refactor: remove the dead copy_files.yaml build-config mechanism

[Jammy2211]

Overview

copy_files.yaml lists script paths that the notebook build copies verbatim into notebooks/ instead of converting to .ipynb. The mechanism is wired into PyAutoHands/autobuild/generate.py, but a deep dive on 2026-07-23 established it is completely dead organism-wide — every config entry is stale or empty, is_copy_file() is unconditionally False, and it has produced zero output in any current build. This task removes the feature outright, mirroring the earlier removals of notebooks_remove.yaml and the autobuild-level env_vars.yaml.

Run mode: --auto, effective autonomy supervised (min(header supervised, refactor cap safe)). Merge and issue-close remain human acts.

Evidence that it is dead

• Every config entry is stale or empty. autofit_workspace lists 3 paths (howtofit/chapter_1_introduction/gaussian.py, profiles.py x2) that exist in no repo — the howtofit content moved to HowToFit/, and they are not there either. autolens_workspace lists imaging/advanced/chaining/hyper_mode/extensions.py; no hyper_mode directory exists at all. The other four workspace files are []. The PyAutoHands fallback keyed dict has 4 keys (howtogalaxy, howtolens, howtofit, BSc_Galaxies_Project), all null — and BSc_Galaxies_Project is not even in the workspace.
• is_copy_file() is therefore unconditionally False.
• Zero output artifacts. No .py file exists in any notebooks/ tree across all 9 build targets (the 6 workspaces + HowToFit/HowToGalaxy/HowToLens). Since generate.py rmtrees notebooks/ and rebuilds it, and copy-as-is is the only path that writes a .py there, this proves the feature produced nothing in any current build.
• No consumer outside generate.py. An exhaustive grep across all repos and file types (incl. .yml/.sh/.toml) found no CI workflow, release pipeline, packaging manifest, or second copy of generate.py. Every remaining hit is a comment, a test, a doc, or a historical PyAutoMind record.
• The HowTo* repos have config/build/ but no copy_files.yaml, so they take the fallback branch — which yields []. The fallback branch, its precedence test, and the "not yet migrated" note in bin/autobuild all guard an empty dict.

Plan

• Delete the copy-as-is notebook-build feature outright; it is provably inert.
• Ship PyAutoHands first, the workspaces second, behind the library-first merge gate (2 PRs).
• Keep copy_to_notebooks() and the <project> CLI arg — both are still load-bearing elsewhere.
• Convert the existing dead-file guard test into the regression that keeps this deleted.
• Defer the PyAutoMind spawn.py half until PyAutoMind PR release.yml: skip PyAutoFit NSS tests in release_test_pypi pytest #93 clears the repo claim.
• Prove behaviour-preservation empirically by diffing generated notebooks/ trees before and after, rather than assuming the removed branch was never taken.

Detailed implementation plan

Affected Repositories

• PyAutoHands (primary)
• PyAutoBrain (one doc line)
• autofit_workspace, autogalaxy_workspace, autolens_workspace
• autofit_workspace_test, autogalaxy_workspace_test, autolens_workspace_test
• PyAutoMind — deferred, see below

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoHands	main	clean
./PyAutoBrain	main	clean
./autofit_workspace	main	clean
./autogalaxy_workspace	main	clean
./autolens_workspace	main	1 untracked dataset.fits (pre-existing, unrelated — not touched)
./autofit_workspace_test	main	clean
./autogalaxy_workspace_test	main	clean
./autolens_workspace_test	main	2 modified dataset/point_source/* (pre-existing, unrelated — not touched)

Suggested branch: feature/remove-dead-copy-files

Implementation Steps

PR 1 — PyAutoHands (+ 1 PyAutoBrain line)

1. autobuild/generate.py:36-45 — delete the copy_files.yaml loader block (the workspace-vs-fallback precedence logic).
2. autobuild/generate.py:48-49 — delete is_copy_file().
3. autobuild/generate.py:142-164 — collapse the if is_copy_file(script_path): ... else: ... branch, keeping only the convert path, de-indented.
4. Delete autobuild/config/copy_files.yaml.
5. bin/autobuild:181-182 — drop the copy_files.yaml rationale from help_generate. Keep the project argument and re-justify it via build_util.inject_colab_setup.
6. docs/internals.md:175,178 — drop the copy_files.yaml bullet and its mention in the fallback paragraph.
7. tests/test_workspace_config_precedence.py — delete test_copy_files_workspace_wins and test_copy_files_falls_back_to_autobuild_keyed; drop "copy_files.yaml" from the tuple in test_actual_workspace_files_exist; add a copy_files.yaml assertion to test_dead_autobuild_files_removed.
8. PyAutoBrain/agents/conductors/build/BUILD_CAPABILITIES.md:49 — {no_run,env_vars,copy_files,visualise_notebooks} -> {no_run,env_vars,visualise_notebooks}.

PR 2 — the 6 workspaces (opened after PR 1 merges)

9. Delete config/build/copy_files.yaml from autofit_workspace, autogalaxy_workspace, autolens_workspace, autofit_workspace_test, autogalaxy_workspace_test, autolens_workspace_test.

Verification

1. pytest tests/test_workspace_config_precedence.py green in PyAutoHands.
2. Behaviour-preservation proof: run autobuild generate autofit in autofit_workspace on main, snapshot the notebooks/ tree (find … | sort | sha256sum), repeat on the branch, diff. Must be byte-identical.
3. Repeat the generate check for one HowTo repo (howtofit), since those exercised the now-deleted fallback path.

Key Files

• PyAutoHands/autobuild/generate.py — the sole consumer; loader + is_copy_file + the dead branch
• PyAutoHands/autobuild/config/copy_files.yaml — the all-null fallback keyed dict
• PyAutoHands/tests/test_workspace_config_precedence.py — 2 tests deleted, 2 assertions adjusted
• PyAutoHands/bin/autobuild — help_generate text only
• PyAutoHands/docs/internals.md — config-file documentation
• <6 workspaces>/config/build/copy_files.yaml — deleted

Guardrails

• Keep the <project> argument to autobuild generate — its help text justifies it via copy_files.yaml, but project is genuinely still used by build_util.inject_colab_setup.
• copy_to_notebooks() stays — the normal conversion path and the .rst/.md sweeps both call it.
• Leave PyAutoMind/complete/** records untouched — historical ledger; several record the original seeding of these configs.

Deferred follow-up (PyAutoMind)

scripts/spawn.py:523 emits autoproject_workspace/config/build/copy_files.yaml into every spawned workspace, and docs/pyautobrain/spawn_spec.md:76 specifies it. Both must go, but PyAutoMind is currently claimed by task arxiv-digest-strong-lensing-term (branch feature/arxiv-digest-strong-lensing-term, PR #93 open). File as a small follow-up once #93 merges. Until then spawn keeps seeding a config nothing reads — harmless once the reader is gone.

Phasing note — Brain override

The Feature Agent scored this too-large (25) and proposed a 4-phase split (design / core-api / workspace / docs). Rejected by the user and by inspection: the score is driven by the 9-repo count, not complexity — the change is ~13 lines removed from one file plus 7 deletions and 4 doc/test edits. The flagged "public-API ripple" risk does not apply: is_copy_file is module-local to generate.py and its branch is provably never taken. Structured instead as the standard library-first 2-PR split.

Original Prompt

Click to expand starting prompt

Check if the copy_files.yaml config/build file is actually used anywhere (it may be in autofit_workspace, but this may also be legacy) and if not remove it or consider if we can remove it somewhere.

Do B, but do one file deep dive to make sure its used nowhere

("B" = full removal of the feature, as opposed to "A" = purging only the stale entries while keeping the mechanism.)

[Jammy2211]

Scope expanded — the sweep found 2 siblings, one of them a live crash

Following a request to extend the sweep to the HowTo* and _test workspaces, I audited every config/build/*.yaml across all build targets. Two files share copy_files.yaml's legacy-fallback pattern and are being folded into this task.

Correction to the plan above: there are 10 build targets, not 9. autobuild/config/workspaces.yaml also lists euclid -> euclid_strong_lens_modeling_pipeline. It has no notebooks/ and no copy_files.yaml, so it falls back to the keyed dict, resolves None or [], and stays inert — the copy_files conclusion is unchanged.

Sibling A — autobuild/config/visualise_notebooks.yaml crashes --visualise

The file is comment-only — it has no keys at all. run.py:63-64 does yaml.safe_load(f).get(project), and safe_load of a comment-only file returns None. Reproduced against the real repo:

HowToLens has own visualise_notebooks.yaml? False
AttributeError: 'NoneType' object has no attribute 'get'

autobuild run <target> --visualise therefore crashes for the 4 targets with no visualise_notebooks.yaml of their own: HowToFit, HowToGalaxy, HowToLens, euclid. (copy_files used .get(...) or [] and survived; this call site omits the or [] after the .get, so the husk file is what kills it.)

Sibling B — autobuild/config/no_run.yaml is an unreachable trap

run.py:42-44 reads the fallback only when the workspace file is missing — and all 10 targets have their own config/build/no_run.yaml (HowToFit's is empty, but it exists). The fallback is never read, yet holds ~50 lines of stale duplicated skip rules for autofit/autogalaxy/autolens/autofit_test/howtolens. Editing it to skip a script silently does nothing. This is also the file behind the known test_parse_no_run_reasons GetDist vs get_dist mismatch.

Revised scope

Delete all three fallbacks and their branches, making workspace-owned config/build/ the single source of truth:

File	Verdict
autobuild/config/copy_files.yaml	DELETE — inert
autobuild/config/visualise_notebooks.yaml	DELETE — crashes --visualise
autobuild/config/no_run.yaml	DELETE — unreachable, stale
autobuild/config/workspaces.yaml	KEEP — live build policy

Code changes: generate.py loses the copy_files loader + is_copy_file + its branch; run.py loses the no_run fallback branch (and with it the isinstance(dict) keyed-dict path, since only flat lists remain) and the visualise fallback branch.

Behaviour after removal:

• Missing no_run.yaml — raise a clear, actionable error naming the workspace rather than falling back silently (all 10 targets have one, so this never fires today).
• Missing visualise_notebooks.yaml — resolve to [] with an explanatory message instead of AttributeError. This is what fixes the crash.

Also folded in: the PyAutoMind piece

PyAutoMind PR #93 merged mid-run, releasing the repo claim that forced the deferral. scripts/spawn.py:523 (stops emitting copy_files.yaml into spawned workspaces) and docs/pyautobrain/spawn_spec.md:76 are back in scope for this task.

Run mode remains --auto at effective autonomy supervised; merge and issue-close stay human.

[Jammy2211]

Shipped — 9 PRs open, awaiting merge

All work is implemented, verified and pushed. Merge and issue-close remain human acts (--auto, effective autonomy supervised).

Merge order — PyAutoHands#176 first

Wave 1 (library + docs)

• refactor: remove the dead autobuild config fallbacks #176 — the change itself
• docs: drop copy_files from the per-workspace build-config list PyAutoBrain#149 — doc line
• refactor: spawn emits no_run.yaml, not copy_files.yaml PyAutoMind#94 — spawn.py seeds no_run.yaml instead

Wave 2 (workspace config deletions, after #176)

• chore: delete dead config/build/copy_files.yaml autofit_workspace#108
• chore: delete dead config/build/copy_files.yaml autogalaxy_workspace#145
• chore: delete dead config/build/copy_files.yaml autolens_workspace#316
• chore: delete dead config/build/copy_files.yaml autofit_workspace_test#65
• chore: delete dead config/build/copy_files.yaml autogalaxy_workspace_test#80
• chore: delete dead config/build/copy_files.yaml autolens_workspace_test#199

Verification

• 132/132 PyAutoHands tests pass (8/8 in the rewritten precedence file).
• Behaviour-preservation proved empirically: regenerated autofit_workspace on the branch and diffed against main — byte-identical (39 files; only the config deletion shows in git status). Repeated for HowToLens, a target that actually used the deleted fallback — also byte-identical.
• --visualise crash reproduced on main (AttributeError) and confirmed fixed (EXIT=0).
• ruff clean; yaml / os.path / CONFIG_PATH / AUTOBUILD_CONFIG removed once unused.
• One pre-existing, unrelated PyAutoBrain failure: test_every_public_agent_has_a_skill_wrapper (missing sizing wrapper) — confirmed failing on main before any edit here.

Net effect

generate.py −46 lines, run.py +3, 60 lines of dead config deleted, 7 files removed. config/build/ is now the single source of truth; autobuild/config/ holds only workspaces.yaml (build policy).

Note for whoever merges

The new test_copy_files_mechanism_fully_removed asserts no workspace still carries a copy_files.yaml. In PyAutoHands CI this passes trivially (sibling repos aren't checked out); in a full local checkout it stays red until the wave-2 PRs merge. That is the intended signal, not a regression.

[Jammy2211]

All 9 PRs merged. autobuild/config/ now holds only workspaces.yaml; each workspace's config/build/ is the single source of truth.

• PyAutoHands#176 (the change), PyAutoBrain#149 (doc), PyAutoMind#94 (spawn seeds no_run.yaml)
• autofit_workspace#108, autogalaxy_workspace#145, autolens_workspace#316, autofit_workspace_test#65, autogalaxy_workspace_test#80, autolens_workspace_test#199

Verified before merge: 132/132 tests; autofit_workspace and HowToLens both regenerate byte-identical notebooks/ trees vs main; the --visualise AttributeError on HowToFit/HowToGalaxy/HowToLens/euclid is fixed.

Closing.