Overview
Workspace example scripts end with a docstring block containing only Finished. or Finish. — 166 of them across 10 repos. This was a crutch against notebook generation "cutting off weird" when a script's last cell wasn't a docstring, and it renders as a pointless final markdown cell in every generated notebook and published markdown page.
Investigating the crutch split the claim in two. The shape it guards against is already safe — deleting the trailing block from real scripts and running the actual add_notebook_quotes → ipynb-py-convert chain gives a complete, unmangled final code cell. But there is a live "cuts off weird" bug in the same family, and it is shipped: a column-0 docstring opened on the line immediately after a non-blank code line never becomes its own markdown cell, so the # %% marker and both ''' delimiters land inside the preceding code cell as literal text. 13 committed notebook code cells are SyntaxError today.
So: harden the generator and pin it with tests, remove all 166 occurrences, and the 13 broken cells get repaired by regeneration.
Plan
- Fix
add_notebook_quotes so a docstring following code still splits into its own markdown cell, and add regression tests — including one pinning that a script ending in a code cell converts correctly, which is what makes the Finish. hack provably unnecessary.
- Merge the PyAutoHands fix first; workspaces invoke
generate.py from the local checkout, so regeneration must run against the fixed generator.
- Sweep the 166 occurrences across 10 repos, one PR per repo, handling each of the five distinct shapes explicitly rather than with a blanket regex.
- Regenerate
notebooks/, llms-full.txt and workspace_index.json in the six artifact-bearing repos; verify no notebook code cell contains a literal # %%.
- Delete the stale paragraph in place in the five curated
markdown/ pages rather than re-rendering them, to avoid churning every figure PNG for a one-paragraph deletion.
Detailed implementation plan
Work Classification
Both — PyAutoHands (library) first, then the workspace sweep.
Worktree root
~/Code/PyAutoLabs-wt/remove-finish-docstring-hack/
Affected Repositories
- PyAutoHands (primary)
- autolens_workspace, autogalaxy_workspace, autofit_workspace, autocti_workspace
- autolens_workspace_test, autofit_workspace_test, autocti_workspace_test
- autolens_workspace_developer
- HowToFit, HowToLens
- euclid_strong_lens_modeling_pipeline
Branch Survey
| Repository |
Current Branch |
Dirty? |
| ./PyAutoHands |
main |
clean |
| ./autolens_workspace |
main |
clean |
| ./autogalaxy_workspace |
main |
clean |
| ./autofit_workspace |
main |
clean |
| ./autocti_workspace |
main |
clean |
| ./autolens_workspace_test |
main |
clean |
| ./autofit_workspace_test |
main |
clean |
| ./autocti_workspace_test |
main |
clean |
| ./autolens_workspace_developer |
main |
clean |
| ./HowToFit |
main |
clean |
| ./HowToLens |
main |
clean |
| ./euclid_strong_lens_modeling_pipeline |
main |
1 changed (test_report.md, pre-existing, unrelated) |
Suggested branch: feature/remove-finish-docstring-hack
Claim contention (hand-read — worktree_check_conflict has never fired)
autolens_workspace — claimed by THREE active tasks: extra-galaxies-point-source (creates only new files), likelihood-function-jax-pointer (5 likelihood_function.py / using_jax.py files, all with zero occurrences), multi-galaxy-imaging-parity (uncommitted edits to scripts/multi_galaxy/{modeling,simulator,start_here}.py).
autogalaxy_workspace — claimed by likelihood-function-jax-pointer (3 files, zero occurrences).
autolens_workspace_test — claimed by vacuous-jax-assertions (status awaiting-merge).
PyAutoHands — claimed by python-312-release-surfaces, live with uncommitted work in .gitignore, bin/autohands, docs/internals.md, pre_build.sh and the run_logs/ + =3.12 cleanup. Zero overlap with the two files this task edits.
Human decision 2026-07-30: proceed in parallel with one carve-out. The only script-level overlap workspace-wide is autolens_workspace/scripts/multi_galaxy/simulator.py (one Finished. line, uncommitted in multi-galaxy-imaging-parity) — skip that file and leave it to that task. Generated artifacts (notebooks/, llms-full.txt, workspace_index.json) collide as usual; whichever PR merges last re-runs generate.py.
Root cause
Generation is build_util.py_to_notebook → autohands/add_notebook_quotes.py → ipynb-py-convert's py2nb. py2nb splits the intermediate .py on the literal '\n\n# %%\n'. The docstring-opener branch (add_notebook_quotes.py:133) emits:
out.extend(["# %%", "\n", "'''\n"])
yielding ...code\n# %%\n'''\n — a single newline before the marker, where py2nb needs \n\n# %%\n. The split never fires. (The closing path is fine: it emits "'''", "\n\n" first, so the following code boundary is always separated.)
Shipped example — autolens_workspace/notebooks/interferometer/features/pixelization/many_visibilities_preparation.ipynb cell 13 ends:
np.save(
file=dataset_path / f"nufft_precision_operator_{mask_radius}.npy",
arr=nufft_precision_operator,
allow_pickle=False,
)
# %%
'''
To load the `nufft_precision_operator` matrix from hard-disk in your model-fit, you can use the code:
'''
All 13 mangled cells: autolens_workspace (guides/hpc/example_cpu_and_gpu ×2, interferometer/features/pixelization/many_visibilities_preparation, interferometer/features/subhalo/simulator), autogalaxy_workspace (guides/hpc/example_cpu_and_gpu ×2, interferometer/features/pixelization/many_visibilities_preparation), autocti_workspace (imaging_ci/modeling/features/{cosmic_rays,non_uniform,serial_cti,visualize_full}), HowToLens (chapter_2_lens_modeling/tutorial_4_dealing_with_failure, simulator/source_complex).
Evidence the hack is obsolete
Deleting the trailing block and running the real conversion chain:
| script |
with block |
without block |
autolens_workspace/scripts/imaging/features/no_lens_light/slam.py |
23 cells, last = markdown Finish. |
22 cells, last = code, complete |
autolens_workspace/scripts/imaging/features/pixelization/source_science.py |
41 cells, last = markdown Finish. |
40 cells, last = code, complete |
Same for every tail shape constructed (trailing newline / none / trailing blank lines / trailing comment).
Implementation Steps
1. PyAutoHands/autohands/add_notebook_quotes.py — opener branch of add_notebook_quotes, emit the separating blank line when missing:
if pending_code_boundary:
out.extend(pending_lines)
pending_lines = []
pending_code_boundary = False
if out and not "".join(out[-3:]).endswith("\n\n"):
out.append("\n")
out.extend(["# %%", "\n", "'''\n"])
Two load-bearing constraints:
- Only when
out is non-empty. py2nb strips a leading # %%\n header; a leading \n defeats that strip and yields a spurious empty first code cell.
- Only when not already blank-terminated. Unconditional emission appends a trailing blank line to every code cell and churns every notebook in every workspace. The adjacent-docstring path already ends
\n\n and must stay untouched — test_adjacent_docstrings_generate_separate_markdown_cells and test_adjacent_docstrings_do_not_emit_an_empty_code_cell_boundary must keep passing unchanged.
2. PyAutoHands/tests/test_add_notebook_quotes.py — three new tests: docstring-immediately-after-code yields a separate markdown cell with no cell source containing # %% or '''; a script ending in a code cell yields a complete final code cell; a script opening with a docstring still yields exactly one leading markdown cell and no empty first code cell.
Merge gate: step 1 must land on PyAutoHands main before any workspace regeneration.
3. Sweep the 166 occurrences. Five shapes — a blanket regex over all of them is wrong:
| shape |
count |
action |
sole-content trailing block at EOF — """\nFinish.\n""" |
126 |
delete block + the blank line before it; file ends on real content with a single trailing newline |
a line inside a block that continues (__Env__, __JAX Variant__, …) |
33 |
delete the line + the blank line after it |
Finished. leading a real sentence |
2 |
drop the word only |
__Finish__ — empty trailing section header |
2 |
delete the header line |
| the same block indented in a function body, or commented out |
5 |
delete; never converted to a cell, pure dead cruft |
On the __Env__ cases: the artifact-visible markdown cell today is exactly Finish., because strip_env_declarations removes the __Env__ section. After removal the block holds only __Env__ and the existing standalone-fallback path drops it whole — correct.
Per-repo, one PR each:
| repo |
occurrences |
regenerate |
| autolens_workspace |
61 (minus 1 carve-out) |
notebooks, llms-full.txt, workspace_index.json |
| autocti_workspace |
35 |
notebooks |
| autocti_workspace_test |
27 |
— |
| autogalaxy_workspace |
21 |
notebooks, llms-full.txt, workspace_index.json |
| autofit_workspace |
11 |
notebooks, llms-full.txt, workspace_index.json |
| autofit_workspace_test |
5 |
— |
| euclid_strong_lens_modeling_pipeline |
3 |
— |
| autolens_workspace_test |
1 |
— |
| autolens_workspace_developer |
1 |
— |
| HowToFit |
1 |
notebooks, llms-full.txt, workspace_index.json |
| HowToLens |
0 |
notebooks — repairs the 2 mangled cells |
4. Generated artifacts. python ../PyAutoHands/autohands/generate.py <project> per artifact-bearing repo, run from the workspace root, committed from the same run so the catalogue cannot drift (conversion only, no script execution). The _test / _developer repos and euclid_strong_lens_modeling_pipeline have no generated artifacts.
5. markdown/ — delete the paragraph in place in the five pages carrying a real one: autolens_workspace/markdown/{point_source/simulator,group/simulator,interferometer/simulator}.md, autogalaxy_workspace/markdown/interferometer/fit.md, autofit_workspace/markdown/overview/overview_1_the_basics.md. Do not re-run generate_markdown.py: it re-executes real model fits and re-quantizes every figure PNG, churning large binaries for a one-paragraph deletion. Most Finish hits under markdown/ are Nautilus status tables (Finished | 18 | 1 | …) and must be left alone. Human-approved deviation from the pages' "do not edit it directly" header.
Key Files
PyAutoHands/autohands/add_notebook_quotes.py — the fix (opener branch of add_notebook_quotes)
PyAutoHands/tests/test_add_notebook_quotes.py — three new regression tests
PyAutoHands/autohands/build_util.py (py_to_notebook) — the caller; unchanged
- 166 workspace example scripts across 10 repos — the sweep
- 5 curated
markdown/*.md pages — in-place paragraph deletion
Verification
pytest PyAutoHands/tests/test_add_notebook_quotes.py green, including the two pre-existing adjacent-docstring tests.
generate.py exits clean in each of the six artifact-bearing repos.
- No generated notebook contains a code cell with a literal
# %% — asserted across every repo's notebooks/ tree. Currently fails at 13 cells; must pass afterwards. Primary proof the fix works.
- Every notebook whose script lost a trailing block has exactly one fewer cell, with a complete code cell last.
grep -rn "Finish" scripts/ *.py per repo returns only genuine prose.
- Curated smoke tests for the repos that have them (the small curated subset, not a full run).
- Pre-flight
git diff --stat before each ship_workspace commit, to catch binary/output leakage.
Notes
- Brain override, recorded: the Feature Agent returned too-large (score 29) and a generic
design / core_api / workspace_examples / docs split off its repo-count proxy. design and core_api are vacuous (no library API is touched; the design is settled above) and docs is empty — the convention is documented nowhere (AGENTS.md, CONTRIBUTING.md, PyAutoHands/docs/ and the Brain skills were all checked). Overridden to one PR per repo behind a single PyAutoHands-first gate.
- Filed separately, not fixed here: a column-0 closing
""" belonging to a triple-quoted string literal in code toggles docstring state and mangles the cell the same way. One occurrence workspace-wide — autolens_workspace_test/gallery/gallery_build.py:42 — outside scripts/, so never converted. A real fix needs tokenization, not a line-prefix test.
Original Prompt
Click to expand starting prompt
Remove the Finished. / Finish. trailing-docstring hack from every workspace
Type: maintenance
Target: workspaces
Repos:
- PyAutoHands
- autolens_workspace
- autogalaxy_workspace
- autofit_workspace
- autocti_workspace
- autolens_workspace_test
- autofit_workspace_test
- autocti_workspace_test
- autolens_workspace_developer
- HowToFit
- euclid_strong_lens_modeling_pipeline
Difficulty: medium
Autonomy: supervised
Priority: medium
Original request (verbatim)
Lots of workspace examples end with a cell Finished., or have the word near
the end. This was a hack becuase notebook generation would cut off weird if
the last bit wasnt a docstring in the Python cell. can you make sure generate
does not do this anymore, than remove all of these Finished. statements
thoruhgout all workspaces.
Investigation — what generation actually does today
Notebook generation is build_util.py_to_notebook →
autohands/add_notebook_quotes.py → ipynb-py-convert's py2nb. py2nb
splits the intermediate .py on the literal '\n\n# %%\n' and treats a chunk
starting with ''' / """ as a markdown cell.
Finding 1 — the crutch is already obsolete. A script ending in a code
cell converts correctly. Verified end-to-end on two real scripts by deleting
their trailing """\nFinish.\n""" block and re-running the real
add_notebook_quotes + ipynb-py-convert chain:
| script |
with block |
without block |
autolens_workspace/scripts/imaging/features/no_lens_light/slam.py |
23 cells, last = markdown Finish. |
22 cells, last = code, complete and unmangled |
autolens_workspace/scripts/imaging/features/pixelization/source_science.py |
41 cells, last = markdown Finish. |
40 cells, last = code, complete and unmangled |
Also verified against every plausible tail shape (trailing newline / no trailing
newline / trailing blank lines / trailing comment): all four produce an
identical, complete final code cell. So there is nothing to fix for the shape
the hack was written for — but it must be pinned by a regression test so the
crutch can never be re-justified.
Finding 2 — there IS a live "cuts off weird" bug, in the same family. A
column-0 docstring opened on the line immediately after a non-blank code line
(no blank line between) is never split into a markdown cell. On a docstring
opener add_notebook_quotes emits (add_notebook_quotes.py:133):
out.extend(["# %%", "\n", "'''\n"])
which yields ...code\n# %%\n'''\n — a single newline before the marker,
where py2nb needs \n\n# %%\n. The split never happens, so the marker and
both ''' delimiters land inside the preceding code cell as literal text.
(The closing path is fine: it emits "'''", "\n\n" first, so the following
code boundary is always correctly separated.)
Reproduced minimally, and shipped in a committed notebook today —
autolens_workspace/notebooks/interferometer/features/pixelization/many_visibilities_preparation.ipynb
has a code cell ending:
np.save(
file=dataset_path / f"nufft_precision_operator_{mask_radius}.npy",
arr=nufft_precision_operator,
allow_pickle=False,
)
# %%
'''
To load the `nufft_precision_operator` matrix from hard-disk in your model-fit, you can use the code:
'''
That cell is a SyntaxError if a user runs it. Source is
scripts/interferometer/features/pixelization/many_visibilities_preparation.py:209
— a """ with no blank line above it.
A scan of the converted-script surface found this shape at 49 sites; after
excluding non-converted paths (.github/scripts/, gallery/, module docstrings
after a shebang) the live converted-script sites are:
autolens_workspace/scripts/interferometer/features/pixelization/many_visibilities_preparation.py:209
autolens_workspace/scripts/interferometer/features/subhalo/simulator.py:117
autogalaxy_workspace/scripts/interferometer/features/pixelization/many_visibilities_preparation.py:193
autocti_workspace/scripts/imaging_ci/modeling/features/{cosmic_rays,serial_cti,visualize_full,non_uniform}.py
HowToLens/scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py:429,
HowToLens/scripts/simulator/source_complex.py:71
- plus
autocti_workspace_test/legacy/** (not converted; ignore)
Fix (PyAutoHands)
In the opener branch of add_notebook_quotes, emit the separating blank line
py2nb requires when it is missing:
if pending_code_boundary:
out.extend(pending_lines)
pending_lines = []
pending_code_boundary = False
if out and not "".join(out[-3:]).endswith("\n\n"):
out.append("\n")
out.extend(["# %%", "\n", "'''\n"])
Two constraints the implementation must respect:
- Only when
out is non-empty. py2nb strips a leading # %%\n
header; a leading \n would defeat that strip and produce a spurious empty
first code cell.
- Only when not already blank-terminated. Emitting the newline
unconditionally appends a trailing blank line to every code cell, churning
every generated notebook in every workspace for no reason. The adjacent-
docstring path already ends with \n\n and must stay untouched (its
regression test test_adjacent_docstrings_generate_separate_markdown_cells
must keep passing).
Tests to add
- a script whose docstring follows code with no blank line yields a separate
markdown cell, and no cell source contains # %% or ''';
- a script ending in a code cell yields a complete final code cell (pins
Finding 1 — the reason the Finish. hack is unnecessary);
- a script starting with a docstring still yields exactly one leading markdown
cell and no empty first code cell.
Out of scope (file separately)
A column-0 closing """ of a triple-quoted string literal in code toggles
docstring state and mangles the cell the same way. One occurrence workspace-wide
— autolens_workspace_test/gallery/gallery_build.py:42 — which sits outside
scripts/ and is therefore never converted. Fixing it needs real tokenization,
not a line-prefix test.
Removal census — 166 occurrences, 166 files, 10 repos
| repo |
occurrences |
| autolens_workspace |
61 |
| autocti_workspace |
35 |
| autocti_workspace_test |
27 |
| autogalaxy_workspace |
21 |
| autofit_workspace |
11 |
| autofit_workspace_test |
5 |
| euclid_strong_lens_modeling_pipeline |
3 |
| autolens_workspace_test |
1 |
| autolens_workspace_developer |
1 |
| HowToFit |
1 |
Five shapes, each needing different handling — a blanket regex over all of them
is wrong:
- 126 × sole-content trailing block at EOF —
"""\nFinish.\n""" / """\nFinished.\n""" as the entire final docstring.
Delete the block and the blank line before it, leaving the file ending on
real content with a single trailing newline.
- 33 × a line inside a block that continues — the block goes on with
__Env__ (developer-only, stripped from artifacts anyway), __JAX Variant__,
or similar. Delete the Finish. line and the blank line after it. Note that
for the __Env__ cases the artifact-visible markdown cell today is exactly
Finish. and nothing else, because strip_env_declarations removes the
__Env__ section; after removal the block holds only __Env__ and the
existing standalone-fallback path drops the whole block — which is correct.
- 2 ×
Finished. leading a real sentence — drop only the word:
autolens_workspace/scripts/cluster/lenstool/modeling.py:503
("Finished. The README in this folder is the narrative companion…")
autolens_workspace/scripts/guides/point_source_pairing.py:182
("Finished. For the production-scale picture — real solver, …")
- 2 ×
__Finish__ — an empty trailing section header (no content follows
it) in {autolens,autogalaxy}_workspace/scripts/imaging/features/multi_gaussian_expansion/modeling.py.
Delete the header line.
- 5 × indented or commented variants — the same block indented inside a
function body, or inside a fully commented-out block. Not converted to
markdown cells (only column-0 delimiters are), so pure dead cruft:
autolens_workspace/scripts/interferometer/features/subhalo/sensitivity/start_here.py:305,
autocti_workspace_test/imaging_ci/profiling/pruning/{parallel_x1,parallel_x3,serial_x1}.py,
autolens_workspace_developer/slam_pipeline/dspl.py:304.
The convention is documented nowhere (AGENTS.md, CONTRIBUTING.md,
PyAutoHands/docs/, the Brain skills were all checked) — so no doc updates are
needed and nothing is authorising new ones.
Generated artifacts
notebooks/, llms-full.txt, workspace_index.json — regenerate with
generate.py in autolens_workspace, autogalaxy_workspace,
autofit_workspace, autocti_workspace and HowToFit. Conversion only, no
script execution — cheap. (llms-full.txt / workspace_index.json contain no
Finish today; they still get rewritten and must be committed from the same
run so the catalogue cannot drift.) The _test / _developer repos and
euclid_strong_lens_modeling_pipeline have no generated artifacts at all.
markdown/ — do not re-run generate_markdown.py. Most Finish
hits in markdown/ are Nautilus sampler status tables (Finished | 18 | 1 | …), not the hack. Only five pages carry a real one:
autolens_workspace/markdown/{point_source/simulator,group/simulator,interferometer/simulator}.md,
autogalaxy_workspace/markdown/interferometer/fit.md,
autofit_workspace/markdown/overview/overview_1_the_basics.md.
A re-render executes real model fits and re-encodes every figure PNG, so it
would churn large binaries for a one-paragraph deletion
([[feedback_ship_workspace_binary_leak]]). Delete the paragraph in place and
say so in the PR.
Validation
pytest PyAutoHands/tests/test_add_notebook_quotes.py green, including the
pre-existing adjacent-docstring tests.
generate.py runs clean in each of the five artifact-bearing repos.
- No generated notebook contains a code cell with a literal
# %% or '''
(this currently fails on many_visibilities_preparation.ipynb and must pass
after the fix) — assert it across the whole notebooks/ tree.
grep -rn "Finish" scripts/ *.py returns nothing but genuine prose in each
swept repo.
- Every notebook whose script lost a trailing block has exactly one fewer cell
than before, with the final cell a complete code cell.
- Curated smoke tests pass for the repos that have them
([[feedback_smoke_tests_small_subset]], [[feedback_two_env_profiles_smoke_vs_release]]).
Claim contention (hand-read — worktree_check_conflict never fires,
[[feedback_worktree_conflict_guard_never_fires]])
autolens_workspace — claimed by THREE active tasks:
extra-galaxies-point-source, likelihood-function-jax-pointer,
multi-galaxy-imaging-parity. This task edits 61 files across the whole repo
and regenerates every notebook, so unlike the previous parallel decisions
there is real overlap risk with all three.
autogalaxy_workspace — claimed by likelihood-function-jax-pointer.
autolens_workspace_test — claimed by vacuous-jax-assertions.
PyAutoHands — claimed by python-312-release-surfaces, live with
uncommitted work in .gitignore, bin/autohands, docs/internals.md,
pre_build.sh and the run_logs/ + =3.12 cleanup. Zero overlap with
autohands/add_notebook_quotes.py / tests/test_add_notebook_quotes.py, so
parallel is safe here on the same precedent as prior decisions.
- Uncontended:
autofit_workspace, autocti_workspace,
autocti_workspace_test, autofit_workspace_test,
autolens_workspace_developer, HowToFit,
euclid_strong_lens_modeling_pipeline.
Suggested phasing follows from that, not from repo count
([[feedback_brain_repo_count_difficulty_proxy]]): PyAutoHands fix first, then
the uncontended sweep, then the contended repos once their branches land.
Overview
Workspace example scripts end with a docstring block containing only
Finished.orFinish.— 166 of them across 10 repos. This was a crutch against notebook generation "cutting off weird" when a script's last cell wasn't a docstring, and it renders as a pointless final markdown cell in every generated notebook and published markdown page.Investigating the crutch split the claim in two. The shape it guards against is already safe — deleting the trailing block from real scripts and running the actual
add_notebook_quotes→ipynb-py-convertchain gives a complete, unmangled final code cell. But there is a live "cuts off weird" bug in the same family, and it is shipped: a column-0 docstring opened on the line immediately after a non-blank code line never becomes its own markdown cell, so the# %%marker and both'''delimiters land inside the preceding code cell as literal text. 13 committed notebook code cells areSyntaxErrortoday.So: harden the generator and pin it with tests, remove all 166 occurrences, and the 13 broken cells get repaired by regeneration.
Plan
add_notebook_quotesso a docstring following code still splits into its own markdown cell, and add regression tests — including one pinning that a script ending in a code cell converts correctly, which is what makes theFinish.hack provably unnecessary.generate.pyfrom the local checkout, so regeneration must run against the fixed generator.notebooks/,llms-full.txtandworkspace_index.jsonin the six artifact-bearing repos; verify no notebook code cell contains a literal# %%.markdown/pages rather than re-rendering them, to avoid churning every figure PNG for a one-paragraph deletion.Detailed implementation plan
Work Classification
Both — PyAutoHands (library) first, then the workspace sweep.
Worktree root
~/Code/PyAutoLabs-wt/remove-finish-docstring-hack/Affected Repositories
Branch Survey
test_report.md, pre-existing, unrelated)Suggested branch:
feature/remove-finish-docstring-hackClaim contention (hand-read —
worktree_check_conflicthas never fired)autolens_workspace— claimed by THREE active tasks:extra-galaxies-point-source(creates only new files),likelihood-function-jax-pointer(5likelihood_function.py/using_jax.pyfiles, all with zero occurrences),multi-galaxy-imaging-parity(uncommitted edits toscripts/multi_galaxy/{modeling,simulator,start_here}.py).autogalaxy_workspace— claimed bylikelihood-function-jax-pointer(3 files, zero occurrences).autolens_workspace_test— claimed byvacuous-jax-assertions(statusawaiting-merge).PyAutoHands— claimed bypython-312-release-surfaces, live with uncommitted work in.gitignore,bin/autohands,docs/internals.md,pre_build.shand therun_logs/+=3.12cleanup. Zero overlap with the two files this task edits.Human decision 2026-07-30: proceed in parallel with one carve-out. The only script-level overlap workspace-wide is
autolens_workspace/scripts/multi_galaxy/simulator.py(oneFinished.line, uncommitted inmulti-galaxy-imaging-parity) — skip that file and leave it to that task. Generated artifacts (notebooks/,llms-full.txt,workspace_index.json) collide as usual; whichever PR merges last re-runsgenerate.py.Root cause
Generation is
build_util.py_to_notebook→autohands/add_notebook_quotes.py→ipynb-py-convert'spy2nb.py2nbsplits the intermediate.pyon the literal'\n\n# %%\n'. The docstring-opener branch (add_notebook_quotes.py:133) emits:yielding
...code\n# %%\n'''\n— a single newline before the marker, wherepy2nbneeds\n\n# %%\n. The split never fires. (The closing path is fine: it emits"'''", "\n\n"first, so the following code boundary is always separated.)Shipped example —
autolens_workspace/notebooks/interferometer/features/pixelization/many_visibilities_preparation.ipynbcell 13 ends:All 13 mangled cells:
autolens_workspace(guides/hpc/example_cpu_and_gpu×2,interferometer/features/pixelization/many_visibilities_preparation,interferometer/features/subhalo/simulator),autogalaxy_workspace(guides/hpc/example_cpu_and_gpu×2,interferometer/features/pixelization/many_visibilities_preparation),autocti_workspace(imaging_ci/modeling/features/{cosmic_rays,non_uniform,serial_cti,visualize_full}),HowToLens(chapter_2_lens_modeling/tutorial_4_dealing_with_failure,simulator/source_complex).Evidence the hack is obsolete
Deleting the trailing block and running the real conversion chain:
autolens_workspace/scripts/imaging/features/no_lens_light/slam.pyFinish.autolens_workspace/scripts/imaging/features/pixelization/source_science.pyFinish.Same for every tail shape constructed (trailing newline / none / trailing blank lines / trailing comment).
Implementation Steps
1.
PyAutoHands/autohands/add_notebook_quotes.py— opener branch ofadd_notebook_quotes, emit the separating blank line when missing:Two load-bearing constraints:
outis non-empty.py2nbstrips a leading# %%\nheader; a leading\ndefeats that strip and yields a spurious empty first code cell.\n\nand must stay untouched —test_adjacent_docstrings_generate_separate_markdown_cellsandtest_adjacent_docstrings_do_not_emit_an_empty_code_cell_boundarymust keep passing unchanged.2.
PyAutoHands/tests/test_add_notebook_quotes.py— three new tests: docstring-immediately-after-code yields a separate markdown cell with no cell source containing# %%or'''; a script ending in a code cell yields a complete final code cell; a script opening with a docstring still yields exactly one leading markdown cell and no empty first code cell.Merge gate: step 1 must land on
PyAutoHandsmainbefore any workspace regeneration.3. Sweep the 166 occurrences. Five shapes — a blanket regex over all of them is wrong:
"""\nFinish.\n"""__Env__,__JAX Variant__, …)Finished.leading a real sentence__Finish__— empty trailing section headerOn the
__Env__cases: the artifact-visible markdown cell today is exactlyFinish., becausestrip_env_declarationsremoves the__Env__section. After removal the block holds only__Env__and the existing standalone-fallback path drops it whole — correct.Per-repo, one PR each:
4. Generated artifacts.
python ../PyAutoHands/autohands/generate.py <project>per artifact-bearing repo, run from the workspace root, committed from the same run so the catalogue cannot drift (conversion only, no script execution). The_test/_developerrepos andeuclid_strong_lens_modeling_pipelinehave no generated artifacts.5.
markdown/— delete the paragraph in place in the five pages carrying a real one:autolens_workspace/markdown/{point_source/simulator,group/simulator,interferometer/simulator}.md,autogalaxy_workspace/markdown/interferometer/fit.md,autofit_workspace/markdown/overview/overview_1_the_basics.md. Do not re-rungenerate_markdown.py: it re-executes real model fits and re-quantizes every figure PNG, churning large binaries for a one-paragraph deletion. MostFinishhits undermarkdown/are Nautilus status tables (Finished | 18 | 1 | …) and must be left alone. Human-approved deviation from the pages' "do not edit it directly" header.Key Files
PyAutoHands/autohands/add_notebook_quotes.py— the fix (opener branch ofadd_notebook_quotes)PyAutoHands/tests/test_add_notebook_quotes.py— three new regression testsPyAutoHands/autohands/build_util.py(py_to_notebook) — the caller; unchangedmarkdown/*.mdpages — in-place paragraph deletionVerification
pytest PyAutoHands/tests/test_add_notebook_quotes.pygreen, including the two pre-existing adjacent-docstring tests.generate.pyexits clean in each of the six artifact-bearing repos.# %%— asserted across every repo'snotebooks/tree. Currently fails at 13 cells; must pass afterwards. Primary proof the fix works.grep -rn "Finish" scripts/ *.pyper repo returns only genuine prose.git diff --statbefore eachship_workspacecommit, to catch binary/output leakage.Notes
design / core_api / workspace_examples / docssplit off its repo-count proxy.designandcore_apiare vacuous (no library API is touched; the design is settled above) anddocsis empty — the convention is documented nowhere (AGENTS.md,CONTRIBUTING.md,PyAutoHands/docs/and the Brain skills were all checked). Overridden to one PR per repo behind a single PyAutoHands-first gate."""belonging to a triple-quoted string literal in code toggles docstring state and mangles the cell the same way. One occurrence workspace-wide —autolens_workspace_test/gallery/gallery_build.py:42— outsidescripts/, so never converted. A real fix needs tokenization, not a line-prefix test.Original Prompt
Click to expand starting prompt
Remove the
Finished./Finish.trailing-docstring hack from every workspaceType: maintenance
Target: workspaces
Repos:
Difficulty: medium
Autonomy: supervised
Priority: medium
Original request (verbatim)
Investigation — what generation actually does today
Notebook generation is
build_util.py_to_notebook→autohands/add_notebook_quotes.py→ipynb-py-convert'spy2nb.py2nbsplits the intermediate
.pyon the literal'\n\n# %%\n'and treats a chunkstarting with
'''/"""as a markdown cell.Finding 1 — the crutch is already obsolete. A script ending in a code
cell converts correctly. Verified end-to-end on two real scripts by deleting
their trailing
"""\nFinish.\n"""block and re-running the realadd_notebook_quotes+ipynb-py-convertchain:autolens_workspace/scripts/imaging/features/no_lens_light/slam.pyFinish.autolens_workspace/scripts/imaging/features/pixelization/source_science.pyFinish.Also verified against every plausible tail shape (trailing newline / no trailing
newline / trailing blank lines / trailing comment): all four produce an
identical, complete final code cell. So there is nothing to fix for the shape
the hack was written for — but it must be pinned by a regression test so the
crutch can never be re-justified.
Finding 2 — there IS a live "cuts off weird" bug, in the same family. A
column-0 docstring opened on the line immediately after a non-blank code line
(no blank line between) is never split into a markdown cell. On a docstring
opener
add_notebook_quotesemits (add_notebook_quotes.py:133):which yields
...code\n# %%\n'''\n— a single newline before the marker,where
py2nbneeds\n\n# %%\n. The split never happens, so the marker andboth
'''delimiters land inside the preceding code cell as literal text.(The closing path is fine: it emits
"'''", "\n\n"first, so the followingcode boundary is always correctly separated.)
Reproduced minimally, and shipped in a committed notebook today —
autolens_workspace/notebooks/interferometer/features/pixelization/many_visibilities_preparation.ipynbhas a code cell ending:
That cell is a
SyntaxErrorif a user runs it. Source isscripts/interferometer/features/pixelization/many_visibilities_preparation.py:209— a
"""with no blank line above it.A scan of the converted-script surface found this shape at 49 sites; after
excluding non-converted paths (
.github/scripts/,gallery/, module docstringsafter a shebang) the live converted-script sites are:
autolens_workspace/scripts/interferometer/features/pixelization/many_visibilities_preparation.py:209autolens_workspace/scripts/interferometer/features/subhalo/simulator.py:117autogalaxy_workspace/scripts/interferometer/features/pixelization/many_visibilities_preparation.py:193autocti_workspace/scripts/imaging_ci/modeling/features/{cosmic_rays,serial_cti,visualize_full,non_uniform}.pyHowToLens/scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py:429,HowToLens/scripts/simulator/source_complex.py:71autocti_workspace_test/legacy/**(not converted; ignore)Fix (PyAutoHands)
In the opener branch of
add_notebook_quotes, emit the separating blank linepy2nbrequires when it is missing:Two constraints the implementation must respect:
outis non-empty.py2nbstrips a leading# %%\nheader; a leading
\nwould defeat that strip and produce a spurious emptyfirst code cell.
unconditionally appends a trailing blank line to every code cell, churning
every generated notebook in every workspace for no reason. The adjacent-
docstring path already ends with
\n\nand must stay untouched (itsregression test
test_adjacent_docstrings_generate_separate_markdown_cellsmust keep passing).
Tests to add
markdown cell, and no cell source contains
# %%or''';Finding 1 — the reason the
Finish.hack is unnecessary);cell and no empty first code cell.
Out of scope (file separately)
A column-0 closing
"""of a triple-quoted string literal in code togglesdocstring state and mangles the cell the same way. One occurrence workspace-wide
—
autolens_workspace_test/gallery/gallery_build.py:42— which sits outsidescripts/and is therefore never converted. Fixing it needs real tokenization,not a line-prefix test.
Removal census — 166 occurrences, 166 files, 10 repos
Five shapes, each needing different handling — a blanket regex over all of them
is wrong:
"""\nFinish.\n"""/"""\nFinished.\n"""as the entire final docstring.Delete the block and the blank line before it, leaving the file ending on
real content with a single trailing newline.
__Env__(developer-only, stripped from artifacts anyway),__JAX Variant__,or similar. Delete the
Finish.line and the blank line after it. Note thatfor the
__Env__cases the artifact-visible markdown cell today is exactlyFinish.and nothing else, becausestrip_env_declarationsremoves the__Env__section; after removal the block holds only__Env__and theexisting standalone-fallback path drops the whole block — which is correct.
Finished.leading a real sentence — drop only the word:autolens_workspace/scripts/cluster/lenstool/modeling.py:503("Finished. The README in this folder is the narrative companion…")
autolens_workspace/scripts/guides/point_source_pairing.py:182("Finished. For the production-scale picture — real solver, …")
__Finish__— an empty trailing section header (no content followsit) in
{autolens,autogalaxy}_workspace/scripts/imaging/features/multi_gaussian_expansion/modeling.py.Delete the header line.
function body, or inside a fully commented-out block. Not converted to
markdown cells (only column-0 delimiters are), so pure dead cruft:
autolens_workspace/scripts/interferometer/features/subhalo/sensitivity/start_here.py:305,autocti_workspace_test/imaging_ci/profiling/pruning/{parallel_x1,parallel_x3,serial_x1}.py,autolens_workspace_developer/slam_pipeline/dspl.py:304.The convention is documented nowhere (
AGENTS.md,CONTRIBUTING.md,PyAutoHands/docs/, the Brain skills were all checked) — so no doc updates areneeded and nothing is authorising new ones.
Generated artifacts
notebooks/,llms-full.txt,workspace_index.json— regenerate withgenerate.pyinautolens_workspace,autogalaxy_workspace,autofit_workspace,autocti_workspaceandHowToFit. Conversion only, noscript execution — cheap. (
llms-full.txt/workspace_index.jsoncontain noFinishtoday; they still get rewritten and must be committed from the samerun so the catalogue cannot drift.) The
_test/_developerrepos andeuclid_strong_lens_modeling_pipelinehave no generated artifacts at all.markdown/— do not re-rungenerate_markdown.py. MostFinishhits in
markdown/are Nautilus sampler status tables (Finished | 18 | 1 | …), not the hack. Only five pages carry a real one:autolens_workspace/markdown/{point_source/simulator,group/simulator,interferometer/simulator}.md,autogalaxy_workspace/markdown/interferometer/fit.md,autofit_workspace/markdown/overview/overview_1_the_basics.md.A re-render executes real model fits and re-encodes every figure PNG, so it
would churn large binaries for a one-paragraph deletion
([[feedback_ship_workspace_binary_leak]]). Delete the paragraph in place and
say so in the PR.
Validation
pytest PyAutoHands/tests/test_add_notebook_quotes.pygreen, including thepre-existing adjacent-docstring tests.
generate.pyruns clean in each of the five artifact-bearing repos.# %%or'''(this currently fails on
many_visibilities_preparation.ipynband must passafter the fix) — assert it across the whole
notebooks/tree.grep -rn "Finish" scripts/ *.pyreturns nothing but genuine prose in eachswept repo.
than before, with the final cell a complete code cell.
([[feedback_smoke_tests_small_subset]], [[feedback_two_env_profiles_smoke_vs_release]]).
Claim contention (hand-read —
worktree_check_conflictnever fires,[[feedback_worktree_conflict_guard_never_fires]])
autolens_workspace— claimed by THREE active tasks:extra-galaxies-point-source,likelihood-function-jax-pointer,multi-galaxy-imaging-parity. This task edits 61 files across the whole repoand regenerates every notebook, so unlike the previous parallel decisions
there is real overlap risk with all three.
autogalaxy_workspace— claimed bylikelihood-function-jax-pointer.autolens_workspace_test— claimed byvacuous-jax-assertions.PyAutoHands— claimed bypython-312-release-surfaces, live withuncommitted work in
.gitignore,bin/autohands,docs/internals.md,pre_build.shand therun_logs/+=3.12cleanup. Zero overlap withautohands/add_notebook_quotes.py/tests/test_add_notebook_quotes.py, soparallel is safe here on the same precedent as prior decisions.
autofit_workspace,autocti_workspace,autocti_workspace_test,autofit_workspace_test,autolens_workspace_developer,HowToFit,euclid_strong_lens_modeling_pipeline.Suggested phasing follows from that, not from repo count
([[feedback_brain_repo_count_difficulty_proxy]]): PyAutoHands fix first, then
the uncontended sweep, then the contended repos once their branches land.