test: run every script in smoke, not a 6-of-40 allowlist - #66
Merged
Conversation
smoke_tests.txt was an allowlist: a script was tested only if someone remembered to add it, so every new tutorial was uncovered from birth. Coverage was 4/26 in HowToGalaxy, 6/40 in HowToLens and 10/15 in HowToFit. That is how a public teaching notebook stayed broken in three places with CI green (HowToGalaxy #56/#57) - no job had ever executed it. The only backstop is PyAutoHeart's workspace-smoke, which runs weekly. Coverage is now opt-OUT. run_smoke.py becomes a thin shim over PyAutoHands' autohands/run_python.py - the same entry point Heart's workspace-validation uses for run_scripts - so the PR gate and the validation runner cannot drift apart, and both read one exclusion list (config/build/no_run.yaml, which the notebook runner already honoured). Discovery, exclusion and env resolution are no longer reimplemented here. --report-dir is passed deliberately, not cosmetically: run_python.py only propagates failures when a report was built, and without it the suite runs to completion and always exits 0. It also switches execute_script from abort-on-first-failure to record-and-continue, matching the old runner's behaviour. A negative control confirms a deliberate failure turns the suite red. Executed-script counts, measured, not assumed: HowToGalaxy 4 -> 26 HowToLens 6 -> 39 (+1 deliberately skipped) HowToFit 10 -> 15 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ous commit) The previous commit deleted smoke_tests.txt but, due to a stale pathspec in the `git add` invocation, staged none of the accompanying changes - so it removed the allowlist while leaving the old runner that requires it, and CI failed with "ERROR: no smoke_tests.txt". This commit carries the work that should have been in it. run_smoke.py becomes a thin shim over PyAutoHands' autohands/run_python.py, the same entry point Heart's workspace-validation uses for run_scripts, so the PR gate and the validation runner share one code path and one exclusion list (config/build/no_run.yaml). --report-dir is passed deliberately: run_python.py only propagates failures when a report was built, and without it the suite always exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Expanding smoke to every script surfaced a pre-existing environment gap. tutorial_7_adaptive_pixelization and tutorial_10_brightness_adaption both do `from autolens import jax_wrapper` at import time, but .github/scripts/smoke_install.sh installs the [optional] extras (which bring jax) only on Python 3.12; the 3.13 leg deliberately exercises the lean, no-optional-deps path. Neither tutorial was in the old 6-script allowlist, so nobody had ever run them there. Both now use the workspace's established optional-dependency idiom - the same importlib.util.find_spec / sys.exit(0) guard used by e.g. autolens_workspace/scripts/interferometer/modeling.py - which is a clean exit 0 as a script, and which build_util.is_clean_skip_exit() already recognises as a PASS on the notebook side. Verified both ways: with jax blocked each exits 0 with an explanatory message; with jax present both still run to completion (25.0s / 78.7s under the smoke profile). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the
smoke_tests.txtallowlist with opt-out coverage: smoke now runs every script underscripts/, minus the exclusions inconfig/build/no_run.yaml.Under the allowlist a script was tested only if someone remembered to add it, so every new tutorial was uncovered from birth:
That is how a public teaching notebook stayed broken in three places while CI was green (#56 / #57) — no job had ever executed it. The only backstop is PyAutoHeart's
workspace-smoke, which runs weekly, and the run that caught it was a manualworkflow_dispatch, not the schedule.Design
run_smoke.pybecomes a thin shim over PyAutoHands'autohands/run_python.py— the same entry point PyAutoHeart's workspace-validation uses for itsrun_scriptsjob. Discovery, exclusion and env resolution are no longer reimplemented per-repo, so the PR gate and the validation runner cannot drift apart, and both read one exclusion list (config/build/no_run.yaml, which the notebook runner already honoured).The delegated runner also orders
simulator*first and thenstart_here.py, which is what tutorials depending on simulated datasets need.Depends on
PyAutoLabs/PyAutoHands#225 —
execute_script()silently rewrote any failure to PASS for scripts withinversionin their path. Without it,chapter_4_pixelizations/tutorial_3_inversions.pywould be unconditionally green in this newly-expanded suite, defeating the purpose.Scripts Changed
.github/scripts/run_smoke.py— delegates torun_python.pysmoke_tests.txt— deletedconfig/build/no_run.yaml— dropped the staletutorial_searchesexclusion (no stated reason, passes in 10.1s); corrected thetutorial_5_bordersreason and tagged itNEEDS_FIXAGENTS.md,.gitignoreOn
tutorial_5_bordersRecorded as "Cant get right masks, need proper update." Controlled re-test on identical dataset files: it fails only with
PYAUTO_SMALL_DATASETS(IndexError: index 371 is out of bounds for axis 0 with size 272) and passes without it. So it is cap-induced, not a mask problem — hardcoded indices outrun the capped mesh, the same root cause as HowToGalaxy #59. The exclusion stays (the cap stays), but the reason is now accurate and tagged so it surfaces rather than sitting silently.Test Plan
EXIT=0tutorial_5_borders, deliberate and documentedGenerated by the PyAutoLabs agent workflow.