fix(ci): stop filtering JAX tracebacks in the script-run harness

[Jammy2211]

Overview

When a workspace script fails under JAX inside the script harness (.github/workflows/workspace-validation.yml, job run_scripts, both the smoke and release legs), JAX's default traceback filter strips its internal frames and, in practice, the exception that matters. The job log records which script failed and nothing about why. That is not hypothetical: run 30516167217 (2026-07-30) failed both interferometer shards with only the filter placeholder, and the real cause — jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes — surfaced only the next night in run 30607596240, where the message happened to survive. One night was spent identifying a failure the harness already had the information to name.

A gate whose error messages are filtered away is the same category of problem as the exit-code contract fixed in PyAutoBrain#196: the signal exists, the harness discards it before a human sees it. The OOM itself was root-caused and fixed on 2026-07-31 (autolens_workspace#450, batch_size=4 on the 48-start MultiStartProdigy); this issue is about the harness, which still filters and will do the same to the next JAX failure.

Plan

• Set JAX_TRACEBACK_FILTERING: "off" once, in the workflow-level env: block of workspace-validation.yml — the block that already carries PYAUTO_SKIP_WORKSPACE_VERSION_CHECK.
• One place covers both the smoke and release legs and every workspace in the matrix; the per-script env profiles inherit it, since a profile only sets the keys it is given and never clears inherited vars.
• Decide whether the sibling script-running workflows (smoke-tests.yml, workspace-smoke.yml) want the same, for the same reason.
• Verify against a real or deliberately-induced JAX failure in CI, not only by reading the diff — the exception type and message must appear in the job log.

Detailed implementation plan

Work Classification

Library / Other (organ repo — PyAutoHeart CI workflow). No PyAuto library or workspace source is touched.

Affected Repositories

• PyAutoHeart (primary)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoHeart	main (d576003)	clean

No active.md worktree claim on PyAutoHeart (worktree_check_conflict → 0).

Suggested branch: feature/jax-traceback-filtering-release-harness

Worktree root: ~/Code/PyAutoLabs-wt/jax-traceback-filtering-release-harness/ (created later by /start_library)

Current state (verified 2026-08-27, PyAutoHeart @ d576003)

grep -rn "JAX_TRACEBACK_FILTERING" . returns no matches anywhere in PyAutoHeart. It is also absent from the workspaces' config/build/profile_release.yaml and profile_smoke.yaml, both of which otherwise pin every JAX-adjacent variable they care about explicitly (JAX_ENABLE_X64, PYAUTO_DISABLE_JAX, ...).

Implementation Steps

1. In .github/workflows/workspace-validation.yml, add to the top-level env: block (currently PYAUTO_SKIP_WORKSPACE_VERSION_CHECK: "1"):

     # JAX's default traceback filter removes the frames that name the real
     # exception, so a failing script reports only "JAX has removed its internal
     # frames". Run 30516167217 cost a night to identify for exactly this reason.
     JAX_TRACEBACK_FILTERING: "off"

   Workflow-level env: is inherited by every job and step, so both the mode=smoke and mode=release run_python.py invocations pick it up, for every project in the matrix.

2. Check smoke-tests.yml and workspace-smoke.yml for the same pattern (they also run workspace scripts under JAX) and add the same line where it applies. If a deliberate reason exists not to, record it in the workflow rather than leaving it implicit.

3. Verify in CI. Induce a JAX exception in a scratch script (or re-run a known JAX failure) and confirm the job log carries the exception type and message rather than the placeholder line. Reading the diff is not verification for this change — the whole defect is that the diff looks obviously right and the log is what has to change.

Key Files

• .github/workflows/workspace-validation.yml — the env: block near the top (above jobs:), and the two Run Python scripts [mode=...] steps that inherit it.
• .github/workflows/smoke-tests.yml, .github/workflows/workspace-smoke.yml — sibling script-runners to check.

Notes / trade-offs

• Turning the filter off makes JAX tracebacks longer and noisier in CI logs. That is the intended trade: a CI log is read only when something has already failed, and the frames are what tell you what failed.
• This is a CI-only change; nothing ships in a wheel and no workspace script or library behaviour changes.

Original Prompt

Click to expand starting prompt

# Release/smoke script harness filters JAX tracebacks away, hiding the real exception

Type: bug
Target: ci
Repos:
- @PyAutoHeart
Difficulty: small
Autonomy: safe
Priority: medium
Status: formalised
Filed: 2026-08-27

## Symptom

When a workspace script fails under JAX inside PyAutoHeart's script harness
(`.github/workflows/workspace-validation.yml`, job `run_scripts`, both the
`smoke` and `release` legs), JAX's default traceback filter removes its own
internal frames *and*, in practice, the exception that matters:

    scripts/interferometer/start_here.py ...  FAIL (19.5s)
    For simplicity, JAX has removed its internal frames from the traceback of the
    following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

That is the whole failure record. The reader is told which script failed and
nothing about why.

## Why it matters — the cost is measured, not hypothetical

PyAutoHeart run 30516167217 (2026-07-30) failed both interferometer shards
(`autogalaxy` and `autolens`) with exactly the message above. Because the real
exception was filtered away, that night read as an unidentified failure. The
same defect surfaced its true cause only the following night, run
30607596240 (2026-07-31), where the message survived intact:

    jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes.

One night was spent identifying a failure the harness already had the
information to name. The underlying OOM was root-caused and fixed the same day
(`autolens_workspace#450`; record:
`complete/2026/08/interferometer-start-here-integrate-oom.md`) — this task is
about the *harness*, which still filters, and will do the same to the next
JAX failure.

A gate whose error messages are filtered away is the same category of problem
as the exit-code contract fixed in PyAutoBrain#196: the signal exists, the
harness discards it before a human sees it.

## Current state (verified 2026-08-27)

`JAX_TRACEBACK_FILTERING` appears nowhere in PyAutoHeart:

    $ grep -rn "JAX_TRACEBACK_FILTERING" .        # PyAutoHeart @ d576003 — no matches

Nor in the workspaces' `config/build/profile_release.yaml` /
`profile_smoke.yaml` defaults, both of which already pin every JAX-adjacent var
they care about (`JAX_ENABLE_X64`, `PYAUTO_DISABLE_JAX`, …) explicitly.

## Proposed fix

Set `JAX_TRACEBACK_FILTERING: "off"` once, in the workflow-level `env:` block of
`PyAutoHeart/.github/workflows/workspace-validation.yml` — the block that
already carries `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK`. One place covers both
the `smoke` and `release` legs and every workspace in the matrix, and
`run_python.py`'s per-script env profiles inherit it (a profile only *sets* the
keys it is given; it never clears inherited vars).

Consider whether the same belongs in the other script-running workflows
(`smoke-tests.yml`, `workspace-smoke.yml`) for the same reason.

## Exit criteria

- A JAX exception raised inside a workspace script under `run_scripts` reports
  its own exception type and message in the job log, not the
  "JAX has removed its internal frames" placeholder.
- Verified against a real (or deliberately induced) JAX failure in a CI run,
  not only by reading the diff.

## Origin

Split out of `draft/bug/autolens/interferometer_release_leg_oom.md` (its
2026-08-04 amendment, consequence 2) when that prompt was retired as superseded
— the OOM it described had already shipped, this harness gap had not.

[Jammy2211]

Shipped — #187 merged as c8b093e1.

Repo	PR	Change
PyAutoHeart	#187	JAX_TRACEBACK_FILTERING: "off" — workflow-level env: in workspace-validation.yml, and the workspace-runner step's env: in smoke-tests.yml. 2 files, +18/−0

workspace-smoke.yml needed no edit of its own: it is a thin caller of workspace-validation.yml, so it inherits the workflow-level value.

CI green before merge: Heart Tests run 33093776599, pytest (3.12) and pytest (3.13) both success; mergeable_state: clean. Locally, python3 -m pytest -q -n auto → 641 passed, and both workflows re-parse under yaml.safe_load (which also confirms "off" stays the string rather than becoming a YAML 1.1 boolean).

The exit criterion is not yet met, deliberately. The issue asks that a real JAX exception print its own type and message in a job log, "verified against a real (or deliberately induced) JAX failure in a CI run, not only by reading the diff". Nothing has failed under JAX since the merge, so that has not been observed. The next JAX failure in run_scripts or the smoke runner is the proof; if it still shows the placeholder, reopen this.

Mind state: PyAutoMind #347.

---

Generated by Claude Code