From eb728401b456ef6666483c0dfa3314e83bd5bc9a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 01:20:43 +0000 Subject: [PATCH 1/4] prompt: confirm root cause of the numba psf_weighted_data garbage Records the confirmed diagnosis on the bug prompt: an unguarded out-of-bounds gather in `psf_weighted_data_from`, not the numba codegen/caching issue the prompt led with. Proven by compiling the shipped source under numba `boundscheck=True`. Notes why the inf suspect is unreachable, and what was not verified here (the profiling-harness corruption counters). Files the kernel-shift axis swap found alongside it as its own prompt, per the one-prompt-one-task rule. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_013unzp382r79c4BN8g4ckb2 --- ...ba_first_call_garbage_psf_weighted_data.md | 50 ++++++++++++++ .../numba_kernel_shift_axes_swapped.md | 67 +++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 draft/bug/autoarray/numba_kernel_shift_axes_swapped.md diff --git a/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md b/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md index dc5a9189..76ae1289 100644 --- a/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md +++ b/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md @@ -55,3 +55,53 @@ as the regression probe. (`_materialize_all`). Worth testing: `cache=False`, numba version pin, and whether symptom 2 reproduces with symptom 1 fixed (they may share a cause). - Keep the profiling-harness corruption counters as the acceptance test. + +## Root cause — confirmed 2026-08-21 + +Not a numba codegen or caching bug. `psf_weighted_data_from` gathers the +weight map at `[ip0_y + k0_y + kernel_shift_y, ip0_x + k0_x + kernel_shift_x]` +with **no bounds check**. numba `@jit()` does not bounds-check array reads, so +for any unmasked pixel within `kernel_shape // 2` of the array edge the gather +reads uninitialized heap memory instead of raising `IndexError`. Negative +indices are unsafe in the same way — they wrap to the opposite edge. + +Proof: compiling the shipped source unchanged under `boundscheck=True` raises +`IndexError: index is out of bounds` for a mask reaching the array edge, and is +clean for an interior-only mask. With the guard added, the numba output matches +the zero-padded numpy twin exactly across array sizes and kernel sizes. + +This explains **both** symptoms, and explains why the inputs were verified +identical between call 1 and call 2 — they were; the function reads memory +*outside* its inputs: + +- **Symptom 1** — a cold-cache first call runs right after numba's compilation + has churned the heap, so the memory next to the freshly allocated weight map + holds compiler garbage (~1e299). Warm cache: no compile, benign neighbour. +- **Symptom 2** — each forked worker has a different heap layout, so whether + the neighbouring memory is poisonous varies per worker and per run. Hence + 2/8 corrupted in one map and 0/24 in the next. + +The `np.isnan` guard was doing real work (masked border is `0/0 = NaN`) but +never protected the array bounds. The sibling `psf_precision_value_from` was +already hardened against exactly this — `psf_weighted_data_from` was missed. + +The inf suspect (finite image / zero noise passing the `isnan` guard) is **not +reachable** via the caller: `.native` zeroes both data and noise outside the +mask, giving `0/0 = NaN`, never `inf`. An inf would require a zero noise value +*inside* the mask, which is a data-validation error and should stay loud. The +`isnan` guard is therefore left as-is. + +Fix: @PyAutoArray branch `claude/autoarray-numba-psf-garbage-hfxnjv` — bounds +guard mirroring the sibling, plus a numba-vs-numpy equivalence regression test +on an edge-touching mask (fails without the fix, passes with it). Full +`test_autoarray` suite: 1034 passed, 3 pre-existing pynufft failures unrelated +to this change. + +Not done: the `parallel_scaling/pixelization_numba.py` corruption counters +named as the acceptance probe could not be run here (needs the profiling +workspace and its datasets). Symptom 2 should be re-measured against this +branch before the prompt is closed. + +Split out while fixing this: `draft/bug/autoarray/numba_kernel_shift_axes_swapped.md` +— both numba gathers derive the y/x kernel shifts from the transposed kernel +axes, harmless for square kernels but wrong for non-square odd PSFs. diff --git a/draft/bug/autoarray/numba_kernel_shift_axes_swapped.md b/draft/bug/autoarray/numba_kernel_shift_axes_swapped.md new file mode 100644 index 00000000..c6d991dd --- /dev/null +++ b/draft/bug/autoarray/numba_kernel_shift_axes_swapped.md @@ -0,0 +1,67 @@ +# Numba PSF gathers derive the y/x kernel shifts from the wrong kernel axes + +Type: bug +Target: autoarray +Repos: +- @PyAutoArray +Difficulty: low +Autonomy: supervised +Priority: medium +Status: formalised + +Found 2026-08-21 while fixing +`draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md` (the +out-of-bounds gather in `psf_weighted_data_from`). Split out under the +one-prompt-one-task rule: separate defect, separate blast radius. + +## Symptom + +Both numba PSF gathers in +`autoarray/inversion/inversion/imaging_numba/inversion_imaging_numba_util.py` +compute their kernel half-widths from the **transposed** kernel axes: + +```python +kernel_shift_y = -(kernel_native.shape[1] // 2) # shape[1] is x +kernel_shift_x = -(kernel_native.shape[0] // 2) # shape[0] is y +``` + +at `psf_weighted_data_from` (line ~48) and `psf_precision_value_from` +(line ~294). The y shift must come from `shape[0]` and the x shift from +`shape[1]`. + +The zero-padded numpy twin +(`imaging/inversion_imaging_util.py:psf_weighted_data_from`) gets it right and +is the reference: + +```python +Ky, Kx = kernel_native.shape +ph, pw = Ky // 2, Kx // 2 +``` + +## Reachability + +Harmless for square kernels (`shape[0] == shape[1]`), which is the common +case and why no test catches it. It is **not** unreachable: kernels are +validated as *odd* in each axis, not square — `exc.KernelException("Convolver +Convolver must be odd")` in `operators/convolver.py:268` and +`structures/grids/uniform_2d.py:1153` check parity only. A non-square odd PSF +(e.g. 3x5) therefore mis-centres the gather, sampling the weight map / noise +map off-centre along both axes. + +With the bounds guard now in place the mis-centred reads are clipped rather +than reading uninitialized memory, so this is a silent wrong-answer bug, not +a crash or a garbage-value bug. + +## Fix + +Swap the two right-hand sides in both functions. Fix them **together** — they +must agree on kernel orientation, and correcting only one would make the +`psf_weighted_data` and `psf_precision_operator` paths disagree. + +## Acceptance + +Extend the numba-vs-numpy equivalence test added by the OOB fix +(`test_autoarray/inversion/inversion/imaging/test_inversion_imaging_util.py:: +test__psf_weighted_data_from__unmasked_pixels_on_array_edge`) to a non-square +odd kernel (e.g. 3x5). It passes today only because that test uses a square +kernel; with a non-square kernel the two implementations diverge. From d8543f51de8c5a68204460f012330d52b3ec8e81 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 01:31:20 +0000 Subject: [PATCH 2/4] prompt: link the PyAutoArray fix PR on the numba psf_weighted_data bug Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_013unzp382r79c4BN8g4ckb2 --- .../bug/autoarray/numba_first_call_garbage_psf_weighted_data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md b/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md index 76ae1289..71a623ed 100644 --- a/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md +++ b/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md @@ -91,7 +91,7 @@ mask, giving `0/0 = NaN`, never `inf`. An inf would require a zero noise value *inside* the mask, which is a data-validation error and should stay loud. The `isnan` guard is therefore left as-is. -Fix: @PyAutoArray branch `claude/autoarray-numba-psf-garbage-hfxnjv` — bounds +Fix: @PyAutoArray PR #456 (branch `claude/autoarray-numba-psf-garbage-hfxnjv`) — bounds guard mirroring the sibling, plus a numba-vs-numpy equivalence regression test on an edge-touching mask (fails without the fix, passes with it). Full `test_autoarray` suite: 1034 passed, 3 pre-existing pynufft failures unrelated From 2562acf4a13850fbd0a556a82d64a7425566dffb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 01:49:12 +0000 Subject: [PATCH 3/4] mind: regenerate the dashboard after filing the kernel-shift prompt `dashboard_refresh.yml` gates pull requests on `draft/**`, so the stale page would have failed CI on the branch rather than self-healing on main. Diff is confined to the new prompt: backlog 145 -> 146, bug 36 -> 37, and its backlog entry. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_013unzp382r79c4BN8g4ckb2 --- dashboard.html | 7 ++++--- dashboard.md | 14 +++++++++++--- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/dashboard.html b/dashboard.html index 78814b26..bfa1c971 100644 --- a/dashboard.html +++ b/dashboard.html @@ -38,7 +38,7 @@

📋 PyAutoMind Dashboard

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.

-

In flight 4 · Parked 3 · Planned 6 · Backlog 145 · markdown version

+

In flight 4 · Parked 3 · Planned 6 · Backlog 146 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 33

TRIAGE: needs manual review before routing — medium · safe · high

@@ -84,9 +84,9 @@

Planned

latent-nan-guard-honest-run

Backlog markdown version

-

145 filed prompts, not started — sorted most-pickable first (priority, then size).

+

146 filed prompts, not started — sorted most-pickable first (priority, then size).

-bug — 36 +bug — 37

Numba sparse-operator likelihood: first-call garbage / intermittent worker corruption — autoarray · medium · supervised · high

EP hierarchical parent-scale collapse: cure the basin, or document the — autofit · too-large · human-required · high

Investigate eager FitImaging.figure_of_merit vs JIT/step-by-step divergence in rectangular pixelization — autolens · too-large · supervised · high

@@ -100,6 +100,7 @@

Backlog

jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium

Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium

jax_grad scripts fail assertions locally that PASS in CI — autolens_workspace_test · medium · supervised · medium

+

Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium

PyNUFFT dev extra is incompatible with current SciPy on Python — autoarray · small · supervised · normal

LogGaussianPrior misreports its own support as (-inf, inf) — autofit · small · supervised · normal

autofit.plot functions accept **kwargs and silently discard them — autofit · small · supervised · normal

diff --git a/dashboard.md b/dashboard.md index 2081918f..67ea26dd 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | [In flight](#in-flight) (`active/`) | 4 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | -| [Backlog](#backlog) (`draft/`) | 145 | +| [Backlog](#backlog) (`draft/`) | 146 | ## Start here @@ -273,10 +273,10 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**145** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**146** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
-bug — 36 +bug — 37
📋 Numba sparse-operator likelihood: first-call garbage / intermittent worker corruption — autoarray · medium · supervised · high @@ -382,6 +382,14 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
+
📋 Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium + +``` +/start_dev draft/bug/autoarray/numba_kernel_shift_axes_swapped.md +``` + +
+
📋 PyNUFFT dev extra is incompatible with current SciPy on Python — autoarray · small · supervised · normal ``` From e993b11901123ffb981ac8643adfe004a6895bf2 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 02:14:46 +0000 Subject: [PATCH 4/4] prompt: record the euclid reproduction of the psf_weighted_data garbage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom 1 reproduced exactly on the real profiling dataset: pre-fix max abs(psf_weighted_data) = 4.901e300, post-fix 298.312. The bug report's own "2.98e02 on fit #2" matches the post-fix value exactly. Corrects an earlier reading of the mask padding: `apply_mask` does not pad, and the padded blurring mask belongs to the dense convolver, not the sparse numba path — which reads the unpadded (71, 71) array, so 1244 of 3841 pixels gather off the array edge. Also records that the named acceptance probe is a weak detector: the corruption counters read zero both pre-fix and post-fix, because the values an out-of-bounds read returns depend on heap state. The direct max abs(psf_weighted_data) check is the reliable probe. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_013unzp382r79c4BN8g4ckb2 --- ...ba_first_call_garbage_psf_weighted_data.md | 44 +++++++++++++++++-- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md b/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md index 71a623ed..de852a53 100644 --- a/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md +++ b/draft/bug/autoarray/numba_first_call_garbage_psf_weighted_data.md @@ -97,10 +97,46 @@ on an edge-touching mask (fails without the fix, passes with it). Full `test_autoarray` suite: 1034 passed, 3 pre-existing pynufft failures unrelated to this change. -Not done: the `parallel_scaling/pixelization_numba.py` corruption counters -named as the acceptance probe could not be run here (needs the profiling -workspace and its datasets). Symptom 2 should be re-measured against this -branch before the prompt is closed. +## Reproduced on the euclid dataset — 2026-08-21 + +Symptom 1 reproduced exactly, on the real profiling dataset +(`autolens_profiling`, `dataset/imaging/euclid`, mask radius 3.5", PSF 21x21), +by calling `psf_weighted_data_from` directly on the masked dataset: + +| | `max abs(psf_weighted_data)` | sum | +|---|---|---| +| pre-fix (`1c33850`) | **4.901e300** | 1.333e301 | +| post-fix | **298.312** | 559433.417 | + +The bug report's own numbers were "max abs = 4.8e299 on fit #1, 2.98e02 on fit +#2". The post-fix value **298.31 = 2.98e02** matches the report's *correct* +value exactly, and the pre-fix value reproduces the uninitialized-memory scale. +1244 of the 3841 unmasked pixels (32%) drive the gather off the array. + +Note on the mask padding — it does **not** protect this path. `apply_mask` +emits no padding warning and leaves `data.native` and `data.mask` at (71, 71); +only `derive_mask.blurring_from(allow_padding=True)` pads, to (89, 89), and +that padded blurring mask is used by the dense convolver, not by the sparse +numba path. `psf_weighted_data_from` reads the unpadded (71, 71) array via +`data.mask.derive_indexes.native_for_slim`, so the mask sits flush against the +array edge and the 21x21 kernel reads past it. + +## The acceptance probe is a weak detector — use the direct check instead + +`parallel_scaling/pixelization_numba.py` was run at P=2, 24 evals, 2 map +repeats, cold `NUMBA_CACHE_DIR`, both pre-fix and post-fix. **Both runs +reported `corrupt_evals_first_map = 0` and `corrupt_evals_steady_maps = [0, 0]`, +and both had a finite warm-up likelihood.** That is not evidence of no bug: the +values an out-of-bounds read returns are whatever the allocator left next to +the weight map, so in that process they happened to be benign. The pre-fix +warm-up likelihood still drifted from the post-fix one in the 6th decimal +(5860.175003698866 vs 5860.175922117387) — the same reads, landing on small +values instead of huge ones. This is exactly the reporter's own intermittency +(2/8 corrupted in one map, 0/24 in the next). + +So the counters can sit at zero on a run where the bug is fully present. Prefer +the direct `max abs(psf_weighted_data)` check above as the regression probe — +it is deterministic within a process and reproduces the reported magnitudes. Split out while fixing this: `draft/bug/autoarray/numba_kernel_shift_axes_swapped.md` — both numba gathers derive the y/x kernel shifts from the transposed kernel