From the August 2026 harness audit (score-affecting; gates the #139 v2 rerun — fix once, before paying for 29 models again).
Symptom
Repair rounds re-ask the model for a full chunk and overwrite cells that already held valid parsed numbers, instead of only filling the missing/invalid cells that triggered the repair.
Evidence (verified against run artifacts)
- 171 published cells across the current board differ from the first valid value the model returned.
- Net effect roughly washes in aggregate (27 cells gained exact, 27 lost exact) but is not a no-op per model.
- Concrete case: claude-haiku-4.5
scenario_003 payroll_tax — first response contained value 0 (exact vs reference); a repair round resampled and the published value is 2618.93 (miss).
Why it matters
The published number for a cell should be a deterministic function of the model's answers, not of how many repair rounds happened to run. Resampling valid answers adds noise and makes reruns non-reproducible.
Fix
In the repair path, merge repair responses cell-wise: a repair may only write cells that are missing or failed validation. Cells with a previously parsed valid value are immutable. Test: a repair response that contradicts an already-valid cell leaves the original value and logs the disagreement.
From the August 2026 harness audit (score-affecting; gates the #139 v2 rerun — fix once, before paying for 29 models again).
Symptom
Repair rounds re-ask the model for a full chunk and overwrite cells that already held valid parsed numbers, instead of only filling the missing/invalid cells that triggered the repair.
Evidence (verified against run artifacts)
scenario_003payroll_tax— first response contained value 0 (exact vs reference); a repair round resampled and the published value is 2618.93 (miss).Why it matters
The published number for a cell should be a deterministic function of the model's answers, not of how many repair rounds happened to run. Resampling valid answers adds noise and makes reruns non-reproducible.
Fix
In the repair path, merge repair responses cell-wise: a repair may only write cells that are missing or failed validation. Cells with a previously parsed valid value are immutable. Test: a repair response that contradicts an already-valid cell leaves the original value and logs the disagreement.