From the August 2026 harness audit (gates the #139 v2 rerun — v2 runs the same models under a new treatment, so identity/treatment confusion becomes an active hazard).
Symptom
Resume validation accepts a run directory regardless of which model or serving treatment produced it.
Evidence (verified)
- The resume probe returns True for a run dir relabeled
definitely-wrong-model.
- The forced-tool and
tool_choice: auto sidecar metadata are byte-identical, so nothing distinguishes a v1-condition partial run from a v2-condition one at resume time.
Why it matters
A resumed run can silently splice rows produced by a different model or under a different condition. The sensitivity runs avoided this only by manual directory discipline.
Fix
run_state.json records a treatment fingerprint: model id, contract type, tool_choice, chunk size, prompt/code version. Resume refuses on mismatch with a clear error. Test: resuming with any fingerprint field changed raises.
From the August 2026 harness audit (gates the #139 v2 rerun — v2 runs the same models under a new treatment, so identity/treatment confusion becomes an active hazard).
Symptom
Resume validation accepts a run directory regardless of which model or serving treatment produced it.
Evidence (verified)
definitely-wrong-model.tool_choice: autosidecar metadata are byte-identical, so nothing distinguishes a v1-condition partial run from a v2-condition one at resume time.Why it matters
A resumed run can silently splice rows produced by a different model or under a different condition. The sensitivity runs avoided this only by manual directory discipline.
Fix
run_state.jsonrecords a treatment fingerprint: model id, contract type, tool_choice, chunk size, prompt/code version. Resume refuses on mismatch with a clear error. Test: resuming with any fingerprint field changed raises.