From the August 2026 harness audit (score-affecting; gates the #139 v2 rerun).
Symptom
When a model's free-text explanation ends with a value = X trailer, the parser lets the trailer override the structured value from the tool call / JSON contract.
Evidence (verified)
- 64 cells board-wide where trailer and structured value disagree and the trailer won.
- Largest single-model effect: +1.897 pts (gpt-5.4-nano).
- Concrete case: gpt-5.4-mini
medicare_eligible — structured tool-call value 1, published value 0 taken from the explanation trailer.
Why it matters
The structured channel is the answer contract; the explanation is commentary. Letting prose override the schema-validated field inverts the contract and rewards/punishes models for formatting quirks.
Fix
Structured value is canonical whenever present and valid. The trailer is a fallback only when the structured cell is missing/unparseable. Log structured-vs-trailer disagreements as a diagnostic column. Test: disagreement case publishes the structured value.
From the August 2026 harness audit (score-affecting; gates the #139 v2 rerun).
Symptom
When a model's free-text explanation ends with a
value = Xtrailer, the parser lets the trailer override the structured value from the tool call / JSON contract.Evidence (verified)
medicare_eligible— structured tool-call value 1, published value 0 taken from the explanation trailer.Why it matters
The structured channel is the answer contract; the explanation is commentary. Letting prose override the schema-validated field inverts the contract and rewards/punishes models for formatting quirks.
Fix
Structured value is canonical whenever present and valid. The trailer is a fallback only when the structured cell is missing/unparseable. Log structured-vs-trailer disagreements as a diagnostic column. Test: disagreement case publishes the structured value.