v2 scoring-pipeline fixes: repair immutability, structured-canonical, resume fingerprints, export guards, spend ledgers

[MaxGhenis]

Implements the harness-audit fixes gating the #139 v2 rerun. Closes #143, closes #144, closes #146, closes #147, closes #148.

• Repair rounds resample and overwrite valid answers #143 — repair immutability: repairs merge cell-wise and may only fill missing/invalid cells; valid cells are immutable and disagreements are logged (repair_disagreements).
• Explanation trailer overrides the structured tool-call value #144 — structured value canonical: a valid structured/tool value always wins; the explanation trailer only rescues a missing/unparseable structured cell. Disagreements logged. (This inverts the previous deliberate trailer-canonical behavior; the old test asserting it is replaced by the structured-wins + trailer-rescue pair.)
• Resume validation ignores model identity and serving treatment #146 — resume treatment fingerprints: run_state records model id, contract, tool_choice mode, and chunk size; resume refuses on mismatch naming the differing field.
• Cost and latency accounting integrity #147 — accounting: PRICE_OVERRIDES canonical before the litellm map (estimated flag honest), nullable latency end-to-end, spend ledger sidecar includes failed + repair calls with stable call keys.
• Serialization order, export guards, batch parity, cache hygiene #148 — integrity guards: modelStats serialized in headline-metric order; fold/export reject duplicate models and out-of-range scores; litellm cache keys namespaced by contract version.

#145 (explanation-contract policy) is deliberately not implemented — it's a methodology decision, tracked separately.

Verification (independent of the implementing agent): full non-slow suite 534 passed, 5 skipped rerun locally; ruff clean; the frozen published-artifact recombination test passes, so published v1 board bytes are unaffected — these changes take effect for v2 reruns only. Four deleted tests each assert behavior an issue intentionally inverted, and each has a stronger replacement (structured-wins/trailer-rescue, repair fill-only pair, duplicate-rejection, override-canonical).

Authored by GPT-5.6 Sol via codex (ultra), reviewed hunk-by-hunk + independently re-tested by Claude.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policybench-site	Ready	Preview	Aug 10, 2026 4:10am