counterfactual: design the offline judge-replay contract — sampling, spend budget, scoring mapping

[JSONbored]

Problem

Counterfactual sub-epic phase 1. Before any harness exists, three contracts need explicit answers, written down where the later issues can cite them:

Requirements

• Fixture contract: which corpus cases are replayable (must carry metadata.diff; bounded size; label present), what a fixture contains, and the deterministic sampling rule when the eligible set exceeds a run's budget (seeded, like splitBacktestCorpus — never 'the first N').
• Spend contract: per-run budget expressed in the existing neuron-estimate terms (mirror the BYOK/free budgeting posture in linked-issue-satisfaction-run), provider order (local ollama first for smoke, BYOK for real runs), and the hard cap semantics when a run exhausts mid-set (persist partial scores + cursor, exactly like calibration: backfill phase 2 — GitHub-truth reversed labels (retro successor scan) + raw-context re-fetch #8170's budget discipline).
• Scoring mapping: how a replayed variant's verdict maps onto the existing BacktestCase classifier contract so scoreBacktest/compareBacktestScores apply UNCHANGED — the variant is just another classifier. Define the abstention case (variant returns unparseable output) explicitly: skipped-and-counted, never coerced.
• Deliverable is a design doc section on the backtest-calibration docs page + the contracts as typed constants/interfaces in a stub module the harness issue implements against.

Links & Resources

The sub-epic (parent), src/services/linked-issue-satisfaction-run.ts (budget posture), scripts/backfill-calibration-corpus-phase2.ts (cursor/budget discipline)

Boundaries

Design + stubs only; no model calls yet.

maintainer-only — spend + safety contract.

[linear-code]

JSO-5703