calibration: backfill the slop_gate_score corpus by deterministic replay over archived diffs

[JSONbored]

Problem

The slop knob (#8224) entered the registry report-only, but its corpus only accumulates while slop gating fires live — weeks to reach the 50/12 sample floors. The slop scorer is deterministic and in-repo, and we hold hundreds of historical PR diffs with human-labeled outcomes (the #8130/#8170 raw-context corpus). The evidence can be manufactured honestly today.

Requirements

⚠️ Required pattern. Mirror the phase-1/phase-2 backfill discipline exactly (scripts/backfill-calibration-corpus*.ts): idempotent synthetic events with an explicit provenance marker, dry-run default, resume cursor, both stores (cloud D1 CLI path + --pg for the ORB Postgres), pacing if any GitHub fetches are needed (prefer none — replay archived diffs only).

• Pure core: recompute slop risk over each archived diff via the shipped deterministic scorer; emit synthetic signal.rule_fired:slop_gate_score events (confidence = risk/100, provenance-tagged) paired with override labels from realized outcomes — the same counterfactual "mapping a" framing the close-confidence backfill used, documented as such in the event metadata.
• Apply on both stores; verify the report-only proposals path and the knobs endpoint surface slop evidence afterward.
• Post the resulting first proposals (or the honest absence) on this issue.

Deliverables

• Backfill script (pure core + thin IO) with 100%-branch unit coverage on the core
• Applied + verified on cloud D1 and the ORB Postgres
• Evidence summary posted here; if proposals clear, file the slop flip-to-live issue citing them

Links & Resources

#8224 (the knob), #8157/#8170 (the backfill discipline), scripts/backfill-calibration-corpus-phase2-core.ts

Boundaries

Report-only evidence generation — no knob movement, no live-mode change.

maintainer-only — synthetic-evidence provenance is gate-adjacent.

[linear-code]

JSO-5724

[JSONbored]

Applied + verified on both stores

slop_replay_backfill_v1 ran over the full archived corpus (460 cases, manifest checksum 86c3b4b55cae…) and applied to cloud D1 (920 rows) and the ORB Postgres (460 fired + 460 override rows, verified by direct count). Rows upsert idempotently by deterministic id.

Replay distribution (labels: 287 confirmed / 173 reversed):

band	cases
zero (0)	165
low (1–30)	3
elevated (31–59)	292
high (≥60)	0

The honest reading: the replay is allowlist-restricted to the three diff-derivable signals (the scorer fires empty_pr_description on an undefined description — right live, where absence is emptiness; inflating here, where the field simply wasn't archived — so unarchived-input signals are excluded by construction, not by trusting undefined-skipping). That makes every replayed score a lower bound, and it shows: no case reaches the 0.60 ceiling, so no mass sits in the ladder band [0.60, 0.70] and the report-only proposals path will (correctly) propose nothing from this data alone.

What the backfill does buy: 460 labeled slop_gate_score cases feeding the knob's reliability curve and drift reads on both stores today, and a corpus mechanism proven end-to-end. The discriminating source for the ladder band remains live full-signal capture (running since #8223) — as live cases with description/commit/cluster signals mix in, the 50/12 floors fill with scores that can actually straddle the ceiling. The flip-to-live issue stays unfiled until proposals clear on that mixed corpus, per this issue's own boundary.