feat(calibration): slop-corpus replay backfill + replay-derived provider track records#8281
Conversation
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8281 +/- ##
=======================================
Coverage 92.16% 92.16%
=======================================
Files 788 788
Lines 79175 79175
Branches 23933 23934 +1
=======================================
Hits 72974 72974
Misses 5062 5062
Partials 1139 1139
Flags with carried forward coverage won't be shown. Click here to find out more. |
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-24 01:12:40 UTC
Review summary Nits — 2 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
…ill family (#8277) Replays the deterministic slop scorer over the archived raw-context diffs (the #8130/#8170 corpus manifest) and synthesizes provenance-tagged slop_gate_score fired/override pairs labeled by realized outcomes — the phase-1 'mapping a' counterfactual framing, applied to the #8224 knob. Signal-subset honesty is enforced by ALLOWLIST: only the three diff-derivable signals may contribute (the scorer fires empty_pr_description on an UNDEFINED description — right live, inflating here), risk/band recompute from exactly their weights, and every row records the computed signal codes beside the provenance tag. Zero GitHub traffic; idempotent upserts; wrangler/--pg dual path. Applied + verified on both stores (460 fired + 460 override rows each): distribution zero 165 / low 3 / elevated 292 / high 0 — all mass below the 0.60 ceiling, so the rows are NEUTRAL to ladder comparisons by construction (they feed the reliability curve and drift reads without being able to fabricate proposals; the discriminating source for the ladder band remains live full-signal capture).
Maps the #8221 harness's cached per-fixture verdicts onto ProviderReviewSignal (would_flag => fail, would_not_flag => pass, abstentions yield NO signal) and aggregates with computeProviderTrackRecords — the identical function live reviewer_vote rows feed. Offline report only: nothing persists, and the rendered table carries the replay-derived disclaimer (#8278's segregation rule). First same-seed two-provider table recorded on #8229.
78732e6 to
e564d71
Compare
…ver process.exit Both wrappers export pure helpers the unit suites import; an import-time main().then(process.exit) fails the whole vitest run as an unhandled rejection even with every test green (CI shard 2's exact failure — all 6568 tests passed). The audit-quality-gate-min-score.ts direct-execution idiom fixes it; dry-run CLI behavior unchanged.
Summary
Closes #8277. Advances #8278 (the adapter + first recorded table; the BYOK multi-provider campaign remains).
Phase-3 backfill (
scripts/backfill-slop-corpus{,-core}.ts): replays the deterministic slop scorer over the archived raw-context corpus manifest and synthesizes provenance-tagged (slop_replay_backfill_v1)slop_gate_scorefired/override pairs labeled by realized outcomes — the phase-1 "mapping a" framing applied to the #8224 knob. Signal-subset honesty is enforced by allowlist: only the three diff-derivable signals may contribute (caught in testing: the scorer firesempty_pr_descriptionon an undefined description — correct live where absence is emptiness, inflating in replay where the field simply wasn't archived); risk/band recompute from exactly those weights, and every row records its computed signal codes. Idempotent upserts, dry-run default, zero GitHub traffic, wrangler/--pgdual path (with the fail-loudresolvePgConnectioncontract honored — its silent-D1-fallthrough warning caught a real wiring bug during rollout).Already applied + verified on both stores (460 fired + 460 override rows each; evidence + the honest lower-bound reading posted on #8277): all replayed mass sits below the 0.60 ceiling, so these rows are neutral to ladder comparisons by construction — they feed the reliability curve and drift reads; the ladder's discriminating source remains live full-signal capture.
#8278 adapter (
scripts/provider-replay-track-record.ts): cached replay verdicts →ProviderReviewSignal→computeProviderTrackRecords(#8228, zero new math), abstentions excluded, replay-derived disclaimer baked into the output. First same-seed two-provider table posted on #8229.Validation
npm run test:ci: green except two known first-test-in-file load timeouts (satisfaction-floor-loosening-run,upstream-ruleset) while a deploy + replay ran concurrently — both pass 76/76 standalone in this worktree; neither file is touched here.npm audit --audit-level=moderateclean.scripts/**is outside the patch-coverage measurement; the cores are unit-covered per the backfill-family convention regardless.