feat(calibration): pure counterfactual fixture assembler per the replay contract

[kai392]

Summary

• Add packages/loopover-engine/src/calibration/counterfactual-fixtures.ts (Epic: AI counterfactual backtesting — prompt/model changes earn the same evidence gate as rule changes #8218 sub-epic phase 2): assembleCounterfactualFixtures(cases, contract) — pure selection, no AI calls, no IO — implementing the counterfactual: design the offline judge-replay contract — sampling, spend budget, scoring mapping #8219 contract's shapes verbatim (CounterfactualFixture, CounterfactualSamplingContract, CounterfactualSkipReason, and the contract's own isReplayableCase narrowing helper; nothing re-invented).
• Eligibility: bounded raw context (metadata.diff, non-empty) + label; every exclusion is accounted in the returned skipped: Record<CounterfactualSkipReason, number> (the Extend the backtest check to logic/regex changes via a dedicated CI job, using captured raw context #8139 skipped-case discipline) — fixtures + skips always sums to the input corpus, pinned by an invariant test.
• Era provenance per the contract: the feat(calibration): phase-2 backfill — retro reversal labels + raw-context re-fetch (GitHub truth) #8207/calibration: backfill phase 2 — GitHub-truth reversed labels (retro successor scan) + raw-context re-fetch #8170 backfill tags re-fetched rows with metadata.rawContextProvenance, the live calibration: capture bounded raw context (issueText/prTitle/prBody/diff) in linked_issue_scope_mismatch's fired-event metadata #8129/calibration: capture bounded raw context for the remaining isConfiguredGateBlocker codes, excluding secret_leak #8130 writers never set it — presence of the key ⇒ raw_context_refetch, otherwise live_capture, so results segment by capture era.
• Seeded deterministic sampling only when the eligible set exceeds maxFixtures, via the splitBacktestCorpus content-hash discipline (sha256(${seed}:${targetKey}), first-8-hex rank, lowest ranks win) — never "the first N"; corpus order preserved within the sample; a same-target rank tie breaks toward the earlier case, deterministically.
• Barrel-exported; no fetches, no model calls, no store reads, per the issue's Boundaries.

Closes #8220

Test plan

• test/unit/counterfactual-fixtures-engine.test.ts (vitest, importing the engine source — the root mirror suite): eligibility arms (missing diff, empty diff, replayable) with skip accounting + the sum invariant; era segmentation both ways; under-budget no-sampling vs over-budget seeded sampling with preserved corpus order; same-seed byte-determinism, cross-seed divergence, and the not-first-N pin; the forced two-element same-target tie-break; empty corpus
• packages/loopover-engine/test/counterfactual-fixtures.test.ts (node:test vs dist): barrel re-export + eligibility/provenance/skip round-trip + sampling determinism
• Local coverage on the new file: 19/19 lines, 8/8 branches; engine workspace suite 677 green; npm run typecheck clean
• CI validate

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.12%. Comparing base (d8f706c) to head (3142b4e).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8247      +/-   ##
==========================================
- Coverage   92.12%   92.12%   -0.01%     
==========================================
  Files         782      783       +1     
  Lines       78531    78551      +20     
  Branches    23720    23725       +5     
==========================================
+ Hits        72347    72363      +16     
  Misses       5062     5062              
- Partials     1122     1126       +4     

Flag	Coverage Δ
shard-1	56.52% <0.00%> (-3.43%)	⬇️
shard-2	53.99% <0.00%> (+6.77%)	⬆️
shard-3	49.91% <100.00%> (-3.62%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...-engine/src/calibration/counterfactual-fixtures.ts	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

[loopover-orb]

Warning

⏸️ LoopOver review result - manual review recommended

_{Review updated: 2026-07-23 14:54:29 UTC}

4 files · 1 AI reviewer · no blockers · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This adds a small, pure `assembleCounterfactualFixtures` function that filters a corpus by `isReplayableCase`, applies seeded deterministic sampling via a content-hash rank (same discipline as `splitBacktestCorpus`) when the eligible set exceeds `maxFixtures`, and tags era provenance via presence of `rawContextProvenance` in metadata. The logic is straightforward and correctly traced: `eligible.filter((_, index) => kept.has(index))` preserves corpus order while `kept` is chosen by rank-then-index sort, giving the deterministic earlier-wins tie-break the tests pin. Test coverage is thorough (skip accounting sum invariant, era segmentation, under/over-budget sampling, same-seed determinism, cross-seed divergence, forced tie-break, empty corpus) and both the vitest source-import suite and the dist-import workspace suite exercise the barrel export added in `index.ts`.

Nits — 4 non-blocking

• packages/loopover-engine/src/calibration/counterfactual-fixtures.ts:35 — the `8` (hex-slice length) and `16` (radix) are already explained by the surrounding file-header comment, but naming them as constants (e.g. `RANK_HEX_LENGTH`) would make the magic numbers self-documenting without needing to re-read the header.
• No validation that `contract.maxFixtures` is non-negative, but this is an internal contract type populated by the harness, not user input, so it's defensive-only.
• Consider extracting `sampleRank`'s magic numbers (8, 16) into named constants for readability, as flagged by the external brief.
• The `metadata!` non-null assertion in `toFixture` (line ~44) is safe given `isReplayableCase` already guarantees it, but a brief inline comment (already present) is the right call — keep it.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#8220
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 106 registered-repo PR(s), 57 merged, 4 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	kai392; Gittensor profile; 106 PR(s), 4 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Linked issue satisfaction

Addressed
The PR adds a pure assembleCounterfactualFixtures function in packages/loopover-engine/src/calibration/ implementing filtering (diff+label present), seeded deterministic sampling with skip accounting, and provenance segmentation, matching the issue's requirements and deliverables including barrel export and root vitest mirror suite with strong coverage.

Review context

• Author: kai392
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: JavaScript, Kotlin, Perl, Python, TypeScript, Vue
• Official Gittensor activity: 106 PR(s), 4 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Keep the PR focused and include validation evidence before maintainer review.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review