feat(benchmark): reversal-aware realized-history ground truth (#9261)

[JSONbored]

Stacked on #9595 (feat/benchmark-action-space-9260), which it imports the action vocabulary from. Based against that branch so the diff shows only this change; retarget to main once #9595 merges.

What

The labels every candidate proposal is scored against: for each work unit in a snapshot, what the maintainer actually did within the prediction horizon — emitted in #9260's exact action vocabulary, so proposal and outcome compare with no translation layer that could drift.

Three rules the surrounding system already believes, now enforced in one place:

• Reversal-aware settlement. A merge later reverted is not a clean merge. The label is the settled state at horizon end with the reversal recorded alongside — the same treatment that makes public-rule-precision.ts's precision honest (it subtracts reversals rather than erasing the decision). The vocabulary is the established reversal_reopened / reversal_reverted / reversal_superseded set, never a benchmark-local re-invention, so benchmark ground truth and internal backtest ground truth cannot disagree about the same event. A reversal counts only inside the horizon and strictly after the action it overturns; the earliest such reversal wins.
• Unresolved is explicit and excluded. A work unit with no action inside the horizon is a first-class unresolved — never silently a correct abstention (which would reward an agent for declining exactly the cases nobody decided) nor an incorrect prediction (which would punish it for the horizon being too short). It leaves the scoring denominator, and the unresolved rate is published so a climbing rate reads as "the horizon is wrong" rather than as a scoring artifact. scoreableGroundTruths is the single definition of that rule, so no scorer can re-implement it slightly differently.
• Stability. Every decision is a pure function of events dated inside the window, so a later event — including a later reversal — cannot retroactively move a settled label. A leaderboard that drifts under its own scores is not one.

The settled state is the last qualifying action in the window (label → request changes → merge settles on the merge), with identical timestamps keeping the earlier-listed event so a stable input order yields a stable label.

Tests (12 node:test cases, 100% of branches)

Settlement ordering and per-action parameter carriage (a merge truth carries no reasonClass/labels keys at all — not present-but-undefined); the reverted-merge regression; all three reversal kinds accepted verbatim with earliest-wins; the four ways a reversal fails to attach (before the action, same instant, past the horizon, before the freeze) plus wrong work unit; the unresolved regression with in/out-of-window events in both directions; inclusive boundaries at both ends; the stability invariant (same input plus a year of post-horizon history → byte-identical output); unparseable timestamps as out-of-window rather than a crash; tie-break determinism; empty roster reporting a null rate rather than 0; and parameters dropped when they don't belong to the settled action.

Closes #9261

[loopover-orb]

Tip

✅ LoopOver review result - approve/merge recommended

_{Review updated: 2026-07-28 23:16:40 UTC}

3 files · 1 AI reviewer · no blockers · readiness 98/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
Adds a pure, well-tested module deriving reversal-aware, unresolved-excluding ground truth for the benchmark, matching the vocabulary from benchmark-proposal.ts and the reversal semantics from public-rule-precision.ts. The core logic (last-event settlement, earliest-after-settlement reversal, inclusive horizon boundaries, exactOptionalPropertyTypes-safe spreads) is sound and the test suite directly exercises every branch including stability, tie-breaking, and boundary inclusivity. No correctness issues found in the traced logic.

Nits — 6 non-blocking

• deriveBenchmarkGroundTruth (benchmark-ground-truth.ts:126-186) nests to depth 5 per the flagged control-flow metric; consider extracting the reversalFor closure's inner loop or the settle-map-building loop into named helpers for readability.
• withinHorizon (benchmark-ground-truth.ts:110) and reversalFor (benchmark-ground-truth.ts:150) each call Date.parse repeatedly on the same strings inside loops — fine at benchmark scale but worth a comment if this ever runs over large event volumes.
• The PR description says it is stacked on feat(benchmark): maintainer action-space schema and proposal validator (#9260) #9595 for the BenchmarkActionKind/BenchmarkCloseReasonClass import — confirm those types still exist under that name once feat(benchmark): maintainer action-space schema and proposal validator (#9260) #9595 lands and this is retargeted to main, since the diff itself can't verify that dependency.
• Consider extracting the per-work-unit settle/reversal derivation in deriveBenchmarkGroundTruth into a small helper to reduce nesting depth at benchmark-ground-truth.ts:126-186.
• The CI failure risk here is more likely explained by the 1-commit-behind base branch status than a defect in this diff — worth confirming once CI finishes.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#9261
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 14 registered-repo PR(s), 13 merged, 310 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 14 PR(s), 310 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Linked issue satisfaction

Partially addressed
The pure-core extractor is thorough and clearly implements reversal-aware settlement and the excluded 'unresolved' class with strong test coverage, satisfying requirements 3 and 4 well. However, the issue's requirement 2 calls for reusing the existing provenance logic from backtest-corpus.ts rather than reimplementing it, and this diff only defines a parallel type union with matching literal strin

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is registered but has no active allocation in the current snapshot.
• Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
• Official Gittensor activity: 14 PR(s), 310 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Then work through the remaining 1 step in the Signals table above.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record

• action: merge · clause: success
• config: a4f18352a69e3bb1f5dbe18798f3cfd5299dc4d8be2b812266b8be2d6307f82d · pack: oss-anti-slop · ci: passed
• record: 21f83dba060e77002e22bbbd73c0bcb31757cc9858f00eddadefb600b3afaea9 (schema v5, head d051556)

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	loopover-ui	d051556	Commit Preview URL Branch Preview URL	Jul 28 2026, 10:58 PM

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.89%. Comparing base (b543e1e) to head (d051556).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9598      +/-   ##
==========================================
+ Coverage   89.87%   89.89%   +0.01%     
==========================================
  Files         882      883       +1     
  Lines      111488   111692     +204     
  Branches    26561    26579      +18     
==========================================
+ Hits       100204   100404     +200     
  Misses       9992     9992              
- Partials     1292     1296       +4     

Flag	Coverage Δ
backend	95.44% <2.94%> (-0.06%)	⬇️
engine	66.35% <100.00%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...r-engine/src/calibration/benchmark-ground-truth.ts	100.00% <100.00%> (ø)
packages/loopover-engine/src/index.ts	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

[loopover-orb]

LoopOver approves — the gate is satisfied and CI is green.

[loopover-orb]

LoopOver approves — the gate is satisfied and CI is green.