Skip to content

fix(engine): guard benchmark ground-truth against invalid window and duplicate roster - #9882

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
RealDiligent:fix/critical-issue-benchmark-groundtruth-guards-9644
Jul 29, 2026
Merged

fix(engine): guard benchmark ground-truth against invalid window and duplicate roster#9882
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
RealDiligent:fix/critical-issue-benchmark-groundtruth-guards-9644

Conversation

@RealDiligent

Copy link
Copy Markdown
Contributor

Summary

benchmarkHorizonEnd and deriveBenchmarkGroundTruth (packages/loopover-engine/src/calibration/benchmark-ground-truth.ts) — the pure core that produces the labels every candidate proposal is scored against — mishandled two inputs their PURE … STABILITY contract does not cover:

  1. Unparseable frozenAt / non-finite horizonDaysnew Date(Date.parse(frozenAt) + horizonDays * MS_PER_DAY).toISOString() throws a bare RangeError: Invalid time value for NaN, an uncaught exception in the harness rather than a named structural failure.
  2. Duplicated workUnitId — the roster loop emitted one truth per element, so a duplicate produced two identical truths and double-counted the unit in workUnits, unresolved, and the unresolvedRate denominator.

Fix

  • benchmarkHorizonEnd now uses negated-compound guards (if (!Number.isFinite(x))) throwing invalid_frozen_at: <value> and invalid_horizon_days: <value>, mirroring splitBacktestCorpus's invalid_held_out_fraction style so NaN fails closed.
  • deriveBenchmarkGroundTruth calls the guarded benchmarkHorizonEnd before any other work, then rejects a duplicated roster entry with invalid_duplicate_work_unit_id: <id> rather than silently de-duplicating a corrupted snapshot (benchmark: frozen-repo snapshot builder — leak-proof repo state at commit T #9259).
  • The module header documents all three rejection cases alongside its existing three rules.

No production logic changed: formulas, thresholds, vocabularies, and valid-input output are byte-identical (asserted by an additive-guards test).

Tests

Added to packages/loopover-engine/test/benchmark-ground-truth.test.ts (the engine Codecov flag's own node:test suite): named invalid_frozen_at / invalid_horizon_days throws including NaN and Infinity, propagation through deriveBenchmarkGroundTruth, the invalid_duplicate_work_unit_id rejection, and an additive-guards test asserting a valid input still yields the exact same BenchmarkGroundTruthSet. Full suite passes (16/16).

Closes #9644

@RealDiligent
RealDiligent requested a review from JSONbored as a code owner July 29, 2026 18:34
@loopover-orb

loopover-orb Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Warning

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-29 18:58:03 UTC

2 files · 1 AI reviewer · no blockers · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This PR adds fail-closed input validation to `benchmarkHorizonEnd` and `deriveBenchmarkGroundTruth` in the pure ground-truth core: a non-finite `frozenAt`/`horizonDays` now throws a named `invalid_frozen_at`/`invalid_horizon_days` error instead of producing a bare `RangeError` from `new Date(NaN).toISOString()`, and a duplicated `workUnitId` in the roster now throws `invalid_duplicate_work_unit_id` instead of silently double-counting the unit in `workUnits`/`unresolved`/`unresolvedRate`. Both failure conditions are genuinely reachable on the stated inputs (invalid date strings, `NaN`/`Infinity` horizonDays, duplicate roster entries), the guards are ordered correctly (window guard before any other computation, duplicate check before the settlement loop), and the additive-guards test confirms valid-input output is unchanged. The change is narrowly scoped to the two documented defects, closes #9644, and is well-covered by new tests exercising both throw paths and their propagation through the public function.

Nits — 4 non-blocking
  • nit: the test file imports from `../dist/index.js` (built output) rather than the source module — fine given CI passed, but worth a comment noting the build step is a prerequisite for local test runs (packages/loopover-engine/test/benchmark-ground-truth.test.ts:4).
  • nit: `benchmarkHorizonEnd`'s two guards could be combined into one early return with a switch on which field is invalid, but the current explicit two-guard form is more readable and matches the sibling `splitBacktestCorpus` style, so this is a non-issue — just noting the negated-compound style is consistent (benchmark-ground-truth.ts:118-123).
  • Consider also asserting in the duplicate-roster test that the roster-position of the duplicate (not just the id) doesn't affect which occurrence is reported, for extra clarity in the error message contract.
  • The module header changes are thorough — consider a one-line pointer in the header to the new tests' names (`invalid_frozen_at`, `invalid_horizon_days`, `invalid_duplicate_work_unit_id`) so future readers can grep straight to them.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #9644
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ❌ 5/25 Preflight is holding this PR: the review lane is unavailable, so it is not ready for automated review.
Contributor workload ✅ 10/10 Author activity: 251 registered-repo PR(s), 99 merged, 38 issue(s).
Contributor context ✅ Confirmed Gittensor contributor RealDiligent; Gittensor profile; 251 PR(s), 38 issue(s).
Improvement ✅ Minor risk: clean · value: minor
Linked issue satisfaction

Addressed
The diff adds the named guards for invalid frozenAt and non-finite horizonDays in benchmarkHorizonEnd, moves the horizon-end computation before other work in deriveBenchmarkGroundTruth, rejects duplicate workUnitIds with invalid_duplicate_work_unit_id, documents all three cases in the module header, and includes tests for each throwing/passing arm plus an additive byte-identical output test, match

Review context
  • Author: RealDiligent
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is registered but has no active allocation in the current snapshot.
  • Public profile languages: Python, Ruby, TypeScript, Svelte, Cuda, JavaScript, Markdown, MDX
  • Official Gittensor activity: 251 PR(s), 38 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Await review-lane availability.
  • Then work through the remaining 2 steps in the Signals table above.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

…duplicate roster

benchmarkHorizonEnd threw a bare RangeError for an unparseable frozenAt or a
non-finite horizonDays (new Date(NaN).toISOString()), and deriveBenchmarkGroundTruth
silently emitted two truths for a duplicated workUnitId, inflating workUnits,
unresolved, and unresolvedRate's denominator.

Add negated-compound guards throwing invalid_frozen_at / invalid_horizon_days in
benchmarkHorizonEnd (called before any other work), and reject a duplicated roster
entry with invalid_duplicate_work_unit_id rather than de-duplicating a corrupted
snapshot. Production formulas, thresholds, and valid-input output are unchanged.

Closes JSONbored#9644
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.28%. Comparing base (8e2d51d) to head (6779a31).
⚠️ Report is 2 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (8e2d51d) and HEAD (6779a31). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (8e2d51d) HEAD (6779a31)
backend 1 0
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #9882       +/-   ##
===========================================
- Coverage   91.75%   79.28%   -12.48%     
===========================================
  Files         919      281      -638     
  Lines      112987    58510    -54477     
  Branches    27193     6777    -20416     
===========================================
- Hits       103667    46387    -57280     
- Misses       8034    11840     +3806     
+ Partials     1286      283     -1003     
Flag Coverage Δ
backend ?
engine 71.83% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...r-engine/src/calibration/benchmark-ground-truth.ts 100.00% <100.00%> (ø)

... and 771 files with indirect coverage changes

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit 300c381 into JSONbored:main Jul 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

benchmark(ground-truth): benchmarkHorizonEnd throws on an unparseable `frozenAt

1 participant