Skip to content

feat(calibration): per-provider reviewer track records from the consensus corpus - #8238

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
kai392:feat/critical-issue-provider-track-records
Jul 23, 2026
Merged

feat(calibration): per-provider reviewer track records from the consensus corpus#8238
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
kai392:feat/critical-issue-provider-track-records

Conversation

@kai392

@kai392 kai392 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add packages/loopover-engine/src/calibration/provider-track-record.ts (Epic: calibration expansion — self-correcting configuration and measured judgment #8211 track F): computeProviderTrackRecords(signals, cases) joins per-provider reviewer signals against the reversal-labeled corpus and reports, per (provider, repo) and per provider overall: {signals, decided, confirmed, reversed, precision|null, agreementRate|null, consensusRate|null, splitRate|null}.
  • Join semantics documented once in the module header and pinned by tests: exact-targetKey join (undecided targets count toward signals but never a rate); precision = P(label confirmed | provider voted fail) — the correct-firing-as-numerator discipline at reviewer grain; agreementRate = symmetric matched-the-human share; consensusRate/splitRate measured only over targets another provider also reviewed, null for one-provider corpora.
  • Null-below-the-sample-floor discipline throughout — a slice that decided nothing (or never voted fail, or shares no targets) reports null, never 0.
  • Providers are opaque ids (none hardcoded); deterministic ordering (providers ascending; overall rollup before repos ascending); aggregates only — an invariant test pins that no target key appears in any returned shape. Vote vocabulary reuses ReviewerConsensusVote from the consensus-calibration module beside it.

Closes #8228

Test plan

  • test/unit/provider-track-record-engine.test.ts (vitest, importing the engine source — the root mirror suite): both-provider consensus + split with per-provider precision/agreement; one-provider null consensus; undecided-target and never-voted-fail null-precision floors; zero-decided null agreement; per-repo→overall rollup equality + deterministic ordering; the no-target-key leak invariant; empty inputs
  • packages/loopover-engine/test/provider-track-record.test.ts (node:test vs dist): barrel re-export + round-trip + null discipline
  • Local coverage on the new file: 55/55 lines, 34/34 branches; engine workspace suite 663 green; npm run typecheck clean
  • CI validate

@kai392
kai392 requested a review from JSONbored as a code owner July 23, 2026 13:49
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.10%. Comparing base (0c8d3d6) to head (bbf74fc).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8238   +/-   ##
=======================================
  Coverage   92.10%   92.10%           
=======================================
  Files         776      777    +1     
  Lines       78328    78387   +59     
  Branches    23668    23683   +15     
=======================================
+ Hits        72147    72202   +55     
  Misses       5062     5062           
- Partials     1119     1123    +4     
Flag Coverage Δ
shard-1 60.33% <0.00%> (+5.41%) ⬆️
shard-2 48.12% <100.00%> (-4.48%) ⬇️
shard-3 53.54% <0.00%> (-1.22%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...er-engine/src/calibration/provider-track-record.ts 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 23, 2026
@loopover-orb

loopover-orb Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 LoopOver is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit f20cdf1 into JSONbored:main Jul 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

calibration: per-provider reviewer track records from the consensus corpus

2 participants