feat(calibration): claimed-confidence reliability curves with derived threshold suggestion - #8252
Conversation
… threshold suggestion Bucket a rule's decided BacktestCases by claimed confidence (metadata.confidence) into fixed 0.05-step edges and report per-bucket empirical precision (null below the sample floor, never 0), then derive the loosest floor whose at-or-above pooled precision meets a target -- never below the hard minimum, null on insufficient pooled density. Engine-exported; no consumer changes (JSONbored#8226, epic JSONbored#8211 track E).
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8252 +/- ##
==========================================
- Coverage 92.12% 92.12% -0.01%
==========================================
Files 782 783 +1
Lines 78531 78580 +49
Branches 23720 23738 +18
==========================================
+ Hits 72347 72392 +45
Misses 5062 5062
- Partials 1122 1126 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
loopover-orb
Bot
added
the
gittensor:feature
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-23 15:13:23 UTC
Review summary Nits — 5 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
![loopover-orb[bot]](https://avatars.githubusercontent.com/in/4139483?s=60&v=4){kind=small}
2 participants
Summary
packages/loopover-engine/src/calibration/reliability-curve.ts— per-rule reliability curves that let an optimal confidence floor FALL OUT of the labeled corpus instead of being guessed from hand-picked candidate ladders (theloosening-knobs.tsregistry's current shape).computeReliabilityCurve(cases, bucketEdges?, sampleFloor?): buckets decided cases by their CLAIMED confidence (metadata.confidence, the same channelbuildConfidenceThresholdClassifierreads, Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138) and reports per-bucket{floor, ceiling, cases, confirmed, reversed, precision|null}. The documentedDEFAULT_RELIABILITY_BUCKET_EDGES([0, 0.3, 0.35, …, 0.95, 1]) step 0.05-wide across [0.3, 1] — the same granularity the registry's candidate ladders step at, so every adoptable floor (both hard minimums, 0.3 and 0.85, included) is exactly a bucket floor — with one catch-all below 0.3; the top bucket is ceiling-inclusive so a claimed confidence of exactly 1 is bucketable. A case with no numeric claimed confidence is dropped, never fabricated as confidence 1 — a deliberate, documented divergence from Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138's degrade-to-1 classifier fallback (that function must decide every case; this one measures claim reliability).deriveThresholdSuggestion(curve, targetPrecision, hardMinimum): the LOOSEST bucket floor at or abovehardMinimumwhose at-or-above buckets' pooled precision meetstargetPrecision; null when no candidate qualifies or pooled density is under the curve's sample floor. Deterministic and conservative: a suggestion is never clamped up to a floor whose own pooled evidence wasn't checked.RELIABILITY_BUCKET_SAMPLE_FLOOR, 5 — the registry's smallest never-on-noise floor) reports null precision, never 0, and an under-density pooled window can never qualify.Scope
type(scope): short summaryConventional Commit format.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #8226).Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally (unsharded) — 100% of the diff's lines AND branches (reliability-curve.ts: 46/46 lines, 42/42 branches, 5/5 functions, verified via lcov)npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateIf any required check was skipped, explain why:
npm run test:cichain was run end-to-end (includes all of the above plus the engine workspace's ownnode --testsuite, worker-pool tests, parity/drift checks, and the UI chain).Safety
UI Evidencesection — not applicable, no visible UI/frontend/docs change.UI Evidence
Not applicable — a pure calibration-engine module with no visible UI, frontend, docs, or extension change.
Notes
packages/loopover-engine/test/reliability-curve.test.ts(node:testagainstdist, gates the engine workspace's ownnpm run test) andtest/unit/reliability-curve-engine.test.ts(the root vitest mirror importing the enginesrcpath directly — the suite that providescodecov/patchcoverage). Coverage includes bucket boundaries (floor-inclusive interior edges, ceiling-inclusive top), every drop path (no/non-numeric/out-of-range/NaN claims), both sides of the sample floor, pooled-density and hard-minimum arms of the suggestion, all throw guards, and a monotonicity invariant (a rising target never loosens the suggestion; a suggestion is never below the hard minimum and is always a real bucket floor).