Skip to content

fix(recap): apply the gate-precision noise floor to the recap's aggregate false-positive rates - #9794

Merged
JSONbored merged 1 commit into
JSONbored:mainfrom
shin-core:fix/recap-min-sample-floor-9691
Jul 29, 2026
Merged

fix(recap): apply the gate-precision noise floor to the recap's aggregate false-positive rates#9794
JSONbored merged 1 commit into
JSONbored:mainfrom
shin-core:fix/recap-min-sample-floor-9691

Conversation

@shin-core

Copy link
Copy Markdown
Contributor

What & why

services/gate-precision.ts nulls a false-positive rate below MIN_SAMPLE = 5 blocks — "a 1-of-1 false positive is noise". buildMaintainerRecap (src/services/maintainer-recap.ts) folds per-repo precision reports into a cross-repo RecapReport and applied that rule inconsistently inside one function:

  • Per-repo path copies overall.falsePositiveRate straight out of the (already-floored) GatePrecisionReport — correct.
  • Aggregate path re-derived totals.gateFalsePositiveRate and the miner/human cohort aggregates with a bare blocked > 0 guard — unfloored.

So one digest could read, over the same window and the same totals: summary "Gate false-positive rate: 100% (2/2 block(s) later merged)"; Totals "Gate false positives: 2/2 (100%)"; and Gate-outcomes "False-positive rate: n/a (fewer than 5 blocks…)". The maintainer sees "the gate is 100% wrong" and "not enough data to say" simultaneously. MIN_SAMPLE = 5 was a private const duplicated in two files, which is what let this third consumer be written without it.

The fix

  • Export the floor once as MIN_GATE_PRECISION_SAMPLE = 5 from gate-precision.ts (replacing its private MIN_SAMPLE).
  • maintainer-recap-gate-outcomes.ts imports it and deletes its duplicate; its rendered "fewer than N blocks" copy uses the imported value.
  • buildMaintainerRecap gates the blended rate and each cohort aggregate on blocked >= MIN_GATE_PRECISION_SAMPLE against its own denominator (not the blended one). The below-floor case subsumes the old blocked === 0 divide-by-zero guard; the two rate-line arms and the number | null type are unchanged.

MIN_SAMPLE/5 now lives in exactly one place (grep-verifiable) — the next consumer of RecapReport.totals inherits it.

Tests

test/unit/maintainer-recap.test.ts:

  • Two repos each blocked 1 / blockedThenMerged 1totals.gateFalsePositiveRate === null (named regression; fails on main).
  • Blended blocked 5, fp 10.2 (floor inclusive at 5).
  • A cohort below its own floor is null even when the blended blocked is above it; a cohort at/above its floor still reports (both >=-arms covered).
  • Updated the existing below-floor cohort assertions (blocked 2/3/4) to null, and aligned the repoInput fixture's cohortReport to gate-precision's real >= 5 floor.

test/unit/maintainer-recap-format.test.ts:

  • A below-floor report renders - Gate false positives: 2/2 (n/a) and the Gate-outcomes "fewer than 5 blocks" line, with no percentage anywhere (Deliverable 5).

Validation

  • npm run typecheck green; all four affected suites (58 tests) green.
  • Diff coverage on all three changed src files is 100% line and branch (both floor arms per rate).
  • git diff --check <base> HEAD clean; no route/schema/migration change (RecapReport type unchanged).

Closes #9691

…gate false-positive rates

`buildMaintainerRecap` re-derived `totals.gateFalsePositiveRate` (and the miner/
human cohort aggregates) with a bare `blocked > 0` guard, while the per-repo path
and the Gate-outcomes section both null the rate below MIN_SAMPLE (5) as
gate-precision does. So one digest could read "Gate false-positive rate: 100%
(2/2)" in its summary and Totals while its own Gate-outcomes section said
"n/a (fewer than 5 blocks)" — the maintainer sees "the gate is 100% wrong" and
"not enough data" about the same window, from the same totals struct.

Export the floor once as `MIN_GATE_PRECISION_SAMPLE` from gate-precision.ts
(replacing its private `MIN_SAMPLE`), import it in maintainer-recap-gate-outcomes.ts
(deleting the duplicate) and maintainer-recap.ts, and gate the blended rate and
each cohort aggregate on `blocked >= MIN_GATE_PRECISION_SAMPLE` against their own
denominator. The below-floor case subsumes the old divide-by-zero guard; the
rate-line arms and the `number | null` type are unchanged. The 5 now lives in one
place, so the next consumer of RecapReport.totals inherits it.

Closes JSONbored#9691
@shin-core
shin-core requested a review from JSONbored as a code owner July 29, 2026 09:06
@superagent-security superagent-security Bot added the contributor:flagged Contributor flagged for review by trust analysis. label Jul 29, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

🚨 Contributor flagged. Click here for more info: Superagent Dashboard

@loopover-orb

loopover-orb Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Warning

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-29 10:28:38 UTC

5 files · 1 AI reviewer · no blockers · CI green · unstable

⏸️ Suggested Action - Manual Review

  • AI review did not produce public notes: The configured AI reviewer returned no usable public assessment for this PR head.

Review summary
AI review is unavailable for this PR head. LoopOver is holding this PR for manual review until the configured AI provider returns a usable public review summary.

Nits — 1 non-blocking
  • AI review did not produce public notes — Fix the configured AI provider, then re-run LoopOver review before relying on the result.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ⚠️ Gate result — Not blocking (Advisory; not blocking this PR.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #9691
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ❌ 5/25 Preflight is holding this PR: the review lane is unavailable, so it is not ready for automated review.
Contributor workload ✅ 10/10 Author activity: 60 registered-repo PR(s), 43 merged, 0 issue(s).
Contributor context ✅ Confirmed Gittensor contributor shin-core; Gittensor profile; 60 PR(s), 0 issue(s).
Improvement ✅ Minor risk: clean · value: minor
Review context
  • Author: shin-core
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is registered but has no active allocation in the current snapshot.
  • Public profile languages: TypeScript, JavaScript, Solidity, Dart, Python, CSS, PHP, Rust
  • Official Gittensor activity: 60 PR(s), 0 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Await review-lane availability.
  • Then work through the remaining 1 step in the Signals table above.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record
  • action: hold · clause: ai_review_inconclusive
  • config: 683cad1c4ae6ca9f40cb11543ec9d21efdaadeb060e3e6351045c0506d5257ec · pack: oss-anti-slop · ci: passed
  • record: f4ef1f83c1efe666d71401a60b422e68b00d36dccff5d66da6c6eaf319f213a2 (schema v5, head d04a613)

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.65%. Comparing base (fef8a8a) to head (d04a613).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9794      +/-   ##
==========================================
+ Coverage   76.58%   76.65%   +0.06%     
==========================================
  Files         282      285       +3     
  Lines       59464    59620     +156     
  Branches     6555     6605      +50     
==========================================
+ Hits        45543    45699     +156     
  Misses      13639    13639              
  Partials      282      282              
Flag Coverage Δ
backend 100.00% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/services/gate-precision.ts 100.00% <100.00%> (ø)
src/services/maintainer-recap-gate-outcomes.ts 100.00% <100.00%> (ø)
src/services/maintainer-recap.ts 100.00% <100.00%> (ø)

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 29, 2026
@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 29, 2026
@JSONbored
JSONbored merged commit afb3b84 into JSONbored:main Jul 29, 2026
7 of 8 checks passed
JSONbored added a commit that referenced this pull request Jul 29, 2026
…vider failure (#9806)

A review that PARSED cleanly but had every public field withheld by the
sanitizer made composeAdvisoryNotes return null, which the orchestration
misreports as "AI review is unavailable for this PR head" -- a PROVIDER
failure -- and holds the PR for manual review. Because the head is unchanged,
every re-run reproduces it: observed live on #9794, which
re-ran with diagnostics `claude-code#0:parsed` and was re-held every time,
including after explicit maintainer re-ticks.

The shape is routine for this project's own scoring/gate code: an honest
narrative about it says "score"/"ranking"/"reward" in every sentence, the
per-sentence sanitizer drops them all, and a clean review (no blockers, no
nits) has nothing left to publish.

Publish a fixed, public-safe sentence instead, worded by the review's REAL
verdict: "found no blocking issues" only when the model returned no blockers;
"raised blocking findings ... read the private review record" when it did --
withheld blockers must never read as a clean result. A truly empty parse (no
assessment at all) still returns null, so a genuine provider failure keeps its
accurate report. One legacy test pinned the old null; updated with the incident
reference.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor:flagged Contributor flagged for review by trust analysis. gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

orb(recap): apply the gate-precision MIN_SAMPLE floor to the recap's aggregate false-positive rates

2 participants