Skip to content

feat(review): ban size and stale-base adjudication at parse time, add the explicit unknown abstention (#8833) - #9592

Merged
JSONbored merged 1 commit into
mainfrom
fix/deterministic-boundary-8833
Jul 28, 2026
Merged

feat(review): ban size and stale-base adjudication at parse time, add the explicit unknown abstention (#8833)#9592
JSONbored merged 1 commit into
mainfrom
fix/deterministic-boundary-8833

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

What

Two more objectively-checkable facts move behind the deterministic boundary, enforced exactly the way the CI-state ban (#8845) and the test-evidence ban already are — demoted at parse time on BOTH the Workers and BYOK paths, so no parse route escapes:

Fact the model can no longer flip Deterministic owner Enforcement
PR size ("too large", "should be split into smaller PRs") sizeGateModeoversized_pr SIZE_CLAIM_PATTERNdemoteSizeClaimBlockers, unconditional
Base staleness / rebase / merge conflicts fetchBaseAheadBystale_base_ref; GitHub's own mergeable_state STALE_BASE_CLAIM_PATTERNdemoteStaleBaseClaimBlockers, unconditional

Both patterns are deliberately claim-shaped: a code-magnitude judgment ("this buffer is too large"), "this function should be split into smaller helpers", or code ABOUT rebasing/conflict-resolution never match — the ban covers assertions about the SUBMISSION and repository state, which are not even visible in a diff.

Explicit abstention (the issue's "unknown → hold" requirement): the rubric now offers "confidence": "unknown" for a model that genuinely cannot judge its blockers. parseReviewConfidence maps it to CONFIDENCE_WHEN_UNSTATED (0.5), under every sane close floor, so an honest abstain routes to the low-confidence hold disposition (a human decides) instead of a guessed verdict.

REVIEW_PROMPT_VERSION bumps to v2 — both edits shape the judge's verdict surface, which the #9477 cache fingerprint (drift = automatic cache miss) and the #8222 counterfactual replay key on.

Tests

  • Pattern both-arm tests: six positive size phrasings + four code-magnitude negatives; seven positive staleness/rebase/conflict phrasings + four code-about-the-topic negatives.
  • Demotion behavior: demoted claims land in nits annotated with the deterministic owner; unrelated blockers untouched; zero-demotion returns the same object (no reallocation), matching every sibling demotion.
  • REGRESSION (parse-path): runWorkersOpinion demotes both claim kinds and logs ai_review_size_claim_demoted / ai_review_stale_base_claim_demoted; the BYOK provider path does the same — the LLM path can no longer flip either fact.
  • Abstention: the rubric offers the exact string the parser maps, and parseReviewConfidence("unknown") lands under the 0.93 default close floor.
  • Prompt-family suites (integrity, cache-input fingerprint, counterfactual replay) green against v2; the deliberate-bump cache test now spoofs v99 so it stays meaningful after this real bump.

Part of #8833 — the structured verifiable-blocker schema ({claim_kind, path, line, evidence_ref}) lands as its own PR next.

… the explicit unknown abstention (#8833)

Two more objectively-checkable facts move behind the deterministic
boundary, enforced the same way the CI-state ban (#8845) and the
test-evidence ban already are -- demoted at parse time on BOTH the
Workers and BYOK paths, so no route escapes:

- SIZE_CLAIM_PATTERN / demoteSizeClaimBlockers: whole-PR size claims
  ('too large', 'should be split') are owned by sizeGateMode ->
  oversized_pr. Anchored on a PR/changeset noun so code-magnitude
  judgments ('this buffer is too large') never match.
- STALE_BASE_CLAIM_PATTERN / demoteStaleBaseClaimBlockers: staleness /
  rebase / merge-conflict claims are owned by stale_base_ref and
  GitHub's own mergeable_state -- none of them is even visible in a
  diff. Claim-shaped so code ABOUT rebasing/conflicts never matches.

The rubric now also OFFERS an explicit abstention: a model that
genuinely cannot judge its blockers sets confidence to the string
"unknown", which parseReviewConfidence maps to CONFIDENCE_WHEN_UNSTATED
(0.5) -- under every sane close floor, so an honest abstain routes to
the low-confidence hold disposition instead of a guessed verdict.

REVIEW_PROMPT_VERSION bumps to v2 (both edits shape the judge's verdict
surface), which the #9477 cache fingerprint and the #8222 counterfactual
replay both key on.

Part of #8833; the structured verifiable-blocker schema lands separately.
@loopover-orb

loopover-orb Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Warning

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-28 21:13:16 UTC

4 files · 1 AI reviewer · 2 blockers · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This PR extends the existing CI-claim/test-evidence-absence demotion pattern to two new claim shapes — whole-PR size claims and base-staleness/rebase/merge-conflict claims — using the identical unconditional demote-to-nit shape, wired into both the Workers and BYOK provider paths, plus adds an explicit "unknown" confidence abstention that already falls through the existing non-number fallback in parseReviewConfidence. The regexes are anchored on submission-noun + claim-verb shapes so code-content sentences ("this buffer is too large", "the rebase helper mutates...") don't false-positive, and both directions are covered by dedicated positive/negative test suites plus end-to-end wiring tests on both call paths. REVIEW_PROMPT_VERSION is correctly bumped to v2 and the two dependent cache/replay tests are updated in lockstep, so no stale-fingerprint gap is introduced.

Nits — 5 non-blocking
  • The nit message baked into demoteSizeClaimBlockers cites `sizeGateMode`/`oversized_pr` as the deterministic owner (src/services/ai-review.ts, demoteSizeClaimBlockers) — that owner isn't visible anywhere in this diff, so it's worth double-checking the field/finding name is accurate before it's surfaced verbatim to contributors.
  • The four new regex constants (SIZE_CLAIM_PATTERN, STALE_BASE_CLAIM_PATTERN) are long single-line alternations with no inline comments breaking down each alternative — consider splitting into named sub-patterns the way test cases already group them (size vs split-phrasing; behind vs rebase vs conflict) for future maintainability.
  • The console.warn blocks in runWorkersOpinion/runProviderReview (src/services/ai-review.ts around the new sizeDemotion/staleBaseDemotion calls) repeat the same four-line pattern four times now (ci, evidence, test-evidence, size, stale-base) — a small helper `warnDemoted(event, count, extra)` would remove the growing duplication.
  • Consider factoring the repeated `if (xDemotion && xDemotion.demoted.length > 0) console.warn(...)` blocks in runWorkersOpinion and runProviderReview into a single small helper now that there are five near-identical instances per call site.
  • If sizeGateMode/oversized_pr isn't the exact live field/finding name, update the demotion nit text in demoteSizeClaimBlockers to match the actual gate-side naming so the surfaced message stays accurate to contributors.

Concerns raised — review before merging

  • No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue: This repo's maintainer focus manifest requires every PR to reference a tracked issue. — Link the relevant issue (for example Closes #123) before opening the PR.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue: This repo's maintainer focus manifest requires every PR to reference a tracked issue. — Link the relevant issue (for example `Closes #123`) before opening the PR.

Decision drivers

  • ❌ Code review — 2 blockers (1 reviewer)
  • ❌ Gate result — Blocking (Repo-configured hard blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 14 registered-repo PR(s), 13 merged, 276 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 14 PR(s), 276 issue(s).
Improvement ✅ Minor risk: clean · value: minor
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is registered but has no active allocation in the current snapshot.
  • Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
  • Official Gittensor activity: 14 PR(s), 276 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Then work through the remaining 3 steps in the Signals table above.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record
  • action: hold · clause: missing_linked_issue
  • config: 844a5e795a8d848738a37ec992189d0e164e822cc111877acbda51bb953e859e · pack: oss-anti-slop · ci: passed
  • record: 2c1bfbc91914442e37b9be93a605bc482a096851293daa516b4301b024da0b89 (schema v5, head fb468c3)

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@JSONbored JSONbored self-assigned this Jul 28, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@github-actions

Copy link
Copy Markdown
Contributor

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (a858cbc) and head (fb468c3) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: linked_issue_scope_mismatch

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.03%. Comparing base (a858cbc) to head (fb468c3).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9592      +/-   ##
==========================================
- Coverage   89.84%   89.03%   -0.81%     
==========================================
  Files         879      879              
  Lines      111071   111095      +24     
  Branches    26439    26449      +10     
==========================================
- Hits        99787    98914     -873     
- Misses       9992    11155    +1163     
+ Partials     1292     1026     -266     
Flag Coverage Δ
backend 94.15% <100.00%> (-1.47%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/services/ai-review.ts 96.88% <100.00%> (+0.09%) ⬆️

... and 3 files with indirect coverage changes

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 28, 2026
@JSONbored
JSONbored merged commit b19111d into main Jul 28, 2026
9 checks passed
@JSONbored
JSONbored deleted the fix/deterministic-boundary-8833 branch July 28, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant