Skip to content

fix(review): report a withheld narrative honestly instead of as a provider failure - #9806

Merged
JSONbored merged 1 commit into
mainfrom
fix/withheld-narrative-honesty
Jul 29, 2026
Merged

fix(review): report a withheld narrative honestly instead of as a provider failure#9806
JSONbored merged 1 commit into
mainfrom
fix/withheld-narrative-honesty

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

The bug (the last of today's review-reliability incidents)

JSONbored/loopover#9794 produced "AI review is unavailable for this PR head" on every run — including forced re-ticks, including after the beta.4 deploy — while its diagnostics said claude-code#0:parsed. The provider was fine every time.

The real path: the model reviewed scoring/proof code, found no blockers and no nits, so its entire output was the narrative — and an honest narrative about this project's own gate/scoring code says "score"/"ranking"/"reward" in every sentence. The per-sentence sanitizer withheld all of them, composeAdvisoryNotes returned null, and the orchestration's null-branch misattributed that to the provider, held the PR, and — the head being unchanged — reproduced identically on every re-run. A permanently wedged PR whose review succeeded every single time.

The fix

When the parse produced a narrative but nothing survived sanitization, publish a fixed, public-safe sentence instead of null — no model text, so the never-echo guarantee is untouched — worded by the review's real verdict:

  • no raw blockers → "completed and found no blocking issues; narrative withheld (non-public project internals)"
  • raw blockers all withheld → "raised blocking findings … withheld; a maintainer should read the private review record" — withheld blockers must never read as a clean result (pinned by an invariant test)

A truly empty parse (no assessment at all) still returns null, so a genuine provider failure keeps its accurate "unavailable" report.

One legacy test explicitly pinned the old null behavior — the exact behavior this incident traces to — and is updated with the incident reference.

Tests

286 pass in test/unit/ai-review.test.ts, including the #9794 regression (all-withheld clean review → honest note, hasPublicReviewAssessment true), the withheld-blockers invariant, and the true-provider-failure null case. tsc clean.

With this, every failure mode from today's investigation has a landed fix: cadence downgrade + false "skipped" bullet (#9800), restart-orphaned reviews (#9802), and this misattribution. Next image picks up all three.

…vider failure

A review that PARSED cleanly but had every public field withheld by the
sanitizer made composeAdvisoryNotes return null, which the orchestration
misreports as "AI review is unavailable for this PR head" -- a PROVIDER
failure -- and holds the PR for manual review. Because the head is unchanged,
every re-run reproduces it: observed live on #9794, which
re-ran with diagnostics `claude-code#0:parsed` and was re-held every time,
including after explicit maintainer re-ticks.

The shape is routine for this project's own scoring/gate code: an honest
narrative about it says "score"/"ranking"/"reward" in every sentence, the
per-sentence sanitizer drops them all, and a clean review (no blockers, no
nits) has nothing left to publish.

Publish a fixed, public-safe sentence instead, worded by the review's REAL
verdict: "found no blocking issues" only when the model returned no blockers;
"raised blocking findings ... read the private review record" when it did --
withheld blockers must never read as a clean result. A truly empty parse (no
assessment at all) still returns null, so a genuine provider failure keeps its
accurate report. One legacy test pinned the old null; updated with the incident
reference.
@JSONbored JSONbored self-assigned this Jul 29, 2026
@loopover-orb

loopover-orb Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Important

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏳ LoopOver is waiting…

LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.51%. Comparing base (ace8d1a) to head (73430bd).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9806      +/-   ##
==========================================
- Coverage   90.33%   89.51%   -0.82%     
==========================================
  Files         918      918              
  Lines      113935   113938       +3     
  Branches    26974    26976       +2     
==========================================
- Hits       102925   101997     -928     
- Misses       9681    10850    +1169     
+ Partials     1329     1091     -238     
Flag Coverage Δ
backend 94.09% <100.00%> (-1.48%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/services/ai-review.ts 96.95% <100.00%> (-0.11%) ⬇️

... and 3 files with indirect coverage changes

@JSONbored
JSONbored merged commit fa4c6d4 into main Jul 29, 2026
9 checks passed
@JSONbored
JSONbored deleted the fix/withheld-narrative-honesty branch July 29, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant