Skip to content

fix(review): stop retrying an AI provider 429 immediately with zero backoff#5481

Merged
JSONbored merged 2 commits into
mainfrom
fix/ai-provider-429-backoff
Jul 12, 2026
Merged

fix(review): stop retrying an AI provider 429 immediately with zero backoff#5481
JSONbored merged 2 commits into
mainfrom
fix/ai-provider-429-backoff

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Sentry issues GITTENSORY-K (selfhost_ai_provider_failed: claude_code_error_429, 2090 occurrences over 2 weeks) and GITTENSORY-8 (ai_review_provider_exhausted: claude_code_error_429, 328 occurrences) are a genuine external Anthropic rate limit — that part is expected. But every retry loop that calls env.AI.run() for the claude-code provider (runWorkersOpinion, runDualAiTieBreakJudgeCall, runWorkersSlopOpinion, runPlannerModel) retries the SAME model immediately with zero backoff on a 429, even though the rate-limit window it just hit cannot possibly have cleared a few hundred milliseconds later — burning the whole per-model attempt budget for zero additional chance of success, and delaying (or, for the tie-break/slop/planner loops, which had no special-casing at all before this) the fallback model's turn.

Changes

  • Added isRateLimitError (exported from src/services/ai-review.ts, matching claude_code_error_429 / ai_http_429 / anthropic_http_429), shared by ai-slop.ts and planner.ts instead of each re-deriving its own copy.
  • Applied the same short-circuit runWorkersOpinion already uses for a non-transient CLI timeout (isSubscriptionCliTimeout) to a rate-limit error too: break out of the current model's retry loop and move straight to the fallback, which may be on a different provider/account entirely.
  • Applied to all 4 affected retry loops: runWorkersOpinion and runDualAiTieBreakJudgeCall (src/services/ai-review.ts), runWorkersSlopOpinion (src/services/ai-slop.ts), runPlannerModel (src/review/planner.ts).
  • src/services/ai-summaries.ts has no retry loop at all today (single unretried attempt) — left out of scope for this fix; flagged as a possible separate follow-up.

Test plan

  • npm run typecheck — clean
  • npm run test:ci (full local gate, unsharded coverage) — green
  • Added regression tests for all 4 retry loops proving a 429 stops after ONE attempt per model (not the full 2-3x budget), while a genuinely transient (non-429, non-timeout) error still retries to the full budget
  • Manually verified (before writing the fix) that reverting it makes the new tests fail with the exact old call-count behavior

Fixes GITTENSORY-K
Fixes GITTENSORY-8

…ackoff

runWorkersOpinion, runDualAiTieBreakJudgeCall, runWorkersSlopOpinion, and
runPlannerModel each retry a failed model call up to 3x (2x for the planner)
before falling through to the fallback model -- but a rate-limit error
(claude_code_error_429 / ai_http_429 / anthropic_http_429) will not have
cleared by the next attempt a few hundred ms later, so retrying the SAME
model burns the whole per-model attempt budget for zero additional chance of
success, delaying (or in the tie-break/slop/planner loops, which had no
special-casing at all, fully exhausting before) the fallback model ever gets
a turn.

Adds isRateLimitError (exported from ai-review.ts, shared by ai-slop.ts and
planner.ts) and applies the same short-circuit runWorkersOpinion already
uses for a non-transient CLI timeout: break out of the current model's retry
loop on a 429 and move straight to the fallback, which may be on a different
provider/account entirely.

Fixes GITTENSORY-K
Fixes GITTENSORY-8
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.73%. Comparing base (849e11e) to head (51fda97).
⚠️ Report is 10 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5481   +/-   ##
=======================================
  Coverage   94.73%   94.73%           
=======================================
  Files         561      561           
  Lines       44794    44798    +4     
  Branches    14667    14671    +4     
=======================================
+ Hits        42437    42441    +4     
  Misses       1622     1622           
  Partials      735      735           
Flag Coverage Δ
shard-1 43.78% <0.00%> (-0.51%) ⬇️
shard-2 35.39% <50.00%> (+0.11%) ⬆️
shard-3 32.11% <50.00%> (+0.08%) ⬆️
shard-4 31.30% <33.33%> (-0.09%) ⬇️
shard-5 33.10% <16.66%> (-0.12%) ⬇️
shard-6 43.86% <0.00%> (+0.27%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/review/planner.ts 100.00% <100.00%> (ø)
src/services/ai-review.ts 96.74% <100.00%> (+<0.01%) ⬆️
src/services/ai-slop.ts 93.50% <100.00%> (+0.08%) ⬆️
src/services/linked-issue-satisfaction-run.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 12, 2026
@loopover-orb

loopover-orb Bot commented Jul 12, 2026

Copy link
Copy Markdown

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-12 23:15:26 UTC

6 files · 1 AI reviewer · 2 blockers · readiness 93/100 · CI green · unknown

⏸️ Suggested Action - Manual Review

Review summary
This adds a shared isRateLimitError helper (matching claude_code_error_429 / ai_http_429 / anthropic_http_429) and wires it into the same break-on-non-transient-error short-circuit already used for isSubscriptionCliTimeout, applied uniformly across the four affected retry loops (runWorkersOpinion, runDualAiTieBreakJudgeCall, runWorkersSlopOpinion, runPlannerModel). The regex `/_(?:http|error)_429$/` correctly matches all three cited error-message shapes, and each site is a straightforward one-line addition to an existing catch block that preserves the fall-through-to-fallback-model behavior. Each of the four loops has a corresponding regression test asserting exactly one attempt on the rate-limited model before falling to the fallback, and the pre-existing 'genuinely transient error still retries the full budget' test is retained/renamed to explicitly exclude 429, so both branches (429 short-circuit vs. other transient error retrying fully) are covered.

Blockers

  • src/services/linked-issue-satisfaction-run.ts:55 still catches every `env.AI.run()` failure and continues the same model's retry loop, so a reachable `claude_code_error_429` in `runWorkersSatisfactionOpinion` will still burn all per-model attempts before falling back; apply the shared guard there too, e.g. `} catch (error) { if (isRateLimitError(error)) break; }`, and add the same real-path regression coverage.
Nits — 5 non-blocking
  • src/services/ai-review.ts: the 429 detection is a bare regex `/_(?:http|error)_429$/` with no named constant for the magic `429`; a small named regex/constant would make the three matched error-shapes self-documenting at the call site.
  • The fix isn't tied to a linked GitHub issue in the PR description — only Sentry issue links (GITTENSORY-K/8) are cited; worth confirming there's an eligible tracked issue per repo convention even though this is a narrow, well-justified bug fix rather than scope creep.
  • src/services/ai-slop.ts and src/review/planner.ts catch blocks now reference `error` but the diff doesn't show whether `error: unknown` typing conflicts with existing lint rules for unused catch bindings in other files — worth a quick lint check (though CI is already green here).
  • Consider extracting the 429-matching error message suffixes into a small array/constant (e.g. RATE_LIMIT_ERROR_SUFFIXES) for readability, per the external brief's magic-number flag at src/services/ai-review.ts:1013.
  • Since src/services/ai-summaries.ts is explicitly called out as out-of-scope (no retry loop today), consider filing the follow-up issue now while the context is fresh, per the PR description's own suggestion.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue — Link the relevant issue (for example `Closes #123`) before opening the PR.
Signal Result Evidence
Code review ❌ 2 blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 44 registered-repo PR(s), 36 merged, 424 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 44 PR(s), 424 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Improvement ✅ Minor risk: clean · value: minor — Code changes are accompanied by test evidence.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 44 PR(s), 424 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 12, 2026
…nion too

Gittensory review flagged that linked-issue-satisfaction-run.ts's own
env.AI.run() retry loop was missed by the original fix -- it still
burned its full per-model attempt budget on a claude_code_error_429
before falling back, same as the four loops already fixed.
@JSONbored
JSONbored merged commit c533294 into main Jul 12, 2026
12 checks passed
@JSONbored
JSONbored deleted the fix/ai-provider-429-backoff branch July 12, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant