fix: probe stale rate-limit cooldown primaries

[joshavant]

Summary

• Allow throttled half-open probes for stale generic rate_limit cooldowns on primary model candidates even when fallbacks are configured.
• Preserve provider-recorded reset windows (blockedUntil with provider block metadata) so known future quota windows still keep the fallback path preferred until near expiry.
• Update model fallback tests for the new probe behavior and the recorded-window guard.

Fixes #87608

Verification

• .agents/skills/autoreview/scripts/autoreview --mode local
  • Clean: no accepted/actionable findings.
• node scripts/run-vitest.mjs src/agents/model-fallback.probe.test.ts src/agents/model-fallback.test.ts -- --reporter=verbose
  • 4 files passed, 170 tests passed.
• AWS Crabbox live repro with Ollama Cloud:
  • provider: aws
  • lease: cbx_33d131797efc
  • run: run_4f42fc0b825a
  • result: passed

Real behavior proof

Behavior addressed: A primary model with a stale generic rate_limit cooldown could remain bypassed behind configured fallbacks even after the provider recovered.

Real environment tested: AWS Crabbox Linux runner using live Ollama Cloud credentials, Gateway, and a real ollama-cloud/gemma3:4b model request.

Exact steps or command run after this patch: seeded an agent auth profile with a 30-minute generic cooldownUntil/cooldownReason: rate_limit, configured primary ollama-cloud/gemma3:4b plus a synthetic missing fallback, started Gateway, and sent a raw Gateway agent model-run request through Ollama Cloud.

Evidence after fix: Crabbox run run_4f42fc0b825a on AWS lease cbx_33d131797efc logged decision=probe_cooldown_candidate for ollama-cloud/gemma3:4b, then decision=candidate_succeeded for the same primary model.

Observed result after fix: The primary Ollama model was probed and succeeded; the seeded cooldown state was cleared (hasCooldownUntil: false, errorCount: 0, failureCounts: null), and the script ended with live_regression_repro_passed=true.

What was not tested: Full repository test suite; this change was covered with focused model fallback tests plus the live Gateway/Ollama regression repro.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed May 28, 2026, 8:52 PM ET / 00:52 UTC.

Summary
The PR changes model fallback cooldown decisions so stale generic rate-limit cooldowns on primary candidates can be half-open probed despite configured fallbacks, while active provider reset windows keep fallback preference, and updates focused tests.

PR surface: Source +38, Tests +10. Total +48 across 3 files.

Reproducibility: yes. from source inspection and PR proof: current main skips a cooldowned primary until near expiry, while the PR body reports a live Gateway/Ollama run where a seeded generic rate_limit cooldown was probed and cleared. I did not execute the repro in this read-only review.

Review metrics: 1 noteworthy metric.

• Fallback cooldown decision: 1 changed decision path. The PR changes whether a cooldowned primary model is tried before configured fallbacks for generic rate_limit state.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Have a maintainer explicitly accept the generic rate-limit primary-probe-before-fallback tradeoff under the protected label.

Risk before merge

• [P2] Merging intentionally changes fallback preference for generic rate_limit cooldowns: existing fallback chains may spend one throttled primary probe before using fallbacks when the local cooldown is still active.

Maintainer options:

1. Accept the bounded probe tradeoff (recommended)
   Merge if maintainers agree that a throttled primary probe for generic rate_limit cooldowns is preferable to staying on fallbacks until the local cooldown nears expiry.
2. Ask for broader provider proof
   If maintainers are not comfortable generalizing from the focused tests and Ollama Cloud live repro, request an additional multi-profile or non-Ollama proof before merge.

Next step before merge

• No ClawSweeper repair is needed; maintainer review should decide the protected fallback-policy tradeoff and normal merge readiness.

Security
Cleared: The diff only touches model fallback runtime and tests; it does not change dependencies, workflows, credentials, scripts, or package resolution.

Review details

Best possible solution:

Land the focused fallback-policy fix after a maintainer accepts the bounded primary-probe tradeoff and required checks pass, keeping the provider reset-window guard intact.

Do we have a high-confidence way to reproduce the issue?

Yes from source inspection and PR proof: current main skips a cooldowned primary until near expiry, while the PR body reports a live Gateway/Ollama run where a seeded generic rate_limit cooldown was probed and cleared. I did not execute the repro in this read-only review.

Is this the best way to solve the issue?

Yes, the implementation is a narrow fallback-decision change with focused tests for stale generic rate_limit cooldowns and provider-recorded reset windows. The remaining question is whether maintainers accept the fallback-policy tradeoff before merge.

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 61cf005437fd.

Label changes

Label changes:

• add P1: The linked bug blocks real agent/message workflows after recovered provider 429s and the PR targets that urgent fallback/auth-provider path.
• add merge-risk: 🚨 auth-provider: The diff changes provider routing and model-choice behavior while auth-profile cooldown state is active.
• add proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes structured post-patch AWS Crabbox live Gateway/Ollama proof with exact setup, run and lease IDs, decision logs, and cleared cooldown state.
• add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): The PR body includes structured post-patch AWS Crabbox live Gateway/Ollama proof with exact setup, run and lease IDs, decision logs, and cleared cooldown state.

Label justifications:

• P1: The linked bug blocks real agent/message workflows after recovered provider 429s and the PR targets that urgent fallback/auth-provider path.
• merge-risk: 🚨 auth-provider: The diff changes provider routing and model-choice behavior while auth-profile cooldown state is active.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): The PR body includes structured post-patch AWS Crabbox live Gateway/Ollama proof with exact setup, run and lease IDs, decision logs, and cleared cooldown state.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes structured post-patch AWS Crabbox live Gateway/Ollama proof with exact setup, run and lease IDs, decision logs, and cleared cooldown state.

Evidence reviewed

PR surface:

Source +38, Tests +10. Total +48 across 3 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	44	6	+38
Tests	2	40	30	+10
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	3	84	36	+48

What I checked:

• Current main behavior: Current main only probes a primary candidate with configured fallbacks when the cooldown is expired or within PROBE_MARGIN_MS, so a 30-minute generic rate_limit cooldown remains skipped behind fallbacks. (src/agents/model-fallback.ts:946, 61cf005437fd)
• PR implementation: The PR adds hasActiveProviderRateLimitResetWindow and passes the inferred cooldown reason into shouldProbePrimaryDuringCooldown so generic rate_limit cooldowns can probe immediately unless a provider-recorded blockedUntil reset window is still active. (src/agents/model-fallback.ts:947, fccb2f576541)
• Focused test coverage: The probe tests now assert that a far-from-expiry generic rate_limit primary is probed and that a blockedUntil window with subscription_limit metadata still suspends instead of probing. (src/agents/model-fallback.probe.test.ts:343, fccb2f576541)
• Provider reset-window contract: The existing auth usage code records blockedUntil with blockedReason, blockedSource, and blockedModel for provider quota windows, which matches the PR's guard for authoritative provider reset windows. (src/agents/auth-profiles/usage.ts:795, 61cf005437fd)
• Real behavior proof in PR body: The PR body reports an AWS Crabbox live Gateway/Ollama Cloud repro, run_4f42fc0b825a on lease cbx_33d131797efc, with probe_cooldown_candidate followed by candidate_succeeded and cleared cooldown state. (fccb2f576541)
• Not already implemented on main: The PR head is not an ancestor of the current main checkout, so the branch still carries the proposed fix rather than being obsolete. (fccb2f576541)

Likely related people:

• Peter Steinberger: git log shows many recent model fallback/auth test and policy commits in the touched files, including shared probe assertions and fallback decision logging work. (role: heavy recent area contributor; confidence: high; commits: 6a87d6e81426, 51c6b1c2bc56, 6739c28718ec; files: src/agents/model-fallback.ts, src/agents/model-fallback.probe.test.ts, src/agents/model-fallback.test.ts)
• Ítalo Souza: Commit 39bb1b3 added the earlier auto-recovery path for primary models after rate-limit cooldown expiry and added the probe regression tests this PR extends. (role: introduced related recovery behavior; confidence: high; commits: 39bb1b33222e; files: src/agents/model-fallback.ts, src/agents/model-fallback.probe.test.ts, src/agents/auth-profiles/usage.ts)
• sebslight: Commit d224776 extracted the cooldown probe decision helper that this PR modifies, and the earlier recovery commit records sebslight as reviewer/co-author. (role: recent refactor/reviewer; confidence: high; commits: d224776ffbb1, 39bb1b33222e; files: src/agents/model-fallback.ts, src/agents/model-fallback.probe.test.ts)
• Vignesh Natarajan: Recent history includes explicit rate-limit cooldown probe behavior and unavailable-reason inference work in the same fallback path. (role: adjacent cooldown probe contributor; confidence: medium; commits: d45353f95b57, 5c7c37a02a3b; files: src/agents/model-fallback.ts, src/agents/model-fallback.probe.test.ts, src/agents/model-fallback.test.ts)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.