fix(quoter-bot): retry transient provider reads with backoff

[haydenshively]

Incident

The production quoter bot on Railway safety-halted 7 times in ~24 hours (2026-08-14 14:21 → 2026-08-15 13:42 UTC), and the final halt left the deployment CRASHED with all offers cancelled. Every halt had the same shape: a single transient SafeProviderError on a read-only HTTP GET (requestJson against the morpho-api/router-api endpoints) surfaced in a ladder/bootstrap cycle read, and the workflow's safety halt cancelled the resting ladder on-chain and exited the process. The bot ran 210+ clean 60s cycles between failures — intermittent API flakiness (roughly one blip every ~4 hours), each costing on-chain cancel gas and full downtime until restart.

Fix

Add retry-with-backoff for transient provider failures at the HTTP transport seam, so one blip no longer triggers the safety halt while genuine/persistent failures still do.

• New retryTransientProviderRead utility (bots/quoter-bot/src/infrastructure/setup-state/provider-retry.utils.ts); requestJson now wraps a single-shot attemptJson in it. All these requests are idempotent read-only GETs, so repetition is safe.
• Retried: SafeProviderError with name TimeoutError/NetworkError, or HTTP status 408/429/5xx.
• Not retried: any other 4xx, and any non-SafeProviderError throw — rethrown immediately.
• Policy: 3 attempts max, half-jittered exponential backoff (capped/2 + random·capped/2 where capped = min(4000ms, 500ms·2^(n-1))) → worst-case ~1.5s added latency, well inside the 60s cycle interval.
• On exhaustion the original sanitized SafeProviderError is rethrown unchanged — no new error type, no URL or response body ever touched, so downstream halt behavior and operator-visible reports are identical.
• Reuses delay from @repo/utils. Evidence for the local utility: retryUntilDefined in @repo/utils has a sync predicate and no backoff/classification; @repo/bot-kit's createBackoff is block-height keyed for the tx queue — neither matches.

Tests

• provider-retry.utils.test.ts (new, 12 tests, fake timers): retries each transient class then succeeds; no retry on 400/403/404 or non-provider errors; exhausts 3 attempts and rethrows the final sanitized error with metadata intact; backoff bounds verified at both jitter extremes.
• viem-setup-state.service.test.ts: 2 loopback-server integration tests (503×2 then success; 400 not retried).
• Mutation check: breaking the retry guard failed 9 tests across both files.

Validation

typecheck ✅ · pnpm lint ✅ (0 warnings) · pnpm format ✅ · pnpm test ✅ (1865 passed / 12 skipped; only pre-existing env-gated fork/e2e failures needing RPC_URL_8453 + anvil) · jsdoc:build ✅ · knip ✅ (via pre-commit)

Follow-ups (deliberately out of scope)

• RPC read path (provider-read.utils.ts / ProviderReadError): left alone — viem's http() transport already retries (default retryCount: 3, exponential backoff) beneath these reads; wrapping again would multiply attempts.
• Richer cycle-log metadata: cycle logs surface only errorName, dropping the sanitized failure metadata (provider/status/code) that would have made this incident diagnosable from logs alone. Threading it through LadderVerboseState ripples across several typed call sites — worth its own PR.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-04T03:36:58.328995Z	1699bf7	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1699bf7f00

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".