Add automatic retry with exponential backoff to all API providers

[micz]

Add automatic retry with exponential backoff to all API providers

Summary

Transient API failures — "model is overloaded" (503), rate limits (429), gateway errors, short network drops — currently surface to the user as a hard error, even though a second attempt a couple of seconds later would usually succeed.

This matters most for the automatic features: the spam filter and add_tags run unattended on incoming mail, so there is nobody around to click "retry". The operation simply fails on a problem that would have resolved by itself.

Scope

Add a shared fetchWithRetry(url, options, retryConfig) helper and use it for fetchResponse() and fetchModels() in all five API modules:

• js/api/anthropic.js
• js/api/google_gemini.js
• js/api/ollama.js
• js/api/openai_comp.js
• js/api/openai_responses.js

Target branch: v5.0.0.

Placement/naming: js/api/api-utils.js already exists for worker-safe shared helpers — either extend it or follow the same naming convention (e.g. js/api/mzta-api-retry.js).

Behaviour

• Retryable HTTP statuses: 408, 429, 500, 502, 503, 504, 529.
• Non-retryable: 400, 401, 403, 404 → fail fast, no retry.
• Network / fetch exceptions: retried.
• Backoff: exponential with jitter. Defaults: 3 retries, initial delay 1s, max delay 10s, factor 2.
• Retry-After header: honoured for both the delta-seconds and the HTTP-date form, capped at 30s.
• Per-attempt timeout via AbortSignal.timeout(). Without it a hung connection never produces a 408/504 and the retry logic never fires. Must not interfere with a user-initiated abort.
• Retry applies only before the response body is consumed. A mid-stream SSE failure is out of scope.
• The helper must always either return a Response or throw — never fall through and return undefined, which would produce a confusing TypeError in the caller.

Logging

• Never log the request URL. For Google Gemini (and some OpenAI-compatible endpoints) the API key travels in the query string, so logging the URL would leak it to the console.
• Use taLogger, consistently with the rest of the codebase, so output respects the debug preference.

UI

Retries must be visible. As designed, the user would see a frozen spinner for up to ~7s (or longer when Retry-After is honoured) with no indication of what is happening and no way to cancel. Suggestion: a newRetryAttempt message from the worker to the webchat controller.

Known trade-off

fetchResponse() issues non-idempotent POSTs. Retrying a 500/504 may re-run a generation that actually succeeded server-side, costing tokens twice. Acceptable, but worth documenting.

Out of scope: automatic model fallback

Rotating to a different model when retries are exhausted is deliberately not part of this issue. It would silently replace the model the user explicitly selected, with different quality, different pricing and no indication in the UI of which model actually answered. Any hardcoded fallback list would also go stale quickly and conflicts with the existing fetchModels() design.

Retry alone covers the common failure modes, which are time-dependent rather than model-dependent. Daily quota exhaustion is the only case a fallback would address, and there the right answer is to tell the user, not to degrade silently.

If model fallback is wanted later, it should be a separate opt-in preference, with a user-chosen list built from fetchModels(), limited to 429/503, and with the effective model reported in the UI.

Documentation

Add an "Automatic Retry Handling" section to claude-spec/04-api-integrations.md.

---

Thanks to @racerm3 for this idea in racerm3/ThunderAI@d7e13be.