Skip to content

v0.1.78

Latest

Choose a tag to compare

@rromenskyi rromenskyi released this 07 Oct 05:30
· 39 commits to main since this release
292b735

Per-(provider, model) unavailability marking, Retry-After aware

Built live against a real production incident: coding-premium's muse-spark-1.3 tier was overloaded upstream (503 service_overloaded/500/504, intermittently hanging up to 90s). A real curl against the live upstream confirmed it sends a standard Retry-After: 60 header on its 503 — this gateway read the error body but never the response headers at all, so that signal was silently discarded.

Operator's own framing, which drove the design: the request is the unit, not the provider. If a target fails, mark it unavailable; the next real request is itself the retry (honoring the vendor's own Retry-After, or a configurable default backoff otherwise) — if it fails too, it falls through to the next tier exactly like any other failed attempt, invisibly to whoever's waiting on it. No separate background prober.

New internal/unavail package

A simpler, always-on sibling to the existing per-(alias,tier) circuit breaker:

  • Keyed by (provider, upstream_model) directly — a target's health is a fact about the vendor/model, not about any one alias's fallback chain, so a failure discovered via one alias immediately protects every other alias routing to the same target.
  • No error-rate window, no consecutive-failure counter, no half-open probe-admission race.
  • Honors the upstream's own Retry-After exactly when present (seconds or HTTP-date, RFC 9110 §10.2.3); absent that, a configurable default backoff doubles from the target's last mark, capped (built-in default 200ms → ~25.6s over 8 doublings).
  • Redis-shared across replicas, fails open if Redis is unreachable.

Also

  • providers.Error.RetryAfter, threaded through every non-2xx upstream call site (chat, audio, model-catalog listing).
  • Vendor X-RateLimit-* headers logged when present (observability only, no behavior tied to them yet).
  • Wired into executePlan alongside the existing breaker, reusing countsAgainstTier's classification so both mechanisms agree on what's attributable to a target.

Test plan

  • internal/unavail: 7 unit tests, -race clean. Two genuine bugs caught and fixed by these tests before reaching executePlan: Retry-After wrongly clamped by the default max, and a unit-scaling bug that silently reset the backoff every other mark.
  • internal/providers: Retry-After parsing (including the exact 60 confirmed live), httpError population, rate-limit-header logging.
  • internal/httpapi: 3 executePlan wiring tests (skip-then-retry, Retry-After-over-default, cross-alias sharing) — surgical-revert proven.
  • Full suite green with -race.

Shipped via #161, chart bump #162.

Full Changelog: v0.1.77...v0.1.78