MaxLLM v0.3.5 — auto: replay when an upstream hangs up, sentinel never leaks upstream

What's in v0.3.5

Patch release on top of v0.3.2, folding in the three auto fixes made
while chasing a five-minute silent turn. No wire-format or configuration
changes.

auto sentinel works on any route (v0.3.3)

A client that selects auto in its model picker only changes the model
string — its base URL stays on the summary route (/v1/responses). The
sentinel was only recognised through route.auto_model, so the request went
upstream verbatim: every provider answered "unknown model" (404/400), the
client got a 502, and codex-style clients retried with backoff — 337 s of
silence in the reported incident.

The sentinel now means "rotate" regardless of the route: the failover walk
enters the [auto_model] pool instead of the route's own chain, the first hop
pins auto_target/auto_requested so later hops stay inside the pool, and
the rewritten body travels non-verbatim. Cost: one rejected attempt
(100–300 ms). Alternative: a hard 502 for a model the client never meant to
name.

Rotation actually rotates (v0.3.4)

On a route without auto_model, the pool was entered mid-request, so no
select_provider slot ever rotated for it: the walk in pool order pinned every
such request to the first working entry and left the rest idle. The sentinel
path now tries the session binding first and then round-robins over the healthy
entries.

A member that accepts the connection and hangs up (v0.3.5)

A model server that crashed — or whose supervisor still holds the listen socket
while the worker is gone — accepts the TCP connection and closes it before
sending a byte. Pingora reports ConnectionClosed while reading response
headers and by default only replays that on a reused connection, so a dead
pool member handed the client a hard 502 while the rest of the pool sat idle.

error_while_proxy now treats "nothing came back and the connection died" like
a connect failure: it counts against the member's circuit breaker, unpins the
session and replays the request on the next pool entry or fallback, bounded by
the route's failover budget. Errors that arrive after the upstream produced
headers are still never replayed.

The sentinel never reaches an upstream (v0.3.5)

A session pinned to a pool member that also sits in the route's fallback chain
was sent "model": "auto" verbatim: strict providers answered 400, and
llama.cpp-style servers accepted it and served whatever model they host. The
pinned entry's model now replaces the sentinel.

Diagnosis and operations

• Every failed attempt is recorded (provider=status, provider=connect,
  provider=dns, provider=closed) and appended to the 502, so a trailing DNS
  failure no longer hides the real answer of the other providers.
• scripts/pool_health.py probes each pool entry through its own
  /v1/responses/<provider> route and prints a pass/fail table; it exits
  non-zero when any member fails, so it can gate a deploy.

Verification on the deployed gateway

• pool probe: 25 members, 12 healthy, 13 failing (10 GPUs down, one 403, one
  quota 503) — all failures are endpoint availability, none is gateway logic;
• 20 fresh sessions on /v1/responses with model=auto: 20/20 200,
  6 distinct models served;
• 10 sessions on /v1/responses/auto: 10/10 200;
• 5 requests on one session: 5/5 200, all on the same member (sticky);
• zero upstream requests carrying "model": "auto".

Tests

Gateway suite: 129 unit + 100 integration + 4 ignored (integration includes
the new test_upstream_hanging_up_before_answering_fails_over,
test_auto_rotation_survives_a_pool_member_that_hangs_up and
test_sentinel_rewritten_when_the_session_is_pinned_to_a_pool_member).
cargo clippy --all-targets -D warnings clean. One pre-existing unit test
(rate_limit_resolver::tests::unclassifiable_error_falls_back_to_default_cooldown)
fails on a clean checkout as well and is unrelated.

Artifact

maxllm-server-linux-x86_64-musl — x86_64 static-pie, glibc-independent
(maxllm 0.3.5, sha256 227fe7c4802e87e50847ec1d8801d8a619b8c6fa40601dd0d78e5fa88ebf3292).