managed failover: advance on response-body read errors

[mostlydev]

Problem

Declared model fallback works for transport failures and upstream 5xx responses, but the managed-tool JSON path does not advance when the upstream returns response headers and then the response body read fails.

internal/proxy/toolmediation.go, inside dispatchJSONWithRetry, currently handles the branch this way:

limited, readErr := readBodyLimited(resp.Body, maxManagedLLMResponseBytes)
resp.Body.Close()
if readErr != nil {
    cancel()
    return dispatchJSONAttemptResult{
        ClientStatus:  http.StatusBadGateway,
        ClientMessage: "failed to read upstream response",
        Err:           readErr,
    }
}

That terminal return ignores canFallback. It differs from the transport-error and 5xx branches immediately around it, which advance to the next declared candidate.

Production evidence

A managed Hermes turn had a primary and a declared fallback candidate. The primary returned headers, then its body read stalled until the configured 150-second candidate deadline. cllama returned:

• status: 502
• error: context deadline exceeded
• client message: failed to read upstream response
• fallback audit events: 0

The runner retried the primary instead of cllama advancing to the declared fallback. Managed tools themselves completed in 5-1364 ms; this was an upstream response-body failure, not a tool timeout.

This behavior is general to managed OpenAI-compatible requests; no deployment-specific policy is required to reproduce it.

Decision

When readBodyLimited returns an error and another declared candidate exists:

1. close/cancel the current response as today;
2. emit the existing candidate-fallback intervention with a stable reason such as response_read_error;
3. return AdvanceToNextCandidate: true and that fallback reason;
4. let dispatchCandidatesJSON emit the existing structured failover event and try the next candidate.

When there is no fallback, preserve the current terminal 502 behavior and message.

Do not retry or replay any managed tools. The failure occurs while reading the model response for the current round, before a usable assistant/tool-call result exists.

Tests

Add focused coverage for both sides of the boundary:

• primary returns HTTP 200 headers and a body whose Read fails; with a declared fallback, the fallback is called and its 200 response is returned;
• the failover/intervention reason is recorded;
• without a fallback, the current 502 failed to read upstream response result remains unchanged.

Non-goals

• No new fallback configuration surface.
• No retry policy or circuit breaker.
• No changes to non-managed streaming behavior.