Make MCP connections survive transient transport failures

[devin-ai-integration]

Summary

A single non-2xx response from a remote MCP server currently takes the connection out for the rest of a run. The streamable-HTTP SDK runs each request inside the transport's task group and calls raise_for_status() there, so an error response tears the group down and the in-flight caller sees a bare CancelledError with no status attached. SupervisedMcpSession._execute then reconnects once, replays the call immediately, and treats a second failure as permanent — so a transient 429 or 5xx is indistinguishable from a revoked token, and both end the connection.

Three things change: failures get classified, retries get a policy, and death gets a cooldown.

Classification. The status is captured before the SDK swallows it, via an httpx response hook installed by _build_server:

class HttpStatusRecorder:
    async def __call__(self, response: httpx.Response) -> None:
        if not 200 <= response.status_code < 300:
            self._failure = _from_status(response.status_code, ..., _retry_after(...))

classify() maps httpx/SDK errors (including nested BaseExceptionGroups, by specificity auth > permission > rate_limit > server > protocol > timeout > transport) to a FailureInfo, and when a cancellation arrives with no exception the supervisor reads the recorder instead of guessing. Only status, reason, Retry-After, request method and URL path are retained — never headers, query strings or bodies.

Timeouts were both defaulted to 5s — the SDK defaults timeout and client_session_timeout_seconds to 5, and _build_server passed neither. Any tool slower than that was killed by our own configuration. They are now explicit and per-connection (http_timeout_seconds=30, session_timeout_seconds=60, sse_read_timeout_seconds=300).

Retry and quarantine replace the one-shot strike counter:

401 (auth)              -> retire immediately; the credential is not accepted
403 (permission)        -> see "a status is not a session verdict" below
429                     -> honor Retry-After, else backoff, up to 3 attempts
5xx/timeout/transport   -> reconnect + settle, backoff w/ jitter, up to 3
attempts exhausted      -> quarantine 30s -> 60s -> 120s, then permanent

Quarantine cleans up the failed server and clears it, so a lazy revive on the next dispatch builds a fresh session rather than reusing a known-broken one, and the supervising task now stays alive across it — including on an idle session death, which previously ended the task outright. is_dead keeps its meaning: permanent for the run.

Concurrency is bounded per connection name (max_concurrent_calls=4) so several agents sharing one provider session cannot fan out unboundedly into a rate limit. Semaphores are keyed per event loop through a WeakKeyDictionary, since one connection name can be served from more than one loop.

A status is not a session verdict

Deciding permanence from the HTTP status alone retires healthy connections. A provider may map a per-request authorization denial onto an HTTP 403 of the tool-call POST — the credential is fine and the session is fine, but this request's arguments named a resource the credential may not read. Observed in practice: the same tool succeeded and then returned 403 twelve seconds later on the same session, differing only in one argument. The connection was retired, and every later call on it failed with "unavailable" for the rest of the run.

So a failure now carries the phase that produced it, and only session-level failures are session verdicts:

                        phase="connect"        phase="call"
                        (connect, rebuild,     (a dispatched
                         list_tools)            tool call)
401 auth                retire                 retire
403 permission          retire                 return to the agent
4xx protocol            retry -> quarantine    return to the agent
other                   unchanged              unchanged

_Outcome grows a third possibility next to value and dead: call_failure, meaning the session is fine and this request was rejected. dispatch() turns it into the standard failed-tool output; it neither retires the connection nor spends a quarantine strike, and the next call rebuilds the session lazily as any post-failure call already did. list_tools() runs in the connect phase, so its behavior is unchanged.

The message the agent gets says whose fault it is, since the previous copy ("connection is unavailable... it will not be retried") sent the model chasing a connection problem instead of its own arguments:

MCP connection 'x' rejected this call (status=403): the provider denied this specific request, not the connection. The connection is still available. Check the arguments — resource and project identifiers, and required fields — and whether the configured credential is allowed to read that resource, then retry.

The classifier splits 403 out of auth into its own permission kind to express this; both remain non-retryable, so retryable is unchanged for every existing caller.

Tests

Cover classification and nested groups, Retry-After in both seconds and HTTP-date form, 429 retry, 5xx quarantine and revival, immediate retirement on 401, cancellation combined with a recorded status, timeout propagation, the concurrency cap, and redaction. One drives a real httpx.AsyncClient built through _build_server over a MockTransport — without it a sync response hook passes every mocked test and fails on every real request, because AsyncClient awaits its hooks.

For the phase split: a 403 and a 400 raised by a dispatched call each leave the connection alive, unquarantined and with no strike, and the next call on the rebuilt session succeeds; a 403 raised by list_tools still retires the connection; a 401 raised by a call still retires it immediately.

Link to Devin session: https://app.devin.ai/sessions/cf9f20a751754792b512e722b852b6f8
Open in Devin Desktop: https://app.devin.ai/desktop/session/cf9f20a751754792b512e722b852b6f8?variant=devin
Requested by: @yoni-at-strix

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

This PR makes MCP HTTP connections recover from transient transport failures while immediately retiring authentication failures.

• Adds sanitized HTTP failure classification, including status and Retry-After handling.
• Introduces bounded retries, quarantine cooldowns, lazy reconnection, and per-connection concurrency limits.
• Adds configurable HTTP, session, and SSE timeouts.
• Resets quarantine strikes after a successful call, resolving the previously reported accumulation across successful revivals.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
strix/tools/mcp/session.py	Implements classified retry, quarantine, reconnection, cleanup, and concurrency behavior; the successful-call path now resets prior quarantine strikes.
strix/tools/mcp/failures.py	Adds sanitized classification for HTTP, timeout, transport, protocol, and nested exception failures.
strix/tools/mcp/client.py	Installs the asynchronous HTTP response recorder and propagates configured transport timeouts.
strix/tools/mcp/config.py	Adds validated timeout and concurrency settings with explicit defaults.
tests/test_mcp_resilience.py	Covers retry, quarantine, revival, classification, redaction, timeout, and concurrency behavior, including regression coverage for resetting quarantine strikes.

_{Reviews (2): Last reviewed commit: "Recover MCP connections from transient d..." | Re-trigger Greptile}

[yoni-at-strix]

@greptile