fix: mcp robustness fixes under high-concurrency

[mikasenghaas]

Companion re-pin: PrimeIntellect-ai/prime-rl#3084

Summary

Make the v1 MCP client paths resilient to connection churn under high rollout concurrency, so per-rollout transport blips no longer fail rollouts — without replay-corrupting the user simulator.

Under high concurrency (e.g. 512 rollouts on the subprocess runtime), the user-sim and tool MCP connections drop at rollout-start churn (httpx.ReadError → anyio.ClosedResourceError). Previously:

• the host-side user path (connect_user) held one connection open for the whole rollout and retried only the initial connect — a mid-session respond() drop propagated. The opening respond("") (interception handle_request) was unwrapped, so a drop there became a 500 to the harness, which exited non-zero → HarnessError → failed rollout. Worse, the harness SDK's retry of that 500 made the interception re-invoke respond("") against a simulator that had already advanced for the lost turn — silently corrupting the trajectory (observed as empty/garbage turns upstream).
• the harness tool path (null/bash program.py) held persistent MCP sessions with no retry, so a mid-loop call_tool transport drop crashed the harness → failed rollout.

Design

Every call now runs on a fresh session (the servers are stateless-HTTP: setup()/setup_task() run once at startup, per-rollout state lives on the shared-state channel), and retries are made safe rather than avoided:

• User turns are exactly-once via a server-side replay cache. The wire respond tool gains a seq conversation position (derived by the interception from the prompt position — byte-identical across SDK retries). The per-rollout user server keeps its last (seq, message) → payload and replays a retried turn instead of advancing the simulator twice; a lock serializes duplicate in-flight attempts (a retry racing a slow first execution) so they join the recorded turn. This dedups the host client's retries and the interception-level re-invocations. The host retries the full call through the existing shared retrying() policy (verifiers.v1.retries).
• A retried opening request injects the cached opening. The opening-injection guard keyed on trace.num_turns == 0, which mutates across attempts: an SDK retry of the opening request after the simulated conversation had committed turns (final response lost) skipped injection and forwarded its empty message list upstream (the renderer's deterministic No messages provided 502). The opening request is now recognized by its own shape — no assistant turn in the request — which is stable across retries.
• Task tool calls are at-least-once, documented. MCP has no idempotency key and tools are arbitrary code, so with_retry in the harness programs replays a call whose response was lost; a tool that fails for real reports through its result, not an exception, so a retried exception is a transport fault. (At-most-once was tried and measurably regressed: 0.4–4.2% of rollouts per step failed at 512 concurrency because drops hit the round-trip itself, not just setup.)
• Teardown failures after a completed body are suppressed — the result is already in hand; closing noise must not fail (or replay) an already-answered call.
• Response parsing/conversion happens outside the retry — a client-side parse failure fails once, never re-invokes the call.

Each session is opened and closed within the caller's task, keeping AnyIO cancellation scopes correctly nested; cancellation is never retried or swallowed. The program.py scripts (deps openai/mcp/httpx, can't import verifiers) carry a small inline with_retry; the null program now declares its direct httpx dependency.

Verification

Semantics (real code, stubbed transports + a real stateless FastMCP server):

• setup churn → retried to success, call runs once
• mid-call drop → replayed with the same seq; the user server's replay cache returns the recorded turn (simulator advances once)
• replay cache: retried (seq, message) → cached payload, simulator invoked once; new seq → advances; seq=-1 → no dedup
• teardown failure after a completed call → swallowed, result usable
• exhaustion raises; cancellation propagates immediately, unretried
• unparseable respond payload → UserError raised once, respond invoked exactly once

End-to-end, 512 concurrency, subprocess runtime, one env per MCP shape:

Env	Shape	Rollout error before	after
alphabet-sort-v1	user simulator	15.1%	0.0% (5/5 steps)
gsm8k-v1	vanilla (no MCP)	0.0%	0.0%
wiki-search-v1	tool (shared, fan-in)	—	0.0%

alphabet-sort-v1 is the reported regression: before the fix, the opening user-sim respond 500'd under churn (ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) → HarnessError('harness 'null' exited 1: ... openai.InternalServerError: 500')). In the final 5-step run: 0 rollout failures, 0 empty-message upstream calls, and 20 respond retries — every one absorbed by the replay cache (the churn still happens; it's now handled). For calibration, the at-most-once alternative (retry setup only, never the call) was measured on the same setup and failed 0.4–4.2% of rollouts per step.

---

Note

Medium Risk
Changes rollout-critical MCP and user-simulator semantics (retries, at-least-once tools, seq-based dedup); behavior is well-tested at 512 concurrency but any custom MCP tool that is not replay-safe could see duplicate side effects.

Overview
Hardens v1 MCP paths so transport churn at high rollout concurrency no longer fails harnesses or double-advances user simulators.

Harness programs (null/bash program.py) drop long-lived MCP sessions: connect_mcp only enumerates tools and returns server specs; each list_tools / call_tool opens a fresh streamable-HTTP session with httpx timeouts, tenacity backoff (6 attempts), and teardown errors suppressed after a successful body. Tool calls are documented as at-least-once on replay.

Host user path replaces connect_user with user_respond + user_session (per-call session, shared retrying()). serve_user yields partial(user_respond, url). The MCP respond tool gains a seq argument; the user server caches (seq, message) → payload under an asyncio.Lock so retries replay instead of advancing twice.

Interception passes len(prompt) as seq for opening and follow-up user turns. Opening injection no longer keys on trace.num_turns == 0 (unstable on SDK retries); it triggers when the request has no assistant messages. Prompt assembly after a model turn appends the assistant message before calling the user sim, then only the user messages (avoiding duplicate assistant in prompt).

^{Reviewed by Cursor Bugbot for commit 7d9adb2. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Fix MCP robustness under high concurrency with per-call sessions, retries, and timeouts

• Replaces long-lived MCP sessions with per-call sessions in both the bash and null harnesses, so a stale or broken connection no longer blocks the entire run.
• Adds with_retry (up to 6 attempts, exponential backoff) and explicit httpx.Timeout (60s connect / 300s read) to all MCP tool enumeration and invocation paths.
• Adds a sequence number (seq) to the user simulator's respond MCP tool, allowing the server to deduplicate retried requests via an asyncio.Lock and a (seq, message) replay cache.
• Updates serve_user in launch.py to yield a partial(user_respond, url) instead of maintaining a persistent connection, aligning with the new per-call model.
• Behavioral Change: connect_mcp no longer accepts an AsyncExitStack and now returns (tool_schemas, dispatch, servers); callers must pass servers and dispatch to call_mcp.

Changes since #2062 opened

• Modified verifiers.v1.mcp.user.User._register method to ensure atomic state synchronization and response generation under a single lock [7d9adb2]
• Replaced manual retry logic with tenacity library in with_retry utility across verifiers.v1.harnesses.bash.program and verifiers.v1.harnesses.null.program modules [7d9adb2]
• Updated MCP_TIMEOUT configuration to use httpx.Timeout with explicit connect timeout in verifiers.v1.harnesses.bash.program and verifiers.v1.harnesses.null.program modules [7d9adb2]

^{Macroscope summarized 9674b53.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR makes significant changes to MCP session management and retry logic. An unresolved P1 review comment identifies a race condition bug in the replay cache logic where the cache can be visible before state commits, contradicting the PR's concurrency robustness goals.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit eaa53e4. Configure here.}