Persistent remote MCP connection → subprocess drain-deadlock SIGKILLs claude and discards completed output (base image 0.8.0)

[nyasour]

Summary

On base image 0.8.0, an agent that has a long-lived remote MCP connection configured in .mcp.json completes its work but then hits a subprocess drain-deadlock: the claude process's stdout reader thread never EOFs, the 90s drain budget is exceeded, Trinity SIGKILLs the process group, and the already-produced output is discarded. Every such execution therefore returns empty/null cost, and the agent's circuit breaker trips. The logs reference the (closed) #728, but the trigger here is different (a persistent MCP connection, not OAuth-failure CPU spin), so raising it as a separate discussion.

Crucially: the MCP tool calls themselves succeed — this is purely an output-capture/subprocess-lifecycle problem, not an MCP connectivity problem.

Environment

• Trinity base image: 0.8.0, runtime claude-code (client UA claude-code/2.1.207 (sdk-cli)), a business-assistant agent.
• .mcp.json configures one remote streamable-HTTP MCP server (FastMCP, running in stateless mode) reached over HTTPS with a bearer header.

Symptom

Every execution (even a no-tool "reply READY" task — claude --print initializes all MCP servers before the first token) stalls, then:

WARNING agent_server.utils.subprocess_pgroup: [Subprocess] Reader thread(s) still busy after process exit (pid=NNN, stuck_count=1) — killing process group, then waiting 30s for natural drain
ERROR   agent_server.utils.subprocess_pgroup: [Subprocess] Reader thread(s) still stuck after ~60s post-kill grace — force-closing pipes; some buffered data may be lost (pid=NNN, stuck_count=1)
WARNING agent_server.services.subprocess_lifecycle: [Subprocess] Drain budget (90s) exceeded — safe_close_pipes may have deadlocked with reader thread's TextIOWrapper lock; reader threads are leaked daemon threads (pid=NNN). Issue #728.
WARNING agent_server.services.subprocess_lifecycle: [Subprocess] #1502: SIGKILLed the process group (pgid=NNN, pid=NNN) after the budget-exceeded drain so the leaked reader can EOF and free the core.
INFO    agent_server.utils.subprocess_pgroup: [METRIC] drain_outcome ... outcome=leaked ... leaked_count=1

The execution record then shows status=running→killed, cost=null, response=null; the circuit breaker opens ("Agent circuit breaker open — agent is unhealthy") and rejects subsequent tasks until an idle health-check resets it.

The work actually completed (key evidence)

The remote MCP server's own request logs show the agent's claude-code (sdk-cli) client performing the full sequence and getting HTTP 200 on every call, ~9s after the run started:

13:19:54  200  claude-code/2.1.207 (sdk-cli)   # initialize
13:19:54  202  claude-code/2.1.207 (sdk-cli)   # notifications/initialized
13:19:54  200  claude-code/2.1.207 (sdk-cli)   # tools/list
13:19:59  200  claude-code/2.1.207 (sdk-cli)   # tool call
13:20:00  200  claude-code/2.1.207 (sdk-cli)   # tool call

So claude connected, listed tools, called them successfully, and (we believe) wrote its final answer — but the SIGKILL/drain-deadlock discarded stdout before Trinity read it.

Reproduces with / does NOT reproduce

• Reproduces with native {"type":"http", ...} config and with an mcp-remote stdio bridge (npx mcp-remote <url> --header ...) to the same server — i.e. independent of transport.
• Does NOT reproduce with an empty .mcp.json ({"mcpServers":{}}) — a trivial task returns cleanly in ~3.6s — or with only Trinity's own auto-injected orchestration MCP server.

That contrast points at a persistent/keep-alive MCP connection (an idle HTTP keep-alive socket or a persistent stdio child) holding the stdout pipe fd / keeping a reader thread alive, so claude never closes stdout on exit and safe_close_pipes deadlocks on the TextIOWrapper lock.

Questions

1. Is this a known issue on 0.8.0, and is it fixed (or mitigated) in a newer base image? It looks adjacent to the closed bug: agent-server.py spins at 90% CPU on OAuth token auth failure, blocking CB recovery #728 but with a different trigger (persistent MCP connection rather than OAuth-failure CPU spin).
2. Is there a recommended pattern/workaround for agents that need a remote MCP server (streamable-HTTP), given this drain path? e.g. a drain strategy that closes the child's inherited stdout fd before waiting, a bounded reader-thread join with an explicit fd close, or a documented config to avoid the persistent connection.
3. Would you prefer this as a formal issue rather than a discussion? Happy to file with a minimal repro (a stateless FastMCP server + a one-server .mcp.json + any trivial task reproduces it deterministically).

Thanks — otherwise 0.8.0 has been solid; this is the one thing blocking an agent that depends on a remote MCP backend.

[nyasour]

Opened a fix PR for this: #1718 — runs the cgroup orphan sweep before the pipe close (so the blocked reader EOFs and releases the TextIOWrapper lock) and bounds every safe_close_pipes path (incl. the 300s asyncio.run teardown-join hazard in terminate(), and the unbounded awaits in claude_code.py/codex_runtime.py). Includes a fork-based regression test that fails pre-fix / passes post-fix.

[vybe]

Thanks — this is a model bug report, and the PR on top of it is even better. Answers in order:

1. Known issue / fixed in a newer image? The class is known — #728 (OAuth-failure CPU spin) and its residual #1502 (budget-exceeded drain leaking a reader thread) are both closed — but your trigger is genuinely distinct: a persistent MCP connection (idle keep-alive socket or persistent stdio child) keeping the reader thread from EOFing, so safe_close_pipes deadlocks on the TextIOWrapper lock exactly as you diagnosed. No released base image fixes this yet — 0.8.0 is current — so your #1718 is the fix, not a duplicate of one already in flight.

2. Recommended pattern / workaround? There's no config-side workaround I'd recommend — the fix belongs in the drain path, and your PR has the right shape (sweep cgroup orphans before the pipe close so the blocked reader EOFs, and bound every close path). If you need to unblock your agent before it ships in a release: the drain code lives entirely in the agent base image, so you can build the base image from your branch (./scripts/deploy/build-base-image.sh) and recreate the agent — no backend changes needed.

3. Separate issue? No need — #1718 is the tracked artifact now, and its fork-based regression test covers the repro. Keep the minimal FastMCP repro handy though; it's useful if review turns anything up.

The one thing blocking merge: as I noted on the PR, the branch was cut from main, which has diverged from dev, so the retarget surfaces a history-only conflict — the change itself cherry-picks onto dev cleanly and the unit suites pass on the rebased tree. I can't push to your fork (org-owned, so GitHub's maintainer-push returns 403), so it needs one action on your side:

git fetch upstream dev && git rebase upstream/dev && git push --force-with-lease

Once that's pushed, the nightly suite re-runs, this can land, and the fix rides the next base-image release.