fix(circuit-breaker): drop-grace + pipe-drop reclassification — #474 follow-up to #798

[AndriiPasternak31]

Summary

Follow-up to #798 (now in dev) which narrowed the circuit-breaker
classifier to ConnectError / ConnectTimeout only. That fix is
correct but doesn't address two adjacent failure modes that became
visible once the classifier got tightened:

1. Sibling-collapse on concurrent transport drops — when one
   caller catches a pipe drop and evicts the pooled httpx client,
   sibling callers race to build a fresh client against a half-closed
   peer and see ConnectError / TimeoutException. Under fix(circuit-breaker): only count TCP unreachability toward circuit (#474) #798's
   classifier those do increment the circuit, so 9-of-10 concurrent
   drops on a healthy agent still trip the breaker.

2. Pipe drop on subprocess writes — when the Claude/Gemini child
   process exits early, headless_executor / gemini_runtime returned
   500 with an ERROR-level [Errno 32] log. That's adjacent to the
   503 status that SUB-003 in task_execution_service interprets as
   auth-class failure and triggers subscription auto-switch.

Composition with #798

This branch is rebased onto current dev (post-#798). The handlers
compose rather than replace:

• CIRCUIT_FAILURE_EXCEPTIONS / TRANSIENT_TRANSPORT_EXCEPTIONS /
  is_circuit_failure() from fix(circuit-breaker): only count TCP unreachability toward circuit (#474) #798 are preserved as the canonical
  classification source.
• The drop-grace logic is layered on top: in _request(), an explicit
  (httpx.ReadError, WriteError, RemoteProtocolError, BrokenPipeError, ConnectionResetError) handler is placed BEFORE the tuple-based
  TRANSIENT_TRANSPORT_EXCEPTIONS handler so the stamp+evict side
  effects always run on the genuine drop signals (and raw OSError
  subclasses, which fix(circuit-breaker): only count TCP unreachability toward circuit (#474) #798 deliberately let propagate, are caught here
  for the drop-coordination).
• ConnectError / ConnectTimeout keep fix(circuit-breaker): only count TCP unreachability toward circuit (#474) #798's record_failure()
  semantics by default, with an in-grace-window override that raises
  AgentConnectionDroppedError instead — that's the actual
  sibling-collapse fix.
• monitoring_service.check_network_health() follows the same layered
  pattern: my /health-specific liveness handler
  (ReadError/WriteError/RemoteProtocolError → record_failure())
  sits BEFORE the shared TRANSIENT_TRANSPORT_EXCEPTIONS handler, so
  Python's first-match enforces the divergent semantics. This is the
  documented /health-specific divergence from _request().

Changes

• agent_client.py — _recent_drops per-base_url grace map
  (2 s window) + AgentConnectionDroppedError (subclass of
  AgentNotReachableError so existing tenacity / catch chains keep
  working) + _acquire_client(base_url) -> (client, is_pooled) so the
  pool isn't repopulated with transient sockets during a burst.
  Explicit drop handler stamps the grace window, evicts the pooled
  client (identity-checked so siblings don't double-close), and skips
  record_failure(). finally closes non-pooled (fresh-during-grace)
  clients on every exit path to prevent connection-handle leaks.
• monitoring_service.check_network_health() — explicit
  (BrokenPipeError, ConnectionResetError) (circuit-neutral) and
  (httpx.ReadError, WriteError, RemoteProtocolError) (record_failure
  because partial response on /health = agent crash) handlers
  layered above the shared TRANSIENT_TRANSPORT_EXCEPTIONS handler.
• headless_executor.py + gemini_runtime.py — catch
  BrokenPipeError / ConnectionResetError from subprocess stdin
  writes, return 502 ("Bad Gateway to Claude subprocess") instead
  of 500. 502 is semantically correct and collision-free with the
  SUB-003 auto-switch path. Log demoted to INFO.
• docker-compose.sibling.yml — minimal Redis-only sibling stack
  on port 6390 (trinity-sibling-* containers) for running the
  Redis-backed integration tests without colliding with the production
  stack on 6379.

Tests

• tests/unit/test_circuit_breaker.py — adds drop-classification
  tests on top of fix(circuit-breaker): only count TCP unreachability toward circuit (#474) #798's classifier tests. Burst of 10 concurrent
  transport drops produces 0 record_failure calls (was 9 under
  fix(circuit-breaker): only count TCP unreachability toward circuit (#474) #798's classifier alone); sibling ConnectError inside the grace
  window is collateral, same ConnectError outside is a real failure;
  pool eviction is idempotent across siblings; non-pooled clients
  are aclose'd on every exit path. 29 tests total.
• tests/integration/test_circuit_breaker.py — adds
  TestConcurrentTransportDrops next to fix(circuit-breaker): only count TCP unreachability toward circuit (#474) #798's
  TestFailureClassification. Real AgentClient + real Redis via
  the sibling compose, drives the drop-burst path. test_raw_broken_pipe
  and test_raw_connection_reset updated to expect
  AgentConnectionDroppedError since the drop handler now catches
  these explicitly (was: expected raw exception to propagate).
• tests/unit/test_monitoring_health_check_classification.py —
  pins the client-vs-agent disconnect split on /health.
• tests/unit/test_headless_executor_pipe_drop.py — pipe drop
  returns 502 (not 500, not 503), log level INFO.
• tests/unit/test_pipe_close_no_auto_switch.py — SUB-003
  auto-switch is not triggered on 502.

Security

docs/security-reports/cso-2026-05-13-474-diff.md — CSO --diff audit
of the working tree pre-PR. 0 critical / 0 high / 0 medium across
secrets, dependencies, auth, injection, and platform patterns.

Test plan

• uv run pytest tests/unit/test_circuit_breaker.py tests/unit/test_monitoring_health_check_classification.py tests/unit/test_headless_executor_pipe_drop.py tests/unit/test_pipe_close_no_auto_switch.py
• Sibling stack up, then
  uv run pytest tests/integration/test_circuit_breaker.py
• Manual: kick a multi-poller MCP-sync drop and confirm the
  per-agent circuit stays closed
• Manual: trigger a Claude subprocess early-exit, confirm 502 and
  no SUB-003 auto-switch

Refs #474, follows #798

🤖 Generated with Claude Code

[vybe]

Solid follow-up to #798. The drop-grace mechanism is correctly designed — per-base_url map bounded by fleet size, identity-checked pool eviction to prevent sibling double-close, aclose() in finally on every exit path. The pipe-drop 502 reclassification closes the SUB-003 false auto-switch path cleanly (502 is collision-free with the 503-or-auth-string predicate). Monitoring /health divergence from _request() semantics is correctly documented and pinned in tests.

Two non-blocking notes:

1. architecture.md entry for agent_client.py doesn't mention AgentConnectionDroppedError, _recent_drops, or _acquire_client — worth a one-liner update in a follow-up or the next architecture pass.
2. Worth commenting on #474 to note what this PR resolved vs. what (if anything) remains, so the issue status is clear.

Two CSO audits included in the PR — appreciated.