Skip to content

fix(mcp): bounded per-peer mount so one hung peer can't darken the aggregator - #347

Merged
ywatanabe1989 merged 3 commits into
developfrom
fix/mcp-aggregator-nonblocking-mount
Jul 6, 2026
Merged

fix(mcp): bounded per-peer mount so one hung peer can't darken the aggregator#347
ywatanabe1989 merged 3 commits into
developfrom
fix/mcp-aggregator-nonblocking-mount

Conversation

@ywatanabe1989

Copy link
Copy Markdown
Collaborator

Problem

The umbrella scitex serve aggregator (src/scitex/_mcp/__init__.py) is the
single MCP surface that will front ALL ~33 packages' tools. At module-load time
it calls register_all_tools(mcp), which iterated the ecosystem registry and
resolved each peer's FastMCP sequentially + synchronously via
importlib.import_module(f"{import_name}._mcp_server").

If ONE peer's _mcp_server import HANGS at init (verified real case:
scitex-todo's store-wedge stalls 20s+ at mcp-start), the whole loop blocks →
scitex._mcp import never completes → every peer's tools go dark. The
aggregator concentrates this cascade 33x versus the current per-server split,
so it must be fixed before fleet-wide deployment.

Fix

Each peer's resolve now runs concurrently in a bounded daemon thread
(_resolve_peers_bounded):

  • All threads start at once and are joined against a single shared deadline, so
    total load time is ~max(peer) bounded by the per-peer budget — never the
    sum. A single hung future can't stall collection of the others.
  • Daemon threads: a truly-wedged import thread can't block interpreter exit.
  • On timeout / exception / hang the peer is SKIPPED with a warning
    (MCP peer '<ns>' resolve timed out ... — its tools will be unavailable this session) and the loop CONTINUES. Healthy peers still mount.
  • Budget configurable via SCITEX_MCP_PEER_TIMEOUT (default 8s).
  • safe_mount stays on the main thread — only the peer IMPORT is
    slow/hang-prone; the mount is fast and mutates shared parent-server state, so
    it is kept single-threaded to avoid concurrent-mutation races.

Existing BaseException handling in _resolve_peer_mcp is preserved, so a peer
that sys.exit()s or raises ImportError at import is skipped, not fatal.

All prior behavior preserved: _is_enabled env gate, _NAMESPACE_ALIASES,
_SKIP_CATEGORIES, register_peer_extras, register_umbrella_tools, the
mounted-count log, and SCITEX_MCP_USE_<NS>=0 gating. register_all_tools
gains injectable iter_registry / peer_timeout kwargs (defaults unchanged) so
tests can drive the real code path with hand-rolled fixture peers instead of
patching module globals (also satisfies the no-monkeypatch house rule).

Tests — tests/scitex/test_mcp_bounded_mount.py (22 tests, all green)

Real fixture peers written to disk and imported for real (no mocks):

  • hung peer (time.sleep(30) at import): _resolve_peers_bounded returns
    within the budget, reports the peer skipped with a "timed out" reason, and the
    co-resident fast peer still resolves to a real FastMCP.
  • full register_all_tools path with an injected registry of one fast +
    one hung peer: does not hang (< 20s despite the 30s sleep), hung namespace NOT
    mounted, fast namespace mounted, a warning naming the hung peer is logged
    ("unavailable").
  • ImportError peer and sys.exit() peer: skipped, don't block the fast
    peer.
  • env gate (SCITEX_MCP_USE_<NS>=0) and _peer_timeout parsing (default /
    override / invalid-fallback).

The whole suite finishes in ~15s even though it imports two 30s-sleep peers —
direct proof the daemon-thread bounding works.

Reviewer notes

  • Module-import-time execution: register_all_tools(mcp) still runs at
    src/scitex/_mcp/__init__.py import (line ~255). The fix bounds that call so a
    hung peer degrades to "its tools missing" instead of hanging the import; it
    does not move the call off import time.
  • No regression: the 5 pre-existing failures in test_mcp_entrypoint.py
    (io/stats/scholar/figrecipe not installed in the sandbox venv) fail identically
    on pristine develop — environmental, not caused by this change.

…gregator

The umbrella `scitex serve` aggregator fronts ~33 packages' MCP tools by
importing each peer's `_mcp_server` and mounting its FastMCP at module-load
time. Previously each peer was resolved SEQUENTIALLY and SYNCHRONOUSLY. If a
single peer's import hangs at init (real case: scitex-todo store-wedge stalls
20s+), the whole `scitex._mcp` import never completes and EVERY peer's tools
go dark — the failure concentrates 33x vs the per-server split.

Fix: resolve each peer's FastMCP concurrently in a bounded DAEMON thread
(`_resolve_peers_bounded`). Threads start at once and are joined against one
shared deadline, so total load time is ~max(peer) bounded by the per-peer
budget — never the sum, and a hung thread can't block interpreter exit. On
timeout / exception the peer is SKIPPED with a warning ("... unavailable this
session") and the healthy peers still mount. Budget is configurable via
`SCITEX_MCP_PEER_TIMEOUT` (default 8s).

Mounting stays on the MAIN thread: only the peer IMPORT is slow/hang-prone;
`safe_mount` is fast and mutates shared parent-server state, so it is kept
single-threaded to avoid concurrent-mutation races.

Preserves all prior behavior: `_is_enabled` env gate, `_NAMESPACE_ALIASES`,
`_SKIP_CATEGORIES`, `register_peer_extras`, `register_umbrella_tools`, the
mounted-count log, and `SCITEX_MCP_USE_<NS>=0` gating. `register_all_tools`
gains injectable `iter_registry` / `peer_timeout` params (defaults unchanged)
so tests drive the real path without patching module globals.

Tests (tests/scitex/test_mcp_bounded_mount.py): real fixture peers written to
disk — a 30s-sleep hung peer, an ImportError peer, a sys.exit peer, fast
peers. Assert the bounded resolve returns within the budget, the hung peer is
skipped + warned, the fast peers still mount, and the env gate holds.
…ked CI collection

PR #347's pytest-matrix hung for 1h30m (job timeout, all 3 py versions) with
"test session starts" printed and then nothing — i.e. a hang during COLLECTION,
before any test ran, where pytest-timeout's per-test SIGALRM cannot fire.

Root cause: the first cut resolved every peer's `_mcp_server` import CONCURRENTLY
(one daemon thread per peer, ~33 fired at once). Importing a large, interdependent
package set from many threads simultaneously races the CPython import machinery and
can deadlock via cross-thread circular imports that never occur when modules are
imported one at a time. That deadlock strikes at `from scitex import _mcp` import
(register_all_tools runs at module load) — which every test module triggers at
collection. `develop` passes because its resolve is sequential; the concurrency
was the regression.

Fix: resolve peers SERIALLY. Each peer still imports in its own bounded daemon
thread (`_resolve_one_peer_bounded`, joined for at most `SCITEX_MCP_PEER_TIMEOUT`,
default 8s) so a hung/wedged import is abandoned + skipped and can never block
interpreter exit — but only ONE import is ever in flight, so imports stay as safe
as develop's sequential loop while gaining the per-peer hang bound. Wall cost is
the sum of the healthy peers' (fast) import times plus the timeout per wedged peer
— a one-time startup cost paid for reliability (the original "~max(peer)"
concurrency goal is knowingly traded away; correctness first).

Verified locally: real `from scitex import _mcp` completes in ~8s and leaves NO
lingering non-daemon threads (so the main-thread timeout signal can always fire);
`test_mcp_entrypoint.py` matches the develop baseline (no new failures, no hang).

Tests: added `_INFINITE_PEER` (import blocks on `threading.Event().wait()`, never
set — the real store-wedge shape, not a bounded sleep) with a fixture asserting the
bounded resolve returns within the budget, skips the wedged peer, and still mounts
the co-resident fast peer. All hang fixtures are written to tmp dirs and imported
ONLY inside the bounded code path at test time — never at module/collection time,
so pytest can never auto-collect or import them into a collection hang.
25 tests pass.
@ywatanabe1989
ywatanabe1989 merged commit f4343f1 into develop Jul 6, 2026
9 of 10 checks passed
@ywatanabe1989
ywatanabe1989 deleted the fix/mcp-aggregator-nonblocking-mount branch July 6, 2026 20:44
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant