Summary
Long-lived Hermes worker processes accumulate orphaned MCP stdio subprocesses over time. On a Minions/gateway deployment running the memory MCP server (mimir), a single worker process (PID 292 in our case) accumulated 53 child mimir processes holding 1.4 GB RSS and 38 open fds each, all against the same SQLite DB. Past ~50 children the DB handle contention makes memory tool calls fail intermittently ("Unknown tool" / lock errors), and the vault appears "down" while hermes mcp test mimir still reports healthy (the gateway bridge stays up).
This is a slow leak: it recurs across sessions and rebuilds after any manual cleanup.
Environment
hermes-agent 0.18.0- Linux container, long-lived worker (gateway spawns a fresh
AIAgent per message; workers are persistent)
- MCP server configured as stdio:
mcp_servers.mimir.command=<binary> --db <path>
Root cause (traced in tools/mcp_tool.py)
MCP stdio servers are process-global and intentionally reused across per-message agents (_servers registry, register_mcp_servers() is idempotent for already-connected names — line ~4304-4311). That reuse is correct.
The leak is in the reconnect / keepalive path of MCPServerTask:
_keepalive_probe() / _wait_for_reconnect_or_shutdown() (~lines 1640, 1762) trigger a reconnect when a probe fails or the transport churns.- Each reconnect enters a fresh
stdio_client(...) (_run_stdio, ~line 1865), spawning a new child process.
- The predecessor child is not deterministically terminated on the reconnect transition. The code acknowledges surviving children exist ("stdio subprocesses that survived the graceful shutdown are now orphaned", ~line 2970) but only reaps them in
_stop_mcp_loop() via _kill_orphaned_mcp_children(include_active=True) — i.e. only when the whole MCP loop is torn down, which never happens in a long-lived worker.
Net: every reconnect over the worker's lifetime can leave one orphaned child behind. They accumulate until DB contention degrades the server.
Additionally, AIAgent.close() (run_agent.py ~line 3432) cleans up background processes, terminal sandboxes, browser daemons, child agents, and HTTP clients — but does not include MCP stdio server teardown. So there is no per-agent path that reaps them either (nor should there be, given global reuse — but it means the only reaper is loop-teardown).
Suggested fix
Reap the predecessor child on every reconnect transition, not only on loop teardown. Concretely, in MCPServerTask's reconnect path, call the existing per-process terminate helper (_terminate_process_tree / _kill_orphaned_mcp_children scoped to this server's prior PID) before/after establishing the new stdio_client, so at most one live child per registered server exists at any time. The orphan-tracking that already feeds _kill_orphaned_mcp_children(include_active=True) has the PIDs; it just needs to run on reconnect, scoped to the reconnecting server, instead of exclusively at loop stop.
Workaround (for others hitting this)
A cron reaper that keeps the child count bounded, protecting the newest N and the parent worker:
- detect processes matching the server command (
<binary> --db)
- if count > threshold, SIGTERM the oldest
(count - keep_newest), then SIGKILL survivors
- never signal the caller's own process ancestry or the parent worker
This keeps the server healthy but is a band-aid; the reconnect-path reap is the real fix.
Repro sketch
- Run a worker with a stdio MCP server configured.
- Induce transport churn / let keepalive probes fail intermittently over a long uptime (or force reconnects).
- Observe child count of the MCP server binary climbing without bound; only a full MCP-loop teardown clears them.
Summary
Long-lived Hermes worker processes accumulate orphaned MCP stdio subprocesses over time. On a Minions/gateway deployment running the memory MCP server (
mimir), a single worker process (PID 292 in our case) accumulated 53 childmimirprocesses holding 1.4 GB RSS and 38 open fds each, all against the same SQLite DB. Past ~50 children the DB handle contention makes memory tool calls fail intermittently ("Unknown tool" / lock errors), and the vault appears "down" whilehermes mcp test mimirstill reports healthy (the gateway bridge stays up).This is a slow leak: it recurs across sessions and rebuilds after any manual cleanup.
Environment
hermes-agent0.18.0AIAgentper message; workers are persistent)mcp_servers.mimir.command=<binary> --db <path>Root cause (traced in
tools/mcp_tool.py)MCP stdio servers are process-global and intentionally reused across per-message agents (
_serversregistry,register_mcp_servers()is idempotent for already-connected names — line ~4304-4311). That reuse is correct.The leak is in the reconnect / keepalive path of
MCPServerTask:_keepalive_probe()/_wait_for_reconnect_or_shutdown()(~lines 1640, 1762) trigger a reconnect when a probe fails or the transport churns.stdio_client(...)(_run_stdio, ~line 1865), spawning a new child process._stop_mcp_loop()via_kill_orphaned_mcp_children(include_active=True)— i.e. only when the whole MCP loop is torn down, which never happens in a long-lived worker.Net: every reconnect over the worker's lifetime can leave one orphaned child behind. They accumulate until DB contention degrades the server.
Additionally,
AIAgent.close()(run_agent.py~line 3432) cleans up background processes, terminal sandboxes, browser daemons, child agents, and HTTP clients — but does not include MCP stdio server teardown. So there is no per-agent path that reaps them either (nor should there be, given global reuse — but it means the only reaper is loop-teardown).Suggested fix
Reap the predecessor child on every reconnect transition, not only on loop teardown. Concretely, in
MCPServerTask's reconnect path, call the existing per-process terminate helper (_terminate_process_tree/_kill_orphaned_mcp_childrenscoped to this server's prior PID) before/after establishing the newstdio_client, so at most one live child per registered server exists at any time. The orphan-tracking that already feeds_kill_orphaned_mcp_children(include_active=True)has the PIDs; it just needs to run on reconnect, scoped to the reconnecting server, instead of exclusively at loop stop.Workaround (for others hitting this)
A cron reaper that keeps the child count bounded, protecting the newest N and the parent worker:
<binary> --db)(count - keep_newest), then SIGKILL survivorsThis keeps the server healthy but is a band-aid; the reconnect-path reap is the real fix.
Repro sketch