[Bug]: MCP stdio watchdog kills live servers when psutil create_time changes

[Bruce-anle]

Bug Description

The POSIX stdio MCP watchdog can classify a live Hermes parent as orphaned and terminate the MCP server process group.

The watchdog records psutil.Process(parent_pid).create_time() when it starts and later requires exact float equality. The public Process.create_time() value is not a stable process-identity contract when the system clock or Linux boot-time basis changes.

This was observed on Windows 11 with WSL2 Ubuntu while both Hermes Gateway parent processes remained alive. The incorrect decision is also reproducible deterministically by holding the PPID constant while returning a different public create-time value.

Steps to Reproduce

1. Run this deterministic probe against the current implementation:

   from unittest.mock import patch
   from tools import mcp_stdio_watchdog as watchdog
   
   with (
       patch.object(watchdog.psutil, "pid_exists", return_value=True),
       patch.object(watchdog.psutil, "Process") as process,
   ):
       process.return_value.create_time.return_value = 101.0
       print(watchdog._is_orphaned(4242, 100.0, getppid=lambda: 4242))

2. The result is True even though the direct parent PID is unchanged and still exists.

3. On WSL2, run a Gateway with a stdio MCP server and sample the Gateway's public create_time() using fresh psutil.Process(pid) instances over time.

4. If the public value drifts, the next watchdog poll follows the same deterministic branch, sends SIGTERM to the MCP process group, and Hermes reconnects the server with a new generation.

Expected Behavior

A watchdog that remains a direct child of the same live Hermes process should not terminate its MCP child process group.

Actual Behavior

In one observed failure window:

• profile A's stdio MCP connection advanced from generation 1 to generation 2;
• profile B's stdio MCP connection advanced from generation 1 to generation 2;
• the generation-1 watchdog exited with returncode=-15, signal=15, parent_alive=False;
• both Gateway MainPIDs remained active;
• profile A's same MainPID create time changed from 1783705305.82 to 1783705306.82;
• profile B's same MainPID create time changed from 1783705305.76 to 1783705306.76.

The resulting chain was:

create_time mismatch -> false orphan -> MCP SIGTERM -> closed stdio transport -> reconnect and generation increment.

Root Cause Analysis

As of upstream main 02063ece119648eeb8e7603c857b03aceda22ffc, tools/mcp_stdio_watchdog.py compares a freshly read public Process.create_time() value with the epoch value captured by tools/mcp_tool.py.

psutil issue giampaolo/psutil#2526 and merged PR giampaolo/psutil#2527 explain that the public create_time() value may change when the system clock changes. The psutil fix made its internal process identity monotonic; it did not make the public epoch timestamp immutable.

For this watchdog, the process is a direct POSIX child of Hermes. The kernel-maintained relationship is sufficient:

def _is_orphaned(original_ppid: int, getppid=os.getppid) -> bool:
    return getppid() != original_ppid

When the Hermes parent exits, the existing watchdog is reparented and its PPID changes. Reusing the old numeric PID cannot attach the existing watchdog to a new process.

Proposed Fix

• Remove the internal --create-time watchdog argument.
• Stop sampling and comparing psutil.Process.create_time() in the MCP watchdog path.
• Use the original direct PPID as the parent-liveness contract.
• Keep the existing process-group termination, signal forwarding, stdio pass-through, and non-POSIX behavior unchanged.

I have a focused patch and regression tests ready and am willing to submit a PR.

Environment

• Hermes Agent 0.18.2
• Windows 11 + WSL2 Ubuntu
• Python 3.11.15
• psutil 7.2.2
• Affected component: Tools / MCP stdio transport and Gateway usage

Related Work

• Hermes PR #42198 handles an unreadable create-time sentinel in the slash worker, not runtime drift in the MCP watchdog.
• Hermes PR #60211 addresses MCP stream ownership and event-loop shutdown.
• Hermes PR #32962 is a broader WSL2 subprocess-resilience change and does not remove this exact comparison.
• Hermes PR #61364 covers desktop backend parent death and currently uses the same public create-time assumption on a different surface.

[Ahmett101]

PR submitted: #62530

[kwt052]

Confirming the same failure on WSL2 (Ubuntu under Win11) — the getppid-only fix you proposed is exactly right; direct-child liveness needs no create_time at all. Two additions from our repro that aren't in this thread yet:

1. A third site has the same defect: hermes_cli/active_sessions.py lease validation compares create_time() with abs(current_start - expected_start) < 0.001. A 1 ms tolerance still breaks under btime drift (we saw 1–4 s steps within minutes on WSL2), so session leases can mis-validate the same way.
2. The codebase already contains the correct primitive: gateway/status.py::_get_process_start_time reads field 22 (starttime) of /proc/<pid>/stat as a raw integer tick count, with a docstring explaining exactly why the epoch-based value is unstable. The watchdog / slash_worker / active_sessions paths could reuse that instead of create_time().

Workaround for anyone blocked before the fix lands, without patching core: inject an import-shadowing psutil stub on PYTHONPATH for the MCP subprocess so the watchdog's built-in getppid-only fallback engages.