MCP stdio keepalive probe triggers infinite reconnect loop on empty exceptions

## MCP stdio keepalive probe triggers infinite reconnect loop on empty exceptions

### Summary

When a stdio MCP server process dies or becomes unresponsive, the keepalive probe (`_keepalive_probe`) fails with an **empty-message exception** (`str(exc) == ""`). The `_is_method_not_found_error` function returns `False` on empty strings (line 506: `if not msg: return False`), so the exception propagates and triggers a reconnect. The reconnect spawns a new process, which also dies, which also triggers a reconnect — **infinite loop, no backoff, no cap.**

Observed with **both** `daedra` (Rust stdio) and `discord` (Node stdio) simultaneously — 3,082 and 3,130 reconnects respectively across ~63 hours. The failures are correlated (daedra fails, discord fails ~19s later, every cycle).

### Steps to reproduce

1. Have two or more stdio MCP servers configured
2. Let one or both become unresponsive (process crash, pipe closure, resource starvation)
3. Watch `~/.hermes/logs/agent.log` — keepalive failures fire every ~30s with no backoff
4. Each failure spawns a new process (which also fails), no orphan cleanup between probes

### Evidence

From `~/.hermes/logs/agent.log`:
```
2026-07-09 20:29:33,028 WARNING tools.mcp_tool: MCP server 'daedra' keepalive failed, triggering reconnect: 
2026-07-09 20:29:52,543 WARNING tools.mcp_tool: MCP server 'discord' keepalive failed, triggering reconnect: 
2026-07-09 20:30:52,218 WARNING tools.mcp_tool: MCP server 'daedra' keepalive failed, triggering reconnect: 
2026-07-09 20:32:11,804 WARNING tools.mcp_tool: MCP server 'discord' keepalive failed, triggering reconnect: 
... (3,082 discord + 3,130 daedra reconnects over 63 hours)
```

Note the **empty exception message** after "triggering reconnect: " — `str(exc)` is `""`.

### Root cause

Two issues in `tools/mcp_tool.py`:

**1. `_is_method_not_found_error` returns `False` on empty `str(exc)` (line 506)**

```python
msg = str(exc).lower()
if not msg:
    return False  # ← empty exception → not "method not found" → propagate → reconnect
```

When the exception has no message (e.g. `CancelledError` with no args, silent pipe closure, process death), the function can't distinguish it from a real failure. The exception propagates to the caller, which triggers reconnect.

**2. `_wait_for_lifecycle_event` has no backoff on keepalive-triggered reconnects (line 1944)**

```python
except Exception as exc:
    logger.warning("MCP server '%s' keepalive failed, triggering reconnect: %s", self.name, exc)
    self._reconnect_event.set()  # ← immediate reconnect, no delay, no cap
    break
```

The reconnect event fires immediately. The outer `run()` loop has exponential backoff for *transport* failures, but keepalive-triggered reconnects bypass that — they fire `_reconnect_event` which causes `_wait_for_lifecycle_event` to return `"reconnect"`, and the `run()` loop immediately re-enters `_run_stdio()` without any delay.

### Suggested fix

**Minimal (what I'll apply locally):** In `_is_method_not_found_error`, treat empty `str(exc)` as a potential "ping unsupported" case rather than a hard failure. If the exception has no message AND no structural `error.code`, it's ambiguous — latch `_ping_unsupported = True` and fall back to `list_tools` instead of triggering reconnect.

```python
msg = str(exc).lower()
if not msg:
    # Empty exception message — ambiguous. Could be pipe closure (real failure)
    # or a server that doesn't implement ping (returns nothing).
    # Latch ping_unsupported so the probe falls back to list_tools on next cycle.
    # If list_tools also fails with empty message, THAT triggers reconnect.
    return True  # treat as "method not found" → fallback to list_tools
```

**Better (for upstream):** Add exponential backoff specifically for keepalive-triggered reconnects, capped at 5 minutes. This prevents the infinite loop even if the server is genuinely dead:

```python
except Exception as exc:
    logger.warning("MCP server '%s' keepalive failed, triggering reconnect: %s", self.name, exc)
    self._keepalive_reconnect_count = getattr(self, '_keepalive_reconnect_count', 0) + 1
    if self._keepalive_reconnect_count > 3:
        logger.error("MCP server '%s': keepalive reconnect loop detected (%d attempts), backing off",
                     self.name, self._keepalive_reconnect_count)
        self._shutdown_event.set()
        return "shutdown"
    self._reconnect_event.set()
    break
```

### Impact

- **3,082 + 3,130 = 6,212 process spawns** in 63 hours across two servers
- Discord token reset by Discord abuse detection (401 after rapid-fire gateway logins)
- Event loop stalls from orphan reaping (each reconnect runs `_kill_orphaned_mcp_children` via `asyncio.to_thread`, blocking up to 2s)
- Related: #60385 (process leak), #30268 (simultaneous keepalive failures), #59568 (lifetime retries counter)

### Environment

- hermes-agent: 0.18.2 (pip)
- OS: Ubuntu (WSL2, 54GB RAM, no OOM)
- Python: 3.11
- MCP servers: daedra (Rust stdio), discord (Node stdio)
- 5 concurrent agent sessions running


Navigation Menu

Uh oh!

MCP stdio keepalive probe triggers infinite reconnect loop on empty exceptions #62212