MCP client: reconnect budget exhaustion unregisters the server's tools permanently — propose slow-retry keepalive instead

[ybkin1]

Summary

@deepseek-ai/dsh-mcp-client reconnects a dropped MCP server with exponential backoff (0.5s doubling to a 30s cap, maxAttempts default 10). Once that budget is exhausted, scheduleReconnect() disposes all of the server's registered tools and gives up permanently — the log reads giving up after 10 consecutive failed reconnect attempts — tools unregistered; reload the plugin or restart the Host to reconnect. With the defaults, the cumulative backoff window is only ~2.5 minutes (0.5+1+2+4+8+16+4x30s, computed from the documented parameters), so one ordinary server outage longer than that — a deploy, a restart, a network blip — permanently disables the server's tools for the rest of a long session, even after the server comes back healthy.

Background & Evidence

• Location: the exhaustion branch in scheduleReconnect() (dsh-mcp-client/lib/index.js, ~L583 in 0.1.1-rc.2) disposes every entry in the disposers map and logs the error quoted above.
• Operational evidence: we run several MCP servers alongside dsh (a mix of resident local HTTP servers and spawned stdio servers) in long-lived orchestration sessions. In practice, a server restart or an outage of a few minutes during a long session reliably landed in the exhaustion branch; the tools stayed gone until manual intervention (plugin reload or Host restart), even though the server had recovered seconds later.
• The module's documented intent — bounding restart attempts for a crash-looping server — is legitimate. The current remedy, though, is disproportionate for transient outages: permanent unregistration after roughly two and a half minutes of unavailability.

Failure Mode or Reproduction

1. Configure an MCP server in dsh; confirm its tools are registered.
2. Stop the server process (or make its endpoint unreachable) for longer than the cumulative backoff window (~2.5 minutes with default settings).
3. Bring the server back and keep the dsh session running.

Observed: the server's tools remain unregistered for the rest of the session; the log shows the giving up ... tools unregistered error. Only a plugin reload or a Host restart brings them back.

Proposed Fix

Keep the fast exponential-backoff budget exactly as-is (it still bounds the crash-loop case), but replace permanent unregistration at exhaustion with a slow-retry keepalive:

• On failedAttempts > maxAttempts, keep the tools registered and re-schedule connectGeneration(false) on a fixed slow interval (we use 30 minutes; anything in the 5-30 minute range seems reasonable), logging one warn line per attempt.
• Preserve the existing stability-window semantics: once a slow retry connects and the connection stays up for at least maxDelayMs, reset failedAttempts so the next outage enjoys a fresh fast budget.
• Tool disposal (including HMR) still clears the pending timer, so teardown behavior is unchanged.
• Optional: make the policy configurable, e.g. reconnect.exhaustion = "unregister" | "slow-retry" (default slow-retry), so operators who prefer fail-fast can opt back in.

We have a working local patch against 0.1.1-rc.2 implementing the slow-retry approach (a SLOW_RETRY_DELAY_MS constant, the rewritten exhaustion branch, and a docstring update) — happy to share the diff in this thread if that would help.

Additional Context

• dsh: @deepseek-ai/dsh 0.1.1-rc.2 (developer preview)
• OS: Windows 10 x64 (build 10.0.18363)
• Node.js: v22.23.2
• Tradeoff we accept locally: a genuinely crash-looping server now costs one spawn attempt every 30 minutes (visible via the warn line) instead of being parked permanently; we consider that preferable to silent, permanent tool loss.

[liyangbing]

这个问题和 SandBase Harness 的 MCP 生命周期设计有一个直接交集：建议把“连接状态”和“工具注册表”分开管理，连接暂时不可用时不要永久丢弃稳定的工具名与 schema。

SandBase Harness 当前的 MCP Manager 采用两层处理：

1. 工具 wrapper 保持 mcp_<server>_<tool> 名称稳定，实际执行时读取当前 live tool；
2. 工具调用遇到 closed/disconnect/socket/transport 等连接错误时，触发指数退避重连（1s、2s、4s…，上限 60s，最多 5 次），成功后刷新工具定义并重试一次；
3. 重连成功会替换旧 client、更新状态和工具数；失败则标记 server 不可用，而不是把异常吞掉。

实现：

• https://github.com/sandbaseai/sandbase-harness/blob/main/src/core/mcp/mcp-manager.ts
• https://github.com/sandbaseai/sandbase-harness/blob/main/tests/integration/mcp.test.ts#L110-L150

对应测试覆盖了“连接断开后自动重连并重试工具调用”以及“永久故障按退避重连后返回失败”。这不能替代你提议的 exhaustion slow-retry：对于跨越整个 fast budget 的长时间故障，仍建议保留工具标识、进入低频后台重试，并在成功 initialize/tools-list 后恢复可用状态；同时要让 close/HMR/会话 teardown 取消慢重试定时器，避免 crash-loop 变成无限 spawn。