fix(daemon): contain zombie generations from abandoned requests, name mute endpoint holders

[OhOkThisIsFine]

Summary

Fixes a daemon zombie failure observed in production on 2026-08-29 (v0.10.8, win32): after the last busy client disconnected, the daemon process stayed alive for 9+ hours with a dead runtime — holding both \\.\pipe\cbm-daemon-* named pipes and the UI port (HTTP still answered) while daemon status said "not running", daemon start timed out after 30 s with no diagnostic, and every MCP client got CONNECTION_CLOSED.

Root cause

The zombie generation's log shows no daemon.runtime_stopping record and a healthy log sink throughout, so the service never left RUNNING. The wedge: a client disconnected while its application request was in flight and the request never observed cancellation. runtime_worker_finish → runtime_worker_reap_application(wait=true) joined that request thread via cbm_thread_join with no deadline (runtime.c). The disconnect path blocked forever, the worker slot never released, later sessions wedged behind the poisoned application state, and no host timeout could fire because the service still reported RUNNING.

Changes

1. Containment (runtime.c) — the disconnect-path reap now waits at most RUNTIME_ABANDONED_REQUEST_JOIN_TIMEOUT_MS (30 s; session_cancel has already run, so a compliant handler returns in milliseconds), then logs daemon.application_request_unresponsive (peer pid, request token) and fail-stops the process (daemon.forced_shutdown component=application_request_join). The kernel releases pipes/claims and the next client starts a fresh generation. Seam-gated test: daemon_runtime_abandoned_request_join_reaches_containment_in_bounded_time.
2. Mute-holder diagnostics (runtime.c/h, bootstrap.c, main.c) — when a transport connect reaches a live process but no valid answer arrives, the holder's kernel pid is resolved via the existing cbm_daemon_ipc_connection_peer_pid (GetNamedPipeServerProcessId / SO_PEERCRED; previously unused client-side). cbm_daemon_bootstrap_classify_failed_connect treats a mute holder as RESERVED — never absence — so the starter stops spawning doomed competitors, and the daemon start timeout message names the pid with recovery guidance. daemon status prints daemon: not responding (endpoint held by pid N) instead of "not running" (also for wrong-op reject frames from a capacity-wedged generation). Tests: daemon_bootstrap_mute_endpoint_holder_is_reserved_and_never_unavailable, daemon_runtime_mute_endpoint_holder_pid_is_reported.
3. Frontend maintenance-monitor cost (frontend.c) — the quiet-state presence poll drops 10 ms → 250 ms (FRONTEND_MAINTENANCE_IDLE_POLL_MS). Each probe try-acquires the maintenance marker lock, which on Windows revalidates the entire ancestor directory chain; at 100 Hz this burned ~20% of a core for the life of every MCP session (7762 CPU-seconds on one 9.5 h session during the incident). The 250 ms detection latency spends under 2% of the 15 s activation drain budget; the post-detection grace loop keeps its 10 ms pacing.
4. Log rewording (version_cohort.c) — version_cohort.claimed_unheld → version_cohort.claimed_fresh prior_holder=none. The old key reads as a stale-claim anomaly and derailed the incident diagnosis; the record marks the healthy fresh-claim path every normal start logs. No test or script references the old key.

Testing

• New regression tests listed above; full daemon_bootstrap (19/19) and daemon_runtime suites pass locally (WSL Ubuntu, ASan+UBSan).
• One pre-existing test fails in the local venue only: daemon_runtime_process_fingerprint_never_hashes_replacement_path copies /bin/cat to a file named image, which cannot run on uutils/Rust-coreutils systems (Ubuntu 25.10+ multi-call binary; fails identically without this PR). CI runners with GNU coreutils are unaffected.

🤖 Generated with Claude Code

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[DeusData]

Merged as ae5de7fe.

This is an incident report with a patch attached rather than a patch with a story, and the difference shows. The root cause is one line's worth of behaviour — runtime_worker_reap_application(wait=true) joining via cbm_thread_join with no deadline — but the failure it produced was almost undiagnosable from the outside: a process holding both named pipes and the UI port, HTTP still answering, daemon status reporting "not running", daemon start timing out after 30 s with no diagnostic, and every MCP client seeing CONNECTION_CLOSED. Getting from that surface to an unbounded join is the hard part.

What made it mergeable without a round trip:

You proved the negative. No daemon.runtime_stopping record and a healthy log sink throughout is what establishes the service never left RUNNING — which is why no host timeout could fire. That is the observation the whole diagnosis rests on, and you led with it.

Containment is bounded and honest about what it does. Fail-stopping after 30 s, with session_cancel already run so a compliant handler returns in milliseconds, is the right trade against an indefinite zombie: the kernel releases the pipes and the next client gets a fresh generation. A wedged process that reports RUNNING is strictly worse than one that exits loudly.

The mute-holder classification is the part that stops the bleeding twice. Treating a live-but-mute holder as RESERVED rather than absence means the starter stops spawning doomed competitors — the failure mode that turns one wedged generation into a pile of them. Reusing cbm_daemon_ipc_connection_peer_pid, which already existed and was simply never called client-side, is better than adding a mechanism.

7762 CPU-seconds on one 9.5-hour session. The 10 ms presence poll try-acquiring a lock that revalidates the whole ancestor directory chain on Windows is a genuinely expensive accident, and you costed both sides of the change: 250 ms detection latency against a 15 s activation drain budget, with the post-detection grace loop keeping its 10 ms pacing.

I verified the one claim that could have bitten quietly: version_cohort.claimed_unheld appears exactly once in the tree, at its own emission site, with no test, script, workflow or doc consuming it. The rename is safe.

And your /bin/cat observation was right — that is fixed on main now by your own #1921 (0a8a1f29), so that local-venue failure should be gone on your next rebase.