[Bug] One unresponsive session worker blocks daemon startup and stampedes rival supervisors

[mdumitrean]

Affected area

Agent core

What happened?

What happened

A session worker alive and still accepting connections but no longer answering IPC wedged the daemon for minutes. The Lock file is already being held error users saw is collateral, not the cause:

• start() takes the exclusive socket-path lease first and holds it until shutdown — it must, since that lease is what guarantees ownership of the socket file.
• start() then awaits adoption of every persisted worker. A wedged one costs connect(2s) + list(5s), then recoverWorker's ladder — 14.5s measured against a stub, longer against a real worker.
• daemon_hello is gated on this.ready, which resolves only when start() finishes. The socket listens the whole time but no client completes the handshake, so every CLI concludes there is no daemon and spawns its own supervisor.
• Each rival waits 600 × 25ms = 15s on the lease, dies with proper-lockfile's bare ELOCKED, and the launcher's 2s post-exit grace blames the exit — so users retry into the same stampede.

Steps to reproduce

The commit gives two, and labels which is which. The deterministic one is npx vitest --run test/daemon-supervisor-startup-adoption.test.ts — a real supervisor against a worker socket that accepts connections and answers nothing. That is the one I actually ran, in both directions (it hangs past the budget with the fix removed).

The manual one — start a session, kill -STOP the worker so its listen backlog keeps completing connections while nothing answers, kill the supervisor only, then run two CLIs at once — is a reconstruction of your incident from the daemon log. I did not run it, and the commit says so rather than presenting it as verified.

Expected behavior

A wedged worker degrades its own sessions only; the supervisor comes up and answers daemon_hello promptly, with that worker's recovery continuing in the background the same way a runtime disconnect is handled. A supervisor losing the lease says which path it lost and that a rival holds it. A CLI losing the spawn race waits for the winner.

Prime Agent version

9e49b73

Environment

macOS

Additional context

No response

[KoLyslaV]

Confirming this on 0.8.1 in production use, with 10 days of numbers from one Linux host, and one extra client-side detail.

Field data (default socket, client-errors.log + daemon log):

• 479x sent no recognizable hello; treating as stale in 10 days (258x on the worst day)
• 59x Daemon supervisor startup failed: Error: Lock file is already being held
• worst observed recovery window: ~2 minutes of every client failing while the new supervisor adopted 7 dead + 1 wedged worker (Timed out connecting to daemon session worker: Timed out waiting for daemon worker hello, then Recovered worker ... without replaying uncertain operations)

The extra detail: the stampede is made worse by the replace path. probeDaemonVersion declares the booting daemon stale after the 2s hello budget, then shutdownStaleDaemonIfNotBusy tries to verify idleness; since the daemon is pre-ready, the session query fails too, and the catch treats "unverifiable" as busy -> refusing to replace stale daemon: busy session(s) present -> StaleDaemonError. So clients neither connect nor replace, in a loop, for the whole recovery window. An early daemon_booting greeting on accept (before this.ready) would let clients distinguish "booting" from "dead" and back off instead of stampeding.

(Also related: the same 2s budget issue was reported in auto-closed issue #1661.)

[mdumitrean]

I’ve solved a few of these bugs on my fork.Best,MariusSent from my iPhoneOn Aug 28, 2026, at 22:02, Štěpán Koláček ***@***.***> wrote: Confirming this on 0.8.1 in production use, with 10 days of numbers from one Linux host, and one extra client-side detail. Field data (default socket, client-errors.log + daemon log): 479x sent no recognizable hello; treating as stale in 10 days (258x on the worst day) 59x Daemon supervisor startup failed: Error: Lock file is already being held worst observed recovery window: ~2 minutes of every client failing while the new supervisor adopted 7 dead + 1 wedged worker (Timed out connecting to daemon session worker: Timed out waiting for daemon worker hello, then Recovered worker ... without replaying uncertain operations) The extra detail: the stampede is made worse by the replace path. probeDaemonVersion declares the booting daemon stale after the 2s hello budget, then shutdownStaleDaemonIfNotBusy tries to verify idleness; since the daemon is pre-ready, the session query fails too, and the catch treats "unverifiable" as busy -> refusing to replace stale daemon: busy session(s) present -> StaleDaemonError. So clients neither connect nor replace, in a loop, for the whole recovery window. An early daemon_booting greeting on accept (before this.ready) would let clients distinguish "booting" from "dead" and back off instead of stampeding. (Also related: the same 2s budget issue was reported in auto-closed issue #1661.) —Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[mdumitrean]

I'm doing long research tasks, prime agent with GPT 5.6 Sol has been incredible.

So far i've run into a few bugs related to long running tasks. I tried to open pull requests but I'm not a trusted person :). I hope they fix these in the main build. You can merge with my fixes, they solve a few of these long task running bugs with sub agents failing or stopping to respond. Saves me from having to waste tokens on the long running tasks and having to resume.