Skip to content

Daemon serves HTTP (and starts minds) while shutting down, after the scheduler is already stopped #893

Description

@psamiton

What

shutdown() in packages/daemon/src/daemon.ts stops the schedulers and managers first and calls server.close() last:

scheduler.stop() / sleepManager.stop() / delivery.dispose()   <- early, synchronous
await bridgeManager.stopAll()
await manager.stopAll()                                       <- can take a long while
server.close()                                                <- last

So between SIGTERM and the final server.close() there is a window — potentially tens of seconds, as long as manager.stopAll() takes to reap live SDK subprocesses — where the daemon:

  • still answers GET /api/health as healthy,
  • still accepts writes and still starts minds on request,
  • but no longer runs the scheduler, sleep manager, or delivery.

Why it matters

A daemon in this state is indistinguishable from a live one to any health poll. volute restart polls health to decide a restart succeeded, so it can be satisfied by the outgoing daemon while the incoming one has already lost the port bind with EADDRINUSE and exited — leaving a half-dead daemon that serves requests and fires no schedules. That is the same class of problem as #510, from the other end of the lifecycle.

Observed for real in CI: the #865 e2e restart test spent its whole 90s window waiting on a schedule that could never fire, because health was answered by a daemon whose scheduler had already stopped (test-side race fixed in #892; this issue is the daemon-side half).

Suggested fix

Close (or at least stop accepting new connections on) the HTTP server early in shutdown(), before the long stopAll() awaits, so "healthy" stops being true the moment the daemon stops doing its job. Needs care with in-flight requests — minds POST log/history events during their own shutdown (turn-lifecycle), and those writes currently land in this window — so "stop listening, drain in flight" is likelier right than a hard close.

Worth checking alongside: whether a shutting-down daemon should refuse POST /api/minds/:name/start outright.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions