daemon supervisor: catch-up is never retried, adoption blocks readiness, one rejection kills all #2261
Dmatut7
started this conversation in
Bug reports
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Three supervisor-side defects we hit in production and verified against
main(fb2db8ee1). Fixes + tests exist on our downstream line; happy to turn any of them into PRs.1. A failed client catch-up is never retried or requeued.
daemon-supervisor.ts~:1505 —catchUpClient(client).catch(log); one transient failure and the client keeps a permanently partial view with no signal. We logged 708 of these in production; 86 client/session pairs failed once and stayed stale forever. Our fix mirrors the worker-side retry (bounded backoff, then a terminalsession_snapshot_failedso the client re-pulls).2. Startup readiness awaits adopting ALL workers (
daemon-supervisor.tsstart()), including create() calls with 24h timeouts — one wedged worker blocks the daemon, and clients attaching during the window see an empty session list. We moved adoption off the readiness path and seed roster rows asrecoveringsolistis honest immediately.3.
unhandledRejection→process.exit(1)(daemon-mode.ts~:646): one stray rejection (a single failed fs write in bookkeeping) kills the daemon and every session. Suggest log-and-isolate with a counter and a degraded state, keepinguncaughtExceptionfatal.Full context: https://github.com/Dmatut7/prime-agent-rlm/blob/merge/repl-kernel/docs/fork/upstream-feedback-20260911.md (§2.13 restates the three from the auto-closed #2238).
All reactions