Repository navigation
Healthy RLM child reported as "Child worker unreachable" after ~2.5 h of continuous work: abandoned wait_for_idle slices exhaust the worker's in-flight permits #4078
viktorhorou
started this conversation in
Bug reports
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Healthy RLM child reported as "Child worker unreachable" after ~2.5 h of continuous work: abandoned
wait_for_idleslices exhaust the worker's in-flight permitsVersion: 0.10.2 release; the same code is on
mainatad1613303.Platform: macOS arm64. The mechanism is platform-independent.
Area:
pa-daemonchild settle watcher, supervisor routing, worker idle wait.What happens
A child session that stays continuously busy for about 2.3-2.5 hours is settled as
errorwith the textChild worker unreachable. The child worker is alive and keeps writing rows, running tools, and sending messages after that verdict. No provider failure is involved.In this park, 138 such failure rows appeared across local sessions; 131 followed more than two hours of work. The median time from the child's first appearance to the failure was 165 minutes. Among 38 failures whose child artifact directory still exists, 34 child sessions have rows after the parent's failure (median +66 minutes, maximum +36 hours). One child wrote its final answer 2.9 minutes before the parent declared it unreachable.
The same windows show hundreds to thousands of
worker_overloaded_counttelemetry events per hour against 0-5 in quiet hours.Mechanism
wait_for_idleslice (crates/pa-daemon/src/rlm_children/lifecycle.rs:394, call at:534).wait_for_idleuntil the session is quiescent, with no deadline (crates/pa-daemon/src/worker/lifecycle.rs:694, handler at:733).WaitForIdlewith a 24 h worker-request budget (crates/pa-daemon/src/supervisor/routing.rs:40) and holds one of the worker's 128 in-flight permits until the worker replies (routing.rs:103; capacity incrates/pa-daemon/src/backpressure.rs:15).crates/pa-daemon/src/supervisor_link.rs:104). The supervisor dispatches commands in a detached task (crates/pa-daemon/src/supervisor/clients.rs:204), so the worker-side wait keeps running and keeps its permit.get_state, gets the instantworker_overloadedrefusal.child_busyread as an unreachable poll (rlm_children/lifecycle.rs:611). After 150 instant refusals (~5 min) it settles the child asChild worker unreachable(:616-621).Predicted trip time is 128 x ~65-67 s + 150 x 2 s ~= 2.4-2.5 h. Observed failures cluster there: 51 of the 138 failures fall in a narrow 145-156 minute window.
This also explains why deleting the child can fail afterward: the worker is saturated, so the kill/shutdown routes cannot acquire permits. It is not a dead process.
Reproduction
Unit level: hold a worker's command channel open without answering, issue 128 client
wait_for_idleroutes with a client-side timeout, let each timeout drop. The nextget_stateon that worker answersworker_overloaded. With a bounded idle wait, each abandoned request releases its permit on its budget and the refusal never appears.End to end: keep one child continuously busy in a long tool call for ~2.5 h (or lower the slice/capacity constants in a test build). The parent watcher reports the child unreachable while its session file keeps growing.
Direction of the fix
I have a focused patch on public fork branches, not submitted as a PR:
main-targeted:
viktorhorou/prime-agent:fix/rlm-unreachable-wait-leak, commitcbfbf7dfc— https://github.com/viktorhorou/prime-agent/commit/cbfbf7dfc.patchv0.10.2 backport for existing installs:
viktorhorou/prime-agent:fix/rlm-unreachable-wait-leak-0.10.2, commitc886e84ee— https://github.com/viktorhorou/prime-agent/commit/c886e84ee.patchadd an optional
timeoutMstoWaitForIdle; the worker answers{"timedOut": true}on its budget instead of parking, so an abandoned request cannot hold a permit;do not advance the give-up streak on
worker_overloadedor supervisor-link reconnects;revive a child whose active id stopped resolving through its durable selector before counting it as gone;
apply the same transient classification to the follow-up usage watcher.
Two regression tests: a bounded
wait_for_idleanswers while the worker is busy, and repeatedworker_overloadedreplies settle a healthy childcompletedinstead ofunreachable. The second test fails with the old classification and passes with the fix.I am opening this as a Discussion because CONTRIBUTING asks for that intake. I can share the branch/patch or open a PR if a maintainer invites it. I have not included session names, prompts, or user data.
Related: #3780 reported the same permit-leak mechanism on 0.10.0. It was auto-closed for vouching, not fixed; the unbounded worker wait and the detached dispatch are unchanged on
main.All reactions