Repository navigation
list_agents: carry a last-activity age per child — bare [running] is indistinguishable from a stalled child #6216
Replies: 1 comment
|
Thanks for this — it is the closest description of our own wall that we have found, and we hit the same "two states look identical" problem from the other seat: the reader sitting in front of the window. Adding the one half I think this proposal cannot cover, plus the measurement that made us keep an external probe. The half that goes blind exactly when you need it most. Our rule has two halves, and both halves have been violated in public reports here.
Baseline round-trip is ~1 ms; a deliberately induced stall produced 63 ms / 476 ms peaks — the ruler does move when the thing it measures moves. The mirror failure, from the other side. #4989 is the same rule broken backwards: a monitor outside the process that treated "health check timed out" as "dead" and force-killed a busy server 8 times in one night, each kill losing in-flight tool calls. Measuring outside is necessary but not sufficient — without a threshold taken from a distribution, "busy" becomes "dead". How this composes with the proposal — two axes, not competitors:
With only the first, a blocked loop freezes both views at once. With only the second, you know the host is alive but not which child is stuck. Ours has fired three times so far in this deployment (two "no answer", one "slow 1.1 s") — small numbers, and we treat them as small numbers. Boundaries. One deployment (Windows + Chromium), 4,386 samples from our own watchdog, readings dated 2026-10-03. A threshold distribution is machine- and load-dependent, so please take the method — sample your own p50/p95/p99 and pick from the tail — rather than the constant. If the shape of the probe is useful I am happy to describe it (~40 lines: an interval sampler, a sliding-window peak, and a sticky-clear rule so a spike does not stay lit after recovery). No patch offered — the repository does not accept external PRs. |
Uh oh!
There was an error while loading. Please reload this page.
The problem
A background subagent (continuable child) emits nothing to its parent between dispatch and final message — by design, and fine. But it means the parent's entire liveness view is
list_agents' status word plusjob_output's(no new output):These are exactly what a healthy, working child looks like. They are also exactly what a stalled child looks like. The model on the parent seat has no data to tell the two apart — and in practice it fills the gap with fabricated elapsed times.
A real incident (2026-09-10)
A manager session dispatched three cross-family review children. Reconstructed afterward from the children's own session transcripts:
job_killlandedEvery
turn/endreadsaborted by parent. Worse, the kills were self-fulfilling: interrupting a child mid-generation persists only the reasoning block, so all three kills produced an empty closing message that was then cited as further evidence of hanging. Every hypothesis the parent recorded (approval-prompt parking, provider outage, background-turn bug) was disproven by the transcripts. The root cause: a parent model judging liveness without a liveness signal.Proposal
Give
list_agentsrows a last-activity age, so running and progressing and running and stalled are one glance apart:The runtime already has the data — every session event carries a timestamp. Two shapes we can see:
listChildren/listDescendantsrows addlastActivity(epoch ms of the child's most recent session event — live children: last log event; cold observation: last observed event; projection-cache hit: omit), and thelist_agentsrender surfaces it humanized. Roughly 30 lines acrossdsh-subagentanddsh-tool-subagent-control; the tool description should also teach the semantics (fresh = progressing, stale = last step hasn't returned).session/event(per-session last-append map) plus atools/post-executewaterfall that rewriteslist_agentscontent lines. Works unmodified on 0.1.5-rc.1.Impact
Any workflow that dispatches background workers — reviews, research fan-out, long builds — hits the same decision point: kill or wait? Guessing wrong either throws away a child's completed work (the incident above lost ~15 minutes of finished review across three children) or parks the parent on a genuinely dead child. A per-row age turns a vibe call into a data read.
Happy to send the native change as a PR if the direction is welcome.
All reactions