Skip to content

[AUTOMATED] fix(status): a dead worker printed 'running' — say DEAD - #437

Merged
mahaloz merged 1 commit into
mainfrom
fix/status-pid-liveness
Sep 6, 2026
Merged

[AUTOMATED] fix(status): a dead worker printed 'running' — say DEAD#437
mahaloz merged 1 commit into
mainfrom
fix/status-pid-liveness

Conversation

@mahaloz

@mahaloz mahaloz commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Found by round 4's captain, which wrote in its own notes that it had checked the
builders with ps, "not with status.py, whose stale_s only ticks on a phase change
and still lists b-r4-function-disasse as running". It was right, and it had to work
around the tool this loop gives it for exactly this question.

stale_s is now - updated_at, and updated_at moves on a state update, i.e. on a
PHASE CHANGE -- not on any sign of life. Neither status module calls os.kill, so
nothing ever asks whether the pid exists. reap() does ask, but only past
stale_seconds (1800) and only when something calls it. So a worker that dies mid-phase
reads running with a growing counter for up to half an hour, and the operator cannot
tell it from one working hard on a long phase. One label, two very different states.

collect() now records alive from a kill(pid, 0): True, False, or None when no pid
was recorded or the check itself failed -- unknowable is not the same as dead, and
PermissionError means the process exists under another uid. The repipe table prints
DEAD in the status column when a row claims running and the pid is gone.

Verified by planting a running worker with pid 999999 in the inventory: the row prints
DEAD, and the inventory was restored afterwards. scripts.pipeline.status --json still
works, so the angr lane is unaffected. tools/repipe/smoke.sh 127/127.

Fourth instance of one pattern this session, and the last three were found the same way
-- a number checked for an unrelated reason. Provider refusals counted as kuna failures
(#419); un-refuted hypotheses indistinguishable from undecided ones (#420); a builder
budget cap indistinguishable from a broken build (#436); now a dead worker
indistinguishable from a busy one. Each time, a status collapsed two causes into one
label, and nothing failed loudly.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

🤖 Generated with Claude Code

https://claude.ai/code/session_01YcFmfZndNjgfqLQVBZCdkY

Found by round 4's captain, which wrote in its own notes that it had checked the
builders with `ps`, "not with status.py, whose stale_s only ticks on a phase change
and still lists b-r4-function-disasse as running". It was right, and it had to work
around the tool this loop gives it for exactly this question.

`stale_s` is `now - updated_at`, and `updated_at` moves on a `state update`, i.e. on a
PHASE CHANGE -- not on any sign of life. Neither status module calls `os.kill`, so
nothing ever asks whether the pid exists. `reap()` does ask, but only past
`stale_seconds` (1800) and only when something calls it. So a worker that dies mid-phase
reads `running` with a growing counter for up to half an hour, and the operator cannot
tell it from one working hard on a long phase. One label, two very different states.

`collect()` now records `alive` from a `kill(pid, 0)`: True, False, or None when no pid
was recorded or the check itself failed -- unknowable is not the same as dead, and
PermissionError means the process exists under another uid. The repipe table prints
`DEAD` in the status column when a row claims `running` and the pid is gone.

Verified by planting a `running` worker with pid 999999 in the inventory: the row prints
DEAD, and the inventory was restored afterwards. `scripts.pipeline.status --json` still
works, so the angr lane is unaffected. tools/repipe/smoke.sh 127/127.

Fourth instance of one pattern this session, and the last three were found the same way
-- a number checked for an unrelated reason. Provider refusals counted as kuna failures
(#419); un-refuted hypotheses indistinguishable from undecided ones (#420); a builder
budget cap indistinguishable from a broken build (#436); now a dead worker
indistinguishable from a busy one. Each time, a status collapsed two causes into one
label, and nothing failed loudly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mahaloz
mahaloz merged commit f34cb87 into main Sep 6, 2026
9 checks passed
@mahaloz
mahaloz deleted the fix/status-pid-liveness branch September 6, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant