feat(jobs): process-group kill for detached jobs; truncate list COMMAND column#153
Merged
Conversation
…ND column
Detached jobs now launch via setsid so the user command leads its own
process group (pgid recorded in metadata). `jobs stop` signals the whole
group — covering children like torchrun workers whose leader already
exited and would previously linger holding GPU memory — and only reports
success once no live group member remains. Group identity is verified via
the inherited OKDEV_JOB_ID env marker before signaling, so recycled
pids/pgids are never hit. Containers without setsid and jobs from older
okdev versions fall back to the previous leader-only behavior.
The scan line gains a groupLive count ("<alive>\t<groupLive>\t<json>",
older shapes still parse) so stop's terminal criterion is "metadata
terminal AND group drained"; jobs list --json exposes pgid/groupLive.
jobs list also truncates the COMMAND column to the terminal width on TTY
output (piped output and --json keep the full command) so long training
commands no longer wrap table rows.
Includes a Linux-only end-to-end test (CI) that launches a real process
tree, kills the group through the actual signal script, and asserts every
member is gone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ibility dash's kill builtin rejects `kill -SIG -- -pgid` with "Illegal number: -"; the bare "-<pgid>" operand after the signal option works in dash, bash, zsh, and busybox ash (verified against real process groups in all three local shells; CI's /bin/sh is dash). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…back in kind smoke Four scenarios on a real pod: detach logs land under /var/okdev/exec; jobs list --output json reports pgid and live group members; jobs stop kills the whole process tree (verified by scanning /proc for survivors); and after crashing the dev container into CrashLoopBackOff, jobs logs still returns the job's output through the okdev-sidecar fallback. The container-crash step runs last before teardown since it leaves the dev container crashlooping. The long sleeps are nominal workloads — the group kill ends them in seconds, so wall-clock cost is the crash-loop induction (~30-40s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
acmore
added a commit
that referenced
this pull request
Jul 5, 2026
…ND column (#153) * feat(jobs): process-group kill for detached jobs; truncate list COMMAND column Detached jobs now launch via setsid so the user command leads its own process group (pgid recorded in metadata). `jobs stop` signals the whole group — covering children like torchrun workers whose leader already exited and would previously linger holding GPU memory — and only reports success once no live group member remains. Group identity is verified via the inherited OKDEV_JOB_ID env marker before signaling, so recycled pids/pgids are never hit. Containers without setsid and jobs from older okdev versions fall back to the previous leader-only behavior. The scan line gains a groupLive count ("<alive>\t<groupLive>\t<json>", older shapes still parse) so stop's terminal criterion is "metadata terminal AND group drained"; jobs list --json exposes pgid/groupLive. jobs list also truncates the COMMAND column to the terminal width on TTY output (piped output and --json keep the full command) so long training commands no longer wrap table rows. Includes a Linux-only end-to-end test (CI) that launches a real process tree, kills the group through the actual signal script, and asserts every member is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(jobs): drop -- before negative pgid in group kill for dash compatibility dash's kill builtin rejects `kill -SIG -- -pgid` with "Illegal number: -"; the bare "-<pgid>" operand after the signal option works in dash, bash, zsh, and busybox ash (verified against real process groups in all three local shells; CI's /bin/sh is dash). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e): cover detach runtime-volume logs, group stop, sidecar fallback in kind smoke Four scenarios on a real pod: detach logs land under /var/okdev/exec; jobs list --output json reports pgid and live group members; jobs stop kills the whole process tree (verified by scanning /proc for survivors); and after crashing the dev container into CrashLoopBackOff, jobs logs still returns the job's output through the okdev-sidecar fallback. The container-crash step runs last before teardown since it leaves the dev container crashlooping. The long sleeps are nominal workloads — the group kill ends them in seconds, so wall-clock cost is the crash-loop induction (~30-40s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes for multi-pod detached-job ergonomics.
1. Process-group kill (the zombie-process problem)
Problem. Detached jobs had no process-group identity: the wrapper launched the command with
(exec "$@") &andjobs stopsent signals to that single PID only. When a rank crashed, sibling ranks' children (pretrain.pyundertorchrun) survived the leader and kept GPU memory allocated — the next job launch then failed with CUDA OOM. Killing required a second manual pass plus pollingnvidia-smi.Change.
setsid, so it leads its own process group (pgid == pid, recorded in job metadata). Images withoutsetsidfall back to the old behavior withpgid=0and a notice in the job log.groupLive— the count of live (non-zombie) processes still in the job's group, identity-verified via the inheritedOKDEV_JOB_IDenv marker (a pgid number cannot be recycled while any member lives, so a mismatch means an unrelated group and is never signaled).jobs stopsignals the group (kill -SIG -- -pgid) instead of the leader, treats "metadata terminal but group members alive" as still-running (this is exactly the leader-died-children-linger case), escalates TERM→KILL as before, and only reports success once the group is drained — a clean exit now means the process tree and its GPU memory are gone.pgidin metadata), sidecar-less containers, and mixed CLI versions all degrade to the previous leader-only path; the scan-line format change (<alive>\t<groupLive>\t<json>) parses all older shapes.2.
jobs listCOMMAND column truncationLong training commands wrapped table rows, breaking one-line-per-job readability. On a terminal the COMMAND column is now truncated with
…to the remaining terminal width (floor of 16 chars); piped/redirected output and--jsonkeep the full command.--jsonalso exposespgid/groupLiveper pod state.Testing
sh -c 'sleep 300 & sleep 300 & wait'tree, assertspgid==pidin metadata, kills via the real signal script, and polls/procuntil every group member is gone.gofmt -lclean;go test -race ./...passes (e2e skips on darwin).🤖 Generated with Claude Code