Skip to content

fix(liveness): a process you cannot signal is not a process that is gone - #505

Merged
fujibee merged 7 commits into
mainfrom
fix/liveness-eperm
Jul 28, 2026
Merged

fix(liveness): a process you cannot signal is not a process that is gone#505
fujibee merged 7 commits into
mainfrom
fix/liveness-eperm

Conversation

@fujibee

@fujibee fujibee commented Jul 27, 2026

Copy link
Copy Markdown
Owner

delivery.sh status reports a running bridge and app-server as stale pidfiles under a sandbox. kill -0 answers "can I signal this", not "is this running", and the two differ exactly where it matters: a failed kill -0 is ESRCH (gone) or EPERM (alive, but not signalable by us). Every shipped caller read the exit status directly.

The misreport is the mild symptom. The same check decides whether to start a second bridge, whether to reuse the app-server, and whether a lock's owner is dead enough to reclaim.

What changed

_agmsg_pid_alive in scripts/lib/instance-id.sh already read the ESRCH/EPERM distinction — it was simply not what anything called. It now also cross-checks ps, per the rule #503 settled on for the test helper, so concluding "dead" needs a source that does not depend on signalling permission at all. A fork-free fast path was added first: callers poll this in loops that exist to be fork-free (#466/#496), so the common answer must not cost a subshell.

All 17 liveness sites now go through it. instance-id.sh is sourced where it was reachable only transitively (codex-bridge-launcher.sh, via a conditional branch of role-session.sh) or not at all (codex-monitor.sh, resolve-project.sh).

file sites what a false "dead" did
scripts/delivery.sh 4 printed live watchers as stale; deleted a live watcher's pidfile without killing it; skipped bridge and app-server teardown
codex/_delivery.sh 1 the reported symptom — "Codex bridge: … stale pidfile (pid N not running)" for a live bridge
codex/_session-start.sh 1 launched a second bridge beside the running one
codex/codex-monitor.sh 2 started a second app-server; broke the startup wait early
codex/codex-bridge-launcher.sh 8 reclaimed a live owner's lock — the #485 duplicate-children shape — and bound child lifetime to the wrong pid
scripts/lib/resolve-project.sh 1 the ancestor walk stopped trusting the real session process

resolve-project.sh was the #500 shape exactly: its marker GC already used the EPERM-aware helper (behind a declare -F guard), while agmsg_pid_is_agent twelve lines up still used a bare kill -0.

What a pid is allowed to be

Review turned the validation into its own concern, _agmsg_pid_valid, because each accepted-but-wrong value fails differently and none of them fail safe:

  • 0 is not a pid. kill -0 0 succeeds because 0 addresses the caller's own process group. A digits-only check called it alive, and callers kill what liveness reports alive — kill 0 TERMs the group, the caller included. A corrupt pidfile holding 0 was all it took. Leading zeros go too: nothing writes them and kill(1) may read them as octal.
  • Above the platform ceiling, kill(1) rejects the argument ("not a pid or valid job spec") instead of reporting ESRCH — and everything that is not ESRCH reads as EPERM, i.e. alive. Unbounded, an oversized value read as alive forever: its lock never reclaimed, its bridge never restarted. Bounding the input is what keeps "not ESRCH" meaning "EPERM".
  • The ceiling is the platform's, not one number. A Windows process id is a DWORD and liveness there reads the native process table through tasklist, not kill(1)'s signed pid_t; the POSIX bound would call a legitimate native pid dead. POSIX gets INT32_MAX, MSYSTEM gets DWORD max.

Comparisons are patterns and ${#pid} only — never $(( )) or -gt on the untrusted value, both of which evaluate what they are given.

_agmsg_pid_valid also gates the two launcher sites that kill a recorded pid without asking about liveness first ([ -n "$old_pid" ] && kill ...), which the helper alone would not have covered. Every other kill of a file-sourced pid was already behind _agmsg_pid_alive; the one remaining (codex-monitor.sh's kill "$server_bg") is our own child from $!, not a file.

Tests

  • pid 1 is a real live process this user cannot signal, so it stages EPERM directly rather than simulating it. The tests skip where no such pid exists (running as root, or a container where pid 1 is ours) instead of quietly passing.
  • A dead-pid case guards the other direction: "assume alive on EPERM" must not become "everything looks alive".
  • 0 / 00 / 000 / 0123, and the ceilings on both platforms — the Windows case runs with MSYSTEM set, so both halves are checkable from either host, and widening or narrowing one side alone fails.
  • The sweep — no shipped script under scripts/ or bin/ may call kill -0 outside the helper — is test: close inherited bats fd 3 in all backgrounded process launches #500's lesson as a test. A partially-hardened file reads like a fixed one; this is what stops the next one.
  • The fast path is asserted structurally (the builtin check must come before, and be able to return without, the subshell), not by counting forks, which would be flaky.

Every new test was checked against a mutation that removes the property it claims: EPERM treated as dead, the fast path deleted, the ps cross-check dropped, the pid ceiling removed, the ceiling collapsed to one number, the DWORD ceiling applied everywhere, and a fresh bare kill -0 reintroduced into a shipped script.

CI caught one thing local runs could not. Two shards failed on an intermediate head while every other shard and the full local suite passed. The pre-existing ESRCH tests stubbed kill and passed the literal 999 — which is a running process on some hosts, so the new ps cross-check correctly called it alive. The cross-check was right and the test's assumption was host-dependent; those tests now use a pid that is genuinely gone, and the cross-check got a test of its own.

One existing test changed meaning rather than breaking: marker-gc: skips when _agmsg_pid_alive is unavailable reached its guard by not sourcing the helper, which is the gap this PR closes. It now unsets the helper to reach the same guard, and a companion test pins the new guarantee that sourcing resolve-project.sh alone provides it.

Verification

Full bats suite locally on macOS: 810 passed, 0 failed, 8 skipped (none of them the new ones), plus every affected suite re-run after each review round.

fujibee added 7 commits July 27, 2026 14:16
_agmsg_pid_alive already read the ESRCH/EPERM distinction. Add the ps
cross-check #503 settled on, so "dead" needs a source that does not depend on
signalling permission at all, and a fork-free fast path, so the common answer
still costs no subshell — callers poll this in loops that exist to be fork-free.
`kill -0` answers "can I signal this", not "is this running". Under a sandbox
a live watcher, bridge or app-server fails it, and every shipped caller read
the exit status directly: status printed live processes as stale pidfiles,
session-start and codex-monitor started a second bridge and app-server beside
the running ones, and the launcher reclaimed a live owner's lock — the #485
duplicate-children shape.

All seventeen sites now call _agmsg_pid_alive; instance-id.sh is sourced where
it was only reachable transitively, or not at all.
pid 1 is a live process this user cannot signal, so it stages the distinction
directly; the suite skips where no such pid exists. A dead-pid case guards the
other direction, since "assume alive" must not make everything look alive.

The sweep is #500's lesson as a test: a partially-hardened file reads like a
fixed one, so no shipped script may call kill -0 outside the helper.
… process

`kill -0 0` succeeds because 0 addresses the caller's own process group, so a
digits-only check called it alive, and callers kill what liveness reports
alive: `kill 0` TERMs the group, the caller included. A pidfile holding 0 was
all it took. Leading zeros go too — nothing writes them and kill(1) may read
them as octal.

The check is split out as _agmsg_pid_valid and also gates the two launcher
sites that kill a recorded pid without asking about liveness first.

Reported by review of #505.
…t a number

Past INT32_MAX kill(1) rejects the ARGUMENT rather than reporting ESRCH, and
everything that is not ESRCH reads as EPERM, i.e. alive. An oversized value in
a pidfile was therefore alive forever: lock never reclaimed, bridge never
restarted. Bounding the input is what keeps "not ESRCH" meaning "EPERM".

The ESRCH tests stubbed kill and passed the literal 999, which is a RUNNING
process on some CI hosts — so the new ps cross-check correctly called it alive
and the tests failed there and only there. They now use a pid that is genuinely
gone, and the cross-check gets a test of its own.

Reported by review of #505.
A Windows process id is a DWORD, and liveness there reads the native process
table through tasklist rather than kill(1)'s signed pid_t. The INT32_MAX bound
sat in front of that branch, so 2147483648..4294967295 — legitimate native pids
— were called dead, and a live watcher or lock owner stale.

Reported by review of #505.
The comment under the ceiling selection still explained only the POSIX side,
which reads as if the Windows branch were an exception to a rule rather than
the other half of it.

Reported by review of #505.
@chemica-tan

Copy link
Copy Markdown
Contributor

Ran this branch on Windows 11 (10.0.26200) + Git Bash (MSYS2 3.4.7, bash 5.2.15), where _agmsg_pid_alive takes the tasklist path. The native-pid probe itself works, but we hit what looks like a blocker for the all-site replacement, so leading with that.

Windows blocker found while validating

On Git Bash there are TWO pid namespaces. kill -0 sees MSYS pids (a bash $! / $$) and never native ones — that is the #415 family this PR fixes. tasklist sees native pids and never MSYS ones. This branch's helper goes tasklist-only under MSYSTEM, so every converted call site that holds an MSYS pid flips from "correctly alive" to "always dead". Measured on this branch, on this host:

$ node -e 'setTimeout(()=>{},20000)' &     # $!=1446912 (MSYS), winpid=50312
$ _agmsg_pid_alive 1446912; echo $?        # 1  — live process reported dead
$ _agmsg_pid_alive 50312;   echo $?        # 0  — same process, native pid: alive
$ kill -0 1446912; echo $?                 # 0  — pre-#505 behavior was correct here

Converted call sites that hold MSYS pids (spot-checked on this branch):

  • codex-monitor.sh:166-174server_bg="$!" goes into the app-server pidfile and the port-wait loop; the loop's _agmsg_pid_alive "$server_bg" || break now exits on its first iteration, so a monitored launch on Windows discovers no port, kills its own fresh app-server, and fails open to plain codex (with its warning printed, but delivery off).
  • codex-monitor.sh:204 passes $$ to the launcher as the parent pid; the launcher's parent/lifetime loops (codex-bridge-launcher.sh:245,291,341,381) and runtime-lock owner checks (:90-120) then see a "dead" parent/owner, so the dispatcher never iterates and role children retire immediately.
  • watch.sh:181 writes $$ into the watcher pidfile, and the generic delivery status/kill paths (delivery.sh:248,308,502 on this branch) judge those MSYS pids through the same helper — so Claude Code watchers read as dead too.

Net: this branch fixes the native-pid reads (bridge pids from writeMeta) but inverts the failure onto the MSYS-pid holders on the same host. We would hold the all-site replacement until pid provenance is handled, and we are not counting #458's read half as fixed yet.

A shape that keeps both namespaces visible — and keeps the hot loops fork-free for the common live-MSYS-parent case, preserving #496's builtins win:

case "${MSYSTEM:-}" in
  MINGW*|MSYS*|CLANGARM*)
    kill -0 "$pid" 2>/dev/null && return 0   # MSYS-namespace pid: builtin, no fork
    MSYS_NO_PATHCONV=1 tasklist /FI "PID eq $pid" 2>/dev/null | grep -q "$pid"
    return $? ;;
esac

Cost: a cross-namespace numeric collision can read "alive" for the wrong process — the same tolerance the pre-#505 code already had for MSYS pids, extended rather than inverted. A Windows test that pins BOTH answers (a live $! and its /proc/<pid>/winpid) in one case would keep this from regressing either way.

A second Windows gap, independent of the above (signal side)

Even with liveness answered correctly, the teardown/replace paths still signal with plain kill, which resolves MSYS pids only:

$ tasklist /FI "PID eq 46764" | findstr node    # native bridge pid: alive
node.exe   46764 ...
$ kill 46764
bash: kill: (46764) - No such process           # signal never delivered
$ taskkill //PID 46764 //T //F                  # what actually works
SUCCESS: ... terminated.

The consequence is worse than "cannot stop": stop_codex_bridge removes the run artifacts even when the kill failed (delivery.sh:359-366), and the launcher's rebind path does the same and then spawns a successor (codex-bridge-launcher.sh:462-500) — so on a rebind event the old native bridge survives as an orphan while a duplicate starts. We run a downstream fix for this half (taskkill on the MSYS branch + a cmdline identity check so a recycled pid is never signalled) and can PR it on top of this branch if welcome; it does not need to hold this PR up.

What you can check without a Windows machine

  • The four-line probe above is the minimal repro; it needs any Windows + Git Bash box, nothing else. There is no macOS/Linux analogue (one namespace there — kill -0 and ps agree by construction).
  • Suites: npx bats tests/test_codex_bridge_launcher.bats tests/test_codex_monitor.bats tests/test_delivery.bats tests/test_instance_id.bats. Note the current windows-latest CI job runs only the Windows-filtered cases of tests/test_install.bats, so a green workflow does not exercise these suites on Windows — the results here are from the stated local machine.
  • Content anchors in case line numbers drift: server_bg="$!", kill "$bpid", kill "$old_pid", echo $$ > "$PIDFILE".

Verification scope: Windows only — we have no macOS/Linux measurements for this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EjjWN3rwUxE6qWBicECXHC

@fujibee
fujibee merged commit c1415e9 into main Jul 28, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants