[Bug] prime-agent status kills active IPython forkservers

[sirouk]

Summary

On Linux v0.8.0, prime-agent status --json can terminate active IPython forkservers.

Daemon discovery treats every Unix listener owned by a prime-agent process as a public daemon. That includes /tmp/prime-agent-forkserver-*/control.sock. The status probe connects and disconnects; ForkServer replaces its established control connection with the probe, then interprets the probe closing as forkserver death and terminates the forkserver. Every IPython kernel forked from it exits.

Reproduction

1. Start a Prime session and run a successful IPython cell.
2. From another shell, run prime-agent status --json.
3. Run another IPython cell in the session.

The next cell returns Kernel has been shut down. In the observed incident, two independent worker forkservers were marked inactive at the same millisecond after one status scan.

Proposed fix

• Exclude internal forkserver control sockets from both ss and lsof daemon discovery.
• Refuse any second connection from replacing the established forkserver control channel.
• Add parser and real forkserver-protocol regressions.

This prevents the kernel death itself. Discussion #1659 and closed PR #1693 address the separate recovery problem after an unexpected death.

[sirouk]

Patch: #1696 / ee8ede1. The contribution gate auto-closed the PR because this account is not vouched; the branch remains available for review or cherry-pick.

Validation: 36 focused tests passed with 1 platform-gated skip, and npm run check passed.

[sirouk]

macOS is fixed on the same branch: bb54dea. The fixtures hardcoded /tmp while isKernelForkServerSocketPath anchors to os.tmpdir(), which is per-user on macOS, so the predicate rejected the fixture and three parser tests failed. Fixtures now derive from os.tmpdir(), matching the defaultDaemonSocketDir() convention already used in that file. No product code changed.

Validation on macOS 27 arm64 and Debian 12 aarch64: test/daemon-ps.test.ts test/kernel-fork-server-protocol.test.ts 36 passed with 1 platform-gated skip on both, npm run check clean from a fresh clone of the branch, and reverting either half of the fix fails the tests on macOS.

I checked this against the recovery half rather than duplicating it. #1693 by @coleleavitt cherry-picks onto current main with this fix on top: no conflicts, no shared files. Prevention sits in daemon-ps.ts and fork-server.ts, recovery in kernel/index.ts and tools/ipython.ts. The combined stack is 59 passed with 1 skip across both PRs' tests, npm run check clean, on both platforms: https://github.com/sirouk/prime-agent/tree/integration/kernel-kill-chain

One gap worth naming: the current tests parse fixture strings, so nothing exercises the real discovery path. Driving a live forkserver socket through the actual ss and lsof branches of scanListeningDaemons confirms the tool reports the socket, discovery excludes it, and both the forkserver and its forked kernel survive the probe. I can add that as a regression if it is wanted.

[coleleavitt]

@sirouk I independently reproduced this and I agree with your diagnosis — it's correct end to end. I traced the whole chain through prime-agent, Node core, libuv, and the kernel AF_UNIX layer, then confirmed it with a live socket repro. Sharing the codepath plus two gaps I think are worth closing.

Confirmed codepath

Discovery has no forkserver exclusion. scanListeningDaemons() enumerates every prime-agent-owned unix listener via ss -lxp / lsof. The forkserver socket is created by the Node daemon itself — createServer() + server.listen(socketPath) in fork-server.ts — so ss attributes it to the daemon pid and it's swept in. probeDaemon() then does client.connect(300) and disconnects.

The forkserver kills itself on the probe (fork-server.ts, unpatched):

server.on("connection", (socket) => {
  this.conn = socket;                        // overwrites the live Python control channel
  socket.on("close", () => this.markDead()); // probe disconnect -> markDead()
});

markDead() -> dispose("forkserver died"), and every forked kernel exits.

Why the second connection is even delivered, layer by layer:

• Node net.js onconnection: maxConnections is null by default, so self.emit('connection', socket) fires for every accept — no dedup.
• libuv uv__server_io (deps/uv/src/unix/stream.c): accepts on every POLLIN on the listen fd.
• kernel unix_poll (net/unix/af_unix.c): a listening AF_UNIX socket reports EPOLLIN whenever sk_receive_queue is non-empty; each connect() queues an skb there, independent of the already-accepted child.

Live repro (faithful replica of the handler): a probe connect->close gives forkserverDead: true, serverAcceptedConns: 2, the child channel displaced but still open. With your guard (if (this.conn) { socket.destroy(); return; }): forkserverDead: false, serverAcceptedConns: 1, child kept. So both halves land exactly as you describe, and they're independent of the recovery work in #1659 / #1693.

Two gaps I'd suggest closing

1. resolve() vs realpath in isKernelForkServerSocketPath — fail-open on macOS. The predicate compares resolve(dirname(socketDir)) === resolve(tmpdir()), but resolve() doesn't collapse symlinks. If tmpdir() is /tmp while ss/lsof report the socket under /private/tmp (or a $TMPDIR with a trailing slash / symlink), the check returns false, the socket isn't excluded, and status still kills the forkserver. Empirically, /private/tmp/prime-agent-forkserver-x/control.sock vs tmpdir '/tmp' -> false. This is the same symlink/tmpdir class that already bit your fixtures in bb54dea, and the string-fixture tests can't catch it. Comparing realpathSync(dirname(socketDir)) against realpathSync(tmpdir()) (guarded for a missing path) closes it.

2. Startup race in the connection guard. The socket listen()s before the Python child connects, and the guard is first-come-first-served. If any probe lands in the listen() -> child-connect window, the probe claims this.conn and the real child is refused/destroyed — I get probeClaimedControlChannel: true, realChildRefused: true. The discovery filter closes this for status itself, but the guard alone flips "kill after startup" into "refuse child during startup" for any other prober. Distinguishing the child (a readiness/hello or a token on the control channel) rather than trusting the first connection would make the guard robust on its own.

Neither is covered by the current string-parsing tests — which lines up with the real-ss/lsof / live-socket regression you already said you'd add. Happy to contribute a driven-discovery + live-forkserver test if useful.

For what it's worth on the recovery side: I hit the permanent-Kernel has been shut down state from a real kernel death and put up #1693 (KernelManager onUnexpectedExit -> provisioner drops the dead handle, plus the lazy ensure() guard). It cherry-picks cleanly onto your branch — disjoint files (kernel/index.ts + tools/ipython.ts vs your daemon-ps.ts + fork-server.ts), no conflicts. Prevention + recovery compose into defense in depth. Nice find.

[sirouk]

Confirmed both, independently. The unix_poll / uv__server_io trace matches what I saw.

Gap 1 reproduces, though the trigger is narrower than stated. On macOS lsof reports the bound path, not the realpath, so with TMPDIR unset or TMPDIR=/tmp discovery sees the same string os.tmpdir() returns and the filter holds. It fails when the two processes disagree by alias. Driving a live socket bound under /private/tmp through real lsof output into parseLsofListeners, with os.tmpdir() resolving to /tmp:

socket in raw lsof  : true
pre-hardening       : socket in discovery -> true    (fail-open, status probes and kills it)
hardened            : socket in discovery -> false

So sameRealDirectory() closes a real fail-open, and falling back to resolve() for a vanished path is the right call.

Gap 2 is worse than described. With a delayed wrapper python so the probe provably wins the window, the pre-hardening guard does not merely refuse the child: the probe's close still runs markDead(), so the original kill stays reachable during startup. Three runs each, deterministic:

pre-hardening : probe landed first -> spawn FAILED, ForkServerUnavailable: forkserver died
hardened      : probe landed first -> kernel spawned, alive: true

The guard I wrote only ever covered the window after the child connects. Promoting solely on the ready line is the right shape: no protocol change, onReady is already assigned before listen(), and replaying only the bytes after the ready line keeps onData from double-firing it.

Cross-checks: both new tests fail on the pre-hardening tree (2 failed | 36 passed | 1 skipped) and pass on yours (38 passed | 1 skipped); npm run check exit 0 on macOS 27 arm64 and Debian 12 aarch64; full npm run test:ci on macOS shows no failure on your branch that is not already on main.

Your branch fast-forwards from the PR branch, so #1696 is now at 8003fef: your f208f84 with authorship intact, plus one line adding your attribution to the hardening bullet per the changelog rule in AGENTS.md. Say the word if you would rather carry the hardening yourself and I will drop it back.

A driven-discovery regression would still be worth having, since neither gap was reachable from the string fixtures. Yours covers the symlink and the race directly, so what is left is the live ss / lsof path. Happy to split that with you rather than both writing it.

[coleleavitt]

main...coleleavitt:prime-agent:fix/forkserver-probe-hardening