[Bug] v0.8.0 stopped IPython kernel remains permanently cached after unexpected exit

[Atroci]

Prime Agent 0.8.0 still leaves a session permanently unable to execute Python after an unexpected IPython kernel exit.

Reproduction

1. Start a session and execute a successful IPython cell.
2. Execute import os; os._exit(23) to simulate an unexpected kernel exit.
3. Execute 1 + 1 in a second explicit IPython call.

Without the patch, every call from step 3 onward returns Kernel has been shut down. The agent session remains responsive, so autonomous or heartbeat prompts can keep consuming turns without a working tool.

Cause

KernelManager correctly becomes non-running after the unexpected exit, but IpythonKernelProvisioner retains both its resolved managerPromise and startedManager. Later ensure() calls return the same stopped manager instead of following the existing full provisioning path.

This is the unrecoverable lifecycle previously discussed in #764. The v0.8.0 cold-boot and parent-watchdog fixes address different failure modes; current main still caches the stopped manager.

Minimal fix tested locally

Before ensure() reuses its memo, discard a resolved startedManager whose isRunning is false. The next call then reuses existing startKernel(), including snapshot restore and runtime bootstrap. Do not replay the failed cell.

Patch commit: Atroci@9c09cc8
Automatically closed PR: #1658

Validation:

• focused provisioner suite: 17 tests passed
• npm run check: passed
• live isolated v0.8.0 smoke: kill kernel with os._exit(23), second IPython call returned 2

If this direction is accepted, I can update the patch to match maintainer feedback.

[coleleavitt]

I hit this exact bug independently on v0.8.0 (Linux) and can confirm it still reproduces on current main (e319a66d7) — the provisioner keeps handing back the stopped manager, so every later cell fails with Kernel has been shut down.

While diagnosing a real session, I traced it back one layer further than the ensure() cache: KernelManager never tells its owner when the kernel dies on its own. The three unexpected-death sites (spawn error, process exit, and checkForkedKernelDeath for forkserver-forked kernels) just set state = "shutdown" and go silent, so the provisioner only finds out lazily on the next ensure().

So I built on the lazy-guard direction from #1658 with a small superset that recovers eagerly too:

• KernelManager.onUnexpectedExit — fired at most once, and only for deaths the caller did not initiate (deliberate shutdown/kill/dispose transition to "shutdown" first, so those stay silent). Wired into all three death sites.
• Provisioner uses that callback to drop startedManager / managerPromise / _lastRestore / kernelManagerRef.current the moment the kernel dies.
• ensure() lazy guard still drops a non-running memoized manager before provisioning, as a belt-and-suspenders for any death that races between calls.
• The failed cell is never replayed; the next explicit call re-runs the normal startKernel() path (snapshot restore + runtime bootstrap), and concurrent recovery calls stay coalesced onto one startup.

Validation:

• test/ipython-provisioner.test.ts + test/kernel-startup.test.ts: 23 passed (lazy guard, eager callback, newer-manager race, once-only firing, and the deliberate-shutdown no-fire path).
• biome + typecheck clean on the changed files.

Branch (single commit, only the fix + a changelog fragment): https://github.com/coleleavitt/prime-agent/tree/fix/ipython-kernel-reprovision

Happy to open a PR or adjust the shape to match your preferred direction (eager callback vs. lazy-only) if a maintainer wants this picked up.

[Atroci]

Update: I tested the eager-callback branch at 3e9b8f7. Its focused kernel tests pass: 23/23.

I also applied the equivalent runtime changes to a local 0.8.0 install, with a rollback copy, and ran an isolated smoke test: os._exit(23) killed the first kernel; the callback cleared the memoized manager; ensure() provisioned a fresh kernel; and 1 + 1 returned 2. This closes the stale-manager path better than the lazy guard alone, including deaths between calls. The currently running CLI still needs a restart to load the patched bundle, but this looks like the right direction.

[KiOs13]

Additional independent validation on current main (9bc005574): I reproduced the stale-manager failure and prepared a local patch that:

• discards a stopped memoized manager before provisioning;
• retries execution at most once only when the manager is no longer running;
• never retries after abort and preserves the existing busy-kernel flow;
• adds PID and stderr-tail context to unexpected-shutdown errors.

Validation: 42 focused and adjacent kernel tests passed, and npm run check passed.

The one-time replay is the material difference from the lazy-only and eager-callback branches already discussed here. Because a cell may have produced partial side effects before the kernel exited, replay may be unsafe. Before I publish a branch, would maintainers prefer the existing no-replay recovery semantics, or is a bounded automatic replay desired for this tool path?