[Bug/RFC] Forced restart race can leave a replacement daemon alive but untracked

[brandon-julio-t]

Summary

A forced daemon restart can race with the old process monitor exit path. The replacement process is successfully spawned and recorded as running, then a late state write from the old monitor clears the replacement PID and changes the daemon back to stopped.

The replacement process remains alive but is no longer tracked by the supervisor. If it later exits, retry logic does not restart it because the supervisor already considers the daemon stopped.

Observed production interleaving

This happened on Pitchfork v2.17.0 during a scheduled restart using retrigger = "always":

old PID enters the stop path
old PID is recorded as stopped
replacement PID is spawned
replacement PID is recorded as running
old PID monitor completes late
daemon is recorded as pid=None / stopped

The replacement continued working for several hours while untracked. When it later exited with an application error, retry = true did not take effect.

The relevant invariant appears to be:

A process exit callback for PID P may mutate daemon state only while the current state still belongs to PID P.

Root cause

In v2.17.0, the monitor exit path reads daemon state and later performs an unconditional upsert. The state read and final write are not one atomic ownership check. A replacement can be installed between them.

Current main includes a pre-drain stopping snapshot that narrows one timing window, but the final state transition is still separate from its ownership check.

There is a second related lifecycle race: concurrent forced starts can both observe a startable state while one stop is in progress. A late stop completion can then overwrite the newly registered owner unless check, stop, and spawn are serialized for a daemon.

Deterministic reproduction

The minimal state-level reproduction is:

1. Register daemon state as PID 200, running, with an active port.
2. Complete the old PID 100 monitor exit transition.
3. On the vulnerable path, state becomes pid=None / stopped.
4. Correct behavior is to ignore PID 100 because the current owner is PID 200.

A behavioral reproduction starts restart A, waits until the daemon is explicitly stopping, then starts restart B. The final tracked PID must be alive and every prior supervisor-recorded PID must be dead.

A retry-specific case is also important: readiness failure can be reported before graceful termination completes. A retry must replace its own still-live failed PID, but must not replace a different PID installed by a newer start.

Proposed direction

I have a Draft RFC implementation that:

• records monitor exits atomically and only when the current PID matches;
• serializes check, stop, and spawn transitions per daemon;
• carries the failed attempt PID through retry ownership decisions;
• refuses to spawn a replacement when stopping the previous process failed;
• logs whether an exit transition was recorded or ignored as stale;
• adds state-level and end-to-end regression tests.

Draft RFC PR: #635

Related work

• Discussion #631 is about supervisor crashes orphaning daemons and is a different failure mode.
• PR #620 refactors monitor exit handling and covers nearby races, but does not appear to make the final exit mutation an atomic PID ownership check or serialize concurrent forced lifecycle transitions.

I am happy to adapt the proposed implementation or combine it with #620 if that better fits the intended supervisor architecture.

---

Authorship note: the investigation, reproduction, and proposed implementation were assisted by OpenAI GPT-5.6 Sol with High reasoning effort. The report and changes were reviewed by @brandon-julio-t.

[jdx]

I believe this will be fixed by #632 but I'm overhauling the restart logic anyhow