fix: an update can't leave the app looking broken — the stampede and the zombie - #268
Merged
Conversation
…the zombie
Two field failure modes from the operator's 0.9.6 update, one incident report:
The stampede (web): a tab open across the update-restart still believed
'unlocked'. Every fetch got 423; each 423 navigated toward /unlock; navigation
re-fired the layout's guarded effects; the guard read the stale state and
fetched again — ~37 req/s observed live, fast enough that the unlock page never
finished loading ('SmartBrain won't start'). Two-part fix: api.ts's 423 path
now notifies a registered lock handler (the account store flips unlocked=false
— a static import would be circular), so guarded effects stop cold on the FIRST
423; and the /unlock navigation is idempotent (no goto when already there).
The zombie (launcher): bifrost.pid recorded a dead process while an Aug-7
0.9.2 bifrost served unrecorded — every Down killed only recorded children,
every Up 'succeeded' against the zombie's health answers while its own fresh
gateway died on the held port. The gateway port now gets the same preflight
discipline as the app port, plus auto-healing: anything serving it at Up time
is by construction a stale instance (a legitimate one was just Down'd and the
records checked), so it is killed by pid — resolved via lsof (darwin), lsof
falling back to ss (linux), or netstat (windows) — and the right version
starts. Refuses to kill self; fails loud with the pid path when lookup fails.
Tests: lock-handler notification + idempotent navigation (DOM-less suite gets
a stand-in window); port-holder resolution against a real listener (asserts
our own pid, skips where no lookup tool exists), kill-self refusal, and the
ss/netstat/lsof output parsers. 190 web + full launcher suite green.
…ated The in-process survivor (same pid as the test) now exercises the fail-loud path: an unkillable gateway-port holder must refuse Up before anything spawns — the self-kill guard makes the test process the perfect unkillable stand-in. The auto-heal path gets its own test with a genuinely separate survivor process (python3 http server): Up must kill it and proceed to the spawn, which the stub gateway's immediate death then proves (awaitChild's error is only reachable past a cleared port). Both skip gracefully where the port is busy — including, fittingly, the operator's machine, where the actual zombie this fix exists for still holds it.
This was referenced Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Operator-directed proper fixes for both halves of today's incident (the workaround was rightly rejected — a user would just give up).
The stampede (web). A tab open across an update-restart still believed "unlocked": fetch → 423 → goto → navigation → guarded effect reads stale state → fetch, at ~37 req/s observed live — so fast the unlock page never finished loading. Now: the first 423 notifies a registered lock handler (the account store flips
unlocked=false; registration avoids a circular import) so every guarded effect stops cold, and the /unlock navigation is idempotent. A tab that relocks lands on the unlock screen once, calmly.The zombie (launcher).
bifrost.pidrecorded a dead process while an unrecorded Aug-7 0.9.2 gateway kept serving — everyDownkilled only recorded children, everyUp"succeeded" against the zombie's health answers while its own fresh gateway died on the held port. Four releases of silent version drift. The gateway port now gets the same preflight the app port has, plus auto-healing: anything serving it at Up time is by construction stale (a legitimate gateway was just Down'd, live records checked), so it's killed by pid —lsof(macOS),lsof→ss(Linux),netstat(Windows) — and the current release's gateway starts. Refuses self-kill; fails loud when lookup is impossible. The gateway now always matches the release, and the operator's live zombie dies at the next update's Up.Tests: handler notification, idempotent navigation, real-listener port-holder resolution (asserts our own pid; skips runners without a lookup tool), kill-self refusal, all three output parsers. 190/190 web, full launcher suite, gofmt/vet clean, bundle rebuilt.