fix(sandbox): a dev server answering 5xx on every route is not "serving" - #7023
Merged
Conversation
The probe's liveness test was "did anything answer": `head()` computed the status code and `State` discarded it, so a dev server returning 500 on every route reported `StatusOnline`. devwatch keys its restart on `!Serving`, so the watchdog never fired and the pod was handed over as ready. Two prod runs today hit this on tenant warm-pool pods whose `.faststore/` was missing `src/pages` — every route 500, `ls .faststore` truthy — and burned 25 of 55 and ~15 of 82 tool calls detecting and rebuilding it by hand. Carry the status code on `probe.State`, and let devwatch treat a 5xx as not-serving so its existing grace window and MaxRestarts budget rebuild the server once. Behind `DEV_RESTART_ON_5XX`, default off: a user's app can legitimately 5xx from its own bug and restarting that fixes nothing. Does not address why the framework directory ends up half-built.
decocms Bot
pushed a commit
that referenced
this pull request
Sep 4, 2026
PR: #7023 fix(sandbox): a dev server answering 5xx on every route is not "serving" Bump type: patch - @decocms/sandbox (packages/sandbox/package.json): 1.61.2 -> 1.61.3 - deploy/helm/sandbox-env (chart 0.16.44) (deploy/helm/sandbox-env/values.yaml deploy/helm/sandbox-env/Chart.yaml): image.tag/appVersion -> 1.61.3 Deploy-Scope: both
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The dev-server liveness probe tested "did anything answer", not "is it usable".
probe.head()already computes the HTTP status code — andprobe.Statethrows it away, keeping it only for a log string. So a dev server returning 500 on every route reportsStatusOnline.devwatchkeys its restart on!Serving(devwatch.go:186), so the watchdog never fires, and the pod is handed over as ready.Two prod runs today (
ELEC-244,ELEC-245) were handed tenant warm-pool pods whose.faststore/was missingsrc/pages:.faststore/still contained404.tsx,500.tsxandapi/, so it looked present; every route 500'd; the watchdog stayed silent. The agents detected and rebuilt it by hand, spending 25 of 55 and ~15 of 82 tool calls on it — plus fighting<defunct>next-serverprocesses left by the pool's own pre-boot.Change
probe.StategainsHTTPStatus, set from the codehead()already returns.devWatchTicktreats a 5xx as not-serving, so devwatch's existing grace window andMaxRestartsbudget rebuild the server once.Gated by
DEV_RESTART_ON_5XX, default off — a user's app can legitimately 5xx from its own bug, and restarting that fixes nothing. Ships dormant; enable per-deployment.Testing
internal/probe/probe_status_test.gostands up real 200 and 500 servers and assertsState.HTTPStatuscarries each code whileStatusstaysonline— the exact pair the old code could not distinguish.go build ./...,go vet ./...,gofmt -l, and theprobe+devwatchsuites pass.Scope
This makes the symptom self-healing. It does not explain why the framework directory ends up half-built on a warm-pool handover — the claim deliberately runs
git checkout -f -Bunder a live dev server (orchestrator.go:265-277), and I have a hypothesis but no reproduction, so I left it alone rather than guess at a second fix. Two follow-ups worth filing:<defunct>next-serverprocesses never clear.Summary by cubic
Makes the dev-server watchdog treat an all-5xx dev server as not serving, so warm-pool pods with a half-built framework directory get rebuilt instead of handed over ready.
The liveness probe previously counted "answered the request" as online, discarding the HTTP status code it already computed. Now
probe.Statecarries the status code, anddevWatchTicktreats a 5xx as not-serving whenDEV_RESTART_ON_5XXis set. The flag defaults off since a user's app can legitimately 5xx from its own bug.Testing
State.HTTPStatuscarries each code whileStatusstaysonline.Does not address why the framework directory ends up half-built on warm-pool handover.
Written for commit 78317fb. Summary will update on new commits.