Skip to content

fix(sandbox): a dev server answering 5xx on every route is not "serving" - #7023

Merged
pedrofrxncx merged 1 commit into
mainfrom
fix/sandbox-probe-5xx-not-online
Sep 4, 2026
Merged

fix(sandbox): a dev server answering 5xx on every route is not "serving"#7023
pedrofrxncx merged 1 commit into
mainfrom
fix/sandbox-probe-5xx-not-online

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Problem

The dev-server liveness probe tested "did anything answer", not "is it usable".

probe.head() already computes the HTTP status code — and probe.State throws it away, keeping it only for a log string. So a dev server returning 500 on every route reports StatusOnline. devwatch keys its restart on !Serving (devwatch.go:186), so the watchdog never fires, and the pod is handed over as ready.

Two prod runs today (ELEC-244, ELEC-245) were handed tenant warm-pool pods whose .faststore/ was missing src/pages:

Error: ENOENT: no such file or directory, scandir '/app/repo/.faststore/src/pages'

.faststore/ still contained 404.tsx, 500.tsx and api/, so it looked present; every route 500'd; the watchdog stayed silent. The agents detected and rebuilt it by hand, spending 25 of 55 and ~15 of 82 tool calls on it — plus fighting <defunct> next-server processes left by the pool's own pre-boot.

Change

  • probe.State gains HTTPStatus, set from the code head() already returns.
  • devWatchTick treats a 5xx as not-serving, so devwatch's existing grace window and MaxRestarts budget rebuild the server once.

Gated by DEV_RESTART_ON_5XX, default off — a user's app can legitimately 5xx from its own bug, and restarting that fixes nothing. Ships dormant; enable per-deployment.

Testing

internal/probe/probe_status_test.go stands up real 200 and 500 servers and asserts State.HTTPStatus carries each code while Status stays online — the exact pair the old code could not distinguish. go build ./..., go vet ./..., gofmt -l, and the probe + devwatch suites pass.

Scope

This makes the symptom self-healing. It does not explain why the framework directory ends up half-built on a warm-pool handover — the claim deliberately runs git checkout -f -B under a live dev server (orchestrator.go:265-277), and I have a hypothesis but no reproduction, so I left it alone rather than guess at a second fix. Two follow-ups worth filing:

  • health-gate the pool handover so a 5xx pod is recycled instead of claimed (fixes it without touching a running user app);
  • add PID-1 reaping — the <defunct> next-server processes never clear.

Summary by cubic

Makes the dev-server watchdog treat an all-5xx dev server as not serving, so warm-pool pods with a half-built framework directory get rebuilt instead of handed over ready.

The liveness probe previously counted "answered the request" as online, discarding the HTTP status code it already computed. Now probe.State carries the status code, and devWatchTick treats a 5xx as not-serving when DEV_RESTART_ON_5XX is set. The flag defaults off since a user's app can legitimately 5xx from its own bug.

Testing

  • Adds a test that spins up real 200 and 500 servers and asserts State.HTTPStatus carries each code while Status stays online.

Does not address why the framework directory ends up half-built on warm-pool handover.

Written for commit 78317fb. Summary will update on new commits.

Review in cubic

The probe's liveness test was "did anything answer": `head()` computed the
status code and `State` discarded it, so a dev server returning 500 on every
route reported `StatusOnline`. devwatch keys its restart on `!Serving`, so the
watchdog never fired and the pod was handed over as ready.

Two prod runs today hit this on tenant warm-pool pods whose `.faststore/` was
missing `src/pages` — every route 500, `ls .faststore` truthy — and burned 25
of 55 and ~15 of 82 tool calls detecting and rebuilding it by hand.

Carry the status code on `probe.State`, and let devwatch treat a 5xx as
not-serving so its existing grace window and MaxRestarts budget rebuild the
server once. Behind `DEV_RESTART_ON_5XX`, default off: a user's app can
legitimately 5xx from its own bug and restarting that fixes nothing.

Does not address why the framework directory ends up half-built.
@pedrofrxncx
pedrofrxncx merged commit c2cbdae into main Sep 4, 2026
33 checks passed
@pedrofrxncx
pedrofrxncx deleted the fix/sandbox-probe-5xx-not-online branch September 4, 2026 20:24
decocms Bot pushed a commit that referenced this pull request Sep 4, 2026
PR: #7023 fix(sandbox): a dev server answering 5xx on every route is not "serving"
Bump type: patch

- @decocms/sandbox (packages/sandbox/package.json): 1.61.2 -> 1.61.3
- deploy/helm/sandbox-env (chart 0.16.44) (deploy/helm/sandbox-env/values.yaml deploy/helm/sandbox-env/Chart.yaml): image.tag/appVersion -> 1.61.3

Deploy-Scope: both
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant