fix(sandbox): force-reprovision a provisioning session orphaned by a crashed start - #5183
Merged
Merged
Conversation
…crashed start
A per-branch shared-sandbox session stuck in `status='provisioning'` never
reboots: `beginStart` only resets the row (bump generation, clear the stale
handle, force a fresh provision) when `desired_state='stopped'` or
`status IN ('missing','failed')`. A `provisioning` row with
`desired_state='running'` re-joins its own generation and returns a stale/null
handle, so nothing ever reprovisions it. Nothing times out `provisioning`
either, so it "reserves forever" — the symptom seen when an old branch's
long-lived session was left half-provisioned by a start that crashed between
`beginStart` and `completeStart`/`failStart` (e.g. the studio pod died mid-provision).
`beginStart` now also treats a `provisioning` row older than 5 minutes on
`updated_at` as stale and resets it, bumping the generation to fence out any
zombie in-flight writes from the dead attempt. 5m is safely longer than the
~180s real provision wait and matches the k8s scheduling timeout.
pedrofrxncx
enabled auto-merge (squash)
July 24, 2026 14:52
decocms Bot
pushed a commit
that referenced
this pull request
Jul 24, 2026
PR: #5183 fix(sandbox): force-reprovision a provisioning session orphaned by a crashed start Bump type: patch - decocms (apps/api/package.json): 4.122.13 -> 4.122.14 Deploy-Scope: server
pedrofrxncx
pushed a commit
that referenced
this pull request
Jul 27, 2026
#5240) Reverts #5112 (share agent sandboxes by branch), #5116 (always share agent sandboxes) and #5132 (use shared staging branch), restoring the pre-#5112 behaviour: hosted agent sandboxes are per-user again, and a new GitHub-backed thread gets a generated branch instead of `staging`. Reconstructed against current paths rather than reverted: #5120 split apps/mesh into apps/api + apps/web and hoisted branch-name.ts, runtime-defaults.ts and thread/schema.ts into packages/shared, and #5174 deleted vm-events.ts outright, so none of the three commits reverse-apply. Core change: SandboxId goes back to a flat `{ userId, projectRef }` and sandboxIdKey back to `userId:projectRef`, so every hosted claim handle changes. Live shared claims must be drained BEFORE this deploys. Also reverted, because they only exist inside the shared model: - #5225 withoutLegacyAgentSandboxEntries — left in place it would strip the sandboxMap entries the reverted server writes back, leaving shouldAutoStart permanently true (silent, compiles fine) - #5219 resolveSharedThreadVm and the shared-scope branch - #5183 stale-provisioning reset (lived inside the deleted storage) Deliberately preserved: - migrations 137/138 and their index.ts entries — Kysely's #ensureNoMissingMigrations runs unconditionally, so deleting an applied migration crash-loops every pod. The tables are dropped by a follow-up forward migration after the drain, not by this commit. - #5114/#5118 withClaimGitLock, #5126 org-context guard, #5143 analytics gate, #5171 owner-keyed thread graft, #5195/#5207/#5210/#5221 client auto-retry, the daemon status-code train and the settings-validation train - #5112's org-fs least-privilege narrowing and the start dedup key fix - #5116's squashed queue-tray pluralization (en + pt-br) - the remote-branch-name validations #5132 deleted, since this restores the dynamic origin/HEAD lookup that made them load-bearing Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Pedro França <pedrofrxncx@deco.cx>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A shared-sandbox session for an old branch reserves forever and never reboots (reported: "Sandbox in Ferragens Floresta org didn't load, reserving forever" — the branch was very old).
Root cause: a per-branch session row stuck in
status='provisioning'is never force-reprovisioned.beginStart(apps/api/src/storage/agent-sandbox-sessions.ts) only resets a row — bump generation, clear the stale handle, force a fresh provision — whendesired_state='stopped'orstatus IN ('missing','failed'). Aprovisioningrow withdesired_state='running'hits everyelsebranch: it re-joins its own generation and returns the existing (stale/null)sandbox_handle, so no reboot happens. Nothing times outprovisioningeither — there is no sweeper — so it hangs indefinitely.An old branch makes this likely because its session row lingers un-GC'd long enough to have been left half-provisioned by an earlier start that crashed between
beginStartandcompleteStart/failStart(e.g. the studio pod died mid-provision, orfailStartno-op'd because its generation fence was already superseded).Fix
beginStartnow also treats aprovisioningrow older than 5 minutes onupdated_atas stale and resets it — bumping the generation to fence out any zombie in-flight writes from the dead attempt, clearing the handle, and forcing a fresh provision on the next start. Self-healing; no sweeper/cron needed. 5m is safely longer than the ~180s real provision wait and matches the k8s scheduling timeout.Testing
Added a real-Postgres integration test (
agent-sandbox-sessions.integration.test.ts) asserting:completeStartis fenced out.bun run checkandbun run fmtpass. (Integration test runs in CI where the test Postgres + deps are available.)Notes / follow-ups (not in this PR)
finishInterruptedSharedTransition(healsstopping/reapingonly) to coverstopping/reaping/deletingwedges, nor relax thestatus==='ready'gate in the sandbox-events self-heal. This fix alone unblocks the reported symptom; those are defense-in-depth.🤖 Generated with Claude Code
Summary by cubic
Stops shared sandbox sessions from “reserving forever” by force-reprovisioning a
provisioningsession that’s stale for over 5 minutes. This self-heals crashed starts so old branches can boot again.beginStart, treatstatus='provisioning'rows withupdated_at < now() - 5mas stale. Bumpgeneration, clear handle/URLs, and restart provisioning to fence zombie writes.Written for commit 70bc3e7. Summary will update on new commits.