Skip to content

fix(sandbox): force-reprovision a provisioning session orphaned by a crashed start - #5183

Merged
pedrofrxncx merged 1 commit into
mainfrom
fix/sandbox-stale-provisioning-reprovision
Jul 24, 2026
Merged

fix(sandbox): force-reprovision a provisioning session orphaned by a crashed start#5183
pedrofrxncx merged 1 commit into
mainfrom
fix/sandbox-stale-provisioning-reprovision

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Problem

A shared-sandbox session for an old branch reserves forever and never reboots (reported: "Sandbox in Ferragens Floresta org didn't load, reserving forever" — the branch was very old).

Root cause: a per-branch session row stuck in status='provisioning' is never force-reprovisioned. beginStart (apps/api/src/storage/agent-sandbox-sessions.ts) only resets a row — bump generation, clear the stale handle, force a fresh provision — when desired_state='stopped' or status IN ('missing','failed'). A provisioning row with desired_state='running' hits every else branch: it re-joins its own generation and returns the existing (stale/null) sandbox_handle, so no reboot happens. Nothing times out provisioning either — there is no sweeper — so it hangs indefinitely.

An old branch makes this likely because its session row lingers un-GC'd long enough to have been left half-provisioned by an earlier start that crashed between beginStart and completeStart/failStart (e.g. the studio pod died mid-provision, or failStart no-op'd because its generation fence was already superseded).

Fix

beginStart now also treats a provisioning row older than 5 minutes on updated_at as stale and resets it — bumping the generation to fence out any zombie in-flight writes from the dead attempt, clearing the handle, and forcing a fresh provision on the next start. Self-healing; no sweeper/cron needed. 5m is safely longer than the ~180s real provision wait and matches the k8s scheduling timeout.

Testing

Added a real-Postgres integration test (agent-sandbox-sessions.integration.test.ts) asserting:

  • a recent provisioning row coalesces (is not reset) — a genuine in-flight start still wins;
  • a backdated (>5m) provisioning row force-reprovisions with a new generation and a null handle;
  • the old generation's completeStart is fenced out.

bun run check and bun run fmt pass. (Integration test runs in CI where the test Postgres + deps are available.)

Notes / follow-ups (not in this PR)

  • Did not extend finishInterruptedSharedTransition (heals stopping/reaping only) to cover stopping/reaping/deleting wedges, nor relax the status==='ready' gate in the sandbox-events self-heal. This fix alone unblocks the reported symptom; those are defense-in-depth.
  • Did not add a migration to GC long-dead sessions — separate concern.

🤖 Generated with Claude Code


Summary by cubic

Stops shared sandbox sessions from “reserving forever” by force-reprovisioning a provisioning session that’s stale for over 5 minutes. This self-heals crashed starts so old branches can boot again.

  • Bug Fixes
    • In beginStart, treat status='provisioning' rows with updated_at < now() - 5m as stale. Bump generation, clear handle/URLs, and restart provisioning to fence zombie writes.
    • Added Postgres integration test to confirm recent provisions coalesce, stale rows reboot, and old-generation completions are ignored.

Written for commit 70bc3e7. Summary will update on new commits.

Review in cubic

…crashed start

A per-branch shared-sandbox session stuck in `status='provisioning'` never
reboots: `beginStart` only resets the row (bump generation, clear the stale
handle, force a fresh provision) when `desired_state='stopped'` or
`status IN ('missing','failed')`. A `provisioning` row with
`desired_state='running'` re-joins its own generation and returns a stale/null
handle, so nothing ever reprovisions it. Nothing times out `provisioning`
either, so it "reserves forever" — the symptom seen when an old branch's
long-lived session was left half-provisioned by a start that crashed between
`beginStart` and `completeStart`/`failStart` (e.g. the studio pod died mid-provision).

`beginStart` now also treats a `provisioning` row older than 5 minutes on
`updated_at` as stale and resets it, bumping the generation to fence out any
zombie in-flight writes from the dead attempt. 5m is safely longer than the
~180s real provision wait and matches the k8s scheduling timeout.
@pedrofrxncx
pedrofrxncx enabled auto-merge (squash) July 24, 2026 14:52
@pedrofrxncx
pedrofrxncx merged commit 507a4d1 into main Jul 24, 2026
15 checks passed
@pedrofrxncx
pedrofrxncx deleted the fix/sandbox-stale-provisioning-reprovision branch July 24, 2026 14:58
decocms Bot pushed a commit that referenced this pull request Jul 24, 2026
PR: #5183 fix(sandbox): force-reprovision a provisioning session orphaned by a crashed start
Bump type: patch

- decocms (apps/api/package.json): 4.122.13 -> 4.122.14

Deploy-Scope: server
pedrofrxncx pushed a commit that referenced this pull request Jul 27, 2026
#5240)

Reverts #5112 (share agent sandboxes by branch), #5116 (always share
agent sandboxes) and #5132 (use shared staging branch), restoring the
pre-#5112 behaviour: hosted agent sandboxes are per-user again, and a
new GitHub-backed thread gets a generated branch instead of `staging`.

Reconstructed against current paths rather than reverted: #5120 split
apps/mesh into apps/api + apps/web and hoisted branch-name.ts,
runtime-defaults.ts and thread/schema.ts into packages/shared, and #5174
deleted vm-events.ts outright, so none of the three commits reverse-apply.

Core change: SandboxId goes back to a flat `{ userId, projectRef }` and
sandboxIdKey back to `userId:projectRef`, so every hosted claim handle
changes. Live shared claims must be drained BEFORE this deploys.

Also reverted, because they only exist inside the shared model:
- #5225 withoutLegacyAgentSandboxEntries — left in place it would strip
  the sandboxMap entries the reverted server writes back, leaving
  shouldAutoStart permanently true (silent, compiles fine)
- #5219 resolveSharedThreadVm and the shared-scope branch
- #5183 stale-provisioning reset (lived inside the deleted storage)

Deliberately preserved:
- migrations 137/138 and their index.ts entries — Kysely's
  #ensureNoMissingMigrations runs unconditionally, so deleting an applied
  migration crash-loops every pod. The tables are dropped by a follow-up
  forward migration after the drain, not by this commit.
- #5114/#5118 withClaimGitLock, #5126 org-context guard, #5143 analytics
  gate, #5171 owner-keyed thread graft, #5195/#5207/#5210/#5221 client
  auto-retry, the daemon status-code train and the settings-validation train
- #5112's org-fs least-privilege narrowing and the start dedup key fix
- #5116's squashed queue-tray pluralization (en + pt-br)
- the remote-branch-name validations #5132 deleted, since this restores
  the dynamic origin/HEAD lookup that made them load-bearing

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Pedro França <pedrofrxncx@deco.cx>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant