Skip to content

fix(sandbox): auto-retry transient claim failures before showing errored - #5210

Merged
pedrofrxncx merged 1 commit into
mainfrom
fix/sandbox-claim-auto-retry
Jul 24, 2026
Merged

fix(sandbox): auto-retry transient claim failures before showing errored#5210
pedrofrxncx merged 1 commit into
mainfrom
fix/sandbox-claim-auto-retry

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #5207 (merged). That PR stopped the infinite spinner by surfacing a terminal claim failed phase as the errored card + Retry. This PR adds bounded auto-recovery so transient infra failures self-heal without a manual click.

Why

#5207 requires a manual Retry for every terminal claim failure, including ones that are just transient infra hiccups (a node had no capacity for a moment, a control-plane blip). Those should self-heal.

What

On a terminal claim failed phase, auto-reprovision in place for the reasons a fresh claim can plausibly resolve, bounded to MAX_CLAIM_AUTO_RETRIES (2 → 3 total attempts), then fall through to the errored card.

ClaimFailureReason Auto-retry? Rationale
scheduling-timeout no capacity — Karpenter may have added a node
reconciler-error transient control-plane hiccup
claim-never-created the claim write was lost — re-issue it
unknown (synthetic "watcher ended") the watch died, not necessarily the boot
image-pull-backoff bad/missing image — fails identically
crash-loop-backoff the container's own code crashes on start

While auto-retry budget remains, deriveStartError returns null (claimRetryExhausted=false) so the booting overlay holds instead of flashing errored between attempts.

Correctness details

  • Fires once per failed episode: a claimFailureHandledRef guard is set when a retry fires and re-armed only when the phase leaves failed, so the persistent failed phase can't re-fire every render once the mutation settles.
  • Bounded budget per boot via claimRetryCountRef; a user-driven retry() resets both refs for a fresh budget.
  • Same gates as auto-start / self-heal: userStopped, autoStartBlocked (others' thread), and isPending all suppress it.
  • Reuses useSandboxStart's module-level inflight dedup, so even a double-fire coalesces to one upstream SANDBOX_START.

Tests

Pure helpers isRetryableClaimFailure + shouldAutoRetryClaim unit-tested (retryable vs non-retryable reasons; budget exhausted; already-handled; pending / stopped / gated). Existing deriveStartError tests updated for the new claimRetryExhausted arg + a new keep-booting case. bun test 55 pass · tsc clean · lint 0 errors · fmt clean.


Summary by cubic

Automatically retries transient claim failures during sandbox boot before showing the errored card. This self-heals common infra hiccups and reduces user clicks and flicker.

  • Bug Fixes
    • Auto-retries in place for retryable failures: scheduling timeout, reconciler error, claim never created, and watcher ended; capped at 2 auto-retries (3 total attempts).
    • Non-retryable failures (image pull backoff, crash loop backoff) surface the error immediately.
    • Keeps the booting overlay while retry budget remains; updated start error derivation to respect retry exhaustion.
    • Fires once per failed episode, honors userStopped and others-thread gates, dedups inflight starts, and resets budget on user Retry; added unit tests for retry logic.

Written for commit 418e4fe. Summary will update on new commits.

Review in cubic

Builds on the terminal-claim-failure surfacing: instead of immediately showing
the errored card on every terminal claim failure, auto-reprovision in place for
the failures that are plausibly transient infra hiccups — scheduling-timeout
(capacity), reconciler-error, claim-never-created, and the synthetic
'watcher ended' (unknown). Bounded to MAX_CLAIM_AUTO_RETRIES (2) per boot, then
falls through to the errored card + Retry.

image-pull-backoff and crash-loop-backoff are excluded: a fresh claim fails
identically, so those surface immediately.

The retry fires once per failed episode (re-armed when the phase leaves
'failed'), reuses useSandboxStart's inflight dedup, and honors the same
userStopped / others-thread gates as auto-start and self-heal. A user-driven
retry() resets the budget. While budget remains, deriveStartError keeps the
booting overlay (claimRetryExhausted=false) instead of flashing errored.

Pure helpers isRetryableClaimFailure + shouldAutoRetryClaim are unit-tested.
@pedrofrxncx
pedrofrxncx enabled auto-merge (squash) July 24, 2026 17:38
@pedrofrxncx
pedrofrxncx merged commit 9e60882 into main Jul 24, 2026
14 checks passed
@pedrofrxncx
pedrofrxncx deleted the fix/sandbox-claim-auto-retry branch July 24, 2026 17:46
decocms Bot pushed a commit that referenced this pull request Jul 24, 2026
PR: #5210 fix(sandbox): auto-retry transient claim failures before showing errored
Bump type: patch

- decocms (apps/api/package.json): 4.123.8 -> 4.123.9

Deploy-Scope: web
pedrofrxncx added a commit that referenced this pull request Jul 24, 2026
…5221)

* fix(sandbox): scope the claim auto-retry budget to the active branch

SandboxLifecycleProvider lives above Chat.ActiveTaskProvider and is not
remounted on a task switch, but its bounded claim-auto-retry budget (added in
#5210) was tracked in plain refs shared across the whole provider lifetime.
Switching to a different task/branch after one branch's boot exhausted its
retry budget carried that exhausted count into the new branch's fresh boot,
so a transient claim failure there skipped auto-retry and surfaced the
errored card immediately instead of retrying.

Replaced the two refs with a small piece of state keyed by branch, reconciled
via the new pure `reconcileClaimRetryEpisode` (reset to a fresh budget
whenever the branch differs from the tracked one), and added unit tests for
it.

* fix

---------

Co-authored-by: Pedro França <pedrofrxncx@deco.cx>
pedrofrxncx pushed a commit that referenced this pull request Jul 27, 2026
#5240)

Reverts #5112 (share agent sandboxes by branch), #5116 (always share
agent sandboxes) and #5132 (use shared staging branch), restoring the
pre-#5112 behaviour: hosted agent sandboxes are per-user again, and a
new GitHub-backed thread gets a generated branch instead of `staging`.

Reconstructed against current paths rather than reverted: #5120 split
apps/mesh into apps/api + apps/web and hoisted branch-name.ts,
runtime-defaults.ts and thread/schema.ts into packages/shared, and #5174
deleted vm-events.ts outright, so none of the three commits reverse-apply.

Core change: SandboxId goes back to a flat `{ userId, projectRef }` and
sandboxIdKey back to `userId:projectRef`, so every hosted claim handle
changes. Live shared claims must be drained BEFORE this deploys.

Also reverted, because they only exist inside the shared model:
- #5225 withoutLegacyAgentSandboxEntries — left in place it would strip
  the sandboxMap entries the reverted server writes back, leaving
  shouldAutoStart permanently true (silent, compiles fine)
- #5219 resolveSharedThreadVm and the shared-scope branch
- #5183 stale-provisioning reset (lived inside the deleted storage)

Deliberately preserved:
- migrations 137/138 and their index.ts entries — Kysely's
  #ensureNoMissingMigrations runs unconditionally, so deleting an applied
  migration crash-loops every pod. The tables are dropped by a follow-up
  forward migration after the drain, not by this commit.
- #5114/#5118 withClaimGitLock, #5126 org-context guard, #5143 analytics
  gate, #5171 owner-keyed thread graft, #5195/#5207/#5210/#5221 client
  auto-retry, the daemon status-code train and the settings-validation train
- #5112's org-fs least-privilege narrowing and the start dedup key fix
- #5116's squashed queue-tray pluralization (en + pt-br)
- the remote-branch-name validations #5132 deleted, since this restores
  the dynamic origin/HEAD lookup that made them load-bearing

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Pedro França <pedrofrxncx@deco.cx>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant