fix(sandbox): make the SIGTERM git-sync survive to completion - #4561
Merged
Conversation
The daemon's shutdown handler pushes the working tree to git — the only durable copy of the user's work, since the pod's /app is ephemeral emptyDir. Three ways it could silently drop that work under SIGTERM (rollout, spot reclaim, idle eviction): - Unmount ran BEFORE the publish, so a hanging rclone stop could eat the push's slice of the grace period. Publish first; unmount is best-effort and already backstopped by the sync exit handler + next-boot reclaim. - The push didn't pass --no-verify, so a repo's native pre-push hook could fail or hang it (the commit already skips hooks; the push now matches). - No terminationGracePeriodSeconds was set, so K8s used the 30s default — too tight to clear commit + 30s-bounded push + unmount, ending in SIGKILL mid-sync. Set it to 90s (chart), inside AWS spot's 120s reclaim notice. Adds an e2e test that publish pushes past a failing pre-push hook (fails without --no-verify, verified).
pedrofrxncx
enabled auto-merge (squash)
July 14, 2026 20:42
decocms Bot
pushed a commit
that referenced
this pull request
Jul 14, 2026
PR: #4561 fix(sandbox): make the SIGTERM git-sync survive to completion Bump type: patch - @decocms/sandbox (packages/sandbox/package.json): 1.15.17 -> 1.15.18 Deploy-Scope: both
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On SIGTERM (rollout, spot reclaim, idle eviction) the sandbox daemon commits + pushes the working tree to git. That pushed branch is the only durable copy of the user's work — the pod's
/appis ephemeralemptyDir. Three ways the sync could silently drop hours of work:shutdown()awaitedmountManager.stop()first, so a hanging rclone unmount could eat the push's slice of the grace period. → Publish first; unmount moved after (best-effort, already backstopped by the syncexithandler + next-boot reclaim).--no-verify, but the push didn't — a repo's nativepre-pushhook could fail or hang it. →--no-verifyon the push too.terminationGracePeriodSeconds. The pod used the K8s default (30s), too tight to clear commit + the 30s-bounded push (git-sync.tsDEFAULT_GIT_TIMEOUT_MS) + unmount → SIGKILL mid-sync. → Set to90s(chart value + template), comfortably inside AWS spot's 120s reclaim notice.Testing
POST /git/publish pushes past a failing pre-push hook— installs apre-pushthatexit 1s and asserts the push still lands. Verified discriminating: fails (500) without--no-verify, passes with it.daemon.git.e2e.test.tsgreen (17/17), including the existingSIGTERM triggers a graceful publishtest after the reorder.helm templaterendersterminationGracePeriodSeconds: 90on theSandboxTemplate.tscclean,bun run lint0 errors, formatted.Notes / follow-up
This hardens the graceful path. It does not cover a hard kill with no SIGTERM (OOM, node loss, non-graceful spot) — that still loses since the last push. A periodic/debounced background push during the session would close that gap; out of scope here.
🤖 Generated with Claude Code
Summary by cubic
Ensures the sandbox completes its SIGTERM git sync so user work is pushed before the pod is killed. We publish before unmount, skip
pre-pushhooks, and extend the pod’s shutdown window to 90s.git push --no-verifyto bypass repopre-pushhooks that could fail or hang.terminationGracePeriodSeconds: 90in Helm to cover commit + 30s-bounded push within AWS spot’s 120s notice.pre-pushhook; existing SIGTERM publish test remains green.Written for commit e39e6c2. Summary will update on new commits.