Skip to content

fix(sandbox): make the SIGTERM git-sync survive to completion - #4561

Merged
pedrofrxncx merged 1 commit into
mainfrom
fix/sandbox-sigterm-git-sync
Jul 14, 2026
Merged

fix(sandbox): make the SIGTERM git-sync survive to completion#4561
pedrofrxncx merged 1 commit into
mainfrom
fix/sandbox-sigterm-git-sync

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

On SIGTERM (rollout, spot reclaim, idle eviction) the sandbox daemon commits + pushes the working tree to git. That pushed branch is the only durable copy of the user's work — the pod's /app is ephemeral emptyDir. Three ways the sync could silently drop hours of work:

  1. Unmount ran before publish. shutdown() awaited mountManager.stop() first, so a hanging rclone unmount could eat the push's slice of the grace period. → Publish first; unmount moved after (best-effort, already backstopped by the sync exit handler + next-boot reclaim).
  2. Push didn't skip hooks. The commit already runs --no-verify, but the push didn't — a repo's native pre-push hook could fail or hang it. → --no-verify on the push too.
  3. No terminationGracePeriodSeconds. The pod used the K8s default (30s), too tight to clear commit + the 30s-bounded push (git-sync.ts DEFAULT_GIT_TIMEOUT_MS) + unmount → SIGKILL mid-sync. → Set to 90s (chart value + template), comfortably inside AWS spot's 120s reclaim notice.

Testing

  • New e2e: POST /git/publish pushes past a failing pre-push hook — installs a pre-push that exit 1s and asserts the push still lands. Verified discriminating: fails (500) without --no-verify, passes with it.
  • Full daemon.git.e2e.test.ts green (17/17), including the existing SIGTERM triggers a graceful publish test after the reorder.
  • helm template renders terminationGracePeriodSeconds: 90 on the SandboxTemplate.
  • tsc clean, bun run lint 0 errors, formatted.

Notes / follow-up

This hardens the graceful path. It does not cover a hard kill with no SIGTERM (OOM, node loss, non-graceful spot) — that still loses since the last push. A periodic/debounced background push during the session would close that gap; out of scope here.

🤖 Generated with Claude Code


Summary by cubic

Ensures the sandbox completes its SIGTERM git sync so user work is pushed before the pod is killed. We publish before unmount, skip pre-push hooks, and extend the pod’s shutdown window to 90s.

  • Bug Fixes
    • Run publish before unmount on shutdown; unmount is best-effort to avoid blocking the push.
    • Use git push --no-verify to bypass repo pre-push hooks that could fail or hang.
    • Set terminationGracePeriodSeconds: 90 in Helm to cover commit + 30s-bounded push within AWS spot’s 120s notice.
    • Add e2e test that confirms publish succeeds past a failing pre-push hook; existing SIGTERM publish test remains green.

Written for commit e39e6c2. Summary will update on new commits.

Review in cubic

The daemon's shutdown handler pushes the working tree to git — the only
durable copy of the user's work, since the pod's /app is ephemeral emptyDir.
Three ways it could silently drop that work under SIGTERM (rollout, spot
reclaim, idle eviction):

- Unmount ran BEFORE the publish, so a hanging rclone stop could eat the
  push's slice of the grace period. Publish first; unmount is best-effort and
  already backstopped by the sync exit handler + next-boot reclaim.
- The push didn't pass --no-verify, so a repo's native pre-push hook could
  fail or hang it (the commit already skips hooks; the push now matches).
- No terminationGracePeriodSeconds was set, so K8s used the 30s default — too
  tight to clear commit + 30s-bounded push + unmount, ending in SIGKILL
  mid-sync. Set it to 90s (chart), inside AWS spot's 120s reclaim notice.

Adds an e2e test that publish pushes past a failing pre-push hook (fails
without --no-verify, verified).
@pedrofrxncx
pedrofrxncx enabled auto-merge (squash) July 14, 2026 20:42
@pedrofrxncx
pedrofrxncx merged commit 30ac696 into main Jul 14, 2026
18 checks passed
@pedrofrxncx
pedrofrxncx deleted the fix/sandbox-sigterm-git-sync branch July 14, 2026 20:47
decocms Bot pushed a commit that referenced this pull request Jul 14, 2026
PR: #4561 fix(sandbox): make the SIGTERM git-sync survive to completion
Bump type: patch

- @decocms/sandbox (packages/sandbox/package.json): 1.15.17 -> 1.15.18

Deploy-Scope: both
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant