Skip to content

fix(sandbox): prevent Daytona heartbeat sleep leak#910

Merged
bingran-you merged 2 commits into
mainfrom
bry/fix-daytona-heartbeat-race
Jul 11, 2026
Merged

fix(sandbox): prevent Daytona heartbeat sleep leak#910
bingran-you merged 2 commits into
mainfrom
bry/fix-daytona-heartbeat-race

Conversation

@bingran-you

Copy link
Copy Markdown
Collaborator

Summary

  • replace signal-based heartbeat teardown with a stop-file and bounded polling
  • prevent the race that leaves sleep 60 orphaned with captured pipes open
  • pin the regression guard to the heartbeat path introduced by PR Fix Toolathlon Daytona blocker retries #894

Root cause

The fast-command path could send TERM after the heartbeat shell spawned sleep 60 but before it stored the child PID. The trap then exited without killing the sleep process. That orphan retained stdout/stderr, so subprocess.run(..., timeout=2) timed out even though the parent shell had already returned 0.

Validation

  • uv run pytest -q tests/test_daytona_dind_compose_up.py — 12 passed
  • 100 fast-command iterations on macOS — passed
  • 100 fast-command iterations in Debian/Linux — passed
  • forced 2-second heartbeat emission — passed
  • uv run ruff format --check src tests tools
  • uv run ruff check src tests tools
  • uv run ty check
  • uv lock --check
  • uv run python -m pytest tests/ -q — 4835 passed, 8 skipped, 7 deselected
  • pip-audit — no known vulnerabilities, one documented ignore

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 10, 2026 23:29 — with GitHub Actions Inactive
@bingran-you bingran-you added the status:ready Triaged, unassigned, available to claim. label Jul 10, 2026
@bingran-you
bingran-you requested a review from xdotli July 10, 2026 23:30

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a560034f73

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

' if kill -0 "$bf_pid" 2>/dev/null; then',
" bf_heartbeat_elapsed=0",
' while [ ! -e "$bf_stop" ] && kill -0 "$bf_pid" 2>/dev/null; do',
" sleep 1",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop waiting a full poll tick after the child exits

On Daytona DinD execs with timeout_sec >= 600, this sleep 1 keeps the wrapper process alive until the next poll even after wait "$bf_pid" has observed the real command exit. _compose_exec passes the same timeout_sec down to _sandbox_exec, which wraps the whole compose command in GNU timeout {timeout_sec}; timeout --help says it kills a command that is still running after the duration and exits 124. As a result, a command that succeeds near the deadline, e.g. at 599.5s for a 600s verifier/agent timeout, can be killed before cat "$bf_out" and exit "$bf_rc", turning a successful run into a timeout and dropping its output.

Useful? React with 👍 / 👎.

@bingran-you
bingran-you requested review from Yiminnn and kywch July 11, 2026 01:42
@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 11, 2026 01:43 — with GitHub Actions Inactive
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@kywch @Yiminnn @xdotli — this is a narrow release-blocking process-cleanup fix. All required CI and the real rollout smoke are green; a human approval is the remaining merge gate.

@bingran-you
bingran-you merged commit b611030 into main Jul 11, 2026
9 checks passed
@bingran-you
bingran-you deleted the bry/fix-daytona-heartbeat-race branch July 11, 2026 04:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:ready Triaged, unassigned, available to claim.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant