When a runner replica dies without cleaning up (a crash, an out-of-memory kill, or docker kill), a Daytona session cannot be continued for the next ~120 seconds. The resume fails with an error that points at the wrong thing:
Agent run failed: the in-sandbox tool MCP shim could not be delivered to the Daytona sandbox,
so the run's gateway/callback tools cannot be advertised to the harness;
run build:extension and retry, or run on the local sandbox / the Pi harness.
Nothing is wrong with the shim, and build:extension does not help. After about 120 seconds the exact same request succeeds. So the session is not lost, but for two minutes the user sees a hard failure with a misleading message.
Reproduce
- Start a Daytona-sandbox session with a tool attached, and let it run one turn. Any harness, any tool permission.
docker kill <runner-container> then docker start <runner-container>, and wait for it to report healthy. Use kill, not restart: on SIGTERM the runner deliberately deletes its sandboxes, so there is nothing left to resume and you measure something else.- Continue the same session (same
session_id).
- Observed: the run fails with the shim error above.
- Expected: the new replica adopts the sandbox and continues, or fails with a message that says the session is held by another replica.
- Wait 140 seconds and send the identical request again. It now succeeds.
Verified on the EE dev stack with the Codex harness on a real Daytona sandbox, on a parked approval resume and on a plain tool run with no approval involved. The runner log shows the new replica reconnecting to the sandbox successfully first:
[sessions/alive] interrupted session=<sid> turn=<new-turn> -> aborting
[sandbox-agent] reconnected sandbox=daytona/<sandbox-id> session=<sid>
[sandbox-agent] [timing] stage=sandbox_start ms=613 sandbox=daytona/<sandbox-id> mode=reconnect
[sandbox-agent] tool MCP shim upload failed: This operation was aborted
The same "interrupted -> aborting" line appears on local-sandbox resumes too, where it is harmless because there is no shim to upload. Only Daytona has an early step the abort can kill.
Reproduces a second way with no crash at all: run two runner replicas, start a session on the first, then point AGENTA_RUNNER_INTERNAL_URL at the second and continue the session there. Same failure. That path is arguably working as designed, since session affinity is meant to route a session back to its owner, but it produces the same misleading error.
Candidate fix (for whoever picks this up)
The runner asks the API for a heartbeat and reads a single boolean, is_current_turn. That one flag covers two different situations, and the runner treats both as a cancellation:
services/runner/src/sessions/alive.ts — sendHeartbeat sets interrupted = body.is_current_turn === false, and handleBeat calls onInterrupted(), which aborts the run signal. On Daytona that signal is what uploadToolMcpAssets is waiting on, so the fresh turn kills itself.api/oss/src/core/sessions/streams/service.py — the heartbeat returns is_current_turn=False from two different places. One means "a cancel or kill took your turn's lock", which is a genuine interruption. The other, the early return at the owner != request.replica_id check, only means "a different replica owns this session". claim_owner (api/oss/src/dbs/redis/sessions/locks.py) never steals from an owner that still looks live, and the owner key lives for OWNER_TTL_SECONDS = 120, so a dead replica keeps the claim until it lapses.
Suggested change, smallest first:
- In
alive.ts, only treat the flag as an interruption when the API also reports this replica as the owner. The response already carries replica_id. When the owner is a different replica, that is an ownership situation, not a cancellation, and the run must not abort itself.
- Decide what should happen instead for a remote sandbox. Retrying until the key lapses is the friendliest, since it self-heals inside two minutes. Failing with a clear "this session is held by another runner, retry shortly" is the cheapest. For a local sandbox the existing single-owner guard already refuses with a good message and should keep doing so.
- Optionally, let the API hand ownership over when the previous owner's key has lapsed, so the handover does not depend on the runner retrying.
Worth checking while in there: whether a graceful stop should really delete a sandbox that is parked waiting on a human approval. Today server.ts drains the pool through pool.destroyAll on SIGTERM, so a normal deploy discards any pending approval card.
Please add a regression test for the ownership case. services/runner/tests/unit/session-ownership.test.ts already covers the guard and is the natural home.
When a runner replica dies without cleaning up (a crash, an out-of-memory kill, or
docker kill), a Daytona session cannot be continued for the next ~120 seconds. The resume fails with an error that points at the wrong thing:Nothing is wrong with the shim, and
build:extensiondoes not help. After about 120 seconds the exact same request succeeds. So the session is not lost, but for two minutes the user sees a hard failure with a misleading message.Reproduce
docker kill <runner-container>thendocker start <runner-container>, and wait for it to report healthy. Usekill, notrestart: on SIGTERM the runner deliberately deletes its sandboxes, so there is nothing left to resume and you measure something else.session_id).Verified on the EE dev stack with the Codex harness on a real Daytona sandbox, on a parked approval resume and on a plain tool run with no approval involved. The runner log shows the new replica reconnecting to the sandbox successfully first:
The same "interrupted -> aborting" line appears on local-sandbox resumes too, where it is harmless because there is no shim to upload. Only Daytona has an early step the abort can kill.
Reproduces a second way with no crash at all: run two runner replicas, start a session on the first, then point
AGENTA_RUNNER_INTERNAL_URLat the second and continue the session there. Same failure. That path is arguably working as designed, since session affinity is meant to route a session back to its owner, but it produces the same misleading error.Candidate fix (for whoever picks this up)
The runner asks the API for a heartbeat and reads a single boolean,
is_current_turn. That one flag covers two different situations, and the runner treats both as a cancellation:services/runner/src/sessions/alive.ts—sendHeartbeatsetsinterrupted = body.is_current_turn === false, andhandleBeatcallsonInterrupted(), which aborts the run signal. On Daytona that signal is whatuploadToolMcpAssetsis waiting on, so the fresh turn kills itself.api/oss/src/core/sessions/streams/service.py— the heartbeat returnsis_current_turn=Falsefrom two different places. One means "a cancel or kill took your turn's lock", which is a genuine interruption. The other, the early return at theowner != request.replica_idcheck, only means "a different replica owns this session".claim_owner(api/oss/src/dbs/redis/sessions/locks.py) never steals from an owner that still looks live, and the owner key lives forOWNER_TTL_SECONDS = 120, so a dead replica keeps the claim until it lapses.Suggested change, smallest first:
alive.ts, only treat the flag as an interruption when the API also reports this replica as the owner. The response already carriesreplica_id. When the owner is a different replica, that is an ownership situation, not a cancellation, and the run must not abort itself.Worth checking while in there: whether a graceful stop should really delete a sandbox that is parked waiting on a human approval. Today
server.tsdrains the pool throughpool.destroyAllon SIGTERM, so a normal deploy discards any pending approval card.Please add a regression test for the ownership case.
services/runner/tests/unit/session-ownership.test.tsalready covers the guard and is the natural home.