fix: fail over unhealthy Daytona sandbox hosts - #183
Merged
Conversation
Preserve the mounted workspace volume while replacing a stopped runtime after Daytona's explicit host-recovery rejection. Extend durable retries for other transient preparation failures.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
iamjr15
added a commit
that referenced
this pull request
Aug 8, 2026
## Why Daytona currently reports an unhealthy sandbox host with a structured HTTP 503 body but no provider error code. The sandbox lifecycle should not depend on provider message text. ## What changed - validate the Daytona error projection at the adapter boundary - normalize the specific host-recovery response to an internal `daytona_host_recovering` code - make lifecycle classification depend only on the internal code and HTTP status - document the normalized adapter-to-lifecycle contract This preserves the guarded same-volume runtime replacement behavior merged in #183 while keeping provider response details isolated to the Daytona adapter. ## Verification - `pnpm lint` - `pnpm typecheck` - `pnpm turbo build --force` - `pnpm deadcode` - `pnpm architecture:check` - `pnpm turbo skills:build` - captured the live Daytona 503 response shape for the affected sandbox without exposing credentials Production behavior will be exercised after merge and exact-SHA Cloudflare deployment.
iamjr15
added a commit
that referenced
this pull request
Aug 8, 2026
## Summary - Treat provider `length` finish reasons as nonterminal model turns. - Checkpoint the partial response and continue durably with an internal instruction to split interrupted operations into smaller complete tool calls. - Validate the SDK finish-reason contract at the agent boundary and accept only `stop` as tool-free semantic completion. - Fail blocked, provider-error, and invalid terminal reasons explicitly instead of publishing incomplete work as successful. ## Root cause The production Pomodoro retry recovered its Daytona runtime correctly, but Claude reached its per-response output limit while forming the next operation. The Workflow used only the absence of parsed tool calls as its completion condition, so it finalized that truncated turn and exposed the sandbox readiness page. ## Architecture The Mastra adapter validates the six AI SDK finish reasons. Cloudflare Workflow owns the policy: tool calls execute durably; `length` appends a non-visible continuation message and starts another checkpointed model turn; only `stop` completes a tool-free run. This keeps semantic completion in charge without adding a step, token, duration, or cost ceiling. ## Decisions | Decision | Choice | Reason | |---|---|---| | Handle truncation in Workflow | Durable continuation | It preserves replay safety and applies to every provider and run type. | | Preserve partial response history | Append an internal user continuation message | The next turn sees the exact work already attempted and avoids repeating completed actions. | | Keep provider output settings unchanged | No fixed output-token override | A larger fixed cap only postpones truncation and does not establish correct completion semantics. | | Reject non-stop terminal reasons | Explicit upstream failure | Content filtering and provider errors are not successful outcomes. | ## Production evidence - Run `019fe1b3-af54-7a69-bd1d-0cd940842e41` completed after model turn 5 returned `finishReason: "length"` with no tool call. - The replacement Daytona sandbox was healthy and mounted the correct durable volume, isolating this from the sandbox recovery issue fixed in #183 and #184. ## Verification - `pnpm lint` - `pnpm typecheck` - `pnpm turbo build --force` - `pnpm deadcode` - `pnpm architecture:check` - `pnpm turbo skills:build` - production flow will be repeated after merge and exact-SHA Cloudflare deployment ## How to review 1. Review the finish-reason schema in `packages/agent-core/src/mastra/durable-agent-step.ts`. 2. Review the durable loop policy in `apps/agent-worker/src/durable-objects/agent-run-workflow.ts`. 3. Confirm the shared schema is enforced by `agent-run-workflow-runtime.ts` and the README contract matches the implementation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Architecture
A run still resolves the single canonical user sandbox first. If Daytona reports that the sandbox's assigned host is recovering, the existing runtime-replacement fence verifies the active-run lease and canonical volume identity, removes the disposable container, and creates one replacement attached to the same isolated volume subpath. Cloudflare Workflow retries remain the durable fallback around preparation.
Decisions Made
Edge Cases Handled
Verification
pnpm lintpnpm typecheckpnpm turbo build --forcepnpm deadcodepnpm architecture:checkpnpm turbo skills:build