Skip to content

Add sandbox retry logic - #196

Open
mendral-app[bot] wants to merge 1 commit into
mainfrom
mendral/add-sandbox-retry-logic
Open

Add sandbox retry logic#196
mendral-app[bot] wants to merge 1 commit into
mainfrom
mendral/add-sandbox-retry-logic

Conversation

@mendral-app

@mendral-app mendral-app Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add exponential backoff retry to all sandbox HTTP operations when the API returns a retryable error (e.g. WORKLOAD_UNAVAILABLE), respecting 500ms→30s backoff and ~60s total budget
  • Eliminates user-facing ResponseError during sandbox cold starts by automatically retrying instead of failing immediately

Details

The sandbox API returns { "error": { "code": "WORKLOAD_UNAVAILABLE", "retryable": true } } when a sandbox is scaling from zero. Previously, all sandbox operations raised ResponseError immediately on any non-2xx response without checking the retryable field.

This PR adds a _request_with_retry method to both SandboxAction (async) and SyncSandboxAction (sync) that:

  1. Makes the HTTP request
  2. Checks if the error response body contains retryable: true
  3. If retryable, logs an info message and sleeps with exponential backoff
  4. Retries until success or the 60s budget is exhausted

All sandbox data-plane operations (process, filesystem, multipart uploads) now use this method.

Resolves: https://app.mendral.com/insights/01KXVWJ0B6X2PV2MJZSNSFDVTY


Note

Created by Mendral. Tag @mendral-app with feedback or questions.

Add _request_with_retry method to SandboxAction (async) and
SyncSandboxAction (sync) that detects retryable API errors
(e.g. WORKLOAD_UNAVAILABLE during cold starts) and automatically
retries with exponential backoff: 500ms → 30s, ~60s total budget.

All sandbox HTTP operations (process exec/get/list/stop/kill/logs,
filesystem mkdir/write/read/rm/ls/find/grep, multipart uploads)
now use this retry wrapper, eliminating user-facing errors when
sandboxes are scaling from zero.
@mendral-app
mendral-app Bot requested a review from a team July 19, 2026 00:38
@mendral-app
mendral-app Bot marked this pull request as ready for review July 19, 2026 00:48
@SystemSculpt

Copy link
Copy Markdown
Member

This is relevant to the dominant SDK-PYTHON-C8 Codspeed WORKLOAD_UNAVAILABLE cluster, but it does not address the Mailai missing-path 404s. Holding merge: the PR changes retry behavior across async/sync filesystem and process mutations (including POST/PUT/DELETE) with no unit tests, and Core CI is red. Please add deterministic async/sync tests for retryable vs non-retryable bodies, budget exhaustion, cancellation, response closing, and mutating-operation replay safety; align/reuse the bounded retry policy in #172 where semantics match. Tracked in ENG-4037.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant