Resume Claude Code sessions after retryable interruptions

[merlerm]

Base status

#121 (runtime-metrics) was squash-merged into main as 48aa4ed. This branch has been rebased onto that updated main, and the PR targets main. The former stacked-PR dependency is satisfied.

Problems

Subscription usage caps

When Claude Code hits the subscription usage cap, the experiment runner sleeps until the reset and launches a new CLI/container process. Previously that process:

• started without the interrupted conversation context;
• received a fresh --max-budget-usd and --max-turns, making a timing-dependent retry a different experimental condition; and
• discarded the interrupted attempt's spend and retry signal (the underlying runtime metrics landed in Record generation-time runtime metrics to results.json #121).

The sandbox files survived, but the agent had to reconstruct its reasoning from disk.

Output-token overflows

Claude can also abort a run with Claude's response exceeded the 32768 output token maximum. Raising CLAUDE_CODE_MAX_OUTPUT_TOKENS is not a reliable fix: the model can consume the larger allowance and still exceed it. Previously this error could leave a partial approach.py and fail the entire experiment.

Context exhaustion and current reset messages

Auto-compaction normally protects Claude's context window, but runs have also stopped with Prompt is too long. Separately, the current subscription message includes minutes (You've hit your session limit · resets 1:30pm (UTC)), which the old hour-only regex did not match or schedule correctly.

What this PR changes

Resume Claude Code context

• Claude sessions are persisted from the first attempt so they are available if a retry becomes necessary.
• A retry uses --continue. Every generation has its own sandbox working directory and isolated session store, so “most recent session in this directory” identifies that generation.
• Docker and Apptainer persist /home/node/.claude/projects under the generation's sandbox directory, across ephemeral container recreation.
• The local backend places Claude state under sandbox-local .agent_home.

Preserve the experimental budget

• Interrupted attempts' reported cost and turns are subtracted from the original limits before resuming.
• A finite exhausted budget is never converted to 0, because both CLI limits interpret zero as unlimited.
• If a finite-budget interrupted attempt does not report its cost, the runner stops rather than silently granting an unknowable remaining budget.
• Retry counts and aborted spend remain visible through generation metrics.

Resuming is intentionally not claimed to be identical to an uninterrupted run: rebuilding context can consume uncached transcript input tokens. The retry metrics let analyses identify or exclude these runs.

Recover output-token overflows

• The exact observed overflow is detected in assistant text, result errors, or stderr.
• A partial output file is not evaluated after this interruption.
• The same isolated conversation resumes immediately, without the rate-limit sleep, using only the remaining dollar and turn budget.
• The continuation prompt asks Claude to avoid repeating reasoning, use tools directly, and finish concisely.
• At most two output-overflow resumptions are allowed, preventing an infinite loop for unlimited-budget configurations.
• gen_output_token_limit_hit and gen_output_token_retries make the condition visible in results.json; aborted tokens and cost use the existing interruption metrics.

Compact an oversized prompt before resuming

• Prompt is too long is detected in assistant text, result errors, or stderr, and partial output is not evaluated.
• The runner resumes the same isolated session with /compact and focused instructions to retain requirements, decisions, failed approaches, and current implementation state while dropping verbose tool output.
• Work resumes only after Claude emits a real compact_boundary event. A normal sandbox “output file not found” result from the local slash command is not treated as failure when that confirmation exists; conversely, an unconfirmed/failed compaction is never assumed to have worked.
• Compaction tokens, turns, and cost count against the original budget. The subsequent work process receives only what remains.
• At most two successful compact-and-resume cycles are allowed. gen_prompt_too_long_hit and gen_prompt_too_long_retries expose the condition in results.

Parse minute-bearing usage resets

• The current resets 1:30pm (UTC) format is recognized.
• Both hour and minute are retained, timezone interpretation is preserved, and the sleep targets five minutes after the stated reset (13:35 UTC in this example).
• Existing hour-only messages such as resets 3am retain their prior behavior.

Keep experiments out of the operator's Claude state

Session persistence must not add experiment conversations to the operator's normal ~/.claude history or session picker.

• Local runs set CLAUDE_CONFIG_DIR to the sandbox-local state directory.
• Host credentials are copied for the lifetime of one local CLI process, never symlinked, and removed immediately afterward.
• Docker and Apptainer receive a writable throwaway config containing only copied credentials when an OAuth environment token is unavailable.
• Host transcripts, history, memory, settings, file history, and job artifacts are neither copied nor live-mounted.
• Token refreshes and other CLI writes cannot modify the live host Claude directory.

Only the experiment's isolated transcript state survives between retry attempts.

Scope: OpenCode

Conversation resume is implemented only for Claude Code. OpenCode is currently used with local/open-source models in this project, where Claude subscription caps and these Claude-specific errors do not apply. OpenCode session persistence/resume semantics are deliberately out of scope rather than being guessed at in this PR.

Validation

• Full repository suite: 544 passed
• Focused sandbox/backend suite: 144 passed
• A local end-to-end fake-CLI test reproduces the exact output-overflow message and exercises subprocess launch, isolated state, stream parsing, budget carry, the concise continuation prompt, --continue, and completion.
• A second end-to-end fake-CLI test reproduces Prompt is too long, requires the isolated /compact continuation and compact_boundary, then requires a separate resumed work process before it completes.
• The existing local rate-limit end-to-end test exercises detection, budget carry, --continue, and completion.
• A parallel end-to-end test forces two independent generations to hit a synthetic cap together and proves each --continue retry recovers only its own prompt-specific session marker.
• The Docker integration test recreates the container with --rm and proves the first container's session marker reaches the resumed container.
• Unit coverage includes assistant/result/stderr detection, partial-output rejection, successful and failed compaction, bounded overflow/compaction retries, exact minute-bearing reset parsing and sleep calculation, host-state exclusion, credentials-only mounts, credential cleanup, retry metrics, budget/turn carry, exhausted limits, and missing interrupted cost.
• Black, isort, mypy, pylint on touched files, and git diff --check pass.

Limitations / not tested

• Real Claude subscription caps, real 32k output overflows, and real context exhaustion cannot be triggered deterministically. Automated end-to-end tests inject a fake Claude CLI while exercising the real runner, parser, retry, compaction orchestration, and local session-persistence paths. We did not spend subscription quota trying to force these failures.
• The overflow recovery depends on Claude Code persisting enough conversation state to make --continue useful after the API abort. The automated test proves our state and invocation plumbing, not the proprietary CLI/API's handling of every real mid-response failure.
• Anthropic documents that /compact can itself fail if the context is already too full to generate its summary. In that case this PR records the failure and stops safely; it does not silently clear the session or restart without context. The automated test covers both confirmed and failed compaction, but not a real Claude-generated summary.
• If Claude does not report interrupted cost and the run has a finite dollar budget, recovery stops safely; guessing would risk granting extra experimental budget.
• After two resumed output overflows, the run remains failed rather than looping forever.
• Apptainer command construction, auth isolation, and bind arguments are unit-tested, but there is no SIF image in this worktree. A live resumed Apptainer run was not possible here. Apptainer is used on a separate cluster and should be smoke-tested there; the implementation mirrors the Docker mount that passed end to end, but writable-tmpfs/nested-bind behavior remains the main platform-specific assumption.
• The agent's transient todo/task UI state is not separately persisted. Conversation transcripts and sandbox files are the state required for reasoning continuity.
• Resume overhead can change token allocation versus an uninterrupted run even though it does not grant a new configured budget. This is why interrupted runs remain explicitly marked in results.

Worktree note

Two unrelated pre-existing formatter-only edits in integration_tests/red_team_sandbox.py and tests/utils/test_env_in_sandbox.py were intentionally left out of every commit in this PR.

[merlerm]

Last one for today I promise! Although there is another one I'm working on since I noticed evaluation object counts were being leaked to the agent unadvertedly, but I will finish that PR once everything else is merged to avoid things getting out of hand. The other two PRs were small enough that I could work on this one as well

[Copilot]

Pull request overview

Adds resumable Claude Code runs across subscription usage-cap resets by persisting CLI session state per sandbox generation, resuming with --continue, and carrying forward remaining budget/turn limits while isolating Claude credentials/config from the operator’s live ~/.claude.

Changes:

• Persist Claude session state per sandbox generation and resume interrupted runs with --continue (local + Docker + Apptainer).
• Carry over remaining experimental max_budget_usd / max_turns across rate-limit retries without accidentally converting exhausted limits into “unlimited”.
• Add/extend unit + integration tests covering resume behavior, budget carry, and host-state exclusion.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/utils/test_sandbox.py	Adds tests for sandbox-local Claude config redirection and credential cleanup.
tests/utils/test_resume_integration.py	New end-to-end test using a fake Claude CLI to validate --continue + session persistence.
tests/utils/test_rate_limit.py	Extends retry-loop tests to cover carried budget/turns and unknown-cost handling.
tests/utils/test_docker_sandbox.py	Adds tests for extra volume mounts, credentials-only auth mounts, and Docker resume across --rm.
tests/utils/test_backends.py	Validates Claude CLI command includes session persistence and adds --continue only when resuming.
tests/utils/test_apptainer_sandbox.py	Updates auth tests to use contextmanager + validates credentials-only bind fallback.
src/robocode/utils/sandbox.py	Redirects local Claude runs to sandbox-local CLAUDE_CONFIG_DIR via a contextmanager.
src/robocode/utils/sandbox_types.py	Adds resume_previous_session to SandboxConfig.
src/robocode/utils/rate_limit.py	Refactors dispatch, carries remaining budget/turns, and resumes sessions across rate-limit sleeps.
src/robocode/utils/docker_sandbox.py	Adds session-store persistence via an extra bind mount and switches to credentials-only throwaway config fallback.
src/robocode/utils/claude_auth.py	New module implementing credentials-only throwaway config + sandbox-local config with credential cleanup.
src/robocode/utils/backends/claude.py	Enables session persistence by default and appends --continue when resuming.
src/robocode/utils/apptainer_sandbox.py	Mirrors Docker auth isolation + adds session-store bind mount persistence for resume.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[merlerm]

validated in a live session so I feel good enough merging this