fix: export the session on every terminal outcome, not only when the verifier runs

[shehabyasser-scale]

The gap

An optimization's durable output (candidate commits, evaluation records, per-case scores) is written by the verifier phase. But the verifier phase does not run on every failure, so whether hours of work survives depends on which exception type harbor caught.

harbor/trial/single_step.py runs _run_agent() -> _collect_artifacts() -> _run_verifier(), and _run_agent swallows only two types:

except (AgentTimeoutError, NonZeroAgentExitCodeError) as exc:
    self._record_exception(exc)

Anything else propagates and _run_verifier() never executes.

Verified against two real runs on 2026-07-31

	Run 1	Run 2
exception	StreamTerminatedError (grpclib)	ApiRateLimitError
in that tuple?	no	yes (ApiRateLimitError is a NonZeroAgentExitCodeError, confirmed via issubclass)
result.json verifier	null	ran, 22:29:05 to 22:57:26
verifier/ contents	empty	7 files, session.tar.gz 8.8M

Run 1 lost everything despite having already earned a validation score of 0.1224 on 49 cases. Run 2 kept a shipped candidate (da6ab2d68f22) that was later recovered and rescored.

The sandbox was reachable the whole time. Run 1's trial.log shows Trial failed: Connection lost immediately followed by Collecting main service artifacts, and its artifacts/manifest.json records status: "ok". Harbor calls _collect_artifacts from _recover_outputs on the failure path. The evidence was sitting on a volume harbor could still read; nothing exported it.

Fix

Hang the export off artifact collection, which runs on every terminal outcome, instead of off the verifier phase, which does not.

• vero harbor archive-session: token-free, HTTP-free, finalize-free snapshot straight off the admin volume. Measured 3.2s on the real 63M / ~2300-file Run 2 session, against the 28 minutes Run 2's verifier phase took.
• task.toml.j2 emits a [[verifier.collect]] hook plus a matching [[artifacts]] entry, landing the archive at <task>/artifacts/session-rescue.tar.gz. Both are best-effort in harbor and cannot fail a trial.
• LAYOUT.session_rescue_archive is a sibling of session_dir, so the verifier's own later export cannot recursively contain it.

Deliberately not done

rescore_candidate.py was not ported into vero. Its value is the scoring, which needs the benchmark's build.yaml and pinned baseline_reward from harness-engineering-bench. A vero command that extracts a worktree but cannot score it would not close the gap. The docs now name the script explicitly so recovery stops being tribal knowledge.

A true "resume" is infeasible and was not attempted: the optimizer's working tree and harness context die with the sandbox (environment.delete: true). What is worth keeping is the candidates and the scores, and those are recoverable.

Verification beyond unit tests

• Rendered task.toml validates against harbor 0.20.0's real TaskConfig; the collect hook parses as ('eval-sidecar', 'vero harbor archive-session', 600.0). Previously sidecars was empty and _collect_artifacts_phased short-circuited at if not sidecars: return.
• Ran the new command against the real Run 2 session: produced an 8.8M archive that round-trips through extract_harbor_session_archive and yields candidate da6ab2d68f22... with parent 43bc6a0f..., matching finalization.json exactly.

Tests

Three new: test_compiled_task_rescues_the_session_outside_the_verifier_phase, test_archive_session_command_snapshots_without_a_token_or_a_live_sidecar, test_archive_session_command_reports_a_missing_session, plus a session_rescue_archive assertion in the pinned-layout test.

Note the suite requires credentials in the environment; without them six tests fail on compiler.py:566 credential validation and look like regressions. With set -a; . heb.secrets.env; set +a the full suite is 485 passed, 15 skipped, one unrelated collection error in test_uv_with_editable.py.

🤖 Generated with Claude Code

Greptile Summary

Adds a best-effort rescue path for optimization sessions when Harbor does not reach the verifier phase.

• Registers a sidecar collection hook that archives session state before artifact download.
• Exposes a token-free, filesystem-only archive-session command.
• Adds the rescue archive path and compiler configuration.
• Documents session recovery and adds configuration and CLI regression coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/build/templates/task.toml.j2	Registers the sidecar collection hook and matching rescue-archive artifact.
vero/src/vero/harbor/cli.py	Adds a filesystem-only command that snapshots a session without finalization or credentials.
vero/src/vero/harbor/build/compiler.py	Supplies the rescue hook timeout and flat artifact destination to the task template.
vero/src/vero/layout.py	Defines the rescue archive beside the session directory to avoid recursive inclusion.
vero/tests/test_v05_harbor_build.py	Verifies emitted TOML and Harbor model parsing route the hook and artifact through the sidecar.
vero/tests/test_v05_harbor_session.py	Covers successful token-free archiving, extraction compatibility, and missing-session failure.

Sequence Diagram

sequenceDiagram
    participant H as Harbor trial
    participant S as Evaluation sidecar
    participant V as Admin volume
    participant A as Trial artifacts
    H->>S: Run agent/verifier lifecycle
    alt Verifier completes
        S->>V: Finalize and export session.tar.gz
    else Terminal outcome skips verifier
        H->>S: Run archive-session collect hook
        S->>V: Create session-rescue.tar.gz
    end
    H->>V: Collect configured archive
    V-->>A: Copy session-rescue.tar.gz

Loading

_{Reviews (2): Last reviewed commit: "test: assert Harbor itself files the res..." | Re-trigger Greptile}