feat: let a build configure its outer optimizer trial's harbor flags

[shehabyasser-scale]

What broke

The tau3 optimizer trial dies at teardown, after codex has already finished optimizing. From the 2026-07-25 run (runs/tau3-rerun/jobs/2026-07-25__07-33-29, ~40 min, $1.57 burned for zero signal):

"n_errored_trials": 1,
"n_retries": 0,
"exception_stats": {"StreamTerminatedError": ["task__cmDWxgt"]}

task__cmDWxgt/exception.txt puts it in harbor's finalization verifier, not in the model:

harbor/trial/single_step.py:105  _run_verifier
harbor/verifier/verifier.py:199  await self.environment.exec(...)
harbor/environments/dind_compose.py:121  _compose_exec
harbor/environments/modal.py:1304  _sdk_exec  ->  grpclib StreamTerminatedError: Connection lost

Harbor prints the remedy itself, as line 4 of that very job's log:

DinD mode uses host networking: no port isolation between services, no Docker DNS service discovery (extra_hosts entries map service names to 127.0.0.1 instead), and no network namespace isolation. Use --ek modal_vm_runtime=true to use the VM runtime instead which does not use host networking.

DinD funnels the whole sandbox through one host-networked gRPC stream, so a single drop takes down a 20+ minute trial with no partial credit.

Why the previous version of this PR was wrong

It raised tau3's max_retries 1 → 3. That knob is the nested per-case retry: build/config.py → HarborBackendConfig.max_retries → the inner harbor run --max-retries in backend.py::_harbor_command. The failing layer is the outer trial, which never gets --max-retries at all. The live rerun confirmed it: still retries=0, still "Not retrying trial because the maximum number of retries has been reached". Reverted to 1.

Same trap applies to extra_harbor_args: it also only reaches the nested eval. There was no way for a build to say anything about the environment the optimizer itself runs in.

The fix

Add optimizer_harbor_args to HarborBuildConfig, forwarded by vero harbor run to the outer harbor run, and set tau3's to ["--ek", "modal_vm_runtime=true"].

• Forwarded before any trailing CLI args, so the command line still wins (harbor's parse_kwargs at cli/utils.py:65 is last-value-wins per key).
• Validator rejects the four flags the CLI owns (-p, -a, -e, -m), mirroring the existing extra_harbor_args guard.
• The compiled compose addresses services by name (http://eval-sidecar:8000, http://inference-gateway:8001), which the VM runtime resolves through real Docker DNS rather than the DinD extra_hosts → 127.0.0.1 remap, so this is the better-supported path for this task shape, not just the more resilient one.

Live verification

Plumbing (from the run log, the outer command vero assembles):

harbor run -p /…/vero-harbor-6zi3nb4v/task -a codex -e modal -m gpt-5.3-codex \
  --ek modal_vm_runtime=true --yes -o ../runs/tau3-vmruntime/jobs

An end-to-end tau3 run on Modal with this branch is in flight; it is a straight A/B against 2026-07-25__07-33-29 (identical config, only the flag differs). Result posted here when it lands.

Tests: test_harbor_run_forwards_build_declared_optimizer_args (outer command carries modal_vm_runtime=true, nested-only flags stay off it, build-declared flags precede CLI args) and an optimizer_harbor_args controlled-flag case in test_load_build_config_rejects_invalid_specs.

Refs #55.

Greptile Summary

This PR adds optimizer_harbor_args to HarborBuildConfig, a new escape hatch that lets a build inject flags directly into the outer harbor run that hosts the optimizer trial — distinct from the already-existing extra_harbor_args, which only reaches the nested per-candidate evaluation sub-run. The field is forwarded in cli.py before the user's trailing CLI args, so a command-line override always wins (harbor's last-value-wins semantics). The tau3 build's comment is notably honest: all four attempted fixes (DinD, VM runtime, sandbox v2, max-retries) failed identically at teardown, so the field is shipped empty there while the upstream root cause is tracked.

• config.py: New optimizer_harbor_args: list[str] field with a Pydantic field_validator that rejects the four flags vero harbor run controls (-p, -a, -e, -m).
• cli.py: One-line insertion appends optimizer_harbor_args to the outer command between agent_env --ae flags and the user's extra args, preserving CLI override priority.
• Tests: A new end-to-end CLI test (test_harbor_run_forwards_build_declared_optimizer_args) confirms the flag reaches the outer command, stays off the nested command, and precedes --yes; existing SimpleNamespace stubs in test_v05_harbor_http.py are updated to include the new attribute.

Confidence Score: 5/5

Safe to merge — the change is additive, the new field defaults to an empty list (no behavior change for existing builds), and the forwarding logic is a single well-tested line.

The change is narrow and well-tested: a new optional list field with a validator, a one-line CLI insertion, and matching test coverage for both the happy path and the rejection case. Existing builds are unaffected by the empty default. The one gap (long-form flag equivalents not in the controlled set) is harmless given last-value-wins ordering.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/build/config.py	Adds optimizer_harbor_args field with a validator that blocks the 4 CLI-owned short flags (-a, -e, -m, -p); long-form equivalents are not blocked, but last-value-wins ordering makes this safe in practice.
vero/src/vero/harbor/cli.py	Correctly inserts optimizer_harbor_args between agent_env flags and the user's extra args so CLI-provided flags always take final precedence.
harness-engineering-bench/tau3/baseline/build.yaml	Adds a comment noting that four live experiments all failed with the same teardown error, so optimizer_harbor_args is left empty; infrastructure is present for future use.
vero/tests/test_v05_cli.py	New regression test verifies outer command carries optimizer_harbor_args, excludes extra_harbor_args, and preserves ordering (build-declared flags before CLI extras).
vero/tests/test_v05_harbor_build.py	Adds round-trip and rejection tests for optimizer_harbor_args; covers the controlled-flag guard and a valid --ek case.
vero/tests/test_v05_harbor_http.py	Updates two existing SimpleNamespace config stubs to include optimizer_harbor_args=[] so they don't crash on the new attribute access in cli.py.

_{Reviews (4): Last reviewed commit: "test: teach the run-command config stubs..." | Re-trigger Greptile}

[shehabyasser-scale]

Live result: the fix did not work. modal_vm_runtime=true does not save the tau3 trial.

runs/tau3-vmruntime/jobs/2026-07-25__08-38-37, 08:38 → 09:18, $1.27:

"n_errored_trials": 1, "n_retries": 0,
"exception_stats": {"StreamTerminatedError": ["task__pnBLeta"]}

The flag itself plumbed through correctly, so the mechanism in this PR works:

harbor run -p …/task -a codex -e modal -m gpt-5.3-codex --ek modal_vm_runtime=true --yes …
Selected strategy: _ModalDinD
Using vm_runtime: True          <- and the host-networking warning is gone

But the trial still ends grpclib.exceptions.StreamTerminatedError: Connection lost, with a tail identical to the previous run's.

The one thing that changed is the most informative part

The failure moved phases:

	previous run (DinD)	this run (vm_runtime)
harbor frame	single_step.py:105 _run_verifier → verifier.py:199 verify	single_step.py:77 _run_agent → codex.py:1102 run
when	after codex finished, during the verifier	during the codex run itself
bottom of stack	modal.py:1304 _sdk_exec → StreamTerminatedError	identical

Same bottom of stack, different long-running exec. That rules out my hypothesis. This is not about host networking or which exec happens to run at teardown — it is that harbor runs each phase as one exec and streams its stdout for the phase's entire duration, and a ~35-minute stream does not survive. Modal's own _stream_stdio_with_retries (task_command_router_client.py:838) tries to recover and re-raises.

So the original version of this PR had the right instinct and the wrong layer: retry is the applicable lever, it just has to reach the outer trial. The outer harbor run receives no --max-retries, which is why both runs log retries=0 and "Not retrying trial because the maximum number of retries has been reached".

What I propose keeping

• optimizer_harbor_args stays. It is verified working, it is the only way to configure the outer trial at all, and it is now the mechanism the actual fix needs.
• tau3's --ek modal_vm_runtime=true should come out, or stay only on its own merits (it does remove the host-networking caveat), not as a fix for this. I am not going to leave a value in a committed config claiming to fix something a live run says it does not fix.

Next experiment, not yet run

optimizer_harbor_args: ["--max-retries", "1"]. Honest caveat before spending it: a retry re-runs the whole ~40-minute codex trial, so worst case is ~80 minutes and ~$2.5, and it only pays off if the stream drop is probabilistic. Two for two so far, at different phases, is weakly consistent with "random drop over a long window" — but two samples is two samples. Flagging the cost rather than burning it unasked.

[shehabyasser-scale]

Final: all four experiments failed. This PR is now plumbing only.

#	outer trial config	elapsed	outcome
1	default (DinD)	39m55s	StreamTerminatedError in the verifier exec
2	--ek modal_vm_runtime=true	39m45s	StreamTerminatedError in the codex agent exec
3	--ek modal_sandbox_v2=true	38m14s	StreamTerminatedError
4	--max-retries 1	58m38s	StreamTerminatedError on both attempts

~$5.30 and four runs. tau3's optimizer trial now has no optimizer_harbor_args value, because none of these is a fix and a committed value implying otherwise is worse than an empty one.

What the experiments did establish

The mechanism works. Run 4 logged:

Trial task__XQNGEdr failed with exception StreamTerminatedError. Retrying in 1.00 seconds...

Every earlier run logged Not retrying trial because the maximum number of retries has been reached. So optimizer_harbor_args genuinely reaches the outer harbor run, and the outer trial is configurable for the first time. That is what this PR is for, and it stands on its own.

Retry is not the cure. The second attempt died identically. On this workload retry converts one guaranteed loss into two, at double the cost.

It is not a networking mode. vm_runtime moved the failure between phases while keeping an identical bottom of stack; sandbox_v2 changed nothing.

It looks deterministic, not flaky. 39m55s, 39m45s and 38m14s across three independent runs is far too tight a spread for a random gRPC drop. That is the single most useful thing these runs produced, and it argues against every "add resilience" fix including the one I proposed.

Where the real cause probably lives

Harbor runs each trial phase as one exec and streams its stdout for the phase's entire duration (codex.py:1102 → base.py:592 exec_as_agent → modal.py:1304 _sdk_exec), so any phase longer than ~40 minutes has a single point of failure with no partial credit. I looked for a matching deadline and did not find one: harbor's agent/verifier defaults are 600s, sandbox_timeout_secs defaults to 24h, and vero's compiled task.toml sets no [agent] timeout_sec at all. So whatever ends it at ~40 minutes is not in harbor's config surface, which is why no --ek fixed it.

This should go upstream to harbor as "long-running agent phases die at ~40 min on the Modal DinD/VM exec stream", with these four tracebacks attached. I have stopped spending on it here.

Review note

The one remaining judgement call is whether optimizer_harbor_args earns its place with no benchmark currently setting it. I think yes: without it there is no way to pass anything to the outer trial, and diagnosing this at all required exactly that. But it is a fair thing to push back on, and I would rather you decide that than have me quietly justify the field with a value that does not work.

[shehabyasser-scale]

Restacked. optimizer_harbor_args landed adjacent to agent_env, which f4e2057 cherry-picked from my fix/forward-agent-env-to-optimizer, so both the field block and the run_command call site conflicted. Resolved by keeping both; they are orthogonal (--ae populates the agent's env, this appends outer --ek flags) and the ordering only matters for flag precedence, which neither contests. Added the field to the Attributes: docstring per the convention 145fa29 introduced.

Also needed a test fix that is worth knowing about generally: test_v05_harbor_http.py builds the build-config as a SimpleNamespace carrying only the attributes run_command reads, so any new field breaks two tests with an AttributeError that surfaces only as a non-zero exit code with empty output. Same thing happened when agent_env was added. Stub updated in both directions.

The PR remains plumbing-only and no benchmark sets the field. The four live experiments behind the original tau3 justification all failed identically (default DinD 39m55s, modal_vm_runtime 39m45s, modal_sandbox_v2 38m14s, --max-retries 1 58m38s on both attempts), every one ending in grpclib StreamTerminatedError. The ~38-40 min consistency across three independent runs is the load-bearing datum: that is deterministic, not a flaky stream drop, so resilience flags are the wrong family of fix. Nothing in pr3-harness-bench touches it either, so it is still open and still belongs upstream in harbor.

That leaves the honest question unchanged: whether a config field earns its place with no caller. I would rather you decide that than have me justify it with a value that does not work.

Full suite matches the base branch: 11 pre-existing failures, none new.