fix: point swe-atlas-qna at a provisioned model and off the DinD SPOF

[shehabyasser-scale]

Refs #51.

swe-atlas-qna has never produced a non-zero score. The 2026-07-25 run scored 0.0 on all 469 cases. There are two independent causes, and the one everybody was chasing is the smaller of the two.

Cause 1: the eval model is not deployed (the one nobody was looking at)

From that run's exported session/artifacts/inference/usage.json:

scope	requests	upstream_errors	tokens
producer (gpt-5.3-codex)	113	0	8,966,339
evaluation (gpt-5.4-mini-2026-03-17)	322	322	0
finalization (gpt-5.4-mini-2026-03-17)	79	79	0

Every one of those 401 failures is the same body:

{"error": {"type": "invalid_request_error", "code": "DeploymentNotFound",
  "message": "The API deployment for this resource does not exist."}}

Confirmed directly against the configured Azure resource — of the entire catalogue, only two deployments answer:

gpt-5.3-codex                HTTP 200
gpt-4o                       HTTP 200
gpt-5.4-mini-2026-03-17      HTTP 404  DeploymentNotFound
gpt-5.4-mini / gpt-5.4 / gpt-5.4-2026-03-05 / gpt-5.4-nano-2026-03-17   404
gpt-5.2-2025-12-11 / gpt-5.1 / gpt-5-mini-2025-08-07                    404
gpt-4.1 / gpt-4o-mini / o4-mini                                         404

GET /openai/v1/models returns 172 entries including gpt-5.4-mini-2026-03-17, which is what made this invisible: that endpoint lists the model catalogue, not the resource's deployments.

So the candidate agent had no working model. The 149 cases that survived infrastructure were duly scored 0.0 and labelled task_failure — the harness reporting "the candidate failed the task" when the truth was "there was no model to call".

Cause 2: sandbox loss (the 68% everyone was looking at)

The other 320 of 469 cases are transient_infra: NotFoundError ×314, RuntimeError ×102, AddTestsDirError ×53.

AddTestsDirError is not a separate tests-directory bug. harbor/verifier/verifier.py:147-157 wraps environment.upload_dir() in a bare try/except and re-raises anything as AddTestsDirError("Failed to add tests directory to environment."). It is the same dead-sandbox event as the NotFoundErrors, observed from a different call site. Ranking them as separate causes in #51 (36% / 32% / 17%) overcounted the problem's dimensionality.

Harbor's DinD strategy runs each sandbox over a single host-networked gRPC stream and warns about it at startup. The same SPOF took out the tau3 optimizer trial (#55).

Changes

1. model: openai/gpt-4o and inference_gateway.evaluation.allowed_models: [gpt-4o], moved together — the allow-list is enforced before the upstream, so a mismatch turns a 404 into a 403 and nothing gets better. The agent strips the provider prefix (atlas_agent/agent.py:80), so the bare form is what the gateway sees.
2. --ek modal_vm_runtime=true appended to the nested eval's extra_harbor_args.
3. The benchmark-config test now parametrizes the expected eval model and asserts model and allowed_models agree, so they cannot drift apart again.

Dependency

gpt-4o is a non-reasoning model, so this is only safe on top of #54. Without that guard, every agent call goes from 404 DeploymentNotFound to 400 unsupported_parameter: reasoning.effort. #54 stops being a nice-to-have and becomes load-bearing.

Wider blast radius (not fixed here)

gaia, officeqa and tau3 all point model and evaluation.allowed_models at the same undeployed gpt-5.4-mini-2026-03-17; tau3 additionally sets TAU2_USER_MODEL and TAU2_NL_ASSERTIONS_MODEL to it. All three are therefore silently producing zeros for the same reason. Only swe-bench-pro is already on gpt-4o. I have not touched them here because I can only prove one benchmark per live run, and a config change I have not run is exactly the kind of thing that produced this bug. The right permanent fix is provisioning the intended reasoning deployment rather than downgrading four benchmarks to gpt-4o.

Verification

A combined live run (#51 + #52 + #53 + #54) is in flight against the same Modal conditions as the failing run. Predicted signal change: eval-scope upstream_errors drops from 322/322 to ~0, transient_infra case count drops well below 320/469, and at least one case scores > 0. Results posted here before merge.

Greptile Summary

This PR fixes a silent zero-score bug in swe-atlas-qna caused by the evaluation model (gpt-5.4-mini-2026-03-17) not being provisioned on the configured Azure resource, swapping it for gpt-4o and keeping the inference-gateway allow-list in lockstep.

• build.yaml: model changed to openai/gpt-4o; inference_gateway.evaluation.allowed_models updated to [gpt-4o] with a comment mandating they move together.
• test_v05_harbor_build.py: the canonical-benchmark test is parametrized with per-benchmark evaluation_models and gains a new assertion that config.model == f\"openai/{evaluation_models[0]}\", preventing future model/allow-list drift.
• The second claimed fix \u2014 appending --ek modal_vm_runtime=true to extra_harbor_args to address the DinD SPOF (Cause 2, sandbox loss on 68% of cases) \u2014 is not present in the actual diff despite being listed in the PR description and title.

Confidence Score: 4/5

Safe to merge for the model-provisioning fix, but the DinD SPOF described in the PR title and body is not addressed by the actual diff.

The model swap from an unprovisioned deployment to gpt-4o is correct and the test guard against future drift is well-designed. However, the PR description explicitly enumerates adding --ek modal_vm_runtime=true to extra_harbor_args as a second required change (fixing the 68% sandbox-loss rate), and that change is entirely absent from the diff. Merging as-is leaves Cause 2 unresolved despite the title claiming otherwise.

Files Needing Attention: harness-engineering-bench/swe-atlas-qna/baseline/build.yaml — the extra_harbor_args line needs the modal_vm_runtime flag that the PR description says was added.

Important Files Changed

Filename	Overview
harness-engineering-bench/swe-atlas-qna/baseline/build.yaml	Switches eval model from unprovisioned gpt-5.4-mini-2026-03-17 to gpt-4o and keeps allowed_models in lockstep — but the second claimed fix (adding --ek modal_vm_runtime=true for the DinD SPOF) is absent from the diff.
vero/tests/test_v05_harbor_build.py	Adds per-benchmark evaluation_models parameter to the canonical-benchmarks test and asserts model/allowed_models stay in lockstep; clean, well-targeted change.

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
harness-engineering-bench/swe-atlas-qna/baseline/build.yaml:56
**Claimed DinD fix absent from the diff**

The PR description explicitly lists Change 2 as "`--ek modal_vm_runtime=true` appended to the nested eval's `extra_harbor_args`", and the PR title reads "off the DinD SPOF". However, `extra_harbor_args` is unchanged in this diff — it remains `["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]`. Cause 2 (sandbox loss on 68% of cases) is therefore not addressed by this PR, even though both the title and description claim it is.

_{Reviews (2): Last reviewed commit: "fix: drop the modal_vm_runtime guess, ke..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

Provider-prefix assumption is implicit

f"openai/{evaluation_models[0]}" hardcodes the openai/ provider prefix. This works for every benchmark today, but the first entry that uses a different provider (e.g., anthropic/…) will appear to pass at the parametrize level — the evaluation_models list would still hold the bare model name — but the assertion would fail with a confusing mismatch. Consider storing the full prefixed form (e.g., "openai/gpt-4o") in evaluation_models and asserting config.model == evaluation_models[0] directly, or at least adding a comment that the openai/ prefix is intentional and must be kept in sync with how agents strip the prefix.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/tests/test_v05_harbor_build.py
Line: 81

Comment:
**Provider-prefix assumption is implicit**

`f"openai/{evaluation_models[0]}"` hardcodes the `openai/` provider prefix. This works for every benchmark today, but the first entry that uses a different provider (e.g., `anthropic/…`) will appear to pass at the parametrize level — the `evaluation_models` list would still hold the bare model name — but the assertion would fail with a confusing mismatch. Consider storing the full prefixed form (e.g., `"openai/gpt-4o"`) in `evaluation_models` and asserting `config.model == evaluation_models[0]` directly, or at least adding a comment that the `openai/` prefix is intentional and must be kept in sync with how agents strip the prefix.

How can I resolve this? If you propose a fix, please make it concise.

[shehabyasser-scale]

Correcting two things in the description above, and adding the control run I should have looked at first.

The per-case terminal exception census (exact, one row per case, 469 total)

n	exception	what it is
165	NotFoundError: Modal Sandbox with container ID ta-… not found	sandbox died
145	NotFoundError: Error code: 404 … 'code': 'DeploymentNotFound'	the missing model deployment
102	RuntimeError: FetchSpec failed: loading container: file does not exist	container image failed to materialize
53	AddTestsDirError: Failed to add tests directory to environment.	sandbox died, seen from the verifier
4	NotFoundError: Task has already finished with status failure	sandbox died

So my "NotFoundError ×314" above double-counted across categories, and — more importantly — I framed the infrastructure half as the DinD gRPC SPOF. That framing is wrong for this layer. The nested eval reports Selected strategy: _ModalDirect on all 469 trials and emits zero DinD host-networking warnings. DinD is the outer trial's strategy (that is #55's failure, correctly diagnosed there). The dominant nested signature is FetchSpec failed: loading container: file does not exist, which is image materialization, not networking.

The control run that settles it

GAIA, same laptop, same day, same Modal account, same shared harness-engineering-bench app, same _ModalDirect nested strategy: runs/gaia-full/jobs/2026-07-24__18-09-28 → 524 of 528 cases succeeded, 4 infra errors, 106 cases scored 1.0, mean reward 0.273.

Modal-in-Modal nesting is therefore not broken, and neither is the shared app or the nested strategy. Whatever is killing swe-atlas-qna's sandboxes is specific to that benchmark's task environment — the prime suspect being its prebuilt ghcr.io/scaleapi/swe-atlas registry image (pulled with no registry secret) and its much larger CPU/memory request, versus GAIA's build-from-Dockerfile.

What this means for this PR

• The model change is well supported and unaffected: GAIA proves gpt-4o both works on this endpoint and yields real scores.
• The --ek modal_vm_runtime=true in extra_harbor_args is a guess I cannot currently justify, and it carries risk of its own (the VM runtime is alpha and disables GPU support). The live run in flight is now the test of exactly that: if the transient_infra count does not move off 320/469, I will drop it from this PR and re-file the image-provisioning hypothesis separately rather than ship an unsupported change.

Same run also gives #54 a proper before-measurement: that GAIA run used gpt-4o without the guard and ate 715 HTTP 400s (514 evaluation + 201 finalization) out of 2376 gateway calls — ~30% of its inference wasted on reasoning.effort.

[shehabyasser-scale]

Live result: model fix proven, infrastructure fix disproven and removed

runs/atlas-all-fixes/jobs/2026-07-25__08-58-16, 35 min, $3.08, on a branch carrying #51 + #52 + #53 + #54. The outer trial completed cleanly (n_errored_trials: 0).

The model half works

	failing run (2026-07-25 07:33)	this run
evaluation scope	322 requests / 322 upstream_errors / 0 tokens	284 requests / 0 upstream_errors / 245,524 tokens
finalization scope	79 / 79 / 0	0 / 0 / 0
producer scope	113 / 0	147 / 0

Full request-log census: 284 × (evaluation, gpt-4o, 200) and 147 × (producer, gpt-5.3-codex, 200). Every single gateway request succeeded. No DeploymentNotFound, and — see #54 — no reasoning.effort 400s either.

For the first time this benchmark produced informative samples: 64 cases reached the judge and were scored, against 0 before.

The infrastructure half does not, so I removed it

I said above that I would drop --ek modal_vm_runtime=true if it did not move the transient_infra count. It did not:

	with the flag	without it
cases	520	469
transient_infra	456 (88%)	320 (68%)
task_failure	64	149
scored > 0	0	0

dead_exception_types: NotFoundError ×411, RuntimeError ×64, AddTestsDirError ×45.

The apparent worsening is mostly not a regression — previously 149 cases died early on the model 404 and never got far enough to hit the sandbox problem. Now they proceed and hit it. The honest reading is that essentially every case still dies to infrastructure, and the model fix only changed where.

Latest commit removes the flag. This PR is now purely the model pin plus the test that keeps model and allowed_models in lockstep.

What remains

infrastructure_invalidity_threshold_exceeded still fires and finalization still reports evaluation did not produce a feasible objective — which is #53 working correctly: an 88%-infra run is refused rather than reported as a clean 0.0.

The open cause is the one filed on #51: ghcr.io/scaleapi/swe-atlas is pulled through Image.from_registry(docker_image, secret=None) because nothing sets registry_secret. Blocked on a Modal secret existing. Not guessing at another --ek in the meantime.

[greptile-apps]

Claimed DinD fix absent from the diff

The PR description explicitly lists Change 2 as "--ek modal_vm_runtime=true appended to the nested eval's extra_harbor_args", and the PR title reads "off the DinD SPOF". However, extra_harbor_args is unchanged in this diff — it remains ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]. Cause 2 (sandbox loss on 68% of cases) is therefore not addressed by this PR, even though both the title and description claim it is.

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-engineering-bench/swe-atlas-qna/baseline/build.yaml
Line: 56

Comment:
**Claimed DinD fix absent from the diff**

The PR description explicitly lists Change 2 as "`--ek modal_vm_runtime=true` appended to the nested eval's `extra_harbor_args`", and the PR title reads "off the DinD SPOF". However, `extra_harbor_args` is unchanged in this diff — it remains `["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]`. Cause 2 (sandbox loss on 68% of cases) is therefore not addressed by this PR, even though both the title and description claim it is.

How can I resolve this? If you propose a fix, please make it concise.

[shehabyasser-scale]

Withdrawing this PR's diagnosis. pr3-harness-bench found the real cause and it is not what I filed.

What I claimed: swe-atlas-qna's 456/520 infrastructure losses came from the prebuilt ghcr.io/scaleapi/swe-atlas image being pulled via Image.from_registry(..., secret=None), and the lever was --ek registry_secret=<modal-secret>.

What it actually was, per b6798bc: these images set ENTRYPOINT ["/bin/bash"], Modal prepends it to the sandbox command, and harbor's default keepalive ["sh","-c","sleep infinity"] then makes bash treat the sh binary as a script. Bash exits immediately, Modal reaps the sandbox, and everything downstream fails against a container that no longer exists. The fix is --ek keepalive=["-c","sleep infinity"].

That explains my census better than my own hypothesis did:

terminal exception	count	explained by keepalive?
NotFoundError: Modal Sandbox with container ID ... not found	165	yes, directly
RuntimeError: FetchSpec failed: loading container: file does not exist	102	yes, one step removed
AddTestsDirError	53	consistent (upload into a just-reaped sandbox), but I would not call it proven
NotFoundError: 404 DeploymentNotFound	145	no, that is the model half

And the decisive evidence is that no registry secret was ever added, yet 98313bc records a completed K=3 held-out run losing 5 of 150 trials, all to gpt-oss 128k context overflow rather than infrastructure. A 456/520 to 5/150 collapse with no registry change means image pulls were never the problem.

The model half of this PR is superseded too: fc0197b moved the target off Azure entirely to fireworks_ai/gpt-oss-120b, so 404 DeploymentNotFound cannot occur.

Recommend closing this as superseded. Nothing in it survives. The modal_vm_runtime half I already reverted after it failed to move the numbers; the gpt-4o pin is replaced by the Fireworks model; the registry-secret follow-up I filed should not be actioned.

One correction worth carrying forward rather than losing: my model: openai/gpt-4o + allowed_models: [gpt-4o] shape would not even have loaded under 800af32's new validator, and the dual-spelling workaround fails 20dd166's invariant. The agents strip only openai/ and the gateway matches the allow-list exactly, so the correct shape is unprefixed on both sides. I have applied that to swe-bench-pro's config in #50.

[shehabyasser-scale]

Closing as superseded by b6798bc (keepalive) and fc0197b (target model) on pr3-harness-bench. Full reasoning and the retraction of my registry-secret diagnosis are in the comment above.