swe-atlas-qna: eval cases fail on Modal sandbox / tests-dir provisioning (flat-zero)

[shehabyasser-scale]

Summary

On a full swe-atlas-qna optimizer run (2026-07-24, job runs/swe-qna/jobs/2026-07-24__20-30-56), every evaluation case failed to run and scored 0.0. Across all 396 case records the terminal exceptions were:

• AddTestsDirError ("Failed to add tests directory to environment"): 144 (36%)
• Modal sandbox NotFoundError ("The Sandbox is unavailable / may have already shut down", "Modal Sandbox with container ID ... not found"): 125 (32%)
• RuntimeError: 67 (17%)
• BadRequestError (the gpt-4o reasoning.effort 400, tracked separately): 60 (15%)

Zero cases produced a score > 0; the baseline and every candidate scored 0.0, so the optimizer had no gradient.

Impact

The task environment cannot be reliably provisioned for swe-atlas-qna on Modal: sandboxes are dying or never created, and the held-out tests directory cannot be added to the environment. This is infrastructure, not candidate quality.

Notes

• The masking that made this look like a clean 0.0 (error_rate 0.0, objective feasible) is addressed by the companion taxonomy fix (Modal sandbox / tests-dir loss now classified as transient_infra). With that fix the run correctly surfaces as invalid; this issue tracks the underlying provisioning root cause.
• Suspects to investigate: Modal sandbox idle-timeout / lifecycle (a sandbox_idle_timeout_secs knob was recently added), the tests-dir fixture step in the swe-atlas-qna verifier, and container/image availability for the pinned task_source.

Evidence

runs/swe-qna/jobs/2026-07-24__20-30-56/task__ztCcXta/verifier/{status.json, experiment.html} and the extracted verifier session evaluations/*/cases/*.json.

[shehabyasser-scale]

Root-caused in #58. The headline correction: there are two independent causes, and the LLM one is bigger than the infrastructure one.

Cause 1 — the eval model does not exist (not in this issue at all).

session/artifacts/inference/usage.json from the 2026-07-25 run:

scope	requests	upstream_errors
producer (gpt-5.3-codex)	113	0
evaluation (gpt-5.4-mini-2026-03-17)	322	322
finalization (gpt-5.4-mini-2026-03-17)	79	79

All 401 failures: 404 DeploymentNotFound. Probing the configured Azure resource directly, only gpt-5.3-codex and gpt-4o answer; every gpt-5.4-*, gpt-5.2-*, gpt-5.1*, gpt-4.1*, gpt-4o-mini and o4-mini returns 404. GET /openai/v1/models lists 172 entries including the missing one — that endpoint is the catalogue, not the deployments, which is what hid this.

So the 149 cases that survived infrastructure were scored 0.0 and labelled task_failure: the harness said the candidate failed when the candidate had no model. Same masking class as #53, one layer up.

Cause 2 — sandbox loss, and it is one failure mode, not three.

The AddTestsDirError in this issue's 36% is not a tests-directory bug. harbor/verifier/verifier.py:147-157 wraps environment.upload_dir() in a bare try/except and re-raises anything as AddTestsDirError("Failed to add tests directory to environment."). It is the same dead-sandbox event as the NotFoundErrors, seen from a different call site. The 36% / 32% / 17% split overstated how many distinct problems there are: it is one problem, harbor's DinD strategy running each sandbox over a single host-networked gRPC stream. Same SPOF that killed the tau3 optimizer trial in #55.

Also worth knowing: gaia, officeqa and tau3 point at the same undeployed model (tau3 also for TAU2_USER_MODEL / TAU2_NL_ASSERTIONS_MODEL), so they are producing zeros for the same reason. The permanent fix is provisioning the intended reasoning deployment, not downgrading four benchmarks to gpt-4o.

[shehabyasser-scale]

The remaining infrastructure losses: a private-registry image pulled anonymously

Filing the hypothesis with the evidence rather than guessing at it again, since testing it needs a Modal secret that does not exist yet.

What is verified

1. The dominant infra signature is an image fetch, not networking. Per-case terminal exceptions, 2026-07-25 run:

• 102 × RuntimeError: FetchSpec failed: loading container: file does not exist
• 165 × NotFoundError: Modal Sandbox with container ID ta-… not found
• 53 × AddTestsDirError (harbor wrapping upload_dir into a dead sandbox)
• 4 × NotFoundError: Task has already finished with status failure

loading container: file does not exist is an image-materialisation failure. The sandbox deaths are consistent with sandboxes that never had a usable image.

2. swe-atlas-qna uses a prebuilt GHCR image. ghcr.io/scaleapi/swe-atlas appears in the run trees. Harbor's image selection (environments/modal.py:255-278):

if use_prebuilt and docker_image:
    registry_secret = Secret.from_name(env._registry_secret) if env._registry_secret else None
    if ".dkr.ecr." in docker_image:
        env._image = Image.from_aws_ecr(docker_image, secret=registry_secret)
    else:
        env._image = Image.from_registry(docker_image, secret=registry_secret)
else:
    env._image = Image.from_dockerfile(...)

Nothing in the build sets registry_secret, so this is Image.from_registry("ghcr.io/scaleapi/swe-atlas…", secret=None) — an anonymous pull of a scaleapi GHCR image.

3. The nesting is exonerated by a control run. runs/gaia-full/jobs/2026-07-24__18-09-28 — same laptop, same day, same Modal account, same shared harness-engineering-bench app, same _ModalDirect nested strategy — got 524/528 cases through, 106 scoring 1.0, mean reward 0.273. GAIA's cached task.toml shows cpus = 1, memory_mb = 2048, and it takes the Image.from_dockerfile branch rather than pulling a prebuilt image.

So: Modal-in-Modal, the shared app, the nested strategy and sandbox_idle_timeout_secs are all cleared. The difference that remains is how the task image is obtained.

4. Not DinD. The nested eval logs Selected strategy: _ModalDirect on 469/469 trials with zero host-networking warnings. DinD is the outer trial's strategy only. I had this wrong in my first pass on #58 and have corrected it there.

What is not verified

I could not check swe-atlas-qna's task.toml — its task package is not in the local harbor cache (only gaia and terminal-bench are), so its CPU/memory request and its exact image tag are unconfirmed. A larger resource request is a plausible second contributor and would be visible in that file.

The lever

--ek registry_secret=<modal-secret-name> in the nested extra_harbor_args, pointing at a Modal secret holding GHCR credentials. Note the ECR branch above is selected by the .dkr.ecr. substring, so a GHCR image correctly takes Image.from_registry, which does accept the same secret.

Needs someone to create the Modal secret first. Happy to wire and test it once it exists.

[shehabyasser-scale]

Separate bug found while measuring the above: infrastructure_max_attempts never fires

Not fixing this in a PR yet — see the caveat at the end — but it is verified and it compounds everything else on this issue.

Evidence

The 2026-07-25 combined run has infrastructure_max_attempts: 3 and lost 456 of 520 cases to transient_infra. Attempts actually made:

attempts-per-case distribution: {1: 520}
harbor/retry-N/ artifact dirs referenced: NONE
retry directories on disk:                NONE

Every case got exactly one attempt. The retry never ran, not once.

Why

harbor/backend.py:1228:

groups = self._trial_groups(attempt_jobs_dir)
# Require FULL coverage before accepting the sub-run.
if requested_tasks <= set(groups):
    jobs_dir = attempt_jobs_dir
    break

_trial_groups builds its keys by walking jobs_dir.rglob("result.json") and reading each trial's task_name. A trial that dies still writes a result.json — that is where exception_info lives, which is exactly what the case census reads. So a task whose sandbox collapsed is still present in groups, requested_tasks <= set(groups) is satisfied, and the loop breaks on attempt 1.

It is a coverage check being relied on as a health check. It only fires when harbor emits no result.json at all for a task, which is far rarer than a trial dying.

Why I have not just fixed it

The obvious one-line fix is to gate on usable trials instead — the predicate already exists as _attempt_is_infra (backend.py:810):

usable = {t for t, trials in groups.items()
          if any(not self._attempt_is_infra(x) for x in trials)}
if requested_tasks <= usable:
    ...

But groups is reassigned on each attempt and only the final attempt's groups survives the loop. Turning retries on without also merging results across attempts would let a later attempt discard good trials an earlier one produced — trading a silent no-op for a silent scoring regression in the most integrity-sensitive path in the backend. The real fix is retry + per-task merge, keeping the best trial across attempts.

I am not shipping that against the scoring path without a live run behind it, and a live run is not informative until the image-pull cause above is fixed: with 88% of cases failing for a deterministic reason, enabling retries would just burn 3× the time and money to fail 3× as often.

Suggested order: registry secret → confirm the infra rate drops → then fix the retry gate with the merge, and validate that retries both fire and preserve earlier successes.