fix: classify Modal sandbox / tests-dir loss as transient_infra

[shehabyasser-scale]

What

The harbor error taxonomy records any unrecognized case failure as TASK_FAILURE: an informative, scoreable sample at the failure value with status=SUCCESS. That is correct for a candidate that crashes on its own, but Modal sandbox lifecycle failures and a missing held-out tests fixture are infrastructure the candidate did not cause, and today they are silently scored 0.

Why

On a swe-atlas-qna optimizer run (2026-07-24), all 396 evaluation cases came back status=success, error_category=task_failure, score=0.0. The terminal exceptions:

exception	count	share
AddTestsDirError ("failed to add tests directory")	144	36%
Modal sandbox NotFoundError / "Sandbox unavailable"	125	32%
RuntimeError	67	17%
BadRequestError (gpt-4o reasoning.effort 400)	60	15%

Because these were classified as task failures, the aggregate reported error_rate 0.0 and objective.feasible=true at 0.0: a fully broken environment masqueraded as a real, shippable zero, and the optimizer ran with no gradient (baseline and every candidate 0.0).

Change

Add the harness/Modal environment-loss signals (sandbox lifecycle, AddTestsDirError / "failed to add tests directory", container fetch failures, StreamTerminatedError) to the TRANSIENT_INFRA patterns in error_taxonomy.py. Those cases are now excluded from the aggregate and counted toward invalidity, so a collapsed environment surfaces as an invalid run instead of a silent zero.

A candidate's own bug (ValueError, KeyError, no answer) is still a task failure, unchanged.

Tests

Two new tests in test_v05_error_taxonomy.py cover the new signals and assert existing classification behavior is unchanged. Verified locally by exercising the classifier directly (new signals map to TRANSIENT_INFRA, all prior cases unchanged).

Related

• The underlying provisioning root cause is tracked in swe-atlas-qna: eval cases fail on Modal sandbox / tests-dir provisioning (flat-zero) #51; this PR only stops it being masked.
• Base branch: feat/swe-bench-pro-baseline-scaffold (stacked on the current integration tip).

🤖 Generated with Claude Code

Greptile Summary

This PR fixes a critical masking bug: Modal sandbox lifecycle failures and missing held-out test fixtures were classified as TASK_FAILURE (scored as an informative 0.0) instead of TRANSIENT_INFRA (excluded from the aggregate), causing a fully broken environment to masquerade as a real zero-score result. It also introduces a new UPSTREAM_MODEL_UNAVAILABLE category for cases where the configured model is not provisioned, which similarly should not be attributed to the candidate.

• New UPSTREAM_MODEL_UNAVAILABLE enum and policy (terminating, non-retryable, counts toward invalidity but never scored), with a new regex for DeploymentNotFound/model_not_found/Azure deployment-not-found phrases.
• New TRANSIENT_INFRA regex block covering Modal sandbox lifecycle signals, AddTestsDirError, FetchSpec/loading container, and StreamTerminatedError.
• classify_case precedence updated to insert UPSTREAM_MODEL_UNAVAILABLE between AUTH_FAILURE and TRANSIENT_INFRA, with four new regression tests covering the actual production error strings.

Confidence Score: 5/5

Safe to merge — the change only widens the set of signals reclassified from task failure to infrastructure, backed by exact production error strings and covered by new regression tests.

The regex additions are anchored tightly to provider-specific error codes and harness-specific exception names. The new UPSTREAM_MODEL_UNAVAILABLE category is wired correctly into both the policy table and the classify_case precedence chain. The four new tests exercise the actual terminal exceptions from the production incident.

Files Needing Attention: No files require special attention beyond the stale classify_case docstring in error_taxonomy.py.

Important Files Changed

Filename	Overview
vero/src/vero/evaluation/scoring/error_taxonomy.py	Adds UPSTREAM_MODEL_UNAVAILABLE category and two new signal-pattern blocks; precedence ordering in classify_case updated, but the docstring still only cites "budget, then auth" as terminating conditions, omitting the new category.
vero/tests/test_v05_error_taxonomy.py	Four new tests covering harness-environment-loss, upstream-model-unavailable policy and precedence, and the fetchspec/container-load disambiguation; good regression coverage for the production failure strings.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["classify_case(signals)"] --> B["classify_signal per signal"]
    B --> C{Pattern matched?}
    C -- "budget/quota" --> D["INFERENCE_BUDGET_EXHAUSTED"]
    C -- "auth/permission" --> E["AUTH_FAILURE"]
    C -- "deploymentnotfound / model_not_found" --> F["UPSTREAM_MODEL_UNAVAILABLE"]
    C -- "rate-limit / timeout / connection / 5xx" --> G["TRANSIENT_INFRA (existing)"]
    C -- "sandbox / addtestsdirerror / streamterminated / loading container / fetchspec" --> H["TRANSIENT_INFRA (new)"]
    C -- no match --> I["None to TASK_FAILURE"]
    D & E & F & G & H & I --> J["Precedence resolution"]
    J --> K{INFERENCE_BUDGET_EXHAUSTED?}
    K -- yes --> L["terminating, not scored"]
    K -- no --> M{AUTH_FAILURE?}
    M -- yes --> N["terminating, not scored"]
    M -- no --> O{UPSTREAM_MODEL_UNAVAILABLE NEW}
    O -- yes --> P["terminating, counts toward invalidity, not scored"]
    O -- no --> Q{TRANSIENT_INFRA?}
    Q -- yes --> R["retryable, counts toward invalidity, not scored"]
    Q -- no --> S["TASK_FAILURE: informative score at failure value"]

Loading

_{Reviews (3): Last reviewed commit: "Merge pull request #60 from scaleapi/fix..." | Re-trigger Greptile}

[greptile-apps]

Bare sandbox term over-classifies candidate errors as infra

The alternation opens with the unanchored word sandbox, which matches any error string containing that substring — e.g. "Stripe sandbox is not enabled for this account", "Reached rate limit on Sandbox plan", or any SWE-bench task whose ground-truth fix touches sandbox-related code and whose candidate crashes with a message that happens to include "sandbox". Those cases would be silently reclassified from TASK_FAILURE (informative, scored) to TRANSIENT_INFRA (excluded, counted toward invalidity), producing a falsely elevated invalidity fraction and suppressing real signal. A more precise anchor — e.g. modal.*sandbox|sandbox.*(unavailable|not.?found|shut.?down) — targets the three actual Modal lifecycle messages from the PR description without catching unrelated uses of the word.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/evaluation/error_taxonomy.py
Line: 170-174

Comment:
**Bare `sandbox` term over-classifies candidate errors as infra**

The alternation opens with the unanchored word `sandbox`, which matches any error string containing that substring — e.g. `"Stripe sandbox is not enabled for this account"`, `"Reached rate limit on Sandbox plan"`, or any SWE-bench task whose ground-truth fix touches sandbox-related code and whose candidate crashes with a message that happens to include "sandbox". Those cases would be silently reclassified from `TASK_FAILURE` (informative, scored) to `TRANSIENT_INFRA` (excluded, counted toward invalidity), producing a falsely elevated invalidity fraction and suppressing real signal. A more precise anchor — e.g. `modal.*sandbox|sandbox.*(unavailable|not.?found|shut.?down)` — targets the three actual Modal lifecycle messages from the PR description without catching unrelated uses of the word.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[shehabyasser-scale]

Restacked onto the new pr3-harness-bench. The taxonomy module moved to vero/src/vero/evaluation/scoring/error_taxonomy.py in 2da2901; content merged clean, since that file is byte-identical to what this PR was written against and no categories or patterns were added upstream.

One interaction worth surfacing, because it partly undercuts this PR's stated motivation. d949a39 gated infrastructure exclusion on trust:

def _attempt_is_infra(self, attempt, *, trusted: bool) -> bool:
    """...For a competitive (agent) evaluation a candidate-controlled trial's own
    transient-infra exception is not trusted as infrastructure - the candidate
    could emit a timeout/connection error to have its dead attempt excluded
    from the mean. Only trusted evaluations honor that signal."""

Every pattern this PR adds maps to TRANSIENT_INFRA, so under the new gate a dead Modal sandbox during search is still scored as an informative task failure. The line in my description, "a run whose sandboxes all collapsed reports a fully successful 0.0 instead of an invalid aggregate," is now only true for trusted evaluations.

I think the gate's reasoning is right and I am not proposing to weaken it. But the two signal classes are not equivalent:

• TimeoutError / ConnectionError are plausibly candidate-emittable, which is exactly the gaming risk the gate exists for.
• Modal Sandbox with container ID ... not found, AddTestsDirError, FetchSpec failed: loading container and StreamTerminatedError are produced by the harness around the candidate, in the sub-run's own exception_info, at points where the candidate is not running.

So there may be room for a narrow exemption: honour harness-origin sandbox-lifecycle signals even when untrusted, keep the gate for the generic network ones. That is a design call on your gate rather than a fix to mine, so I have left the code alone and am raising it here. If you would rather keep it uniform, this PR is still worth having for the trusted finalization path and for correct diagnostics.

Full suite matches the base branch: 11 pre-existing failures, none new.