fix(eval): capture stdout when a scenario subprocess fails

[kovtcharov-amd]

A failing eval run told you nothing. Every scenario reported ERRORED with an empty error field, so triaging a CI failure meant re-running the eval locally and hoping it reproduced.

The runner read only stderr — but --output-format json puts the CLI's own error on stdout, so stderr is routinely empty on exactly the failures worth reading. It now reports both streams, and when a process dies silently it names the exit code instead of storing an empty string.

Test plan

• PYTHONPATH=$(pwd)/src python -m pytest tests/test_eval.py -q — 143 pass
• Check out main, copy tests/test_eval.py over, re-run: the three new cases fail (test_nonzero_exit_captures_stdout, test_nonzero_exit_captures_both_streams, test_nonzero_exit_error_never_empty)
• python -m black --check and python -m isort --check-only on both files — clean

[github-actions]

Verdict: Approve

A failing eval scenario now tells you why it failed. The runner only read stderr, but the Claude CLI in JSON mode writes its own error to stdout — so every failed scenario recorded an empty error and CI failures had to be reproduced locally to triage. This reports both streams, and names the exit code when a process dies silently.

The change is small, the three new tests pin each branch (stdout-only, both streams, and the never-empty fallback), and the before/after evidence shows the real fix landing. Nothing blocking.

One thing worth a follow-up, not a merge blocker: a scenario that times out still records no explanation at all, which is the same triage gap this PR closes for the crash path.

Real-world evidence

The PR ships an evidence bundle, and the fix is exercised through the real gaia eval agent command against a live backend on 127.0.0.1:4200 (/api/health → HTTP 200). Same command run at HEAD~1 and HEAD:

Before (HEAD~1):
[ERROR] concise_response — exit code 1
                                          ← stderr line printed empty
{ "status": "ERRORED", "error": "" }

After (HEAD):
[ERROR] concise_response — exit code 1
stdout: {"type":"result","subtype":"error_during_execution","is_error":true,"result":"Not logged in - please run /login"}

All four branches were exercised live — stdout-only/exit 1, both streams/exit 2, silence/exit 3 (no output on either stream (exit 3)), and the untouched success path (PASS 9.0/10, 1/1 passed (100%)). The JUnit consumer of the changed field was checked too, and now carries the captured text instead of a blank <error> body. Spot regression poked --compare, --audit-only, and three sibling backend routes — all clean. pytest tests/test_eval.py::TestRunScenarioSubprocess -q → 20 passed.

The bundle is explicit that no real LLM turn ran (no Lemonade/GPU on this runner) and that a scorecard-vs-baseline comparison is pending strix-halo lane, and that the failing claude subprocess was a PATH stand-in — everything downstream of it (the CLI, runner, backend, trace files) was real. That's the right shape for this change: the code under test is the failure-handling path, which the stub reaches faithfully. Agent UI screenshot is correctly N/A — this writes a field consumed by CI, not the UI. The verdict rests on this evidence plus static review; nothing here contradicts the fix.

🔍 Technical details

Issues

🟢 Minor — a timed-out scenario still records no explanation (src/gaia/eval/runner.py:1096)

The TimeoutExpired branch builds a result dict with no error key at all, so a TIMEOUT trace has exactly the triage problem this PR fixes for the non-zero-exit path: the console prints [TIMEOUT] … exceeded Ns but the persisted trace/JUnit carries nothing. Pre-existing, and adjacent enough to be worth a one-line fix while the file is open:

    except subprocess.TimeoutExpired:
        elapsed = time.time() - start
        print(f"[TIMEOUT] {scenario_id} — exceeded {timeout}s", file=sys.stderr)
        result = {
            "scenario_id": scenario_id,
            "status": "TIMEOUT",
            "overall_score": None,
            "turns": [],
            "error": f"subprocess exceeded {timeout}s timeout",
            "elapsed_s": elapsed,
            "cost_estimate": {"turns": 0, "estimated_usd": 0.0},
        }

🟢 Minor — the JSON-parse failure path still reads stdout only (src/gaia/eval/runner.py:1091)

error: f"JSON parse error: {e}. stdout: {proc.stdout[:300]}" drops stderr. This is the exit-0-but-garbage case, so stderr is the more likely place a warning explaining the malformed output lives. Lower value than the timeout one — the parse error itself plus 300 chars of stdout is usually enough — but the same detail construction would cover it.

Strengths

• The comment earns its place. Two lines naming the non-obvious invariant (--output-format json puts the error on stdout) is exactly the WHY-only style CLAUDE.md asks for — no history dump, no issue-number tagging.
• Per-stream truncation, not a shared budget. Slicing each stream to 500 chars independently means a chatty stderr can't crowd out the stdout payload that actually explains the failure — a subtle improvement over the single-slice original that the evidence bundle calls out explicitly.
• The tests pin behaviour, not implementation. test_nonzero_exit_error_never_empty asserts the invariant ("must say something") rather than the exact string, and the _run helper gained a stderr parameter with a default so all 17 existing cases are untouched. The docstrings say what regression each guards.
• No silent-fallback violation. The or f"no output on either stream (exit {N})" branch is the opposite of a silent default — it converts a silent failure into a named one.