fix(eval): drive the agent eval with the OAuth token, and probe it for real

[kovtcharov-amd]

The Agent Eval — Gemma-4-E4B consolidation gate is red on main and on every open PR, and has been for weeks. It is not measuring anything: all five tool_selection scenarios die about three seconds in, which is a launch failure rather than a bad answer. A gate that fails identically everywhere teaches reviewers to ignore it, which is worse than having no gate on the day it catches something real.

The cause is that the account behind ANTHROPIC_API_KEY is out of credit. A sibling job that does not discard its subprocess output says so outright: 400 — Your credit balance is too low to access the Anthropic API. Every other Claude workflow in this repo already prefers CLAUDE_CODE_OAUTH_TOKEN and falls back to the key; this one asked for the key alone.

Closes nothing on its own — #3341 stays open for the stdout-swallowing half (#3375, #3368) — but it should turn the gate green again.

Test plan

• Dispatch this workflow on the branch and confirm the tool_selection scenarios run for their usual 200–680s rather than erroring in ~3s
• Confirm the preflight's live probe passes, and that it fails loudly if the credential is rejected — temporarily point it at a bogus token to check the error is the one a reader can act on
• Confirm rag_quality and context_retention still report their existing embedder diagnosis (they stay blocked on ci(eval): the agent eval gate has never produced a scorecard — embedder will not load on its runner #3016; this PR does not touch them)

Notes for the reviewer

Why the CLI credential is sufficient here. gaia eval agent drives and scores each scenario through claude -p — the score comes back in the --json-schema payload. src/gaia/eval/runner.py never imports ClaudeClient, so the SDK judge is not on this path at all. The email quality evals do use the SDK and are unaffected by this change; they will keep failing until the account is funded.

Why ANTHROPIC_API_KEY is blanked rather than left alongside. runner.py adds --bare only when that variable is set, and --bare restricts Anthropic auth to the key or an apiKeyHelper — OAuth is never read. Setting both would silently keep using the empty account.

The preflight change is the part worth keeping regardless of the credential. It asserted the variable existed. Asserting that a request is accepted — one line, one call — would have turned this into a startup error the first day instead of weeks of a red gate.

[github-actions]

Verdict: Request changes

Switching the eval gate to the OAuth token is the right fix and the reasoning behind blanking the API key checks out against the code it cites. The new live probe, though, is written in a way that breaks exactly when it is supposed to work.

The blocking issue — the probe can't print its own error message. This step runs under PowerShell's "stop on any error" mode, and the probe captures the CLI's error output. In Windows PowerShell that combination makes the very first line of error output kill the step outright, so a rejected credential ends the run with a raw PowerShell stack trace instead of the actionable message the PR adds one line below it. The same step is also now fragile in the other direction: if the CLI prints any harmless warning to its error stream on a successful call, the preflight goes red for no reason — trading one always-red gate for another. This workflow already solves this exact problem elsewhere in the file, so the fix is a few lines and follows an established local pattern.

Worth calling out that the PR's own test plan item — "temporarily point it at a bogus token to check the error is the one a reader can act on" — is the check that would have caught this. Running it before merge is the whole verification here.

Real-world evidence

N/A — CI-workflow-only change; no evidence bundle was produced for this PR and this surface can't be exercised from the review lane. My verdict rests on static review plus reading the credential-selection code the PR's rationale depends on. The only meaningful proof is a dispatch of this workflow on the branch, and all three test-plan boxes are still unchecked — including the bogus-token check that targets the issue above.

🔍 Technical details

🟡 Important

Probe runs under $ErrorActionPreference = "Stop" with 2>&1, so its own ::error:: is unreachable (.github/workflows/test_eval_agent_gemma_consolidation.yml:512)

The step sets Stop at line 393 and never relaxes it at top-level scope. In Windows PowerShell 5.1 (pinned at line 291), merging a native command's stderr into the pipeline under Stop raises a terminating NativeCommandError on the first stderr line. Two consequences:

1. A rejected credential — the case this probe exists for — terminates at line 512, so the actionable message at line 514 never prints.
2. Any benign stderr from claude on a successful invocation also fails the preflight, i.e. a new false-red.

This file already documents the hazard and works around it in the sibling step:

# Deliberately NOT "Stop": the eval's stderr is piped through `2>&1`
# below, and under Stop the first stderr line from a native command
# becomes a terminating NativeCommandError.

— lines 808–811

Invoke-EmbedProbe (lines 403–426) shows the in-file pattern: flip to Continue, capture, restore Stop. Applying it, and asserting output was actually produced so a launch failure can't read as a pass:

          # Assert the credential is ACCEPTED, not merely present. #3341 sat red
          # for weeks because the key existed and the account behind it was out
          # of credit: the preflight passed and all five scenarios then died in
          # three seconds each with their stdout discarded.
          # Continue, not Stop: under Stop the first stderr line from `claude`
          # becomes a terminating NativeCommandError and the ::error:: below
          # never prints. Same reason as the eval step.
          $ErrorActionPreference = "Continue"
          $global:LASTEXITCODE = 0
          $probe = (& claude -p --model $env:EVAL_MODEL "reply with: ok" 2>&1 | Out-String)
          $probeCode = $LASTEXITCODE
          $ErrorActionPreference = "Stop"
          if ($probeCode -ne 0 -or -not $probe.Trim()) {
            Write-Host "::error::The Claude credential is present but a one-line probe was rejected, so every scenario would error. Exit $probeCode. Response: $probe"
            exit 1
          }
          Write-Host "Claude credential accepted by a live probe."

Assigning to $probe directly also beats -OutVariable, which yields an ArrayList that flattens multi-line CLI output onto one line when interpolated.

🟢 Minor

Backticks in the missing-credential message escape unevenly (line 504) — inside a PowerShell double-quoted string, `gaia eval agent` renders as plain gaia eval agent (the backticks are consumed as escapes) while claude -p correctly renders as `claude -p`. Double the first pair for consistency:

            Write-Host "::error::Neither CLAUDE_CODE_OAUTH_TOKEN nor ANTHROPIC_API_KEY is set. ``gaia eval agent`` drives AND scores each scenario through ``claude -p`` (src/gaia/eval/runner.py), so without a credential every scenario errors in a few seconds and the scorecard is meaningless. Add either repository secret, or run the eval locally on AMD hardware."

The probe has no timeout (line 512) — a claude invocation that hangs (auth prompt, network stall) blocks the preflight against only the 450-minute job cap, holding the serialised lemonade-eval pool. A short bounded wait would fail fast.

Same dead account still gates test_eval_rag.yml (.github/workflows/test_eval_rag.yml:57) — it drives gaia eval agent on ANTHROPIC_API_KEY alone, so it stays red for the credit-balance reason this PR diagnoses. Out of scope here, but worth the same two-line treatment as a follow-up.

Strengths

• The --bare rationale is correct, not assumed: src/gaia/eval/runner.py:949 appends --bare only when ANTHROPIC_API_KEY is set, and --bare restricts auth to the key — so leaving both set really would pin the eval to the empty account. Blanking the key is the right call, and the comment is duplicated at both use sites so neither can drift alone.
• The secrets.X == '' && secrets.Y || '' selection degrades correctly in all three cases (OAuth present, key-only, neither), and the neither-case is caught by the explicit guard rather than silently producing an empty credential.
• New run: body lines respect this file's pure-ASCII constraint (lines 284–290) — easy to violate given the em dashes elsewhere in the file's name: fields.
• Asserting the credential is accepted rather than present is the durable half of this change, independent of which secret ends up funded.

[kovtcharov-amd]

Verified on this PR's own run, and the credential half is fixed.

Claude credential accepted by a live probe.
  CLAUDE_CODE_OAUTH_TOKEN: ***
  ANTHROPIC_API_KEY:

The scenarios now reach the agent instead of dying at the door. Before, all five errored in 3 seconds; now they run 14–18s, and multi_step_plan ran for 993 seconds before timing out — that is a scenario genuinely executing, in the 200–680s band #3341 recorded for real runs.

The gate is still red, on a different and deeper cause. Four scenarios return INFRA_ERROR at 14–18s and one times out. The backend comes up healthy first, so this is past auth and past startup.

And the new failure is still invisible for the same reason the old one was. The runner prints the subprocess's stderr while claude -p reports on stdout, so all we can see is the shape. That makes #3375 the critical path now rather than a nice-to-have — until it lands, the next person gets to guess from timings the way this one did.

I would still merge this: it removes one real blocker, it is what every other Claude workflow in the repo already does, and the live probe means the next credential failure announces itself in the preflight instead of thirty minutes later as five mystery errors.

[kovtcharov-amd]

Second fix pushed, for the cause the first one exposed.

With the credential working, the scenarios got far enough to say what was actually wrong — the trace artifacts carry the agent's own diagnosis, identical across all four:

The gaia-agent-ui MCP server failed to connect (CONNECTION_CLOSED) before Phase 1 could begin. system_status() was not callable, so no session could be created and no turns were executed.

The eval's MCP config launched that server with uv run, which resolves the project environment rather than the active one. On a checkout without a synced .venv it creates an empty one, and the server dies before writing a single protocol frame. A stdio server that exits is CONNECTION_CLOSED to the client.

Reproduced directly rather than inferred:

$ uv run python -m gaia.mcp.servers.agent_ui_mcp --stdio
Creating virtual environment at: .venv
...Error while finding module specification for 'gaia.mcp.servers.agent_ui_mcp'
(ModuleNotFoundError: No module named 'gaia')     # exit 1

It now launches with python, inheriting the environment the eval is already running in — the one that has GAIA installed.

That also explains why this looked environment-specific and was not: the config is checked in, so every runner and every branch hit it the same way.

[kovtcharov-amd]

Third run, and the gate is measuring for the first time. Not green yet, and the residual is a different shape of problem.

Two scenarios ran end to end and produced real scores — 9.93 PASS and 3.08 FAIL — where every scenario previously died before Phase 1. The 3.08 is a genuine product finding rather than infrastructure: the agent reports that search_file excludes .md from its default scope and never scans the eval corpus directory. That is exactly the kind of signal this gate exists to produce and has not produced once in weeks.

Three still report CONNECTION_CLOSED. So the uv run fix was necessary but not sufficient: the MCP server now starts, and sometimes stays up. Intermittent rather than absolute points at a startup race or contention on the runner, not a config error — a checked-in config fails identically every time, and this no longer does.

Where that leaves things, honestly:

• The two fixes here are both verified and worth merging on their own. Auth was genuinely broken; the uv run launch was genuinely broken; each was invisible behind the other.
• The remaining flake needs someone who can watch the runner while it happens. fix(eval): capture stdout when a scenario subprocess fails #3375 would help — the MCP server's own stderr is still discarded, so all anyone can see is that the connection closed, not why.
• Until this merges, the other five open PRs keep showing the same red gate, because the fix has to be on their branch or on main.

I have not merged it; that is a call for a human.

[kovtcharov-amd]

Correction: I pushed a third fix on a warm-up hypothesis, the next run contradicted it, and I have reverted it. Recording that here so nobody re-derives it.

The hypothesis was that the MCP server's first launches were slow rather than broken — scenarios 1-3 had failed at 19s/17s/36s while 4 and 5 completed in 197s and 336s, which reads as a warm-up curve. So I pre-warmed the import and set MCP_TIMEOUT=120000.

The next run failed all five at 15-22s. That is the disproof: had the wider startup window been in effect, a slow server would have been given 120 seconds before being dropped. Failing at the same ~15-20s means the server is exiting, not timing out — and the run where two scenarios passed was variance, not warming.

So this PR is back to the two fixes that are each independently reproduced:

1. Authenticate with CLAUDE_CODE_OAUTH_TOKEN — verified by the live probe passing and by scenarios reaching the agent at all.
2. Launch the MCP server with python rather than uv run — reproduced directly on this machine, where uv run creates an empty venv and exits with ModuleNotFoundError: No module named 'gaia'.

What is left is a server that starts and then exits, and its stderr is discarded, so there is nothing to read. That is #3375's fix, and it is now the blocker rather than a convenience — I have spent three runs inferring from timings what one line of captured stderr would have said outright.

[kovtcharov-amd]

Root cause found, and it is one line.

=== MCP server launch 2026-09-05T15:55:42+00:00
    python=C:\actions-runner\_work\gaia\gaia\.venv\Scripts\python.exe ===
ModuleNotFoundError: No module named 'mcp'

The job installed -e .[dev,eval,ui,api]. mcp is an optional extra and was not in that list — while every scenario in this gate is driven through the Agent UI MCP server. The server exited before writing a protocol frame, and the client reported that as CONNECTION_CLOSED.

That is why it failed identically on main and on every PR, why re-running never helped, and why it looked like a runner fault: a missing dependency is perfectly deterministic.

Four commits, each of which had to come before the next could be seen:

1. Authenticate with the OAuth token. The API account is out of credit. Until this, nothing ran at all.
2. Launch the MCP server with python, not uv run. uv run built an empty venv and the server died with a different ModuleNotFoundError, for gaia.
3. Keep the server's stderr. It is a grandchild — the runner starts claude -p, which starts the server — so nothing was holding its output. fix(eval): capture stdout when a scenario subprocess fails #3375 captures the scenario subprocess and would not have reached it.
4. Install the mcp extra. The actual fault, visible only once (3) was in place.

I also pushed and reverted a warm-up hypothesis in between; the next run disproved it and I took it out rather than leave a speculative change in the PR.

Worth keeping regardless of this gate: the preflight now proves the credential is accepted rather than merely present, and the launcher means the next MCP failure explains itself instead of costing five runs of inference.

[kovtcharov-amd]

The gate is measuring. With the mcp extra installed the server starts cleanly — zero ModuleNotFoundError, and CONNECTION_CLOSED is gone from every scenario.

scenario	before	now
no_tools_needed	INFRA_ERROR	PASS 9.9
known_path_read	INFRA_ERROR	FAIL 6.9
smart_discovery	INFRA_ERROR	FAIL 3.7
data_vs_recall_disambiguation	INFRA_ERROR	SETUP_ERROR
multi_step_plan	INFRA_ERROR	SETUP_ERROR

Three of five now produce real scores against the baseline, where the gate had measured nothing at all.

The last two are a path guard doing its job. Indexing refuses the eval corpus: "Access denied: path must be within home directory (C:\Windows\System32\config\systemprofile)". The runner's service account has that as HOME, while the corpus lives under the workspace.

I would not take the fix the agent suggests. Both traces recommend widening the allowlist in src/gaia/ui/server.py / routers/documents.py. That weakens a real containment guard in shipped product code to suit a test's directory layout — the guard is right and the environment is wrong. The fix belongs in the eval setup: give the backend a HOME that contains the corpus, or stage the fixtures inside the allowed root before the run. Worth its own issue and its own review rather than being folded in here.

Scope-wise this PR is done: it takes the gate from measuring nothing to measuring three of five, and the residual is a separate, well-understood environmental fault.