feat(eval): add Claude Code CLI agent harness by Rodriguespn · Pull Request #47 · supabase/evals

Rodriguespn · 2026-06-18T16:25:07Z

Adds claude-code as an eval agent alongside ai-sdk, so scenarios run through a real agent harness — the Claude Code CLI drives its own loop, tools, and MCP client — rather than a bare model loop. Its transcript is parsed back into the same surface scorers already use, so scoring is unchanged. The framework is generic: a runner + a parser + a registry entry adds the next agent (Codex follows in #49).

How it runs

The agent runs in a Docker sandbox in both modes. The base sandbox carries the agent's own tooling (shell/file tools) and skills, but not the Supabase CLI. The only thing that varies by mode is the Supabase local stack:

local-stack mode — the Supabase CLI is installed at setup and the local stack is started (supabase start); the agent works with the CLI + its own tools, scored against the running stack.
tools mode — no Supabase CLI and no local stack; the agent works through the Supabase MCP against platform-lite (the mocked hosted platform), with its shell/file tools still available.

Main difference: tools mode has no Supabase CLI (or local stack) — it's MCP-driven by environment, not by a tool restriction.

Live runs

Closes AI-848

linear-code · 2026-06-18T16:25:11Z

AI-848 Add claude-code agent support to the eval framework

Add a claude-code agent option (claudeCodeAgent() AgentHarness in packages/core/src/index.ts) that shells out to the Claude Code CLI, alongside the existing ai-sdk agent.

Review in Linear

vercel · 2026-06-18T16:25:14Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored	Preview	Jun 23, 2026 11:47pm

…brittle test check) (#50) Fixes false-negative eval scoring found while auditing [claude-code-haiku-4.5](https://github.com/supabase/evals/actions/runs/27825575802/job/82349020404) and [claude-code-sonnet-4.6](https://github.com/supabase/evals/actions/runs/27825575802/job/82349020420) runs. ### Fixed - seed never applied in `investigate-auth-001-deleted-user-access` and `investigate-realtime-001-subscribed-no-events` because their seed was still under the old `seed/` driectory (renamed to `remote/`. So both ran against an empty DB (scorer threw `relation "profiles" does not exist`; the realtime judge penalized the agent for rebuilding the scenario it was forced to recreate). - `build-cli-002`'s "supabase db diff used" check matched `tc.endpoint === "bash"`, but Claude Code emit `"Bash"` instead. Lowercased `tc.endpoint` - `discoverStackConfig()` read the API keys once and threw if absent. Now retries briefly (5×2s) before failing. - The agent wrote passing tests and correctly diagnoses `posts` still failed it, because the "negative tests catch the bug (`failedTests > 0`)" check hard-coded one test style contradicted the diagnosis judge. Dropped it and kept only the judge assessment. - the mock edge runtime's `require` shim only resolved `@supabase/supabase-js` and threw "import not supported" for `npm:`/`node:`/`esm.sh` imports the real Deno runtime accepts (`build-functions-004` used `npm:jwt-decode`). Now resolves `node:` builtins, `npm:`/`jsr:`/CDN URLs/bare specifiers (version-stripped) against the runtime's modules; unresolvable packages error clearly instead of being rejected. Verified via `bootPlatformBackend` (`npm:zod` + `node:crypto` → 200). **Notes** These are framework/eval-definition fixes, independent of the Claude Code feature work (#47). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add `claude-code` as an agent option alongside `ai-sdk`. Unlike the in-process `aiSdkAgent` loop, a CLI agent is its own harness (own tools, loop, MCP client), so it runs inside the local-stack Docker sandbox and its transcript is parsed after the run rather than intercepted live. - Generic, extensible transcript layer in core (transcript vocabulary + per-agent parsers + registry), ported/adapted from @supabase/agent-evals. Adding Codex/Gemini/OpenCode/Cursor is one spec + one parser + one registry entry. - adaptTranscript bridges canonical TranscriptEvent[] to the existing TranscriptPart[]/ToolCallRecord[] surface, so scorers, serializeTranscript, and judges keep working unchanged. - createCliAgent orchestration: install pinned CLI in the sandbox, translate the MCP server map into the CLI's own config (rewriting loopback hosts to host.docker.internal), run it, capture + parse the session JSONL. - claudeCodeAgent factory + experiments/claude-code.ts. - Wire AgentRunArgs.sandbox, AgentHarness.requiresSandbox, and LocalStackSession.sandbox; run-eval passes the sandbox in local-stack mode and skips tools-mode evals for CLI agents. CLI agents run only against local-stack evals in this phase (tools-mode is skipped with a clear message). Verified e2e on Haiku against build-tests-001-rls-tenant-isolation (CLI installed, ran its own loop, transcript parsed into 14 tool calls + 23 transcript parts, scorer consumed it). Unit tests cover the parser + adapter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…iku experiment - Source the model-id type from the official Anthropic SDK (`Model` from `@anthropic-ai/sdk/resources/messages`) instead of a free `string`, so model ids autocomplete and validate (the SDK's `(string & {})` fallback still allows newer ids). `CliAgentSpec`/`createCliAgent` are now generic over the agent's model type with a `defaultModel`; new CLI agents bring their own SDK model type. - `claudeCodeAgent({ model? })` defaults to the spec's model. - Add experiments/claude-code-haiku.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Per the Claude Code headless/CLI docs: - Capture the transcript from `--output-format stream-json --verbose` on stdout instead of reading the on-disk session JSONL — Anthropic's recommended programmatic path (no session-file race, no escaped-cwd guessing). The parser already handles these event envelopes. CliAgentSpec.exec now returns { command, raw }; the disk-read captureTranscript is gone. - Add `--strict-mcp-config` so only our `--mcp-config` servers load (ignore any .mcp.json in the workspace), and tag MCP servers `"type": "stdio"`. - Derive stoppedReason from the final `result` event's subtype (success → stop, error_max_turns etc. surfaced verbatim), falling back to the process exit. - Keep `--print`, `--append-system-prompt` (preserve Claude Code's default agent prompt), `--model`, and `--dangerously-skip-permissions` (sanctioned for isolated containers). Also rename experiments to include the model version: claude-code-sonnet-4.6.ts and claude-code-haiku-4.5.ts. Add a deriveStopReason unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CLI agents previously skipped tools-mode evals. Now they run there too, in a minimal sandbox, confined to the MCP surface so they can't bypass the tools the eval measures. - sandbox: createBareSandbox() — a bare Node container (no Supabase stack) wrapped as an AgentSandbox; reaches host-side platform-lite via host.docker.internal. - core: AgentRunArgs.restrictToMcp confines a CLI agent to the MCP tool surface; EvalSessionArgs.hostname binds platform-lite to 0.0.0.0 so the sandbox's in-container MCP servers can reach it. cli-agent derives an allowlist (mcp__<server>) and, when restricted, runs Claude Code with --allowedTools and no --dangerously-skip-permissions (print mode auto-denies non-allowed tools). rewriteLoopback also maps 0.0.0.0 -> host.docker.internal. - Pass the prompt on stdin instead of a positional arg: `--allowedTools` is variadic and would otherwise swallow the trailing prompt (caused an exit-1, empty-transcript run). Verified against claude 2.1.x. - harness: tools mode boots a bare sandbox + binds 0.0.0.0 + sets restrictToMcp when the agent requiresSandbox; removed the old tools-mode skip. Verified e2e: claude-code-haiku-4.5 on investigate-db-001 passed 3/3 using only mcp__supabase-mcp__* tools (no native Bash/Read), confirming the restriction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

From the branch review: - Move tool-permission/allowlist construction out of the generic createCliAgent into the runner (spec.exec). The generic layer now passes restrictToMcp + mcpServerNames; each runner translates that into its own permission flags and MUST throw if it can't enforce confinement (so a future Codex/Gemini can't silently run unrestricted in tools mode). Removes the Claude-specific `mcp__<server>` allowlist format from the framework. - Pass the system prompt via --append-system-prompt-file instead of "$(cat ...)" — removes the ARG_MAX / shell-expansion surface (verified the flag against claude 2.1.x). - Tighten rewriteLoopback to rewrite only a URL authority (after ://), not any loopback-looking substring (could have clobbered a token/ref/password). - Dedup: shared isRecord + parseJsonlRecords (new json.ts) used by the parser, cli-agent stop-reason scan, and index; alias SandboxCommandResult to core's CommandResult so the identical shapes can't drift. Deferred (documented for when the 2nd agent lands): full runner/parser/agent registry split with a captureTranscript hook for disk-only CLIs, positional tool-call/result pairing fallback for non-id'd transcripts (Codex), and cross-package shellQuote dedup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Restructure the CLI-agent framework so each per-agent concern lives in one place — prep for adding more agents (Codex, Gemini, …) with minimal new code. - runners/ (new): AgentRunner interface + shared helpers (writeSandboxFile, npmInstallGlobal, findNewestFile, rewriteLoopback, processStopReason, shellQuote, scratch paths). runners/claude-code.ts owns all CLI-shaped decisions: install, exec flags, permission model, MCP-config format/placement. - parsers/ (existing): unchanged role — raw transcript → canonical events. - cli-agent.ts: now just createCliAgent(runner, parser, options) composition + the claudeCodeAgent factory. The Claude-specific MCP-config/permission logic that lived in the generic layer moved into the runner. - AgentRunner.canRestrictToMcp + AgentHarness.supportsToolsMode: a CLI that can't be confined to the MCP surface reports supportsToolsMode=false and the harness skips tools-mode evals for it (Claude Code supports it). Adding an agent is now a runner + a parser + a registry entry. Behavior is unchanged for Claude Code; core tests 8/8, typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…extractor - createBareSandbox now passes DockerSandboxOptions through to DockerSandbox.create (image/timeout/network) and its comment no longer references the removed NET_ADMIN/DNAT mechanism (AI-844 replaced it with host networking). It stays a thin composition of DockerSandbox.create + toAgentSandbox — the primitives we already have. - shared/normalize.ts is now just the normalization *algorithm* + the AgentToolMap shape; the Claude Code tool-name map moved into parsers/claude-code.ts. Each agent owns its mapping and passes it in; shared uses it but doesn't define it (and there's no shared↔parser import cycle). - Removed the unused toISO from shared/extract.ts. The remaining extractors are bespoke arg-shape lookups with no sensible native/npm replacement. Typecheck clean; core tests 8/8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ug publish copy Branch dispatches (ref != main) intentionally skip the commit/refresh-PR steps, so they previously produced no eval-results.json artifact. Upload it on workflow_dispatch too. Also log the downloaded-artifacts and reconstructed results/ trees in the publish step to diagnose 'no result files matched'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ng in both modes Make the CLI agent's environment identical across modes except the Supabase local stack — the one real difference. - New createAgentEnvironment(sandbox pkg): the single builder — image + DockerSandbox.create (host net iff localStack) + optional setupSupabaseSandbox + installSkills. `localStack` present → local-stack mode; omitted → tools mode. localStackRuntime and createBareSandbox are now just configurations of it, so adding/removing an environment component happens in one place. - Skills now install in BOTH modes. A CLI agent reads SKILL.md with its file tools in tools mode too (bare sandbox uses the same image, which carries the skills CLI); createBareSandbox returns the skills-discovery promptAddendum. In-process (ai-sdk) agents keep the load_skill tool (no filesystem). - Drop the MCP-only restriction: CLI agents get their full native toolset in both modes (the sandbox is the isolation boundary). Removes restrictToMcp (AgentRunArgs/RunnerExecArgs/claude runner), canRestrictToMcp (AgentRunner), supportsToolsMode (AgentHarness) and the tools-mode skip. Claude Code always runs --dangerously-skip-permissions. Tradeoff: tools mode no longer isolates MCP usage for CLI agents; with no local stack and a state-based scorer this is low-risk, and strict MCP-only can return later as an opt-in env component. core/sandbox/framework typecheck clean; core tests 8/8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e base image Move the Supabase CLI out of the Dockerfile and install it at setup time, only in local-stack mode. The base image now carries just the common agent tooling (node, git, psql, curl, the skills CLI), shared by both modes; tools-mode sandboxes genuinely lack the Supabase CLI. This makes the running stack + the CLI the *only* difference between the two environments — exactly the boundary. - Dockerfile: drop the supabase `.deb` install + CLI_VERSION arg. - ensureSupabaseSandboxImage(): no longer keyed by CLI version (tag is base-skills-<v>); the image is reused across modes and CLI versions. - New installSupabaseCli(sandbox, cliVersion); setupSupabaseSandbox runs it first (everything below it needs the CLI). createAgentEnvironment threads cliVersion through only on the localStack path. Tradeoff: the CLI `.deb` now downloads per local-stack session instead of once at image build. Marginal next to `supabase start`; if it ever matters, a local-stack image variant FROM the base would restore build-time caching. sandbox/framework typecheck clean; core tests 8/8. (docker integration test unchanged — it installs the CLI via setupSupabaseSandbox before using it.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…h upload Single-experiment dispatches are a test-only path; their publish step is expected to fail (real refreshes run multiple experiments, where download-artifact nests per artifact). Revert the single-artifact flat-layout handling and the debug logging added while diagnosing. Keep only the workflow_dispatch artifact upload of eval-results.json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…atrix Both the workflow_dispatch default and the PR (run-evals label) default now run claude-code-haiku-4.5 and claude-code-sonnet-4.6 alongside the openai experiments, so labelling the PR refreshes eval-results.json with Claude Code runs too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The Vercel project's Ignored Build Step was cancelling preview builds even for commits that change apps/web (including eval-results.json refreshes), so the branch preview was stale. Add a vercel.json ignoreCommand (the documented monorepo pattern) that proceeds only when apps/web changed — so results refreshes (and app changes) deploy a fresh preview. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The Option-B commit re-introduced `import { runScorer } from "../lib/scorer.js"` during conflict resolution, but later commits on this branch removed that usage and `lib/scorer.js` does not exist here — it broke `pnpm typecheck`. Remove it and fold `createBareSandbox` into the existing `@supabase-evals/sandbox` import. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The comments claimed Claude Code "only runs against local-stack evals; tools-mode evals are skipped" and pointed at a nonexistent claude-code.ts. Since the bare-sandbox work, CLI agents run in BOTH modes (full sandbox for local-stack, bare sandbox + MCP for tools). Describe that accurately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`requiresSandbox` was misleading: in local-stack mode even an in-process ai-sdk agent operates a sandbox (via its tools). The bit actually means "the agent itself runs inside the sandbox" — it brings its own harness (loop + tools + MCP client) and needs a container to run in, as every CLI agent does. Rename the flag and the derived `cliAgent` local (→ agentRunsInSandbox) to match, and expand the doc comment to state that in-process agents leave it false (no sandbox booted in tools mode; local-stack always has one regardless). Pure rename — behaviour unchanged. Verified with a live claude-code-haiku-4.5 tools-mode run (bare sandbox + MCP), 3/3 checks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The local-stack scorer's `query()` hardcoded the DB at 127.0.0.1:54322, but the DB port is agent-controlled (supabase/config.toml) — an agent that remaps ports to dodge a conflict left psql connecting to a dead port, false-failing every SQL-based check while the host-side supabase-js checks (which already derive their endpoint from `supabase status`) passed. Read `DB_URL` from `supabase status -o json` instead, mirroring how getClient resolves API_URL/keys. DB_URL is reported as soon as the database is up (it doesn't wait on gotrue), so DB-only evals aren't coupled to auth readiness. Verified: build-cli-001 now scores 7/7 with claude-code-haiku-4.5 (previously 4/7 with three psql checks false-failing on a remapped port). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The Dockerfile test still asserted the Supabase CLI was installed in the image (ARG CLI_VERSION + the release .deb), but that install was relocated to installSupabaseCli when the CLI became a local-stack component — so the test was red on this branch. Assert the new reality (CLI-free base with common tooling; the CLI pin lives in SUPABASE_CLI_VERSION). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`shared/extract.ts` hardcoded the union of agents' arg-key guesses (args.path ?? args.file_path ?? args.filePath ?? …), so the shared layer knew harness-specific field names — and the extracted values were then stripped by adapt before any consumer saw them (dead code). Restructure so the shared extractor only extracts and each harness's parser owns its keys, mirroring how AgentToolMap already works for tool names: - shared/extract.ts: generic extractArgs(args, ArgFieldMap) — no hardcoded keys. - claude-code.ts: declares CLAUDE_CODE_ARG_FIELDS (file_path/notebook_path, command, url) and hands it in. - Normalized path/command/url now live on the canonical TranscriptEvent.tool and are surfaced on ToolCallRecord (no longer stripped), so scorers can read a call's path/command/url without knowing the harness's raw keys. Adding a harness (Codex, Gemini CLI) = its runner + parser + tool map + arg-field map; shared, adapt, and scorers stay unchanged. Verified: core tests 8/8 and a live claude-code-haiku-4.5 run (parse→adapt→score intact; MCP calls correctly yield no normalized fields). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

vitest was the one shared dev dependency left out of the pnpm catalog, and the workspace had drifted across majors — apps/framework on ^4.1.5, core/sandbox/ platform-lite on ^3.1.3 — so two vitest majors were installed at once. Add `vitest: ^4.1.5` to the catalog and switch every workspace package to `catalog:`, standardizing on v4 (the newest, already used by apps/framework). The lockfile now resolves a single vitest (4.1.8). Verified on v4: core 8/8, sandbox 25/25, platform-lite 35/35, and `pnpm typecheck` clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Address PR #47 review (Matt) plus self-review findings: - parsers/claude-code: dedup the terminal `result` line against the streamed final assistant message so the closing turn isn't doubled; still emit it in plain `--print` mode. Adds regression tests. - run-eval: convert cliSandbox/session/hostedBackend to `await using` via a `disposable()` adapter, removing the try/finally blocks. Fixes the leak where a throw in startSession skipped earlier-resource cleanup. - runners/shared: remove unreferenced `findNewestFile` (dead code). - parsers/claude-code: fix docstring — parses stream-json on stdout, not the on-disk session file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds `codex` (OpenAI Codex) as an eval agent on the runner/parser framework from #47: a runner + a parser + a registry entry + a factory + an experiment. Orchestration is unchanged; Codex's transcript is parsed into the same surface scorers already use. - runners/codex.ts: install + `codex login --with-api-key` (key via stdin, never argv), `codex exec --json` (--skip-git-repo-check --dangerously-bypass-approvals-and-sandbox -m <model>, prompt = system+user on stdin), MCP via a generated ~/.codex/config.toml. Model type from openai's ChatModel widened with (string & {}), exported as CodexModel. - runners/codex.ts deriveStopReason: `codex exec` exits 0 even on a failed turn, so the stop reason comes from the terminal turn.completed/turn.failed event, not the exit code — a clean stop and an agent failure are distinct. - parsers/codex.ts: the thread/turn/item event schema; paired tool_call/ tool_result by item id; owns its CODEX_TOOLS map; normalized command/path via the shared extractArgs (raw args left untouched). Tool success is tri-state — a missing status is unknown, not auto-success. - Registered in parsers/registry.ts; codexAgent factory; experiments/codex-gpt-5.4.ts. Adds the openai SDK as a compile-time dep. Codex always has shell access and no per-tool allowlist, so it can't be confined to the MCP-only surface — it runs local-stack evals only. Model note: Codex's tool_search isn't supported by gpt-5.4-nano (API 400); use gpt-5.4-mini or larger. Verified e2e: codex/gpt-5.4 x build-tests-001 → 3/3, stoppedReason "stop" (from turn.completed), 27 tool calls incl. a search_docs MCP call. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds `codex` (OpenAI Codex) as an eval agent on the runner/parser framework from #47: a runner + a parser + a registry entry + a factory + an experiment. Orchestration is unchanged; Codex's transcript is parsed into the same surface scorers already use. - runners/codex.ts: install + `codex login --with-api-key` (key via stdin, never argv), `codex exec --json` (--skip-git-repo-check --dangerously-bypass-approvals-and-sandbox -m <model>, prompt = system+user on stdin), MCP via a generated ~/.codex/config.toml. Model type from openai's ChatModel widened with (string & {}), exported as CodexModel. - runners/codex.ts deriveStopReason: `codex exec` exits 0 even on a failed turn, so the stop reason comes from the terminal turn.completed/turn.failed event, not the exit code — a clean stop and an agent failure stay distinct. - parsers/codex.ts: the thread/turn/item event schema; paired tool_call/ tool_result by item id; owns its CODEX_TOOLS map; normalized command/path via the shared extractArgs (raw args left untouched). Tool success is tri-state — a missing status is unknown, not auto-success. - Registered in parsers/registry.ts; codexAgent factory; experiments/codex-gpt-5.4.ts. Adds the openai SDK as a compile-time dep. Like Claude Code, Codex runs in BOTH modes: the sandbox carries its shell/file tools either way, and tools mode just drops the Supabase CLI + local stack so Supabase access goes through MCP. Which mode an eval uses is a property of the eval (interface/local dir), not the agent. Model note: Codex enables a `tool_search` tool that gpt-5.4-nano rejects with `400 invalid_request_error: "Tool 'tool_search' is not supported with gpt-5.4-nano."`, failing the turn immediately — use gpt-5.4-mini or larger. Verified e2e (CLI 0.138, gpt-5.4): local-stack build-tests-001 -> 3/3, and tools-mode investigate-db-001 -> 3/3 using MCP (execute_sql / list_tables / list_projects), both stoppedReason "stop". gpt-5.4-nano reproduces the tool_search 400 -> stoppedReason "error". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds `codex` (OpenAI Codex) as an eval agent on the runner/parser framework from #47: a runner + a parser + a registry entry + a factory + experiments. Orchestration is unchanged; Codex's transcript is parsed into the same surface scorers already use. - runners/codex.ts: install + `codex login --with-api-key` (key via stdin, never argv), `codex exec --json` (--skip-git-repo-check --dangerously-bypass-approvals-and-sandbox -m <model>, prompt = system+user on stdin), MCP via a generated ~/.codex/config.toml. Model type from openai's ChatModel widened with (string & {}), exported as CodexModel. - runners/codex.ts deriveStopReason: `codex exec` exits 0 even on a failed turn, so the stop reason comes from the terminal turn.completed/turn.failed event, not the exit code — a clean stop and an agent failure stay distinct. - parsers/codex.ts: the thread/turn/item event schema; paired tool_call/ tool_result by item id; owns its CODEX_TOOLS map; normalized command/path via the shared extractArgs (raw args left untouched). Tool success is tri-state — a missing status is unknown, not auto-success. - Registered in parsers/registry.ts; codexAgent factory; experiments for gpt-5.4 and gpt-5.5. Adds the openai SDK as a compile-time dep. - eval-refresh workflow: run codex-gpt-5.4 + codex-gpt-5.5 alongside the claude-code experiments. Like Claude Code, Codex runs in BOTH modes: the sandbox carries its shell/file tools either way, and tools mode just drops the Supabase CLI + local stack so Supabase access goes through MCP. Which mode an eval uses is a property of the eval (interface/local dir), not the agent. Model note: Codex enables a `tool_search` tool that gpt-5.4-nano rejects with `400 invalid_request_error: "Tool 'tool_search' is not supported with gpt-5.4-nano."`, failing the turn immediately — use gpt-5.4-mini or larger. Verified e2e (CLI 0.138, gpt-5.4): local-stack build-tests-001 -> 3/3, and tools-mode investigate-db-001 -> 3/3 using MCP (execute_sql / list_tables / list_projects), both stoppedReason "stop". gpt-5.4-nano reproduces the tool_search 400 -> stoppedReason "error". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…am (#49) Adds **`codex`** (OpenAI Codex) as an eval agent alongside `claude-code` and `ai-sdk`, built on the runner/parser framework from [#47](#47): a runner + a parser + a registry entry. The orchestration is unchanged and Codex's transcript is parsed back into the same surface scorers already use, so scoring is unchanged. Codex runs headless via **`codex exec --json`** in a Docker sandbox. Like Claude Code, it runs in **both** modes — the sandbox carries its shell/file tools either way; tools mode just drops the Supabase CLI + local stack, so Supabase access goes through **MCP**. Which mode an eval uses is a property of the eval (`interface`/`local/` dir), not the agent. - Install + `codex login --with-api-key` (key piped via stdin, never in argv). - Prompt (system + user) on stdin; `--skip-git-repo-check --dangerously-bypass-approvals-and-sandbox -m <model>`. - MCP servers written to a generated `~/.codex/config.toml`. - `codex exec` exits `0` even on a failed turn, so the stop reason is derived from the terminal `turn.completed` / `turn.failed` event rather than the exit code. ## Important Codex enables a `tool_search` tool that small models reject — `gpt-5.4-nano` returns `400 invalid_request_error: "Tool 'tool_search' is not supported with gpt-5.4-nano."` and fails the turn immediately. **Use `gpt-5.4-mini` or larger**; the committed experiment uses `gpt-5.4`. Closes [AI-849](https://linear.app/supabase/issue/AI-849) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Rodriguespn added the run-evals Add to a PR to refresh benchmark evals label Jun 19, 2026

Rodriguespn force-pushed the pedrorodrigues/ai-848-add-claude-code-agent-support branch from e5080c4 to 3355b2f Compare June 19, 2026 09:56

Rodriguespn added run-evals Add to a PR to refresh benchmark evals and removed run-evals Add to a PR to refresh benchmark evals labels Jun 19, 2026

Rodriguespn marked this pull request as ready for review June 19, 2026 12:26

Rodriguespn removed the run-evals Add to a PR to refresh benchmark evals label Jun 19, 2026

vercel Bot deployed to Preview June 19, 2026 13:11 View deployment

Rodriguespn mentioned this pull request Jun 19, 2026

fix(evals): correct false-negative scoring (seed dir, endpoint case, brittle test check) #50

Merged

Rodriguespn marked this pull request as draft June 22, 2026 15:24

Rodriguespn and others added 18 commits June 22, 2026 17:04

chore: refresh eval results

8b94a15

chore: refresh eval results

ce52d06

Rodriguespn and others added 2 commits June 22, 2026 19:44

Rodriguespn force-pushed the pedrorodrigues/ai-848-add-claude-code-agent-support branch from 3768353 to 6e5ed14 Compare June 22, 2026 20:41

Rodriguespn marked this pull request as ready for review June 22, 2026 20:49

Rodriguespn requested a review from mattrossman June 22, 2026 21:52

Rodriguespn commented Jun 22, 2026

View reviewed changes

Comment thread apps/web/vercel.json

mattrossman reviewed Jun 23, 2026

View reviewed changes

Comment thread packages/core/src/parsers/claude-code.ts

mattrossman reviewed Jun 23, 2026

View reviewed changes

Comment thread apps/framework/harness/run-eval.ts Outdated

mattrossman reviewed Jun 23, 2026

View reviewed changes

Comment thread apps/framework/harness/run-eval.ts Outdated

mattrossman approved these changes Jun 23, 2026

View reviewed changes

Rodriguespn merged commit f4ec0fd into main Jun 23, 2026
5 checks passed

Rodriguespn changed the title ~~feat(eval): add Claude Code CLI agent harness (AI-848)~~ feat(eval): add Claude Code CLI agent harness Jun 23, 2026

Rodriguespn self-assigned this Jun 23, 2026

Rodriguespn mentioned this pull request Jun 24, 2026

feat(eval): add Codex agent & add reasoningEffort as experiment param #49

Merged

This was referenced Jun 24, 2026

feat(eval): add OpenCode agent harness #56

Closed

feat(eval): add Gemini CLI agent harness #57

Draft

raulb deleted the pedrorodrigues/ai-848-add-claude-code-agent-support branch July 29, 2026 11:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(eval): add Claude Code CLI agent harness - #47

feat(eval): add Claude Code CLI agent harness#47
Rodriguespn merged 23 commits into
mainfrom
pedrorodrigues/ai-848-add-claude-code-agent-support

Rodriguespn commented Jun 18, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Rodriguespn commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it runs

Live runs

Uh oh!

linear-code Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Rodriguespn commented Jun 18, 2026 •

edited

Loading

linear-code Bot commented Jun 18, 2026 •

edited

Loading

vercel Bot commented Jun 18, 2026 •

edited

Loading