feat(eval): add Codex agent & add reasoningEffort as experiment param - #49
Merged
Merged
Conversation
AI-849 Add codex agent support to the eval framework
Add a |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Rodriguespn
marked this pull request as ready for review
June 18, 2026 23:36
Rodriguespn
marked this pull request as draft
June 18, 2026 23:38
Rodriguespn
changed the title
Rodriguespn
force-pushed
the
pedrorodrigues/ai-849-add-codex-agent-support
branch
from
June 18, 2026 23:42
8dbc447 to
342c0bb
Compare
Rodriguespn
marked this pull request as ready for review
June 18, 2026 23:42
Rodriguespn
marked this pull request as draft
June 19, 2026 09:50
Rodriguespn
force-pushed
the
pedrorodrigues/ai-848-add-claude-code-agent-support
branch
from
June 19, 2026 09:56
e5080c4 to
3355b2f
Compare
Rodriguespn
force-pushed
the
pedrorodrigues/ai-849-add-codex-agent-support
branch
2 times, most recently
from
June 19, 2026 12:10
29c8514 to
48d685d
Compare
Rodriguespn
force-pushed
the
pedrorodrigues/ai-848-add-claude-code-agent-support
branch
from
June 22, 2026 20:41
3768353 to
6e5ed14
Compare
Rodriguespn
added a commit
that referenced
this pull request
Jun 23, 2026
Adds **`claude-code`** as an eval agent alongside `ai-sdk`, so scenarios run through a **real agent harness** — the Claude Code CLI drives its own loop, tools, and MCP client — rather than a bare model loop. Its transcript is parsed back into the same surface scorers already use, so scoring is unchanged. The framework is generic: a runner + a parser + a registry entry adds the next agent (Codex follows in [#49](#49)). ## How it runs The agent runs in a Docker sandbox in **both** modes. The base sandbox carries the agent's own tooling (shell/file tools) and skills, but **not** the Supabase CLI. The only thing that varies by mode is the Supabase local stack: - **local-stack mode** — the Supabase CLI is installed at setup and the local stack is started (`supabase start`); the agent works with the CLI + its own tools, scored against the running stack. - **tools mode** — **no Supabase CLI and no local stack**; the agent works through the Supabase **MCP** against platform-lite (the mocked hosted platform), with its shell/file tools still available. **Main difference: tools mode has no Supabase CLI (or local stack)** — it's MCP-driven by environment, not by a tool restriction. ## Live runs - [claude-code-haiku-4.5](https://github.com/supabase/evals/actions/runs/27825575802/job/82349020404) - [claude-code-sonnet-4.6](https://github.com/supabase/evals/actions/runs/27825575802/job/82349020420) Closes [AI-848](https://linear.app/supabase/issue/AI-848) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Rodriguespn
changed the title
Rodriguespn
changed the base branch from
pedrorodrigues/ai-848-add-claude-code-agent-support
to
main
June 24, 2026 00:04
Rodriguespn
force-pushed
the
pedrorodrigues/ai-849-add-codex-agent-support
branch
3 times, most recently
from
June 24, 2026 00:25
1213d51 to
277c55f
Compare
Rodriguespn
marked this pull request as ready for review
June 24, 2026 00:30
Rodriguespn
force-pushed
the
pedrorodrigues/ai-849-add-codex-agent-support
branch
from
June 24, 2026 09:44
0f567ac to
9cfcf6a
Compare
mattrossman
reviewed
Jun 24, 2026
| // property of the eval (interface/local dir), not the agent. | ||
| export default defineExperiment({ | ||
| agent: codexAgent({ | ||
| model: "gpt-5.4", |
Collaborator
There was a problem hiding this comment.
Per sync, I'd remove this experiment or replace w/ 5.4-mini to keep experiments reflective of minimal clusters of users. I expect power users will use 5.5, and those on a token budget will use 5.4-mini
Collaborator
There was a problem hiding this comment.
Also, I don't see a way to specify reasoning level on codexAgent (or the claude code one).
Maybe that can be a separate PR but I think we need a way to control reasoning level in Codex and Claude Code agents since that's a pretty common setting users may change in those agents.
Collaborator
There was a problem hiding this comment.
see motivation on model selection here https://supabase.slack.com/archives/C0BAGJBL49E/p1782333782523379
mattrossman
approved these changes
Jun 24, 2026
mattrossman
left a comment
mattrossman
left a comment
Collaborator
There was a problem hiding this comment.
Looks good + tested locally, though I'd like to remove or replace the 5.4 experiment. If replacing, I'd pick 5.4-mini (medium) to mirror sonnet tier. Eventually will want a way to control reasoning explicitly though that can be a follow up PR.
Adds `codex` (OpenAI Codex) as an eval agent on the runner/parser framework from #47: a runner + a parser + a registry entry + a factory + experiments. Orchestration is unchanged; Codex's transcript is parsed into the same surface scorers already use. - runners/codex.ts: install + `codex login --with-api-key` (key via stdin, never argv), `codex exec --json` (--skip-git-repo-check --dangerously-bypass-approvals-and-sandbox -m <model>, prompt = system+user on stdin), MCP via a generated ~/.codex/config.toml. Model type from openai's ChatModel widened with (string & {}), exported as CodexModel. - runners/codex.ts deriveStopReason: `codex exec` exits 0 even on a failed turn, so the stop reason comes from the terminal turn.completed/turn.failed event, not the exit code — a clean stop and an agent failure stay distinct. - parsers/codex.ts: the thread/turn/item event schema; paired tool_call/ tool_result by item id; owns its CODEX_TOOLS map; normalized command/path via the shared extractArgs (raw args left untouched). Tool success is tri-state — a missing status is unknown, not auto-success. - Registered in parsers/registry.ts; codexAgent factory; experiments for gpt-5.4 and gpt-5.5. Adds the openai SDK as a compile-time dep. - eval-refresh workflow: run codex-gpt-5.4 + codex-gpt-5.5 alongside the claude-code experiments. Like Claude Code, Codex runs in BOTH modes: the sandbox carries its shell/file tools either way, and tools mode just drops the Supabase CLI + local stack so Supabase access goes through MCP. Which mode an eval uses is a property of the eval (interface/local dir), not the agent. Model note: Codex enables a `tool_search` tool that gpt-5.4-nano rejects with `400 invalid_request_error: "Tool 'tool_search' is not supported with gpt-5.4-nano."`, failing the turn immediately — use gpt-5.4-mini or larger. Verified e2e (CLI 0.138, gpt-5.4): local-stack build-tests-001 -> 3/3, and tools-mode investigate-db-001 -> 3/3 using MCP (execute_sql / list_tables / list_projects), both stoppedReason "stop". gpt-5.4-nano reproduces the tool_search 400 -> stoppedReason "error". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rodriguespn
force-pushed
the
pedrorodrigues/ai-849-add-codex-agent-support
branch
from
June 25, 2026 14:36
a414785 to
8290576
Compare
Rodriguespn
changed the title
reasoningEffort as experiment param
Rodriguespn
force-pushed
the
pedrorodrigues/ai-849-add-codex-agent-support
branch
2 times, most recently
from
June 25, 2026 14:50
f4dc184 to
9c274fd
Compare
Rodriguespn
force-pushed
the
pedrorodrigues/ai-849-add-codex-agent-support
branch
from
June 25, 2026 14:52
9c274fd to
d34f45a
Compare
Split the CLI-agent layer so each agent owns its own factory. The generic orchestration (createCliAgent) moves to agents/engine.ts and no longer imports any concrete runner, parser, or model/effort type. Each agent now lives under agents/<name>/ (runner + parser + an index.ts that wires them into the public claudeCodeAgent/codexAgent factory). Collapse the separate parsers/registry.ts into agents/registry.ts, derived from the AgentDefinition each agent module exports (id comes from runner.id, so it is no longer written twice). Generic transcript tooling stays in parsers/. Adding an agent is now one folder + one registry line; no shared file changes. Public API is unchanged; createCliAgent and AgentDefinition are newly exported. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
codex(OpenAI Codex) as an eval agent alongsideclaude-codeandai-sdk, built on the runner/parser framework from #47: a runner + a parser + a registry entry. The orchestration is unchanged and Codex's transcript is parsed back into the same surface scorers already use, so scoring is unchanged.Codex runs headless via
codex exec --jsonin a Docker sandbox. Like Claude Code, it runs in both modes — the sandbox carries its shell/file tools either way; tools mode just drops the Supabase CLI + local stack, so Supabase access goes through MCP. Which mode an eval uses is a property of the eval (interface/local/dir), not the agent.codex login --with-api-key(key piped via stdin, never in argv).--skip-git-repo-check --dangerously-bypass-approvals-and-sandbox -m <model>.~/.codex/config.toml.codex execexits0even on a failed turn, so the stop reason is derived from the terminalturn.completed/turn.failedevent rather than the exit code.Important
Codex enables a
tool_searchtool that small models reject —gpt-5.4-nanoreturns400 invalid_request_error: "Tool 'tool_search' is not supported with gpt-5.4-nano."and fails the turn immediately. Usegpt-5.4-minior larger; the committed experiment usesgpt-5.4.Closes AI-849