diff --git a/.github/pages-stub/index.html b/.github/pages-stub/index.html index 738657ec..4d3677df 100644 --- a/.github/pages-stub/index.html +++ b/.github/pages-stub/index.html @@ -3,7 +3,7 @@ - Coder Eval — evaluate AI coding agents and Claude Code skills + Coder Eval — evaluate AI coding agents and their skills

Coder Eval

- An open-source framework for evaluating and benchmarking AI coding agents and their Claude - Code skills: it runs a real agent — Claude Code, Codex, or Gemini — in a sandbox against - declarative YAML tasks, then scores the files and commands the agent actually produced. + Playwright for coding agents. An open-source, agent-agnostic framework for + evaluating and benchmarking AI coding agents and their skills: it runs a real agent — Claude + Code, Codex, Antigravity (Gemini), or OpenCode — in a sandbox against declarative YAML + tasks, then scores the files and commands the agent actually produced.

The documentation has moved to diff --git a/CLAUDE.md b/CLAUDE.md index ac00d5ef..1e988f96 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -216,7 +216,7 @@ make plugin-reference # the plugin's bundled criteria reference from the models Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction. -Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's). +Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed). When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033–CE036 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE036 enforces the `live_verdict` determinism + monotonicity contract (`criteria/base.py`) that `EarlyStopWatcher`'s latching, deferred fail-stop, and flip-attribution silently depend on: monotonicity over arbitrary Python is undecidable, so instead of a static check it REPLAYS each live criterion against every prefix of recorded trajectories (`tests/lint/live_verdict_contract.py::CASES`) — on the authored ordering AND under seeded shuffles (`permuted_violations`, which catch order-sensitive bugs the authored walk misses) — and asserts the property directly, plus registry-derived coverage — every `LiveSuccessCriterion` in the union must have cases, and every polarity its instances claim via `live_decidable_polarities()` must actually be reached by one (otherwise a single always-`undecided` fixture would "cover" a type while proving nothing). Adding a live criterion therefore means adding `ContractCase`s in the same change. CE035 resolves every `steps..outputs.` / `needs..outputs.` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.) diff --git a/README.md b/README.md index 1273b57c..00f7d841 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills +# Coder Eval — evaluate and benchmark AI coding agents and their skills [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/) [![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval) @@ -8,21 +8,28 @@ [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/) [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml) -**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for -**evaluating and benchmarking AI coding agents and their skills** — built for CLI -and skill builders — with sandboxing, reproducibility, and data-driven analysis. -It runs a real agent (**Claude Code**, **Codex**, **Google Antigravity / -Gemini**, or **OpenCode**) in a sandbox against declarative YAML tasks, then scores the files and -commands it actually produced. Not an "agentic coding" benchmark: it measures how -effective your CLI and skills are when used by coding agents. - -Reach for it when you want to **test whether a Claude Code skill triggers**, -**A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode** (or model vs. model, -prompt vs. prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench, -SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the -tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a -`skill_triggered` activation check, an A/B experiment layer, and per-tool cost -telemetry. See [How it compares](https://coder-eval.com/docs/comparison). +

+ Playwright for coding agents — one declarative test file, any agent + runtime, a real sandbox, and a pass/fail gate in CI. +

+ +**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an +open-source, **agent-agnostic** framework for **evaluating and benchmarking AI coding +agents and their skills** — built for benchmark authors, CLI builders, and skill +builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real +agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, or +**OpenCode** — in a sandbox against declarative YAML tasks, then scores the files and +commands it actually produced. Changing harness is one field (`agent.type`); the +tasks, criteria, scoring, telemetry, and reports stay the same. + +Reach for it when you want to **benchmark agents on your own domain tasks**, +**test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code +vs. Codex vs. Gemini vs. OpenCode** (or model vs. model, prompt vs. prompt), or +**gate CI on coding-agent quality**. It is **not a fixed leaderboard**: unlike +SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks +and you bring the scoring — weighted 0.0–1.0 criteria, a `skill_triggered` activation +check, an A/B experiment layer, and per-tool cost telemetry, over whatever work *you* +care about. See [How it compares](https://coder-eval.com/docs/comparison). 📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**.

@@ -33,15 +40,26 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison). - **Sandboxed execution** in isolated environments with resource limits - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics -- **Agent abstraction** — Claude Code, Codex, Antigravity (Gemini), and OpenCode today, extensible via a plugin SPI +- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), and OpenCode today; add your own harness through the plugin SPI - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming +## Watch the intro + +

+ + Video: Coder Eval — UiPath open-source framework to test AI coding agents + +

+ +▶ **[Coder Eval: UiPath open-source framework to test AI Coding Agents](https://www.youtube.com/watch?v=Iyq-5m1CnuI)** +— what the framework does, and how a run works end to end. + ## What you can do with it - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds - **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt -- **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style) +- **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on whichever harness your users run - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites @@ -53,15 +71,27 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison). ## Quick Start -**Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and the -[Claude CLI](https://docs.anthropic.com/claude/docs/claude-code) (`brew install claude`). -Developed on macOS; CI runs on Linux. +**Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and **the +runtime of at least one coding agent** — plus that agent's own model credentials. +Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra, +the other two are separate CLIs you install yourself: + +| Agent | `agent.type` | Runtime | Guide | +| --- | --- | --- | --- | +| Claude Code (default) | `claude-code` | `brew install claude` — separate CLI | [Claude Code](docs/agents/CLAUDE_CODE.md) | +| OpenAI Codex | `codex` | `uv sync --extra codex` — the extra ships the Codex SDK + CLI | [Codex](docs/agents/CODEX.md) | +| Google Antigravity (Gemini) | `antigravity` | `uv sync --extra antigravity` — the extra ships the harness binary | [Antigravity](docs/agents/ANTIGRAVITY.md) | +| OpenCode (open-weight models) | `opencode` | `npm install -g opencode-ai` — separate CLI | [OpenCode](docs/agents/OPENCODE.md) | + +The examples below use the default `claude-code` agent. Developed on macOS; CI runs on +Linux. ```bash git clone https://github.com/UiPath/coder_eval.git cd coder_eval -uv sync --extra dev # install core + dev tools +uv sync # install the framework (add --extra codex / + # --extra antigravity for those runtimes) cp .env.example .env # then set ANTHROPIC_API_KEY — or skip that: an # existing Claude Code login (`claude login`) is # picked up automatically @@ -73,13 +103,14 @@ uv run coder-eval report runs/latest # view the result New here? Follow **[Tutorial 01 — Your First Evaluation](docs/tutorials/01-first-evaluation.md)**. -The optional `[uipath]` extra (`uv sync --extra dev --extra uipath`) adds the in-host -`uipath` SDK for local sandbox parity; it installs from public PyPI (no credentials -required). Without it the framework runs end-to-end; uipath-dependent features fail -at dispatch with a clear hint. +Two more extras you only need on purpose: `--extra dev` adds the contributor +toolchain (pytest, ruff, pyright, pre-commit — see +[CONTRIBUTING.md](CONTRIBUTING.md)), and `--extra uipath` adds the in-host `uipath` +SDK for local sandbox parity (public PyPI, no credentials). Without either, the +framework still runs end-to-end. -**Using Coder Eval in CI or another project?** Install the published package -instead of cloning: +**Just want the CLI, without cloning?** Install the published package — this is also +what a CI job or another repo does: ```bash uv tool install coder-eval # puts the `coder-eval` CLI on your PATH, @@ -98,9 +129,10 @@ the full setup. ## Use inside Claude Code -This repo is also a **Claude Code plugin marketplace**, so the whole loop — -scaffold a suite, author a task, check whether a skill triggers, read the -results — runs inside the agent: +Coder Eval evaluates any of the supported agents, and it also ships an authoring +front-end for one of them: this repo is a **Claude Code plugin marketplace**, so the +whole loop — scaffold a suite, author a task, check whether a skill triggers, read the +results — runs inside Claude Code. The suites you author this way run on every harness: ``` /plugin marketplace add UiPath/coder_eval @@ -275,9 +307,13 @@ success_criteria: description: "Script must execute successfully" ``` -Tasks can omit the `agent` section entirely — defaults resolve from the experiment -layer (`experiments/default.yaml`). For the full schema and every criterion type, -see the [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md). +`agent.type` is the only harness-specific line: swap it for `codex`, `antigravity`, or +`opencode` — or override it per run with `coder-eval run … -D agent.type=opencode` — and +the same criteria score the same way. Tasks can omit the `agent` section entirely — +defaults resolve from the experiment layer (`experiments/default.yaml`). For the full +schema and every criterion type, see the +[Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md); for what each `run_limits` field +means on each harness, see [Run-Limit Parity](docs/agents/HARNESS_PARITY.md). > **Tip:** With the [Claude Code plugin](docs/PLUGIN.md) installed, use > `/coder-eval:task` to scaffold a task from a natural-language description, and @@ -301,8 +337,11 @@ extension points (new criteria, new agents). - **Tasks execute real code** — run untrusted tasks only under the container driver (see [Docker Isolation](docs/DOCKER_ISOLATION.md)); the `tempdir` driver is not a security boundary. -- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; Coder Eval - does not proxy or supply model access. +- **Bring your own agent runtime and model credentials** — Coder Eval never supplies + model access, and it only ships an agent runtime where an extra says so (`codex`, + `antigravity`); Claude Code and OpenCode are separate CLIs. Supply the runtime (see + [Quick Start](#quick-start)) and the keys it needs — Anthropic, Bedrock, OpenAI, + Gemini, or an OpenRouter key for open-weight models via OpenCode. - **Python 3.13+ only.** ## Support & security @@ -318,6 +357,8 @@ extension points (new criteria, new agents). ## Acknowledgments -Built with the [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk), -[Pydantic](https://pydantic.dev/), [Typer](https://typer.tiangolo.com/), and -[Rich](https://rich.readthedocs.io/). +Built with [Pydantic](https://pydantic.dev/), [Typer](https://typer.tiangolo.com/), +and [Rich](https://rich.readthedocs.io/), on top of the harnesses it drives — the +[Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk), the +[Codex SDK](https://github.com/openai/codex), [Google Antigravity](https://antigravity.google/), +and [OpenCode](https://opencode.ai). diff --git a/docs/comparison.md b/docs/comparison.md index 6dad1f4b..745e7109 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -2,14 +2,15 @@ title: "Coder Eval vs. SWE-bench, SkillsBench, Harbor & OpenAI Evals — how it compares" description: >- How Coder Eval compares to SWE-bench, SkillsBench, Harbor, and OpenAI Evals for - evaluating AI coding agents and Claude Code skills — with when to choose each. + evaluating AI coding agents and their skills — with when to choose each. Grounded in each project's own docs. --- # How Coder Eval compares -**Coder Eval** is a framework for evaluating AI coding agents and their skills. It -runs an agent (Claude Code, Codex, Gemini) in a sandbox against declarative YAML +**Coder Eval** is an agent-agnostic framework for evaluating AI coding agents and +their skills. It runs a real agent (Claude Code, OpenAI Codex, Antigravity/Gemini, or +OpenCode — one `agent.type` field apart) in a sandbox against declarative YAML tasks and scores the files and commands it produces on a weighted 0.0–1.0 scale, with cost/token telemetry, an A/B experiment layer, skill-activation checks, and CI pass/fail gates. @@ -24,7 +25,7 @@ in its own documentation; see [Sources](#sources). | --- | --- | --- | --- | --- | --- | --- | | **What it grades** | Files + commands the agent produced | Whether a patch passes the repo's tests | Skill value on a fixed task set | Agent task success in sandboxes | Model text output | Whatever you wire up | | **Task source** | Your own (YAML) | Fixed benchmark (+ collection script) | Fixed (87 tasks / 8 domains) | Your own (framework) + Terminal-Bench 2.0 | Your own (YAML/JSON) + registry | Manual | -| **Runs a real agent + tools** | ✅ Claude Code, Codex, Gemini | Runs your patch/scaffold | ✅ multi-harness | ✅ Claude Code, OpenHands, Codex | ❌ grades model output | ❌ | +| **Runs a real agent + tools** | ✅ Claude Code, Codex, Antigravity (Gemini), OpenCode | Runs your patch/scaffold | ✅ multi-harness | ✅ Claude Code, OpenHands, Codex | ❌ grades model output | ❌ | | **Sandboxed & reproducible** | ✅ tempdir / Docker | ✅ Docker | ✅ deterministic verifiers | ✅ Docker + cloud (Daytona/Modal) | — (grades text) | ❌ | | **Scoring** | Weighted 0.0–1.0 + thresholds | Pass/fail (tests) | Pass/fail (verifiers) | Task-level | Match / model-graded | ❌ | | **A/B experiments (model / tool / prompt)** | ✅ built-in | ❌ | Skills on/off only | — | Compare model versions | ❌ | @@ -120,8 +121,9 @@ focus on tasks and criteria rather than harness infrastructure. ## When to choose Coder Eval -- Evaluating how well agents use a CLI or Claude Code skills. -- A/B-testing models, tools, or prompts on the same tasks (Claude Code vs. Codex vs. Gemini). +- Benchmarking coding agents on your own domain tasks instead of a fixed leaderboard. +- Evaluating how well agents use a CLI or a skill (Claude Code skills included). +- A/B-testing agents, models, tools, or prompts on the same tasks (Claude Code vs. Codex vs. Antigravity/Gemini vs. OpenCode). - Detecting skill regressions when a model or prompt changes. - Gating CI on coding-agent quality. - Producing reproducible, sandboxed runs scored per file and per command, with a dashboard for review. diff --git a/docs/index.md b/docs/index.md index 5f22ce7e..7db93bb1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,24 +1,27 @@ --- -title: "Evaluate AI coding agents & Claude Code skills — Coder Eval" +title: "Evaluate AI coding agents and their skills — Coder Eval" description: >- - Coder Eval is an open-source framework to evaluate, benchmark, and A/B-test AI - coding agents and Claude Code skills in a sandbox — declarative YAML tasks, - weighted scoring, cost/token telemetry, and CI gates for Claude Code, Codex, - Gemini, and OpenCode. + Coder Eval is Playwright for coding agents: an open-source, agent-agnostic + framework to evaluate, benchmark, and A/B-test AI coding agents and their skills + in a sandbox — declarative YAML tasks, weighted scoring, cost/token telemetry, and + CI gates for Claude Code, Codex, Antigravity (Gemini), and OpenCode. --- -# Evaluate AI coding agents & Claude Code skills — Coder Eval +# Evaluate AI coding agents and their skills — Coder Eval -**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for -**evaluating AI coding agents and their skills** — built for CLI and skill -builders — with sandboxing, reproducibility, and data-driven analysis. It is not -an "agentic coding" benchmark: it measures how effective *your* CLI and skills -are when used by coding agents such as **Claude Code**, **Codex**, **Google -Antigravity (Gemini)**, and **OpenCode**. +**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an +open-source, **agent-agnostic** framework for **evaluating AI coding agents and their +skills** — built for benchmark authors, CLI builders, and skill builders — with +sandboxing, reproducibility, and data-driven analysis. It runs a real agent — +**Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, or **OpenCode** — +against declarative YAML tasks in a sandbox. Changing harness is one field +(`agent.type`); the tasks, criteria, scoring, telemetry, and reports stay the same. +It is **not a fixed leaderboard**: you bring the tasks and you bring the scoring. -If you have ever asked *"how do I test whether my Claude Code skill actually -triggers?"*, *"how do I benchmark Claude Code vs. Codex on my own tasks?"*, or -*"how do I gate CI on coding-agent quality?"* — this is the framework for that. +If you have ever asked *"how do I benchmark coding agents on my own domain tasks?"*, +*"how do I test whether my skill actually triggers?"*, *"how do I compare Claude Code +vs. Codex vs. Gemini vs. OpenCode on the same suite?"*, or *"how do I gate CI on +coding-agent quality?"* — this is the framework for that.

Coder Eval running a sandboxed coding-agent evaluation from a YAML task and browsing the scored result in evalboard @@ -30,7 +33,7 @@ triggers?"*, *"how do I benchmark Claude Code vs. Codex on my own tasks?"*, or - **Sandboxed execution** in isolated environments with resource limits - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics -- **Agent abstraction** — Claude Code, Codex, Antigravity (Gemini), and OpenCode today, extensible via a plugin SPI +- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), and OpenCode today; add your own harness through the plugin SPI - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming @@ -38,8 +41,9 @@ triggers?"*, *"how do I benchmark Claude Code vs. Codex on my own tasks?"*, or - **Benchmark coding agents** — score an agent across a suite of tasks with weighted pass/fail thresholds - **Compare models & configs** — A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt -- **Test whether a Claude Code skill triggers** — verify an agent actually engages a - target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style) +- **Test whether a skill triggers** — verify an agent actually engages a target + skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on + whichever harness your users run - **Keep skills fresh in CI** — re-validate skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites diff --git a/docs/llms.txt b/docs/llms.txt index 9dc3865f..baec1eac 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -1,22 +1,23 @@ # Coder Eval -> Coder Eval (`pip install coder-eval`) is an open-source framework for evaluating, -> benchmarking, and A/B-testing AI coding agents and their Claude Code skills in a -> sandbox. It uses declarative YAML tasks with weighted, continuous scoring -> (0.0–1.0), runs real agents (Claude Code, Codex, Google Antigravity/Gemini, -> OpenCode) with -> full tool use, captures per-tool cost/token telemetry, and provides CI-ready -> pass/fail gates. It is not a fixed benchmark or leaderboard — it scores your own -> tasks, and can verify whether a Claude Code skill actually triggers. - -Choose Coder Eval to: benchmark coding agents on your own tasks; A/B-test model vs. -model, tool-on vs. tool-off, or prompt vs. prompt; test whether a Claude Code skill -triggers and catch regressions in CI; gate a build on coding-agent quality. It -differs from fixed datasets (SWE-bench, SkillsBench) that give a shared leaderboard, -from OpenAI Evals (which grades model text), and from large-scale agent-eval / RL -frameworks like Harbor — Coder Eval is purpose-built for coding-agent/skill suites -with weighted 0.0–1.0 file/command scoring, a skill-activation check, an experiment -layer, deep cost telemetry, and the evalboard dashboard. It also differs from +> Coder Eval (`pip install coder-eval`) is an open-source, agent-agnostic framework +> for evaluating, benchmarking, and A/B-testing AI coding agents and their skills +> (including Claude Code skills) in a sandbox. It uses declarative YAML tasks with +> weighted, continuous scoring (0.0–1.0), runs real agents (Claude Code, OpenAI +> Codex, Google Antigravity/Gemini, OpenCode) with full tool use — switching harness +> is one field, `agent.type` — captures per-tool cost/token telemetry, and provides +> CI-ready pass/fail gates. It is not a fixed benchmark or leaderboard: you bring the +> tasks and the scoring, and it can verify whether a skill actually triggers. + +Choose Coder Eval to: benchmark coding agents on your own domain tasks; A/B-test +agent vs. agent, model vs. model, tool-on vs. tool-off, or prompt vs. prompt; test +whether a skill triggers and catch regressions in CI; gate a build on coding-agent +quality. It differs from fixed datasets (SWE-bench, SkillsBench) that give a shared +leaderboard, from OpenAI Evals (which grades model text), and from large-scale +agent-eval / RL frameworks like Harbor — Coder Eval is purpose-built for +coding-agent/skill suites with weighted 0.0–1.0 file/command scoring, a +skill-activation check, an experiment layer, deep cost telemetry, and the evalboard +dashboard. It also differs from hand-rolled scripts, which lack reproducible sandboxes, weighted criteria, telemetry, and A/B plumbing. diff --git a/mkdocs.yml b/mkdocs.yml index db2d2728..ebd33b6d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,8 +1,9 @@ site_name: Coder Eval site_description: >- - Evaluate & benchmark AI coding agents and Claude Code skills — a sandboxed, - reproducible framework with declarative YAML eval suites for Claude Code, - Codex & Gemini, A/B experiments, weighted scoring, and CI gates. + Evaluate and benchmark AI coding agents and their skills — a sandboxed, + agent-agnostic framework with declarative YAML eval suites for Claude Code, + Codex, Antigravity (Gemini) and OpenCode, A/B experiments, weighted scoring, + and CI gates. site_url: https://coder-eval.com/docs/ repo_url: https://github.com/UiPath/coder_eval repo_name: UiPath/coder_eval diff --git a/pyproject.toml b/pyproject.toml index 5eea61f9..3b22030f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "coder-eval" version = "0.11.6" -description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites." +description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode) with sandboxed, reproducible YAML task suites." readme = "README.md" license = "Apache-2.0" requires-python = ">=3.13" @@ -9,7 +9,7 @@ authors = [{ name = "UiPath", email = "coder-eval@uipath.com" }] keywords = [ "ai", "llm", "agent", "coding-agent", "evaluation", "eval", "evals", "benchmark", "swe-bench", "claude", "claude-code", "codex", "anthropic", - "gemini", "antigravity", "sandbox", "code-generation", "agent-evaluation", + "gemini", "antigravity", "opencode", "sandbox", "code-generation", "agent-evaluation", "llm-evaluation", "llm-eval", "ai-evaluation", "skills-evaluation", "claude-skills", "claude-code-skills", "agent-skills", "skillsbench", "agent-testing", "llmops", diff --git a/tests/lint/agent_roster_parity.py b/tests/lint/agent_roster_parity.py new file mode 100644 index 00000000..873f7c4f --- /dev/null +++ b/tests/lint/agent_roster_parity.py @@ -0,0 +1,117 @@ +"""CE047 — every marketing/onboarding surface must name every built-in agent. + +The roster of supported harnesses is restated in prose on a handful of surfaces +that nothing mechanically ties to the code: the README, the docs home, the +comparison page, ``llms.txt``, the ``mkdocs.yml`` site description, the Pages +stub, and the packaging metadata. Adding a harness means remembering all seven — +which is exactly how **OpenCode shipped while being absent from most of them**: +the agent worked, but a reader (or a crawler, or an LLM answering "which agents +does Coder Eval support?") was told it did not exist. There is no error, no test +failure, and no user report for that; the surface just quietly under-sells the +framework. + +``AgentKind`` is the framework's own list of built-ins (it is deliberately NOT the +closed set of valid ``agent.type`` values — the ``AgentRegistry`` is authoritative +and plugins extend it — but every built-in *is* in the enum, and a built-in is +what these surfaces promise). This rule derives the expected roster from that +enum and asserts each surface mentions each agent by name. + +Scope note: this is a **presence** check over a file (or, where a file is mostly +unrelated content, over one extracted region — ``mkdocs.yml``'s +``site_description``, ``pyproject.toml``'s ``description`` + ``keywords``). It +cannot tell a good sentence from a bad one; it only makes "we forgot this harness +exists" impossible to ship. Third-party plugin agents are out of scope — they own +their own docs. + +It is intentionally NOT a ``BaseRule`` in ``tests/lint/runner.py``: that runner is +AST-only over ``.py`` files, whereas this reasons over Markdown/YAML/TOML/HTML +surfaces. It is wired as a dedicated test in +``tests/test_custom_lint.py::TestCE047AgentRosterParity`` (precedent: CE026-CE031). +""" + +from __future__ import annotations + +import re +import tomllib +from collections.abc import Callable +from pathlib import Path + + +# Agent kinds that are NOT part of the user-facing roster: `none` is the agentless +# system-task escape hatch and `unknown` is an internal sentinel for a task whose +# type could not be resolved. Neither is a harness anyone installs. +NON_ROSTER_KINDS: frozenset[str] = frozenset({"none", "unknown"}) + +# How each built-in agent is spelled in prose. Keyed by the `AgentKind` VALUE so a +# new built-in fails `test_every_builtin_kind_has_display_names` until its prose +# name is declared here — the enum stays the trigger, this table is only the +# spelling. Matching is case-insensitive and substring-based, so "Codex" also +# covers "OpenAI Codex" and `coder-eval[codex]`. +AGENT_DISPLAY_NAMES: dict[str, tuple[str, ...]] = { + "claude-code": ("Claude Code",), + "codex": ("Codex",), + # Google's harness is named on some surfaces by its model ("Gemini"), which is + # an acceptable spelling of the same row. + "antigravity": ("Antigravity", "Gemini"), + "opencode": ("OpenCode",), +} + + +def _whole_file(text: str) -> str: + return text + + +def _mkdocs_site_description(text: str) -> str: + """The `site_description:` block scalar — the rest of mkdocs.yml is nav/theme.""" + match = re.search(r"^site_description:.*?(?=^\S)", text, re.MULTILINE | re.DOTALL) + return match.group(0) if match else "" + + +def _pyproject_marketing_text(text: str) -> str: + """`description` + `keywords` — the strings PyPI shows and search engines index.""" + project = tomllib.loads(text).get("project", {}) + return project.get("description", "") + "\n" + "\n".join(project.get("keywords", [])) + + +# (repo-relative path, why it matters, region extractor). Every entry must exist. +ROSTER_SURFACES: tuple[tuple[str, str, Callable[[str], str]], ...] = ( + ("README.md", "the GitHub landing page", _whole_file), + ("docs/index.md", "the docs home on coder-eval.com", _whole_file), + ("docs/comparison.md", "the 'how it compares' page", _whole_file), + ("docs/llms.txt", "what LLMs read to answer questions about the project", _whole_file), + ("mkdocs.yml", "the docs-site description (search results, link previews)", _mkdocs_site_description), + (".github/pages-stub/index.html", "the coder-eval.com root stub", _whole_file), + ("pyproject.toml", "the PyPI description and keywords", _pyproject_marketing_text), +) + + +def roster_kinds() -> list[str]: + """The user-facing built-in agent kinds, derived from ``AgentKind``.""" + from coder_eval.models import AgentKind + + return sorted(k.value for k in AgentKind if k.value not in NON_ROSTER_KINDS) + + +def missing_agents_in(text: str, kinds: list[str] | None = None) -> list[str]: + """Roster kinds that ``text`` names by none of their accepted spellings.""" + haystack = text.lower() + return [ + kind + for kind in (kinds if kinds is not None else roster_kinds()) + if not any(name.lower() in haystack for name in AGENT_DISPLAY_NAMES.get(kind, (kind,))) + ] + + +def find_roster_gaps(repo_root: Path) -> dict[str, list[str]]: + """Map each surface that under-sells the roster to the agents it never names.""" + kinds = roster_kinds() + gaps: dict[str, list[str]] = {} + for rel, _why, extract in ROSTER_SURFACES: + path = repo_root / rel + if not path.is_file(): + gaps[rel] = [""] + continue + missing = missing_agents_in(extract(path.read_text(encoding="utf-8")), kinds) + if missing: + gaps[rel] = missing + return gaps diff --git a/tests/test_custom_lint.py b/tests/test_custom_lint.py index 64508fe7..adfbd60c 100644 --- a/tests/test_custom_lint.py +++ b/tests/test_custom_lint.py @@ -3600,3 +3600,70 @@ def test_yaml_walk_flags_a_literal_tail_behind_a_variable(self, tmp_path: Path): encoding="utf-8", ) assert not self._offending_paths_in(task) + + +@pytest.mark.lint +class TestCE047AgentRosterParity: + """CE047 — every onboarding/marketing surface must name every built-in agent. + + The roster is restated in prose on seven surfaces with nothing tying them to + the code, which is how OpenCode shipped while most of them still listed three + harnesses. Derives the expectation from `AgentKind` and asserts presence. A + whole-tree doc rule, so it lives here rather than in the AST-only runner. + """ + + REPO_ROOT = Path(__file__).parent.parent + + def test_every_surface_names_every_builtin_agent(self): + from tests.lint.agent_roster_parity import ROSTER_SURFACES, find_roster_gaps + + why = {rel: reason for rel, reason, _ in ROSTER_SURFACES} + gaps = find_roster_gaps(self.REPO_ROOT) + assert not gaps, ( + "\nSurface(s) that never name a built-in agent — a reader is told it does not " + "exist. Add it to the prose, or (if it is not user-facing) add its kind to " + "NON_ROSTER_KINDS in tests/lint/agent_roster_parity.py:\n\n" + + "\n".join( + f" {rel} ({why.get(rel, '?')}): missing {', '.join(names)}" for rel, names in sorted(gaps.items()) + ) + ) + + def test_every_builtin_kind_has_display_names(self): + # A new built-in agent must declare how it is spelled in prose, otherwise + # the presence check would silently fall back to matching its raw kind. + from tests.lint.agent_roster_parity import AGENT_DISPLAY_NAMES, roster_kinds + + undeclared = [k for k in roster_kinds() if k not in AGENT_DISPLAY_NAMES] + assert not undeclared, f"add prose spellings to AGENT_DISPLAY_NAMES for: {undeclared}" + + def test_roster_excludes_the_non_harness_kinds(self): + from tests.lint.agent_roster_parity import roster_kinds + + kinds = roster_kinds() + assert "claude-code" in kinds and "opencode" in kinds + assert "none" not in kinds and "unknown" not in kinds + + def test_catches_the_opencode_regression(self): + # The exact historical gap: a surface listing three of the four harnesses. + from tests.lint.agent_roster_parity import missing_agents_in + + three_of_four = "runs Claude Code, Codex, or Antigravity (Gemini) in a sandbox" + assert missing_agents_in(three_of_four) == ["opencode"] + + def test_model_name_counts_as_naming_the_antigravity_row(self): + from tests.lint.agent_roster_parity import missing_agents_in + + assert missing_agents_in("Claude Code, Codex, Gemini, and OpenCode") == [] + + def test_extractors_narrow_to_the_marketing_region(self, tmp_path: Path): + from tests.lint.agent_roster_parity import _mkdocs_site_description, _pyproject_marketing_text + + mkdocs = "site_name: X\nsite_description: >-\n Claude Code and OpenCode\nnav:\n - Codex: agents/codex.md\n" + region = _mkdocs_site_description(mkdocs) + assert "OpenCode" in region + assert "agents/codex.md" not in region # the nav must not satisfy the check + + pyproject = '[project]\ndescription = "Claude Code"\nkeywords = ["opencode"]\n[tool.x]\nz = "codex"\n' + region = _pyproject_marketing_text(pyproject) + assert "Claude Code" in region and "opencode" in region + assert "[tool.x]" not in region