How well do AI coding agents build on Notion's developer platform?
An open benchmark that runs coding agents — Claude Code, Codex, and friends — against real tasks on the Notion developer platform: the ntn CLI, Workers, and Notion-as-Code. Every task is scored by deterministic, programmatic verification — no LLM judges.
Inspired by Supabase Evals, Terminal-Bench, MCPMark, and τ-bench. Notion's platform shipped in May 2026 and is explicitly "built for AI coding agents" — this measures that, publicly and reproducibly.
Coming soon — first run in progress.
| Agent config | Score (avg@3) | pass^3 | Tool errors | Tokens | API-equiv cost | Time |
|---|---|---|---|---|---|---|
| OpenCode × Kimi K3 | – | – | – | – | – | – |
| Claude Code × Opus 5 (high) | – | – | – | – | – | – |
| Claude Code × Sonnet 5 (high) | – | – | – | – | – | – |
| Codex × GPT-5.6 Sol (medium) | – | – | – | – | – | – |
| Codex × GPT-5.6 Luna (high) | – | – | – | – | – | – |
| Claude Code × Fable 5 | – | – | – | – | – | – |
| Codex × GPT-5.6 Sol (xhigh) | – | – | – | – | – | – |
Grouped by product area, stage (build / investigate / resolve / operate), and agent config — all tables are generated by notionbench score <runDir> from that run's results.jsonl, alongside results/<run>/summary.md.
- 38 tasks across 4 stages and every programmable Notion surface — see docs/COVERAGE.md.
- Two suites (
benchmark= published, frozen per version;regression= known failure modes, grows freely, never affects published scores). - Three verification layers: static (typecheck), behavioral (
ntn workers exec --local, canonical intents comparison for Notion-as-Code — both offline & deterministic), and live workspace state assertions (host-side, never visible to the agent). - k=3 independent trials per task for v1 (extending to 5 via checkpoint/resume); we report avg@k with Wilson intervals and pass^k (reliability) — no "retry then grade."
- Docs-provided by default: agents get the AGENTS.md/skills Notion ships in its templates — the condition real developers actually run. (A docs-withheld ablation is supported by the harness for future experiments.)
- QC per task (CI): the oracle solution must pass, a null agent must fail, and a plausibly-wrong solution must fail.
- Every task in long form — what it asks for, what the verifier checks, what the trial starts from: docs/TASKS.md (generated; regenerate with
node scripts/gen-task-docs.mjs).
evals/<id>/ one task: PROMPT.md (frontmatter + instruction), EVAL.ts (scorer),
fixture/ (workspace state), solution/ (oracle), wrong/ (QC foil)
packages/core/ task metadata schema, frontmatter parsing, canonical types
packages/runner/ spawns agent CLIs headless in the sandbox, scores each trial,
checkpoint/resume; the `notionbench` CLI
packages/sandbox/ Docker environment (ntn, Node 24, templates preinstalled)
packages/scoring/ verification layers: intents canonicalizer, exec-local driver,
EVAL.ts subprocess driver, results.jsonl, the report tables,
live-state assertion helpers, stats (pass^k, Wilson)
docs/ COVERAGE.md (task × dimension matrix), methodology notes
web/ results site + live run dashboard (static, Cloudflare Pages)
.github/workflows/ CI: build, test, and the per-task QC gate — offline, no tokens
The published notionbench package is a pre-release placeholder today; npx notionbench will become the one-command entry point at v0.1.0 (first results release). Until then, run from the repo:
pnpm install && pnpm -r build
# See exactly what would happen — the grid, the argv per config, the child env.
node packages/runner/dist/cli.js run --dry-run --trials 3
# Run it, then aggregate. `run` does spawn -> score -> checkpoint per trial and
# appends every verified rollout to results/<runId>/results.jsonl.
node packages/runner/dist/cli.js run --tasks '*nac*' --configs claude-code-opus-5 --trials 3
node packages/runner/dist/cli.js score results/latest
# Watch it while it runs: hosts web/ + a bearer-gated /api/status, and prints a
# ready-to-open http://127.0.0.1:8377/#api=…&key=… link.
node packages/runner/dist/cli.js serve results/latest
# Audit a finished run before publishing it: per-task failure patterns, tasks
# whose failures share a diagnostic, verifier crashes, and a verdict.
node packages/runner/dist/cli.js doctor results/latestThe 798-cell grid runs for days, and a task whose verifier is wrong invalidates every cell that touched it. Three things keep that cheap:
-
Cells execute trial-major, task-major. The first pass covers every task, so all seven configs' verdicts on task N land minutes apart instead of days apart. Configs still run concurrently and never overlap with themselves; a config in a rate-window cooldown is skipped and the block proceeds without it, picking its missed cell back up when the window reopens.
-
A deterministic watchdog (no LLM) halts the run when ≥3 configs fail the same task in the same trial with the same normalized diagnostic — the signature of a verifier bug, which is how two real ones presented — or when a verifier crashes even once. A task every config fails for different reasons is flagged SUSPECT but never halts: that is also what a very hard task looks like, and this benchmark is meant to contain those. In-flight trials always finish and are scored; nothing is killed.
-
A reviewed failure stops the halt without blinding the watchdog. When the shared diagnostic turns out to be a real agent failure — a prompt several models misread the same way —
--ack <taskId>:<substring> --ack-reason "<why>"records exactly that signature asacknowledgedand lets the grid continue, while every other task stays protected. Nothing is hidden: the reason is stored inrun-spec.json, replayed on--resume, shown inALERT.jsonand listed bydoctor, which stops calling the run clean. Verifier crashes and fixture failures can never be acknowledged. -
Fix it and re-run only it, in one command.
--redoinvalidates that task's cells and retires its stale rows toresults.superseded.jsonlrather than leaving them to be averaged in:node packages/runner/dist/cli.js run --resume <runId> --redo <taskId>
Details: packages/runner/README.md.
Presets exist for Claude Code and Codex configs; any prompt-in/files-out CLI works via the command-template harness (see packages/runner). Auth is your subscription — the runner strips ANTHROPIC_API_KEY / OPENAI_API_KEY from every child so a stray key cannot silently change what is being measured. Live-suite tasks additionally need a Notion workspace integration token (docs coming). Community submissions (PR with trajectories) planned post-v1.
Verifying task authoring needs no agent at all:
pnpm --filter @notionbench/evals run qc # every task: oracle=1, wrong=0, null=0🚧 Early scaffold. Task authoring in progress. Watch the repo for the first results post.
MIT