Model Benchmarks

Reproducible coding-model benchmarks with recorded prompts, transcripts, timing, verification evidence, generated applications, and a browser-based comparison workbench.

Repository layout

• .agents/skills/run-benchmark/ — canonical benchmark runner and validators.
• .agents/rules/benchmark-runs.md — routing, evidence, naming, and proof rules.
• benchmark-compare/ — TanStack Start comparison application.
• benchmark-plans/ — reusable model/task matrices.
• benchmarks/ — every benchmark task, seed, and generated application.
• prompts/ — canonical, immutable versioned task prompts.
• prompts.json — deterministic prompt paths, public URLs, sizes, and hashes.
• runs/ — immutable run-level evidence and metadata.
• transcripts.json — portable index of every primary and resumed transcript.
• benchmarks/<task>/<date>-<harness>-<model>/ — historical generated applications.

Historical result directories are evidence. Do not rename or rewrite them.

Two task directories intentionally contain source material in addition to dated results:

• benchmarks/thumbfast/base-repo/ is the seed copied into each Thumbfast run.
• benchmarks/openclaw-benchmark/ is the OpenClaw CLI evaluation suite itself; its dated children are recorded runs of that suite.

All other task directories contain only dated result directories. Older YouTube Thumbnail results that predate the current naming convention retain their original names because they are historical evidence.

See CONTRIBUTE.md to propose prompts or tasks and to run a recorded benchmark with the repository's Codex skill.

Requirements

• Node.js 22 or newer
• pnpm 10.12.1 through Corepack
• One or more supported harnesses: codex, ccx, cc, cursor-agent

Install and open the comparison app

corepack enable
pnpm install --frozen-lockfile
pnpm dev

The comparison app opens on http://127.0.0.1:9080 and reads benchmark data from the repository root. Override the data directory with BENCHMARK_ROOT. Canonical prompts are available at /prompts and each immutable version at /prompts/<slug>/v<number>.

Production:

pnpm build
BENCHMARK_ROOT="$PWD" HOST=127.0.0.1 PORT=9080 pnpm start

pnpm build builds every buildable benchmark preview before building the comparison site. pnpm start verifies those preview artifacts and starts only the comparison site.

Run benchmarks

List canonical tasks:

.agents/skills/run-benchmark/scripts/run-benchmark.sh --list-tasks

Run one model/task combination:

.agents/skills/run-benchmark/scripts/run-benchmark.sh \
  --model gpt-5.6-terra \
  --harness codex \
  --tasks elysian-taste-challenge \
  --effort medium

Run an enabled Markdown matrix:

.agents/skills/run-benchmark/scripts/run-benchmark-plan.sh \
  benchmark-plans/example.md

Every real run must have a session ID, parseable transcript, a captured final response, and task-appropriate build/runtime verification. A successful process exit alone is not a successful benchmark.

Validation

pnpm audit:layout
pnpm test
pnpm typecheck
pnpm lint
pnpm build
pnpm validate:skill
pnpm verify:prompts
pnpm verify:transcripts

Repository hygiene

The Git repository contains source and benchmark evidence, not installed dependencies or reproducible framework output. Generated applications retain their own package.json and pnpm lockfile where available, while node_modules, build caches, local environment files, and nested Git histories stay outside the published repository.

Run pnpm audit:layout after moving or importing benchmark applications. It fails when source files or an unexpected directory are left directly inside a task directory, while preserving the documented seeds and legacy result names.

Deployment

See deploy/README.md for the hardened systemd deployment used by benchmark.melvynx.dev.