A blind-rebuild benchmark for coding agents: distill a real app into a dataset of requirements + deterministic checks, hand an agent only the public requirements, and score how faithfully it rebuilds the app — by behavior and pixels, never by reading its code.
The full design lives in MANIFESTO.md, and a feature-by-feature architecture
walkthrough (with diagrams) lives in TECHNICAL.md. This README covers what is
built today and how to run it.
The core engine is implemented for the web surface (MANIFESTO §15 step 1): the
protocol, the black-box defineCheck contract, a Playwright web driver, the screen-space
visual oracle, and deterministic scoreRebuild. It is exercised end-to-end by a hand-written
toy dataset (datasets/toy-web, a tiny 2048).
The multi-iteration feedback loop (build → score → QA hints → fix → revise → re-score) is
implemented: with --max-iterations > 1, a non-scoring QA role turns each unmet requirement's
evidence into a no-leak hint that explains in detail how the observed behavior diverges from the
requirement (describing the phenomenon, not how to fix it), and the developer hands the agent just
that feedback as the next round. To mimic real agent usage, the system keeps one conversation
across the loop: it persists its coding-agent session and resumes it each pass (--resume), so the
build prompt and conformance contract are sent only once and every fix pass revises the existing
app/ in place with full prior context — repeating until every requirement is met or the cap is hit.
The review and audit report lanes are implemented too: after the loop, an LLM review writes a
per-iteration and overall narrative root-cause, and an audit flags check-quality smells (both
report-only — they never change the mechanical score, and run on a dedicated --review-model). Every
run also records provider-reported tokens/cost split by harness vs system lane, the exact
model each provider used, timestamped conversation dumps, and a snapshot of each iteration's
app/.
Deferred (not yet built): non-web surfaces.
- Node ≥ 22, pnpm
- Playwright's Chromium:
pnpm exec playwright install chromium
pnpm install
pnpm build # tsc -b across the workspace
pnpm test # node:test suiteThe engine has two halves — offline authoring (turn a real app into a gated dataset) and the online run (rebuild it blind, then score):
# 1. Fill: distill a real app into a gated + locked dataset. An LLM QA author writes the
# requirements, a single build prompt, conformance protocol, design spec, and one check per
# requirement; each authoring stage self-repairs against the live original (design colors,
# check type-checks, and the reference gate) before the dataset is locked.
pnpm fill --repo <dir|url> --out datasets/<slug> --surface web --model sonnet
# 2. Gate + lock an existing dataset: record references from the ORIGINAL, prove every check
# passes on it (a check that fails on the original is broken), then stamp + write checks.lock.json.
pnpm tsx scripts/gate.ts --dataset datasets/toy-web --app examples/toy-app
# 3. Score a rebuilt app against a validated dataset (quick dev view).
pnpm score --dataset datasets/toy-web --app <rebuild-dir>
# 4. Online run (§10): developer delivers the dataset's single build prompt → a system rebuilds the
# app blind in a Docker container → score → review + audit → emit the run directory (reports,
# per-lane metrics, conversation dumps, per-iteration app snapshots) under
# results/<dataset>/<runId>/ — a permanent, git-tracked home; handpick which runs to commit
# (see results/README.md). Override with --out.
pnpm benchmark <repo-url> --system-model sonnet
pnpm benchmark <repo-url> --system-model sonnet --max-iterations 3
pnpm benchmark <repo-url> --system-model sonnet --review-model opus
pnpm benchmark --list-systems # list available systems
# 5. Build the results viz site: snapshot completed runs into a static, publishable site, then
# preview it locally (a landing page + a reports browser with full per-run detail: scores,
# verdicts, metrics, review/audit, reference/candidate/diff screenshots, the build code, and a chronological
# conversation timeline: build prompt → agent's build → review → audit).
pnpm site # generate apps/site/data/ from results/**
pnpm site:serve # preview at http://localhost:4173fill needs the chosen model's API key (or pass --subscription to bill a logged-in Claude
Pro/Max plan via the claude CLI — no API key needed), and exits 0 only if the gate is fully green;
pnpm validate --dataset <dir> re-gates a dataset
after edits with zero LLM calls. Pass --fake <ref-dataset> to fill to replay an existing
dataset's artifacts tokenlessly.
Pass --reference to score to inject the dataset's reference adapter — use it only when
scoring the original app (it exposes window.__bench over code a rebuild must implement
itself). score exits non-zero unless every requirement is met.
The online run resolves the dataset by repoUrl (or --dataset <dir>), needs Docker plus the
chosen models' API keys, and runs --max-iterations feedback passes (default 1 = a single
build → score), then a review + audit pass. --system-model is required; the developer
--authoring-model (alias --benchmark-model) defaults to opus, and --review-model (review + audit) to the authoring model. Each pass writes verdicts-NN.json /
hints-NN.json / report-iter-NN.*, conversations/{harness,system}-NN.jsonl, and
apps/iter-NN/; the aggregate report.md shows models (exact), per-lane metrics
(tokens/cost/duration), the review narrative, audit flags, and the iteration
progression. Try score against the bundled apps: examples/toy-app scores 5/5;
examples/toy-app-broken (a deliberately broken merge) scores 3/5.
packages/
protocol/ surfaces, state schema, design spec, requirements, verdicts, conformance prose
check/ the black-box defineCheck SDK + CheckContext
harness/ dataset loader, driver registry + web driver, visual oracle, scoring, reports,
file bus, developer role, run lifecycle
llm/ provider-agnostic LLM client + model roster resolution
apps/
site/ static results-viz site (landing + reports browser); data/ generated by `pnpm site`
datasets/
toy-web/ hand-written toy dataset (manifest, requirements, prompts, checks, references, lock)
examples/
toy-app/ a tiny working 2048 (the "original")
toy-app-broken/ the same app with a seeded defect (for testing the scorer)
rosters/
bare-agent/ baseline system: wraps a coding-agent CLI in a Docker container
superpowers/ bare-agent + the obra/superpowers Claude Code plugin (skills framework) seeded in
models/ per-provider model rosters (alias → provider/key/concrete id)
results/ permanent, git-tracked home for benchmark runs (handpick what to commit)
scripts/
gate.ts / score.ts / benchmark.ts the dev entry points above
build-site.ts / serve-site.ts generate + preview the apps/site results viz
MIT