Repository navigation
Benchmarks
What the auto-pilot modes cost on real projects. Each comparison builds the same README and the same five foundation documents from an empty repository, once per mode, and every product passes its own gates and smoke test at the end.
How to read the numbers:
- Dollars are the API-equivalent cost each run reports, summed from the run records on disk. On a subscription plan it is not what you were billed.
- Agent time is the sum of run durations. Runs did not overlap, so it is also how long the machine was busy.
- All runs used Claude Code with Opus.
A small command-line project, 5 September 2026, effort high. Express was measured before it was built, by pasting its three instructions into the project's own skill files.
| Standard | Express | Express as a share | |
|---|---|---|---|
| Runs | 80 | 24 | 30% |
| Cost | $41.07 | $14.06 | 34% |
| Agent time | 109 min | 37 min | 34% |
| Cards | 39 | 12 | 31% |
| Commits | 14 | 3 |
Both products pass their smoke test. Almost all the saving is break-downs, and the implement, review and checkup runs that each extra card brings with it.
A small browser RPG: a village, a quest, a sewer of rats, save and load. Standard ran from 19 to 27 September; Mini ran on 27 September, twice.
| Standard | Mini, first build | Mini, repeat | |
|---|---|---|---|
| Cost | $470.10 | $7.94 | $8.20 |
| Agent time | 17.4 h | 37 min | 35.5 min |
| Calendar time | 7 active days | 40 min | 36 min |
| Runs | 711 | 3 | 2 |
| Model turns | 8,485 | 224 * | 148 |
| Output tokens | 4.55 M | 0.21 M * | 0.16 M |
| Features / stories / tasks | 14 / 170 / 406 | 10 / 14 / 22 | 7 / 14 / 23 |
| Cards per feature | 42 | 4.6 | 6.3 |
| Stops that needed a person | 17, and spent attempts cleared 5 times | 1 † | 0 |
* Across all eight of that project's runs, including re-tests after the first finished game. † The build used different sprite packs from the ones the documents name, and stopped to say so.
The product came out the same size. Standard's and the first Mini build's games are 79.6 and 79.5 KB of JavaScript gzipped, on the same three runtime dependencies, and both pass tests, lint, typecheck and the browser playthrough. Standard wrote far more tests (1,065 against 123), mostly because it writes one card per acceptance criterion and every card tests its own.
The repeat run is the one in the README: same documents, no person involved, one review round, and the review closed all 44 cards. It leaves out a $0.36 first attempt that could not reach the board API, a bug fixed since.
VibeBoard's loop was being rebuilt while simple-rpg ran, so its $470 spans eight versions of the
machinery. Per feature it closed:
| Version of the loop | Cost per feature closed |
|---|---|
| A: one run per task | $94.86 |
| C: effort medium | $49.26 |
| D: Opus 5.5 | $14.52 |
| E: done means judged | $14.31 |
| F: a leaner harness | $23.35 |
| G: each run sees only its slice of the board | $9.26 |
| H: break-down folded in, fewer comments | $3.37 |
| Mini, first build | $0.79 |
Version B closed no feature. At version H's rate, all 14 features would cost about $47, against Mini's $7.94.
- Mini gives up checks between the build and the review. One run carries the whole plan, and it has no time limit or spending cap while it runs.
- Features are not the same size across modes. The same README became 14 features in Standard and 7 or 10 in Mini.
- These are two small projects. Neither is a large codebase, and Mini's figures come from one kind of project.
- Not measured yet: Express against Mini on the same README, the OpenCode backend, and any model other than Opus.