Benchmarks

Benchmarks

Every published number has a JSONL receipt in benchmarks/*/results/. Two harnesses produce them. Nothing is edited by hand.

billing-sweep — is the cost math exact?

Sends real pages to the provider's count_tokens endpoint and compares the measured token count against the predicted formula, across geometries and models. The formula ships only when the residual is zero.

node benchmarks/billing-sweep/run.mjs --dry-run   # predictions only, $0

density-frontier — how much can a page hold and still read at 100%?

Renders needles (hex / camelCase / digit strings) among distractors at increasing density, then asks the model to read them back. It measures the frontier where accuracy is still perfect — and, crucially, whether a model abstains when it can't read (good) or confabulates a wrong answer (disqualifying).

pnpm exec tsx benchmarks/density-frontier/run.ts --dry-run   # cost table, $0

What the receipts show

Claim	Measured
Sample UI render	1015 chars → 438×120 PNG, 254 → 84 tokens (66.9% saved)
Standard page cost	1568×728 = 1456 image tokens, flat
Claude Fable 5 read accuracy	100% at production density
Opus 4.8	77–87% only at 4× glyph size (opt-in safe mode)
GPT-5.5	0/60 — blocked
Gemini 2.5-flash	confabulates — blocked

Text vs. image, visually

same 28,080-char context

 as dense TEXT   ██████████████████████████████████████████████  ~14,040 tokens
 as ONE IMAGE    █████                                              1,460 tokens   (one 1568×728 page)
                                                                   ▼
                                                             ~10× fewer tokens

Ground rules

• Measurement before claims. A number without a receipt does not go in the docs.
• Measured decisions are not re-litigated without new data (see the decision log in ROADMAP.md).
• Never touch benchmarks/*/results/ except by running the harnesses.

Live runs need an API key (env only), or run at $0 via the Claude Code CLI (--via-cli) or through OmniRoute (--via-omniroute). Full methodology: BENCHMARKS.md.