Measured Results

Measured results

Goal of this page: every claim this project makes, where it was measured, and what it does not prove.

House rule: every number here is either measured and cited, or explicitly labelled a target. Nothing is estimated. The counting uses the exact cl100k_base tokenizer, not an approximation — and a helper in the code refuses to let a benchmark report savings computed from the estimator.

Reproduce all of it:

uv run --python 3.12 --with pytest --with pytest-timeout --with tiktoken pytest tests/ -q -s

---

From the test suite

Claim	Test	Target	Measured
Tool descriptions re-sent every turn	test_server_budget.py	≤600 tokens	469
Compact table vs JSON, 30 records	test_render.py	≥35% smaller	44.6% smaller (516 vs 932)
Cost to resume from a mid-run board	test_resume.py	<3,000 tokens	2,679
That cost stays flat as history grows	test_resume.py	≤1.35×	1.00× (2,679 → 2,679 across 120 added entries)
Concurrent writes: exactly one winner of eight	test_concurrency.py	1	1 (the other seven rejected, each carrying the current version)
No version lost under 30 concurrent writes	test_concurrency.py	30 in history	30
Network connections in local mode	test_local_posture.py	0	0

The headline: resume cost does not grow with the board

Board content	Reading everything	Resuming	Ratio
10,506 tokens	10,506	2,679	25.5%
229,566 tokens	229,566	2,679	1.2%
448,626 tokens	448,626	2,679	0.6%
667,686 tokens	667,686	2,679	0.4%
886,746 tokens	886,746	2,679	0.3%

The honest reading: resume cost is flat, so the size of the saving is a function of how much work the board holds. The ratio is a property of the workload, not of this software. What the software guarantees is the flat line.

One live five-agent run — not from the test suite

Everything above comes from tests. This does not: it is a single live run on 2026-09-07 with real agents doing real analysis, recorded because the four-arm benchmark had produced nothing yet and one honest data point beats an assumption. n=1. Treat it as a data point, not a result.

Five agents answered five questions about this repository. Identical prompts and identical requested depth in both arms; only the reporting path differed. Arm A replied in prose. Arm B wrote a summary plus a body to the board and replied in three lines.

	Arm A (no board)	Arm B (board)	Change
Parent agent context	36,766 tokens	2,357	−93.6%
Total spend inside all agents	607,120 tokens	372,112	−38.7%
Wall clock, slowest agent	483 s	376 s	−22.2%

Quality held. Both arms independently found the same core defects. Arm B additionally found three that Arm A missed, including that the server resolved an access grant once at process start — making revocation and expiry inert for a running server.

A prediction this run falsified. The expectation was that Arm B would cost more internally, since each worker loads about 2,600 tokens of protocol before starting. It cost 39% less: composing a 9,500-token prose report is more expensive than writing a summary plus a body, and the write cost dominates the load cost.

What it does not establish. One run, one repository, one task shape. No repetition, so run-to-run variance is unmeasured. Both arms used the same model, so this isolates the reporting path and nothing else. Arm A's prose is richer per finding; the honest claim is that the summaries answered the same questions at decision level, with detail retrievable at 3,000–6,000 tokens per entry.

What a write costs, by shape

Write	Before	After	Saved
A set of 12 findings — objects → columns+rows	536	385	28%
A body that already summarizes itself → digest_from	589	489	17%
One small fact — body restating the summary → summary alone	101	58	43%
Adding one finding — re-read and re-emit → append	1,209	78	94%
The result echo alone — summary echoed back → not echoed	81	26	68%

Four writes end to end: 2,435 → 1,010 tokens, 59% less. Strip the append row and the other three save 22%.

Each row counts a whole write — arguments out plus result back. The 34% quoted elsewhere for columns+rows measures the row payload alone, which is why the same change reads lower here: the summary and the reply are identical on both sides and dilute the ratio. Neither number is wrong; they measure different brackets.

What it cost to get there: the per-turn tool surface grew from 395 to 469 tokens (budget 600, so 131 left), and the test count grew from 84 to 120. That 74 tokens a turn is paid by every agent whether or not it uses a new shape. It sits in the cacheable prefix, so the real bill is smaller than the number suggests — but it is a standing cost, and the next argument for that surface has to displace something.

Corrections that measurement forced

The token estimator was wrong by up to +106%, not the ±15% originally documented. Caught by validating it against a real tokenizer. An estimator that over-counts by 2× would have truncated every read at half its real budget and inflated every benchmark figure. It was rebuilt and refitted: max +26.2%, mean 11.6%, biased high — the safe direction, so budgets under-fill rather than overflow.

The tool surface was 1,036 estimated tokens on first write, over the 600 budget. The build test rejected it. The fix was moving prose out of the machine schema into the protocol instructions, which load once per session instead of being re-sent every turn. Real measurement afterwards: 395.

Bugs the tests found before any agent did

Bug	Found by	Fix
Concurrent writers of identical content raced on one temporary path and renamed a file that had vanished	a concurrency test	a unique temporary name per writer
Exporting a workspace and importing it under a different name was rejected, making a "portable dump" unportable	a command-line test	rebase the workspace on import; a flag for exact restore
A topic pattern did not match the bare topic itself	an address test	the pattern now compiles to make the tail optional

Not yet measured

The four-arm comparison, task success rate, and the other planned metrics in the benchmark plan. The single live run above covers roughly one cell of that grid, on one task, at n=1, and does not substitute for it. The stop criteria — what result would mean this project should be abandoned — are stated in advance in docs/10-is-it-worth-building.md and decided from those runs, not from this page.