Benchmarks

Benchmarks

What has been measured so far, including where Scholion did not help. Details, raw results and how to rerun them are in bench/README.md and bench/retrieval/.

Experiment	Setup	Result
E1: retrieval (offline)	42 tasks from six public repositories; first pack scored against three baselines	The first pack names the files a fix touches best of all methods (0.96–0.98). With BM25 and rank fusion it also shows more of the changed symbols' signatures than BM25 over symbols at 4k and 8k tokens (0.73 vs 0.64 at 4k; level at 2k).
E2: merge fuzzing	About 1,700 random cases of concurrent edits, conflicts and reverts, each checked against a line model and git merge-tree	No lost work, no false conflict, no false clean merge.
E3: agent sessions	Claude Haiku on a 25-file Python project, 160 sessions	About 12 % fewer turns and 14 % less wall time at equal cost (the cost interval contains zero).
E4: knowledge relay	90 sessions on DeepSeek Flash; a later agent gets the earlier agent's lessons	Lessons cut cost per solved task by about 42 % (30/30 solved vs 18/30 without them). Scholion delivered them at the same cost as a plain NOTES.md.
Stress runs	Two eight-agent runs (cheap models, a small Python web app)	Every feature merged through Scholion with main green after every merge. This shows the workflow holds up, not that it is cheaper.

Not yet measured: larger repositories, whether records carry knowledge to the next agent at scale, and whether routing by strength pays off.

Running the harness

scholion-bench check|run|report|calibrate. Only run --confirm spends money; the analyzers are pure. retrieval runs E1 offline (--no-pack needs no daemon), and mine builds retrieval tasks from a repository's history. See bench/README.md.

A recorded run is at https://openevaluations.si/demo: Claude records a decision and a dead end and hands off, Codex picks the task up from a context pack, and Gemini reviews.