Skip to content

Benchmarks

Arun Soman edited this page Aug 31, 2026 · 1 revision

Benchmarks — headline numbers, all measured

Raw methodology + artifacts stay in the repo: docs/benchmarks.md, docs/benchmarks-requests.md, semantic raw JSON next to them. Dated records live in the CHANGELOG.

What Number Where measured
Semantic retrieval recall@8 = 1.0 on 16/16 labeled questions (requests + urllib3); hybrid MRR 0.854 / 0.781 docs/benchmarks-semantic-*.json
Caller resolution vs grep 1 cie call vs 3 grep+reads, still not guaranteed correct docs/benchmarks-requests.md (includes the 3-of-6 under-resolution miss)
Index speed 110 files → 1,902 nodes / 6,581 edges in ~1.9s the README demo take (uncut cast in docs/demo)
Surface health 135 tools, conformance: 0 crashes, honest-empty envelopes tool-test-lab harness
TESTS edges after v0.1.4 1 → 562 on cie itself CHANGELOG [0.1.4]

The losses and ties are published too — the benchmarks docs report a tie and a real loss, not just the wins.

Clone this wiki locally