Repository navigation
v1.0.3 — matched baselines: the gap is the two rules, not the ranker
What you will notice
- The comparison no longer flatters KRouter. An outside review showed that the "25% vs 60%" gap was mostly
structural: at their defaults the other systems cannot refuse and do not know a page was replaced. With the same two
rules given to them (superseded pages left out, a refusal floor picked on the dev set only), plain BM25 reaches 19%
wrong citations on held-out and basic-memory 30%. At the same coverage as KRouter they cite slightly fewer wrong
pages (2 vs 3 of 12). The README anddocs/BENCHMARK.mdnow say so plainly: KRouter's advantage on this bench is
those two rules on by default, plus receipts, not a better ranker. Reproduced, not taken on trust; KRouter was
not changed in response, because the held-out set is not a tuning target. - Release cadence written down as practised: minor releases at most weekly, patch releases when ready.
Details
run.py --adapter bm25-floor;bench/memory_honesty/floor.pyapplies "skip superseded + dev-picked floor" to any
adapter output that carries scores; the basic-memory adapter now records scores.- Third-party systems run in a Linux container (
bench/memory_honesty/adapters/Dockerfile): basic-memory was re-run
there, with identical answers to the host run. A container does not share the host's keychain or launchd. - HELDOUT.md log: the reviewer's and the maintainer's baseline runs (aggregates only; no
--reveal; no KRouter
change).