Skip to content

v1.0.3 — matched baselines: the gap is the two rules, not the ranker

Choose a tag to compare

@398894496-arch 398894496-arch released this 12 Oct 11:14
· 94 commits to main since this release
f7e2df7

What you will notice

  • The comparison no longer flatters KRouter. An outside review showed that the "25% vs 60%" gap was mostly
    structural: at their defaults the other systems cannot refuse and do not know a page was replaced. With the same two
    rules given to them (superseded pages left out, a refusal floor picked on the dev set only), plain BM25 reaches 19%
    wrong citations on held-out and basic-memory 30%. At the same coverage as KRouter they cite slightly fewer wrong
    pages (2 vs 3 of 12). The README and docs/BENCHMARK.md now say so plainly: KRouter's advantage on this bench is
    those two rules on by default, plus receipts, not a better ranker. Reproduced, not taken on trust; KRouter was
    not changed in response, because the held-out set is not a tuning target.
  • Release cadence written down as practised: minor releases at most weekly, patch releases when ready.

Details

  • run.py --adapter bm25-floor; bench/memory_honesty/floor.py applies "skip superseded + dev-picked floor" to any
    adapter output that carries scores; the basic-memory adapter now records scores.
  • Third-party systems run in a Linux container (bench/memory_honesty/adapters/Dockerfile): basic-memory was re-run
    there, with identical answers to the host run. A container does not share the host's keychain or launchd.
  • HELDOUT.md log: the reviewer's and the maintainer's baseline runs (aggregates only; no --reveal; no KRouter
    change).