Skip to content

v2.1.2 — LongMemEval answer accuracy with 2.1.1's pages: 449 of 500

Latest

Choose a tag to compare

@398894496-arch 398894496-arch released this 13 Nov 11:19
· 78 commits to main since this release
1b25b7e

2.1.2 — 2026-11-13

What you will notice: nothing in how KRouter looks things up; the engine is the same as 2.1.1. This release carries the
set 22 run and the docs that report it. With 2.1.1's default pages (about 5.6 a question), Claude Sonnet 5.5 as reader
and as judge, run once: 449 of 500 (89.8%), and on the held-out half 201 of 235 (85.5%). Plain BM25 was not rerun
at the same page count, so there is no keyword-search row to set beside it.

Set 22 run (2026-11-13): LongMemEval S answer accuracy with 2.1.1's default pages (about 5.6 a question), Sonnet 5.5 reader and judge as in sets 11 and 12: 449 of 500 (89.8%, 87.2–92.4), held-out half 201 of 235 (85.5%); 1.3.0 at three pages was 400 of 500. BM25 was not rerun at the same page count. Results: bench/longmemeval/results/2026-11-13-sonnet-qa.set22.jsonl.

server.json now points at v2.1.1 (KRouter.mcpb, SHA-256 a2d4ffe3…, checked against the published SHA256SUMS); nothing else changed.