Evidence Map

Evidence Map

A routing table for a sceptical reader. Every published claim, where it was measured, and the command that reproduces it.

This page deliberately contains no figures. The numbers live in the repository, versioned alongside the code that produced them, and quoting them here would create a second copy with no CI, no review, and no link to a commit. Follow the links.

---

Where each claim is measured

Claim	Measured in	Reproduce with
Hybrid retrieval + reranking beat dense-only — and where they don't	FINDINGS §1, RESULTS	make eval
A fixed gap threshold does not transfer across embedders	FINDINGS §2	make eval
The in-sample correction to the original calibration claim	FINDINGS §2b	python -m recall.eval.calibrate
Fine-tuning is a null on a rich corpus	FINDINGS §3, RAG_TRAINING_STUDY	python finetune/train.py
Fine-tuning does pay on an opaque-jargon corpus	RAG_TRAINING_STUDY	python finetune/train.py --corpus finetune/confusable_corpus
Validity beats similarity — the trust layer kills stale-memory false positives	FINDINGS §4	make eval
Timestamps cannot replace declared supersession (steelmanned)	FINDINGS §4, ENTAILMENT_SUPERSESSION_STUDY	make eval
The near-miss class needs a judge — and the judge needs the threshold	FINDINGS §5, ENTAILMENT_SUPERSESSION_STUDY	make eval
The supersession result holds at scale, at full coverage	FINDINGS §5b	python -m recall.eval.scale
The abstention threshold, measured and then rebuilt	FINDINGS §6	python -m recall.eval.calibrate
Real-corpus retrieval on paraphrased questions, not headings	FINDINGS §7	python -m recall.eval.labelled
Reranking, candidate pool and chunk size were not the lever on that corpus — where the right document was often not retrieved at any depth	FINDINGS §7	python -m recall.eval.labelled
Reranking is the lever on the standard public benchmark — the largest single retrieval gain measured here, on the default local embedder	RESULTS §11, FINDINGS §11	python -m recall.eval.locomo --rerank
A bigger cross-encoder buys nothing over the shipped one	RESULTS §11	python -m recall.eval.locomo --rerank --reranker-model …
The embedder is the lever — and the win is corpus-specific	FINDINGS §8, gap study	see the PEPs command below
Head-to-head against Mem0 on LOCOMO — accuracy, cost and speed	FINDINGS §9d, RESULTS §9	benchmarks/
MTRAG external benchmark: RE-call retrieval is close to the benchmark retrieval, does not top the benchmark, and correct refusal is scored by an independent judge	docs/MTRAG_BENCHMARK.md, results/mtrag_generation/	commands in the MTRAG document
SPLADE and Voyage rerank on MTRAG Task A	docs/MTRAG_BENCHMARK.md §2	benchmarks/mtrag/run.py and committed Task A artifacts
Hosted Voyage rerank beats the shipped local reranker on the ladder, while abstention separation does not follow	RESULTS §12, FINDINGS §12	python benchmarks/rerank_pool_arms.py
Scale and latency characteristics	results/scale/	python -m recall.eval.scale --embedder hashing --filler 50000
Production posture, itemised	README	—

The public replication anyone can check

The strongest single thing to verify, because the corpus is public and the labelled questions ship in this repo:

git clone --depth 1 https://github.com/python/peps
python -m recall.eval.labelled --corpus peps/peps \
  --questions recall/eval/peps_questions.json --glob '**/*.rst'

---

What was corrected or withdrawn

The current correction history lives in CHANGELOG.md, results/FINDINGS.md and the git log. Earlier versions also carried a README withdrawal list; the durable source now is the versioned result narrative plus the changelog.

Those correction notes are the place to read if you are assessing rigour. A claims table without withdrawals is marketing; a project that publishes its own retractions is doing something else.

Categories of withdrawal, as a guide to what to look for:

• a threshold fitted and scored on the same samples (on separable data that result is arithmetic, not evidence);
• "unanswerable" queries that were an answerable query plus a nonsense suffix, so nothing could separate them;
• retrieval measured using document headings as queries — which is known-item retrieval, not the task;
• a conclusion drawn from one corpus that a larger corpus family restated;
• raw MTRAG metrics compared against conditioned baselines, which flattered RE-call until the join and conditioning path were audited.

---

How the evaluation is built

Three properties do most of the work, and they are the transferable part.

Held-out measurement. The fine-tuning study trains on one set of queries and measures on a held-out set of differently-phrased queries for the same documents — so a lift is generalisation, not memorisation. The calibration rule was validated by fitting on half the queries and scoring on the other half, across multiple index rebuilds, because HNSW builds are nondeterministic and a single-build result would confound the rule with the index.

Hypotheses eliminated one at a time. On the real-corpus investigation, four candidate levers were tested individually against the same held-out questions. Three were eliminated; one was confirmed. The eliminations are what make the fourth result a diagnosis rather than a lucky guess — and the diagnostic move that made it interpretable was measuring recall far beyond the top-k, which showed the right document was often nowhere in the candidate pool at all. A reranker only reorders what was retrieved, so no amount of reordering could have helped.

Analytic baselines are labelled as such. Where a reference point is true by definition rather than observed — a system with no gap guard never abstains, so its false-confidence rate is 1.00 by arithmetic — it is marked in the results table rather than presented as a measurement. Likewise a metric that is mechanically capped by the experiment's design is flagged, so it is not read as poor performance.

What the evaluation still cannot measure

Stated in FINDINGS §"What this evaluation still cannot measure" and echoed in the README's "What this does not do". The main ones:

• Successor and abstention accuracy on generated corpora are not meaningful. Every synthetic document is the same sentence with a different opaque token, so those columns measure token discrimination, not the trust layer. Supersession-trust rate, latency and scale figures are unaffected — the distinction matters and is stated per column.
• Gap detection is bounded by the embedder. With a weak one, no threshold separates answerable from unanswerable. Measured, not assumed.
• ANN recall is tuned on the filtered path only. Every query now also carries a tenant predicate, and that combination has not been measured on a multi-tenant table. → #11
• Validity is authored, not inferred — and the tooling to recover undeclared edges could safely declare none of them on a real corpus. → #29

Reproducing everything

make eval                                                     # ablations + trust + near-miss
python -m recall.eval.scale --embedder hashing --filler 50000  # scale + latency

make eval needs the same Docker Postgres as the test suite and runs key-free with the local embedder; the cloud-embedder row appears only when an API key is set. The larger LOCOMO, MTRAG, ladder and promotion studies have their own scripts and retained artifacts linked above.

---

See also: The-Problem for what these measurements are testing · The-Trust-Layer for the mechanisms behind them · docs/WRITEUP.md for the design story · docs/CASE_STUDY.md for where the real corpus came from and exactly what is public versus private.