Three different LoCoMo number pairs published across README / docs/benchmarks / CLAUDE.md-referenced baseline — reconcile before arXiv submission

[cdeust]

Raised by an external reader who diffed the repo against numbers I quoted him. He is right, and there are more than the two he found. Verified 2026-08-02.

Three distinct LoCoMo pairs are in circulation

Source	MRR	R@10
README.md:351-352 (headline table)	0.805	91.5%
docs/benchmarks/e1-v3-locomo-results.md:5 (pre-fix sweep)	0.8278	94.2%
docs/benchmarks/e1-v3-locomo-results-post-fix.md:5 (post-fix, authoritative)	0.8279	94.35%
"CLAUDE.md established baseline", cited by three docs	0.794	92.6%

The README publishes the lowest pair while linking, at README.md:521, to the document carrying the highest. A reviewer diffing the paper against the repo lands on this immediately.

The 0.794 / 0.926 pair is cited as the "CLAUDE.md established LoCoMo baseline" by e1-v3-locomo-results.md:6, e1-v3-locomo-results-post-fix.md:6, and e1-v3-locomo-smoke-finding.md:16, but the current CLAUDE.md no longer carries those numbers — it defers to the arXiv sources. So three documents cite a baseline against a location that no longer states it.

Per-mechanism deltas also differ between the two sweeps, with three sign flips

The post-fix re-run (code SHA 2f45bcb39dbe15fa0ef857cc8c8c3783175d05db, a descendant of the plasticity result-shape fix 5f737fe) supersedes the pre-fix sweep (ef178da7, which its own Method section flags as pre-fix). Authoritative values:

• RECONSOLIDATION +0.0091 MRR / +0.0146 R@10 (pre-fix said +0.0076 / +0.011)
• ADAPTIVE_DECAY −0.0163 MRR (unchanged)
• HOMEOSTATIC_PLASTICITY +0.0017 (pre-fix −0.0025 — sign flipped; the negative reading was a plasticity-bug artefact, per -post-fix.md:46,69)
• SCHEMA_ENGINE +0.0017 (pre-fix −0.0004 — sign flipped)
• SYNAPTIC_PLASTICITY also flipped, per the post-fix headline

The qualitative conclusion is unchanged — one mechanism clearly above the ±0.002 per-row floor, one clearly negative, the rest at the floor — but any external citation of a specific delta must come from the post-fix document.

Why this is worth fixing now

Papers are going to arXiv. Two documents describing the same 14-row sweep, differing per-mechanism including in sign, with the README quoting neither, costs credibility for a reason that has nothing to do with the science. The pre-fix run is not wrong; it is superseded, and nothing says so at the top of the file.

Proposed fix

1. Update README.md:351-352 to the post-fix pair (0.8279 / 94.35%), or state explicitly which configuration the README's pair measures and why it differs.
2. Add a supersession banner to the first line of docs/benchmarks/e1-v3-locomo-results.md pointing at the post-fix document.
3. Resolve the 0.794 / 0.926 "CLAUDE.md established baseline" citations — either restore the number to CLAUDE.md with its provenance or repoint the three citing documents at the real source.
4. Diff every LoCoMo/LongMemEval/BEAM figure in docs/arxiv-thermodynamic/main.tex and docs/arxiv-context-assembly/ against the post-fix documents before submission, and record the check.

Acceptance

• Every LoCoMo pair published in the repo traces to one named run with its code SHA, or carries an explicit note of which configuration it measures.
• grep -rn "0.805\|91.5\|0.794\|0.926" README.md docs/ returns only occurrences that are either the authoritative value or explicitly labelled superseded.

[cdeust]

Checked agent/issue-347-post-fix-docs at 7e5cdbfd against this issue's acceptance criteria. The LoCoMo work is done and in one respect done better than asked. Two residuals remain, and the second one says this issue was scoped too narrowly.

Verified done

• 0.805 and 91.5% no longer occur anywhere in README.md or docs/.
• Every surviving 0.794 / 0.926 is relabelled as the historical April 2026 clean-DB comparator (n=1982), traced to commit b4057a, with the disclosure that the original per-query artefact was never committed. That disclosure was not requested here and is the right call: a number whose artefact is lost should be marked unverifiable rather than quietly dropped.
• docs/benchmarks/arxiv-figure-audit-2026-08-02.md records the check, satisfying step 4.

Credit where the fix beat the spec: acceptance criterion 2 as I wrote it was too coarse. Several 0.926 occurrences are not the LoCoMo R@10 at all — they are the BEAM abstention mean retrieval similarity (docs/arxiv-thermodynamic/main.tex:539,546, docs/papers/thermodynamic-memory-vs-flat-importance.md:143). Applying my grep mechanically would have corrupted a correct number and broken a live argument in the paper. The audit doc calls this out at line 29. My criterion was wrong; do not tighten it into a lint rule in this form.

Also checked and correctly left alone: README.md:392 | Temporal reasoning | 0.926 | 98.5% | is a LongMemEval per-category MRR row (n=500), unrelated to the LoCoMo pair.

Residual 1 — an unlabelled current-value claim survives in a paper appendix

docs/papers/appendix-popper-falsifiability.md is not in 7e5cdbfd's changed files, and at lines 70 and 116 it reads "Current scores (LongMemEval 97.8%, LoCoMo 92.6%, BEAM 0.543)" — presented as current, with no historical label, where the authoritative LoCoMo figure is 94.35%. This is the exact defect this issue was filed against, surviving in arXiv-bound material.

Residual 2 — LongMemEval has the same three-way split, and this issue never asked anyone to look

On the fix branch, simultaneously:

Source	LongMemEval R@10
docs/arxiv-thermodynamic/main.tex:50,105,134	98.4%
benchmarks/README.md:87	98.4%
README.md:382	98.2%
docs/papers/appendix-popper-falsifiability.md:58,60,70	97.8%

All cited at n=500. The appendix does not merely mention 97.8% — falsifiability claim C5 is titled with it: "97.8% LongMemEval R@10 generalizes beyond the benchmark". The document whose entire purpose is epistemic rigour is built on a third value that neither the paper nor the benchmarks README carries.

That is my fault, not the fix's: I scoped this issue to LoCoMo because that was the discrepancy I had been handed. The same audit has to run on LongMemEval and on BEAM before submission, or the reviewer who diffs the paper against the repo simply finds the next one.

Suggested closure conditions

1. Resolve appendix-popper-falsifiability.md — either relabel its figures as historical with provenance, or restate C5 against the authoritative values. If C5's number moves, check whether the claim's wording still holds.
2. Extend docs/benchmarks/arxiv-figure-audit-2026-08-02.md to LongMemEval and BEAM, with the same "one named run + code SHA, or explicitly labelled historical" rule.
3. Reconcile 98.4 / 98.2 / 97.8 to one authoritative LongMemEval pair with its run and SHA, exactly as was done for LoCoMo.

[cdeust]

Opus 5 review follow-up is now pushed in b55e206 on draft PR #348.

Resolved:

• stale 92.6% LongMemEval value in the Popper appendix;
• current LongMemEval headline split, now sourced to the clean run as R@10 98.2%, MRR 0.9167, n=500, code SHA 28145f0 (the 98.4% / 0.9124 pair remains only as the explicitly named historical E1 v3 ablation snapshot);
• BEAM provenance, now labeled retrieval-proxy MRR 0.591, n=100, five-conversation protocol, code SHA a071d89;
• invalid comparison between that BEAM proxy and the paper end-to-end metric removed;
• arXiv audit extended across LoCoMo, LongMemEval, and BEAM.

Both arXiv PDFs were regenerated. Local claim checks, diff checks, link checks, LaTeX builds, and extracted-PDF figure checks pass. Final CI on b55e206 is fully green: 19 applicable checks passed, one scheduled-only job skipped, no failures.