Repository navigation
Patch for a knowledge-update regression introduced in 2.0.0. Upgrade if you are on 2.0.0.
2.0.0 shipped rank_weight_recency = 0.5. At that weight a stale memory beat the memory that replaced it whenever the stale row matched one more query token.
Two rows, 947 and 795 days old, asked "frontend framework team":
"The team uses React for the frontend." lexical=1.0000 recency=0.0051 -> 1.3038 ← won
"We migrated the frontend from React to Svelte." lexical=0.4735 recency=0.8412 -> 1.1940
Lexical ratio-to-best handed the older row a 0.53 edge while recency returned only 0.42 after weighting. The row saying the team migrated away from React lost to the row saying it uses React — the knowledge-update case this project is built around, regressed against 1.5.0.
memory_stress_bench.py catches it. 2.0.0 published that suite's result as 23/23 while the shipped code scored 22/23, confirmed by A/B against the commit before the architecture-review branch.
The fix
rank_weight_recency 0.5 → 0.75. The inversion flips below 0.63, so this leaves margin without reaching 1.0, which would assert recency and relevance matter equally — a stronger claim than anything measured here.
| 2.0.0 | 2.0.1 | |
|---|---|---|
| memory stress suite | 22/23 | 23/23 |
| pooled LongMemEval R@5 | 79.1% | 79.1% |
| CodeMemEval R@5, questions as written | 100% | 100% |
| CodeMemEval R@5, questions restated | 75.0% | 79.2% |
| temporal ranking suite top-1 | 80.0% | 80.0% |
Nothing regressed, including the two cases the temporal suite already fails.
The temporal suite cannot select this value — it returns an identical 80.0% top-1 with identical per-category results for every weight in [0.25, 1.5], and its own cross-validation prefers 0.25, which fails the case harder. The choice rests on the two suites that can see the difference.
Also fixed
RankingWeightscarried a second, independent default.from_settingsreads config, but a bareRankingWeights()used the dataclass default — changing onlyconfig.pyleft every caller that skips settings on the old weights. A test now asserts the two agree.ranking.py's module docstring described an algorithm the module does not implement (Reciprocal Rank Fusion plus a bare hyperbolic recency map; the code has used ratio-to-best and a span-weighted blend since the rewrite). It is the first thing read when a ranking result looks wrong, and it sent two separate investigations down the wrong path.- Removed a duplicated
tierassignment infuse().
Docs
REPORT and README carry numbers re-measured against the fixed scorer: the restated CodeMemEval split rises to 79.2% R@5 (19/24, 95% CI [59.5%, 90.8%]) and pooled R@1 to 52.3%.
Tests 417 → 420, including a regression test pinning the 0.63 boundary so a future weight change fails loudly instead of silently degrading knowledge-update recall.
Full changelog: https://github.com/opendb-ai/openDB/blob/main/CHANGELOG.md