Skip to content

v2.0.1 — a superseded fact could outrank the fact that replaced it

Latest

Choose a tag to compare

@wuwangzhang1216 wuwangzhang1216 released this 04 Aug 20:25
· 4 commits to main since this release
c2a5a66

Patch for a knowledge-update regression introduced in 2.0.0. Upgrade if you are on 2.0.0.

2.0.0 shipped rank_weight_recency = 0.5. At that weight a stale memory beat the memory that replaced it whenever the stale row matched one more query token.

Two rows, 947 and 795 days old, asked "frontend framework team":

"The team uses React for the frontend."          lexical=1.0000  recency=0.0051  -> 1.3038  ← won
"We migrated the frontend from React to Svelte."  lexical=0.4735  recency=0.8412  -> 1.1940

Lexical ratio-to-best handed the older row a 0.53 edge while recency returned only 0.42 after weighting. The row saying the team migrated away from React lost to the row saying it uses React — the knowledge-update case this project is built around, regressed against 1.5.0.

memory_stress_bench.py catches it. 2.0.0 published that suite's result as 23/23 while the shipped code scored 22/23, confirmed by A/B against the commit before the architecture-review branch.

The fix

rank_weight_recency 0.5 → 0.75. The inversion flips below 0.63, so this leaves margin without reaching 1.0, which would assert recency and relevance matter equally — a stronger claim than anything measured here.

2.0.0 2.0.1
memory stress suite 22/23 23/23
pooled LongMemEval R@5 79.1% 79.1%
CodeMemEval R@5, questions as written 100% 100%
CodeMemEval R@5, questions restated 75.0% 79.2%
temporal ranking suite top-1 80.0% 80.0%

Nothing regressed, including the two cases the temporal suite already fails.

The temporal suite cannot select this value — it returns an identical 80.0% top-1 with identical per-category results for every weight in [0.25, 1.5], and its own cross-validation prefers 0.25, which fails the case harder. The choice rests on the two suites that can see the difference.

Also fixed

  • RankingWeights carried a second, independent default. from_settings reads config, but a bare RankingWeights() used the dataclass default — changing only config.py left every caller that skips settings on the old weights. A test now asserts the two agree.
  • ranking.py's module docstring described an algorithm the module does not implement (Reciprocal Rank Fusion plus a bare hyperbolic recency map; the code has used ratio-to-best and a span-weighted blend since the rewrite). It is the first thing read when a ranking result looks wrong, and it sent two separate investigations down the wrong path.
  • Removed a duplicated tier assignment in fuse().

Docs

REPORT and README carry numbers re-measured against the fixed scorer: the restated CodeMemEval split rises to 79.2% R@5 (19/24, 95% CI [59.5%, 90.8%]) and pooled R@1 to 52.3%.

Tests 417 → 420, including a regression test pinning the 0.63 boundary so a future weight change fails loudly instead of silently degrading knowledge-update recall.

Full changelog: https://github.com/opendb-ai/openDB/blob/main/CHANGELOG.md