Skip to content

v4.1.0 — Retrieval benchmark + deterministic recall

Choose a tag to compare

View all tags
@LuiggiVal08 LuiggiVal08 released this 01 Aug 08:19
v4.1.0
a5dcaa3

What's new

LongMemEval-style retrieval benchmark

Retrieval is now measured against a frozen snapshot of real project memory — the first public retrieval benchmark for toon-memory.

  • Corpus: 187 real data.toon entries (snapshot 2026-08-01)
  • Gold set: 42 hand-authored queries across 6 categories (core-fact, temporal, knowledge-updating, multi-hop, meta/session, distractor)
  • Measured code = production pipeline (src/lib), bundled in-memory with esbuild — no faithful copies
  • Artifacts: benchmarks/retrieval-corpus.toon, gold-queries.json, retrieval-results.json · Reproducible via npm run bench:retrieval
Mode R@5 nDCG@5 MRR@5 answerable
linear 0.643 0.654 0.776 81.0%
rrf 0.861 0.764 0.788 97.6%
smart (unified) 0.829 0.739 0.760 92.5%

Deterministic recall

New optional today parameter on memory_recall / memory_smart_recall pins the reference date for recency, staleness decay, and TTL — fully reproducible results (default stays the wall clock, backward compatible).

Docs

Retrieval metrics block added to the docs home (English; other locales guarded).

Assets 2
Loading