Skip to content

memory-bench v0.1

Choose a tag to compare

@tonydzi tonydzi released this 10 Oct 00:54
· 104 commits to main since this release

memory-bench v0.1 is a public benchmark for retrieval from an agent's markdown memory. It lives in bench/.

  • 202 synthetic notes about a fictional lab, linked with [[wikilinks]], with no personal data.
  • 91 questions in seven classes: fact, bridge, paraphrase, paraphrase+bridge, temporal, absent entity, absent attribute. 22 of them have no answer in the vault.
  • Metrics: recall@k on support notes, verbatim citation, abstaining on a missing fact.
  • python bench/run.py --system bm25 runs end to end with nothing installed.

Baseline, with its weak spots (leaderboard):

system R@5 answer abstains when absent
sqlite-graph-memory (vector) 0.88 0.36 0.45
sqlite-graph-memory (graph) 0.85 0.36 0.45
BM25, no model 0.77 0.45 0.36

On "what is the current deadline" questions, retrieval is perfect (R@5 1.00). The system still quotes the superseded decision 6 times out of 6.

CI recomputes every number from the per-question outputs. It reran all three rows from a clean index on Linux and matched exactly: drift 0.0000, 0 of 91 rankings changed. A forged result is rejected (#20).

Add your system: bench/CONTRIBUTING.md. Closes #16.

Built by AI agents (Mycroft, Anton's synthetic AI cofounder, with Codex reviewing). Maintained by a human.