Repository navigation
memory-bench v0.1
memory-bench v0.1 is a public benchmark for retrieval from an agent's markdown memory. It lives in bench/.
- 202 synthetic notes about a fictional lab, linked with
[[wikilinks]], with no personal data. - 91 questions in seven classes: fact, bridge, paraphrase, paraphrase+bridge, temporal, absent entity, absent attribute. 22 of them have no answer in the vault.
- Metrics: recall@k on support notes, verbatim citation, abstaining on a missing fact.
python bench/run.py --system bm25runs end to end with nothing installed.
Baseline, with its weak spots (leaderboard):
| system | R@5 | answer | abstains when absent |
|---|---|---|---|
| sqlite-graph-memory (vector) | 0.88 | 0.36 | 0.45 |
| sqlite-graph-memory (graph) | 0.85 | 0.36 | 0.45 |
| BM25, no model | 0.77 | 0.45 | 0.36 |
On "what is the current deadline" questions, retrieval is perfect (R@5 1.00). The system still quotes the superseded decision 6 times out of 6.
CI recomputes every number from the per-question outputs. It reran all three rows from a clean index on Linux and matched exactly: drift 0.0000, 0 of 91 rankings changed. A forged result is rejected (#20).
Add your system: bench/CONTRIBUTING.md. Closes #16.
Built by AI agents (Mycroft, Anton's synthetic AI cofounder, with Codex reviewing). Maintained by a human.