Skip to content

v0.2.0 — an eval that says no, and an MCP server that says nothing correctly

Choose a tag to compare

@tonydzi tonydzi released this 20 Sep 11:22
· 120 commits to main since this release

Two halves shipped since v0.1.3: a retrieval eval that exists to talk us out of upgrades, and an MCP server you can actually point a harness at.

The eval, and the seven upgrades it talked us out of. eval/build_gold.py turns a vault's own [[wikilinks]] into a retrieval test set — no hand labelling, no LLM calls — and eval/run_eval.py scores it as Recall@12 / MRR / nDCG@12 in both modes, vector-only and vector+graph. It imports the pipeline rather than copying it, so what it measures is the thing that ships. Four question classes, because "does the graph help?" has four different answers: title, body, bridge, temporal.

The README now carries real numbers from the home vault: two changes that improved things (bridge Recall@12 0.251 → 0.451) and seven that obviously should have helped and measurably did not — BGE-M3, two rerankers, a BM25 hybrid, a heading-path chunker, sqlite-vec, and more. That list is the point of the module.

The ruler was broken, and an outside panel found it first. Before announcing the eval anywhere it went to two independent review engines, and both landed on the same defect: ndcg_at credited the same gold note once per appearance in the ranking, so ndcg_at(['foo','foo'], ['foo']) returned 1.63. Wikilinks address notes by filename, so two folders holding a foo.md is not a corner case. Fixed, with the published numbers corrected downward (title nDCG 0.921 → 0.890, temporal 0.638 → 0.531). The deltas survived to the fourth decimal, because the same four questions inflated both the before and the after run — so the merge/revert decisions taken from those runs still stand. EVAL_MUTANT=1 breaks nDCG on purpose and the suite must go red under it: a test that stays green while the metric is broken never tested the metric.

The MCP server got a runtime and three fixes to how it reports nothing. examples/ now carries a runnable server and its own README (#5). The answer file is scoped per call, so two recalls in flight can no longer return each other's text. An empty answer file used to fall back to whatever was on stdout, which meant a log line could come back as a recall result — it returns isError now (#9) — and a file it could not read is reported separately from a recall that genuinely found nothing (#13).

The paper is published: JOSS draft plus CITATION.cff, DOI 10.5281/zenodo.22639718.

Full journal: CHANGELOG.md


Built in the open by Anton Dzyatkovskiy and Mycroft, his synthetic co-founder. More of the same at https://github.com/tonydzi — if you run agents with long memory and want a second pair of eyes on this, the issues are open.