Skip to content

v0.1.0 — Small-model recall + benchmark-driven retrieval

Choose a tag to compare

View all tags
@fonsecabc fonsecabc released this 02 Jul 05:58
· 40 commits to main since this release
v0.1.0
6d9a96d

First tagged release. Focus: make recall fit small / local models, and put retrieval quality under continuous measurement.

Highlights

  • Compact recall output — memory_recall gains format:'compact' + maxChars: terse, id-free text with relations rendered by name, bounded to a hard character budget. JSON stays the default, so tool-using clients are unaffected.
  • Retrieval quality overhaul — weighted Reciprocal Rank Fusion, own-relevance-gated graph spreading activation, an exact name/alias match bonus, and an optional local-LLM reranker (RERANK_ENABLED). Every change is gated by an option, so the prior behaviour is exactly recoverable.
  • RAG benchmark (packages/core/src/eval/) — a synthetic, privacy-safe brain + 40 gold queries across 9 query kinds + standard IR metrics (Recall@k / MRR / nDCG), runnable against local Ollama, with a hermetic CI regression guard.
  • Env-configurable recall defaults (TARS_RECALL_*).

Benchmark results

Measured on the bundled synthetic benchmark (40 queries over a 26-entity abstract graph; local Ollama nomic-embed-text embeddings). Headline = a top-heavy nDCG@3 composite, which models what a small model with a tight context budget actually reads.

Full pipeline vs the original retriever

Composite hit@1
Original retriever 0.737 0.65
Improved pipeline 0.964 0.93
Δ +22.7 pp +28 pp

Isolating the LLM reranker (retrieval tuning held constant)

Query kind Without reranker With reranker
exact 1.00 1.00
alias 1.00 0.89
semantic 0.71 0.99
attribute 0.85 1.00
multi-hop 0.64 0.94
two-hop 0.66 0.88
collision 0.28 1.00
negative 0.89 1.00
temporal 1.00 1.00
overall 0.743 0.964

Reranker delta: +22.1 pp (0.743 → 0.964, reproducible at temperature 0). The reranker is what cracks relational ("who does X's manager report to") and name-collision ("the Alex who works at Acme") queries that pure lexical/vector/graph ranking cannot.

Enabling

  • Compact output: pass format:'compact' per call, or set TARS_RECALL_FORMAT=compact.
  • Reranker: RERANK_ENABLED=1 (optional OLLAMA_RERANK_MODEL, default qwen2.5:14b-instruct). Off by default — it adds one local LLM call per recall, worth it on the small-model serving path.

Notes

  • The numbers come from a synthetic benchmark and measure the query shapes it covers; re-measure against real data as a brain grows.
  • Retrieval runs fully local (Ollama); no data leaves the machine.

🤖 Generated with Claude Code

Assets 2
Loading