-
Notifications
You must be signed in to change notification settings - Fork 0
wiki vs rag qmd followup
Nicolas Cravino edited this page Jul 29, 2026
·
1 revision
id: wiki-vs-rag-qmd-followup title: Wiki vs RAG: Where the Old Approach Still Wins tags: [rag, benchmark, llm] category: agentic-frameworks published: 2026-07-29 created: 2026-07-29 updated: 2026-07-29 freshness: fresh date_source: git-commit source_file: wiki-vs-rag.md
Follow-up to the wiki-vs-RAG benchmark adding a fourth arm: qmd, Tobi Lütke's local-first BM25 + on-device embeddings + local reranker stack. Same 30 queries, same judge — qmd is the cheapest arm and the lowest quality overall.
- qmd costs ~922 tokens / $0.0052 per run with no embedding API, but scores 3.56 overall vs 4.45+ for the stronger systems. [^src:articles-published §wiki-vs-rag-qmd-followup]
- By category: narrow queries roughly tie (~4.9); branch queries drop to 2.83; cross-branch queries to 3.37; against agentic RAG it goes 24 losses, 0 wins, 6 ties. [^src:articles-published §wiki-vs-rag-qmd-followup]
- Failure mode: qmd's reranker ranks peripheral stubs above branch landing pages — a genuine Pareto point for grep-shaped questions, not a drop-in replacement when synthesis matters. [^src:articles-published §wiki-vs-rag-qmd-followup]
- Retrieval design is not one-size-fits-all: different architectures fit different query shapes. [^src:articles-published §wiki-vs-rag-qmd-followup]