wiki vs rag.stub

---

id: wiki-vs-rag name: Wiki-fed Agents vs. RAG Benchmark repo_path: ~/Documents/sw30labs/repos/wiki-vs-rag remote_url: https://github.com/sw30labs/wiki-vs-rag.git primary_language: Python framework: Anthropic SDK + OpenAI embeddings + FAISS + QMD category: agentic-frameworks stacks: [agentic, rag] ingested: 2026-04-22 last_commit_date: 2026-04-20

Purpose

Reproducible head-to-head benchmark of four retrieval/agent patterns over the sw30labs/.github.wiki corpus pinned at commit c130a3b (89 markdown files, ~45 K corpus tokens). Compares single-shot RAG (FAISS top-k), agentic-RAG (same FAISS, ≤8-turn tool loop), wiki navigation (precomputed sitemap + page-read tools), and QMD (local hybrid retrieval — BM25 FTS5 + EmbeddingGemma + Qwen3 reranker, no embedding API). Headline finding: agentic-RAG is the Pareto winner — same quality as wiki-nav at 40% of the token cost (4.77 vs 4.69 quality, $0.029 vs $0.064 per query). QMD is the cheapest arm ($0.005/query, 0 API spend) and ties on narrow factoids but collapses on branch and cross-branch synthesis (quality 2.83 / 3.37 vs 4.5+ for the other three) — confirming its "AI grep" framing rather than RAG-replacement.

Architecture

• Phased Makefile pipeline: index (FAISS) → sitemap (hierarchical) → qmd-index (BM25+vectors+reranker) → queries (Opus synthesizes 30 ground-truthed queries) → eval (run all four agents) → judge (Opus 1–5 absolute scoring) → report (PNGs + CSV + summary.json).
• Four agent modules under rag/, rag_agentic/, wiki/, qmd/ — each exposes a single agent.py with the same interface, so eval/run.py iterates uniformly.
• Idempotent targets: each phase guards its output file; partial reruns work without recomputing upstream artefacts.
• Section-aware chunking (rag/chunker.py): 800-token window, 150-token overlap; section headings preserved as chunk metadata.
• Tool loop budget: agentic arms cap at ≤8 turns; the loop terminates when the agent emits a final answer or hits the budget.
• QMD bootstrap is intentionally out-of-band (make qmd-setup): installs Bun + @tobilu/qmd globally and downloads ~2 GB of GGUF models on first use — kept out of make all to avoid surprising contributors.

Dependencies

• anthropic>=0.40 — Claude Sonnet 4.6 (agents, temp 0) + Claude Opus 4.7 (judge + query synthesizer)
• openai>=1.50 — text-embedding-3-small for the FAISS RAG arms
• faiss-cpu>=1.7 — IndexFlatIP cosine similarity for RAG and agentic-RAG
• tiktoken>=0.7 — token counting for cost accounting
• @tobilu/qmd (npm, external) — local hybrid retriever; pulls EmbeddingGemma + Qwen3 reranker GGUFs
• numpy, matplotlib, pyyaml — eval bookkeeping and chart rendering

CI/CD

No GitHub Actions. Reproducibility is via the Makefile + checked-in artefacts (indices, sitemaps, run logs, judgments, charts). Full pipeline runtime ~10 min, ~$8 of API spend.

Key findings

• Agentic-RAG beats single-shot RAG 11–2 (17 ties) on 30 queries with mean Δ +0.32 quality, costing 4.2× more tokens.
• Agentic-RAG vs wiki-nav: 10–6 (14 ties), Δ −0.08 — statistically a tie on quality, but agentic-RAG runs at 45% of the cost.
• QMD wins 0/30 head-to-head against agentic-RAG (24–0, 6 ties). On narrow factoids it ties (4.47 vs 5.00); on branch and cross-branch it loses by ~1.2 quality points.
• Two diagnosed QMD failure modes: compact-snippet trap (reranker returns front-matter rather than chunk body) and missing landing pages (BM25+rerank scores leaf stubs higher than the branch overview the question needed).
• Setup-cost asymmetry called out as out-of-scope: RAG embedding build = $0.0009 / 9.1 s; QMD = $0 API + ~2 GB local model download + 9 s local compute.

Entities and topics

• agentic-frameworks — primary category landing page
• agent-stack — the 10-layer mental model this benchmark instantiates at the retrieval layer
• deepagent-azure-cli — another tool-loop agent in the same family
• wiki-curator (this skill) — the wiki being benchmarked is built and maintained here
• Tags: [python, llm, rag, faiss, openai, benchmark, langchain]