rag-demo

End-to-end RAG pipeline demo. Wires together several focused utilities into a working question-answering agent, then measures what it actually produces.

corpus → BM25 index → retrieval tool → ToolLoop agent → answer
                                                        ↓
                               rageval: faithfulness, context recall, NDCG
                                                        ↓
                               agent-scratchpad: observations persisted to disk

What's wired together

Repo	Role in this demo
tool-loop	Agentic loop — model calls search(), gets passages, answers
rag-eval	Measures retrieval recall and answer faithfulness per query
agent-scratchpad	Stores Q&A observations; demo shows retrieval from memory
llm-gateway	Rate limiting and cost tracking around the Anthropic client

Setup

git clone https://github.com/shadowmodder/rag-demo
cd rag-demo

# Install the utility libs from source
pip install -e ../tool-loop
pip install -e ../rag-eval
pip install -e ../agent-scratchpad
pip install -e ../llm-gateway
pip install anthropic

Run

export ANTHROPIC_API_KEY=sk-ant-...
python demo.py                      # all 6 queries, k=3 retrieved docs
python demo.py --queries 2          # first 2 queries only
python demo.py --dry-run            # skip API calls, test the pipeline end-to-end

Sample output

Corpus: 15 documents

Running queries...

Q1: Why does attention scale quadratically?
  retrieved: ['Attention complexity', 'Flash attention', 'Transformer architecture']
  recall=1.00  faith=0.84  relevance=0.71  (1243ms)

Q2: How does prompt caching reduce cost?
  retrieved: ['Prompt caching', 'Attention complexity', 'Evaluation without labels']
  recall=1.00  faith=0.91  relevance=0.67  (987ms)

Q3: What metrics measure RAG retrieval quality?
  retrieved: ['RAG retrieval quality', 'BM25 retrieval', 'Precision vs recall tradeoff']
  recall=1.00  faith=0.88  relevance=0.73  (1104ms)

Q4: How does token bucket rate limiting work?
  retrieved: ['Token bucket rate limiting', 'Attention complexity', 'Evaluation without labels']
  recall=1.00  faith=0.79  relevance=0.62  (1056ms)

Q5: What is the difference between Platt and isotonic calibration?
  retrieved: ['Model calibration', 'Isotonic regression for calibration', 'Transformer architecture']
  recall=1.00  faith=0.85  relevance=0.68  (1189ms)

Q6: How do you accumulate streaming tool calls?
  retrieved: ['Streaming and SSE', 'Tool use in agents']
  recall=1.00  faith=0.77  relevance=0.59  (1312ms)

── Summary ──────────────────────────────────────────────
Q   recall  faithfulness  relevance  ms
--  ------  ------------  ---------  ----
Q1  1.00    0.84          0.71       1243
Q2  1.00    0.91          0.67        987
Q3  1.00    0.88          0.73       1104
Q4  1.00    0.79          0.62       1056
Q5  1.00    0.85          0.68       1189
Q6  1.00    0.77          0.59       1312

Memory: 6 observations stored in demo_memory.json

── Scratchpad recall: 'quadratic attention' ──────────
  [0.847] q1_answer: Q: Why does attention scale quadratically?
A: Self-attention computes pairwise dot...
  [0.613] q2_answer: Q: How does prompt caching reduce cost?
A: Prompt caching works by reusing the ...

How the retriever works

BM25 — same algorithm used in Elasticsearch and most production search stacks. Term-frequency saturation prevents a word appearing 10 times from scoring 10× better than once. Document length normalisation prevents long documents from dominating.

No vector embeddings needed to run this demo — the agent-scratchpad uses a bag-of-words hash embedding for illustration (swap in any real embedder for production).

What the metrics mean

• Context recall — what fraction of the relevant documents were actually retrieved. 1.00 means the BM25 index found all the right passages.
• Faithfulness — fraction of answer sentences with at least one trigram present in the retrieved context. Measures whether the model stays grounded.
• Answer relevance — Jaccard similarity between non-stopword tokens in the question and the answer. Measures whether the answer addresses what was asked.

Extending this

• Swap BM25 for a dense retriever (add a voyage-3 embedding call) and compare NDCG scores
• Add prompt-cache-bench to measure latency savings when the system prompt is cached across calls
• Try stream-parse to process the agent's response incrementally and extract code blocks in real time