Local RAG Experiment

A from-scratch, fully local Retrieval-Augmented Generation (RAG) system over a Wikipedia subset. Built end to end on Apple Silicon to explore the practical trade-offs of small-model RAG: chunking, embeddings, vector search, prompt construction, and on-device generation. Everything runs offline after the initial model and dataset downloads. No API keys, no cloud.

Why RAG?

A language model only knows what was baked into its weights at training time. RAG bolts on a retrieval step so the model can answer from a specific corpus you control: chunk the documents, embed them, and at query time pull the most relevant chunks into the prompt as context. The interesting question this repo probes is whether that grounding actually happens. With a small 500-article corpus and a base (not instruction-tuned) model, it often does not: in the five-probe eval below, only 1 of 5 answers was genuinely grounded in the retrieved text. The other correct answers came from the model's own parametric knowledge, which is exactly the behaviour RAG is meant to replace. Documenting that gap is the point of the project.

Hardware & Stack

• Hardware: Apple M4 MacBook Pro (arm64)
• LLM runtime: MLX via mlx-lm
• LLM: Phi-2 (microsoft/phi-2), a small base model, not instruction-tuned
• Embeddings: sentence-transformers/all-MiniLM-L6-v2 (384-dim)
• Vector store: ChromaDB (local, persistent)
• Dataset: HuggingFace wikimedia/wikipedia, config 20231101.simple
• Language: Python 3.11

What This Demonstrates

• A complete RAG pipeline built from parts: streaming ingest, word-window chunking, sentence-transformer embeddings, persistent vector search, and retrieval-augmented prompting.
• Running an LLM natively on Apple Silicon with MLX, no GPU or cloud.
• An honest evaluation that surfaces the failure modes of small-corpus, base-model RAG rather than hiding them.

How RAG Works (5 steps)

1. Chunk: Split each source document into overlapping word windows (here, 300 words with 50-word overlap) so each chunk fits the embedder and stays semantically coherent.
2. Embed: Run every chunk through a sentence-transformer to produce a dense vector that captures meaning rather than surface tokens.
3. Index: Store the chunks and vectors in a vector database (ChromaDB), keyed for fast nearest-neighbor lookup.
4. Retrieve: Embed the user's question with the same model and pull the top-k most similar chunks from the index.
5. Generate: Feed those chunks plus the question into a local LLM, instructing it to ground its answer in the provided context.

Installation

python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Requirements:

• Python 3.11 (arm64 build on Apple Silicon)
• ~2 GB free disk for the Phi-2 weights and the Chroma index
• A working HuggingFace cache (~/.cache/huggingface). Phi-2 is pulled on first run if not already present.

Usage

Run in this order:

# 1. Build the index (streams 500 wiki articles, then chunk, embed, store)
python src/ingest.py

# 2. Ask one-off questions
python src/query.py "Your question here"

# 3. Run the canned evaluation suite (writes data/eval_results.txt)
python src/evaluate_rag.py

# 4. Regenerate the results chart from the committed judgments
python src/plot_results.py

Results

Five canned probes from src/evaluate_rag.py. Grading is by hand from the committed retrieval transcript in data/eval_results.txt, recorded in data/eval_summary.csv.

#	Question	Retrieved sources	Retrieval	Answer	Grounded in context?
1	Who was Albert Einstein?	Physics, Physics, String theory	weak	correct	no, answered from model weights
2	What is the capital of France?	France, France, France	partial	correct	no, context never states the capital
3	What is photosynthesis?	Plant, Plant, Tree	good	correct	yes
4	When did World War II end?	Cold War, September, Japan	weak	correct	no, context has the start date only
5	What is the Great Wall of China?	Beijing, Wall, China	weak	wrong	no, grounded in the wrong "wall" chunk

Only probe 3 is a clean win where retrieval found the right material and the answer used it. Probes 1, 2, and 4 look correct but the retrieved chunks do not contain the answer, so the model is reciting parametric knowledge. Probe 5 shows the opposite failure: retrieval pulls a generic "wall" definition and the model faithfully grounds its answer in the wrong chunk.

Regenerate this chart with python src/plot_results.py.

Key Findings

1. Ungrounded generation is the dominant failure. Three of five answers were correct only because Phi-2 already knew the fact. A base model does not reliably defer to retrieved context, so a right answer is not evidence that retrieval worked. This is the single most important thing the eval reveals.
2. Weak retrieval on a thin corpus. With 500 simple-Wikipedia articles, there is often no dedicated article for the entity in question (no Einstein page, no Great Wall page), so nearest-neighbor search returns topically-adjacent chunks that miss the target fact.
3. Grounding in the wrong chunk still fails. Probe 5 is the clearest case of the system working as designed and still being wrong: retrieval decided a generic definition of "wall" was most similar, and generation dutifully used it.

Known Limitations

• Tiny eval set. Five hand-graded probes are illustrative, not a benchmark. There is no automated scoring, so grading is subjective and not regression-safe.
• Base model, not instruction-tuned. Phi-2 does not reliably follow the "answer only from context" instruction. An instruction-tuned model would defer to retrieval far more often (see below).
• Small corpus. 500 articles means thin coverage; many questions have no answer-bearing chunk to retrieve.
• Re-ingest is not idempotent. ingest.py uses collection.add, which raises on duplicate IDs. Delete db/ before re-running, or switch to upsert.
• Eval output is overwritten each run, so it is fine for iteration but does not track results over time.
• Chunking cuts across facts. Word-window chunking is dependency-light but can split a fact across a chunk boundary.

Potential Improvements

1. Instruction-tuned LLM: swap Phi-2 for Mistral-Instruct, Llama-3-Instruct, or Phi-3-mini so the model actually follows "answer only from context". This directly targets the dominant failure mode above.
2. Hybrid search: combine BM25 (lexical) with semantic search to catch cases where the question shares rare terms with the source.
3. Reranking: run a cross-encoder (e.g. bge-reranker) over the top-k retrievals to push the most relevant chunk to position 1.
4. Larger corpus: ingest the full simple-Wikipedia dump (~200k articles) for serious coverage.

Project Structure

rag-experiment/
├── src/
│   ├── ingest.py         # stream, chunk, embed, persist to ChromaDB
│   ├── query.py          # CLI: question, retrieval, Phi-2 answer
│   ├── evaluate_rag.py   # batch eval over the fixed probe set
│   └── plot_results.py   # render data/eval_summary.png from the CSV
├── data/
│   ├── eval_results.txt  # full retrieval + answer transcript (committed)
│   ├── eval_summary.csv  # hand-graded outcome per probe (committed)
│   └── eval_summary.png  # results chart (committed)
├── db/                   # ChromaDB persistent store (gitignored)
├── requirements.txt
├── README.md
└── CLAUDE.md             # notes for future Claude sessions

Acknowledgements

Built with assistance from Claude (Anthropic) for code generation and project scaffolding. All system design, evaluation, and analysis are my own.