Local RAG for PDF documents

A Retrieval-Augmented Generation (RAG) pipeline that processes a PDF, creates local embeddings, and answers questions using only the retrieved chunks. The entire inference workflow runs with Ollama.

The repository includes document 2605.24708 and its processed artifacts as an example.

Project workflow

1. Extracts text from the PDF page by page with PyMuPDF.
2. Normalizes ligatures, hyphenated words, and line breaks.
3. Splits the content into chunks of up to 500 tokens with overlap.
4. Generates embeddings with embeddinggemma and maintains a hash-based cache.
5. Retrieves chunks with dense cosine search or hybrid Dense + BM25 search.
6. Generates grounded claims with qwen3:4b and verifies quoted evidence.

Requirements

• Python 3.11 or later.
• Ollama running at http://localhost:11434.
• The local embeddinggemma and qwen3:4b models.

Installation

Clone the repository and run all commands from its root directory. Install the application dependencies with pip:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Alternatively, install the application and notebook dependencies from pyproject.toml with uv:

uv sync --all-groups
source .venv/bin/activate

Start Ollama and download the two models in a separate terminal if necessary:

ollama serve
ollama pull embeddinggemma
ollama pull qwen3:4b

Quick start

With Ollama running, you can query the included embeddings directly:

python src/chunking/search_chunks.py \
  "What is the main contribution of the paper?" \
  --top-k 3

To generate an answer with verified citations:

python src/answer_question.py \
  "What is the main contribution of the paper?"

The JSON output contains sentence-level claims, exact evidence excerpts and chunk IDs, an insufficient_evidence flag, and metadata for the retrieved chunks. The command fails if a citation refers to an unknown chunk or its evidence is not present in that chunk.

Run the complete pipeline

The following commands regenerate all the example artifacts:

python src/extraction/extract_pages.py data/raw/2605.24708.pdf \
  --source-url https://arxiv.org/pdf/2605.24708

python src/extraction/normalize_pages.py
python src/chunking/chunk_pages.py
python src/embeddings/embed_chunks.py

You can then run the search or answer generator described in the previous section.

Command reference

Extract PDF pages

python src/extraction/extract_pages.py INPUT_PDF \
  [--output-dir OUTPUT_DIRECTORY] \
  [--source-url SOURCE_URL]

Writes one JSON object per page and a manifest to data/extracted/ by default. --source-url records provenance in the manifest; it does not download the PDF. Example:

python src/extraction/extract_pages.py data/raw/2605.24708.pdf \
  --output-dir data/extracted \
  --source-url https://arxiv.org/pdf/2605.24708

Normalize pages

python src/extraction/normalize_pages.py

Reads data/extracted/2605.24708.pages.jsonl, normalizes PDF line breaks, hyphenation, whitespace, and ligatures, and writes data/normalized/2605.24708.pages.jsonl.

Create chunks

python src/chunking/chunk_pages.py

Reads the normalized pages and writes paragraph-aware, overlapping chunks to data/chunks/2605.24708.chunks.jsonl. Chunk size and overlap are configured by constants in the script.

Generate embeddings

python src/embeddings/embed_chunks.py

Uses Ollama's embeddinggemma model and writes data/embeddings/2605.24708.embeddings.jsonl. Existing vectors are reused when their content hash and embedding model match.

Dense search

python src/chunking/search_chunks.py "QUESTION" [--top-k N]

Embeds the question and ranks every stored chunk by cosine similarity. --top-k defaults to 3.

Hybrid search

python src/chunking/hybrid_search.py "QUESTION" \
  [--top-k N] \
  [--candidate-k N]

Combines dense and BM25 candidates using reciprocal rank fusion. --top-k defaults to 3, while --candidate-k defaults to 10 candidates from each retriever.

Generate an answer

python src/answer_question.py "QUESTION"

Retrieves five dense-search chunks, asks qwen3:4b for grounded claims, checks all citation IDs and evidence excerpts, and prints the validated result as JSON.

Evaluate retrieval

python src/evaluation/evaluate_retrieval.py [--k N]

Evaluates dense retrieval against answerable cases in data/evals/retrieval_golden.jsonl and reports Hit Rate, MRR, and NDCG at N. The default is 3.

Run retrieval experiments

The notebook compares Dense and Hybrid retrieval, tunes reciprocal rank fusion on the development cases, and evaluates the selected configuration on the held-out test cases:

jupyter lab src/notebooks/retrieval_experiments.ipynb

Notebook dependencies are installed by uv sync --all-groups. Start Ollama before running the cells.

Use the OpenCode agents

The project includes rag-prompt-engineer, a primary agent for controlled generation-prompt experiments, and rag-pdf-critic, its read-only review subagent. Restart OpenCode after first checking out these agent files, then select rag-prompt-engineer in the agent picker and provide the prompt-change objective. The primary agent runs the benchmark and delegates the independent PDF review automatically.

Structure

RAG/
├── data/
│   ├── raw/          # Original PDF
│   ├── extracted/    # Page-level text and manifest
│   ├── normalized/   # Normalized text
│   ├── chunks/       # Chunks ready for retrieval
│   ├── embeddings/   # Chunks with their vectors
│   └── evals/        # Development and frozen retrieval cases
└── src/
    ├── extraction/   # Extraction and normalization
    ├── chunking/     # Chunking, dense search, and hybrid search
    ├── embeddings/   # Embedding generation
    ├── evaluation/   # Retrieval metrics and evaluation CLI
    ├── notebooks/    # Retrieval tuning and comparison
    └── answer_question.py

Current configuration

The example document paths and model names are defined as constants in the scripts. Retrieval loads the embeddings into memory and computes cosine similarity exhaustively, a simple implementation intended for small collections and local experimentation.