rag-eval-lab

A provider-agnostic RAG evaluation lab. Build a retrieval-augmented pipeline against any LLM provider — commercial (Anthropic, OpenAI, Cohere) or local open-source models (Llama / Mistral via Ollama) — score answer quality with reproducible metrics, track experiments in MLflow, and A/B-test two configurations with a statistical significance test.

Designed to demonstrate the end-to-end loop an LLM engineer owns: pick a model → build a pipeline → measure it → prove one option is actually better.

Runs free, out of the box. The default echo provider is a dependency-free extractive baseline — python -m rageval.evaluate --provider echo computes real metrics with no API key, no GPU, and no model download.

Quick start

pip install -e ".[dev]"

# 1) Free baseline — extractive, no model needed
python -m rageval.evaluate --provider echo

# 2) Local open-source model (free): install Ollama, then `ollama pull mistral`
python -m rageval.evaluate --provider ollama:mistral --mlflow

# 3) A/B test: is the LLM significantly better than the extractive baseline?
python -m rageval.evaluate --provider ollama:mistral --provider-b echo

A/B output reports the mean difference, a 95% bootstrap confidence interval, a two-sided p-value, and a SIGNIFICANT / not-significant verdict.

How it maps to an LLM-engineering role

Capability	Where it lives
Multiple LLM providers — Anthropic, OpenAI, Cohere, + open-source via Ollama	providers.py — one interface, lazily-loaded backends
Open-source models (Llama / Mistral)	OllamaProvider — local, free, offline
RAG pipeline	pipeline.py (default, dependency-free) + LangChain & LlamaIndex variants in integrations.py
Quality evaluation metrics	metrics.py — answer token-F1, context recall (deterministic, no judge); semantic metrics via Ragas in integrations.py
A/B testing of models	abtest.py — paired bootstrap significance test
Experiment tracking	tracking.py — MLflow runs
Justifying model/method choice	run any two configs through the A/B harness and let the evidence decide
CI/CD + Python ecosystem	tested core, green CI, optional extras for each integration

Architecture

corpus ──chunk──▶ retriever (lexical, or FAISS / LlamaIndex) ──▶ top-k contexts
                                                                     │
question ────────────────────────────────────────────────────────┐  ▼
                                                                   provider.answer()
                                                                   (echo | ollama | anthropic | openai | cohere)
                                                                     │
                                                                     ▼
                                              metrics (token-F1, context recall, or Ragas)
                                                     │
                                          MLflow run   +   A/B significance test

What's tested vs. optional

The core is dependency-free and fully unit-tested (chunking, metrics, the A/B test, the provider factory, and the offline pipeline) — so CI is fast and green without any API key. The semantic integrations (LangChain, LlamaIndex, Ragas, MLflow, and the commercial providers) are imported lazily and live behind the rag / eval / providers extras; install what you need:

pip install -e ".[rag,eval,providers]"

Develop

ruff check .
pytest -q

Next steps (ideas)

• Add a retrieval-only metric suite (MRR, hit@k) alongside the answer metrics
• Wire the Ragas judge to a stronger model and compare to the lexical metrics
• Promote the A/B harness into a CI gate that fails a PR on a significant regression