RAG Eval Workbench

Offline, standard-library RAG evaluation workbench for testing retrieval quality, answer grounding, latency, and hallucination risk without paid API keys.

Engineering Scope

RAG Eval Workbench implements a compact evaluation harness for retrieval-augmented generation systems. It focuses on operational pieces that are easy to miss in prototypes:

• deterministic document loading and chunking;
• local vector-style retrieval with hashed TF-IDF and cosine similarity;
• citation-first answer synthesis;
• retrieval, grounding, hallucination, latency, and cost metrics;
• trace files and reproducible run artifacts;
• CI that runs tests and the demo without network services.

The core demo uses only Python 3 standard library modules. Optional integrations are documented for FastAPI, Streamlit, LangChain, and MLflow, but they are not required to run the project.

Quickstart

cd rag-eval-workbench
python -m rag_eval_workbench demo --output runs/demo
python -m unittest discover -s tests

If your system only exposes Python as python3, use python3 -m ... for the same commands.

Open the generated dashboard:

open runs/demo/dashboard.html

The command works from a fresh checkout because the repo uses a src/ package layout plus a thin top-level checkout shim for local execution. Installing the package is optional.

Demo Output

The demo writes a portable run folder:

runs/demo/
  dashboard.html
  metrics.csv
  metrics.json
  results.json
  run_manifest.json
  summary.md
  traces.jsonl

Example terminal output:

Run complete: runs/demo
Dashboard: runs/demo/dashboard.html
Metrics: retrieval@k=1.000 groundedness=0.900 risk=0.100 latency_ms=1.200 cost=$0.000000

Actual latency varies by machine, but the corpus, benchmark questions, retrieval algorithm, and evaluation logic are deterministic.

Metrics

Metric	What It Answers	Why It Matters
retrieval@k	Did the expected source document appear in the retrieved top-k contexts?	Catches retrieval and chunking regressions.
groundedness	Are answer terms supported by retrieved context terms?	Flags unsupported generated claims.
answer_relevance	Does the answer overlap with the reference answer?	Keeps responses aligned to expected facts.
hallucination_risk	How risky is the answer based on grounding and retrieval quality?	Gives release owners a triage signal.
latency_ms	How long did the local pipeline take per question?	Makes evaluation performance visible.
estimated_cost_usd	What is the configured per-run token cost estimate?	Defaults to zero for the offline standard-library demo.

Architecture

examples/docs
    -> load markdown/text documents
    -> create overlapping word chunks
    -> build local hashed TF-IDF vectors
    -> retrieve top-k chunks by cosine similarity
    -> synthesize extractive cited answers
    -> compute heuristic metrics
    -> write traces, metrics, and static dashboard

Key modules live under src/rag_eval_workbench/:

Module	Role
documents.py	Document loading and deterministic chunking.
retrieval.py	Local hash-vector retriever.
synthesis.py	Extractive answer synthesis with citations.
evaluator.py	Heuristic RAG metrics.
benchmark.py	End-to-end benchmark orchestration.
tracing.py	JSON, JSONL, CSV, and Markdown artifact writing.
dashboard.py	Static HTML dashboard generation.
cli.py	python -m rag_eval_workbench entrypoint.

See docs/architecture.md and docs/eval-methodology.md for more detail.

CLI Usage

Run the bundled demo:

python -m rag_eval_workbench demo --output runs/demo

Run against your own local corpus:

python -m rag_eval_workbench benchmark \
  --docs examples/docs \
  --questions examples/questions.json \
  --output runs/custom \
  --top-k 3

Optional cost modeling:

python -m rag_eval_workbench demo \
  --output runs/demo-costed \
  --cost-per-1k-tokens 0.002

Repository Layout

.
  .github/workflows/ci.yml
  docs/
  examples/
  src/rag_eval_workbench/
  tests/
  .env.example
  pyproject.toml

Production Concerns

The project covers common operating concerns for RAG systems:

• eval harness design for RAG quality gates;
• deterministic local baseline before paid model integrations;
• trace-first debugging for retrieval failures;
• CI-friendly tests and demo artifacts;
• static dashboard generation for stakeholder review;
• clean extension path for model providers, vector stores, and experiment tracking.

Optional Integrations

The standard demo intentionally avoids third-party packages. Suggested production wrappers are documented in docs/optional-integrations.md:

• FastAPI for benchmark service endpoints;
• Streamlit for analyst exploration;
• LangChain for provider-backed retrievers;
• MLflow for experiment tracking and artifact logging.

Limitations

• The synthesizer is extractive and deterministic; it is not a generative LLM.
• The evaluator is heuristic and transparent; it is not a replacement for human review or an LLM judge.
• Hash-vector retrieval is a local baseline; production systems can swap in embeddings while keeping the same benchmark shape.