Repository navigation
RAG Evidence Lab: offline citation and number checks for exported RAG answers #1841
manularrea
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I'm sharing the first prototype of RAG Evidence Lab, a small Python tool for inspecting exported RAG answers alongside their retrieved sources.
A simple example: the answer says 90 days and cites a policy that says 30 days. The citation resolves, and lexical overlap is high, but the numeric literal deserves a closer look. The included synthetic demo makes that mismatch visible.
The current version provides:
It runs locally on Python 3.10+, with no API keys, model downloads, or runtime dependencies:
git clone https://github.com/manularrea/rag-evidence-lab.git cd rag-evidence-lab python -m rag_evidence_lab examples/demo.json --html report.html --json report.jsonOpen
report.htmlto inspect the four demo cases. The project is MIT licensed; tests, installation and smoke checks pass on Python 3.10 and 3.12.Scope: this is heuristic triage for human review. It does not establish semantic entailment: negation, units and paraphrases can produce false negatives or positives. A clean report is not proof of factual correctness. It consumes JSON exports and is currently a standalone tool, without a dedicated OpenAI Evals adapter.
I'd welcome feedback from people evaluating RAG systems: would better numeric normalization or framework exporter recipes be more useful first? There are small contribution issues for both, and English/Spanish onboarding in the repository.
All reactions