Repository navigation
Benchmarking
Anjana Tiha edited this page Mar 28, 2026
·
2 revisions
source rag_env_311/bin/activate
python main.py --benchmark # custom 7-metric pipeline + agent tool tests
python main.py --ragas # RAGAS LLM-as-judge evaluation (see below)--benchmark runs two phases automatically:
- RAG pipeline — 15 questions across 4 domains, 7 metrics
- Agent tools — 18 tests across 5 tools
Results are saved to:
-
benchmark_results.json— full history (all runs appended) -
benchmark_results.csv— latest run only -
tool_benchmark_results.json— agent tool results
| Metric | Kind | What it measures | Scale |
|---|---|---|---|
faithfulness_llm |
LLM-as-judge | Every claim in the answer is grounded in the retrieved context | 0–1 |
answer_relevancy_llm |
LLM-as-judge | The answer directly addresses the question | 0–1 |
ground_truth_match |
F1 word overlap | Lexical similarity to the known correct answer | 0–1 |
keyword_recall |
Exact match | Fraction of expected keywords found in the response | 0–1 |
context_relevance |
Cosine similarity | Mean similarity of top retrieved chunks to the query | 0–1 |
precision_at_5 |
Exact match | Fraction of top-5 chunks containing an expected keyword | 0–1 |
mrr |
Rank | Reciprocal rank of the first relevant chunk | 0–1 |
| Tool | Tests | What is verified |
|---|---|---|
calculator |
5 | Arithmetic correctness, percentage normalisation, unsafe expression rejection |
sentiment |
4 | 4-field structured output, short query search-first routing |
summarise |
3 | Keyword coverage, adaptive length |
translate |
3 | Non-empty output, language routing |
topic_search |
3 | End-to-end pipeline with mocked network |
RAGAS adds four industry-standard LLM-as-judge metrics alongside the custom benchmark. Install the optional dependencies first:
pip install -e ".[eval]"Then run:
python main.py --ragas| Metric | What it measures |
|---|---|
| Faithfulness | Every claim in the answer is grounded in the retrieved context |
| ResponseRelevancy | The answer directly addresses the question asked |
| ContextPrecision | The most relevant chunks are ranked highest |
| ContextRecall | Retrieved context covers the ground truth answer |
RAGAS uses the same DEFAULT_TEST_CASES as --benchmark. To run on custom questions, call run_ragas_evaluation() directly — see docs_technical/BENCHMARK.md.
Measured against 15 questions across 4 domains: cat facts, Python language, team members CSV, and machine learning.
| Metric | Score |
|---|---|
| Faithfulness | 0.967 |
| Answer Relevancy | 0.867 |
| Ground Truth Match | 0.748 |
| Keyword Recall | 0.967 |
| Context Relevance | 0.656 |
| Precision@5 | 0.453 |
| MRR | 1.000 |
| Overall | 0.808 |
Agent tool benchmark: 18 / 18 (calculator 5/5 · sentiment 4/4 · summarise 3/3 · translate 3/3 · topic_search 3/3)
Test cases are defined in Benchmarker.DEFAULT_TEST_CASES in src/rag/benchmarker.py. Each entry is a dict:
{
'question': 'Who created Python and when?',
'ground_truth': 'Python was created by Guido van Rossum in 1991.',
'expected_keywords': ['Guido', '1991'],
'query_type': 'factual',
}To run a custom set of test cases:
from src.rag.benchmarker import Benchmarker
from src.rag.vector_store import VectorStore
store = VectorStore()
store.build_or_load(chunks)
benchmarker = Benchmarker(store)
results = benchmarker.run(test_cases=[
{'question': 'What is Alice's role?', 'ground_truth': 'Senior Engineer',
'expected_keywords': ['Senior', 'Engineer'], 'query_type': 'factual'},
])-
High MRR + Low Precision@5: Retrieval finds the right document but also returns many irrelevant chunks. Consider tightening
SIMILARITY_THRESHOLDinconfig.py. - High Faithfulness + Low Relevancy: The answer is grounded in context but doesn't directly address the question. The query classifier may be mis-routing the query type.
- Low Context Relevance: The embedding model is not finding semantically similar chunks. Check that the query topic matches what is in the documents.
-
Low Ground Truth Match: The model is correct but uses different words. Check the
expected_keywordslist — it may need broader synonyms.