Skip to content

Benchmarking

Anjana Tiha edited this page Mar 28, 2026 · 2 revisions

Benchmarking

Running the Benchmark

source rag_env_311/bin/activate

python main.py --benchmark     # custom 7-metric pipeline + agent tool tests
python main.py --ragas         # RAGAS LLM-as-judge evaluation (see below)

--benchmark runs two phases automatically:

  1. RAG pipeline — 15 questions across 4 domains, 7 metrics
  2. Agent tools — 18 tests across 5 tools

Results are saved to:

  • benchmark_results.json — full history (all runs appended)
  • benchmark_results.csv — latest run only
  • tool_benchmark_results.json — agent tool results

Metrics

RAG Pipeline (7 metrics)

Metric Kind What it measures Scale
faithfulness_llm LLM-as-judge Every claim in the answer is grounded in the retrieved context 0–1
answer_relevancy_llm LLM-as-judge The answer directly addresses the question 0–1
ground_truth_match F1 word overlap Lexical similarity to the known correct answer 0–1
keyword_recall Exact match Fraction of expected keywords found in the response 0–1
context_relevance Cosine similarity Mean similarity of top retrieved chunks to the query 0–1
precision_at_5 Exact match Fraction of top-5 chunks containing an expected keyword 0–1
mrr Rank Reciprocal rank of the first relevant chunk 0–1

Agent Tools (18 tests)

Tool Tests What is verified
calculator 5 Arithmetic correctness, percentage normalisation, unsafe expression rejection
sentiment 4 4-field structured output, short query search-first routing
summarise 3 Keyword coverage, adaptive length
translate 3 Non-empty output, language routing
topic_search 3 End-to-end pipeline with mocked network

RAGAS Evaluation

RAGAS adds four industry-standard LLM-as-judge metrics alongside the custom benchmark. Install the optional dependencies first:

pip install -e ".[eval]"

Then run:

python main.py --ragas
Metric What it measures
Faithfulness Every claim in the answer is grounded in the retrieved context
ResponseRelevancy The answer directly addresses the question asked
ContextPrecision The most relevant chunks are ranked highest
ContextRecall Retrieved context covers the ground truth answer

RAGAS uses the same DEFAULT_TEST_CASES as --benchmark. To run on custom questions, call run_ragas_evaluation() directly — see docs_technical/BENCHMARK.md.


Current Scores (v1.3.1)

Measured against 15 questions across 4 domains: cat facts, Python language, team members CSV, and machine learning.

Metric Score
Faithfulness 0.967
Answer Relevancy 0.867
Ground Truth Match 0.748
Keyword Recall 0.967
Context Relevance 0.656
Precision@5 0.453
MRR 1.000
Overall 0.808

Agent tool benchmark: 18 / 18 (calculator 5/5 · sentiment 4/4 · summarise 3/3 · translate 3/3 · topic_search 3/3)


Adding Test Cases

Test cases are defined in Benchmarker.DEFAULT_TEST_CASES in src/rag/benchmarker.py. Each entry is a dict:

{
    'question':          'Who created Python and when?',
    'ground_truth':      'Python was created by Guido van Rossum in 1991.',
    'expected_keywords': ['Guido', '1991'],
    'query_type':        'factual',
}

To run a custom set of test cases:

from src.rag.benchmarker import Benchmarker
from src.rag.vector_store import VectorStore

store = VectorStore()
store.build_or_load(chunks)

benchmarker = Benchmarker(store)
results = benchmarker.run(test_cases=[
    {'question': 'What is Alice's role?', 'ground_truth': 'Senior Engineer',
     'expected_keywords': ['Senior', 'Engineer'], 'query_type': 'factual'},
])

Interpreting Results

  • High MRR + Low Precision@5: Retrieval finds the right document but also returns many irrelevant chunks. Consider tightening SIMILARITY_THRESHOLD in config.py.
  • High Faithfulness + Low Relevancy: The answer is grounded in context but doesn't directly address the question. The query classifier may be mis-routing the query type.
  • Low Context Relevance: The embedding model is not finding semantically similar chunks. Check that the query topic matches what is in the documents.
  • Low Ground Truth Match: The model is correct but uses different words. Check the expected_keywords list — it may need broader synonyms.

Clone this wiki locally