How can I evaluate a LlamaIndex retriever without calling an LLM?

[nina7-code]

I’m exploring how to test retrieval separately from answer generation in LlamaIndex.

I want a small test dataset where each question has known relevant document IDs, then compare those IDs with the retriever’s results—without calling a generation model.

For example:

relevant_ids = {"cancel-plan", "refund-policy"}
retrieved_ids = ["billing", "cancel-plan", "upgrade-plan"]

My main questions are:

1. Is Recall@k a sensible starting metric, alongside the rank of the first relevant result?
2. When several retrieved nodes belong to the same source document, should I evaluate node IDs or deduplicate by source document ID?
3. Does LlamaIndex have a built-in evaluator suitable for manually labeled examples like these, without generating questions through an LLM?

Small examples or documentation pointers would be appreciated.

[Cha-Imaa]

Yes, you can evaluate retrieval against manually labeled relevant IDs without calling a generation LLM.

1. Which metrics?

Metric	What it measures
Recall@k	The fraction of relevant items retrieved in the top k results
MRR@k	The average reciprocal rank of the first relevant result across queries
Hit Rate@k	Whether at least one relevant item appears in the top k results, averaged across queries

For your example:

• Recall@3 = 0.5: one of the two relevant documents was retrieved.
• RR@3 = 0.5: the first relevant result appears at rank 2.
• Hit@3 = 1: at least one relevant document was found.

2. Node IDs or document IDs?

Choose the granularity that matches your labels:

• Node-level: compare retrieved node IDs against relevant chunk IDs.
• Document-level: map retrieved nodes to their source document IDs before calculating your metrics.

Take the original top k results first. For document-level recall, count each relevant document only once. Duplicate chunks can still occupy retrieval slots; deduplicating before selecting the top k measures a different retrieval policy.

3. LlamaIndex’s built-in evaluator

RetrieverEvaluator supports manually labeled queries and expected node IDs:

from llama_index.core.evaluation import RetrieverEvaluator

# Assumes you already have an index.
retriever = index.as_retriever(similarity_top_k=3)

evaluator = RetrieverEvaluator.from_metric_names(
    ["hit_rate", "mrr"],
    retriever=retriever,
)

# Run in a notebook with top-level await, or inside an async function.
result = await evaluator.aevaluate(
    query="How do I cancel my plan?",
    expected_ids=["cancel-plan", "refund-policy"],
)

print(result.metric_vals_dict)

Replace those example IDs with the actual node IDs in your index. The built-in evaluator compares node IDs; it does not automatically group results by source document.

Also, k comes from the retriever configuration, not from aevaluate(). The example above calculates hit rate and reciprocal rank; it does not calculate recall.

These metrics need no generation LLM. Retrieval may still make embedding API calls, or LLM calls if you configure query rewriting or another LLM-based retrieval step.

Official LlamaIndex retrieval-evaluation guide