Evaluation

Evaluation

Strata-RAG is eval-first: retrieval quality and generation quality are measured, not assumed, and — crucially — they are measured separately, because they fail for different reasons and a single end-to-end score hides which half is broken.

The two separated concerns

A RAG answer can be wrong two ways:

1. Retrieval failed — the right passage was never pulled into the context. No amount of careful generation can recover; the model is answering from the wrong material.
2. Generation failed — the right passage was retrieved, but the model ignored it, embellished it, or contradicted it (a faithfulness failure / hallucination).

Conflating these is the classic RAG evaluation mistake. Strata-RAG keeps them apart: a retrieval metric suite over a golden set, and an independent faithfulness judge over generation. You always know which stage to fix.

Retrieval metrics (over a hand-labelled golden set)

The golden set is a set of questions each mapped to the source ids that are genuinely relevant. Pure functions compute, per question and aggregated:

• Recall@K — did we find it? Of the relevant items, how many landed in the top K. The headline retrieval number — if recall is low, retrieval is the bottleneck.
• Precision@K — how much junk? Of the top K returned, how many are actually relevant. Trades against recall; the rerank floor moves this knob.
• MRR (Mean Reciprocal Rank) — is the best one near the top? Rewards putting the first relevant item high. Matters because the generator reads the top hits most.
• nDCG (normalised Discounted Cumulative Gain) — rank-aware quality. Credits relevant items more when they appear higher, normalised against the ideal ordering. The most complete single signal of ranking quality.

These functions are pure and unit-tested — an evaluation harness you can't trust is worthless, so the metrics themselves are pinned by tests independent of any index.

Slicing the eval by question kind

Because the engine answers two classes of question, the eval can be sliced:

• --kind retrieval evaluates only the semantic questions — the real test of the embedder and reranker. Aggregation questions are answered by the sidecar, not the embedding index, so scoring them as a retrieval test would be meaningless.
• --dense-only turns off BM25 + rerank to isolate the raw embedder contribution — the cleanest A/B signal when comparing embedding models.

Each result table is tagged with its model / collection / mode, so two runs sit side by side and the winner is unambiguous. The two A/B axes are model × mode (dense-only vs hybrid+rerank).

Generation: LLM-as-judge faithfulness

Retrieval being correct doesn't make the answer correct. A separate LLM-as-judge gate reads each generated answer against the retrieved context and judges whether every claim is grounded in that context — catching invented facts, unsupported additions, and contradictions.

The decisive lesson the design encodes: grounding is orthogonal to injection defense. "Answer only from the context" stops hallucination; it does nothing about an instruction that is itself sitting in the retrieved context. Faithfulness eval measures one thing (invented facts); the injection guardrails measure another (an attack obeyed). Keeping them separate is deliberate.

Adversarial / guardrail eval

The injection defenses are themselves measured. Running the injection eval pushes an adversarial fixture set through the input scanner and reports an attack-success-rate (attacks missed / total) plus the false-positive rate on clean text. Toggling a single defense layer off and re-running shows that layer's exact contribution — the residual risk is quantified, not asserted.

The principle

Everything that matters is turned into a number against a fixed reference set:

• retrieval → golden-set Recall@K / Precision@K / MRR / nDCG,
• generation → LLM-judge faithfulness,
• defenses → attack-success-rate over adversarial fixtures.

Measured, separated, and reproducible out of the box on the synthetic sample corpus.