How are people currently debugging retrieval failures and RAG quality issues in Haystack?

[rautaditya2606]

Hi everyone,

I'm a haystack contributor and also use haystack daily in production for rag applications.

One thing I've noticed is that when retrieval quality drops or a query produces a poor answer, I often end up manually inspecting retrieved documents, pipeline outputs, evaluation metrics, and document stores to understand what went wrong.

I'm curious how others approach this today:

how do you debug retrieval failures?
how do you inspect pipeline execution step-by-step?
how do you validate document stores and catch issues such as duplicates, malformed documents, missing metadata, etc.?
are there existing tools or workflows you're using for observability and diagnostics around haystack pipelines?

I'd love to understand the current pain points and workflows before investing time into building anything around this area.

thanks!

[krotname]

What works best for me is to separate this into three layers: single-query debugging, corpus health checks, and regression/evaluation.

For a single bad query, I usually rerun the pipeline with intermediate outputs included:

result = pipe.run(
    data,
    include_outputs_from={"retriever", "ranker", "prompt_builder"},
)

Then I inspect, in this order:

1. query as seen by the retriever, including filters
2. retrieved document ids, scores, metadata, and chunk text
3. ranker output, if any
4. final prompt assembled by PromptBuilder
5. generated answer and citations/references

That usually tells me whether the failure is retrieval, ranking, prompt construction, or generation.

For step-by-step execution, Haystack's LoggingTracer is the quickest local tool because it does not need a backend. I enable content tracing only in dev/safe data environments, because it logs component inputs/outputs. For production, I prefer OpenTelemetry/Langfuse/Phoenix-style tracing and keep content tracing off unless the data is scrubbed.

For document store validation, I keep a separate ingestion audit that checks:

• duplicate document ids and duplicate content hashes
• empty/very short chunks
• missing metadata keys used by filters
• unexpected metadata cardinality, for example tenant/source/type
• null embeddings or wrong embedding dimensions
• chunk parent ids / source ids missing after preprocessing

For quality regression, I keep a small golden set and run it on every retriever/indexing change. With labels, I use document-level metrics such as recall/MRR/MAP. Without labels, I use model-based checks such as context relevance and faithfulness, but I treat them as triage signals rather than truth.

The tool I would personally want is a "RAG debug bundle" around one query: run the pipeline with selected intermediate outputs, attach the trace id, show retrieved docs + scores + metadata, show prompt, run a few corpus checks for the touched docs, and diff this run against a previous known-good run. That would cover most of the manual work I currently do.

[rautaditya2606]

This is exactly the workflow I'm trying to automate. Building a tool around this, will share when it's ready.
Thank you

[rautaditya2606]

Built what you described.

Three tools: validate_document_store, inspect_pipeline, diagnose_retrieval_failure.

diagnose_retrieval_failure runs pipeline.run(include_outputs_from={retriever_name}),
extracts intermediate retriever outputs, and classifies failures in sequence:
NO_RESULTS → RANKING_FAILURE → EMPTY_CONTEXT → GENERATOR_FAILURE.

Demo output on a local InMemory pipeline with zero external API dependencies:

============================================================
HAYSTACK DIAGNOSTICS DEMO RUN

[Step 2] Running validate_document_store()...
7 documents indexed. 12 issues found across: content=None, empty content,
short chunks, duplicates, null embeddings.

[Step 5] Failure diagnostics:

CASE A: SUCCESS (Query: 'Paris')
Failure Type: SUCCESS
Checks: {no_results: False, empty_context: False, generator_failure: False, ranking_failure: False}

CASE B: NO RESULTS (Query: 'Tokyo')
Failure Type: NO_RESULTS
Checks: {no_results: True, empty_context: False, generator_failure: False, ranking_failure: False}

CASE C: GENERATOR FAILURE (Query: 'Rome')
Failure Type: GENERATOR_FAILURE
Answer: I'm sorry, I do not know the answer based on the provided context.
Checks: {no_results: False, empty_context: False, generator_failure: True, ranking_failure: False}

CASE D: RANKING FAILURE (Query: 'Berlin', Threshold: 5.0)
Failure Type: RANKING_FAILURE
Top Score: 1.165
Checks: {no_results: False, empty_context: False, generator_failure: False, ranking_failure: True}

Repo: https://github.com/rautaditya2606/haystack-diagnostics

[rautaditya2606]

Quick update: I ended up building the debug bundle you described.

It captures pipeline configuration, retriever outputs, retrieved documents + scores + metadata, prompt context, answer state, and diagnostic metadata into a structured JSON artifact that can be persisted and diffed across runs.

I also added a diff CLI tool (python -m diagnostics.debug_bundler diff bundle_a.json bundle_b.json) for comparing bundles across executions, including document-level changes, score deltas, configuration changes, and answer diffs.

Your original "single-query debug bundle" suggestion ended up shaping most of the design, so thanks for the detailed feedback.

[GioiaZheng]

One thing I would make first-class is a reproducible failure artifact, not only a dashboard.

For each bad query, I’d want a compact “debug bundle” that captures the pipeline version, embedding model/version, document store snapshot or index version, retriever filters, raw top-k before reranking, reranked top-k, final prompt, answer, and citations. That makes the failure diffable against a known-good run instead of only inspectable by eye.

I’d also separate corpus-health checks from query-level eval. Duplicates, missing metadata, empty chunks, wrong embedding dimensions, and unexpected filter cardinality are ingestion problems; retrieval relevance/MRR/faithfulness are query problems. Mixing them makes the debugging surface feel larger than it is.

The smallest useful tool, to me, would be: run one query, persist the full intermediate trace, attach corpus checks for the touched documents, and allow diffing against a previous run.

[rautaditya2606]

The debug bundle is now implemented.

I took your suggestion of treating reproducibility as a first-class concern rather than just inspection. The bundle captures the full query execution state into a structured artifact that can be persisted and diffed against previous runs.

I also separated corpus-health diagnostics from query-level diagnostics in the implementation, with the bundle acting as the bridge between them.

Additionally, I implemented the retrieval-failure taxonomy split you suggested. The diagnoser now distinguishes SCORE_BELOW_CUTOFF (expected document not retrieved with sufficient score) from CONTEXT_LOSS (expected document retrieved but dropped by reranking). The context-packing truncation sub-case is still deferred for now.

Your corpus/query separation point significantly influenced the final design, so I wanted to thank you for pushing on that.

[GioiaZheng]

Thanks for implementing it so quickly. I pulled the repo and ran the early version locally.

Quick results from a Windows/Python 3.12.13 environment:

• pip install -e .[dev] worked; it pulled haystack-ai==2.30.2 from the broad pyproject.toml constraint rather than the pinned 2.29.0 in requirements.txt.
• pytest tests/ passed: 45 passed in ~44s.
• demo/sample_pipeline.py passed with HAYSTACK_TELEMETRY_ENABLED=false set.
• collect_debug_bundle_tool returned valid JSON and wrote a bundle when I used a workspace-local output_dir.
• python -m diagnostics.debug_bundler diff bundle_a bundle_b worked.

Two small things I found while testing:

1. tests/smoke_mcp.py hardcodes output_dir=/tmp/mcp_smoke_bundles. On Windows / locked-down environments that becomes \tmp and fails with [WinError 5] Access is denied. Using tempfile.mkdtemp() or Path(tempfile.gettempdir()) / mcp_smoke_bundles would make the smoke test portable.

2. The diff output is useful, but it currently reports noisy config_changes for equivalent InMemory runs because InMemoryDocumentStore.index is a random UUID each time. It might be worth normalizing or ignoring known volatile fields, or adding an ignore_config_paths option, so config diffs stay focused on user-meaningful changes.

Overall, the direction looks solid. The corpus/query separation is much clearer in the current design, and the bundle + diff path feels like the right debugging artifact rather than just another diagnostic report.

[rautaditya2606]

Hey Gioia, thanks again for taking the time to pull the repo, run the tests, and provide such detailed feedback. Both fixes are pushed.

• /tmp hardcode — all four paths in smoke_mcp.py now use Path(tempfile.gettempdir())
• Noisy config diff — added ignore_config_paths to diff_debug_bundles(), defaults to frozenset({"*.index"}). Pass ignore_config_paths=set() to see everything.

Really appreciate the feedback, both suggestions improved the project substantially. Let me know if you spot anything else while testing.

[GioiaZheng]

Thanks for turning those around. I pulled the latest main (dd31847) and re-ran the checks on the same Windows / Python 3.12.13 setup.

• pytest tests/ passes: 48 passed in 22.07s.
• tests/smoke_mcp.py now writes under Path(tempfile.gettempdir()) and passes when I run it with PYTHONIOENCODING=utf-8.
• The new ignore_config_paths default covers the noisy *.index case, and the added diff tests make that behavior explicit.

One small extra Windows note: running the smoke script in the default cp1252 console failed before the checks completed because the status output prints a checkmark glyph. Not related to the /tmp fix, but either documenting PYTHONIOENCODING=utf-8 / chcp 65001 or falling back to ASCII status markers would make the manual smoke path more robust.

Other than that, both fixes look good from my side. Nice turnaround.

[rautaditya2606]

Thanks for turning those around. I pulled the latest main (dd31847) and re-ran the checks on the same Windows / Python 3.12.13 setup.

• pytest tests/ passes: 48 passed in 22.07s.
• tests/smoke_mcp.py now writes under Path(tempfile.gettempdir()) and passes when I run it with PYTHONIOENCODING=utf-8.
• The new ignore_config_paths default covers the noisy *.index case, and the added diff tests make that behavior explicit.

One small extra Windows note: running the smoke script in the default cp1252 console failed before the checks completed because the status output prints a checkmark glyph. Not related to the /tmp fix, but either documenting PYTHONIOENCODING=utf-8 / chcp 65001 or falling back to ASCII status markers would make the manual smoke path more robust.

Other than that, both fixes look good from my side. Nice turnaround.

Thanks for checking again, Gioia!

I also replaced the Unicode checkmark/cross glyphs with ASCII-safe [PASS] and [FAIL] markers to avoid Windows console encoding issues out of the box.

Appreciate the catch!

[GioiaZheng]

Confirmed on latest main (4d6bb1b) from the same Windows / Python 3.12.13 setup.

• pytest tests/ passes: 48 passed in 22.94s.
• tests/smoke_mcp.py passes without PYTHONIOENCODING; the [PASS] markers avoid the cp1252 crash path.

One tiny leftover: the two smoke section headings still use an em dash, which renders as a replacement character in my captured console output. It does not fail the run, but if the goal is fully ASCII-safe smoke output, those separators could be changed to - too.

Looks good otherwise.

[rautaditya2606]

Confirmed on latest main (4d6bb1b) from the same Windows / Python 3.12.13 setup.

• pytest tests/ passes: 48 passed in 22.94s.
• tests/smoke_mcp.py passes without PYTHONIOENCODING; the [PASS] markers avoid the cp1252 crash path.

One tiny leftover: the two smoke section headings still use an em dash, which renders as a replacement character in my captured console output. It does not fail the run, but if the goal is fully ASCII-safe smoke output, those separators could be changed to - too.

Looks good otherwise.

Thanks again for testing this so thoroughly, Gioia!

I've pushed the final ascii safe tweak and replaced the remaining em dashes with plain -, so the smoke output is now fully ascii safe.

I really appreciate you taking the time to test everything on windows and report these issues. Your feedback genuinely helped make the project more robust.

[VectorPeak]

A workflow that’s been useful in production is treating retrieval debugging as a lineage problem. Every chunk gets stable metadata at indexing time: source_id, chunk_id, content_hash, embedding_model, index_version, indexed_at, and any filter fields used at query time. When a bad answer happens, the key question becomes whether the wrong chunks were retrieved, the right chunks were filtered out, or the generator ignored good context.

For document store validation, a small offline checker catches a lot before it becomes a RAG issue:

from collections import Counter

def validate_documents(docs):
 errors = []

 ids = [doc.id for doc in docs]
 duplicate_ids = [doc_id for doc_id, count in Counter(ids).items() if count > 1]
 if duplicate_ids:
 errors.append(f"duplicate document ids: {duplicate_ids[:10]}")

 for doc in docs:
 if not doc.content or not doc.content.strip():
 errors.append(f"{doc.id}: empty content")

 if "source_id" not in doc.meta:
 errors.append(f"{doc.id}: missing source_id")

 if "chunk_id" not in doc.meta:
 errors.append(f"{doc.id}: missing chunk_id")

 if "content_hash" not in doc.meta:
 errors.append(f"{doc.id}: missing content_hash")

[rautaditya2606]

Thanks for the detailed breakdown, really helpful.
added what you suggested, duplicate ID checks, lineage metadata validation, and a new FILTER_EXCLUSION failure type for cases where the doc exists but gets filtered out by metadata filters
https://github.com/rautaditya2606/haystack-diagnostics