Haystack Diagnostics Engine — open-source inspect/validate/diagnose toolchain for RAG pipelines

[rautaditya2606]

Haystack Diagnostics

Hey everyone,

As a Haystack contributor and daily Haystack user, I kept running into the same debugging workflow: a retrieval pipeline fails, results look wrong, and I end up manually inspecting document stores, pipeline graphs, and intermediate outputs to find the root cause.

Haystack already provides great building blocks like LoggingTracer, OpenTelemetry support, and pipeline visualization, but I couldn't find a single tool that brings validation, inspection, and retrieval diagnostics together.

So I built haystack-diagnostics.

Features

• validate_document_store

  • Detects null content, empty chunks, duplicates, missing metadata keys, null embeddings, and embedding dimension mismatches.

• inspect_pipeline

  • Shows components, connections, sockets, and generates a Mermaid.js pipeline graph.

• diagnose_retrieval_failure

  • Classifies failures as:

    • NO_RESULTS
    • RANKING_FAILURE
    • EMPTY_CONTEXT
    • GENERATOR_FAILURE

All tools are also exposed through a lightweight MCP server for clients like Claude Desktop and Cursor.

Repo: https://github.com/rautaditya2606/haystack-diagnostics

The demo runs fully offline with no API keys required.

Would appreciate feedback from maintainers and production Haystack users on the approach, failure classification logic, and anything missing from the debugging workflow.

[GioiaZheng]

This looks very aligned with the failure mode people hit in production RAG. One thing I’d add is a “debug bundle” output for a single query, not only individual tools.

For one failed query, persist: pipeline graph/version, component params, retrieved docs before/after ranker, scores, metadata, final prompt, answer, and document-store checks for touched docs. Then allow diffing that bundle against a known-good run.

For the failure taxonomy, I’d consider splitting RANKING_FAILURE into “right doc retrieved but below cutoff” vs “right doc present but lost after rerank/context packing.” Those require different fixes and are easy to conflate when looking only at final answer quality.

[rautaditya2606]

Thanks — both of these are going on the roadmap.

The debug bundle makes sense as a first-class tool rather than something assembled manually from the three existing ones. Planning to implement it as collect_debug_bundle(query, pipeline, document_store) that serializes to JSON on disk — pipeline graph/version, component params, raw top-k before reranking, reranked top-k, scores, metadata, final prompt, answer, and corpus checks scoped to the touched document IDs. File-based so failures are diffable across runs and reproducible outside the original environment. The diff would be a CLI command comparing two bundle files: score deltas per doc, which docs appeared or disappeared between stages, config changes.

On the taxonomy split — that distinction is real and currently gets conflated. Plan is to split RANKING_FAILURE into SCORE_BELOW_CUTOFF (right doc retrieved but didn't clear the threshold) and CONTEXT_LOSS (right doc survived retrieval but was demoted by the reranker or truncated during context packing). Those two require completely different fixes and right now the classifier can't tell them apart. Will need a relevant_doc_id hint to trace the doc through each stage — without it the classifier can't know which doc was expected. Thinking of an optional param on both the CLI and MCP tool.

If you're open to it, would value your feedback on an early version once the bundle is built.