Your MLflow metrics are green. Your RAG answers are trash. Here is the 16-problem map I ended up using.

[onestardao]

quick context: I have been debugging RAG and LLM pipelines that log into MLflow for the past year. The same pattern kept showing up.

The MLflow UI looks fine. Hit-rate is fine. Latency is fine. Your eval score is “good enough”. Every scalar metric sits in the green zone.

Then a user sends you a screenshot.

The answer cites the wrong document. Or it blends two unrelated support tickets. Or it invents a parameter that never existed in your codebase. You dig into artifacts and the retrieved chunks look “sort of related” but not actually on target. You tweak a threshold, change top-k, maybe swap the embedding model, re-run, and a different weird failure appears.

Most teams call all of this “hallucination” and start tuning everything at once. That word is too vague to fix anything.

I eventually gave up on that label and built a failure map instead.

Over about a year of reviewing real pipelines, I collected 16 very repeatable failure modes for RAG and agent-style systems. I kept reusing the same map with different teams. Last week I finally wrote it up for MLflow users and compressed it into two things:

• one hi-res debug card PNG that any strong LLM can read
• one system prompt that turns any chat box into a “RAG failure clinic for MLflow runs”

article (full write-up and prompt):
https://psbigbig.medium.com/the-16-problem-rag-map-how-to-debug-failing-mlflow-runs-with-a-single-screenshot-6563f5bee003

the idea is very simple:

1. Download the full-resolution debug card from GitHub.
2. Open your favourite strong LLM (ChatGPT, Claude, Gemini, Grok, Kimi, Perplexity, your internal assistant).
3. Upload the card.
4. Paste the context for one failing MLflow run:
   • task and run id
   • key parameters and metrics
   • question (Q), retrieved evidence (E), prompt (P), answer (A)
5. Ask the model to use the 16-problem map and tell you:
   • which numbered failure modes (No.1–No.16) are likely active here
   • which one or two structural levers you should try first

If you tag the run with something like:

• wfgy_problem_no = 5,1
• wfgy_lane = IN,RE

you suddenly get a new axis for browsing your MLflow history. Instead of “all runs with eval_score > 0.7”, you can ask “all runs that look like semantic mismatch between query and embedding” or “all runs that show deployment bootstrap issues”.

The map itself is designed to sit before infra. You do not have to change MLflow or adopt a new service. You keep logging as usual, then add a very small schema on top:

• question
• retrieval queries and top chunks
• prompt template
• answer
• any eval signals you already track

The debug card is the visual version. The article also includes a full system prompt called “RAG Failure Clinic for MLflow (ProblemMap edition)” which you can paste into any system field. That version makes the model behave like a structured triage assistant: it has names and definitions for the 16 problems, uses a simple semantic stress scalar for “how bad is this mismatch”, and proposes minimal repairs instead of “rebuild everything”.

This is not a brand new idea out of nowhere. Earlier versions of the same 16-problem map have already been adapted into a few public projects:

• RAGFlow ships a failure-modes checklist in their docs, adapted from this map as a step-by-step RAG troubleshooting guide.
• LlamaIndex integrated a similar 16-problem checklist into their RAG troubleshooting docs.
• Harvard MIMS Lab’s ToolUniverse exposes a triage tool that wraps a condensed subset of the map for incident tags.
• QCRI’s multimodal RAG survey cites this family of ideas as a practical diagnostic reference.

None of them uses the exact same poster you see in the article. Each team rewrote it for their stack. The MLflow piece is the first time I aimed the full map directly at MLflow users and attached a ready-to-use card and clinic prompt.

If you want to try it in a very low-risk way, here is a minimal recipe that takes about 5 minutes:

1. Pick three to five MLflow runs that look fine in metrics but have clear user complaints.
2. Download the debug card, upload it into your favourite LLM.
3. For one run, paste task, run id, key config, metrics, and one or two bad Q/A pairs.
4. Ask the model to classify the run into problem numbers No.1–No.16 and suggest one or two minimal structural fixes.
5. Write those numbers back as tags on the run. Repeat for a few runs and see which numbers cluster.

If you do try this on real MLflow runs, I would honestly be more interested in your failure distribution than in stars. For example:

• do you mostly see input / retrieval problems, or reasoning / state, or infra and deployment?
• does your “hallucination” bucket secretly split into three or four very different patterns?
• does tagging runs this way actually change what you fix first?

The article has all the details, the full prompt, and the GitHub links to the card. Everything is MIT licensed and you can fork or drop it into your own docs if it turns out to be useful.

Happy to answer questions or hear counter-examples if you think the 16-problem taxonomy is missing something important.

WFGY 3.0 · RAG 16 Problem Map — Global Debug Card
https://github.com/onestardao/WFGY/blob/main/ProblemMap/wfgy-rag-16-problem-map-global-debug-card.md