For Dask users running RAG on clusters: a 16 problem map and one debug card to name your failures.

[onestardao]

Hi all,

this is for people who run RAG or agent style pipelines on top of Dask.

I kept running into the same pattern last year. The Dask dashboard is green. Graphs complete, workers scale up and down, CPU and memory stay inside alerts. But users still send screenshots of answers that are subtly wrong.

Sometimes the model keeps quoting last month instead of last week. Sometimes it blends tickets from two customers. Sometimes every sentence is locally correct, but the high level claim is just wrong.

Most of the time we just say “hallucination” or “prompt issue” and start guessing. After a while that felt too coarse. Two jobs that both look like hallucination can have completely different root causes, especially once you have retrieval, embeddings, tools and long running graphs in the mix.

So I spent about a year turning those failures into a concrete map.
The result is a 16 problem failure vocabulary for RAG and LLM pipelines, plus a global debug card you can feed into any strong LLM.

For Dask users I just published a Dask specific guide here:

https://psbigbig.medium.com/your-dask-dashboard-is-green-your-rag-answers-are-wrong-here-is-a-16-problem-map-to-debug-them-f8a96c71cbf1

What is inside:

• a single visual debug card (poster) that lists the 16 problems and the four lanes
  (IN = input and retrieval, RE = reasoning, ST = state over time, OP = infra and deployment)
• an appendix system prompt called “RAG Failure Clinic for Dask pipelines (ProblemMap edition)”
• three levels of integration, from “upload the card and paste one failing job”
  up to “small internal assistant that tags Dask jobs with wfgy_problem_no and wfgy_lane”

The intended workflow is deliberately low tech.

You download the PNG once, open your favourite LLM, upload the image, paste a short job context
(question, chunks, prompt template, answer, plus a small sketch of the Dask graph)
and ask the model to tell you which problem numbers are active and what small structural fix to try first.

I tested this card and prompt on several LLMs (ChatGPT, Claude, Gemini, Grok, Kimi, Perplexity).
They can all read the poster and return consistent problem labels when given the same failing run.

Under the hood there is some structure (ΔS as a semantic stress scalar, four zones, and a few optional repair operators),
but you do not need any of that math to use the map. The main thing is that your team gets a shared language like
“this group of jobs is mostly No.5 plus a bit of No.1” instead of only “RAG is weird again”.

The map comes from an open source project I maintain called WFGY
(about 1.6k stars on GitHub right now, MIT license, focused on RAG and reasoning failures).

I would love feedback from Dask users:

• does this failure vocabulary feel useful on top of your existing dashboards
• are there Dask specific failure patterns I missed
• if you try the card on one of your own broken jobs, do the suggested problem numbers and fixes make sense

If it turns out to be genuinely helpful, I am happy to adapt the examples or the prompt so it fits better with how Dask teams actually run things in production.

WFGY 3.0 · RAG 16 Problem Map — Global Debug Card Home page
https://github.com/onestardao/WFGY/blob/main/ProblemMap/wfgy-rag-16-problem-map-global-debug-card.md