[Proposal] Example: streaming RAG diagnostics with the WFGY 16-problem map in Pathway

[onestardao]

Is your feature request related to a problem? Please describe.

Many Pathway users build RAG systems, but when the answers look wrong it is hard to tell why they failed. It is not obvious whether the root cause is retrieval collapse, chunk drift, embedding mismatch, or something in the orchestration logic. People often end up debugging by trial and error on raw logs instead of following a structured checklist.

Describe the solution you'd like

I would like to contribute (or help design) a small example that uses Pathway to stream RAG diagnostics using the WFGY 16-problem map:

https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

The example could show a simple text RAG pipeline and log, for each query, which WFGY problem codes are suspected (for example No.1 hallucination and chunk drift, No.2 retrieval interpretation collapse, No.5 semantic vs embedding mismatch, and so on). The implementation can stay inside Pathway primitives, with a light layer of heuristics on top of existing logs and metrics.

Describe alternatives you've considered

Right now I run WFGY diagnostics separately in notebooks and external evaluation tools, then manually correlate those results with Pathway logs. This works for my own experiments, but it is not reproducible for other users and it does not show up as an official example in the docs.

Additional context

WFGY and the 16-problem map are MIT licensed and already referenced by several research groups, for example:

• Harvard MIMS Lab ToolUniverse
• University of Innsbruck Data Science Group (Rankify RAG toolkit)
• QCRI LLM Lab Multimodal RAG Survey

If you feel this fits Pathway, I am happy to adapt the taxonomy and open a PR as a self contained example or small docs page, without adding any heavy dependencies.

[zxqfd555]

Hey @onestardao!

If you’re ready to contribute an example, we’d be happy to support that.

As I understand it, you’re proposing to write an article (please correct me if I’m mistaken) that demonstrates a RAG example built with Pathway (ideally leveraging Pathway’s native tools like llm-app wherever possible) and shows how it can be debugged using the WGFY tool.

If you’re willing to take this on, I suggest publishing the article under our Developers Templates section.

Please take a look at the existing examples there to get a sense of the style and structure we follow. We would also appreciate it if you could include a couple of illustrations, as they significantly improve clarity and readability.

As for where to commit the content, please use the following directory in the repository.

The content format should be Markdown. You can refer to the publicly available files in that folder for formatting guidance and conventions.

If you need any help or clarification along the way, feel free to reach out!

[onestardao]

Thanks, this is super helpful.

Yes, I can take this on. I’ll draft a Pathway-native RAG example (ideally using lm-app where it makes sense) and show how to stream diagnostics that map each query/run to a small set of suspected failure modes (retrieval hallucination, chunking issues, embedding/metric mismatch, config drift, observability gaps, and index lifecycle/deploy ordering).

Plan:

1. Start from an existing Developers Templates example and follow the same structure/style.
2. Add a small reproducible RAG pipeline + a “debug view” section that explains what to log/inspect and how to map symptoms to minimal fixes.
3. Include 1–2 simple illustrations (flow + symptom→inspect→fix table).

Two quick questions before I open the PR:

• Which exact directory should the Markdown live in for Developers Templates (and where should images/assets go)?
• Do you prefer the example to use a specific dataset/source (or should I keep it fully self-contained with a tiny local corpus)?

If you confirm the path + preferred example baseline, I’ll open a PR with a first version and iterate from there.

[zxqfd555]

Hey,

The markdown file should be here: https://github.com/pathwaycom/pathway/tree/main/docs/2.developers/7.templates/rag

As for the images, this part is not replicated to GitHub yet. To resolve this problem, you can host them somewhere - I'll then check them out, move to the static folder, and edit the links. Please use the SVG format.

We don't have a preference for the dataset as long as its license allows the usage in the tutorials, it's easily downloadable, and it's (at least mostly) in English.

[onestardao]

Thanks again for the guidance and for pointing me to the templates path.

I’ve opened PR #195 ([Docs] Add WFGY-based RAG diagnostics example) with a first docs-only version of the RAG diagnostics template. It follows the existing templates style, keeps everything implementation-agnostic, and does not add any new runtime dependencies.

If anything about the wording, placement, or level of detail would fit better with your docs conventions, I’m very happy to adjust.