-
Notifications
You must be signed in to change notification settings - Fork 1
Vanilla RAG Failure Mode
type: concept up: "Markov-Retrieval-Tutorial" tags: [rag, embeddings, failure-mode, motivation] related: ["MiniLM-Embeddings", "Heterogeneous-Graph", "Bridge-Entity"]
The structural failure of nearest-neighbor retrieval on tail-vocabulary items is the central problem the tutorial motivates.
Vanilla retrieval-augmented generation embeds chunks into a vector space and answers queries by retrieving the nearest neighbors of the query embedding. This works well when the corpus and the query share canonical vocabulary. It fails predictably when they do not.
The example from the tutorial: a query for "Indiana manufacturers with additive manufacturing capability" will find a company whose chunk says "additive manufacturing" explicitly. It will miss a company that does the same thing but writes "metal 3D printing," "laser sintering," or "powder bed fusion," because those chunks sit further from the query in embedding space.
This is a property of nearest-neighbor retrieval over any non-uniform distribution, not a property of any specific encoder. Embedding spaces have geometry. That geometry favors items whose vocabulary is typical of the corpus. Items in the long tail of the vocabulary distribution sit further from typical queries by construction, regardless of their actual relevance.
Swapping in a stronger encoder shifts the geometry but does not change the principle.
Figure 1 of the tutorial PDF plots the corpus in 2D using UMAP. For the query "metal 3D printing in Indiana":
- AM-related entities (TailCo, Sintron, PowderTech, ITAMCO's processes chunk) sit near the query.
- Canonical-only manufacturers (PrecisionWorks, MidwestPrecision, AcmeMfg, HoosierMetals) sit in a band on the left, far from the query.
- Vanilla RAG retrieves only what is geographically near the query. It cannot connect the two clusters.
For any query whose answer requires combining information from both clusters, no single nearest-neighbor lookup will work.
The corpus has structure beyond the embedding space. Documents contain chunks. Chunks mention concepts. Concepts co-occur. These relationships form a graph that the embedding space discards. Graph retrieval can reach items via paths through other items, not only items that are directly near the query.
See Heterogeneous Graph for the substrate, Random Walk With Restart for the algorithm that turns "a path exists" into "this item is relevant," and Bridge Entity for the corpus structure that makes paths form in the first place.
If retrieval is bottlenecked by what the embedding sees as similar, the fix can live at retrieval time (graph traversal, concept-boost re-ranking) or at extraction time (canonicalize vocabulary, tag related concepts at indexing). The tutorial argues both fixes are doing the same job in different parts of the pipeline. See Extraction-Retrieval Co-Design.