-
Notifications
You must be signed in to change notification settings - Fork 1
Tutorial Queries
type: entity up: "Markov-Retrieval-Tutorial" tags: [queries, evaluation, findings] related: ["Tutorial-Corpus", "Path-Length-Decomposition", "Bridge-Entity"]
The five queries the tutorial runs against the tutorial corpus, and the empirical findings the writeup tabulates.
| Outcome | Count |
|---|---|
| Chain produced clearly better rankings than vanilla RAG | 3 |
| Chain and vanilla performed similarly | 1 |
| Chain partially bridged but could not reach the broader cluster | 1 |
The chain is not uniformly better. It is better where the graph contains paths the chain can exploit, and not better when the right answers are already near the query in embedding space.
"DLA suppliers with defense procurement experience."
This query is acronym-heavy. The encoder treats "DLA" as opaque and only weakly connects it to canonical defense vocabulary like "defense primes" or "ITAR registered." Most companies in the corpus do not use the term DLA.
The chain recovers HoosierMetals (the bridge entity, its overview chunk ranked fourth) because its chunks use both vocabularies. It does not bridge into the broader canonical cluster (ITAMCO, AcmeMfg, AeroForge) because one bridge is not enough signal.
Path-length decomposition for two chunks under this query, from Figure 3 of the writeup:
| Chunk | Total stationary mass | Path-length signature |
|---|---|---|
| HoosierMetals overview (recovered, rank 4) | 0.0315 | Clean spike at k=1, sustained through k=3 |
| ITAMCO capabilities (not recovered) | 0.013 | Flat, small contributions across all k, none above 0.0022 |
The contrast is not about ranking. It is about reachability. HoosierMetals is reachable from the query concepts through short paths; ITAMCO is not.
The query Figure 1 visualizes. AM-related entities (TailCo, ITAMCO's processes chunk, Sintron, PowderTech) sit near the query in embedding space. Vanilla RAG handles this kind of query well; the chain's contribution here is reordering, not pure recovery.
These are the three observations the writeup foregrounds in section 5, each with a specific consequence for how the architecture should be presented.
-
Reordering wins, not pure recovery wins. On most queries, the items the chain promotes are also in the vanilla top-5; the chain just orders them better by reasoning over multiple query anchors and document-level reinforcement. Pure tail recovery (a chunk with zero direct query similarity surfacing into the top-5 entirely through multi-hop paths) does happen, most clearly with TailCo's secondary chunks and PowderTech's quality chunk, but the magnitudes are modest. At larger corpus scale this should grow.
-
The chain inherits encoder limitations. The DLA query brought up FoodCo's distribution chunk at rank 5 because the encoder placed "cold chain logistics" near "Defense Logistics Agency" in the starting distribution by surface word overlap. The chain walked from there. No structural reasoning can correct a starting distribution that already misweights the query. This is the closest thing to an honest limitation of the architecture, and it points at the importance of how the seed distribution π₀ is constructed.
-
The architecture has costs. Concept extraction at indexing time is non-trivial. The chain is more expensive than nearest-neighbor lookup (still under a second per query on 10⁴ to 10⁵ nodes, but construction needs care at very large scale). The transition matrix has parameters (type weights, sparsification thresholds, restart probability) that benefit from calibration against the query distribution. None of these costs are prohibitive, but they are real, and a system that does not need them should not pay them.
QUERIES in corpus.py. The orchestration that runs all five and prints the rankings, the starting distributions, the chain rankings, and the path-length decompositions is run_tutorial.py.
When tuning chain hyperparameters, evaluate against all five queries, not one. The path-length decompositions should still tell a coherent story after any change.