-
Notifications
You must be signed in to change notification settings - Fork 1
Tutorial Corpus
type: entity up: "Markov-Retrieval-Tutorial" tags: [corpus, data, manufacturing] related: ["Bridge-Entity", "Heterogeneous-Graph", "Tutorial-Queries"]
The synthetic corpus of 24 chunks across 12 manufacturing entities, defined in corpus.py. Designed so every chunk and every concept can be inspected by hand.
Real industrial-base corpora contain entities that use canonical vocabulary, entities that use tail or specialist vocabulary, bilingual bridges, acronym-only suppliers, and off-topic distractors. The vocabulary distribution is genuinely non-uniform. The tutorial corpus is sized to make every behavior of the chain inspectable while keeping that realistic shape.
Each entity is motivated by a corpus-realism argument (this kind of entity should exist in any reasonable corpus of this type), not by a result-engineering argument (this entity will make a specific query work better). CLAUDE.md is explicit about this distinction.
| Category | Vocabulary used | Entities |
|---|---|---|
| Canonical manufacturers | Standard industry vocabulary | ITAMCO, AcmeMfg, AeroForge, MidwestPrecision, PrecisionWorks, ChicagoCo |
| Vocabulary-tail | Same capabilities described in different words | TailCo, PowderTech, Sintron |
| Bilingual bridge | Both canonical and procurement vocabularies in the same chunk | HoosierMetals |
| Acronym-tail | Defense procurement jargon only | LegacyShop |
| Off-topic distractor | Unrelated to manufacturing | FoodCo |
See Bridge Entity for why the bilingual category matters more than the others for graph retrieval.
Entities are placed in Indiana, Michigan, Ohio, and Illinois so that location-anchored queries (for example "Indiana manufacturers...") have meaningful right and wrong answers.
| Entity | Approximate location |
|---|---|
| ITAMCO, HoosierMetals, TailCo | Indiana |
| MidwestPrecision | Michigan |
| PrecisionWorks | Ohio |
| ChicagoCo | Illinois |
| AcmeMfg, AeroForge, PowderTech, Sintron, LegacyShop, FoodCo | Various |
The README and the writeup both treat geography as a query-time filter, not as a graph-time concept.
Each chunk in CORPUS carries a concepts list. These are the open-vocabulary terms that drive chunk-to-concept edges in the graph. They are manually curated for the tutorial. A real pipeline would extract them with NER or LLM-driven extraction; that step is out of scope here.
DIAGNOSTIC_CONCEPT_PAIRS in corpus.py records pairs whose cosine similarity is printed at run time as a sanity check on what the encoder treats as related.
Add chunk dicts to CORPUS. Each new concept must appear in at least one chunk's concepts list before it can appear in DIAGNOSTIC_CONCEPT_PAIRS. Changes to corpus shape can move the headline numbers cited in the writeup, so regenerate figures and update docs/tutorial.tex to match. See the reproducibility note in Markov Retrieval Tutorial.