-
Notifications
You must be signed in to change notification settings - Fork 1
Heterogeneous Graph
type: concept up: "Markov-Retrieval-Tutorial" tags: [graph, data-structure, retrieval] related: ["Random-Walk-With-Restart", "MiniLM-Embeddings"]
The heterogeneous graph is the substrate on which the Random Walk With Restart operates.
| Type | What it represents |
|---|---|
| Chunk | A passage of original prose from a source document |
| Document | A source artifact that one or more chunks come from |
| Concept | An open-vocabulary term extracted and attached to one or more chunks |
Chunks remain as text-bearing nodes. The chain runs on the original prose, not on triple-extracted facts. This is the text fidelity commitment of the project.
- Chunk to document: each chunk belongs to exactly one document.
- Chunk to concept: each chunk lists the concepts attached to it (open vocabulary, see below).
- Chunk to chunk and concept to concept: weighted by MiniLM embedding cosine similarity, sparsified by threshold.
Edge weights between types are scaled by per-type weights set in build_transition_matrix.
Concepts are not declared by a formal ontology. They are attached to chunks during corpus construction and may include terms the system did not anticipate. This is the open vocabulary commitment: an ontology cannot host connections that have not been named, and discovery often turns on connections that have not yet been named.
In the tutorial corpus the concepts are manually curated as part of CORPUS in corpus.py. A real pipeline would extract them with NER or LLM-driven extraction. That extraction step is out of scope for the tutorial.
graph.py builds the graph deterministically from the corpus and the embedding model. corpus.py defines CORPUS (chunk dicts) and DIAGNOSTIC_CONCEPT_PAIRS (pairs whose embedding similarity is reported as a sanity check).
The tutorial graph is small on purpose: 12 entities, 24 chunks, and the concepts those chunks declare. Small enough that every chunk and every transition is inspectable. Two-stage scaling (concept-level filtering followed by the chain on the active subgraph) is the path to larger corpora and is mentioned in the writeup but not implemented here.