Core Concepts

Core Concepts

The pipeline

Documents
    -> parser (entities, relations)
    -> semantic graph
    -> canonicalizer (merge duplicates against what is already stored)
    -> canonical graph + fact vector index        (saved in a .noema bundle)

Question
    -> vector search over facts
    -> graph expansion (hops)
    -> sub-graph encoded as notation under max_tokens
    -> your LLM prompt

Entities, relations and events

Type	Meaning	Example
Entity	A thing with a name and a type	Google, type ORG
Relation	A subject, predicate and object linking two entities	Larry Page -found-> Google
Event	A fact involving several entities and/or a time	*ipo (#1) @2004-08-19
Document	A source text, referenced by mentions

Entity types follow spaCy's labels: PERSON, ORG, GPE (countries, cities, states), LOC, DATE, EVENT, PRODUCT, MONEY, and OTHER for anything else.

The built-in parser extracts entities and relations. Events are part of the data model, the notation and the storage backends, and survive canonicalization, but the built-in parser does not create them. You can add events to a SemanticGraph yourself.

Every entity, relation and event carries its mentions (the source document and character span it came from) and a confidence value, so any fact can be traced back to its text.

Parsing

Entities come from spaCy named entity recognition. Relations come from rule-based patterns over the spaCy dependency parse, which handle simple subject-verb-object sentences and conjunctions. Predicates are normalized to lowercase snake_case, for example found or ceo_of.

Relation patterns are English-specific. A non-English spaCy model gives weaker relations than entities. For sentences the rule-based patterns miss, see Advanced NLP.

Canonicalization

The parser creates fresh entities for every mention, so the same real-world thing in two documents starts as two nodes. Canonicalization merges them:

1. Exact match after normalizing case, punctuation and business suffixes, so Google, Inc. and Google match.
2. Embedding similarity within the same entity type. Entities merge when similarity reaches the threshold (default 0.85).

Entities only merge within the same type, so a person named Washington never merges with the city. Relations and events that referenced a merged entity are remapped to the canonical one. Alternate surface forms are kept as the entity's aliases.

Tune the threshold with KnowledgeBase(similarity_threshold=...). Higher values merge less.

Notation

Notation is the compact text form of a sub-graph: one fact per line, no filler words. See Notation Format.

Retrieval

Each relation is embedded as its own tiny notation block. At query time Noema finds the facts closest to the question, expands outward through the graph, and assembles as many facts as fit in a token budget. See Retrieval.

Why this saves tokens

• Structure: no filler words or restated subjects
• Deduplication: a fact repeated across documents is stored and retrieved once
• Query-time slicing: only the relevant sub-graph reaches the prompt, not whole chunks

How much you save depends on your text. Fact-dense prose compresses well. Short or fact-sparse text may not. Measure on your own data, and see Benchmarks for a way to do it.

The .noema bundle

A knowledge base saves to one zip file:

kb.noema
|-- graph.json        canonical entities, relations and events, with provenance
|-- embeddings.bin    float32 vectors, one row per fact
|-- index.faiss       similarity index over those vectors
`-- manifest.json     schema version, counts, embedding and spaCy model names

graph.json is the source of truth. The embeddings and index are built from the notation of each fact, not from raw sentences. Loading a bundle reuses the saved index, so the first retrieve() after load() does not re-embed anything.