Knowledge Bases

Knowledge Bases

A KnowledgeBase holds a canonical graph plus the settings used to build it. It grows as you ingest documents, saves to a .noema bundle, and answers questions through retrieve().

import noema

kb = noema.KnowledgeBase()

Constructor options

Option	Default	Meaning
embedding_model	all-MiniLM-L6-v2	sentence-transformers model for deduplication and retrieval
similarity_threshold	0.85	Entity merge threshold. Higher merges less
spacy_model	en_core_web_sm	spaCy pipeline for parsing

The embedding model is recorded in saved bundles. KnowledgeBase.load() uses the model the bundle was built with, so queries are embedded the same way as the stored facts.

Ingesting documents

report = kb.ingest("Larry Page founded Google in 1998.")
report = kb.ingest(["Sundar Pichai is the CEO of Google.", "Google acquired YouTube in 2006."])

ingest() accepts a string, a Document, or a list mixing both. Use Document to attach an id or metadata:

from noema import Document

kb.ingest(Document(id="doc-42", text="...", metadata={"source": "annual-report"}))

New facts merge into the existing graph and entities are canonicalized against what is already stored, so known entities are merged (by exact match or embedding similarity) instead of duplicated.

The ingest report

ingest() returns an IngestReport:

Field	Meaning
documents_ingested	Documents processed in this call
entities_before / entities_after	Entity count before and after
entities_parsed	Entities the parser found in the new documents
relations_before / relations_after	Relation count before and after
entities_merged	Entities folded into another entity (property)

entities_merged is not just before - after, because the graph usually also grows with new entities. It compares the whole candidate pool with what remained.

For large document sets use ingest_stream(). See Large Datasets and Backends.

Saving and loading

kb.save("company_kb.noema")
kb = noema.KnowledgeBase.load("company_kb.noema")

The bundle layout is described in Core Concepts. You can inspect one without loading the whole thing:

noema inspect company_kb.noema

Looking at the graph

graph = kb.graph
len(graph)                      # number of nodes
graph.entities                  # list of Entity
graph.relations                 # list of Relation
graph.relations_of(entity_id)   # relations touching an entity
graph.neighbors(entity_id, hops=2)

Entity ids are short prefixed strings such as ent_9f3a2b1c.

Retrieving

context = kb.retrieve("Who is Google's CEO?")

See Retrieval.

Using coreference resolution

Pronouns like "It" are not recognized as entities, so a sentence like "It is headquartered in Mountain View." produces no relation on its own. Resolve coreferences first, then ingest the rewritten document. See Advanced NLP.