Large Datasets and Backends

Large Datasets and Backends

A .noema bundle loads the whole graph into memory. This page covers ingesting large document sets in batches, and moving the graph into an external store when a bundle stops being practical.

Streaming ingestion

import noema

def read_documents():
    for path in paths:
        yield path.read_text(encoding="utf-8")   # read lazily, one at a time

kb = noema.KnowledgeBase()
report = kb.ingest_stream(
    read_documents(),
    batch_size=100,
    checkpoint_path="kb.noema",
    on_batch=lambda index, batch_report: print(index, batch_report.documents_ingested),
)

Option	Default	Meaning
documents		Any iterable of strings or Document objects, including a generator
batch_size	100	Documents parsed and canonicalized together
checkpoint_path	None	If set, the knowledge base is saved here after every batch
on_batch	None	Called after each batch as on_batch(batch_index, batch_report), with batch_index starting at 1

Returns one IngestReport with totals across all batches.

• The input never has to fit in memory at once, because a generator is read one batch at a time.
• Each batch is a normal ingest() call, so the cost per batch still grows with the size of the knowledge base. Batching bounds memory use, not total time.
• With checkpoint_path, a crash only loses the batch in progress. To resume, load() the checkpoint, skip the documents already ingested, and call ingest_stream() again.
• The CLI wraps this with a progress bar: noema build-graph. See CLI Reference.

Persistent backends

When a knowledge base outgrows a single in-memory bundle, push the graph to an external store.

kb.sync_to_backend(backend)                              # write the whole graph
kb = noema.KnowledgeBase.load_from_backend(backend)      # read it all back into memory

sync_to_backend() replaces whatever the backend held. load_from_backend() reads the backend's full contents into a new in-memory knowledge base, so retrieval still runs in memory. Backends take over storage and durability, not query execution.

load_from_backend() accepts embedding_model and spacy_model. Backends store the graph, not the embedding index, so use the same embedding model you built the graph with. The index is built on the first retrieve().

Neo4j

pip install "noema-rag[neo4j]"

from noema.storage.neo4j_backend import Neo4jBackend

with Neo4jBackend("bolt://localhost:7687", auth=("neo4j", "password"), database="neo4j") as backend:
    kb.sync_to_backend(backend)

Entities become :Entity nodes with a type label such as :PERSON. Relations become relationships between them, and events become :Event nodes linked to their participants. Writes use MERGE, so repeated writes of the same ids update rather than duplicate.

RDF

pip install "noema-rag[rdf]"

from noema.storage.rdf_backend import RDFBackend

backend = RDFBackend("kb.ttl", format="turtle")
kb.sync_to_backend(backend)

Option	Default	Meaning
path	None	File to load on open and write on flush(). None keeps the graph in memory only
format	"turtle"	Any rdflib serialization format

Entities become subjects in the http://noema.dev/kg# namespace. A relation is a triple using its predicate as the RDF predicate, plus a reified node that carries the relation's id, confidence and mentions. A path that does not exist yet starts an empty graph, and the file is created on the first flush.

Writing your own backend

Subclass noema.GraphBackend and implement:

upsert_entities, upsert_relations, upsert_events, get_entity, all_entities, all_relations, all_events, entity_count and clear.

flush() and close() are optional and do nothing by default. Backends work as context managers, closing on exit. Calls to upsert_* may be batched internally, so treat writes as visible to reads only after flush(). Relations and events refer to entities by id, and entities are always written first.

Related

Knowledge Bases, CLI Reference, Installation