Compressing Text

Compressing Text

noema.compress() is the one-off path: it takes a single text, builds a graph, canonicalizes it, encodes it as notation and scores the result. It does not keep a knowledge base. For many documents or later retrieval, use Knowledge Bases.

import noema

result = noema.compress(
    "Larry Page founded Google in 1998. Sundar Pichai is the CEO of Google."
)

The result

compress() returns a CompressResult:

Field	Meaning
notation	The compact logical form. See Notation Format
decoded_text	The notation rendered back into sentences, which confidence was scored against
confidence	Round-trip validation score from 0 to 1. See Round-Trip Validation
saved_percent	Token reduction versus the original text, never below 0
original_tokens	Token count of the input
notation_tokens	Token count of the notation

Options

All options are keyword-only.

Option	Default	Meaning
similarity_threshold	the canonicalizer default (0.85)	Entity merge threshold
validation_threshold	0.75	Confidence needed to count as passed
token_model	cl100k_base	tiktoken encoding used for token counts
spacy_model	en_core_web_sm	spaCy pipeline for entities and relations

Reading the numbers

• A low confidence score means the decoded text drifted from the original. Compare result.decoded_text with your input to see what was lost.
• saved_percent of 0 with non-empty notation means the notation was not shorter than the text. This happens with short or fact-sparse input.
• Empty notation means no facts were extracted, which is an extraction miss, not full compression. Noema reports 0 saved in that case.
• Token counts use tiktoken. For counts that match a specific model, see Integrations.

Decoding

text = noema.decode(result.notation)

Decoding is template-based, not a generative model, so sentences are readable but can be grammatically rough. See Notation Format.

From the command line

noema compress "Larry Page founded Google in 1998."
noema compress --file article.txt

See CLI Reference.