Advanced NLP

Advanced NLP

The default pipeline is spaCy named entity recognition plus rule-based relation patterns. It is fast and precise on simple sentences. This page covers three optional tools for harder text.

These tools are separate functions. They are not switched on inside noema.compress() or KnowledgeBase.ingest(), which always use the default pipeline. You call them yourself, as shown below.

Coreference resolution

Pronouns and descriptions like "It" or "the company" are not entities, so a sentence such as "It is headquartered in Mountain View." produces no relation by itself. Coreference resolution rewrites each mention to the cluster's most specific name first ("It" becomes "Google").

Install the extra:

pip install "noema-rag[advanced-nlp]"

Resolve, then ingest:

import noema
from noema import Document
from noema.parser.coref import resolve_coreferences

doc = Document(text="Google was founded in 1998. It is headquartered in Mountain View.")
resolved = resolve_coreferences(doc)      # a new Document, the input is not changed

print(resolved.text)

kb = noema.KnowledgeBase()
kb.ingest(resolved)

The default model is biu-nlp/f-coref. It downloads on first use. Resolution happens within one document, not across documents.

Transformer relation extraction

Rule-based patterns miss sentences with unusual structure. extract_relations_transformer() uses a REBEL model (Babelscape/rebel-large by default) to extract triplets. A triplet is kept only if both its head and tail match an entity already extracted from the same document.

extract_relations_hybrid() is usually the better choice. It runs the rule-based pass first, then runs the transformer only on sentences where the rule-based pass found nothing.

import noema
from noema import Document, ParsedDocument, SemanticGraph
from noema.parser.transformer_relations import extract_relations_hybrid

doc = Document(text=text)
entities = noema.extract_entities(doc)
relations = extract_relations_hybrid(doc, entities)

graph = SemanticGraph()
graph.add_document(ParsedDocument(document=doc, entities=entities, relations=relations))
graph = noema.canonicalize_entities(graph)

noema.save_bundle("hybrid_kb.noema", graph)
kb = noema.KnowledgeBase.load("hybrid_kb.noema")

Transformer relations are stored with confidence 0.75, and rule-based relations with 1.0. The REBEL model is large, so the first call downloads a lot of data and runs slowly on CPU.

KnowledgeBase has no option for a different relation extractor. The example above builds the graph yourself, saves it as a bundle, and loads it as a knowledge base. Documents ingested into that knowledge base later use the default rule-based pipeline.

NLI validation

For a stronger meaning-preservation check than embedding similarity, see the entailment check in Round-Trip Validation.

Pinned dependency

Noema pins transformers>=4.40,<5.0. Version 5 breaks fastcoref model loading, so do not upgrade transformers past the pin.

Related

Core Concepts, Knowledge Bases