Round Trip Validation

Round-Trip Validation

Compression can lose meaning. Round-trip validation measures how much: the notation is decoded back into natural language and compared with the original text.

original text -> notation -> decoded text -> compare with original -> confidence

noema.compress() runs this for you and returns the score as result.confidence.

Embedding check (default)

import noema

check = noema.validate(original_text, decoded_text)

check.confidence   # 0 to 1
check.passed       # confidence >= threshold
check.threshold    # the threshold used (default 0.75)

The score is the cosine similarity of sentence embeddings, clipped to the range 0 to 1. Empty decoded text scores 0.0 without loading a model.

Options: threshold (default 0.75) and model_name (default all-MiniLM-L6-v2).

What it can and cannot catch

Embedding similarity is a fast, cheap signal. It detects text that drifted or lost most of its content. It does not detect a reversed claim: "Google acquired YouTube" and "YouTube acquired Google" have very similar embeddings. For that, use the entailment check below.

Entailment check (stronger)

An NLI model checks whether the original text actually entails the decoded text, which catches contradictions that similarity misses.

from noema.validator.nli import validate_nli

check = validate_nli(original_text, decoded_text)

check.confidence       # entailment probability
check.entailment
check.neutral
check.contradiction
check.passed

confidence is the entailment probability specifically. A neutral pair (unrelated but not contradicting) also scores low. contradiction and neutral are exposed separately so you can tell the two failure modes apart.

The check runs cross-encoder/nli-deberta-v3-base by default and is slower than the embedding check, so it is not used inside compress(). It tests that the decoded text does not distort a claim the original made. It does not test completeness, meaning whether every original detail was restated.

Using the scores

• Treat confidence as a diagnostic, not a proof. A high score means the decoded text is close to the original, not that it is correct.
• When confidence is low, read result.decoded_text next to your input to see what is missing.
• Decoding is template-based and grammatically rough, which lowers similarity a little even when no fact was lost. Compare scores across your own texts rather than against a fixed target, and set validation_threshold to suit your data.

Related

Compressing Text, Notation Format, Advanced NLP