Faster semantic deduplication

Faster semantic deduplication
The 1.0.0 macOS wheels were built with an older Rust compiler that made the dense comparison loop slower. Deduplicating 100,000 names with embeddings now takes 4.6 s instead of 7.6 s, with exactly the same results. Wheels are now built with a pinned compiler. Linux and Windows wheels were not affected.

No API or behavior changes: upgrading from 1.0.0 is safe.

pip install -U univo

v1.0.0

Univo 1.0 is the first stable release: fast, local duplicate detection for Python data.

pip install univo

What's in it

• Find duplicates in strings, records and nested Python objects, field by field, with an explanation for every match.
• Exact, fuzzy and semantic comparison, with a 15 MB multilingual model that runs locally and works offline.
• Partitions, typed IDs and vetoes are applied before any score.
• Complete search by default: every pair above the threshold, and a flag in the report whenever a limit is reached.
• Indexes you can update, save and load.
• Wheels for Linux, macOS and Windows, on x86_64 and arm64, for Python 3.11+. No dependencies, no Rust needed.

Good to know

• Scores rank candidates; they are not probabilities. Univo never merges or deletes records for you.
• What stays compatible across 1.x: stability.
• Benchmarks against SemHash: benchmarks.

v1.0.0-rc.1

First release candidate of Univo 1.0, published to TestPyPI for testing.