Pari 0.2.0 alpha

Pari 0.2.0 alpha

Pari 0.2.0 turns the post-0.1 engine work into a coordinated Rust, Python, and CLI release. It keeps the 0.1 signature and file-format contracts while adding higher-level deduplication, operational tooling, dataset adapters, and current native-Linux benchmark evidence.

Install

Python 3.10 or newer:

python -m pip install "pari-similarity==0.2.0"

The distribution name remains pari-similarity; Python code continues to import pari.

Rust 1.81 or newer:

cargo add pari-core@0.2.0 pari-index@0.2.0 pari-store@0.2.0

The four public crates are released together. Add pari-format@0.2.0 only when working directly with the codecs or persisted-format metadata.

Signed-tag release artifacts include CLI archives for Linux x86-64, macOS arm64, and Windows x86-64, plus SHA256SUMS, a CycloneDX SBOM, and build provenance.

Highlights

• DedupeIndex and deduplicate provide typed, bounded-batch Python deduplication with memory or local persistence, exact verification, representative selection, and progress cancellation.
• The text workload builds reusable reference indexes and runs mutating deduplication or non-mutating cross-corpus audits. Code-corpus and entity-matching examples add domain-specific feature extraction and evaluation.
• Datasketch 2.x affine32 adapters can import compatible signatures. Golden tests pin value-level interoperability before the migration benchmark reports performance.
• Batch signature construction uses deterministic bounded CPU parallelism in Rust and Python. The automatic policy remains capped and preserves input order.
• Query observability, exact bucket distributions, and progress events expose operational state without changing persisted data.
• plan_lsh, Index.explain, and the CLI plan/explain commands expose the versioned analytical planner and storage recommendation model.
• Optional PyArrow/Parquet, Polars, and Hugging Face adapters consume bounded batches rather than materializing whole datasets.
• Reference workloads publish outputs transactionally and use atomic no-replace claims so failures or concurrent writers cannot leave misleading final artifacts.

Compatibility and migration

Existing 0.1 MinHash, Index, IndexStats, error classes, Rust MinHash/index/store types, CLI commands, and machine-readable revision-1 fields remain supported. The pari-affine32-v1 and pari-affine64-v1 signature schemes are unchanged. Format-v1 .pari files remain readable; 0.2 does not silently reinterpret existing bytes.

Python users can upgrade in place. The new top-level deduplication and progress types are additive. Planner types are available at the top level but remain experimental because their model coefficients and recommendations may evolve at a future minor release.

Rust applications should update pari-core, pari-format, pari-index, and pari-store together when more than one is used. Published inter-crate dependencies require the exact coordinated 0.2.0 version. The minimum Rust version remains 1.81.

CLI machine-readable output remains revision 1. Existing fields keep their meaning and type; consumers should continue to ignore unknown fields. stats adds observability and bucket-distribution fields. plan and explain are experimental commands with a model-labeled output contract. Progress is written to stderr and does not contaminate stdout JSON or JSONL.

Datasketch migration is opt-in. Pari's affine32 adapter can reproduce Datasketch 2.x affine signatures for imported permutations, but Pari and Datasketch use different default seed-to-permutation mappings and LSH implementations. Do not assume byte-identical default signatures or identical candidate sets; follow the interoperability guide.

The complete supported/experimental classification is in the v0.x compatibility contract.

Performance evidence

The selected native-Linux 100K and 1M campaign report records exact commands, environment, cache policy, checksums, correctness gates, storage measurements, text audits, and the Datasketch semantic baseline. On the recorded four-CPU runners, the automatic policy used four signature threads; the 1M bundle recorded 322.04K signatures/s, 0.57 near-threshold candidate recall, and persistent/lazy candidate parity. These numbers apply only to the linked environment and workload.

Historical WSL bundles remain available under their original source SHAs. They are not treated as same-machine baselines for a speedup claim.

Experimental surfaces and known limits

• Planner coefficients and storage recommendations, optional Datasketch and dataset adapters, Redis namespace/descriptor bytes, observability measurement policy beyond supported fields, and direct builder/lazy-store APIs are experimental.
• The committed scale envelope is 1M items. This release makes no 10M, distributed sharding, GPU, out-of-core clustering, Weighted MinHash, or SimHash performance claim.
• Redis is a shared runtime backend rather than an archival format. Use the versioned .pari format for durable interchange.
• CLI archives cover Linux x86-64, macOS arm64, and Windows x86-64. Linux arm64 artifacts remain optional future work.
• LSH results are approximate candidates. Applications that need an exact similarity threshold must retain or reconstruct source signatures and verify candidates.

Security and provenance

The release workflow builds and install-tests artifacts without publication credentials, publishes through short-lived PyPI and crates.io OIDC tokens, attests the assembled files, and creates the GitHub prerelease only after publication succeeds. Report vulnerabilities through GitHub's private process described in SECURITY.md.

Pari 0.1.0 alpha

Pari 0.1.0 alpha

Pari 0.1.0 is the first public alpha of the Rust-powered similarity indexing and deduplication engine.

Highlights

• Batch-first MinHash and LSH in safe Rust.
• Persistent .pari indexes with crash-safe commits and lazy on-disk bucket reads.
• Bounded-memory external index construction.
• Native duplicate grouping without materializing candidate graphs in Python.
• Typed Python API with abi3 wheels for CPython 3.10+.
• Streaming CLI for index, search, dedup, stats, and verify workflows.
• Pluggable shared storage with memory and Redis backends.
• Reproducible benchmark and storage-layout tooling.

Install

Python distribution:

python -m pip install pari-similarity==0.1.0

The import package remains:

from pari import Index, MinHash

CLI archives for Linux, macOS, and Windows are attached to the GitHub Release.

Compatibility

This is an alpha release. See docs/compatibility.md for the exact v0.x compatibility contract, including Python, Rust, CLI machine-readable output, signature schemes, remote backend descriptors, and .pari format v1.

Security and provenance

Release artifacts are built and install-tested before publication. GitHub release artifacts include SHA-256 checksums, a CycloneDX dependency SBOM, and GitHub build provenance attestations where supported. Python publication uses PyPI Trusted Publishing rather than a stored API token.