Pari 0.2.0 alpha

Pari 0.2.0 alpha

Pari 0.2.0 turns the post-0.1 engine work into a coordinated Rust, Python, and CLI release. It keeps the 0.1 signature and file-format contracts while adding higher-level deduplication, operational tooling, dataset adapters, and current native-Linux benchmark evidence.

Install

Python 3.10 or newer:

python -m pip install "pari-similarity==0.2.0"

The distribution name remains pari-similarity; Python code continues to import pari.

Rust 1.81 or newer:

cargo add pari-core@0.2.0 pari-index@0.2.0 pari-store@0.2.0

The four public crates are released together. Add pari-format@0.2.0 only when working directly with the codecs or persisted-format metadata.

Signed-tag release artifacts include CLI archives for Linux x86-64, macOS arm64, and Windows x86-64, plus SHA256SUMS, a CycloneDX SBOM, and build provenance.

Highlights

• DedupeIndex and deduplicate provide typed, bounded-batch Python deduplication with memory or local persistence, exact verification, representative selection, and progress cancellation.
• The text workload builds reusable reference indexes and runs mutating deduplication or non-mutating cross-corpus audits. Code-corpus and entity-matching examples add domain-specific feature extraction and evaluation.
• Datasketch 2.x affine32 adapters can import compatible signatures. Golden tests pin value-level interoperability before the migration benchmark reports performance.
• Batch signature construction uses deterministic bounded CPU parallelism in Rust and Python. The automatic policy remains capped and preserves input order.
• Query observability, exact bucket distributions, and progress events expose operational state without changing persisted data.
• plan_lsh, Index.explain, and the CLI plan/explain commands expose the versioned analytical planner and storage recommendation model.
• Optional PyArrow/Parquet, Polars, and Hugging Face adapters consume bounded batches rather than materializing whole datasets.
• Reference workloads publish outputs transactionally and use atomic no-replace claims so failures or concurrent writers cannot leave misleading final artifacts.

Compatibility and migration

Existing 0.1 MinHash, Index, IndexStats, error classes, Rust MinHash/index/store types, CLI commands, and machine-readable revision-1 fields remain supported. The pari-affine32-v1 and pari-affine64-v1 signature schemes are unchanged. Format-v1 .pari files remain readable; 0.2 does not silently reinterpret existing bytes.

Python users can upgrade in place. The new top-level deduplication and progress types are additive. Planner types are available at the top level but remain experimental because their model coefficients and recommendations may evolve at a future minor release.

Rust applications should update pari-core, pari-format, pari-index, and pari-store together when more than one is used. Published inter-crate dependencies require the exact coordinated 0.2.0 version. The minimum Rust version remains 1.81.

CLI machine-readable output remains revision 1. Existing fields keep their meaning and type; consumers should continue to ignore unknown fields. stats adds observability and bucket-distribution fields. plan and explain are experimental commands with a model-labeled output contract. Progress is written to stderr and does not contaminate stdout JSON or JSONL.

Datasketch migration is opt-in. Pari's affine32 adapter can reproduce Datasketch 2.x affine signatures for imported permutations, but Pari and Datasketch use different default seed-to-permutation mappings and LSH implementations. Do not assume byte-identical default signatures or identical candidate sets; follow the interoperability guide.

The complete supported/experimental classification is in the v0.x compatibility contract.

Performance evidence

The selected native-Linux 100K and 1M campaign report records exact commands, environment, cache policy, checksums, correctness gates, storage measurements, text audits, and the Datasketch semantic baseline. On the recorded four-CPU runners, the automatic policy used four signature threads; the 1M bundle recorded 322.04K signatures/s, 0.57 near-threshold candidate recall, and persistent/lazy candidate parity. These numbers apply only to the linked environment and workload.

Historical WSL bundles remain available under their original source SHAs. They are not treated as same-machine baselines for a speedup claim.

Experimental surfaces and known limits

• Planner coefficients and storage recommendations, optional Datasketch and dataset adapters, Redis namespace/descriptor bytes, observability measurement policy beyond supported fields, and direct builder/lazy-store APIs are experimental.
• The committed scale envelope is 1M items. This release makes no 10M, distributed sharding, GPU, out-of-core clustering, Weighted MinHash, or SimHash performance claim.
• Redis is a shared runtime backend rather than an archival format. Use the versioned .pari format for durable interchange.
• CLI archives cover Linux x86-64, macOS arm64, and Windows x86-64. Linux arm64 artifacts remain optional future work.
• LSH results are approximate candidates. Applications that need an exact similarity threshold must retain or reconstruct source signatures and verify candidates.

Security and provenance

The release workflow builds and install-tests artifacts without publication credentials, publishes through short-lived PyPI and crates.io OIDC tokens, attests the assembled files, and creates the GitHub prerelease only after publication succeeds. Report vulnerabilities through GitHub's private process described in SECURITY.md.