v0.7.0 -- Ray Distributed Backend

Ray Backend for Large-Scale Entity Resolution

GoldenMatch now scales to 10M+ records with an optional Ray backend.

pip install goldenmatch[ray]
goldenmatch dedupe huge_dataset.parquet --backend ray

Or in config:

backend: ray

Zero configuration needed. Ray auto-starts locally and uses all CPU cores. For datasets over 50M records, point at a Ray cluster.

Records	Backend	User Setup
< 500K	Polars (default)	Nothing
500K - 2M	DuckDB or Ray local	pip install goldenmatch[ray]
2M - 50M	Ray local	Same
50M+	Ray cluster	User provides infra

How it works

Block scoring (the bottleneck) is distributed as independent Ray tasks. Each block is scored on a separate CPU core. Results are collected and merged into the pipeline seamlessly.

For small workloads (<= 4 blocks), falls back to the existing ThreadPoolExecutor automatically.

Stats

• 909 tests passing (6 new)
• CI green on Python 3.11/3.12/3.13

pip install --upgrade goldenmatch