Skip to content

v1.2.4

Choose a tag to compare

View all tags
@benzsevern benzsevern released this 31 Mar 18:51
v1.2.4
125cf25

New Feature: Compound Blocking Keys

Auto-configure now generates compound blocking keys when single-column blocking produces oversized blocks. This prevents OOM on datasets like the Kaggle Blue Book for Bulldozers (401K rows) where fiBaseModel="580" has 19,798 records.

How it works

  1. LLM-assisted (optional): sends column profiles with cardinality stats to LLM for compound key suggestions, validates against actual block sizes
  2. Greedy fallback: pairs the highest-cardinality column with the column that reduces max block most
  3. Multi-pass: generates 2-3 compound key passes for recall coverage

Example

# Before: OOM on fiBaseModel (19,798 records per block)
# After: compound [fiModelDesc, state] (max 1,198 per block)
result = goldenmatch.dedupe_df(df)  # just works

Guard ensures compound blocking only activates when ALL single-column candidates are oversized — existing datasets are unaffected.

Other improvements

  • All blocking fallbacks now set skip_oversized=True as safety net
  • _detect_llm_provider() threads LLM provider from dedupe_df(llm_scorer=True) through auto-configure
  • 1,190 tests passing
Assets 2
Loading