Skip to content

v1.4.3 — auto-config regression fixes

Choose a tag to compare

@benzsevern benzsevern released this 08 Apr 00:44

Bug-fix release: three v1.4 auto-config regressions

Fixes three independent bugs observed when calling dedupe_df(df) on medium-sized person datasets (5K–50K rows). See #40 for the full diagnosis and before/after numbers.

Fixes

  1. Learned blocking auto-upgrade — previously gated at >= 5000 rows with learned_sample_size = 5000, so on a 5K dataset the learner trained its predicates on 100% of its own input, producing overfit predicates and multi-minute runtimes. Gate raised to >= 50_000; sample size capped at min(total_rows // 4, 5000) so the learner always has held-out rows.

  2. Geo/zip/low-cardinality promoted to exact matchkeys_SCORER_MAP routed col_type zip and geo to scorer="exact", causing the auto-config emitter to build exact matchkeys that asserted "two records sharing a city/zip are the same entity." This collapsed every record per city into one mega-cluster. Also caught the birth_year misclassified-as-phone failure mode triggered by upstream date transforms. Fix: exclude col_type in {"zip", "geo"} from exact matchkeys unconditionally (they remain blocking candidates), and raise the cardinality guard from < 0.01 to < 0.5.

  3. DedupeResult.total_records double-counted the golden rollup_extract_stats summed golden + dupes + unique, but golden is a per-cluster rollup, not an independent row population. Whenever duplicates existed, total_records > df.height by exactly the number of multi-member clusters. Fix: total_records = dupes + unique.

Impact on NCVR 5K repro

pre-fix post-fix
elapsed >15 min ~15 s
total_records 5085 5000
clusters (size ≥ 2) 18 61
biggest cluster 100 3

Other changes

  • Aggregate "all exact-eligible columns excluded" warning now enumerates skipped columns and their reasons.
  • _extract_stats now warns loudly on contract-violating shapes (golden present, dupes/unique absent) instead of silently returning zeroed stats.
  • New tests/test_autoconfig_regressions.py with 15 tests covering all three bugs, their boundaries (50K and 0.5 exact), positive cases (high-cardinality email/phone still promoted), and an end-to-end interaction test.
  • Full test suite: 1319 passed (up from 1310).

Upgrade

pip install --upgrade goldenmatch

No API changes — drop-in upgrade from 1.4.2.