v1.4.3 — auto-config regression fixes
Bug-fix release: three v1.4 auto-config regressions
Fixes three independent bugs observed when calling dedupe_df(df) on medium-sized person datasets (5K–50K rows). See #40 for the full diagnosis and before/after numbers.
Fixes
-
Learned blocking auto-upgrade — previously gated at
>= 5000rows withlearned_sample_size = 5000, so on a 5K dataset the learner trained its predicates on 100% of its own input, producing overfit predicates and multi-minute runtimes. Gate raised to>= 50_000; sample size capped atmin(total_rows // 4, 5000)so the learner always has held-out rows. -
Geo/zip/low-cardinality promoted to exact matchkeys —
_SCORER_MAProutedcol_typezipandgeotoscorer="exact", causing the auto-config emitter to build exact matchkeys that asserted "two records sharing a city/zip are the same entity." This collapsed every record per city into one mega-cluster. Also caught thebirth_yearmisclassified-as-phone failure mode triggered by upstream date transforms. Fix: excludecol_type in {"zip", "geo"}from exact matchkeys unconditionally (they remain blocking candidates), and raise the cardinality guard from< 0.01to< 0.5. -
DedupeResult.total_recordsdouble-counted the golden rollup —_extract_statssummedgolden + dupes + unique, butgoldenis a per-cluster rollup, not an independent row population. Whenever duplicates existed,total_records > df.heightby exactly the number of multi-member clusters. Fix:total_records = dupes + unique.
Impact on NCVR 5K repro
| pre-fix | post-fix | |
|---|---|---|
| elapsed | >15 min | ~15 s |
total_records |
5085 | 5000 |
| clusters (size ≥ 2) | 18 | 61 |
| biggest cluster | 100 | 3 |
Other changes
- Aggregate "all exact-eligible columns excluded" warning now enumerates skipped columns and their reasons.
_extract_statsnow warns loudly on contract-violating shapes (golden present, dupes/unique absent) instead of silently returning zeroed stats.- New
tests/test_autoconfig_regressions.pywith 15 tests covering all three bugs, their boundaries (50K and 0.5 exact), positive cases (high-cardinality email/phone still promoted), and an end-to-end interaction test. - Full test suite: 1319 passed (up from 1310).
Upgrade
pip install --upgrade goldenmatchNo API changes — drop-in upgrade from 1.4.2.