Skip to content

v1.2.7

Choose a tag to compare

View all tags
@benzsevern benzsevern released this 03 Apr 15:32
v1.2.7
fc11707

Fix: Auto-config cardinality guards

Auto-config (dedupe_df(df) with no config) failed on datasets with near-unique ID columns, low-cardinality categorical columns, or long-text description fields. This release fixes all three root causes.

Bug fixes

  • Blocking on unique IDs — Columns with cardinality_ratio >= 0.95 (e.g., rec_id, id) are now excluded from blocking key selection. Previously, these produced single-row blocks with zero comparisons.
  • Exact matchkeys on low-cardinality columns — Columns with cardinality_ratio < 0.01 (e.g., state, county) no longer get exact matchkeys. Previously, these caused quadratic pair explosions (50M+ pairs → MemoryError).
  • Description columns routed to fuzzy matching — Long-text columns (avg_len > 50) now get a token_sort fuzzy scorer in addition to record_embedding, ensuring they contribute to matching even without a sentence-transformer model.
  • Safe defaults — Guards only fire when cardinality_ratio was actually measured (> 0), so manually-constructed ColumnProfile objects are unaffected.
  • Config-altering decisions now log warnings instead of info-level messages.

Testing

  • 64 autoconfig tests (was 49), including boundary value tests and integration tests on real Febrl and DBLP-ACM benchmark datasets.
  • Full suite: 1244 tests passing.
Assets 2
Loading