ROADMAP

[NickCrews]

Features

• better connected_components() API #5
• EM algorithm of FS model for unlabeled data #9
• API like dedupe's Variable for doing blocking and comparison based on one config
• save/load FS weights
• levenshtein(), etc implementations for Ibis
• TF adjustments
• HDBSCAN for clustering

Comparisons

• Name
• Location
  • Address
  • lat long
• String
• email
• date
• number

Eval

• features: cardinality, median collisions?
• blocking: reduction ratio, TP rate, FP rate, etc
• comparisons: f1 score, FP, TP, etc
• clustering: sillouette, rand index, adjusted rand index, Mutual Information, etc. reference #6

Plot

• features viz?? looking at the record tables pre-comparison. Might be out of scope.
• blocking rules viz (a la splink) : viz: plot blocking with https://upset.app/ #7
• weights viz showing m, u, odds of weights
• comparisons viz (a la splink, but not limited to the FS model)
• scored viz (showing weights applied to real data)
• clustering viz (a la splink, although ideally with less javascript and)
• m u weights and match weights chart (a la splink)

Datasets

• deduplication task with labels (patents)
• linkage task with labels

Testing

• 50% coverage
• 70% coverage
• 90% coverage

Docs

How-to guides

• strings
• names
• locations
• collections
• dates
• emails
• improving cleaning
• improving blocking
• improving comparing
• improve clustering