automating the search for the best combination of blockers and comparisons

[theimanph]

Hi,

Would it be possible to automate the search for the best combination of blockers and comparisons? One could use clustering metrics to determine the quality of the answer.. What do you think?

tom

[RobinL]

Hello. Yes - good idea. We've been thinking about this a bit and have done some work on auto blocking rules (but not auto-comparisons). It's included in the codebase, but we haven't put it in the public API yet because we're unsure exactly how to make them available to users.

The best place to look for how they work are the three PRs that added the functionality - there's a fairly decent description in the comment.
#1665
#1667
#1668

All of which relied on this one: #1388

I'm sure there's loads of room for improvement. At the moment I think they're 'good enough to be useful' but not yet good enough that we want to firmly add them to the API and document them.

[bbanzai88]

Cool!! Thank you!!

[theimanph]

Hi Robin, Have you seen this: https://github.com/databricks-industry-solutions/auto-data-linkage ? There are some interesting ideas there..