Data cleansing problem statement: Data in a record are often duplicated. How do we find the duplicate probability ? [Work In Progress]
-
Updated
Nov 20, 2018 - Scala
Entity resolution (also known as data matching, data linkage, record linkage, and many other terms) is the task of finding entities in a dataset that refer to the same entity across different data sources (e.g., data files, books, websites, and databases). Entity resolution is necessary when joining different data sets based on entities that may or may not share a common identifier (e.g., database key, URI, National identification number), which may be due to differences in record shape, storage location, or curator style or preference.
Data cleansing problem statement: Data in a record are often duplicated. How do we find the duplicate probability ? [Work In Progress]
Examples of spark-lucenerdd
Distributed Bayesian Entity Resolution in Apache Spark
Spark RDD with Lucene's query and entity linkage capabilities
Created by Halbert L. Dunn
Released 1946