STA 790 Entity Resolution - Mini Project

• Qinzhi Peng
• Xinyan (Hathaway) Liu
• Yujie(Johnny) Ye
• Zhankai Ye

Google Slides

Introduction

This vignette demonstrates the use of the RecordLinkage package in R for performing entity resolution tasks. It focuses on two key functionalities: weight-based deduplication and supervised classification. These techniques are commonly used in record linkage to identify and remove duplicate records from datasets and to classify matches and non-matches based on various supervised machine learning models.

The vignette covers the steps from generating record pairs to performing deduplication using both the Expectation-Maximization (EM) algorithm and various supervised learning methods. Additionally, it explores the performance metrics associated with classification, such as alpha and beta error rates, accuracy, precision, and recall.

Outline

Deduplication & Weigted Deduplication

• Comparison Pattern
• Comparison Functions
• Blocking
• Stochastic Record Linkage and Weight Calculation

Supervised ML for ER

• Use case of supervised methods for entity resolution
• Generate paris of record
• Model training & Classification of unlabled records
• Interpretation