docs/guides/data_deduplication.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b81941f1",
   "metadata": {},
   "source": [
    "Data deduplication\n",
    "==================\n",
    "\n",
    "Introduction\n",
    "------------\n",
    "\n",
    "This example shows how to find records in datasets belonging to the same\n",
    "entity. In our case, we try to deduplicate a dataset with records of\n",
    "persons. We will try to link within the dataset based on attributes like\n",
    "first name, surname, sex, date of birth, place and address. The data\n",
    "used in this example is part of\n",
    "[Febrl](https://sourceforge.net/projects/febrl/) and is fictitious.\n",
    "\n",
    "First, start with importing the ``recordlinkage`` module. The submodule\n",
    "``recordlinkage.datasets`` contains several datasets that can be used\n",
    "for testing. For this example, we use the Febrl dataset 1. This dataset\n",
    "contains 1000 records of which 500 original and 500 duplicates, with\n",
    "exactly one duplicate per original record. This dataset can be loaded\n",
    "with the function ``load_febrl1``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d867f434",
   "metadata": {},
   "outputs": [],
   "source": [
    "import recordlinkage\n",
    "from recordlinkage.datasets import load_febrl1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f638bf",
   "metadata": {},
   "source": [
    "The dataset is loaded with the following code. The returned datasets are\n",
    "of type ``pandas.DataFrame``. This makes it easy to manipulate the data\n",
    "if desired. For details about data manipulation with ``pandas``, see\n",
    "their comprehensive documentation http://pandas.pydata.org/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e26f41fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>given_name</th>\n",
       "      <th>surname</th>\n",
       "      <th>street_number</th>\n",
       "      <th>address_1</th>\n",
       "      <th>address_2</th>\n",
       "      <th>suburb</th>\n",
       "      <th>postcode</th>\n",
       "      <th>state</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>soc_sec_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>rec-223-org</th>\n",
       "      <td>NaN</td>\n",
       "      <td>waller</td>\n",
       "      <td>6</td>\n",
       "      <td>tullaroop street</td>\n",
       "      <td>willaroo</td>\n",
       "      <td>st james</td>\n",
       "      <td>4011</td>\n",
       "      <td>wa</td>\n",
       "      <td>19081209</td>\n",
       "      <td>6988048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-122-org</th>\n",
       "      <td>lachlan</td>\n",
       "      <td>berry</td>\n",
       "      <td>69</td>\n",
       "      <td>giblin street</td>\n",
       "      <td>killarney</td>\n",
       "      <td>bittern</td>\n",
       "      <td>4814</td>\n",
       "      <td>qld</td>\n",
       "      <td>19990219</td>\n",
       "      <td>7364009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-373-org</th>\n",
       "      <td>deakin</td>\n",
       "      <td>sondergeld</td>\n",
       "      <td>48</td>\n",
       "      <td>goldfinch circuit</td>\n",
       "      <td>kooltuo</td>\n",
       "      <td>canterbury</td>\n",
       "      <td>2776</td>\n",
       "      <td>vic</td>\n",
       "      <td>19600210</td>\n",
       "      <td>2635962</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-10-dup-0</th>\n",
       "      <td>kayla</td>\n",
       "      <td>harrington</td>\n",
       "      <td>NaN</td>\n",
       "      <td>maltby circuit</td>\n",
       "      <td>coaling</td>\n",
       "      <td>coolaroo</td>\n",
       "      <td>3465</td>\n",
       "      <td>nsw</td>\n",
       "      <td>19150612</td>\n",
       "      <td>9004242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-227-org</th>\n",
       "      <td>luke</td>\n",
       "      <td>purdon</td>\n",
       "      <td>23</td>\n",
       "      <td>ramsay place</td>\n",
       "      <td>mirani</td>\n",
       "      <td>garbutt</td>\n",
       "      <td>2260</td>\n",
       "      <td>vic</td>\n",
       "      <td>19831024</td>\n",
       "      <td>8099933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-188-dup-0</th>\n",
       "      <td>stephanie</td>\n",
       "      <td>geu</td>\n",
       "      <td>28</td>\n",
       "      <td>bainton crescent</td>\n",
       "      <td>masonic memorial village</td>\n",
       "      <td>maryborough</td>\n",
       "      <td>2541</td>\n",
       "      <td>sa</td>\n",
       "      <td>19421008</td>\n",
       "      <td>3997529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-334-dup-0</th>\n",
       "      <td>nicholas</td>\n",
       "      <td>NaN</td>\n",
       "      <td>289</td>\n",
       "      <td>britten-jonues drive</td>\n",
       "      <td>jabaru court</td>\n",
       "      <td>paddington</td>\n",
       "      <td>2000</td>\n",
       "      <td>vic</td>\n",
       "      <td>19970422</td>\n",
       "      <td>5062738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-469-dup-0</th>\n",
       "      <td>lachlan</td>\n",
       "      <td>katsiavos</td>\n",
       "      <td>29</td>\n",
       "      <td>paul coe cdrescent</td>\n",
       "      <td>NaN</td>\n",
       "      <td>casual</td>\n",
       "      <td>2913</td>\n",
       "      <td>nsw</td>\n",
       "      <td>19380406</td>\n",
       "      <td>4112327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-350-dup-0</th>\n",
       "      <td>monique</td>\n",
       "      <td>gergely</td>\n",
       "      <td>21</td>\n",
       "      <td>harwoos court</td>\n",
       "      <td>hyberni a park</td>\n",
       "      <td>sherwood</td>\n",
       "      <td>2207</td>\n",
       "      <td>nsw</td>\n",
       "      <td>19790807</td>\n",
       "      <td>7375144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-212-org</th>\n",
       "      <td>NaN</td>\n",
       "      <td>mcveigh</td>\n",
       "      <td>45</td>\n",
       "      <td>bougainville street</td>\n",
       "      <td>kimberley</td>\n",
       "      <td>ourimbah</td>\n",
       "      <td>6060</td>\n",
       "      <td>wa</td>\n",
       "      <td>19360219</td>\n",
       "      <td>8243761</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              given_name     surname street_number             address_1  \\\n",
       "rec_id                                                                     \n",
       "rec-223-org          NaN      waller             6      tullaroop street   \n",
       "rec-122-org      lachlan       berry            69         giblin street   \n",
       "rec-373-org       deakin  sondergeld            48     goldfinch circuit   \n",
       "rec-10-dup-0       kayla  harrington           NaN        maltby circuit   \n",
       "rec-227-org         luke      purdon            23          ramsay place   \n",
       "...                  ...         ...           ...                   ...   \n",
       "rec-188-dup-0  stephanie         geu            28      bainton crescent   \n",
       "rec-334-dup-0   nicholas         NaN           289  britten-jonues drive   \n",
       "rec-469-dup-0    lachlan   katsiavos            29    paul coe cdrescent   \n",
       "rec-350-dup-0    monique     gergely            21         harwoos court   \n",
       "rec-212-org          NaN     mcveigh            45   bougainville street   \n",
       "\n",
       "                              address_2       suburb postcode state  \\\n",
       "rec_id                                                                \n",
       "rec-223-org                    willaroo     st james     4011    wa   \n",
       "rec-122-org                   killarney      bittern     4814   qld   \n",
       "rec-373-org                     kooltuo   canterbury     2776   vic   \n",
       "rec-10-dup-0                    coaling     coolaroo     3465   nsw   \n",
       "rec-227-org                      mirani      garbutt     2260   vic   \n",
       "...                                 ...          ...      ...   ...   \n",
       "rec-188-dup-0  masonic memorial village  maryborough     2541    sa   \n",
       "rec-334-dup-0              jabaru court   paddington     2000   vic   \n",
       "rec-469-dup-0                       NaN       casual     2913   nsw   \n",
       "rec-350-dup-0            hyberni a park     sherwood     2207   nsw   \n",
       "rec-212-org                   kimberley     ourimbah     6060    wa   \n",
       "\n",
       "              date_of_birth soc_sec_id  \n",
       "rec_id                                  \n",
       "rec-223-org        19081209    6988048  \n",
       "rec-122-org        19990219    7364009  \n",
       "rec-373-org        19600210    2635962  \n",
       "rec-10-dup-0       19150612    9004242  \n",
       "rec-227-org        19831024    8099933  \n",
       "...                     ...        ...  \n",
       "rec-188-dup-0      19421008    3997529  \n",
       "rec-334-dup-0      19970422    5062738  \n",
       "rec-469-dup-0      19380406    4112327  \n",
       "rec-350-dup-0      19790807    7375144  \n",
       "rec-212-org        19360219    8243761  \n",
       "\n",
       "[1000 rows x 10 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfA = load_febrl1()\n",
    "dfA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d7be0b0",
   "metadata": {},
   "source": [
    "Make record pairs\n",
    "-----------------\n",
    "\n",
    "It is very intuitive to start with comparing each record in DataFrame\n",
    "``dfA`` with all other records in DataFrame ``dfA``. In fact, we want to\n",
    "make record pairs. Each record pair should contain two different records\n",
    "of DataFrame ``dfA``. This process of making record pairs is also called\n",
    "\"indexing\". With the ``recordlinkage`` module, indexing is easy. First,\n",
    "load the ``recordlinkage.Index`` class and call the ``.full`` method.\n",
    "This object generates a full index on a ``.index(...)`` call. In case of\n",
    "deduplication of a single dataframe, one dataframe is sufficient as\n",
    "input argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "430c201e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:recordlinkage:indexing - performance warning - A full index can result in large number of record pairs.\n"
     ]
    }
   ],
   "source": [
    "indexer = recordlinkage.Index()\n",
    "indexer.full()\n",
    "candidate_links = indexer.index(dfA)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e1da188",
   "metadata": {},
   "source": [
    "With the method ``index``, all possible (and unique) record pairs are\n",
    "made. The method returns a ``pandas.MultiIndex``. The number of pairs is\n",
    "equal to the number of records in ``dfA`` choose ``2``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ed98b94c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 499500\n"
     ]
    }
   ],
   "source": [
    "print(len(dfA), len(candidate_links))\n",
    "# (1000*1000-1000)/2 = 499500"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdd38fac",
   "metadata": {},
   "source": [
    "Many of these record pairs do not belong to the same person. The\n",
    "``recordlinkage`` toolkit has some more advanced indexing methods to\n",
    "reduce the number of record pairs. Obvious non-matches are left out of\n",
    "the index. Note that if a matching record pair is not included in the\n",
    "index, it can not be matched anymore.\n",
    "\n",
    "One of the most well known indexing methods is named *blocking*. This\n",
    "method includes only record pairs that are identical on one or more\n",
    "stored attributes of the person (or entity in general). The blocking\n",
    "method can be used in the ``recordlinkage`` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6b6bd8ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2082"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexer = recordlinkage.Index()\n",
    "indexer.block(\"given_name\")\n",
    "candidate_links = indexer.index(dfA)\n",
    "len(candidate_links)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f2cadc9",
   "metadata": {},
   "source": [
    "The argument \"given\\_name\" is the blocking variable. This variable has\n",
    "to be the name of a column in ``dfA``. It is possible to\n",
    "parse a list of columns names to block on multiple variables. Blocking\n",
    "on multiple variables will reduce the number of record pairs even\n",
    "further.\n",
    "\n",
    "Another implemented indexing method is *Sorted Neighbourhood Indexing*\n",
    "(``recordlinkage.index.sortedneighbourhood``). This method is very\n",
    "useful when there are many misspellings in the string were used for\n",
    "indexing. In fact, sorted neighbourhood indexing is a generalisation of\n",
    "blocking. See the documentation for details about sorted neighbourd\n",
    "indexing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f8beda5",
   "metadata": {},
   "source": [
    "Compare records\n",
    "---------------\n",
    "\n",
    "Each record pair is a candidate match. To classify the candidate record\n",
    "pairs into matches and non-matches, compare the records on all\n",
    "attributes both records have in common. The ``recordlinkage`` module has\n",
    "a class named ``Compare``. This class is used to compare the records.\n",
    "The following code shows how to compare attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "205405a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "compare_cl = recordlinkage.Compare()\n",
    "compare_cl.exact(\"given_name\", \"given_name\", label=\"given_name\")\n",
    "compare_cl.string(\n",
    "    \"surname\", \"surname\", method=\"jarowinkler\", threshold=0.85, label=\"surname\"\n",
    ")\n",
    "compare_cl.exact(\"date_of_birth\", \"date_of_birth\", label=\"date_of_birth\")\n",
    "compare_cl.exact(\"suburb\", \"suburb\", label=\"suburb\")\n",
    "compare_cl.exact(\"state\", \"state\", label=\"state\")\n",
    "compare_cl.string(\"address_1\", \"address_1\", threshold=0.85, label=\"address_1\")\n",
    "features = compare_cl.compute(candidate_links, dfA)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1e32c3a",
   "metadata": {},
   "source": [
    "The comparing of record pairs starts when the ``compute`` method is\n",
    "called. All attribute comparisons are stored in a DataFrame with\n",
    "horizontally the features and vertically the record pairs. The first 10\n",
    "comparison vectors are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d4c10fef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>given_name</th>\n",
       "      <th>surname</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>suburb</th>\n",
       "      <th>state</th>\n",
       "      <th>address_1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec_id_1</th>\n",
       "      <th>rec_id_2</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>rec-183-dup-0</th>\n",
       "      <th>rec-122-org</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">rec-248-org</th>\n",
       "      <th>rec-122-org</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-183-dup-0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">rec-122-dup-0</th>\n",
       "      <th>rec-122-org</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-183-dup-0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-248-org</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">rec-469-org</th>\n",
       "      <th>rec-122-org</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-183-dup-0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-248-org</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-122-dup-0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             given_name  surname  date_of_birth  suburb  \\\n",
       "rec_id_1      rec_id_2                                                    \n",
       "rec-183-dup-0 rec-122-org             1      0.0              0       0   \n",
       "rec-248-org   rec-122-org             1      0.0              0       0   \n",
       "              rec-183-dup-0           1      0.0              0       0   \n",
       "rec-122-dup-0 rec-122-org             1      1.0              1       1   \n",
       "              rec-183-dup-0           1      0.0              0       0   \n",
       "              rec-248-org             1      0.0              0       0   \n",
       "rec-469-org   rec-122-org             1      0.0              0       0   \n",
       "              rec-183-dup-0           1      0.0              0       0   \n",
       "              rec-248-org             1      0.0              0       0   \n",
       "              rec-122-dup-0           1      0.0              0       0   \n",
       "\n",
       "                             state  address_1  \n",
       "rec_id_1      rec_id_2                         \n",
       "rec-183-dup-0 rec-122-org        0        0.0  \n",
       "rec-248-org   rec-122-org        1        0.0  \n",
       "              rec-183-dup-0      0        0.0  \n",
       "rec-122-dup-0 rec-122-org        1        1.0  \n",
       "              rec-183-dup-0      0        0.0  \n",
       "              rec-248-org        1        0.0  \n",
       "rec-469-org   rec-122-org        0        0.0  \n",
       "              rec-183-dup-0      1        0.0  \n",
       "              rec-248-org        0        0.0  \n",
       "              rec-122-dup-0      0        0.0  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d0cdce02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>given_name</th>\n",
       "      <th>surname</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>suburb</th>\n",
       "      <th>state</th>\n",
       "      <th>address_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2082.0</td>\n",
       "      <td>2082.000000</td>\n",
       "      <td>2082.000000</td>\n",
       "      <td>2082.000000</td>\n",
       "      <td>2082.000000</td>\n",
       "      <td>2082.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.144092</td>\n",
       "      <td>0.139289</td>\n",
       "      <td>0.108549</td>\n",
       "      <td>0.327089</td>\n",
       "      <td>0.133045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.351268</td>\n",
       "      <td>0.346331</td>\n",
       "      <td>0.311148</td>\n",
       "      <td>0.469263</td>\n",
       "      <td>0.339705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       given_name      surname  date_of_birth       suburb        state  \\\n",
       "count      2082.0  2082.000000    2082.000000  2082.000000  2082.000000   \n",
       "mean          1.0     0.144092       0.139289     0.108549     0.327089   \n",
       "std           0.0     0.351268       0.346331     0.311148     0.469263   \n",
       "min           1.0     0.000000       0.000000     0.000000     0.000000   \n",
       "25%           1.0     0.000000       0.000000     0.000000     0.000000   \n",
       "50%           1.0     0.000000       0.000000     0.000000     0.000000   \n",
       "75%           1.0     0.000000       0.000000     0.000000     1.000000   \n",
       "max           1.0     1.000000       1.000000     1.000000     1.000000   \n",
       "\n",
       "         address_1  \n",
       "count  2082.000000  \n",
       "mean      0.133045  \n",
       "std       0.339705  \n",
       "min       0.000000  \n",
       "25%       0.000000  \n",
       "50%       0.000000  \n",
       "75%       0.000000  \n",
       "max       1.000000  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a95f7c",
   "metadata": {},
   "source": [
    "The last step is to decide which records belong to the same person. In\n",
    "this example, we keep it simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "072e23b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.0     142\n",
       "5.0     145\n",
       "4.0      30\n",
       "3.0       9\n",
       "2.0     376\n",
       "1.0    1380\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.sum(axis=1).value_counts().sort_index(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a90814ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>given_name</th>\n",
       "      <th>surname</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>suburb</th>\n",
       "      <th>state</th>\n",
       "      <th>address_1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec_id_1</th>\n",
       "      <th>rec_id_2</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>rec-122-dup-0</th>\n",
       "      <th>rec-122-org</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-183-org</th>\n",
       "      <th>rec-183-dup-0</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-248-dup-0</th>\n",
       "      <th>rec-248-org</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-373-dup-0</th>\n",
       "      <th>rec-373-org</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-10-org</th>\n",
       "      <th>rec-10-dup-0</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-184-dup-0</th>\n",
       "      <th>rec-184-org</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-252-org</th>\n",
       "      <th>rec-252-dup-0</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-48-dup-0</th>\n",
       "      <th>rec-48-org</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-298-dup-0</th>\n",
       "      <th>rec-298-org</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-282-org</th>\n",
       "      <th>rec-282-dup-0</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>317 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                             given_name  surname  date_of_birth  suburb  \\\n",
       "rec_id_1      rec_id_2                                                    \n",
       "rec-122-dup-0 rec-122-org             1      1.0              1       1   \n",
       "rec-183-org   rec-183-dup-0           1      1.0              1       1   \n",
       "rec-248-dup-0 rec-248-org             1      1.0              1       1   \n",
       "rec-373-dup-0 rec-373-org             1      1.0              1       1   \n",
       "rec-10-org    rec-10-dup-0            1      1.0              1       1   \n",
       "...                                 ...      ...            ...     ...   \n",
       "rec-184-dup-0 rec-184-org             1      1.0              1       0   \n",
       "rec-252-org   rec-252-dup-0           1      1.0              1       1   \n",
       "rec-48-dup-0  rec-48-org              1      1.0              1       1   \n",
       "rec-298-dup-0 rec-298-org             1      1.0              1       1   \n",
       "rec-282-org   rec-282-dup-0           1      1.0              1       1   \n",
       "\n",
       "                             state  address_1  \n",
       "rec_id_1      rec_id_2                         \n",
       "rec-122-dup-0 rec-122-org        1        1.0  \n",
       "rec-183-org   rec-183-dup-0      1        1.0  \n",
       "rec-248-dup-0 rec-248-org        1        1.0  \n",
       "rec-373-dup-0 rec-373-org        1        1.0  \n",
       "rec-10-org    rec-10-dup-0       1        1.0  \n",
       "...                            ...        ...  \n",
       "rec-184-dup-0 rec-184-org        1        1.0  \n",
       "rec-252-org   rec-252-dup-0      1        1.0  \n",
       "rec-48-dup-0  rec-48-org         1        1.0  \n",
       "rec-298-dup-0 rec-298-org        1        0.0  \n",
       "rec-282-org   rec-282-dup-0      1        0.0  \n",
       "\n",
       "[317 rows x 6 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matches = features[features.sum(axis=1) > 3]\n",
    "matches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07ef07fb",
   "metadata": {},
   "source": [
    "Full code\n",
    "---------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "24782f23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "317\n"
     ]
    }
   ],
   "source": [
    "import recordlinkage\n",
    "from recordlinkage.datasets import load_febrl1\n",
    "\n",
    "dfA = load_febrl1()\n",
    "\n",
    "# Indexation step\n",
    "indexer = recordlinkage.Index()\n",
    "indexer.block(left_on=\"given_name\")\n",
    "candidate_links = indexer.index(dfA)\n",
    "\n",
    "# Comparison step\n",
    "compare_cl = recordlinkage.Compare()\n",
    "\n",
    "compare_cl.exact(\"given_name\", \"given_name\", label=\"given_name\")\n",
    "compare_cl.string(\n",
    "    \"surname\", \"surname\", method=\"jarowinkler\", threshold=0.85, label=\"surname\"\n",
    ")\n",
    "compare_cl.exact(\"date_of_birth\", \"date_of_birth\", label=\"date_of_birth\")\n",
    "compare_cl.exact(\"suburb\", \"suburb\", label=\"suburb\")\n",
    "compare_cl.exact(\"state\", \"state\", label=\"state\")\n",
    "compare_cl.string(\"address_1\", \"address_1\", threshold=0.85, label=\"address_1\")\n",
    "\n",
    "features = compare_cl.compute(candidate_links, dfA)\n",
    "\n",
    "# Classification step\n",
    "matches = features[features.sum(axis=1) > 3]\n",
    "print(len(matches))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}