quickstart_demo_linking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splink data linking demo (link only)\n",
    "\n",
    "In this demo we link two small datasets.  \n",
    "\n",
    "The larger table contains duplicates, but in this notebook we use the `link_only` setting, so `splink` makes no attempt to deduplicate these records.    \n",
    "\n",
    "Note it is possible to simultaneously link and dedupe using the `link_and_dedupe` setting.\n",
    "\n",
    "**Important** Where deduplication is not required, `link_only` can provide an important performance boost by dramatically reducing the number of records which need to be compared.\n",
    "\n",
    "For example, if you wanted to link 10 records to 1,000, then the maximum number of comparisons that need to be made (i.e. with no blocking rules) is 10,000.  If you need to dedupe as well, that number would be n(n-1)/2 = 509,545.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1:  Imports and setup\n",
    "\n",
    "The following is just boilerplate code that sets up the Spark session and sets some other non-essential configuration options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RendererRegistry.enable('mimetype')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd \n",
    "pd.options.display.max_columns = 500\n",
    "pd.options.display.max_rows = 100\n",
    "import altair as alt\n",
    "alt.renderers.enable('mimetype')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging \n",
    "logging.basicConfig()  # Means logs will print in Jupyter Lab\n",
    "\n",
    "# Set to DEBUG if you want splink to log the SQL statements it's executing under the hood\n",
    "logging.getLogger(\"splink\").setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "22/01/11 05:40:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "22/01/11 05:40:57 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.\n",
      "22/01/11 05:40:57 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.\n"
     ]
    }
   ],
   "source": [
    "from utility_functions.demo_utils import get_spark\n",
    "spark = get_spark()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Read in the data\n",
    "\n",
    "In this example, we link two datasets, but you can link as many as you like.\n",
    "\n",
    "⚠️ Note that `splink` makes the following assumptions about your data:\n",
    "\n",
    "-  There is a field containing a unique record identifier in each dataset.  By default, this should be called `unique_id`, but you can change this in the settings\n",
    "-  There is a field containing a dataset name in each dataset, to disambiguate the `unique_id` column if the same id values occur in more than one dataset.  By default, this column is called `source_dataset`, but you can change this in the settings.\n",
    "-  The two datasets being linked have common column names - e.g. date of birth is represented in both datasets in a field of the same name.   In many cases, this means that the user needs to rename columns prior to using `splink`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The count of rows in `df_1` is 181\n",
      "+---------+----------+-------+----------+------------+--------------------+-----+--------------+\n",
      "|unique_id|first_name|surname|       dob|        city|               email|group|source_dataset|\n",
      "+---------+----------+-------+----------+------------+--------------------+-----+--------------+\n",
      "|        0|    Julia |   null|2015-10-29|      London| hannah88@powers.com|    0|          df_1|\n",
      "|        4|      oNah| Watson|2008-03-23|      Bolton|matthew78@ballard...|    1|          df_1|\n",
      "|       13|    Molly |   Bell|2002-01-05|Peterborough|                null|    2|          df_1|\n",
      "|       15| Alexander|Amelia |1983-05-19|     Glasgow|ic-mpbell@alleale...|    3|          df_1|\n",
      "|       20|    Ol vri|ynnollC|1972-03-08|    Plymouth|derekwilliams@nor...|    4|          df_1|\n",
      "+---------+----------+-------+----------+------------+--------------------+-----+--------------+\n",
      "only showing top 5 rows\n",
      "\n",
      "The count of rows in `df_2` is 819\n",
      "+---------+----------+-------+----------+------+--------------------+-----+--------------+\n",
      "|unique_id|first_name|surname|       dob|  city|               email|group|source_dataset|\n",
      "+---------+----------+-------+----------+------+--------------------+-----+--------------+\n",
      "|        1|    Julia | Taylor|2015-07-31|London| hannah88@powers.com|    0|          df_2|\n",
      "|        2|    Julia | Taylor|2016-01-27|London| hannah88@powers.com|    0|          df_2|\n",
      "|        3|    Julia | Taylor|2015-10-29|  null|  hannah88opowersc@m|    0|          df_2|\n",
      "|        5|     Noah | Watson|2008-03-23|Bolton|matthew78@ballard...|    1|          df_2|\n",
      "|        6|    Watson|  Noah |2008-03-23|  null|matthew78@ballard...|    1|          df_2|\n",
      "+---------+----------+-------+----------+------+--------------------+-----+--------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import lit \n",
    "df_1 = spark.read.parquet(\"data/fake_df_l.parquet\")\n",
    "df_1 = df_1.withColumn(\"source_dataset\", lit(\"df_1\"))\n",
    "df_2 = spark.read.parquet(\"data/fake_df_r.parquet\")\n",
    "df_2 = df_2.withColumn(\"source_dataset\", lit(\"df_2\"))\n",
    "print(f\"The count of rows in `df_1` is {df_1.count()}\")\n",
    "df_1.show(5)\n",
    "print(f\"The count of rows in `df_2` is {df_2.count()}\")\n",
    "df_2.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3:  Configure splink using the `settings` object\n",
    "\n",
    "Most of `splink` configuration options are stored in a settings dictionary.  This dictionary allows significant customisation, and can therefore get quite complex.  \n",
    "\n",
    "💥 We provide an tool for helping to author valid settings dictionaries, which includes tooltips and autocomplete, which you can find [here](http://robinlinacre.com/splink_settings_editor/).\n",
    "\n",
    "Customisation overrides default values built into splink.  For the purposes of this demo, we will specify a simple settings dictionary, which means we will be relying on these sensible defaults.\n",
    "\n",
    "To help with authoring and validation of the settings dictionary, we have written a [json schema](https://json-schema.org/), which can be found [here](https://github.com/moj-analytical-services/splink/blob/master/splink/files/settings_jsonschema.json).  \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The comparison expression allows for the case where a first name and surname have been inverted \n",
    "sql_case_expression = \"\"\"\n",
    "CASE \n",
    "WHEN first_name_l = first_name_r AND surname_l = surname_r THEN 4 \n",
    "WHEN first_name_l = surname_r AND surname_l = first_name_r THEN 3\n",
    "WHEN first_name_l = first_name_r THEN 2\n",
    "WHEN surname_l = surname_r THEN 1\n",
    "ELSE 0 \n",
    "END\n",
    "\"\"\"\n",
    "\n",
    "settings = {\n",
    "    \"link_type\": \"link_only\", \n",
    "    \"max_iterations\": 20,\n",
    "    \"blocking_rules\": [\n",
    "    ],\n",
    "    \"comparison_columns\": [\n",
    "       {\n",
    "            \"custom_name\": \"name_inversion\",\n",
    "            \"custom_columns_used\": [\"first_name\", \"surname\"],\n",
    "            \"case_expression\": sql_case_expression,\n",
    "            \"num_levels\": 5\n",
    "        },\n",
    "        {\n",
    "            \"col_name\": \"city\",\n",
    "            \"num_levels\": 3\n",
    "        },\n",
    "        {\n",
    "            \"col_name\": \"email\",\n",
    "            \"num_levels\": 3\n",
    "        },\n",
    "        {\n",
    "            \"col_name\": \"dob\"\n",
    "        }\n",
    "    ],\n",
    "    \"additional_columns_to_retain\": [\"group\"],\n",
    "    \"em_convergence\": 0.01,\n",
    "    \"max_iterations\": 4,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In words, this setting dictionary says:\n",
    "\n",
    "- We are performing a data linking task (the other options are `dedupe_only`, or `link_and_dedupe`)\n",
    "- Since the input datasets are so small, we do not specify any blocking rules and instead generate all possible comparisons.\n",
    "- When comparing records, we will use information from the `first_name`, `surname`, `city` and `email` columns to compute a match score.\n",
    "- For the comparisons on the `first_name` and `surname` column we allow the possibility that the names have been inputted in the wrong order.  \n",
    "  - The highest level of similarity is that both `first_name` and `surname` both match.\n",
    "  - There are other levels of similarity for the names being inverted, and just first name, or just surname matching.\n",
    "- We will retain the `group` column in the results even though this is not used as part of comparisons.  This is a labelled dataset and `group` contains the true match - i.e. where group matches, the records pertain to the same person"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4:  Estimate match scores using the Expectation Maximisation algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/robinlinacre/anaconda3/lib/python3.8/site-packages/splink/default_settings.py:262: UserWarning: You have not specified any blocking rules, meaning all comparisons between the input dataset(s) will be generated and blocking will not be used.For large input datasets, this will generally be computationally intractable because it will generate comparisons equal to the number of rows squared.\n",
      "  warnings.warn(\n",
      "/Users/robinlinacre/anaconda3/lib/python3.8/site-packages/splink/default_settings.py:185: UserWarning: No -1 level found in case statement. You usually want to use -1 as the level for the null value. e.g. WHEN col_l is null or col_r is null then -1 Case statement is:\n",
      " \n",
      "CASE \n",
      "WHEN first_name_l = first_name_r AND surname_l = surname_r THEN 4 \n",
      "WHEN first_name_l = surname_r AND surname_l = first_name_r THEN 3\n",
      "WHEN first_name_l = first_name_r THEN 2\n",
      "WHEN surname_l = surname_r THEN 1\n",
      "ELSE 0 \n",
      "END\n",
      ".\n",
      "  warnings.warn(\n",
      "INFO:splink.iterate:Iteration 0 complete                                        \n",
      "INFO:splink.model:The maximum change in parameters was 0.40458029469636825 for key name_inversion, level 4\n",
      "INFO:splink.iterate:Iteration 1 complete\n",
      "INFO:splink.model:The maximum change in parameters was 0.07434341748102571 for key email, level 1\n",
      "INFO:splink.iterate:Iteration 2 complete\n",
      "INFO:splink.model:The maximum change in parameters was 0.025150011310513642 for key dob, level 1\n",
      "INFO:splink.iterate:Iteration 3 complete\n",
      "INFO:splink.model:The maximum change in parameters was 0.009595088165527288 for key name_inversion, level 0\n",
      "INFO:splink.iterate:EM algorithm has converged\n"
     ]
    }
   ],
   "source": [
    "from splink import Splink\n",
    "\n",
    "linker = Splink(settings, [df_1, df_2], spark)\n",
    "df_e = linker.get_scored_comparisons()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Inspect results \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>match_weight</th>\n",
       "      <th>match_probability</th>\n",
       "      <th>source_dataset_l</th>\n",
       "      <th>unique_id_l</th>\n",
       "      <th>source_dataset_r</th>\n",
       "      <th>unique_id_r</th>\n",
       "      <th>surname_l</th>\n",
       "      <th>surname_r</th>\n",
       "      <th>first_name_l</th>\n",
       "      <th>first_name_r</th>\n",
       "      <th>gamma_name_inversion</th>\n",
       "      <th>city_l</th>\n",
       "      <th>city_r</th>\n",
       "      <th>gamma_city</th>\n",
       "      <th>email_l</th>\n",
       "      <th>email_r</th>\n",
       "      <th>gamma_email</th>\n",
       "      <th>dob_l</th>\n",
       "      <th>dob_r</th>\n",
       "      <th>gamma_dob</th>\n",
       "      <th>group_l</th>\n",
       "      <th>group_r</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>93101</th>\n",
       "      <td>37.816346</td>\n",
       "      <td>1.0</td>\n",
       "      <td>df_1</td>\n",
       "      <td>664</td>\n",
       "      <td>df_2</td>\n",
       "      <td>668</td>\n",
       "      <td>Taylor</td>\n",
       "      <td>Ivy</td>\n",
       "      <td>Ivy</td>\n",
       "      <td>Taylor</td>\n",
       "      <td>3</td>\n",
       "      <td>Lonon</td>\n",
       "      <td>London</td>\n",
       "      <td>1</td>\n",
       "      <td>jonesjennmfer@pitt.coi</td>\n",
       "      <td>jonesjennifer@pitts.com</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-13</td>\n",
       "      <td>1980-01-13</td>\n",
       "      <td>1</td>\n",
       "      <td>113</td>\n",
       "      <td>113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93106</th>\n",
       "      <td>37.816346</td>\n",
       "      <td>1.0</td>\n",
       "      <td>df_1</td>\n",
       "      <td>664</td>\n",
       "      <td>df_2</td>\n",
       "      <td>673</td>\n",
       "      <td>Taylor</td>\n",
       "      <td>Ivy</td>\n",
       "      <td>Ivy</td>\n",
       "      <td>Taylor</td>\n",
       "      <td>3</td>\n",
       "      <td>Lonon</td>\n",
       "      <td>London</td>\n",
       "      <td>1</td>\n",
       "      <td>jonesjennmfer@pitt.coi</td>\n",
       "      <td>jonesjennifer@pitts.com</td>\n",
       "      <td>1</td>\n",
       "      <td>1980-01-13</td>\n",
       "      <td>1980-01-13</td>\n",
       "      <td>1</td>\n",
       "      <td>113</td>\n",
       "      <td>113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79930</th>\n",
       "      <td>37.816346</td>\n",
       "      <td>1.0</td>\n",
       "      <td>df_1</td>\n",
       "      <td>581</td>\n",
       "      <td>df_2</td>\n",
       "      <td>585</td>\n",
       "      <td>Shaw</td>\n",
       "      <td>Eleanor</td>\n",
       "      <td>Eleanor</td>\n",
       "      <td>Shaw</td>\n",
       "      <td>3</td>\n",
       "      <td>Birmingham</td>\n",
       "      <td>Birmingha</td>\n",
       "      <td>1</td>\n",
       "      <td>stephaniewebbhart.net</td>\n",
       "      <td>stephaniewebb@hart.net</td>\n",
       "      <td>1</td>\n",
       "      <td>1979-03-31</td>\n",
       "      <td>1979-03-31</td>\n",
       "      <td>1</td>\n",
       "      <td>97</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73327</th>\n",
       "      <td>37.373057</td>\n",
       "      <td>1.0</td>\n",
       "      <td>df_1</td>\n",
       "      <td>517</td>\n",
       "      <td>df_2</td>\n",
       "      <td>526</td>\n",
       "      <td>Martha</td>\n",
       "      <td>Brown</td>\n",
       "      <td>Brown</td>\n",
       "      <td>Martha</td>\n",
       "      <td>3</td>\n",
       "      <td>Southend-on-Sea</td>\n",
       "      <td>Southend-on-Sea</td>\n",
       "      <td>2</td>\n",
       "      <td>watsonthomas@jones-stuart.biz</td>\n",
       "      <td>watsonthomas@jones-s.urttbiz</td>\n",
       "      <td>1</td>\n",
       "      <td>2002-09-01</td>\n",
       "      <td>2002-09-01</td>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79105</th>\n",
       "      <td>37.373057</td>\n",
       "      <td>1.0</td>\n",
       "      <td>df_1</td>\n",
       "      <td>574</td>\n",
       "      <td>df_2</td>\n",
       "      <td>578</td>\n",
       "      <td>Williams</td>\n",
       "      <td>George</td>\n",
       "      <td>George</td>\n",
       "      <td>Williams</td>\n",
       "      <td>3</td>\n",
       "      <td>London</td>\n",
       "      <td>London</td>\n",
       "      <td>2</td>\n",
       "      <td>desek58gibbr.biz</td>\n",
       "      <td>derek58@gibbs.biz</td>\n",
       "      <td>1</td>\n",
       "      <td>1981-08-06</td>\n",
       "      <td>1981-08-06</td>\n",
       "      <td>1</td>\n",
       "      <td>96</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       match_weight  match_probability source_dataset_l  unique_id_l  \\\n",
       "93101     37.816346                1.0             df_1          664   \n",
       "93106     37.816346                1.0             df_1          664   \n",
       "79930     37.816346                1.0             df_1          581   \n",
       "73327     37.373057                1.0             df_1          517   \n",
       "79105     37.373057                1.0             df_1          574   \n",
       "\n",
       "      source_dataset_r  unique_id_r surname_l surname_r first_name_l  \\\n",
       "93101             df_2          668    Taylor      Ivy          Ivy    \n",
       "93106             df_2          673    Taylor      Ivy          Ivy    \n",
       "79930             df_2          585      Shaw  Eleanor      Eleanor    \n",
       "73327             df_2          526   Martha      Brown        Brown   \n",
       "79105             df_2          578  Williams   George       George    \n",
       "\n",
       "      first_name_r  gamma_name_inversion           city_l           city_r  \\\n",
       "93101       Taylor                     3            Lonon           London   \n",
       "93106       Taylor                     3            Lonon           London   \n",
       "79930         Shaw                     3       Birmingham        Birmingha   \n",
       "73327      Martha                      3  Southend-on-Sea  Southend-on-Sea   \n",
       "79105     Williams                     3           London           London   \n",
       "\n",
       "       gamma_city                        email_l  \\\n",
       "93101           1         jonesjennmfer@pitt.coi   \n",
       "93106           1         jonesjennmfer@pitt.coi   \n",
       "79930           1          stephaniewebbhart.net   \n",
       "73327           2  watsonthomas@jones-stuart.biz   \n",
       "79105           2               desek58gibbr.biz   \n",
       "\n",
       "                            email_r  gamma_email       dob_l       dob_r  \\\n",
       "93101       jonesjennifer@pitts.com            1  1980-01-13  1980-01-13   \n",
       "93106       jonesjennifer@pitts.com            1  1980-01-13  1980-01-13   \n",
       "79930        stephaniewebb@hart.net            1  1979-03-31  1979-03-31   \n",
       "73327  watsonthomas@jones-s.urttbiz            1  2002-09-01  2002-09-01   \n",
       "79105             derek58@gibbs.biz            1  1981-08-06  1981-08-06   \n",
       "\n",
       "       gamma_dob  group_l  group_r  \n",
       "93101          1      113      113  \n",
       "93106          1      113      113  \n",
       "79930          1       97       97  \n",
       "73327          1       89       89  \n",
       "79105          1       96       96  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect main dataframe that contains the match scores\n",
    "df_e.toPandas().sort_values(\"match_probability\", ascending=False).head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `params` property of the `linker` is an object that contains a lot of diagnostic information about how the match probability was computed.  The following cells demonstrate some of its functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.vegalite.v4+json": {
       "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
       "config": {
        "header": {
         "title": null
        },
        "title": {
         "anchor": "middle",
         "offset": 10
        },
        "view": {
         "continuousHeight": 300,
         "continuousWidth": 400,
         "height": 300,
         "width": 400
        }
       },
       "data": {
        "name": "data-f306f12e00d3f5f049f79d2780a7ab73"
       },
       "datasets": {
        "data-f306f12e00d3f5f049f79d2780a7ab73": [
         {
          "bayes_factor": 7866.369234599884,
          "column_name": "name_inversion",
          "comparison_vector_value": "4",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 4,
          "is_null": false,
          "label": "level_4",
          "level_name": "level_4",
          "level_proportion": 0.0009781501494208627,
          "log2_bayes_factor": 12.941482190295073,
          "m_probability": 0.1735820750664971,
          "num_levels": 5,
          "sql_expr": "WHEN first_name_l = first_name_r AND surname_l = surname_r THEN 4",
          "u_probability": 0.000022066352327196117
         },
         {
          "bayes_factor": 10816.87507346276,
          "column_name": "name_inversion",
          "comparison_vector_value": "3",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 3,
          "is_null": false,
          "label": "level_3",
          "level_name": "level_3",
          "level_proportion": 0.000802757708835054,
          "log2_bayes_factor": 13.400996153467629,
          "m_probability": 0.14333415629686924,
          "num_levels": 5,
          "sql_expr": "WHEN first_name_l = surname_r AND surname_l = first_name_r THEN 3",
          "u_probability": 0.00001325097639784281
         },
         {
          "bayes_factor": 84.21275991056915,
          "column_name": "name_inversion",
          "comparison_vector_value": "2",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 2,
          "is_null": false,
          "label": "level_2",
          "level_name": "level_2",
          "level_proportion": 0.003082859436450116,
          "log2_bayes_factor": 6.3959669417774965,
          "m_probability": 0.178015412463166,
          "num_levels": 5,
          "sql_expr": "WHEN first_name_l = first_name_r THEN 2",
          "u_probability": 0.0021138769546587928
         },
         {
          "bayes_factor": 43.14154400089717,
          "column_name": "name_inversion",
          "comparison_vector_value": "1",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 1,
          "is_null": false,
          "label": "level_1",
          "level_name": "level_1",
          "level_proportion": 0.005140347681783208,
          "log2_bayes_factor": 5.43100590519603,
          "m_probability": 0.1799810780566816,
          "num_levels": 5,
          "sql_expr": "WHEN surname_l = surname_r THEN 1",
          "u_probability": 0.004171873821969346
         },
         {
          "bayes_factor": 0.32715524872500057,
          "column_name": "name_inversion",
          "comparison_vector_value": "0",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 0,
          "is_null": false,
          "label": "level_0",
          "level_name": "level_0",
          "level_proportion": 0.9899958850235109,
          "log2_bayes_factor": -1.6119526780809075,
          "m_probability": 0.3250872781167861,
          "num_levels": 5,
          "sql_expr": "ELSE 0",
          "u_probability": 0.9936789318946468
         },
         {
          "bayes_factor": 7.5754215135378,
          "column_name": "city",
          "comparison_vector_value": "2",
          "gamma_column_name": "gamma_city",
          "gamma_index": 2,
          "is_null": false,
          "label": "level_2",
          "level_name": "level_2",
          "level_proportion": 0.09247482158475576,
          "log2_bayes_factor": 2.9213261656247806,
          "m_probability": 0.6760480820908364,
          "num_levels": 3,
          "sql_expr": "WHEN JARO_WINKLER_SIM(city_l, city_r) >= 1.0 THEN 2",
          "u_probability": 0.0892423056436783
         },
         {
          "bayes_factor": 10.300307763779484,
          "column_name": "city",
          "comparison_vector_value": "1",
          "gamma_column_name": "gamma_city",
          "gamma_index": 1,
          "is_null": false,
          "label": "level_1",
          "level_name": "level_1",
          "level_proportion": 0.02173640315205869,
          "log2_bayes_factor": 3.364615539348775,
          "m_probability": 0.2129801834705903,
          "num_levels": 3,
          "sql_expr": "WHEN JARO_WINKLER_SIM(city_l, city_r) >= 0.88 THEN 1",
          "u_probability": 0.020677069885185802
         },
         {
          "bayes_factor": 0.12467604775074131,
          "column_name": "city",
          "comparison_vector_value": "0",
          "gamma_column_name": "gamma_city",
          "gamma_index": 0,
          "is_null": false,
          "label": "level_0",
          "level_name": "level_0",
          "level_proportion": 0.8857887752631854,
          "log2_bayes_factor": -3.003743767733968,
          "m_probability": 0.11097173443857297,
          "num_levels": 3,
          "sql_expr": "ELSE 0",
          "u_probability": 0.8900806244711358
         },
         {
          "bayes_factor": 8537.588804915724,
          "column_name": "email",
          "comparison_vector_value": "2",
          "gamma_column_name": "gamma_email",
          "gamma_index": 2,
          "is_null": false,
          "label": "level_2",
          "level_name": "level_2",
          "level_proportion": 0.0036458147411618713,
          "log2_bayes_factor": 13.059612964535702,
          "m_probability": 0.6481277887828767,
          "num_levels": 3,
          "sql_expr": "WHEN JARO_WINKLER_SIM(email_l, email_r) >= 1.0 THEN 2",
          "u_probability": 0.00007591461753343069
         },
         {
          "bayes_factor": 26193.64119339461,
          "column_name": "email",
          "comparison_vector_value": "1",
          "gamma_column_name": "gamma_email",
          "gamma_index": 1,
          "is_null": false,
          "label": "level_1",
          "level_name": "level_1",
          "level_proportion": 0.0018015059731803925,
          "log2_bayes_factor": 14.676929003060637,
          "m_probability": 0.32479282624986167,
          "num_levels": 3,
          "sql_expr": "WHEN JARO_WINKLER_SIM(email_l, email_r) >= 0.88 THEN 1",
          "u_probability": 0.000012399682192018664
         },
         {
          "bayes_factor": 0.02708177667540402,
          "column_name": "email",
          "comparison_vector_value": "0",
          "gamma_column_name": "gamma_email",
          "gamma_index": 0,
          "is_null": false,
          "label": "level_0",
          "level_name": "level_0",
          "level_proportion": 0.9945526792856578,
          "log2_bayes_factor": -5.2065338010569935,
          "m_probability": 0.02707938496726161,
          "num_levels": 3,
          "sql_expr": "ELSE 0",
          "u_probability": 0.9999116857002746
         },
         {
          "bayes_factor": 14971.363071299604,
          "column_name": "dob",
          "comparison_vector_value": "1",
          "gamma_column_name": "gamma_dob",
          "gamma_index": 1,
          "is_null": false,
          "label": "level_1",
          "level_name": "level_1",
          "level_proportion": 0.00333920223422985,
          "log2_bayes_factor": 13.869917957437533,
          "m_probability": 0.5989503315946443,
          "num_levels": 2,
          "sql_expr": "WHEN dob_l = dob_r THEN 1",
          "u_probability": 0.00004000639946691586
         },
         {
          "bayes_factor": 0.4010657136005066,
          "column_name": "dob",
          "comparison_vector_value": "0",
          "gamma_column_name": "gamma_dob",
          "gamma_index": 0,
          "is_null": false,
          "label": "level_0",
          "level_name": "level_0",
          "level_proportion": 0.9966607977657703,
          "log2_bayes_factor": -1.318089456914419,
          "m_probability": 0.40104966840535583,
          "num_levels": 2,
          "sql_expr": "ELSE 0",
          "u_probability": 0.9999599936005332
         }
        ]
       },
       "hconcat": [
        {
         "encoding": {
          "color": {
           "value": "red"
          },
          "row": {
           "field": "column_name",
           "header": {
            "labelAlign": "left",
            "labelAnchor": "middle",
            "labelAngle": 0
           },
           "sort": {
            "field": "gamma_index"
           },
           "type": "nominal"
          },
          "tooltip": [
           {
            "field": "column_name",
            "type": "nominal"
           },
           {
            "field": "level_name",
            "type": "ordinal"
           },
           {
            "field": "u_probability",
            "format": ".4f",
            "type": "quantitative"
           },
           {
            "field": "bayes_factor",
            "format": ".4f",
            "type": "quantitative"
           },
           {
            "field": "level_proportion",
            "format": ".2%",
            "title": "Percentage of record comparisons in this level",
            "type": "nominal"
           },
           {
            "field": "log2_bayes_factor",
            "format": ".4f",
            "type": "quantitative"
           }
          ],
          "x": {
           "axis": {
            "title": "proportion"
           },
           "field": "u_probability",
           "type": "quantitative"
          },
          "y": {
           "axis": {
            "title": null
           },
           "field": "label",
           "sort": {
            "field": "gamma_index"
           },
           "type": "nominal"
          }
         },
         "height": 50,
         "mark": "bar",
         "resolve": {
          "scale": {
           "y": "independent"
          }
         },
         "title": {
          "fontWeight": "normal",
          "text": "Non-matches"
         },
         "transform": [
          {
           "filter": "(datum.bayes_factor != 'unnecessary filter2 due to vega lite issue 4680')"
          }
         ],
         "width": 150
        },
        {
         "encoding": {
          "color": {
           "value": "green"
          },
          "row": {
           "field": "column_name",
           "header": {
            "labels": false
           },
           "sort": {
            "field": "gamma_index"
           },
           "type": "nominal"
          },
          "tooltip": [
           {
            "field": "column_name",
            "type": "nominal"
           },
           {
            "field": "level_name",
            "type": "ordinal"
           },
           {
            "field": "m_probability",
            "format": ".4f",
            "type": "quantitative"
           },
           {
            "field": "bayes_factor",
            "format": ".4f",
            "type": "quantitative"
           },
           {
            "field": "level_proportion",
            "format": ".2%",
            "title": "Percentage of record comparisons in this level",
            "type": "nominal"
           },
           {
            "field": "log2_bayes_factor",
            "format": ".4f",
            "type": "quantitative"
           }
          ],
          "x": {
           "axis": {
            "title": "proportion"
           },
           "field": "m_probability",
           "type": "quantitative"
          },
          "y": {
           "axis": {
            "title": null
           },
           "field": "label",
           "sort": {
            "field": "gamma_index"
           },
           "type": "nominal"
          }
         },
         "height": 50,
         "mark": "bar",
         "resolve": {
          "scale": {
           "y": "independent"
          }
         },
         "title": {
          "fontWeight": "normal",
          "text": "Matches"
         },
         "transform": [
          {
           "filter": "(datum.bayes_factor != 'unnecessary filter due to vega lite issue 4680')"
          }
         ],
         "width": 150
        }
       ],
       "title": {
        "subtitle": "Estimated proportion of matches λ =  0.00551",
        "text": "Probability distributions of non-matches and matches "
       },
       "transform": []
      },
      "text/plain": [
       "<VegaLite 4 object>\n",
       "\n",
       "If you see this message, it means the renderer has not been properly enabled\n",
       "for the frontend that you are using. For more information, see\n",
       "https://altair-viz.github.io/user_guide/troubleshooting.html\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = linker.model\n",
    "model.probability_distribution_chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative representation of the parameters displays them in terms of the effect different values in the comparison vectors have on the match probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.vegalite.v4+json": {
       "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
       "config": {
        "header": {
         "title": null
        },
        "mark": {
         "tooltip": null
        },
        "title": {
         "anchor": "middle"
        },
        "view": {
         "continuousHeight": 300,
         "continuousWidth": 400,
         "height": 300,
         "width": 400
        }
       },
       "data": {
        "name": "data-f306f12e00d3f5f049f79d2780a7ab73"
       },
       "datasets": {
        "data-f306f12e00d3f5f049f79d2780a7ab73": [
         {
          "bayes_factor": 7866.369234599884,
          "column_name": "name_inversion",
          "comparison_vector_value": "4",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 4,
          "is_null": false,
          "label": "level_4",
          "level_name": "level_4",
          "level_proportion": 0.0009781501494208627,
          "log2_bayes_factor": 12.941482190295073,
          "m_probability": 0.1735820750664971,
          "num_levels": 5,
          "sql_expr": "WHEN first_name_l = first_name_r AND surname_l = surname_r THEN 4",
          "u_probability": 0.000022066352327196117
         },
         {
          "bayes_factor": 10816.87507346276,
          "column_name": "name_inversion",
          "comparison_vector_value": "3",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 3,
          "is_null": false,
          "label": "level_3",
          "level_name": "level_3",
          "level_proportion": 0.000802757708835054,
          "log2_bayes_factor": 13.400996153467629,
          "m_probability": 0.14333415629686924,
          "num_levels": 5,
          "sql_expr": "WHEN first_name_l = surname_r AND surname_l = first_name_r THEN 3",
          "u_probability": 0.00001325097639784281
         },
         {
          "bayes_factor": 84.21275991056915,
          "column_name": "name_inversion",
          "comparison_vector_value": "2",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 2,
          "is_null": false,
          "label": "level_2",
          "level_name": "level_2",
          "level_proportion": 0.003082859436450116,
          "log2_bayes_factor": 6.3959669417774965,
          "m_probability": 0.178015412463166,
          "num_levels": 5,
          "sql_expr": "WHEN first_name_l = first_name_r THEN 2",
          "u_probability": 0.0021138769546587928
         },
         {
          "bayes_factor": 43.14154400089717,
          "column_name": "name_inversion",
          "comparison_vector_value": "1",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 1,
          "is_null": false,
          "label": "level_1",
          "level_name": "level_1",
          "level_proportion": 0.005140347681783208,
          "log2_bayes_factor": 5.43100590519603,
          "m_probability": 0.1799810780566816,
          "num_levels": 5,
          "sql_expr": "WHEN surname_l = surname_r THEN 1",
          "u_probability": 0.004171873821969346
         },
         {
          "bayes_factor": 0.32715524872500057,
          "column_name": "name_inversion",
          "comparison_vector_value": "0",
          "gamma_column_name": "gamma_name_inversion",
          "gamma_index": 0,
          "is_null": false,
          "label": "level_0",
          "level_name": "level_0",
          "level_proportion": 0.9899958850235109,
          "log2_bayes_factor": -1.6119526780809075,
          "m_probability": 0.3250872781167861,
          "num_levels": 5,
          "sql_expr": "ELSE 0",
          "u_probability": 0.9936789318946468
         },
         {
          "bayes_factor": 7.5754215135378,
          "column_name": "city",
          "comparison_vector_value": "2",
          "gamma_column_name": "gamma_city",
          "gamma_index": 2,
          "is_null": false,
          "label": "level_2",
          "level_name": "level_2",
          "level_proportion": 0.09247482158475576,
          "log2_bayes_factor": 2.9213261656247806,
          "m_probability": 0.6760480820908364,
          "num_levels": 3,
          "sql_expr": "WHEN JARO_WINKLER_SIM(city_l, city_r) >= 1.0 THEN 2",
          "u_probability": 0.0892423056436783
         },
         {
          "bayes_factor": 10.300307763779484,
          "column_name": "city",
          "comparison_vector_value": "1",
          "gamma_column_name": "gamma_city",
          "gamma_index": 1,
          "is_null": false,
          "label": "level_1",
          "level_name": "level_1",
          "level_proportion": 0.02173640315205869,
          "log2_bayes_factor": 3.364615539348775,
          "m_probability": 0.2129801834705903,
          "num_levels": 3,
          "sql_expr": "WHEN JARO_WINKLER_SIM(city_l, city_r) >= 0.88 THEN 1",
          "u_probability": 0.020677069885185802
         },
         {
          "bayes_factor": 0.12467604775074131,
          "column_name": "city",
          "comparison_vector_value": "0",
          "gamma_column_name": "gamma_city",
          "gamma_index": 0,
          "is_null": false,
          "label": "level_0",
          "level_name": "level_0",
          "level_proportion": 0.8857887752631854,
          "log2_bayes_factor": -3.003743767733968,
          "m_probability": 0.11097173443857297,
          "num_levels": 3,
          "sql_expr": "ELSE 0",
          "u_probability": 0.8900806244711358
         },
         {
          "bayes_factor": 8537.588804915724,
          "column_name": "email",
          "comparison_vector_value": "2",
          "gamma_column_name": "gamma_email",
          "gamma_index": 2,
          "is_null": false,
          "label": "level_2",
          "level_name": "level_2",
          "level_proportion": 0.0036458147411618713,
          "log2_bayes_factor": 13.059612964535702,
          "m_probability": 0.6481277887828767,
          "num_levels": 3,
          "sql_expr": "WHEN JARO_WINKLER_SIM(email_l, email_r) >= 1.0 THEN 2",
          "u_probability": 0.00007591461753343069
         },
         {
          "bayes_factor": 26193.64119339461,
          "column_name": "email",
          "comparison_vector_value": "1",
          "gamma_column_name": "gamma_email",
          "gamma_index": 1,
          "is_null": false,
          "label": "level_1",
          "level_name": "level_1",
          "level_proportion": 0.0018015059731803925,
          "log2_bayes_factor": 14.676929003060637,
          "m_probability": 0.32479282624986167,
          "num_levels": 3,
          "sql_expr": "WHEN JARO_WINKLER_SIM(email_l, email_r) >= 0.88 THEN 1",
          "u_probability": 0.000012399682192018664
         },
         {
          "bayes_factor": 0.02708177667540402,
          "column_name": "email",
          "comparison_vector_value": "0",
          "gamma_column_name": "gamma_email",
          "gamma_index": 0,
          "is_null": false,
          "label": "level_0",
          "level_name": "level_0",
          "level_proportion": 0.9945526792856578,
          "log2_bayes_factor": -5.2065338010569935,
          "m_probability": 0.02707938496726161,
          "num_levels": 3,
          "sql_expr": "ELSE 0",
          "u_probability": 0.9999116857002746
         },
         {
          "bayes_factor": 14971.363071299604,
          "column_name": "dob",
          "comparison_vector_value": "1",
          "gamma_column_name": "gamma_dob",
          "gamma_index": 1,
          "is_null": false,
          "label": "level_1",
          "level_name": "level_1",
          "level_proportion": 0.00333920223422985,
          "log2_bayes_factor": 13.869917957437533,
          "m_probability": 0.5989503315946443,
          "num_levels": 2,
          "sql_expr": "WHEN dob_l = dob_r THEN 1",
          "u_probability": 0.00004000639946691586
         },
         {
          "bayes_factor": 0.4010657136005066,
          "column_name": "dob",
          "comparison_vector_value": "0",
          "gamma_column_name": "gamma_dob",
          "gamma_index": 0,
          "is_null": false,
          "label": "level_0",
          "level_name": "level_0",
          "level_proportion": 0.9966607977657703,
          "log2_bayes_factor": -1.318089456914419,
          "m_probability": 0.40104966840535583,
          "num_levels": 2,
          "sql_expr": "ELSE 0",
          "u_probability": 0.9999599936005332
         }
        ]
       },
       "encoding": {
        "color": {
         "field": "log2_bayes_factor",
         "scale": {
          "domain": [
           -10,
           0,
           10
          ],
          "range": [
           "red",
           "orange",
           "green"
          ]
         },
         "title": "Match weight",
         "type": "quantitative"
        },
        "row": {
         "field": "column_name",
         "header": {
          "labelAlign": "left",
          "labelAnchor": "middle",
          "labelAngle": 0
         },
         "sort": {
          "field": "gamma_index"
         },
         "type": "nominal"
        },
        "tooltip": [
         {
          "field": "column_name",
          "type": "nominal"
         },
         {
          "field": "label",
          "title": "level label",
          "type": "ordinal"
         },
         {
          "field": "sql_expr",
          "type": "nominal"
         },
         {
          "field": "m_probability",
          "format": ".4f",
          "type": "quantitative"
         },
         {
          "field": "u_probability",
          "format": ".4f",
          "type": "quantitative"
         },
         {
          "field": "bayes_factor",
          "format": ".4f",
          "title": "Bayes factor = m/u",
          "type": "quantitative"
         },
         {
          "field": "log2_bayes_factor",
          "format": ".4f",
          "title": "Match weight = log2(m/u)",
          "type": "quantitative"
         },
         {
          "field": "level_proportion",
          "format": ".2%",
          "title": "Percentage of non-null comparisons in this level",
          "type": "nominal"
         }
        ],
        "x": {
         "axis": {
          "title": "Match weight = log2(m/u)",
          "values": [
           -10,
           -5,
           0,
           5,
           10
          ]
         },
         "field": "log2_bayes_factor",
         "scale": {
          "domain": [
           -10,
           10
          ]
         },
         "type": "quantitative"
        },
        "y": {
         "axis": {
          "title": null
         },
         "field": "label",
         "sort": {
          "field": "gamma_index"
         },
         "type": "nominal"
        }
       },
       "height": 50,
       "mark": {
        "clip": true,
        "type": "bar"
       },
       "resolve": {
        "scale": {
         "y": "independent"
        }
       },
       "selection": {
        "selector076": {
         "bind": "scales",
         "encodings": [
          "x"
         ],
         "type": "interval"
        }
       },
       "title": {
        "subtitle": "Use mousewheeel to zoom",
        "text": "Influence of comparison vector values on match probability"
       }
      },
      "text/plain": [
       "<VegaLite 4 object>\n",
       "\n",
       "If you see this message, it means the renderer has not been properly enabled\n",
       "for the frontend that you are using. For more information, see\n",
       "https://altair-viz.github.io/user_guide/troubleshooting.html\n"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.bayes_factor_chart()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If charts aren't displaying correctly in your notebook, you can write them to a file (by default splink_charts.html)\n",
    "model.all_charts_write_html_file(\"splink_charts.html\", overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also generate a report which explains how the match probability was computed for an individual comparison row.  \n",
    "\n",
    "Note that you need to convert the row to a dictionary for this to work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Initial probability of match (prior) = λ = 0.005509\n",
      "------\n",
      "Comparison of name_inversion.  Values are:         \n",
      "name_inversion_l:                                  Thomas, J cob\n",
      "name_inversion_r:                                  Bond, Harry \n",
      "Comparison has:                                    5 levels\n",
      "Level for this comparison:                         gamma_name_inversion = 0\n",
      "m probability = P(level|match):                    0.3251\n",
      "u probability = P(level|non-match):                0.9937\n",
      "Bayes factor = m/u:                                0.3272\n",
      "New probability of match (updated belief):         0.001809\n",
      "\n",
      "------\n",
      "Comparison of city.  Values are:                   \n",
      "city_l:                                            Sudlreand\n",
      "city_r:                                            None\n",
      "Comparison has:                                    3 levels\n",
      "Level for this comparison:                         gamma_city = -1\n",
      "m probability = P(level|match):                    1\n",
      "u probability = P(level|non-match):                1\n",
      "Bayes factor = m/u:                                1\n",
      "New probability of match (updated belief):         0.001809\n",
      "\n",
      "------\n",
      "Comparison of email.  Values are:                  \n",
      "email_l:                                           hknapp@davis-allen.info\n",
      "email_r:                                           alexanderthomas@reyes.com\n",
      "Comparison has:                                    3 levels\n",
      "Level for this comparison:                         gamma_email = 0\n",
      "m probability = P(level|match):                    0.02708\n",
      "u probability = P(level|non-match):                0.9999\n",
      "Bayes factor = m/u:                                0.02708\n",
      "New probability of match (updated belief):         4.907e-05\n",
      "\n",
      "------\n",
      "Comparison of dob.  Values are:                    \n",
      "dob_l:                                             2007-05-28\n",
      "dob_r:                                             1979-12-08\n",
      "Comparison has:                                    2 levels\n",
      "Level for this comparison:                         gamma_dob = 0\n",
      "m probability = P(level|match):                    0.401\n",
      "u probability = P(level|non-match):                1\n",
      "Bayes factor = m/u:                                0.4011\n",
      "New probability of match (updated belief):         1.968e-05\n",
      "\n",
      "\n",
      "Final probability of match = 1.968e-05\n",
      "\n",
      "Reminder:\n",
      "\n",
      "The m probability for a given level is the proportion of matches which are in this level.\n",
      "We would generally expect the highest similarity level to have the largest proportion of matches.\n",
      "For example, we would expect first name field to match exactly amongst most matching records, except where nicknames, aliases or typos have occurred.\n",
      "For a comparison column that changes through time, like address, we may expect a lower proportion of comparisons to be in the highest similarity level.\n",
      "\n",
      "The u probability for a given level is the proportion of non-matches which are in this level.\n",
      "We would generally expect the lowest similarity level to have the highest proportion of non-matches, but the magnitude depends on the cardinality of the field.\n",
      "For example, we would expect that in the vast majority of non-matching records, the date of birth field would not match.  However, we would expect it to be common for gender to match amongst non-matches.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from splink.intuition import intuition_report\n",
    "row_dict = df_e.toPandas().sample(1).to_dict(orient=\"records\")[0]\n",
    "print(intuition_report(row_dict, model))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "data": {
      "application/vnd.vegalite.v4+json": {
       "$schema": "https://vega.github.io/schema/vega-lite/v4.8.1.json",
       "config": {
        "title": {
         "fontSize": 14
        },
        "view": {
         "continuousHeight": 300,
         "continuousWidth": 400
        }
       },
       "data": {
        "name": "data-b6f9d8ee7ef48a0713fc85e66a024fff"
       },
       "datasets": {
        "data-b6f9d8ee7ef48a0713fc85e66a024fff": [
         {
          "binwidth": 0.564527783571009,
          "count_rows": 81038,
          "freqdensity": 143550.06495407797,
          "normalised": 0.5466712538535742,
          "splink_score_bin_high": -18.071904405390065,
          "splink_score_bin_low": -18.636432188961074
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -17.507376621819056,
          "splink_score_bin_low": -18.071904405390065
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -16.942848838248047,
          "splink_score_bin_low": -17.507376621819056
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -16.378321054677038,
          "splink_score_bin_low": -16.942848838248047
         },
         {
          "binwidth": 0.5645277835710072,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -15.81379327110603,
          "splink_score_bin_low": -16.378321054677038
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 23697,
          "freqdensity": 41976.67624098306,
          "normalised": 0.15985671786776753,
          "splink_score_bin_high": -15.249265487535022,
          "splink_score_bin_low": -15.81379327110603
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -14.684737703964013,
          "splink_score_bin_low": -15.249265487535022
         },
         {
          "binwidth": 0.5645277835710072,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -14.120209920393005,
          "splink_score_bin_low": -14.684737703964013
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -13.555682136821996,
          "splink_score_bin_low": -14.120209920393005
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 23868,
          "freqdensity": 42279.58427310561,
          "normalised": 0.1610102604577742,
          "splink_score_bin_high": -12.991154353250987,
          "splink_score_bin_low": -13.555682136821996
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 8852,
          "freqdensity": 15680.361990344012,
          "normalised": 0.05971438015636908,
          "splink_score_bin_high": -12.426626569679978,
          "splink_score_bin_low": -12.991154353250987
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 2047,
          "freqdensity": 3626.039425467035,
          "normalised": 0.013808781764582865,
          "splink_score_bin_high": -11.86209878610897,
          "splink_score_bin_low": -12.426626569679978
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 305,
          "freqdensity": 540.2745602185861,
          "normalised": 0.0020574882453335487,
          "splink_score_bin_high": -11.29757100253796,
          "splink_score_bin_low": -11.86209878610897
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -10.733043218966952,
          "splink_score_bin_low": -11.29757100253796
         },
         {
          "binwidth": 0.5645277835710072,
          "count_rows": 5171,
          "freqdensity": 9159.868035705958,
          "normalised": 0.034882858087278076,
          "splink_score_bin_high": -10.168515435395944,
          "splink_score_bin_low": -10.733043218966952
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -9.603987651824935,
          "splink_score_bin_low": -10.168515435395944
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -9.039459868253926,
          "splink_score_bin_low": -9.603987651824935
         },
         {
          "binwidth": 0.5645277835710072,
          "count_rows": 92,
          "freqdensity": 162.96806406593467,
          "normalised": 0.0006206194051497936,
          "splink_score_bin_high": -8.47493208468292,
          "splink_score_bin_low": -9.039459868253926
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -7.91040430111191,
          "splink_score_bin_low": -8.47493208468292
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 1683,
          "freqdensity": 2981.252737206165,
          "normalised": 0.011353287596381515,
          "splink_score_bin_high": -7.345876517540901,
          "splink_score_bin_low": -7.91040430111191
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 357,
          "freqdensity": 632.3869442558532,
          "normalised": 0.00240827312650517,
          "splink_score_bin_high": -6.781348733969892,
          "splink_score_bin_low": -7.345876517540901
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 74,
          "freqdensity": 131.083008053034,
          "normalised": 0.0004991938693596151,
          "splink_score_bin_high": -6.216820950398883,
          "splink_score_bin_low": -6.781348733969892
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 64,
          "freqdensity": 113.36908804586724,
          "normalised": 0.0004317352383650725,
          "splink_score_bin_high": -5.652293166827874,
          "splink_score_bin_low": -6.216820950398883
         },
         {
          "binwidth": 0.5645277835710072,
          "count_rows": 101,
          "freqdensity": 178.9105920723848,
          "normalised": 0.0006813321730448821,
          "splink_score_bin_high": -5.087765383256867,
          "splink_score_bin_low": -5.652293166827874
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 24,
          "freqdensity": 42.51340801720021,
          "normalised": 0.00016190071438690218,
          "splink_score_bin_high": -4.523237599685858,
          "splink_score_bin_low": -5.087765383256867
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 2,
          "freqdensity": 3.5427840014333514,
          "normalised": 0.000013491726198908516,
          "splink_score_bin_high": -3.9587098161148493,
          "splink_score_bin_low": -4.523237599685858
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 3,
          "freqdensity": 5.314176002150027,
          "normalised": 0.000020237589298362772,
          "splink_score_bin_high": -3.3941820325438403,
          "splink_score_bin_low": -3.9587098161148493
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 27,
          "freqdensity": 47.82758401935024,
          "normalised": 0.00018213830368526493,
          "splink_score_bin_high": -2.8296542489728314,
          "splink_score_bin_low": -3.3941820325438403
         },
         {
          "binwidth": 0.5645277835710072,
          "count_rows": 11,
          "freqdensity": 19.485312007883493,
          "normalised": 0.00007420449409399706,
          "splink_score_bin_high": -2.265126465401824,
          "splink_score_bin_low": -2.8296542489728314
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -1.7005986818308152,
          "splink_score_bin_low": -2.265126465401824
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": -1.1360708982598062,
          "splink_score_bin_low": -1.7005986818308152
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 1,
          "freqdensity": 1.7713920007166757,
          "normalised": 0.000006745863099454258,
          "splink_score_bin_high": -0.5715431146887973,
          "splink_score_bin_low": -1.1360708982598062
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 29,
          "freqdensity": 51.370368020783594,
          "normalised": 0.00019563002988417347,
          "splink_score_bin_high": -0.007015331117788293,
          "splink_score_bin_low": -0.5715431146887973
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 12,
          "freqdensity": 21.256704008600106,
          "normalised": 0.00008095035719345109,
          "splink_score_bin_high": 0.5575124524532207,
          "splink_score_bin_low": -0.007015331117788293
         },
         {
          "binwidth": 0.5645277835710054,
          "count_rows": 2,
          "freqdensity": 3.5427840014333736,
          "normalised": 0.000013491726198908599,
          "splink_score_bin_high": 1.122040236024226,
          "splink_score_bin_low": 0.5575124524532207
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 1.686568019595235,
          "splink_score_bin_low": 1.122040236024226
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 7,
          "freqdensity": 12.399744005016728,
          "normalised": 0.0000472210416961798,
          "splink_score_bin_high": 2.251095803166244,
          "splink_score_bin_low": 1.686568019595235
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 8,
          "freqdensity": 14.171136005733405,
          "normalised": 0.000053966904795634064,
          "splink_score_bin_high": 2.815623586737253,
          "splink_score_bin_low": 2.251095803166244
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": 3.380151370308262,
          "splink_score_bin_low": 2.815623586737253
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": 3.944679153879271,
          "splink_score_bin_low": 3.380151370308262
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 7,
          "freqdensity": 12.399744005016728,
          "normalised": 0.0000472210416961798,
          "splink_score_bin_high": 4.50920693745028,
          "splink_score_bin_low": 3.944679153879271
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 5.073734721021289,
          "splink_score_bin_low": 4.50920693745028
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 42,
          "freqdensity": 74.39846403010037,
          "normalised": 0.0002833262501770788,
          "splink_score_bin_high": 5.638262504592298,
          "splink_score_bin_low": 5.073734721021289
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 6.202790288163307,
          "splink_score_bin_low": 5.638262504592298
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 5,
          "freqdensity": 8.856960003583378,
          "normalised": 0.00003372931549727129,
          "splink_score_bin_high": 6.767318071734316,
          "splink_score_bin_low": 6.202790288163307
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 21,
          "freqdensity": 37.199232015050185,
          "normalised": 0.0001416631250885394,
          "splink_score_bin_high": 7.331845855305325,
          "splink_score_bin_low": 6.767318071734316
         },
         {
          "binwidth": 0.5645277835710054,
          "count_rows": 31,
          "freqdensity": 54.91315202221729,
          "normalised": 0.0002091217560830833,
          "splink_score_bin_high": 7.89637363887633,
          "splink_score_bin_low": 7.331845855305325
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 8.46090142244734,
          "splink_score_bin_low": 7.89637363887633
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 1,
          "freqdensity": 1.7713920007166757,
          "normalised": 0.000006745863099454258,
          "splink_score_bin_high": 9.025429206018348,
          "splink_score_bin_low": 8.46090142244734
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 3,
          "freqdensity": 5.314176002150027,
          "normalised": 0.000020237589298362772,
          "splink_score_bin_high": 9.589956989589357,
          "splink_score_bin_low": 9.025429206018348
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 10.154484773160366,
          "splink_score_bin_low": 9.589956989589357
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 8,
          "freqdensity": 14.171136005733405,
          "normalised": 0.000053966904795634064,
          "splink_score_bin_high": 10.719012556731375,
          "splink_score_bin_low": 10.154484773160366
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": 11.283540340302384,
          "splink_score_bin_low": 10.719012556731375
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 2,
          "freqdensity": 3.5427840014333514,
          "normalised": 0.000013491726198908516,
          "splink_score_bin_high": 11.848068123873393,
          "splink_score_bin_low": 11.283540340302384
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 7,
          "freqdensity": 12.399744005016728,
          "normalised": 0.0000472210416961798,
          "splink_score_bin_high": 12.412595907444402,
          "splink_score_bin_low": 11.848068123873393
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 18,
          "freqdensity": 31.88505601290016,
          "normalised": 0.00012142553579017663,
          "splink_score_bin_high": 12.97712369101541,
          "splink_score_bin_low": 12.412595907444402
         },
         {
          "binwidth": 0.5645277835710054,
          "count_rows": 5,
          "freqdensity": 8.856960003583433,
          "normalised": 0.0000337293154972715,
          "splink_score_bin_high": 13.541651474586416,
          "splink_score_bin_low": 12.97712369101541
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 27,
          "freqdensity": 47.82758401935024,
          "normalised": 0.00018213830368526493,
          "splink_score_bin_high": 14.106179258157425,
          "splink_score_bin_low": 13.541651474586416
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 11,
          "freqdensity": 19.485312007883433,
          "normalised": 0.00007420449409399683,
          "splink_score_bin_high": 14.670707041728434,
          "splink_score_bin_low": 14.106179258157425
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 27,
          "freqdensity": 47.82758401935024,
          "normalised": 0.00018213830368526493,
          "splink_score_bin_high": 15.235234825299443,
          "splink_score_bin_low": 14.670707041728434
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 15.799762608870452,
          "splink_score_bin_low": 15.235234825299443
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 8,
          "freqdensity": 14.171136005733405,
          "normalised": 0.000053966904795634064,
          "splink_score_bin_high": 16.36429039244146,
          "splink_score_bin_low": 15.799762608870452
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 3,
          "freqdensity": 5.314176002150027,
          "normalised": 0.000020237589298362772,
          "splink_score_bin_high": 16.92881817601247,
          "splink_score_bin_low": 16.36429039244146
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 14,
          "freqdensity": 24.799488010033457,
          "normalised": 0.0000944420833923596,
          "splink_score_bin_high": 17.49334595958348,
          "splink_score_bin_low": 16.92881817601247
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 17,
          "freqdensity": 30.113664012183484,
          "normalised": 0.00011467967269072237,
          "splink_score_bin_high": 18.057873743154488,
          "splink_score_bin_low": 17.49334595958348
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": 18.622401526725497,
          "splink_score_bin_low": 18.057873743154488
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 10,
          "freqdensity": 17.713920007166756,
          "normalised": 0.00006745863099454258,
          "splink_score_bin_high": 19.186929310296506,
          "splink_score_bin_low": 18.622401526725497
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 19.751457093867515,
          "splink_score_bin_low": 19.186929310296506
         },
         {
          "binwidth": 0.5645277835710019,
          "count_rows": 20,
          "freqdensity": 35.42784001433396,
          "normalised": 0.00013491726198908686,
          "splink_score_bin_high": 20.315984877438517,
          "splink_score_bin_low": 19.751457093867515
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 69,
          "freqdensity": 122.22604804945061,
          "normalised": 0.00046546455386234375,
          "splink_score_bin_high": 20.880512661009526,
          "splink_score_bin_low": 20.315984877438517
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 18,
          "freqdensity": 31.88505601290016,
          "normalised": 0.00012142553579017663,
          "splink_score_bin_high": 21.445040444580535,
          "splink_score_bin_low": 20.880512661009526
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 12,
          "freqdensity": 21.256704008600106,
          "normalised": 0.00008095035719345109,
          "splink_score_bin_high": 22.009568228151544,
          "splink_score_bin_low": 21.445040444580535
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 38,
          "freqdensity": 67.31289602723368,
          "normalised": 0.00025634279777926177,
          "splink_score_bin_high": 22.574096011722553,
          "splink_score_bin_low": 22.009568228151544
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 34,
          "freqdensity": 60.22732802436697,
          "normalised": 0.00022935934538144474,
          "splink_score_bin_high": 23.13862379529356,
          "splink_score_bin_low": 22.574096011722553
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 23.70315157886457,
          "splink_score_bin_low": 23.13862379529356
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": 24.26767936243558,
          "splink_score_bin_low": 23.70315157886457
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 1,
          "freqdensity": 1.7713920007166757,
          "normalised": 0.000006745863099454258,
          "splink_score_bin_high": 24.83220714600659,
          "splink_score_bin_low": 24.26767936243558
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 25.396734929577597,
          "splink_score_bin_low": 24.83220714600659
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 6,
          "freqdensity": 10.628352004300053,
          "normalised": 0.000040475178596725545,
          "splink_score_bin_high": 25.961262713148606,
          "splink_score_bin_low": 25.396734929577597
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 8,
          "freqdensity": 14.171136005733405,
          "normalised": 0.000053966904795634064,
          "splink_score_bin_high": 26.525790496719615,
          "splink_score_bin_low": 25.961262713148606
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": 27.090318280290624,
          "splink_score_bin_low": 26.525790496719615
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 3,
          "freqdensity": 5.314176002150027,
          "normalised": 0.000020237589298362772,
          "splink_score_bin_high": 27.654846063861633,
          "splink_score_bin_low": 27.090318280290624
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 31,
          "freqdensity": 54.91315202221694,
          "normalised": 0.00020912175608308196,
          "splink_score_bin_high": 28.219373847432642,
          "splink_score_bin_low": 27.654846063861633
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 22,
          "freqdensity": 38.970624015766866,
          "normalised": 0.00014840898818799367,
          "splink_score_bin_high": 28.78390163100365,
          "splink_score_bin_low": 28.219373847432642
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 29.34842941457466,
          "splink_score_bin_low": 28.78390163100365
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 21,
          "freqdensity": 37.199232015050185,
          "normalised": 0.0001416631250885394,
          "splink_score_bin_high": 29.91295719814567,
          "splink_score_bin_low": 29.34842941457466
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 13,
          "freqdensity": 23.028096009316783,
          "normalised": 0.00008769622029290535,
          "splink_score_bin_high": 30.477484981716678,
          "splink_score_bin_low": 29.91295719814567
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 12,
          "freqdensity": 21.256704008600106,
          "normalised": 0.00008095035719345109,
          "splink_score_bin_high": 31.042012765287687,
          "splink_score_bin_low": 30.477484981716678
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 1,
          "freqdensity": 1.7713920007166757,
          "normalised": 0.000006745863099454258,
          "splink_score_bin_high": 31.606540548858696,
          "splink_score_bin_low": 31.042012765287687
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": 32.171068332429705,
          "splink_score_bin_low": 31.606540548858696
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 12,
          "freqdensity": 21.256704008600106,
          "normalised": 0.00008095035719345109,
          "splink_score_bin_high": 32.735596116000714,
          "splink_score_bin_low": 32.171068332429705
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 6,
          "freqdensity": 10.628352004300053,
          "normalised": 0.000040475178596725545,
          "splink_score_bin_high": 33.30012389957172,
          "splink_score_bin_low": 32.735596116000714
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 0,
          "freqdensity": 0,
          "normalised": 0,
          "splink_score_bin_high": 33.86465168314273,
          "splink_score_bin_low": 33.30012389957172
         },
         {
          "binwidth": 0.5645277835710019,
          "count_rows": 3,
          "freqdensity": 5.314176002150093,
          "normalised": 0.000020237589298363026,
          "splink_score_bin_high": 34.429179466713734,
          "splink_score_bin_low": 33.86465168314273
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 2,
          "freqdensity": 3.5427840014333514,
          "normalised": 0.000013491726198908516,
          "splink_score_bin_high": 34.99370725028474,
          "splink_score_bin_low": 34.429179466713734
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 9,
          "freqdensity": 15.94252800645008,
          "normalised": 0.000060712767895088313,
          "splink_score_bin_high": 35.55823503385575,
          "splink_score_bin_low": 34.99370725028474
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 33,
          "freqdensity": 58.455936023650295,
          "normalised": 0.0002226134822819905,
          "splink_score_bin_high": 36.12276281742676,
          "splink_score_bin_low": 35.55823503385575
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 6,
          "freqdensity": 10.628352004300053,
          "normalised": 0.000040475178596725545,
          "splink_score_bin_high": 36.68729060099777,
          "splink_score_bin_low": 36.12276281742676
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 11,
          "freqdensity": 19.485312007883433,
          "normalised": 0.00007420449409399683,
          "splink_score_bin_high": 37.25181838456878,
          "splink_score_bin_low": 36.68729060099777
         },
         {
          "binwidth": 0.564527783571009,
          "count_rows": 16,
          "freqdensity": 28.34227201146681,
          "normalised": 0.00010793380959126813,
          "splink_score_bin_high": 37.81634616813979,
          "splink_score_bin_low": 37.25181838456878
         }
        ]
       },
       "encoding": {
        "tooltip": [
         {
          "field": "count_rows",
          "title": "count",
          "type": "quantitative"
         }
        ],
        "x": {
         "axis": {
          "title": "splink score"
         },
         "bin": "binned",
         "field": "splink_score_bin_low",
         "type": "quantitative"
        },
        "x2": {
         "field": "splink_score_bin_high"
        },
        "y": {
         "axis": {
          "title": "probability density"
         },
         "field": "normalised",
         "type": "quantitative"
        }
       },
       "height": 200,
       "mark": "bar",
       "selection": {
        "selector076": {
         "bind": "scales",
         "encodings": [
          "y"
         ],
         "type": "interval"
        }
       },
       "title": {
        "subtitle": "Use mousewheeel to zoom",
        "text": "Histogram of splink scores"
       },
       "width": 700
      },
      "text/plain": [
       "<VegaLite 4 object>\n",
       "\n",
       "If you see this message, it means the renderer has not been properly enabled\n",
       "for the frontend that you are using. For more information, see\n",
       "https://altair-viz.github.io/user_guide/troubleshooting.html\n"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from splink.diagnostics import splink_score_histogram\n",
    "splink_score_histogram(df_e, spark, 100, \"match_weight\", symmetric=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}