docs/source/example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2ac196e4",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "source": [
    "# Spark-Matcher example "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "609d5c78",
   "metadata": {},
   "source": [
    "This notebook shows how to use the `spark_matcher`. First we create a Spark session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06615ef2",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "%config Completer.use_jedi = False  # for proper autocompletion\n",
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5da9661",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "21/11/24 13:44:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n"
     ]
    }
   ],
   "source": [
    "spark = (SparkSession\n",
    "             .builder\n",
    "             .master(\"local\")\n",
    "             .enableHiveSupport()\n",
    "             .getOrCreate())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b939e181",
   "metadata": {},
   "source": [
    "In this notebook we use some example data that comes with `spark_matcher`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23ab7967",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "from spark_matcher.data import load_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "420b9232",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "a, b = load_data(spark)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a4d79e9",
   "metadata": {},
   "source": [
    "The two dataframes `a` and `b` both contain records North Carolina Voter Registry data. For each person there is a name, suburb and postcode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4793d94",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>suburb</th>\n",
       "      <th>postcode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>kiera matthews</td>\n",
       "      <td>charlotte</td>\n",
       "      <td>28216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>khimerc thomas</td>\n",
       "      <td>charlotte</td>\n",
       "      <td>2826g</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             name     suburb postcode\n",
       "0  kiera matthews  charlotte    28216\n",
       "1  khimerc thomas  charlotte    2826g"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.limit(2).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a52091b",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>suburb</th>\n",
       "      <th>postcode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>kiea matthews</td>\n",
       "      <td>charlotte</td>\n",
       "      <td>28218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chimerc thmas</td>\n",
       "      <td>chaflotte</td>\n",
       "      <td>28269</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name     suburb postcode\n",
       "0  kiea matthews  charlotte    28218\n",
       "1  chimerc thmas  chaflotte    28269"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.limit(2).toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "320e6045",
   "metadata": {},
   "source": [
    "We use the `spark_matcher` to link the records in dataframe `a` with the records in dataframe `b`. First import the `Matcher` and create an instance. The fields that are used for matching are given as the `col_names` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e9303ee",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "from spark_matcher.matcher import Matcher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc3e9994",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "myMatcher = Matcher(spark, col_names=['name', 'suburb', 'postcode'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c2edaad",
   "metadata": {},
   "source": [
    "Now we are ready for fitting the `Matcher` object using 'active learning'; this means that the user has to enter whether a pair is a match or not. \n",
    "You enter 'y' if a pair is a match or 'n' when a pair is not a match. You will be notified when the model has converged and you can stop training by pressing 'f'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b2803d3",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 1 (0+/0-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: kiea matthews\n",
      "suburb_1: charlotte\n",
      "postcode_1: 28218\n",
      "\n",
      "name_2: kiea matthews\n",
      "suburb_2: charlotte\n",
      "postcode_2: 28218\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " y\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 2 (1+/0-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: khimerc thomas\n",
      "suburb_1: charlotte\n",
      "postcode_1: 2826g\n",
      "\n",
      "name_2: kiea matthews\n",
      "suburb_2: charlotte\n",
      "postcode_2: 28218\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " n\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 3 (1+/1-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: john bentzen\n",
      "suburb_1: waxhaw\n",
      "postcode_1: 28173\n",
      "\n",
      "name_2: john hanegraaff\n",
      "suburb_2: waxhaw\n",
      "postcode_2: 28173\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " n\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 4 (1+/2-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: willie greene\n",
      "suburb_1: mooresville\n",
      "postcode_1: 28115\n",
      "\n",
      "name_2: lois greene\n",
      "suburb_2: mooresboro\n",
      "postcode_2: 28114\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " n\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 5 (1+/3-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: jennifer hannen\n",
      "suburb_1: greensboro\n",
      "postcode_1: 27405\n",
      "\n",
      "name_2: jennifer bentz\n",
      "suburb_2: greensboro\n",
      "postcode_2: 27407\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " n\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 6 (1+/4-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: crystal boone\n",
      "suburb_1: green mountain\n",
      "postcode_1: 28740\n",
      "\n",
      "name_2: crystnal boone\n",
      "suburb_2: green mountain\n",
      "postcode_2: 28750\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " y\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 7 (2+/4-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: latonja yarborovgh\n",
      "suburb_1: knightdale\n",
      "postcode_1: 27945\n",
      "\n",
      "name_2: latonja yarborough\n",
      "suburb_2: knivhtdale\n",
      "postcode_2: 2754s\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " y\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 8 (3+/4-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: jerome oliveah\n",
      "suburb_1: selms\n",
      "postcode_1: 27576\n",
      "\n",
      "name_2: jerome oliver\n",
      "suburb_2: selma\n",
      "postcode_2: 27576\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " y\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 9 (4+/4-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: latoyw oneal\n",
      "suburb_1: smihtfield\n",
      "postcode_1: 27577\n",
      "\n",
      "name_2: lato6a oneal\n",
      "suburb_2: smithfield\n",
      "postcode_2: 27537\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " y\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 10 (5+/4-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: patricia adams\n",
      "suburb_1: rocky mount\n",
      "postcode_1: 27804\n",
      "\n",
      "name_2: patricia barus\n",
      "suburb_2: valdese\n",
      "postcode_2: 28690\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " n\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nr. 11 (5+/5-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: kimberly gay\n",
      "suburb_1: kinston\n",
      "postcode_1: 28504\n",
      "\n",
      "name_2: kimbeahly gav\n",
      "suburb_2: kinston\n",
      "postcode_2: 28504\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " y\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classifier converged, enter 'f' to stop training\n",
      "\n",
      "Nr. 12 (6+/5-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: de1>ra benf:eld\n",
      "suburb_1: concord\n",
      "postcode_1: 28025\n",
      "\n",
      "name_2: debra benfield\n",
      "suburb_2: concord\n",
      "postcode_2: 28025\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " y\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classifier converged, enter 'f' to stop training\n",
      "\n",
      "Nr. 13 (7+/5-)\n",
      "Is this a match? (y)es, (n)o, (p)revious, (s)kip, (f)inish\n",
      "\n",
      "name_1: ruth edwards\n",
      "suburb_1: henderson\n",
      "postcode_1: 27536\n",
      "\n",
      "name_2: raechaun edwards\n",
      "suburb_2: lillington\n",
      "postcode_2: 27546\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " f\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<spark_matcher.matcher.matcher.Matcher at 0x7fab47c8e610>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myMatcher.fit(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b24a66d",
   "metadata": {},
   "source": [
    "The `Matcher` is now trained and can be used to predict on all data. This can be the data used for training or new data that was not seen by the model yet.\n",
    "By default the `threshold` is 0.5. A lower threshold results in more matches but also in more incorrect matches. By setting the `top_n` you can choose how \n",
    "many matches you want to see if there are more than one matche for a particular record."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56551b1d",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "result = myMatcher.predict(a, b, threshold=0.5, top_n=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71a04f95",
   "metadata": {},
   "source": [
    "Now let's have a look at the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd336dd8",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "result_pdf = result.toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd4ce794",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name_1</th>\n",
       "      <th>suburb_1</th>\n",
       "      <th>postcode_1</th>\n",
       "      <th>name_2</th>\n",
       "      <th>suburb_2</th>\n",
       "      <th>postcode_2</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>teresina fontana</td>\n",
       "      <td>newport</td>\n",
       "      <td>28570</td>\n",
       "      <td>teres'lna fontana</td>\n",
       "      <td>newport</td>\n",
       "      <td>28571</td>\n",
       "      <td>0.802804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>721</th>\n",
       "      <td>melissa wa5d</td>\n",
       "      <td>greensboro</td>\n",
       "      <td>274|0</td>\n",
       "      <td>melissa ward</td>\n",
       "      <td>greensboro</td>\n",
       "      <td>27410</td>\n",
       "      <td>0.954172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1026</th>\n",
       "      <td>thomas dy5on</td>\n",
       "      <td>statesville</td>\n",
       "      <td>z8677</td>\n",
       "      <td>thomas dyson</td>\n",
       "      <td>statesvile</td>\n",
       "      <td>28697</td>\n",
       "      <td>0.958674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>512</th>\n",
       "      <td>judirh coile</td>\n",
       "      <td>charlotte</td>\n",
       "      <td>2821q</td>\n",
       "      <td>judith coile</td>\n",
       "      <td>charlott</td>\n",
       "      <td>28224</td>\n",
       "      <td>0.959523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>340</th>\n",
       "      <td>ge0ffrey ryan</td>\n",
       "      <td>wilmington</td>\n",
       "      <td>z8403</td>\n",
       "      <td>geoffrey ryan</td>\n",
       "      <td>wilmnigton</td>\n",
       "      <td>28400</td>\n",
       "      <td>0.962752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>helem farmer</td>\n",
       "      <td>lincolnton</td>\n",
       "      <td>2809z</td>\n",
       "      <td>helen farmer</td>\n",
       "      <td>lincolnton</td>\n",
       "      <td>28092</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>367</th>\n",
       "      <td>heather stewart</td>\n",
       "      <td>concord</td>\n",
       "      <td>28027</td>\n",
       "      <td>heather stewart</td>\n",
       "      <td>concord</td>\n",
       "      <td>28027</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>366</th>\n",
       "      <td>heather caywood</td>\n",
       "      <td>charlotte</td>\n",
       "      <td>28278</td>\n",
       "      <td>heather cayw0od</td>\n",
       "      <td>char1otte</td>\n",
       "      <td>28278</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>henry wyatt</td>\n",
       "      <td>waynesville</td>\n",
       "      <td>28786</td>\n",
       "      <td>henty wyatt</td>\n",
       "      <td>waynesvillte</td>\n",
       "      <td>28786</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1105</th>\n",
       "      <td>zedekiah dawkins</td>\n",
       "      <td>hikh point</td>\n",
       "      <td>27z60</td>\n",
       "      <td>zedekiah dawkins</td>\n",
       "      <td>hikh point</td>\n",
       "      <td>27z60</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1106 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                name_1     suburb_1 postcode_1             name_2  \\\n",
       "1011  teresina fontana      newport      28570  teres'lna fontana   \n",
       "721       melissa wa5d   greensboro      274|0       melissa ward   \n",
       "1026      thomas dy5on  statesville      z8677       thomas dyson   \n",
       "512       judirh coile    charlotte      2821q       judith coile   \n",
       "340      ge0ffrey ryan   wilmington      z8403      geoffrey ryan   \n",
       "...                ...          ...        ...                ...   \n",
       "368       helem farmer   lincolnton      2809z       helen farmer   \n",
       "367    heather stewart      concord      28027    heather stewart   \n",
       "366    heather caywood    charlotte      28278    heather cayw0od   \n",
       "372        henry wyatt  waynesville      28786        henty wyatt   \n",
       "1105  zedekiah dawkins   hikh point      27z60   zedekiah dawkins   \n",
       "\n",
       "          suburb_2 postcode_2     score  \n",
       "1011       newport      28571  0.802804  \n",
       "721     greensboro      27410  0.954172  \n",
       "1026    statesvile      28697  0.958674  \n",
       "512       charlott      28224  0.959523  \n",
       "340     wilmnigton      28400  0.962752  \n",
       "...            ...        ...       ...  \n",
       "368     lincolnton      28092  1.000000  \n",
       "367        concord      28027  1.000000  \n",
       "366      char1otte      28278  1.000000  \n",
       "372   waynesvillte      28786  1.000000  \n",
       "1105    hikh point      27z60  1.000000  \n",
       "\n",
       "[1106 rows x 7 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_pdf.sort_values('score')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf80c73",
   "metadata": {},
   "source": [
    "If you want to use the `Matcher` later without having to retrain, you can save the `Matcher` and load it later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "249d4b6a",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "myMatcher.save('myMatcher.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "593c1dec",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "myRestoredMatcher = Matcher(spark)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "198ae237",
   "metadata": {
    "tags": [
     "keep_output"
    ]
   },
   "outputs": [],
   "source": [
    "myRestoredMatcher.load('myMatcher.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0192ad55",
   "metadata": {},
   "source": [
    "This `Matcher` object can be used to predict on new data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}