k-nearest-neighbours.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# k-Nearest Neighbours (kNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "kNN is a supervised method that can be used to solve classification problems and regression problems.\n",
    "\n",
    "Given\n",
    "\n",
    "- a training set $\\mathcal{T} = \\left\\{(x_1, y_1), (x_2, y_2), \\cdots, (x_n, y_n) \\right\\}$\n",
    "- a positive integer $k$ \n",
    "- a test observation $x'$\n",
    "\n",
    "Let $\\mathcal{N}$ denote the $k$ data points in the training set $\\mathcal{T}$ that are closest to $x'$.\n",
    "\n",
    "In kNN classification, the method selects the most frequent $y_i \\in \\mathcal{N}$ where $i \\in {1,2, \\cdots, k}$.\n",
    "\n",
    "In kNN regression, the method estimates $\\hat{y}$ for $x'$ by computing the mean of the $k$ points in the training set that are closest to $x'$:\n",
    "\n",
    "$$\n",
    "\\hat{y} = \\frac{1}{k} \\sum_{x_i \\in \\mathcal{N} } y_i\n",
    "$$\n",
    "\n",
    "Performance of the kNN algorithm depends on several factors including:\n",
    "\n",
    "- a suitable distance measure: we can use the Euclidean distance for numeric features but it has issues in higher dimensions. There exist many <a href=\"https://www.rdocumentation.org/packages/fingerprint/versions/3.5.4/topics/distance-methods\">distance metrics</a>.\n",
    "- an appropriate $k$ value: Choosing an appropriate $k$ can be difficult and depends on the <a href=\"http://scott.fortmann-roe.com/docs/BiasVariance.html\">bias-variance tradeoff</a>. A small $k$ value has low bias but high variance whereas a large $k$ value provide a smoother and less variable fit. There are some rule-of-thumb for picking the $k$ value. The value of $k$ must not be a multiple of the number of classes to avoid ties. For instance, the value of $k$ must be odd if problems with only two classes.\n",
    "\n",
    "Unlike other methods that build a model of the training (labelled) data, kNN has no explicit knowledge representation of a model. This means that predicting using kNN is associated with a high computational cost especially for large data sets. In addition, kNN performs poorly for high dimensional data because \"the ratio of the distances of the nearest and farthest neighbors to a given target in high dimensional space is almost 1 for a wide variety of data distributions and distance functions\" [<a href=\"https://bib.dbvis.de/uploadedFiles/155.pdf\">#<a>]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore Credit Card Fraud Detection Data Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we explore the <a href=\"https://www.kaggle.com/dalpozz/creditcardfraud\">Credit Card Fraud Detection Dataset</a> on Kaggle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:37.302624Z",
     "start_time": "2017-08-03T17:40:36.334632Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.424065Z",
     "start_time": "2017-08-03T17:40:37.304123Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/creditcard.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.454574Z",
     "start_time": "2017-08-03T17:40:40.426066Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>...</th>\n",
       "      <th>V21</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Amount</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.359807</td>\n",
       "      <td>-0.072781</td>\n",
       "      <td>2.536347</td>\n",
       "      <td>1.378155</td>\n",
       "      <td>-0.338321</td>\n",
       "      <td>0.462388</td>\n",
       "      <td>0.239599</td>\n",
       "      <td>0.098698</td>\n",
       "      <td>0.363787</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.018307</td>\n",
       "      <td>0.277838</td>\n",
       "      <td>-0.110474</td>\n",
       "      <td>0.066928</td>\n",
       "      <td>0.128539</td>\n",
       "      <td>-0.189115</td>\n",
       "      <td>0.133558</td>\n",
       "      <td>-0.021053</td>\n",
       "      <td>149.62</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.191857</td>\n",
       "      <td>0.266151</td>\n",
       "      <td>0.166480</td>\n",
       "      <td>0.448154</td>\n",
       "      <td>0.060018</td>\n",
       "      <td>-0.082361</td>\n",
       "      <td>-0.078803</td>\n",
       "      <td>0.085102</td>\n",
       "      <td>-0.255425</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.225775</td>\n",
       "      <td>-0.638672</td>\n",
       "      <td>0.101288</td>\n",
       "      <td>-0.339846</td>\n",
       "      <td>0.167170</td>\n",
       "      <td>0.125895</td>\n",
       "      <td>-0.008983</td>\n",
       "      <td>0.014724</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.358354</td>\n",
       "      <td>-1.340163</td>\n",
       "      <td>1.773209</td>\n",
       "      <td>0.379780</td>\n",
       "      <td>-0.503198</td>\n",
       "      <td>1.800499</td>\n",
       "      <td>0.791461</td>\n",
       "      <td>0.247676</td>\n",
       "      <td>-1.514654</td>\n",
       "      <td>...</td>\n",
       "      <td>0.247998</td>\n",
       "      <td>0.771679</td>\n",
       "      <td>0.909412</td>\n",
       "      <td>-0.689281</td>\n",
       "      <td>-0.327642</td>\n",
       "      <td>-0.139097</td>\n",
       "      <td>-0.055353</td>\n",
       "      <td>-0.059752</td>\n",
       "      <td>378.66</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.966272</td>\n",
       "      <td>-0.185226</td>\n",
       "      <td>1.792993</td>\n",
       "      <td>-0.863291</td>\n",
       "      <td>-0.010309</td>\n",
       "      <td>1.247203</td>\n",
       "      <td>0.237609</td>\n",
       "      <td>0.377436</td>\n",
       "      <td>-1.387024</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.108300</td>\n",
       "      <td>0.005274</td>\n",
       "      <td>-0.190321</td>\n",
       "      <td>-1.175575</td>\n",
       "      <td>0.647376</td>\n",
       "      <td>-0.221929</td>\n",
       "      <td>0.062723</td>\n",
       "      <td>0.061458</td>\n",
       "      <td>123.50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>-1.158233</td>\n",
       "      <td>0.877737</td>\n",
       "      <td>1.548718</td>\n",
       "      <td>0.403034</td>\n",
       "      <td>-0.407193</td>\n",
       "      <td>0.095921</td>\n",
       "      <td>0.592941</td>\n",
       "      <td>-0.270533</td>\n",
       "      <td>0.817739</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.009431</td>\n",
       "      <td>0.798278</td>\n",
       "      <td>-0.137458</td>\n",
       "      <td>0.141267</td>\n",
       "      <td>-0.206010</td>\n",
       "      <td>0.502292</td>\n",
       "      <td>0.219422</td>\n",
       "      <td>0.215153</td>\n",
       "      <td>69.99</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Time        V1        V2        V3        V4        V5        V6        V7  \\\n",
       "0   0.0 -1.359807 -0.072781  2.536347  1.378155 -0.338321  0.462388  0.239599   \n",
       "1   0.0  1.191857  0.266151  0.166480  0.448154  0.060018 -0.082361 -0.078803   \n",
       "2   1.0 -1.358354 -1.340163  1.773209  0.379780 -0.503198  1.800499  0.791461   \n",
       "3   1.0 -0.966272 -0.185226  1.792993 -0.863291 -0.010309  1.247203  0.237609   \n",
       "4   2.0 -1.158233  0.877737  1.548718  0.403034 -0.407193  0.095921  0.592941   \n",
       "\n",
       "         V8        V9  ...         V21       V22       V23       V24  \\\n",
       "0  0.098698  0.363787  ...   -0.018307  0.277838 -0.110474  0.066928   \n",
       "1  0.085102 -0.255425  ...   -0.225775 -0.638672  0.101288 -0.339846   \n",
       "2  0.247676 -1.514654  ...    0.247998  0.771679  0.909412 -0.689281   \n",
       "3  0.377436 -1.387024  ...   -0.108300  0.005274 -0.190321 -1.175575   \n",
       "4 -0.270533  0.817739  ...   -0.009431  0.798278 -0.137458  0.141267   \n",
       "\n",
       "        V25       V26       V27       V28  Amount  Class  \n",
       "0  0.128539 -0.189115  0.133558 -0.021053  149.62      0  \n",
       "1  0.167170  0.125895 -0.008983  0.014724    2.69      0  \n",
       "2 -0.327642 -0.139097 -0.055353 -0.059752  378.66      0  \n",
       "3  0.647376 -0.221929  0.062723  0.061458  123.50      0  \n",
       "4 -0.206010  0.502292  0.219422  0.215153   69.99      0  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if there are missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.575103Z",
     "start_time": "2017-08-03T17:40:40.456574Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Time      False\n",
       "V1        False\n",
       "V2        False\n",
       "V3        False\n",
       "V4        False\n",
       "V5        False\n",
       "V6        False\n",
       "V7        False\n",
       "V8        False\n",
       "V9        False\n",
       "V10       False\n",
       "V11       False\n",
       "V12       False\n",
       "V13       False\n",
       "V14       False\n",
       "V15       False\n",
       "V16       False\n",
       "V17       False\n",
       "V18       False\n",
       "V19       False\n",
       "V20       False\n",
       "V21       False\n",
       "V22       False\n",
       "V23       False\n",
       "V24       False\n",
       "V25       False\n",
       "V26       False\n",
       "V27       False\n",
       "V28       False\n",
       "Amount    False\n",
       "Class     False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalise the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T09:41:03.693445Z",
     "start_time": "2017-08-03T09:41:03.688443Z"
    }
   },
   "source": [
    "The features V1, V2, ... and V28 are normalised because of the PCA transformation. However, the Amount feature is not. Since the kNN method relies distances between the observation, we need to rescale this feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.617637Z",
     "start_time": "2017-08-03T17:40:40.577104Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "ft_amount = df['Amount'].values.reshape(-1, 1)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(ft_amount)\n",
    "\n",
    "df['AmountNormalised'] = scaler.transform(ft_amount)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.645646Z",
     "start_time": "2017-08-03T17:40:40.619139Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>...</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Amount</th>\n",
       "      <th>Class</th>\n",
       "      <th>AmountNormalised</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.359807</td>\n",
       "      <td>-0.072781</td>\n",
       "      <td>2.536347</td>\n",
       "      <td>1.378155</td>\n",
       "      <td>-0.338321</td>\n",
       "      <td>0.462388</td>\n",
       "      <td>0.239599</td>\n",
       "      <td>0.098698</td>\n",
       "      <td>0.363787</td>\n",
       "      <td>...</td>\n",
       "      <td>0.277838</td>\n",
       "      <td>-0.110474</td>\n",
       "      <td>0.066928</td>\n",
       "      <td>0.128539</td>\n",
       "      <td>-0.189115</td>\n",
       "      <td>0.133558</td>\n",
       "      <td>-0.021053</td>\n",
       "      <td>149.62</td>\n",
       "      <td>0</td>\n",
       "      <td>0.244964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.191857</td>\n",
       "      <td>0.266151</td>\n",
       "      <td>0.166480</td>\n",
       "      <td>0.448154</td>\n",
       "      <td>0.060018</td>\n",
       "      <td>-0.082361</td>\n",
       "      <td>-0.078803</td>\n",
       "      <td>0.085102</td>\n",
       "      <td>-0.255425</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.638672</td>\n",
       "      <td>0.101288</td>\n",
       "      <td>-0.339846</td>\n",
       "      <td>0.167170</td>\n",
       "      <td>0.125895</td>\n",
       "      <td>-0.008983</td>\n",
       "      <td>0.014724</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.342475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.358354</td>\n",
       "      <td>-1.340163</td>\n",
       "      <td>1.773209</td>\n",
       "      <td>0.379780</td>\n",
       "      <td>-0.503198</td>\n",
       "      <td>1.800499</td>\n",
       "      <td>0.791461</td>\n",
       "      <td>0.247676</td>\n",
       "      <td>-1.514654</td>\n",
       "      <td>...</td>\n",
       "      <td>0.771679</td>\n",
       "      <td>0.909412</td>\n",
       "      <td>-0.689281</td>\n",
       "      <td>-0.327642</td>\n",
       "      <td>-0.139097</td>\n",
       "      <td>-0.055353</td>\n",
       "      <td>-0.059752</td>\n",
       "      <td>378.66</td>\n",
       "      <td>0</td>\n",
       "      <td>1.160686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.966272</td>\n",
       "      <td>-0.185226</td>\n",
       "      <td>1.792993</td>\n",
       "      <td>-0.863291</td>\n",
       "      <td>-0.010309</td>\n",
       "      <td>1.247203</td>\n",
       "      <td>0.237609</td>\n",
       "      <td>0.377436</td>\n",
       "      <td>-1.387024</td>\n",
       "      <td>...</td>\n",
       "      <td>0.005274</td>\n",
       "      <td>-0.190321</td>\n",
       "      <td>-1.175575</td>\n",
       "      <td>0.647376</td>\n",
       "      <td>-0.221929</td>\n",
       "      <td>0.062723</td>\n",
       "      <td>0.061458</td>\n",
       "      <td>123.50</td>\n",
       "      <td>0</td>\n",
       "      <td>0.140534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>-1.158233</td>\n",
       "      <td>0.877737</td>\n",
       "      <td>1.548718</td>\n",
       "      <td>0.403034</td>\n",
       "      <td>-0.407193</td>\n",
       "      <td>0.095921</td>\n",
       "      <td>0.592941</td>\n",
       "      <td>-0.270533</td>\n",
       "      <td>0.817739</td>\n",
       "      <td>...</td>\n",
       "      <td>0.798278</td>\n",
       "      <td>-0.137458</td>\n",
       "      <td>0.141267</td>\n",
       "      <td>-0.206010</td>\n",
       "      <td>0.502292</td>\n",
       "      <td>0.219422</td>\n",
       "      <td>0.215153</td>\n",
       "      <td>69.99</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.073403</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Time        V1        V2        V3        V4        V5        V6        V7  \\\n",
       "0   0.0 -1.359807 -0.072781  2.536347  1.378155 -0.338321  0.462388  0.239599   \n",
       "1   0.0  1.191857  0.266151  0.166480  0.448154  0.060018 -0.082361 -0.078803   \n",
       "2   1.0 -1.358354 -1.340163  1.773209  0.379780 -0.503198  1.800499  0.791461   \n",
       "3   1.0 -0.966272 -0.185226  1.792993 -0.863291 -0.010309  1.247203  0.237609   \n",
       "4   2.0 -1.158233  0.877737  1.548718  0.403034 -0.407193  0.095921  0.592941   \n",
       "\n",
       "         V8        V9        ...              V22       V23       V24  \\\n",
       "0  0.098698  0.363787        ...         0.277838 -0.110474  0.066928   \n",
       "1  0.085102 -0.255425        ...        -0.638672  0.101288 -0.339846   \n",
       "2  0.247676 -1.514654        ...         0.771679  0.909412 -0.689281   \n",
       "3  0.377436 -1.387024        ...         0.005274 -0.190321 -1.175575   \n",
       "4 -0.270533  0.817739        ...         0.798278 -0.137458  0.141267   \n",
       "\n",
       "        V25       V26       V27       V28  Amount  Class  AmountNormalised  \n",
       "0  0.128539 -0.189115  0.133558 -0.021053  149.62      0          0.244964  \n",
       "1  0.167170  0.125895 -0.008983  0.014724    2.69      0         -0.342475  \n",
       "2 -0.327642 -0.139097 -0.055353 -0.059752  378.66      0          1.160686  \n",
       "3  0.647376 -0.221929  0.062723  0.061458  123.50      0          0.140534  \n",
       "4 -0.206010  0.502292  0.219422  0.215153   69.99      0         -0.073403  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can't use the Time feature, so let us get rid of it. We don't need the Amount feature either."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.753672Z",
     "start_time": "2017-08-03T17:40:40.647146Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>V10</th>\n",
       "      <th>...</th>\n",
       "      <th>V21</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Class</th>\n",
       "      <th>AmountNormalised</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.359807</td>\n",
       "      <td>-0.072781</td>\n",
       "      <td>2.536347</td>\n",
       "      <td>1.378155</td>\n",
       "      <td>-0.338321</td>\n",
       "      <td>0.462388</td>\n",
       "      <td>0.239599</td>\n",
       "      <td>0.098698</td>\n",
       "      <td>0.363787</td>\n",
       "      <td>0.090794</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.018307</td>\n",
       "      <td>0.277838</td>\n",
       "      <td>-0.110474</td>\n",
       "      <td>0.066928</td>\n",
       "      <td>0.128539</td>\n",
       "      <td>-0.189115</td>\n",
       "      <td>0.133558</td>\n",
       "      <td>-0.021053</td>\n",
       "      <td>0</td>\n",
       "      <td>0.244964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.191857</td>\n",
       "      <td>0.266151</td>\n",
       "      <td>0.166480</td>\n",
       "      <td>0.448154</td>\n",
       "      <td>0.060018</td>\n",
       "      <td>-0.082361</td>\n",
       "      <td>-0.078803</td>\n",
       "      <td>0.085102</td>\n",
       "      <td>-0.255425</td>\n",
       "      <td>-0.166974</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.225775</td>\n",
       "      <td>-0.638672</td>\n",
       "      <td>0.101288</td>\n",
       "      <td>-0.339846</td>\n",
       "      <td>0.167170</td>\n",
       "      <td>0.125895</td>\n",
       "      <td>-0.008983</td>\n",
       "      <td>0.014724</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.342475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.358354</td>\n",
       "      <td>-1.340163</td>\n",
       "      <td>1.773209</td>\n",
       "      <td>0.379780</td>\n",
       "      <td>-0.503198</td>\n",
       "      <td>1.800499</td>\n",
       "      <td>0.791461</td>\n",
       "      <td>0.247676</td>\n",
       "      <td>-1.514654</td>\n",
       "      <td>0.207643</td>\n",
       "      <td>...</td>\n",
       "      <td>0.247998</td>\n",
       "      <td>0.771679</td>\n",
       "      <td>0.909412</td>\n",
       "      <td>-0.689281</td>\n",
       "      <td>-0.327642</td>\n",
       "      <td>-0.139097</td>\n",
       "      <td>-0.055353</td>\n",
       "      <td>-0.059752</td>\n",
       "      <td>0</td>\n",
       "      <td>1.160686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.966272</td>\n",
       "      <td>-0.185226</td>\n",
       "      <td>1.792993</td>\n",
       "      <td>-0.863291</td>\n",
       "      <td>-0.010309</td>\n",
       "      <td>1.247203</td>\n",
       "      <td>0.237609</td>\n",
       "      <td>0.377436</td>\n",
       "      <td>-1.387024</td>\n",
       "      <td>-0.054952</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.108300</td>\n",
       "      <td>0.005274</td>\n",
       "      <td>-0.190321</td>\n",
       "      <td>-1.175575</td>\n",
       "      <td>0.647376</td>\n",
       "      <td>-0.221929</td>\n",
       "      <td>0.062723</td>\n",
       "      <td>0.061458</td>\n",
       "      <td>0</td>\n",
       "      <td>0.140534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.158233</td>\n",
       "      <td>0.877737</td>\n",
       "      <td>1.548718</td>\n",
       "      <td>0.403034</td>\n",
       "      <td>-0.407193</td>\n",
       "      <td>0.095921</td>\n",
       "      <td>0.592941</td>\n",
       "      <td>-0.270533</td>\n",
       "      <td>0.817739</td>\n",
       "      <td>0.753074</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.009431</td>\n",
       "      <td>0.798278</td>\n",
       "      <td>-0.137458</td>\n",
       "      <td>0.141267</td>\n",
       "      <td>-0.206010</td>\n",
       "      <td>0.502292</td>\n",
       "      <td>0.219422</td>\n",
       "      <td>0.215153</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.073403</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         V1        V2        V3        V4        V5        V6        V7  \\\n",
       "0 -1.359807 -0.072781  2.536347  1.378155 -0.338321  0.462388  0.239599   \n",
       "1  1.191857  0.266151  0.166480  0.448154  0.060018 -0.082361 -0.078803   \n",
       "2 -1.358354 -1.340163  1.773209  0.379780 -0.503198  1.800499  0.791461   \n",
       "3 -0.966272 -0.185226  1.792993 -0.863291 -0.010309  1.247203  0.237609   \n",
       "4 -1.158233  0.877737  1.548718  0.403034 -0.407193  0.095921  0.592941   \n",
       "\n",
       "         V8        V9       V10        ...              V21       V22  \\\n",
       "0  0.098698  0.363787  0.090794        ...        -0.018307  0.277838   \n",
       "1  0.085102 -0.255425 -0.166974        ...        -0.225775 -0.638672   \n",
       "2  0.247676 -1.514654  0.207643        ...         0.247998  0.771679   \n",
       "3  0.377436 -1.387024 -0.054952        ...        -0.108300  0.005274   \n",
       "4 -0.270533  0.817739  0.753074        ...        -0.009431  0.798278   \n",
       "\n",
       "        V23       V24       V25       V26       V27       V28  Class  \\\n",
       "0 -0.110474  0.066928  0.128539 -0.189115  0.133558 -0.021053      0   \n",
       "1  0.101288 -0.339846  0.167170  0.125895 -0.008983  0.014724      0   \n",
       "2  0.909412 -0.689281 -0.327642 -0.139097 -0.055353 -0.059752      0   \n",
       "3 -0.190321 -1.175575  0.647376 -0.221929  0.062723  0.061458      0   \n",
       "4 -0.137458  0.141267 -0.206010  0.502292  0.219422  0.215153      0   \n",
       "\n",
       "   AmountNormalised  \n",
       "0          0.244964  \n",
       "1         -0.342475  \n",
       "2          1.160686  \n",
       "3          0.140534  \n",
       "4         -0.073403  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop(['Time', 'Amount'], axis=1, inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Re-sample the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us check the size of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.759173Z",
     "start_time": "2017-08-03T17:40:40.755172Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(284807, 30)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data set contains 284,807 examples. As kNN does not perform well with large data sets, we must find a way to reduce of size. We also need to ensure that the data is balanced before we run our kNN classifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.779180Z",
     "start_time": "2017-08-03T17:40:40.761174Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    284315\n",
       "1       492\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Class'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just by looking at the numbers above, we can see that the data is not balanced. Let us print out the proportion of data that is in class 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.784182Z",
     "start_time": "2017-08-03T17:40:40.780180Z"
    }
   },
   "outputs": [],
   "source": [
    "def print_class0_proportion(target_column):\n",
    "    class_0, class_1 = pd.Series(target_column).value_counts()\n",
    "    class_0_pct = round(class_0 / target_column.shape[0] * 100, 2)\n",
    "    print('Class 0 represents {0}% of all the observations.'.format(class_0_pct))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.802186Z",
     "start_time": "2017-08-03T17:40:40.785181Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class 0 represents 99.83% of all the observations.\n"
     ]
    }
   ],
   "source": [
    "print_class0_proportion(df['Class'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data set is highly unbalanced. Only a very small proportion (0.17%) of the transactions are fraudulent while the majority of the examples are normal. In the real world, this makes sense. However, training a classifier with unbalanced data results in predictions that are biased towards predicting the majority class. Therefore, we must re-sample the data set so both classes have approximately 50-50 ratio. We can either:\n",
    "\n",
    "- Undersample the majority class or\n",
    "- Oversample the minority class\n",
    "\n",
    "Undersampling the majority class is better for this data set because we want to reduce the number of examples to get kNN to perform well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2017-08-03T12:45:37.748Z"
    },
    "collapsed": true
   },
   "source": [
    "We use the *imbalance-learn* library from *scikit-learn-contrib* project. Remember to install it: `conda install -c glemaitre imbalanced-learn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:40.829193Z",
     "start_time": "2017-08-03T17:40:40.803186Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df.drop('Class', axis=1)\n",
    "y = df['Class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:41.124274Z",
     "start_time": "2017-08-03T17:40:40.830192Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from imblearn.under_sampling import RandomUnderSampler\n",
    "sampler = RandomUnderSampler(random_state=42)\n",
    "X_resampled, y_resampled = sampler.fit_sample(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the ratio should be around 50 percent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:41.130776Z",
     "start_time": "2017-08-03T17:40:41.125775Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class 0 represents 50.0% of all the observations.\n"
     ]
    }
   ],
   "source": [
    "print_class0_proportion(y_resampled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Partition the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T10:06:00.108432Z",
     "start_time": "2017-08-03T10:06:00.104430Z"
    }
   },
   "source": [
    "Now we are ready to partition the data into training set and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:41.138777Z",
     "start_time": "2017-08-03T17:40:41.132276Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_resampled_train, X_resampled_test, y_resampled_train, y_resampled_test = train_test_split(X_resampled, y_resampled, test_size=.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T10:09:18.757040Z",
     "start_time": "2017-08-03T10:09:18.726530Z"
    }
   },
   "source": [
    "### Pick an Appropriate $k$ Value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An appropriate $k$ value for the kNN algorithm is the one that yields the lowest error or the highest score. We can use 10-Fold Cross Validation to estimate the score associated with each $k$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:41.149281Z",
     "start_time": "2017-08-03T17:40:41.140278Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:42.353076Z",
     "start_time": "2017-08-03T17:40:41.150781Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x19600967fd0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmoAAAEHCAYAAAAeUGpmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4XNW18P/vFPVeRl2yXLct2ZbcbcDYgLEBJ5SATULC\nTX0DqZQkN7fk5l5ubr+/0N6QhJDGmwRCTxxMbGOaAfciyZbsY8u2ei9Wb1N+f4wkC6OumTlT1ud5\neB50Zs6ctSVZs2bvs9cyOBwOhBBCCCGE9zHqHYAQQgghhBidJGpCCCGEEF5KEjUhhBBCCC8liZoQ\nQgghhJeSRE0IIYQQwkuZ9Q7AHRobO6a8lTUuLpzW1m53hOMTAnn8MvbAHDsE9vgDeewQ2OOXsXvf\n2C2WKMNYj8mM2iCz2aR3CLoK5PHL2ANXII8/kMcOgT1+GbtvkURNCCGEEMJLSaImhBBCCOGlJFET\nQgghhPBSkqgJIYQQQngpSdSEEEIIIbyUX5bncLdDJfXsPFBGTVM3aYnhbF2XzZqcZL3DEkIIIYSf\nkURtig6V1PP0juLhr6sau4a/lmRNCCGEEK4kS59TtPNA2RjHyz0ahxBCCCH8nyRqU1TTNHpF49rm\nLg9HIoQQQgh/J4naFKUlho96PDUhwsORCCGEEMLfSaI2RVvXZY9xfJZnAxFCCCGE35PNBFM0tGFg\n54Fyqho7AbjtmtmykUAIIYQQLieJ2jSsyUlmTU4yWkUr//3cieGETQghhBDClWTpcwYWZMYyKzmK\n42cbabjUo3c4QgghhPAzkqjNgMFgYPPqTBwO2Hu0Uu9whBBCCOFnJFGboVULk4iLCuH9olq6ewf0\nDkcIIYQQfkQStRkym4xsWpFBX7+N9wpr9A5HCCGEEH5EEjUXuDY/jZAgE3uPVmG12fUORwghhBB+\nQhI1F4gIDeKapam0dvRxVGvQOxwhhBBC+AlJ1FzkxpUZGIA9hytxOBx6hyOEEEIIPyCJmoskxYWz\nfIGFsroOzlZe0jscIYQQQvgBSdRcaPPqTAD2HJFSHUIIIYSYOUnUXGheegyzU6MpONdEfUu33uEI\nIYQQwsdJouZCBoOBLaszcQBvSgFcIYQQQsyQJGoutkJZSIgO4YOTtXT2SAFcIYQQQkyfJGouZjIa\n2bQyk/4BO+8VVOsdjhBCCCF8mCRqbrB+aRqhwSb2HpMCuEIIIYSYPknU3CA81My1eWm0dfZz+HS9\n3uEIIYQQwkeZ3fXCSikj8FMgD+gDvqJpWumIx+8Fvge0Ab/VNO1XSikT8AygAAdwv6Zpp5RS+cD/\nBWyDr/U3mqZ5dQa0aWUGbx6tZPfhStblpmAwGPQOSQghhBA+xp0zarcDoZqmrQP+Dvjx0ANKqUTg\nR8BGYAPwWaVUNvBJAE3TrgZ+APz74ClPAN/SNG0j8CrwfTfG7RKJMWGsVElUNnRyprxV73CEEEII\n4YPcmahdA+wC0DTtILByxGNzgEJN01o0TbMDR4C1mqb9Cfjq4HNmAUMl/j+taVrB4P+bgV43xu0y\nQwVwd0sBXCGEEEJMg9uWPoFonMuaQ2xKKbOmaVbgHJCrlEoGOoAbgLMAmqZZlVLPAncAdw0eqwVQ\nSl0FfBO4drwLx8WFYzabphywxRI15XMmer1F2RcpOt9Mrx0yk137+q7m6vH7Ehl74Ark8Qfy2CGw\nxy9j9x3uTNTagZHfDeNgkoamaa1KqYeAV4Bm4DjQNPRETdM+r5T6PnBIKZWjaVqXUupu4B+BrZqm\nNY534dbWqXcFsFiiaGzsmPJ5E7l+WRqny1p4Yc8ZPn/TQpe/vqu4a/y+QMYemGOHwB5/II8dAnv8\nMnbvG/t4yaM7E7UPcd5z9qJSai1wcugBpZQZWA6sB4KBN4F/GNxgkKFp2n8C3YAdsCulPgfcB2zU\nNK3FjTG73LL5FhJjQtl/qo47rp1DdHiw3iEJL3eopJ6dB8qoaeomLTGcreuyWZOTrHdYQgghdODO\nRO014Eal1H7AAHxRKXUPEKlp2i+UUuCcSesFfqxpWpNS6lXgN0qpfUAQ8CDQDzwJVACvDp73nqZp\n/+zG2F3GaDRw46pMnt97jndPVHPr1bP1DukjhpOC5m7SEiQp0Nuhknqe3lE8/HVVY9fw1/JzEUKI\nwOO2RG1wk8D9Vxw+M+LxR4BHrjinC9g+ysvFuzxAD7pmSSp/ev8ibx+r4uY1WQRN4/45d5CkwPvs\nPFA2xvFy+ZkIIUQAkoK3HhAWYmZjfhrt3QMcLPGe8m/jJQVCHzVNo99fWdvc5eFIhBBCeANJ1Dzk\nhhUZmIwG9hypxOFw6B0OADVNo7/5S1Kgn7TE8FGPpyZEeDgSIYQQ3kASNQ+Jjw5l1cIkqhu7KC7T\nfz9Ee3c/RuPo3RIkKdDP1nXZYxyf5dlAhBBCeAVJ1DzoxlXOArh7DutbALe338rjLxZitY0+sydJ\ngX5WLUoiLMSE0QBDXceuWpwi96cJIUSAkkTNg2anRrMgM5ZTF1uoauzUJQarzc5Tr52irK6Da5ak\n8tVP5hAS5NzckJ4YwX235kpSoKPyug56+mysy03h8W9dQ5DZyNnKS9jsdr1DE0IIoQNJ1Dxsy2Bb\nqT06tJWyOxz8eudpii+2kDc3gc/frFibm8J1y9IBuHeLkiRNZyfOOes+589PJCo8mGuWpNLU1sux\n8Ws8CyGE8FOSqHlY3rxEkuLCOFhcR1tXv8eu63A4eOGtUg6W1DM3PZr7b1+Myej88WckOe9Jq2zQ\nZ5ZPXFZY2oTZZCB3trMizeZVmRiA3YcrvGYTihBCCM+RRM3DjAYDm1dlYrU5eOd4lceuu+tQBW8e\nrSQ1IZwH7sobXu4EyExytq6QRE1fzW29VDZ0sjArjtBgZ4nD5Phwli2wcLG2g7OVl3SOUAghhKdJ\noqaDqxenEhFq5u3j1fQP2Nx+vQ9P1vLSu+eJiwrhO3fnExkW9JHHUxPCMZsMut03J5wKzzuXPfPm\nJX7k+E2rswDYrfMmFCGEEJ4niZoOQoJNbFyWTmfPAPuL69x6rcLSJn7zxhkiQs08vD2P+OjQjz3H\nbDKSkRRFVWMndrssr+mlYOj+tCsStXkZMcxNj6agtElq3AkhRICRRE0n1y93FsB980gldjfde3S+\nuo2f/ekUZpOBB+7KI90SOeZzZ6dF0z9gp+FSj1tiEePr6bNypqKVzKRIEmI+nkzLrJoQQgQmSdR0\nEhcVwpqcZGqbuzl1odnlr1/T1MXjLzlrpd1/22LmZcSM+/zZac7Hq+Q+NV2UlLVgtTk+tuw5ZNl8\nC0mxYew/VUe7BzehCCGE0JckajraPFgA19WzJC3tvTz6YgFdvVY+f7Mif/7ob/4jZadGA1AhiZou\nhpY9l43xszIaDWxenYnVZudtD25CEUIIoS9J1HSUlRzFollxnC5vpaK+wyWv2dkzwKMvFtLS3sed\nG+awfmnapM6TGTX92O0OCs83ExMRzKyUqDGfd/WSVCLDgnj7eDV9HtiEIoQQQn+SqOnMlQVw+wZs\nPPlKETVNXWxamcEtayffCio2KoSYiGAp0aGDCzXtdPYMkDcvAaNh9P6rACFBJq4b2oRystaDEQoh\nhNCLJGo6WzwngdSEcA6V1NPa0Tft17HZ7Tz952JKq9pYk5PMp2+Yj2GcN/3RZCRF0tzeS3fvwLTj\nEFN3otTZdSB/nmXC516/IgOzycjuI5WyQ1cIIQKAJGo6MxoM3LgqE5vdMe17jxwOB8/u0igobSIn\nO44vb1007szMWDKTnLtCZVbNswpLmwkyG1mUHTfhc2MigrlqcTINrT3D7aaEEEL4L0nUvMBVuSlE\nhgXx7olq+vqnfu/Rq/su8EFRLbNSovjGHUswm6b3Yx1K1KoapVaXpzRc6qGmqYucWXEf6RYxns2r\nhkp1VLgzNCGEEF5AEjUvEBxk4vrl6XT1Wvnw1NTuPdp7tJKdB8pJigvjoW15hIWYpx1HpmVoRs01\nGxvExAoHZ8XyJrEzd0haYgR5cxMorW6jtKrNXaEJIYTwApKoeYnrlmdgNhnYM4UCuIdP1/P83nNE\nRwTz8N35REcEzyiGlIRwTEaDLH16UEHpYKI2d/KJGsBNa2RWTQghAoEkal4iJiKYtbkpNLT2UFg6\n8b1HxWUtPPOXEkKCTTy8PY+k2LAZx2A2GUlLjKC6sUtuVPeA7l4rZysvkZ0SRVxUyJTOXZAZS3ZK\nFMfPNlLf2u2mCIUQQuhNEjUvsmWSBXDL6zr4yasnMRjgW3cuJSt57NpbU5WZFEm/1S5v/h5w6mIz\nNrvjY709J8NgMHDTmiwcuKa0ixBCCO8kiZoXSbdEsnh2PGcrL3Gxtn3U59S3dvPYiwX099v46idz\nWTRr4p2CUyEbCjxnuAn7FO5PG2mFspAQHcqHRbV0dEtbKSGE8EeSqHmZzYMFcN8cZZakrbOPR18o\noL17gM9tXsDKhUkuv35Gkmwo8ASb3c7JC83ER4cMJ8dTZTIa2bwqk36rnXdOVLs4QiGEEN5AEjUv\nk5sdT7olgiNnGmhp7x0+3tNn5bEXC2m81MutV2dz3fIMt1x/uJZavWwocKfSqja6eq3kzU2ccmHi\nka5Zmkp4iJm3jlUxYJW2UkII4W+mX8tBuIXBYGDzqkx+88YZ9h6rYvt18xiw2vm/rxRR0dDJhvw0\nbrtmttuuHx0eTExkMFWNkqi504kZLnsOCQsxs3FZOm8cLGf/qTo25Ke7IjyXOVRSz84DZdQ0dZOW\nGM7WddmsyUnWO6wp85dxCCF8jyRqXmhtTgp/fOscuw9XsOdwBSHBJnr6bCxfYOHezWpGMzCTkWmJ\n5NTFFrp6B4gIDXLrtQJVYWkTIUEmFmbFzvi1bliRwe7DFew+XMn6vLRpdaVwh0Ml9Ty9o3j466rG\nruGvfSnJ8ZdxCCF8kyx9eqHjZxvp6bPhcIDdAT19ziWt5QsSMRrd/yY8vKFA6qm5RW1zF/WtPeTO\njifIPLluBOOJiwphbW4ydS3dFJU2uyBC19h5oGyM4+UejWOm/GUcQgjfJImaFxrrjWHXIc+UYZCe\nn+5VOJhMTacsx1i2rHYWwN3lRQVwa5pGL/FS2+xbO4r9ZRxCCN8kiZoX0vuNIUMSNbcqONeIAVg6\nN8Flr5lhiWTxnPFLu3iSw+EgLGT02cLUhAgPRzMzKQnhox5PHeO4EEK4kiRqXigtcaw3Bs+8waXE\nh2M2SSspd+jsGeBcdRtz0qNn3PLrSjet9p62Un/+4CJdvdZRH7tuuXdteBjPgNWGY4yWbsFmo3Tw\nEEK4nds2EyiljMBPgTygD/iKpmmlIx6/F/ge0Ab8VtO0XymlTMAzgAIcwP2app1SSs0Dfjt47BTw\nDU3T7O6KXW9b12V/5Obly8dneeT6w62kmpytpDxxX1ygOHm+GYfDtcueQxbNiiMrKZIjZxq4a0MP\niS5oKzYdbx+vYseHZVhiQ7lpdRbvnKihtrmL8FAzHd0DvFdQzdqcZMJCvHsvk93u4Bc7Sqht7mZ2\nahQDVju1zd2kJoRjtTu4UNvBc3vP8tkbF7h9g48QInC58y/l7UCopmnrlFJrgR8DtwEopRKBHwHL\ngUvAXqXUW0A+gKZpVyulNgL/PnjOo8APNE17Vyn188Fjr7kxdl0N7STbeaCc2uYuUhMi2Lpulkd3\nmGVaIqmo76S+tdvnlqq82YnBPq7uSNQMBgNb1mTxzF9K2HO0kns2LXD5NSZy9EwDf9hzlujwIB6+\nO5/kuPDhmn8Oh4Nnd51hX2EtP3n1JA9uyyPI7J2T+g6Hg9/v0Th2tpGFWbE8tD3vIxs/unsH+K8/\nHOft49XERATzyavdVzJHCBHY3JmoXQPsAtA07aBSauWIx+YAhZqmtQAopY4AazVN+6NS6vXB58zC\nmcQBrADeG/z/vwKbGSdRi4sLxzyN3XQWi+t6Zs7UJzZE8YkN8zx6zZHjXzgnkQ9P1dHWY2OpF31f\n3MUTP/sBq53iiy0kx4eTtyjFLbMwt6yP4LV9F/igqJYv37aEyPCJl1ddNfai0kZ+8ZcSQkNMPHLf\nVczL+HjpkYc/u5I+6xEOFdfxuzfP8r3PrdR9xna08T+/+wzvFtQwOy2af/nqVUSEfbxMzb997Wr+\n9v++z2vvXyQ9JZota7M9EK1redPfPD0E8vhl7L7DnYlaNM5lzSE2pZRZ0zQrcA7IVUolAx3ADcBZ\nAE3TrEqpZ4E7gLsGzzVomjZ0M0gHEDPehVun0VDcYomisTFw2yZdOf74cOevRvH5RhZmROsVlkd4\n6mdfXNZCT5+Vqxen0NTkvvv/rl+ewYvvlPLyXo2t67LHfa6rxl5e18F/P3cch8PBN+5YSkyIaczX\n/eJNipa2Hj4orCHEfJx7Ns3XbelwtPG/c6Ka5/ZoJMaE8q1PLaG7s5fuzt5Rz39wWx7/8btjPPVy\nIdjsLF9g8UTYLiF/8wJ3/DJ27xv7eMmjO9cd2oGRVzYOJmlomtYKPAS8AjwPHAeahp6oadrngQXA\nM0qpCGDk/WhRXJ5pE24iOz9db6ZN2Cfr2rw0QoNN7D1WxYDV/bdyNlzq4bGXCunrt/F/PplDTnb8\nuM8PDjLx7buWkm6J4K1jVbxx0HvqkR3TGvj9Ho2o8CC+c3c+sZEh4z4/JT6cB7flEWw28fSOYs5W\nyp8mIYRruTNR+xC4BWDwHrWTQw8opcw4709bD2wHFgIfKqXuVUr9/eDTunEmaHbgxOA9awA3A++7\nMW4BRIUHExsZLImaizgcDgpLmwgLMbEgc+bdCMYTHmpmQ34abZ39HCqpd+u12rv6efSFAtq7+vnM\npvmsXjS5+ygjQoN4eHs+CdEhvPLeBfYV1rg1zsnQKlp5ekcJwUEmHtqeR3L85MpvzEmL5ht3LMZu\nd/Dky0VSKFoI4VLuTNReA3qVUvuBx4CHlFL3KKW+OjSzhnMm7V3gSU3TmoBXgWVKqX3AbuBBTdN6\ngO8AjyilDgDBwMtujFsMykyKorWjj86eAb1D8XnVTV00tfWyeHYCZpP7b6C/cWUmJqOB3Ycrxiwv\nMVM9fVYee6mQhtYePnHVLDatzJzS+XFRITx8dz6RYUE8u+sMJ841uiXOyaio7+DJV4pwOBx8844l\nZKdMbbl/8ZwEvrR1Ed19Vh59sYCmth43RSqECDRuu0dtsHzG/VccPjPi8UeAR644pwvnDNuVr3UW\n2OCGMMU4MpIiOHmhmaqGThbOitM7HJ9WWOqZZc8h8dGhrFqUxMHiek5dbGHJHNcV1wWw2uw89dpJ\nyus6WL80lTvWz5nW66QmRPDAtqX87/Mn+Pmfi/nup/OZP8omBHdqvNTDYy8W0tNn475bc8mdPf7S\n7VjW5abQ3tXPC2+X8ugLhfz955YTNYnNHEIIMR7v3BsvvIK0knKdgnNNGA0GlydM49myarCt1CHX\nFsC1Oxz88vUSSspayZ+XyN/cpGa0GWBuWgxfv30JdruDJ14qorrRc79vbZ19PPpCAW1d/Xzmhvkz\nLoGzZXUWN63Joq6lm8dfKqKv3+aiSIUQgUoSNTGmzCTnXpBKD75x+qP2rn4u1LQzLyOGyFHKPLjL\nrJQoFs2K43R5K+V1rtnl5HA4+OPecxw+3cC8jBjuuy0Xk3Hmf0aWzk3gCzcvHFw6LKS5bfRdlq7U\n22/lX355kPrWHm5ZO4sbV01t6XYsd22cy1WLU7hY285TfzqJ1ea3tbmFEB4giZoYU0p8GGaTUWbU\nZqjwfBMO3FPkdiI3rRlsK3XENbNqbxwsZ++xKtITI/j2nUsJCZp6vcKxXL0kle3XzaO1o49HXyxw\n672RzqXbU5RWXuKaJancuWF6S7ejMRoMfOHmhSyZk8CpCy385o3T2N10n6AQwv9JoibGZDIaSU+M\noLqxC5tdZgWmy1NlOUazeHY86ZYIDpc00NI+s1mq94tqeOW9C8RHh/DQ9jy3zA7etCaLLaszqW3u\n5onBkh+uZnc4+PXO0xRfbGFVTjKfv3lmS7ejMZuMfP32xcxJi+ZAcT0vv3Pepa8vhAgckqiJcWUk\nRWC12alvkV1s0zFgtVFc5uxGkDLJcg+uZDAY2LIqC7vDwZtHK6f9OgXnmnj2rxoRoWYe3p5PfHSo\nC6P8qG3XzWNdbjLna9r52Z9PuXTp0OFw8OLbpRwsqWduejR/e+9KlyzdjiYk2MSD2/JITQhn1+EK\nl98rKIQIDJKoiXEN36cmy5/Tcrr8Ev0DdvLneW4TwZXW5CQTExnMewU1dPdaJz7hCqVVbfzsz6cw\nmww8uC2PtET39n41Ggx88ZZFLJ4TT9H5Zp796xmXlRjZdbiCPUcqSU0I54G78ggNdm9j+MgwZ724\nuKgQXnynlP2nat16PSGE/5FETYxraOdnlWwomJZCNzZhn6wgs5FNKzLo7bdNubBsdWMnT7xciM3m\n4Ot3LGZu+rjd21xmaOlwdmo0H56q4+X3Zr50+OHJWl565zxxUSF8Z7B+myckxITy0PY8wkPM/OaN\nM5y80OyR6woh/IMkamJcUqJj+hwOBwWlTUSEmpmX4ZkEZywbl6UTEmTizaOVk15KbGnv5dEXC+nq\ntfLFWxaydK5nk83QYDMPbltKSnw4fz1YwZ7D0186LCxt4jdvnHEu3d7t3qXb0WRYIvn2XUsxGg08\n9dpJzte0TXySEEIgiZqYQGRYEHFRIZKoTUNFfSetHX0smZvgtvugJisiNIj1eam0dvRx5EzDhM/v\n7Bngxy8U0NrRx7aNc7l6SaoHovy4qPBgHr47j5jIYP74dikHiuum/Brnq9v42Z+cS7cP3JVHupuX\nbseyIDOW+2/LZcBq54mXiqht7tIlDiGEb5FETUwoMylSWklNgzcse460eWUmBgPsPjR+W6m+ARtP\nvFxIbXM3m1dlDpf40EtiTBgPb88nLMTMr3ee5tTFyS8d1jR18fhLhVhtDu6/bbHuM5vL5lv4/E0L\n6ewZ4NHBRFgIIcYjiZqYUIZFlj+n40RpEyajgcWz9dtIMFJibBirFiZR0dDJ6fLWUZ9jtdn52Z9O\ncb66nbU5yWy/fp7LS1dMR2ZSJN++cwkGg4GnXj3Fxdr2Cc9xLt0W0NVr5fM3K13Ko4zm2rw07rh2\nDs3tznpx3b3yAUgIMTZJ1MSE5D61qWvt6KO8roMFmbGEh7p3Z+FUbFk92FZqlPu9HA4Hz+46Q9H5\nZnJnx/OlrYswekGSNkRlxXH/bbn0W2089mIhdS3dYz63s2eAR18spKW9jzs3zGH90jQPRjqxT6yb\nxQ3LM6hu7OLJl4voH5BWU0KI0UmiJiY0vPNTErVJKzzvXcueQ2anRqMyYzl1oeVjO3lfee8CH56s\nY3ZqFN+4YzFmk/f9eVi+wMK9W9Tw0uGlzo8vHfYP2HjylSJqmrrYtDKDW9bO0iHS8RkMBj6zaT6r\nFiZxtqqNp3cUS1FpIcSovO8vsfA6ydJKasqGuhHkecly20hDs2q7R8yq7TlSyRsHy0mOC+OBbe6v\nLzYTG/PTuX39bJraenn0hcKP1Iaz2e38/M/FlFa1sSYnmU/fMN8rlm5HYzQa+Monclg0K44T55r4\n3e6zLqsXJ4TwH97711h4DZPRSLrlcispvXcweru+ARuny1tJT4wgKTZM73A+Zum8BGIigvnwZB23\nfXcHsZHBtHT0ERMZzHfuzic6PFjvECf0yauyaevq553j1fz7745iNBiobe4mNMREd6+VnOw4vuxl\nS7ejCTIb+eanlvDfzx1nX2ENnT39NLT2UNPUTVpiOFvXZbMmJ1nvMIUQOpJ3XDEpmUmRWG126qSV\n1IRKyloYsNrJ87JlzyFHTjfQ1tUPOPtetgzuPLxxZSaJXphYjsZgMPDZTQuYkxpFbXM31U1d2B2O\n4dm1NTnJXrl0O5qwEDMPbc8nOjyI42ebqGp0jqWqsYundxRzqKRe7xCFEDryjb9kQneZwzs/O3SO\nxPsNl+XwwmVPgJ0HykY9frDYtxICo9FA38Do93W9eaTKw9HMTExEMOGho3dK2Hmg3MPRCCG8iSRq\nYlJk5+fk2B0OCkqbiQoPYk5qtN7hjKqmafTdkr5YgLW22X/G0tA6+my1L45FCOE6kqiJSckY3vkp\nbxrjKavtoL2rn6VzEzAavfP+qLTE8FGPpyboU7F/JmQsQgh/J4mamJTLraRk6XM8BcPdCCw6RzK2\nreuyxzjufWUsJhIIY7lumXfVgBNCeNakd30qpbKBXGAXkKVp2kV3BSW8U2ZSJEXnm+no7ifKB3YG\n6qHgXBNmk4Hc2XF6hzKmoV2EOw+UU9vcRWpCBFvXzfLJ3YX+PJaI0CDau/vZV1jLusUpXl0yRQjh\nPpP6l6+Uuhv4ARAOrAMOKKW+q2na790ZnPAuQ4laVUMni7Lj9Q7H6zS19VDV2MniOfFe/6a6JieZ\nNTnJWCxRNDb69izp0Fj8wcixOBwOfvvXM7xfVMvP/1zMt+5cIqVxhAhAk/1X/33gKqBd07QGYBnw\n926LSngl2VAwvsJSZ7Nwb+tGIHyTwWDg3i2KxbPjKTrfLAVxhQhQk03UbJqmDX/s1jStFpB+JwFG\nErXxXb4/TRI14Rpmk5Gv3b6YrORI9hXW8LqU6hAi4Ew2UStWSn0TCFJK5SulfgEUuDEu4YWS48IJ\nMhupbJRE7Uo9fVa0ilaykiKJjw7VOxzhR8JCzDy4LY+E6BBe23eB/adq9Q5JCOFBk03UvgGkAz3A\nr4F24OvuCkp4J6PRQHpiBDVNXVhtMqE6UvHFFqw2h9d2IxC+LTYyhIe25xMeYuY3b5yhuKxF75CE\nEB4y2UTtJ5qm/b2maas0TVuuadp3Ry6FisDhbCXloK5l9EKjgcrbuxEI35eWGMG37lyCwQBPvXpS\nbkEQIkBMNlFbrJSKdGskwidcLnwrbxJD7HYHheebiYkMZlZKlN7hCD+msuL48tYcevttPP5SIS3t\nvXqHJIRws8kmanagQil1QCn19tB/7gxMeKcs2VDwMedr2ujsGSBvbiJGg3d2IxD+Y01OMtuum0tr\nRx+PvVQ43IheCOGfJlvs6W/dGoXwGUMzarKh4LICWfYUHnbT6ixa2vp463gVT712koe252E2SY01\nIfzRpBLHlrEcAAAgAElEQVQ1TdPeU0rdDNwweM47mqb9ebxzlFJG4KdAHtAHfEXTtNIRj98LfA9o\nA36radqvlFJBODcrZAMhwL9pmrZDKZUP/BywAmcHX0vuZtdBRGgQ8dEhMqM2QsG5JoLNRnJmeW83\nAuFfDAYDn9k0n5aOXk6ca+I3b5zmK5/IwSAzukL4nUl9BFNK/S3wL0AFcBH4R6XUP0xw2u1AqKZp\n64C/A3484vUSgR8BG4ENwGcHW1R9DmjWNG09cBPwk8FT/hn4V03TrsGZwG2dTNzCPTItkbR19tPe\n3a93KLpraO2mtrmbnOx4goNMeocjAojRaOCrt+YyNy2aA8X1vLrvgt4hCSHcYLJz5Z8DNmqa9qSm\naU/gTLDuneCca3D2BUXTtIPAyhGPzQEKNU1rGZwZOwKsBV4C/mnwOQacM2gAJ4B4pZQBiAIGJhm3\ncIPMZNlQMKRgqBuBLHsKHYQEmfjWXUtJigtj54Fy3i2o1jskIYSLTfYeNaOmaT0jvu7lchI1lmic\ny5pDbEops6ZpVuAckKuUSgY6cC6pntU0rRNAKRUFvIyzvyiDz39q8Os24N3xLhwXF47ZPPXZDYsl\nsHfsTXb8OXMtvL6/nNbuAb/5nk13HCXlrQBct3qWzxa69Zef4XT5+vgtwI/uv4rvPfk+v9+tkZ0e\ny6qclMmd6+Njn6lAHr+M3XdMNlF7Syn1CvDbwa+/AEy067Md5+zXEONgkoamaa1KqYeAV4Bm4DjQ\nBKCUygReA36qadpzg+c+AazXNK1YKfUNnMuo3xjrwq2tU6/x5Q/NqWdiKuOPCXUmwacvNHO1HzTD\nnu7Pvrt3gOILzcxOjcLWN0Bjo+9N9MrvvX+MPwj41p1L+N/nTvBf/+8I379nObNTo8c9x1/GPl2B\nPH4Zu/eNfbzkcbJLnw8Ce4G/wZmkvQV8Z4JzPgRuAVBKrQVODj2glDIDy4H1wHZgIfDh4AzbHuD7\nmqb9esRrteBM/ABqALlrW0fJceEEm40Bv/R58kILNrt0IxDeYW5aDPfdmsuA1c4TLxXSeKln4pOE\nEF5vsolaBM4ZsW3At4EUIHiCc14DepVS+4HHgIeUUvcopb46NLOGcybtXeBJTdOagH/AmYT9k1Lq\n3cH/woCvAH9USr2Hs3XVRBsZhBsZjQbSLRHUNAd2K6lCacIuvMyyBRbu2bSA9u4BHnuxkM4e35vl\nFUJ81GSXPp8Digb/vwNngvc74M6xThjcJHD/FYfPjHj8EeCRK855AHhglJf7ALh6krEKD8hMiuRi\nbQd1zd3DtdUCidVmp+h8M/HRIWQG4PiF97phRQbN7b3sOlTBk68U8d2782VHshA+bLIzarM0TfsB\ngKZp7YP/P9d9YQlvl5nkXE8P1MK3pVVtdPdZyZuXKLWrhNe5a+NcVi9KorSqjV++XoLd4dA7JCHE\nNE02UXMopZYMfaGUWoiUyAhoGZYIIHBbSQ11I1gmy57CCxkNBr68NQeVGctRrZEX3y6d+CQhhFea\n7NLnd4E3lVJVg19bcNZWEwEqM4B7fjocDgpKmwgJNqGyZF+L8E5BZiPfvHMJ//n74+w5UklCdCg3\nrsrUOywhxBRNOKOmlPoEcAHIAl7AufvyBeCAe0MT3iw8NIiE6NCA3PlZ19JNQ2sPi7PjCTJLf0Xh\nvSJCg3hw21JiIoL541vnOHqmQe+QhBBTNO67jFLquzjbN4XiLKHxLzg3FpiB/8/dwQnvlpkUSVtX\nP+1dgdNK6lBJPf/z3AkALtS2c6ikXueIhBhfYkwYD27LIzjYxDOvl3Cu6pLeIQkhpmCipc97gXWa\npnUrpf4L2KFp2i8HWzmVuD884c0ykiIpKG2isqGT3NnxeofjdodK6nl6R/Hw160dfcNfr/GDwr/C\nf81KieIbty/m8ZeKePSFAmKjQmi81EtaQjhb12XL768QXmyidRuHpmlDZf6v43LvTtlCJMgKsPvU\ndh4oG+N4uUfjEGI6Fs9JYP3SVPoG7NS39GC3O6hq7OLpHcUyMyyEF5toRs2qlIoFIoFlOLsGoJSa\nxcS9PoWfywiwRK2mafTWZLXNXR6ORIjpOV/TNurxnQfKZVZNCC810YzafwEFwEHgl5qm1SqltuNs\nIfU/7g5OeLek2DCCg4wBk6ilJYaPejw1IcLDkQgxPfJhQwjfM26ipmnay8BVwC2apn198HAn8BVN\n037n7uCEdzMaDWRYIqkNkFZSN63JGvX41nWzPByJENMjHzaE8D0T1lHTNK0GZyP0oa/fcGtEwqdk\nWCK5UNNObXO337dSiokIASAyLIiePiupCRFsXTdLloyEz9i6LvsjG2IuH5cPG0J4q8kWvBViVJcL\n33b4faI21I3ga7flsijb/3e5Cv8z9KFi54Fyapq7sNsdZKdEyYcNIbyYJGpiRoaSs6oG/77HxeFw\nUHCuibAQM/MzY/UOR4hpW5OTzJqcZBITI3ngx+9SVttOXUs3KfGjL4sKIfQlZdXFjGRYLs+o+bPq\nxi6a23tZMices0n+2QjfZzAYuGlNFg5gz5FKvcMRQoxB3nHEjISHmkmMCaWy0b9n1IaWPfOlCbvw\nI8sXJJIYE8qHJ2tp7w6cDiNC+BJJ1MSMZVgiae/qp82PW0kVljZhNBhYMjdB71CEcBmT0cjmVZkM\nWO28c7xa73CEEKOQRE3M2MgNBf6oraufCzXtzM+IISI0SO9whHCpa5amEhFq5q1jVfQP2PQORwhx\nBUnUxIz5+4aCotImHECeLHsKPxQabGbjsnQ6ewbYf6pO73CEEFeQRE3MmL/PqA3dn7ZsviRqwj/d\nsCIDs8nA7sMV2B3SylkIbyKJmpgxS1wYIUEmv2wlNWC1UVzWQkp8OMlSvkD4qdjIENbmplDf2kPh\nuSa9wxFCjCCJmpgxo8FAhiWC2uZuv2sldbq8lf4Bu+z2FH5vy6pMAHYdrtA5EiHESJKoCZfISIrE\nZndQ0+Rf96kVlDYDkC/LnsLPpVsiWTo3gXNVbZyvbtM7HCHEIEnUhEtcvk/Nf5Y/HQ4HhaVNRISa\nmZserXc4QrjdltVZAOyWWTUhvIYkasIlhnd+NvpPolZR30lrRx9L5yZgMso/FeH/FmbFMis5imNn\nG2m41KN3OEIIJFETLnK5lZT/JGrD3QjmW3SORAjPMBgMbFmTicMBb0pbKSG8giRqwiXCQgZbSTV0\n4vCT7f0FpU2YjAYWz47XOxQhPGalSiIhOoT3i2ro7BnQOxwhAp4kasJlMpMi6egeoN0PWkm1dvRR\nXteByoolLMSsdzhCeIzZZOTGlZn0D9h594S0lRJCb5KoCZfxpw0FhYPLntKNQASi9XlphIWY2Xus\nigGrf5XcEcLXSKImXMafErXh+9MkURMBKCzEzIb8NNq7+jlYLG2lhNCTJGrCZYYTNR/f+dnXb6Ok\nrJV0SwSW2DC9wxFCF5tWZGAyGtglbaWE0JUkasJlEmPDCAn2/VZSJWUtWG3SjUAEtvjoUFYvSqa2\nuZtTF5r1DkeIgOW2u6SVUkbgp0Ae0Ad8RdO00hGP3wt8D2gDfqtp2q+UUkHAr4FsIAT4N03Tdiil\nkoBngDjABPyNpmnn3RW7mJ6hVlJltR0MWO0EmX3zc4AsewrhtGV1JgeK69h1qIKlc+XfgxB6cOc7\n6e1AqKZp64C/A3489IBSKhH4EbAR2AB8VimVDXwOaNY0bT1wE/CTwVP+B/iDpmnXAj8AFroxbjED\nmUlR2OwOapt9s5WU3eGg8Hwz0eFBzE6TbgQisGUlR5GbHceZikuU1bXrHY4QAcmddQeuAXYBaJp2\nUCm1csRjc4BCTdNaAJRSR4C1wEvAy4PPMQDWwf+/GihSSu0FyoAH3Bi3mIFMSwTg3FCQlRylczRT\nd7G2nfaufq5ZkorRYNA7HCF0t2VNFsVlrew+XMl9t+bqHY7HHSqpZ+eBMmqauklLDGfrumzW5CT7\n7HWE73FnohaNc1lziE0pZdY0zQqcA3KVUslAB3ADcFbTtE4ApVQUzoTtB4PnZgOtmqZtUkr9EPg+\n8MOxLhwXF47ZbJpywBaL7yUWruSK8S9ZkAx7ztLU0e9T38+hWHcdrQLg2hUZPhX/TATKOMcSyOOf\nzNg3Jkby6r6LHDnTwFfvWEpSfLgHIvOMica/70QVT+8oHv66qrGLp3cUEx0dyrXLMlwWh6euM5L8\n3vsOdyZq7cDI74ZxMElD07RWpdRDwCtAM3AcaAJQSmUCrwE/1TTtucFzm4Edg///F+Dfx7twa2v3\nlIO1WKJobOyY8nn+wlXjjwhyzkKdLW/xme/nyLHvL6zBbDKSER/mM/HPhPzeB+74pzL2G5an86ud\np3lhzxk+fcN8N0fmGZMZ//O7z4x6/PE/nuClvWddFkt10+i3ijy/W2NRRozLrjNEfu+9b+zjJY/u\nTNQ+BD4JvKiUWgucHHpAKWUGlgPrgWDgTeAfBmfY9gDf1DTtrRGv9QFwC/A74FqgGOGVwkLMWGIv\nt5Iy+NDyYVNbD1WNnSyZk0BosHQjEGLImpxkXnnvPO8V1nDr1dmEhwbpHZJH1DSN/qF/wGqnxoX3\n4Y5VVNhX7/UVruXOd6PXgBuVUvtx3m/2RaXUPUCkpmm/UEqBcyatF/ixpmlNSqkncO7s/Cel1D8N\nvs7NwHeAXyqlvoZzOfUeN8YtZigzKYrjZxtp6+onNjJE73AmrbDUWYIgf16CzpEI4V2G2kq99O55\n3iuo4ea1s/QOySPSEsOpavx4spRhieRfv7zaZdf54a8OjXqd1IQIl11D+C63JWqaptmB+684fGbE\n448Aj1xxzgOMvlGgHLjR1TEK98iwRHD8bCOVDZ0+lagVSNsoIca0IT+NHfvLePNoJTeuysRs8s3y\nO1OxdV32R+4du3zctYmqp64jfJP//0sTHpeZ5Fxr96XCtz19Vs6Ut5KVHEl8dKje4QjhdcJDg9iQ\nl8alzn4OldTrHY5HJMQ4/xaEBpswGQ1kWCK579Zcl+/GXJOTzH235pJhuTyD9plN82XXpwDcu/Qp\nAlRmsrOVVJUPJWrFF1uw2R1S5FaIcWxamcHeo1XsPlzBVYtTfOoe1OnYc6QSgG/duZRFs+Lceq01\nOcmsyUnmwKk6nnm9hKZLvW69nvAdMqMmXC4xJtTnWkkNdyOYL4maEGNJjAlj1aIkqhq7KC5r0Tsc\nt2q81MMxrYGspEgWZsV67LqrFiURGxnMvqIaunsHPHZd4b0kURMuZzQYyLREUtvczYDVpnc4E7LZ\nHRSdbyY2MphZPlikVwhP2rI6E4Ddhyp0jsS99h6twuGALauzPDpzaDYZ2bQyk75+G/sKaz12XeG9\nJFETbpGZFInd4Rhze7s3OVPWQmfPAHnzEv1+KUeImcpOiWZhVizFZa1U1HtfPSpX6O61sq+ohtjI\nYFYtSvL49TfkpxEcZGTvsUqsttFLd4jAIYmacIuMJOd9ar6w/HmkpA6Q3Z5CTNaW1VkA7D5cqXMk\n7rGvsIa+fhs3rMjQZXdrRGgQ65ek0dLexzGt0ePXF95FEjXhFpk+lKgdLqkj2Gwkx803CwvhL5bM\nTSA1IZzDp+tpafevm96tNjt7j1USHGRkQ366bnFsWpWBAdh9uAKHw6FbHEJ/kqgJtxjaZl7V6N2J\nWn1rN5X1neRkxxMcNPX+sEIEIqPBwJbVWdjsDt46VqV3OC51TGukpb2P9UvSiAzTrwNDclw4yxZY\nKKvr4FxV28QnCL8liZpwi9BgM0mxYcOtpLxV4TnZ7SnEdKzLTSY6Iph3C6rp6bPqHY5LOBwOdh+u\nwIBzRktvm1cNbtw47N8bN8T4JFETbpOZFElnzwCXOvv1DmVMw90I5krbKCGmIshs4oYVGfT02Xi/\nsEbvcFziXFUbZXUd5M9PJDkuXO9wmJ8Rw+zUKArONVHf6v0bs4R7SKIm3Mbb71Pr6h3gbGUbC7Ji\nifGhVldCeIvrlqUTHGTkzaP+sTtxaOZqaLOE3gwGA5tXZeEA3jzinxs3xMQkURNuc3nnp3du4T95\noRm7w8HqnBS9QxHCJ0WGOXcnNrf3cVRr0DucGalv7abgXBOzU6OYnxGjdzjDVi60kBAdwgcna+ns\nkQK4gUgSNeE23j6jVljaDMDqXEnUhJiuG1dlYDDA7kOVXn0/6kT2HqnCAWxe5dkCtxMxGY3csCKT\n/gE77xVU6x2O0IEkasJtEmNCCQ02UdXYpXcoH2O12Tl5vpmE6BCyU6P1DkcIn5UUF86KBRbK6zs4\nU3FJ73Cmpat3gPdP1hAfHcIKZdE7nI+5Ni+NkGATbx2r8oslZjE1kqgJtzEYDGQkRVLnha2kSqva\n6O6zSjcCIVxgy5qhAri+uTvxvYIa+gfsbFqRqUuB24mEh5q5dmkalzr7OXLat5eYxdR532+k8CtD\nraSqm7xrVm24Cbt0IxBixuamxTA/I4ai881Ue3ntxCtZbXb2Hq0kJNjEtXlpeoczphtXDi4xSwHc\ngCOJmnArb7xPzeFwUFDaREiwCZUl3QiEcIWbhtpK+djuxCOnG7jU2c+1S9MIDzXrHc6YEmPDWKGS\nqGjo9NklZjE9kqgJt8q0eF+iVtfSTUNrD4tnxxNkln8CQrhC3vxEkuPCOFhcx6XOPr3DmRSHw8Hu\nIxUYDLBppf4FbieyZbAA7h4fXWIW0yPvUsKt0i0RGIAqL0rUCs7JsqcQrjbUVspq8522UlrFJSrq\nO1mxwIIlNkzvcCY0Nz2GuenRFJ5vprbZu24nEe4jiZpwq9BgM5Y472olVVDahMHgbCwthHCdqxan\nEBkWxLsnqunt9/62Ut5W4HYytqxyxioFcAOHJGrC7TKTIunqtdLaof9ySEd3P6XVbcxNjyE6PFjv\ncITwK8FBzrZSXb1WPiiq1TuccVU1dFB4vpm56dHMTfeeArcTWb7AQmJMKB+eqqOj23vb8wnXkURN\nuN1Q8Yvv/Ww/P/zVIQ6V1OsWS9H5ZhwOWfYUwl2uW55OkNnIniOV2OzeW/Nrx74LwOUZKl9hNBq4\ncWUmA1Y7756QAriBQBI14VaHSuo5qjUC4HBAVWMXT+8o1i1ZK5SyHEK4VXR4MFcvTqGprZfv//wA\nX/nvd3T/gHalju5+3jpaSWJMKMsW+N7fgmuWphIWYuat49UMWL03GRau4b17kYVf2HmgbIzj5azJ\nSfZoLFabnVMXW0iKDSM1Idyj1xYikCTHO/99tbQ7b3cY+oAGePzf/WjePVFN/4CNG1fOwWT0vfmK\nsBAzG/LT2HWogoMldaxf6r3139zlUEk9Ow+UUdPUTVpiOFvXZbv8d8sT15gM3/sNFT6lpql71ON6\n7FjSKi7R22+TbgRCuNmHJ0e/P23ngXIPR/JxA1Y7bx2vJjzUzDVLU/UOZ9o2rcjAaDCw54hv91id\njkMl9Ty9o5iqxi7sDodbVmo8cY3Jkhk14VZpieGj9vpMTYjweCyXy3LIbk8h3MmbPqBd6VBJPe1d\n/dyxcR5hIb77FhgfHcqqRUkcKqmnpKyV3NnxeofkMWOt1PxiRzG/+evpcc81GAyTSmwHBkZfUtZj\nNch3f0uFT9i6Lnt4yWOkNTlJHo1jqBtBWIiZ+ZmxHr22EIHGmz6gjeRwONhzpAKjwcAnrpkNXtaD\neKo2r8rkUEk9u49UBFSiVjNGS0IHkDbB75g5yIR1YOKfe1ldx6jH9fiwIYmacKuhTx47D5RT29xF\nbGQwze197D1axepFyR4rMlnd2EVzey+rFyV5ZdNlIfzJWB/QNg9W1tdLSVkrVY1drMlJJikunMbG\n0d+MfcXs1GgWZMZy6kIL1Y2dpA92gvFnnT0DGI0G7LaPz4plWCL54RdWjXu+xRI1qZ/7D391yGs+\nbMg7lnC7NTnJ/OuXV/PM317H/379au7ZNJ+2rn5+/EIB7V2eqQN0QnZ7CuExa3KSue/WXDIskZiM\nhuEemke1Bl1Lduw+4ixwq3fC6ErDbaUCoABu34CNJ18pwjpKkgawdd0sl11r67pst19jsmRGTXjc\nppWZtHf38/r+ch5/qZDvfWaZ2+8VKSxtwmgwSDcCITxkTU7y8Iy61WbniZeLKDrfzB/2nOXeLcrj\nG3qqGzs5daGFBRkxzE6N9ui13SlvXiJJcWEcKK7nUxvmEhPhn4W8bXY7T/+5mNKqNtbkJJM3N4E3\nDlZQ29xFakIEW9fNcum9Y1euBrnjGpMliZrQxR3r59DW2c/7RbX89LWTPLAtz21Lkm1d/VysaUdl\nxRIRGuSWawghxmY2Gfn67Yv5rz8c592CGhJiQsecsXCXN486Z5w2+1C7qMkYKoD7hzfP8s7xKm5f\nP0fvkFzO4XDw7C6NgtImcrLj+PLWRZhNRtbmprj1uiM/bOjJbUufSimjUurnSqkDSql3lVLzrnj8\nXqVUkVLqfaXUlwePBSmlfjd47LBS6tYrzrlHKXXAXTELzzEYDPzNTYr8eYkUl7Xyq52nsbtpi3lR\naRMOnJ88hRD6CAsx8+C2POKjQ3jlvQscKK7z2LXbuvrZf6qepLgwv7z94ZolqUSEmnn7uLM+nL95\ndd8FPiiqZVZKFN+4Y0nA3WfsztHeDoRqmrYO+Dvgx0MPKKUSgR8BG4ENwGeVUtnA54BmTdPWAzcB\nPxlxzjLgy1zuSCR8nMlo5P7bcpmfEcOhknr++NY5t9QDKpD704TwCnFRITy0LY+wEDO/3nma02Ut\nHrnuO8ersNrs3LgyE6PR/95CQoJNbFyWTmfPgEcTYE/Ye7SSnQfKSYoLG/7dCTTuHPE1wC4ATdMO\nKqVWjnhsDlCoaVoLgFLqCLAWeAl4efA5BsA6+HgC8B/Ag8AzE104Li4cs9k05YAtlqgpn+NP9Br/\nv953Fd9/6gP2Hq0iPTmau66f77LX7h+wUVLeSkZSJIvV2FPYgfyzD+SxQ2CPX4+xWyxR/OBLq/nn\nXxzgqT+d4r+/uZ5sN94z1jdg473CGiLDgrj9uvmEjnij96ef/bYbFbsPV/D2iWo+dYOaMCH1hbG/\nf6Ka5986R1xUCP/+tatJcdGOS18Y+0juTNSigbYRX9uUUmZN06zAOSBXKZUMdAA3AGc1TesEUEpF\n4UzYfqCUMgG/Ah4GeiZz4dbW0YstjmeyW3b9ld7j//anlvAfvz/GsztLMDrsLmuJUnS+ib5+G4tn\nx485Pr3HrqdAHjsE9vj1HHtqTChfumURv/hLCT98ej//eO8K4qND3XKtfYU1tHX2c8vaWXS09zA0\nYn/82a9amMyB4jreOVzO0nE2TvnC2IvLWnj8xUJCg008cNdSTHa7S2L21rGPlzy6c+mzHRh5ZeNg\nkoamaa3AQ8ArwPPAcaAJQCmVCbwD/E7TtOeAFcB84GfAH4EcpdTjboxb6CA+OpSHt+cTEWrm2b9q\nw8uVM3W5G4EsewrhTdbmpnDXxrm0dvTx+EtF9PRZXX4Nu8PB7sMVmIwGbliR4fLX9zZbVjtLdew+\nXKFzJDNTXtfBT149icEA3/rUUrKSfWsGzNXcmah9CNwCoJRaC5wcekApZQaWA+uB7cBC4MPBGbY9\nwPc1Tfs1gKZphzVNy9U0bSPwaaBE07QH3Ri30ElaYgQPbsvDbDLw8z+dorSqbeKTxuFwOCg830xk\nWBBz0/1nO74Q/uLmNVlctyydqsZOnnrtJFaba2usnbrQQm1zN6sXJRMXFeLS1/ZGWclRLJoVx+ny\nVirqvW/WaDLqW7t57MUC+vttfPWTuSycFad3SLpzZ6L2GtCrlNoPPAY8NLhr86tDM2s4Z9LeBZ7U\nNK0J+AcgDvinwZ2i7yqlPFO6XniFuekxfP2OxVhtDp54uZDqMVqFTEZFfSetHX0smZOAyRhYu4SE\n8AUGg4F7bpxP/rxESspa+e1fz7h0Q9EePyxwO5Ghsb7pgwVw2zr7ePSFAtq7B/jc5gWsXOjZVoPe\nym33qGmaZgfuv+LwmRGPPwI8csU5DwAPjPOaZTg3HQg/tnRuIl+8ZSG/2nmaR18omPb9K8O7PefL\nsqcQ3spkNHLfrbn8z/PH2X+qjoToUO64dua1wCobOikpa2VhViyzUgJn6WzJ3ARS4sM5WFLPnRvn\nEhvpGzOJPX1WHnuxkMZLvdx6dTbXLff/perJkmkG4ZWuXpLKtuuc96/8+IUCOnsGpvwaBeeaMBkN\nLA6gZsVC+KKQYBMP3JWHJTaUv+wvY19hzYxfc2g2bYufFbidiNFgYPPqTGx2B28fr9I7nEkZsNr5\nyasnqWjoZEN+GrddM1vvkLyKJGrCa920OovNqzKpbe7miZcL6ZtCIcfWjj7K6ztYmBUbkHV3hPA1\n0RHBPLQ9n8iwIP7fLo2i883Tfq1LnX0cLK4nJT48INvGXZWbQmRYEO8cr6av37sL4NrtDp55vYTT\n5a0sX2Dh3s2eby/m7SRRE17LYDCw/fp5rM1N5nx1Oz//06lJN3QuHFz2lG4EQviOlPhwvn3XUkwm\nAz/70ynK6tqn9TpvH6/CZneweVUmxgB80w8OMnHdsnS6eq3sP1WrdzhjcjgcPLf3LEfPNLAgM5b7\nbs3xy4LEMyWJmvBqRoOBL92yiMWz4yk838yzf9UmdbOxdCMQwjfNS4/hq5/MpX/AxuMvFdF0aVLl\nM4f1Ddh453g1kWFBrFvs3l6Q3uz65emYTQb2HKl0W3u+mXr9QDlvH68mwxLBt+9cQtA0CtUHAknU\nhNczm4x8/Y7FzE6N4oOTtby678K4z+/rt1FS1kqGJYLEWNk0LISvWaEsfHrTfNq7+nnspcIp3aO6\n/1QdXb1WrluWTkhQ4L7xx0SGsDY3hfrWHopKp7+M7C77Cmt4bd8FEqJDeWh7PuGhQXqH5LUkURM+\nITTYzAPb8kiOC2PngfJxt56XlLVgtdll2VMIH3bjyszhe1R/8koRA9aJ77WyOxzsOVKJ2WTg+uXp\nHojSuw2V6vC2Argnzjby7K4zRIYF8Z1P5wdEjbuZkERN+Izo8GC+c3c+MZHBPP/WOQ6V1I/6PFn2\nFDH/aRAAAAivSURBVMI/bL9+HisXJnG2qo1fvn56wiW8otJm6lu6WZuTQoyPlKVwpwxLJLmz49Eq\nL037fj9XO1t5iZ/vKCbIbOTBbXmkxIfrHZLXk0RN+JTE2DAe3p5PWIiJX75eQvHFlo88bh/sRhAd\nHsTsNOlGIIQvMxoM/J9PLGJ+RgxHzjTw8jvnx31+IBa4nciWwe/FHi8ogFvV2MmTLxdhtzv4xh1L\nmCN/oydFEjXhczKTIvn2nUsxGAz85LWTXKy9/EnxYm077V39LJ2XGJC7vYTwN0FmE9+6cympCeHs\nOlzB3qOjJxzldR2cqbhE7ux4MpIiPRyl98qdHU96YgRHTjfQ0t6rWxxNbT08+kIB3X1WvnTLIpbM\nCbyyKdMliZrwSSorjvtuzRncGVZIfUs3cLkshyx7CuE/IsOCeGhbHtERwTy/9xzHtMaPPWf3UIFb\nmU37CIPBwOZVzgK4bx3TpwBuR3c/j75QyKXOfu6+fl5A78adDknUhM9aoZK4d7Oio3uA//jdMf7x\nmYO8vr8cgO5e6wRnCyF8SWJsGA9uW0pwkIlf/KWY0uq24cda2ns5crqBtMQIcqUTyceszU0mOjyI\ndwtq6O337N/Gvn4bT7xcRF1LNzetyQq4ThGuIIma8Gkbl6WzUlno6Bmgtrl7+Piv3zg95mYDIYRv\nyk6J5mu352KzOXjy5aLhmfS3RhS4lar2HxdkNnH98gx6+qx8UOS5ArhWm52f/ukUF2rauWpxCndt\nnOuxa/sT6a0jfF5dS/eox3ceKGdNTrKHoxFCuNPSuYncu2UBz+7S+M/fHyM8NIi6lm6MBjCbJEkb\ny8bl6fxlfxkvvF3KH98uJS0hnK3rsl3+N/JQST07D5RR09RNaLCJ7j4rS+Yk8IWbF8p9w9MkiZrw\neTVNoydqtc1dHo5ECOEJG/LTKbrQzImzTbR3O4vh2h3wy9dPYzIa5QPaKE6XtWKzXy5vUtXYxdM7\nigFc9v06VFI//JoA3X3OZdZVC5Mwm2QBb7okURM+Ly0xnKrGjydlqQkROkQjhPCExtbRW0vJTPro\ndh4oG/X4L3YU85u/nnbJNQYGRu/FvOdIJdcsTXXJNQKRJGrC521dl/2RT3GXj8/SIRohhCfITPrU\njPX9cgBpLvpQW1bXMepx+ZnMjCRqwucNfXreeaCc2uYuUhMi2LpulnyqFsKPyUz61Iz1/cqwRPLD\nL6z6/9u7kxA56iiO49+JRmMgigcTFERB8YGXKA5GUWPAfcPl7BpEFA+KwRWXix4EE0HFhZi4RhCX\neBDUHFzigluioigvboeIC0FciYrR9lA12mgnYyBTVd3/7wcGqqoH5r1+s/z6X/9mtsnXuH7ZG85k\nChjUNBLm7T/HYCYVxJX0rdPE8+VMpoZBTZI0dFxJ3zpNPF/OZGoY1CRJQ8mV9K0z8XztttssNmwY\nvJ9sW30NbTu+X1aSJKmjDGqSJEkdZVCTJEnqKIOaJElSRxnUJEmSOmqs1+tN/lmSJElqnCtqkiRJ\nHWVQkyRJ6iiDmiRJUkcZ1CRJkjrKoCZJktRRBjVJkqSOMqhJkiR11PZtF9C2iJgG3AnMBX4Dzs/M\nT9qtqjkRsRb4sT79PDPPa7OeJkTEPODmzFwQEfsC9wM94APg4sz8s836ptq/+j8QeBr4uH74rsx8\ntL3qpkZETAeWA3sDOwI3Ah9SyOw30/96ypj9dsBSIKhmfSHwK+XMflD/0ylg9hMiYjawBjgG2MSQ\nzd4VNTgNmJGZhwJXAYtbrqcxETEDGMvMBfVHCSHtCuBeYEZ9aQlwbWYeAYwBp7ZVWxMG9H8QsKTv\ne2BUf1mfCXxbz/l44A7Kmv2g/kuZ/SkAmXkYcC1wE2XNflD/pcx+4kXKPcAv9aWhm71BDQ4HngXI\nzNeB8XbLadRcYGZErIqI5yPikLYLasCnwBl95wcBL9XHzwBHN15Rswb1f1JErI6IZRExq6W6ptpj\nwHX18RjVq+qSZr+5/kd+9pn5FHBBfboX8D0FzX4L/Y/87Gu3AHcDX9bnQzd7gxrsDPzQd/5HRJRy\nS3gj1TfxcVTL4StGvffMfAL4ve/SWGZO/B+1n4Bdmq+qOQP6fxO4PDPnA58BN7RS2BTLzJ8z86f6\nD9LjVCsLxcx+M/0XMXuAzNwUEQ8AtwMrKGj2MLD/ImYfEecCGzLzub7LQzd7g1q1P6v/1cS0zNzU\nVjENWwc8nJm9zFwHfAvs3nJNTevfmzCL6tVmSVZm5pqJY+DANouZShGxJ/AC8FBmPkJhsx/QfzGz\nB8jMc4D9qPZr7dT30MjPHv7T/6pCZr8QOCYiXgQOAB4EZvc9PhSzN6jBq8CJAPWtv/fbLadRC6n3\n5EXEHlSri1+1WlHz3omIBfXxCcDLLdbShuci4uD6+CiqDbcjJyLmAKuAKzNzeX25mNlvpv9SZn9W\nRFxdn26kCuhvFzT7Qf0/WcLsM3N+Zh6ZmQuAd4GzgWeGbfYjfZvrf1pJlbhfo9q7MfIb6vssA+6P\niFeo3gGzsKDVxAmLgKURsQPwEdVtoZJcBNweEb8DX/PPXpZRcw2wK3BdREzs1boEuK2Q2Q/q/zLg\n1gJm/yRwX0Sspnq346VU8y7l535Q/+sp4+d+kKH7nT/W6/Um/yxJkiQ1zlufkiRJHWVQkyRJ6iiD\nmiRJUkcZ1CRJkjrKoCZJktRRBjVJkqSOMqhJkiR1lEFNkiYREbMj4ou265BUHoOaJE1uHHir7SIk\nlcegJkmT+zuoRcTJEfFWRIy3XJOkAvi/PiVpcuPAPRGxGNgHODYzv2u5JkkFcEVNkiY3DiwGvgFO\nN6RJaopBTZK2ICL2AH4F3gNmZmav5ZIkFcSgJklbNg6sAc4GjoqI81quR1JBDGqStGXjwNrM/A04\nDVgUEce1XJOkQoz1eq7iS5IkdZErapIkSR1lUJMkSeoog5okSVJHGdQkSZI6yqAmSZLUUQY1SZKk\njjKoSZIkddRfT0E31Jyzg4kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x19600910fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "k_values = np.arange(1, 42, 2) # odd values from 1 to 42\n",
    "k_scores = []\n",
    "for k in k_values:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    scores = cross_val_score(knn, X_resampled_train, y_resampled_train, cv=10, scoring='accuracy')\n",
    "    avg_score = np.mean(scores)\n",
    "    k_scores.append(avg_score)\n",
    "    \n",
    "# Plot the scores\n",
    "fig, ax = plt.subplots(figsize=(10,4))\n",
    "ax.plot(k_values, k_scores, 'o-')\n",
    "ax.set_xlabel('$k$')\n",
    "ax.set_ylabel('Score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "It is clear that the score peaks at $k=5$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is time to evaluate kNN where $k=5$ using a classification report. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:42.374082Z",
     "start_time": "2017-08-03T17:40:42.354576Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.98      0.95       153\n",
      "          1       0.98      0.92      0.95       143\n",
      "\n",
      "avg / total       0.95      0.95      0.95       296\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "knn = KNeighborsClassifier(n_neighbors=5)\n",
    "knn.fit(X_resampled_train, y_resampled_train)\n",
    "y_resampled_predictions = knn.predict(X_resampled_test)\n",
    "\n",
    "print(classification_report(y_resampled_test, y_resampled_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results for the under-sampled data are quite good. Here is the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:42.382083Z",
     "start_time": "2017-08-03T17:40:42.375582Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[150,   3],\n",
       "       [ 12, 131]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "confusion_matrix(y_resampled_test, y_resampled_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us predict on a subset of the whole data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:42.485615Z",
     "start_time": "2017-08-03T17:40:42.384084Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:44.683403Z",
     "start_time": "2017-08-03T17:40:42.487116Z"
    }
   },
   "outputs": [],
   "source": [
    "predicted = knn.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:44.706408Z",
     "start_time": "2017-08-03T17:40:44.684902Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      0.97      0.99     85282\n",
      "          1       0.06      0.91      0.11       161\n",
      "\n",
      "avg / total       1.00      0.97      0.98     85443\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T17:40:44.767424Z",
     "start_time": "2017-08-03T17:40:44.707908Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[83013  2269]\n",
      " [   14   147]]\n"
     ]
    }
   ],
   "source": [
    "print(confusion_matrix(y_test, predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "170px",
    "width": "543px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_position": {
    "height": "659px",
    "left": "0px",
    "right": "1096px",
    "top": "106px",
    "width": "280px"
   },
   "toc_section_display": "block",
   "toc_window_display": true,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}