09_classification_metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating a classification model\n",
    "\n",
    "*From the video series: [Introduction to machine learning with scikit-learn](https://github.com/justmarkham/scikit-learn-videos)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "- What is the purpose of **model evaluation**, and what are some common evaluation procedures?\n",
    "- What is the usage of **classification accuracy**, and what are its limitations?\n",
    "- How does a **confusion matrix** describe the performance of a classifier?\n",
    "- What **metrics** can be computed from a confusion matrix?\n",
    "- How can you adjust classifier performance by **changing the classification threshold**?\n",
    "- What is the purpose of an **ROC curve**?\n",
    "- How does **Area Under the Curve (AUC)** differ from classification accuracy?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review of model evaluation\n",
    "\n",
    "- Need a way to choose between models: different model types, tuning parameters, and features\n",
    "- Use a **model evaluation procedure** to estimate how well a model will generalize to out-of-sample data\n",
    "- Requires a **model evaluation metric** to quantify the model performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model evaluation procedures\n",
    "\n",
    "1. **Training and testing on the same data**\n",
    "    - Rewards overly complex models that \"overfit\" the training data and won't necessarily generalize\n",
    "2. **Train/test split**\n",
    "    - Split the dataset into two pieces, so that the model can be trained and tested on different data\n",
    "    - Better estimate of out-of-sample performance, but still a \"high variance\" estimate\n",
    "    - Useful due to its speed, simplicity, and flexibility\n",
    "3. **K-fold cross-validation**\n",
    "    - Systematically create \"K\" train/test splits and average the results together\n",
    "    - Even better estimate of out-of-sample performance\n",
    "    - Runs \"K\" times slower than train/test split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model evaluation metrics\n",
    "\n",
    "- **Regression problems:** Mean Absolute Error, Mean Squared Error, Root Mean Squared Error\n",
    "- **Classification problems:** Classification accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification accuracy\n",
    "\n",
    "[Pima Indian Diabetes dataset](https://archive.ics.uci.edu/ml/datasets/Pima+Indians+Diabetes) from the UCI Machine Learning Repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# read the data into a Pandas DataFrame\n",
    "import pandas as pd\n",
    "url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data'\n",
    "col_names = ['pregnant', 'glucose', 'bp', 'skin', 'insulin', 'bmi', 'pedigree', 'age', 'label']\n",
    "pima = pd.read_csv(url, header=None, names=col_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pregnant</th>\n",
       "      <th>glucose</th>\n",
       "      <th>bp</th>\n",
       "      <th>skin</th>\n",
       "      <th>insulin</th>\n",
       "      <th>bmi</th>\n",
       "      <th>pedigree</th>\n",
       "      <th>age</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pregnant  glucose  bp  skin  insulin   bmi  pedigree  age  label\n",
       "0         6      148  72    35        0  33.6     0.627   50      1\n",
       "1         1       85  66    29        0  26.6     0.351   31      0\n",
       "2         8      183  64     0        0  23.3     0.672   32      1\n",
       "3         1       89  66    23       94  28.1     0.167   21      0\n",
       "4         0      137  40    35      168  43.1     2.288   33      1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the first 5 rows of data\n",
    "pima.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** Can we predict the diabetes status of a patient given their health measurements?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define X and y\n",
    "feature_cols = ['pregnant', 'insulin', 'bmi', 'age']\n",
    "X = pima[feature_cols]\n",
    "y = pima.label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# split X and y into training and testing sets\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train a logistic regression model on the training set\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression()\n",
    "logreg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# make class predictions for the testing set\n",
    "y_pred_class = logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Classification accuracy:** percentage of correct predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.692708333333\n"
     ]
    }
   ],
   "source": [
    "# calculate accuracy\n",
    "from sklearn import metrics\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Null accuracy:** accuracy that could be achieved by always predicting the most frequent class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    130\n",
       "1     62\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the class distribution of the testing set (using a Pandas Series method)\n",
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3229166666666667"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the percentage of ones\n",
    "y_test.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6770833333333333"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the percentage of zeros\n",
    "1 - y_test.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6770833333333333"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate null accuracy (for binary classification problems coded as 0/1)\n",
    "max(y_test.mean(), 1 - y_test.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.677083\n",
       "Name: label, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate null accuracy (for multi-class classification problems)\n",
    "y_test.value_counts().head(1) / len(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing the **true** and **predicted** response values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True: [1 0 0 1 0 0 1 1 0 0 1 1 0 0 0 0 1 0 0 0 1 1 0 0 0]\n",
      "Pred: [0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "# print the first 25 true and predicted responses\n",
    "from __future__ import print_function\n",
    "print('True:', y_test.values[0:25])\n",
    "print('Pred:', y_pred_class[0:25])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:**\n",
    "\n",
    "- Classification accuracy is the **easiest classification metric to understand**\n",
    "- But, it does not tell you the **underlying distribution** of response values\n",
    "- And, it does not tell you what **\"types\" of errors** your classifier is making"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion matrix\n",
    "\n",
    "Table that describes the performance of a classification model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[118  12]\n",
      " [ 47  15]]\n"
     ]
    }
   ],
   "source": [
    "# IMPORTANT: first argument is true values, second argument is predicted values\n",
    "print(metrics.confusion_matrix(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Small confusion matrix](images/09_confusion_matrix_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Every observation in the testing set is represented in **exactly one box**\n",
    "- It's a 2x2 matrix because there are **2 response classes**\n",
    "- The format shown here is **not** universal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Basic terminology**\n",
    "\n",
    "- **True Positives (TP):** we *correctly* predicted that they *do* have diabetes\n",
    "- **True Negatives (TN):** we *correctly* predicted that they *don't* have diabetes\n",
    "- **False Positives (FP):** we *incorrectly* predicted that they *do* have diabetes (a \"Type I error\")\n",
    "- **False Negatives (FN):** we *incorrectly* predicted that they *don't* have diabetes (a \"Type II error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True: [1 0 0 1 0 0 1 1 0 0 1 1 0 0 0 0 1 0 0 0 1 1 0 0 0]\n",
      "Pred: [0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "# print the first 25 true and predicted responses\n",
    "print('True:', y_test.values[0:25])\n",
    "print('Pred:', y_pred_class[0:25])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# save confusion matrix and slice into four pieces\n",
    "confusion = metrics.confusion_matrix(y_test, y_pred_class)\n",
    "TP = confusion[1, 1]\n",
    "TN = confusion[0, 0]\n",
    "FP = confusion[0, 1]\n",
    "FN = confusion[1, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Large confusion matrix](images/09_confusion_matrix_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metrics computed from a confusion matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Classification Accuracy:** Overall, how often is the classifier correct?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.692708333333\n",
      "0.692708333333\n"
     ]
    }
   ],
   "source": [
    "print((TP + TN) / float(TP + TN + FP + FN))\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Classification Error:** Overall, how often is the classifier incorrect?\n",
    "\n",
    "- Also known as \"Misclassification Rate\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.307291666667\n",
      "0.307291666667\n"
     ]
    }
   ],
   "source": [
    "print((FP + FN) / float(TP + TN + FP + FN))\n",
    "print(1 - metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Sensitivity:** When the actual value is positive, how often is the prediction correct?\n",
    "\n",
    "- How \"sensitive\" is the classifier to detecting positive instances?\n",
    "- Also known as \"True Positive Rate\" or \"Recall\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.241935483871\n",
      "0.241935483871\n"
     ]
    }
   ],
   "source": [
    "print(TP / float(TP + FN))\n",
    "print(metrics.recall_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Specificity:** When the actual value is negative, how often is the prediction correct?\n",
    "\n",
    "- How \"specific\" (or \"selective\") is the classifier in predicting positive instances?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.907692307692\n"
     ]
    }
   ],
   "source": [
    "print(TN / float(TN + FP))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**False Positive Rate:** When the actual value is negative, how often is the prediction incorrect?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0923076923077\n"
     ]
    }
   ],
   "source": [
    "print(FP / float(TN + FP))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Precision:** When a positive value is predicted, how often is the prediction correct?\n",
    "\n",
    "- How \"precise\" is the classifier when predicting positive instances?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.555555555556\n",
      "0.555555555556\n"
     ]
    }
   ],
   "source": [
    "print(TP / float(TP + FP))\n",
    "print(metrics.precision_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many other metrics can be computed: F1 score, Matthews correlation coefficient, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:**\n",
    "\n",
    "- Confusion matrix gives you a **more complete picture** of how your classifier is performing\n",
    "- Also allows you to compute various **classification metrics**, and these metrics can guide your model selection\n",
    "\n",
    "**Which metrics should you focus on?**\n",
    "\n",
    "- Choice of metric depends on your **business objective**\n",
    "- **Spam filter** (positive class is \"spam\"): Optimize for **precision or specificity** because false negatives (spam goes to the inbox) are more acceptable than false positives (non-spam is caught by the spam filter)\n",
    "- **Fraudulent transaction detector** (positive class is \"fraud\"): Optimize for **sensitivity** because false positives (normal transactions that are flagged as possible fraud) are more acceptable than false negatives (fraudulent transactions that are not detected)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adjusting the classification threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 1, 0, 1], dtype=int64)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the first 10 predicted responses\n",
    "logreg.predict(X_test)[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.63247571,  0.36752429],\n",
       "       [ 0.71643656,  0.28356344],\n",
       "       [ 0.71104114,  0.28895886],\n",
       "       [ 0.5858938 ,  0.4141062 ],\n",
       "       [ 0.84103973,  0.15896027],\n",
       "       [ 0.82934844,  0.17065156],\n",
       "       [ 0.50110974,  0.49889026],\n",
       "       [ 0.48658459,  0.51341541],\n",
       "       [ 0.72321388,  0.27678612],\n",
       "       [ 0.32810562,  0.67189438]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the first 10 predicted probabilities of class membership\n",
    "logreg.predict_proba(X_test)[0:10, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.36752429,  0.28356344,  0.28895886,  0.4141062 ,  0.15896027,\n",
       "        0.17065156,  0.49889026,  0.51341541,  0.27678612,  0.67189438])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the first 10 predicted probabilities for class 1\n",
    "logreg.predict_proba(X_test)[0:10, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# store the predicted probabilities for class 1\n",
    "y_pred_prob = logreg.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# allow plots to appear in the notebook\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['font.size'] = 14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xab7bda0>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAElCAYAAAAlet80AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XvcZXPd//HXG+OQUA6Z3HLKoUhKFIm5um9EB6WiAxGh\nolK5OygxqYi6JakUUXQn+lWKDhIuuhUaiXLOOcyYCMOM08zn98f3u2fWLHtfe+91Xdde65p5Px+P\n/biuvY6f9d1rr89a3+93r6WIwMzMrIol6g7AzMwmLicRMzOrzEnEzMwqcxIxM7PKnETMzKwyJxEz\nM6vMSaQiSXdIOrXuOBZ1klaRdJakmZLmSvpI3TGNRNLakuZJ2qswbKqkeXXGVdbE/VfSlFx2u4/h\nMvfOy3xlD9N+X9LtpWELlVO7z7fLMoclXdx/5BOHkwjddzRJ50m6rTR4HtDXj2wkbS3pCEkrVo11\nMXQM8Ib89z3Ab+sNp5Ig7S99kfQuSQePQzzQ5747QOMRV6/LbPc5tZt3oWFdvteVPvuJZKm6A2iQ\nkXa0duM2ov+d49XA4cBpwCN9zru4mgKcHxFfrTuQUfgCcHSF+d4NbAJ8fWzDaTTVuO796HJiHRF3\nSloOeKoweKTv9Q5jGmED+Uqkooh4KiLm9jlbnV8QJE2StGSdMVTwPODh8V6JpGeN17IjYl5EPDle\ny2+q8SzT8RARcyPiqR6mezIWvtVHx+91RDwdEU+PSYAN5SRSUbs6ZUkHSrpW0qOSHpJ0taT987gj\ngGPzpHfk6rO5krYrzP8BSX+TNEfSfZJOkvTcNus+SNKtkmZLulzStrnu9aLCNK365XfnOvk7gdnA\nf+Rk8nlJV0p6MC/nCklvbrOueZK+Jeltkv5eWOdmefz+km7OMQ9LWrfH8ls7t3X8Ky/zyuL6W1WM\nwLOB97bKq8syW7HuLum6HNNfJb2uNF2r+vK1kk6QNB2YVRi/oqTjJN0p6Ylc1odJWqK0nJVyPfpD\nkv4t6TTgOW3iatsmImkHSRdJeljSI5KmSdo3j7uYVI23To71Gdsv6cN5f5sjaYakUySt0mY9h0m6\nW9Jjki6UtPFI5ThOZTri510QwJJ5/7wnx/xbSeuX1reppFMl/SPHNFPSmZJe0GFTlpN0Yp7ukRzL\n80rLfEabSJvyWKhNRF2+1yp9LwvL6frZSXq5pF9Luj9Pd4ek0yUtM1KMg+bqrIWt1OZLKGBSm2nL\n9aLvA04Ezga+kefZhHSpezLwM2BD4J3AwcADedYb8vyHAUcCvwdOAl4IfAh4paRXtc6QJH0wL/9S\n4DhgbeAc4EHg7jZxfgaYCxyft+VRYEVgf+DHwKnAsqSqk59Jen1EnF9axjakA9o383Z/BjhX0peA\nDwPfIh08P026pB9qE0exrFYD/gQsT6qq+RewZ17/uyPiLOCSPOx7wBXAd0daZsFrgN3zch8FDsix\nDkXEH0vTfoNUbl8EVsqxLQsMA2sB3wbuBF4JTM3DDijM/0vS53sS6XN8M/ADnln9GeVhkt4DfB+4\nntTe8wDwUlI5n1qI6T+Aj1I625V0ErBPXsY3gBcAHwG2lLRl68pH0heAzwLnAb8BXgacDyzdrvA6\nGG2Z9vJ5z9800n60BPAV4Ll5+y+S9NKIeChPtwPp+/QD4F7S9+WDeftfEhGPl5Z5Qo7986TvzEeA\nDXNZta4UnvE59WDE73W75fXy2UlaFbgAmEnaP/6dp9uFVI5P9Bnn+ImIxf4F7E1q3xjpdVtpntuB\nUwvvfwZc22U9h5AO6GuVhq8KPE76cqtNXAfm95NIO9WfgSUL070nT3dRYdiUPOwOYNnS+gRMKg1b\nCvgb8LvS8Hk5tnUKw/bPw6cDzy4M/1LevvW6lMNxebptC8OWBa4D7ilt26xiOXdZ7ry83FcWhq1M\n+mJf2qZc/wgsUVrGZ0gHmw3aDJ/bGk5KGPOAQ0rlelGebq/C8COAuYX3KwAPkZLjMiNsz7nl/S4P\nf3Ve9x4dhu9X2q9+UZru83m6ruU6RmXa0+dd2GfvA1YoTPvaPPzI4vxtYt0qT/fuNnH9tbjPkw7i\n84B9C8NOK5c3z/yer53nK36+bb/XedzFLPy97PWz2yUv8+W97Pt1vlydtUCQzvy3b/O6oof5HwbW\nlLRFhXVvT0oQX4+8B2VnADNIZ6cAWwKrACfHwu0xPyKdqbTzg1j4rIxIWlc2k5SqzJ5Durp5RZtl\nXBQRdxTet8rjpxHxaJvh63WIpeX1wF8i4g+FmB4nXdFMBjbvMv9IpkXElYXlPkgqn20krVSa9uSI\nKFcz7Qb8H/CgUvfiVfLV6YWkJDFU2Ia5pKuQ1rqCdLXWre1rR1Ii+XJEVDmj3J2UXH9XivFm0v7y\n2jzdDqT96lul+U/oc32jLdN+P+8fRMSswrQXkxLOG0vzAyBpeUkrA/8gJed2+/BJsXB7x+l52je2\nmXY89frZPUzaj3aR1Ogao0YHV4OFviwtkv4FrN5l3mOA/wSuVOoOfAFwVkQM97DetfPfm4sDI2Ke\npFuAdfKgtUjJ7tbSdHMl3dFh2eWuyQBI2o9UTfBiFj7otetxVq4mazV0/7PNcJGqIEayNvDTNsNv\nyPOvQ7raquKWNsNa5bo2cG3+P2hfNhuSqpVmthkXpIZ+SJ/F9Ih4rMO6RvLC/Pe6HqZtZwNSEprR\nZlw5RkgH1wUTRDwgqdNJRzujLdN+P+9/tJn2ZhYcYJH0HNJ37m2kK6OWIFejlYaVy2Bubv9Yp826\nxlNPn11EXCLpJ6ReXx+XdAmp+vRHETF7UMH2wklkjETEjZI2Ip11vY509fB+Sd+MiA/XGNqc8gBJ\ne5DaGH4BfBm4H3ga2Bd4V5tldGrQ7jS81l5ofXhG2ZDq4i8CjqL9drRNygO2BKld4R20j7GfBDHW\n2pXpePgJsDXwVVJV1SzSQfgsmt1hqOfPLiLekWs23ki6qvwu8GlJW0XEvwYRbC+cRMZQvsT+GanB\ncAlSo9+Bko6KiPvo3Gh3J2mH2ojCGZMkkc5c/lKabn1S9UpruiVJZ1TX9Bjq24FbI2LX4sDcOWAQ\n7iRta9mLSWV0xyiWvUGbYa113dnD/LeS6uO7/cr4TmB7ScuXrkbabVe7dQh4CSNfuXTaX24lV7N2\nOSttbe8GFK5ec/VJt6vFotGWab+fd7v1bdiaLl+F/BdweER8sTVB7rXUbrta36PfF6ZdEliX1Ili\ntPppjO/1s0sLjpgGTAOm5h5xvyG1SVb53dG4aHLGnlBynex8uV747/ltq9tn62BT3tEvAJ4EPpIT\nR8uepGq0c/P7aaQGzf218O899myzzJE84wpC0nrAW/pYxmicB2wuaZvC+pch9a6ZDlw1imVvIWmr\nwnJXIV1dXRYRvfze5CxSL5mdyyMkPVtSq1fTr4Elc8yt8QIOovtB5XekH6V9OvcG6+Qx2nQZzjEu\nSarqKMe4RD7IQjpoPg0cWJqs31/Bj7ZM+/289yq2tUj6T1JPx/PyoNb+Wz5+fbzNsJb3Fz47SA3u\nzyksczQ6fa/b6emzK3yGRX/Nf9uNq42vRBYYbRXM7yTdT2qUnU468/kQcE1EtLr7Tcvr+bKkH5ES\nx4UR8a/cFfPIvJxzSPXmBwFXk7q5EhFPSZpKahi9WNLZpPrmfUhXML2eEf0SeKukc/P/a5K+0DeS\nuoCOt2NIB6FfSzqB1P7wHuBFpJ41o7lNxN9J3U9PZEF31GcDh5am6/R5fwV4E3COpNNJB7jlgE1J\n9e+bAneREvtlwNFKv425jpSEu37BI2KW0u1MvgdMy/vCA6QD5RoR8fY86TRgd0nHkzotzIuIsyLi\nD5K+CXxC6fc655O6fG6QY/wccHrer75KSlbnkRLfZsDOtG/z6WS0Zdrv5z0DuEzS90gH5oNJvbi+\nBvPLbxj4ZE5Gd5K6IW9Hqirq5GJJZ5KuQD5Eass5feRN70nH73V5wl4/O2BvSQcBPyddvSxH+p4/\nDfy/MYh57AyyKxipq2O56+y9pWmmknaY2aTucRsPIK69KXVjLI0/l1T9Uxx2G/C9wvv9SHXpM0j1\nwreSdvpVS/N9knRZ/lRe53aFcQeQutnOIXVz/Dbw3DbxHJTXP5t0cNmG1DD5q8I0U/Lyd++wTYeQ\nEs9s0pfp3ZS6oubp5gLfLA1bOw//RGn4iOtss4wfk770s4ErgV3aTPdIsZy7LHMeqcfPbqTfX8wh\nJeGd+vy8lyMl9BvzMu4nJYxDgKUL0z2H1Nf/36TfRpxGapRv18X36Tbr2Yn0e5hZpJ5CVwJ7l+I4\njXTQfbrNZ/Ne4HLSgf2h/DkeA6xZmu4wUieIR0lXJy8u778DKNOun3dh/3kXC44Dj5GqcMpdrlfP\ny5uZt/0XpF6B5e9lK64ppN9k3J/3qbOA1UvLPI3u3/PWvr9Xabq232vSMezCNuUx4mdHOpk7gwXf\n8+m5HLbp5bswyJdywAOh9OvOd5A+0NZZy9yIeCCP/xSpP/7epLriI0hnGBvGM3vBWJarUWaSuty+\nv+546qL0q/CTIqJcfWMVuUytmzraRJ6OiJkRcX9+PVAYdzBwdEScExHXk5LJCqSzZGN+XXLZ3qRu\njt0ag83MxlQdSWQ9pXvi3KZ0r5t1AfLfyaRGZmB+b6dLSb/mtGQrSX+RdKikAyR9h3RblWtpWl2p\nmS3yBt2wfjmpLvBG0o9qPkdqQNuElECCZ/4IZwawxgBjbLo7SA27HyZdfTxIqpc/NBbxu4X2IGju\nczImKpepjWigSSRKN/aTdDnp3jR709utRRZ7EXEng+uKO6FExES7zX3juUytm1q7+EbEbEnXkbq3\n/YLU2L46C99OY3VSz4S2JPksycysgogY9d0lav2xYf6h1YtI3XxvJyWLHUrjtyV1r+yo7i5uTXkd\nccQRtcfQlJfLwmXhshj5NVYGeiUi6Suk31zcRbrC+BzwLBb84Od44FBJN5Fu+nYYqQ/9mYOM08zM\nejPo6qw1SbeQXpX0u4bLga0i4m6AiDg2X32cSPql6hXAjuHfiJiZNdKgG9bb3SG2PM2RpF8LW5+G\nhobqDqExXBYLuCwWcFmMvYH+Yn08SIqJvg1mZoMmiZjoDetmZjaxOYmYmVllTiJmZlaZk4iZmVXm\nJGJmZpU5iZiZWWVOImZmVpmTiJmZVeYkYmZmlTmJmJlZZU4iZmZWmZPIYmDy5HWQVNtr8uR16i4C\nMxsnvgHjYkAS9T4mW2P6EBwzGz3fgNHMzGrnJGJmZpU5iZiZWWVOImZmVpmTiJmZVeYkYmZmlTmJ\nmJlZZU4iZmZWmZOImZlV5iRiZmaVOYmYmVllTiJmZlbZUnUHsKibPHkdZsy4s+4wzMzGhe/iO87q\nv4MuQN0x+C6+Zk3ju/iamVntnETMzKwyJxEzM6vMScTMzCpzEjEzs8qcRMzMrLJak4ikQyXNk3RC\nafhUSfdImi3pYkkb1xWjmZl1VlsSkbQVsD9wTWn4p4CPAQcBWwD3AxdIWn7gQZqZ2YhqSSKSVgJ+\nCOwDPFQafTBwdEScExHXA3sDKwDvHmyUZmbWTV1XIt8Fzo6IS4oDJa0LTAYuaA2LiMeBS4FXDzRC\nMzPrauD3zpK0P7Ae8K42oyeT7s8xozR8BrDGOIdmZmZ9GmgSkbQh8CVgm4iYN1bLnTp16vz/h4aG\nGBoaGqtFm5ktEoaHhxkeHh7z5Q70BoyS9gZOBYoJZEnS1cdc4CXAjcCWEXFVYb7zgJkRsU+bZfoG\njN2jqDkG34DRrGkm6g0Yfw5sCmxWeE0DzgQ2i4ibgenADq0ZJC0LbAtcNuBYzcysi4FWZ0XEI8D1\nxWGSHgMejIgb8qDjgUMl3QTcAhwGzCIlGjMza5AmPJRqoXqOiDg2X32cCDwXuALYMSIeqyM4MzPr\nzA+lGmduE0nrb/JnZLY4mqhtImZmtghxEjEzs8qcRMzMrDInETMzq8xJxMzMKnMSMTOzypxEzMys\nMicRMzOrzEnEzMwqcxIxM7PKnETMzKwyJxEzM6vMScQWC5Mnr4Ok2l6TJ69TdxGYjQvfxXec+S6+\naf11f0b1fw71l4FZke/ia2ZmtXMSMTOzypxEzMysMicRMzOrzEnEzMwqcxIxM7PKnETMzKwyJxEz\nM6vMScTMzCpzEjEzs8qcRMzMrDInETMzq8xJxMzMKnMSMTOzyvpKIpJWHq9AzMxs4un3SuReST+W\ntMO4RGNmZhNKv0lk1zzPuZLukHSEpLXHIS5bpCxT61MF0wOpzGw8VHqyYa7Weg/wXmBT4CLge8DP\nI+LJsQywh1j8ZMPuUdQcQ93rb0IMfrKhNctYPdlw1I/HlfQh4KvA0sCDwEnAURExe7TB9bh+J5Hu\nUdQcQ93rb0IMTiLWLLU+HlfS8yT9t6TrgGOB/wf8F3Aw8GbgnA7zHSjpGkkP59cfJb2+NM1USfdI\nmi3pYkkbV4nRzMzGX19XIpJ2AfYFdgZuAk4BzoiIfxemeSFwQ0Qs3Wb+NwFPAreQEth7gU8Cm0fE\n3yV9CvgMsDdwM3AE8Bpgw4h4rENMvhLpHkXNMdS9/ibE4CsRa5ZaqrMkzQLOAk6OiCs6TLMc8JmI\n+FyPy3wA+HREnCzpXuCEiPhyHrcscD9wSESc3GF+J5HuUdQcQ93rb0IMTiLWLHUlkWdHxKOjXWle\n1hLA7sD3gc2BOcCtwJYRcVVhuvOAmRGxT4flOIl0j6LmGOpefxNicBKxZqmrTWRHSW9sE8ybJO3a\nywIkvSRf0TwBfAvYNSKuByaTvuUzSrPMyOPMzKxh+k0iR5IO/mWP53G9uBHYDHgl8G3gdDeem5lN\nTEv1Of0LSQ3eZbfkcV1FxNPAbfnt1ZJeCXwMOIpU57A68M/CLKsD00da5tSpU+f/PzQ0xNDQUC+h\nmJktNoaHhxkeHh7z5fbbJnIfsGdEXFgavgPww4hYve8ApAuBeyJirw4N6zNIDeundJjfbSLdo6g5\nhrrX34QY3CZizTJWbSL9Xon8EviapF0j4tYcyPrA/+RxI5J0NPAr4G5gBWAPYArQ+q3I8cChkm4i\nXd0cBswCzuwzTjMzG4B+k8gngfOBGyW1qpzWBP4CfKKH+ScDZ+S/DwPXAjtFxO8BIuLYfPVxIvBc\n4Apgx06/ETEzs3r1fdsTpfqZnYCX5UFXA+fXVafk6qyeoqg5hrrX34QYXJ1lzdKYe2fVzUmkpyhq\njqHu9TchBicRa5a62kSQ9ArSfbKeR6mLcER8fLQBmZnZxNFXEpH0MVIj+h3AvSx8aufTLDOzxUy/\nXXzvAo6LiOPHL6T+uDqrpyhqjqHu9TchBldnWbPUdduTleihK6+ZmS0e+k0iZwM7jkcgZmY28fTb\nsH4r8AVJWwF/A54qjoyIE8YqMDMza75+20TuHmF0RMRaow+pP24T6SmKmmOoe/1NiMFtItYs/p1I\n5iTSUxQ1x1D3+psQg5OINUutz1jPAaySf71uZmaLqb6SiKRJko6S9BDp7rrr5uFHS/rAeARoZmbN\n1e+VyOeAtwHvY+GHU10FtH18rZmZLbr6TSJ7AO+PiJ8C8wrD/wZsNGZRmZnZhNBvElmDdMuTsiWp\ncB8uMzOb2PpNItcD27YZvhvplvBmZrYY6ffq4Ujg+5LWICWgt0raCNgLeNNYB2dmZs1W5aFUrwc+\nC7yC1Pn+auDzEfGbsQ+vp3j8O5HuUdQcQ93rb0IM/p2INYt/bJg5ifQURc0x1L3+JsTgJGLNUvuP\nDc3MzPp9KNW/GeF0LiJWHnVEZmY2YfTbsP7fpfeTgJcDbwGOHpOIzMxswugriUTE99oNlzQNmDIm\nEZmZ2YQxJg3rktYDromIFUYfUt/rdsN69yhqjqHu9TchBjesW7M0rWF9N+CBMVqWmZlNEP02rF/N\nwqdzAiYDqwEfGsO4zMxsAui3Yf280vt5wEzg4oi4bmxCMjOzicI/NhxnbhNpwvqbEIPbRKxZmtYm\nYmZmi6F+20SeosfTuYhYulJEZmY2YfTbJnIIcDhwLvCnPGxr0h18p5LaR8zMbDHRV5uIpHOA30TE\nd0rDPwC8PiJ2GeP4eonJbSLdo6g5hrrX34QY3CZizVLLXXwlPQq8LCL+URq+PunHhsuPNqB+OYn0\nFEXNMdS9/ibE4CRizVJXw/oDwFvbDN8V+NdogzEzs4ml3zaRqcApkqawoE1kK2An4IAxjMvMzCaA\nvq5EIuI00jPWHwV2z6/HgCkRcWq3+SUdKulKSQ9Lul/SLyVt0ma6qZLukTRb0sWSNu4nTjMzG4yB\n/thQ0m+AM4FppErqL5B6d704Ih7K03wK+AywN3AzcATwGmDDiHiszTLdJtI9ippjqHv9TYjBbSLW\nLLU9HlfSasAewHqkZ6s/IGkr4L6IuLPPZS0PPAy8OSJ+lYfdC5wQEV/O75cF7gcOiYiT2yzDSaR7\nFDXHUPf6mxCDk4g1Sy0N65JeDtwEvA/4ALBSHrUzcFSF9a+YY/h3Xv66pBs6XtCaICIeBy4FXl1h\n+WZmNo767Z31P8C3ImJT4InC8N+Sqpz69XXgLyxopJ9MOl2cUZpuRh5nZmYN0m/vrFcA+7cZfi+w\nej8LknQc6epim0bXR5mZWUf9JpHHSVVQZRvRxy1PJH2N1LNrqNSOMp1Ueb068M/C8NXzuLamTp06\n//+hoSGGhoZ6DcXMbLEwPDzM8PDwmC+331+snwKsQkoADwIvBeYC5wB/iIiDe1jG10lPQhyKiJvb\njG/XsD6D1LB+SpvpG30h44b1Jqy/CTG4Yd2apa7bnqxEav94EbACqRprMnAlsFNEPNpl/m8CewJv\nBm4ojHq01X1X0ieBQ4F9gVuAw0jtLRu5i2/lKGqOoe71NyEGJxFrljq7+ArYAdic1DD/F+D8Xo7k\nkubR/pv8+Yg4sjDd4cD7gecCVwAHRcT1HZbpJNI9ippjqHv9TYjBScSaZeBJRNIkYBjYNyJuGu2K\nx4qTSE9R1BxD3etvQgxOItYsA/+dSEQ8BWxAeq66mZlZ378TOYP0Q0MzM7O+u/guDewnaXvgKtLN\nF+eLiI+PVWBmZtZ8/SaRlwHX5v/Ld9Z1ha+Z2WKmp4Z1SS8F/h4RjWsPccN6T1HUHEPd629CDG5Y\nt2YZdMP61cCqhZX/StLzR7tyMzOb2HpNIuVstR2w3BjHYmZmE0y/vbPMzMzm6zWJBM+sUHYFr5nZ\nYq7X3lkCfiip9QyRZYGTJc0uThQRu4xlcGZm1my9JpEflN7/cKwDMTOziafvGzA2jbv49hRFzTHU\nvf4mxOAuvtYstTxj3czMrMhJxMzMKnMSMTOzypxEzMysMicRMzOrzEnEzMwqcxIxM7PKnETMzKwy\nJxEzM6vMScTMzCpzEjEzs8qcRMzMrDInETMzq8xJxMzMKnMSMTOzypxEzMysMicRMzOrzEnEzMwq\ncxIxW0xMnrwOkmp9TZ68Tt3FYGPMz1gfZ37GehPW34QY6n/GelP2xbrLwRI/Y93MzGrnJGJmZpUN\nPIlI2lbSLyT9U9I8SXu1mWaqpHskzZZ0saSNBx2nmZl1V8eVyLOBvwEfAWaXR0r6FPAx4CBgC+B+\n4AJJyw8ySDMz667WhnVJs4CDIuL0wrB7gRMi4sv5/bKkRHJIRJzcZhluWO8eRc0x1L3+JsRQf4Ny\nU/bFusvBkkWyYV3SusBk4ILWsIh4HLgUeHVdcZmZWXuNSiKkBBLAjNLwGXmcmZk1yFJ1BzAWpk6d\nOv//oaEhhoaGaovFzKyJhoeHGR4eHvPlNqpNJFdn3QpsGRFXFaY7D5gZEfu0WYbbRLpHUXMMda+/\nCTHU3xbQlH2x7nKwZJFsE4mI24HpwA6tYblhfVvgsrriMjOz9gZenZW76q5POjVcAlhL0mbAgxFx\nN3A8cKikm4BbgMOAWcCZg47VzMxGNvDqLElTgIt55nX1DyJi3zzN4cD7gecCV5CqvK7vsLyO1Vlf\n+coJHHPMcWMVeiUPPHAnTahCqLsqx2VQfzWOq7OsaKyqsxbpGzDuuutenHPOZsDbBhvUfPeReibX\nXcZ1HzzqXn8TYlgWeKLG9bfU/zlM9GPOomKsksgi0TtrZKsB69S07sWgeK1HT9CEA7jZWGtUw7qZ\nmU0sTiJmZlaZk4iZmVXmJGJmZpU5iZiZWWVOImZmVpmTiJmZVeYkYmZmlTmJmJlZZU4iZmZWmZOI\nmZlV5iRiZmaVOYmYmVllTiJmZlaZk4iZmVXmB16Y2QAtk5+wWI/VV1+b6dPvqG39iyInETMboHof\nzjVjhh/MNdZcnWVmZpU5iZiZWWVOImZmVpmTiJmZVeYkYmZmlTmJmJlZZU4iZmZWmZOImZlV5iRi\nZmaVOYmYmVllTiJmZlaZk4iZmVXmJGJmZpU5iZiZWWVOImZmVlljk4ikAyXdJmmOpGmSXlN3TGZm\ntrBGJhFJ7wCOB74IvAz4I/AbSWvWGpiZmS2kkUkE+BhwakScGhE3RcRHgPuAD9YcV8MN1x1AgwzX\nHUCDDNcdgC3CGpdEJE0CXgFcUBr1O+DVg49oIhmuO4AGGa47gAYZrjsAW4Q1LokAqwJLAjNKw2cA\nkwcfjpmZdbJU3QGMp2WWmcRyyx3HpEk/qWX9EXOYNauWVZuZDYQiou4YFpKrs2YD74yInxaGnwhs\nEhGvLU3frA0wM5sgIkKjXUbjrkQi4ilJVwE7AD8tjNoBeMYlxVgUgpmZVdO4JJIdB5wu6c/AZaRe\nWc8HvlNrVGZmtpBGJpGIOFvSysBnScnj78DOEXF3vZGZmVlR49pEzMxs4mhiF9+F9Hv7E0kvkTQs\nabakuyV9blCxjrd+ykLSFEnnSLpX0mOSrpG0zyDjHU9Vb4sjaQNJsyQ9Mt4xDkqVspD0UUk3SHpc\n0j2SjhpErOOtwvHidZL+KOkRSTPzd2aDQcU7HiRtK+kXkv4paZ6kvXqYp/pxMyIa+wLeATwJ7Ats\nBJwAzALW7DD9CqRftp8JvBh4K/AI8LG6t6WGsjgUOBLYGlgH+ADwFKnXW+3bM8iyKMw3CZgGnAs8\nUvd21FUWpDbHG4E35n1jM2Cnurdl0GWRt30OcDSwHvBS4LfAzXVvyyjLYWfSLaPeCjwK7NVl+lEd\nN2vf4C794aUvAAANK0lEQVQbdzlwUmnYzcCXOkz/QeAhYOnCsM8Cd9e9LYMuiw7LOAv4Sd3bUldZ\nAF8DvgfsvQglkX6/IxvlA+2GdcfegLJ4Wz6xUmHYEDAXWLnu7RmjMpnVQxIZ1XGzsdVZFW9/shXw\nh4h4sjDsfGANSWuPfZSDMYa3glkR+PdYxVWHqmUh6Q3A64EPj190g1WxLHYBbgVeL+lWSbdL+r6k\n1cYx1HFXsSz+TEoi+0laQtIKwHuBKyPiwfGKtYFGddxsbBKh2u1PJneYXiPMMxGM+lYwkt4I/CcT\nv5t032UhaQ3gu8AeETF7fMMbqCr7xXqkapx3AHsBewIvAn45PiEOTN9lERF3ATuSqn2fIJ2NbwK8\nafzCbKRRHTebnERsjEjaBvhf4MMRcVXd8dTgDOBbETEtv1+cf6C6BLA0sGdEXBYRlwHvAV4lact6\nQxssSauTqjd/AGwBTCFV/9Rzn6QJqslJ5F+kusnVS8NXB6Z3mGd6h+ljhHkmgiplAUDunfJr4LCI\n+O74hDdQVcritcARkp6S9BRwCvBsSU9K2m/8Qh13VcriPuDpiLi1NSAibsnLWWs8ghyQKmVxEPBo\nRHw6Iq6JiP8jJdQpkhanO4aP6rjZ2CQSEU8BrdufFO1A+hV7O38CtpW0dGHYjsC9EXHn2Ec5GBXL\nAknbkRLI4RHxjfGLcHAqlsVLSA832yy/Difdn20zJvBZZ8WyuAxYStK6rQGSXkiqClrcviPPIiWe\nonn5b2OPjeNgdMfNunsPdOk1sDvwOPA+Ur3t10ldz9bM448Gfl+YfkXgXuBHpLrNtwIPAx+te1tq\nKIshUve+Y0hnFa3XqnVvy6DLos38i1LvrH73C5EalC8mJdaXkx44clnd21JDWbwWeBr4HLA+sDmp\ni+8dwHJ1b88oymF50gnSy4DHgMPy+xd0KIdRHTdr3+AeCuQDwG2k/tx/BrYpjDsNuLU0/Sb5SzEb\nuIdUjVP7dgy6LPL7uW1et9W9HXXsF6V5F5kkUqUsSCcTZ+UDxXTgdGC1urejprLYnfTboUdyWZwD\nvKju7RhlGUwhXVGVv/unjlAOlY+bvu2JmZlVtjjV+5mZ2RhzEjEzs8qcRMzMrDInETMzq8xJxMzM\nKnMSMTOzypxEzMysMicRA0DS3yQdXnh/u6SP1xDHK/LT2Gq5j5OkiyWdMMplTMnbsHKXaea2pinP\nUx4/aJKmSpqeY+j6ZLw8zyp5G7bL79fO7zfvY72nSZrodxRerDiJNFT+Ms3LX+In87MfviLpWQMK\nYQvgW71MKGlvSbPGcN2Lwi9gu23DZcDzY+HnVkSn8eNQxh1J2oR0f7EDSLcCP6uP2YvbcFee/69j\nF113VZKXVbdU3QHYiC4gPe9haWBb0m2rlwM+1G5iSUtFxNNjseKIeKCPyUXDD/ySJkW6SV8j5M/p\n/j7GD7KMN0ghRJUrgvm32Y90O4yO2ziOGr8/Lkp8JdJsT0TEzIi4JyJ+DPwQeAuApKF8trWzpCsk\nPU668yaS3iRpmqQ5+Qrmi/nJb+Txq0n6haTZudpqn/KKy9VZklaU9G1J9+blXidpN0lTgFOB5QtX\nTofneSZJOkbS3ZIey3HuWFrPTpJuyMu8BNiwW6Hk2I6QdIakWZLuk3RIaZp5kg6U9FNJjwJfysO3\nk3R5Xt90SccVyyZbStLxkh7Mr2NLy95D0pWSHpE0Q9LZ+cFXZVtLujqva1rxzLhblVdxfKcylvQ5\nSX9rM+9lko4fofxeIumC/Pk/kK96V8jjjgB+VijD8l1ui8vZsrCfXQW8qjR+oSsCpacHniLptrzu\nmyV9osOyP5s/n1mSTpW0TGn8JyX9Iy/nGkl7FEbflv9Oy+u/qDDfPnnfnSPpRkkfLS33/ZJuyuNn\nSvqNJB8nR1L3zcL86ngTtdOAX5aGnQDMjIVvsnYNsD3paXWrAK8j3VhvrzxsCnADcGxhOb8G/kZ6\nLOZmpDu6PkK6ZXxrmtuBjxfeXwb8nXRr7bXzOnchXc1+hPQwn9WA5wHPyvP8L/BHYJscy4Gku6xu\nmsevSbpR3vGk5PF24G7ysy1GKJvbSU+h+zTp7qv7k55M95bCNPNIN9TbN697bWAN0p2Nv0l61vjr\nSc/X+EphvlZZfL0Q00MU7mhKeoTqTnm5WwAXAsOF8a3P5vpcThsDZ5NubLdsYZr5z/Ie6T0wqV0Z\nA/9Bel76FoV1b5Tne0mHsntWjuOnOa5tgZuAnxTGvy8vYzXgeR2Wszzp6Xc/Bl6c94vr83zb5WnW\nzu83z++XAqaS7pa7Vi7bB4F9Svv9I6QqtI3zcv8JHF+Y5kukfbq1L74zl83OefwWufy3z2X1nDx8\n/7ztu+b53kC6e+2Bhfmeyst7AbApcDCwRN3Hgya/ag/Arw4fTCmJAK8kPXjnR/l960D1ltJ8lwCf\nLQ17MzAr/79hnm+rwvi1SLfEbptE8pf1aWDDDrE+4664pMewziXfhrsw/OfAifn/o4AbS+M/S29J\n5PzSsJOBSwvv5xUPPHnYl4Cb2sQ+hwUH94s7xHTXCPG8KK9vjdJn887CNMuTnm+/b2GanpJIpzLO\nw88lPbWx9f4Y0jPCO8W6f47jWYVhrXjXy+/fBsztsn8eQEoAyxWG7cEzk8g8chLpsJyjgd+V9vt2\ny51Dqsp9FulOs9uUlvM14LyR1kt6XsoepWEHA9fl/3fNZbP8WH2PF4eX20SabWelxtSl8usc0hlp\nS5AexFP0CmBLSZ8uDFsCWEbpcaAvIn3R/zx/IRF3Sbp3hDheBtwXETf3EfvmpLrp6yUVH0e7NOnM\nnRzL5aX5/tTj8svT/Yl0ECgql0279f1fjml90pUWHWI6UtKzI+LRXD1zOKlcVmZBHfxapDNb8vv5\ny4mIx3LV08bdN60vJwPfl/Qx0ln0nsDnR5j+RcC1sfCz5v9IOuhuzIKqoG5ay5lTGPYnujx6WNIH\nSFc6a5OSwiTS8zuK2i13aeCFwLL59duFdyuWIp1cdFrvqqSri+9IOqk0X6v95AJSorlD0vnA74Cf\nRcSjI23T4s5JpNkuIZ05Pk16yli7+unHSu+XIB1E2j2xb2bh//FueFyCdGDaghR/0ZxnTj4uymXT\nSc8NsUq9435LOsDsSWo4Xg34A+lAN2i/Ip2Zv41UDbQScGbFZY3rPiHpHaQrho+TEsMjpE4ib+ll\n9vy31T7xRlLVZ9FIHSda872fDicqhROE7UhX358GjpK0RURM5MdrjysnkWabHREdz646+AvpoTpt\nzygl3Uj6Qr2SfKas9JuMdg3DLVcDz5e0UUTc1Gb8k6THq5bnEamb6iUdlnsD6SlqRVuPEEfRVm3m\nu6HLPDcAu5WGbUtqT7m1MOxVpWm2JiXx1kFmFVKV4Z2QGqp55gFYOcY78jTLkx7T+/0uMXbSroyJ\niLmSfkA6u3+YdOY8UlfgG4B9JC0fEa0ku02Ot1v5lZezt6TlClcNWzNyItoGuDwivt0aIGn9NtNt\n2ma5rc9oyfz/OiPsV0/mv/PLKyLuz1fb60fE/3YKMCLmkR7ONCxpKukk4Y3AKSNs1+Kt7vo0v9q/\naNOwXhrfqsdeuTR8R9KX7POkp5VtRDpLPaYwza9JDfJbkapkLmSEhnXSAeaPpOqeHUkNytsDb87j\ntyZVkW1POsAul4efkZfzNmBdUlXbIeR2HFL1Qrlh/S56b1j/FAsa1ucAuxammQe8tTTfGqQG2G+T\nqmPeQGpYL3Y6uJh0MP5aIaZ/Ax/L41clnfl/NW/TG3K5FNsCWp/N33OZbEJqKL6vUDYLfX49vG9b\nxnncuqSrvSeAKV32q+VIDdU/JSW17YAbgbML0/TSJrI86QD7YxY0gF/HCG0ipKuOh0mdEtYnPZb2\nIQpP2yTt9w+TrqZay70L+Hphmi+Qrqr3IVVxbUa6wtgvj1+SBY+FfR6wYh7+vjz8o/mz3QR4D/Cp\nPP4NpOril5GqJt9LurrZZqSyWNxftQfgV4cPprckMr/htTRue1JV2KP5S3oluQdKHr8aqX3lMVId\n8L7AtSycRG5j4d5ZKwLfIfXImU06QL69MP6b+Ys9t7Wc/GU+HPgHqVfWvXm9Ly/MtzPprHY2qUro\nXfSWRA4n9f6aRTo4/3dpmrmUkkge/hpSdcacPN9XgUmF8ReRfmR5Ail5PAAcC+kpoHma3YBbcsyX\n5wNdOYnMJZ3BXpPXNY1CQ2/58+v2vlMZF8ZdCNzS4761Can+/7G8fd8DViiM75pE8nRb5u2aQ7ry\nfAMj986aRGrDeYDUeH4y6UBfTiK/zMNnkE5uTiV3fChMd1DeB+fk6c4H/qswfl/SVeBTwEWF4e/I\nMc/OcVwK7J7HbZM//5m5bK4F9qr7WND0lx+PaxOOpNuBb0TEcXXH0hSSrgPOiIgv1x2LLV7cJmI2\ngeVeR7uRzvq/W3M4thhyErGJyJfPC9xPqn45IBa+D5fZQLg6y8zMKvM9YczMrDInETMzq8xJxMzM\nKnMSMTOzypxEzMysMicRMzOr7P8DZygQ6cD29GEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xab553c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# histogram of predicted probabilities\n",
    "plt.hist(y_pred_prob, bins=8)\n",
    "plt.xlim(0, 1)\n",
    "plt.title('Histogram of predicted probabilities')\n",
    "plt.xlabel('Predicted probability of diabetes')\n",
    "plt.ylabel('Frequency')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Decrease the threshold** for predicting diabetes in order to **increase the sensitivity** of the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# predict diabetes if the predicted probability is greater than 0.3\n",
    "from sklearn.preprocessing import binarize\n",
    "y_pred_class = binarize([y_pred_prob], 0.3)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.36752429,  0.28356344,  0.28895886,  0.4141062 ,  0.15896027,\n",
       "        0.17065156,  0.49889026,  0.51341541,  0.27678612,  0.67189438])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the first 10 predicted probabilities\n",
    "y_pred_prob[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.,  0.,  0.,  1.,  0.,  0.,  1.,  1.,  0.,  1.])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the first 10 predicted classes with the lower threshold\n",
    "y_pred_class[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[118  12]\n",
      " [ 47  15]]\n"
     ]
    }
   ],
   "source": [
    "# previous confusion matrix (default threshold of 0.5)\n",
    "print(confusion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[80 50]\n",
      " [16 46]]\n"
     ]
    }
   ],
   "source": [
    "# new confusion matrix (threshold of 0.3)\n",
    "print(metrics.confusion_matrix(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.741935483871\n"
     ]
    }
   ],
   "source": [
    "# sensitivity has increased (used to be 0.24)\n",
    "print(46 / float(46 + 16))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.615384615385\n"
     ]
    }
   ],
   "source": [
    "# specificity has decreased (used to be 0.91)\n",
    "print(80 / float(80 + 50))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:**\n",
    "\n",
    "- **Threshold of 0.5** is used by default (for binary problems) to convert predicted probabilities into class predictions\n",
    "- Threshold can be **adjusted** to increase sensitivity or specificity\n",
    "- Sensitivity and specificity have an **inverse relationship**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ROC Curves and Area Under the Curve (AUC)\n",
    "\n",
    "**Question:** Wouldn't it be nice if we could see how sensitivity and specificity are affected by various thresholds, without actually changing the threshold?\n",
    "\n",
    "**Answer:** Plot the ROC curve!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAElCAYAAAAskX9OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXfYHFX1xz9fYpAWTAIICS30KgQCSBMCCEgELCgIIgRB\nKf5AUESUJliQjoJIkRCaIE1AOqgvagAhIE2qCS2UFBJIIISSnN8f925msmyZ3Z1tb87neebZnTv3\nzj1zdnbO3HNukZnhOI7jOHmwQLsFcBzHcXoPblQcx3Gc3HCj4jiO4+SGGxXHcRwnN9yoOI7jOLnh\nRsVxHMfJDTcqTq9B0hBJt0uaJmm2pF3bJMccScen9kfGtBXqONfPYtlP5ytl59KIvnKU4UVJo4rS\nlpD0J0mT4/11mKSto6xbtUvWTsONSs5I2jfeZIXtQ0kTJF0iaXCFcjtLukPSm5Lek/SspFMlDaxQ\nZgVJ50p6TtJMSTMkPSjpp5I+1Zwr7Gj+AGwAHAd8CxjbXnHmYnFrddmSSNpJ0gl5njNncr/mOphT\nQoZTgC/Gz28Bd8T0dsvaUXyi3QL0Ugw4ARgPLARsCuwHbCFpXTP7IJ1Z0unAD4BHgV8B04BhwKHA\nnpK2NbPni8rsCFwHfARcDjxO+D03Ao4GtgK+0KwL7DQkCfgccI6ZndtueYq4DLiq+HdvIyOAQ4AT\n2y1IB7MGwbCk2Rq408xOT6U9J2nhDvpt244bleZxl5k9GL+PkvQmcBSwK8EYACBpT4JBuQrY25Ip\nDkZJugToAa6VtKGZzYllVgSuAV4BtjWzN9IVS/opcEDTriwj8c/2XouqGwD0BabndUJJi5jZzEbP\nE3/TTnroqN0CdDpm9mGJ5E8Db5fIm9tvm9c9107c/dU6/kn4M69SlH4CMBU40IrmzDGzsYSm9meA\nr6UO/RhYDNi/2KDEcpPM7FfVBJK0jKQLJL0iaZakFyRdKGnRePxnkorf1kr6vKMP+jZJ20p6QNJ7\nwI8k/UXSS2Xqv0fSuKK0vaILb6akqZKukTSkynWcAEwhtBALMYjxqePrRdnelvSOpL9L2rLoHAW3\n5TaSfivpDWBGlXoXlHSWpEmSpku6UdKyGfW1paSro95mSXot6n5AmeqWkPRHSW9FvZxf+J2K6tpB\nUk90hc5QiDGtnzp+CaGVUoj9zInxgbRsVX8DSavE9Nei/K9Kuk7S0pV0FsuuJukqSROjq/c5SWdV\nKZNJX5IWlXS6pPEx36Ti3zuL7ErFVAr3BuE/V/gtZ8djw1UiphKv8RpJU+I1PiJpt6I8Nd9z3YC3\nVFrHSvFzWiFB0qrA6sAoM3unTLnLCG6KnQmtE4BdgBfM7P56hYl/oIeAgcAFwFPAYOArwBLAu5T3\nbZdKN2A14FrgIkJ842VgHDBC0qZm9kCq/qUI7oRTU2lHA7+M1zmK0Pr4P+BfktY3szfLXM71wOvA\n+cANcXsnnnMt4F+EP+spwPvAd4B7JH3ezP5VdK5zCEb+F0C1uNTFwF7AlcD9wHDg1jK6KU77OrA4\nQfeTgPWiXOsAWxTlFXA1MAH4CTAU+C6wHOG+IF7rXgRX6F0EF+gnY75/StrIzJ4j6Ggw8HngmySt\nlsnxHFV/A0mfiHV8EjiXoPtBBHfrYGBiOYVJWgcYQ3DbXgC8AAwB9gCOKFeuBn2dT3gBO5dwTw8A\nPgusH68hq+zp3+teYG/C7/1v4MLUsY/9tvGeGxPPfQrhXvwKweOwt5n9sejaarnnOh8z8y3HDdgX\nmA1sT3g4LwvsRrhZ3wUGp/LuSvDbHlblnG8BD8Xv/WKZGxqUczTwIbBRhTwnALMrXOMKqbQXYtoX\ni/IuFq/7rKL0Q2L+9eL+8gQX0TFF+VYC3gN+UeV6loh6Ob4o/QZgFrByUd7JwINF1zQHuA9YIIP+\n1ov5zylKvzRe1/FF5y7W10IlzrlnzLd50W8wB7ilKO+JMe+2cX8R4E3goqJ8n4r33hWptHPK/K6Z\nfoPUtX+ljvuuh+CiXKFCnkb0NRX4bYbfraLs8X4eVZQ2o0Ta1lGGrVJpdxFinAsW5b0TeLnee65b\nNnd/NQcRbqDJhLjHtYQbclczey2Vr1/8rNbknUF4SyP1WXczWZKALwO3WXCx5cUEM7s1nWChBXYb\n4U0zze7AM2b2eNzfDegDXKPQdXMJSUsQrvMJYJtahZG0ALADcLOZzXWHWWjxjAaGxRZTmossxq6q\nMILwhvq7ovTfkiFmYWazUnL2i9d6fyw7rDg74a26VD2FlsoOQH/gqiL99SW4XrPoL+tvUIhbfUHS\nIhnOW7jOJQkdSC4xs5ezloOa9PU28FmV72lZl+xZie647Qj/+X5FerwTWFbSakXFst5zXYEbleZg\nBJfB5wl/1L8Q3o5nFeUrGIZ+VKZfKu/0VFq9LEUwTv9t4BylGF8m/WpgUMHvLGkQsGVML7Aa4QHx\nLMEYF7ZJhIdGPeM0liK8wT9X4tjT8XNIKs0qXEMxK8b8/ytKL1XXx5C0XIwRvE14EE6OdRulXSDz\n1BMN4zQS+Qv6u4eP6+8rBF1UI9NvYGYvAmcQOoNMkXS3wpiNst3fIyvHz5rvuxr09SOCS+xlSQ9J\n+rmk1QsHG5A9K6sSdPgz5tXhZKDQayx9L9dyz3UFHlNpHmMt9v6SdBPBL3uVpDUs6RFVeLCtV+4k\nMYC6OME/jJnNkPQaIXjfbMr1v+9TJr1cT69bCX7lPYB/xE8Bf0rlWSDW9wWCOyHrufOm6fXEFtTd\nhBeNXwLPEFyECxDeZut52Svob1/gtSp5q52j6m9gZj+KgexdCa2kM4BjJW1lZs/UWX9JatGXmV0n\n6R/Al6JchwJHSdrXzK5ugewFWc4itNBL8WTRfqvu7ZbgRqUFmNmcGAD9J+EmPzWmPy/pOeDLkr5v\nZu+WKL4v4Y/+l1TazcCBkjaz+oL1kwktnnWr5JsGIGlxM0t31R1SS2VmNkvSzcBukg4lGJXHLASO\nCxR6gb2S40NpMjCTMOagmLXi54t1nvslgmFclfCQK1CqrmI+E/PtY2ZXFBJjx41yrEaqtRLdKQNI\n5B8X5ZliZn+rUn+5l4WafgMze5rwYnSKpHWBRwjB9gOrnL/afVdMTfoys0mEziIXSVqcEFw/kVTL\nuA7Zs1JodXyU4Xfolbj7q0WY2RiCD/hwSQumDp1E6IF1fnwjm4ukjQhjW54gBJwLnEZ4U7s4upIo\nKre0pGMqyGLAn4GdJG1cQezCg2pud0mFbqz7VChTjqsJLpj9CL1xrio6fj0x0F6qcHyI1kT0U98B\n7CKp0PuO6OrYh9D5YXKt543cTtDN/xWlH0r1EdaFVkDx/+9HZcqWquf7MW8hhnUnoUPHTyX1/dgJ\nQjyjwLsxrdjNluk3iDGN4tbqs4Q37v6lysJcl929hG65Q8rlK0EmfUlaIBqRdJ3TCUH3/o3InpV4\nP/0d+E6puE7R79Ar8ZZKcygXqD2d8Mf9NqHrI2b2x/hgPwxYW9IfCS2EjYCRhLftr5nZXHeEmb0g\naQ9Ct8+nJKVH1G8IfIPQpbESPyH0ULtXUqFL8TIE//uXYyD1LkK34FGSTiM8cPYj+NiXz6yNQOGh\ndybhQZB2fRWu6Wjg1PjAuTHmX4ngyriaYIBr5VjCdY6R9DtCl+IDCH74HxblzTwo0Mwek3QVcLCk\n/gR9b0MSl6jEM8DzwJmSlif0WNqJ0FOwXNllJd0G3ELoUnwAcIeZ/TXKM0PSQcAVwH+ibBOBFQju\nrCcJ9x2E6WsE/E7S7YTuvTfX8BtsG8teR3ggi3DPLca8cbJSHEposT8c77vxhPjUHma2epkyWfXV\nD3hV0vXAY4TW+JbAjoQebzQoezmKf7ODCd3YH5d0EeHl7NOEl6m1CMMIypXtflrd3YwwlcZNhD73\ncwhN2mpl1iV0RZxJ6E11XLu7zVWQtdAdcpMSx0QI5I6jqAshYezJnYRuoe8RbvhTgIEV6lqR0Cvo\n+aibGYQHxtFAvwyyDib0vX891jkOOA9YJJVnKKHL43uEN77DKN3lczxwa5X6LorlxlTIswvhTe/t\neD1Px2tcq8q5l4jn/ti9QXCf3JI659+ALbL+bhXqXJDgO59EeID9Oep0HjnK6Gs1gs99GmHg5uWE\nllxx2RMID/21CONhphEequcDi5WQaQtC6+VNQovkeUJPt88W3YdnEGIvH5WQreJvQHB/Xhjv0Xei\n/PcCO2fU2xqEF6IpUcZngNMa1Rehp9uvgYejjmYQXrYOJ/7fsspOuJ8vLkqbXiLtY12KY/oKhP/W\nBEIHnVfiPbh7I/dcN2yKF9cyJO1EuPEfIQzsO8TMLquQvx/hQdxDeEtai/AnOcHMKo7CdRzHcVpL\ny43KPJVLM4DvVTEqBwMnA5+2OMdOjBccZGa1umAcx3GcJtINgfpNgX/avJO23QkMVphY0XEcx+kQ\nusGoLMPH5xKaSPALL9N6cRzHcZxydINRcRzHcbqEbuhS/AZQPJ320oRuqR+b9l1S+4JEjuM4XYyZ\nNdzFuRtaKvcDnysaMLgD8JqZlVyno91d6jplO+GEE9ouQ6dsrgvXhesi2aBUWj603KgoLKKzvqSh\nsf4V4v7y8fjJku5JFfkjYQzGaEnrSPoqYZGqM1ote7fx4osvtluEjsF1keC6SHBd5E87WiobAf8h\nDFBaiDAnzyMk62UvQ7KgFRamWdieMKjsIcLI2NPM7OwWyuw4znzOwIEg9Y5tQLn1RXOg5TEVM7uX\nCsbMzPYrkfZfwqp6Tg2MHDmy3SJ0DK6LBNdFQi26mDYNcvQS9VoyDX6UtDlhOoIhwMKE+ageAe4x\ns7JLh7YDSZanf9BxHAfCG35vfrRIwpoZqJe0oKQfSnqBMDfO7oQ1sRcBNiBMDPiKpBskbdCoIE7+\n9PT0tFuEjsF1keC6SHBd5E8l99dzhCnXjyQsO/uxhWQkrUlYJ/pWSceY2SXNEdNxnPmVgQOD66nd\nNDMO0Zso6/6SNMzMHs50EmlhYIiFhW/airu/HKd30dvdTp1CXu6vrDGVrnlSd5GojuNkwI1Ka2h6\nTKWICZJ+IWmVRit0Wof7ixNcFwmN6qLVXWub6Xby+yJ/shqVXwMjgOck/V3SNyUt1ES5HMfpUApd\na1u1TZ3a7it2aqGm9VRiL69vE4LzfQjrjF+cNfbSCtz95TjNxd1RvZOWxlRKVL4gYR3mUwhLeD5J\nWFb10nY/0d2oOE5zcaPSO2l1TKVQaR9JXwauA04HHgUOAm4gGJiyKzg6rcf9xQmui3QspKdjYxyt\nxu+L/Mk0TYuktQlur70JLZMrgQ3N7IlUnj8DDwDfaoKcjuM0SCEW0tMDw4e3Wxqnt5K1S/Ecwqj6\nPwDXmdn7JfIsBlxkZnvmLmUNuPvLcUrjbiunEq0ep7KamT3faGWtwI2K45TGjYpTiVbHVG6SNLCE\nEJ+S9FSjQjjNwf3FCa6LBNdFgusif7IalTUpHX9ZCPABkY7jOA5Qxf0laUT8egvwTeDt1OE+wOeB\nEWa2WtMkrBF3fzlOadz95VSiJTGVGKAHMKC4MgMmAIeb2Z8bFSQv3Kg4TkJ6ht8BA3x0ulOeVsVU\nFiasnzIJWCHuF7YFzWzFTjIozry4vzhhftVFekqVgkGZX3VRCtdF/lQcp5LqOjyoBbI4juM4XU6l\n9VQOAUaZ2az4vSxmdl4zhKsHd385ToLHUZysND2mIul1YF0zezN+L4eZ2eBGBckLNyqOk+BGxclK\n02MqZjbIzN5MfS+3dYxBcebF/cUJrosE10WC6yJ/Mo1TiWvRO47jOE5Fapn76xHgcuBqM5vYbMHq\nxd1fzvxAuqtwJbwbsZOVVs/9tR5h8OM3CD3B/kYwMH82s5mNCpEnblSc+QGPlTh509K5v8zscTP7\nsZmtCGwPvAz8Fpgo6fJGhXCag/uLE1wXCa6LBNdF/tS0SBeAmd1rZt8lGJf/AXvlLpXjOI7TldS6\nRv2yhPXpvwmsR1iU6woz+31zxKsdd385vRWfcsVpJq2OqexPMCRbEVonVxKMyQuNCpA3blSc3orH\nUZxm0ur1VH4JPAFsZmZrmtnPO9GgOPPi/uIE10WC6yLBdZE/mdaoB5Y1s9lNlcRxHMfpeipN07I2\n8IyZzYnfy2JmHbP6o7u/nG4g6ziTNB5HcZpJK+b+mgMsY2aT4vfCmirpAiLM/dWnUUHywo2K0w14\nfMTpNFoRU1kLmJz6vnbqc+2ifacDcX9xgusiwXWR4LrIn0oTSj6beuWfFvc/tgE1NuLDtPqSxkt6\nT9JYSVtWyb+jpPskTZc0WdKNkjpmCWPHGTgwtD6ybgMGtFtix2kOWbsUzwYGmdmkovQlgEm1uL8k\n7UGY4uUgYAzwPWA/YC0zm1Ai/xDgaeBs4CJgMeBUYGUzW71Efnd/OS3H3VlOt9PqcSpz4ytF6SsA\nT5vZopkrlB4AHjWzg1JpzwHXmtkxJfLvBlxNWL7YYtpw4K/AUmY2tSi/GxWn5bhRcbqdloxTkXSq\npFMJwfnjC/txOwO4ijB+JROS+gLDgLuLDt0FbF6m2EPAh8ABkhaQ1A8YCTxYbFCceXF/cYLrIsF1\nkeC6yJ9q41Q+Fz8FbEp4uBf4gDC6/tc11Lck0Aconjp/IrBdqQJm9rKkHYBrgfMIhvARYKca6nWc\nhqjWBdhjJI4TqGhUzGwzAElXAQea2fSWSJVC0tLAxcClhJZRP+AkgpHZplSZkSNHMmTIEAD69+/P\n0KFDGT58OJC8mcwP+8OHD+8oebp5f9q04Zh1jjyN7hfoFHnatV9I6xR5Wrnf09PD6NGjAeY+L/Og\npgklG64suL9mAt8ws+tT6ecC65jZx4yEpJOAL5rZsFTassArwJZmdl9Rfo+pOLnjMROnt9P0mIqk\nayQtnvpedstamZl9CDxMmDY/zfaEnmClWAQoniJmTjX5HfcXp6lFF6W6B/cm95bfFwmui/yp9FCe\nTTJ6fnaVrRbOBEZK2l/SmpJ+Q1hN8nwASSdLuieV/1ZgQ0nHSVpV0obAJYSFwh6usW7Hqcq0aaFV\nkt58ehTHyUZL3V9zK5UOAo4iGJMngcPNbEw8dgmwlZmtksq/e8y/OsF99gBwtJk9U+Lc7v5yGsJd\nXc78SEvHqZSofEFgE2Ccmb3eqBB54kbFaRQ3Ks78SEvXU5F0oaQD4/dPAPcD/wDGSyqOjzgdgvuL\nE1wXCa6LBNdF/mQNdH8RGBu/7wp8GhgCnEzo3us4juM4madpmQWsamYTJF0IvGtmR8R5uR43s8Wb\nK2Z23P3lNIq7v5z5kVYvJzwRWFPSAsCOhHm3ABal9t5fjuM4Ti8lq1G5DPgT8B/CNCuFubs2Bp5t\nglxODsyv/uLS09D3+LT0kfn1viiF6yJ/Mq1Rb2bHSXoGWAG42szeT5U/vVnCOU49FMaZpOnpgdTM\nHI7jNIm2jFNpJh5TcTwm4ji1k1dMJVNLJVa4NLAFoefXPG4zMzuvUUEcx3Gc7ifrOJWvAy8SFsv6\nGXBcaju2SbI5DeL+4gTXRYLrIsF1kT9ZWyonE9Yy+YmZfdBEeRzHcZwuJus4lXeBz5jZ+OaL1Bge\nU3E8puI4tdPqcSp3EpYBdpy2Ubqr8PzXJdhxOpmsRuVm4DRJx0r6kqQR6a2ZAjr109v8xaWmpC+1\nlZqmvrfpohFcFwmui/zJGlMZFT9LzfNlhAGRjuM4znxO1pjKJysdTw2GbDseU+m9eKzEcZpHS8ep\ndJLRcBzHcTqXzGu8S/q2pIclTY2zEyPpSElfaZZwTmO4vzjBdZHgukhwXeRP1sGP3yOMVbkGWDhV\nbjLw/eaI5jiO43QbWWMqTxHWhL9Z0gxgfTMbL2ldoMfMlmy2oFnxmErvxWMqjtM8Wj1OZSXgsRLp\n7xPWVHEcx3GczEblRWD9Euk7Ak/nJo2TK+4vTnBdJLguElwX+ZN1nMpZwLmS+gICNoyTTB4LHNws\n4RzHcZzuIvN6KpIOBY4hTH0PMAU40cx+1yTZ6sJjKr0Xj6k4TvPIK6ZS8yJdkpYjuM1e6cSntxuV\n3osbFcdpHq0O1M/FzCYAywDDJS3WqABO83B/cYLrIsF1keC6yJ+KRkXSgZJ+XJR2A3A/8FfgKUmr\nNVE+Zz6i2izEPvuw43Q+Fd1fkv4NnGdml8b9XYEbgAMJvb5+AzxtZvu0QNZMuPure3H3luO0j1bN\n/bUq8HBq/4vALWZ2cRTiaODiRoVwHMdxegfVYioLAzNS+5sBPan954Glc5bJyQn3Fye4LhJcFwmu\ni/ypZlReIq74KGlJYG1gTOr40sBbzRHNcRzH6TaqxVSOAb4HnAtsCyxnZmumjh8KfMnMPt9sQbPi\nMZXuxWMqjtM+WhVT+TXwKeBbwBvA7kXHtwOua1QIx3Ecp3dQ0f1lZrPN7CgzW8vMtjGzx4uOf9nM\nzm+uiE69uL84wXWR4LpIcF3kT82DH/NA0iGSxkt6T9JYSVtmKHO4pKclzZL0qqRftUJWx3EcJztl\nYyqSngBOBG40s4/KnkBaCfgBMMHMTqlaobQHcDlwECHo/z1gP2CtOFq/VJkzgRHAkcCTBJfcIDO7\no0Rej6l0KR5TcZz20fS5vyTtCJwKDAbuAsYCrwGzgAGEnmBbAkOB84Gfm9m0DII/ADxqZgel0p4D\nrjWzY0rkXwN4AljXzJ7LcH43Kl2KGxXHaR9Nn/vLzO40s/WBPYCZhFH0owkj6k8HNojfVzSzH2Q0\nKH0JXZTvLjp0F7B5mWK7AuOAEZLGSXpB0mhJS1Wrb37H/cUJrosE10WC6yJ/qq6nYmZ/A/5W2Fdj\nTYElgT7AxKL0iYSeZKVYGRhCMG6F6WDOAG4mDMZ0HMdxOoSap75vqDJpEPAqsJWZ/SuVfhywl5mt\nVaLMBcABwOpmNi6mrQY8C3zWzB4qyu/ury7F3V+O0z5aNU4lb6YAs/n41C5LE8bBlOJ14KOCQQEw\ns+clzQZWAB4qLjBy5EiGDBkCQP/+/Rk6dCjDhw8Hkuau7/u+7/v+/Lzf09PD6NGjAeY+L/OgpS0V\nKBuof5YQqD+2RP7tgTuAVc3shZi2CmHesU3MbGxRfm+pRHp6eubeTJ3GwIEwrSgKN2AATJ3anPo6\nWRetxnWR4LpI6NaWCsCZwGWSHiJ0KT4YGEToQYakk4GNU1O/3AM8AoySdAQg4Czg/mKD4nQP06a5\nq8txeiMtb6kASDoIOIpgTJ4EDjezMfHYJYSYyyqp/EsDvwW+ALxH6C32QzObXOLc3lLpAjx+4jid\nRcvXqI/dgXcEVgEuMbPpkpYH3jaz6Y0KkhduVLoDNyqO01m0dI16SUOAp4A/ErrzLhkP/RA4rVEh\nnOZQCMo5ros0rosE10X+ZJ376zeE+McSBPdTgT9TfnyJ4ziOM5+Ryf0l6U1gczN7VtIMYH0zG19o\nwZjZIs0VMzvu/uoO3P3lOJ1FS91fMV+fEunLMe9yw47jOM58TFajcjdwaGrfJC0KnEAYQ+J0IO4v\nTnBdJLguElwX+ZN1nMqRQI+kx4GFgMuA1QmtlG81STbHcRyny6ilS/FiBAMyjNDCeQS41Mw6yv3l\nMZXOo9Wj5x3HqZ2WjlORtAnwsJnNLkrvAwwzswcbFSQv3Kh0Hh6Ud5zOp9WB+vsJ3YmL6R+POR2I\n+4sTXBcJrosE10X+ZDUqAkq9aw4gLODlOI7jOJXdX5KuiV93A24B3k8d7gOsD7xgZts3TcIacfdX\n6ygVKymFx08cp/Np1SzFhRiKgDmpfQgj668Eft+oEE534jMNO45TTEX3l5ntaWZ7AqcAexf247av\nmZ1gZuUW13LajPuLE1wXCa6LBNdF/mQap2JmP2m2II7jOE73U8s4lT2BPQlL+C6YPmZma+cvWn14\nTKV1eFdhx+k9tHrq+8MJKzOOA9YE/ga8AgwGrmtUCMdxHKd3kLVL8cHAd83sCOBD4Ewz25GwGuNS\nzRLOaQz3Fye4LhJcFwmui/zJalSWBx6I398D+sXvlwO75y2U0z4GDgxurSzbgAHtltZxnE4j6zQt\nLwBfNbP/SBoLXGBmF0n6PHCNmQ1stqBZ8ZhKY3icxHHmT1o9TcvfgZ3j90uBsyXdDlwD3NSoEI7j\nOE7vIKtROYiwNj1mdg5wCCFQ/6t4zOlA3F+c4LpIcF0kuC7yJ+s4lQ+AD1L7lxJaLE4HknX6lFJ4\nnMRxnEbIPE6lZGFpZ+AkM9swP5Eaw2MqHhdxHKd2WhZTkfQtSZdLGiVpw5i2qaQHgOuBJxoVwnEc\nx+kdVDQqkr4PjAI2IIymvzem3UYI3q9kZvs2XUqnKumuwAUXlvuLE1wXCa6LBNdF/lSLqXwX+D8z\nu0DS9sCdwFeB1c1sStOlczLjMwY7jtMJVFtP5V1gbTN7Ke5/AGxtZh272uP8GlPxOIrjOI3QqpjK\nwoQR9AXeByY2WqnjOI7TO8kyTmWkpEMkHUJwl+1d2E+lOy2g0hQqpboCu784wXWR4LpIcF3kT7WY\nyiTgiNT+W4TJJdMYcF6eQjml8biJ4zidTkPjVDqR3hxT8biJ4zjNotVzfzmO4zhOVdyodDilxp9k\nxf3FCa6LBNdFgusifzLN/eW0D4+jOI7TTbQlphJ7jB0JDAL+CxxuZv/KUG414BHAzGzxMnl6VUzF\n4yiO47SCro2pSNoDOBv4BTAUuA+4XdJyVcr1Ba4Cepoto+M4jlMfmY2KpL6Sdpb0fUmLx7TlC99r\n4AhglJmNMrNnzeww4HU+3lW5mFOBx4DraqxvvsX9xQmuiwTXRYLrIn8yxVQkDQHuBpYGFgH+AkwH\nfkgYdX9gxvP0BYYBpxUdugvYvEK5LwIjCBNbfj1LXY7jOE7rybpG/U3ANOA7wFRgfTMbL2lr4GIz\nWzVTZdIg4FVgq3QMRdJxwF5mtlaJMoOBh4AvmdlYSfsC53hMxXEcJz/yiqlk7f21JbC5mX0ozVPn\nS8DgRoWowuXAeWY2Nu43fNGO4zhOc8hqVBYA+pRIXw6YUUN9U4DZBDdamqWBN8qU2Qb4nKSfxX0B\nC8QZkw8xsz8UFxg5ciRDhgwBoH///gwdOpThw4cDiQ+1W/ahh56e+sqn/cWdcj3t2i+kdYo87dx/\n9NFHOfx3Nyl6AAAa2klEQVTwwztGnnbun3322V39fGhkv6enh9GjRwPMfV7mQVb31zXAm2Z2sKQZ\nwHqEecFuBF6rZaGuuGLko2Z2UCrtWeBaMzu2RP61i5K+DPwU2DjW/XZRfnd/RXp6eubeTPM7rosE\n10WC6yIhL/dXVqOyAqEr7zvAWsADwOqEVsqWZlaulVHqXLsDlwHfA8YQen3tR1i3ZYKkk4GNzezz\nZcr3+pjKwIFh0COEUfRTp7ZXHsdxej8tjamY2cuS1gP2ATYkuMP+BFxqZrW4vzCzayQNBI4hDH58\nEtjJzCbELMsAK9Vyzt6Gj6J3HKdbydpS+VSxm6lT6Q0tlbx6fHnTPsF1keC6SHBdJLR6RP0bkq6T\n9KU41sRxHMdxPkbWlsquwF7ALsAs4FrgcjMb01zxasdbKo7jOLXT0kB9qtLFgN0IBmZb4BXgSjM7\nrlFB8sKNiuM4Tu20ZUJJM3vHzC41sx2B9YG3Cd17nQ4kPUZjfsd1keC6SHBd5E9NRkXSJyV9TdKf\nCVPQLwGc3hTJHMdxnK4ja0xlO+CbwFdj0vXAFUBPp/ma3P3lOI5TO60e/Pg+cAfBkNxsZu83WnGz\ncKPiOI5TO62OqQwysy+Z2bWdbFCceXF/cYLrIsF1keC6yJ+yI+olLWJmM+PuLEmLlMubyuc0QGF6\nlgED2i2J4zhOfZR1f0maTWihTJI0ByjrkDGzUjMYt4Vudn+528txnHbRirm/RhAW5Cp898ed4ziO\nU5GyMRUzu9PMPorf74j7JbfWievUgvuLE1wXCa6LBNdF/mQK1EuaKWmpEukDJXk8pQ4GDgzurvTm\nsRTHcbqdrF2K5wDLmNmkovTBwHgzW6hJ8tVMt8RUPH7iOE4n0ZL1VCQdEr8aMFLSO6nDfYCtgeca\nFcJxHMfpHVRzfx0XNwE/TO0fBxwJLAUcUra001bcX5zgukhwXSS4LvKnYkvFzAYBSLofGGFm01oi\nleM4jtOV1DT1fTfgMRXHcZzaaXpMRdKpwIlm9m78XhYzO6pRQRzHcZzup1JM5XNA39T3ctuWzRSw\nN5HuRtyK7sPuL05wXSS4LhJcF/lTtqViZpuV+u7Uz7Rp7vJyHKd3U3dMRdJywBuFUfedQifHVDyO\n4jhOp9LSqe8l/UzS3qn9W4CXgTckbdSoEI7jOE7vIOt6KiOBcQCSdgQ2A4YD1wK/boZg3UipqVfa\nOQ2L+4sTXBcJrosE10X+VBynkmIZYEL8PgK41sz+Iel14MGmSNaFeMzEcZz5naxzf70G7GZm90t6\nBjjOzK6VtAbwkJkt3mxBs9LOmIrHTBzH6VZaMvdXihuBKyQ9DXyasF49wPpEt5jjOI7jZI2pHA6M\nAl4FvmBmM2L6isCFzRDMaRz3Fye4LhJcFwmui/zJ1FIxsw+AX5ZIPy13iRzHcZyuJfM4FUkDgYOA\ntQlT4f8XuNDMplYs2GI8puI4jlM7rR6n8llC7OQg4JPAQoQp7/8naeNGheh2Cl2JfeVGx3Hmd7LG\nVM4gBOtXNrOvm9nXgZWBm4GzmiVct1DoSjy1o9ps7i9O47pIcF0kuC7yJ2vvr2HAAekpWczsozh7\n8dimSOY4juN0HVnHqUwCvmlmdxel7wBcbmZLN0m+mmlVTGXgwNBCgeD26rRWiuM4Ti20NKYCXANc\nLGk3SYPi9jXgonisJiQdImm8pPckjZVUdvp8SVtLulHSa5LelfSYpP1qrTNvCi6vTnR7OY7jtIus\nRuVI4HbgasJ0LROAq4DbgB/VUqGkPYCzgV8AQ4H7gNvjrMel2Bx4HNgNWAf4PXChpG/UUu/8iPuL\nE1wXCa6LBNdF/mQdpzILOFDSj4HVYvLzZvZWHXUeAYwys1Fx/zBJXwAOBo4pUffJRUnnS9qGYGSu\nrqN+x3Ecp0lUjalIGgxsS+hKfK+Z/a/uyqS+wEzgG2Z2fSr9XGAdM9sm43luB14xs++WONbUmEoh\nluJxFMdxehMtmftL0uYEF1dhwsgPJO1tZtfVWd+SQB9gYlH6RGC7LCeQtDPByG1epwwN4TMRO47j\nlKea++sXwAME19T7cf90oF6j0hCStgCuBA41s4fL5Rs5ciRDhgwBoH///gwdOpThw4cDiQ+13n3o\noaen/vKt3E/7iztBnnbuF9I6RZ527j/66KMcfvjhHSNPO/fPPvvsXJ8P3bTf09PD6NGjAeY+L/Og\novtL0pvANmb2eNzvB7wFLFFPPKUR91fsIXYrcKyZnVMhX1PdX900FUtPT8/cm2l+x3WR4LpIcF0k\n5OX+qmZU5gDLmNmkVNoMYD0ze6GuCqUHgEfN7KBU2rOEhb+OLVNmK+AWwjouv6ly/pqNSnrMSTU8\nluI4Tm+kleuprC5pyXTdwGqSFi4kmNlTNdR5JnCZpIeAMQTX2iDgfABJJwMbm9nn4/5wgkH5HXC1\npMJAy9lmNqWGesvicRLHcZx8yDJO5V7gidS2CGGRrieAJ+NnZszsGsL6LMcA/yEE3Hcys8JyxcsA\nK6WK7AssTBgr81pq82WMq5COJ8zvuC4SXBcJrov8qdZSWasZlZrZ+cSWSYlj+5XYb8oI+nT3YMdx\nHKdxMq+n0i3UElPppqC74zhOM2n13F+O4ziOUxU3Kr0Y9xcnuC4SXBcJrov8caPiOI7j5IbHVHrX\n5TuO49RFW2IqkhaTtH4cGe84juM485DJqEhaVNJlwHTgYWD5mH6upI9NV+90Bu4vTnBdJLguElwX\n+ZO1pXIysAZhoOKsVPpdwNfzFspxHMfpTrKuUf8SsLuZ/TvO/bW+mY2XtCrwiJktXuUULcNjKo7j\nOLXT6pjKUsCkEumLNiqA4ziO03vIalQeBkak9gvv998G7s9VIic33F+c4LpIcF0kuC7yJ9Ma9YTJ\nH2+TtGYs8z1J6wDDga2bJJvjOI7TZWQepyJpGHAUMIzQwnkE+JWZPdI88WrHYyqO4zi105JFuroR\nNyqO4zi109JAvaRFKm2NCuE0B/cXJ7guElwXCa6L/MkaU3mHJDhfij45yOI4juN0OVnHqexYlNQX\n2AA4gLBu/GVNkK0u3P3lOI5TOx0RU5G0B7C3me3SqCB5Uc2oFFZ7hLDi49SpLRLMcRyng+mURbrG\nAts2KkQrmTYttE7Mer9BcX9xgusiwXWR4LrIn7qNiqQFge8Br+YnjuM4jtPNZI2pTGbeQL2A/sAH\nwD5mdn1zxKudau4vj6M4juN8nLzcX1l7fx1btD8HmAzcZ2al5gTrOAqxlAED2i2J4zhO76Wq+0vS\nJ4APgZvM7IK4XWRmN3aLQYEkltLb4yhp3F+c4LpIcF0kuC7yp6pRMbOPgHOBTzZfHMdxHKebyRpT\n6QHONLObmy5Rg6RjKt592HEcJxutjqmcC5whaTBhGvx30wfN7KlGBWkGBZeX4ziO0xqydim+BlgF\nOA/4N/BE3J6Mn04H4v7iBNdFgusiwXWRP1lbKms1VQrHcRynV1AxpiJpFPB9M5vROpEaIx1T8TEp\njuM42WjJ3F+SZgODuqnrsBsVx3Gc2mnV3F8NV+C0D/cXJ7guElwXCa6L/MkSqPd3fcdxHCcT1dxf\nc8hgVMysYxbpcveX4zhO7bRynMp3gbcarSiNpEOAI4FBwH+Bw83sXxXyr0sYK7MJ8CZwoZn9PE+Z\nHMdxnMbJ4v76i5ldX2mrpcK4sNfZwC+AocB9wO2SliuTvx9wN/A6MAz4PvAjSUfUUu/8iPuLE1wX\nCa6LBNdF/lQzKs1wHh0BjDKzUWb2rJkdRjAYB5fJvzewMLCvmT1tZjcApwA/aIJsvYpHH3203SJ0\nDK6LBNdFgusif1ra+0tSX0Jr4+6iQ3cBm5cptinwTzP7IJV2JzBY0op5ytfbeOutXL2WXY3rIsF1\nkeC6yJ+KRsXMFsh5jMqSQB9gYlH6RGCZMmWWKZNfFco4juM4baDRNeqdDubFF19stwgdg+siwXWR\n4LrIn0xT3+dWWXB/zQS+kQ7wSzoXWMfMtilR5lJgoJntkkrbiDCx5cpm9lJRfu9E7DiOUwetnPo+\nF8zsQ0kPA9sD6V5j2wPXlil2P/BrSQum4io7AK8VG5RYh88C4DiO0yba4f46ExgpaX9Ja0r6DWG8\nyvkAkk6WdE8q/x8JrZvRktaR9FXgx8AZrRbccRzHqUxLWyoAZnaNpIHAMQRj8iSwk5lNiFmWAVZK\n5Z8uaXvgd8BDwDTgNDM7u7WSO47jONVoaUzFcRzH6d10Xe8vSYdIGi/pPUljJW1ZJf+6knokzZT0\niqTjWiVrs6lFF5K2lnSjpNckvSvpMUn7tVLeZlLrfZEqt5qkGZKmN1vGVlGPLiQdLulpSbMkvSrp\nV62QtdnU8bzYUdJ9kqZLmhz/M6u1St5mIelzkm6SNEHSHEn7ZChT37PTzLpmA/YAPgC+DawB/BaY\nASxXJn8/wmj9qwirV34VmA4c0e5raYMufgKcBGwGDAEOAj4k9MRr+/W0Uhepcn2BscBfgOntvo52\n6YIQ53wG2DneG+sDX2j3tbRaF/Ha3wNOBlYG1gPuAJ5r97XkoIudCFNjfRV4B9inSv66n51tv9ga\nFfMAcH5R2nPAL8vkP5gwGeaCqbRjgFfafS2t1kWZc/wJuLbd19IuXQBnARcD+/Yio1Lrf2SN+OBd\nvd2yd4AudosvWkqlDQdmE4Y1tP2actLLjAxGpe5nZ9e4v3yKl4Q6dVGKxQkdH7qWenUh6YvACODQ\n5knXWurUxa7AOGCEpHGSXpA0WtJSTRS16dSpi4cIRuUASQvEyWxHAg+a2dRmydqh1P3s7Bqjgk/x\nkqYeXcyDpJ2BbYEL8hWt5dSsC0mDgQuBb5rZzOaK11LquS9WJrh99gD2IUzguiZwc3NEbBk168LM\nXiaMgTsJeJ/wpr4OsEup/L2cup+d3WRUnJyQtAVwJXComT3cbnnawOXAeWY2Nu7PzwNmFwAWBPY2\nszFmNgb4FvBZSRu3V7TWImlpgjv0UmAjYGuCq6jcwGynBN1kVKYQfJtLF6UvDbxRpswbZfJbhTLd\nQD26ACD2frkNONbMLmyOeC2lHl1sA5wg6UNJHwJ/ABaT9IGkA5onatOpRxevAx+Z2bhCgpk9H8+z\nQjOEbBH16OJ7wDtmdrSZPWZh4cBvAVtLqsWt3Buo+9nZNUbFzD4EClO8pNkeGFOm2P3A5yQtmEor\nO8VLt1CnLpC0FcGgHG9m5zRPwtZRpy7WJSwQt37cjifM2rA+XfxWWqcuxgCfkDR3wLGkVQiuo/nt\nP7IIwRClmRM/u+ZZmRP1Pzvb3ROhxl4LuwOzgP0Jft/fELq5LRePnwzck8q/OPAaYaqXdQjd4t4m\nLF/c9utpsS6GE7oSnkJ44yhsS7b7WlqtixLle1Pvr1rvCxEC1H8nGNoNgB5gTLuvpQ262Ab4CDgO\nWBXYkNCl+EVg4XZfT4O6WJTw0jQUeBc4Nu4vX0YXdT87236xdSjnIGA8oT/5Q8AWqWOXAOOK8q8T\n/yQzgVcJbp+2X0erdRH3Z5fYxrf7OtpxXxSV7TVGpR5dEF4u/hQfGm8AlwFLtfs62qSL3Qljl6ZH\nXdwIrNnu68hBD1sTWl3F//9RFXRR17PTp2lxHMdxcmN+8xM6juM4TcSNiuM4jpMbblQcx3Gc3HCj\n4jiO4+SGGxXHcRwnN9yoOI7jOLnhRsVxHMfJDTcqvQhJfeKqbru2W5Z6kbRKvIb1quS7XNINrZKr\n05B0maSj2y1Hqyh1b0taW9L9cVXH52q9/yXtL6nhKe0l3SDpsEbP01two9JBSLok/ilmx8/C94oP\n2FYi6ecpuT6S9JKkCyQNzKmK8YSptZ+M9W0X61u8KN8hhLUumkaq7sLvMUXSPZI+W+N5cjX2koYS\nVvL7bSptN0l3SpoU68ptAkRJ20j6a7z+dyX9Lxq1RfKqoxpmNptwX9yeSv4lYRaA1YBNy+SpxBXA\n6oWdeG//pw7xTgKOk7RoHWV7HW5UOo+7CX+MwjaI+IDtIJ4kyLY8YWbXrwCj8jixBSaZWWEiPxFm\nRlVRvhlm1op15Y3w4FmGMH/aNOC2Go1o3lPrH0pYsTO9FsyiwL+AHxJkzgVJ6xImIR0LbEWYuuMg\nwpTwC1YomjvxvvgwlbQqYSGpCRYX0SqRp9L53jezKcXJdcj1KDAB2KvWsr2Sds9J49s8c+1cAtxc\n4fhOwD8JD7Y3CX/21VPH+xDm99k1lfYzwoR4swgTxF2cOibC2vXjCPP7PEaVNeuBnwOPFKUdR1jU\n6BNxfz3gr/GcUwhrVPRL5S8cf5swx9IjwOfisVXiNayX+j479XlhzHcFcEP8fjDwaglZrwGuS+1/\niTBz7Xvxmk8C+la41u1inYun0oZGWXZMpW1CWFFwcrymfwAbp46/krqGOaTWPK9Dpj5RZzuVOb50\nrGPznO7JH1JlfriopzmElTQfjdfyIDC0KN+WUTczo07OBRYruh+PIiz5O4swS/JJxfd26nv6vvgp\npe//ZQnrrE8hTKT4cOpeOwCYFr/vX+KcexHWVvlz0XUsQDAi/5dKOxH4W7ueHZ20eUulu1gEOJ2w\nTOpwwp/kZkl9SmWWtAfwfeC7hLe6XQiT6hX4NWGlvwOBtQgzGP9BUvF04dV4n/CH7htdAHcSjN5G\nhNlNtyKstFjgauDleHwo4UE6K3W88LY4njDBHwQXxyDgB0V5IEyGuISkbQsJcSnYnQkLciFpBDAa\nODte6/6E1Q5PzHB9iudYFNgv1p1+G+4Xz70FwcA8Dtwu6VPx+MbxHPsSWjybNiDTBoRWydgKefLk\nDWCZuGxCNU4l/D7DCEbjL5I+CXNddncQlhZYl7Ae/DDgoqLyRxFeXNYCvkZ4eM+DJW6ucYR7eBBw\nVnE+SYsRXsIGE+79dZhXt0ZyH11J+B3+SzDMg4DronwjJC2ZKrcTMJDwYlPgQWBTSZ8oo5v5h3Zb\nNd+SjdBS+ZDgWihst1bIvzjhjWqTuD/PmxrwI4Krqk+JsosR3ig/W5R+DnBjhTrnaakQ/vzjgH/E\n/YMJb4ULpfIU3mRXjPvvAHuWOf/clkqq7DythZh+ObGlEvdvYt5W2EiCYesb98cAPy46x27AWxWu\ntSD39PhbFFoa9wELVCgnYBKwe6nfJZWvHpl2Az6ocDzvlsoCBMM3m2BgbiK8qCxRQk9fS6X1I7Ta\n9on7VwK/Lzr3RrFc/3gvzwL2KyNHqVbI08BPy+WJ9+JbwKfKnHN/YGq5ezuV/hTwg9T+dcAfi/Js\nEHW0fB567+bNWyqdx70E109hAam5KxFKWlXSHyWNk/Q2YTpqKL9C358If9YXJF0Ug7l947F1gU8C\nd0uaUdhifStXkXE9SdMlzQSeIBiVfeKxNYHHzCzd8igsirRW/DwTuFTS3ZJ+Imm1KvVl4Qrgq6lF\nhfYixB0KLYphwPFF13oZYcXHJSqc1whumw2AbxBaT/taEvNB0qclXSjpWUlvEYzQQKqvnFiPTAsT\nWoYNIWmlVL3TJR1ZKp+ZzTGzkcByBFfYK8DRwDOSVk9nBR5IlZtBeOtfOyYNA0YWXWsPwQisQmhF\n9AX+1ui1pRgK/MfM3m7wPH8gtFCJLZZdYlqa9+Lnwg3W1fV4U63zmGlmL5Q5dhvhAX4AIT4yh/C2\nVjJgamYvxwf25wlvk2cBx0rajKSTxoh4rjQfVJHxGcIfaw5hJbhMgVGiq8HMjpd0Wax7R+Bnkg4w\ns8sznqcUNxNcFbtIGgNsy7yuDgEnAKW6IVfrVvqihU4B/4sulRslrWfBDQPhLXxx4DCCW+99wstB\ntUB2PTJNARaR9Akz+6jK+SvxMuGlpcCblTKb2euE67xS0rHA/4AjCa7VLCwAXEBYKKu448IEgtHp\nVC4DfilpE4KL81UzKzZ+hY4bk1sqWQfiRqVLkPRpQlxkPzMbE9M2oUoPPjN7H7gVuFXS6YQ/8KaE\n4PgHBJfUv2oU54MKhu9p4JuSFjazwtvblgSD8nRKrv8RusT+VtKFBFdEwaik4yUFA1cybpQ63/uS\nrifEiJYHXi7oKfIfYA0zG1/16iozmtAx4WBCoBnCg+Y7ZnYngKRBBJ9/QbbZkmaXuIZ6ZCp0eV2b\nELupi2gQ69KFmb0laSLBhVpAhPvqOpgb01oHOD8efwRYu9x9I+m/BNfvduTUk5Cgq90l9TeztzLk\n/4AS95mZTZF0E+Ee3ZRwDxSzLvCSmU1rQN5egRuV7mEK4e31u5LeIDw4T+Xja2rPRdJ+8euDhKD+\nNwl/nP+Z2XRJZwFnxeDiPwlv25sB75tZvX/sywlrvl8q6URgKeD3wJ9iy2lRwtKl1xF6pS1LeCj3\npEVPfS+sh72zpNuB98zs3TJ1X0EYo7AGYRnUNCcSWhgTCMHi2cBngGFm9pOsF2dmcyT9BvippD9E\nN99zwLckPUzQ4anM2/EAQstgO0n3EfT7Vj0ymdlESU8QDPVcoyJpAMHdVggorybpXeB1M5uU9fqK\nkXQw4YH5Z0IreRGCK2hNQisrzfGSphFiLycS4lB/isdOBu6TdC6hRfkOwTCOMLOD4/14LnCqpI8I\n3aOXJPQgu5D6uIIQV7xR0jGEFvl6hDjKP0vkfxFYSdL6hJevGWZWeKn5A3AL4Zm5c4mynyN0UHHa\nHdTxLdmo3qV4W0IMo9D9d9v4fa94vA/hwVQIVH4FuJ9gjKYTfN47Fp3zMILvexYwkdBDZ5sKMpQM\nZhbl+QxwD8GQTSE8RBaLxz5JeOC/QPBDTwDOAxaJx1eJ17Be6nzHEx4IH5F0KZ4nUB/TRHh4f0SJ\nJWCBHQjG8x1CAPffwEEVrqNcJ4HFok6PjPtDo25nEgzMNwjB3XQQeVfgWYJr7Ll6ZYplDgHuK0pL\nd4lNbz+tdK4M9+SGhG61z8ffczIhRvaNVJ6CnkYQDF25LsUbxfur0JX8UeC4ojxHE4zXLMJD/oRS\n93ZMK9ZxqTzLEQzbVIKRewjYMqWzdKB+IcLLzrR4nr2KZHsBuL2EjhaO17NBK58Xnbr5csKO02VI\nWogQ19rDzP7dAfJsRxinM8BaMyC15UhamPBi8x0zu67o2GHADmZWqgUz3+HuL8fpMsxslqR9SFxd\nTpOQJIIL9weEFlapThWzCN2sHdyoOE5XYmb/aLcM8wkrE1x/LwMjLdWVvIDVH/Pplbj7y3Ecx8kN\nH/zoOI7j5IYbFcdxHCc33Kg4juM4ueFGxXEcx8kNNyqO4zhObrhRcRzHcXLj/wElxmlZNiDK3AAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xadf99b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# IMPORTANT: first argument is true values, second argument is predicted probabilities\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.0])\n",
    "plt.title('ROC curve for diabetes classifier')\n",
    "plt.xlabel('False Positive Rate (1 - Specificity)')\n",
    "plt.ylabel('True Positive Rate (Sensitivity)')\n",
    "plt.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ROC curve can help you to **choose a threshold** that balances sensitivity and specificity in a way that makes sense for your particular context\n",
    "- You can't actually **see the thresholds** used to generate the curve on the ROC curve itself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define a function that accepts a threshold and prints sensitivity and specificity\n",
    "def evaluate_threshold(threshold):\n",
    "    print('Sensitivity:', tpr[thresholds > threshold][-1])\n",
    "    print('Specificity:', 1 - fpr[thresholds > threshold][-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 0.241935483871\n",
      "Specificity: 0.907692307692\n"
     ]
    }
   ],
   "source": [
    "evaluate_threshold(0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity: 0.725806451613\n",
      "Specificity: 0.615384615385\n"
     ]
    }
   ],
   "source": [
    "evaluate_threshold(0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AUC is the **percentage** of the ROC plot that is **underneath the curve**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.724565756824\n"
     ]
    }
   ],
   "source": [
    "# IMPORTANT: first argument is true values, second argument is predicted probabilities\n",
    "print(metrics.roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- AUC is useful as a **single number summary** of classifier performance.\n",
    "- If you randomly chose one positive and one negative observation, AUC represents the likelihood that your classifier will assign a **higher predicted probability** to the positive observation.\n",
    "- AUC is useful even when there is **high class imbalance** (unlike classification accuracy)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.73782336182336183"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate cross-validated AUC\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "cross_val_score(logreg, X, y, cv=10, scoring='roc_auc').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Confusion matrix advantages:**\n",
    "\n",
    "- Allows you to calculate a **variety of metrics**\n",
    "- Useful for **multi-class problems** (more than two response classes)\n",
    "\n",
    "**ROC/AUC advantages:**\n",
    "\n",
    "- Does not require you to **set a classification threshold**\n",
    "- Still useful when there is **high class imbalance**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion Matrix Resources\n",
    "\n",
    "- Blog post: [Simple guide to confusion matrix terminology](http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/) by me\n",
    "- Videos: [Intuitive sensitivity and specificity](https://www.youtube.com/watch?v=U4_3fditnWg) (9 minutes) and [The tradeoff between sensitivity and specificity](https://www.youtube.com/watch?v=vtYDyGGeQyo) (13 minutes) by Rahul Patwari\n",
    "- Notebook: [How to calculate \"expected value\"](https://github.com/podopie/DAT18NYC/blob/master/classes/13-expected_value_cost_benefit_analysis.ipynb) from a confusion matrix by treating it as a cost-benefit matrix (by Ed Podojil)\n",
    "- Graphic: How [classification threshold](https://media.amazonwebservices.com/blog/2015/ml_adjust_model_1.png) affects different evaluation metrics (from a [blog post](https://aws.amazon.com/blogs/aws/amazon-machine-learning-make-data-driven-decisions-at-scale/) about Amazon Machine Learning)\n",
    "\n",
    "\n",
    "## ROC and AUC Resources\n",
    "\n",
    "- Lesson notes: [ROC Curves](http://ebp.uga.edu/courses/Chapter%204%20-%20Diagnosis%20I/8%20-%20ROC%20curves.html) (from the University of Georgia)\n",
    "- Video: [ROC Curves and Area Under the Curve](https://www.youtube.com/watch?v=OAl6eAyP-yo) (14 minutes) by me, including [transcript and screenshots](http://www.dataschool.io/roc-curves-and-auc-explained/) and a [visualization](http://www.navan.name/roc/)\n",
    "- Video: [ROC Curves](https://www.youtube.com/watch?v=21Igj5Pr6u4) (12 minutes) by Rahul Patwari\n",
    "- Paper: [An introduction to ROC analysis](http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf) by Tom Fawcett\n",
    "- Usage examples: [Comparing different feature sets](http://research.microsoft.com/pubs/205472/aisec10-leontjeva.pdf) for detecting fraudulent Skype users, and [comparing different classifiers](http://www.cse.ust.hk/nevinZhangGroup/readings/yi/Bradley_PR97.pdf) on a number of popular datasets\n",
    "\n",
    "## Other Resources\n",
    "\n",
    "- scikit-learn documentation: [Model evaluation](http://scikit-learn.org/stable/modules/model_evaluation.html)\n",
    "- Guide: [Comparing model evaluation procedures and metrics](https://github.com/justmarkham/DAT8/blob/master/other/model_evaluation_comparison.md) by me\n",
    "- Video: [Counterfactual evaluation of machine learning models](https://www.youtube.com/watch?v=QWCSxAKR-h0) (45 minutes) about how Stripe evaluates its fraud detection model, including [slides](http://www.slideshare.net/MichaelManapat/counterfactual-evaluation-of-machine-learning-models)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comments or Questions?\n",
    "\n",
    "- Email: <kevin@dataschool.io>\n",
    "- Website: http://dataschool.io\n",
    "- Twitter: [@justmarkham](https://twitter.com/justmarkham)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    @font-face {\n",
       "        font-family: \"Computer Modern\";\n",
       "        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "    }\n",
       "    div.cell{\n",
       "        width: 90%;\n",
       "/*        margin-left:auto;*/\n",
       "/*        margin-right:auto;*/\n",
       "    }\n",
       "    ul {\n",
       "        line-height: 145%;\n",
       "        font-size: 90%;\n",
       "    }\n",
       "    li {\n",
       "        margin-bottom: 1em;\n",
       "    }\n",
       "    h1 {\n",
       "        font-family: Helvetica, serif;\n",
       "    }\n",
       "    h4{\n",
       "        margin-top: 12px;\n",
       "        margin-bottom: 3px;\n",
       "       }\n",
       "    div.text_cell_render{\n",
       "        font-family: Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
       "        line-height: 145%;\n",
       "        font-size: 130%;\n",
       "        width: 90%;\n",
       "        margin-left:auto;\n",
       "        margin-right:auto;\n",
       "    }\n",
       "    .CodeMirror{\n",
       "            font-family: \"Source Code Pro\", source-code-pro,Consolas, monospace;\n",
       "    }\n",
       "/*    .prompt{\n",
       "        display: None;\n",
       "    }*/\n",
       "    .text_cell_render h5 {\n",
       "        font-weight: 300;\n",
       "        font-size: 16pt;\n",
       "        color: #4057A1;\n",
       "        font-style: italic;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 0.5em;\n",
       "        display: block;\n",
       "    }\n",
       "\n",
       "    .warning{\n",
       "        color: rgb( 240, 20, 20 )\n",
       "        }\n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"]\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}