module-3/day-6-pipelines-roc/pipelines_roc_auc-enkeboll.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src= \"img/pipelines.png\" style=\"height:450px\">\n",
    "\n",
    "\n",
    "[Image Source](https://towardsdatascience.com/using-functiontransformer-and-pipeline-in-sklearn-to-predict-chardonnay-ratings-9b13fdd6c6fd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Agenda__\n",
    "\n",
    "- Pipelines and Composite estimators\n",
    "\n",
    "- Why do we need them?\n",
    "\n",
    "- How to use them in sklearn: accessing a particular object in pipe and changing parameters\n",
    "\n",
    "- ROC Curve\n",
    "\n",
    "- Area under ROC Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pipelines\n",
    "\n",
    "__What is a Pipeline?__\n",
    "\n",
    "_Transformers:_ Any object with .transform method. Ex: StandardScaler, OneHotEncoder.\n",
    "\n",
    "_Estimators:_ Any object with predict method. Ex: RandomForestClassifier, LinearRegression etc.\n",
    "\n",
    "_Pipelines:_ A tool for combining transformers with estimators. \n",
    "\n",
    "\n",
    "__Why do we need pipelines?__\n",
    "\n",
    "- Convenience and encapsulation\n",
    "\n",
    "Even though we train 10 transformers and 5 estimators we will call fit and predict once.\n",
    "\n",
    "- Joint parameter selection - here emphasize preprocessing part\n",
    "\n",
    "We can put pipelines into gridsearch and find best parameters for all the estimators at once.\n",
    "\n",
    "- Safety\n",
    "\n",
    "Pipelines help avoid leaking statistics from your test data into trained model.\n",
    "\n",
    "[Pipelines and Composite Estimators](https://scikit-learn.org/stable/modules/compose.html#combining-estimators)\n",
    "\n",
    "\n",
    "\n",
    "## Usage of Pipelines\n",
    "\n",
    "The Pipeline is built using a list of (key, value) pairs, where the key is a string containing the name you want to give this step and value is an estimator object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('imputer', SimpleImputer()), ('scaler', StandardScaler()),\n",
       "                ('clf', LogisticRegression(penalty='l1'))])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "estimators = [('imputer', SimpleImputer()),\n",
    "              ('scaler', StandardScaler()),\n",
    "              ('clf', LogisticRegression(penalty='l1'))]\n",
    "\n",
    "pipe = Pipeline(estimators)\n",
    "\n",
    "pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Your Turn__\n",
    "\n",
    "- Create your own pipeline. You can use the same transformers and estimators with different parameters and 'name'.\n",
    "\n",
    "- You can create a new pipe with a scaler also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.impute import KNNImputer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('knnimputer', KNNImputer()),\n",
       "                ('scaler', MinMaxScaler(feature_range=(0, 100))),\n",
       "                ('clf', DecisionTreeClassifier(max_depth=3))])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "your_estimators = [('knnimputer', KNNImputer()),\n",
    "                   ('scaler', MinMaxScaler((0, 100))),\n",
    "                   ('clf', DecisionTreeClassifier(max_depth=3))]\n",
    "your_pipe = Pipeline(your_estimators)\n",
    "your_pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sklearn also gives us \"make_pipeline\" which is almost the same thing but with make_pipeline you don't have to give names.\n",
    "\n",
    "__Your Turn__\n",
    "\n",
    "-  [Check documentation: 6.1.1.1.1 Construction](https://scikit-learn.org/stable/modules/compose.html) and use `make_pipeline` to construct a pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('knnimputer', KNNImputer()),\n",
       "                ('minmaxscaler', MinMaxScaler(feature_range=(0, 100))),\n",
       "                ('decisiontreeclassifier',\n",
       "                 DecisionTreeClassifier(max_depth=3))])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "your_make_pipe = make_pipeline(KNNImputer(), MinMaxScaler(feature_range=(0, 100)),\n",
    "                               DecisionTreeClassifier(max_depth=3))\n",
    "your_make_pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('imputer', SimpleImputer()), ('scaler', StandardScaler()),\n",
       "                ('clf', LogisticRegression(penalty='l1'))])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing steps\n",
    "\n",
    "We have multiple ways to access and object in the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SimpleImputer()"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## note that these will all give the simple imputer object\n",
    "# pipe[0]\n",
    "\n",
    "# pipe['imputer']\n",
    "\n",
    "pipe.steps[0][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SimpleImputer()"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## We can also access a particular object by named_steps\n",
    "\n",
    "pipe.named_steps.imputer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('scaler', StandardScaler()),\n",
       "                ('clf', LogisticRegression(penalty='l1'))])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## We can 'slice' pipelines to create sub-pipes\n",
    "\n",
    "pipe[1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access to the parameters\n",
    "\n",
    "Parameters of the estimators in the pipeline can be accessed using the \n",
    "\"estimator__parameter\" syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StandardScaler(with_std=False)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.named_steps.scaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'copy': True, 'with_mean': True, 'with_std': False}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('imputer', SimpleImputer()), ('scaler', StandardScaler()),\n",
       "                ('clf', LogisticRegression(C=10, penalty='l1'))])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.set_params(clf__C = 10)\n",
    "\n",
    "pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('imputer', SimpleImputer()),\n",
       "                ('scaler', StandardScaler(with_std=False)),\n",
       "                ('clf', LogisticRegression(C=10, penalty='l1'))])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.set_params(scaler__with_std=False)\n",
    "pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipelines in action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Dataset info](https://archive.ics.uci.edu/ml/datasets/Heart+Disease)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('./data/heart.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(303, 15)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>ChestPain</th>\n",
       "      <th>RestBP</th>\n",
       "      <th>Chol</th>\n",
       "      <th>Fbs</th>\n",
       "      <th>RestECG</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>ExAng</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>Slope</th>\n",
       "      <th>Ca</th>\n",
       "      <th>Thal</th>\n",
       "      <th>AHD</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>typical</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>fixed</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>asymptomatic</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>normal</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>asymptomatic</td>\n",
       "      <td>120</td>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>reversable</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  Age  Sex     ChestPain  RestBP  Chol  Fbs  RestECG  MaxHR  \\\n",
       "0           1   63    1       typical     145   233    1        2    150   \n",
       "1           2   67    1  asymptomatic     160   286    0        2    108   \n",
       "2           3   67    1  asymptomatic     120   229    0        2    129   \n",
       "\n",
       "   ExAng  Oldpeak  Slope   Ca        Thal  AHD  \n",
       "0      0      2.3      3  0.0       fixed   No  \n",
       "1      1      1.5      2  3.0      normal  Yes  \n",
       "2      1      2.6      2  2.0  reversable  Yes  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df.shape)\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>ChestPain</th>\n",
       "      <th>RestBP</th>\n",
       "      <th>Chol</th>\n",
       "      <th>Fbs</th>\n",
       "      <th>RestECG</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>ExAng</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>Slope</th>\n",
       "      <th>Ca</th>\n",
       "      <th>Thal</th>\n",
       "      <th>AHD</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>typical</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>fixed</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>asymptomatic</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>normal</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>asymptomatic</td>\n",
       "      <td>120</td>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>reversable</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>nonanginal</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>nontypical</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  Sex     ChestPain  RestBP  Chol  Fbs  RestECG  MaxHR  ExAng  Oldpeak  \\\n",
       "0   63    1       typical     145   233    1        2    150      0      2.3   \n",
       "1   67    1  asymptomatic     160   286    0        2    108      1      1.5   \n",
       "2   67    1  asymptomatic     120   229    0        2    129      1      2.6   \n",
       "3   37    1    nonanginal     130   250    0        0    187      0      3.5   \n",
       "4   41    0    nontypical     130   204    0        2    172      0      1.4   \n",
       "\n",
       "   Slope   Ca        Thal  AHD  \n",
       "0      3  0.0       fixed   No  \n",
       "1      2  3.0      normal  Yes  \n",
       "2      2  2.0  reversable  Yes  \n",
       "3      3  0.0      normal   No  \n",
       "4      1  0.0      normal   No  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop(columns='Unnamed: 0')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our dataset we have 303 patients and 13 independent variables and 1 binary target variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are working with classification problems it is always good practice to check the class balance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "No     0.541254\n",
       "Yes    0.458746\n",
       "Name: AHD, dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['AHD'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that approximately 54% of the patients are in the class 0 which refers to 'no presence' of a heart disease. Consequently, 45% of the patients have a heart disease. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Train-Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 303 entries, 0 to 302\n",
      "Data columns (total 14 columns):\n",
      "Age          303 non-null int64\n",
      "Sex          303 non-null int64\n",
      "ChestPain    303 non-null object\n",
      "RestBP       303 non-null int64\n",
      "Chol         303 non-null int64\n",
      "Fbs          303 non-null int64\n",
      "RestECG      303 non-null int64\n",
      "MaxHR        303 non-null int64\n",
      "ExAng        303 non-null int64\n",
      "Oldpeak      303 non-null float64\n",
      "Slope        303 non-null int64\n",
      "Ca           299 non-null float64\n",
      "Thal         301 non-null object\n",
      "AHD          303 non-null object\n",
      "dtypes: float64(2), int64(9), object(3)\n",
      "memory usage: 33.3+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For model evaluation we split our data into two parts: Train - Test\n",
    "\n",
    "X = df.drop('AHD', axis=1)\n",
    "y = df['AHD']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=77, stratify=y,\n",
    "                                                    test_size=.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "No     0.541322\n",
       "Yes    0.458678\n",
       "Name: AHD, dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's check number of 1 and 0 in both datasets\n",
    "y_train.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "No     0.540984\n",
       "Yes    0.459016\n",
       "Name: AHD, dtype: float64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choosing a Perfomance Metric for Model Evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Model Selection vs Model Evaluation__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Model Selection/Model Comparison: What is the best parameters for a given model. Between different models which one is better models the reality.\n",
    "\n",
    "Ex: If we are working with an app that runs a machine learning algorithm model selection is choosing the process of choosing a final algorithm to deploy.\n",
    "\n",
    "\n",
    "- Model Evaluation: After selecting a 'best' model with model selection how this model will perform in the 'real' case.\n",
    "\n",
    "Ex: Model evaluation is where we want to predict how successful this algorithm will be."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Available tools in sklearn](https://scikit-learn.org/stable/model_selection.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src= 'img/table.png' width = 450 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Prep Before Training a Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[A good blog post on handling categorical variables](https://www.bogotobogo.com/python/scikit-learn/scikit_machine_learning_Data_Preprocessing-Missing-Data-Categorical-Data.php)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Your Turn__\n",
    "\n",
    "- Convert Categorical Variables to OneHotEncoding\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "si = SimpleImputer(strategy='most_frequent')\n",
    "X_train_imputed = si.fit_transform(X_train)\n",
    "X_test_imputed = si.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps.append(('imputer', si))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>ChestPain</th>\n",
       "      <th>RestBP</th>\n",
       "      <th>Chol</th>\n",
       "      <th>Fbs</th>\n",
       "      <th>RestECG</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>ExAng</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>Slope</th>\n",
       "      <th>Ca</th>\n",
       "      <th>Thal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>typical</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>fixed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>asymptomatic</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>asymptomatic</td>\n",
       "      <td>120</td>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>reversable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>nonanginal</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>nontypical</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  Sex     ChestPain  RestBP  Chol  Fbs  RestECG  MaxHR  ExAng  Oldpeak  \\\n",
       "0   63    1       typical     145   233    1        2    150      0      2.3   \n",
       "1   67    1  asymptomatic     160   286    0        2    108      1      1.5   \n",
       "2   67    1  asymptomatic     120   229    0        2    129      1      2.6   \n",
       "3   37    1    nonanginal     130   250    0        0    187      0      3.5   \n",
       "4   41    0    nontypical     130   204    0        2    172      0      1.4   \n",
       "\n",
       "   Slope   Ca        Thal  \n",
       "0      3  0.0       fixed  \n",
       "1      2  3.0      normal  \n",
       "2      2  2.0  reversable  \n",
       "3      3  0.0      normal  \n",
       "4      1  0.0      normal  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0    176\n",
       "1.0     65\n",
       "2.0     38\n",
       "3.0     20\n",
       "Name: Ca, dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.Ca.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an encoder object. This will help us to convert\n",
    "# categorical variables to new columns\n",
    "encoder = OneHotEncoder(drop='first')\n",
    "\n",
    "# Create an columntransformer object.\n",
    "# This will help us to merge transformed columns\n",
    "# with the rest of the dataset.\n",
    "catvars = [2, 6, 10, 12]\n",
    "ct = ColumnTransformer([('encoder', encoder, catvars)],\n",
    "                       remainder='passthrough', sparse_threshold=0)\n",
    "\n",
    "X_train_transformed = ct.fit_transform(X_train_imputed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct = ColumnTransformer([('encoder', encoder, catvars)],\n",
    "                       remainder='passthrough', sparse_threshold=0.9)\n",
    "\n",
    "X_train_transformed_sparse = ct.fit_transform(X_train_imputed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>217.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>111.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>281.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>226.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>178.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>256.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>202.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>249.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>144.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>238</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>138.0</td>\n",
       "      <td>234.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>239</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>220.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>271.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>241</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>212.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>132.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>242 rows × 18 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       0    1    2    3    4    5    6    7    8     9   10     11     12  \\\n",
       "0    0.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0  1.0  55.0  1.0  140.0  217.0   \n",
       "1    0.0  1.0  0.0  0.0  1.0  1.0  0.0  0.0  1.0  62.0  1.0  120.0  281.0   \n",
       "2    0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0  42.0  1.0  140.0  226.0   \n",
       "3    0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0  1.0  48.0  1.0  130.0  256.0   \n",
       "4    0.0  1.0  0.0  0.0  1.0  0.0  0.0  1.0  0.0  29.0  1.0  130.0  204.0   \n",
       "..   ...  ...  ...  ...  ...  ...  ...  ...  ...   ...  ...    ...    ...   \n",
       "237  0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0  1.0  46.0  1.0  120.0  249.0   \n",
       "238  0.0  0.0  0.0  0.0  1.0  0.0  0.0  1.0  0.0  53.0  0.0  138.0  234.0   \n",
       "239  0.0  1.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0  44.0  1.0  120.0  220.0   \n",
       "240  0.0  1.0  0.0  0.0  0.0  1.0  0.0  1.0  0.0  49.0  0.0  134.0  271.0   \n",
       "241  0.0  0.0  0.0  0.0  1.0  0.0  0.0  1.0  0.0  66.0  1.0  112.0  212.0   \n",
       "\n",
       "      13     14   15   16   17  \n",
       "0    0.0  111.0  1.0  5.6  0.0  \n",
       "1    0.0  103.0  0.0  1.4  1.0  \n",
       "2    0.0  178.0  0.0  0.0  0.0  \n",
       "3    1.0  150.0  1.0  0.0  2.0  \n",
       "4    0.0  202.0  0.0  0.0  0.0  \n",
       "..   ...    ...  ...  ...  ...  \n",
       "237  0.0  144.0  0.0  0.8  0.0  \n",
       "238  0.0  160.0  0.0  0.0  0.0  \n",
       "239  0.0  170.0  0.0  0.0  0.0  \n",
       "240  0.0  162.0  0.0  0.0  0.0  \n",
       "241  0.0  132.0  1.0  0.1  1.0  \n",
       "\n",
       "[242 rows x 18 columns]"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_train_transformed_sparse.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps.append(('onehot', ct))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.0, 0.0, 0.0, ..., 1, 5.6, 0.0],\n",
       "       [0.0, 1.0, 0.0, ..., 0, 1.4, 1.0],\n",
       "       [0.0, 0.0, 0.0, ..., 0, 0.0, 0.0],\n",
       "       ...,\n",
       "       [0.0, 1.0, 0.0, ..., 0, 0.0, 0.0],\n",
       "       [0.0, 1.0, 0.0, ..., 0, 0.0, 0.0],\n",
       "       [0.0, 0.0, 0.0, ..., 1, 0.1, 1.0]], dtype=object)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_transformed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Don't forget!!__\n",
    "\n",
    "- Apply the same transformations to the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(61, 18)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_transformed = ct.transform(X_test_imputed)\n",
    "X_test_transformed.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['encoder__x0_nonanginal',\n",
       " 'encoder__x0_nontypical',\n",
       " 'encoder__x0_typical',\n",
       " 'encoder__x1_1',\n",
       " 'encoder__x1_2',\n",
       " 'encoder__x2_2',\n",
       " 'encoder__x2_3',\n",
       " 'encoder__x3_normal',\n",
       " 'encoder__x3_reversable',\n",
       " 'x0',\n",
       " 'x1',\n",
       " 'x3',\n",
       " 'x4',\n",
       " 'x5',\n",
       " 'x7',\n",
       " 'x8',\n",
       " 'x9',\n",
       " 'x11']"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct.get_feature_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Scaling Features__ \n",
    "\n",
    "-- Let's go back to the column transformer.\n",
    "\n",
    "[Different Scalers and Their Effect on Data](https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "ss = StandardScaler()\n",
    "X_train_final = ss.fit_transform(X_train_transformed)\n",
    "X_test_final = ss.transform(X_test_transformed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps.append(('scaler', ss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('imputer', SimpleImputer(strategy='most_frequent')),\n",
       " ('onehot', ColumnTransformer(remainder='passthrough', sparse_threshold=0,\n",
       "                    transformers=[('encoder', OneHotEncoder(drop='first'),\n",
       "                                   [2, 6, 10, 12])])),\n",
       " ('scaler', StandardScaler())]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(242, 18)"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_final.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(61, 18)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_final.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Check sklearn for documentation of Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)\n",
    "\n",
    "\n",
    "[For solvers](https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg = LogisticRegression(penalty='none', max_iter=10000)\n",
    "steps.append(('lr', log_reg))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('imputer', SimpleImputer(strategy='most_frequent')),\n",
       " ('onehot', ColumnTransformer(remainder='passthrough', sparse_threshold=0,\n",
       "                    transformers=[('encoder', OneHotEncoder(drop='first'),\n",
       "                                   [2, 6, 10, 12])])),\n",
       " ('scaler', StandardScaler()),\n",
       " ('lr', LogisticRegression(max_iter=10000, penalty='none'))]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline(steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('imputer', SimpleImputer(strategy='most_frequent')),\n",
       "                ('onehot',\n",
       "                 ColumnTransformer(remainder='passthrough', sparse_threshold=0,\n",
       "                                   transformers=[('encoder',\n",
       "                                                  OneHotEncoder(drop='first'),\n",
       "                                                  [2, 6, 10, 12])])),\n",
       "                ('scaler', StandardScaler()),\n",
       "                ('lr', LogisticRegression(max_iter=10000, penalty='none'))])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('imputer', SimpleImputer(strategy='most_frequent')),\n",
       "                ('onehot',\n",
       "                 ColumnTransformer(remainder='passthrough', sparse_threshold=0,\n",
       "                                   transformers=[('encoder',\n",
       "                                                  OneHotEncoder(drop='first'),\n",
       "                                                  [2, 6, 10, 12])])),\n",
       "                ('scaler', StandardScaler()),\n",
       "                ('lr', LogisticRegression(max_iter=10000, penalty='none'))])"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8760330578512396\n"
     ]
    }
   ],
   "source": [
    "# What is this score?\n",
    "print(pipe.score(X_train, y_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8360655737704918\n"
     ]
    }
   ],
   "source": [
    "print(pipe.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = pipe.predict(X_train)\n",
    "\n",
    "score = pipe.score(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_train, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAItCAYAAAA9haCkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de7zsc73H8ddnE7YQksshB8UuqUh0LyFFQqcUnXNCjh1RqaPrUemq00U3JbvcKhGiVFLarjkl13LdEZFS7nLb2Ht/zh/f39hjrJtlzZq1f9/Xcz/mMWt+v9/8ft+ZPWvNZ97f7+87kZlIkiS12bRBN0CSJKnfLHgkSVLrWfBIkqTWs+CRJEmtZ8EjSZJab/FBN0CSJA3G9I32mbRTte+/+OCYrGMNxYRHkiS1ngWPJElqPbu0JEmqVdSTe9TzSCVJUrVMeCRJqlUMdBzxpDLhkSRJrWfCI0lSrRzDI0mS1B4mPJIk1coxPJIkSe1hwiNJUq0cwyNJktQeJjySJNXKMTySJEntYcEjSZJazy4tSZJq5aBlSZKk9jDhkSSpVg5aliRJag8THkmSauUYHkmSpPYw4ZEkqVaO4ZEkSWoPEx5JkmrlGB5JkqT2MOGRJKlWjuGRJElqDxMeSZJq5RgeSZKk9jDhkSSpViY8kiRJ7WHBI0mSWs8uLUmSajXN09IlSZJaw4RHkqRaOWhZkiSpPUx4JEmqlV8tIUmS1B4mPJIk1coxPJIkSe1hwiNJUq0cwyNJktQeJjySJNXKMTySJEntYcIjSVKtHMMjSZLUHiY8kiTVyjE8kiRJ7WHBI0mSWs8uLUmSauWgZUmSpPYw4ZEkqVYOWpYkSWoPEx5JkmrlGB5JkqT2MOGRJKlWjuGRJElqDxMeSZJqZcIjSZLUHiY8kiTVyrO0JEmS2sOER5KkWjmGR5IkqT1MeCRJqpVjeCRJktrDgkeSJLWeXVqSJNXKQcuSJEntYcIjSVKtHLQsSZLUHiY8kiRVKkx4JEmS2sOER5KkSpnwSJIktYgJjyRJtaon4DHhkSRJ7WfCI0lSpRzDI0mS1CImPJIkVcqER5IkqUVMeCRJqpQJjyRJUotY8EiSpNazS0uSpErZpSVJktQiJjySJNWqnoDHhEeSJLWfCY8kSZVyDI8kSVKLmPBIklQpEx5JkqQWMeGRJKlSJjySJEktYsEjSVKlImLSLmNoy+ERcXNEXNa1bMWIOC0irm6uV2iWR0R8NSKuiYg/RMTzRtu/BY8kSZoKjgRe07Psg8DszFwXmN3cBtgaWLe5zAQOGW3nFjySJNUqJvEyisw8G7i9Z/H2wFHNz0cBO3Qt/04WvwWWj4jVRtq/BY8kSeq7iJgZERd0XWaO4W6rZOZNAM31ys3y1YG/dG13Y7NsWJ6lJUlSpSbzLK3MnAXMmqDdDdXwHOkOJjySJGmq+kenq6q5vrlZfiPw1K7t1gD+NtKOLHgkSarUVDpLaxgnA7s0P+8C/Lhr+Vubs7VeCNzV6foajl1akiRp4CLiGGAzYKWIuBH4GPBZ4LiI2B24Adix2fwUYBvgGuA+YLfR9m/BI0mSBi4zdx5m1RZDbJvA3o9l/xY8kiRVyq+WkCRJahETHkmSalVPwGPCI0mS2s+ER5KkSjmGR5IkqUVMeCRJqpQJjyRJUouY8EiSVCkTHkmSpBYx4ZEkqVImPJIkSS1iwiNJUq3qCXhMeCRJUvuZ8EiSVCnH8EiSJLWIBY8kSWo9u7QkSaqUXVqSJEktYsIjSVKlTHgkSZJaxIRHkqRa1RPwmPBIkqT2M+GRJKlSjuGRJElqERMeSZIqZcIjSZLUIiY8kiRVyoRHkiSpRUx4JEmqlAmPJElSi5jwSJJUq3oCHhMeSZLUfhY8kiSp9ezSkiSpUg5aliRJahETHkmSKmXCI0mS1CImPJIkVaqigMeER5IktZ8FT0tFcV1EZEQ8fdDtmWoiYqWIODgiro2IuRHxt4j4RUTsMOi2jVdEvCQizouI+5v/+3eN8X6viohzI+KuiPhHRJwUETO61q/VvI6GuswZYn8zI+Ky5nn9R0T8YIht9oiIq5ttLoyILXrWvyIizoiImyPigeb/6YsRsVzPdt+MiKsi4p6IuCMizo6ILYc43urN47onIm5t/u+X7tnmzGEe41Jd2xw5wnOxc9d2y0XElyPizxFxX0RcGRH7RteAiYjYbJj9fLanXW+OiBMj4qZm/a5DPL4tI+IHEXF9c7zLImKfiFisZ7uPR8SlEfHPiLg7Ii6IiDcPsb9nRcQvm33dGhGHRMQyQ2z35Ig4NCL+3rzuroqIt/Zup6krIibtMmh2abXXi4C1mp93Aj41uKZMLRHxBOAMYGng08CfgDWArYAtgB8NrnXjE6Wo/QXwU+BDwKbAQRFxX2Z+e4T7bQz8jPKYPw4sC3wUOC0iNsjMfwI3UV5P3aYDvwR+3rO/TwH7UF5v5wOrAK/o2WYn4JvAAcCvgd2An0bEJpl5WbPZisDFwDeAW4BnNe2bAWzb046DgTnAEsDuwM8j4mWZ+dvmeIs3z82DwJuB5YGDmuv/6HlcZwAf7ln2QNfPn2za3m0v4C3Ar7qWHQm8vNnXNcArm2MG8KWe+/87cG3X7b/2rH8j5Xf5p8B/MbSZlNfz/sBfgJcCXwTWBv67a7vlmrZdAcxv9n1sRMzPzBMAIuJJwOnAHynP15OBzwGrAQ9/IGiKz7OBe4B3ArcC61P+H6QpJzJz0G1QH0TE1yhvJJcBy2bmswbcpIdFxFKZOXeAx38V5c1608w8v2ddZJ9/KSJiembeP8H7PJTyprp+Zs5rln0DeB2w5nCPqUkTdgXW6Lrfc4DfA9tk5s+Hud+OwHHACzPzvGbZs4A/AK/JzNNGaOsc4NzMfFtze1pzvN9nZm8B0n2/PYBZwJMz8/ZhtlkMuA74UWa+q1m2M/A94OmZeV2z7E3AscCMzLy6WXYmcGtmvnG4NgxzzMuBGzJz6+b20sDdwL6Z+bWu7U4EVs/MFzS3N6MUWM/uKvSG2v+0zFzQJCx3A7tl5pE926yUmbf2LPsM8B5g+czsLtp6938ucFtmbtfc/hClaF4zM+9slm0H/BjYJDMvaJZ9llIwPXuiX8+aPOu9/9RJKwL++LnXDDTmsUurhZo/+jsCJwOHA+s3b2K92/1rRBzTRNb3RcQfIuItXeunR8Tnmpj8gSjdJAd2rc+I2KdnnwdExK1dt3dttts0SpfB/cD7mnWfbeL1eyLixog4OiJWHaKdezTbdbpIToiIJ0XEayNiQUSs3bP92s3y7YZ5ipZvrv/eu6K3MIiI50TETyLizqadv2sKpu5j/SgWdhH8JHq6EJvH/94oXRy3AJd2rds+SrfC3CjdAp9rEqjHamvgxE7R0jiWklxtMML9ngDc13O/OzvNG+F+OwPXdYqdxi7An0YpdtYB1qMUSwBk5gLg+OYxjOS25nrYBCEz51Pa373N1sD5nWKn8SNK4vOaUY45oub3an3gmK7Fi1P+tt7Vs/mdjOObi5rnZ7Rtbh1i8cXAUpRUZyS38cjna0Pggk6x0/glkMBru5btBhxmsaNFhQVPO21O6Uo4FjgBeIjyBvWwiFgZ+A2wCbAfJQk4DHhqsz4on+j2Ar4ObAN8DFhpnG06hhLJb9NcA6wMfIbyR3RfYB3g9OgadxAR+wOHAmdR4vS9KG8kywCnAn+jvNF225XSDXLKMG25BFgAHB4RL226PB4lIp4BnEuJ8vcEXg+cxMLnaElgNvBMYI/muGsDZ0XEij27e1+zn/8EOsnDm4ATgd8B21G6bGYC3UVlZ/zMrsM8FiLiiU2brupZdWVz/Yzh7ktJPv4lIj4QEStExFMpXS9XNY9tqOMtRykijulZ9QLg0k7R2xTJv4qIZ3Zt02nLUG1dMSKe0nOsxSJiyYjYkNJdc2Jm/r1nm4iIxaOMJ3kPsC6l0O8+5iOOl5kPUroye5+brZri/74oY7oe9UGhx07AXLq6QZtuwOOA90fEhhGxbERsC7yJ8rvU6/SImB9lvM/+0TPu5nF4MSWxuqV3RfN8LR8R/07pyu3upluKUgx2m0f5nXlmc/+1Kb+/d0bEKRHxYETcEhEHRYRdWouQcAyPFnE7Uz5NnpqZD0bEacBOEfHhrgTjPcCTgI0z86ZmWfcb3FbAq4DtM/PkruXfGWebvpqZX+le0OnSgIdTqd8ANwIvAc6OiOUpYyC+nJnv7brriV33OxLYJSI+npnZFGq7AN/tSS26j3t1RLwP+CxwDjA3Is6ifFo9vmvTj1GKq5d1fYrtTi92A9YE1svMa5v2nEcZj/F2ugoX4O+Z+fDg0Kadnwe+k5nv6Fr+APD1iDgwM2+jfKqeT3mzGU4nsbqzZ/kdzfUKw90xMy9u3oyPpzwfUIqDV4/QDbID5U3x2J7lqwLPY2EBOI8ylufUiJjRdGN22jJSW7vfoC+njNuBMg7nP4doz5tZWHzdC7w5M3/XtX6FIY7XOWb3c3MWcBRlzM2/Av8DnBMRz83MPw9x/86xT2mKnG5vBY6mpCxQ/h8/lJlHdW1zFwtfgw9SxiZ9HHgK8O5hjjcmEbE+pUj/3yHWvZDyuwbl/2ifzOwet3YN8JaIeEJmPtQs2xhYjDK2Csr/NZSxPcdSkrLnUj7AzAPe/3jaL/WDCU/LNKnD64GTmk+xUN4M1gJe2LXp5pSC6CaGtjlwe0+x83j8bIi2bh0R/xcRd1H+SN7YrFqvuX4RZVDqESPs93DKm9Nmze1XNrdHug+ZeRAljdkb+AklnTguurrsKM/BD0aI7DcFLuoUO81+b6SkQi/t2bb38a9HKZaOaz5tL94kTadTiokNmv1dn5mLZ+ZYCs3h+uKH7aOPMu7m+5Qicktge0ohcEr0nBHVZWfg8sy8tGf5NOCJwBsy86TM/Anltbg6ZWDuSG2KYZa/gVIA7wk8Gzg+Hv1R8ReUpHJrSgJ3bJTxMSMdr3PMh5dn5scy84jMPCczv0d5LSUlfXz0nSNeQEkle5MuKAOTX0Apil9BSacOiIjdu453cWZ+KDNPycxfZea+wBeAd0TEeJNUImIF4IeU8VSfGWKTSynP16soA74Pjq4zzIBvUYqur0XEqs1r5BuUwnt+s03nvePyzNwjM0/PzC9Rivx3Rc8ZcJq6IibvMmgWPO2zNeUT/ylNZL08cCblTJPuP2pPppx9M5zR1j9W/+i+ERGbUMYY3Uj51P4iFhZkndOAn9xcD9uOptg4k/LGQnP9u8y8fLQGZeZfM/MbmfkmyliXU4H3RUTnuKM9B6v1Pq7GP1j4Sbh7WbfOG9oplC7HzqUzzuSpo7W/Sye9WL5n+XBpSrdPAldn5u6ZObspcF9LKQYfdUZQ89xsydBv8rcD/8jMTlda5//nz5RxLrAwyelt65ApVWZenpn/l5mHUl6/21AKke5t7sjMCzLz1Mz8T0p68YmuTe4Y4nidYw773DRdZ+dSUquh7EQZRPyIYrbpBtsLeFtmHpmZZ2fmZ4AvA1+IMkh7OCdQkvfRutKGFOUU+h8DSwLbdX3oeVhm3ts8X7/KzPcA36UrCcrMqyhdqztTXv9/oHS7XsLC13Fn0PgZPbs/vTn208bTfqmfLHjap1PUHE/5Q38H5TTVJYE3dY0PuI3yhj2c0dZDKaJ6++t73+g7ej9hv57SdfHmzDw5yynEvYOIO4NUR2vHt4E3RMTqwL8xSrozZOMy76V8il0M6Aw6Hu05uIkyjqHXKix8Q3j4ED23O+tnUj5t916GPDtqhLb/hUePRxluvEzvNpf07O8O4HqGftN6I+UNubc7CxaOGeoVLOyS67RlqLbePtR4ky4XNdfrjLANlG6k7m2u6j1eM85kHUZ+bjoelQ41RcuOlLPBehPAzrEu6Vl+MaXIejKje8xnzjS/29+nnMK/dWYOVYwP5SLgqdE1WD4zD6e8jp8D/AtlqoGnA79tNvkTjx7nAwuTulEHWmtqmDYtJu0yaBY8LRLltNVtKZ++X9lzeS/lD1jn0/Fs4NURscowu5tNGUS67TDroaQzDw9Ibd4ENh9jc6cDD/WcFdXb7fEb4H4ePSi514mUP77HUl7TQ70ZPywiVoyhByqv21zf3FzPphSJSw2xLcB5wMbRdZZYU3S9mDK/zEjmUOZbWav5tN17uW2U+/f6OfD6ngGvb6YUQsOe8kwpbDbqXtCkOGtRkpleO1MStD8Nse6nwKrN+JHOvp5G6WL8PTyc+PyRUix0tukUD6MVeS9prq8bboOmu+tFPdv8HNgkIv61a9l2lA8Bp46wr1WaY144xOqXU7rqhkq6rm+ue5OhjSljjIY6o6rjDZTu3T+MsM1wvkEZS/O6zHzUhJAjeAlwY9d4HQAyc25mXtoUTv9B+d06rln3IGU8W+/v+xbAfZRxQNKU4qDldtmeMvnYV3pOF+7MtfE/lDesX1HGGLyVMijz05Q3xmcCT8zMz1H+mP0C+H5EfILyKXA14OWZ+fZmtycBe0fExZSBuv/F6KfAdpwG7BsRX6aMoXkxPZPAZeadEfFJ4NPNJ/JTKG9SrwU+npl/bbabGxFHU8bjHNNzOu1QNgcOjIgjKJPjLWiO/0Hgp12nL3+8WX92RHyRkvhsRJmz5HDKBG4foEx091HK+IYDKG9oh47UgCzzqvw38N1mrMzPKUXbOpRBwW/MzPuaN+k/UbpHRhrH83lKwfjdiPgWJSV6O7BXd1EZEfOAT2Rmp8vnm8CPmsHfx1DG4HygacvR3QeIiH8BXsYjJ7LrdhLldXJilLPr5lO6lv4IdM+2fADwvYj4M6XLaBdKsdk9JcJ3m/tdQnkDfR5lIOxvaLpRIqLTlhOBGyjJyS6UrtHXdR3vBMpr/8SI+AhlsP6XgO/nwjl4nkMZf3I8pWBZkzIXzQJKV1SvnSivh6FOwb+guRzevC6uo4zp2pfyu5nNMQ+hpJznU57vbShJype7C96mgFyfhV29z4+Ie4BbMvOsZpsPs/AMvwXNwOSOKzLzn81r6QhKCnQt5UzH1zePZa+u4y3XPF9nU4qvVzbP8x75yPmPPgH8uvk9OoaSBn0Q+OQIA96lgbHgaZedKeMxzutdkZkPRcRxwM4R8Y7MvCUiXkI5y+LLlELiapozi5oznl5PGeOxL2UQ498ofyw7Pk7p0vkU5Q/2wZQ04RFz8wwlM0+JiA9QZmjdg/JGti3lTa57uwMj4nbKWStvp3TRnU0ZO9HtR5SC53BGdx5lnMObKG+ii1HSjE8BD59JlplzIuKllDNpOrMVX0EzE29mPhDlawwOopzSH5TxRP+Ww0yM1/PYfhAR/2z29zZKgXAtJSnpdBdE074R09jMvCYiXtO05eeU7sH/zkfPsvyIfWXmj6N8tcD7KG/2cylv1ptl5t967vum5vo4hpCZ8yNiG8rr6bDmOL8C3t2dHmTmMU0a+QHgI5QzsbbNR06+9zvKaf77NW2+Dvgq8KVcOC/NXyhvyJ+hvD5voRRIL83MzllIndf+ayivz+MoXbHHNo+54zbKc30gpXC6m/J/uUNm3tD9OJt08A3ACTnEmYDN8/A6yuvpo03brqcUel/s2vRKyoeE91C6hq+hFBZf7dnlmyhnDHbs3VzOYuFg/a2a686kgd1e2TyWOym/w/tTzrK6k/J6fm1mdk/hMJ9S2O9BSWIvA3bMR57JRWb+rnmcB1KK1ZspM5d3D/zXFDcVBhNPFmdaVitExOcoXThr5xgmapMkwbP+55eTVgRc/umtBlpemfBoPA6npDE3s3AW3x0pn2CfSTld+4Jm+RKU7p3nU7oH3k35tDkhonzJ5fqUSP7jFjuqyYwZMx7+XZwzZ84GzbLPU7r0OpMr7jZnzpzRunlVqakwIeBkcdCyxuNIHj0l/2WUM6TO7lm+R3P9bMq8H19kYl93h1K62U7h0V0BUtsdyaN/F08DNpgzZ85zKF3EvV1cUpVMeDQeZ7Pwm9g7hjsleX0WzuB8M2XcwPMpYzQet8zcbCL2Iy2K5syZc/aMGTPW6ln2y66bv6VMJSANqaKAp38FT5TvIdqecupmUgbLndw9KZmq8HvK6+BYymR6GzfXE1LwSBrR23jkGXJStfpS8DRn3+xMeZPrvLGtARwTEcdm5meHud9MyqmVHHrooRu/+5DxTEWhybDmaity4lf35Pk7fuYRA95+8a1386EvnXT+RVeUE1sWW2waB+67Ay/fZL033nDT7Txh8cU47Ie/PuGnZ/Z+K4GmivsvPpi5Q34LmaaiU345m3e+Y0/mznvkZIXfOvQQXrnFlnzpKwdvO3ce3x1U+zR2S5V35EnNXGoaw9OvhGd34Fm9E1lFxEGUU1CHLHgycxYwq3Pz3YeMenazprj58xfw/i8+/F2fnHHke7nmhpEm05X0eJ38o5M4+6wzmXXYkVW9oUkj6VfBs4AyHfn1PctXwynHqzJ9qScQBPfNfZDNX/AM5s1fwFXX9n6DhKSJcu45Z3PEYd/isKO+x/Tp0wfdHE1xNRXE/Sp49gVmR8TVlMnBoMxc+nTGMCmdprajDtyVl228ListvwzXnPpJPvnNU7jjrns56AM7stIKy3DiV/fkD3P+ynZ7f52nrLAsP/nG3ixYkPztljvZff+jBt18qTU+sN97ueD833HnnXfwqs1fzl57v5PDvzWLBx96kD3/q3yf7rOf+1w+8rFPjLInqf36NvFg8/04m1IGLQfle5fOz8z5Y9xFTt/I2kiabI7hkQZjEGN4Njxg9qRNPHjJAVu0c+LBZgK43466oSRJUp85D48kSZWqaQyPMy1LkqTWM+GRJKlSFQU8JjySJKn9LHgkSVLr2aUlSVKlHLQsSZLUIiY8kiRVqqKAx4RHkiS1nwmPJEmVcgyPJElSi5jwSJJUqYoCHhMeSZLUfiY8kiRVyjE8kiRJLWLCI0lSpSoKeEx4JElS+5nwSJJUKcfwSJIktYgJjyRJlaoo4DHhkSRJ7WfBI0mSWs8uLUmSKuWgZUmSpBYx4ZEkqVIVBTwmPJIkqf1MeCRJqpRjeCRJklrEhEeSpEqZ8EiSJE2iiHhPRFweEZdFxDERsVRErB0R50XE1RHxg4hYYrz7t+CRJKlSEZN3GbkdsTrwLuD5mbkBsBiwE/C/wJcyc13gDmD38T5WCx5JkjQVLA5Mj4jFgaWBm4DNgROa9UcBO4x35xY8kiRVKiIm8zIzIi7ouszstCMz/wp8AbiBUujcBVwI3JmZ85rNbgRWH+9jddCyJEnqu8ycBcwaal1ErABsD6wN3AkcD2w91G7Ge3wLHkmSKjWFTtLaErguM28BiIgTgRcDy0fE4k3Kswbwt/EewC4tSZI0aDcAL4yIpaOcK78FcAVwBvDGZptdgB+P9wAmPJIkVWqqzMOTmedFxAnARcA84GJK99fPgGMj4lPNssPGewwLHkmSNHCZ+THgYz2LrwU2nYj926UlSZJaz4RHkqRKTZEerUlhwiNJklrPhEeSpEpNqyjiMeGRJEmtZ8IjSVKlKgp4THgkSVL7mfBIklSpqTLx4GQw4ZEkSa1nwiNJUqWm1RPwmPBIkqT2M+GRJKlSjuGRJElqERMeSZIqVVHAY8IjSZLaz4RHkqRKBfVEPCY8kiSp9Sx4JElS69mlJUlSpZx4UJIkqUVMeCRJqpQTD0qSJLWICY8kSZWqKOAx4ZEkSe1nwiNJUqWmVRTxmPBIkqTWM+GRJKlSFQU8JjySJKn9THgkSaqU8/BIkiS1iAmPJEmVqijgMeGRJEntZ8IjSVKlnIdHkiSpRSx4JElS69mlJUlSperp0DLhkSRJFTDhkSSpUk48KEmS1CImPJIkVWpaPQHP8AVPRKw40h0z8/aJb44kSdLEGynhuRBIhh7EncA6fWmRJEmaFDWN4Rm24MnMtSezIZIkSf0y6qDlKP4jIj7S3F4zIjbtf9MkSVI/RUzeZdDGcpbWN4AXAW9pbt8NfL1vLZIkSZpgYzlL6wWZ+byIuBggM++IiCX63C5JktRnNY3hGUvC81BELEYZqExEPAVY0NdWSZIkTaCxJDxfBU4CVomITwNvBPbva6skSVLfOQ9Pl8w8OiIuBLZoFu2QmVf2t1mSJEkTZ6wzLS8NdLq1pvevOZIkabI4hqdLRHwUOApYEVgJOCIi7NKSJEmLjLEkPDsDG2XmXICI+CxwEfCpfjZMkiRpooyl4PkzsBQwt7m9JPCnfjVIkiRNjno6tEb+8tCvUcbsPABcHhGnNbdfBfx6cponSZL0+I2U8FzQXF9IOS2948y+tUaSJE2aaRUNWh7py0OPmsyGSJIk9cuoY3giYl3gQGB9ylgeADJznT62S5Ik9VlFAc+YvlriCOAQYB7wSuA7wHf72ShJkqSJNJaCZ3pmzgYiM6/PzAOAzfvbLEmS1G8RMWmXQRvLaelzI2IacHVE7AP8FVi5v82SJEmaOGMpePalfLXEu4BPUtKdXfrZKEmS1H9TIHiZNGP58tDzmx/vAXbrb3MkSZIm3kgTD/6EMtHgkDJzu760SJIkTQrn4Sm+MGmtkCRJ6qORJh48azIbIkmSJldFAc+YTkuXJElapI3lLC1JktRCU2F+nMkypQue+y8+eNBNkKq01JT+yyBJj51naUmSVKmaxrVM6bO0brt33qCbIFXnyU9cnLX3/dmgmyFV57ovv3bQTWg1z9KSJEmtN2pPfUSsCxwIrA8s1Vmemev0sV2SJKnPahq0PJbuuyOAQ4B5wCuB7wDf7WejJEmSJtJYCp7pmTkbiMy8PjMPoHyBqCRJWoRNi8m7DNpYTj6dGxHTgKsjYh/gr8DK/W2WJEnSxBlLwbMvsDTwLuCTlHRnl342SpIk9d9USF4my6gFT2ae3/x4D7Bbf5sjSZI08cZyltYZDDEBYWY6jkeSpEVYTWdpjaVLa7+un5cC3kA5Y0uSJGmRMJYurQt7Fp0bEU5KKEnSIs4xPF0iYsWum9OAjYFV+9YiSZKkCTaWLq0LKWN4gtKVdR2wez8bJUmS+q+iITxjKniemZlzuxdExJJ9ao8kSdKEG0vB83/A83qW/WaIZZIkaREyraKIZ9iCJyJWBVYHpkfERpQuLYDlKBMRSpIkLRJGSnheDewKrAF8kYUFzz+BD08sN40AABEJSURBVPe3WZIkqd/G8oWabTFswZOZRwFHRcQbMvOHk9gmSZKkCTWW4m7jiFi+cyMiVoiIT/WxTZIkSRNqLAXP1pl5Z+dGZt4BbNO/JkmSpMkQMXmXQRtLwbNY92noETEd8LR0SZK0yBjLaenfA2ZHxBGUCQjfBnynr62SJEl952npXTLzcxHxB2BLyplan8zMX/S9ZZIkSRNkLAkPmXkqcCpARLwkIr6emXv3tWWSJKmvKgp4xnYKfkRsGBH/GxF/Bj4FXNXXVkmSpKpExPIRcUJEXBURV0bEiyJixYg4LSKubq5XGO/+hy14ImK9iPhoRFwJHAzcCERmvjIzvzbeA0qSpKlhWkzeZQy+Apyamc8AngtcCXwQmJ2Z6wKzm9vje6wjrLsK2AJ4XWa+tCly5o/3QJIkSUOJiOWAlwOHAWTmg82UONsDRzWbHQXsMN5jjFTwvAH4O3BGRHwrIrZg4ddLSJKkRdy0iEm7RMTMiLig6zKzqynrALcAR0TExRHx7Yh4IrBKZt4E0FyvPO7HOtyKzDwpM98MPAM4E3gPsEpEHBIRW433gJIkqT6ZOSszn991mdW1enHgecAhmbkRcC+Po/tqKKMOWs7MezPz6MzclvJFopdMdCMkSdLkm0IzLd8I3JiZ5zW3T6AUQP+IiNVKW2M14ObxPtbH9EWpmXl7Zh6amZuP94CSJEndMvPvwF8iYkazaAvgCuBkYJdm2S7Aj8d7jDHNwyNJktpnjGdPTZZ3AkdHxBLAtcBulGDmuIjYHbgB2HG8O7fgkSRJA5eZlwDPH2LVFhOxfwseSZIqFRWdfP2YxvBIkiQtiix4JElS69mlJUlSpabYoOW+MuGRJEmtZ8IjSVKlTHgkSZJaxIRHkqRKxRi+86EtTHgkSVLrmfBIklQpx/BIkiS1iAmPJEmVqmgIjwmPJElqPxMeSZIqNa2iiMeER5IktZ4JjyRJlfIsLUmSpBYx4ZEkqVIVDeEx4ZEkSe1nwSNJklrPLi1Jkio1jXr6tEx4JElS65nwSJJUKQctS5IktYgJjyRJlXLiQUmSpBYx4ZEkqVJ+eagkSVKLmPBIklSpigIeEx5JktR+JjySJFXKMTySJEktYsIjSVKlKgp4THgkSVL7mfBIklSpmlKPmh6rJEmqlAWPJElqPbu0JEmqVFQ0atmER5IktZ4JjyRJlaon3zHhkSRJFTDhkSSpUn61hCRJUouY8EiSVKl68h0THkmSVAETHkmSKlXREB4THkmS1H4mPJIkVcqZliVJklrEhEeSpErVlHrU9FglSVKlTHgkSaqUY3gkSZJaxIJHkiS1nl1akiRVqp4OLRMeSZJUARMeSZIq5aBlSZKkFjHhkSSpUjWlHjU9VkmSVCkTHkmSKuUYHkmSpBYx4ZEkqVL15DsmPJIkqQImPJIkVaqiITwmPJIkqf1MeCRJqtS0ikbxmPBIkqTWM+GRJKlSjuGRJElqEQseSZLUenZpSZJUqXDQsiRJUnuY8EiSVKmaBi1b8Ohx+/QB+3PuOWexwoorcvTxPwbgj3Ou5POf/gQPPvgAiy22OPt9aH/W3+A5A26p1C67vnwtdnrRmgRw7G9v4Iiz/sx7t16PVz17FRZkctvdD7Lf93/Pzf98YNBNlQbOLi09btu8bge+dPChj1j29a8cxNve/g6OOvZE/muvffj6Vw4aUOukdlpv1WXY6UVrssNBv2abz5/D5uuvwlorLc2s069l68+dw2s//2tOv+Jm3vXqdQfdVE1h04hJuwyaBY8et402fj7LPelJj1gWwL333APAPffczUpPecoAWia119NXWYZL/nwHcx9awPwFye/+dBuvfs6q3PPAvIe3mb7EYuQA2yhNJZPepRURu2XmEZN9XE2ufff7IO/ZZyYHf/kLLFiwgEOPOHrQTZJaZc7f72G/185g+aWfwNyH5rPZ+itz6Q13AbDfNjN4/Sarc/fcebzl4N8OuKWaymoawxOZk1v/R8QNmbnmMOtmAjObm7Myc9bktUyPx4wZM9YCfjpnzpwNANZcc83Z06dP/8acOXN+OGPGjDcBM+fMmbPlQBsptc/uwN7APcAVwP0RcWXX384PAUsBHxtQ+zTF/eKKWyatCHj1+k8ZaHnVl4QnIv4w3CpgleHu1/ySWuS0wJJLLvkKoFPgHA98e4DNkdrqsOYC8BngRsqHxs7f0e8DP8OCR8OoKeHpV5fWKsCrgTt6lgfwf306pqaQefPmPbTEEku8AjgT2By4erAtklppZeBmYE3g34AXbbDBBnt1rd8OuGoQDZOmmn4VPD8FlsnMS3pXRMSZfTqmBmTGjBnHAJsBK82YMeNG4GM333zz9WusscYXZ8yYsTgwl4VdlZImzg+BJwMPUbq27vjCF76wBnAZsAC4HthzcM3TVFfTTMuTPoZHdYiImY7Bkiafv3t6LE678tZJKwJe9cyV2jeGR/IPrjQY/u7psZhWT8DjPDySJKn9THgkSapUTWN4THg0oSLiNRExJyKuiYgPDro9Ui0i4vCIuDkiLht0W6SpyIJHEyYiFgO+DmwNrA/sHBHrD7ZVUjWOBF4z6EZIU5UFjybSpsA1mXltZj4IHAtsP+A2SVXIzLOB2wfdDi1aIibvMmgWPJpIqwN/6bp9Y7NMkqSBctCyJtJQNbwTPUnSFOWgZWl8bgSe2nV7DeBvA2qLJEkPM+HRRDofWDci1gb+CuwEvGWwTZIkDceJB6VxyMx5wD7AL4ArgeMy8/LBtkqqQ0QcA/wGmBERN0bE7oNuk/RYRcRiEXFxRPy0ub12RJwXEVdHxA8iYolx79vv0pIkqU7n/PGOSSsCXrbeCqPmSRHxXuD5wHKZuW1EHAecmJnHRsQ3gd9n5iHjOb4JjyRJGriIWAN4LfDt5nYAmwMnNJscBeww3v07hkeSpEpN5vw4ETETmNm1aFbPl91+GXg/sGxz+8nAnc1wCXicU51Y8EiSpL5riptZQ62LiG2BmzPzwojYrLN4qN2M9/gWPJIkVWoKnaT1EmC7iNgGWApYjpL4LB8Rizcpz+Oa6sQxPJIkaaAy80OZuUZmrkWZ0uT0zPx34Azgjc1muwA/Hu8xLHikAYuI+RFxSURcFhHHR8TSj2Nfm3WdzrndSN9YHxHLR8Q7xnGMAyJiv7Eu79nmyIh440jb9Gy/lt/+LfXPtIhJu4zTB4D3RsQ1lDE9h437sY73jpImzP2ZuWFmbgA8COzZvTKKx/y7mpknZ+ZnR9hkeeAxFzyS1E+ZeWZmbtv8fG1mbpqZT8/MHTPzgfHu14JHmlrOAZ7eJBtXRsQ3gIuAp0bEVhHxm4i4qEmClgGIiNdExFUR8Wvg3zo7iohdI+Lg5udVIuKkiPh9c3kx8FngaU269Plmu/dFxPkR8YeI+HjXvv4nIuZExK+AGaM9iIjYo9nP7yPihz2p1ZYRcU5E/LEZqNiZbOzzXcd+++N9IiWNLibxMmgWPNIUERGLA1sDlzaLZgDfycyNgHuB/YEtM/N5wAWUmHcp4FvA64CXAasOs/uvAmdl5nOB5wGXAx8E/tSkS++LiK2AdYFNgQ2BjSPi5RGxMaVPfSNKQbXJGB7OiZm5SXO8K4HuWX/XAl5BmW/jm81j2B24KzM3afa/R/MVJZI0ITxLSxq86RFxSfPzOZQ+6n8Brs/M3zbLXwisD5xb5uJiCcrXCDwDuC4zrwaIiO/xyHkuOjYH3gqQmfOBuyJihZ5ttmouFze3l6EUQMsCJ2Xmfc0xTh7DY9ogIj5F6TZbhvJ1Ix3HZeYC4OqIuLZ5DFsBz+ka3/Ok5th/HMOxJGlUFjzS4N2fmRt2L2iKmnu7FwGnZebOPdttyOOYl6JHAAdm5qE9x9h3HMc4EtghM38fEbsCm3Wt691XNsd+Z2Z2F0ZExFqP8biSHoup0Nc0SezSkhYNvwVeEhFPB4iIpSNiPeAqYO2IeFqz3c7D3H82sFdz38UiYjngbhbOaAolhXlb19ig1SNiZeBs4PURMT0ilqV0n41mWeCmiHgC8O8963aMiGlNm9cB5jTH3qvZnohYLyKeOIbjSNKYmPBIi4DMvKVJSo6JiCWbxftn5h+b6dp/FhG3Ar8GNhhiF+8GZkX5Bu35wF6Z+ZuIOLc57fvnzTieZwK/aRKme4D/yMyLIuIHwCXA9ZRut9F8BDiv2f5SHllYzQHOAlYB9szMuRHxbcrYnoua78+5hcfxnTmSxiYqinj8tnRJkip13p/umrQi4AVPe9JAqysTHkmSKjWZXx46aI7hkSRJrWfCI0lSpSoKeEx4JElS+5nwSJJUq4oiHhMeSZLUeiY8kiRVqqZ5eEx4JElS65nwSJJUKefhkSRJahETHkmSKlVRwGPCI0mS2s+CR5IktZ5dWpIk1aqiPi0THkmS1HomPJIkVcqJByVJklrEhEeSpEo58aAkSVKLmPBIklSpigIeEx5JktR+JjySJNWqoojHhEeSJLWeCY8kSZVyHh5JkqQWMeGRJKlSzsMjSZLUIiY8kiRVqqKAx4RHkiS1nwmPJEm1qijiMeGRJEmtZ8EjSZJazy4tSZIq5cSDkiRJLWLCI0lSpZx4UJIkqUVMeCRJqlRFAY8JjyRJaj8THkmSalVRxGPCI0mSWs+ER5KkSjkPjyRJUouY8EiSVCnn4ZEkSWoREx5JkipVUcBjwiNJktrPhEeSpFpVFPGY8EiSpNaz4JEkSa1nl5YkSZVy4kFJkqQWMeGRJKlSTjwoSZLUIiY8kiRVqqKAx4RHkiS1nwmPJEm1qijiMeGRJEmtZ8IjSVKlnIdHkiSpRUx4JEmqlPPwSJIktYgJjyRJlaoo4DHhkSRJ7WfCI0lSrSqKeEx4JElS61nwSJKk1rNLS5KkSjnxoCRJUouY8EiSVCknHpQkSWoREx5JkipVUcBjwiNJktrPhEeSpEo5hkeSJKlFTHgkSapWPRGPCY8kSWo9Ex5JkirlGB5JkqQWMeGRJKlSFQU8JjySJKn9THgkSaqUY3gkSZJaxIJHkiS1nl1akiRVKioatmzCI0mSWs+CR5KkWsUkXkZqRsRTI+KMiLgyIi6PiHc3y1eMiNMi4urmeoXxPlQLHkmSNGjzgP/OzGcCLwT2joj1gQ8CszNzXWB2c3tcLHgkSarUFAl4yMybMvOi5ue7gSuB1YHtgaOazY4CdhjvY7XgkSRJfRcRMyPigq7LzGG2WwvYCDgPWCUzb4JSFAErj/f4nqUlSVKlJnPiwcycBcwaaZuIWAb4IbBvZv4zJrCBJjySJGngIuIJlGLn6Mw8sVn8j4hYrVm/GnDzePdvwSNJUqViEv+N2I4S5RwGXJmZB3WtOhnYpfl5F+DH436smTne+0qSpEXYLXfPm7Qi4CnLLj5s1RMRLwXOAS4FFjSLP0wZx3McsCZwA7BjZt4+nuNb8EiSVKlb7pnEgmeZ4QueyWCXliRJaj3P0pIkqVL1fJOWCY8kSaqACY8kSZWazHl4Bs2ER5IktZ4FjyRJaj27tCRJqtRoEwK2iQmPJElqPRMeSZIq5aBlSZKkFrHgkSRJrWfBI0mSWs8xPJIkVcoxPJIkSS1iwiNJUqWch0eSJKlFTHgkSaqUY3gkSZJaxIRHkqRKVRTwmPBIkqT2M+GRJKlWFUU8JjySJKn1LHgkSVLr2aUlSVKlnHhQkiSpRUx4JEmqlBMPSpIktYgJjyRJlaoo4DHhkSRJ7WfCI0lSrSqKeEx4JElS65nwSJJUKefhkSRJahETHkmSKuU8PJIkSS0SmTnoNkiSJPWVCY8kSWo9Cx5JktR6FjySJKn1LHgkSVLrWfBIkqTWs+CRJEmt9/+zB63uAFVBfwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure(figsize=(10, 10))\n",
    "sns.heatmap(cm, annot=True, fmt=\".0f\", linewidths=.5,\n",
    "            square=True, cmap='Blues')\n",
    "plt.ylabel('Actual label')\n",
    "plt.xlabel('Predicted label')\n",
    "all_sample_title = 'Accuracy Score: {0}'.format(score)\n",
    "plt.title(all_sample_title, size=15)\n",
    "plt.savefig('toy_Digits_ConfusionSeabornCodementor.png')\n",
    "# plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ROC Curves for Model Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting ROC curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg_vanilla = pipe\n",
    "\n",
    "steps_l2 = steps.copy()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps_l2[-1] = ('lr', LogisticRegression(max_iter=10000, penalty='l2', C=.01))\n",
    "log_reg_l2 = Pipeline(steps_l2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg_vanilla.fit(X, y_train)\n",
    "\n",
    "y_probs_vanilla = log_reg_vanilla.predict_proba(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg_l2.fit(X, y_train)\n",
    "y_probs_l2 = log_reg_l2.predict_proba(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = y_train.map({'Yes': 1, 'No': 0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpr_v, tpr_v, thresholds_v = roc_curve(y_train, y_probs_vanilla[:,1])\n",
    "fpr_l2, tpr_l2, thresholds_l2 = roc_curve(y_train, y_probs_l2[:,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xUZfb48c9JQhWQpYkSFHRRsbAoAfUH+8VOFxUQEJRioxpRYVFBETsuIkERECkKUhRBFkEWFcHVGImIKAjSAoQiTYpCgDDn98cMcYwpk2Tu3MnMeb9eeZm5c+fOyZXMyXmee88jqooxxhiTmxi3AzDGGBPeLFEYY4zJkyUKY4wxebJEYYwxJk+WKIwxxuTJEoUxxpg8OZYoRGSSiOwRkR9zeV5EJElENorIahG50qlYjDHGFJ6TFcUUoHkez7cA6vi+7gfecDAWY4wxheRYolDV5cCBPHZpC7ytXl8DFUXkbKfiMcYYUzhxLr53DWC73+N037Zd2XcUkfvxVh2cccYZDS6++OKQBGiMKb5+2HEox+2Xy+YQR+KurYeUgxlKpod9qlq1MMdwM1FIDtty7CeiqhOACQAJCQmamprqZFzGFG/TO8CG/7odRRgr5/3PsJwTSSTweDwAxMTE0LlzZ3bv3s3nn3++tbDHc/Oqp3Sgpt/jeGCnS7EYEzksSeSvzs1uR+CY1NRUzjnnHPr16wfAjBkzWLp0aZGO6WZFMR/oJyIzgauAQ6r6l2EnY4Khx+RvWLp+ryPHnlRiBNfHrnLk2EVRK+PdPJ9Pe7FViCIxoeDxeOjevTvvvPMOAIcOBa9icixRiMgM4FqgioikA08BJQBUdRywEGgJbASOAj2cisUYp5IEEJZJ4rNT9fN8/rqLCjVUbcLUZ599Rrt27Th48CAVK1Zkzpw5XH/99UE7vmOJQlU75/O8An2den8TvfKqHvL9K7oo4/thNOZ9PZDmdhAmZJYsWcLBgwe56667mDJlCjExwZ1VsDuzTcTJLUkE9Fd0YZNEBI95m/A0d+5c7r//fgBeeOEFfv75Z95+++2gJwlwd47CmFwFY06hQGPw2SuJMKoOjPH322+/0bp1a5YtW0ZsbCwjRoygYsWK1KlTx7H3tIrChKWiJokCj8H7JwmrDkyYeuutt6hSpQrLli2jdu3arFmzhooVKzr+vlZRmCJz8oqiQlcFW4FhhXhDqyRMmEpNTeXee+8lNjaWp556imHDhoXsva2iMEXmVJIoUlVQGFZJmDC0ePFiABISEhg0aBBpaWkhTRJgFYUJUCBVQ9hcl29VgYkAW7dupXnz5qxbt46kpCT69+/PSy+95EosVlGYgOSXJOy6fGOC54knnuD8889n3bp1NGrUiC5durgaj1UUJlc5VREhrxqsb5GJMhdddBE///wzpUuXZvz48dx9991uh2SJwuQue5JwpWooaJKweQZTDPk38WvQoAHx8fF8+OGHlCtXzuXIvCxRRBCnrj5ypIooaKVg8w4mQqWkpNCmTRtuv/12xo0bx7vv5t2jyw02RxFBnEgSjlURBUkSViWYCJSZmUnnzp25+uqr2bt3L8eOHXM7pFxZRRGBglYBnP6rf1hwDpcjqxRMFFqyZAkdOnTg0KFDVKpUiblz5/J///d/boeVK6soTO6cnkS2SsFEqaVLl3L48GF69OjB3r17wzpJgFUUYc/Ju55zZX2PjAm62bNns2TJEt58802ef/557rvvPmrXru12WAGxiiLMFTRJBGVOwfoeGRM0hw8f5p///CcdO3Zk8uTJHDx4EKDYJAmwiiIshPyu50CvOLJKwpgimTBhAv379+fEiROcf/75fPzxxyFp4hdsVlGEgZDf9RxIkrBKwpgiSU1N5YEHHuDUqVM8/fTTbNq0ydFW4E6yiiLEirT6WqCsYjDGNQsWLKB169YkJCQwePBg+vfvzznnnON2WEViFUWIFWn1tUBZxWBMyG3ZsoULL7yQNm3a8NprrwHeleeKe5IAqyiCojBXJoWkerCKwZiQ+Ne//sW///1vPB4P11xzDV27dnU7pKCyRBEErlyZdFpuScIqBmNC4sILL2TDhg2UKVOGN9980/VOr06wRBFEIe2savc6GOMa/yZ+jRo1onbt2sydO5eyZcu6HJkzbI6iuLJ7HYxxxZdffkm1atXo1asXANOmTWPx4sURmyTAKoriJaf5CKskjAmJzMxM7rzzTt577z0ATpw44XJEoWOJojjJniSskjAmJBYtWkSnTp04fPgwlStXZt68eTRp0sTtsELGEkURhKQPk1URxrjuf//7H0eOHOHee+9l/PjxxMRE16h9dP20QeafJEK2boNVEcaExIwZM+jZsycAzz33HFu2bOHNN9+MuiQBVlEELCR3VPuzq5qMccXBgwdp2bIlycnJxMbG8sorr1CxYkXOO+88t0NzTfSlxkIKyR3V/uyqJmNC7o033uCss84iOTmZOnXqsGHDhmLZxC/YrKLIQdCrh4KuD+3PKgljQiI1NZU+ffoQFxfH888/z2OPPeZ2SGHDKoocBL16KGySsErCGMfNnz8fgISEBIYMGcL27dstSWQT9RVFUKoH69ZqTLGzadMmmjVrxqZNm0hKSqJ///4888wzbocVlqK+oghK9WDdWo0pVgYOHMiFF17Ipk2baNy4MXfddZfbIYW1qK8oTgvKlUtWMRgT9v7+97+zadMmypQpw+TJk+nYsaPbIYU9SxTGmIjn38Tv6quvpk6dOsydO5fSpUu7HFnx4OjQk4g0F5H1IrJRRAbn8Py5IrJURL4TkdUi0tLJeIwx0Wf58uVUrVqVBx54APA28Vu0aJEliQJwLFGISCzwOtACuAToLCKXZNttCDBbVa8AOgFjnYrHEdM7wLAz3Y7CGJODEydO0K5dO5o2bcqBAwdQVbdDKracHHpqBGxU1c0AIjITaAus9dtHgQq+788EdjoYT/DZTXHGhKUFCxbQuXNnfvvtN6pWrcqHH37INddc43ZYxZaTiaIGsN3vcTpwVbZ9hgH/FZH+wBnAjTkdSETuB+4HOPfcc4MeaJHZJLYxYSUlJYXff/+dXr168frrr0dlf6ZgcvLsSQ7bstd+nYEpqhoPtATeEZG/xKSqE1Q1QVUTqlZ1qGWGMaZYmzZtGj169ADgmWeeYdu2bbzxxhuWJILAyYoiHajp9zievw4t3QM0B1DVZBEpDVQB9jgYV9EUpR2HMSboDhw4QMuWLUlJSSEuLo5Ro0ZRsWJF4uPj3Q4tYjiZalcAdUSktoiUxDtZPT/bPtuAGwBEpC5QGnB4gYcisrbfxoSNMWPGUL16dVJSUrjwwgvZuHGjNfFzgGMVhapmikg/YDEQC0xS1TUiMhxIVdX5wCPAmyIyAO+wVHcNp0sT8qoebF7CGFelpqby4IMPEhcXx4gRIxg4cKDbIUUsR2+4U9WFwMJs2570+34t0NjJGIoktyRhVYQxrpkzZw7t2rUjISGBoUOH0q9fP6pVq+Z2WBHN7swOhFUPxrhu/fr1tGjRgi1btmQ18Rs+fLjbYUUFuxzAGBPWPB4PAwYMoG7dumzZsoWmTZvSrVs3t8OKKlZR5MSubDImbNSpU4fNmzdTtmxZ3n77bdq1a+d2SFHHEkVO7I5rY1zl38Tvn//8J3Xr1uX999+3/kwusURxWk5VhM1NGBNyn3/+Obfffju33norkyZNYsqUKW6HFPVsjuI0uz/CGFedOHGC2267jeuuu45ff/2V2NhYt0MyPlZRZGdVhDEhN3/+fO68805+//13qlWrxoIFC2jYsKHbYRkfqyiMMa5buXIlR48epV+/fuzatcuSRJixRGGMccXUqVO5++67ARg2bBjp6emMGTPGmviFIfs/YowJqX379pGQkED37t2ZMWMGBw8eBOCcc85xOTKTm6iZo+gx+RuWrg/vfoPGRLpXX32VgQMHkpmZSd26dfn444+tiV8xEDUVRV5J4rqLbI0LY5y2YsUKBgwYAMDIkSNZu3ZteC5EZv4iaiqK09JebPXnDXYXtjGOmj17NnfccQcNGzZk2LBh9O3blypVqrgdlimAqEsUf2F3YRvjiJ9++okWLVqwdetWdu3aRWJiIk899ZTbYZlCiPhEcXpuYlKJEVwfu8q7SndO7P4JY4LC4/GQmJjI66+/jqpy3XXXcc8997gdlimCgBKFb4W6c1V1o8PxBN3puYnrY1flvpNVEsYEzQUXXEBaWhpnnHEG06dPp23btm6HZIoo30QhIq2AV4CSQG0RqQ88paq3OR1cYeV5hZNVDsYEXWZmJjExMcTExNC0aVP+8Y9/MHv2bEqWLOl2aCYIArnqaThwFXAQQFVXAX93Mqiiyp4k7KomY5zz6aefUrVq1azhpSlTpjBv3jxLEhEkkKGnk6p6UET8t4XPutZ5+NMVTsNcC8OYiJSRkUH79u356KOPAKwFeAQLJFH8JCJ3ADEiUhtIBL52NixjTDibO3cuXbt25ejRo1SvXp2PPvqIK6+80u2wjEMCGXrqBzQAPMAHQAbeZBF2ekz+hlqDP3I7DGMi3urVqzl27BgPPvggO3bssCQR4QJJFM1U9V+qeoXvazDQwunACsN/bsLmJYwJrrfeeouuXbsC8NRTT7Fz505Gjx5tTfyiQCBDT0PwVhL+nshhW9j4y93XxphC27NnDy1atGDlypXExcUxduxYKlSoQPXq1d0OzYRIrolCRJoBzYEaIvKK31MV8A5DGWMi3MiRIxk8eDCZmZlceumlfPzxx1SoUMHtsEyI5VVR7AF+xDsnscZv+xFgsJNBFZn1bzKmyFasWMGjjz5KiRIlSEpKon///m6HZFySa6JQ1e+A70RkuqpmhDCmosstSdgd2Mbka8aMGXTu3JmGDRvyzDPP0KdPHypVquR2WMZFgcxR1BCR54BLgKwLpVX1QseiCha7C9uYgP3444+0bNmS7du3s2fPHhITExkyZIjbYZkwEMjlClOAyYDgvdppNjDTwZiMMSHk8Xjo3bs39erVY/v27dx4443cd999bodlwkggFUVZVV0sIv9W1U3AEBH5wunAjDGhUbt2bbZt20a5cuWYMWMGrVu3djskE2YCSRTHxdu/Y5OI9AJ2ANWcDcsY4yT/Jn433HADhw4dYsaMGdafyeQokKGnAUA54EGgMXAf0NPJoIwxzlm8eDFVqlShZ0/vr/GkSZOYM2eOJQmTq3wrClVN8X17BLgLQETinQzKGBN8GRkZ3H777SxatAgR4YwzznA7JFNM5JkoRKQhUAP4n6ruE5FLgX8B1wOWLIwpJt5//326devG0aNHOfvss1m4cCH169d3OyxTTOQ69CQiLwDTgS7AxyLyBLAU+B4I/0tjjTFZ1qxZw7Fjx3j44YdJT0+3JGEKRFRzXlpCRNYCDVT1mIhUAnYC/1DV9QEfXKQ5MBqIBSaq6os57HMH3tUiFPheVe/M65gJCQmampqa43OfDW365yVP7T4KE8XGjx/PsmXLePfddwFvz6Zq1ew6lGglIt+qakJhXpvX0FOGqh4DUNUDIrKugEkiFngduAlIB1aIyHxVXeu3Tx3gMaCxqv4qIkX6V/ynJGF3YZsotXv3bpo3b873339PiRIlGDduHBUqVLAkYQotr0Rxvoic7hArQC2/x6jq7fkcuxGwUVU3A4jITKAtsNZvn/uA11X1V98x9xQw/pxZJWGi1AsvvMDQoUM5deoUl19+uTXxM0GRV6Jol+3xawU8dg1gu9/jdLxrb/u7EEBEvsQ7PDVMVT/OfiARuR+4H+Dcc88tYBjGRIeUlBQef/xxSpYsSVJSEn369HE7JBMh8moK+GkRjy05bMs+IRIH1AGuxXsV1RcicpmqHswWywRgAnjnKIoYlzERZdq0aXTt2pWrrrqKF154gV69elGxYkW3wzIRJJA7swsrHajp9zge74R49n2+VtWTwBYRWY83cawI9E16TP4ma2W7NFvb3USR1atX07JlS3bs2MH+/ftJTExk8ODwXgHAFE9OrmG4AqgjIrVFpCTQCZifbZ95wHUAIlIF71DU5oK8if/yp8ZEA4/Hw3333Uf9+vXZsWMHN998szXxM44KuKIQkVKqejzQ/VU1U0T6AYvxzj9MUtU1IjIcSFXV+b7nbvZdinsKGKiq+wv2I3ilvdjKe5GtMRGuVq1abN++nfLlyzNz5kxatmzpdkgmwuWbKESkEfAWcCZwroj8A7hXVfNd7kpVFwILs2170u97BR72fRljcuHfxK9Zs2YcPHiQGTNmEBfn5OixMV6BDD0lAa2B/QCq+j2+4SJjjPMWLlxIpUqV6NGjBwBvvvkm7733niUJEzKB/EuLUdWt3k7jWU45FI8xxufo0aPceuutLFmyBBHhzDPPdDskE6UCSRTbfcNP6rvbuj/ws7NhGRPdZs2aRY8ePTh27Bg1atRg4cKF1KtXz+2wTJQKZOipN945hHOBX4CrfduMMQ7ZsGEDx48fZ+DAgaSnp1uSMK4KpKLIVNVOjkdiTJQbO3Ysy5cvZ+bMmQwZMoRevXpRpUoVt8MyJqBEscJ3I9ws4ANVPeJwTMZElZ07d9K8eXN++OEHSpQoweHDh6lQoYIlCRM28h16UtULgGeBBsAPIjJPRKzCMCYInnvuOc4991x++OEH6tevz7Zt26yJnwk7Ad2ZrapfqeqDwJXAYbwLGhljiiAlJYUhQ4YQGxvLuHHj+O6776hevbrbYRnzF/kmChEpJyJdROQ/wDfAXuD/OR6ZMRHI4/EwefJkAK666ipeeuklfvnlFx544AGXIzMmd4HMUfwI/AcYoapfOByPMRFr1apVtGzZkl27dnH48GESExMZNGiQ22EZk69AEsX5qupxPBJjIpTH4+Hee+9lypQpqCotWrSwCsIUK7kmChEZqaqPAHNE5C9rQASwwp0xBu9iWzt27KBChQrMnj2bZs2auR2SMQWSV0Uxy/ffgq5sZ0zUO3HiBDExMcTFxdGyZUuOHDnCO++8Y/2ZTLGU62S2qn7j+7auqn7q/wXUDU14xhQ/CxYsoHLlyvTs2ROACRMmWKdXU6wFcnlszxy23RPsQIwp7o4ePcqNN95ImzZt+P3336lcubLbIRkTFHnNUXTEuypdbRH5wO+p8sDBnF9lTHSaMWMGPXv2JCMjg5o1a7Jo0SIuvfRSt8MyJijyqoW/wbsGRTzwut/2I8B3TgZVEJNKjOD62FW2up1x1aZNmzhx4gSPPfYYzz//vNvhGBNU4l1krvhISEjQ1NTUPzYMy9ajv87N0OW90AZlotKYMWP44osvmD17NgAHDhygUqVKLkdlTM5E5FtVTSjMa/Maelqmqk1F5FfAP5sI3lVMw+s3YtghtyMwUSI9PZ3mzZuzZs2aPzXxsyRhIlVek9mnlzutAlT1+zr92Jio8/TTT1OrVi3WrFlDgwYNSE9PtyZ+JuLldXns6buxawKxqnoKuAZ4ADgjBLEZE1ZSUlIYNmwYcXFxTJw4kdTUVKpVq+Z2WMY4LpDLY+fhXQb1AuBtvPdQvOtoVMaECY/Hw8SJEwFvE7+RI0eyb98+7rnHrhA30SOQO4A8qnpSRG4HXlXVJBEJm6uejHHKypUradWqFbt37+b3338nMTGRhx9+2O2wjAm5QCqKTBHpANwFLPBtK+FcSMa4y+Px0K1bNxo0aMDu3btp3bq1NfEzUS2QiqIn0Advm/HNIlIbmOFsWMa4p2bNmuzcuZMzzzyTOXPmcMMNN7gdkjGuyjdRqOqPIvIg8HcRuRjYqKrPOR+aMaHj38SvTZs2HDlyhKlTp1p/JmMIbIW7fwIbgbeAScDPItLY6cCMCZW5c+dSqVKlrCZ+48aNY/r06ZYkjPEJ5DdhFNBSVdcCiEhd4B2gUHf4GRMufvvtN9q0acPnn3+OiFC1qt0eZExOAkkUJU8nCQBV/UlESjoYkzGOmzZtGvfeey/Hjx/nvPPOY9GiRdSta93zjclJIFc9rRSR8SLSxPf1BmHUFNCYwti+fTuZmZkMHTqUtLQ0SxLG5CGQRNEL2AQMAv4FbMZ7d7Yxxcorr7xCu3btAHjsscfYt28fw4cPdzkqY8JfnkNPInI5cAEwV1VHhCYkY4Jr27ZtNGvWjHXr1lGyZMmsJn4VK1Z0OzRjioVcKwoReRxv+44uwBIRyWmlO2PC2tChQ6lduzbr1q0jISGBHTt2WBM/Ywoor6GnLkA9Ve0ANAR6hyYkY4IjJSWFZ599lpIlSzJlyhRWrFhBlSpV3A7LmGInr0RxXFV/B1DVvfnsa0xY8Hg8jB8/HvA28Rs1ahR79+6lW7duLkdmTPGV1xzF+X5rZQtwgf/a2ap6e34HF5HmwGggFpioqi/msl974D2goaqm5rSPMflZsWIFrVu3Zs+ePWRkZJCYmMhDDz3kdljGFHt5JYp22R6/VpADi0gs3rW2bwLSgRUiMt//ngzffuWBB4GUghzfmNMyMzPp1q0b777r7X7ftm1beve2kVJjgiXXRKGqnxbx2I3w9oXaDCAiM4G2wNps+z0DjAAeLeL7mSh17rnnsmvXLv72t7/xwQcfcO2117odkjERxclmNjWA7X6P04Gr/HcQkSuAmqq6QERyTRQicj9wP3g/FIzJyMggLi6OuLg42rZty9GjR5k8eTIxMTaVZkywOflbJTls06wnRWLw9pF6JL8DqeoEVU1Q1QTrx2Pef/99KleunDVB/cYbbzB16lRLEsY4JOCKQkRKqerxAhw7He9626fFAzv9HpcHLgM+FxGA6sB8EbkloAnt6R1gw38LEI4p7g4fPkybNm1Yvnw5IkKNGjXcDsmYqBBIm/FGIvIDsMH3+B8iMiaAY68A6ohIbV8TwU7A/NNPquohVa2iqrVUtRbwNRBYkoA/JYnPTtUP6CWm+Jo6dSrVqlVj+fLlnH/++axfv54RI6xZgDGhEEitngS0BvYDqOr3wHX5vUhVM4F+wGLgJ2C2qq4RkeEickvhQ/6zWhnv0vPkoGAdzoSpnTt3kpmZydNPP82mTZuoU6eO2yEZEzUCGXqKUdWtvuGh004FcnBVXQgszLbtyVz2vTaQY5roMXLkSL788ks++OADHnvsMfr27WvtN4xxQSCJYruINALUd29Ef+BnZ8MqmOsusgnuSLJ161aaNWvG+vXr/9TEz5KEMe4IZOipN/AwcC7wC3A1YdT3Ke3FVkzu0cjtMEyQPP7441lzEI0aNWLXrl2WIIxxWb4VharuwTsRbYyjkpOTeeGFFyhdujTjx4/n7rvvdjskYwwBJAoReRO/+x9OU9X7HYnIRBWPx8PYsWPp168f11xzDUlJSfTo0YNy5cq5HZoxxieQOYpP/L4vDdzGn++4NqZQkpOTadu2LXv37uXUqVMkJibSv39/t8MyxmQTyNDTLP/HIvIOsMSxiEzEy8zMpGvXrsya5f2n1a5dO2viZ0wYK0yvp9rAecEOxESPmjVrsnv3bipVqsSHH35IkyZN3A7JGJOHQOYofuWPOYoY4AAw2MmgTOTxb+LXvn17fv/9dyZOnGj9mYwpBvL8LRXvXXb/AKr6vv6mquer6uxQBGciw+zZs6lUqVLWVUxjxoxh0qRJliSMKSbyrChUVUVkrqo2CFVAJnIcPHiQ1q1b8+WXXyIinHeejVgaUxwFMkfxjYhcqaorHY/GRIzJkyfTq1cvTpw4wQUXXMDixYu54IIL3A7LGFMIudb+InI6iTTBmyzWi8hKEflORCxpmDydvuT12WefZePGjZYkjCnG8qoovgGuBG4NUSymmHvhhRdISUlh3rx5DBo0iF69eln7DWMiQF6JQgBUdVOIYjHF1JYtW7j55pvZuHEjpUqVsiZ+xkSYvBJFVRF5OLcnVfUVB+IxxcygQYMYOXIkHo+Ha665hoULF1qCMCbC5HV9YixQDu+SpTl9mSiXnJzMyy+/TKlSpXj33Xf56quvqFixotthGWOCLK+KYpeqDg9ZJKZY8Hg8jBkzhsTERK655hpef/11unfvTtmyZd0OzRjjkHznKIw57csvv6Rt27bs378fgMTERPr06eNyVMYYp+U19HRDyKIwYS0zM5MOHTrQpEkT9u/fT/v27enbt6/bYRljQiTXikJVD4QyEBO+4uPj+eWXX6hcuTIffvghjRs3djskY0wIFaZ7rIkCR48epWTJksTFxdGhQweOHz/OuHHjrD+TMVHIfuvNX0yfPp0qVarQtWtXwNvEb8KECZYkjIlSVlGYLAcOHKBVq1Z8/fXXxMTEUKdOHbdDMsaEAUsUBoCJEyfSp08fTp48SZ06dVi8eDG1a9d2OyxjTBiwsQQDwK+//oqq8tJLL/Hzzz9bkjDGZBFVzX+vMFLq7Dp6drdXSSt9p3fDsEPuBlSMPfvss6SkpPCf//wHgN9++41y5cq5HJUxxgki8q2qJhTmtTb0FIU2bNhA8+bN2bx5M6VLl85KEJYkjDE5KZZDT2kvtnI7hGLJ4/HwyCOPcNFFF7F582aaNGnCL7/8YgnCGJOnYpkoTOEkJyfzyiuvUKZMGWbNmsUXX3xhnV6NMfmyRBHhPB4PI0eOBKBx48aMGzeO/fv3c8cdd7gcmTGmuLA5igi2fPlybrvtNg4cOEBcXByJiYk88MADbodljClmrKKIQCdOnOD222+nadOmHDhwgM6dO1sTP2NMoVlFEYFq1qzJnj17qFq1Kv/5z3+46qqr3A7JGFOMFbtEcblshmFnuh1G2PFv4tepUydOnjzJa6+9Zv2ZjDFF5uiniIg0F5H1IrJRRAbn8PzDIrJWRFaLyKcicl6B3qDOzUGLtTh7++23qVy5Mnfe6b0JcfTo0YwdO9aShDEmKByrKEQkFngduAlIB1aIyHxVXeu323dAgqoeFZHewAigY74Ht7uxAW8TvxYtWvDNN98QExPDxRdf7HZIxpgI5OSfnI2Ajaq6WVVPADOBtv47qOpSVT3qe/g1EO9gPBFlwoQJVK9enW+++SbrBrrhw22Jc2NM8DmZKGoA2/0ep/u25eYeYFFOT4jI/SKSKiKpQYyvWDty5AiqyogRI1i3bh3nnVewUTtjjAmUk4lCctiWYwdCEekKJAAv5/S8qk5Q1YTCNrSKFE8//TStWkBW42IAABKdSURBVHnblzzyyCMcOnSIgQMHuhyVMSbSOXnVUzpQ0+9xPLAz+04iciPwBNBUVY87GE+xtX79epo3b05aWtqfmviVLVvW7dCMMVHAyYpiBVBHRGqLSEmgEzDffwcRuQIYD9yiqnscjKVY8ng8PPjgg9StW5e0tDSaNm3K3r17rYmfMSakHEsUqpoJ9AMWAz8Bs1V1jYgMF5FbfLu9DJQD3hORVSIyP5fDRaXk5GTGjBlD2bJlef/99/n8888tSRhjQq7YLVyUcE6spu485XYYjvF4PPz73/9m0KBBgHeJ0rvvvpuSJUu6HJkxpjizhYsixGeffUa7du04ePAgpUqVIjExkXvvvdftsIwJqpMnT5Kenk5GRobboUSk0qVLEx8fT4kSJYJ2TEsUYeDEiRN06NCB+fO9I29dunSxJn4mYqWnp1O+fHlq1aqFSE4XR5rCUlX2799Penp6UNe9t0QRBuLj49m7dy/VqlVjwYIFNGzY0O2QjHFMRkaGJQmHiAiVK1dm7969QT2uJQqX/Pbbb5QuXZq4uDi6dOlCZmYmo0ePtv5MJipYknCOE+fWPpVcMHnyZKpUqULnzp0BGDVqFGPGjLEkYYwJS/bJFEL79u0jISGBnj17cvLkSS6//HK3QzIm6lx77bUsXrz4T9teffVV+vTpU+BjPfnkk3zyySdZx01N9XYZqlWrFvv27St6sGHCEkWIjB07lrPPPptvv/2WunXrsmXLFp588km3wzIm6nTu3JmZM2f+advMmTOzKvyCGD58ODfeeGOwQgtbNkcRIhkZGYgIo0aN4qGHHnI7HGPCQq3BHzly3LQXW+X6XPv27RkyZAjHjx+nVKlSpKWlsXPnTurXr88NN9zAr7/+ysmTJ3n22Wdp27YtaWlptGjRgiZNmvDVV19Ro0YNPvzwQ8qUKUP37t1p3bo17du3z/X9br31VrZv305GRgaJiYncf//9TvzIjrKKwkFDhw6lZcuWADz88MMcPnzYkoQxLqtcuTKNGjXi448/BrzVRMeOHSlTpgxz585l5cqVLF26lEceeYTTNyRv2LCBvn37smbNGipWrMicOXMCfr9Jkybx7bffkpqaSlJSEvv373fk53KSVRQO+Omnn2jRogVbt26lTJkyWU38Spcu7XZoxoSVvP7yd9Lp4ae2bdsyc+ZMJk2ahKry+OOPs3z5cmJiYtixYwe//PILALVr16Z+/foANGjQgLS0tIDfKykpiblz5wKwfft2NmzYQOXKlYP+MznJKoog8ng89O3bl0svvZStW7dy/fXXs2fPHuvPZEyYufXWW/n0009ZuXIlx44d48orr2T69Ons3buXb7/9llWrVnHWWWdl3T1eqlSprNfGxsaSmZkZ0Pt8/vnnfPLJJyQnJ/P9999zxRVXFMs70i1RBFFycjJjx46lbNmyzJs3j08//dSShDFhqFy5clx77bX07NkzaxL70KFDVKtWjRIlSrB06VK2bt1a5Pc5dOgQf/vb3yhbtizr1q3j66+/LvIx3WCJoogyMzN57rnnAGjcuDGTJk3iwIEDtG3bNp9XGmPc1LlzZ77//ns6deoEeFvnpKamkpCQwPTp04OyBn3z5s3JzMykXr16DB06lKuvvrrIx3SDdY8tgiVLltChQwcOHTpEUlIS/fv3dzskY8LeTz/9RN26dd0OI6LldI6te2yIZWRk0K5dOxYuXAhA9+7drYmfMSZiWaIohPj4ePbv30/16tX56KOPuPLKK90OyRhjHGOJIkCHDx+mbNmyxMXF0a1bNzweDyNHjrT+TMaYiGefcgGYOHEi1apVy5r0GjlyJKNGjbIkYYyJClZR5GHPnj00b96c7777jtjY2KwbbowxJppYosjFa6+9xoABA8jMzOSyyy5j0aJFxMfHux2WMcaEnI2d5OLEiROICElJSfzwww+WJIyJIDndCPvKK69wySWXUK9ePW644YZcb7gbNmwYNWrUoH79+lx22WVZSxgH0+k25QcPHmTs2LFBP35BWaLw89hjj9GsWTPgjyZ+dm+EMdHhiiuuIDU1ldWrV9O+fXsGDRqU674DBgxg1apVvPfee/Ts2ROPx+NITOGSKGzoCfjxxx9p0aIF6enp1sTPmFAadqZDxz1U4Jdcd911Wd9fffXVTJs2Ld/X1K1bl7i4OPbt24eI0KtXL7Zt2wZ4F0Nq3Lgxw4YNY9u2bWzevJlt27bx0EMP8eCDDwL5tyAfPHgwmzZton79+tx0003s3r2b9u3bZ3V+6NKlCx07duSWW24p8M9bEFGdKDweD3369GHChAmoKjfddBPz5s2jbNmybodmjHHRW2+9RYsWLfLdLyUlhZiYGKpWrUqXLl0YMGAATZo0Ydu2bTRr1oyffvoJgHXr1rF06VKOHDnCRRddRO/evSlRogSTJk2iUqVKHDt2jIYNG9KuXbs/dZZ98cUX+fHHH1m1ahUAy5YtY9SoUbRt25ZDhw7x1VdfMXXqVGdOgp+oThTJycmMHz+e8uXLM3PmzKy1I4wxIVKIv/ydNm3aNFJTU1m2bFmu+4waNYpp06ZRvnx5Zs2ahYjwySefsHbt2qx9Dh8+zJEjRwBo1aoVpUqVolSpUlSrVo1ffvmF+Pj4Arcgb9q0KX379mXPnj188MEHtGvXjrg45z/Goy5RZGZm8vzzz/Pkk0/SuHFjpk6dSqdOnShZsqTboRljXPbJJ5/w3HPPsWzZsqzW4k888QQffeRdie/0X/YDBgzg0Ucf/dNrPR4PycnJlClT5i/HzalNuX8L8rJly3LttdcG1IL8rrvuYvr06VnraIRCVE1mL168mMqVK/PUU08xevRoAO6++25LEsYYvvvuOx544AHmz59PtWrVsrY/99xzrFq1KitJ5Obmm2/mtddey3qc3/6BtCAvX758VlVyWvfu3Xn11VcBuPTSS/P9uYIhKhJFRkYGLVq0oHnz5hw5coR77rnHrmYyJoodPXqU+Pj4rK9XXnmFgQMH8ttvv9GhQwfq169f4AnipKQkUlNTqVevHpdccgnjxo3Lc/9AWpBXrlyZxo0bc9lllzFw4EAAzjrrLOrWrUuPHj0KFF9RREWb8SpVqrB//37OOeccFi1aRL169RyKzhiTH2szXjRHjx7l8ssvZ+XKlZx5Zs5XjQW7zXjEVhQHDx7kxIkTAPTo0YNHH32UHTt2WJIwxhRbn3zyCRdffDH9+/fPNUk4ISIns9944w0eeughWrduzZw5c3j55ZfdDskYY4rsxhtvzLpPI5QiKlHs3r2bZs2asXr1amJjY2nYsKHbIRljcqCqiIjbYUQkJ6YTImboafTo0cTHx7N69Wrq1atHeno6gwcPdjssY0w2pUuXZv/+/Y58oEU7VWX//v1B7yoRURVFbGwsY8aMoXfv3m6HYozJRXx8POnp6ezdu9ftUCJS6dKlg97EtFhf9TRo0CBWrVrFf//7X8Db8dXuiTDGmL8qylVPjlYUItIcGA3EAhNV9cVsz5cC3gYaAPuBjqqalt9xV69eTcuWLdmxYwdly5bNauJnScIYY4LPsTkKEYkFXgdaAJcAnUXkkmy73QP8qqp/B0YBL+V33P1Hlfr167Njxw6aNWvG3r17c+wtb4wxJjicnMxuBGxU1c2qegKYCbTNtk9b4HTrw/eBGySfSyG2HlLKly/PwoUL+fjjj63TqzHGOMzJoacawHa/x+nAVbnto6qZInIIqAzs899JRO4HTjdqP3748OEfrdMrAFXIdq6imJ2LP9i5+IOdiz9cVNgXOpkocqoMss+cB7IPqjoBmAAgIqmFnZCJNHYu/mDn4g92Lv5g5+IPIpJa2Nc6OfSUDtT0exwP7MxtHxGJA84EDjgYkzHGmAJyMlGsAOqISG0RKQl0ArKvQj4f6Ob7vj3wmRa363WNMSbCOTb05Jtz6Acsxnt57CRVXSMiw4FUVZ0PvAW8IyIb8VYSnQI49ASnYi6G7Fz8wc7FH+xc/MHOxR8KfS6K3Q13xhhjQitiej0ZY4xxhiUKY4wxeQrbRCEizUVkvYhsFJG/tIEVkVIiMsv3fIqI1Ap9lKERwLl4WETWishqEflURM5zI85QyO9c+O3XXkRURCL20shAzoWI3OH7t7FGRN4NdYyhEsDvyLkislREvvP9nkTkjVgiMklE9ojIj7k8LyKS5DtPq0XkyoAOrKph94V38nsTcD5QEvgeuCTbPn2Acb7vOwGz3I7bxXNxHVDW933vaD4Xvv3KA8uBr4EEt+N28d9FHeA74G++x9XcjtvFczEB6O37/hIgze24HToX/wdcCfyYy/MtgUV472G7GkgJ5LjhWlE40v6jmMr3XKjqUlU96nv4Nd57ViJRIP8uAJ4BRgAZoQwuxAI5F/cBr6vqrwCquifEMYZKIOdCgQq+78/kr/d0RQRVXU7e96K1Bd5Wr6+BiiJydn7HDddEkVP7jxq57aOqmcDp9h+RJpBz4e8evH8xRKJ8z4WIXAHUVNUFoQzMBYH8u7gQuFBEvhSRr33dnCNRIOdiGNBVRNKBhUD/0IQWdgr6eQKE78JFQWv/EQEC/jlFpCuQADR1NCL35HkuRCQGbxfi7qEKyEWB/LuIwzv8dC3eKvMLEblMVQ86HFuoBXIuOgNTVHWkiFyD9/6ty1TV43x4YaVQn5vhWlFY+48/BHIuEJEbgSeAW1T1eIhiC7X8zkV54DLgcxFJwzsGOz9CJ7QD/R35UFVPquoWYD3exBFpAjkX9wCzAVQ1GSiNt2FgtAno8yS7cE0U1v7jD/meC99wy3i8SSJSx6Ehn3OhqodUtYqq1lLVWnjna25R1UI3QwtjgfyOzMN7oQMiUgXvUNTmkEYZGoGci23ADQAiUhdvoojGtVjnA3f7rn66Gjikqrvye1FYDj2pc+0/ip0Az8XLQDngPd98/jZVvcW1oB0S4LmICgGei8XAzSKyFjgFDFTV/e5F7YwAz8UjwJsiMgDvUEv3SPzDUkRm4B1qrOKbj3kKKAGgquPwzs+0BDYCR4EeAR03As+VMcaYIArXoSdjjDFhwhKFMcaYPFmiMMYYkydLFMYYY/JkicIYY0yeLFGYsCMip0Rkld9XrTz2rZVbp8wCvufnvu6j3/taXlxUiGP0EpG7fd93F5Fz/J6bKCKXBDnOFSJSP4DXPCQiZYv63iZ6WaIw4eiYqtb3+0oL0ft2UdV/4G02+XJBX6yq41T1bd/D7sA5fs/dq6prgxLlH3GOJbA4HwIsUZhCs0RhigVf5fCFiKz0ff2/HPa5VES+8VUhq0Wkjm97V7/t40UkNp+3Ww783ffaG3xrGPzg6/Vfyrf9RfljDZB/+7YNE5FHRaQ93p5b033vWcZXCSSISG8RGeEXc3cRGVPIOJPxa+gmIm+ISKp415542rftQbwJa6mILPVtu1lEkn3n8T0RKZfP+5goZ4nChKMyfsNOc33b9gA3qeqVQEcgKYfX9QJGq2p9vB/U6b52DR2Bxr7tp4Au+bx/G+AHESkNTAE6qurleDsZ9BaRSsBtwKWqWg941v/Fqvo+kIr3L//6qnrM7+n3gdv9HncEZhUyzuZ423Sc9oSqJgD1gKYiUk9Vk/D28rlOVa/ztfIYAtzoO5epwMP5vI+JcmHZwsNEvWO+D0t/JYDXfGPyp/D2LcouGXhCROKBD1R1g4jcADQAVvjam5TBm3RyMl1EjgFpeNtQXwRsUdWffc9PBfoCr+Fd62KiiHwEBNzSXFX3ishmX5+dDb73+NJ33ILEeQbedhX+K5TdISL34/29PhvvAj2rs732at/2L33vUxLveTMmV5YoTHExAPgF+AfeSvgvixKp6rsikgK0AhaLyL142ypPVdXHAniPLv4NBEUkx/VNfL2FGuFtMtcJ6AdcX4CfZRZwB7AOmKuqKt5P7YDjxLuK24vA68DtIlIbeBRoqKq/isgUvI3vshNgiap2LkC8JsrZ0JMpLs4EdvnWD7gL71/TfyIi5wObfcMt8/EOwXwKtBeRar59Kknga4qvA2qJyN99j+8ClvnG9M9U1YV4J4pzuvLoCN625zn5ALgV7xoJs3zbChSnqp7EO4R0tW/YqgLwO3BIRM4CWuQSy9dA49M/k4iUFZGcqjNjsliiMMXFWKCbiHyNd9jp9xz26Qj8KCKrgIvxLvm4Fu8H6n9FZDWwBO+wTL5UNQNvd833ROQHwAOMw/uhu8B3vGV4q53spgDjTk9mZzvur8Ba4DxV/ca3rcBx+uY+RgKPqur3eNfHXgNMwjucddoEYJGILFXVvXivyJrhe5+v8Z4rY3Jl3WONMcbkySoKY4wxebJEYYwxJk+WKIwxxuTJEoUxxpg8WaIwxhiTJ0sUxhhj8mSJwhhjTJ7+P4KKvJX6MybUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_roc_curve(fpr, tpr, label=None):\n",
    "    plt.plot(fpr, tpr, linewidth=2, label=label)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.axis([0, 1, 0, 1])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "\n",
    "\n",
    "plot_roc_curve(fpr_v, tpr_v, label='Vanilla')\n",
    "plot_roc_curve(fpr_l2, tpr_l2, label='L2-Penalty')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also we can measure the __A__rea __U__nder __C__urve scores\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9401004057492607"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "roc_auc_score(y_train, y_probs_vanilla[:,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9216697613644179"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_train, y_probs_l2[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Default Measure (in most prebuilt models) - Accuracy\n",
    "\n",
    "$$ \\frac{(TP + TN)}{(TP + FP + TN + FN)} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Category definitions - possible outcomes in binary classification\n",
    "\n",
    "- TP = True Positive (class 1 correctly classified as class 1) - e.g. Patient with cancer tests positive for cancer\n",
    "- TN = True Negative (class 0 correctly classified as class 0) - e.g. Patient without cancer tests negative for cancer\n",
    "- FP = False Positive (class 0 incorrectly classified as class 1) - e.g. Patient without cancer tests positive for cancer\n",
    "- FN = False Negative (class 1 incorrectly classified as class 0) - e.g. Patient with cancer tests negative for cancer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " $$ \\text{Possible misclassifications} $$\n",
    "\n",
    "<img src='./img/type-1-type-2.jpg' width=400/>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that Logistic Regression gives probability predictions for each class, in addition to the final classification. By default, threshold for the prediction is set to 0.5, but we can adjust that threshold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The AUC / ROC curve (Area Under Curve of the Receiver Operating Characteristic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/pop-curve.png' width=500/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Cross Validation Scores for Model Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.95652174, 0.88047138, 0.88986014, 0.86188811, 0.86538462])"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(pipe, X, y_train, cv=5, scoring='roc_auc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8908251973469365"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.95819398, 0.88552189, 0.86713287, 0.88811189, 0.91608392])"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(pipe_l2, X, y_train, cv=5, scoring='roc_auc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9030089073567333"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}