1-data-science-and-ml-tools/scikit-learn-exercises-solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scikit-Learn Practice Solutions\n",
    "\n",
    "This notebook offers a set of potential solutions to the Scikit-Learn excercise notebook.\n",
    "\n",
    "Exercises are based off (and directly taken from) the quick [introduction to Scikit-Learn notebook](https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/section-2-data-science-and-ml-tools/introduction-to-scikit-learn.ipynb).\n",
    "\n",
    "Different tasks will be detailed by comments or text.\n",
    "\n",
    "For further reference and resources, it's advised to check out the [Scikit-Learn documnetation](https://scikit-learn.org/stable/user_guide.html).\n",
    "\n",
    "And if you get stuck, try searching for a question in the following format: \"how to do XYZ with Scikit-Learn\", where XYZ is the function you want to leverage from Scikit-Learn.\n",
    "\n",
    "Since we'll be working with data, we'll import Scikit-Learn's counterparts, Matplotlib, NumPy and pandas.\n",
    "\n",
    "Let's get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup matplotlib to plot inline (within the notebook)\n",
    "%matplotlib inline\n",
    "\n",
    "# Import the pyplot module of Matplotlib as plt\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Import pandas under the abbreviation 'pd'\n",
    "import pandas as pd\n",
    "\n",
    "# Import NumPy under the abbreviation 'np'\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End-to-end Scikit-Learn classification workflow\n",
    "\n",
    "Let's start with an end to end Scikit-Learn workflow.\n",
    "\n",
    "More specifically, we'll:\n",
    "1. Get a dataset ready\n",
    "2. Prepare a machine learning model to make predictions\n",
    "3. Fit the model to the data and make a prediction\n",
    "4. Evaluate the model's predictions \n",
    "\n",
    "The data we'll be using is [stored on GitHub](https://github.com/mrdbourke/zero-to-mastery-ml/tree/master/data). We'll start with [`heart-disease.csv`](https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/heart-disease.csv), a dataset which contains anonymous patient data and whether or not they have heart disease.\n",
    "\n",
    "**Note:** When viewing a `.csv` on GitHub, make sure it's in the raw format. For example, the URL should look like: https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/heart-disease.csv\n",
    "\n",
    "### 1. Getting a dataset ready"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n",
       "0   63    1   3       145   233    1        0      150      0      2.3      0   \n",
       "1   37    1   2       130   250    0        1      187      0      3.5      0   \n",
       "2   41    0   1       130   204    0        0      172      0      1.4      2   \n",
       "3   56    1   1       120   236    0        1      178      0      0.8      2   \n",
       "4   57    0   0       120   354    0        1      163      1      0.6      2   \n",
       "\n",
       "   ca  thal  target  \n",
       "0   0     1       1  \n",
       "1   0     2       1  \n",
       "2   0     2       1  \n",
       "3   0     2       1  \n",
       "4   0     2       1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the heart disease dataset and save it to a variable\n",
    "# using pandas and read_csv()\n",
    "# Hint: You can directly pass the URL of a csv to read_csv()\n",
    "heart_disease = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/heart-disease.csv\")\n",
    "\n",
    "# Check the first 5 rows of the data\n",
    "heart_disease.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal here is to build a machine learning model on all of the columns except `target` to predict `target`.\n",
    "\n",
    "In essence, the `target` column is our **target variable** (also called `y` or `labels`) and the rest of the other columns are our independent variables (also called `data` or `X`).\n",
    "\n",
    "And since our target variable is one thing or another (heart disease or not), we know our problem is a classification problem (classifying whether something is one thing or another).\n",
    "\n",
    "Knowing this, let's create `X` and `y` by splitting our dataframe up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create X (all columns except target)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "\n",
    "# Create y (only the target column)\n",
    "y = heart_disease[\"target\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we've split our data into `X` and `y`, we'll use Scikit-Learn to split it into training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import train_test_split from sklearn's model_selection module\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Use train_test_split to split X & y into training and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((227, 13), (76, 13), (227,), (76,))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the different shapes of the training and test datasets\n",
    "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the different shapes of the data?\n",
    "\n",
    "Since our data is now in training and test sets, we'll build a machine learning model to fit patterns in the training data and then make predictions on the test data.\n",
    "\n",
    "To figure out which machine learning model we should use, you can refer to [Scikit-Learn's machine learning map](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html).\n",
    "\n",
    "After following the map, you decide to use the [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).\n",
    "\n",
    "### 2. Preparing a machine learning model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the RandomForestClassifier from sklearn's ensemble module\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Instantiate an instance of RandomForestClassifier as clf\n",
    "clf = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you've got a `RandomForestClassifier` instance, let's fit it to the training data.\n",
    "\n",
    "Once it's fit, we'll make predictions on the test data.\n",
    "\n",
    "### 3. Fitting a model and making predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,\n",
       "                       criterion='gini', max_depth=None, max_features='auto',\n",
       "                       max_leaf_nodes=None, max_samples=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=1, min_samples_split=2,\n",
       "                       min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                       n_jobs=None, oob_score=False, random_state=None,\n",
       "                       verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit the RandomForestClassifier to the training data\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the fitted model to make predictions on the test data and\n",
    "# save the predictions to a variable called y_preds\n",
    "y_preds = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Evaluating a model's predictions\n",
    "\n",
    "Evaluating predictions is as important making them. Let's check how our model did by calling the `score()` method on it and passing it the training (`X_train, y_train`) and testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate the fitted model on the training set using the score() function\n",
    "clf.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8026315789473685"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate the fitted model on the test set using the score() function\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* How did you model go? \n",
    "* What metric does `score()` return for classifiers? \n",
    "* Did your model do better on the training dataset or test dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experimenting with different classification models\n",
    "\n",
    "Now we've quickly covered an end-to-end Scikit-Learn workflow and since experimenting is a large part of machine learning, we'll now try a series of different machine learning models and see which gets the best results on our dataset.\n",
    "\n",
    "Going through the [Scikit-Learn machine learning map](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html), we see there are a number of different classification models we can try (different models are in the green boxes).\n",
    "\n",
    "For this exercise, the models we're going to try and compare are:\n",
    "* [LinearSVC](https://scikit-learn.org/stable/modules/svm.html#classification)\n",
    "* [KNeighborsClassifier](https://scikit-learn.org/stable/modules/neighbors.html) (also known as K-Nearest Neighbors or KNN)\n",
    "* [SVC](https://scikit-learn.org/stable/modules/svm.html#classification) (also known as support vector classifier, a form of [support vector machine](https://en.wikipedia.org/wiki/Support-vector_machine))\n",
    "* [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) (despite the name, this is actually a classifier)\n",
    "* [RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) (an ensemble method and what we used above)\n",
    "\n",
    "We'll follow the same workflow we used above (except this time for multiple models):\n",
    "1. Import a machine learning model\n",
    "2. Get it ready\n",
    "3. Fit it to the data and make predictions\n",
    "4. Evaluate the fitted model\n",
    "\n",
    "**Note:** Since we've already got the data ready, we can reuse it in this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import LinearSVC from sklearn's svm module\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "# Import KNeighborsClassifier from sklearn's neighbors module\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# Import SVC from sklearn's svm module\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "# Import LogisticRegression from sklearn's linear_model module\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Note: we don't have to import RandomForestClassifier, since we already have"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to the consistency of Scikit-Learn's API design, we can use virtually the same code to fit, score and make predictions with each of our models.\n",
    "\n",
    "To see which model performs best, we'll do the following:\n",
    "1. Instantiate each model in a dictionary\n",
    "2. Create an empty results dictionary\n",
    "3. Fit each model on the training data\n",
    "4. Score each model on the test data\n",
    "5. Check the results\n",
    "\n",
    "If you're wondering what it means to instantiate each model in a dictionary, see the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EXAMPLE: Instantiating a RandomForestClassifier() in a dictionary\n",
    "example_dict = {\"RandomForestClassifier\": RandomForestClassifier()}\n",
    "\n",
    "# Create a dictionary called models which contains all of the classification models we've imported\n",
    "# Make sure the dictionary is in the same format as example_dict\n",
    "# The models dictionary should contain 5 models\n",
    "models = {\"LinearSVC\": LinearSVC(),\n",
    "          \"KNN\": KNeighborsClassifier(),\n",
    "          \"SVC\": SVC(),\n",
    "          \"LogisticRegression\": LogisticRegression(),\n",
    "          \"RandomForestClassifier\": RandomForestClassifier()}\n",
    "\n",
    "# Create an empty dictionary called results\n",
    "results = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since each model we're using has the same `fit()` and `score()` functions, we can loop through our models dictionary and, call `fit()` on the training data and then call `score()` with the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'RandomForestClassifier': 0.7631578947368421}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# EXAMPLE: Looping through example_dict fitting and scoring the model\n",
    "example_results = {}\n",
    "for model_name, model in example_dict.items():\n",
    "    model.fit(X_train, y_train)\n",
    "    example_results[model_name] = model.score(X_test, y_test)\n",
    "\n",
    "example_results    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/daniel/Desktop/ml-course/zero-to-mastery-ml/env/lib/python3.6/site-packages/sklearn/svm/_base.py:947: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "/Users/daniel/Desktop/ml-course/zero-to-mastery-ml/env/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:939: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html.\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'LinearSVC': 0.7631578947368421,\n",
       " 'KNN': 0.6447368421052632,\n",
       " 'SVC': 0.618421052631579,\n",
       " 'LogisticRegression': 0.7894736842105263,\n",
       " 'RandomForestClassifier': 0.7631578947368421}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Loop through the models dictionary items, fitting the model on the training data\n",
    "# and appending the model name and model score on the test data to the results dictionary\n",
    "for model_name, model in models.items():\n",
    "    model.fit(X_train, y_train)\n",
    "    results[model_name] = model.score(X_test, y_test)\n",
    "    \n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Which model performed the best? \n",
    "* Do the results change each time you run the cell? \n",
    "* Why do you think this is?\n",
    "\n",
    "Due to the randomness of how each model finds patterns in the data, you might notice different results each time.\n",
    "\n",
    "Without manually setting the random state using the `random_state` parameter of some models or using a NumPy random seed, every time you run the cell, you'll get slightly different results.\n",
    "\n",
    "Let's see this in effect by running the same code as the cell above, except this time setting a [NumPy random seed equal to 42](https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.random.seed.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/daniel/Desktop/ml-course/zero-to-mastery-ml/env/lib/python3.6/site-packages/sklearn/svm/_base.py:947: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n",
      "/Users/daniel/Desktop/ml-course/zero-to-mastery-ml/env/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:939: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html.\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'LinearSVC': 0.7236842105263158,\n",
       " 'KNN': 0.6447368421052632,\n",
       " 'SVC': 0.618421052631579,\n",
       " 'LogisticRegression': 0.7894736842105263,\n",
       " 'RandomForestClassifier': 0.7763157894736842}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run the same code as the cell above, except this time set a NumPy random seed\n",
    "# equal to 42\n",
    "np.random.seed(42)\n",
    "\n",
    "for model_name, model in models.items():\n",
    "    model.fit(X_train, y_train)\n",
    "    results[model_name] = model.score(X_test, y_test)\n",
    "    \n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Run the cell above a few times, what do you notice about the results? \n",
    "* Which model performs the best this time?\n",
    "* What happens if you add a NumPy random seed to the cell where you called `train_test_split()` (towards the top of the notebook) and then rerun the cell above?\n",
    "\n",
    "Let's make our results a little more visual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAFlCAYAAAD76RNtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dfbxVZZ3+8c8lSuRjidSooJBiRSk+4EPp2IM6WVOYD2OYZWrqzBRlD69+0WT+zH79prGmxhorNc1qDFIniQqzmdRKywIfUsEoJM2ToyE+RyLINX+sdXCz2fucjR3O2qx9vV+v82Kvey02X7dwnfvc6173LdtERMTGb5OqC4iIiKGRQI+IqIkEekRETSTQIyJqIoEeEVETCfSIiJrYtKo/eLvttvP48eOr+uMjIjZKN91004O2x7Q6V1mgjx8/nvnz51f1x0dEbJQk3dPuXIZcIiJqIoEeEVETHQW6pMMlLZK0WNKMFud3knStpFsk3SbpDUNfakREDGTQMXRJI4DzgMOAPmCepDm2FzZcdgZwme0vSZoEzAXGr28xK1eupK+vjyeffHJ9f2uURo0axdixY9lss82qLiUihlknN0X3AxbbXgIgaRZwBNAY6Aa2Ll9vA9z3bIrp6+tjq622Yvz48Uh6Nm/R02yzbNky+vr6mDBhQtXlRMQw62TIZUfg3objvrKt0VnA2yT1UfTO39PqjSSdJmm+pPlLly5d5/yTTz7J6NGjE+bPkiRGjx6dn3AielQngd4qXZvX3D0OuMT2WOANwDckrfPeti+wPcX2lDFjWk6jTJj/hfL5RfSuTgK9DxjXcDyWdYdU3glcBmD758AoYLuhKLAKV155JZL49a9/XXUpEREd62QMfR4wUdIE4A/ANOCtTdf8HjgEuETSSykCfd0xlfU0fsb3/9K3WMvdn/rbjq6bOXMmBx10ELNmzeKss84a0hr6Pf3004wYMWKDvHdEv6H+N/RsdPrvLv5yg/bQba8CpgNXA3dSzGZZIOlsSVPLyz4InCrpV8BM4ERvpFshPfHEE9xwww1cdNFFzJo1a037Oeecw+67787kyZOZMaOYubl48WIOPfRQJk+ezN57781dd93Fddddxxvf+MY1v2/69OlccsklQPF07Nlnn81BBx3E5ZdfzoUXXsi+++7L5MmTOfroo1m+fDkADzzwAEceeSSTJ09m8uTJ/OxnP+NjH/sY55577pr3/ehHP8rnP//5YfhEImJj0dGj/7bnUtzsbGw7s+H1QuDAoS2tGrNnz+bwww9nt912Y9ttt+Xmm2/mgQceYPbs2fziF79g880356GHHgLg+OOPZ8aMGRx55JE8+eSTrF69mnvvvXfA9x81ahTXX389AMuWLePUU08F4IwzzuCiiy7iPe95D+9973t51atexZVXXsnTTz/NE088wQ477MBRRx3F6aefzurVq5k1axa//OUvN+yHEREblcrWculWM2fO5H3vex8A06ZNY+bMmaxevZqTTjqJzTffHIBtt92Wxx9/nD/84Q8ceeSRQBHUnXjLW96y5vUdd9zBGWecwSOPPMITTzzB6173OgCuueYavv71rwMwYsQIttlmG7bZZhtGjx7NLbfcwgMPPMBee+3F6NGjh+y/OyI2fgn0BsuWLeOaa67hjjvuQBJPP/00kjj66KPXmT3SbkRp0003ZfXq1WuOm6cQbrHFFmten3jiicyePZvJkydzySWXcN111w1Y3ymnnMIll1zC/fffz8knn7ye/3URva0X7idkLZcGV1xxBSeccAL33HMPd999N/feey8TJkxg22235eKLL14zxv3QQw+x9dZbM3bsWGbPng3AihUrWL58OTvvvDMLFy5kxYoVPProo/zoRz9q++c9/vjjbL/99qxcuZJLL710TfshhxzCl770JaC4efrYY48BcOSRR/KDH/yAefPmrenNR0T0S6A3mDlz5pohlH5HH3009913H1OnTmXKlCnsueeefOYznwHgG9/4Bp///OfZY489eOUrX8n999/PuHHjOPbYY9ljjz04/vjj2Wuvvdr+eZ/4xCfYf//9Oeyww3jJS16ypv3cc8/l2muvZffdd2efffZhwYIFAIwcOZLXvOY1HHvssZkhExHrUFWTUaZMmeLm9dDvvPNOXvrSl1ZSz8Zg9erV7L333lx++eVMnDix7XX5HKNfLwwzdKoun4Wkm2xPaXUuPfSNxMKFC9l111055JBDBgzziOhduSm6kZg0aRJLliypuoyI6GLpoUdE1ETXBfpG+oBp18jnF9G7uirQR40axbJlyxJKz1L/euidPuQUEfXSVWPoY8eOpa+vj1ZrpUdn+ncsioje01WBvtlmm2WnnYiIZ6mrhlwiIuLZS6BHRNREVw25rK+6PPkVETEU0kOPiKiJBHpERE10FOiSDpe0SNJiSTNanP+cpFvLr99IemToS42IiIEMOoYuaQRwHnAY0AfMkzSn3HYOANvvb7j+PUD7NWMjImKD6KSHvh+w2PYS208Bs4AjBrj+OIqNoiMiYhh1Eug7Ao07H/eVbeuQtDMwAbjmLy8tIiLWRyeBrhZt7RZbmQZcYfvplm8knSZpvqT5ebw/ImJodRLofcC4huOxwH1trp3GAMMtti+wPcX2lDFjxnReZUREDKqTQJ8HTJQ0QdJIitCe03yRpBcDzwd+PrQlRkREJwYNdNurgOnA1cCdwGW2F0g6W9LUhkuPA2Y5a99GRFSio0f/bc8F5ja1ndl0fNbQlRUREesrT4pGRNREAj0ioiYS6BERNZFAj4ioiQR6RERNJNAjImoigR4RURMJ9IiImkigR0TUxEa9SXQ8IxtmR0R66BERNZFAj4ioiQR6RERNJNAjImoigR4RURMJ9IiImsi0xaidTOGMXtVRD13S4ZIWSVosaUaba46VtFDSAknfHNoyIyJiMIP20CWNAM4DDgP6gHmS5the2HDNROAjwIG2H5b0gg1VcEREtNZJD30/YLHtJbafAmYBRzRdcypwnu2HAWz/cWjLjIiIwXQS6DsC9zYc95VtjXYDdpN0g6QbJR0+VAVGRERnOrkpqhZtbvE+E4FXA2OBn0p6ue1H1noj6TTgNICddtppvYuNiIj2Oumh9wHjGo7HAve1uOY7tlfa/h2wiCLg12L7AttTbE8ZM2bMs605IiJa6CTQ5wETJU2QNBKYBsxpumY28BoASdtRDMEsGcpCIyJiYIMGuu1VwHTgauBO4DLbCySdLWlqednVwDJJC4FrgQ/ZXrahio6IiHV19GCR7bnA3Ka2MxteG/hA+RURERXIo/8RETWRQI+IqIkEekRETSTQIyJqIoEeEVETCfSIiJpIoEdE1EQCPSKiJhLoERE1kUCPiKiJBHpERE0k0CMiaiKBHhFREwn0iIiaSKBHRNREAj0ioiYS6BERNdFRoEs6XNIiSYslzWhx/kRJSyXdWn6dMvSlRkTEQAbdgk7SCOA84DCgD5gnaY7thU2Xfsv29A1QY0REdKCTHvp+wGLbS2w/BcwCjtiwZUVExPrqJNB3BO5tOO4r25odLek2SVdIGjck1UVERMc6CXS1aHPT8XeB8bb3AP4b+FrLN5JOkzRf0vylS5euX6URETGgTgK9D2jscY8F7mu8wPYy2yvKwwuBfVq9ke0LbE+xPWXMmDHPpt6IiGijk0CfB0yUNEHSSGAaMKfxAknbNxxOBe4cuhIjIqITg85ysb1K0nTgamAEcLHtBZLOBubbngO8V9JUYBXwEHDiBqw5IiJaGDTQAWzPBeY2tZ3Z8PojwEeGtrSIiFgfeVI0IqImEugRETWRQI+IqIkEekRETSTQIyJqIoEeEVETCfSIiJpIoEdE1EQCPSKiJhLoERE1kUCPiKiJBHpERE0k0CMiaiKBHhFREwn0iIiaSKBHRNREAj0ioiYS6BERNdFRoEs6XNIiSYslzRjgumMkWdKUoSsxIiI6MWigSxoBnAe8HpgEHCdpUovrtgLeC/xiqIuMiIjBddJD3w9YbHuJ7aeAWcARLa77BHAO8OQQ1hcRER3qJNB3BO5tOO4r29aQtBcwzvb3BnojSadJmi9p/tKlS9e72IiIaK+TQFeLNq85KW0CfA744GBvZPsC21NsTxkzZkznVUZExKA6CfQ+YFzD8VjgvobjrYCXA9dJuhs4AJiTG6MREcOrk0CfB0yUNEHSSGAaMKf/pO1HbW9ne7zt8cCNwFTb8zdIxRER0dKggW57FTAduBq4E7jM9gJJZ0uauqELjIiIzmzayUW25wJzm9rObHPtq//ysiIiYn3lSdGIiJpIoEdE1EQCPSKiJhLoERE1kUCPiKiJBHpERE0k0CMiaiKBHhFREwn0iIiaSKBHRNREAj0ioiYS6BERNZFAj4ioiQR6RERNJNAjImoigR4RURMJ9IiImugo0CUdLmmRpMWSZrQ4/w+Sbpd0q6TrJU0a+lIjImIggwa6pBHAecDrgUnAcS0C+5u2d7e9J3AO8NkhrzQiIgbUSQ99P2Cx7SW2nwJmAUc0XmD7sYbDLQAPXYkREdGJTjaJ3hG4t+G4D9i/+SJJ7wY+AIwEXjsk1UVERMc66aGrRds6PXDb59neBfgwcEbLN5JOkzRf0vylS5euX6URETGgTgK9DxjXcDwWuG+A62cBb251wvYFtqfYnjJmzJjOq4yIiEF1EujzgImSJkgaCUwD5jReIGliw+HfAr8duhIjIqITg46h214laTpwNTACuNj2AklnA/NtzwGmSzoUWAk8DLxjQxYdERHr6uSmKLbnAnOb2s5seH36ENcVERHrKU+KRkTURAI9IqImEugRETWRQI+IqIkEekRETSTQIyJqIoEeEVETCfSIiJpIoEdE1EQCPSKiJhLoERE1kUCPiKiJBHpERE0k0CMiaiKBHhFREwn0iIiaSKBHRNRER4Eu6XBJiyQtljSjxfkPSFoo6TZJP5K089CXGhERAxk00CWNAM4DXg9MAo6TNKnpsluAKbb3AK4AzhnqQiMiYmCd9ND3AxbbXmL7KWAWcETjBbavtb28PLwRGDu0ZUZExGA6CfQdgXsbjvvKtnbeCVz1lxQVERHrb9MOrlGLNre8UHobMAV4VZvzpwGnAey0004dlhgREZ3opIfeB4xrOB4L3Nd8kaRDgY8CU22vaPVGti+wPcX2lDFjxjybeiMioo1OAn0eMFHSBEkjgWnAnMYLJO0FnE8R5n8c+jIjImIwgwa67VXAdOBq4E7gMtsLJJ0taWp52aeBLYHLJd0qaU6bt4uIiA2kkzF0bM8F5ja1ndnw+tAhrisiItZTnhSNiKiJBHpERE0k0CMiaiKBHhFREwn0iIiaSKBHRNREAj0ioiYS6BERNZFAj4ioiQR6RERNJNAjImoigR4RURMJ9IiImkigR0TURAI9IqImEugRETWRQI+IqImOAl3S4ZIWSVosaUaL8wdLulnSKknHDH2ZERExmEEDXdII4Dzg9cAk4DhJk5ou+z1wIvDNoS4wIiI608meovsBi20vAZA0CzgCWNh/ge27y3OrN0CNERHRgU6GXHYE7m047ivbIiKii3QS6GrR5mfzh0k6TdJ8SfOXLl36bN4iIiLa6CTQ+4BxDcdjgfuezR9m+wLbU2xPGTNmzLN5i4iIaKOTQJ8HTJQ0QdJIYBowZ8OWFRER62vQQLe9CpgOXA3cCVxme4GksyVNBZC0r6Q+4O+A8yUt2JBFR0TEujqZ5YLtucDcprYzG17PoxiKiYiIiuRJ0YiImkigR0TURAI9IqImEugRETWRQI+IqIkEekRETSTQIyJqIoEeEVETCfSIiJpIoEdE1EQCPSKiJhLoERE1kUCPiKiJBHpERE0k0CMiaiKBHhFREwn0iIiaSKBHRNRER4Eu6XBJiyQtljSjxfnnSPpWef4XksYPdaERETGwQQNd0gjgPOD1wCTgOEmTmi57J/Cw7V2BzwH/MtSFRkTEwDrpoe8HLLa9xPZTwCzgiKZrjgC+Vr6+AjhEkoauzIiIGIxsD3yBdAxwuO1TyuO3A/vbnt5wzR3lNX3l8V3lNQ82vddpwGnl4YuBRUP1H/IX2A54cNCrekM+i0I+h2fks3hGt3wWO9se0+rEph385lY97ebvAp1cg+0LgAs6+DOHjaT5tqdUXUc3yGdRyOfwjHwWz9gYPotOhlz6gHENx2OB+9pdI2lTYBvgoaEoMCIiOtNJoM8DJkqaIGkkMA2Y03TNHOAd5etjgGs82FhOREQMqUGHXGyvkjQduBoYAVxse4Gks4H5tucAFwHfkLSYomc+bUMWPcS6agioYvksCvkcnpHP4hld/1kMelM0IiI2DnlSNCKiJhLoERE1kUCPiKiJBHpERAuSRkh6f9V1rI+euikq6XXAVravaGo/Hvij7f+qprLhJ+nMAU7b9ieGrZgKSdoX2M72VU3tU4E/2L6pmsqqI2k34EPAzjTMhLP92sqKqoik62y/uuo6OtVrgX4j8CbbS5va/wq40vYrqqls+En6YIvmzYFTgNG2txzmkioh6TrgRNt3N7XvClzQoyH2K+DLwE3A0/3tPfrN7ZMUD0p+C/hTf7vtmysragC9Fui32d5jfc/VnaStgNMpVs28DPhX23+stqrhIel227u3Ofcr25OHu6aqSbrJ9j5V19ENJF3botnd+o2+k7Vc6mSUpE1tr2pslLQZ8NyKaqqMpG2BDwDHU6yWubfth6utatgN9P99i2Grort8V9K7gCuBFf2NtntuOQ/br6m6hvXRazdFvw1cKGnNP9Ty9ZfLcz1D0qcplnV4HNjd9lk9GOYA/y3pk83LPUv6OHBNRTVV7R0UY+g/oxh2uQmYX2lFFZH0QkkXSbqqPJ4k6Z1V19VOrw25bAr8P4px4nvK5p0oli74mO2VVdU23CStpuh9rWLtlTFF8SPl1pUUNszKb+gXAfsCt5bNkykC7BTbT1RVW1SvDPKvAh+1PbnMkFvaDdNVrdcCfTPbKyU9F9i1bF5s+89V1hXVk/Qi4GXl4QLbS6qsp0rlEOQ/AgeXTdcB5/dSh6efpHm295V0i+29yrZbbe9ZdW2t9NoY+h8kfQf4JnBdVoQMSQuB/wC+Zfu7VdfTJb4EbAZ8sTx+e9l2SmUVVedPkkZT/hQr6QDg0WpLaq/XeuijKZb3nQZMpNgub6btX1RaWAUkPU7xl7Rx7NgU3+RH2u6Jb/aSJlP8fTiWYjeamcBltpvX/O8ZrWb39PCMn72BLwAvB+4AxgDH2L6t0sLa6KlAbyRpB+DvKP4xvwCYZfuj1VZVnXLq4ruAv6eYk99qnnqtlb2vtwBHA4spvtlfWG1Vw0/SzcDf2b6rPH4RcIXtvautrBrluPmLKTo/i7p56KlnAx1A0pbAURRT97a3/cKKSxp2kp4HvA84gWIo6nO2l1VbVbUkvRr4HDDJ9nMqLmfYSTqE4kbgEooQ2xk4yXarOdm1JOm1tq+RdFSr87a7clZcT/xY3UjSKOBNwHHAgcAPgI8AP6yyruEmaTvggxQ90ouBvWx37djghlYuAXAcRe/8borNDC6vsqaq2P6RpIk80yv9te0Vg/y2ujmYYtrqm1qcM106zbmnAl3SN4FDgZ9Q9EbfavvJaquqzD3AUoqe2HLgnY1TsW1/tqK6hpWk/08xfv4IMAs40HZftVVVY4Be6S6SurZXuoH0P5Nxke3rK61kPfRUoFNso/f3th+vupAu8GmemX++VdO5XhqH2xM42fZPACSdIOloim94Z/XY05GvYiPslW4gJwHnAp8HNpp7Bz01hi7pTcBttu8pj8+k+BH7HuB027+rsr7hJGlsu56opDf1yhS+8gbgobYfknQwRS/9PRRB/1Lbx1RaYFRC0kzgFRSzWu5qPEXx4F1XrvvUa4F+G3CA7eWS3gh8lmLcdC+Ku/qvq7TAYSRpEfC6FqsMngScYXuXSgobZo0PiUg6D1hq+6zmc71E0ukUQ3GPAxdS9FBn2O61+0x/RfFT/dTmc/2dwm7Ta2u52Pby8vVRFONjN9n+CsV34l7yfuC/yptfAEj6CMWMn1dVVtXw27SclgZwCGuv39JrQ5L9Trb9GPA3FFN6TwI+VW1Jw8/2/bYn276n+avq2trptb+wKqcqLqf4x/vFhnOjqimpGrbnSloBXCXpzRRPAe4LHNxji3TNBH4s6UHgz8BPYc166L0666f/7vgbgK/a/lXz4mV1J+ky28dKup3Wax1lyKVqkk4G/gl4jGKHosPL9r2Az9g+pMr6qiDpIGA2xcp6x/birJ/ygaLtgR/a/lPZthuwZbduZLAhSfoqsCMwgWKhshEUS2X0zBrpkra3/T+Sdm51vlt76T0V6ACSxlH8Rb3e9uqybXtgM9u/r7S4YdT06P9zgJUUu9P01GqLsS5Jm1DcFF5i+5Fy3fyx3fq4+4ZUrsb5Z9ury2/yLwGu6tanRXsu0CE7skQMRNKBwK22/yTpbRQ3Rc/t1l7phiTpJuCvgecDN1Isq7zc9vGVFtZGr90U7Xdj+WRgRKzrS8DycuGy/0Mxrffr1ZZUGZUTKY4CvmD7SGBSxTW11auB/hrg55LuknSbpNvLKY0RAavKpaWPoOiZn8u6D5/1Ckl6BcU2jd8v27p2MknXFraBvb7qAiK62OPlFNa3A38taQTF+ui96H0Uaz1daXtBufJk1y5S1pNj6P0kvYCG6Yq9dFM0op3ygZq3AvNs/1TSTsCrbffqsAuw5mbxluUc/a7Uk0MukqZK+i3wO+DHFKvrXVVpURFdwvb9wH9SzH6CYuOPK6urqDqSvilp63K2y0JgkaQPVV1XOz0Z6MAngAOA39ieQPGQ0Q3VlhTRHSSdSrGb1/ll044Uzyr0okllj/zNwFyKTeXfXm1J7fVqoK8sN3HYRNIm5cL9PbdmR0Qb76bYK+AxANu/pVgCoBdtVm6a/WbgO+X8864dp+7Vm6KPlEsA/BS4VNIfgVUV1xTRLVbYfqr/af9yrZuuDbEN7HyKIdlfAT8pnxzt2jH0nrwp2v/0F8VPKMcD2wCX9vrWaxEAks6h2PDjBIqlhN8FLOzlPXcbSdrUdld2AHsy0AHK77QTbf+3pM2BEdn4ImLNbI53Uqy2KIolZL/iHg0LSX8LvIy1Z8SdXV1F7fXkkEt50+c0YFtgF4qbPl+muDka0bPKOedfs/02irXQe5qkLwObUzyM+BXgGOCXlRY1gF69KZqbPhEt2H4aGCNpZNW1dIlX2j4BeNj2xyl2MRpXcU1t9WQPndz0iRjI3cANkuYAf+pv7JWNw5v8ufx1uaQdgGUUq7V2pV4N9B9L+ifguZIOo7jp0xN7aEZ04L7yaxN6dw2Xft+T9DyKTdVvpuj4faXaktrryZuiuekTEetL0nOAUba7dierngz0iGhP0ndZdwjyUYq1wM/vhV2tJB010Hnb3x6uWtZHTw65lAv4nwXsTPEZ9O/S86Iq64roEksoNk2fWR6/BXgA2I1i5kvXPvo+hN40wDkDXRnoPdlDl/Rril3vb6LYdg2APFgUAZJ+YvvgVm2SFth+WVW1xcB6socOPGo7qytGtDZG0k79y0mXy+duV557qrqyho+kD1DkxEVN7e+heAjx36qpbGC92kP/FMVO5t8GVvS39+IO7xHNJL2B4kG7uyiGIydQzAS7Dji1W8NsKEm6A9jb9lNN7c+hWCd+j2oqG1ivBnqrHUds+7XDXkxEFyqD6yUUgf7rXrgR2kjS7bZ3X99zVevJIRfbr6m6hohuVa5t9AFgZ9unSpoo6cW2v1d1bcNJ0gttP9DcVlU9neipQJf0Ntv/UY6PraNHn4SLaPZVigkDryiP+4DLgV4K9E8D35f0QYoHigD2Ac4BPlNZVYPoqUAHtih/bfX0W++NPUW0tovtt0g6DsD2n9W/TkaPsP11SUuBs4GXU+TDAuD/dvOEip4KdNvnl79+vPmcpPcNf0URXekpSc+l7ORI2oWGyQO9wvZVkh63fX1ju6QDbXfllpU9eVO0FUm/t71T1XVEVK1c3+gMYBLwQ4qVSU+0fV2VdVVB0s229x6srVv0VA99ED31I2VEO7b/S9LNFBupCzjd9oMVlzWsJL0CeCXFnPzGe25bU0x57kq9uh56K/lRJaJke5nt75czW0ZL6rXNLkYCW1J0erdq+HqMYpOLrtRTQy6SHqd1cAt4ru38xBI9S9IeFDM4dgBmA18AvgjsD/yr7c9VWF4lJO1s+57y9SbAlra7dpPonuqh297K9tYtvrZKmEdwIfBN4GhgKcV0vSXArr0Y5qV/lrR1ubH8QmCRpA9VXVQ7PdVDj4j2JN1qe8+G43uB8eW2dD2p/zORdDzFPPQPAzd166P/6ZVGRL9RkvbimQkCTwB79M9B79G1jjaTtBnwZuDfba+U1LW94AR6RPT7H6Dxaen7G44N9OJaR+dT7LH6K+Anknam3Fy+G2XIJSJiPUja1PaqqutopaduikbE4CS9u9wYuf/4+ZLeVWVNVZH0QkkXSbqqPJ4EvKPistpKoEdEs1NtP9J/YPth4NQK66nSJRSbyO9QHv8G6NplQhLoEdFsk8bFuCSNoHjQphdtZ/syYDVAOdTStbN+clM0IppdDVwm6csUN0P/AfhBtSVV5k+SRvPMQmUHAI9WW1J7uSkaEWspn4j8e+AQiimMPwS+0ovz0SXtTfHE7MuBO4AxwDG2b6u0sDYS6BERLZTf2A4Afgm8mOKb2yLbKystbAAJ9IgAQNJlto+VdDst1jzq1qcjNyRJP7f9isGv7A4J9IgAQNL2tv+nfHhmHf2LVPUSSR8HbgO+7Y0gLBPoEbEWSf9i+8ODtfWCcoXWLShmtvyZYtjFtreutLA2EugRsZY2u/Tc1otDLhubTFuMCAAk/SPwLmAXSY2zOLYCunIPzeEgaSpwcHl4XbnpR1dKDz0iAJC0DfB84J+BGQ2nHrf9UDVVVUvSp4B9gUvLpuMols+d0f53VSeBHhFrkbQL0Gd7haRXA3sAX29cDqBXlD+p7Gl7dXk8ArilW4ef8uh/RDT7T+BpSbsCFwETKHYy6lXPa3i9TWVVdCBj6BHRbLXtVZKOAv7N9hck3VJ1URX5Z+AWSddSzHA5GPhItSW1l9j1PosAAAR5SURBVECPiGYrJR0HnAC8qWzbrMJ6KmN7pqTrKMbRBXzY9v3VVtVehlwiotlJwCuAT9r+naQJwH9UXNOwkjS94XBb23Nsf6ebwxxyUzQiYh2Nc/FbzcvvVhlyiQgga7kMQINf0h0S6BHR7/Ty1zdWWkV3eJ6kIymGpbcubxCvYfvb1ZQ1sAy5REQ0kfTVAU7b9snDVsx6SKBHxFrKBamag+FRYD7wQdtLhr+q6ESGXCKi2WeB+ygeJhIwDfgrYBFwMfDqyiobZpKeRzF9czwNeWn7vVXVNJD00CNiLZJ+YXv/prYbbR8g6Ve2J1dV23CT9DPgRuB2yo2iAWx/rbKiBpAeekQ0Wy3pWOCK8viYhnO91gMcZfsDVRfRqfTQI2Itkl4EnEvxcBHAz4H3A38A9rF9fVW1DTdJ7weeAL4HrOhv79bVJxPoERFtSHo38EngEZ756cS2X1RdVe0l0CNiLZLGAl8ADqQIseuB0233VVpYBSTdBexv+8Gqa+lE1nKJiGZfBeYAOwA7At8t23rRAmB51UV0Kj30iFiLpFtt7zlYWy+QdCXwMuBa1h5D78ppi5nlEhHNHpT0NmBmeXwcsKzCeqo0u/zaKKSHHhFrkbQT8O8Us1wM/Ax4r+3fV1pYRSSNBHYrDxfZXlllPQNJoEfEoCS9z/a/VV3HcCv3VP0acDfFU7PjgHfY/kmFZbWVQI+IQUn6ve2dqq5juEm6CXir7UXl8W7ATNv7VFtZa5nlEhGd2GjWBB9im/WHOYDt39DF2/HlpmhEdKJXf5SfL+ki4Bvl8fHATRXWM6AMuUQE0HbZXCh658+13XMdQEnPAd4NHETxOfwE+KLtFQP+xook0CMiaqLnvuNGRAym3b6q/bp1f9UEekTEuvr3VX13+WvjGHrXLgWQIZeIiDYk3WD7wMHaukWmLUZEtLeFpIP6DyS9EtiiwnoGlCGXiIj23glcLGmb8vgR4OQK6xlQhlwiIgYhaWuKvHy06loGkkCPiGijnId+NDCehhEN22dXVdNAMuQSEdHed4BHKZ4O7cqHiRqlhx4R0YakO2y/vOo6OpVZLhER7f1M0u5VF9Gp9NAjItqQtBDYFfgdxZCLAHfrk6IJ9IiINiTt3Krd9j3DXUsnclM0IqKN/uCW9AJgVMXlDCpj6BERbUiaKum3FEMuP6bYiu6qSosaQAI9IqK9TwAHAL+xPQE4BLih2pLaS6BHRLS30vYyYBNJm9i+Ftiz6qLayRh6RER7j0jakmKnoksl/RFYVXFNbWWWS0REG5K2AP5MMZpxPLANcGnZa+86CfSIiA5JGgFMs31p1bW0kjH0iIgmkraW9BFJ/y7pb1SYDiwBjq26vnbSQ4+IaCLpO8DDwM8pZrY8HxgJnG771iprG0gCPSKiiaTbbe9evh4BPAjsZPvxaisbWIZcIiLWtbL/he2ngd91e5hDeugREeuQ9DTwp/5D4LnAcp5ZnGvrqmobSAI9IqImMuQSEVETCfSIiJpIoEdE1EQCPSKiJhLoERE18b8yLw6ItMWQegAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a pandas dataframe with the data as the values of the results dictionary,\n",
    "# the index as the keys of the results dictionary and a single column called accuracy.\n",
    "# Be sure to save the dataframe to a variable.\n",
    "results_df = pd.DataFrame(results.values(), \n",
    "                          results.keys(), \n",
    "                          columns=[\"Accuracy\"])\n",
    "\n",
    "# Create a bar plot of the results dataframe using plot.bar()\n",
    "results_df.plot.bar();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `np.random.seed(42)` results in the `LogisticRegression` model perfoming the best (at least on my computer).\n",
    "\n",
    "Let's tune its hyperparameters and see if we can improve it.\n",
    "\n",
    "### Hyperparameter Tuning\n",
    "\n",
    "Remember, if you're ever trying to tune a machine learning models hyperparameters and you're not sure where to start, you can always search something like \"MODEL_NAME hyperparameter tuning\".\n",
    "\n",
    "In the case of LogisticRegression, you might come across articles, such as [Hyperparameter Tuning Using Grid Search by Chris Albon](https://chrisalbon.com/machine_learning/model_selection/hyperparameter_tuning_using_grid_search/).\n",
    "\n",
    "The article uses [`GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) but we're going to be using [`RandomizedSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html).\n",
    "\n",
    "The different hyperparameters to search over have been setup for you in `log_reg_grid` but feel free to change them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Different LogisticRegression hyperparameters\n",
    "log_reg_grid = {\"C\": np.logspace(-4, 4, 20),\n",
    "                \"solver\": [\"liblinear\"]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we've got a set of hyperparameters we can import `RandomizedSearchCV`, pass it our dictionary of hyperparameters and let it search for the best combination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 5 candidates, totalling 25 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done  25 out of  25 | elapsed:    0.3s finished\n"
     ]
    }
   ],
   "source": [
    "# Setup np random seed of 42\n",
    "np.random.seed(42)\n",
    "\n",
    "# Import RandomizedSearchCV from sklearn's model_selection module\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "# Setup an instance of RandomizedSearchCV with a LogisticRegression() estimator,\n",
    "# our log_reg_grid as the param_distributions, a cv of 5 and n_iter of 5.\n",
    "rs_log_reg = RandomizedSearchCV(estimator=LogisticRegression(),\n",
    "                                param_distributions=log_reg_grid,\n",
    "                                cv=5,\n",
    "                                n_iter=5,\n",
    "                                verbose=True)\n",
    "\n",
    "# Fit the instance of RandomizedSearchCV\n",
    "rs_log_reg.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once `RandomizedSearchCV` has finished, we can find the best hyperparmeters it found using the `best_params_` attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'solver': 'liblinear', 'C': 0.23357214690901212}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the best parameters of the RandomizedSearchCV instance using the best_params_ attribute\n",
    "rs_log_reg.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7894736842105263"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Score the instance of RandomizedSearchCV using the test data\n",
    "rs_log_reg.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After hyperparameter tuning, did the models score improve? What else could you try to improve it? Are there any other methods of hyperparameter tuning you can find for `LogisticRegression`?\n",
    "\n",
    "### Classifier Model Evaluation\n",
    "\n",
    "We've tried to find the best hyperparameters on our model using `RandomizedSearchCV` and so far we've only been evaluating our model using the `score()` function which returns accuracy. \n",
    "\n",
    "But when it comes to classification, you'll likely want to use a few more evaluation metrics, including:\n",
    "* [**Confusion matrix**](https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/) - Compares the predicted values with the true values in a tabular way, if 100% correct, all values in the matrix will be top left to bottom right (diagnol line).\n",
    "* [**Cross-validation**](https://scikit-learn.org/stable/modules/cross_validation.html) - Splits your dataset into multiple parts and train and tests your model on each part and evaluates performance as an average. \n",
    "* [**Precision**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score) - Proportion of true positives over total number of samples. Higher precision leads to less false positives.\n",
    "* [**Recall**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score) - Proportion of true positives over total number of true positives and false positives. Higher recall leads to less false negatives.\n",
    "* [**F1 score**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score) - Combines precision and recall into one metric. 1 is best, 0 is worst.\n",
    "* [**Classification report**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) - Sklearn has a built-in function called `classification_report()` which returns some of the main classification metrics such as precision, recall and f1-score.\n",
    "* [**ROC Curve**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_score.html) - [Receiver Operating Characterisitc](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) is a plot of true positive rate versus false positive rate.\n",
    "* [**Area Under Curve (AUC)**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html) - The area underneath the ROC curve. A perfect model achieves a score of 1.0.\n",
    "\n",
    "Before we get to these, we'll instantiate a new instance of our model using the best hyerparameters found by `RandomizedSearchCV`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a LogisticRegression classifier using the best hyperparameters from RandomizedSearchCV\n",
    "clf = LogisticRegression(solver=\"liblinear\", C=0.23357214690901212)\n",
    "\n",
    "# Fit the new instance of LogisticRegression with the best hyperparameters on the training data \n",
    "clf.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's to import the relative Scikit-Learn methods for each of the classification evaluation metrics we're after."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import confusion_matrix and classification_report from sklearn's metrics module\n",
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "\n",
    "# Import precision_score, recall_score and f1_score from sklearn's metrics module\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score\n",
    "\n",
    "# Import plot_roc_curve from sklearn's metrics module\n",
    "from sklearn.metrics import plot_roc_curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluation metrics are very often comparing a model's predictions to some ground truth labels.\n",
    "\n",
    "Let's make some predictions on the test data using our latest model and save them to `y_preds`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions on test data and save them\n",
    "y_preds = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to use the predictions our model has made to evaluate it beyond accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[24, 11],\n",
       "       [ 5, 36]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a confusion matrix using the confusion_matrix function\n",
    "confusion_matrix(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge:** The in-built `confusion_matrix` function in Scikit-Learn produces something not too visual, how could you make your confusion matrix more visual?\n",
    "\n",
    "You might want to search something like \"how to plot a confusion matrix\". Note: There may be more than one way to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANQAAADQCAYAAAB/T08OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAOcUlEQVR4nO3deZBV5Z3G8e/TjcgmCIqIKIqGhFEHE4MCBpUYGZ2JRh2jhqiQcUFRxjjlOjWSQWMSK1rOlI6iGDUuUdRJGfeVqNFUBDS44K4oIO4sESwEuvnNH/c203Y1tw/Ne/r2aZ5PVVffc053nweqnnrPfe9ZFBGYWRo11Q5g1pG4UGYJuVBmCblQZgm5UGYJuVBmCXWqdoD1+XTM/p7PT2T8G92rHaHDeXDBg2puvUcos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXKiMavr2pdel/03v62+m93W/pesRR35le9cfHkPfx55CPXtVKWHxnHnpmdz219u4+rGr160b9f1RTH18Kve/dz+Dhw6uYrrWcaGyqq/ni2uvYumJ41h2xkS6/OAIagfuCJTK1vnbw6j/+KMqhyyWx+96nMnjJn9l3fw35nPxhIuZO3NulVJtnNyeAi9pCHAYMAAI4APg3oh4La995mntkiWsXbIEgFi5kvoF86nZui/1C+bT49RJrLjuGnpd+IsqpyyWubPmss3223xl3cK3F1YpTRq5jFCSzgOmAwJmAbPLr2+XdH4e+2xLNf22pdPXBlP3+qt0HrkP9Ys/o37eO9WOZe1AXiPUicBuEbGm8UpJlwOvAJc090uSJgATAC4bMphx2/fPKd5G6NKVnj+7iBVTryTq6+k29nj+dv7Z1U5l7URe76HWAts1s75/eVuzImJaRAyLiGHtsky1tfT6z4tY9cfHWf3M09T2H0Dttv3pfe319LllOjV9+9J76nWod59qJ7UqyWuEOhOYIektoOGgeCDwNWBSTvvM3RZnnUfdgvms/P2dANS/N4/FRx++bnufW6az9PRTiM//Vq2IVmW5FCoiHpb0dWBvSpMSAt4HZkdEfR77zFun3f6eLmMOom7eO3S+5jcAfHHDdayeNbPKyYrr3CvPZejIofTs3ZObZ97MrZffyvJly5l40UR69enFlBunMO/VeUw+fnLLf6ydUERUO0OzPh2zf/sMVkDj3+he7QgdzoMLHlRz69c7Qkmq+EYgIpZsbCizjqbSId/zlD4/aq6JAeycSyKzAltvoSJiUFsGMesIWpw2V8lxkiaXlwdK2jv/aGbFk+VzqKuBkcCPy8vLgatyS2RWYFmmzYdHxJ6S5gBExFJJnXPOZVZIWUaoNZJqKU1EIKkvFc52MNuUZSnUFcDdQD9JvwCeAX6ZayqzgmrxkC8ififpeeB75VWHF/USDLO8ZT31qBvQcNjXNb84ZsWWZdr8Z8BNQB9ga+BGSRfkHcysiLKMUGOBb0XElwCSLgH+ClycZzCzIsoyKfEe0KXR8uaAL081a0alk2OvpPSeaRXwiqTHystjKM30mVkTlQ75nit/f57StHmDJ3NLY1ZwlU6Ovaktg5h1BC1OSkgaDPwK2JVG76UiwpdvmDWRZVLiRmAqUAd8F7gZuCXPUGZFlaVQXSNiBqXL5edHxBTggHxjmRVTls+hvpRUA7wlaRKwCNimhd8x2yRlGaHOpHTq0RnAt4HjgfF5hjIrqiwnx84uv1wB/Eu+ccyKrdIHu/dRvgaqORHxg1wSmRVYpRHqsjZLYdZBVPpg96m2DGLWEfiBa2YJuVBmCblQZgl5ls8soSyzfP8MbAvcWl4eS+miQzNrosVZPkk/j4j9Gm26T9Kfck9mVkBZ3kP1lbTuUg1Jg4C++UUyK64sJ8f+G/CkpHnl5Z2AU3JLZFZgWc7le7h8keGQ8qrXI2JVvrHMiinLffm6AecAkyLiRWCgpENyT2ZWQFmv2F1N6ZE2UHr4tO/JZ9aMLO+hdomIYySNBYiIlZKafWBvSv2fejvvXWwyVn7wdLUjbDKyjFCrJXXl/x9nswule/WZWRNZRqgpwMPADpJ+B3wHX2ho1qwss3yPlh9nM4LSE+F/GhGf5Z7MrICyzPLNiIjFEfFARNwfEZ9JmtEW4cyKptLJsV0o3Zxla0m9KY1OAD2B7dogm1nhVDrkO4XSHY+2o3R/84ZCfY6fAm/WLEWs9wqN0g9I/xoRV7ZRnnU6dR5QOZhl5mnz9DbbeudmPzrKMm2+VtKWDQuSeks6LVkysw4kS6FOjohlDQsRsRQ4Ob9IZsWVpVA1jc+MkFQLdM4vkllxZflg9xHgTknXUDpb4lRKH/SaWRNZCnUepRm/iZRm+h4FfpNnKLOianGWr1o8y5eOZ/nSW98sX6UPdu+MiKMlvUwzdz+KiKEJ85l1CJUO+X5a/u6LCc0yqnTXow/L3+e3XRyzYqt0yLecyje67JlLIrMCqzRCbQEg6SLgI0oPqhZwLLBFm6QzK5gsH+weFBFXR8TyiPg8IqYCR+YdzKyIshSqXtKxkmol1Ug6FqjPO5hZEWUp1I+Bo4GPy19HldeZWRNZLoF/Dzgs/yhmxZflEvivS5ohaW55eaikC/KPZlY8WQ75rgP+HVgDEBEvAT/KM5RZUWUpVLeImNVkXV0eYcyKLkuhPivf3LLhRpc/BD7MNZVZQWW5fON0YBowRNIi4F1KH+6aWRMVCyWpBhgWEQdK6g7URMTytolmVjwVD/kiYi0wqfz6C5fJrLIs76Eek3S2pB0k9Wn4yj2ZWQFleQ91Qvn76Y3WBbBzMz9rtknLcqbEoLYIYtYRtFio8j3OTwNGURqZngauiYgvc85mVjhZDvluBpYDDbdjHkvp2qij8gplVlRZCvWNiNij0fITkl7MK5BZkWWZ5ZsjaUTDgqThwJ/zi2RWXFlGqOHAOEkLyssDgdcabi+2Kd9O7O03n2X5ihXU16+lrq6OESP/qdqRCmPVqtWMP/0cVq9ZQ31dPWO+O4pJJx1PRHDFtJt49IlnqKmp4Zgjvs9xRxXn6qEshTo49xQFduCYo1i8eGm1YxRO586bccMVl9CtW1fW1NUxbuLZ7DtiGPPmL+SjTz7jvtumUVNTw+Kly1r+Y+1Ilmlz30bMkpNEt25dAairq6Ourg5J3HH3A/x6ynnU1JTejWzVe8tKf6bdyfIeKilJHeYJ8hHBQw/ezsxnH+KkE32+8Iaqr6/nyPGns98hYxm517cYutsQFi76kIdmPMXRJ5zBqWdNZv7CRdWOuUHavFDAhevbIGmCpOckPbd27RdtmalV9ht9OHsPP5hDDj2OiRN/wr6jhlc7UqHU1tby+5uuYsbdt/Dyq2/y1rz3WL1mDZt37sydN1zBkYcezORf/le1Y26QXAol6aX1fL0M9Fvf70XEtIgYFhHDamq65xEtqQ8//BiATz9dzD33PMRee32zyomKqecWPdhrz6E88+xzbNt3a8aMHgXAgfvvw5vvvFvldBsmrxGqHzAOOLSZr8U57bNNdevWlR49uq97PebA/XnllTeqnKo4lixdxufLVwDw5apVPDt7DoN23IED9hvJzOdfAGD2nJfZcYcB1Yy5wbLM8rXG/UCPiHih6QZJT+a0zzbVr19f/veu6wHo1KmW6dP/wCOPPlndUAXy6eKl/MfFl1G/di2xNjjogH0Z/Z3h7Dl0N8678Nfccscf6Na1Cxeef2a1o24QPx9qE+DnQ6W3MU+BN7OMXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4RcKLOEXCizhFwos4Ta7UOri0LShIiYVu0cHUXR/z89Qm28CdUO0MEU+v/ThTJLyIUyS8iF2niFPd5vpwr9/+lJCbOEPEKZJeRCbQRJB0t6Q9Lbks6vdp4ik3SDpE8kza12lo3hQrWSpFrgKuAfgV2BsZJ2rW6qQvstcHC1Q2wsF6r19gbejoh5EbEamA4cVuVMhRURfwKWVDvHxnKhWm8AsLDR8vvldbYJc6FaT82s85TpJs6Far33gR0aLW8PfFClLNZOuFCtNxsYLGmQpM7Aj4B7q5zJqsyFaqWIqAMmAY8ArwF3RsQr1U1VXJJuB/4CfEPS+5JOrHam1vCZEmYJeYQyS8iFMkvIhTJLyIUyS8iFMkvIhWonJG0p6bQc//5PJP1PCz8zRdLZG/h3V2xcso7FhWo/tgSaLVT5zHYrABeq/bgE2EXSC5IulTRa0hOSbgNelrRT42uFJJ0taUr59S6SHpb0vKSnJQ2ptCNJh0qaKWmOpMcl9Wu0eQ9Jf5T0lqSTG/3OOZJmS3pJ0oVp/+kdR6dqB7B1zgd2j4hvAkgaTekSkd0j4l1JO1X43WnAqRHxlqThwNXAARV+/hlgRESEpJOAc4GzytuGAiOA7sAcSQ8AuwODy3kE3Ctpv/IlF9aIC9W+zYqIdyv9gKQewD7AXdK6E+A3b+Hvbg/cIak/0BlovI97ImIlsFLSE5RKNAr4B2BO+Wd6UCqYC9WEC9W+fdHodR1fPUTvUv5eAyxrGNkyuhK4PCLuLY+EUxpta3ouWlAalX4VEdduwD42SX4P1X4sB7aosP1jYBtJW0naHDgEICI+B96VdBSASvZoYV+9gEXl1+ObbDtMUhdJWwGjKZ1V/whwQnk0RNIASdtk/6dtOjxCtRMRsVjSn8sTDw8BDzTZvkbSRcBMSodorzfafCwwVdIFwGaULsd/scLuplA6RFwEPAsMarRtVnnfA4GfR8QHwAeS/g74S/mwcgVwHPBJK/+5HZbPNjdLyId8Zgm5UGYJuVBmCblQZgm5UGYJuVBmCblQZgm5UGYJ/R+LJa+d/4LICwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import seaborn for improving visualisation of confusion matrix\n",
    "import seaborn as sns\n",
    "\n",
    "# Make confusion matrix more visual\n",
    "def plot_conf_mat(y_test, y_preds):\n",
    "    \"\"\"\n",
    "    Plots a confusion matrix using Seaborn's heatmap().\n",
    "    \"\"\"\n",
    "    fig, ax = plt.subplots(figsize=(3, 3))\n",
    "    ax = sns.heatmap(confusion_matrix(y_test, y_preds),\n",
    "                     annot=True, # Annotate the boxes\n",
    "                     cbar=False)\n",
    "    plt.xlabel(\"Predicted label\")\n",
    "    plt.ylabel(\"True label\")\n",
    "    \n",
    "    # Fix the broken annotations (this happened in Matplotlib 3.1.1)\n",
    "    bottom, top = ax.get_ylim()\n",
    "    ax.set_ylim(bottom + 0.5, top - 0.5);\n",
    "    \n",
    "plot_conf_mat(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about a classification report?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.83      0.69      0.75        35\n",
      "           1       0.77      0.88      0.82        41\n",
      "\n",
      "    accuracy                           0.79        76\n",
      "   macro avg       0.80      0.78      0.78        76\n",
      "weighted avg       0.79      0.79      0.79        76\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# classification report\n",
    "print(classification_report(y_test, y_preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge:** Write down what each of the columns in this classification report are.\n",
    "\n",
    "* **Precision** - Indicates the proportion of positive identifications (model predicted class 1) which were actually correct. A model which produces no false positives has a precision of 1.0.\n",
    "* **Recall** - Indicates the proportion of actual positives which were correctly classified. A model which produces no false negatives has a recall of 1.0.\n",
    "* **F1 score** - A combination of precision and recall. A perfect model achieves an F1 score of 1.0.\n",
    "* **Support** - The number of samples each metric was calculated on.\n",
    "* **Accuracy** - The accuracy of the model in decimal form. Perfect accuracy is equal to 1.0.\n",
    "* **Macro avg** - Short for macro average, the average precision, recall and F1 score between classes. Macro avg doesn’t class imbalance into effort, so if you do have class imbalances, pay attention to this metric.\n",
    "* **Weighted avg** - Short for weighted average, the weighted average precision, recall and F1 score between classes. Weighted means each metric is calculated with respect to how many samples there are in each class. This metric will favour the majority class (e.g. will give a high value when one class out performs another due to having more samples).\n",
    "\n",
    "The classification report gives us a range of values for precision, recall and F1 score, time to find these metrics using Scikit-Learn functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7659574468085106"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the precision score of the model using precision_score()\n",
    "precision_score(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8780487804878049"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the recall score\n",
    "recall_score(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8181818181818182"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the F1 score\n",
    "f1_score(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confusion matrix: done.\n",
    "Classification report: done.\n",
    "ROC (receiver operator characteristic) curve & AUC (area under curve) score: not done.\n",
    "\n",
    "Let's fix this.\n",
    "\n",
    "If you're unfamiliar with what a ROC curve, that's your first challenge, to read up on what one is.\n",
    "\n",
    "In a sentence, a [ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) is a plot of the true positive rate versus the false positive rate.\n",
    "\n",
    "And the AUC score is the area behind the ROC curve.\n",
    "\n",
    "Scikit-Learn provides a handy function for creating both of these called [`plot_roc_curve()`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_roc_curve.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3wV1bn/8c8DomABW7n4iyANIBYiQsTIAbEVDyLgBazFC+AFUUEqQmuptb+2yvFyqtVyWhW1WAXkBUgVFWqpVK2XFkUJEG4BESFqgJ+igHK0yO35/TGbdCfZ2dkhmb2TzPf9eu0Xe2bWnnkmIfPstdbMWubuiIhIdDXIdAAiIpJZSgQiIhGnRCAiEnFKBCIiEadEICIScUdkOoCqatmypWdnZ2c6DBGROmXZsmWfunurRNvqXCLIzs4mPz8/02GIiNQpZvZBRdvUNCQiEnFKBCIiEadEICIScUoEIiIRp0QgIhJxoSUCM3vCzD4xszUVbDcze8DMNprZKjPrEVYsIiJSsTBrBNOBgUm2DwI6xV6jgUdCjEVERCoQ2nME7v6GmWUnKTIEeNKDcbCXmNk3zSzL3beFFZNIVM1++0PmF2zJdBhSTTnHN+f2C0+u8f1mso+gDfBR3HJxbF05ZjbazPLNLH/79u1pCU6kPplfsIXCbV9kOgyppTL5ZLElWJdwlhx3nwpMBcjLy9NMOiKHISerOXPH9M50GFILZbJGUAycELfcFtiaoVhERCIrk4lgAXBV7O6hXsDn6h8QEUm/0JqGzGwO0BdoaWbFwO1AIwB3fxRYCJwHbAS+Aq4JKxaRTKhNHbSF274gJ6t5psOQWirMu4aGVbLdgRvDOr5Iph3qoK0NF+CcrOYMyU14L4ZI3RuGWqQuUQet1AUaYkJEJOKUCEREIk5NQ5IWtanjNF1qS/+ASGVUI5C0iOKTreqglbpCNQJJG3WcitROqhGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRNwRmQ5Aar/Zb3/I/IIt1dpH4bYvyMlqXkMRiUhNUo1AKjW/YAuF276o1j5yspozJLdNDUUkIjVJNQJJSU5Wc+aO6Z3pMEQkBKoRiIhEXKiJwMwGmtm7ZrbRzG5NsL2dmb1qZivMbJWZnRdmPCIiUl5oicDMGgJTgEFADjDMzHLKFPsl8Cd3PxW4HHg4rHhERCSxMGsEPYGN7r7J3fcCTwFDypRx4NCtJMcAW0OMR0REEggzEbQBPopbLo6tizcJuMLMioGFwE2JdmRmo80s38zyt2/fHkasIiKRFWYisATrvMzyMGC6u7cFzgNmmlm5mNx9qrvnuXteq1atQghVRCS6wkwExcAJccttKd/0cy3wJwB3fwtoDLQMMSYRESkjzESwFOhkZu3N7EiCzuAFZcp8CPQDMLMuBIlAbT8iImkUWiJw9/3AOGARsI7g7qC1ZnaHmQ2OFfsJcL2ZrQTmACPdvWzzkYiIhCjUJ4vdfSFBJ3D8utvi3hcCfcKMQUREktOTxSIiEadEICIScUoEIiIRp0QgIhJxSgQiIhGnRCAiEnGamCbiUpmGUtNMitRvqhFEXCrTUGqaSZH6TTUC0TSUIhGXUo3AzI40sxPDDkZERNKv0kRgZucDq4GXYsu5ZvZc2IGJiEh6pFIjuAP4D2AXgLsXAKodiIjUE6kkgn3uvqvMOo0QKiJST6TSWbzOzC4FGphZe2ACsCTcsEREJF1SqRGMA04DDgLPAnsIkoGIiNQDqdQIBrj7z4CfHVphZhcTJAUREanjUkkEv6T8Rf8XCdZJLaOnhkUkFRUmAjMbAAwE2pjZ5LhNzQmaiaSWO/TUcLILvZ4aFpFkNYJPgDUEfQJr49bvBm4NMyipOXpqWEQqU2EicPcVwAozm+Xue9IYk4iIpFEqfQRtzOxuIAdofGilu58UWlQiIpI2qSSC6cBdwP3AIOAa1EeQceoIFpGakspzBEe7+yIAd3/f3X8JnB1uWFIZDR8tIjUllRrB12ZmwPtmdgOwBWgdbliSCnUEi0hNSCUR/BhoCowH7gaOAUaFGZSIiKRPpYnA3d+Ovd0NXAlgZm3DDEpERNInaSIws9OBNsA/3f1TMzuZYKiJ/wSUDEJUWWewOoJFpKZU2FlsZr8GZgEjgBfN7BfAq8BKQLeOhqyyzmB1BItITUlWIxgCdHf3f5nZscDW2PK76QlN1BksIumQ7PbRPe7+LwB33wGsVxIQEal/ktUIOpjZoRFGDciOW8bdL65s52Y2EPg90BD4o7vfk6DMpcAkglnPVrr78NTDr5v0MJiI1CbJEsEPyiw/VJUdm1lDYArQHygGlprZAncvjCvTCfg50Mfdd5pZJJ5P0KigIlKbJBt07pVq7rsnsNHdNwGY2VME/Q6FcWWuB6a4+87YMT+p5jHrDLX/i0htkcoQE4erDfBR3HJxbF28k4CTzGyxmS2JNSWVY2ajzSzfzPK3b98eUrgiItEUZiKwBOu8zPIRQCegLzAM+KOZfbPch9ynunueu+e1atWqxgMVEYmyVIaYAMDMjnL3r6uw72LghLjltgS3oJYts8Td9wGbzexdgsSwtArHqVXUESwidU2lNQIz62lmq4H3YsvdzezBFPa9FOhkZu3N7EjgcmBBmTLPExvJ1MxaEjQVbapC/LWORgUVkbomlRrBA8AFBBdt3H2lmVU6DLW77zezccAigttHn3D3tWZ2B5Dv7gti2841s0LgAPBTd//sMM+l1lBHsIjUJakkggbu/kEwEnWJA6ns3N0XAgvLrLst7r0DN8deIiKSAakkgo/MrCfgsWcDbgI2hBuWiIikSyp3DY0l+MbeDvgY6BVbJyIi9UAqNYL97n556JGIiEhGpFIjWGpmC83sajNrFnpEIiKSVpUmAnfvCNwFnAasNrPnzUw1BBGReiKlJ4vd/U13Hw/0AL4gmLBGRETqgVQeKGtqZiPM7M/AO8B24IzQIxMRkbRIpbN4DfBn4Dfu/o+Q4xERkTRLJRF0cPeDoUciIiIZUWEiMLPfuvtPgHlmVnbU0JRmKBMRkdovWY1gbuzfKs1MJiIidUuyGcreib3t4u6lkkFsMLnqzmBWJ1U2zLSGmBaRuiaV20dHJVh3bU0HUldUNsy0hpgWkbomWR/BZQRzCLQ3s2fjNjUDdoUdWG2mYaZFpD5J1kfwDvAZwcxiU+LW7wZWhBmUiIikT7I+gs3AZuDl9IUjIiLplqxp6HV3P8vMdlJ60nkjmFPm2NCjExGR0CVrGjo0HWXLdAQiIiKZUeFdQ3FPE58ANHT3A0BvYAzwjTTEJiIiaZDK7aPPE0xT2RF4EugCzA41KhERSZtUEsFBd98HXAz8zt1vAnSjvIhIPZFKIthvZpcAVwIvxNY1Ci8kERFJp1SfLD6bYBjqTWbWHpgTblgiIpIulQ5D7e5rzGw8cKKZdQY2uvvd4YcmIiLpUGkiMLPvAjOBLQTPEPwfM7vS3ReHHZyIiIQvlYlp/gc4z90LAcysC0FiyAszMBERSY9U+giOPJQEANx9HXBkeCGJiEg6pVIjWG5mfyCoBQCMQIPOiYjUG6kkghuA8cAtBH0EbwAPhhmUiIikT9JEYGanAB2B59z9N+kJSURE0qnCPgIz+78Ew0uMAF4ys0QzlYmISB2XrLN4BNDN3S8BTgfGVnXnZjbQzN41s41mdmuSckPNzM1MdyKJiKRZskTwtbt/CeDu2yspW46ZNSSY2WwQkAMMM7OcBOWaEfRBvF2V/YuISM1I1kfQIW6uYgM6xs9d7O4XV7LvngRPIW8CMLOngCFAYZlydwK/ASZWJXAREakZyRLBD8osP1TFfbcBPopbLgb+I76AmZ0KnODuL5hZhYnAzEYDowHatWtXxTBERCSZZHMWv1LNfVui3ZZsNGtA8NTyyMp25O5TgakAeXl5XklxERGpgiq1+1dRMcHsZoe0BbbGLTcDugKvmVkR0AtYoA5jEZH0CjMRLAU6mVl7MzsSuBxYcGiju3/u7i3dPdvds4ElwGB3zw8xJhERKSPlRGBmR1Vlx+6+HxgHLALWAX9y97VmdoeZDa5amCIiEpZUhqHuCTwOHAO0M7PuwHWxKSuTcveFwMIy626roGzfVAIWEZGalUqN4AHgAuAzAHdfSTBjmYiI1AOpJIIG7v5BmXUHwghGRETSL5XRRz+KNQ957Gnhm4AN4YYlIiLpkkqNYCxwM9AO+JjgNs8qjzskIiK1UyqT139CcOuniIjUQ6ncNfQYcU8EH+Luo0OJSERE0iqVPoKX4943Br5P6TGERESkDkulaWhu/LKZzQReCi0iERFJq8MZYqI98O2aDkRERDIjlT6Cnfy7j6ABsAOocLYxERGpWyqbvN6A7sCW2KqD7q5hoEVE6pGkTUOxi/5z7n4g9lISEBGpZ1LpI3jHzHqEHomIiGREhU1DZnZEbCjpM4Hrzex94EuCmcfc3ZUcRETqgWR9BO8APYCL0hRLxs1++0PmF2xJWqZw2xfkZDVPU0QiIuFLlggMwN3fT1MsGTe/YEulF/qcrOYMyW2TxqhERMKVLBG0MrObK9ro7pNDiCfjcrKaM3dM70yHISKSNskSQUOgKbGagYiI1E/JEsE2d78jbZGIiEhGJLt9VDUBEZEISJYI+qUtChERyZgKE4G770hnICIikhmHM/qoiIjUI0oEIiIRp0QgIhJxSgQiIhGnRCAiEnFKBCIiEadEICIScUoEIiIRF2oiMLOBZvaumW00s3IT3pvZzWZWaGarzOwVM/t2mPGIiEh5oSUCM2sITAEGATnAMDPLKVNsBZDn7t2AZ4DfhBWPiIgkFmaNoCew0d03ufte4ClgSHwBd3/V3b+KLS4B2oYYj4iIJBBmImgDfBS3XBxbV5Frgb8m2mBmo80s38zyt2/fXoMhiohImIkg0TDWnrCg2RVAHnBfou3uPtXd89w9r1WrVjUYooiIJJuYprqKgRPiltsCW8sWMrNzgF8AZ7n71yHGIyIiCYRZI1gKdDKz9mZ2JHA5sCC+gJmdCvwBGOzun4QYi4iIVCC0RODu+4FxwCJgHfAnd19rZneY2eBYsfsI5kV+2swKzGxBBbsTEZGQhNk0hLsvBBaWWXdb3Ptzwjx+vNlvf8j8gi1JyxRu+4KcrOZpikhEpHaIzJPF8wu2ULjti6RlcrKaMyQ32Y1NIiL1T6g1gtomJ6s5c8f0znQYIiK1SmRqBCIikpgSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRFyk5iyW+mffvn0UFxezZ8+eTIciUis0btyYtm3b0qhRo5Q/o0QgdVpxcTHNmjUjOzsbM8t0OCIZ5e589tlnFBcX0759+5Q/p6YhqdP27NlDixYtlAREADOjRYsWVa4hKxFInackIPJvh/P3oEQgIhJxSgQi1dS0adNq72Pr1q0MHTq0wu27du3i4YcfTrk8QN++ffnOd75D9+7dOf300ykoKKh2nDXptttu4+WXX66Rfa1YsYLrrruu1LohQ4bQu3fvUutGjhzJM888U2pd/O9vw4YNnHfeeZx44ol06dKFSy+9lI8//rhase3YsYP+/fvTqVMn+vfvz86dOxOWu+WWWzj55JPp0qUL48ePx90B2Lt3L6NHj+akk06ic+fOzJs3D4CHHnqIadOmVSu2Q5QIRGqB448/vtwFKl7ZRFBZ+UNmzZrFypUr+eEPf8hPf/rTGol1//79NbKfO+64g3POOadG9vXf//3f3HTTTSXLu3btYvny5ezatYvNmzentI89e/Zw/vnnM3bsWDZu3Mi6desYO3Ys27dvr1Zs99xzD/369eO9996jX79+3HPPPeXKvPnmmyxevJhVq1axZs0ali5dyuuvvw7A3XffTevWrdmwYQOFhYWcddZZAIwaNYoHHnigWrEdoruGpN74rz+vpXDrFzW6z5zjm3P7hSdX+XMffPABo0aNYvv27bRq1Ypp06bRrl073n//fUaMGMGBAwcYNGgQkydP5n//938pKiriggsuYM2aNaxdu5ZrrrmGvXv3cvDgQebNm8evfvUr3n//fXJzc+nfvz833nhjSfkDBw7ws5/9jEWLFmFmXH/99aUuigC9e/fmvvvuK1n+29/+xu23387XX39Nx44dmTZtGk2bNmXhwoXcfPPNtGzZkh49erBp0yZeeOEFJk2axNatWykqKqJly5bMnDmTW2+9lddee42vv/6aG2+8kTFjxrBt2zYuu+wyvvjiC/bv388jjzzCGWecwbXXXkt+fj5mxqhRo/jxj3/MyJEjueCCCxg6dCivvPIKEydOZP/+/Zx++uk88sgjHHXUUWRnZ3P11Vfz5z//mX379vH000/TuXPnUue2e/duVq1aRffu3UvWzZs3jwsvvJDjjjuOp556ip///OeV/s5mz55N7969ufDCC0vWnX322VX+3Zc1f/58XnvtNQCuvvpq+vbty7333luqjJmxZ88e9u7di7uzb98+jjvuOACeeOIJ1q9fD0CDBg1o2bIlAEcffTTZ2dm888479OzZs1oxqkYgEoJx48Zx1VVXsWrVKkaMGMH48eMBmDBhAhMmTGDp0qUcf/zxCT/76KOPMmHCBAoKCsjPz6dt27bcc889dOzYkYKCglIXdICpU6eyefNmVqxYUXK8sl588UUuuugiAD799FPuuusuXn75ZZYvX05eXh6TJ09mz549jBkzhr/+9a/885//LPdNeNmyZcyfP5/Zs2fz+OOPc8wxx7B06VKWLl3KY489xubNm5k9ezYDBgygoKCAlStXkpubS0FBAVu2bGHNmjWsXr2aa665ptR+9+zZw8iRI5k7dy6rV68uSSCHtGzZkuXLlzN27Fjuv//+cueWn59P165dS62bM2cOw4YNY9iwYcyZM6eiX1Mpa9as4bTTTqu03O7du8nNzU34KiwsLFf+448/JisrC4CsrCw++eSTcmV69+7N2WefTVZWFllZWQwYMIAuXbqwa9cuAH71q1/Ro0cPLrnkklJNVXl5efzjH/9I6fySUY1A6o3D+eYelrfeeotnn30WgCuvvJJbbrmlZP3zzz8PwPDhw5k4cWK5z/bu3Zu7776b4uJiLr74Yjp16pT0WC+//DI33HADRxwR/Dkfe+yxJdtGjBjBl19+yYEDB1i+fDkAS5YsobCwkD59+gBBG3Tv3r1Zv349HTp0KLn/fNiwYUydOrVkX4MHD6ZJkyZAUKNYtWpVSfPU559/znvvvcfpp5/OqFGj2LdvHxdddBG5ubl06NCBTZs2cdNNN3H++edz7rnnlor/3XffpX379px00klA8K15ypQp/OhHPwLg4osvBuC0004r+ZnG27ZtG61atSpZ/vjjj9m4cSNnnnkmZsYRRxzBmjVr6Nq1a8I7aqp6l02zZs1qvL/lUFNUcXExAP379+eNN94gJyeH4uJi+vTpw+TJk5k8eTITJ05k5syZALRu3bqktlAdodYIzGygmb1rZhvN7NYE248ys7mx7W+bWXaY8YhkSlUuNsOHD2fBggU0adKEAQMG8Pe//z1peXevcP+zZs1i8+bNDB8+nBtvvLGkfP/+/SkoKKCgoIDCwkIef/zxks7JinzjG98odcwHH3ywZB+bN2/m3HPP5Xvf+x5vvPEGbdq04corr+TJJ5/kW9/6FitXrqRv375MmTKlXKduZcc96qijAGjYsGHC/okmTZqUum9+7ty57Ny5k/bt25OdnU1RURFPPfUUAC1atCjVWbtjx46SppaTTz6ZZcuWJY0Fql4jOO6449i2bRsQJK3WrVuXK/Pcc8/Rq1cvmjZtStOmTRk0aBBLliyhRYsWHH300Xz/+98H4JJLLilJ6BDUpg4l5+oILRGYWUNgCjAIyAGGmVlOmWLXAjvd/UTgf4B7EakHzjjjjJKLz6xZszjzzDMB6NWrV8ldH4e2l7Vp0yY6dOjA+PHjGTx4MKtWraJZs2bs3r07Yflzzz2XRx99tOQiuWPHjlLbGzVqxF133cWSJUtYt24dvXr1YvHixWzcuBGAr776ig0bNtC5c2c2bdpEUVEREFxQKzJgwAAeeeQR9u3bBwR323z55Zd88MEHtG7dmuuvv55rr72W5cuX8+mnn3Lw4EF+8IMfcOedd5a6kAF07tyZoqKiknhmzpxZ0iGaii5dupR8FoJmoRdffJGioiKKiopYtmxZyc+6b9++zJ07l7179wIwffr0kn6A4cOH8+abb/KXv/ylZF8vvvgiq1evLnW8QzWCRK+cnLKXuKAmNWPGDABmzJjBkCFDypVp164dr7/+Ovv372ffvn28/vrrdOnSBTPjwgsvLOljeOWVV0odY8OGDeWaxQ5HmDWCnsBGd9/k7nuBp4CyP4EhwIzY+2eAfqang6SO+eqrr2jbtm3Ja/LkyTzwwANMmzaNbt26MXPmTH7/+98D8Lvf/Y7JkyfTs2dPtm3bxjHHHFNuf3PnzqVr167k5uayfv16rrrqKlq0aEGfPn3o2rVrubt/rrvuOtq1a0e3bt3o3r07s2fPLrfPJk2a8JOf/IT777+fVq1aMX36dIYNG0a3bt3o1asX69evp0mTJjz88MMMHDiQM888k+OOOy5hfIeOmZOTQ48ePejatStjxoxh//79vPbaa+Tm5nLqqacyb948JkyYwJYtW+jbty+5ubmMHDmSX//616X21bhxY6ZNm8Yll1zCKaecQoMGDbjhhhtS/vl37tyZzz//nN27d1NUVMSHH35Ir169Sra3b9+e5s2b8/bbb3PBBRfw3e9+l9NOO43c3FwWL15c0nHbpEkTXnjhBR588EE6depETk4O06dPT/gNvipuvfVWXnrpJTp16sRLL73ErbcGjSP5+fkltaOhQ4fSsWNHTjnlFLp370737t1LOq3vvfdeJk2aVPJ/6be//W3JvhcvXlwzd165eygvYCjwx7jlK4GHypRZA7SNW34faJlgX6OBfCC/Xbt2fjgmLVjjkxasOazPSu1VWFiY6RCq5Msvv/SDBw+6u/ucOXN88ODBGY6otN27d7u7+8GDB33s2LE+efLkDEeUmsmTJ/tjjz2W6TDSavny5X7FFVck3Jbo7wLI9wqu12F2Fif6Zl+2MTCVMrj7VGAqQF5eXvIGxQrUpo5Eia5ly5Yxbtw43J1vfvObPPHEE5kOqZTHHnuMGTNmsHfvXk499VTGjBmT6ZBSMnbsWJ5++ulMh5FWn376KXfeeWeN7Mu8ko6aw96xWW9gkrsPiC3/HMDdfx1XZlGszFtmdgTw/4BWniSovLw8z8/PDyVmqXvWrVtHly5dMh2GSK2S6O/CzJa5e16i8mH2ESwFOplZezM7ErgcWFCmzALg6tj7ocDfkyUBkUT0X0bk3w7n7yG0RODu+4FxwCJgHfAnd19rZneY2eBYsceBFma2EbgZKHeLqUgyjRs35rPPPlMyEOHf8xE0bty4Sp8LrWkoLGoakniaoUyktIpmKEvWNKQni6VOa9SoUZVmYhKR8jTWkIhIxCkRiIhEnBKBiEjE1bnOYjPbDnxwmB9vCXxag+HUBTrnaNA5R0N1zvnb7t4q0YY6lwiqw8zyK+o1r690ztGgc46GsM5ZTUMiIhGnRCAiEnFRSwRTKy9S7+ico0HnHA2hnHOk+ghERKS8qNUIRESkDCUCEZGIq5eJwMwGmtm7ZrbRzMqNaGpmR5nZ3Nj2t80sO/1R1qwUzvlmMys0s1Vm9oqZfTsTcdakys45rtxQM3Mzq/O3GqZyzmZ2aex3vdbMys9bWcek8H+7nZm9amYrYv+/z8tEnDXFzJ4ws0/MbE0F283MHoj9PFaZWY9qH7Siqcvq6gtoSDDlZQfgSGAlkFOmzA+BR2PvLwfmZjruNJzz2cDRsfdjo3DOsXLNgDeAJUBepuNOw++5E7AC+FZsuXWm407DOU8Fxsbe5wBFmY67muf8PaAHsKaC7ecBfyWY4bEX8HZ1j1kfawQ9gY3uvsnd9wJPAUPKlBkCzIi9fwboZ2aJps2sKyo9Z3d/1d2/ii0uAdqmOcaalsrvGeBO4DdAfRinOpVzvh6Y4u47Adz9kzTHWNNSOWcHmsfeHwNsTWN8Nc7d3wB2JCkyBHjSA0uAb5pZVnWOWR8TQRvgo7jl4ti6hGU8mEDnc6BFWqILRyrnHO9agm8UdVml52xmpwInuPsL6QwsRKn8nk8CTjKzxWa2xMwGpi26cKRyzpOAK8ysGFgI3JSe0DKmqn/vlaqP8xEk+mZf9h7ZVMrUJSmfj5ldAeQBZ4UaUfiSnrOZNQD+BxiZroDSIJXf8xEEzUN9CWp9/zCzru6+K+TYwpLKOQ8Dprv7b2Nzpc+MnfPB8MPLiBq/ftXHGkExcELcclvKVxVLypjZEQTVyWRVsdoulXPGzM4BfgEMdvev0xRbWCo752ZAV+A1MysiaEtdUMc7jFP9vz3f3fe5+2bgXYLEUFelcs7XAn8CcPe3gMYEg7PVVyn9vVdFfUwES4FOZtbezI4k6AxeUKbMAuDq2PuhwN891gtTR1V6zrFmkj8QJIG63m4MlZyzu3/u7i3dPdvdswn6RQa7e12e5zSV/9vPE9wYgJm1JGgq2pTWKGtWKuf8IdAPwMy6ECSC7WmNMr0WAFfF7h7qBXzu7tuqs8N61zTk7vvNbBywiOCOgyfcfa2Z3QHku/sC4HGC6uNGgprA5ZmLuPpSPOf7gKbA07F+8Q/dfXDGgq6mFM+5XknxnBcB55pZIXAA+Km7f5a5qKsnxXP+CfCYmf2YoIlkZF3+Ymdmcwia9lrG+j1uBxoBuPujBP0g5wEbga+Aa6p9zDr88xIRkRpQH5uGRESkCpQIREQiTolARCTilAhERCJOiUBEJOKUCKTWMbMDZlYQ98pOUja7olEaq3jM12IjXK6MDc/wncPYxw1mdit0z9MAAAOXSURBVFXs/UgzOz5u2x/NLKeG41xqZrkpfOZHZnZ0dY8t9ZcSgdRG/3L33LhXUZqOO8LduxMMSHhfVT/s7o+6+5OxxZHA8XHbrnP3whqJ8t9xPkxqcf4IUCKQCikRSJ0Q++b/DzNbHnudkaDMyWb2TqwWscrMOsXWXxG3/g9m1rCSw70BnBj7bL/YOPerY+PEHxVbf4/9e36H+2PrJpnZRDMbSjCe06zYMZvEvsnnmdlYM/tNXMwjzezBw4zzLeIGGzOzR8ws34J5CP4rtm48QUJ61cxeja0718zeiv0cnzazppUcR+o5JQKpjZrENQs9F1v3CdDf3XsAlwEPJPjcDcDv3T2X4EJcHBty4DKgT2z9AWBEJce/EFhtZo2B6cBl7n4KwZP4Y83sWOD7wMnu3g24K/7D7v4MkE/wzT3X3f8Vt/kZ4OK45cuAuYcZ50CCISUO+YW75wHdgLPMrJu7P0AwDs3Z7n52bNiJXwLnxH6W+cDNlRxH6rl6N8SE1Av/il0M4zUCHoq1iR8gGEOnrLeAX5hZW+BZd3/PzPoBpwFLY0NrNCFIKonMMrN/AUUEQxl/B9js7hti22cANwIPEcxv8Ecz+wuQ8jDX7r7dzDbFxoh5L3aMxbH9ViXObxAMuRA/O9WlZjaa4O86i2CSllVlPtsrtn5x7DhHEvzcJMKUCKSu+DHwMdCdoCZbbqIZd59tZm8D5wOLzOw6giF7Z7j7z1M4xoj4QenMLOEcFbHxb3oSDHR2OTAO+M8qnMtc4FJgPfCcu7sFV+WU4ySYqeseYApwsZm1ByYCp7v7TjObTjD4WlkGvOTuw6oQr9RzahqSuuIYYFtsjPkrCb4Nl2JmHYBNseaQBQRNJK8AQ82sdazMsZb6fM3rgWwzOzG2fCXweqxN/Rh3X0jQEZvozp3dBENhJ/IscBHBOPpzY+uqFKe77yNo4ukVa1ZqDnwJfG5mxwGDKohlCdDn0DmZ2dFmlqh2JRGiRCB1xcPA1Wa2hKBZ6MsEZS4D1phZAdCZYDq/QoIL5t/MbBXwEkGzSaXcfQ/ByI5Pm9lq4CDwKMFF9YXY/l4nqK2UNR149FBncZn97gQKgW+7+zuxdVWOM9b38FtgoruvJJireC3wBEFz0yFTgb+a2avuvp3gjqY5seMsIfhZSYRp9FERkYhTjUBEJOKUCEREIk6JQEQk4pQIREQiTolARCTilAhERCJOiUBEJOL+Px0xPWmtSgc2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot a ROC curve using our current machine learning model using plot_roc_curve\n",
    "plot_roc_curve(clf, X_test, y_test);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful! We've gone far beyond accuracy with a plethora extra classification evaluation metrics.\n",
    "\n",
    "If you're not sure about any of these, don't worry, they can take a while to understand. That could be an optional extension, reading up on a classification metric you're not sure of.\n",
    "\n",
    "The thing to note here is all of these metrics have been calculated using a single training set and a single test set. Whilst this is okay, a more robust way is to calculate them using [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html).\n",
    "\n",
    "We can calculate various evaluation metrics using cross-validation using Scikit-Learn's [`cross_val_score()`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) function along with the `scoring` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import cross_val_score from sklearn's model_selection module\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.81967213, 0.90163934, 0.8852459 , 0.88333333, 0.75      ])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# EXAMPLE: By default cross_val_score returns 5 values (cv=5).\n",
    "cross_val_score(clf, \n",
    "                X, \n",
    "                y, \n",
    "                scoring=\"accuracy\",\n",
    "                cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8479781420765027"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# EXAMPLE: Taking the mean of the returned values from cross_val_score \n",
    "# gives a cross-validated version of the scoring metric.\n",
    "cross_val_acc = np.mean(cross_val_score(clf,\n",
    "                                        X,\n",
    "                                        y,\n",
    "                                        scoring=\"accuracy\",\n",
    "                                        cv=5))\n",
    "\n",
    "cross_val_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the examples, the cross-validated accuracy is found by taking the mean of the array returned by `cross_val_score()`.\n",
    "\n",
    "Now it's time to find the same for precision, recall and F1 score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8215873015873015"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the cross-validated precision\n",
    "cross_val_precision = np.mean(cross_val_score(clf,\n",
    "                                              X,\n",
    "                                              y,\n",
    "                                              scoring=\"precision\",\n",
    "                                              cv=5))\n",
    "\n",
    "cross_val_precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9272727272727274"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the cross-validated recall\n",
    "cross_val_recall = np.mean(cross_val_score(clf,\n",
    "                                           X,\n",
    "                                           y,\n",
    "                                           scoring=\"recall\",\n",
    "                                           cv=5))\n",
    "\n",
    "cross_val_recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8705403543192143"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the cross-validated F1 score\n",
    "cross_val_f1 = np.mean(cross_val_score(clf,\n",
    "                                       X,\n",
    "                                       y,\n",
    "                                       scoring=\"f1\",\n",
    "                                       cv=5))\n",
    "\n",
    "cross_val_f1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exporting and importing a trained model\n",
    "\n",
    "Once you've trained a model, you may want to export it and save it to file so you can share it or use it elsewhere.\n",
    "\n",
    "One method of exporting and importing models is using the joblib library.\n",
    "\n",
    "In Scikit-Learn, exporting and importing a trained model is known as [model persistence](https://scikit-learn.org/stable/modules/model_persistence.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the dump and load functions from the joblib library\n",
    "from joblib import dump, load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['trained-classifier.joblib']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use the dump function to export the trained model to file\n",
    "dump(clf, \"trained-classifier.joblib\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7894736842105263"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use the load function to import the trained model you just exported\n",
    "# Save it to a different variable name to the origial trained model\n",
    "loaded_clf = load(\"trained-classifier.joblib\")\n",
    "\n",
    "# Evaluate the loaded trained model on the test data\n",
    "loaded_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the loaded trained model results versus the original (pre-exported) model results?\n",
    "\n",
    "\n",
    "## Scikit-Learn Regression Practice\n",
    "\n",
    "For the next few exercises, we're going to be working on a regression problem, in other words, using some data to predict a number.\n",
    "\n",
    "Our dataset is a [table of car sales](https://docs.google.com/spreadsheets/d/1LPEIWJdSSJYrfn-P3UQDIXbEn5gg-o6I7ExLrWTTBWs/edit?usp=sharing), containing different car characteristics as well as a sale price.\n",
    "\n",
    "We'll use Scikit-Learn's built-in regression machine learning models to try and learn the patterns in the car characteristics and their prices on a certain group of the dataset before trying to predict the sale price of a group of cars the model has never seen before.\n",
    "\n",
    "To begin, we'll [import the data from GitHub](https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/car-sales-extended-missing-data.csv) into a pandas DataFrame, check out some details about it and try to build a model as soon as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Colour</th>\n",
       "      <th>Odometer (KM)</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>35431.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15323.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>Blue</td>\n",
       "      <td>192714.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>19943.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>84714.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>28343.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>White</td>\n",
       "      <td>154365.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>13434.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nissan</td>\n",
       "      <td>Blue</td>\n",
       "      <td>181577.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>14043.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Make Colour  Odometer (KM)  Doors    Price\n",
       "0   Honda  White        35431.0    4.0  15323.0\n",
       "1     BMW   Blue       192714.0    5.0  19943.0\n",
       "2   Honda  White        84714.0    4.0  28343.0\n",
       "3  Toyota  White       154365.0    4.0  13434.0\n",
       "4  Nissan   Blue       181577.0    3.0  14043.0"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in the car sales data\n",
    "car_sales = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/car-sales-extended-missing-data.csv\")\n",
    "\n",
    "# View the first 5 rows of the car sales data\n",
    "car_sales.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000 entries, 0 to 999\n",
      "Data columns (total 5 columns):\n",
      "Make             951 non-null object\n",
      "Colour           950 non-null object\n",
      "Odometer (KM)    950 non-null float64\n",
      "Doors            950 non-null float64\n",
      "Price            950 non-null float64\n",
      "dtypes: float64(3), object(2)\n",
      "memory usage: 39.2+ KB\n"
     ]
    }
   ],
   "source": [
    "# Get information about the car sales DataFrame\n",
    "car_sales.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the output of `info()`,\n",
    "* How many rows are there total?\n",
    "* What datatypes are in each column?\n",
    "* How many missing values are there in each column?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             49\n",
       "Colour           50\n",
       "Odometer (KM)    50\n",
       "Doors            50\n",
       "Price            50\n",
       "dtype: int64"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find number of missing values in each column\n",
    "car_sales.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make              object\n",
       "Colour            object\n",
       "Odometer (KM)    float64\n",
       "Doors            float64\n",
       "Price            float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the datatypes of each column of car_sales\n",
    "car_sales.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowing this information, what would happen if we tried to model our data as it is?\n",
    "\n",
    "Let's see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "could not convert string to float: 'Honda'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-44-476d8071e1b5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensemble\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mRandomForestRegressor\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mcar_sales_X\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcar_sales_y\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcar_sales\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdrop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Price\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcar_sales\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPrice\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mrf_regressor\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mRandomForestRegressor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcar_sales_X\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcar_sales_y\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/Desktop/ml-course/zero-to-mastery-ml/env/lib/python3.6/site-packages/sklearn/ensemble/_forest.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[1;32m    293\u001b[0m         \"\"\"\n\u001b[1;32m    294\u001b[0m         \u001b[0;31m# Validate or convert input data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 295\u001b[0;31m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcheck_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccept_sparse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"csc\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mDTYPE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    296\u001b[0m         \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcheck_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccept_sparse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'csc'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mensure_2d\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    297\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0msample_weight\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/ml-course/zero-to-mastery-ml/env/lib/python3.6/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36mcheck_array\u001b[0;34m(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)\u001b[0m\n\u001b[1;32m    513\u001b[0m                     \u001b[0marray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0marray\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcasting\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"unsafe\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    514\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 515\u001b[0;31m                     \u001b[0marray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    516\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mComplexWarning\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    517\u001b[0m                 raise ValueError(\"Complex data not supported\\n\"\n",
      "\u001b[0;32m~/Desktop/ml-course/zero-to-mastery-ml/env/lib/python3.6/site-packages/numpy/core/_asarray.py\u001b[0m in \u001b[0;36masarray\u001b[0;34m(a, dtype, order)\u001b[0m\n\u001b[1;32m     83\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m     \"\"\"\n\u001b[0;32m---> 85\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morder\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: could not convert string to float: 'Honda'"
     ]
    }
   ],
   "source": [
    "# EXAMPLE: This doesn't work because our car_sales data isn't all numerical\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "car_sales_X, car_sales_y = car_sales.drop(\"Price\", axis=1), car_sales.Price\n",
    "rf_regressor = RandomForestRegressor().fit(car_sales_X, car_sales_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see, the cell above breaks because our data contains non-numerical values as well as missing data.\n",
    "\n",
    "To take care of some of the missing data, we'll remove the rows which have no labels (all the rows with missing values in the `Price` column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove rows with no labels (NaN's in the Price column)\n",
    "car_sales.dropna(subset=[\"Price\"], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a pipeline\n",
    "Since our `car_sales` data has missing numerical values as well as the data isn't all numerical, we'll have to fix these things before we can fit a machine learning model on it.\n",
    "\n",
    "There are ways we could do this with pandas but since we're practicing Scikit-Learn, we'll see how we might do it with the [`Pipeline`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) class. \n",
    "\n",
    "Because we're modifying columns in our dataframe (filling missing values, converting non-numerical data to numbers) we'll need the [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html), [`SimpleImputer`](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html) and [`OneHotEncoder`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) classes as well.\n",
    "\n",
    "Finally, because we'll need to split our data into training and test sets, we'll import `train_test_split` as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Pipeline from sklearn's pipeline module\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# Import ColumnTransformer from sklearn's compose module\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "# Import SimpleImputer from sklearn's impute module\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# Import OneHotEncoder from sklearn's preprocessing module\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "# Import train_test_split from sklearn's model_selection module\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we've got the necessary tools we need to create our preprocessing `Pipeline` which fills missing values along with turning all non-numerical data into numbers.\n",
    "\n",
    "Let's start with the categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define different categorical features \n",
    "categorical_features = [\"Make\", \"Colour\"]\n",
    "\n",
    "# Create categorical transformer Pipeline\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    # Set SimpleImputer strategy to \"constant\" and fill value to \"missing\"\n",
    "    (\"imputer\", SimpleImputer(strategy=\"constant\", fill_value=\"missing\")),\n",
    "    # Set OneHotEncoder to ignore the unknowns\n",
    "    (\"onehot\", OneHotEncoder(handle_unknown=\"ignore\"))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would be safe to treat `Doors` as a categorical feature as well, however since we know the vast majority of cars have 4 doors, we'll impute the missing `Doors` values as 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define Doors features\n",
    "door_feature = [\"Doors\"]\n",
    "\n",
    "# Create Doors transformer Pipeline\n",
    "door_transformer = Pipeline(steps=[\n",
    "    # Set SimpleImputer strategy to \"constant\" and fill value to 4\n",
    "    (\"imputer\", SimpleImputer(strategy=\"constant\", fill_value=4))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now onto the numeric features. In this case, the only numeric feature is the `Odometer (KM)` column. Let's fill its missing values with the median."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define numeric features (only the Odometer (KM) column)\n",
    "numeric_features = [\"Odometer (KM)\"]\n",
    "\n",
    "# Crearte numeric transformer Pipeline\n",
    "numeric_transformer = Pipeline(steps=[\n",
    "    # Set SimpleImputer strategy to fill missing values with the \"Median\"\n",
    "    (\"imputer\", SimpleImputer(strategy=\"median\"))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to put all of our individual transformer `Pipeline`'s into a single `ColumnTransformer` instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup preprocessing steps (fill missing values, then convert to numbers)\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        # Use the categorical_transformer to transform the categorical_features\n",
    "        (\"cat\", categorical_transformer, categorical_features),\n",
    "        # Use the door_transformer to transform the door_feature\n",
    "        (\"door\", door_transformer, door_feature),\n",
    "        # Use the numeric_transformer to transform the numeric_features\n",
    "        (\"num\", numeric_transformer, numeric_features)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boom! Now our `preprocessor` is ready, time to import some regression models to try out.\n",
    "\n",
    "Comparing our data to the [Scikit-Learn machine learning map](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html), we can see there's a handful of different regression models we can try.\n",
    "\n",
    "* [RidgeRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html)\n",
    "* [SVR(kernel=\"linear\")](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html) - short for Support Vector Regressor, a form form of support vector machine.\n",
    "* [SVR(kernel=\"rbf\")](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html) - short for Support Vector Regressor, a form of support vector machine.\n",
    "* [RandomForestRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) - the regression version of RandomForestClassifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Ridge from sklearn's linear_model module\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "# Import SVR from sklearn's svm module\n",
    "from sklearn.svm import SVR\n",
    "\n",
    "# Import RandomForestRegressor from sklearn's ensemble module\n",
    "from sklearn.ensemble import RandomForestRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, thanks to the design of the Scikit-Learn library, we're able to use very similar code for each of these models.\n",
    "\n",
    "To test them all, we'll create a dictionary of regression models and an empty dictionary for regression model results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create dictionary of model instances, there should be 4 total key, value pairs\n",
    "# in the form {\"model_name\": model_instance}.\n",
    "# Don't forget there's two versions of SVR, one with a \"linear\" kernel and the\n",
    "# other with kernel set to \"rbf\".\n",
    "regression_models = {\"Ridge\": Ridge(),\n",
    "                     \"SVR_linear\": SVR(kernel=\"linear\"),\n",
    "                     \"SVR_rbf\": SVR(kernel=\"rbf\"),\n",
    "                     \"RandomForestRegressor\": RandomForestRegressor()}\n",
    "\n",
    "# Create an empty dictionary for the regression results\n",
    "regression_results = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our regression model dictionary is prepared as well as an empty dictionary to append results to, time to get the data split into `X` (feature variables) and `y` (target variable) as well as training and test sets.\n",
    "\n",
    "In our car sales problem, we're trying to use the different characteristics of a car (`X`) to predict its sale price (`y`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create car sales X data (every column of car_sales except Price)\n",
    "car_sales_X = car_sales.drop(\"Price\", axis=1)\n",
    "\n",
    "# Create car sales y data (the Price column of car_sales)\n",
    "car_sales_y = car_sales[\"Price\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((760, 4), (190, 4), (760,), (190,))"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use train_test_split to split the car_sales_X and car_sales_y data into \n",
    "# training and test sets.\n",
    "# Give the test set 20% of the data using the test_size parameter.\n",
    "# For reproducibility set the random_state parameter to 42.\n",
    "car_X_train, car_X_test, car_y_train, car_y_test = train_test_split(car_sales_X,\n",
    "                                                                    car_sales_y,\n",
    "                                                                    test_size=0.2,\n",
    "                                                                    random_state=42)\n",
    "\n",
    "# Check the shapes of the training and test datasets\n",
    "car_X_train.shape, car_X_test.shape, car_y_train.shape, car_y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* How many rows are in each set?\n",
    "* How many columns are in each set?\n",
    "\n",
    "Alright, our data is split into training and test sets, time to build a small loop which is going to:\n",
    "1. Go through our `regression_models` dictionary\n",
    "2. Create a `Pipeline` which contains our `preprocessor` as well as one of the models in the dictionary\n",
    "3. Fits the `Pipeline` to the car sales training data\n",
    "4. Evaluates the target model on the car sales test data and appends the results to our `regression_results` dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting Ridge...\n",
      "Scoring Ridge...\n",
      "Fitting SVR_linear...\n",
      "Scoring SVR_linear...\n",
      "Fitting SVR_rbf...\n",
      "Scoring SVR_rbf...\n",
      "Fitting RandomForestRegressor...\n",
      "Scoring RandomForestRegressor...\n"
     ]
    }
   ],
   "source": [
    "# Loop through the items in the regression_models dictionary\n",
    "for model_name, model in regression_models.items():\n",
    "    \n",
    "    # Create a model pipeline with a preprocessor step and model step\n",
    "    model_pipeline = Pipeline(steps=[(\"preprocessor\", preprocessor),\n",
    "                                      (\"model\", model)])\n",
    "    \n",
    "    # Fit the model pipeline to the car sales training data\n",
    "    print(f\"Fitting {model_name}...\")\n",
    "    model_pipeline.fit(car_X_train, car_y_train)\n",
    "    \n",
    "    # Score the model pipeline on the test data appending the model_name to the \n",
    "    # results dictionary\n",
    "    print(f\"Scoring {model_name}...\")\n",
    "    regression_results[model_name] = model_pipeline.score(car_X_test, \n",
    "                                                          car_y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our regression models have been fit, let's see how they did!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Ridge': 0.25402611057943925,\n",
       " 'SVR_linear': -0.489452821008145,\n",
       " 'SVR_rbf': 0.0018546241516633755,\n",
       " 'RandomForestRegressor': 0.2291358152962253}"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the results of each regression model by printing the regression_results\n",
    "# dictionary\n",
    "regression_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Which model did the best?\n",
    "* How could you improve its results?\n",
    "* What metric does the `score()` method of a regression model return by default?\n",
    "\n",
    "Since we've fitted some models but only compared them via the default metric contained in the `score()` method (R^2 score or coefficient of determination), let's take the `RidgeRegression` model and evaluate it with a few other [regression metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics).\n",
    "\n",
    "Specifically, let's find:\n",
    "1. **R^2 (pronounced r-squared) or coefficient of determination** - Compares your models predictions to the mean of the targets. Values can range from negative infinity (a very poor model) to 1. For example, if all your model does is predict the mean of the targets, its R^2 value would be 0. And if your model perfectly predicts a range of numbers it's R^2 value would be 1. \n",
    "2. **Mean absolute error (MAE)** - The average of the absolute differences between predictions and actual values. It gives you an idea of how wrong your predictions were.\n",
    "3. **Mean squared error (MSE)** - The average squared differences between predictions and actual values. Squaring the errors removes negative errors. It also amplifies outliers (samples which have larger errors).\n",
    "\n",
    "Scikit-Learn has a few classes built-in which are going to help us with these, namely, [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html), [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html) and [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import mean_absolute_error from sklearn's metrics module\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "# Import mean_squared_error from sklearn's metrics module\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# Import r2_score from sklearn's metrics module\n",
    "from sklearn.metrics import r2_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the evaluation metrics we're concerned with compare a model's predictions with the ground truth labels. Knowing this, we'll have to make some predictions.\n",
    "\n",
    "Let's create a `Pipeline` with the `preprocessor` and a `Ridge()` model, fit it on the car sales training data and then make predictions on the car sales test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([18514.480873  , 22204.86580267, 11045.72092225,  6891.87687957,\n",
       "        8793.93970278, 10926.96786577, 15267.76229871, 13834.80312146,\n",
       "       20207.44377898, 14412.59968701,  6216.01228306, 16543.06707068,\n",
       "       11783.69374936, 13492.13284255, 14321.17899086, 16425.45131776,\n",
       "       15999.88230172,  9925.04252351, 11576.64865889, 11580.55130633,\n",
       "       10647.22402588, 13113.54464064, 17874.90033257, 23447.29246732,\n",
       "       11797.33019118, 14481.85336505, 18430.527126  , 14680.33044208,\n",
       "       20585.63901269, 19953.91395769, 18163.1170151 , 22366.43390174,\n",
       "       12435.69707467, 14265.25495748, 18351.62656383, 19564.58267503,\n",
       "       12213.05747919, 12479.20772529, 18677.93478691, 11284.89072638,\n",
       "       15278.54001605, 17378.81425165, 19247.77623181, 17343.90692672,\n",
       "       15048.12797569, 12724.44477165, 12389.86021752,  8475.63778401,\n",
       "       15255.49324105, 18602.79970861])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create RidgeRegression Pipeline with preprocessor as the \"preprocessor\" and\n",
    "# Ridge() as the \"model\".\n",
    "ridge_pipeline = Pipeline(steps=[(\"preprocessor\", preprocessor),\n",
    "                                 (\"model\", Ridge())])\n",
    "\n",
    "# Fit the RidgeRegression Pipeline to the car sales training data\n",
    "ridge_pipeline.fit(car_X_train, car_y_train)\n",
    "\n",
    "# Make predictions on the car sales test data using the RidgeRegression Pipeline\n",
    "car_y_preds = ridge_pipeline.predict(car_X_test)\n",
    "\n",
    "# View the first 50 predictions\n",
    "car_y_preds[:50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! Now we've got some predictions, time to evaluate them. We'll find the mean squared error (MSE), mean absolute error (MAE) and R^2 score (coefficient of determination) of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49950182.633374564"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# EXAMPLE: Find the MSE by comparing the car sales test labels to the car sales predictions\n",
    "mse = mean_squared_error(car_y_test, car_y_preds)\n",
    "# Return the MSE\n",
    "mse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5713.8215208551555"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the MAE by comparing the car sales test labels to the car sales predictions\n",
    "mae =  mean_absolute_error(car_y_test, car_y_preds)\n",
    "# Return the MAE\n",
    "mae"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25402611057943925"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the R^2 score by comparing the car sales test labels to the car sales predictions\n",
    "r2 = r2_score(car_y_test, car_y_preds)\n",
    "# Return the R^2 score\n",
    "r2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boom! Our model could potentially do with some hyperparameter tuning (this would be a great extension). And we could probably do with finding some more data on our problem, 1000 rows doesn't seem to be sufficient.\n",
    "\n",
    "* How would you export the trained regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extensions\n",
    "\n",
    "You should be proud. Getting this far means you've worked through a classification problem and regression problem using pure (mostly) Scikit-Learn (no easy feat!).\n",
    "\n",
    "For more exercises, check out the [Scikit-Learn getting started documentation](https://scikit-learn.org/stable/getting_started.html). A good practice would be to read through it and for the parts you find interesting, add them into the end of this notebook.\n",
    "\n",
    "Finally, as always, remember, the best way to learn something new is to try it. And try it relentlessly. If you're unsure of how to do something, never be afraid to ask a question or search for something such as, \"how to tune the hyperparmaters of a scikit-learn ridge regression model\"."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}