1-data-science-and-ml-tools/scikit-learn-metric-comparison.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing the metrics of different Scikit-Learn models\n",
    "\n",
    "One of the most important things when comparing different models is to make sure they're compared on the same data splits.\n",
    "\n",
    "For example, let's say you have `model_1` and `model_2` which each differ slightly.\n",
    "\n",
    "If you want to compare and evaulate their results, `model_1` and `model_2` should both be trained on the same data (e.g. `X_train` and `y_train`) and their predictions should each be made on the same data, for example:\n",
    "* `model_1.fit(X_train, y_train)` -> `model_1.predict(X_test)` -> `model_1_preds`\n",
    "* `model_2.fit(X_train, y_train)` -> `model_2.predict(X_test)` -> `model_2_preds`\n",
    "\n",
    "Note the differences here being the two models and the 2 different sets of predictions which can be compared against each other.\n",
    "\n",
    "This short notebook compares 3 different models on a small dataset.\n",
    "1. A baseline `RandomForestClassifier` (all default parameters)\n",
    "2. A `RandomForestClassifier` tuned with `RandomizedSearchCV` (and `refit=True`)\n",
    "3. A `RandomForestClassifier` tuned with `GridSearchCV` (and `refit=True`)\n",
    "\n",
    "The most important part is they all use the same data splits created using `train_test_split()` and `np.random.seed(42)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import and split data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "heart_disease = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/heart-disease.csv\")\n",
    "\n",
    "# Split into X & y\n",
    "X = heart_disease.drop(\"target\", axis =1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Split into train & test\n",
    "np.random.seed(42) # seed for reproducibility\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make evaluation function\n",
    "\n",
    "Our evaluation function will use all of the major classification metric functions from Scikit-Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_preds(y_true, y_preds):\n",
    "    \"\"\"\n",
    "    Performs evaluation comparison on y_true labels vs. y_pred labels\n",
    "    on a classification.\n",
    "    \"\"\"\n",
    "    accuracy = accuracy_score(y_true, y_preds)\n",
    "    precision = precision_score(y_true, y_preds)\n",
    "    recall = recall_score(y_true, y_preds)\n",
    "    f1 = f1_score(y_true, y_preds)\n",
    "    metric_dict = {\"accuracy\": round(accuracy, 2),\n",
    "                   \"precision\": round(precision, 2),\n",
    "                   \"recall\": round(recall, 2),\n",
    "                   \"f1\": round(f1, 2)}\n",
    "    print(f\"Acc: {accuracy * 100:.2f}%\")\n",
    "    print(f\"Precision: {precision:.2f}\")\n",
    "    print(f\"Recall: {recall:.2f}\")\n",
    "    print(f\"F1 score: {f1:.2f}\")\n",
    "    \n",
    "    return metric_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline model\n",
    "Create model with default hyperparameters. See [RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) documentation for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 83.61%\n",
      "Precision: 0.84\n",
      "Recall: 0.84\n",
      "F1 score: 0.84\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "# Make & fit baseline model\n",
    "clf = RandomForestClassifier()\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Make baseline predictions\n",
    "y_preds = clf.predict(X_test)\n",
    "\n",
    "# Evaluate the classifier on validation set\n",
    "baseline_metrics = evaluate_preds(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RandomizedSearchCV\n",
    "Find hyperparameters with [RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html).\n",
    "\n",
    "**Note:** Although best parameters are found on different splits of `X_train` and `y_train`, because `refit=True`, once the best parameters are found, they are refit to the entire set of `X_train` and `y_train`. See the [RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html) and [cross-validation documentation](https://scikit-learn.org/stable/modules/cross_validation.html) for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n",
      "[CV] n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10 \n",
      "[CV]  n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10 \n",
      "[CV]  n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10 \n",
      "[CV]  n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10 \n",
      "[CV]  n_estimators=100, min_samples_split=6, min_samples_leaf=1, max_features=auto, max_depth=10, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.1s\n",
      "[CV] n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30 \n",
      "[CV]  n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30, total=   1.2s\n",
      "[CV] n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30 \n",
      "[CV]  n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30, total=   1.2s\n",
      "[CV] n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30 \n",
      "[CV]  n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30, total=   1.2s\n",
      "[CV] n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30 \n",
      "[CV]  n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30, total=   1.2s\n",
      "[CV] n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30 \n",
      "[CV]  n_estimators=1200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=30, total=   1.2s\n",
      "[CV] n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.2s\n",
      "[CV] n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.2s\n",
      "[CV] n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.2s\n",
      "[CV] n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.2s\n",
      "[CV] n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=200, min_samples_split=6, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   0.2s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30 \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30 \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30 \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30 \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30, total=   0.1s\n",
      "[CV] n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30 \n",
      "[CV]  n_estimators=100, min_samples_split=2, min_samples_leaf=4, max_features=auto, max_depth=30, total=   0.1s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   1.2s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   1.2s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   1.2s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   1.4s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=2, max_features=sqrt, max_depth=None, total=   1.2s\n",
      "[CV] n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10 \n",
      "[CV]  n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10, total=   0.5s\n",
      "[CV] n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10 \n",
      "[CV]  n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10, total=   0.5s\n",
      "[CV] n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10 \n",
      "[CV]  n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10, total=   0.6s\n",
      "[CV] n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10 \n",
      "[CV]  n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10, total=   0.6s\n",
      "[CV] n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10 \n",
      "[CV]  n_estimators=500, min_samples_split=4, min_samples_leaf=2, max_features=sqrt, max_depth=10, total=   0.5s\n",
      "[CV] n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20, total=   1.1s\n",
      "[CV] n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20, total=   1.2s\n",
      "[CV] n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20, total=   1.1s\n",
      "[CV] n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20, total=   1.0s\n",
      "[CV] n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1000, min_samples_split=2, min_samples_leaf=4, max_features=sqrt, max_depth=20, total=   1.0s\n",
      "[CV] n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20 \n",
      "[CV]  n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20, total=   0.0s\n",
      "[CV] n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20 \n",
      "[CV]  n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20, total=   0.0s\n",
      "[CV] n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20 \n",
      "[CV]  n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20, total=   0.0s\n",
      "[CV] n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20 \n",
      "[CV]  n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20, total=   0.0s\n",
      "[CV] n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20 \n",
      "[CV]  n_estimators=10, min_samples_split=2, min_samples_leaf=2, max_features=auto, max_depth=20, total=   0.0s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20, total=   1.2s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20, total=   1.4s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20, total=   1.3s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20, total=   1.3s\n",
      "[CV] n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20 \n",
      "[CV]  n_estimators=1200, min_samples_split=2, min_samples_leaf=1, max_features=sqrt, max_depth=20, total=   1.3s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  50 out of  50 | elapsed:   29.7s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=5, error_score=nan,\n",
       "                   estimator=RandomForestClassifier(bootstrap=True,\n",
       "                                                    ccp_alpha=0.0,\n",
       "                                                    class_weight=None,\n",
       "                                                    criterion='gini',\n",
       "                                                    max_depth=None,\n",
       "                                                    max_features='auto',\n",
       "                                                    max_leaf_nodes=None,\n",
       "                                                    max_samples=None,\n",
       "                                                    min_impurity_decrease=0.0,\n",
       "                                                    min_impurity_split=None,\n",
       "                                                    min_samples_leaf=1,\n",
       "                                                    min_samples_split=2,\n",
       "                                                    min_weight_fraction_leaf=0.0,\n",
       "                                                    n_estimators=100, n_jobs...\n",
       "                                                    random_state=None,\n",
       "                                                    verbose=0,\n",
       "                                                    warm_start=False),\n",
       "                   iid='deprecated', n_iter=10, n_jobs=None,\n",
       "                   param_distributions={'max_depth': [None, 5, 10, 20, 30],\n",
       "                                        'max_features': ['auto', 'sqrt'],\n",
       "                                        'min_samples_leaf': [1, 2, 4],\n",
       "                                        'min_samples_split': [2, 4, 6],\n",
       "                                        'n_estimators': [10, 100, 200, 500,\n",
       "                                                         1000, 1200]},\n",
       "                   pre_dispatch='2*n_jobs', random_state=42, refit=True,\n",
       "                   return_train_score=False, scoring=None, verbose=2)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "# Setup the parameters grid\n",
    "grid = {\"n_estimators\": [10, 100, 200, 500, 1000, 1200],\n",
    "        \"max_depth\": [None, 5, 10, 20, 30],\n",
    "        \"max_features\": [\"auto\", \"sqrt\"],\n",
    "        \"min_samples_split\": [2, 4, 6],\n",
    "        \"min_samples_leaf\": [1, 2, 4]}\n",
    "\n",
    "# Instantiate RandomForestClassifier\n",
    "clf = RandomForestClassifier(n_jobs=1)\n",
    "\n",
    "# Setup RandomizedSearchCV\n",
    "rs_clf = RandomizedSearchCV(estimator=clf,\n",
    "                            param_distributions=grid, \n",
    "                            n_iter=10, # number of models to try\n",
    "                            cv=5,\n",
    "                            verbose=2,\n",
    "                            random_state=42, # set random_state to 42 for reproducibility\n",
    "                            refit=True) # set refit=True (default) to refit the best model on the full dataset \n",
    "\n",
    "# Fit the RandomizedSearchCV version of clf\n",
    "rs_clf.fit(X_train, y_train) # 'rs' is short for RandomizedSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': 100,\n",
       " 'min_samples_split': 2,\n",
       " 'min_samples_leaf': 4,\n",
       " 'max_features': 'auto',\n",
       " 'max_depth': 30}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check best parameters of RandomizedSearchCV\n",
    "rs_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 85.25%\n",
      "Precision: 0.85\n",
      "Recall: 0.88\n",
      "F1 score: 0.86\n"
     ]
    }
   ],
   "source": [
    "# Evaluate RandomizedSearch model\n",
    "rs_y_preds = rs_clf.predict(X_test)\n",
    "\n",
    "# Evaluate the classifier on validation set\n",
    "rs_metrics = evaluate_preds(y_test, rs_y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GridSearchCV\n",
    "Find best hyperparameters using [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html).\n",
    "\n",
    "**Note:** Although best parameters are found on different splits of `X_train` and `y_train`, because `refit=True`, once the best parameters are found, they are refit to the entire set of `X_train` and `y_train`. See the [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) and [cross-validation documentation](https://scikit-learn.org/stable/modules/cross_validation.html) for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 12 candidates, totalling 60 fits\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.6s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.6s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.6s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.6s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.6s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.6s\n",
      "[CV] max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=100, total=   0.1s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=200, total=   0.2s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.5s\n",
      "[CV] max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500 \n",
      "[CV]  max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=500, total=   0.5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  60 out of  60 | elapsed:   17.1s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score=nan,\n",
       "             estimator=RandomForestClassifier(bootstrap=True, ccp_alpha=0.0,\n",
       "                                              class_weight=None,\n",
       "                                              criterion='gini', max_depth=None,\n",
       "                                              max_features='auto',\n",
       "                                              max_leaf_nodes=None,\n",
       "                                              max_samples=None,\n",
       "                                              min_impurity_decrease=0.0,\n",
       "                                              min_impurity_split=None,\n",
       "                                              min_samples_leaf=1,\n",
       "                                              min_samples_split=2,\n",
       "                                              min_weight_fraction_leaf=0.0,\n",
       "                                              n_estimators=100, n_jobs=1,\n",
       "                                              oob_score=False,\n",
       "                                              random_state=None, verbose=0,\n",
       "                                              warm_start=False),\n",
       "             iid='deprecated', n_jobs=None,\n",
       "             param_grid={'max_depth': [None], 'max_features': ['auto', 'sqrt'],\n",
       "                         'min_samples_leaf': [1, 2], 'min_samples_split': [6],\n",
       "                         'n_estimators': [100, 200, 500]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=2)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# Setup grid-2 (refined version of grid)\n",
    "grid_2 = {'n_estimators': [100, 200, 500],\n",
    "          'max_depth': [None],\n",
    "          'max_features': ['auto', 'sqrt'],\n",
    "          'min_samples_split': [6],\n",
    "          'min_samples_leaf': [1, 2]}\n",
    "\n",
    "# Instantiate RandomForestClassifier\n",
    "clf = RandomForestClassifier(n_jobs=1)\n",
    "\n",
    "# Setup GridSearchCV\n",
    "gs_clf = GridSearchCV(estimator=clf,\n",
    "                      param_grid=grid_2, \n",
    "                      cv=5,\n",
    "                      verbose=2,\n",
    "                      refit=True) # set refit=True (default) to refit the best model on the full dataset\n",
    "\n",
    "# Fit the GridSearchCV version of clf\n",
    "gs_clf.fit(X_train, y_train) # 'gs' is short for GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': None,\n",
       " 'max_features': 'auto',\n",
       " 'min_samples_leaf': 2,\n",
       " 'min_samples_split': 6,\n",
       " 'n_estimators': 200}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find best parameters of GridSearchCV\n",
    "gs_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 85.25%\n",
      "Precision: 0.85\n",
      "Recall: 0.88\n",
      "F1 score: 0.86\n"
     ]
    }
   ],
   "source": [
    "# Evaluate GridSearchCV model\n",
    "gs_y_preds = gs_clf.predict(X_test)\n",
    "\n",
    "# Evaluate the classifier on validation set\n",
    "gs_metrics = evaluate_preds(y_test, gs_y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare metrics\n",
    "Compare all of the found metrics between the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAH1CAYAAADS7HuGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de7RddX3v/c+XcBGIAm2wWokEWrwQEiAkyE1BaUGONqjH5wgVhaKAWIRa9XlwnFOxiNYKx1qtglgLWkClWBVBK1VQhHpJIBgJl8pAbpUKKhcDBAz8nj/2TgwhkA2/nayV5PUaI8O95pp77u8Ki+2bOeeas1prAQDgqVlv0AMAAKzJxBQAQAcxBQDQQUwBAHQQUwAAHcQUAECH9Qf1gydNmtSmTJkyqB8PADBmV1xxxS9aa1uu6LmBxdSUKVMyd+7cQf14AIAxq6qbH+85h/kAADqIKQCADmIKAKCDmAIA6CCmAAA6iCkAgA5iCgCgg5gCAOggpgAAOogpAIAOYgoAoIOYAgDoIKYAADqIKQCADmIKAKCDmAIA6CCmAAA6iCkAgA7rD3oAAJ6C9262yjY9bZvnrrJtJ8mPD/3xKt0+rG72TAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0cAV0AFjbraFXzF9TrpZvzxQAQAcxBQDQQUwBAHRwztSqtoYep07WnGPVa5U19P3ivQKsy+yZAgDoIKYAADqIKQCADmIKAKCDE9CTTDn+wlW27Zuetso2zQCsyvdK4v2ytvG7hbHyu2XNZs8UAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0GFNMVdXLq+r6qrqhqo5fwfPPrapLqmpeVc2vqv8x/qMCAAyflcZUVU1I8vEkByTZPsnBVbX9cqv9nyTnttZ2TnJQkk+M96AAAMNoLHumdk1yQ2vtxtbaQ0k+n+TA5dZpSZ4x+vVmSX42fiMCAAyvscTUc5Lcuszj20aXLeu9SQ6pqtuSfC3J21a0oao6sqrmVtXcO++88ymMCwAwXMYSU7WCZW25xwcnObO1tlWS/5Hkn6vqMdturZ3eWpvZWpu55ZZbPvlpAQCGzFhi6rYkk5d5vFUeexjvTUnOTZLW2veSPC3JpPEYEABgmI0lpuYk2a6qtqmqDTNygvn5y61zS5J9k6SqXpiRmHIcDwBY6600plpri5Mck+QbSa7NyKf2FlTViVU1e3S1dyQ5oqp+lORzSQ5rrS1/KBAAYK2z/lhWaq19LSMnli+77D3LfH1Nkj3HdzQAgOHnCugAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3GFFNV9fKqur6qbqiq4x9nnf9VVddU1YKqOmd8xwQAGE7rr2yFqpqQ5ONJ/jjJbUnmVNX5rbVrlllnuyTvTrJna+2uqnrmqhoYAGCYjGXP1K5Jbmit3dhaeyjJ55McuNw6RyT5eGvtriRprd0xvmMCAAynscTUc5Lcuszj20aXLet5SZ5XVZdX1fer6uUr2lBVHVlVc6tq7p133vnUJgYAGCJjialawbK23OP1k2yXZJ8kByf5x6ra/DHf1NrprbWZrbWZW2655ZOdFQBg6Iwlpm5LMnmZx1sl+dkK1vlKa+03rbWfJrk+I3EFALBWG0tMzUmyXVVtU1UbJjkoyfnLrfPlJC9NkqqalJHDfjeO56AAAMNopTHVWluc5Jgk30hybZJzW2sLqurEqpo9uto3kvyyqq5JckmSd7XWfrmqhgYAGBYrvTRCkrTWvpbka8ste88yX7ckfzn6BwBgneEK6AAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcYUU1X18qq6vqpuqKrjn2C911ZVq6qZ4zciAMDwWmlMVdWEJB9PckCS7ZMcXFXbr2C9pyc5NskPxntIAIBhNZY9U7smuaG1dmNr7aEkn09y4ArWe1+SDyVZNI7zAQAMtbHE1HOS3LrM49tGly1VVTsnmdxau+CJNlRVR1bV3Kqae+eddz7pYQEAhs1YYqpWsKwtfbJqvSR/l+QdK9tQa+301trM1trMLbfccuxTAgAMqbHE1G1JJi/zeKskP1vm8dOT7JDk21V1U5LdkpzvJHQAYF0wlpiak2S7qtqmqjZMclCS85c82Vq7p7U2qbU2pbU2Jcn3k8xurc1dJRMDAAyRlcZUa21xkmOSfCPJtUnOba0tqKoTq2r2qh4QAGCYrT+WlVprX0vyteWWvedx1t2nfywAgDWDK6ADAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQYU0xV1cur6vqquqGqjl/B839ZVddU1fyq+lZVbT3+owIADJ+VxlRVTUjy8SQHJNk+ycFVtf1yq81LMrO1Nj3JeUk+NN6DAgAMo7Hsmdo1yQ2ttRtbaw8l+XySA5ddobV2SWvt/tGH30+y1fiOCQAwnMYSU89Jcusyj28bXfZ43pTk6yt6oqqOrKq5VTX3zjvvHPuUAABDaiwxVStY1la4YtUhSWYmOXlFz7fWTm+tzWytzdxyyy3HPiUAwJBafwzr3JZk8jKPt0rys+VXqqo/SvK/k+zdWntwfMYDABhuY9kzNSfJdlW1TVVtmOSgJOcvu0JV7Zzkk0lmt9buGP8xAQCG00pjqrW2OMkxSb6R5Nok57bWFlTViVU1e3S1k5NMTPIvVXVVVZ3/OJsDAFirjOUwX1prX0vyteWWvWeZr/9onOcCAFgjuAI6AEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHQQUwAAHcQUAEAHMQUA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdBBTAAAdxBQAQAcxBQDQQUwBAHRYf9ADAKvX0yc8PUc894hM3nhyKjUu27z22mvHZTtrm0/Nfva4baul5ea7f5OP/eCu3PvgI+O2XaCfmIJ1zBHPPSI7brVjNnz6hqkan5h64aQXjst21ja/ue3ucdtWay2/+7v35m1J3n/pL8dtu0A/h/lgHTN548njGlKsHlWV9Td5RrbefINBjwIsR0zBOqZSQmoNVVXjdmgWGD9iCgCgg3OmYB33ilNuGoet/HYbN33wFStf+6ab8spXvjJXX331OPzsR/v2t7+dU045JRdccEHOP//8XHPNNTn++OPH/ecALCGmgLXW7NmzM3v27EGPAazlHOYDBmLx4sU59NBDM3369Lz2ta/N/fffnxNPPDGzZs3KDjvskCOPPDKttSTJRz/60Wy//faZPn16DjrooCTJfffdl8MPPzyzZs3KzjvvnK985SuP+RlnnnlmjjnmmCTJYYcdlmOPPTZ77LFHtt1225x33nlL1zv55JMza9asTJ8+PSeccMJqePXA2kRMAQNx/fXX58gjj8z8+fPzjGc8I5/4xCdyzDHHZM6cObn66qvzwAMP5IILLkiSfPCDH8y8efMyf/78nHbaaUmS97///XnZy16WOXPm5JJLLsm73vWu3HfffU/4M2+//fZcdtllueCCC5Ye+rvooovyk5/8JD/84Q9z1VVX5Yorrsill166al88sFYRU8BATJ48OXvuuWeS5JBDDslll12WSy65JC960Ysybdq0XHzxxVmwYEGSZPr06Xn961+fs846K+uvP3J2wkUXXZQPfvCD2WmnnbLPPvtk0aJFueWWW57wZ77qVa/Keuutl+233z4///nPl27noosuys4775wZM2bkuuuuy09+8pNV+MqBtY1zpoCBWP7yDFWVt771rZk7d24mT56c9773vVm0aFGS5MILL8yll16a888/P+973/uyYMGCtNbyxS9+Mc9//vMftZ0lkbQiG2200dKvlxxCbK3l3e9+d4466qjxemnAOsaeKWAgbrnllnzve99Lknzuc5/LXnvtlSSZNGlSFi5cuPScpkceeSS33nprXvrSl+ZDH/pQ7r777ixcuDD7779/Pvaxjy2Nonnz5j2lOfbff//80z/9UxYuXJgk+a//+q/ccccdvS8PWIfYMwXruAvfOaV7G1MnTX3S3/PCF74wn/nMZ3LUUUdlu+22y9FHH5277ror06ZNy5QpUzJr1qwkycMPP5xDDjkk99xzT1prefvb357NN988f/VXf5W/+Iu/yPTp09Nay5QpU5aeY/Vk7Lfffrn22muz++67J0kmTpyYs846K8985jOf9LaAdZOYAla7KVOm5JprrnnM8pNOOiknnXTSY5Zfdtllj1m28cYb55Of/ORjlu+zzz7ZZ599kox8gu+www5LMvLJvmUt2ROVJMcdd1yOO+64J/EKAH7LYT4AgA5iCgCgg5gCAOggpgAAOogpAIAOYgoAoINLI8A6buo/7DG+G3zvPeO7vTGYMmVK5s6dm0mTJq32nz0e5nzvsnzmkx/LP5z5hUGPAjwF9kwBA9NayyOPPDLoMVabhx9+eNAjAKuAmAJWq5tuuikvfOEL89a3vjUzZszIrbfemqOPPjozZ87M1KlTc8IJJyxdd8qUKTnhhBMyY8aMTJs2Ldddd12S5Je//GX222+/7LzzzjnqqKOW3lImST784Q9nhx12yA477JCPfOQjS3/mC17wgrz5zW/ODjvskNe//vX55je/mT333DPbbbddfvjDHz5mzgULFmTXXXfNTjvtlOnTpy+9+fFZZ521dPlRRx21NJAe7zUcsPv0nPaRD+XQ17w8F13w5dzy0xtz5MGvyv+z31553QF759abfpokeeC++/KOow7Ngfvsmne/7YhHvSZguIkpYLW7/vrr88Y3vjHz5s3L1ltvnfe///2ZO3du5s+fn+985zuZP3/+0nUnTZqUK6+8MkcffXROOeWUJMlf//VfZ6+99sq8efMye/bs3HLLLUmSK664ImeccUZ+8IMf5Pvf/34+9alPLb1n3w033JDjjjsu8+fPz3XXXZdzzjknl112WU455ZR84AMfeMyMp512Wo477rhcddVVmTt3brbaaqtce+21+cIXvpDLL788V111VSZMmJCzzz47SZ7wNWy00Ub5zL/+Ww448H/m3ccemde98c35l4suy2e/9I1M+r3fS5Jct2B+/t/3fiBfuvj7ue2WmzNvzvdXzV8+MO7EFLDabb311tltt92WPj733HMzY8aM7LzzzlmwYMGjbjXzmte8Jkmyyy675KabbkqSXHrppTnkkEOSJK94xSuyxRZbJBm57cyrX/3qbLrpppk4cWJe85rX5Lvf/W6SZJtttsm0adOy3nrrZerUqdl3331TVZk2bdrS7S5r9913zwc+8IH87d/+bW6++eZsvPHG+da3vpUrrrgis2bNyk477ZRvfetbufHGG1f6Gvb/k1cnSe5b+Ovc8d+3Z98DXpkk2ehpT8vGG2+SJNlhp13ye89+TtZbb708f/sd8rPbbun+ewZWDyegA6vdpptuuvTrn/70pznllFMyZ86cbLHFFjnssMOyaNGipc9vtNFGSZIJEyZk8eLFS5dX1WO2+0SHxpZsJ0nWW2+9pY/XW2+9R213iT/90z/Ni170olx44YXZf//984//+I9preXQQw/N3/zN3zxq3ZW9ho032XSl822w4Ya/nW/ChDy82PlVsKawZwoYqHvvvTebbrppNttss/z85z/P17/+9ZV+z0te8pKlh9e+/vWv56677lq6/Mtf/nLuv//+3HffffnSl76UF7/4xU9prhtvvDHbbrttjj322MyePTvz58/Pvvvum/POOy933HFHkuRXv/pVbr755jG/holPf0Z+79m/n4v/7cIkyUMPPpgHHrj/Kc0HDA97pmAdt+CY/+jextRJU5/y9+64447ZeeedM3Xq1Gy77bbZc889V/o9J5xwQg4++ODMmDEje++9d5773OcmSWbMmJHDDjssu+66a5LkzW9+c3beeecVHsZbmS984Qs566yzssEGG+RZz3pW3vOe9+R3fud3ctJJJ2W//fbLI488kg022CAf//jHs9tuu435Nbz/70/L+45/ez7xfz+Q9TfYIKeceuaTng0YLjWoT4zMnDmzzZ07dyA/e3lTjr9wlW37pqf96Srb9rRtnrvKtp0kPz70x6t0+2uiVfleSVbP++Uj238kz9rmWeO67Z6YWpvNv+3ucd/mz2+5MUecf7vfLWuZteF3y6owTO+VqrqitTZzRc85zAcA0EFMAQB0EFMAAB3EFABABzEFANBBTAEAdHCdKVjHHXThQeO6vfH8KPNpp52WTTbZJG984xsftfymm27KK1/5ylx99dXj9rOerGGYARgOYgoYSosXL85b3vKWQY+RxYsXZ/31/aoEHp/fEMBq9773vS9nn312Jk+enEmTJmWXXXbJO9/5zuyzzz7ZY489cvnll2f27Nn59a9/nYkTJ+ad73xnrrjiihx++OHZZJNNstdee61wu7fffnte97rX5d57783ixYtz6qmn5sUvfnEuuuiinHDCCXnwwQfzB3/wBznjjDMyceLEnHjiifnqV7+aBx54IHvssUc++clPpqoeM8chhxySt7zlLUtvanzqqafm93//9/Pwww/niCOOyH/8x3/kOc95Tr7yla9k4403Xp1/lcAQcM4UsFrNnTs3X/ziFzNv3rz867/+a5a/E8Ldd9+d73znO3nHO97xqOV/9md/lo9+9KP53ve+97jbPuecc7L//vvnqquuyo9+9KPstNNO+cUvfpGTTjop3/zmN3PllVdm5syZ+fCHP5wkOeaYYzJnzpxcffXVeeCBB3LBBRescI5jjz02e++9d370ox/lyiuvzNSpI1d8/8lPfpI///M/z4IFC7L55pvni1/84nj9NQFrEHumgNXqsssuy4EHHrh0D86f/MmfPOr5173udY/5nnvuuSd333139t577yTJG97whhXeTHjWrFk5/PDD85vf/CavetWrstNOO+U73/lOrrnmmqX3y3vooYey++67J0kuueSSfOhDH8r999+fX/3qV5k6derSeZad4+KLL85nP/vZJMmECROy2Wab5a677so222yTnXbaKUmyyy67PKV7AAJrPjEFrFYrux/opptuusLvqaqVbvslL3lJLr300lx44YV5wxvekHe9613ZYost8sd//Mf53Oc+96h1Fy1alLe+9a2ZO3duJk+enPe+971ZtGjRE86xvI022mjp1xMmTMgDDzyw0u8B1j4O8wGr1V577ZWvfvWrWbRoURYuXJgLL1z5DV4333zzbLbZZrnsssuSJGefffYK17v55pvzzGc+M0cccUTe9KY35corr8xuu+2Wyy+/PDfccEOS5P77789//ud/Lg2nSZMmZeHChTnvvPMe9+fvu+++OfXUU5MkDz/8cO69994n9ZqBtZs9U7CO+/wrPt+9jamTpo553VmzZmX27NnZcccds/XWW2fmzJnZbLPNVvp9Z5xxxtIT0Pfff/8VrvPtb387J598cjbYYINMnDgxn/3sZ7PlllvmzDPPzMEHH5wHH3wwSXLSSSflec97Xo444ohMmzYtU6ZMyaxZsx73Z//93/99jjzyyHz605/OhAkTcuqpp+bZz372mF8zsHarle1yX1VmzpzZlj/xdFCmHL/y/zJ+qm562p+usm1P2+a5q2zbyfheL2htsSrfK8nqeb98ZPuP5FnbPGtct/1kYipJFi5cmIkTJ+b+++/PS17ykpx++umZMWPGuM40DObfdve4b/Pnt9yYI86/3e+Wtcza8LtlVRim90pVXdFam7mi5+yZAla7I488Mtdcc00WLVqUQw89dK0MKWDdIaaA1e6cc84Z9AgA48YJ6LCOaWkr/UQdw6m1lhb/7GDYiClYx9z6wK156NcPCao1TGsti++/Nzff/ZtBjwIsx2E+WMd86pZP5YgckckbT05l5dduGov17vTfZSvy87vG77pTLS033/2bfOwHd43bNoHxIaZgHfPrh3+dD//0w+O6zWH6xM0wOWAVf0ILGA5j+s/Jqnp5VV1fVTdU1fEreH6jqvrC6PM/qKop4z0oAMAwWmlMVdWEJB9PckCS7ZMcXFXbL7fam5Lc1Vr7wyR/l+Rvx3tQAIBhNJY9U7smuaG1dmNr7aEkn09y4HLrHJjkM6Nfn5dk3xrLjbQAANZwYzln6jlJbl3m8W1JXvR467TWFlfVPUl+N8kvll2pqo5McuTow4VVdf1TGXpN8hSKclKW+3t7fFc/+a0/CXWYHl7dnuTf+JN4rySr8v3ivbL6+d3Ck+F3y7jY+vGeGEtMreiVLP+Z6rGsk9ba6UlOH8PPXGdV1dzHu1w9LMt7hSfD+4Wx8l558sZymO+2JJOXebxVkp893jpVtX6SzZL8ajwGBAAYZmOJqTlJtquqbapqwyQHJTl/uXXOT3Lo6NevTXJxc0VAAGAdsNLDfKPnQB2T5BtJJiT5p9bagqo6Mcnc1tr5ST6d5J+r6oaM7JE6aFUOvZZzGJSx8l7hyfB+Yay8V56ksgMJAOCpcw8IAIAOYgoAoIOYAgDoIKYAADqM5aKdrEJV9cUk/5Tk6621RwY9D8OtqjZK8j+TTMky//621k4c1EzA2qmqJrbWFg56jjWBmBq8U5P8WZKPVtW/JDmztXbdgGdieH0lyT1Jrkjy4IBnYUhV1VezgrtQLNFam70ax2HNdU2S5w56iDWBmBqw1to3k3yzqjZLcnCSf6+qW5N8KslZrbXfDHRAhs1WrbWXD3oIht4pgx6ANUNV/eXjPZVk4uqcZU3mOlNDoKp+N8khSd6QkVv1nJ1kryTTWmv7DHA0hkxVnZ7kY621Hw96FmDNV1WLkpycZPEKnn57a23z1TzSGklMDVhV/WuSFyT554wc4rt9mefcbJJHqaprkvxhkp9m5DBfJWmttekDHYyhUlU/zhMf5vN+IUlSVf+R5G2ttStW8NytrbXJK/g2luMw3+D9Q2vt4hU9IaRYgQMGPQBrhFcOegDWGP+V5OaqOq619vfLPef/g8ZITA3eC6vqytba3UlSVVskObi19okBz8UQaq3dXFU7Jnnx6KLvttZ+NMiZGD6ttZsHPQNrjO2TbJrk8Kr6bEb2di/hnN0xcp2pwTtiSUglSWvtriRHDHAehlhVHZeRc+qeOfrnrKp622CnYlhV1W5VNaeqFlbVQ1X1cFXdO+i5GCqfTPJvGTnd5Irl/swd4FxrFOdMDVhVzU+yYxv9B1FVE5LMb61NHexkDKPR98vurbX7Rh9vmuR7zoFhRapqbpKDkvxLRg7ZvDHJH7bW/vdAB2PoVNWprbWjBz3HmsphvsH7RpJzq+q0jJww+paM/FcCrEgleXiZxw/n0bvl4VFaazdU1YTW2sNJzhg94RgeRUj1EVOD9/8lOSrJ0Rn5P8WLkvzjQCdimJ2R5AdV9aXRx69K8ukBzsNwu7+qNkxyVVV9KMntGTk/BhhHDvPBGqaqZmTkOmSV5NLW2rwBj8SQqqqtk/w8yYZJ3p5ksySfaK3dMNDBYC0jpgasqpzkUHcAAAWBSURBVLZL8jcZ+UTF05Ysb61tO7ChGDpV9YzW2r1V9Tsrer619qvVPRPDb/ScugeW3Pdz9JzMjVpr9w92Mli7+DTf4J2RkfvzLU7y0iSfzcgFPGFZ54z+75JP2Cz54xM3PJFvJdlkmccbJ/nmgGaBtZY9UwNWVVe01napqh+31qaNLvtua+3FK/tegCdSVVe11nZa2TKgjz1Tg7eoqtZL8pOqOqaqXp2R6wfBY1TVnqOHblJVh1TVh6vKXd15PPeNnmOXJKmqXZI8MMB5YK1kz9SAVdWsJNcm2TzJ+5I8I8nJrbXvD3QwhtKS65IlmZ6Rw8GfTvKa1treAx2MoTT6++XzGbmBepI8O8nrVnQfNuCpE1MDNHoy6Adba+8a9CysGUZvPTSjqt6T5L9aa59esmzQszGcqmqDJM/PyKc/r2utuUUIjDOH+QZo9CJ6u1SViy4yVr+uqncnOSTJhaNBvsGAZ2JIVdUmGbmW3XGttR8nmVJVboIM40xMDd68JF+pqjdU1WuW/Bn0UAyt1yV5MMmbWmv/neQ5SU4e7EgMsTOSPJRk99HHtyU5aXDjwNrJYb4Bq6ozVrC4tdYOX+3DAGuVqprbWptZVfNaazuPLvtRa23HQc8GaxO3kxmw1tqfDXoGhl9VXdZa26uqfp2RezgufSoj8f2MAY3GcHuoqjbO6Humqv4gI3s2gXFkz9SAje6Zesw/BHumgB6j52K+IcmbMnKHhYuS7JnksNbatwc4Gqx17JkavAuW+fppSV6d336MGR6lqnZLsqC19uvRxxOTTG2t/WCwkzFsWmutqo5Lsl+S3TKyF/O41tovBjsZrH3smRoyoxfw/GZr7WWDnoXhU1Xzksxoo//ijr5f5ro0AitSVR9PcmZrbc6gZ4G1mT1Tw2e7JK5ozeOptsx/AbXWHqkq/x7zeF6a5KiqujnJffntOXbTBzsWrF38Eh6wFZxQ/N8ZuS4MrMiNVXVsRm6OnSRvTXLjAOdhuB0w6AFgXeAwH6xBquqZST6a5GUZifBvJfmL1todAx0MYB0mpgZs9MbGF7fW7hl9vHmSfVprXx7sZADAWLgC+uCdsCSkkqS1dneSEwY4D0Osqp5XVd+qqqtHH0+vqv8z6LkA1mViavBW9M/AuWw8nk8leXeS3yRJa21+koMGOhHAOk5MDd7cqvpwVf1BVW1bVX+X5IpBD8XQ2qS19sPlli0eyCQAJBFTw+BtGbkR6ReSnJvkgSR/PtCJGGa/GL0lyJLrTL02ye2DHQlg3eYEdFiDVNW2SU5PskeSu5L8NMnrW2s3D3QwgHWYPVMDVlX/PvoJviWPt6iqbwxyJobT6NXOZ7bW/ijJlkle0FrbS0gBDJaYGrxJo5/gS5K01u5K8swBzsOQaq09kuSY0a/vW3J/PgAGS0wN3iNVtfT2MVU1JY++Ijos69+r6p1VNbmqfmfJn0EPBbAuc87UgFXVyzNyDsx3Rhe9JMmRrTWH+niMqvppVhDbrbVtBzAOABFTQ2H0FiFHJrkqydOS3NFau3SwUzGMqmrjjNyPb6+MRNV3k5zWWntgoIMBrMPE1IBV1ZuTHJdkq4zE1G5Jvtdae9lAB2MoVdW5Se5NcvboooOTbN5a+1+Dmwpg3eZK24N3XJJZSb7fWntpVb0gyV8PeCaG1/Nbazsu8/iSqvrRwKYBwAnoQ2BRa21RklTVRq2165I8f8AzMbzmVdVuSx5U1YuSXD7AeQDWefZMDd5to9eZ+nJGPql1V5KfDXgmhteLkryxqm4ZffzcJNdW1Y+TtNba9MGNBrBucs7UEKmqvZNsluTfWmsPDXoehk9Vbf1Ez7uAJ8DqJ6YAADo4ZwoAoIOYAgDoIKYAADqIKQCADmIKAKDD/w+BR4gqoHmzpwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "compare_metrics = pd.DataFrame({\"baseline\": baseline_metrics,\n",
    "                                \"random search\": rs_metrics,\n",
    "                                \"grid search\": gs_metrics})\n",
    "\n",
    "compare_metrics.plot.bar(figsize=(10, 8));"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}