Modeling_Final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Modeling "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In this notebook, we move forward with our cleaned data and start our preprocessing and modeling \n",
    "\n",
    "\n",
    "After cleaning our data, and determining what preprocessing steps we want to integrate into our code via pipelines, we then create our baseline models. Through an iterative process we work through better performing modeling types before using a combination of high performing models. In evaluating these models, we kept in mind that simply guessing the waterpoint is functioning each time will give you a model that performs with 54% accuracy. As we began modeling, it became very clear that non-parametric models were the best fit for our data, as they better tolerate a large proportion of categorical features. As we iterated through the models, this assumption was backed up by our results. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Importing the relevant libraries\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from category_encoders.count import CountEncoder\n",
    "from category_encoders.one_hot import OneHotEncoder\n",
    "from category_encoders.cat_boost import CatBoostEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.experimental import enable_iterative_imputer\n",
    "from sklearn.impute import IterativeImputer\n",
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "from sklearn.model_selection import cross_validate\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score\n",
    "from sklearn.metrics import roc_auc_score, plot_confusion_matrix, plot_roc_curve\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# settings to display all columns\n",
    "pd.set_option(\"display.max_columns\", None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/data_cleaned.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look again at our final cleaned dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>non_functional</th>\n",
       "      <th>amount_tsh</th>\n",
       "      <th>funder</th>\n",
       "      <th>gps_height</th>\n",
       "      <th>installer</th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>basin</th>\n",
       "      <th>subvillage</th>\n",
       "      <th>region</th>\n",
       "      <th>lga</th>\n",
       "      <th>ward</th>\n",
       "      <th>population</th>\n",
       "      <th>public_meeting</th>\n",
       "      <th>scheme_management</th>\n",
       "      <th>permit</th>\n",
       "      <th>construction_year</th>\n",
       "      <th>extraction_type</th>\n",
       "      <th>management</th>\n",
       "      <th>payment</th>\n",
       "      <th>water_quality</th>\n",
       "      <th>quantity</th>\n",
       "      <th>source</th>\n",
       "      <th>waterpoint_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>69572</td>\n",
       "      <td>0</td>\n",
       "      <td>6000.0</td>\n",
       "      <td>Roman</td>\n",
       "      <td>1390</td>\n",
       "      <td>Roman</td>\n",
       "      <td>34.938093</td>\n",
       "      <td>-9.856322</td>\n",
       "      <td>Lake Nyasa</td>\n",
       "      <td>Mnyusi B</td>\n",
       "      <td>Iringa</td>\n",
       "      <td>Ludewa</td>\n",
       "      <td>Mundindi</td>\n",
       "      <td>109</td>\n",
       "      <td>True</td>\n",
       "      <td>VWC</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>gravity</td>\n",
       "      <td>vwc</td>\n",
       "      <td>pay annually</td>\n",
       "      <td>soft</td>\n",
       "      <td>enough</td>\n",
       "      <td>spring</td>\n",
       "      <td>communal standpipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8776</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Grumeti</td>\n",
       "      <td>1399</td>\n",
       "      <td>GRUMETI</td>\n",
       "      <td>34.698766</td>\n",
       "      <td>-2.147466</td>\n",
       "      <td>Lake Victoria</td>\n",
       "      <td>Nyamara</td>\n",
       "      <td>Mara</td>\n",
       "      <td>Serengeti</td>\n",
       "      <td>Natta</td>\n",
       "      <td>280</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Other</td>\n",
       "      <td>True</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>gravity</td>\n",
       "      <td>wug</td>\n",
       "      <td>never pay</td>\n",
       "      <td>soft</td>\n",
       "      <td>insufficient</td>\n",
       "      <td>rainwater harvesting</td>\n",
       "      <td>communal standpipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>34310</td>\n",
       "      <td>0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>Lottery Club</td>\n",
       "      <td>686</td>\n",
       "      <td>World vision</td>\n",
       "      <td>37.460664</td>\n",
       "      <td>-3.821329</td>\n",
       "      <td>Pangani</td>\n",
       "      <td>Majengo</td>\n",
       "      <td>Manyara</td>\n",
       "      <td>Simanjiro</td>\n",
       "      <td>Ngorika</td>\n",
       "      <td>250</td>\n",
       "      <td>True</td>\n",
       "      <td>VWC</td>\n",
       "      <td>True</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>gravity</td>\n",
       "      <td>vwc</td>\n",
       "      <td>pay per bucket</td>\n",
       "      <td>soft</td>\n",
       "      <td>enough</td>\n",
       "      <td>dam</td>\n",
       "      <td>communal standpipe multiple</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>67743</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Unicef</td>\n",
       "      <td>263</td>\n",
       "      <td>UNICEF</td>\n",
       "      <td>38.486161</td>\n",
       "      <td>-11.155298</td>\n",
       "      <td>Ruvuma / Southern Coast</td>\n",
       "      <td>Mahakamani</td>\n",
       "      <td>Mtwara</td>\n",
       "      <td>Nanyumbu</td>\n",
       "      <td>Nanyumbu</td>\n",
       "      <td>58</td>\n",
       "      <td>True</td>\n",
       "      <td>VWC</td>\n",
       "      <td>True</td>\n",
       "      <td>1986.0</td>\n",
       "      <td>submersible</td>\n",
       "      <td>vwc</td>\n",
       "      <td>never pay</td>\n",
       "      <td>soft</td>\n",
       "      <td>dry</td>\n",
       "      <td>machine dbh</td>\n",
       "      <td>communal standpipe multiple</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>19728</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Action In A</td>\n",
       "      <td>0</td>\n",
       "      <td>Artisan</td>\n",
       "      <td>31.130847</td>\n",
       "      <td>-1.825359</td>\n",
       "      <td>Lake Victoria</td>\n",
       "      <td>Kyanyamisa</td>\n",
       "      <td>Kagera</td>\n",
       "      <td>Karagwe</td>\n",
       "      <td>Nyakasimbi</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>gravity</td>\n",
       "      <td>other</td>\n",
       "      <td>never pay</td>\n",
       "      <td>soft</td>\n",
       "      <td>seasonal</td>\n",
       "      <td>rainwater harvesting</td>\n",
       "      <td>communal standpipe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  non_functional  amount_tsh        funder  gps_height     installer  \\\n",
       "0  69572               0      6000.0         Roman        1390         Roman   \n",
       "1   8776               0         0.0       Grumeti        1399       GRUMETI   \n",
       "2  34310               0        25.0  Lottery Club         686  World vision   \n",
       "3  67743               1         0.0        Unicef         263        UNICEF   \n",
       "4  19728               0         0.0   Action In A           0       Artisan   \n",
       "\n",
       "   longitude   latitude                    basin  subvillage   region  \\\n",
       "0  34.938093  -9.856322               Lake Nyasa    Mnyusi B   Iringa   \n",
       "1  34.698766  -2.147466            Lake Victoria     Nyamara     Mara   \n",
       "2  37.460664  -3.821329                  Pangani     Majengo  Manyara   \n",
       "3  38.486161 -11.155298  Ruvuma / Southern Coast  Mahakamani   Mtwara   \n",
       "4  31.130847  -1.825359            Lake Victoria  Kyanyamisa   Kagera   \n",
       "\n",
       "         lga        ward  population public_meeting scheme_management  \\\n",
       "0     Ludewa    Mundindi         109           True               VWC   \n",
       "1  Serengeti       Natta         280        Unknown             Other   \n",
       "2  Simanjiro     Ngorika         250           True               VWC   \n",
       "3   Nanyumbu    Nanyumbu          58           True               VWC   \n",
       "4    Karagwe  Nyakasimbi           0           True           Unknown   \n",
       "\n",
       "    permit  construction_year extraction_type management         payment  \\\n",
       "0  Unknown             1999.0         gravity        vwc    pay annually   \n",
       "1     True             2010.0         gravity        wug       never pay   \n",
       "2     True             2009.0         gravity        vwc  pay per bucket   \n",
       "3     True             1986.0     submersible        vwc       never pay   \n",
       "4     True                NaN         gravity      other       never pay   \n",
       "\n",
       "  water_quality      quantity                source  \\\n",
       "0          soft        enough                spring   \n",
       "1          soft  insufficient  rainwater harvesting   \n",
       "2          soft        enough                   dam   \n",
       "3          soft           dry           machine dbh   \n",
       "4          soft      seasonal  rainwater harvesting   \n",
       "\n",
       "               waterpoint_type  \n",
       "0           communal standpipe  \n",
       "1           communal standpipe  \n",
       "2  communal standpipe multiple  \n",
       "3  communal standpipe multiple  \n",
       "4           communal standpipe  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling \n",
    "\n",
    "Because of the large number of categorical features in our dataset, we will choose model types that have high functionality with categories. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we start with selecting our X and y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(columns='non_functional')\n",
    "y = df['non_functional']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For preprocessing purposes, we have to split our features into three different groups. One group holds our numerical data. The next deals with categorical columns that have less than 25 unique values. Finally, out last column deals with any categorical columns that have 25 or more unique values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Lists to hold column names\n",
    "num_cols = []\n",
    "sm_cat_cols = []\n",
    "lg_cat_cols = []\n",
    "\n",
    "for c in X.columns:\n",
    "    if X[c].dtype in ['float64', 'int64']:\n",
    "        num_cols.append(c)\n",
    "    elif X[c].nunique() < 25:\n",
    "        sm_cat_cols.append(c)\n",
    "    else:\n",
    "        lg_cat_cols.append(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our deliverable to our stakeholders is the final model. With this in consideration, we decided to separate off a holdout set of values before modeling to use on our best model. After this first separation we proceed with a traditional train test split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#First holdout set\n",
    "X_train_big, X_holdout, y_train_big, y_holdout = train_test_split(X, y,\n",
    "                                                          test_size = .10, random_state = 396)\n",
    "#Performing train test split on the remaining data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_train_big, y_train_big,\n",
    "                                                    test_size = .25, random_state = 396)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(estimator, X_tr, X_te, y_tr, y_te, cv=5, grid_search=False):\n",
    "    '''\n",
    "Function takes in estimator, training data, test data, a Boolean value indicating if the estimator is a grid search, \n",
    "and the cross validation splitting strategy if the estimator is not a grid search, \n",
    "and returns the accuracy, precision, recall, f1, and the ROC-AUC scores for the model \n",
    "and a confusion matrix visualization.  \n",
    "\n",
    "If ‘grid_search parameter is set to “True” then the function will not perform cross validation on the model. \n",
    "Based off of Lindsey Berlin’s evaluate function found at: \n",
    "https://github.com/lindseyberlin/Cat-in-the-Dat-Project/blob/main/notebooks/Lindsey/EDA-Initial-Models.ipynb\n",
    "------------------------------------------------------------------------------------------\n",
    "Inputs: \n",
    "-Estimator - Estimator object  \n",
    "-X_tr – X_train dataframe\n",
    "-X_te – X_test dataframe\n",
    "-Y_tr – y_train dataframe\n",
    "-Y_te – y_test dataframe\n",
    "-Cv – If cross_val  set to true this determines the cross-validation splitting strategy.  \n",
    "        Takes in all value options for sklearn.model_selection_cross_val_score “cv” parameter:\n",
    "        - None, to use the default 5-fold cross validation,\n",
    "        - int, to specify the number of folds in a (Stratified)KFold,\n",
    "        - CV splitter,\n",
    "        - An iterable yielding (train, test) splits as arrays of indices\n",
    "- grid_search – “ Boolean indicating whether a the estimator is a GridSearchCV object, \n",
    "        if set to “False” a cross validation will be performed with the number of iterations set by the “cv” parameter.  \n",
    "        Default value is “False”.\n",
    "\n",
    "Returns – nothing is returned \n",
    "\n",
    "\n",
    "    '''\n",
    "    #If no grid search is being performed, go through evaluation steps as normal, including cross validation\n",
    "    if grid_search == False:\n",
    "        #Cross-Validate\n",
    "        output = cross_validate(estimator, X_tr, y_tr, cv=cv,\n",
    "                                scoring=['accuracy', 'precision','recall', 'f1', 'roc_auc'])\n",
    "        #Printing out the mean of all of our evaluating metrics across the cross validation. \n",
    "        #Accuracy, precisionc recall, f1, and roc auc\n",
    "        print('Results of Cross-Validation:\\n')\n",
    "        print(f'Average accuracy: {output[\"test_accuracy\"].mean()}\\\n",
    "        +/- {output[\"test_accuracy\"].std()}')\n",
    "        print(f'Average precision: {output[\"test_precision\"].mean()}\\\n",
    "        +/- {output[\"test_precision\"].std()}')\n",
    "        print(f'**Average recall: {output[\"test_recall\"].mean()}\\\n",
    "        +/- {output[\"test_recall\"].std()}')\n",
    "        print(f'Average f1 score: {output[\"test_f1\"].mean()}\\\n",
    "        +/- {output[\"test_f1\"].std()}')\n",
    "        print(f'Average roc_auc: {output[\"test_roc_auc\"].mean()}\\\n",
    "        +/- {output[\"test_roc_auc\"].std()}\\n')\n",
    "        print('+'*20)\n",
    "    \n",
    "        \n",
    "        #Fitting the estimator to our X and y train data\n",
    "        estimator.fit(X_tr, y_tr)\n",
    "        #getting predictions for X train\n",
    "        tr_preds = estimator.predict(X_tr)\n",
    "        #getting predictions for X test\n",
    "        te_preds = estimator.predict(X_te)\n",
    "        \n",
    "        #Creating a confusion matrix from our data with custom labels\n",
    "        categories = ['Functional', 'Non-Functional']\n",
    "        print('\\nResults of Train-Test Split Validation:')\n",
    "        plot_confusion_matrix(estimator, X_te, y_te, display_labels= categories,\n",
    "                              cmap='mako')\n",
    "        \n",
    "        #Printing our final evaluating metrics across X train\n",
    "        #Evaluating using accuracy, precision, recall, f1, roc auc\n",
    "        print(\"\\nTraining Scores:\")\n",
    "        print(f\"Train accuracy: {accuracy_score(y_tr, tr_preds)}\")\n",
    "        print(f\"Train precision: {precision_score(y_tr, tr_preds)}\")\n",
    "        print(f\"**Train recall: {recall_score(y_tr, tr_preds)}\")\n",
    "        print(f\"Train f1 score: {f1_score(y_tr, tr_preds)}\")\n",
    "        print(f\"Train roc_auc: {roc_auc_score(y_tr, tr_preds)}\\n\")\n",
    "        print(\"<>\"*10)\n",
    "        #Printing our final evaluating metrics across X test\n",
    "        #Evaluating using accuracy, precision, recall, f1, roc auc\n",
    "        print(\"\\nTesting Scores:\")\n",
    "        print(f\"Test accuracy: {accuracy_score(y_te, te_preds)}\")\n",
    "        print(f\"Test precision: {precision_score(y_te, te_preds)}\")\n",
    "        print(f\"**Test recall: {recall_score(y_te, te_preds)}\")\n",
    "        print(f\"Test f1 score: {f1_score(y_te, te_preds)}\")\n",
    "        print(f\"Test roc_auc: {roc_auc_score(y_te, te_preds)}\")\n",
    "    \n",
    "    #If a grid search is being performed, do not perform a cross validate.\n",
    "    else:\n",
    "        #Fitting the estimator to our X and y train data\n",
    "        estimator.fit(X_tr, y_tr)\n",
    "        #getting predictions for X train\n",
    "        tr_preds = estimator.predict(X_tr)\n",
    "        #getting predictions for X test\n",
    "        te_preds = estimator.predict(X_te)\n",
    "        \n",
    "        #Creating a confusion matrix from our data with custom labels\n",
    "        categories = ['Functional', 'Non-Functional']\n",
    "        print('\\nResults of Train-Test Split Validation:')\n",
    "        plot_confusion_matrix(estimator, X_te, y_te, display_labels= categories,\n",
    "                              cmap='mako')\n",
    "        \n",
    "        #Printing our final evaluating metrics across X train \n",
    "        #Evaluating using accuracy, precision, recall, f1, roc auc\n",
    "        print(\"\\nTraining Scores:\")\n",
    "        print(f\"Train accuracy: {accuracy_score(y_tr, tr_preds)}\")\n",
    "        print(f\"Train precision: {precision_score(y_tr, tr_preds)}\")\n",
    "        print(f\"**Train recall: {recall_score(y_tr, tr_preds)}\")\n",
    "        print(f\"Train f1 score: {f1_score(y_tr, tr_preds)}\")\n",
    "        print(f\"Train roc_auc: {roc_auc_score(y_tr, tr_preds)}\\n\")\n",
    "        print(\"<>\"*10)\n",
    "        \n",
    "        #Printing our final evaluating metrics across X test\n",
    "        #Evaluating using accuracy, precision, recall, f1, roc auc\n",
    "        print(\"\\nTesting Scores:\")\n",
    "        print(f\"Test accuracy: {accuracy_score(y_te, te_preds)}\")\n",
    "        print(f\"Test precision: {precision_score(y_te, te_preds)}\")\n",
    "        print(f\"**Test recall: {recall_score(y_te, te_preds)}\")\n",
    "        print(f\"Test f1 score: {f1_score(y_te, te_preds)}\")\n",
    "        print(f\"Test roc_auc: {roc_auc_score(y_te, te_preds)}\")\n",
    "        \n",
    "        print('\\nGrid Search Results (you animal):\\n')\n",
    "        return pd.DataFrame(estimator.cv_results_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing\n",
    "\n",
    "Our preprocessing involved splitting our data into three different categories; numerical, small categorical, and large categorical. The preprossing steps for each group was as follows: \n",
    "\n",
    "- **Numeric Columns**\n",
    "Our numerical columns were first run through an Iterative Imputer to deal with missing values by modeling each feature with missing values as a function of other features. We then use a MinMaxScaler on all numeric data to better scale our non normal distributions. \n",
    "\n",
    "- **Small Categorical Columns**\n",
    "Our small categorical columns were run through OneHotEncoder to split each unique value into its own binary column. Columns in this category were selected due to having less than 25 unique values in their column.\n",
    "\n",
    "- **Large Categorical Columns**\n",
    "Our large categorical columns were run through a categorical encoder called Count Encoder. This was done because of their large number of unique values. The Count Encoder keeps a single column and reduces the categories in the column to their value counts. \n",
    "\n",
    "- **All Categorical Columns**\n",
    "Alternatively we explored the use of [catboost encoding](https://contrib.scikit-learn.org/category_encoders/catboost.html) for all of our categorical encoders. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first step in our modeling process is to set up some pipelines to deal with our data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating Pipelines for each column type:\n",
    "# numeric, categorical with under 25 categories, categorical with over 25 categories\n",
    "\n",
    "\n",
    "num_col_pipe = Pipeline(steps = [\n",
    "    ('ii', IterativeImputer(random_state=396, add_indicator=True)),\n",
    "    ('mm', MinMaxScaler())\n",
    "])\n",
    "\n",
    "sm_cat_col_pipe = Pipeline(steps = [\n",
    "    ('ohe', OneHotEncoder(handle_unknown = 'indicator'))\n",
    "])\n",
    "\n",
    "lg_cat_col_pipe = Pipeline(steps = [\n",
    "    ('count', CountEncoder(min_group_size=10, min_group_name='Other'))\n",
    "])\n",
    "\n",
    "#We created category that would use catboost for all categorical features for further evaluation.\n",
    "\n",
    "all_cat_col_pipe = Pipeline(steps = [\n",
    "    ('catboost', CatBoostEncoder(random_state=396))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our next step is to create preprocessors that will easily streamline the handling of our data for use in future pipelines. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Preprocessor1 makes a distinction between our small and large categorical columns \n",
    "preprocessor1 = ColumnTransformer(transformers=[\n",
    "    ('numeric', num_col_pipe, num_cols),\n",
    "    ('small_cat', sm_cat_col_pipe, sm_cat_cols),\n",
    "    ('large_cat', lg_cat_col_pipe, lg_cat_cols )\n",
    "])\n",
    "#Preprocessor2 takes in all categorical features no matter the size\n",
    "# and handles them using CatBoostEncoder\n",
    "preprocessor2 = ColumnTransformer(transformers=[\n",
    "    ('numeric', num_col_pipe, num_cols),\n",
    "    ('cat', all_cat_col_pipe, [*sm_cat_cols, *lg_cat_cols]),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline Evaluation\n",
    "\n",
    "To get an initial understanding of what a modelless prediction looks like, we used DummyClassifier. Our most frequent value in our predictor column is `functioning`. If we were to predict functioning on our test dataset, we would be right around 54% of the time. Any model we create moving forward should outperform this baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of Cross-Validation:\n",
      "\n",
      "Average accuracy: 0.5431494883953081        +/- 4.9912652857475586e-05\n",
      "Average precision: 0.0        +/- 0.0\n",
      "**Average recall: 0.0        +/- 0.0\n",
      "Average f1 score: 0.0        +/- 0.0\n",
      "Average roc_auc: 0.5        +/- 0.0\n",
      "\n",
      "++++++++++++++++++++\n",
      "\n",
      "Results of Train-Test Split Validation:\n",
      "\n",
      "Training Scores:\n",
      "Train accuracy: 0.5431494883953082\n",
      "Train precision: 0.0\n",
      "**Train recall: 0.0\n",
      "Train f1 score: 0.0\n",
      "Train roc_auc: 0.5\n",
      "\n",
      "<><><><><><><><><><>\n",
      "\n",
      "Testing Scores:\n",
      "Test accuracy: 0.5414389458710789\n",
      "Test precision: 0.0\n",
      "**Test recall: 0.0\n",
      "Test f1 score: 0.0\n",
      "Test roc_auc: 0.5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnfklEQVR4nO3de5xVdb3/8debARFNVEQmBBRU0hQVlUiznzdMsTIpL9GpvOQJU0vzpCc8drwVZplhnhSjNNEuRiqJF7ycSbzfQFAE5YiigiCIVt6RGT6/P9Z3YIF79t7j7GFmD++nj/XYa33XWt/13bNxf/b6fr/r+1VEYGZm1qmtC2BmZu2DA4KZmQEOCGZmljggmJkZ4IBgZmZJ57YugDWtR88e0Wfrfm1dDGuGOTNmtXURrJlWEssiYsuW5LHf5w6IN15/o+RxT8946s6IGN6Sa7UmB4R2rM/W/Zh8/+1tXQxrhl02HdDWRbBmeqdhxUstzeON199g8gNTSh637cZ9erb0Wq3JAcHMrMWClbGyrQvRYg4IZmYtFED9yoa2LkaLuVHZzKylIrtDKLWUImkHSTNzy5uSvi+ph6S7JT2XXjfPnXOWpHmS5ko6JJe+p6RZad9lklTq+g4IZmYtFMDKMv4rmU/E3IgYHBGDgT2Bd4FJwGigLiIGAnVpG0k7ASOBnYHhwBWSalJ244BRwMC0lGzMdkAwM6uAlREll2YaBjwfES8BhwMTUvoEYERaPxy4PiKWR8R8YB4wVFJvoHtEPBzZgHXX5s5pktsQzMxarOxG5Z6SpuW2x0fE+CaOHQn8Oa3XRsRigIhYLKlXSu8DPJI7Z2FKW5HW104vygHBzKyFAsoNCMsiYkipgyRtAHwJOKvUoU0Up6n0ohwQzMxaKCKoj4r2MjoUeCIilqTtJZJ6p7uD3sDSlL4QyD+92hdYlNL7Fkgvym0IZmYVUIleRjlfY3V1EcBk4Ni0fixwcy59pKSukgaQNR4/lqqX3pK0V+pddEzunCb5DsHMrAI+QqNxQZI2Aj4HnJhLvgiYKOkE4GXgKICImC1pIjAHqAdOiVh1q3IScA3QDZiSlqIcEMzMWigq+KRyRLwLbLFW2utkvY4KHT8GGFMgfRowqDnXdkAwM6sAD11hZmat0ajcJhwQzMxaKIAG3yGYmRlUrlG5LTkgmJm1WJQ1VlF754BgZtZC2ZPKvkMwMzPchmBmZjT2MnJAMDNb77mXkZmZreI2BDMzy+4QSo8u3e45IJiZtdhHmhGt3XFAMDNroZXABx66wszMCLchmJkZ2fDXDQ4IZmYGblQ2MzMah65o61K0nAOCmVkFuMrIzMwI4AM/qWxmZhFuVDYzMxqfVK5+ndq6AGZmHUFDuksotpRD0maSbpD0rKRnJO0tqYekuyU9l143zx1/lqR5kuZKOiSXvqekWWnfZZJU6toOCGZmLZSNdlqZgAD8CrgjInYEdgOeAUYDdRExEKhL20jaCRgJ7AwMB66QVJPyGQeMAgamZXipCzsgmJlVQEOUXkqR1B3YF7gKICI+iIh/AocDE9JhE4ARaf1w4PqIWB4R84F5wFBJvYHuEfFwRARwbe6cJrkNwcyshbKxjMq6A+gpaVpue3xEjM9tbwu8Bvxe0m7AdOA0oDYiFgNExGJJvdLxfYBHcucvTGkr0vra6UU5IJiZtVSU/RzCsogYUmR/Z2AP4HsR8aikX5Gqh5pQqF0giqQX5SojM7MWauxlVGopw0JgYUQ8mrZvIAsQS1I1EOl1ae74frnz+wKLUnrfAulFOSCYmVVAJdoQIuJVYIGkHVLSMGAOMBk4NqUdC9yc1icDIyV1lTSArPH4sVS99JakvVLvomNy5zTJVUZmZi2U9TKqWHbfA/4oaQPgBeB4sh/vEyWdALwMHAUQEbMlTSQLGvXAKRGrJmY4CbgG6AZMSUtRDghmZi0UwIoKjVwRETOBQu0Mw5o4fgwwpkD6NGBQc67tgGBm1kIeusLMzFbx8NdmZgaIlVFyZIh2zwHBzKyFOsrgdg4IZmYV4CojMzMjAuqrf34cBwQzs5bynMpmZrZKB5hB0wHBKmP+c89z5vHfW7W98KUFnHLW6Sxd/CpT76ijS5cu9BuwDT++/GK6b9adWdNncv5p/wVkfbhPHv19hh12CO+9+x4/OO4UFsx/iZqaGvYbPozTz/thW70tA/Y5aD9++LNzqKmp4aYJf+GqsePaukjtUke4Q6jasYwkNUiamVv6VzDvEWniicbtCyQdVKn8U577S7q1knm2pQEDt+OGB27nhgdu5y/33sKG3TZk2BcPZu8DPsukh+/kpofuYJvtB/C7sVcAsP0nd+D6qZO54YHbufLGCVxw+tnU19cDcNx3v80tj9fx1/tuZeaj07j/7qlt+M7Wb506deLsSy7g5COO4/BPfY5Dj/wS2+6wfVsXq92JKG9p76r5DuG9iBjcSnmPAG4lGx+EiDinla7TIT1674P0G7ANW23dl622Xj3g4m5DdueuydlwKt026rYqffn7y1cN1ttto24M3XdvALpssAGf3HUQSxYtXneFtzXsMmQwL7/wEgtfXADAlBtv4YAvHMwLc+e1ccnaH98htDOSXpTUM60PkTQ1rZ8n6WpJUyW9IOnU3DnHSHpK0pOSrpP0GeBLwMXpzmM7SddIOjIdP0zSjDRX6dWSuuaufb6kJ9K+HVP6UEkPpXMeyo1i2GFNufFWDj3isA+lT/rDRD570H6rtp+aNoMRex3MV/YZzjm/HEPnzmv+Pnnzn28y9Y46Pr3fPq1eZiusV+9aXl24etTkJYsWU7tVbRuWqP1aubL00t5Vc0DolqsumlTG8TsChwBDgXMldZG0M3A2cGBE7AacFhEPkQ0pe2ZEDI6I5xszkLQh2eiBX42IXcjusE7KXWNZROxBNpfpGSntWWDfiNgdOAe4sFghJY2SNE3StDeWvV7G22pfVnzwAVOn/C8Hj/j8Gunjf/Frajp35otHj1iVtuuQ3fnbI3dx/d9v5ndjr8juFJL6+nr+899P5esnHke//luvq+LbWgrNyx7VUPexjjX2Miq1tHfVHBDeS1/YgyPiy2Ucf1uad3QZ2eQStcCBwA0pjYh4o0QeOwDzI+L/0vYEsvlPG92UXqcD/dP6psBfJT0NjCWbDLtJETE+IoZExJAePbco4221L/ffPZVP7rYzPXttuSrt5j/dyL13/p2LfntpwS+YbXfYnm4bbcS8Z+auSjv/tP9im237882Tv7VOym2FLVn0Kh/vu9Wq7dqterN08dIiZ6ynOkgbQjUHhELqWf2eNlxr3/LcegPZr3tRxrRyOaUGK2m8RmP+AD8G7omIQcBhBcrVoUy58RYOPeJLq7Yf+N97ufpXV/I/f/7tGu0GC19csKoRedHLC3lx3gur2hsu+8kvePvNt/jhRW66aWtPT3+SbbbtT59t+tK5SxcOPeIwpt5+d1sXq13qCAGhmhuVC3kR2JNsIogjyji+DpgkaWxEvC6pR7pLeAvYpMDxzwL9JW0fEfOAbwL3lrjGpsAraf24MspUtd579z0evucBzhm7emj2C888lw8++IBRI74JwK6f2p1zxo5hxiOPc9WlV9K5c+esJ8svfszmW/Tg1VcW89tfXM6AT2zH0ft+EYCvjTqGI44Z2SbvaX3X0NDAhWeew5WTrqWmpoZJ103k+Wefa+titUvV8IVfSkcLCOcDV0n6L+DRUgen2YbGAPdKagBmkH1pXw/8NjU+H5k7/n1Jx5NVAXUGHgeuLHGZnwMTJP0H8PeP8J6qRreNuvHA/BlrpN0+Y2rBYw8b+RUOG/mVD6V/vE9vZv1zfmsUzz6i+++ayv13TW3rYrRrQXU0GpciNxC1X7vssVtMvv/2ti6GNcMumw5o6yJYM73TsGJ6RBSaoaxs3T+xfQy9/Oclj6s7+IgWX6s1dbQ7BDOzNtERfls7IJiZVYADgpmZEXSMgNDRup2ama17FXwOIY16MCs9dDstpfWQdLek59Lr5rnjz5I0T9JcSYfk0vdM+cyTdJkKPQS0FgcEM7MKiJWll2Y4ID1029gAPRqoi4iBZN3lRwOkQThHkj3wOhy4QlJNOmccMAoYmJbhpS7qgGBmVgGt/GDa4WQjI5BeR+TSr0+jMMwH5gFDJfUGukfEw5F1Jb02d06THBDMzCqgzIDQs3GssrSMKpQVcJek6bn9tRGxOLtOLAZ6pfQ+wILcuQtTWp+0vnZ6UW5UNjNrqaDcQXCWlfEcwj4RsUhSL+BuSc8WObZQu0AUSS/KdwhmZi0UVK4NISIWpdelwCSyEZqXpGog0mvjCIMLgX650/sCi1J63wLpRTkgmJlVQCXaECRtLGmTxnXgYOBpsiH5j02HHQvcnNYnAyMldZU0gKzx+LFUrfSWpL1S76Jjcuc0yVVGZmYtFUFUZsKDWrIBNyH7fv5TRNwh6XFgoqQTgJeBo7LLxmxJE8lmd6wHTomIhpTXSWTzt3QjG/BzSqmLOyCYmVVCBeJBRLwA7FYg/XVgWBPnjAHGFEifBgxqzvUdEMzMKqAjPKnsgGBmVgkOCGZmBr5DMDMzyO4OGkoe1e45IJiZtVA22mn13yI4IJiZVUL1xwMHBDOzFit/6Ip2zQHBzKwCOkCNkQOCmVlFOCCYmRkBVGboijblgGBmVgnNmxGtXXJAMDOrhOq/QXBAMDOrBDcqm5lZx+92Kul/KPIWI+LUVimRmVk16uCNytPWWSnMzKpd9ceDpgNCREzIb0vaOCLeaf0imZlVoQ4QEErOqSxpb0lzgGfS9m6Srmj1kpmZVYvG5xBKLe1cyYAAXAocArwOEBFPAvu2YpnMzKpPlLG0c2X1MoqIBWnS50YdYORvM7MKqoIv/FLKCQgLJH0GCEkbAKeSqo/MzIzsIYQqqBIqpZwqo+8ApwB9gFeAwWnbzMwaVbDKSFKNpBmSbk3bPSTdLem59Lp57tizJM2TNFfSIbn0PSXNSvsu01rVPIWUDAgRsSwivh4RtRGxZUR8IyJeL/+tmZmtByrbhnAaa9bEjAbqImIgUJe2kbQTMBLYGRgOXCGpJp0zDhgFDEzL8FIXLaeX0baSbpH0mqSlkm6WtG3578vMbD1QoYAgqS/wBeB3ueTDgcZHASYAI3Lp10fE8oiYD8wDhkrqDXSPiIcjm9vz2tw5TSqnyuhPwESgN7AV8Ffgz2WcZ2a2/ogovZTnUuA/WXP81NqIWJxdJhYDvVJ6H2BB7riFKa1PWl87vahyAoIi4rqIqE/LH+gQ7elmZhUSZF/fpRboKWlabhmVz0bSF4GlETG9zCsXaheIIulFFRvLqEdavUfSaOD6lOFXgdtKl9PMbD1S3nwIyyJiSJH9+wBfkvR5YEOgu6Q/AEsk9Y6Ixak6aGk6fiHQL3d+X2BRSu9bIL2oYncI08nGM/oqcCJwDzAVOAk4vlTGZmbrlQpUGUXEWRHRNyL6kzUW/z0ivgFMBo5Nhx0L3JzWJwMjJXWVNICs8fixVK30lqS9Uu+iY3LnNKnYWEYDSpbezMzWxZPIFwETJZ0AvAwcBRARsyVNBOYA9cApEdH44PBJwDVAN2BKWooq60llSYOAnchuYUgFubbcd2Jm1uFVOCBExFSyWhlSV/9hTRw3BhhTIH0aMKg51ywZECSdC+xPFhBuBw4FHiDrxmRmZtAhutqU08voSLLI9GpEHA/sBnRt1VKZmVWTABqi9NLOlVNl9F5ErJRUL6k7Weu2H0wzM8tr/9/3JZUTEKZJ2gz4LVnPo7eBx1qzUGZm1UbrQ0CIiJPT6pWS7iB7HPqp1i2WmVmVKf9J5Har2INpexTbFxFPtE6RzMyqUPXHg6J3CJcU2RfAgRUui63ljQ/e4Q8LprV1McyslCqZEa2UYg+mHbAuC2JmVr06xgQ5ZT2YZmZmJVR/PHBAMDNrscbRTqucA4KZWSV0gF5G5cyYJknfkHRO2t5a0tDWL5qZWRWp7BSabaKcoSuuAPYGvpa23wIub7USmZlVow4QEMqpMvp0ROwhaQZARPxD0gatXC4zs+oRoPWkl9EKSTWk+CZpSzpE84mZWQVVfzwoq8roMmAS0EvSGLKhry9s1VKZmVWb9aHKKCL+KGk62RDYAkZExDOtXjIzs2rSAepNypkgZ2vgXeCWfFpEvNyaBTMzqxpVcgdQSjltCLeRvVWRTaE5AJgL7NyK5TIzqy7rQ6NyROyS306joJ7YaiUyM6tG1R8Pmv+kckQ8IelTrVEYM7NqJNaTCXIk/UdusxOwB/Baq5XIzKzaRGVGO5W0IXAf2bz1nYEbIuJcST2AvwD9gReBoyPiH+mcs4ATgAbg1Ii4M6XvCVwDdANuB06LKD6+RjndTjfJLV3J2hQOb86bNDPr8CrT7XQ5cGBE7AYMBoZL2gsYDdRFxECgLm0jaSdgJFmb7nDgivTcGMA4YBQwMC3DS1286B1CyvhjEXFmWW/FzGx9VYEqo/QL/u202SUtQfYjfP+UPgGYCvwwpV8fEcuB+ZLmAUMlvUg23fHDAJKuBUYAU4pdv8k7BEmdI6KBrIrIzMyakoauKLWUQ1KNpJnAUuDuiHgUqI2IxQDptVc6vA+wIHf6wpTWJ62vnV5UsTuEx8iCwUxJk4G/Au807oyIm0plbma23ijv+76npPy8uOMjYvwa2WQ/xAdL2gyYJGlQkfzUREmaSi+qnF5GPYDXyeZQbrxQAA4IZmaNyrsDWBYRQ8o5MCL+KWkqWd3/Ekm9I2KxpN5kdw+Q/fLvlzutL7AopfctkF5UsUblXqmH0dPArPQ6O70+Xc4bMjNbXyhKLyXzkLZMdwZI6gYcBDwLTAaOTYcdC9yc1icDIyV1lTSArPH4sVSt9JakvSQJOCZ3TpOK3SHUAB/jI956mJmtNyo3dEVvYELq0NMJmBgRt0p6GJgo6QTgZeAogIiYLWkiMAeoB05JVU4AJ7G62+kUSjQoQ/GAsDgiLvho78nMbP2iCkyhGRFPAbsXSH+dbIDRQueMAcYUSJ8GFGt/+JBiAaHQnYGZmRXSUP0VJ8UCQsFoZGZmaymzjaC9azIgRMQb67IgZmbVK7LhK6pcswe3MzOzAhwQzMxsvRnt1MzMSghQQ/XPoemAYGZWCa4yMjMzcJWRmZlBdnfgOwQzMwMqMmNaW3NAMDOrgEoMXdHWHBDMzFoqAlY2lD6unXNAMDOrAN8hmJlZxgHBzMwACD+YZmZmHtzOzMwgTZHZ4EZlMzMjXGVkZmaJq4zMzIzAdwhmZgauMjIzs9U6QJVRp7YugJlZ9QuI+tJLCZL6SbpH0jOSZks6LaX3kHS3pOfS6+a5c86SNE/SXEmH5NL3lDQr7btMkkpd3wHBzKyFAohYWXIpQz3wg4j4JLAXcIqknYDRQF1EDATq0jZp30hgZ2A4cIWkmpTXOGAUMDAtw0td3AHBzKylsohQeimVTcTiiHgirb8FPAP0AQ4HJqTDJgAj0vrhwPURsTwi5gPzgKGSegPdI+LhiAjg2tw5TXIbgplZi5XdqNxT0rTc9viIGF/oQEn9gd2BR4HaiFgMWdCQ1Csd1gd4JHfawpS2Iq2vnV6UA4JVzPtvvsOU88ax7LkFIPH5C07irSVv8MC4ibz+wisc8+ef0nvn7QCY/9CT3HvpH2lYUU9Nl84c8INvss2ndwHgT8efyzvL/kHnrhsAcPRv/puNt9i0zd7X+m6fg/bjhz87h5qaGm6a8BeuGjuurYvUPpUXEJZFxJBSB0n6GHAj8P2IeLNI9X+hHVEkvahWCwiSAvhlRPwgbZ8BfCwizqtA3ucB3wZeS0l3RMToluab8t4M+LeIuCJtbwVcFhFHViL/3HWmAmdExLRSx1aLup/9nm332Z0v//IMGlasYMV7H9C1+8Z8eewZ3HnBmj+CNtq8O0f8ejSb9OrBa8+9zMTv/IRT6lYf88WLTlsVPKztdOrUibMvuYBRh3+DV195leunTuae2+/mhbnz2rpo7U+Fup1K6kIWDP4YETel5CWSeqe7g97A0pS+EOiXO70vsCil9y2QXlRrtiEsB74iqWcr5T82IganpSLBINkMOLlxIyIWVToYdETL336XBdPnsOtXDgSgpksXNuy+MT237csWAz58p1r7yQFs0qsHAD2370f98hXUf7BinZbZSttlyGBefuElFr64gPoVK5hy4y0c8IWD27pY7U8EsbKh5FJK6gl0FfBMRPwyt2sycGxaPxa4OZc+UlJXSQPIGo8fS9VLb0naK+V5TO6cJrVmQKgHxgOnr71D0jaS6iQ9lV63TunXpO5RD0l6QVLZX8SSjpP069z2rZL2T+tvSxoj6UlJj0iqTem1kial9CclfQa4CNhO0kxJF0vqL+npdPyGkn6funLNkHRA7to3SbojdQv7ea4c4yRNS13Izm/+n7E6/HPhEjbavDu3/+hyfn/UmUw5dxwfvPt+WefOvfsRanccQOcNuqxKu/1Hl/P7I8/gwStvIDpA/+5q1at3La8uXP3DcsmixdRuVduGJWrPVpaxlLQP8E3gwPQdNFPS58m+lz4n6Tngc2mbiJgNTATmAHcAp0REY+Q5CfgdWUPz88CUUhdv7TaEy4Gn8l+Qya+BayNigqRvAZexugW8N/BZYEey6HdDE3mfLukbaf2HJcqxMfBIRJydyvJt4CfpuvdGxJdTV62PkXXnGhQRg2FVw06jUwAiYhdJOwJ3SfpE2jeYrAFoOTBX0v9ExALg7Ih4I+VfJ2nXiHiqqYJKGkXWVYxNt9qyxNtqP1Y2rOTVZ+Zz0FknsNWuA/nfi67mkav+xr7fG1n0vNfmLeDesX/k6PE/WpV22EWnskntFix/5z3+dvovmH3LfQz60n6t/RasgEJ11w7QhUS53UqL5xLxAIXr/wGGNXHOGGBMgfRpwKDmXL9Vu51GxJtk3Z1OXWvX3sCf0vp1ZAGg0d8iYmVEzAGK/RTJVxndWaIoHwC3pvXpQP+0fiBZX10ioiEi/lUin8+m8hIRzwIvAY0BoS4i/hUR75NF621S+tGSngBmkPUV3qnYBSJifEQMiYghG2/evURx2o9NanuwSe0WbLXrQAB2+NzeLHnmhaLnvPnq60z6/sV84cLvsnm/j+fy2gKArht3Y6fPf5bFs55rvYJbUUsWvcrH+261art2q94sXby0yBnrsVhZemnn1sVzCJcCJ5D9Sm9K/ifH8ty6AFJ1z0xJM4vkUc+a72fD3PqKWP2zpoGPfmdU7Em/fLkbgM6pTu8MYFhE7Arctla5OoyP9dyc7h/fgtfnvwLAS4/Ooud2fZs8/v033+GGU37Kfqf9G31333FV+sr6Bt79x5sANKyoZ9590+k5cOvWLbw16enpT7LNtv3ps01fOnfpwqFHHMbU2+9u62K1Q9EhAkKrdztN1SUTyYLC1Sn5IbKn664Dvg48UCKPs4GzS1zqReBkSZ3I+tsOLaN4dWT1bJemKp2NgbeATZo4/r5U3r+nqqKtgbnAHk0c3x14B/hXarc4FJhaRrmq0kFnfYtbR19Gw4p6Nutby+d/fDL/V/cod194Ne/9401uOPmn9NqxP1/9zY944s938M8Fr/LQb27god9ktYJH/+a/6dKtKxNP/Akr6xtYuXIl/ffahd2OKHinbOtAQ0MDF555DldOupaamhomXTeR55/1HduHBGU1Grd36+o5hEuA7+a2TwWulnQmWdfR4ytwjQeB+cAs4GngiTLOOQ0YL+kEsl/1J0XEw5IeTA3JU8jaQRpdAVwpaRbZHclxEbG8qT7CEfGkpBnAbOCFVMYOq3bHARz7l5+tkfaJYZ/mE8M+/aFjP3PiEXzmxCMK5nPcxLWbnKwt3X/XVO6/a2pbF6OdC6K8RuN2TW4gar/6DNo+vnPDJW1dDGuGnw06qq2LYM30TsOK6eU8LFbMJhv1jME7HFbyuAdmXtPia7UmP6lsZtZC2VBG1f/j2gHBzKzFPEGOmZk1ckAwMzMiWP2AcPVyQDAzq4BKPKnc1hwQzMwqwAHBzMzIGpXdy8jMzPAdgpmZAdmTCA4IZmbrvQhY6bGMzMzMD6aZmdkqHWFwOwcEM7NKcC8jMzOr1BSabc0BwcyshdyobGZmScfodrou5lQ2M+vwIlaWXMoh6WpJS9OsjY1pPSTdLem59Lp5bt9ZkuZJmivpkFz6npJmpX2XqampHXMcEMzMKiAiSi5lugYYvlbaaKAuIgaSzQU/GkDSTmTz0++czrkizQ8PMA4YBQxMy9p5fogDgplZi0XF7hAi4j7gjbWSDwcmpPUJwIhc+vURsTwi5gPzgKGSegPdI+LhyCLRtblzmuQ2BDOziijrC7+npGm57fERMb6M82ojYjFARCyW1Cul9wEeyR23MKWtSOtrpxflgGBm1kIRUW4vo2URMaSCly7ULhBF0otylZGZWQVEGf+1wJJUDUR6XZrSFwL9csf1BRal9L4F0otyQDAzq4BKtSE0YTJwbFo/Frg5lz5SUldJA8gajx9L1UtvSdor9S46JndOk1xlZGbWYpV7UlnSn4H9ydobFgLnAhcBEyWdALwMHAUQEbMlTQTmAPXAKbF6cueTyHosdQOmpKUoBwQzs0qo0FhGEfG1JnYNa+L4McCYAunTgEHNubYDgplZBXi0UzMza04vo3bNAcHMrAJ8h2BmZgDNGZqi3XJAMDNrMc+HYGZmZI8A+w7BzMwggpXhRmUzMwNXGZmZWcZVRmZmloau8x2CmZnhOwQzM0vchmBmZqmXkQOCmdl6L3sOwQHBzMxwG4KZmQEeusLMzFbxHYKZmUG4DcHMzMgeTPNYRmZmBrjKyMzMko5QZdSprQtgZlb9oqz/yiFpuKS5kuZJGt3KBV+DA4KZWQs1TpBTailFUg1wOXAosBPwNUk7tW7pV3OVkZlZSwU0VKZReSgwLyJeAJB0PXA4MKcSmZeijtAQ0lFJeg14qa3L0Up6AsvauhBWto78eW0TEVu2JANJd5D9jUrZEHg/tz0+Isbn8jkSGB4R/562vwl8OiK+25Lylct3CO1YS/+RtmeSpkXEkLYuh5XHn1dxETG8QlmpUPYVyrsktyGYmbUfC4F+ue2+wKJ1dXEHBDOz9uNxYKCkAZI2AEYCk9fVxV1lZG1lfOlDrB3x57UORES9pO8CdwI1wNURMXtdXd+NymZmBrjKyMzMEgcEMzMDHBCsAEkNkmbmlv4VzHtE/slLSRdIOqhS+ac895d0ayXzXNckhaRLcttnSDqvQnmfJ+mV3Od7USXyTXlvJunk3PZWkm6oVP65fKdKcjfYCnOjshXyXkQMbqW8RwC3kp68jIhzWuk61W458BVJP42I1nggbGxE/KIV8t0MOBm4AiAiFgFHtsJ1rBX4DsHKIulFST3T+hBJU9P6eZKuTr/YXpB0au6cYyQ9JelJSddJ+gzwJeDi9Mt0O0nXpKczkTRM0gxJs1KeXXPXPl/SE2nfjil9qKSH0jkPSdphHf9ZWlM9Wc+e09feIWkbSXXpb1snaeuUfo2ky9Lf4oXGv2s5JB0n6de57Vsl7Z/W35Y0Jn2Oj0iqTem1kial9CfT53sRsF36fC+W1F/S0+n4DSX9Pn2GMyQdkLv2TZLukPScpJ/nyjFO0jRJsyWd3/w/ozWHA4IV0i1XnTCpjON3BA4hG4flXEldJO0MnA0cGBG7AadFxENkfarPjIjBEfF8YwaSNgSuAb4aEbuQ3b2elLvGsojYAxgHnJHSngX2jYjdgXOAC1vwntujy4GvS9p0rfRfA9dGxK7AH4HLcvt6A58Fvkj25dyU03Of8SElyrEx8Ej6HO8Dvp3SLwPuTel7ALOB0cDz6fM9c618TgFIn+/XgAnpcwcYDHwV2AX4qqTGh7POTk9I7wrsJ2nXEmW1FnBAsELeS/9DD46IL5dx/G0RsTxVbSwFaoEDgRsaqzsi4o0SeewAzI+I/0vbE4B9c/tvSq/Tgf5pfVPgr+kX6Fhg5zLKWjUi4k3gWuDUtXbtDfwprV9HFgAa/S0iVkbEHLLPoSljc5/xnSWK8gFZNR+s+fc/kCxAExENEfGvEvl8NpWXiHiWbJyuT6R9dRHxr4h4n6w6cZuUfrSkJ4AZZJ/vOhv5c33kgGDlqmf1v5cN19q3PLfeQPbrXjRvDJZCY7gUukZj/gA/Bu6JiEHAYQXK1RFcCpxA9iu9Kfm/c/6zEECq7pkpaWaRPPKfL6z5t1wRqx9Yyv/9m6vYZ/yhf0OSBpDdDQ5Ld0O30TE/43bDAcHK9SKwZ1o/oozj68h+3W0BIKlHSn8L2KTA8c8C/SVtn7a/Cdxb4hqbAq+k9ePKKFPVSXdWE8mCQqOHyIY0APg68ECJPM5uvBsoctiLwGBJnVJ1zdAyildHqtaTVCOpO01/vpBVN309Hf8JYGtgbpH8uwPvAP9K7RaHllEmawEHBCvX+cCvJN1P9guuqPS4/RjgXklPAr9Mu64HzkyNitvljn8fOJ6sCmgWsBK4ssRlfg78VNKDZI/5d1SXsObQyqcCx0t6iixwnlaBazwIzAdmAb8AnijjnNOAA9LnNR3YOSJeBx6U9LSki9c6/gqgJh3/F+C4iFhOEyLiSbKqotnA1amM1oo8dIWZmQG+QzAzs8QBwczMAAcEMzNLHBDMzAxwQDAzs8QBwaqaVo/M+rSkv0raqAV55cdV+p1yo7IWOHb/NHZPc6+xakyoctLXOubtZl7rPElnlD7SLOOAYNWucZiNQWRDLHwnv1PSR3o+ISL+PQ3/0JT9gWYHBLP2zAHBOpL7ge3Tr/d7JP0JmJWeor1Y0uNphNATAZT5taQ5km4DejVmpNx4+5KGKxtp9Ullo4v2Jws8jQPE/T9JW0q6MV3jcUn7pHO3kHRXehDvN5QeogNJf5M0PY3wOWqtfZekstRJ2jKlbZdGCp0u6X6l0WDNmsvzIViHIKkz2dAGd6SkocCgiJifvlT/FRGfUjak9oOS7gJ2JxtUbxeygeDmkD0Rm893S+C3ZKOqzpfUIyLekHQl8HbjnAIp+IyNiAeUDUd9J/BJ4FzggYi4QNIXgDW+4JvwrXSNbsDjkm5MTwBvDDwRET+QdE7K+7tkw2R/JyKek/RpsieCD/wIf0ZbzzkgWLXrlhu07X7gKrKqnMciYn5KPxjYVavnB9gUGEg2muqfI6IBWCTp7wXy3wu4rzGvIqO2HgTsJK26AeguaZN0ja+kc2+T9I8y3tOpkhpHme2Xyvo62XAef0npfwBukvSx9H7/mrt21zKuYfYhDghW7T40u1v6YnwnnwR8b+1hniV9ntIjspY7amsnYO+IeK9AWcoeH0bZpDQHpbzeVTYRUVMjfEa67j9bcYY7W4+4DcHWB3cCJ0nqAtlIm5I2Jht9c2RqY+gNHFDg3IfJJmYZkM5tatTWu8iqb0jHDU6r+RE+DwU2L1HWTYF/pGCwI9kdSqNOrJ6O8t/IqqLeBOZLOipdQ5J2K3ENs4IcEGx98Duy9oEnlE2m8xuyu+NJwHNkI3yOo8Bw2xHxGlm9/01p1NbGKptbgC83NiqTjUA6JDVaz2F1b6fzgX2VTfJyMPByibLeQTYXwFNk8z08ktv3DrCzpOlkbQQXpPSvAyek8s0GDi/jb2L2IR7t1MzMAN8hmJlZ4oBgZmaAA4KZmSUOCGZmBjggmJlZ4oBgZmaAA4KZmSX/H8fRsy8MM2YJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Pipeline for preprocessing and modeling\n",
    "baseline_evaluation = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor1),\n",
    "    ('classifier', DummyClassifier(strategy='most_frequent'))\n",
    "])\n",
    "\n",
    "#Putting our model through our evaluate function\n",
    "evaluate(baseline_evaluation, X_train, X_test, y_train, y_test, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results**\n",
    "\n",
    "- After evaluating our baseline model, we see that it has an accuracy of 54% on the testing data. All other evaluating metric scores are 0 and our roc_auc shows a value of .5, confirming this is a poor model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parametric vs. Non-parametric Modeling\n",
    "\n",
    "As we move forward in our modeling, we want to make an initial evaluation of parametric and non-parametric models against one another to provide direction for future modeling decisions. We will compare a logistic regresison model to a decision tree model. We expect the non-parametric decision tree to perform better because it is well equiped to handle categorical data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression Model\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of Cross-Validation:\n",
      "\n",
      "Average accuracy: 0.7174943848265535        +/- 0.015325501044653429\n",
      "Average precision: 0.7358609891647862        +/- 0.01680253180110356\n",
      "**Average recall: 0.5948851690908241        +/- 0.025143502775855472\n",
      "Average f1 score: 0.6578082563573079        +/- 0.02160212307885155\n",
      "Average roc_auc: 0.7809795185127753        +/- 0.015509342705686022\n",
      "\n",
      "++++++++++++++++++++\n",
      "\n",
      "Results of Train-Test Split Validation:\n",
      "\n",
      "Training Scores:\n",
      "Train accuracy: 0.5673072123783379\n",
      "Train precision: 0.5509473684210526\n",
      "**Train recall: 0.28591718562220036\n",
      "Train f1 score: 0.3764655110407825\n",
      "Train roc_auc: 0.544952711539275\n",
      "\n",
      "<><><><><><><><><><>\n",
      "\n",
      "Testing Scores:\n",
      "Test accuracy: 0.5666691622370292\n",
      "Test precision: 0.5527386541471049\n",
      "**Test recall: 0.2883265306122449\n",
      "Test f1 score: 0.3789699570815451\n",
      "Test roc_auc: 0.5453662520317862\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsp0lEQVR4nO3debxVVf3/8dfbiwKKVyUEEVAQUXNEJZxzTNEs0SwpU0sLNafMIc1vhvpD/ZpTpFJW5lB+FQeSDHAgMQeUAFFAMVFQEWRSmTSUez+/P/a6eLzee87Bcy534P3ssR/ss/bea69zj53P2Wvt/VmKCMzMzNZp7AaYmVnT4IBgZmaAA4KZmSUOCGZmBjggmJlZ0qqxG2D1a9+hfXTZoltjN8NWw8svTGnsJthqqiYWRsSmpdSx/9cOjPcWvVdwv6kvvPRIRPQr5VwNyQGhCeuyRTdGPDWysZthq2GnjXo0dhNsNS2v+uTNUut4b9F7jHh6VMH9ttqgS4dSz9WQHBDMzEoWVEd1YzeiZA4IZmYlCmBldVVjN6NkDghmZqUKXyGYmRnZFUI1DghmZgZUt4C8cA4IZmYlc5eRmZmRuowcEMzMLCJYGb7LyMzM8BWCmZklLWFQ2cntzMxKFGlQudBSDEmzJE2RNFnShFQ2SNI7qWyypCNy9r9Y0gxJr0o6LKd891TPDElDJKnQuX2FYGZWBmXuMjowIhbWKrshIq7NLZC0PTAA2AHYHHhc0jYRUQUMBQYCzwEjgX5A3oRLvkIwMytRzaByoaUBHAXcExErImImMAPoK6kzUBkR4yIigDuB/oUqc0AwMytRAFVRXXABOkiakLMMrKe6RyVNrLX9TEkvSbpN0iaprAvwds4+s1NZl7ReuzwvdxmZmZVBkYPKCyOiT4F99omIOZI6Ao9Jmk7W/XMFWbC4ArgOOBmoa1wg8pTn5SsEM7OSBdVF/K+omiLmpH/nA8OBvhExLyKqIqIa+APQN+0+G8idRasrMCeVd62jPC8HBDOzEmVPKkfBpRBJG0jasGYdOBSYmsYEahwNTE3rI4ABklpL6gH0AsZHxFxgqaQ9091FJwIPFTq/u4zMzMqgqjx3GXUChqc7RFsBd0fEaEl3SepNFntmAacCRMQ0ScOAl4GVwBnpDiOA04HbgbZkdxcVnNLNAcHMrETZXUalB4SIeAPYpY7yE/IcMxgYXEf5BGDH1Tm/A4KZWYlq7jJq7hwQzMzKoCWkrnBAMDMrUQBVhe/qbPIcEMzMSlbcXURNnQOCmVmJqoGPPR+CmZkRHkMwMzOy9NdVDghmZgYeVDYzM2pSVzR2K0rngGBmVgbuMjIzMwL42E8qm5lZhAeVzcyMmieVmz8HBDOzMvAVgpmZpWynDghmZgZUNf944Ck0zcxKleUyioJLMSTNkjRF0mRJE1JZe0mPSXot/btJzv4XS5oh6VVJh+WU757qmSFpSJpKMy8HBDOzUkXWZVRoWQ0HRkTviOiTXl8EjImIXsCY9BpJ2wMDgB2AfsAtkirSMUOBgWTzLPdK2/NyQDAzK1HNXUaFlhIcBdyR1u8A+ueU3xMRKyJiJjAD6CupM1AZEeMiIoA7c46plwOCmVkZVEXhBeggaULOMrCOqgJ4VNLEnO2dImIuQPq3YyrvArydc+zsVNYlrdcuz8uDymZmJcruMipq14U53UD12Sci5kjqCDwmaXqefesaF4g85Xk5IJiZlSiAT8qUuSIi5qR/50saDvQF5knqHBFzU3fQ/LT7bKBbzuFdgTmpvGsd5Xm5y8jMrERRxIByMYPKkjaQtGHNOnAoMBUYAZyUdjsJeCitjwAGSGotqQfZ4PH41K20VNKe6e6iE3OOqZevEMzMyqBM6a87AcPTHaKtgLsjYrSkfwPDJJ0CvAV8GyAipkkaBrwMrATOiFg1l+fpwO1AW2BUWvJyQDAzK5mojoK3+RcUEW8Au9RRvgg4uJ5jBgOD6yifAOy4Oud3QDAzK5GT25mZ2SqeMc3MzIiAlc1/fhwHBDOzUnlOZTMzW6UFzKDp5xCsfA7baV+O3rsfx+57BMcd8E0Apr/0MscfcvSqsikTJ6/a/4/X38IRux7AN/ocxDNjnlxVftq3TuJb+xxO/z0P5fJzL6GqqiUM1zU9l998DWNfn8CDzz3yuW0nnfVjpiyZxcbts6Saex24L/c++XceHDeae5/8O32/uteqfQ875kgeeHYUw59/lHMvv2iNtb+pqY7CS1PXbAOCpKqUHrZm6V7GuvunLII1ry+XdEi56k91HiDp4XLW2RTc9ve7uf/pkdw7dgQA1//qKk77+Tnc//RIzvjFuVx/6dUAvD79NUY98Hf+9twjDL3/Dv7feZeu+uK/9s838cAzoxg+7hHeX/gej/5tZKO9n5bsob/ez+nHnPS58k5dOrPXQfsx561PU+G8v+h9zjzuFI7Zqx+XnHYeV956AwAbtd+Y8664mB9943iO3uNQvtRxU/bYf+819h6aiojilqau2QYE4KOUHrZmmVXGuvsDqwJCRFwaEY+Xsf61hiSWL10GwLIlS9m0cycAnhj5GId/6xus17o1Xbt3Y4uttmTKxBcBaFe5IQArV67kk48/pog07vYFTHx2PIvfX/y58guv+iXX//Kqz3yBTX9pGgvezbIlzHjlP7Ru05p111uPrt234M0ZM3l/0XsAPDf2aQ456vA10v6mxlcITUyaWKJDWu8jaWxaHyTpNkljJb0h6eycY06U9JKkFyXdJWlv4JvAr9OVR09Jt0s6Nu1/sKQX0sQTt0lqnXPuyyRNStu2S+V9JT2bjnlW0rZr+M+yxkji1KNP5Dv7f4P7br8bgJ9fdSnXXXoVh+ywN9f98kp+eukFAMyb+y6dunRedWynzTszf+67q16fesyJ7L91H9bfsB1fW0u/YBrDAYcfwvy58/jP1Ffq3edrRx3O9Ben8cnHH/P2G7PosU1PNt+iKxUVFRz09UPZLOdzXZtUVxdemrrmPKjcVtLktD4zIo4usP92wIHAhsCrkoYC2wCXkGUXXCipfUS8J2kE8HBE3A+s+oUqqQ3Zo+AHR8R/JN1J9nj4jekcCyNiN0k/Ac4HfgRMB74aEStTt9OVwLfqa2RKdzsQYPNuBbPVNil3PnI/HTt3YtGChQzsfwI9evXksYdGceHg/+FrRx3O6OEPc+lZF/HHh/5C1HH9rJwEjb9/8E5W/HcFF/34pzz/r2fZ+8D91uRbWSu1aduGH19wJqf2P6HefXpu14tzL7+IgWmfJR8s4Ypz/4df334TUV3N5Ocn0rX7FmuqyU1GS7nLqDlfIeR2GRUKBgD/SJNILCTLFNgJOAi4P5UREe8VqGNbsuDzn/T6DuCrOdsfTP9OBLqn9Y2A+yRNBW4gm9moXhFxa0T0iYg+7Tt8qYi31XR0TN1BX9q0AwcfeRhTJ73IiHse5JBvZhM1Hdb/60ydlHULbbZ5Z+a9M3fVsfPmzF3VnVSjdZvWHHD4ITwx8rE19A7Wbt16bEmXLbty/zOjGD3laTp12YxhTz3MlzpuCkCnzTfjxrt/zy8G/ozZM99addyTo8dw/EH9+f4hxzDrtTd48/WZjfUWGo/HEJqklXz6ntrU2rYiZ72K7OpIFJEjPEehzuyac9TUD3AF8ERE7Ah8o452tQgfLv9w1VjBh8s/5NknnmLrL2/Lppt1ZMLTzwPw/L+eZYutugNZ18SoB/7OxytWMHvW27z5+ix22n0XPly2fFVf9cqVK3nqsSfo0atno7yntc1rL7/KAT370G+nfem3077Me+ddvrPfkSyav4ANN6rk5vv+zG8GXcPk5yd+5riaHy6VG1dy3I9O4ME7722M5je6lhAQmnOXUV1mAbuTZfWrt1smxxiyzII3RMSimi4jYClZ11Jt04HukraOiBnACcCTdeyXayPgnbT+gyLa1CwtWrCQnx5/KgBVVVUccew32feQ/Vl/g/W5+qLLqVq5ktZtWvOr31wJwNZf3obDjv46R+1xKK1aVXDJtZdTUVHBhx9+yFnf/TEfr1hBdXU1fffbi++cfHxjvrUW639vG8JX9t2Tjb+0CY+/Mo6br7yB4XcNq3Pf7w48kW5bbcmpF57NqRdmQ3Cn9j+B9xYu4ufX/Iptd/wyAL/73yG8OWMtvEKgeXzhF6K6+nKbA0nLIqJdrbL9gD8B84DngT4RcYCkQcCyiLg27TcVODIiZkk6CbiA7Ff9CxHxA0n7AH8g+8V/LPBL0piCpIOBa8mC6b+B0yNihaRZ6XwLJfUBrk3n3ousa2kB8E/ghIjoLukA4PyIOLK+97jTbrvEiKd8y2VzstNGPRq7Cbaalld9MrGIWczyqtx269jjlmsK7vf4Id8q+VwNqdkGhLWBA0Lz44DQ/JQlIGyzdfS9uXBAGHNo0w4ILa3LyMysUbSE39YtbVDZzKxRlHNQWVJFenbp4fR6kKR3cjIzHJGz78WSZkh6VdJhOeW7p2eiZkgaoiKe8HRAMDMrUVD2u4zOAWo/HXhDzq32IwFSip0BZLez9wNukVSR9h9K9kxTr7T0K3RSBwQzs1KV8TkESV2BrwN/LGL3o4B70jNWM4EZQF9JnYHKiBgX2UDxnWQpefJyQDAzK4OoLrwAHSRNyFkG1lHVjcCFQO1kF2emNDu3SdoklXUB3s7ZZ3Yq65LWa5fn5YBgZlYGRV4hLKzJRJCWW3PrkHQkMD8iJtaqfijQE+gNzAWuqzmkrqbkKc/LdxmZmZVBme4y2gf4Zho0bgNUSvpLRHy/ZgdJfwBqUufPBrrlHN8VmJPKu9ZRnpevEMzMShVFLoWqibg4IrpGRHeyweJ/RsT305hAjaOBqWl9BDBAUmtJPcgGj8dHxFxgqaQ9091FJwIPFTq/rxDMzEoUNPgUmtdI6p1ONQs4FSAipkkaBrxMlsvtjIiomWLwdLLszG3J0vmMKnQSBwQzszIo94NpETEWGJvW681JHhGDgcF1lE8AdlydczogmJmVKoJoARMiOCCYmZVD848HDghmZuXQEnIZOSCYmZWDA4KZmYGvEMzMDLKrg6qCezV5DghmZiXKsp02/0sEBwQzs3Jo/vHAAcHMrGRFpqZo6hwQzMzKoAX0GDkgmJmVhQOCmZkRgFNXmJkZ8Pn5zZohBwQzs3Jo/hcIDghmZuXgQWUzM2v5t51K+i153mJEnN0gLTIza47KOKgsqQKYALwTEUdKag/cC3QnmzHtOxHxftr3YuAUsuQZZ0fEI6l8dz6dMW0kcE4UeJw63xXChBLej5nZ2qW8VwjnAK8Alen1RcCYiLha0kXp9c8lbU829/IOwObA45K2SdNoDgUGAs+RBYR+FJhGs96AEBF35L6WtEFELP8i78zMrMUrU0CQ1BX4Otm0mD9LxUcBB6T1O8im1vx5Kr8nIlYAMyXNAPpKmgVURsS4VOedQH8KBIR1imjcXpJeJotWSNpF0i3Fvz0zsxau5jmEQgt0kDQhZxlYR203Ahfy2RtZO0XEXID0b8dU3gV4O2e/2amsS1qvXZ5XMYPKNwKHASNSY16U9NUijjMzW3sUd4WwMCL61LdR0pHA/IiYKOmAIupTPS2przyvou4yioi3pc/U3wIyf5uZlVF5uoz2Ab4p6QigDVAp6S/APEmdI2KupM7A/LT/bKBbzvFdgTmpvGsd5XkV7DIC3pa0NxCS1pN0Pqn7yMzMyB5CKK7LqEA1cXFEdI2I7mSDxf+MiO+T9dCclHY7CXgorY8ABkhqLakH0AsYn7qVlkraU9mv+RNzjqlXMVcIpwG/Iet/egd4BDijiOPMzNYeDfscwtXAMEmnAG8B3waIiGmShgEvAyuBM9IdRgCn8+ltp6MoMKAMRQSEiFgIHP8F3oCZ2dqjzAEhIsaS3U1ERCwCDq5nv8FkdyTVLp8A7Lg65yzmLqOtJP1d0gJJ8yU9JGmr1TmJmVmLF0UsTVwxYwh3A8OAzmQPPtwH/F9DNsrMrNmJKLw0ccUEBEXEXRGxMi1/oVnEOjOzNSTInhootDRx+XIZtU+rT6RHpe8he9vHAf9YA20zM2s+msEXfiH5BpUn8tkHHE7N2RbAFQ3VKDOzZqcZdAkVki+XUY812RAzs2armQwaF1LUk8qSdgS2J3tyDoCIuLOhGmVm1uysDQFB0q/IsuxtT5ZC9XDgacABwcysRgsICMXcZXQs2QMR70bED4FdgNYN2iozs+YkgKoovDRxxXQZfRQR1ZJWSqokS6rkB9PMzHI1/e/7gooJCBMkbQz8gezOo2XA+IZslJlZc6O1ISBExE/S6u8kjSabheelhm2WmVkz05JvO5W0W75tETGpYZpkZtYMNf94kPcK4bo82wI4qMxtsVpmLl7G90Y+19jNsNXwpY0KzlJoTczy92aVXklLfw4hIg5ckw0xM2u+ipsAp6kr6sE0MzMroPnHg6KeQzAzs3zKlO1UUhtJ4yW9KGmapMtS+SBJ70ianJYjco65WNIMSa9KOiynfHdJU9K2IWkqzbx8hWBmVg7luctoBXBQRCyTtC7wtKSaqS9viIhrc3eWtD3Z3Ms7kM1X87ikbdI0mkOBgcBzZFkm+lFgGs1iZkyTpO9LujS93kJS39V6i2ZmLV0ZZkyLzLL0ct205DvyKOCeiFgRETOBGUBfSZ3JHhEYFxFBlmqof6HzF9NldAuwF/Dd9HopcHMRx5mZrT2KCwgdJE3IWQbWrkZShaTJZFkhHouI59OmMyW9JOk2SZuksi7A2zmHz05lXdJ67fK8iuky2iMidpP0AkBEvC9pvSKOMzNbOwSouLuMFkZEn7xVZd09vVOGiOEp2/RQsjloauaiuQ44mU/nq6nVmnrL8yrmCuETSRU1lUnalBYxN5CZWRmVocvoM9VFfACMBfpFxLyIqIqIarI0QjXd9rOBbjmHdQXmpPKudZTnVUxAGAIMBzpKGkyW+vrKIo4zM1t7lCEgSNo0XRkgqS1wCDA9jQnUOBqYmtZHAAMktZbUA+gFjI+IucBSSXumu4tOBB4qdP5ichn9VdJEshTYAvpHxCuF35qZ2VqkPP0mnYE7Uq/MOsCwiHhY0l2SepOFlVmkKY0jYpqkYcDLwErgjNTlBHA6cDvQluzuorx3GEFxE+RsAXwI/D23LCLeKvINmpm1bGVKXZESh+5aR/kJeY4ZDAyuo3wCsOPqnL+YQeV/8OkgRRugB/Aq2X2vZmYGa0fqiojYKfd1yoJ6aoO1yMysOWr+8WD1n1SOiEmSvtIQjTEza47EWjJBjqSf5bxcB9gNWNBgLTIza25i7cl2umHO+kqyMYUHGqY5ZmbNVPOPB/kDQrr1qV1EXLCG2mNm1jy15IAgqVVErMw3laaZmbE6qSuatHxXCOPJxgsmSxoB3Acsr9kYEQ82cNvMzJqP5h8PihpDaA8sIptDueZ5hAAcEMzMarTwK4SO6Q6jqXw+e17zf+dmZmXU0m87rQDa8QXTqJqZrTXKlLqiseULCHMj4vI11hIzs2ZM5ZlCs1HlCwgFJ2Q2M7OkqmUHhIPXWCvMzJqzaOFjCBHx3ppsiJlZ8xVZ+opmbrWT25mZWR1aQEAoZgpNMzPLoybbaaGlYD1SG0njJb0oaZqky1J5e0mPSXot/btJzjEXS5oh6VVJh+WU7y5pSto2JE2lmZcDgplZqQJUVV1wKcIK4KCI2AXoDfSTtCdwETAmInoBY9JrJG0PDCCbsKwfcEvKQQcwFBhINs9yr7Q9LwcEM7NyiCi8FKwiIiKWpZfrpiWAo4A7UvkdQP+0fhRwT0SsiIiZwAygr6TOQGVEjIuIAO7MOaZeDghmZmVQZJdRB0kTcpaBn6tHqpA0GZgPPBYRzwOdImIuQPq3Y9q9C/B2zuGzU1mXtF67PC8PKpuZlarIKwBgYUT0yV9VVAG9JW0MDJe0Y57d68sk8YUyTPgKwcysHKqj8LIaIuIDYCxZ3/+81A1E+nd+2m020C3nsK7AnFTetY7yvBwQzMzKQBEFl4J1SJumKwMktQUOAaYDI4CT0m4nAQ+l9RHAAEmtJfUgGzwen7qVlkraM91ddGLOMfVyl5GZWakioLqqHDV1Bu5IdwqtAwyLiIcljQOGSToFeAv4dnbamCZpGPAy2RTHZ6QuJ4DTgduBtsCotOTlgGBmVgblSG4XES8Bu9ZRvoh60glFxGBgcB3lE4B84w+f44BgZlYOLeBJZQcEM7NyiKIePGvSHBDMzErm5HZmZkZ68KyqLIPKjcoBwcysZOEuIzMzS9xlZGZmBL5CMDMzcJeRmZl9yl1GZmaWXSGsbOxGlMwBwcysRAGEu4zMzCxFhMZuRckcEMzMSuZBZbPPiOpqZlzyC9Zt357uF1zIu8OGsXTiBFhnHVpVVtL1tNNYd5P2fLxgAf85/zxab745AOtvvTVdTvkRAB+MG8eCvw0nqqvZcNdd6fy94xvzLbVoF157MXsdvA8fLHqfHx5yAgCX3nI5W2y1BQDtKtuxbMkyftTvBwBstV1Pzrv6QtZvtwER1Zx25I/4eMXHtFq3Fedc8TN677UrUR388Zpb+deosY30rhqRA0L9JAVwfUScl16fD7SLiEFlqHsQ8GNgQSoaHREXlVpvqntj4HsRcUt6vTkwJCKOLUf9OecZC5yfUtS2CAtHjaJ1ly5Uf/QRAJseeSSbfec72bbRo5n/4IOrvvjX69SJXldd/ZnjVy5dyrt3/5WtB19Jq8pK3h56C8umTqXdjquVwdeKNPq+kQy//QF+ceMvV5Vd/pNLV62f/sszWb5kOQAVFRVcMuRSrjznCl5/ZQaVG1ey8pNsEPX7Z53EB4ve54T9v4skKjeuXLNvpKloAQGhIWdMWwEcI6lDA9V/Q0T0TktZgkGyMfCTmhcRMafcwaAl+mTRIpZOfoH2Bx64qqxi/fVXrVev+C91T/P6qY/nz2e9zTrTqjL7Qmm3404sHv98g7TX4KXnX2TpB0vq3X7gkQcx5qHHAOjz1b688crrvP7KDACWfLCE6ursC/CI477OX2+6C4CIYPH7ixu45U1QBFFdVXApRFI3SU9IekXSNEnnpPJBkt6RNDktR+Qcc7GkGZJelXRYTvnukqakbUPSzGl5NWRAWAncCpxbe4OkLSWNkfRS+neLVH57avizkt6QVPQXsaQfSLop5/XDkg5I68skDZb0oqTnJHVK5Z0kDU/lL0raG7ga6Jn+6L+W1F3S1LR/G0l/Tn/kFyQdmHPuByWNlvSapGty2jFU0oT04V62+n/G5mHOXXfS+bvfA332P6l3772X6WeewQfPPEOnb397VfnHCxbw2sUX8cbll7F8+nQAWnfqxIq5c/h4wQKiqoolEybwyaL31uj7sMzOe+zC+wvf551ZswHotlU3IoJr/nI9t468jQGnfQ/IupUATr7gx9w68jYGDb2CTTps0mjtblzVRSwFrQTOi4gvA3sCZ0jaPm3L/RE8EiBtGwDsQDb38i1ptjWAocBAsmk1e6XteTX0nMo3A8dL2qhW+U3AnRGxM/BXYEjOts7AvsCRZF/O9Tk3J1oelmc/gA2A5yJiF+BfZN1NpPM+mcp3A6YBFwGvpz/6BbXqOQMgInYCvks21V2btK03cBywE3CcpJqJry+JiD7AzsD+knbO11BJA1MAmfDJkvp/vTUlSyZNolVlJW232upz2zY77ji2u+lmNt5nHxY9+ggArTbemO2G/JZeV11N5++fwNs3/ZaqDz+kol07uvzwZN4a8htev+wy1tu0A6rwtN+N4eCjvrbq6gCgolUFO31lZwafdRlnHXM6+/Xbn9322Z2Kigo6bt6Jqf+ewsAjTmbapKmc/j9nNmLLG0sQUV1wKVhLxNyImJTWlwKvAF3yHHIUcE9ErIiImcAMoK+kzkBlRIyLiADuBPoXOn+D/r8tIpakhpxda9NewN1p/S6yAFDjbxFRHREvA53yVJ8bLR8p0JSPgYfT+kSge1o/iCyKEhFVEVHoWnff1F4iYjrwJrBN2jYmIhZHxH/J5jfdMpV/R9Ik4AWyKL49eUTErRHRJyL6rFvZPPpiP/zPqyyZNInpZ5/F278dwrJp03j75ps+s8/Ge+/D4vHjAVhn3XVpteGGALTdaivW69SJFe/OBaBy993Z+or/x9aXX07rzpuz3mabrdk3Y1RUVLBfv/15YsSYVWUL5s7nxecns/j9xaz47wqee2IcvXbclsXvL+ajDz/iqdFPAjD24SfoteO2jdX0xhXVhZfVIKk72XSaNf2mZ6Zeldsk1VyGdQHezjlsdirrktZrl+e1Jn5+3QicQvYrvT65N/CuyFkXQOrumSxpcp46VvLZ99MmZ/2TFCUBqvjig+n5+uBy210FtJLUAzgfODhdDf2jVrtahM0GfJcv33Qz2w35Ld3OOpt2O+xAtzPOZMXcuav2WTJp4qq7ilYuWUKk/ueP581jxbvvsl7HLPavXJzF5Kply1j0+GO0P/CgNfxubPf9+vDW62+y4N0Fq8rGPzmerbbrSes2ramoqKD3Hr1587WZAIx7/Bl675VNA7z7vn1Wla9dotiA0KGmByAtA+uqTVI74AHgp+mH9VCgJ1lPxFzguppd625MveV5NfhtpxHxnqRhZEHhtlT8LFm/113A8cDTBeq4BLikwKlmAT+RtA5ZJOxbRPPGAKcDN6Z+tw2ApcCG9ez/r9Tef0raBtgCeJWsu6kulcByYHEatzgcGFtEu1qEd++5hxVz5yCJdTtsSpdTTgFg+fRXmHfffaiiAtZZhy4nn0Krdllf9Jw77+C/b70FQMejj6F1586N1v6W7pc3DaL3nruyUfuNuW/8cP583Z8Yee/DHPTNQ/jnQ49/Zt9li5dy3x/u4XcP/wkInvvnOJ775zgAfn/lLfziN5dy5qBz+GDRB/zveVc2wrtpZEFRg8bAwtSFXC9J65IFg79GxIMAETEvZ/sf+LTHYzbQLefwrsCcVN61jvK81tRzCNcBuR2LZwO3SbqA7NbRH5bhHM8AM4EpwFRgUhHHnAPcKukUsl/1p0fEOEnPpIHkUWTjIDVuAX4naQrZFckPImJFfYP3EfGipBfIxibeSG1s0dptvz3tts96xbY893P3EwCwUd892KjvHnVu2+Ks2r2L1lCuOHNQneVX/2xwneWPDX+Ux4Y/+rnyee/M45xjzyhn05qhIIobNM4r3Qn0J+CViLg+p7xzRNRcch9N9h0HMAK4W9L1wOZkg8fjI6JK0lJJe5J1OZ0I/Lbg+aMFPG7dUrXr2TN2uvqawjtakzHntPMbuwm2mt56b9bEQr/aC9lw/Q7Re9tvFNzv6cm35z2XpH2Bp8h+2NZEmF+Q3cTSm6zbZxZwak2AkHQJcDLZj9SfRsSoVN4HuB1oS/bj9qwo8IXvJ5XNzEqUpTIq/cd1RDxN3f3/I/McMxj43GVdeuh1tZ7qdEAwMyuZcxmZmVkNBwQzMyOCiKLuMmrSHBDMzMrAE+SYmRnggGBmZkA2qNz8b+F3QDAzKwNfIZiZGdmTCA4IZmZrvQioLi6XUZPmgGBmVjI/mGZmZkk5kts1NgcEM7Ny8F1GZmZWM4Vmc+eAYGZWIg8qm5lZ4ttOzcwsaQldRusU3sXMzAqJiIJLIZK6SXpC0iuSpkk6J5W3l/SYpNfSv5vkHHOxpBmSXpV0WE757pKmpG1DVN9cvzkcEMzMSpYNKhdairASOC8ivgzsCZwhaXvgImBMRPQCxqTXpG0DgB2AfsAtkipSXUOBgWTzLPdK2/NyQDAzK4vqIpb8ImJuRExK60uBV4AuwFHAHWm3O4D+af0o4J6IWBERM4EZQF9JnYHKiBiX5lG+M+eYenkMwcysRBFR7F1GHSRNyHl9a0TcWteOkroDuwLPA50iYm4611xJHdNuXYDncg6bnco+Seu1y/NyQDAzK4OgqAfTFkZEn0I7SWoHPAD8NCKW5On+r2tD5CnPywHBzKwMynWXkaR1yYLBXyPiwVQ8T1LndHXQGZifymcD3XIO7wrMSeVd6yjPy2MIZmYlK8+gcroT6E/AKxFxfc6mEcBJaf0k4KGc8gGSWkvqQTZ4PD51Ly2VtGeq88ScY+rlKwQzs3IoTy6jfYATgCmSJqeyXwBXA8MknQK8BXw7O2VMkzQMeJnsDqUzIqJmMON04HagLTAqLXk5IJiZlUE5sp1GxNPU3f8PcHA9xwwGBtdRPgHYcXXO74BgZlai1bjLqElzQDAzKwPPh2BmZgBFpaZo6hwQzMxK5vkQzMyM7IkvXyGYmRlEUB0eVDYzM1rGfAgOCGZmZeAuIzMzIwjfdmpmZhlfIZiZGeAxBDMzg3SXkQOCmdlaL3sOwQHBzMzwGIKZmQFOXWFmZqu0hCsET6FpZlaqoCxTaAJIuk3SfElTc8oGSXpH0uS0HJGz7WJJMyS9KumwnPLdJU1J24akqTTzckAwMytRkOUyKrQU6XagXx3lN0RE77SMBJC0PTAA2CEdc4ukirT/UGAg2TzLveqp8zMcEMzMyiAiCi5F1vMv4L0iT3sUcE9ErIiImcAMoK+kzkBlRIyL7MR3Av0LVeaAYGZWBkV2GXWQNCFnGbgapzhT0kupS2mTVNYFeDtnn9mprEtar12elwOCmVnJoqj/AQsjok/OcmuRJxgK9AR6A3OB61J5XeMCkac8L99lZGZWooaeICci5tWsS/oD8HB6ORvolrNrV2BOKu9aR3levkIwMytVQFVUFVy+qDQmUONooOYOpBHAAEmtJfUgGzweHxFzgaWS9kx3F50IPFTwPC3h3tmWStIC4M3GbkcD6QAsbOxGWNFa8ue1ZURsWkoFkkaT/Y0KWRgRee/2kfR/wAGpvnnAr9Lr3mQXI7OAU9OXPpIuAU4GVgI/jYhRqbwP2R1LbYFRwFlR4AvfAcEahaQJEdGnsdthxfHntXZwl5GZmQEOCGZmljggWGMp9nY7axr8ea0FPIZgZmaArxDMzCxxQDAzM8ABweogqSonze5kSd3LWHf/lKGx5vXlkg4pV/2pzgMkPVx4z6ZLUki6Luf1+ZIGlanu2qmUry5HvanujSX9JOf15pLuL1f9OfWOTffZWxk5dYXV5aOI6N1Adfcne+z+ZYCIuLSBztPcrQCOkXRVRDTEA2E3RMS1DVDvxsBPgFsAImIOcGwDnMcagK8QrCiSZknqkNb7SBqb1gel7ItjJb0h6eycY05M2RlflHSXpL2BbwK/Tr9Me0q6XdKxaf+DJb2QJvW4TVLrnHNfJmlS2rZdKu8r6dl0zLOStl3Df5aGtJLszp5za2+QtKWkMelvO0bSFqn89jQRyrPpsyj6i1jSDyTdlPP6YUkHpPVlkganz/E5SZ1SeSdJw1P5i+nzvRromT7fX0vqrjTRi6Q2kv6cPsMXJB2Yc+4HJY2W9Jqka3LaMTRlBZ0m6bLV/zPa6nBAsLq0zelOGF7E/tsBhwF9gV9JWlfSDsAlwEERsQtwTkQ8S5Z75YI0ycfrNRVIakP2mP1xEbET2dXr6TnnWBgRu5FlfTw/lU0HvhoRuwKXAleW8J6bopuB4yVtVKv8JuDOiNgZ+CswJGdbZ2Bf4EiyL+f6nJvzGR+WZz+ADYDn0uf4L+DHqXwI8GQq3w2YBlwEvJ4+3wtq1XMGQPp8vwvckT53yNIyHAfsBBwnqSZh2yXpCemdgf0l7VygrVYCBwSry0c5MzMdXcT+/0gTdCwE5gOdgIOA+2u6OyKi0IQf2wIzI+I/6fUdwFdztj+Y/p0IdE/rGwH3pV+gN5DNGtViRMQSsolNzq61aS/g7rR+F1kAqPG3iKiOiJfJPof65M6+9UiBpnzMp9k1c//+B5EFaCKiKiIWF6hn39ReImI6WZ6ubdK2MRGxOCL+S9aduGUq/46kScALZJ/v9liDcUCwYq3k0/9e2tTatiJnvYrs170oIv96jkLzvdaco6Z+gCuAJyJiR+AbdbSrJbgROIXsV3p9cv/OuZ+FAFJ3z2RJk/PUkfv5wmf/lp/kJEXL/fuvrnyf8ef+G1KWvfN84OB0NfQPWuZn3GQ4IFixZgG7p/VvFbH/GLJfd18CkNQ+lS8FNqxj/+lAd0lbp9cnAE8WOMdGwDtp/QdFtKnZSVdWw8iCQo1nyebRBTgeeLpAHZfUXA3k2W0W0FvSOqm7pm8RzRtD6taTVCGpkvo/X8i6m45P+28DbAG8mqf+SmA5sDiNWxxeRJusBA4IVqzLgN9IeorsF1xeETENGAw8KelF4Pq06R7ggjSo2DNn//8CPyTrApoCVAO/K3Caa4CrJD0DVBTYtzm7js+mVj4b+KGkl8gC5zllOMczwExgCnAtMKmIY84BDkyf10Rgh4hYBDwjaaqkX9fa/xagIu1/L/CDiFhBPSLiRbKuomnAbamN1oCcusLMzABfIZiZWeKAYGZmgAOCmZklDghmZgY4IJiZWeKAYM2aPs3MOlXSfZLWL6Gu3LxKf1ROVtY69j0g5e5Z3XOsyglVTHmtfZat5rkGSTq/8J5mGQcEa+5q0mzsSJZi4bTcjZK+0PMJEfGjlP6hPgcAqx0QzJoyBwRrSZ4Ctk6/3p+QdDcwJT1F+2tJ/04ZQk8FUOYmSS9L+gfQsaYi5eTbl9RPWabVF5VlF+1OFnhqEsTtJ2lTSQ+kc/xb0j7p2C9JejQ9iPd7CqfoQNLfJE1MGT4H1tp2XWrLGEmbprKeKVPoRElPKWWDNVtdng/BWgRJrchSG4xORX2BHSNiZvpSXRwRX1GWUvsZSY8Cu5Il1duJLBHcy2RPxObWuynwB7KsqjMltY+I9yT9DlhWM6dACj43RMTTytJRPwJ8GfgV8HREXC7p68BnvuDrcXI6R1vg35IeSE8AbwBMiojzJF2a6j6TLE32aRHxmqQ9yJ4IPugL/BltLeeAYM1d25ykbU8BfyLryhkfETNT+aHAzvp0foCNgF5k2VT/LyKqgDmS/llH/XsC/6qpK0/W1kOA7aVVFwCVkjZM5zgmHfsPSe8X8Z7OllSTZbZbausisnQe96byvwAPSmqX3u99OeduXcQ5zD7HAcGau8/N7pa+GJfnFgFn1U7zLOkICmdkLTZr6zrAXhHxUR1tKTo/jLJJaQ5JdX2obCKi+jJ8RjrvBw04w52tRTyGYGuDR4DTJa0LWaZNSRuQZd8ckMYYOgMH1nHsOLKJWXqkY+vL2vooWfcNab/eaTU3w+fhwCYF2roR8H4KBtuRXaHUWIdPp6P8HllX1BJgpqRvp3NI0i4FzmFWJwcEWxv8kWx8YJKyyXR+T3Z1PBx4jSzD51DqSLcdEQvI+v0fTFlba7ps/g4cXTOoTJaBtE8atH6ZT+92ugz4qrJJXg4F3irQ1tFkcwG8RDbfw3M525YDO0iaSDZGcHkqPx44JbVvGnBUEX8Ts89xtlMzMwN8hWBmZokDgpmZAQ4IZmaWOCCYmRnggGBmZokDgpmZAQ4IZmaW/H85N4DMZG2UDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Pipeline for preprocessing and modeling\n",
    "logreg = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor1),\n",
    "    ('classifier', LogisticRegression(random_state=396))\n",
    "])\n",
    "\n",
    "#Putting our logistic regression model through our evaluate function\n",
    "evaluate(logreg, X_train, X_test, y_train, y_test, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results**\n",
    "- Our baseline logistic regression shows only a slight improvement over our dummy classifier model in every single metric used to evaluate.\n",
    "- All metrics are quite low -- in particular our recall score is unacceptably low at 29%. Given the large number of categorical features, this is all to be expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of Cross-Validation:\n",
      "\n",
      "Average accuracy: 0.750262041427502        +/- 0.006833022123599001\n",
      "Average precision: 0.7218269684965126        +/- 0.009263669744064939\n",
      "**Average recall: 0.7377904673987746        +/- 0.004731566221769646\n",
      "Average f1 score: 0.7297018074987742        +/- 0.006378236817575044\n",
      "Average roc_auc: 0.749271185122341        +/- 0.006563798539038056\n",
      "\n",
      "++++++++++++++++++++\n",
      "\n",
      "Results of Train-Test Split Validation:\n",
      "\n",
      "Training Scores:\n",
      "Train accuracy: 1.0\n",
      "Train precision: 1.0\n",
      "**Train recall: 1.0\n",
      "Train f1 score: 1.0\n",
      "Train roc_auc: 1.0\n",
      "\n",
      "<><><><><><><><><><>\n",
      "\n",
      "Testing Scores:\n",
      "Test accuracy: 0.7573556936437823\n",
      "Test precision: 0.7296909843899331\n",
      "**Test recall: 0.7479183673469387\n",
      "Test f1 score: 0.7386922518745465\n",
      "Test roc_auc: 0.7566334093371861\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEICAYAAABfz4NwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqsklEQVR4nO3debxVVf3/8dcbMMEBBVFCBkc0QRQTidKvYwli5ZyYOffFHL6av7SvZpnDlzTNNDIxHBIsIxxInFMKzVkQlEFNElKERJAMJ+Te+/n9sdfFzeXecw7dc7n3XN5PH/tx91577bXXOUfO5+y11l5bEYGZmVmb5q6AmZm1DA4IZmYGOCCYmVnigGBmZoADgpmZJQ4IZmYGOCCYmbUokuZJmiFpuqQpKe1iSW+ltOmShubyXyBpjqRXJQ3Ope+eypkjaaQkFTt3u6Z5SVYOnbt0ju69ejZ3NWwNvPLirOaugq2hqprqxRGxeWPK2Ocr+8W7S94tmm/mtJcejoghJRS5X0QsrpN2TUT8LJ8gqQ8wDOgLbAk8KmmHiKgGRgHDgWeAB4AhwIOFTuqA0IJ179WTiX99oLmrYWtg4Bb9mrsKtoYWf7j0H40t490l7zLxiYLftQBsu2H3Lo09Vx2HAOMiYjkwV9IcYKCkeUDHiHgaQNJY4FCKBAQ3GZmZNVpQEzVFF6CLpCm5ZXi9hcGfJE2ts/9MSS9JukVSp5TWHXgzl2d+Suue1uumF+QrBDOzRgqgqqa6lKyLI2JAkTx7RsQCSVsAj0h6haz557J0qsuAq4GTgfr6BaJAekG+QjAza6wo+QqhhKJiQfq7CJgADIyItyOiOiJqgBuBgSn7fCDf0dgDWJDSe9STXpADgplZIwVQU8J/xUjaUNLGtevAgcBMSd1y2Q4DZqb1icAwSetL2gboDTwXEQuBZZIGpdFFxwP3FDu/m4zMzMqgpjwzR3cFJqQRou2A2yPiIUm3SepPFnvmAacCRMQsSeOB2UAVcEYaYQRwGnAr0IGsM7lor7cDgplZo0XJTUIFS4l4Hdi1nvTjChwzAhhRT/oUYOc1Ob8DgplZIwWUJSA0NwcEM7NGigiqoqRRRi2aA4KZWRn4CsHMzICydSo3KwcEM7NGijJ1Kjc3BwQzszJwQDAzM3cqm5lZJoBqXyGYmRm4U9nMzACIkuYqaukcEMzMGim7U9lXCGZmhvsQzMyM2lFGDghmZus8jzIyM7OV3IdgZmbZFULxRxa3eA4IZmaNFr5CMDMzqAE+aQVTV7Rp7gqYmVW8yPoQii2lkDRP0gxJ0yVNSWmdJT0i6bX0t1Mu/wWS5kh6VdLgXPruqZw5kkYqPai5EAcEM7NGCoLqKL6sgf0ion9EDEjb5wOTIqI3MCltI6kPMAzoCwwBrpfUNh0zChgO9E7LkGIndUAwMyuDaqLo0giHAGPS+hjg0Fz6uIhYHhFzgTnAQEndgI4R8XREBDA2d0yDHBDMzBopm7qi+LIGxf1J0lRJw1Na14hYCJD+bpHSuwNv5o6dn9K6p/W66QW5U9nMrAxKbBLqUtsvkIyOiNF18uwZEQskbQE8IumVAuXV1y8QBdILckAwM2ukAD4p7U7lxbl+gfrLiliQ/i6SNAEYCLwtqVtELEzNQYtS9vlAz9zhPYAFKb1HPekFucnIzKyRooQO5VKuICRtKGnj2nXgQGAmMBE4IWU7AbgnrU8EhklaX9I2ZJ3Hz6VmpWWSBqXRRcfnjmmQrxDMzBopu1O5LLoCE9II0XbA7RHxkKTngfGSTgHeAI4CiIhZksYDs4Eq4IyIlTdEnAbcCnQAHkxLQQ4IZmZlsIbDSusVEa8Du9aTvgQ4oIFjRgAj6kmfAuy8Jud3QDAza6RstlNPXWFmZkB15ccDBwQzs8bK5jKq/IjggGBm1ljhJiMzM6Oso4yalQOCmVkZuA/BzMzSKKPmrkXjOSCYmTVSACtKmrmiZXNAMDNrpFjz5x20SA4IZmZlsAbTW7dYDghmZo0maqLoEypbPAcEM7NG8rBTMzNbyU1GZmZGBFR5lJGZmdU+U7nSOSCYmZVBaU/QbNkcEKxsBvfbiw023oi2bdrQtl07/jB5Itdffi13jR1Hp806A3DWReex94H7MWPqdC45+wdANob79PO/ywFfGwzASQcPY/Hbi1i/fXsAfj1hLJtt3qV5XlQr9sNfXMZeX9mbpYvf5Zi9D1uZ/o1vf5OjTjmG6qpqnnzkcX556c9pt147Lrj6x+y0a1+iJrj6wit44ann2WDDDRh939iVx27RrSsP3nkf1/zwp83xkpqVrxCakaRqYEYu6dCImFemsg8F/hYRs9P2pcDjEfFoOcpPZe4LnBsRXy1XmS3BLffevvLLv9Zxp5/Mif8zfJW07XfakXGTJ9KuXTve+ecijtxrKPscdADt2mX/S15x47X03W2XtVbvddH94/7IHTffzsXX/WRl2u577sHeQ/bjm/sczopPVtCpS/ZZHnrckQB8c5/D6dSlM9eOG8WJXxnGhx98yLf2O3Ll8WMe/QOT7y/bP5OKEZEtla5Nc1egET6KiP65ZV4Zyz4U6FO7EREXlTMYGHTYoMPKL//lHy+Hyh/CXXGmPT2Vfy99b5W0I046mjEjb2bFJysAWLr4XQC22XE7nn/82ZVp77+3jJ36913l2J7b9qJzl82Y9vTUtVD7lqcmii8tXSUHhNVImiepS1ofIGlyWr9Y0i2SJkt6XdJZuWOOl/SSpBcl3SbpS8DXgaskTZe0naRbJR2Z8h8gaZqkGanM9XPnvkTSC2nf51L6QElPpWOekrTjWn5b1hpJnHrY8Xxjn69xx623r0z//eixHP6lIfzojO/z3r8+/QJ6aco0Dh10IIfvOYSLfj5iZYAA+OEZ3+fIvYZyw5Ujidbw06tC9Npua/oP2p1bHrqdG+75DTv1zx7J+9rMV9nnoP1o27YtW/bqzud27UPX7p9d5dgDDxvKI398qDmq3SLU1BRfSiWpbfrOuC9tXyzprfSdNF3S0FzeCyTNkfSqpMG59N3Td9EcSSMlFf3ZVckBoUPuzZlQQv7PAYOBgcCPJa0nqS9wIbB/ROwKnB0RTwETgfPSlcffawuQ1B64FTg6IvqRNbmdljvH4oj4PDAKODelvQLsHRG7ARcBP6EAScMlTZE05d3FS0p4WS3H2IfvZPzj9zHqzt8w7sbbmPLks3zjlGN5YPpj3PnEA2z+2c352YWfPgt8lwG78cdn/sS4P9/DTddcn10pkDUXTXjqIcY8OJ4Xnn6ee8fd3VwvaZ3Ttm1bOm7akZOHfJORF1/N5Tf9DIB7b5/AogVvM+bRP3DO//0vLz0/neqqVW/F+sphB/Gnux9ojmo3u9pRRmW8QjgbeLlO2jW5FpEHACT1AYYBfYEhwPWS2qb8o4DhQO+0DCl20koOCPkmo8OKZ+f+iFgeEYuBRUBXYH/gzpRGRLxbpIwdgbkR8be0PQbYO7e/9ptrKrB1Wt8EuEPSTOAasg+uQRExOiIGRMSAzl02K+FltRxbdOsKwGabd+GArw5m5gsv0mWLzWnbti1t2rThiOOPYeYLL6523LY7bk+HDTZgzsuvAtB1y+yX54Ybb8TQIw9hRj3HWNNYtPBt/nJf1jo6e9pMamqCTTfrRHV1Ndf86Eq+td+RnHf8WWzcsSNvvv6Plcf17rsj7dq15ZWXZjdX1ZtXfNqPUGgphaQewMHATSVkPwQYl77b5gJzgIGSugEdI+LpyC6xx5I1hRdUyQGhPlV8+pra19m3PLdeTfbrXmTBvVTFLrlqz1FbPsBlwF8iYmfga/XUq1X48IMP+WDZ+yvXn/rLX9l+px1555+LVuaZdN/DbL/TDgDMn/cmVVVVACx4Yz7z5rzOlr16UFVVxdIlWVxesWIFjz88id47tdpWthbnsQf+zID/GghAr223Yr3PrMe/lixl/Q7tab9BBwAG7vNFqqurmPu311ced+DhB/Hw3Q82S51bihIDQpfaFoC0DK+nqGuB75M9qjnvzNS8fYukTimtO/BmLs/8lNY9rddNL6hiRxk1YB6wO/AgcEQJ+ScBEyRdExFLJHVOVwnLgI3ryf8KsLWk7SNiDnAc8FiRc2wCvJXWTyyhThVpyTuL+e6xpwJQXV3N0CO/zl5f3ocLhp/DKzNfRkD3Xj246NqsxWzaM89z87U30K5dO9q0acOFP7uMTpt15sMPPuTUw0+gasUKampqGLTPnhxxwrBmfGWt12W/vpLd99yDTTtvyr0vPsqNV17PxNvv5ke/+D9+//gEVqxYwSVnZkODO3fpzMjxv6amJnhn4dv8+PQLVinry18fzHePOb05XkaLUeIVwOKIGNDQTklfBRZFxNQ0ErHWKLIfl5H+Xg2cTP0/UqNAekGtLSBcAtws6QfAs8UyR8QsSSOAx9Iw1mlkX9rjgBtT5/ORufwfSzqJrAmoHfA8cEOR01wJjJH0/4A//wevqSL03LoXdz25+i/Ey0dfU2/+rw07nK8NO3y19A023IDxj91b9vrZ6n506vfrTf/x6eevlrbwzQUc9cWvNVjWYXscVLZ6VaJgzTqNC9gT+HrqNG4PdJT024j4Vm0GSTcC96XN+UDP3PE9gAUpvUc96QXJIzharn6f3zUm/nXd7KSrVAO36NfcVbA1tPjDpVML/WovRccdto+Bv7qyaL5JBx5R8rny9ypJ6hYRC1P6OcAXImJYGhhzO9lgmS3JWj16R0S1pOeB/yH7cfwA8MvazuiGtLYrBDOzZtHEv62vlNSf7GJkHnBqds6YJWk8MJusD/WMiKgd/nUa2ajIDmTN6EU7eRwQzMzKoNwBISImA5PT+nEF8o0ARtSTPgXYeU3O6YBgZtZIQeuYusIBwcyssVrJXEYOCGZmZeDpr83MDPAVgpmZJQ4IZmaWepWbuxKN54BgZtZIgfsQzMwscZORmZlBBFEJj0QrwgHBzKwcKj8eOCCYmZWDm4zMzCzjgGBmZuArBDMzg+zqoLporhbPAcHMrJGy2U4r/xLBAcHMrBwqPx44IJiZNZqnrjAzs1qtoMWINs1dATOzViFKWEokqa2kaZLuS9udJT0i6bX0t1Mu7wWS5kh6VdLgXPrukmakfSMlqdh5HRDMzBorgJoovpTubODl3Pb5wKSI6A1MSttI6gMMA/oCQ4DrJbVNx4wChgO90zKk2EkdEMzMyqGmhKUEknoABwM35ZIPAcak9THAobn0cRGxPCLmAnOAgZK6AR0j4unIhj+NzR3TIAcEM7NyKK3JqIukKblleD0lXQt8n1VDSNeIWAiQ/m6R0rsDb+byzU9p3dN63fSC3KlsZlYGJXYqL46IAQ3tlPRVYFFETJW0bwnl1dcvEAXSC3JAMDNrrPINO90T+LqkoUB7oKOk3wJvS+oWEQtTc9CilH8+0DN3fA9gQUrvUU96QQ0GBEm/pMBLjIizihVuZrbOKMPzECLiAuACgHSFcG5EfEvSVcAJwBXp7z3pkInA7ZJ+DmxJ1nn8XERUS1omaRDwLHA88Mti5y90hTDlP3pFZmbroqa9D+EKYLykU4A3gKMAImKWpPHAbKAKOCMiamdVOg24FegAPJiWghoMCBExJr8tacOI+GDNX4eZ2TqgzAEhIiYDk9P6EuCABvKNAEbUkz4F2HlNzll0lJGkL0qaTRoTK2lXSdevyUnMzFq18t+H0CxKGXZ6LTAYWAIQES8CezdhnczMKk8Z71RuLiWNMoqIN+vc9dwKZv42MyujCvjCL6aUgPCmpC8BIekzwFmseku1mdm6LSqjSaiYUpqMvgOcQXaX21tA/7RtZma11oUmo4hYDBy7FupiZla5KuALv5hSRhltK+leSe9IWiTpHknbro3KmZlVjFZwhVBKk9HtwHigG9mdcHcAv2/KSpmZVZyI4ksLV0pAUETcFhFVafktFRHrzMzWkqBs0183p0JzGXVOq3+RdD4wjuxlHw3cvxbqZmZWOSrgC7+YQp3KU1l1GtVTc/sCuKypKmVmVnEqoEmomEJzGW2zNitiZlaxKqTTuJiS7lSWtDPQh2x+bgAiYmxTVcrMrOKsCwFB0o+BfckCwgPAQcATZM/oNDMzaBUBoZRRRkeSTbv6z4g4CdgVWL9Ja2VmVkkCqI7iSwtXSpPRRxFRI6lKUkeyR7f5xjQzs7yW/31fVCkBYYqkTYEbyUYevQ8815SVMjOrNFoXAkJEnJ5Wb5D0ENAxIl5q2mqZmVWYVjDstME+BEmfr7sAnYF2ad3MzGqVYS4jSe0lPSfpRUmzJF2S0i+W9Jak6WkZmjvmAklzJL0qaXAufXdJM9K+karzUJv6FLpCuLrAvgD2L/7yrDFmT5tBv018O0gl2f+xcc1dBVtD9+51ROMLKd99CMuB/SPifUnrAU9IejDtuyYifpbPLKkPMAzoSzbX3KOSdoiIamAUMBx4hmyE6BDgQQoodGPafv/hCzIzW8eU5wE5ERFk/bQA66WlUMGHAOMiYjkwV9IcYKCkeWTN+08DSBoLHEqRgFDKsFMzMyumtCajLpKm5JbhdYuR1FbSdLIRnY9ExLNp15mSXpJ0i6ROKa078Gbu8PkprXtar5tekAOCmVljlT7b6eKIGJBbRq9WVER1RPQHepD92t+ZrPlnO7InVi7k0yb9+voFokB6QQ4IZmblUObnIUTEv4DJwJCIeDsFihqyWwAGpmzzgZ65w3oAC1J6j3rSCyrliWmS9C1JF6XtXpIGFjvOzGydUp5RRpun+76Q1AH4MvCKpG65bIcBM9P6RGCYpPUlbQP0Bp6LiIXAMkmD0uii44F7ip2/lBvTrie72NkfuBRYBtwF7FHCsWZm64byjDLqBoyR1JbsB/v4iLhP0m2S+qezzCM9jiAiZkkaD8wGqoAz0ggjgNOAW4EOZJ3JBTuUobSA8IWI+LykaakCSyV9pvTXZ2bWygWoPKOMXgJ2qyf9uALHjABG1JM+Bdh5Tc5fSkBYkaJVQHZJQ6t4NpCZWRlV/o3KJXUqjwQmAFtIGkE29fVPmrRWZmaVpgx9CM2tlLmMfidpKtkU2AIOjYiXm7xmZmaVpBW0m5TygJxewIfAvfm0iHijKStmZlYxKuQKoJhS+hDu59MbHdoD2wCvks2dYWZmUJapK5pbKU1G/fLbaabTU5usRmZmlajy40FJVwiriIgXJPkeBDOzRKwjD8iR9P9ym22AzwPvNFmNzMwqTZRnttPmVsoVwsa59SqyPoW7mqY6ZmYVqvLjQeGAkG5I2ygizltL9TEzq0ytOSBIahcRVX5cpplZEWWauqK5FbpCeI6sv2C6pInAHcAHtTsj4u4mrpuZWeWo/HhQUh9CZ2AJ2WyntfcjBOCAYGZWq5VfIWyRRhjNZPUn8FT+KzczK6PWPuy0LbAR/+Gj2MzM1hnrwNQVCyPi0rVWEzOzCqY1fERmS1QoINR3ZWBmZvWpbt0B4YC1Vgszs0oWraMPocEH5ETEu2uzImZmlSuy6SuKLUVIai/pOUkvSpol6ZKU3lnSI5JeS3875Y65QNIcSa9KGpxL313SjLRvpKSirT6lPDHNzMyKKUNAAJYD+0fErkB/YIikQcD5wKSI6A1MSttI6gMMI3scwRDg+jTDBMAoYDjQOy1Dip3cAcHMrJFqZzstthQTmffT5nppCeAQYExKHwMcmtYPAcZFxPKImAvMAQZK6gZ0jIinIyKAsbljGuSAYGbWWAGqrim6AF0kTcktw+sWJamtpOnAIuCRiHgW6BoRCwHS3y1S9u7Am7nD56e07mm9bnpBa/w8BDMzq0dpTUKLI2JA4WKiGugvaVNggqSdC2Rv6D6x/+j+MV8hmJmVQTmajPIi4l/AZLK2/7dTMxDp76KUbT7QM3dYD2BBSu9RT3pBDghmZo1VSodyaaOMNk9XBkjqAHwZeAWYCJyQsp0A3JPWJwLDJK0vaRuyzuPnUrPSMkmD0uii43PHNMhNRmZm5VCeye26AWPSSKE2wPiIuE/S08B4SacAbwBHAUTELEnjgdlkDzA7IzU5AZwG3Ap0AB5MS0EOCGZmZVCOqSsi4iVgt3rSl9DAzcIRMQIYUU/6FKBQ/8NqHBDMzBorAmqqi+dr4RwQzMzKoLVPbmdmZqVyQDAzMwCiprlr0GgOCGZmjVbyXEUtmgOCmVkjKUDV7lQ2MzPCTUZmZpa4ycjMzAh8hWBmZuAmIzMz+5SbjMzMLLtCqGruSjSaA4KZWSMFEG4yMjOzFBGauxaN5oBgZtZo7lQ2W+nSX13J3kP25913lnD4oMEAnHbBdznihGEsXfwuACMvvZK//mnyymM+22NL7nnuEa6//FrG/PLGVcobOe5Gemzda2VZ1jSiuobnT/0B63fpxK5X/C+v/+YOFtz/Zz6zSUcAtv3vYXQZtBs1VVW8ctVolv1tLlFdzWcH783Wxx4KwN9vGsc/H36cqmUfsM9DY5rx1TSzVhAQmuwRmpJC0tW57XMlXVymsi+W9Jak6Wm5ohzlprI3lXR6bntLSXeWq/xcuZMlFXzYdiW553d3ctrhJ6yWftuvbuaovYZy1F5DVwkGAN+//Ec88cjk1Y454GuD+eiDD5uoppb35l0PsuFWW66S1uvIoQy8+acMvPmndBmUPatl0eRnqPlkBV/4zVXsMfpyFkx8lI8WZo/17fLF3Rlww2rPZ1n3RE3xpYVrymcqLwcOl9Slicq/JiL6p+X8Mpa7KbAyIETEgog4sozlt0pTn3qO95a+V3L+/Q8+kPnz3mDOK6+tkt5hww04/sxv8+srf1nuKlodHy9awpJnXqDbwfsXzyxR/fFyaqqqqVn+CVqvHe023ACATfr2Zv3NOjVxbVu4CKKmuujS0jVlQKgCRgPn1N0haStJkyS9lP72Sum3Shop6SlJr0sq+YtY0omSrstt3ydp37T+vqQRkl6U9Iykrim9q6QJKf1FSV8CrgC2S1ceV0naWtLMlL+9pN9ImiFpmqT9cue+W9JDkl6TdGWuHqMkTZE0S9Ila/42VrZjhp/AXU89yKW/upKOm2bNEB026MDJ53yHUVf8YrX8//PD7zHmlzfx8Ucfr+2qrnNeu24M2516LNkz2D81f8LDPHvy93n5pzewYtn7AGyxzxdo2359njziOzx59Jn0OvqrrNdxo+aodgtWU8LSsjVlQAD4FXCspE3qpF8HjI2IXYDfASNz+7oBewFfJftybsg5uSajYg3NGwLPRMSuwOPAf6f0kcBjKf3zwCzgfODv6crjvDrlnAEQEf2AY8geht0+7esPHA30A46W1DOlXxgRA4BdgH0k7VKoopKGpwAyJajsUQvjb/otQ3fdmyP3HMo7/1zEuSN+CMDpPziH235182rNQjv260PPbbfiz/c93BzVXacsfmoqn+m0CR133HaV9B6HfIUv3j6SgTddwWc225Q51/8WgH+//HfUtg173jWKL/1+JG+Ov5+PFrzdHFVvoYKImqJLMZJ6SvqLpJfTj8izU3rdZvKhuWMukDRH0qv570JJu6cfr3PSD23Vd868Ju1Ujoh/SxoLnAV8lNv1ReDwtH4bcGVu3x8je+dm1/6Sb8A1EfGz2g1JJxbI+wlwX1qfCnwlre8PHJ/qWg28J6nQte9ewC9T/lck/QPYIe2bFBHvpbrMBrYC3gS+IWk42XvdDegDvNTQCSJiNNmVFW3VpqIjwpJ3Fq9cv2vMOK4bfzMA/Qb05yuHDOWcSy9g4006ElHDJ8uXU11dQ5/+/XhoxhO0a9eWzptvxi33j+Pkg4c110totd6b+TcWPzmVJc9Mo+aTFVR9+BGz/u86+v7wzJV5tjx4f166IPun+fakJ+k8cFfatGvHZzptwiY778i/X32dDlsW+ie6jilPH0EV8L2IeEHSxsBUSY+kfat85wFI6gMMA/oCWwKPStohfZ+NAoYDzwAPAEOABwudfG2MMroWeAH4TYE8+S++5bl1AUgaARwMEBH9GyijilWveNrn1ldErBwkXM1//roLRdh8vauBdpK2Ac4F9oiIpZJurVOvVq1L181Z/PY7QNZRPOflvwFw4pBvrMxz2gXf5cP3P+D3o8cCMP7m7Bfplr16cN34mx0Mmsh2w49hu+HHALB02ize+MN99P3hmSxfsnRlf8A7TzzPhttkF7rtt9iMpS/M4rNf+S9qPl7Oe7Nfo+eRBzVb/Vue8gw7jYiFwMK0vkzSy0D3AoccAoyLiOXAXElzgIGS5gEdI+JpgPTD/FCaOyBExLuSxgOnALek5KfIotptwLHAE0XKuBC4sMip5gGnS2pD9gYOLKF6k4DTgGsltSVrWloGbNxA/sdTff8saQegF/AqWXNTfToCH5BdeXQFDgIml1CvivPTW0ayx16D2HSzTjz68tP86ifXsMd/DeJz/foQEbz1xnwuPfsHzV1NK2LODb/j/Tn/AIkOn92cHb/3bQC6HzqYl386iudOOo+IoNtB+7LRdlutPObtR5+kevknPHnk6XQ7eD+2Pemo5nwZa19QaqdxF0lTctujU6vAaiRtDewGPAvsCZwp6XhgCtlVxFKy77pncofNT2kr0nrd9ILW1n0IVwNn5rbPAm6RdB7wDnBSGc7xJDAXmAHMJLsqKeZsYLSkU8h+1Z8WEU9LejJ1JD9I1g9S63rgBkkzyK5IToyI5Q01zUXEi5KmkfVNvJ7q2Cr978lnrZY24bbxRY8bdfm19aYveGO+70FYSzrt1pdOu/UFoO+FZ9abp90G7el3yWrjQwDY/jvHsv13jm2y+lWGIErrNF6c+hQLkrQRcBfw3dT0Pgq4jKw15TKy79STqb/VIgqkF9RkASEiNsqtvw1skNueR9Z+X/eYExsqo076xfWkBdmv92J1uRO4M1evQ+rJ/806STun9I+BE+vJfytwa277q7n11fKn9H3rSzezClWm+wwkrUcWDH4XEXfDyu+q2v038mmf6HygZ+7wHsCClN6jnvSCmnqUkZlZq5dNZRRFl2LSSKCbgZcj4ue59G65bIeRtYIATASGSVo/9Vn2Bp5LfRHLJA1KZR4P3FPs/J66wsys0co2l9GewHHADEnTU9oPgGMk9c9OxDzgVICImJX6aGeTNWOfkUYYQdY/eivQgaz5u2CHMjggmJmVR3lGGT1B/e3/DxQ4ZgSw2twhETGF1NxdKgcEM7PGiuDTH+aVywHBzKwM/IAcMzMDHBDMzAzIOpUreqYZwAHBzKwsfIVgZmZko0EdEMzM1nkRUFMBD8ApxgHBzKzRynZjWrNyQDAzK4MSJ7dr0RwQzMzKwaOMzMys9hGalc4BwcyskdypbGZmiYedmplZ4iYjMzMDKOkBOC2dA4KZWaO5U9nMzFZyQDAzW+dFRKsYZdSmuStgZtYaRAn/FSOpp6S/SHpZ0ixJZ6f0zpIekfRa+tspd8wFkuZIelXS4Fz67pJmpH0jJdX3aM5VOCCYmZVBRE3RpQRVwPciYidgEHCGpD7A+cCkiOgNTErbpH3DgL7AEOB6SW1TWaOA4UDvtAwpdnIHBDOzRouyBISIWBgRL6T1ZcDLQHfgEGBMyjYGODStHwKMi4jlETEXmAMMlNQN6BgRT0c2/Gls7pgGuQ/BzKwcSht22kXSlNz26IgYXV9GSVsDuwHPAl0jYmF2mlgoaYuUrTvwTO6w+SltRVqvm16QA4KZWRmUONvp4ogYUCyTpI2Au4DvRsS/CzT/17cjCqQX5IBgZtZI5RxlJGk9smDwu4i4OyW/LalbujroBixK6fOBnrnDewALUnqPetILch+CmVkZBDVFl2LSSKCbgZcj4ue5XROBE9L6CcA9ufRhktaXtA1Z5/FzqXlpmaRBqczjc8c0yFcIZmZlUKapK/YEjgNmSJqe0n4AXAGMl3QK8AZwVDrnLEnjgdlkI5TOiIjaS5XTgFuBDsCDaSnIAcHMrNHKM3VFRDxB/e3/AAc0cMwIYEQ96VOAndfk/A4IZmaNFHhyOzMzA4igJip/6goHBDOzMvBsp2ZmBrjJyMzMqJ3YzlcIZmaGrxDMzCxxH4KZmaVRRg4IZmbrvOw+BAcEMzPDfQhmZgaUa+qK5uaAYGZWBr5CMDMzCPchmJkZ2Y1pnsvIzMwANxmZmVniJiMzM6N2NqNK54BgZtZIfkCOmZllAqpbQaeyWkNUa60kvQP8o7nr0US6AIubuxJWstb8eW0VEZs3pgBJD5G9R8UsjoghjTlXU3JAsGYhaUpEDGjuelhp/HmtG9o0dwXMzKxlcEAwMzPAAcGaz+jmroCtEX9e6wD3IZiZGeArBDMzSxwQzMwMcECwekiqljQ9t2xdxrIPldQnt32ppC+Xq/xU5r6S7itnmWubpJB0dW77XEkXl6nsiyW9lft8ryhHuansTSWdntveUtKd5So/V+5kSR4GW2a+U9nq81FE9G+isg8F7gNmA0TERU10nkq3HDhc0uUR0RQ3hF0TET9rgnI3BU4HrgeIiAXAkU1wHmsCvkKwkkiaJ6lLWh8gaXJav1jSLekX2+uSzsodc7yklyS9KOk2SV8Cvg5clX6ZbifpVklHpvwHSJomaUYqc/3cuS+R9ELa97mUPlDSU+mYpyTtuJbflqZURTay55y6OyRtJWlSem8nSeqV0m+VNDK9F6/Xvq+lkHSipOty2/dJ2jetvy9pRPocn5HUNaV3lTQhpb+YPt8rgO3S53uVpK0lzUz520v6TfoMp0naL3fuuyU9JOk1SVfm6jFK0hRJsyRdsuZvo60JBwSrT4dcc8KEEvJ/DhgMDAR+LGk9SX2BC4H9I2JX4OyIeAqYCJwXEf0j4u+1BUhqD9wKHB0R/ciuXk/LnWNxRHweGAWcm9JeAfaOiN2Ai4CfNOI1t0S/Ao6VtEmd9OuAsRGxC/A7YGRuXzdgL+CrZF/ODTkn9xkPLlKPDYFn0uf4OPDfKX0k8FhK/zwwCzgf+Hv6fM+rU84ZAOnzPQYYkz53gP7A0UA/4GhJPVP6hekO6V2AfSTtUqSu1ggOCFafj9I/6P4RcVgJ+e+PiOWpaWMR0BXYH7iztrkjIt4tUsaOwNyI+FvaHgPsndt/d/o7Fdg6rW8C3JF+gV4D9C2hrhUjIv4NjAXOqrPri8Dtaf02sgBQ648RURMRs8k+h4Zck/uMHy5SlU/Imvlg1fd/f7IATURUR8R7RcrZK9WXiHiFbJ6uHdK+SRHxXkR8TNacuFVK/4akF4BpZJ9vH6zJOCBYqar49P+X9nX2Lc+tV5P9uhes0QTxKrK/9hy15QNcBvwlInYGvlZPvVqDa4FTyH6lNyT/Puc/CwGk5p7pkqYXKCP/+cKq7+WK+PSGpfz7v6YKfcar/T8kaRuyq8ED0tXQ/bTOz7jFcECwUs0Ddk/rR5SQfxLZr7vNACR1TunLgI3ryf8KsLWk7dP2ccBjRc6xCfBWWj+xhDpVnHRlNZ4sKNR6ChiW1o8FnihSxoW1VwMFss0D+ktqk5prBpZQvUmkZj1JbSV1pOHPF7LmpmNT/h2AXsCrBcrvCHwAvJf6LQ4qoU7WCA4IVqpLgF9I+ivZL7iCImIWMAJ4TNKLwM/TrnHAealTcbtc/o+Bk8iagGYANcANRU5zJXC5pCeBtmv6girI1aw6tfJZwEmSXiILnGeX4RxPAnOBGcDPgBdKOOZsYL/0eU0F+kbEEuBJSTMlXVUn//VA25T/D8CJEbGcBkTEi2RNRbOAW1IdrQl56gozMwN8hWBmZokDgpmZAQ4IZmaWOCCYmRnggGBmZokDglU0fToz60xJd0jaoBFl5edVukm5WVnrybtvmrtnTc+xck6oUtLr5Hl/Dc91saRzi+c0yzggWKWrnWZjZ7IpFr6T3ynpP7o/ISK+naZ/aMi+wBoHBLOWzAHBWpO/AtunX+9/kXQ7MCPdRXuVpOfTDKGnAihznaTZku4HtqgtSLn59iUNUTbT6ovKZhfdmizw1E4Q91+SNpd0VzrH85L2TMduJulP6Ua8X1N8ig4k/VHS1DTD5/A6+65OdZkkafOUtl2aKXSqpL8qzQZrtqb8PARrFSS1I5va4KGUNBDYOSLmpi/V9yJiD2VTaj8p6U/AbmST6vUjmwhuNtkdsflyNwduJJtVda6kzhHxrqQbgPdrnymQgs81EfGEsumoHwZ2An4MPBERl0o6GFjlC74BJ6dzdACel3RXugN4Q+CFiPiepItS2WeSTZP9nYh4TdIXyO4I3v8/eBttHeeAYJWuQ27Str8CN5M15TwXEXNT+oHALvr0+QCbAL3JZlP9fURUAwsk/bme8gcBj9eWVWDW1i8DfaSVFwAdJW2cznF4OvZ+SUtLeE1nSaqdZbZnqusSsuk8/pDSfwvcLWmj9HrvyJ17/RLOYbYaBwSrdKs93S19MX6QTwL+p+40z5KGUnxG1lJnbW0DfDEiPqqnLiXPD6PsoTRfTmV9qOxBRA3N8BnpvP9qwifc2TrEfQi2LngYOE3SepDNtClpQ7LZN4elPoZuwH71HPs02YNZtknHNjRr65/Imm9I+fqn1fwMnwcBnYrUdRNgaQoGnyO7QqnVhk8fR/lNsqaofwNzJR2VziFJuxY5h1m9HBBsXXATWf/AC8oepvNrsqvjCcBrZDN8jqKe6bYj4h2ydv+706yttU029wKH1XYqk81AOiB1Ws/m09FOlwB7K3vIy4HAG0Xq+hDZswBeInvewzO5fR8AfSVNJesjuDSlHwuckuo3CzikhPfEbDWe7dTMzABfIZiZWeKAYGZmgAOCmZklDghmZgY4IJiZWeKAYGZmgAOCmZkl/x8lZpyhuZpUGQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Pipeline for preprocessing and modeling \n",
    "dt = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor1),\n",
    "    ('classifier', DecisionTreeClassifier(random_state=396))\n",
    "])\n",
    "\n",
    "#Putting our decision tree baseline through our evaluate function\n",
    "evaluate(dt, X_train, X_test, y_train, y_test, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results**\n",
    "- Our decision tree model performed significantly better than our baseline model and logistic regression in every metric of evaluation. It is worth noting that this model is overfit on the training data but it stills performs much much better than previous models on the testing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further Non-parametric Modeling \n",
    "\n",
    "Because our initial Decision Tree performs better than Logistic Regression across all metrics, we will continue moving forward with non-parametric modeling. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest\n",
    "\n",
    "We found that a single decision tree gave us a significant improvement in results. To move forward, we will start working with Random Forest which uses multiple decision trees in its modeling. We will also compare our different preprocessing options to see if there are any differences in results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of Cross-Validation:\n",
      "\n",
      "Average accuracy: 0.8163214374844022        +/- 0.002409287424237006\n",
      "Average precision: 0.826344324310344        +/- 0.005677841054201111\n",
      "**Average recall: 0.7571287745079245        +/- 0.006276391866409888\n",
      "Average f1 score: 0.790188166446984        +/- 0.002844295514499265\n",
      "Average roc_auc: 0.894404629088819        +/- 0.0024281142451172293\n",
      "\n",
      "++++++++++++++++++++\n",
      "\n",
      "Results of Train-Test Split Validation:\n",
      "\n",
      "Training Scores:\n",
      "Train accuracy: 1.0\n",
      "Train precision: 1.0\n",
      "**Train recall: 1.0\n",
      "Train f1 score: 1.0\n",
      "Train roc_auc: 1.0\n",
      "\n",
      "<><><><><><><><><><>\n",
      "\n",
      "Testing Scores:\n",
      "Test accuracy: 0.8209927378902449\n",
      "Test precision: 0.8333928571428572\n",
      "**Test recall: 0.7619591836734694\n",
      "Test f1 score: 0.7960767590618337\n",
      "Test roc_auc: 0.8164746139606285\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoSUlEQVR4nO3deZwUxf3/8debQ8ADFFFEQEBFDaLiRbzifWDiQYwHxgRN/IaIGo2JJhqPePyIV7wTNcYY8YrxQokKakg0Il4oKAIiKB4IihwCXsjufn5/dC0OuLszOLMss/t+Ph792O7q6uqaGZjPVFV3tSICMzOzZg1dATMzWzU4IJiZGeCAYGZmiQOCmZkBDghmZpa0aOgKWO3ad2gfnTfq2tDVsBUwadyEhq6CraAqYk5ErFdMGXvst1fMmzsvb77Xxr36WET0K+Zc9ckBYRXWeaOuDH/60Yauhq2Ardr1aOgq2Ar6tHLJO8WWMW/uPIaPHpE338ZrdO5Q7LnqkwOCmVnRgqqoauhKFM0BwcysSAFUVFU2dDWK5oBgZlascAvBzMzIWghVOCCYmRlQ1QjmhXNAMDMrmruMzMyM1GXUCAKC71Q2MytSRFARlXmXQkhaW9J9kl6XNFnSzpLaS3pC0tT0d52c/GdJmiZpiqQDctK3lzQh7btWkvKd2wHBzKwEqqIq71Kga4CREbEFsA0wGTgTGBURPYFRaRtJvYABwJZAP+B6Sc1TOTcAg4Ceacl7h7QDgplZCVRF5F3ykdQW2B34G0BEfBkRHwOHAkNTtqFA/7R+KHB3RCyOiOnANKCvpE5A24h4NrKnoN2Wc0ytHBDMzIoUaVC5gBZCB0ljc5ZByxW1MfAR8HdJ4yTdLGkNoGNEzAJIf9dP+TsD7+UcPyOldU7ry6fXyYPKZmYlUGCX0JyI2KGO/S2A7YBfRMTzkq4hdQ/VoqZxgagjvU5uIZiZFamEg8ozgBkR8Xzavo8sQHyYuoFIf2fn5M+dErkLMDOld6khvU4OCGZmRQqgMqryLnnLifgAeE/S5ilpH2ASMBw4NqUdCzyU1ocDAyS1ktSDbPD4hdSttEjSTunqooE5x9TKXUZmZiVQwjuVfwHcKWk14C3gJ2Q/3u+RdDzwLnAEQERMlHQPWdCoAE6KWNoUGQzcCrQBRqSlTg4IZmZFi5LNZRQR44Gaxhn2qSX/EGBIDeljgd4rcm4HBDOzImV3KnsuIzMzg4LGCFZ1DghmZkXKrjJyQDAza/KqrzIqdw4IZmYl4DEEMzPLWgj5bwRe5TkgmJkVrbDJ61Z1DghmZkWqAr4s8HkHqzIHBDOzYoXHEMzMjGz660oHBDMzAw8qm5kZ1VNXNHQtiueAYGZWAu4yMjMzAvjSdyqbmVmEB5XNzIzqO5XLnwOCmVkJuIVgZmZptlMHBDMzAyrLPx44IJiZFSuby6j8I4IDgplZscJdRmZmhq8yMjOzHB5DMDOzdJVRQ9eieA4IZmZFCmBJ+c9c4YBgZlYsT11hZmZLefprMzMDRFWooStRNAcEM7Mi+bJTMzNbyl1GZmZGBFT4KiMzM2ssz1Ru1tAVMDNrDKIq/1IISW9LmiBpvKSxKa29pCckTU1/18nJf5akaZKmSDogJ337VM40SddKyjvq7YBgJbPw44X8auBgDt5xHw7puy/jX3iZK879AwfvuA+H7dKPU4/5OQs/XgjAki+/5JwTz+D7u/TjB7seyItPP7e0nJEPPMxhu/Sj/077c+V5FzfUy2lSjhn8Ex547jGGPf84PzrxpwCcfM6vuH/MCO4d/Sh/efA21ttgfQB6b78N945+lHtHP8p9z4xg74MOqKvoJqMq8i8rYK+I6BMRO6TtM4FREdETGJW2kdQLGABsCfQDrpfUPB1zAzAI6JmWfvlOWrYBQVJliqDVS/cSlt0/vdHV2xdK2rdU5acy95T0cCnLbGiXnnkBu+67B/96cRT3j36UjTfblJ332o1hzz7GA2NG0m3THtx81fUA3Df0bgCGjRnJTQ/ezuXnDKGqqoqP583nivMu5ubhd/Lgc48zd/YcnnvqmYZ8WY3ept/ajB8cO4Af7nUoh+9yIHscsDcbbdKdv19zEz/Y5UCO2O27PDXyP5zw21MBmDZpCgP2OJgjdvsuJxw2kPOuGULz5s3znKVxiyhsKcKhwNC0PhTon5N+d0QsjojpwDSgr6ROQNuIeDYiArgt55halW1AAD5PEbR6ebuEZfcHlgaEiDgvIv5dwvIbnU8WLuKlMS9w2I+PAqDlaqvRdu227LL37rRokQ1VbbPDtnw48wMA3pwylW/vsQsA667Xgbbt2jJx3KvMePtdum3Sg/Yd1gVgpz135d/DRzbAK2o6Nt58U159cRxffP4FlZWVjH3mefY56AA+XfTJ0jxtVl+dSN9o1fkAWrVuVfQ3XWNRYAuhg6SxOcugGooK4HFJL+Xs7xgRswDS3/VTemfgvZxjZ6S0zml9+fQ6lXNA+JrU99Yhre8g6cm0fr6kWyQ9KektSafkHDNQ0quSXpF0u6RdgEOAy1PLYxNJt0o6POXfR9K41Dd3i6RWOee+QNLLad8WKb2vpDHpmDGSNl/Jb8tKMePt91inQ3vOOfEMjvjO9/j9L37LZ59+tkyeYXfcw2777gHA5r2/xX8ffYKKigpmvP0ek8ZP4IMZs+i6cXemT32T99+ZQUVFBf955Ak+mDGzIV5SkzF10hS237Uv7dqvTes2rfnO/nuxQZdOAPzi3NN5YtIYvnfkofx5yJVLj9lqhz4Me/5xHnj2MS785TlLA0RTVlWVfwHmRMQOOctNNRS1a0RsBxwInCRp9zpOW9O4QNSRXqdyDghtcrqLhhWQfwvgAKAv8HtJLSVtCZwN7B0R2wCnRsQYYDhwRmp5vFldgKTWwK3AURGxFdlVWoNzzjEnfZA3AKentNeB3SNiW+A84A91VVLSoOpfD/PmzC3gZa0aKisrmPzKRI46/hjuffoR2qy+On+76oal+2/6459o3qIFBx3ZH4Dv/+hIOm7YiQF7HsKlZ13INt/enuYtmtNu7Xace8VFnPHTkzn2wCPZcKPONG/hi+Hq0/Q33uSWq27kpgfv4MYHhjJlwmQqK7Iv+Osu+iP79dqFR+55iKN/fuzSYyaMHc/3v70/A/Y8hP/79WBWa9Wqoaq/Sqi+yqgUYwgRMTP9nQ0MI/vO+jB1A5H+zk7ZZwBdcw7vAsxM6V1qSK9TOQeE3C6j7xeQ/5HUzzaH7M3sCOwN3JfSiIh5ecrYHJgeEW+k7aFAbvR+IP19Ceie1tsB90p6DbiKbPCnVhFxU/Wvh+puk3LQccNOdNxwA7beYVsA9jv0QCa/OhGAh+66n6ce+w+X/PVqqi90aNGiBb+9+FzuG/0o1/3jryxasJBum/QAYM8D9+WuUQ9y5xMP0L3nxnTbpHuDvKamZNjt93DU7gdx3IFHsWD+x7zz5vRl9j9670Pse8jXxySnv/Emn3/6OZv22mxlVXXVVKIxBElrSFqreh3YH3iN7EdqdUQ+FngorQ8HBkhqJakH2eDxC6lbaZGkndLVRQNzjqlVOQeEmlTw1Wtqvdy+xTnrlWS/7kUBzagc+S7bqj5HdfkAFwH/jYjewME11KtR6NBxPTbo0onpU7MG1fNPjWGTzTdl9L+f4pZrbuS6f/yVNqu3WZr/888+X9qlNOa/T9O8eXM22aInAHM/mgPAgo8X8M+b7+CwgUet5FfT9FT/+Nigy4bse0g/Rtw3nI1yAvFe392X6W9kn23nbl2WDiJ36tqZ7j03ZuY7M75WZlNTokHljsBoSa8AL5D9kB0JXALsJ2kqsF/aJiImAvcAk4CRwEkRUd1/Nxi4mWyg+U1gRL6TN7a2+NvA9mQv/AcF5B8FDJN0VUTMldQ+tRIWAWvVkP91oLukTSNiGvBj4Kk852gHvJ/WjyugTmXrrEsv4MyfncaSL7+kS/eNuOj6yzl6r0P58ssvGdT/xwBsveO2nHfVEOZ9NJcTfjAQNWvG+p024OK/fNU/femZFzLltckAnPCbU+i+6cYN8nqakivvuIG1269DxZIKhvz6XBZ+vJDzr7uU7j03JqqqmPne+1z0y7MB2HbnHTn+tMFULKmgqqqKIb86l4/nzW/gV9DwSjG2HhFvAdvUkD4X2KeWY4YAQ2pIHwv0XpHzN7aAcAHwN0m/A57PlzkiJkoaAjwlqRIYR/alfTfw1zT4fHhO/i8k/YSsC6gF8CJwY57TXAYMlfQr4D/f4DWVjS227sU/nxy+TNqj456sMW/nbl3419ia347L/nZtqatmeRzX78ivpf3qx4NryAkP3z2Mh+8uZNiu6QiWDhqXNYUvGVtlbbXdNjH86Ucbuhq2ArZq16Ohq2Ar6NPKJS/l3AD2jbTdbNPo++fL8uYbtf8Pij5XfWpsLQQzswbRGH5bOyCYmZWAA4KZmRE4IJiZGSy9D6HcOSCYmZVAodNbr8ocEMzMSsAtBDMzAxwQzMwM0qhyQ1eieA4IZmZFCjyGYGZmibuMzMwMIogVfGjyqsgBwcysFMo/HjggmJmVgruMzMws44BgZmbgFoKZmUHWOqjMm2uV54BgZlakbLbT8m8iOCCYmZVC+ccDBwQzs6J56gozM6vWCHqMHBDMzErCAcHMzAjAU1eYmRkAnu3UzMwAdxmZmVnGg8pmZtb4LzuVdB11vMSIOKVeamRmVo4a+aDy2JVWCzOzclf+8aD2gBARQ3O3Ja0REZ/Wf5XMzMpQIwgIzfJlkLSzpEnA5LS9jaTr671mZmblovo+hHxLgSQ1lzRO0sNpu72kJyRNTX/Xycl7lqRpkqZIOiAnfXtJE9K+ayUp33nzBgTgauAAYC5ARLwC7F7wKzMzawqigKVwp5J+hCdnAqMioicwKm0jqRcwANgS6AdcL6l5OuYGYBDQMy398p20kIBARLy3XFIjmPnbzKyEShQQJHUBvgfcnJN8KFDdjT8U6J+TfndELI6I6cA0oK+kTkDbiHg2snm5b8s5plaFXHb6nqRdgJC0GnAKy0YuM7OmLVasSyiPq4HfAGvlpHWMiFnZqWKWpPVTemfguZx8M1LakrS+fHqdCmkhnACclAp7H+iTts3MrFphLYQOksbmLINyi5B0EDA7Il4q8Kw1jQtEHel1yttCiIg5wDH562Vm1oQV1kCYExE71LF/V+AQSd8FWgNtJd0BfCipU2oddAJmp/wzgK45x3cBZqb0LjWk16mQq4w2lvQvSR9Jmi3pIUkb5zvOzKxJKcEYQkScFRFdIqI72WDxfyLiR8Bw4NiU7VjgobQ+HBggqZWkHmSDxy+k7qVFknZKVxcNzDmmVoV0Gd0F3AN0AjYE7gX+UcBxZmZNR0T+5Zu7BNhP0lRgv7RNREwk+36eBIwEToqI6ot+BpMNTE8D3gRG5DtJIYPKiojbc7bvkHRyoa/CzKzRC0o+/XVEPAk8mdbnAvvUkm8IMKSG9LFA7xU5Z11zGbVPq/+VdCZwN9nLPgp4ZEVOYmbW6DXy5yG8xLKj1T/P2RfARfVVKTOzstMI5r+uay6jHiuzImZmZauxT3+dS1JvoBfZZVAARMRt9VUpM7Oy0xQCgqTfA3uSBYRHgQOB0WS3QpuZGTSKgFDIZaeHk41ufxARPwG2AVrVa63MzMpJAJWRf1nFFdJl9HlEVEmqkNSW7A4535hmZpZr1f++z6uQgDBW0trAX8muPPoEeKE+K2VmVm7UFAJCRJyYVm+UNJJsStVX67daZmZlpjFfdippu7r2RcTL9VMlM7MyVP7xoM4WwhV17Atg7xLXxZYzdcIb7L/xfg1dDVsB2w3zNF/l5ulDDi++kMZ+H0JE7LUyK2JmVr5K+oCcBlPQjWlmZpZH+ccDBwQzs6LVw2ynDcEBwcysFBrBVUaFPDFNkn4k6by0vZGkvvVfNTOzMlKCJ6Y1tEKmrrge2Bk4Om0vAv5cbzUyMytHjSAgFNJl9O2I2E7SOICImC9ptXqul5lZ+QhQE7nKaImk5qT4Jmk9GsXwiZlZCZV/PCioy+haYBiwvqQhZFNf/6Fea2VmVm6aQpdRRNwp6SWyKbAF9I+IyfVeMzOzctII+k0KeUDORsBnwL9y0yLi3fqsmJlZ2SiTFkA+hYwhPEL2UkX2CM0ewBRgy3qsl5lZeWkKg8oRsVXudpoF9ef1ViMzs3JU/vFgxe9UjoiXJe1YH5UxMytHook8IEfSr3I2mwHbAR/VW43MzMpNNJ3ZTtfKWa8gG1O4v36qY2ZWpso/HtQdENINaWtGxBkrqT5mZuWpMQcESS0ioqKuR2mamRlNYuqKF8jGC8ZLGg7cC3xavTMiHqjnupmZlY/yjwcFjSG0B+aSPUO5+n6EABwQzMyqNfIWwvrpCqPX+CoQVCv/V25mVkKN4bLTuia3aw6smZa1ctarFzMzg8ImtisgYEhqLekFSa9ImijpgpTeXtITkqamv+vkHHOWpGmSpkg6ICd9e0kT0r5rJammc+aqq4UwKyIuzP8SzMxMpXmE5mJg74j4RFJLYLSkEcBhwKiIuETSmcCZwG8l9QIGkE0ltCHwb0mbRUQlcAMwCHgOeBToB4yo6+R1tRDyRhMzM0sqI/+SR2Q+SZst0xLAocDQlD4U6J/WDwXujojFETEdmAb0ldQJaBsRz0ZEALflHFOrugLCPnlrb2Zm2WWnBSxAB0ljc5ZByxclqbmk8cBs4ImIeB7oGBGzANLf9VP2zsB7OYfPSGmd0/ry6XWqtcsoIublO9jMzAAim74ivzkRsUOdJWXdPX0krQ0Mk9S7juw19eQsfxFQbnqdCnlimpmZ5RORf1mh4uJj4Emyvv8PUzcQ6e/slG0G0DXnsC7AzJTepYb0OjkgmJkVqXq20wK6jOouR1ovtQyQ1AbYF3gdGA4cm7IdCzyU1ocDAyS1ktQD6Am8kLqVFknaKV1dNDDnmFqt8PTXZma2nABVluQZmp2AoWkeuWbAPRHxsKRngXskHQ+8CxwBEBETJd0DTCKbfPSk1OUEMBi4FWhDdnVRnVcYgQOCmVlplOCy04h4Fdi2hvS51HKhT0QMAYbUkD4WqGv84WscEMzMSqAx3KnsgGBmVqxvMGi8KnJAMDMrhUY+uZ2ZmRWoRFNXNCgHBDOzYkVAVWX+fKs4BwQzsxJwC8HMzDIOCGZmBkCU5Ma0BuWAYGZWNF92amZmpLmKKj2obGZmhLuMzMwscZeRmZkRuIVgZmbgLiMzM/uKu4zMzCxrIVQ0dCWK5oBgZlakAMJdRmZmliJCQ9eiaA4IZmZF86Cy2VK/uvR0dtr723w892MG9fvZMvsO/9kRDPrdzzl8u8NYOH8hHTt35OZ/38KMt94DYPK4yVx7zjUA7HnwXhx94g8JgrkfzuXS0y5m4fyFK/31NBVRWcXrv/kdLdu3Z9OzfwPA7EdG8tGIx1HzZrTdflu6DDwGgM/efof3bvwblZ9/BmrGFpf9P6KykjfOvmBpeV/OnUv73Xej6/HHNsjraVAOCLWTFMCVEfHrtH06sGZEnF+Css8HfgZ8lJJGRsSZxZabyl4b+GFEXJ+2NwSujYjDS1F+znmeBE5PD8Iue0/c/xjDb3uQ31zx22XS1+u0Htvttj0fvv/hMumz3pnJ4O+dsExas+bNOPG8E/m//Y9n4fyF/N+ZP+PQgf25/Zrb6r3+TdXsR0bQuktnKj/7HIBFEyay4MWX+NZVl9KsZUuWfLwAgKis5O1r/kz3U05i9R7dqFi0CDVvQbPVVuNbV16ytLzJp/+OtXfq2yCvpcE1goDQrB7LXgwcJqlDPZV/VUT0SUtJgkGyNnBi9UZEzCx1MGiMJrwwgUUfL/pa+gnnDubmS24iCuhflQQSrVdvDcDqa67O3NlzS15Xy3w5Zy4LXxpHh333Wpr20WNP0PH7h9CsZUsAWq7dDoCF41+lTbeNWL1HNwBarLUWar7s18cXM2dRsWABa/baYiW9glVIBFFVmXdZ1dVnQKgAbgJOW36HpG6SRkl6Nf3dKKXfKulaSWMkvSWp4C9iScdJ+lPO9sOS9kzrn0gaIukVSc9J6pjSO0oaltJfkbQLcAmwiaTxki6X1F3Sayl/a0l/lzRB0jhJe+Wc+wFJIyVNlXRZTj1ukDRW0kRJX7Wtm4Cd9t2ZOR/M4a3Jb31t3wZdN+D6h2/kj3dfQe8dewNQWVHJdedew19G/JV/PP9PuvXsxsh/jljZ1W4yZtxyG50H/hD01dfA4pkf8Mnk13n9t+fwxjkX8OnUN1P6LCQx9cKLmfzrs/hg2PCvlTd/9BjW2XXnLLA3SVUFLKu2+gwIAH8GjpHUbrn0PwG3RcTWwJ3AtTn7OgG7AQeRfTnX5rT0pT1e0gF56rEG8FxEbAP8j6y7iXTep1L6dsBE4EzgzdTyOGO5ck4CiIitgKOBoZJap319gKOArYCjJHVN6WdHxA7A1sAekrauq6KSBqUAMrayqnyva27VuhU/POmHDL1q6Nf2zftoHsfsegwnHnQCf/l/N3LW1b9j9TVXp3mL5hx0zMGceNAJHP3to3jr9bcYcOLRDVD7xm/B2Jdp0a4tq2+y8TLpUVlJ5SefsvklF9H52GOYfsU1RARRWcUnk6fQ45cnsfkfzmfB82NZ+Opryxw7f/SztP/OLivzZaxCgoiqvMuqrl4HlSNioaTbgFOAz3N27QwcltZvBy7L2fdgZO/cpOpf8rW4KiL+WL0h6bg68n4JPJzWXwL2S+t7AwNTXSuBBZLWqaOc3YDrUv7XJb0DbJb2jYqIBakuk4BuwHvAkZIGkb3XnYBewKu1nSAibiJrWdG6ZZuyvY6tU7cN2aDLBtz46F8AWG+D9bj+Xzfyi/4nMX/OfJZ8uQSAqa9NZea7s+jco8vSX5az3p0FwP8eeYqjThjQMC+gkfvk9SksePFlXnt5PFVLllD52edMv/pPrLZue9beqS+SWKPnpiBRsXARLTu0Z80tv0WLtm0BaLtdHz5/azptt85ad59Nf4eorPxagGlSyuALP5+VcZXR1cDLwN/ryJP7xbc4Z10AkoYA3wOIiD61lFHBsi2e1jnrS+KrTuxKvvnrrqstnFvvSqCFpB7A6cCOETFf0q3L1avRenvKdI7c8Yil27c9fQcnH3IiC+cvpF37diz6eBFVVVVs0LUTnbt35oN3Z9Gy1Wps1LMb7dq3Y8G8BWy32/a8++a7DfgqGq/OPzqazj/KWl+LXpvEhw89TI9fnsxHjz3BogkTWat3L76YOYuoqKBF27Vo22drPhz2L6oWL0YtWvDJpMmsf9CBS8ubP3oM6zTZ1gH4stMCRcQ8SfcAxwO3pOQxwACy1sExwOg8ZZwNnJ3nVG8DJ0pqBnQGCrnUYRQwGLhaUnOyrqVFwFq15P9fqu9/JG0GbARMIetuqklb4FOylkdH4EDgyQLqVXbOuuZ3bL3TNrRbpx13jvkHt189lJH3jKwx71Z9t2bgacdSWVlJVWUV155zNYsWZAPSd1xzO1f880oqKiqZ/f6HXH765SvzZTR56+69F+/8+UYmnXoGatGC7qcMRhIt1lyT9Q/5Lq//5mxAtN2+D+12+Oqf/fwxzy29bLVJCspi0DiflXUfwhXAyTnbpwC3SDqD7NLRn5TgHM8A04EJwGtkrZJ8TgVuknQ82a/6wRHxrKRn0kDyCLJxkGrXAzdKmkDWIjkuIhbXNogWEa9IGkc2NvFWqmOjdPGpf6hz/8Dv/Gjp+uiRTzN65NM15nvkrod55K6Ha9xn9WOt3r1Yq3cvAJq1bEGPX55cY7519/gO6+7xnRr39b7hmnqrX3kIogwGjfNRIZcDWsNo3bJNdG3fo6GrYSug080XNXQVbAU9fcjhL6ULP76xtVbvEH02PzhvvtHjby36XPXJdyqbmRUpm8qo/H9cOyCYmRWtcQwq1/d9CGZmTUNU5V/ykNRV0n8lTU43s56a0ttLeiLd+PpE7uXxks6SNE3SlNx7siRtn26inZZu+M17x6ADgplZsSKIqMy7FKAC+HVEfAvYCThJUi+yG2ZHRURPsqsjzwRI+wYAWwL9gOvTFZMANwCDgJ5p6Zfv5A4IZmYlUIo7lSNiVkS8nNYXAZPJLqM/FKi+7X8o0D+tHwrcHRGLI2I6MA3oK6kT0DYink33YN2Wc0ytPIZgZlYCpZ6aQlJ3YFvgeaBjRMzKzhOzJK2fsnUGnss5bEZKW5LWl0+vkwOCmVnRotAnpnWQlDvl/U1pupplSFoTuB/4ZZoCqLbyatoRdaTXyQHBzKwECmwhzMl3H4KklmTB4M6IeCAlfyipU2oddAJmp/QZQNecw7sAM1N6lxrS6+QxBDOzogWlmP46XQn0N2ByRFyZs2s4UP0YumOBh3LSB0hqleZO6wm8kLqXFknaKZU5MOeYWrmFYGZWpAioKs1cRrsCPwYmSBqf0n5H9iiAe9I0O+8CR2TnjYlprrhJZFconRRfXc40GLgVaEM2DU/eh4s4IJiZFa00N6ZFxGhqn1V5n1qOGQIMqSF9LNB7Rc7vgGBmVgKNYXI7BwQzs1LwXEZmZlb9CM1y54BgZlakEg4qNygHBDOzolVfdlreHBDMzErAXUZmZgb4ATlmZgZ4UNnMzHI4IJiZNXkR4auMzMwsE/lnl17lOSCYmZWAxxDMzAwPKpuZ2Vd82amZmYFnOzUzM3yVkZmZ5XALwczMAE9dYWZmgK8yMjMzIJv82i0EMzODCKrCg8pmZobvVDYzs8RdRmZmRhC+7NTMzDJuIZiZGeAxBDMzg3SVkQOCmVmTl92H4IBgZmZ4DMHMzABPXWFmZku5hWBmZhCNYwyhWUNXwMys3AXZXEb5lkJIukXSbEmv5aS1l/SEpKnp7zo5+86SNE3SFEkH5KRvL2lC2netJOU7twOCmVkJRETepUC3Av2WSzsTGBURPYFRaRtJvYABwJbpmOslNU/H3AAMAnqmZfkyv8YBwcysBCKq8i6FlRP/A+Ytl3woMDStDwX656TfHRGLI2I6MA3oK6kT0DYino0sEt2Wc0ytPIZgZla0bDajAnSQNDZn+6aIuKmA4zpGxCyAiJglaf2U3hl4LiffjJS2JK0vn14nBwQzsyKtwANy5kTEDiU8dU3jAlFHep0cEMzMihVQWb8PyPlQUqfUOugEzE7pM4CuOfm6ADNTepca0uvkgLAKW1zxxZxpsye/09D1qCcdgDkNXYlSm3bI4Q1dhfrSKD+vpFuxBVQRj31asaRDAVm/6Xs4HDgWuCT9fSgn/S5JVwIbkg0evxARlZIWSdoJeB4YCFyX7yRqDDdTWPmRNLbETWerR/68Vh5J/wD2JAvCHwK/Bx4E7gE2At4FjoiIeSn/2cBPgQrglxExIqXvQHbFUhtgBPCLyPOF74BgDcJfMOXFn1fT4MtOzcwMcECwhlPIpXa26vDn1QS4y8jMzAC3EMzMLHFAMDMzwAHBaiCpUtL4nKV7Ccvunybkqt6+UNK+pSo/lbmnpIdLWebKJikkXZGzfbqk80tU9vmS3s/5fC8pRbmp7LUlnZizvaGk+0pVfk65T6bLKq2EfGOa1eTziOhTT2X3Bx4GJgFExHn1dJ5ytxg4TNLFEVEfN4RdFRF/rIdy1wZOBK4HiIiZQKO9W6+xcQvBCiLpbUkd0voOkp5M6+en+duflPSWpFNyjhko6VVJr0i6XdIuwCHA5emX6SaSbpV0eMq/j6RxaQ73WyS1yjn3BZJeTvu2SOl9JY1Jx4yRtPlKflvqUwXZlT2nLb9DUjdJo9J7O0rSRin91jTv/Zj0WRT8RSzpOEl/ytl+WNKeaf0TSUPS5/icpI4pvaOkYSn9lfT5XgJskj7fyyV1r57XX1JrSX9Pn+E4SXvlnPsBSSOVzfd/WU49bpA0VtJESRes+NtoK8IBwWrSJqc7YVgB+bcADgD6Ar+X1FLSlsDZwN4RsQ1wakSMIbvV/oyI6BMRb1YXIKk12V2VR0XEVmSt18E555gTEduRzfF+ekp7Hdg9IrYFzgP+UMRrXhX9GThGUrvl0v8E3BYRWwN3Atfm7OsE7AYcRPblXJvTcj7jA+rIB7AG8Fz6HP8H/CylXws8ldK3AyaSzdP/Zvp8z1iunJMA0ud7NDA0fe4AfYCjgK2AoyRVz89zdrohbmtgD0lb56mrFcEBwWryefoP3Scivl9A/kfSfOxzyCbd6gjsDdxX3d1RfZt9HTYHpkfEG2l7KLB7zv4H0t+XgO5pvR1wb/oFehXZQ0IajYhYSDaP/SnL7doZuCut304WAKo9GBFVETGJ7HOozVU5n/FjearyJVk3Hyz7/u9NFqCJiMqIWJCnnN1SfYmI14F3gM3SvlERsSAiviDrTqyeX+hISS8D48g+315YvXFAsEJV8NW/l9bL7Vucs15J9uteFDDdbo58j/erPkd1+QAXAf+NiN7AwTXUqzG4Gjie7Fd6bXLf59zPQgCpu2e8pPF1lJH7+cKy7+WSnDlwct//FVXXZ/y1f0OSepC1BvdJraFHaJyf8SrDAcEK9TawfVr/QQH5R5H9ulsXsmfCpvRFwFo15H8d6C5p07T9Y+CpPOdoB7yf1o8roE5lJ7Ws7iELCtXGkD02EeAYYHSeMs6ubg3Uke1toI+kZqm7pm8B1RtF6taT1FxSW2r/fCHrbjom5d+MbKK2KXWU3xb4FFiQxi0OLKBOVgQHBCvUBcA1kp4m+wVXp4iYCAwBnpL0CnBl2nU3cEYaVNwkJ/8XwE/IuoAmAFXAjXlOcxlwsaRngOZ58pazK8hmvqx2CvATSa+SBc5TS3COZ4DpwATgj8DLBRxzKrBX+rxeAraMiLnAM5Jek3T5cvmvB5qn/P8EjouIxdQiIl4h6yqaCNyS6mj1yFNXmJkZ4BaCmZklDghmZgY4IJiZWeKAYGZmgAOCmZklDghW1vTVzKyvSbpX0upFlJU7r9LNypmVtYa8e6a5e1b0HEvnhCokfbk8n6zguc6XdHr+nGYZBwQrd9XTbPQmm2LhhNydkr7R/QkR8X9p+ofa7AmscEAwW5U5IFhj8jSwafr1/l9JdwET0l20l0t6Mc0Q+nMAZf4kaZKkR4D1qwtSznz7kvopm2n1FWWzi3YnCzzVE8R9R9J6ku5P53hR0q7p2HUlPZ5uxPsL+afoQNKDkl5KM3wOWm7fFakuoyStl9I2STOFviTpaaXZYM1WlJ+HYI2CpBZkUxuMTEl9gd4RMT19qS6IiB2VTan9jKTHgW3JJtXbimwiuElkd8Tmlrse8FeyWVWnS2ofEfMk3Qh8Uv1MgRR8roqI0cqmo34M+Bbwe2B0RFwo6XvAMl/wtfhpOkcb4EVJ96c7gNcAXo6IX0s6L5V9Mtk02SdExFRJ3ya7I3jvb/A2WhPngGDlrk3OpG1PA38j68p5ISKmp/T9ga311fMB2gE9yWZT/UdEVAIzJf2nhvJ3Av5XXVYds7buC/SSljYA2kpaK53jsHTsI5LmF/CaTpFUPcts11TXuWTTefwzpd8BPCBpzfR67805d6sCzmH2NQ4IVu6+9nS39MX4aW4S8Ivlp3mW9F3yz8ha6KytzYCdI+LzGupS8Pwwyh5Ks28q6zNlDyKqbYbPSOf9uB6fcGdNiMcQrCl4DBgsqSVkM21KWoNs9s0BaYyhE7BXDcc+S/Zglh7p2NpmbX2crPuGlK9PWs2d4fNAYJ08dW0HzE/BYAuyFkq1Znz1OMofknVFLQSmSzoinUOStslzDrMaOSBYU3Az2fjAy8oepvMXstbxMGAq2QyfN1DDdNsR8RFZv/8DadbW6i6bfwHfrx5UJpuBdIc0aD2Jr652ugDYXdlDXvYH3s1T15FkzwJ4lex5D8/l7PsU2FLSS2RjBBem9GOA41P9JgKHFvCemH2NZzs1MzPALQQzM0scEMzMDHBAMDOzxAHBzMwABwQzM0scEMzMDHBAMDOz5P8DB517rcUQbUAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Pipeline for preprocessing and modeling \n",
    "rf1 = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor1),\n",
    "    ('classifier', RandomForestClassifier(random_state=396))\n",
    "])\n",
    "#Putting our random forest through our evaluate function\n",
    "evaluate(rf1, X_train, X_test, y_train, y_test, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results**\n",
    "- Random Forest is performing better than a single decision tree on all metrics. \n",
    "- Like the decision tree, this model is also overfit on the training data but performs well on testing data.\n",
    "- **Note:** Not presented in this notebook is further hyperparameter tuning to try to improve metric scores. Ultimately the default hyperparameters performed best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the model through our other preprocessor to evaluate differences with the type of categorical preprocssing. Let's see if CatBoost does any better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of Cross-Validation:\n",
      "\n",
      "Average accuracy: 0.8017718991764411        +/- 0.003928224081846972\n",
      "Average precision: 0.8253557646774837        +/- 0.006802787290684527\n",
      "**Average recall: 0.7181249180439877        +/- 0.007776437575354086\n",
      "Average f1 score: 0.7679744663185317        +/- 0.004911935790124266\n",
      "Average roc_auc: 0.8786947891492594        +/- 0.0025936439275698344\n",
      "\n",
      "++++++++++++++++++++\n",
      "\n",
      "Results of Train-Test Split Validation:\n",
      "\n",
      "Training Scores:\n",
      "Train accuracy: 0.865710007486898\n",
      "Train precision: 0.8979126901052891\n",
      "**Train recall: 0.7966240576860046\n",
      "Train f1 score: 0.8442411786841116\n",
      "Train roc_auc: 0.8602216042886924\n",
      "\n",
      "<><><><><><><><><><>\n",
      "\n",
      "Testing Scores:\n",
      "Test accuracy: 0.8046717077187991\n",
      "Test precision: 0.8297074268567142\n",
      "**Test recall: 0.7222857142857143\n",
      "Test f1 score: 0.7722789560967094\n",
      "Test roc_auc: 0.7983663084702908\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnTklEQVR4nO3deZgU1dn+8e8NKqCCgiBBUME1EXcJcYsbGjAuqFEhMRGNCQkx6uv2Rl9/bjEYEzUmmOCWGMG4RI0oUSEqLlFRERBEcAEFFUVWF3BBmHl+f9QZaHBmurF7mOmZ+3NddU3VqVOnTndDP13nnDqliMDMzKxZfVfAzMwaBgcEMzMDHBDMzCxxQDAzM8ABwczMknXquwJWs3bt20XnLTav72rYGpj24pT6roKtoUpiQUR0KKaM/Q85MBYtXJQ338svvvSfiOhTzLnqkgNCA9Z5i80Z+dRD9V0NWwM7bdStvqtga+iTimVvFVvGooWLGPn0qLz5ttqgc/tiz1WXHBDMzIoWVEZlfVeiaA4IZmZFCmB5ZUV9V6NoDghmZsUKXyGYmRnZFUIlDghmZgZUNoJ54RwQzMyK5iYjMzMjNRk5IJiZWUSwPMp/lJGnrjAzK4HKqMy7FELSxpLukfSqpFck7SWpnaRHJE1Pf9vm5D9f0gxJr0nqnZO+h6Qpad8QScp3bgcEM7MSqIzIuxToT8DoiPg6sAvwCnAeMCYitgXGpG0k7QD0B7oDfYChkpqncq4DBgLbpiXvlBkOCGZmRYrUqVzsFYKkNsB+wN8AIuKLiPgQ6AsMS9mGAUel9b7AnRGxNCJmAjOAnpI6AW0i4tnIHos5POeYGjkgmJmVQIEBob2k8TnLwNWK2QqYD/xd0ouS/ippA6BjRMwBSH83Tfk7A+/kHD87pXVO66un18qdymZmRVqDTuUFEdGjlv3rALsDp0XE85L+RGoeqkF1/QJRS3qtfIVgZlakACqiMu9SgNnA7Ih4Pm3fQxYg5qZmINLfeTn5c+fI7wK8l9K7VJNeKwcEM7MSKEWnckS8D7wjafuU1AuYBowEBqS0AcD9aX0k0F9SC0ndyDqPx6VmpcWS9kyji07MOaZGbjIyMytalHIuo9OA2yStB7wJnEz24/0uSacAbwPHAUTEVEl3kQWN5cCpESvargYBtwCtgFFpqZUDgplZkbI7lUszl1FETAKq62foVUP+wcDgatLHAzuuybkdEMzMSqDAPoIGzQHBzKxI2SgjBwQzsyavapRRuXNAMDMrAT8PwczMsiuE/Pd9NXgOCGZmRVujyesaLAcEM7MiVQJfNILnITggmJkVK9yHYGZmZNNfVzggmJkZuFPZzMyomrqivmtRPAcEM7MScJORmZkRwBe+U9nMzCLcqWxmZlTdqVz+HBDMzErAVwhmZpZmO3VAMDMzoKL844EDgplZsbK5jMo/IjggmJkVK9xkZGZmeJSRmZnlcB+CmZmlUUb1XYviOSCYmRUpgGXlP3OFA4KZWbE8dYWZma3g6a/NzAwQlaH6rkTRHBDMzIrkYadmZraCm4zMzIwIWO5RRmZm1lieqdysvitgZtYYRGX+pRCSZkmaImmSpPEprZ2kRyRNT3/b5uQ/X9IMSa9J6p2TvkcqZ4akIZLy9no7IFjJfPzhx5x14iCO+GYvjux5MJPGTeTa31zNMXv34dh9v8vAo3/EvDlzARj7+FMcv/8RHL13H47f/wief3Lsl8o7rf9POHqv3l9Kt9I7YdDJ3Pvcfxjx/MP88Bc/BqBN24248b5beeDFx7nxvltps3EbANZZZx1+c/3V3PvsaO5/4VFOOesX9Vn1BqMy8i9r4MCI2DUieqTt84AxEbEtMCZtI2kHoD/QHegDDJXUPB1zHTAQ2DYtffKdtGwDgqSKFEGrlq4lLPuo9EZXbf9a0sGlKj+VeYCkB0pZZn373XmXss/B+/PvF8bwr6cfYqvttuHk0wdy79jR3PP0Q+zf+yCu//0QANq2a8ef7/wrI8aOZvB1V/F/Pz9rlbIeHTmaVhtuUB8vo8nZ5hvb8b0B/fnBgX05du9D2b/3QWyxdVdOOXMQzz85lsN3O5DnnxzLKWdmX/zfOfq7rNdiPY7Zqw/99juc407+AZtt0aWeX0X9iihsKUJfYFhaHwYclZN+Z0QsjYiZwAygp6ROQJuIeDYiAhiec0yNyjYgAJ+lCFq1zCph2UcBKwJCRFwUEY+WsPxGZ8nHi5kwdhzH/KgfAOuutx5tNm7Dhm1ar8jz2aefUXXV+o1durNpp45A9oW09POlfLF0KQCfLvmE4UP/xs/O+eVafhVN01bbb8NLL7zI5599TkVFBeOfeZ5eh/fmwMMO4f7b7wHg/tvv4cDDDwGyL7ZW67eiefPmtGjVkmXLvmDJ4sX1+RIahBJeIQTwsKQJkgamtI4RMQcg/d00pXcG3sk5dnZK65zWV0+vVTkHhC9JbW/t03oPSU+k9Usk3SzpCUlvSjo955gTJb0kabKkWyXtDRwJXJmuPLaWdIukY1P+XpJeTG1zN0tqkXPuSyVNTPu+ntJ7Shqbjhkrafu1/LasFbNnvUPb9u34f784l+O+fRgXn/YrPv3kUwCGXHYlB3ffmwfvvp9T/+/MLx37yMhRfH3n7qzXogUA1w7+AwNO/QktW7Vaq6+hqZo+7TX22KcnG7XbmJatWvLt7xzI17p0YpMOHVgwdz4AC+bOZ5P27QF45L6H+OzTz3hs+jgenjqWYUNu4uMPPqrPl9AgVFbmX4D2ksbnLAOrKWqfiNgdOBQ4VdJ+tZy2un6BqCW9VuUcEFrlNBeNKCD/14HeQE/gYknrSuoOXAAcFBG7AGdExFhgJHBuuvJ4o6oASS2BW4B+EbET2SitQTnnWJA+yOuAc1Laq8B+EbEbcBFweW2VlDSw6h/LogULC3hZDUNFxXJemTyVfqecwN1PPUir9dfnb9dcB8DpF57Lo1PHcthxfbnjxuGrHDfjlde55uLfcfEfBwPw6kvTeOfNWfQ6wn0Ha8vM19/g5muu58b7/sH19w7jtSmvULG85tusdtxjFyorKui13bc4dKdvc+JpP6FL183XYo0bnqpRRgVcISyIiB45y41fKivivfR3HjCC7DtrbmoGIv2dl7LPBnLf/C7Aeym9SzXptSrngJDbZHR0AfkfTO1sC8jezI7AQcA9KY2IWJSnjO2BmRHxetoeBuRG73vT3wlA17S+EXC3pJeBa8g6f2oUETdW/WNp136TAl5Ww9Bxs0503Oxr7NxjNwAO6Xsor7w0dZU83z32SB799+gV2++/O4f/+eHPuPz6q9m825YATH5hItMmv0zvnfblxEOPY9aMmZx8WP+190KaqBG33kW//Q7npEP78dEHH/LWGzNZOH8+7Tt2AKB9xw4sXLAAgMOO78vTjz7J8uXLWbRgIZOem0D33Xauz+rXvxL1IUjaQFLrqnXgO8DLZD9SB6RsA4D70/pIoL+kFpK6kXUej0vNSosl7ZlGF52Yc0yNyjkgVGc5K19Ty9X2Lc1ZryD7dS8KuIzKkW/YVtU5qsoHuAx4PCJ2BI6opl6NQvuOHfhal07MnJ5dUD3/5Fi23n4b3npj5oo8j496lG7bbgVkI5JOPf7HnHHR/7Lbnj1W5Ol3yg957NXn+c+Upxk+6m66btONvz9459p9MU1Q1Y+Pr3XZjIOP7MOoe0byxEOP0vcHxwLQ9wfH8viDjwAw5533+NZ+ewNZX8LO39yNma+/UX3BTUiJOpU7Ak9LmgyMI/shOxq4AjhE0nTgkLRNREwF7gKmAaOBUyOi6vJuEPBXso7mN4BR+U7e2G5MmwXsQfbCv1dA/jHACEnXRMRCSe3SVcJioHU1+V8FukraJiJmAD8Cnsxzjo2Ad9P6SQXUqWyd/7tLOe+nZ7Lsiy/o0nULLht6JZecdh6zZryJJDbbvDMXXpM1Dd1x0zDemfkWN1x5LTdceS0AN4wYziYd2tfnS2iy/vCP69i4XVuWL1vO4LMv5OMPP+Zv11zHVbf8haNPPJ4577zH2QOyUUZ33DSc3wy9khHPP4wk7vvH3bw+9dV6fgX1rxSzX0fEm8Au1aQvBHrVcMxgYHA16eOBHdfk/IoyncNb0pKI2HC1tG8DfwPmAs8DPSLiAEmXAEsi4qqU72Xg8IiYJWkAcC7Zr/oXI+IkSfsAN5H94j8WuBB4ICLukdQLuIosmL4ADIqIpZJmpfMtkNQDuCqdey+ypqX5wGPAjyKiq6QDgHMi4vCaXuNOu+8SI596qATvlq0tO23Urb6rYGvok4plE3LG+38lbbbfJr419Pd58z168PeKPlddKtuA0BQ4IJQfB4TyU5KAsN020fMv+QPCmO807IDQ2JqMzMzqRWP4be2AYGZWAg4IZmZG4IBgZmaw4j6EcueAYGZWAoVOb92QOSCYmZWArxDMzAxwQDAzM0i9yvVdieI5IJiZFSlwH4KZmSVuMjIzM4gg1vChyQ2RA4KZWSmUfzxwQDAzKwU3GZmZWcYBwczMwFcIZmYG2dVBRd5cDZ4DgplZkbLZTsv/EsEBwcysFMo/HjggmJkVzVNXmJlZlUbQYuSAYGZWEg4IZmZGAJ66wszMAPBsp2ZmBrjJyMzMMu5UNjOzxj/sVNK11PISI+L0OqmRmVk5auSdyuPXWi3MzMpd+ceDmgNCRAzL3Za0QUR8UvdVMjMrQ40gIDTLl0HSXpKmAa+k7V0kDa3zmpmZlYuq+xDyLQ1c3oAA/BHoDSwEiIjJwH51WCczs/ITBSwFktRc0ouSHkjb7SQ9Iml6+ts2J+/5kmZIek1S75z0PSRNSfuGSFK+8xYSEIiId1ZLagQzf5uZlVAJAwJwBqlVJjkPGBMR2wJj0jaSdgD6A92BPsBQSc3TMdcBA4Ft09In30kLCQjvSNobCEnrSTpntYqamTVtUUBzUYFNRpK6AIcBf81J7gtU9esOA47KSb8zIpZGxExgBtBTUiegTUQ8G9mDGobnHFOjQgLCz4FTgc7Au8CuadvMzKoUdoXQXtL4nGVgNSX9EfhfVp0Mo2NEzAFIfzdN6Z2B3Bac2Smtc1pfPb1WeW9Mi4gFwAn58pmZNWmFXQAsiIgeNe2UdDgwLyImSDqggPKq6xeIWtJrVcgoo60k/VvSfEnzJN0vaasCKmpm1nSUpg9hH+BISbOAO4GDJP0DmJuagUh/56X8s4HNc47vAryX0rtUk16rQpqMbgfuAjoBmwF3A3cUcJyZWdMRkX/JW0ScHxFdIqIrWWfxYxHxQ2AkMCBlGwDcn9ZHAv0ltZDUjazzeFxqVlosac80uujEnGNqVEhAUETcGhHL0/IPGsUtGGZmJRJkLf75lq/uCuAQSdOBQ9I2ETGV7Af7NGA0cGpEVI0CHUTWMT0DeAMYle8ktc1l1C6tPi7pPLLLlwD6AQ9+hRdkZtZ4lfh5CBHxBPBEWl8I9Koh32BgcDXp44Ed1+SctXUqT2DVzomf5Z4LuGxNTmRm1qg1gvmva5vLqNvarIiZWdlq7NNf55K0I7AD0LIqLSKG11WlzMzKTlMICJIuBg4gCwgPAYcCT5Pd+WZmZtAoAkIho4yOJevMeD8iTgZ2AVrUaa3MzMpJABWRf2ngCmky+iwiKiUtl9SG7IYI35hmZpar4X/f51VIQBgvaWPgJrKRR0uAcXVZKTOzcqOmEBAi4hdp9XpJo8lm0HupbqtlZlZmGvOwU0m717YvIibWTZXMzMpQ+ceDWq8Qrq5lXwAHlbgutpo3pr7J0d3713c1bA10H357fVfB1tC4E44rvpDGfh9CRBy4NitiZla+yuOZyfkUdGOamZnlUf7xwAHBzKxoVbOdljkHBDOzUmgEo4wKeWKaJP1Q0kVpewtJPeu+amZmZaQ0T0yrV4VMXTEU2Av4ftpeDPylzmpkZlaOGkFAKKTJ6FsRsbukFwEi4gNJ69VxvczMykeAmsgoo2WSmpPim6QONIruEzOzEir/eFBQk9EQYASwqaTBZFNfX16ntTIzKzdNockoIm6TNIFsCmwBR0XEK3VeMzOzctII2k0KeUDOFsCnwL9z0yLi7bqsmJlZ2SiTK4B8CulDeJDspYrsEZrdgNeA7nVYLzOz8tIUOpUjYqfc7TQL6s/qrEZmZuWo/OPBmt+pHBETJX2zLipjZlaORBN5QI6ks3I2mwG7A/PrrEZmZuUmms5sp61z1peT9Sn8q26qY2ZWpso/HtQeENINaRtGxLlrqT5mZuWpMQcESetExPLaHqVpZmY0iakrxpH1F0ySNBK4G/ikamdE3FvHdTMzKx/lHw8K6kNoBywke4Zy1f0IATggmJlVaeRXCJumEUYvszIQVCn/V25mVkKNYdhpbZPbNQc2TEvrnPWqxczMoLCJ7QoIGJJaShonabKkqZIuTentJD0iaXr62zbnmPMlzZD0mqTeOel7SJqS9g2RpOrOmau2K4Q5EfHr/C/BzMxUmkdoLgUOioglktYFnpY0CjgGGBMRV0g6DzgP+JWkHYD+ZFMJbQY8Kmm7iKgArgMGAs8BDwF9gFG1nby2K4S80cTMzJKKyL/kEZklaXPdtATQFxiW0ocBR6X1vsCdEbE0ImYCM4CekjoBbSLi2YgIYHjOMTWqLSD0ylt7MzPLhp0WsADtJY3PWQauXpSk5pImAfOARyLieaBjRMwBSH83Tdk7A+/kHD47pXVO66un16rGJqOIWJTvYDMzA4hs+or8FkREj1pLypp7dpW0MTBC0o61ZK+uJWf1QUC56bUq5IlpZmaWT0T+ZY2Kiw+BJ8ja/uemZiDS33kp22xg85zDugDvpfQu1aTXygHBzKxIVbOdFtBkVHs5Uod0ZYCkVsDBwKvASGBAyjYAuD+tjwT6S2ohqRuwLTAuNSstlrRnGl10Ys4xNVrj6a/NzGw1AaooyTM0OwHD0jxyzYC7IuIBSc8Cd0k6BXgbOA4gIqZKuguYRjb56KmpyQlgEHAL0IpsdFGtI4zAAcHMrDRKMOw0Il4CdqsmfSE1DPSJiMHA4GrSxwO19T98iQOCmVkJNIY7lR0QzMyK9RU6jRsiBwQzs1Jo5JPbmZlZgUo0dUW9ckAwMytWBFRW5M/XwDkgmJmVgK8QzMws44BgZmYAREluTKtXDghmZkXzsFMzMyPNVVThTmUzMyPcZGRmZombjMzMjMBXCGZmBm4yMjOzldxkZGZm2RXC8vquRNEcEMzMihRAuMnIzMxSRKjvWhTNAcHMrGjuVDZb4Ze/+Tk99t+djxZ9zBl9zwHg7KvPoHO3zQDYoPX6fLL4U8465lcAHPPTozj4ewdSWVHJXy+/hUnPTAbghDP6ccCR+7HBRhvygx4D6ufFNCFRWcmbF1/AOm3bseVZ565IX/DQA8z95+1s/+frWad1G5a8PIW5d91BVFSg5s3p2P8ENtyh+yplvX3NVXwxfx7bXP77tf0yGoZGEBCa1VXBkkLS1Tnb50i6pERlXyLpXUmT0nJFKcpNZW8s6Rc525tJuqdU5eeU+4SkHqUut748NuJJfj3wt6ukXX32nzjrmF9x1jG/4tlHxvHcI+MA6LJ1Z/Y9dG9OP+Jsfj3wcn524Y9p1kwAvPD4RP633wVrvf5N1cKHR9Fis86rpC1buJAlU6ew7ibtV6Q1b92aLc48l20G/47OAwfx7g1DVznm4/HjaNay5Vqpc4MVlfmXBq7OAgKwFDhGUvu8Ob+aayJi17ScV8JyNwZWBISIeC8iji1h+Y3StAmvsPijJTXu36f3njz10DMA9Dzomzw9aizLly1n3rvzmfP2XLbdaRsAXn9pOh8s+HBtVLnJW7ZoIUsmT2Lj/Q9cJf3922/la/1+AFqZ1mrLrqzbti0ALTp3IZYto3LZMgAqPv+chaMfov2RR62tqjc8EURlRd6loavLgLAcuBE4c/UdkraUNEbSS+nvFin9FklDJI2V9Kakgr+IJZ0k6c852w9IOiCtL5E0WNJkSc9J6pjSO0oakdInS9obuALYOl15XCmpq6SXU/6Wkv4uaYqkFyUdmHPueyWNljRd0u9z6nGdpPGSpkq6dM3fxvK3wx7f4MOFHzHnrfcB2GTTtix8f8GK/QvnLqRdx3b1Vb0m6/3bbqXj8d9HWvnN//HECazTti0tt9iyxuM+Hj+OlltuSbN11wVg3r/uZpM+h9FsvRZ1XueGrbKApWGry4AA8BfgBEkbrZb+Z2B4ROwM3AYMydnXCdgXOJzsy7kmZ+Y0GfXOU48NgOciYhfgv8BPU/oQ4MmUvjswFTgPeCNdeZy7WjmnAkTETsD3gWGSqq6TdwX6ATsB/SRtntIviIgewM7A/pJ2rq2ikgamADJ+ecWyPC+rPHz7sL156qGxK7Zzv4CqRCMYoVFOFk+aSPM2bWjVbasVaZVLl7Lg3/ex6THH1Xjc57NnM/efd7DZST8B4LO3ZvHFvPdp0+ObdV7nhi2IqMy7NHR12qkcER9LGg6cDnyWs2sv4Ji0fiuQ2wt1X2Tv3LSqX/I1uCYirqrakHRSLXm/AB5I6xOAQ9L6QcCJqa4VwEeS2tZSzr7AtSn/q5LeArZL+8ZExEepLtOALYF3gOMlDSR7rzsBOwAv1XSCiLiR7MqK9Vu0LvtvyWbNm7HnwT0557jzV6QtmLuITb62siVxk46b8MG8D+qjek3Wp6+/zuIXJ/L6S5OIZcuo+OwzZt84lC/mz+eNC7MW2GWLFvHmRRfQ7eLLWHfjjVm2aCHvDPkDXQYOYr2O2X/Nz2ZM5/NZM3n97NOJikoqPv6Imb+9jG7nX1ifL69+lMEXfj5rY5TRH4GJwN9ryZP7xbc0Z10AkgYDhwFExK41lLGcVa94cnu4lsXKn6AVfPXX/eWftivl1rsCWEdSN+Ac4JsR8YGkW1arV6O3y1478e7M91g4d9GKtBceH8+Zvz+dkbc8QLtN29Jpy68xfcqMeqxl09Px+P50PL4/AJ+8Mo0Fox5ki9NWbd19/ezT2eqS37BO6zZUfPIJb/3hSjoe15/1t9t+RZ52vQ6hXa/s99UX8+fz9jVXNs1g0EiGndZ1kxERsQi4CzglJ3ks0D+tnwA8naeMC6o6kGvJNgvYVVKz1FzTs4DqjQEGAUhqLqkNsBhoXUP+/6b6Imk7YAvgtVrKbwN8Qnbl0RE4tIA6laWzrjydK+64jM26duKmx4bS65iso3LfQ/de0Zlc5Z0Zsxn7n2e59t9Xc9GN/8dNv7mZysosXp949gnc9NhQWrRcj5seG0q/U92f3xAsevRhvpg7l/kjR/DGhefzxoXns/zjj+q7Wg1H0Cg6ldfWfQhXA7/M2T4duFnSucB84OQSnOMZYCYwBXiZ7KoknzOAGyWdQvarflBEPCvpmdSRPIqsH6TKUOB6SVPIrkhOioil1bWJA0TEZEkvkvVNvJnq2Cj94dwh1aZfe8F11abfc8MI7rlhxJfSh199G8Ovvq2kdbPabfCNHdjgGzt8KX27q1d+ph36Hk2HvkfXWs56HTo03XsQCKIMOo3zkTvzGq71W7SO7TrtWt/VsDXQ4vIz6rsKtobGnXDchDTw4ytrvX772HX7I/Lme3rSLUWfqy75TmUzsyJlUxmV/49rBwQzs6K5U9nMzKqUYOoKSZtLelzSK+lm1jNSejtJj6QbXx/JHR4v6XxJMyS9lntPlqQ90k20M9INv7WNkgQcEMzMihdBREXepQDLgbMj4hvAnsCpknYgu2F2TERsSzY68jyAtK8/0B3oAwyV1DyVdR0wENg2LX3yndwBwcysBEpxp3JEzImIiWl9MfAK0BnoCwxL2YYBR6X1vsCdEbE0ImYCM4CekjoBbSLi2XQP1vCcY2rkPgQzsxIocGqK9pLG52zfmGYn+BJJXYHdgOeBjhExJztPzJG0acrWGXgu57DZKW1ZWl89vVYOCGZmRYtCn5i2oJBhp5I2BP4F/E+aAqjGrNVXpsb0WjkgmJmVQKkmr5O0LlkwuC0i7k3JcyV1SlcHnYB5KX02sHnO4V2A91J6l2rSa+U+BDOzogWlmP46jQT6G/BKRPwhZ9dIoOoRggOA+3PS+0tqkeZO2xYYl5qXFkvaM5V5Ys4xNfIVgplZkSKgsjRzFe0D/AiYImlSSvs/skcB3JWm2XkbOC47b0yVdBcwjWyE0qmxcjjTIOAWoBXZNDyj8p3cAcHMrGiluTEtIp6m5lmVe9VwzGBgcDXp44Ed1+T8DghmZiXQGCa3c0AwMysFz2VkZmZVj9Asdw4IZmZFKmGncr1yQDAzK1rVsNPy5oBgZlYCbjIyMzPAD8gxMzPAncpmZpbDAcHMrMmLCI8yMjOzTOSfXbrBc0AwMysB9yGYmRnuVDYzs5U87NTMzMCznZqZGR5lZGZmOXyFYGZmgKeuMDMzwKOMzMwMyCa/9hWCmZlBBJXhTmUzM8N3KpuZWeImIzMzIwgPOzUzs4yvEMzMDHAfgpmZQRpl5IBgZtbkZfchOCCYmRnuQzAzM8BTV5iZ2QqN4QqhWX1XwMys7EXWh5BvKYSkmyXNk/RyTlo7SY9Imp7+ts3Zd76kGZJek9Q7J30PSVPSviGSlO/cDghmZkUKsrmM8i0FugXos1raecCYiNgWGJO2kbQD0B/ono4ZKql5OuY6YCCwbVpWL/NLHBDMzEogIvIuBZbzX2DRasl9gWFpfRhwVE76nRGxNCJmAjOAnpI6AW0i4tnITjw855gauQ/BzKwECmwSai9pfM72jRFxYwHHdYyIOdl5Yo6kTVN6Z+C5nHyzU9qytL56eq0cEMzMipbNZlSABRHRo4Qnrq5fIGpJr5WbjMzMilT1gJxSNBnVYG5qBiL9nZfSZwOb5+TrAryX0rtUk14rBwQzs2IFVERF3qUII4EBaX0AcH9Oen9JLSR1I+s8HpealxZL2jONLjox55gaqTGMnW2sJM0H3qrvetSR9sCC+q6EFawxf15bRkSHYgqQNJrsPcpnQUTUOtpH0h3AAam8ucDFwH3AXcAWwNvAcRGxKOW/APgxsBz4n4gYldJ7kI1YagWMAk6LPF/4DghWLySNL3FbqtUhf15Ng5uMzMwMcEAwM7PEAcHqSyFjr63h8OfVBLgPwczMAF8hmJlZ4oBgZmaAA4JVQ1KFpEk5S9cSln1UmqGxavvXkg4uVfmpzAMkPVDKMtc2SSHp6pztcyRdUqKyL5H0bs7ne0Upyk1lbyzpFznbm0m6p1Tl55T7RBpnbyXkuYysOp9FxK51VPZRwAPANICIuKiOzlPulgLHSPptRNTFDWHXRMRVdVDuxsAvgKEAEfEecGwdnMfqgK8QrCCSZklqn9Z7SHoirV+SHujxhKQ3JZ2ec8yJkl6SNFnSrZL2Bo4Erky/TLeWdIukY1P+XpJeTA/1uFlSi5xzXyppYtr39ZTeU9LYdMxYSduv5belLi0nG9lz5uo7JG0paUx6b8dI2iKl35IehDI2fRYFfxFLOknSn3O2H5B0QFpfImlw+hyfk9QxpXeUNCKlT06f7xXA1unzvVJS16oHvUhqKenv6TN8UdKBOee+V9JoZQ+A+X1OPa6TNF7SVEmXrvnbaGvCAcGq0yqnOWFEAfm/DvQGegIXS1pXUnfgAuCgiNgFOCMixpLNvXJuROwaEW9UFSCpJdlt9v0iYieyq9dBOedYEBG7kz3045yU9iqwX0TsBlwEXF7Ea26I/gKcIGmj1dL/DAyPiJ2B24AhOfs6AfsCh5N9OdfkzJzPuHct+QA2AJ5Ln+N/gZ+m9CHAkyl9d2Aq2YNb3kif77mrlXMqQPp8vw8MS587wK5AP2AnoJ+kqgnbLkh3SO8M7C9p5zx1tSI4IFh1Pkv/oXeNiKMLyP9gekDHArJZGDsCBwH3VDV3VM27UovtgZkR8XraHgbsl7P/3vR3AtA1rW8E3J1+gV5D9tSoRiMiPiZ7sMnpq+3aC7g9rd9KFgCq3BcRlRExjexzqMk1OZ/xf/JU5QuyZj5Y9f0/iCxAExEVEfFRnnL2TfUlIl4lm6dru7RvTER8FBGfkzUnbpnSj5c0EXiR7PPdAaszDghWqOWs/PfScrV9S3PWK8h+3YsC5l/Pke95r1XnqCof4DLg8YjYETiimno1Bn8ETiH7lV6T3Pc597MQQGrumSRpUi1l5H6+sOp7uSxnUrTc939N1fYZf+nfkLLZO88BeqWroQdpnJ9xg+GAYIWaBeyR1r9XQP4xZL/uNoHsIeEpfTHQupr8rwJdJW2Ttn8EPJnnHBsB76b1kwqoU9lJV1Z3kQWFKmPJnqMLcALwdJ4yLqi6Gqgl2yxgV0nNUnNNzwKqN4bUrCepuaQ21Pz5QtbcdELKvx3ZzJ2v1VJ+G+AT4KPUb3FoAXWyIjggWKEuBf4k6SmyX3C1ioipwGDgSUmTgT+kXXcC56ZOxa1z8n8OnEzWBDQFqASuz3Oa3wO/lfQM0DxP3nJ2NatOrXw6cLKkl8gC5xklOMczwExgCnAVMLGAY84ADkyf1wSge0QsBJ6R9LKkK1fLPxRonvL/EzgpIpZSg4iYTNZUNBW4OdXR6pCnrjAzM8BXCGZmljggmJkZ4IBgZmaJA4KZmQEOCGZmljggWFnTyplZX5Z0t6T1iygrd16lvypnVtZq8h6Q5u5Z03OsmBOqkPTV8ixZw3NdIumc/DnNMg4IVu6qptnYkWyKhZ/n7pT0le5PiIifpOkfanIAsMYBwawhc0CwxuQpYJv06/1xSbcDU9JdtFdKeiHNEPozAGX+LGmapAeBTasKUs58+5L6KJtpdbKy2UW7kgWeqgnivi2pg6R/pXO8IGmfdOwmkh5ON+LdQP4pOpB0n6QJaYbPgavtuzrVZYykDilt6zRT6ARJTynNBmu2pvw8BGsUJK1DNrXB6JTUE9gxImamL9WPIuKbyqbUfkbSw8BuZJPq7UQ2Edw0sjtic8vtANxENqvqTEntImKRpOuBJVXPFEjB55qIeFrZdNT/Ab4BXAw8HRG/lnQYsMoXfA1+nM7RCnhB0r/SHcAbABMj4mxJF6Wyf0k2TfbPI2K6pG+R3RF80Fd4G62Jc0CwctcqZ9K2p4C/kTXljIuImSn9O8DOWvl8gI2AbclmU70jIiqA9yQ9Vk35ewL/rSqrlllbDwZ2kFZcALSR1Dqd45h07IOSPijgNZ0uqWqW2c1TXReSTefxz5T+D+BeSRum13t3zrlbFHAOsy9xQLBy96Wnu6Uvxk9yk4DTVp/mWdJ3yT8ja6GztjYD9oqIz6qpS8Hzwyh7KM3BqaxPlT2IqKYZPiOd98M6fMKdNSHuQ7Cm4D/AIEnrQjbTpqQNyGbf7J/6GDoBB1Zz7LNkD2bplo6tadbWh8mab0j5dk2ruTN8Hgq0zVPXjYAPUjD4OtkVSpVmrHwc5Q/ImqI+BmZKOi6dQ5J2yXMOs2o5IFhT8Fey/oGJyh6mcwPZ1fEIYDrZDJ/XUc102xExn6zd/940a2tVk82/gaOrOpXJZiDtkTqtp7FytNOlwH7KHvLyHeDtPHUdTfYsgJfInvfwXM6+T4DukiaQ9RH8OqWfAJyS6jcV6FvAe2L2JZ7t1MzMAF8hmJlZ4oBgZmaAA4KZmSUOCGZmBjggmJlZ4oBgZmaAA4KZmSX/H1DBlRQbrg0KAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Pipeline for preprocessing and modeling\n",
    "rf2 = Pipeline(steps=[\n",
    "    ('preprocessor2', preprocessor2),\n",
    "    ('classifier', RandomForestClassifier(random_state=396))\n",
    "])\n",
    "#Putting our random forest with preprocessor2 through our evaluate function\n",
    "evaluate(rf2, X_train, X_test, y_train, y_test, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results**\n",
    "- The catboost encoding did not lead to an increase in model performance in any metric despite reducing overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We decided to explore other non-parametric ensemble methods, beyond random forest. Adaboost is included here because it contributes to our final model, despite its lower scoring in comparison to random forest. \n",
    "\n",
    "A gradient boost classifier was also examined but is not included here because it did not do as well as other models. Look further in personal notebooks if you're curious."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of Cross-Validation:\n",
      "\n",
      "Average accuracy: 0.7467681557274769        +/- 0.003431472196308603\n",
      "Average precision: 0.7844656567033667        +/- 0.0024194159171063044\n",
      "**Average recall: 0.6146060643943401        +/- 0.012452073374971798\n",
      "Average f1 score: 0.6891391631449573        +/- 0.007173486992449794\n",
      "Average roc_auc: 0.8244678458973114        +/- 0.0032480373855838635\n",
      "\n",
      "++++++++++++++++++++\n",
      "\n",
      "Results of Train-Test Split Validation:\n",
      "\n",
      "Training Scores:\n",
      "Train accuracy: 0.7478911904167707\n",
      "Train precision: 0.784623924507355\n",
      "**Train recall: 0.6177209658035617\n",
      "Train f1 score: 0.6912402958616052\n",
      "Train roc_auc: 0.737550062023266\n",
      "\n",
      "<><><><><><><><><><>\n",
      "\n",
      "Testing Scores:\n",
      "Test accuracy: 0.7475481021187392\n",
      "Test precision: 0.7864724245577523\n",
      "**Test recall: 0.6169795918367347\n",
      "Test f1 score: 0.691491308325709\n",
      "Test roc_auc: 0.737555061405093\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAneklEQVR4nO3deZgU1b3/8fcHkEVkUAQRxQUVNW6gItFoUETjEo1ojJoYt5jgevUaY67GG0XzQ01cYxQT4oaJXuOGCyrGEEUNKoIbghpRXBAEARVFZZn5/v6oM9DAzHRPuoeZnvm8nqeeqT516tSpbuhvn3OqTikiMDMza9XYFTAzs6bBAcHMzAAHBDMzSxwQzMwMcEAwM7OkTWNXwGrXpWuX2HDjjRq7GlYPU1+a3NhVsHqqIuZGRLdiythz34Exf978vPlee+nVxyJi/2KO1ZAcEJqwDTfeiAeffqSxq2H1sH3nXo1dBaunhZVL3iu2jPnz5vPgM4/mzbdZxw27FnushuSAYGZWtKAqqhq7EkVzQDAzK1IAS6sqG7saRXNAMDMrVriFYGZmZC2EKso/IPiyUzOzEqiKyLsUQtLaku6R9Iak1yXtJqmLpMclvZX+rpOT/zxJ0yS9KWm/nPSdJU1O266VpHzHdkAwMyta1mWUbynQ74ExEbE10Ad4HTgXGBsRvYGx6TWStgGOArYF9geGS2qdyrkBGAL0Tkvey10dEMzMihRQkoAgqQIYANwEEBGLI+JT4BBgZMo2Ehic1g8B7oyIRRExHZgG9JfUA6iIiGcjm9L6tpx9auUxBDOzIkUES6Ogq4y6SpqY83pERIzIeb0Z8DFwi6Q+wCTgTKB7RMxKx5olab2Uf0PguZz9Z6S0JWl95fQ6OSCYmZVAgV1CcyOiXx3b2wA7Af8VEc9L+j2pe6gWNY0LRB3pdXKXkZlZCZRoUHkGMCMink+v7yELELNTNxDp75yc/Lnz2/QEZqb0njWk18kBwcysSFGiQeWI+Aj4QNJWKWkQMBV4EDgupR0HPJDWHwSOktROUi+yweMJqXvpc0m7pquLjs3Zp1buMjIzK4ES3pj2X8DtktoC7wAnkP14v0vSicD7wA8AImKKpLvIgsZS4LSIZYMZpwC3Ah2AR9NSJwcEM7Mi1WNQuZCyXgZqGmcYVEv+YcCwGtInAtvV59gOCGZmRQqg0lNXmJkZUPCdyE2ZA4KZWdGiWcxl5IBgZlak7E5ltxDMzAyPIZiZGdVXGTkgmJm1eL7KyMzMlvEYgpmZZS2E/HPHNXkOCGZmRSv8iWhNmQOCmVmRqoDFJZq6ojE5IJiZFSs8hmBmZmTTX1c6IJiZGXhQ2czMqJ66orFrUTwHBDOzEnCXkZmZEcBi36lsZmYRHlQ2MzOq71Qufw4IZmYl4BaCmZml2U4dEMzMDKgs/3jggGBmVqxsLqPyjwgOCGZmxQp3GZmZGb7KyMzMcngMwczM0lVGjV2L4jkgmJkVKYAl5T9zhQOCmVmxPHWFmZkt4+mvzcwMEFWhxq5E0RwQzMyK5MtOzcxsmebQZdSqsStgZlbuImBpVf6lEJLelTRZ0suSJqa0LpIel/RW+rtOTv7zJE2T9Kak/XLSd07lTJN0raS8fVoOCGZmRap+pnK+pR4GRkTfiOiXXp8LjI2I3sDY9BpJ2wBHAdsC+wPDJbVO+9wADAF6p2X/fAd1QDAzK4Goyr8U4RBgZFofCQzOSb8zIhZFxHRgGtBfUg+gIiKejYgAbsvZp1YOCFYyCz5dwM+PPYWDdxnE9/rvw8sTXuTKX1/CwbsM4rBv7c+ZR5/Egk8XLMt/41XDOXDHvTi43978a+y4ZelLFi9m6JnncdDOAzl4l0E8/sCjjXA2zd/F1/+OJ9+eyH3PPbYsrWKdzoy4/y+MfukJRtz/FyrWrgBgt4F78LdxD3Hfs2P427iH6D9gt2X7tFljDS78/SU89OI/eXDiWPb5Xt4fos1SgS2ErpIm5ixDaigqgL9LmpSzvXtEzAJIf9dL6RsCH+TsOyOlbZjWV06vU9kGBEmVqY+tetm0hGUPTk2x6tcXS9qnVOWnMveSNLqUZTa23557EbvvsycPvTCWe595hM223ILdBu7BqGcf477xY9hki17cePVwAN5+4y0evfch7n/uMW64ZyT/7+wLqKzMrtMYccX1dOm2LqMnPcEDzz9Ovz2+2Zin1Ww9cPs9nHLYcSuknXjWKTw/bjwH7TiQ58eN58SzTgXgk3mfcPqRJ3LYbvtz/slnc8mIq5ftM+Sc05k/dx4H77Q3h+yyDxOfeX61nkdTEFHYAsyNiH45y4gaits9InYCDgBOkzSgjkPXNC4QdaTXqWwDAvBV6mOrXt4tYdmDgWUBISIuiIh/lLD8ZueLBZ8zafwEDjvmSADWaNuWirUr+NbeA2jTJruYrU+/HZk98yMAnnjkcQ74/sG0bdeOnptuxMabbcLkSa8AMOr2u/lp+iJq1aoV66zbpRHOqPmbNH4Cn33y2QppA7+7Lw/ccQ8AD9xxDwMP2heAN16dwscfzQFg2uv/pl37dqzRti0Ah/74B9x4ZRboI4JP53+yuk6hSSnVGEJEzEx/5wCjgP7A7NQNRPo7J2WfAWyUs3tPYGZK71lDep3KOSCsIo3Od03r/SQ9mdaHSrpZ0pOS3pF0Rs4+x0p6VdIrkv4i6VvA94DLU8tjc0m3Sjo85R8k6aU0en+zpHY5x75I0otp29Ypvb+k8Wmf8ZK2Ws1vy2ox490PWKdrF/731HP4wbe/y4X/9T98ufDLFfKM+utd7LHPngDMnvUR3TfssWxb9w16MGfWR8u6lK4bdhVHDDiInx93KnPnfLz6TqSFW7dbN+bOzt7vubM/Zt2uXVfJs+8hB/DGK1NYsngxnTpnXUqn/+/Z/O2p0Vw58nrW7bbqPi1BVVX+JR9JHSV1ql4HvgO8BjwIVDfnjgMeSOsPAkdJaiepF9ng8YTUrfS5pF3T1UXH5uxTq3IOCB1yuotGFZB/a2A/smh7oaQ1JG0LnA/sHRF9gDMjYjzZm3xOanm8XV2ApPbArcCREbE92X0cp+QcY25q6t0A/CKlvQEMiIgdgQuAS+qqpKQh1f2L8+fOK+C0mobKyqW8/soUjjzxaO5++mE6rLkmN119w7LtI664jtZt2nDQEYOB7JfkyoSorFzK7A9nseM3d+aup0bTZ5eduPJ/63zLbDXafOvenHXxuVz0378CoHXr1qzfcwNeem4iRw44iFcmvMjZw37VyLVc/Up4lVF34BlJrwATgIcjYgxwGbCvpLeAfdNrImIKcBcwFRgDnBYR1ffInQLcSDbQ/DaQdzCunG9M+yoi+tYj/8MRsQhYJGkO2Ru/N3BPRMwFiIj5ecrYCpgeEf9Or0cCpwHXpNf3pb+TgMPSemdgpKTeZP9u1qjrAKlPcQTA9jv1KZtbXbpv0IPuG6zPDv12BLJfkTdd80cAHrjjXsY99k9ufOB2qi+FXn+DHsz+cNay/WfPnEW3Ht1Zu8s6dFizA4MOzi6n3m/wgYz6612r+Wxarnkff0zX7lkroWv3bsybO3fZtu4brM81d/yJXw35OTOmvw/Ap/M/4cuFXzL2oWxg+rH7H+HQY49slLo3quVjBMUVE/EO0KeG9HnAoFr2GQYMqyF9IrBdfY5fzi2Emixl+Tm1X2nbopz1SrJgKAoYaMmR78aO6mNUlw/wG+CJiNgOOLiGejULXbt3Y/2ePZj+Vtagen7ceDbfague+cc4bv79H/nD//2ZDmt2WJZ/rwP24dF7H2LxokXMePcD3nv7XbbfuQ+S2HP/Qbzw9HMAPDduPJtttUWjnFNL9OQj/+CQHx0OwCE/OpwnHn4cgE6dK7j+7lv4/dDf8fLzk1bYZ9yYsezy7V0B2HXP3XnnjbdWb6WbiAIHlZu0cm4h1ORdYGeyptH3C8g/Fhgl6eqImCepS2olfA50qiH/G8CmkraIiGnAMcC4GvLl6gx8mNaPL6BOZeu8317EuT87iyWLF9Nz0435zfDL+eHAQ1i8eDFDBh8DwA677MgFVw9ji29syX6HfpdDvvkd2rRpzflXXEzr1tn9NGcN/R/OO+nn/Pa8i+nSdV1+c/3vGvO0mq3f3nwtu+yxK2uvuw7/eP1Zrr/kam66+gauuPV6Dj32CGZ9MJOzj8sG93845Fg22mwTTvrlGZz0y2wI7qTBxzB/7jyuvuAyLh1xFf9z2QXMnzufX596TmOeVqMphy/8fFRTX245kPRFRKy1Utq3gZuA2cDzQL+I2EvSUOCLiLgi5XsNOCgi3pV0HHAO2a/6lyLieEm7A38m+8V/OPBrYHRE3CNpEHAFWTB9ATglIhZJejcdb66kfsAV6di7kXUtfQz8EzgmIjaVtBfwi4g4qLZz3H6nPvHg04+U4N2y1WX7zr0auwpWTwsrl0zKuSP4P1Kx1RbxzeH5f7j8Y5/vF32shlS2AaElcEAoPw4I5ackAWHLLaJ/AS3Zsd9p2gGhuXUZmZk1iubw29oBwcysBBwQzMyMwAHBzMygZPchNDYHBDOzEihyeusmwQHBzKwE3EIwMzPAAcHMzCCNKjd2JYrngGBmVqTAYwhmZpa4y8jMzCCCKPSRaE2YA4KZWSmUfzxwQDAzKwV3GZmZWcYBwczMwC0EMzODrHVQmTdXk+eAYGZWpGy20/JvIjggmJmVQvnHAwcEM7OieeoKMzOr1gx6jBwQzMxKwgHBzMwIwFNXmJkZAJ7t1MzMAHcZmZlZxoPKZmbW/C87lfQH6jjFiDijQWpkZlaOmvmg8sTVVgszs3JX/vGg9oAQESNzX0vqGBELG75KZmZlqBkEhFb5MkjaTdJU4PX0uo+k4Q1eMzOzclF9H0K+pUCSWkt6SdLo9LqLpMclvZX+rpOT9zxJ0yS9KWm/nPSdJU1O266VpHzHzRsQgGuA/YB5ABHxCjCg4DMzM2sJooClcGeSfoQn5wJjI6I3MDa9RtI2wFHAtsD+wHBJrdM+NwBDgN5p2T/fQQsJCETEByslNYOZv83MSqhEAUFST+C7wI05yYcA1d34I4HBOel3RsSiiJgOTAP6S+oBVETEs5HNy31bzj61KuSy0w8kfQsISW2BM1gxcpmZtWxRcJdQV0m5F+yMiIgRK+W5Bvgl0CknrXtEzMoOFbMkrZfSNwSey8k3I6UtSesrp9epkIBwMvD7VNiHwGPAaQXsZ2bWchTWApgbEf1q2yjpIGBOREyStFcB5dU0LhB1pNcpb0CIiLnA0fnrZWbWgpXmKqPdge9JOhBoD1RI+iswW1KP1DroAcxJ+WcAG+Xs3xOYmdJ71pBep0KuMtpM0kOSPpY0R9IDkjYr6NTMzFqKEowhRMR5EdEzIjYlGyz+Z0T8GHgQOC5lOw54IK0/CBwlqZ2kXmSDxxNS99LnknZNVxcdm7NPrQoZVL4DuAvoAWwA3A38XwH7mZm1HBH5l//cZcC+kt4C9k2viYgpZN/PU4ExwGkRUX3RzylkA9PTgLeBR/MdpJAxBEXEX3Je/1XS6YWehZlZsxeUfPrriHgSeDKtzwMG1ZJvGDCshvSJwHb1OWZdcxl1SatPSDoXuJPstI8EHq7PQczMmr1m/jyESaw4Wn1SzrYAftNQlTIzKzvNYP7ruuYy6rU6K2JmVraa+/TXuSRtB2xDdhkUABFxW0NVysys7LSEgCDpQmAvsoDwCHAA8AzZrdBmZgbNIiAUctnp4WSj2x9FxAlAH6Bdg9bKzKycBFAZ+ZcmrpAuo68iokrSUkkVZHfI+cY0M7NcTf/7Pq9CAsJESWsDfya78ugLYEJDVsrMrNyoJQSEiDg1rf5R0hiyKVVfbdhqmZmVmeZ82amkneraFhEvNkyVzMzKUPnHgzpbCFfWsS2AvUtcF1vJu2/O4viBlzV2NawevjH0lsaugtXTxF//uPhCmvt9CBExcHVWxMysfNXvmclNVUE3ppmZWR7lHw8cEMzMitYAs502BgcEM7NSaAZXGRXyxDRJ+rGkC9LrjSX1b/iqmZmVkRI8Ma2xFTJ1xXBgN+CH6fXnwPUNViMzs3LUDAJCIV1G34yInSS9BBARn0hq28D1MjMrHwFqIVcZLZHUmhTfJHWjWQyfmJmVUPnHg4K6jK4FRgHrSRpGNvX1JQ1aKzOzctMSuowi4nZJk8imwBYwOCJeb/CamZmVk2bQb1LIA3I2Br4EHspNi4j3G7JiZmZlo0xaAPkUMobwMNmpiuwRmr2AN4FtG7BeZmblpSUMKkfE9rmv0yyoJzVYjczMylH5x4P636kcES9K2qUhKmNmVo5EC3lAjqSf57xsBewEfNxgNTIzKzfRcmY77ZSzvpRsTOHehqmOmVmZKv94UHdASDekrRUR56ym+piZlafmHBAktYmIpXU9StPMzGgRU1dMIBsveFnSg8DdwMLqjRFxXwPXzcysfJR/PChoDKELMI/sGcrV9yME4IBgZlatmbcQ1ktXGL3G8kBQrfzP3MyshJr7ZaetgbVYMRBUawanbmZWIi1g6opZEXHxaquJmVkZUwkeoSmpPfAU0I7s+/meiLhQUhfgb8CmwLvAERHxSdrnPOBEoBI4IyIeS+k7A7cCHYBHgDMj6q5kXdNf19QyMDOzmlRG/iW/RcDeEdEH6AvsL2lX4FxgbET0Bsam10jaBjiKbG65/YHh6XYBgBuAIUDvtOyf7+B1BYRBhdTezKzFi2wMId+St5jMF+nlGmkJ4BBgZEofCQxO64cAd0bEooiYDkwD+kvqAVRExLOpVXBbzj61qjUgRMT8/NU3MzOIbPqKfAt0lTQxZxmyckmSWkt6GZgDPB4RzwPdI2IWQPq7Xsq+IfBBzu4zUtqGaX3l9DrVe3I7MzOrQWFjCHMjol/dxUQl0FfS2sAoSdvVkb22i37+o4uBCnmEppmZ1aF6ttNiu4xyRcSnwJNkff+zUzcQ6e+clG0GsFHObj2BmSm9Zw3pdXJAMDMrVoAqq/Iu+UjqlloGSOoA7AO8ATwIHJeyHQc8kNYfBI6S1E5SL7LB4wmpW+lzSbtKEnBszj61cpeRmVkplOCyU6AHMDJdKdQKuCsiRkt6FrhL0onA+8APskPGFEl3AVPJZqM+LXU5AZzC8stOH01LnRwQzMxKoBR3KkfEq8CONaTPo5YrPyNiGDCshvSJQF3jD6twQDAzK9byq4jKmgOCmVkpNPPJ7czMrEClmLqisTkgmJkVKwKqKvPna+IcEMzMSsAtBDMzyzggmJkZAJH/xrOmzgHBzKxovuzUzMxIcxVVelDZzMwIdxmZmVniLiMzMyNwC8HMzMBdRmZmtpy7jMzMLGshLG3sShTNAcHMrEgBhLuMzMwsRYTGrkXRHBDMzIrmQWWzZbp068TPfnUwnbt0JKqCJ0e/zOP3TuSwnwxgx917ExEs+ORLbrxsNJ/O+2L5futVcMnIn3H/rU8z5m8TAGjdphXHnPkdtu67CRHBvTeOY+JTbzbWqTVLVUuX8N4tlxKVS4mqSiq+0Y9uAw9lxj3DWTz3oyzP11/Sqv2abHbyxXz26rPMG7/8kbyLZs+g10lDab/+xix47XnmPj2aiCrW6t2H7vse0Vin1bgcEGonKYCrIuLs9PoXwFoRMbQEZQ8FfgZ8nJLGRMS5xZabyl4b+FFEDE+vNwCujYjDS1F+znGeBH6Rnnta9iorq7hz+Fjee2s27Tu0ZeiIE5gycTqP3Pkc9938FAD7HNaPQ47bnZFXPbZsvx+dNojJz7+9QlkH/3h3Fnz6Jece8yck6FjRYbWeS0ug1m3Y5Lhf0qpte6JyKe/ecilr9d6BnoefuizP7MfupFX77L3vvMNudN5hNwC+nv0BM+78A+3X35ilX37B7MfvoteQC2nTsYKZ9/+Zhe9MpeNm2zTKeTWqZhAQWjVg2YuAwyR1baDyr46IvmkpSTBI1gaW/a+IiJmlDgbN0WfzF/LeW7MB+Pqrxcx8by7rdO3E118uXpanXfs1Vuhm3WmP3nw861M+fHfuCmV9+8AdGH37s0DWLfvFZ181/Am0MJJo1bY9AFFVSVSueIVMRLBg6gQ6b/fNVfZd8NrzVKT0JZ/Moe2669OmYwUAHXtty4LXm8VvnPqJyN7HPEtT15ABYSkwAjhr5Q2SNpE0VtKr6e/GKf1WSddKGi/pHUkFfxFLOl7SdTmvR0vaK61/IWmYpFckPSepe0rvLmlUSn9F0reAy4DNJb0s6XJJm0p6LeVvL+kWSZMlvSRpYM6x75M0RtJbkn6XU48bJE2UNEXSRfV/G8tP1/U7s0nv7rz9+kwAvn/iAK686zR223dbRqXWQtv2a3DgD3fj/pHPrLDvmmu1A+Cwnwxg6IgTOG3oYCrWWXP1nkALEVVVvPPHC/j35Wey1mbb0qHn5su2ffX+v2nTsTNt111/lf0WTJlAxfZZQGjbpTuL585i8adziapKPn/zRZYumL/azqFpqSpgadoaMiAAXA8cLanzSunXAbdFxA7A7cC1Odt6AHsAB5F9OdfmrPSl/bKk/fLUoyPwXET0AZ4i624iHXdcSt8JmAKcC7ydWh7nrFTOaQARsT3wQ2CkpPZpW1/gSGB74EhJG6X08yOiH7ADsKekHeqqqKQhKYBMXLK0/H4Zt+uwBqdfdCh3XPePZa2De296irOPuJ5nH5/CoEP7AXDoCd/msbsnsOirJSvs36p1K9Zdr4Jpr81g6JBbmDblQ446ZdBqP4+WQK1asdnJF9P751fx1czpfD1nxrJtn01e3grI9dWMt2m1Rlvar9cTgNYdOrL+d4/lw3tu4L1bLmWNzl2hVevVdg5NRxBRlXdp6hp0UDkiFki6DTgDyP122w04LK3/Bfhdzrb7I3vnplb/kq/F1RFxRfULScfXkXcxMDqtTwL2Tet7A8emulYCn0lap45y9gD+kPK/Iek9YMu0bWxEfJbqMhXYBPgAOELSELL3ugewDfBqbQeIiBFkLSs6dVyvrK5ja926FadfdBjP/mMKk57+9yrbnxs7hbMuO4L7b32azb6xAbvsuRVHnjyQNddqT1VVsGRxJWNHTWLRV4uZ9HQ2iPzCk28w4MA+q/tUWpTW7ddkzU22YuG0ybRfr2f2S/+NSfQacuEqeRe8NoGK7XZdIa3TVn3ptFVfAD6Z9CS0aujfmU1UGXzh57M6rjK6BngRuKWOPLlffIty1gUgaRjwXYCI6FtLGUtZscXTPmd9ScSy3utK/vPzVh3bcutdCbSR1Av4BbBLRHwi6daV6tWs/OSXBzLr/Xk8dvcLy9K6b7gOsz/8BIAdv9WbWe/PA+DSM/66LM/g4/fg668WM3bUJABefnYaW/fdhNdfeo9tdt6Ume+tOMZgxVu6cAFq3YbW7dekasliFk6fyrq7HwjAwnem0q5rD9ao6LLCPhFVLJj6ApuccO4qZbXpWEHlVwv55IV/suEPTqXl8WWnBYmI+ZLuAk4Ebk7J44GjyFoHRwPP1LJ7dRnnA+fnOdS7wKmSWgEbAv0LqN5Y4BTgGkmtybqWPgc61ZL/qVTff0raEtgYeJOsu6kmFcBCspZHd+AA4MkC6lV2em/fk933254P3p7DxTf+BIB7/jyOAQfuwPobr0tUBfNmf8atV43JW9Zdf3qCIb86mB+dvg+ff/olN/724Yaufouz9IvPmHn/jVBVBRF02nYXOm3ZF1hx0DjXl+/9mzYV69B2nfVWSJ895g6+/ugDALru+T3a1TDu0OwFZTFonM/qug/hSuD0nNdnADdLOofs0tETSnCMfwHTgcnAa2StknzOBEZIOpHsV/0pEfGspH+lgeRHycZBqg0H/ihpMlmL5PiIWCTV3HCIiFckvUQ2NvFOqmOz9NbkGRy/16WrpL+60iWlNbn/1hV/D8ybvYBLz7y9ZHWzVbXvvhGbnVTzNQ4bDP5pjekdN92aXj/99SrpG37/5JLWrTwFUQaDxvkomsHt1s1Vp47rxU7bHtnY1bB6+PJ7u+bPZE3KxF//eFK68OM/1mnNrtF3q4Pz5nvm5VuLPlZD8p3KZmZFyqYyKv8f1w4IZmZF86CymZlVc0AwMzMiyG5lKm8OCGZmJVAOdyLn00JvKTQzK61STF0haSNJT0h6Pc1/dmZK7yLp8TRX2uO5MypIOk/SNElv5k7jI2nnNO/atDRHXF031gIOCGZmJRDZ1Lz5lvyWAmdHxDeAXYHTJG1DNsfa2IjoTXZD7bkAadtRwLbA/sDwdJMtwA3AEKB3WvbPd3AHBDOzEihFCyEiZkXEi2n9c+B1spkXDgFGpmwjgcFp/RDgzohYFBHTgWlAf0k9gIqIeDZN23Nbzj618hiCmVnRggKnt+4qKfeBESPShJarkLQpsCPwPNA9ImZBFjQkVc8fsiHwXM5uM1LakrS+cnqdHBDMzIoUAVWFzWU0t5A7lSWtBdwL/HeaNbrWrDVVp470OrnLyMysaOnGtHxLASStQRYMbo+I+1Ly7NQNRPo7J6XPADbK2b0nMDOl96whvU4OCGZmJRBU5V3ySVcC3QS8HhFX5Wx6EDgurR8HPJCTfpSkdmm6/d7AhNS99LmkXVOZx+bsUyt3GZmZlUJp5jLaHTgGmCzp5ZT2K7KnR96VZmZ+H/hBdsiYkh4vMJXsCqXTYvkdcqcAtwIdyGZufjTfwR0QzMyKFiW5MS0inqH2B3HV+CzZiBgGDKshfSKwXX2O74BgZlakegwqN2kOCGZmRSv4stMmzQHBzKwEmsNcRg4IZmYl4AfkmJkZpRpUbmwOCGZmJeGAYGbW4kWErzIyM7NM5J8qqMlzQDAzKwGPIZiZGR5UNjOz5XzZqZmZAQXNZtrUOSCYmRXJVxmZmdkybiGYmRngqSvMzAzwVUZmZgZkk1+7hWBmZhBBVXhQ2czM8J3KZmaWuMvIzMwIwpedmplZxi0EMzMDPIZgZmaQrjJyQDAza/Gy+xAcEMzMDI8hmJkZ4KkrzMxsGbcQzMwMwmMIZmZGdmOa5zIyMzPAXUZmZpa4y8jMzKiezajcOSCYmRXJD8gxM7NMQGUzGFRWc4hqzZWkj4H3GrseDaQrMLexK2EFa86f1yYR0a2YAiSNIXuP8pkbEfsXc6yG5IBgjULSxIjo19j1sML482oZWjV2BczMrGlwQDAzM8ABwRrPiMaugNWLP68WwGMIZmYGuIVgZmaJA4KZmQEOCFYDSZWSXs5ZNi1h2YMlbZPz+mJJ+5Sq/FTmXpJGl7LM1U1SSLoy5/UvJA0tUdlDJX2Y8/leVopyU9lrSzo15/UGku4pVfk55T4pyZfBlpjvVLaafBURfRuo7MHAaGAqQERc0EDHKXeLgMMkXRoRDXFD2NURcUUDlLs2cCowHCAiZgKHN8BxrAG4hWAFkfSupK5pvZ+kJ9P6UEk3p19s70g6I2efYyW9KukVSX+R9C3ge8Dl6Zfp5pJulXR4yj9I0kuSJqcy2+Uc+yJJL6ZtW6f0/pLGp33GS9pqNb8tDWkp2ZU9Z628QdImksam93aspI1T+q2Srk3vxTvV72shJB0v6bqc16Ml7ZXWv5A0LH2Oz0nqntK7SxqV0l9Jn+9lwObp871c0qaSXkv520u6JX2GL0kamHPs+ySNkfSWpN/l1OMGSRMlTZF0Uf3fRqsPBwSrSYec7oRRBeTfGtgP6A9cKGkNSdsC5wN7R0Qf4MyIGA88CJwTEX0j4u3qAiS1B24FjoyI7clar6fkHGNuROwE3AD8IqW9AQyIiB2BC4BLijjnpuh64GhJnVdKvw64LSJ2AG4Hrs3Z1gPYAziI7Mu5NmflfMb75alHR+C59Dk+BfwspV8LjEvpOwFTgHOBt9Pne85K5ZwGkD7fHwIj0+cO0Bc4EtgeOFLSRin9/HSH9A7AnpJ2yFNXK4IDgtXkq/Qfum9EHFpA/ocjYlHq2pgDdAf2Bu6p7u6IiPl5ytgKmB4R/06vRwIDcrbfl/5OAjZN652Bu9Mv0KuBbQuoa9mIiAXAbcAZK23aDbgjrf+FLABUuz8iqiJiKtnnUJurcz7jx/JUZTFZNx+s+P7vTRagiYjKiPgsTzl7pPoSEW+QzdO1Zdo2NiI+i4ivyboTN0npR0h6EXiJ7PPdBmswDghWqKUs//fSfqVti3LWK8l+3QvqNUG88myvPkZ1+QC/AZ6IiO2Ag2uoV3NwDXAi2a/02uS+z7mfhQBSd8/Lkl6uo4zczxdWfC+XxPIblnLf//qq6zNe5d+QpF5krcFBqTX0MM3zM24yHBCsUO8CO6f17xeQfyzZr7t1ASR1SemfA51qyP8GsKmkLdLrY4BxeY7RGfgwrR9fQJ3KTmpZ3UUWFKqNB45K60cDz+Qp4/zq1kAd2d4F+kpqlbpr+hdQvbGkbj1JrSVVUPvnC1l309Ep/5bAxsCbdZRfASwEPkvjFgcUUCcrggOCFeoi4PeSnib7BVeniJgCDAPGSXoFuCptuhM4Jw0qbp6T/2vgBLIuoMlAFfDHPIf5HXCppH8Bret7QmXkSlacWvkM4ARJr5IFzjNLcIx/AdOBycAVwIsF7HMmMDB9XpOAbSNiHvAvSa9Junyl/MOB1in/34DjI2IRtYiIV8i6iqYAN6c6WgPy1BVmZga4hWBmZokDgpmZAQ4IZmaWOCCYmRnggGBmZokDgpU1LZ+Z9TVJd0tas4iycudVulE5s7LWkHevNHdPfY+xbE6oQtJXyvNFPY81VNIv8uc0yzggWLmrnmZjO7IpFk7O3SjpP7o/ISJ+mqZ/qM1eQL0DgllT5oBgzcnTwBbp1/sTku4AJqe7aC+X9EKaIfQkAGWukzRV0sPAetUFKWe+fUn7K5tp9RVls4tuShZ4qieI+7akbpLuTcd4QdLuad91Jf093Yj3J/JP0YGk+yVNSjN8Dllp25WpLmMldUtpm6eZQidJelppNliz+vLzEKxZkNSGbGqDMSmpP7BdRExPX6qfRcQuyqbU/pekvwM7kk2qtz3ZRHBTye6IzS23G/BnsllVp0vqEhHzJf0R+KL6mQIp+FwdEc8om476MeAbwIXAMxFxsaTvAit8wdfiJ+kYHYAXJN2b7gDuCLwYEWdLuiCVfTrZNNknR8Rbkr5Jdkfw3v/B22gtnAOClbsOOZO2PQ3cRNaVMyEipqf07wA7aPnzAToDvclmU/2/iKgEZkr6Zw3l7wo8VV1WHbO27gNsIy1rAFRI6pSOcVja92FJnxRwTmdIqp5ldqNU13lk03n8LaX/FbhP0lrpfO/OOXa7Ao5htgoHBCt3qzzdLX0xLsxNAv5r5WmeJR1I/hlZC521tRWwW0R8VUNdCp4fRtlDafZJZX2p7EFEtc3wGem4nzbgE+6sBfEYgrUEjwGnSFoDspk2JXUkm33zqDTG0AMYWMO+z5I9mKVX2re2WVv/TtZ9Q8rXN63mzvB5ALBOnrp2Bj5JwWBrshZKtVYsfxzlj8i6ohYA0yX9IB1DkvrkOYZZjRwQrCW4kWx84EVlD9P5E1nreBTwFtkMnzdQw3TbEfExWb//fWnW1uoum4eAQ6sHlclmIO2XBq2nsvxqp4uAAcoe8vId4P08dR1D9iyAV8me9/BczraFwLaSJpGNEVyc0o8GTkz1mwIcUsB7YrYKz3ZqZmaAWwhmZpY4IJiZGeCAYGZmiQOCmZkBDghmZpY4IJiZGeCAYGZmyf8H23YBqdXXejUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Pipeline for preprocessing and modeling\n",
    "adbst = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor1),\n",
    "    ('classifier', AdaBoostClassifier(random_state=396))\n",
    "])\n",
    "\n",
    "#Putting adaboost through our evaluate function\n",
    "evaluate(adbst, X_train, X_test, y_train, y_test, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results**\n",
    "- Due to the significant drop in recall score, adaboost on it's own was discarded as a standalone potential final model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Considerations\n",
    "- We performed a gridsearch on a few of our models in notebooks found in personal folders. The results were subpar and not fit for inclusion in our final model.\n",
    "- Other Model types were examined, please reference personal notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Voting Classifier - Final Model\n",
    "\n",
    "We felt that the logic behind AdaBoost was sound, despite its low scores, so we wanted to see if it could help improve our top performer, RandomForest. We used a VotingClassifier and weighed it more heavily in favor of RandomForest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "w_avg = VotingClassifier(estimators=[\n",
    "    ('ada', adbst),\n",
    "    ('rndfor', rf1)],\n",
    "    weights=[0.25, 0.75], voting = 'soft')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of Cross-Validation:\n",
      "\n",
      "Average accuracy: 0.816246568505116        +/- 0.0025942578477496534\n",
      "Average precision: 0.8250221415085994        +/- 0.005128887445870006\n",
      "**Average recall: 0.7587675814760242        +/- 0.006842820600317659\n",
      "Average f1 score: 0.7904750774468964        +/- 0.0033039917197465215\n",
      "Average roc_auc: 0.8947367035469279        +/- 0.0024034085678559212\n",
      "\n",
      "++++++++++++++++++++\n",
      "\n",
      "Results of Train-Test Split Validation:\n",
      "\n",
      "Training Scores:\n",
      "Train accuracy: 1.0\n",
      "Train precision: 1.0\n",
      "**Train recall: 1.0\n",
      "Train f1 score: 1.0\n",
      "Train roc_auc: 1.0\n",
      "\n",
      "<><><><><><><><><><>\n",
      "\n",
      "Testing Scores:\n",
      "Test accuracy: 0.8212173392228794\n",
      "Test precision: 0.8322959274408679\n",
      "**Test recall: 0.7640816326530612\n",
      "Test f1 score: 0.7967313585291114\n",
      "Test roc_auc: 0.8168444667690085\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoLElEQVR4nO3deZgU1fn28e/NIuACiiAiqKDigqCoiPuGGyYuxLhgjGJiJOJuEvPTGE1MXqLRuMaocYtgYgwuKFFBDYlG3EFEBCWguCAKAiqICMzM8/5RZ7CFmenG6WHomftzXXV11amqU6enoZ+uc6qeUkRgZmbWpL4bYGZmawYHBDMzAxwQzMwscUAwMzPAAcHMzJJm9d0Aq17bdm2j02ab1nczbBVMmTCpvptgq6iCmBsR7WtTx34HHxDz583Pu93rE157PCL61eZYdckBYQ3WabNNGfnMY/XdDFsFPdt0re8m2CpaVL7s3drWMX/efEaOHZV3uy3W6dSutseqSw4IZma1FlRERX03otYcEMzMaimAsory+m5GrTkgmJnVVvgMwczMyM4QKnBAMDMzoKIB5IVzQDAzqzV3GZmZGanLqAEEBN+pbGZWSxFBWZTnnQohaX1J90t6U9IbkvaQ1FbSk5KmpdcNcra/SNJ0SVMlHZpTvoukSWndDZKU79gOCGZmRVARFXmnAl0PjI6IbYEdgTeAC4ExEdENGJOWkdQdGABsD/QDbpLUNNVzMzAI6JamvHdIOyCYmRVBRUTeKR9JrYF9gTsAImJpRHwKHAUMTZsNBfqn+aOAeyNiSUTMAKYDfSR1BFpHxPORPQVtWM4+1XJAMDOrpUiDygWcIbSTNC5nGrRCVVsAHwN/kTRB0u2S1gE6RMSHAOl1o7R9J+D9nP1nprJOaX7F8hp5UNnMrAgK7BKaGxG9a1jfDNgZODsiXpR0Pal7qBpVjQtEDeU18hmCmVktFXFQeSYwMyJeTMv3kwWI2akbiPQ6J2f73JTInYFZqbxzFeU1ckAwM6ulAMqjIu+Ut56Ij4D3JW2Tig4EpgAjgYGpbCDwcJofCQyQ1EJSV7LB45dSt9JCSbunq4tOztmnWu4yMjMrgiLeqXw28DdJawFvAz8g+/E+XNKpwHvAsQARMVnScLKgUQacGbH8VGQwcBfQChiVpho5IJiZ1VoULZdRRLwKVDXOcGA12w8BhlRRPg7osSrHdkAwM6ul7E5l5zIyMzMoaIxgTeeAYGZWS9lVRg4IZmaNXuVVRqXOAcHMrAg8hmBmZtkZQv4bgdd4DghmZrVWWPK6NZ0DgplZLVUASwt83sGazAHBzKy2wmMIZmZGlv663AHBzMzAg8pmZkZl6or6bkXtOSCYmRWBu4zMzIwAlvpOZTMzi/CgspmZUXmnculzQDAzKwKfIZiZWcp26oBgZmZAeenHAwcEM7PaynIZlX5EcEAwM6utcJeRmZnhq4zMzCyHxxDMzCxdZVTfrag9BwQzs1oKYFnpZ65wQDAzqy2nrjAzs+Wc/trMzABREarvRtSaA4KZWS35slMzM1vOXUZmZkYElPkqIzMzayjPVG5S3w0wM2sIoiL/VAhJ70iaJOlVSeNSWVtJT0qall43yNn+IknTJU2VdGhO+S6pnumSbpCUd9TbAcGKZsGnC/jJyYM5YtcDObLPQbz60itcfcnvOGLXAzl6z36ce+KPWfDpAgCWLVvGxaf/lO/s2Y8j+xzE7dfctLye0Q8+wtF79qP/7odwzaWX19fbaVROHPwDHnzhcUa8+ATfP+OHX1s38OzTmLTgHdZvm30H9dhlR+4b+xj3jX2M+58dRd/DD62qykanIvJPq+CAiOgVEb3T8oXAmIjoBoxJy0jqDgwAtgf6ATdJapr2uRkYBHRLU798By3ZgCCpPEXQyqlLEevun/7Qlcu/kXRQsepPde4v6ZFi1lnffn/hZex10H788+UxPDD2MbbYeiv2OGBvRjz/OA8+N5rNt+rK7ddmX/xPPPQYS5cuZcRzo/nHU//kvr/cwwfvzuTT+Z9w9aWXc/vIv/HQC08wb85cXnj62Xp+Zw3bVtttzXcHDuB7BxzFMXsexn6H9mWzLbsA0KFTR/bouw+z3pu5fPvpU6YyYL8jOHbvb3H60Sdz6fVDaNq0aTW1Nw4RhU21cBQwNM0PBfrnlN8bEUsiYgYwHegjqSPQOiKej4gAhuXsU62SDQjA4hRBK6d3ilh3f2B5QIiISyPiX0Wsv8H5fMFCxj/3EkefdDwAzddai9brt2bPvvvSrFk2VLVj752YPesjACSxeNEXlJWVseTLL2m+VnPWbb0uM995j8237ErbdhsCsPv+e/GvkaPr5001EltssxWvvTyBLxd/SXl5OeOefZED06/+n19+CddccvnXvswqtwNo0bJFrb/pGooCzxDaSRqXMw2qoqoAnpA0Pmd9h4j4ECC9bpTKOwHv5+w7M5V1SvMrlteolAPCSlLfW7s031vSU2n+15LulPSUpLclnZOzz8mSXpM0UdLdkvYEjgSuSmceW0q6S9IxafsDJU1IfXN3SmqRc+zLJL2S1m2byvtIei7t85ykbVbzn2W1mPnO+2zQri2/POMCjt3n2/zq7P/ji0VffG2bEX8dzt4H7QfAwUcdRqt11qbvNrtxSI+9GHj2abTZYH023aILM6a9xQfvzqSsrIx/P/okH82cVR9vqdGYNmUqu+zVhzZt16dlq5bsc8gBbNy5I/sfdhBzPpzN/15/Y6V9evbuxYgXn+DB5x/nN+f9cnmAaMwqKvJPwNyI6J0z3VpFVXtFxM7AYcCZkvat4bBVjQtEDeU1KuWA0Cqnu2hEAdtvCxwK9AF+Jam5pO2Bi4G+EbEjcG5EPAeMBC5IZx5vVVYgqSVwF3B8RPQku0prcM4x5qYP8mbgZ6nsTWDfiNgJuBT4XU2NlDSo8tfD/LnzCnhba4by8jLemDiZ4089kfueeZRWa6/NHdfevHz9rX+4kabNmnH4cf0BeH38RJo0bcqYN19g1MT/MuzG23n/nfdos34bLrn6t1zww7MYeNhxbLJZJ5o288VwdWnG/97izmtv4daH/sotDw5l6qQ3KC8r57QLzuJPQ66pcp9J417lO7sdwoD9j+RHPx3MWi1arOZWr1kqrzIqxhhCRMxKr3OAEWTfWbNTNxDpdU7afCawac7unYFZqbxzFeU1KuWAkNtl9J0Ctn809bPNJftjdgD6AvenMiJifp46tgFmRMT/0vJQIDd6P5hexwNd0nwb4D5JrwPXkg3+VCsibq389VDZbVIKOmzSkQ6bbMwOvXcCsjOAN16bDMDD9zzA04//mytuu47KCx0evf9h9j5wX5o3b86G7dvRa7feTJ7wGgD7H3YQ94x5iL89+SBdum3B5qk/2+rOiLuHc/y+h3PKYcfz2Sef8sF7M+m0eWfuf3YUoyeNpUOnjRn+zCNsuFH7r+03439vsXjRYrbqvnU9tXwNUaQxBEnrSFqvch44BHid7EfqwLTZQODhND8SGCCphaSuZIPHL6VupYWSdk9XF52cs0+1SjkgVKWMr95TyxXWLcmZLyf7dS8KOI3Kke+yrcpjVNYP8FvgPxHRAziiinY1CO06tGfjzh2ZMS07oXrx6efYcputGPuvp7nz+lv4499vo9XarZZv37FzJ1787/NEBF8s+oLXxk2ga7ctAZj38VwAPvv0M/5x+185+uTjV/8bamQqf3xs3HkTDjqyH//8+wPsv2Vv+vXcm34992b2Bx9x3D6HM2/Ox3TavPPyQeSOm3aiS7ctmPXuzJqqbxSKNKjcARgraSLwEtkP2dHAFcDBkqYBB6dlImIyMByYAowGzoyIyv67wcDtZAPNbwGj8h28oZ2LvwPsQvbGv1vA9mOAEZKujYh5ktqms4SFwHpVbP8m0EXSVhExHTgJeDrPMdoAH6T5UwpoU8m66PeXceFp57Ns6VI6d9mM3950FScccBRLly5lUP+TANhh15249NohnPCjk/jlmRfwnT0OJSLof+IxbNNjOwB+f+FvmJr6rU//+Tl02WqLentPjcU1f72Z9dtuQNmyMob89JLllwdXZac9duXU8wdTtqyMiooKhvzkEj6d/8lqbO2aqRhj6xHxNrBjFeXzgAOr2WcIMKSK8nFAj1U5fkMLCJcBd0j6BfBivo0jYrKkIcDTksqBCWRf2vcCt6XB52Nytv9S0g/IuoCaAS8Dt+Q5zJXAUEk/Af79Dd5Tydh2h+7846mRXyt7bMJTVW679rrrcM3Qm6pcd+UdNxS7aZbHKf2Oq3F9v557L59/5N4RPHJvIcN2jUewfNC4pCl8ydgaq+fOO8bIZx6r72bYKujZpmt9N8FW0aLyZeNzbgD7RlpvvVX0+dOVebcbc8h3a32sutTQzhDMzOpFQ/ht7YBgZlYEDghmZkbggGBmZrD8PoRS54BgZlYEhaa3XpM5IJiZFYHPEMzMDHBAMDMzSKPK9d2I2nNAMDOrpcBjCGZmlrjLyMzMIIJYxYcmr4kcEMzMiqH044EDgplZMbjLyMzMMg4IZmYGPkMwMzPIzg7K8261xnNAMDOrpSzbaemfIjggmJkVQ+nHAwcEM7Nac+oKMzOr1AB6jBwQzMyKwgHBzMwIwKkrzMwMAGc7NTMzwF1GZmaW8aCymZk1/MtOJf2RGt5iRJxTJy0yMytFDXxQedxqa4WZWakr/XhQfUCIiKG5y5LWiYhFdd8kM7MS1AACQpN8G0jaQ9IU4I20vKOkm+q8ZWZmpaLyPoR8U4EkNZU0QdIjabmtpCclTUuvG+Rse5Gk6ZKmSjo0p3wXSZPSuhskKd9x8wYE4DrgUGAeQERMBPYt+J2ZmTUGUcBUuHNJP8KTC4ExEdENGJOWkdQdGABsD/QDbpLUNO1zMzAI6JamfvkOWkhAICLeX6GoAWT+NjMroiIFBEmdgW8Dt+cUHwVUduMPBfrnlN8bEUsiYgYwHegjqSPQOiKejywv97CcfapVyGWn70vaEwhJawHn8PXIZWbWuEXBXULtJOVesHNrRNy6wjbXAT8H1ssp6xARH2aHig8lbZTKOwEv5Gw3M5UtS/MrlteokIBwOnB9quwD4HHgzAL2MzNrPAo7A5gbEb2rWynpcGBORIyXtH8B9VU1LhA1lNcob0CIiLnAifnbZWbWiBXnKqO9gCMlfQtoCbSW9FdgtqSO6eygIzAnbT8T2DRn/87ArFTeuYryGhVyldEWkv4p6WNJcyQ9LGmLgt6amVljUYQxhIi4KCI6R0QXssHif0fE94GRwMC02UDg4TQ/EhggqYWkrmSDxy+l7qWFknZPVxednLNPtQoZVL4HGA50BDYB7gP+XsB+ZmaNR0T+6Zu7AjhY0jTg4LRMREwm+36eAowGzoyIyot+BpMNTE8H3gJG5TtIIWMIioi7c5b/KumsQt+FmVmDFxQ9/XVEPAU8lebnAQdWs90QYEgV5eOAHqtyzJpyGbVNs/+RdCFwL9nbPh54dFUOYmbW4DXw5yGM5+uj1T/OWRfAb+uqUWZmJacB5L+uKZdR19XZEDOzktXQ01/nktQD6E52GRQAETGsrhplZlZyGkNAkPQrYH+ygPAYcBgwluxWaDMzgwYREAq57PQYstHtjyLiB8COQIs6bZWZWSkJoDzyT2u4QrqMFkdEhaQySa3J7pDzjWlmZrnW/O/7vAoJCOMkrQ/cRnbl0efAS3XZKDOzUqPGEBAi4ow0e4uk0WQpVV+r22aZmZWYhnzZqaSda1oXEa/UTZPMzEpQ6ceDGs8Qrq5hXQB9i9wWW8G0SVM5uGuVd6vbGmrnEU7zVWqeOfKY2lfS0O9DiIgDVmdDzMxK16o9M3lNVdCNaWZmlkfpxwMHBDOzWquDbKf1wQHBzKwYGsBVRoU8MU2Svi/p0rS8maQ+dd80M7MSUoQnptW3QlJX3ATsAZyQlhcCf6qzFpmZlaIGEBAK6TLaLSJ2ljQBICI+kbRWHbfLzKx0BKiRXGW0TFJTUnyT1J4GMXxiZlZEpR8PCuoyugEYAWwkaQhZ6uvf1WmrzMxKTWPoMoqIv0kaT5YCW0D/iHijzltmZlZKGkC/SSEPyNkM+AL4Z25ZRLxXlw0zMysZJXIGkE8hYwiPkr1VkT1CsyswFdi+DttlZlZaGsOgckT0zF1OWVB/XGctMjMrRaUfD1b9TuWIeEXSrnXRGDOzUiQayQNyJP0kZ7EJsDPwcZ21yMys1ETjyXa6Xs58GdmYwgN10xwzsxJV+vGg5oCQbkhbNyIuWE3tMTMrTQ05IEhqFhFlNT1K08zMaBSpK14iGy94VdJI4D5gUeXKiHiwjttmZlY6Sj8eFDSG0BaYR/YM5cr7EQJwQDAzq9TAzxA2SlcYvc5XgaBS6b9zM7MiagiXndaU3K4psG6a1suZr5zMzAwKS2xXQMCQ1FLSS5ImSpos6bJU3lbSk5KmpdcNcva5SNJ0SVMlHZpTvoukSWndDZJU1TFz1XSG8GFE/Cb/WzAzMxXnEZpLgL4R8bmk5sBYSaOAo4ExEXGFpAuBC4H/k9QdGECWSmgT4F+Sto6IcuBmYBDwAvAY0A8YVdPBazpDyBtNzMwsKY/8Ux6R+TwtNk9TAEcBQ1P5UKB/mj8KuDcilkTEDGA60EdSR6B1RDwfEQEMy9mnWjUFhAPztt7MzLLLTguYgHaSxuVMg1asSlJTSa8Cc4AnI+JFoENEfAiQXjdKm3cC3s/ZfWYq65TmVyyvUbVdRhExP9/OZmYGEFn6ivzmRkTvGmvKunt6SVofGCGpRw2bV9WTs+JFQLnlNSrkiWlmZpZPRP5plaqLT4GnyPr+Z6duINLrnLTZTGDTnN06A7NSeecqymvkgGBmVkuV2U4L6DKquR6pfTozQFIr4CDgTWAkMDBtNhB4OM2PBAZIaiGpK9ANeCl1Ky2UtHu6uujknH2qtcrpr83MbAUBKi/KMzQ7AkNTHrkmwPCIeETS88BwSacC7wHHAkTEZEnDgSlkyUfPTF1OAIOBu4BWZFcX1XiFETggmJkVRxEuO42I14CdqiifRzUX+kTEEGBIFeXjgJrGH1bigGBmVgQN4U5lBwQzs9r6BoPGayIHBDOzYmjgye3MzKxARUpdUa8cEMzMaisCKsrzb7eGc0AwMysCnyGYmVnGAcHMzACIotyYVq8cEMzMas2XnZqZGSlXUbkHlc3MjHCXkZmZJe4yMjMzAp8hmJkZuMvIzMy+4i4jMzPLzhDK6rsRteaAYGZWSwGEu4zMzCxFhPpuRa05IJiZ1ZoHlc2W++mVF7Bb3935dN6nDDr01K+tO+a04/jxxafz3Z36s+CTBcvL22+yEXc8+ReGXTeU+28bDsAf7r2Gtu03ZOmSJQBceNLP+XTep6vtfTQ2UV7Bmz//Bc3btmWri38OwJxHR/PxqCdQ0ya03mUnOp98IlFWxrs33coXb79DlJez4f77sPF3+wPwxVtv884fbyGWLqX1zr3ofOpAJNXju6onDSAgNKmriiWFpKtzln8m6ddFqvvXkj6Q9GqarihGvanu9SWdkbO8iaT7i1V/Tr1PSepd7HrryxP3P84vBl64Unn7ju3ZZZ9dmD1z9krrBl9yBi8/9dJK5VecN4TTvzWI0781yMGgjs15dBQtO3davrxw0mQ+e3k82137e7pf/wc6HHk4AJ889yKxrIzu113Jdn/4HXOfGMOSOR8D8N6f72SzwT+i+5+uZcmHH7FgwsR6eS/1LiryT2u4OgsIwBLgaEnt6qj+ayOiV5pW/ib65tYHlgeEiJgVEccUsf4GadJLr7HwswUrlZ9+yRncdvmfCb7ev7rnIXvx4Xsf8s60d1ZTC21FS+fOY8H4CbQ76IDlZR8//iQdvnMkTZo3B6D5+m2yFYLyJUuI8nIqli5FzZrRtFUrls3/hPLFi1l3m62RRNv99+GzF8fVx9upXxFERXneaU1XlwGhDLgVOH/FFZI2lzRG0mvpdbNUfpekGyQ9J+ltSQV/EUs6RdKNOcuPSNo/zX8uaYikiZJekNQhlXeQNCKVT5S0J3AFsGU687hKUhdJr6ftW0r6i6RJkiZIOiDn2A9KGi1pmqQrc9pxs6RxkiZLumzV/4yla4+D9mTe7Lm8/cbbXytv2aolx58+gLuvH1rlfj+76ufc8titnHj291dHMxutmXcOo9PJ3wN99TWwZNZHfP7Gm7z5f7/kf7+8jEXT3gJggz12o2mLFkw6dTCvDzqbDkcdTrP11mXp/PmstWHb5fuvteGGLJ0/f7W/lzVDRQHTmq0uAwLAn4ATJbVZofxGYFhE7AD8DbghZ11HYG/gcLIv5+qcn9NldGiedqwDvBAROwL/BU5L5TcAT6fynYHJwIXAW+nM44IV6jkTICJ6AicAQyW1TOt6AccDPYHjJW2ayi+OiN7ADsB+knaoqaGSBqUAMq68BH5RVKdFyxaccNaJ3HXNXSutO/n8U3jgjvv58osvV1p3+bm/Y1C/H3H+sefSc9cdOOjog1dDaxufz8a9QrM2rVl7yy2+Vh7l5ZR/vohtrvgtnQaeyIyrrycissDQpAk9b7+J7W++ntkjH2XJR7OhqgtrGuP4AUFERd5pTVeng8oRsUDSMOAcYHHOqj2Ao9P83cCVOeseiuwvN6Xyl3w1ro2IP1QuSDqlhm2XAo+k+fFA5bdMX+Dk1NZy4DNJG9RQz97AH9P2b0p6F9g6rRsTEZ+ltkwBNgfeB46TNIjsb90R6A68Vt0BIuJWsjMrWjZvWbLXsXXcfBM27rwxfx51GwDtN27PzY/8mbP6n8G2vbZln2/ty2kX/Zh1W69LRUUFy5Ys5eFhDzFv9lwAFi9azL9HjmHbHbfjXw8+WZ9vpUH6/M2pfPbyK7z+yqtULFtG+ReLmXHdjay1YVvW370Pklin21YgUbZgIfOfeZbWO+2ImjWj+fptWHfbrfnirbdZd7ttWTrvqzOCpfPmsdYGNf0XasBK4As/n9VxldF1wCvAX2rYJveLb0nOvAAkDQG+DRARvaqpo4yvn/G0zJlfFrH8IuFyvvn7rumnT267y4FmkroCPwN2jYhPJN21QrsarHemzuC43t9dvnz32Hs484jTWfDJAn5y3HnLy086byCLFy3m4WEP0aRpE9ZtvS4LPllA02ZN2a3v7kx49pV6aH3D1+n7J9Dp+ycAsPD1Kcx++BG6nncWHz/+JAsnTWa9Ht35ctaHRFkZzVqvx1rt2rFw0mTa7rc3FUuWsOh/02l/+GE0b7sBTVu2ZNHUaay99VbMf+oZ2n8r3wl7Q+TLTgsSEfMlDQdOBe5Mxc8BA8jODk4Exuap42Lg4jyHegc4Q1IToBPQp4DmjQEGA9dJakrWtbQQWK+a7f+b2vtvSVsDmwFTybqbqtIaWER25tEBOAx4qoB2lZxf3PBLdth9R9ps0IZ7nv8Hw669i9HDR61SHWuttRaXD7uSZs2a0qRpUyY8O57H/v5oHbXYqrJh3wN490+3MOXcC1CzZnQ5ZzCSaH/YIbx74y28cd4FELBh3/1Yu8vmAGz64x/y7h9voWLpUtrs3IvWO/eq3zdRH4KSGDTOZ3Xdh3A1cFbO8jnAnZIuAD4GflCEYzwLzAAmAa+TnZXkcy5wq6RTyX7VD46I5yU9mwaSR5GNg1S6CbhF0iSyM5JTImJJdddcR8RESRPIxibeTm1skH53zv+rcf1Je3+vyvK7r/tqYPnLxV9y5hGnF7Vdlt96PbqzXo/uADRp3oyu55210jZNW7VkiwvOq3L/dbbaku7XX1WXTSwBQZTAoHE+igZwu3VD1bJ5y+i8QZf6boatgk3uGFLfTbBV9MyRx4xPF358Y+ut3S56bXNE3u3GvnpXrY9Vl3ynsplZLWWpjEr/x7UDgplZrXlQ2czMKjWAgFDXN6aZmTV8EUSU553ykbSppP9IeiNlNzg3lbeV9GTKhPBk7v1Ski6SNF3S1NybdCXtkrIqTE8ZIPLeMeiAYGZWBEW6U7kM+GlEbAfsDpwpqTtZBoUxEdGN7HL5CwHSugHA9kA/4KZ0CT3AzcAgoFua+uU7uAOCmVkRFCMgRMSHEfFKml8IvEF2X9VRQOU12kOB/mn+KODeiFgSETOA6UAfSR2B1hHxfLopd1jOPtXyGIKZWa1FoU9MaycpNx3srSldzUokdQF2Al4EOkTEh5AFDUkbpc06AS/k7DYzlS1L8yuW18gBwcysCArsEppbyH0IktYFHgDOSznhqt20qqbUUF4jdxmZmdVaUKz015KakwWDv0XEg6l4duoGIr3OSeUzgU1zdu8MzErlnasor5EDgplZLUVARUV53imfdCXQHcAbEXFNzqqRwMA0PxB4OKd8gKQWKZlmN+Cl1L20UNLuqc6Tc/aplruMzMxqrWg3pu0FnARMkvRqKvsF2bNhhqe8a+8BxwJExOSUPHQK2RVKZ8ZX17cOBu4CWpHlZcubbdIBwcysCIqR3C4ixlJ9mv0Dq9lnCLBSEq2IGAf0WJXjOyCYmRWDcxmZmVnlIzRLnQOCmVktVQ4qlzoHBDOzWqu87LS0OSCYmRWBu4zMzAzwA3LMzAzwoLKZmeVwQDAza/QiwlcZmZlZJvInE13jOSCYmRWBxxDMzAwPKpuZ2Vd82amZmUFxsp3WNwcEM7Na8lVGZma2nM8QzMwMcOoKMzMDfJWRmZkBWfJrnyGYmRlEUBEeVDYzM3ynspmZJe4yMjMzgvBlp2ZmlvEZgpmZAR5DMDMzSFcZOSCYmTV62X0IDghmZobHEMzMDHDqCjMzW85nCGZmBtEwxhCa1HcDzMxKXZDlMso3FULSnZLmSHo9p6ytpCclTUuvG+Ssu0jSdElTJR2aU76LpElp3Q2SlO/YDghmZkUQEXmnAt0F9Fuh7EJgTER0A8akZSR1BwYA26d9bpLUNO1zMzAI6JamFetciQOCmVkRRFTknQqrJ/4LzF+h+ChgaJofCvTPKb83IpZExAxgOtBHUkegdUQ8H1kkGpazT7U8hmBmVmtZNqMCtJM0Lmf51oi4tYD9OkTEhwAR8aGkjVJ5J+CFnO1mprJlaX7F8ho5IJiZ1dIqPCBnbkT0LuKhqxoXiBrKa+SAYGZWWwHldfuAnNmSOqazg47AnFQ+E9g0Z7vOwKxU3rmK8ho5IKzBlpQtmfvWx1Pfre921JF2wNz6bkSxvXXkMfXdhLrSID+vZPPaVlBBPL6obFm7Ajb9pn/DkcBA4Ir0+nBO+T2SrgE2IRs8fikiyiUtlLQ78CJwMvDHfAdRQ7iZwkqPpHFFPnW2OuTPa/WR9Hdgf7IgPBv4FfAQMBzYDHgPODYi5qftLwZ+CJQB50XEqFTem+yKpVbAKODsyPOF74Bg9cJfMKXFn1fj4MtOzcwMcECw+lPIpXa25vDn1Qi4y8jMzACfIZiZWeKAYGZmgAOCVUFSuaRXc6YuRay7f0rIVbn8G0kHFav+VOf+kh4pZp2rm6SQdHXO8s8k/bpIdf9a0gc5n+8Vxag31b2+pDNyljeRdH+x6s+p96l0WaUVkW9Ms6osjohedVR3f+ARYApARFxaR8cpdUuAoyVdHhF1cUPYtRHxhzqod33gDOAmgIiYBTTYu/UaGp8hWEEkvSOpXZrvLempNP/rlL/9KUlvSzonZ5+TJb0maaKkuyXtCRwJXJV+mW4p6S5Jx6TtD5Q0IeVwv1NSi5xjXybplbRu21TeR9JzaZ/nJG2zmv8sdamM7Mqe81dcIWlzSWPS33aMpM1S+V0p7/1z6bMo+ItY0imSbsxZfkTS/mn+c0lD0uf4gqQOqbyDpBGpfGL6fK8Atkyf71WSulTm9ZfUUtJf0mc4QdIBOcd+UNJoZfn+r8xpx82SxkmaLOmyVf8z2qpwQLCqtMrpThhRwPbbAocCfYBfSWouaXvgYqBvROwInBsRz5Hdan9BRPSKiLcqK5DUkuyuyuMjoifZ2evgnGPMjYidyXK8/yyVvQnsGxE7AZcCv6vFe14T/Qk4UVKbFcpvBIZFxA7A34AbctZ1BPYGDif7cq7O+Tmf8aE1bAewDvBC+hz/C5yWym8Ank7lOwOTyfL0v5U+3wtWqOdMgPT5ngAMTZ87QC/geKAncLykyvw8F6cb4nYA9pO0Q562Wi04IFhVFqf/0L0i4jsFbP9oysc+lyzpVgegL3B/ZXdH5W32NdgGmBER/0vLQ4F9c9Y/mF7HA13SfBvgvvQL9Fqyh4Q0GBGxgCyP/TkrrNoDuCfN300WACo9FBEVETGF7HOozrU5n/HjeZqylKybD77+9+9LFqCJiPKI+CxPPXun9hIRbwLvAlundWMi4rOI+JKsO7Eyv9Bxkl4BJpB9vt2xOuOAYIUq46t/Ly1XWLckZ76c7Ne9KCDdbo58j/erPEZl/QC/Bf4TET2AI6poV0NwHXAq2a/06uT+nXM/CwGk7p5XJb1aQx25ny98/W+5LCcHTu7ff1XV9Bmv9G9IUleys8ED09nQozTMz3iN4YBghXoH2CXNf7eA7ceQ/brbELJnwqbyhcB6VWz/JtBF0lZp+STg6TzHaAN8kOZPKaBNJSedWQ0nCwqVniN7bCLAicDYPHVcXHk2UMNm7wC9JDVJ3TV9CmjeGFK3nqSmklpT/ecLWXfTiWn7rckStU2tof7WwCLgszRucVgBbbJacECwQl0GXC/pGbJfcDWKiMnAEOBpSROBa9Kqe4EL0qDiljnbfwn8gKwLaBJQAdyS5zBXApdLehZommfbUnY1WebLSucAP5D0GlngPLcIx3gWmAFMAv4AvFLAPucCB6TPazywfUTMA56V9Lqkq1bY/iagadr+H8ApEbGEakTERLKuosnAnamNVoecusLMzACfIZiZWeKAYGZmgAOCmZklDghmZgY4IJiZWeKAYCVNX2VmfV3SfZLWrkVduXmVbldOVtYqtt0/5e5Z1WMszwlVSPkK23y+isf6taSf5d/SLOOAYKWuMs1GD7IUC6fnrpT0je5PiIgfpfQP1dkfWOWAYLYmc0CwhuQZYKv06/0/ku4BJqW7aK+S9HLKEPpjAGVulDRF0qPARpUVKSffvqR+yjKtTlSWXbQLWeCpTBC3j6T2kh5Ix3hZ0l5p3w0lPZFuxPsz+VN0IOkhSeNThs9BK6y7OrVljKT2qWzLlCl0vKRnlLLBmq0qPw/BGgRJzchSG4xORX2AHhExI32pfhYRuypLqf2spCeAnciS6vUkSwQ3heyO2Nx62wO3kWVVnSGpbUTMl3QL8HnlMwVS8Lk2IsYqS0f9OLAd8CtgbET8RtK3ga99wVfjh+kYrYCXJT2Q7gBeB3glIn4q6dJU91lkabJPj4hpknYjuyO47zf4M1oj54Bgpa5VTtK2Z4A7yLpyXoqIGan8EGAHffV8gDZAN7Jsqn+PiHJglqR/V1H/7sB/K+uqIWvrQUB3afkJQGtJ66VjHJ32fVTSJwW8p3MkVWaZ3TS1dR5ZOo9/pPK/Ag9KWje93/tyjt2igGOYrcQBwUrdSk93S1+Mi3KLgLNXTPMs6Vvkz8haaNbWJsAeEbG4irYUnB9G2UNpDkp1faHsQUTVZfiMdNxP6/AJd9aIeAzBGoPHgcGSmkOWaVPSOmTZNwekMYaOwAFV7Ps82YNZuqZ9q8va+gRZ9w1pu15pNjfD52HABnna2gb4JAWDbcnOUCo14avHUX6PrCtqATBD0rHpGJK0Y55jmFXJAcEag9vJxgdeUfYwnT+TnR2PAKaRZfi8mSrSbUfEx2T9/g+mrK2VXTb/BL5TOahMloG0dxq0nsJXVztdBuyr7CEvhwDv5WnraLJnAbxG9ryHF3LWLQK2lzSebIzgN6n8RODU1L7JwFEF/E3MVuJsp2ZmBvgMwczMEgcEMzMDHBDMzCxxQDAzM8ABwczMEgcEMzMDHBDMzCz5/+l0etxklcABAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate(w_avg, X_train, X_test, y_train, y_test, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results**\n",
    "- This composite classifier did indeed perform marginally better than the lone RandomForest on the testing scores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing on Holdout Set\n",
    "Now that we have our top performer, let's see how it does on the holdout set. This simulates the accuracy of the model on unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of Cross-Validation:\n",
      "\n",
      "Average accuracy: 0.8210455096697835        +/- 0.0037431677412490313\n",
      "Average precision: 0.8299271714429152        +/- 0.0026153039664668603\n",
      "**Average recall: 0.7655433174516902        +/- 0.009866371532775137\n",
      "Average f1 score: 0.7963999654838649        +/- 0.005407150440460352\n",
      "Average roc_auc: 0.8989890413424089        +/- 0.002955958156236045\n",
      "\n",
      "++++++++++++++++++++\n",
      "\n",
      "Results of Train-Test Split Validation:\n",
      "\n",
      "Training Scores:\n",
      "Train accuracy: 0.9999812828719561\n",
      "Train precision: 1.0\n",
      "**Train recall: 0.9999590683967091\n",
      "Train f1 score: 0.9999795337794969\n",
      "Train roc_auc: 0.9999795341983546\n",
      "\n",
      "<><><><><><><><><><>\n",
      "\n",
      "Testing Scores:\n",
      "Test accuracy: 0.8271854471955533\n",
      "Test precision: 0.8342674139311449\n",
      "**Test recall: 0.7729970326409495\n",
      "Test f1 score: 0.8024643819792068\n",
      "Test roc_auc: 0.8226293401402835\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlLElEQVR4nO3debxVVf3/8debIUAmJYYQVDAxE1FUIs0JhxTNFM0cvpZafVPJ0vg5hFmI9sMJzTSDwjSHBrPSwjnj50Q4gaIMiqJgoiQCioCK3Mvn98deF49w7zkHzrnce+59Px+P/WCftfdea5179HzOWmvvtRQRmJmZtWjoCpiZWePggGBmZoADgpmZJQ4IZmYGOCCYmVnSqqErYHXr0rVL9Np6q4auhm2A2c/OaOgq2AZaQyyOiG6l5LHfl/ePpUuWFjxv5rPPPxARQ0spqz45IDRivbbeiomP3dvQ1bANMKBz34augm2gldWrXys1j6VLljJx8n0Fz9u2fa+upZZVnxwQzMxKFqyJNQ1diZI5IJiZlSiAqjXVDV2NkjkgmJmVKtxCMDMzshbCGhwQzMwMWNME5oVzQDAzK5m7jMzMjNRl5IBgZmYRQVX4LiMzM8MtBDMzSzyobGZmhAeVzcyshgOCmZl5UNnMzDIBVLuFYGZm4EFlMzMDIDyXkZmZ1Typ7BaCmZnhMQQzM6PmLiMHBDOzZs93GZmZ2VoeQzAzs6yFgAOCmZkRbiGYmRmsAT7y1BVmZkY0jTGEFg1dATOzShcE1VF4K0TSVpIekvSCpFmSzkrpoyW9IWl62g7LueZ8SXMlzZF0SE767pJmpGPXSlKh8t1CMDMrgzINKlcBZ0fEM5I6AtMkPZiOXR0RV+aeLGlH4HigP7Al8C9J20dENTAeOBV4ArgXGArcl69wtxDMzEqUTV1ReCuYT8TCiHgm7S8HXgB65bnkSOC2iFgVEfOAucBgST2BThHxeEQEcAswrFD5DghmZmVQZJdRV0lTc7ZT68pPUh9gV+DJlPR9Sc9LulHSFimtF/B6zmULUlqvtL9uel7uMjIzK1EAHxX3pPLiiBhU6CRJHYC/AT+MiPckjQd+lor6GXAV8G2gtnGByJOelwOCmVmJoshB42JIak0WDP4QEXek/N/KOX49cHd6uQDYKufy3sCbKb13Lel5ucvIzKxE2ZPKhbdC0p1ANwAvRMTPc9J75px2FDAz7U8EjpfURlJfoB/wVEQsBJZL2iPleRLwj0Llu4VgZlYGZWoh7AV8E5ghaXpK+zFwgqSBZLFnPnAaQETMknQ7MJvsDqUz0h1GAMOBm4B2ZHcX5b3DCBwQzMxKls12WnpAiIjJ1N7/f2+ea8YAY2pJnwrstCHlOyCYmZVBdeU/qOyAYGZWqmwuo8qPCA4IZmalirKNITQoBwQzsxLV3GVU6RwQzMzKwGMIZmaW7jJq6FqUzgHBzKxEAawuauaKxs0BwcysROWcuqIhOSCYmZVBMdNbN3YOCGZmJRNrouCCZI2eA4KZWYl826mZma3lLiMzMyMCqnyXkZmZ1aypXOkcEMzMyqC4FTQbN6+YZmXx3wVv8u3DT+CIwQcxbI+D+f343wHw4vOzOfGgozhm78M4bsgRzJg2HYApDz3Gsft9laO+NJRj9/sqTz4yBYCVy1dwzN6Hrd322XY3Lh95cUO9rWalRYsW3P7YPVx3+w0AHDzsMO588p889+6r7LjrgPXO/0zvLXnyzVmc/IPvbuqqNkprovDW2FVsC0FSNTAjJ2lYRMwvU97DgJciYnZ6fTHwaET8qxz5pzyHAOdExOHlyrMhtWzVinP+7wXsOHAnVi5fwXFDvsqe++/Nzy+8lNN/dBb7fHkIj/7zIX4+6jJ+d89tbNGlC9fd9lu69+zBy7PncPrXTmbSC0/QvmMH/jr547VAjt3vqxz41UMa8J01H98Y/i3mvTSX9h07APDy7DmMOPF0Rl1zSa3nn3fpT5n84MObsIaNV0S2VbqKDQjABxExsJ7yHka2iPVsgIgYVU/lNBndPtOdbp/pDkD7jh3ou/12vLXwv0hi5fIVAKx4bzndevYA4PO79F977Xaf355VH67io1Wr+FSbNmvTX3tlHksXL2H3Lw3ehO+keeqx5WfY55ADuP7K6zjp+/8LwLyXXqnz/AO+cjAL5v+HD97/YFNVsdGrhBZAIU2qy0jSfEld0/4gSQ+n/dGSbpT0sKRXJZ2Zc81Jkp6X9JykWyV9CTgCGCtpuqTPSrpJ0jHp/AMlPStpRsqzTU7ZF0l6Jh3bIaUPljQlXTNF0uc28Z9lk3vjtQW8OGM2O+8+kB9dOoqrRl3KQf2/xFU/vYQfjjp3vfMfnHgfO+zc/xPBAODev97F0KO+QrZGuNWn8y4bxdWjLmVNEd9q7TZrx7dHnM74y67ZBDWrHGvWFN4au0oOCO3SF/Z0SXcWcf4OwCHAYOBCSa0l9QcuAA6IiF2AsyJiCjARODciBkbE2p9JktqSLVp9XEQMIGthDc8pY3FE7AaMB85JaS8C+0bErsAooPb298dlnCppqqSpSxcvKeJtNS7vr1jJiJOG86NLfkqHTh358w2/57wxP+Ffs6Zw7iU/YdQPRn7i/LkvvMTVF17Ohb9Yb0lY7r/jLg495ohNVfVma9+hB7B08RJmT59Z1Pnf+/EIbv3VDXyw8v16rlnlqLnLyGMIDWdDu4zuiYhVwCpJi4AewAHAXyNiMUBELC2Qx+eAeRHxUnp9M3AG8Iv0+o707zTg6LTfGbhZUj+y/25a5ysgIiYAEwAG7LZLBfwn9LHVq1cz4qThfOXrR3LQEUMBmHjbHYy8/EIADhn2FUafef7a8//7xkJ++I3TuOTXV7FV320+kdecGbOprqqi/8D1BzOtvHb94iD2P/Qg9vny/rRp24b2HTtw6fVXc/53R9R6/oBBA/nykYcx4uLz6di5ExFr+GjVKv404ZZNXPNGxGMIjVIVH7d62q5zbFXOfjXZexfZl3SxCvVd1JRRkz/Az4CHIuIoSX2AhzegvIoREVz4/R+x7fbbcXLqg4ZsbGHq5Cf5wj578OSjU9h62z4AvPfue5xx7Lc5a9R57LrHoPXyu/dvd3Ho19w62BSuuegKrrnoCgAG7b0Hp5z53TqDAcApQ49duz/8/B/y/oqVzTsYJE0hIFRyl1Ft5gO7p/2vFXH+JOBYSZ8GkNQlpS8HOtZy/otAH0nbpdffBB4pUEZn4I20f0oRdapIzz4xlbv+fCdPPjpl7S2jj/7zIUZfcyljfzKGr+11KNdePJYL0x0rf7r+Zl6f9xq/GfvLtecveXvx2vweuPMedxc1sAMOP4R/vfA4uwzelXF/uZFf3+kv/Xxq7jTKtzV2ikqoZS0krYiIDuuk7QPcALwFPAkMioghkkYDKyLiynTeTODwiJgv6WTgXLJf9c9GxCmS9gKuJ/vFfwzwU+DuiPirpAOBK8laAE8DwyNilaT5qbzFkgYBV6ay9yTrWnob+H/ANyOiTzG3nQ7YbZeY+Ni9dR22RmhA574NXQXbQCurV0+LiPWbqRug0+e2iy+Ou6Lgef866Gsll1WfKjYgNAcOCJXHAaHylCUgbL9dDP5V4YAw6eDGHRCa2hiCmVmDaAq/rR0QzMzKwAHBzMwIHBDMzAz8HIKZmX2sKUx/7YBgZlYGbiGYmRnggGBmZpBGlRu6EqVzQDAzK1HgMQQzM0vcZWRmZhBBVMKCBwU0tdlOzcwaRhSxFSBpK0kPSXpB0ixJZ6X0LpIelPRy+neLnGvOlzRX0hxJh+Sk755Wb5wr6VoVsfSgA4KZWRmUafrrKuDsiPg8sAdwhqQdgZHApIjoRzZt/0iAdOx4oD8wFBgnqWXKazxwKtAvbUMLFe6AYGZWDmVoIUTEwoh4Ju0vB14AegFHkk2jT/p3WNo/ErgtIlZFxDxgLjBYUk+gU0Q8HtmU1rfkXFMnjyGYmZVBkS2ArpKm5ryekJbNXU9aYXFXsrVdekTEwqycWCipezqtF/BEzmULUtrqtL9uel4OCGZmpQqyJbYKW1zMegiSOgB/A34YEe/l6f6v7UDkSc/LXUZmZiXKZjuNglsxJLUmCwZ/iIg7UvJbqRuI9O+ilL4A2Crn8t7Amym9dy3peTkgmJmVQ3nuMhLZMsAvRMTPcw5NBE5O+ycD/8hJP15SG0l9yQaPn0rdS8sl7ZHyPCnnmjq5y8jMrFTlm7piL+CbwAxJ01Paj4HLgNslfQf4D/B1gIiYJel2YDbZHUpnRERN59Vw4CagHXBf2vJyQDAzK4NyPKkcEZOpvf8f4MA6rhkDjKklfSqw04aU74BgZlYOlf+gsgOCmVnJAmgCU1c4IJiZlYNnOzUzM8BdRmZmlvH012Zm1vRXTJP0S/K8xYg4s15qZGZWiZr4oPLUPMfMzCxX5ceDugNCRNyc+1pS+4hYWf9VMjOrQE0gIBScy0jSnpJmk83LjaRdJI2r95qZmVWKmucQCm2NXDGT2/0COARYAhARzwH71mOdzMwqTxkmt2toRd1lFBGvrzMfd3Ezf5uZNRcV8IVfSDEB4XVJXwJC0qeAM0ndR2ZmRvYQQgV0CRVSTJfR6cAZZMuvvQEMTK/NzKxGc+gyiojFwImboC5mZpWrAr7wCynmLqNtJd0l6W1JiyT9Q9K2m6JyZmYVowm0EIrpMvojcDvQE9gS+Avwp/qslJlZxYkovDVyxQQERcStEVGVtt9TEbHOzGwTCbLprwttjVy+uYy6pN2HJI0EbiN728cB92yCupmZVY4K+MIvJN+g8jSyAFDzAMJpOccC+Fl9VcrMrOJUQJdQIfnmMuq7KStiZlaxKmTQuJCinlSWtBOwI9C2Ji0ibqmvSpmZVZzmEBAkXQgMIQsI9wKHApMBBwQzsxpNICAUc5fRMcCBwH8j4lvALkCbeq2VmVklCaA6Cm+NXDFdRh9ExBpJVZI6AYsAP5hmZpar8X/fF1RMQJgqaXPgerI7j1YAT9VnpczMKo2aQ0CIiO+l3V9Luh/oFBHP12+1zMwqTFO+7VTSbvmORcQz9VMlM7MKVPnxIG8L4ao8xwI4oMx1sXW89PwcDthmv4auhm2AXe/wNF+VZvKRx5SeSVN/DiEi9t+UFTEzq1xNY4Gcoh5MMzOzAio/HjggmJmVrGa20wrngGBmVg5N4C6jYlZMk6RvSBqVXm8taXD9V83MrII0kxXTxgF7Aiek18uBX9VbjczMKlETCAjFdBl9MSJ2k/QsQES8I+lT9VwvM7PKEaAmcJdRMS2E1ZJakuKbpG40ieETM7MyKlMLQdKNkhZJmpmTNlrSG5Kmp+2wnGPnS5oraY6kQ3LSd5c0Ix27VpLWLWtdxQSEa4E7ge6SxpBNfX1JcW/NzKyZKF+X0U3A0FrSr46IgWm7F0DSjsDxQP90zbj0Ax5gPHAq0C9tteX5CcXMZfQHSdPIpsAWMCwiXij4lszMmpMy9ZtExKOS+hR5+pHAbRGxCpgnaS4wWNJ8snnnHgeQdAswDLgvX2bF3GW0NfA+cBcwEViZ0szMDIprHWQthK6SpuZsp25AKd+X9HzqUtoipfUCXs85Z0FK65X2103Pq5hB5XvI3orIltDsC8wha6KYmRkUO3XF4ogYtBG5jwd+RvZd/DOyuea+Tfa9vK7Ik55XMV1GA3Jfp1lQTyt0nZlZs1KPNxlFxFs1+5KuB+5OLxcAW+Wc2ht4M6X3riU9r2IGldet2DPAFzb0OjOzpkpkC+QU2jY6f6lnzsujgJo7kCYCx0tqI6kv2eDxUxGxEFguaY90d9FJwD8KlVOwhSDp/+S8bAHsBrxd3NswM2sGonyznUr6EzCEbLxhAXAhMETSQLJ2yHxSL01EzJJ0OzAbqALOiIjqlNVwsjuW2pENJucdUIbixhA65uxXkY0p/K2I68zMmo8ydRlFxAm1JN+Q5/wxwJha0qcCO21I2XkDQrqftUNEnLshmZqZNTuV/6By3iU0W0VEVb6lNM3MjCYzdUW+FsJTZOMF0yVNBP4CrKw5GBF31HPdzMwqR+XHg6LGELoAS8jWUK65vzUABwQzsxpNvIXQPd1hNJP1H3So/HduZlZGpdxW2ljkCwgtgQ5s5BNvZmbNRoWsd1BIvoCwMCIu3mQ1MTOrYGoCS2jmCwgF5842M7OkumkHhAM3WS3MzCpZiVNTNBZ1BoSIWLopK2JmVrkim76iwhVz26mZmRXigGBmZjWznVY6BwQzs1IFqLpMa2g2IAcEM7NycJeRmZmBu4zMzAyy1oFbCGZmBjT5ye3MzKxITX3qCjMzK0YErKkufF4j54BgZlYGbiGYmVnGAcHMzAAIP5hmZmae3M7MzCB7KE3VHlQ2MzPCXUZmZpa4y8jMzAjcQjAzM3CXkZmZfcxdRmZmlrUQqhq6EiVzQDAzK1EA4S4jMzNLEaGha1EyBwQzs5J5UNmsTu07deCcK86j7/Z9iYCx515Gt57dOHnEt9h6u2343hGn8dLzcwDYfZ9BfHfkabRq3Zqq1av5zZjxPDvlmQZ+B03bR4uX8Nq141j97rtI4tNfPpDuhx9K1fIVzP/5NXy0aDGf6t6VPmefRasOHYiqKv4zfgLvvzqfqK6my5B9+MzRwz6R5yuXjuWjtxbx+V+MbZg31dAcEOomKYCfR8TZ6fU5QIeIGF2GvEcD3wXeTkn3R8TIUvNNeW8O/E9EjEuvtwSujYhjypF/TjkPA+dExNRy5ttYfH/0mTz98JNcdPooWrVuRZt2bVnx3gouPPUnjLj0nE+cu2zpMi749kiWvLWEPtv35YrfX8mxg7/WQDVvHtSyBb1O+QabbduX6g8+YM65P6bjLgNY+tAjdBiwE585+kj+e8c/eOvOifT65v/wzuNPsmZ1FZ+/+grWrFrFC2edwxZ770Wb7t0AePeJp2jZtm0Dv6sG1gQCQot6zHsVcLSkrvWU/9URMTBtZQkGyebA92peRMSb5Q4GTd1mHTZj58G7cO9t9wBQtbqKle+t4D9zX+P1V19f7/y5s15myVtLAJj/0jxat/kUrT/VepPWublpvcUWbLZtXwBatmtH2969WL10Kcuensan998XgE/vvy/Lnvr498qaD1cR1dWs+egj1KoVLdu1A6D6gw9ZdNe99DjmqE3/RhqLCGJNdcGtsavPgFAFTABGrHtA0jaSJkl6Pv27dUq/SdK1kqZIelVS0V/Ekk6RdF3O67slDUn7KySNkfScpCck9UjpPSTdmdKfk/Ql4DLgs5KmSxorqY+kmen8tpJ+J2mGpGcl7Z9T9h2S7pf0sqQrcuoxXtJUSbMkXbThf8bK03PrLVm29F3Ou+p8fnPvbzn78vNo2664X4/7HrYfc2e9zOqPVtdzLa3GqkVv8/68+bTvtx1V7y6j9RZbAFnQqFr2HgBb7PlFWrRtw8z/Hc6s035A9yMOp1XHDgAsvO12uh/xFVq0adNg76FxWFPEVpikGyUtqvneSWldJD2Yvl8elLRFzrHzJc2VNEfSITnpu6fvqrnpe1WFyq7PgADwK+BESZ3XSb8OuCUidgb+AFybc6wnsDdwONmXc11GpC/t6bl/hDq0B56IiF2AR8m6m0jlPpLSdwNmASOBV1LL49x18jkDICIGACcAN0uq+aYbCBwHDACOk7RVSr8gIgYBOwP7Sdo5X0UlnZoCyNTqaPy/KGrTslVL+u3Uj4m3/p3TDvtfPvzgQ0743okFr+uzfR9OPf90rj7/yk1QS4Ps1/28sVfT+1sn0XKzzeo8b+XcV1CLFux0/Th2HH8Ni+66h1X/fYv3581n1cK32PyLX9iEtW6Mgog1Bbci3QQMXSdtJDApIvoBk9JrJO0IHA/0T9eMk9QyXTMeOBXol7Z181xPvQaEiHgPuAU4c51DewJ/TPu3kgWAGn+PiDURMRvokSf73C6jBwpU5SPg7rQ/DeiT9g8g+6MREdURsaxAPnun+hIRLwKvAdunY5MiYllEfAjMBrZJ6cdKegZ4luxD2zFfARExISIGRcSglms/18ry9sK3eXvh27w4/QUAHr33YfrttH3ea7p+phsXTRjDpSPG8OZrb26KajZ7UVXFvLFX02Wfvdh8j8EAtNq8M6vfeQeA1e+8Q6vOnQB457F/02ngLqhVK1p37kz7Hbbn/VdeZeWcl3n/1VeZdfoPePmC0axauJCXR13cYO+pQcWawlsx2UQ8CixdJ/lI4Oa0fzMwLCf9tohYFRHzgLnAYEk9gU4R8XhEBNn38DAKqO8WAsAvgO+Q/UqvS+4NvKty9gWQunumS5qeJ48qPvl+cvsoVqc/CkA1Gz+Ynq/JlVvvaqCVpL7AOcCBqTV0zzr1apLeeXspixYuYqtts0bSbnvtzmsvz6/z/PadOnDpTZfz28snMGvqzDrPs/KJCF4bN4G2vbek+xFfWZveedDuLHnoUQCWPPQonb+wOwCf6tqV5TNnERFUf/gh7780l7a9tqTb0C8z4Lfj6f/rX9JvzGja9OxJv4tHNch7alhRbEDoWtMDkLZTiyygR0QsBEj/dk/pvYDcgbkFKa1X2l83Pa96DwgRsRS4nSwo1JhC1swBOBGYXCCPC2paA3lOmw8MlNQiddcMLqJ6k4DhAJJaSuoELAc61nH+o6m+SNoe2BqYkyf/TsBKYFkatzi0iDo1Cb8cdQ0/vvanXP/A7/jsjtvxh1/dyt6H7MOfn/wrO+7Wn0t+dzmX35p1DR118tFs2acX3zzzJCbcdwMT7ruBzT+9ecO+gSZu5YtzeOeRx1g+YxYvnj2SF88eybJpz9Lj6CNY/vwMZp8xguXPz6DHUUcC0HXowaz58ENe/OG5vPSjn9Bl//1o12ebAqU0I0Gxg8qLa3oA0jahxJJr+5EaedLz2lTPIVwFfD/n9ZnAjZLOJbt19FtlKOPfwDxgBjATKOZG9rOACZK+Q/arfnhEPC7p32lA5z6ycZAa44BfS5pB1iI5JSJW1TVWExHPSXqWbGzi1VTHZuGV2XMZfvgnf/xMfuAxJj/w2Hrn/v6Xt/D7X96yqapmQIfP78Cuf/tTrcf6jf7Jemkt27Wl7zk/zJtnm+7dmu8zCARR5KDxRnpLUs+IWJi6gxal9AXAVjnn9QbeTOm9a0nPS9EEHrduqtq0ahtbbt678InWaPS+8fKGroJtoMlHHjMt3fix0Tpu1jUGfu6rhcuaflNRZUnqA9wdETul12OBJRFxmaSRQJeIOE9Sf7Lx2MHAlmS9Hv0iolrS08APgCeBe4FfRsS9+cr1k8pmZiXKpjIqz49rSX8ChpCNNywALiS74/L21JvxH+DrZGXOknQ72Y0sVcAZEWtvTxxOdsdSO7LejvsKle2AYGZWsvLNZRQRJ9Rx6MA6zh8DjKklfSqw04aU7YBgZlYOTWDqCgcEM7NSRRAV+iBpLgcEM7My8AI5ZmYGOCCYmRmQDSpX/i38DghmZmXgFoKZmZE9ieCAYGbW7EXAmgpYAKcQBwQzs5KV78G0huSAYGZWBvU8ud0m4YBgZlYOvsvIzMxqltCsdA4IZmYl8qCymZklvu3UzMwSdxmZmRlQvgVyGpIDgplZyTyobGZmazkgmJk1exHhu4zMzCwTeAzBzMzwXUZmZgZ4UNnMzD7m207NzAw826mZmeG7jMzMLIdbCGZmBnjqCjMzA3yXkZmZAdnk124hmJkZRLAmPKhsZmb4SWUzM0vcZWRmZgTh207NzCzjFoKZmQEeQzAzM0h3GVV+QGjR0BUwM6t02XMIawpuxZA0X9IMSdMlTU1pXSQ9KOnl9O8WOeefL2mupDmSDinlfTggmJmVQUQU3DbA/hExMCIGpdcjgUkR0Q+YlF4jaUfgeKA/MBQYJ6nlxr4HBwQzs5JF2VoIdTgSuDnt3wwMy0m/LSJWRcQ8YC4weGMLcUAwMyuDIlsIXSVNzdlOrS0r4J+SpuUc7xERC1M5C4HuKb0X8HrOtQtS2kbxoLKZWami6LuMFud0A9Vlr4h4U1J34EFJL+Y5V7XXZuM4IJiZlSgo31xGEfFm+neRpDvJuoDektQzIhZK6gksSqcvALbKubw38ObGlu0uIzOzMijHoLKk9pI61uwDBwMzgYnAyem0k4F/pP2JwPGS2kjqC/QDntrY9+AWgplZGZTpwbQewJ2SIPt+/mNE3C/paeB2Sd8B/gN8PSszZkm6HZgNVAFnRGx8U8UBwcysZNlsRiXnEvEqsEst6UuAA+u4ZgwwpuTCcUAwMyuZF8gxM7NMQHUTWCBHTSGqNVWS3gZea+h61JOuwOKGroQVrSl/XttERLdSMpB0P9nfqJDFETG0lLLqkwOCNQhJU4u4H9saCX9ezYNvOzUzM8ABwczMEgcEaygTGroCtkH8eTUDHkMwMzPALQQzM0scEMzMDHBAsFpIqk7L99VsfcqY97C0ylPN64slHVSu/FOeQyTdXc48NzVJIemqnNfnSBpdprxHS3oj5/O9rBz5prw3l/S9nNdbSvprufLPyfdhSb4Ntsz8pLLV5oOIGFhPeQ8D7iabjIuIGFVP5VS6VcDRki6NiPp4IOzqiLiyHvLdHPgeMA7WTuV8TD2UY/XALQQrSlr4u2vaHyTp4bQ/WtKN6Rfbq5LOzLnmJEnPS3pO0q2SvgQcAYxNv0w/K+kmScek8w+U9GxaYPxGSW1yyr5I0jPp2A4pfbCkKemaKZI+t4n/LPWpiuzOnhHrHpC0jaRJ6W87SdLWKf0mSdemv8WrNX/XYkg6RdJ1Oa/vljQk7a+QNCZ9jk9I6pHSe0i6M6U/lz7fy4DPps93rKQ+kmam89tK+l36DJ+VtH9O2XdIul/ZIvJX5NRjfFpZbJakizb8z2gbwgHBatMupzvhziLO3wE4hGwhjwsltZbUH7gAOCAidgHOiogpZPO3n5sWEH+lJgNJbYGbgOMiYgBZ63V4ThmLI2I3YDxwTkp7Edg3InYFRgGXlPCeG6NfASdK6rxO+nXALRGxM/AH4NqcYz2BvYHDyb6c6zIi5zM+pEA92gNPpM/xUeC7Kf1a4JGUvhswi2zx91fS53vuOvmcAZA+3xOAm9PnDjAQOA4YABwnqWbRlwvSE9I7A/tJ2rlAXa0EDghWmw/S/9ADI+KoIs6/Jy3yvZhsJacewAHAX2u6OyJiaYE8PgfMi4iX0uubgX1zjt+R/p0G9En7nYG/pF+gVwP9i6hrxYiI94BbgDPXObQn8Me0fytZAKjx94hYExGzyT6Hulyd8xk/UKAqH5F188En//4HkAVoIqI6IpYVyGfvVF8i4kWyebq2T8cmRcSyiPiQrDtxm5R+rKRngGfJPt8dsXrjgGDFquLj/17arnNsVc5+Ndmve7Fha7vWtjZsbWXU5A/wM+ChiNgJ+Got9WoKfgF8h+xXel1y/865n4UAUnfPdEnT8+SR+/nCJ/+Wq+PjB5Zy//4bKt9nvN5/Q8pWADsHODC1hu6haX7GjYYDghVrPrB72v9aEedPIvt192kASV1S+nKgYy3nvwj0kbRdev1N4JECZXQG3kj7pxRRp4qTWla3kwWFGlOA49P+icDkAnlcUNMayHPafGCgpBapu2ZwEdWbROrWk9RSUifq/nwh6246MZ2/PbA1MCdP/p2AlcCyNG5xaBF1shI4IFixLgKukfQY2S+4vCJiFtkqTo9Ieg74eTp0G3BuGlT8bM75HwLfIusCmgGsAX5doJgrgEsl/RtouaFvqIJcxSenVj4T+Jak58kC51llKOPfwDxgBnAl8EwR15wF7J8+r2lA/7Sy178lzZQ0dp3zxwEt0/l/Bk6JiFXUISKeI+sqmgXcmOpo9chTV5iZGeAWgpmZJQ4IZmYGOCCYmVnigGBmZoADgpmZJQ4IVtH08cysMyX9RdJmJeSVO6/Sb5UzK2st5w5Jc/dsaBlr54QqJn2dc1ZsYFmjJZ1T+EyzjAOCVbqaaTZ2Ipti4fTcg5I26vmEiPjfNP1DXYYAGxwQzBozBwRrSh4Dtku/3h+S9EdgRnqKdqykp9MMoacBKHOdpNmS7gG612SknPn2JQ1VNtPqc8pmF+1DFnhqJojbR1I3SX9LZTwtaa907acl/TM9iPcbCk/RgaS/S5qWZvg8dZ1jV6W6TJLULaV9Ns0UOk3SY0qzwZptKK+HYE2CpFZkUxvcn5IGAztFxLz0pbosIr6gbErtf0v6J7Ar2aR6A8gmgptN9kRsbr7dgOvJZlWdJ6lLRCyV9GtgRc2aAin4XB0Rk5VNR/0A8HngQmByRFws6SvAJ77g6/DtVEY74GlJf0tPALcHnomIsyWNSnl/n2ya7NMj4mVJXyR7IviAjfgzWjPngGCVrl3OpG2PATeQdeU8FRHzUvrBwM76eH2AzkA/stlU/xQR1cCbkv5fLfnvATxak1eeWVsPAnaU1jYAOknqmMo4Ol17j6R3inhPZ0qqmWV2q1TXJWTTefw5pf8euENSh/R+/5JTdpsiyjBbjwOCVbr1VndLX4wrc5OAH6w7zbOkwyg8I2uxs7a2APaMiA9qqUvR88MoW5TmoJTX+8oWIqprhs9I5b5bjyvcWTPiMQRrDh4AhktqDdlMm5Lak82+eXwaY+gJ7F/LtY+TLczSN11b16yt/yTrviGdNzDt5s7weSiwRYG6dgbeScFgB7IWSo0WfLwc5f+QdUW9B8yT9PVUhiTtUqAMs1o5IFhz8Fuy8YFnlC2m8xuy1vGdwMtkM3yOp5bptiPibbJ+/zvSrK01XTZ3AUfVDCqTzUA6KA1az+bju50uAvZVtsjLwcB/CtT1frK1AJ4nW+/hiZxjK4H+kqaRjRFcnNJPBL6T6jcLOLKIv4nZejzbqZmZAW4hmJlZ4oBgZmaAA4KZmSUOCGZmBjggmJlZ4oBgZmaAA4KZmSX/HwasEHP55L0rAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate(w_avg, X_train_big, X_holdout, y_train_big, y_holdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Evaluation and Conclusion\n",
    "\n",
    "As we can see, the composite VotingClassifier model performs very well with the holdout data! The scores are all the highest of any model, and importantly the recall is the highest. \n",
    "\n",
    "The problem we are trying to address is to create a model that can predict the function state of a water pump based on the data collected about it. In particular, we care about whether a pump is correctly identified as non-functional so we can reduce the number of people suffering without water. Because of this, recall was deemed the most important metric, as it gives the correct guess frequency among the pumps that are indeed non-functional.\n",
    "\n",
    "As our final composite classifier performed so well on the holdout data, we are highly confident that our model is generalizable to unseen data and would be an effective tool for the government of Tanzania. However, it is worth noting that this model can only do so much with the data it received. If the Tanzanian government wished to improve their predictive capabilities, we would recommend they revisit their data collection methods, as more robust descriptors and information would lead to a more robust predictive model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (learn-env)",
   "language": "python",
   "name": "learn-env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}