Advanced Learning Algorithms/C2_W4_Lab_02_Tree_Ensemble.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ungraded Lab - Trees Ensemble\n",
    "\n",
    "In this notebook, you will:\n",
    "\n",
    " - Use Pandas to perform one-hot encoding of a dataset\n",
    " - Use scikit-learn to implement a Decision Tree, Random Forest and XGBoost models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import the libraries we will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "from xgboost import XGBClassifier\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('./deeplearning.mplstyle')\n",
    "\n",
    "RANDOM_STATE = 55 ## We will pass it to every sklearn call so we ensure reproducibility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Datatset\n",
    "- This dataset is obtained from Kaggle: [Heart Failure Prediction Dataset](https://www.kaggle.com/datasets/fedesoriano/heart-failure-prediction)\n",
    "\n",
    "#### Context\n",
    "- Cardiovascular disease (CVDs) is the number one cause of death globally, taking an estimated 17.9 million lives each year, which accounts for 31% of all deaths worldwide. Four out of five CVD deaths are due to heart attacks and strokes, and one-third of these deaths occur prematurely in people under 70 years of age. Heart failure is a common event caused by CVDs.\n",
    "- People with cardiovascular disease or who are at high cardiovascular risk (due to the presence of one or more risk factors such as hypertension, diabetes, hyperlipidaemia or already established disease) need early detection and management.  \n",
    "- This dataset contains 11 features that can be used to predict possible heart disease.\n",
    "- Let's train a machine learning model to assist with diagnosing this disease.\n",
    "\n",
    "#### Attribute Information\n",
    "- Age: age of the patient [years]\n",
    "- Sex: sex of the patient [M: Male, F: Female]\n",
    "- ChestPainType: chest pain type [TA: Typical Angina, ATA: Atypical Angina, NAP: Non-Anginal Pain, ASY: Asymptomatic]\n",
    "- RestingBP: resting blood pressure [mm Hg]\n",
    "- Cholesterol: serum cholesterol [mm/dl]\n",
    "- FastingBS: fasting blood sugar [1: if FastingBS > 120 mg/dl, 0: otherwise]\n",
    "- RestingECG: resting electrocardiogram results [Normal: Normal, ST: having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV), LVH: showing probable or definite left ventricular hypertrophy by Estes' criteria]\n",
    "- MaxHR: maximum heart rate achieved [Numeric value between 60 and 202]\n",
    "- ExerciseAngina: exercise-induced angina [Y: Yes, N: No]\n",
    "- Oldpeak: oldpeak = ST [Numeric value measured in depression]\n",
    "- ST_Slope: the slope of the peak exercise ST segment [Up: upsloping, Flat: flat, Down: downsloping]\n",
    "- HeartDisease: output class [1: heart disease, 0: Normal]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now load the dataset. As we can see above, the variables:\n",
    "\n",
    "- Sex\n",
    "- ChestPainType\n",
    "- RestingECG\n",
    "- ExerciseAngina\n",
    "- ST_Slope\n",
    "\n",
    "Are *categorical*, so we must one-hot encode them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset using pandas\n",
    "df = pd.read_csv(\"heart.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>ChestPainType</th>\n",
       "      <th>RestingBP</th>\n",
       "      <th>Cholesterol</th>\n",
       "      <th>FastingBS</th>\n",
       "      <th>RestingECG</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>ExerciseAngina</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>ST_Slope</th>\n",
       "      <th>HeartDisease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>40</td>\n",
       "      <td>M</td>\n",
       "      <td>ATA</td>\n",
       "      <td>140</td>\n",
       "      <td>289</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>172</td>\n",
       "      <td>N</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Up</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49</td>\n",
       "      <td>F</td>\n",
       "      <td>NAP</td>\n",
       "      <td>160</td>\n",
       "      <td>180</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>156</td>\n",
       "      <td>N</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37</td>\n",
       "      <td>M</td>\n",
       "      <td>ATA</td>\n",
       "      <td>130</td>\n",
       "      <td>283</td>\n",
       "      <td>0</td>\n",
       "      <td>ST</td>\n",
       "      <td>98</td>\n",
       "      <td>N</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Up</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>48</td>\n",
       "      <td>F</td>\n",
       "      <td>ASY</td>\n",
       "      <td>138</td>\n",
       "      <td>214</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>108</td>\n",
       "      <td>Y</td>\n",
       "      <td>1.5</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>54</td>\n",
       "      <td>M</td>\n",
       "      <td>NAP</td>\n",
       "      <td>150</td>\n",
       "      <td>195</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>122</td>\n",
       "      <td>N</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Up</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age Sex ChestPainType  RestingBP  Cholesterol  FastingBS RestingECG  MaxHR  \\\n",
       "0   40   M           ATA        140          289          0     Normal    172   \n",
       "1   49   F           NAP        160          180          0     Normal    156   \n",
       "2   37   M           ATA        130          283          0         ST     98   \n",
       "3   48   F           ASY        138          214          0     Normal    108   \n",
       "4   54   M           NAP        150          195          0     Normal    122   \n",
       "\n",
       "  ExerciseAngina  Oldpeak ST_Slope  HeartDisease  \n",
       "0              N      0.0       Up             0  \n",
       "1              N      1.0     Flat             1  \n",
       "2              N      0.0       Up             0  \n",
       "3              Y      1.5     Flat             1  \n",
       "4              N      0.0       Up             0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We must perform some data engineering before working with the models. There are 5 categorical features, so we will use Pandas to one-hot encode them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. One-hot encoding using Pandas\n",
    "\n",
    "First we will remove the binary variables, because one-hot encoding them would do nothing to them. To achieve this we will just count how many different values there are in each categorical variable and consider only the variables with 3 or more values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_variables = ['Sex',\n",
    "'ChestPainType',\n",
    "'RestingECG',\n",
    "'ExerciseAngina',\n",
    "'ST_Slope'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a reminder, one-hot encoding aims to transform a categorical variable with `n` outputs into `n` binary variables.\n",
    "\n",
    "Pandas has a built-in method to one-hot encode variables, it is the function `pd.get_dummies`. There are several arguments to this function, but here we will use only a few. They are:\n",
    "\n",
    " - data: DataFrame to be used\n",
    " - prefix: A list with prefixes, so we know which value we are dealing with\n",
    " - columns: the list of columns that will be one-hot encoded. 'prefix' and 'columns' must have the same length.\n",
    " \n",
    "For more information, you can always type `help(pd.get_dummies)` to read the function's full documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will replace the columns with the one-hot encoded ones and keep the columns outside 'columns' argument as it is.\n",
    "df = pd.get_dummies(data = df,\n",
    "                         prefix = cat_variables,\n",
    "                         columns = cat_variables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>RestingBP</th>\n",
       "      <th>Cholesterol</th>\n",
       "      <th>FastingBS</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>HeartDisease</th>\n",
       "      <th>Sex_F</th>\n",
       "      <th>Sex_M</th>\n",
       "      <th>ChestPainType_ASY</th>\n",
       "      <th>...</th>\n",
       "      <th>ChestPainType_NAP</th>\n",
       "      <th>ChestPainType_TA</th>\n",
       "      <th>RestingECG_LVH</th>\n",
       "      <th>RestingECG_Normal</th>\n",
       "      <th>RestingECG_ST</th>\n",
       "      <th>ExerciseAngina_N</th>\n",
       "      <th>ExerciseAngina_Y</th>\n",
       "      <th>ST_Slope_Down</th>\n",
       "      <th>ST_Slope_Flat</th>\n",
       "      <th>ST_Slope_Up</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>40</td>\n",
       "      <td>140</td>\n",
       "      <td>289</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49</td>\n",
       "      <td>160</td>\n",
       "      <td>180</td>\n",
       "      <td>0</td>\n",
       "      <td>156</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37</td>\n",
       "      <td>130</td>\n",
       "      <td>283</td>\n",
       "      <td>0</td>\n",
       "      <td>98</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>48</td>\n",
       "      <td>138</td>\n",
       "      <td>214</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>54</td>\n",
       "      <td>150</td>\n",
       "      <td>195</td>\n",
       "      <td>0</td>\n",
       "      <td>122</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  RestingBP  Cholesterol  FastingBS  MaxHR  Oldpeak  HeartDisease  \\\n",
       "0   40        140          289          0    172      0.0             0   \n",
       "1   49        160          180          0    156      1.0             1   \n",
       "2   37        130          283          0     98      0.0             0   \n",
       "3   48        138          214          0    108      1.5             1   \n",
       "4   54        150          195          0    122      0.0             0   \n",
       "\n",
       "   Sex_F  Sex_M  ChestPainType_ASY  ...  ChestPainType_NAP  ChestPainType_TA  \\\n",
       "0      0      1                  0  ...                  0                 0   \n",
       "1      1      0                  0  ...                  1                 0   \n",
       "2      0      1                  0  ...                  0                 0   \n",
       "3      1      0                  1  ...                  0                 0   \n",
       "4      0      1                  0  ...                  1                 0   \n",
       "\n",
       "   RestingECG_LVH  RestingECG_Normal  RestingECG_ST  ExerciseAngina_N  \\\n",
       "0               0                  1              0                 1   \n",
       "1               0                  1              0                 1   \n",
       "2               0                  0              1                 1   \n",
       "3               0                  1              0                 0   \n",
       "4               0                  1              0                 1   \n",
       "\n",
       "   ExerciseAngina_Y  ST_Slope_Down  ST_Slope_Flat  ST_Slope_Up  \n",
       "0                 0              0              0            1  \n",
       "1                 0              0              1            0  \n",
       "2                 0              0              0            1  \n",
       "3                 1              0              1            0  \n",
       "4                 0              0              0            1  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's choose the variables that will be the input features of the model.\n",
    "- The target is `HeartDisease`.\n",
    "- All other variables are features that can potentially be used to predict the target, `HeartDisease`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = [x for x in df.columns if x not in 'HeartDisease'] ## Removing our target variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We started with 11 features.  Let's see how many feature variables we have after one-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n"
     ]
    }
   ],
   "source": [
    "print(len(features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Splitting the Dataset\n",
    "\n",
    "In this section, we will split our dataset into train and test datasets. We will use the function `train_test_split` from Scikit-learn. Let's just check its arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function train_test_split in module sklearn.model_selection._split:\n",
      "\n",
      "train_test_split(*arrays, test_size=None, train_size=None, random_state=None, shuffle=True, stratify=None)\n",
      "    Split arrays or matrices into random train and test subsets.\n",
      "    \n",
      "    Quick utility that wraps input validation and\n",
      "    ``next(ShuffleSplit().split(X, y))`` and application to input data\n",
      "    into a single call for splitting (and optionally subsampling) data in a\n",
      "    oneliner.\n",
      "    \n",
      "    Read more in the :ref:`User Guide <cross_validation>`.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    *arrays : sequence of indexables with same length / shape[0]\n",
      "        Allowed inputs are lists, numpy arrays, scipy-sparse\n",
      "        matrices or pandas dataframes.\n",
      "    \n",
      "    test_size : float or int, default=None\n",
      "        If float, should be between 0.0 and 1.0 and represent the proportion\n",
      "        of the dataset to include in the test split. If int, represents the\n",
      "        absolute number of test samples. If None, the value is set to the\n",
      "        complement of the train size. If ``train_size`` is also None, it will\n",
      "        be set to 0.25.\n",
      "    \n",
      "    train_size : float or int, default=None\n",
      "        If float, should be between 0.0 and 1.0 and represent the\n",
      "        proportion of the dataset to include in the train split. If\n",
      "        int, represents the absolute number of train samples. If None,\n",
      "        the value is automatically set to the complement of the test size.\n",
      "    \n",
      "    random_state : int, RandomState instance or None, default=None\n",
      "        Controls the shuffling applied to the data before applying the split.\n",
      "        Pass an int for reproducible output across multiple function calls.\n",
      "        See :term:`Glossary <random_state>`.\n",
      "    \n",
      "    shuffle : bool, default=True\n",
      "        Whether or not to shuffle the data before splitting. If shuffle=False\n",
      "        then stratify must be None.\n",
      "    \n",
      "    stratify : array-like, default=None\n",
      "        If not None, data is split in a stratified fashion, using this as\n",
      "        the class labels.\n",
      "        Read more in the :ref:`User Guide <stratification>`.\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    splitting : list, length=2 * len(arrays)\n",
      "        List containing train-test split of inputs.\n",
      "    \n",
      "        .. versionadded:: 0.16\n",
      "            If the input is sparse, the output will be a\n",
      "            ``scipy.sparse.csr_matrix``. Else, output type is the same as the\n",
      "            input type.\n",
      "    \n",
      "    Examples\n",
      "    --------\n",
      "    >>> import numpy as np\n",
      "    >>> from sklearn.model_selection import train_test_split\n",
      "    >>> X, y = np.arange(10).reshape((5, 2)), range(5)\n",
      "    >>> X\n",
      "    array([[0, 1],\n",
      "           [2, 3],\n",
      "           [4, 5],\n",
      "           [6, 7],\n",
      "           [8, 9]])\n",
      "    >>> list(y)\n",
      "    [0, 1, 2, 3, 4]\n",
      "    \n",
      "    >>> X_train, X_test, y_train, y_test = train_test_split(\n",
      "    ...     X, y, test_size=0.33, random_state=42)\n",
      "    ...\n",
      "    >>> X_train\n",
      "    array([[4, 5],\n",
      "           [0, 1],\n",
      "           [6, 7]])\n",
      "    >>> y_train\n",
      "    [2, 0, 3]\n",
      "    >>> X_test\n",
      "    array([[2, 3],\n",
      "           [8, 9]])\n",
      "    >>> y_test\n",
      "    [1, 4]\n",
      "    \n",
      "    >>> train_test_split(y, shuffle=False)\n",
      "    [[0, 1, 2], [3, 4]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(train_test_split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(df[features], df['HeartDisease'], train_size = 0.8, random_state = RANDOM_STATE)\n",
    "\n",
    "# We will keep the shuffle = True since our dataset has not any time dependency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train samples: 734\n",
      "validation samples: 184\n",
      "target proportion: 0.5518\n"
     ]
    }
   ],
   "source": [
    "print(f'train samples: {len(X_train)}')\n",
    "print(f'validation samples: {len(X_val)}')\n",
    "print(f'target proportion: {sum(y_train)/len(y_train):.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Building the Models\n",
    "\n",
    "## 4.1 Decision Tree\n",
    "\n",
    "In this section, let's work with the Decision Tree we previously learned, but now using the [Scikit-learn implementation](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html). \n",
    "\n",
    "There are several hyperparameters in the Decision Tree object from Scikit-learn. We will use only some of them and also we will not perform feature selection nor hyperparameter tuning in this lab (but you are encouraged to do so and compare the results 😄 )\n",
    "\n",
    "The hyperparameters we will use and investigate here are:\n",
    "\n",
    " - min_samples_split: The minimum number of samples required to split an internal node. \n",
    "   - Choosing a higher min_samples_split can reduce the number of splits and may help to reduce overfitting.\n",
    " - max_depth: The maximum depth of the tree. \n",
    "   - Choosing a lower max_depth can reduce the number of splits and may help to reduce overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_samples_split_list = [2,10, 30, 50, 100, 200, 300, 700] ## If the number is an integer, then it is the actual quantity of samples,\n",
    "max_depth_list = [1,2, 3, 4, 8, 16, 32, 64, None] # None means that there is no depth limit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f45b7524490>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEXCAYAAAC6baP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/T0lEQVR4nO3deXyU1dXA8d9J2PdVQFBABARlX1UQEFHEVhA3cEWttva12kWt1ioUSzdtq7ZWX/fKq4CiKCoUBUGtKyA7CARkSdj3fQk57x/3hjwzmSyTTGYmyfl+PvPJzH2WOTNJ5sxz7/PcI6qKMcYYU1gpiQ7AGGNM6WKJwxhjTFQscRhjjImKJQ5jjDFRscRhjDEmKpY4jDHGRMUSh4kbEekvIioizRIdS1GJyBwReSHw+BURmVnANqNEJDNGz68ickMs9lUaiEgL/5r7JDoWk8MSh8nF/6Pmd1tXxF1/ATQBNsUu2oKJyBU+7vZ5LP+XiKwXkaL8P9wDXF28CCPGNFNEXomwqAkwOdbPF0/5vLZINuJe89clF5GJliUOE0mTwO1K39Y10NYjuLKIVCrMTlX1mKpuUdWsGMZaGO8Bm4HbwxeISDXgOuDFosSlqntVdXfxQyz0821R1SPxer5EEpFKqnrCv+bjiY7H5LDEYXLx/6hbVHULsMs3bw+0bRORu0XkdRHZC4wHEJFxIrJCRA6JyEYReVZEamfvN7yrKvB4kIh86rdbLiKX5hWbOB+IyFwRqejbUvy32E9FJDXC68kEXgJuFJHKYYuvAWoAL4pIVxGZLiLbROSAf47B+b1X4V1VPpZHA/uYBNQN26aliLwtIpv8a14iIjcG9wkMBG4OHOX198tCuqpEpImITBSRPSJy2HeldY/wnhf6PfbbjRGRNBG5RkRW++3eEZFaIjJcRFaKyH4RmRz8HfttR4jIQhE5IiLrRORvIlI9v9cW6JK6XkSmichB4NFIXVUicoqIvCwiW/1zrBSRW/2yiv750kXkqIhsFpGJ+b1WUwSqaje75XkD+gMKNAu0KbATuAtoBbT27b8F+gItcB8O3wH/zmtfgceLgMFAa+BlYB9QN5+YGuK6ux73jx/y8ZyWzzYtgBPAyLD2z4F3A/GMAs4G2gC/B44BbQLrzwFeCDx+BZgZeHwPcBC42e/jfmAPkBlYp4N/7zr59+9nQCYwwC+vDXwKTAIa+1ulwHt/g78vuC6chUAfv99JwG6gQTHf4zH+dXwAdAT6AduBD4FpPvY+wFbgz4HtRvnnvxE4A7gAWAyMz++1+d+PAunA9UBLf8tu7+O3rwqsAL4FLvLPcTEwwi//pd9Hf+B03NHxzxP9f1TWbgkPwG7JfSPvxPFiIba9AjgKpETaV+Dx8MA2jXzbJQXse4D/sB0NHAeGFSKe6cDHgcft/HNdls82i4CHAo/nkH/iSAfGhe1jMoHEkcfzvAs8H3g8E3glwnrBxDHQP24fWF4Z1y33SHHeY1ziyMQnIN/2NC75Ngy0PQnMCzxeB/wkbF8X+Oerm9drCySIh/Noz04ctwFHgn+PYes/CXwMSKL/d8ryzbqqTFF9E97guzA+9V0wB4DXcN8mGxewr4XZd1R1K+7DqVF+G6jqbOCvuA+4F1T1nULE/BzQX0TO9I9vBzbgEgoi0lDcQPl3vuvnAO7oo3kh9o2I1AKa4k4CCPpv2HrVRORPIrJMRHb55xlS2OcJOBvYqarLsxtU9SjuKOTssHUXBtYp1HsMZKjqjsDjLcAWVd0e1nYKuPfPv4a/+W66A/61TffrnknBcv1dhekGLFfV9DyWv4w78krzXaVXSiHH4EzhWeIwRXUw+EBEegFv4rohrsANpv/ELy7oH/dYhLZ8/zb9WMb5uA/AViIihYj5PdwH3e3+w+QmQgfFX8F1td3vf3bGfeDG+oPnMeAG4He4I6fOuO6fkvyAi/o9xh3JBWkebdn7yf55D+41Zd864brIlhQizoMFr5I3VV2I6+K6F/eanwQW+qRuYsQSh4mVPsAOVf2tqn6tqquAkrxeYwzuG+z5QE/ch32+NGeQfBRuULwO8GJglQuAf6nqVFVdguvyOaOwAanqPiADOC9s0flhjy8AXlPVN1R1EbAWNx4SdAzINdAfZhlQXwKnGfvB/17A0sLGHSv+SGYj0FZV0yLcss8GK8xry8t8oL3kcy2Qqh5Q1SmqejfQHdcl2a+Iz2cisMRhYmUl0FBEbhORM0TkJuCnJfFEItIPeAC4WVW/Bu7AnYHTsxCbvwA0AJ4CPlDVjMCylcD1ItJBRDoDE4j+A+6vwD0icqOItBaRX+EGcYNWAkNFpKf/0H8OODVsne+BbiLSSkQaiD+DLMzHuK6d10XkfBE5B3gVqAI8E2XcsfIQcLeIPCQi54hIWxEZJiL/G1inMK8tLxOA9cBUEbnIn6E2UESuBRCR+/yZWWeLSEvgVtxR6aoYvT6DJQ4TI6r6PjAO+AOuS2IEcF+sn0dE6gH/BzypqjP8c7+B62aaICI1C4hzHe7MoLq4D+ygW3D/E98A7wD/AeZGGeKTuKT0d1w317nA2LB1foH78JsNzMIdpYRf1PdXYAducH47uY9aUDcaPAx39toHPtbGwKCwsYm4UdXxuKO5H+Dex7m4o8Nggi7wteWz/0O4o4elwETcGVZP4862Ane22C+BL3F/h1cAV6rqyqK+JpObuL89Y4wxpnDsiMMYY0xULHEYY4yJiiUOY4wxUbHEYYwxJioVEh1ASdq7d6+N/BtjTDHUrl0718W1dsRhjDEmKpY4jDHGRKVMd1UF1a5du+CVjDHGsHfv3nyX2xGHMcaYqFjiMMYYE5Vy01VljCkbVJVdu3aRlRXv0vVlU0pKCvXq1aNwlQmcuCUOEXkJN/HZNlU9J8JywU0QNwQ4BIxS1W/9sptxZUkBfq+q//bt3XCT21XF1TO4R23yLWPKtF27dlG9enWqVKmS6FDKhCNHjrBr1y7q169f6G3i2VX1Cq7mcV4uxRV7aY2bJvsZODkb6mhcjYGewGgRqeu3eQZXxS17u/z2b4wpA7KysixpxFCVKlWiPnqLW+JQ1U+BXfmsMhR4VZ2vgDoi0gS4BPhIVXep6m7gI2CwX1ZLVb/yRxmv4qaYjqn0A7BqT6z3aowxpVcyDY43xVUPy5bu2/JrT4/QHjOfbIJuk+GH02Dv0Vju2RhTGu3cuZPOnTvTuXNnGjduTNOmTU8+PnYsUnXeHPPmzePuu++OU6QlywbHI1CFp5bAr76AEwrbDsPNH8PbgyGl8ONHxpgypn79+ixcuBCAMWPGUKNGDe69996TyzMzM6lQIfLHavfu3enevXs8wixxyZQ4MoDTAo+b+bYMoH9Y+xzf3izC+jGxcIdLGtneXQd/+hZ+0y1Wz2CMiQWJcZFcvTO69UeNGkWVKlVYsGAB559/PiNGjOCee+7hyJEjVK1alZdffpm2bdsyZ84cHn/8cd5//33GjBnDhg0bWLt2LRs2bODnP/95qToaSabEMRW4S0Qm4gbC96rqZhGZAfwhMCB+MfCgqu4SkX0i0hv4GrgJ+EcsAhGBf10Ai3fCt4ECnL/9Bro1hEtOj8WzGGPKivT0dL744gtSU1PZt28fn332GRUqVGDmzJn85je/4a233sq1zXfffcfs2bPZv38/bdu25c4776RixWjKrydOPE/HnYA7cmggIum4M6UqAqjqs7jTaYcAabjTcW/xy3aJyKPk1H4eq6rZg+w/Jed03On+FhNVK7iuqW6TYecR16bAyJkw7yo4o1asnskYU9pdffXVpKamAm66jptvvpnVq1cjIhw/fjziNpdddhmVK1emcuXKnHLKKWzdupVmzZpFXDfZxC1xqOrIApYr8D95LHsJeClC+zwg1zUhsdK8Jky4CAZ/AFm+22r3URj+H/jiCqhWOr4cGGNKWPXq1U/ef/jhhxkwYABTpkxh3bp19O/fP+I2lStXPnk/NTWVzMzMkg4zZpKpqyopDToN/tALHvgqp23RTrjjExg/0HVrGWMSJ9oxiZK2d+9emjZ1J3i+8soriQ2mhCTT6bhJ6/7OcOUZoW2vrYZ/Lk1IOMaYJHb//ffz4IMP0qVLl1J1FBENKcszdAQrABZ3WvX9x6DX27Bid05bhRT4+IfQ99Ri7doYE4Xt27fTsGHDRIdRpoS/p8Fp1a0CYDHUrARTLoGagXGNzCy4+kPYdDBxcRljTLxZ4ohC27rw6sDQtq2H4aoZcOxEYmIyxph4s8QRpWEt4aGuoW1fboWff56YeIwxJt4scRTB73rAJaeFtj2zDF7+LjHxGGNMPFniKILUFHj9ImhZM7T9zk9h/vbExGSMMfFiiaOI6lWBKYPdFebZjp5wFwfuOJy4uIwxpqRZ4iiGTg3guX6hbRsOwIiP3BlXxpiyZ8CAAcyYMSOk7YknnuDOOyNfidi/f3/mzZsHwJAhQ9izZ0+udcaMGcPjjz+e7/O+8847LF++/OTjRx55hJkzZ0YZfWxY4iimG9rA3R1C22ZlwENfJyYeY0zJGjlyJBMnTgxpmzhxIiNH5jurEgDTpk2jTp06RXre8MQxduxYLrrooiLtq7gsccTA4+dC3yahbX9ZCG+uSUg4xpQv/5TY3gpw1VVX8cEHH5ws3LRu3To2bdrEhAkT6N69O2effTajR4+OuG2LFi3YscNNuT1u3DjatGlDnz59WLly5cl1nn/+eXr06EGnTp248sorOXToEF988QVTp07lvvvuo3PnzqxZs4ZRo0YxefJkAGbNmkWXLl3o0KEDt956K0ePHj35fKNHj6Zr16506NCB776LzRk8ljhioGIqvHExNKkW2n7Lx7A8v2K5xphSp169evTs2ZPp091k3BMnTuSaa65h3LhxzJs3j8WLF/PJJ5+wePHiPPcxf/58Jk6cyMKFC5k2bRpz5849uWz48OHMnTuXRYsW0a5dO1588UXOO+88Lr/8ch577DEWLlxIq1atTq5/5MgRRo0axaRJk1iyZAmZmZk880xOkZIGDRrw7bffcueddxbYHVZYljhipHE1eOsSqBh4Rw9mwhX/sbKzxpQ1we6q7G6qN954g65du9KlSxeWLVsW0q0U7rPPPuOKK66gWrVq1KpVi8svv/zksqVLl9K3b186dOjAa6+9xrJly/KNZeXKlbRs2ZI2bdoAcPPNN/Ppp5+eXD58+HAAunXrxrp164r6kkNY4oihcxvDU31C21btdWVns8rulGDGlDtDhw5l1qxZfPvttxw6dIh69erx+OOPM2vWLBYvXsxll13GkSNHirTvUaNG8c9//pMlS5YwevToIu8nW/b07bGcut2mVY+xH7eHb7aFXgz47jr447fwkJWdNSb27or/t7IaNWowYMAAbr31VkaOHMm+ffuoXr06tWvXZuvWrUyfPj3POhwAF1xwAaNGjeLBBx8kMzOT9957jx//+McA7N+/nyZNmnD8+HFee+21k1O016xZk/379+faV9u2bVm3bh1paWmceeaZjB8/nn79+uVaL5bidsQhIoNFZKWIpInIAxGWNxeRWSKyWETmiEgz3z5ARBYGbkdEZJhf9oqIfB9Y1jlerycvIvB0X1diNujhb2D6+sTEZIyJvZEjR7Jo0SJGjhxJp06d6NKlC2eddRbXXXcd559/fr7bdu3alWuvvZZOnTpx6aWX0qNHj5PLHn30UXr16sX555/PWWeddbJ9xIgRPPbYY3Tp0oU1a3LOvKlSpQovv/wyV199NR06dCAlJYWf/OQnsX/BAXGZVl1EUoFVwCAgHVcGdqSqLg+s8ybwvqr+W0QuBG5R1RvD9lMPV1q2maoeEpFX/DaTIz1vLKdVj9b6/aFlZwHqVHJlZ1vFNxRjyhSbVj32knVa9Z5AmqquVdVjwERgaNg67YGP/f3ZEZYDXAVMV9VDJRZpjDSvCRMHQUrgLd9zDIbPgEORSxAbY0ypEK/E0RTYGHic7tuCFgHD/f0rgJoiUj9snRHAhLC2cb576+8iUpkkclEz+GOv0LbFO+H2T6AM188yxpRxyXRW1b1APxFZAPQDMoCTVS5EpAnQAQhe6/8gcBbQA6gH/Dpu0RbSfZ3hqrCys6+vhn8sSUg4xhhTbPFKHBlAcCLyZr7tJFXdpKrDVbUL8JBv2xNY5RpgiqoeD2yzWZ2jwMu4LrGkIgIvDYD2dUPbf/UlfLopMTEZY0xxxCtxzAVai0hLEamE63KaGlxBRBqISHY8DwIvhe1jJGHdVP4oBBERYBiwNPahF1/NSvD2JVCrUk5bdtnZjAOJi8uY0iglJaXY1zaYHEeOHCElJbpUEJezqgBEZAjwBJAKvKSq40RkLDBPVaeKyFXAHwEFPgX+xx9JICItgM+B01Q1K7DPj4GGgAALgZ+o6smP4kSeVRXJu9/DsP+EtvVuBHOGQuXUxMRkTGmjquzatYusLJuCOhZSUlKoV68e7vu3U9BZVXFLHImQbIkD3PUcv58f2vaT9vBMyV6vY4wxhZYsp+Mab0x3GBxWdvbZ5fDSisTEY4wx0bLEEWepKfDaRXBGrdD2n34G87YlJiZjjImGJY4EqFfFDZbnKjs7A7Zb2VljTJKzxJEgnRrA82HjGhut7KwxphSwxJFA17eBe8LKzn6cAb+xsrPGmCRmiSPBHotQdvaxhVZ21hiTvCxxJFh22dlTq4e23/IxLLOys8aYJGSJIwk0rgZvXWxlZ40xpYMljiTRuzH8I6zs7Oq9cJOVnTXGJBlLHEnkjvZw61mhbVPXwbj5EVc3xpiEsMSRRLLLznYPK242ei5Ms7KzxpgkYYkjyVSpAG9dAg2q5LQpcP1MWLM3z82MMSZuLHEkodNrwqQIZWev+A8ctLKzxpgEs8SRpC5sBn/uHdq2ZBfcYWVnjTEJZokjif2qE1zdKrTt9dXwlJWdNcYkkCWOJJZn2dkv4BMrO2uMSRBLHEmuRkWYMji07OwJhWs+hHQrO2uMSYC4JQ4RGSwiK0UkTUQeiLC8uYjMEpHFIjJHRJoFlp0QkYX+NjXQ3lJEvvb7nOTrmZc5berA+AtD27YdhqtmuOnYjTEmnuKSOEQkFXgauBRoD4wUkfZhqz0OvKqqHYGxuPrj2Q6ramd/uzzQ/mfg76p6JrAbuK3EXkSCXd4SHu4W2vb1Nrjnv4mJxxhTfsXriKMnkKaqa1X1GDARGBq2TnvgY39/doTlIcRVVr8QmOyb/g0Mi1XAyWhMDxhyemjb/y6HF63srDEmjuKVOJoCGwOP031b0CJguL9/BVBTROr7x1VEZJ6IfCUiw3xbfWCPqmbms88yJUXg/yKUnf2fz2CulZ01xsRJMg2O3wv0E5EFQD8gA8juwW+uqt2B64AnRKRVHvso8+pWdoPl4WVnr5wB2w4lLi5jTPkRr8SRAZwWeNzMt52kqptUdbiqdgEe8m17/M8M/3MtMAfoAuwE6ohIhbz2WVZ1rA8vWNlZY0yCxCtxzAVa+7OgKgEjgKnBFUSkgYhkx/Mg8JJvrysilbPXAc4Hlquq4sZCrvLb3Ay8W+KvJElc1wZ+3jG0bfYmeOCrxMRjjCk/4pI4/DjEXcAMYAXwhqouE5GxIpJ9llR/YKWIrAIaAeN8eztgnogswiWKP6nqcr/s18AvRSQNN+bxYjxeT7L4S2/od2po218XwRtpiYnHGFM+iJbhiY/27t178sXVrl07kaGUmK2HoNtkyDiY01a9Anw1HM6pn/d2xhiTl717c6birl27toQvT6bBcVMEjaq5adgrhZWdHT4D9ljZWWNMCbDEUQb0agT/6Bvatnov3DjLys4aY2LPEkcZcXs7uC2s7Oz76+H3VnbWGBNjljjKCBH4Z4Sys2PmwgdWdtYYE0OWOMqQvMrO3jAT0qzsrDEmRixxlDGn14Q3Ls5ddna4lZ01xsSIJY4yaEBTd41H0JJdcPscKztrjCk+Sxxl1C87wTVhM3pNSIMnFycmHmNM2WGJo4wSgRcHwNlhZWfv/RLmlIsZvYwxJcUSRxlmZWeNMSXBEkcZ17oOvDYwtG37ETcNu5WdNcYUhSWOcuAHLWB099C2b7bB3VZ21hhTBJY4yolHuucuO/vccnhheeT1jTEmL5Y4yonssrOtIpSd/WZrYmIyxpROljjKkbqV4e3BUC1QdvZYlpWdNcZExxJHOdOxPrzQP7Qt/SBca2VnjTGFFLfEISKDRWSliKSJyAMRljcXkVkislhE5ohIM9/eWUS+FJFlftm1gW1eEZHvRWShv3WO1+spzUa2hl+ElZ2dY2VnjTGFFJfEISKpwNPApUB7YKSItA9b7XHgVVXtCIwF/ujbDwE3qerZwGDgCRGpE9juPlXt7G8LS/BllCl/ORf6Ryg7O8nKzhpjClDoxCEiU0RkmIhULMLz9ATSVHWtqh4DJgJDw9ZpD3zs78/OXq6qq1R1tb+/CdgGhE0ebqJVIQUmDYKm1UPbb50NS3cmJiZjTOkQzRHHZ8AjwBYReUZEzoti26bAxsDjdN8WtAgY7u9fAdQUkZCq2SLSE6gErAk0j/NdWH8XkcpRxFTunRKh7OyhTLjCys4aY/JR6MShqn9T1a7ABcAeYIKIrBaRR0SkVf5bF8q9QD8RWQD0AzKAk9c2i0gTYDxwi6pmD+M+CJwF9ADqAb+OQRzlSq9GrgBUUJqVnTXG5CPqMQ5VXaaqDwI34MYfRgPfishMEemUx2YZwGmBx818W3C/m1R1uKp2AR7ybXsARKQW8AHwkKp+FdhmszpHgZdxXWImSre3hx+1C217fz08Oi8x8RhjkltUiUNE2orIoyKyBngOmAS0ABoB04B38th0LtBaRFqKSCVgBDA1bN8NRCQ7ngeBl3x7JWAKbuB8ctg2TfxPAYYBS6N5PSbHP/pAj1NC28bMg/fXJSQcY0wSi2ZwfB7wOa5L6DpVbaeqf1DVjap6RFX/lte2qpoJ3AXMAFYAb6jqMhEZKyKX+9X6AytFZBUuEY3z7dfgusdGRTjt9jURWQIsARoAvy/s6zGhssvONqwS2n7DLCs7a4wJJVrIknAichUw1Z8VVSrs3bv35IurXbt2IkMpNWZnwKD33PTr2c6pB18Nh+pFOZ/OGFPq7N2b822xdu3aEr48mq6qfbhuqZN819WgogZnks+Apu4aj6Clu+BHc6zsrDHGiSZxPA3sD2vb79tNGfKLjnDtmaFtE9PgCSs7a4whusRxiqpuDmvbDDSOYTwmCYjAi/1dF1XQfVZ21hhDdIljrYhcGNbWH/g+duGYZFHdl52tHaHs7EYrO2tMuRZN4hgDvC0ifxWRn4rIX4G3cFeTmzLozNrw2kWhbdllZ49kJiYmY0ziRXPl+LvAxUB14DL/8xLfbsqoy5rDmLCys3Ot7Kwx5VqhT8ctjex03NjIUhg63V1NHvRcP3fVuTGmbCnodNyoEoe/8K4v7mK7kztT1aTsrrLEETt7jkL3ybBmX05bpRT4bBj0bJSwsIwxJSBm13GIyB24K8cvxE0m2AH4FXBmftuZsqFOZTdYbmVnjTHRDI7fDwxW1SuAw/7nVcDxEonMJJ0O9d1pukFWdtaY8ifa6zg+8/ezRCRFVacDPyyBuEySGtEafhU2B/KcTfBrKztrTLkRTeJIF5EW/v4qYKiI9AVKzdxVJjb+1Dt32dm/LYKJqxMTjzEmvqJJHH8Bsqs2jAX+D1fq9XexDsokt+yys83Cys7eNgeWWNlZY8q8Qp1V5etdtAQ2+CnSs+tkVFLVpL2O2M6qKlnfbIW+77hB8mytasG8q9xgujGmdIrJWVXqsssSICvQdiyZk4YpeT0bwdNhZWfX7HM1PKzsrDFlVzRdVQuANiUViCmdftQebg8rO/uBlZ01pkyLJnHMAf4jImNE5DYRuTX7VpiNRWSwiKwUkTQReSDC8uYiMktEFovIHBFpFlh2s4is9rebA+3dRGSJ3+dTvkvNxNk/+kJPKztrTLkRTQXA2XksUlUNnzU3fNtU3JlYg4B0XA3ykaq6PLDOm8D7qvpvPwvvLap6o4jUA+YB3QEF5gPdVHW3iHwD3A18jat5/pQ/RRiwMY54Sj8AXd90kyBmq10J5l4JreskLCxjTBHE7MpxVR2Qxy3fpOH1BNJUda0vPTsRGBq2TnvcWVoAswPLLwE+UtVdqrob+AgYLCJNgFqq+pUfg3kVGFbY12Niq1kNeONiSA38ie09BsNnwAG7RNSYMiWaKUdS8roVYvOmwMbA43TfFrQIGO7vXwHUFJH6+Wzb1N/Pb58mjvo3hces7KwxZV40YxyZuOlFIt1i4V6gn4gsAPoBGcCJGO3bxMnPO8KIsNnLJqXB363srDFlRjSJoyVwRuB2PvAecEchts0ATgs8bubbTlLVTao6XFW7AA/5tj35bJvh7+e5TxN/IvBCf+gQVnb2/i9htv12jCkTohnjWB92+wq4GTdTbkHmAq1FpKW/cHAEMDW4gog0CHR7PQi85O/PAC4WkboiUhdXTGqGr3++T0R6+7OpbgKsqFQSyC47Wyes7Oy1VnbWmDIhmiOOSGoBDQtayV9tfhcuCawA3lDVZSIyVkQu96v1B1aKyCqgETDOb7sLeBSXfOYCY30bwE+BF4A0YA1w8owqk1itfNnZ4OkYVnbWmLIhmtNxx+NOh81WDbgAmKSqPyuB2IrNTsdNvLHzYPTc0LYftYPn+yckHGNMIRR0Om6F8IZ8pIU9Pgg8q6ozixaaKQ9+283VKA+WnX1hhbtg0MrOGlM6Wc1xU+L2HIUeb0FazpcYKqXAp8Ogl5WdNSbpxLJ07FMicl5Y23ki8kRxAjRlX53KMOUSqB5WdvaH0+Dvi2CfVXQxplSJZoxjO9DUX/md3VYZ2Kiqp+S9ZeLYEUdyeSPNlZkNV7OiG/e4uwO0qBX/uIwxoWJ2xIEbGA9fPzXKfZhy7Joz4d5Oudv3H3cXCLZ6Ha6eAV9ssSvNjUlm0Xzofwb8PvtaC/9zjG83plD+2Bse6AKVU3Mvy1KYvBbOnwK933ZXnGdm5V7PGJNY0XRVNQPeB5oA64HTgc3AD1U1Pb9tE8W6qpLXtkPw7HJ4eilsO5z3eqfVcF1YP2pnVQWNiZeCuqqiOqvKH2X0xE0BshH4RlWT9juhJY7kd/QETFgNf1sES3blvV71CnCbHwdpZb9KY0pUzBKHiHQGdqrqxkDbaUA9VV1U7EhLgCWO0kMVZmW4s6ymbch7PQGGtYRfdIQ+TdzcWMaY2Ipl4lgKXK6qawNtrYApqtqx+KHGniWO0um73fDEYnh1FRzOZ3qSbg3hlx3h6lZQMcKYiTGmaGKZOPapaq6TJfNqTwaWOEq3HYfhf5fDP5fClkN5r9e0OvysA9zRHuraOIgxxRbL03HTRaRrsME/3lTk6IzJR4Oq8FA3WHcD/PtC6Nwg8noZB+GBr6DZq3DXZ7B6T1zDNKbcieaI43bgEeAvuJloW+GKL41T1edKLMJisCOOskUVPtnkBtLfXx8642aQAD9oDr/sBP1OtXEQY6IV67OqrgZuw51VtQF4UVUnFz/MkmGJo+xatQeeXAyvrIRD+YyDdG7gxkGuPRMq2TiIMYUS68TRCOgFhHQaqOpLkbdILEscZd+uI/DccvjHUth0MO/1mlSDu86BH58N9avELz5jSqNYDo4PA8bjplc/G1gGnAP8V1UHxCDWmLPEUX4cOwFvrnFTl8zfnvd6VSvAzW1cbfS2deMXnzGlSSwHx38P3Oprgh/0P+8A5hdmYxEZLCIrRSRNRB6IsPx0EZktIgtEZLGIDPHt14vIwsAty19TgojM8fvMXpaUky2aklcpFa5vA3OvhE+Hums9Ig1tHM50V6yfNRF+MA1mpdu8WMZEq0in44rIblWt668k31LQ7LgikgqsAgYB6bgSsCNVdXlgneeABar6jIi0B6apaouw/XQA3lHVVv7xHOBeVZ0X6XntiKN8W7MXnlwCL62Ag/mMg3Ss7y4oHNk68hxaxpQ3sTzi2ObHOADWici5uDOrCvOv1hNIU9W1flr2icDQsHUUV8McoDaRT/Md6bc1pkCtasNTfSD9JnjsXDfvVSSLd8Its6H5eHh0HmzPZ+4sY0x0ieN5oI+//3dgNrAI+Fchtm2Km9sqW7pvCxoD3CAi6cA0IFId82uBCWFtL/tuqodF7MRLk1udynBvZ1hzHUy4yJWtjWTrYXhkLpw+Hu6YA8vzmTvLmPKs0IlDVf+sqm/5+68CbYBuqvpwjGIZCbyiqs2AIcD47CncAUSkF3BIVZcGtrleVTsAff3txhjFYsqgiqkwojV8NRw+vwKuPANSInzVOHICnl8BZ0+Cwe/DhxttHMSYoCIXYVLVDaq6opCrZ+Cu/cjWzLcF3Qa84ff9JVCF0NN+RxB2tKGqGf7nfuB1XJeYMfkSgfMaw+RLIO06d4ZVzYqR152xES55Hzq+ARNXw4mknQvamPiJV/W+uUBrEWkpIpVwSWBq2DobgIEAItIOlzi2+8cpwDUExjdEpIKINPD3KwI/AJZiTBRa1oK/nw8bb4S/ngfNa0Zeb+kuGDnTJZA317iiU8aUV1FdAFisJ3Kn1z6BG0x/SVXHichYYJ6qTvVnUj0P1MANlN+vqh/6bfsDf1LV3oH9VQc+BSr6fc4EfqmqJ7LXsbOqTLQys+Cd7920Jl9uzXu9DvVgTA932m+k7i5jSrOYXjle2ljiMMXx1RZ3QeHktXkfYXRuAL/rDj9sYXNimbLDEodnicMU1eo98Oh8eG113gmkW0P4XQ8YcrolEFP6WeLwLHGY4lq5G8bOd6Vu8/qv6XmKSyCXnGYJxJReljg8SxwmVpbvcglkUlre65zbCMb2hIFNLYGY0scSh2eJw8Takp3wu3nw1tq81+nT2CWQAeGXuxqTxCxxeJY4TElZuMMlkHe+z3ud/qe6LqwLTo1fXMYUlSUOzxKHKWnzt8OYua46YV4GNnVHIOc1jl9cxkTLEodnicPEyzdbYcw8mL4h73UuOc0dgfRqlPc6xiSKJQ7PEoeJty+3uATy4ca81xlyuksg3a2SjEkiljg8SxwmUf67GUbPhY/DZ2cL+GFzl0C6NIxfXMbkxRKHZ4nDJNonm+CRb+DTzXmvc0VLN5VJx/rxi8uYcJY4PEscJhmowuwMV/fj8y15r3fVGTC6O5xjCcQkgCUOzxKHSSaqMDPdJZCv8phMUYBrznQJpF3duIZnyjlLHJ4lDpOMVOE/G2D0PJi7LfI6AlzXGh7pDm3qxDM6U15Z4vAscZhkpgofrHeD6N/uiLxOisANreHh7nCm/TmbEmSJw7PEYUoDVZi6ziWQRTsjr5MqcFNb+G03OKNWXMMz5YQlDs8ShylNshSmrHXXgSzdFXmdCilwS1t4qFvelQuNKQpLHJ4lDlMaZSlMXuMSyIrdkdepmAK3nQW/6Qan1YhvfKZsKihxxKvmOCIyWERWikiaiDwQYfnpIjJbRBaIyGJfahYRaSEih0Vkob89G9imm4gs8ft8SsQmsDZlS4q4M6uWXAOvXwRtInz/OZ4Fzy6HM1+Duz6DjAPxj9OUL3E54hCRVGAVMAhIB+YCI1V1eWCd54AFqvqMrz8+TVVbiEgL4H1VPSfCfr8B7ga+BqYBT6nq9OzldsRhyprMLFdIaux8SNsbeZ3KqXBPBzeZYuXU+MZnyoZkOeLoCaSp6lpVPQZMBIaGraNA9lBfbWBTfjsUkSZALVX9Sl32exUYFtOojUkyFVLgxrawYgS8PABaRhjbOHoC/rIQer/tqhYaE2sV4vQ8TYHgVG/pQK+wdcYAH4rIz4DqwEWBZS1FZAGwD/itqn7m95ketk8rl1MabJsP66fBiWOJjqTwqjaEU/tC/Y6Qkviv8RVSYNRZcH1reHWVq4m+fn/oOgt3QNfJ8M++MKqtVSI0sROvxFEYI4FXVPWvInIuMF5EzgE2A6er6k4R6Qa8IyJnJzRSU3SrJsBHN4BmJTqSoqlcF069AJoNgKb9oX4HkLgNFeZSMRVuawc3toGXv3MJJONgzvJDmXDrbDdD77MXQO3KCQvVlCHxShwZwGmBx818W9BtwGAAVf1SRKoADVR1G3DUt88XkTVAG799swL2aZLJ6kmlO2kAHN0N37/rbgBV6sOp/VwSaTYA6p2dkK/2lVLhx2fDiDPhx5/mroc+MQ2+3goTBlkNEFN88fqqNBdoLSItRaQSMAKYGrbOBmAggIi0A6oA20WkoR9cR0TOAFoDa1V1M7BPRHr7s6luAt6Nz8sxUUt7Ez68vnQnjUiO7IS1b8Nnd8OEDvBSI5h+NSz5F+xa4a7oi6PalWHCRfBif6gW9rXw+/3Q5x3407fuNF9jiipu13H402ufAFKBl1R1nIiMBeap6lR/JtXzQA3cQPn9qvqhiFwJjAWOA1nAaFV9z++zO/AKUBWYDvxMAy/IzqpKEmlvwYxrQU/ktEkKdPoFVC4Fv5esE7BzMWTMcUcc0ajWyB2NNO0PTQdAnTZxOyJZsRtGfhT5CvSBTWH8QGhSPS6hmFLGLgD0LHEkyJopMOMayMrMaZMUuGg8tL0ucXEVhWbBDp9AMmZDxidwLI9zYvNSrUlOt1bTAVC7VYkmkiOZ8Ouv4KkluZc1qAL/vhCGNC+xpzellCUOzxJHAqx9F/5zVWjSQGDQq9D2hoSFFTNZJ2DHIpdE0mfDpk/h+P6Ctwuq0SznaKRpf6jVskQSyXvr4JbZsPNI7mW/6Ah/7G3XfJgcljg8Sxxx9v17MP1KyDoeaBS46BU466ZERVWysjJh+wJ/NDIHNn0Gx6O8jLvm6TlJpOkAqBW7w4GMA3DDLJgT4QqpLg1g4iCbtt04ljg8SxxxtO4DmDYcsoLXaQgMfAnajUpUVPF34jhsn++SSPps2PxfyDwU3T5qtQw9Iql5WkFb5B9SFvxpgZt990TYv371CvB0Xzfzrl3zUb5Z4vAsccTJ+unwwbCwpAFc+CK0vzUhISWNE8dg27ycrq3Nn8OJCH1H+andKvSIpMapRQrliy1w3czcFw2CKxr1zAVQq1KRdm3KAEscniWOOFg/A6YNhRNHQ9sHPAdn356YmJLZiaOw9RuXRDLmwJYvcr93BanTJvSIpHrjQm+65yjc8Qm8uSb3sjNqudN6e9o1H+WSJQ7PEkcJ2/ARfPDD3B98/Z+Fc36cmJhKm8wjsOWrnLO2tnyV+8itIHXPckmk2QB3YWK1U/JdXRVeWAH3fA6HM0OXVUiBcT3h3s5ull5Tflji8CxxlKCNs+D9H+Tudun3NHT4aWJiKgsyD8OWL3OOSLZ+HXayQSHUOzvQtdUPqjaIuNryXTDiI1gSoWjUoGbw6kBoXC3qV2BKKUscniWOEpI+G96/zH3IBV3wD+h4V2JiKquOH4TNX+SctbVtbtipzoVQv2POdSSnXgBV6p1cdCQT7vsS/rk092anVHXXfAw+vVivwJQSljg8SxwlIOMTeO/S3Emj75PQ6e7ExFSeHDvgztTK7traNj/06vwCCTToFOja6guV6/Du925ixF0Rhlt+1Qn+0MvNjWXKLkscniWOGMv41CeNsNNL+/wNOv8iMTGVd8f2wab/5py1tWNBdHODSQo06ALNBrCjXn9u+q4v07fUyrVat4Zu4Lx1ndiFbpKLJQ7PEkcMbfovvDfYdZ0Enf84dPlVYmIyuR3d4y5CzD4i2b4QNw1c4aiksKlaN147NICPZQD/pQ8HxRU1r1ER/tXXFZUyZY8lDs8SR4xs/gKmXpL7iujz/gxd709MTKZwjuzyicQfkexcHNXmmaQylx7MlgHMlgF8wXkMb1udf/WFmnbNR5liicOzxBEDW76Cdy/OPR/TuX+Ebg8kJiZTdId3wqZPcq5s3xVhVDwfx6jIN/RkcZX+DOw1gLbtzoMKVUsmVhNXljg8SxzFtPUbeHeQ60cP6v176P5QYmIysXVoW2gi2b0iqs0zpRKpjXsh2TP/Nu4NFaqUTKymRFni8CxxFMPWefDuRbmnEO/5O+j5SGJiMiXv4BaXSLKvI9mzMrrtUytD43NzriNp3Mu1maRnicOzxFFE2+a7pHF0T2h7j9HQa0wiIjKJcmCTH2ifQ1b6bFL2pRW4SYgKVaHxeTnXkZzSA1JtcCQZJU3iEJHBwJO4CoAvqOqfwpafDvwbqOPXeUBVp4nIIOBPQCXgGHCfqn7st5kDNAGyLyS42NcoByxxFNv2BfDOwNxV77r/FnqNtSlUy7sD6cxbMIfvls7mvBOzOYPvo9u+QjVocj7UO8edCmxiq1JN6Dm6SJsmReLwNcNXAYOAdFwN8pGqujywznPAAlV9xpeRnaaqLUSkC7BVVTeJyDnADFVt6reZA9yrqvMiPa8ljmLYvtAnjbA5KLr9xo1rWNIw3ob9rs7H+k3r6a9zGMBsBuhsmrMh0aGVb9Uawa1birRpQYkjXmm+J5CmqmtV9RgwERgato4C2Vcb1QY2AajqAlXNLj2zDKgqItZRWpJ2LPbdU2FJo+uvLWmYXE6vCR9fDrf2aM7/pd7MLSmv0CJ1PS1T1nKrvMiElBs5WLlZosM0MRSvxNEU2Bh4nO7bgsYAN4hIOjAN+FmE/VwJfKuqwckQXhaRhSLysIh9ohXbzqXuSOPIztD2Lve6027tLTYRVEiB0T1g9uXQrLprWycteTnlVq6TV6lxfAP3tUrjyAXPQ5vrXO11U2pVSHQAASOBV1T1ryJyLjBeRM5RdXMmiMjZwJ+BiwPbXK+qGSJSE3gLuBF4Nd6Blxk7l8E7F8KRHaHtnX8J5/3FkoYp0AWnwqJr4EdzYEpwyEOEx9e14t3drZg46Ed0HaSwZ7Wba+tIhCl5TfFVLLnpjOOVODKAYM3LZr4t6DZgMICqfikiVYAGwDYRaQZMAW5S1ZNlZ1Q1w//cLyKv47rELHEUxa7lLmkc3h7a3ukeN5WIJQ1TSPWqwFuXwLPL4BdfwNHAvIur90Lvt+HPvYV7OrYhpW6bxAVqiixeXVVzgdYi0lJEKgEjgKlh62wABgKISDugCrBdROoAH+DOsvo8e2URqSAiDfz9isAPgOgufTXO7u980tgW2t7xZ9Dn75Y0TNRE4M5zYO6V0L5u6LLjWfDLL+AH02BblCXYTXKIS+JQ1UzgLmAGsAJ4Q1WXichYEbncr/Yr4HYRWQRMAEapO+XrLuBM4BE/lrFQRE4BKgMzRGQxsBB3BPN8PF5PmbJ7JUwZAIe2hrZ3+B83PbolDVMMHeq75PHj9rmXTd8And6EDze6SoSm9LALAMuzPavh7X5waHNo+zk/gX7/sqRhYuqtNW7sY0+Earj1KkPnBu7Wxd/a1nGD7ib+kuI6jkSxxJGPPWkwpT8cDBtqOvsO6P+MXZBlSsT6/XD9TPi8EJcXVEl1RyxdGkBn/7NjfahWseTjLO8scXiWOAL2rnFJ40B6aHv7H8GA/7WkYUpUZhaMnQe/nx9NdRAnRaBNbZ9MAkcnDWxS3piyxOFZ4vD2fe+6pw5sDG1vdwtc+IIlDRM3n22C338LX2yBA8eLt6+m1XOSSHZCaVHTeluLyhKHZ4kD2LfOHWnsXx/aftbNMPAlSxomIbIU1uyFhTthwXZYsMPd31LMM65qV8pJIp3rQ5eG0K4OVLR66QWyxOGV+8Sxb71PGutC29veCANfhhT7bzLJZcshWLjDJZIFO9z91XsL3i4/lVLgnHr+6KShSyidGrhSuCaHJQ6vXCeO/RthSj/XTRXU5jq46FVLGqbU2H8MFu0MTShLd7lrQ4pKgNa1A0cn/mejkrvwOulZ4vDKbeI4kO7GNPatDW1vPQIGjYeUZJp1xpjoHTsBK3bnHJVkd3Xti3DabzSaVAsdgO/cAM6o5QboyzpLHF6REkfWCTe6Vlr7/g9kuO6pvWEFd868Bi5+zZKGKbNU4fv9bsxk4c6co5NNB4u332oV3GnCpcEpVWHFyKJtW1DisE+O/Gz8CKYNg5otoFYL/7Nl6P2qDZPz1I0Dm9w0IuFJo9VVljRMmSfijg7OqAVXtspp33YoJ5FkH52s2lP404IPZbpbaVCxBL/v2qdHfvavgxNHXa3lvOotV6iak1hqtQwkGZ9gqtSPf2I5uNkljT2rQtvPGA4Xv25Jw5Rbp1SDi6vBxYEpVw8chyVhyWTJrtDJGU0o+wTJz751Ba+TeRh2r3C3SCrWyP+IpXKd2CaWQ1t90ghLdGcMg0smQKqdPmJMUI2KcG5jd8t2/ASs3BN6RteCHZGnSymPLHHk51DRyi6GOH4Adi11t0gq1YqcULLvV45ibObQNpc0dn8X2t7ycrhkEqRWKtJLMKa8qZgK59R3txvbujZVlziySsmwcEn2c9jgeEGO7nFHHvvXudNZQ+5/7xJDSapcNzShhHeLVarh1ju83c1yu2tZ6PbNL4Mhb0GqVds1xhSOnVXllcjpuKpwdHeEhBK4n1nCBQeq1HdJ5MjO3NdpNB8CQ962pGGMiYqdVVWSRKBKPXc7pVvu5aquDOu+de5Dff+63PdPHCleDEd25q4PDnD6YLjUjjSMMbFniaMkibjTdas2hEY9ci9XdYPZEbvB1rk5pbKKMBp32sUwZApUqFK8+I0xJoK4JQ4RGQw8CaQCL6jqn8KWnw78G6jj13lAVaf5ZQ/iapKfAO5W1RmF2WfSE4Hqjd2tce/cyzXLnVob6Uhl3/dwYANkhZ1UftrFcNk7ljSMMSUmLmMcIpIKrAIGAem4GuQjVXV5YJ3ngAWq+oyItAemqWoLf38C0BM4FZgJZFe4z3efZX7KkawTcHCTTyjr3am9zYfY3FPGmGJJljGOnkCaqq4FEJGJwFBgeWAdBWr5+7WBTf7+UGCiqh4FvheRNL8/CrHPsi0lFWqe5m7GGBMn8ZqEqSkQrByU7tuCxgA3iEg6MA34WQHbFmafxhhjYiyZZu8bCbyiqs2AIcB4kdI6u6AxxpRd8eqqygCC/SnNfFvQbcBgAFX9UkSqAA0K2LagfRpjjImxeH2jnwu0FpGWIlIJGAFMDVtnAzAQQETaAVWA7X69ESJSWURaAq2Bbwq5T2OMMTEWlyMOVc0UkbuAGbhTZ19S1WUiMhaYp6pTgV8Bz4vIL3AD5aPUnfK1TETewA16ZwL/o6onACLtMx6vxxhjyrNyM+WIMcaY6EU6HdcGn40xxkTFEocxxpiolOmuKmOMMbFnRxzGGGOiYokjAhE5TURmi8hyEVkmIvckOqZwIvKSiGwTkaWBtnoi8pGIrPY/6yYyxmwiUkVEvhGRRf79/J1vbykiX4tImohM8qdVJ5yIrBORJSKyUETm+bakeW+j+d2L85R/jxeLSNc4xxrxfymJ443qb9VfJjDJt38tIi3iGGtb/zeafdsnIj+Py3urqnYLuwFNgK7+fk3cZIrtEx1XWIwXAF2BpYG2v+BmFQZ4APhzouP0sQhQw9+vCHwN9AbeAEb49meBOxMdq49lHdAgrC1p3ttofve4WRim+99Bb+DrOMca8X8pieON6m8V+CnwrL8/ApiUoL+JVGAL0Dwe721C/vBL2w14FxiU6DgixNUi7MNjJdDE328CrEx0jBFirgZ8C/QCdgAVfPu5wIxEx+djiZQ4kuq9LezvHvhf3KzRudZLUNzv4ma0Tvp4C/O3iruO7Fx/v4JfTxIQ68XA5/H6W7CuqgL4Q88uuG8eya6Rqm7297cAjRIZTJCIpIrIQmAb8BGwBtijqtkFRZJpkkoFPhSR+SJyh29L2vfWyyu+pJkMNOx/KWnjjfJv9WS8fvleoH484/VG4MpPQBzeW6sAmA8RqQG8BfxcVfclOp5oqKqKSNKcMqfuav/OIlIHmAKcldiI8tVHVTNE5BTgIxH5Lrgw2d7bcMkYX/j/kkjONWXJFm8p+1vFj7dcDjwYvqyk3ls74siDiFTE/aG/pqpvJzqeQtoqIk0A/M9tCY4nF1XdA8zGHe7XEZHsLy9JM0mlqmb4n9twHxw9Sf73Nq/4CjPBaInK438paePNVsi/1ZPx+uW1gZ3xjZRLgW9Vdat/XOLvrSWOCMR9HXoRWKGqf0t0PFGYCtzs79+M609OOBFp6L+9ISJVcX3cK3D/lFf51ZIiXhGpLiI1s+/j+o6XkqTvbUBe8U0FbvJn1PQG9ga6MUpcPv9LyRpvtH+rwddxFfCx+gGEOBpJTjdVeEwl894mYtAp2W9AH1w/92Jgob8NSXRcYTFOADYDx3F9lbfh+lZnAatxJXbrJTpOH2tHYIF/P5cCj/j2M3AzHacBbwKVkyDWM4BF/rYMeMi3J817G83vHncGzdO4fvolQPc4xxrxfymJ443qbxU3i/ebvv0b4Iw4x1sdd4RTO9BW4u+tXTlujDEmKtZVZYwxJiqWOIwxxkTFEocxxpioWOIwxhgTFUscxhhjomKJw5QpInK6iBwQkdREx1Jc4mbpvSjRcQSJyBwR+ZG/f72IfJjomEz8WeIwZYqqblDVGuqmjTAlSFVfU9WLsx+LiIrImYmMycSHJQ5jjDFRscRhSgXfbXOfL0BzUEReFJFGIjJdRPaLyEwRqSsiLfw33wp+uzki8qiIfO7X+1BEGhTwXFVE5P9EZKeI7BGRuSLSyC+7RURW+H2tFZEfB7brLyLpInK/uEJLm0VkmIgMEZFVIrJLRH4TWH+MiEwWVwhov4h8KyKd8ogpRUQeEJE1Pq43RKReQfHm8xpH+fj3i8j3InJ9oP1zEfmniOwVke9EZGA++/ivv/+pb17kuwqvze/5TelmicOUJlfi5g5qA/wQV5TmN0BD3N/y3Xlsdx1wC3AKUAm4t4DnuRk3Wd1puOkbfgIc9su2AT8Aavl9/l1CK6k1xk1D0RR4BHgeuAHoBvQFHhaRloH1h+KmrKgHvA684ycFDPczYBjQDzgV2I2bPqKgeHPxc3A9BVyqqjWB83BTgWTrhZuWogEwGng7O0nlRVUv8Hc7+a7CSfmtb0o3SxymNPmHqm5VN3vtZ7gKZgtU9QhuFtsueWz3sqquUtXDuEpunQt4nuO4D+AzVfWEqs5XP62+qn6gqmvU+QT4EJcQgtuOU9XjwETch++TqrpfVZcBy4HgUcV8VZ3s1/8bLun0jhDTT3DzZqWr6lFgDHCVP7LKM958ZAHniEhVVd3sY8u2DXhCVY/7BLASuKyA/ZlyxBKHKU22Bu4fjvC4Rh7bbQncP5TPetnG4yq7TRSRTSLyl+yjABG5VES+8t1Oe3AT9gW7vnYGBuazv/XnF+fJwjqqmoWbtPDUCDE1B6b4rqg9uBlbT+CK9OQZbySqehC4FpeMNovIByISrDmRoaGT2K3PIyZTTlniMCaM/6b9O1Vtj+vG+QFuOurKuLoSj+OqrNUBpuFmHS2qk/URRCQFVyNhU4T1NuK6luoEblVUNSOveAt4jTNUdRCutOh3uC61bE39dOjZTs8jJlNOWeIwJoyIDBCRDv5akH24rqAs3PhIZWA7kCkil+LqdRRHNxEZ7rucfg4cBb6KsN6zwDgRae5jbCgiQwuIN6/X10hEhvqxjqPAgbD1TwHuFpGKInI10A6XIAuyFTf9uCnjLHEYk1tjYDLuQ3gF8AkwXlX34wbg38ANTl+HK45THO/iuo12AzcCw/14R7gn/XN9KCL7ccmlV37x5vOcKcAvcUcRu3AD7ncGln8NtAZ2AOOAq1S1MFXtxgD/9t1p1xRifVNKWT0OYxJERMbgBrRvSHQs2URkFPAjVe2T6FhM8rIjDmOMMVGxxGHKJXHzLB2IcFtW8NalQx6v74CI9C14a2PyZl1VxhhjomJHHMYYY6JiicMYY0xULHEYY4yJiiUOY4wxUbHEYYwxJiqWOIwxxkTl/wGNi9SOD0jQowAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracy_list_train = []\n",
    "accuracy_list_val = []\n",
    "for min_samples_split in min_samples_split_list:\n",
    "    # You can fit the model at the same time you define it, because the fit function returns the fitted estimator.\n",
    "    model = DecisionTreeClassifier(min_samples_split = min_samples_split,\n",
    "                                   random_state = RANDOM_STATE).fit(X_train,y_train) \n",
    "    predictions_train = model.predict(X_train) ## The predicted values for the train dataset\n",
    "    predictions_val = model.predict(X_val) ## The predicted values for the test dataset\n",
    "    accuracy_train = accuracy_score(predictions_train,y_train)\n",
    "    accuracy_val = accuracy_score(predictions_val,y_val)\n",
    "    accuracy_list_train.append(accuracy_train)\n",
    "    accuracy_list_val.append(accuracy_val)\n",
    "\n",
    "plt.title('Train x Validation metrics')\n",
    "plt.xlabel('min_samples_split')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xticks(ticks = range(len(min_samples_split_list )),labels=min_samples_split_list)\n",
    "plt.plot(accuracy_list_train)\n",
    "plt.plot(accuracy_list_val)\n",
    "plt.legend(['Train','Validation'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how increasing the the number of `min_samples_split` reduces overfitting.\n",
    "- Increasing `min_samples_split` from 10 to 30, and from 30 to 50, even though it does not improve the validation accuracy, it brings the training accuracy closer to it, showing a reduction in overfitting.\n",
    "\n",
    "Let's do the same experiment with `max_depth`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f45b7a06110>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEXCAYAAAC6baP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA8f0lEQVR4nO3dd5xU1f3/8dd7F5bO0hUBAZEiWLBgl6KoWCKCDWygxhgTWyyJJsbuz+QbUzQajRU1Kio2LAQFRTQ2kCogiIoUAVGkl2XZz++Pe3f3zuzA7sDu3C2f5+MxD+aee+6dzyyz+5l7zj3nyMxwzjnnyior7gCcc85VLZ44nHPOpcUTh3POubR44nDOOZcWTxzOOefS4onDOedcWjxxuIyR1FeSSWobdyw7StIESY9EtkdIGlfKMcMl5ZfT65ukc8vjXFWBpA7hez4y7lhcMU8croTwF3V7jwU7eOoPgdbAd+UXbekkDQrj7r6N/f+S9K2kHfl9uBI4Y+ciTBnTOEkjUuxqDYwq79fLpO28t1QWEbznTyouIpcuTxwuldaRx2lh2QGRsl7RypJyynJSM8szs2VmVlCOsZbFa8BS4OLkHZLqA2cDj+5IXGa22sx+2vkQy/x6y8xsU6ZeL06Scsxsa/iet8QdjyvmicOVEP6iLjOzZcDKsHhFpOx7SVdIekbSauApAEl3SpojaYOkRZIelJRbeN7kpqrI9rGSJobHzZZ0wrZiU+ANSZMk1Q7LssJvsRMlZad4P/nAY8B5kuok7T4TaAg8KukASWMkfS9pXfgaA7b3s0puqgpjuT1yjueApknHdJT0kqTvwvc8U9J50XMCxwDDIld5fcN9CU1VklpLGilplaSNYVPaQSl+5mX+GYfH3SJpvqQzJX0ZHveKpMaSBkuaK2mtpFHR/+Pw2CGSpknaJGmBpL9JarC99xZpkjpH0puS1gO3p2qqktRK0uOSloevMVfSheG+2uHrLZa0WdJSSSO3917dDjAzf/hjmw+gL2BA20iZAT8ClwGdgM5h+Y3AUUAHgj8OXwBPbOtcke3pwACgM/A4sAZoup2YWhI0d90dbv8hjKfddo7pAGwFhiaV/w94NRLPcKAH0AW4A8gDukTqTwAeiWyPAMZFtq8E1gPDwnP8FlgF5Efq7BP+7PYLf36XA/lAv3B/LjAReA7YNXzkRH7254bPRdCEMw04Mjzvc8BPQIud/BnfEr6PN4B9gT7ACuAt4M0w9iOB5cCfI8cND1//PGAPoDcwA3hqe+8t/P8xYDFwDtAxfBSWHxkeXw+YA0wB+oevcRwwJNx/dXiOvsDuBFfHV8X9e1TdHrEH4I/K/WDbiePRMhw7CNgMZKU6V2R7cOSYXcKy40s5d7/wj+3NwBbg1DLEMwZ4J7K9V/haJ23nmOnAHyLbE9h+4lgM3Jl0jlFEEsc2XudV4OHI9jhgRIp60cRxTLjdPbK/DkGz3E078zMmSBz5hAkoLLufIPm2jJTdA0yObC8Afpl0rt7h6zXd1nuLJIg/bqO8MHFcBGyKfh6T6t8DvAMo7t+d6vzwpiq3oz5NLgibMCaGTTDrgKcJvk3uWsq5phU+MbPlBH+cdtneAWb2LvBXgj9wj5jZK2WI+SGgr6Q9w+2LgYUECQVJLRV0lH8RNv2sI7j6aF+GcyOpMdCG4CaAqA+S6tWX9CdJsyStDF/nxLK+TkQP4Eczm11YYGabCa5CeiTVnRapU6afMbDEzH6IbC8DlpnZiqSyVhD8/ML38LewmW5d+N7GhHX3pHQlPldJDgRmm9nibex/nODKa37YVHqaytgH58rOE4fbUeujG5IOAV4gaIYYRNCZ/stwd2m/uHkpyrb72Qz7Mo4g+APYSZLKEPNrBH/oLg7/mJxPYqf4CIKmtt+G//Yk+INb3n94/gKcC9xKcOXUk6D5pyL/wKX9Mya4kouybZQVnqfw3ysJ3lPhYz+CJrKZZYhzfelVts3MphE0cV1L8J7vAaaFSd2VE08crrwcCfxgZjea2SdmNg+oyPEatxB8gz0COJjgj/12WXEn+XCCTvEmwKORKr2Bf5nZaDObSdDks0dZAzKzNcAS4PCkXUckbfcGnjaz581sOvA1QX9IVB5QoqM/ySyguSK3GYed/4cAn5c17vISXsksArqa2fwUj8K7wcry3rblM6C7tjMWyMzWmdnLZnYFcBBBk2SfHXw9l4InDlde5gItJV0kaQ9J5wO/qogXktQHuB4YZmafAL8guAPn4DIc/gjQArgXeMPMlkT2zQXOkbSPpJ7As6T/B+6vwJWSzpPUWdI1BJ24UXOBgZIODv/oPwTsllTnG+BASZ0ktVB4B1mSdwiadp6RdISkvYEngbrAA2nGXV7+AFwh6Q+S9pbUVdKpkv4dqVOW97YtzwLfAqMl9Q/vUDtG0lkAkq4L78zqIakjcCHBVem8cnp/Dk8crpyY2evAncD/I2iSGAJcV96vI6kZ8B/gHjMbG7728wTNTM9KalRKnAsI7gxqSvAHO+oCgt+JT4FXgP8Ck9IM8R6CpPR3gmauw4Dbkur8huCP37vAeIKrlORBfX8FfiDonF9ByasWLOgNPpXg7rU3wlh3BY5N6pvIGDN7iuBq7mSCn+MkgqvDaIIu9b1t5/wbCK4ePgdGEtxhdT/B3VYQ3C12NfARwedwEHCamc3d0ffkSlLw2XPOOefKxq84nHPOpcUTh3POubR44nDOOZcWTxzOOefSUivuACrS6tWrveffOed2Qm5ubonBtX7F4ZxzLi2eOJxzzqWlWjdVReXm5pZeyTnnHKtXr97ufr/icM45lxZPHM4559JSY5qqosyMlStXUlCQ6aWvq6+srCyaNWtG2WY3d85VZRlLHJIeI5j47Hsz2zvFfhFMEHcisAEYbmZTwn3DCJYlBbjDzJ4Iyw8kmNyuHsF6BldaGSbfWrlyJQ0aNKBu3bo7/b5cYNOmTaxcuZLmzZvHHYpzroJl8opjBHAfwbTPqZxAsNhLZ4L1BB4ADglnQ72ZYF59Az6TNNrMfgrrXEyw4tmbBGsqj0lx7gQFBQWeNMpZ3bp1Wbt2bdxhuHLwxU/w3U4tp+Qqg5xsOLJ1xZw7Y4nDzCZK6rCdKgOBJ8Mrho8lNZHUmmDN5LfNbCWApLeBAZImAI3N7OOw/EmCKaZLTRzOuZKWroezx8GE7+KOxJWHXerBsuEVc+7K1DnehmD1sEKLw7LtlS9OUV7p/fjjj/Ts2ZOePXuy66670qZNm6LtvLxUK3wWmzx5MldccUWGInU1xaTvodeLnjRc2dTIzvG4NW/enGnTpgFwyy230LBhQ6699tqi/fn5+dSqlfq/5qCDDuKggw7KRJiuhnhmHlw0ATZtjTsSV1VUpsSxBGgX2W4bli0haK6Klk8Iy9umqJ82lfMim3Zp+scMHz6cunXrMnXqVI444giGDBnClVdeyaZNm6hXrx6PP/44Xbt2ZcKECdx99928/vrr3HLLLSxcuJCvv/6ahQsXctVVV/nViCuzrQXwh0/hz1NL7uvRFFrVK1nuqo6mdSru3JUpcYwGLpM0kqBzfLWZLZU0Fvh/kpqG9Y4DbjCzlZLWSDqUoHP8fOCfsUReThYvXsyHH35IdnY2a9as4f3336dWrVqMGzeO3//+97z44osljvniiy949913Wbt2LV27duXSSy+ldu10lnB2NdHqzXDOeHjj25L7rtkP/nwoZFemhmxXqWTydtxnCa4cWkhaTHCnVG0AM3uQ4K6oE4H5BLfjXhDuWynpdorXfr6tsKMc+BXFt+OOoYp3jJ9xxhlkZ2cDwZD/YcOG8eWXXyKJLVu2pDzmpJNOok6dOtSpU4dWrVqxfPly2rZtm7KucwBfroJTxsAXqxLLc7Lg4b5wftcYgnJVSibvqhpayn4Dfr2NfY8Bj6UonwyUGBNSVTVo0KDo+R//+Ef69evHyy+/zIIFC+jbt2/KY+rUKb4ezc7OJj8/v6LDdFXY24vgzLdgVdI9GLvWh5ePh0N3jScuV7VUpqaq2OxIn0RFW716NW3aBDeJjRgxIt5gXJVnBvfMgGs+goKkIbK9WgVJo03DeGJzVY+3YlZSv/3tb7nhhhvYf//9/SrC7ZTNW+HCd+E3H5ZMGud0hvcGetJw6VEZZuiosqIrAEanVV+xYgUtW7aMJabqzH+ulc+yDTD4v/DR8sRyAX86FK7rCT69mEsWnVY91QqA3lTlXDX12Qo4dQwsTpo+pHEOPNsfTmwfT1yu6vPE4Vw19OyXQfNU8qC+zrkw+gTo1jT1cc6VhScO56qRAoMbP4G7UgzqO64djDy2YgeGuZrBE4dz1cSaPDhnHLyeYlDf1eGgvlp+O4wrB544nKsG5q8OBvXN+SmxPCcLHuoDw7rFE5ernjxxOFfFjVscDOr7aXNiuQ/qcxXFL1xj0K9fP8aOHZtQ9o9//INLL009ErFv375MnjwZgBNPPJFVq1aVqHPLLbdw9913b/d1X3nlFWbPnl20fdNNNzFu3Lg0o3eVhRncOwMGvF4yaRzYEiad5knDVQxPHDEYOnQoI0eOTCgbOXIkQ4dud1YWAN58802aNGmyQ6+bnDhuu+02+vfvv0PncvHavBV+PgGu/B9sTRqKdXZneP9UaOuD+lwF8aYqgPvKeQTUZdsfVHn66adz4403kpeXR05ODgsWLOC7777j2Wef5eqrr2bjxo2cfvrp3HrrrSWO7dChA5MnT6ZFixbceeedPPHEE7Rq1Yp27dpx4IEHAvDwww/z0EMPkZeXx5577slTTz3FtGnTGD16NO+99x533HEHL774Irfffjsnn3wyp59+OuPHj+faa68lPz+fXr168cADD1CnTh06dOjAsGHDeO2119iyZQsvvPAC3bp5g3mclm2A08bCh8sSywXcdSj8tqcP6nMVy684YtCsWTMOPvhgxowJJvMdOXIkZ555JnfeeSeTJ09mxowZvPfee8yYMWOb5/jss88YOXIk06ZN480332TSpElF+wYPHsykSZOYPn06e+21F48++iiHH344p5xyCn/5y1+YNm0anTp1Kqq/adMmhg8fznPPPcfMmTPJz8/ngQeKFylp0aIFU6ZM4dJLLy21OcxVrCkroNeokkmjUe1gfMbv9vek4SqeJ46YRJurCpupnn/+eQ444AD2339/Zs2aldCslOz9999n0KBB1K9fn8aNG3PKKacU7fv888856qij2GeffXj66aeZNWvWdmOZO3cuHTt2pEuXLgAMGzaMiRMnFu0fPHgwAAceeCALFizY0bfsdtJz8+HIV0qOBN8zFz45DU7uEEdUribyxBGTgQMHMn78eKZMmcKGDRto1qwZd999N+PHj2fGjBmcdNJJbNq0aYfOPXz4cO677z5mzpzJzTffvMPnKVQ4dbtP2x6PAoM/fAJD3oaNST/+Y9vCp6fBXj4S3GWQ93FAqX0SFaFhw4b069ePCy+8kKFDh7JmzRoaNGhAbm4uy5cvZ8yYMdtcgwOgd+/eDB8+nBtuuIH8/Hxee+01LrnkEgDWrl1L69at2bJlC08//XTR9OyNGjVi7dq1Jc7VtWtXFixYwPz584v6RPr06VMh79ulZ00enDsOXksxqO+qfeEvh/mgPpd5njhiNHToUAYNGsTIkSPp1q0b+++/P926daNdu3YcccQR2z32gAMO4KyzzmK//fajVatW9OrVq2jf7bffziGHHELLli055JBDipLFkCFDuPjii7n33nsZNWpUUf26devy+OOPc8YZZxR1jv/yl7+smDftyuyrcFDf7BSD+h7sAxf4PQouJhmbVl3SAOAeIBt4xMz+lLS/PcEqfy2BlcC5ZrZYUj/g75Gq3YAhZvaKpBFAH6BwDuDhZjatsKJPq55Z/nMtP+MXwxkpBvXtUg9eGgCH+/gMV4EqxbTqkrKB+4FjgcXAJEmjzSza+3s38KSZPSHpaOAu4DwzexfoGZ6nGcGa5G9FjrvOzEbhXDVgBv+cCVd/WHJ8xoEt4eUB0M7HZ7iYZap19GBgvpl9bWZ5wEhgYFKd7sA74fN3U+wHOB0YY2YbKixS52KyeStcPCH1oL4he8LEgZ40XOWQqcTRBlgU2V4clkVNBwaHzwcBjSQ1T6ozBHg2qexOSTMk/V2STxjtqqTlG+CY0fDoF4nlAu46BJ7pD/VrxxKacyVUpvsxrgX6SJpK0G+xBChahkZSa2AfIDrJ0w0EfR69gGbA7zIWrXPlZMoK6PUi/C/FoL5XT4DrD/BBfa5yydRdVUuAdpHttmFZETP7jvCKQ1JD4DQzWxWpcibwspltiRyzNHy6WdLjBMmnVFlZWWzatIm6deum+z7cNmzatImsrMr0PaRqeH4+DH+35PiMTo2DkeDdm8UTl3Pbk6nEMQnoLKkjQcIYApwdrSCpBbDSzAoIriQeSzrH0LA8ekxrM1sqScCpwOdlCaZZs2asXLky5ZgGt2OysrJo1sz/ypVVgcFNn8KdU0ruO6YNPH8cNPPvNa6SykjiMLN8SZcRNDNlA4+Z2SxJtwGTzWw00Be4S5IBE4FfFx4vqQPBFct7Sad+WlJLgqbgaUCZBh9Ionnz5O4T5zLDDIa/A0/NK7nvin3gr4f7oD5XuWVsHEcctjWOw7k4jV0IA95ILKudBQ/2hgv3iicm56IqxTgO51yx+5IaVFvVg5eOhyNaxxOPc+nyxOFcBn2zBt5ImnfqlQFwmI8Ed1WIt6Q6l0EPzIJo4/CBLeHQXWILx7kd4onDuQzZmA+Pzkksu2xvH6Phqh5PHM5lyMj5sDIyaWGzOnDWnvHF49yO8sThXAaYwX0zE8su2gvqeS+jq4I8cTiXAZ8shyk/FG8LuLRHbOE4t1M8cTiXAcm34J7UHjo2jicW53aWJw7nKtjyDfDCV4lll+0dTyzOlQdPHM5VsEfmQF5B8faeuXBsu23Xd66y88ThXAXKL4AHZyWW/boHZPktuK4K88ThXAUavQAWry/erl8LhneLLRznyoUnDucq0P1JneLndoEmvk6lq+I8cThXQWavhHeWJJb92jvFXTXgicO5CvKvpL6No1rDvr4MjKsGPHE4VwHW5METcxPL/GrDVReeOJyrAE/Ng3Vbirdb14dBHeOLx7nylLHEIWmApLmS5ku6PsX+9pLGS5ohaYKktpF9WyVNCx+jI+UdJX0SnvM5STmZej/ObYtZyU7xX3SHnOx44nGuvGUkcUjKBu4HTgC6A0MldU+qdjfwpJntC9wG3BXZt9HMeoaPUyLlfwb+bmZ7Aj8BF1XYm3CujN5dAnN+Kt6ulRUkDueqi0xdcRwMzDezr80sDxgJDEyq0x14J3z+bor9CSQJOBoYFRY9AZxaXgE7t6OS56Ua3BF2axBPLM5VhEwljjbAosj24rAsajowOHw+CGgkqfAelLqSJkv6WNKpYVlzYJWZ5W/nnM5l1MK18OqCxDKfl8pVN5VpNYBrgfskDQcmAkuAreG+9ma2RNIewDuSZgKr4wnTuW3792woiKwNu08zOLJ1fPE4VxEylTiWANFp3dqGZUXM7DvCKw5JDYHTzGxVuG9J+O/XkiYA+wMvAk0k1QqvOkqc07lM2rwVHp6dWHbZPr40rKt+MtVUNQnoHN4FlQMMAUZHK0hqIakwnhuAx8LyppLqFNYBjgBmm5kR9IWcHh4zDHi1wt+Jc9vwwlewYlPxdm4OnNM5vnicqygZSRzhFcFlwFhgDvC8mc2SdJukwruk+gJzJc0DdgHuDMv3AiZLmk6QKP5kZoXf634HXC1pPkGfx6OZeD/OpZJ8C+4F3aBB7Xhica4iKfjiXj2tXr266M3l5ubGGYqr5j5bAQeNSiybNxQ6N4klHOd2yurVxV3Iubm5JRpbfeS4c+Ug+Wrj+HaeNFz15YnDuZ304yZ49svEMp+XylVnnjic20mPzYFNW4u3OzSCE3ePLx7nKponDud2wtaCktOnX9oDsv03y1Vj/vF2bieMWQgL1hZv182Gi/aKLx7nMsETh3M7IXleqiF7QvO68cTiXKZ44nBuB325CsYuSiy7bJ9YQnEuozxxOLeDkvs2DmkFB7aMJxbnMskTh3M7YP0WePyLxDK/2nA1hScO53bA01/C6rzi7ZZ14YxO8cXjXCZ54nAuTamWhr24O9TxpWFdDeGJw7k0fbAUZvxYvJ0luMSXhnU1iCcO59J0f1Kn+CkdYPdGsYTiXCw8cTiXhqXr4cWvE8t8aVhX03jicC4ND82G/ILi7W5N4Ghf6d7VMJ44nCujLVuDNcWjfr23Lw3rah5PHM6V0cvfwNINxdsNa8P5XeOLx7m4lDlxSHpZ0qmSdmgxTEkDJM2VNF/S9Sn2t5c0XtIMSRMktQ3Le0r6SNKscN9ZkWNGSPpG0rTw0XNHYnOuLJLnpTq/CzTOiScW5+KUzhXH+8BNwDJJD0g6vKwHSsoG7gdOALoDQyUl38B4N/Ckme0L3AbcFZZvAM43sx7AAOAfkppEjrvOzHqGj2lpvB/nymzGj/D+0sSyX3mnuKuhypw4zOxvZnYA0BtYBTwr6UtJN0kqbczswcB8M/vazPKAkcDApDrdgXfC5+8W7jezeWb2Zfj8O+B7wGcEchmVPOCv327Qo1k8sTgXt7T7OMxslpndAJxLcDVwMzBF0jhJ+23jsDZAdB7RxWFZ1HRgcPh8ENBIUvNoBUkHAznAV5HiO8MmrL9LqpPu+3GuNKs2w3/mJZb50rCuJksrcUjqKul2SV8BDwHPAR2AXYA3gVd2IpZrgT6SpgJ9gCVA0YKckloDTwEXmFnhDZE3AN2AXkAz4Hc78frOpTTiC9iQX7zdtgEM7BhfPM7FrVZZK0qaTJAkngPONrNPkqr8TdLl2zh8CdAust02LCsSNkMNDl+rIXCama0KtxsDbwB/MLOPI8cUtjpvlvQ4QfJxrtwUWMnp0y/pAbX8fkRXg5U5cQB/AkaHfRQpmdm2vodNAjpL6kiQMIYAZ0crSGoBrAyvJm4AHgvLc4CXCTrORyUd09rMlkoScCqQ1BLt3M55exF8ubp4u3YWXOxLw7oaLp3vTWsIrjiKhE1Xx5Z2oJnlA5cBY4E5wPNmNkvSbZJOCav1BeZKmkfQ9HVnWH4mQYf88BS33T4taSYwE2gB3JHG+3GuVMmd4md0gl3qxxOLc5WFzKxsFaUvgd6R5iEk7QZMMLMuFRTfTlm9enXRm8vNzY0zFFcFfbMGOj0N0d+QDwfBYbvGFpJzGbF6dfFldm5ubom5EdK54mgVTRqhpYD/Grlq6cFZiUlj/xZw6C6xheNcpZFO4vha0tFJZX2Bb8ovHOcqh4358MicxLLLfF4q54D0OsdvAV6S9CjBOIpOwAXhw7lq5bn5sHJz8XbTOjBkz/jica4ySWfk+KvAcUAD4KTw3+PDcueqDbOS81Jd1A3q79Asbc5VP+lccWBmnwKfVlAszlUKn34Pn60o3hZwqY8Ud65IWokjvA32KIJbX4tae83spvINy7n4JF9tnNge9mgcTyzOVUbpTKv+C+B/wNEEU3vsA1wDeMuvqza+3wDPz08s+3WPeGJxrrJK566q3wIDzGwQsDH893RgS4VE5lwMHpkDeZGlYTs1huN3jy8e5yqjdMdxvB8+L5CUZWZjgJ9VQFzOZVx+ATyYtDTsr/aGLL8F17kE6fRxLJbUwcwWAPOAgZJ+ALY5d5VzVclrC2DRuuLterXggm6xheNcpZVO4vg/YC9gAcEKfaMI1sa4ovzDci7zkuelOqdzMH7DOZeoTIkjnH12IrAQwMzGSGoK5JjZuu0e7FwVMOcnGL8kscwXa3IutTL1cVgwE+JMoCBSludJw1UX/0q62jhiV+jZIp5YnKvs0ukcnwpUyllwndsZa/PgibmJZZf51YZz25ROH8cE4L+SRhCsH140caiZPVa+YTmXOU/Ng7WRm8p3qQeD94gvHucqu3QSxxEEM+H2SSo3wtX6nKtqUs1LdUl3yMmOJx7nqoIyJw4z61eRgTgXhwnfBR3jhbIFv+geXzzOVQXpTDmSta1HGY8fIGmupPmSrk+xv72k8ZJmSJogqW1k3zBJX4aPYZHyAyXNDM95b3j3l3Nldt/MxO3Be0CbhvHE4lxVkU7neD7B9CKpHtslKRu4HzgB6A4MlZT8ve5u4Ekz25dgnMhd4bHNgJuBQ4CDgZvDW4EBHgAuBjqHjwFpvB9Xwy1aB68uSCzzW3CdK106iaMjsEfkcQTwGvCLMhx7MDDfzL42szxgJDAwqU534J3w+buR/ccDb5vZSjP7CXgbGCCpNdDYzD4Obxd+Ejg1jffjarh/z4KtkbVhezSF3q3ji8e5qiKdhZy+TXp8DAwjmCm3NG0I7sQqtDgsi5oODA6fDwIaSWq+nWPbhM+3d07nUtq8FR5OXhp2H18a1rmySOeKI5XGQMvyCAS4FugjaSrBnVtLgK3ldG7nEoz6Cr7fWLzdOAfO9VFKzpVJme+qkvQUkbEbQH2gN/CfMhy+BGgX2W4blhUxs+8IrzgkNQROM7NVkpYAfZOOnRAe3zapPGnSCOdSS56XanhXaOhLwzpXJulcccwHvoo8PgbONrPLy3DsJKCzpI6ScoAhwOhoBUktIndo3UDx2JCxwHGSmoad4scBY81sKbBG0qHh3VTnA77+uSvVlBXw0fLEsl/5Yk3OlVk64zhu3dEXMbN8SZcRJIFs4DEzmyXpNmCymY0muKq4S5IRTKj46/DYlZJuJ0g+ALeZ2crw+a+AEUA9YEz4cG67kq82jm0LXZumruucK0nBDUllqCjdC4w0sw8jZYcDZ5rZVRUT3s5ZvXp10ZvLzc2NMxRXSfy4Cdo+CZsivWevDICBHeOLybnKZvXq1UXPc3NzS9wykk5T1VBgclLZZ8DZOxSZczF4/IvEpLF7Qzi5fXzxOFcVpZM4LEX97DTP4VxsthaUnD790h6Q7Z9g59KSzq/M+8AdhR3Y4b+3hOXOVXr/XQTfrC3erpMNF+0VXzzOVVXpzI57JfA6sFTSt8DuwFLgZxURmHPlLXleqrM6Qct68cTiXFWWzl1ViyUdQDB9SDuC0dyfmlnB9o90Ln7zVwdXHFGX7RNPLM5VdekMAOwJ/BhONfJxWNZOUjMzm15B8TlXLpL7Nnq1Ch7OufSl08fxHyB5bG0O8FT5heNc+Vu/JbibKsqXhnVux6WTOHY3s6+jBWb2FdChXCNyrpw98yWsyiveblEXzuwUXzzOVXXpJI7CPo4i4fZ35RuSc+XHrORI8Z/vBXXTuS3EOZcgnV+fvwOvSvo/grmqOhHMaHtnRQTmXHn43zKY/mPxdpbglz4vlXM7JZ27qh6WtAq4iOCuqoXANWY2qoJic26nJV9t/Kw9tG8UTyzOVRfpXrBPBDYDLcLtxpIuNLPHtnOMcxm3tQD+MQNe+Cqx3JeGdW7npXM77qkEd1DNB3oAs4C9gQ8ongLdudjNWwUXvAsfLkss79oEjmmb6gjnXDrS6Ry/A7jQzPYH1of//oJgokPnYre1AP4+HfZ7vmTSALjj4KCPwzm3c9KZVn2NmTUOn/9kZk3D+aqWmVmlHErl06rXHPNXwwXvwAcpEkbTOnDfkXC2Lw3rXJmUNq16On0c30vaxcyWAwskHQb8QDBDrnOxKLBgDqrrP4GN+SX3n9we/t0HdmuQ+dicq67SSRwPA0cCLxLcmvsuUAD8tQLicq5UX62GC9+FiUtL7svNgXuPhPO6gLx5yrlyVeamqhIHSrsDDcxsThnrDwDuIbhCecTM/pTifE8ATcI615vZm5LOAa6LVN0XOMDMpkmaALQGNob7jjOz7wsrelNV9VRgwdxTv/sYNqS4yjhxd3ioD7RpmPnYnKsOSmuq2uHEkQ5J2cA84FhgMcH64UPNbHakzkPAVDN7QFJ34E0z65B0nn2AV8ysU7g9AbjWzJJXJgQ8cVRH36wJrjImpJivoHEO3HMEDOvqVxnO7Yzy7OPYGQcD8wvnupI0EhgIzI7UMaBx+DyX1FOZDAVGVmCcrpIqMPj3LLjuI1if4ipjQDt4uC+09asM5ypcphJHG4L1OwotBg5JqnML8Jaky4EGQP8U5zmLIOFEPS5pK0Hfyx2WiUsol1EL1sBFE+CdJSX3NaoNfz8CLuzmVxnOZUplWm15KDDCzNoCJwJPFS5TCyDpEGCDmUUnkTjHzPYBjgof52UyYFexLLzK2Of51Enj2Lbw+VnB8q+eNJzLnEwljiUE81sVahuWRV0EPA9gZh8BdSme2gRgCPBs9AAzWxL+uxZ4hqBJzFUD366F416HX06EdVsS9zWsHdxiO/Zk2N3nnXIu4zKVOCYBnSV1lJRDkARGJ9VZCBwDIGkvgsSxItzOAs4k0r8hqZakFuHz2sDJQNKUdq6qMYOHZ8M+z8G4xSX3H9MmuMr4RXe/ynAuLhnp4zCzfEmXAWMJbrV9zMxmSboNmGxmo4FrgIcl/Yago3x4pL+iN7AoaSGpOsDYMGlkA+MIxpq4KmrROrh4AoxdVHJfg1pw9+FwSWkJwwpg3RJYNa/4sWEZtDoIev4GVJlaZ52rmjJyO25c/HbcqsEsWNr1Nx/CmryS+/vtBo/2g46NI4Ubf0xMDoWP1V9C/saSJwHY7zdw1N8q5D04V51UlttxnUtp8Tr4xXswZmHJfS2y13PfvvM5o/k8subNg58iCWLzyvRfbPo/YI9ToU3vnQ3buRrNrzhcLMzgiblw7QdbaJq3gC7Mo4vNK/p3n+x5tNqaopNjZzXeA4ZMhxwf8OHctvgVh4ufGaz/ruhqYd2Kecz+Zh6HbZjHUr6mNilG9G3dider0wSadIUmXaBeS5gWaZ5a8zV89Dvoc/9OvIBzNZsnDld+Nq+Cn+am7nfYsr6oWkPK4b7p7LrQpHOQHJIfdZsn9qAXbIEZ/yzenvkv2GMwtDtmZ6Nwrkbypiq38/I3wcQrYM5jYDtzqZBEWdC4Y+rk0LBt2e+Q2rIeRvaE1fOLyxrtDkNnQk7jbR7mXE3lTVWuYm3NgzGnw7dv7PAp8uu1plbTSFIofN54D8jO2fkYazeAYx6Hl3oT3OkNrF0IH1wLRz+08+d3robxxOF2XEE+vHV2mZLGahozl67MUxfm0YUFWV04bt8uDD2wM7XqZmD4925HBuM4ov0dsx+GTqdB++Mr/vWdq0a8qcrtmIKtMO58mPdMYnn91tiuhzLHuvDAki5MzQ8SxQpaFvU7HLYLPN4PujbNcMz5G2Hk/rBqbnFZgzZw9udBh7pzDvCmKlcRrAAmXFIyaeR25ocTJnLJ5F156ZuwLPKRq5MNdxwMv9kXsuMYwF2rHvQfAS8eEbwHgPVL4P3fQP/HYwjIuarJ519w6TGDiVfC7EcTyxt14LWe4+n2WiRpRBzSCqadAdf2jClpFNr1UNj/usSyL0bAN6/HEo5zVZEnDld2ZvDh72DmfYnFDdpwTat3OOWDdvy4KfGQnCz486HwwSDolummqW055FZo1iOx7N2LYdMOjEZ3rgbyxOHK7tNbYepfEsvq78I/9niHvy3oWKJ6r1Yw9Qz47f5QqzJ90rLrQP8nQNnFZRuWwcTL44vJuSqkMv06u8rssz/DpFsTy+o2571e47h6dpeE4pwsuOsQ+HAQdG+WwRjT0epAOOj3iWXznoGvXoonHueqEE8crnTT/wkfXZ9YlpPLd8e8xaBJeycU71ofPjsdrj+gkl1lpHLQjdBiv8SyCb+EjSviice5KqKy/2q7uM16GN6/IrGsdkPyTvovAz87gJ82FxdnC547FvZuntkQd1h2TtBklRW5uXDjCnjv1/HF5FwV4InDbdvc/8C7lySWZdeFk1/nyvmHMjnpi/ldh0Dv3TIXXrlosR/0uimxbP4L8OXz8cTjXBXgicOlNn8UjBtG0RQdAFk5cNKr/Gd9Hx6cnVj91I7BrbZV0gHXQ8sDE8ve+xVsWB5PPM5VchlLHJIGSJorab6k61Ps313Su5KmSpoh6cSwvIOkjZKmhY8HI8ccKGlmeM57JV+Fulx88zq8NbR4kBwEzTknjGJWw+O45L3E6p0aByPBq+xPP7t2MDAwKzIv1qYfg6utajyzgnM7KiOJQ1I2cD9wAtAdGCqpe1K1G4HnzWx/YAjwr8i+r8ysZ/j4ZaT8AeBioHP4GFBR76HGWPg2jDktmIeqkLLguGdY2+ZnnDYWNkR21cmGUcdDkzqZD7VcNd87GN8R9c2rMO/peOJxrhLL1BXHwcB8M/vazPKAkcDApDoGFM5xnQt8t70TSmoNNDazjy2YcOtJ4NRyjbqmWTIR3hwIBdGFvwXHjMA6ncHPJ8DcVYmH3H8U9GyRwRgr0v7Xwi6HJJZNvBzWbfej6FyNk6nE0QZYFNleHJZF3QKcK2kx8CYQHY3VMWzCek/SUZFzRtcWTXVOV1bLPoHXTwomAozq+yB0O4/7Pofnv0rcdUE3uGivzIVY4bJqBU1W2XWLyzavCkaVe5OVc0UqU+f4UGCEmbUFTgSekpQFLAV2D5uwrgaekeSr75SnFVPhtQGwZV1i+VH/gL1/wcfL4JoPE3ft2xzuOzJjEWZO025w6J2JZd++CXNGxBKOc5VRphLHEqBdZLttWBZ1EfA8gJl9BNQFWpjZZjP7MSz/DPgK6BIe37aUc7rS/DgLXj0u+GYdddhdsN+V/LARzngLtkT6yRvnwIvHQ/3aGY00c/a7ElofkVj2wVWwdlHK6s7VNJlKHJOAzpI6Ssoh6PwenVRnIXAMgKS9CBLHCkktw851JO1B0An+tZktBdZIOjS8m+p84NXMvJ1qYtWX8Gp/2PRDYnmvP8KB17O1AM4ZD4vXJ+4e0Q/2rM7Lm2RlwzEjgmnYC+WtgXd+7k1WzpGhxGFm+cBlwFhgDsHdU7Mk3SbplLDaNcDFkqYDzwLDw07v3sAMSdOAUcAvzaxwGtNfAY8A8wmuRMZk4v1UC2sWwCtHB5P7RfW8Bg4O7i66/TN4K+lL9jX7waA9MhNirJrsCYf9ObFs0VvBSHrnajhfAbAmWrcEXjoK1iQtnLHPr6D3fSAxdiGc8EbC8D+O3BXeOQVqZ1MzWAG8cgwsmVBcVrshDJ0JjTvEFZVzFa60FQArU+e4y4QNy4M/hslJY68LoPc/QWLh2qCJKpo0WtWD546rQUkDgvErxzwGtRsUl21ZB+MvTBwc6VwN44mjJtn4I7zSP3HNbYDOQ6Hfw6As8rbCmW+RsCBTlmDksbBbA2qexh3hiLsTy5a8CzP/lbq+czWAJ46aYvMqGH0crPw8sXyPQeEMscGlxLUfwSffJ1a5vRf0q8kjZHpcAu36J5Z9+DtYNT+eeJyLmSeOmiBvLbx2IqyYklje/gQ4/tlgribgufnwz5mJVU5qH6ytUaNJcPSjkBMZPpS/AcZfAAVb44vLuZh44qjutmyAN06BZR8llrc9Gk54MVhGFfjiJ/j5hMQqHRrBk0cHTVU1XqPd4ci/J5Yt/QBm3BtPPM7FyBNHdbZ1M4wZnHhXEASD2058tWicwrotcNrY4N9COVnwwnHQrC6u0F4XQPsTE8s++j38NDd1feeqKU8c1dXWLfDfs2Dh2MTyVgfByW9ATkMgGM/2y/dg9k+J1e45Eg5qlaFYqwoJ+j0EdZoUl23dBOOGe5OVq1E8cVRHBVvh7XODacGjmu8Lp4yFOsVjWh6cBU9/mVjt3C5wSfKk9y7QsA0cldQ8tfxjmPbXeOJxLgaeOKobK4B3LoL5SUufNu0GA9+Gus2KiiZ9D1f9L7Faj6bwYO8qvChTJnQ9Fzqeklj28R+Deb+cqwE8cVQnZvDer+GLJxLLczvBwPFQv7jt6cdNweSFeZFxbA1rB5MXNqiukxeWFwn6/hvqFCdhCvKCpXa3btn2cc5VE544qgsz+OBq+PzBxPJGuwdJo+FuRUUFBuePh2/XJlZ9tC90bVrxoVYLDXaFPvcnlq34DKb8OXV956oRTxzVxSd/hOn/SCyr3zpIGo3bJxTfNQXeXJhY9Yp94Mw9KzbEaqfzWdDp9MSySbfBD9Pjice5DPHEUR1MvjN4RNVrCaeOD2Z5jRi/GG6alFj10F3gL4dVcIzVkQR9/xX8rAsVbAmbrPK2fZxzVZwnjqpu6t/g4xsTy+o0DTrCmyWu67pkHQx9O2iqKtSiLjx/HOTUpMkLy1O9ltDngcSyH6aXTOTOVSOeOKqymQ/A/65JLKvdKLjltsV+CcVbtsJZb8OKyOSFAp7uD+0aVnyo1dqepwUTRUZNvhO+n5K6vnNVnCeOqmrOCHjvV4llterDz96EXXqVqH79J/C/pDWbbj4IjmtXoqrbEb3/CfV3Ld62rTDu/GD0vnPVjCeOqmjeyGCsRlR2HThpNOx2ZInqL34Ff0vqrz2+HfzxoAqMsaap1xz6/TuxbOUs+PTWeOJxrgJlbAVASQOAe4Bs4BEz+1PS/t2BJ4AmYZ3rzexNSccCfwJygDzgOjN7JzxmAtAa2Bie5jgzK5oUfKdXAFwyEb6tZKvR5m+AmfcH32gLZdWGE1+BDieWqP7lKjhwFKyNDC9o1xCmnA4t6pWo7nbW28Ng7pPF28qC0z6EXQ+JLybn0lTaCoC1MhGEpGzgfuBYYDEwSdJoM5sdqXYjwVrkD0jqDrwJdAB+AH5mZt9J2ptg3fLo6hDnmNnkCgl8+Scw5U+l14uTsuH4kSmTxoZw8sJo0qidBc8f60mjwvS+BxaPg/XfBdtWAOOHw1lTiiaVdK6qy1RT1cHAfDP72szygJHAwKQ6BhQueJALfAdgZlPNLPwtZBZQT1KdDMRcBQj6PwmdBpfYYwa/eh9mrkws/+vhcOiuJaq78lKnSbB2R9RPXwRTkjhXTWQqcbQBFkW2F5N41QBwC3CupMUEVxuXpzjPacAUM4v2OD4uaZqkP0o1aYYlwdGPQNezU+59dA48kTTb91l7wmV7ZyC0mq79AOj+88SyaX+Dpf9LXd+5KiYjTVVlNBQYYWZ/lXQY8JSkvc2sAEBSD+DPwHGRY84xsyWSGgEvAucBTyafeIftdhQc+v/K7XTlJqsWtOkHu6Tu3Z66Ai77ILGsWxN4uI9PXpgxR/4VFr0FawuH6Fsw/fqQaVC7Ji7e7qqTTCWOJUD0xs+2YVnURcAAADP7SFJdoAXwvaS2wMvA+Wb2VeEBZrYk/HetpGcImsTKL3HsemjwqEJWbYbT34LNkb7z+rVg1PHQKCe+uGqcnMZBk9WrxxaXrZ4fLPzU+5744nKuHGSqqWoS0FlSR0k5wBBgdFKdhcAxAJL2AuoCKyQ1Ad4guMuq6FpfUi1JLcLntYGTgc8r+o1UZgUGw96Br9cklj/cB3o0S32Mq0Dt+sPelyaWzbgXlrwXTzzOlZOMJA4zywcuI7gjag7B3VOzJN0mqXBhg2uAiyVNB54Fhltwr/BlwJ7ATWFfxjRJrYA6wFhJM4BpBFcwD2fi/VRWd0+D0QsSyy7tAWd3iSMaB8Dh/weNOyaWjb8A8tbFE49z5SBj4zjisNPjOKqQ976Do0cnzkN1UEv4YBDU8Xmo4rXkPXi5b2LZ3pcGEyQ6VwmVNo7DR45XA0vXw1lvJSaNpnXgheM8aVQKbfrAvlckln3+ACwaF088zu0kTxxVXH4BDHkblm9MLP/PMdChcepjXAwOuwtykxY8GX8h5K1JXd+5SswTRxV346cwcWli2R8OgBPbp67vYlK7PvQfQTAncWjdInj/KijIjyko53aM93FUYaO/gYH/TSw7ug28dTJk+1eCyul/18HUuxPLsmpB407QpEvJR4PWPvjGZVxpfRyeOKqor9fAAS/A6shCc7s1gKmnQ6v68cXlSpG/EZ47IJiGpCxqN0idUJp0CaY3ca4CVIpJDl352pQPp49NTBrZCiYv9KRRydWqB/2fgFGHJ85wvC1b1sOKqcEjWb2WqRNK7p5Qq275x+5cyBNHFXT5BzD1h8Sy/zsMjmgdTzwuTbscDCe9GsxftXI2bFhW+jGpbFwRPErMgSVotHvqpNKoPWT5rXZu53hT1XZ8uhwmfFd6vUxash7unZlYNrhjMKWIN4VXUXlrYNWXsGpe4uOnubBlbfm+VlYO5Eb6U5p2LX5er5V/iBzgfRw7lTj+MhV++3G5hlTu9syFyadBrk80X/2YwcbvSyaUVfNg1XwoyCv9HOnIaRw2dXWGuk3L99wu82o3gsN3bD0h7+OoxurVgheP96RRbUlQf5fgsdtRifsKtsK6hfBTiqSy9luC5W3SlLcGvp8cPFzVV3+XHU4cpfHEUYU92Bv2bR53FC4WWdnBHFiNO0L74xP35W+C1V+lvlLZ+H3q8zmXBk8c29GrFVzXM+4oSqqdBf3bQr/kpbCcg+COquY9gkeyzatS96esmgdbfOJFVzbex+GcC/pTNiwLEsjq+cFVi6vaatWH7hfs0KHex+GcK50UjFJv0DqYlNG57fCJKZxzzqXFE4dzzrm0eOJwzjmXFk8czjnn0uKJwznnXFpqzF1V0dvLnHPO7Ti/4nDOOZcWTxzOOefSUq1HjjvnnCt/fsXhnHMuLZ44UpD0mKTvJX0edyxRktpJelfSbEmzJF0Zd0wAkupK+lTS9DCuW+OOKUpStqSpkl6PO5ZCkn4T/qw+l/SspNjWet3W513S5ZK+COP8vwzHlPIzJelpSXPDn9tjkmpnMq4whiaSRoU/mzmSDovsu0aSSWqR6bgyyRNHaiOAAXEHkUI+cI2ZdQcOBX4tqXvMMQFsBo42s/2AnsAASYfGG1KCK4E5cQdRSFIb4ArgIDPbG8gGhsQY0giSPu+S+gEDgf3MrAdwd4Zj2tZn6mmgG7APUA/4eYbjArgH+K+ZdQP2I/xsSWoHHAcsjCGmjPLEkYKZTQRWxh1HMjNbamZTwudrCT6wsU+uboHCOblrh49K0XkmqS1wEvBI3LEkqQXUk1QLqA/EtkjxNj7vlwJ/MrPNYZ2MLuSxrc+Umb0Z7jPgU6BtJuOSlAv0Bh4N48wzs1Xh7r8Dv6WSfPYrkieOKkpSB2B/4JOYQwGKmoOmAd8Db5tZpYgL+AfBL3NBzHEUMbMlBN/gFwJLgdVm9la8UZXQBThK0ieS3pPUK9MBbO8zFTZRnQf8N8NhdQRWAI+HzZ+PSGogaSCwxMymZzieWHjiqIIkNQReBK4yszVxxwNgZlvNrCfBN8CDJe0dc0hIOhn43sw+izuWKElNCZqBOgK7AQ0knRtvVCXUApoRNIleBzwvqcS6DBWplM/Uv4CJZvZ+JmMi+LkcADxgZvsD64FbgN8DN2U4lth44qhiwm9aLwJPm9lLcceTLLxsf5fK0Ud0BHCKpAXASOBoSf+JNyQA+gPfmNkKM9sCvAQcHnNMyRYDL4WtQp8SXLHF0uGb/JmSdDPQErg6hnAWA4sjVz+jCBJJR2B6+FlrC0yRtGsM8WWEJ44qJPzG9ygwx8z+Fnc8hSS1lNQkfF4POBb4ItagADO7wczamlkHgs7nd8ysMnyzXwgcKql++H96DJWo8z70CtAPQFIXIAf4IVMvvq3PlKSfA8cDQ80s482PZrYMWCSpa1h0DDDFzFqZWYfws7YYOCCsWy3VmLmq0iHpWaAv0ELSYuBmM3s03qiA4Bv0ecDMsO0X4Pdm9mZ8IQHQGnhCUjbBl5HnzazS3Ppa2ZjZJ5JGAVMI7pSbCjwUVzypPu/AY8Bj4S26ecAwy+xo4ZSfKUn5wLfAR2HL2UtmdlsG4wK4HHhaUg7wNbBj67NWYT5y3DnnXFq8qco551xaPHE455xLiycO55xzafHE4ZxzLi2eOJxzzqXFE4dzlYCkvuGtsBVx7g7hjK1++70rF544nKtmJC2Q1D/uOFz15YnDOedcWjxxuBor/GZ+naQZktZLelTSLpLGSForaVw4ISGSXpC0TNJqSRMl9QjLcyRNk3R5uJ0t6X+StjvhnaR6kkZI+knSbKBX0v7dJL0oaYWkbyRdEdl3S7iQ0HNhnFMk7RfuewrYHXhN0jpJv42c9hxJCyX9IOkP5fEzdDWTJw5X051GMA9SF+BnwBiCmU5bEvx+FP7BHgN0BloRTBXyNATrMQDnArdJ2gu4nmBhpjtLed2bgU7h43hgWOEOSVnAa8B0gvVWjgGuknR85PiBwAsEM9g+A7wiqbaZnUcwF9bPzKyhmUVX7jsS6Bqe76YwXufS5onD1XT/NLPl4RoZ7wOfmNlUM9sEvEyw5glm9piZrQ0XNroF2C9c1Acz+xy4g2BiwGuB88xsaymveyZwp5mtNLNFwL2Rfb2AlmZ2W7hQ0NfAwySuEviZmY0KZ9f9G1CXYAr07bnVzDaGa0ZMJ1i9zrm0eeJwNd3yyPONKbYbhs1Pf5L0laQ1wIJwf3Sa8SeA9sCbZvZlGV53N2BRZPvbyPP2wG6SVhU+CK6CdonUKTo2nCV2cXjO7YnO1roBaFiGOJ0rwROHc6U7m6BpqD+QC3QIy6MLG/0LeB04XtKRZTjnUqBdZHv3yPNFBOt1NIk8GpnZiZE6RceGTVttKV5+1mcudRXKE4dzpWsEbAZ+JFgf/P9Fd0o6DzgQGE7QJ/JEuErj9jwP3CCpqYJ10S+P7PsUWCvpd2EnerakvZOWbz1Q0uBwbMZVYXwfh/uWA3vswPt0rkw8cThXuicJmpKWALMp/gONpN0J1jU/38zWmdkzwGTg76Wc89bwnN8AbwFPFe4I+0dOBnqG+38AHiG42in0KnAW8BPBGi2Dw/4OgLuAG8NmrmvTf7vObZ+vx+FcFSPpFmDPSrKaoauB/IrDOedcWjxxOFdBwoGE61I8fh93bM7tDG+qcs45lxa/4nDOOZcWTxzOOefS4onDOedcWjxxOOecS4snDuecc2nxxOGccy4t/x8Lf/Sys7JQhAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracy_list_train = []\n",
    "accuracy_list_val = []\n",
    "for max_depth in max_depth_list:\n",
    "    # You can fit the model at the same time you define it, because the fit function returns the fitted estimator.\n",
    "    model = DecisionTreeClassifier(max_depth = max_depth,\n",
    "                                   random_state = RANDOM_STATE).fit(X_train,y_train) \n",
    "    predictions_train = model.predict(X_train) ## The predicted values for the train dataset\n",
    "    predictions_val = model.predict(X_val) ## The predicted values for the test dataset\n",
    "    accuracy_train = accuracy_score(predictions_train,y_train)\n",
    "    accuracy_val = accuracy_score(predictions_val,y_val)\n",
    "    accuracy_list_train.append(accuracy_train)\n",
    "    accuracy_list_val.append(accuracy_val)\n",
    "\n",
    "plt.title('Train x Validation metrics')\n",
    "plt.xlabel('max_depth')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xticks(ticks = range(len(max_depth_list )),labels=max_depth_list)\n",
    "plt.plot(accuracy_list_train)\n",
    "plt.plot(accuracy_list_val)\n",
    "plt.legend(['Train','Validation'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that in general, reducing `max_depth` can help to reduce overfitting.\n",
    "- Reducing `max_depth` from 8 to 4 increases validation accuracy closer to training accuracy, while significantly reducing training accuracy.\n",
    "- The validation accuracy reaches the highest at tree_depth=4. \n",
    "- When the `max_depth` is smaller than 3, both training and validation accuracy decreases.  The tree cannot make enough splits to distinguish positives from negatives (the model is underfitting the training set). \n",
    "- When the `max_depth` is too high ( >= 5), validation accuracy decreases while training accuracy increases, indicating that the model is overfitting to the training set.\n",
    "\n",
    "So we can choose the best values for these two hyper-parameters for our model to be:\n",
    "- `max_depth = 4`\n",
    "- `min_samples_split = 50` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "decision_tree_model = DecisionTreeClassifier(min_samples_split = 50,\n",
    "                                             max_depth = 3,\n",
    "                                             random_state = RANDOM_STATE).fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics train:\n",
      "\tAccuracy score: 0.8583\n",
      "Metrics validation:\n",
      "\tAccuracy score: 0.8641\n"
     ]
    }
   ],
   "source": [
    "print(f\"Metrics train:\\n\\tAccuracy score: {accuracy_score(decision_tree_model.predict(X_train),y_train):.4f}\")\n",
    "print(f\"Metrics validation:\\n\\tAccuracy score: {accuracy_score(decision_tree_model.predict(X_val),y_val):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No sign of overfitting, even though the metrics are not that good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Random Forest\n",
    "\n",
    "Now let's try the Random Forest algorithm also, using the Scikit-learn implementation. \n",
    "- All of the hyperparameters found in the decision tree model will also exist in this algorithm, since a random forest is an ensemble of many Decision Trees.\n",
    "- One additional hyperparameter for Random Forest is called `n_estimators` which is the number of Decision Trees that make up the Random Forest. \n",
    "\n",
    "Remember that for a Random Forest, we randomly choose a subset of the features AND randomly choose a subset of the training examples to train each individual tree.\n",
    "- Following the lectures, if $n$ is the number of features, we will randomly select $\\sqrt{n}$ of these features to train each individual tree. \n",
    "- Note that you can modify this by setting the `max_features` parameter.\n",
    "\n",
    "You can also speed up your training jobs with another parameter, `n_jobs`. \n",
    "- Since the fitting of each tree is independent of each other, it is possible fit more than one tree in parallel. \n",
    "- So setting `n_jobs` higher will increase how many CPU cores it will use. Note that the numbers very close to the maximum cores of your CPU may impact on the overall performance of your PC and even lead to freezes. \n",
    "- Changing this parameter does not impact on the final result but can reduce the training time.\n",
    "\n",
    "We will run the same script again, but with another parameter, `n_estimators`, where we will choose between 10, 50, and 100. The default is 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_samples_split_list = [2,10, 30, 50, 100, 200, 300, 700]  ## If the number is an integer, then it is the actual quantity of samples,\n",
    "                                             ## If it is a float, then it is the percentage of the dataset\n",
    "max_depth_list = [2, 4, 8, 16, 32, 64, None]\n",
    "n_estimators_list = [10,50,100,500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f45b6b5b1d0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEXCAYAAABLZvh6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0P0lEQVR4nO3deZwU5ZnA8d8zN9cMDKdcOtyCyCkeoIIHoiYeiBwmRmIuzSZqdqMbk1WJrrvJxpiYjYkbY3TXKIdniKAoeGBEEUQOuUHu+5wZGJjz2T/emqG76Tl6prurZ/r5fj71ma63rqdruuvpeqvqfUVVMcYYk5xS/A7AGGOMfywJGGNMErMkYIwxScySgDHGJDFLAsYYk8QsCRhjTBKzJGDqRURGi4iKSFe/Y6kvEXlfRP4cMP6ciMyvZZmpIlIWpe2riHw9GutqDETkLO89j/I7FnOKJYEmzvvS1TRsreeqFwFnALujF23tRORGL+7+1Uz/g4hsE5H6fLbvBm5uWIRhY5ovIs+FmXQG8HK0txdPNby3cHbg3vPi2EVkImVJoOk7I2C4ySsbGlB2XuDMIpJRl5Wqaomq7lXViijGWhd/B/YA3wmdICLNgVuAZ+oTl6rmq+qRhodY5+3tVdWT8dqen0QkQ1XLvfdc6nc85hRLAk2c96Xbq6p7gcNe8YGAsv0icpeIvCgi+cDzACLyqIisFZEiEdkhIk+JSE7lekOrgwLGrxSRhd5ya0Tk6upiE2eOiCwRkXSvLMX7dblQRFLDvJ8y4C/ArSKSGTJ5ItASeEZEhorImyKyX0SOedsYV9O+Cq0O8mJ5JGAdM4E2IcvkicirIrLbe8+rROTWwHUClwO3BZx9jfamBVUHicgZIjJDRI6KyAmvump4mH1e533sLTdNRDaJyEQR2egt97qIZIvIeBFZLyKFIvJy4P/YW3ayiCwXkZMislVEHheRFjW9t4Bqn6+JyFwROQ48Eq46SEQ6iMizIrLP28Z6Ebndm5bubW+niBSLyB4RmVHTezX1oKo2JMkAjAYU6BpQpsAh4AdAT6C3V/5vwMXAWbgv+jrgf6tbV8D4CmAc0Bt4FigA2tQQU3tcldJj3vjPvHi61bDMWUA5MCWk/CPgbwHxTAUGAH2AfwdKgD4B878P/Dlg/DlgfsD43cBx4DZvHfcBR4GygHkGevtukLf/fgiUAWO86TnAQmAm0MkbMgL2/de914KrJlkOjPLWOxM4ArRr4D6e5r2POcC5wKXAAeBtYK4X+yhgH/DLgOWmetu/FegBXAKsBJ6v6b15/x8FdgJfA/K8obJ8lLd8M2AtsAy4wtvGWGCyN/2fvXWMBrrjzlrv8ft71NQG3wOwIY7/7OqTwDN1WPZGoBhICbeugPHxAct09MquqmXdY7wD50NAKXBDHeJ5E3g3YPxsb1vX1rDMCuBnAePvU3MS2Ak8GrKOlwlIAtVs52/A0wHj84HnwswXmAQu98b7B0zPxFV9PdiQfYxLAmV4ycQrexKXSNsHlD0BLA0Y3wrcEbKuS7zttanuvQUc7B+oprwyCXwLOBn4eQyZ/wngXUD8/u405cGqgwzAp6EFXjXBQq+a4xjwAu5XXqda1rW88oWq7sMdaDrWtICqvgf8Gnew+rOqvl6HmP8EjBaRXt74d4DtuOSAiLQXd5F4nVe9cgx3VnBmHdaNiGQDXXAXwAP9I2S+5iLyCxFZLSKHve1cU9ftBBgAHFLVNZUFqlqMOzsYEDLv8oB56rSPgV2qejBgfC+wV1UPhJR1ALf/vPfwuFcVdsx7b2968/aidqd9rkIMA9ao6s5qpj+LOyPa5FVH3iR1vGZl6s6SgAFXVVBFRM4HXsKd6t+Iu5B8hze5ti9hSZiyGj9nXt3/SNzBrKeISB1i/jvuoPUd78DwDYIvCD+Hq866z/s7GHfwjPZB5FfA14Gf485oBuOqWGJ5sIp4H+POsAJpNWWV66n8ezfuPVUOg3DVUKvqEOfx2mepnqoux1Uj/Rj3np8AlnsJ2kSJJQETzijgoKr+m6ouVtUNQCyfB5iG+2U5EhiBO3DXSE9dIJ6KuyDcGngmYJZLgD+o6mxVXYWrVulR14BUtQDYBVwUMmlkyPglwAuqOktVVwBf4q4fBCoBTrvIHWI10FYCbn31LnyfD3xR17ijxTvD2AH0VdVNYYbKu5rq8t6q8xnQX2p41kRVj6nqa6p6FzAcV+13aT23Z8KwJGDCWQ+0F5FviUgPEfkG8P1YbEhELgV+AtymqouB7+LuJBlRh8X/DLQDfgfMUdVdAdPWA18TkYEiMhiYTuQHq18Dd4vIrSLSW0T+BXcBM9B64HoRGeEdwP8EdA6ZZwswTER6ikg78e6ECvEurvrkRREZKSLnAP8HZAF/jDDuaPkZcJeI/ExEzhGRviJyg4j8T8A8dXlv1ZkObANmi8gV3p1Wl4vIJAARude7w2iAiOQBt+POFjdE6f0ZLAmYMFT1DeBR4D9wp/2TgXujvR0RyQX+CjyhqvO8bc/CVeVMF5FWtcS5FXeHSxvcwTfQN3Gf70+B14G3gCURhvgELsH8BleVdCHwcMg8P8IdyN4DFuDOHkIfAPs1cBB3YfoAp59NoO5K6A24u7DmeLF2Aq4MqcuPG1V9HneW9RXcflyCO2sLTLa1vrca1l+E+1X/BTADd6fQk7i7hsDd9fTPwMe4z+GNwE2qur6+78mcTtxnzxhjTDKyMwFjjElilgSMMSaJWRIwxpgkZknAGGOSWJrfAUQiPz/frmIbY0w95eTknPYgpp0JGGNMErMkYIwxSaxRVQcFysnJqX0mY4xJcvn5+TVOtzMBY4xJYpYEjDEmiTXa6iBjTOOnqhw+fJiKinh3Vd00paSkkJubS91aY3fikgRE5C+4Rqj2q+o5YaYLrrGua4AiYKqqLotHbMYY/xw+fJgWLVqQlZXldyhNwsmTJzl8+DBt27at8zLxqg56DtcnanWuxnVU0RvXlLBfTecaY+KooqLCEkAUZWVlRXxWFZckoKoLgcM1zHI98H/qfAK0FpEzoh3Hy5vhoU+hzM48jTEGSJwLw11wvRhV2umVRc3OY/DdD+Dhz2DUa7Cp5rumjDFJ4NChQwwePJjBgwfTqVMnunTpUjVeUhKuF89Tli5dyl133RWnSGMnKS4MVyjc9i4cKXbji/fD4FnwxCi4vR9EcA3FGNOEtG3bluXLlwMwbdo0WrZsyY9//OOq6WVlZaSlhT9MDh8+nOHDh8cjzJhKlCSwC+gWMN6V4N6LGmRrIawOqYw6Xgbffh/e2AZPXwrtmoVd1BgTRxLlq4F6Z+TLTJ06laysLD7//HNGjhzJ5MmTufvuuzl58iTNmjXj2WefpW/fvrz//vs89thjvPHGG0ybNo3t27fz5Zdfsn37du65555Gc5aQKElgNvADEZmB61g7X1X3RGvlPbJh1SR30J+9NXja61vgk33w3Bi4qnu0tmiMacx27tzJokWLSE1NpaCggA8//JC0tDTmz5/PT3/6U1555ZXTllm3bh3vvfcehYWF9O3blzvvvJP09Ei6XPZHvG4RnQ6MBtqJyE7gISAdQFWfAubibg/dhLtF9JvRjqF9M3h9HPx5LdzzERSVnZq2twjGzYG7BsIvLoBmiZIajTG+uPnmm0lNTQVcswu33XYbGzduREQoLS0Nu8y1115LZmYmmZmZdOjQgX379tG1a9d4hl0vcTncqeqUWqYr8E+xjkMEvtMfRneGry2AJfuDp/9uFczfCS9cAYPbxToaY0yiatGiRdXrBx54gDFjxvDaa6+xdetWRo8eHXaZzMzMqtepqamUlZWFnS/RJOVv3t6t4aMb4JHP4NFl7sJxpTVHYMQr8B/nwz8PghS7aGxM3NSnDj/W8vPz6dLF3az43HPP+RtMDCTKLaJxl54KD4+AD2+AvFbB00or4N6P4YrZsOOYL+EZYxLEfffdx/3338+QIUMaza/7SIiriWkcAnsWi2ZT0gUlcPc/4Ln1p09rnQFPXQqTekVtc8YYz4EDB2jfvr3fYTQpofs0sClp61msGtkZ8Oxl8NJYaJMZPO1oCUx+B76xAPKL/YnPGGNixZJAgAk9YdVEuDzMs8rPb4BBL8GHu+MflzHGxIolgRBdWsLbX4XHL4KMkL2zrRBGz4afLYaScn/iM8aYaLIkEEaKwI8GwdIJcE5u8LQKhf9YBhe9BuuP+BOfMcZEiyWBGgxsC0tugh+de/q0zw7AkJfhqdXQiK6tG2NMEEsCtchKg8dHwttfgc4tgqedKIM7F8J1b8L+In/iM8aYhrAkUEdXdoOVE+GmHqdPe2MbDJwFb2yNe1jGmAYYM2YM8+bNCyr77W9/y513hn9qbfTo0SxduhSAa665hqNHj542z7Rp03jsscdq3O7rr7/OmjVrqsYffPBB5s+fH2H00WFJIAJts9xtpM+OgZYh7ULtPwFffRO+vxCKwjctYoxJMFOmTGHGjBlBZTNmzGDKlBpbugFg7ty5tG7dul7bDU0CDz/8MFdccUW91tVQlgQiJAJT+8GKiXBhx9On/3E1DH3ZXTMwxkTo9xLdoRYTJkxgzpw5VR3IbN26ld27dzN9+nSGDx/OgAEDeOihh8Iue9ZZZ3Hw4EEAHn30Ufr06cOoUaNYv/7UU6dPP/005513HoMGDeKmm26iqKiIRYsWMXv2bO69914GDx7M5s2bmTp1Ki+//DIACxYsYMiQIQwcOJDbb7+d4uLiqu099NBDDB06lIEDB7Ju3boG7epKlgTqqUc2LLwBHj4PUkM+a+uPwgWvwn8ug3LrytKYhJWbm8uIESN48803AXcWMHHiRB599FGWLl3KypUr+eCDD1i5cmW16/jss8+YMWMGy5cvZ+7cuSxZsqRq2vjx41myZAkrVqzg7LPP5plnnuGiiy7iuuuu41e/+hXLly+nZ8+eVfOfPHmSqVOnMnPmTFatWkVZWRl//OOpThbatWvHsmXLuPPOO2utcqorSwINkJYCDwyHRTdCr5BWLMoq4KeLYcxs2FrgT3zGmNoFVglVVgXNmjWLoUOHMmTIEFavXh1UdRPqww8/5MYbb6R58+ZkZ2dz3XXXVU374osvuPjiixk4cCAvvPACq1evrjGW9evXk5eXR58+fQC47bbbWLhwYdX08ePHAzBs2DC2bt1a37ccxJJAFIzoCJ/fDN85+/RpH+5xTxr/dYPdSmpMIrr++utZsGABy5Yto6ioiNzcXB577DEWLFjAypUrufbaazl58mS91j116lR+//vfs2rVKh566KF6r6dSZXPV0WyqOimbko6Flunwp9FwzZmuB7NDAf/rghK4dQHM2QZ/uOT09omMMZ4fxP+XUsuWLRkzZgy33347U6ZMoaCggBYtWpCTk8O+fft48803q+1DAOCSSy5h6tSp3H///ZSVlfH3v/+d733vewAUFhZyxhlnUFpaygsvvFDVJHWrVq0oLCw8bV19+/Zl69atbNq0iV69evH8889z6aWXxuR9V7IzgSi7Ic+1PzSu2+nTZmyCc2fCe1HrPdkYEw1TpkxhxYoVTJkyhUGDBjFkyBD69evHLbfcwsiRI2tcdujQoUyaNIlBgwZx9dVXc95551VNe+SRRzj//PMZOXIk/fr1qyqfPHkyv/rVrxgyZAibN2+uKs/KyuLZZ5/l5ptvZuDAgaSkpHDHHXdE/w0HsKakY0QVnvzC9UtwMqSdIQF+PBgeGQGZqX5EZ0xisKako8+akk4QIvCDgfDZhNO7qlTgV8vdHURrDvsRnTHGOJYEYqx/LnwyHu4b7M4AAi0/CMNehv9eZReNjTH+sCQQB5mp8MsL4d3roFvL4Gkny+Guf8DVc2DPcX/iM8YkL0sCcTS6i3vSeHKYrirn7XDtD72+Jf5xGWOSlyWBOGuTCdOvhL9e7rq1DHToJNz4FnznfThm7Q+ZJJCSktLge+fNKSdPniQlJbLDut0d5KNtha7v4oV7Tp/WKwf+exSM7eY6uTGmKVJVDh8+TEWFta8SDSkpKeTm5iJy6qBR291BlgR8Vl7h7hR6YIlraiJU7xy4c4BrtM4eMjPGRMqSQCOx7ADcMt81PhdOszSY0gu+fw4Ms9uqjTF1ZEmgESkqdQ+X/aHmNqYY0QG+PwAm9nLJwRhjqmNJoBFavA9+twpe2gylNVSV5mbCt86GOwa4pq2NMSaUJYFGbF8R/GWd68x++7Hq5xNgXHd3dnB1d0i1e76MMR5LAk1AeQXM3Q5/+ALe2lHzvGe2gjv6uzOE9s3iE58xJnFZEmhiNuW7M4O/rIMjxdXPl5ECN/eEfzoHLujo2jIyxiQfSwJN1IkymLnJXUResr/meQe3c1VFt/SGFunxic8YkxgsCSSBJftdVdGMTac3Wx0oOwOm9nXPHfRrE7/4jDH+sSSQRA6dhOfWwR9Xw+Za+jW+rIs7O7g+z/WVbIxpmiwJJKEKhXd2uKqiN7a58ep0bgHf6w/fPtu9NsY0LQmTBERkHPAEkAr8WVV/ETL9TOAvQHvgMPB1Vd0ZOI8lgchtK4Q/rYGn18CBGtrpSkuBG/Pc2cGlne1CsjFNRUIkARFJBTYAVwI7gSXAFFVdEzDPS8Abqvq/InIZ8E1VvTVwPZYE6q+4HF790l07+Mfemuc9u41LBrf2gRxrr8iYRi1RksCFwDRVvcobvx9AVf8zYJ7VwDhV3SGuCbx8VQ16DtaSQHSsOOiuG/x1Axwvq36+Fmnw9T6uvaJz28YvPmNM9CRKH8NdgMDHnHZ6ZYFWAOO91zcCrUTEDj0xMKgdPHUp7PqGa6767GruFDpeBv+zBgbNgotfg+kb3RmFMabpSKT7Qn4MXCoinwOXArsAO+TEUE4m/GAgrJ4E710HE3pAajXXAv6x17Vy2v15+Nli2F4Y31iNMbGRMNVBIfO3BNapatfA8gZVB508Arvei2wZP6VmQbvB0LJzXDe7+7i7iPynte51dVIEvnKmu3ZwpXV8Y0zCSpRrAmm4C8OX437hLwFuUdXVAfO0Aw6raoWIPAqUq+qDgetpUBLYtwReGlHv9+CbVt2h4wXQ6ULodAG0HwKpsb9aW1oOs7e620zf3VXzvD2zYXwP6NICzmjuhk7N4YwW0NKeUDbGVwmRBABE5Brgt7hbRP+iqo+KyMPAUlWdLSITgP8EFFgI/JOqBrWOk5RJIFRKBrQf6hJCZWJo2S2m93SuPeLaK3puPRSURLZsy/SApFA5tDi9rG2W3ZZqTCwkTBKIBksC1WjROfhsocMwSIt+E6LHSt3F4Se/gBWHorvu9BTo2OxUggiXODo1h07NID01uts2pimzJFDp6EZY9K/RDil2ju+GA59DRYQ/vQFS0tz1hMDEkJ0XtZ/aqvDxPvfMwUuboSTOfYS3ywpJEmHOLKwqyhjHkkBjVl7sEsHeT2Dvx7DvEyjcXr91Netwqgqp4wXQYThktGxwiPuL4M3tsO0Y7DkOe4pgb9Gpv/FOEIFapIU/s+jaEga0cbfGZln3nKaJsyTQ1Bzb7ZJBZWLYvxTKa2gPojqSAm3PDU4MrXtHtWJeFQ4Xn0oKoUmismzvicivNURDqkDf1u5BuMChawu7PmGaDksCTV15KRxc4SWGj11yKPiyfuvKzA2+4NxxBGTEp/Pi46XBZxDVJY0DJ9ydAxFTpTlFZFNAOakckA7Vzto64/TEcE6u9cVgGidLAsmoaB/sWxxQjfQplBXVY0UCuQOCE0Obfu4sIl5UofQYlBRASQFlxQXkFxZw9FgBhccLKCoqoOREAWUn86koKUBKCkgrKyCjvIBWWkA2p4a0gGcPN9GTeXIVb8tY3mMMhVJzshOgZw6cmxucHPKy7RkJk9gsCRioKINDX7izhT3etYWjG+q3rowc6Hh+wNnC+ZAVpt0JrXAH7+L8qgM4JQVQWhA8XlIAJflhygKG+v32r7NS0viYC6uSwmcMQ+uY6FqkwcDAs4ZcN97aGt4zCcKSgAnvxCF3tlBVjbQYSuvZFkSbfi45BB3sG2+7EgdpyztyJW8zlrdlLLsltJmr2nVveXqVUu8c68DHxJ8lAVM3FeVwZK2rQqpMDIfX1L5cY5Ka5a5xFB+BitI6L7YhdQBvVFzFPBnLQi7hpNTvGYzMVHdXUmhyaB/9RzqMqWJJwNRf8VF3PSHwFtXio/GPI625O3gHDTlhyrIhPeB1Zk5AWStIzXDrKz0Ouz6A7fPccHR9nUMpS8lkU4tLeDflKl4ovopFJQMafCtRp+anqpIqE0O/Ni5pGNNQlgRM9GiFu5awN+BOpMNfuPJw0lvWfvBOD3MgD5qnlXv4LZYKtsGOt2H727BzfkSJrrx5Z/a0HctnWWOZW3Eliwrase4olDXw+Yi0FOjXGgbmwtD2cGFHGNbenmswkbMkYGKrpBAOrXKvAw/e6S0hpRH+lK0oc89eVJ4l7FtcfZI7jUD7oZR1u4otra9isVzA8iMZrDwEKw/BvhMNCy09BQa3cwnhgo7u75mt7JkGUzNLAsY0RPFR2LHAO1OYB4Xb6r5sekvoehl0Gwvdr2Jfek9WHZGqpLDyEKw+3LCnqjs1P5UQKs8WmtvzDCaAJQFjokXVtUG1fZ5LCrvec9cX6io7D7pf5ZJC18sgM4eyCthw9FRSWHnY/d1xrH4hpqXAoLYBiaET5NnZQlKzJGBMrJQXw55F7lrCjnmunae6klT3nEVlUugwPKj67EgxrDoEnx+Exftcg31b63nXbYdmLilUJobzOtjTz8nEkoAx8VK0H3a8c+pMoWhf3ZfNzIVuV0D3sS4ptOp22ix7i+CTffDxXvd3yQE4URZ5mKniHmgLvLbQK8fOFpoqSwLG+EEVDq10Zwnb58HuDyNrFrzN2e4softYaDvw1MX2gCeZS8th1WF3llCZGDYX1C/ctlnB1xbO6wCtMuq3LpNYLAkYkwhKi2D3B6eSwpG19VtPeqtTz0CEub32uGSztTib9UU5rDqWzecF2ewvD2xBKZtCWlEhNd+5lSKu0bzAxNC7tbWT1BhZEjAmERXuOHXH0Y757inmODpGC/LJIbiJvWwKJHi8ah7JRtOzyWuXTd/22Zx7RjZDO2eT0yzxH1xQhdIKKC4PGCpceY/spl8NZknAmERXUe6eTahMCns/AS2vfbkEUERzStLcWUhmVjZZWc0BQYEKdYMqVNQwXoFXVpfxCNYbGEM4ZaSxLPMyJk38V87MaYTPtNSRJQFjGpvio7DzPZcQ9nwEJw+51lcjuR3V1NkL7X7B1yY3oq5nI2RJwJimoqLMPaFdY7PcdWieuxG38BoL2+lO7ne30DKjaTbxWlsSSPwKPWOMk5Lm+m4I139DJCr7eigpCO7vodqkko+WFHCiqIDiEwVUlBSQXlZASy0gJcZ9PcRDd7bzzvKPuXLESL9D8YUlAWOSjaScuqOoZde6LQI094ZK+SeVz/YcZ/meAtbtK2DLwXxKS10DSWni2jrKSHV/0wXSUyEjxStPceNVryvHJXhadfNmePNWLle1jnDjKe7ZiEDr3v0l/QreqhovXfciJGkSsOogY0xUqLp2kNJTEv9W0k2fv06vj26sGt9Pe7K+vZvsJthMa23VQU2zEswYE3cirg+ERE8AAD0HXk2BnPoh2YEDfPrZAh8j8k+dk4CIvCYiN4iItTpijGnUJC2TDbnjg8s2vuhTNP6K5EzgQ+BBYK+I/FFELopRTMYYE3OtB94SNH7esdc4cryBnT40QnVOAqr6uKoOBS4BjgLTRWSjiDwoIj1jFaAxxsRCz7PHcEA6Vo1nU8jypXN9jMgfEV8TUNXVqno/8HWgCHgIWCYi80VkULQDNMaYWJDUVDZ1mBhUlr45+aqEIkoCItJXRB4Rkc3An4CZwFlAR2Au8Hq0AzTGmFjpcG5wldDwojkcLsivZu6mKZILw0uBj4Bc4BZVPVtV/0NVd6jqSVV9PGZRGmNMDPTscz47UvKqxrMoZtWS1/0LyAeRnAn8Auisqv+kqovDzaCqeeHKjTEmIYmwtdPkoKIWW5KrSiiSJFCAq/qp4lUPXRnViIwxJo66DAmuEhp8cgEHD+/3KZr4iyQJPAmEtjxV6JUbY0yj1CPvHDamnlM1nkY5a5e85GNE8RVJEuigqntCyvYAnaIYjzHGxN3OzlOCxltvm+5TJPEXSRL4UkQuCykbDWypy8IiMk5E1ovIJhH5SZjp3UXkPRH5XERWisg1EcRmjDH1dtaw4CQwsOQj9u/f5lM08RVJEpgGvCoivxaR74vIr4FXcE8R10hEUnHVRlcD/YEpItI/ZLZ/A2ap6hBgMvCHCGIzxph6y+uax8r0C4LKNi2Z4VM08RXJE8N/A8YCLYBrvb9XeeW1GQFsUtUvVbUEmAFcH7oJINt7nQPsrmtsxhjTUPu7Bp8NtN+ZHFVCEbWbqqqfAp/WYztdgB0B4zuB80PmmQa8LSI/xCWYK+qxHWOMqZfewydSvuVHpFLhxktXsG/nGjp2Da20aFoifWJ4sIj8UER+LiIPVw5RimUK8JyqdgWuAZ4XEWvq2hgTF2d27MTSjODLnluXNf2zgUieGP4u7onhy4B/BQYC/wL0qsPiu4BuAeNdvbJA3wJmAajqx0AW0K6u8RljTEMd6R5cJdRl13TXW04TFskv7fuAcap6I3DC+zsBKK3DskuA3iKSJyIZuAu/s0Pm2Q5cDiAiZ+OSwIEI4jPGmAbpf954ismoGu9avpm9W5b4GFHsRfqcwIfe6woRSVHVN4Gv1ragqpYBPwDmAWtxdwGt9qqTrvNm+xfgOyKyApgOTNXG1PelMabR6962NZ9kBd+dvmt5064SiuTC8E4ROUtVtwIbgOtF5CBQUpeFVXUurqXRwLIHA16vAZKzp2djTMI4dtYUWPd61Xj3vTOh4jFISfUvqBiK5Ezgv4CzvdcPA38F3gV+Hu2gjDHGL0OGf4VCWlaNt6/Yw56NH/gYUWzVKQmIiAALgXcAvGqgNkAbVf1j7MIzxpj46ty6OYua3RBUtn9l060SqlMS8OrmV4F3A60rK1HVY7EKzBhj/FLSM/guobP2vwLlxT5FE1uRVAd9DvSJVSDGGJMohg+7koO0rRrP0SPsXjvPx4hiJ5Ik8D7wlohME5FvicjtlUOMYjPGGF+c0Sqdj1rcHFR2ZFXTrBKK5O6gkbgWQy8NKVfgL1GLyBhjEoD2ngLLn6oa73FoNpQeh/QWPkYVfXVOAqo6JpaBGGNMIrloyCh2LO9KN3YC0Iwidn0xmy5DptSyZOMSSbMRKdUNsQzQGGP80KFFCh9nB/c/fHx10+t/OJIDeBmuiYhwgzHGNDlpfYN/9ecdnQcnD/sUTWxEkgTygB4Bw0jg78B3YxCXMcb47tJzh7A+4KbIdErZvfIVHyOKvkg6ldkWMnwC3IZrUdQYY5qcts2ET1vfElRWvLZpVQk1tD4/G2gfjUCMMSYRNT87uErozMIP0MLQlvAbr0guDD8vIv8XMLwMfIZrQ8gYY5qky87pwzKGVo2noOxeOcvHiKIrkjOBTcDmgOET4BZV/WEsAjPGmETQJhM+zw2uEtL1TadKKJLnBKy1UGNMUmo9YBIVH95LCq6Lk65FS9EjG5E2vX2OrOEiqQ76nYhcFFJ2kYj8NupRGWNMArmiX1f+IRcHle1eMcOnaKIrkuqgKcDSkLLPgFvCzGuMMU1GTiasbht8gTh904tNov/hSJKAhpk/NcJ1GGNMo9ThnAmUBtSgdzi5Dj2wwseIoiOSA/iHwL9XNhPh/Z3mlRtjTJM2tk875svYoLI9Kxp/y6KRJIG7gSuAPSLyKbAbuBKwu4OMMU1eqwzY0CG4SihrywzQimqWaBwiuTtop4gMBUYA3YAdwKeqjXwPGGNMHXUbeD1F+5rRnBMA5JZsp2L3IlK6jPI5svqL5O6gwUAXVf1EVV/ymo3oIiKDYhadMcYkkKt6tmJuyleDyvY18v6HI6kO+iuQHlKWATwfvXCMMSZxtUiHLZ2Cq4RabnsJyhtvY8qRJIHuqvplYIGqbgbOimpExhiTwHoPvJqj5FSNtyo7QMWOBT5G1DCRJIHKawJVvPHd0Q3JGGMS11V5mcxOvSmo7EAjrhKKJAn8BvibiPxQRK4RkR8CrwGPxyY0Y4xJPM3SYGfn4Cqh7J2vQdkJnyJqmEjuDnpaRI4C38LdHbQd+BdVfTlGsRljTEIaMHAMe3d0pBP7AGhWUUj5ljmk9p7gc2SRi/Rp34XAH4BfAy8B2SJye9SjMsaYBDbuzFReS50UVHZoVeOsEorkFtEbcM1J/xx4CveQ2P8At8YkMmOMSVCZqXCgW3CVUJs9c6A436eI6i+SM4F/B25X1SHAce/vd3GNyBljTFIZds75fEle1Xi6FlO++TUfI6qfSG8RfSmk7H+Bb0QxHmOMaRSu7Ca8lhZ8NnCkEVYJRZIE9otIR+/1VhG5EOiJa0nUGGOSSkYqHD0zpErowAIo2udTRPUTSRJ4GqhsIOM3wHvACtyFYmOMSTqjBpzDSgZWjadSTvmG0AqTxFbnJKCqv1TVV7zX/wf0AYap6gOxCs4YYxLZZV3gb+nBZwMFqxtXlVC9O4RR1e2qujaawRhjTGOSngpFeZODytocWQQF23yKKHJx6xVMRMaJyHoR2SQiPwkz/TcistwbNngPphljTEK7vH8eH3NBUFnZhsbT/3BckoCIpAJPAlcD/YEpItI/cB5V/ZGqDlbVwcB/A6/GIzZjjGmI0Z1hdkZwV+vHV7/oUzSRi9eZwAhgk6p+qaolwAzg+hrmnwI0roo1Y0xSSkuBip4TKQ84nOYUroTDa3yMqu7ilQS64Hoiq7TTKzuNiJwJ5AHvxiEuY4xpsKv7deRdLgsqK1vXOH7Hxu2aQAQmAy+rarnfgRhjTF1cfAbMyQquEipe9yKo+hRR3cUrCezCtTxaqatXFs5krCrIGNOIpKZARq8bKSajqqxF0Zewf4mPUdVNvJLAEqC3iOSJSAbuQD87dCYR6Qe0AT6OU1zGGBMV1/VtzVyuCSorbQRVQnFJAqpaBvwAmAesBWap6moReVhErguYdTIwQ7URnEMZY0yAizrBW82Cq4TKNsyAisSu2ZbGdLzNz8+vCjYnJ6emWY0xJu7u+/AED6zoQCuOnSq8YQF0vaz6hWIsP/9U89Y5OTkSOj0RLwwbY0yjNL53M16XG4LKEr1KyJKAMcZEyfkdYX5IlZBuehnKi32KqHaWBIwxJkpEoFPfKzhI26qyjLKjsH2ef0HVwpKAMcZE0YTe6bwkNweVJXKVkCUBY4yJouHt4f0WwVVCsmU2lByrZgl/WRIwxpgoEoEefUeyg65VZWkVRbDltEejEoIlAWOMibKJvVKYIcH9DJSuT8wqIUsCxhgTZYPbwT9aBVcJpe54C04c8imi6lkSMMaYKBOBgX0Gs46+VWUpWgabX/ExqvAsCRhjTAxM6i1Ml+D+hxOxSsiSgDHGxMA5ufBpTnASSNvzARyrrgFlf1gSMMaYGBCB8/v2YSnDTpWhsHGmj1GdzpKAMcbEyMSeJHyVkCUBY4yJkf65sKLNJCo41Xhn+sGlcHSjj1EFsyRgjDExNLpvVxZySXDhhsQ5G7AkYIwxMRSuSqhs/fSE6X/YkoAxxsRQn9awvt0ESkmrKkvLXwcHV/gXVABLAsYYE2NX9W7LPK4KLtzwoj/BhLAkYIwxMRa2SmjDDNAKnyI6xZKAMcbEWM8c2N7+eopoVlWWdnwH7FnkY1SOJQFjjImD63q3ZLZcF1y40f+7hCwJGGNMHEzsdXqVUPmGWVBe6lNEjiUBY4yJgzNbweEO4zhC66qy1OKDsHOBf0FhScAYY+JmfO9MXpGbggt9fnDMkoAxxsTJhDB3CVVsfhXKTvgUkSUBY4yJm24tobTTaPbQqaospewYbJ3jW0yWBIwxJo4m9EplpkwKLvSxSsiSgDHGxNGEnjAjtEpo6xwozvclHksCxhgTR51bQFbnEWymR1VZSkUxfPmaL/FYEjDGmDib2EuYIZODC32qErIkYIwxcXZTD5iRcktQme6cD0X74h6LJQFjjImzjs2hQ5cBrGRgVZloBWx6Ke6xWBIwxhgfTArTjIQfVUKWBIwxxgfj8+CllJDrAnsXQcHWuMZhScAYY3zQrhn07JbHIi4MnrBxRlzjiFsSEJFxIrJeRDaJyE+qmWeiiKwRkdUikhjd7hhjTIxMCtOMhMa5SiguSUBEUoEngauB/sAUEekfMk9v4H5gpKoOAO6JR2zGGOOXG/Lg1dSJlAcciuXQSji0Om4xxOtMYASwSVW/VNUSYAZwfcg83wGeVNUjAKq6P06xGWOML3KzYHC3jizg8uAJcexsJl5JoAuwI2B8p1cWqA/QR0Q+EpFPRGRcnGIzxhjfhOt/WDdMB9W4bD+RLgynAb2B0cAU4GkRae1nQMYYE2vX58EbqeM5SWZVmRR8CfuXxGX78UoCu4BuAeNdvbJAO4HZqlqqqluADbikYIwxTVbrTLiwew5zuSZ4wob43BsTrySwBOgtInkikgFMBmaHzPM67iwAEWmHqx76Mk7xGWOMbyb1gukpIVVCG2dCRXnMtx2XJKCqZcAPgHnAWmCWqq4WkYdF5DpvtnnAIRFZA7wH3Kuqh+IRnzHG+Om6s+Cd1K9QSMuqMinaC7s/iPm2ReN08SEa8vPzq4LNycnxMxRjjImq8W/BDZu/wTf0+VOF/b8Fl/25QevNzz/VT0FOTo6ETk+kC8PGGJO0wrUlpJtfgfLimG7XkoAxxiSAa8+Ej9Ku4ADtqsqk+ChsnxfT7VoSMMaYBNAyHcadlc5LcnPwhBjfJWRJwBhjEkTYB8e2zIaSYzHbpiUBY4xJENd0h+VpI9lB16oyKTsBW0LvqI8eSwLGGJMgmqfDV/NSTu9sJoZtCVkSMMaYBDIx3F1C29+CE7F5bMqSgDHGJJBx3WBz+mDW0beqTCrKYPMrMdmeJQFjjEkgWWlwfQ/hRbkleEKMqoQsCRhjTIIJ2+PYrg/gWGi7mw1nScAYYxLMld3gQGZvljKsqkxQ2Dgz6tuyJGCMMQkmMxVuzCMuVUKWBIwxJgFN6gUzZRIVuDbfPk4ZRXnfb0a9x7G0qK7NGGNMVFzeBU5mdeHWE8/zoVzMDunOm21g3GntgDaMnQkYY0wCSk+F8T3gxZSvsUO6AzBrc/S3Y0nAGGMS1MSewePv7YKKKHcBY9VBxhiToMZ0gR7ZcFEnlxDGdoOUKFcHWRIwxpgElZYCG2+J/oE/kFUHGWNMAotlAgBLAsYYk9QsCRhjTBKzJGCMMUnMkoAxxiQxSwLGGJPELAkYY0wSa7TPCeTn5/sdgjHGNHp2JmCMMUnMkoAxxiQx0Si3TW2MMabxsDMBY4xJYk0+CYhINxF5T0TWiMhqEbnb75hCichfRGS/iHwRUJYrIu+IyEbvbxs/Y6wkIlki8qmIrPD258+98jwRWSwim0Rkpohk+B1rJRHZKiKrRGS5iCz1yhJi/0byvxfnd94+XikiQ32IN+z3KRFjjvSzKiKZ3vgmb/pZ8YrV235f7zNaORSIyD0x37eq2qQH4AxgqPe6FbAB6O93XCExXgIMBb4IKPsv4Cfe658Av/Q7Ti8WAVp6r9OBxcAFwCxgslf+FHCn37EGxLwVaBdSlhD7N5L/PXAN8Kb3P7gAWOxDvGG/T4kYc6SfVeD7wFPe68nATB8/s6nAXuDMWO9bX96gnwPwN+BKv+MIE9dZIQeC9cAZ3uszgPV+xxgm5ubAMuB84CCQ5pVfCMzzO76AOMMlgYTZv3X93wP/A0wJN5+Psf8NuDLRY67LZxWYB1zovU7z5hOf9utY4KN4fB6afHVQIO/0bgjuF0Gi66iqe7zXe4GOfgYTSERSRWQ5sB94B9gMHFXVMm+WnUAXn8ILR4G3ReQzEfmuV5aw+5fqY+sC7AiYz9f9HPJ9SsiYI/ysVsXqTc8H2sYr1hCTgene65ju20b7sFikRKQl8Apwj6oW+B1PJFRVRSRhbuNS1XJgsIi0Bl4D+vkbUa1GqeouEekAvCMi6wInJtr+DZSosYV+n0RONXqfSDE3ws8q3jWK64D7Q6fFYt8mxZmAiKTjPrAvqOqrfsdTR/tE5AwA7+9+n+M5jaoeBd7DnVK3FpHKHxVdgV1+xRVKVXd5f/fjDgQjSOz9W11su4BuAfP5sp+r+T4ldMx1/KxWxepNzwEOxTdSAK4GlqnqPm88pvu2yScBcT9RngHWqurjfscTgdnAbd7r23B1r74TkfberypEpBmuPngt7gs2wZstkeJtISKtKl/j6lq/IEH3r6e62GYD3/DuCrkAyA+oJoiLGr5PCRdzPT6rge9hAvCuepXtcTaFU1VBEOt968dFjzhfYBmFqxNeCSz3hmv8jiskxunAHqAUV6/3LVxd5AJgIzAfyPU7Ti/Wc4HPvf35BfCgV94D+BTYBLwEZPoda0BcK7xhNfAzrzwh9m8k/3vcXSBP4uq1VwHDfYg37PcpEWOO9LMKZHnjm7zpPXzYvy1wZx85AWUx3bf2xLAxxiSxJl8dZIwxpnqWBIwxJolZEjDGmCRmScAYY5KYJQFjjElilgRMwhKR7iJyTERS/Y6locS1ZHqF33EEEpH3ReTb3uuvicjbfsdk4s+SgElYqrpdVVuqe/TfxJCqvqCqYyvHRURFpJefMZn4sCRgjDFJzJKAiTuvauReryOM4yLyjIh0FJE3RaRQROaLSBsROcv7RZrmLfe+iDwiIh95870tIu1q2VaWiPxVRA6JyFERWSIiHb1p3xSRtd66vhSR7wUsN1pEdorIfeI6fdkjIjeIyDUiskFEDovITwPmnyYiL4vrlKRQRJaJyKBqYkoRkZ+IyGYvrlkikltbvDW8x6le/IUiskVEvhZQ/pGI/F5E8kVknYhcXsM6/uG9XugVr/Cq4ybVtH3TuFkSMH65CdeWSx/gq7jOMX4KtMd9Lu+qZrlbgG8CHYAM4Me1bOc2XENg3XCP398BnPCm7Qe+AmR76/yNBPfO1AnXlEAX4EHgaeDrwDDgYuABEckLmP96XLMDucCLwOteY2uhfgjcAFwKdAaO4B7/ry3e03jtIf0OuFpVWwEX4ZpyqHQ+rlmBdsBDwKuVCac6qnqJ93KQVx03s6b5TeNmScD45b9VdZ+6Fj4/xPWK9LmqnsS19DmkmuWeVdUNqnoC10PU4Fq2U4o7mPZS1XJV/Uy9psRVdY6qblbnA+Bt3ME9cNlHVbUUmIE7kD6hqoWquhpYAwT+2v9MVV/25n8cl0AuCBPTHbg2jHaqajEwDZjgnfFUG28NKoBzRKSZqu7xYqu0H/itqpZ6B/P1wLW1rM8kEUsCxi/7Al6fCDPesprl9ga8LqphvkrP43qMmiEiu0Xkvyp/nYvI1SLyiVe1cxTXEFpg9dKhgIvSlb/Ga4qzqoMPVa3ANQjXOUxMZwKvedU9R3EtW5bjOgupNt5wVPU4MAmXWPaIyBwRCWwzf5cGNxC2rZqYTJKyJGCaNO8X8M9VtT+uquQruOZ3M3Ft4j+G67mpNTAX1zJjfVW17S4iKbj23XeHmW8HrvqmdcCQpaq7qou3lvc4T1WvxHU9uA5XbVWpi9f8c6Xu1cRkkpQlAdOkicgYERnoPWtQgKtuqcBdT8gEDgBlInI1rq+BhhgmIuO9ap17gGLgkzDzPQU8KiJnejG2F5Hra4m3uvfXUUSu964NFAPHQubvANwlIukicjNwNi7Z1WYfrsll08RZEjBNXSfgZdwBdS3wAfC8qhbiLj7Pwl2YvQXXSUdD/A1XNXMEuBUY710fCPWEt623RaQQlyjOryneGraZAvwz7tf9YdzF5jsDpi8GeuM6TX8UmKCqdektaxrwv16V1cQ6zG8aKetPwJgoEJFpuIu5X/c7lkoiMhX4tqqO8jsWk7jsTMAYY5KYJQHT6Ilr9+ZYmGF17Us3DtW8v2MicnHtSxtTPasOMsaYJGZnAsYYk8QsCRhjTBKzJGCMMUnMkoAxxiQxSwLGGJPELAkYY0wS+3+44gDiv0LeuwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracy_list_train = []\n",
    "accuracy_list_val = []\n",
    "for min_samples_split in min_samples_split_list:\n",
    "    # You can fit the model at the same time you define it, because the fit function returns the fitted estimator.\n",
    "    model = RandomForestClassifier(min_samples_split = min_samples_split,\n",
    "                                   random_state = RANDOM_STATE).fit(X_train,y_train) \n",
    "    predictions_train = model.predict(X_train) ## The predicted values for the train dataset\n",
    "    predictions_val = model.predict(X_val) ## The predicted values for the test dataset\n",
    "    accuracy_train = accuracy_score(predictions_train,y_train)\n",
    "    accuracy_val = accuracy_score(predictions_val,y_val)\n",
    "    accuracy_list_train.append(accuracy_train)\n",
    "    accuracy_list_val.append(accuracy_val)\n",
    "\n",
    "plt.title('Train x Validation metrics')\n",
    "plt.xlabel('min_samples_split')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xticks(ticks = range(len(min_samples_split_list )),labels=min_samples_split_list) \n",
    "plt.plot(accuracy_list_train)\n",
    "plt.plot(accuracy_list_val)\n",
    "plt.legend(['Train','Validation'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that, even though the validation accuraty reaches is the same both at `min_samples_split = 2` and `min_samples_split = 10`, in the latter the difference in training and validation set reduces, showing less overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f45b6ae5550>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEXCAYAAABGeIg9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA4sElEQVR4nO3deXwV1fnH8c83gbATQEQRUBYVRSyLgisCCoiioGhV3ECtv2rrVrUqahVxX2o3rW3dUEvFXQFBFATB1gVkFRFFRAjIIkjYl5Dn98dMyL03IeRCcifL83697it3zpyZeW4g89w5c+YcmRnOOedcrLSoA3DOOVf2eHJwzjlXgCcH55xzBXhycM45V4AnB+eccwV4cnDOOVeAJwdXoiR1k2SSmkYdy56SNEnSMzHLwySN3802gyTllNDxTdLFJbGv8kBS8/Aznxh1LC6fJ4dKKvxjLOq1aA93/T+gMbCs5KLdPUlnh3G32cX6v0v6QdKe/J+/Hvjl3kVYaEzjJQ0rZFVj4PWSPl4qFfHZCrOE4DN/VnoRuWR5cqi8Gse8zgnLOsaUdYqtLCmjODs1s21mttzMcksw1uIYBfwIXJm4QlJN4ELg2T2Jy8yyzeznvQ+x2MdbbmZbUnW8KEnKMLMd4WfeHnU8Lp8nh0oq/GNcbmbLgTVh8aqYspWSrpP0H0nZwEsAku6XNE/SJklLJP1DUmbefhOblWKWe0qaHG73laTTdhWbAu9KmiqpaliWFn4bnSwpvZDPkwM8B1wiqVrC6vOA2sCzkjpKGitppaQN4TF6F/W7SmxWCmO5N2YfrwD1E7ZpIelNScvCzzxH0iWx+wROAQbGXK11C9fFNStJaixphKS1kjaHzV5HF/I7L/bvONxuiKQFks6T9G243duS6krqL2m+pPWSXo/9Nw63vUDSTElbJC2S9LikWkV9tpjmo4skjZG0Ebi3sGYlSY0kPS9pRXiM+ZIuD9dVDY+XJWmrpB8ljSjqs7o9YGb+quQvoBtgQNOYMgNWA9cArYBDwvI7gS5Ac4ITwNfAC7vaV8zyLKA3cAjwPLAOqF9ETPsSNE09Fi7fEcbTrIhtmgM7gAEJ5f8F3omJZxBwBHAocB+wDTg0pv4k4JmY5WHA+Jjl64GNwMBwH7cAa4GcmDpHhr+7duHv71ogB+gers8EJgOvAPuHr4yY3/3F4XsRNLfMBE4M9/sK8DPQcC9/x0PCz/Eu8AugK7AKeB8YE8Z+IrACeDhmu0Hh8S8BWgInAbOBl4r6bOG/jwFZwEVAi/CVV35iuH0NYB4wHegRHqMXcEG4/sZwH92AAwmucm+I+u+oor0iD8Bf0b/YdXJ4thjbng1sBdIK21fMcv+YbfYLy07dzb67hyfUu4HtwFnFiGcs8GHM8uHhsfoUsc0s4I6Y5UkUnRyygPsT9vE6MclhF8d5B3g6Znk8MKyQerHJ4ZRwuU3M+moETWh37c3vmCA55BAmmbDsSYIEu29M2V+AaTHLi4CrEvZ1Uni8+rv6bDFJ4A+7KM9LDlcAW2L/PybU/wvwIaCo/3Yq8sublVxRPk8sCJsbJofNJRuA4QTfCvffzb5m5r0xsxUEJ6D9itrAzCYCfyQ4iT1jZm8XI+Z/Ad0kHRwuXwksJkgaSNpXwc3pr8Nmmg0EVxEHFWPfSKoLNCG48R7r44R6NSU9JGmupDXhcU4v7nFiHAGsNrOv8grMbCvB1cQRCXVnxtQp1u8YWGpmP8UsLweWm9mqhLJGEPz+ws/weNiktiH8bGPDugezewX+XyU4CvjKzLJ2sf55giuoBWGz5jkq5j0xV3yeHFxRNsYuSDoGeI2gyeBsghvYV4Wrd/fHua2QsiL//4X3Fk4gOMm1kqRixDyK4GR2ZXjCuJT4G9HDCJrFbgl/tic4qZb0yeVR4GLgHoIroPYETTWleRJL+ndMcEUWy3ZRlrefvJ/XE3ymvFc7guasOcWIc+Puq+yamc0kaI66meAz/wWYGSZuV0I8ObhknAj8ZGZ3mtlnZvYNUJrPMwwh+CZ6AtCZ4IReJMu/MT2I4EZ0PeDZmConAX83s5FmNoegeaZlcQMys3XAUuD4hFUnJCyfBAw3s1fNbBawkOD+RKxtQIGb6wnmAvsopotueMP9GODL4sZdUsIrkiVAazNbUMgrr5dVcT7brnwBtFERz8qY2QYze8vMrgOOJmg+7LqHx3OF8OTgkjEf2FfSFZJaSroU+E1pHEhSV+A2YKCZfQb8H0HPls7F2PwZoCHwV+BdM1sas24+cJGkIyW1B14m+ZPYH4HrJV0i6RBJNxHcOI01H+gnqXN4Yv8XcEBCne+BoyS1ktRQYc+sBB8SNMP8R9IJktoCLwLVgaeSjLuk3AFcJ+kOSW0ltZZ0lqR/xtQpzmfblZeBH4CRknqEPb9OkXQ+gKTfhz2ejpDUAric4OrymxL6fA5PDi4JZjYauB94gKD54ALg9yV9HEkNgH8DfzGzceGxXyVoEnpZUp3dxLmIoMdNfYKTcqzLCP7ffw68DbwHTE0yxL8QJJ4/ETRJHQcMTajzO4IT3ERgAsHVRuKDbX8EfiK4Ib6KglcfWHAH9iyCXmHvhrHuD/RMuFeQMmb2EsFV2RkEv8epBFd5sUl4t5+tiP1vIrgK+BIYQdBz6UmCXkwQ9MK6EfiE4P/h2cA5ZjZ/Tz+TK0jB/z3nnHMun185OOecK8CTg3POuQI8OTjnnCvAk4NzzrkCqkQdQEnIzs72u+rOObcXMjMz4x4y9SsH55xzBXhycM45V0CFaFaKlZmZuftKzjnnyM7O3uU6v3JwzjlXgCcH55xzBVS4ZqVYZsaaNWvIzU31dMYVV1paGg0aNKB4o2c758qrlCQHSc8RDNK10szaFrJeBIOZnQ5sAgaZ2fRw3UCCqSkB7jOzF4p73DVr1lCrVi2qV6++tx/BhbZs2cKaNWvYZ599og7FOVeKUnXlMAx4gmCo4cKcRjBRyCEE49Q/BRwTjs55N8F47QZ8IWmkmf1cnIPm5uZ6Yihh1atXZ/369VGHUW6s2QKzV0OuP4njSsmBdeDgUuiHk5LkYGaTJTUvoko/4MVweOJPJdWT1JhgbtwPzGwNgKQPCCZQf7mUQ3Zur439AS4YD+sKm5/NuRJyawd46NiS329ZuSHdhGB2qTxZYdmuysuF1atX0759e9q3b8/+++9PkyZNdi5v21b0GWPatGlcd911KYrUlbRXF0Df9zwxuPKrQt+Qjto+++zDzJkzARgyZAi1a9fm5ptv3rk+JyeHKlUK/yc4+uijOfroo1MRpithz82DKz/ypiRXvpWV5LAUaBaz3DQsW0rQtBRbPmlPD6ISnlTRrk5+m0GDBlG9enVmzJjBCSecwAUXXMD111/Pli1bqFGjBs8//zytW7dm0qRJPPbYY4wePZohQ4awePFiFi5cyOLFi7nhhhv8qqKM+vMs+N3/4ssEdGkM6d7By5WCVnVLZ79lJTmMBK6RNILghnS2mf0oaRzwgKT6Yb1ewOCogiwpWVlZ/O9//yM9PZ1169YxZcoUqlSpwvjx47n99tt54403Cmzz9ddfM3HiRNavX0/r1q25+uqrqVo1mWl5XWkyg3u/gLsTJhxNFww7GS4+NJq4nNtTqerK+jLBFUBDSVkEPZCqApjZP4AxBN1YFxB0Zb0sXLdG0r3kz/E7NO/mdHn2y1/+kvT0YE777OxsBg4cyLfffosktm/fXug2ffr0oVq1alSrVo1GjRqxYsUKmjZtmsqw3S6Ywc2fwOOz4ssz0uDVXtCvRTRxObc3UtVbacBu1hvw212sew54rjTiikqtWrV2vv/DH/5A9+7deeutt1i0aBHdunUrdJtq1artfJ+enk5OTk5ph+mKYUcuXDUZnpkXX16zCrxzGvTw/O3KqbLSrJQSe3KPoLRlZ2fTpEnQAWvYsGHRBuOSsn0HXPIhvLIgvjwzA8b0geP3jyYu50pCWenKWmndcsstDB48mA4dOvjVQDmyOQfOHlcwMexbHSb188Tgyj8FLTrlW+xMcLFDdq9atYp99903kpgqssr+e12/DfqOhUnL4sub1oLxZ0Lr+oVv51xZEztkd+JMcJWqWcm5vbVmC5z2Lny+Mr68VV2Y0BcOqhNNXM6VNE8OzhXT8k3QcxR8mdBfrm0DeP8MaFyr8O2cK488OThXDD+shx6jYEHCxFmdGsF7faCBj+/oKhhPDs7txvyfg8SQtTG+vOsBMOo0qJMRTVzOlSZPDs4VYdZPQVPSqi3x5X0Ogtd6QQ3/C3IVlHdldW4XPlkO3d4pmBjOPxjePNUTg6vYPDmUou7duzNu3Li4sj//+c9cfXXhT+N169aNadOmAXD66aezdu3aAnWGDBnCY489VuRx3377bb766qudy3fddRfjx49PMvrKbUJWcMWwNmHI7V8dDsNPgYz0aOJyLlU8OZSiAQMGMGLEiLiyESNGMGBAkaOJADBmzBjq1au3R8dNTA5Dhw6lR48ee7Svyuid7+H0d2FjwjOJv/sF/KsrpPtfjasEKteF8RMlPGbyNUU/QHjuuedy5513sm3bNjIyMli0aBHLli3j5Zdf5sYbb2Tz5s2ce+653HPPPQW2bd68OdOmTaNhw4bcf//9vPDCCzRq1IhmzZpx1FFHAfD000/zr3/9i23btnHwwQfz0ksvMXPmTEaOHMlHH33EfffdxxtvvMG9997LGWecwbnnnsuECRO4+eabycnJoVOnTjz11FNUq1aN5s2bM3DgQEaNGsX27dt57bXXOOyww0r291UODP8GBn4IOxL+aYccDXcdDfJht10l4d+BSlGDBg3o3LkzY8eOBYKrhvPOO4/777+fadOmMXv2bD766CNmz569y3188cUXjBgxgpkzZzJmzBimTs0fE7p///5MnTqVWbNmcfjhh/Pss89y/PHH07dvXx599FFmzpxJq1atdtbfsmULgwYN4pVXXmHOnDnk5OTw1FP5k1w0bNiQ6dOnc/XVV++26aoi+sdcuGRCwcTw+PFwdydPDK5y8eRQymKblvKalF599VU6duxIhw4dmDt3blwTUKIpU6Zw9tlnU7NmTerWrUvfvn13rvvyyy/p0qULRx55JMOHD2fu3LlFxjJ//nxatGjBoYcGkwsMHDiQyZMn71zfv39/AI466igWLVq0px+5XHpkBlw9GWLzgoBnusHv2kUUlHMR8uRQyvr168eECROYPn06mzZtokGDBjz22GNMmDCB2bNn06dPH7Zs2bL7HRVi0KBBPPHEE8yZM4e77757j/eTJ29Y8Mo0JLgZ3PkZ3PppfHmVNHi5J1xxeDRxORe1ynXPYTf3CEpD7dq16d69O5dffjkDBgxg3bp11KpVi8zMTFasWMHYsWN3OYcDwEknncSgQYMYPHgwOTk5jBo1il//+tcArF+/nsaNG7N9+3aGDx++c+jvOnXqsH79+gL7at26NYsWLWLBggU771F07dq1VD53eZBrcMN/4W9z4surp8PrpwbPMjhXWaXsykFSb0nzJS2QdFsh6w+SNEHSbEmTJDWNWfeIpLmS5kn6q1S+Wn8HDBjArFmzGDBgAO3ataNDhw4cdthhXHjhhZxwwglFbtuxY0fOP/982rVrx2mnnUanTp12rrv33ns55phjOOGEE+JuHl9wwQU8+uijdOjQge+++25nefXq1Xn++ef55S9/yZFHHklaWhpXXXVVyX/gciAnFy6fWDAx1K4KY/t4YnAuJUN2S0oHvgF6AlkE034OMLOvYuq8Bow2sxcknQxcZmaXSDoeeBQ4Kaz6MTDYzCblbetDdqdWef+9bt0BF42HNxbGl9evFoyT1Hm/aOJyLtWKGrI7VVcOnYEFZrbQzLYBI4B+CXXaAB+G7yfGrDegOpABVCOYe3pFqUfsKqRN26Hf2IKJYf+aMLmfJwbn8qQqOTQBlsQsZ4VlsWYB/cP3ZwN1JO1jZp8QJIsfw9c4M0uYsde53cveCqeOhnFL4ssPqgNTzoK2+0QSlnNlUlnqrXQz0FXSDKArsBTYIelg4HCgKUFCOVlSl+jCdOXRqs1w8kj4eHl8eet6QWI4OLOwrZyrvFLVW2kp0CxmuWlYtpOZLSO8cpBUGzjHzNZKuhL41Mw2hOvGAscBU1IRuCv/lm6AnqNh3s/x5e0bwrg+0KhmNHE5V5al6sphKnCIpBaSMoALgJGxFSQ1lJQXz2DgufD9YoIriiqSqhJcVRSrWSktLW2v+/67eFu2bCEtrSxdcBZt4Tro8nbBxHDcfjCxrycG53YlJVcOZpYj6RpgHJAOPGdmcyUNBaaZ2UigG/CgJAMmA78NN38dOBmYQ3Bz+j0zG1Wc4zZo0IA1a9YU2uff7Zm0tDQaNGgQdRjF8tWaYJKeHzfFl/doCm/1DrqtOucKl5KurKVtV11ZXeX1xarg5vPqhAvHfs1hRE+oXrke/3SuUEV1ZfU/EVfhTFkGfcbA+u3x5RcfCs91g6o+F4Nzu1V+Go+dK4b3FsOp7xZMDFcfAS+c7InBueLy5OAqjDe+g75jYXPCmIG3doAnu0BauRp0xbloebOSqxCGfQ1XTAoG04v1wDEwuGMUETlXvnlycOXe3+bAdR8XLH+iC/y2berjca4i8OTgyi0zeGA63Pl5fHma4PnucGnraOJyriLw5ODKJbNggp5HZ8aXV00Luqr2bxlJWM5VGJ4cXLmzIxd+OwX+mTC7ao0q8HZv6NWs8O2cc8XnycGVK9t3wKCJ8J9v48vrZsC7p8OJjaOJy7mKxpODKze25MD5H8DIRfHlDavDuDOgY/mdf8i5MseTgysXNoST9Hy4NL78gFow/kw4vH40cTlXUXlycGXez1vh9Hfh04T5/1rWDRJDi7rRxOVcRebJwZVpKzZBr9Ewe3V8eZv68MGZwZWDc67keXJwZdbi9dBzFHyTHV9+1L7wXh9oWCOauJyrDDw5uDLp27VwyihYsiG+/KTGMOr0oHeSc670eHJwZc7s1dBrFKzYHF/euxm8cSrU9El6nCt1KRuVVVJvSfMlLZB0WyHrD5I0QdJsSZMkNY1Zd6Ck9yXNk/SVpOapitul1rKNcMrIgonh3JbwzmmeGJxLlZQkB0npwJPAaUAbYICkNgnVHgNeNLNfAEOBB2PWvQg8amaHA52BlaUftYvC3VPhp4TZ2y47DF7uCRk+F4NzKZOqK4fOwAIzW2hm24ARQL+EOm2AD8P3E/PWh0mkipl9AGBmG8wsYVZgVxHM+xme+zq+7Jq28Ew3qOIzjziXUqn6k2sCLIlZzgrLYs0C+ofvzwbqSNoHOBRYK+lNSTMkPRpeibgK5vbP4udjODQTHj/eJ+lxLgpl6fvYzUBXSTOArsBSYAfBTfMu4fpOQEtgUEQxulLyv+Xw9vfxZQ8e69N6OheVVCWHpUDsWJlNw7KdzGyZmfU3sw7AHWHZWoKrjJlhk1QO8Dbgc3tVIGZwyyfxZcc0grNbRBOPcy51yWEqcIikFpIygAuAkbEVJDWUlBfPYOC5mG3rScobVu1kIGGwZleejVoE/10eX/bwcSBvTnIuMilJDuE3/muAccA84FUzmytpqKS+YbVuwHxJ3wD7AfeH2+4gaFKaIGkOIODpVMTtSl9OLgz+LL6sz0HQ9YBo4nHOBWRmu69VxmVnZ+/8EJmZmVGG4pL03Dy4YlL+soBZ58GR+0QUkHOVSHZ2/tg0mZmZcdfqZemGtKtkNufAXVPjyy5t7YnBubLAk4OLzF/nwNKN+cvV0mFop+jicc7l8+TgIrFmCzw4Pb7s2rZwYJ1o4nHOxfPk4CLx4HTI3pa/nJkBg72DsnNlhicHl3KL18PfvowvG9wRGlSPJh7nXEGeHFzK3T0Vtu7IX25SC647Mrp4nHMFeXJwKTVnNbwwP75saCeo4TOLOFemeHJwKTX4M4h9sqZN/aD7qnOubPHk4FLmo2Xw7g/xZQ8e48NxO1cW+Z+lSwkzuDVhcL0T9oczm0cSjnNuNzw5uJR463v4LGH+voeP9cH1nCurPDm4UpeTC4M/jS/r1xxOaBxJOM65YvDk4Erds/Pgm/zxvUhTMJGPc67s8uTgStXG7TBkWnzZ5YfB4fWjicc5VzyeHFyp+vNsWL4pf7l6Ogw5Orp4nHPF48nBlZqfNsPDM+LLbvgFNKkdTTzOueJLWXKQ1FvSfEkLJN1WyPqDJE2QNFvSJElNE9bXlZQl6YlUxez2zv3TYf32/OUG1eDWDtHF45wrvpQkB0npwJPAaUAbYICkNgnVHgNeNLNfAEOBBxPW3wtMLu1YXcn4fh08mTC43h1HQb1q0cTjnEtOqq4cOgMLzGyhmW0DRgD9Euq0AT4M30+MXS/pKIJ5pd9PQayuBPzhc9iem798YG34zRHRxeOcS06xk4OktySdJanqHhynCbAkZjkrLIs1C+gfvj8bqCNpH0lpwB+Bm/fguC4CM1bB8G/jy+7tDNV9cD3nyo1krhymAHcByyU9Jen4Eo7lZqCrpBlAV2ApsAP4DTDGzLJK+HiulAz+LH75yAZw0SHRxOKc2zPF/i5nZo8Dj0s6ArgYeFnSNuAlYLiZfVfE5kuBZjHLTcOy2P0vI7xykFQbOMfM1ko6Dugi6TdAbSBD0gYzK3BT20VvQhaMWxJf9tCxkO794pwrV2Rmu69V2IZSF+AJoC2wAZgK3GRmswqpWwX4BjiFIClMBS40s7kxdRoCa8wsV9L9wA4zuythP4OAo83smtjy7OzsnR8iMzNzjz6P23u5Bp3fgC9W5Zd1PQAm9vUxlJwri7Kz84cuyMzMjPsrTer7nKTWku6V9B3wL+AVoDnBzeIxwNuFbWdmOcA1wDhgHvCqmc2VNFRS37BaN2C+pG/C/d2fTGwueq99F58YAB7xwfWcK5eKfeUgaRpBIniFoMvpZ4XU+d7MWpRohMXgVw7R27YD2oyA79bll53bEl47NbqYnHNFK+rKIZn+Iw8BI8OuqIWKIjG4suHpefGJIV1w/zHRxeOc2zvJNCutI7hy2ClsZupZohG5cmf9NhiaMLjelYfDofUiCcc5VwKSSQ5PAusTytaH5a4Se3wWrNycv1yzCtzlg+s5V64lkxwamdmPCWU/AvuXYDyunFmxCR5L6J92UztoXCuaeJxzJSOZ5LBQ0skJZd2A70suHFfe3PsFbIgZXK9hdbi5fWThOOdKSDI3pIcAb0p6FvgOaAVcFr5cJbQgG/75VXzZH46CuhnRxOOcKznFvnIws3eAXkAtoE/489Sw3FVCd34ezA+dp0Ud+LUPrudchZDUUGhm9jnweSnF4sqRaSvhlQXxZfd1hmrp0cTjnCtZSSUHSe2BLkBDYOcDE4nDXLiKzQxu/TS+rENDuMAH13OuwkhmyO7/A/4LnAzcChwJ3AQcXDqhubLq/SXw4dL4soePhTQfJsO5CiOZ3kq3AL3N7Gxgc/jzXGB70Zu5iiS3kKuGHk2hZ7PC6zvnyqdkn3OYEr7PlZRmZmOBM0shLldGvfwtzFodX/bQsdHE4pwrPcncc8iS1NzMFhEMv91P0k/ALsdachXL1h1BD6VYFxwMR+0bTTzOudKTTHJ4BDgcWAQMBV4HMoDrSj4sVxb9Yy4sihlApWpa0EPJOVfxFCs5SBIwGVgMYGZjJdUHMsxsQynG58qI7K3B09CxrjoCWvkI6c5VSMW652DBpA9zgNyYsm2eGCqPR2fC6i35y7Wrwp0dIwvHOVfKkrkhPQM4dE8PJKm3pPmSFkgqMP+zpIMkTZA0W9IkSU3D8vaSPpE0N1x3/p7G4PbMjxvh8dnxZb9vD41qRhKOcy4FkrnnMAl4T9IwYAmwc/Y1M3uuqA0lpRMM7d0TyAKmShppZrEj8zxGMMPcC+EAfw8ClwCbgEvN7FtJBwBfSBpnZmuTiN3thXumweac/OVGNeDGdtHF45wrfckkhxMIRmDtmlBuQJHJAegMLDCzhQCSRgD9gNjk0Aa4MXw/kXA+ajP7ZueBzJZJWgnsC6xNIna3h+b/DM/Miy+7++igWck5V3EVOzmYWfe9OE4TgquNPFlA4iSSs4D+wF+As4E6kvYxs5296iV1Jugh9d1exOKScMfnsCNmmvGDM4NZ3pxzFVsyw2ek7epVQrHcDHSVNIPg6mQpsCPm+I2Bl4DLzCy38F24kvTpcnhjYXzZA8dAVR9cz7kKL5lmpRxi7jMk2N3pYikQO8BC07BsJzNbRnDlgKTawDl59xUk1QXeBe4ws4TBG1xpMINbEn7TnRrBuS2jicc5l1rJJIcWCcuNgduAUcXYdipwiKQWBEnhAuDC2AqSGgJrwquCwYT3MSRlAG8R3Kx+PYl43V4YsximJEwK+/CxIB9cz7lKIZnJfn5IeH0KDCQYoXV32+YA1wDjgHnAq2Y2V9JQSX3Dat2A+ZK+AfYD7g/LzwNOAgZJmhm+2hc3bpe8HblwW8JVQ+9m0L1JNPE451JPwfNte7ix1AyYbWb1Sy6k5GVnZ+/8EJmZ/sju3nrhaxg0MX9ZwIxfQruGkYXknCsF2dnZO99nZmbGtQsUu1lJ0kvE33OoSfCN/t97GZ8rQ7bkwB+mxpddfKgnBucqm2TuOSRMCslG4B9mNr4E43ERe+JLWBIzKEpGGgztFF08zrloJPOcwz2lGYiL3s9b4YHp8WW/bQvN60YTj3MuOsk85/BXSccnlB0v6c8lHpWLxMMzggSRp24G3O6D6zlXKSXzANsAYFpC2RckdEl15VPWBvhLwuB6t7aHhjUiCcc5F7FkkoMVUj89yX24MmrIVNiyI3+5cU24/hfRxeOci1YyJ/YpwH15w2WEP4eE5a4c+2oNPD8/vuyeTlDLB9dzrtJKprfS9cBo4EdJPwAHAj8CZ5ZGYC51Bn8GuTGdlFvXg8sOiywc51wZkExvpSxJHQmG325GMMrq5z4IXvn28Y8wclF82YPHQBVvLHSuUkvmIbj2wOpw2IxPw7JmkhqY2axSis+VIjO4NWGYjGP3g7MSR9FyzlU6yXw//DeQ2AqdQTCMtiuHRi6C/y2PL3vEB9dzzpFccjgwbya3PGb2HdC8RCNyKZGTG9xriHXmQdDlgGjicc6VLckkh7x7DjuFy8tKNiSXCsO+hnk/5y+nKZjIxznnILneSn8C3pH0CME0na0IZm+7v8itXJmzaTvcnfA448DW0HafaOJxzpU9yfRWelrSWuAKgt5Ki4GbfAKe8uevc2DZxvzlaunBcw3OOZcnmSsHgMnAViBvAOe6ki43s+dKNixXWlZvgYdmxJdddyQ0qx1NPM65simZrqxnEfRMWgAcAcwF2gIfE07p6cq+B6dD9rb85XoZcFuH6OJxzpVNydyQvg+43Mw6ABvDn/9HMPjebknqLWm+pAWSbitk/UGSJkiaLWmSpKYx6wZK+jZ8DUwiZhfjh/XwtznxZbd3hAbVo4nHOVd2FXuaUEnrzKxu+P5nM6sfjq+03Mwa7WbbdOAboCeQBUwFBpjZVzF1XgNGm9kLkk4GLjOzSyQ1IBgN9miCwf++AI4ys519bXya0OIZOAFe/CZ/uWkt+OZCqJFs46JzrkIoaprQZK4cVkraL3y/SNJxBD2W0ouxbWdggZktNLNtwAigX0KdNsCH4fuJMetPBT4wszVhQvgA6J1E3A6YvRpe+ia+bGhnTwzOucIlkxyeBk4M3/+J4AQ+C/h7MbZtQjAWU56ssCzWLKB/+P5soI6kfYq5rduNwZ/GTwB+RH249NDIwnHOlXHJdGV9OOb9i5ImAbXMbF4JxXIz8ISkQQS9opYCO4rcwhXLpKUwZnF82YPHQroPruec24U9blQws8W7r7XTUoJnI/I0Dcti97eM8MpBUm3gHDNbK2kp0C1h20l7EHKlVNjgel0awxkHRROPc658SNV3x6nAIZJaSMoALgBGxlaQ1DBvIiFgMPndY8cBvSTVl1Qf6BWWuWJ4YyF8vjK+7GEfXM85txspSQ5mlgNcQ3BSnwe8amZzJQ2V1Des1g2YL+kbYD/CYTnMbA1wL0GCmQoMDcvcbmzfAbcnDK53dgs4bv9o4nHOlR/F7spalnlX1sL9Yy5cPTl/OU0w93w4rH50MTnnyo6S6srqypEN22HI1PiyKw7zxOCcKx5PDhXUn2fDis35yzWqwBAfXM85V0yeHCqgVZvhkYTB9X73CzigVjTxOOfKH38+tgK67wtYvz1/uUE1uKV9ZOFUDts3wbrvYd1CyP4u/LkQ1n0Hm1ZEHZ2ryNpdD53vLvHdenKoYBaug6fmxpfdeRRkVosmngrDLDjJr1uYnwDyTv7ZC2HTj1FH6CqrnM27r7MHPDlUMH/4HLbn5i8fVAd+0za6eMqVHVth3aKEk3/MlUDOpqgjdC5lPDlUIDNWwX++jS+7t1Mw05sj+Pa/ZXV8k092zMl/QxbxI1A5V3l5cqhAbkt44K3dPnBRZRtcb8d22LA4/uQf++1/27rSOa7SoPaBkNkS6raEzFbBz7otoc6BkOZ/aq6UpJfOhCz+P7aCGJ8F7y+JL3vo2ODBtwpna3bMTd/v4u8DrF8MVkrjNVatnX/Sjz35Z7YKEkB6Rukc17kIeHKoAFZugus+ji/rfgCc2qzw+mVe7o6giWddwrf+vPsAW0tx9JTaTeNP+nk/M1tC9YY+KJWrNDw5lHNLNkDPUTB/bXz5w8eV8fPYtg27PvmvXwS523e7iz1SpUYhJ/+8b//NoYrPmeoceHIo1xZkQ49RwdzQsS46BDoVOXFrClgubPyxYL//vPebV+5+H3uq5v6Fn/zrtgzWlems6VzZ4MmhnPpyNfQcDcsTelf2agb/7JqiIHI2Bw9+FdbzZ933sGNL6Rw3LQPqtghv/raKP/nXbQFV/VFw5/aWJ4dyaOpK6D0a1myNL+/fAv7TswS7rpoF3/B31fNn47ISOlAhqu8TnvBbFewBVLtJ0DvIOVdqPDmUM5OWwpljg1FXY116KDzbHaoke87csRXW/VCw7T/vfsD2jSUWe5y0KlDnoMJP/pktIaNu6RzXOVcsnhzKkTE/wDnjYEtCT83ftoW/nriLbqtmsGVN4d0+1y2E9UsotQe/MjLDnj6tCp78azfzvv/OlWEp++uU1Bv4C5AOPGNmDyWsPxB4AagX1rnNzMZIqgo8A3QM433RzB5MVdxlxSsL4OIJkJMbX357R7ivc3iPdeUXsHJawSeAt2UXus+9prTgJJ/Y5z/vZ3WfPMK58iolyUFSOvAk0BPIAqZKGmlmX8VUu5Ng+tCnJLUBxgDNgV8C1czsSEk1ga8kvWxmi1IRe1nw7Dy4clLB7/cPHQu3diAYEfSDi2Dh2yV/8Kq1C57087791znIH/xyroJK1ZVDZ2CBmS0EkDQC6AfEJgcD8hqaM4FlMeW1JFUBagDbgFIaA6Hs+dMsuPF/8WUCnuwCV7cleFr43TNh2ZQ9P0itJvEn/dgeQP7gl3OVUqqSQxMgdnCHLOCYhDpDgPclXQvUAnqE5a8TJJIfgZrA78ysFB+RLRvMYOg0GDItvjxdMOxkuPhQYPNPMPJUWDW96J3FPfjVMv4mcN0W/uCXc66AsnRHcAAwzMz+KOk44CVJbQmuOnYABwD1gSmSxuddhVREZnDzJ/D4rPjyjDR4tRf0awFsWArv9ISf58VXatAGWp0b3wzkD34555KUquSwFIgd6adpWBbrCqA3gJl9Iqk60BC4EHjPzLYDKyX9FzgaqJDJYUcuXDUZnkk459esAu+cBj2aEtxkfqdH8KBZrP2PgzPe9RvBzrm9lqoniaYCh0hqISkDuAAYmVBnMXAKgKTDgerAqrD85LC8FnAs8HWK4k6pbTvgogkFE0NmBnxwZpgYVs+FN04smBia9YC+73ticM6ViJQkBzPLAa4BxgHzCHolzZU0VFLfsNpNwJWSZgEvA4PMzAh6OdWWNJcgyTxvZrNTEXcqbc6B/uOCLqux9q0Ok/rB8fsDK6bCmycVnJKy5VnQZxRk1E5VuM65Ck7B+bd8y87O3vkhMjMzowxlj6zfFjz1/FHCaBRNawVXDIfVB7ImBb2Stm+Ir9T6EjjlOX+gzDmXtOzs/GegMjMz425M+hklYmu2wGnvwucJg5S2qgsT+gZzQLPoXRh7bsGB7I78LZz0Vx9nyDlX4jw5ROjHjdBrNHyZ0DG3bQN4/wxoXAv49hX44GLIzYmvdNTtcOx93gvJOVcqPDlE5If1wVwMCxJGtujUCN7rAw2qA3Ofhom/psCz0cc/DB1vSVWozrlKyJNDBOb/HCSGrIQBT7seAKNOgzoZwIw/wn9vTthS0O3v0PaqVIXqnKukPDmk2MyfoNcoWJVw+6DPQfBaL6iRbvDZ3TD13vgKSoceL0LrC1MXrHOu0vLkkEKfLA9uPmdviy8//2B48WTISMuFKTfA7L/FV0ivBr1fgxZnpixW51zl5skhRcZnwVljYWPCfeVfHQ7/OAnSyYEJV8LXw+IrVK0FfUZC05NTFqtzznlySIF3vofz3odtCXMx3NgOHjsOlLsV3r8IvnsjvkK1+sEDEPsnjlHonHOly5NDKRv+DQz8EHYkdDi6pxP84ShQzkYY0x+WvB9foeZ+0PcDaHhk6oJ1zrmQJ4dS9I+58JvJBSfpefx4+F07grkYRveBH/8bX6HOQdBvPNQ7OFWhOudcHE8OpeThGXDbp/FlAp7uBlccDmxaCaN6w6oZ8ZXqtYZ+H0CdZjjnXFQ8OZQwM7jzc3ggYf6dKmnw71OCnklsyIK3e8Da+fGVGraHvuOgZqNUheucc4Xy5FCCcg2u/xie+DK+vHo6vH5q8CwDaxcEczGs/yG+0v7Hw5nvQrV6qQrXOed2yZNDCcnJhV9NghcSLgZqV4XRpwdPP/PTHBjZCzYtj6/UrCec/lbQbdU558oATw4lYOsOuPADeDNh/p0G1WBsH+i8H7DicxjZG7b+HF+pZX849T/Bg27OOVdGeHLYSxu3B5P0vL8kvnz/mvDBGdB2HyBrIrzbt+BcDIcNhJOf8bkYnHNlTsomApDUW9J8SQsk3VbI+gMlTZQ0Q9JsSafHrPuFpE8kzZU0J5xfOnLZW+HU0QUTw0F1YMpZYWL4flQwml5iYvjFtT5Jj3OuzErJmUlSOsF0nz2BLGCqpJFm9lVMtTsJpg99SlIbYAzQXFIV4N/AJWY2S9I+wPZUxF2UVZuh92iY/lN8eet6MP5MaFob+OZlGH9pwbkYjr4TjhnqczE458qsVF05dAYWmNlCM9sGjAD6JdQxoG74PhPImzSzFzDbzGYBmNlqM9uRgph3aekG6PpOwcTQviFM7hcmhi//GQyJkZgYjn8Ujr3XE4NzrkxLVXJoAsQ2vmSFZbGGABdLyiK4arg2LD8UMEnjJE2XFOksNwvXQZe3YV7CfeXj94eJfaFRTWD6IzDpKuKfjRZ0+yd0TJyjwTnnyp6yNPnwAGCYmTUFTgdekpRG0PR1InBR+PNsSadEEeDcNXDiW/D9+vjyHk2DaT3rZRh8cgf879b4CmlVoNdwaPt/qQvWOef2QqqSw1IgdjyIpmFZrCuAVwHM7BOgOtCQ4Cpjspn9ZGabCK4qOpZ6xAm+WBU0Jf24Kb78rBbB/eZaVXJh8nXwxQPxFdKrw2lvwaEDUhesc87tpVQlh6nAIZJaSMoALgBGJtRZDJwCIOlwguSwChgHHCmpZnhzuivwFSk0ZRl0fwdWJ8zedvGhwext1dNyYPxlMOeJ+ApVawdDbrc4I3XBOudcCUhJbyUzy5F0DcGJPh14zszmShoKTDOzkcBNwNOSfkfQWD/IzAz4WdLjBAnGgDFm9m4q4gZ4b3HwHMPmhPvKVx8BT3SBtNytMO4CWPh2fIVqDaDvWNivc6pCdc65EqPg/Fu+ZWdn7/wQmZmZJbbf17+DC8fD9oRJem7rAA8ckzcXw1mwZHx8hZr7ByOr7tO2xGJxzrmSlp2dvfN9ZmZmXBdKfwJrF4Z9DVdMCgbTi/XgMXBbR2DrWhh1Oiz/JL5CneZw1njIbJWSOJ1zrjR4cijEX2fD9f8tWP5kF/hNW4K5GEb2gp9mxVeof1hwxVC7aUridM650uLJIYYZ3D8d/vB5fHma4PnucGlrYP2SYMjttd/EV9q3QzAXQ419Uxavc86VFk8OITO45RN4LOFiICMNRvSEs1sCa78N52JYHF+p8YlwxmioVnL3O5xzLkqeHIAdufDbKfDPhA6yNarA272hVzPgp9nhXAwr4isd2BtOewOq1kxZvM45V9oqfXLYvgMGTYT/fBtfXjcDxpwOJzQGln8aPOm2dW18pVbnBk8+p2ekKlznnEuJSp0ctuTAee/DqIQZOxtWD4bD6LAvsGQCjOkH2zfGVzr8Muj+Lx9y2zlXIVXqM1uVtOAVq0kt+OBMOLw+sPAdeO88yN0WX6nd9XDi46CyNDSVc86VnEp9dquSBi/3hJ5hz9OWdeHjs8LEMH84jD2nYGLodBec+CdPDM65Cq3Sn+GqpcNbveHyw4LZ25rXBeY8BR9cAonTRpzwRzjmHp+LwTlX4VXqZqU8tarCs93DhS8egk8GJ9RQcH/hiF+lOjTnnIuEJ4c8ZvDJ7TD9ofjytCrQczgccl40cTnnXAQ8OQBYLnx0DXz5VHx5evXgGYbmp0cTl3PORcSTw47tMOEy+GZ4fHnVOnDGKGjSNZq4nHMuQpU7OezYCu+dD9+/E19erUEwTtJ+R0cTl3PORaxyJwelF+x5VLNxOBfDEdHE5JxzZUDKurJK6i1pvqQFkm4rZP2BkiZKmiFptqTTC1m/QdLNJRZUWhU4dQQ06xEs120B53zsicE5V+mlJDlISgeeBE4D2gADJLVJqHYn8KqZdSCYY/rvCesfB8aWeHDp1eD0t+GwQdB/CmS2LPFDOOdceZOqZqXOwAIzWwggaQTQD4gdB9WAuuH7TGBZ3gpJZwHfAwkDHJWQqrWgx/OlsmvnnCuPUtWs1ARYErOcFZbFGgJcLCkLGANcCyCpNnArcE/ph+mccw7K1vAZA4BhZtYUOB14SVIaQdL4k5ltiDI455yrTFLVrLQUaBaz3DQsi3UF0BvAzD6RVB1oCBwDnCvpEaAekCtpi5k9UepRO+dcJZWq5DAVOERSC4KkcAFwYUKdxcApwDBJhwPVgVVm1iWvgqQhwAZPDM45V7pS0qxkZjnANcA4YB5Br6S5koZK6htWuwm4UtIs4GVgkJlZKuJzzjkXTxXh/Judnb3zQ2RmZkYZinPOlRvZ2dk732dmZsY9EVzhnpCO/bDOOef2TFnqreScc66M8OTgnHOugApxz8E551zJ8isH55xzBVTq5CCpWTgS7FeS5kq6PuqY9oak9HBU29FRx7I3JP0u/Pf4UtLL4QOR5YKk5yStlPRlQvm1kr4OP9cjUcVXXJKqS/pc0qww5nvC8uHh6Mpfhp+1atSxFoekepJeD/8N5kk6LmbdTZJMUsMoYyxrKnVyAHKAm8ysDXAs8NtCRostT64neI6k3JLUBLgOONrM2gLpBA9NlhfDCJ/0zyOpO8FAk+3M7AjgsQjiStZW4GQzawe0B3pLOhYYDhwGHAnUAH4VWYTJ+QvwnpkdBrQj/DuR1AzoRfAQrotRqZODmf1oZtPD9+sJ/sMkDghYLkhqCvQBnok6lhJQBaghqQpQk5gRess6M5sMrEkovhp4yMy2hnVWpjywJFkgbzyzquHLzGxMuM6AzwmGwinTJGUCJwHPApjZNjNbG67+E3ALwajQLkalTg6xJDUHOgCfRRzKnvozwX/y3Ijj2CtmtpTgm/Vi4Ecg28zejzaqvXYo0EXSZ5I+ktQp6oCKI2ymnAmsBD4ws89i1lUFLgHeiyi8ZLQAVgHPh82uz0iqJakfsNTMZkUcX5nkyYGdw4K/AdxgZuuijidZks4AVprZF1HHsrck1SdogmkBHADUknRxtFHttSpAA4Kmy98Dr0qJ89OWPWa2w8zaE1wddJbUNmb134HJZjYlkuCSUwXoCDwVTia2kWC059uBuyKMq0yr9Mkh/Ab0BjDczN6MOp49dALQV9IiYARwsqR/RxvSHusBfG9mq8xsO/AmcHzEMe2tLODNsDXmc4Kru3Jz8zNsgplIeC9F0t3AvsCNEYaVjCwgK+bK53WCZNECmBX+3TQFpkvaP5oQy55KnRzCb2/PAvPM7PGo49lTZjbYzJqaWXOCm7cfmll5/ba9GDhWUs3w3+cUyvlNduBtoDuApEOBDOCnKAPaHUn7SqoXvq8B9AS+lvQr4FRggJmViyZMM1sOLJHUOiw6BZhuZo3MrHn4d5MFdAzrOirg2EpJOoGg3XRO2LYKcLuZjYkupMrNzD6T9DownaA32QzgX9FGVXySXga6AQ3DWQ3vBp4Dngu7t24DBpaDEYcbAy+E87+nEYykPFpSDvAD8EnYMvammQ2NMM7iuhYYLikDWAhcFnE8ZZ4/Ie2cc66ASt2s5JxzrnCeHJxzzhXgycE551wBnhycc84V4MnBOedcAZ4cnEshSd3CLq6lse/m4eiilb2LuisBnhycK6ckLZLUI+o4XMXkycE551wBnhxchRd+w/69pNmSNkp6VtJ+ksZKWi9pfDjgH5Jek7RcUrakyZKOCMszJM2UdG24nC7pv5KKHLhNUg1JwyT9LOkroFPC+gMkvSFplaTvJV0Xs25IOEHNK2Gc0yW1C9e9BBwIjJK0QdItMbu9SNJiST9JuqMkfoeu8vHk4CqLcwjGBzoUOBMYSzAq574Efwd5J+WxwCFAI4IhPIZDMAcAcDEwVNLhwG0EExHdv5vj3g20Cl+nAgPzVkhKA0YBswjmETkFuEHSqTHb9wNeIxjV9T/A25KqmtklBONQnWlmtc0sdna5E4HW4f7uCuN1LimeHFxl8TczWxHOFzEF+MzMZpjZFuAtgrk8MLPnzGx9ODHPEKBdOFkMZvYlcB/BQHo3A5eY2Y7dHPc84H4zW2NmS4C/xqzrBOxrZkPDCWgWAk8TP/PdF2b2ejhC7eNAdYKhv4tyj5ltDucpmEUw85lzSfHk4CqLFTHvNxeyXDtsKnpI0neS1gGLwvWxw2u/ABwEjDGzb4tx3AOAJTHLP8S8Pwg4QNLavBfB1cx+MXV2bhuOgpoV7rMosSOLbgJqFyNO5+J4cnAu34UEzTg9gEygeVgeOzHP34HRwKmSTizGPn8EmsUsHxjzfgnB3BX1Yl51zOz0mDo7tw2boZqSP22qj5rpSo0nB+fy1QG2AqsJ5q5+IHalpEuAo4BBBPcoXghnESzKq8BgSfXDeb6vjVn3ObBe0q3hjet0SW0TphE9SlL/8NmFG8L4Pg3XrQBa7sHndG63PDk4l+9FgmafpcBX5J+EkXQgwTzdl5rZBjP7DzCNYIL6otwT7vN74H3gpbwV4f2KM4D24fqfgGcIrlryvAOcD/xMMPdI//D+A8CDwJ1hk9TNyX9c53bN53NwroySNAQ4uBzP6ufKMb9ycM45V4AnB+f2Uvgw3YZCXrdHHZtze8qblZxzzhXgVw7OOecK8OTgnHOuAE8OzjnnCvDk4JxzrgBPDs455wrw5OCcc66A/weCGyK41srzMQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracy_list_train = []\n",
    "accuracy_list_val = []\n",
    "for max_depth in max_depth_list:\n",
    "    # You can fit the model at the same time you define it, because the fit function returns the fitted estimator.\n",
    "    model = RandomForestClassifier(max_depth = max_depth,\n",
    "                                   random_state = RANDOM_STATE).fit(X_train,y_train) \n",
    "    predictions_train = model.predict(X_train) ## The predicted values for the train dataset\n",
    "    predictions_val = model.predict(X_val) ## The predicted values for the test dataset\n",
    "    accuracy_train = accuracy_score(predictions_train,y_train)\n",
    "    accuracy_val = accuracy_score(predictions_val,y_val)\n",
    "    accuracy_list_train.append(accuracy_train)\n",
    "    accuracy_list_val.append(accuracy_val)\n",
    "\n",
    "plt.title('Train x Validation metrics')\n",
    "plt.xlabel('max_depth')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xticks(ticks = range(len(max_depth_list )),labels=max_depth_list)\n",
    "plt.plot(accuracy_list_train)\n",
    "plt.plot(accuracy_list_val)\n",
    "plt.legend(['Train','Validation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f45b6a14f10>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEXCAYAAABGeIg9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsu0lEQVR4nO3deZyVdd3/8debYZNtYFgURQM3EM0FFUxNMVssfUiaFbQI1c+2n5aWdz8tbyPUu+5u23xkdruiVqJZGZZmaZK2qKAiCooSkbIjy7ALA5/fH9c1zDXnzDDnDGfODDPv5+NxHueca/1c58y5PvNdru+liMDMzCyrU2sHYGZmbY+Tg5mZ5XFyMDOzPE4OZmaWx8nBzMzyODmYmVkeJwcrKUljJYWkIa0dS3NJmiHp1sz7qZIebWKdSZJqSrT/kPSJUmxrbyBpaHrMp7Z2LFbHyaGDSn+Mu3ssauam/w4MBpaWLtqmSTovjXtkI/N/IunfkprzN/9l4MN7FmGDMT0qaWoDswYD95d6f+W0m2NryBskx/x0y0VkxXJy6LgGZx4fSqeNykw7MbuwpK6FbDQitkXE8ojYWcJYC/EgsAy4KHeGpB7Ax4DbmhNXRFRHxNo9D7Hg/S2PiK3l2l9rktQ1Inakx7y9teOxOk4OHVT6Y1weEcuBNenkVZlpKyV9SdIvJFUDdwNIuk7Sy5I2S3pD0k8lVdZuN7daKfP+PZKeSNebJ+n9jcWmxO8lzZTUJZ3WKf1v9AlJFQ0cTw1wO/BJSd1yZn8E6AXcJmmUpIclrZS0Md3HWbv7rHKrldJYrsls416gX846wyT9WtLS9JhflPTJ7DaBM4GJmdLa2HRevWolSYMlTZO0TtKWtNrrhAY+84I/43S9yZIWSPqIpNfS9R6Q1EfS+ZLmS9og6f7sd5yuO17SbElbJS2S9H1JPXd3bJnqo49LekjSJuCahqqVJA2SdIekFek+5kv6dDqvS7q/xZLekrRM0rTdHas1Q0T40cEfwFgggCGZaQGsBi4GDgEOS6dfBbwTGEpyAngFuLOxbWXevwCcBRwG3AGsB/rtJqaBJFVT16fvv5HGc+Bu1hkK7AAm5Ez/G/DbTDyTgCOBw4FrgW3A4ZnlZwC3Zt5PBR7NvP8ysAmYmG7ja8A6oCazzNvTz+6Y9PO7BKgBzkjnVwJPAPcC+6WPrpnP/hPpa5FUt8wGTk23ey+wFhiwh5/x5PQ4fg8cDZwOrAL+CDyUxn4qsAL478x6k9L9fxI4GDgNmAPcvbtjS7+fABYDHweGpY/a6aem6+8DvAw8B7w73cd7gfHp/K+k2xgLHERSyr20tX9H7e3R6gH40foPGk8OtxWw7nnAW0CnhraVeX9+Zp1902nva2LbZ6Qn1G8C24EPFhDPw8CfM++PSPd19m7WeQH4Rub9DHafHBYD1+Vs434yyaGR/fwWuCXz/lFgagPLZZPDmen7kZn53Uiq0K7ek8+YJDnUkCaZdNqNJAl2YGbaj4BZmfeLgM/nbOu0dH/9Gju2TBL4z0am1yaHzwBbs3+POcv/CPgzoNb+7bTnh6uVbHeeyZ2QVjc8kVaXbAR+TvJf4X5NbGt27YuIWEFyAtp3dytExOPA90hOYrdGxAMFxHwzMFbSoen7i4DXSZIGkgYqaZx+Ja2m2UhSinhbAdtGUh/gAJKG96y/5izXQ9J3JM2VtCbdzwcK3U/GkcDqiJhXOyEi3iIpTRyZs+zszDIFfcbAkoh4M/N+ObA8IlblTBsEyeeXHsP30yq1jemxPZwueyhNy/u7ynE8MC8iFjcy/w6SEtSCtFrzQyqwTcwK5+Rgu7Mp+0bSGOCXJFUG55E0YH8+nd3Uj3NbA9N2+/eXti2cQnKSO0SSCoj5QZKT2UXpCeNC6jdETyWpFvta+nwsyUm11CeX/wE+AXyLpAR0LElVTUuexIr+jElKZFnRyLTa7dQ+f5nkmGofx5BUZ71YQJybml6kcRExm6Q66nKSY/4RMDtN3FYiTg5WjFOBNyPiqoh4OiJeBVryeobJJP+JngKMJjmh71bUNUxPImmI7gvcllnkNOAnETE9Il4kqZ45uNCAImI9sAQ4OWfWKTnvTwN+HhH3RcQLwEKS9omsbUBe43qOuUB/Zbropg3uY4CXCo27VNISyRvA8IhY0MCjtpdVIcfWmGeBkdrNtTIRsTEifhMRXwJOIKk+PL2Z+7MGODlYMeYDAyV9RtLBki4EvtgSO5J0OnAFMDEingY+S9KzZXQBq98KDABuAH4fEUsy8+YDH5f0dknHAvdQ/Ense8CXJX1S0mGSvkrScJo1HxgnaXR6Yr8Z2D9nmX8Bx0s6RNIApT2zcvyZpBrmF5JOkXQUcBfQHbipyLhL5RvAlyR9Q9JRkoZL+qCk/80sU8ixNeYe4N/AdEnvTnt+nSnpowCS/iPt8XSkpGHAp0lKl6+W6PgMJwcrQkT8DrgO+C+S6oPxwH+Uej+SqoCfAT+KiEfSfd9HUiV0j6TeTcS5iKTHTT+Sk3LWp0j+7p8BHgD+AMwsMsQfkSSeH5BUSb0DmJKzzGUkJ7jHgcdIShu5F7Z9D3iTpEF8FfmlDyJpgf0gSa+w36ex7ge8J6etoGwi4m6SUtk5JJ/jTJJSXjYJN3lsu9n+ZpJSwEvANJKeSzeS9GKCpBfWV4B/kPwdngd8KCLmN/eYLJ+Svz0zM7M6LjmYmVkeJwczM8vj5GBmZnmcHMzMLE/n1g6gFKqrq92qbma2ByorK+tdZOqSg5mZ5XFyMDOzPO2iWimrsrKy6YXMzIzq6upG57nkYGZmeZwczMwsj5ODmZnlKUtykHS7kvvtNjjEsBI3pPeznSNpVGbexPT+tq9JmliOeM3MOrpyNUhPBX5MMtRwQ95PcqOQw0jGqb8JGJOOzvlNkvHaA3hW0vSIWNviEVurWfsWzF0D23a0diRmbd9BveHQFuiHU5bkEBFPSBq6m0XGAXelwxM/JamvpMEk98b9U0SsAZD0J5IbqN/TwiFbmWzbAXNWw9Mr4ekVyePVxjtQmFmO/3ccfOek0m+3rXRlPYDk7lK1FqfTGptue6EIWLQhTQJpMnjuTXjLJQSzNqetJAdrh9a9BTNX1i8VrNra9Hpm1vraSnJYAhyYeT8knbaEpGopO31G2aKygm3fAS+uqV8qeGVd87bVSXBUFfTvVtIQzdqlQ/q0zHbbSnKYDlwsaRpJg3R1RCyT9AjwX5L6pcu9F7iytYK0RAS8vrF+Inh2FWxtZvXQkJ4wZl8YMyh5Pn4g9CzmjsNmVnJlSQ6S7iEpAQyQtJikB1IXgIj4KfAQ8AFgAbCZ5D6/RMQaSddQd4/fKbWN01Y+67el1UOZZLBiS/O21bMznDAITsokg/17ljZeM9tz7eIe0tkhuz220p6p2Qkv5VQPvbw26UdcLAFHVtUlgTH7wpH9oMKXXpq1CdmxlXKH7G4r1UrWCiJg8aa6xuKnVybVQ5trmre9wT3qVw+dMBB6dy1tzGZWHk4OHciGbTBrVf1SwbLNzdtWj85J20C2VDCkJ0hNr2tmbZ+TQzu1YyfMXVu/VDBvLexsRv2QgCP61S8VHFUFnV09ZNZuOTm0E0s2Zq4nWAmzVsKmZlYP7btPfvVQpbuVmnUoTg57oU3bM9VDaTJYsql52+pekVYPZZLBQb1cPWTW0Tk5tHE7dia9hbKlgpfWNK96CGBE3/qJ4O1V0KWipCGbWTvg5NDGLNtUf7iJWatgw/bmbWtg97rG4jGD4MRB0NfVQ2ZWACeHVrR5e9J1NFsqeGNj87bVrQKOG1C/99Cw3q4eMrPmcXIok50Br+RUD724GnY0s3rosMr61UPH9Ieurh4ysxJxcmghKzbXv55g5qpkGIrmqOpWPxGMHgRV3Usbr5lZlpNDCWypgedyqof+vaF52+raCY4dUD8ZHNLH1UNmVl5ODkXaGfDquvqlgjlrkjGJmuOQPvUTwbEDkvYDM7PW5OTQhFVbcqqHVsK6ZlYP9euWVAllq4cG7FPaeM3MSsHJIWNrDTz/Zv1k8K9mVg917gTH9q9fKjis0tVDZrZ36PDJYflmuO5ZeGoFvLAatjezemhY7/qJ4LgB0L3Df7pmtrfq8KevbhXw45eKW6dP1/rVQ2MGwaAeLROfmVlr6PDJoV83GN4X5q9reH6F4Oj+9S8uG943uc+xmVl71eGTAyQn/trkcFCv+tVDowZAD9/P2Mw6GCcH4AtHwvkHJ8lgP1cPmZk5OQCctF9rR2Bm1raU7V5eks6SNF/SAklXNDD/bZIekzRH0gxJQzLzvitprqSXJd0guUOomVlLKktykFQB3Ai8HxgJTJA0Mmex64G7IuJoYArw7XTdk4FTgKOBo4ATgdPLEbeZWUdVrpLDaGBBRCyMiG3ANGBczjIjgT+nrx/PzA+gO9AV6AZ0AVa0eMRmZh1YuZLDAcAbmfeL02lZLwDnp6/PA3pL6h8R/yBJFsvSxyMR8XILx2tm1qGVrc2hAJcDp0t6nqTaaAmwQ9KhwBHAEJKE8i5J72y9MM3M2r9y9VZaAhyYeT8knbZLRCwlLTlI6gV8KCLWSboIeCoiNqbzHgbeATxZjsDNzDqicpUcZgKHSRomqSswHpieXUDSAEm18VwJ3J6+fp2kRNFZUheSUoWrlczMWlBZkkNE1AAXA4+QnNjvi4i5kqZIOjddbCwwX9KrwL7Aden0+4F/Ai+StEu8EBEPliNuM7OOShHNvIlxG1JdXb3rICorK1szFDOzvUZ1dfWu15WVlfWuH2tLDdJmZtZGODmYmVkeJwczM8vj5GBmZnmcHMzMLI+Tg5mZ5XFyMDOzPE4OZmaWx8nBzMzyODmYmVkeJwczM8vj5GBmZnmcHMzMLI+Tg5mZ5XFyMDOzPE4OZmaWx8nBzMzyODmYmVkeJwczM8vj5GBmZnnKlhwknSVpvqQFkq5oYP7bJD0maY6kGZKGZOYdJOmPkl6WNE/S0HLFbWbWEZUlOUiqAG4E3g+MBCZIGpmz2PXAXRFxNDAF+HZm3l3A/0TEEcBoYGXLR21m1nGVq+QwGlgQEQsjYhswDRiXs8xI4M/p68dr56dJpHNE/AkgIjZGxObyhG1m1jGVKzkcALyReb84nZb1AnB++vo8oLek/sDhwDpJv5b0vKT/SUsiZmbWQtpSg/TlwOmSngdOB5YAO4DOwDvT+ScCBwOTWilGM7MOoVzJYQlwYOb9kHTaLhGxNCLOj4jjgG+k09aRlDJmp1VSNcADwKhyBG1m1lGVKznMBA6TNExSV2A8MD27gKQBkmrjuRK4PbNuX0kD0/fvAuaVIWYzsw6rLMkh/Y//YuAR4GXgvoiYK2mKpHPTxcYC8yW9CuwLXJeuu4OkSukxSS8CAm4pR9xmZh2VIqK1Y9hj1dXVuw6isrKyNUMxM9trVFdX73pdWVmp7Ly21CBtZmZthJODmZnlcXIwM7M8Tg5mZpbHycHMzPI4OZiZWR4nBzMzy+PkYGZmeZwczMwsj5ODmZnlcXIwM7M8BScHSb+R9EFJXVoyIDMza33FlByeBK4Glku6SdLJLRSTmZm1soKTQ0R8PyJGAacB64B7JL0m6WpJh7RUgGZmVn5FtzlExNyIuBL4BLAZ+CbwnKRHJR1T6gDNzKz8ikoOkoZLukbSP4GbgXuBoSQ353mI5BaeZma2l+tc6IKSZpEkgnuBj0XE0zmLfF/SJSWMzczMWknByQH4DjA9IrY1tkBEDNvzkMzMrLUVU620nqTksEtazfSekkZkZmatrpjkcCOwIWfahnS6mZm1I8Ukh0ERsSxn2jJgv0JWlnSWpPmSFki6ooH5b5P0mKQ5kmZIGpIzv4+kxZJ+XETMZmbWDMUkh4WS3pUzbSzwr6ZWlFRBUsJ4PzASmCBpZM5i1wN3RcTRwBTg2znzrwGeKCJeMzNrpmIapCcDv5Z0G/BP4BDgU+mjKaOBBRGxEEDSNGAcMC+zzEjgK+nrx8l0i5V0PEl32T8AJxQRs5mZNUMxV0j/Fngv0BM4O31+Xzq9KQcAb2TeL06nZb0AnJ++Pg/oLam/pE7A94DLC43VzMz2TDElByLiGeCZForlcuDHkiaRVB8tAXYAXwQeiojFklpo12ZmllVUcpB0LPBOYACw60wdEVc3seoS4MDM+yHptF0iYilpyUFSL+BDEbFO0juAd0r6ItAL6CppY0TkNWqbmVlpFHOF9GeBHwB/JGlYfpikmqmQaqWZwGGShpEkhfHAx3K2PwBYExE7gSuB2wEi4uOZZSYBJzgxmJm1rGJ6K30NOCsizgO2pM8XANubWjEiaoCLgUeAl4H7ImKupCmSzk0XGwvMl/QqSePzdUXEZmZmJaSIKGxBaX1E9ElfrwYGRsROSWsioqolg2xKdXX1roOorKxszVDMzPYa1dXVu15XVlbWa9Qtps1hsaShEbEIeBUYJ+lNoNGxlszMbO9UTHL4LnAEsIjkIrX7ga7Al0oflpmZtaaCkoOSPqRPAK8DRMTDkvoBXSNiYwvGZ2ZmraCgBulIGiZeBHZmpm1zYjAza5+K6a30PHB4SwViZmZtRzFtDjOAP0iaSjIUxq4eQhFxe2nDMjOz1lRMcjiFZATW03OmB+kFa2Zm1j4UnBwi4oyWDMTMzNqOYobPaLR9Ih3ywszM2oliqpVqyLQz5KgoQSxmZtZGFJMchuW8HwxcATxYunDMzKwtKHhspQZXliqBmRHRql1cPbaSmVnxdje2UjHXOTSkDzBwD7dhZmZtTDEN0ndTv82hB3Aa8LNSB2VmZq2rmDaHBTnvNwE/jYhHSxiPmZm1AcVc5/CtlgzEzMzajoLbHCTdIOnknGknS/phyaMyM7NWVUyD9ARgVs60Z8m5F7SZme39ikkO0cDyFUVuw8zM9gLFnNifBK6tHUYjfZ6cTjczs3akmOTwZeDdwDJJzwBLgfcAlxSysqSzJM2XtEDSFQ3Mf5ukxyTNkTRD0pB0+rGS/iFpbjrvo0XEbGZmzVDUFdJpaWE0cCDJPR2eKWTQPUkVwKskyWQxMBOYEBHzMsv8EvhdRNwp6V3ApyLik5IOJ7kZ3WuS9idp5zgiItbVrusrpM3Mire7K6SLuQjuWGB1RDwFPJVOO1BSVUS80MTqo4EFEbEwXW8aMA6Yl1lmJPCV9PXjwAMAEfFq7QIRsVTSSpKrstcVGruZmRWnmGqlnwFdcqZ1Be4uYN0DSEoatRan07JeAM5PX58H9JbUP7uApNHpPv9ZYMxmZtYMxSSHg2r/868VEf8EhpYolsuB0yU9T3K3uSXAjtqZkgaTJKJP+f4RZmYtq5jhMxZLGhURz9VOkDSKpGG6KUtI2ilqDUmn7RIRS0lLDpJ6AR+qbVeQ1Af4PfCNtFrLzMxaUDHJ4QfAbyV9l6Ra5xCS//avK2DdmcBhkoaRJIXx5Fw8J2kAsCYtFVxJel9qSV2B3wB3RcT9RcRrZmbNVMzYSrdIWgd8hqQU8Drw1UJO2BFRI+li4BGSC+duj4i5kqYAsyJiOjAW+LakAJ4A/m+6+kdIRn/tL2lSOm1SRMwuNHYzMytOsV1Z9wXGAAOy0yPi9hLHVRR3ZTUzK16purJ+kKRBeAFwJDAXOAr4K2kVkJmZtQ/F9Fa6Fvh0RBwHbEqfP0tyUZqZmbUjxXZl/WXOtDuBC0sYj5mZtQHFJIeVaZsDwCJJ7yDpsVRR+rDMzKw1FZMcbgFOTV//gGSIixeAn5Q6KDMza11F9Vaqt6J0ENAzIl4ubUjFc28lM7PilaS3Uq6IeH0PYjIzszbMd3EzM7M8Tg5mZpbHycHMzPI4OZiZWR4nBzMzy+PkYGZmeZwczMwsj5ODmZnlcXIwM7M8Tg5mZpbHycHMzPI4OZiZWR4nBzMzy+PkYGZmecqWHCSdJWm+pAWSrmhg/tskPSZpjqQZkoZk5k2U9Fr6mFiumM3MOqqyJAdJFcCNwPuBkcAESSNzFrseuCsijgamAN9O160CvgmMAUYD35TUrxxxm5l1VOUqOYwGFkTEwojYBkwDxuUsMxL4c/r68cz89wF/iog1EbEW+BNwVhliNjPrsMqVHA4A3si8X5xOy3oBOD99fR7QW1L/Atc1M7MSaksN0pcDp0t6HjgdWALsaN2QzMw6pmbfQ7pIS4ADM++HpNN2iYilpCUHSb2AD0XEOklLgLE5685oyWDNzDq6cpUcZgKHSRomqSswHpieXUDSAEm18VwJ3J6+fgR4r6R+aUP0e9NpZmbWQsqSHCKiBriY5KT+MnBfRMyVNEXSueliY4H5kl4F9gWuS9ddA1xDkmBmAlPSaWZm1kIUEa0dwx6rrq7edRCVlZWtGYqZ2V6jurp61+vKykpl57WlBmkzM2sjnBzMzCyPk4OZmeVxcjAzszxODmZWmG3rYdNy2LG9tSOxMijXRXBmtjeInbBxMax9Jf+xeVndcl0rYZ8B0L320b/ufUPP3aqgU0XrHZcVzcnBrCOq2QLrXs1PAOvmJ/Oasq06eVT/s8AdCrr1azyB5E3rD936gly50VqcHMzaqwjYvCI96b8Ca+fXJYEN/wbKeY1TwFtrkgevFraKOiVJYrdJJOe5S2+Qmt62NcnJwWxvt2M7rP9nw1VB26qbXr9QnTpDlz7w1lrKklhiJ2xZlTzWFrhOpy6NlEz6N55gOvdwQmmAk4PZ3mLr2kz1TyYBrF8IO2tKt59ufaHfEdBvRN2j73DoczBUdIGdO+CtdbD1zeSxpYnnrW8my5fDzu1J20i2faQpFd0LqObKSTSdu7fcMbQRHj7DrC3ZuSOp8qmt/8+WArasLOGOBH2GQb/h0HdE/USwz8DS/ye9Y3tSpbTlTdi6uvEkkp22fUNpYyilLj0LSCI5CaWiS2tHnWd3w2e45GDWGrZtrGsQzpYC1r0KO94q3X4696h/4u83IkkGfQ+FzvuUbj9NqegCPfZNHoXa8RZsWd1ECSVnfs3mljuGrO2bkseGfxe+Tm4Pr9wG+DbWw8vJwaylRMCmpTm9gdKG4Y1vNL1+MXoekF8N1G8E9Dpg7+3xU9ENeu2fPAq1fXPhJZPa553bWu4Yslqqh1floVB1RMnDdXIw21M73oJ1ryUn/WwpYO0rsH1j6fbTqSv0PSy/FNBvOHTtXbr97M269EgevQ9sellIEvj2jfklkd21n2x5E6IcN6kssIfX8AvhPXeWfO9ODmaF2vJmA9cFvALr/5X0rCmV7gPyq4L6jYDeQ30hWalJSWLt2jtpgylERFIC2F0C2bo6v/qrpXp47TOgRTbr5GCWtbMmOdnnXhi29pX0B14i6gR9Dsmc/IfXVQe10I/dSkRKenR16wscWtg6LdnDq7uTg1npbFtf/6Kw2lLAuteS7pCl0qV3w6WAykOSOnXrGDpVwD79kwfDC1untodXXikkJ4m0QHsDODlYe1boOEGl0Pug/C6h/UZAj/18gZU1T3N6eJWQk4Pt/fLGCZpf1yuolF0bK7rXr/7Z1Sh8eNLv3UomIlizZg07d5awLaeD69SpE1VVVajAf1acHGzvEJFcBNZQKaDU4wT12LfhUkDvg/bebqF7mTVr1tCzZ0+6d2//VyKXy9atW1mzZg39+/cvaPmyJQdJZwE/AiqAWyPiOznzDwLuBPqmy1wREQ9J6gLcCoxK470rIr5drritFWxcCiufaflxgioPzXQHzTQMd+tbuv1Ys+zcudOJocS6d+/Ohg2FX3VeluQgqQK4EXgPsBiYKWl6RMzLLHYVcF9E3CRpJPAQMBT4MNAtIt4uqQcwT9I9EbGoHLFbC4udyYl/6V9hWfpY/6/Sbb92nKBsNVC/EXXjBJlZg8pVchgNLIiIhQCSpgHjgGxyCKBP+roSWJqZ3lNSZ2AfYBuwvhxBWwvY8RasnAXL/pYmhL+lF/nsCUGfoQ2UAlponCCzDqBcyeEAIDtewGJgTM4yk4E/SroE6Am8O51+P0kiWQb0AC6LiD09m1i5bF0Ly/9eVzJYObP5Ywc1OE7Q8OSq4XKOE2Tt3urVqznzzDMBWL58ORUVFQwcOBCAZ555hq5duza67qxZs7jrrru44YYbyhJrS2lLDdITgKkR8T1J7wDulnQUSaljB7A/0A94UtKjtaUQa0MiksbhZX+tKxmsean47XTqAgOPh4HH1R86em8eJ8j2Kv3792f27NkATJ48mV69enH55Zfvml9TU0Pnzg2fPk844QROOOGEcoTZosqVHJYA2cFOhqTTsj4DnAUQEf+Q1B0YAHwM+ENEbAdWSvobcALg5NDadu6A1S/WtRUs/Stsyv1aC9C1EgafDINPTR77nuiSgNWjm0q7vfhC8etMmjSJ7t278/zzz3PKKacwfvx4vvzlL7N161b22Wcf7rjjDoYPH86MGTO4/vrr+d3vfsfkyZN5/fXXWbhwIa+//jqXXnopX/rSl0p7MC2kXMlhJnCYpGEkSWE8yUk/63XgTGCqpCOA7sCqdPq7SEoSPYGTgB+WKW7L2r4JVjxTlwiW/6N5Y+73PihNBKckz1VHeswg2yssXryYv//971RUVLB+/XqefPJJOnfuzKOPPsrXv/51fvWrX+Wt88orr/D444+zYcMGhg8fzhe+8AW6dGn7nSHKkhwiokbSxcAjJN1Ub4+IuZKmALMiYjrwVeAWSZeRNEJPioiQdCNwh6S5gIA7ImJOOeLu8DavTKqHaksGq55rxh3HBP3fDvufWpcQeh/UIuGatbQPf/jDVFQk/8hUV1czceJEXnvtNSSxfXvDw66cffbZdOvWjW7dujFo0CBWrFjBkCFDyhl2s5StzSEiHiLpnpqddnXm9TzglAbW20jSndVaUkQyrlC2iqj6teK3U9Ed9h2TJoNTYL93+LoBazd69qy7Ev4///M/OeOMM/jNb37DokWLGDt2bIPrdOtWN4ZWRUUFNTUlvKVrC2pLDdJWTju2warn65LBsr8lN3IvVvcBddVD+58KA0dBReM9OcyaozltBC2turqaAw44AICpU6e2bjAtwMmho3irGpY/VZcMVjydjElUrMpD6xLB4FOTcYV8HYF1QF/72teYOHEi1157LWeffXZrh1NyimihG1CUUXV19a6DqKysbM1Q2o6Ni+tfdfzmHIoef0gVSUlgV+PxKdBzvxYJ1yxr1apVu64rsNLJ/Vyrq+uGpKmsrKz3X55LDu1B7ITVc+uqh5b9tbgbn9fq0itpI6gtGew7xqONmnVQTg57o5qtyZXGtSWD5X8v/K5RWT0Gw/7vrEsG/d+eDEhnZh2ezwR7gy1v5gxBMat5dyurOrKuimj/U5N7Eru9wMwa4OTQ1kTA+oV1g9It+yusfbn47XTqmlxpXHvV8eCToXtV6eM1s3bJyaG17ayBN1+ou7Zg2V9h8/Lit9OtX06X0uOhs8fDN7PmcXIot20bYcVTdYlgxVPJsBTF6jOsLhkMPjW5ybgHpTOzEvHZpKVtWgYLfglPXgr3ngC39IXfvgdmfgsWP1ZYYlCnZITSoy+B990LkxbDhQvhPXfDUZ+D/kc6MZiV0BlnnMEjjzxSb9oPf/hDvvCFhq/GGzt2LLNmzQLgAx/4AOvWrctbZvLkyVx//fW73e8DDzzAvHl1t7m5+uqrefTRR4uMvjRcciiliOSuZtkqovXNGDy2cw/Y76S6ksF+J0HXPk2vZ2YlMWHCBKZNm8b73ve+XdOmTZvGd7/73SbXfeihh5pcpjEPPPAA55xzDiNHjgRgypQpzd7WnnJy2BM73oKVz2YGp/sbbF1d/Hb2GVT/quMBx/oWlmZZPy5xr7qLd39B6AUXXMBVV13Ftm3b6Nq1K4sWLWLp0qXcc889fOUrX2HLli1ccMEFfOtb38pbd+jQocyaNYsBAwZw3XXXceeddzJo0CAOPPBAjj/+eABuueUWbr75ZrZt28ahhx7K3XffzezZs5k+fTp/+ctfuPbaa/nVr37FNddcwznnnMMFF1zAY489xuWXX05NTQ0nnngiN910E926dWPo0KFMnDiRBx98kO3bt/PLX/6SESNG7PFH5ORQjK1rk2Gqdw1B8Uzz7mrWd3j9UUorD3WXUrM2pKqqitGjR/Pwww8zbtw4pk2bxkc+8hG+/vWvU1VVxY4dOzjzzDOZM2cORx99dIPbePbZZ5k2bRqzZ8+mpqaGUaNG7UoO559/PhdddBEAV111FbfddhuXXHIJ55577q5kkLV161YmTZrEY489xuGHH86FF17ITTfdxKWXXgrAgAEDeO655/jJT37C9ddfz6233rrHn4GTQ2MiYMPr9UcpXTOXooeg6NQZBp5Qd23B4FOS+xqbWZtWW7VUmxxuu+027rvvPm6++WZqampYtmwZ8+bNazQ5PPnkk5x33nn06NEDgHPPPXfXvJdeeomrrrqKdevWsXHjxnrVVw2ZP38+w4YN4/DDDwdg4sSJ3HjjjbuSw/nnnw/A8ccfz69//es9PXTAyaHOrruaZe5fsHFx8dvp2gf2O7muZDDoROjSo/TxmlmLGjduHJdddhnPPfccmzdvpqqqiuuvv56ZM2fSr18/Jk2axNatW5u17UmTJvHAAw9wzDHHMHXqVGbMmLFHsdYOC17KIcGdHKoXwl++mFQXbVtf/Pq9hsDgd9aVCqqO8l3NzEqtiTaCltCrVy/OOOMMPv3pTzNhwgTWr19Pz549qaysZMWKFTz88MON3sMB4LTTTmPSpElceeWV1NTU8OCDD/K5z30OgA0bNjB48GC2b9/Oz3/+811Df/fu3ZsNG/Lvrjh8+HAWLVrEggULdrVRnH766S1y3LWcHLpXwet/pLDqIkH/o+quLdj/VN/VzKwdmzBhAueddx7Tpk1jxIgRHHfccYwYMYIDDzyQU07JuzdZPaNGjeKjH/0oxxxzDIMGDeLEE0/cNe+aa65hzJgxDBw4kDFjxuxKCOPHj+eiiy7ihhtu4P7779+1fPfu3bnjjjv48Ic/vKtB+vOf/3zLHHTKQ3YDTDs2uUo5V0V32Hd0XSLwXc3MysJDdrcMD9ldrMGnJsmhe//MWESnwKBRUNGt6fXNzNoZJweAYy+Doy9Oupi6S6mZWfmGz5B0lqT5khZIuqKB+QdJelzS85LmSPpAZt7Rkv4haa6kFyWVdkS5ykOg3wgnBjOzVFmSg6QK4Ebg/cBIYIKkkTmLXQXcFxHHAeOBn6TrdgZ+Bnw+Io4ExgLNuJmBme0tOnXq1OxuotawrVu30qlT4af8clUrjQYWRMRCAEnTgHHAvMwyAdQOIFQJLE1fvxeYExEvAEREM8anMLO9SVVVFWvWrGmwW6c1T6dOnaiqKvyeLuVKDgcAb2TeLwbG5CwzGfijpEuAnsC70+mHAyHpEWAgMC0imh79ysz2WpLo379/a4fRobWlcZ4nAFMjYgjwAeBuSZ1IEtipwMfT5/Mkndl6YZqZtX/lSg5LgAMz74ek07I+A9wHEBH/ALoDA0hKGU9ExJsRsRl4CBjV4hGbmXVg5apWmgkcJmkYSVIYD3wsZ5nXgTOBqZKOIEkOq4BHgK9J6gFsA04HftDYjrIXdZiZWfOUJTlERI2ki0lO9BXA7RExV9IUYFZETAe+Ctwi6TKSxulJkVy+vVbS90kSTAAPRcTvyxG3mVlH1e6GzzAzs+LlDp/RLpKDmZmVVlvqrWRmZm1Eh0sOkm6XtFLSS5lpVZL+JOm19Llfa8bYEUlalA6NMlvSrHSav5cyK+b3ocQN6ZA4cyS5F2ELKeb3UarvpcMlB2AqcFbOtCuAxyLiMOCx9L2V3xkRcWxEnJC+9/dSflMp/PfxfuCw9PFZ4KYyxdhRFfr7KMn30uGSQ0Q8AazJmTwOuDN9fSfwwXLGZI3y91JmRf4+xgF3ReIpoK+kwWUJ1KCFv5cOlxwasW9ELEtfLwf2bc1gOqggGT7lWUmfTaf5e2kbGvseGhoW54ByBtaBFPP7KMn34vs55IiIkOQuXOV3akQskTQI+JOkV7Iz/b20Df4eWk3Zfx8uOSRW1Ba70ueVrRxPhxMRS9LnlcBvSEby9ffSNjT2PRQyLI6VQJG/j5J8L04OienAxPT1ROC3rRhLhyOpp6Teta9Jhml/CX8vbUVj38N04MK0d8xJQHWmmsNKpBm/j5J8Lx3uIjhJ95DcMGgAsAL4JvAAyaB/BwH/Bj4SEbmNctZCJB1M8t8QJFWdv4iI6yT1x99LWRXz+5Ak4MckvZs2A5+KiFmtEHa7Vuzvo1TfS4dLDmZm1jRXK5mZWR4nBzMzy+PkYGZmeZwczMwsj5ODmZnlcXIwM7M8Tg5mJSDpYUkTm17SbO/g6xzMiiRpMnBoRHyiDPsaCvwL6BIRNS29P7NaLjmYtWOSPLimNYuTg7Ub6d2yLk/vflUt6V5J3ZtY55z07lrrJP1d0tGZef9P0hJJGyTNl3SmpLOArwMflbRR0gvpsjMk/Z/09SRJf5P0g3S7CyWdnE5/I73T2sTMfs6W9Lyk9en8yZkQn0if16X7e4ekTpKukvTvdFt3SapMtzVUUkj6jKTXgT9L6i7pZ5JWp/HMlOThz223nBysvfkIyZgyw4CjgUmNLSjpOOB24HNAf+B/gemSukkaDlwMnBgRvYH3AYsi4g/AfwH3RkSviDimkc2PAeak2/0FMA04ETgU+ATwY0m90mU3ARcCfYGzgS9I+mA677T0uW+6v3+kxzQJOAM4GOhFMpZO1unAEWncE4FKkpE6+wOfB7Y09rmYgZODtT83RMTSdIC+B4Fjd7PsZ4H/jYinI2JHRNwJvAWcBOwAugEjJXWJiEUR8c8i4vhXRNwRETuAe0lOzFMi4q2I+COwjSRREBEzIuLFiNgZEXOAe0hO7o35OPD9iFgYERuBK4HxOVVIkyNiU0RsAbaTJIVD0+N8NiLWF3Es1gE5OVh7szzzejPJf9WNeRvw1bSqZZ2kdSQn8f0jYgFwKTAZWClpmqT9i4hjReb1FoCIyJ3WC0DSGEmPS1olqZrkP/sBu9n2/iSjcNb6N8londmqouydwO4GHgGmSVoq6buSuhRxLNYBOTlYR/YGcF1E9M08ekTEPQAR8YuIOJUkiQTw3+l6pe7i9wuSMfgPjIhK4KeAdrOvpWlMtQ4CaqifkHatFxHbI+JbETESOBk4h6Qay6xRTg7Wkd0CfD79z13pTVXOltRb0nBJ75LUDdhK8p/+znS9FcBQSaX6/fQG1kTEVkmjgY9l5q1K93twZto9wGWShqXtFrVtIA12dZV0hqS3S6oA1pNUM+1saFmzWk4O1mGlN0C5iKQxdy2wgLoG7G7Ad4A3SaqqBpHU7QP8Mn1eLem5EoTyRWCKpA3A1SQ3cKmNcTNwHfC3tOrrJJJG9LtJejL9iyR5XbKb7e8H3E+SGF4G/pKub9YoXwRnZmZ5XHIwM7M8Tg7Wrkn6enrxWO7j4daOzawtc7WSmZnlccnBzMzyODmYmVkeJwczM8vj5GBmZnmcHMzMLM//B/CoSA/b9uG5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracy_list_train = []\n",
    "accuracy_list_val = []\n",
    "for n_estimators in n_estimators_list:\n",
    "    # You can fit the model at the same time you define it, because the fit function returns the fitted estimator.\n",
    "    model = RandomForestClassifier(n_estimators = n_estimators,\n",
    "                                   random_state = RANDOM_STATE).fit(X_train,y_train) \n",
    "    predictions_train = model.predict(X_train) ## The predicted values for the train dataset\n",
    "    predictions_val = model.predict(X_val) ## The predicted values for the test dataset\n",
    "    accuracy_train = accuracy_score(predictions_train,y_train)\n",
    "    accuracy_val = accuracy_score(predictions_val,y_val)\n",
    "    accuracy_list_train.append(accuracy_train)\n",
    "    accuracy_list_val.append(accuracy_val)\n",
    "\n",
    "plt.title('Train x Validation metrics')\n",
    "plt.xlabel('n_estimators')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xticks(ticks = range(len(n_estimators_list )),labels=n_estimators_list)\n",
    "plt.plot(accuracy_list_train)\n",
    "plt.plot(accuracy_list_val)\n",
    "plt.legend(['Train','Validation'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's then fit a random forest with the following parameters:\n",
    "\n",
    " - max_depth: 16\n",
    " - min_samples_split: 10\n",
    " - n_estimators: 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_forest_model = RandomForestClassifier(n_estimators = 100,\n",
    "                                             max_depth = 16, \n",
    "                                             min_samples_split = 10).fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics train:\n",
      "\tAccuracy score: 0.9319\n",
      "Metrics test:\n",
      "\tAccuracy score: 0.8804\n"
     ]
    }
   ],
   "source": [
    "print(f\"Metrics train:\\n\\tAccuracy score: {accuracy_score(random_forest_model.predict(X_train),y_train):.4f}\\nMetrics test:\\n\\tAccuracy score: {accuracy_score(random_forest_model.predict(X_val),y_val):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we are searching for the best value one hyperparameter while leaving the other hyperparameters at their default values.\n",
    "- Ideally, we would want to check every combination of values for every hyperparameter that we are tuning.\n",
    "- If we have 3 hyperparameters, and each hyperparameter has 4 values to try out, we should have a total of 4 x 4 x 4 = 64 combinations to try.\n",
    "- When we only modify one hyperparameter while leaving the rest as their default value, we are trying 4 + 4 + 4 = 12 results. \n",
    "- To try out all combinations, we can use a sklearn implementation called GridSearchCV. GridSearchCV has a refit parameter that will automatically refit a model on the best combination so we will not need to program it explicitly. For more on GridSearchCV, please refer to its [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 XGBoost\n",
    "\n",
    "Next is the Gradient Boosting model, called XGBoost. The boosting methods train several trees, but instead of them being uncorrelated to each other, now the trees are fit one after the other in order to minimize the error. \n",
    "\n",
    "The model has the same parameters as a decision tree, plus the learning rate.\n",
    "- The learning rate is the size of the step on the Gradient Descent method that the XGBoost uses internally to minimize the error on each train step.\n",
    "\n",
    "One interesting thing about the XGBoost is that during fitting, it can take in an evaluation dataset of the form `(X_val,y_val)`.\n",
    "- On each iteration, it measures the cost (or evaluation metric) on the evaluation datasets.\n",
    "- Once the cost (or metric) stops decreasing for a number of rounds (called early_stopping_rounds), the training will stop. \n",
    "- More iterations lead to more estimators, and more estimators can result in overfitting.  \n",
    "- By stopping once the validation metric no longer improves, we can limit the number of estimators created, and reduce overfitting.\n",
    "\n",
    "First, let's define a subset of our training set (we should not use the test set here)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = int(len(X_train)*0.8) ## Let's use 80% to train and 20% to eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_fit, X_train_eval, y_train_fit, y_train_eval = X_train[:n], X_train[n:], y_train[:n], y_train[n:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then set a large number of estimators, because we can stop if the cost function stops decreasing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note some of the `.fit()` parameters:\n",
    "- `eval_set = [(X_train_eval,y_train_eval)]`:Here we must pass a list to the eval_set, because you can have several different tuples ov eval sets. \n",
    "- `early_stopping_rounds`: This parameter helps to stop the model training if its evaluation metric is no longer improving on the validation set. It's set to 10.\n",
    "  - The model keeps track of the round with the best performance (lowest evaluation metric).  For example, let's say round 16 has the lowest evaluation metric so far.\n",
    "  - Each successive round's evaluation metric is compared to the best metric.  If the model goes 10 rounds where none have a better metric than the best one, then the model stops training.\n",
    "  - The model is returned at its last state when training terminated, not its state during the best round.  For example, if the model stops at round 26, but the best round was 16, the model's training state at round 26 is returned, not round 16.\n",
    "  - Note that this is different from returning the model's \"best\" state (from when the evaluation metric was the lowest)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-logloss:0.64479\n",
      "[1]\tvalidation_0-logloss:0.60569\n",
      "[2]\tvalidation_0-logloss:0.57481\n",
      "[3]\tvalidation_0-logloss:0.54947\n",
      "[4]\tvalidation_0-logloss:0.52973\n",
      "[5]\tvalidation_0-logloss:0.51331\n",
      "[6]\tvalidation_0-logloss:0.49823\n",
      "[7]\tvalidation_0-logloss:0.48855\n",
      "[8]\tvalidation_0-logloss:0.47888\n",
      "[9]\tvalidation_0-logloss:0.47068\n",
      "[10]\tvalidation_0-logloss:0.46507\n",
      "[11]\tvalidation_0-logloss:0.45832\n",
      "[12]\tvalidation_0-logloss:0.45557\n",
      "[13]\tvalidation_0-logloss:0.45030\n",
      "[14]\tvalidation_0-logloss:0.44653\n",
      "[15]\tvalidation_0-logloss:0.44213\n",
      "[16]\tvalidation_0-logloss:0.43948\n",
      "[17]\tvalidation_0-logloss:0.44088\n",
      "[18]\tvalidation_0-logloss:0.44358\n",
      "[19]\tvalidation_0-logloss:0.44493\n",
      "[20]\tvalidation_0-logloss:0.44294\n",
      "[21]\tvalidation_0-logloss:0.44486\n",
      "[22]\tvalidation_0-logloss:0.44586\n",
      "[23]\tvalidation_0-logloss:0.44680\n",
      "[24]\tvalidation_0-logloss:0.44925\n",
      "[25]\tvalidation_0-logloss:0.45383\n",
      "[26]\tvalidation_0-logloss:0.45547\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', callbacks=None,\n",
       "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
       "              early_stopping_rounds=None, enable_categorical=False,\n",
       "              eval_metric=None, gamma=0, gpu_id=-1, grow_policy='depthwise',\n",
       "              importance_type=None, interaction_constraints='',\n",
       "              learning_rate=0.1, max_bin=256, max_cat_to_onehot=4,\n",
       "              max_delta_step=0, max_depth=6, max_leaves=0, min_child_weight=1,\n",
       "              missing=nan, monotone_constraints='()', n_estimators=500,\n",
       "              n_jobs=0, num_parallel_tree=1, predictor='auto', random_state=55,\n",
       "              reg_alpha=0, reg_lambda=1, ...)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb_model = XGBClassifier(n_estimators = 500, learning_rate = 0.1,verbosity = 1, random_state = RANDOM_STATE)\n",
    "xgb_model.fit(X_train_fit,y_train_fit, eval_set = [(X_train_eval,y_train_eval)], early_stopping_rounds = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though we initialized the model to allow up to 500 estimators, the algorithm only fit 26 estimators (over 26 rounds of training).\n",
    "\n",
    "To see why, let's look for the round of training that had the best performance (lowest evaluation metric).  You can either view the validation log loss metrics that were output above, or view the model's `.best_iteration` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb_model.best_iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best round of training was round 16, with a log loss of 4.3948.  \n",
    "- For 10 rounds of training after that (from round 17 to 26), the log loss was higher than this.\n",
    "- Since we set `early_stopping_rounds` to 10, then by the 10th round where the log loss doesn't improve upon the best one, training stops.\n",
    "- You can try out different values of `early_stopping_rounds` to verify this.  If you set it to 20, for instance, the model stops training at round 36 (16 + 20)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics train:\n",
      "\tAccuracy score: 0.9251\n",
      "Metrics test:\n",
      "\tAccuracy score: 0.8641\n"
     ]
    }
   ],
   "source": [
    "print(f\"Metrics train:\\n\\tAccuracy score: {accuracy_score(xgb_model.predict(X_train),y_train):.4f}\\nMetrics test:\\n\\tAccuracy score: {accuracy_score(xgb_model.predict(X_val),y_val):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, both Random Forest and XGBoost had similar performance (test accuracy).  \n",
    "\n",
    "Congratulations, you have learned how to use Decision Tree, Random Forest from the scikit-learn library and XGBoost!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "56d44d6a8424451b5ce45d1ae0b0b7865dc60710e7f74571dd51dd80d7829ee9"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}