notebooks/parameter_tuning_grid_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a44e2290",
   "metadata": {},
   "source": [
    "# Hyperparameter tuning by grid-search\n",
    "\n",
    "In the previous notebook, we saw that hyperparameters can affect the\n",
    "statistical performance of a model. In this notebook, we will show how to\n",
    "optimize hyperparameters using a grid-search approach."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37bd450c",
   "metadata": {},
   "source": [
    "## Our predictive model\n",
    "\n",
    "Let us reload the dataset as we did previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "49dfb463",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:28.825450Z",
     "start_time": "2021-05-31T09:56:28.533112Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "\n",
    "set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e488c933",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:29.990202Z",
     "start_time": "2021-05-31T09:56:29.808608Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1056503a",
   "metadata": {},
   "source": [
    "We extract the column containing the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3ba361d8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:30.813904Z",
     "start_time": "2021-05-31T09:56:30.802398Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         <=50K\n",
       "1         <=50K\n",
       "2          >50K\n",
       "3          >50K\n",
       "4         <=50K\n",
       "          ...  \n",
       "48837     <=50K\n",
       "48838      >50K\n",
       "48839     <=50K\n",
       "48840     <=50K\n",
       "48841      >50K\n",
       "Name: class, Length: 48842, dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "target"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3204e92e",
   "metadata": {},
   "source": [
    "We drop from our data the target and the `\"education-num\"` column which\n",
    "duplicates the information from the `\"education\"` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "27c0e716",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:32.667985Z",
     "start_time": "2021-05-31T09:56:32.651626Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25</td>\n",
       "      <td>Private</td>\n",
       "      <td>11th</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Farming-fishing</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28</td>\n",
       "      <td>Local-gov</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Protective-serv</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>44</td>\n",
       "      <td>Private</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>7688</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>18</td>\n",
       "      <td>?</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>?</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age   workclass      education       marital-status          occupation  \\\n",
       "0   25     Private           11th        Never-married   Machine-op-inspct   \n",
       "1   38     Private        HS-grad   Married-civ-spouse     Farming-fishing   \n",
       "2   28   Local-gov     Assoc-acdm   Married-civ-spouse     Protective-serv   \n",
       "3   44     Private   Some-college   Married-civ-spouse   Machine-op-inspct   \n",
       "4   18           ?   Some-college        Never-married                   ?   \n",
       "\n",
       "  relationship    race      sex  capital-gain  capital-loss  hours-per-week  \\\n",
       "0    Own-child   Black     Male             0             0              40   \n",
       "1      Husband   White     Male             0             0              50   \n",
       "2      Husband   White     Male             0             0              40   \n",
       "3      Husband   Black     Male          7688             0              40   \n",
       "4    Own-child   White   Female             0             0              30   \n",
       "\n",
       "   native-country  \n",
       "0   United-States  \n",
       "1   United-States  \n",
       "2   United-States  \n",
       "3   United-States  \n",
       "4   United-States  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = adult_census.drop(columns=[target_name, \"education-num\"])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0709aeb",
   "metadata": {},
   "source": [
    "Once the dataset is loaded, we split it into a training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b0171360",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:36.387201Z",
     "start_time": "2021-05-31T09:56:36.365188Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1406f09",
   "metadata": {},
   "source": [
    "We will define a pipeline as seen in the first module. It will handle both\n",
    "numerical and categorical features.\n",
    "\n",
    "As we will use a tree-based model as a predictor, here we apply an ordinal\n",
    "encoder on the categorical features: it encodes every category with an\n",
    "arbitrary integer. For simple models such as linear models, a one-hot encoder\n",
    "should be preferred. But for complex models, in particular tree-based models,\n",
    "the ordinal encoder is useful as it avoids having high-dimensional\n",
    "representations.\n",
    "\n",
    "First we select all the categorical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e7a04223",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:39.993780Z",
     "start_time": "2021-05-31T09:56:39.986153Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_selector as selector\n",
    "\n",
    "categorical_columns_selector = selector(dtype_include=object)\n",
    "categorical_columns = categorical_columns_selector(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e58feb1",
   "metadata": {},
   "source": [
    "Then we build our ordinal encoder, giving it the known categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bb38224d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:42.112435Z",
     "start_time": "2021-05-31T09:56:42.106154Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "categorical_preprocessor = OrdinalEncoder(handle_unknown=\"use_encoded_value\",\n",
    "                                          unknown_value=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3afed25",
   "metadata": {},
   "source": [
    "We now use a column transformer with code to select the categorical columns\n",
    "and apply to them the ordinal encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f9f0aa14",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:44.881188Z",
     "start_time": "2021-05-31T09:56:44.874539Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "preprocessor = ColumnTransformer([\n",
    "    ('cat-preprocessor', categorical_preprocessor, categorical_columns)],\n",
    "    remainder='passthrough', sparse_threshold=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fea5e46",
   "metadata": {},
   "source": [
    "Finally, we use a tree-based classifier (i.e. histogram gradient-boosting) to\n",
    "predict whether or not a person earns more than 50 k$ a year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5fadd6aa",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:56:48.830314Z",
     "start_time": "2021-05-31T09:56:48.789186Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 {color: black;background-color: white;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 pre{padding: 0;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-toggleable {background-color: white;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-estimator:hover {background-color: #d4ebff;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-item {z-index: 1;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-parallel-item:only-child::after {width: 0;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43 div.sk-container {display: inline-block;position: relative;}</style><div id=\"sk-36b3b78c-ce01-44cd-ade6-fae7fd48af43\" class\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"886868a5-d464-4187-9504-01c7205e79e0\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"886868a5-d464-4187-9504-01c7205e79e0\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('preprocessor',\n",
       "                 ColumnTransformer(remainder='passthrough', sparse_threshold=0,\n",
       "                                   transformers=[('cat-preprocessor',\n",
       "                                                  OrdinalEncoder(handle_unknown='use_encoded_value',\n",
       "                                                                 unknown_value=-1),\n",
       "                                                  ['workclass', 'education',\n",
       "                                                   'marital-status',\n",
       "                                                   'occupation', 'relationship',\n",
       "                                                   'race', 'sex',\n",
       "                                                   'native-country'])])),\n",
       "                ('classifier',\n",
       "                 HistGradientBoostingClassifier(max_leaf_nodes=4,\n",
       "                                                random_state=42))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"87cc7d45-55ea-4032-b2f7-6f3f55b007c4\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"87cc7d45-55ea-4032-b2f7-6f3f55b007c4\">preprocessor: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(remainder='passthrough', sparse_threshold=0,\n",
       "                  transformers=[('cat-preprocessor',\n",
       "                                 OrdinalEncoder(handle_unknown='use_encoded_value',\n",
       "                                                unknown_value=-1),\n",
       "                                 ['workclass', 'education', 'marital-status',\n",
       "                                  'occupation', 'relationship', 'race', 'sex',\n",
       "                                  'native-country'])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"60a215ef-375a-46c8-a416-e47c094949a1\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"60a215ef-375a-46c8-a416-e47c094949a1\">cat-preprocessor</label><div class=\"sk-toggleable__content\"><pre>['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8402ce7b-5a92-4d19-bdb9-7c0dd0b0f997\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"8402ce7b-5a92-4d19-bdb9-7c0dd0b0f997\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b4f24523-27fc-4e05-812b-bcabac589ca2\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"b4f24523-27fc-4e05-812b-bcabac589ca2\">remainder</label><div class=\"sk-toggleable__content\"><pre></pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"80cb6b36-c16b-4c7b-b8b0-1758dd0866d4\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"80cb6b36-c16b-4c7b-b8b0-1758dd0866d4\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b86c6ec1-ed84-4ed5-82da-c3216d0e0296\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"b86c6ec1-ed84-4ed5-82da-c3216d0e0296\">HistGradientBoostingClassifier</label><div class=\"sk-toggleable__content\"><pre>HistGradientBoostingClassifier(max_leaf_nodes=4, random_state=42)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('preprocessor',\n",
       "                 ColumnTransformer(remainder='passthrough', sparse_threshold=0,\n",
       "                                   transformers=[('cat-preprocessor',\n",
       "                                                  OrdinalEncoder(handle_unknown='use_encoded_value',\n",
       "                                                                 unknown_value=-1),\n",
       "                                                  ['workclass', 'education',\n",
       "                                                   'marital-status',\n",
       "                                                   'occupation', 'relationship',\n",
       "                                                   'race', 'sex',\n",
       "                                                   'native-country'])])),\n",
       "                ('classifier',\n",
       "                 HistGradientBoostingClassifier(max_leaf_nodes=4,\n",
       "                                                random_state=42))])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the moment this line is required to import HistGradientBoostingClassifier\n",
    "from sklearn.experimental import enable_hist_gradient_boosting\n",
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "model = Pipeline([\n",
    "    (\"preprocessor\", preprocessor),\n",
    "    (\"classifier\",\n",
    "     HistGradientBoostingClassifier(random_state=42, max_leaf_nodes=4))])\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86cd81f3",
   "metadata": {},
   "source": [
    "## Tuning using a grid-search\n",
    "\n",
    "Instead of manually writing the two `for` loops, scikit-learn provides a\n",
    "class called `GridSearchCV` which implement the exhaustive search implemented\n",
    "during the exercise.\n",
    "\n",
    "Let see how to use the `GridSearchCV` estimator for doing such search.\n",
    "Since the grid-search will be costly, we will only explore the combination\n",
    "learning-rate and the maximum number of nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a9d6e751",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:57:02.446910Z",
     "start_time": "2021-05-31T09:56:57.650246Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.42 s, sys: 109 ms, total: 5.53 s\n",
      "Wall time: 4.77 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 {color: black;background-color: white;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 pre{padding: 0;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-toggleable {background-color: white;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-item {z-index: 1;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-parallel-item:only-child::after {width: 0;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1 div.sk-container {display: inline-block;position: relative;}</style><div id=\"sk-c4c00eee-d95c-4f7c-a5bf-e46754fbc9e1\" class\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dbc73a3e-5ff2-4ec2-acad-8a01e8c515f6\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"dbc73a3e-5ff2-4ec2-acad-8a01e8c515f6\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=2,\n",
       "             estimator=Pipeline(steps=[('preprocessor',\n",
       "                                        ColumnTransformer(remainder='passthrough',\n",
       "                                                          sparse_threshold=0,\n",
       "                                                          transformers=[('cat-preprocessor',\n",
       "                                                                         OrdinalEncoder(handle_unknown='use_encoded_value',\n",
       "                                                                                        unknown_value=-1),\n",
       "                                                                         ['workclass',\n",
       "                                                                          'education',\n",
       "                                                                          'marital-status',\n",
       "                                                                          'occupation',\n",
       "                                                                          'relationship',\n",
       "                                                                          'race',\n",
       "                                                                          'sex',\n",
       "                                                                          'native-country'])])),\n",
       "                                       ('classifier',\n",
       "                                        HistGradientBoostingClassifier(max_leaf_nodes=4,\n",
       "                                                                       random_state=42))]),\n",
       "             n_jobs=4,\n",
       "             param_grid={'classifier__learning_rate': (0.05, 0.1, 0.5, 1, 5),\n",
       "                         'classifier__max_leaf_nodes': (3, 10, 30, 100)})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e10d5518-e6e3-4c0c-85e7-0d2e90c2d209\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"e10d5518-e6e3-4c0c-85e7-0d2e90c2d209\">preprocessor: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(remainder='passthrough', sparse_threshold=0,\n",
       "                  transformers=[('cat-preprocessor',\n",
       "                                 OrdinalEncoder(handle_unknown='use_encoded_value',\n",
       "                                                unknown_value=-1),\n",
       "                                 ['workclass', 'education', 'marital-status',\n",
       "                                  'occupation', 'relationship', 'race', 'sex',\n",
       "                                  'native-country'])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"14703427-84eb-4e91-a247-0aa384eb120d\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"14703427-84eb-4e91-a247-0aa384eb120d\">cat-preprocessor</label><div class=\"sk-toggleable__content\"><pre>['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"1da82126-ceed-4f72-ad4e-c7749342b822\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"1da82126-ceed-4f72-ad4e-c7749342b822\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"02955a2c-bfd3-4433-a993-36b52728f472\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"02955a2c-bfd3-4433-a993-36b52728f472\">remainder</label><div class=\"sk-toggleable__content\"><pre></pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"049a7715-4851-4094-9f6f-b6efc09fefd5\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"049a7715-4851-4094-9f6f-b6efc09fefd5\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"9339bee6-dcd8-4f3b-8896-a3574f7ec040\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"9339bee6-dcd8-4f3b-8896-a3574f7ec040\">HistGradientBoostingClassifier</label><div class=\"sk-toggleable__content\"><pre>HistGradientBoostingClassifier(max_leaf_nodes=4, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=2,\n",
       "             estimator=Pipeline(steps=[('preprocessor',\n",
       "                                        ColumnTransformer(remainder='passthrough',\n",
       "                                                          sparse_threshold=0,\n",
       "                                                          transformers=[('cat-preprocessor',\n",
       "                                                                         OrdinalEncoder(handle_unknown='use_encoded_value',\n",
       "                                                                                        unknown_value=-1),\n",
       "                                                                         ['workclass',\n",
       "                                                                          'education',\n",
       "                                                                          'marital-status',\n",
       "                                                                          'occupation',\n",
       "                                                                          'relationship',\n",
       "                                                                          'race',\n",
       "                                                                          'sex',\n",
       "                                                                          'native-country'])])),\n",
       "                                       ('classifier',\n",
       "                                        HistGradientBoostingClassifier(max_leaf_nodes=4,\n",
       "                                                                       random_state=42))]),\n",
       "             n_jobs=4,\n",
       "             param_grid={'classifier__learning_rate': (0.05, 0.1, 0.5, 1, 5),\n",
       "                         'classifier__max_leaf_nodes': (3, 10, 30, 100)})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid = {\n",
    "    'classifier__learning_rate': (0.05, 0.1, 0.5, 1, 5),\n",
    "    'classifier__max_leaf_nodes': (3, 10, 30, 100)}\n",
    "model_grid_search = GridSearchCV(model, param_grid=param_grid,\n",
    "                                 n_jobs=4, cv=2)\n",
    "model_grid_search.fit(data_train, target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57cfb31",
   "metadata": {},
   "source": [
    "Finally, we will check the accuracy of our model using the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "17f1b6fd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:57:04.649784Z",
     "start_time": "2021-05-31T09:57:04.541501Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The test accuracy score of the grid-searched pipeline is: 0.88\n"
     ]
    }
   ],
   "source": [
    "accuracy = model_grid_search.score(data_test, target_test)\n",
    "print(\n",
    "    f\"The test accuracy score of the grid-searched pipeline is: \"\n",
    "    f\"{accuracy:.2f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dd7d616",
   "metadata": {},
   "source": [
    "<div class=\"admonition warning alert alert-danger\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Warning</p>\n",
    "<p>Be aware that the evaluation should normally be performed in a\n",
    "cross-validation framework by providing <tt class=\"docutils literal\">model_grid_search</tt> as a model to\n",
    "the <tt class=\"docutils literal\">cross_validate</tt> function.</p>\n",
    "<p class=\"last\">Here, we are using a single train-test split to highlight the specificities\n",
    "of the <tt class=\"docutils literal\">model_grid_search</tt> instance. We will show such examples in the last\n",
    "section of this notebook.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be4a5433",
   "metadata": {},
   "source": [
    "The `GridSearchCV` estimator takes a `param_grid` parameter which defines\n",
    "all hyperparameters and their associated values. The grid-search will be in\n",
    "charge of creating all possible combinations and test them.\n",
    "\n",
    "The number of combinations will be equal to the product of the\n",
    "number of values to explore for each parameter (e.g. in our example 4 x 4\n",
    "combinations). Thus, adding new parameters with their associated values to be\n",
    "explored become rapidly computationally expensive.\n",
    "\n",
    "Once the grid-search is fitted, it can be used as any other predictor by\n",
    "calling `predict` and `predict_proba`. Internally, it will use the model with\n",
    "the best parameters found during `fit`.\n",
    "\n",
    "Get predictions for the 5 first samples using the estimator with the best\n",
    "parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "421400e3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:57:52.521386Z",
     "start_time": "2021-05-31T09:57:52.501647Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' <=50K', ' <=50K', ' >50K', ' <=50K', ' >50K'], dtype=object)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_grid_search.predict(data_test.iloc[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db13b9f7",
   "metadata": {},
   "source": [
    "You can know about these parameters by looking at the `best_params_`\n",
    "attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "05ce9f20",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:57:56.247298Z",
     "start_time": "2021-05-31T09:57:56.245148Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best set of parameters is: {'classifier__learning_rate': 0.1, 'classifier__max_leaf_nodes': 30}\n"
     ]
    }
   ],
   "source": [
    "print(f\"The best set of parameters is: \"\n",
    "      f\"{model_grid_search.best_params_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4dee09d",
   "metadata": {},
   "source": [
    "The accuracy and the best parameters of the grid-searched pipeline are\n",
    "similar to the ones we found in the previous exercise, where we searched the\n",
    "best parameters \"by hand\" through a double for loop.\n",
    "\n",
    "In addition, we can inspect all results which are stored in the attribute\n",
    "`cv_results_` of the grid-search. We will filter some specific columns\n",
    "from these results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7cf24f0c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:58:10.753589Z",
     "start_time": "2021-05-31T09:58:10.732973Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_classifier__learning_rate</th>\n",
       "      <th>param_classifier__max_leaf_nodes</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.366540</td>\n",
       "      <td>0.040701</td>\n",
       "      <td>0.110018</td>\n",
       "      <td>0.009910</td>\n",
       "      <td>0.1</td>\n",
       "      <td>30</td>\n",
       "      <td>{'classifier__learning_rate': 0.1, 'classifier...</td>\n",
       "      <td>0.868912</td>\n",
       "      <td>0.867213</td>\n",
       "      <td>0.868063</td>\n",
       "      <td>0.000850</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.456051</td>\n",
       "      <td>0.012136</td>\n",
       "      <td>0.132355</td>\n",
       "      <td>0.007661</td>\n",
       "      <td>0.05</td>\n",
       "      <td>30</td>\n",
       "      <td>{'classifier__learning_rate': 0.05, 'classifie...</td>\n",
       "      <td>0.868530</td>\n",
       "      <td>0.866667</td>\n",
       "      <td>0.867598</td>\n",
       "      <td>0.000932</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.145535</td>\n",
       "      <td>0.000616</td>\n",
       "      <td>0.064308</td>\n",
       "      <td>0.001329</td>\n",
       "      <td>0.5</td>\n",
       "      <td>30</td>\n",
       "      <td>{'classifier__learning_rate': 0.5, 'classifier...</td>\n",
       "      <td>0.867056</td>\n",
       "      <td>0.865902</td>\n",
       "      <td>0.866479</td>\n",
       "      <td>0.000577</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.325728</td>\n",
       "      <td>0.022468</td>\n",
       "      <td>0.104191</td>\n",
       "      <td>0.001981</td>\n",
       "      <td>0.1</td>\n",
       "      <td>10</td>\n",
       "      <td>{'classifier__learning_rate': 0.1, 'classifier...</td>\n",
       "      <td>0.866783</td>\n",
       "      <td>0.866066</td>\n",
       "      <td>0.866425</td>\n",
       "      <td>0.000359</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.182203</td>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.084572</td>\n",
       "      <td>0.000317</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3</td>\n",
       "      <td>{'classifier__learning_rate': 0.5, 'classifier...</td>\n",
       "      <td>0.864872</td>\n",
       "      <td>0.866776</td>\n",
       "      <td>0.865824</td>\n",
       "      <td>0.000952</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "6        0.366540      0.040701         0.110018        0.009910   \n",
       "2        0.456051      0.012136         0.132355        0.007661   \n",
       "10       0.145535      0.000616         0.064308        0.001329   \n",
       "5        0.325728      0.022468         0.104191        0.001981   \n",
       "8        0.182203      0.000471         0.084572        0.000317   \n",
       "\n",
       "   param_classifier__learning_rate param_classifier__max_leaf_nodes  \\\n",
       "6                              0.1                               30   \n",
       "2                             0.05                               30   \n",
       "10                             0.5                               30   \n",
       "5                              0.1                               10   \n",
       "8                              0.5                                3   \n",
       "\n",
       "                                               params  split0_test_score  \\\n",
       "6   {'classifier__learning_rate': 0.1, 'classifier...           0.868912   \n",
       "2   {'classifier__learning_rate': 0.05, 'classifie...           0.868530   \n",
       "10  {'classifier__learning_rate': 0.5, 'classifier...           0.867056   \n",
       "5   {'classifier__learning_rate': 0.1, 'classifier...           0.866783   \n",
       "8   {'classifier__learning_rate': 0.5, 'classifier...           0.864872   \n",
       "\n",
       "    split1_test_score  mean_test_score  std_test_score  rank_test_score  \n",
       "6            0.867213         0.868063        0.000850                1  \n",
       "2            0.866667         0.867598        0.000932                2  \n",
       "10           0.865902         0.866479        0.000577                3  \n",
       "5            0.866066         0.866425        0.000359                4  \n",
       "8            0.866776         0.865824        0.000952                5  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_results = pd.DataFrame(model_grid_search.cv_results_).sort_values(\n",
    "    \"mean_test_score\", ascending=False)\n",
    "cv_results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd6e128b",
   "metadata": {},
   "source": [
    "Let us focus on the most interesting columns and shorten the parameter\n",
    "names to remove the `\"param_classifier__\"` prefix for readability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ce3f4c51",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:58:17.552885Z",
     "start_time": "2021-05-31T09:58:17.547069Z"
    }
   },
   "outputs": [],
   "source": [
    "# get the parameter names\n",
    "column_results = [f\"param_{name}\" for name in param_grid.keys()]\n",
    "column_results += [\n",
    "    \"mean_test_score\", \"std_test_score\", \"rank_test_score\"]\n",
    "cv_results = cv_results[column_results]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9a810763",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:58:18.989357Z",
     "start_time": "2021-05-31T09:58:18.980336Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>max_leaf_nodes</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.1</td>\n",
       "      <td>30</td>\n",
       "      <td>0.868063</td>\n",
       "      <td>0.000850</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.05</td>\n",
       "      <td>30</td>\n",
       "      <td>0.867598</td>\n",
       "      <td>0.000932</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.5</td>\n",
       "      <td>30</td>\n",
       "      <td>0.866479</td>\n",
       "      <td>0.000577</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.1</td>\n",
       "      <td>10</td>\n",
       "      <td>0.866425</td>\n",
       "      <td>0.000359</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0.865824</td>\n",
       "      <td>0.000952</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.5</td>\n",
       "      <td>10</td>\n",
       "      <td>0.865824</td>\n",
       "      <td>0.000031</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.05</td>\n",
       "      <td>100</td>\n",
       "      <td>0.865797</td>\n",
       "      <td>0.001259</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.1</td>\n",
       "      <td>100</td>\n",
       "      <td>0.864732</td>\n",
       "      <td>0.000795</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.05</td>\n",
       "      <td>10</td>\n",
       "      <td>0.862029</td>\n",
       "      <td>0.000222</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.5</td>\n",
       "      <td>100</td>\n",
       "      <td>0.859491</td>\n",
       "      <td>0.001069</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>0.858863</td>\n",
       "      <td>0.004036</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0.857389</td>\n",
       "      <td>0.003545</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.1</td>\n",
       "      <td>3</td>\n",
       "      <td>0.853266</td>\n",
       "      <td>0.000515</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>0.851028</td>\n",
       "      <td>0.002707</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>0.835194</td>\n",
       "      <td>0.004609</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.05</td>\n",
       "      <td>3</td>\n",
       "      <td>0.827196</td>\n",
       "      <td>0.000214</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>5</td>\n",
       "      <td>100</td>\n",
       "      <td>0.727579</td>\n",
       "      <td>0.055780</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>5</td>\n",
       "      <td>30</td>\n",
       "      <td>0.638062</td>\n",
       "      <td>0.144696</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>0.527527</td>\n",
       "      <td>0.175411</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>0.283476</td>\n",
       "      <td>0.003775</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   learning_rate max_leaf_nodes  mean_test_score  std_test_score  \\\n",
       "6            0.1             30         0.868063        0.000850   \n",
       "2           0.05             30         0.867598        0.000932   \n",
       "10           0.5             30         0.866479        0.000577   \n",
       "5            0.1             10         0.866425        0.000359   \n",
       "8            0.5              3         0.865824        0.000952   \n",
       "9            0.5             10         0.865824        0.000031   \n",
       "3           0.05            100         0.865797        0.001259   \n",
       "7            0.1            100         0.864732        0.000795   \n",
       "1           0.05             10         0.862029        0.000222   \n",
       "11           0.5            100         0.859491        0.001069   \n",
       "13             1             10         0.858863        0.004036   \n",
       "12             1              3         0.857389        0.003545   \n",
       "4            0.1              3         0.853266        0.000515   \n",
       "14             1             30         0.851028        0.002707   \n",
       "15             1            100         0.835194        0.004609   \n",
       "0           0.05              3         0.827196        0.000214   \n",
       "19             5            100         0.727579        0.055780   \n",
       "18             5             30         0.638062        0.144696   \n",
       "17             5             10         0.527527        0.175411   \n",
       "16             5              3         0.283476        0.003775   \n",
       "\n",
       "    rank_test_score  \n",
       "6                 1  \n",
       "2                 2  \n",
       "10                3  \n",
       "5                 4  \n",
       "8                 5  \n",
       "9                 6  \n",
       "3                 7  \n",
       "7                 8  \n",
       "1                 9  \n",
       "11               10  \n",
       "13               11  \n",
       "12               12  \n",
       "4                13  \n",
       "14               14  \n",
       "15               15  \n",
       "0                16  \n",
       "19               17  \n",
       "18               18  \n",
       "17               19  \n",
       "16               20  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def shorten_param(param_name):\n",
    "    if \"__\" in param_name:\n",
    "        return param_name.rsplit(\"__\", 1)[1]\n",
    "    return param_name\n",
    "\n",
    "\n",
    "cv_results = cv_results.rename(shorten_param, axis=1)\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71da101f",
   "metadata": {},
   "source": [
    "With only 2 parameters, we might want to visualize the grid-search as a\n",
    "heatmap. We need to transform our `cv_results` into a dataframe where:\n",
    "\n",
    "- the rows will correspond to the learning-rate values;\n",
    "- the columns will correspond to the maximum number of leaf;\n",
    "- the content of the dataframe will be the mean test scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f484c24e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:58:25.941695Z",
     "start_time": "2021-05-31T09:58:25.915632Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>max_leaf_nodes</th>\n",
       "      <th>3</th>\n",
       "      <th>10</th>\n",
       "      <th>30</th>\n",
       "      <th>100</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>learning_rate</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.05</th>\n",
       "      <td>0.827196</td>\n",
       "      <td>0.862029</td>\n",
       "      <td>0.867598</td>\n",
       "      <td>0.865797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.10</th>\n",
       "      <td>0.853266</td>\n",
       "      <td>0.866425</td>\n",
       "      <td>0.868063</td>\n",
       "      <td>0.864732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.50</th>\n",
       "      <td>0.865824</td>\n",
       "      <td>0.865824</td>\n",
       "      <td>0.866479</td>\n",
       "      <td>0.859491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.00</th>\n",
       "      <td>0.857389</td>\n",
       "      <td>0.858863</td>\n",
       "      <td>0.851028</td>\n",
       "      <td>0.835194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5.00</th>\n",
       "      <td>0.283476</td>\n",
       "      <td>0.527527</td>\n",
       "      <td>0.638062</td>\n",
       "      <td>0.727579</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "max_leaf_nodes       3         10        30        100\n",
       "learning_rate                                         \n",
       "0.05            0.827196  0.862029  0.867598  0.865797\n",
       "0.10            0.853266  0.866425  0.868063  0.864732\n",
       "0.50            0.865824  0.865824  0.866479  0.859491\n",
       "1.00            0.857389  0.858863  0.851028  0.835194\n",
       "5.00            0.283476  0.527527  0.638062  0.727579"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivoted_cv_results = cv_results.pivot_table(\n",
    "    values=\"mean_test_score\", index=[\"learning_rate\"],\n",
    "    columns=[\"max_leaf_nodes\"])\n",
    "\n",
    "pivoted_cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ace547f5",
   "metadata": {},
   "source": [
    "We can use a heatmap representation to show the above dataframe visually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "11997cb9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-31T09:58:28.747173Z",
     "start_time": "2021-05-31T09:58:28.385695Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEfCAYAAAB8jtA1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABOYElEQVR4nO3dd3xUVdrA8d8zIZQk1NCLUhVERV1R6U1dG4ii7tpZ9RXXsrqudUFF17LWVdeydnRRFEGUpoBUESurovROkIRAqAlFkjzvH+cmTIZJMpnMZDLh+e7nfsLce+bec4+z88wp9xxRVYwxxphI88U6A8YYY6omCzDGGGOiwgKMMcaYqLAAY4wxJioswBhjjIkKCzDGGGOiwgKMMcbEIRFJEZHnRSRdRPaKyPciMijE9/5JRH4Wkf0isllEXhWR1GLS/kVEVnhpV4vIXSISUuywAGOMMfFpAnA5MAI4F1gCTBCRc0p6k4j8DXgT+BoYBAz3/s4QkcSAtCOAfwHvA78H3gAeAR4NJYNiD1oaY0x88YLIFOBCVZ3g7RPgCyBVVTsV876aQCYwS1UH++3vA8wBrlfV17x9qcBG4FVVvdUv7SPAXUAbVd1YUj6tBmOMMfHnAmAn8EnBDnW1hbeBjiJyTDHv6wzUBib771TVucAu4CK/3WcBNb1z+hsFVMPVekpkAcYYY+LPscASVc0P2L/I73gw1b2/+4Mc2x/wvmMBBRb7J1LVlcDeEq5RqFppCYwxxkSHiOwoLY2q1guyOxVYEWT/Nr/jwSwH8oHuwH/98nEU0Aj4LeAae1Q1WDDaXsI1Ch0mAWaFdTQZU4LMfctinYVKo3HNQVLecyQdeXmo3zk7y3GZkq4R9JiqbhORd4FrROR/wHigFfAakIcLPuW6hr/DJMAYY0zFkRB7H4qpnYQii+A1iAbe321BjhX4MyDAf4BXcUHlv0A6RZu9soBkEakRpBZTv5RrANYHY4wxESfiC2krh8VApyDPoxzn/f2luDeqao6qXgk0BLoATVR1KNARNwrN/xqCGxjgd2/SHqhV0jUKWIAxxpgIq4AAMwGoBwwM2H8VsFxVl5R2AlXdrqqLVHWriFwAdABe8kvyKa7j/8qAt14N5AKTSruGNZEZY0yEuUdSomoqMBt4w3teZS3ui78ncL5fPuYAfVRV/PZdAjQBluJqIv2AvwAPqeq3BelUNUtEHgPuE5Gd3vW6AXcDz6pqWmmZtABjjDERF93GIVVVERmMe6L+UVxtZgnuwcvSahZ5wLVAe+/1IuAyVR0bJO1DuIEINwH3ApuAB4DHQ8nnYfIkv40iM6YkNorsoEiMIqvX/oaQvnN2rPpP1Ks6sWQ1GGOMibBQR5FVdRZgjDEmwsrZgV9lWIAxxpgIswDjWIAxxpgIswDjWIAxxpgI80lCrLNQKViAMcaYCLMajGMBxhhjIswCjGMBxhhjIs4CDFiAMcaYiLMajGMBxhhjIswCjGMBJgLmzVvII4+8Rn5+PhdffAbXX39xkeMTJ87htdfGA5CcXJORI2+kY8c2AIwa9TEffjgdEeGoo1rz2GO3UqNG9UOuES9KK4tvvvmZG298mJYtmwBwxhnduPnmS9m//zcuv/wefvvtAHl5efz+9z34y18uj8UtRExpZQGuPB599DVyc3OpX78Oo0f/s/BYXl4eQ4bcTpMmDXjllQcqMusR982Xy3ju8Ynk5+dz3gWncMW1/Yscf2/UHGZM/R8Aebn5rF+byaQ5I6lRM5Fb/vQyvx3IJS83n75nHMe1N/4+FrdQJj6xr1awAFNueXl5PPTQf3jrrX/QpEkqF110O/37n0r79kcUpmnZsgmjRz9G3bopzJ37Pffd9wIffvg0mzdn8c47k5g69SVq1qzBrbf+kylT5nHhhafH8I7CF0pZAJx88jGHfGFWr57I228/QnJyLQ4cyOWyy+6md+/fccIJHSvyFiImlLLYtSubBx98mddfH0nz5o3JytpR5BzvvDOJdu1akp29p4JzH1l5efk88+gE/vXK9TRqUpf/u+x5evTtTJt2TQrTXDa0L5cN7QvAl3OWMHb0POrUTUJVefb1YSQl1SD3QB43Dn2R03p2pPPxR8bobkJTAbMpx4VKU48TkYYicrSIHCUiDWOdn1AtWrSSI49sRqtWTalePZFzz+3NzJnfFElz0kmdqFs3BYATTuhIRsbWwmN5efns2/cbubl57Nu3n8aNGxCvQimL4ogIycm1AMjNzSU3Nzeu/08aSllMmjSXM87oRvPmjQFITa1XeCwjYytz5nzHRRedWZHZjoqlv2ygRauGNG+ZSmJiNQacdQLz5ywuNv3nn/3AgLNPBNznIimpBgC5uXnk5ubj1sCq3CpgPZi4ENM7FJHmIvKiiKQDm3HTTS8FNotIhnesZSnn2FHaFs172Lw5i6ZND8bDJk1S2bw5q9j048ZNp3fv3xWmveaaC+jX7xp69ryKlJRkevY8KZrZjapQy+LHH5czaNAtXHfdA6xcub5wf15eHuef/xe6d7+S7t1PpEuXoysk39EQSlmsW7eJXbuyufLKe7nwwtv4+ONZhcceffQ17rzzT/h88f8ltCVzF42b1it83ahxXbZuDr4U/b69v/HNl8vpe/pxhfvy8vL50yXPMKjfg3Q9rQOdjz8i6HsrE8EX0lbVxayJTEQ6AXOBGriV0RZxcI3nBrilPy8HLhGR3qq6NCYZLUWw5Q6K++X99deLGDduBu+955ZS2Lkzm5kzv2HmzNepXTuZW2/9J598Mpvzz+8X1TxHSyhl0blzO2bNeoPk5FrMnfs9N930CNOnvwpAQkICn3zyPLt2ZXPTTY+yYsV6jjqqcjeFFCeUssjLy2Px4tWMGvUw+/bt549/vJMuXY5m3bpfadCgLsce255vvvm5orIcPcGWBCnm/yNfzl3CcSe0pk7dpMJ9CQk+3hp7O7t37WX4X99mzcoM2nZoGq3cRsThUDsJRSz7YJ4CVgPnquq2YAlEpAEwBXgSOC9YGlWtV/qlorceTNOmDYs0eW3enBW0mWvZsrWMGPFvXnttJPXr1wFgwYIfadmyCQ0a1AXgzDO788MPS+M2wIRSFikpB784+vQ5mQcffJlt23YWlgFAnTopnHrqcXzxxcK4DTChlEXTpqnUr1+HpKSaJCXV5OSTj2XZsrUsWbKaWbO+Zd68hezf/xvZ2Xu4446neeqpv1X0bUREoyZ1yczYUfh6S+ZOGjauEzTtzM9+5HSveSxQ7Tq1OLFrW75ZsMwCTJyIZSn0Av5RXHAB8I49DPSusFyV0XHHdWDduk2kpWXw228HmDJlHv37n1IkzaZNmdxyy2M88cTttGnTonB/8+aN+OmnZezduw9V5auvfqJdu1YVfQsRE0pZbNmyvfDX/aJFK8jPz6d+/Tps27aTXbuyAdi3bz8LFvxI27Ylto5WaqGUxYABp/H994vJzc1j7959LFq0nHbtWvG3v13NvHmjmDXrDZ555i5OO+34uA0uAB07t2Ljhq1s2riNAwdymfnZj/Tsc8wh6bJ37+XHhWvo2bdz4b7t27LZvWsvAPv3HeD7r1dxROvGFZb3cIlUC2mr6mJ5h/lAKDPCJXhpK6Vq1RK4//4buO66B8jLy2fIkNPp0OFIxoz5FIBLLz2bF198nx07dvHggy8Drinoo4/+RZcuR/P73/fgggtuo1q1BDp1assf/nBWLG+nXEIpi2nTvmTMmKkkJCRQs2YNnnnmLkSEzMxt3HPPs+Tl5aOaz1ln9aRfv1NKuWLlFUpZtGvXil69fsegQbfg8wkXXXRm3NbYSlKtWgJ/vXcwf/uzG7J97uBTaNO+KR+P/QqAwZd0A2DerF/o2u0oaiUdHKaftXUXj474gLz8fDRf6XdmF3oECU6VjdVgnJgtmSwiHwNHAGer6uZi0jQGPgU2qOoF4V/Nlkw2piS2ZPJBkVgyuf3Jz4X0nbPq+1sr/5C4cohlDeZ2YD6wRkRmAL/gOvkVSAWOBc4AdgKXxCqTxhhTVofDCLFQxCzAqOoaEekC3AkMBgZycIC7AmuAF4GnVDUzJpk0xpgwWBOZE9NeJlXdAtwF3CUiNYH63qHtqrovdjkzxphySKjSLV8hqzTDGLyAkh7rfBhjTLnF8SwUkVRpAowxxlQZFmCAOAgwIpIP5Ktqpc+rMcYAlWiWx9iKhy/tebhOf2OMiQtqNRggDgKMqvaNdR6MMaZMLL4AcRBgjDEm7iRYGxnEsKXQW/elesC+riIyXUR2ichOEflMRILPfGeMMZWVhLhVcbEMs0uBEwpeiMhJuP6WE3HT908FTga+EJHKP/mQMcYU8EloWxUXyyaywNJ9EPgV6FEwN5mINAUWAMNxa8MYY0zlZ538QOUaTNcL+Kf/xJeqmgE8AcTnAinGmMOTNZEBlauTPxnXbBZoKW7yS2OMiQ+HQfNXKGIdYHp5zWAAW3FLJQeqD+RUXJaMMaZ81AIMEPsA8wRFK4p9cR38/k4C1lVQfowxpvwswACxDTDB+lV2BtnXFng/ynkxxpjIsfgCxHY9mLkhpruivNdqf+E35T1F1ZFvs+4U0NrVS090mMg7sm6ss1BprHs4AiexUWRA7JvIjDGm6rEmMsACjDHGRJ4FGMACjDHGRJ41kQEWYIwxJvIq0yPsMWQBxhhjIs1qMIDFWWOMibwKmCpGRFJE5HkRSReRvSLyvYgMCvG9Q0RkgYhs97avROSSIOm0mO2GUK5jNRhjjImwCnqSfwLuQfS7gLXAUGCCiAxU1anFvUlErgZGAeOBf3i7rwU+EJEUVX0z4C0fAM8G7FsTSgYtwBhjTKRFOcCIyDnA6cCFqjrB2zcb92D607jlTorzJ2A9cImq5nvvnYYLGlcBgQEmQ1W/Dief1kRmjDGRFv31YC7AzXzyScEOVVXgbaBjKWtoHQCyC4KL9958IBvYX55MBQo7wIhIbxF5WEReE5GO3r4Ub3+9iOXQGGPijUhIm4jsKG0r5grHAkv8g4Rnkd/x4rwAdBKR4SLSUEQaichw4GjgX0HSX+X18ewTkW+C9dUUp8xNZCKSALwHXITrplJgDLAMyAU+Bp4CHi3ruY0xpkqIfhdMKrAiyP5tfseDUtVPvMEAo4GCiXFygItV9bOA5O/imtvSgGbAjbi+mmaq+lxpmQynD+ZuYAhwO/AZfmu4qOo+EZkAnIMFGGPM4SrE5i9VrVeOq5Q0sWCxx0TkDFwlYQyuoz8Bt2LwGBG5SFWn+OXvioD3jgPmAA+LyKuqurekDIYTYK4C3lHV50QkWJRcigswxhhzeIr+KLIsgtdSCtbU2hbkGCIiuH6aWarqP9T4MxFpCfwbmBLsveD6akRkNG4F4mOB70rKZDh9MK2Br0o4vgO3SJgxxhyWNEFC2sphMa4fJfA7/Djv7y/FvK8Jrqnr+yDHvgfaiEjNUq5dcM3A/p9iE5bFboKvPFmgPbAljPMaY0zVEGInfzlMAOoBAwP2XwUsV9UlxbxvO7APOCXIsdOALFXdV9xFvYB2OS4OLC4tk+E0kc0HrhCRJ4JcvD5wDa5vxhhjDk/RbyKbCswG3vC6KtYCVwM9gfMLEonIHKCPqgqAqu4Xkf8At4nI68A4XB/MVd57R/i99w7cyLJZQDrQFPizl+6mkgJRgXACzCO4IDML9zQoQBcR6QDcAyQD/wzjvHGr94nNGXHNyST4hLGfr+KVCUUDe0pSIs/c2pNmjZKo5vPx+sQljJ+1GoDaSYk8dlM3OrSqhwL3vrCAH1ZsjcFdREbvE5sz4rqurixmrOKVj4rW1FOSEnnmrz1p1jCZagk+Xv948cGySE7ksZu60+GIeqiqK4vlcVwWxzXlvitPIsEnfDBnDa9MXlrkeEqtRJ7582k0T00iwefj9anLGP/FWgDmPjOQnH0HyMtX8vKUwQ9Mj8UtREyfDg25/5xOriwWbuTleUUfBK9doxr/urgLLerWJMEnvPblWj7836+Fx30Ck/7cg4xd+7h29MKKzn7ZRfkJQ1VVERmMG0z1KK42swT34GXgsvOB7sCN+r0eNxo4Hzci7UrcqLECy3HBarB3/hxgITAohGsAYQQYVf1eRC4E3gDe8nY/hRuYlwlcUEL1rMrx+YSR/3cKVz/4ORlZe/joibOZ+d1GVm08uPrzlWcfzcqNO7j+sdk0qFOD6f8+n4nz1nIgN5/7ru3KvB82cfOT80is5qNm9YQY3k35+HzCyGGncvUDM1xZPHkOM79NK1oW5xzNyrSdXP+IVxYvDvYri1OY979fufmJua4sasRxWYgw8uqTufrx2WRs28uEh85g5v9+ZdWmXYVprjy9A6t+3cX1z3xBg9o1mPHEOUxcsJ4Dea5p+/JHZ7E9+7dY3ULE+AQeGtiZK976loxd+5h4Q3dmLM1k1ZbswjRXnnYEqzKzuW70QhokVWfWbb34+KdNHMhzg6H+1K01q7Zkk1IjTiYfqYDJLlV1F3CztxWXpm+QfXnAK95W0vknASEFkuKEFWe9eW5aA4Nww5bvxQ1dbquq8f1Tq4y6tE9lffpu0jZncyA3nynz13P6Ka2KpFF1v1YBkmomsjN7P7l5+aTUSqTrMU0Y+/kqAA7k5rN7z4EKv4dI6dIhsCzWcfqpZSiLzo2LlkVOHJdFuwas37ybtC05HMjLZ/LXGzj9dy2KpFGU5JruCzOpZjV25vxGbn6p/aZx54SW9ViflUPa9r0cyFMm/ZzOmZ0aF02kkOz9oEiqkcCOvQfI9Zb3blqnJv2PbsT7C9MqOuth0wRfSFtVF86DlkcAW7zxz5O9zf94LaCRqm4o43kb4obdKbBNVeOibaRJahLpWTmFrzOycujSoWGRNP+duoxX7u3HgjeGkFwzkVuf+QJVaNUkhW279vH4zd3p1Lo+v6zJ4h9vfM/e/bkVfRsR0aRBEulb/ctiz6FlMWUZrwzvz4I3LyK5ViK3PjXPlUXTFLbt3M/jf+lOp9YN+GV1Fv94/bv4LYv6tUjftqfwdca2vXRpV3RszH9nrOTVv/biq3+fT3LNavzlhQWo9/SCooy6uy8ojJm9mvdnr67A3EdWkzo12bTzYHN9+q59nNCyXpE0b3+9ntev+B3f3t2P5OrVuPmDHwvL4v5zOvHYtOXxU3sBm4TLE04xrMXNg1OcQV6aUolIcxF5UUTSgc24NsSlwGYRyfCOtSzlHOFOtRARwSrCgU849TqxOUvXbaf7teMZ9LcpPHDdKaTUSiQhQejctgHvTVvBoDumsGdfLsMu7BzN7EaVBGkWCFoWa7fR/ZpxDPrrZB643isLn4/O7Rrw3qcrGHT7ZFcWQ0qa7aJyC1YWgYXR67imLNmwg263fMLA4dMYefXvSPFqNJc89Dnn3zeda56ayxWnt6fr0Y0qINfREfT/I1q0MHp3aMSS9F2c8vhsznnxSx4aeAwpNarR/+hGZOXs5xe/psW4EP25yOJCOAGmtFLxUfITpu4kIp2AH4ErgJm4ZrZh3nYvMAM3HO4HL22llJG1h2apyYWvm6Ymk7mt6MOtQ/q3Y9rXrkK3PmM3GzOzaduiDhlZe8jI2sNPK11l7bOvNtC5bUkjwCu3jKwcmjX0L4skMv1+xQMMGdC+aFlszqZtyzpkZOUElMX6+C6LbXto1iCp8HXTBrXYvKPo5+Ki3m2Z9t1GANZnZrNxSw5tm9cBIHOH+8WftWs/07//9ZDaTzzJ2LWP5nUPPlrRrE5NMncXnVPx4pNa8NmSzQCs37aHtO17adcwmZOPqM/pHZsw/299+PclJ9C9bSr/uuj4Cs1/WKI/TDkuhFuRKymAdMI9bFmap4DVQBtVvUJVn1DV173tCVW9Ejf19CrgyWIzolqvtC302yq7RauyOLJZbVo2TiGxmo9zex7JzO+KthVv2pJD9+ObApBatyZtmtchbXM2W3fsI31rDm28L5XuxzdlVdrOQ64RLxatDCyL1sz8NlhZNAO8smhRl7SMYGXRLL7LYs02WjetTctGySQm+DjvtCOY6TcqCmBTVg7dOzcBILVODdo0rU1aZja1aiQU9s3UqpFAr+OasiKOy+KnX3fSOjWZlvVrkZggDDyuGTOWZRZJs2nHPnq0cw+mN0yuTtuGyWzYvocnZqyg25Oz6fn0XG4Z+yML1mTx13GLgl2mcrEaDBBiH4y3QM3VfrtGiMj/BUnaADd9wIQQTtsL+KOqBp3SAEBVt4nIw7g5cyqlvHzlwde/5a37B5DgEz6cuYqVaTu59MwOAIyZvpIXP/yZJ27pzpR/nYeI8OR//8d27xfcQ69/xzO39SSxmo+0zdnc/cKCWN5OueTlKw++9i1vPXA6CQnCh597ZfH7owAYM20FL45dxBO39mDKcwMR4Ml3Fh4si9e+5Znbe5JYLYG0zbu5+/k4L4t3FjLqzj74fD7GzVvDyl93cWn/dgCMmbWaFz5ezBPXn8bUR89CBJ744Ce2Z/9Gq0bJvHxbTwASfD4mfbWeeT9nxPJ2yiUvX7l/8hLeudobvr5wIyszs7m8qxsA8u53aTw/ZxVPDTmez27uiQj8c9pytsfxgJcKmOwyLkhgW2jQRCK3Ard5L4/AzYOTE5BMcesJfA0MV9USn+b3+kauLG08tTfr5zvlqYm0v/C/pd/k4SLfiqKA1q4e6yxUGnlH1o11FiqNdQ+fXe7w0Hr41JD+j7bukXOqdCgKqQbjTcv8HICI5AO3qep75bz2HOBBEflWVTcHSyAijYEHcE+sGmNMfDgM+ldCEc6DlpEagHc7bkaANSIyAzc52zZcTSgV19R2Bm7VtpAXuDHGmJizYcpAeFPFRISqrhGRLsCduKkIBnKw5VJx60O/CDylqplBT2KMMZWR1WCAMAOMiLQD/gqcipuaPzBeq6q2K+08Xj/NXcBd3hTRBdP8bw9lIjVjjKmUDoMRYqEI50n+43BNWzVwk6G1xU3bnIqbbXM1sLGs5/UCSnpZ32eMMZXN4TANTCjCKYWHgN+ALsAAb9+tqtoc95BkPeCmiOTOGGPikS/ErYoL5xZ7Aq+q6nIOPnBZsNbAa8CnRHC6fhHJF5H4nJDKGHN4sif5gfD6YGrjmsHA1WTArQFT4EvgsfJkKsA8Qph6xhhjKg3rgwHCCzCbcX0tqOpuEckBjvI7Xh+3QlpEBFvPwBhjKjULMEB4AeZHoKvf67nArSLyLa7J7Wbgp/JnzRhj4pTFFyC8Ppj3gFRv3ReA+4C6uKftZ+I6+f8ekdwBInKWiKwpPaUxxlQOtuCYE86T/B8AH/i9/kFEOuPWiMkDPlXVSAaEZODICJ7PGGOiy5rIgDIGGBGpgXu4Ml1VVxbsV9U04Pkynqt3iEnjdwUuY8zhyeILUPYaTB6uGexvwMpS0pZmDqGNDpMQ0xljTKXgq/qtXyEpU4BR1VwRySAy8TkH+BwYVUq67sAdEbieMcZUiMPgEZeQhDOK7EPgEhH5t6rml+PaPwI1VPWTkhKJSMwm5DTGmHBYgHHC+fJ+HegHzBCRZ3FNZXsCE6nqhlLO8z/gDyFe0/5zGWPihs86+YHwAswvuD4RAfqWkK60hy2fAD4q7WKqOp7DYtYeY0xVYTUYJ5wA8xAR6HRX1V+BX8t7nlDIrv0VcRkTZ+xzcZDstLKIJLGfxEB4z8GMjEI+jDGmyrAajBP1OCsiTUQkT0T6R/taxhhTGfgktK2qq6gRWodBURpjjGPPwTg2BNgYYyJMrI0MsABjjDERZ538jgUYY4yJMKvAOBZgjDEmwizAOBZgjDEmwizAOBZgjDEmwg6DtcRCYgHGGGMizGowTkUEmJ3An4DFFXAtY4yJOTkcnqIMQZkDjIgcUUoSBfYCWersA94OJ3PGGBOPrAbjhFODWUdok13uEZGZwAOq+lMY1zHGmLhkAcYJdzblc4ETgWnAcm9/R+BM3Dovc73X5wIDRKS3qv5Q/uwaY0zlZwHGCSfALAFuArqoapF+FRE5DpgNPKGqd4rI8cCXwP3ABeXNrDHGxAMbReaEUwx/B14MDC4Aqvoz8BIwwnu9CHgN6FWeTBpjTDwRX2hbua4hkiIiz4tIuojsFZHvRWRQiO8dIiILRGS7t30lIpcUk/YvIrJCRPaLyGoRuUsktNyHU4M5GthSwvFML02BpUDtMK4TN3p1bcmIG08jwSeM/XQ5r76/qMjxlOREnr6nH80aJ1MtwccbHy5i/LSVtGlZl+dGHFzFoFWz2jz39kJGfRS/A+6sLA7q1bUlI27u5spi6nJeHVO0KzIlOZGn/96PZo1TXFmMXcT4z1bQplVdnrtvQGG6Vs1q89yohYwa/0tF30LE9D6pOSOuO4WEBGHs9JW8EnAvKUmJPHN7L5o1cp+L1ycsZvzMVQDUTk7ksZu70+HI+qgq9z6/gB+Wl/QVFHsV1EQ2ATgJuAtYCwwFJojIQFWdWnze5GpgFDAe+Ie3+1rgAxFJUdU3/dKOAB4EHgFmAd29fzcA7iktg+EEmM3AYODFIBkXXFPYZr/djYBtYVwnLvh8wshbujP07k/J2JLD+BfPZ9aCDazasKMwzRWDjmHV+u0Mu286DerWZNpbFzFx5mrWbtzJoBsmFJ5n/vuXMn3++hjdSflZWRzk8wkjb+3B0DunurJ4eTCzFqxn1fodhWmuOL8zq9btYNhwryzevpiJn69ibdpOBl3/UeF55o+9jOnz18XmRiLA5xNGDjuNq++fTkbWHj56+lxmfpvGqrSdhWmuPLcjK9N2cP3Ds2hQpwbTX76AiXPXcCA3n/v+7xTm/W8TNz8+l8RqPmrWKG019tiL9mzKInIOcDpwoapO8PbNBtoCTwPFBhjcYyPrgUtUNd977zRgDXAV8Ka3LxUYDrygqvd7750jIsnAXSLygqpuLCmf4VTS3sB13E8RkTNFpLW3/R6YAvT10hQ4F/gxjOvEheOPbsT6TbtIS9/Ngdx8psxZw4AeRxZJo0ByUiIASbWqsXP3fnLz8ouk6X5iczZs2s2mzOyKynrEWVkcdHzHRqz/1a8sZq1mQPeAslD1K4vE4GVxUnM2bNrFps3xWxZdOjRkffou0jZnu7L4Yi2nn9qqSBpVJaWWX1lku7JIqZVI185NGDtjJQAHcvPZnXOgwu+hrERC28rhAtwzhp8U7FBVxT0S0lFEjinhvQeA7ILg4r03H8gG/NfOPguoyaGPmYzCVU5KbY4LpwbzCNAcGOZlwJ8ArwIPA4hITeAd3MiyUolIXdwAgm5APvAV8JKq7irhPTtKO2/7Aa+FcvmwNG2YRHpmTuHrjC05dOnYqEia0R8v4T//OIMvP7iM5KREbnt4Fhow0Pvcfm2ZPHt11PJZEawsDmraMJl0vwCZsTWHLp0aF0kz+uMl/OfhM/nyw8tdWTw0M0hZtGPyrPguiyapSaRv9ftcbN1Dl6OLfi7+O2UZrwzvz4JRF5NcK5Fbn5yLKrRqmsK2nft5/NYedGpTn19WZfGP175j7/7cir6NMgl1wbFQvr9UtV6Q3ccCS/yDhGeR//FiTvkC8JGIDAdewX1vX4/r2rgj4BpKwEPyqrpSRPZ6x0tU5hqMquar6p+9k9+LCyiv4Tr/j1XVG7xIiqruU9VXVPW7wPOIyCYROcHvdSNgIS44nQScAjwKfCsi9cqazwoT5GdI4ENCvU5uwdLVWfT4w3sMGjaB+2/uTor3yxUgsZqP/t2O5NO5a6Oc2SizsjgoyK/TwODRq2tLVxYXv8ug//uI+//S49Cy6B7/ZRHsl7oGFEavE1uwdO12ug/9kEG3TeKBYaeSUiuRhAQfnds14L1PlzPotsns2ZfLsItK/V6LuQpYMjmV4F0P2/yOB6Wqn+BqH3fg+tMzcd/lF6vqZwHX2KOq+w89C9tLukaBsMcxqOoSVX1CVf/sBZXHVbW4iBlMU6C63+tHgMbAmaraQlWb4WpIzfFGpRWTj3qlbWHcXsgytuTQrHFy4eumjZLJzNpTJM2Qs45i+hfrANiwaRcbM3bTttXBbPU+pSVLVm4la8feaGY16qwsDnJlkVL4umnDZDL9fsVDMWVxRL3C471PaeXKYnucl8XWPTRr6Pe5aJhE5raAz8WA9kz7yvW5rU/fzcbN2bRtWZeMrTlkbN3DTyu2AvDZgvV0blvq91rMhRpgyvn9VdID78UeE5EzgPeAD3DPLp6NGzAwRkTOjcQ1ClSm0drnAY+r6ucFO1R1Oq7D6vyY5aoUPy/fQusWdWjZNIXEaj7O7duWmQuKdk5vysyh20ktAEitV4s2reqSln6w1e+8fu3ivkkIrCz8/bysoCxqu7Lo346ZX20okmZTZjbdTmoOQGp9ryw2+ZVF//hvHgNYtHIrRzavQ8sm3ueiVxtmflO0b3jT1hy6d2kGQGq9mrRpUZe0jN1s3bGP9K05tGlRB4DuXZqxKm1HRd9CmflEQ9rKIYvgNYgG3t+gA6u8gVhvA7O8isEMVf1MVa8Evgb+HXCNZBGpEeRU9Yu7hr+wJrsUkW7AzUAH3E0GVvZUVduV8bSNcQ9lBvoSV32rlPLylQf/vYA3/3k2CT5h3GcrWLV+B5ee1xGAMZOX8eLoH3j8zt5Mfu1CBHjyte/YvsvVOmvWSKDH71pw37PzY3gXkWFlcVBhWTx+NgkJwrhPl7Nq3XYuHdgJgDGTlvLif3/g8bv7MPn1IYjAk69+e2hZ/OuLWN5GROTlKw++8g1vjTydBJ+PDz9fycq0HVx61lEAjPlsBS9+8BNP3NqTKc8PcmXx9kK273Zl8dCr3/DM7b1ITPSRlpHN3c8F+5qoXCpgrsvFwBAR8QX0wxzn/S1uTHsToBnwfZBj3wN9RaSmN4fkYtx3e2f8+tFFpD1Qq4RrFJLAttBS3yByFfAWbiTCClyUO4Sq9ivlPPnAAFy/C8AyYKhXa/FPdzYwVlXDfpamw+mvl+ungqmi8u1jUUBTqpee6DCxauLV5Q4P506fH9KHa8qZPcO6lteUNRkY7PWpFOyfBzRW1Y7FvK8GsAOYrqrnBxz7Auikqg2916nAr8DLqvpXv3T/wD0D01ZV00rKZzg1mOG4+cdOV9VNYbzf3+cBr08EpgfsOxoo73WMMabCVPNF/cfLVNy0XG94gWAtcDXQE78uBRGZA/RRVQFQ1f0i8h/gNhF5HRgHJOCef+mJX3+3qmaJyGPAfSKy07teN+Bu4NnSgguEF2COBO6MQHB5MMi+X4Psu4rg1TljjKmUot25raoqIoNxI20fBerhhiVfqKqTSnn7HbgWo+uBi3CPhKwArgTeDUj7EO55m5twXRWbgAeAx0PJZzgBZiMQrNOnTFQ1WIAJlu6k8l7LGGMqUkWsN+Y9H3iztxWXpm+QfXm4519eCeEaCjzrbWUWTqD9D3C5iFT++RqMMSYGRDSkraoLpwazEBiCewDyRVzbX15gIlWdV868GWNMXLIVk51wAsxMv3+/zqEP24i3LyI1HG+0Wb6qhjWk2hhjKlplesAwlsL50v5TxHNRsnmEtkSzMcZUChUwiiwulDnAqGrgzJpRFayTyhhjKjOrwTiVotlJRDriJs9MxdVWtgG/qOqymGbMGGPCYH0wTqkBRkR6w8FO+4LXpQmlk19EzsLNNdaRINPNiMhy4I6SVmczxpjKppzzjFUZodRg5uC+7Gup6m8Fr0tIH1Inv4gMxM3g+RVubZlFHJw8rQFwPO4hy4kickEIDw8ZY0ylYDUYJ5QAcw0uYBQsIxepTv77gdGqOrSY498Cr4vIKNyToxZgjDFxoZrVYIAQAoyqjgp4HalO/s7AnSGkG0XJ60sbY0ylYjUYJ5ad/DuA1iGka+OlNcaYuGABxgk7wIjIUUB7gq8Hg6q+U8opxgFPisg2VZ1YzDUG4iZVez/cfBpjTEWzYcpOmQOMiDTBrYh2RsGuIMkUKC3A3At0Aj4WkR24mUC3ee9N9Y7Vww0q+HtZ82mMMbFio8iccGowL+CCy8vALIpZcKw0qpoDnCEi5wCDcc/BHO0dzgI+AibYEGVjTLyxJjInnABzBvAfVS12iuiy8AJIVINIfqOkaJ4+rvg258Q6C5WHrWhZSPblxjoLVUo1CzBAeAHGB/wU6YwYY0xVcThMxR+KcALMF0CXSGfEGGOqCmsic8IZ7HA7cIGIDIl0ZowxpirwhbhVdeHUYF4GsoGxIrIJWMOhC46pqg4ob+aMMSYe2SgyJ5wA0xY3lHiD9/qIyGXHGGPiX7XDoXoSgnDWg2kdhXwYY0yVEZHlfKuAMsVZEUkRkVkicm20MmSMMfHOJxrSVtWVKcCoajbQNUp5McaYKsEnoW1VXTh9MD/ipnExxhgTxOEQPEIRTlfUA8D/iUi/SGfGGGOqggQJbavqwqnBXIEbQfa5iPwErAD2BKRRVbV+GmPMYamar+r3r4QinAAz1O/fJ3hbIAUswBhjDkvWROaEM0zZRngbY0wJbJiyE8sVLY0xpkqyGoxjAcYYYyLscHjGJRRhBRgRqY/rYzkVqM+ho9FsLjJjzGHrcBghFopwlkw+EvgSaA7sBOrgljouCDRbgcNqVaveXZpx31W/I8EnfDB7Na9MXFLkeEqtRJ65qTvNGyaRkCC8PnkZ4+euAWDu84PI2ZtLXr6Sl5/P4OHTYnELEdOra0tG3NyNBJ8wdupyXh1TdOmglOREnv57P5o1TqFago83xi5i/GcraNOqLs/dd/A3SatmtXlu1EJGjf+lom8hYnqd0pIRN3cnIUEYO2UZr74XpCyG9/fKQnjjA7+yeMC/LOrw3FvfM2pcHJfFyS0Y8efTSPD5GPvZcl79YFGR4ylJiTx9T1+aNUp2n4txPzN++koAaidX59Hbe9KhdX1QuOfpL/hxaWYsbiNkNheZE04N5mGgHjAA+BnIBP4AfA0MB/4I9IlQ/io9nwgj/3QyVz86i4ysvUx45PfMXLiRVb/uKkxz5ZkdWPXrTq5/ai4NatdgxjPnMXH+Og7k5QNw+cMz2b57f6xuIWJ8PmHkrT0YeudUMrbkMP7lwcxasJ5V63cUprni/M6sWreDYcOn06BuTaa9fTETP1/F2rSdDLr+o8LzzB97GdPnr4vNjUSAK4ueDL1jiiuL/1zArC8DymJwZ1at286wv09zZfHfSw6WxXV+ZTHucqZ/sS42NxIBPp8w8ubuDL3nMzK25jD+34OY9dUGVm3YUZjmikHHsGr9DobdP8OVxRsXMXHWag7k5jPixtOY991GbvnHLBKr+ahZo/K37FsfjBNOnB0AvKaqs3HDkQFEVfeo6nBc0Hk8Uhms7Lq0T2V9RjZpmTkcyMtn8lfrOf3klkXSKJBcy/2fIqlmNXZm/0Zufn4Mchtdx3dsxPpfd5GWvpsDuflMmbWaAd2PLJJGVUlOSgQgqVYiO3fvJzevaFl0P6k5GzbtYtPm7ArLe6S5sthZtCx6tC6SRpXQyuLXOC+LoxuxftMu0jK8spi7hgHdi07Crvh/LqoVlkVKUiJdj2vKh5+tAOBAbj67c36r8HsoqwTRkLaqLpwAkwoU1NUPeH9r+R2fAZxRnkzFkyb1a5GedbBFMCNrD03qJxVJ899pK2jfvC5fvXQBU584h4feWYh6ny1VGHVvPz555Cz+2L9dRWY94po2TCY98+AXYcbWHJo0Si6SZvTHS2h3RD2+/PByJr8xhIdf+KqwLAqc268dk2etrogsR03TRsmkb/H7XGwJUhYTFtPuyPp8Of4KJr91EQ//e8GhZdG/ffyXRcOkgLLYQ5PUgLL4ZCntWtXlyzGXMvmVC3n45a9RhVZNa7Ntxz4ev6MXn7w0mEf+2pNaNeOgBhPiVtWFc49bgAbev3cD+4DWfserUzTglIuI/E5E3izh+I7StkjlJfj1g+0t+i3R6/hmLFm/nW43TmDgPZ8ycujJpHg1mktGzuD8v3/GNY/P5oozj6Jrx0bRzG50BSmLwC/MXl1bsnR1Fj0ufpdB//cR9/+lByneL1eAxGo++nc/kk/nro1yZiueBhRGr1NasnRVFj2GjGbQdeO5/9YgZdHjSD6ds6aisxp1h5TFyS1YumYbPS4dw6A/T+D+m7uRkpRIQoKPzh1SeW/yMs6/8WP27stl2B+Oj1GuQ2eTXTrhBJjFQBdwQ8WAb4EbReQIEWkNXA8si1gOXfC6OoLni6iMbXtp5vdrrGlqEpu37y2S5qK+bZn2bRoA6zdns3FLNm2b1wUg00ubtWs/07/bSJd2qRWU88jL2JJDs8Ypha+bNkwmc2vR8R5DzjqqsD9hw6ZdbMzYTdsj6hUe731KK5as3EpWQBnGm4wtOTTzq7E0bZRM5taiMyoNOetops9zgXTDr7vYmB5QFqe2YsmKKlAWW/cElEUSmdsCyuLMowr73DZs2u0+F63qkrE1h4wtOfy0bAsAn32xls7tG1ZY3sOV6NOQtqounADzCdBNRApqKQ8BHYC1wGrv3/+ITPZKp6r1Stuief1Fq7No3bQ2LRslk5jg47xuRzJz4a9F0mzauofuxzYFILVuTdo0q0NaZja1aiSQ7FX3a9VIoNfxTVmxcWc0sxtVPy/bQusWdWjZtDaJ1Xyc278dM7/aUCTNpsxsup3UHIDU+rVo06ouaZsODog4r3/8N48B/Lx8C61b1i1aFgvWF0mzKTObbr9rARSURT3S0v3KYkB7Js9cVaH5joaflxd8LlJcWfRpG/xzcaL3uahXkzYt65KWvput2/eSviWHNi3dD7JuJzZn1YbtFX4PZWU1GEcCq6phnUTkZOAyIA+YoKoLQnhPXlmuoaphz77Q7tL3ovpToe8JzRlx1Un4fMK4OWt46ePFXHp6ewDGfL6KxvVr8cQNp9G4Xi1E4D8Tl/DJ/HW0apzMy7f3BiAhQZj05Xpe+nhxNLOKb3N0R5D3ObUVw2/sRkKCMO7T5bz87o9cOtCt7jBm0lIapybx+N19aNQgCRF4ZcxPTPzcfYnWrJHAvPcvo/8V75Odc6Cky0RGfnR/QfY5tRXDb+5Ggs/nymL0D1w6yCuLiV5Z3NOXRqleWbz3IxNn+JXF2Mvpf9mYiimL6tGd3KRP15YM//NpJPiEcdNW8PKYn7j03I4AjJmyjMYNknj8zt40alALEeGVD35i4kz3Q6NT2wY8cntPEqslkJaxm3uemseu7Oh19K+cfm25v/onbfg0pA/XwCPOrtJhJiIBJqwLi/yGW1vmy1KStgfOqcwBJp5EO8DElSgHmLgS5QATTyIRYKakhRZgzm1VtQNMuYZjiEh7oAnwi6qWtW1nMbBOVf9ayjWGAOeEmUVjjKlw9iS/E9ZIORE5T0RWA8uBecDvvP2NRWSViFwUwmn+B5wU6iXDyacxxsSCTzSkrTxEJEVEnheRdBHZKyLfi8igEN63TkS0mG1ZQNri0t0QSh7DmSqmLzAB17z1NjCy4JiqZnqB54/AuFJO9SawvpQ0AHMBWz3TGBM3qlXMT+IJuB/pd+EGWQ0FJojIQFWdWsL7LgBqBOw7DngV+DhI+g+AZwP2hTR2PpwmsvuBnzg40eXIgONfAVeVdhJV/ZLS+19Q1a24IGOMMXEh2k1kInIOcDpwoapO8PbNBtoCTwPFBhhV/SHI+S7z/hnsmcMMVf06nHyG00R2MvCuqhY318lGoGk4mTHGmKqgAprILsBNNvxJwQ7vucS3gY4ickyoJxKR6rhRwPNVdUV5MhUonACTAJQ0M2NDoPJPFmSMMVFSAc/BHAssCfJDf5Hf8VANxk0BVtyMKVd5fTz7ROQbEbkk1BOH00S2FOgFvFTM8fNwTWgRISL5QL6qVv4JiIwxhtCDRyhTWRXzsHgqEKy2sc3veKiuAbKBsUGOvYtrbksDmgE3Ah+ISDNVfa60E4fzpf0G8LyIfA5M9PapiCQB/wS6EUIfTBnMI3ByL2OMqcQqaCLLkr4XQ/rOFJGWuMmJ31LVQx6SU9UrAtKPA+YAD4vIq6pa4jxGZQ4wqvqyiPQAXsN1JikwBhcxE7yMvlvW85Zwvb6ROpcxxlSEUGsw5ZjKKovgtZSCiYi3BTkWzFBcPCx2QmF/qpovIqNxrVjHAt+VlD6sZidVvUJExgNXAB1xz6l8A7yjquPLej4R6YjLbCouYG3DPbwZyUkzjTGmQgSfZT2iFgNDRMQX0A9znPe31OVPRURwAWZZKNN7+SmooJW6qFXY/Rre0LgJ4b4fQETOwtWCCoJUkUuIyHLgjlLGdBtjTKVSAU1kE4BrgYH4jSTDdU8sV9UlQd9VVB+gHe45mpCIiA+4HLdUS6kTJ5YaYEQkrP4UVX2nlPMOxBXSV8Aw3OiHgmpdA+B4XGFNFJELVHVSOPkwxpiKJtFfrXIqMBt4Q0RScQ9aXg30BM4/mA+ZA/RR1WB1qmuAXCDod7WI3AEcDcwC0nGPn/zZu8ZNqrqvtEyGUoMZhWu2KkulTykm037uB0ar6tBijn8LvC4io4AHAAswxpi4EO0WMlVVERkMPOpt9YAluAcvS/2uFJHawBBgqqpuLibZclywGuydPwdYCAwK9Qd/KAEmWtO0dAbuDCHdKEp4KtUYYyqbiljrRVV3ATd7W3Fp+hazfzeQHOyYX5pJlPOHfakBRlWjNU3LDooutVycNl5aY4yJC4fDYmKhqKDh2kGNA54safZPr5/mcUqfONMYYyoNCXGr6mL5dPy9QCfgY+9p1iW4Tn7FDVfuhGv3mwP8PSY5NMaYMFTAMOW4ELMajKrmqOoZuKllxnl5ORo3ZNkHfAQMVNUBqpodq3waY0xZWQ3Gifn8Xt4zLlHtxD/QtVk0Tx9Xqn++LtZZqDRkb26ss1B52PLREXU4BI9QxDzAGGNMVWNLJjsWYIwxJsIq4EHLuGABxhhjIswqMI4FGGOMiTAbReZYgDHGmAiL5QOGlYkFGGOMiTCrwTgWYIwxJsJsqhjHAowxxkSYxRfHAowxxkSY1WAcCzDGGBNhFl8cCzDGGBNh9qClYwHGGGMizJrIHAswxhgTYRZfHAswxhgTYfagpWMBxhhjIswetHQswBhjTMRZhAELMBHRp3UDRvbtQIIP3v85nZe+21DkeO3qCTx39jE0r1OTaiK8snADHy7OoEaCjw//cCLVE3xUE2Hqykye+WpdbG4iQnr/rgUjhp1Kgk8YO20Fr3z4c5HjKUmJPHNnb5o1SqFagvD6R78wfsYqAGonV+exW3vQ4ch6qMK9z87nh2VbYnEbEdGra0tG3NyNhARh7JTlvDrmpyLHU5ITefrv/WjWJIVqCT7e+GAR4z9bQZtWdXnu/gGF6Vo1q81zby1k1PhfKvoWIqZX15aMuPE097n4dDmvvr+oyPGU5ESevqcfzRonu7L4cBHjp62kTcu6PDeif2G6Vs1q89zbCxn10eKKvoUyEQswgAWYcvMJPNz/KC4f/yPpu/cz6fKTmbF6Kyu37SlMc9UJLVm5LYdrPvmZBrUSmfOnU/l46Wb25+Xzxw9/ZM+BPKr5hPF/OInZ67bxQ/quGN5R+Hw+YeSNp3H18GlkbN3DR88OZObXG1iVtrMwzZXndWLlhp1c/+BMGtSpwfTXhjBx9hoO5OZz37BTmbdwIzc/OpvEaj5q1ojfj6fPJ4y8tQdD75xKxpYcxv9nMLMWrGfV+h2Faa4Y3JlV63cwbPh0GtStybR3Lmbi56tYm7aTQf/3UeF55n94GdPnr4vNjUSAzyeMvKU7Q+/+1JXFi+cza8EGVm3YUZjmikHHsGr9dobd55XFWxcxceZq1m7cyaAbJhSeZ/77lzJ9/voY3UnoRBJinYVKwfqiyumEpnVYt2MvG3bu40C+MmnZZs5s17BoIlWSE92XZXJiAjv2HSDXW6J2z4E8AKr5hGo+QeN4+HyXoxqyftNu0jKyOZCbz5R5azi92xFF0ihKSi1XFkm1Etm5ez+5efmk1Eqk67FNGDttJQAHcvPZnfNbhd9DpBzfsRHrN+0iLX23K4tZqxnQ48giaVSV5KREoGhZ+Ot+UnM2bNrFps3ZFZb3SDv+6ICymLPm0LIAv7KoFrwsTmzOhk272ZRZ+ctCQvxfVVfpAoyI1BCR34nIiSJSPYT0O0rbopnfpik12LR7X+Hr9Oz9NKldo0iaUT/+SvvUJL6/vjvTr+rKyNmrKIgjPoFPrziZH27owfwN2/gxIz5rLwBNUpNI35pT+Dpj6x6apCYXSfPfSUtp16oeC0b/gSkvDeYfr3yDqmv62LZzH4//tScT/z2IR2/tQa04rsE0bZhMut8XYcaWHJo0LFoWoycsod0R9fhy3OVMfnMID7/w1SE/MM7t347JM1dXRJajpmnDJNIz/T4XW3JokppUJM3oj72y+OAyJr82hIdf+vrQsujXlsmz46UsJMStaotZgBGR60WkYcC+W4BM4FvgeyBTRIbFIn+hCvYRCfw/Rp/WDViSmc3Jry7grNHf81D/DqRUd1XofIWzR3/Pqa99RZemdTgq4As5nkiQoTMaUBi9TmrB0jXb6H7FBwy6+RMe+PNppNRKJCFB6Nw+lfemLmPQLRPZsy+XYZccV1FZj7wgH4zAz0Wvri1ZuiqLHhe9y6DrPuL+v/QgxfsVD5BYzUf/7kfy6dy1Uc5slAX7XAS87nVyC5auzqLHH95j0LAJ3H9z90PLolv8lIWIL6StqovlHb4MtC14ISIXA88BPwI3ADcCPwMvicg5xZ1EVeuVtkXzJtKz99O8ds3C181SapCZvb9Imos7N+OzVa6zev2OvaTt3Ee7BkV/we3an8vXaTvo27pBNLMbVRlbc2jm9yu9acMkMv36ogCGnNGBaQtcG/r69N1s3JxN21Z1ydi6h4ytOfy0fCsAn81fR+d2qRWX+QjL2JJDs8Ypha+bNkomMyunSJohZx/F9C/WAbBh0y42pu+m7RH1Co/3PrUVS1ZsJWv73orIctS4svD7XDRKJjMr4HNxVkBZZOymbat6hcd7n9KSJSu3krUjXsrCajAQ2wATWLp/A+aoah9VfU1VXwF6A/OBv1Z47kL0U8Zu2tSrRas6NUn0CQM7NmHGmq1F0mzavY8eR9QHoGFSIu0aJLFhxz4a1EqkjtcMVKOaj55H1Gd1wBdyPFm0YitHNq9DyyYpJFbzcW7vtsz8Oq1Imk1bcuh+QjMAUuvVpE2LOqRl7Gbr9r2kb8mhTYs6AHQ/oVmRTuB48/OyLbRuUYeWTWu7sujfjpkLio4u3LQ5m24nNQcgtX4t2rSqS9qmg02k5/Vvx+RZ8dIkVLyflxeUhfe56NuWmQuKdtRvysyh20ktAEit55WF32CX8/q1i6PmMfCF+L+qTgKbMCrswiL5wGmq+q33ei9wpaqOC0j3R+AlVQ37p/0Rz8yO6k32a9OAB/p2IEGED35J54Vv13PF8e6LY/SiTTRJrs7Tv+9E45TqCPDSdxuYsHQzHRsm88xZnUgQwScwecUWnvt6XTSzSvXPo3v+Pie3ZMSwU0jwCR9OX8nLHyzi0nOOBmDM1OU0blCLJ27vRaMGSQjwyoeL+GT2GgA6tW3Ao7f2ILGaj7SM3dz9r/nsyo5eR7/szY3auQH6nNqK4Td1I8EnjPt0OS+/+yOXDuwEwJhJS2mcmsTjd/ehUWoSIvDKez8x8XM3ZLtmjQTmfXAZ/S9/n+ycA1HNJwAJ0f013eeUlgy/0SuLz1bw8ns/cul5HQEYM3mZK4s7e7uyAF55fxETZ/qVxZhL6X/lBxVSFis/v67chZF9YE5I3zkpiX2rdDWmMgWYPcAZqvplQLo+wHRVrRHkNCGJdoCJJ9EOMPEk2gEmrkQ5wMSTSASYnNy5IX3nJFfrU6ULPtbDdG4TkQzv33uAVkHSNAN2VFiOjDGm3Kp03AhZrAPMHwNenwO8H7CvP7CsYrJjjDHldzg84xKKmAUYVQ21h2s5MDGaeTHGmMiq+h34oYh1DaZUqvp0rPNgjDFl4TsMnnEJRaUJMN5Dl6m4Z7C2qerWUt5ijDGVlDWRQYzrcSLSXEReFJF0YDOwBFgKbBaRDO9Yy1jm0RhjykrwhbRVdTGrwYhIJ2AuUAOYBCwCtnmHGwDHAZcDl4hIb1VdGpOMGmNMmVkNBmLbRPYUsBo4V1W3BUsgIg2AKcCTwHkVmDdjjAlbsHn5DkexDDC9gD8WF1wAVHWbiDwMjKm4bBljTHlZgIHYBph8IJRVeRK8tMYYExckpK+2qi+WvUxzgAdFpElxCUSkMfAAMLuiMmWMMeVlC445sazB3I6bKXmNiMwAfsF18ituuPKxwBnATuCSWGXSGGPKyvpgnJjVYFR1DdAFeBE4BrgX1/H/tPfvzt6xE1Q1fubpNsYYfCFu4RORFBF5XkTSRWSviHwvIoNCeN86EdFitkOm5RKRv4jIChHZLyKrReQuCXG1tJg+aKmqW4C7gLtEpCZQ3zu0XVX3Ff9OY4ypvCqo+WsCcBLuO3QtMBSYICIDVXVqCe+7APd4iL/jgFeBj/13isgI4EHgEWAW0N37dwPgntIyWGme5PcCSnqs82GMMeUV7eWQvVV+TwcuVNUJ3r7ZuFWCnwaKDTCq+kOQ813m/fNNv32pwHDgBVW939s9R0SScZWCF1R1Y0n5rPqPkhpjTIWLehPZBbj+6U8Kdqhb3OttoKOIHBPqiUSkOnAZMF9VV/gdOguo6Z3T3yhc5aTU5rhKU4MpjrcwWb6qVvq8GmMMhN5EJiI7SkujqvWC7D4WWKKqgY9wLPI/HlImYDBuYNWbAfuPxQ26WhyQn5XeCsTHlnbiePjSnoe7ybBtuL1fTId0FHyIivmgVKzbY3v5SlUWMWZlcVDVK4ujQv3O2RHmBVKBFUH2b/M7HqprgGxgbJBr7FHV/UHesz2Ua1T6AKOqfWOdB2OMiYZyBtSSfniH9KPcm0z4DOAtVc2J9DUqRYARkY646lbhdP3AL6pqK1kaY8yhsgheg2jg/S12Cq4AQ3GdQYHNYwXXSBaRGkFqMfVDuUZMA4yInIUb8dCRQyfvURFZDtxRypA7Y4w53CwGhoiIL6Af5jjv7y+lnUDc06BDgWWquqCYawjumcT/+b2vPVArlGvEbBSZiAwEJuOi4DDgNOAobzvN27cVmOilNcYY40wA6gGB341XActVNZQO/j5AO4LXXgA+BfYDVwbsvxrIxS2zUqJY1mDuB0ar6tBijn8LvC4io3DzkZV6M8YYc5iYipuj8Q3veZW1uC/+nsD5BYlEZA7QR1WDDTq4Bhco3gl2AVXNEpHHgPtEZKd3vW7A3cCzqppWWiZjGWA6A3eGkG4UJTw0ZIwxhxtVVREZDDzqbfVww5IvVNVSf4yLSG1gCDBVVTeXkPQh3PM2N+Gm8NqE+8H/eCj5jGWA2QG0DiFdG8IfymeMMVWSqu4Cbva24tL0LWb/biA5hGso8Ky3lVksn+QfBzxZ0uRsXt/L415aY4wxcURcgIrBhd18Nh8DA3A1lCUUna6/E67aNwc4X1WzY5BNY4wxYYpZgCnMgJu0bTAHn4MBN/56MTDBhigbY0x8inmAMcYYUzXZbMrGGGOiwgJMlIhIdxGZJiK/isg+EdkiIrNE5OxY5y3aRKSliDwnIvNFJNtbKa9vMWnPEJGvvRX5MkXkFRGpV6EZjpKyfAaqUjlE479/uKs3mtiyABM99YHlwN9w6ypcj3sqdqqI/DGWGasA7YFLcTO0ziwukfelMxVIwz2RfAdujYkpoS7JWsmF9BmoguUQjf/+E4DLgRHAubhBQRO8PlxTWamqbRW04Z47SgNmxTovUb5Pn9+/B+NGBvYNku5b4IeA9Gd46f8Q6/uoqM9AVSuHSP/3B87x9l3gt0+A+cDSWN+vbcVv8fjrKG6pai7uqdgDsc5LNOmhiyAdQkRaAF2B//qnV9UZwK+4p4yrnMDPQFUshyj894/Y6o2mYlWK6fqrMq+q7wMa4ybwPArXFHC4K1gNL9iMrD8Twmp58aKUz8BhUw4BynLfkVy90VQgq8FE31jcr9VfgduAS1T1s5jmqHIoeOYp2JoS2yjbinyVXUmfgcOpHPyV5b5TS0gHVbeM4p4FmOi7CzgF13k5FRgrIpfGNkuVSnEPYlWlB7RC+QwcDuUQTKj3Xe7VG03FsyayKFPVNcAa7+UkEZkEvCgiH4TSVl2FZXl/i1uVL9QV+Sq9kj4DHEblEKAs9x2p1RtNBbMaTMX7Fjd8tVGsMxJji72/wfoYjiOE1fLimP9n4HAth7Lc92KgU5ChyyGv3mhiwwJMBfKWKO2Lm9wzq8TEVZyqbgS+By73/+IQkQFAC+CjWOUtmgI/A4drOZTxviOxeqOJAWsiixIReRdYDyzELf3cDLfiXH/gFm+4apUlIhd5/+zq/e0jIg2BHFX91Nt3NzAdGCMirwLNccszfAN8WJH5jYYyfAaqXDlE+L9/SKs3mkoo1g/iVNUNtwjQV7iaSq73dxowMNZ5q6D712K2dQHpzsJ9oewDtgCvAfVjnf+K/gxUtXKI9H9/oA7wApDhpf0fMDjW92lbyZvNpmyMMSYqrA/GGGNMVFiAMcYYExUWYIwxxkSFBRhjjDFRYQHGGGNMVFiAMcYYExUWYEylJiLrRGROBV6vn7eE725vqd+hFXXtshKRoSUtR2xMrNmT/MZ4RKQ+boqSjbhljvcAC2KaKWPimAUYYw7qipvz6lpVrZJzgBlTkayJzJiDmnp/bfp3YyLAAkwV59dOP0BE7heR9SKyV0S+EZHTvDR9RGS+iOSISLqI3BdwjjNF5AMRWeO9d4eITBeRPgHpThSRfSIyw5s1uGB/gojME5E9ItI5Qvd1sohMEJGtIrJfRJaLyHARqRaQ7hQRGSUiK7zr7xaRL0XkgoB063BrvAPM9sos5HmURKRvQZ+NiPxJRBZ7+VovIncV857BXl6yve1LEQk6eaOIXCciy7xzrhKRWwEpJm1dEXncS7dfRLaIyBgRaRuQrqaIjPTKbo/33/VnEXky1Ps2piTWRHb4+CeQADwHVMf1MUwTkauBN4BXgXeBS4CHRGStqo723jsUt7jTO7j+iRbAdcBMEemnql8AqOoP3pfpc8A9wGPe+x8AegE3qGrBOiBhE5FzcFO4rwKextU4ugEPAScAF/slvwDoiFu2eD1u4aqrgY9E5HJVfc9LdxtwNnA98CiwNMzs3QA0wZXpDuAK4HER2eh3LUTkRuBFYBnwMG4iyKHAxyIyTFVf9Ut7G/Av4Cfg70AScCeQGXhxEamL6zc6AngTt5ZKM+BG4BsROVlV13vJXwSuwf13/Rfu89EBN9uzMeUX69k2bYvuhvvSUtzss9X99g/y9ucCXf32VwfSga/89iUHOW8T3BT0U4Mc+wS3Bn03oA+QB4wLM//rgDl+r2viZtSdB1QLSPtX7576lpL3JGA5sKSYsuobRj77eu/dBNQLuNaWgPKsD2TjAmQdv/11gNXA7oJz4PqEcoAlQJJf2pbeOQLv9zlgL9AlIH9HAruAUX77tgX772ebbZHarIns8PGyqv7m9/oL7+/XqvpdwU4vzbe4X7IF+3IK/i0iKd6aHHm4adZPDXKtP+GCwHu4WlEarsYTCWfggttbQD0RaViw4dYNATizmLwneXlPAmbhVkmsE6F8FXhLVXf4XX8P8DV+5endQzLwvKru8ku7C/g3kAKc7ncvScCL3rkK0m7ElW0hr1nyclzw/TWgbHK8fJzp95adQGcRCbaqpDHlZk1kh481/i9UdbvXTbI2SNrt+K2BLiLtgEeA3+N+URc5VeCbVXWbiPwJmOHt6un/pVtOnby/b5aQpknBP0SkMa4J6nygcZC09XC/7CNlTZB9gWvKt/H+BmsuLFj+t23A32VB0gau5NjIu86ZuFpTMPl+/74N+C/ws4iswS3qNQmYpKr5Qd5rTJlYgDl85JVxP+BqLLhfxMnAs8DPuCacfOBeim+vH+T37xOAL0PPaokKOrbvBH4sJs0mKPxFPx0XlJ4HvsP9as/D1bIuI/IDXUosT0/QzvlS0gYbcBB4noLXn+NWhiyRqn4iIq2Bc3BNmacD1wJfiMjpATVeY8rMAowpzQDcUrbXqOpb/gdE5OFgbxCRgcAtuGas9sBTIjJPVX+OQH5Wen9zVPXzUtIeD3QBHlLVBwLyGKkmu3Cs9v52BmYGHDvG+7smIG0nXLOev04Br7fgBhbUCaFsAFfbBEYDo72A/E/gLlyNLy6XazaVh/XBmNIU/CIv8mtZRM4kSP+LiLTABZaluCWDL8c9Ef++iNSKQH6m4UZP3SMiDYJcv5aI1C4l78fiRpfFygxcn8gtfnnF+/ctuM77GX5p9wI3iUiSX9qWuBpYIa9Z613gFBG5KNiFvSbDgqHj9QLer8AP3stDytaYsrIajCnNfFyH/dNec8pGXJPXlbjmsuMKEoqID/cFlwz80euU3iMi1+KGFT8LDCtPZlQ1R0SuAj4GlovIm7jRWPVww5EvxAWPObggtxi4y/tyXg4c5eXhF+Ck8uQlXKq6wxvO/SJu6PAo79BQXI1vmKru9NJuF/dc0lPAAhF5B9fpfwOuNndiwOmHAz2AsSIyFtex/xtuFNk5wELvOrWBdBGZiAsqmbi+oT/j+uAmRfzGzWHHAowpkfdl+HvgCdyv62q4L6lzcO31x/klH4Fry79RVRf5neNjEXkJuFFEpqvq+HLmaZqIdMU9a3MFrnN7O6456RlgkZcuT0TOxX05X40LfL94/+5CjAKMl7eXRCQd15dU0Hz3E3CBqn4ckPZpEckGbsc9W5SGu6edBAx2UNWdItID95zTJbimrlzcD4P5wOte0j24gD8A1/eSghuePhF4TFU3RfB2zWFKXK3YGGOMiSzrgzHGGBMV1kRmKpyINMJNS1KSbFXNroj8FEdEEnDNb6XZZkN6jTmUNZGZCudNLHlkKckeVNWR0c9N8bxBDcEeRA3UT1XnRDc3xsQfq8GYWLgcKG3IcrAn4itaBm5al9L8FO2MGBOPrAZjjDEmKqyT3xhjTFRYgDHGGBMVFmCMMcZEhQUYY4wxUWEBxhhjTFT8P57RnsPyflctAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "ax = sns.heatmap(pivoted_cv_results, annot=True, cmap=\"YlGnBu\", vmin=0.7,\n",
    "                 vmax=0.9)\n",
    "ax.invert_yaxis()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e915398b",
   "metadata": {},
   "source": [
    "The above tables highlights the following things:\n",
    "\n",
    "* for too high values of `learning_rate`, the statistical performance of the\n",
    "  model is degraded and adjusting the value of `max_leaf_nodes` cannot fix\n",
    "  that problem;\n",
    "* outside of this pathological region, we observe that the optimal choice\n",
    "  of `max_leaf_nodes` depends on the value of `learning_rate`;\n",
    "* in particular, we observe a \"diagonal\" of good models with an accuracy\n",
    "  close to the maximal of 0.87: when the value of `max_leaf_nodes` is\n",
    "  increased, one should increase the value of `learning_rate` accordingly\n",
    "  to preserve a good accuracy.\n",
    "\n",
    "The precise meaning of those two parameters will be explained in a latter\n",
    "notebook.\n",
    "\n",
    "For now we will note that, in general, **there is no unique optimal parameter\n",
    "setting**: 6 models out of the 16 parameter configuration reach the maximal\n",
    "accuracy (up to small random fluctuations caused by the sampling of the\n",
    "training set)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac38c19e",
   "metadata": {},
   "source": [
    "In this notebook we have seen:\n",
    "\n",
    "* how to optimize the hyperparameters of a predictive model via a\n",
    "  grid-search;\n",
    "* that searching for more than two hyperparamters is too costly;\n",
    "* that a grid-search does not necessarily find an optimal solution."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}