Week2/HyperParameterOptimizationPresentation/HyperparameterOptimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hyperparameter Optimization Example\n",
    "\n",
    "This Jupyter Notebook is made for illustrating - through a mixture of slides and code in an interactive fashion - the different methods for optimising Hyperparameters for Machine Learning models. First it shows the most naive, manual approach, then grid search, and finally bayesian optimization. \n",
    "\n",
    "\n",
    "### Authors and Date:\n",
    "- Christian Michelsen & Troels Petersen (Niels Bohr Institute)   \n",
    "- 2024-04-27 (latest update)\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "This notebook uses the __[HTRU2 Pulsar dataset](https://archive.ics.uci.edu/ml/datasets/HTRU2)__ dataset as example data for Hyperparameter Optimization (HPO).\n",
    "\n",
    "The focus on this small example is neither the actual code nor getting any specific results, but - hopefully - getting a better understanding of HPO. This is also why we don't describe the code in great detail - and simply load the dataset from a csv file directly - but the first part of the code should hopefully look familiar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. Naive, manual approach\n",
    "2. Grid search\n",
    "3. Random search\n",
    "4. Bayesian optimization\n",
    "5. \"Full\" scan over parameter space\n",
    "6. New methods\n",
    "7. New software\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- __[HTRU2 Pulsar dataset](https://archive.ics.uci.edu/ml/datasets/HTRU2)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Focus on the understanding of HPO, not the actual code nor the data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Nomenclature (i.e. naming scheme)\n",
    "- Machine Learning Model: $\\mathcal{A}$\n",
    "- $N$ hyperparameters\n",
    "- Domain: $\\Lambda_n$\n",
    "- Hyperparameter configuration space: $\\mathbf{\\Lambda}=\\Lambda_1 \\times \\Lambda_2 \\times \\dots \\times \\Lambda_N $\n",
    "- Vector of hyperparameters: $\\mathbf{\\lambda} \\in \\mathbf{\\Lambda}$\n",
    "- Specific ML model: $\\mathcal{A}_\\mathbf{\\lambda}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Domain of hyperparameters:\n",
    "\n",
    "1. real\n",
    "2. integer\n",
    "3. binary\n",
    "4. categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Goal:\n",
    "\n",
    "Given a dataset $\\mathcal{D}$, find the vector of HyperParameters $\\mathbf{\\lambda}^{*}$, which performes \"best\", i.e. minimises the expected loss function $\\mathcal{L}$ for the model $\\mathcal{A}_\\mathbf{\\lambda}$ on the test set of the data $D_\\mathrm{test}$:\n",
    "\n",
    "$$ \\mathbf{\\lambda}^{*} = \\mathop{\\mathrm{argmin}}_{\\mathbf{\\lambda} \\in \\mathbf{\\Lambda}} \\mathbb{E}_{D_\\mathrm{test} \\thicksim \\mathcal{D}} \\, \\left[ \\mathbf{V}\\left(\\mathcal{L}, \\mathcal{A}_\\mathbf{\\lambda}, D_\\mathrm{test}\\right) \\right]  $$\n",
    "\n",
    "In practice we have to approximate the expectation above. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First, we import the modules we want to use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
    "from sklearn import tree\n",
    "from sklearn.datasets import load_iris, load_wine\n",
    "from sklearn.metrics import accuracy_score\n",
    "from IPython.display import SVG\n",
    "from graphviz import Source\n",
    "from IPython.display import display                               \n",
    "from ipywidgets import interactive\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV\n",
    "from scipy.stats import randint, poisson\n",
    "\n",
    "import warnings\n",
    "# warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We read in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mean_SNR</th>\n",
       "      <th>STD_SNR</th>\n",
       "      <th>Kurtosis_SNR</th>\n",
       "      <th>Skewness_SNR</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>27.555184</td>\n",
       "      <td>61.719016</td>\n",
       "      <td>2.208808</td>\n",
       "      <td>3.662680</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.358696</td>\n",
       "      <td>13.079034</td>\n",
       "      <td>13.312141</td>\n",
       "      <td>212.597029</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>73.112876</td>\n",
       "      <td>62.070220</td>\n",
       "      <td>1.268206</td>\n",
       "      <td>1.082920</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>146.568562</td>\n",
       "      <td>82.394624</td>\n",
       "      <td>-0.274902</td>\n",
       "      <td>-1.121848</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.071070</td>\n",
       "      <td>29.760400</td>\n",
       "      <td>5.318767</td>\n",
       "      <td>28.698048</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>32.919732</td>\n",
       "      <td>65.094197</td>\n",
       "      <td>1.605538</td>\n",
       "      <td>0.871364</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>34.101171</td>\n",
       "      <td>62.577395</td>\n",
       "      <td>1.890020</td>\n",
       "      <td>2.572133</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>50.107860</td>\n",
       "      <td>66.321825</td>\n",
       "      <td>1.456423</td>\n",
       "      <td>1.335182</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>176.119565</td>\n",
       "      <td>59.737720</td>\n",
       "      <td>-1.785377</td>\n",
       "      <td>2.940913</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>183.622910</td>\n",
       "      <td>79.932815</td>\n",
       "      <td>-1.326647</td>\n",
       "      <td>0.346712</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Mean_SNR    STD_SNR  Kurtosis_SNR  Skewness_SNR  Class\n",
       "0   27.555184  61.719016      2.208808      3.662680      1\n",
       "1    1.358696  13.079034     13.312141    212.597029      1\n",
       "2   73.112876  62.070220      1.268206      1.082920      1\n",
       "3  146.568562  82.394624     -0.274902     -1.121848      1\n",
       "4    6.071070  29.760400      5.318767     28.698048      1\n",
       "5   32.919732  65.094197      1.605538      0.871364      1\n",
       "6   34.101171  62.577395      1.890020      2.572133      1\n",
       "7   50.107860  66.321825      1.456423      1.335182      1\n",
       "8  176.119565  59.737720     -1.785377      2.940913      1\n",
       "9  183.622910  79.932815     -1.326647      0.346712      1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('./data/Pulsar_data.csv')\n",
    "\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We then divide the dataset in input features (X) and target (y):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3278, 4)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mean_SNR</th>\n",
       "      <th>STD_SNR</th>\n",
       "      <th>Kurtosis_SNR</th>\n",
       "      <th>Skewness_SNR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>159.849498</td>\n",
       "      <td>76.740010</td>\n",
       "      <td>-0.575016</td>\n",
       "      <td>-0.941293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>831</th>\n",
       "      <td>4.243311</td>\n",
       "      <td>26.746490</td>\n",
       "      <td>7.110978</td>\n",
       "      <td>52.701218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2658</th>\n",
       "      <td>1.015050</td>\n",
       "      <td>10.449662</td>\n",
       "      <td>15.593479</td>\n",
       "      <td>316.011541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2495</th>\n",
       "      <td>2.235786</td>\n",
       "      <td>19.071848</td>\n",
       "      <td>9.659137</td>\n",
       "      <td>99.294390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2603</th>\n",
       "      <td>2.266722</td>\n",
       "      <td>15.512103</td>\n",
       "      <td>9.062942</td>\n",
       "      <td>99.652157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>121.404682</td>\n",
       "      <td>47.965569</td>\n",
       "      <td>0.663053</td>\n",
       "      <td>1.203139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1370</th>\n",
       "      <td>35.209866</td>\n",
       "      <td>60.573157</td>\n",
       "      <td>1.635995</td>\n",
       "      <td>1.609377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1124</th>\n",
       "      <td>199.577759</td>\n",
       "      <td>58.656643</td>\n",
       "      <td>-1.862320</td>\n",
       "      <td>2.391870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2170</th>\n",
       "      <td>0.663043</td>\n",
       "      <td>8.571517</td>\n",
       "      <td>23.415092</td>\n",
       "      <td>655.614875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2177</th>\n",
       "      <td>3.112876</td>\n",
       "      <td>16.855717</td>\n",
       "      <td>8.301954</td>\n",
       "      <td>90.378150</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Mean_SNR    STD_SNR  Kurtosis_SNR  Skewness_SNR\n",
       "233   159.849498  76.740010     -0.575016     -0.941293\n",
       "831     4.243311  26.746490      7.110978     52.701218\n",
       "2658    1.015050  10.449662     15.593479    316.011541\n",
       "2495    2.235786  19.071848      9.659137     99.294390\n",
       "2603    2.266722  15.512103      9.062942     99.652157\n",
       "111   121.404682  47.965569      0.663053      1.203139\n",
       "1370   35.209866  60.573157      1.635995      1.609377\n",
       "1124  199.577759  58.656643     -1.862320      2.391870\n",
       "2170    0.663043   8.571517     23.415092    655.614875\n",
       "2177    3.112876  16.855717      8.301954     90.378150"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df.drop(columns='Class')\n",
    "y = df['Class']\n",
    "feature_names = df.columns.tolist()[:-1]\n",
    "\n",
    "print(X.shape)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, \n",
    "                                                    y, \n",
    "                                                    test_size=0.20, \n",
    "                                                    random_state=42)\n",
    "X_train.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And check out the y values (which turns out to be balanced):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "233     1\n",
       "831     1\n",
       "2658    0\n",
       "2495    0\n",
       "2603    0\n",
       "111     1\n",
       "1370    1\n",
       "1124    1\n",
       "2170    0\n",
       "2177    0\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1319\n",
       "1    1303\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Part A: Naïve Approach\n",
    "\n",
    "- Manual configuration\n",
    "- _\"[Babysitting is also known as Trial & Error or Grad Student Descent in the academic field](https://blog.floydhub.com/guide-to-hyperparameters-search-for-deep-learning-models/)\"_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "416a2d408b784cbd83d55cb475845b3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=1, description='max_depth', max=10, min=1), IntSlider(value=1, descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def fit_and_grapth_estimator(estimator):\n",
    "    \n",
    "    estimator.fit(X_train, y_train)\n",
    "    \n",
    "    accuracy = accuracy_score(y_train, estimator.predict(X_train))\n",
    "    print(f'Training Accuracy: {accuracy:.4f}')\n",
    "    \n",
    "    class_names = [str(i) for i in range(y_train.nunique())]\n",
    "    graph = Source(tree.export_graphviz(estimator, \n",
    "                                        out_file=None, \n",
    "                                        feature_names=feature_names, \n",
    "                                        class_names=class_names, \n",
    "                                        filled = True))\n",
    "    display(SVG(graph.pipe(format='svg')))\n",
    "    return estimator\n",
    "\n",
    "\n",
    "def plot_tree(max_depth=1, min_samples_leaf=1):\n",
    "    \n",
    "    estimator = DecisionTreeClassifier(random_state=42, \n",
    "                                       max_depth=max_depth, \n",
    "                                       min_samples_leaf=min_samples_leaf)\n",
    "    \n",
    "    return fit_and_grapth_estimator(estimator)\n",
    "\n",
    "display(interactive(plot_tree, \n",
    "                    max_depth=(1, 10, 1), \n",
    "                    min_samples_leaf=(1, 100, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "__(Test this interactively in notebook)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And test this configuration out on the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Manual: 0.8201\n"
     ]
    }
   ],
   "source": [
    "clf_manual = DecisionTreeClassifier(random_state=42, \n",
    "                                    max_depth=10, \n",
    "                                    min_samples_leaf=5)\n",
    "\n",
    "clf_manual.fit(X_train, y_train)\n",
    "accuracy_manual = accuracy_score(y_test, clf_manual.predict(X_test))\n",
    "print(f'Accuracy Manual: {accuracy_manual:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Part B: Grid Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Grid Search: \n",
    "\n",
    "- _full factorial design_ \n",
    "- Cartesian product\n",
    "- Curse of dimensionality (grows exponentially)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![title](./images/GridSearch.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "__[Grid Search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV\n",
    ")__ with Scikit Learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "parameters_GridSearch = {'max_depth':[1, 10, 100], \n",
    "                         'min_samples_leaf':[1, 10, 100],\n",
    "                        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "clf_DecisionTree = DecisionTreeClassifier(random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "GridSearch = GridSearchCV(clf_DecisionTree, \n",
    "                          parameters_GridSearch, \n",
    "                          cv=5, \n",
    "                          return_train_score=True, \n",
    "                          refit=True, \n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "GridSearch.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "GridSearch_results = pd.DataFrame(GridSearch.cv_results_)        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grid Search: \tBest parameters:  {'max_depth': 1, 'min_samples_leaf': 1} , Best scores: 0.8551\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Grid Search: \\tBest parameters: \", GridSearch.best_params_, f\", Best scores: {GridSearch.best_score_:.4f}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_max_depth</th>\n",
       "      <th>param_min_samples_leaf</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>...</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "      <th>split0_train_score</th>\n",
       "      <th>split1_train_score</th>\n",
       "      <th>split2_train_score</th>\n",
       "      <th>split3_train_score</th>\n",
       "      <th>split4_train_score</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>std_train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.002607</td>\n",
       "      <td>0.000216</td>\n",
       "      <td>0.000733</td>\n",
       "      <td>2.392449e-05</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>{'max_depth': 1, 'min_samples_leaf': 1}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.002216</td>\n",
       "      <td>0.000166</td>\n",
       "      <td>0.000658</td>\n",
       "      <td>4.802662e-05</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>{'max_depth': 1, 'min_samples_leaf': 10}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.001994</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000606</td>\n",
       "      <td>4.527192e-06</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>{'max_depth': 1, 'min_samples_leaf': 100}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.006125</td>\n",
       "      <td>0.000096</td>\n",
       "      <td>0.000628</td>\n",
       "      <td>1.497299e-06</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>{'max_depth': 10, 'min_samples_leaf': 1}</td>\n",
       "      <td>0.847619</td>\n",
       "      <td>0.822857</td>\n",
       "      <td>0.858779</td>\n",
       "      <td>...</td>\n",
       "      <td>0.841729</td>\n",
       "      <td>0.011991</td>\n",
       "      <td>8</td>\n",
       "      <td>0.956128</td>\n",
       "      <td>0.948021</td>\n",
       "      <td>0.954242</td>\n",
       "      <td>0.962345</td>\n",
       "      <td>0.957102</td>\n",
       "      <td>0.955568</td>\n",
       "      <td>0.004633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.005641</td>\n",
       "      <td>0.000071</td>\n",
       "      <td>0.000627</td>\n",
       "      <td>3.944801e-06</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>{'max_depth': 10, 'min_samples_leaf': 10}</td>\n",
       "      <td>0.845714</td>\n",
       "      <td>0.847619</td>\n",
       "      <td>0.853053</td>\n",
       "      <td>...</td>\n",
       "      <td>0.846300</td>\n",
       "      <td>0.008757</td>\n",
       "      <td>6</td>\n",
       "      <td>0.896042</td>\n",
       "      <td>0.898903</td>\n",
       "      <td>0.898475</td>\n",
       "      <td>0.893232</td>\n",
       "      <td>0.895615</td>\n",
       "      <td>0.896453</td>\n",
       "      <td>0.002066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.003822</td>\n",
       "      <td>0.000082</td>\n",
       "      <td>0.000619</td>\n",
       "      <td>8.395585e-07</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "      <td>{'max_depth': 10, 'min_samples_leaf': 100}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.006462</td>\n",
       "      <td>0.000273</td>\n",
       "      <td>0.000660</td>\n",
       "      <td>7.911454e-05</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>{'max_depth': 100, 'min_samples_leaf': 1}</td>\n",
       "      <td>0.826667</td>\n",
       "      <td>0.796190</td>\n",
       "      <td>0.841603</td>\n",
       "      <td>...</td>\n",
       "      <td>0.824190</td>\n",
       "      <td>0.015056</td>\n",
       "      <td>9</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.005753</td>\n",
       "      <td>0.000704</td>\n",
       "      <td>0.000724</td>\n",
       "      <td>2.285865e-04</td>\n",
       "      <td>100</td>\n",
       "      <td>10</td>\n",
       "      <td>{'max_depth': 100, 'min_samples_leaf': 10}</td>\n",
       "      <td>0.845714</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.854962</td>\n",
       "      <td>...</td>\n",
       "      <td>0.845154</td>\n",
       "      <td>0.009863</td>\n",
       "      <td>7</td>\n",
       "      <td>0.896996</td>\n",
       "      <td>0.899857</td>\n",
       "      <td>0.898475</td>\n",
       "      <td>0.894185</td>\n",
       "      <td>0.896568</td>\n",
       "      <td>0.897216</td>\n",
       "      <td>0.001909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.003517</td>\n",
       "      <td>0.000086</td>\n",
       "      <td>0.000574</td>\n",
       "      <td>2.499764e-05</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>{'max_depth': 100, 'min_samples_leaf': 100}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "0       0.002607      0.000216         0.000733    2.392449e-05   \n",
       "1       0.002216      0.000166         0.000658    4.802662e-05   \n",
       "2       0.001994      0.000007         0.000606    4.527192e-06   \n",
       "3       0.006125      0.000096         0.000628    1.497299e-06   \n",
       "4       0.005641      0.000071         0.000627    3.944801e-06   \n",
       "5       0.003822      0.000082         0.000619    8.395585e-07   \n",
       "6       0.006462      0.000273         0.000660    7.911454e-05   \n",
       "7       0.005753      0.000704         0.000724    2.285865e-04   \n",
       "8       0.003517      0.000086         0.000574    2.499764e-05   \n",
       "\n",
       "  param_max_depth param_min_samples_leaf  \\\n",
       "0               1                      1   \n",
       "1               1                     10   \n",
       "2               1                    100   \n",
       "3              10                      1   \n",
       "4              10                     10   \n",
       "5              10                    100   \n",
       "6             100                      1   \n",
       "7             100                     10   \n",
       "8             100                    100   \n",
       "\n",
       "                                        params  split0_test_score  \\\n",
       "0      {'max_depth': 1, 'min_samples_leaf': 1}           0.849524   \n",
       "1     {'max_depth': 1, 'min_samples_leaf': 10}           0.849524   \n",
       "2    {'max_depth': 1, 'min_samples_leaf': 100}           0.849524   \n",
       "3     {'max_depth': 10, 'min_samples_leaf': 1}           0.847619   \n",
       "4    {'max_depth': 10, 'min_samples_leaf': 10}           0.845714   \n",
       "5   {'max_depth': 10, 'min_samples_leaf': 100}           0.849524   \n",
       "6    {'max_depth': 100, 'min_samples_leaf': 1}           0.826667   \n",
       "7   {'max_depth': 100, 'min_samples_leaf': 10}           0.845714   \n",
       "8  {'max_depth': 100, 'min_samples_leaf': 100}           0.849524   \n",
       "\n",
       "   split1_test_score  split2_test_score  ...  mean_test_score  std_test_score  \\\n",
       "0           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "1           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "2           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "3           0.822857           0.858779  ...         0.841729        0.011991   \n",
       "4           0.847619           0.853053  ...         0.846300        0.008757   \n",
       "5           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "6           0.796190           0.841603  ...         0.824190        0.015056   \n",
       "7           0.849524           0.854962  ...         0.845154        0.009863   \n",
       "8           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "\n",
       "   rank_test_score  split0_train_score  split1_train_score  \\\n",
       "0                1            0.862184            0.858369   \n",
       "1                1            0.862184            0.858369   \n",
       "2                1            0.862184            0.858369   \n",
       "3                8            0.956128            0.948021   \n",
       "4                6            0.896042            0.898903   \n",
       "5                1            0.862184            0.858369   \n",
       "6                9            1.000000            1.000000   \n",
       "7                7            0.896996            0.899857   \n",
       "8                1            0.862184            0.858369   \n",
       "\n",
       "   split2_train_score  split3_train_score  split4_train_score  \\\n",
       "0            0.858913            0.859390            0.860343   \n",
       "1            0.858913            0.859390            0.860343   \n",
       "2            0.858913            0.859390            0.860343   \n",
       "3            0.954242            0.962345            0.957102   \n",
       "4            0.898475            0.893232            0.895615   \n",
       "5            0.858913            0.859390            0.860343   \n",
       "6            1.000000            1.000000            1.000000   \n",
       "7            0.898475            0.894185            0.896568   \n",
       "8            0.858913            0.859390            0.860343   \n",
       "\n",
       "   mean_train_score  std_train_score  \n",
       "0          0.859840         0.001340  \n",
       "1          0.859840         0.001340  \n",
       "2          0.859840         0.001340  \n",
       "3          0.955568         0.004633  \n",
       "4          0.896453         0.002066  \n",
       "5          0.859840         0.001340  \n",
       "6          1.000000         0.000000  \n",
       "7          0.897216         0.001909  \n",
       "8          0.859840         0.001340  \n",
       "\n",
       "[9 rows x 22 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GridSearch_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "clf_GridSearch = GridSearch.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Manual:      0.8201\n",
      "Accuracy Grid Search: 0.8430\n"
     ]
    }
   ],
   "source": [
    "accuracy_GridSearch = accuracy_score(y_test, clf_GridSearch.predict(X_test))\n",
    "print(f'Accuracy Manual:      {accuracy_manual:.4f}')\n",
    "print(f'Accuracy Grid Search: {accuracy_GridSearch:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Part C: Random Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- $B$ function evaluations, $N$ hyperparameters, $y$ number of different values:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ y_{\\mathrm{Grid Search}} = B^{1/N}, \\quad y_{\\mathrm{Random Search}} = B  $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/RandomSearch.png\" alt=\"Random Search\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- _\"This failure of grid search is the rule rather than the exception in high dimensional\n",
    "hyper-parameter optimization\"_ [Bergstra, 2012]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- useful baseline because (almost) no assumptions about the ML algorithm being optimized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "__[Random Search](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf)__ with __[Scikit Learn](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html#sklearn.model_selection.RandomizedSearchCV)__ using __[Scipy Stats](https://docs.scipy.org/doc/scipy/reference/stats.html)__ as PDFs for the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# specify parameters and distributions to sample from\n",
    "parameters_RandomSearch = {'max_depth': poisson(25), \n",
    "                           'min_samples_leaf': randint(1, 100)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# run randomized search\n",
    "n_iter_search = 9\n",
    "RandomSearch = RandomizedSearchCV(clf_DecisionTree, \n",
    "                                  param_distributions=parameters_RandomSearch, \n",
    "                                  n_iter=n_iter_search, \n",
    "                                  cv=5, \n",
    "                                  return_train_score=True,\n",
    "                                  random_state=42,\n",
    "                                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# fit the random search instance\n",
    "RandomSearch.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random Search: \tBest parameters:  {'max_depth': 26, 'min_samples_leaf': 83} , Best scores: 0.855\n"
     ]
    }
   ],
   "source": [
    "RandomSearch_results = pd.DataFrame(RandomSearch.cv_results_)                 \n",
    "print(\"Random Search: \\tBest parameters: \", RandomSearch.best_params_, f\", Best scores: {RandomSearch.best_score_:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_max_depth</th>\n",
       "      <th>param_min_samples_leaf</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>...</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "      <th>split0_train_score</th>\n",
       "      <th>split1_train_score</th>\n",
       "      <th>split2_train_score</th>\n",
       "      <th>split3_train_score</th>\n",
       "      <th>split4_train_score</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>std_train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.005301</td>\n",
       "      <td>0.000451</td>\n",
       "      <td>0.000794</td>\n",
       "      <td>0.000081</td>\n",
       "      <td>23</td>\n",
       "      <td>72</td>\n",
       "      <td>{'max_depth': 23, 'min_samples_leaf': 72}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.854308</td>\n",
       "      <td>0.010440</td>\n",
       "      <td>7</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.004121</td>\n",
       "      <td>0.000081</td>\n",
       "      <td>0.000621</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>26</td>\n",
       "      <td>83</td>\n",
       "      <td>{'max_depth': 26, 'min_samples_leaf': 83}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.005163</td>\n",
       "      <td>0.000049</td>\n",
       "      <td>0.000628</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>{'max_depth': 17, 'min_samples_leaf': 24}</td>\n",
       "      <td>0.847619</td>\n",
       "      <td>0.860952</td>\n",
       "      <td>0.868321</td>\n",
       "      <td>...</td>\n",
       "      <td>0.854310</td>\n",
       "      <td>0.012162</td>\n",
       "      <td>6</td>\n",
       "      <td>0.875536</td>\n",
       "      <td>0.879351</td>\n",
       "      <td>0.878456</td>\n",
       "      <td>0.875596</td>\n",
       "      <td>0.881792</td>\n",
       "      <td>0.878146</td>\n",
       "      <td>0.002373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003965</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000617</td>\n",
       "      <td>0.000003</td>\n",
       "      <td>27</td>\n",
       "      <td>88</td>\n",
       "      <td>{'max_depth': 27, 'min_samples_leaf': 88}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.004162</td>\n",
       "      <td>0.000120</td>\n",
       "      <td>0.000621</td>\n",
       "      <td>0.000038</td>\n",
       "      <td>31</td>\n",
       "      <td>64</td>\n",
       "      <td>{'max_depth': 31, 'min_samples_leaf': 64}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.861296</td>\n",
       "      <td>0.860031</td>\n",
       "      <td>0.001460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.003765</td>\n",
       "      <td>0.000068</td>\n",
       "      <td>0.000586</td>\n",
       "      <td>0.000027</td>\n",
       "      <td>27</td>\n",
       "      <td>89</td>\n",
       "      <td>{'max_depth': 27, 'min_samples_leaf': 89}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.860343</td>\n",
       "      <td>0.859840</td>\n",
       "      <td>0.001340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.004323</td>\n",
       "      <td>0.000066</td>\n",
       "      <td>0.000582</td>\n",
       "      <td>0.000012</td>\n",
       "      <td>22</td>\n",
       "      <td>42</td>\n",
       "      <td>{'max_depth': 22, 'min_samples_leaf': 42}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.836190</td>\n",
       "      <td>0.856870</td>\n",
       "      <td>...</td>\n",
       "      <td>0.845540</td>\n",
       "      <td>0.015574</td>\n",
       "      <td>9</td>\n",
       "      <td>0.866476</td>\n",
       "      <td>0.865999</td>\n",
       "      <td>0.862726</td>\n",
       "      <td>0.865110</td>\n",
       "      <td>0.864156</td>\n",
       "      <td>0.864893</td>\n",
       "      <td>0.001343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.003994</td>\n",
       "      <td>0.000063</td>\n",
       "      <td>0.000567</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>21</td>\n",
       "      <td>62</td>\n",
       "      <td>{'max_depth': 21, 'min_samples_leaf': 62}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.840000</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.850500</td>\n",
       "      <td>0.009778</td>\n",
       "      <td>8</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.861296</td>\n",
       "      <td>0.860031</td>\n",
       "      <td>0.001460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.003972</td>\n",
       "      <td>0.000113</td>\n",
       "      <td>0.000570</td>\n",
       "      <td>0.000026</td>\n",
       "      <td>20</td>\n",
       "      <td>64</td>\n",
       "      <td>{'max_depth': 20, 'min_samples_leaf': 64}</td>\n",
       "      <td>0.849524</td>\n",
       "      <td>0.862857</td>\n",
       "      <td>0.862595</td>\n",
       "      <td>...</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.862184</td>\n",
       "      <td>0.858369</td>\n",
       "      <td>0.858913</td>\n",
       "      <td>0.859390</td>\n",
       "      <td>0.861296</td>\n",
       "      <td>0.860031</td>\n",
       "      <td>0.001460</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "0       0.005301      0.000451         0.000794        0.000081   \n",
       "1       0.004121      0.000081         0.000621        0.000002   \n",
       "2       0.005163      0.000049         0.000628        0.000002   \n",
       "3       0.003965      0.000053         0.000617        0.000003   \n",
       "4       0.004162      0.000120         0.000621        0.000038   \n",
       "5       0.003765      0.000068         0.000586        0.000027   \n",
       "6       0.004323      0.000066         0.000582        0.000012   \n",
       "7       0.003994      0.000063         0.000567        0.000002   \n",
       "8       0.003972      0.000113         0.000570        0.000026   \n",
       "\n",
       "  param_max_depth param_min_samples_leaf  \\\n",
       "0              23                     72   \n",
       "1              26                     83   \n",
       "2              17                     24   \n",
       "3              27                     88   \n",
       "4              31                     64   \n",
       "5              27                     89   \n",
       "6              22                     42   \n",
       "7              21                     62   \n",
       "8              20                     64   \n",
       "\n",
       "                                      params  split0_test_score  \\\n",
       "0  {'max_depth': 23, 'min_samples_leaf': 72}           0.849524   \n",
       "1  {'max_depth': 26, 'min_samples_leaf': 83}           0.849524   \n",
       "2  {'max_depth': 17, 'min_samples_leaf': 24}           0.847619   \n",
       "3  {'max_depth': 27, 'min_samples_leaf': 88}           0.849524   \n",
       "4  {'max_depth': 31, 'min_samples_leaf': 64}           0.849524   \n",
       "5  {'max_depth': 27, 'min_samples_leaf': 89}           0.849524   \n",
       "6  {'max_depth': 22, 'min_samples_leaf': 42}           0.849524   \n",
       "7  {'max_depth': 21, 'min_samples_leaf': 62}           0.849524   \n",
       "8  {'max_depth': 20, 'min_samples_leaf': 64}           0.849524   \n",
       "\n",
       "   split1_test_score  split2_test_score  ...  mean_test_score  std_test_score  \\\n",
       "0           0.862857           0.862595  ...         0.854308        0.010440   \n",
       "1           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "2           0.860952           0.868321  ...         0.854310        0.012162   \n",
       "3           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "4           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "5           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "6           0.836190           0.856870  ...         0.845540        0.015574   \n",
       "7           0.840000           0.862595  ...         0.850500        0.009778   \n",
       "8           0.862857           0.862595  ...         0.855072        0.009121   \n",
       "\n",
       "   rank_test_score  split0_train_score  split1_train_score  \\\n",
       "0                7            0.862184            0.858369   \n",
       "1                1            0.862184            0.858369   \n",
       "2                6            0.875536            0.879351   \n",
       "3                1            0.862184            0.858369   \n",
       "4                1            0.862184            0.858369   \n",
       "5                1            0.862184            0.858369   \n",
       "6                9            0.866476            0.865999   \n",
       "7                8            0.862184            0.858369   \n",
       "8                1            0.862184            0.858369   \n",
       "\n",
       "   split2_train_score  split3_train_score  split4_train_score  \\\n",
       "0            0.858913            0.859390            0.860343   \n",
       "1            0.858913            0.859390            0.860343   \n",
       "2            0.878456            0.875596            0.881792   \n",
       "3            0.858913            0.859390            0.860343   \n",
       "4            0.858913            0.859390            0.861296   \n",
       "5            0.858913            0.859390            0.860343   \n",
       "6            0.862726            0.865110            0.864156   \n",
       "7            0.858913            0.859390            0.861296   \n",
       "8            0.858913            0.859390            0.861296   \n",
       "\n",
       "   mean_train_score  std_train_score  \n",
       "0          0.859840         0.001340  \n",
       "1          0.859840         0.001340  \n",
       "2          0.878146         0.002373  \n",
       "3          0.859840         0.001340  \n",
       "4          0.860031         0.001460  \n",
       "5          0.859840         0.001340  \n",
       "6          0.864893         0.001343  \n",
       "7          0.860031         0.001460  \n",
       "8          0.860031         0.001460  \n",
       "\n",
       "[9 rows x 22 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RandomSearch_results.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Manual:        0.8201\n",
      "Accuracy Grid search:   0.8430\n",
      "Accuracy Random Search: 0.8430\n"
     ]
    }
   ],
   "source": [
    "clf_RandomSearch = RandomSearch.best_estimator_\n",
    "\n",
    "accuracy_RandomSearch = accuracy_score(y_test, clf_RandomSearch.predict(X_test))\n",
    "print(f'Accuracy Manual:        {accuracy_manual:.4f}')\n",
    "print(f'Accuracy Grid search:   {accuracy_GridSearch:.4f}')\n",
    "print(f'Accuracy Random Search: {accuracy_RandomSearch:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Part D: Bayesian Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Expensive black box functions $\\Rightarrow$ need of smart guesses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. Probabilistic Surrogate Model (to be fitted)  \n",
    "   - Often Gaussian Processes \n",
    "2. Acquisition function  \n",
    "   - Exploitation / Exploration  \n",
    "   - Cheap to Computer\n",
    "  \n",
    "_[Brochu, Cora, de Freitas, 2010]_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./images/BO1.png\" alt=\"Bayesian Optimization 1\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./images/BO2.png\" alt=\"Bayesian Optimization 2\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./images/comparison_rs_bo.png\" alt=\"BO vs. RS\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "__[Bayesian Optimization](https://arxiv.org/pdf/1012.2599v1.pdf)__ with the Python package __[BayesianOptimization](https://github.com/fmfn/BayesianOptimization)__:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from bayes_opt import BayesianOptimization\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "def DecisionTree_CrossValidation(max_depth, min_samples_leaf, data, targets):\n",
    "    \"\"\"Decision Tree cross validation.\n",
    "       Fits a Decision Tree with the given paramaters to the target \n",
    "       given data, calculated a CV accuracy score and returns the mean.\n",
    "       The goal is to find combinations of max_depth, min_samples_leaf \n",
    "       that maximize the accuracy\n",
    "    \"\"\"\n",
    "    \n",
    "    estimator = DecisionTreeClassifier(random_state=42, \n",
    "                                       max_depth=max_depth, \n",
    "                                       min_samples_leaf=min_samples_leaf)\n",
    "    \n",
    "    cval = cross_val_score(estimator, data, targets, scoring='accuracy', cv=5)\n",
    "    \n",
    "    return cval.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def optimize_DecisionTree(data, targets, pars, n_iter=5):\n",
    "    \"\"\"Apply Bayesian Optimization to Decision Tree parameters.\"\"\"\n",
    "    \n",
    "    def crossval_wrapper(max_depth, min_samples_leaf):\n",
    "        \"\"\"Wrapper of Decision Tree cross validation. \n",
    "           Notice how we ensure max_depth, min_samples_leaf \n",
    "           are casted to integer before we pass them along.\n",
    "        \"\"\"\n",
    "        return DecisionTree_CrossValidation(max_depth=int(max_depth), \n",
    "                                            min_samples_leaf=int(min_samples_leaf), \n",
    "                                            data=data, \n",
    "                                            targets=targets)\n",
    "\n",
    "    optimizer = BayesianOptimization(f=crossval_wrapper, \n",
    "                                     pbounds=pars, \n",
    "                                     random_state=42, \n",
    "                                     verbose=2)\n",
    "    optimizer.maximize(init_points=4, n_iter=n_iter)\n",
    "\n",
    "    return optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   | max_depth | min_sa... |\n",
      "-------------------------------------------------\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 0.8551  \u001b[0m | \u001b[0m 38.08   \u001b[0m | \u001b[0m 95.12   \u001b[0m |\n",
      "| \u001b[0m 2       \u001b[0m | \u001b[0m 0.849   \u001b[0m | \u001b[0m 73.47   \u001b[0m | \u001b[0m 60.27   \u001b[0m |\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m 0.8524  \u001b[0m | \u001b[0m 16.45   \u001b[0m | \u001b[0m 16.44   \u001b[0m |\n",
      "| \u001b[0m 4       \u001b[0m | \u001b[0m 0.8551  \u001b[0m | \u001b[0m 6.75    \u001b[0m | \u001b[0m 86.75   \u001b[0m |\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m 0.8551  \u001b[0m | \u001b[0m 1.56    \u001b[0m | \u001b[0m 99.21   \u001b[0m |\n",
      "| \u001b[0m 6       \u001b[0m | \u001b[0m 0.8551  \u001b[0m | \u001b[0m 2.084   \u001b[0m | \u001b[0m 88.69   \u001b[0m |\n",
      "| \u001b[0m 7       \u001b[0m | \u001b[0m 0.8551  \u001b[0m | \u001b[0m 1.206   \u001b[0m | \u001b[0m 97.77   \u001b[0m |\n",
      "| \u001b[0m 8       \u001b[0m | \u001b[0m 0.8551  \u001b[0m | \u001b[0m 1.452   \u001b[0m | \u001b[0m 98.8    \u001b[0m |\n",
      "| \u001b[0m 9       \u001b[0m | \u001b[0m 0.8551  \u001b[0m | \u001b[0m 1.165   \u001b[0m | \u001b[0m 98.39   \u001b[0m |\n",
      "=================================================\n",
      "{'target': 0.8550716103235187, 'params': {'max_depth': 38.07947176588889, 'min_samples_leaf': 95.1207163345817}}\n"
     ]
    }
   ],
   "source": [
    "parameters_BayesianOptimization = {\"max_depth\": (1, 100), \n",
    "                                   \"min_samples_leaf\": (1, 100),\n",
    "                                  }\n",
    "\n",
    "BayesianOptimization = optimize_DecisionTree(X_train, \n",
    "                                             y_train, \n",
    "                                             parameters_BayesianOptimization, \n",
    "                                             n_iter=5)\n",
    "print(BayesianOptimization.max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "params = BayesianOptimization.max['params']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "for key, val in params.items():\n",
    "    params[key] = int(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "clf_BO = DecisionTreeClassifier(random_state=42, **params)\n",
    "clf_BO = clf_BO.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Manual:                0.8201\n",
      "Accuracy Grid Search:           0.8430\n",
      "Accuracy Random Search:         0.8430\n",
      "Accuracy Bayesian Optimization: 0.8430\n"
     ]
    }
   ],
   "source": [
    "accuracy_BayesianOptimization = accuracy_score(y_test, clf_BO.predict(X_test))\n",
    "print(f'Accuracy Manual:                {accuracy_manual:.4f}')\n",
    "print(f'Accuracy Grid Search:           {accuracy_GridSearch:.4f}')\n",
    "print(f'Accuracy Random Search:         {accuracy_RandomSearch:.4f}')\n",
    "print(f'Accuracy Bayesian Optimization: {accuracy_BayesianOptimization:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Part D: Full Scan over Parameter Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Only possible in low-dimensional space, slow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "max_depth_array = np.arange(1, 30)\n",
    "min_samples_leaf_array = np.arange(2, 31)\n",
    "Z = np.zeros((len(max_depth_array), len(min_samples_leaf_array)))\n",
    "\n",
    "for i, max_depth in enumerate(max_depth_array):\n",
    "    for j, min_samples_leaf in enumerate(min_samples_leaf_array):\n",
    "        \n",
    "        clf = DecisionTreeClassifier(random_state=42, \n",
    "                                     max_depth=max_depth, \n",
    "                                     min_samples_leaf=\n",
    "                                     min_samples_leaf)\n",
    "        clf.fit(X_train, y_train)\n",
    "        acc = accuracy_score(y_test, clf.predict(X_test))\n",
    "        Z[i, j] = acc\n",
    "        \n",
    "# Notice: have to transpose Z to match up with imshow\n",
    "Z = Z.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Plot the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcYAAAF0CAYAAAC5YJlXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjrUlEQVR4nO3de7RedX3n8feHGBMIBNAERC6CghdKW3QiloW1VsVS1rSo47RitdCySrsq1gu9YJeDlGk76lSd2loUqwPFC7KqtnRKCy6GeqGMJUC4Ba1AvQSRkHIHhVy+88ezj2zSJOfZ5zlPnnP2eb9Ye53n2c9v7/3deUK+5/fbv0uqCkmSNLDLpAOQJGkuMTFKktRiYpQkqcXEKElSi4lRkqQWE6MkSS1PmnQAw3rS8t1q8T57DVV28aLNY4tjt0Ubhy67qYb/vWPpLsOfV5IA7v3u93n43scyyRh+5qeX1b/fM9q/udfc8OilVXXcLIU0snmTGBfvsxfPfN+vDVX2acsfHFscR+69buiy9zy2bOiyz15210zCkbSAfegXrpx0CGy4ZzNfvfSAkc6xeL/bVsxSOLNi3iRGSdJcVGyuLZMOYlb5jFGSpBZrjJKkGStgC/2aWtQaoyRpJFtG/G86SY5L8vUktyY5YxufH5TkiiTXJbkhyfGtz34syVVJbk5yY5Kl013PGqMkacaKYvMYF6NIsgj4EHAssA64OsnFVbW2VeydwEVVdU6Sw4FLgIOTPAn4BPDGqro+yVOBaYcAjLXGmGRpkn9Jcn2Trf+g2X9Ikq822f8zSZ48zjgkSeOzhRppm8ZRwK1VdXtVPQZcCJywVZkCljev9wS+27x+JXBDVV0PUFX/XlXTji0Zd1Pqo8DLqurHgSOB45L8BPAe4ANVdShwL3DKmOOQJM1dK5Ksbm2ntj7bH/hO6/26Zl/bWcAbkqxjUFt8c7P/2UAluTTJtUl+d5hgxtqUWoPFHh9q3i5utgJeBry+2X8+g5s6Z5yxSJJmXwGbR+98s6GqVo1w/InAeVX1viRHAxckOYJBjnsx8ELgEeDyJNdU1eU7OtnYO98kWZRkDbAe+AJwG3BfVW1qimwr+0uS5okxN6XeARzYen9As6/tFOAigKq6ClgKrGCQX75UVRuq6hEGtckXTHfBsSfGqtpcVUcyuJmjgOcOe2ySU6eq1psfeGRcIUqSZqiAzVUjbdO4Gjis6ZvyZOB1wMVblfk28HKAJM9jkBjvBi4FfjTJbk1HnJ8C1jKNnTZco6ruA64Ajgb2aoKEbWf/qWPOrapVVbVq0fLddk6gkqROtoy47UjTungagyR3C4PepzcnOTvJzzfFTgd+Lcn1wKeBk2vgXuD9DJLrGuDaqvr76e5nrM8Yk6wENlbVfUl2ZdDd9j0MEuRrGfQuOgn423HGIUmav6rqEgbNoO19Z7ZerwWO2c6xn2AwZGNo4x7HuB9wfjMOZRcGmf7/JFkLXJjkD4HrgI+NOQ5J0hgUNRudb+aUcfdKvQF4/jb2387geaMkaT4r2NyvvOjMN5KkmRvMldovzpUqSVKLNUZJ0gjCZjLpIGaViVGSNGMFbPEZoyRJj7PGKElSYzBXar8So51vJElqscYoSRrJlupXjdHEKEmasT42pZoYJUkzVoTNPXsqZ2KUJI2kb02p/UrzkiSNyBqjJGnGfMYoSdIThM3Vr8ZHE6MkacYGq2v0KzH2624kSRqRNUZJ0kh8xihJUqPKZ4ySJD3BFmuMkiQNDIZr9KvG2K+7kSRpRNYYJUkj8BmjJEk/1MdxjCZGSdJINjuJuCRJ/WWNUZI0Y67HKEnSVrbY+UaSpIE+jmM0Mc5TGzbuPukQJE3Yplo06RAGTal2vpEkqb+sMUqSRuI4RkmSGlU4840kSY+Lq2tIkjSl6F+NsV93I0nSiKwxSpJG4jhGSZIaRdjSs3GMJkZJ0kj6VmPs191IkjQia4ySpBkrnERckqSWsNlxjJIkDVhjlCRpK32rMfYrzUuSNCJrjJKkGauKTamSJLX1ba5UE6MkacYKXF1DkqTHpXc1xrHeTZIDk1yRZG2Sm5O8pdl/VpI7kqxptuPHGYckScMad41xE3B6VV2bZA/gmiRfaD77QFX9yZivL0kao8E4RptSh1ZVdwJ3Nq8fTHILsP84rylJ2rn6Non4TnvGmORg4PnAV4FjgNOS/DKwmkGt8t5tHHMqcCrA4pV77qxQZ839G5cOXXbDxt07nfuzNz6/aziSeube7//zpEPo5bJTOyXNJ9kd+Czw1qp6ADgHeBZwJIMa5fu2dVxVnVtVq6pq1aLlu+2MUCVJC9zYE2OSxQyS4ier6nMAVXVXVW2uqi3AR4Gjxh2HJGk8trDLSNt0khyX5OtJbk1yxjY+P6jp6HldkhumOnQmOTjJ91sdPT88zP2MtSk1SYCPAbdU1ftb+/drnj8CvBq4aZxxSJLGowo2j7EpNcki4EPAscA64OokF1fV2laxdwIXVdU5SQ4HLgEObj67raqO7HLNcT9jPAZ4I3BjkjXNvt8HTkxyJIMOTd8Efn3McUiSxmTMzxiPAm6tqtsBklwInAC0E2MBy5vXewLfHeWC4+6V+hXY5pQIl4zzupKknWPQ+WasT+X2B77Ter8OeNFWZc4CLkvyZmAZ8IrWZ4ckuQ54AHhnVX15ugv2q4+tJGk+WpFkdWs7tePxJwLnVdUBwPHABUl2YdC586Cqej7wduBTSZbv4DyAU8JJkkY0C+sxbqiqVdv57A7gwNb7A5p9bacAxwFU1VVJlgIrqmo98Giz/5oktwHPZjBMcLusMUqSZmxq5ptRtmlcDRyW5JAkTwZeB1y8VZlvAy8HSPI8YClwd5KVTecdkjwTOAy4fboLWmOUJI1gvM8Yq2pTktOAS4FFwMer6uYkZwOrq+pi4HTgo0nexiBXn1xVleQlwNlJNgJbgN+oqnumu6aJUZI0knEvO1VVl7BVp82qOrP1ei2DURBbH/dZBuPoO7EpVZKkFmuMkqQZG/cA/0kwMUqSRjLmcYw7nYlRkjRjrq4hSVLPWWOUJI1k3L1SdzYToyRpxqYG+PeJiVGSNBI730iSNGW4ad3mlX6leUmSRmSNUZI0Y4WdbyRJeoK+NaWaGCVJM2avVEmSttK3xGjnG0mSWqwxSpJmrI9zpZoYJUkjsVeqJElTymeMkiT1mjVGSdKMOVxDkqStmBglSWrYK1WSpK2UiVHDeuFe3xq67HOW3tnp3F9e+ayu4UjqmbsXb550CL1kYpQkjcRxjJIkNaqH4xhNjJKkkfiMUZKkH+pfr1RnvpEkqcUaoyRpJDalSpLUcEo4SZLaatAztU98xihJUos1RknSSBzgr7G48sHDOpVff/fyMUUiab7YtHHRpEOgsPONJEkt/RvHaGKUJI3EzjeSJPWYNUZJ0kh8xihJUqPKxChJ0hPY+UaSpJYF1/kmyTHNzyVdT57kwCRXJFmb5OYkb2n2PyXJF5J8o/m5d/fQJUmafcP0Sv1g8/OqGZx/E3B6VR0O/ATwpiSHA2cAl1fVYcDlzXtJ0jxUlZG2uWaYptSNSc4F9k/ywa0/rKrf2t6BVXUncGfz+sEktwD7AycAL22KnQ/8E/B7nSKXJE1cMTeT2yiGSYz/GXgF8DPANTO9UJKDgecDXwX2bZImwPeAfbdzzKnAqQCLV+4500tLksaoZ48Yp0+MVbUBuDDJLVV1/UwukmR34LPAW6vqgeTx3y6qqpJs88+1qs4FzgXY9dCn9+3PXpI0B3Xplfr1JG8CfgRYOrWzqn51RwclWcwgKX6yqj7X7L4ryX5VdWeS/YD1HeOWJM0FPRzH2GVKuAuApzFoUv0icADw4I4OyKBq+DHglqp6f+uji4GTmtcnAX/bIQ5J0lxSI25zTJca46FV9V+TnFBV5yf5FPDlaY45BngjcGOSNc2+3wfeDVyU5BTgW8AvdIxbkjRH9K3G2CUxbmx+3pfkCAadZvbZ0QFV9RXY7gqWL+9wbUnSHNW3Af5dEuO5zUD8/8agKXR34MyxRCVJ0oQMnRir6i+bl18EnjmecCRJ80nRv6bUoTvfJNk3yceS/EPz/vDmGaEkaaEqoDLaNsd06ZV6HnAp8PTm/b8Cb53leCRJ88xg6amZb3NNl8S4oqouArYAVNUmYPNYopIkaUK6JMaHkzyVZtRJkp8A7h9LVJKk+WPM4xiTHJfk60luTfIfFp1IclCzktN1SW5Icvw2Pn8oyW8PcztdeqW+nUFv1GcluRJYCby2w/GSpN4Z7yTiSRYBHwKOBdYBVye5uKrWtoq9E7ioqs5pVnC6BDi49fn7gX8Y9ppdeqVem+SngOcwGJv49araOM1hkqS+G+9zwqOAW6vqdoAkFzJYoamdGAtY3rzeE/ju1AdJXgX8G/DwsBecNjEmec12Pnp2Elrzn0qSFprxz5W6P/Cd1vt1wIu2KnMWcFmSNwPLGKwINbWAxe8xqG0O1YwKw9UYf24HnxVgYpQkjWJFktWt9+c2qysN60TgvKp6X5KjgQuaGdrOAj5QVQ+1V3WazjDLTv3KMCdKclJVnT/0lSVJ/TB6U+qGqlq1nc/uAA5svT+g2dd2CnAcQFVdlWQpsIJBzfK1Sd4L7AVsSfKDqvrzHQXTpVfqdN4yi+eSJM0bGXHboauBw5IckuTJwOsYdARt+zbN/NtJnsdgacS7q+onq+rgqjoY+F/AH0+XFKFbr9TpzL3pCyRJ4zfGzjdVtSnJaQwmmFkEfLyqbk5yNrC6qi4GTgc+muRtTTQnV8186oDZTIxzcP4CSdLYjflf/6q6hMEQjPa+M1uv1zJY5nBH5zhr2OvNZlOqNUZJ0rw3mzXGK2fxXJKk+WBqEvEe6bK6xluSLM/Ax5Jcm+SVU59X1WnjCVGSNJf1bRLxLjXGX62qP03yM8DewBuBC4DLxhLZArPm3gM6ld9l/ZKhy+566PBT2j78nT06xSFpgjbPkZraHExuo+jyjHHqGzgeuKCqbsbnipKknulSY7wmyWXAIcA7kuxBswSVJGkB69kzxi6J8RTgSOD2qnqkWYJqqFlxJEn9lQXclFrA4cBvNe+XMZhdQJK0UI26FuMcTKpdEuNfAEczmKwV4EEGa2RJkhasDJpSR9nmmC5NqS+qqhckuQ6gqu5t5q2TJKk3uiTGjc1KygWQZCV2vpEkzcHm0FF0aUr9IPB5YJ8kfwR8BfjjsUQlSZo/evaMcegaY1V9Msk1DJb2CPCqqrplbJFJkuaHOZjcRjFtYkzylNbb9cCn259V1T3jCEySNA/0cK7UYWqM1zC49W3deQHPnNWIeuTVy9cMXXbFkx7sdO733L13x2iGU0uGf2y874H3djr3Hkse7RqOpB24Z9fHJh1CL02bGKvqkJ0RiCRpfurbAP9Oy04leQ3wYgY1xS9X1d+MIyhJ0jzSs8TYZdmpvwB+A7gRuAn4jSQO8Jck9UqXGuPLgOdV1dQ4xvOBm8cSlSRJE9JlHOOtwEGt9wc2+yRJC1hqtG2u6VJj3AO4Jcm/NO9fCKxOcjFAVf38bAcnSZoHFuBwjSlnji0KSdL8NEdnrxlFl5lvvgiQZHn7OAf4S9ICt1ATY5JTgbOBHzCYPDw4wF+S1DNdmlJ/BziiqjaMKxhJ0vwzFzvQjKJLYrwNeGRcgSx0GzbtMbZzP/ro8F/z7vs8PHTZhx91OU5pkjZXl4EFY7SAE+M7gH9O8lXgh5NeVtVvzXpUkqT5YwEnxo8A/5fBzDcuUCxJmrNjEUfRJTEurqq3jy0SSZLmgC6J8R+anql/xxObUh2uIUkL2QIe4H9i8/MdrX0O15CkhW6hNqW6LqMkaVsW8jNGkhwBHA4sndpXVX8120FJkjQpXdZjfBfwZ83208B7gR1OHJ7k40nWJ7mpte+sJHckWdNsx88wdknSXFAjbnNMl9GhrwVeDnyvqn4F+HFgz2mOOQ84bhv7P1BVRzbbJR1ikCTNJSMuOTUXm2G7JMbvV9UWYFMzkfh6BmsybldVfQmw16ok9VnPaoxdnjGuTrIX8FHgGuAh4KoZXve0JL8MrAZOr6p7Z3ie3rj6vmd0Kr/x/iVDl82jw//+s2nPjUOXrR8sGroswMNLl05fSNLQNm3s9v/g2MzB5DaKof/FrKrfrKr7qurDwLHASU2TalfnAM8CjgTuBN63vYJJTk2yOsnqzQ84Taskafy6dL45Jsmy5u2LgZOTdKvmAFV1V1VtbpplPwoctYOy51bVqqpatWj5bl0vJUnaCRbyM8ZzgEeS/DhwOoPVNjoP1UiyX+vtq4GbtldWkqSdrcszxk1VVUlOAP68qj6W5JQdHZDk08BLgRVJ1gHvAl6a5EgGrdLfBH59JoFLkuaIOVjrG0WXxPhgkncAbwBekmQXYPGODqiqE7ex+2MdrilJmsvmaHPoKLo0pf4ig8nDT6mq7wEHAP9zLFFJkjQhXeZK/R7w/tb7b9N6xpjkqqo6enbDkyTNeT2rMXaaK3UaDlKTpIXIxLhdPfujkSRNJyzsZ4ySJPXebNYY+7WEsyRpOD2rMc5mYnzjLJ5rpzpo2fBTtT531zuHLvvsxcumL9Q4ZNm/D10WYM0Dhw1ddtdD7x+67LIlj3WKo4sD9rhvbOeWFqIrxvj/69AW8nCNJK9J8o0k9yd5IMmDSR6Y+ryqnMFGkhaiBby6xnuBn6uqW8YVjCRpHpqDyW0UXTrf3GVSlCT1Xdf1GD8D/A2DGXAAqKrPzXZQkqT5o2/PGLskxuXAI8ArW/sKMDFK0kK2UBPjDBclliT12RztQDOKaRNjkt+tqvcm+TO2cftV9VtjiUySNC/0rSl1mM43Ux1uVm9nkyRpbJIcl+TrSW5NcsY2Pj8oyRVJrktyQ5Ljm/1HJVnTbNcnefUw15u2xlhVf9e8XAv8PnBw67iitcKGJGkBGmONMcki4EPAscA64OokF1fV2laxdwIXVdU5SQ4HLmGQq24CVlXVpiT7Adcn+buq2rSja3bpfPMJ4HeAG4EtHY6TJPXYmJtSjwJurarbAZJcCJzAoLI2pRh0EAXYE/guQFU90iqzlCFTeJfEeHdVXdyhvMZo8/Id/sLzBE9b/uBYYjhy73Wdyq9Y/NBY4pAWqn9Z9Oj0hXaG8SbG/YHvtN6vA160VZmzgMuSvBlYBrxi6oMkLwI+DjwDeON0tUXolhjfleQvgctxHKMkafasSNLus3JuVZ3b4fgTgfOq6n1JjgYuSHJEVW2pqq8CP5LkecD5Sf6hqn6wo5N1SYy/AjwXWMzjTamOY5SkhWx2hmtsqKpV2/nsDuDA1vsDmn1tpwDHAVTVVUmWAiuA9T8Ms+qWJA8BRzBNx9EuifGFVfWcDuUlST0Xxr7m4NXAYUkOYZAQXwe8fqsy3wZeDpzX1AyXAnc3x3yn6XzzDAaVu29Od8EuifGfkxy+VU8gSdJCN8ZnjE1SOw24FFgEfLyqbk5yNrC66ftyOvDRJG9rojm5qirJi4Ezkmxk0NL5m1W1YbprdkmMPwGsSfJvDJ4xZhBz/ViXm5Qk9cu4B/hX1SUMhmC0953Zer0WOGYbx10AXND1el0S43FdTy5J0nzTZa7Ub40zEEnSPNWzKeG61BglSfqPTIySJDVqYU4iLknSgmGNcY5Yc+8Bkw4BgNvWrRy67Lfu3rvTuTfev6RrOJJ24O6Hb5h0CAM9qzGaGCVJI+lbU6qJUZI0GhOjJEmP61uN0c43kiS1WGOUJM3c7KyuMaeYGCVJozExSpI0EPr3jNHEKEkaTc8So51vJElqscYoSRpJql9VRhOjJGnm7JWqLq78wZahy3add3T3WxcPXfa2pcPPf7rL+uHnM920ZPgYYPCQXtIs2jI3/q/qW+cbnzFKktRijVGSNJqe1RhNjJKkkfStKdXEKEkajYlRkqRG9a/GaOcbSZJaxpoYk3w8yfokN7X2PSXJF5J8o/nZbZyCJGluqRG3OWbcNcbzgOO22ncGcHlVHQZc3ryXJM1DU5OIj7LNNWNNjFX1JeCerXafAJzfvD4feNU4Y5AkjVnVaNscM4nON/tW1Z3N6+8B+04gBknSLJmLtb5RTLRXalVVsv0/0iSnAqcCLF65506Laz548v3D/018dMmmoctu2mcm0Qzp/m5TyA2rlgw/9R5AHrXPmaTtm8S/EHcl2Q+g+bl+ewWr6tyqWlVVqxYt322nBShJGtKoHW/mYG1zEonxYuCk5vVJwN9OIAZJ0izJltG2uWbcwzU+DVwFPCfJuiSnAO8Gjk3yDeAVzXtJ0nzVsxrjWJ8xVtWJ2/no5eO8riRJM+WUcJKkkdgrVZKkKcWcHIs4ChOjJGkk1hglSWrrWWJ0pLMkSS3WGCVJMzY1iXifmBjHaOWi7w9ddkmHadsA7j9s+LJdmgXqB4uGLrt4z0c7nBk2Lhn+3OPUdQo5ac7aZQ5kpDk6EfgoTIySpJFYY5Qkqa1nidHON5IktVhjlCSNxKZUSZKmFLClX5nRxChJGk2/8qLPGCVJarPGKEkaic8YJUlqc4C/JEmPs8aooa3cJUOXfc6K9Z3Ofe0DS4cue8jKe4cue+v9Txu67DM6nBeAlcMX/dbdew9dduP9SzqF0WUqu6csf6TTuaWdacOTN086hGY9xkkHMbvsfCNJUos1RknSjA1W1+hXldHEKEkaTc8WrDExSpJGYo1RkqQpdr6RJGnnSnJckq8nuTXJGdv4/KAkVyS5LskNSY5v9h+b5JokNzY/XzbM9awxSpJGUGMd4J9kEfAh4FhgHXB1kouram2r2DuBi6rqnCSHA5cABwMbgJ+rqu8mOQK4FNh/umuaGCVJIxnzAP+jgFur6naAJBcCJwDtxFjA8ub1nsB3AarqulaZm4Fdkyypqh0OZjYxSpJGM3qNcUWS1a3351bVuc3r/YHvtD5bB7xoq+PPAi5L8mZgGfCKbVzjvwDXTpcUwcQoSZq8DVW1aoTjTwTOq6r3JTkauCDJEVW1BSDJjwDvAV45zMlMjJKkmSvIeMcx3gEc2Hp/QLOv7RTgOICquirJUmAFsD7JAcDngV+uqtuGuaCJcY742RU3dSq/9tJnD132e8v3GLpsHh2+o/JtNz996LIAtWT4/3u6xDH8jLQDXeZWvavjPKzSzrTpsUWTDmFgvOMYrwYOS3IIg4T4OuD1W5X5NvBy4LwkzwOWAncn2Qv4e+CMqrpy2As6XEOSNJoacdvRqas2Aacx6FF6C4PepzcnOTvJzzfFTgd+Lcn1wKeBk6uqmuMOBc5MsqbZ9pnudqwxSpJGMu6Zb6rqEgZDMNr7zmy9Xgscs43j/hD4w67Xs8YoSVKLNUZJ0micK1WSpEbh6hqSJE0J5eoakiQ9Qc8So51vJElqscYoSRpNz2qMJkZJ0szZ+UYbNg0/vdrajUvHFsce3x7+N7T7lu45dNmDv7Rx6LKLHtk0dFmAR54+/J/HHrc9OHwc/z58WYAt6zcMX/aRRzqdW9qZ7q258fezb51vfMYoSVKLNUZJ0mh6VmM0MUqSRlAmRkmSfqgwMUqS9AQ965Vq5xtJklqsMUqSRtK34RoTS4xJvgk8CGwGNlXVqknFIkkagYlxVv10VQ0/2lqSNLcUsMXEKElSw+Eas6mAy5IU8JGqOnfrAklOBU4FWLxy+GnNFoJH98rQZfd+wfqhy37zwOXDB3H/kuHLAvscOnzjwL9du3Losk++v9vfjc1LDhi67P7/NPyUW12mvJNmw5bL/t+kQ+ilSSbGF1fVHUn2Ab6Q5GtV9aV2gSZZnguw66FP79evJJLUFz2rMU5suEZV3dH8XA98HjhqUrFIkkZQNdo2x0wkMSZZlmSPqdfAK4GbJhGLJGkEU51vRtnmmEk1pe4LfD7JVAyfqqp/nFAskiT90EQSY1XdDvz4JK4tSZpNBdWvOeEcriFJGs0cfE44ChOjJGnmHOAvSdJWelZjdHUNSZJarDFKkkbTsxqjiXGMVi76/tBlvzbGODbcPPz0asu/M/xUc13d+8DwcSz6wfBxPLZnt/8pH1u5aeiy3zt6t6HLbu4wQ96u67vF/NBBnYprgdj05UlHAM6VKklSWwFb+jVcw2eMkiS1WGOUJI3GplRJklpMjJIkTZmbE4GPwsQoSZq5gurZXKl2vpEkqcUaoyRpNDalSpLUYucbSZIaVb0b4G9ilCSNxhqjxuGpT3qoU/nHlo8njocOHP4v+Oblw8852lUeHb5f2C4dynY9d5f5TzcvHf7P7t4juv2GvfiB8cSs+a3sPjkWJkZJ0kjKplRJkqa4uoYkSY8rejdcwxZqSZJarDFKkkbTsynhTIySpBkroHrWlGpilCTNXJU1RkmS2vpWY7TzjSRJLdYYJUmj6VlTamqeDMxMcjfwrUnH0dEKYMOkgxgj72/+6/s99v3+nlNVe0wygCT/yODPeRQbquq42YhnNsybxDgfJVldVasmHce4eH/zX9/v0fvTTPiMUZKkFhOjJEktJsbxOnfSAYyZ9zf/9f0evT915jNGSZJarDFKktRiYhyDJN9McmOSNUlWTzqe2ZDk40nWJ7mpte8pSb6Q5BvNz70nGeMotnN/ZyW5o/ke1yQ5fpIxjiLJgUmuSLI2yc1J3tLs78V3uIP769N3uDTJvyS5vrnHP2j2H5Lkq0luTfKZJE+edKzznU2pY5Dkm8CqqurN+KkkLwEeAv6qqo5o9r0XuKeq3p3kDGDvqvq9ScY5U9u5v7OAh6rqTyYZ22xIsh+wX1Vdm2QP4BrgVcDJ9OA73MH9/QL9+Q4DLKuqh5IsBr4CvAV4O/C5qrowyYeB66vqnEnGOt9ZY9RQqupLwD1b7T4BOL95fT6Df4jmpe3cX29U1Z1VdW3z+kHgFmB/evId7uD+eqMGHmreLm62Al4G/HWzf95+h3OJiXE8CrgsyTVJTp10MGO0b1Xd2bz+HrDvJIMZk9OS3NA0tc7LZsatJTkYeD7wVXr4HW51f9Cj7zDJoiRrgPXAF4DbgPuqalNTZB09+4VgEkyM4/HiqnoB8LPAm5pmul6rQZt839rlzwGeBRwJ3Am8b6LRzIIkuwOfBd5aVQ+0P+vDd7iN++vVd1hVm6vqSOAA4CjguZONqJ9MjGNQVXc0P9cDn2fwF7iP7mqe7Uw941k/4XhmVVXd1fxDtAX4KPP8e2yeS30W+GRVfa7Z3ZvvcFv317fvcEpV3QdcARwN7JVkakGIA4A7JhVXX5gYZ1mSZc3Df5IsA14J3LTjo+ati4GTmtcnAX87wVhm3VTCaLyaefw9Nh03PgbcUlXvb33Ui+9we/fXs+9wZZK9mte7AscyeJZ6BfDapti8/Q7nEnulzrIkz2RQS4TBsl6fqqo/mmBIsyLJp4GXMphF/y7gXcDfABcBBzFY+eQXqmpedmDZzv29lEETXAHfBH699TxuXknyYuDLwI3A1BpBv8/gOdy8/w53cH8n0p/v8McYdK5ZxKBSc1FVnd38m3Mh8BTgOuANVfXo5CKd/0yMkiS12JQqSVKLiVGSpBYToyRJLSZGSZJaTIySJLWYGCVJajExSiNolhhbMcNjT07y9Nk4l6TZY2KUJudk4OnTFZK0c5kY1QtJDk7ytSTnJfnXJJ9M8ookVzaL8B7VbFcluS7JPyd5TnPs25J8vHn9o0luSrLbdq7z1CSXNQvF/iWQ1mdvaBaSXZPkI0kWNfsfSvKB5pjLm6m9XgusAj7ZlN+1Oc2bk1ybwULXThAtTYCJUX1yKIPVE57bbK8HXgz8NoPpwb4G/GRVPR84E/jj5rg/BQ5N8mrgfzOYNuyR7VzjXcBXqupHGEz9dxBAkucBvwgc06x+sBn4peaYZcDq5pgvAu+qqr8GVgO/VFVHVtX3m7IbmpVZzmnilrSTPWn6ItK88W9VdSNAkpuBy6uqktwIHAzsCZyf5DAGc2cuBqiqLUlOBm4APlJVV+7gGi8BXtMc9/dJ7m32vxz4T8DVg/ms2ZXHV6rYAnymef0J4HNs39Rn10xdR9LOZWJUn7QnTt7Ser+Fwd/1/w5cUVWvbhaz/adW+cOAh5j5M78A51fVO4You6MJiqdi3oz/f0oTYVOqFpI9eXytupOndibZE/ggg9rgU5vnf9vzJQZNtCT5WWBqRfjLgdcm2af57ClJntF8tguPLwv0euArzesHgT1GuB9JY2Bi1ELyXuB/JLmOJ9bGPgB8qKr+FTgFePdUgtuGPwBe0jTVvgb4NkBVrQXeCVyW5AbgC8DUWoAPA0cluQl4GXB2s/884MNbdb6RNGEuOyWNWZKHqmr3ScchaTjWGCVJarHGKG1Dkl8B3rLV7iur6k2TiEfSzmNilCSpxaZUSZJaTIySJLWYGCVJajExSpLUYmKUJKnl/wPj8pLIuoFglgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "# notice that we are setting the extent and origin keywords\n",
    "CS = ax.imshow(Z, extent=[1, 30, 2, 31], cmap='viridis', origin='lower')\n",
    "ax.set(xlabel='max_depth', ylabel='min_samples_leaf')\n",
    "\n",
    "fig.colorbar(CS);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sum up:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"./images/Chart.png\" alt=\"Chart\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "__[Guide To Hyperparameters Search For Deep Learning Models](https://blog.floydhub.com/guide-to-hyperparameters-search-for-deep-learning-models/)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Part E: New Methods\n",
    "\n",
    "Bayesian Optimization meets HyperBand (BOHB)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "HyperBand:\n",
    "\n",
    "\n",
    "<img src=\"./images/sh.gif\" alt=\"BO vs. RS\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./images/comparison_rs_bo.png\" alt=\"BO vs. RS\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./images/comparison-1.png\" alt=\"BOHB\" width=\"600\"/>\n",
    "\n",
    "__[BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://www.automl.org/blog_bohb/)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Part F: New Software\n",
    "\n",
    "__[Optuna](https://optuna.org/)__ is a HyperParameter Optimisation framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/lightgbm/__init__.py:48: UserWarning: Starting from version 2.2.1, the library file in distribution wheels for macOS is built by the Apple Clang (Xcode_8.3.3) compiler.\n",
      "This means that in case of installing LightGBM from PyPI via the ``pip install lightgbm`` command, you don't need to install the gcc compiler anymore.\n",
      "Instead of that, you need to install the OpenMP library, which is required for running LightGBM on the system with the Apple Clang compiler.\n",
      "You can install the OpenMP library by the following command: ``brew install libomp``.\n",
      "  \"You can install the OpenMP library by the following command: ``brew install libomp``.\", UserWarning)\n"
     ]
    }
   ],
   "source": [
    "import optuna\n",
    "from optuna.samplers import TPESampler\n",
    "from optuna.integration import LightGBMPruningCallback\n",
    "from optuna.pruners import MedianPruner\n",
    "import lightgbm as lgb\n",
    "\n",
    "lgb_data_train = lgb.Dataset(X_train, label=y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def objective(trial):\n",
    "\n",
    "    boosting_types = [\"gbdt\", \"rf\", \"dart\"]\n",
    "    boosting_type = trial.suggest_categorical(\"boosting_type\", boosting_types)\n",
    "\n",
    "    params = {\n",
    "        \"objective\": \"binary\",\n",
    "        \"metric\": 'auc',\n",
    "        \"boosting\": boosting_type,\n",
    "        \"max_depth\": 5,\n",
    "        \"max_depth\": trial.suggest_int(\"max_depth\", 2, 63),\n",
    "        \"min_child_weight\": trial.suggest_loguniform(\"min_child_weight\", 1e-5, 10),\n",
    "        \"scale_pos_weight\": trial.suggest_uniform(\"scale_pos_weight\", 10.0, 30.0),\n",
    "        \"bagging_freq\": 1, \"bagging_fraction\": 0.6,\n",
    "        \"verbosity\": -1\n",
    "    }\n",
    "\n",
    "    N_iterations_max = 10_000\n",
    "    early_stopping_rounds = 50\n",
    "\n",
    "    if boosting_type == \"dart\":\n",
    "        N_iterations_max = 100\n",
    "        early_stopping_rounds = None\n",
    "\n",
    "    cv_res = lgb.cv(\n",
    "        params,\n",
    "        lgb_data_train,\n",
    "        num_boost_round=N_iterations_max,\n",
    "        early_stopping_rounds=early_stopping_rounds,\n",
    "        verbose_eval=False,\n",
    "        seed=42,\n",
    "        callbacks=[LightGBMPruningCallback(trial, \"auc\")],\n",
    "    )\n",
    "\n",
    "    num_boost_round = len(cv_res[\"auc-mean\"])\n",
    "    trial.set_user_attr(\"num_boost_round\", num_boost_round)\n",
    "\n",
    "    return cv_res[\"auc-mean\"][-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2024-04-27 23:24:42,308]\u001b[0m A new study created in memory with name: no-name-6ea2c63c-26ba-43b4-9c3a-8490a9df1fd1\u001b[0m\n",
      "/opt/anaconda3/lib/python3.7/site-packages/optuna/progress_bar.py:47: ExperimentalWarning: Progress bar is experimental (supported from v1.2.0). The interface can change in the future.\n",
      "  self._init_valid()\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d39a5751534843978d312933d4caa46b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2024-04-27 23:24:42,613]\u001b[0m Trial 0 finished with value: 0.9257274934861961 and parameters: {'boosting_type': 'rf', 'max_depth': 39, 'min_child_weight': 8.632008168602535e-05, 'scale_pos_weight': 13.119890406724053}. Best is trial 0 with value: 0.9257274934861961.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:42,889]\u001b[0m Trial 1 finished with value: 0.925645717219157 and parameters: {'boosting_type': 'rf', 'max_depth': 45, 'min_child_weight': 1.3289448722869181e-05, 'scale_pos_weight': 29.398197043239886}. Best is trial 0 with value: 0.9257274934861961.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:43,210]\u001b[0m Trial 2 finished with value: 0.9327382743380188 and parameters: {'boosting_type': 'gbdt', 'max_depth': 13, 'min_child_weight': 0.0006690421166498799, 'scale_pos_weight': 20.495128632644757}. Best is trial 2 with value: 0.9327382743380188.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:43,546]\u001b[0m Trial 3 finished with value: 0.9293024673628942 and parameters: {'boosting_type': 'dart', 'max_depth': 10, 'min_child_weight': 0.0005660670699258885, 'scale_pos_weight': 17.327236865873836}. Best is trial 2 with value: 0.9327382743380188.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:43,785]\u001b[0m Trial 4 finished with value: 0.9261444485440651 and parameters: {'boosting_type': 'rf', 'max_depth': 33, 'min_child_weight': 0.035849855803404704, 'scale_pos_weight': 10.929008254399955}. Best is trial 2 with value: 0.9327382743380188.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:44,122]\u001b[0m Trial 5 finished with value: 0.9311118522518811 and parameters: {'boosting_type': 'gbdt', 'max_depth': 60, 'min_child_weight': 6.220025976819156, 'scale_pos_weight': 26.16794696232922}. Best is trial 2 with value: 0.9327382743380188.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:44,255]\u001b[0m Trial 6 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:44,404]\u001b[0m Trial 7 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:44,539]\u001b[0m Trial 8 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:44,679]\u001b[0m Trial 9 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:44,733]\u001b[0m Trial 10 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:45,067]\u001b[0m Trial 11 finished with value: 0.9309290578399502 and parameters: {'boosting_type': 'gbdt', 'max_depth': 63, 'min_child_weight': 6.894420684704665, 'scale_pos_weight': 25.370137676514304}. Best is trial 2 with value: 0.9327382743380188.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:45,412]\u001b[0m Trial 12 finished with value: 0.9323419999728675 and parameters: {'boosting_type': 'gbdt', 'max_depth': 63, 'min_child_weight': 0.38601839366790247, 'scale_pos_weight': 23.675597363736365}. Best is trial 2 with value: 0.9327382743380188.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:45,766]\u001b[0m Trial 13 finished with value: 0.9327630495788028 and parameters: {'boosting_type': 'gbdt', 'max_depth': 21, 'min_child_weight': 0.47333235522568584, 'scale_pos_weight': 22.116723260828707}. Best is trial 13 with value: 0.9327630495788028.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:46,112]\u001b[0m Trial 14 finished with value: 0.9318985480712734 and parameters: {'boosting_type': 'gbdt', 'max_depth': 20, 'min_child_weight': 0.0018017260723627204, 'scale_pos_weight': 20.87074372077165}. Best is trial 13 with value: 0.9327630495788028.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:46,481]\u001b[0m Trial 15 finished with value: 0.933907588579396 and parameters: {'boosting_type': 'gbdt', 'max_depth': 23, 'min_child_weight': 0.5262050790257533, 'scale_pos_weight': 17.101924194245463}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:46,828]\u001b[0m Trial 16 finished with value: 0.9323761434494685 and parameters: {'boosting_type': 'gbdt', 'max_depth': 25, 'min_child_weight': 0.7139629318981857, 'scale_pos_weight': 15.599819505406096}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:46,914]\u001b[0m Trial 17 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:47,261]\u001b[0m Trial 18 finished with value: 0.9320047563876231 and parameters: {'boosting_type': 'gbdt', 'max_depth': 20, 'min_child_weight': 0.07473651026301241, 'scale_pos_weight': 18.745829052448148}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:47,436]\u001b[0m Trial 19 pruned. Trial was pruned at iteration 61.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:47,796]\u001b[0m Trial 20 finished with value: 0.9328372233375839 and parameters: {'boosting_type': 'gbdt', 'max_depth': 21, 'min_child_weight': 1.5423777057795445, 'scale_pos_weight': 13.233732216784787}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:48,088]\u001b[0m Trial 21 finished with value: 0.9321245240568861 and parameters: {'boosting_type': 'gbdt', 'max_depth': 22, 'min_child_weight': 1.5417332097862932, 'scale_pos_weight': 10.107893557559724}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:48,477]\u001b[0m Trial 22 finished with value: 0.9323386454194766 and parameters: {'boosting_type': 'gbdt', 'max_depth': 26, 'min_child_weight': 0.13230118984954353, 'scale_pos_weight': 12.488404178785878}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:48,659]\u001b[0m Trial 23 pruned. Trial was pruned at iteration 66.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:49,080]\u001b[0m Trial 24 finished with value: 0.9324599798628743 and parameters: {'boosting_type': 'gbdt', 'max_depth': 29, 'min_child_weight': 0.22938248553821958, 'scale_pos_weight': 18.52693332188188}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:49,385]\u001b[0m Trial 25 finished with value: 0.9323709490119938 and parameters: {'boosting_type': 'gbdt', 'max_depth': 10, 'min_child_weight': 0.014223320052048582, 'scale_pos_weight': 11.577261921985883}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:49,523]\u001b[0m Trial 26 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:49,868]\u001b[0m Trial 27 finished with value: 0.932944700613427 and parameters: {'boosting_type': 'gbdt', 'max_depth': 37, 'min_child_weight': 0.018459882906772026, 'scale_pos_weight': 17.679098751711575}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:50,222]\u001b[0m Trial 28 finished with value: 0.9327459260646458 and parameters: {'boosting_type': 'gbdt', 'max_depth': 38, 'min_child_weight': 0.006874274339719773, 'scale_pos_weight': 14.404674789343591}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:50,354]\u001b[0m Trial 29 pruned. Trial was pruned at iteration 52.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:50,753]\u001b[0m Trial 30 finished with value: 0.9327176880494863 and parameters: {'boosting_type': 'gbdt', 'max_depth': 51, 'min_child_weight': 0.003257102075046086, 'scale_pos_weight': 17.435919490699508}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:50,879]\u001b[0m Trial 31 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:51,026]\u001b[0m Trial 32 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:51,381]\u001b[0m Trial 33 finished with value: 0.9326676138454196 and parameters: {'boosting_type': 'gbdt', 'max_depth': 23, 'min_child_weight': 2.9350751934489323, 'scale_pos_weight': 17.738460794183766}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:51,513]\u001b[0m Trial 34 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:51,657]\u001b[0m Trial 35 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:51,794]\u001b[0m Trial 36 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:51,949]\u001b[0m Trial 37 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:52,077]\u001b[0m Trial 38 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:52,439]\u001b[0m Trial 39 finished with value: 0.9331977959415845 and parameters: {'boosting_type': 'gbdt', 'max_depth': 35, 'min_child_weight': 0.000341792895071912, 'scale_pos_weight': 10.091883058577956}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:52,578]\u001b[0m Trial 40 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:52,949]\u001b[0m Trial 41 finished with value: 0.9330434512538387 and parameters: {'boosting_type': 'gbdt', 'max_depth': 36, 'min_child_weight': 0.0010904325188218595, 'scale_pos_weight': 10.103269089092729}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2024-04-27 23:24:53,302]\u001b[0m Trial 42 finished with value: 0.9330358105424482 and parameters: {'boosting_type': 'gbdt', 'max_depth': 35, 'min_child_weight': 0.0004148881183051914, 'scale_pos_weight': 10.017562759585447}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:53,447]\u001b[0m Trial 43 pruned. Trial was pruned at iteration 56.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:53,595]\u001b[0m Trial 44 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:53,954]\u001b[0m Trial 45 finished with value: 0.9329793149860837 and parameters: {'boosting_type': 'gbdt', 'max_depth': 41, 'min_child_weight': 0.001167290226730846, 'scale_pos_weight': 10.044747367447195}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:54,099]\u001b[0m Trial 46 pruned. Trial was pruned at iteration 56.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:54,261]\u001b[0m Trial 47 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:54,394]\u001b[0m Trial 48 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:54,751]\u001b[0m Trial 49 finished with value: 0.9331977959415845 and parameters: {'boosting_type': 'gbdt', 'max_depth': 31, 'min_child_weight': 0.0029537732097114785, 'scale_pos_weight': 10.091379098007323}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:55,106]\u001b[0m Trial 50 finished with value: 0.9332437207336415 and parameters: {'boosting_type': 'gbdt', 'max_depth': 31, 'min_child_weight': 0.0036815043901656367, 'scale_pos_weight': 11.101495735354064}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:55,312]\u001b[0m Trial 51 pruned. Trial was pruned at iteration 80.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:55,457]\u001b[0m Trial 52 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:55,593]\u001b[0m Trial 53 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:55,952]\u001b[0m Trial 54 finished with value: 0.9326659755677553 and parameters: {'boosting_type': 'gbdt', 'max_depth': 32, 'min_child_weight': 0.005148615159009996, 'scale_pos_weight': 10.013233337332471}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:56,299]\u001b[0m Trial 55 finished with value: 0.932887914967943 and parameters: {'boosting_type': 'gbdt', 'max_depth': 35, 'min_child_weight': 0.0007769349556585813, 'scale_pos_weight': 10.919188105436339}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:56,431]\u001b[0m Trial 56 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:56,573]\u001b[0m Trial 57 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:56,706]\u001b[0m Trial 58 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:56,852]\u001b[0m Trial 59 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:57,207]\u001b[0m Trial 60 finished with value: 0.9326659755677553 and parameters: {'boosting_type': 'gbdt', 'max_depth': 32, 'min_child_weight': 0.0125312356630232, 'scale_pos_weight': 10.012683183019575}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:57,478]\u001b[0m Trial 61 pruned. Trial was pruned at iteration 78.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:57,614]\u001b[0m Trial 62 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:57,960]\u001b[0m Trial 63 finished with value: 0.9337278447512649 and parameters: {'boosting_type': 'gbdt', 'max_depth': 43, 'min_child_weight': 0.0025822330028222257, 'scale_pos_weight': 10.12268219759839}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:58,107]\u001b[0m Trial 64 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:58,243]\u001b[0m Trial 65 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:58,381]\u001b[0m Trial 66 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:58,525]\u001b[0m Trial 67 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:58,655]\u001b[0m Trial 68 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:58,815]\u001b[0m Trial 69 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:58,947]\u001b[0m Trial 70 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:59,098]\u001b[0m Trial 71 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:59,458]\u001b[0m Trial 72 finished with value: 0.9330532260771051 and parameters: {'boosting_type': 'gbdt', 'max_depth': 41, 'min_child_weight': 0.0010124114585983697, 'scale_pos_weight': 10.019540916241073}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:59,662]\u001b[0m Trial 73 pruned. Trial was pruned at iteration 78.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:59,801]\u001b[0m Trial 74 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:24:59,936]\u001b[0m Trial 75 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:00,098]\u001b[0m Trial 76 pruned. Trial was pruned at iteration 52.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:00,243]\u001b[0m Trial 77 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:00,589]\u001b[0m Trial 78 finished with value: 0.9332219468158403 and parameters: {'boosting_type': 'gbdt', 'max_depth': 48, 'min_child_weight': 0.00021426558938719107, 'scale_pos_weight': 10.049409464075314}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:00,725]\u001b[0m Trial 79 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:00,872]\u001b[0m Trial 80 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:01,228]\u001b[0m Trial 81 finished with value: 0.9335821924646988 and parameters: {'boosting_type': 'gbdt', 'max_depth': 40, 'min_child_weight': 8.589036966510786e-05, 'scale_pos_weight': 10.070391130777391}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:01,363]\u001b[0m Trial 82 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:01,516]\u001b[0m Trial 83 pruned. Trial was pruned at iteration 53.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:01,875]\u001b[0m Trial 84 finished with value: 0.9333552874637998 and parameters: {'boosting_type': 'gbdt', 'max_depth': 45, 'min_child_weight': 2.4458989204236287e-05, 'scale_pos_weight': 11.595407882622062}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:02,008]\u001b[0m Trial 85 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:02,147]\u001b[0m Trial 86 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:02,295]\u001b[0m Trial 87 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:02,438]\u001b[0m Trial 88 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:02,570]\u001b[0m Trial 89 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:02,798]\u001b[0m Trial 90 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:03,148]\u001b[0m Trial 91 finished with value: 0.932749962586399 and parameters: {'boosting_type': 'gbdt', 'max_depth': 38, 'min_child_weight': 0.0006070626829421298, 'scale_pos_weight': 10.455425864624141}. Best is trial 15 with value: 0.933907588579396.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:03,295]\u001b[0m Trial 92 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:03,450]\u001b[0m Trial 93 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:03,619]\u001b[0m Trial 94 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:03,760]\u001b[0m Trial 95 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:03,892]\u001b[0m Trial 96 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:04,033]\u001b[0m Trial 97 pruned. Trial was pruned at iteration 50.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:04,171]\u001b[0m Trial 98 pruned. Trial was pruned at iteration 51.\u001b[0m\n",
      "\u001b[32m[I 2024-04-27 23:25:04,340]\u001b[0m Trial 99 pruned. Trial was pruned at iteration 56.\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "study = optuna.create_study(\n",
    "    direction=\"maximize\",\n",
    "    sampler=TPESampler(seed=42),\n",
    "    pruner=MedianPruner(n_warmup_steps=50),\n",
    ")\n",
    "\n",
    "study.optimize(objective, n_trials=100, show_progress_bar=True);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'boosting_type': CategoricalDistribution(choices=('gbdt', 'rf', 'dart')),\n",
       " 'max_depth': IntUniformDistribution(high=63, low=2, step=1),\n",
       " 'min_child_weight': LogUniformDistribution(high=10.0, low=1e-05),\n",
       " 'scale_pos_weight': UniformDistribution(high=30.0, low=10.0)}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To see all info at the best trial use:\n",
    "study.best_trial\n",
    "\n",
    "# To print metric values for all trials:\n",
    "study.best_trial.intermediate_values\n",
    "\n",
    "# To see distributions from which optuna samples parameters:\n",
    "study.best_trial.distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'boosting_type': 'gbdt',\n",
       " 'max_depth': 23,\n",
       " 'min_child_weight': 0.5262050790257533,\n",
       " 'scale_pos_weight': 17.101924194245463}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To simply get the optimized parameters:\n",
    "study.best_trial.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Happy HyperParameter Optimisation!\n",
    "\n",
    "...and remember, that this is useful but not essential in this course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Converting notebook HyperparameterOptimization.ipynb to slides\n",
      "[NbConvertApp] Writing 400283 bytes to HyperparameterOptimization.slides.html\n"
     ]
    }
   ],
   "source": [
    "!jupyter nbconvert --to slides --SlidesExporter.reveal_scroll=True HyperparameterOptimization.ipynb"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "interpreter": {
   "hash": "6aba6d678e50d757e9e1bf8fa8538a573156bd3852124f6ff591560922470a56"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}