Ch6.3and6.4-practical-boosting-with-XGBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains code and comments from Sections 6.3 and 6.4 of the book [Ensemble Methods for Machine Learning](https://www.manning.com/books/ensemble-methods-for-machine-learning). Please see the book for additional details on this topic. This notebook and code are released under the [MIT license](https://github.com/gkunapuli/ensemble-methods-notebooks/blob/master/LICENSE)._\n",
    "\n",
    "## 6.3 XGBoost: A Framework for Newton Boosting\n",
    "[XGboost](https://xgboost.readthedocs.io/en/latest/), or eXtreme Gradient Boosting is a popular open-source gradient boosting framework. It was originally developed by Tianqi Chen and gained widespread popularity when it won the Higgs Boson Kaggle Challenge in 2014. Like LightGBM, XGBoost is a highly-scalable tree boosting package that can be extended to work with millions of data points. XGBoost offers the following advantages:\n",
    "* Parallelization and cache-aware optimization for highly efficient tree learning, to enable support for ver large data sets and lower memory usage (Section 6.3.1)\n",
    "* Newton boosting with regularized learning objectives (Section 6.3.1)\n",
    "* Support for a large number of loss functions for classification, regression and ranking as well as application-specific custom loss functions (Section 6.3.2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3.2 Newton Boosting with XGBoost\n",
    "See the documentation of XGBoost for [installation instructions](https://xgboost.readthedocs.io/en/latest/build.html); it can be installed quickly with pip or built from the source up. \n",
    "\n",
    "We kick off our explorations of XGBoost with the breast cancer data set, which we have used several times in the past as a benchmark data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "Xtrn, Xtst, ytrn, ytst = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a simple Newton boosting classifier using XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>XGBClassifier(base_score=0.5, booster=&#x27;gbtree&#x27;, callbacks=None,\n",
       "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
       "              early_stopping_rounds=None, enable_categorical=False,\n",
       "              eval_metric=None, gamma=0, gpu_id=-1, grow_policy=&#x27;depthwise&#x27;,\n",
       "              importance_type=None, interaction_constraints=&#x27;&#x27;,\n",
       "              learning_rate=0.300000012, max_bin=256, max_cat_to_onehot=4,\n",
       "              max_delta_step=0, max_depth=1, max_leaves=0, min_child_weight=1,\n",
       "              missing=nan, monotone_constraints=&#x27;()&#x27;, n_estimators=20, n_jobs=0,\n",
       "              num_parallel_tree=1, predictor=&#x27;auto&#x27;, random_state=0,\n",
       "              reg_alpha=0, reg_lambda=1, ...)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">XGBClassifier</label><div class=\"sk-toggleable__content\"><pre>XGBClassifier(base_score=0.5, booster=&#x27;gbtree&#x27;, callbacks=None,\n",
       "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
       "              early_stopping_rounds=None, enable_categorical=False,\n",
       "              eval_metric=None, gamma=0, gpu_id=-1, grow_policy=&#x27;depthwise&#x27;,\n",
       "              importance_type=None, interaction_constraints=&#x27;&#x27;,\n",
       "              learning_rate=0.300000012, max_bin=256, max_cat_to_onehot=4,\n",
       "              max_delta_step=0, max_depth=1, max_leaves=0, min_child_weight=1,\n",
       "              missing=nan, monotone_constraints=&#x27;()&#x27;, n_estimators=20, n_jobs=0,\n",
       "              num_parallel_tree=1, predictor=&#x27;auto&#x27;, random_state=0,\n",
       "              reg_alpha=0, reg_lambda=1, ...)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', callbacks=None,\n",
       "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
       "              early_stopping_rounds=None, enable_categorical=False,\n",
       "              eval_metric=None, gamma=0, gpu_id=-1, grow_policy='depthwise',\n",
       "              importance_type=None, interaction_constraints='',\n",
       "              learning_rate=0.300000012, max_bin=256, max_cat_to_onehot=4,\n",
       "              max_delta_step=0, max_depth=1, max_leaves=0, min_child_weight=1,\n",
       "              missing=nan, monotone_constraints='()', n_estimators=20, n_jobs=0,\n",
       "              num_parallel_tree=1, predictor='auto', random_state=0,\n",
       "              reg_alpha=0, reg_lambda=1, ...)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from xgboost import XGBClassifier\n",
    "ens = XGBClassifier(n_estimators=20, max_depth=1, objective='binary:logistic', verbosity=0)\n",
    "ens.fit(Xtrn, ytrn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9649122807017544"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "ypred = ens.predict(Xtst)\n",
    "accuracy_score(ytst, ypred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9649122807017544"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import xgboost as xgb\n",
    "trn = xgb.DMatrix(Xtrn, label=ytrn)\n",
    "tst = xgb.DMatrix(Xtst, label=ytst)\n",
    "\n",
    "params = {'max_depth': 1, 'objective':'binary:logistic'}\n",
    "ens2 = xgb.train(params, trn, num_boost_round=20)\n",
    "\n",
    "ypred_proba = ens2.predict(tst)\n",
    "ypred = (ypred_proba >= 0.5).astype(int)\n",
    "accuracy_score(ytst, ypred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 6.4 XGBoost in Practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.1 Learning Rate\n",
    "Like LightGBM, XGBoost also plays nicely with scikit-learn, and we can combine the relevant functionalities from both packages to perform effective model learning.\n",
    "\n",
    "In the experiment below, we repeat the experiment from the previous chapter to combine ``scikit-learn``'s [``StratifiedKFold``](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html) class to split the training data into several folds of training and validation sets. ``StratifiedKFold`` ensures that we preserve class distributions, that is, the fractions of different classes across the folds.\n",
    "\n",
    "**Listing 6.5**: Cross Validation with XGBoost and scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.02 0.04 0.06 0.08 0.1  0.2  0.4  0.6  0.8  1.   1.2  1.4  1.6  1.8 ]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "# Initialize choices of learning rates, number of cross-validation folds\n",
    "n_folds = 10\n",
    "learning_rates = np.concatenate([np.linspace(0.02, 0.1, num=5),\n",
    "                                 np.linspace(0.2, 1.8, num=9)])\n",
    "n_learning_rate_steps = len(learning_rates)\n",
    "print(learning_rates)\n",
    "\n",
    "# Split the data into training and validation folds\n",
    "splitter = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\n",
    "\n",
    "# Initialize some structures to save training and validation errors\n",
    "trn_err = np.zeros((n_learning_rate_steps, n_folds))\n",
    "val_err = np.zeros((n_learning_rate_steps, n_folds))\n",
    "\n",
    "# Train LightGBM classifier for different learning rates and plot the results\n",
    "for i, rate in enumerate(learning_rates):\n",
    "    for j, (trn, val) in enumerate(splitter.split(X, y)):\n",
    "        gbm = XGBClassifier(n_estimators=10, max_depth=1, learning_rate=rate, verbosity=0, use_label_encoder=False)\n",
    "        gbm.fit(X[trn, :], y[trn])\n",
    "\n",
    "        trn_err[i, j] = (1 - accuracy_score(y[trn], gbm.predict(X[trn, :]))) * 100\n",
    "        val_err[i, j] = (1 - accuracy_score(y[val], gbm.predict(X[val, :]))) * 100\n",
    "    \n",
    "trn_err = np.mean(trn_err, axis=1)\n",
    "val_err = np.mean(val_err, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEYCAYAAABiECzgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/W0lEQVR4nO3deXiU1fXA8e/JQvYFkgBhTdhVENCACwQFV3BvFYsLiODWKi7dbKm/om21WpfWXQQFFHFr64qCKwRlb1AQJAkQwm7CkoSEhCz398c7E0KYJJNkZt6Z5HyeZ55kZt55780wnNzc995zxBiDUkop3wuyuwNKKdVWaQBWSimbaABWSimbaABWSimbaABWSimbhNjdgaZKTEw0KSkpdndDKaXctnbt2gJjTFLdxwMuAKekpLBmzRq7u6GUUm4Tke2uHtcpCKWUsokGYKWUsokGYKWUsokGYKWUsknAXYRTnlNdXc3OnTspKSmxuyvKj4SGhtKxY0diY2Pt7kqr13oD8FODoDCv4WPiesC9633THz9UUFCAiNC/f3+CgvSPIQXGGI4cOcKuXbsANAh7WesNwIV5MKOw4WNmxPmmL37q0KFDpKSkaPBVNUSEyMhIunbtyu7duzUAe5n+z2vDqqqqCA0Ntbsbyg9FRERQUVFhdzdaPQ3AbZyI2N0F5Yf0c+EbGoCVUsomGoBVsxSWVrBgVR5Pf5HNglV5FJb655+rY8eOZe7cuR4/VilPaL0X4ZRXGGN45sscZmVsZVS/JFISoliWU8AjCzcxNb0Xd43p0+I/X6Ojo2u+Ly0tJSwsjODgYABeeuklrr/+erfP9cknn3jlWKU8QQOwapJnvsxh8ca9fHbfOXSKDa95fF9RGVPmrgZg2nl9W9TG4cOHa75PSUlh1qxZnH/++SccV1lZSUhI6/oIu/qZmvpztsb3pbXSKQjltsLSCmZlbGX2pGHHBV+ATrHhzJ40jFkZWyk84p3piK+//ppu3brx6KOP0rlzZyZPnszBgwe59NJLSUpKon379lx66aXs3Lmz5jXnnnsus2bNAmDOnDmMHDmS3/zmN7Rv357U1NTjRr1NOXbbtm2MGjWKmJgYzj//fH71q19xww031Nv3jz76iCFDhhAfH8/ZZ5/N999/X/NcSkoKjz76KKeeeipRUVHk5OQgIsyePZsePXowZswYqqur+etf/0rPnj3p2LEjEydOpLDQWmaZm5t7wvEqMLTeX5NxPRpf5xvXwzd9CRAPfvgDG3cX1fv8T0VlDE/tcELwdeoUG87w1A5c+ewyOtZzzMldYvnzZac0u4979+7lwIEDbN++nerqakpLS5k8eTJvv/02VVVV3Hzzzdx555289957Ll+/cuVKJk2aREFBATNnzmTKlCns2rXL5bRJQ8ded911jBgxgs8//5xVq1Yxbtw4Lr/8cpdtZmZmcvPNN/Phhx+SlpbG66+/zuWXX87mzZsJCwsDYMGCBXz88cckJiayb98+AJYsWcKmTZsICgpizpw5zJkzh6+++qomAN9555289tprNe3UPl4Fhtb7L3XvemsjxoxCGDQe2qccu++8teFdcM1RUWXo1ymmwWP6doyhosp4rQ9BQUE8+OCDhIWFERERQUJCAj//+c+JjIwkJiaG6dOns2TJknpf37NnT2655RaCg4OZNGkSe/bsqQl47h6bl5fH6tWreeihh2jXrh0jR46sN/gCzJw5k9tuu40zzjij5lxhYWGsWLGi5php06bRvXt3IiIiah6bMWMGUVFRREREMH/+fO677z569epFdHQ0jzzyCG+++SaVlZUuj1eBofWOgGuLSoLD+Xb3wu81NjJdsCqPZTkFDR6Td7CUX43pw4Th3vnrIikpifDwY6Pr0tJS7r33Xj799FMOHjwIQHFxMVVVVTUX7mrr3LlzzfeRkZHA8XPO7hxbUFBAhw4dah4D6N69Ozt27HB5nu3btzN37lyeeeaZmseOHj3K7t27j3t9XbUf2717Nz179qy537NnTyorK4/75eHqHMq/tY0AHJ0EFSVwtATaRdndm4A1bmAyjyzcxL6iMpfTEHsLy8jIyufhqwZ5rQ91pwqeeOIJNm/ezMqVK+ncuTPr1q1j6NChGOO9UXhycjIHDhygtLS0JgjXF3zBCozTp09n+vTp9R7jagqk9mNdunRh+/ZjRRXy8vIICQmhU6dONXPeunnCC7ycU6b1TkHUFuUoxVTS8OhNNSwuMpSp6b2YMnc1+4rKjntuX1EZU+etZmp6L+IifLe9ubi4mIiICOLj4zlw4AAPPvig19vs2bMnaWlpzJgxg6NHj7J8+XI+/PDDeo+/5ZZbePHFF1m5ciXGGEpKSvj4448pLi52u80JEybw1FNPsW3bNg4fPswf//hHrr32Wl3t4G3OnDIN3RoL0A1oG/96UR2tryX50L5nw8eqBt01pg8AFzy5hPR+SaQkRJK7v5SMrPyadcC+dM8993DdddeRmJhIly5d+PWvf13vBThPmj9/PjfddBMJCQkMHz6ca6+9lqqqKpfHpqWl8fLLL3PnnXeSnZ1NREQEI0eOZNSoUW63d/PNN7N7925GjRpFWVkZF1100XFTGiowiTf/VPOGtLQ00+SinLv+By+PhglvQv+x3ulYANq0aRMnnXRSs15bWFrBwg17yC8uJykmjHGDkn068vU31157LQMGDPDJCNxXWvL5aDVmxLmXVbGRY0RkrTEmre7jbWQE7JiCOPyTvf1oReIiQ712oS0QrF69mg4dOpCamsrixYt5//33uf/+++3ulgowPgnAIvIKcCnwkzFmoOOxDsBbQAqQC4w3xhz0Sgdq5oB1JYTyjL179/Kzn/2M/fv3061bN1544QWGDh1qd7dUgPHVRbg5wMV1Hrsf+MIY0xf4wnHfO0LDISxWL8Ipj7nsssvYsWMHpaWlZGVlMXnyZLu7pAKQTwKwMWYpcKDOw1cAztRTc4ErvdqJqCQo0SkIpZT/sHMZWidjzB7H93uBTvUdKCK3isgaEVmTn9/MaYSoJJ2CUEr5Fb+4CGeMMSJS73IMY8xMYCZYqyCa1UhUIuzf0rwOKqXaJi/nlLEzAO8TkWRjzB4RSQa8Oz8Q3RHyVjR+nFJKOd3zPTx1CnQ9Da593eOnt3MK4gNgkuP7ScD7Xm0tKglK90O168XySil1gn0/QNEu6HuRV07vkwAsIguA5UB/EdkpIlOAvwMXiEg2cL7jvvdEJQHGCsKqVRMRcnJyALj99tv5y1/+4taxTTV//nwuvPDCZr1WBYjsRdbXvhd45fQ+mYIwxkyo56nzfNE+cPxa4OiOPmu21fFychKAiy++mOHDh/PQQw8d9/j777/Pbbfdxs6dO93OgfDiiy82ux+15ebmkpqaSkVFRU3b119/fZPKI6kAlLUYkgdDTOfGj20Gv7gI5xO1d8N1an5C8DbPmZykIY1dtGjEpEmTmD59Og8++OBxGb5ee+01rr/+ek1AU4uWMPKi0gOwcxWk/8ZrTbSNbGhwbNSrmzH83pVXXsn+/fvJyMioeezgwYN89NFHTJw4kVWrVnHWWWcRHx9PcnIyd955J0ePHnV5rptuuok//elPNff/8Y9/kJycTJcuXXjllVeOO/bjjz9m6NChxMbG0r17d2bMmFHznDNxTnx8PNHR0SxfvrymbJHTt99+y7Bhw4iLi2PYsGF8++23Nc+de+65PPDAA4wYMYKYmBguvPBCCgrq/yxqCSM/sOVLMNXQzzvzv9CmRsCJ1lddC1y/T+6HvR6qEvLqJa4f7zwIxjY83R8REcH48eOZN29eTeB7++23GTBgAIMHD2bt2rU89dRTpKWlsXPnTsaOHcvzzz/PPffc0+B5P/30Ux5//HG++OILUlNTueWWW457Pioqinnz5nHKKaewYcMGLrjgAoYMGcKVV17J0qVLSU1N5dChQzWjxc2bN9e89sCBA1xyySU8/fTTTJgwgXfeeYdLLrmEnJwcEhISAHjjjTf45JNP6N69O2PHjuXxxx/n738/8b3QEkZ+ImsRRCZCl9O81kTbeefD4yEoVHfDBYhJkybx7rvvUlZm5R2eN28ekyZZi2ZOP/10zjzzTEJCQkhJSeG2225rsAyR09tvv83kyZMZOHAgUVFRx41wwRqlDho0iKCgIE499VQmTJjg1nnBGj337duXG2+8kZCQECZMmMCAAQOOyxM8efJk+vXrV/MLZt26dS7PpSWM/EB1FeR8Dn3OBy/+gmo7I2AR3Q3XmEZGpoD787uTP25RV0aOHEliYiLvvfcew4YNY9WqVfznP/8BICsri/vuu481a9ZQWlpKZWUlp59+eqPn3L1793HH1S7xA1YRzvvvv58NGzZw9OhRysvLueaaa9zqb92SQc7z79q1q+Z+3RJH9ZVC0hJGfmDXWjhyAPp5d5VL2xkBgzUNoXPAAWPixInMmzeP119/nYsuuohOnazd6nfccQcDBgwgOzuboqIiHn74YbdKECUnJx9XOigv7/jVHNdddx2XX345O3bsoLCwkNtvv73mvI2V+6lbMsh5/q5du7r1s9bmLGF06NChmltpaSkTJhxbTNSSEkYNnUM5ZC0CCYbe3l2o1bYCcHRHzQkcQCZOnMjnn3/Oyy+/XDP9AFYZotjYWKKjo/nxxx954YUX3Drf+PHjmTNnDhs3bqS0tPSE5OnFxcV06NCB8PBwVq1axRtvvFHzXFJSEkFBQWzdutXluceNG0dWVhZvvPEGlZWVvPXWW2zcuJFLL720yT+3ljDyA9mLoMeZEBHv1WbaVgCOStIRcABJSUnh7LPPpqSk5Liy748//jhvvPEGMTEx3HLLLVx77bVunW/s2LHcc889jBkzhj59+pxw9f/555/n//7v/4iJieGhhx5i/PjxNc9FRkYyffp0RowYQXx8/HHzsQAJCQl89NFHPPHEEyQkJPDYY4/x0UcfkZiY2OSfu3YJo/bt29OnTx/mzJnTpHPcfPPN3HjjjYwaNYrU1FTCw8O1hJG7inZbF6O9tPmitrZRkshp8Z9g1cswfa81J9zGNavkjA82Yij/0GZLEq2dCx9OgzuWQ6eTPXLKtl2SqG7QeDD+xGM0aLhH3yPV2mUvhrju0NH7v3zaRgD2we4tpVQrUFkOW76Cwb/wyV/JbWsOWCmlGrL9G6gogb6+SbKkAVgppZyyP4OQcEgd5ZPmNAC3cYF2EVb5RnV1td1dsEfWIkhJh3aRPmlOA3AbFh4ezv79+zUIqxrGGI4ePcquXbuIioqyuzu+tX8LHNjis+kHaCsX4ZRL3bp1Y+fOnTS70KlqlUJCQoiLi2vWGuaAluVIvu7l7ce1aQBuw0JDQ0lNTbW7G0r5h+zFkNgf2qf4rEmdglBKqfLD1goIH45+oa2MgL1cWlopFeC2fg1VR306/wttJQDX3b31eH/oPRqu8ky9MKVUgMteBGGx0OMsnzbbNqcgkgfDnu/s7oVSyh8YY63/7T0agkN92nTbDcD5P8LRUrt7opSy2971ULwH+nqv9lt92mYA7jLEKra37we7e6KUslu2Y/lZn/N93nTbDMDJg62ve9bZ2g2llB/IWgxdhkJMp8aP9bC2GYBju1rVTjUAK9W2leyHnattmX6AtrIKorbauYEzX7dudWluYKXahi1fAMbn63+d2l4A1tzASimnrEVWqbLkobY03zanIJRSqqoScj6HPhdAkD2hUAOwUqpt2rUGyg7ZNv0AGoCVUm1V1iKQYOg12rYuaABWSrVN2YutrccR8bZ1QQOwUqrtKdwF+zbYOv0AGoCVUm1R9mLrq03rf500ACul2p7sxdZ6/6T+tnaj7a0D1tzASrVtFWVW/t8h14GIrV2xPQCLyL3AVMAA64HJxpgyrzVYd4fbxg/g7Rvh5kXQ40yvNauU8hPbv4GKUtunH8DmKQgR6QpMA9KMMQOBYOAXPu1EykhArN+ISqnWL3sxhIRDarrdPbF/BIzVhwgRqQAigd0+a7l2XoiMJ+HrRxo+XnNEKBXYjLHW/6aOgtAIu3tjbwA2xuwSkceBPOAIsNgYs7jucSJyK3ArQI8eHpyfrZ0XYkac5ohQqrXbnwMHt8FZv7K7J4D9UxDtgSuAVKALECUiN9Q9zhgz0xiTZoxJS0pK8nU3lVKthXP5WT/753/B/mVo5wPbjDH5xpgK4D/A2Tb3SSnVWmUtgqSTIN4/VjrZHYDzgDNFJFJEBDgP2GRzn5RSrVF5MWz/FvpeYHdPatg9B7xSRN4F/gdUApnATDv7pJRqpbZ8BdUVzZp+KCytYOGGPeQXl5MUE8a4gcnERba8grLdI2CMMX82xgwwxgw0xtxojCm3u09KqVYoezGExUH3M9x+iTGGp7/IJv2xL/kmp4CjldUsyykg/bEvefqLbIwxLeqSPyxDU0op7zIGsj+DPmMg2P2R6zNf5rB4414+u+8cOsWG1zy+r6iMKXNXAzDtvL7N7pbtI2CllPK6Pd/B4b3Q1/3sZ4WlFczK2MrsScOOC74AnWLDmT1pGLMytlJ4pKLZ3WrbI+DaeSGCwzRHhFKtVfZiQKzyQ25auGEP6f2STgi+Tp1iw0nvl8TC9XuYMLx5saFtB2B3drUd2AbPDYdB4+HK57zfJ6WU52Uvhq6nQbT7+wjyi8tJTYhq8JiUhEjyi5t/2UqnIBrTIRXOuA3WzYc939vdG6VUU5UUwM41TZp+AEiKCWPb/pIGj8ndX0pSTFizu6YB2B3pv4GI9rB4ujWZr5QKHDmfA6bJAXjcwGQysvLZV+Q6OePewjIysvIZNyi52V3TAOyOiHg49w+wbSlkfWp3b5RSTZG1CKI6QvKQJr0sLjKUqem9mDJ39QlBeF9RGVPnrWZqei/iIpq/HrhtzwE3RdpkWDUTFj8Afc5v0lIWpZRNqiphyxcw4DIIavp4864xfQA45x9fMbJPIv07x5C7v5SMrHympveqeb65dATsruBQuPAvsD8b1rxqd2+UUu7YuQrKCpu9/VhEuP2c3lRVG8orqggLCWZkn0Qyfj+Gaef1RVpYUUNHwE3R72JISbfyBp863tZy1kopN2QtgqAQ6D262afYtKeIiirDhDN6tmi+1xUdATeFCFz0NzhyEDIet7s3SqnGZC+GHmdBePNzeWfmHQRgaI94D3XqGA3ATZU82Crmt/Ila42wUso/HdoBP21sce7fzB2H6BQbRnKc5ytoaABujjF/sv6s+XyG3T1RStXHmXy9icvP6lq34xBDu7f3QIdOpAG4OWK7wNnTYON7kLfS7t4opVzJXgzxPSGxX7NPsf9wOdv3l3pl+gE0ADffiGkQ3RkW/VE3ZyjlbyqOwNYl1vRDC1YqfLfzEABDusd7pl91NBqAReQ0EXlARN4Vkc8dXx8QkTSv9ChQtIuC8x6AXWtgw7/t7o1Sqrbcb6DyCPRt4fxv3iGCg4RB3bxTkLfeACwiF4nIGmAB0B34BnjT8bU7MF9E1ojIxV7pmYcUllawYFUeT3+RzYJVeRSWNj913AkGT4DOg+DzB6HC9XZFpZQNshdBSASkjGjRaTLzDjGgcwyR7byzYrehEfAtwB3GmP7GmFuNMU8ZY2Y5vt5qjOkP3AFM9UrPWsjbmewBCAqGC/9mlbdf+ULLz6eUajljrPW/vc6B0OavXKiuNny345DXph+ggY0YxpirG3uxMWY10OhxdvB2Jvsavc6xNmgsfQKG3NCkdHdKKS8oyIZD22HE3S06zZb8wxSXVzK0h3dWQEAzLsKJyIUi8hsR+Zm0dB+el/gik/1xLvgLVJRaO+SUUvbKXmR9beHys8y8Q4D3LsBBE7cii8hDwJnAGuAy4CrgRi/0q0V8kcn+OK//HEwVrJlt3VyJ6+FeAnilVMtkLYKOJ0N89xadJnPHIWLDQ+iV2HBS9pZoMACLyJXGmPdqPTTKGHOu47lQ4Cev9awFfJHJ/jiFeTCjsOFjGit3pJRqubJCyFsOZ93Z4lNl5h1kSI/2BAV57w/9xqYgxorIByLS03F/k4i8KCJTgVeBVV7rWQv4IpO9UsoPbf0aqitbvP24pLySrH3FXp1+gEYCsDHmNuAR4E0ReQD4HfAdcJrj6zVe7V0z+SKTvVLKD2UtthLvdBveotN8v7OQauOdBDy1NXoRzhizHBgBFANfAVuMMb80xvzDGFPk1d41ky8y2Sul/Ex1tbX9uPd5ENyydbuZO6wMaEO6xXugY/VrbA5YsC609QJ+AK4AnnJMQdxtjNnj1d61gDNT/QVPLuGs3on0Soxi+4ESlmUXeCSTvVLKz+xZByU/tXj6AawVEKmJUbSPatfyfjWgsRHwPOA+IAGYDvzSGDMeeAVYKCK/9mrvWkBEmHZeXzJ+N4a+HaN5YckW4iPaeSyTfbP8+xYoyPF9u0q1BdmfAWKVDGsBY4wjA1q8R7rVkMYC8CXAaGPMH4DzHfcxxnwKnAXEerd7LRcXGcqdY/oQHCR0iGpn77TDjx/Bc8PgP7fB/i329UOp1ih7EXQ9HaISW3SaXYeOkF9c7vX5X2h8HfAq4EER+QorANfkXjTGlAF/9mLfPCY8NJjeSVH8uNfmKeu7v4dv/gmrZ8P6d+DUa+Gc30KHXvb2q7meGmQtwWuIrn9WvnA4H3b9D0b/scWnWrfjEABDvJQDuLbGAvB44DaseeANwP95vUdeMqBzLGu3H/TOyeN6NL7ON66HtU35or9ZuYS/+Ze1aeP7t2DIBBj1W2if4p3+eYuuf1b+IuczwLR49xtY879hIUEMSI5peb8a0WAAdqxy+IfXe+EDJyXH8sF3uyk8UuH5aYimjvBiOsHFD1s5hZc9ZVVZ/u5Nq9RR+m+gfc/Gz6GUOiZ7sZWfO3lwi0+VmXeQQV3jCA32frr0htJRPikinRt6sYh0FpEnPd8tz3P+Nvtxjx+tnIvpDGMfhbvXQdrNVhB+5nT48B6rnpVSqnFVFZDzJfQ9v0XJ1wGOVlazYXeRT+Z/oeER8GZglYhsApY47hcDMUA/4FygP/BXL/fRI05Otq4XbtpTxBm9EmzuTR2xXWDcP2DEPbDsSVg7FzJfh9MmQvqvIa6r3T20lB+GvBWQmwG5y+zujVKWHSuhvLDFydfBig9HK6u9mgGttobSUb4kIq9grf0dC1wJxAMHge+BF4EPjTGV3u9my3WMCSMuIoSP1++hqKySpJgwxg1MJi7SjzZjxHWFS56wAnHGE/C/eZD5Gpx+E4y8D2J9vHOv/LD14XYG3F3/s5IOBYVaV5vd9e2z1gVHTdWpvCFrkfWZ7D26xadylqD39hZkp8bmgCuAdx23gGWM4ZkvczhaaYiLCK1Jzv7Iwk01mzL8KrNmfHe47J+Qfh8sfRzWvGKNitMmw8h7rakLbzhaYgXcbY6Au/t/1r76oBAr4I68B1LSoftwqySTuxfYFk+Hz/9sjVCGXGctlA/2o198KrBlfwY9z4awll80W1dTgt51JkVP806dDT/jTM7+9W/P9W5ydk+L7wGXP+0IxP+AVS/D2jmQNsUKhtEdW3b+o6WOEe4ya5S7a+2xgNvlNGu1RspI6HGmFXCb65crYN18+O4t2PwxRCZaI+Kh10OnU1r2M6i27VAe5G+CoTd45HSZjhL0vhqQ2R6ARSQemAUMBAxwsyP/hEc4k7PXrYwBx5KzX/DkEiadneK/uSHap8AVz1nzwUsft8ofrXkFhk2BH96Dop0Nv965FrfiSK2Auwx2roHqCpBg6DIUzr7LCrjdz4SwaM/1v+NJcOFf4bwZkPM5rHsdVs2EFc9B8hAYcj0MuhoiO3iuTW/S9c/+I8uRfN0D24+dJeg9kiPcTbYHYOBfwKfGmKtFpB0Q6cmT+zw5uzd16AVXPm8F4iWPwYrnwVS7txb3lbFWBeeqoyBBVsA965eQMgp6nNG8P9/cXf/sFBwC/S+2biX7Yf3bkDkfPvmtNU3Rf5w1kuk9xqq35690/bP/yF4M7VMhoeW5XZwl6H2xBdmp0QAsIkFYKx6WGWOOerJxEYkDRgE3ATjO79E2fJ6c3RcSesPPXoJRv4Fn09x7TWUZnHG7NYfb40wI98Au8paM8KIS4Mw7rNue760piu/fho3vQUyyY4riBkj0w6kh5R8qjsC2pXDapBYvPwPvl6B3pdEAbIypFpH3jTHe2BaSCuQDr4rIYGAtVpa147Kpi8itwK0APXo0bZSaFBPGspyCBo/J3V/KyD4t2z9ui6YEp1u/8l4/Wir5VOt2wUOQ9ak1Kv72GWvbdrfh1lzxKT/zzC+NpiortOYZD+VZa7MPNTL1oHxnW4Y1sOjX8t1vYAXg/p28V4LeFXdbWioiZxpjVnih/dOAu4wxK0XkX8D9wAO1DzLGzARmAqSlpTWpnvy4gck8snAT+4rKXE5DOJOzP3zVoOb+DMpTQsLg5CusW/Fea5t25nz48G745H446TIrGKeMgn8N9sw87JFDxwJs4Y5awXa79bWszlRDqEdnyAKHP857Zy+y/j16jmzxqZwl6C8f0sUDHXOfuwF4O/CJiLwP7MC6WAaAMaYl+SF2AjuNMc4kP+9iBWCPqZ2cvXaV5MLSCt5cncfsZdsYntqh1k+k/EJMZ6us+NnTrPXH616H9f+25o3jerg/D3vk4PGj17q38roBNspafRLfw7oYGd/92P34nhCZAA/Ge+3H9lv+Nu9tjFX9ote5ENryJWO+KEHvirsBOAJ4z/F9t1qPtyhsGWP2isgOEelvjNkMnAdsbMk5XamdnH1k30QOlBzlux2FnN07gWtO70bu/lLSH/vSP9cEt3Ui0O1063bRw/Djx9YuwcZGY06Pphx/PzTKyrUR3wN6ngVxdQNsB4/MJwJQVdniygyqHvmbrc9A+r0eOZ0vStC74tanwxgz2Yt9uAuY71gBsRXweFvO5OyTzkrht+9+x4GSo4G3JlhBaIS1XG3Q1e6Pti786/EBNqK95wJsY2adZ63j9kCCGFVHtmP5mQeyn4FvStC74vavZxHpC0wAugK7gAXGmOyWdsAYsw5w81J+y63Yuj+w1wSrpjn7LvvaLtoNM0dby/3O/UPLNrPYqbIJK4TenmittEkZCUkDvPfLLmsxdBoIcd0aP9YNmXkHGdw93qsl6F1xKwCLyGXAfOAjrPng/sAaEbnRGPOBF/vnUa1qTbBTU9fiqpZz9z2/fSl89mdrRcfGD+DSJ1tcLsenqipg3RvWLkx37VwLG9+3vo9MtAJxykhIHQWJ/TwTkI8cgrzl1jUCD3CWoL/wFC9t8W+AuyPgh4ErjDE1a5lE5FzgWSBgAnCrXBOsu618rynv+eVPW2uaP7wbXv85DBpvzWX7c2KiqkprBcqSR63VIF1Pt1aIuOPeDXAw99huy9wMa203QFTHYwE5Jd1aRtmcgLz1KysplIemH3xVgt4VdwNwNyCjzmPLOP6CnN+rb01wYWkFCzfsIb+4nKVZBVw11E/SP6rWIWUE3PENZDxpZbnL+cwKwoMn+G4+2h3VVbDh3/D13+HAFmvuetw/rEDn7soPEeiQat1Ou9FarXBwmxWMt2VYAfmH/1jHRneqFZBHWRuMXL0f9S2Be/XiY9+3YAmcr0rQu+JuAF4H/Bp4tNZj9zkeDxh11wQ7s6TNytjKqH5J9EyIJCkmjH9+nsXh8kpdEaE8JyQMRv8BTrnKGg2/dwd8twAu/acVeOxUXW2NUr/+OxRstuZWr50PAy5p+S8IEWsLfYdeVn5rY+DA1mMpTrdlWEEfrIoWKSMhNd0aIXfoZb3ey0vg1vmoBL0r7gbgO4APReRurHXA3YFS4DJvdcwb6q4Jfmv1DhZv3HvCRTldEREAAnXuu+MAmPwJrH0VPp8BL5wN5/zeuljo6xSdxliVur96BH76wbpods0cOOkKCKpTLMdT77eI9QsnobeV59oYq0K4MyDnZsAGR/bbmC5WQPYiYwyZOw6RbtNOWHdzQXQBhgJDHN/vBlY68gUHFOea4POf+JrKak5Yjga6IiIgBPLcd1CQlcmu/zj45HfwxYOw/l1rvribDxYEGWNlEfvqb7D3eyuRzc9nW6Pz+pIgeev9FoHEPtYtbbIjIOdYOR5yl8HWr73TroOzBP0QG+Z/oem5IAK+Do1zTXB0WAgrtx04IfjWng/umRDFv9fu5OaRqTb1VrVqsclw7WvW5pKPfwOzzofht8J5D3gkufgJjIGcL6zAu/t/VprTK1+EQdf4z4YREeviXGJf65eUMV7deegsQT/UByXoXbE7F4RtDpdX0rfjsZy3deeDUxKiSI4L54nFm3U+WHnXgEusOc8v/2LlSf7xI6s0Vf+xnjm/MbBtCXz1sJUPOq4HXP6MdRHQ3yuTePn/nC9L0Ltidy4I29RdEeGsmqHzwcoW4bHWioNB4+HDabDgF1ZSorGPtawEVe431oh3+zfWnOolT8LQGyHE9xec/JEvS9C74m6rzlwQBmvpWXfHLaCWodU2bmAyGVn57Csqq6maUTtZj5NzPnhWxlYKjwTclLcKNN2HwW1L4bz/g82fwrPDreon1dVNO8+OVTD3cpgzzppTHfsYTMu0/qzX4Av4vgS9K+5chAvGGvX+zRgTQDsUGlZ7RcRlp3ZpfTvkVOAKDrWqnpx8JXx0D3x0r5Ws/uB2KN7d8Gtjkq06ezmfQ1SStd447WYrj4Y6jq9L0LvizkW4KhG5A5jh/e74lnNFxJOfZTGlkQttAbdDTgW+hN4w8QNrO/Di6VZqTXfWw1aWw/kPwvBbAjf/RG1eWnLo6xL0rrg7B/wacDvwvBf74nN1V0Q0JGCrZqjAJmIloe97ITzuZt2ze773zioKu3hpCZyvS9C74u4c8HDgXyKSKyIZIrLUefNm53zl56d1Y/mWAvYVlbl83lk1Y9ygZB/3TCmHpuSOaE3B14sydxxiSPd4W1c3uTsCftlxa5Xqq5oB1iqI62atoHuHSCLb+XGlXqWU2w6UHPV5CXpX3E3IPtfbHbFb7aoZ6f2SSEmIJHd/KRlZ+QzpHs/S7AJue20tz19/GuGhGoiVCmTrHAl4fFmC3pUGpyBE5Ok696fUuf9vb3TKDs754IzfjWFkn0TCQoIZ2SeRjN+PYd6UM/jbVQP5avNPTHxlFUVluhxNqUBmRwl6VxobAd8ETKt1/x/A7Fr3L/B0h+wWFxnq8s+S68/oSUx4KPe9tY7rXl7B3MnDSYgOs6GHSqmWWrfD9yXoXWnsIlzd2ek2vRf38sFdeHliGtn7DnPNS8vZdeiI3V1SSjVRdbVhXd4hWzdgODUW/utWPW7zxdtHD+jIa1POYMqc1Vzzwre8NvUMeidFN/5CpVoqUFNw+hlnCXo71/86NRaAQ0RkNMdGvnXvt8mrUcNTO7Dg1jOZ9Moqxr+4nLk3D2dg17jjMqklxYQxbmAycZF+nuxEBY5ATsHpRzKdGdBs3AHn1FgA/gl4pdb9/XXu/+TxHgWIgV3jeOf2s7hx9ip+8dJyLhnchU/W76nJpLYsp4BHFm5ianovzaSmlB/JzLOnBL0rDQZgY0yKj/oRkHolRfPO7WdxxbPLWJN7QDOpKRUA7CpB74o9Odhakah2IZRXVvPGLWdqJjWl/JyzBL0/TD+ABuAWW7hhj9uZ1JRS9qopQe8HF+BAA3CL5ReXk5rQ8FySZlJTyj/UlKDXANw6JMWEsW1/SYPH5O4vJSlGN20oZTc7S9C7ogG4hWpX1nBFM6kp5R+cJej9ZfQLGoBbrHYmtbpBeF9RGTfMWskNZ/bU0vZK2cxZgt4fdsA5+Ukt6sBWXya1JZvzqaiqZt2Og1RUVdtW+E8pZX8Jelc0InhAfZnUvrl/DH+7ahDfbjnAn/67AWPa/E5upWxjdwl6V3QE7EGuMqldfXo3cgtKeParHFKTorj9nN429U6ptm3djkO2lqB3xX960ordd0E/Lj01mb9/8iOf6HpgpXzuaGU163cV+tX8L2gA9omgIOHxawYztEc897y1rmYuSinlG84S9EP8aP4XNAD7THhoMC9PTCMpJoypc9ew82Cp3V1Sqs2ouQCnI+ATiUiwiGSKyEd298WbEqPDePWmYZRXVjFlzhqKtbSRUj6RmXfQ9hL0rvhFAAbuBjbZ3Qlf6NsphheuP50t+Ye5841MKquq7e6SUq2eP5Sgd8X2ACwi3YBLgFl298VXRvZN5C9XDmRJVj4zPvxBl6cp5UXOEvT+kgGtNn9YhvZP4HeA/yzO84EJw3uQW1DCS0u3kpoYzZSRqXZ3SalWaZ2fJeCpzdYRsIhcCvxkjFnbyHG3isgaEVmTn5/vo9553+8vHsDFp3Tmrx9v5LON++zujlKtUmbeIYIETrW5BL0rdk9BjAAuF5Fc4E1gjIi8XvcgY8xMY0yaMSYtKSnJ1330mqAg4alrhzCoaxzTFmSyYVeh3V1SqtVZt+MQAzrH2l6C3hVbA7Ax5g/GmG6O0ke/AL40xtxgZ598LaJdMLMmptE+MpQpc1ezp1BL3SvlKc4S9EP8bPmZk90jYAV0jA1n9k3DKCm3lqeVlFfa3SWlWgVnCXp/qYBRl98EYGPM18aYS+3uh11OSo7lmeuG8uPeIu5+M5Oqal0ZoVRL+VMJelf8JgArGN2/Iw9efgqfb/qJv3680e7uKBXwMvMOEeMnJehd8b9Z6TbuxrNS2FpQwqvf5JKaGMXEs1Ls7pJSAWudYwOGP5Sgd0VHwH7oT5eczHkDOjLjgx/4avNPdndHqYBUUl7J5r1Ffjv9ABqA/VJwkPD0hKEM6BzLnfP/x6Y9RXZ3SamA428l6F3RAOynosJCmH1TGtHhIUyZs5qf6in6qZRyzZkBzR93wDlpAPZjyXERzJ40jIOlFUydt4bSo7o8TSl3ZeYd9KsS9K5oAPZzA7vG8fSEoazfVci9b62jWpenKdUofyxB74oG4ABwwcmd+NMlJ7Poh308+umPdndHKb+3u7DM70rQu6LL0ALEzSNS2FZwmJeWbiUlMeqE4p9KqWMy8/w3A1ptGoADhIgw47JT2HHgCH96bwPd20cysm+i3d1Syi/VlKDvHGt3VxqkUxABJCQ4iGevG0qfpGjumL+W7H3FdndJKb/kLEHfLsS/Q5x/906dICY8lNk3pREWEszkOaspOFxud5eU8ivOEvT+Pv0AGoADUrf2kcyelEbB4XJumbeGsooqu7uklN9wlqD35x1wThqAA9Tg7vE8NX4ImXmH+M0733Gw5CgLVuXx9BfZLFiVR2GpVlxWbZO/lqB3RQNwABs7KJnfX9yfTzfsZcSjX/JNTgFHK6tZllNA+mNf8vQX2VrwU7U5mXkH6RjjfyXoXdFVEAHuaGU1vZKieG3KGXSKPfaB21dUxpS5qwGYdl5fu7qnlM9l7jjE0B7+V4LeFR0BB7DC0gpmL9t2QvAF6BQbzuxJw5iVsZXCIzododoGfy5B74oG4AC2cMMe0vslnRB8nTrFhpPeL4mF6/f4uGdK2cOfS9C7ogE4gOUXl5Oa0HCm/5SESPKLdamaahvW+XEJelc0AAewpJgwtu0vafCY3IJSkmLCfNQjpeyVueMQ/f20BL0rGoAD2LiByWRk5bOvnlzBewvL+OLHfQzpHhijAaVawlmCPhCWnzlpAA5gcZGhTE3vxZS5q08IwvuKypg8ZxXBItwwaxWrth2wqZdK+cbWAv8uQe9KYIzTVb3uGtMHgAueXEJ6vyRSEiLJ3V9KRlY+U9N7MW5QZ26dt5brXl7BQ1cM5LozNIuaap3+l3cICIwNGE4agAOciDDtvL5MOiuFhRv2kF9czsg+iTx81SDiIkIB+O+vRjBtQSZ//O96ftxbxAOXnkxosP7xo1qXYyXoo+3uits0ALcScZGh9eYIjosI5ZWbhvHopz8yc+lWsvcd5rnrT6ODH5dqUaqp/L0EvSs6DGojgoOEP447iSeuGczavINc8dwyftyr1ZZV61BTgj6A5n9BA3Cb8/PTu/HWrWdSXlHNz57/lkU/7LW7S0q1WE0J+gDZAeekAbgNGtqjPR/eNZK+HaO57bW1PKNJe1SAC4QS9K5oAG6jOsWG89ZtZ3HV0K488VkWd76RqWXvVcDKzDtISkKkX5egd0UDcBsWHhrMk+MH88dxA/hkwx6ufmE5Ow+W2t0tpZrEWYI+0KYfQANwmyci3DqqN7NvGsaOA6Vc8ew3rM7VTRsqcARKCXpXNAArAEb378h/fzWCuIhQrnt5BQtW5dndJaXcEigl6F3RAKxq9OkYzX9/NYKzeifyh/+s58/vb6CiqtrubinVoHUBUoLeFQ3A6jhxEaG8etMwbklPZe7y7UycvYqDJUft7pZS9crccYiBAVCC3pXA67HyuuAgYfolJ9ds2rj8uWVs3ltsd7eUOoGzBH2gbcBw0gCs6nX8po1vWKybNpSf+XFv4JSgd8XWACwi3UXkKxHZKCI/iMjddvZHnci5aaNPx2hufW0tz36pmzaU/8h0ZEAbEoArIMD+EXAl8GtjzMnAmcCvRORkm/uk6qi9aePxxVncuUA3bSj/4CxB3yUAStC7Yms2NGPMHmCP4/tiEdkEdAU22tkvdSLnpo0BnWP4+6c/kltQwsyJaXSNj6CwtKImFWZSTBjjBiYTFxlqd5dVG7AugErQu+I36ShFJAUYCqx08dytwK0APXpoQnG7iAi3ndObfp1imLYgk8ufyeCCkzuzcP0eRvVLIiUhimU5BTyycBNT03tx15g+Xv+PocG/7TpQcpTc/aVcOyxwY4JfBGARiQb+DdxjjDkhR6IxZiYwEyAtLU0nIG02eoC1aWPCzOWszj3AZ/edQ6fYY38C7isqY8rc1QBMO6+vV/pgjOGZL3OYlbHVtuCv7OUsQR+IO+CcbA/AIhKKFXznG2P+Y3d/lHuSosMor6zmjVvOPC74gjVnPHvSMC54cgmTzk6pqczhSc98mcPijXttCf7KPwRaCXpXbA3AYg1RZgObjDFP2tkX1TQLN+whvV/SCcHXqVNsOMNTOzBh5nIGJMcSGx5KdFgI0eEhxISHEB0WYj1W636M45jgRioaFJZWMCtj6wnB19mut4O/8g+BVoLeFbt7PgK4EVgvIuscj/3RGLPQvi4pd+QXl5OaENXgMX07xrBuxyGKyg5QXFbJ4fJKqqobn0GKahdMdK2gHFMnSOcWlHBW74QGg396vyQWrt9Tb5kmFdicJegvG9LF7q60iN2rIJYBOlEXgJJiwliWU9DgMXkHS/n1hf1rgqAxhiMVVRwuq6TIEZCLyyo4XFZJcXmlFaTLHI857hc7jtlbWFYTxA+XV/LLc3s32HbPDpFk7yvGGKNzwa2QswR9ICbgqc3uEbAKUOMGJvPIwk3sKypzORLdW1hGRlY+D181qOYxESGyXQiR7ULo2IK8KfNXbufbnP0NHrNpTzFfbf6Jd9buZGCXOAZ2jWVg1zgGdo0jNSEqoAo3qhM5S9CfFsAX4EADsGqmuMhQpqb3Ysrc1cyeNOyEC2FT561manovr8zBXjqoC49+8mODwX/N9gP8+dKTyck/zIbdRcxdvp2jlVZmt6h2wZzSxQrGg7rFMrBLHL2Sohude67LriVwuvQuMEvQu6IBWDXbXWP6AHDBk0tI75dESkIkuftLycjKr1kK5g3uBP9b0nsxeWRqzeMVVdVk7zvMht2FbNhl3d5YtZ2yb6ygHBEazMldYhnkGCUP7BpLn6RoQoJP3Cxq1xI4XXp3TCCWoHdFAm1ff1pamlmzZo3d3VC1nDAiG5Ts9dUHtYNRfcG/sWBUWVXN1oIS1u8sZP2uQn7YXcgPu4soPVoFQFhIECclW0F5UNc4TukaS79OMbzw9RYWb9zrMvhPmbuaC0/u7JUlcE9/kW1Lu/6mpLySQTMWcefoPtx3YX+7u+MWEVlrjEk74XENwCqQeTr4V1UbthUcZsOuItY7Rso/7C7icLmV+yI0SAgOFpb8drTL6Y99RWVc8OQS3r9zBO0j2xEUJIQECUFifQ0OkmaNUgtLK0h/7EuXS+9qt5vx+zGtfundiq37+cXMFbx60zBGD+hod3fcUl8A1ikIFdDiIkM9utQsOEjo0zGGPh1juHJoV8Ba8pS7v4QNu4t4Z80OwkKCGlwCd3rPDox+fEm9bQQJhAQFERRkfQ12BObgICFYjn0fUuvxwiMVDEvt0PDSu75tY+mdMwPa4ABfAQEagJVqVFCQ0Cspml5J0eQWlNRczKvPSckxiBjS+yZRVW2oqjZUVhuqnV+N9bWqzs16rJqqaqyvxvpaWWXI+ekw/TvFNNhujw6RPPzxJj7dsJd+naLp1ymGfp1i6Nsp2mObFey8AOhs+921O0iIakdwK5jv1gCsVBO4s/55+4FSLji5s0dHogtW5TXa7paCw/TuGEV+cTnLt+4/7hdF9w4R9OsYQ99OMfTvHE3fjjH06RhNeGiwW+3beQGwbtsXndKZ7J8Ok/7YlwF/8VEDsFJN0Jz1z75qd8WW/TVzwFXVhu37S8jad5jsfcVs3ldM9r7DLM3Op6LKuu4TJNAzIYq+HaPp39kKzv06RdMrMfqE+mp25t5ozXk/9CKcUk1k12oET7RbUVVNboEVmK2gXEzWvmJy95fWbBMPCRJSEqPo75i+6BYfwYMfbeRzGy4AtpaLj3oRTikPsWv9syfaDQ0Oom8na7R7Cck1j5dXVrE1v4QsR0DO2neYH3YXsnDDHoyB0f07NngB8MxeCdzzZiYDu3o2M9mGXYWc2av15v3QAKxUE4kI087ry6SzUmouSI3sk8jDVw3y6ijMm+2GhQRzUnIsJyUfv0f8yNEq/vrxRuIbudDWOymaF5dsYUlWfov6UZcxcEcjeT9SEiLJLy73aLu+ogFYqWby9BI4f2w3ol0wA7vGuZV46eGfDfJ4v9y5+Ji7v5SRfRI92q6v2F2UUynl58YNTCYjK599RWUun3deeBw3KNnl84Hati9oAFZKNah27o26gdDbiZfsbNsXdApCKdUouy482t22t+kyNKWU2+xIvOQPbbeULkNTSrWYXRce7W7bW3QOWCmlbKIBWCmlbKIBWCmlbKIBWCmlbBJwqyBEJB/Y7sahiUDDW2i8R9vWtltzu9p20/U0xiTVfTDgArC7RGSNq2Uf2ra23Vrabos/c2trW6cglFLKJhqAlVLKJq05AM/UtrXtVt52W/yZW1XbrXYOWCml/F1rHgErpZRf0wCslFI2CfgALCIXi8hmEckRkftdPB8mIm85nl8pIik+bPs+EdkoIt+LyBci0tNXbdc67uciYkTEY0tn3GlbRMY7fvYfROQNX7UtIj1E5CsRyXS87+M81O4rIvKTiGyo53kRkacd/fpeRE7zRLtutn29o831IvKtiAz2Rbu1jhsmIpUicrUn2nW3bRE5V0TWOT5jS3zVtojEiciHIvKdo+3JzW7MGBOwNyAY2AL0AtoB3wEn1znml8CLju9/Abzlw7ZHA5GO7+/wZduO42KApcAKIM2HP3dfIBNo77jf0YdtzwTucHx/MpDrobZHAacBG+p5fhzwCSDAmcBKD37OG2v77Frv9VhPtd1Yu7X+Tb4EFgJX+/Bnjgc2Aj08+Rlzs+0/Ao86vk8CDgDtmtNWoI+AhwM5xpitxpijwJvAFXWOuQKY6/j+XeA8ERFftG2M+coYU+q4uwLo5oF23Wrb4S/Ao4Drei7ea/sW4DljzEEAY8xPPmzbAM7KknHAbk80bIxZivUfrT5XAPOMZQUQLyIeqZPTWNvGmG+d7zUe/Jy58TMD3AX8G/DUv7G7bV8H/McYk+c43mPtu9G2AWIccSTacWxlc9oK9ADcFdhR6/5Ox2MujzHGVAKFQIKP2q5tCtYIyRMabdvxJ3B3Y8zHHmrT7baBfkA/EflGRFaIyMU+bHsGcIOI7MQald3lobYb09TPg7d48nPWIBHpClwFvOCL9uroB7QXka9FZK2ITPRh288CJ2H9cl8P3G2MqW7OiTQhuw+IyA1AGnCOj9oLAp4EbvJFey6EYE1DnIs1GlsqIoOMMYd80PYEYI4x5gkROQt4TUQGNvc/SCARkdFYAXikj5r8J/B7Y0y1Z/6obJIQ4HTgPCACWC4iK4wxWT5o+yJgHTAG6A18JiIZxpiipp4o0EfAu4Dute53czzm8hgRCcH6s3S/j9pGRM4HpgOXG2PKPdCuO23HAAOBr0UkF2tO8gMPXYhz5+feCXxgjKkwxmwDsrACsi/angK8DWCMWQ6EYyVQ8Ta3Pg/eIiKnArOAK4wxnvh8uyMNeNPxGbsaeF5ErvRR2zuBRcaYEmNMAda1Do9cfHTDZKzpD2OMyQG2AQOac6JAD8Crgb4ikioi7bAusn1Q55gPgEmO768GvjSO2XNvty0iQ4GXsIKvJ+fIGmzbGFNojEk0xqQYY1Kw5gUvN8Z4opieO+/5e1ijX0QkEevPxa0+ajsPa1SEiJyEFYDzPdB2Yz4AJjpWQ5wJFBpj9vigXUSkB/Af4EYfjQABMMak1vqMvQv80hjzno+afx8YKSIhIhIJnAFs8lHbtT9jnYD+NPfz7akrh3bdsK4+Z2FdHZ/ueOwhrIAD1n/Ad4AcYBXQy4dtfw7sw/pzZR3WqNAnbdc59ms8tArCzZ9bsKZANmLNkf3Ch22fDHyDtUJiHXChh9pdAOwBKrBGX1OA24Hba/3Mzzn6td7D73djbc8CDtb6nK3xRbt1jp2DZ1dBNNo28FvHZ2wDcI8P3+8uwGLHv/MG4IbmtqVbkZVSyiaBPgWhlFIBSwOwUkrZRAOwUkrZRAOwUkrZRAOwUkrZRAOwalNEJF1ENtvdD6VAA7DyIRHJdewMtI0xJsMY09/OPjg50inutLsfyj4agFWrIiLBdvcBavID6/8v1SD9gCjbiUiQiNwvIltEZL+IvC0iHWo9/46I7BWRQhFZKiKn1Hpujoi8ICILRaQEGO0Yaf/GkaS8UKyE/OGO448bdTZ0rOP534nIHhHZLSJTxUpu36een+NrEfmbiHwDlAK9RGSyiGwSkWIR2SoitzmOjcLKWtZFRA47bl0aey9U66IBWPmDu4ArsbLFdcHaVvtcrec/wUrm0xH4HzC/zuuvA/6GlYRomeOx8cDFQCpwKg1nhnN5rCON5n3A+UAfHPktGnEjcKujL9ux8uReipWjeDLwlIicZowpwUqevtsYE+247XbjvVCtiAZg5Q9ux8rrsNNYGeNmAFc7stdhjHnFGFNc67nBIhJX6/XvG2O+McZUG2OcyeefNsbsNsYcAD4EhjTQfn3HjgdeNcb8YKzE+jPc+FnmOI6vNFY2uI+NMVuMZQlWDoH05r4XqnXRAKz8QU/gvyJySEQOYWW1qgI6iUiwiPzd8Sd5EZDreE3tFJM7ONHeWt+XYlUuqE99x3apc25X7dR13DEiMlaspPQHHD/bOBpOj1nve+FG2yrAaABW/mAHMNYYE1/rFm6M2YU1vXAF1jRAHJDieE3tDODeyii1h+PL+3Sv70BXfRGRMKxyPY8DnYwx8VhVOqTusbU09F6oVkYDsPK1UBEJr3ULAV4E/iaOqtEikiQizlpvMUA5VhL9SOBhH/b1bWCyiJzkyDn7QBNf3w4Iw8pHXCkiY4ELaz2/D0ioM53S0HuhWhkNwMrXFgJHat1mAP/CSmi+WESKsRLIn+E4fh7WxaxdWLlfV/iqo8aYT4Cnga+w8kk723arsokxphiYhhXID2KN5msnzv8RK/fsVseUQxcafi9UK6P5gJVyk6PCxgYgzFgFXpVqER0BK9UAEblKRMJEpD3wKPChBl/lKRqAlWrYbVhrebdgrUa4w97uqNZEpyCUUsomOgJWSimbaABWSimbaABWSimbaABWSimbaABWSimb/D+2PF31/HohEQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5, 4))\n",
    "\n",
    "ax.plot(learning_rates, trn_err, linewidth=1.5, marker='o', markersize=9, mfc='w', label='Training error')\n",
    "ax.plot(learning_rates, val_err, linewidth=1.5, marker='s', markersize=9, mfc='w', label='Validation error')\n",
    "ax.legend(fontsize=12)\n",
    "ax.set_xlabel('Learning rate', fontsize=12)\n",
    "ax.set_ylabel('Error (%)', fontsize=12)\n",
    "ax.set_xticks(np.arange(0, 2.0, 0.2), fontsize=12)\n",
    "\n",
    "fig.tight_layout()\n",
    "# plt.savefig('./figures/CH06_F14_Kunapuli.png', format='png', dpi=300, bbox_inches='tight')\n",
    "# plt.savefig('./figures/CH06_F14_Kunapuli.pdf', format='pdf', dpi=300, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Listing 6.6**: Cross Validation with XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "\n",
    "# Convert the training and test data into XGBoost.DMatrix format\n",
    "trn = xgb.DMatrix(Xtrn, label=ytrn)\n",
    "tst = xgb.DMatrix(Xtst, label=ytst)\n",
    "\n",
    "params = {'learning_rate': 0.25, 'max_depth': 2, 'objective': 'binary:logistic'}\n",
    "cv_results = xgb.cv(params, trn, num_boost_round=60, nfold=10, metrics={'error'}, seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train-error-mean</th>\n",
       "      <th>train-error-std</th>\n",
       "      <th>test-error-mean</th>\n",
       "      <th>test-error-std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.052257</td>\n",
       "      <td>0.013810</td>\n",
       "      <td>0.085700</td>\n",
       "      <td>0.031741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.042248</td>\n",
       "      <td>0.004104</td>\n",
       "      <td>0.085749</td>\n",
       "      <td>0.036140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.032970</td>\n",
       "      <td>0.005720</td>\n",
       "      <td>0.065990</td>\n",
       "      <td>0.034372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.032482</td>\n",
       "      <td>0.004791</td>\n",
       "      <td>0.065990</td>\n",
       "      <td>0.034187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.030523</td>\n",
       "      <td>0.004249</td>\n",
       "      <td>0.063865</td>\n",
       "      <td>0.035097</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   train-error-mean  train-error-std  test-error-mean  test-error-std\n",
       "0          0.052257         0.013810         0.085700        0.031741\n",
       "1          0.042248         0.004104         0.085749        0.036140\n",
       "2          0.032970         0.005720         0.065990        0.034372\n",
       "3          0.032482         0.004791         0.065990        0.034187\n",
       "4          0.030523         0.004249         0.063865        0.035097"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_results.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAAsTAAALEwEAmpwYAABYv0lEQVR4nO2dd3ykdbX/32cmyaS3Tbb3xrJ0RJpIV5qIXkFBBVEvyLVd/d2rgiJNxV6vBZGmoIJiARSl9770XXaB3WV7ye6ml+nn98f3yWZ2MklmkpnMJDnv12teM/N92nmeTJ7Pc77f8z1HVBXDMAzDKDR8+TbAMAzDMFJhAmUYhmEUJCZQhmEYRkFiAmUYhmEUJCZQhmEYRkFSlG8DckFDQ4POnTs332YYhmEYafD888/vVNXG5PZxKVBz585l2bJl+TbDMAzDSAMRWZ+qPa0uPhHxi8jDIhLIrlmGYRiGkZq0BEpVY8C8dNc3DMMwjJGSieBcCfxKROZ4HpWv95Ur4wzDMIyJSyZjUNd57+cmtAmggD9rFhmGYRgGmQnUvJxZYRiGMU5ob2+nqamJSCSSb1MKhuLiYiZPnkx1dXVG26UtUKq6HsDr0psCbFfVeEZHMwzDGMe0t7ezfft2ZsyYQVlZGSKSb5PyjqrS09PD5s2bATISqbTHj0SkWkR+BwSBzUCPiPxWRGoytNcwDGNc0tTUxIwZMygvLzdx8hARysvLmTFjBk1NTRltm0mAw8+ACmBfoAzYDyj32g3DMCY8kUiEsrKyfJtRkJSVlWXc7ZnJGNTJwHxV7fa+vyEiHwfWZHREwzCMcYx5TqkZznXJxIMKAsmpKBqAUDobi8jJIvK6iKwWkYtTLD9aRF4QkaiInJnQfqCIPCUiK0TkFRH5UAY2Zw9V6MzMPTUMwzCGTyYCdR1wn4hcJCKniMhFwD3AtUNtKCJ+4BfAKcBS4BwRWZq02gbgfOAPSe3dwHmqug/Oi/uJiNRmYHd26NgG3btG/bCGYRiFwkUXXcQ3vvGNUTteJlF83xSRLcCHgenAFuB7wA1pbH4osFpV1wKIyK3AGcBrCftf5y3bIzJQVd9I+LxFRJpwnlxruraPmFAHdG4DxHlS5sIbhpEmr25qy+n+95uZfpza3Llzue666zjxxBOHdaxrrrlmWNsNl7QEyvOAHgBOUtV0BCmZGcDGhO+bgMMy3YmIHAqUkGLcS0QuBC4EmD179jBMHIBYFFp68xgqxMJQZCkJDcMYX0SjUYqKCit/eKa5+PLmOojINOBm4OOp5l+p6rWqeoiqHtLY2C9r+/BpXQ/xhMiTaDB7+zYMwxglzj33XDZs2MDpp59OZWUl3/ve9xARrr/+embPns3xxx8PwFlnncXUqVOpqanh6KOPZsWKFbv3cf7553PppZcC8PDDDzNz5kx++MMfMnnyZKZNm8aNN96YVZszzcV3zTBz8W0GZiV8n+m1pYWIVAP/BL6mqk9nYPPI6NwBofY926JpxYQYhmEUFDfffDOzZ8/mrrvuorOzkw9+8IMAPPLII6xcuZJ77rkHgFNOOYU333yTpqYmDj74YD7ykY8MuM9t27bR1tbG5s2buf766/nMZz5DS0tL1mzONEjiPGAtEAYiQNR7H4rngEUiMk9ESoCzgTvTOai3/t+A36nq7RnYOzLC3dCeQkPNgzIMYxxxxRVXUFFRsXv+1ic+8QmqqqoIBAJcccUVvPzyy7S1pR5HKy4u5rLLLqO4uJhTTz2VyspKXn/99azZlolALcJ1881PePV+HxRVjQKfxUX9rQT+pKorROQqEXkvgIi8XUQ2AWcBvxaRXr/yg8DRwPki8pL3OjADu4dH63pcHtwkouGcH9owDGO0mDWrr3MrFotx8cUXs2DBAqqrq+mtTL5z586U206aNGmPcavy8nI6OzuzZlsmQRLLgVpVHVYfl6reDdyd1HZZwufncF1/ydvdAtwynGOOiIE8JfOgDMMYo6SaLJvY9oc//IE77riD+++/n7lz59LW1kZdXR2qKR7WR4FMgiTeACbl1pwxQDwC8Vi+rTAMw8iYKVOmsHbt2gGXd3R0EAgEmDRpEt3d3Xz1q18dRev6k0lM4e+Bf4jIT3Fh4rslVVUfzLZhBU00BCXl+bbCMIwxQCbzlHLNJZdcwuc+9zm+/OUv747GS+S8887jnnvuYcaMGdTX1/ONb3yDX/3qV3mw1CHpum4i8tYAi1RVhxyHGk0OOeQQXbZs2ch2suXFgZfVzoHy+pHt3zCMccfKlSvZe++9821GwTLQ9RGR51X1kOT2TDJJWMHCXizU3DAMI+dkEsWHiBSLyDt7E7aKSIWIVOTGtALGAiUMwzByTiYFC/fDBUr8Brjeaz6G9HLxjS9iFmpuGIaRazLxoH4FXKaqS+ibnPsIcFTWrSp0zIMyDMPIOZkI1D70zUdSAFXtwlXXnVho3CbsGoZh5JhMBGod8LbEBi+7+OpsGjRmiFmghGEYRi7JZB7U14F/isg1QImIXAJcBFyQE8sKnWgIAlX5tsIwDGPckrYHpar/wFW0bcSNPc0B/kNV782RbYWNjUMZhmHklIyqU6nqi8Cnc2TL2MLmQhmGYeSUjOZBGQmYQBmGMcaYO3cu999//4j2cdNNN3HUUaMTvG0CNVxiIchThl/DMIyJgAnUSDAvyjCMMUKqku9PP/00Rx55JLW1tRxwwAE8/PDDu9e/6aabmD9/PlVVVcybN4/f//73rFy5kosuuoinnnqKyspKamtrc2pzRmNQRhLRIBSX5tsKwzAKlLkX/3PUjrXuO6cNuvzmm2/mscce47rrruPEE09k8+bN7L///tx8882cfPLJPPDAA3zgAx9g1apVlJeX8/nPf57nnnuOvfbai61bt9Lc3Mzee+/NNddcw3XXXcfjjz+e83PKJNVRvYhcLSJ3i8ijia9cGljQmAdlGMYY5ZZbbuHUU0/l1FNPxefz8a53vYtDDjmEu+92dWV9Ph/Lly+np6eHadOmsc8++4y6jZl08f0BOAK4C5eLL/E1MbFQc8Mwxijr16/nz3/+M7W1tbtfjz/+OFu3bqWiooLbbruNa665hmnTpnHaaaexatWqUbcxky6+I4HG4ZZ8H5dY0ljDMAZhqG630SaxvPusWbM499xz+c1vfpNy3ZNOOomTTjqJnp4eLr30Ui644AIee+yxlGXjc0UmHtQrwMxcGTImMQ/KMIwxRGLJ949+9KPcdddd3HPPPcRiMYLBIA8//DCbNm1i+/bt3HHHHXR1dREIBKisrMTn8+3ex6ZNmwiHc/+AnolAPQj8W0S+KiKfSHzlyriCJx6FWDTfVhiGYaTFJZdcwje/+U1qa2u57bbbuOOOO7j66qtpbGxk1qxZfP/73ycejxOPx/nRj37E9OnTqa+v55FHHtld+v34449nn332YerUqTQ0NOTU3kxKvj80wCJV1eOzZ9LIyXnJ90QaFkPJxKvZaBhGf6zk++DksuT7cSO0bXwSDZlAGYZh5ICM5kGJSB1wOjAD2AzcpaotuTBszGDjUIZhGDkhk3lQRwBrcCU29gc+Bazx2icuJlCGYRg5IRMP6ifAp1X11t4GEfkQ8DPg7Vm2a+xglXUNwzByQiZRfIuBPyW13Q4sTGdjETlZRF4XkdUicnGK5UeLyAsiEhWRM5OWfUxE3vReH8vA5twTDVrSWMMwdhOPx/NtQkEynOuSiUC9CZyd1HYWrttvUETED/wCOAVYCpwjIkuTVtsAnI/LWJG4bT1wOXAYcChwuTcWViCoTdg1DAOAiooKNm/eTDgcJt0I6fGOqhIOh9m8eTMVFZkFlGXSxfcF4B8i8nlgPTAXWAS8J41tDwVWq+paABG5FTgDeK13BVVd5y1LltmTgPtUtdlbfh+usu8fM7A9t+xaDb4iED/4fN67H0oqobQGRnHmtWEY+WPmzJns3LmT9evXE43aHMleioqKqKmpyXjeVCZh5k+KyALgNGA6Liff3b3CMQQzgI0J3zfhPKJ0SLXtjOSVRORC4EKA2bNnp7nrLBELp/aiunY44Sqrh/J6KC4bXbsMwxhVfD4fkydPZvLkyfk2ZVyQacn3FuCWHNkyIlT1WuBacBN182xOH/EodDW5V3E5VE6Bstp8W2UYhlHwDCpQIvJvVT3Z+/wYkPLGr6pHD3GczcCshO8zvbZ02Awcm7Ttw2luW1hEuqF1gxOqopJ8W2MYhlHQDOVB/S7h83UjOM5zwCIRmYcTnLOBD6e57T3A1QmBEe8GLhmBLflFY9C2ESYtyLclhmEYBc2gAqWqiRF1q1T1meR1ROTQoQ6iqlER+SxObPzADaq6QkSuApap6p0i8nbgb0AdcLqIXKmq+6hqs4h8AydyAFelOe5VuITaoacFykYhGDFiVX8NwxibZJIstl1Vq1O0N6tqfdYtGwGjmix2uPiKYfLeLtovVwTboPktqJkFFZNydxzDMIwRMFCy2CHnQYmIz5vHJB6+hNciwGIph0M8Au3pDsMNg3A3tKwDFNo2QPfYdjoNw5h4pBPFF6UvOCJZjOLAt7Jq0USie5fr5gtUZXe/0TA0rwVNmFLWut69lxeUs2sYhjEg6QjUPECAR4DEaD0FdqhqTy4MmzC0boTGJW6CbzaIx6B5jfPQ+h1rvZs0PBpjX4ZhGCNkSIFSVe/Rmzk5tmViEgtB5zaonj7yfam6MafBMqy3rAfE5mIZhlHwZFoP6r3AMUADzqsCQFXPy7JdE4vOJi8tUr8YlMxoXQ/hjiFWUjc2JfNHfjzDMIwckkk9qMuBX3vbnAXswuXJa82JZRMKhZa3IDSUuAy0uTrPqCfd2pGeSEVDwzueYRjGKJDJwMcngHep6heBsPd+Oi5prDFSNO4CG0KdmW0Xj8GuNdCTYZSextzx4rHMtjMMwxglMhGoWlVd7n0Oi0ixqj6L6/IzsoHGXYBDuCu99aNh2PlmGt16A20f9ELRDcMwCo9MBGqNiOzjfV4O/JeInAuk269kpIPGnUc0lEiFu2HnGxAdYRBlqB3at45sH4ZhGDkgkyCJS4HedAQX4woLVgKfybZREx71uu0mLYSScq9N+97DHc7z0SxV7uzc5tIhWfi5YRgFRCb1oO5O+PwsaZZ6N4aJxmDn67hgyVGoHtK6AfyBPkEcjEiPm78VqMxOeLxhGEYKMoniO09E9k9qO8Dr5jNyxiiVtuod/+rYBrEUk3wB4nFo3wI7XodIF3Rud0Jlpa0Nw8gBmYxBfYM9K9viff9m9swx8ko8Ch1bYfsKN+E3Mew91AE7VjlRShTN7p1u/pWJlGEYWSaTMahqoD2prQ2ozZo1RoGgEGx1L3/AjU8F2wZevafFhavXzcteyibDMCY8mdxNXgM+kNT2fmBl9swxCo5YaHBx6iXUbvOqDMPIKpl4UF8B7haRDwFrcEESJwCn5sIwYwwS9roBfcX9lwUqoWqaS1ZrGIaRBml7UKr6OLAfrrJtBfAssK+qPpEj24yxSCzsAiiSX53b3bytyCCJbA3DMBLIKFmsl9n8OzmypXCIx+GJn7nItnjMhXzHYyA+2P+DUD8v3xaOTSLdLnS+eqZV+DUMY0gGFSgRuVZVL/Q+38wAMc/jLpu5CKz4a+pl8Qgcf+no2jOe0Lir8Btqh9rZuS15bxjGmGYoD+qthM+rc2lIQSECR37OeUzidzfRrp3w/I2Wuy5bBFuhqQsqGqCsHopK8m2RYRgFxlAClVgf/DFVfTCXxhQU+yYFLEZDMOdI99RvZId4xM276tgKJVUu1VJZrXlVhmEAQwdJXJjw+e85tKPwKQpAwyL3bmSfcIfr+tu+HLozLB1iGMa4ZCgP6mURuR03ByogIlelWklVL8u6ZcbEROPQvQvK64de1zCMcc1QAnUmzouag8taOivnFhUybz0Gr/0N5h4N+7wv39aMX8KdLh+gP8V8KsMwJgyDCpSqNuHl2hORIlX9+KhYVaiE2mDzC1A2yQQq1wTbXACFYRgTlkHHoET2mPb/SRHxpXqlcyAROVlEXheR1SJycYrlARG5zVv+jIjM9dqLReS3IvKqiKwUkUsyOcGsUjvXvbeuz5sJE4ae1nxbYBhGnhlKXBKTsEWBSNKrt21QRMQP/AI4BVgKnCMiS5NW+yTQoqoLgR8D3/XazwICqrof8DbgU73iNerUzXHvLeuzVyzQSE24E2LRfFthGEYeGWoMap+EzyNJn3AosFpV1wKIyK3AGbjgi17OAK7wPt8O/Nzz4BSoEJEioAwI0z+r+ugQqILySW4Qv2M7VE/LixkTA3WTeS1YwjAmLIN6UKq6MeHz+sQX0ARs8z4PxQz2rCW1yWtLuY6qRnHe2yScWHUBW4ENwA9UtV8csohcKCLLRGTZjh070jBpmPR6Ua3rcncMw9HTkm8LDMPII5lU1P2BiBzqfT4NaAZaROT0XBnncSgQA6bjvLj/EZH5ySup6rWqeoiqHtLY2Jg7a3rHoVpsHCrnhDrGTvmOxOKOhmFkhUzqQX0EWO59vgz4KPBe4Oo0tt3MniHqM722lOt43Xk1wC7gw8C/VTXiRRU+ARySgd3ZZcbbYMnpMGlB3kyYOGh6tajyTaTHPbDEbVzSMLJJJgJVrqrdIjIJmK+qf1HV+3FzpIbiOWCRiMwTkRLgbODOpHXuBD7mfT4TeFBVFdetdzyAiFQAhwOrMrA7u8x9Bxz9PzDz7XkzYUIxFgSqp8WlberemW9LDGNckUm5jTdE5CO4QoX3AYhIA9Az1IaqGhWRzwL3AH7gBlVd4WWmWKaqdwLXAzeLyGpc9+HZ3ua/AG4UkRW4ycI3quorGdhtjGVC7c4zKeRS8r1jZZ3bXRCN5RI0jKyQiUB9GvgpLoruk17bScC96WysqncDdye1XZbwOYgLKU/erjNVe15p2wS71sD0g6C0Ot/WjG807kSqrDb18lgU4inC0f0loyNqoU5XpBGcHV07oGpq7o9rGBOAtAVKVZ8Djkxq+z3w+2wbVfA8+gPY+hKc8l2YdVi+rRn/BFtTC1SoE1reSi1QFZOhJjlQNAckRxp2NkF5A/gzqgVqGEYKMoniO05E5nmfp3nZHW4UkYn3uFg3171bJN/oEGwHTaqV2dMCu1anFidw40GxIeeQjwxVJ557tMWgqym3xzWMCUImfSC/xIV7A/wQKAbiwLXZNqrg2Z1RYl1ezZgwaGzPMO7OJu/apyzw7G0Td+vlklB7aoHs2pF7cTSMCUAm/RAzVHWDFwJ+Ei56LwxsyYllhUyvB2U5+UaPYKvL5NG2Kf1oue6dUDk5d1nRB6pbpXEXMFEzMzfHNYwJQiYeVLuITAGOAV7zghfAeVITi9oEDyq568nIDcE2aF6bWSh3Lr2oeMx5UAPRtROi4dwc2zAmCJkI1P/h5jP9Hhf6DfAO8jknKV+U1UGgGsJdLi+fkXvi0cEFYSByNRYVbBsiYbBC57bsH9cwJhCZRPF9V0T+BsRUdY3XvBn4z5xYVsiIuG6+7cuhfYvVLSpker2obEf0pZMnsHtXZuXr6+fbtAXDSCCjWFhVfWOw7xOKEy5zYyJFgXxbYgxF906onJK90O9YJIPcexl0AQdbTaCM/kzg6tJp/8eKSDWuHMYxQAMuqwMAqjo765YVOuY1jR12By1kyYvqaSUj4UmXYJsb09yjTugIUHWeXuf2gcPxR0rtnJGJatdO8BUNPBE7ed2Orf3b/SXQsHj4103VTbyPpkiKUzcPApXD2282iATddIrSGqidNfT644xMHil/iUvyehVwCy5Z7JeAv+TALsPILtn0onJVBiQedZ5ZNryoYBu0b019080mbZugZMnwsnZEw9C+2QmEzh689ldnk1s3FfGo60qtmJS5DQAd2yA8gEccbM2fQEV6nHDuzvOoUDuxfIFMflXvBj6gqnfgxqHuAD4EnJsTywqdSDfc9QX403kWyTcW6PWiRko0BJGuke9nIJIn/mZKuAt2vukiHnMtTgCx0PCDQdo2eoEm6qZsdA0QcNS+dWBx6qVz2/D+D0Odg9ufr2TFkR5vInpCgE/3Lq+a98S532TyOOmjrwR8p4jU4IoILsy6VWOBojLY9aa7IfS0WOXXsUBXE/RkELSQilzfHEbSzdexHTryMC2xs8lFthaXpb9NT0v/qMy2DW5SduXkhLZNbuLzUMTC7gaeSdd7PDb0XMZYGMLdUFKe/n5HSrjbiZOmqIXW04zzpOZkryu4gMnEg3oZN/4E8Biuy+9XwMQMlOiN5AObsDuWiEdH9kp108i2fcMpfti5Iz/iBDgPaGP64h2PQdsAHlH7ZucxqULrhvTEqZeObZnV5Grb1JfodzBG04sKdw0sTr30tLgclBPAk8rEg7qAvsCI/8YVKqwFzsuyTWOHurmwfYWbsDv9oHxbY4wXMo3m626G9k05MyctIl0uiKEyjWrW7Zv37LpKpnObuwbRYGY2xCPOi0rHhp6W9L3pYCtUT8vMlsGIhty1ikdcF6eq9x535zzo/Lpem9qg6TWQAihD07BXzioHZDIPam3C5yYm4vynZKz8u5ELMunm62kpHA++Y4uLNisqGXidUGd6k9szFadedtfkGuSGGQ07jy9dokEXTVdcOjybegl3e93MWQqyScf7G+MMKlAi8ol0dqKqN2THnDGGJY01ckG60Xw9rYX1cKRx58nVzx9gubrAiFzSG/GWOI6VTOuGzLtqg23DF6hQhxunG04mlAnOUB5UOhF6CkxQgZrr3gvlCdYYPwzVzRdsHzqjez4ItjkPIVDTf1nn9uF7RpkwUGXjUKebRxXuTL3dYARboWrKwMvjMdixaoC0WgX2NxpDDCpQqnrcaBkyJqlohH3eDzWz3NPjhqdh26t9y2e8DWYekj/7jLHLYN18vYUaC/XGl+8ehXjUjfH0Ckok6LofRxLsEOl2XYMDdV92bJsQXW6jTSaZJN4NrEtMbyQii4E5qnpfLozLJ/F4GuN+IvCO/+77vvkFWH573/eX/wgLT4QjPwultbkw0xivDNTNF+6C5jXpDaRPZDq3u/GwrqbsJXQOtqUOwIgEM4s2NNImkyi+XwBHJ7V1eu2Ls2ZRgRCOxSnNNDJl9uFQXuc+Bztgxd9g9f2waRkc82WYc2T2DTXGL8ndfOFul1nAxGloNAY7VmZ3n8HW1ALVvpmC9WbHOJkI1GRVTU6EtRUYlyXfI7E4pcUZCtTMQ/bs0tv7PfDo92Hry9k1zpgY9LS67mMR95TevCb387CMgQl39k/cGmyz4IcckolArRWR41X1wYS2Y4G3smtSYRCKxqka6U5qZsJ7fuy6/hKFK5sJQY3xS2+p+6KAl/YmRwlfjfQJtvfl/FMdeMKxkRUyEagrgL+KyPXAGmAB8HHvNe6IxLLUjSK+PcVpw9Ow7AY49QdWWsEYmq4dLvJtsImtxugRbO0TqK4dLhehkTPS7sPyksO+G6gATvPeT/Laxx3hbAlUIhqHF2+GnW+4rr8JkKrEGCGhdosOKyRCHS6kPBZJXfrDyCqZFix8Fng2R7YUFJFoPL1IvkwQHxx3KfzlP2HdY/DaHbDP+7J4gBRYd6JhZBH1xp06LFhlFCiARE6FiQKRTBJPpkv1NDj6f93np3/horJyxRv3wu/eC0/+3OX/Mgxj5HRmISu+kRajJlAicrKIvC4iq0Xk4hTLAyJym7f8GRGZm7BsfxF5SkRWiMirIjLCpFjpEY7m6AlpwXGw5DTXTfDAVa72S7bZtRoe+7570lt+O/z1Ape92TCMkTEadbYMYJQESkT8uPlSpwBLgXNEZGnSap8EWlR1IfBj4LvetkW4Cr4Xqeo+uMjBURkxzsk4VC9Hfs7VdGldD0/9Irv7DnfD/Vc6AZx7lDtOLAJlVrPKMIyxQxbqX6fFocDq3ozoInIrcAbwWsI6Z+AiBQFuB34uIoILzHhFVV8GUNUsTQsfmpx5UABFpXDi5XDv1122iWyhCo//yCXlrJsHx18KiOuW6C26FulxM+17cwkahmEUIENlM7+ZNKZIq+pQNaFmAIlpjDcBhw20jqpGRaQNmITLUqEicg/QCNyqqt9LYeuFwIUAs2fPHsrktMipBwUu6/MHf9eX1DIedYEUI63xMnU/2PgsnHiFE0KA2ll9y5/5Nbzxb3jfLwfOPG0YhpFnhroTrsbNeVqDK/f+PsCPExgfzutpzZ15gBPRo4CPeO/vF5ETkldS1WtV9RBVPaSxMY2CZWkQicZzHwmemHH5pT/AP77opU4ZJiKw9Aw459a+ciCJaNzlJosGXTdgLsa/DMMwssCgAqWqV/a+cJ7Maar6EVX9qqp+FDcfaq80jrMZSHiEZ6bXlnIdb9ypBtiFE8NHVXWnqnYDdwMHp3HMEaOMghfVSzQEK//h0iLd/klY/tfMwlgj3XsGQfR25yUjPjjuq33jX0/8ZERmG4Zh5IpMxqAOB55OansGOCKNbZ8DFonIPJwQnQ18OGmdO4GPAU8BZwIPqmpv196XRaQcCAPH4IIoRoVwNE6gaBRiSYoC8IHfwBM/gzUPwJM/g7cegdlHgL8E9v2PvnVf+j2EkzyfHStdCejjvz50UtriMtf997eL4I17XLn6xSenb6vGYc1DUFYHM0blWcEwjAlIJgL1InC1iFymqj0iUgZcCbw01IbemNJngXtwXYQ3qOoKEbkKWKaqdwLXAzeLyGqgGSdiqGqLiPwIJ3IK3K2q/8zA7hExah4UuPIAJ3wd5h8Dj//YeVNbX3alOhIFasXfU6f3LyqF6unpHat+nisV8uj34PGfQOOS9IMmXv2Lm8Mlfjjj/2ByckCmYRjGyBFNc5DFm5f0B+AQoAWoA5YBH1bVdTmyb1gccsghumzZshHt482XHgOgMlDEtJpRmXa1J8FWWHGHq/9TXAaHJKQ8XP5X157M3HdkFvSgCg9/G968Fxac4MQxHSLdcOOp7nPVVPiP30BgxKl1DcMYi0w9YMQpd0TkeVXtV901bQ/KE6EjRWQ2MA3YqqobRmTVGGBUPahESmvhbR9LvSzRmxoJInDUF6BqGhyY3OOaQPsWeP4mt25xuXt98j644zMur+BjP4ITLrOUSoZhZJWhwsxTyeIm77V7uer4TUrVG8k3bu+9xeV7emdP/p+bQxWPu3IP8ZgToWjQTfQ9/CK3nr/YidJfL4C1D8HMt8GS9+TnHAzDGJcM5UFFGXwelHjL/YOsM6bpjeQblUCJQmD7ctjxev/2BcfDgWfv2VYzE476Ijx0NTzxfy7YonrG6NhpGMa4ZyiBmjcqVhQ4oxbJVwgc/hk3xuTzuyAI8bngjfoBfgqL3g1bXnJiVTVtVE01DGN8M6hAqer65DavW28KsH08d+1pUTkS7QayWLxwLDBt/8y3OfpL47gP1DCMfJF2kISIVAM/x4V/FwERL6fe51W1LUf2jTqxuPL8+hZ+/4py/iKhslgJ5TIn33ggUZxa1sF9l/Uvxlg5GQ7/L5i0cFRNMwxj7JLJPKif4aro7gusB+YA3/LaBwg3G3t8/KbnePQNN8doYf1MTp7aPrE8qJHQ2eRSNfW09F/WthEe+jZ84DrztgzDSItMBOpkYL6XbgjgDRH5OC5P37jhsHn1uwXqiQ3dvGtuA+HgzvEdyZctKic7AUqeo6UKK/4Gi9/ddxHtghqGMQSZjPwHcdnEE2kAxlWp1lP2nbr78wvrW2kvnkzcX5a/+VBjjfJJUDt7z1fdHDeHKjHjxGM/hBd+5zK4G4ZhpCATgboOuE9ELhKRU0TkIlzqomtzY1p+mN9YyV5TXFaEcCzO8xtbCVfPJhy3p/2s0bwWVv0Dlt0Af/+0G7cyDMNIIhOB+hbwHVwi1x9679/z2scVJyd4UU+u2Yn6AwTL08xxZwxN/Xw47YdQOcVNAv7rBfDKnzLL3m4YxrhnUIESkdsSvp6vqjeo6omqutR7v17TTeY3hjhlvz6BWrauhVA0RrCoGsob8mjVOGPG2+DMG2CvU105+qd/6dXC2ppvywzDKBCG8qBO8squA/w018YUCntNqWK6lyC2JxLjxQ2thKIxKKvNr2HjjZIKOObLcNLVrnTH1pfh5T/m26rM0TgEx81MC8MoGIaK4nsMeEpE3gBKReR3qVZKo+T7mEJEeMfCBv78vCsA+OSanRyxYJKbvJtn28Ylc46Es26EZTfBoRfk25rM2bUG/vUVWHgivP2TrraXYRgjZigP6izc5NzXcWnp1gzwGnccuaCvO+/Zt5oJR+OE1Qe+4jxaNY4prXWRfiUV7ns8Ch3b8mlR+tTOggXHwat/gr9eCDtW5dsiwxgXDJXqKAjcAiAixV7p9wnBgsYKJlcFaOoI0RWO8cqmNhZMriRQXAahSL7NG9/0tLpsFF1NhV1rKhp0RSKLSp33tPFZaF3vIhMPOhcOPhd8mUw1NAwjkbSj+FT1it7PIvLLnFhTQIgIRy6YtPv7k2t2unGo4rI8WjVBKCl3N/+ObfDoD/qnTeolGnQl65f/te81Wl5XsB3+fD4su9GVJJm8t5ukvO+Zbkzqhd86oUr0pjqbYM2DA7+i42pKoWGMmOE+3n0U+HQ2DSlEjlzQwN9f2gLA02t30RGMMLkiD9V1Jxr+EjjxcvjLBfDWI7DyTlh6xp7rbFsOj3wH2jbt2V4zw1X5Bdi1GooroDrLWdZV4ZHvOjHc9Cwc9FHA78aejvysq2z88HdcCP3G56Bxidtuxyp44KqB93vu3238yjASGK5ATYhYgb2mVlFfUUJzV5j2YJSn1zQz64AGSvJt2ESgega883/gwW/AUz+HKfu4RLOq8Myv4JU/Awp1c2HagX3bVU5x79Eg3H8FdO9yJUSWnJa91Eor/grrn3DjZSdc5oo3JjL9IBdC/9IfXBaNXioaYP5xA+83eT+GMcEZrkBdnVUrChSfCEfMn8Q/X3Vzc55Ys4sT957MlN11Go2csvAE2PKiyzpx/5XwH792FYAjPU5sDvgwvO1jzuNKJhaB+gXOw3rsB7DucTjmSy4V00jY+QY8fY37fPSXB66BVVLRPyJx8lLnGRqGkRYZCZSI1AFVwDW5MafwOHJBn0A9vWYXOzrDTC4uRaI9ebZsgnDkZ2H7Cmh5C167Ew44Gw77L9jrNJi8ZODtAlVw4hVubOeJn8DGp92Y0fxjnSfW22UY6oDnrht4P0vPcJkvANY/CU/+DOIRWPo+mH9MVk5xN+FOePUvEGqHIz+X2bbxqLs+rf1KuMGsw1woP7g0U6/dMfB+3v6ffUEpr93h1k9F3VzY5/2Z2ZhMNOjGDWtmwbx3pr/dU7+AWLjvu68Ylr7X5X3MBqv+ATvfdJ/r5rq/tSU2zgtDCpSIFANXAh8HJuOVeReR7cCNwBWqOm7D2vaZXkN1aRHtwSjN3WFWbGln1owiavJt2EShqNR5HeufgP0/6NpKygcXp15EnBc27QB49Hsuym7lXTD7yD6BigYHv2HPPrxPoHa87sadJi1wta2yTbDdBVcA7PsfrpszHeJRuONzsGNl6uWB6j6B6mwa/HwP/EifQG181l33VMw6rE+gNA5dO102+3TZvgIe/rbzcBefNLhARbqd59zLqrshkpQxf+3DLkgl08n0sTA88j048MN9f+dNz8Pah/rWqZ3tMp8Yo046HtSvgAXAR4CXgTagGjgQ+Jq3/D9zZF/e8fuEw+dP4t7XtgPw5OqdHNpYS40NRI0edXPda7hUNMDJ34UNT7kbdFVfKitKKuAdXxj82L3MPtzdhOcdnZtghurpsOgkeONf8MItcOxX0tvOVwTTD4SeZtjvzP5z9RoX932umzv4+ZZU9n3e+70w8+2p10sUoxV/g2d/A4d/GvY+fXBvIxaG53/rMoZoHMQHh3yib3moo08gNe7EdNkNcPpP+wTkiP+CWEIW/DfvgaaV8PDVcPJ33D7T5Zlfw+r7XUDNmTe4bZec6h5qml6DN+91WfdNoPKCDJVKT0RagTmpquaKSC2wTlVrc2HccDnkkEN02bJlI9rHq5v6TveF9S1cftcKABqrAtx49l4sKNpOWbF/RMcwjH60bYI/eYlZPvT7gSMQ27c6QZqyj/seC7txt96JzqPJw991ogpO0I75MlQkV+bBZdx46GpoXgOI66495ON9Y4jRkJvoXDcXDjgHnv21G4MEOPhjbt1UdDa5hMN7neoyeaQ79+ytR918O18RvPfn/b3ycBf88Wwnmu/5sQt+Mfoz9QDwZZJ3vD8i8ryqHpLcns5fsgeYhvOckpmGqxM1rtlvZg0VAT9doRg7OkKs3BWjoTpiAmVkn5qZsPBdzit46RY4+kt7Lld13ZRP/9J5O2fd6DwOf0nqYJHR4NivwKy3w+M/gU3PwZ8/7rpBwUU5lk9yuQrv+IzrUq2eDsdeAlP323M/u1ZD53Y3jvbWI66ttBaO+uLg432Vk+GDN0Npdfo2t291UwUADvtU6i7jkgrY7yznwT3/2+wJ1JqHnNe58MShPc6mVfDcb6BhMbzt/Ak3DSEd2fse8JCIfFNEzhKRd4vImSLyDeABXAmOIRGRk0XkdRFZLSIXp1geEJHbvOXPiMjcpOWzRaRTRP43neNlk2K/jyPn96U+enR1Mx1hIRK3SD4jBxz0UdfV9Pq/95x43LEN7v5fePxH7kY/dd+BJzGPNguOd2I563AX7LH1ZffqDWYorXHjW0vf58aKksUJnDd41o0w/WD3ff5xcNZN6QWjJIpTsHXw5L2xCDxwpfOQ5rzDTa4eiH3/wwXVLDxh5Nc62A73Xe6Ove0V93e8+0tOlPvZGIZnr4M7Pg2bn3ddor3jkxOIIT0oVf2xiLwGnAe8B6gEOoEVwMdV9Z6h9iEifuAXwLuATcBzInKnqr6WsNongRZVXSgiZwPfBT6UsPxHwL/SO63sc8ziRu5b6X5Ij63eyX/t10hHT5j6ChuMMrJM7Sx3w199v3sd+BFY9U/nNUW6XdDDUV9w6xQS5ZPg5G/DrjfdzR+grL5v+UEfHToarmqaqxXW0zy8KQE7VsG9l0L9Qjj56tTjUc/82q1XOQWOvXhwm0oqXbqtbETx+YtddGBxmRvfe/1fsHmZ8ziP8ObqgQvGefg7LnIVcQE97Vtct+cEI63OWk+EhhSiQTgUWK2qawFE5FbgDCBRoM4ArvA+3w78XEREVVVE3ge8BSSF7owe+86ooa68mJbuCK3dEV7coRzZEKGuvMQiUI3s87bzXVffrENdlFnvGM/co504ldcPtnX+EHHdUQMtS3cfw52vVlYP0bCbVvDSH2Hv0/qWFZe7btCGxU54Trw8vTyPiXarDnweoQ7QWFJbp/tbFZc7YTrxcuftVU2D/T/kukXXPdo3PWDz886r0rjr7j3mK/29zWjIZSTZ/0NQlyK0vrR2cJt68Zf0RUfGo87zHYiSyr6xvXA3xBPC/HPoxY8ok6VXK+qdqvroEKvOADYmfN8EHDbQOqoaFZE2YJKIBIGv4LyvAbv3RORC4EKA2bOzNB8iAb9PeOeiRu582aU+enhdD4fWK52hKFWllhDUyDI1M90LXKb09U/AO/7beU32RDQwlZOdV3TPV93YzXO/6Vt20tUu3H7xu106qkwCSkIdLjNI22Z4d1K6Ko274214OvW2S89w42gAjXv1tZfXw7uuhPWPw8xDXdvU/d0E8+kHeqVbUqRWW/4X93tINQWgqBQ+8e++73//NLRt7L8ewH4fhCO8jHXbXnUFQwfig7/rm2f22A9hzQN9yy7eCP4Mxv8yYKR31hLgISCX0QJXAD9W1U4Z5B9TVa8FrgUXxZcLQ45Z3CdQT6zrIHxAOa09ERMoI7fMOhTOudXN/zKGZs6RLuT95T+6RL69JEb3ZRrtqOpC3iPdbg5Xb/QkuG7ESQudQAVS3Kg7t7txr1SprERgbsIcMH8xvO8Xgwe87PsBJ5hv3OM8n0SSk1mXVKa2CaA4Qfx8RQOvB3t2lRaXDb5uFkknzHywYoQlwK9VdVCBEpEjcBN6T/K+XwKgqt9OWOceb52nRKQI2AY0Ao8Cs7zVaoE4cJmq/nyg42U7zLwXVeVTtzzP1jYXuHj5UeUcNbOIqdWlJlKGMd559joXWTnrUDj2q9CxxaWvAidAoY7C7XrNJXkOM78ReB5IVQsg3b6G54BFIjIP2AycDXw4aZ07gY8BTwFnAg+qU8/djxcicgXQOZg4ZYul06uJxOJE40rUe+8OxTh6USO3LXMu84MbYhw1s4ht7UGau33UlBZTXVo80r+VYRiFyP5nwYq/uAwbfzoPfH4XdVha6zyfiShOOSYdgXoT+IqqPpS8QERKge6hduCNKX0WF2jhB25Q1RUichWwTFXvBK4HbhaR1UAzTsTyht8n+H17Oobh0jjHLO4TqKc3heiKlFBRLISjcXZ0htjVFaK6rJjashKK/TZWYBjjhtIal97ppT+4fInTDnSek5Ez0hGoR4AluLGmZGLe8iFR1buBu5PaLkv4HMSVmB9sH1ekc6xcUVLkY/HUSuY1VPDWzi4icXhyU4R3zevrL44rtHZHaOuOMKkyQF25lVAwjHHDAee4is8Ni12C2kzSKhkZk848qE8NsiwCDFLgZvxRVVrMMYsbeWuni3h/cP2eAtWLAjs7QwQjMSZXleK337FhjH0CVS6VkzEq2G0zQ6pKi3jnor6sEi9sj9ESjA+4fmcoysaWbkLRgdcxDMMw+mMClSHlJUVMry1jn+kuzDKu8OjG6KDbRGJxNjZ3094z+HqGYRhGHyZQw6CqtIijF/Vla35w/dADpQps7wiyozNVMKRhGIaRzKAC5VXQNZKoKi3mHQsb8PtclN5rO2Ns60yvC6+1O8KurvDQKxqGYUxwhgqSWI8rToiI3K+qJ+bepMKnKlBEbXkxB82qZdn6FgD+34NdzK/1M7PKt/u1uN5PeXH/UPPmrjAiUF9uiWYNwzAGYiiB6haRfYGVwKFe7r1+d1xVnVARAD6fUBEo4ujFjbsFake3sqM7yjMJ65UXw8f2DXDGopLd3lYvuzrD+BBqLQzdMAwjJUONQV0JPAuEgQogCkQSXr3fJxzVpUUctbCBt88duBe0OwK/ejHEp+/tYvmO/gESOzpDdAQtcMIwDCMV6eTiKwKmAquAfXAe1B4bqer6XBk4HLKRi28owtE4r2/rACDW/BZb28Nsao+xqSPO5vYoy7cH2da5p/i8a24xFxwYoK6077lAgCk1pVQFLJefYRhjkHzm4lPVKLBJRA4qNCHKJyVFPkqLfQQjcfz185hZDzMTlkdice58YT23LttM0EuofN+6CE9ujvCZg0t3T+5VYHtbEF9tKRUlJlKGYRi9ZCJ760TkShF5S0SCIrLW+z5hR/qrSgcePyr2+/jA2+fxy3P256hZfZeoKwLfeybINS8GiXkl4xXY2hqkI2TdfYZhGL1kIlDfA04EPgUcAFwEHI8rzT4hSafERmNtNRe/50CuPq6G6ZV9l/svr4f5+mPddIb7RGp7W9DGpAzDMDwyEaizgPeq6r2q+rqq3gu8H/hgbkwrfMpL/P2i81Kh/hL223tvfnXqJN4xo0/Untsa4/P3dbGpw/UBKrCt3UTKMAwDMhOoge7EE7amhIikX6jQV4SvYSGXHtvAh5f2dflt7IjzuXu7eH5bnyhtaw/SZmmRDMOY4GQiUH8G7hKRk0RkbxE5Gfg78KecWDZGqB5kHKofPj/Rmnmcf1ANlxxRRolXbqozAl99pJv73urLMNHUEaS1e0JG8BuGYQCZCdSXgfuBX+Aq7P4frkbUV3Jg15ihsrQISfAhi/xCecBPbXkxJUUpLq/PT7h6LsfPLeNHx1cwqcxtHFf4/jNB/rWmT6R2dIZoMZEyDGOCMuQ8qLHIaMyDSqQzFKXIJ5T4ffgSxqRUlfZglF2dIbpCsT22kUg3gbY17OyJ8bVHulnb2peM4/OHlHL6wr5uwPqKEiZVTNhgScMwCpkczoOybOZZoDJQRGmxfw9xAjdGVVNWzPzGShZOrqS2vHi3t6XF5YSrZtFQ5uP7x5WzqK7vT/GzZUH+/kafJ9XcFbYs6IZhTDhMoEaJshI/s+rLWTylanf+vXiglmj5FKoDPr53XAVL6vv+HL94Icjtq/pEqbU7QlPHMESqvAEC1UzgWBbDMMYoJlCjTEmRj1n15cxrrKC02Ee0fAqxQB2VJcJ3jq1g6ST/7nV//VKIW1/rE6W2ngjb2oOk3StbUgW1s2DSApiyL9TMdm2GYRhjABOoPFEZKGLh5Eqm1ZYSq55JpGI6ZRVVfPvYcvZr7BOp618JcfPyPpHqCEbZ3pGGSInPiVMv/iKomAQNC51YmVAZhlHgDDqJR0Q+kc5OVPWG7JgzsRARGioD1JYVs6G5iK5QA76qKFee2spV97zFS9ucMP1ueYhoXDl/vwAiQkcwihBkSnXpwDuvmg5FgdTL/MVQOxt2rAKNpV7HMAwjzww1y/TchM8CvAPYBmwEZgFTgCcAE6gRUOT3MXdSBet2ddEVguKqBr52Rh1X372SFze2AvCH18IEAw2cf8Rc/NFu2jvW4+sI0ViVQoSKK6CiYYiDlkDNTGgdRv5f8TvvrHMHRLoy394wDCMNBu3iU9Xjel/Aq8CXVHWWqh6pqrOAL3ntxgjx+YS5kyqo9DJTlBb7ufS0pRwyp6/e1F9f3MJ1T6wjVlxBqGYBLSFhZ2dy+Xhx3pGkERRRXg+lNZkZWlIFjUugrA6qp2e2rWEYRgZkMgb1Udzk3ER+zp5eljECfD5hTn35bpEqKfLx1VP35vD59bvXufPlLfzqkTXE/AFCtQtoDvlp7koQqappUDxI118yNbPBl042DIHqGW4Mq8ibkxWozFzgDMMw0iQTgdoGvDep7XSgKXvmGL0i1Zvjr9jv4ysnLeEdC/u67P61fBvf+/cquiJCqHYBO8KlLuNEcTlUTs7sgP6iPYMpUlFUCo17pd539QwshN0wjFyQiUB9HvitiDwpIreJyFPAb4HPpbOxiJwsIq+LyGoRuTjF8oC339Ui8oyIzPXa3yUiz4vIq9778RnYPCbx+YQ5k/pEqsjv40vv3otjFjfuXueJNbv44p9eYvWObsLVc9gerWJTfBLR+DAyg5TWQPmk/u3FFVA313XpFZel3rYoMPR4l2EYxjBIW6BU9T5gHvAr4AXvfb5XdmNQRMSPy+F3CrAUOEdEliat9kmgRVUXAj+mr87UTuB0Vd0P+Bhwc7o2j2VE9hQpv0/44omLec9+03avs7UtyJduf5m7l28jXDGNlkgxb2zvpLU7eVwqDapngL8EEDe+1LAYGhe7z0ONZ1VOdYEThmEYWSRtgRKRn6nqLlW9WVW/q6q/U9VdIvKTNDY/FFitqmtVNQzcCpyRtM4ZOI8M4HbgBBERVX1RVbd47SuAMhEZIH56fJFKpD51zAK+fNJelBU7QYjGlV89sobv3/s63eEosbiysbmHdTu7CEfjg+1+T3x+qJ8Pk5c6r6mkIv1t/UVQNTWDMzMMwxiaTLr4zh+gPZ0giRm40PReNnltKddR1SjQBiT3O30AeEFV++X8EZELRWSZiCzbsWNHGiaNDZJFCuCdixr5yYcOZH5Dn4g89uZOvnjbS2xp7QHchN43tnewszNE2gmBi8v6AiAypaIR/BPiucEwjFFiyGp7CZN1i1JM3J2P64LLOSKyD67b792plqvqtcC14LKZj4ZNo0WvSK3f1b272u702jK+f+YBXPf4Wv61fBsAW9qCfPkvr3D5e5ayaEoVqrC1NUhzV5ipNaWZ1a7K3EiongYt63J3jLGE+KBiMpSUp79N9y4ItuXOJsMYY6RTDrbXQyphT29Jge24caGh2Iyb2NvLTK8t1TqbRKQIqAF2AYjITOBvwHmquiaN4407UolUSZGPTx+7kH2m1/CzB98kHI3T1hPhq39/lUtO2ZuDZ7s5VKFInPU7u6ksLWJaTSmlxTkaLyqrs8m7AGX1Ltw/U280UA3NayHUnhu7DGOMMaRAeZN0EZFvquqlwzzOc8AiEZmHE6KzgQ8nrXMnTuyeAs4EHlRVFZFa4J/Axar6xDCPPy7oFamtbc4r6u25O2ZxI1OqAlz1j9foCEUJRuJc9Y/X+MIJizh2r77Q8M5glNWhTuoqSphSFaDIn4NUjJMWgqYY+4p0QdsmiA0jgCNb+IqgaIBoxLRRiEfdeSSfZ0mVm7ycideUiAjUzYPmNRDuzHx7f4mzLdX1N4wxSNoFC0WkEehR1U4vKu88IAbcojr0f4SInAr8BPADN6jqt0TkKmCZqt4pIqW4CL2DgGbgbFVdKyKXApcAbybs7t2qOuD8q9EuWJgPorE4u7rC7OoME/NCyzc2d3PZnSvYmVA76hPvmMv7D5rZb3ufDyZXldJQWYKkk3UiG8Tj0LkNOptwDvgoUt7gxMOXRe8xFnFCFQu7KMbS6uzsNx6DXash0j30uv4AlNW6qQIlFc6mjm2uu3C0r7ExMclhwcJMBOoZ4CJVfVFEvgu8B4gAD6nqF0dkXZaZCALVSzyutHSH2dkZJhyNs6szxOV3rmB9c9/N7ehFDZz5tlnMa+gfmVdcJEyrLqOmPIfjU8lEepw3NRwvIVOKSqFmlst6MZaIRWHXmxAN9l9WXO4EqbRm4Plp0RC0b4Fga07N3BNxthWXuTG4ZDQG4W6I9oyiTUbOKRCBagHqvW63TcCRQCewQlWnDb716DKRBKoXVd09PtUZjPLNu19jxZY9xzIOnFXL+w+awUGzavt5TWUlfqbVlFIRSGdYMkt07XRClZMnfYHKKS78fbQ8xGwTi8DONyEegUAVBGqcl+bP4GEi3O281niOstb7i92E7pJyJ07pXOt4zHmH4W7X9RvqSK9bUvypBVnj6Xmbg+EPpL6ukR7L+D8UBSJQO3Gh4IuBW1V1HxHxAW2qWlDFhSaiQAHE4sraHZ0EI3FC0Rg/uf9NHl/dP8hy7qRyzjhgBkctaugXMFFZWkRjVYDK0RKqnlYv8m8YIlVU6sLbfSlsLSrNLCdhoRKLum7JsSqy6RCPQU+L65ZMJTQlVV5i49qBb4Tdze5hZzhiUlTqJqan6v6N9MCuNe4hwUhNgQjUzUA1bm7SPar6DRHZF7hdVZeMyLosM1EFCiAcjbNmRyfRmPu7rtrWzt9f3MxTa3eRnAWprNjPOxc1cMLeU9h7atUeXlVZiZ/GqgA1ZaPQ9ZepSJVUuryAlqh2/BHudkIV6nBja2X16T9oREPQsj6zKFLxQcNegx8jGnIiFes3/dKAghGoAC7KLgLcrKpRETkWmKqqt47IuiwzkQUKoCccY82Ozj2q7m5rC3LHy5u5f+V2gpH+3SnTako5Ye8pHL/X5D1qTAWKfdSWFyMIIn1pYUUEvwh+v1DkE4r9Pvy+ETzlpyNSgWrXbTfWxpOM0UPVBYl0bktv/bp5TgiHIhZxUwBG2pUIriu0ZqYTvfHQfZhPgRKRW4C7gX+ravOIrBglJrpAAbQHI6zf2f+fqTMY5d8rtnH/yu1sbu0/WC3AAbNqOWHJZA6fPymjOVMiLvt6WbGf8oCfipIiSot96UcJphIpX5FLZFs+aeAKwYaRTKjTFeMcbFpD5ZTMaprFY9D8FoQ7hm9XoNqlEvP5XbfmeJjYnmeBOhs4DXgXsAYnVner6osjsiiHmEA5dnaG2NqaIgoMF1Tx+rYO7l/VxGNv7qA73P9JrrzEzzsXNnDcksksnFxJoCjzEG0Rt5/KQBEVgSLKS/yDC1avSAUSxh3G8/iLkTvicejeCZ3b3fywRALVLvdkpr+teNwJ33CiI8sbnOeUeMzWDd6UgDFMgXTxCS7p62nAqcBU4N84wbpXVUchZjg9TKD6aOoIsqszvHtMKhXBSIyn1+7igVVNvLyxdcBOtsaqADNqy3a/5jdWsGRqdUZdeyJQGSiisrSIykBRym01FiMuPmJxJaZKLObeU/1UfQJ15SX4RtK9aIxv4jEnUl07XMSfP+Dqm41kTlwk6ESqpyX1VIBkqqZD1ZQUtsVh5+vp7aNQKQSBSrHDqcDJOME6Bvi6qv56RFZmCROo/nSForT1RGjriQwqVjs6Qjz0ehMPrNzOlrah/2kqA0W8bU4dh86t5+A5daMX/ZdAoNjHjNqy0Q2RN8Ye0bAbm6poHHj+2HCI9DihCnWQ8imqaopLAzbY9jteZ8xOrC5EgUrauR83R6og0oibQA1OdzhKRzBKMBKjJxIjEu3/G1BVVm3r4IGV23llcxvb24P9ogCT8fuEfaZVs2hKJTNqy5heW8bMunKqS4tGJVvFpMoSplSXjixYwzDyQddOaNuYellJlYsgzGeasMHIoUCl/cgpIvXA/wIHAnuEUanq0UBBiJMxNOUlRZSX9P3po7E4PZ5YdYVidIWigLD3tGr2nubS90Ricba1B9nS2sPmlh42NHfz0sZWdnX1/dPE4sorm9t4ZfOeGbkrAv49ugZn1JUzo7aUaTVlWU1cu6szTHswwozaMqpymbndMLJNRYNLEtybzb6ozI3BltW5CcSqLvNKd7PrWpwg+RYz6RP5AxAA/gRkIdbSKBSK/D6q/D53U69y6ZM6wy4jRWcoSigSp9jvY1ZdObPqyl1dZZyXtXZnF8++1cyz65pZ3ZR6GLIrFOON7Z28sb3/8qIMx6+S8fuEfabX8L4DZ3DAzBoiUVi3s5vygJ+yYj+lxe49UOSzcSqjsKmZDUVNTpSSuyBFvGwiVRCf5UQq1J66S3EckUmQRDvQmKpYYKFhXXzZJRSN0Rl03YKdoeiA/xO7OkMs39LO5pZuNrf2sLm1hy2tQXoiozPXY15DBe87cAbvXNRAcVKmdhEIFPmoryihvmIUE+QahjEk2Yjiexz42Fiox2QClTvicaXLG8Nq6Q4TH6KnQVVp7gqzpbWHTa097r3FvW9LY1xrOEyqKOGU/aax15QqZtSWMamyBF+CIAWKfUypLh2dLBmGYQzJiMeggAeBf4vIjcAe07RV9YYR2meMEXw+oaq0mKrSYqZUl7KrM8SOztCAQiUiTKoMMKkywH4za/dYFosr8TQfkAZabUdHiLte2cL9K7cTijojdnWFueXp9bvXKSnyMb2mlBl15SyZWsUxixoJReKUB1yC3MTxOMMwCodMPKiHBlikqnp89kwaOeZBjS6xuLKzM8SOjlDeusQ7ghH+tXwbd72yhdbuwRN7+gTeNqeOE5ZM4dB59dRVlFDhjVmVl6Sem2UYRu7IaZh5oWEClR+isTjNXWFC0XjfJNt432s0fmqRWJzH3tzBq5vb2NzixsHag9EB168KFHHM4kYOnlPHkqlVVJUWU1Lko7zE328cC9xYVonfR1mJC7ywsSzDGDlZFSgvq8Tu/8x0KuqOJiZQhUvc69ZTcO8D/PziqkTjSjxR4FKsp+rmdXWHYwPuqyMYYXNrD2/t7OKRN3b0q5OVyMy6MpZMrWLJ1Gpm1pXtMXYF7kdfX1lCQ2UAv08oK+mLFiz2CyVFPop9FjFoGJmQjSCJGcDPgaOB2sRlqprFOtojxwRq4qGqdIfdHK7O0OCCtbWthwdWNfHgqiZ2dAwvKLXE72N6benuCcnTalzQRbU3PldXUUxdRQnVpcXUlBVTVlJQ/yKGUVBkI0jiGtz8pxOAR3BCdQUuF59h5BURocJLSDvZa4vG4kRiSiQeJxKNE47FCUXi+H3lfPSwOXz40Nm8uqmNZ9c1s2pbO2t2dBFLM6wwHIuzblc363YNPCVQgPqKEmbUlTGrrpyFkytZMq2KpdOqmVFbRlGKLsShUO3zJlN1QRrGeCITD2oXMFtVu0SkVVVrvewST1rBQmOsEY562TPCvRk0XOqn1U2drNrWwapt7SmDLWJxpakjRFvPyCqsFvuFaTVlzKorY2Z9ObPry2msDKScjBwo9rls8CVFlBb7d3c79o6VVQaKKA/4h5Vt3jAKgWx4UDGgd7S5VUQagXZcGXjDGFOUFPkoKfLtngulqvREYsyZVMHh8yfRFY4OOserMxj1JiN3s7k1SFN7kPZglI5ghA7vvStFCZNeIjFlQ3M3G5q7YU365RZ8AhWBIurKS5xHNrWKvadWM6u+vE/IAkVUmGAZ44BMBOoZXJmNvwH3ALcBPYC5KsaYR0R25yjsrSgcjaVWqGjcBXAsmVZFJB4nGtPdc7pU2R0EEo7G2N4eYlOLy1+4qaWbjV4ew+au4SX+jCueAEbZ0NzNg6uaACgr9rPX1Cr2nV7N2+fWM6+hgpJiHxUlRbtFq6TIugSNsUUmXXy1gE9Vm0WkDPgfoAr4iapuzZ2JmWNdfEah0xGM8Pq2DlZu7WB1Uwdv7exKGQ7fG/zREYzSHoykLCyZiobKAG+fW8eh8+rZf0btbo+xItBXPNLGsIxCweZBGUYBE48r4QE8tlAkTlc4SreXwLc9GGVbW5BV29pZudWNl7UMMjm5tNjHkqnV7O2Fzy+eWrW7WGSRX/D7BL/I7u/lJUVUBYosVN4YNbJRbiMAXAacA0xS1RoReTewWFV/nj1TDWPi4fMJpQNUeC0t9lNT7sbK4nGlOxKjOxTlsPn1dIdjRKJxtneEeG1LO8+vb+H5Dc10hfo8rWAkzksbW3lpYyvgogtn1ZezoLGC2vISqkqLvPB4995YFaCxKuDC5stcu3lbRj7IpIvvl7iAiO8A//Ki+Gbgyr3vk8b2JwM/BfzAdar6naTlAeB3wNuAXcCHVHWdt+wS4JO4QI3Pq+o9gx3LPChjIhGJxekOu4jEzlCEjp4oK7e28+y6Zp59qzmtysjJFPuF6TW9RSfdq9g/Mo/K73PZ5CdVBJhUWUxDVYD68pKMPDW/yLDC843CJhsTdbcCC70w82ZVrffaW1W1doht/cAbwLuATcBzwDmq+lrCOp8G9lfVi0TkbOD9qvohEVkK/BE4FJgO3I/z2gbsjDeBMiYywUiM9p4IrT0RQpE429uDLnR+azsrt7Xz1s6unGSRHw5+n1CSgeAU+WX35Ofa8mLqK0qoKy/JKACkMlBEQ2UJteWu9Eptudtf0Qirwk5kygN+qkdQJDQbYebh5PW9UPN0YmQPBVar6lpvu1uBM4DXEtY5AzfxF+B24OdeSqUzgFu9OlRvichqb39PZWC7YUwYSr3US5OrSwlGYkyuDjC/sYLjljQSj0NPOMabTR1sbQvSnhAW394Tpa0nwvb2IK0jnOeVLrG40hPPoF5YxEUxbm7tyZ1RRsZc8M55fO20pVnfbyYC9WfgtyLyRQARmQb8BLg1jW1nABsTvm8CDhtoHVWNikgbMMlrfzpp235zr0TkQuBCgNmzZ6dhkmGMf3rFqpdQNEYwHGf2pHJC0RhRL89hJBbfY95XZyjKFq/o5ObWHpcSaoReVyQe3x2N2CuKwUhBpfE0CoxMBOqrwHeBV4Fy4E3gN8CVObArY1T1WuBacF18eTbHMAqSQJGbwFtD/+4Y9RL09iLU9X3OQtZ2TUoSrAo94egexxyKnkiM1u4IbT0RWrvDtHif42nuQ/G6QIMRWrsjNHeFd+8vdTpiIx0qA7kp/pm2QKlqGPgi8EWva2+nph+jvhmYlfB9pteWap1NIlIE1OC6D9PZ1jCMESIiIw6EGOII/VqGk0R3zqRs2GKMBYYcFRSR2ckvoAyYlfB9KJ4DFonIPBEpAc4G7kxa507gY97nM4EHPQG8EzhbRAIiMg9YBDyb3ukZhmEYY5V0PKh19PU+p3q8Ulzo+IB4Y0qfxaVI8gM3qOoKEbkKWKaqdwLXAzd7QRDNOBHDW+9PuICKKPCZwSL4DMMwjPHBkGHmIvIizmP6LXALsCV5nUITDAszNwzDGDsMFGY+ZBefqh6E63KrB57A1X86GyhR1VihiZNhGIYxPkhrZpqqLlfVLwFzgR8B7wG2isjBObTNMAzDmMBkOnV6EXAMcATwItCSdYsMwzAMgzSCJLyquefgIuyqgJuBo1V1Q45tMwzDMCYw6UTxbQHewglTb0aHhSKysHcFVX0wB7YZhmEYE5h0BGobUApc4L2SUWB+No0yDMMwjHFZsFBEdgDrR7ibBmBnFswpVOz8xjZ2fmOf8X6OmZzfHFVtTG4clwKVDURkWaq4/PGCnd/Yxs5v7DPezzEb52cFUAzDMIyCxATKMAzDKEhMoAbm2nwbkGPs/MY2dn5jn/F+jiM+PxuDMgzDMAoS86AMwzCMgsQEyjAMwyhITKBSICIni8jrIrJaRC7Otz0jRURuEJEmEVme0FYvIveJyJvee91g+yhkRGSWiDwkIq+JyAoR+W+vfVyco4iUisizIvKyd35Xeu3zROQZ73d6m1cMdMwiIn4ReVFE/uF9HzfnJyLrRORVEXlJRJZ5bePi9wkgIrUicruIrBKRlSJyRDbOzwQqCRHxA78ATgGWAueIyNL8WjVibgJOTmq7GHhAVRcBD3jfxypR4H9UdSlwOPAZ7282Xs4xBByvqgcABwIni8jhwHeBH6vqQlzi5k/mz8Ss8N/AyoTv4+38jlPVAxPmBo2X3yfAT4F/q+oS4ADc33Hk56eq9kp44TK135Pw/RLgknzblYXzmgssT/j+OjDN+zwNeD3fNmbxXO8A3jUezxEoB14ADsPN0i/y2vf43Y61FzDTu4kdD/wDV717PJ3fOqAhqW1c/D6BGly+Vsn2+ZkH1Z8ZwMaE75u8tvHGFFXd6n3eBkzJpzHZQkTmAgcBzzCOztHr/noJaALuA9YAraoa9VYZ67/TnwBfBuLe90mMr/NT4F4ReV5ELvTaxsvvcx6wA7jR66K9TkQqyML5mUAZqHvEGfPzDUSkEvgL8AVVbU9cNtbPUV316gNxnsahwJL8WpQ9ROQ9QJOqPp9vW3LIUap6MG7o4DMicnTiwjH++ywCDgZ+pa4CexdJ3XnDPT8TqP5sBmYlfJ/ptY03tovINADvvSnP9owIESnGidPvVfWvXvO4OkcAVW0FHsJ1edWKSG9FgrH8O30H8F4RWQfciuvm+ynj5/xQ1c3eexPwN9xDxnj5fW4CNqnqM97323GCNeLzM4Hqz3PAIi+CqAQ4G7gzzzblgjtxRSjx3u/Ioy0jQkQEuB5Yqao/Slg0Ls5RRBpFpNb7XIYbX1uJE6ozvdXG7Pmp6iWqOlNV5+L+3x5U1Y8wTs5PRCpEpKr3M/BuYDnj5PepqtuAjSKyl9d0AvAaWTg/yySRAhE5Fdcn7gduUNVv5deikSEifwSOxaW/3w5cDvwd+BMwG1ea5IOq2pwnE0eEiBwFPAa8St8Yxldx41Bj/hxFZH/gt7jfow/4k6peJSLzcR5HPfAi8FFVDeXP0pEjIscC/6uq7xkv5+edx9+8r0XAH1T1WyIyiXHw+wQQkQOB64ASYC3wcbzfKiM4PxMowzAMoyCxLj7DMAyjIDGBMgzDMAoSEyjDMAyjIDGBMgzDMAoSEyjDMAyjIDGBMgZERG4SkW/m6dgiIjeKSIuIPJti+fki8ng+bEuwYbaIdHoJhrO972tE5OvZ3m+GNqzwwr5H85iNXkbssizv950i8no29zkaiMj+IvJkvu3IFyZQYwgvZX+TN9mvt+0/ReThPJqVK47CTUidqaqH5tsY2H39T+z9rqobVLVSVWPZPpaqXqSq3/COe6yIbMr2MRJJ9TCiqvuo6sO5PG4KLgZuUtWekexERFREFvZ+V9XHVHWvwbYZwbFy9iCnqq8ArSJyei72X+iYQI09/LiyBGOKYXgZc4B1qtqVC3smEgnpggoaEQngMg7ckm9bRpM0/jd+D3xqNGwpOPKdqt1eGaW1X4d7wmwGar22/wQe9j7PxSVkLErY5mHgP73P5wNPAD8GWnEzvo/02jficmV9LGHbm4BrcNmzO4BHgDkJy5d4y5pxqfU/mLTtr4C7cckjT0xxPtNx6VCagdXABV77J4EgEAM6gStTbNt7Lj8H2oBVwAlD7dtbFsBlCtnivX4CBLxlDbhyD63eto/hHuRuxmWp6PFs+nLy9fau9Tc8uzqAe0kosQCch5tRvwv4uvf37HddEq7fN4EK75hx77id3rn5cL+FNd7+/gTUJ/0OPglsAB712v+MyyrdBjwK7OO1XwhEgLC3/7sSfm8npnHNjsXlY/sf3G9oK/DxhHM5FZf6pgOXT+9/Bzjno4HVSW01uDRWW71tvwn4vWULcb/JNlxpjtu89ke98+/yzudDvTYm/S99CXjFW+96XLbtf3l23g/UJayf6bXb2/s9tAIrgPcO9r8x2DXCZXHv6b3eE+mVdwPslcEfy7thAH8Fvum1ZSpQUVwaEr/3z74BV6AxgMsR1gFUeuvf5H0/2lv+U+Bxb1kFTtQ+jkvfcpB3k1iasG0bLhGoDyhNcT6PAr8ESnGF+HbgCvP12vr4INei91y+CBR7N6E2+m7Sg+37KuBpYDLQCDwJfMNb9m2cKBd7r3fSl3FlHQmCkny9vWu9BlgMlHnfv+MtW4q7gR2FSwfzA9yNbVCB8j4fS8LN1Wv7b+8cZnp/m18Df0yy63fe36nMa/8EUEWf2LyU6njJv7c0rtmx3t/iKu+anQp0493gceLyTu9zHXDwAOf8GeCfSW1/886twjv2s8CnvGV/BL6G9/vCZQzv3U6BhQnf97iG3rk9jROlGThhfQH3Oy4FHgQuT1g/7WvnXYPVuHRbJbjktx3AXgP9bwx1jYB2YP9834NG+2VdfGOTy4DPiUjjMLZ9S1VvVDduchsuc/tVqhpS1XtxT4ILE9b/p6o+qi4H2teAI0RkFvAeXBfcjaoaVdUXcdnEz0rY9g5VfUJV46oaTDTC28c7gK+oalBVX8Ll8jovg3NpAn6iqhFVvQ3nxZ2Wxr4/4p1zk6ruAK4EzvWWRXDF1eZ4+31MvTtEmtyoqm+oG0P5E04cwSU9vUtVH1fVMO5vOJI8YxcBX1PVTd7f5grgzKTuvCtUtcuzBVW9QVU7EtY/QERq0jzeYNcM3HW7yrtmd+PEeK+EZUtFpFpVW1T1hQGOUYu7kQMgIlNwYvcF7zyacN7/2Qn7nQNM9/7OmQbN/J+qbleXafwx4BlVfdH7rf4NJ1ZAxtfucKAS93ASVtUHcV75OQnrJP9vDHWNOrzrM6EwgRqDqOpy3A9+OCWityd87r1xJbdVJnzfXbxRVTtx3V7TcTeGw0SktfeFu4lNTbVtCqYDzarakdC2nsyK0m1OEo/13n6H2vd073vydgDfxz393isia0Uk02u8LeFzN33Xcjp7XstuXNfccJkD/C3h2q/EdYkmFoXbfTyv4OF3RGSNiLTjPAhwXZrpMNg1A9ilfcUFYc9z/wBOaNaLyCMicsQAx2jBeSm9zMF5I1sTzvPXOE8KXDerAM96EYefSPNcekn+3af8PxjGtZsObFTVeEJb8m87+X9jqGtUhesunFCYQI1dLgcuYM8ffW9AQXlCW6JgDIfdtbG8goD1uDGIjcAjqlqb8KpU1f9K2HYwD2ELUN9bhsBjNpnV/JnhldpI3L53jGSwfW/B3fySt8N7Sv4fVZ0PvBf4fyJyQhrnMxRbcd1xwO6yGZPS3DbVcTcCpyRd/1LPG0i13YeBM3BdxDW4bkBwN/iBjpHIgNdsSONVn1PVM3DC8necZ5mKV3Ddo71sBEK4cbzec6xW1X28/W5T1QtUdTouiOCXiZF7WSTTa7cFmCUiiffX5N/2HtsMdo1EZAauq3DMhcmPFBOoMYqqrsZ10X0+oW0H7p/go95T3yeABSM81KkicpRXG+sbwNOquhHnwS0WkXNFpNh7vV1E9k7T/o24cYxvi0ipV1Lik2QWwTUZ+Lx37LNwA9N3p7HvPwKXenNuGnDdbbeAq+4qIgs94WvDeSW9T8LbgfkZ2JfI7cDpInKkdy2voO8GNxTbgUlJXUrXAN8SkTme3Y0icsYg+6jC3ex34R5grk5xjMHObcBrNhgiUiIiHxGRGlWN4MZS4gOs/iyuSOEMAHXlwu8Ffigi1SLiE5EFInKMt++zRKRX9FtwN/1s/K2SyfTaPYPzIL/s/TaPBU7HlQ7pRxrX6BhcjawxV2pkpJhAjW2uwg0eJ3IBLjppF7AP7kY9Ev6A89aagbcBHwXnaeCCKs7GPTFuA76LG0ROl3NwT6NbcH3+l6vq/Rls/wywCBec8S3gTFXt7TYbbN/fBJbhnthfxQ2O985jWYSL4OoEngJ+qaoPecu+jbtJt4rI/2ZgJ6q6Avgc7ia11dt/E+7GN9S2q3ACsdY79nRcwMqduK7IDtyA/2GD7OZ3uG6mzbhosaeTll+PGwNpFZG/p9h+sGs2FOcC67zusYtwXcH98MbmbsL7jXmch/MeXsOJ0O24MUKAtwPPiEgn7lr8t6qu9ZZdAfzWO58PpmnnQGR07bzzOB1X3n0nLljnPO/vOBCDXaOP4B5IJhxWD8ow8oDXXdoKLFLVt/JsTsHgBf48BhykI5ysOx7wvP9fq+pA43bjGhMowxglvGwAD+C69n6I83gOzjBK0DAmDNbFZxijxxn0BXEsAs42cTKMgTEPyjAMwyhIzIMyDMMwChITKMMwDKMgMYEyDMMwChITKMMwDKMgMYEyDMMwCpL/D+tC7mcGCsXLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "n_estimators = 60\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(6, 4))\n",
    "m = cv_results['train-error-mean']\n",
    "s = cv_results['train-error-std']\n",
    "ax.fill_between(range(n_estimators), m + s, m - s, alpha=0.2)\n",
    "ax.plot(range(n_estimators), m, linewidth=3)\n",
    "\n",
    "m = cv_results['test-error-mean']\n",
    "s = cv_results['test-error-std']\n",
    "ax.fill_between(range(n_estimators), m + s, m - s, alpha=0.2)\n",
    "ax.plot(range(n_estimators), m, linestyle='--', linewidth=2)\n",
    "\n",
    "ax.set_xlabel('Number of boosting iterations (estimators)', fontsize=12)\n",
    "ax.set_ylabel('Mean/std of 10-fold classification error', fontsize=12)\n",
    "ax.legend(['train', 'test'], fontsize=12)\n",
    "\n",
    "fig.tight_layout()\n",
    "# plt.savefig('./figures/CH06_F15_Kunapuli.png', format='png', dpi=300, bbox_inches='tight')\n",
    "# plt.savefig('./figures/CH06_F15_Kunapuli.pdf', format='pdf', dpi=300, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.2 Early Stopping\n",
    "Early stopping with ``XGBoost`` works pretty similarly to LightGBM, where we specify a value for the parameter ``early_stopping_rounds``. As long as the overall score (say accuracy) improves over the last ``early_stopping_rounds``, ``XGBoost`` will continue to train. However, when the score does not improve after ``early_stopping_rounds``, ``XGBoost`` stops.\n",
    "\n",
    "As with ``LightGBM``, ``XGBoost`` requires a validation set\n",
    "\n",
    "**Listing 6.7**: Early Stopping with XGBoost\n",
    "\n",
    "_NOTE: This listing is slightly modified from the print version as XGBoost has deprecated the usage of certain parameters such as_ ``eval_metric`` _and_ ``early_stopping_rounds`` _in the_ ``fit()`` _function. These parameters are now initialized in the constructor instead, for better compatibility with scikit-learn._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-auc:0.95480\n",
      "[1]\tvalidation_0-auc:0.96725\n",
      "[2]\tvalidation_0-auc:0.96757\n",
      "[3]\tvalidation_0-auc:0.99017\n",
      "[4]\tvalidation_0-auc:0.99099\n",
      "[5]\tvalidation_0-auc:0.99181\n",
      "[6]\tvalidation_0-auc:0.99410\n",
      "[7]\tvalidation_0-auc:0.99640\n",
      "[8]\tvalidation_0-auc:0.99476\n",
      "[9]\tvalidation_0-auc:0.99148\n",
      "[10]\tvalidation_0-auc:0.99050\n",
      "[11]\tvalidation_0-auc:0.99050\n",
      "[12]\tvalidation_0-auc:0.98985\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>XGBClassifier(base_score=0.5, booster=&#x27;gbtree&#x27;, callbacks=None,\n",
       "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
       "              early_stopping_rounds=5, enable_categorical=False,\n",
       "              eval_metric=&#x27;auc&#x27;, gamma=0, gpu_id=-1, grow_policy=&#x27;depthwise&#x27;,\n",
       "              importance_type=None, interaction_constraints=&#x27;&#x27;,\n",
       "              learning_rate=0.300000012, max_bin=256, max_cat_to_onehot=4,\n",
       "              max_delta_step=0, max_depth=2, max_leaves=0, min_child_weight=1,\n",
       "              missing=nan, monotone_constraints=&#x27;()&#x27;, n_estimators=50, n_jobs=0,\n",
       "              num_parallel_tree=1, predictor=&#x27;auto&#x27;, random_state=0,\n",
       "              reg_alpha=0, reg_lambda=1, ...)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">XGBClassifier</label><div class=\"sk-toggleable__content\"><pre>XGBClassifier(base_score=0.5, booster=&#x27;gbtree&#x27;, callbacks=None,\n",
       "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
       "              early_stopping_rounds=5, enable_categorical=False,\n",
       "              eval_metric=&#x27;auc&#x27;, gamma=0, gpu_id=-1, grow_policy=&#x27;depthwise&#x27;,\n",
       "              importance_type=None, interaction_constraints=&#x27;&#x27;,\n",
       "              learning_rate=0.300000012, max_bin=256, max_cat_to_onehot=4,\n",
       "              max_delta_step=0, max_depth=2, max_leaves=0, min_child_weight=1,\n",
       "              missing=nan, monotone_constraints=&#x27;()&#x27;, n_estimators=50, n_jobs=0,\n",
       "              num_parallel_tree=1, predictor=&#x27;auto&#x27;, random_state=0,\n",
       "              reg_alpha=0, reg_lambda=1, ...)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', callbacks=None,\n",
       "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
       "              early_stopping_rounds=5, enable_categorical=False,\n",
       "              eval_metric='auc', gamma=0, gpu_id=-1, grow_policy='depthwise',\n",
       "              importance_type=None, interaction_constraints='',\n",
       "              learning_rate=0.300000012, max_bin=256, max_cat_to_onehot=4,\n",
       "              max_delta_step=0, max_depth=2, max_leaves=0, min_child_weight=1,\n",
       "              missing=nan, monotone_constraints='()', n_estimators=50, n_jobs=0,\n",
       "              num_parallel_tree=1, predictor='auto', random_state=0,\n",
       "              reg_alpha=0, reg_lambda=1, ...)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a single split of training and validation sets\n",
    "Xtrn, Xval, ytrn, yval = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)\n",
    "ens = XGBClassifier(n_estimators=50, max_depth=2, objective='binary:logistic', eval_metric='auc', early_stopping_rounds=5)\n",
    "ens.fit(Xtrn, ytrn, eval_set=[(Xval, yval)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}