Pytorch/TabNet in Pytorch - New Solution to Tabular ML problems.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TabNet in Pytorch - New Solution to Tabular ML problems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tabnet has basic GLU (Gated Linear Unit) Activuation\n",
    "### with shared layers "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter to Tune at https://github.com/dreamquark-ai/tabnet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It outperfomrs in many tree-based boosting algorithms !!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![a](https://image.slidesharecdn.com/tabnetpresentation-200606103121/95/tabnet-presentation-7-638.jpg?cb=1591439795)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Arc2](https://image.slidesharecdn.com/tabnetpresentation-200606103121/95/tabnet-presentation-8-638.jpg?cb=1591439795)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Advantage:\n",
    "1. We can stack as much as we want (More steps, bigger the model)\n",
    "2. All steps contribute equally to the final layer (depending on the problems like regression or classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Explanability](https://image.slidesharecdn.com/tabnetpresentation-200606103121/95/tabnet-presentation-11-638.jpg?cb=1591439795)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Has a robust explanability, and it's sk-learn compatible!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## let's dive into examples!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pytorch Version: 1.5.0\n"
     ]
    }
   ],
   "source": [
    "from pytorch_tabnet.tab_model import TabNetClassifier, TabNetRegressor\n",
    "\n",
    "import torch\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.metrics import accuracy_score, mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import os\n",
    "import math\n",
    "import random\n",
    "import pathlib\n",
    "import json\n",
    "import wget\n",
    "from pathlib import Path\n",
    "import shutil\n",
    "import gzip\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "print('Pytorch Version:', torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (1) Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.data.gz\"\n",
    "dataset_name = 'forest-cover-type'\n",
    "tmp_out = Path('./data/'+dataset_name+'.gz')\n",
    "out = Path(os.getcwd()+'/data/'+dataset_name+'.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading file...\n",
      "100% [........................................................................] 11240707 / 11240707"
     ]
    }
   ],
   "source": [
    "out.parent.mkdir(parents=True, exist_ok=True)\n",
    "if out.exists():\n",
    "    print(\"File already exists.\")\n",
    "else:\n",
    "    print(\"Downloading file...\")\n",
    "    wget.download(url, tmp_out.as_posix())\n",
    "    with gzip.open(tmp_out, 'rb') as f_in:\n",
    "        with open(out, 'wb') as f_out:\n",
    "            shutil.copyfileobj(f_in, f_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (2) Train/Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = \"Covertype\"\n",
    "\n",
    "categorical = [\n",
    "    \"Wilderness_Area1\", \"Wilderness_Area2\", \"Wilderness_Area3\",\n",
    "    \"Wilderness_Area4\", \"Soil_Type1\", \"Soil_Type2\", \"Soil_Type3\", \"Soil_Type4\",\n",
    "    \"Soil_Type5\", \"Soil_Type6\", \"Soil_Type7\", \"Soil_Type8\", \"Soil_Type9\",\n",
    "    \"Soil_Type10\", \"Soil_Type11\", \"Soil_Type12\", \"Soil_Type13\", \"Soil_Type14\",\n",
    "    \"Soil_Type15\", \"Soil_Type16\", \"Soil_Type17\", \"Soil_Type18\", \"Soil_Type19\",\n",
    "    \"Soil_Type20\", \"Soil_Type21\", \"Soil_Type22\", \"Soil_Type23\", \"Soil_Type24\",\n",
    "    \"Soil_Type25\", \"Soil_Type26\", \"Soil_Type27\", \"Soil_Type28\", \"Soil_Type29\",\n",
    "    \"Soil_Type30\", \"Soil_Type31\", \"Soil_Type32\", \"Soil_Type33\", \"Soil_Type34\",\n",
    "    \"Soil_Type35\", \"Soil_Type36\", \"Soil_Type37\", \"Soil_Type38\", \"Soil_Type39\",\n",
    "    \"Soil_Type40\"\n",
    "]\n",
    "\n",
    "integer = [\n",
    "    \"Elevation\", \"Aspect\", \"Slope\", \"Horizontal_Distance_To_Hydrology\",\n",
    "    \"Vertical_Distance_To_Hydrology\", \"Horizontal_Distance_To_Roadways\",\n",
    "    \"Hillshade_9am\", \"Hillshade_Noon\", \"Hillshade_3pm\",\n",
    "    \"Horizontal_Distance_To_Fire_Points\"\n",
    "]\n",
    "\n",
    "feature_columns = (\n",
    "    integer + categorical + [target])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "581012\n"
     ]
    }
   ],
   "source": [
    "train = pd.read_csv(out, header=None, names=feature_columns)\n",
    "train.head()\n",
    "\n",
    "n_total = len(train)\n",
    "print(n_total)\n",
    "\n",
    "# Train, val and test split follows\n",
    "train_val_indices, test_indices = train_test_split(\n",
    "    range(n_total), test_size=0.2, random_state=0)\n",
    "\n",
    "train_indices, valid_indices = train_test_split(\n",
    "    train_val_indices, test_size=0.2 / 0.6, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (3) Pre-Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_columns = []\n",
    "categorical_dims =  {}\n",
    "for col in train.columns[train.dtypes == object]:\n",
    "    print(col, train[col].nunique())\n",
    "    l_enc = LabelEncoder()\n",
    "    train[col] = train[col].fillna(\"VV_likely\")\n",
    "    train[col] = l_enc.fit_transform(train[col].values)\n",
    "    categorical_columns.append(col)\n",
    "    categorical_dims[col] = len(l_enc.classes_)\n",
    "\n",
    "for col in train.columns[train.dtypes == 'float64']:\n",
    "    train.fillna(train.loc[train_indices, col].mean(), inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (4) Define categorical features for categorical embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "unused_feat = []\n",
    "\n",
    "features = [ col for col in train.columns if col not in unused_feat+[target]] \n",
    "\n",
    "cat_idxs = [ i for i, f in enumerate(features) if f in categorical_columns]\n",
    "\n",
    "cat_dims = [ categorical_dims[f] for i, f in enumerate(features) if f in categorical_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (5) Create a Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device used : cuda\n"
     ]
    }
   ],
   "source": [
    "clf = TabNetClassifier(\n",
    "    n_d=64, n_a=64, n_steps=5,\n",
    "    lr=0.02,\n",
    "    gamma=1.5, n_independent=2, n_shared=2,\n",
    "    cat_idxs=cat_idxs,\n",
    "    cat_dims=cat_dims,\n",
    "    cat_emb_dim=1,\n",
    "    lambda_sparse=1e-4, momentum=0.3, clip_value=2.,\n",
    "    optimizer_fn=torch.optim.Adam,\n",
    "    scheduler_params = {\"gamma\": 0.95,\n",
    "                     \"step_size\": 20},\n",
    "    scheduler_fn=torch.optim.lr_scheduler.StepLR, epsilon=1e-15\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (6) Train a Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train[features].values[train_indices]\n",
    "y_train = train[target].values[train_indices]\n",
    "\n",
    "X_valid = train[features].values[valid_indices]\n",
    "y_valid = train[target].values[valid_indices]\n",
    "\n",
    "X_test = train[features].values[test_indices]\n",
    "y_test = train[target].values[test_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_epochs = 1000 if not os.getenv(\"CI\", False) else 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Will train until validation stopping metric hasn't improved in 100 rounds.\n",
      "---------------------------------------\n",
      "| EPOCH |  train  |   valid  | total time (s)\n",
      "| 1     | 0.56079 |  0.22032 |   32.7      \n",
      "| 2     | 0.66891 |  0.24418 |   61.4      \n",
      "| 3     | 0.69981 |  0.15226 |   96.2      \n",
      "| 4     | 0.71060 |  0.15770 |   131.4     \n",
      "| 5     | 0.71834 |  0.16619 |   166.8     \n",
      "| 6     | 0.72683 |  0.20677 |   203.1     \n",
      "| 7     | 0.73037 |  0.18427 |   239.1     \n",
      "| 8     | 0.73657 |  0.21702 |   280.2     \n",
      "| 9     | 0.73895 |  0.25973 |   317.0     \n",
      "| 10    | 0.74255 |  0.30849 |   352.5     \n",
      "| 11    | 0.74675 |  0.38309 |   390.7     \n",
      "| 12    | 0.74869 |  0.40913 |   428.0     \n",
      "| 13    | 0.75236 |  0.50597 |   462.0     \n",
      "| 14    | 0.75527 |  0.55599 |   496.0     \n",
      "| 15    | 0.75886 |  0.58694 |   530.7     \n",
      "| 16    | 0.76262 |  0.61613 |   568.2     \n",
      "| 17    | 0.76595 |  0.67300 |   606.0     \n",
      "| 18    | 0.77195 |  0.70184 |   643.8     \n",
      "| 19    | 0.77594 |  0.72445 |   681.7     \n",
      "| 20    | 0.77722 |  0.73927 |   719.7     \n",
      "| 21    | 0.78065 |  0.74689 |   757.1     \n",
      "| 22    | 0.78209 |  0.76143 |   794.3     \n",
      "| 23    | 0.78187 |  0.76658 |   832.2     \n",
      "| 24    | 0.78596 |  0.78265 |   869.9     \n",
      "| 25    | 0.79321 |  0.79628 |   907.3     \n",
      "| 26    | 0.79918 |  0.80144 |   950.4     \n",
      "| 27    | 0.80229 |  0.80338 |   987.6     \n",
      "| 28    | 0.80510 |  0.81055 |   1025.0    \n",
      "| 29    | 0.80616 |  0.81145 |   1062.6    \n",
      "| 30    | 0.81095 |  0.81672 |   1099.9    \n",
      "| 31    | 0.81402 |  0.81978 |   1136.9    \n",
      "| 32    | 0.81880 |  0.82688 |   1174.7    \n",
      "| 33    | 0.82193 |  0.82416 |   1211.9    \n",
      "| 34    | 0.82329 |  0.83007 |   1249.8    \n",
      "| 35    | 0.82738 |  0.83422 |   1287.5    \n",
      "| 36    | 0.82662 |  0.83504 |   1325.1    \n",
      "| 37    | 0.81632 |  0.81775 |   1362.5    \n",
      "| 38    | 0.81864 |  0.83353 |   1400.4    \n",
      "| 39    | 0.82153 |  0.82960 |   1437.7    \n",
      "| 40    | 0.82536 |  0.83887 |   1475.2    \n",
      "| 41    | 0.83080 |  0.84250 |   1512.4    \n",
      "| 42    | 0.83675 |  0.84585 |   1550.4    \n",
      "| 43    | 0.83838 |  0.84490 |   1587.7    \n",
      "| 44    | 0.84197 |  0.85027 |   1624.9    \n",
      "| 45    | 0.83709 |  0.83093 |   1662.5    \n",
      "| 46    | 0.83164 |  0.84261 |   1699.8    \n",
      "| 47    | 0.84031 |  0.84882 |   1736.7    \n",
      "| 48    | 0.84699 |  0.85345 |   1777.5    \n",
      "| 49    | 0.85042 |  0.85676 |   1818.5    \n",
      "| 50    | 0.85277 |  0.86401 |   1858.6    \n",
      "| 51    | 0.85702 |  0.86235 |   1898.1    \n",
      "| 52    | 0.85887 |  0.86449 |   1936.0    \n",
      "| 53    | 0.86169 |  0.86221 |   1973.5    \n",
      "| 54    | 0.85367 |  0.86279 |   2011.1    \n",
      "| 55    | 0.86234 |  0.87215 |   2048.6    \n",
      "| 56    | 0.86772 |  0.87694 |   2086.2    \n",
      "| 57    | 0.86114 |  0.84975 |   2123.6    \n",
      "| 58    | 0.84709 |  0.82954 |   2161.4    \n",
      "| 59    | 0.82541 |  0.82969 |   2198.7    \n",
      "| 60    | 0.82445 |  0.83110 |   2237.7    \n",
      "| 61    | 0.82976 |  0.83738 |   2275.2    \n",
      "| 62    | 0.84168 |  0.85149 |   2312.4    \n",
      "| 63    | 0.85045 |  0.86303 |   2349.6    \n",
      "| 64    | 0.85670 |  0.85963 |   2387.3    \n",
      "| 65    | 0.84662 |  0.83808 |   2424.6    \n",
      "| 66    | 0.83459 |  0.84749 |   2461.6    \n",
      "| 67    | 0.84967 |  0.86076 |   2499.9    \n",
      "| 68    | 0.85924 |  0.86884 |   2537.1    \n",
      "| 69    | 0.86368 |  0.86680 |   2574.2    \n",
      "| 70    | 0.86624 |  0.87257 |   2611.5    \n",
      "| 71    | 0.86084 |  0.80604 |   2649.3    \n",
      "| 72    | 0.78755 |  0.79039 |   2686.6    \n",
      "| 73    | 0.79845 |  0.81654 |   2731.9    \n",
      "| 74    | 0.81347 |  0.82462 |   2780.3    \n",
      "| 75    | 0.80435 |  0.81170 |   2825.8    \n",
      "| 76    | 0.81315 |  0.82020 |   2869.0    \n",
      "| 77    | 0.76613 |  0.73253 |   2910.5    \n",
      "| 78    | 0.74520 |  0.75580 |   2950.9    \n",
      "| 79    | 0.76391 |  0.77470 |   2990.7    \n",
      "| 80    | 0.77417 |  0.78356 |   3029.0    \n",
      "| 81    | 0.78030 |  0.78679 |   3067.1    \n",
      "| 82    | 0.79052 |  0.80036 |   3104.1    \n",
      "| 83    | 0.79617 |  0.80115 |   3140.5    \n",
      "| 84    | 0.80335 |  0.81539 |   3175.9    \n",
      "| 85    | 0.81257 |  0.81993 |   3211.6    \n",
      "| 86    | 0.81782 |  0.82759 |   3258.2    \n",
      "| 87    | 0.81985 |  0.82836 |   3305.9    \n",
      "| 88    | 0.81448 |  0.81699 |   3351.3    \n",
      "| 89    | 0.81445 |  0.82814 |   3394.7    \n",
      "| 90    | 0.82311 |  0.83207 |   3435.9    \n",
      "| 91    | 0.82845 |  0.83883 |   3476.1    \n",
      "| 92    | 0.83530 |  0.84278 |   3515.4    \n",
      "| 93    | 0.83665 |  0.84330 |   3554.5    \n",
      "| 94    | 0.83847 |  0.84212 |   3592.3    \n",
      "| 95    | 0.84160 |  0.84950 |   3629.2    \n",
      "| 96    | 0.83277 |  0.82186 |   3665.7    \n",
      "| 97    | 0.82517 |  0.83948 |   3701.4    \n",
      "| 98    | 0.83600 |  0.84271 |   3738.0    \n",
      "| 99    | 0.84181 |  0.85051 |   3775.4    \n",
      "| 100   | 0.84552 |  0.85646 |   3813.6    \n",
      "| 101   | 0.84413 |  0.84983 |   3850.8    \n",
      "| 102   | 0.84334 |  0.85278 |   3888.3    \n",
      "| 103   | 0.85102 |  0.85706 |   3925.6    \n",
      "| 104   | 0.85529 |  0.86591 |   3964.0    \n",
      "| 105   | 0.85823 |  0.86616 |   4001.4    \n",
      "| 106   | 0.86250 |  0.87401 |   4039.0    \n",
      "| 107   | 0.86450 |  0.86988 |   4078.1    \n",
      "| 108   | 0.86136 |  0.87007 |   4115.3    \n",
      "| 109   | 0.85649 |  0.85979 |   4152.6    \n",
      "| 110   | 0.85660 |  0.86848 |   4189.8    \n",
      "| 111   | 0.85975 |  0.87195 |   4236.1    \n",
      "| 112   | 0.83871 |  0.81298 |   4285.1    \n",
      "| 113   | 0.77762 |  0.76619 |   4322.9    \n",
      "| 114   | 0.77367 |  0.79184 |   4358.8    \n",
      "| 115   | 0.78503 |  0.79092 |   4395.4    \n",
      "| 116   | 0.78224 |  0.79060 |   4432.9    \n",
      "| 117   | 0.78784 |  0.80241 |   4470.6    \n",
      "| 118   | 0.79801 |  0.80728 |   4508.4    \n",
      "| 119   | 0.80377 |  0.81314 |   4549.2    \n",
      "| 120   | 0.80935 |  0.82154 |   4586.3    \n",
      "| 121   | 0.81842 |  0.82606 |   4623.2    \n",
      "| 122   | 0.82005 |  0.82428 |   4661.2    \n",
      "| 123   | 0.82221 |  0.83111 |   4698.1    \n",
      "| 124   | 0.82709 |  0.83432 |   4735.3    \n",
      "| 125   | 0.83141 |  0.83737 |   4772.9    \n",
      "| 126   | 0.82774 |  0.81675 |   4810.3    \n",
      "| 127   | 0.81730 |  0.82057 |   4848.1    \n",
      "| 128   | 0.82198 |  0.83455 |   4885.6    \n",
      "| 129   | 0.83105 |  0.84259 |   4923.4    \n",
      "| 130   | 0.83048 |  0.82661 |   4960.8    \n",
      "| 131   | 0.81176 |  0.81864 |   4998.4    \n",
      "| 132   | 0.80519 |  0.80451 |   5035.9    \n",
      "| 133   | 0.80869 |  0.82106 |   5073.5    \n",
      "| 134   | 0.80447 |  0.80800 |   5110.9    \n",
      "| 135   | 0.80967 |  0.81942 |   5148.1    \n",
      "| 136   | 0.81861 |  0.82294 |   5185.7    \n",
      "| 137   | 0.82262 |  0.83176 |   5223.1    \n",
      "| 138   | 0.82634 |  0.83813 |   5260.8    \n",
      "| 139   | 0.83047 |  0.84066 |   5298.1    \n",
      "| 140   | 0.83445 |  0.84517 |   5335.5    \n",
      "| 141   | 0.83588 |  0.83788 |   5373.3    \n",
      "| 142   | 0.83382 |  0.84270 |   5410.2    \n",
      "| 143   | 0.83858 |  0.84466 |   5447.1    \n",
      "| 144   | 0.84247 |  0.84404 |   5484.5    \n",
      "| 145   | 0.84592 |  0.85278 |   5521.8    \n",
      "| 146   | 0.84407 |  0.84722 |   5560.1    \n",
      "| 147   | 0.83919 |  0.83735 |   5597.4    \n",
      "| 148   | 0.83654 |  0.84401 |   5635.1    \n",
      "| 149   | 0.83958 |  0.85223 |   5672.5    \n",
      "| 150   | 0.84586 |  0.85073 |   5709.7    \n",
      "| 151   | 0.85107 |  0.86265 |   5748.1    \n",
      "| 152   | 0.84332 |  0.84044 |   5785.5    \n",
      "| 153   | 0.83859 |  0.85237 |   5824.2    \n",
      "| 154   | 0.84708 |  0.86097 |   5862.2    \n",
      "| 155   | 0.84437 |  0.84612 |   5899.6    \n",
      "| 156   | 0.83533 |  0.83891 |   5937.0    \n",
      "Early stopping occured at epoch 156\n",
      "Training done in 5937.047 seconds.\n",
      "---------------------------------------\n"
     ]
    }
   ],
   "source": [
    "clf.fit(\n",
    "    X_train=X_train, y_train=y_train,\n",
    "    X_valid=X_valid, y_valid=y_valid,\n",
    "    max_epochs=max_epochs, patience=100,\n",
    "    batch_size=16384, virtual_batch_size=256\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1c3db2b4788>]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD8CAYAAACINTRsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHKNJREFUeJzt3X2QJHd93/H3d6bncWf37vbhniVOkkGgoHBSDiLABgHGCMVBpuw4EAqLRFiulElB7GAkU5WQP1IBg3lwFcEWIFuxiRwejQLEmCg8GizYU/SIEELScbrTPezt3e3j7Dz0fPNH9572Vre3e7uz2zM9n1fVVM/09G5/77c3n/3td3q6zd0REZF0yCRdgIiItI9CXUQkRRTqIiIpolAXEUkRhbqISIoo1EVEUkShLiKSIgp1EZEUUaiLiKRIsJE7Gx4e9j179mzkLkVEut7+/ftPuPvISrbd0FDfs2cPo6OjG7lLEZGuZ2Y/X+m2ar+IiKSIQl1EJEUU6iIiKaJQFxFJEYW6iEiKKNRFRFJEoS4ikiLdEeqP/i1898NJVyEi0vG6I9Qfvxv+/mNJVyEi0vG6I9TzfVCfSboKEZGOt2yom9lFZvZNM3vEzB42s3fG699nZofN7L74dv26VZnvg1YDmvV124WISBqs5NwvTeD33f1eM+sH9pvZN+LnPuLuH1q/8mL5SrSsT0MwuO67ExHpVsvO1N39iLvfG9+fAh4Bdq13YWfJ90VLtWBERM7rgnrqZrYHuAq4J171DjN7wMxuN7Mtba7tGQp1EZEVWXGom1kF+ALwLnefBD4BXAbsBY4Af7zE191sZqNmNjo2Nra6Ks+0XxTqIiLns6JQN7McUaB/xt2/CODux9w9dPcW8EngJef6Wne/zd33ufu+kZEVneP92Rb21EVEZEkrOfrFgE8Dj7j7hxes37FgszcCD7W/vJjaLyIiK7KSo19eDrwVeNDM7ovX/SHwZjPbCzhwAPiddakQNFMXEVmhZUPd3b8H2Dme+lr7y1nCmZm6Ql1E5Hy65xOloPaLiMgyFOoiIinSHaGeyUJQUvtFRGQZ3RHqoJN6iYisgEJdRCRFuijUKwp1EZFldFGo96mnLiKyjC4Ldc3URUTOR6EuIpIiXRTqFbVfRESW0UWhrpm6iMhyujPUf/I1uOe2ZOsREelAXRTqFWjOQdiE0dvhHz6edEUiIh2ni0I9Pv9LYwamjkCjmmw9IiIdaCXnU+8MhQWXtJs6AmEj2XpERDpQ94T6/IUyZk/C7Dhkuqd0EZGN0n3tl/GfRctWE5r15OoREelAXRjqjz2zrjGbTC0iIh2qC0P98WfW6c1SEZGzdFGoxz31E5qpi4gspYtCfVFPHTRTFxFZpPtCfe70M+sU6iIiZ+meUM/1PXPfstGyoXPBiIgs1D2hHuQhm4/ub744WmqmLiJylu4JdXimBTN0WbTUG6UiImfpslCPj4AZvDRaaqYuInKWLgv1eKY+OD9TV6iLiCzUpaEez9R10QwRkbN0d6hrpi4icpYuC/W4pz6wE4KS3igVEVlk2VA3s4vM7Jtm9oiZPWxm74zXD5rZN8zssXi5Zd2rzfdBcRPky5AraaYuIrLISmbqTeD33f0FwDXA75rZFcAtwN3u/lzg7vjx+tr7r+CV8W5yZYW6iMgiy15pwt2PAEfi+1Nm9giwC7gBuDbe7A7gW8B71qXKeZe9OrpBPFPXG6UiIgtdUE/dzPYAVwH3ANviwJ8P/q3tLu681H4REXmWFYe6mVWALwDvcvfJC/i6m81s1MxGx8bGVlPjueXKeqNURGSRFYW6meWIAv0z7v7FePUxM9sRP78DOH6ur3X329x9n7vvGxkZaUfNkbx66iIii63k6BcDPg084u4fXvDUXcCN8f0bgS+3v7zz0BulIiLPsuwbpcDLgbcCD5rZffG6PwTeD3zWzG4CDgL/Yn1KXEJOx6mLiCy2kqNfvgfYEk+/pr3lXIBcCeoKdRGRhbrrE6ULqf0iIvIsXRzqar+IiCzWxaHeB60GhI2kKxER6RhdHOqlaKkWjIjIGSkIdbVgRETmdXGol6OlQl1E5IwuDnW1X0REFuviUJ+fqSvURUTmdW+o59V+ERFZrHtDXe0XEZFn6eJQj2fqdV0oQ0RkXheHumbqIiKLdXGoq6cuIrJYF4e6ZuoiIot1cajrkEYRkcW6N9QzWcgWoKE3SkVE5nVvqEN8+l3N1EVE5nV5qJf1RqmIyAJdHuqaqYuILNTloa5L2omILNTdoZ5X+0VEZKHuDvVcCeoKdRGReV0e6pqpi4gs1N2hnq9AbSrpKkREOkZ3h3qhH+rTSVchItIxujzUNVMXEVmoy0O9H8I6NGtJVyIi0hG6O9Tz/dGyphaMiAh0e6gX4lCvqwUjIgJdH+qVaKm+uogIsIJQN7Pbzey4mT20YN37zOywmd0X365f3zKXUFD7RURkoZXM1P8CuO4c6z/i7nvj29faW9YKnempa6YuIgIrCHV3/w5wcgNquXDqqYuInGUtPfV3mNkDcXtmS9squhDqqYuInGW1of4J4DJgL3AE+OOlNjSzm81s1MxGx8bGVrm7JainLiJyllWFursfc/fQ3VvAJ4GXnGfb29x9n7vvGxkZWW2d55aPZ+o6VYCICLDKUDezHQsevhF4aKlt11UmC7k+tV9ERGLBchuY2Z3AtcCwmR0C/hNwrZntBRw4APzOOtZ4fjr/i4jIGcuGuru/+RyrP70OtaxOoV+hLiIS6+5PlELUV1dPXUQESEOoa6YuInJGSkJdM3UREUhDqOcrUJtMugoRkY7Q/aGuS9qJiJyRglDXIY0iIvNSEOq6pJ2IyLzuD3Vd0k5E5IzuD3WdfldE5IwUhPr86Xc1UxcRSUGo6+pHIiLzuj/U53vqOqxRRCQFoX5mpq4PIImIpCDU1VMXEZmXglBXT11EZF73h7ouaScickb3h3omC7myZuoiIqQh1EHnVBcRiaUj1PM6qZeICKQl1HX6XRERIE2hPqfj1EVE0hHqla0wfSzpKkREEpeOUB/YBZNPg3vSlYiIJCo9oR7WYPZk0pWIiCQqJaG+M1pOHkq2DhGRhKUk1HdFy8mnk61DRCRhKQn1+Zn64WTrEBFJWDpCvbIVMoFm6iLS89IR6pks9O9QqItIz0tHqEPUglH7RUR63LKhbma3m9lxM3towbpBM/uGmT0WL7esb5krMLATJhTqItLbVjJT/wvgukXrbgHudvfnAnfHj5OlDyCJiCwf6u7+HWDxp3puAO6I798B/Fqb67pwA7ugWYXqqaQrERFJzGp76tvc/QhAvNzavpJW6cxhjXqzVER617q/UWpmN5vZqJmNjo2Nrd+O9AEkEZFVh/oxM9sBEC+PL7Whu9/m7vvcfd/IyMgqd7cCOlWAiMiqQ/0u4Mb4/o3Al9tTzhpUtoFlNFMXkZ62kkMa7wR+AFxuZofM7Cbg/cBrzewx4LXx42RlA30ASUR6XrDcBu7+5iWeek2ba1m7gV1w8smkqxARSUx6PlEKcNFL4PB+aMwlXYmISCLSFeqXvCK6WMahHyZdiYhIItIV6he/FCwLT34n6UpERBKRrlAvDsCuqxXqItKz0hXqELVgDu+H2lTSlYiIbLh0hnqrCQf/IelKREQ2XPpC/aJ/Ctk8PPntpCsREdlw6Qv1XAm2XwlHH0y6EhGRDZe+UIfok6VTx5KuQkRkw6U01LfD9NGkqxAR2XDpDPXK9uhiGc1a0pWIiGyodIZ6/7ZoOa0WjIj0lnSGemV7tFRfXUR6TDpD/cxMXX11Eekt6Qz1itovItKb0hnqfSPRVZDUfhGRHpPOUM9ko2BX+0VEekw6Qx2iFoxm6iLSY9Ib6voAkoj0oPSGumbqItKD0hvq/dth5ji0wqQrERHZMOkN9co28BbMjiddiYjIhkl3qANMqa8uIr0jvaHeH58qQB9AEpEekt5Q10xdRHpQ+kNdhzWKSA9Jb6jnilDcrMMaRaSnpDfUAQZ2wsShpKsQEdkw6Q71rS+A4w8nXYWIyIZJd6hveyGcPghzE0lXIiKyIdYU6mZ2wMweNLP7zGy0XUW1zbYXRstjmq2LSG9ox0z9Ve6+1933teF7tdf2ONSPPpRsHSIiGyTd7Zf+HVAahGMPJl2JiMiGWGuoO/B3ZrbfzG4+1wZmdrOZjZrZ6NjY2Bp3d4HMotm62i8i0iPWGuovd/ergdcDv2tmr1i8gbvf5u773H3fyMjIGne3CtuuhGM/1tkaRaQnrCnU3f3peHkc+BLwknYU1Vbb/hE0q3DyiaQrERFZd6sOdTPrM7P++fvArwCd947kmTdL1VcXkfRby0x9G/A9M7sf+CHwVXf/2/aU1UYjz4dMAMc67/eNiEi7Bav9Qnd/AnhRG2tZH0EBhi+HIw8kXYmIyLpL9yGN83a8CI7cD+5JVyIisq56J9Rnjuvc6iKSer0R6jv3Rssj9ydbh4jIOuuNUN/2QsDgyH1JVyIisq56I9QLFRh+nmbqIpJ6vRHqEPXVn9ZMXUTSrXdCfedemHoapo8nXYmIyLrpnVDfER9SrxaMiKRY74T69iujpVowIpJivRPqxU2w9Qp48ttJVyIism56J9QBLn89/Pz7MHsy6UpERNZFj4X6PwMP4bG/S7oSEZF10VuhvvMqqGyHn3w16UpERNZFb4V6JgPPvx5+djc05pKuRkSk7Xor1CFqwTRm9IapiKRS74X6Jb8UHQnzvY9A2Ey6GhGRtuq9UA8K8PoPwsEfwLc/kHQ1IiJt1XuhDvCifwl73wLf+SAc+F7S1YiItE1vhjrA9R+ETbvhW+9PuhIRkbbp3VDP98GLb4ID34VjP066GhGRtujdUAe46rcgW4AffSrpSkRE2qK3Q71vCK78Dbj/r2FuIulqRETWrCtCvdVyDo7Prs83f/Hbo+PW/9c7oXpqffYhIrJBuiLU3/OFB/jNP/sBc42w/d9819Vw7a3w47vg49fAw18C9/bvR0RkA3RFqP/6P9nN0ck57vj+gfXZwbW3wG/fDX3D8Lm3wR3/HJ74FrRa67M/EZF10hWhfs2lQ7zyeSP8t289zkS1sT472XkV3PxtuP5DcOxh+O83wMdeBA9+fn32JyKyDroi1AH+4LrLmag2+NDXH2Vqbp2CPRvAS34bfu8R+PVPR2+kfuGmaPZ+8B4I12m/IiJtYr6B/eN9+/b56Ojoqr/+3Z+7n8/tP0SQMS7f3s9If4HhSnTbXM7Rl89Sygf05bOUCwHlfJZyPktfPqCQy5DPZsgH8S2bwczOv8OwCX//0egDSq0G5Cuw5xfhklfAyPNh8BLo2xod877c95LO0grh+CP4pl3UcwMUgmzSFYksycz2u/u+FW3bTaEetpwfHTjJtx4d4ydHJzkxXWN8us6J6RqN8ML/HfMhn8vaWWGfD7LkgwyF+PktNs0L6w9wxdy9PL96LyP1Q2d9n2amQDW3hVp+C7O5LVRzW2gUBmmWhqn1X0SrspNssUJQKJEr9pErlCmWKxSKJYq5gGIuSyHIkMnoF8OaHH2QxuRxRg+cpDH4XEZ2XsLuwTL9xRz1ZouJaoOBUsDc+CFm7vzX7Dy9H4DDPsQTmecwXroE6xshPzAM5SGsPES2Mky+vIm+QkC5XKbUP0ilmKNSCCjmVjAx6DI+/gSP3f3nTIw9xbHiLzBd3k3/wAD5YoW6FbB8H8W+CuW+ASrlMpvKefqLAZVCQJBdvz/8w7BFozZLvthHJmO0Wk7LfV332UlSG+pLcXeqjZDZeshsLWSm3mS23mS2HjJTC5mtN6k1W9Tnb2Fr0eOQerNFI3Tqzfi5sEW9GZ7Zfn7bRuhUGuPsDA+xvXWUTa0JtjDJsE0yyCSDNsWQTTLIFGWrLVt7zQPq5KgT0CBHg4CG5WgS0LQcTQtoWp6m5QgtR5jJ0bIcYSZPX/MkI40j1Ak4kRmmkSmSNaOVzdPIFAmDMmG2RJgr0wrKWFAgE+TJBAWy2Szl6tOUqk8z53lmMhXC/CYobiZbqpAv9pEr9pMr9xHkK1i+RBBkyWUzBBkjyGbIxb+EpmpNmp6hPLCFgVKBgVJAYMbM0Z8y+8Q9NJ/6EfVTTzMxFzLXdLAs9WyJ6dwQ9cIwmYFt5DfvoDS4m4H+fvrzDsd/jB/8IdO1kMM+xFx5J+WtexgaGmHb0CCVSj+FfEAhyJBvTBJ+9d0ED33urLEd8wFO+CYaVmCQ0wwxyWkqVKiSocVn+9/KroE8FzeeZNPUowzPPUWO87fYZr3ASfqZ8zxz5GlkCjQzRZqZAnkabGmdooVxnCGms/14tojliliuhOWKEES3aN0z9zPZHKVTj7L55ANMU+SwbSfMRb/484UixWIRsgVa2TyeydHKFPD4fr5QolwuUSqVKRZLZHMFMrlC9HMOAjJmZDPRzQzqs5M0q1PUM0XCbJl8Pkfj4H5K3/0vbBv7Pi03ZijSb9XzjkXLjSp5qhSY8SJVK1HN9FHPliGTIcBpZQLCeIzCbJFWUIRMwObaEYZrT+G0qJPnVGYLJzNDNHP9BLkifa0J+mpj5KvHGWiOs5VTFKzBuPdzwjezyaYpUeMEmzllm6lmytQzfTSCMs1sGQ+KkCvF41uilS0QZKAvnMJoMZMdoJktkMtEk7pcNkMmKECxn5ZlCRt1snMnKcwexd1pBH3UsxUauQpWGCDXt5l8eRPFygDZiYNkx3/KTJhlKjOAlYco9m+hNXua1uQRbPoowexxnvPqm7jo0hcsmwnnsmGhbmbXAR8DssCn3P28J1JZr1BPWjOMgr8ROkHGyJhRa4bUpidonnyS1unDNGuzNOtVWvUZWvUqrXoVb8zhzdqZm4V1rNXAwgaZVj2+Nci26mS8QdBqRMv4Npvt53RhN1labGocI9uq4Q5Zr1NozVH0OQrUz1v7tBcp0CBnaz9ctOXGNEUm6aOPObbYNBAF4SEfjl48Gch4i6LPstknyLD0/7/QDQMydu5tah5QI0+WkAINPh7+Gg8Xr+aml+9hV/1xWk8/SDgzTqs+S704RKs0hM2dxsMmxde8h8tecNXZ39Ad6tP47Di1yTHmTo9RnzpBfXaCWrNFc26GzMxRMtWTeGMOmlUyzTkyzSqZsEaTLJPBEFlzBpvHKYbTZFt1cl4j53XyLH+q5wO+jbI12Ep7rqPb9Ax1cjTJ4kCO8FmTjarnKVmdk17hzuwNXPyqt3H9y/aRnXwKnzzM1NQUteoMQRj9/61VZ2hUZ2jUZghrM7Rqs1CfJtOYJohvjhN6hqw3yHmNfKtOnhp5r5OnwTGGeSq7m9ACStQZ9FMMtcYp+SxZWkxRZty2MJsfIezbSq20jTBXoTR3jFJtnHpugGa2SKE2TrE2Ti6cIR/OUAirFFpVCiw/odpoD7zyk/zjV/3mqr52Q0LdzLLAT4HXAoeAHwFvdvclT6SS1lDvaK0QGrNQn4WwBmGDsFmn0WgQbN5FUBmKwqwxC9XTNGdPMTczTbU6RW1minp1Cq/PQqNK6E7YclotP3MfoBBkydEgrE7g1UmoTdAkYGLwShrbr2bg4ivZMzLAUKVwdm1hE2bHCSePMDV2iJnxw8xVZ6mGEG66mNKlL2Pb4AAD9eOEp55i4ugTTE6cYnpqkla9Cs0q1qzSCkMe33UD4fareN0Lt7OplEtgoFegFUKzBs05GvEv+eZclUZ9lrA+R37rL9A/tCtqwzXmop9JWKc2V2VqZoawWYfmHDTr0KxjrTrerFGvVanVatTnqjTqc1j8c7awhrXqWNjAwjoOhBbQKA7huQp5r5ELZ7HGDGFpiMI1b2fPzu3JtjTcowMSgnwbvk8dGtVozBpVMMNLW8AyWPU0NKo0W049jG9zszRnT2M4QS5PUBki2LyTbDYLc1NQm4TaFLWZU9SmJ6jPnKZZnaTVvxvb+nzKOaPcPE198gS16ZNky5vJb9lFeWgXxc3bsaCwfN1L2KhQfynwPnd/Xfz4VgB3/69LfY1CXUTkwl1IqK/lV/Iu4KkFjw/F6xYXc7OZjZrZ6NjY2Bp2JyIiy1lLqJ/rbf9nTfvd/TZ33+fu+0ZGRtawOxERWc5aQv0QcNGCx7uBp9dWjoiIrMVaQv1HwHPN7BIzywNvAu5qT1kiIrIawWq/0N2bZvYO4OtEhzTe7u4Pt60yERG5YKsOdQB3/xrwtTbVIiIia9Qbn7EVEekRCnURkRTZ0HO/mNkY8PNVfvkwcKKN5bRTJ9cGnV2falu9Tq5Pta3OUrU9x91XdEz4hob6WpjZ6Eo/UbXROrk26Oz6VNvqdXJ9qm112lGb2i8iIimiUBcRSZFuCvXbki7gPDq5Nujs+lTb6nVyfaptddZcW9f01EVEZHndNFMXEZFldEWom9l1Zvaomf3MzG5JuJaLzOybZvaImT1sZu+M1w+a2TfM7LF4uSXBGrNm9v/M7Cvx40vM7J64tv8Zn6snibo2m9nnzewn8fi9tMPG7d/HP9OHzOxOMysmNXZmdruZHTezhxasO+dYWeRP4tfHA2Z2dQK1fTD+uT5gZl8ys80Lnrs1ru1RM3vdeta2VH0LnvsPZuZmNhw/Tnzs4vX/Lh6fh83sjxasv/Cxc/eOvhGdV+Zx4FIgD9wPXJFgPTuAq+P7/URXf7oC+CPglnj9LcAHEqzx94D/AXwlfvxZ4E3x/T8F/m1Cdd0BvD2+nwc2d8q4EV0L4EmgtGDM3pbU2AGvAK4GHlqw7pxjBVwP/G+i02FfA9yTQG2/AgTx/Q8sqO2K+DVbAC6JX8vZja4vXn8R0bmqfg4Md9DYvQr4P0Ahfrx1LWO3YS+aNQzCS4GvL3h8K3Br0nUtqOfLRJf0exTYEa/bATyaUD27gbuBVwNfif+znljwgjtrPDewroE4NG3R+k4Zt/mLvgwSnRPpK8Drkhw7YM+iF/85xwr4M6JLST5ru42qbdFzbwQ+E98/6/Uah+pLN3rs4nWfB14EHFgQ6omPHdHE4ZfPsd2qxq4b2i8rusJSEsxsD3AVcA+wzd2PAMTLrQmV9VHgD4BW/HgIOO3u81c9Tmr8LgXGgD+PW0OfMrM+OmTc3P0w8CHgIHAEmAD20xljN2+pseq018i/IZr9QofUZmZvAA67+/2LnuqE+p4H/FLc5vu2mb14LbV1Q6iv6ApLG83MKsAXgHe5+2TS9QCY2a8Cx919/8LV59g0ifELiP7s/IS7XwXMELUQOkLcn76B6M/cnUAf8PpzbJr4/71z6JSfMWb2XqAJfGZ+1Tk229DazKwMvBf4j+d6+hzrNnrsAmALUfvn3cBnzcxYZW3dEOodd4UlM8sRBfpn3P2L8epjZrYjfn4HcDyB0l4OvMHMDgB/TdSC+Siw2czmT7Oc1PgdAg65+z3x488ThXwnjBvALwNPuvuYuzeALwIvozPGbt5SY9URrxEzuxH4VeAtHvcLOqS2y4h+Wd8fvzZ2A/ea2fYOqe8Q8EWP/JDor+zh1dbWDaHeUVdYin+Dfhp4xN0/vOCpu4Ab4/s3EvXaN5S73+ruu919D9E4/V93fwvwTeA3Eq7tKPCUmV0er3oN8GM6YNxiB4FrzKwc/4zn60t87BZYaqzuAn4rPpLjGmBivk2zUczsOuA9wBvcfXbBU3cBbzKzgpldAjwX+OFG1ubuD7r7VnffE782DhEd7HCUDhg74G+IJmCY2fOIDiI4wWrHbr3fsGjTGwvXEx1l8jjw3oRr+UWiP4EeAO6Lb9cT9a7vBh6Ll4MJ13ktzxz9cmn8n+FnwOeI32VPoKa9wGg8dn9D9Cdnx4wb8J+BnwAPAX9JdNRBImMH3EnU228QhdBNS40V0Z/pH49fHw8C+xKo7WdE/d/518SfLtj+vXFtjwKvT2LsFj1/gGfeKO2EscsDfxX/v7sXePVaxk6fKBURSZFuaL+IiMgKKdRFRFJEoS4ikiIKdRGRFFGoi4ikiEJdRCRFFOoiIimiUBcRSZH/D57E3OtQjEttAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot losses\n",
    "plt.plot(clf.history['train']['loss'])\n",
    "plt.plot(clf.history['valid']['loss'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1c386056308>]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VFX6x/HPk15ICGmUhBBK6J3QRIooCqigoi5gb1jX7i6uruu6u7+1dyyoa0UQUREVxY6g9F4DASkhQHrvyfn9cQcJIcAAUzKT5/165TVz79zMfbgk35w599xzxRiDUkop7+Pj7gKUUko5hwa8Ukp5KQ14pZTyUhrwSinlpTTglVLKS2nAK6WUl9KAV0opL6UBr5RSXkoDXimlvJSfPRuJyGjgBcAXeNMY83id19sA/wNigBzgSmNM2vHeMzo62iQmJp5KzUop1WitWrUqyxgTY8+2Jwx4EfEFpgGjgDRghYjMM8ZsrrXZ08B7xph3RWQk8F/gquO9b2JiIitXrrSnRqWUUjYistvebe3pohkApBpjdhpjKoBZwPg623QFfrA9/6me15VSSrmYPQEfB+yttZxmW1fbOmCC7fnFQJiIRJ1+eUoppU6VPQEv9ayrOwXl/cBwEVkDDAf2AVVHvZHIFBFZKSIrMzMzT7pYpZRS9rMn4NOA1rWW44H02hsYY9KNMZcYY/oAD9nW5dd9I2PMdGNMsjEmOSbGrnMESimlTpE9Ab8CSBKRtiISAEwE5tXeQESiReTQez2INaJGKaWUG50w4I0xVcAdwAJgCzDbGLNJRB4TkXG2zUYAKSKyDWgO/MdJ9SqllLKTuOuOTsnJyUaHSSql1MkRkVXGmGR7ttUrWZVzZW6DjZ+6uwqlGiUNeOUc1VWw6Fl4bQjMuQ6yd7hmv1nboaLYNfvyJNk74LNbYfdvx9+upkaPnxfRgFfO8c1U+OGfkDDYWv59oXP2k7UdslKtYPrpv/ByMix+3jn78lQF++H9i2Ddh/D2GJhxOXx+O3z798NhXlMN62bBKwPh2a5QWerempVD2DUXjVLHZAyUF0JQ+OF16z+GFW9QNfA2/lk2mTtkPIGbvici+XrH7nvPUnhvPFSVQXAklOZgEHL2bqHRX2VXWQYr/wdFByDlGyjOhmu/wvy+CLP6XXwQKEy3Av78Z+DLe2D1uxDcDMryIG8PxHRy979CnSYNeHVyUr6Gr+6H8FYQnQS7f4XcXVTFdONg9CBahQqy7kPKWw3k0pRz2XBgD/2DezBs50IWbTvI0I7N7dvP6vchcysMvBkiEo5+PTMFPvwThMext+NV7Fn9HfMrO3Gh7xJ8d2xjxg/buf2sDvj61HednpczxgrsdR+CbyCERHFg7Fu8sLopv2wbyL68ngzrGMN/4mbSeuVbUF4AGz6GIXdDx/OsVn7eXg34k7XiLWjZC+LrnP+sKAb/EBDX/yxqF42yX8rX8NFVEBgGItRs/Zr9AW2YH3k1aw5W0Xzz25St/Zjq2O5cW3gru/Iq+N+1yQw/7zIipIhn3/uE/fl2fPTP2GoF1JKX4YXeR3e5VFXArMkYX3+mxT3BsIUduY/76Hj+XfTo0oV2gfk8+902Plxm95xM3mXpq1a4j3iQ7Hv28mS3zxg2Bz5fu48ecU25eVg7tuwv4Jy1w9gX0M4K984XkNbvAV5aXQFAdW4jPXanaufP8NW98P2jR67ftxqe7gi/PO2OqrQFr+yQlQpLXoI1M6BFD7Iu/ojHf97PnO1pkAeRoQGM738L7xeWM2/9fpJCmrAzu5h3r+vHmUnRUHgOfAMD2cC0n1L590U9jr2vmhor3AObkH/ZHPZ+dD/tf3iCwAE34xMQbG2z6h3ITuXnvi/z1G8VTBrQmofO70qTQD8oTiAk9Usig31JOVjoksPjFulrYN6d0GsSDL7NWleaBz//F5ZPp6TdGB4+cC5f/vdHKqpruKRPHFPHdCY2PAiA+87txJuLd3Ldd3dwuc+PvL75MrLWLURMDbcE+rJ/1zYSBrjx3+dJqipg/gPW892/QnEWhEZD/j6YOQkqiuC3F2HATRAc4dLSNOCV9YMY1gJ8fI9cX1ECP/0Hlr6C8fEnve0EXvG7is9fXkN5VTU3DW3LRX3i6NIiHB8fobK6hoLyKn5OyeRf47tZ4Q7We8d04ZLSVM5fsZdbhrcnvllI/bWsnQF7fiNr5DNc9mkRbQrP5Z2ANSz4YibnTbje6u9f+ATFLQdx8/IohneM5j8X9cDnUFdMeBxSXUGHsAoyCsqdd8zcaeOnMPc2MDWw4EEo2Ad+gbDqHUxJDutbXMLV2y6k0ieDyQMTmDwwgY7Nw454iwA/H24b0YGzOzfn45WDGQdEBPtzQa9WHHwpiqx9qdTTMdZ4GGOFdt5uGPeS9TNcW02N9buRYxsdlrWNHT3uof2G52DrV9B7MsyabIX7xa/DZzfDijdg2AMu/WdowDdGhy5uMzWw8AlY+CQM/yuc9eDhbQ5utn5Ac39nZ8KlPJB1Ias2+RMeVMx53Vpw21ntaR/T5Ii39ff14bUr+7F5fwF9E5oduc92I0ha+TYhlPPyj6k8PqHn0XVVlsKP/6ayVX8u+jWR4soKnr7pBorff4WydXPYOGgC3bdNg5Is/iJTiQwJ5NnLex0Od4DwlgB0DClgY6EHnmot2A8zJ0LnC2B4PWGwYQ58ciO0HgiXvws/P251ZYkPJulcnq+awAubQxnXqxV/G9uFFk2Djru7Ti3CePiCrkesSw2NQ/L3Ul5VTaCf7zG+08v9+jyseAMjvshrQ+Hy96DN4MOvf/8Pq1XepAUUHWBNk6FcvCKZxUHNiVz3GSHlBbB/LVz2LstDhtEjcRTBS16BgbdCYJNj79fBNOAbk7J8WD4dlr4G1RUQGA4FaRDUFNa8D8P/YrXid/wIs6/B+IfwRtsX+b8t0XRrFc7Lk9tzbtcWBPgd+9RNkL/v0eEO0OUCZNmrPNzhdx5cFcjUMZ2JCAk4chvbqI9/B9xDZnElc245gx7xTSnvdiHnbPiMf8ycwVMlz5IeP5avUlvx8uQuRDUJPPI9wlsBkBiQz0+ZDbwFX5prnSwuzYWAUAiNhdlXQdY265NK3YDf/j18djM1CYOZ2fE53np9K6EBE7ikTWd8WnRjfWE4n6xP466zk7hnVMdTLiu0eSKhOxfyy7YsRnW186S4s1SWgn+wY99v0TOwfz1c9CqERkFVOfz+i/Vzn5+GCQiFdbP4yW8oL1WO59Xy54j84HL8b1uMNGsDy163wr3/jezo/yh3vf0j23OEW4Z34Iflg5i85ytq9i/HJ+k85pYnc/f7S+gtQ5kb+B15n91HxOWvgo9rTn9qwDcGJTmw9BVYNh3K8yHpXGiWSHXePhbF38wvOwt4pOApSrf9RHBEC5hxOSY6iYdDHmHGlhr+PLID947qiJzOKICEMyA8nvNqfuGBmk58u+kgl/evNUlpRTEsfo6dYcm8m96aFyf1okd8UwACe00gcMOH/LvwEYqCY7in8CraRYcwpnvLo/cTbt2qoLVvHpmF5RhjTq9uZ6gogaXTYPELUHHkeYIq32CWB4/gjJyfIT8NmsZD+lpY9DRs+ZLSqG5MyLqdzdt20DchguAAX17c247clAKggOuHtOXuc5JOq7zY1h2R3+fy1Lrd7g34bd/CR1fCBc9BnytO771Kc2HLF7D4OcjZCT5+8P54OPtR65qN7O0YvyCKg1pSVZzN+uruPB9+J13btOCu7X/njdJ72PPaJJr3v4iwxf+BTuezuMNfuP3VJfj5hPDhzf3o1yaS1ObX4z/vc8orhRlht/D4JxsY2DaSoUljeWvROm7YOouyzwMJGv+8S0JeA94bVJVbP7B1+9ABijJh+nCrn7bLOKrPvI/3dzXlh60ZrNmTR1F5FX1aBZFfGsKy2S9wRlQRTYLCebz508xYkcfD53fhxqHtTr9GHx/ocSlhv71E94hr+GrD/iMDftEzUJzJAxW3c83gNozr1erwa22HY4Kb4VtaxKSi21iTZ3jy0vb1D4EMjQHxpQXZVFTXkFdSSbPQgKO3c5eC/fD+xZC5haqO57Mt7hJSCgPZlbaP8gMp/FLSgeAqf86Qn1m98HP6jpoMb48FX3+2d7yJK7cMwCckmHev78mwpOg//niVVFRRWFZF8/Djd8nYw7dZAmDYtGUz1TX9nTPU1BhIWwFrPoBeE6HNGUe+nrYKPr4GqsutP269Jtb/823Pfn57CX78l/WpNaYzXP05VZUV+Hx0BT4zJmDC4zg4+g3uWR3Dkj0ltI8J5e5zOjK3R0t8fITqmh4s/CSHkZv+Bos3sSz0LJ7MnsKqt1fSLiaUd64dQEKUdU6pQ5/hlK4czEcFPXjst1LaRofy+lX9iAgJYF2HZ3jtzSpuWfcu1dHt8B16twMO5PFpwHuq/DT44THrY2VxJrQ7C66ea7225gOrxdLvWmuagJJsuOF78qJ6cdestSzctplOzcMY17sVF/WOY0DbSDJnXsy5KTMgE96N/Suvr8hjyrB2jgn3Q3pejvz6PHc238Bt2wPJK6kgIsgXFjwEy17lp8Cz2O3Xg7fPqzP+2tcfGfcS2UVVbJwbSFxEEBf3qXtTMRsfXwhrSWRNNgAZheUNJ+BzdsJ7F0FJNhnjPmTC98HsXW8NG41v1pHB3QZzb7cWnNkhkoL//ps9K7+meUA5cZXFpJw/nzEf5dGvTTNeuaIfMWFHdk2FBPgREuCgX+em1h/eyKoMMgvLT9iPf1KqKmDzXGsoZ/pqAEzu78g1XxzepiQHPrzc+mM9+Hb4+i/WEN0uF5zcvmqqrdb58unWOY2h93EgtAvPfr+NuWvT6VfzAEN8NvJ+/sXkfhlEsH8lT17akwl944/4o+brI4y87HbyQrPYlJbLX7PGEuQr/HNcNy7tF09oYK3jLkLwlG+4usYQs/EAfdtE/NEV2SuhGSkX/puH5kbRoWIE153yQbSfBrynMQaWTIMf/w0Y6D7B6lvf+iUc3AQhUdYww+oK69L9ymK46FU2+3bilpd/ZX9+Kf93cQ8mDWh9RNdFzJnXQcoMtgX34h97ejKycyx/Hd3ZsbU37wbNuzM8Zzb/9llFyTuvEVGSAkUH2BA/iRtSz+eZP3UmPMj/6O/tciEtgDebZhIZEoC/73E+3oa3IrzSumPYwYIyOrUIO/a2jlacZX1aCmoKzRKtdWUFVtfA0lfAP5i9F87k0i8qKK+q4rUr+5Gc2IzoOucSpOMIhqQsomjpTkpi+3LHj1br/K1r+9d/fBzJdmFZvGSyL6/EcQGft8f69JKdionswG8dp7I9ZSNX//4Vv6zawLC+3a2fyd9ehJJsdp3/IVfPK2AWMWTP+Q9y/VC6xzW1f3/fPGiF++A7KBz2CK//sos3F/9MTQ1clhzPgLY9CfTz5eztmRhjuGdUR2LDjv1vjRj7D4YAi+3YtY+PcH7Po7sQL++fANzL2HpecwYN+IbOGCscUn+AHhOsy/PXf0Re63OYWnoli9eEkBRWzsc+3+G34k0IioDqSmto1ur3Ma36MqdqKH9/9VeaBvvz0c2D6z8JGp8M5z9DUtK5fJQTSs/4COd8NB98OwHfPcI5fusoyQ7nQOuBfBfek7+nduHszrFc1PsYLXOb4R3tuBNYeCtC0jcAVgv+CPn7rBOxju6Xr6mBL/5sfXoCCAiD25dCWCtrVMzuX6HHZXzX8mbumZNDkL8vH00ZfMw/PoFJZxGT8jkxwAPpF7K9qoh3rnNBuAOEx2EQ4iSLfXll9Gtziu9TVQ7L37AaIM27wYK/QUURu0e9yS3LY9iyvpjLEzvgc+BLFn76Oksy7mDq0EhY9joVXS/huq9LKakW1raaxNj0F3nl20/ofp2d012sfg+Wv071wFuZEX4TLzz9C9nFFYzv3Yr7z+1E68jDw3RHd29xnDdyvCO6Jp1MA74hM8ZqqS96GhMai3x5DwAzQq7ioe2jiQ0L4uI+LUg5WMhneQO5eO0s/Pz8ocuFVPf4Ez/5n8Vz329jU/p6BiRG8vIVfY7dQhGB/jciwEBnXovRezLSezIzvt/Oc99vg63gI3D3OUn8eWSSY06IhrfCb/t3gCGjsOzw+nWzrPHII/8Ow+631hlz+mFvjNWNsOYD6H+T9cfyi7utcdQdzrbC/cIXeSJzIK/Os06OvjS5L3ERxxkd0nYYANWBTVnsM5Qr+iUwolPs6dVpL78ATFgL4nKzSM87xUnH9iyzJjTL3g7iA6aGyqAo3m3/Io9/FUBkaBXTr+rHqK7NMa+/yLX5Kxn+yw6uKXidFlVl/CP/QvbmlDBzyiD6txxE1jOfMnnXwxSnDSA0vru1D2OsUUixtT5pVpRY1wN89wjZsUOYuHEUqdmbGNQukrfHdqFnvGsvNHI3DfiGyBhr2NbSV2DbN+xqcynXZU6mSflmggP8oGlv7usXzfVntiU00I+KqhoeePFiLiv4BargQ/+LmfbkT+zLKyW+WTDP/akX43rFNah5We48uwMTB7Rmb04JTYP9SWruwG6U8FZIZTEtAysPX+yU+oMVOL4B1jTGfa60wuHja+DKTyCu36ntqzjb6ufdMJuifrfyZsB1LPw1i4uDJnN1yluQ+j20G8GHlSN49eeNTBqQwGPjux2/iwkgsh3ED8A3aRQ/DxlDwIm2dzCfiATaFGaz4VQCPn2NNXtlaDQrhrzBu2nNqdq7klV5zcle7c9FvVvxyIVdDw+T7XEpCd89wushrxOzZRFf+45g5o4A/nVRN/onRgKQcdFMoj+6EP8PLoHbF0FYc9gyD2ZfDZe8CT0vg/3r4INLoTiDTQG9mLznGmJj/XjrmmRGdo5teKOpXEADvqEpyoC5t0Lq91QHRTKv2fXcmzKSbnEBTDlvAqO7tzgqHAL8fLjjistY/sp0Kmp8eGhFIEPah/K3sV0Y1bX5ccetu4uI0Dw8yCGjPo5iGwvfJbSQzMJyq4tg9jUQ05nC856nyQdjkHl3wt6l1msHN59awO9ZZnW/lBewvcvtTFg1jMLyVHrFRzCt9FySzQ90MAd4L+IuHp+3ieEdY/j3Rd3t+0MrAjd+B0DgCTZ1iogEWu/75eRb8Plp8OFECInit7M+4opZvxMXUc3gLufwYPsozuoUe/RJ7+4T4Lt/MKrmVz6oPptZIdcwc/IgBrc/fKFaly49uDXsUV4qut+6OO+CZ61zUWBNS91pDMy9nRoR/tLkvywoas+DF3fh8uR4/Fz8x7Eh0YBvCEpzrSl2c3fBho+pKctndtRtPJo+AFMUxEPnd+K6IW2PGwxJzcNYNHEO6XllLO/e+qhRFo1KmBXwHYLyWVNYBjsXQkUhuwf+g1FvZfFB/AQGbJ9FZUAE/sCetD0k9D3JfRgD8++HgFA+6v4af11URa/4JrwwsQ+J0aFkFJbxyMyn2fb7Lnb+Vkq3VuG8NLlPg/oUdVxNWxNTk0V6TpH935OZYv3Bqyxh/7jPuXVmGp2ah/HpbWccf4RP03i49kskPI5hJpbLwoMI8j9ySKSIkDxwOB8tGM7k1e/h03YY7F0GXS60xre/PRoObuDNlv/k011tePu6vvadr/FydgW8iIwGXgB8gTeNMY/XeT0BeBeIsG0z1Rgz38G1eqfiLHh3HGRswvgFsz84iZvK7iU9tz3XD0/gqsFtaNnUviv5hnaJd3KxHuKPq1kL+Kag3BpKGhDGc1ubUVGdwU27RzK7dS6PZgzjbR4lN3P/yc+7svUrOLCelEFPMHVhFef3bMlzl/f+49NSbFgQr00ZRVllNQABvj5HTqnQ0EW1x49qJH+Pfdtv+RI+uwX8g9g95j2u+TwfYwyvX9XPvuGbiWcCcLzzuZf2i+e6xRO5vGwhfp/cSKVvKNccvJqnY/KIP7CItaFD+L/fO/DgmM4a7jYnPPIi4gtMA0YBacAKEZlnjNlca7OHgdnGmFdFpCswH0h0Qr3eo6oCk7ac0rn34pe/izvM3/i2qBsUCRf3iWNG7T5KdXLCWoL40r5mFxmFfTA7fqA07gzmbcjg+iFtWZ+Wx3m7ryG6SSC5VeFUFWae3PvX1MDPj1PZtC1XLGtDx9hQnrq0Z71dYXVboh4jyroaNrZiL4VllYQda/ROWb41HHHtDGjZi+97PsufP8miSZAfb183gDZRoQ4rKSIkgFdvG8f8aZ9zUcWXfFg1koyKAK7IvpgH/Kp4PHcSU8d0YcowB1674eHsacEPAFKNMTsBRGQWMB6oHfAGOHRLn6ZAuiOL9Co1NVT9+iLm5yfwry7BmEBu5a/E9BzFP1uG0yO+af3DGJX9/AKg6zj6bJ1Ph6qeSN4eFgRfip+vD7eMaAcG/jFvE1OGtaP0vWZISZZ975u315qON2sbHNzA4wF3U258ePXKvo67yKihiLYCvp2ksz+/rP6AN8aaDnfPUmrOvI/HS8Yz/fM028VYfZ1yfqVF0yDOuukJUmZXMGTUv7g2qSMLNnXik9W9mHZWB3q1blyjZE7Enp/KOGBvreU0YGCdbR4FvhWRPwOhwDn1vZGITAGmACQkNMLJSEvzqJh1FQG7f+G76r4sbTqGLoPP58XkTkdeDadO3xl3ErDpM57zfxWAl/ck8Kf+rf8YJvrqldZJ1ZSgSAKLsu17z/n3w86fKY/tyWd+45lTPoj3bxxIuxjXzQ7oMiGRVAZG0r4qnX25pUdNNwzAupmw+1fMBS9w+9YefL0xjasHt+Hh87s69cR+05h4mt7+8R/Lo7u3ZHR98xIpuwK+vo5DU2d5EvCOMeYZERkMvC8i3Y0xNUd8kzHTgekAycnJdd/D6+XNf4wmuxfzcM0UBky4m4d7tWqUQ7dcIq4veS0Gk3RgCXtoTnZgHHeeffQkXD6h0TQt3GlNm3C8LrE9S2HbN/ze634u3jCQ6hrD+zcOpLcXtxhNVBLtS/ezPa/U6oo5sMGazC2sJVSVWjftjh/AzKoRfL1xEw+c14nbz+rg7rJVLfYEfBpQ+9KreI7ugrkBGA1gjFkiIkFANJDhiCK9we4dW2m54V2+kJH8acrDf8yUqJynJPkOIr5cwsKqHjw6oVu9I4sCmsYSeaCQLZnF9GtzjIA3Br5/lLLAGMYu70ZCbBCvXdWPttGO619uiPyad6TdvnkszCuF75617qT1BwER0oZ8yL8+3MqZHaK5dXh7d5WqjsGegF8BJIlIW2AfMBGYXGebPcDZwDsi0gUIAk7yzJX3+jU1i5wZU2mB0Puq/6OthrtLhHc/j/+bO5mixPP4T+9W9W4TFtmCECln94FM+rWp59xH1nbr5g97lvAUN9K5dXNm3DjQ+/rc6+ETnUSM5JObnQEHvoXEoWS1v4jsA3tp39SHvaHduGJuEYH+PjxT98YrqkE44U+pMaZKRO4AFmANgfyfMWaTiDwGrDTGzAPuA94QkXuwum+uNcY0ui4YYwyrdueydm8e2w9a44fT80tJT13Ht4ELKeh1I23b6Z3qXaVJkD/n3PAfurQMO2ZXWHi01Xd78MA+oM5NMtbOhLm3YnwD+DbkAuYUncUXf+rTKMIdgGjreCQc/AEK06kc9lcmLmpLakZLwoP8KKusISZMeP/6gc65YE2dNrt+Um1j2ufXWfdIreebgSGOLc0zGGNIyy1l5e4cpv/yO1v2FwAQ3SQQXx/wEeGtNovxyQqg2bl/dXO1jc+AtpHHfd2viTVeOjejTq9j2kr44k5q2pzJowH38t6GUp66tOcf8343CrahkqMKPwPg3Yz2pGYU8LexndmyvxBfH+GhsV0aznTM6iiNpCnieBVVNby/dDev/JRKdnEFAO1jQnnyUmuq3T+mfy3KgOe+tuY+CY12Y8WqXiHW/0lRzsHD64oyYdYVVIe14qbSP/NjSin3jurIpf0a2YVkzdpQI350MLv53acNT/5WxLherZgyTPvaPYUG/ClIOVDILR+s4vesYoZ0iGJsj5Z0bhFOn9YRR/dDrnjTmpt90O3uKVYdn+2PbmVhBlXVNda8JavehqIDPNHmDX7ZXs0LE3sz/gTTGHslX38kqh1kbWOpT2+aBPnx9zo36FYNmwb8Sfp+80HumrWG0EA/3r6uPyM6xhx7qGN5kRXwHcdAtA4fa5BsAd/UFJCWW0piZDCsnUFG9ECmp4TywHkdG2e420hUEmRt47I/Xcv5rYe6Zj565TAa8HZatTuH57/fzqLtWXSPC+fNq/sf/043BzbCJzdYt8s70/n3XlSnKDCcGh9/oqSA9fvySSxcDbm7eNZcSHKbZtzS2If+teoDe5bgl3gG4f4a7p5GA/4Elu3M5sUft/NrajZRoQE8OKYzVw9OJDjgOHOMHNgIb4yE4GZw1WeQMMh1BauTI4KERtPKFDN//X7GNZlBuW8o80r6Mv+yXp4z+6OzDLkLkq8Hfx0l44k04OthjOG3Hdm89ON2lu7MIbpJAA+N7cIVgxLsGyL383/BLwhuWQRNXHQXHnXKJCSazj4VPJqyCxM0l/lmKAM6xpPo5Rcy2cUvAPyiTrydapA04Gspq6xm1vI9vLd0Nzszi4kJC+TvF3Rl8oCE47fYazuw0boB9vCpGu6eIjSKuJo8htesRKpKeb/8DG5Mdt19M5VylkYf8MYYth4o5OeUTN757XcOFpTTJyGCpy/rxQU9W578dK+/PAmB4TDoFucUrBwvJJrQ3F2MC1rDweoIdgd35Zwuzd1dlVKnrVEH/M8pGfzry83syCwGYEBiJM//qc8Rtwo7Kdu/g82fw7AHrP535RlCo5GiDIaQwZzqwVzUv3WDvM2hUierUQb82r15PPfdNhZuy6RtdChPTOjB0KQYWh3vLvcnsmsxfHQltOwFZ9zpuGKV84VEQ2UJgcDm8DO5cdDx7iuklOdoNAGfV1LBF+v38+W6dJb9nkNEiD8PjunMdUPann5r7dCNhpslwpWfQVD4Cb9FNSChtk9sAU34z723g18jvp+t8iqNIuA3pxdw/TsrOFBQRruYUP4yuhNXD06kiaNusrH4eagqg8kfHQ4L5Tls0xXQ4RwNd+VVvDrga2oMX6xP52+fbiA82J/PbjuD3q0jHHuTjcIDsPo96D3JasErz2O7STedL3BvHUo5mNcG/G87svjH55vYnlHHNcSGAAAW4klEQVREz/imvHF1snOmNP3tJaipgjPvcfx7K9eI62ddkNZ2hLsrUcqhvDLgN6Xnc+O7K4kNC+SlSX0Y26Olc65ILM2Dlf+DHpdBpN7J3WOJQPuR7q5CKYfzuoA/WFDGDe+spGmwP7NvHkysM29EsG0BVJZA/xudtw+llDpFXjXYN7+0kmvfXkFhWSVvXdPfueEOsGUehLWyPuIrpVQD4zUBX1pRzQ3vrCA1o5DXrupH11ZOHqpYUQKpP0Dn88HHaw6jUsqL2JVMIjJaRFJEJFVEptbz+nMistb2tU1E8hxf6vE9+Ol6Vu/J5YWJfRiaFOP8He74AapKocuFzt+XUkqdghP2wYuILzANGAWkAStEZJ7tPqwAGGPuqbX9n4E+Tqj1mBZsOsDctencfU4SY3u0dM1Ot3xhTUfQplHeilYp5QHsacEPAFKNMTuNMRXALGD8cbafBMx0RHH2yC2u4KHPNtK1ZTi3n+WiuyZVV8K2b6w7Nfl63XlqpZSXsCfg44C9tZbTbOuOIiJtgLbAj6dfmn3eWLST3JIKnrqsJ/6+LuoLT18DZfnQ8TzX7E8ppU6BPYlY3wByc4xtJwJzjDHV9b6RyBQRWSkiKzMzM+2t8biW7symV3xTurVq6pD3s8uuRdZj4pmu26dSSp0kewI+Dah994N4IP0Y207kON0zxpjpxphkY0xyTMzpnwgtq6xmw758+idGnvZ7nZRdiyG26x83bFZKqYbInoBfASSJSFsRCcAK8Xl1NxKRTkAzYIljSzy29Wn5VFYbkl0Z8NWVsGeptt6VUg3eCQPeGFMF3AEsALYAs40xm0TkMREZV2vTScAsY8yxum8cbsWuHAD6tXHhzTXS11hXr2rAK6UaOLuGgBhj5gPz66x7pM7yo44ryz6rdufSPiaUyNAA1+30UP+7Do9USjVwHnsJZk2NYeWuHDf0v/+q/e9KKY/gsQGfmllEQVmVi/vfq7T/XSnlMTw24A/1v/dPdGH/e8E+qCyGFj1ct0+llDpFHhvwe3JKCPD1ISEyxHU7zdttPUboTZmVUg2fxwZ8QWkl4cH+jr393onk7bEeIxJct0+llDpFHhvw+aWVNA128TwweXtAfKBpvGv3q5RSp8DDA97ftTvN3Q3hceDr4v0qpdQp0IA/GXl7tHtGKeUxNOBPRt5uDXillMfw3IAvcXHAV1VAQbqOoFFKeQyPDPiaGkNheZVrAz5/L2C0Ba+U8hgeGfCFZVUYA+GuDPhDQySbaQteKeUZPDLg80srAVzbgv/jIidtwSulPIMGvL3y9oD4Qlgr1+1TKaVOgwa8vfL2WBc46U22lVIewrMDPsSFAZ+rQySVUp7FswPe1S14PcGqlPIgGvD2KM2DogPQrK1r9qeUUg5gV8CLyGgRSRGRVBGZeoxtLheRzSKySUQ+dGyZR8ovrcTfVwj293Xmbg7bu8x6bD3QNftTSikHOOEZQxHxBaYBo4A0YIWIzDPGbK61TRLwIDDEGJMrIrHOKhgOT1PgsqmCd/8KPv4Qn+ya/SmllAPY04IfAKQaY3YaYyqAWcD4OtvcBEwzxuQCGGMyHFvmkQ7NBe8yu5dAqz7gH+y6fSql1GmyJ+DjgL21ltNs62rrCHQUkV9FZKmIjHZUgfVx6URjFSWQvgbaDHbN/pRSykHsGdRdXz+Iqed9koARQDywSES6G2PyjngjkSnAFICEhFMfcphfWklUk4BT/v6Tsm8l1FRCmyGu2Z9SSjmIPS34NKB1reV4IL2ebT43xlQaY34HUrAC/wjGmOnGmGRjTHJMTMyp1uzaFvzuJYDoCVallMexJ+BXAEki0lZEAoCJwLw628wFzgIQkWisLpudjiy0tvzSSiJcFfB7foPm3SA4wjX7U0opBzlhwBtjqoA7gAXAFmC2MWaTiDwmIuNsmy0AskVkM/AT8IAxJtsZBdfUGArKXNSCr66EvcuhzRnO35dSSjmYXROrGGPmA/PrrHuk1nMD3Gv7cqrCchdOFbx/PVSWQIKeYFVKeR6Pu5K1wJVXse75zXrUFrxSygN5XMC7dJqC3Uus6QnCWjh/X0op5WAa8MdSU2O14HV4pFLKQ3luwDt7quCsFCjN1QuclFIey3MD3tkt+N22/nc9waqU8lAa8Mey+zdo0gIi2zl3P0op5SQed/+5S/rE0bt1hPOnCt67zOqecdWMlUop5WAe14KPDQ9iULso504VXF4E+XuheXfn7UMppZzM4wLeJXJssyxEtXdvHUopdRo04OuTs8N6jNSAV0p5Lg34+mQfCng9waqU8lwa8PXJ2WmNoAls4u5KlFLqlGnA1ydnp7belVIeTwO+Ptk7IEoDXinl2TTg6yorgOIMPcGqlPJ4GvB16RBJpZSX0ICvS4dIKqW8hAZ8Xdm2FryeZFVKeTi7Al5ERotIioikisjUel6/VkQyRWSt7etGx5fqIjk7IKwVBIS4uxKllDotJ5xsTER8gWnAKCANWCEi84wxm+ts+pEx5g4n1Oha2Tu0/10p5RXsacEPAFKNMTuNMRXALGC8c8tyo7zd0KyNu6tQSqnTZk/AxwF7ay2n2dbVNUFE1ovIHBFp7ZDqXM0YKMmG0Bh3V6KUUqfNnoCvb15eU2f5CyDRGNMT+B54t943EpkiIitFZGVmZubJVeoK5YVQUwXBke6uRCmlTps9AZ8G1G6RxwPptTcwxmQbY8pti28A/ep7I2PMdGNMsjEmOSamAbaSS3OsxxANeKWU57Mn4FcASSLSVkQCgInAvNobiEjLWovjgC2OK9GFSmwBry14pZQXOOEoGmNMlYjcASwAfIH/GWM2ichjwEpjzDzgThEZB1QBOcC1TqzZebQFr5TyInbdk9UYMx+YX2fdI7WePwg86NjS3KAk13rUFrxSygvolay1aQteKeVFNOBrO9QHHxTh3jqUUsoBNOBrK82BoKbga1fPlVJKNWga8LWV5Gj/u1LKa2jA11aao/3vSimvoQFfm7bglVJeRAO+Nm3BK6W8iAZ8bSW52oJXSnkNDfhDqiqgolBb8Eopr6EBf0jpoatYm7m3DqWUchAN+EP0KlallJfRgD9EZ5JUSnkZDfhDtAWvlPIyGvCHaAteKeVlNOAP0Ra8UsrLaMAfUpIDvoHgH+LuSpRSyiE04A85dBWr1HePcaWU8jwa8IfoVaxKKS+jAX+IzkOjlPIydgW8iIwWkRQRSRWRqcfZ7lIRMSKS7LgSXaQ4S69iVUp5lRMGvIj4AtOAMUBXYJKIdK1nuzDgTmCZo4t0uuJsyE6F5t3dXYlSSjmMPS34AUCqMWanMaYCmAWMr2e7fwFPAmUOrM81dv4EGOhwtrsrUUoph7En4OOAvbWW02zr/iAifYDWxpgvj/dGIjJFRFaKyMrMzMyTLtZpdvxodc+06uPuSpRSymHsCfj6xg2aP14U8QGeA+470RsZY6YbY5KNMckxMTH2V+lMxlgB324E+Pi6uxqllHIYewI+DWhdazkeSK+1HAZ0B34WkV3AIGCex5xozdgMhfuhvXbPKKW8iz0BvwJIEpG2IhIATATmHXrRGJNvjIk2xiQaYxKBpcA4Y8xKp1TsaDt+tB7bj3RvHUop5WAnDHhjTBVwB7AA2ALMNsZsEpHHRGScswt0uh0/QkxnaBp34m2VUsqD+NmzkTFmPjC/zrpHjrHtiNMvy4WytkPime6uQimlHK5xX8lqDBQdhCbN3V2JUko5XOMO+NJcqK6AsBburkQppRyucQd80UHrsUmse+tQSiknaNwBX3jAemyiLXillPdp3AF/qAWvXTRKKS+kAQ/aRaOU8kqNO+ALD4J/KASGubsSpZRyuMYd8EUHIEyHSCqlvFMjD/gMHQOvlPJajTvgCw9owCulvFbjDviigzqCRinltRpvwFeUQHmBjqBRSnmtxhvwfwyR1Ba8Uso7acDrKBqllJdqvAH/xzQFGvBKKe/UeAO+KMN61C4apZSXasQBfwB8/CAkyt2VKKWUUzTegC88CKGx4NN4D4FSyrvZlW4iMlpEUkQkVUSm1vP6LSKyQUTWishiEenq+FIdrOigDpFUSnm1Ewa8iPgC04AxQFdgUj0B/qExpocxpjfwJPCswyt1tIJ9EN7K3VUopZTT2NOCHwCkGmN2GmMqgFnA+NobGGMKai2GAsZxJTpBTQ3k7ITIdu6uRCmlnMbPjm3igL21ltOAgXU3EpHbgXuBAGCkQ6pzloJ9UFUGUR3cXYlSSjmNPS14qWfdUS10Y8w0Y0x74K/Aw/W+kcgUEVkpIiszMzNPrlJHyk61HjXglVJezJ6ATwNa11qOB9KPs/0s4KL6XjDGTDfGJBtjkmNiYuyv0tE04JVSjYA9Ab8CSBKRtiISAEwE5tXeQESSai2eD2x3XIlOkL3DupOTziSplPJiJ+yDN8ZUicgdwALAF/ifMWaTiDwGrDTGzAPuEJFzgEogF7jGmUWftuxUiGoHUl/vk1JKeQd7TrJijJkPzK+z7pFaz+9ycF3OlbMDWvZydxVKKeVUje8yzqoKyN2t/e9KKa/X+AI+bzeYag14pZTXa3wBryNolFKNROMNeL2KVSnl5RphwO+A4EgIiXR3JUop5VSNMOBTtXtGKdUoNL6Az9gCMZ3cXYVSSjld4wr44iwoyYLYLu6uRCmlnK5xBXzGFusxprN761BKKRfw3IDf+hW8fwmYk5h6PnOr9agteKVUI+C5Ab/8DdjxA5QXnHjbQzI2Q1BTCGvpvLqUUqqB8MyALyuAXYut5yXZ9n9fxlaI6aKTjCmlGgXPDPgdP0BNpfW8JMe+7zEGMrdArPa/K6UaB88M+JRvDj+3twVflAGluVYLXimlGgHPC/jqKtj+LbQeZC3bG/CZthE02oJXSjUSnhfwacuhNAf6XGEt2xvwfwyR1Ba8Uqpx8LyA37kQfPyh63jr8WQCPjgSmsQ6tz6llGog7LqjU4MyYir0vNwa7hgSZX/AH9gAzbvpCBqlVKNhVwteREaLSIqIpIrI1Hpev1dENovIehH5QUTaOL7UP3YGUe2t5yFR9o2iqSyFA+shPtlpZSmlVENzwoAXEV9gGjAG6ApMEpGudTZbAyQbY3oCc4AnHV1ovUIij92C3/YtLHnFer5/PdRUQXx/l5SllFINgT0t+AFAqjFmpzGmApgFjK+9gTHmJ2NMiW1xKRDv2DKP4XhdNEtehu/+DqV5kLbCWhenLXilVONhT8DHAXtrLafZ1h3LDcDXp1OU3Y4X8Ac3Wa32bd9YAR+RAGHNXVKWUko1BPacZK3vrGS9M3yJyJVAMjD8GK9PAaYAJCQk2FnicYREWRcv1VSDj+/h9UUZ1rTAAFu+gPQ1kDDo9PenlFIexJ4WfBrQutZyPJBedyMROQd4CBhnjCmv742MMdONMcnGmOSYmJhTqfdIIVFgaqAs/8j1BzdZjzFdYNsCKNgH8QNOf39KKeVB7An4FUCSiLQVkQBgIjCv9gYi0gd4HSvcMxxf5jGERFmPdbtpDgX88L8cnrNGT7AqpRqZEwa8MaYKuANYAGwBZhtjNonIYyIyzrbZU0AT4GMRWSsi847xdo516MbZdQM+YzOExkKXcdbFTb6B0KKHS0pSSqmGwq4LnYwx84H5ddY9Uuv5OQ6uyz7Ha8E37wa+fjDoVsjfC34Brq9PKaXcyPOuZK0tNNp6rB3wNdXWnZv632gtD/+L6+tSSqkGwPPmoqkt2NZFU5x1eF3OTqgqs1rwSinViHl2wAeEgH/IkS34gxutx9i6F9sqpVTj4tkBD0fPR7N/PYgPxHRyX01KKdUAeEHA15qPZt8qWPoqtB0G/sHurUsppdzMCwLeNl1B3l6YOQmaxMAlb7q7KqWUcjvvCPjsVHhjJFSUwOTZVsgrpVQj5x0BX5Zn3QDkhgUQq7fkU0op8PRx8AB9roKgCDjjzxDYxN3VKKVUg+H5Ad+iu/WllFLqCJ7fRaOUUqpeGvBKKeWlNOCVUspLacArpZSX0oBXSikvpQGvlFJeSgNeKaW8lAa8Ukp5KTHGuGfHIpnA7lP89mgg64RbuU9Drk9rOzUNuTZo2PVpbafmWLW1McbYNeGW2wL+dIjISmNMsrvrOJaGXJ/Wdmoacm3QsOvT2k6NI2rTLhqllPJSGvBKKeWlPDXgp7u7gBNoyPVpbaemIdcGDbs+re3UnHZtHtkHr5RS6sQ8tQWvlFLqBDwu4EVktIikiEiqiEx1cy2tReQnEdkiIptE5C7b+kgR+U5Ettsem7mxRl8RWSMiX9qW24rIMlttH4lIgBtrixCROSKy1XYMBzeUYyci99j+TzeKyEwRCXLXsROR/4lIhohsrLWu3uMklhdtvx/rRaSvm+p7yvb/ul5EPhORiFqvPWirL0VEznN1bbVeu19EjIhE25ZdeuyOVZuI/Nl2bDaJyJO11p/8cTPGeMwX4AvsANoBAcA6oKsb62kJ9LU9DwO2AV2BJ4GptvVTgSfcWOO9wIfAl7bl2cBE2/PXgFvdWNu7wI225wFAREM4dkAc8DsQXOuYXeuuYwcMA/oCG2utq/c4AWOBrwEBBgHL3FTfuYCf7fkTterravu9DQTa2n6ffV1Zm219a2AB1rU40e44dsc4bmcB3wOBtuXY0zluLvulcdABGQwsqLX8IPCgu+uqVc/nwCggBWhpW9cSSHFTPfHAD8BI4EvbD25WrV+8I46ni2sLt4Wo1Fnv9mNnC/i9QCTWXc++BM5z57EDEusEQb3HCXgdmFTfdq6sr85rFwMzbM+P+J21hexgV9cGzAF6AbtqBbzLj109/6+zgXPq2e6UjpunddEc+sU7JM22zu1EJBHoAywDmhtj9gPYHmPdVNbzwF+AGttyFJBnjKmyLbvz+LUDMoG3bV1Ib4pIKA3g2Blj9gFPA3uA/UA+sIqGc+zg2MepIf6OXI/VMoYGUJ+IjAP2GWPW1XnJ7bUBHYGhtq7AhSLS/3Rq87SAl3rWuX0YkIg0AT4B7jbGFLi7HgARuQDIMMasqr26nk3ddfz8sD6evmqM6QMUY3U1uJ2tP3s81kfhVkAoMKaeTd3+s1ePhvR/jIg8BFQBMw6tqmczl9UnIiHAQ8Aj9b1czzpXHzs/oBlWF9EDwGwREU6xNk8L+DSsvrND4oF0N9UCgIj4Y4X7DGPMp7bVB0Wkpe31lkCGG0obAowTkV3ALKxumueBCBE5dLN1dx6/NCDNGLPMtjwHK/AbwrE7B/jdGJNpjKkEPgXOoOEcOzj2cWowvyMicg1wAXCFsfUr4P762mP94V5n+92IB1aLSIsGUBu2Gj41luVYn76jT7U2Twv4FUCSbTRDADARmOeuYmx/Wd8Cthhjnq310jzgGtvza7D65l3KGPOgMSbeGJOIdZx+NMZcAfwEXOrO2mz1HQD2ikgn26qzgc00gGOH1TUzSERCbP/Hh2prEMfO5ljHaR5wtW1EyCAg/1BXjiuJyGjgr8A4Y0xJrZfmARNFJFBE2gJJwHJX1WWM2WCMiTXGJNp+N9KwBkocoGEcu7lYjTFEpCPW4IMsTvW4OfMEgpNOSozFGq2yA3jIzbWcifUxaT2w1vY1Fquv+wdgu+0x0s11juDwKJp2th+MVOBjbGfr3VRXb2Cl7fjNxfpo2iCOHfBPYCuwEXgfa/SCW44dMBPrXEAlViDdcKzjhPVRfprt92MDkOym+lKx+owP/V68Vmv7h2z1pQBjXF1bndd3cfgkq0uP3TGOWwDwge3nbjUw8nSOm17JqpRSXsrTumiUUkrZSQNeKaW8lAa8Ukp5KQ14pZTyUhrwSinlpTTglVLKS2nAK6WUl9KAV0opL/X/lW4KeNKDpY0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot accuracies\n",
    "plt.plot([-x for x in clf.history['train']['metric']])\n",
    "plt.plot([-x for x in clf.history['valid']['metric']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (7) Evaluate and Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BEST VALID SCORE FOR forest-cover-type : -0.8769435318871541\n",
      "FINAL TEST SCORE FOR forest-cover-type : 0.877946352503808\n"
     ]
    }
   ],
   "source": [
    "# To get final results you may need to use a mapping for classes \n",
    "# as you are allowed to use targets like [\"yes\", \"no\", \"maybe\", \"I don't know\"]\n",
    "\n",
    "preds_mapper = { idx : class_name for idx, class_name in enumerate(clf.classes_)}\n",
    "\n",
    "preds = clf.predict_proba(X_test)\n",
    "\n",
    "y_pred = np.vectorize(preds_mapper.get)(np.argmax(preds, axis=1))\n",
    "\n",
    "test_acc = accuracy_score(y_pred=y_pred, y_true=y_test)\n",
    "\n",
    "print(f\"BEST VALID SCORE FOR {dataset_name} : {clf.best_cost}\")\n",
    "print(f\"FINAL TEST SCORE FOR {dataset_name} : {test_acc}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or you can simply use the predict method\n",
    "\n",
    "y_pred = clf.predict(X_test)\n",
    "test_acc = accuracy_score(y_pred=y_pred, y_true=y_test)\n",
    "print(f\"FINAL TEST SCORE FOR {dataset_name} : {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (8) ML Explanation\n",
    "\n",
    "##### Global explainability : feat importance summing to 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.06985060e-01, 2.05323116e-02, 8.68474068e-04, 7.32807603e-02,\n",
       "       1.06426423e-02, 7.81555935e-02, 1.22019225e-02, 1.87041664e-02,\n",
       "       1.67281014e-02, 3.83164078e-03, 7.75130077e-02, 2.43340951e-03,\n",
       "       1.09689558e-02, 5.94868120e-04, 4.12367991e-03, 6.77477575e-03,\n",
       "       8.66506355e-03, 1.76662957e-02, 8.33596323e-03, 1.35711919e-02,\n",
       "       6.68019313e-03, 2.82436978e-03, 1.44143876e-02, 2.10648958e-02,\n",
       "       9.58632393e-03, 5.10320237e-02, 1.63746540e-03, 2.73632031e-03,\n",
       "       2.65570922e-03, 2.12177448e-02, 2.18405046e-02, 3.22798531e-04,\n",
       "       2.70298286e-03, 3.81003808e-03, 6.26933684e-04, 1.54110747e-02,\n",
       "       7.34738564e-02, 1.56653783e-02, 1.09570278e-02, 5.22872951e-06,\n",
       "       8.67283266e-03, 8.37724958e-03, 2.26903581e-02, 1.50456830e-02,\n",
       "       5.19529021e-03, 2.67514849e-02, 1.43090885e-02, 2.96755618e-03,\n",
       "       2.34175126e-04, 1.35612338e-03, 2.87748463e-03, 1.10573521e-02,\n",
       "       2.11722111e-04, 9.01045860e-03])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.feature_importances_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Local explainability and masks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "explain_matrix, masks = clf.explain(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABIEAAADhCAYAAABfjaWIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcXFWZ//Hv09Xp7nT2fY8JkCAIBASy6YwKOijqiKP+xmVm0NFBHXUcR38qzuoybqOoL1QURYXf4DCCiuiADLKoCCQESMISSAiE7CvZO1tXnd8fXblVt9JVXVV9695TdT/v1yuvnNP3Vt2nqvupqj59znPMOScAAAAAAAC0trakAwAAAAAAAEDjMQgEAAAAAACQAgwCAQAAAAAApACDQAAAAAAAACnAIBAAAAAAAEAKMAgEAAAAAACQAgwCpZiZvdzMNiYdB4AwchPwE7kJ+IncBPxEbvqJQSBUzczebmbPmdlBM7vZzMYmHROQdmY2xcxuMbPNZubMbFbSMQGQzOy1Znavme0xs61m9j0zG5F0XEDamdkrzOzRfG7uMrOfm9m0pOMCUGBmP8x/rj0l6VhaEYNAqIqZvUjSdyX9paRJknokfTvRoABIUk7SryW9KelAAISMkvQ5SVMlnSZpuqT/SDQiAJL0hKSLnHOj1ZefayRdlWxIAI4zs5dKOjnpOFoZg0CeMrN1ZvZ/zWxlfubNNWY2ycxuM7P9ZvYbMxtTdP6N+b807jWz3+UHbY4fu9jMnsjfbpOZfazMNf8uf970fg6/Q9IvnXO/c84dkPTPkv6Mv2oibXzLTefcNufctyU92JAHDDQJD3Pzx865XzvnepxzuyV9T9JLGvHYAZ95mJvbnHObi76UlcRsA6SOb7mZP94u6UpJH4z68aKAQSC/vUnSqyTNlfR6SbdJ+pSk8er73v1d0bm3SZojaaKkhyVdX3TsGknvdc6NkHSGpLtKL2Rm/yzpnZJe5pzrb93miyStON5xzq2VdDQfG5A2PuUmgAKfc/OPJT1e28MBWoZXuWlmM81sj6RDkj4m6cuDeGxAM/MqNyV9RNLvnHMr639IGEh70gGgoiudc9skycx+L2m7c+6RfP/nki48fqJz7gfH22b2b5J2m9ko59xeSccknW5mK/J/jdxddA0zsyskzZf0ivz5/RkuqfTYXknMBEIa+ZSbAAq8zE0ze5WkSyUtGOwDBJqUV7npnFsvabT11bf8G0lPRvQ4gWbjTW6a2QxJ75V0bpQPECdiJpDfthW1D/XTHy5JZpYxsy+a2Voz2ydpXf6c8fn/3yTpYknPmdlvzWxR0f2MlnSZpC8M8EH2gKSRJV8bKWl/DY8HaBU+5SaAAu9y08wWSvqxpDc751bX8ZiAVuBdbkqSc+55SddK+kV+GQqQNj7l5tclfYbPvY3HIFBreLukN0h6pfoKUc7Kf90kyTn3oHPuDeqbunezpJ8U3Xa3pNdJ+qGZVapV8Likecc7ZnaSpE5JfKAFyosjNwHULpbcNLNzJN0i6a+dc3dG+QCAFpXE+2Z7/v5K/9gJoCCO3LxQ0n/k6w5tzX/tfjN7e2SPApIYBGoVIyQdkbRLUrekzx8/YGYdZvaO/FS9Y5L2qa8AXsA5d4/6Cj//3MzKTVW/XtLrzeyPzGyYpM9I+plzjplAQHlx5KbMrEt9g7KS1JnvAyiv4blpZmeob+e+DznnftmQRwG0njhy88/M7FQzazOzCZKukPRIflYQgP7F8Zl2rvomHZyd/yf11Sn6eYSPA2IQqFVcJ+k5SZvUt+3lAyXH/1LSuvzUvfdJ+ovSO3DO3SHpXZJuMbMT1mE65x7P3/Z6SdvV90LwtxE+BqAVNTw38w6pb8mm1FfX4NDgQwdaWhy5+VFJEyRdY2YH8v8oDA1UFkduTlPfAO1+SY9Kykl6Y1QPAGhRcfy+ud05t/X4v/yXdzrn+FwbMXPOJR0DAAAAAAAAGoyZQAAAAAAAACnAIBAAAAAAAEAKMAgEAAAAAACQAoMaBDKzV5vZU2b2tJl9MqqgAAwOuQn4idwE/ERuAn4iN4Ho1V0Y2swyklZLepWkjZIelPQ259wT0YUHoFbkJuAnchPwE7kJ+IncBBqjfRC3nS/paefcM5JkZjdIeoP6tozrV4d1ui4Nq+rOs+MK52V2HQwfG1t07PnwMXV3hfs9h6u6nqykXzQ2Zu2Z8KHebHX3OdAl2wtPv+vtrf52Q4aE4zl2LJp42goTw1wuF8l9ouCwDuqoO1L6k9YIDc1NoBEmn9ETtLc+1h06Zp0dob47cjTSa/ucm5kRw1z7+DFBv3NdT7lTY5EbXXidaNtzsMKZiMywoYX2wXTtkutzbkb2vjmi6PVuf7L5nYThp4c/bx54gkoVzcDX3KwpL4cXvbYeCL+22pCi39GOVf87Whr0Tgg/v+07/P0scGxyONYhW/2NNSr7tXunc27CQOcNZhBomqQNRf2NkhaUnmRml0m6TJK61K0FdmFVd77ndYuC9ujr7g8d2/eahUF75I8fCF/vRWeE+m7ZY1Vdr3hARgoPymRGjw0dy+56vqr7lCS1FQ0g5cKDR5nxEwv3uW171XfZPnlaqN+7cVP18VTQNrTwQSTXk74PIo22xN0Z16UamptAI3z8lkeD9pdPPjN0LDNzdqifffrZSK/tc25mxo3WlE9/MDg2550PNTjEynouLITb/bMlCUaSHm7evKBt969IMJL4+ZybUb1vZs9/cdDO3P3woO+v2Sy8IfyHzAfmDSlzJnziU27Wm5e5c84O2m33Lg8dax8/KWj3bt1WW8QtbsdbFoX6E666v8yZydv0rsWh/rQv3pdQJPH5jbvpuWrOG8xwe3+jvyesLXPOXe2cO885d94QdQ7icgCqRG4Cfqo5NzMjmKEHxID3TcBPA+YmeQnUbjAzgTZKmlHUny5p8+DCKSid/VOsdPZPsWpn/pxwuwrLsWqa+VMqV37pWC2zf4pFNfOnfcrk8P1u2RrJ/dbL7irMcHIXhB/jkdecH+p33vZgLDE1qYbmJlrD4dfPD/W7frk0oUj63LBzYVGvZAlwxDN/ElRfblp9tfsaoe2YP7GkhcsUfgeKY+1FSiX2vpn91K6gnbk7jiv65bav/XGoP0b+zipAIhqWmxsvKKyAmHlv+Bizf8o7OD3cH3DdUYJm/jT8u200BV1aw2BmAj0oaY6ZzTazDklvlXRLNGEBGARyE/ATuQn4idwE/ERuAg1Q90wg51yvmX1Q0u2SMpJ+4Jx7PLLIANSF3AT8RG4CfiI3AT+Rm0BjDGY5mJxzt0q6NaJYELOkl3+VKl0CVqzS8q/MKY0tGjuQj699NNQvLWqbBHITA0l6+Vep7834Q9C+SGdXOLO51ZWbzp9FQL1d/sSSFm1HmcAeh6TeNzteVVUNz9jcvrlQIPeiqY1/LR7zI5Z/obJG5WZ2aPnlzTvfWyh+PP67/IwWm/GbI0mHULXsmmeSDsFb7MMIAAAAAACQAgwCAQAAAAAApACDQAAAAAAAACkwqJpASTl0SWFr46E311/XYt/bClsSj/yv8tvON0ruZecE7bbfPlL3/Rx9dWH79I5ft+bW6dZe8qNqhfHLOGoAtXV3h/q5np6g7UMNIFRh4Vnh/gMrk4mjCplJE0P97LbtCUUSn0q1JzZevjjUn/6F++q6xrYPFe5n0pX13UfcLJNT13B/1t+PvPWxoJ1LMI402XHu8KA9wa9SXmhBcdQBikIt7ws7L1sU6o+/uvVqvPT82YJQv/tnSxKKpHkM2Ve+xt3EB/YEbd7rwjZc0Bnqz7o7oUAwKMwEAgAAAAAASAEGgQAAAAAAAFKgKZaDbfin8JTPGZ+LZhp/EkvAig1mCVixapeArbn2xaH+nEsfjuT6cXC9vYlev3j5F5qUx8u/SrXK8q/irYal+pcZ1Lv8q1SzLAEr1tXeq1Mm7Az6iS8Mmz2j0H7syeTiSJGJS/cF7fIbGgPRiHuL+HrV8r7Qisu/SrH8q3bHRpZ/Rd2+cHTQHr8ijmiax4y7Ev8kgggwEwgAAAAAACAFGAQCAAAAAABIAQaBAAAAAAAAUqApagJ1b2neVfDts2YG7d516xOMJJkaQNYZ3kawbeTIoJ3dsSOSa2SK7lOSsvv2lTkzfs9+obAt6dErk61BlXbtJ80K9XufWZdIHGky99r3h/qzVb4uw/p/LdR+m/np5qvd0yhHsxlt3Dsq6E/Q1gSjkY5OGha02x+rcCIiY9nCZ6Dm/TSEZhFHHaBmqTuE1pY5XH6L+K7dvNqWc2BqR6g/qsx58BszgQAAAAAAAFKAQSAAAAAAAIAUaIrlYOOuaZ6tHVdffX6oP/ey6rZvH0hmzklBO7vmmUjus9SO9xeWLk24Kprn3B0JbyMY1RKw0H16tPyr1OzLC8/jdncwwUjQTMu/ipcRSuGfo6isvmp+qD/3/Usjv0YtcbMErH/ZbJv27u0O+hMSjEWShuw7GrSZLB+PAyeNCNrdyyucCDSJ4iVgxUvDSo8BjdRR4deHnomFeRLDY4ilmYy6vnnKWxx884JQf9hNSxKKxD/MBAIAAAAAAEgBBoEAAAAAAABSgEEgAAAAAACAFGiKmkClotp2/fDrCzUxun4ZTT2MWmoA7X5noe7HmB9Vrp1RqQ5QVLV8oqoDFAdrL/zout7eBCMBoteIGkClGlEDqFHSXDPCTOro8uc1ru2p54J2NsE40qRnQuHvdd0VzgOaUdxb0sd1Tfive1uu7LED0wtV7ybGEUwTOfy6cE3Jrl/5+3mSGkDlMRMIAAAAAAAgBRgEAgAAAAAASAEGgQAAAAAAAFKgKWsCDaYOULGo6gBVq336tFDfWTT3W1zLp33a1NCx3k2bo7mIZ+KuA7T6qvD612aqpwJUY/f/zAnaY167Jvbr73xvobbZ+O+GayKluX6Dy5kOH+hIOozAs39/RtCe+Zn7EowkPUrzAajVrKVDg/a6+YcSjOREcdQXSfN7CMrb/cLCL2IjS46d8p+7g3b5ykHptPPM8PDB9F8lFEgVimvIStSRLcZMIAAAAAAAgBRgEAgAAAAAACAFEl0O5hbPC9p234pI7rN98qRQv+fswnbyHb+ufvv2qNi5LwravQ89Hjo29oebIrlGZsKEwjUatPwrM25s0M7uer76200Kb6yY3bY9spjiVO/yr2OvPDdouweY0p+k3MvOCfXbfvtIQpH44Y9WHg71f39W/EvAirHkpX+d63o0910PJR1GgCVgiAJbdsfLtyVgxXzeXto35E202o6Vr8thW3fEGElzOTQlm3QIVWP5V3nMBAIAAAAAAEgBBoEAAAAAAABSYMBBIDP7gZltN7PHir421szuMLM1+f/HNDZMAKXITcBP5CbgJ3IT8BO5CcTLnHOVTzD7Y0kHJF3nnDsj/7UvS3reOfdFM/ukpDHOuU8MdLGRNtYtsAuDfvHaVta1otUtcXdqn3u+/ALkGjUyN4G4Jfl+QG4C1Ym7JonPuXnevC639PYZQZ/PsUgTX3OT98zWRD2s6v3G3fSQc+68gc4bcCaQc+53kkorAb9B0rX59rWSLqk5QgCDQm4CfiI3AT+Rm4CfyE0gXvXWBJrknNsiSfn/J5Y70cwuM7NlZrbsmI7UeTkAVSI3AT+Rm4Cf6srNHbuaZ4ccoElVlZu8ZwK1a/gW8c65qyVdLfVN0Ss+Vu9UruJt113JtuvP//WiUH/sD6rbdjhz+txQP/vE6rpiK9XW3V3onDQzdCz32JORXCMq2Ze/OGhn7nk4wUgGp97t7NOmUm6i9bV1dYX6ucOHy5wZj705f7cwjltxbnbOnu7WfPrc4Nicdya7XfzBNy8I2sNuWpJgJCky/8xCe+mjycUhpuCXvm+m/fmIAss8MFj1fp7d8M+Lg/aMz94XOnb49fODdtcvlw42xJZiQzpCfXfsaMOvWe/rwvPvKhkX+GF14wJpUO9MoG1mNkWS8v9vjy4kAINAbgJ+IjcBP5GbgJ/ITaBB6h0EukXSpfn2pZJ+EU04AAaJ3AT8RG4CfiI3AT+Rm0CDVLNF/H9Jul/SqWa20czeLemLkl5lZmskvSrfBxAjchPwE7kJ+IncBPxEbgLxGrAmkHPubWUOJbb/XmkdoGLV1gAqFVUNoKMXhXdk67h9WaFTQw2gthEjQv3c/v2DiqsaSdcB2vBPRWtzP3dfhTPDdr07vN5z3DXpWO/pY26ieWy/9JxQf/x3k82bV634q6A9RmsSjGTwIs1Nk9TmT8mu9p5c0iGkTm5o4aNavdO30Yf3Tf/EUQOIukP+SyI3xz/aW/bY+jcU3uvm/rJRETSn3peeEepn7va3jiw1gMrj8wQAAAAAAEAKMAgEAAAAAACQAgwCAQAAAAAApMCANYFQm1ANIEmrvzM/aM9939Kq7yeOGkC+qaUOULHSGkD737owaI+44YFBxTRYuZcWrTt/pL7Hh+q1z5oZ6veuW59QJPHJTJoYtLPbqt89ddJPwjXKspFFVJ9XTC3UAVpe4TwkK9vF347i1naofN0KoBkV1+iJoz4PNYDQnx3zCr8Gz7w5fGzq//IrcjkHpnaE+qMSigODw6c5AAAAAACAFGAQCAAAAAAAIAWacq6bWzwvaNt9K+q+n+Jt2Bu1/KrSErAd7y9sbT7hqvq3sGufNjVo927aXPf9+Mzawz+qrrcwPX7ne8NbxMex1XVxPMWxlGq7t2hhizvUyJAgv5d/HXjLglB/+I1Lyp476t5xof7el+4qe24tS8BCt9u9u67bNcrycwY+J5VykjuUSTqKQNeOo0mHkDoHXtAdtIdXv6oc8Fbcy7PYIh79yRwuf2z7eRa0h/8khmCayLi71oX6Pi9YzowZE+r79tk3ScwEAgAAAAAASAEGgQAAAAAAAFKAQSAAAAAAAIAUaMqaQMV1gDIjR4aOZfftq/p+nvrmnKA959KHBx9YjQZTB6jYhj+fFbSnXNGaNYEq1d2JowZQqUrxAP2pVAOoVKUaQGmU5noObe05jZx0IOkwAm2/fyTpEFKnZ1KhJtTwBOMAotI+eVLQ7t26reHXa5X3jCvWhT/v/sOsRWXORDVyQ8ofm/uN54I2n/jDnvz4rFD/lI9sTSaQKlADqDxmAgEAAAAAAKQAg0AAAAAAAAApEOtysLln9ej22wvT+qOYnlnL8q9SSSwBa4QpV9yXdAhIqTQv00F80vxzNWTtYU2+ZFXSYQSe+WJh+cFJn4x/KW4a7T0tG7QnJhgHEJXiJWB8jqgey7+iNeRg+WOb3zgraE/8ZmuW2qjXKR95IOkQqnbwzQtC/WE3VV+aodUxEwgAAAAAACAFGAQCAAAAAABIAQaBAAAAAAAAUiDWmkCrV3ZHvtY3c8rsUD/79LOR3K8N6Qja7tjRSO4TlW356OKgPeWr4TpHxd8PKfw9eebL4TXSJ32cOhVxYe0+alFc+4Gfneq4OR068u1ZQb/zT9YlFoskTb03O/BJiNRpn98QtNmqGK2A9wL44OjI8se6t+fiC6TJ7H3HwlB/1PX+1giiBlB5zAQCAAAAAABIAQaBAAAAAAAAUiDW5WC12P/WwlSzETeEp5ltvLywbGj6F8LLhtonTwr1i7ehrEVUS8Dahg0L2rmDFfYiHITM6XODdvaJ1Q25RhxKl4AVy0wt+b4+V5gez/IvoHYzlwwL9dcvaMzrU7FK0/4Pv35+0O765dKGx9IsjvZmtGHbmKB/itYlF4ykzt0sj47bgXOmB+2uTWxVjNptufm0oD3lklUJRtKn+L1g24cWh45NurL8Z0EgSrNu3h20Sxd/Df+Jv0uckvb863tC/VHXJxRIFXreGN4ivvvnLA87jplAAAAAAAAAKcAgEAAAAAAAQAowCAQAAAAAAJAC3tYEKq0DVKy0DlCxDe84OdSf8tX6agJFpbgOUPusmaFjvevWR3KN4jpAa74ZXvs454OtsfaxuAYQgMGLowZQLX773auD9kW/ZMvggDO5rD9/rzk4tTNoD08wjjTp2n4o6RDQQAfeUvjcNvzGxnxm86EOUDlz//ypUH/vlQkFgtTJrSjkxe2bl4eOVaphmHaz37oy6RCqRg2g8vz5ZAkAAAAAAICGGXAQyMxmmNndZrbKzB43sw/nvz7WzO4wszX5/8cMdF8AokNuAn4iNwE/kZuAn8hNIF7VzATqlfRR59xpkhZK+oCZnS7pk5LudM7NkXRnvg8gPuQm4CdyE/ATuQn4idwEYjRgTSDn3BZJW/Lt/Wa2StI0SW+Q9PL8addKukfSJxoSZQ1WfvTbof5FX/VnTWdUNYAqaZUaQBhYs+Um/NJ2xgtD/dxjTyYUSZ+fHBiV6PWjFHlummtEmHVp6006gvRxmcLf6yzBOFqBj++bjaoDVK/i2ihx1EXZ+9JdDb8G/JdEbj77hUVB+6Kp4WM731s4Nv6790dxuZaRfcWLQ/3M3Q8nFAkGo6aaQGY2S9I5kpZImpRP2OOJOzHq4ABUh9wE/ERuAn4iNwE/kZtA41U9CGRmwyX9VNLfO+f21XC7y8xsmZktO6Yj9cQIoAJyE/BTFLmZ3e/XLm5AK+B9E/BTPblJXgK1q2qLeDMbor6EvN4597P8l7eZ2RTn3BYzmyJpe3+3dc5dLelqSRppYyOZ0775Y4uD9tSvhLeLj2rqamZSYaA5u63fh4aIZV9emF6YuSc8tbB06cqhGSOCdudtDzY2MI/5lps+W/3d80P9ue9N78+NJD3z1nBtxVn/lFAgedfMnZ1sABGLKjc7Z85w2tMRS8zVGLr1cNIhpM7zp3cH7XGsShi0pN83fd+K2rd4kB715ma9eTl0a/kFtiwBK2/TH3WF+jPvTiiQKvRecG6o337XQwlF4p9qdgczSddIWuWcu6Lo0C2SLs23L5X0i+jDA1AOuQn4idwE/ERuAn4iN4F4VTMT6CWS/lLSo2Z2/M8Xn5L0RUk/MbN3S1ov6S2NCRFAGeQm4CdyE/ATuQn4idwEYlTN7mD3qvyGFBdGGw6AapGbgJ/ITcBP5CbgJ3ITiFdVNYF809s98DmD5VsdoMzIkUE7u698nbTeC0vWPt5Z/drH9imTC/ezZWsN0UWjtA5QMdu6I9TvjHk7a7ZDbH5prwFU6qQb94T6uYTiOO6vntoQtK87dUaCkXgm46TRR5OOInBocqEWQAxvxZA0dlVP0iEgQtTcCfO9RhJa177TeoP25JJjbcOGBe3cQTZoKDbzM/cNfJIntrykM9SfcVdCgXiopi3iAQAAAAAA0JwYBAIAAAAAAEiBRJeDXbGusP3eP8xaVPXtmmkaWlQqLQErVsvyr1JJLAGrVnbnrkSvX2n5V/vkSaF+79ZtjQ4HGLTcilVJhxDyjhGFHL9OLAc7rnNdj+Zc6s/y0+6fL0k6hNSx+1YkHUKApTsFc8/q0e23F56PND8Xg8HzhqQMe7b8r8FPf29O0D7p7cvLnge/zfhs+sYMqsVMIAAAAAAAgBRgEAgAAAAAACAFGAQCAAAAAABIgURrAtVSB6ictq6uUN9GjQz13eRxQbu0Bkbm9LlBO/vE6kHH0h+3aF4htvv9Wdc/EBvSEeq7Y/5sUVzqyGvPD/U7/6e6rcCPvKbkdrfVt4U4NYBQjcz4caF+pTpXW28+LdSffIlf9XuK7fnL8Ov46P93f5kzK6MuRP+OzB6qtZ8vPDcnJ1yboG3EiKCd278/wUjSY+uHFwftyd9Itr4BeVqwemU3z0eToJYV+jPqmWzZY9ljzJMox84/M9R3Dz6aUCQYDH7CAQAAAAAAUoBBIAAAAAAAgBRIdDlYFHKHD4f629/z4lB/4jfLT51u1BKwYnEvAYtqGYnPy79KVbv864Tb1bn8C6hHpeVfpXxe/lWq3uVftUj1VP6cKXtwSNJRBLJnnRy07Q9smxuH4VtySYeAFCl+vW2V19pWeRyI1uaLCsvB5t4YPta5tkvoXzMt/7L28FCH6+1NKBL/MBMIAAAAAAAgBRgEAgAAAAAASAEGgQAAAAAAAFLA25pAxWv4alm/N/Fbja9PkbTibeelcN2hqGqJbP3I4lB/8tfq25a2bV64RtHuM0cF7VH/+UBd9wlg8Jqpzo7PsTXakI5ezZi5M+kwAofHdwTtoQnGkSaj/7A+aFPNAPVoO+uFQTu38smK58b9evuup54L9X946gtivT7Sa+TjHWWPDdvsYoykuWTGjwv1a6l5Gbee14ZrBQ/9xdKEIvEPM4EAAAAAAABSgEEgAAAAAACAFGAQCAAAAAAAIAW8rQlUSx2g8A3DazhXf2d+0J77vvA6wOf/elHQHvuD5qklVFwDqFHqrQFUKrciXKNoVANCt3NeFOq7Rx6P/iLAYJmF+y7Z9ebNVGfn1k0Ph/oXT3txmTNbTzbXpr2HuoJ+0nV4WE8fv81vnBW0J35zc3KBoGkNVAcoSXHUAGqmGniIz6hny/+uuWvhsaA97vtxRNM8jp4RztnMPf7WBOIzS3nMBAIAAAAAAEgBBoEAAAAAAABSwFyMSxJG2li3wC4M+sXTM5maiVa3xN2pfe55G/jM+JXmJhC3JN8PyE2gOnEvq/E5N8+b1+WW3j4j6PM5Fmnia27yntmaWNJZvd+4mx5yzp030HnMBAIAAAAAAEgBBoEAAAAAAABSgEEgAAAAAACAFEh0i/hGrOc78OuTQv3hr34m8mv4bNd7FoX6475/f0KRAAAG68isbq359LlBf847H0owGunAWxYE7eE3LkkwkhSZf2ahvfTR5OIQdRiKrV7Z3ZLPB/U6kRarv18omzL3PctCx9b/y+KgPfMz98UWE/pX72uRtYeHOlxvbxThtARmAgEAAAAAAKTAgINAZtZlZkvNbIWZPW5mn85/fbaZLTGzNWb232bW0fhwARxHbgJ+IjcBP5GbgJ/ITSBe1SwHOyLpAufcATMbIuleM7tN0j9I+ppz7gYz+46kd0u6KqrAdryvsKxpwneqX9Lk2/Kvnj8rTJ3v/lnjp84nsfxr0ycWh/rTvsS0yZgkkpvwS/tJs4J27zPrqr7d0YvCu0d23L6szJnxePljlwTtTq1LLpBoRJub5hoabC3aD/sTS1rkOjJBm+nbg8b75gDiXgLG1s/Iiz03T//stqBdukgoO5T3unLap0wCogcwAAAWwElEQVQO9Xu3bE0okoFt+eD8UH/y1/kd9bgBP0+4Pgfy3SH5f07SBZJuyn/9WkmX9HNzAA1CbgJ+IjcBP5GbgJ/ITSBeVf1RycwyZrZc0nZJd0haK2mPc+74wOlGSdPK3PYyM1tmZsuO6UgUMQPIIzcBP0WVm9n9B+MJGEgJ3jcBP9Wbm+QlULuqBoGcc1nn3NmSpkuaL+m0/k4rc9urnXPnOefOG6LO+iMFcAJyE/BTVLmZGTGskWECqcP7JuCnenOTvARqV9MW8c65PWZ2j6SFkkabWXt+dHa6pM1RBlapDlD75ElB++ipU0PH2n77SJRhSJJsSLgGmTt2tOrbxlEHqBG2fyBc52fit8qvoYyjBlBm5MhQP7tvX9DOveyc0LFG/AyUaptXeF/KrVjV8OsNJM7cRONlJkwI9d2UcUE7t/LJ0LFa6gAVi6oG0Pobzwz1Z76l/BbWuT8q5Grb78N52vkn6yKJxzeDzk1zahuSa2yQNRh2/9qgnU0wjjTZeVZ30J54b4KBtBjeN/1Qbw2gNd9aEOrP+UD5z9vHXnluqD/kNw/VdU2fHXhL+PkYfmNz/v4hxZebGy+ZEbQnf2ND6NjEZf687/pm45+fFOpP/rq/NYGoAVReNbuDTTCz0fn2UEmvlLRK0t2S3pw/7VJJv2hUkABORG4CfiI3AT+Rm4CfyE0gXtXMBJoi6Vozy6hv0OgnzrlfmdkTkm4ws89JekTSNQ2ME8CJyE3AT+Qm4CdyE/ATuQnEaMBBIOfcSknn9PP1Z9S3XhNAAshNwE/kJuAnchPwE7kJxKummkCNtOtvFoX6475XviZQ79ZtQbutqN0otdQAapR9t50ctEe+Zm2FM6NRqQZQEoprAJWKowZQKR/qAKE2t29eHurXW4cgDtkdO8JfKO17pFINoFKldYAwMDvcpiFPdQ98YkyOnTYzaLf9fleCkaRHjjqnwAkq1QAqddd14ckjPr//16uZawAlZdRzvWWPbXpNoerd3J/GEU3zmLDicNIhIAJV7Q4GAAAAAACA5sYgEAAAAAAAQAp4sxys0vKvStZ+ZWGof/LHHqj6tm3DhgXt3MGDdV0/LpWWgLXPKkzP3/Cm6aFjU77q17IuICmtOP271LP/NS/Un/22FaH+5o8vDtpTv8xrQzNwXU5HTvZn6vXBaYW1SSMSjCNNpt61O2izaTFQuzS8/zfTkndf7Dyj8GvwjJvDx4av7og5muZhx3gnagXMBAIAAAAAAEgBBoEAAAAAAABSgEEgAAAAAACAFPCmJlAt2s4+PWjXUgOoVLV1gNq6w9vz7nhHuO5GvfWMotK7bn3QnvLV9RXOrF5xnaHSa/im7awXhvq5lU82/JoH37wgaA+7iW054YfSGkClGlEHaPPHFof6U79S/hpPf61Qw+2Uj4RfuzNzTw7a2dXla6CljbXl1NV9NOkwAiPX7A/aLsE40uTQ1OFBu7NyigPoRxrq5bTiY2q04RvLv4v1dsUYSJM5NKkz1B9W5jz4jZlAAAAAAAAAKcAgEAAAAAAAQAo05XKw3PInyh6rZWlC1dfr6Qn1G7H8K6q4j7zm/FC/87YH67ofn5d/lYpj+VcploABfWp5rSpdAlbsrbf8Nmhf/8LpoWPFS4Clyu8BrcZl23T4QOfAJ8bk4MzC0qTuhxIMJEU69vqzHBDRm/NgIb/XnH8kwUj6HLm48Dmy89b6PkP6hqVS6M95H3gkaK/9UfjYzM9Ev3y+VbQfZov4VsBMIAAAAAAAgBRgEAgAAAAAACAFGAQCAAAAAABIgaasCZQ59ZSgnX3q6dCxKGoAJSGquOutASRJz35+UdCe/an4t73f+pFCXaTJXws/H5nRo0L97J69scQENJO9t54S6o+6+OkyZ/rnG1e8JWiPV/j1J001gPrjPNqL/dDYwt+OuhOMI03ad+wP2tkE40Bj+FAHqFgcdYCKt2ynXg+Scsfd5wTtk0o/d7y08HPZdu9yoWD/9PDwgT9VC1ELZgIBAAAAAACkAINAAAAAAAAAKdCUy8FKl4DV6+irC9tgdvw6/m0wj73y3KA95Df177W7/t8Ky6hm/lv9y8qSWAJWrHQJWLHS5V9tZ70waCexRTzgo2Za/lVq/NXlX3+Klw5I6Vo+YG1OncP82SJ84s1PBW2WJsVj2ysmBe3xa55JMBK0oszck0P97Oq1Db9mml7Do7TpE4tD/WlfKnxuXvuVhaFjJ3/sgVhiambdm6zssSHP9wRt3uvCdp8eXqM+PqE4MDjMBAIAAAAAAEgBBoEAAAAAAABSgEEgAAAAAACAFGjKmkBu8bygbfetCB3b+vfh9bKTv16+zkzsdYDaMqHuYOoAFSuuA3TwTQtCx4b9dEnV99M+ZXLQ7t2ydfCBNVBxHaD2yZNCx9zhw0GbreSB5lBpy+A0149wOdPRno6kwwjsvWBO0B5+Y/XvL6jf+OUHkg4BLSyOGkCVpLnmW62KawCVogZQ7Y6MLX+sZ9aooN35RAzBNJE5H26en7Vdf7Mo1B/3vWTr3/qEmUAAAAAAAAApwCAQAAAAAABACjAIBAAAAAAAkAJNWROotA5QsdIaQEcuPj9od94argFk7YWH73p7I4qugly2IXe7/YOFOkgTv1l+vfBAGlEHKHPanFA/u2pNJPfbPvsFQfvY1DGhY/aH5aWnA9p4ebhe2PQv1J8rreDp/zwn1D/lLx5JKJI+1IHoX+e6Hs15ZzT146JAHaAELH006QgiRx0YHMf3vnrkTbSswq9l3UsKtbIa89tb81pzZbj+7JwP+fu5gBpA5TETCAAAAAAAIAWqHgQys4yZPWJmv8r3Z5vZEjNbY2b/bWb+bF8CpAi5CfiJ3AT8Q14CfiI3gfjUshzsw5JWSRqZ739J0tecczeY2XckvVvSVbVc3M55UdB2jzxey03LKt0uXLeW3wY+liVgMSheAtY+Y3roWO+GjXXdZ/usmeH7Wbc+aGcmTAgdy+7YUf6OtmwP329E29D3Pvtc0Laitm82/HNhCdLR7zVsS8XIc7MVpX35V6mkl3+VqrRFfBMbdG4emT1Uaz9feD5Ofnuyy133vX1h0B754+bZJraZZU49JWhnn3o6wUiik3CO856ZsBZ9vW+4FDxXseZmttuVPbbrtacG7dHXsaSo2MzbckmHgAhUNRPIzKZLeq2k7+f7JukCSTflT7lW0iWNCBBAeeQm4CdyE/APeQn4idwE4lXtcrCvS/q4pONDf+Mk7XHOHZ9Ks1HStP5uaGaXmdkyM1t2TEcGFSyAE5CbgJ8iyc3s/oONjxRIj7rzUuJ9E2ggPs8CMRpwEMjMXidpu3OueHsS6+fUfufUOeeuds6d55w7b4g66wwTQClyE/BTlLmZGTGsITECaTPYvJR43wQagc+zQPyqqQn0Ekl/amYXS+pS3zrNr0sabWbt+RHa6ZI213rxqOoAFevdui3U3/NXi4J26ZrOQ5fMD9pDb14aeSwD2f3OQmxjfhTNetN6awCdcD9FNYBKVawBVHrunr3hL5T2I9A277RQP7diVeTXqNeMzxbq0Gx2kf9Fv2G5iei1z35BqF9c12r1d+aHjs19X/yvR3Gw9sJbTmlNtkq1Drb/7eJQf+K3va/vFFluWptTR4c/9evG3MG2uXHb8ZJCHb6xLVITKCG8Z3oi7to2bK3uvURyc/zy8jWBdv7J4aA9+roor9r8hq7fH+pTIag5DTgTyDl3uXNuunNulqS3SrrLOfcOSXdLenP+tEsl/aJhUQI4AbkJ+IncBPxDXgJ+IjeB+FW9RXw/PiHpH8zsafWt27wmmpAADBK5CfiJ3AT8Q14CfiI3gQapZYt4OefukXRPvv2MpPmVzm+UQ28oWsb1i/CyifaTZoX6Xc+Xn7Ae9xKwzNyTQ/2oloAVO3Lx+aF+560PVn3b4u3lo1pWFgefln8lxZfcRHnFy79KJbH8K4np8aVLwKrVBMu/yhpsbrqc6cihIdEHVqfc9ImFTg1Lg1G/jv1Mto8a75n+iOO9iOVfzSPO3Nw5r1B2aMQN4WMnf5vX3XJyjz2ZdAhV2/7BknIC32zez5NRG8xMIAAAAAAAADQJBoEAAAAAAABSgEEgAAAAAACAFKipJpAvSusAFet9Zl2o786c2P+JknreuCBod/98yaDjGkh29dpQv+2sFwbt3Mpo1lfWUgOoVDPVASq29cPh9Z6Tv8F6T/inlp/T9f8aPnfmp6P/mU6iRkL25S8O2pl7Ho79+s3ITMoM8Wcz9rb9PUHbn6haW7bDBj4JaFJxvBexRTz6M3x9+WO7Lj8UtMe/PoZg0BDUACqPmUAAAAAAAAApwCAQAAAAAABACjAIBAAAAAAAkAJNWROo2Ob/G66dMfU/wmv/KtUPiqMOUCVPvn9k0J77/gQDaXJJ1ABqnzI5aPdu2Rr79dF8avk5bUQNoIFc+dwfgvaHXvCShlyjUh2gzIQJQTu7Y0dDrt+MXK/p2J6upMMIrH1X4bVv1j8+m2Ak6THq+geSDgEAWs7wLeUr2z2/rfA72vg4gmki2Ve8ONTP3E2Nx2bETCAAAAAAAIAUYBAIAAAAAAAgBZp+OVjp8q9Smz9eWC429cvhcw++qbBF/LCfxr80bO77yy9VS6PcSwtbdrbdu7zCmSXmnxnuL300oojKYwkYWk2jloBVq9ISsDRv79s2JKfuCQeTDiNw0r+vCNq5BONIkx3vXxS0J1x1f4KRoBXZeWeE+m7ZYwlFUps9f7Uo1B99XfncaJX3jPbp00L93o2bgva6fw8/H7P+kdeKgew5ufBr8NCSY7NujDeWZrJ1fniJ+rS7EwoEg8JMIAAAAAAAgBRgEAgAAAAAACAFGAQCAAAAAABIgaavCTSQ0jpAxUbesyZol98ksHHWXFvYYm/OpWyvV1MdoLZM0Ny2YETo0CRKLcFDaa5rM1hpfq5yRzM6vGHEwCfGJHvO3KBtf6jhNRt1yxxOOgK0smapAVSqUg2gUq3y/ltcA6gUNYBq197jyh7b98F9QXv87XFE0zymfalyPV40B2YCAQAAAAAApACDQAAAAAAAAClgzpWfChe1kTbWLbALg/5znylsZ/iCf2EaYyosPKvQfmBlcnHEZPPHFgftZ667Qoe2brAEwymrNDeBuBVP1497qv4Sd6f2uefJTWAAcS+r8Tk3z5vX5ZbePiPoN+sSI6AevuYm75mtqVWWdMbhN+6mh5xz5w10HjOBAAAAAAAAUoBBIAAAAAAAgBRgEAgAAAAAACAFEt0ivto6QEcvCi9r23lWR9Ce+h/RbFPXdsYLQ/3cY09Gcr9RyYwZE7Szu3cnGElt2qdPC/V7K9QByrzo1KCdffyphsUUp6lfKfx8bnAHE4wESUiyzk2z4fnpX3ZOp/ZdeXLQH/matQlGI+16T6GW37jvU8svFlZUdiPGOo79IU8LVq/sbprn4/+s2hq0f3La5AQjOZGd+6JQ3z30eEKRIG3W/0uhbufMz7DtebVW//DcUH/uux5q+DWb5bW2mTATCAAAAAAAIAUYBAIAAAAAAEiBRJeDVavj9mWh/pih8yO/RlTLv6w9/JS63t5I7jeOJWAbP1WYFjn989FMi+zduKnqc49MHh6020tmA2dOmxPqZ1etGVRcQBzinr7KFpqtp/doRls3F5YDj0wwFknKdni3I3DLa585PWj3PrchwUjQrHxbAhZeKp1gIEi1IxOzZY+1Tyv8YPZu2hxHOE1j5MrOpEOomnWGY3VHjiQUiX+YCQQAAAAAAJACDAIBAAAAAACkAINAAAAAAAAAKWAuxu1GzWyHpOckjZe0M7YLV+ZTLBLxVOJTLFLt8bzAOTehUcEMBrlZFZ/i8SkWqfnj8T03D6q5n99G8yken2KRmj8ecrM2zf79biSfYpH8iqeeWLzMTU8/z0p+xeNTLBLxVNKw3Ix1ECi4qNky59x5sV+4Hz7FIhFPJT7FIvkXTxR8ekw+xSL5FY9PsUjE02i+PR7iKc+nWCTiaTTfHg/xlOdTLJJf8fgUS1R8e0w+xeNTLBLxVNLIWFgOBgAAAAAAkAIMAgEAAAAAAKRAUoNAVyd03f74FItEPJX4FIvkXzxR8Okx+RSL5Fc8PsUiEU+j+fZ4iKc8n2KRiKfRfHs8xFOeT7FIfsXjUyxR8e0x+RSPT7FIxFNJw2JJpCYQAAAAAAAA4sVyMAAAAAAAgBSIdRDIzF5tZk+Z2dNm9sk4r52//g/MbLuZPVb0tbFmdoeZrcn/PyamWGaY2d1mtsrMHjezDyccT5eZLTWzFfl4Pp3/+mwzW5KP57/NrCOOePLXzpjZI2b2Kw9iWWdmj5rZcjNblv9aIt+rRiA3Q7GQmwPHRG7GhNwMxUJuDhwTuRkTcjMUC7k5cEzkZkzIzVAs5ObAMaUyN2MbBDKzjKRvSXqNpNMlvc3MTo/r+nk/kvTqkq99UtKdzrk5ku7M9+PQK+mjzrnTJC2U9IH885FUPEckXeCcmyfpbEmvNrOFkr4k6Wv5eHZLendM8UjShyWtKuonGYskvcI5d3bRVn1Jfa8iRW6egNwcGLkZA3LzBOTmwMjNGJCbJyA3B0ZuxoDcPAG5ObB05qZzLpZ/khZJur2of7mky+O6ftF1Z0l6rKj/lKQp+fYUSU/FHVP+2r+Q9Cof4pHULelhSQsk7ZTU3t/3sMExTM//oF8g6VeSLKlY8tdbJ2l8ydcS/15F9NjIzcpxkZvhGMjN+B4buVk5LnIzHAO5Gd9jIzcrx0VuhmMgN+N7bORm5bjIzXAMqc3NOJeDTZO0oai/Mf+1pE1yzm2RpPz/E+MOwMxmSTpH0pIk48lPh1suabukOyStlbTHOdebPyXO79nXJX1cUi7fH5dgLJLkJP2vmT1kZpflv5b4z05EyM0yyM1+kZvxITfLIDf7RW7Gh9wsg9zsF7kZH3KzDHKzX6nNzfYo7qRK1s/XUr81mZkNl/RTSX/vnNtn1t/TFA/nXFbS2WY2WtLPJZ3W32mNjsPMXidpu3PuITN7+fEvJxFLkZc45zab2URJd5jZkzFeu9GSfm69RG6eiNyMXdLPrZfIzRORm7FL+rn1Erl5InIzdkk/t14iN0+U9tyMcybQRkkzivrTJW2O8frlbDOzKZKU/397XBc2syHqS8jrnXM/Szqe45xzeyTdo761o6PN7PhgYVzfs5dI+lMzWyfpBvVN0ft6QrFIkpxzm/P/b1ffC9Z8efC9igi5WYLcLIvcjBe5WYLcLIvcjBe5WYLcLIvcjBe5WYLcLCvVuRnnINCDkuZYX8XtDklvlXRLjNcv5xZJl+bbl6pvrWTDWd8Q7DWSVjnnrvAgngn5EVmZ2VBJr1Rfkay7Jb05znicc5c756Y752ap7+fkLufcO5KIRZLMbJiZjTjelvQnkh5TQt+rBiA3i5Cb5ZGbsSM3i5Cb5ZGbsSM3i5Cb5ZGbsSM3i5Cb5aU+N6MoLFTtP0kXS1qtvrV//xjntfPX/y9JWyQdU99I8bvVt/bvTklr8v+PjSmWl6pvetlKScvz/y5OMJ6zJD2Sj+cxSf+S//pJkpZKelrSjZI6Y/6evVzSr5KMJX/dFfl/jx//2U3qe9Wgx0huFmIhN6uLi9yM5zGSm4VYyM3q4iI343mM5GYhFnKzurjIzXgeI7lZiIXcrC6u1OWm5e8cAAAAAAAALSzO5WAAAAAAAABICINAAAAAAAAAKcAgEAAAAAAAQAowCAQAAAAAAJACDAIBAAAAAACkAINAAAAAAAAAKcAgEAAAAAAAQAowCAQAAAAAAJAC/x80ZhO/jCB9dwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1440x1440 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axs = plt.subplots(1, 5, figsize=(20,20))\n",
    "\n",
    "for i in range(5):\n",
    "    axs[i].imshow(masks[i][:50])\n",
    "    axs[i].set_title(f\"mask {i}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation in XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_estimators = 1000 if not os.getenv(\"CI\", False) else 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-merror:0.220102\n",
      "Will train until validation_0-merror hasn't improved in 40 rounds.\n",
      "[10]\tvalidation_0-merror:0.195292\n",
      "[20]\tvalidation_0-merror:0.184288\n",
      "[30]\tvalidation_0-merror:0.17753\n",
      "[40]\tvalidation_0-merror:0.170727\n",
      "[50]\tvalidation_0-merror:0.162376\n",
      "[60]\tvalidation_0-merror:0.15656\n",
      "[70]\tvalidation_0-merror:0.151268\n",
      "[80]\tvalidation_0-merror:0.14644\n",
      "[90]\tvalidation_0-merror:0.141348\n",
      "[100]\tvalidation_0-merror:0.136733\n",
      "[110]\tvalidation_0-merror:0.132015\n",
      "[120]\tvalidation_0-merror:0.127006\n",
      "[130]\tvalidation_0-merror:0.123005\n",
      "[140]\tvalidation_0-merror:0.119055\n",
      "[150]\tvalidation_0-merror:0.11464\n",
      "[160]\tvalidation_0-merror:0.110348\n",
      "[170]\tvalidation_0-merror:0.107198\n",
      "[180]\tvalidation_0-merror:0.103397\n",
      "[190]\tvalidation_0-merror:0.099834\n",
      "[200]\tvalidation_0-merror:0.097485\n",
      "[210]\tvalidation_0-merror:0.093922\n",
      "[220]\tvalidation_0-merror:0.091353\n",
      "[230]\tvalidation_0-merror:0.089004\n",
      "[240]\tvalidation_0-merror:0.086467\n",
      "[250]\tvalidation_0-merror:0.084066\n",
      "[260]\tvalidation_0-merror:0.081427\n",
      "[270]\tvalidation_0-merror:0.079264\n",
      "[280]\tvalidation_0-merror:0.077548\n",
      "[290]\tvalidation_0-merror:0.075502\n",
      "[300]\tvalidation_0-merror:0.07352\n",
      "[310]\tvalidation_0-merror:0.071849\n",
      "[320]\tvalidation_0-merror:0.070422\n",
      "[330]\tvalidation_0-merror:0.069054\n",
      "[340]\tvalidation_0-merror:0.067802\n",
      "[350]\tvalidation_0-merror:0.066627\n",
      "[360]\tvalidation_0-merror:0.065401\n",
      "[370]\tvalidation_0-merror:0.064555\n",
      "[380]\tvalidation_0-merror:0.063277\n",
      "[390]\tvalidation_0-merror:0.062258\n",
      "[400]\tvalidation_0-merror:0.061083\n",
      "[410]\tvalidation_0-merror:0.060373\n",
      "[420]\tvalidation_0-merror:0.059302\n",
      "[430]\tvalidation_0-merror:0.058708\n",
      "[440]\tvalidation_0-merror:0.05794\n",
      "[450]\tvalidation_0-merror:0.057055\n",
      "[460]\tvalidation_0-merror:0.056404\n",
      "[470]\tvalidation_0-merror:0.055584\n",
      "[480]\tvalidation_0-merror:0.055184\n",
      "[490]\tvalidation_0-merror:0.054519\n",
      "[500]\tvalidation_0-merror:0.054125\n",
      "[510]\tvalidation_0-merror:0.053415\n",
      "[520]\tvalidation_0-merror:0.052976\n",
      "[530]\tvalidation_0-merror:0.052331\n",
      "[540]\tvalidation_0-merror:0.051821\n",
      "[550]\tvalidation_0-merror:0.051214\n",
      "[560]\tvalidation_0-merror:0.05053\n",
      "[570]\tvalidation_0-merror:0.050078\n",
      "[580]\tvalidation_0-merror:0.049827\n",
      "[590]\tvalidation_0-merror:0.049549\n",
      "[600]\tvalidation_0-merror:0.048904\n",
      "[610]\tvalidation_0-merror:0.04842\n",
      "[620]\tvalidation_0-merror:0.048058\n",
      "[630]\tvalidation_0-merror:0.047548\n",
      "[640]\tvalidation_0-merror:0.047148\n",
      "[650]\tvalidation_0-merror:0.046838\n",
      "[660]\tvalidation_0-merror:0.046509\n",
      "[670]\tvalidation_0-merror:0.046199\n",
      "[680]\tvalidation_0-merror:0.046057\n",
      "[690]\tvalidation_0-merror:0.045954\n",
      "[700]\tvalidation_0-merror:0.045728\n",
      "[710]\tvalidation_0-merror:0.045315\n",
      "[720]\tvalidation_0-merror:0.045018\n",
      "[730]\tvalidation_0-merror:0.044618\n",
      "[740]\tvalidation_0-merror:0.044341\n",
      "[750]\tvalidation_0-merror:0.044263\n",
      "[760]\tvalidation_0-merror:0.043928\n",
      "[770]\tvalidation_0-merror:0.043798\n",
      "[780]\tvalidation_0-merror:0.043579\n",
      "[790]\tvalidation_0-merror:0.04345\n",
      "[800]\tvalidation_0-merror:0.04325\n",
      "[810]\tvalidation_0-merror:0.043063\n",
      "[820]\tvalidation_0-merror:0.043088\n",
      "[830]\tvalidation_0-merror:0.042733\n",
      "[840]\tvalidation_0-merror:0.042488\n",
      "[850]\tvalidation_0-merror:0.042275\n",
      "[860]\tvalidation_0-merror:0.042127\n",
      "[870]\tvalidation_0-merror:0.041991\n",
      "[880]\tvalidation_0-merror:0.041772\n",
      "[890]\tvalidation_0-merror:0.041746\n",
      "[900]\tvalidation_0-merror:0.04163\n",
      "[910]\tvalidation_0-merror:0.041527\n",
      "[920]\tvalidation_0-merror:0.041339\n",
      "[930]\tvalidation_0-merror:0.041165\n",
      "[940]\tvalidation_0-merror:0.041036\n",
      "[950]\tvalidation_0-merror:0.040881\n",
      "[960]\tvalidation_0-merror:0.040797\n",
      "[970]\tvalidation_0-merror:0.040655\n",
      "[980]\tvalidation_0-merror:0.040649\n",
      "[990]\tvalidation_0-merror:0.040539\n",
      "[999]\tvalidation_0-merror:0.040429\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBClassifier(max_depth=8, n_estimators=1000, n_jobs=-1,\n",
       "              objective='multi:softprob', subsample=0.7, verbosity=0)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from xgboost import XGBClassifier\n",
    "\n",
    "clf_xgb = XGBClassifier(max_depth=8,\n",
    "    learning_rate=0.1,\n",
    "    n_estimators=n_estimators,\n",
    "    verbosity=0,\n",
    "    silent=None,\n",
    "    objective=\"multi:softmax\",\n",
    "    booster='gbtree',\n",
    "    n_jobs=-1,\n",
    "    nthread=None,\n",
    "    gamma=0,\n",
    "    min_child_weight=1,\n",
    "    max_delta_step=0,\n",
    "    subsample=0.7,\n",
    "    colsample_bytree=1,\n",
    "    colsample_bylevel=1,\n",
    "    colsample_bynode=1,\n",
    "    reg_alpha=0,\n",
    "    reg_lambda=1,\n",
    "    scale_pos_weight=1,\n",
    "    base_score=0.5,\n",
    "    random_state=0,\n",
    "    seed=None,)\n",
    "\n",
    "clf_xgb.fit(X_train, y_train,\n",
    "            eval_set=[(X_valid, y_valid)],\n",
    "            early_stopping_rounds=40,\n",
    "            verbose=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9596481150403067\n",
      "0.9604399197955302\n"
     ]
    }
   ],
   "source": [
    "preds_valid = np.array(clf_xgb.predict_proba(X_valid, ))\n",
    "valid_acc = accuracy_score(y_pred=np.argmax(preds_valid, axis=1) + 1, y_true=y_valid)\n",
    "print(valid_acc)\n",
    "\n",
    "preds_test = np.array(clf_xgb.predict_proba(X_test))\n",
    "test_acc = accuracy_score(y_pred=np.argmax(preds_test, axis=1) + 1, y_true=y_test)\n",
    "print(test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (1) Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\"\n",
    "dataset_name = 'census-income'\n",
    "out = Path(os.getcwd()+'/data/'+dataset_name+'.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File already exists.\n"
     ]
    }
   ],
   "source": [
    "out.parent.mkdir(parents=True, exist_ok=True)\n",
    "if out.exists():\n",
    "    print(\"File already exists.\")\n",
    "else:\n",
    "    print(\"Downloading file...\")\n",
    "    wget.download(url, out.as_posix())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (2) Train/Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv(out)\n",
    "target = ' <=50K'\n",
    "if \"Set\" not in train.columns:\n",
    "    train[\"Set\"] = np.random.choice([\"train\", \"valid\", \"test\"], p =[.8, .1, .1], size=(train.shape[0],))\n",
    "\n",
    "train_indices = train[train.Set==\"train\"].index\n",
    "valid_indices = train[train.Set==\"valid\"].index\n",
    "test_indices = train[train.Set==\"test\"].index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([    0,     1,     3,     4,     5,     6,     7,     8,     9,\n",
       "               10,\n",
       "            ...\n",
       "            32548, 32549, 32550, 32551, 32553, 32554, 32555, 32556, 32557,\n",
       "            32559],\n",
       "           dtype='int64', length=26094)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (3) Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label encode categorical features and fill empty cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " State-gov 9\n",
      " Bachelors 16\n",
      " Never-married 7\n",
      " Adm-clerical 15\n",
      " Not-in-family 6\n",
      " White 5\n",
      " Male 2\n",
      " United-States 42\n",
      " <=50K 2\n",
      "Set 3\n"
     ]
    }
   ],
   "source": [
    "categorical_columns = []\n",
    "categorical_dims =  {}\n",
    "for col in train.columns[train.dtypes == object]:\n",
    "    print(col, train[col].nunique())\n",
    "    l_enc = LabelEncoder()\n",
    "    train[col] = train[col].fillna(\"VV_likely\")\n",
    "    train[col] = l_enc.fit_transform(train[col].values)\n",
    "    categorical_columns.append(col)\n",
    "    categorical_dims[col] = len(l_enc.classes_)\n",
    "\n",
    "for col in train.columns[train.dtypes == 'float64']:\n",
    "    train.fillna(train.loc[train_indices, col].mean(), inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (4) Define categorical features for categorical embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "unused_feat = ['Set']\n",
    "\n",
    "features = [ col for col in train.columns if col not in unused_feat+[target]] \n",
    "\n",
    "cat_idxs = [ i for i, f in enumerate(features) if f in categorical_columns]\n",
    "\n",
    "cat_dims = [ categorical_dims[f] for i, f in enumerate(features) if f in categorical_columns]\n",
    "\n",
    "# define your embedding sizes : here just a random choice\n",
    "cat_emb_dim = [5, 4, 3, 6, 2, 2, 1, 10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (5) Create a Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device used : cuda\n"
     ]
    }
   ],
   "source": [
    "clf = TabNetRegressor(cat_dims=cat_dims, \n",
    "                      cat_emb_dim=cat_emb_dim, \n",
    "                      cat_idxs=cat_idxs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (6) Train a Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train[features].values[train_indices]\n",
    "y_train = train[target].values[train_indices].reshape(-1, 1)\n",
    "\n",
    "X_valid = train[features].values[valid_indices]\n",
    "y_valid = train[target].values[valid_indices].reshape(-1, 1)\n",
    "\n",
    "X_test = train[features].values[test_indices]\n",
    "y_test = train[target].values[test_indices].reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_epochs = 1000 if not os.getenv(\"CI\", False) else 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Will train until validation stopping metric hasn't improved in 50 rounds.\n",
      "---------------------------------------\n",
      "| EPOCH |  train  |   valid  | total time (s)\n",
      "| 1     | -0.17033 |  -0.27162 |   5.8       \n",
      "| 2     | -0.13373 |  -0.18774 |   11.4      \n",
      "| 3     | -0.12592 |  -0.16804 |   17.5      \n",
      "| 4     | -0.12041 |  -0.13407 |   23.4      \n",
      "| 5     | -0.11810 |  -0.12819 |   29.7      \n",
      "| 6     | -0.11661 |  -0.13072 |   36.0      \n",
      "| 7     | -0.11288 |  -0.12724 |   42.6      \n",
      "| 8     | -0.11267 |  -0.11723 |   48.7      \n",
      "| 9     | -0.11055 |  -0.11717 |   54.6      \n",
      "| 10    | -0.10884 |  -0.11294 |   60.4      \n",
      "| 11    | -0.10612 |  -0.10995 |   66.9      \n",
      "| 12    | -0.10412 |  -0.11029 |   72.7      \n",
      "| 13    | -0.10403 |  -0.10978 |   78.5      \n",
      "| 14    | -0.10333 |  -0.11039 |   84.4      \n",
      "| 15    | -0.10214 |  -0.10864 |   90.2      \n",
      "| 16    | -0.10224 |  -0.10830 |   96.2      \n",
      "| 17    | -0.10264 |  -0.11234 |   102.2     \n",
      "| 18    | -0.10092 |  -0.10992 |   108.0     \n",
      "| 19    | -0.10119 |  -0.10671 |   113.7     \n",
      "| 20    | -0.10048 |  -0.10866 |   119.6     \n",
      "| 21    | -0.10043 |  -0.10844 |   125.3     \n",
      "| 22    | -0.09999 |  -0.10824 |   133.4     \n",
      "| 23    | -0.10069 |  -0.11013 |   139.8     \n",
      "| 24    | -0.09963 |  -0.10872 |   147.0     \n",
      "| 25    | -0.09986 |  -0.10996 |   153.7     \n",
      "| 26    | -0.09862 |  -0.10796 |   159.9     \n",
      "| 27    | -0.09944 |  -0.10788 |   166.3     \n",
      "| 28    | -0.09945 |  -0.11139 |   172.5     \n",
      "| 29    | -0.09961 |  -0.10769 |   178.7     \n",
      "| 30    | -0.09869 |  -0.10631 |   185.4     \n",
      "| 31    | -0.09925 |  -0.10965 |   191.7     \n",
      "| 32    | -0.09855 |  -0.10618 |   198.1     \n",
      "| 33    | -0.09776 |  -0.10694 |   204.5     \n",
      "| 34    | -0.09751 |  -0.11263 |   210.7     \n",
      "| 35    | -0.09854 |  -0.10681 |   217.1     \n",
      "| 36    | -0.09889 |  -0.11326 |   231.2     \n",
      "| 37    | -0.09779 |  -0.10878 |   253.0     \n",
      "| 38    | -0.09959 |  -0.10904 |   272.3     \n",
      "| 39    | -0.10012 |  -0.11136 |   284.4     \n",
      "| 40    | -0.09936 |  -0.11480 |   295.7     \n",
      "| 41    | -0.09880 |  -0.10566 |   307.4     \n",
      "| 42    | -0.09910 |  -0.10637 |   320.0     \n",
      "| 43    | -0.09860 |  -0.11286 |   331.7     \n",
      "| 44    | -0.09752 |  -0.10692 |   347.6     \n",
      "| 45    | -0.09709 |  -0.10574 |   362.0     \n",
      "| 46    | -0.09864 |  -0.10788 |   372.2     \n",
      "| 47    | -0.09982 |  -0.10625 |   384.8     \n",
      "| 48    | -0.10007 |  -0.10679 |   404.7     \n",
      "| 49    | -0.09880 |  -0.10719 |   420.3     \n",
      "| 50    | -0.09849 |  -0.10834 |   435.8     \n",
      "| 51    | -0.09859 |  -0.10491 |   450.7     \n",
      "| 52    | -0.09793 |  -0.10846 |   462.6     \n",
      "| 53    | -0.09666 |  -0.10467 |   474.9     \n",
      "| 54    | -0.09738 |  -0.10861 |   485.4     \n",
      "| 55    | -0.09655 |  -0.10514 |   501.4     \n",
      "| 56    | -0.09598 |  -0.10585 |   518.4     \n",
      "| 57    | -0.09621 |  -0.10478 |   534.8     \n",
      "| 58    | -0.09562 |  -0.10574 |   549.1     \n",
      "| 59    | -0.09630 |  -0.10773 |   560.1     \n",
      "| 60    | -0.09528 |  -0.10568 |   570.8     \n",
      "| 61    | -0.09512 |  -0.10639 |   580.2     \n",
      "| 62    | -0.09528 |  -0.10697 |   591.0     \n",
      "| 63    | -0.09628 |  -0.10640 |   602.1     \n",
      "| 64    | -0.09627 |  -0.11002 |   611.6     \n",
      "| 65    | -0.09465 |  -0.10631 |   621.4     \n",
      "| 66    | -0.09558 |  -0.10475 |   630.8     \n",
      "| 67    | -0.09424 |  -0.10887 |   640.1     \n",
      "| 68    | -0.09478 |  -0.10519 |   650.2     \n",
      "| 69    | -0.09416 |  -0.10839 |   661.3     \n",
      "| 70    | -0.09585 |  -0.10855 |   671.7     \n",
      "| 71    | -0.09437 |  -0.10641 |   681.6     \n",
      "| 72    | -0.09354 |  -0.10729 |   691.4     \n",
      "| 73    | -0.09435 |  -0.10699 |   700.8     \n",
      "| 74    | -0.09361 |  -0.10725 |   712.1     \n",
      "| 75    | -0.09345 |  -0.10687 |   723.4     \n",
      "| 76    | -0.09330 |  -0.10610 |   732.4     \n",
      "| 77    | -0.09315 |  -0.10658 |   741.5     \n",
      "| 78    | -0.09260 |  -0.10697 |   751.0     \n",
      "| 79    | -0.09355 |  -0.10681 |   760.2     \n",
      "| 80    | -0.09351 |  -0.10547 |   768.7     \n",
      "| 81    | -0.09430 |  -0.10766 |   777.3     \n",
      "| 82    | -0.09319 |  -0.10632 |   786.6     \n",
      "| 83    | -0.09442 |  -0.10672 |   795.9     \n",
      "| 84    | -0.09525 |  -0.10904 |   805.0     \n",
      "| 85    | -0.09369 |  -0.10876 |   813.5     \n",
      "| 86    | -0.09443 |  -0.10789 |   822.4     \n",
      "| 87    | -0.09460 |  -0.10650 |   831.1     \n",
      "| 88    | -0.09292 |  -0.11067 |   839.5     \n",
      "| 89    | -0.09527 |  -0.10737 |   848.6     \n",
      "| 90    | -0.09433 |  -0.10631 |   857.1     \n",
      "| 91    | -0.09311 |  -0.10794 |   865.4     \n",
      "| 92    | -0.09274 |  -0.10513 |   873.6     \n",
      "| 93    | -0.09225 |  -0.10776 |   882.4     \n",
      "| 94    | -0.09253 |  -0.10827 |   891.2     \n",
      "| 95    | -0.09350 |  -0.10614 |   899.4     \n",
      "| 96    | -0.09187 |  -0.10743 |   908.1     \n",
      "| 97    | -0.09233 |  -0.10824 |   917.9     \n",
      "| 98    | -0.09296 |  -0.10749 |   930.0     \n",
      "| 99    | -0.09288 |  -0.10805 |   939.8     \n",
      "| 100   | -0.09284 |  -0.10626 |   948.5     \n",
      "| 101   | -0.09153 |  -0.10851 |   957.1     \n",
      "| 102   | -0.09421 |  -0.10710 |   966.1     \n",
      "| 103   | -0.09561 |  -0.11011 |   974.9     \n",
      "Early stopping occured at epoch 103\n",
      "Training done in 974.867 seconds.\n",
      "---------------------------------------\n"
     ]
    }
   ],
   "source": [
    "clf.fit(\n",
    "    X_train=X_train, y_train=y_train,\n",
    "    X_valid=X_valid, y_valid=y_valid,\n",
    "    max_epochs=max_epochs,\n",
    "    patience=50,\n",
    "    batch_size=1024, virtual_batch_size=128,\n",
    "    num_workers=0,\n",
    "    drop_last=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (7) Evaluate and Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BEST VALID SCORE FOR census-income : 0.10466502606868744\n",
      "FINAL TEST SCORE FOR census-income : 0.10429346327336872\n"
     ]
    }
   ],
   "source": [
    "preds = clf.predict(X_test)\n",
    "\n",
    "y_true = y_test\n",
    "\n",
    "test_score = mean_squared_error(y_pred=preds, y_true=y_true)\n",
    "\n",
    "print(f\"BEST VALID SCORE FOR {dataset_name} : {clf.best_cost}\")\n",
    "print(f\"FINAL TEST SCORE FOR {dataset_name} : {test_score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (8) ML Explanation\n",
    "\n",
    "##### Global explainability : feat importance summing to 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([7.60197223e-02, 4.53381126e-03, 3.51288006e-04, 8.90989974e-02,\n",
       "       1.80659371e-03, 1.29987365e-01, 3.12216862e-01, 6.25388060e-02,\n",
       "       1.81649025e-02, 3.80424200e-05, 2.06811124e-01, 1.20327338e-05,\n",
       "       3.20366949e-03, 9.52167841e-02])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.feature_importances_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Local explainability and masks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "explain_matrix, masks = clf.explain(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABGkAAARuCAYAAABgEWoTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3X2QnXd12PFztCvJlm1sGSzXi01tJ1YGQ41oXWHD9C3EWUrLmAkwLUmI09JxIW1DO6Etk5ZO0iYtpCV0Jg2TQE1wWkggNMROC2ypm5QS22ANyEpkg2yMqbHAL0J+lS1Lq1//0NVEyPJqr/Q8es7d+/nMeLR79+7Zs9Jqf7pfP7ubrbUAAAAAYFirhl4AAAAAAJEGAAAAoASRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkYYVKzP/amZ+a+g9AKjJOQHAUpwTDEGkgZHM/NHM/GZmPpmZv5eZZw+9EwA1ZOZ5mXljZu7MzJaZFw69EwB1ZObfyMwvZOYjmfmdzPxQZp4x9F5MHpEGIiIzXxIRvx4Rb4mIcyNiT0R8YNClAKjkQER8NiLeMPQiAJR0ZkT8QkTMRcSLI+L8iPj3g27ERBJpGERm3puZ/zQzt42uXLkuM8/NzM9k5uOZ+b8yc/1h9/+dUZF+NDM/P4oqh1722sy8Y/R692fmO5/jbf706H7nH+XFPxYRv99a+3xr7YmIeHdE/Ij6DTCMaudEa+2B1toHIuK2Xt5hAMZS8Jz4WGvts621Pa213RHxoYh4VR/vOyubSMOQ3hARV0XExoh4XUR8JiJ+NiJeEAc/Nn/6sPt+JiIuiYgNEfHliPjoYS+7LiL+fmvtjIh4aUT87yPfUGa+OyJ+MiL+SmvtaF9X+pKIuP3QM621r0fEM6PdABhGpXMCgHoqnxN/OSK2j/fuQMTs0Asw1X6ltfZARERm/t+IeLC19pXR85+KiFcfumNr7cOHns7Mn4uI3Zl5Zmvt0YjYFxGXZubto2q9+7C3kZn5yxGxOSL+2uj+R3N6RBz5skcjwpU0AMOpdE4AUE/JcyIzr4qIayLiFSf6DjJ9XEnDkB447OmnjvL86RERmTmTme/JzK9n5mMRce/oPi8Y/fqGiHhtRHwzM/9PZl552JyzIuLaiPh3x/iE+kREPO+I254XEY+P8f4A0K1K5wQA9ZQ7JzLzioj4WES8sbW24zjeJ6acSMMk+NGIuDoifigOfkOuC0e3Z0REa+221trVcfDSxd+LiE8c9rq7I+JvRsRvZOZSXxO6PSJeduiZzLw4ItZGhE+sAPWdjHMCgMl1Us6JzHx5RNwYEX+3tXZTl+8A00OkYRKcERF7I2JXRKyLiH976AWZuSYzf2x0qeK+iHgsIhYPf+XW2h/GwW8M/KnMfK5LDj8aEa/LzL+UmadFxL+OiN9trbmSBqC+k3FORGaeEgcDfkTE2tHzANTX+zmRmS+Ngz8F8B+11n6/l/eCqSDSMAl+MyK+GRH3R8QdEXHrES9/S0TcO7p08W0R8eNHDmitfS4i/k5E3JiZf+EoL98+et2PRsSDcfAT+U91+D4A0J/ez4mRp+Lgl8dGRHx19DwA9Z2Mc+JnIuKciLguM58Y/ecbBzO2bK0NvQMAAADA1HMlDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAAbMn842tybXtlDjtZL5JYAptvGxP5zPvvW9fPPzdxex8MN/jBWfPtAsvWN353B3b1nU+E+BwT8eT8Uzb65zomccTEbm6+3MyIqLt29fL3GfO6+fPa813uv/3XvjJxxERsfeF/fyZrb3/yV7mTorlnhMnNdKcEqfFK/LVJ/NNAlNoYWFr5zM3z9/X+Uye7cILVseXFi7ofO783KbOZwIc7ovtpqFXmAoeT0TMnjvXy9z99+/sZe59f++Vvcx90Xu3dD6z7Xum85mT6J53XNnL3Iv/+S29zJ0Uyz0nfLkTAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABJxRpMvM1mfm1zLw7M9/V1VIArAzOCQCW4pwA+F7HHWkycyYifjUi/npEXBoRb87MS7taDIDJ5pwAYCnOCYBnO5EraTZHxN2ttXtaa89ExG9HxNXdrAXACuCcAGApzgmAI5xIpHlhRNx32PPfGt32PTLz2szckplb9sXeE3hzAEyYsc+Jh3YtnrTlABicxxMARziRSJNHua0964bWPthau7y1dvnqWHsCbw6ACTP2OXHO82dOwloAFOHxBMARTiTSfCsiLjjs+fMjYueJrQPACuKcAGApzgmAI5xIpLktIi7JzIsyc01E/O2IuLGbtQBYAZwTACzFOQFwhNnjfcXW2v7M/IcRsRARMxHx4dba9s42A2CiOScAWIpzAuDZjjvSRES01j4dEZ/uaBcAVhjnBABLcU4AfK8T+XInAAAAADoi0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUcEI/3amK3ddc2fnM9dff0vlM/tRTV2/ufOapN3yp85lMph+47u2dz7zv4fd3PhMAYFwHzjmrn8H37+xl7PodB3qZu/iKSzufueoLWzufOYnW7s6hV5hqrqQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChgdugFurD++luGXoExnfaNxzqfeaDzifRt5twNvcx90+u+0PnM3/jYE53PBAAY14E1M0OvMJZTdu3rZe7TG9Z2PnNd5xMn08zeoTeYbq6kAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKGB26AWYTge2fbXzmQs7t3Y+MyJifm5TL3OJWHzgwV7m3rZppvOZT7bsfCYAwLiy9TO3p7HRZvr5N1Qu9rUx/tk7LFfSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFDA79ALQlfm5Tb3MXdi5tZe5fe0LAMDKNbNzVy9z9/cyNeI7m9f0MvdF793S+czW+cTJ9NS5fieG5EoaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAmaHXgCqm5/bNPQKjGlh59bOZ26e39P5TACAce2/f+fQK4xl79kHepnb9j3Ty1wiDqxpQ68w1VxJAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUMDs0AtMmyff+Ipe5p72yS/2Mrcvd/3mn+985iU/8eXOZzKZXnP1WzqfuePuX+98JgDAuHL1ml7mtn3P9DL3vJtbL3Mf/fErOp955n+9tfOZk2jtLtdyDMnvPgAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAAbNDLzBtTvvkF4deoYRLfuLLQ6/ACvbZG/5L5zM3z+/qfCbPtmPbupif2zT0GqxQCzu39jLXxyxwMq36/j/by9zFO+/qZe7T6/u5LmDDH3X/b7PFzidOpuZSjkH57QcAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChgdugFmE5PfPbizmee/pp7Op/JZLr8X72985lf3fn+zmfybBsv2xMLC1s7nzs/t6nzmUweHwfASrB45qlDrzCWM+95ppe5j714feczT7uj85ETac1jQ28w3VxJAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUMDs0AswnU7/udOHXoEV7Jp/8unOZ77v5kc7nwkAMK5VT+3rZe6BXqZG7Nmwupe5Z9z7VC9zidh/6tAbTDdX0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQwO/QCTKlbtw29wbLt+LXNvczd+LYv9TKXiF+54bWdz3zwka91PhMAYFyrHnmil7kHepka8fiL+rkuYP1n7u185mLnEyfTvue1oVeYaq6kAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAACjgmJEmMz+cmQ9m5p8cdtvZmfm5zLxr9Ov6ftcEoCrnBABLcU4ALN9yrqT5SES85ojb3hURN7XWLomIm0bPAzCdPhLOCQCe20fCOQGwLMeMNK21z0fEd4+4+eqIuH709PUR8fqO9wJgQjgnAFiKcwJg+Y73e9Kc21r7dkTE6NcNz3XHzLw2M7dk5pZ9sfc43xwAE+a4zomHdi2etAUBGJTHEwBH0fs3Dm6tfbC1dnlr7fLVsbbvNwfAhDn8nDjn+TNDrwNAMR5PANPkeCPNA5l5XkTE6NcHu1sJgBXAOQHAUpwTAEdxvJHmxoi4ZvT0NRFxQzfrALBCOCcAWIpzAuAolvMjuH8rIm6JiB/IzG9l5lsj4j0RcVVm3hURV42eB2AKOScAWIpzAmD5Zo91h9bam5/jRa/ueBcAJpBzAoClOCcAlq/3bxwMAAAAwLGJNAAAAAAFiDQAAAAABYg0AAAAAAUc8xsHQx8e/KlXdj5zwwdu7nxmRMTGt32pl7n05/SXfrfzmatO3d/5TACAcS1uOKufwd+8r5exL9i2r5e5e161sfOZa//HbZ3PnESnPJxDrzDVXEkDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQwOzQCzCdNnzg5qFXYAX78uUf73zm5nW7O5/Js+3Yti7m5zYNvQYr1MLOrb3M9TELnEzttj/uZW5/nyN7GUuPznufx2pDciUNAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAGzQy8AAIdsvGxPLCxs7Xzu/NymzmcyeXwcACvBzKUbe5k7P9fL2Nj11it7mbvh1u92PnNx+9c6nzmJ7nv3K3uZe8G/ubmXuSuNK2kAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAqYHXoBAAAAlqetmqz/z7728dbL3L1/5vTOZ85u73zkRJrZO/QG022y/oYDAAAArFAiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAAbNDL9CFb/zWyzqfedGbb+98JpNp5z97ZS9z537p5l7mEjE/t6nzmTvars5nAgCMK/ftH3qFsTy9PnuZe+bWRzqfudj5xMm0/9ShN5hurqQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoYHboBbpw0ZtvH3oFVrC5X7p56BUY08LOrZ3P3Dy/p/OZPNuObetifm7T0GuwQvXxuSEifMwCJ9Xi1+7uZW5/nyN7GRuL/YwlIl708x7/DMmVNAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAXMDr3AtLnvX7yyl7kX/OLNvcyNVTP9zD2w2M9ciIiL/+dbO5/57cf+U+czAQCqmJ/b1Mvc77yjn8c/5/3RY53PbFv+pPOZk+ie91zZy9yL33VLL3NXGlfSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFDA79ALT5oJfvHnoFcZzYHHoDWBs9/zwdZ3P3Py+hzufCQAwrplLN/Yyd/GOHb3MXb2n9TJ31ZN7O5/pkc9Bq5/MoVeYaq6kAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAACjgmJEmMy/IzD/IzDszc3tmvmN0+9mZ+bnMvGv06/r+1wWgGucEAEtxTgAs33KupNkfET/TWntxRFwREf8gMy+NiHdFxE2ttUsi4qbR8wBMH+cEAEtxTgAs0zEjTWvt2621L4+efjwi7oyIF0bE1RFx/ehu10fE6/taEoC6nBMALMU5AbB8Y31Pmsy8MCJeHhFfjIhzW2vfjjj4iTciNjzH61ybmVsyc8u+2Hti2wJQmnMCgKU4JwCWtuxIk5mnR8R/i4h/3Fp7bLmv11r7YGvt8tba5atj7fHsCMAEcE4AsBTnBMCxLSvSZObqOPgJ9aOttd8d3fxAZp43evl5EfFgPysCUJ1zAoClOCcAlmc5P90pI+K6iLiztfbLh73oxoi4ZvT0NRFxQ/frAVCdcwKApTgnAJZvdhn3eVVEvCUi/jgzt45u+9mIeE9EfCIz3xoR/y8i3tTPigAU55wAYCnOCYBlOmakaa19ISLyOV786m7XAWDSOCcAWIpzAmD5xvrpTgAAAAD0Q6QBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKGA5P4IbJsLCzq3HvtNxmJ/b1Mtc+tPHn9mOtqvzmTzbxsv2xMJC93+X/T0mwscBwCAO9DR3lesNWJl8ZAMAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQwO/QCXci1azuf2fbu7Xwmf2r3T17Z+cz5uc5HRkTEo5/+/l7mnvnau3uZCwDAynXgtO4f+/Tp9J37e5m758LndT5z7fbOR06k2T1DbzDdXEkDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFDA7NALdKHt3Tv0Coxp/UduGXqFZTvztXcPvQJjWti5tfOZm+f3dD6TZ9uxbV3Mz20aeg1WqD4+N0SEj1ngpFr1ZD+PfRZ7mRrxxPn9POTc8IWHO5/Z1+/BpNm/bugNppsraQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAqYHXoBADhk42V7YmFha+dz5+c2dT6TyePjAFgJFu/YMfQKY9n9ktbL3Od/6K5e5hLxzFkHhl5hqrmSBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoIDZoRcAgEN2bFsX83Obhl6DFWph59Ze5vqYBU6mmRdf0svcxTvv6mXu+u3Zy9yZSzd2PnPxjh2dz5xEqx91LceQ/O4DAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUMDv0AgBwyMbL9sTCwtbO587Pbep8JpPHxwGwErSZmaFXGMvsU62XuQdOW9vLXCJW7R96g+nmShoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJmh14AAACAZZrJoTcYSy72M/fAbPfXG0zW72yP2tALTDdX0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQwO/QCTKev/4crOp/5fe+8tfOZTKaLf+dtnc/89u7/2PlMAIBxtTWT9RDulO/u72Xu0+eu7XzmqZ1PnEyzTw+9wXRzJQ0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg34ldnLAAAbMElEQVQAAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABs0MvwHT6vnfeOvQKrGD3vOnXOp+5+T8/1PlMAIBx5TP7e5nbepka8fTZ/TzkPOPep3qZS8T+U4beYLq5kgYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKCA2aEXYDo9tXBR5zNPnf9G5zOZTH/xX76985lf3fn+zmcCAIwtc+gNxjKzr/Uy98Damc5ndj9xMuWBoTeYbq6kAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKOCYkSYzT8nML2Xm7Zm5PTN/fnT7RZn5xcy8KzM/nplr+l8XgGqcEwAsxTkBsHzLuZJmb0T8YGvtZRGxKSJek5lXRMR7I+L9rbVLImJ3RLy1vzUBKMw5AcBSnBMAy3TMSNMOemL07OrRfy0ifjAiPjm6/fqIeH0vGwJQmnMCgKU4JwCWb1nfkyYzZzJza0Q8GBGfi4ivR8QjrbX9o7t8KyJe2M+KAFTnnABgKc4JgOVZVqRprS221jZFxPkRsTkiXny0ux3tdTPz2szckplb9sXe498UgLK6Oice2rXY55oADMTjCYDlGeunO7XWHomIP4yIKyLirMycHb3o/IjY+Ryv88HW2uWttctXx9oT2RWA4k70nDjn+TMnZ1EABuHxBMDSlvPTnc7JzLNGT58aET8UEXdGxB9ExBtHd7smIm7oa0kA6nJOALAU5wTA8s0e+y5xXkRcn5kzcTDqfKK19t8z846I+O3M/IWI+EpEXNfjngDU5ZwAYCnOCYBlOmakaa1ti4iXH+X2e+Lg15MCMMWcEwAsxTkBsHxjfU8aAAAAAPoh0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFHDMH8ENffj8n/tU5zPnY1PnM5lMZ3/4ls5nzrQnO58JADCuljn0CmOZ2dt6mbv/1JnOZ3Y/cTKt2j/0BtPNlTQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAXMDr3AtHn8b13Ry9wzPn5rL3P78iN3X9XD1Id6mAkAAHW0r2wfeoWx7FvXz3UBbab7mWu7HzmR9p0+9AbTzZU0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDT8//buPdbvur7j+OtDD7fiBdxFPRQFA3hBWXEMFDdivKwOFJVsicYRkpltf8xNpwlotv+WLMo25xYXJ1OH2YxzYyw6s60aEZeNiwo0ICCXIUotE3CCSifS9rM/etwql9Nz5vfb7/vb83gkhJ6T8uad8ut59zz5nhYAAAAoYGHqBdaax3/syqlXKOGB0++ZegUKWDjm6aPMPf8zHx985q+ftX3wmQAAq/aCE8eZe+V1o4w99N6HRpn74OE+lR3LQfdPvcHa5kkaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAhamXgBYu3Z85aujzD39kOFnPr4NPxMAYNWuvG7qDVbljlcdOMrcY99y5ShzSR44atfUK6xpnqQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoYGHqBVibtp132uAzFy+4fPCZzNPxnzt38Jlbv/u+wWcCAKzaKc8bZ+7nrx9l7NH/+NAoc7effergM9dfctXgM+fosK2e5ZiSH30AAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAIWpl5gCOtOeObgM3fecPPgM/k/ixdcPvUKK3b7u144ytxnnH/FKHNJDr72sMFntu2aNgBQwOevn3qDVbnjzANHmXvsW68cZS7Jd5+2a+oV1jSfdQAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABSwMPUCQ9h5w81Tr8AqbT/71MFnrr/kqsFnJsmpP3fTKHPvGWUqSbJ4weWDz7yzPzD4TACA1Tpg43NGmbtry42jzH3KlaOMza6f3Tj4zAP+bcvgM+fo0G94lmNKfvQBAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKWJh6Adam9ZdcNfUKK3bPafdNvQKrtHnblsFnnrJp++AzeaRbrlufTYsbp16D/dQYHxuSeM0C+9SuLTeOMne8j5GjjGVER77z8qlXWNM8SQMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFDAwtQLQHV3vfW0UeY+9d2XjzKX5Nn/fs7gM7/63fcPPpNHOv7E7dm8ecvgczctbhx8JvPjdQDsD9pJJ4wyd9PiKGOz/bWnjjL3oPt3DD5z4dKrB585R9vOG+fzn8ULfP6zEp6kAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoYGHqBVibHjzjZwafefA/fWHwmUly+uuvHmXure8eZSxJnvZL1w8+867+34PP5JFuuW59Ni1uHHzu5m1bBp85xp4AsDcHPPC9UebuHGVqsuvANsrcA3buGmUuyc6Dp95gbfMkDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABK440rbV1rbVrW2ufXHr7mNbaVa21W1trH2utHTTemgBU504AsBx3AmDvVvMkzZuT3LTH2+9K8se99+OSfCvJG4dcDIDZcScAWI47AbAXK4o0rbUNSc5M8oGlt1uSlyS5eOm7fDjJa8ZYEID63AkAluNOAKzMSp+keU+S85LsWnr7x5Lc13vfsfT21iRHPto/2Fr7tdbaF1trX3woD/5IywJQljsBwHLcCYAV2Gukaa29Msndvfer93z3o3zX/mj/fO/9wt77yb33kw/Mwf/PNQGoyp0AYDnuBMDKLazg+7woyVmttTOSHJLkCdldwg9vrS0s1e8NSbaNtyYAhbkTACzHnQBYob0+SdN7f0fvfUPv/egkr0tyae/9DUk+m+QXl77buUk+PtqWAJTlTgCwHHcCYOVW86c7Pdz5Sd7aWrstu7+m9IPDrATAfsKdAGA57gTAw6zky53+V+/9siSXLX379iSnDL8SAHPlTgCwHHcCYHk/ypM0AAAAAAxEpAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoYFV/uhMM5TtHDf/SO3jwibt9/u6njTL3iNw6ylyYs+NP3J7Nm7cMPnfT4sbBZwLAFPpBB069wqrsPGicueu27xh8Zh984jy1nVNvsLZ5kgYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKCAhakXWGu+fskJo8w98uwbRpm7cNSGUeY++WM3Dj5z5+ATdzvizFtHmsxYvveqUwaf2T93xeAzAQBWa9eXvjz1CqtyyDfH+VX6N5/3uMFnPukLg4+cpR3r+9QrrGmepAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChgYeoF1pojz75h6hVWZcedW6deAVbtc++/cPCZp2y6d/CZPNIt163PpsWNU6/Bfmrzti2jzPWaBfYH432MHGVsDh5nLEmO/t0rpl5hTfMkDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAASINAAAAQAEiDQAAAEABIg0AAABAAQtTLwAAP3D8iduzefOWweduWtw4+Ezmx+sA2B+0k04YZe6mxVHGZvtrTx1l7kH37xh85sKlVw8+c462nXfaKHMXL7h8lLn7G0/SAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFCDSAAAAABQg0gAAAAAUINIAAAAAFLAw9QIAAACsTOt9lLnjTE36unHmtl1jbUxvU2+wtnmSBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoACRBgAAAKAAkQYAAACgAJEGAAAAoICFqRdYa3a++PmjzF132TWjzB3LbX910uAzjz3n2sFnMk8vfuOvDj7z5jv+dPCZAACr1W7fOvUKq7LzoDbK3O8fPvynsocOPnGedh049QZrmydpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAAChBpAAAAAAoQaQAAAAAKEGkAAAAACliYeoG1Zt1l10y9QgnHnnPt1CuwH7vsg38x+MxTNt07+Ewe6Zbr1mfT4sap12A/tXnbllHmes0C+9SGp4wz98ZvjzL2ocPaKHOPuPZbg8/cOfjEmRrnPxkr5EkaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACRBoAAACAAkQaAAAAgAJEGgAAAIACFqZeAAAAgJXZeeMto8zdvG3LKHM3LY4yNjvHGUuSo37v8qlXWNM8SQMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFCASAMAAABQgEgDAAAAUIBIAwAAAFDAwtQLDGHdsccMPnPnbV8ZfGaSrDv8iaPM3Xnf/aPMHcut7z118JnHvemqwWcyrode9tOjzD3jZccPPvPW//jQ4DN5pONP3J7Nm7cMPnfT4sbBZzI/XgfA/uCAE581ytxNi6OMzUM/f/Iocx984rrBZz7u73w+kSR3ve20UeY+9Y8uH2Xu/saTNAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAWINAAAAAAFiDQAAAAABYg0AAAAAAW03vu++5e1dk+Sr67wu/94kntHXGdIc9o1mde+c9o1mde+c9o1mX7fp/fef2LCf/+a4E6UMad957RrMq9957RrMv2+7sQ+sIo7MfXrYbXmtO+cdk3mte+cdk3mtW+FXVd0J/ZppFmN1toXe+8nT73HSsxp12Re+85p12Re+85p12R++zK+Ob0m5rRrMq9957RrMq9957RrMr99GdfcXg9z2ndOuybz2ndOuybz2ndOu/pyJwAAAIACRBoAAACAAipHmgunXmAV5rRrMq9957RrMq9957RrMr99Gd+cXhNz2jWZ175z2jWZ175z2jWZ376Ma26vhzntO6ddk3ntO6ddk3ntO5tdy/6eNAAAAABrSeUnaQAAAADWjHKRprX2itbaza2121prb596n+W01o5qrX22tXZTa+2G1tqbp95pb1pr61pr17bWPjn1LnvTWju8tXZxa+3LSz/GL5x6p8fSWvvtpdfAl1prH22tHTL1TntqrX2otXZ3a+1Le7zvSa21T7fWbl36+xFT7vgDj7HrHyy9Dq5rrf1Da+3wKXdkWu7EuNyJcbgTw3En2Bt3YlxzuRNzuhGJOzGkud+JUpGmtbYuyZ8l+YUkz0ny+tbac6bdalk7kryt9/7sJC9I8hvF902SNye5aeolVuhPkvxL7/1ZSX4qRfdurR2Z5LeSnNx7f26SdUleN+1Wj3BRklc87H1vT/KZ3vtxST6z9HYFF+WRu346yXN77ycmuSXJO/b1UtTgTuwT7sTA3InBXRR3gsfgTuwTc7kTs7gRiTsxgosy4ztRKtIkOSXJbb3323vv30/yN0lePfFOj6n3flfv/Zqlb38nu3/iHzntVo+ttbYhyZlJPjD1LnvTWntCktOTfDBJeu/f773fN+1Wy1pIcmhrbSHJ+iTbJt7nh/Te/zXJfz3s3a9O8uGlb384yWv26VKP4dF27b1/qve+Y+nNK5Ns2OeLUYU7MSJ3YlTuxEDcCfbCnRjRXO7EDG9E4k4MZu53olqkOTLJnXu8vTWFP0jtqbV2dJKTklw17SbLek+S85LsmnqRFXhGknuS/OXS45QfaK0dNvVSj6b3/vUkf5jka0nuSnJ/7/1T0261Ik/uvd+V7P4FQpKfnHiflfqVJP889RJMxp0YlzsxAndin3Mn1jZ3YlxzuROzuRGJOzGB0neiWqRpj/K+8n/8VGvtcUn+Pslbeu/fnnqfR9Nae2WSu3vvV0+9ywotJHl+kvf13k9K8kDqPD73Q5a+9vLVSY5JspjksNbaL0+71f6ptfY72f1Y8Eem3oXJuBMjcSfG407sO+4EcSdGM7M7MZsbkbgT+9Ic7kS1SLM1yVF7vL0hxR7zerjW2oHZ/QH1I733S6beZxkvSnJWa+2O7H7s8yWttb+edqVlbU2ytff+g/+TcHF2f6Ct6GVJvtJ7v6f3/lCSS5KcNvFOK/GN1tpTk2Tp73dPvM+yWmvnJnllkjf03sv/YovRuBPjcSfG407sA+4ES9yJ8czpTszpRiTuxD4xlztRLdJ8IclxrbVjWmsHZfdvlvSJiXd6TK21lt1f53hT7/3dU++znN77O3rvG3rvR2f3j+ulvfeydbb3/p9J7mytPXPpXS9NcuOEKy3na0le0Fpbv/SaeGkK/8Zke/hEknOXvn1uko9PuMuyWmuvSHJ+krN679un3odJuRMjcSdG5U6MzJ1gD+7ESOZ0J2Z2IxJ3YnRzuhOlIs3Sb+TzpiSbs/tF+be99xum3WpZL0pyTnZX5C1Lf50x9VL7kd9M8pHW2nVJNib5/Yn3eVRLhf7iJNckuT67f15dOOlSD9Na+2iSK5I8s7W2tbX2xiTvTPLy1tqtSV6+9PbkHmPX9yZ5fJJPL/08+/NJl2Qy7gQP404MxJ1gf+FOsIdZ3IjEnRja3O9EK/yUDwAAAMCaUepJGgAAAIC1SqQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKECkAQAAAChApAEAAAAoQKQBAAAAKOB/AB4oYlG3N+nxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1440x1440 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axs = plt.subplots(1, 3, figsize=(20,20))\n",
    "\n",
    "for i in range(3):\n",
    "    axs[i].imshow(masks[i][:50])\n",
    "    axs[i].set_title(f\"mask {i}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation in XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-rmse:0.470985\n",
      "Will train until validation_0-rmse hasn't improved in 40 rounds.\n",
      "[10]\tvalidation_0-rmse:0.339601\n",
      "[20]\tvalidation_0-rmse:0.314615\n",
      "[30]\tvalidation_0-rmse:0.309246\n",
      "[40]\tvalidation_0-rmse:0.308169\n",
      "[50]\tvalidation_0-rmse:0.307491\n",
      "[60]\tvalidation_0-rmse:0.307688\n",
      "[70]\tvalidation_0-rmse:0.307697\n",
      "[80]\tvalidation_0-rmse:0.307683\n",
      "[90]\tvalidation_0-rmse:0.308334\n",
      "Stopping. Best iteration:\n",
      "[53]\tvalidation_0-rmse:0.307343\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBRegressor(max_depth=8, n_estimators=1000, n_jobs=-1, subsample=0.7,\n",
       "             verbosity=0)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from xgboost import XGBRegressor\n",
    "\n",
    "clf_xgb = XGBRegressor(max_depth=8,\n",
    "    learning_rate=0.1,\n",
    "    n_estimators=1000,\n",
    "    verbosity=0,\n",
    "    silent=None,\n",
    "    objective='reg:linear',\n",
    "    booster='gbtree',\n",
    "    n_jobs=-1,\n",
    "    nthread=None,\n",
    "    gamma=0,\n",
    "    min_child_weight=1,\n",
    "    max_delta_step=0,\n",
    "    subsample=0.7,\n",
    "    colsample_bytree=1,\n",
    "    colsample_bylevel=1,\n",
    "    colsample_bynode=1,\n",
    "    reg_alpha=0,\n",
    "    reg_lambda=1,\n",
    "    scale_pos_weight=1,\n",
    "    base_score=0.5,\n",
    "    random_state=0,\n",
    "    seed=None,)\n",
    "\n",
    "clf_xgb.fit(X_train, y_train,\n",
    "        eval_set=[(X_valid, y_valid)],\n",
    "        early_stopping_rounds=40,\n",
    "        verbose=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.09445981424093695\n",
      "0.09347489128752377\n"
     ]
    }
   ],
   "source": [
    "preds = np.array(clf_xgb.predict(X_valid))\n",
    "valid_auc = mean_squared_error(y_pred=preds, y_true=y_valid)\n",
    "print(valid_auc)\n",
    "\n",
    "preds = np.array(clf_xgb.predict(X_test))\n",
    "test_auc = mean_squared_error(y_pred=preds, y_true=y_test)\n",
    "print(test_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> FastAi wrapper https://pypi.org/project/fast-tabnet/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Paper https://arxiv.org/pdf/1908.07442.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Slide https://www.slideshare.net/SebastienFischman/tab-netpresentation/SebastienFischman/tab-netpresentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}