tutorials/segment_model_building/h2o-segment-model-building.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# H2O Segment Model Building"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "H2O-3 Release 3.30 (Zahradnik) introduced Segment Model Building to scale building multiple models on dataset segments (partitions). Both the Python and R API were extended with a new function `train_segments()`: with any H2O algorithm specify `segment_columns` (columns that will be used to make groups of observations, e.g. a Product Category, Region, etc.) and train many models in parallel on all available nodes of your H2O cluster. You can then access each individual model and work with them in the way you are already familiar with. For more details sse:\n",
    "\n",
    "  - [H2O Release 3.30 (Zahradnik)](https://www.h2o.ai/blog/h2o-release-3-30-zahradnik/))\n",
    "  - [Python Reference](http://docs.h2o.ai/h2o/latest-stable/h2o-py/docs/modeling.html#h2o.estimators.estimator_base.H2OEstimator.train_segments)\n",
    "  - [R Reference](http://docs.h2o.ai/h2o/latest-stable/h2o-r/docs/reference/h2o.train_segments.html)\n",
    "  - [H2O-3 source code](https://github.com/h2oai/h2o-3/blob/455c3bec27a3cdbf9c6e77b4c22da5305f2649ac/h2o-py/h2o/estimators/estimator_base.py#L117)\n",
    "  \n",
    "This example illustrates complete workflow to build multiple models on dataset segments using H2O-3 Python API\n",
    "\n",
    "### Initialize H2O and load packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321 . connected.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>32 secs</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>America/Chicago</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.30.0.3</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>7 days, 3 hours and 43 minutes </td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>H2O_from_python_gkanevsky_8sv77s</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>7.963 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>16</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>16</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>H2O_API_Extensions:</td>\n",
       "<td>Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.7.4 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ------------------------------------------------------------------\n",
       "H2O_cluster_uptime:         32 secs\n",
       "H2O_cluster_timezone:       America/Chicago\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.30.0.3\n",
       "H2O_cluster_version_age:    7 days, 3 hours and 43 minutes\n",
       "H2O_cluster_name:           H2O_from_python_gkanevsky_8sv77s\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    7.963 Gb\n",
       "H2O_cluster_total_cores:    16\n",
       "H2O_cluster_allowed_cores:  16\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://localhost:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "H2O_API_Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4\n",
       "Python_version:             3.7.4 final\n",
       "--------------------------  ------------------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import h2o\n",
    "\n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from h2o.estimators.gbm import H2OGradientBoostingEstimator\n",
    "import pandas as pd\n",
    "import datatable as dt\n",
    "from plotnine import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import data with white and red wine quality\n",
    "\n",
    "First, import wine quality data from your local repository or from H2O public S3 bucket to the H2O cluster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "import_from_s3 = True\n",
    "if import_from_s3:\n",
    "    winequality_df = h2o.import_file(\"https://h2o-public-test-data.s3.amazonaws.com/smalldata/wine/winequality-redwhite.csv\")\n",
    "else:\n",
    "    winequality_df = h2o.upload_file(\"../data/winequality-redwhite.csv\")\n",
    "\n",
    "predictors = winequality_df.names\n",
    "predictors.remove('type')\n",
    "predictors.remove('quality')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split data into train, validation, and test partitions\n",
    "\n",
    "Before modeling split dataset into train, validation, and test partitions. H2O-3 has function `split_frame()` with the simplest possible random split by handling each row independently with a \"coin flip\". The thinking is, since H2O-3 is intended to handle big data, there are enough samples to not worry about it (reference: [stackoverflow.com](https://stackoverflow.com/a/53510090/59470))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df,valid_df,test_df = winequality_df.split_frame(ratios=[.7, .15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stratified split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we intend to use segmented models let's alternatively create stratified partitions to assure that between segments (variable `type`) and dependent variable (`quality`) the train and validation sets are well represented. \n",
    "First, create empty H2O frames to hold train, valid, and test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = winequality_df[\"quality\"] < 0\n",
    "train_df = winequality_df[mask,:]\n",
    "valid_df = winequality_df[mask,:]\n",
    "test_df = winequality_df[mask,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, obtain all unique values from column `quality` that defines our segments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   quality  nrow\n",
      "0        3    30\n",
      "1        4   216\n",
      "2        5  2138\n",
      "3        6  2836\n",
      "4        7  1079\n",
      "5        8   193\n",
      "6        9     5\n"
     ]
    }
   ],
   "source": [
    "quality_values = winequality_df.group_by(\"quality\").count().get_frame().as_data_frame()\n",
    "print(quality_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, iterate over quality segments to split each and bind results to train, valid, and test frames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "for _, value in quality_values[\"quality\"].iteritems(): \n",
    "    segment = winequality_df[winequality_df[\"quality\"] == value]\n",
    "    train_segment,valid_segment,test_segment = segment.split_frame(ratios=[.5, .25])\n",
    "    train_df = train_df.rbind(train_segment)\n",
    "    valid_df = valid_df.rbind(valid_segment)\n",
    "    test_df = test_df.rbind(test_segment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review the sizes of partitions visually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAHACAYAAACyIiyEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhlVZnv8e9bVFlKkYJSbCXSitpiowgO7dStiK20E66LInQ3Dl2ANo73OoHgACqiqLf1KoqzlAq2aLXockDFq4iK3XiFoilBUQERgwoCVWEqAqz7x94Fh5CsSk5OcpKT7+d5znPO2VPefVZ28svaa+9EKQVJkiRNbEm/C5AkSZrPDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5I0xyLikog4vd91SJoaw5Kk20TEnhFROh63RMSGiPhlRHwxIg6IiLv04Ou8OiJW96DknouIfSLirV2s95CI+GxEXBgRN0TEVRHx84hYExFPnoVSJc2R8H/DSdosIvYEvg+sBb7aTt4GuD/wDOBhwAXAvqWUC2bwdS4BLiml7DmDcmdFRKwB/qWUEtNY5+nA14BR4LM0n9HdgJ2BZwNfLaW8smP55UAppdzUw9IlzZKl/S5A0rx0binlxHHT3tD2Bn0S+FZE7FpKGZ370uald9P8PN2zlPLfnTMi4pXAvTunlVI2zWFtkmbI03CSpqyUsgb4N+C+wCs2T4+IJRHxxog4PSIuj4ibIuL3EfGZiLhvx3I7RUQB7gc8adwpv53aZR4TEZ9uT/1d1z5+GhEHjq8nIlZFxHsj4lftqa8NEXF+RLxvgmWfHBGnRsTVEbEpIi6IiDdExFYdy1wC/Ev7urO21Vv4aB4M/Hl8UGo/s1tLKSPjarnDmKWIeOu4rzf+sbpj2YiIl0TEWR2fz5kRsc8E+/z0iPheRPyp3efLI+JbEfF3W9gfSR3sWZI0XR8DDqM5vXRsO+0u7bS13H46ajfgIOApEbFbKeUq4ArghcD7gSuBYzq2e0X7/BzgIcDJwO+A7YD9gU9HxD1LKe/pWOeLwJ7AJ4BzgOXAXwF/31lwRBxE0yN2TlvzNcDfAe8CHgH8U7voq4HXAk9s69zszC18JhcBu0TEc0spX97CshP5MvDrCaa/heZU3h87pp0AvIjmNOlJ7bTnAqdExMtKKR8FiIg9gK8D5wPvBf5M08P1BGB34Mdd1CktTqUUHz58+KCUAk3wKMCbt7DcRuDKjvcB3G2C5Z7abu/QcdMvAU6fZNsrJpi2BDiDJuQsa6dtC9wKHL+FWu8N3ACcQjtOs2Pe69r6ntQxbU3zo3Fan9t+bS0FuBD4NPAyYJdJlp90/zuWObLd3rs7pu3TTnvNBMtnYAMw1L5/X7vsX/T7+8qHj4X+8DScpG5spAkrQJMsSik3wG2n5LaLiO2BdTS/wB871Q2XUq7b/Doi7hYR9wDuDnyr/ZoPbmdfD2wCHhcRD6hs8nnAXWl6lu4REdtvftD0vAA8bar1TVLzl2h6o74EbA8cCBwPnB8RZ2yhvjtpT7u9DfgCcHjHrBfQBL+TO/ej3ZevACuBx7fLXt0+7x8Ry7rbM0ngaThJ3VlJE4JuExHPAQ4FHkVzWq7T3ae64fYX/9tpelF2mGCRuwOUUsbawdMfAn4TERcCPwS+SXP12S3t8ru0z1+/05Zud6+p1jeZUsqPgR9HRAA70YSnf22fvxoRjypTuPotIp5Gc1rxB8DqUkrnJcu70Fxl9/vKJjbvy4eAvYHjgGMj4ifA6cDnSykXT2PXpEXPsCRpWiLigcAQHWNe2sHFXwb+H82Yn0tpekCg6R2ZUi92GzS+A+xK80v+pzQ9JLcAzwRe07mtUsqnIuJr7bw9gL2Ag4GzIuJJpZQbO5Z/MfDbSb70yCTTp60NNxcDF0fEiTQB7m+BxwA/qq0bEY+gGfd1IbBPufNVc0toQurzKpv5eVvH1RHx+PZr70UzVuko4KiIeGEp5eTp7pu0WBmWJE3Xv7bPnT01LwJupBn7c/3miRGxAlg1wTYmu8Hbw2gGXB9dSjmyc0ZE7DXRCqWUP9GMM1rThq33AK+nGUf0OZrgAXB1KeW7k+/WFmubtlLKrRHxnzSB5T61ZSPifsA3aAbHP7OUcs0Ei10I/DVwTinlz1P5+jQB7UcdX+Ns4J00A+glTYFjliRNWTuW5nU0PUcf7ph1C03IGP8z5S0TTAO4lolPzW0+dXaHG0JGxH1oeoY6p20dEVt3Tmt7dc5u327e/hdpgtxbI2KbCfbpbhExNK42ImI6pw6f0Qa18dO35vbxUOdX1l8FnErTY/esUspkPWCfaZ/fM8nXu1fH63tOsP6lNFcdTnnfJNmzJGliu0fEC9rXK7j9Dt67cfsdvDtvSPklmlNDP4jmDtgBPJ1mjM2VE2z/P4EXR8TR7fZupbnlwC+A9cBhbbD5efu1DwF+wx1/ye8MnBERX2nXuRJ4IPBSmlNVpwCUUn4fEYfQXKH2y4j4DM2l/ndv63sOzfio0ztqeyVwfER8AxgD/msL43z+HdgQEV9va94E/CVwAPAgYE0p5bzK+h9pa/ks8NCIeOi4+WeWUi4qpXw5Ij4BvATYrd33PwLDwN/QtNHmwdwfj+YeV9+mOf24FEg0A+TfX6lF0nj9vhzPhw8f8+fB7bcO2Py4lebKtwtpemgOAJZPsu5BwHk0Y5X+RHMPoB2Z4DJ54C+A/wCu4vZL7ndq592XJnz8sd3Wunbbq9vl9myXuwfN5fFnt9u5kWas0KeBnSeo77E0oe4PwE3t85nAm4G7dyy3BPjfwGXc3mO2eguf23NpBmWvb2sZowlv/7ete8m45e/wmdAEtVJ5rB63/j/R/Fuaa2iC2aU0A9tfOq6mr9Dcq+rGtq7/oglaS2r748OHjzs+/N9wkiRJFY5ZkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuaNSMjI8tHRkbeOjIysrzftah3bNfBZdsOJtt15gxLmk3LgaPaZw0O23Vw2baDyXadIcOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkiqW9rsA3W7HEw4v/a5hlmzodwG9dNmBx0a/a5AkzR17liRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVLF0n4XsFlKaV/gbcD9gSuB1+Scv5xS2hX4JLAbcBHwspzzDzvWeyVwBLAS+CbwkpzzxnbedsDHgWcAG4Fjcs7Hz91eSZKkhW5e9CyllP4e+D/AS4Eh4NHAupTSMiADpwCrgGOBr6aUVrXr7QUcBTwLuA+wHDiuY9MfogmEw8DewNEppSfPxT5JkqTBMF96lt4OvD3n/KP2/Z+AP7VhaGvgvTnnW4ETU0qvAZ4LfApYDZyQc14HkFJ6C/DTlNLLgAD2Ax6Rcx4FzkkprQEOAr4/Z3smSZIWtL6HpZTSVsBjgG+klH5NE45OA14N7Aqc1walzda102mfv9kxbz2wFfAgml6zyDmfP27d187GfkiSpMHU97AE3AtYBuwP7AlcC3ye5rTcr4EN45a/Bti2fb1N5/ycc0kpbaQ5lRc045TGrzvU2/JhZGRkOc0pQC0CIyMjK/tdQ59tPoaGRkZG+lqIes62HUy26ySGh4fH54QJzYewdH37/KGc82UAKaVjgK8A76QZuN1pW2C0fX3tBPNXtvOXbGHdXjqCZuyUFofxAX6xuqzfBWjW2LaDyXa9s5jKQn0PSznna1JKvwPKBLPXA4ellJZ0nIp7OPCRjvm70/RE0V45dwvwK5oPoKSUdsk5X9Cx7vpZ2I13Ae/rwXb8JbwwbLvlRQbaEM0P3R2ZnT8+1D+27WCyXWeo72Gp9UnglSmlbwLXAYfTXAV3OnAj8LqU0geBfYEH0FwdB7AGOCmldBJwMXA0cHLO+XqAlNJamivgDmzXW01zuq+nhoeHNwGber1dzU9T7bYdVB3d+KOL/bMYNLbtYLJdZ25e3DqA5nTbj4Dzgd9w+32WxoAEPI9mvNGbgH1yzlcB5JxPowlI3wIuB8aAV3Vs9xU0PVaXA6cCR+acvzcXOyRJkgZDlDLR2S/1w44nHG5jLACXHXjslM5xD6p2gPsGYFv/Sh0stu1gsl1nbr70LEmSJM1LhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqWNrvAgbB2NjYDsAO/a5Dc2NsbOyR/a6hn1atWrVidHSUoaGh3cfGxq7rdz3qHdt2MNmuk1u2bNnZU1nOsNQbhwBH9bsIzZmf9buAflq6dCmrVq0COKPftai3bNvBZLtWxVQWMiz1xseA3IPtLOpfwgvIo/pdQD/dfPPNK0ZHR88YGhraY+nSpf6VOkBs28Fku86cYakHli1bdjlweb/r0NyYarftoLriiitWAlx99dXnDg8Pb+x3Peod23Yw2a4z5wBvSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVLG03wV0SindA/gl8Ouc8+PaabsCnwR2Ay4CXpZz/mHHOq8EjgBWAt8EXpJz3tjO2w74OPAMYCNwTM75+LnbI0mStND1pGcpIp4QES+OiAfPcFP/Bpy3+U1KaRmQgVOAVcCxwFdTSqva+XsBRwHPAu4DLAeO69jeh2gC4TCwN3B0SunJM6xRkiQtItMOSxHx+Yg4oeP9S4EzaHpw1kXEU7opJKW0J/BA4LMdk/cEtgbem3PelHM+EbgYeG47fzVwQs55Xdub9BbgH1NKW6eUVgD7AW/OOY/mnM8B1gAHdVOfJElanLrpWXoCcGrH+yNoTpOtBNbS9PRMS0ppOU0v0MuB0jFrV+C8nPOtHdPWtdM3zz+3Y956YCvgQcDOQOScz59kXUmSpC3qZszSPYHLASLiocBfAh8opVwbEZ8BvtTFNo8Avp1zPi+l9KiO6dsAG8Ytew2w7UTzc84lpbQRGAKCZpzS+HWHuqivamRkZDnNKUAtAiMjIyv7XUOfbT6GhkZGRvpaiHrOth1MtuskhoeHx+eECXUTlv4M3A/4IfB04PJSys/beVsxzd6qlNKDgBcCu08w+1qaHqtO2wKjlfkr2/lLtrBuLx1BFz1qWrDGB/jF6rJ+F6BZY9sOJtv1zmIqC3UTlk4F3h0Ru9OMGfpcx7xdacYUTcff0QzOviSlBE0PzV1TSlcCBwMPSykt6TgV93DgI+3r9TQh6/Nw25VztwC/ovkASkppl5zzBR3rrp9mfVPxLuB9PdiOv4QXhm23vMhAG6L5obsjs/PHh/rHth1MtusMdROWXk/Tg/R0mkv1O3tUngN8e5rb+yLw3Y73+9H0NCXgj8CNwOtSSh8E9gUeQHN1HDQDtk9KKZ1EE9KOBk7OOV8PkFJaS3MF3IHtequB/adZ3xYNDw9vAjb1eruan6babTuoOrrxRxf7ZzFobNvBZLvO3LTDUillA5NcUVZKecJ0t9cGm+s3v08pXQ3clHO+rH2faAaQv53mPkv75Jyvatc9LaV0NPAtbr/P0qs6Nv8K4BM0Y6w2AkfmnL833RolSdLiFaWULS/VuULERcBzSinnTjBvVyCXUh7Qo/oWlR1POHx6jaG+uOzAY6d0jntQtQPcNwDb+lfqYLFtB5PtOnPd3DpgJya/8mtrmqvjJEmSBsKUTsNFxF1pgtDmv6hXRsTdxy12V2AfwOsSJUnSwJjqmKU3AEe2rwv1QdxvnUlBkiRJ88lUw9JXgEtoepY+DbwD+M24ZW4CLiilrOtZdZIkSX02pbDUDuY+FyAiCvCNUsqVs1mYJEnSfNDNrQM+MxuFSJIkzUfTvhouIu4WEe+MiAsj4vqIuGXc4+bZKFSSJKkfurmD94eBA2j+Ye4JNGOVJEmSBlI3YenZwKGllON6XYwkSdJ8081NKW8BftnrQiRJkuajbsLSR2n+0a0kSdLA6+Y03HXAEyPiJ8BpwDXj5pdSyvtnXJkkSdI80E1Yenf7fF/gsRPML4BhSZIkDYRu7rPUzak7SZKkBcngI0mSVDHtnqWI2GNLy5RSzuiuHEmSpPmlmzFLp9OMS4qOaWXcMlt1W5AkSdJ80k1YevQE0+4O/APwHOClM6pIkiRpHulmgPfPJpl1WkSMAgcD351RVZIkSfNErwd4/xh4Vo+3KUmS1De9Dkv7AFf1eJuSJEl9083VcHmCyXcBHkxzo8rDZlqUJEnSfNHNAO+V3PnqtxtpximtLaV8e8ZVSZIkzRPdDPDecxbqkCRJmpe66Vm6TUTch+a2AVeVUn7fm5IkSZLmj64GeEfE8yPiIuBSYB1waURcFBEH9LQ6SZKkPutmgPc/A58DvgO8DfgDcG/gn4DPRcStpZQv9LRKSZKkPunmNNwRwAmllIPHTf9MRHwKeCNgWJIkSQOhm9NwOzN5GDq5nS9JkjQQuglLVwIPnWTeQ9v5kiRJA6Gb03AnA8dExA3AF0spV0fEdsD+wNHAR3pZoCRJUj91E5beCOxEE4qOj4ib2+0E8B/Am3pWnSTNU4e95uvjb847SDbA2f2uoWfe8/69o981aGHr5qaUm4B9I+JhwBOBVTT/D+5HpZTzelyfJElSX3V9U8o2GBmOJEnSQJv2AO+I+MeIOHSSea+PiP1mXpYkSdL80M3VcIcDmyaZd0M7X5IkaSB0e5+l9ZPMOx/vsyRJkgZIN2HpRuBek8zbAbi5+3IkSZLml27C0g+AwyNiRefE9v1hwOk9qEuSJGle6PY+Sz8BfhMRa4ERYBh4HnAXmn+oK0mSNBC6uc/SLyLi0cDbgH2BewB/Bk4D3lZK+XVvS5QkSeqfru6z1Aai5/e4FkmSpHmnmzFLkiRJi4ZhSZIkqcKwJEmSVNH1/4bT7cbGxnaguceUFoGxsbFH9ruGflq1atWK0dFRhoaGdh8bG7uu3/VIW+Ix6zE7mWXLlp09leUMS71xCHBUv4vQnPlZvwvop6VLl7Jq1SqAM/pdizRFHrMes5OJqSw07bAUEW+kuTHlWaWUsemuP6A+BuQebGdRH9ALyKP6XUA/3XzzzStGR0fPGBoa2mPp0qWL+a9Uj9eFw2PWY3ZGuulZeinwDuDGiDiLJjidAZxZSrmhl8UtFMuWLbscuLzfdWhuTLXbdlBdccUVKwGuvvrqc4eHhzf2ux5pSzxmPWZnatoDvEsp9wUeCLwc+A1wAM0NKa+JiJ9ExLG9LVGSJKl/uroarpRycSllTSnl4FLKg4CnAGcCjwUO7WWBkiRJ/dTVAO+IeCjwpPaxB7A9sB44jua0nCRJ0kDoZoD3FcBK4ByaYPQ54IellA09rk2SJKnvujkNt4TmUrvSPm5tnyVJkgbOtHuWSin3iIiH0Zx+2wN4EXDPiPhvmqviziilnNLbMiVJkvqj2wHe55VSPlxK+cdSyjDwD8BG4H8Ba3tZoCRJUj91O8D7ftxxgPcDgFuAn+IAb0mSNEC6GeD9W2BH4CaacPQFbr8ppXcGlSRJA6WbnqVP0ISj/yqlbOpxPZIkSfNKNwO83zEbhUiSJM1HXQ3wjoidI+LTEfGriPhz+/ypiNi51wVKkiT1Uzdjlh4FnA5sAr4G/AG4N5CA/SPiSaWURf1PCyVJ0uDoZszSe4Bzgad1DuiOiBXAt9v5T+1NeZIkSf3VzWm4xwHvHn/lW/v+Pe18SZKkgdBNWLoJWDHJvBXAWPflSJIkzS/dhKXvAu+MiL/unNi+Pxr4Ti8KkyRJmg+6CUuvpflHuusj4tyI+HZErAPWt9t7XS8LlCRJ6qdph6VSyu+Ah9GEpl+22/gl8Bpgt1LKZT2tUJIkqY+mdTVcRNwFeDawrpTyQeCDs1KVJEnSPDGtnqVSyk3AScBfzk45kiRJ80s3Y5Z+gWFJkiQtEt2EpSOAN7d38pYkSRpo3d7Be3vgrIi4EvgTUDrml1LK7r0oTpIkqd+6CUs/A/5frwuRJEmaj6Ydlkopq2ehDkmSpHmpmzFLkiRJi4ZhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKliab8LSCktBz4MPAXYHrgUeGfO+aR2/q7AJ4HdgIuAl+Wcf9ix/iuBI4CVwDeBl+ScN7bztgM+DjwD2Agck3M+fo52TZIkDYD50LO0FBihCUvbAocAx6eUHp9SWgZk4BRgFXAs8NWU0iqAlNJewFHAs4D7AMuB4zq2/aF2+8PA3sDRKaUnz8VOSZKkwdD3nqWc83XAkR2TfpRS+jHwt8A2wNbAe3POtwInppReAzwX+BSwGjgh57wOIKX0FuCnKaWXAQHsBzwi5zwKnJNSWgMcBHx/LvZNkiQtfPOhZ+kOUkorgL8B1gO7Aue1QWmzde102udzO+atB7YCHgTsDETO+fxJ1pUkSdqivvcsdUopBXACcBbwHeAxwIZxi11Dc7oOmp6n2+bnnEtKaSMwRNOztHGCdYd6XffIyMhymlOAWgRGRkZW9ruGPtt8DA2NjIz0tRBpKjxmPWYnMzw8PD4nTGjehKU2KH2UZuzRP7TB51qagdudtgVG29cTzV/Zzl+yhXV76QiasVNaHMYH+MXqsn4XIE2Rx2zDY/bOYioLzYuw1AalDwOPAJ7ajmOC5rTaYSmlJR2n4h4OfKRj/u7A59vt7ArcAvyK5gMoKaVdcs4XdKy7fhZ24V3A+3qwHQ/ohWHbLS8y0IZofujuyOz88bFQeLwuHB6zHrMzMi/CEs1Va48DnrL5sv/W6cCNwOtSSh8E9gUeQHN1HMAa4KSU0knAxcDRwMk55+sBUkpraa6AO7BdbzWwf6+LHx4e3gRs6vV2NT9Ntdt2UHV0448u7s/i7H4XoCla3N+nHrO90PcB3iml+wEvBx4C/C6ldG37eGPOeQxIwPNoxhu9Cdgn53wVQM75NJqA9C3gcmAMeFXH5l8BlHbeqcCROefvzc2eSZKkQdD3nqWc82+pnDPMOZ8HPLYy/zjueG+lznnX0Nw+QJIkqSt971mSJEmazwxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSapY2u8CpEF34eqtSr9rmCUbLux3BT2285pbot81SJp/DEuSJLX+8JD3DtwfNx2nkDb8oX9l9Ny9zz90zv648TScJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVLO13AYNgbGxsB2CHftehuTE2NvbIfteg2WHbDibbdTD1ol2XLVt29lSWMyz1xiHAUf0uQnPmZ/0uQLPGth1Mtutg6kW7xlQWMiz1xseA3IPteEAvDI+a5vK268Ixnba1XRcO23UwTfdncdcMSz2wbNmyy4HL+12H5sZUu2218Ni2g8l2HUxz2a4O8JYkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRVGJYkSZIqDEuSJEkVS/tdwGxLKW0HfBx4BrAROCbnfHx/q5IkSQvFYuhZ+hBNKBwG9gaOTik9ub8lSZKkhWKge5ZSSiuA/YBH5JxHgXNSSmuAg4Dv97M2SZK0MAx0WAJ2BiLnfH7HtHXAa3v5RUZGRpYDy3u5Tc1fIyMjK/tdg2aHbTuYptOui+F0y6DoxfE6PDy8cSrLDXpY2oZmnFKna4ChHn+dI4CjZrqRs572P3tQiubAhuksvM07fzdbdaj3ph0ToGIAAAXYSURBVNy2rz70kbNZh3pryu1663efP5t1qLem9bN4EjGVhQY9LF0LjE+e2wKjPf467wLe1+NtDoIh4DJgR3r/mat/bNfBZdsOJtt1hgY9LF0IlJTSLjnnC9ppDwfW9/KLDA8PbwI29XKbg2BkZGTzy9GpdnVq/rNdB5dtO5hs15kb6NOzOefrgLU0V8ANpZR2B1YDJ/S1MEmStGAMdFhqvQIowOXAqcCROefv9bckSZK0UAz6aThyztfQ3D5AkiRp2hZDz5IkSVLXDEuSJEkVhiVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpIoopfS7BkmSpHnLniVJkqQKw5IkSVKFYUmSJKnCsCRJklRhWJIkSaowLEmSJFUYliRJkioMS5IkSRWGJUmSpArDkiRJUoVhSZIkqcKwJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUsbTfBWjwpZRWAy/NOT+ufV+AXXLOv+hrYZKqUkrPBw7OOf99v2tR91JKOwEXA3fLOd+YUjoVWJtz/tQEy94VuAG4f875kjktdB4zLEm6TUrpEppg+60ZbGMnOn4w96g0dWGm7ZlzPgk4qadFqe9yzs/odw0LjafhJGkRSikt63cN0kJhz5J6JqV0KHAIcC/gMuDNOef/6G9VmqqU0ueA+wKnpJRuAT4ArAE+CDwa2AD8W875+Hb5xwAfBh4M3Ah8Kef8CuCMdpNXppQAnjeTnip1Z5L2fCPwqvaxA7CydtxOcgr9EOC17fpfAw7KOd80h7u2aKWU3gA8Mee8d8e0NwKPBT4KHAP8Fc2xugY4MudcJtjO6cAXcs4fTSktadd7Mc3pt7fN8m4sSPYsqZcuBp4EbAu8FTgxpbRjXyvSlOWcXwhcCjwn57wNzQ/Q7wKZ5hfjM4HDU0p7tat8APhgznklcH/gxHb6Hu3z9jnnbQxK/TG+PXPOb2pnPQ94Ik04gukft88F/hZ4EPB44AW9r16TOAnYK6W0fce059Mce9cBq4HtgGcB/wrsO4VtHgzsRxO4Hgo8u4f1Dgx7ltQzOee1HW9PTikdQXMAamHaG/hDzvkj7ftfppQ+AfwzcBowBvxVSumeOecrgJ/0qU5Nz7E55z9tflM5bi+rrH8VQErpG8AjgU/PVrG6Xc75spTSmcD+wPEppYcDOwJfGzc+8L9TSv9OE4LXTrCpTv8MfCDnfBFASuko4H/0vvqFzbCknkkpvYime36ndtI2wPbApn7VpBnZCXhESumajmlbAT9sXx9M02V/fkrpUuAdOedT5rZEdeHSzjeV43Yyf+h4fT1w714Wpy06ETgQOJ6mV2lte4XbY4FjgV2BuwDLgakcj8Pc8Xvit70tdzAYltQTKaX7AZ8Engr8OOd8S0rpHCD6W5mmqXN8w6XAmTnnPSdaMOf8K+CAdsxDAr6YUrrXuG2ovyZqi1s3v/C4XZDWAsellB5A0yv0wnb654GPAM/MOd+QUno/zenzLRmhGdu22X0nW3AxMyypV1a0z1fAbX+t7tq/ctSlPwIPbF9/HTg2pXQwzV+ztwC7AHfNOf80pfQC4Ns55ytSShtpfsHeTPM9cGu7nZ/P9Q7oDjrbcyIetwtMznlDe/rzozRh+AftrCHg6jYo/Q1wAPD9KWzyZOD1KaWvA1fSjFvTOA7wVk/knM8H3gv8mOYH9MOBM/talLrxLuCw9tTbG4C9aHqNfkfzC/XjwMp22acBP08pXUtzVdwBOefRnPP1wDuA76eUrkkpPW2ud0K3ua09U0pHj5/pcbtgfY7m2Px8znlzT+HLgSNTSqM0gedLU9zWJ2lO150FnA98o7elDoYoxR5zSZKkydizJEmSVGFYkiRJqjAsSZIkVRiWJEmSKgxLkiRJFYYlSZKkCsOSJElShWFJkiSpwrAkSZJUYViSJEmqMCxJkiRV/H/+CWZQiS+1KgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<ggplot: (8771582291209)>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_toplot = pd.DataFrame({\"datasets\": [\"all\",\"train\",\"valid\",\"test\"],\n",
    "                            \"size\": [winequality_df.shape[0],\n",
    "                                     train_df.shape[0],\n",
    "                                     valid_df.shape[0],\n",
    "                                     test_df.shape[0]]})\n",
    "ggplot(data_toplot, aes(x='datasets', y='size', fill='datasets')) + \\\n",
    "    geom_bar(stat = 'identity') + \\\n",
    "    scale_fill_brewer(type = 'qual', palette = 2) + \\\n",
    "    labs(title = 'Dataset Sizes', x = None, y = 'row counts') + \\\n",
    "    theme_minimal() + \\\n",
    "    theme(legend_position = 'none')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create segmented models\n",
    "\n",
    "Wine dataset contains two types of wines: white and red. We will create a model for each type using H2O egmented model training function `train_segments()` by defining segment with column `type`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm Segment Models Build progress: |██████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "wine_gbm = H2OGradientBoostingEstimator(seed = 2020)\n",
    "wine_models = wine_gbm.train_segments(x = predictors, y = 'quality', \n",
    "                                      training_frame = train_df, \n",
    "                                      validation_frame = valid_df,\n",
    "                                      segments = ['type'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieve models using H2O frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_models_h2o = wine_models.as_frame()\n",
    "model_id = (wine_models_h2o[wine_models_h2o[\"type\"] == \"red\"][\"model\"])\n",
    "wine_red_gbm = h2o.get_model(model_id.flatten())\n",
    "model_id = (wine_models_h2o[wine_models_h2o[\"type\"] == \"white\"][\"model\"])\n",
    "wine_white_gbm = h2o.get_model(model_id.flatten())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, retrieve models using Pandas frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_models_pd = wine_models.as_frame().as_data_frame()\n",
    "model_id = wine_models_pd.loc[wine_models_pd['type'] == \"red\", 'model'].values[0]\n",
    "wine_red_gbm = h2o.get_model(model_id)\n",
    "model_id = wine_models_pd.loc[wine_models_pd['type'] == \"white\", 'model'].values[0]\n",
    "wine_white_gbm = h2o.get_model(model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, retrieve models using datatable (Pydatatable):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_model_dt = dt.Frame(wine_models.as_frame().as_data_frame())\n",
    "model_id = wine_model_dt[dt.f.type == 'red', 'model'].to_tuples()[0][0]\n",
    "wine_red_gbm = h2o.get_model(model_id)\n",
    "model_id = wine_model_dt[dt.f.type == 'white', 'model'].to_tuples()[0][0]\n",
    "wine_white_gbm = h2o.get_model(model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model performance\n",
    "\n",
    "Predicting and scoring segmented models should take place on the respective segments of validation and test sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_white_df = valid_df[valid_df[\"type\"] == 'white']\n",
    "valid_white_perf = wine_white_gbm.model_performance(valid_white_df)\n",
    "\n",
    "test_white_df = test_df[test_df[\"type\"] == 'white']\n",
    "test_white_perf = wine_white_gbm.model_performance(test_white_df)\n",
    "\n",
    "valid_red_df = valid_df[valid_df[\"type\"] == 'red']\n",
    "valid_red_perf = wine_red_gbm.model_performance(valid_red_df)\n",
    "\n",
    "test_red_df = test_df[test_df[\"type\"] == 'red']\n",
    "test_red_perf = wine_red_gbm.model_performance(test_red_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute standard deviations on red and white segments for all data as baseline metrics to compare RMSE scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdev_data = winequality_df.group_by(\"type\").sd().get_frame()[['type','sdev_quality']].as_data_frame()\n",
    "sdev_data.columns = ['type', 'value']\n",
    "sdev_data[\"dataset\"] = \"all: std.dev\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize RMSE scores on all segments using validation and test predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAAHICAYAAACI+LbJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeZhkRZ2v8fcH3QI2zaIg0ulFQEBQXHAB90EdF5Q56riMgCigKG6juA2gAuKw6FUcR0YBZVgERYcRPeoFAQFxHxQVHZAdGrIQGmm6m72AuH/ESTvJri2qqzorq9/P89TTlefEiRMZmVn9zcg4kZFSQpIkSdLErdHvBkiSJEmDxhAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtjSMiToqISX+1Z0TsFREpInaewmZNuYh4SkScFxGLm/Ye2u82zWYr+7ySJPWXIVozVkTs3IS5FBEnj1ImIuL6pswDq7qNU60rcHf/LIuI/4mI90XEmtN03jnAd4BtgE8Ceza3NYtExLoR8S8RcUlE3B4RdzWvn7Mi4mP9bt9MEREbRMShM/2Nr6T+mtPvBkgTcC/whoh4f0ppac++lwGPb8rMXeUtmz5fAX4BBNAC9ga+BGwP7DcN59sS2Ar4UErpmGmoX30WEfOBXwPbAd8Hvg7cCWwBvBD4BPDZvjVwZtkAOKT5/cI+tkPSDGaI1iD4DrA7sBtwXM++dwALgRuA563idk2nX6SUTu3ciIivAJcB74yIQ1JKt0zFSSJi3ZTSncBjm02Lp6LeUc6h/noHOUAfnVL6cO/OiHjcqm+SJA0up3NoEFxOHpV9e/fGiNgIeA1wIvDQSAdGxLYRcXpE3BIR90XEtRHxuYhYb4Sym0TEyRHx1+Zj7p9HxItHa1REPKGZ1zoUEfdHxE0R8eWmXVMqpbQE+CV5ZHrLrjbMj4jDI+KK5v7dHhHfjYin9rS1MzVmr4h4V0RcGhH3AsdExPXAT5qiJ3ZNI9m8OXbtiDgkIv4cEfc25/h+RDxrhD5JTZ/sHBEXRsRS4I/Nvs5UlZdGxEHNY3FvRPwhInZpyjwpIn4QEUuan1OaEdTuc2wbEf8REX9qytwTEX+MiI/0TnfpOudLmv1XdT0P9h+pryPiqRHxza7HtR0R34uIZ/aUm5LHPyI2aupZ1NyXX/Q+75r72h5pOk9E7NTcx38d51RPbP69YKSdKaWbRqh7k4j4UjPl4/7mdXRq57nRU3bTiPh67+snRpj73Tw3ro+IzSLivyLPw18SEf8dEY9pyuzT3O97I+K6iHh77zmbci+OPB1lcfPYXh55ykrvc6FzzgUR8Y3meXxPRPyk+7GNiL2A65qbh3S9Hq7vKvPKiDg/Im5tznlzRJwdEc8fqY2SZidHojUovgb8Z0Q8JaX0x2bbW8nP4ROBnXsPiIinAxcBa5KnR1wLvAD4MPD3EfG8lNLdTdn5wE/JUxpOAi4Gngz8ALhmlLovBO4G/pM8Er418G7gpRGxYxN8p0RERFM/wKJm23rAz5o2nwz8AdgQ2Bf4ZUS8MKV0SU9VHyCPOh8P3AQsA74LPBs4qNn+0855miDy/4AXk/vimOb4dwM/i4hdUkq9oexZwBvIj8s3gPk9+48CHkF+TB4E/hn4XkS8ATgB+DZ5usFzgbcB9zX3qWNn4O+aMjcA6wCvAv4v+Q3Ge0bowiOadvwneQrDW4GjI+LmlNLpnUJNmD+zOecJwJ+BjZrzPQ/4bVNuKh//HwFLgE8DjyJP1zmn6dvzmjLHAf8OvBqoe45/B5Ca9o7l2ubft0TEj1NK94xVOCL+D/nN67pN3VeSpxa9G3h5RDwrpbSwKbs++XmzJbk/fsvyaSNXj3KKeeQ+/DlwILAt8D7gsRFxJvBe4Kvk5+i+wNci4s8ppZ93tXEf8t+G35GfV3cAzweOBHYA3jzCOS8CfkOevrIJsD9wdkRsmVJa1uzfH/gC+bnQuTbgzuacLyK/Fi4jP+f+Sn5NvAB4WnN/JK0OUkr++DMjf8hhKZH/s5sHLAX+rWv//wI/an6/EHig5/iLyCPUz+nZfnCn3q5thzXbPthT9s3N9tSz/XfkUPKonu07AQ8Ah3Rt26upY+cJ3OdO2f3I4W1j4OnkYJKAX3aV/TfgfmCnnjo2AG4ELhihLxcDjx2jr/fq2b5Ps/3LPdu3Ic9DvxJYo2t7an5eOcZ9+wOwVtf2pzbbHwLe1HPMd5v7uG7Xtnmj9N1pTd9vOoFzzgNuI0+b6Wx7JHBr00ebj1B/9/2c8OM/xmN9UtO2uqfuxwN3NX0bzbb1m211Tx3rkkPmORM434bkqU+JHDZ/QL6I9O+BuSOU/y5wO7Blz/bNm3Oe2LXt8Kbe9/SU/UdGfv1c2Gw/oGf70c32m4D1u7Zv0jzfvtm17bHAPeSgGz31fLip5+9GOOeBo7zG39lzHxNw6Aj90mnjY8brc3/88Wd2/zidQwMhpXQXcDp5FO0REfE84EnkUagVRMTG5IulfpRS+lXP7s+RA8nru7a9nhwsvtxz3tOBq3rq3p4cbE8H1mg+jt+o+Rj/GvLI2ysmdUeX+wp5xPlWcmB7Gzlsva5pQwB7kKd4XNPThjnAOcALI2KdnnpPTin9paAdnT46rHtjSulK8ijz1sBTeo75Q0rp7DHqPCaldF9XXZeS3yDdnFL6dk/Zn5AvGN28q/xdnd8jYq2IeFRzv88mf+rwTFbUe867yH23TVeZl5PftPxbSun63gpSSg8155zqx//ITt3NeW4gvyHYmvwGg5RHtU8HXhURC7qO3Y0cpI8f7yQppcXkvjkSuIU8qn0YcC5wU0Ts0SnbjCz/A/lTiKU99/FO4Fc99/F15DcfD2tHSuk7wBWjNOkh8hvBbp1PQU5OXSP5KV8DcAXLP42B/GnH2uS/AY/uaeMPmjK9j8ND5BHmbuc2/27DxHSuG3hTRMymi5klFTJEa5CcADwaeC35I+zbgO+NUrYzb/h/e3ekPIXjGuAJXZufAFydUrp/hLou67m9XfPvgeSg2/vzRPLI2co4irzyyN8DzwEenVJ6TVcA3qj5edEobdiHHCh75+deWdiOLYG/jhK8O9NqntCzfbxzXDvCtsVjbIf8uAMQEY+MiCMj4jry6ORfyff5lKbIoyZ4zr9218vyEPW70ZsOTP3j3/v86t62Vde2Y8mP6d5d2/Ylv9Ea7XXwMCmlRSmlg1JKTyT308vIbxw3BE7pmtO7Dfn/hz0Y+T7+PQ+/j1sC16SURlpm8s+jNGcopXRvz7bO4z3ac6H78eo8Dj8YoX2dc/Y+DiucM6X01+bXRzMxxwD/Q14tZ3FEnBsRH4+ILSZ4vKRZwjnRGhgppV9HxJ/Ic2ifDhw/SuiFfAEe5I9dV1b03O68+fwSK85P7RhzvukE/G9aPh92JJ02XESeSzuaRT237y5sR1Deh+Od48HC7Z12dHwDqMgjkBeR30w9QB5lPYqRBwfGqnukc4xlVTz+HX/r+5TSxRFxCbBPRBxBHqV+NvDZlNJwccV5ZPo84LyIuJQc0vcmz+nt3Mdvk+clF7V1gsZ6TEbb1/0Yddr4DvKc9JEMFZxzQo9/SmlxRDyXPEf+ZeS50IeQL0LcM6X0rYnUI2nwGaI1aE5g+cexY11I1bkYcPveHc0Uhy15+AVP1wBbRcQjRgjm2/Xc/ttI6zhBdzotIo/MbTjNbbgaeGJEbJJWXFav07crXHg5XZppBhVwakrpnT37th75qAnrTDvYgdHDMUz94/8k8tSS3m2wYt8eS54y8VLyJzKJUaY0FfpF82+r+fdq8tSHdSZ4H68lv37mjhDot52C9o2k8zgsnobXwJhvCJrpNz9rfoiIxwOXkC9gNURLqwmnc2jQnAJ8Ctg/pbTCVI2OlNIi8vzKV0TEjj27P0yeR/rfXdu+Q74g72ErO0TEm3n4PEyA3wOXAm+PiN6A3fkWxY0ndncmp/lP/FTgKRHxtpHKRMTKTimB5SsTfLKn7q3Ia3dfRe6LVaUzd/hho4bN6iofWsm6zyG/OflgjLyEW+fv5VQ//gd21d0JZHuQg2xv336DPH/8/U2ZC1NKVzEBEfGciNhwlN2va/69DP42xeGHwKtjlGUee55f32X5yjDdZf6R5UvrTbVvk6fzHBoR647QvnWiZ3nEAp11zVeYGjTKY7uQ/NwZaSqRpFnKkWgNlJTS7cChEyz+z+SP+8+P/GUlnSXudiev1nB0V9nPkS/SOjryGsv/Qx5p3Zs89/dvF8+llFJE7AmcD1wSEScCfyJfALcFeYTwpIJ2TtYnyMt5nRQRryGPit0NbEYeqbybvDTdyjgZeAvw3ojYjLwcW2eJuwDelVKaiikzE5JSWhYRZwN7RMR95G/g25S8hvhKfQFNSunuiNib/MbhDxHxNfLo9IbkJe7OAr40DY9/izyl4kxyCHs3eQnA9/X2bUrprog4rSkDE59qAXkVinc2/fcrcujbkLwyy65AG/h8V/l3k59T5zbn/A35TczjyRclXkxe/QTyNx2+GfhSROxAXuLuSeTXzx/IS79NqZRSOyLeRV655oqIOJlmxRTyp0evIz8WF06i7r9GxNXAmyPiGvJz666U0veB47teCzeQ/x+tyG8Wei9alDSLGaI1a6WUfh8RO5HDzF7kZcKGyOH5sOYCw07ZJRHxQnIYeC05EPyOHC7eRs8KFCmlS5u1gg8AdiGHuLvJS8t9lzxKNq1SSksj4gXAB4F/Al5J/hh6iBwuTxnj8Ime48GIeBX5fu5GXu3gbnK4OiyldPHKnmMS3kL+2HzX5vfryRd7XUKe4ztpKaUfNvNdDwL2JH86sYjcnz/vKjeVj/8ryesNH0L+hOR3wB4ppR+PUv5YcsD9K8s/KZiI48nrUb+EvF74xuTlA68jv4n8XErp1k7hJqQ+A/gY+TXxT035NvlTnhO6yt7RvH4+Q141Y3fy4/Eq8vNzoitfFEkpnRIRVwAfIV9M+yjysnzXNvdpZT4l2YMcio8gL394A8u/Lv2t5OfHxuTH/SrgnYy/VrekWSRW4SCSJGklRcS25G/x/EJKaWWnsEy75mLgNVNKK0x9kaRB5pxoSRosHyR/4nBcvxvSLSIeOcK215G/+fNHq75FkjS9nM4hSTNcRMwjf/nJtuSL976dUhrtS0z65fsRMUSeDz1MXnLwreT5xJ/pZ8MkaTo4nUOSZrhmtZDryN+0eR7w9q4vCZkRIuKD5NC8BXlu963kFU8OSSkt7GfbJGk6GKIlSZKkQs6JliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFamqRWq/WXVqu1c7/bIWniWq3W9a1W65Wj7Nus1Wrd2Wq11lrV7ZI0eOb0uwGSJM0E7XZ7IbBu53ar1boQOL3dbh/bt0ZJmrEciZaAVqu1ZqvV8vUgSZImxJForbZardb1wLHAPwHbAdu0Wq2PA68mvzbOAD7SbrfvbcrvD3wEmAt8ph9tljSyVqu1J/DWdrv9sub2r4Dhdrv9wub2j4H/bIo/udVqHQ5sDfwM2KPdbi9utVqbA9cB6wCfBF4IPKfVan0OOLPdbu/ZarUeC3wR2Bm4HzgBOKzdbj+0au6ppJnCkTet7t4CvB5Yj/wf41rAk4Btyf/BfhKg1Wq9DPgE8A/AZs2+jfrQXkkjuwB4XqvVekSr1VoX2ALYutVqPbLVaj0CeC5wYVN2d+C1wAJgA2D/3sra7fbHgZ8CH2y32+s2AXoNoAauBh4P7AS8Bnj7tN4zSTOSIVqru2Pa7fa15P9IdwXe3263l7bb7TuAfwV2a8rtBpzUbrcvaUamD8DXjzRjtNvtm4C/ADsCzwd+CfwaeB457Lbb7Xa7Kf6ldrt9Y7vdvpP8idMzJniaZwH/B/hEu92+t91uDwFHs/zvhKTViNM5tLpb2Py7ObAmcGOr1ersi2Yb5BGrP3R2tNvtO1qt1tJV1EZJE3Mh8GLgkc3v0dy+j+Wj0JDDdsfddF1MOI7NgY2BxV1/J9YAbpxccyUNMkO0VnedeYwLgQeAx7Tb7ftHKDdEnsYBQKvVWp88BUTSzHEhsBc5RL+bHKKPIYfoEyZRX+q5vRC4qd1ubz7pFkqaNQzREtBut//SarV+CHyx1WodBNwBPA54crvdPhv4FnBaq9X6OvBn4EiWB3BJM8MFwHHAMuDSZttW5De8e06ivluAJ3TdvhhY1Gq1PkmexnFPs39Bu93+yWQbLWkwOadTWu5t5NHo3wNLgB8B2wC02+0fkYPzD8mjUdcAt/WnmZJG0syLHgJ+2W63H2pWzPglefS4PfbRI/oiULVarcWtVuvkdrv9IPni4m2Aq4DFwLeBTafmHkgaJJFS76dVkiRJksbiSLQkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhl7hbjQwNDa0FHAgcuWDBgvv63Z6Zzv4qZ5/NDENDQ+uRV5hZf8GCBX4p0ATYZ2XsL8mR6NXNWsAhzb8an/1Vzj6TJK0WDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBWa0+8GSNJs8rgTD0j9bkOXJf1uAMBNex8V/W6DJE01R6IlSZKkQoZoSZIkqZDTOSZpeHh4U2DTfrejxIYbbjhv2bJlzJ8//2nDw8N39bs9M539VW5Q+2zu3LmXjLZveHj4GauyLbPRIPThoD53+2WQ+2us17tUIlKaSdP3Bsfw8PChwCH9boeklTd37txR5+wODw8X/ZHc4tRPrnyDZpnr3vLpfjdB+puxXu9SCUeiJ+84oO53I0o88MAD85YtW3bR/PnzXzRnzpyBGjnoB/ur3Czts2cWlv/ttLRisJX24So3S5+708b+kgzRkzZ37tybgZv73Y4SixYtWg9g8eLFf1iwYMHSfrdnprO/ys3GPvOj35U3CH04G5+708n+krywUJIkSSpmiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQnP63YDVweNOPCD1uw09lvS7ATftfVT0uw2SJEmT5Ui0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSoTn9bgBAVVUbAMcDuwBLgcPruv7yCOUOAg7q2rQGsA7wzLquL6mqai/gBOCerjLvquv6tOlquyRJklY/MyJEA8eQ27IA2Ao4r6qqy+u6vqC7UF3XRwBHdG5XVfVu4EN1XV/SVeziuq6fswraLEmSpNVU30N0VVXzgDcCO9R1vQz4XVVVJwH7ABeMdWxT5sTpbaEkSZL0cH0P0cA2QNR1fVnXtt8DHxrroKqqtgeeAbyuZ9dTq6paBCwB/hs4tK7re3qPlyRJkiZrJoTodcnzoLvdAcwf57h9gHPqur6pa9tFwPbA9cDWwCnAZ4H3T0lLuwwNDa0FrDXV9a4uhoaG1ut3Gyag8xycPzQ01NeGDJCB7LMFCxb0/g36mwF5rs5oA9KHA/nc7aOB7a+xXu9SiZkQou8Eev/Arg8sG+2AqqrmAm8B3tu9va7ra7tuXlFV1QHAN5mGEA0cCBwyDfWuLpb0uwEFbhq/iHoMWp/FGPsG6bk6Uw1SHw7ac7ffBrG/xnq9SxM2E0L0lUCqqmq7uq4vb7Y9HfjTGMfsCqwJ1OPU/RDT92I5Ejh6gmUH6T+QVWX9fjdgAuaT/4N4HGO8qdPDzMY+K32u+npfka/32cf+0mqv7yG6ruu7qqo6A/h0VVV7A1sCewFvGuOwvYHT6rq+r3tjVVW7AL+v6/rmqqq2BI4CzpyOdi9YsOA+4L5xC2pEg/BxWtdHlMsGob0zwWzss9lyP/ppEPpwNj53p5P9Jc2AEN14L/BV4Gby/OiD67o+v6qqzYDLgCfVdb0QoKqqTcjrST97hHpeApxYVdV84DbgDODgVdB+SZIkrUZmRIiu6/oO8jJ3vdsXki887N52CzB3lHo+Cnx0OtooSZIkdfi135IkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVKhOf1ugCRp9XXlXmumfrehx5Ir+9yAbU56MPrcBEkT4Ei0JEmSVMiRaEmSBsjH9v/BTBq9XwKX9LUBn/3Cro7cqy8ciZYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKzel3AyStvI/t/4PU7zb0WAKX9LUBn/3CrtHXBkiSZjVHoiVJkqRChmhJkiSpkCFakiRJKuSc6EkaHh7eFNi03+0YVMPDw8/odxvGs+GGG85btmwZ8+fPf9rw8PBd/W6PypQ8x+bOnTvqBO5BeK7OdPZhGfurTGl/jfV6l0oYoifvXcAh/W7EAPttvxswnjlz5rDhhhsCXNTvtmhSSp5jY12EOOOfqwPAPixjf5Up7S8vOtaUMERP3nFAPcGy/kFc0TPH2nndvmvbZz22+Oq9Y/WZ/bWiMZ9j01iPj8WKfO6WGe85Z5893FS91qUihuhJmjt37s3Azf1ux6Dy47Ry9lmZqeov+33l2Ydl7K8y9pf6xQsLJUmSpEKGaEmSJKmQIVqSJEkqZIiWJEmSChmiJUmSpEKGaEmSJKmQIVqSJEkqZIiWJEmSChmiJUmSpEKGaEmSJKmQIVqSJEkqZIiWJEmSChmiJUmSpEKGaEmSJKmQIVqSJEkqZIiWJEmSChmiJUmSpEKGaEmSJKmQIVqSJEkqZIiWJEmSChmiJUmSpEKGaEmSJKmQIVqSJEkqZIiWJElaxSJio4hIEbFXwTE7R8RB09isibThtRHxnn62YaYwREuSJA2GnYG+hmjgtYAhGkO0JEmSVMwQLUmSNM0iYt+IuD4i7o6IHwNb9ex/a0T8LCJuj4jFEXFhROzYtf9Q4BBgXjMNJEXEhc2+bSPi9Ii4san/soj4cESs0XOOAyLi6oi4NyJujYjzImKLrv1rRcQREXFDRNwXEZdHxO5d+08C3gY8uasNJ015Zw2IOf1ugCRJ0mwWEbsCxwMnAacDz2r+7bY5cApwDfAIYHfgooh4akrpSuBrwOOa7S9pjlna/NsCrgBOA5YBTwc+BcwDDmva8Fbg08DBwC+B9YEXAut1teHbwAuaYy8HXgWcGhGLU0pnNcdvDGwL7NEcs2hSnTILGKIlSZKm1yeAn6aU9m5u/ygi5gEHdgqklA7r/N6MIJ8LPBvYCzgopXRTRNwEPJRS+lV35SmlHwM/bo4N4GfAI4H30YRoYEfg0pTSkV2Hfq/rnC8GKuAVKaVzms3nRkSLHKrPSildExGLgMf3tmF15HQOSZKkaRIRawLPBM7s2XVGT7ntIuLMiLgFeBAYBp4IbDOBc6wdEZ+KiKuB+5pjDwc2jYh1m2KXADtExNER8YKImNtTzcuB24HzI2JO54cczndo7oe6OBItSZI0fTYm561be7bf0vklIuYD55CnRnwIuAG4lzyFY+0JnOMzwL7kEePfAncAryGPgK8N3EmeSjIfeCewP7AkIk4GDkgp3QNsBDyKHMBHsilw0wTastowREuSJE2fRcADwGN6tm/S9ftzyfOdd00p/aGzMSLWZ2LB9Y3AcSmlz3Qd++ruAimlh4AvAl9spmi8GTgKuI081/n2pq2vGuUcvW8CVntO55AkSZomKaUHyVMpXtez6w1dv6/T/Ht/Z0NEPI98sWG3+4G1RjjNOj3HrkkOyaO1qZ1S+jxwKbBds/k88qj5/Sml34zw06n/fiY2Oj7rORItSZI0vQ4HvhcRJ7J8dY7du/b/ijzl4j8i4ijyahuHAu2eei4H5kTEB4BfAEtTSleQL0LcNyIuI48mv5eesB0RxwGLm3MtBp4PPA34MkBK6dyI+D5wdkR8lhyw5wFPBrZKKb2jqw37RMRuwFXAbSml6yffNYPLkWhJkqRplFKqgf2AlwLfBV4G7Na1/xbylIzHkFfM+GBT/uqeqr5PDr0HAr8Gjmu2vx/4CfAl4D+BPwJH9Bz7C/LydScAZ5OXqNs/pXRCV5k3AMeSv5HwrKbsy5u6O04A/qs518XksL9aciRakiRpmqWUjmN56O2Irv1nk8Ntt7N66niAPMr83p7tt7DidBHIFyZ2ypwMnDxOG+8nL4l32BhlltL1BmB15ki0JEmSVMgQLUmSJBVyOockSdI0eNyJB3xm/FLFlt2091H/Og31qpAhWpIkaXp8bBrq/AtgiJ4BZkSIrqpqA+B4YBdgKXB4XddfHqHc5sB1wF1dm0+t63q/rjLvI1+1uh7w/4B967peOn2tlyRJ0upmRoRo4BhyWxYAWwHnVVV1eV3XF4xSfqO6ru/t3VhV1cuAQ8hLx1wLnEJeguVt09JqSZIkrZb6fmFhVVXzyGsjfqKu62V1Xf+O/P3u+0yiur2AE+u6/n0z+vxJ4J+qqnrkVLVXkiRJmgkj0dsAUdf1ZV3bfg98aIxjrq6qag3gIuCjdV3f2GzfnjyFo+NPwJrA1sAfmEJDQ0NrMfJXb2oChoaG1ut3GwaNfVampL8WLFgw6pQv+33l2Ydl7OIMqGkAABlfSURBVK8ypf011utdKjETQvS65HnQ3e4A5o9Q9jbg2eSQvT7wWeD7VVU9s67rB5u6lnQK13WdqqpaOkpdK+tA8tQRTc6S8Yuoh31WpqS/Yox99vvKsw/L2F9lSvtrrNf7rBQRewH7pZSe09xOwHYppT9P83m3BS5PKU2oz5uvPH9sSmmv6WzXVJkJIfpO8kWA3dYHlvUWrOv6TuA3zc2/VlX1nqbc1sCfR6lrvZHqmgJHAkdPsKx/EFe0/jj77bMVjdVn9teKxnuOTVc9PhYr8rlbxr+PZabqta4CvcF8dTThEB0RC4Bbm6+cHKvcfGCHlNJFE6z6SiBVVbVdXdeXN9ueTp6KMZ7U/HTe4fwJeBrwDYCqqrYHHgSummBbJmzBggX3AfdNdb2ri/E+TrtyVTVkgIzdZ5esuoYMiKn6yNaPflfeWH3oa31F4z/nfL138zWqfim5sPBG4BmdGxGxRkRcGxFP7in3JGC0VTVWUNf1XcAZwKerqppfVdXTaC4Q7C1bVdVOVVVtV1XVGs2yeMcAV7P87/BJwN5VVT21qqr5wKeBb9V1ffdE2yNJkjRoIuKjEXF1RCyLiMsj4vVTVO/bmry3LCKuj4i9I2I74Fjg2RFxZ/MzLyLWjoivRcTiiLgK+Ptx6n5qRFzc1P1DYMOe/c+OiJ829V0eEf/YbN8pIm6PiEd0lX1eRNwWEXOn4n5PREmI7p3PEsDmTM3Fde8ljyjfDJwFHFzX9flVVW1WVdWdVVVt1pTbEvgheXrGn4FHAbs286Gp6/pccnA+u6lrGHj/FLRPkiRpJrsO+Dvy9JZDgVMj4nHjHRQRu0fEpaPsm0deKniXlNJ8YCfgNymly4H9gItTSus2P3eRV0V7CvBE4PnAHmOcdy7wPeBMcp77IrBn1/5NyXnu88BG5AHWr0XEdimlX5Ovk9ulq8o9gG+llIbHu89TZSbMiaau6zvIy9z1bl9Ivliwc/ubwDfHqetL5AdckiRptZBSOqPr5rci4kBy6L1pnOO+QTMNdhQPAdtHxMKU0i3ALWOU3Q34QErpVvjbhYLfHaXsc4F5wFEppYeAcyLiR1379wTOSyl1jv91RJxJzouHAaeRg/P3ImIO8CagGqNtU67v60RLkiRp5UTEWyPi9xFxR0TcQV72d6OVqbMZXX4T8C7g5og4OyK2H+OQBcDCrts3jFO23QTokcpvDrymc3+a+/RPwKbN/lOBXZtr8V4BLEkp/XIi92uqGKIlSZIGWEQ8Hvga8M/Ao1NKGwB/ZAqW80spnZNSejnwWPJ3bnSuWUsjFB8CNuu6vdkIZbrLtiKiO4t2l18InJ5S2qDrZ92U0rubdl3TtOcfgbeQR6ZXqdLpHB+OiM4wfueB+WhELOoqs8nKN0uSJEkTNK/5dxHkUWnySPRKiYhNyFNCfgzcA9xNXvUM8rSOVkSslVLqrFb2LeCgiPhVc/tfxqj+l019H4uIz5Pnc78S+Haz/1TgtxHxD+Tr5dYAdgCWNnOyO2X2bbY/fWXu62SUhOiFwI49224ARlofcOEI2yRJkjTFUkqXRcT/BX5OnsN8CvCLiRwbEXsAB6WUeldbgxxcPwx8nTzy/Efgnc2+88lffndzM5rcIs9V3oS8atpt5IsFn9d1rmOb9u6XUhqOiNeSR9A/AfykOc9aTZmbIuLVwGfIq69BHnnu/kbr04EvAL9LKU35csbjmXCITiltPo3tkCRJ0iSllD4OfHyUfSexPIjS/Q2CKaXTGGUqRErpZvII8Uj77gf+YYRd+zQ/Hcd0HbNfTx2/B541Uv3N/t8ALx1j/1+BR4y2f7o5J1qSJEkqVPKNhXOBdVJKS3u2Pxb4CLAdeW3mY5t3DpIkSdKsVDIn+mjg5eQFtAGIiEeTv3/0scDt5AW+94iI5zZD9JIkSdKsUzKd44XkCd/dPkwO0PumlDYiTyq/CjhwaponSZIkzTwlIXoz8lWY3V4DXJFSOgGg+Yaaz7PiKh6SJEnSrFEynWMueT0/ACJiA2Bb4NiectfiWtGSJGk1d9PeR630l51o5ioZib6G/D3nHa9o/v1xT7lHAYtXplGSJEnSTFYyEn0CcFREAPwF+CT522rO6in3YuDPU9I6SZIkaQYqCdFfBp4MHEye2rEQ2C2ldE+nQDPF463AkVPZSEmSpEFz5V5r/mQaqr19m5MefN001KtCJd9Y+CCwX0TsD8xLKd02QrE7ga2BpSPskyRJWp28aBrq/Ms01KlJKBmJBqAZeb5nlH0PAH9d2UZJkiRJM1nJNxb+Y0nFKaXvlDdHkiRJmvlKRqLPAFLz+3hLtiRgzUm1SJIkSZrhSkL0Q+R1os8EvoErcEiSJGk1VRKiW8Cbgd2B/0f+9sLTgG+mlG6ehrZJkiRJM9KEv2wlpXRLSumLKaWdgCcC3wP2BW6MiPMj4h3NEneSJElahSLi+oh45UrWsXlEpIhYe6raNZuVfGPh36SUrk4pHZZS2g7YEbgc+Ar5C1kkSZKkWW1SIRogItaIiFcAHwD2BJYAP52qhkmSJGl8EfF1YDPgzIi4MyIOj4itI+KsiLgtIq6JiPd0ld8xIi6OiKURcWtE/Eez66Lm39uaelZqZHu2K14nOiKeB+wGvAmYB9TkedJnN+tES5IkaRVJKe0ZES8E9kspnR0RjyTPEjgKqIAtgXMj4qqU0rnAF4F/Tyl9PSLmAU9tqnoRcB2wUUrp3lV/TwbLhEeiI+KIiLgWuADYAvgQsElKafeU0g8M0JIkSTPCrsBfUkpfSSkNp5SuAL5KHgQFGAa2ioiNU0p3pZR+2beWDrCSkegDgGXAfwO3ATsBO0WMuGR0Sil9YOWbJ0mSpEKbAztExB1d29Zk+bTbtwOfAi6LiIXAv6aUzly1TRx8JSF6IflLVJ47gbKJPFdakiRJ0y91/b4Q+EVKaecRC6Z0FbB7RKxBnu7x7YjYpKcOjWPCITqltPlEy0bE/Em1RpIkSZNxC/CE5vcfAEdFxNuBU4EHge2AtVNKF0fEW4AfpZQWRcRS8jdRPwAsIn+53hOA/13Vd2DQTHp1jpFExGMi4gjghqmsV5IkSWM6EvhYM4XjX4CXkUeZbySH4+OB9ZqyrwD+NyLuBP4D2D2ltCyldDfwr8AFEXFHswqbRlG0OkdEPAd4G3kZlavJV3Ze03wEcDCwN/AI4JtT3VBJkiSNLKX0PfIX4XV7zShl9xyjnkOAQ6awabPWhEN0ROwCfJ885L+I/A5n94jYE/g6sCE5PH86pXTlNLRVkiRJmhFKpnMcBPwWaKWUHgs8CjiHvE703cCOKaU9DdCSJEma7UpC9LbAkSmlvwCklO4kL3s3BzggpXTJNLRPkiRJmnFKQvSjgaGebZ3bV01NcyRJkqSZr3R1jtHWD3xwZRsiSZIkDYqi1TnIS548NML2n/ZsTyml9VeiXZIkSQNtm5MeHPFrnTU7lIToT01bKyRJkqQBUvKNhYZoSZIkifLpHJIkSZqAj+3/g/unodq/fPYLu242DfWqkCFakiRpeswdkDo1CaWrc0iSJEmrPUeiJ2l4eHhTYNN+t2NQDQ8PP6PfbRg09lmZkv6aO3fuqF8WZb+vPPuwjP1VprS/xnq9SyUM0ZP3LuCQfjdigP223w0YQPZZmZL+GmsZKvt95dmHZeyvMqX95bJzmhKG6Mk7DqgnWNY/iCt65jj77bMVjdVn9teKxnuOTVc9PhYr8rlbxr+PZabqtb7aiYjNgeuAdVJK90bEWcAZKaUTRii7NnAPsEVK6fpV2c6ZyhA9SXPnzr0ZuLnf7RhUfpxWzj4rM1X9Zb+vPPuwjP1Vxv6aOimlXfrdhkHihYWSJElSIUO0JEnSAIuIf4mIH/RsOygivhcRu0TEJRGxNCJujIhPR8SI88Ij4sKI2K/5fY2IODIiFkXEQmCPVXBXBoohWpIkabCdBrwsIjbq2rYHcCpwF7AXsAHwauCdwOsnUOfbgTcCOwFPBv5hCts7KxiiJUmSBlhK6SbgF8CbACLi6cDjgO+nlC5KKV2aUnoopXQp8E3g7yZQ7W7AF1NK16aUluGKZCswREuSJA2+U4Hdm9/3IK+ycW9E7BQRFzTTMpYA+wEbjVrLcguAhV23b5ja5g4+Q7QkSdLgOwN4RkRsSR5FPrXZ/g3gh8BmKaX1ga8wsbWyh4DNum5vNlrB1ZUhWpIkacCllJaQw/KxQAJ+0uyaDyxOKd0TEc9i+Wj1eL4F/HNEbBER84FDp7jJA88QLUmSNDt8HXgZ8I2U0kPNtvcAB0fEMnIQ/q8J1vU14Ezgf4DLyAFdXfyyFUmSpFkgpVTTM1UjpXQGearHSOWv7y6fUtq56/cHgY81Px0rfJPh6syRaEmSJKmQIVqSJEkqZIiWJEmSChmiJUmSpEKGaEmSJKmQIVqSJEkq5BJ3kiRJ0+CzX9h1It8MqAHlSLQkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVKhOf1uAEBVVRsAxwO7AEuBw+u6/vII5V4NHAA8Bbgf+DHwwbqub2n27wWcANzTddi76ro+bVrvgCRJklYrMyJEA8eQ27IA2Ao4r6qqy+u6vqCn3PrAUcCFQADHAieRw3fHxXVdP2e6GyxJkqTVV99DdFVV84A3AjvUdb0M+F1VVScB+wAPC9F1XX+j59h/J49GS5IkSatM30M0sA0QdV1f1rXt98CHJnDs3wF/6tn21KqqFgFLgP8GDq3r+p4VjlxJQ0NDawFrTXW9q4uhoaH1+t2GQWOflSnprwULFiydino0MvuwjP1VprS/xnq9SyVmQohelzwPutsdwPyxDqqqakfg48CruzZfBGwPXA9sDZwCfBZ4/xS1tduBwCHTUO/qYkm/GzCA7LMyJf0VU1SPRmYflrG/ypT211ivd2nCZkKIvhPofRe5PrBstAOqqnoqUAPvqOv6553tdV1f21XsiqqqDgC+yfSE6COBoydY1j+IK1p/nP322YrG6jP7a0XjPcemqx4fixX53C3j38cyU/Val4rMhBB9JZCqqtquruvLm21PZ8VpGgBUVbU9cA7w4bquzxin7oeYpnecCxYsuA+4bzrqXh2M93HalauqIQNk7D67ZNU1ZEBM1Ue2fvS78sbqQ1/rKxr/OefrvZuvUfVL39eJruv6LuAM4NNVVc2vquppwF7Aib1lq6p6MnAecOBIy9ZVVbVLVVWbNr9vSV7J48xpbL4kSZJWQzNhJBrgvcBXgZvJ86MPruv6/KqqNgMuA55U1/VC4CPAY4AvVVX1pc7BdV2v2/z6EuDEqqrmA7eRw/nBq+5uSJIkaXUwI0J0Xdd3kJe5692+kHzhYef23sDeY9TzUeCj09FGSZIkqaPv0zkkSZKkQWOIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKmSIliRJkgoZoiVJkqRChmhJkiSpkCFakiRJKjSn3w0YVMPDw5sCm/a7HYNqeHj4Gf1uw6Cxz8qU9NfcuXMvmYp6NDL7sIz9Vaa0v8Z6vUslDNGT9y7gkH43YoD9tt8NGED2WZmS/oopqkcjsw/L2F9lSvtrrNe7NGGG6Mk7DqgnWNY/iCt65jj77bMVjdVn9teKxnuOTVc9PhYr8rlbxr+PZabqtS4VMURP0ty5c28Gbu53OwaVH6eVs8/KTFV/2e8rzz4sY3+Vsb/UL15YKEmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVMgQLUmSJBUyREuSJEmFDNGSJElSIUO0JEmSVGhOvxsw1aqq2gA4HtgFWAocXtf1l/vbKkmSJM0ms3Ek+hjym4MFwK7Ap6uqenF/myRJkqTZZFaNRFdVNQ94I7BDXdfLgN9VVXUSsA9wQT/bJkmSpNljVoVoYBsg6rq+rGvb74EPTfWJhoaG1gLWmup6VxdDQ0Pr9bsNg8Y+K1PSXwsWLFg6FfVoZPZhGfurTGl/jfV6l0rMthC9LnkedLc7gPnTcK4DgUMmUvB/XvHP03D6gbdkrJ3rHnHjqmrHIBm1zz740WesynYMijGfYz1iiurx9T6yUfvQ1/qIxnzO+XpfQdFrlLFf79KEzbYQfSfQ+450fWDZNJzrSODoaah3Os0HbgIex/T0yWxjf5WbjX22fr8bMAmz8XGYbvZZGftLq73ZFqKvBFJVVdvVdX15s+3pwJ+m+kQLFiy4D7hvquudTkNDQ51fl/lx1vjsr3Kzsc8G8X7MxsdhutlnZewvaZatzlHX9V3AGeQVOeZXVfU0YC/gxL42TJIkSbPKrArRjfcCCbgZOAs4uK7r8/vbJEmSJM0ms206B3Vd30Fe5k6SJEmaFrNxJFqSJEmaVoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSpkiJYkSZIKGaIlSZKkQoZoSZIkqZAhWpIkSSoUKaV+t0GSJEkaKI5ES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7SkGSsiNoqIFBF7FRyzc0QcNI3NmkgbXhsR7+lnGyRJ08sQLWm22Rnoa4gGXgsYoiVpFjNES5IkSYUM0ZJmjIjYNyKuj4i7I+LHwFY9+98aET+LiNsjYnFEXBgRO3btPxQ4BJjXTANJEXFhs2/biDg9Im5s6r8sIj4cEWv0nOOAiLg6Iu6NiFsj4ryI2KJr/1oRcURE3BAR90XE5RGxe9f+k4C3AU/uasNJU95ZkqS+mtPvBkgSQETsChwPnAScDjyr+bfb5sApwDXAI4DdgYsi4qkppSuBrwGPa7a/pDlmafNvC7gCOA1YBjwd+BQwDzisacNbgU8DBwO/BNYHXgis19WGbwMvaI69HHgVcGpELE4pndUcvzGwLbBHc8yiSXWKJGnGipRSv9sgSUTEr4D7U0ov6tp2BHAgsHdK6aSe8muQP037E/CdlNJBzfZDgY+klNYd41wBrAl8DHhfSmlBs/0Y4LkppWeOctyLgfOBV6SUzuna/m1g85TSjs3tk4BnpZS2L+kDSdLgcDqHpL6LiDWBZwJn9uw6o6fcdhFxZkTcAjwIDANPBLaZwDnWjohPRcTVwH3NsYcDm0ZEJ3BfAuwQEUdHxAsiYm5PNS8HbgfOj4g5nR/gx81xa5bcb0nS4HI6h6SZYGPy36Nbe7bf0vklIuYD55CnRnwIuAG4lzyFY+0JnOMzwL7kaRi/Be4AXgN8ojn+TvJUkvnAO4H9gSURcTJwQErpHmAj4FHkAD6STYGbJtAWSdKAM0RLmgkWAQ8Aj+nZvknX788lz3feNaX0h87GiFifiQXXNwLHpZQ+03Xsq7sLpJQeAr4IfDEiWsCbgaOA28hznW9v2vqqUc7R+yZAkjRLOZ1DUt+llB4kT6V4Xc+uN3T9vk7z7/2dDRHxPPLFht3uB9Ya4TTr9By7Jjkkj9amdkrp88ClwHbN5vPIo+b3p5R+M8JPp/77mdjouCRpQDkSLWmmOBz4XkScyPLVOXbv2v8r8pSL/4iIo8irbRwKtHvquRyYExEfAH4BLE0pXQGcC+wbEZeRR5PfS0/YjojjgMXNuRYDzweeBnwZIKV0bkR8Hzg7Ij5LDtjzgCcDW6WU3tHVhn0iYjfgKuC2lNL1k+8aSdJM4+ockmaMiHgX8HHyaO+vyStz/IJmdY6IeCXwOeAJwJXAAcC/AHemlHZt6phDnpLxevL0kItSSjtHxCbAscBLgbvJ85+vBr4KbJxSui0i3kaeN70d8EjgWuDYlNKXutr4iOa8bwEeDywhrxByYkrp602Z9YDjgJcBjwZOTintNdX9JUnqH0O0JEmSVMg50ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSZIkFTJES5IkSYUM0ZIkSVIhQ7QkSZJUyBAtSfr/GwWjYBSMglFAIgAAxxTk2iqkBnoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<ggplot: (8771598392229)>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_toplot = pd.DataFrame({\"type\": ['white','red','white','red'],\n",
    "                            \"value\": [valid_white_perf.rmse(),\n",
    "                                     valid_red_perf.rmse(),\n",
    "                                     test_white_perf.rmse(),\n",
    "                                     test_red_perf.rmse()],\n",
    "                            \"dataset\": ['valid','valid','test','test'],})\n",
    "data_toplot = data_toplot.append(sdev_data)\n",
    "ggplot(data_toplot, aes(x='dataset', y='value', fill='dataset')) + \\\n",
    "    geom_bar(stat = 'identity') + \\\n",
    "    scale_fill_brewer(type = 'qual', palette = 2) + \\\n",
    "    facet_wrap('~type', ncol = 2) + theme_minimal() + \\\n",
    "    labs(title = 'Model Performance by Segments', y = 'RMSE') + \\\n",
    "    theme_minimal() + \\\n",
    "    theme(legend_position = 'right',\n",
    "          axis_text_x = element_blank())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}