examples/text/text_regression_example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\";\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\";"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using Keras version: 2.2.4-tf\n"
     ]
    }
   ],
   "source": [
    "import ktrain\n",
    "from ktrain import text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Wine Prices from Textual Descriptions\n",
    "\n",
    "This notebook shows an example of **text regression** in *ktrain*.  Given a textual description of a wine, we will attempt to predict its price.  The data is available from FloydHub [here](https://www.floydhub.com/floydhub/datasets/wine-reviews/1/wine_data.csv).\n",
    "\n",
    "## Clean and Prepare the Data\n",
    "\n",
    "We will simply perform the same data preparation as performed by the [original FloydHub example notebook](https://github.com/floydhub/regression-template) that inspired this exmaple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>country</th>\n",
       "      <th>description</th>\n",
       "      <th>designation</th>\n",
       "      <th>points</th>\n",
       "      <th>price</th>\n",
       "      <th>province</th>\n",
       "      <th>region_1</th>\n",
       "      <th>region_2</th>\n",
       "      <th>variety</th>\n",
       "      <th>winery</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8486</th>\n",
       "      <td>8486</td>\n",
       "      <td>Italy</td>\n",
       "      <td>Made entirely from Nero d'Avola, this opens wi...</td>\n",
       "      <td>Violino</td>\n",
       "      <td>89</td>\n",
       "      <td>20.0</td>\n",
       "      <td>Sicily &amp; Sardinia</td>\n",
       "      <td>Vittoria</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Nero d'Avola</td>\n",
       "      <td>Paolo Calì</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148584</th>\n",
       "      <td>148585</td>\n",
       "      <td>Portugal</td>\n",
       "      <td>Warre's seems to have found just the right for...</td>\n",
       "      <td>Otima 20-year old tawny</td>\n",
       "      <td>90</td>\n",
       "      <td>42.0</td>\n",
       "      <td>Port</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Port</td>\n",
       "      <td>Warre's</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18353</th>\n",
       "      <td>18353</td>\n",
       "      <td>Italy</td>\n",
       "      <td>A more evolved and sophisticated expression of...</td>\n",
       "      <td>Campogrande</td>\n",
       "      <td>87</td>\n",
       "      <td>23.0</td>\n",
       "      <td>Veneto</td>\n",
       "      <td>Soave Superiore</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Garganega</td>\n",
       "      <td>Sandro de Bruno</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5281</th>\n",
       "      <td>5281</td>\n",
       "      <td>Spain</td>\n",
       "      <td>Red-fruit and citrus aromas create an astringe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>84</td>\n",
       "      <td>12.0</td>\n",
       "      <td>Northern Spain</td>\n",
       "      <td>Ribera del Duero</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tempranillo</td>\n",
       "      <td>Condado de Oriza</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87768</th>\n",
       "      <td>87768</td>\n",
       "      <td>US</td>\n",
       "      <td>Lightly funky and showing definite signs of ea...</td>\n",
       "      <td>Lia's Vineyard</td>\n",
       "      <td>89</td>\n",
       "      <td>35.0</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>Chehalem Mountains</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Pinot Noir</td>\n",
       "      <td>Seven of Hearts</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Unnamed: 0   country  \\\n",
       "8486          8486     Italy   \n",
       "148584      148585  Portugal   \n",
       "18353        18353     Italy   \n",
       "5281          5281     Spain   \n",
       "87768        87768        US   \n",
       "\n",
       "                                              description  \\\n",
       "8486    Made entirely from Nero d'Avola, this opens wi...   \n",
       "148584  Warre's seems to have found just the right for...   \n",
       "18353   A more evolved and sophisticated expression of...   \n",
       "5281    Red-fruit and citrus aromas create an astringe...   \n",
       "87768   Lightly funky and showing definite signs of ea...   \n",
       "\n",
       "                    designation  points  price           province  \\\n",
       "8486                    Violino      89   20.0  Sicily & Sardinia   \n",
       "148584  Otima 20-year old tawny      90   42.0               Port   \n",
       "18353               Campogrande      87   23.0             Veneto   \n",
       "5281                        NaN      84   12.0     Northern Spain   \n",
       "87768            Lia's Vineyard      89   35.0             Oregon   \n",
       "\n",
       "                  region_1           region_2       variety            winery  \n",
       "8486              Vittoria                NaN  Nero d'Avola        Paolo Calì  \n",
       "148584                 NaN                NaN          Port           Warre's  \n",
       "18353      Soave Superiore                NaN     Garganega   Sandro de Bruno  \n",
       "5281      Ribera del Duero                NaN   Tempranillo  Condado de Oriza  \n",
       "87768   Chehalem Mountains  Willamette Valley    Pinot Noir   Seven of Hearts  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "path = 'data/wine/wine_data.csv'  # ADD path/to/dataset\n",
    "data = pd.read_csv(path)\n",
    "data = data.sample(frac=1., random_state=0)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: 95646\n",
      "Test size: 23912\n"
     ]
    }
   ],
   "source": [
    "# this code was taken directly from FloydHub's regression template for\n",
    "# wine price prediction: https://github.com/floydhub/regression-template\n",
    "\n",
    "# Clean it from null values\n",
    "data = data[pd.notnull(data['country'])]\n",
    "data = data[pd.notnull(data['price'])]\n",
    "data = data.drop(data.columns[0], axis=1) \n",
    "variety_threshold = 500 # Anything that occurs less than this will be removed.\n",
    "value_counts = data['variety'].value_counts()\n",
    "to_remove = value_counts[value_counts <= variety_threshold].index\n",
    "data.replace(to_remove, np.nan, inplace=True)\n",
    "data = data[pd.notnull(data['variety'])]\n",
    "\n",
    "# Split data into train and test\n",
    "train_size = int(len(data) * .8)\n",
    "print (\"Train size: %d\" % train_size)\n",
    "print (\"Test size: %d\" % (len(data) - train_size))\n",
    "\n",
    "# Train features\n",
    "description_train = data['description'][:train_size]\n",
    "variety_train = data['variety'][:train_size]\n",
    "\n",
    "# Train labels\n",
    "labels_train = data['price'][:train_size]\n",
    "\n",
    "# Test features\n",
    "description_test = data['description'][train_size:]\n",
    "variety_test = data['variety'][train_size:]\n",
    "\n",
    "# Test labels\n",
    "labels_test = data['price'][train_size:]\n",
    "\n",
    "x_train = description_train.values\n",
    "y_train = labels_train.values\n",
    "x_test = description_test.values\n",
    "y_test = labels_test.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 1: Preprocess the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "task: text regression (supply class_names argument if this is supposed to be classification task)\n",
      "language: en\n",
      "Word Counts: 30953\n",
      "Nrows: 95646\n",
      "95646 train sequences\n",
      "train sequence lengths:\n",
      "\tmean : 41\n",
      "\t95percentile : 62\n",
      "\t99percentile : 74\n",
      "Adding 3-gram features\n",
      "max_features changed to 1769319 with addition of ngrams\n",
      "Average train sequence length with ngrams: 120\n",
      "train (w/ngrams) sequence lengths:\n",
      "\tmean : 121\n",
      "\t95percentile : 183\n",
      "\t99percentile : 219\n",
      "x_train shape: (95646,200)\n",
      "y_train shape: 95646\n",
      "23912 test sequences\n",
      "test sequence lengths:\n",
      "\tmean : 41\n",
      "\t95percentile : 62\n",
      "\t99percentile : 73\n",
      "Average test sequence length with ngrams: 111\n",
      "test (w/ngrams) sequence lengths:\n",
      "\tmean : 112\n",
      "\t95percentile : 172\n",
      "\t99percentile : 207\n",
      "x_test shape: (23912,200)\n",
      "y_test shape: 23912\n"
     ]
    }
   ],
   "source": [
    "trn, val, preproc = text.texts_from_array(x_train=x_train, y_train=y_train,\n",
    "                                          x_test=x_test, y_test=y_test,\n",
    "                                          ngram_range=3, \n",
    "                                          maxlen=200, \n",
    "                                          max_features=35000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 2: Create a Text Regression Model and Wrap in Learner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fasttext: a fastText-like model [http://arxiv.org/pdf/1607.01759.pdf]\n",
      "linreg: linear text regression using a trainable Embedding layer\n",
      "bigru: Bidirectional GRU with pretrained word vectors [https://arxiv.org/abs/1712.09405]\n",
      "standard_gru: simple 2-layer GRU with randomly initialized embeddings\n",
      "bert: Bidirectional Encoder Representations from Transformers (BERT) [https://arxiv.org/abs/1810.04805]\n",
      "distilbert: distilled, smaller, and faster BERT from Hugging Face [https://arxiv.org/abs/1910.01108]\n"
     ]
    }
   ],
   "source": [
    "text.print_text_regression_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "maxlen is 200\n",
      "done.\n"
     ]
    }
   ],
   "source": [
    "model = text.text_regression_model('linreg', train_data=trn, preproc=preproc)\n",
    "learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lower the `batch size` above if you run out of GPU memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 3: Estimate the LR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "simulating training for different learning rates... this may take a few moments...\n",
      "Train on 95646 samples\n",
      "Epoch 1/1024\n",
      "95646/95646 [==============================] - 8s 81us/sample - loss: 2627.6407 - mae: 34.2769\n",
      "Epoch 2/1024\n",
      "95646/95646 [==============================] - 7s 70us/sample - loss: 2610.0313 - mae: 34.0299\n",
      "Epoch 3/1024\n",
      "95646/95646 [==============================] - 7s 70us/sample - loss: 2148.5174 - mae: 26.8848\n",
      "Epoch 4/1024\n",
      "95646/95646 [==============================] - 7s 71us/sample - loss: 1158.6146 - mae: 15.1160\n",
      "Epoch 5/1024\n",
      "15360/95646 [===>..........................] - ETA: 5s - loss: 4022.5116 - mae: 36.6476\n",
      "\n",
      "done.\n",
      "Please invoke the Learner.lr_plot() method to visually inspect the loss plot to help identify the maximal learning rate associated with falling loss.\n"
     ]
    }
   ],
   "source": [
    "learner.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEKCAYAAADXdbjqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dd3gc5bW437Or3otlWZbcuw3GGNNLIIAxLZByAyQBkpAQEnLTbpJL2oUQIPxyE0KANDo3QBKSAKH3XoyxjXHFvVuyLBf1ttrv98fM7M6udlcraVdaSed9nn088007O1rPmfOdJsYYFEVRFKU/eAZbAEVRFGXoo8pEURRF6TeqTBRFUZR+o8pEURRF6TeqTBRFUZR+o8pEURRF6Tdpgy1AMhg1apSZOHHiYIuhKIoypFi2bFmdMaasL8cOS2UyceJEli5dOthiKIqiDClEZHtfj9VpLkVRFKXfqDJRFEVR+o0qE0VRFKXfqDJRFEVR+o0qE0VRFKXfqDJRFEVR+o0qE0VRlGHCsu0HWLL1wKBcW5VJGM+vqeHBxX0OtVYURRk0bn1pIzc9s25Qrj0skxb7w9f+sgyAzx87HhEZZGkURVHip62zi+x076BcWy2TKHR0+QdbBEVRlF7R2tlFToYqk5Sis0vbGSuKMrRo7egiS5VJatHpU8tEUZShRVunX6e5Uo1OneZSFGWI0dLhU2WSaqjPRFGUoUZrZxfZOs2VWqjPRFGUoYTfb2jr9JOllklqodNciqIMJdptP69Oc6UYHeqAVxRlCNHa2QVAdvrgPNZVmURBLRNFUYYSAWWiPpPUQn0miqIMJVo7LGWiPpMUQy0TRVGGEm22ZZKTMThVspKmTEQkS0SWiMiHIrJGRH5uj08SkfdEZJOI/F1EMuzxTHt9k719outcP7LH14vIWcmSGSDDa90SDQ1WFGUoEfSZDD/LpB34uDHmCGAesEhEjgP+H/BbY8xU4CBwhb3/FcBBe/y39n6IyGzgYmAOsAj4g4gk7W6lea3ijpoBryjKUKKlw/GZDDMHvLFoslfT7Y8BPg780x5/ALjQXr7AXsfefrpYZXsvAP5mjGk3xmwFNgHHJEtur8dWJuozURRlCDGsfSYi4hWRFUAt8CKwGThkjPHZu+wCKu3lSmAngL29Hih1j0c4JuGk29Nc6jNRFGUo0TaMp7kwxnQZY+YBVVjWxMxkXUtErhSRpSKydN++fX0+T5ptmYwEn8ndb27h3yt2D7YYiqIkgBERGmyMOQS8ChwPFImIE25QBThPs93AOAB7eyGw3z0e4Rj3Ne40xiwwxiwoKyvrs6yOMlmx81CfzzFUuOHpdXz7bysGWwxFURKAM8017CwTESkTkSJ7ORs4E1iHpVQ+Y+92OfBve/kJex17+yvGGGOPX2xHe00CpgFLkiV3pv2HePi9Hcm6RMph3WZFUYYyjmUyWD6TZAYkVwAP2JFXHuARY8xTIrIW+JuI3AB8ANxj738P8BcR2QQcwIrgwhizRkQeAdYCPuBqY0xXsoTOTBt5qTf7GtsZXZA12GIoitIPWju6EBm8Z1jSlIkxZiVwZITxLUSIxjLGtAH/EeVcNwI3JlrGSDihwSOJjbVNA6JM2jq7mPmz57j2/Nl86cRJSb+eoowkmtp95GWmYQXBDjwj7zW8B0bSjE+u7ajbsLdxQK7340dXAfDzJ9cOyPUUZSTRbCuTwUKVSRj+EaRMSvMyAcsyGQje3bI/sNzuS9pMpaKMSJrafeSqMkkd3M7o4V6G3mB91017B0aZVNe3BZZrG9qTdp13NtUx8Zqn2ThAFpeipAJNapmkLk6o3XDFb+vKddUNdA2wSVbT0NbzTn3kjY11ADy3uiZp11CUVEOVSYrhd1kmLZ2+GHumBu2+rn6F9uZmeGls9/H6htoEStUzbisl0VQWWcEE1UlUWIqSaqjPJMVwP5dbUtwy6fD5mfHT57j52Y/6dLzfGM6aM4bcDC9PflidYOm6M3lULh+bbiWU3vT0uqTltxRkpwOwvyl5U2mKkmo0t3epzySV8BsTKEOfatNcbZ1dvLO5LrDulHz58xtb+nQ+Y6xQ6OaOLh77YDcvrd2bEDmj0dHlpyQ3A7CmuTYl2fG/v6kjqedXlFSisa2TvMzBSVgEVSbdMECO/QdJNcvk5mc/4nN3vcfaPQ0AdLkqG/flLd9g8Lhi0pfvONh/IaPwlQeWsutgK+muPJ4dB1qSci1nqrJOLRNlhGCMobmji7wstUxSBwO5dqeylo7U8pnstB++uw5a//r8wWizvkx1+Q2IwFlzygH4w2ubA5VHE81L6/YGrnnrRfMA2JukiC7ntuxrVGWijAzaOv10+Y1Oc6USfmPItS2TVJvmKrR9Adv2NzP/Fy/y1yXB+mF9meoyBkSEP1+6IDD29/d3xjgCvv7gMv73+b75aAA27m3knMMrAKhtTI6D3LFMmlPs76coyaKp3XrxVQd8CmEI9lBOtWmusgIryfD1Dfs40NzBr1/YELK9ppcRUsYYwgsvxPrOfr/h2dU1/P7Vzb26jpuahjYy0jyU5mZQmyTLwT3j90gPylFRhgPOLMpg9X8HVSbdMIaAZdKSpCmfvlJVlA3A25v2h4zbVfO56sFlvTqfgYDP5FzbWvDH8L10ubbVt3b26lqTRuUCUJxjOeDL8jOTlrjolvPJlXuSco1wDjR38N2/r6CxrXf3RVESgfMSmDNIvUxAlUk3/MYEfCatKeYzifaYd7pD9rYHi98YHP/7rRdbfoyGGA9Dd2LjtrrmXl0rzSNkpHm470tHAzC6IIt9SZ7mAnhzY92AJGQ+8M42HvtgN/e8tTXp11KUcIL931WZpAzGBLV7qk1z+aM8FK85O9jAcmcvIqSMCVom6V4PFYVZ1DVGD6d1tzLeebB3kVhdfsPC2eVUFFrW1ZiCTPYkKXEx/Db97qUNkXdMIGX51hRkb6caFSUROIEzOYPUywRUmXTDGEOa10O6V7j1pY0pFREU7f26ojCLP33hKIBeteENn9IalZcZM5zW/YZf14v78vqGfew61IrXE/TQTCjNZV9jO83tibf+nDDp6y+YA8C6muTX6HKUSTIz+xUlGsFpLvWZpAwGEKDTzuH473+tHNDr//n1zUy85umIUzPuoVsvmhdoMez1eALhvb9+YUP80zp2aLBDWX4me2OUIPG5znvdk2vZeaCFK+5/n18+Y2Wzt3T4IhbHvPzeJXT4/HhdF5tYavlQtu3v3XRZPDgW3Plzx3J4ZSEHmpOfvOhYeP21TFbvrmfiNU/zUU1DIsRSRgiOA16nuVII99QPEPPhmgzuetMK8Y1kETlv3F89eRLnzq3guMmlgGXiuhviXPvE6rhyZNwOeIDp5flsqm0KTGfd/eYWJl7zNL9+fj1ANyV1/h1v8fJHtfz5jS1s2NvEvJ+/yAW/fzvq9dyWyayKfAA+3Fnfo5y9xRHTI8LxU0pZtv0g25OgtNw4f5v+JmKed/tbAHz7ryv6LZMycmhVB3zq4XZKQ2yHdDIotzse7qlv7bbNmZX69hnTSfd6mDnGeiDvPmTte/OnDgfgwcU7OPe2t3q8lj8sNHhcSTY+v2Gr7Vy/4el1ANzx6iaMMSGWCcChluC9Wbr9AB1dftZVN0S9Z25lMmlULplpnuRYJvaNEg+cfdgYAD5K8lSXc2taO7uiKvJDLT1bSONLcgAozk1PmGzK8CfggFefSepgsKZ+zp1rhco2tA5sRNcYW5lUH+puEQUekvb61z42hY9NL+MzR1UBsMh+cAJsrWsOcZhHwhjwuB7wo/Otay/87Ru8s6mOSjsUGSxLySnf8v2F00PO4/UID78XTKB019xyBw24ryUilBdkJcXyc+6TR4Spo/MA2Lwv2T1bgt9z+/7u1sn2/c3Mu/5F7ns7drTXefbvriw/+W2UleFDa6dGc6UcTlb4/5w3G4CTpo4a0OuXF9rKJJJlYv/rTE2V5WfywJePYZTdMbHIzuFweGHNXv7w2qao1wq3TBwnMsDn7n6P5g5fIIfllY9q6bTrlIwryQnkpQCcOr2MNXuCc/yf+sM7gWV3zocvTLmNKchKaPSTM9Xk6C+vCPlZ6YzOz2TLvuROc7mNtkhh07sPWn/Ph1xKN9Z5arV8vtILWuz/q5lpg/dIV2UShpMVXl6QxYzy/JD6VwNBvl0OYU8syyQ8bT0KVz+8nF89tz5qvS3LCguezLGKHA61dLJwtmXtXPPoKny2ZeL1CJXFQatlTmVhVBncfpbwWlyjCzJ7nQW/Zk99xKm0L9//PnN//gLQ/T5NGpUbmLpLFu7AuK0Rpu4ceXtK9nRkT1Z1AGV40tLRRU5GWsj/54Fm8OLIUhRnmgsgLystUPOmr/i6/BiCiYXxXB9gz6HoPpPe/l7W1zRyxLiiCOcL9Q+NKczirssWMH98EUff+BJ+E6wHBsFaWmkeT8gU2GQ7ux1g5ph8ml0+A7efJfw7leRmBKbj4r0/bl/QtpvPDSy/8lFt4Ds598mx4CqLs1m8ObRqQKJxh1lHskwa2qx7crCHyDJnWnCgAz+UoU1rR9egTnGBWibdMCZYlj0vM42mtp6VydMrqzni5y/Q4fNT39IZEol13u1vccYtr8d9fedhEnGay+ULiMYk14Pd4a1NdRH2tKf0wsbOnF1OaV4mk8ssX4P7B+q83ad5hIuOHhcYH2c7jQGmlOWx51BbYErLXSZ/d5gyca79+vp9Ub9PLMKnzZxr/K8dfeZM0VUVZVPT0NajD6k/ON+yJDeDbXVBn8lf3t3GrS9tCFzb5zcxQ7edTS0dXf1+kVFGDpZlosokpfC7HrD5WWk0xqFMfvnsOupbO/nWXz/gpF+9wtE3vhTY9lFNYzeH7Ic7D0WNeHIeJpGyw/1hb9yR+NuVx/GnL8wPGYtWZsVvTNRzOUqpLD+TX9pRYs5D0usVslxRI+NDlEkuXX4TSN5z+0zcfhaAS4+fCMD+5vimdN4Nsy4iRWgt3nIgsOx8t6riHPzG6nWfLBxFP3lUbmCaq93Xxc/+vYZbX9pIW2dQkdXEsDrcFo5aJ0q8tHR0DWokF6gy6YY19WM9hPKz0mh0vR3ua2zntpc3ditrUpBlTQU9t6YmqvJxMr2NMVzw+7e5+M+LI+7nbuwUngAYHs0VifKCLBYdVhGIRgPYsDdyWKx7Si+cXPstJy8zjU8cMRYgkKvhJEseNaEYgFF5Qcf/zIoCIJjs6ficrjl7ZkApOThKqKa+HWNMj5bDJXeF3rOv/SVY2NJxPH7/Hx8GxpzvdvwUKx9n5a7E57Q4ODpgWnk++xrbmXjN09z+cjD4YZer/MyKHdFrqLmbnCWrEKYy/GjrVMsk5TCurPDwaa6fPr6KW17cwOKtoW/IY4u6h3GGT1HssqN5nCmOtVHekh2FYUx3JdAbn8nHZ4wOLO880BLRCe9ErkXi8hMmkp3u5fRZo8nNTGNUXkYgvNbJF/n7lcex4YazQ87hKBgnPNj5vkXZ6aSF+UWcUvR7G9v4zt9XMPt/novZMXJcSXbIenlBMPrMaQfsxpFrbFE2aR4J/A16yy0vbuCWF9bH3Mf5uy2wvz9Y+TkOO1zW6dUPL49xnuDya+treyuqMkJp6fANaikVUGXSDauciuMzSae1syswN+/8R3fPiQPMtt/G3by+fh8/fXxVYN1piet2SE+85umQN1brGsHtTnfCgGyBKKWetUm5KzLLb4IO6m7ninL8keOLWfeLRVQVW9bDpFG5bLMfiI6zPM3rISMsFDE/K42Tpo6isjib6vrWgKXgTlh0M7ogi731bfx7xR46u0zUvJ4Pdhyk3Z4qSvMIJ0wpZfmOQ9S3dvKl+5Z0q4l1zKSSwLLXI1QUZXXz2cTLbS9v5LZXoodYQ1DRHzEucmRb+MtDNCvMbwyluRnMH1/En9/YklQ/jzJ8aFEHfOphOeCtZaefsmNlONNZP35sVcgxXYGHfHBs2/5mHlwczClw8gzCs8h/9vjqkHW/sQouHlZZwOItoRaQVf4kvu9x/JRSvn7qFO653Oqi+I2Hgm/Dm2obA8ollv/FjVPtFyIrhqtPm8LsigKy072U5mXwwY5D3Pf2tkDvlTRv5OuMKcgM8SFE85988g/vBMJlf/PZI/jY9DIAfvviBl61HfhXnjI5MOX25RMnhRxfVZTD7l5WOg4nVtFP5yUgM83L3ZctCNlWlJPeTdkt334wynms5M4CO4pu2k+e7Y/IygihtVN9JimH3zXNlW8rE8cPMrmse6QUWAoiIy00XHZ9mHPYsUDc0U3WeOjbsqPMjplovXn7/YYVOw9x1xtbYjrMw/F6hP9eNJOPzxxtvZnbyZCPvL+TM255gyseWArEH2Y8pjBo6aRFUCY/OGsmT3/rJEQkUNDRHQablRb5hz6mMCtkOi9Sfk14bsYF8ypZOMfKf7n/nW2B8ca2Tn56rpVsGj4lVlmc3WfLxCFWrxL3X/WM2eV854xpgfUqOyfH6xFe+/6pQHdLMXAe++/vLoqpDbeUntBorhTEEHTAF9lvh07VWfcfyx2W6vcb0jzC4a7kvSc+DO3w9+TKatZVNwQc0sdOKuGMWaO7PSj9fstamDgqhw6fn7qmdi6/dwk3PrOOQy2dvc4xERGuPGUy1fVtdPkNt7+6MWR7vJaOO6Ex2pSVc98+d+x4AP6xbFdg22kzR0c8ZnR+VqBCM8Da6u5O8keX7+o25o4gCxy7p4ELj6zkg5+dyZyxodNNlUXZ1DZ2D2qIhwx7Wi9msUgn0s6+N5fbkWrOtcFSwhNH5TKhNIe/L43cTth5Yfieq2TNAy6FqSiR0DyTFMTtgB9faj2wnEqw7hmqhjYfdU3t/HPZLnx+g1ckJN8inC6/4ezfvRkobHjBvErmjSuiNqynh99+M3Ue3sfc9HJAiW3Z19ynDFfnYfbSur3dLJt4zxdqmcT+2cytCk2Q/Nopk0NCiaOdF2DV7u6BCZHyMrwe4fnvnBIydsXJkwEojuCMryzOxpjI+Ts9YWxN8ezqmqgdJsMj7YpzMzj7sDFcdvwEKous30W7rci272/hUEsnD723vdt389tVq+eMLeSp/zwJ6H8lYmV447R/UMskxbAS+ezaV3bNq/12wyh3SHB9ayffe+RDvv+PD7nv7W14vRKIZArnbFcBxl8/b3X9c95SIbSnR5cdmjzWNWU22lYs727ZHzMsOBrn2Pkd66obmDUmNFgg3lo+5XFYJg4ZaZ6A72JWRQE/OmdWjPNmhqw/+eEequtbec/2F9U2tvH6hmBS4yxXsMOMMfk8+c2TuO9LR7Pt5nMDIcyRqLLv5/19eMt3P+/fjJYAav/rVtZ//MJRXH/BYSGlZwB+Yt+Pnzy2mvve3srq3UFrzF21+rDKQo6ZVJL0umLK0Kbd58dvBrcxFqgy6YYh6IB3Sok4jl93Al5DaycHXM5irwgnTxvFzDH53G/3OQe4+7IFXH3aVAps/8u79kPS65FggyhXdJgx1jb3G3uGy3nd3odpmpLcDCaW5vDEij0caO5g0qhcvn7qFMCK2oqHCpc86VGc6W7u/aJ1D6762OSY+zlv7WCFYgMc/8tXuOjOxTS1+7jsniW8ubGOrHQPS35yOg995diQ4w+vKuS0GZGn0ML3g8jVmHvCbwzfPG0quRle1u6JHdIdydBz5+EAfHxWUN4bnl7Hebe/xSPvW9Ne4f10ppTlDkDF477z2vpazrv9zYjVCJSBoTUFys+DKpNuuB3wTl7EH17bzNJtB0LCdutbO5lilxwBSwHkZKTx3HdO4dQZo/noF4vYfNM5nDG7nMMqC1l53Vkh10nzRrZMnGmuElcF4NURpn56y7TyfLbUNbNk2wFKcjP470Uz+eBnZ0a1psJxVxTuyTIBa6rrg5+dGdNagFDfxw8XzQjZ9tr62kCWe1unn9H5WRHzSeIhPyuduVWFtEQpehkLp1T/lNF5/HXJDh56b3vEfSCyMnEi4b53puUHcQdqOPzQTvL0u6IJwSpPc7Cls8eaXoPFjx5dxerdDezVwpSDhvObHrbTXCIyTkReFZG1IrJGRL5tj18nIrtFZIX9Ocd1zI9EZJOIrBeRs1zji+yxTSJyTbJkBqdqcPcnwoqdh0KmuXYebGFccfBBGD6rn5Xu7fbQPXFqaWDZ6xHyMtMoy88MKfPhzJl7PMIvLjwMsML++ltauso11eKIFcm3EI10r4eTp1nl+J2S9z1RnJvRo0/G7TR030+Abz78QdzyxcPYwuxehwcH66ERCLD43Usbo+4X6bdzzKQS/vrV47j6tKkAUf1HEPz7OzgRhNGSXAcbx3qvb9GIs8EiYJkMV2UC+ID/MsbMBo4DrhaR2fa23xpj5tmfZwDsbRcDc4BFwB9ExCsiXuD3wNnAbOAS13kSTrRcjn1N7TiWfHa6lw01jSEJZfH0Gb/z0mD+gdPH49TpZTy1spqr/rKM6vrWkDnzS4+bEGju5EyJ9RV3NNb72yLnOPTEX644li03nRPzYdgXnDf1jDQPv/zU4fzknFmBLpKJZFZFAZv3NfcqOspdD+0HZ1mWk9sidQj6TCKf5/gppSEvF7/6zNxuIdbtvq5unT4PsxXY5+9+L26ZBxInH6an0vpK8gi27B2mPhNjTLUxZrm93AisAypjHHIB8DdjTLsxZiuwCTjG/mwyxmwxxnQAf7P3TZLchMxVXHrcBMB6+DvTXGle4YF3t/PnN7YE9otVCdYhNzONI+y5e6eQ4qfmW10Sn1tTw5m3vEFXV2guyWFjLYdzf3MNTnfN03/D9pf0BU+8scS9wJlqy0zzcMkx4/nqKZMD3SMBTp42KhDZ1B+crPhrn1gTs2yLG7/LMinKyeATR4xl16Hu1o1jtcYbHffZBeO44qTQxMrHP9gdUrUagt0vgah9acAq7//YB91DqJNNoSqTQcdpEz3Y01wDospEZCJwJPAecCLwTRG5DFiKZb0cxFI07kp+uwgqn51h46Fe2AQRqcTILy48jHXVDextaGN8SQ4iRCzm6CiJnvj3N0/iUEtHoCuiuxRLU7uPLXVNIQ+TT8wby+Mr9kSsItwbpo7OZ8tN5yRFGfSXmz51OMdMKgkJBjh5WhnW+4flzI+330ksppcHLYo3N9Zxip1FH4ugY926b+NKsnlmVTW+Ln9IrbGeLJNIHD2xJOSF5L//tYqq4mxyo7xh1tS3Bfxs4Zxw8ysAzBtXHLENQbJwlEmDKpNB42CLNSuSmzlMLRMHEckD/gV8xxjTAPwRmALMA6qB3yToOleKyFIRWbpvX9/6Y4Q3VXIozs3gUEsnXXY+yVfC3ihXXreQh756XNzXcbfXLcxJD9m2YW9TiBUyuyKopO743JF88YSJcV8nnFRUJGBFcX3huAkh00DuB38iFAlAqcvXs8oVjvvqR7Ws2RO5onD4b2LSqDx8fsMbG0N/Y45hGslnEo0zZpd3G9t1sLWbE/+v9m8rUsO0cKJ9j2ThlBhSy2TwWLunARGYVZH4qeHekFRlIiLpWIrkIWPMowDGmL3GmC5jjB+4C2saC2A3MM51eJU9Fm08BGPMncaYBcaYBWVlPb9xRiJaeGdxTjqHWjoDdZO+f1Yw6qiqOJuCrPRAWGtf+Mk5s7j+gjmB9QMtQf+LO0T4vLljue4TcxgJiAh3X7aA685PrHvsSydOBOBfrqz6L93/Pufe9lbEHjPuaS6wKhcAfPn+pSH7BazaXv6P+t6Z07n8+AkhY+F9Whyf0q44lMnm2oHNSXGmVg62pGa02Uhgf3MHxTkZZEYpWTRQJDOaS4B7gHXGmFtc4+4OSZ8EnEqHTwAXi0imiEwCpgFLgPeBaSIySUQysJz0TyRD5mhTFUU5GdQ0tPGn1zfT4fOTle4NVMv94aKZ/b7uV0+ZHPDNABwzqTRk+7KfnsHyn53Z7+sMNc6YXc4Xwwo29pdrz5/Dd8+Yzta6Zlo7ukLyI+Ze90I3X0p4Q7JxJTnk2y8O7mMDocG9lOdbp0/j5xccFnOfMYVZiMRnmfz2pQ28szlyYmUycKoDHNRorkHjQHNHn0PmE0kyLZMTgUuBj4eFAf9KRFaJyErgNOC7AMaYNcAjwFrgOeBq24LxAd8EnseaRH/E3jfhhM+PO7j7oDtMtEutJCpRSET451XHM2lULjdeGPpwKc3LTIkfy3BhyuhcjLHaEIdbAbVh+RKRrNXPLLCCA9y1x5yHaryFOMP54+fnB0Kuw0PKM9I8FGan89zqmqjHO0mxAJ+7a+AivxwlWt+qlslgsb+5IyQvbbBImsfGGPMWkV/UnolxzI3AjRHGn4l1XKKIFuBTlNNdmRw2tpANe5sYnR9fzkU8LJhYwqt2VVkleTgO6nNue5Pz5oa2Et6yrzmkdEwwGTH4U/726dO47+1tbK1rZsXOQ2SmeYI+kz66pc4+vIJJZbl8+b73uePz87ttz81I46OaRstvF8H3FU80YTJwLnuwWS2TweJgc0fEcPWBRjPgI9DNAR9B6//0vNn85j+OYG6cUVxK6uDOun9qZTUA/7jqeAC21IWWLjFhPhOwpj2nlOWyfX8zl9y5mLN/9yY3P/sR0DsHfDgzxxTwzo9OZ36EEjdX2eHc0Yo++vyGC+cFqw0MlEPcuT/vbtkfyJ1SBpaGtk4Ksgc3kgtUmYQQzQFf5JrmcraV5Gbw6aOq+lTFVxlc8rPSuzn254wtICvdw9awoorhPhOHiaW5bN/fQmtY7keyfg7T7OTVnQda2H2olbqm0Ok4n99QWZzNnZceBVhTeAOBu8TQA+9uG5BrKqG0dnQNesIiqDIJIZoT1R2++9g3Thw4gZSk8cUTJwVyg+aPLyInI42Jpbk89sHukDLz4dFcbsL9LZA8ZeK0N7jv7a2cePMrLLjhpcA2Y4w9/eVhsj3dsWWAikO6Z9diJVUqyaOt009m+uA/ygdfghQiUhlxgNLcoF8knoq5ytDg2MlW1NzyHYcAq5jl/uYOzr/jrcA+0YIy5kcpkNlXB3xPOOVwnBbFAHe9sYVdB1sC/pJ0jwSm8L73yIexm3klCLdlEmdRASWBdPkNHV3+Qa8YDKpMQog2zVXqKiHenzlxJbX41HyrwMJVH7P8EU4dtMY2XyAMN1oiq7skzbVcUQYAACAASURBVI/Pmcm3Pj6VP196VMISLMPxeqRbsMCNz6zjpP/3aqD1s9crgZB1iN4aOJEYA/mZaYwtzOrWglpJPo41mArKZPAn2lKISJE7EJqBrZbJ8GHmmAK23XxuYP07Z0yns8vPg4t3cMLNr7D2+rOiTnOJCF87ZTLvbT3AZcdPTHjxy0jMqigIBAy4Of92y5IKLxwZ7ldJBsYYvF7h6EklLO1jAVGl7zjKZCB+fz2hlomLSLW5HDbccDYPf+VYppUPbskCJXkUZqfzySODBSZn/8/zLNtuPSAjTV/96JxZPH71iQP2H/mIsHbIDo1222ev3U7Zqbg8EJaCUzJ/bFE2exvaQkKUb372o5AumUriaU0hy0SViYtYDY4y0jycMHXUwAqkDDhHjisKaRj2uuOjSAGD9IQppZw3tyKkk6cbx2r+25XHUZqbwfoIAQKJxmnmNbYwC5/fhFhDf3p9M5ffuyTpMoxkHMtEHfApRjQHvDJy8HisSgQOK3dZhRNT4Tfh8Qh3fG4+p84YHbGNgFMfrigng0/MG8uOAy1xl9rvK5YhIoFukvGUfFESR1unVdJHLZMUI1Yfb2XkIK5GWOv3Wm/3qVZw+YeLZrLt5nMpL4jcTnlCSQ4tHV3sS7LfxNiWSUWRFW1WHSFxsbZBkxmTRav6TFKTvMw0br/kSLuXhjKSufq0qTz0lWDbnFSwTCJx56ULAnXifF1BK2SCXTJmx/7etSnuLcb2mTiWSaQs+P724lGikyote0GVSQhZ6V7OP2LsgDYXUlKXeeOCDu/9cbRlHgyOGFfEs98+hR+cNYPzjwiWU3HaPG9LsjJxfCZF2emkeyWiJbSvMflRZSOVQ3bZnKIIxWgHGlUmihKF3My0gG/C6WOSimRneLn6tKkhOSaVRdl4BHYkOXHRb6xpQY/Hsk5W7+7enGsgQpRHKgftl5xUqCqueSaKEoMfLpqZkJ41A01GmofK4uykWybGGOyIZD42vYy/LN5OW2dXyBx+nVomSWP3oVYy0jwh3VsHC7VMFGWYYhWjTLZlYgJVIeaMLQAsS8QdRbYqgrWiJIYd+1sYX5ITsS3BQKPKRFGGKRNKc9gepWR9ojAEI92c5l51TR0hBSBfWLs3qTKMZFo7u8hNAec7qDJRlGHLxNJcDrV0ciiJ/dmdDHiAUXajuLrG9pACkADtPq0onAzaOrvITIGwYFBloijDlsMqrRL7b25MXk94vzGBvKxRdkHUuqagMnGqHR9I0Wi4oU6bz58SOSagykRRhi3zxxeT5hHWVTck7RpW0qJtmQSmudoDpYlm2HXCIvV+UfpPe2cXWWmp8RhPDSkURUk4GWkeJpflsmFv8hpl+f3Baa6sdC/5mWnUNXUECj46OVvVhzRxMRmER84NJqpMFGUYM708nw17k2cVuKe5wPKb7HNNc5XZfpQDzRoenAzaOv1kqmWiKEqymVGez44DLTy4eHtSzm8I7f8zKi+D/U3tgWiurHQveba1oiSeNp9aJoqiDABO/52fPr46KRWEnUKPDqPyMqlr6ghcyyNWp1J1wCcHa5orNR7jqSGFoihJ4czZ5UywC0H+/Mm1CT+/OzQYLGWyr7Gdu9/cCjhFILNYvkO7MCYaYwxtnRrNpSjKAOD1CH/8/FEA/Gv5roSf3x/BMqlv7eSOVzcBlmVyzMQSdh1spcPnT/j1RzLt9v1UZaIoyoAwe2wBlx43ISll9J1Cjw6j8kNrRIkIlcVWefq92tckobR0pE7LXlBloigjgsribOpbO2ls60zoeU1YNFdFYVbIdo8IYwpVmSQDp6tl+D0fLFSZKMoIoMq2DnYnuK2uCfOZzK0qCtnukWAWfKQujErf2WnXXRtXkjPIkljEpUxE5NsiUiAW94jIchFZmGzhFEVJDJVFtjI5mFhlEu4zKQ4rhV7f2skY+805UhdGpe84LwbOi8JgE69l8mVjTAOwECgGLgVuTppUiqIkFMdvsSsJysTtMwkvhV5d30ZBVhrZ6V5qdJoroTS0diIChSnQZRHiVybOL+Qc4C/GmDWuMUVRUpyyvEwy0zwJn+ayQoNDx1Zet5BffXouAIdaOhA7PFgtk8TS2O4jLzMtRJkPJvEqk2Ui8gKWMnleRPIBjfNTlCGCiFBZlM2ug8H+Js+uqub3dghvX3EXenQoyEpntt0oa+roPADKC7LUMkkwTW0+8jNTp1luvJJcAcwDthhjWkSkBPhS8sRSFCXRbKlrZktdM41tneRnpfP1h5YDcPZhYyjLz2RbXQuHVxX26pxWaHD38cMqC3n22ycz3c7AryjM4r2tB/r9HZQgTe0+8rKGnjI5HlhhjGkWkS8A84HfJU8sRVGSxaPLd4eUN/n4b17nP46q4h/LdvGnLxzFosPGxH2uSJaJw6yKgsByeWEWexva8PsNnhRoMTscaLKnuVKFeKe5/gi0iMgRwH8Bm4H/i3WAiIwTkVdFZK2IrBGRb9vjJSLyoohstP8ttsdFRG4TkU0islJE5rvOdbm9/0YRubxP31RRRjjPf+cUAK59Yg2/e3ljyLYddpjpVQ8u69U5w5MWo1FRmIXPb6jT6sEJo7HNR15WajjfIX5l4jNW5bYLgDuMMb8H8ns6BvgvY8xs4DjgahGZDVwDvGyMmQa8bK8DnA1Msz9XYikw7Cm1a4FjgWOAax0FpChK/Dj+C4ey/Exy7P7h7imoZdvjn44KL/QYjXI712RvvSqTRNHY1kleZmpkv0P8yqRRRH6EFRL8tIh4gJgq0RhTbYxZbi83AuuASiyF9IC92wPAhfbyBcD/GYvFQJGIVABnAS8aYw4YYw4CLwKL4v6GiqIAVtjumz88jamj83j86hNZ8uPTeeKbJ3Xb79N/fDfuc4YXeoyGk6W9Yteh+AVWYnKwpZOS3Iyedxwg4lUmFwHtWPkmNUAV8L/xXkREJgJHAu8B5caYantTDVBuL1cCO12H7bLHoo2HX+NKEVkqIkv37dsXr2iKMqIYV5LDS9/7GPPGFSEiTLY7IYZT3xpf2RW/MXHlCDhZ8D97fHW8oiox6Ozyc6C5g7K81CilAnEqE1uBPAQUish5QJsxJqbPxEFE8oB/Ad+xEx/d5zVY/XX6jTHmTmPMAmPMgrKyskScUlGGPR6PcOtF87qNv7WxLq7j4/WZlNr94ZXEcKjFUvYluUPMZyIinwWWAP8BfBZ4T0Q+E8dx6ViK5CFjzKP28F57+gr731p7fDcwznV4lT0WbVxRlARw4ZFBQ/8n58wCYH1NQ7TdQ4jXZ+L1CF87ZTIAT63c03shlRCcgp0FKZL9DvFPc/0EONoYc7kx5jIsR/jPYh0g1uvKPcA6Y8wtrk1PAE5E1uXAv13jl9lRXccB9fZ02PPAQhEpth3vC+0xRVESxEvf+xhPfPNEvnrKZCaU5nDf29u4/sm11DbGTjQML/QYi5kVVszONx/+gEW3vtFvmUcyDW0+wEoQTRXiDVL2GGNqXev76VkRnYjlsF8lIivssR9j1fR6RESuALZjWToAz2Bl2G8CWrCTIo0xB0TkF8D79n7XG2M0+0lREog70svrERrbfdz79lb2NbVz+yVHRj3ObwyeOF9JJ5YG/TMf1TRS39JJYU7qPAyHEg22Tyt/CCYtPicizwN/tdcvwnr4R8UY8xbR63edHmF/A1wd5Vz3AvfGKauiKP3gV5+ey2f+ZEV0rdldH3Pf8EKPsZgU5ux/7INdfPHESX0TcoTTMFSnuYwxPwDuBObanzuNMf+dTMEURRkcFkwsobzAcphvqWvmzjc2R93XmPgrvhblZIRUuL0uCT3pRwoNrak3zRV3cyxjzL+MMd+zP48lUyhFUQaXf151AodVWuVQbnrmI5bvOBhxP3+MciqR+P7C6SHr+xo1ibEvBB3wqTPNFVOZiEijiDRE+DSKSHzhHoqiDDnGleTw1H+ezNWnTQFg5c7QZMPaxjZO/81rbNvfQkZa/A1bJ5SGTnWtrdbHSF9oaOvE65GU6f8OPSgTY0y+MaYgwiffGFMQ61hFUYY+3184g5wML9sPtISMP7+6hs37moHeTbWE+0021zb1X8gRSEOrj4Ks1OllAtoDXlGUGIgI40ty2LE/VJlUFAZbxaanxf9AqyzK5rQZZfzpC0cBcP1T6jfpCzUNbRTlpE4pFYg/mktRlBHKhNKcgBXi4PUGFUinL/4iFh6PcN+XjkmYbCOVVbvqOX5K6WCLEYJaJoqixGRCaS47DrTg9weVRleXtTyuJJtv2H6V3vK9My1nfIdPm7b2hg6fn72NbYwvyRlsUUJQZaIoSkzGl+TQ4fOHOMt9fksB3HnpAkb1se5WaZ41TXOwpaOHPRU3+5vbMSZY1j9VUGWiKEpMHGVx3u1vBcZ8tpWS1o+uiaV2+fS6Jg0P7g37myzl6yjjVEGViaIoMTlhqjU37y5X32UrE29/lImtpNwthJWe2Wcr31GqTBRFGUoUZKWzaM6YkFT3Tttnku7t+yPEsUycN20lPpz71dfpxWShykRRlB6ZVJbLln3N7LTzTbpsn0l/LJOy/ExENHGxt9TUtwLW/UslVJkoitIjTtvdk3/1KuDymXj7rkzys9I5clwRy7dHLtWiRGbb/hbKCzLJyUitzA5VJoqi9EhZ2JSKr8txwPfvETJ7bAHr9zZiFQ1X4qG+tZPiFEtYBFUmiqLEwVlzxpCfab0J+7r8rN1jTU2l98MyAZhRnk9jm4+ahthNuJQgjW2dKdXHxEGViaIoPeLxCNdfOAewGlv9felOAPIy+/dQm1JmNeXaEpZhr0Snqd1HfgqVnndQZaIoSlxMG2213XWXo+9vocFy2xfTU3tgJUhLexfZGalTLdhBlYmiKHExpSwPEfhgh1WO/ry5Ff0+p5PFXdugiYvx0u7zk5WmykRRlCFKdoaX8SU5LLOjrz5xxNh+nzMvM42cDC97VZnETVtnF1npqffoTj2JFEVJWdK9HnbYuSajEpTnMK44h837tK9JvLT7/GSqZaIoylDm5GmjAsszyvMTcs4FE4tZtv1goERLIthU24iva3hWI1bLRFGUIc+XT5wUWM7tZySXw6yKAprafQnrB1/b0MYZt7zBD/65MiHnSyV8XX58fqOWiaIoQ5vKIqvD4riS7B727MU5i61z7TrY0sOeoWyqbeSdzXXdxhvafAA89sHu/guXYnTY1lYqWiapl/miKErK4vEIb/zgNLIyEvcwG2crk92HWlnQi+POuOUNAF763ilMHR2ccnN6rQB8sOMgR44vToicqUBbp/XdMtNST5mknkSKoqQ040tzGJ2fuMZMY4scy6S1T8ev2RNaKNIp9QJQN8wqErf7ugDIStdpLkVRlBByMtIozc3otTI5aaoVDBDua+l0Od4b2zr7L2AKEbBMUnCaK/UkUhRlxFFZnN1rn4lTn2pvWF0vnysqbLg54QOWiTrgFUVRuuM3hjc31vUqosspNFzTEN0ySWS4cSqglomiKEoM5lYVAfCPZTtDxjt8fupbIk9VGSxFsbc+zDKxfSZzxhYARD1+KNLQan0XLfSoKIoSgRsuOIzKomx+9dx6Wju6AuPffWQFR1z/Av4IFoY/YJmEKpOrHlwGwMnTygDY2cvps1SmLtD/PbW6LIIqE0VRUgCPR/jMUVUAPPDutsD486trACtsOBzjUibu5lottjKaXJYLECj/MhxwlElpnjbHUhRFichXTray61fuOhQYqyiyQpD3RFAm2NNcHT4/hyJMZR1eWQjANx5aHlI2fyhT19RBRpon0KgslVBloihKSpCflc4p08t4ZlUNL67dC0BVUQ4AF925uNv+7pkv91TXsZNKGFuYxayKgsDYjx9dlSSpB5a6xnbK8jL73UcmGagyURQlZZg8ypqa+ur/LQVgTGEwOTLckW6MIc1jPVQ37G0MjmMlVgIU2OHDH9U0Mhyoa+5gVApOcUESlYmI3CsitSKy2jV2nYjsFpEV9ucc17YficgmEVkvIme5xhfZY5tE5JpkyasoyuAzviQnZN2dM/LQku0h2/wGppfn4xHYXBssYW+MwWO/uRfmBKOe2jq7GOrsb2qnNAWd75Bcy+R+YFGE8d8aY+bZn2cARGQ2cDEwxz7mDyLiFREv8HvgbGA2cIm9r6Iow5AyV4+Uzi4/vi4/E20ro/pQaNSWATLSPFQUZrPTlT3vNwSUSVF28C3+qZXVSZR8YKhrah95lokx5g3gQJy7XwD8zRjTbozZCmwCjrE/m4wxW4wxHcDf7H0VRRmGzK0qDCxv2NuIz2/Izkjj6InFrK0OrcFljEEEqoqz2emK2PLb4wBfP3VKYPz7//gwucInGWMM+5s6RqRlEo1vishKexrMKedZCbizlXbZY9HGuyEiV4rIUhFZum/fvmTIrShKkplQmsvvLp4HwLm3vUVLh490rzBnbCHrqhtCMtqNAQGqinNC6nq5LZNzDq9g803WbLpbUQ1FGtt9+PyGkpwRZplE4Y/AFGAeUA38JlEnNsbcaYxZYIxZUFZWlqjTKooywMwbVxRYfnvTfnYdbGXmmHxaOrpCQoQNlm+ksjibvY1tgTIqls8keD6vRzh3bkUge3yocqjZkr8oJ/Wy32GAlYkxZq8xpssY4wfuwprGAtgNjHPtWmWPRRtXFGWYMqE0lwxXv44DzR2U21FdtY1Bv4nfjzXNVZSNMVBjl1XxuxzwDpVF2eypD01uHGocbLHK6RerZQIiUuFa/STgRHo9AVwsIpkiMgmYBiwB3gemicgkEcnActI/MZAyK4oy8HzlpEkh604Drdtf2RQYMxhEJNAPZeFvrWZZlpIJVSYVhVl0+Pw88M62mNdduesQs//nuYBiSiUO2ZZVce4Is0xE5K/Au8AMEdklIlcAvxKRVSKyEjgN+C6AMWYN8AiwFngOuNq2YHzAN4HngXXAI/a+iqIMY76/cAbnHzEWgIWzy5lSlgfAkq3BmB6/7TMZXWA5pFvt0F9/2DQXBBtwXffk2pjXvf+dbbR0dPHI0p3dStsPNodsy6Qwe4RZJsaYS4wxFcaYdGNMlTHmHmPMpcaYw40xc40xnzDGVLv2v9EYM8UYM8MY86xr/BljzHR7243JkldRlNTB4xFOmFIKQEF2OiLCt06fRmtnV7DEvLGmuRxFAVYuiXE54B2qioP7+Fwl6sPJ8FqPxFte3MCxN72cqK+TEJzy/KW5I0yZKIqi9IejJljBnsdNtpRKuG/EccDnZabxySOtIM9dB1styyTsyTajPNgjvjZGzxSnkKJDKvlYtu9voSArTR3wiqIovWF6eT4r/udMPj3fUhSVtnXhVAH225YJwOePHQ9Y5eatPJNQyyTN6+G+Lx0NRCsaadHuC7VaetOsK9nUt3ZSkpuRknW5QJWJoigpTFFO8OF52NhC0jzCO5vrgNCyKePsMizLth2MOM0FVkQXwJ4YznV/mCWSSuXrWzp85GSkXrVgB1UmiqIMCQpz0pkxJp+Vu+qB0KrBZXZW+B2vborogAcrogtiWyb+MHdKKjXWam7vIjcz9Xq/O6gyURRlyDC3qpCVu+oxxmra61ggHpf2aPf5I1om+Vnp5GelUR1DmXSFWSY7D0Tfd6Bp6fCRm4J9TBxUmSiKMmSYW1VEfWsnW+qaA7W5HJwyLNX1bURzK1QWZYeUXgnHGEOJK1pqVypZJh1d5Oo0l6IoSv85ZlIJYOWbOLW5HGa7mmFFskwAJo3KZWtdc9Tz+40V+XXhPCvHJaUsk3YfORmpO82VumpOURQljMmjchmVl2kpE0LLpkwaZZVh6fD5I/pMAKaU5fHC2r10+PwhJVsc/MaQ5hVuvfhIAJZuT512v80dXTrNpSiKkghEhGMnlfDu5v2B2lwOaV4P08utTPlolsmU0bl0+Q07DkS2Tvz+0Aix6vq2mEmOA4kVzZW6lokqE0VRhhSnTB9FTUMbH9U0dMu5mG4nJza2+SIeO3mUpWw21UZRJoaAVTO+JIcuv2HJtnjbMiWP5nYfnV2GguzUTFgEVSaKogwxTp0xGgjW5nLj+E1W7DwU8diJdo/551ZH7rroNwavrU0WzhmDR+CfS3clQOr+4dQJG52fmo2xQJWJoihDjPKCLOaPt3qehM9mXWiXVdkdJfy30H6zf3zFnoilUqysegnse+LUUWze19Rtv4HGKQFTXpA1yJJER5WJoihDjmmjremscN+IUwTxuMklUY+dbFsnr3xU222b5TMJrlcV54T0lx8sHGWilomiKEoCmTDKKp/S0tEVMi4iLP7R6dz7xaOjHvv8d08hK93DO5v3d9sW3lhrXEk2B5o7aGqP7IMZKGqdaS61TBRFURLHhBLLuohU7mRMYVbMGlbpXg9tnX7ueWtrsJy9jVVxOKhM5oy1+sa/t6W74hlIahvbyUzzUJClocGKoigJY+poKyqrtqFvVX2doo/h2fDhRSKPtH0zG2sH12+yt6GN8oKslK0YDKpMFEUZgkwdnceUslyuPGVyn46/1S69ssmlJP78+ma21DWH+EwKstIpzc1g+/7oWfMDQW1De0r7S0Az4BVFGYJ4PcLL/3Vqn4+fW1VIuldYvuMgZ84uB+CXz35knTvs7X9CaU7MEiwDwZ76Vg6vLBxUGXpCLRNFUUYcmWleppTlsb6mESDEwR4+lTRtdH6IBTPQdPkNuw+2Mt7u2ZKqqDJRFGVEMmNMPh9VNwDQ2NYZGG/rDI0Qm1aeR11TBweaOwJjtQ1t1Ld2MhBU17fi8xtVJoqiKKnIjDH57Km3lIKvK5jAGJ6kOMV29v/40VUYY2jr7OKYm17mc3ctHhA5d9tBAk7b4lRFlYmiKCOSWWOs0ivrqhvocIUIh09pTbOVyXNrali9u4FVu61Oj2v2NAxIv5MaO8ekolCViaIoSsoxb5wV9rtk64EQy6QoJyNkv7Guh/ja6no6fUHFs3hL8otAVts968cUpm7CIqgyURRlhFJsl1655cUNgeTFL54wkUe/fkLIfh6PsOWmc8jPSuOpldX4XM3n1+ypT4psb22sY+cBy+rZdbCF/Kw08lK4lwmoMlEUZQQzy64y/PtXNwHwselljC/t7uj2eITDKwt5b+sBWm0HvdcjrN6deGVS19TOF+55j5N/9Sp3v7mFD3fWc9jY1A4LBlUmiqKMYC4/fgIAz66uAaxSK9H4zFFVdPj8gZyTI8cVsWZPA13+7tWH+8OhlmDU2A1Pr2PV7nrmTyhK6DWSgSoTRVFGLBcdPY50bzCvJM0bvVzJBNticRTP7LEFtHR08fyamoTK1NnVXTktmBi9CnKqoMpEUZQRi4jw5RMnBdbTYyiTeeOKKcnN4EO78dbxk0sB+MZDy/En0Dpx/Dc/PXcWv/zU4fzsvNmcMq0sYedPFqpMFEUZ0RzteuvPTo/u5PZ6hIV26RWAmba/BeD9BLb2dSyTaeX5XHLMeK44aVKg+2Mqo8pEUZQRzRkuBVFVEjuXY0pZXmA5zSPc/KnDgehtgvuCY5nEspJSEVUmiqKMeE6bUUZBVhoFWekx95tclhtY9nqEi48ZT1VxNit39S2qq8tv+PFjqwI1wiCoTDJiBAOkIkNLWkVRlCRw9+VHs/SnZ/a4X4hlYlsOR1QV8fSqanxhjbbioaahjYff28FFd74bGLv0niVA7MiyVGRoSasoipIEvB4hI63nx+E4V7FFp1T9zDFWP/q3NtVFPKbLb6IqmtYOq1rxoZbuRSOHgp/EjSoTRVGUOPF6hMe+cQLnHzE2UHblshMmArDWrkAczqm/fpXTfvNaxG1N7cEKxa+trw1ROtkZ3sQIPUAkTZmIyL0iUisiq11jJSLyoohstP8ttsdFRG4TkU0islJE5ruOudzef6OIXJ4seRVFUeLhyPHF3H7JkQHLoTA7nXEl2azZE1mZ7DzQys4DrRjTPXz4xbXBHJUnVuyhucNSLpccMy5kSm0okEzL5H5gUdjYNcDLxphpwMv2OsDZwDT7cyXwR7CUD3AtcCxwDHCto4AURVFShdkVBayNokwcnIKNbn7/6mYAFkwo5tEPdvPEit0AzK1K/Yz3cJKmTIwxbwDhwdcXAA/Yyw8AF7rG/89YLAaKRKQCOAt40RhzwBhzEHiR7gpKURRlUJkztpBt+5tDOjaGEyvi6ysnW4mTP/v3GoCU7/ceiYH2mZQbY6rt5RrACfCuBHa69ttlj0Ub74aIXCkiS0Vk6b59+xIrtaIoSgxmVxRgDIHOjW4KsqxEyKseXMa2sF7yWekevnryJM6aM4YzZgXzXeYMgcKO4QyaA95YE4gJq0FgjLnTGLPAGLOgrCz1Sw8oijJ8mFNpZcOHO+G7/IaGtqC1cv1TawPLVtdGP9npXkSE2y6Zx5iCLL57xvSU710SiYEukL9XRCqMMdX2NFatPb4bGOfar8oe2w2cGjb+2gDIqSiKEjdjCrIozklnze5QZbJ4y/6Qdb/LCd9uN9nKzrAewzkZaSz+8elJljR5DLRl8gTgRGRdDvzbNX6ZHdV1HFBvT4c9DywUkWLb8b7QHlMURUkZRIQ5Ywu7WSZfeWApADd98nDOm1sR0l++xY7cyk4fHhkayQwN/ivwLjBDRHaJyBXAzcCZIrIROMNeB3gG2AJsAu4CvgFgjDkA/AJ43/5cb48piqKkFLPHFrC+pjFQDqW+tTPQSOuEKaVMLstj54FWlu84CBDYNtTySaKRtGkuY8wlUTZ1s+Ns/8nVUc5zL3BvAkVTFEVJOHPGFtDR5WdTbROzKgr4r0dWBLZNHJXLJ4+s5LaXN/Lkh3uYP76YVscyyUjtdrzxMjzsK0VRlEFmzljLCe9YHi+ts1zCXznJCvudNCqXj88czQtr9uL3G9ocyyRdLRNFURTFZkpZHpVF2by0di91jcHWuz9YNCOwfP4RFbzyUS0//fdqHn5vB6DKRFEURXEhIhw5voinVlbz6nor1+1T8yvJTAsqi4WzxwAfBhQJ3UfrSwAACe9JREFUQHFu7LL3QwWd5lIURUkQZ7oabQEs3XYwZD03M43PLqgKrGene5lenj8gsiUbtUwURVESxCeOGEtpbiYHWjp44J1t/OfHp3bb56ZPHs600fnsqW/l2vPnDIKUyUEiVbIc6ixYsMAsXbp0sMVQFEUZUojIMmPMgr4cq9NciqIoSr9RZaIoiqL0G1UmiqIoSr9RZaIoiqL0G1UmiqIoSr9RZaIoiqL0G1UmiqIoSr9RZaIoiqL0m2GZtCgi+4DtrqFCoD5sOdLYKKCul5dzn6c3+/Q01tNyf2SOdv14toePx1of6nIn6jcSS65Y2xP1G4Hk/Lb1NzK0fiPu9Why5xpj+tb33Bgz7D/AneHLUcaW9ufcvdmnp7GelvsjczxyR9sePh5rfajLnajfSF/lTtRvpK9y629keP1GkiW38xkp01xPRliONNbfc/dmn57Gelruj8zxHB9te/h4rPWhLneifiPxHK+/kaEp91D6jbjXEyk3MEynufqKiCw1faxLM1gMRZlB5R5ohqLcQ1FmGLlyjxTLJF7uHGwB+sBQlBlU7oFmKMo9FGWGESq3WiaKoihKv1HLRFEURek3qkwURVGUfqPKRFEURek3qkziQEROFpE/icjdIvLOYMsTLyLiEZEbReR2Ebl8sOWJFxE5VUTetO/5qYMtT7yISK6ILBWR8wZblngRkVn2ff6niHx9sOWJFxG5UETuEpG/i8jCwZYnXkRksojcIyL/HGxZYmH/lh+w7/Hn4zlm2CsTEblXRGpFZHXY+CIRWS8im0TkmljnMMa8aYy5CngKeCCZ8rrk67fcwAVAFdAJ7EqWrG4SJLcBmoAsBkDuBMkM8N/AI8mRsjsJ+m2vs3/bnwVOTKa8LvkSIffjxpivAlcBFyVTXpd8iZB7izHmiuRKGpleyv8p4J/2Pf5EXBfoT8bjUPgApwDzgdWuMS+wGZgMZAAfArOBw7EUhvsz2nXcI0D+UJEbuAb4mn3sP4eQ3B77uHLgoSEi85nAxcAXgfOGyr22j/kE8CzwuaEkt33cb4D5Q1DuAfn/2A/5fwTMs/d5OJ7zpzHMMca8ISITw4aPATYZY7YAiMjfgAuMMb8EIk5RiMh4oN4Y05hEcQMkQm4R2QV02KtdyZM2SKLut81BIDMZcrpJ0L0+FcjF+o/YKiLPGGP8qS63fZ4ngCdE5Gng4eRJHLheIu63ADcDzxpjlidXYosE/7YHnN7IjzUjUAWsIM4ZrGGvTKJQCex0re8Cju3hmCuA+5ImUXz0Vu5HgdtF5GTgjWQK1gO9kltEPgWcBRQBdyRXtKj0SmZjzE8AROSLQF2yFUkMenuvT8Wa0sgEnkmqZLHp7W/7P4EzgEIRmWqM+VMyhYtBb+93KXAjcKSI/MhWOoNJNPlvA+4QkXOJs9zKSFUmvcYYc+1gy9BbjDEtWEpwSGGMeRRLEQ45jDH3D7YMvcEY8xrw2iCL0WuMMbdhPfCGFMaY/Vh+npTGGNMMfKk3xwx7B3wUdgPjXOtV9liqo3IPHENRZlC5B5qhKrdDwuQfqcrkfWCaiEwSkQwsx+kTgyxTPKjcA8dQlBlU7oFmqMrtkDj5BzqiYBAiGP4KVBMMj73CHj8H2IAVyfCTwZZT5VaZVW6VeyjLr4UeFUVRlH4zUqe5FEVRlASiykRRFEXpN6pMFEVRlH6jykRRFEXpN6pMFEVRlH6jykRRFEXpN6pMlJRARJoG4BqfiLOUfCKveaqInNCH444UkXvs5S+KyGDVKAtBRCaGlzCPsE+ZiDw3UDIpqYEqE2VYISLeaNuMMU8YY25OwjVj1bg7Fei1MgF+zBCsPQVgjNkHVIvIgPRHUVIDVSZKyiEiPxCR90VkpYj83DX+uIgsE5E1InKla7xJRH4jIh8Cx4vINhH5uYgsF5FVIjLT3i/whi8i94vIbSLyjohsEZHP2OMeEfmDiHwkIi+KyDPOtjAZXxORW0VkKfBtETlfRN4TkQ9E5CURKbfLfV8FfFdEVojVsbNMRP5lf7/3Iz1wRSQfmGuM+TDCtoki8op9b162WyMgIlNEZLH9fW+IZOmJ1T3vaRH5UERWi8hF9vjR9n34UESWiEi+fZ037Xu4PJJ1JSJeEflf19/qa67NjwNxdehThgmDneKvH/0YYwCa7H8XAncCgvWy8xRwir2txP43G1gNlNrrBvis61zbgP+0l78B3G0vfxG4w16+H/iHfY3ZWD0dAD6DVYrdA4zB6qnymQjyvgb8wbVeDIGKEl8BfmMvXwd837Xfw8BJ9vJ4YF2Ec58G/Mu17pb7SeBye/nLwOP28lPAJfbyVc79DDvvp4G7XOuFWA2RtgBH22MFWNXEc4Ase2wasNRenojdXAm4EvipvZwJLAUm2euVwKrB/l3pZ+A+WoJeSTUW2p8P7PU8rIfZG8C3ROST9vg4e3w/VuOvf4WdxylhvwyrX0ckHjdW35G1IlJuj50E/MMerxGRV2PI+nfXchXwdxGpwHpAb41yzBnAbBFx1gtEJM8Y47YkKoB9UY4/3vV9/gL8yjV+ob38MPDrCMeuAn4jIv8PeMoY86aIHA5UG2PeBzDGNIBlxWD1s5iHdX+nRzjfQmCuy3IrxPqbbAVqgbFRvoMyDFFloqQaAvzSGPPnkEGridMZwPHGmBYReQ2rRzxAmzEmvJNku/1vF9F/5+2uZYmyTyyaXcu3A7cYY56wZb0uyjEe4DhjTFuM87YS/G4JwxizQUTmYxX2u0FEXgYei7L7d4G9wBFYMkeSV7AswOcjbMvC+h7KCEF9Jkqq8TzwZRHJAxCRShEZjfXWe9BWJDOB45J0/beBT9u+k3IsB3o8FBLsA3G5a7wRyHetv4DVJRAA+80/nHXA1CjXeQerTDhYPok37eXFWNNYuLaHICJjgRZjzIPA/2L1A18PVIjI0fY++XZAQSGWxeIHLsXqFR7O88DXRSTdPna6bdGAZcnEjPpShheqTJSUwhjzAtY0zbsisgr4J9bD+DkgTUTWYfX+XpwkEf6FVZ57LfAgsByoj+O464B/iMgyoM41/iTwSccBD3wLWGA7rNcSoeueMeYjrHa0+eHbsBTRl0RkJdZD/tv2+HeA79njU6PIfDiwRERWANcCNxhjOoCLsNo7fwi8iGVV/AG43B6bSagV5nA31n1abocL/5mgFXga8HSEY5RhipagV5QwHB+GWP26lwAnGmNqBliG7wKNxpi749w/B2g1xhgRuRjLGX9BUoWMLc8bwAXGmIODJYMysKjPRFG685SIFGE50n8x0IrE5o/Af/Ri/6OwHOYCHMKK9BoURKQMy3+kimQEoZaJoiiK0m/UZ6IoiqL0G1UmiqIoSr9RZaIoiqL0G1UmiqIoSr9RZaIoiqL0G1UmiqIoSr/5/03V0tkb+BK2AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.lr_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 4: Train and Inspect the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "begin training using onecycle policy with max lr of 0.03...\n",
      "Train on 95646 samples, validate on 23912 samples\n",
      "Epoch 1/10\n",
      "95646/95646 [==============================] - 8s 79us/sample - loss: 1556.0435 - mae: 19.2369 - val_loss: 984.7442 - val_mae: 15.1122\n",
      "Epoch 2/10\n",
      "95646/95646 [==============================] - 8s 79us/sample - loss: 1052.5454 - mae: 13.0505 - val_loss: 808.2142 - val_mae: 12.5382\n",
      "Epoch 3/10\n",
      "95646/95646 [==============================] - 7s 76us/sample - loss: 809.7949 - mae: 9.4578 - val_loss: 695.8532 - val_mae: 10.8098\n",
      "Epoch 4/10\n",
      "95646/95646 [==============================] - 8s 80us/sample - loss: 616.9707 - mae: 6.6427 - val_loss: 621.5498 - val_mae: 9.9253\n",
      "Epoch 5/10\n",
      "95646/95646 [==============================] - 7s 78us/sample - loss: 471.5737 - mae: 4.8021 - val_loss: 582.4865 - val_mae: 9.9948\n",
      "Epoch 6/10\n",
      "95646/95646 [==============================] - 8s 79us/sample - loss: 369.3043 - mae: 4.1017 - val_loss: 572.5836 - val_mae: 10.4219\n",
      "Epoch 7/10\n",
      "95646/95646 [==============================] - 8s 79us/sample - loss: 304.9035 - mae: 3.7351 - val_loss: 563.6406 - val_mae: 10.3136\n",
      "Epoch 8/10\n",
      "95646/95646 [==============================] - 8s 79us/sample - loss: 257.2997 - mae: 2.8500 - val_loss: 562.7244 - val_mae: 10.0789\n",
      "Epoch 9/10\n",
      "95646/95646 [==============================] - 8s 79us/sample - loss: 226.9375 - mae: 1.9855 - val_loss: 559.9848 - val_mae: 9.7024\n",
      "Epoch 10/10\n",
      "95646/95646 [==============================] - 8s 79us/sample - loss: 211.9495 - mae: 1.3842 - val_loss: 561.2627 - val_mae: 9.6977\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f653d0d1cf8>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.fit_onecycle(0.03, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our MAE is roughly 10, which means our model's predictions are about $10 off on average.  This isn't bad considering there is a wide range of wine prices and predictions are being made purely from text descriptions. \n",
    "\n",
    "Let's examine the wines we got the most wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------\n",
      "id:6695 | loss:675000.75 | true:980.0 | pred:158.42)\n",
      "\n",
      "this was a great vintage port year and this white port which was bottled in 2015 has hints of the firm tannins and structure that marked out the year it also has preserved an amazing amount of freshness still suggesting orange marmalade flavors these are backed up by the fine concentrated old wood tastes the wine is of course ready to drink\n",
      "----------\n",
      "id:19469 | loss:524528.9 | true:775.0 | pred:50.76)\n",
      "\n",
      "perfumed florals mingle curiously with deep dusty mineral notes on this bracing tba sunny nectarine and tangerine flavors are mouthwatering and juicy struck with acidity then plunged into of sweet honey and nectar it's a delightful sensory roller coaster that feels endless on the finish\n",
      "----------\n",
      "id:3310 | loss:400394.03 | true:848.0 | pred:215.23)\n",
      "\n",
      "full of ripe fruit opulent and concentrated this is a fabulous and impressive wine it has a beautiful line of acidity balanced with ripe fruits the wood aging is subtle just a hint of smokiness and toast this is one of those wines from a great white wine vintage that will age many years drink from 2024\n"
     ]
    }
   ],
   "source": [
    "learner.view_top_losses(n=3, preproc=preproc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like our model has trouble with expensive wines, which is understandable given the descriptions of them, which may  not differ much from less expensive wines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 5: Making Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = ktrain.get_predictor(learner.model, preproc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a prediction for a random wine in the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: This Millesimato sparkling blend of Pinot Nero and oak-aged Chardonnay delivers a generous and creamy mouthfeel followed by refined aromas of dried fruit and baked bread. This is a beautiful wine to serve with tempura appetizers.\n",
      "Actual Price: 52.0\n"
     ]
    }
   ],
   "source": [
    "idx = np.random.randint(len(x_test))\n",
    "print('Description: %s' % (x_test[idx]))\n",
    "print('Actual Price: %s' % (y_test[idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our prediction for this wine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([52.698753], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict(x_test[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Transfomer API for Text Regression\n",
    "\n",
    "*ktrain* includes a simplified interface to the Hugging Face transformers library.  This interface can also be used for text regression. Here is a short example of training a [DistilBERT model](https://arxiv.org/abs/1910.01108) for a single epoch to predict wine prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preprocessing train...\n",
      "language: en\n",
      "train sequence lengths:\n",
      "\tmean : 41\n",
      "\t95percentile : 61\n",
      "\t99percentile : 73\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preprocessing test...\n",
      "language: en\n",
      "test sequence lengths:\n",
      "\tmean : 41\n",
      "\t95percentile : 62\n",
      "\t99percentile : 73\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "begin training using onecycle policy with max lr of 0.0001...\n",
      "Train for 748 steps, validate for 187 steps\n",
      "748/748 [==============================] - 310s 415ms/step - loss: 1443.0076 - mae: 18.3470 - val_loss: 879.1416 - val_mae: 13.9600\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f743e36e748>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MODEL_NAME = 'distilbert-base-uncased'\n",
    "t = text.Transformer(MODEL_NAME, maxlen=75)\n",
    "trn = t.preprocess_train(x_train, y_train)\n",
    "val = t.preprocess_test(x_test, y_test)\n",
    "model = t.get_regression_model()\n",
    "learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=128)\n",
    "learner.fit_onecycle(1e-4, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For the Prediction part as discussed in the [issue](https://github.com/amaiya/ktrain/issues/417) ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = ktrain.get_predictor(learner.model, t)\n",
    "p.predict(['This is first document.', 'This is second document.', 'This is third document.'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}