tutorials/tutorial-01-introduction.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\";\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"; "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *ktrain*: A Simple Library to Help Train Neural Networks in Keras\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*ktrain* is a lightweight wrapper for Keras to help build, train, and deploy neural models in a way that requires a minimal amount of code and a reduced cognitive load. By facilitating the loading and preprocessing of data, tuning learning rates, fitting models with different learning rate policies, inspecting misclassifications, and making predictions on new raw data, *ktrain* allows you to focus more on architecting a good Keras model for your problem.  Inspired by the *fastai* library, it supports:\n",
    "- a **learning rate finder** to help find a good initial learning rate for your model\n",
    "- a variety of demonstrably effective **learning rate schedules** to improve performance including SGDR and Leslie Smith's 1cycle and triangular learning rate policies.\n",
    "-  fast and easy-to-use **pre-canned models** for `text` data (e.g., text classification, NER), `vision` data (e.g., image classification), `graph` data (e.g., link prediction), and `tabular` data.\n",
    "-  methods to help you **easily load and preprocess text and image data** from a variety of formats\n",
    "- easily inspecting data points that were misclassified to help improve your model\n",
    "- a simple prediction API for **saving and deploying models and data-preprocessing steps** to easily make predictions on new raw data\n",
    "\n",
    "We will begin by importing the *ktrain* module.  The *ktrain* library contains sub-modules to handle specific types of data. Currently, we have the *ktrain.vision* module for image classification and the *ktrain.text* module for text classification. Additional sub-modules will be added over time for tackling various other types of problems.  We will import the *vision* module here, as this tutorial example involves image classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import ktrain\n",
    "from ktrain import vision as vis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data:  An Obligatory MNIST Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*ktrain* has convenience functions to help you easily load data from a variety of formats (e.g., training data in folders or CSVs). Examples include:\n",
    "\n",
    "`ktrain.vision` module:\n",
    "- `images_from_folder`:  labels are represented as subfolders containing images [ [example notebook] ](https://github.com/amaiya/ktrain/blob/master/examples/vision/dogs_vs_cats-ResNet50.ipynb)\n",
    "- `images_from_csv`: labels are mapped to images in a CSV file [ [example notebook](https://github.com/amaiya/ktrain/blob/master/examples/vision/planet-ResNet50.ipynb) ]\n",
    "- `images_from_fname`: labels are included as part of the filename and must be extracted using a regular expression [ [example notebook](https://github.com/amaiya/ktrain/blob/master/examples/vision/pets-ResNet50.ipynb) ]\n",
    "- `images_from_array`: images and labels are stored in array [ [example notebook](https://github.com/amaiya/ktrain/blob/master/examples/vision/mnist-images_from_array_example.ipynb) ]\n",
    "\n",
    "`ktrain.text` module:\n",
    "- `texts_from_folder`:  labels are represented as subfolders containing text files [ [example notebook] ](https://github.com/amaiya/ktrain/blob/master/examples/text/IMDb-BERT.ipynb)\n",
    "- `texts_from_csv`: texts and associated labels are stored in columns in a CSV file [ [example notebook](https://github.com/amaiya/ktrain/blob/master/examples/text/toxic_comments-fasttext.ipynb) ]\n",
    "- `texts_from_df`: texts and associated labels are stored in columns in a *pandas* DataFrame [ [example notebook](https://github.com/amaiya/ktrain/blob/master/examples/text/ArabicHotelReviews-nbsvm.ipynb) ]\n",
    "- `texts_from_array`: texts and labels are loaded and preprocessed from an array [ [example notebook](https://github.com/amaiya/ktrain/blob/master/examples/text/20newsgroup-distilbert.ipynb) ]\n",
    "\n",
    "`ktrain.tabular` module:\n",
    "- `tabular_from_csv`: dependent and independent variables are stored as columns in a CSV file [ [example notebook](https://github.com/amaiya/ktrain/blob/master/examples/tabular/tabular_classification_and_regression_example.ipynb) ]\n",
    "- `tabular_from_df`: dependent and independent variables are stored as columns in a *pandas* DataFrame [ [example notebook](https://github.com/amaiya/ktrain/blob/develop/examples/tabular/tabular_classification_and_regression_example.ipynb) ]\n",
    "\n",
    "\n",
    "We will load the MNIST image classification dataset here, since no neural network tutorial is complete without an obligatory [MNIST](http://en.wikipedia.org/wiki/MNIST_database) example.  \n",
    "\n",
    "First, download a PNG version of the **MNIST** dataset from [here](https://s3.amazonaws.com/fast-ai-imageclas/mnist_png.tgz) and set DATADIR to the extracted folder.\n",
    "\n",
    "Next, use the ```images_from_folder``` function to load the data as a generator (i.e., Keras DirectoryIterator object).  This function assumes the following directory structure:\n",
    "```\n",
    "  ├── datadir\n",
    "    │   ├── train\n",
    "    │   │   ├── class0       # folder containing documents of class 0\n",
    "    │   │   ├── class1       # folder containing documents of class 1\n",
    "    │   │   ├── class2       # folder containing documents of class 2\n",
    "    │   │   └── classN       # folder containing documents of class N\n",
    "    │   └── test \n",
    "    │       ├── class0       # folder containing documents of class 0\n",
    "    │       ├── class1       # folder containing documents of class 1\n",
    "    │       ├── class2       # folder containing documents of class 2\n",
    "    │       └── classN       # folder containing documents of class N\n",
    "```\n",
    "The *train_test_names* argument can be used, if the train and test subfolders are named differently (e.g., *test* folder is called *valid*).  The **data_aug** parameter can be used to employ [data augmentation](https://arxiv.org/abs/1712.04621). We set this parameter using the ```get_data_aug``` function, which is a simple wrapper around Keras ImageDataGenerator and returns a data augmentation scheme with the following defaults:\n",
    "```\n",
    "# default data augmentation in ktrain\n",
    "def get_data_aug(rotation_range=40,\n",
    "                 zoom_range=0.2,\n",
    "                 width_shift_range=0.2,\n",
    "                 height_shift_range=0.2,\n",
    "                 horizontal_flip=False,\n",
    "                 vertical_flip=False,\n",
    "                 **kwargs):\n",
    "\n",
    "```\n",
    "Additional arguments can be supplied to further configure data augmentation.  See [Keras documentation](https://keras.io/preprocessing/image/#imagedatagenerator-class) for a full set of augmentation parameters.  Since the defaults are designed for \"normal-sized\" photos and may be too aggressive for little 28x28 MNIST images, we have reduced some of the values slightly. We have also set the image size to 28x28 and the color mode to grayscale.  If using alternative values for these parameters, the image will automatically be adjusted (e.g., resized, converted from 1-channel to 3-channel image).  This is necessary, for instance, if a particular model only supports 3-channel images ('rgb' color_mode) or a minimum image size (e.g., 32x32). The defaults are ```target_size=(224,224)``` and ```color_mode='rgb'```, which tend to work well for most other image classification problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 60000 images belonging to 10 classes.\n",
      "Found 60000 images belonging to 10 classes.\n",
      "Found 10000 images belonging to 10 classes.\n"
     ]
    }
   ],
   "source": [
    "DATADIR = 'data/mnist_png'\n",
    "data_aug = vis.get_data_aug(  rotation_range=15,\n",
    "                              zoom_range=0.1,\n",
    "                              width_shift_range=0.1,\n",
    "                              height_shift_range=0.1)\n",
    "(train_data, val_data, preproc) = vis.images_from_folder(\n",
    "                                              datadir=DATADIR,\n",
    "                                              data_aug = data_aug,\n",
    "                                              train_test_names=['training', 'testing'], \n",
    "                                              target_size=(28,28), color_mode='grayscale')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(The training folder is scanned twice - once to inspect the data for normalization and a second time to return a DirectoryIterator generator).  \n",
    "\n",
    "All *ktrain* data-loading functions return a Preprocessor instance (stored in the variable *preproc* above).  Preprocessor instances are used to automaticlaly preprocess and appropriately normalize raw data when making predictions on new examples.  This is demonstrated a little later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a Model\n",
    "\n",
    "Having loaded the data, we now have to define a model.  You can either define your own model or, if appropriate, can use a pre-canned model included in *ktrain*.  The list of available image classification models can be viewed using the ```print_image_classifiers``` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pretrained_resnet50: 50-layer Residual Network (pretrained on ImageNet)\n",
      "resnet50: 50-layer Resididual Network (randomly initialized)\n",
      "pretrained_mobilenet: MobileNet Neural Network (pretrained on ImageNet - TF only)\n",
      "mobilenet: MobileNet Neural Network (randomly initialized - TF only)\n",
      "pretrained_inception: Inception Version 3  (pretrained on ImageNet)\n",
      "inception: Inception Version 3 (randomly initialized)\n",
      "wrn22: 22-layer Wide Residual Network (randomly initialized)\n",
      "default_cnn: a default Convolutional Neural Network\n"
     ]
    }
   ],
   "source": [
    "vis.print_image_classifiers()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If opting to use a pre-canned model, the model can be loaded using the ```image_classifier``` function.\n",
    "For the MNIST problem, we will use the 'default_cnn' model, which is a simple CNN with the structure shown below. Note that the ```image_classifier``` function accepts the training data as an argument, as the data will be inspected so that the model can be automatically configured correctly for your problem (e.g., model will be configured for multilabel classification if classes are not mutually exclusive)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is Multi-Label? False\n",
      "default_cnn model created.\n"
     ]
    }
   ],
   "source": [
    "model = vis.image_classifier('default_cnn', train_data, val_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 24, 24, 32)        9248      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 12, 12, 32)        0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 12, 12, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 12, 12, 64)        18496     \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 12, 12, 64)        36928     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 6, 6, 64)          0         \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 6, 6, 64)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 6, 6, 128)         73856     \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 6, 6, 128)         0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 4608)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 128)               589952    \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 128)               512       \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 730,602\n",
      "Trainable params: 730,346\n",
      "Non-trainable params: 256\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping Your Data and Model in a Learner Object\n",
    "\n",
    "Armed with some data and a model, you would normally call the ```fit``` method of the model instance when using Keras.  With *ktrain*, we will instead first wrap the model and data in a Learner object using the ```get_learner``` function.  The Learner object will help us train our model in various ways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner = ktrain.get_learner(model, train_data=train_data, val_data=val_data, \n",
    "                             workers=8, use_multiprocessing=True, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional arguments for ```get_learner``` include *batch_size*, *workers*, and *use_multiprocessing*.  The *workers* and *use_multiprocessing* arguments only take effect if train_data is a generator, and they each map directly to the *workers* and *use_multiprocessing* arguments to ```model.fit_generator``` in Keras. These values can be adjusted based on your system and data.  Here, we will use eight workers and a larger batch size since the MNIST images are quite small.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Rate Finder\n",
    "\n",
    "The Learner object can be used to find a good learning rate for your model using the ```lr_find``` and ```lr_plot``` methods.  The ```lr_find``` method simulates training at different learning rates and tracks the loss.  After visually inspecting the plot generated by ```lr_plot```, we choose the highest learning rate still associated with a falling loss. We will choose a rate that is aggressively high: **0.001** (or 1e-3) for use in training.  (The learning rate of 0.001 happens to be the default learning rate for the Adam optimizer in Keras.  The default is a good fit for this particular instance, but this is most definitely not always the case.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "simulating training for different learning rates... this may take a few moments...\n",
      "Epoch 1/5\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 3.2559 - acc: 0.0982\n",
      "Epoch 2/5\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 2.0976 - acc: 0.3472\n",
      "Epoch 3/5\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.3393 - acc: 0.8939\n",
      "Epoch 4/5\n",
      "780/937 [=======================>......] - ETA: 2s - loss: 0.3107 - acc: 0.9132\n",
      "\n",
      "done.\n",
      "Please invoke the Learner.lr_plot() method to visually inspect the loss plot to help identify the maximal learning rate associated with falling loss.\n"
     ]
    }
   ],
   "source": [
    "learner.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEOCAYAAABmVAtTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XecVNXdx/HPbzsLS92lNxEQke6KIGIvoIkNezTwWIh5NBpNI+qTWKMpliAxhmBsiR1jiKCCgEqRsiC9gyAgZelLW7ac54+5jMu6C7vL3Lk7s9/36zUv7r1zZ+Z7dtn5zZ1z7znmnENERAQgIegAIiJSfagoiIhImIqCiIiEqSiIiEiYioKIiISpKIiISJiKgoiIhPlWFMwszcxmmdl8M1tsZg+Xsc8QM8s1s3ne7Ta/8oiIyLEl+fjc+cB5zrm9ZpYMTDWzD51zM0rt95Zz7i4fc4iISAX5VhRc6FLpvd5qsnfT5dMiItWYn0cKmFkiMAdoD/zFOTezjN0GmdlZwArgXufc+qM9Z2Zmpmvbtm3Es4qIxLM5c+Zsc85lHWs/i8bYR2ZWH/g38BPn3KIS2xsBe51z+Wb2I+A659x5ZTx+KDAUoHXr1qeuW7fO98wiIvHEzOY457KPtV9Uzj5yzu0CJgMDSm3f7pzL91ZHAaeW8/iRzrls51x2VtYxC52IiFSRn2cfZXlHCJhZLeBCYFmpfZqVWL0MWOpXHhEROTY/+xSaAa94/QoJwNvOuQ/M7BEgxzk3BrjbzC4DCoEdwBAf84iIyDFEpU8hkrKzs11OTk7QMUREYkq16lMQEZHYoKIgIiJhKgpVsDp3L5t3Hww6hohIxPl68Vo8Kip2nP/UZwCsffLSgNOIiESWjhTK4Zxj+MSVtB02lvvemscnS7bwn3kbOfH+ceF9npu4kj0HC9h/qBCA175Yy+2v5jB99baAUouIHB+dfeQ5WFDEtr35TF+1nYFdm/LajHX84aPlVX6+ox1FbNp9gC178unRqn6Vn19EpDIqevZRjS0KD/93MS9NW8urt/Smf4dM2t0/jor+KB67ogtrt+1j1NSvyt2ncUYqI27sRe8TGoa3zV67g2te+CK8/qOz2/HrgSdXuQ1lGb94Mz1a16dxRlpEn1dEYltFi0KN6lNoO2wsZ3fMYvj1PXlp2loAfviPWeXuf1WvFjx9bQ8g9HWSc5CQYECob6HIOQ4WFNOwdjL78osYNrATBw4V0fPRCWzNy+fav33BgFOasuib3bxxe58jCgLA3z5bw3XZrWiXVYfdBwrYue8QAJ+tyOWKHi2ol558zDYVFTs27zlIvycnhbf1a9+If93Wp1I/GxERqEFHCgVFxXR44MOj7vPG7X3oe2Ijtu45yNLNeZzdsWrjLM1Zt5NBf51e7v2/GtCJ33+0rNz7SzshszZfbdvH4L5t+NXATtRKTmTGmh3c8PfSU1N866Y+rbmyZwtObdOw3H1EpObQ10el/HPGOh58f9ER2+b+34XkFxbx/eemcePprbnvwo6RignA8IkreXrCiiO2jbmrH91a1udQYTEdHzx6kaqsa05tyeJv9rBk057wtjW/uyR8dCMiNZeKQinvzd3AfW/PP2JbNE8pHTVlDac0r0ffExsdsd05x+uzvubjxVu4rHtzuresR4cmGezcd4gLn/mcbXvz6dikDo1qp/LFmu1HPDYlKYEFv72I/IJi6qQlkZhgFBW7I86QMoOvntCpsyI1nYpCKfmFRTwxbhmN66by3MRV3HdhR24/q50PCf1TWFTM4+OWsnjjHl64+VQa1k4pc7+DBUVs33co3M/w2q29yW7TkFopidGMKyLViIrCUTjnMIv/r1RWbMnjomc+D6+3aZTOczf0pFvL+hw4VMTbOeu5Jrsl6Sk16nwDkRpJRUEA+M1/FvHqF0efqW7WA+eTkpjA1rx82mfVUR+ESBxSUZCwgwVFPD52Ka/NOPY0pg3Sk5n9wAUkJepid5F4oqIgZRq/eDMfLNjEvvxCTmlel+GTVn1nnwGnNOWFm8ucGVVEYpSKglTYwYIinp6wgj7tGnLLy6Gf7bRh59Gifq2Ak4lIpGiSHamwtORE7r/kZM7r1IRnrwtdwd3vyUmszt3LhCVbiLUPDiJSdTrtRI4wsGtTfvpWaPnwEOGH6ehBJP7pSEGOkJqUWO5Fff2enMSzn6wIDxUuIvFHfQpSrumrt9G7bUMSE4yrX/iCOet2ApCYYIy9+0w6Na0bcEIRqSj1KchxO+PETJISEzAz3hzah77tQkN0FBU7Bjw7hU+Xbw04oYhEmm9FwczSzGyWmc03s8Vm9nAZ+6Sa2VtmtsrMZppZW7/yyPFJTkzgtVt788x13cPbhrw0m137DwWYSkQizc8jhXzgPOdcd6AHMMDMSg/yfyuw0znXHngG+L2PeeQ4JSUmcGXPlqx98lLO8Ab2GzZ6YcCpRCSSfCsKLmSvt5rs3Up3YFwOvOItvwucbzVhUKI48Prtofr+0eLN3PX6XP748TIKi4oDTiUix8vXPgUzSzSzecBWYIJzbmapXVoA6wGcc4XAbqAREhMOf5X0wYJN/GXyato/8CEFKgwiMc3XouCcK3LO9QBaAr3NrEtVnsfMhppZjpnl5ObmRjakVNmVPVsy7u7+R2zr8MCHPDRmcUCJROR4ReXsI+fcLmAyMKDUXRuBVgBmlgTUA7aX2gfn3EjnXLZzLjsrq2pTZIo/Ojevy9onL2XKL88Nb3t5+lpWbc0LMJWIVJWfZx9lmVl9b7kWcCFQemLiMcBgb/lqYJKLtQsnBIBWDdOZ8+AF3NC7FQAXPP25vkoSiUF+Hik0Ayab2QJgNqE+hQ/M7BEzu8zb50WgkZmtAu4DhvmYR3zWqE4qT1zVjfM6NQZCXyVNX70t4FQiUhm6olkibsPO/Zz5+8nh9ck/P4cTMmsHmEhEdEWzBKZlg3T+c2e/8PpT45cHmEZEKkNFQXzRvVV9vnriEiB0yuqEJVvYsHN/wKlE5FhUFMQ3ZsawgZ0AuP3VHM78/WQOFhQFnEpEjkZFQXx1c582R6zP/GpHQElEpCJUFMRXtVOTWPboABY9fDEAg/8xi4UbdgecSkTKo6IgvktLTqROahI/v6gjAN8fMZX8Qn2NJFIdqShI1Nx1XgfuPr8DANe88EXAaUSkLCoKElX3XRg6WliwYTdth40lNy8/4EQiUpKKgkTdvRd0DC//c8a6AJOISGkqChJ191zQgSWPXEzD2in8eeJKlm3eE3QkEfGoKEgg0lOSuPeCUP/CgGenBJxGRA5TUZDA3Ny3Le28MZHW79DVziLVgYqCBGrkD0Pjc/X/w+Rj7Cki0aCiIIFq37hOeFl9CyLBU1GQwE0fdh4AU1dq7gWRoKkoSOCa169Fi/q1+GyF5t8WCZqKglQLnZpmMGXlNtoOG8uhQk3jKRIUFQWpFu698NsL2jo++CE3jZoZYBqRmktFQaqFLi3qsejhi2laNw2Aqau2kXewIOBUIjWPioJUG3VSk5hx//mc36kxAF0fGh9wIpGaR0VBqp1Rg7+dW3zLnoMBJhGpeVQUpNoxM8bc1Q+AX7+3MOA0IjWLb0XBzFqZ2WQzW2Jmi83snjL2OcfMdpvZPO/2G7/ySGzp1rI+AJOWbWXnvkMBpxGpOfw8UigEfuac6wz0Ae40s85l7DfFOdfDuz3iYx6JMc/d0BOAS4ZPwTkXcBqRmsG3ouCc2+Scm+st5wFLgRZ+vZ7En0u7NgNg0+6DzF67M+A0IjVDVPoUzKwt0BMo6+TzvmY238w+NLNTopFHYkNCgrHkkYsBGDVlTcBpRGoG34uCmdUBRgM/dc6VHvFsLtDGOdcdeA54v5znGGpmOWaWk5uroRBqkvSUJADGL9nCmty9AacRiX++FgUzSyZUEP7lnHuv9P3OuT3Oub3e8jgg2cwyy9hvpHMu2zmXnZWV5WdkqYYevaILAOc99Rm79qvTWcRPfp59ZMCLwFLn3NPl7NPU2w8z6+3l2e5XJolNN/dpQ/8Ooc8Klw6fGnAakfjm55FCP+Bm4LwSp5xeYmZ3mNkd3j5XA4vMbD4wHLje6TQTKcNrt55OreRENu46wDMTVgQdRyRuJfn1xM65qYAdY58RwAi/Mkh8eXJQV+55cx5/nriStpnpXNmzZdCRROKOrmiWmHFZ9+Y8eOnJANz71nwNmCfiAxUFiRlmxm3923HH2ScC8NqMdQEnEok/KgoSc4YN7ETvtg15a/Z6iovVBSUSSSoKEpNuPL0167bvZ/pqnawmEkkqChKTBnRpSv30ZN6Y9XXQUUTiioqCxKS05ESu6NGCsQs3MXHplqDjiMQNFQWJWTf1aQPAra/kBJxEJH6oKEjMat+4Dic3qwugDmeRCFFRkJh2s3e0cP3IGRQWFQecRiT2qShITDv/5MYAzFq7g0uHT+VgQVHAiURim4qCxLQmddO4+7z2ACzfkkfPRyZoljaR46CiIDHvvotOYuFDFwFwoKCIBRt2B5xIJHapKEhcyEhLZuqvzsUM3pmzPug4IjFLRUHiRssG6fQ7MZN563cFHUUkZqkoSFw5qWkGizbuoe2wsdz39ryg44jEHBUFiSsDuzQNL783dyO7D2h4bZHKUFGQuJLdtiGzHjifH/YNXb9w/lOfBZxIJLaoKEjcaZyRxrCBnQDYtjef/EJduyBSUSoKEpfSU5L4282nAvDr9xYGnEYkdqgoSNw656QsAGZozgWRClNRkLiVmpTIj885ka15+ezNLww6jkhMUFGQuNa/fSaFxY6Za3S0IFIRKgoS13q1aQDAb/6zOOAkIrHBt6JgZq3MbLKZLTGzxWZ2Txn7mJkNN7NVZrbAzHr5lUdqprTkRE5qksHGXQc0UJ5IBfh5pFAI/Mw51xnoA9xpZp1L7TMQ6ODdhgJ/9TGP1FA39G4FwJpt+wJOIlL9+VYUnHObnHNzveU8YCnQotRulwOvupAZQH0za+ZXJqmZLu3WnKQE4725G4KOIlLtRaVPwczaAj2BmaXuagGUHNJyA98tHJjZUDPLMbOc3Nxcv2JKnMrKSKVz87r8d/4mfYUkcgy+FwUzqwOMBn7qnNtTledwzo10zmU757KzsrIiG1BqhFOa1+PrHfs17IXIMfhaFMwsmVBB+Jdz7r0ydtkItCqx3tLbJhJR95zfAQj1K2zdczDgNCLVl59nHxnwIrDUOfd0ObuNAX7onYXUB9jtnNvkVyapuZrWS+MfQ7IBmK+Z2UTKleTjc/cDbgYWmtnhge3vB1oDOOdeAMYBlwCrgP3A//iYR2q4vu0yARgxeRUXdm4ScBqR6sm3ouCcmwrYMfZxwJ1+ZRApqVZKIgDz1+9i2qpt9GufGXAikepHVzRLjfLSkNMA+MGomWzbmx9wGpHqp0JFwczuMbO63nf/L5rZXDO7yO9wIpF2bqfGDOrVEoDsxz4JOI1I9VPRI4VbvNNJLwIaEOoreNK3VCI+eura7jSvlwbA6ty9AacRqV4qWhQO9w1cArzmnFvMMfoLRKqz12/vA8BT45cHnESkeqloUZhjZuMJFYWPzSwDKPYvloi/2jRKB2D+ep2eKlJSRYvCrcAw4DTn3H4gGZ0+KjHMzBjctw0bdx2g7bCxFBdr+AsRqHhR6Assd87tMrObgAcBfcSSmHZ4rgWAdvePY8e+QwGmEakeKloU/grsN7PuwM+A1cCrvqUSiYLLujfnhZtODa+/NO2rANOIVA8VLQqF3oVmlwMjnHN/ATL8iyXiPzNjQJemfHhPfwCem7Qq4EQiwatoUcgzs18TOhV1rJklEOpXEIl5JzerS/vGdQB4d47mXJCaraJF4Togn9D1CpsJjWb6R99SiUTZm0NDp6gu21Sl0d1F4kaFioJXCP4F1DOz7wEHnXPqU5C4kVknlSZ1Uxk19Su2aGhtqcEqOszFtcAs4BrgWmCmmV3tZzCRaLuiZ2jSv0uHTwk4iUhwKvr10QOErlEY7Jz7IdAb+D//YolE37ABnaiTmsS2vYfYqdNTpYaqaFFIcM5tLbG+vRKPFYkJZsY/bzsdgI8Xbw44jUgwKvrG/pGZfWxmQ8xsCDCW0AQ5InGle8t6ZNZJZcrKbUFHEQlEhSbZcc79wswGEZpNDWCkc+7f/sUSCYaZcUnXprw5az1b9hykSd20oCOJRFWFvwJyzo12zt3n3VQQJG4N6tWSQ0XF/Hf+N0FHEYm6oxYFM8szsz1l3PLMTCd0S1w6uVldAB4buzTgJCLRd9Svj5xzGspCapyUpG8/K+UdLCAjTRfvS82hM4hEyvDqLb0BmPv1roCTiESXioJIGXq1aUCCwayvtgcdRSSqfCsKZvYPM9tqZovKuf8cM9ttZvO822/8yiJSWXVSk+jSoh5/mbyajbsOBB1HJGr8PFJ4GRhwjH2mOOd6eLdHfMwiUmnf79YcgH5PTqLtsLEUaXY2qQF8KwrOuc+BHX49v4jfBp/Rlj7tGobXT7xf12tK/Au6T6Gvmc03sw/N7JSAs4gcISUpgTeH9mXSz84Ob/tsRW6AiUT8F2RRmAu0cc51B54D3i9vRzMbamY5ZpaTm6s/Somudll1+OAnZwIw+B+zCE1CKBKfAisKzrk9zrm93vI4INnMMsvZd6RzLts5l52VlRXVnCIAXVrU45KuTQGYvHzrMfYWiV2BFQUza2pm5i339rLo/D+ptv5wdXcAbnk5h2178wNOI+IPP09JfQP4AjjJzDaY2a1mdoeZ3eHtcjWwyMzmA8OB652Oy6Uaq5OaFD5ayH7sk4DTiPijQqOkVoVz7oZj3D8CGOHX64v44dnrejJu4YcAfL19P60bpQecSCSygj77SCSmpCQl8PaP+gLw5uyvA04jEnkqCiKV1PuEhvTvkMnzn66msKg46DgiEaWiIFIFg3q1BGDIS7MDTiISWSoKIlVwWffQEBhTV21j0cbdAacRiRwVBZEqSEgw/nbzqQA8+8nKgNOIRI6KgkgVXXxKU3q1rs+KLXlBRxGJGBUFkeNweY8WfL1jP/PWazIeiQ8qCiLH4fyTGwPw5iydnirxQUVB5Di0bJDOOSdl8c6cDezYdyjoOCLHTUVB5Dj97MKTKCp29Hp0gkZQlZinoiBynLq0qBteHj13Y4BJRI6fioLIcTIzFj50EQBTVmq+D4ltKgoiEZCRlsyVPVvw6fJc8guLgo4jUmUqCiIRcnmP5uw+UMDEpZqER2KXioJIhPTvkEXTumnc8+aXFBerw1lik4qCSIQkJhiXdmtGQZFj4jIdLUhsUlEQiaBfDjiJjLQkRs/ZEHQUkSpRURCJoNSkRC7r3pzPVuSSd7Ag6DgilaaiIBJhl3VvzoGCIqat2h50FJFKU1EQibBebRpQOyVR1yxITFJREImw5MQE+rXP5JOlWyjSWUgSY1QURHxwVa8WbNmTr6MFiTm+FQUz+4eZbTWzReXcb2Y23MxWmdkCM+vlVxaRaDu3U2hI7SEvzdYgeRJT/DxSeBkYcJT7BwIdvNtQ4K8+ZhGJqtSkRM49KQvQIHkSW3wrCs65z4EdR9nlcuBVFzIDqG9mzfzKIxJtowafRv30ZH7+znz25RcGHUekQoLsU2gBrC+xvsHbJhIXEhOMu85tD8DMr3R6qsSGmOhoNrOhZpZjZjm5ueq4k9hxU582ZKQmMW7h5qCjiFRIkEVhI9CqxHpLb9t3OOdGOueynXPZWVlZUQknEglpyYlc0rUZHyz4hr36CkliQJBFYQzwQ+8spD7AbufcpgDziPji2tNacrCgmI8X6WhBqj8/T0l9A/gCOMnMNpjZrWZ2h5nd4e0yDlgDrAL+DvyvX1lEgtSrdQNaNazFnyeupKCoOOg4IkeV5NcTO+duOMb9DrjTr9cXqS7MjGtObcXTE1bwzIQV/HJAp6AjiZQrJjqaRWLd4bOQnv90NXO/3hlwGpHyqSiIREFCgvHEVV0BuO+tebrKWaotFQWRKLmhd2tu738Ca7fvZ/E3e4KOI1ImFQWRKBp61onUTklk5Odrgo4iUiYVBZEoyspI5ZrsVny0aDN7NDObVEMqCiJRdknXZhwqKmbysq1BRxH5DhUFkSjLbtOANo3SefWLdUFHEfkOFQWRKEtIMAb3bcucdTv5SFc5SzWjoiASgKuzWwIwYvJKijVlp1QjKgoiAaiblswTV3Vl0cY9vDtnQ9BxRMJUFEQCclWv0PQhvxy9gJ37DgWcRiRERUEkIKlJiYy4sScAPR+dwPa9+QEnElFREAnU97o1Z1CvUP/Cs5+sDDiNVGcPjVnM+MX+n5igoiASsD9c3Y2erevz2ox1TFu1Leg4Ug3l5uXz8vS1vDHra99fS0VBJGCJCcaIG3sB8HoU/ugl9kxYsgWA9BTfZjsIU1EQqQZa1K/FlT1bMHbBJu7/98Kg40g1s/tAaEiUP1zdzffXUlEQqSZ+cl5ozoXXZ34d/mQoAvD1jn00rJ1C7VQdKYjUGO2y6jD6x2cAcPurORTpojbxrM7dR/usOlF5LRUFkWrk1DYN+N9zTgTgXzM1NpKEbN+bT2ZGSlReS0VBpJr5xcUn0b9DJr/5z2JueXk2hUXFQUeSABUVO1bn7mPX/ugMta6iIFLNmBmPXxGaunPSsq1c+fx0jY9Ugy3bHJqlr7AoOv8HVBREqqHWjdJZ9fhA2mXWZuHG3XR7eLzmda6h/uxd1PjEoK5ReT1fi4KZDTCz5Wa2ysyGlXH/EDPLNbN53u02P/OIxJKkxATG3t0fgL35hbyTo4HzapriYsd470y0dpm1o/KavhUFM0sE/gIMBDoDN5hZ5zJ2fcs518O7jfIrj0gsqpWSyJrfXUJmnRR+OXoBL079KuhIEkW9HpsAwEPf74yZReU1/TxS6A2scs6tcc4dAt4ELvfx9UTiUkKCMWrwaQA8+sESdkepw1GCNWHJlnDn8qBTW0btdf0sCi2A9SXWN3jbShtkZgvM7F0za+VjHpGY1aNVfcbefSYA97z1pTqea4D73poHwD9vPZ2MtOSovW7QHc3/Bdo657oBE4BXytrJzIaaWY6Z5eTm5kY1oEh1cUrzelzUuQmfLs+l3f3jmL5qmzqf49Th3+sVPZpzZofMqL62n0VhI1Dyk39Lb1uYc267c+7wIPKjgFPLeiLn3EjnXLZzLjsrK8uXsCKx4Nnre3Ba2wYA3DhqJr1/N5EDh4oCTiWR9uX6XeTlF3JCZnSuYi7Jz6IwG+hgZieYWQpwPTCm5A5m1qzE6mXAUh/ziMS89JQk3rnjDP7oDYyWm5fPyb/5iK17DgacTCLlnZz1XPX8dAC6t6oX9df3rSg45wqBu4CPCb3Zv+2cW2xmj5jZZd5ud5vZYjObD9wNDPErj0g8uSa7FWufvJQLTm4CQO/fTeSL1dsDTiWR8It3F4SXz+4Y/W9GLNa+k8zOznY5OTlBxxCpNoaNXsCbs0PndDSpm8qMX58ftdMXJbIWbNjFZSOmUSc1iYk/O5smddMi9txmNsc5l32s/YLuaBaR4/TkoG588JPQmUlb9uRzwq/H8cGCbwJOJVUxakroOpRXbukd0YJQGSoKInGgS4t6rHp8YHj9rte/ZPLyrQEmkspyzjFmfqiYd20R/b6Ew1QUROJEUmICyx4dwIOXngzA/7w0m6fGLw84lVTU4YIw5Iy2pCQF99asoiASR9KSE7mtfzteuCl0dvdzk1bRdthYfv7OfHbtPxRwOjmaGWtCJwr8/OKTAs2hoiAShwZ0acrUX50bXn93zgZ6PDKBtsPG6vTVEqat2hYemjpob8xaT3abBtSJwpSbR6OiIBKnWjZIZ9mjA3j/zn5cl/3tdaR9n5zEpGWaA/rpCSv4waiZDHh2Chc981mgkxl9+fVOAFo3TA8sw2EqCiJxLC05kR6t6vP7q7ux9slL+dM13XHOccvLObQdNpZFG3cHHTEQU1bmMnziyvD6ii17eXn62kCyrN+xnyu9i9XuvbBjIBlKCvY4RUSi6upTW9K/QyY3/H0Ga3L38b3npobva1o3jWtPa8VPz+9AQkL8XudQXOwYNnoh6SmJfPqLc8iqk8p1I2fw2Nil1EpJ5Aent4lalrELNnHn63MBuPiUJrSqBkcKunhNpIaav34Xvx2zmHnrdx2xPbNOKm8O7cOJWbXj8iK4RRt3873npvL0td25qldoSOqd+w7R89EJ4X0e+n5n+rXPpHHdNOrV+naEUufccf1MVm3NIzkxgfSUJGZ9tSNcEAZ2acpfbypz6LeIqejFayoKIjVcQVExeQcLyVm7g1e+WMu0Vd8dLqN1w3Tuv6QTi78JdcreeuYJ1E9PiXLSyHhr9tf8avRCPvvFObRp9O1sZrv2H+LCZz4nNy//iP3PbJ/Ji0OyOenBj8Lb/jCoG9eeFuqnOVhQxIadB6iblsTyLXk0rJ1Cs3q1eHnaV3y2IpfHr+zKKc3rcvULXzBn3c7v5HlzaB/6tGvkU2u/paIgIpVWXOyYtGwrt7167L+xX1x8Ejf2bs3bOesZ2KUZzeunkZQYvW7K4mLH6ty9dGiSEd72za4DbM3Lp0er+uU+7qnxy3n+09Usf3RAmXnX5O7lV6MXMHvtd9/AS0tJTODQcXRQv3pLb86K0vhGKgoiUmVFxY5i5ygoKiY9JYn563fx7y83MvfrnTTOSOWTpeVfLX36CQ1ZumkPqcmJPP+DXmzfm8+0VdsZ0KUp/dpnsnHXAd6bs4ELT2lCfkExJzXNIC05keJix5JNe2hcN5XGGeUP8VBQVMybs9fzf+8vAkIXez102Snsyy/klN9+DEC7rNqc3TGLu85tT62URF74dDW3n9WOeet38btxy9hzoIBpw8475s+hsKiY/3l5NlNWbuO+Czvyk/Pas2HnAf40fjn/mfftUCIZaUnkHSwkMcEoKjEB0tCz2jHy8zXh9UUPXxzYKacqCiLim137D/HA+4sYu2ATpzSvG/5aKZI6N6vLD/q05rrsVsxbv4ulm/Po3rIeg/46nYKiI9+3fnR2O/722Zpynum7ureqz3/u7Ffh/YuL3Xc63wuKihm/eAv9O2ZSt9TMaMXFjoLiYlKTEgHILywKLwdFRUFEou5QYTFj5n9Dywa1aJCewts562mQnkxaciKPjf2wLAFkAAAJrklEQVR2upQLTm5MxyYZPP/p6u88R+dmdVmy6dhFZtYD59MwPYVbXsnh8xWhGRlvPL01j1/RheVb8vhkyRb+NH4FEDqzarN30V7/Dpn8+OwTOaN9dGc0C5qKgohUOx8t2kzD2in0PqFheJtzjgMFRaSnfPu1ys59h0hMNF6dvpb35m4kKdE4uVldPly4mU7NMnj3jjPC4wMdKixmxKSVbN93iMeu6PKds4PK+pRfE6koiIhImOZTEBGRSlNREBGRMBUFEREJU1EQEZEwFQUREQlTURARkTAVBRERCVNREBGRsJi7eM3McoF1JTbVA3YfZbnktkxgWxVfuuTzVPb+0vdVZr10O/xsw9H2KWt7RdtRVntipR3HWlY71I7ytle2HX6/V7Vxzh17SFbnXEzfgJFHWy61LScSr1PZ+0vfV5n10u3wsw1H26es7RVtRzntiYl2VOD/l9qhdkSkHdF6rzrWLR6+PvrvMZZLbovU61T2/tL3VWY9ku2oyHOUt09Z2yvajvJ+R1UVzXZUZLmq1A61o+RytN6rjirmvj46HmaW4yow9kd1Fg9tALWjulE7qpcg2xEPRwqVMTLoABEQD20AtaO6UTuql8DaUaOOFERE5Ohq2pGCiIgchYqCiIiEqSiIiEiYigJgZv3N7AUzG2Vm04POU1VmlmBmj5vZc2Y2OOg8VWVm55jZFO93ck7QeY6HmdU2sxwz+17QWarKzE72fhfvmtmPg85TVWZ2hZn93czeMrOLgs5TFWbWzsxeNLN3/XqNmC8KZvYPM9tqZotKbR9gZsvNbJWZDTvaczjnpjjn7gA+AF7xM295ItEO4HKgJVAAbPAr69FEqB0O2AukEdvtAPgV8LY/KY8tQn8fS72/j2uBfn7mLU+E2vG+c+524A7gOj/zliVCbVjjnLvV16DHc+VbdbgBZwG9gEUltiUCq4F2QAowH+gMdCX0xl/y1rjE494GMmK1HcAw4EfeY9+N4XYkeI9rAvwrhttxIXA9MAT4Xqy2w3vMZcCHwI2x3A7vcU8BvWK8Db79fScR45xzn5tZ21KbewOrnHNrAMzsTeBy59wTQJmH8WbWGtjtnMvzMW65ItEOM9sAHPJWi/xLW75I/T48O4FUP3IeS4R+H+cAtQn9kR8ws3HOuWI/c5cWqd+Hc24MMMbMxgKv+5e4bBH6fRjwJPChc26uv4m/K8J/G76J+aJQjhbA+hLrG4DTj/GYW4GXfEtUNZVtx3vAc2bWH/jcz2CVVKl2mNlVwMVAfWCEv9EqpVLtcM49AGBmQ4Bt0S4IR1HZ38c5wFWECvQ4X5NVTmX/Pn4CXADUM7P2zrkX/AxXQZX9XTQCHgd6mtmvveIRUfFaFCrNOffboDMcL+fcfkLFLaY5594jVODignPu5aAzHA/n3KfApwHHOG7OueHA8KBzHA/n3HZCfSK+ifmO5nJsBFqVWG/pbYs1akf1onZUL/HQjmrXhngtCrOBDmZ2gpmlEOrsGxNwpqpQO6oXtaN6iYd2VL82BHEmQYR79N8ANvHtaZi3etsvAVYQ6tl/IOicaofaoXbU7HbEShs0IJ6IiITF69dHIiJSBSoKIiISpqIgIiJhKgoiIhKmoiAiImEqCiIiEqaiIL4zs71ReI3LKjiUdSRf8xwzO6MKj+tpZi96y0PMrFqM72RmbUsP61zGPllm9lG0Mkn0qShIzDCzxPLuc86Ncc496cNrHm18sHOAShcF4H5idAwe51wusMnMAplXQfynoiBRZWa/MLPZZrbAzB4usf19M5tjZovNbGiJ7XvN7Ckzmw/0NbO1Zvawmc01s4Vm1snbL/yJ28xeNrPhZjbdzNaY2dXe9gQze97MlpnZBDMbd/i+Uhk/NbNnzSwHuMfMvm9mM83sSzP7xMyaeEMg3wHca2bzLDR7X5aZjfbaN7usN04zywC6Oefml3FfWzOb5P1sJnrDuWNmJ5rZDK+9j5V15GWhGd7Gmtl8M1tkZtd520/zfg7zzWyWmWV4rzPF+xnOLetox8wSzeyPJX5XPypx9/vAD8r8BUvsC/qSat3i/wbs9f69CBgJGKEPJB8AZ3n3NfT+rQUsAhp56w64tsRzrQV+4i3/LzDKWx4CjPCWXwbe8V6jM6Hx6gGuJjT0cwLQlNB8DVeXkfdT4PkS6w0gfPX/bcBT3vJDwM9L7Pc6cKa33BpYWsZznwuMLrFeMvd/gcHe8i3A+97yB8AN3vIdh3+epZ53EPD3Euv1CE3asgY4zdtWl9DIyOlAmretA5DjLbfFmwAGGAo86C2nAjnACd56C2Bh0P+vdPPnpqGzJZou8m5feut1CL0pfQ7cbWZXettbedu3E5osaHSp5zk8rPYcQuP8l+V9F5q/YImZNfG2nQm8423fbGaTj5L1rRLLLYG3zKwZoTfar8p5zAVA59BcLgDUNbM6zrmSn+ybAbnlPL5vifa8BvyhxPYrvOXXgT+V8diFwFNm9nvgA+fcFDPrCmxyzs0GcM7tgdBRBTDCzHoQ+vl2LOP5LgK6lTiSqkfod/IVsBVoXk4bJMapKEg0GfCEc+5vR2wMTeJyAdDXObffzD4lND8zwEHnXOlZ5PK9f4so//9wfollK2efo9lXYvk54Gnn3Bgv60PlPCYB6OOcO3iU5z3At22LGOfcCjPrRWhwtcfMbCLw73J2vxfYAnQnlLmsvEboiOzjMu5LI9QOiUPqU5Bo+hi4xczqAJhZCzNrTOhT6E6vIHQC+vj0+tOAQV7fQhNCHcUVUY9vx7gfXGJ7HpBRYn08odm9APA+iZe2FGhfzutMJzR0MoS+s5/iLc8g9PUQJe4/gpk1B/Y75/4J/JHQXMDLgWZmdpq3T4bXcV6P0BFEMXAzoXmCS/sY+LGZJXuP7egdYUDoyOKoZylJ7FJRkKhxzo0n9PXHF2a2EHiX0JvqR0CSmS0lNIfuDJ8ijCY0ZPES4J/AXGB3BR73EPCOmc0BtpXY/l/gysMdzcDdQLbXMbuEMmbIcs4tIzQdZEbp+wgVlP8xswWE3qzv8bb/FLjP296+nMxdgVlmNg/4LfCYc+4QcB2hKVrnAxMIfcp/HhjsbevEkUdFh40i9HOa652m+je+PSo7FxhbxmMkDmjobKlRDn/Hb6G5bmcB/Zxzm6Oc4V4gzzk3qoL7pwMHnHPOzK4n1Ol8ua8hj57nc0KTy+8MKoP4R30KUtN8YGb1CXUYPxrtguD5K3BNJfY/lVDHsAG7CJ2ZFAgzyyLUv6KCEKd0pCAiImHqUxARkTAVBRERCVNREBGRMBUFEREJU1EQEZEwFQUREQn7fw741Eq6ByuVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.lr_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **n_skip_beginning** and **n_skip_end** arguments to ```lr_plot``` can be used to zoom into the plot, but we have not used it here.  Using the plot, we select the maximal learning rate associated with a still falling loss. In this case, we will choose 1e-3, which happens to be the default learning rate for the Adam optimizer.  This will not always be the case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with Learning Rate Schedules\n",
    "\n",
    "Varying the learning rate cyclically during training has been shown to be [effective](https://arxiv.org/abs/1506.01186) and a good general practice.  *ktrain* allows you to easily employ a variety of demonstrably effective learning rate policies during training. These include:\n",
    "\n",
    "* a [triangular learning rate policy](https://arxiv.org/abs/1506.01186) available via the ```autofit``` method\n",
    "* a [1cycle policy](https://arxiv.org/abs/1803.09820) available via the ```fit_onecycle``` method\n",
    "* an [SGDR](https://arxiv.org/abs/1608.03983) (Stochastic Gradient Descent with Restart) schedule available using the ```fit``` method by supplying a *cycle_len* argument.\n",
    "\n",
    "The ```autofit``` and ```fit_onecycle``` methods tend to be good choices that produce pleasing results.  For more information on learning rate schedules, see the [*ktrain* tutorial notebook on tuning learning rates](https://github.com/amaiya/ktrain/blob/master/tutorials/tutorial-02-tuning-learning-rates.ipynb).\n",
    "\n",
    "The ```autofit``` method in *ktrain* employs a triangular learning rate schedule and uses the supplied learning rate as the maximum learning rate.  The ```autofit``` method accepts two primary arguments. The first (required) is the learning rate (**lr**) to be used, which can be found using the learning rate finder above. The second is optional and indicates the number of epochs (**epochs**) to train.  If **epochs** is not supplied as a second argument, then ```autofit``` will train until the validation loss no longer improves after a certain period. This period can be configured using the **early_stopping** argument.  The ```autofit``` method will also reduce the learning rate when validation loss no longer improves, which can be configured using the reduce_on_plateau argument to ```autofit```.  At the end of training, the weights producing the lowest validation loss are automatically loaded into the model.\n",
    "\n",
    "\n",
    "The ```autofit``` method also accepts a **checkpoint_folder** argument representing the path to a directory. If supplied, the weights of the model after each epoch will be saved to this folder.  Thus, the model state of any epoch can be easily restored using the ```learner.model.load_weights``` method, if the final validation accuracy is not desired. In this case, the final accuracy is quite satisfactory for this model.  \n",
    "\n",
    "As shown below, this learning rate scheme achieves an accuracy of **99.64%** after 10 epochs despite the simplicity of our model. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "begin training using triangular learning rate policy with max lr of 0.001...\n",
      "Epoch 1/10\n",
      "937/937 [==============================] - 17s 18ms/step - loss: 0.5395 - acc: 0.8311 - val_loss: 0.0342 - val_acc: 0.9882\n",
      "Epoch 2/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.1266 - acc: 0.9612 - val_loss: 0.0243 - val_acc: 0.9924\n",
      "Epoch 3/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0924 - acc: 0.9715 - val_loss: 0.0213 - val_acc: 0.9936\n",
      "Epoch 4/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0769 - acc: 0.9769 - val_loss: 0.0171 - val_acc: 0.9942\n",
      "Epoch 5/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0686 - acc: 0.9791 - val_loss: 0.0160 - val_acc: 0.9950\n",
      "Epoch 6/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0632 - acc: 0.9810 - val_loss: 0.0137 - val_acc: 0.9952\n",
      "Epoch 7/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0535 - acc: 0.9830 - val_loss: 0.0128 - val_acc: 0.9954\n",
      "Epoch 8/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0543 - acc: 0.9835 - val_loss: 0.0140 - val_acc: 0.9955\n",
      "Epoch 9/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0481 - acc: 0.9849 - val_loss: 0.0128 - val_acc: 0.9959\n",
      "Epoch 10/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0438 - acc: 0.9866 - val_loss: 0.0126 - val_acc: 0.9964\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f79843fcfd0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.autofit(0.001, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will demonstrate training using the ```fit_onecycle``` method, where similar results are achieved after 10  epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "begin training using onecycle policy with max lr of 0.001...\n",
      "Epoch 1/10\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.7958 - acc: 0.7482 - val_loss: 0.0804 - val_acc: 0.9739\n",
      "Epoch 2/10\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.1713 - acc: 0.9479 - val_loss: 0.0363 - val_acc: 0.9876\n",
      "Epoch 3/10\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.1110 - acc: 0.9664 - val_loss: 0.0327 - val_acc: 0.9893\n",
      "Epoch 4/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0889 - acc: 0.9726 - val_loss: 0.0228 - val_acc: 0.9919\n",
      "Epoch 5/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0750 - acc: 0.9772 - val_loss: 0.0289 - val_acc: 0.9915\n",
      "Epoch 6/10\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0668 - acc: 0.9790 - val_loss: 0.0232 - val_acc: 0.9925\n",
      "Epoch 7/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0535 - acc: 0.9842 - val_loss: 0.0159 - val_acc: 0.9941\n",
      "Epoch 8/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0436 - acc: 0.9864 - val_loss: 0.0181 - val_acc: 0.9946\n",
      "Epoch 9/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0399 - acc: 0.9877 - val_loss: 0.0143 - val_acc: 0.9951\n",
      "Epoch 10/10\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0315 - acc: 0.9906 - val_loss: 0.0123 - val_acc: 0.9963\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f3bf463ac18>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.reset_weights(verbose=0)\n",
    "learner.fit_onecycle(0.001, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we invoke ```autofit``` **without** supplying the number of epochs. As mentioned, the training will automatically stop when the validation loss fails to improve. Moreover, the maximum learning rate will automatically decrease periodically. We supply the *checkpoint_folder* argument to ensure we can restore the weights from any epoch after training completes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "early_stopping automatically enabled at patience=5\n",
      "reduce_on_plateau automatically enabled at patience=2\n",
      "\n",
      "\n",
      "begin training using triangular learning rate policy with max lr of 0.001...\n",
      "Epoch 1/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.3721 - acc: 0.8839 - val_loss: 0.0337 - val_acc: 0.9892\n",
      "Epoch 2/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.1040 - acc: 0.9680 - val_loss: 0.0230 - val_acc: 0.9928\n",
      "Epoch 3/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0791 - acc: 0.9764 - val_loss: 0.0200 - val_acc: 0.9942\n",
      "Epoch 4/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0689 - acc: 0.9790 - val_loss: 0.0169 - val_acc: 0.9947\n",
      "Epoch 5/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0608 - acc: 0.9820 - val_loss: 0.0153 - val_acc: 0.9948\n",
      "Epoch 6/1024\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0550 - acc: 0.9832 - val_loss: 0.0160 - val_acc: 0.9952\n",
      "Epoch 7/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0532 - acc: 0.9838 - val_loss: 0.0132 - val_acc: 0.9947\n",
      "Epoch 8/1024\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0473 - acc: 0.9854 - val_loss: 0.0126 - val_acc: 0.9957\n",
      "Epoch 9/1024\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0477 - acc: 0.9860 - val_loss: 0.0141 - val_acc: 0.9947\n",
      "Epoch 10/1024\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0424 - acc: 0.9877 - val_loss: 0.0138 - val_acc: 0.9949\n",
      "\n",
      "Epoch 00010: Reducing Max LR on Plateau: new max lr will be 0.0005 (if not early_stopping).\n",
      "Epoch 11/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0351 - acc: 0.9894 - val_loss: 0.0131 - val_acc: 0.9956\n",
      "Epoch 12/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0328 - acc: 0.9898 - val_loss: 0.0133 - val_acc: 0.9953\n",
      "\n",
      "Epoch 00012: Reducing Max LR on Plateau: new max lr will be 0.00025 (if not early_stopping).\n",
      "Epoch 13/1024\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0302 - acc: 0.9907 - val_loss: 0.0122 - val_acc: 0.9961\n",
      "Epoch 14/1024\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0290 - acc: 0.9912 - val_loss: 0.0141 - val_acc: 0.9954\n",
      "Epoch 15/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0280 - acc: 0.9911 - val_loss: 0.0121 - val_acc: 0.9959\n",
      "Epoch 16/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0281 - acc: 0.9917 - val_loss: 0.0126 - val_acc: 0.9962\n",
      "Epoch 17/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0258 - acc: 0.9920 - val_loss: 0.0115 - val_acc: 0.9961\n",
      "Epoch 18/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0257 - acc: 0.9921 - val_loss: 0.0113 - val_acc: 0.9964\n",
      "Epoch 19/1024\n",
      "937/937 [==============================] - 15s 17ms/step - loss: 0.0258 - acc: 0.9920 - val_loss: 0.0115 - val_acc: 0.9963\n",
      "Epoch 20/1024\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0227 - acc: 0.9930 - val_loss: 0.0119 - val_acc: 0.9961\n",
      "\n",
      "Epoch 00020: Reducing Max LR on Plateau: new max lr will be 0.000125 (if not early_stopping).\n",
      "Epoch 21/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0218 - acc: 0.9935 - val_loss: 0.0127 - val_acc: 0.9958\n",
      "Epoch 22/1024\n",
      "937/937 [==============================] - 15s 16ms/step - loss: 0.0225 - acc: 0.9931 - val_loss: 0.0116 - val_acc: 0.9960\n",
      "\n",
      "Epoch 00022: Reducing Max LR on Plateau: new max lr will be 6.25e-05 (if not early_stopping).\n",
      "Epoch 23/1024\n",
      "937/937 [==============================] - 16s 17ms/step - loss: 0.0200 - acc: 0.9938 - val_loss: 0.0116 - val_acc: 0.9959\n",
      "Restoring model weights from the end of the best epoch\n",
      "Epoch 00023: early stopping\n",
      "Weights from best epoch have been loaded into model.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f3f30070208>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.reset_weights(verbose=0)\n",
    "learner.autofit(0.001, checkpoint_folder='/tmp/mnist')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After training completes, weights producing the lowest validation loss are loaded into the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final loss:0.011306069766094152, final score:0.9964\n"
     ]
    }
   ],
   "source": [
    "loss, acc = learner.model.evaluate_generator(val_data, steps=len(val_data))\n",
    "print('final loss:%s, final score:%s' % (loss, acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the loweset validation loss produces the highest accuracy (**99.64%**).  If this is not the case, then weights producing the highest accuracy can be re-loaded into the model from the checkpoint folder:\n",
    "\n",
    "```\n",
    "# loading weights from the end of 18th epoch\n",
    "learner.model.load_weights('weights-18.hdf5')\n",
    "```\n",
    "\n",
    "Note that, in the demonstrations above, the ```fit_onecycle```,and ```autofit``` methods produced similar validation accuracies in a similar number of epochs.  This may not always be the case. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspecting Misclassifications\n",
    "\n",
    "With a trained model in hand, let's view the top 3 most misclassified examples in our validation set. This can be done using the ```learner.view_top_losses``` method, which displays the validation examples with the highest loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEICAYAAAAkx4P5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAHEdJREFUeJzt3XmcHWWd7/HPlySEJQkEg5kQdmQRFYJGwTuojAhIfDmAcBm4yCs4anDBZcQZEWcGHEQZxg2XQYMgeFEQL6uICARRUbbAMOwiIoGELGSSEEAIIfndP56nsXLSp04n6e46/eT7fr3Oq0/Xr5bfearOr6qeqq5WRGBmVqoNmk7AzGwguciZWdFc5MysaC5yZlY0FzkzK5qLnJkVbY2LnKTzJX1hIJIZSJK2lfSspGFN59JtJJ0s6XtN57G+krSlpIckbdx0LkOFpEslHdyXcQf0SE7STZI+sBbT/ULSgf2ZS0Q8HhGjImJFh2UfJ+nm/lz22pK0oaSFkkatbVv2Ms/9JM2uDouIL0bEOs+7soytqsuQdJSkByU9J+mPkt7SMv5nJX1R0u6SZkpanF83SNq9l/lvmOdXXcYukq6U9JSkRXkb2rUSnyrpTklLJc2WdKak4f31mdfRScD5EfE81H9vJI2U9CVJj0t6XtIfJP2jJFXGeY2k63I7LMmfe0olfrKkP+Wd/mxJP+5ropJeLelGSU9LekTSYTXj9qnNJe0s6QVJF1aGvUvSzTn/eZK+J2l0ZbJ/B/p0sNV1p6uSNgUmA79qOpcu8Fbg7oh4tulE1tAU4FoASQeQNsj3AaNJn+nRlvHfBVwDPAkcAWwBjAOuAi7uZf7/CDzVMmzzPP6uwHjgduDKSnwT4JN5vnsD+wOfXpsP158kjQSmAhd2Gjf7CSn3KaT2PBaYBpxVGeenwPXAXwGvBD4OLM3Lm5qneUdEjCJ912b0MdfhpDa9mrSOpgEXStqlzSR9bfNvA3e0DNuMVMS2Al4NTAT+oycYEbcDYyRN7ph4RNS+gL2Au4BngB+TNrov5NjY/IGfAhbn91vn2OnACuAF4FngW3n4WcATpEa/E3hLy/L+Frgqvz+VtFIvzMu/F9gF+CywIM/nwMq0NwGnAb/N418HjMux7YEAhuffjyN92Z4B/gQckxvzhZz3s8CSDm1zPvAd0gb1DKkwb1eJB/Ah4A/AEtLKVI4NA74CLMzLP6GaXx7nq8Cnatpyt7zsRcDvgSMr004BHsh5zSFtXJsCzwMr83yeJW1EpwIXtrTTVODxnN/nKvPdGLggr+8HgX8CZre0y2XAe/L73wHvr2nDsXldDmsZPhz4KPDnluE75OUe3LrclvG2yJ/jFW3inwJ+2ibW0wbTSIV3LvDpSvxU4BLgB7l97wcmV+KvB/4rx35C+t58oc2y3go80jLsJuADvYy7f94GtmkZvnfePl5FKigBbN5med8Cvt7pe99m2tfmbUaVYdcBp/Vx+tXaHDgqt+XL22Cbad8D3Nsy7BzglI7L7ZDUhsAs4B+AEaS97HL+UuReARxOqtij8wq9om5lAe/N0w0HTgTmARtV4t8Bjq9sTC8AB+Xxf0AqCJ/L+XwQ+FPL8v5IKoQb59/PaNlwh5O+7EuBXXNsAvCa/P444OY+rrTz84b8VmAkqYDfXIkHqfBvDmxL2hm8M8c+RCpCW5O+6DewepF7qJLjKm2ZP8MTpCOk4aSd0UJg9xyfS96B5Pm/Pr/fj9WL0ssbWKWdzsltuCewDHh1jp9BKuZjc+73VOeX18vCvD0MA14knY49Aswmfck2btnIL2rJZwnwEqkY/3NL7GrgsN4+R8t4hwJza+JX9GwbvcR62uCi3M6vy+vuHS3b5ZT8Gb8E3NrynflEbov35DZoV+Q+CvysZdgq67oy/AzgV23mMws4HhBpp3p1boPxvXz/FpGOhiez+s7lJODqNsvorchdD1zex+/LKm0OjAEeztvRy9tgm2m/DlzcMuxTwGUdl9shqbeS9mTVD/W7mhU2CVjcaWW1TLMY2LPy++PkPVX+4NdXYu/OjTws/z6ayl4rL++fK+N/BLi2ZcPtKXJLSAV645Z8jmPNitzFld9HkfaoPfkHsG8lfglwUn5/I7mY59/fwapHmjtR2cO3tiXwd8BvWvL5LnnPltvxeGBMyzj70bcit3UlfjtwVH7/KHBQJfYBVi1y+wMz8vut8rxmknYk40hH2adXxv+/wLG9tO2mef29qzLsMODn7T5HZbytSUevR7eJ/z2p4I5rE+9pg90qw84Ezq201w2V2O7A85XvzBxW/c7cTPvvzOdY/cu7yrquDP9e67iV2K3kI+78+b9F2uGvBH4N7FwZ9xjSTvU54H+Az/Rxex+R1/8/5fcHkgr4L/ow7WptTjoo+EzrNtjLtAeQ6sQuLcM/CNzYadmd+uS2AuZEnmM2q+eNpE0kfVfSLElLc2NuXncFU9Knc6fx05KWkM69x+XY64CnI+KJyiTzK++fBxbGXy4ePJ9/jqqMM6/y/s8tMQAi4jlSkfgQMFfSzyTt1i7nDl7ONVLf2SJSu3XKZ6vqtC3vIR0l/LxmudsBe+eO2SW5LY8h9cNAKuBTgFmSfiXpzX38PP2R9zX5fc/6+WZEzI2IhaRT8CkAkjYgbcDXti48r6PvAD+Q9MrcV3smqX+pLUlbkk6h/jMiLuolfijpyOvgnE+d6mebRf163Sj3WfX2nWlto6rFpJ11Xywk7Sx6MyHHiYjZEXFCROxE2k6eI50FkeM/jIh3kM4wPgScJumgTguPiOWko8N3kT7/iaQd9+y66Xprc0mTSDv2r3WYdh/gR8AREfFwS3g06WClVqciNxeYWL1yQzrt6nEiqaN374gYQ9qLQTpkhrQ3rCb8FtJe4EhgbERsDjxdGb/6BRlQEfGLiDiAtHE8RDo9Wy3nPtim542kUaS+oCf7MN1c0h53tflkrW3RmtcTpFOXzSuvURHxYYCIuCMiDiF1PF9B2hh7m8+a6nPeEbGY9AWoLrP6/o3ArIhovYjQYwNSV8hEYGfSEdZvJM0j9ftNyFfetgeQNJZU4K6KiNNbZybpnaT1/O6IuLfTB235bNvS9/Xa+p1pbaOqe0jdK31xA2nHtsr8JO2dl3Fj6wT5gOHbpFPN1tjyiPhJzmG1eG8i4p6IeFtEvCIiDgJ2JB3p96qmzfcjrc/H8/r8NHC4pLsq0+5Fupj09xHR28WRVwP/3SnnTkXuFlLfyMcljZD0HuBNlfho0t56iaQtgFNapp9PaoTq+C+R+jeGS/pX0nl5jynAzzolva4kjZd0SD46WEY6BV5ZyXlrSRv2cXZTJO2bxz+N1DdTt+fucQnwCUkTJW0OfKaS3yakdv5lZfzWtrwa2EXSsXndjJD0xnyJf0NJx0jaLO99l7Z8vldI2qyPn6+3vD8raaykiaQLJj157wCMjIgHK+N/H/hYPhobS+rfvTrHVlnfkg6QtJekYZLGkI76ei5w3Ef6Ik/Krw/kzzIJeCKP/wvgtxFxUmvSkt4O/BA4PNKVub74l3y28hpS32dfbrW4hdRlcYKk4ZIOYdXvTKvbSWc/E1uGD5e0UeU1IiJuIF0JvTTfJjIsH+lcCJwdEX/I6+Xzkl4laQNJ40inirfmdjgu354xOscPBl4D3NaXBpG0R85nE0mfJh0knN9m3Lo2n07qkulZn98hbQsH5WlfSzrC/1hE/LRNOm+j/mwH6FDkIuJFUsfpcaTTsL8j7UF7fJ3UOb2Q1Iitpx1nAUco3fP0DdJGeC2ps3EWqfP2ifyhNif1bfyuU9L9YANSp+WTpM/1NuDDOXYj6WrZPEmdTmcgHUqfkufzBlLHbl+cQzrquId0Je4a0g5gBfB24JaIeKEy/iptGRHPkPpEjsqfYx7pVo2RefxjgcdyN8KHSKeyRMRDpA71R/NpbvUUrC/+jXR09ifSkcX/I+0o4C+3glSdRro94GFSsfov0tXi3sbfPOf2NKk/aSfShZoXIuKliJjX8yK198r8+wpSf90bgfcp3f/V8+o58/gXUtfINZXYy18QST+XdHJL7r8iXTCZAXw5Iq7r1DiV78z7SadS7yUV9WU145/P6tvN2aQDiJ7X9/Pww0k7v2tJO+cLgXOBj+X4i6QjpBtIO7f78rKPy/GlwMmkPtslpC6AD0fEzbkdTq62Sy+OJR2tLiD1vx4QEcvytNv2tc0j4s8t6/NZ4IXKUf2JwJbAuZVp7+9JQtIbgWf7tMPqS4fjYLxIp7CXNJ3HGuZ8Pm06lNdiXgeTTt0A/hP4SNOfr495f5h8xY9UsKb0cbrxtFzU6pYXLbcb9cP8bgPeVxPfktRlsnF/LG99eAGX9nVb66abgZfQoROyJJI2ljQln9JMJB0NXp7Dd1fedxVJEyT9dT7V2ZW0x+3J9SZWPcWusxlwYuQttiSS3ibpr/K6nQrsQS8XV3pExFMRsVvkv3iwziLi8IjoU/99t/xZC9GHU4Em5EPk7XoJHb+uswY+T+rneZ7UH/GvABExfR3nPZA2JN2qsgNpx3Qx6ciTiDizrzOJdKWs9WpZKXYl9V1uSrrl4oiImNtsSusvFbgjNTN7WTedrpqZ9buuOV1dWxtqZGzEpk2nYVasF3iOF2OZOo/ZnbquyOWbB88i/U3g9yLijLrxN2JT9tb+g5Kb2frotl7vwx06uup0VenPwb5Nup1id+Bo9fI8MTOzvuqqIke6M/yRiHg00k2SFwOHNJyTmQ1h3VbkJrLqHzPPzsNWIWma0hNkZy7v/UZyMzOg+4pcn0TE9IiYHBGTR7z8V0xmZqvrtiI3h1Wf2NDzXDAzs7XSbUXuDmBnSTvkp3ocRXrUipnZWumqW0gi4iVJJ5CeVjIMOC8i7u8wmZlZW11V5ADyH90OyoMzzax83Xa6ambWr1zkzKxoLnJmVjQXOTMrmoucmRXNRc7MitZ1t5DY0LfyLXvVxk+74Hu18X/43Edr42MuunWNc7L1l4/kzKxoLnJmVjQXOTMrmoucmRXNRc7MiuYiZ2ZF8y0ktub22aM2/Ojx9ZPvNXJlbfyU075fGz9py/e3jY3/xu/qF27rHR/JmVnRXOTMrGgucmZWNBc5Myuai5yZFc1FzsyK5iJnZkXzfXK2xh6etmF9/G++Wxuvv0sOXj9yUf0I0WEGZhU+kjOzornImVnRXOTMrGgucmZWNBc5Myuai5yZFc1FzsyK5vvk1ldSbXj41hPbxo57w8A+s+3Jl+o3yzGPvzSgy7eydF2Rk/QY8AywAngpIiY3m5GZDWVdV+Syv4mIhU0nYWZDn/vkzKxo3VjkArhO0p2SpvU2gqRpkmZKmrmcZYOcnpkNJd14urpvRMyR9ErgekkPRcSvqyNExHRgOsAYbeE/1zaztrruSC4i5uSfC4DLgTc1m5GZDWVdVeQkbSppdM974EDgvmazMrOhrNtOV8cDlyvdwzUc+FFEXNtsSmVaevTetfGb/uOb6zD3ddt33vr8jrXxja+8fZ3mb+uXripyEfEosGfTeZhZObrqdNXMrL+5yJlZ0VzkzKxoLnJmVjQXOTMrWlddXbX+M+8T/6s2/u8fO3eQMllz22z4P7Xxpf/n3W1jY350a3+nY0Ocj+TMrGgucmZWNBc5Myuai5yZFc1FzsyK5iJnZkVzkTOzovk+uaGqw78UHD1nRX18g+f7M5t+9Y6Nn6mP1zwG6nV7fLx22h1Puas2Hsv8OP3S+EjOzIrmImdmRXORM7OiuciZWdFc5MysaC5yZlY0FzkzK5oihvY/oB+jLWJv7d90GoOvw31yV81u7t/2bdBh37mSlY0te89bptbGtzmzvl25/d41TWnIuy1msDQWdWiY7uUjOTMrmoucmRXNRc7MiuYiZ2ZFc5Ezs6K5yJlZ0VzkzKxofp5cl1p83Jtr47sdf39tvNP9YgNphIbVxpcP4K2ZnZb932++oH4Gl9eH9zz7Y21j23zhd/UTWyMa+SZIOk/SAkn3VYZtIel6SX/IP8c2kZuZlaWp3f35wDtbhp0EzIiInYEZ+Xczs3XSSJGLiF8Di1oGHwL0nEtcABw6qEmZWZG6qU9ufETMze/nAePbjShpGjANYCM2GYTUzGyo6sqrq5GeGtC2ezoipkfE5IiYPIKRg5iZmQ013VTk5kuaAJB/Lmg4HzMrQDcVuauAnufgTAWubDAXMytEI8+Tk3QRsB8wDpgPnAJcAVwCbAvMAo6MiNaLE6sp9Xlyw365VW388l2a2wd8Y/FutfEZC+rjT169XW18jyMeqI2fu931bWMD/Sy7mcva34f3bzu+fp3m3a2G+vPkGrnwEBFHtwmVV63MrFHddLpqZtbvXOTMrGgucmZWNBc5Myuai5yZFa2b/qxrvTNs883axsZt9OwgZrJmvn/xQbXxTo8cmsCc2vii89q3C8BBl/zvtrHTX3VZ7bST1/EPZCaPXNE29qcz6h+PteMpd9XGY9mytcrJ6vlIzsyK5iJnZkVzkTOzornImVnRXOTMrGgucmZWNBc5Myua75Nr0OIpr24bu3zbbw5iJt1lxZKna+Mrv71r29ipi95fO+0fP1j/xKAH9/9ubbzO/cd+qzb+mjihNr7DZ29Z62Vbez6SM7OiuciZWdFc5MysaC5yZlY0FzkzK5qLnJkVzUXOzIrWyL8k7E9D+V8SHv3Qk21jx4yeO6DLfvfv/7Y2Hm+vf+bbUFXX5rBu7T5C7f9dIcBXF+1YG7/hyMm18RUPPLzGOfWHof4vCX0kZ2ZFc5Ezs6K5yJlZ0VzkzKxoLnJmVjQXOTMrmoucmRXNz5Nr0Ipov49ZycoBXXbEkL3taZ189/PvqY3vePp3auN7j1zeNra8wy2nK1g/27xpjRzJSTpP0gJJ91WGnSppjqS782tKE7mZWVmaOl09H3hnL8O/FhGT8uuaQc7JzArUSJGLiF8Di5pYtpmtX7rtwsMJku7Jp7Nj240kaZqkmZJmLmfZYOZnZkNMNxW5s4GdgEnAXOAr7UaMiOkRMTkiJo9g5GDlZ2ZDUNcUuYiYHxErImIlcA7wpqZzMrOhr2uKnKQJlV8PA+5rN66ZWV81cp+cpIuA/YBxkmYDpwD7SZoEBPAYcHwTua0vPrfD1bXxD196TNvYdh95qnbaFfMXrFVOg2HMRbfWxk/Y6iO18Ts+ddZaL/vjYx+qjV++xwG18dEPrPWi12uNFLmIOLqXwecOeiJmVryuOV01MxsILnJmVjQXOTMrmoucmRXNRc7MiuZHLTXo9BsPaRt776HfGtBl1z0yCOCufc5vG3v/5fW3Ovz292+ojW/y8Lr9lcqfd36xfVD1zzva5or6fxt48rSL1iYl62I+kjOzornImVnRXOTMrGgucmZWNBc5Myuai5yZFc1FzsyK5vvkGjRi3PNtYxt08f7n+9vNqI2P2P6m2vjyA1f0YzYty1b9fXD7jD+iNn74qIUdltB+vXRadqd/Wbie/pfIAde93yQzs37gImdmRXORM7OiuciZWdFc5MysaC5yZlY0FzkzK5rvk2vQ1tNHtI3d8sb6e646PQ+uSZ3uB1vJysaW/as9658Xty6ZXfLsFrXxc594S218k/ndu06HMh/JmVnRXOTMrGgucmZWNBc5Myuai5yZFc1FzsyK5iJnZkVTRIcbiwZiodI2wA+A8UAA0yPiLElbAD8GtgceA46MiMV18xqjLWJv7T+wCTdg6dH71Man/stPa+Pv2+yxfsxmzXR6Ft5A3ie3rsv+4sJJtfHrv9T+XrfRjz5XOy2331sf71K3xQyWxqIh+7S7po7kXgJOjIjdgX2Aj0raHTgJmBEROwMz8u9mZmutkSIXEXMj4q78/hngQWAicAhwQR7tAuDQJvIzs3I03icnaXtgL+A2YHxEzM2heaTTWTOztdZokZM0CrgU+GRELK3GInUW9tphKGmapJmSZi5n2SBkamZDVWNFTtIIUoH7YURclgfPlzQhxycAC3qbNiKmR8TkiJg8gpGDk7CZDUmNFDlJAs4FHoyIr1ZCVwFT8/upwJWDnZuZlaWpW0j2BX4D3Mtfnm5zMqlf7hJgW2AW6RaSRXXzKvUWko5Uf0V/8dWvqo3/ZtKP+jObVazrbRy3LWv/CCqA6fP2a79s1c975s9eWxvf/rL6f0m44oGHa+MlGuq3kDTyPLmIuBlo12jrYcUys4HS+NVVM7OB5CJnZkVzkTOzornImVnRXOTMrGgucmZWtEbuk+tP6+19cmaDZKjfJ+cjOTMrmoucmRXNRc7MiuYiZ2ZFc5Ezs6K5yJlZ0VzkzKxoLnJmVjQXOTMrmoucmRXNRc7MiuYiZ2ZFc5Ezs6K5yJlZ0VzkzKxoLnJmVjQXOTMrmoucmRXNRc7MiuYiZ2ZFc5Ezs6K5yJlZ0VzkzKxojRQ5SdtI+qWkByTdL+kTefipkuZIuju/pjSRn5mVY3hDy30JODEi7pI0GrhT0vU59rWI+HJDeZlZYRopchExF5ib3z8j6UFgYhO5mFnZGu+Tk7Q9sBdwWx50gqR7JJ0naWybaaZJmilp5nKWDVKmZjYUNVrkJI0CLgU+GRFLgbOBnYBJpCO9r/Q2XURMj4jJETF5BCMHLV8zG3oaK3KSRpAK3A8j4jKAiJgfESsiYiVwDvCmpvIzszI0dXVVwLnAgxHx1crwCZXRDgPuG+zczKwsTV1d/WvgWOBeSXfnYScDR0uaBATwGHB8M+mZWSmaurp6M6BeQtcMdi5mVrbGr66amQ0kFzkzK5qLnJkVzUXOzIrmImdmRXORM7OiuciZWdFc5MysaC5yZlY0FzkzK5qLnJkVzUXOzIrmImdmRXORM7OiKSKazmGdSHoKmFUZNA5Y2FA6nXRrbt2aFzi3tdWfuW0XEVv207wG3ZAvcq0kzYyIyU3n0Ztuza1b8wLntra6ObfB5tNVMyuai5yZFa3EIje96QRqdGtu3ZoXOLe11c25Dari+uTMzKpKPJIzM3uZi5yZFa2YIifpnZJ+L+kRSSc1nU+VpMck3SvpbkkzG87lPEkLJN1XGbaFpOsl/SH/HNtFuZ0qaU5uu7slTWkgr20k/VLSA5Lul/SJPLzxdqvJrfF26xZF9MlJGgY8DBwAzAbuAI6OiAcaTSyT9BgwOSIav3FU0luBZ4EfRMRr87AzgUURcUbeQYyNiM90SW6nAs9GxJcHO59KXhOACRFxl6TRwJ3AocBxNNxuNbkdScPt1i1KOZJ7E/BIRDwaES8CFwOHNJxTV4qIXwOLWgYfAlyQ319A+pIMuja5NS4i5kbEXfn9M8CDwES6oN1qcrOslCI3EXii8vtsumtFB3CdpDslTWs6mV6Mj4i5+f08YHyTyfTiBEn35NPZRk6le0jaHtgLuI0ua7eW3KCL2q1JpRS5brdvRLweOBj4aD4t60qR+i+6qQ/jbGAnYBIwF/hKU4lIGgVcCnwyIpZWY023Wy+5dU27Na2UIjcH2Kby+9Z5WFeIiDn55wLgctLpdTeZn/t2evp4FjScz8siYn5ErIiIlcA5NNR2kkaQisgPI+KyPLgr2q233Lql3bpBKUXuDmBnSTtI2hA4Criq4ZwAkLRp7hBG0qbAgcB99VMNuquAqfn9VODKBnNZRU8RyQ6jgbaTJOBc4MGI+Gol1Hi7tcutG9qtWxRxdRUgXyL/OjAMOC8iTm84JQAk7Ug6egMYDvyoydwkXQTsR3oUz3zgFOAK4BJgW9Jjq46MiEG/ANAmt/1Ip1wBPAYcX+kHG6y89gV+A9wLrMyDTyb1fTXabjW5HU3D7dYtiilyZma9KeV01cysVy5yZlY0FzkzK5qLnJkVzUXOzIrmImdmRXORM7Oi/X9xlJcwwjDguwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEICAYAAAAkx4P5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAHONJREFUeJzt3Xm4HVWZ7/HvjwwESMJgMIRAABEhoAzeNGgzenHAdCuK3bQ02sG+GFBzWwWHGAfwCsq1RbRlaKNEcGimVpFWVBBBBJUQaG4AEwQhAUIIhASSIIQM7/1jrQN1ds6uc8hwap/F7/M8+zl711vDu1fVfqtqVZ29FRGYmZVqs6YTMDPblFzkzKxoLnJmVjQXOTMrmoucmRXNRc7Mivaii5ykiySdsSmS2ZQkjZO0QtKgpnPpNJKmSfp203m8VEnaXtJcSVs0nctAIelsSR/oy7ib9EhO0g2STlyP6X4p6c0bM5eIeDAihkfEml6WfYKkmzbmsteXpKGSFksavr5t2cM8j5D0cHVYRHwxIjZ43pVl7Ni1jJz3s3kHs0LSPT2M/01Jk5V8WtKDkpZJulTSyMp4d1fms0LSakn/lWOvkvQTSY9LWpK3oT0r024u6RxJj0haKul8SUM21nveQFOBiyLiGaj/3OT38aXcRs9IulfSxyWpMs4+kq7J7fCkpNskTazEp0l6ILfhw5IuezHJSnq3pDmSnpb0Z0mH1ox3j6SnJD0m6eKu9Znfx4WS5ktaLukOSW9tmX7LvJ4W53ncWAl/BZgmaWhv+Xbc6aqkrYAJwG+azqUDHAbcERErmk7kRZoI/KLyekrewQyPiD17GP+twNXAPwHvBQ4GdgS2AL7RNVJE7NM1H2AE8BBwRQ5vA1wF7AmMBmYCP6ksYyppu3o18CrgtcBnNvB9bjBJmwOTgO/3cZIrgCNJbTyC1F6Tga9Xxvkv4FpgB+DlwL8Ay/LyJuVp3pjbcQJw3YvI903A/wXel5d/GHB/m9FvBg6OiK2BVwCDga6zwMGk9Xc4sDVpXVwuadfK9NOB7YDx+e9HuwIRsRCYC7y916QjovYBHADcDiwHLgMuBc7IsW2BnwKPA0vz851y7ExgDfAssAI4Nw//en5zy4DbgENblvd24Kr8/HTSSv1+Xv6dpA30U8BjeT5vrkx7A/CF3LjLgWuAUTm2KxDA4Pz6BNLKWQ48AByfG/PZnPcK4Mle2uYi4N9JG9RyUmHepRIP4GTgXuBJ4DxAOTYIOBtYnJc/pZpfHuerwCk1bblXXvYS4B7g2Mq0E4E/5rwWAB8DtgKeAdbm+awgFZPTge+3tNMk4MGc36cr890CuDiv7znAJ4CHW9rlR8AxlXVyYk0b7gvMzs//E/h4JfbX+T1v2cN0h+f3tlWb+W6X38fL8utZwN9X4v8IPNRm2q42mAw8AiwEPlaJnw5cDnw353A3MKESfy3w3zl2Belzc0abZR0G3NcyrMc2IxW3Z4GdW4YflLePVwKjcu7btFneucDXevvc16yv3wH/az2mG57b6+qacWYD76ps28uAkTXjfxr4Tq/L7iWxocB8UgUdAvwdsIoXitzLgHcBW5Kq+hXAlXUrC3hPnm4wcCrwKDCsEv934KTKxvQs8JY8/ndJBeHTOZ/3Aw+0LO/PpEK4RX59VsuGO5j0YV8G7JljY4B98vMTgJv6uOIuyhvyYcDmpAJ+UyUepMK/DTCOtDM4KsdOJhWhnUg7i1+xbpGbW8mxW1vm9/AQaY86mLQzWgzsneMLyTuQPP/X5udHsG5ROp11i9y3chvuB6wExuf4WaRivm3OfXZ1fnm9LAZGVPJ+PA+7GTiiZdlTgS/l5/8JfKISOzjnsl8PbT+DdIrXbt28A1hYeT2L7juB4/O8t+5h2q42uCS382vye3hjy3Y5kbSz+hLwh5bPzIdzWxwDPEf7Ivch4Gctw7qt68rws4DftJnPfOAkQKSd6k9zG4zu4fO3BPg46ShuUA/r46dtljEov5epwH3Aw6SiuUXNejgEeCq359NUDkpaxhud23Sv/PqfSAc15+Rt505yAaxMcwxw+4YWucNIezJVhv2uZoXtDyztbWW1TLO0uhGTjh52rmxM11ZibyMdfQzKr0dQ2Wvl5X2mMv4HgV+0bLhdRe5JUoHeoiWfE3hxRe7SyuvhpD1qV/4BHFKJXw5Mzc9/TS7m+fUb6X6kuTuVPXxrWwL/APy2JZ9vAqdV2vEkWvaE9L3I7VSJzwTenZ/fD7ylEjuR7kXuSOC6yuuD8nrqOi1bDuxeif+WF4rxicCfcg5bk04/A3h9S75bknZS3QpmJb4T6ej1uMqwM0hFdnvSadwted5jepi+qw32qgz7MnBhpb1+VYntDTxT+cwsoPtn5ibaf2Y+Xd2G6j43wLdbx63E/kA+4s7v/1zSDn8tcCOwR2Xc40k71aeBJ4BP9nF73zG3yyzSgcGo3KZn9mHasbndXtVDbEjO55uVYdPysk4n7TgOJ332x1fGeRNwf2/L7q1PbkdgQeQ5ZvO7nuSOwW/mzsNluTG3qbuCKeljudPyKUlPkjbmUTn2GuCpiHioMsmiyvNngMXxwsWDZ/Lf4ZVxHq08/0tLDICIeJpUJE4GFkr6maS92uXci+dzjdR3toTUbr3ls2N12pbnkI4Sfl6z3F2Ag3LH8pO5LY8nfYAhFfCJwHxJv5H0+j6+n42R99VdLyLilohYHhErI+Ji0odiIoCkbUinJb/Lo88gHT3dQDoFvD4P73ahhLQHX0IP/baStid1U5wfEZdUQmeSTiHvyMu7knRWsqh1Hm3e23zq1+swSYPp+TPT2kZVS0k7gb5YTCouPRmT40TEwxExJSJ2J20nT5POgsjxH0TEG0lnGCcDX5D0lj4sv+vz9o2IWBgRi0ldKhNrpula5gJSP+2l1eGSNgO+RzpCnNKyrK6zxuci4jek7aF6QXIE6WClVm9FbiEwtnrlhnTa1eVUUkfvQRExkrQXg3TIDKkSV9/QoaQ+nGOBbSNiG9KhbNf43T4gm1JE/DIi3kTaOOaSTs/WybkPdu56Imk4qS/okT5Mt5C0x11nPllrW7Tm9RDp1GWbymN4RHwAICJujYijSR3PV5KOInuaz4v1YvNuFbywvt8C/LprpxURayPitIjYNSJ2IhW6BflRNQn4bkshQdK2pAJ3VUSc2W2hEc/kD/7YiHgF6QjmtohYW5Nr9b2No+/rtfUz09pGVbNJ3St98SvSjq3b/CQdlJfx69YJ8gHDeaQLLq2xVRFxRc5hnXgP4y8l7XCq7f5itqfBpDOUrrwFXEg6VX1XRKyqjDu7pxRaXo8H/l9vC+2tyP0eWA38i6Qhko4BDqzER5Aq7pOStgNOa5l+EemqSnX81aT+jcGSPgeMrMQnAj/rLekNJWm0pKPzldyVpMPgro19EbBTXy5NZxMlHZLH/wKpb6Zuz93lcuDDksbmI5pPVvLbktTO11fGb23LnwKvkvTevG6GSPorSeOVbj05XtLWecNZ1vL+XiZp6z6+v57y/pSkbSWNpbL3lbQbsHlEzMmvt5H0FknDJA2WdDxpR9h15bXb+pa0naTdlexNOkr4P9VCJGkn4A2kix9Uho8EfgncHBFTW5PO7bxjnvfrgM+y7vba6rP5bGUfUt9nX261+D2py2JKfs9H0/0z02om6exnbMvwwbnduh5DIuJXpCuhP1S6TWRQfi/fBy6IiHvzevm8pFdK2kzSKOCfSaezXbdI/Y2kETn+VmAf0ul7X3wH+N+SXp53Kh8lbYvryNvguPx8F9LRdPVK7gWkQvW2yLfPVNxI6nL5VG7Hg0nr/ZeVcQ6n/mwn6cO59AReuFJ0GZUrRaRD8xtIReJPpD6gar/S6/PwpcC/kTouZ5A+dAtJR3XzSP1R25CLX2XZp5P7iuKFfqt5ldeDqfQfsW6/1Qnk/jW698mNIZ3qPEU63L2BFzrsh5I+eEtIp8a99cl1XV1dkVfMbpV4AK9sGf+MSu7nkI4oHiBtLKtIRzl/S0vnb2tb5mF75lwfz/P5NalfdCipkCzNbX0r3fsGZ+Txn6T91dXqeni+XUn9md/L084hXfr/c45NIV/5za+3z8tensf/A/CmHBPplO/llfFfRbpK/BfS6eEpPbT5p2jpi8zDJ/FC5/aKymNcvNBXNi/P+x7g+Jbpfw5Ma2mDrqurj9L9gsjz7dVTm5E+M3fk5V9Butr82Zrt6F+p9Ivl9o6WR9f6GUa6heMh0gHGfaQLAZtV1s/F+b2uyLlfAozN8WNIXQZd28adwAmVZU8Dfl6T6xDg/Lw+HyV9rofl2LiWNj+TdOT3dP47nReudu+S31fXHQNdj+Mry9qHtNN4mnSR7p2V2Jg8z6G91rDeRuivB+kU9vKm83iROV9Emw7l9ZjXW4H5+fn5wAebfn99zPsD5Ct+pNPUiX2c7kBgZtP5t8mtW9HaCPO7BXhfTXx7UpdJ26uUfqzTZmf39TMymM7xJOnI5iVB6V943kDqQxpNOnX6cQ7fQbqhs+NIGkM6bf49sAepX/bcHL6B7qfYventdHFAknQ46WhxMeli0L50vzm6m4h4nHQBxvooIk7t67gdU+Qi4pqmc+iJpLtJh9atTtrQWQOfJ53+P0M67fwcQERM38B5b0pDSbeq7EbaMV1KOvIkIr7c15lExMxNkl1n2JPUd7kV6Zabv4t0h741oOvuezOzInXc/66amW1MHXO6ur6GavMYxlZNp2FWrGd5mudipXofszN1XJGTdBTpf0AHAd+OiLPqxh/GVhykI/slN7OXoluiz19S0pE66nQ1/zvYeaTbKfYGjss3hZqZrZeOKnKke6fui4j7I+I50pW7oxvOycwGsE4rcmPp/s/MD+dh3Sh9i+wsSbNWsbLfkjOzgafTilyfRMT0iJgQEROGsHnT6ZhZB+u0IreA7t/Y0PW9YGZm66XTitytwB6Sdsvf6vFu0hcnmpmtl466hSQiVkuaQvo6lUHAjIi4u+G0zGwA66giBxARV9NPX5xpZuXrtNNVM7ONykXOzIrmImdmRXORM7OiuciZWdFc5MysaC5yZlY0FzkzK5qLnJkVzUXOzIrmImdmRXORM7OiuciZWdE67ltIzJo0aOTI2vhfDt2zbeyRQ+s/ToOerf9Vv9G3rqqNb/mnJ2rja+69vzb+UuUjOTMrmoucmRXNRc7MiuYiZ2ZFc5Ezs6K5yJlZ0VzkzKxovk/OXlI223LL2vjcM8bXxu951/nt5039fXBrido4768Pz1xZP//P3f+OtrHBb3ywfuYF85GcmRXNRc7MiuYiZ2ZFc5Ezs6K5yJlZ0VzkzKxoLnJmVjTfJ2dFeer419XGX3fKrNr4VTu0vw+uN0fefUxtfG3U3+e2+MYxtfHhD/dyn12NbXnp3ifXcUVO0jxgObAGWB0RE5rNyMwGso4rctkbImJx00mY2cDnPjkzK1onFrkArpF0m6TJPY0gabKkWZJmrWJlP6dnZgNJJ56uHhIRCyS9HLhW0tyIuLE6QkRMB6YDjNR2698ba2bF67gjuYhYkP8+BvwYOLDZjMxsIOuoIidpK0kjup4DbwbuajYrMxvIOu10dTTwY0mQcvuPiPhFsylZp1l76AFtY+ed8W+10+47dFBtfNqi+juWZp/86raxzWfeWTttb3bmgQ2a3nrWUUUuIu4H9ms6DzMrR0edrpqZbWwucmZWNBc5Myuai5yZFc1FzsyK1lFXV80A9D/2qY0/OGVN29iOg56rnXb8pR+rjb9y2n/Xxlm5YbeJWP/zkZyZFc1FzsyK5iJnZkVzkTOzornImVnRXOTMrGgucmZWNN8nZ/1u0MiRtfGlZzxbG79nvx+2jR1253trp9391D/Uxv010+XxkZyZFc1FzsyK5iJnZkVzkTOzornImVnRXOTMrGgucmZWNN8nZ/1u0Q92qI3/Yd9LauPjbz6hbewVpyytnXZ1bdRK5CM5Myuai5yZFc1FzsyK5iJnZkVzkTOzornImVnRXOTMrGi+T85etM223LI2/sSx+9XGL9n3K70sYVhtdLvL2y9/9UP+XVTrrpEjOUkzJD0m6a7KsO0kXSvp3vx32yZyM7OyNHW6ehFwVMuwqcB1EbEHcF1+bWa2QRopchFxI7CkZfDRwMX5+cXAO/o1KTMrUif1yY2OiIX5+aPA6HYjSpoMTAYYRn3/kJm9tHXk1dWICGp+UyQipkfEhIiYMITN+zEzMxtoOqnILZI0BiD/fazhfMysAJ1U5K4CJuXnk4CfNJiLmRVC6cywnxcqXQIcAYwCFgGnAVcClwPjgPnAsRHRenFiHSO1XRykIzddsi9Rdb+NuuB9r66ddtYnvrGx0+lm9nNr2sbWoNppT77zPbXxHU55rja+5r4HauMluiWuY1ksqW/YDtbIhYeIOK5NyNXKzDaqTjpdNTPb6FzkzKxoLnJmVjQXOTMrmoucmRWtkVtINibfQrJpLPv57m1jN+57eT9msq7Nam4TWdv+H2X65IuLX1Mbn3n0K9vGVs97cIOW3akG+i0kPpIzs6K5yJlZ0VzkzKxoLnJmVjQXOTMrmoucmRXNRc7MitZJX39uG1HdVyUBPHxi/dclzd7v/LaxNbFht0yd+uiBtfGrr/2r2vhuU3/fNrb2kP1rpz3kvJm18c+Muqs2Pv6kw9vn9aky75Mb6HwkZ2ZFc5Ezs6K5yJlZ0VzkzKxoLnJmVjQXOTMrmoucmRXN98kNUINGvaw2Pvdz7b/3DOCed9X/bOBTa9v/NN9J8/+mdtqHvrZHbXzEj2+rje+2uv19cL0ZMvv+2vi1C/eqjU8bdWdtfPWOK190TtYsH8mZWdFc5MysaC5yZlY0FzkzK5qLnJkVzUXOzIrmImdmRfN9cg0aPHbHtrF7PrJL7bTfPGZ6bfywYdesV05djrjtfW1jLz96bu20w7mlNr4pf+lXI0fUxq9/zRUbNP9xlw3aoOmt/zVyJCdphqTHJN1VGXa6pAWS7siPiU3kZmZlaep09SLgqB6GnxMR++fH1f2ck5kVqJEiFxE3AkuaWLaZvbR02oWHKZJm59PZbduNJGmypFmSZq3C/0toZu11UpG7ANgd2B9YCJzdbsSImB4REyJiwhA276/8zGwA6pgiFxGLImJNRKwFvgXU/6STmVkfdEyRkzSm8vKdQP1vw5mZ9UEj98lJugQ4Ahgl6WHgNOAISfuTbqOaB5zURG79ae5Zo9vH/ue5m3TZFz41rja+wz8+1Da2dmMnsxHd89GdN2j6aYsm1MaHXTe7bWxT3v9n66+RIhcRx/Uw+MJ+T8TMitcxp6tmZpuCi5yZFc1FzsyK5iJnZkVzkTOzovmrlho0eOia9Z52z1+9vzZ+4gE318Y//rI/1sa/9sm3t43t8rn1/8nAjeGJE1/fNnbD3/9rL1NvURudffKr6ydfWf+ThdZ5fCRnZkVzkTOzornImVnRXOTMrGgucmZWNBc5Myuai5yZFc33yXWozVBtfNSv678R+cZz679z9JNXzqmNX/ie89rGvvidd9ZOu/qB+bXxQSNH1saXXrZ9bXzWfhe0jf15Ve2kHPGRD9bGh8+s/zlFG3h8JGdmRXORM7OiuciZWdFc5MysaC5yZlY0FzkzK5qLnJkVzffJNeg1Oz7SNra2lx+4e2L/+vj2v3mqNv62P/1tbXzqLle3jT1+2I61064+qj7+vint5w1w8jbX18Z/9pet2sbOef8/1047/HrfB/dS4yM5Myuai5yZFc1FzsyK5iJnZkVzkTOzornImVnRXOTMrGiN3CcnaWfgu8BoIIDpEfF1SdsBlwG7AvOAYyNiaRM59oflp45pG/vGjD1qp517bPvvewPYiw/VxvfZbF5t/ODN17aN3fzFc2un3VD/8OejauNPfHm3trFh18/c2OnYANfUkdxq4NSI2Bt4HfAhSXsDU4HrImIP4Lr82sxsvTVS5CJiYUTcnp8vB+YAY4GjgYvzaBcD72giPzMrR+N9cpJ2BQ4AbgFGR8TCHHqUdDprZrbeGi1ykoYDPwQ+EhHLqrGICOj5HzglTZY0S9KsVazsh0zNbKBqrMhJGkIqcD+IiB/lwYskjcnxMcBjPU0bEdMjYkJETBhC/Q+6mNlLWyNFTpKAC4E5EfHVSugqYFJ+Pgn4SX/nZmZlUTor7OeFSocAvwXuBLruVZhG6pe7HBgHzCfdQrKkbl4jtV0cpCM3YbbNeOTjf10bP+ukGbXxN2/x9AYtv+4nEXv7Gqif/2VEbfyUKyfVxvf4Xv1dQ2tnz62N28Z1S1zHslhS/xuZHayR++Qi4iZo+ykqr2KZWWMav7pqZrYpuciZWdFc5MysaC5yZlY0FzkzK5qLnJkVrZH75DamUu+T683gMTvUxu+f/Ira+JrxK9Z72atX1t95NH5q+59aBFi98NH1Xrb1v4F+n5yP5MysaC5yZlY0FzkzK5qLnJkVzUXOzIrmImdmRXORM7OiNfJVS7bhervXbNznm7sXbXVjSzZbl4/kzKxoLnJmVjQXOTMrmoucmRXNRc7MiuYiZ2ZFc5Ezs6K5yJlZ0VzkzKxoLnJmVjQXOTMrmoucmRXNRc7MiuYiZ2ZFc5Ezs6I1UuQk7Szpekl/lHS3pA/n4adLWiDpjvyY2ER+ZlaOpr40czVwakTcLmkEcJuka3PsnIj4SkN5mVlhGilyEbEQWJifL5c0BxjbRC5mVrbG++Qk7QocANySB02RNFvSDEnbtplmsqRZkmatYmU/ZWpmA1GjRU7ScOCHwEciYhlwAbA7sD/pSO/snqaLiOkRMSEiJgxh837L18wGnsaKnKQhpAL3g4j4EUBELIqINRGxFvgWcGBT+ZlZGZq6uirgQmBORHy1MnxMZbR3Anf1d25mVpamrq4eDLwXuFPSHXnYNOA4SfsDAcwDTmomPTMrRVNXV28C1EPo6v7OxczK1vjVVTOzTclFzsyK5iJnZkVzkTOzornImVnRXOTMrGgucmZWNBc5Myuai5yZFc1FzsyK5iJnZkVzkTOzornImVnRXOTMrGiKiKZz2CCSHgfmVwaNAhY3lE5vOjW3Ts0LnNv62pi57RIR22+kefW7AV/kWkmaFRETms6jJ52aW6fmBc5tfXVybv3Np6tmVjQXOTMrWolFbnrTCdTo1Nw6NS9wbuurk3PrV8X1yZmZVZV4JGdm9jwXOTMrWjFFTtJRku6RdJ+kqU3nUyVpnqQ7Jd0haVbDucyQ9JikuyrDtpN0raR7899tOyi30yUtyG13h6SJDeW2s6TrJf1R0t2SPpyHN9p2NXl1RLt1giL65CQNAv4EvAl4GLgVOC4i/thoYpmkecCEiGj8xlFJhwErgO9GxKvzsC8DSyLirLyD2DYiPtkhuZ0OrIiIr/R3Pi25jQHGRMTtkkYAtwHvAE6gwbaryetYOqDdOkEpR3IHAvdFxP0R8RxwKXB0wzl1pIi4EVjSMvho4OL8/GLSh6TftcmtI0TEwoi4PT9fDswBxtJw29XkZVkpRW4s8FDl9cN01ooO4BpJt0ma3HQyPRgdEQvz80eB0U0m04Mpkmbn09lGTqWrJO0KHADcQge1XUte0GHt1pRSilynOyQiXgu8FfhQPi3rSJH6LzqpD+MCYHdgf2AhcHaTyUgaDvwQ+EhELKvGmmy7HvLqqHZrUilFbgGwc+X1TnlYR4iIBfnvY8CPSafXnWRR7tvp6uN5rOF8nhcRiyJiTUSsBb5Fg20naQipkPwgIn6UBzfedj3l1Unt1rRSitytwB6SdpM0FHg3cFXDOQEgaavcIYykrYA3A3fVT9XvrgIm5eeTgJ80mEs3XQUkeycNtZ0kARcCcyLiq5VQo23XLq9OabdOUMTVVYB8ifxrwCBgRkSc2XBKAEh6BenoDWAw8B9N5ibpEuAI0lfxLAJOA64ELgfGkb626tiI6PcLAG1yO4J0yhXAPOCkSh9Yf+Z2CPBb4E5gbR48jdT/1Vjb1eR1HB3Qbp2gmCJnZtaTUk5Xzcx65CJnZkVzkTOzornImVnRXOTMrGgucmZWNBc5Myva/weTE4E+x/5TkgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEICAYAAADLBejHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAG/dJREFUeJzt3XmYXVWZ7/Hvj0xAQkwQiBiGIAKSFgl0BG1Q40VBYtug2AgPrcEJELhqi14RvRCvE42i2NcIBqETlMHYKkOLA6CADKKBiwzGVoREhgyEACEQAkne+8daRXYOdXZVkkrtUyu/z/PUU+fsdw/v2Wfvd6+1z6pTigjMzEq0WdMJmJltLC5wZlYsFzgzK5YLnJkVywXOzIrlAmdmxVrnAidphqQvboxkNiZJO0laJmlQ07l0GkmnSfpu03lsqiSNlzRbkprOZSCQNEbSHEnDepp3o7bgJF0v6UPrsdwvJB3cl7lExN8iYkRErOph28dKuqkvt72+JA2VtFjSiPXdl92sc5Kkh6rTIuLLEbHB665s4+Wt25C0m6RnJX2/m/m/I+k4SW+XdJOkJyQtkPRdSVtV5jtS0i2SnpF0fcs6tpF0s6TH8vK3SjqgZZ5/zetdKunC3pwg/eQLwNciD0qVNFfSW7qbUdIoSefm1/GMpLslvb9lngPzfnpS0pK8X16bY0MlnS3poXzBnyvpnN4mmudfnpddJumXNfPOkPRcZd5uGxiSTpcUra9Z0lsk3SHp6ZzvkQARsRD4NXBcT/l2XBdV0nBgInBD07l0gDcCd0bEsqYTWUeTgZ+3TJsG/L7N/IcCVwMvAb4IvBzYExgLfLUy3xLgHODMbtaxDPgAsC0wGvg34CpJgwEkHQKcChwE7Ay8Avj8Or6uPidpe+DNwOW9mHcocC0p/9eT9tengDMlfSLPMxL4L+D/AluT9uHngRV5NZ8hnV/7AVsBk4A71jHtd+TGwoiI6KkhclZl3hc1MCTtCvwzML9l+njgEuCz+XXuDdxemeVi4PgeM42I2h9gH9IOeAr4AXAZ8MUcG03amY8Cj+fHO+TYl4BVwLOkg+9befo3gQeBpTnhN7Rs75+AK/PjqcAPge/n7d8N7E56kxbl9RxcWfZ60tXw5jz/L4FtcmwcEMDg/PxY4P483wPAMaST6tmc9zLgiR72zQzgPOCavJ4bgJ0r8QBOAP4CPEE6yZVjg4CzgcV5+ydX88vzfB34RM2+fFXe9hLgv4EjK8tOBv6Y83oY+CQwHFgOrM7rWUYqJlOB77fspynA33J+n62sdwtgZn6/5wD/C3ioZb/8GHhX5flRwKzqdiqx1wB3tdm/7wLu7mb6h4Dra96XzYB35NexXZ52CfDlyjwHAQvaLD8JeAg4Lb/+ucAxLe/7NOCnef/eBuxaiR+c348ngW/n4+JDbbb1PuDalmlzgbd0M+8HScf98Jbp78nv5UhS8Wp73JLO0Y/3dN7XLN9tbjXnxxd7mOfn+Vhda735/fpCzXKDgWeonG/dztfDxocC84B/BYYA7waeZ02BeylwBLAl6WrwQ+DyyvLXt76xwL/k5QYDpwALgM0r8fOA4/PjqaST+pA8/0WkYvDZnM+HgQdatvdXUhHcIj8/s+XEHUw60ZcCe+TY9sDf5cfHAjetwxv4FKmlNYxUvG+qxCMfUKOAnUgXgrfl2AmkArQD6UJxLS8ucH+q5LjWvsyv4UHg/fk17UM6Gcfn+HzyxSOvf9/qydvyOqby4gJ3ft6He5Ou/nvm+JmkE3Z0zv2u6vry+7IY2Co/Hwn8Oc/7wnYq858KfKXN/j0HuKyb6W0LXM7nua7XUJn+B+A9lefb5Hle2s06JgErSReYYcCbgKcr78UM4DFSK2gwqTVxWWW9S0nFeTDwMdI5067AfRWY1psiQmpczGxzsq8knScjc24zSS3j0S3zfo504ToR2It8wa3Evw18u+aYnwssJB3LvwT27uH8WJJ/bgeOaIn/M3BFd6+Z1Pj4AqlRM5/UyNm6m/f6nzakwL0ReKS6E4BbaFOVgQnA4y0Fp9s3tjLP49WdlHf+jpUT75pK7B2kK9Wg/HyrfJCOqmzvc5X5TwR+3nLidhW4J0jFeYuWfI5l3QrcZZXnI0gtra78AziwEp8FnJof/4pcyPPzt7B2C3NX4L52+5J01f5NSz7fAc6o7MfjgZHdnLy9KXA7VOK/A46qHHiHVGIfYu0CdxBwXeX5N4FPt26nEv8NLa34PP2t+djYvZtYTy24zYGjgSmVaX8lX1zy8yH5dY7rZvlJpIIxvDJtFvC/K+/7dyuxycCf8uP3AbdWYiJdiNoVuPPJF+HKtLl0X+CubZ23EltAbmWSeiIzSK3QlcCVwJgcGwScROrlrCCd31O6W2eb7RxAuvBtSepJLSCff93Muy9rGjOTSY2BAyrn7l+69n/rayZdpOaSGisjgB8BF7es/2bgfXX59nQP7uXAw5HXls3reiBpy3yDeJ6kpcCNwKi6TyolfTJ/AvKkpCdI/ettcmwv4MmIeLCyyMLK4+XA4ljTj1+ef4+ozLOg8viZlhgAEfE0qUCcAMyX9FNJr2qXcw9eyDXSvbIlpP3WUz4vry7b8hjSAfGzmu3uDOyfb6g/kfflMcDLcvyIvI55km6Q9Ppevp6+yPtqAEkTSIX7G91tQNIoUjf7lpbpryN1Ud4dEX9ex7yJiGcj4lLgVEl758ldXbguXY+farOax/Nx0mUe6/G+5nNnrQ9cWrdDOtl7YzGpt7GWfJ9xmxwnIuZExLERsQPw6pzTOTm2KiKmRcQBpJ7Fl4ALJe3ZmwQi4uaIWB4Rz0TEV0gNhTe0mfeOiHgsIlZGxNWklu67cngq8L2ImNtmU8uB/4iIP+fz6sukY6tqq7z9tnoqcPOBsS0fX+9UeXwKsAewf0SMJLX4IF21IF0hXyDpDaR7NkeSms6jSPcpuuZ/4eTY2CLiFxHxVtIB8yfSlfRFOffCjl0PJI0g3dh9pBfLzSd12160nqx1X7Tm9SBwQ0SMqvyMiIiPAETE7yPiMGA70g3sWW3Ws67WJe9JpBbh3yQtIN0HPEJS103tQ4BfVS5YSNqH1OL4QERct4G5DiF9mABwL6m73WVvYGFEPNZm2dH5A68uO7Ee72s+d3ZoPzt3kVopvXEtcGhLXpAuZiuA37YuEBF/IrXmXt1NbHlETCMV2fG9zOFFq2HN+bsu8x4EfDR/GryAdBzNkvTpHL+LtY/V1loyGHgl6dZDWz0VuFtJTdyPShoi6V2k+w5dtiJV2ickbQ2c0bL8QtYcYF3zryT13wdLOp21r6qTSTduNyqlcTSH5QNlBenqvrqS8w75E6vemJw/lh9Kumfw25YWaDuzgI9JGptbMl1vLJK2JO3nX1fmb92X/wXsLum9+b0ZIum1kvbMQwGOkfSSiHiedE+o+vpeKuklvXx93eX9GUmjJY0lfTjSlfcuwLCImJMnTSd1tSfkn/NI7+8hOb7W+y3p1aSbzv8zIq5q3bCkQZI2J3V5NpO0uaQhOfa6rvdB0hb5RBlD+gAA0v3bDyqNORtFuhc1o4fX+vm8vjcA/0i6x9yTnwJ7STo8n4QnsaZV3Z1rgH3z66oakl9f189g4Huk1uAPJY3L7/khwL8DUyPiSUmvknSKpB3yftmR1F3/bX7+caWhQltIGixpCum8/H89vTClsaQH5H2yuaRPkVqON7eZ/91KQ5w2Uxr29S+kixekAvdq1hwbj5BuqUzL8f8A3i/pFfl8OJV0zHfZD5gbEfOoUVvgIuI5UpPyWFLX6z2kT8i6nEPqjy8m7cDWoQHfBN4t6XFJ/w78Is/zZ1KT/1lycz4fdONp6a5sJJuRPp18hPS63gR8JMd+RbraL5C0uBfruoRU2JcAf096E3vjfNJN2rtIB9fVpOK/CvgfpPs4z1bmX2tfRsRTpE/rjsqvYwFpaETX2K73AnPzrYMTSN3Xriv6pcD9uWtb7Xb1xv8hnWQPkFoU/8maIQhvp9LqzN2YBV0/pAvJsxHxaG7ZHMLax8wppGEeF2jNuKl7K/H3ki6o55K6RctZ0/IeRjo5HiN9ajwZeHtEPJJz+TlwFumi8TfS8ffCBVnSvZKOqWxrAall8wipa3VC3ne1ImIx6eb5WTmX8cDsyj5qnX8h6Zg7rCV0dX59XT9TI2IFqcv/IKlwLyV9EPLZiOgaTvMUsD9wm6SnSeflPaR9C6k7fXZ+fYtJBfiIiLg/74fzJJ3X5uVtRdr3j5P28duAQ7tawfmiWn2/Ppbne4L0YcqHI+L6/Lofazk2VpFuCyzL8QtJF6XbSO/VCuCjlXUfQ7pg1uvtzcWN/UPqts5qOo91zHkGPXwMvg7rOhSYlx9/Gzix6dfXy7w/QuoqQzopJ/dyuf2A3zWdf5vcJtHyQcwGrGszUpF8c80840ljBNUX2yz9h3TbZQ6V0RftfjppoO8TtLkZXaLcRZicuwljSa2Jn+TwnZXHHUXS9rmbspmkPUgtg65cr2ftbnVPWm9pFEHSIUp/cTCMNJZOdHN/rEtE/DEiXhv57LV6EbEoIvaMtXs43RrcHwn1RkS0/ZOPJuUm987dhHoeRd3DqkkjzH9A6oL8FDgdICKmb+C6N6ahpOEou5AuSpeRWpxExFm9XUlE/G6jZNcZXk+6dTGUNNbx8IhYXr+IbQzyRcPMStVJXVQzsz7VMV3U9TVUw2JzWocFmVlfeZaneS5WDMivcuq4AifpbaQhEYNIfw7T3TdHvGBzhrO/DuqX3Mw2Rbdt8Hjr5nRUF1XpT7ymkYZMjAeOVvraFDOzddZRBY40Nuq+iLg/0iDjy3jxAEgzs17ptAI3lrX/ePuhPG0tSt/+OlvS7Oe7HyBuZtZxBa5XImJ6REyMiIlDXvjLJDOztXVagXuYtb+dYoc8zcxsnXVagfs9sJukXfK3cxzFmm8fMDNbJx01TCQiVko6mfStI4OACyPi3h4WMzPrVkcVOIBI3/zZL196aWZl67QuqplZn3GBM7NiucCZWbFc4MysWC5wZlYsFzgzK5YLnJkVywXOzIrlAmdmxXKBM7NiucCZWbFc4MysWC5wZlYsFzgzK5YLnJkVywXOzIrlAmdmxXKBM7NiucCZWbFc4MysWC5wZlYsFzgzK5YLnJkVywXOzIrlAmdmxXKBM7NiucCZWbFc4MysWC5wZlYsFzgzK9bgphNoJWku8BSwClgZERObzcjMBqqOK3DZmyNicdNJmNnA5i6qmRWrEwtcAL+UdLuk47qbQdJxkmZLmv08K/o5PTMbKDqxi3pgRDwsaTvgGkl/iogbqzNExHRgOsBIbR1NJGlmna/jWnAR8XD+vQj4CbBfsxmZ2UDVUQVO0nBJW3U9Bg4G7mk2KzMbqDqtizoG+IkkSLldEhE/bzYlazVom5fWxud8dVxt/O9fOa82/uhZr6iNb/GzO9rGYuXK2mVt09JRBS4i7gf2bjoPMytDR3VRzcz6kgucmRXLBc7MiuUCZ2bFcoEzs2J11Keo1jkGbbtt29icL+xSu+x9B5+3YRvvYfF9v3py29jLzrllw7a931614SdeNaJtbOT9y2uX3eymO9crJVt/bsGZWbFc4MysWC5wZlYsFzgzK5YLnJkVywXOzIrlAmdmxfI4uE3UZsOH18ZHXr6qbey+cRs4zm0DnXLCrLaxi8/ZoXbZ1W/Ypzb+1YvqX9teQ4e0jf1t5TO1y56458G18dXP1C9v684tODMrlgucmRXLBc7MiuUCZ2bFcoEzs2K5wJlZsVzgzKxYHgdXqJ7+td+oK1bXxr837tq+TGcti1bVj/e6dOlrauPfmv3mtrE9JtR/J9uXL/pObbxunFtPdhq8ZW38gVPr/2Hczqffut7btu65BWdmxXKBM7NiucCZWbFc4MysWC5wZlYsFzgzK5YLnJkVy+PgCvXQd7erjV817uKNtu3znty5Nv7dae+ojW83rf5/m+5yUPvYfjP/ULvshKHNHfLPjVvR2LY3VY204CRdKGmRpHsq07aWdI2kv+Tfo5vIzczK0VQXdQbwtpZppwLXRcRuwHX5uZnZemukwEXEjcCSlsmHATPz45nA4f2alJkVp5PuwY2JiPn58QJgTLsZJR0HHAewOfV//2dmm66O/BQ1IgKImvj0iJgYEROHMKwfMzOzgaSTCtxCSdsD5N+LGs7HzAa4TipwVwJT8uMpwBUN5mJmBWjkHpykS4FJwDaSHgLOAM4EZkn6IDAPOLKJ3DqFBte/NVv+qn4UzQ27XNDDFjZfx4zW+NqSPWrjN75jz9r4yxbdWRtfcPI/1MZ//Kmz2sZ6+k62jWl5PFcb3+4XQ/spE+vSSIGLiKPbhGqGcJqZrZtO6qKamfUpFzgzK5YLnJkVywXOzIrlAmdmxeqkP9Wyivkn7Vcbv2PXb/WwhvUfBgJw1TMj28Z6Ggay/JXb1sY/+Iuba+NHjbipNk6Df55X91VQP/xk6/dHrO0lV/+2r9OxHrgFZ2bFcoEzs2K5wJlZsVzgzKxYLnBmViwXODMrlgucmRXL4+A61Oknfb/R7T8Xg9rGlpw7pHbZa/aaVhvfQp37tUGLVj1TG5951j+2jY2++ta+Tsc2kFtwZlYsFzgzK5YLnJkVywXOzIrlAmdmxXKBM7NiucCZWbE8Ds66dcTwx9vHXvPDHpbu3HFuPXnPnPfWxkfP8Fi3gcQtODMrlgucmRXLBc7MiuUCZ2bFcoEzs2K5wJlZsVzgzKxYHgfXoU6d/a7a+OFvurCfMtm0DP3K6B7meKBf8rC+0UgLTtKFkhZJuqcybaqkhyXdmX8mN5GbmZWjqS7qDKC7fwP+jYiYkH+u7ueczKwwjRS4iLgRWNLEts1s09FpHzKcLOmu3IVtezNE0nGSZkua/Twr+jM/MxtAOqnAnQvsCkwA5gNnt5sxIqZHxMSImDiEYf2Vn5kNMB1T4CJiYUSsiojVwPnAfk3nZGYDW8cUOEnbV56+E7in3bxmZr3RyDg4SZcCk4BtJD0EnAFMkjQBCGAucHwTuXWK3c5YWhufceXLa+PHjnykL9NZy+VPj6qNf+aOw2vju7/s0dr4Fbv9dJ1z6q1XXnVCbXyPm+6ojUdfJmMbXSMFLiKO7mbyBf2eiJkVrWO6qGZmfc0FzsyK5QJnZsVygTOzYrnAmVmx/HVJHWrVX+6vjf9o/93r41u8pjZ+/0m71sYHLVfb2M7n3lu77K6r6r9S6E23brwhLFc9M7I2Pv7fFtbGV65c2ZfpWMPcgjOzYrnAmVmxXODMrFgucGZWLBc4MyuWC5yZFcsFzsyK5XFwA9SqpfVfp0QP8Z1PX7T+2+4hvujEf6iNf2L0Deu9bYBl0f5r6j93wftqlx37wC0btG0bWNyCM7NiucCZWbFc4MysWC5wZlYsFzgzK5YLnJkVywXOzIrlcXDW5z5w0sb7t38Ac54b2jY29kyPc7M13IIzs2K5wJlZsVzgzKxYLnBmViwXODMrlgucmRXLBc7MitXIODhJOwIXAWOAAKZHxDclbQ38ABgHzAWOjIjHm8jR2nvsw6+vjZ84atpG3f6US09uGxvHrRt12zawNNWCWwmcEhHjgdcBJ0kaD5wKXBcRuwHX5edmZuulkQIXEfMj4o78+ClgDjAWOAyYmWebCRzeRH5mVobG78FJGgfsA9wGjImI+Tm0gNSFNTNbL40WOEkjgB8BH4+Itf6JQEQE6f5cd8sdJ2m2pNnP0/77+c1s09ZYgZM0hFTcLo6IH+fJCyVtn+PbA93+Z5SImB4REyNi4hCG9U/CZjbgNFLgJAm4AJgTEV+vhK4EpuTHU4Ar+js3MytHU1+XdADwXuBuSXfmaacBZwKzJH0QmAcc2VB+VuPZQ3v4l4U9GKT66+qqWF0b3/qebu9cmL1IIwUuIm4C1CZ8UH/mYmblavxTVDOzjcUFzsyK5QJnZsVygTOzYrnAmVmxXODMrFj+t4G2zt644183aPmexrn15JIzv9Y2duIfP1y77Oo/zNmgbdvA4hacmRXLBc7MiuUCZ2bFcoEzs2K5wJlZsVzgzKxYLnBmViyPg7MBZ6fBW7aNLXz9qNplt/1DX2djncwtODMrlgucmRXLBc7MiuUCZ2bFcoEzs2K5wJlZsVzgzKxYHgdn6+zmS/atjS875de18REatkHbX7r62fbrXrBqg9ZtZXELzsyK5QJnZsVygTOzYrnAmVmxXODMrFgucGZWLBc4MytWI+PgJO0IXASMAQKYHhHflDQV+DDwaJ71tIi4uokcrb2XfeOW2viE8R+tjd83+TsbtP19f/axtrHdL//dBq3bytLUQN+VwCkRcYekrYDbJV2TY9+IiPb/2dfMrJcaKXARMR+Ynx8/JWkOMLaJXMysXI3fg5M0DtgHuC1POlnSXZIulDS6zTLHSZotafbzrOinTM1soGm0wEkaAfwI+HhELAXOBXYFJpBaeGd3t1xETI+IiRExcQgb9neNZlauxgqcpCGk4nZxRPwYICIWRsSqiFgNnA/s11R+ZjbwNVLgJAm4AJgTEV+vTN++Mts7gXv6OzczK4ciov83Kh0I/Aa4G1idJ58GHE3qngYwFzg+fyDR1khtHfvroI2XrNkm7ra4jqWxRE3nsT6a+hT1JqC7HeYxb2bWZxr/FNXMbGNxgTOzYrnAmVmxXODMrFgucGZWLBc4MyuWC5yZFcsFzsyK5QJnZsVygTOzYrnAmVmxXODMrFgucGZWLBc4MytWI98H15ckPQrMq0zaBljcUDp1OjUvcG7ra1PJbeeI2LaP1tWvBnyBayVpdkRMbDqPVp2aFzi39eXcOp+7qGZWLBc4MytWiQVuetMJtNGpeYFzW1/OrcMVdw/OzKxLiS04MzPABc7MClZMgZP0Nkn/Lek+Sac2nU+VpLmS7pZ0p6TZDedyoaRFku6pTNta0jWS/pJ/j+6g3KZKejjvuzslTW4grx0l/VrSHyXdK+ljeXrj+60mt8b3Wyco4h6cpEHAn4G3Ag8BvweOjog/NppYJmkuMDEiGh8UKumNwDLgooh4dZ52FrAkIs7MF4fREfHpDsltKrAsIr7W3/lU8toe2D4i7pC0FXA7cDhwLA3vt5rcjqTh/dYJSmnB7QfcFxH3R8RzwGXAYQ3n1JEi4kZgScvkw4CZ+fFM0gnS79rk1riImB8Rd+THTwFzgLF0wH6ryc0op8CNBR6sPH+IznqTA/ilpNslHdd0Mt0YExHz8+MFwJgmk+nGyZLuyl3YRrrPXSSNA/YBbqPD9ltLbtBB+60ppRS4TndgROwLHAqclLtiHSnSPYtOum9xLrArMAGYD5zdVCKSRgA/Aj4eEUursab3Wze5dcx+a1IpBe5hYMfK8x3ytI4QEQ/n34uAn5C61J1kYb6X03VPZ1HD+bwgIhZGxKqIWA2cT0P7TtIQUgG5OCJ+nCd3xH7rLrdO2W9NK6XA/R7YTdIukoYCRwFXNpwTAJKG55u/SBoOHAzcU79Uv7sSmJIfTwGuaDCXtXQVkOydNLDvJAm4AJgTEV+vhBrfb+1y64T91gmK+BQVIH8Mfg4wCLgwIr7UcEoASHoFqdUGMBi4pMncJF0KTCJ9nc5C4AzgcmAWsBPpq6eOjIh+v9nfJrdJpG5WAHOB4yv3vforrwOB3wB3A6vz5NNI97oa3W81uR1Nw/utExRT4MzMWpXSRTUzexEXODMrlgucmRXLBc7MiuUCZ2bFcoEzs2K5wJlZsf4/zQxAeILidf0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.view_top_losses(n=3, preproc=preproc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen, such images are legitmately challenging to classify - even for humans. In some cases, inspecting misclassifications can help shed light on how to improve your model or improve data preprocessing strategies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting New Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that our call to ```images_from_folder``` returned a Preprocessor instance as a third return value. We can take our model and the Preprocessor instance and wrap them in a Predictor object to easily make predictions on new raw data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = ktrain.get_predictor(learner.model, preproc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.get_classes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['7']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict_filename(DATADIR+'/testing/7/7021.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f42147caa20>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAADZBJREFUeJzt3W2MXOV5xvHrMvFLYgj1xrA1xsWJ66IiEKTZGkhRRUSTGiutSZuSWG1wJRSnKo5CFdQi90OIqkpWGhOhNk3lBBdTEUiUgLAqh4S6RCQNWKyRARNDINQUO36BOrWdQPy2dz/sMdrAzpn1nDNzZn3/f9JqZ8595jw3g685M/PMzuOIEIB8pjTdAIBmEH4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0m9pZeDTfP0mKGZvRwSSOUX+rmOxGFPZN9K4be9WNJtkk6T9JWIWF22/wzN1KW+qsqQAEpsjk0T3rfjp/22T5P0RUlXS7pA0jLbF3R6PAC9VeU1/yJJz0fECxFxRNI9kpbW0xaAbqsS/rmSXhpzfWex7ZfYXmF72PbwUR2uMByAOnX93f6IWBsRQxExNFXTuz0cgAmqEv5dkuaNuX5usQ3AJFAl/I9JWmj7nbanSfqopA31tAWg2zqe6ouIY7ZXSvq2Rqf61kXE07V1BqCrKs3zR8RGSRtr6gVAD/HxXiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5KqtEqv7R2SDkk6LulYRAzV0RSA7qsU/sL7IuKVGo4DoId42g8kVTX8Iek7trfYXlFHQwB6o+rT/isiYpftsyU9aPuZiHh47A7Fg8IKSZqht1UcDkBdKp35I2JX8XufpPskLRpnn7URMRQRQ1M1vcpwAGrUcfhtz7R9xonLkj4gaVtdjQHoripP+wcl3Wf7xHG+GhEP1NIVgK7rOPwR8YKki2vsBS28Ze45pfUDl81rWdu1eKTS2LMGD5bWN7/nqx0f+zc2/kVpfe4D5U9Mz/jWU6X1kVdfPemeMmGqD0iK8ANJEX4gKcIPJEX4gaQIP5BUHX/Vh4r23Pje0vrf3XBHaf3333ag47GntHn8H1H5VGGVicQfLfmX8mMvKT/6b37jk6X1hZ969KR7yoQzP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kxTx/H7j4I+XfgVJlHv9Uds8f/mNp/bO3/lHL2rEXX6q7nUmHMz+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJMU8fx/43rbzy3f4tU29aaQDix67rrR+0dm7W9b+9bxq/10XTyuvP3Pj3Ja1X/8r5vk58wNJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUm3n+W2vk/RBSfsi4sJi24Ckr0maL2mHpGsj4qfda/PUdtZ/lf9vmLKk88foJc9cU1rf8+3Wy3tL0jmf+0Fp/Ve1vbT+6JrLWtamzv9u6W2PRmm5rXdsdbUDnOIm8q/qDkmL37DtZkmbImKhpE3FdQCTSNvwR8TDkva/YfNSSeuLy+sllZ9eAPSdTp9PDkbEic9t7pE0WFM/AHqk8ht+ERGSWr46s73C9rDt4aM6XHU4ADXpNPx7bc+RpOL3vlY7RsTaiBiKiKGpmt7hcADq1mn4N0haXlxeLun+etoB0Cttw2/7bkmPSDrf9k7b10taLen9tp+T9HvFdQCTSNt5/ohY1qJ0Vc29pDX73qdL6+cP/WVpfeCJ1o/hZ91TvibAOYeq/V374at/u7T+n3/y+Za1o/HW0tuOaKSjnk6Y/UjLV6M6XunIpwY+4QckRfiBpAg/kBThB5Ii/EBShB9Iiq/u7gPHDx4srS9cubnjY1ebLGvv0Lzyf0KDp3XvU51/+sLVpfX4n11dG/tUwJkfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Jinh+VnPdnzzc29osHBkrrA794pUedTE6c+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKeb5UerQR1ovsS1J/77gi22O0Pr8MkXlS2gfGDlSWj9zzeltxkYZzvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kFTbeX7b6yR9UNK+iLiw2HaLpI9LernYbVVEbOxWk2jOsev+t7RebRnt8nPP5XfdVFp/10OPVBgbEznz3yFp8TjbvxARlxQ/BB+YZNqGPyIelrS/B70A6KEqr/lX2n7S9jrbs2rrCEBPdBr+L0laIOkSSbslrWm1o+0VtodtDx/V4Q6HA1C3jsIfEXsj4nhEjEj6sqRFJfuujYihiBiaqu4t2gjg5HQUfttzxlz9kKRt9bQDoFcmMtV3t6QrJc22vVPSZyRdafsSSSFph6RPdLFHAF3QNvwRsWyczbd3oRcks/f4a6X1c797rEed5MQn/ICkCD+QFOEHkiL8QFKEH0iK8ANJ8dXdyf3fxy4vrX/jon9oc4TOP7V55dfL/2R3wQOPdnxstMeZH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSYp4/uR+sLl9ie0Rv7drYC25iHr9JnPmBpAg/kBThB5Ii/EBShB9IivADSRF+ICnm+U9xP/nr95bWR7SlTb3KEtzSQ6+dXun26B7O/EBShB9IivADSRF+ICnCDyRF+IGkCD+QVNt5ftvzJN0paVBSSFobEbfZHpD0NUnzJe2QdG1E/LR7raKVn3/40pa1TZ9s9737MyqN/a1XZ5XW//m6P25Zs56oNDaqmciZ/5ikT0fEBZIuk3SD7Qsk3SxpU0QslLSpuA5gkmgb/ojYHRGPF5cPSdouaa6kpZLWF7utl3RNt5oEUL+Tes1ve76kd0vaLGkwInYXpT0afVkAYJKYcPhtny7pm5JujIiDY2sRERp9P2C8262wPWx7+KgOV2oWQH0mFH7bUzUa/Lsi4t5i817bc4r6HEn7xrttRKyNiKGIGJpaYVFHAPVqG37blnS7pO0RceuY0gZJy4vLyyXdX397ALplIn/S+zuSPibpKdtbi22rJK2W9HXb10t6UdK13WkR7fzkfa1rZ06Z1tWx//7ZJaX1gUeYzutXbcMfEd+X5Bblq+ptB0Cv8Ak/ICnCDyRF+IGkCD+QFOEHkiL8QFJ8dfck8N+rLy+tP3vNP5VUyx/fp7ScxR11YORIaf3MNXw192TFmR9IivADSRF+ICnCDyRF+IGkCD+QFOEHkmKefxL4lYteKa1XW0a7/PH/D7ZdV1p/+0OPVxgbTeLMDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJMc8/CYzcO7t8h0s6P/be46+V1mfcNtDmCD/ufHA0ijM/kBThB5Ii/EBShB9IivADSRF+ICnCDyTVdp7f9jxJd0oalBSS1kbEbbZvkfRxSS8Xu66KiI3dajSzs+4uX+P+syvf07L2mbO3lN72w6tuKq2f+cCjpXVMXhP5kM8xSZ+OiMdtnyFpi+0Hi9oXIuLz3WsPQLe0DX9E7Ja0u7h8yPZ2SXO73RiA7jqp1/y250t6t6TNxaaVtp+0vc72rBa3WWF72PbwUR2u1CyA+kw4/LZPl/RNSTdGxEFJX5K0QKOfLN8tac14t4uItRExFBFDUzW9hpYB1GFC4bc9VaPBvysi7pWkiNgbEccjYkTSlyUt6l6bAOrWNvy2Lel2Sdsj4tYx2+eM2e1DkrbV3x6AbnFElO9gXyHpe5Kekl7/juhVkpZp9Cl/SNoh6RPFm4Mtvd0DcamvqtgygFY2xyYdjP3l664XJvJu//elcRdxZ04fmMT4hB+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiCptn/PX+tg9suSXhyzabakV3rWwMnp1976tS+J3jpVZ2/nRcRZE9mxp+F/0+D2cEQMNdZAiX7trV/7kuitU031xtN+ICnCDyTVdPjXNjx+mX7trV/7kuitU4301uhrfgDNafrMD6AhjYTf9mLbz9p+3vbNTfTQiu0dtp+yvdX2cMO9rLO9z/a2MdsGbD9o+7ni97jLpDXU2y22dxX33VbbSxrqbZ7th2z/0PbTtj9VbG/0vivpq5H7redP+22fJulHkt4vaaekxyQti4gf9rSRFmzvkDQUEY3PCdv+XUk/k3RnRFxYbPucpP0Rsbp44JwVEX/TJ73dIulnTa/cXCwoM2fsytKSrpH052rwvivp61o1cL81ceZfJOn5iHghIo5IukfS0gb66HsR8bCk/W/YvFTS+uLyeo3+4+m5Fr31hYjYHRGPF5cPSTqxsnSj911JX41oIvxzJb005vpO9deS3yHpO7a32F7RdDPjGByzMtIeSYNNNjOOtis399IbVpbum/uukxWv68Ybfm92RUT8lqSrJd1QPL3tSzH6mq2fpmsmtHJzr4yzsvTrmrzvOl3xum5NhH+XpHljrp9bbOsLEbGr+L1P0n3qv9WH955YJLX4va/hfl7XTys3j7eytPrgvuunFa+bCP9jkhbafqftaZI+KmlDA328ie2ZxRsxsj1T0gfUf6sPb5C0vLi8XNL9DfbyS/pl5eZWK0ur4fuu71a8joie/0haotF3/H8s6W+b6KFFX++S9ETx83TTvUm6W6NPA49q9L2R6yW9Q9ImSc9J+g9JA33U279pdDXnJzUatDkN9XaFRp/SPylpa/GzpOn7rqSvRu43PuEHJMUbfkBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkvp/TGkZU3sa4u4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "vis.show_image(DATADIR+'/testing/7/7021.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictor object can be saved and reloaded later for use within a deployed application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.save('/tmp/mymnist')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = ktrain.load_predictor('/tmp/mymnist')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the `load_predictor` and `get_predictor` functions accept an optional `batch_size` argument that is set to 32 by default.  For instance, the `batch_size` used for inference and predictions can be increased with either of the following:\n",
    "```python\n",
    "# you can set the batch_size as an argument to load_predictor\n",
    "predictor = ktrain.load_predictor('/tmp/mymnist', batch_size=64)\n",
    "\n",
    "# you can also set the batch_size used for predictions this way\n",
    "predictor.batch_size = 64\n",
    "```\n",
    "Larger batch sizes can potentially speed predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1010 images belonging to 1 classes.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('3/1020.png', '3'),\n",
       " ('3/1028.png', '3'),\n",
       " ('3/1042.png', '3'),\n",
       " ('3/1062.png', '3'),\n",
       " ('3/1066.png', '3'),\n",
       " ('3/1067.png', '3'),\n",
       " ('3/1069.png', '3'),\n",
       " ('3/1072.png', '3'),\n",
       " ('3/1092.png', '3'),\n",
       " ('3/1095.png', '3')]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's make predictions for all images depicting 3 in our validation set\n",
    "predictor.predict_folder(DATADIR+'/testing/3/')[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using *ktrain* with Your Own Custom Keras Models\n",
    "\n",
    "\n",
    "In the examples above, we employed the use of a pre-canned model that we loaded using the ```image_classifier``` function.  This is not required, as *ktrain* is designed to work seamlessly with Keras.\n",
    "\n",
    "For instance, in the example below, we use *ktrain* with a custom model that we define ourselves.   The code below was copied directly from the [Keras MNIST example](https://github.com/keras-team/keras/blob/master/examples/mnist_cnn.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import some necessary modules\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Dropout, Flatten\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D\n",
    "from tensorflow.keras import optimizers\n",
    "from tensorflow.keras import backend as K\n",
    "\n",
    "# load data as you normally would in Keras\n",
    "NUM_CLASSES = 10\n",
    "\n",
    "\n",
    "# define a model as you normally would in Keras\n",
    "def load_model(input_shape):\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                     activation='relu',\n",
    "                     input_shape=input_shape))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "    model.compile(loss=tf.keras.losses.categorical_crossentropy,\n",
    "                  optimizer='adam',\n",
    "                   metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "# load the data and the model    \n",
    "if K.image_data_format() == 'channels_first':\n",
    "    input_shape=(1,28,28)\n",
    "else:\n",
    "    input_shape=(28,28,1)\n",
    "\n",
    "model = load_model(input_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this custom model, we can follow the exact same training procedure as above and take advantage of various *ktrain* features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wrap model and data in Learner instance\n",
    "learner = ktrain.get_learner(model, train_data=train_data, val_data=val_data, \n",
    "                             workers=8, use_multiprocessing=True, batch_size=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "simulating training for different learning rates... this may take a few moments...\n",
      "Epoch 1/5\n",
      "937/937 [==============================] - 12s 12ms/step - loss: 2.3264 - acc: 0.1166\n",
      "Epoch 2/5\n",
      "  1/937 [..............................] - ETA: 11s - loss: 2.2100 - acc: 0.1875\n",
      "937/937 [==============================] - 12s 13ms/step - loss: 1.5400 - acc: 0.4883\n",
      "Epoch 3/5\n",
      "937/937 [==============================] - 12s 12ms/step - loss: 0.3973 - acc: 0.8754\n",
      "Epoch 4/5\n",
      "469/937 [==============>...............] - ETA: 6s - loss: 0.6880 - acc: 0.7913\n",
      "\n",
      "done.\n",
      "Please invoke the Learner.lr_plot() method to visually inspect the loss plot to help identify the maximal learning rate associated with falling loss.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEOCAYAAABmVAtTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8VNXdx/HPLwsJayAS9iXsiCyCQUBFcUfcFXetVq3F2qqt9qm1ta1Wq61Ln0etWqqWarV1oVoRFGgVAVlk3/c9yBK2hJCEbOf5Y4YYMAkJzJ07y/f9es2LO3fuzP2eDJlf7txzzzHnHCIiIgAJfgcQEZHIoaIgIiIVVBRERKSCioKIiFRQURARkQoqCiIiUsGzomBm7c3sczNbbmbLzOy+KrYZZma5ZrYwePuVV3lEROTokjx87VLgAefcfDNrDMwzs8nOueVHbDfNOXeJhzlERKSWPDtScM5tc87NDy7vB1YAbb3an4iIHL+wnFMws0ygPzC7ioeHmNkiM/vEzE6q5vl3mdnc4O0uD6OKiMQ183qYCzNrBHwBPOGc+9cRjzUByp1z+WY2Avg/51y3ml6vefPmLjMz07O8IiKxaN68ebuccxlH287LcwqYWTIwFnjryIIA4JzLq7Q8wcxeMrPmzrld1b1mZmYmc+fO9SawiEiMMrNNtdnOy95HBrwGrHDOPVfNNq2C22Fmpwbz7PYqk4iI1MzLI4XTgVuAJWa2MLjuYaADgHPuFWAkcLeZlQKFwPVOw7aKiPjGs6LgnJsO2FG2eRF40asMIiJSN7qiWUREKqgoiIhIBRUFERGpEDdFIbewhA8WZFNervPYIiLViZui8NnKHfz4nUV0fngCm3Yf8DuOiEhE8vTitUhy4Umt6NZiHWt25nPW01MAaNYgmVsGd+S7p3eipLyc8nJo0TiFhATjUM/YjbsLaNE4hQb1EgleUnGY4tJyAJISjISEGjtbiYhEPM+HuQi1rKwsdzxXNL87Zwv/M3Zxjdv0a5fGouzcw9bVS0rglA7N6NsujcnLd7D7QDEZjVNYuzMfgOaNUjjvxBaUlTsapSZRVu7IPKEhAzPT6dMu7ZjzioiEgpnNc85lHXW7eCsKh5SVOz5e/DVjZmxkweZ9XJfVnm4tG/HVhj1MWr6D5ESjpMzxk/O7k723gDkb97Jh1zdfOyUmGAMzm5GYYLRsnMrC7H3szi8mwaCwpIyikvLD9vf9szpzZrcMTunYjHU5+bRPb0CT1OTDttmeWwRAw5REGtZL0pGHiISMioIHikvLKSoto6i4jNR6id/6UK/MOcf2vCL+OHk1n63MYVf+wW9t0yWjIaXljvyiUkrKyskrKq14zAwu7duG/UUlHCwt5+weLbhxUAcapsTNN34iEkIqChFmW24hE5Zs5+t9hRSXllNSVs72vCLW5xxg854ChnZrTrMG9ejUvCFb9hTwydLtlDlHebmjtFKPqfN7tSQ50UhOTOCmQR05tVO6j60SkWihohBD9hwoZsqqnYydn83Mdbtp0TiVwpIycgtL6N6yEfXrJdEkNYni0nKGdDmB/KJSWjetzy2DO5KUYGzLK6J1k1R9HSUSx1QUYlRpWTmJCUZRSTmvf7mB9+dlk723gKSEBFqlpbJx9wGqe0t/MeJEfQUlEqdUFOJIWbkjwcDM2F9UwqbdBYyZsZH6yYls2HWA4rJy1u3MZ/eBYhqlJHHdwPb88uITq+xiKyKxSUVBvmX+5r08Om45i7bsA+DarHac2LoJAEUl5ZzUpgldWjSiTVqqCoZIjFFRkCqVlpXz8pR1PDt5dY3b9WmbRlZmM05s3YSO6Q04pWMzkhLj5gJ4kZijoiA1Ki93ZO8tpEFKIkUlZewrKGH5tjy25xYxe8NutucWkb23kIPBK7a7tmjET87vzvCTWumEtUgUUlGQ41ZW7ti8p4A5G/fw7KRV7MgLXGvRrll9Lu7bmusHdqBT84Y+pxSR2lBRkJAqK3d8uGAr09fuYvWO/Sz7Og+Aod2a850hmZx3YgudhxCJYCoK4hnnHFv2FPLWV5t4Z84W9hWU0KNlY569th+922qcJ5FIpKIgYXGwtIwnJ6xkzIyNADRKSeLuYV24ZUjHGocBEZHwUlGQsPp81U7+PnMT/125E4DWaamc07MFo87qQvv0Bj6nExEVBfFFQXEpU1fn8IsPlrL7QDEQuJL69jM6kaheSyK+UVEQXx06Mf3nqetYvSO/okvrRb1b6YS0iA9UFCQiOOf4cOFWHv7XUgpLyji96wn86JxuDO58gt/RROJKbYuCLlEVT5kZV/Zvx8yfn8ONgzrw5drd3PzqbN6ctYmikjK/44nIEVQUJCyaNqjH767sw/xHzqdXmyY88uFSRr4yg515RX5HE5FKVBQkrNIb1uPDH5zOKzcPYH3OAYY9M4WHxi6mpKz86E8WEc9pYH0Ju4QEY3jv1nRIb8hLU9byzzlbWLEtj9HfyaJlk1S/44nENR0piG96tWnCizcO4Mmr+rDs6zxuGD2L3MISv2OJxDUVBfHdDad24MUbB7Bh9wEe/3i533FE4pqKgkSE4b1bcdeZnXlvXjZ//XID0dZVWiRWqChIxHjwgh4M7pzOo+OW8/j4FX7HEYlLKgoSMZITExjz3VO5rF8bXpu+gWlrcvyOJBJ3VBQkoqQmJ/LkVX3o0bIxP3x7ARt3HfA7kkhcUVGQiNMwJYlXb82ioLiUS1+Yrh5JImGkoiARqX16Ax65pBf7D5byW/VIEgkbFQWJWN8Zksm1We34YMFWtu4r9DuOSFxQUZCIdu+53XDO8bfgzG4i4i3PioKZtTezz81suZktM7P7qtjGzOx5M1trZovNbIBXeSQ6tWvWgEv7tWHMlxtZtX2/33FEYp6XRwqlwAPOuV7AYOAeM+t1xDYXAd2Ct7uAlz3MI1HqkUt60SAlkd9+vFwXtYl4zLOi4Jzb5pybH1zeD6wA2h6x2eXAGy5gFtDUzFp7lUmiU/NGKfzk/O5MX7uLC/44lX0FxX5HEolZYTmnYGaZQH9g9hEPtQW2VLqfzbcLhwi3DO7I4M7prNmZz8mPTWZbrk48i3jB86JgZo2AscD9zrm8Y3yNu8xsrpnNzcnRVa7xyMx4+87BDO3WHICRL8+kuFRzMIiEmqdFwcySCRSEt5xz/6pik61A+0r32wXXHcY5N9o5l+Wcy8rIyPAmrES8hATjzTsG8fgVvdm6r5Afv7tQ5xhEQszL3kcGvAascM49V81mHwHfCfZCGgzkOue2eZVJYsNNgzpwVvcMxi/extxNe/2OIxJTvDxSOB24BTjHzBYGbyPMbJSZjQpuMwFYD6wF/gL8wMM8EiPMjJdvHkDjlCRen77B7zgiMcWz6Tidc9MBO8o2DrjHqwwSuxrUS+LGQR3489T1vDV7EzcN6uh3JJGYoCuaJWr98JyuADw3aTWlZTrpLBIKKgoStRqnJvPMNf3YfaCYz1bu9DuOSExQUZCodmm/1mQ0TuG345ezeXeB33FEop6KgkS1lKREXrl5AHvyi3nqU03hKXK8VBQk6p3SMZ1rB7ZnwpLtzFi3y+84IlFNRUFiwv3ndad+ciIfzP/WtY8iUgcqChIT0uonc+6JLXhvXjY5+w/6HUckaqkoSMy47bRMAP7w6Up/g4hEMRUFiRlZmemcmpnO56t2UlauMZFEjoWKgsSUW4Z0ZFd+MTPX7fY7ikhUUlGQmHJOzxY0bZDMX6at9zuKSFRSUZCY0jAliRtP7cC0NTk64SxyDFQUJOZc2b8t5Q7+vVDdU0XqSkVBYk63lo3p0zaNx8ev0CQ8InWkoiAxaXjvVgCMX6I5m0TqQkVBYtL3hnamXmIC4xZ97XcUkaiioiAxqV5SAlf2b8vEZTvYla8TziK1paIgMWtE39YAPDpuuc9JRKKHioLErLO6Z9A6LZVxi75m7sY9fscRiQoqChLTRt+SBcDIV2aqJ5JILagoSEzr0y6Nmwd3AGDhln0+pxGJfCoKEvN+emFP6iUm8PFidU8VORoVBYl5h+Za+PusTRSXlvsdRySiqShIXBh5SjsOlpYzZsYGv6OIRDQVBYkL5/RsQa/WTRg9dQP7Cor9jiMSsVQUJC6YGUO7NWdX/kFOfmwyRSVlfkcSiUgqChI37hzauWL59S/1NZJIVVQUJG5kNE5hw5MjaNkkhcnLd/gdRyQiqShIXDEzbji1Aws272Ptzv1+xxGJOCoKEncGdGgGwHnPTfU5iUjkUVGQuHNG1+YVy1+szvExiUjkUVGQuJOQYHz+4DAAJugqZ5HDqChIXOrUvCFZHZsxb/Nev6OIRBQVBYlbl53chrU785m1frffUUQihoqCxK0RfQKT8PxH3VNFKqgoSNxq3iiFgZnNeHX6BsrKNdeCCKgoSJw7uX1TAN6bu8XnJCKRQUVB4tr/DO9JcqIxbvHXfkcRiQgqChLXkhMTKq5w1iB5Ih4WBTN73cx2mtnSah4fZma5ZrYwePuVV1lEajKiT2sKissYOz/b7ygivvPySGEMMPwo20xzzp0cvD3mYRaRag3qlE7fdmm8Nl0jp4p4VhScc1OBPV69vkiomBlX9W/L+pwDZO8t8DuOiK/8PqcwxMwWmdknZnZSdRuZ2V1mNtfM5ubkaKwaCb3TguMhvTV7s89JRPzlZ1GYD3R0zvUDXgA+rG5D59xo51yWcy4rIyMjbAElfnRv2ZirB7TjlS/WaZA8iWu+FQXnXJ5zLj+4PAFINrPmR3maiGceu/wkumQ04lf/XopzuphN4pNvRcHMWpmZBZdPDWbRIDTim4YpSdx9Vhc27S7Q0YLELS+7pP4DmAn0MLNsM7vDzEaZ2ajgJiOBpWa2CHgeuN7pzzPx2WUnt6FJahL/XqiL2SQ+JXn1ws65G47y+IvAi17tX+RYJCcmcEX/trw5axP3nduNzOYN/Y4kElZ+9z4SiTijzuqCczDsmSnsyCvyO45IWKkoiByhTdP6XHFyGwCufnkGuYUlPicSCZ9aFQUzu8/MmljAa2Y238wu8DqciF/+9/r+AGTvLaTfo5N4bvJqnxOJhEdtjxRud87lARcAzYBbgKc8SyUSAZ6/of83y/9dw6dLt/uYRiQ8alsULPjvCOBN59yySutEYtJl/dqw8amLefCC7gBM1gxtEgdqWxTmmdkkAkVhopk1Bsq9iyUSOX54TjcGd05n7PxsDa8tMa+2ReEO4CFgoHOuAEgGvutZKpEIc/WAdgC8qxnaJMbVtigMAVY55/aZ2c3AL4Fc72KJRJYr+7clvWE9fvXvZewrKPY7johnalsUXgYKzKwf8ACwDnjDs1QiESYpMYEnrugNwN9nbfI5jYh3alsUSoNDUFwOvOic+xPQ2LtYIpHnoj6t6dmqMf+av9XvKCKeqW1R2G9mPyfQFXW8mSUQOK8gElfO7J7B+l0HmLtR80dJbKptUbgOOEjgeoXtQDvgac9SiUSoe4Z1BWDkKzN9TiLijVoVhWAheAtIM7NLgCLnnM4pSNxJa/DNAfKCzXt9TCLijdoOc3Et8BVwDXAtMNvMRnoZTCRSzX74XAAeeG+Rz0lEQq+2Q2f/gsA1CjsBzCwD+A/wvlfBRCJVyyapAKzPOcCs9bsZ3PkEnxOJhE5tzykkHCoIQbvr8FyRmPP29wYBcP3oWTw3aZXPaURCp7Yf7J+a2UQzu83MbgPGAxO8iyUS2U7r0pwx3x0IwPOfrWXtzv0+JxIJjdqeaP4pMBroG7yNds79zMtgIpFuWI8WvHTTAADemKkL2iQ21Ho6TufcWGCsh1lEos6IPq3p2y6NN2Zu4tJ+bRiYme53JJHjUuORgpntN7O8Km77zSwvXCFFItkdZ3QC4OlPdW5Bol+NRwrOOQ1lIXIUl5/clsnLdzB97S7Kyh2JCZpqRKKXehCJhMDw3q3YV1DCwi26oE288dDYxYxfvM3z/agoiITA0G4ZJCYYnyzRlJ0SekUlZfxzzhbW5eR7vi8VBZEQSKufzNBuzXl1+gZWbNPpNgmt7L2FALRPr+/5vlQURELkoYt6AvCm5luQENuytwCADukNPN+XioJIiPRs1YSrBrTl7dmb2bDrgN9xJIZs21cEQJumOlIQiSqH5nI++5kp/gaRmLI9rwgzyGiU4vm+VBREQmhIpcHxApMVihy/HblFNG+UQlKi9x/ZKgoiIZSQYPz+6j4ALNyyz+c0Eiu25xXRKjg6r9dUFERCbGi3DACufGkGny5VF1U5fjvyimjZxPuvjkBFQSTk2jStz82DOwDw0/c1EY8cH+ccK7fvp1mDemHZn4qCiAcev6IPl/Vrw/6iUt6Zs5nSsnK/I0mUWhD8GrIsTOeoVBREPHLToMDRws/GLuEXHyz1OY1Eq43B7s2jzuoSlv2pKIh4ZFDnEyq+B56w1PsxayQ2fb0vcDVzOC5cAxUFEU/N+vm5/PLiE9lfVMqOvCK/40gUyisqJTU5gdTkxLDsT0VBxENmRlZw4p3Hxi33OY1Eo7zCEpqkJodtfyoKIh7r1y4NgPFLtrFlT4HPaSTa5BaW0KS+ioJIzDAzXg7O5fzDt+f7nEaiTV5RCWmxUBTM7HUz22lmVXa7sIDnzWytmS02swFeZRHx20V9WtOySQqLsnOZujrH7zgSRfIKS2mSWuMkmSHl5ZHCGGB4DY9fBHQL3u4CXvYwi4jvfnlxLwBe/Hytz0kkWjjn+HpfIekNw3M1M3hYFJxzU4E9NWxyOfCGC5gFNDWz1l7lEfHbpf3aMKJPK+Zt2quL2aRW9haUsPtAMT1bNQ7bPv08p9AW2FLpfnZw3beY2V1mNtfM5ubk6NBbotfAzHTKygPDFogczecrdwLhmUfhkKg40eycG+2cy3LOZWVkZPgdR+SYnXdiSwDmbKzpIFokYFtu4MK1wZ3Tw7ZPP4vCVqB9pfvtgutEYlb79AbUT07k0XHLySsq8TuORLDC4jKembQagBPCMLnOIX4WhY+A7wR7IQ0Gcp1zGgtAYt7lJ7cB0HhIUqOnPlnhy3697JL6D2Am0MPMss3sDjMbZWajgptMANYDa4G/AD/wKotIJHniysAkPOMWfc2KbXk+p5FI9cGCwBcnKx6rqRNn6HnW+dU5d8NRHnfAPV7tXyRSJSYYF/VuxSdLt/Pjdxby6f1n+h1JIsy+gmLyikrp174p9euFZ8yjQ6LiRLNIrHn55lPo1qIRK7fvZ9nXuX7HkQizOTgcyvUD2x9ly9BTURDxySOXBC5mu/j56RQWl/mcRiLJ2p35APQI4/UJh6goiPjkzO4ZXNk/cGnO85+t8TmNRJKtewNdUds1C9/1CYeoKIj46MmrAiedX56yjn0FxT6nkUiQf7CUZycHuqK2aJwa9v2rKIj4KDU5seLCpA8X6DIdgffmbjn6Rh5SURDx2V9vOxWALcGvDCS+TVuzC4A7z+jky/5VFER8Vr9eImbw2vQNFJdqoLx45pzjs+B4R78MdkQINxUFkQhwVf92APxnxQ6fk4ifXvliPQDXZYW/K+ohKgoiEeB3V/UG4AdvzSdwXafEo3eD5xPuOquzbxlUFEQiQEpSYsXsWp+v2ulzGvHD+px8Nuw6QJu0VLpkNPIth4qCSISY/fB5NEpJ4q9fbvQ7ivjgxr/MBuCBC3r4mkNFQSRC1K+XyK2ndWTaml3MWLvL7zgSZofGOLr6lHa+5lBREIkgNw3qCMD4JRpFPp4459iZV8Rtp2X6HUVFQSSStGlan/NObMFbszezdKsGyosXb87axIHiMrq19O9cwiEqCiIR5p6zuwJwyQvTmbdJ03bGMuccHy7Yyq/+vQyAAR2a+ZxIRUEk4vTv0IxBnQJDX1z98kxyCzRtZ6wat3gb97+zEIB+7ZtyYusmPidSURCJSO98f0jF8utfbvAxiXhp7sZvjgSHdc/wMck3VBREItS6342gZ6vGvDFzoy5oi1FfbdhD33ZpPD2yL/ef183vOICKgkjESkwwrslqz96CEjr9fAKlZRoXKZbsOVDMyu37ufCkVlyT1R4z8zsSoKIgEtFuGdyxYnn0tPU+JpFQ25YbGBW3c/OGPic5nIqCSASrl5TAyt8OB2DcIl27EEvW7AhMudkpQ0VBROogNTmR20/vxIpteewvUk+kWJG9twCAjukqCiJSR0O7NwfgjN9/7nMSCYW9B4p5ZlJgys1Dw1tEChUFkShwVrdAd8XcwhLKytUTKdoNeHyy3xGqpaIgEgUSEoxHgjNxdXl4ArvyD/qcSI7Vvxdu5VAP47VPXORvmCqoKIhEiZsGdajoqfLQ2CU+p5G6+nLtLp76ZCX3/TNwBfO/7zmdpMTI+wiOvEQiUqXU5EQ+e3AYEJi2s6C41N9AUmvPTV7NTa/O5pUv1gHwt9tPpV/7pj6nqpqKgkiU+cf3BgPwxsxNPieR2igqKeP5/645bN1ZETKkRVVUFESizODO6SQlGE99spIDB3W0EOlGTw1cdHj3sC50yWjI23cO8jlRzVQURKKMmTHqrC4AvBec6F0i14x1gVn0/ufCHvz3gWGc1rW5z4lqpqIgEoUeuKA7jVKSmLR8h99R5CgSEwJjGkXK2EZHo6IgEoXMjB+c3YUZ63bz5iydW4hUzjnmbNjL8JNa+R2l1lQURKLUd0/rBMAT45f7nESq84eJqyguK6dHq8Z+R6k1FQWRKFW/XiK/vrQXRSXlnP/cF37HkSMUFJfy8pRAF9TrBrb3OU3tqSiIRLGbBgWG1l6zM5//+8+ao2wt4XKwtIxev5oIBM4ptGla3+dEtaeiIBLF6iUl8OE9pwPwx/+sZua63T4nksLiMnr88tOK+5E4lEVNVBREotzJ7Zsy9u7TALj19a/ILdTw2n6qPKf2Vw+fGzW9jg5RURCJAad0bMYfRvaluKycfo9OoqikzO9IcWnSsu08PXEVAH+6cQAtmqT6nKjuPC0KZjbczFaZ2Voze6iKx28zsxwzWxi83ellHpFYdmnfNiQF+8Rf9uJ0Nu464HOi+HLx89O46815AFzWrw0X923tc6Jj41lRMLNE4E/ARUAv4AYz61XFpu84504O3l71Ko9IrKtfL5E1T1xEgsHqHfkMe2YKM9ftxjnNv+C1guJSln2dV3H/Zxf19DHN8fHySOFUYK1zbr1zrhj4J3C5h/sTiXtmxuyHz6u4f8NfZtHp5xP4zUfLVBw8NOiJ/wLw+BW9mf/I+bSNot5GR/KyKLQFKg/Mkh1cd6SrzWyxmb1vZtHTmVckQmU0TmHjUxfTp21axboxMzbys7GLfUwV2/YHBya8tF8b0hvW8znN8fH7RPM4INM51xeYDPytqo3M7C4zm2tmc3NycsIaUCRa/f2OQUx5cBj/e93JALw7N5sPFmTz3KRVOmoIofU5+QDcc3YX0uon+5zm+CV5+Npbgcp/+bcLrqvgnKvcqfpV4A9VvZBzbjQwGiArK0v/m0VqIa1BMmkNksls3pAm9ZO4fcxcfvzOIgDapzfgmiwdmIfC94Mnl8/oGrlzJNSFl0cKc4BuZtbJzOoB1wMfVd7AzCqfnr8MWOFhHpG4dU7PlhWT8wD89P3FlJaV+5goNrw/L5s1O/NJMBjS5QS/44SEZ0XBOVcK/BCYSODD/l3n3DIze8zMLgtudq+ZLTOzRcC9wG1e5RGJd0O6nMDGpy7m4RGBnjFdf/EJmQ+NZ9nXufo6qY5Ky8p5euJKHnwvcOQ1+Sdn+ZwodCza/jNkZWW5uXPn+h1DJGo55+j08wnfWv+nGwdEbd/6cFq6NZdLXphecb9BvUSWPzbcx0S1Y2bznHNZR9vO7xPNIhJmZsaKx4Yz8f4zufCklhXr73l7Pp+v2sm8TXt15FCNsnJ3WEE4p2cLJtw71MdEoacjBRHhucmrvzW5PMATV/auGIm1rNxx7z8WcOOgDpwe4VNKemXgE/8hZ/9BAL746TA6ntDQ50S1pyMFEam1n5zfnV9f+u0BB37xwVIyHxrPm7M20eXhCYxfso2bXp3tQ0L/bd5dUFEQFjxyflQVhLrQkYKIHKaopIw/f7Ge4rLAv6Xl3/6MOLN7BkO7NmdIlxPo2qIRqcmJPiQNn+tHz2TW+j0A/OsHpzGgQzOfE9VdbY8UvLxOQUSiUGpyIved1w2A+87tznOTV/PKF4EZxCbcO5QRz09j6uocpq7+5kLSV24+heG9o2ce4rpwzlUUBID+7Zv6mMZ7+vpIRKpVLymBhy7qycanLmbjUxfTq00Tfnphj29tN+rv89iVf5CSKLv2wTlHfnCICoAV2/K46P+msfdAMQDl5Y4/Tl4NwO+u7MPGpy6OuvkR6kpfH4nIMVufk8+lL0znQPE38zf85Pzu3Dm0E/WTEzEzZq7bzT++2sxHi76mZZMUduQd5Lqs9vx+ZF8fk0P+wVJ6/zowZeYD53dn9LT17C/6pkCc2T2Dzs0bMmbGRgA+/tEZ9K40nlS0qe3XRyoKInLczn12Cutyvj1/w73ndquyVxPA1QPa8ey1/byOVmF7bhHPTFpFSVk595zdlQv+OLXWzz01M523vzeIpMTo/XJFRUFEwqaopIxJy3dwUpsmXPfnWezKP/itbc7t2YLisnKuHtCO+99ZCED/Dk157LLeLMzex82DOpB/sJTZ6/dw5xuB3/H3Rw0hKzO9xn0757jp1dnkFpbw4IU9aN+sAXsLinl64ireunMQ63LyeenzdXy06Osqn9+/Q1MWbN4HwKBO6bzz/SFk7y3gjN9/DsCn9w+lZ6smx/yziRQqCiLiqy9W53Dr619xZf+2/DE4Uusha3fu57znaveX+mX92vD8Df0B2LT7AGn1k2na4Jvhqfv+ZiJ5lb72qck1p7RjwpJtFV93bXzq4lo9LxaoKIhIRCssLuPEX31a5WODO6dzUps0Xpu+AYAfnt2Vu4d14aTgOYCxd5/Gkux9/Gbc8lrv7/ErenPz4I4UFpfx+Pjl3DKkY0wcAdSWioKIRI3SsnIWb81l0+4DXNm/XcX6bbmFDHnys6M+v/LVxf/4ajOTlm3ntVsHApBXVHLYkUW8UlEQkZiwZU8BQ/8Q+H6/W4tGtG5av+Iaic7NG/LuqCE0b5TiZ8SooIvXRCQmtE9vwMMjerJmRz5PXxO+3krxSkVLrvP3AAAJHUlEQVRBRCLeXWd28TtC3IjeTrciIhJyKgoiIlJBRUFERCqoKIiISAUVBRERqaCiICIiFVQURESkgoqCiIhUiLphLswsB9gEpAG5lR6qfL+q5ebAruPc/ZH7rOt2tV0f7rYdb7uqe+xo62qz7HfbIvU9qylbbbeJ1fesusdi9T2ran1V+Ts65zKOmtA5F5U3YHR196taBuaGep913a6268PdtuNtV3WPHW1dLZd9bVukvme1bVs8vme1bVusvGd1aUttbtH89dG4Gu5XtxzqfdZ1u9quD3fbjrdd1T12tHV6z45PbV4rHt+z6h6L1fesqvXHnD/qvj46VmY219VihMBopLZFn1htF8Ru22K1XUeK5iOFuhrtdwAPqW3RJ1bbBbHbtlht12Hi5khBRESOLp6OFERE5ChUFEREpIKKgoiIVFBREBGRCioKgJkNNbNXzOxVM5vhd55QMrMEM3vCzF4ws1v9zhMqZjbMzKYF37dhfucJNTNraGZzzewSv7OEipmdGHy/3jezu/3OE0pmdoWZ/cXM3jGzC/zOczyiviiY2etmttPMlh6xfriZrTKztWb2UE2v4Zyb5pwbBXwM/M3LvHURirYBlwPtgBIg26usdRGidjkgH0glQtoFIWsbwM+Ad71JWXch+j1bEfw9uxY43cu8dRGitn3onPseMAq4zsu8Xov6LqlmdiaBD4c3nHO9g+sSgdXA+QQ+MOYANwCJwJNHvMTtzrmdwee9C9zhnNsfpvg1CkXbgre9zrk/m9n7zrmR4cpfnRC1a5dzrtzMWgLPOeduClf+moSobf2AEwgUvF3OuY/Dk756ofo9M7PLgLuBN51zb4crf01C/BnyLPCWc25+mOKHXJLfAY6Xc26qmWUesfpUYK1zbj2Amf0TuNw59yRQ5eG4mXUAciOlIEBo2mZm2UBx8G6Zd2lrL1TvWdBeIMWLnMciRO/ZMKAh0AsoNLMJzrlyL3MfTajeM+fcR8BHZjYeiIiiEKL3zICngE+iuSBADBSFarQFtlS6nw0MOspz7gD+6lmi0Klr2/4FvGBmQ4GpXgY7TnVql5ldBVwINAVe9DbacatT25xzvwAws9sIHhF5mu7Y1fU9GwZcRaCIT/A02fGr6+/Zj4DzgDQz6+qce8XLcF6K1aJQZ865X/udwQvOuQICBS+mOOf+RaDgxSzn3Bi/M4SSc24KMMXnGJ5wzj0PPO93jlCI+hPN1dgKtK90v11wXSyI1bbFarsgdtsWq+2C2G5bjWK1KMwBuplZJzOrB1wPfORzplCJ1bbFarsgdtsWq+2C2G5bjaK+KJjZP4CZQA8zyzazO5xzpcAPgYnACuBd59wyP3Mei1htW6y2C2K3bbHaLojtth2LqO+SKiIioRP1RwoiIhI6KgoiIlJBRUFERCqoKIiISAUVBRERqaCiICIiFVQUxHNmlh+GfVxWyyGpQ7nPYWZ22jE8r7+ZvRZcvs3MImLsJjPLPHL46Cq2yTCzT8OVScJPRUGiRnA44yo55z5yzj3lwT5rGh9sGFDnogA8TJSOk+OcywG2mVnEzIcgoaWiIGFlZj81szlmttjMHq20/kMzm2dmy8zsrkrr883sWTNbBAwxs41m9qiZzTezJWbWM7hdxV/cZjbGzJ43sxlmtt7MRgbXJ5jZS2a20swmm9mEQ48dkXGKmf2vmc0F7jOzS81stpktMLP/mFnL4FDLo4Afm9lCC8zel2FmY4Ptm1PVB6eZNQb6OucWVfFYppl9FvzZ/Dc4nDtm1sXMZgXb+3hVR14WmKltvJktMrOlZnZdcP3A4M9hkZl9ZWaNg/uZFvwZzq/qaMfMEs3s6Urv1fcrPfwhEBHzV4gHnHO66ebpDcgP/nsBMBowAn+QfAycGXwsPfhvfWApcELwvgOurfRaG4EfBZd/ALwaXL4NeDG4PAZ4L7iPXgTGxQcYSWDI5gSgFYG5GEZWkXcK8FKl+8345ur/O4Fng8u/AR6stN3bwBnB5Q7Aiipe+2xgbKX7lXOPA24NLt8OfBhc/hi4Ibg86tDP84jXvRr4S6X7aUA9YD0wMLiuCYGRkRsAqcF13YC5weVMYGlw+S7gl8HlFGAu0Cl4vy2wxO//V7p5c9PQ2RJOFwRvC4L3GxH4UJoK3GtmVwbXtw+u301gYqCxR7zOoSGz5xEYn78qH7rAPATLLTA7G8AZwHvB9dvN7PMasr5Tabkd8I6ZtSbwQbuhmuecB/Qys0P3m5hZI+dc5b/sWwM51Tx/SKX2vAn8odL6K4LLbwPPVPHcJcCzZvZ74GPn3DQz6wNsc87NAXDO5UHgqAJ40cxOJvDz7V7F610A9K10JJVG4D3ZAOwE2lTTBolyKgoSTgY86Zz782ErA5OvnAcMcc4VmNkUAlNRAhQ5546cMe5g8N8yqv8/fLDSslWzTU0OVFp+gcCUnx8Fs/6mmuckAIOdc0U1vG4h37QtZJxzq81sADACeNzM/gt8UM3mPwZ2EJj2MwGoKq8ROCKbWMVjqQTaITFI5xQknCYCt5tZIwAza2tmLQj8Fbo3WBB6AoM92v+XwNXBcwstCZworo00vhlL/9ZK6/cDjSvdn0RgBi4Agn+JH2kF0LWa/cwgMEQzBL6znxZcnkXg6yEqPX4YM2sDFDjn/g48DQwAVgGtzWxgcJvGwRPnaQSOIMqBWwjMO3ykicDdZpYcfG734BEGBI4sauylJNFLRUHCxjk3icDXHzPNbAnwPoEP1U+BJDNbQWCe21keRRhLYFrF5cDfgflAbi2e9xvgPTObB+yqtH4ccOWhE83AvUBW8MTscgLf/x/GObeSwJSNjY98jEBB+a6ZLSbwYX1fcP39wE+C67tWk7kP8JWZLQR+DTzunCsGriMwHesiYDKBv/JfAm4NruvJ4UdFh7xK4Oc0P9hN9c98c1R2NjC+iudIDNDQ2RJXDn3Hb2YnAF8Bpzvntoc5w4+B/c65V2u5fQOg0DnnzOx6AiedL/c0ZM15phKYxH6vXxnEOzqnIPHmYzNrSuCE8W/DXRCCXgauqcP2pxA4MWzAPgI9k3xhZhkEzq+oIMQoHSmIiEgFnVMQEZEKKgoiIlJBRUFERCqoKIiISAUVBRERqfD/4x0hnMFgmksAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# find a good learning rate\n",
    "learner.lr_find()\n",
    "learner.lr_plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "begin training using triangular learning rate policy with max lr of 0.001...\n",
      "Epoch 1/3\n",
      "937/937 [==============================] - 12s 13ms/step - loss: 0.1412 - acc: 0.9585 - val_loss: 0.0253 - val_acc: 0.9914\n",
      "Epoch 2/3\n",
      "937/937 [==============================] - 12s 13ms/step - loss: 0.1272 - acc: 0.9620 - val_loss: 0.0247 - val_acc: 0.9916\n",
      "Epoch 3/3\n",
      "937/937 [==============================] - 12s 13ms/step - loss: 0.1172 - acc: 0.9644 - val_loss: 0.0226 - val_acc: 0.9921\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f3e41e7c0f0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train using a triangular learning rate policy\n",
    "learner.autofit(0.001, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get a Predictor instance\n",
    "predictor = ktrain.get_predictor(learner.model, preproc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1032 images belonging to 1 classes.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('2/1.png', '2'),\n",
       " ('2/1002.png', '2'),\n",
       " ('2/1016.png', '2'),\n",
       " ('2/1031.png', '2'),\n",
       " ('2/1036.png', '2'),\n",
       " ('2/1049.png', '2'),\n",
       " ('2/1050.png', '2'),\n",
       " ('2/1053.png', '2'),\n",
       " ('2/1056.png', '2'),\n",
       " ('2/106.png', '2')]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make predictions on new data\n",
    "predictor.predict_folder(DATADIR+'/testing/2')[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}