05b_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "### We will share our pipeline and it's source code on https://github.com/ilos-vigil/scl-2020-product-detection\n",
    "\n",
    "For current best submission (0.85325 on public leaderboard), see https://www.kaggle.com/ilosvigil/scl2020-2-5-model?scriptVersionId=37953296"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# This notebook in nutshell\n",
    "\n",
    "> This notebook source code based on [Ensemble, LAMB & GridMask on TPU](https://www.kaggle.com/ilosvigil/ensemble-lamb-gridmask-on-tpu) version 37 and 43\n",
    "\n",
    "* Multimodal Deep Learning\n",
    "    * Image\n",
    "    * Text (from OCR)\n",
    "* Dynamic image augmentation\n",
    "    * GridMask (https://arxiv.org/abs/2001.04086) from [GridMask data augmentation with tensorflow](https://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow)\n",
    "    * Rotate, shear, zoom, shift from [Rotation Augmentation GPU/TPU - [0.96+]](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)\n",
    "    * [tf.image](https://www.tensorflow.org/api_docs/python/tf/image) functions\n",
    "* TF-IDF word representation\n",
    "    * L2 normalization\n",
    "    * Sublinear Term Frequency\n",
    "* EfficientNet B7 (https://arxiv.org/abs/1905.11946)\n",
    "* Global Average Pooling (https://arxiv.org/abs/1312.4400)\n",
    "* LAMB optimizer (https://arxiv.org/abs/1904.00962)\n",
    "* MLP / Multi Layer Perceptron\n",
    "* TTA / Test Time Augmentation\n",
    "* TPU v3-8"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# Library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting efficientnet\r\n",
      "  Downloading efficientnet-1.1.0-py3-none-any.whl (18 kB)\r\n",
      "Requirement already up-to-date: tensorflow_addons in /opt/conda/lib/python3.7/site-packages (0.10.0)\r\n",
      "Requirement already up-to-date: tensorflow in /opt/conda/lib/python3.7/site-packages (2.2.0)\r\n",
      "Requirement already satisfied, skipping upgrade: scikit-image in /opt/conda/lib/python3.7/site-packages (from efficientnet) (0.16.2)\r\n",
      "Collecting keras-applications<=1.0.8,>=1.0.7\r\n",
      "  Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)\r\n",
      "\u001b[K     |████████████████████████████████| 50 kB 2.2 MB/s \r\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: typeguard>=2.7 in /opt/conda/lib/python3.7/site-packages (from tensorflow_addons) (2.9.1)\r\n",
      "Requirement already satisfied, skipping upgrade: tensorflow-estimator<2.3.0,>=2.2.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (2.2.0)\r\n",
      "Requirement already satisfied, skipping upgrade: scipy==1.4.1; python_version >= \"3\" in /opt/conda/lib/python3.7/site-packages (from tensorflow) (1.4.1)\r\n",
      "Requirement already satisfied, skipping upgrade: wheel>=0.26; python_version >= \"3\" in /opt/conda/lib/python3.7/site-packages (from tensorflow) (0.34.2)\r\n",
      "Requirement already satisfied, skipping upgrade: six>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (1.14.0)\r\n",
      "Requirement already satisfied, skipping upgrade: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (1.1.0)\r\n",
      "Requirement already satisfied, skipping upgrade: tensorboard<2.3.0,>=2.2.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (2.2.2)\r\n",
      "Requirement already satisfied, skipping upgrade: h5py<2.11.0,>=2.10.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (2.10.0)\r\n",
      "Requirement already satisfied, skipping upgrade: wrapt>=1.11.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (1.11.2)\r\n",
      "Requirement already satisfied, skipping upgrade: gast==0.3.3 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (0.3.3)\r\n",
      "Requirement already satisfied, skipping upgrade: grpcio>=1.8.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (1.29.0)\r\n",
      "Requirement already satisfied, skipping upgrade: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (3.2.1)\r\n",
      "Requirement already satisfied, skipping upgrade: protobuf>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (3.12.2)\r\n",
      "Requirement already satisfied, skipping upgrade: numpy<2.0,>=1.16.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (1.18.1)\r\n",
      "Requirement already satisfied, skipping upgrade: keras-preprocessing>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (1.1.2)\r\n",
      "Requirement already satisfied, skipping upgrade: google-pasta>=0.1.8 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (0.2.0)\r\n",
      "Requirement already satisfied, skipping upgrade: astunparse==1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (1.6.3)\r\n",
      "Requirement already satisfied, skipping upgrade: absl-py>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow) (0.9.0)\r\n",
      "Requirement already satisfied, skipping upgrade: matplotlib!=3.0.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-image->efficientnet) (3.2.1)\r\n",
      "Requirement already satisfied, skipping upgrade: networkx>=2.0 in /opt/conda/lib/python3.7/site-packages (from scikit-image->efficientnet) (2.4)\r\n",
      "Requirement already satisfied, skipping upgrade: pillow>=4.3.0 in /opt/conda/lib/python3.7/site-packages (from scikit-image->efficientnet) (5.4.1)\r\n",
      "Requirement already satisfied, skipping upgrade: imageio>=2.3.0 in /opt/conda/lib/python3.7/site-packages (from scikit-image->efficientnet) (2.8.0)\r\n",
      "Requirement already satisfied, skipping upgrade: PyWavelets>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from scikit-image->efficientnet) (1.1.1)\r\n",
      "Requirement already satisfied, skipping upgrade: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (1.6.0.post3)\r\n",
      "Requirement already satisfied, skipping upgrade: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (0.4.1)\r\n",
      "Requirement already satisfied, skipping upgrade: google-auth<2,>=1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (1.14.0)\r\n",
      "Requirement already satisfied, skipping upgrade: requests<3,>=2.21.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (2.23.0)\r\n",
      "Requirement already satisfied, skipping upgrade: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (3.2.1)\r\n",
      "Requirement already satisfied, skipping upgrade: setuptools>=41.0.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (46.1.3.post20200325)\r\n",
      "Requirement already satisfied, skipping upgrade: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (1.0.1)\r\n",
      "Requirement already satisfied, skipping upgrade: cycler>=0.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->efficientnet) (0.10.0)\r\n",
      "Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->efficientnet) (1.2.0)\r\n",
      "Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->efficientnet) (2.4.7)\r\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->efficientnet) (2.8.1)\r\n",
      "Requirement already satisfied, skipping upgrade: decorator>=4.3.0 in /opt/conda/lib/python3.7/site-packages (from networkx>=2.0->scikit-image->efficientnet) (4.4.2)\r\n",
      "Requirement already satisfied, skipping upgrade: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow) (1.2.0)\r\n",
      "Requirement already satisfied, skipping upgrade: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow) (3.1.1)\r\n",
      "Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow) (0.2.7)\r\n",
      "Requirement already satisfied, skipping upgrade: rsa<4.1,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow) (4.0)\r\n",
      "Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow) (3.0.4)\r\n",
      "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow) (1.24.3)\r\n",
      "Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow) (2.9)\r\n",
      "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow) (2020.4.5.2)\r\n",
      "Requirement already satisfied, skipping upgrade: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow) (3.0.1)\r\n",
      "Requirement already satisfied, skipping upgrade: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow) (0.4.8)\r\n",
      "Installing collected packages: keras-applications, efficientnet\r\n",
      "Successfully installed efficientnet-1.1.0 keras-applications-1.0.8\r\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade efficientnet tensorflow_addons tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import re\n",
    "import random\n",
    "import os\n",
    "import tensorflow as tf\n",
    "import tensorflow_addons as tfa\n",
    "import numpy as np\n",
    "import tensorflow.keras.backend as K\n",
    "import efficientnet.tfkeras as efn\n",
    "import efficientnet\n",
    "import itertools\n",
    "import matplotlib\n",
    "import scipy\n",
    "import pandas as pd\n",
    "import sklearn\n",
    "from matplotlib import pyplot as plt\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy version : 1.18.1\n",
      "Tensorflow version : 2.2.0\n",
      "Tensorflow Addons version : 0.10.0\n",
      "EfficientNet (library) version : 1.1.0\n",
      "Matplotlib version : 3.2.1\n",
      "Scipy version : 1.4.1\n",
      "Pandas version : 1.0.3\n",
      "Scikit-Learn version : 0.23.1\n"
     ]
    }
   ],
   "source": [
    "print(f'Numpy version : {np.__version__}')\n",
    "print(f'Tensorflow version : {tf.__version__}')\n",
    "print(f'Tensorflow Addons version : {tfa.__version__}')\n",
    "print(f'EfficientNet (library) version : {efficientnet.__version__}')\n",
    "print(f'Matplotlib version : {matplotlib.__version__}')\n",
    "print(f'Scipy version : {scipy.__version__}')\n",
    "print(f'Pandas version : {pd.__version__}')\n",
    "print(f'Scikit-Learn version : {sklearn.__version__}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip freeze > requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "PRE_TRAINING_TIME_START = datetime.now()\n",
    "AUTO = tf.data.experimental.AUTOTUNE"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## Set seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "SEED = 42\n",
    "\n",
    "os.environ['PYTHONHASHSEED']=str(SEED)\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "\n",
    "os.environ['TF_DETERMINISTIC_OPS']=str(SEED)\n",
    "tf.random.set_seed(SEED)\n",
    "# tf.config.threading.set_inter_op_parallelism_threads(1)\n",
    "# tf.config.threading.set_intra_op_parallelism_threads(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## TPU or GPU detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on TPU  grpc://10.0.0.2:8470\n",
      "REPLICAS:  8\n"
     ]
    }
   ],
   "source": [
    "# Detect hardware, return appropriate distribution strategy\n",
    "try:\n",
    "    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n",
    "    print('Running on TPU ', tpu.master())\n",
    "except ValueError:\n",
    "    tpu = None\n",
    "\n",
    "if tpu:\n",
    "    tf.config.experimental_connect_to_cluster(tpu)\n",
    "    tf.tpu.experimental.initialize_tpu_system(tpu)\n",
    "    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n",
    "else:\n",
    "    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n",
    "\n",
    "print(\"REPLICAS: \", strategy.num_replicas_in_sync)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 24K\r\n",
      "drwxr-xr-x 6 root   root    4.0K Jul  4 07:34 .\r\n",
      "drwxr-xr-x 6 root   root    4.0K Jul  4 07:34 ..\r\n",
      "drwxr-xr-x 4 nobody nogroup 4.0K Jun 24 14:24 shopee-product-detection-student\r\n",
      "drwxr-xr-x 2 nobody nogroup 4.0K Jul  1 11:57 tfrecords\r\n",
      "drwxr-xr-x 2 nobody nogroup 4.0K Jul  1 11:56 tfrecords-2\r\n",
      "drwxr-xr-x 2 nobody nogroup 4.0K Jul  1 11:57 tfrecords-3\r\n"
     ]
    }
   ],
   "source": [
    "!ls -lha /kaggle/input/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "104\n",
      "12\n"
     ]
    }
   ],
   "source": [
    "from kaggle_datasets import KaggleDatasets\n",
    "IMAGE_SIZE = (512, 512)\n",
    "\n",
    "GCS_TRAIN_PATHS = [\n",
    "    KaggleDatasets().get_gcs_path('tfrecords'),\n",
    "    KaggleDatasets().get_gcs_path('tfrecords-2')\n",
    "]\n",
    "TRAINING_FILENAMES = []\n",
    "for i in GCS_TRAIN_PATHS:\n",
    "    TRAINING_FILENAMES.append(tf.io.gfile.glob(i + '/*.tfrecords'))\n",
    "TRAINING_FILENAMES = list(itertools.chain.from_iterable(TRAINING_FILENAMES))\n",
    "\n",
    "GCS_TEST_PATH = KaggleDatasets().get_gcs_path('tfrecords-3')\n",
    "TEST_FILENAMES = tf.io.gfile.glob(GCS_TEST_PATH + '/*.tfrecords') # predictions on this dataset should be submitted for the competition\n",
    "\n",
    "print(len(TRAINING_FILENAMES))\n",
    "print(len(TEST_FILENAMES))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "EPOCHS = 12\n",
    "DO_AUG = True\n",
    "\n",
    "BATCH_SIZE = 256\n",
    "current_epoch = 0 # used to determine augmentation rate\n",
    "chance = 0\n",
    "\n",
    "NUM_TRAINING_IMAGES = 105390\n",
    "NUM_TEST_IMAGES = 12186\n",
    "STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASSES = [str(c).zfill(2) for c in range(0, 42)]"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# TFRecord function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_image(image_data):\n",
    "    image = tf.image.decode_jpeg(image_data, channels=3)\n",
    "    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n",
    "    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n",
    "    return image\n",
    "\n",
    "def read_labeled_tfrecord(example):\n",
    "    LABELED_TFREC_FORMAT = {\n",
    "        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n",
    "        \"words\": tf.io.FixedLenFeature([6633], tf.float32),  # shape [] means single element\n",
    "        \"label\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n",
    "    }\n",
    "    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n",
    "    image = decode_image(example['image'])\n",
    "    words = example['words']\n",
    "    label = tf.cast(example['label'], tf.int32)\n",
    "    \n",
    "    return ((image, words), label) # returns a dataset of (image, label) pairs\n",
    "\n",
    "def read_unlabeled_tfrecord(example):\n",
    "    UNLABELED_TFREC_FORMAT = {\n",
    "        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n",
    "        \"words\": tf.io.FixedLenFeature([6633], tf.float32),  # shape [] means single element\n",
    "        \"filename\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n",
    "        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n",
    "    }\n",
    "    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n",
    "    image = decode_image(example['image'])\n",
    "    words = example['words']\n",
    "    filename = example['filename']\n",
    "    return ((image, words), filename) # returns a dataset of image(s)\n",
    "\n",
    "def load_dataset(filenames, labeled=True, ordered=False):\n",
    "    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n",
    "    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n",
    "\n",
    "    ignore_order = tf.data.Options()\n",
    "    if not ordered:\n",
    "        ignore_order.experimental_deterministic = False # disable order, increase speed\n",
    "\n",
    "    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n",
    "    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n",
    "    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n",
    "    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n",
    "    return dataset\n",
    "\n",
    "def get_training_dataset(do_aug=True):\n",
    "    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n",
    "    if do_aug:\n",
    "        dataset = dataset.map(image_augmentation, num_parallel_calls=AUTO)\n",
    "    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n",
    "    dataset = dataset.shuffle(2048)\n",
    "    dataset = dataset.batch(BATCH_SIZE)\n",
    "    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n",
    "    return dataset\n",
    "\n",
    "def get_test_dataset(ordered=False, tta=None):\n",
    "    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n",
    "    if tta == 0:\n",
    "        dataset = dataset.map(image_augmentation_tta_0, num_parallel_calls=AUTO)    \n",
    "    elif tta == 1:\n",
    "        dataset = dataset.map(image_augmentation_tta_1, num_parallel_calls=AUTO)\n",
    "    elif tta == 2:\n",
    "        dataset = dataset.map(image_augmentation_tta_2, num_parallel_calls=AUTO)\n",
    "    dataset = dataset.batch(BATCH_SIZE)\n",
    "    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# Image augmentation functions\n",
    "\n",
    "* Chance of applying image augmentation based on training epoch with this formula\n",
    "* `maxval` for most augmentation is 11\n",
    "\n",
    "$$\n",
    "\\text{chance} =\n",
    "\\begin{cases}\n",
    "    0 & \\text{epoch} < 2 \\\\\n",
    "    epoch - 1 & \\text{epoch} < 9 \\\\\n",
    "    8 & \\text{epoch} \\geq 9\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\text{chance'} = \\frac{\\text{chance}}{\\text{maxval}}\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "| Function   | Sub-chance | Range                                    |\n",
    "| ---------- | ---------- | ---------------------------------------- |\n",
    "| Flip       | -          | Left to right                            |\n",
    "| Brightness | -          | 0.9 to 1.1                               |\n",
    "| Contrast   | -          | 0.9 to 1.1                               |\n",
    "| Saturation | -          | 0.95 to 1.05                             |\n",
    "| Hue        | -          | 0.95 to 1.05                             |\n",
    "| Rotate     | 50%        | 17 degrees * random normal distribution  |\n",
    "| Shear      | 50%        | 5.5 degrees * random normal distribution |\n",
    "| Zoom Out   | 33%        | 1.0 - (random normal distribution / 8.5) |\n",
    "| Shift      | 33%        | 18 pixel * random normal distribution    |\n",
    "| GridMask   | -          | d (square size) : 112 - 352 px           |\n",
    "|            |            | r (keep ratio) : 0.4                     |\n",
    "|            |            | 45 degress * random normal distribution  |"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## Rotate, shear, zoom, shift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n",
    "    # returns 3x3 transformmatrix which transforms indicies\n",
    "        \n",
    "    # CONVERT DEGREES TO RADIANS\n",
    "    rotation = math.pi * rotation / 180.\n",
    "    shear = math.pi * shear / 180.\n",
    "    \n",
    "    # ROTATION MATRIX\n",
    "    c1 = tf.math.cos(rotation)\n",
    "    s1 = tf.math.sin(rotation)\n",
    "    one = tf.constant([1],dtype='float32')\n",
    "    zero = tf.constant([0],dtype='float32')\n",
    "    rotation_matrix = tf.reshape( tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3] )\n",
    "        \n",
    "    # SHEAR MATRIX\n",
    "    c2 = tf.math.cos(shear)\n",
    "    s2 = tf.math.sin(shear)\n",
    "    shear_matrix = tf.reshape( tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3] )    \n",
    "    \n",
    "    # ZOOM MATRIX\n",
    "    zoom_matrix = tf.reshape( tf.concat([one/height_zoom,zero,zero, zero,one/width_zoom,zero, zero,zero,one],axis=0),[3,3] )\n",
    "    \n",
    "    # SHIFT MATRIX\n",
    "    shift_matrix = tf.reshape( tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3] )\n",
    "    \n",
    "    return K.dot(K.dot(rotation_matrix, shear_matrix), K.dot(zoom_matrix, shift_matrix))\n",
    "\n",
    "\n",
    "@tf.function\n",
    "def transform(image):\n",
    "    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n",
    "    # output - image randomly rotated, sheared, zoomed, and shifted\n",
    "    DIM = IMAGE_SIZE[0]\n",
    "    XDIM = DIM%2 #fix for size 331\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=2, dtype=tf.int32, seed=SEED) == 0: # 50% chance\n",
    "        rot = 17. * tf.random.normal([1],dtype='float32')\n",
    "    else:\n",
    "        rot = tf.constant([0],dtype='float32')\n",
    "    \n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=2, dtype=tf.int32, seed=SEED) == 0: # 50% chance\n",
    "        shr = 5.5 * tf.random.normal([1],dtype='float32') \n",
    "    else:\n",
    "        shr = tf.constant([0],dtype='float32')\n",
    "    \n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=3, dtype=tf.int32, seed=SEED) == 0: # 33% chance\n",
    "        h_zoom = tf.random.normal([1],dtype='float32')/8.5\n",
    "        if h_zoom > 0:\n",
    "            h_zoom = 1.0 + h_zoom * -1\n",
    "        else:\n",
    "            h_zoom = 1.0 + h_zoom\n",
    "    else:\n",
    "        h_zoom = tf.constant([1],dtype='float32')\n",
    "    \n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=3, dtype=tf.int32, seed=SEED) == 0: # 33% chance\n",
    "        w_zoom = tf.random.normal([1],dtype='float32')/8.5\n",
    "        if w_zoom > 0:\n",
    "            w_zoom = 1.0 + w_zoom * -1\n",
    "        else:\n",
    "            w_zoom = 1.0 + w_zoom\n",
    "    else:\n",
    "        w_zoom = tf.constant([1],dtype='float32')\n",
    "    \n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=3, dtype=tf.int32, seed=SEED) == 0: # 33% chance\n",
    "        h_shift = 18. * tf.random.normal([1],dtype='float32') \n",
    "    else:\n",
    "        h_shift = tf.constant([0],dtype='float32')\n",
    "    \n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=3, dtype=tf.int32, seed=SEED) == 0: # 33% chance\n",
    "        w_shift = 18. * tf.random.normal([1],dtype='float32') \n",
    "    else:\n",
    "        w_shift = tf.constant([0],dtype='float32')\n",
    "  \n",
    "    # GET TRANSFORMATION MATRIX\n",
    "    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n",
    "\n",
    "    # LIST DESTINATION PIXEL INDICES\n",
    "    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n",
    "    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n",
    "    z = tf.ones([DIM*DIM],dtype='int32')\n",
    "    idx = tf.stack( [x,y,z] )\n",
    "    \n",
    "    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n",
    "    idx2 = K.dot(m,tf.cast(idx,dtype='float32'))\n",
    "    idx2 = K.cast(idx2,dtype='int32')\n",
    "    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n",
    "    \n",
    "    # FIND ORIGIN PIXEL VALUES           \n",
    "    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n",
    "    d = tf.gather_nd(image,tf.transpose(idx3))\n",
    "        \n",
    "    return tf.reshape(d,[DIM,DIM,3])"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## GridMask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def transform_grid_mark(image, inv_mat, image_shape):\n",
    "    h, w, c = image_shape\n",
    "    \n",
    "    cx, cy = w//2, h//2\n",
    "\n",
    "    new_xs = tf.repeat( tf.range(-cx, cx, 1), h)\n",
    "    new_ys = tf.tile( tf.range(-cy, cy, 1), [w])\n",
    "    new_zs = tf.ones([h*w], dtype=tf.int32)\n",
    "\n",
    "    old_coords = tf.matmul(inv_mat, tf.cast(tf.stack([new_xs, new_ys, new_zs]), tf.float32))\n",
    "    old_coords_x, old_coords_y = tf.round(old_coords[0, :] + tf.cast(w, tf.float32)//2.), tf.round(old_coords[1, :] + tf.cast(h, tf.float32)//2.)\n",
    "    old_coords_x = tf.cast(old_coords_x, tf.int32)\n",
    "    old_coords_y = tf.cast(old_coords_y, tf.int32)    \n",
    "\n",
    "    clip_mask_x = tf.logical_or(old_coords_x<0, old_coords_x>w-1)\n",
    "    clip_mask_y = tf.logical_or(old_coords_y<0, old_coords_y>h-1)\n",
    "    clip_mask = tf.logical_or(clip_mask_x, clip_mask_y)\n",
    "\n",
    "    old_coords_x = tf.boolean_mask(old_coords_x, tf.logical_not(clip_mask))\n",
    "    old_coords_y = tf.boolean_mask(old_coords_y, tf.logical_not(clip_mask))\n",
    "    new_coords_x = tf.boolean_mask(new_xs+cx, tf.logical_not(clip_mask))\n",
    "    new_coords_y = tf.boolean_mask(new_ys+cy, tf.logical_not(clip_mask))\n",
    "\n",
    "    old_coords = tf.cast(tf.stack([old_coords_y, old_coords_x]), tf.int32)\n",
    "    new_coords = tf.cast(tf.stack([new_coords_y, new_coords_x]), tf.int64)\n",
    "    rotated_image_values = tf.gather_nd(image, tf.transpose(old_coords))\n",
    "    rotated_image_channel = list()\n",
    "    for i in range(c):\n",
    "        vals = rotated_image_values[:,i]\n",
    "        sparse_channel = tf.SparseTensor(tf.transpose(new_coords), vals, [h, w])\n",
    "        rotated_image_channel.append(tf.sparse.to_dense(sparse_channel, default_value=0, validate_indices=False))\n",
    "\n",
    "    return tf.transpose(tf.stack(rotated_image_channel), [1,2,0])\n",
    "\n",
    "\n",
    "@tf.function\n",
    "def random_rotate(image, angle, image_shape):\n",
    "    def get_rotation_mat_inv(angle):\n",
    "          #transform to radian\n",
    "        angle = math.pi * angle / 180\n",
    "\n",
    "        cos_val = tf.math.cos(angle)\n",
    "        sin_val = tf.math.sin(angle)\n",
    "        one = tf.constant([1], tf.float32)\n",
    "        zero = tf.constant([0], tf.float32)\n",
    "\n",
    "        rot_mat_inv = tf.concat([cos_val, sin_val, zero,\n",
    "                                     -sin_val, cos_val, zero,\n",
    "                                     zero, zero, one], axis=0)\n",
    "        rot_mat_inv = tf.reshape(rot_mat_inv, [3,3])\n",
    "\n",
    "        return rot_mat_inv\n",
    "    angle = float(angle) * tf.random.normal([1],dtype='float32')\n",
    "    rot_mat_inv = get_rotation_mat_inv(angle)\n",
    "    return transform_grid_mark(image, rot_mat_inv, image_shape)\n",
    "\n",
    "\n",
    "@tf.function\n",
    "def grid_mask():\n",
    "    h = tf.constant(IMAGE_SIZE[0], dtype=tf.float32)\n",
    "    w = tf.constant(IMAGE_SIZE[1], dtype=tf.float32)\n",
    "    \n",
    "    image_height, image_width = (h, w)\n",
    "    d1 = 112 # 105\n",
    "    d2 = 352 # 210\n",
    "    rotate_angle = 45\n",
    "    ratio = 0.6 # this is delete ratio, so keep ratio = 1 - delete ratio\n",
    "\n",
    "    hh = tf.math.ceil(tf.math.sqrt(h*h+w*w))\n",
    "    hh = tf.cast(hh, tf.int32)\n",
    "    hh = hh+1 if hh%2==1 else hh\n",
    "    d = tf.random.uniform(shape=[], minval=d1, maxval=d2, dtype=tf.int32)\n",
    "    l = tf.cast(tf.cast(d,tf.float32)*ratio+0.5, tf.int32)\n",
    "\n",
    "    st_h = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n",
    "    st_w = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n",
    "\n",
    "    y_ranges = tf.range(-1 * d + st_h, -1 * d + st_h + l)\n",
    "    x_ranges = tf.range(-1 * d + st_w, -1 * d + st_w + l)\n",
    "\n",
    "    for i in range(0, hh//d+1):\n",
    "        s1 = i * d + st_h\n",
    "        s2 = i * d + st_w\n",
    "        y_ranges = tf.concat([y_ranges, tf.range(s1,s1+l)], axis=0)\n",
    "        x_ranges = tf.concat([x_ranges, tf.range(s2,s2+l)], axis=0)\n",
    "\n",
    "    x_clip_mask = tf.logical_or(x_ranges <0 , x_ranges > hh-1)\n",
    "    y_clip_mask = tf.logical_or(y_ranges <0 , y_ranges > hh-1)\n",
    "    clip_mask = tf.logical_or(x_clip_mask, y_clip_mask)\n",
    "\n",
    "    x_ranges = tf.boolean_mask(x_ranges, tf.logical_not(clip_mask))\n",
    "    y_ranges = tf.boolean_mask(y_ranges, tf.logical_not(clip_mask))\n",
    "\n",
    "    hh_ranges = tf.tile(tf.range(0,hh), [tf.cast(tf.reduce_sum(tf.ones_like(x_ranges)), tf.int32)])\n",
    "    x_ranges = tf.repeat(x_ranges, hh)\n",
    "    y_ranges = tf.repeat(y_ranges, hh)\n",
    "\n",
    "    y_hh_indices = tf.transpose(tf.stack([y_ranges, hh_ranges]))\n",
    "    x_hh_indices = tf.transpose(tf.stack([hh_ranges, x_ranges]))\n",
    "\n",
    "    y_mask_sparse = tf.SparseTensor(tf.cast(y_hh_indices, tf.int64),  tf.zeros_like(y_ranges), [hh, hh])\n",
    "    y_mask = tf.sparse.to_dense(y_mask_sparse, 1, False)\n",
    "\n",
    "    x_mask_sparse = tf.SparseTensor(tf.cast(x_hh_indices, tf.int64), tf.zeros_like(x_ranges), [hh, hh])\n",
    "    x_mask = tf.sparse.to_dense(x_mask_sparse, 1, False)\n",
    "\n",
    "    mask = tf.expand_dims( tf.clip_by_value(x_mask + y_mask, 0, 1), axis=-1)\n",
    "\n",
    "    mask = random_rotate(mask, rotate_angle, [hh, hh, 1])\n",
    "    mask = tf.image.crop_to_bounding_box(mask, (hh-tf.cast(h, tf.int32))//2, (hh-tf.cast(w, tf.int32))//2, tf.cast(image_height, tf.int32), tf.cast(image_width, tf.int32))\n",
    "\n",
    "    return mask\n",
    "\n",
    "\n",
    "@tf.function\n",
    "def apply_grid_mask(image):\n",
    "    mask = grid_mask()\n",
    "    mask = tf.concat([mask, mask, mask], axis=-1)\n",
    "\n",
    "    return image * tf.cast(mask, 'float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## Augmentation function & tf.image functions (flip, brightness, contrast, saturation, hue)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for train\n",
    "\n",
    "@tf.function\n",
    "def image_augmentation(iw, label):\n",
    "    global current_epoch\n",
    "    global chance\n",
    "    \n",
    "    image, words = iw\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < chance:\n",
    "        image = tf.image.flip_left_right(image)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < chance:\n",
    "        image = tf.image.random_brightness(image, 0.1)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < chance:\n",
    "        image = tf.image.random_contrast(image, 0.9, 1.1)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < chance:\n",
    "        image = tf.image.random_saturation(image, 0.95, 1.05)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < chance:\n",
    "        image = tf.image.random_hue(image, 0.05)\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < chance:\n",
    "        image = transform(image)\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=10, dtype=tf.int32, seed=SEED) < chance:\n",
    "        image = apply_grid_mask(image)\n",
    "\n",
    "    return ((image, words), label)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for test / tta\n",
    "\n",
    "# only flip left to right, brightness & contrast\n",
    "@tf.function\n",
    "def image_augmentation_tta_0(iw, filename):\n",
    "    max_chance = 8\n",
    "    \n",
    "    image, words = iw\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.flip_left_right(image)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_brightness(image, 0.1)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_contrast(image, 0.9, 1.1)\n",
    "\n",
    "    return ((image, words), filename)\n",
    "\n",
    "# all augmentation, except gridmask\n",
    "@tf.function\n",
    "def image_augmentation_tta_1(iw, filename):\n",
    "    max_chance = 8\n",
    "    \n",
    "    image, words = iw\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.flip_left_right(image)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_brightness(image, 0.1)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_contrast(image, 0.9, 1.1)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_saturation(image, 0.95, 1.05)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_hue(image, 0.05)\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = transform(image)\n",
    "\n",
    "    return ((image, words), filename)\n",
    "\n",
    "# all augmentation\n",
    "@tf.function\n",
    "def image_augmentation_tta_2(iw, filename):\n",
    "    max_chance = 8\n",
    "    \n",
    "    image, words = iw\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.flip_left_right(image)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_brightness(image, 0.1)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_contrast(image, 0.9, 1.1)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_saturation(image, 0.95, 1.05)\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = tf.image.random_hue(image, 0.05)\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=11, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = transform(image)\n",
    "\n",
    "    if tf.random.uniform(shape=[], minval=0, maxval=10, dtype=tf.int32, seed=SEED) < max_chance:\n",
    "        image = apply_grid_mask(image)\n",
    "\n",
    "    return ((image, words), filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## Show augmentated image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_augmented_image(same_image=True):\n",
    "    row, col = 3, 5\n",
    "    if same_image:\n",
    "        all_elements = get_training_dataset(do_aug=False).unbatch()\n",
    "        one_element = tf.data.Dataset.from_tensors( next(iter(all_elements)) )\n",
    "        augmented_element = one_element.repeat().map(image_augmentation).batch(row*col)\n",
    "        for iw, label in augmented_element:\n",
    "            image, words = iw\n",
    "            plt.figure(figsize=(15,int(15*row/col)))\n",
    "            for j in range(row*col):\n",
    "                plt.subplot(row,col,j+1)\n",
    "                plt.axis('off')\n",
    "                plt.imshow(image[j,])\n",
    "            plt.suptitle(CLASSES[label[0]])\n",
    "            plt.show()\n",
    "            break\n",
    "        \n",
    "        del all_elements\n",
    "        del one_element\n",
    "        del augmented_element\n",
    "    else:\n",
    "        all_elements = get_training_dataset(do_aug=True).unbatch()\n",
    "        augmented_element = all_elements.batch(row*col)\n",
    "\n",
    "        for iw, label in augmented_element:\n",
    "            image, words = iw\n",
    "            plt.figure(figsize=(15,int(15*row/col)))\n",
    "            for j in range(row*col):\n",
    "                plt.subplot(row,col,j+1)\n",
    "                plt.title(CLASSES[label[j]])\n",
    "                plt.axis('off')\n",
    "                plt.imshow(image[j,])\n",
    "            plt.show()\n",
    "            break\n",
    "        \n",
    "        del all_elements\n",
    "        del augmented_element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell again to see different image\n",
    "# chance = 8\n",
    "# show_augmented_image()\n",
    "# chance = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell again to see different batch of image\n",
    "# chance = 8\n",
    "# show_augmented_image(same_image=False)\n",
    "# chance = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# Model function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Model, Sequential\n",
    "from tensorflow.keras.layers import Input, Flatten, Dense, Dropout, AveragePooling2D, GlobalAveragePooling2D, SpatialDropout2D, BatchNormalization, Activation, Concatenate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plt_lr(epoch_count):\n",
    "    if epoch_count > 50:\n",
    "        epoch_count = 50\n",
    "    \n",
    "    rng = [i for i in range(epoch_count)]\n",
    "\n",
    "    plt.figure()\n",
    "    y = [lrfn(x) for x in rng]\n",
    "    plt.title(f'Learning rate schedule: {y[0]} to {y[epoch_count-1]}')\n",
    "    plt.plot(rng, y)\n",
    "\n",
    "def plt_acc(h):\n",
    "    plt.figure()\n",
    "    plt.plot(h.history[\"sparse_categorical_accuracy\"])\n",
    "    if 'val_sparse_categorical_accuracy' in h.history:\n",
    "        plt.plot(h.history[\"val_sparse_categorical_accuracy\"]) \n",
    "        plt.legend([\"training\",\"validation\"])       \n",
    "    else:\n",
    "        plt.legend([\"training\"])\n",
    "    plt.xlabel(\"epoch\")\n",
    "    plt.title(\"Sparse Categorical Accuracy\")\n",
    "    plt.show()\n",
    "\n",
    "def plt_loss(h):\n",
    "    plt.figure()\n",
    "    plt.plot(h.history[\"loss\"])\n",
    "    if 'val_loss' in h.history:\n",
    "        plt.plot(h.history[\"val_loss\"]) \n",
    "        plt.legend([\"training\",\"validation\"])       \n",
    "    else:\n",
    "        plt.legend([\"training\"])\n",
    "    plt.legend([\"training\",\"validation\"])\n",
    "    plt.xlabel(\"epoch\")\n",
    "    plt.title(\"Loss\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EpochCallback(tf.keras.callbacks.Callback):  \n",
    "    def on_epoch_begin(self, epoch, logs=None):\n",
    "        global current_epoch\n",
    "        global chance\n",
    "\n",
    "        current_epoch = epoch       \n",
    "        if current_epoch < 2:\n",
    "            chance = 0\n",
    "        elif current_epoch < 9:\n",
    "            chance = current_epoch - 1 # possible chance between 1 to 7\n",
    "        else:\n",
    "            chance = 8\n",
    "        print(f'Epoch #{current_epoch}')\n",
    "        print(datetime.now())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "es_val_acc = tf.keras.callbacks.EarlyStopping(\n",
    "    monitor='val_sparse_categorical_accuracy', min_delta=0.001, patience=5, verbose=1, mode='auto',\n",
    "    baseline=None, restore_best_weights=True\n",
    ")\n",
    "\n",
    "es_val_loss = tf.keras.callbacks.EarlyStopping(\n",
    "    monitor='val_loss', min_delta=0.001, patience=5, verbose=1, mode='auto',\n",
    "    baseline=None, restore_best_weights=True\n",
    ")\n",
    "\n",
    "es_acc = tf.keras.callbacks.EarlyStopping(\n",
    "    monitor='sparse_categorical_accuracy', min_delta=0.001, patience=5, verbose=1, mode='auto',\n",
    "    baseline=None, restore_best_weights=False\n",
    ")\n",
    "\n",
    "es_loss = tf.keras.callbacks.EarlyStopping(\n",
    "    monitor='loss', min_delta=0.001, patience=5, verbose=1, mode='auto',\n",
    "    baseline=None, restore_best_weights=False\n",
    ")\n",
    "\n",
    "epoch_cb = EpochCallback()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# Create model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://github.com/qubvel/efficientnet/releases/download/v0.0.1/efficientnet-b7_noisy-student_notop.h5\n",
      "258072576/258068648 [==============================] - 3s 0us/step\n",
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "mlp-words_input (InputLayer)    [(None, 6633)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "mlp-words_dense_1 (Dense)       (None, 331)          2195854     mlp-words_input[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "mlp-words_bn_1 (BatchNormalizat (None, 331)          1324        mlp-words_dense_1[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "mlp-words_act_1 (Activation)    (None, 331)          0           mlp-words_bn_1[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "efficientnet-b7_input (InputLay [(None, 512, 512, 3) 0                                            \n",
      "__________________________________________________________________________________________________\n",
      "mlp-words_dense_2 (Dense)       (None, 110)          36520       mlp-words_act_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "efficientnet-b7 (Model)         (None, 16, 16, 2560) 64097680    efficientnet-b7_input[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "mlp-words_bn_2 (BatchNormalizat (None, 110)          440         mlp-words_dense_2[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "efficientnet-b7_gap (GlobalAver (None, 2560)         0           efficientnet-b7[1][0]            \n",
      "__________________________________________________________________________________________________\n",
      "mlp-words_act_2 (Activation)    (None, 110)          0           mlp-words_bn_2[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 2670)         0           efficientnet-b7_gap[0][0]        \n",
      "                                                                 mlp-words_act_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "output (Dense)                  (None, 42)           112182      concatenate[0][0]                \n",
      "==================================================================================================\n",
      "Total params: 66,444,000\n",
      "Trainable params: 66,132,398\n",
      "Non-trainable params: 311,602\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "with strategy.scope():\n",
    "    # phase 1\n",
    "    efn7 = efn.EfficientNetB7(weights='noisy-student', include_top=False, input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3))\n",
    "    for layer in efn7.layers:\n",
    "        layer.trainable = True\n",
    "\n",
    "    model_image = Sequential([\n",
    "        efn7,\n",
    "        GlobalAveragePooling2D(name='efficientnet-b7_gap'),\n",
    "    ], name='b7-image')\n",
    "\n",
    "    model_words = Sequential([\n",
    "        Input((6633, ), name='mlp-words_input'),\n",
    "\n",
    "        Dense(331, name='mlp-words_dense_1'),\n",
    "        BatchNormalization(name='mlp-words_bn_1'),\n",
    "        Activation('relu', name='mlp-words_act_1'),\n",
    "\n",
    "        Dense(110, name='mlp-words_dense_2'),\n",
    "        BatchNormalization(name='mlp-words_bn_2'),\n",
    "        Activation('relu', name='mlp-words_act_2'),\n",
    "    ], name='mlp-words')\n",
    "    concatenate = Concatenate(name='concatenate')([model_image.output, model_words.output])\n",
    "    output = Dense(len(CLASSES), activation='softmax', name='output')(concatenate)\n",
    "\n",
    "    model = Model(inputs=[model_image.input, model_words.input], outputs=output)\n",
    "\n",
    "    # phase 2\n",
    "#     model = tf.keras.models.load_model('/kaggle/input/train-phase-1-085009/model.h5')\n",
    "#     model.load_weights('/kaggle/input/train-phase-1-085009/model_weights.h5')\n",
    "\n",
    "    model.compile(optimizer=tfa.optimizers.LAMB(0.01), loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\n",
    "\n",
    "model.summary()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pre training time : 103.180323 seconds\n"
     ]
    }
   ],
   "source": [
    "print(f'Pre training time : {(datetime.now() - PRE_TRAINING_TIME_START).total_seconds()} seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# Train model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEICAYAAAB8lNKlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXwU9fnA8c+zubjCmQDhvsJ9CERAEMWbw4pUQfFCq0WKWu2N9tfbtrZaa6kHaquiooAnKHgVRUUFE65wSwQlgRDCfZPr+f0xk3ZZks0SNpns5nm/Xvva3Znvd+aZ2d15Zr4zO19RVYwxxphw8XkdgDHGmOhiicUYY0xYWWIxxhgTVpZYjDHGhJUlFmOMMWFlicUYY0xYWWIpg4gMF5FNXsfhFRG5WUSWhGlaHURERSQ2nGWNMTVXjUssIvKNiFzsZQyq+qmqdvMyhlIiMkJEcryOIxKJyI9EZKeIHBCRZ0QkIUjZs0RkuYgcdZ/PCnVaItJURN4QkSMi8q2IXOc3rjRZHvZ7/CpIHItF5LYzWOZyYymnfKWWyx1/kYhsdNfZRyLS3m/cPSKyRUQOisgOEfm7/w6Du14+cutuDPzNi8hdIrLVrZ8hIucGxDVHRHa7j1ki0tBv/HdEZK27rj8XkZ5+4xLcWHaIyD4ReVxE4vzG/dtd1kMislJERvnVjReRV91tlIrIiDLW5wAR+cSdd56I3O03bqiIfOlOOzNgme4L+I4cE5ESEUlyx68LGF8kIm+VMf9Jbmy3BQwP9jn3EJEP3XFZIjLOb1xPd/3vcx//8V+f5VLVGvUAvgEuruJ5xHi9nG4cAvgqKDMCyKnmuG4GloRpWh0ABWLDWTaEaV0G5AG9gCbAYuCBcsrGA98CPwISgB+67+NDmRbwMjAHaACcCxwAelVmmdxp33YGy11uLKe7jipYriT3/XigDvAgsNSvbmegsfu6KfAh8GO/8V8ADwN1gauA/UCyO24wcAQY6P5GfgDkl/5ugceB94GGQCPgP8DD7rhU4KAbbyxwL5BVuv6B3wCfujElA0uB37nj6gO/dT8zH3A5cAjo4Pc9uceddi4wImB9JgG7gOvd71Ei0MNvHex211cMcAOwD2hSzmfzW+DDcsYJsAW4KWB4E2AjsNb/OxTsc3bX0VfAj924LnTXfVd3fGN3fYg7/odAZoXfw3BsPML5oJzE4n7Q04CvgT3AXKCp3/hXgJ04X/ZP8PsxAc8BTwAL3ZV2sTufnwKZbp05QB23/Aj8NubByrrjf+5+0XYAt+FsSLoE2XD8EfgMOAZ0AW4BNrhf4i3A7X5f9GNACXDYfbSqaF2U8WV/G+eHuxfnR+Vzx7UFXsf50e4BHnWH3wwsAR7C+fJvBUb5TbMR8G93mbcD9/O/H32MW2+3uyx34LdhDfx8cX5AL7qvOwSULXc+IXyPXgL+5Pf+ImBnOWUvdacvfsO2ASMrmpb7GRXg/hDdYS/wvx/uSctUQcx/BIqB4+5nXfp5DAXScb576cDQcuoHjeV01lEIyzUZ+Dxg3seA7mXMpxnOxv9x931X4ASQ6FfmU2CK+/oa4MuAaSuQ4r5/B5jqN/4O4D339Z3AgoDtxjHgIvd9BjDeb/x1QHaQzyQTuKqM4Tmcmlj+BLxQznQuB9YFDPsKuLWMsoLz255UzrTOd78f9QOGzwCmErBzUsHn3Nudlv93/33gD2XMN9Zd10cr+i7XuKawIH4IXImzUlvhbPAe8xv/Ds7eSnNgBTAroP51OD/cRJyNJsAEYCTQEeiLs0EtT5llRWQkTra/GCdJnB/CstyI88NMxNkz3oXzxWuIk2T+LiIDVPUIMArYoaoN3MeOENaFv5/g/AiSgRbAfYCKSAxOwvkWZ+PXGpjtV28wsAknMf0V+LeIiDtuJlDkLm9/nA1z6aH3991l6Q+kAVeHsD7KU+58RKSdiOwXkXbl1O0FrPZ7vxpoISLNyimbqe6vx5XpDq9oWl2BYlX9KmB8L072rYjkiMizpc0bgVT1lzgb2Dvdz/pOEWkKLACm42ygHwYWlLMcocbiv9yVXa6T6rrf1a/95yUi14nIQZydjH7Ak351t6jqoXKm/Q4QIyKD3e/p94BVODuO4HzXLxeRJiLSBOeI553S2boPAt73DjK+jYg0Clw5ItLCXQ/rAseVYwiw121+2yUib/l9PwPnWzqsN6cajvNbfa2c+UwCXnXXeWmsg3B+bzPKKB/scw6Mqcy4RGQ/zg7PP3ESaFCRlFhuB36pqjmqegJnT/fq0nZbVX1GVQ/5jesX8GWZp6qfqWqJqh53h01X1R2quhd4CzipXT1AeWUnAM+q6jpVPQr8LoRlec4tX6Sqhaq6QFW/VsfHOHsMwyu7LgIUAilAe3den7ob0EE4SelnqnpEVY+rqv8J+29V9WlVLcbZwKfgfBlb4CS7e9x6u4C/A9f6rY9HVDXbXVd/DmF9nKKi+ajqNlVtrKrbyplEA5w9/FKlrxNDKFtaPrGc8f7TqqjubuBsoD1O004ip+70BDMG2KyqL7jfl5dxmju+U4nlqKj86SxXhfNS1ZdUtSHOxnkGTnNMKHUP4WxUl+Ac2fwGmOyX+FfgNEvtcR/FOM1jAB8A54tzbjIeZ0cqHqjnjn8HuFtEkkWkJc5OGn7jAXDPu8wCZqrqRkLTBmejfzfQDudI/2V33OdAKxGZKCJxIjIJp7mwXhnTKU0chwNHiEg9nJ215/yGxbjLf5eqlpQxvWCf80acHdufuXFdirPDelJcqtoYpwXhTmBleSugVCQllvbAG+5e6n6cpqNinI1djIg8ICJfu3tI37h1/PcMs8uY5k6/10dxPoDylFe2VcC0y5pPoJPKiMgoEVkqInvdZRvNybEHKnddlFH2QZw25vfFOZk6zR3eFid5FJUzj/8ur5swwVnm9kAckOs3/ydxjhTh1PXxbZDlCKai+VTkMM4RYKnS14dCKFta/lA54/2nFbSuqh5W1Qw3KeTh/DAvFb+TzRVoxanr8FucI8zTXY6Kyoe8XKczL1XdjLPXX7rxr6jubThHKb1wksINwNsi0sod/wpOM1KiW+9r4EV3XhtxNsyP4jShJgHrcY7awWm1WIlzBPQ58CbOzteu0kBExIfT7FeA83mF6hjwhqqmuzuvvwOGikgjVd0DjMVp3cjDaf34j19cpfOui3MeZmY58/guTpP2x37DpuIccX9RTp1yP2dVLcRp/RiD85v/CU7T+ikXDLlHSDOA50Uk6O8wkhJLNk47f2O/Rx1V3Y7TzDUWpzmqEU7TDpx8mFdVt3HOxdlTKdU2hDr/jcW9OuM1nPMSLdw9g4X8L/ay4g62Lk6ekXMU9xNV7YSzl/tjEbnInUa7co5ygsnG2YtM8pt3Q1UtbcbI5eR1ENhUdYST94ZaVnI+FVmH0/xSqh+Q5/7Ayyrb16+pD5zmznV+48ub1ldArIikBowvr/mk9PMsqwnCf3ypHThJ1l87nHNCgU43ljNZrpPqikh9nD3w8uYV644vrdtJRPyPpPyn3Q94S1W/clsY3sX5Xg31G/+keyR7GGdjN7p0Qqr6qqr2VtVmOEc77XHOTaGqx1T1TlVt7f4m9gDL3SNz3O/Av3F20q5yN7yhyuTkz++kz1pVP1bVs1W1KU5zeDfgy4BplCaOxeXMYxLwfECz7UXAOPeqr5046+lvIvKoOz7ob0FVM1X1fFVtpqqXAZ3KiKuUD+f3W9aOzf9oBSdhqvuBc7QxCudKk9JHLM4VO4txmnTAOWcw1n09FWcPpCHOib7H8TuBjnPYeH8Z8ynvJPIITj15X17ZUThf+h7uCn+eik/e+59YS8Q52jgf5ws4CueI6H53fHecPaFGfnXKXRdlzO9ynHMUgrPBz3WXLwanrfUhd53VAYa5dW4m4KqwgPU5D/iHu759OBuM891xP8DZQ2yDcwXKIk4+IT8L52RiHE6b8G7KP3lf7nxC+B6NxNkD6+nG8SEVXxV2N87VPHdy8lVhQaeFc27qZXc9DuPkq6cG42xAfDjnSOYAHwWJezYnn2hthnPhxXU4v4Nr3PdJQeqXGcvprqMKlivZfX+V+935CydfFXYb0Nx93RNn4/aw3/ilON+9OsA4Tr4qbBJOYuuE8729BOc30d0d/xFOW39d9/E48JnftAfifL+T3fX9kt+41jhHgYJzTiQbuNRv/Aw3tgblrLMEN+YcnHN+dXBPfONcUbUPp5k8Dqfp9lO/uv3d4Q2BR/xj9ivzPvD7cubdBuecY+eA4Y1xdtBKH5/jHBk1CvFz7usuRz2ci5S2AgnuuEvcuGPcuKfj7OzUKSvG/04zlB9pdT5wNuIa8Lgf54f5Y5wTyodwDn//5NZpgLMROoSzQbiJakos7vt73Q9uB86GVYG25SzfYgIuJ8W50iIP58f1As4P+n6/8c/g7Fnt539XhZW5LsqY34/c+I/g/Bh+5TeuHU5TwB6cDfx0d/jNBE8sjXCussvB2bisBK51x8Xi/KD2uF/QwKvCOgHLcA7PS09KB7sqrLz5tHOn0S7Id6m02eEg8Czuj8Ud9w5wX8CPfjlOEl8B9D+NaTV11+MRnKvJrvMbN9FdD0dwkvrzQMsgMZ+Ds1Hd5/d5nOvGdsB9PjdI/WCxnLLOKrtc7viLcdroj+F8rzv4jXvWne4RnO/fg5x8JWUHt84xnO+x/+9LgN+78zyE09R7o9/4jjjnOffg7N2/C6T6jV/i1tuL03xa32/ceW48R935Xu83rj3O96/0qrzSh3+Zbzh1++S/3D/AOZrc58bY1m/cy+5nWHplafOA9dka92KVcj7be/FLVEG+A4s5dRsT7HN+0I33MM7voovfuPHuZ3wY5+rRhUDfimIozbQmTESkB8515Ala/vkLY4yJWpF0jqXGEpFx4vwrtwlOk8BbllSMMbWVJZbwuB3nMPFrnPMlP/A2HGOM8Y41hRljjAkrO2IxxhgTVlFxe/KkpCTt0KGD12EYY0xEWb58+W5VTQ73dKMisXTo0IGMjAyvwzDGmIgiIpW9M0ZQ1hRmjDEmrCyxGGOMCStLLMYYY8LKEosxxpiwssRijDEmrEJKLCIyUkQ2iUiWX38e/uNFRKa74zNFZEBFdUVkvIisE5ESEUkLmN69bvlNInLZmSygMcaY6lVhYnF7J3sM53buPYGJItIzoNgonG6BU3G63H0ihLprcfoe+CRgfj1xegnshXO758fd6RhjjIkAoRyxDAKyVHWLqhbg3NJ9bECZsbidz6jqUqCxiKQEq6uqG1R1UxnzGwvMVtUTqroVp/fDQZVaulruo427WLcjsAdYY4ypWqEkltac3NVsDqf2HlZemVDqVmZ+iMhkEckQkYz8/PwKJln7HDhWyJQXl/O959I5cOx0OsEzxpgzE0piKasL1cA7V5ZXJpS6lZkfqvqUqqapalpyctjvSBDx5q/ewYmiEnYdOsEf3l7vdTjGmFoklMSSw8l9mLfB6SkxlDKh1K3M/EwF5qZn0yOlIXde0IVXl+fw4cY8r0MyxtQSoSSWdCBVRDqKSDzOifX5AWXmAze5V4cNAQ6oam6IdQPNB64VkQQR6YhzQcCXp7FMtd76HQdZs/0A16S14a4LU+neMpFpr63hwFFrEjPGVL0KE4vbE+KdwHs4fU/PVdV1IjJFRKa4xRYCW3BOtD8NTA1WF/7b62IOTh/fC0TkPbfOOmAusB6nL+s7VLU4TMtbK8zNyCY+1seV/VsTH+vjofH92HukgN+9tc7r0IwxtUBUdPSVlpamdndjx/HCYgb/aRHndU3mnxP7/3f4wx98xfRFm/nXTWlc3LOFhxEaY2oKEVmuqmkVlzw99s/7KPPeup0cOFbINWltTxp+5wVd6JHSkHvfWMP+owUeRWeMqQ0ssUSZuRnZtGlSl6Gdm5003GkS68u+IwX8dr41iRljqo4lliiSvfcon2XtYfzAtvh8p1613atVI+68sAtvrtrBe+t2ehChMaY2sMQSRV7JyEYErk5rU26ZOy7oQs+UhvzyjbXsO2JNYsaY8LPEEiWKS5RXlucwPDWZ1o3rllsuLsbH3yb048CxAn5jTWLGmCpgiSVKfLo5n9wDx085aV+WHikNuevCVOav3sG7a3OrITpjTG1iiSVKzM3Ipkm9OC7u2Tyk8j8Y0ZnerRvyf2+uZa81iRljwsgSSxTYc/gEH6zPY1z/NiTEhtbDQFyM88fJA8cK+fW8tVUcoTGmNrHEEgXeWLmdwmLlmrMrbgbz171lQ+65uCtvZ+aycI01iRljwsMSS4RTVeZmZNOvbWO6tUw87fq3n9eJvm0a8as317Ln8IkqiNAYU9tYYolwq7L381XeYa49zaOVUrFuk9ih40X8ep5dJWaMOXOWWCLc3Ixs6sbFcHnflEpPo2uLRO65JJUFa3J5O9N6KDDGnBlLLBHsaEERb63OZUzfFBLrxJ3RtCYP70S/to351ZtryT9kTWLGmMqzxBLBFmTmcvhE0WmftC9LbIyPh67uy5ETxfzqzbVEw12vjTHesMQSweakZ9MpuT5p7ZuEZXqpLRL58aVdeXfdTt7KtKvEjDGVY4klQmXtOkzGt/uYkNYWkVNvOFlZ3x/eibPaNubX89ay69DxsE3XGFN7WGKJUK9kZBPjE747oHVYpxvjEx4a34+jBcX83xvWJGaMOX2WWCJQYXEJr63I4cLuzWmeWCfs0+/SvAE/vbQr76/PY/5qu0rMGHN6LLFEoA837mL34YKQbjhZWbee24kB7Rrz63nr2HXQmsSMMaGzxBKB5qZn0zwxgRHdkqtsHjE+4cHx/TheWMx91iRmjDkNISUWERkpIptEJEtEppUxXkRkujs+U0QGVFRXRJqKyAcistl9buIOjxeRZ0VkjYisFpERYVjOqJF38DgfbdrFVQPbEBtTtfsFnZMb8LPLuvGfDXm8uWp7lc7LGBM9KtwyiUgM8BgwCugJTBSRngHFRgGp7mMy8EQIdacBi1Q1FVjkvgf4PoCq9gEuAf4mInZk5Xp1eQ4lChOqsBnM3y3DOpLWvgm/mbeOPGsSM8aEIJQN9iAgS1W3qGoBMBsYG1BmLPC8OpYCjUUkpYK6Y4GZ7uuZwJXu6544iQZV3QXsB9IqtXRRRlV5JSObQR2b0jGpfrXMs7RJrKC4hPteX2NNYsaYCoWSWFoD2X7vc9xhoZQJVreFquYCuM+lPVStBsaKSKyIdAQGAqfsnovIZBHJEJGM/Pz8EBYj8i3bupdv9hyt0pP2ZemYVJ+fXdadRRt38doKaxIzxgQXSmIp6993gbut5ZUJpW6gZ3ASUAbwCPA5UHTKRFSfUtU0VU1LTq66k9g1ydz0bBITYhndp/I3nKysW4Z2YFCHpvzurXXsPGBNYsaY8oWSWHI4+YihDRD454byygSrm+c2l+E+7wJQ1SJV/ZGqnqWqY4HGwObQFid6HTxeyMK1uVxxVivqxofWS2Q4+XzCX6/uS2FxCfe+nmlNYsaYcoWSWNKBVBHpKCLxwLXA/IAy84Gb3KvDhgAH3OatYHXnA5Pc15OAeQAiUk9E6ruvLwGKVHV95RcxOsxftYPjhSVhueFkZXVIqs8vRnbno035vLI8x7M4jDE1W2xFBVS1SETuBN4DYoBnVHWdiExxx88AFgKjgSzgKHBLsLrupB8A5orIrcA2YLw7vDnwnoiUANuBG8OypBFubkY23Vsm0qd1I0/jmHROB95Zu5M/vLWe4alJpDSq62k8xpiaR6KhSSMtLU0zMjK8DqPKrN9xkNHTP+U33+nJLcM6eh0O2/Yc5bJHPuHsjk2ZecvZYb0JpjGm+ojIclUN+1W39v+QCDA3I5v4GB9XnhXeG05WVrtm9bh3dHc++SqfuRnZFVcwxtQqllhquOOFxbyxcjuX9mpBk/rxXofzXzcMbs+QTk25/+0NbN9/zOtwjDE1iCWWGu799XkcOFbo6Un7svh8woNX96NYlWmv2VVixpj/scRSw81Nz6Z147oM65zkdSinaNu0HveO7sGnm3czO92axIwxDkssNVj23qMsydrN+LQ2+Hw18wT59YPaMbRzM+5/ez05+456HY4xpgawxFKDvbI8BxEYX823cDkdPp/wl6v6AjDtNbuXmDHGEkuNVVyivJqRzbldkmjduGb/V6Rt03rcN6YHS7J289QnW7wOxxjjMUssNdSSrN3sOHC8xp20L891g9oxpk8Kf3l3Ix9/VTtuCmqMKZsllhpqbno2TerFcUnPFl6HEhIR515iXVskctdLK/hm9xGvQzLGeMQSSw2090gB76/fybj+bUiIrf4bTlZW/YRYnroxDZ9PmPxCBodPnHJTamNMLWCJpQZ6Y+V2Cos1YprB/LVrVo9HJw4ga9dhfjJ3FSUldjLfmNrGEksNo6rMTc+mX9vGdGuZ6HU4lXJuahL3je7Be+vyePSjLK/DMcZUM0ssNczqnANsyjtU7b1Ehtut53ZkXP/WPPzBV3ywPs/rcIwx1cgSSw0zJz2bunExfKdf9fcSGU4iwp+/24c+rRvxozmryNp1yOuQjDHVxBJLDXK0oIi3Vu9gdJ8UEuvEeR3OGasTF8OTNw6kTpyP7z+/nAPHCr0OyRhTDSyx1CALMnM5fKIoIk/al6dV47o8fv1Asvce5Z7ZKym2k/nGRD1LLDXI3IxsOiXV5+wOTbwOJawGdWzKb67oxUeb8nn4g01eh2OMqWKWWGqIr/MPk/7NPsantY3KHhlvGNyOiYPa8thHX7MgM9frcIwxVcgSSw0xNyObGJ9w1cCa0UtkuIkIv72iFwPaNeanr6xmQ+5Br0MyxlSRkBKLiIwUkU0ikiUi08oYLyIy3R2fKSIDKqorIk1F5AMR2ew+N3GHx4nITBFZIyIbROTecCxoTVZYXMJry7dzQbfmNE+s43U4VSYhNoYZNwykYd1YJr+Qwb4jBV6HZIypAhUmFhGJAR4DRgE9gYki0jOg2Cgg1X1MBp4Ioe40YJGqpgKL3PcA44EEVe0DDARuF5EOlVy+iPDRxl3sPnwiqk7al6d5wzrMuGEgeQdOcOfLKygqLvE6JGNMmIVyxDIIyFLVLapaAMwGxgaUGQs8r46lQGMRSamg7lhgpvt6JnCl+1qB+iISC9QFCoCobjeZm5FNcmICF3RL9jqUatG/XRPuH9ebz7L28MA7G70OxxgTZqEkltaAf7+zOe6wUMoEq9tCVXMB3Ofm7vBXgSNALrANeEhV9wYGJSKTRSRDRDLy8yP3Nu27Dh7no035XD2wDbExteeU14S0ttw8tAP/WrKV11fkeB2OMSaMQtmSlXWJUuCfEcorE0rdQIOAYqAV0BH4iYh0OmUiqk+papqqpiUnR+6e/qsrciguUSZE+C1cKuOXY3owpFNTpr2+hsyc/V6HY4wJk1ASSw7gv9VrA+wIsUywunlucxnu8y53+HXAu6paqKq7gM+AtBDijDiqyisZOQzq2JSOSfW9DqfaxcX4eOy6ASQ3SOD2F5aTf+iE1yEZY8IglMSSDqSKSEcRiQeuBeYHlJkP3OReHTYEOOA2bwWrOx+Y5L6eBMxzX28DLnSnVR8YAkRlQ/yXW/eydfeRiL/h5Jlo1iCBJ28cyL6jBUydtZyCIjuZb0ykqzCxqGoRcCfwHrABmKuq60RkiohMcYstBLYAWcDTwNRgdd06DwCXiMhm4BL3PThXkTUA1uIkpmdVNfNMF7QmmpORTWJCLKP7RPYNJ89U79aN+MtVfUn/Zh+/f3tdxRWMMTVabCiFVHUhTvLwHzbD77UCd4Ra1x2+B7iojOGHcS45jmoHjxeycE0u3x3QhrrxkdNLZFUZe1Zr1u84yJOfbKFXq0ZMHNTO65CMMZVUey5DqmHeWr2D44UltboZLNDPR3ZneGoSv563luXfnnIhoDEmQlhi8cjc9Gy6t0ykb5tGXodSY8T4hEcnDqBV47pMeXEFOw8c9zokY0wlWGLxwIbcg6zOOcCEKL3h5JloVC+Op29K48iJIm5/cTnHC4u9DskYc5ossXhgTno28TE+xvWPzhtOnqmuLRJ5eMJZrM7ez6/eXItzCs8YEykssVSzE0XFvLlqO5f0akGT+vFeh1Njjezdkh9elMory3N4/otvvQ7HGHMaLLFUs/fX5bH/aKGdtA/BPRelcnGPFvz+7fV88fUer8MxxoTIEks1m5uRTevGdTm3S5LXodR4Pp/w92v60aFZPe54aQU5+456HZIxJgSWWKpRzr6jLMnazdUD2+Dz2Un7UCTWcU7mFxaXcPsLyzlWYCfzjanpLLFUo1cynLv4jk9r43EkkaVTcgOmX9uf9bkH+cVrmXYy35gazhJLNSkuUV5dnsO5XZJo06Se1+FEnAu6N+enl3Zj/uodPPXJFq/DMcYEYYmlmnyWtZvt+4/Vil4iq8rUEZ0Z0yeFv7y7kY+/itw+eIyJdpZYqsmcjGya1Ivjkp4tvA4lYokID47vS9cWidz10gq+2X3E65CMMWWwxFIN9h0p4IN1eVzZvzUJsXbDyTNRLz6Wp29Kw+cTbns+g71HCrwOyRgTwBJLNXhj5XYKikusGSxM2jatxxPXDyR771Fu/PcyDhwt9DokY4wfSyxVTFWZm5FNvzaN6N6yodfhRI1zOjfjyRsHsjnvMDc9+yWHjltyMaamsMRSxTJzDrBx5yEm2NFK2I3o1pzHrx/Auu0HuOXZdI6cKPI6JGMMlliq3JyMbOrE+fhOv1ZehxKVLu7Zgn9O7M/K7P3cOjPd/kBpTA1giaUKHSso5q1VOxjdJ4WGdeK8DidqjeqTwsMT+rFs614mv5Bht9o3xmOWWKrQwjW5HDpRZDecrAZjz2rNX6/qy6ebdzN11goKikq8DsmYWssSSxWak55Nx6T6DOrY1OtQaoXxaW3507g+fLhxF3e+tILCYksuxnghpMQiIiNFZJOIZInItDLGi4hMd8dnisiAiuqKSFMR+UBENrvPTdzh14vIKr9HiYicFY6FrU5b8g/z5Td7GZ/WxnqJrEbXDW7H767oxfvr87hnziqKLLkYU+0qTCwiEgM8BowCegITRaRnQLFRQKr7mAw8EULdacAiVU0FFrnvUdVZqnqWqp4F3Ah8o6qrzmgpPTA3I6KSLbkAABzMSURBVIcYn3D1ALvhZHWbNLQDvxzdgwWZufz81UyKS+ymlcZUp9gQygwCslR1C4CIzAbGAuv9yowFnlfntrNLRaSxiKQAHYLUHQuMcOvPBBYDvwiY90Tg5dNeKo8VFZfw2oocLuiWTPOGdbwOp1b6/nmdOFFUzEPvf0VcjI8/f7ePdVVgTDUJJbG0BrL93ucAg0Mo07qCui1UNRdAVXNFpHkZ874GJwGdQkQm4xwd0a5duxAWo/p8tCmf/EMnmGAn7T1154WpFBSVMP3DLOJihT+M7W3NksZUg1ASS1m/xMC2hfLKhFK37JmKDAaOqurassar6lPAUwBpaWk1qq1jTno2SQ0SuKB7WbnSVKcfXdKVE8UlPPnxFuJjYvjV5T0suRhTxUJJLDmA/653G2BHiGXig9TNE5EU92glBdgVMM1ricBmsF0Hj/PRpl3cNrwjcTF20Z3XRIRpI7tTUFTCM59tJSHOx88v62bJxZgqFMqWLx1IFZGOIhKPs8GfH1BmPnCTe3XYEOCA28wVrO58YJL7ehIwr3RiIuIDxgOzK7lcnnltxXaKS9SawWoQEeHXl/fk+sHteGLx1zzyn81eh2RMVKvwiEVVi0TkTuA9IAZ4RlXXicgUd/wMYCEwGsgCjgK3BKvrTvoBYK6I3Apsw0kkpc4DckpP+kcKVeWVjGzO7tCEzskNvA7H+BFxzrEUFpfwj0WbiY/1cccFXbwOy5ioFEpTGKq6ECd5+A+b4fdagTtCresO3wNcVE6dxcCQUGKrSdK/2ceW3UeYahusGsnnE/783b4UFisPvreJhFgftw3v5HVYxkSdkBKLCc2c9GwaJMQyuk9Lr0Mx5YjxCQ9e3ZeCohLuX7CB+FgfN53TweuwjIkqlljC5NDxQhauyeXK/q2pF2+rtSaLjfHxyLVnUVBcwq/nrSMuxsfEQTXrknVjIpldthQmb63O5VhhsfUSGSHiYnw8el1/LuiWzH1vrOHV5Tleh2RM1LDEEiZzMrLp1iKRfm0aeR2KCVFCbAxP3DCQYZ2T+Pmrq5m/OvAqemNMZVhiCYNNOw+xOns/E85ua/+PiDB14mJ4+qY00jo05UdzVvHOmlyvQzIm4lliCYM56dnExQjj+rf2OhRTCXXjY3jm5rPp16YRd728kv+sz/M6JGMimiWWM3SiqJg3VuZwac+WNK0f73U4ppIaJMTy3PcG0atVQ6bOWsHHX+V7HZIxEcsSyxn6z/pd7DtayAQ7aR/xGtaJ4/nvDaZL8wZMfj6Dz7N2ex2SMRHJEssZmpORTatGdTi3S5LXoZgwaFQvjhdvG0yHZvW5dWYGX27d63VIxkQcSyxnIGffUT7dnM/VaW2Jsb4+okbT+vG8eNtgUhrX4ZZnv2TFtn1eh2RMRLHEcgZK//swfqD1EhltkhMTeOm2ISQlJjDpmS9Zk3PA65CMiRiWWCqppER5JSOHYZ2TaNu0ntfhmCrQslEdXvr+EBrVjePGZ5axboclF2NCYYmlkj77ejfb9x+zk/ZRrnXjurz8/SHUjYvh2ieX2tVixoTAEkslzUnPplHdOC7t2cLrUEwVa9u0Hq9PHUqbpvX43nPpvLj0W69DMqZGs8RSCfuOFPD+ujzG9W9NnbgYr8Mx1SClUV1emXIO53dN5v/eXMv9b6+nuKRG9YhtTI1hiaUS3ly1nYLiEuslspZpkBDLUzcO5OahHfjXkq1MeXE5RwuKvA7LmBrHEstpUlXmpGfTp3UjerZq6HU4pprFxvj47RW9+O13erJoQx4TnvyCvIPHvQ7LmBrFEstpWrP9ABt3HrKT9rXczcM68q9JaWzNP8LYRz+zK8aM8WOJ5TTNSc8mIdbHFf1aeR2K8diF3VvwypShiMD4GV/w4Ua7eaUxEGJiEZGRIrJJRLJEZFoZ40VEprvjM0VkQEV1RaSpiHwgIpvd5yZ+4/qKyBcisk5E1ohInTNd0HA4VlDM/FU7GNMnhUZ147wOx9QAPVs15M07htEpuT63zczguc+2eh2SMZ6rMLGISAzwGDAK6AlMFJGeAcVGAanuYzLwRAh1pwGLVDUVWOS+R0RigReBKaraCxgBFFZ+EcPnnbW5HDpRZM1g5iQtGtZh7u3ncFGPFvz2rfX8Zt5aiopLvA7LGM+EcsQyCMhS1S2qWgDMBsYGlBkLPK+OpUBjEUmpoO5YYKb7eiZwpfv6UiBTVVcDqOoeVS2u5PKF1Zz0bDo0q8fgjk29DsXUMPXiY5lxw0C+P7wjM7/4lu8/n8HhE3bFmKmdQkksrYFsv/c57rBQygSr20JVcwHc5+bu8K6Aish7IrJCRH4eyoJUtW92H2HZ1r2MT7NeIk3ZYnzCL8f05I/jevPJ5t1c/cTn7Nh/zOuwjKl2oSSWsraigf8MK69MKHUDxQLnAte7z+NE5KJTghKZLCIZIpKRn1/1t9mYm5GNT+Bqu+GkqcD1g9vz7M1ns33fMa587DO7gaWpdUJJLDmA/0mFNsCOEMsEq5vnNpfhPu/ym9bHqrpbVY8CC4EBBFDVp1Q1TVXTkpOTQ1iMyisqLuHV5Tlc0K05LRrWiOsITA13XtdkXv3BUOJifEx48gveW7fT65CMqTahJJZ0IFVEOopIPHAtMD+gzHzgJvfqsCHAAbd5K1jd+cAk9/UkYJ77+j2gr4jUc0/knw+sr+TyhcXHX+Wz69AJO2lvTku3lom8eccwurVMZMqLy3n6ky2o2m1gTPSrMLGoahFwJ84GfwMwV1XXicgUEZniFlsIbAGygKeBqcHqunUeAC4Rkc3AJe57VHUf8DBOUloFrFDVBWFY1kqbk55NUoMELuzevOLCxvhJTkxg9uQhjOrdkj8u3MAv31xLoV0xZqKcRMMeVFpammZkZFTJtHcdOs7QP3/Ired25N7RPapkHib6lZQoD76/iScWf83w1CQeu34ADevYf6GMt0RkuaqmhXu69s/7Cry+YjtFJcp4u+GkOQM+n/CLkd3561V9+eLrPVz9xOdk7z3qdVjGVAlLLEGoKnPTs0lr34QuzRt4HY6JAhPObsvztw5i54HjjHv8M1Zu2+d1SMaEnSWWIDK+3ceW3UfspL0Jq6Gdk3h96jDqxcdy7VNLWbgm1+uQjAkrSyxBzEnPpn58DGP6pHgdiokyXZo34I2pQ+nduhFTZ63g8cVZdsWYiRqWWMpx6HghCzJz+U6/VtRPiPU6HBOFmjVIYNZtg7miXyv++u4mfvFaJgVFdsWYiXy2xSzH25m5HCsstmYwU6XqxMXwj2vPokNSfaYv2kz23mPMuGEgjerZFWMmctkRSznmpGeT2rwB/ds29joUE+VEhB9f0pWHJ/Qj49u9jHviM77dc8TrsIypNEssZfgq7xCrsvdzzdl2w0lTfb47oA0v3jqYvUcKuPKxz/j4q6q/B54xVcESSxnmpGcTFyOM6x94E2djqtbgTs14Y+owkhokMOmZL/ndW+s4Xlgjeo0wJmSWWAIUFJXwxsrtXNyjBc0aJHgdjqmFOibV5627zmXSOe159rNvGPvoZ2zcedDrsIwJmSWWAP/ZkMfeIwV20t54qk5cDL8b25tnbz6bPUdOcMWjn/HMkq2UlNglyabms8QSYE56NimN6nBeatXeit+YUFzQvTnv3nMew7sk8fu31zPp2S/ZdfC412EZE5QlFj879h/jk835jB/YhhifnbQ3NUNSgwT+NSmNP1zZm/Rv9nLZI5/wvvXvYmowSyx+Xl2egyp2w0lT44gINw5pz9t3nUurxnWZ/MJy7n19DUcLirwOzZhTWGJxlZQoczOyGdalGW2b1vM6HGPK1KV5Im9MHcbt53didvo2Lp++hMyc/V6HZcxJLLG4vtiyh5x9x5hgRyumhouP9XHvqB7Mum0wxwqL+e7jn/PYR1kU24l9U0NYYnHNSc+mUd04LuvV0utQjAnJ0M5JvHv3eVzWqyUPvreJiU8vZfv+Y16HZYwlFoD9Rwt4d91OrjyrFXXiYrwOx5iQNaoXx6PX9eeh8f1Yt/0AIx/5hHmrtnsdlqnlLLEAb67cTkFRif13xUQkEeHqgW1YePdwujRvwN2zV/GjOas4eLzQ69BMLVXrE4uqMicjh96tG9KrVSOvwzGm0to3q88rt5/DPRenMm/VdkY98inp3+z1OixTC4WUWERkpIhsEpEsEZlWxngRkenu+EwRGVBRXRFpKiIfiMhm97mJO7yDiBwTkVXuY0Y4FrQ8a7cfZEPuQa6xk/YmCsTG+Ljn4q68MmUoPh9c8+QX/O39TRQWWz8vpvpUmFhEJAZ4DBgF9AQmikjPgGKjgFT3MRl4IoS604BFqpoKLHLfl/paVc9yH1Mqu3ChmJOxjYRYH1ecZTecNNFjYPsmLPzhcMb1b8M/P8zi6hlf8M1uuxW/qR6hHLEMArJUdYuqFgCzgbEBZcYCz6tjKdBYRFIqqDsWmOm+nglceYbLctqOFxYzb9UORvVuSaO61rGSiS6JdeL424R+PHpdf7bmH2b09E+Zm55tXSCbKhdKYmkNZPu9z3GHhVImWN0WqpoL4D439yvXUURWisjHIjK8rKBEZLKIZIhIRn5+5fqtyMw5wIlCO2lvotvlfVvx7j3n0bdNI37+WiZTZ61g35ECr8MyUSyUxFLWTbMCd3nKKxNK3UC5QDtV7Q/8GHhJRBqeMhHVp1Q1TVXTkpMrd8PIQR2bsuy+ixjSsVml6hsTKVo1rstLtw3h3lHd+c+GPEb+4xOWbN7tdVgmSoWSWHIA/136NsCOEMsEq5vnNpfhPu8CUNUTqrrHfb0c+BroGsrCVEaT+vH47IaTphbw+YTbz+/MG1OHUT8hlhv+vYw/LljPiSLrSMyEVyiJJR1IFZGOIhIPXAvMDygzH7jJvTpsCHDAbd4KVnc+MMl9PQmYByAiye5Jf0SkE84FAVsqvYTGmJP0bt2IBXcN54Yh7Xj6061c+djnfJV3yOuwTBSpMLGoahFwJ/AesAGYq6rrRGSKiJResbUQZ+OfBTwNTA1W163zAHCJiGwGLnHfA5wHZIrIauBVYIqq2sX4xoRR3fgY7r+yD/+6KY1dB49z+fQl/PXdjRw5YXdLNmdOouEKkbS0NM3IyPA6DGMiUv6hE/x54QZeX7mdlg3rcO/o7lzRrxUi1kQc7URkuaqmhXu6tf6f98bUdsmJCTx8zVm89oNzSEqM5+7Zq7jmyaWs33HQ69BMhLLEYowBYGD7psy741z+/N0+ZOUf5vJ/fsqv3lzL/qN2abI5PZZYjDH/FeMTJg5qx0c/GcFN53Rg1rJvGfHQYl5c+q3192JCZonFGHOKRvXi+O0VvVh493C6tUjk/95cy3f+ucRuamlCYonFGFOu7i0bMnvyEB69rj/7jhYwfsYX3DN7JTsPHPc6NFODWWIxxgQlIlzetxWLfnI+d17QhYVrdnLh3xbzxOKv7c+VpkyWWIwxIakXH8tPL+vGBz8+j2FdkvjLuxsZ+cinfLRxl9ehmRrGEosx5rS0b1afp29K47lbzkaAW55L59bn0u22/Oa/LLEYYyplRLfmvHvPedw3ujtLt+zh0r9/Yv/eN4AlFmPMGYiP9TH5vM589NMRXN43hccXf81Ff/uYeau2W78vtZglFmPMGWvesM6p/95/yv69X1tZYjHGhE3pv/f/NK4Pm/MO2b/3aylLLMaYsIrxCdcNbsfin17w33/vX/DQYmYts3/v1xaWWIwxVcL/3/tdWyTyyzfWcsWjS8iwf+9HPUssxpgq5f/v/b1HCrh6xhdMnbWcDbl2/iVaxXodgDEm+pX+e//C7s2Z8fEWnlmylYVrdnJJzxb88MJU+rRp5HWIJoysoy9jTLU7cLSQZz/fyjNLtnLweBEjuiVz14WpDGzfxOvQapWq6ujLEosxxjOHjhfy/Bff8u8lW9l7pIBhXZpx14WpDOnUzOvQagVLLEFYYjEmsh0tKGLW0m08+ckWdh8+waAOTbnroi6c2yXJukiuQpZYgrDEYkx0OF5YzOwvtzHj4y3sPHics9o25ocXdeGCbs0twVQBT/u8F5GRIrJJRLJEZFoZ40VEprvjM0VkQEV1RaSpiHwgIpvd5yYB02wnIodF5KdnsoDGmMhRJy6Gm4d15OOfj+CP43qTf+gE33sug+88uoR31+6kxP4HExEqTCwiEgM8BowCegITRaRnQLFRQKr7mAw8EULdacAiVU0FFrnv/f0deKcSy2SMiXAJsTFcP7g9i382gr9e3ZfDx4uY8uJyRk//lLdW77A/WtZwoRyxDAKyVHWLqhYAs4GxAWXGAs+rYynQWERSKqg7Fpjpvp4JXFk6MRG5EtgCrKvkchljokBcjI8JaW35z4/P55FrzqKoRLnr5ZVc+vePeX1FDkXFJV6HaMoQSmJpDWT7vc9xh4VSJljdFqqaC+A+NwcQkfrAL4DfBQtKRCaLSIaIZOTn54ewGMaYSBUb4+PK/q15757zeOy6AcTF+Pjx3NVc9PDHzEnfRkGRJZiaJJTEUtYZs8Dj0PLKhFI30O+Av6vq4WCFVPUpVU1T1bTk5OQKJmmMiQYxPmFM3xQW/nA4T944kMQ6sfzitTVc8NBiXlj6rXWVXEOE8s/7HKCt3/s2wI4Qy8QHqZsnIimqmus2m5X2bzoYuFpE/go0BkpE5LiqPhrKAhljop/PJ1zWqyWX9mzB4k35TP9wM796cy2PfriZ28/rzMRB7agbH+N1mLVWKEcs6UCqiHQUkXjgWmB+QJn5wE3u1WFDgANu81awuvOBSe7rScA8AFUdrqodVLUD8AjwJ0sqxpiyiAgXdG/O6z8Yyou3DqZ9s/r8/u31DP/rhzz58dfWm6VHKjxiUdUiEbkTeA+IAZ5R1XUiMsUdPwNYCIwGsoCjwC3B6rqTfgCYKyK3AtuA8WFdMmNMrSEinJuaxLmpSSzbsod/fpjFn9/ZyIyPv+bWczty/eD2NKkf73WYtYb9QdIYE5WWf7uPRz/czEeb8omP9TGmTwrXD27HwPZN7M+WLvvnfRCWWIwx5dm48yAvLdvG6yu2c/hEEd1aJHLd4HaMG9CahnXivA7PU5ZYgrDEYoypyJETRby1egezlm1jzfYD1I2L4Yp+rbh+SDv6tmnsdXiesMQShCUWY8zpyMzZz0vLtjFv1Q6OFRbTu3VDrh/cniv6taJ+Qu3ppsoSSxCWWIwxlXHweCFvrtzOrKXb2JR3iAYJsYzr35rrBrejR0pDr8OrcpZYgrDEYow5E6rKim37mLV0G2+vyaWgqIQB7Rpz/eD2jOmbQp246PxPjCWWICyxGGPCZd+RAl5bkcNLy7axZfcRGtWN46oBbbhucDu6NG/gdXhhZYklCEssxphwU1W+2LKHWcu28d7anRSVKEM6NeX6we25rFdL4mND6nWkRquqxFJ7zlIZY8xpEBGGdk5iaOck8g+d4JXl2by0bBt3vbySZvXjGZ/WlusGtaNds3peh1rj2BGLMcaEqKRE+WRzPrOWbWPRhjxKFM7rmsz1g9txUffmxMZE1lGMNYUFYYnFGFPdcg8cY056NrO/zGbnweO0aJjANWe345qz29K6cV2vwwuJJZYgLLEYY7xSVFzChxt3MWvZNj7ZnI8qnN2hCWP6pDCqTwotGtbxOsRyWWIJwhKLMaYmyN57lDdWbmdBZi6b8g4hAmd3aMrlfVMY2bslzRNrVpKxxBKEJRZjTE2zOe8QC9bk8nZmLlm7DuMTGNyxGWPcJJPUIMHrEC2xBGOJxRhTk32Vd4i3M3N5O3MHW/KP4BM4p3MzxvRpxcjeLWnq0S39LbEEYYnFGBMJVJWNOw+xwE0y3+w5SoxPGNq5GZf3TeHSni2rtd8YSyxBWGIxxkQaVWV97kE3yeSybe9RYn3CsC5JjOmbwmU9W9KoXtXe1t8SSxCWWIwxkUxVWbfj4H+by3L2HSMuRhiemsyYPilc0qtFlfQdY4klCEssxphooapk5hxgwZpcFmTmsn3/MeJjfJzX1TmSubhHCxLDlGQssQRhicUYE41UlVXZ+1mQmcuCNbnkHjhOfKyPEV2TGdM3hYt6tKDBGfQf42liEZGRwD+AGOBfqvpAwHhxx48GjgI3q+qKYHVFpCkwB+gAfANMUNV9IjIIeKp00sBvVfWNYPFZYjHGRLuSEmVl9n7eztzBwjW55B08QUKsj5vOac8vx/Ss1DQ9SywiEgN8BVwC5ADpwERVXe9XZjRwF05iGQz8Q1UHB6srIn8F9qrqAyIyDWiiqr8QkXpAgaoWiUgKsBpopapF5cVoicUYU5uUlCjLt+1jQWYu7ZvV45ZhHSs1HS/vbjwIyFLVLW4gs4GxwHq/MmOB59XJUktFpLGbFDoEqTsWGOHWnwksBn6hqkf9plsHiPy2OmOMCSOfTzi7Q1PO7tDU61DKFMqtOFsD2X7vc9xhoZQJVreFquYCuM/NSwuJyGARWQesAaaUdbQiIpNFJENEMvLz80NYDGOMMdUhlMQiZQwLPIoor0wodU8toLpMVXsBZwP3isgpN9hR1adUNU1V05KTkyuapDHGmGoSSmLJAdr6vW8D7AixTLC6eW5zGe7zrsAZq+oG4AjQO4Q4jTHG1AChJJZ0IFVEOopIPHAtMD+gzHzgJnEMAQ64zVvB6s4HJrmvJwHzANyyse7r9kA3nKvGjDHGRIAKT967V2fdCbyHc8nwM6q6TkSmuONnAAtxrgjLwrnc+JZgdd1JPwDMFZFbgW3AeHf4ucA0ESkESoCpqro7LEtrjDGmytkfJI0xppaqqsuNI6uDZmOMMTWeJRZjjDFhFRVNYSKSD3x7BpNIAqL1PI4tW+SK5uWzZasZ2qtq2P+vERWJ5UyJSEZVtDPWBLZskSual8+WLbpZU5gxxpiwssRijDEmrCyxOJ6quEjEsmWLXNG8fLZsUczOsRhjjAkrO2IxxhgTVpZYjDHGhFWtTiwiMlJENolIltuLZVQQkbYi8pGIbBCRdSJyt9cxhZuIxIjIShF52+tYws3tKO9VEdnofobneB1TuIjIj9zv5FoRebmsLjEiiYg8IyK7RGSt37CmIvKBiGx2n5t4GaMXam1icbtNfgwYBfQEJopI5TqOrnmKgJ+oag9gCHBHFC1bqbuBDV4HUUX+Abyrqt2BfkTJcopIa+CHQJqq9sa5Me213kZ1xp4DRgYMmwYsUtVUYJH7vlaptYkFvy6XVbUAKO02OeKpaq6qrnBfH8LZMAX2+hmxRKQNMAb4l9exhJuINATOA/4NoKoFqrrf26jCKhao63aNUY9T+3aKKKr6CbA3YPBYnO7WcZ+vrNagaoDanFhC6XI54olIB6A/sMzbSMLqEeDnON0qRJtOQD7wrNvU9y8Rqe91UOGgqtuBh3C6ycjF6bfpfW+jqhLldrteW9TmxFKpbpMjiYg0AF4D7lHVg17HEw4icjmwS1WXex1LFYkFBgBPqGp/nB5Uo6IpxT3XMBboCLQC6ovIDd5GZapCbU4soXS5HLFEJA4nqcxS1de9jieMhgFXiMg3OM2XF4rIi96GFFY5QI6qlh5hvoqTaKLBxcBWVc1X1ULgdWCoxzFVhQq7XY92tTmxhNLlckQSEcFpo9+gqg97HU84qeq9qtpGVTvgfGYfqmrU7PWq6k4gW0S6uYMuAtZ7GFI4bQOGiEg99zt6EVFyYUKAMrtdr00q7Jo4WlXQbXKkGwbcCKwRkVXusPtUdaGHMZnQ3QXMcnd4tuB29R3pVHWZiLwKrMC5cnElEX77ExF5GRgBJIlIDvAbyu92vdawW7oYY4wJq9rcFGaMMaYKWGIxxhgTVpZYjDHGhJUlFmOMMWFlicUYY0xYWWIxxhgTVpZYjDHGhNX/A77Xj3RLhHqqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "LR_START = 0.0005\n",
    "LR_MAX = 0.001\n",
    "LR_MIN = 0.00015\n",
    "LR_RAMPUP_EPOCHS = 2\n",
    "LR_SUSTAIN_EPOCHS = 0\n",
    "LR_EXP_DECAY = 0.83\n",
    "\n",
    "def lrfn(epoch):\n",
    "    if epoch < LR_RAMPUP_EPOCHS:\n",
    "        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n",
    "    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n",
    "        lr = LR_MAX\n",
    "    else:\n",
    "        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n",
    "    return lr\n",
    "lr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n",
    "\n",
    "plt_lr(EPOCHS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch #0\n",
      "2020-07-04 07:36:26.339532\n",
      "\n",
      "Epoch 00001: LearningRateScheduler reducing learning rate to 0.0005.\n",
      "Epoch 1/12\n",
      "411/411 [==============================] - 768s 2s/step - loss: 1.4918 - sparse_categorical_accuracy: 0.6371 - lr: 5.0000e-04\n",
      "Epoch #1\n",
      "2020-07-04 07:55:38.701107\n",
      "\n",
      "Epoch 00002: LearningRateScheduler reducing learning rate to 0.00075.\n",
      "Epoch 2/12\n",
      "411/411 [==============================] - 769s 2s/step - loss: 0.7336 - sparse_categorical_accuracy: 0.7991 - lr: 7.5000e-04\n",
      "Epoch #2\n",
      "2020-07-04 08:08:29.905076\n",
      "\n",
      "Epoch 00003: LearningRateScheduler reducing learning rate to 0.001.\n",
      "Epoch 3/12\n",
      "411/411 [==============================] - 768s 2s/step - loss: 0.5952 - sparse_categorical_accuracy: 0.8340 - lr: 0.0010\n",
      "Epoch #3\n",
      "2020-07-04 08:21:20.059565\n",
      "\n",
      "Epoch 00004: LearningRateScheduler reducing learning rate to 0.0008554999999999999.\n",
      "Epoch 4/12\n",
      "411/411 [==============================] - 767s 2s/step - loss: 0.4441 - sparse_categorical_accuracy: 0.8721 - lr: 8.5550e-04\n",
      "Epoch #4\n",
      "2020-07-04 08:34:08.766752\n",
      "\n",
      "Epoch 00005: LearningRateScheduler reducing learning rate to 0.000735565.\n",
      "Epoch 5/12\n",
      "411/411 [==============================] - 769s 2s/step - loss: 0.3277 - sparse_categorical_accuracy: 0.9027 - lr: 7.3556e-04\n",
      "Epoch #5\n",
      "2020-07-04 08:46:59.476429\n",
      "\n",
      "Epoch 00006: LearningRateScheduler reducing learning rate to 0.0006360189499999999.\n",
      "Epoch 6/12\n",
      "411/411 [==============================] - 769s 2s/step - loss: 0.2437 - sparse_categorical_accuracy: 0.9253 - lr: 6.3602e-04\n",
      "Epoch #6\n",
      "2020-07-04 08:59:50.461033\n",
      "\n",
      "Epoch 00007: LearningRateScheduler reducing learning rate to 0.0005533957284999999.\n",
      "Epoch 7/12\n",
      "411/411 [==============================] - 769s 2s/step - loss: 0.1829 - sparse_categorical_accuracy: 0.9422 - lr: 5.5340e-04\n",
      "Epoch #7\n",
      "2020-07-04 09:12:40.931356\n",
      "\n",
      "Epoch 00008: LearningRateScheduler reducing learning rate to 0.000484818454655.\n",
      "Epoch 8/12\n",
      "411/411 [==============================] - 770s 2s/step - loss: 0.1404 - sparse_categorical_accuracy: 0.9547 - lr: 4.8482e-04\n",
      "Epoch #8\n",
      "2020-07-04 09:25:32.351368\n",
      "\n",
      "Epoch 00009: LearningRateScheduler reducing learning rate to 0.00042789931736364993.\n",
      "Epoch 9/12\n",
      "411/411 [==============================] - 770s 2s/step - loss: 0.1078 - sparse_categorical_accuracy: 0.9643 - lr: 4.2790e-04\n",
      "Epoch #9\n",
      "2020-07-04 09:38:23.784520\n",
      "\n",
      "Epoch 00010: LearningRateScheduler reducing learning rate to 0.0003806564334118294.\n",
      "Epoch 10/12\n",
      "411/411 [==============================] - 769s 2s/step - loss: 0.0848 - sparse_categorical_accuracy: 0.9722 - lr: 3.8066e-04\n",
      "Epoch #10\n",
      "2020-07-04 09:51:15.105931\n",
      "\n",
      "Epoch 00011: LearningRateScheduler reducing learning rate to 0.0003414448397318184.\n",
      "Epoch 11/12\n",
      "411/411 [==============================] - 770s 2s/step - loss: 0.0681 - sparse_categorical_accuracy: 0.9786 - lr: 3.4144e-04\n",
      "Epoch #11\n",
      "2020-07-04 10:04:06.849536\n",
      "\n",
      "Epoch 00012: LearningRateScheduler reducing learning rate to 0.0003088992169774093.\n",
      "Epoch 12/12\n",
      "411/411 [==============================] - 768s 2s/step - loss: 0.0547 - sparse_categorical_accuracy: 0.9826 - lr: 3.0890e-04\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f7fca4d1c10>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(\n",
    "    get_training_dataset(do_aug=DO_AUG), steps_per_epoch=STEPS_PER_EPOCH,\n",
    "    epochs=EPOCHS,\n",
    "    callbacks=[es_acc, epoch_cb, lr_schedule], verbose=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEWCAYAAABollyxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXxV9Z3/8deHkAAJYUlYhEDYZREFFVERFUUt7kut22jV2rHa+qud2lbttLXbTJ2pM506tUXrONqpuLRKpZaqNVQURWUxyJIgOwnBrKwhIdvn98c54CUGcsGEm3vv+/l43EfuPed77vmce5P3Pfnec77H3B0REUlcnWJdgIiItC8FvYhIglPQi4gkOAW9iEiCU9CLiCQ4Bb2ISIJT0Iu0ITP7q5nd/Bmf4xYzW9BWNYko6BOYmU01s3fMbIeZVZnZ22Z2Sqzrao2ZTTazuWa2Paz7fTO7Ncpl3zCzL7d3jQfj7he6+1PtvR4zyzCz3WY2t73XJfFPQZ+gzKwH8DLw30AWkAP8CNjbDuvq3IbPdTowD5gPjASygTuBC9tqHe3BAkfz7+lqgvfyAjMbcBTX26bvtxwl7q5bAt6AScD2Q8y/BXib4INgB1AITI+YfytQAOwC1gNfiZg3DSgG7gU+Bv4P6EPwwbIdqALeAjqF7QcCLwDlwAbg64eoawHwyCHm9w7XUw5sC+8PCuf9C9AI1AK7gV+F08cAfwvrWg1cE/F82cCfgZ3AIuCnwIKI+VPC6TvCn1Mi5r0RrvNtoIbgg+kN4MsRbf4x4nVcBZwUTr8PWBcx/cpm782Cg70GYZt54bqXAt9qNm8q8E74XhQBt4TTuwH/AWwKt2dBOG0aUNzsOTYC54X3fwj8Efh9+Dp9GZgMLAzXsRX4FZAWsfxxEa95KfBd4BhgD5Ad0e7k8L1MjfXfTCLfYl6Abu30xkIPoBJ4imBvuHez+bcADcA/AanAteEff1Y4/2JgBGDA2eEf6L6QmhYu+29AlzAsfgbMDJ8rFTgzXLYTsAT4AZAGDCf44PhcCzWnEwT1OYfYrmzg82HbTOAPwJ8i5jcP2oww7G4FOgMnARXAceH8Z8NbOjAubLsgnJdF8GFyU7js9eHj7Ih1bQ5DrXO43fvXD3wB2AKcEr4WI4EhEfMGhq/PtUA1MCDivTlo0AO5QFNY7z3Ah83m7QprTQ1fr4nhvEfC+nKAFIIPsS5EF/T1wBVhvd0IAvq0cLuHEnyYfSNsn0kQ/vcAXcPHp4bz5gJ3RqznF8B/x/rvJdFvMS9At3Z8c2Es8CTB3ncDMAfoH867BSgBLKL9+8BNB3muPwF3h/enAXVA14j5PwZeAkY2W+5UYHOzafcD/9vCOnIAB8YcxjZOBLZFPN4ftOHja4G3mi3zKPBAGHb1wOiIefv36AkC/v1myy7kkz3kN4AfN5u/f/3Aq/tesyi2Ix+4POK9OVTQfw/ID+8PJPhwPDHitZ3dwjKdCP7rmNDCvGm0HvRvtlL/N/atl+BD5oODtLsWeDu8n0LwH+HkWP+tJPpNffQJzN0L3P0Wdx8EjCcIhf+KaLLFw7+40KawDWZ2oZm9G34Zuh24iKB7Zp9yd6+NePxzYC3wmpmtN7P7wulDgIHhF6vbw+f6LtC/hZK3EeypHrTP2czSzexRM9tkZjuBN4FeZpZykEWGAKc2W/8/EHQj9CXYIy2KaB95f2D4mkTaRPCB1FL75gYTdM+0tB1fNLP8iJrGc+DreyhfBJ4GcPcSgu8z9h3pc7B19iHYu26xnigcsJ1mdqyZvWxmH4fvw7/ySf0H3W6CnYFxZjYcOB/Y4e7vH2FNEiUFfZJw90KCvfvxEZNzzMwiHucCJWbWhaBP/SGC/wB6EfzLHdn2gGFP3X2Xu9/j7sOBS4Fvmtl0goDY4O69Im6Z7n5RCzXuIdhj/vwhNuUeYDRBV0AP4Kxw+r7amg/HWgTMb7b+7u5+J0HfcAMwKKL94Ij7JQQfFJFyCbpjWnwdWlj3iOYTzWwI8FvgLoJuoF7ACg58fVtkZlOAUcD9Ych+TPBf0/Xhl6QtrpOgu6r2IPOqCbqu9q0jheBDMFLz7fwNwfc6o8L34bsR9R+sBsKdg+cJPmxvIvh+R9qZgj5BmdkYM7vHzAaFjwcT/Ev9bkSzfsDXzSzVzL5A0NUzl6AvvQthEJrZhcAFrazvEjMbGX5w7CToTmgk6A7aaWb3mlk3M0sxs/GHOMzzO8AtZvZtM8sOn3uCmT0bzs8k6ILYbmZZBF0wkUoJvgfY52XgWDO7KdzOVDM7xczGunsj8CLww/A/hTEEe8v7zA2XvcHMOpvZtQT94i8f6rWI8DjwLTM7OTwqZ2QY8hkEwVkebt+tHPgBfCg3E3zJOY6g22piuGw6wXcxTwPnmdk1Yc3ZZjbR3ZuAJ4D/NLOB4ftwevih/hHQ1cwuNrNUgq6hLq3UkUnwPu8OX7c7I+a9DBxjZt8wsy5mlmlmp0bM/x1B99RlBF/wSjtT0CeuXQR7eu+ZWTVBwK8g2CPe5z2CvcMKgiM4rnb3SnffBXydYM9rG3ADQf/+oYwCXic42mUh8Gt3fyMM00sJAmlDuK7HgZ4tPYm7vwOcG97Wm1kV8BhB6ELQ9dQtfJ53gVeaPcUvgavNbJuZPRxuywXAdQR76B/zyZfIEOxV9+STo4eeITwE1d0rgUvC16yS4EPoEnevaOW12LctfyB4XWcRvB9/IviyexXB0S8LCT6Yjic4cueQzKwrcA3Bl5cfR9w2hLXf7O6bCbrZ7iE44iUfmBA+xbeA5QRHD1WFr0Mnd98BfJXgfdlCsIdf3Eo53yL4vdhF8N/JcxHbvYugW+ZSgtd1DXBOxPy3Cbrolrr7xta2Wz47O7CLVpKFmd1C8KXh1FjX0pGY2b8Bx7j7Zzq7VQ7NzOYBs9z98VjXkgy0Ry9JLeziOiHsWpkM3AbMjnVdiSzstjuJiP8CpH3pDDdJdpkE3TUDgTKCLpWXYlpRAjOzpwiOx7877OKRo0BdNyIiCU5dNyIiCa5Ddt306dPHhw4dGusyRETixpIlSyrcvfn5D0AHDfqhQ4eyePHiWJchIhI3zKz5Wdz7qetGRCTBKehFRBKcgl5EJMF1yD76ltTX11NcXExtbW3rjZNY165dGTRoEKmpqbEuRUQ6iLgJ+uLiYjIzMxk6dCgHDrgo+7g7lZWVFBcXM2zYsFiXIyIdRNx03dTW1pKdna2QPwQzIzs7W//1iMgB4iboAYV8FPQaiUhzcdN1IyKSCNyd6rpGqnbXUVm9l6rqOiqr66iqrsMd7pzW4jVbPhMFfZS2b9/OrFmz+OpXv3pYy1100UXMmjWLXr16HbTND37wA8466yzOO++8z1qmiBxl7s7O2gaqquuoqt5L5e66A8K7qrqOit1799+vrK6jrqGpxefql9lFQR9L27dv59e//vWngr6xsZGUlINdrhTmzp170Hn7/PjHP/7M9YlI22lqcj7eWUvxthoqd+89ILQrmwX6tj111De2PDhkeloKWRlpZGek0S+zC2OO6UF29zSyMtL2Tw9+diGrexoZaQfPks9CQR+l++67j3Xr1jFx4kRSU1Pp3r07AwYMID8/n1WrVnHFFVdQVFREbW0td999N7fffjvwyXAOu3fv5sILL2Tq1Km888475OTk8NJLL9GtWzduueUWLrnkEq6++mqGDh3KzTffzJ///Gfq6+v5wx/+wJgxYygvL+eGG26gsrKSU045hVdeeYUlS5bQp0+015MWkUh76hooqqphU2U1m6v2UFS1h01Ve9hctYfiqhrqGj+9153ZtfP+cB7UO50Jg3qR1f2TwI4M7eyMNLqmtk9wH664DPof/Xklq0p2tulzjhvYgwcuPe6g8x988EFWrFhBfn4+b7zxBhdffDErVqzYfxjjE088QVZWFjU1NZxyyil8/vOfJzs7+4DnWLNmDc888wy//e1vueaaa3jhhRe48cYbP7WuPn36sHTpUn7961/z0EMP8fjjj/OjH/2Ic889l/vvv59XXnmFxx57rE23XyTRuDtlu/ayuWoPmyuDEC8Kg3xT5R4qdu89oH1ml87kZqczun8m54/tT252OoN6p9O3exeyu6fROz2NtM5xdfzKfnEZ9B3B5MmTDzhW/eGHH2b27ODCREVFRaxZs+ZTQT9s2DAmTpwIwMknn8zGjRtbfO6rrrpqf5sXX3wRgAULFux//hkzZtC7d+823R6ReFRb30jxtiC4N4chvjm8X7RtD7X1n+yVm8HAnt3IzUpn+ph+5GanMzgrnSFZ6eRmpdMrPTVhj1qLy6A/1J730ZKRkbH//htvvMHrr7/OwoULSU9PZ9q0aS0ey96lS5f991NSUqipqWnxufe1S0lJoaGhAQj2TkSSUV1DE5sqq1lbtpu1ZbvZWLmvm6Wa0p0H7pWnp6WQm5XOsD4ZnH1sX4aEYZ6blU5O72506dwxulKOtrgM+ljIzMxk166Wr3y2Y8cOevfuTXp6OoWFhbz77rttvv6pU6fy/PPPc++99/Laa6+xbdu2Nl+HSCzV1DWyrnz3/kBfW7abNWW72FS5h4amT3Z0junRldysdM4c1ZfcMMRzs4Of2RlpCbtX/lko6KOUnZ3NGWecwfjx4+nWrRv9+/ffP2/GjBnMnDmTE044gdGjR3Paaae1+fofeOABrr/+ep577jnOPvtsBgwYQGZmZpuvR6S97aipZ23ZbtaFQR4E+m62bK9h3z+uKZ2MIdnpjOzbnRnjj2Fkv+6M7JvJiH4ZpKcptg5Xh7xm7KRJk7z5hUcKCgoYO3ZsjCqKvb1795KSkkLnzp1ZuHAhd955J/n5+S22TfbXSmLP3anYXceasl2s2793Hvws2/VJd0ta504M75PBqP6ZjOzbnVH9uzOyX3eGZmfE7RefsWJmS9x9Ukvz9NEYJzZv3sw111xDU1MTaWlp/Pa3v411SSK4O1u21xzQ3bIv1HfU1O9v171LZ0b0685Zx/YN986DUB/UO52UTupqaW8K+jgxatQoPvjgg1iXIUnM3dlUuYflW3awYssOloe3XbUN+9tkZ6Qxol93Lj5hAKP6BXvnI/t155geXdV3HkNxFfTurl+WVnTErjiJP62FelpKJ8YMyOTSCQMZN6AHx/bPZGS/7mRlpMW4cmlJ3AR9165dqays1FDFh7BvPPquXbvGuhSJI81D/cPiHawoaTnUT8jpyficnhzbP1N96HEkboJ+0KBBFBcXU15eHutSOrR9V5gSaUm0oX7ZhIEcr1BPGHET9KmpqbpqkshhiCbUxyrUk0LcBL2IHNqu2noWbazivfVVCnU5QFRBb2YzgF8CKcDj7v5gs/m9gSeAEUAt8CV3XxHO2wjsAhqBhoMd5ykih6emrpElm7bxzroK3llXyfItO2hscoW6fEqrQW9mKcAjwPlAMbDIzOa4+6qIZt8F8t39SjMbE7afHjH/HHevaMO6RZJOXUMT+UXbeWddBQvXVfLB5u3UNTbRuZMxcXAvvjptBKePyOak3N4dZnhc6Rii2aOfDKx19/UAZvYscDkQGfTjgJ8BuHuhmQ01s/7uXtrWBYski4bGJlaU7GThukreWVfB4o3bqKlvxAyOz+nJrWcM5fQR2ZwyNIuMLuqFlYOL5rcjByiKeFwMnNqszTLgKmCBmU0GhgCDgFLAgdfMzIFH3b3FgdTN7HbgdoDc3NzD2QaRhNDU5Kwu3cU76ypZuK6C99ZXsWtv0Mc+un8m154ymCkjsjl1WDY901NjXK3Ek2iCvqWD1puflfMg8EszyweWAx8A+06XO8PdS8ysH/A3Myt09zc/9YTBB8BjEIx1E+0GiMQrd2d9RfX+YF+4rpJte4JhA4b1yeDSiQOZMiKb04Zn06d7l1aeTeTgogn6YmBwxONBQElkA3ffCdwKYMHZTBvCG+5eEv4sM7PZBF1Bnwp6kWRQVLVnf1fMwvWV+8dTH9izK9PH9mfKiGxOH5HNgJ7dYlypJJJogn4RMMrMhgFbgOuAGyIbmFkvYI+71wFfBt50951mlgF0cvdd4f0LAF0JW5LGnroG5hWW8dZHFbyzvoKiquBiM326d9kf6lNGZJObla4zvqXdtBr07t5gZncBrxIcXvmEu680szvC+TOBscDvzKyR4Eva28LF+wOzw1/gzsAsd3+l7TdDpOOoa2jirTXlzFlWwmsrS6mpb6Rnt1ROG57Fl6cOZ8qIbEb2665gl6MmbsajF+nImpqc9zZUMWdZCX9dsZXte+rpnZ7KRccP4LIJA5k0NEvD8Uq70nj0Iu3A3VmxZScv5W/h5Q+38vHOWtLTUrhgXH8un5jD1FF9SE3RSUoSewp6kcO0rnw3c/JLmLOshA0V1aSmGNNG9+OfJ4zlvLH96Zamk5WkY1HQi0Rh644aXl62lZeWbWHFlp2YwenDs7nj7OHMOG6AjmuXDk1BL3IQ26rrmLtiKy/ll7BoYxXuMGFQT75/yTguOWEA/Xto3H+JDwp6kQjVexv426pS5iwr4c2Pymlockb26843zzuWSycMZGifjFiXKHLYFPSS9Ooampj/UTkv5W/h9YJSauubyOnVjdvOHMblE3IYOyBTh0JKXFPQS1JqbHLeW1/JnGUlzF2+lZ21DWRlpPGFkwdz2cSBnJzbm046HFIShIJekkrF7r08+fZGnl9cRNmuvWSkpfC5447hsokDOWOkDoeUxKSgl6RQVLWHx95cz/OLi6hrbGL6mH5ceeIgzh3TT4dDSsJT0EtCW1Wyk5nz1/GX5VvpZPD5kwbxj2cNZ0Tf7rEuTeSoUdBLwnEPhiOYOX8db6wuJyMthdumDuNLZwzjmJ46JFKSj4JeEkZTk/O3glJmzl/HB5u3k52Rxrc/N5obTx2iE5okqSnoJe7VNTTxUv4WZs5fx7ryagZndeMnV4znCycP0rVTRVDQSxyr3tvAM+9v5n8WbGDrjlrGDujBw9efyEXjj6Gzjp4R2U9BL3GncvdennpnI08t3MSOmnpOG57Fz646nrOP7asTm0RaoKCXuFFUtYfH31rPc4uLqK1v4oJx/blj2ghOyu0d69JEOjQFvXR4hR/v5NH565mzrIROBldMzOErZw9nZL/MWJcmEhcU9NJhLdpYxW/eWMe8wjLS01K4dcpQbjtzmC6cLXKYFPTSoTQ1OfMKy/jN/HUs2bSNrIw0vnn+sXzx9CH0Sk+LdXkicSmqoDezGcAvCS4O/ri7P9hsfm/gCWAEUAt8yd1XRLOsCEB9YxNz8kuYOX8da8p2k9OrGz+67DiumTRYQxSIfEatBr2ZpQCPAOcDxcAiM5vj7qsimn0XyHf3K81sTNh+epTLShJzd+Yu/5h/nVvAlu01jO6fyX9dO5GLTxigAcZE2kg0e/STgbXuvh7AzJ4FLgciw3oc8DMAdy80s6Fm1h8YHsWykqRKttfw/T+tIK+wjOMG9uCnV4xn2mgdIinS1qIJ+hygKOJxMXBqszbLgKuABWY2GRgCDIpyWUkyjU3OU+9s5D9eW02Tw/cuHsstU4bqJCeRdhJN0Le0e+XNHj8I/NLM8oHlwAdAQ5TLBisxux24HSA3NzeKsiQerSzZwXdfXM6y4h1MG92Xn1w+nsFZ6bEuSyShRRP0xcDgiMeDgJLIBu6+E7gVwIL/uzeEt/TWlo14jseAxwAmTZrU4oeBxK+aukb+K+8jHn9rA73TU3n4+hO59IQB6qYROQqiCfpFwCgzGwZsAa4DbohsYGa9gD3uXgd8GXjT3XeaWavLSuJ786Ny/vlPyymqquHaSYO5/6IxOlRS5ChqNejdvcHM7gJeJThE8gl3X2lmd4TzZwJjgd+ZWSPBF623HWrZ9tkU6Wgqd+/lp38pYPYHWxjeJ4Nnbz+N04Znx7oskaRj7h2vl2TSpEm+ePHiWJchR8jdeWHpFn76l1VU723gzrNH8NVzRmrIYJF2ZGZL3H1SS/N0Zqy0qY0V1Xx39nLeWVfJpCG9+dlVxzOqv8akEYklBb20ifrGJh57cz0P560hLaUT/3LleK4/JZdOnfRlq0isKejlM1u6eRv3v7Cc1aW7uOj4Y3jg0uPo30PXZhXpKBT0csR21dbz0Kur+d27mzimR1d++8VJnD+uf6zLEpFmFPRyRF5d+TEPvLSS0l213Hz6UL71udF076JfJ5GOSH+Zclg+3lHLA3NW8OrKUsYck8nMm05m4uBesS5LRA5BQS9RaWpynn5vE//+ymrqGpu4d8YYvnzmMI0wKRIHFPTSqo9Kd3HfCx+ydPN2po7sw79cOZ4h2RmxLktEoqSgl4OqrW/kkb+vZeb8dWR2TeU/r5nAlSfmaHwakTijoJcWLVxXyXdnL2dDRTVXnZTD9y4eR1aGxqcRiUcKejlAU5PzyN/X8p+vf0RuVjq/v+1Upo7qE+uyROQzUNDLfjtr6/nmc8t4vaCUKyYO5F+vOp70NP2KiMQ7/RULEHzh+pX/W0JR1R5+eOk4bp4yVH3xIglCQS/85cOtfPuPy0hP68ysfzyNycOyYl2SiLQhBX0Sa2hs4t9fXc1jb67npNxe/ObGkzVGjUgCUtAnqcrde7lr1gcsXF/JTacN4fuXjCOts05+EklECvoktKxoO3f+fgkV1XX8/OoT+MKkwa0vJCJxS0GfZJ5btJnv/2klfTO78OKdUxif0zPWJYlIO1PQJ4m9DY38cM4qnnl/M1NH9uHh60/UCVAiSUJBnwS27qjhjt8vZVnRdr46bQT3XDCaFF35SSRpRBX0ZjYD+CWQAjzu7g82m98T+D2QGz7nQ+7+v+G8jcAuoBFoONjFa6V9LFxXyV2zllJb38jMG09ixvgBsS5JRI6yVoPezFKAR4DzgWJgkZnNcfdVEc2+Bqxy90vNrC+w2syedve6cP457l7R1sXLwbk7/7NgAz/7ayFDs9N59KbTGNlPF+kWSUbR7NFPBta6+3oAM3sWuByIDHoHMi04lbI7UAU0tHGtEqU9dQ18548f8vKHW/nccf156AsTyOyaGuuyRCRGogn6HKAo4nExcGqzNr8C5gAlQCZwrbs3hfMceM3MHHjU3R9raSVmdjtwO0Bubm7UGyAH2lhRzVf+bwlrynbxnRmjufPsERrKQCTJRRP0LaWEN3v8OSAfOBcYAfzNzN5y953AGe5eYmb9wumF7v7mp54w+AB4DGDSpEnNn1+ikFdQyjeeyyelk/HUlyZz5qi+sS5JRDqAaE6FLAYiz6gZRLDnHulW4EUPrAU2AGMA3L0k/FkGzCboCpI21NTk/OJvH3HbU4vJzUrnz3dNVciLyH7RBP0iYJSZDTOzNOA6gm6aSJuB6QBm1h8YDaw3swwzywynZwAXACvaqniBHXvq+fLvFvPLvDV8/qRBvHDnFAZnpce6LBHpQFrtunH3BjO7C3iV4PDKJ9x9pZndEc6fCfwEeNLMlhN09dzr7hVmNhyYHfYRdwZmufsr7bQtSadg607u+P0SSrbX8JMrxnPjqbnqjxeRTzH3jtcdPmnSJF+8eHGsy+jQXsrfwn0vLCeza2d+c+NJnDxEQwuLJDMzW3Kw85R0ZmycqW9s4sG/FvI/CzZwytDePPIPJ9EvU0MLi8jBKejjSPmuvdw1aynvbajililD+eeLx5KaoqGFReTQFPRxYsmmbXzt6aVsr6njF9dO4MoTB8W6JBGJEwr6Dq6xyfnNG2v5xetrGNirKy/cOYXjBmpoYRGJnoK+A9u6o4Z/ei6fd9dXcdmEgfz0yvH00FAGInKYFPQd1CsrPubeFz6kvrGJh74wgc+flKNDJ0XkiCjoO5iaukZ+8pdVzHpvM8fn9OTh609kWJ+MWJclInFMQd+BFGzdydef+YA1Zbv5ylnDueeC0bpgt4h8Zgr6DsDdeeqdjfzrXwvp2S2V/7tNA5KJSNtR0MdY5e69fOePH5JXWMa5Y/rx86tPILt7l1iXJSIJREEfQwvWVPDN5/PZXlPPDy8dx81ThuoLVxFpcwr6GKhraOI/XlvNo2+uZ2S/7jz1pcmMHdAj1mWJSIJS0B9lGyqqufvZD/iweAc3nJrL9y8eR7e0lFiXJSIJTEF/lLg7Lyzdwg9eWkFqSidm3ngSM8YPiHVZIpIEFPRHwc7aer43ewVzlpVw6rAsfnHtRAb26hbrskQkSSjo29nSzdv4+jMfsHVHLfecfyxfPWckKZ30hauIHD0K+nYSORjZgJ5def4rp3PykN6xLktEkpCCvh1s3VHDN57N570NGoxMRGJPQd/GNBiZiHQ0Cvo2osHIRKSjimrELDObYWarzWytmd3XwvyeZvZnM1tmZivN7NZol00EBVt3ctmvFjDrvc185azhvHDnFIW8iHQYre7Rm1kK8AhwPlAMLDKzOe6+KqLZ14BV7n6pmfUFVpvZ00BjFMvGLQ1GJiLxIJqum8nAWndfD2BmzwKXA5Fh7UCmBZ3R3YEqoAE4NYpl41JNXSN3zVqqwchEpMOLJuhzgKKIx8UEAR7pV8AcoATIBK519yYzi2ZZAMzsduB2gNzc3KiKj6XZH2whr7CM7108ltumDtMXriLSYUXTR99Sgnmzx58D8oGBwETgV2bWI8plg4nuj7n7JHef1Ldvx+/+yCsoZVDvbgp5Eenwogn6YmBwxONBBHvukW4FXvTAWmADMCbKZeNOTV0jC9ZWMH1MP4W8iHR40QT9ImCUmQ0zszTgOoJumkibgekAZtYfGA2sj3LZuPPOugr2NjRx7tj+sS5FRKRVrfbRu3uDmd0FvAqkAE+4+0ozuyOcPxP4CfCkmS0n6K65190rAFpatn025ejJKywjPS2F04ZnxboUEZFWRXXClLvPBeY2mzYz4n4JcEG0y8Yzd2deQRlnjupDl84aR15EOr6oTpiST6ws2cnHO2uZrm4bEYkTCvrDNK+wDDM4Z3S/WJciIhIVBf1hyisoZcKgXvTN1MlRIhIfFPSHoWxXLcuKdzB9jPbmRSR+KOgPw98LywDUPy8icUVBfxjyCsoY2LMrYwdkxroUEZGoKeijVFsfnA177lidDSsi8UVBH6V311eyp66R6WPUbSMi8UVBH6V5hWV0Te3E6SOyY12KiMhhUdBHwd3JKyhj6si+dE3V2bAiEo6AdL0AAA4CSURBVF8U9FFYXbqLLdtrmD5Wh1WKSPxR0EchryA4rPJcHT8vInFIQR+FvIJSjs/pSf8eXWNdiojIYVPQt6Ji914+KNqubhsRiVsK+la8sbocd3RYpYjELQV9K+YVltK/RxfG5/SIdSkiIkdEQX8IdQ1NvPlRBefq2rAiEscU9Ifw/oYqdu9t4Fx124hIHFPQH8LrBaV06dyJqSP7xLoUEZEjFlXQm9kMM1ttZmvN7L4W5n/bzPLD2wozazSzrHDeRjNbHs5b3NYb0F7cnbzCUqaMyKZbms6GFZH41WrQm1kK8AhwITAOuN7MxkW2cfefu/tEd58I3A/Md/eqiCbnhPMntWHt7Wpt2W6Kqmo09ryIxL1o9ugnA2vdfb271wHPApcfov31wDNtUVws5RXqbFgRSQzRBH0OUBTxuDic9ilmlg7MAF6ImOzAa2a2xMxuP9hKzOx2M1tsZovLy8ujKKt9zSsoY9yAHgzs1S3WpYiIfCbRBH1LxxX6QdpeCrzdrNvmDHc/iaDr52tmdlZLC7r7Y+4+yd0n9e3bN4qy2s+26joWb6rS2bAikhCiCfpiYHDE40FAyUHaXkezbht3Lwl/lgGzCbqCOrT5H5XT5Lo2rIgkhmiCfhEwysyGmVkaQZjPad7IzHoCZwMvRUzLMLPMffeBC4AVbVF4e3q9oJQ+3dM4IadnrEsREfnMOrfWwN0bzOwu4FUgBXjC3Vea2R3h/Jlh0yuB19y9OmLx/sDs8KzSzsAsd3+lLTegrdU3NjH/o3JmHHcMnTrpbFgRiX+tBj2Au88F5jabNrPZ4yeBJ5tNWw9M+EwVHmWLNlaxq7ZB3TYikjB0Zmwz8wrKSEvpxJmjdDasiCQGBX0z8wrLOG1ENhldovpnR0Skw1PQR1hfvpv1FdVM10lSIpJAFPQR5ulsWBFJQAr6CK8XlDK6fyaDs9JjXYqISJtR0Id21NSzaOM2nQ0rIglHQR+a/1E5jU2uoBeRhKOgD80rKCUrI42Jg3vHuhQRkTaloAcaGpv4++pypo3uS4rOhhWRBKOgB5Zu3s6Omnqm69qwIpKAFPRAXmEpnTsZZx2rs2FFJPEo6IG8gjJOHZ5FZtfUWJciItLmkj7oN1VWs7Zst7ptRCRhJX3Q5xUEZ8PqsEoRSVRJH/TzCssY2a87Q7IzYl2KiEi7SOqg31Vbz3sbKjWImYgktKQO+rfWVFDf6BrETEQSWlIHfV5BGT27pXLyEJ0NKyKJK2mDvrHJ+fvqMqaN7kvnlKR9GUQkCSRtwuUXbaequk7XhhWRhBdV0JvZDDNbbWZrzey+FuZ/28zyw9sKM2s0s6xolo2VvIJSUjoZZ4/qG+tSRETaVatBb2YpwCPAhcA44HozGxfZxt1/7u4T3X0icD8w392rolk2VuYVlnHK0N70TNfZsCKS2KLZo58MrHX39e5eBzwLXH6I9tcDzxzhskdF8bY9FH68S2fDikhSiCboc4CiiMfF4bRPMbN0YAbwwhEse7uZLTazxeXl5VGUdeT2XRtWZ8OKSDKIJuhbGqDdD9L2UuBtd6863GXd/TF3n+Tuk/r2bd9+87yCMob1yWB43+7tuh4RkY4gmqAvBgZHPB4ElByk7XV80m1zuMseFdV7G1i4TmfDikjyiCboFwGjzGyYmaURhPmc5o3MrCdwNvDS4S57NC1YW0FdYxPnqttGRJJE59YauHuDmd0FvAqkAE+4+0ozuyOcPzNseiXwmrtXt7ZsW2/E4cgrKCWza2dOGZoVyzJERI6aVoMewN3nAnObTZvZ7PGTwJPRLBsrTU3OvMJyzj62L6k6G1ZEkkRSpd2HW3ZQsXuvjrYRkaSSVEE/r6CUTgbTjlXQi0jySKqgf72gjJOH9KZ3RlqsSxEROWqSJui37qhh1dadGsRMRJJO0gT9/rNhdfy8iCSZpAn6vIIycrPSGdlPZ8OKSHJJiqCvqWvk7bUVnDumH2YtjcogIpK4kiLo315bwd6GJh1WKSJJKSmCPq+wjIy0FE4dlh3rUkREjrqED3p3Z15hKWcd25e0zgm/uSIin5LwybeyZCelO/fqsEoRSVoJH/R5BWWYwbTRujasiCSnxA/6wlJOHNyLPt27xLoUEZGYSOigL9tZy4fFO9RtIyJJLaGDXteGFRFJ8KDPKywjp1c3RvfPjHUpIiIxk7BBX1vfyII1OhtWRCRhg37h+kpq6hvVbSMiSS9hg35eQRnpaSmcNlxnw4pIcosq6M1shpmtNrO1ZnbfQdpMM7N8M1tpZvMjpm80s+XhvMVtVfihuDt5BaVMHdmHrqkpR2OVIiIdVqsXBzezFOAR4HygGFhkZnPcfVVEm17Ar4EZ7r7ZzJr3l5zj7hVtWPchFX68i5Idtdx93qijtUoRkQ4rmj36ycBad1/v7nXAs8DlzdrcALzo7psB3L2sbcs8PHkFpQCco4uMiIhEFfQ5QFHE4+JwWqRjgd5m9oaZLTGzL0bMc+C1cPrtn63c6OQVljFhUE/6ZXY9GqsTEenQWu26AVo6NtFbeJ6TgelAN2Chmb3r7h8BZ7h7Sdid8zczK3T3Nz+1kuBD4HaA3Nzcw9mGA1Ts3kt+0Xb+6bxjj/g5REQSSTR79MXA4IjHg4CSFtq84u7VYV/8m8AEAHcvCX+WAbMJuoI+xd0fc/dJ7j6pb98jH4Ds74VluMO56rYREQGiC/pFwCgzG2ZmacB1wJxmbV4CzjSzzmaWDpwKFJhZhpllAphZBnABsKLtyv+0vIIyjunRleMG9mjP1YiIxI1Wu27cvcHM7gJeBVKAJ9x9pZndEc6f6e4FZvYK8CHQBDzu7ivMbDgwOzwztTMwy91faa+N2dvQyFtryrn8xBydDSsiEoqmjx53nwvMbTZtZrPHPwd+3mzaesIunKPh/Q1VVNc1Ml3dNiIi+yXUmbF5BWV0Te3EGSP7xLoUEZEOI2GC3t3JKyzljBE6G1ZEJFJUXTfxoLa+iSnD+zBlpMa2ERGJlDBB3y0thX+7+oRYlyEi0uEkTNeNiIi0TEEvIpLgFPQiIglOQS8ikuAU9CIiCU5BLyKS4BT0IiIJTkEvIpLgzL35NURiz8zKgU1HuHgf4Khdn/Yo07bFr0TePm1bxzDE3Vu8mEeHDPrPwswWu/ukWNfRHrRt8SuRt0/b1vGp60ZEJMEp6EVEElwiBv1jsS6gHWnb4lcib5+2rYNLuD56ERE5UCLu0YuISAQFvYhIgkuYoDezGWa22szWmtl9sa6nLZnZYDP7u5kVmNlKM7s71jW1NTNLMbMPzOzlWNfSlsysl5n90cwKw/fv9FjX1JbM7J/C38kVZvaMmXWNdU1HysyeMLMyM1sRMS3LzP5mZmvCn71jWeORSoigN7MU4BHgQmAccL2ZjYttVW2qAbjH3ccCpwFfS7DtA7gbKIh1Ee3gl8Ar7j4GmEACbaOZ5QBfBya5+3ggBbgutlV9Jk8CM5pNuw/Ic/dRQF74OO4kRNADk4G17r7e3euAZ4HLY1xTm3H3re6+NLy/iyAscmJbVdsxs0HAxcDjsa6lLZlZD+As4H8A3L3O3bfHtqo21xnoZmadgXSgJMb1HDF3fxOoajb5cuCp8P5TwBVHtag2kihBnwMURTwuJoGCMJKZDQVOBN6LbSVt6r+A7wBNsS6kjQ0HyoH/DbulHjezjFgX1VbcfQvwELAZ2ArscPfXYltVm+vv7lsh2OEC+sW4niOSKEFvLUxLuONGzaw78ALwDXffGet62oKZXQKUufuSWNfSDjoDJwG/cfcTgWri9F//loT91ZcDw4CBQIaZ3RjbqqQliRL0xcDgiMeDiON/IVtiZqkEIf+0u78Y63ra0BnAZWa2kaDL7Vwz+31sS2ozxUCxu+/77+uPBMGfKM4DNrh7ubvXAy8CU2JcU1srNbMBAOHPshjXc0QSJegXAaPMbJiZpRF8ITQnxjW1GTMzgn7eAnf/z1jX05bc/X53H+TuQwnet3nunhB7he7+MVBkZqPDSdOBVTEsqa1tBk4zs/Twd3Q6CfRlc2gOcHN4/2bgpRjWcsQ6x7qAtuDuDWZ2F/AqwTf/T7j7yhiX1ZbOAG4ClptZfjjtu+4+N4Y1SXT+H/B0uAOyHrg1xvW0GXd/z8z+CCwlODLsA+J4yAAzewaYBvQxs2LgAeBB4Hkzu43gg+0LsavwyGkIBBGRBJcoXTciInIQCnoRkQSnoBcRSXAKehGRBKegFxFJcAp6kTZkZtMSbQROiX8KehGRBKegl6RkZjea2ftmlm9mj4bj4e82s/8ws6VmlmdmfcO2E83sXTP70Mxm7xuT3MxGmtnrZrYsXGZE+PTdI8agfzo8a1QkZhT0knTMbCxwLXCGu08EGoF/ADKApe5+EjCf4MxIgN8B97r7CcDyiOlPA4+4+wSCMV62htNPBL5BcG2E4QRnNovETEIMgSBymKYDJwOLwp3tbgSDVTUBz4Vtfg+8aGY9gV7uPj+c/hTwBzPLBHLcfTaAu9cChM/3vrsXh4/zgaHAgvbfLJGWKeglGRnwlLvff8BEs+83a3eo8UEO1R2zN+J+I/o7kxhT140kozzgajPrB/uvCzqE4O/h6rDNDcACd98BbDOzM8PpNwHzw+sBFJvZFeFzdDGz9KO6FSJR0p6GJB13X2Vm3wNeM7NOQD3wNYILgxxnZkuAHQT9+BAMTzszDPLIEShvAh41sx+HzxGXIxtK4tPolSIhM9vt7t1jXYdIW1PXjYhIgtMevYhIgtMevYhIglPQi4gkOAW9iEiCU9CLiCQ4Bb2ISIL7/yV+QPJu5G7FAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEWCAYAAAB2X2wCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXxU9b3/8dcnk30hIQsICRAgEdkRArKEVlxR22Krte7X1mpd297b21/tr7V2u7f2/myr3uuGXlt3a9W22qq12KKgoAZEZVMCsgQwhEACIYRs398fM2CIIRnIJCdz5v18PPLIzDnfmfmMy3tOPvM932POOUREJPrFeV2AiIhEhgJdRMQnFOgiIj6hQBcR8QkFuoiITyjQRUR8QoEuIuITCnSJCWa20cxO87oOkZ6kQBcR8QkFusQsM0sys9vNbFvo53YzSwrtyzWzv5hZjZntMrNFZhYX2vc9M9tqZnvN7AMzO9XbdyISFO91ASIe+gEwHZgEOODPwA+Bm4HvABVAXmjsdMCZ2SjgBmCqc26bmRUCgd4tW6RjOkKXWHYJ8FPn3A7nXBXwE+Cy0L4mYBAwzDnX5Jxb5IILH7UAScAYM0twzm10zq33pHqRdhToEssGA5va3N8U2gbw/4By4GUz22BmNwE458qBbwM/BnaY2ZNmNhiRPkCBLrFsGzCszf2hoW045/Y6577jnBsBfB74t4O9cufc48650tBjHfDL3i1bpGMKdIklCWaWfPAHeAL4oZnlmVku8CPgUQAz+5yZFZmZAXsItlpazGyUmZ0S+vK0Adgf2ifiOQW6xJIXCAbwwZ9koAx4D3gfWA78PDS2GFgA1AFLgLudcwsJ9s9vBXYCHwMDgP/ba+9ApBOmC1yIiPiDjtBFRHxCgS4i4hMKdBERn1Cgi4j4hGen/ufm5rrCwkKvXl5EJCotW7Zsp3Mur6N9ngV6YWEhZWVlXr28iEhUMrNNR9qnlouIiE8o0EVEfEKBLiLiE1oPXUR6VVNTExUVFTQ0NHhdSp+WnJxMQUEBCQkJYT9GgS4ivaqiooKMjAwKCwsJrn0m7TnnqK6upqKiguHDh4f9OLVcRKRXNTQ0kJOTozDvhJmRk5Nz1H/FKNBFpNcpzLt2LP+Moi7QP6zcy8//spqGJi1BLSLSVtQFesXueh5Y/BHLN+32uhQRiUI1NTXcfffdR/24s88+m5qamk7H/OhHP2LBggXHWlq3RV2gTxueQ3ycsbh8p9eliEgUOlKgt7R0/lf/Cy+8QFZWVqdjfvrTn3Laaad1q77uiLpAT0+K58ShWbyuQBeRY3DTTTexfv16Jk2axNSpU5kzZw4XX3wx48ePB+Dcc89lypQpjB07lvnz5x96XGFhITt37mTjxo2MHj2aq666irFjx3LGGWewf/9+AK644gqefvrpQ+NvueUWJk+ezPjx41m7di0AVVVVnH766UyePJlvfOMbDBs2jJ07I5NnUTltcVZRLne8so7a+iYyU8OfoykifctPnl/F6m17IvqcYwb345bPjz3i/ltvvZWVK1eyYsUKFi5cyDnnnMPKlSsPTQ988MEHyc7OZv/+/UydOpXzzjuPnJycw55j3bp1PPHEE9x///1ccMEFPPPMM1x66aWfeq3c3FyWL1/O3XffzW233cYDDzzAT37yE0455RS+//3v89JLLx32odFdUXeEDlBalItzsGSDjtJFpHumTZt22FzvO++8k4kTJzJ9+nS2bNnCunXrPvWY4cOHM2nSJACmTJnCxo0bO3zuL33pS58as3jxYi688EIA5s6dS//+/SP2XqLyCH3ikCzSEgMsLt/J3HGDvC5HRI5RZ0fSvSUtLe3Q7YULF7JgwQKWLFlCamoqJ598codzwZOSkg7dDgQCh1ouRxoXCARobm4GgicN9ZSoPEJPCMQxfUQOr5dXe12KiESZjIwM9u7d2+G+2tpa+vfvT2pqKmvXrmXp0qURf/3S0lKeeuopAF5++WV2747cjL2oDHQI9tE/2rmPit31XpciIlEkJyeHWbNmMW7cOL773e8etm/u3Lk0NzczYcIEbr75ZqZPnx7x17/lllt4+eWXmTx5Mi+++CKDBg0iIyMjIs9tPXn435mSkhLXnQtcfPDxXs68/TX+67wJXDB1SAQrE5GetGbNGkaPHu11GZ45cOAAgUCA+Ph4lixZwrXXXsuKFSs6HNvRPyszW+acK+lofFT20AGOH5hObnoSr6/fqUAXkaixefNmLrjgAlpbW0lMTOT++++P2HN3Gehm9iDwOWCHc25cJ+OmAkuBrzjnno5YhUd+PUqLclhcvhPnnNaGEJGoUFxczDvvvNMjzx1OD/13wNzOBphZAPgl8LcI1BS2WUW57Kxr5IPKjr/gEJG+yatWbzQ5ln9GXQa6c+41YFcXw24EngF2HHUF3TCrKBeAxes0H10kWiQnJ1NdXa1Q78TB9dCTk5OP6nHd7qGbWT7wReAUYGoXY68GrgYYOnRod1+awVkpjMhL4/XynXx99ohuP5+I9LyCggIqKiqoqqryupQ+7eAVi45GJL4UvR34nnOupas+tnNuPjAfgrNcIvDalBbl8vSyChqbW0mMj9pZmCIxIyEh4aiuwiPhi0QClgBPmtlG4HzgbjM7NwLPG5ZZRbnUN7awYkvny1qKiPhdtwPdOTfcOVfonCsEngauc879qduVhWn6iBziDC2nKyIxr8tAN7MngCXAKDOrMLMrzewaM7um58vrWmZKAhMKtJyuiEiXPXTn3EXhPplz7opuVXOMSotyuefV9extaCIjWcvpikhs8sW3iLOKcmlpdby5oavZlSIi/uWLQJ88LIvkhDj10UUkpvki0JPiA0wtzFYfXURimi8CHYJ99HU76qjc8+nF6EVEYoFvAv3gMgBvrNdRuojEJt8E+phB/eifmsDidbqKkYjEJt8EelycMbMol9dDy+mKiMQa3wQ6BPvoH+9pYH3VPq9LERHpdb4LdECzXUQkJvkq0IdkpzI0O1Xz0UUkJvkq0CE422Xp+mqaW1q9LkVEpFf5LtBLi3LZe6CZ97bWel2KiEiv8l2gzxiZgxm8rsvSiUiM8V2gZ6clMnZwP/XRRSTm+C7QIdhHX755N/WNzV6XIiLSa/wZ6CNzaWpxvPWRltMVkdjhy0CfWphNYiBO89FFJKb4MtBTEgNMGdaf18u1rouIxI5wrin6oJntMLOVR9h/iZm9F/p5w8wmRr7Mo1danMvq7XuorjvgdSkiIr0inCP03wFzO9n/EfBZ59wE4GfA/AjU1W2fLKero3QRiQ1dBrpz7jXgiN8uOufecM7tDt1dChREqLZuGZ+fSUZyvProIhIzIt1DvxJ48Ug7zexqMyszs7KqqqoIv/ThAnHGzJE5LFqn5XRFJDZELNDNbA7BQP/ekcY45+Y750qccyV5eXmReukjKi3KZWvNfjbvqu/x1xIR8VpEAt3MJgAPAPOcc32maX2wj66zRkUkFnQ70M1sKPAscJlz7sPulxQ5w3PTGJyZrD66iMSE+K4GmNkTwMlArplVALcACQDOuXuBHwE5wN1mBtDsnCvpqYKPhpkxqyiXv6+ppKXVEYgzr0sSEekxXQa6c+6iLvZ/Hfh6xCqKsNLiXP6wrILV2/YwviDT63JERHqML88UbWvmSPXRRSQ2+D7Q8zKSGDUwQ310EfE93wc6BGe7vLVxFw1NLV6XIiLSY2Ii0EuLc2hsbmXZpt1dDxYRiVIxEejThucQH2dqu4iIr8VEoKcnxXPi0CwFuoj4WkwEOgT76O9traW2vsnrUkREekTMBHppUS7OwZINOkoXEX+KmUCfOCSLtMSA5qOLiG/FTKAnBOKYPiJHl6UTEd+KmUCHYB/9o537qNit5XRFxH9iKtBLi0OXpdNRuoj4UEwFevGAdPIyktRHFxFfiqlANzNKi3J5vXwnra26LJ2I+EtMBTrAzJE5VO9r5IPKvV6XIiISUTEX6AcvS6ezRkXEb2Iu0AdnpTAiL019dBHxnZgLdAieNfrWR7tobG71uhQRkYiJyUCfVZRLfWMLK7bUeF2KiEjEdBnoZvagme0ws5VH2G9mdqeZlZvZe2Y2OfJlRtb0ETnEmS5LJyL+Es4R+u+AuZ3sPwsoDv1cDdzT/bJ6VmZKAhMKtJyuiPhLl4HunHsN2NXJkHnAwy5oKZBlZoMiVWBPKS3KZcWWGvY2aDldEfGHSPTQ84Etbe5XhLZ9ipldbWZlZlZWVVUVgZc+drOKcmlpdby5obPPKhGR6BGJQLcOtnV4GqZzbr5zrsQ5V5KXlxeBlz52k4dlkZwQpz66iPhGJAK9AhjS5n4BsC0Cz9ujkuIDTBueoz66iPhGJAL9OeDy0GyX6UCtc257BJ63x5UW5bBuRx2Vexq8LkVEpNvCmbb4BLAEGGVmFWZ2pZldY2bXhIa8AGwAyoH7get6rNoI0zIAIuIn8V0NcM5d1MV+B1wfsYp60ejj+pGdlsji8p18aXKB1+WIiHRLTJ4pelBcnDFjZLCPHvxcEhGJXjEd6BCcj1655wDrq+q8LkVEpFsU6KE++uJ16qOLSHSL+UAfkp3K0OxUXl+v64yKSHSL+UCH4GyXpeuraW7RcroiEr0U6ATbLnsPNPPe1lqvSxEROWYKdGDGyBzM4HX10UUkiinQgey0RMYO7qd1XUQkqinQQ2YV5bJ8827qG5u9LkVE5Jgo0ENKi3JpanG89ZGW0xWR6KRAD5lamE1ifJzWdRGRqKVAD0lOCFAyrD+LyzUfXUSikwK9jVlFuazZvoeddQe8LkVE5Kgp0Ns4uJzuGzprVESikAK9jfH5mWQkx2s+uohEJQV6G4E4Y+bIHBZrOV0RiUIK9HZKi3LZWrOfzbvqvS5FROSoKNDbOdhH11mjIhJtwgp0M5trZh+YWbmZ3dTB/kwze97M3jWzVWb21ciX2juG56YxODNZ89FFJOqEc5HoAHAXcBYwBrjIzMa0G3Y9sNo5NxE4GfiVmSVGuNZeYWbMKsrljfXVtLSqjy4i0SOcI/RpQLlzboNzrhF4EpjXbowDMszMgHRgFxC1i6KUFudSU9/E6m17vC5FRCRs4QR6PrClzf2K0La2/gcYDWwD3ge+5Zz71NUizOxqMyszs7KqqqpjLLnnzRypPrqIRJ9wAt062Na+F3EmsAIYDEwC/sfM+n3qQc7Nd86VOOdK8vLyjrrY3pKXkcQJx2Wojy4iUSWcQK8AhrS5X0DwSLytrwLPuqBy4CPghMiU6I1ZRbm8tXEXDU0tXpciIhKWcAL9baDYzIaHvui8EHiu3ZjNwKkAZjYQGAVsiGShva20KJfG5laWbdrtdSkiImHpMtCdc83ADcDfgDXAU865VWZ2jZldExr2M2Cmmb0PvAJ8zzkX1f2KacOziY8z9dFFJGrEhzPIOfcC8EK7bfe2ub0NOCOypXkrLSmeE4dmqY8uIlFDZ4p2YlZRLu9vraWmvtHrUkREuqRA70RpUS7OwRItpysiUUCB3omJQ7JISwyojy4iUUGB3omEQByzinJ5ZnkFjy7dpCV1RaRPU6B34WfnjmNqYTY//NNKrvjt21TuafC6JBGRDinQuzCwXzIPf20aP5s3ljc/quaM37zG8++2P69KRMR7CvQwmBmXzSjkhW/OZnhuGjc+8Q43PvGOZr+ISJ+iQD8KI/LSefqaGfz7Gcfz4vvbOeM3r7Hwgx1elyUiAijQj1p8II4bTinmT9fPIis1gSt++zY/+OP71DdG7WrBIuITCvRjNC4/k+duKOWq2cN5/K3NnHXHIpZt2uV1WSISwxTo3ZCcEOAH54zhiaum09zi+PK9S/ivl9bS2PyppeBFRHqcAj0Cpo/I4aVvz+bLU4Zw98L1zLvrddZ+rKsdiUjvUqBHSEZyAr88fwL3X15C1d4GvvDfr3Pfq+t1XVIR6TUK9Ag7fcxA/vbtzzDnhDx+8eJaLpy/hM3V9V6XJSIxQIHeA3LSk7j30in8+oKJrN2+l7PueI0n39qspQNEpEcp0HuImfGlyQW89K+fYeKQLG569n2ufKiMHXu1dICI9AwFeg/Lz0rh0StP4pbPj+H18p2c+ZvXeOH97V6XJSI+pEDvBXFxxldnDeev35zNkOxUrntsOd9+8h1q65u8Lk1EfESB3ouKBqTzzLUz+dfTjuf597Zz5u2vsWhdlddliYhPhBXoZjbXzD4ws3Izu+kIY042sxVmtsrMXo1smf6REIjjW6cV88frZpKWFOCy/32LW/68kv2NLV6XJiJRrstAN7MAcBdwFjAGuMjMxrQbkwXcDXzBOTcW+HIP1OorEwqy+Os3Z/O1WcN5aMkmzrlzEe9s3u11WSISxcI5Qp8GlDvnNjjnGoEngXntxlwMPOuc2wzgnNMShGFITgjwo8+P4fGrTuJAcyvn3fMGv3r5Ay0dICLHJJxAzwe2tLlfEdrW1vFAfzNbaGbLzOzyjp7IzK42szIzK6uqUu/4oJkjc3nx27P50uQC/vsf5Vxw3xIqdutkJBE5OuEEunWwrf0ZMvHAFOAc4EzgZjM7/lMPcm6+c67EOVeSl5d31MX6Wb/kBG778kTuvmQy63fUcfYdi/j76kqvyxKRKBJOoFcAQ9rcLwDaX4OtAnjJObfPObcTeA2YGJkSY8vZ4wfxl2+WMjQnlaseLuPnf1mtFoyIhCWcQH8bKDaz4WaWCFwIPNduzJ+B2WYWb2apwEnAmsiWGjuG5aTxzLUz+ZcZw3hg8UdqwYhIWLoMdOdcM3AD8DeCIf2Uc26VmV1jZteExqwBXgLeA94CHnDOrey5sv0vKT7AT+aN466LJ1O+o45z7lzMArVgRKQT5tWCUSUlJa6srMyT1442G3fu4/rHl7Nq2x6u/swIvnvmKBICOidMJBaZ2TLnXElH+5QKUaAwN9iCuXzGMOa/toEL7lvC1pr9XpclIn2MAj1KJCcE+Om8cfzPxSeyrjI4C+aVNWrBiMgnFOhR5nMTBvOXG0vJz0rhyofK+M8X1tDUolkwIqJAj0qFuWk8e91MLp0+lPmvbeAr9y1hm1owIjFPgR6lkhMC/Pzc8fz3RSfyYWUdZ9+5iH+sVQtGJJYp0KPc5ycO5vkbSxmcmcLXflfGL15UC0YkVinQfWB4qAVzyUlDue/VDVw4f6laMCIxSIHuE8kJAf7ji+O586ITWbt9D+fcuYh/rtWilyKxRIHuM18ItWCOy0zhq797m1tfXKsWjEiMUKD70Ii8dP543UwuPmko9766novmL2V7rVowIn6nQPep5IQA//nF8dxx4STWbN/D2Xcs4p8fqAUj4mcKdJ+bNymf528sZWC/ZL7627f55UtraVYLRsSXFOgxYEReOn+6fhYXTRvKPQvXc9H9S/m4tsHrskQkwhToMSI5IcAvvjSe278yiVXb9nD2nYtYqBaMiK8o0GPMuScGWzADMpK4ItSCOdDc4nVZIhIBCvQYNDLUgrlw6hDuWbieM37zGi+v+hiv1sYXkchQoMeo5IQAt543gYe/No2EQBxXP7KMyx98iw8r93pdmogcIwV6jPvM8Xm8+K3Z/PjzY3h3Sw1n3bGIHz+3ipr6Rq9LE5GjFFagm9lcM/vAzMrN7KZOxk01sxYzOz9yJUpPSwjEccWs4Sz87hwumjaEh5ds5OTbFvLIko2a4igSRboMdDMLAHcBZwFjgIvMbMwRxv2S4MWkJQplpyXy83PH89dvzmb0cf24+c+rOOfOxbxRvtPr0kQkDOEcoU8Dyp1zG5xzjcCTwLwOxt0IPANoLlyUGz2oH49fdRL3XjqZfY3NXPzAm3zjkTI2V9d7XZqIdCKcQM8HtrS5XxHadoiZ5QNfBO7t7InM7GozKzOzsqqqqqOtVXqRmTF33CAW/Ntn+e6Zo1i0bien/fpV/uultew70Ox1eSLSgXAC3TrY1n5+2+3A95xznU5ods7Nd86VOOdK8vLywq1RPJScEOD6OUX84zsn87kJg7h74Xrm3LaQZ5ZV0NqqaY4ifUk4gV4BDGlzvwDY1m5MCfCkmW0EzgfuNrNzI1Kh9AnHZSbz669M4tnrZjIoK4Xv/OFdvnjPG7yzebfXpYlISDiB/jZQbGbDzSwRuBB4ru0A59xw51yhc64QeBq4zjn3p4hXK56bPLQ/f7x2Jr/68kS21ezni3e/wb/9fgWVe7Q2jIjXugx051wzcAPB2StrgKecc6vM7Bozu6anC5S+Jy7OOG9KAf/895O57uSR/OW97cy5bSF3/bOchiYtIyDiFfPqdO+SkhJXVlbmyWtLZG2uruc/XljN31ZVMiQ7hR+cPYYzxw7ErKOvX0SkO8xsmXOupKN9OlNUum1oTir3XVbCY18/idSEeK55dBmXPPAmaz/e43VpIjFFgS4RM6sol79+s5SfzRvL6tBVkm7+00p279MyAiK9QYEuERUfiOOyGYUs/PeTuXxGIY+/tZmTb1vI717/SBerFulhCnTpEVmpifz4C2N58VuzGZ+fyY+fX83ZdyzilTWVmr8u0kMU6NKjjh+YwSNXTuP+y0tobGnlyofKOOVXC3lg0QZq65u8Lk/EVzTLRXpNY3MrL636mEeWbOTtjbtJTohj3sR8LpsxjHH5mV6XJxIVOpvlokAXT6zetodHlm7iT+9sZX9TC5OHZnHZjGGcPX4QSfEBr8sT6bMU6NJn1e5v4pllFTy6dBMbdu4jJy2Rr0wdwiXTh5GfleJ1eSJ9jgJd+rzWVsfr63fy8JJNvLKmEoBTRw/k8hnDKC3K1UlKIiGdBXp8bxcj0pG4OGN2cR6zi/Oo2F3P429u5vdvb+HvqysZkZvGpdOHcd6UAjJTErwuVaTP0hG69FkHmlt44f3tPLxkE+9sriElIcC5J+Zz+YxhjB7Uz+vyRDyhlotEvZVba3l4yUb+vGIbB5pbmVrYn8tmFDJ37HEkxmv2rcQOBbr4Rk19I38oq+DRNzexqbqevIwkLpo6hItPGsZxmclelyfS4xTo4jutrY5X11XxyJJN/PODHcSZccaYgVw2YxgzRuToS1TxLX0pKr4TF2fMGTWAOaMGsLm6nsfe3MTvy7bw4sqPKR6QzmUzhvHFE/PJSNaXqBI7dIQuvtHQ1MLz727jkaWbeK+ilrTEAKePGchpYwbymePz6KdwFx9Qy0VizootNTy2dBML1lSyu76J+Dhj+ogcTh09gNNGD2RIdqrXJYocEwW6xKyWVsfyzbtZsKaSBasrWV+1D4BRAzOC4T5mIJMKsoiLU89dooMCXSRk4859wXBfU8nbG3fT0urITU9kzqhguM8uziU1UV8tSd/V7UA3s7nAHUAAeMA5d2u7/ZcA3wvdrQOudc6929lzKtDFa7X1TSz8cAcL1uxg4Qc72NvQTGJ8HDNH5nDa6IGcOnoAgzK1noz0Ld0KdDMLAB8CpwMVwNvARc651W3GzATWOOd2m9lZwI+dcyd19rwKdOlLmlpaefujXSxYs4MFayrZvKsegHH5/Tj1hIGcPmYgYwf303RI8Vx3A30GwYA+M3T/+wDOuV8cYXx/YKVzLr+z51WgS1/lnKN8Rx0L1uzglTWVLNu8G+fguH7JnDJ6AKePHsiMkTkkJ2iZX+l93Z2Hng9saXO/Aujs6PtK4MUjFHI1cDXA0KFDw3hpkd5nZhQPzKB4YAbXnjyS6roD/PODKl5ZU8mf39nK429uJiUhQGlxLqePHsicEwaQl5HkddkiYQV6R39jdnhYb2ZzCAZ6aUf7nXPzgfkQPEIPs0YRT+WkJ3H+lALOn1LAgeYWlm7YxYLVlbyyppK/r67EDCYNyeKUUQOYUtif8fmZOqFJPBFOoFcAQ9rcLwC2tR9kZhOAB4CznHPVkSlPpG9Jig/w2ePz+Ozxefx03ljWbN/LgjXBcP/V3z8EwAyK8tKZOCSLiUOymFSQxajjMrSImPS4cHro8QS/FD0V2ErwS9GLnXOr2owZCvwDuNw590Y4L6weuvjN7n2NvLe1lne31PDulhpWbKmhel8jAInxcYwb3C8Y8EOymFiQxbCcVH3JKkctEtMWzwZuJzht8UHn3H+Y2TUAzrl7zewB4DxgU+ghzUd6wYMU6OJ3zjm21uzn3S21vFsRDPj3K2rZ39QCQGZKQugIPpOJQ7KYUJClXrx0SScWifQRzS2trNtRFzyKr6hhxZZaPqzcS0tr8P/D/KyU4BH8kEwmFmQxLj+TtCSd6CSfUKCL9GH1jc2s2rbnUJvm3YoatuzaD0CcwfEDM5hYkBXqyWcyamAG8QH142OVls8V6cNSE+OZWpjN1MLsQ9uq6w7wXkXtoYB/efXH/L4sOHs4OSGOcYMzGZefSdGAdIoHpFM0IJ2cdLVrYp0CXaQPyklPYs4JA5hzwgAg2I/fsms/Kyo++cL1qbIt1De2HHpMdloiRXnpFA1MD/4ekE7xwHSO65esL19jhAJdJAqYGUNzUhmak8oXJg4GgiG/rbaB8h11oZ+9lO+o44X3t1NT33ToselJ8YwcEAz54oGf/C7on0pAq0z6igJdJEqZGflZKeRnpfDZ4/MObXfOUb2vkXWVdZRX1VFeuZfyqjoWravimeUVh8YlxscxIjeN4oEZn4T9gHQKc9I0Zz5KKdBFfMbMyE1PIjc9iRkjcw7bV7u/ifIddazfUce60BH9O5t38/y7n5wrGIgzhuWkHhbyxQMyGJKdSr/keLVv+jAFukgMyUxJYMqw/kwZ1v+w7fsbW1hfVXeofXMw7F9Zu+PQlEqAtMQAg7JSGJyVwuDMZAZlpjA4Kzl4PyuFQZnJWrTMQwp0ESElMcC4/ODMmbYam1vZVL2PdTvq2Lp7P9tq97OtZj/baxtYva2WnXWNn3qu7LREBmeFwj4zGPaDslLID20bkJGkaZc9RIEuIkeUGB93aOXJjjQ0tVC5p4GtNfvZXtPA9tr9bA393lxdz9L11ew90HzYYwJxxsCMpHZH+ocf5WenJaq1cwwU6CJyzJITAgzLSWNYTtoRx+xpaGJ7TQPbaoOhv61m/6Hb71XU8LdVDTQ2tx72mMT4OHLTEslOTyQnLYmctESyQ/dz05IO3c5JSyQnPYm0xIA+AFCgi0gP65ecQL/jEhh1XMdH+Qdn5fWgvNQAAAcoSURBVGyr2c+20NH99toGqusa2bXvALv2NVK+o45d+xoPrYPTXmJ8XCjcE8lu8wGQEwr97NCHQG56cHt6kj+/3FWgi4in2s7KmVDQ+dj6xuZQ0Ad/qvc1Ul134FO3N1QFPwDannjVVmIg7lDgZ4fCPzMl4dBPv5QE+iV/cj8zNfi7r/8loEAXkaiRmhhPanY8Q7JTwxq/v7GF6n1tAz941P/J7eD2TdX17GloYs/+Jlo7Wd4qPs7o1yb0P/kQiD/sA+HT+xN65a8CBbqI+FZKYoCCxFQK+of3AdDa6qhrbKa2vona/cGAr+3sp76RzdX7gmMbmg+b4tleIM7olxwM/kunD+Prs0dE6m0eokAXEQmJi7Ngzz854bDLtIXDOUfdgebDAn/Pod+Hb++pde8V6CIiEWBmZCQnkJGcQEH/rsf3BM3uFxHxCQW6iIhPhBXoZjbXzD4ws3Izu6mD/WZmd4b2v2dmkyNfqoiIdKbLQDezAHAXcBYwBrjIzMa0G3YWUBz6uRq4J8J1iohIF8I5Qp8GlDvnNjjnGoEngXntxswDHnZBS4EsMxsU4VpFRKQT4QR6PrClzf2K0LajHYOZXW1mZWZWVlVVdbS1iohIJ8IJ9I5ObWo/ez6cMTjn5jvnSpxzJXl5eR08REREjlU4gV4Bh82xLwC2HcMYERHpQeZcJwsXAGYWD3wInApsBd4GLnbOrWoz5hzgBuBs4CTgTufctC6etwrYdIx15wI7j/Gx0cDP70/vLXr5+f1F03sb5pzrsMXR5ZmizrlmM7sB+BsQAB50zq0ys2tC++8FXiAY5uVAPfDVMJ73mHsuZlbmnCs51sf3dX5+f3pv0cvP788v7y2sU/+dcy8QDO222+5tc9sB10e2NBERORo6U1RExCeiNdDne11AD/Pz+9N7i15+fn++eG9dfikqIiLRIVqP0EVEpB0FuoiIT0RdoHe18mO0MrMhZvZPM1tjZqvM7Fte1xRpZhYws3fM7C9e1xJpZpZlZk+b2drQv8MZXtcUKWb2r6H/Jlea2RNmlux1Td1hZg+a2Q4zW9lmW7aZ/d3M1oV+e3SJiu6JqkAPc+XHaNUMfMc5NxqYDlzvo/d20LeANV4X0UPuAF5yzp0ATMQn79PM8oFvAiXOuXEEz0W50Nuquu13wNx2224CXnHOFQOvhO5HnagKdMJb+TEqOee2O+eWh27vJRgIn1rgLFqZWQFwDvCA17VEmpn1Az4D/C+Ac67ROVfjbVURFQ+khM4aTyXKl/Vwzr0G7Gq3eR7wUOj2Q8C5vVpUhERboIe1qmO0M7NC4ETgTW8riajbgf8DtHpdSA8YAVQBvw21lB4wszSvi4oE59xW4DZgM7AdqHXOvextVT1ioHNuOwQProABHtdzTKIt0MNa1TGamVk68AzwbefcHq/riQQz+xywwzm3zOtaekg8MBm4xzl3IrCPKP2Tvb1QL3keMBwYDKSZ2aXeViVHEm2B7utVHc0sgWCYP+ace9breiJoFvAFM9tIsE12ipk96m1JEVUBVDjnDv5F9TTBgPeD04CPnHNVzrkm4Flgpsc19YTKgxflCf3e4XE9xyTaAv1toNjMhptZIsEvZ57zuKaIMDMj2INd45z7tdf1RJJz7vvOuQLnXCHBf2f/cM755ijPOfcxsMXMRoU2nQqs9rCkSNoMTDez1NB/o6fiky9823kO+JfQ7X8B/uxhLccsrMW5+oojrfzocVmRMgu4DHjfzFaEtv3f0MJo0vfdCDwWOtDYQBgrjkYD59ybZvY0sJzgTKx3iPLT5M3sCeBkINfMKoBbgFuBp8zsSoIfYl/2rsJjp1P/RUR8ItpaLiIicgQKdBERn1Cgi4j4hAJdRMQnFOgiIj6hQBc5BmZ2sh9XjZTopkAXEfEJBbr4mpldamZvmdkKM7svtCZ7nZn9ysyWm9krZpYXGjvJzJaa2Xtm9seDa2KbWZGZLTCzd0OPGRl6+vQ2a6A/FjqTUsQzCnTxLTMbDXwFmOWcmwS0AJcAacBy59xk4FWCZwoCPAx8zzk3AXi/zfbHgLuccxMJrmOyPbT9RODbBNfmH0HwbF8Rz0TVqf8iR+lUYArwdujgOYXgokutwO9DYx4FnjWzTCDLOfdqaPtDwB/MLAPId879EcA51wAQer63nHMVofsrgEJgcc+/LZGOKdDFzwx4yDn3/cM2mt3cblxn61901kY50OZ2C/r/STymlov42SvA+WY2AA5dN3IYwf/uzw+NuRhY7JyrBXab2ezQ9suAV0Nr0leY2bmh50gys9RefRciYdIRhfiWc261mf0QeNnM4oAm4HqCF6AYa2bLgFqCfXYILpt6byiw266YeBlwn5n9NPQcUbkSn/ifVluUmGNmdc65dK/rEIk0tVxERHxCR+giIj6hI3QREZ9QoIuI+IQCXUTEJxToIiI+oUAXEfGJ/w/R0oJG3GjvzwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "h = model.history\n",
    "plt_acc(h)\n",
    "plt_loss(h)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "POST_TRAINING_TIME_START = datetime.now()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "# Submit Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(tta=None):\n",
    "    test_ds = get_test_dataset(ordered=True, tta=tta) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n",
    "\n",
    "    print(f'Computing predictions for TTA {tta}...')\n",
    "    test_images_ds = test_ds.map(lambda iw, filename: [iw])\n",
    "    model_pred = model.predict(test_images_ds)\n",
    "\n",
    "    return model_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing predictions for TTA None...\n",
      "Computing predictions for TTA 0...\n",
      "Computing predictions for TTA 1...\n",
      "Computing predictions for TTA 2...\n"
     ]
    }
   ],
   "source": [
    "model_pred = test(tta=None)\n",
    "model_pred_tta_0 = test(tta=0)\n",
    "model_pred_tta_1 = test(tta=1)\n",
    "model_pred_tta_2 = test(tta=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## No TTA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fd663cf2b6e1d7b02938c6aaae0a32d2.jpg</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cce40cdfd9ef53882386d319d3179696.jpg</td>\n",
       "      <td>04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5989e05d9f73748f938a6da6b9d5fd7c.jpg</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0c95bf0651bd02f7a5a193b9f430ed52.jpg</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>eaa93b6461bfa7868273b759b9e3c125.jpg</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12181</th>\n",
       "      <td>3feb40ea8c99f2b0c3d6a1bcaca7f8cf.jpg</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12182</th>\n",
       "      <td>ceb6db5d91b0e0934f25b9a97a0b0d4e.jpg</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12183</th>\n",
       "      <td>18cbecd37fa0eb34743070354edd01f8.jpg</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12184</th>\n",
       "      <td>f9d85e33f40d1702ec514e5e82cd3f6c.jpg</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12185</th>\n",
       "      <td>f614e1799f92933b16d69a39d0aa9cc5.jpg</td>\n",
       "      <td>05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12186 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   filename category\n",
       "0      fd663cf2b6e1d7b02938c6aaae0a32d2.jpg       20\n",
       "1      cce40cdfd9ef53882386d319d3179696.jpg       04\n",
       "2      5989e05d9f73748f938a6da6b9d5fd7c.jpg       10\n",
       "3      0c95bf0651bd02f7a5a193b9f430ed52.jpg       33\n",
       "4      eaa93b6461bfa7868273b759b9e3c125.jpg       23\n",
       "...                                     ...      ...\n",
       "12181  3feb40ea8c99f2b0c3d6a1bcaca7f8cf.jpg       18\n",
       "12182  ceb6db5d91b0e0934f25b9a97a0b0d4e.jpg       37\n",
       "12183  18cbecd37fa0eb34743070354edd01f8.jpg       34\n",
       "12184  f9d85e33f40d1702ec514e5e82cd3f6c.jpg       40\n",
       "12185  f614e1799f92933b16d69a39d0aa9cc5.jpg       05\n",
       "\n",
       "[12186 rows x 2 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_plain = np.argmax(model_pred, axis=-1)\n",
    "\n",
    "test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n",
    "\n",
    "test_ids_ds = test_ds.map(lambda iw, filename: filename).unbatch()\n",
    "test_ids = next(iter(test_ids_ds.batch(pred_plain.shape[0]))).numpy().astype('U') # all in one batch\n",
    "\n",
    "df_submission = pd.DataFrame({'filename': test_ids, 'category': pred_plain})\n",
    "df_submission = df_submission.drop_duplicates()\n",
    "df_submission['category'] = df_submission['category'].apply(lambda c: str(c).zfill(2))\n",
    "df_submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "filename,category\r\n",
      "fd663cf2b6e1d7b02938c6aaae0a32d2.jpg,20\r\n",
      "cce40cdfd9ef53882386d319d3179696.jpg,04\r\n",
      "5989e05d9f73748f938a6da6b9d5fd7c.jpg,10\r\n",
      "0c95bf0651bd02f7a5a193b9f430ed52.jpg,33\r\n",
      "eaa93b6461bfa7868273b759b9e3c125.jpg,23\r\n",
      "3d7b10a9f6d55bbd14a72d803aec2ce4.jpg,05\r\n",
      "24fb326378e62fa541ee7f0f3d329afa.jpg,28\r\n",
      "a76a602031927190740b582ee87d3bf8.jpg,07\r\n",
      "cf5441e526c9db16787de0277938be76.jpg,39\r\n"
     ]
    }
   ],
   "source": [
    "df_submission.to_csv('submission.csv', index=False)\n",
    "!head submission.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## No TTA + TTA type 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fd663cf2b6e1d7b02938c6aaae0a32d2.jpg</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cce40cdfd9ef53882386d319d3179696.jpg</td>\n",
       "      <td>04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5989e05d9f73748f938a6da6b9d5fd7c.jpg</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0c95bf0651bd02f7a5a193b9f430ed52.jpg</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>eaa93b6461bfa7868273b759b9e3c125.jpg</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12181</th>\n",
       "      <td>3feb40ea8c99f2b0c3d6a1bcaca7f8cf.jpg</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12182</th>\n",
       "      <td>ceb6db5d91b0e0934f25b9a97a0b0d4e.jpg</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12183</th>\n",
       "      <td>18cbecd37fa0eb34743070354edd01f8.jpg</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12184</th>\n",
       "      <td>f9d85e33f40d1702ec514e5e82cd3f6c.jpg</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12185</th>\n",
       "      <td>f614e1799f92933b16d69a39d0aa9cc5.jpg</td>\n",
       "      <td>05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12186 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   filename category\n",
       "0      fd663cf2b6e1d7b02938c6aaae0a32d2.jpg       20\n",
       "1      cce40cdfd9ef53882386d319d3179696.jpg       04\n",
       "2      5989e05d9f73748f938a6da6b9d5fd7c.jpg       10\n",
       "3      0c95bf0651bd02f7a5a193b9f430ed52.jpg       33\n",
       "4      eaa93b6461bfa7868273b759b9e3c125.jpg       23\n",
       "...                                     ...      ...\n",
       "12181  3feb40ea8c99f2b0c3d6a1bcaca7f8cf.jpg       18\n",
       "12182  ceb6db5d91b0e0934f25b9a97a0b0d4e.jpg       37\n",
       "12183  18cbecd37fa0eb34743070354edd01f8.jpg       34\n",
       "12184  f9d85e33f40d1702ec514e5e82cd3f6c.jpg       40\n",
       "12185  f614e1799f92933b16d69a39d0aa9cc5.jpg       05\n",
       "\n",
       "[12186 rows x 2 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tta_0 = np.mean(np.array([model_pred, model_pred_tta_0]), axis=0)\n",
    "pred_tta_0 = np.argmax(pred_tta_0, axis=-1)\n",
    "\n",
    "test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n",
    "\n",
    "test_ids_ds = test_ds.map(lambda iw, filename: filename).unbatch()\n",
    "test_ids = next(iter(test_ids_ds.batch(pred_tta_0.shape[0]))).numpy().astype('U') # all in one batch\n",
    "\n",
    "df_submission_tta_0 = pd.DataFrame({'filename': test_ids, 'category': pred_tta_0})\n",
    "df_submission_tta_0 = df_submission_tta_0.drop_duplicates()\n",
    "df_submission_tta_0['category'] = df_submission_tta_0['category'].apply(lambda c: str(c).zfill(2))\n",
    "df_submission_tta_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "filename,category\r\n",
      "fd663cf2b6e1d7b02938c6aaae0a32d2.jpg,20\r\n",
      "cce40cdfd9ef53882386d319d3179696.jpg,04\r\n",
      "5989e05d9f73748f938a6da6b9d5fd7c.jpg,10\r\n",
      "0c95bf0651bd02f7a5a193b9f430ed52.jpg,33\r\n",
      "eaa93b6461bfa7868273b759b9e3c125.jpg,23\r\n",
      "3d7b10a9f6d55bbd14a72d803aec2ce4.jpg,05\r\n",
      "24fb326378e62fa541ee7f0f3d329afa.jpg,28\r\n",
      "a76a602031927190740b582ee87d3bf8.jpg,07\r\n",
      "cf5441e526c9db16787de0277938be76.jpg,39\r\n"
     ]
    }
   ],
   "source": [
    "df_submission_tta_0.to_csv('submission_tta_0.csv', index=False)\n",
    "!head submission_tta_0.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## No TTA + TTA type 0 + TTA type 1 + TTA type 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fd663cf2b6e1d7b02938c6aaae0a32d2.jpg</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cce40cdfd9ef53882386d319d3179696.jpg</td>\n",
       "      <td>04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5989e05d9f73748f938a6da6b9d5fd7c.jpg</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0c95bf0651bd02f7a5a193b9f430ed52.jpg</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>eaa93b6461bfa7868273b759b9e3c125.jpg</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12181</th>\n",
       "      <td>3feb40ea8c99f2b0c3d6a1bcaca7f8cf.jpg</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12182</th>\n",
       "      <td>ceb6db5d91b0e0934f25b9a97a0b0d4e.jpg</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12183</th>\n",
       "      <td>18cbecd37fa0eb34743070354edd01f8.jpg</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12184</th>\n",
       "      <td>f9d85e33f40d1702ec514e5e82cd3f6c.jpg</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12185</th>\n",
       "      <td>f614e1799f92933b16d69a39d0aa9cc5.jpg</td>\n",
       "      <td>05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12186 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   filename category\n",
       "0      fd663cf2b6e1d7b02938c6aaae0a32d2.jpg       20\n",
       "1      cce40cdfd9ef53882386d319d3179696.jpg       04\n",
       "2      5989e05d9f73748f938a6da6b9d5fd7c.jpg       10\n",
       "3      0c95bf0651bd02f7a5a193b9f430ed52.jpg       33\n",
       "4      eaa93b6461bfa7868273b759b9e3c125.jpg       23\n",
       "...                                     ...      ...\n",
       "12181  3feb40ea8c99f2b0c3d6a1bcaca7f8cf.jpg       18\n",
       "12182  ceb6db5d91b0e0934f25b9a97a0b0d4e.jpg       37\n",
       "12183  18cbecd37fa0eb34743070354edd01f8.jpg       34\n",
       "12184  f9d85e33f40d1702ec514e5e82cd3f6c.jpg       40\n",
       "12185  f614e1799f92933b16d69a39d0aa9cc5.jpg       05\n",
       "\n",
       "[12186 rows x 2 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tta_all = np.mean(np.array([model_pred, model_pred_tta_0, model_pred_tta_1, model_pred_tta_2]), axis=0)\n",
    "pred_tta_all = np.argmax(pred_tta_all, axis=-1)\n",
    "\n",
    "test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n",
    "\n",
    "test_ids_ds = test_ds.map(lambda iw, filename: filename).unbatch()\n",
    "test_ids = next(iter(test_ids_ds.batch(pred_tta_all.shape[0]))).numpy().astype('U') # all in one batch\n",
    "\n",
    "df_submission_tta_all = pd.DataFrame({'filename': test_ids, 'category': pred_tta_all})\n",
    "df_submission_tta_all = df_submission.drop_duplicates()\n",
    "df_submission_tta_all['category'] = df_submission_tta_all['category'].apply(lambda c: str(c).zfill(2))\n",
    "df_submission_tta_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "filename,category\r\n",
      "fd663cf2b6e1d7b02938c6aaae0a32d2.jpg,20\r\n",
      "cce40cdfd9ef53882386d319d3179696.jpg,04\r\n",
      "5989e05d9f73748f938a6da6b9d5fd7c.jpg,10\r\n",
      "0c95bf0651bd02f7a5a193b9f430ed52.jpg,33\r\n",
      "eaa93b6461bfa7868273b759b9e3c125.jpg,23\r\n",
      "3d7b10a9f6d55bbd14a72d803aec2ce4.jpg,05\r\n",
      "24fb326378e62fa541ee7f0f3d329afa.jpg,28\r\n",
      "a76a602031927190740b582ee87d3bf8.jpg,07\r\n",
      "cf5441e526c9db16787de0277938be76.jpg,39\r\n"
     ]
    }
   ],
   "source": [
    "df_submission_tta_all.to_csv('submission_tta_all.csv', index=False)\n",
    "!head submission_tta_all.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "source": [
    "## Save model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('model.h5')\n",
    "model.save_weights('model_weights.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Post training time : 249.875486 seconds\n"
     ]
    }
   ],
   "source": [
    "print(f'Post training time : {(datetime.now() - POST_TRAINING_TIME_START).total_seconds()} seconds')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}