week_1/C1M1_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chest X-Ray Medical Diagnosis with Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FZYK-0rin5x7"
   },
   "source": [
    "<img src=\"xray-header-image.png\" style=\"padding-top: 50px;width: 87%;left: 0px;margin-left: 0px;margin-right: 0px;\">\n",
    "\n",
    "__Welcome to the first assignment of course 1!__ \n",
    "\n",
    "In this assignment! You will explore medical image diagnosis by building a state-of-the-art chest X-ray classifier using Keras. \n",
    "\n",
    "The assignment will walk through some of the steps of building and evaluating this deep learning classifier model. In particular, you will:\n",
    "- Pre-process and prepare a real-world X-ray dataset\n",
    "- Use transfer learning to retrain a DenseNet model for X-ray image classification\n",
    "- Learn a technique to handle class imbalance\n",
    "- Measure diagnostic performance by computing the AUC (Area Under the Curve) for the ROC (Receiver Operating Characteristic) curve\n",
    "- Visualize model activity using GradCAMs\n",
    "\n",
    "In completing this assignment you will learn about the following topics: \n",
    "\n",
    "- Data preparation\n",
    "  - Visualizing data\n",
    "  - Preventing data leakage\n",
    "- Model Development\n",
    "  - Addressing class imbalance\n",
    "  - Leveraging pre-trained models using transfer learning\n",
    "- Evaluation\n",
    "  - AUC and ROC curves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "Use these links to jump to specific sections of this assignment!\n",
    "\n",
    "- [1. Import Packages and Function](#1)\n",
    "- [2. Load the Datasets](#2)\n",
    "    - [2.1 Preventing Data Leakage](#2-1)\n",
    "        - [Exercise 1 - Checking Data Leakage](#Ex-1)\n",
    "    - [2.2 Preparing Images](#2-2)\n",
    "- [3. Model Development](#3)\n",
    "    - [3.1 Addressing Class Imbalance](#3-1)\n",
    "        - [Exercise 2 - Computing Class Frequencies](#Ex-2)\n",
    "        - [Exercise 3 - Weighted Loss](#Ex-3)\n",
    "    - [3.3 DenseNet121](#3-3)\n",
    "- [4. Training [optional]](#4)\n",
    "    - [4.1 Training on the Larger Dataset](#4-1)\n",
    "- [5. Prediction and Evaluation](#5)\n",
    "    - [5.1 ROC Curve and AUROC](#5-1)\n",
    "    - [5.2 Visualizing Learning with GradCAM](#5-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XI8PBrk_2Z4V"
   },
   "source": [
    "<a name='1'></a>\n",
    "## 1. Import Packages and Functions¶\n",
    "\n",
    "We'll make use of the following packages:\n",
    "- `numpy` and `pandas` is what we'll use to manipulate our data\n",
    "- `matplotlib.pyplot` and `seaborn` will be used to produce plots for visualization\n",
    "- `util` will provide the locally defined utility functions that have been provided for this assignment\n",
    "\n",
    "We will also use several modules from the `keras` framework for building deep learning models.\n",
    "\n",
    "Run the next cell to import all the necessary packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Je3yV0Wnn5x8",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.applications.densenet import DenseNet121\n",
    "from keras.layers import Dense, GlobalAveragePooling2D\n",
    "from keras.models import Model\n",
    "from keras import backend as K\n",
    "\n",
    "from keras.models import load_model\n",
    "\n",
    "import util"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6PMDCWQRn5yA"
   },
   "source": [
    "<a name='2'></a>\n",
    "## 2 Load the Datasets\n",
    "\n",
    "For this assignment, we will be using the [ChestX-ray8 dataset](https://arxiv.org/abs/1705.02315) which contains 108,948 frontal-view X-ray images of 32,717 unique patients. \n",
    "- Each image in the data set contains multiple text-mined labels identifying 14 different pathological conditions. \n",
    "- These in turn can be used by physicians to diagnose 8 different diseases. \n",
    "- We will use this data to develop a single model that will provide binary classification predictions for each of the 14 labeled pathologies. \n",
    "- In other words it will predict 'positive' or 'negative' for each of the pathologies.\n",
    " \n",
    "You can download the entire dataset for free [here](https://nihcc.app.box.com/v/ChestXray-NIHCC). \n",
    "- We have provided a ~1000 image subset of the images for you.\n",
    "- These can be accessed in the folder path stored in the `IMAGE_DIR` variable.\n",
    "\n",
    "The dataset includes a CSV file that provides the labels for each X-ray. \n",
    "\n",
    "To make your job a bit easier, we have processed the labels for our small sample and generated three new files to get you started. These three files are:\n",
    "\n",
    "1. `nih/train-small.csv`: 875 images from our dataset to be used for training.\n",
    "1. `nih/valid-small.csv`: 109 images from our dataset to be used for validation.\n",
    "1. `nih/test.csv`: 420 images from our dataset to be used for testing. \n",
    "\n",
    "This dataset has been annotated by consensus among four different radiologists for 5 of our 14 pathologies:\n",
    "- `Consolidation`\n",
    "- `Edema`\n",
    "- `Effusion`\n",
    "- `Cardiomegaly`\n",
    "- `Atelectasis`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sidebar on meaning of 'class'\n",
    "It is worth noting that the word **'class'** is used in multiple ways is these discussions. \n",
    "- We sometimes refer to each of the 14 pathological conditions that are labeled in our dataset as a class. \n",
    "- But for each of those pathologies we are attempting to predict whether a certain condition is present (i.e. positive result) or absent (i.e. negative result). \n",
    "    - These two possible labels of 'positive' or 'negative' (or the numerical equivalent of 1 or 0) are also typically referred to as classes. \n",
    "- Moreover, we also use the term in reference to software code 'classes' such as `ImageDataGenerator`.\n",
    "\n",
    "As long as you are aware of all this though, it should not cause you any confusion as the term 'class' is usually clear from the context in which it is used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read in the data\n",
    "Let's open these files using the [pandas](https://pandas.pydata.org/) library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224
    },
    "colab_type": "code",
    "id": "5JRSHB7i0t_6",
    "outputId": "69830050-af47-4ebc-946d-d411d0cbdf5b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Image</th>\n",
       "      <th>Atelectasis</th>\n",
       "      <th>Cardiomegaly</th>\n",
       "      <th>Consolidation</th>\n",
       "      <th>Edema</th>\n",
       "      <th>Effusion</th>\n",
       "      <th>Emphysema</th>\n",
       "      <th>Fibrosis</th>\n",
       "      <th>Hernia</th>\n",
       "      <th>Infiltration</th>\n",
       "      <th>Mass</th>\n",
       "      <th>Nodule</th>\n",
       "      <th>PatientId</th>\n",
       "      <th>Pleural_Thickening</th>\n",
       "      <th>Pneumonia</th>\n",
       "      <th>Pneumothorax</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00027079_001.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27079</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00004477_001.png</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4477</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00018530_002.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>18530</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00026928_001.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26928</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00016687_000.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>16687</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Image  Atelectasis  Cardiomegaly  Consolidation  Edema  \\\n",
       "0  00027079_001.png            0             0              0      0   \n",
       "1  00004477_001.png            1             0              0      0   \n",
       "2  00018530_002.png            0             0              0      0   \n",
       "3  00026928_001.png            0             0              0      0   \n",
       "4  00016687_000.png            0             0              0      0   \n",
       "\n",
       "   Effusion  Emphysema  Fibrosis  Hernia  Infiltration  Mass  Nodule  \\\n",
       "0         0          0         0       0             0     0       0   \n",
       "1         1          0         0       0             0     0       0   \n",
       "2         0          0         0       0             0     0       0   \n",
       "3         0          0         0       0             1     0       0   \n",
       "4         0          0         0       0             0     0       1   \n",
       "\n",
       "   PatientId  Pleural_Thickening  Pneumonia  Pneumothorax  \n",
       "0      27079                   1          0             0  \n",
       "1       4477                   0          0             0  \n",
       "2      18530                   0          0             0  \n",
       "3      26928                   0          0             0  \n",
       "4      16687                   0          0             0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = pd.read_csv(\"nih/train-small.csv\")\n",
    "valid_df = pd.read_csv(\"nih/valid-small.csv\")\n",
    "\n",
    "test_df = pd.read_csv(\"nih/test.csv\")\n",
    "\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mrDoMlsun5yE"
   },
   "outputs": [],
   "source": [
    "labels = ['Cardiomegaly', \n",
    "          'Emphysema', \n",
    "          'Effusion', \n",
    "          'Hernia', \n",
    "          'Infiltration', \n",
    "          'Mass', \n",
    "          'Nodule', \n",
    "          'Atelectasis',\n",
    "          'Pneumothorax',\n",
    "          'Pleural_Thickening', \n",
    "          'Pneumonia', \n",
    "          'Fibrosis', \n",
    "          'Edema', \n",
    "          'Consolidation']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iKwFwpHLn5yG"
   },
   "source": [
    "<a name='2-1'></a>\n",
    "### 2.1 Preventing Data Leakage\n",
    "It is worth noting that our dataset contains multiple images for each patient. This could be the case, for example, when a patient has taken multiple X-ray images at different times during their hospital visits. In our data splitting, we have ensured that the split is done on the patient level so that there is no data \"leakage\" between the train, validation, and test datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='Ex-1'></a>\n",
    "### Exercise 1 - Checking Data Leakage\n",
    "In the cell below, write a function to check whether there is leakage between two datasets. We'll use this to make sure there are no patients in the test set that are also present in either the train or validation sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li> Make use of python's set.intersection() function. </li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Jz6dwTSrUcKc"
   },
   "outputs": [],
   "source": [
    "# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def check_for_leakage(df1, df2, patient_col):\n",
    "    \"\"\"\n",
    "    Return True if there any patients are in both df1 and df2.\n",
    "\n",
    "    Args:\n",
    "        df1 (dataframe): dataframe describing first dataset\n",
    "        df2 (dataframe): dataframe describing second dataset\n",
    "        patient_col (str): string name of column with patient IDs\n",
    "    \n",
    "    Returns:\n",
    "        leakage (bool): True if there is leakage, otherwise False\n",
    "    \"\"\"\n",
    "\n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "    \n",
    "    df1_patients_unique = set(df1[patient_col].values)\n",
    "    df2_patients_unique = set(df2[patient_col].values)\n",
    "    \n",
    "    patients_in_both_groups = df1_patients_unique.intersection(df2_patients_unique)\n",
    "\n",
    "    # leakage contains true if there is patient overlap, otherwise false.\n",
    "    leakage = len(patients_in_both_groups) > 0 # boolean (true if there is at least 1 patient in both groups)\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return leakage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 544
    },
    "colab_type": "code",
    "id": "Rh2p1krrV1g5",
    "outputId": "9ee44d93-8ef1-4c98-f9fa-65b309b9b889"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test case 1\n",
      "df1\n",
      "   patient_id\n",
      "0           0\n",
      "1           1\n",
      "2           2\n",
      "df2\n",
      "   patient_id\n",
      "0           2\n",
      "1           3\n",
      "2           4\n",
      "leakage output: True\n",
      "-------------------------------------\n",
      "test case 2\n",
      "df1:\n",
      "   patient_id\n",
      "0           0\n",
      "1           1\n",
      "2           2\n",
      "df2:\n",
      "   patient_id\n",
      "0           3\n",
      "1           4\n",
      "2           5\n",
      "leakage output: False\n"
     ]
    }
   ],
   "source": [
    "# test\n",
    "print(\"test case 1\")\n",
    "df1 = pd.DataFrame({'patient_id': [0, 1, 2]})\n",
    "df2 = pd.DataFrame({'patient_id': [2, 3, 4]})\n",
    "print(\"df1\")\n",
    "print(df1)\n",
    "print(\"df2\")\n",
    "print(df2)\n",
    "print(f\"leakage output: {check_for_leakage(df1, df2, 'patient_id')}\")\n",
    "print(\"-------------------------------------\")\n",
    "print(\"test case 2\")\n",
    "df1 = pd.DataFrame({'patient_id': [0, 1, 2]})\n",
    "df2 = pd.DataFrame({'patient_id': [3, 4, 5]})\n",
    "print(\"df1:\")\n",
    "print(df1)\n",
    "print(\"df2:\")\n",
    "print(df2)\n",
    "\n",
    "print(f\"leakage output: {check_for_leakage(df1, df2, 'patient_id')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Expected output\n",
    "\n",
    "```Python\n",
    "test case 1\n",
    "df1\n",
    "   patient_id\n",
    "0           0\n",
    "1           1\n",
    "2           2\n",
    "df2\n",
    "   patient_id\n",
    "0           2\n",
    "1           3\n",
    "2           4\n",
    "leakage output: True\n",
    "-------------------------------------\n",
    "test case 2\n",
    "df1:\n",
    "   patient_id\n",
    "0           0\n",
    "1           1\n",
    "2           2\n",
    "df2:\n",
    "   patient_id\n",
    "0           3\n",
    "1           4\n",
    "2           5\n",
    "leakage output: False\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FCWkiLudW_Il"
   },
   "source": [
    "Run the next cell to check if there are patients in both train and test or in both valid and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "AMF3Wd3yW-RS",
    "outputId": "e417c9ea-c06b-49a7-af35-d802bc1725eb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "leakage between train and test: False\n",
      "leakage between valid and test: False\n"
     ]
    }
   ],
   "source": [
    "print(\"leakage between train and test: {}\".format(check_for_leakage(train_df, test_df, 'PatientId')))\n",
    "print(\"leakage between valid and test: {}\".format(check_for_leakage(valid_df, test_df, 'PatientId')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zRUvYHpYXhlQ"
   },
   "source": [
    "If we get `False` for both, then we're ready to start preparing the datasets for training. Remember to always check for data leakage!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JBWZ5l4ln5yH"
   },
   "source": [
    "<a name='2-2'></a>\n",
    "### 2.2 Preparing Images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SPjuZHPpn5yH"
   },
   "source": [
    "With our dataset splits ready, we can now proceed with setting up our model to consume them. \n",
    "- For this we will use the off-the-shelf [ImageDataGenerator](https://keras.io/preprocessing/image/) class from the Keras framework, which allows us to build a \"generator\" for images specified in a dataframe. \n",
    "- This class also provides support for basic data augmentation such as random horizontal flipping of images.\n",
    "- We also use the generator to transform the values in each batch so that their mean is $0$ and their standard deviation is 1. \n",
    "    - This will facilitate model training by standardizing the input distribution. \n",
    "- The generator also converts our single channel X-ray images (gray-scale) to a three-channel format by repeating the values in the image across all channels.\n",
    "    - We will want this because the pre-trained model that we'll use requires three-channel inputs.\n",
    "\n",
    "Since it is mainly a matter of reading and understanding Keras documentation, we have implemented the generator for you. There are a few things to note: \n",
    "1. We normalize the mean and standard deviation of the data\n",
    "3. We shuffle the input after each epoch.\n",
    "4. We set the image size to be 320px by 320px"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nAgVGOAju8pX"
   },
   "outputs": [],
   "source": [
    "def get_train_generator(df, image_dir, x_col, y_cols, shuffle=True, batch_size=8, seed=1, target_w = 320, target_h = 320):\n",
    "    \"\"\"\n",
    "    Return generator for training set, normalizing using batch\n",
    "    statistics.\n",
    "\n",
    "    Args:\n",
    "      train_df (dataframe): dataframe specifying training data.\n",
    "      image_dir (str): directory where image files are held.\n",
    "      x_col (str): name of column in df that holds filenames.\n",
    "      y_cols (list): list of strings that hold y labels for images.\n",
    "      sample_size (int): size of sample to use for normalization statistics.\n",
    "      batch_size (int): images per batch to be fed into model during training.\n",
    "      seed (int): random seed.\n",
    "      target_w (int): final width of input images.\n",
    "      target_h (int): final height of input images.\n",
    "    \n",
    "    Returns:\n",
    "        train_generator (DataFrameIterator): iterator over training set\n",
    "    \"\"\"        \n",
    "    print(\"getting train generator...\") \n",
    "    # normalize images\n",
    "    image_generator = ImageDataGenerator(\n",
    "        samplewise_center=True,\n",
    "        samplewise_std_normalization= True)\n",
    "    \n",
    "    # flow from directory with specified batch size\n",
    "    # and target image size\n",
    "    generator = image_generator.flow_from_dataframe(\n",
    "            dataframe=df,\n",
    "            directory=image_dir,\n",
    "            x_col=x_col,\n",
    "            y_col=y_cols,\n",
    "            class_mode=\"raw\",\n",
    "            batch_size=batch_size,\n",
    "            shuffle=shuffle,\n",
    "            seed=seed,\n",
    "            target_size=(target_w,target_h))\n",
    "    \n",
    "    return generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vpRXR-3_u7cl"
   },
   "source": [
    "#### Build a separate generator for valid and test sets\n",
    "\n",
    "Now we need to build a new generator for validation and testing data. \n",
    "\n",
    "**Why can't we use the same generator as for the training data?**\n",
    "\n",
    "Look back at the generator we wrote for the training data. \n",
    "- It normalizes each image **per batch**, meaning that it uses batch statistics. \n",
    "- We should not do this with the test and validation data, since in a real life scenario we don't process incoming images a batch at a time (we process one image at a time). \n",
    "- Knowing the average per batch of test data would effectively give our model an advantage.  \n",
    "    - The model should not have any information about the test data.\n",
    "\n",
    "What we need to do is normalize incoming test data using the statistics **computed from the training set**. \n",
    "* We implement this in the function below. \n",
    "* There is one technical note. Ideally, we would want to compute our sample mean and standard deviation using the entire training set. \n",
    "* However, since this is extremely large, that would be very time consuming. \n",
    "* In the interest of time, we'll take a random sample of the dataset and calcualte the sample mean and sample standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UtWEAfAnrhMq"
   },
   "outputs": [],
   "source": [
    "def get_test_and_valid_generator(valid_df, test_df, train_df, image_dir, x_col, y_cols, sample_size=100, batch_size=8, seed=1, target_w = 320, target_h = 320):\n",
    "    \"\"\"\n",
    "    Return generator for validation set and test test set using \n",
    "    normalization statistics from training set.\n",
    "\n",
    "    Args:\n",
    "      valid_df (dataframe): dataframe specifying validation data.\n",
    "      test_df (dataframe): dataframe specifying test data.\n",
    "      train_df (dataframe): dataframe specifying training data.\n",
    "      image_dir (str): directory where image files are held.\n",
    "      x_col (str): name of column in df that holds filenames.\n",
    "      y_cols (list): list of strings that hold y labels for images.\n",
    "      sample_size (int): size of sample to use for normalization statistics.\n",
    "      batch_size (int): images per batch to be fed into model during training.\n",
    "      seed (int): random seed.\n",
    "      target_w (int): final width of input images.\n",
    "      target_h (int): final height of input images.\n",
    "    \n",
    "    Returns:\n",
    "        test_generator (DataFrameIterator) and valid_generator: iterators over test set and validation set respectively\n",
    "    \"\"\"\n",
    "    print(\"getting train and valid generators...\")\n",
    "    # get generator to sample dataset\n",
    "    raw_train_generator = ImageDataGenerator().flow_from_dataframe(\n",
    "        dataframe=train_df, \n",
    "        directory=IMAGE_DIR, \n",
    "        x_col=\"Image\", \n",
    "        y_col=labels, \n",
    "        class_mode=\"raw\", \n",
    "        batch_size=sample_size, \n",
    "        shuffle=True, \n",
    "        target_size=(target_w, target_h))\n",
    "    \n",
    "    # get data sample\n",
    "    batch = raw_train_generator.next()\n",
    "    data_sample = batch[0]\n",
    "\n",
    "    # use sample to fit mean and std for test set generator\n",
    "    image_generator = ImageDataGenerator(\n",
    "        featurewise_center=True,\n",
    "        featurewise_std_normalization= True)\n",
    "    \n",
    "    # fit generator to sample from training data\n",
    "    image_generator.fit(data_sample)\n",
    "\n",
    "    # get test generator\n",
    "    valid_generator = image_generator.flow_from_dataframe(\n",
    "            dataframe=valid_df,\n",
    "            directory=image_dir,\n",
    "            x_col=x_col,\n",
    "            y_col=y_cols,\n",
    "            class_mode=\"raw\",\n",
    "            batch_size=batch_size,\n",
    "            shuffle=False,\n",
    "            seed=seed,\n",
    "            target_size=(target_w,target_h))\n",
    "\n",
    "    test_generator = image_generator.flow_from_dataframe(\n",
    "            dataframe=test_df,\n",
    "            directory=image_dir,\n",
    "            x_col=x_col,\n",
    "            y_col=y_cols,\n",
    "            class_mode=\"raw\",\n",
    "            batch_size=batch_size,\n",
    "            shuffle=False,\n",
    "            seed=seed,\n",
    "            target_size=(target_w,target_h))\n",
    "    return valid_generator, test_generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ga4RZN5On5yL"
   },
   "source": [
    "With our generator function ready, let's make one generator for our training data and one each of our test and  validation datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "rNE3HWRbn5yL",
    "outputId": "4c6b1c25-a33d-42e0-f442-40971ca52a3f",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "getting train generator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/keras_preprocessing/image/dataframe_iterator.py:273: UserWarning: Found 866 invalid image filename(s) in x_col=\"Image\". These filename(s) will be ignored.\n",
      "  .format(n_invalid, x_col)\n",
      "/opt/conda/lib/python3.6/site-packages/keras_preprocessing/image/dataframe_iterator.py:273: UserWarning: Found 866 invalid image filename(s) in x_col=\"Image\". These filename(s) will be ignored.\n",
      "  .format(n_invalid, x_col)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 9 validated image filenames.\n",
      "getting train and valid generators...\n",
      "Found 9 validated image filenames.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/keras_preprocessing/image/dataframe_iterator.py:273: UserWarning: Found 108 invalid image filename(s) in x_col=\"Image\". These filename(s) will be ignored.\n",
      "  .format(n_invalid, x_col)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1 validated image filenames.\n",
      "Found 420 validated image filenames.\n"
     ]
    }
   ],
   "source": [
    "IMAGE_DIR = \"nih/images-small/\"\n",
    "train_generator = get_train_generator(train_df, IMAGE_DIR, \"Image\", labels)\n",
    "valid_generator, test_generator= get_test_and_valid_generator(valid_df, test_df, train_df, IMAGE_DIR, \"Image\", labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pYtXacDgn5yN"
   },
   "source": [
    "Let's peek into what the generator gives our model during training and validation. We can do this by calling the `__get_item__(index)` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 303
    },
    "colab_type": "code",
    "id": "Jh77vpN-n5yO",
    "outputId": "c4e68e79-e8f2-4bb9-8909-072c9dd2f805"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAD8CAYAAAB3lxGOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOy9bWxk2Xke+ByObtGo6qCqDRaFqpaKbZC9IcdgS+kZuGe8A8MaJ5ITCCtFPyx/IJGNAAqwNpAAARQnfzZYIIBhrBMgWKwBGTbiANl4bSSxBdlZr1czi2hgzcRiw+q2h/QO6TQpdZXdVetmxV0XYN0Znv1RfE49561LNr+KLM7UCxBVvHU/zz3v1/N+HOe9x5SmNKUPLs1c9g1MaUpTulyaCoEpTekDTlMhMKUpfcBpKgSmNKUPOE2FwJSm9AGnqRCY0pQ+4DQ2IeCc+2Hn3J845zadcz87rutMaUpTOhu5ceQJOOeeA/D/AvgbAL4D4A8A/Jj3/u1zv9iUpjSlM9G4LIHvA7Dpvf9T730fwK8B+MyYrjWlKU3pDPShMZ33BoBvy//fAXD3sJ2dc9O0xStAxWIR7777Lj70oQ9hZmYGhUIhfHfOgVYl/3/uuefw3HPPhe8zMzP4ru/6LnjvkSQJ+v0+0jRFoVDA/v4+nnvuOezv7yNJEhQKBbz33ntwzqHf7+O9994L2wDg3XffRa/Xg/ce7733Ht577z1kWYZ+vw/nXLRtSoE63vuq3TguIfBMcs59EcAXL+v6UzqaCoUCFhYWUCwWUS6XMTc3h2KxCADhe5ZlKJfLqFarKJfLSJIk7JOmKYrFImq1GtI0RbfbRblcxssvvxxd5+HDh8iyDEmSIEkSpGmKWq2Ga9euhX329/cxMzMTPvf29rCzswMA6HQ6aLVa6Ha7yLIM7XYbnU4HOzs74brdbhfNZhNpmqLf71/QCE4kbedtHJcQeATgo/L/Rw62BfLefxnAl4GpJTBJpBq9WCwiSRIAA2ar1WoAEBi22+0iTVNUKhU0m01Uq1V0u13Mzc0hTdNwHIVBHqVpinK5jCzLkKYpsixDlmXY29sL2wAgyzLUajU8evQoMH2n0wmMz/+73S42NjbCuSkcdnd3xzpuV5nGBQx+CANg8IcwYP4/APDj3vs/PmT/qRCYUKJQqFQqWFpaCpqeWp9CghYDLQL+VSoVJEmCcrmMYrEY7UctDiBYAzzX4uJiEDS8VqfTQZqm6HQ66HQ6aDabyLIsaHtqfxUo29sD5fcBtwBIa977F+3GsQCD3vt3AfwMgN8FsA7g1w8TAFOabEqSBPPz8yiVSkFLq8a2LkC32w3HZVmGXq8XmJnHkHEtw9K94PkoNMj4/OTvlUol/M7z633zHPw+pXwaGybgvf8dAL8zrvNPafxUKpVQKpUiZgQGmAAwNOWBIdNnWYatrS0AwPLyMur1ejgmSRLUajUkSYK33noLzWYzuBLlchmFQgHAwCrodDpBILRarSBs6DoQWKRroExPy6BYLAbhxXufls6P0qUBg1OaXCIz0qRXEJDbsiwLAGGapoHBgQHDdzodbG1tIU3TIDT0fDym3W4Hzd7r9YJGp3ChlUDQkG6EWhVZlgWNz79arRasknK5jMePH8M5d9FDeSVoKgSmFJFzLvLl1ZSmlk2SBMvLy+F/xQJorpPxgYFQ6XQ66PV6wRXY3NyMLAlgICS2t7extLSEVquFWq0WMTpdAjX9CUJS8zMqodZBlmWYn5/H7u4usiybeGvg5Zdfxje+8Y0Lu95UCEzpSKLWpzWwvLwcmItampqcmjdNU6RpikajEZB5muZk0izLQpgPGDDz3NwclpaWIgbf2NiIGL/dbkcRBVoJtBDoPijGQFcAQLj2JAOFv//7vw8AF2a5TAuIphQRmQRAYLTFxUWsrq7ilVdeCb49gMg1oN+tuQStVito+kKhELAFIv21Wi3kIFBwrK6uRtEEYgi0LHhvFDgkAo2dTgdzc3Mol8vBkigWi+j3+9F9T12DIU0tgUsmmt/1eh3FYhHLy8sAECW8PH78+ELuw3uPLMsCmEdNTsGwsbGBRqOBcrkcRQHI2ACCWU6tzDyCXq+Hfr8fCYmdnZ0A9pFhHzx4EAQHcYetra0oKkDLglYAw4wqKCicgIFgq1QqIVqhwOKUxpQncOKb+IDnCVQqFQBAtVoN2ooaV1H5jY2N4NeOawLzXl566aWIqQi8afiOjEaBMDc3h2q1ivX19ei4YrGIpaWlABImSYLXXnsNANBqtUIkYG5uDp/+9KeDm7G8vIxut4v19fWAUaRpGmELDD/SBbC4gIYXmVQ06QKAPDkGayU3T2BqCVwyOeeQZRmq1UFKtybAAAPG4vcsy7CwsBDMY0XuAYRQWqvVCszS6/WOdQ/02T/xiU+MgHUalkvTFEtLSwHZ5++lUgntdhvNZhONRiOE/jQKsLCwEFyHxcXFwLiknZ0dbG5uAgAWFhZQLpcDJlCv10NWoj4ThSatE/X/aUVQcN2+fTsI0ikNaSoELpm89+j1eiHxZWFhIWS5kfmo3TjRa7VaCJMxHMf9i8UiGo0GAASmpaCgdgSGoB59+XK5jNXV1aD95+bmArBWKpWiLL80TUcYiZEB1gooYEjqdDpoNBrodrsh/wAAarVacA14f1/96ldDCJJWAEHDQqEQ3A7NYKSgUldkcXERW1tb4b6KxWJwTaY0oKk7MCFE4IzMQTNX02015EUTmpoWQJRiq6a4pvhq8Q8/0zTFCy+8EMXnAYTvv/u7vxsECoBwTS38ITPynjSb0AqTLMuwtraGpaUl3L9/P+xP4nPfuXMnCldyH9X2FEaaVqzZiBwvPq/u0+12LwRvOSlN3YEJp0KhgEqlcu6Th2AcJ7Um41AQUDszTq/mLxlT3QMyribraIydmvmFF17A7u5uSBKiZZCmKba3t0dy93l/FBK8R+bvU/hQwPA+SqUSms0mSqUSut0u2u02gKEF0+l0IlenVCpF0QgyNlORFfgrFAqo1+vo9XpYWloKeQg8J++n1WoFgdVoNEL0YX19HWmaXkoOQd41Hz16lLPneGhqCZyCbt68CWBQBnueVCgUsLS0BABR6IwFM48ePQpCqNFoYHl5OdL4ZEzVhJpF12q1cPfu3QhUW1lZGam5pw//4MGDYJ5rfQBxAACRoErTFBsbG8GsV8HF/Yh9aMmvAo90O6rValSNqPe2ubmJer2O7e3tcF+aVKShy2q1GkUVGHFhxSFDk2maotlsHgtDOW/K48Hv//7vH0fCUK4lMBUCp6BSqYRqtXruQoCk5bzH2ffVV18NTKYMQA1MIE63AwhmNjBI2S0UCiEZZ2dnB1tbW+EcZH41/XlNxvYpaKh9tayYAqteryNJEty/fz9KGLKC4Pbt21GikpJGCABErhEwxAco6LIsw8rKCvr9PkqlUpSeTGFA4XD//v0wJu9D3GAqBI6iGzduIEmSiLGdc1GyC/PPvffB/Gw2m5c6WVggU6lUwiRfXl7G8vJyQOk56ZWBGY5sNpsBSCPDvPXWWwF4tNV5ZDYm5HS7XSwuLqJcLmNxcRGlUgnb29tRRAMYZgQqGEn3QkN6tIAYbtSQH92anZ0d9Hq9IJg0FAgMMxaBOI9BMx4VJ+h2uyFy0ev1gmCiAGW0hVgC3ZjzsBq89/j5n/95fOlLXxoR/lNM4ALoxo0bke+1vb0dADr65ru7u3jxxRfR7XaxurqKr33tawAQJnKlUkG73b4QX5KJRQACYLa7uxsEQLVaRa1WQ6PRCKZ5o9GIzHAeS+YB4nJbMge/qz/OhB5GJ3gubSDC7WReTfFlNh9/16QgDUXaa3OsVbBQqJH5uQ/vSTEL7pckScAslpaWgvAul8tROjKzHXmfTJqisFlcXIyiLPzOOXNSbOFLX/oSgKGbedH0gRYCjx49CtKXwqDf74c+ddVqNUwUarGXX34ZOzs7IWuNGjBN0wBGHTYB5ufno3MBwDvvvPPM+yTzM3qgSUTUYHfu3IkQfzIBJz219Obm5ogQUACv1WrhwYMH2N3dHdF0KjBpCS0vLwdm5/U1vAggaj1Gc1zrBjQBCRgwLa2AbrcbAYG7u7tBYKRpGphVBYJiCXpvfG4ee/v2bRSLxSAANBSrAo3nt8lb1kWidaSCjM95HCD5F37hF565zzjoAy0EgHxQhoyWJEnUUouTmZpQS1o1u259fX3kvC+//HIoiCGTdLvdoM2VmLzD89F0V/9YUfrV1VUAw8QiflJrLywsYG1tLYogaLIRzXDt8lMqlQAcbvL2+308fPgQDx8+xMc+9rEoBElGpCDQVGQSr6fAJfenJUbkXnELZjTSxeFYKGCpYCnHRSMmzCVot9tYWVlBvV5HqVQKAp/uji1j1lRlYGjtKAZhw52cF7u7u0G5eO9DMZXSJz/5ydyxHjd94IVAHvV6PczPz+PRo0fo9XqoVqtR3LtcLgcTERia0Kurq8FcVKHQ7XaDif7KK6+El08EfXFxEa+//jr6/X7IcVcEHogz4zjxaOrrBGeJLy2EZrMZfHRm6JEJ+AkMJim1NM+3srKCNE2faa20Wq2gBXd2diKQTpmagqJer4dwJM16jg8xjEqlEjQyBRKtEzKdJgXluQL05bmd11KNTqHJpCQAwQrhJy0FnkPfJ0ObvB6vwcYqzWYTwBBkpHLgscDAwrpx40bUXPXJkycoFAoXgjdNhUAOFQoF9Ho9OOdQKBRCtZtaA0mS4BOf+AR++7d/O0LhOaGoFdSH1zCe5tEXi8UQ9ydZDal+rWpZ9ZupBZlZ12w2ce/ePdy7dy/SlDyG1yPQZbGNb33rWwAGvqoFTHW/Xq8XFRBpNiIw7EOwuLgYhAGtEbUQKBjq9XpUoAQMBACfgXkGan3YXoVqpXG7/q5jS4akIFQwk92PiP1o/oNmPNKN0O/6WSgUgjXAMUmSBN/zPd8TAEu+11qthlarhdu3b+Ob3/zmCWbu6WgqBHIoy7KgldnCSpmWn9T+2sACGDI8GYKpstp+mxONTNloNLCxsTFiyioyrqanJtFQq9XrdczNzWF7exs7OztotVq4d+9e9Fz6x/TfLDu60Uaz2YwEQd6+FFLU3js7O8FtYhpzp9PB7du3AQyYWjU5P2/fvh2F9hiBKBQKEUbDsQMQBIAKHyuA1DLh91qtFq6rBUiKSVAhcOxpqVlrTnMOdC7wPXE7LQOeh/eYZVmoc6AAUwxlnDQNER5CBAYVG6D5PTc3F5DySqWC1157LbgB6jJUq9WAqq+vr0chLJKCdDQjVbvzfHnxeTIKNT/j5wzxKXLd6XTQ7/dD1x4F/qj9GPLU6IglaxXoeJXL5agjEV0d9dNrtRoqlQrW19dHIhS0Fng8n4suFTAsI9aogGYy3rt3L8qXoDVzGLGPogp4ujBqvem9ktI0xdbW1kgolS4k3xUFGdc/oDABELka1Wo1FGVxbp1zZuo0RHgSqlar2N3dDQtr0BIAhpqX+1CTMduNPqyCSWQCRZwBhDZaAIKfzmuoK6HfyVR6b5yorL2nAODEZQ4B23ypIKIwIHMf5Yc+fPgwVzsxd4ITms/EZ7fPQsuGlpCCfMBQY3e7XWxvb4+Y1/xNkfq1tbVwD8ViMZQ0H0Wq5ZMkCWXLTCmmFaNuBJ+Jz0gNroAghanODT5zsVgcwVn6/X4UoQKOF1E4D/rAC4G8BA1qAJbikrlp4gIIMfm5uTksLCxgfX09lNhq8g0Qg0CcvLwGASYAgTksgEVNQm3FbDo28uRke+211yKE3sb/e71e0PrlcvnUJbVZluHGjRvo9XrROVSLtdvtUBmpgo2ulYb2gCGCz2fudDrBEqOFQaFmfX5m/QE4dvm0EkPEfPd0ATudDu7duxfG9JOf/CSSJAn1B7zfu3fvIkkSvP766+Gc5XIZ29vbI2N869atkQzIy6YPRHuxozKvrDtEzURfuVAoREg0tQ/r3snIjCBouFB9QyLiGjPneUk8zsbXgYH/Wq1Wo5p8xQ4UMFTGAxBpS+bHn6Wmvt/vR76tEvEFvT4tHA3PKVDX7XaD9aJYgIKi2pWYro/mNqyvr+fmNpyEvPdBC6sFQ/ziN3/zN/HgwYMQ/VGQFwAWFxcjELdSqaBQKETz75133gkRD1qMl00feEyAGYJEplXCcyKm6TD3Xf1FamYywiuvvBImtDbCpKYmMNhqtUJiETW5ItXqerDtGIDIxOYxCjR+/etfj9bgI9MDA993HOYlMQJaVBoHX1lZCYzP519dXY3ak1GLKyLP96CCgOOi2YcMyd2/f38s4BnnRL/fD8Bkr9fDysoKGo0GGo0GFhYWQr2BRoh+4zd+IwDMTDmfAJpiAkraWrvX6wUNaxNFFNghMGh9fVoBbKypSD5N8yRJQgw9SQaxchty0jCkBRap8dSSIAOR0S1Qpokr44o38/wUAGrlAENryFoC2gzFks0x0L4KAIKg5TOPCz1XH5/WD+cCAcFarRZAYt67Civ+zc/PT4ogGKEzWQLOuYcA/hLAewDe9d6/6Jz7bgD/B4CbAB4C+BHv/ZNnnGdslkAeiMVsLb7QcrkcLAJuo6ndaDQiYE5NdQoEaq+trS00Gg289NJLUViP/i8wnODakJMTSBtkLC0tBUuDzM6oAP9vt9sh/pxlg8IfalWmC+/u7o69nRazHmn2rqyshOfgOPEZ2UhVBZQyC5OwKAh1Hx6TpikePHgQXICLpvn5eSRJEmL7TNnWDEgAASPIssGaiLz34/Dc888/j83NzfMW3mNbi/AT3vuPy8l/FsDXvPe3AHzt4P+JIfb0o5m5sLAQACybEAQMJx+FgsZ2CdJVKpXQEahUKkXgF9F/TTLi8cDAj1xeXg4x64WFBSwtLQX/klqdLbhoArN/H9HtnZ2dcG5Onna7faFM4r2H9z60R6NLYnsScFw5pjq+tJrI8BZ1z7JBR+Isu/iVhp1zUecnhik5l4DY4mGJNTBw65gF+Sws4POf/3yoAOV1x0njcAc+A+AHD77/KoD/B8A/HsN1jkUaCmMsmzFcmm7akw8YJpFQAOiy2gr61ev1KLzE/cgEdB+IESwvL4d0Ug3r6R81JZtt2BCaVu/RwtBQILXnRTbHsMzIvH4NXy4vL0dhPlpUGjkBhiFQZS6ODQXg3Nwc3nrrrQt7PhL7QZKJyfjEC3j/igOpa2nDm5qodfPmzVCIRoHO+dNut8fav+KsloAH8H8559acc1882PZh7z3t3z8D8OEzXuPURB9V19ZjHJmpsjS3gRiN10w9AJEloGCVZg8WCoWAM7BrMI/ly11cXIwW3VDrQEuAeT95zTLa7XYwnzmpeM+0Di6beA86ngoA8l6tEKAQ094DFIKab3CZ9PjxY/T7/ajpqs4nYCjk2RtBwWTOs0qlEoTHnTt3AMQZjlQsnCOnsQhKpdIzjzurJfCK9/6Rc24ewO855zb0R++9P8zfPxAaX8z77bzohRdeQKvVQrvdjtwATjrNCiOD8YUBGNmfPqumhup3ZWibFkpBwPRhphJTAPG8BCBJ1ApkmM3NzRBv5zX5G/GFcXc+Ooy0ItJm0dEC4EKhdlz5v4Y5+XxMxtFMwcsmpjKzrTmfjXUGrKdgtEbfL+cJx8jOQwoCDQEvLy8HXOE4mMLLL78cKiWpSA7DF84kBLz3jw4+Hzvn/iOA7wPw5865mve+5ZyrAciFRL33XwbwZWB8wKCucsMMPybjKMoOxDXtNN1I3E9DiUw15XFaPqsJPNQEar4z6UgtDV5HhdHGxkY4jvFq3hcRaTIEr5skSejuc5I2ZedBaZqGa7LaDxj2S+Q46djaVYNILO3V0Kqa2pNgEbC5iIZw9Tm0EEqxDQWkNYRqo1K0NDSV+llAYaVSwauvvhrSz3VeHEanFgLOuRKAGe/9Xx58/ySA/xnAVwB8AcDPHXz+1mmvcVai9mm326G4RkEo9T3V11e/1Ha7JZPZTD8VHvqpIS5gWDq7sbERVZ0BsSCi76tmP31RxQIIPgIDLKJarV5aKMpOUEYmeP9AvD4AMHSVlEE4VqVSKRTwKG5Cf/qyiM1eNY2ZpcRUDHQ7WSm5ubkZuY4q/HiMCjp+X11dDdGgT3/609ja2sL9+/dzhcH8/DxeeeWVaGFWAM8UIGexBD4M4D8e+BsfAvC/e+//T+fcHwD4defc3wOwDeBHznCNM1MeMq0WgNW+ZGBN0+ULWVhYAIAoLqxFJSpkgKF/q1YAr0dBxBRUTm4FJHke7cKjgKIyE5lsEvAAS48fPw6CUJF+YIgVcDz0PSmjK/6hQuUyqN/vR4KW71EtQwpsXYhV36lVPhQYnHfcRpeB84PzUGsPWOz2yiuvABjiXGpl0EXJo1MDg977P/Xef+zg73u99//8YPv/573/Ie/9Le/9X/fe/8Vpr3FWIkPzRdBHtXX7CggCGGFEBXMs09Gi0AnOfTS/XcNKih7zPJax1TLhd32JuoQXhdQkd8e1zKutzbTTErfps6p7pTjKZdPjx4+jhWOtoOIzpGkallXjO1M3kEJC3SdmV2o+i2JPN2/eDK7pwsJCWD5OcRju/yzr8H2dMaiAk7UIaMapWc8JyfAMmVa1lJqxmhWmWAOZMi8MZjPf+MnjVPvT5+c1XnjhhZCpxoQgDT3Nz8+HNNVJEwh8VoZArTanFaXJV0qtVgs7OzvodDpoNpuXskhIHj169CisoUCLhq4kV3UCBsVVLDCz81LdChv6vXPnTgBGVfA3Go2wVDwwjLbUarWoqpELseS1sSO9b4UAE4A0js8XwHg+B0pdA1KWDXv72dJgMr+a5mrKW6uCpNtp4lNQEDDTfdSc4zE2t4ARg4vODTgpUXBZt4DvxCbb8BhgOFYc40kRAKR+vx/6HFAZAAOMhunoTC3XngE6Xzi31P1MkgRvvPFGwI00JE0lo+4rrQDNeu12B+3UGcrMo/etEHj11VejXH0yEuOuan6rSU+mZK8ACgD+ZhmdL0Mluvp/1pzjaryKF6gFouakhpNqtRoePHiAZrOJjY0NbG1theYgk6b184hdghnp4MRW3xXAyPPzM01TrK+vT6Sgc86Fxqtczk0tRvZqZIg5SZKomYjOB9XswEAQMsGM/SOZRMT5oZaTWgS8ZqvVwvLy8qG9It+3QoB92lRrqkmfhwFYDUQBQG2uElr9ekW2+ZsyvloQ2gjDAo8a69fWW8CgY06/3w9WB++LSSqTTsy0W1xcBBADtoqGaywdQDTOkygAgLgc3QqwRqOBTqeDlZWVoIkZsQIQBGJeh2ZqczI9k8x0Dus6D4pvUfloodVh9L4UAp/73OcCmKdhQB08C+ypv39Yx1+aeyq5NdyowoCMT+nObXZxDv1NkfNyuYyvf/3rURxaM+wajQa++tWvTpxpfBgx7k+zlREWji8FAbcrHtLtdkMW5aRTv9+PrLh79+6F8mkFN5MkCT0TFTxW5cH/+f327dsjQmZhYSHMM1q5nU4nWA8891ENS9+XQkBrARSEIbVarahltL4EHqf5ATTZuJ8FDHV7Hj7Afan99LzWB+S9vvHGG8GSIYPQiiAItbS0dKzFSyaFOHEpcHV8rI9sBd9VIu99tDiNRg8U/9A2bxSGjGSRyuVywLc0LK2rJwHD5i6sr9jY2AguR5ZlRyaOvW+FANtCcUBVa9tmknNzcyGHm9ltHHCmhAJx738123RS58XwiQPwHOpO2DZg9+7dQ6/XQ6vVwuLiIt56660IfKRvfJnJMqchxrc5RjY0asuGtVbgqgkBFhRRqwPD5+TCr9VqNVIKGtFZWVkJSoMgdpqmAfWnW0ABuba2NlK5mmWDRVjffPPNZ6YNv+/ai2mZpkpdFQZW+yrDaoiOJpU19/MSj9QSoIDhC1MNqCa9CileX5l7Y2NjpE35VWMIknZu1rHWLDlgqNl0u473VSHiF0wPZ+ZkqVQKVoJ2e2ZolwlHxeKgUzOBRGCooBhO3dzcDM1UbZ8LZjCyMY26mpbeV0LAOYdPfepToejGtqoCYouADMtMQO5jtbYi/0mShPMruMhjFam1ueQqcCiIeD8bGxt46623InCMfjFdgI2NjSgZ5CpRXp6GvhdaaMBoh2JgYN2Nu67+vIgMre8XGK6gRC2vJckLCwuhwpRE7c95uri4iGKxGDpK37t3LygGjUxpYhwrThXbsvS+EgJkfDUrOZFsVZaWcwID5iWWoMCMgjPAsB5BmVmZUqWuxsZJmnjE+9B7tamxPJaaka3ErhqptaRhMDu+3FcFuArcSWnO+SyykSYgTiVmXgcL0TinNFGI42IFf5IkYd0GJQ0PEhSv1+sj+1l6XwmBn/iJnwgIqgoDnWQqFRcWFiKTfGdnJzSNtJOP+xCBBYZpwQroAYOXxFJhC4ABw4VDuZ0dg7jIKS0PCoO5ubkQWuPkuWpEE3hraytqhqpLgAPDMQWG5i/XALh9+zaWlpZCF99JJOdcVEauYWOuatVoNDA3N4eVlZUQvSKgp5qd8yFN05AolmUZNjc3oyIhCli1BKgwaOnartBK7xshwMEFhqWaysRqfnLAlMGzLAsLRiipOacIbV6tADWYbTypJq9qfmA46a2vz/ixZoMpgHjViOaxAq8aXbEWlLpL/NP4+aS6RIwMcL7l5TxQ+GlHKH1OXVFJ29BrDokqOWDYxUoVEcFHGx2z9L4QAjSzqtXqSEcffREEW8rlcliEg63E19bWkGVZ0DB6Dhs+JGn/AEp7y9QKbjGWqyExvnBej0CSugYaL2cU46oRgbFutxutkqTCgECqCjuOAVOHuf7CpIOF77zzDr71rW8F5qTFo0k7tA6otbMsC+NEC5GWIQUH13ZUhZNnbep8/0AIAaKuJKvh+V0ZWcNSfEFsIqn76HfrGuhv1FS3bt0KE1eP0TRiPY5hHfpwFhHX71w05KoAZEpcZkv9fZK1gjRyYt8ft1+FKEmpVMIbb7wREp00KlSr1UIcn+CdrhBFa0cjXACi2D+VjioqRqMABCXDuX4YXXkhUCgUUK/X8eM//uNBqtrEG0Vcs2yw0CN9TVallcvl6HjuawWBhrXU1EvTFNeuXcP+/n40YXkenfjWjbBhQJtzoPdSrVavTJZgHvHZte+emsQUoMRFdHkyjg/xgUkXhmky04wAACAASURBVGRMCnuGCpvNZlhmTZOh+M41i1CtTOYYkDgfW61WqFAEEAGNFLpHuZFXXgj0+/3ga9PU0ow9/iljKkOura1FWlelNYkCwOZ2W4tgf38/SgBRRlbEn2QZnmvg2eYbei9XQQMeRsyGs8+m3ZU4VtaP1rGiizTpLgHvnQBfq9WK3E0bOiYuxcQeugb8Y46BAtf8jRYCMLSabJPTw+hKCwGu88b+7jSj1PzhNk4sXXjEvgjuT7LfLThnQ30aziOpgOEL0fMqPsDz6ApCFD4UXJO6is1xqN/vR8lAWiVoQ60WUNNw2Z07d440byeF7Jzg8vDAUNjxNxsOJtOr1agCQJVbkgwLiVTh8X8Vqnl0pdOGOTnu3r0b0GdOLIZOWCOgqb8UAOzzlyTDtuR0HVTzW03MYzhx1ZzXcmO1BKx/xxdHv67b7YZnoDXCQhBqh6vsBgDxwi9AbLaqltfxIqmLx5V9Jj1Uyh6LpVIpPNOdO3ciAa9RLMU/kiQJz8cMQboPtp07v9N94jGcQ+STw+jKWgJcUwCIzUwLxlG6EjjU39XcooS25rq6EXp+BfBs5MDmB6hLoK6K5iL0er3g2hSLRdy7dy8qO77qAgAYnbRWS6lwsECs9ZGvEqlAIybF7/qsGjLVYzgHuHYlx8CCzFR+ukCNzsnD6EoKAQJCWZbhU5/6FMrlMu7fvz+CymvoqVgshmqsYrGI1157LTI5gUGLaJIOrI1ha8jxxo0boXiDpHkB3K6IrQoENf3SNMUbb7wRioZ2dnaCCXjVqVQqhfHVd6MakGOu5dPUYtz/OCGvSSWGR9lKPcuyoN2tS0RXQLEkJiHREqW2t92GkiTBgwcPsLW1FS1xdxhdSSFAplhYWMDy8nLUWEP9LCD26wkgqnbmOvGUzopYq5bnORWpnpubw97eXvjdvjRrCfAFUljRfKMQY544J8Xu7m6EHl9VUhRf3wcQA4GalMX/tTcDiduuChHtBwZzSDsPAYhcSG63rg4xARUSFmzmuFj39X0pBACEgop6vR5SKq02VhSVRD9JMwZ1P24nqKMWAE14TkprFdi8A2DUDGPclveys7ODN998E0kyXGtPBQgwvmXFL4o0KsPQLL/Pzc1hdXUVwDBSYBcksdmdSZJcmY5K8/PzqFQq0XvV96n+vcUEVHAkyaDOhQKlWCyGcmWOEefgzs5OwLPoVt6+ffvQe7xyQoAJPVmWYXV1dUTCKahkfyMD379/P/ymuQEK5CkKy/OqNmdLKBKPzatWJCnCrd+5sKmavb1eb+KBr+NSpVIJvQRU8wP5uRgUstrzgf8DA8E6yfUDSqwX0EVuFUDmHABiV5EKjQVA/J+WK8eQVgXnKUPMGurmmgiH0ZWLDjCrSmsFdCAV/Scj6pJhb775JoBh9pYChgrG6DmsD8riIGCo6TUcZO8HGBYN8Xur1cLGxgayLIsASj2G1WZXlfie1BzVVXj4qb0Y1TRmGrh1pbTGvt1uT7SlRIYul8toNBohTZ3vtVAohLlmc1RUKBBHUEAaGPafUAtUQ9nMUTiqx+CVswToX9I8Wltbi4A7BfPI5JxYQFywo76omu0aXbAxV1oTPA9/5z2o8FBXgufjp42DW0Scn1c5KsDnsfnvOq4EtvIiMFx+TN0yZZBer3clhKTOyTRNg+VnXYO8yJPOYy52yt8VNNZqzDyr9qhOVM8UAs65X3HOPXbO/ZFs+27n3O855945+Lx+sN055/6Vc27TOXffOXfntAN3GHnvkSQJPvvZzwYmtm2rgWG8X7EB1T6cgAq2WF9MzTf+TzdAQzlAXBVoXyZNMn0pFCKU8EBsjRAdnvTU2KOIY6tAnrpLmg9htWClUgkCXBd1ZY0Fa+U1ojOppJYohZdVLOoKAQigNbepxcjfNdSo4VN1Z7nvUY1aj2MJ/GsAP2y2/SyAr3nvbwH42sH/APA3Adw6+PsigF88xvlPRPQva7VaLoKqA8KQILdr405KWV1XQKWnBRmzLAtugAUJ9TggzjWwoCSAyPzld5stppGMq0y9Xi+YotSENgZO5qCPz0/Nodf3okDjJGcOkkl1bnEM7NzU9w4gKCiLDRHD4hgRdFTlBYzOuzMlC3nv/zMAu57gZwD86sH3XwXwWdn+b/yA3gRQOVie/FyIGXWf/exnUa/XQ0kqB05RdTW1icLTdVANzX3UHdAXRkCl0WiEZhhW2JBs2IoCQ1cXpjDKM295PU4ONoNgevRVJh0rBfq0pp4mq05g62JpJMdGfiaNuDBMu90OFo1aA6VSKeo/CcQuI1dlBoZRBI4Vk8t0zGyPC9ur4DA6LSbwYe8942B/hsEKxQBwA8C3Zb/vHGw7F1IEnhpBB80yv0pH6/Nzf/qd+psKEvXX7Xk1HVmPsRMzD/ABEAFCvJ+5uTncuXMnOq7f719ZbKBer49obsVZLCNrvjyA6B3wOLWkrgImsLu7G7VLy7Jh3woVjIpRKfCpvQg5bhQQFghXhtdVko+iM0cHvPfeOXfiGeqc+yIGLsOJqFwuo1qt4v79+1GttJriutYgf1OUWbULJ50CLPrCyKC8Fq0GNe15XxraUW2nWkyLg/hSeT9M+aRFYF/sVSsecs6FnogUBgBCK20KURXWvV4PlUolCompoASGuR4ct1KpNPHhVK4puLy8HPnynHfMXQGG/QWBwbjpakXA0F3SyAA/OZ5WQI6jduDPaeYffHJ2PgLwUdnvIwfbRsh7/2Xv/Yve+xePe1HvPV599dVQUAMMq7NU+qm2tmCLAnpH+WTAcFA1wce6AbZDroKOh+EF1hXQ0mR7T9yfteRXpdEmgBGLiBNR268B8ZiS6YkH6Lhr4Zb6uVdhTDgfqEAotNRKsvOErgS7SVnQWqsw+bvOOQULj8qwPK0Q+AqALxx8/wKA35Ltf/cgSvASgK64DWemQqGA1dVV7OzshAwsm9CjjKjaWaUlB1AxBD2Pno/bLd6gaK6a9dZt0Benx/C8bHZqBZfeD4/V9NOrQBp6taErjhvdA3XPaA1QIFA4quWk7l25XJ54zITvVRvHJEkSwoU2O1WRfWp8Jklp7YGdq0Ccdm0jBXl0nBDhvwPwDQB/1Tn3Hefc3wPwcwD+hnPuHQB//eB/APgdAH8KYBPALwH4H084VkfST/7kTwaz2TI3Y9H0H4Gh1t3c3Az/K4OpplIm14lH4DEPhNJzqOvBbSqBeX5FcskkTAqiRtAYusU2JhkIs0Sztlqtht6OTBnWDDp97kqlEmk+Pm+1Wh1ZR4JgGzBYtWeSie+fK0lrIphaOeoKWKVDCyJNBwuP2CxAtQBUQfG3w+iZmID3/scO+emHcvb1AH76Wec8LTUajVA2TE3A8JPWpdtViNikkll7HBx1I1RL8X/+TlNN030VyOHLs9rbZhGqaaa1CUyYYdSAAo515ATUKEAm3f8lEcy0Ag2IS2oBREU1OvnpInCMVehy7NgmbpJJXUgbQiaDK4Csc9HiVtzPtqPT8ylw/izFcWXShm/duoUsG3QF1uXBgFgA0Gyu1+vIsiwU5+jAWCS21+uFpJNut4vV1dVoWXNgOPBK1nzj//Zlq4lPptaab2uV0ES+d+8eOp3OlQMESTdv3kStVgtan8lWZGpgCKiqr5sH4GZZhnq9HmUJ6vvk2E76Aq1WW/d6vSiZShmfKxkxgmUxLu5v3Und7ziC4MqkDd+9ezfS4larMEoAIKRW2tg/EJv9/J8D1u12sbi4OIINaEILt1ugCohXy1FwTyeyXZpMEXOGItN00Feg2Wyi1+tNvL97GGmeBSskW60Wms1mVCbNd2cbYtpKUG20qa4Ax3Bubg7z8/MX/JTPJudc1DDGKpRqtRopMZaR8zNJBslBjJoc5uNrwpkKFWuhWroSQuDzn/98MPno++eZQ5xMnDxcS8BmqmVZvMAD0XdKZE3myRvEPEzBanT1+3ntPAkODFHzbnewpDQtHU74q5Yj4JxDqVTC4uJicKWICSwsLISwnoKuuiCH9YU58TXngIxAwcFVmphjMQlUKpUwPz8fsiEt82qsXxWHgoKq2bmKE4AwfzVBSIFBna9HdRUCrog7sLi4GIUC7UNzADVfQJcIUw2u1oO6BiwPtouR6svRP+1ZqPvz3GR8XW1HgR9FuJvNJjY2NsLxdBf6/f6V8f+VyLBAvJrQ3NxcQP4VbFVzVQUr3zOxEf5PhFytKc3pePnll/GNb3zjIh/5SNIcCc4Ndg4mfqU5EMS1NG2aFkGeJaDPbiNh/BtHiPDC6Kd+6qeCAGCKqUo51dwaKWi324HpdIBJNqxXr9eD5WD3VdBFowR5GWvWt1W/lf/rMZubmwG45L3y/6Ne3CQTa+AJvgKjAjsPC7FulbpgwLAMuVarBUHCxhqqBY+TJTduogtHZUFLRd1WkoK/wNAypSXIaAmZXCNMmqxmw+U8xo6zpYkXAhwYNeeVkbQ5g/X1AeQyoPpMwDDP2vYW0EG3Jr2arPaelPn13mlxWA1orYhJmMRnIX1nytiaDKQulTKwWlg69jxeGYjvSwUM/79sHKVYLEYp0DpfiWWUSqWgrFTocU5R+2dZvMIW3VbFUBTIpvBVvOXKCoGPfexjSNM0tEjigJDo22tcvdPpYHt7O6zdbjEATVNN00Ftd6VSCcUeWZZFGYnAcKHRPA2WhwNYQJGk59zY2ECz2US32w1WDoDwPwXHVeieY2lzcxPb29thPMi4CwsLUa2GCksyhgpECxZaPIDbGTbWCf/SSy/hxo1zK1s5MdGc1/BoHoLP6BSL46yiAQaVg/1+P+qFoa6lYgMcLwpUCtIr2214dXU1enC7FHgeaEeGt/uRrHmUJAk2NzfDwNNn1fNZM0v/V0GjPeStZletqOFBkgom1Y6T3DXnKNIwIBldF97gp3XRuJ3jqYxDq0DHN0kGIWHbeEQ17mUQXUy+f+2DwGdiJSEwXEkrTYddha32thYieYKMDgyxMDK+tYzzaGKFwOc+97kQAVCpp1pANYSipLQOVKPaiQMMctjX1tbCOVQCWyQ/L8SipruaqTaSYI9hHzgSNSLvqd1uY3t7G977K2kJsBkIGZFt4KrVKhYWFgCMLq2mQh0YWgEce4uYW2FJ8E0jDnnv7KLo0aNHePvtt6Owr96zgn8kFQ58jkKhEFkJ6qbS7OcYaHsxZmaq5XQYTawQ0Iw6fmqGnw0lcZv67zoZdKLRbOJE5bHAaHYWBZBaFHkNRhUFt5YJGZzmGfdj6iiFGK/BpbcLhcKVtARUWFsfn+6BflrhrETGtlpNTWyOkxUU3W730rGBb3zjG0Ho54U3OQcUMwCGqcSspeAcUr+fTG+Fi3U/nkUTGyLkA3KpMCCOvWsbL066TqeDjY0NNBqNEdBJgbg0HZQbv/7666HdNTBEZUkWILQgF7/zk+CMvV+dvJqfwPtXwQUMwU4KOt7TUX3iJok4vhx/ovl2QqovmwcUqmDVEKsyEv+nS6BmsEYNLjOTcHNzM4SgCfLpfFKXQK1RCjKbfq6uEfe1Ss6C0kfRRFoCWk+ulXOq8VWLKBrdbDZHIgBWczQaDdy7dy8MrrYMA+KFHVSyWmBQmVlfKo/lcVoUAsRotoa9tBe/ag0Cl1eBLEhHi4ZjoeY+tbxaDioArNnP96Pvg+PKrr3WqrCY0GUQ36OGS9VCpJC3wpAWq85F66qqUOA5+Gkt0sNoIoXA0tJSYHJdGYifh4XWtra2Rpa/5r5coFQnnUpTYDRDS6Ur9+e1AETI9WGuif6uk1oFFAUAmUeLotR/vGzT9jhk8wEICKo21wmqrpuOMQWkDW+pNaD70mTWqAFTwNnY5LKIqzHr/FGhR2tPXSb9I1FIcAzJIwoOahdsq6wOo4kTAs8//zzm5uaws7ODjY2N0FVF8QBrOpLhNjc3wzrtWquuGmh5eRkbGxsjzM/BsvUGFg/Q1lb6v1oJAKLJS22lGY/aOYZosPUZ9T4uW5sdh27cuBEBdOyVoF2DFSlncZEKT46bulQqvPldFYGOj44tk8b6/f6lCVE2giGAt7W1FXIDeK/K2HmJbUyTV+uWadY633Tu7uzsBEFpf7M0cUKA5b4coLwySp0w3J8PqXn2ylD8rn3vuQ8wbBVu/VGdZCph84qY9BgbXdBPvnjtJqsditjW3Fozk1BDcFSBDt9HrVaLzHSLfVi3yfYWyBOEVsjynaZpGnXpUeHL62ne/kVHW4rFYsig5D3pArYENhlS1YQhO4+swFM3Sq1W8gKtX17nMJo4IaAMo/XzaZqi3W5HyRDKJBsbG4FJCMBpEtHi4mLoGAwgcgXUNweGOASFkTWnrJ9pJ6buw980WYnfq9VqmJTaM08z5GziyGWHDPPKmp1zoVAGGEzO5eXl8LtaYxZX0UnLsdNx1O/AsJ2bulNaYacYT5IkqFarqFarIWZ/kdEW51wIiVogT3EBKgNtLqOAniohjXbp3NTxUoB0Z2cnrNV5GE2cEAAGYZVut4t2u412ux2QZb5IPjwHstPpYH19PRzPQeaksH66mlzqg/G3NE2j5A2NDliQEcjP6baTnWQRXdbH27JkvT9+Z0nqJGIDtiMSn10tJ9Xu+rwWzLOKQE18JW5nRp2eg9ei66HW30WRtX70OYrFYsABiHtpmrDOLd637UplBQUwzJHRsXwWJjBRIcIf+IEfwMbGBkqlEh4/fhyWXtrd3Q011wCiGH+n08Gbb74ZmcqtVgt3796NNM/W1lZgSE3WydNMQLwOoEYfeIyGrOiS8H9OPt2mL9f6+0mSYH19PZy3VqtFwspq30lwC5ReeOGFENXIsgzLy8uReQvEgvmoCkxgWDVoXTO1KFRAW+tI1ztkV+NOp4NarYatra1w7LibtSwtLUX3QKFGHEQ1v64upFZMu92OOlBxHNidWK0oHW9VSnz+w2iiLIEkSUL5LLWdan7uY1t2WRNP+7wDCBlrfPk25szf8jSOVhuqhtc0VbUOVGrz/vMmr+7DXgY05Qj4PKsOfBKIMW4uzEIGJuVZBsr0Fm9RQWr34//WiuDkJx6hPQwbjUaUFLa8vIxGo4FarTZ2i6rZbIbFU3VuWYtSXVwNk2o6tAo9dRuTZBhNOiyhyrqvlibGEnjxxRdDF2FgyJTM5SczctCAgcZ/8ODByLn4wEmSYHl5OQzOzs7OSEaWNfPtgGniCe+LnzrQem3L+EAMaPKcxWIR7XYbhUIhMH+n0wnAUZIM1kuY5DwBNgDVCchlyA8jdecsIAgM/VpaVNxGwagJRjovNHNQAUeSasM0TbGysoK333773MekUCigWq2i0WhEz6j3rnPUzh9aCwoO2/Uv1XLSc+SFpXd3dwM2kUcTYwnwpdLUVQYiMKhagI03Hj58OHKupaUlJEkSFiEhU5HpVbPrwNJsshNSrQMLPOX5WzbyoN+tb9pqtbC+vj6SSQgMzNqXXnopKnOeNFLTlMzLtmhWEKomV2GsY6PMYUO2Kqi1GEd9YI1G2FAiM0nHZWGxoxKvRXM9TdPQS4DPZQFk/c5xzJs/KgTs+Ko7wVC5jt1hNDFCIE3TAO455yIzUfOrFWFn8Y8l+tRZloXVWNXHVl9JB5XCRa+d58urdtdJBsQvUrEEFRhJMqhcXFtbiywfGxnY3t7G2toavPcT2WGIiLyGden/AnEfBcVROBaaO2AZm2OniV16PN9BnimtTGIjEwxf8jws0DkP4j1b64jveHV1NaoR0T+NBAE4VOtr7ovNaVHFxNA6KxWPAkUnQgjMzs5ic3MzMLwCX3wwvjzVKocxBgdZ04Ft2q5OJF7HSlmdfGolqDmn8X0rWNTiUCHCPgIqGLQ7kr74o8zqy6bd3V202+2ROnclmxKsglTHOg98VQbSKlFeQ7cp8Xw6vipAiBkkSRKVjZ+V6AbQArX4FT81x0T/eJ8krZy1Y6bnUrxKBYl2JzqKJkIIOOfgvYf3Pgr1KONoCLDb7R663vrzzz+PYnGQF2DN9rwBp4CxyT2sQeD1VKuomaqor1oOOgFUcG1vbwf3xGoKFRx6nkklhm55nwRgy+UyKpVKYAY+F60xany+Cx23crmM69evh7Fh0Y0toKFQ1jFSBrJWnkXIKQjotpwHKVPyfolLKMCnrhDzX/ibzicSlQ/nIZ81b+7wWCrIdruN3d3dyY8OPPfccyPbtHAoywbLdamZySpDJedcCLGpb69rCappyglCn9YKCLvsVx6T8nf7UtTSsElHahLrS9cXf9Ex7dOSxSpardaI+alWADEEfVYVBGmaYn9/H9evXw/MoxZgnmZXSwOIIzLU+BQw1v3jezkvl8AWo1ksi/cOIOSIEEdQRaUYi2p+tbYUQ9FnsxiM5lHk0XGWIfsV59xj59wfybZ/5px75Jz7w4O/vyW//RPn3KZz7k+cc586zsBZk46hG+99BPBQim5sbOTGeLkUVZqm2NrairSBJunokmWUkCpwuC+LkSxZU12TjDg5CfRxcmroL88qUT+Z57RNJyaR+E50wlILqZ9vMQENl1rXAAD29/dHgEJej6TmvQqZSqWCVqsVMuVUwHBfvi9tWHpW1+DRo0fY3t7G5uZmaI9vXUJVHtp9STV6mqYh5Kr7c15aEJnzWM8PILTM43kPo+NYAv8awA/nbP+X3vuPH/z9DgA4554H8KMAvvfgmP/NOTeq5g3ZGyQmQOlsGeSoJA+2DadZppNMi314XtvDjsd0u130+/2wSIYVEhYotMVKiksAQzBSTWgAISSox/A5WTw1qZEB658CcYNNtXj4GxCvOnT9+nWkaYpr165hZmYmjMvMzEwkQBQ0JFnsh9egECKArHUhCuZaV/E8iozo0m5vbwfzXedYtVqNxopzxi68Yt0kvX8FTfU8KkxPgiU9Uwh47/8zgL845vk+A+DXvPd73vv/isHCpN93jGuMbGOEgK2VCAwe9XDKmPqi7SRU6amWgE4QDjgFkTK0mp3qn+oLUiZQ39/6wnz5ak1w/4WFBdy4cSPCKy67dkCJz64TXc1fq4UVR+E47O/vR5N/dnYWMzPDaamakO+Xf1ag871pTwMgXq5eTW3eA6s4zxODYdIbF5PhnOHSYvpcdAvUTeX45s1payHZe+Y8UoE2rhDhzzjn7h+4C9cPtt0A8G3Z5zsH205M+hI5SN1uN6zOY8k5h+Xl5UhTU0No2q76bMqM6scpjkBtbDW8CgDVWGRmnl+TgJTxeRy7Eelzl8vlUDfBlGlthz4plNfwBYhXVOJvCohp+OtIM3VmBjMzMxHDK36iORd6Hu3hz/fCnnvqTjDVWe+PgveswrZQKIRUYALZNhpVLpdDEhjninUX7TyztTAARtbdyKOjUs1PKwR+EcAigI8DaAH4hZOewDn3RefcN51z38z7vdfrBdOJC0ywPjyP6vV6YCpg1GzXOKyVtECctKEaQQEnANH5dVJSQqvwovZToEo1l70277Pb7aJer2NhYQFLS0uRqThJQgBAWBrLjh/9bwWyLGiW90z7+/vY29vD/v5+2EY3Qd0vHee8SEOlUonWKtT74LGcDyocyJzF4qBg67TugWI5RP8VzOa9ViqVIOCVyfV5dX7ZuaDuoypNRttIRz3HqdKGvfd/Lif/JQBfPfj3EYCPyq4fOdiWd44vA/jywTlGxBSXaaIPnSSDIpvDgLJarRakrZruJOsvcaJSqzQajTApNMZMwIvpqJqlpiEbixvwU8FDSnui54pX5IWOeH8q+CYpfbhQKGBlZQVJMijQWl1djRhUXTJrwtuJTpqZmcHs7Gz4f39/P7gHNg9BmYHvA0AkCBRXKpVKuZagauEkSUJEYXt7OwB4xIdOQuy9QPBRo0F877QWVItbv15dALUU6F6qNclKREvnbgk452ry798GwMjBVwD8qHNu1jn3PQBuAfgvp7mG+kMchMMEQKVSCYUawFBTkGHp+9tJxHPnJaKolqcfp/iCgowWsFE/mP9rWEhfFM3UxcXF6L5Uu54U6Lko4kSmhgLi1uv6v46dWkTq/1va399HlmV4+vRpEAw6J/i/nptEpmGpcZIkoXTXhhLpLrCwiOdixIA4w0ldBBtippvJKAQFk12PkAwOjLZr4/m0lkAtJO3PeVx6piXgnPt3AH4QwJxz7jsA/icAP+ic+zgAD+AhgL8PAN77P3bO/TqAtwG8C+CnvffvneiODqjf70dtt9gMJI+WlpYi4EknJ1NTKTVtDzbrfwFxI1Ng2LiC++kxKtlVG/E8QCzJFaTU0la2g+J9MKGInYcmkWhyawaeLr1lTXT9n+9mb2/vUBeHVkGSJNjb2wMQx8P1HfDdKoNTu5MxlMl4rsPCcBr2ZEEPE4uOaxEoNqTKo9lsRguT0F3gvdlcAD6PVTQ6f7mvhgWPS8eJDvyY977mvU+89x/x3v+y9/7veO9Xvfe3vff/g/e+Jfv/c+/9ovf+r3rv/9OJ7sYQB6nT6RzJCHNzcwEc0UGkhgbiBpU6Mfm7fmqdNo/lb8rYNotN6xMUnNQMLjshuK+CZbROeMykhgjValKrSBkUiIWyTmZrmR1GFAbXrl0bETD8zgiFEjMWd3d3o4Ydak7rO9PzEsBV4X7S99Dv90MasQqsbrcbAYIqxGi9WAtC57Mmv6mlqPPzJDQxpcSWKpVKkOBcjSePnHOhYYiaTqrNrURWhlOfHojXdtcXB8TMqOE+7q/H8ngAAQVO0+Hah4x4EDwjvqB5BfV6PbgjpVJpoiwCNg8l89El2N3djZbiBhCeE4gbtBSLxSPdAWCICdASULwAQMQQpCRJIoFARdLv98N2azXyXWt6rWI+vBbfRZY9u1WZcy5ULlo8SpvkVCoVtNvtMJ60Ci2QnafUKEw5pkmSBPf1uBbBRKQN55ECIUeBGisrK5E5RcluwRRlVn3pFszjdzUnLbKsCK6+FK3/tj4wX3yz2Rx5OdQ4miuQpoM1FCg8TurnjZsUvNT7I3Prpx7Dsb927RrSNMXThOD5AwAAIABJREFUp0+jSIAlConZ2dkgAOhCULjoe7RVemoVAsM1JVTbKqZDza+5BRratB2TjjtWqnR4X3y3VAhA3GtCLVb+pu4Lx9u6mdoV6zg0sZYA21Ufp+kD/XUl9e108pEss2rM2ZqaFCjW/9LJlhfLVkZQyZ9lWYh+aJgKQGh8opbAuNtgnYayLAs188oUqlkVMLS/PXnyBOVy+ZmWADBgegoAJhfpu7DMpaE1vn+t9qzX66HDswKbaZpGwKBqcb4TdRWPAqvZeFV7LeTNFVqJGm2yysVaKTbqQveCXYv57MelibUErGmWRzdu3MD29jaAYZaUBY2UVDOr1uUxvC6Pt5pM781+V4GgGkTNTu7Ldel5HPEMapz19XU8fvwY77zzThAAk5QpCAxNbGA4KdVasv46if8fVwAAsQugCUTA6KIvGsLVSIUqAQKY/G6TcPKsPGU8HpMkyaHxd7oelql5j8rYFifJs1itFWBxJc1steX4z6KJFALOOXQ6nWdqQA4GUzR1AClV1XRSbWDj/dxuz60WANt8qfmmppoeS0GThxvoslN80Z1OJ2RD5mmXSSsk0so3YNi6q1qtRqapPjswZLDjCoA80twBvkNrGdiojTIu74HPYAU+3QEeq+fR1X6ZoEa6desW5ufncfPmTXz605/G6upqZNorQ+vY5SVb6f1aq0CP4XMy/AkgdKQ+LpA5kUJAGeUoevz4Mbz34YUCsSa3yK8mhKhPpSarbtcX2Gq1RkI5up++LL4YFTr6p+Dkzs5OCAdy+6Rp/Tyyk9H6yHbyk4lUUJ+WKACUefXcylR23HnPClDyeMuMljGty6cWBc/LEGKn08HOzk4QGDYcCSDkL/A+eB1aJpqCzvMDw8xCtQaSZJgHQff4uEDyxGECXEji0aPcRMNcyrIsrHenL0b9MCC/T1uev89jVYDYEJ4FbKz5qeAkgOD/8zwPHjyIio4YIUjTNKDFk9ZanMQszjwBwJi6ugS21v3atWtnuj4xAs0j0PdSLA5XiFbzWfECAFG0QF0bBQrp0ysdJtBYvcocAGAoPPLSfUlWwOi1tXuzzhErjCgAuOzaSWjiLAEmT5yEmJnFTCwOQh4yDcQ+vYYP1ZVQIcFJZCc8Tf48M5QdcTjJGFIDEPoZaKMJagvuU61WTzQGF00qvDg5gWFTWJLFBorF4pHRgOOQRgmePHkSzq+Mx09qVPuedB9rJWiY1oZ9eS2ei63LNb1d54udN6oUdN0BizvYZ1KXIA87oOI7jZU1cUKAWv0k5JwLIRcOHFeC1UnI8+snJwj3sZMDyO/yY5M9rN9GsI9ZbP1+P5iInFy0BMhEvA+mus7Pzwc/0za8uEyXQWsfrE/NLEI73vx+VlfgMFJgVX1qYCiAFDBUs1k1tQpn1fbqf/O5G40GFhcXAw6ilgiQ33CFJfFa7syW8zpX9c/OP9ushJ2D1Do9CU2UO+CcO3VCDLsQcWDq9XpIJOHaBSQOnB10i+TSjOVvfFFMQ6a2tuAU/+f1t7e3Q6eYvAIW/qbXeOmll8I2mqrEJYrFwSKX4+iZfxzieKpGBBCNBye/+uZJkmB2djYK+Z2GaEkoc/J/YLTfQ17oFxgKcs0kVR9d31Gr1RrppKxWgwoZq0zyGJP7E2TWd69mfp6lqs+bZYNogK50fFKaKEugXq+f6jjvfWg+wgGjb6QvQ7sEUZvRZASGbZu0QYY1dy2zkzTCwD92FdZ8epICUbqs2t27d3H37t0RTUYXY2VlBdVqFd1u99KsAdVCFGrcbvENEgXp/v7+mQQAMAwTAqOl3hryVQzG7kurBYiLniwz27wHmyhGt4iFRmRi7YikVgoFOMeL9TG8D4s/KPHZLMaiC66cBkeaGEugVCqdCAy0pC6EStZyuTzChLu7uyFWrxJdfdw8zcBzU0iw1BkYXR8eGGjG7e3tEZNSBUaxOCgk0jbVwHAyKjDF49X0pg/Ol8/OzeMkvW8yQZIkI4uCWovnvMm6Yvre+buOuwpuAm8WN7LnVISf7952o+K82dzcRLE4yCpks1seqxEotgJnsQ9Dz5qEZN1WkgpYWs3c/7TvfWIsgfPMi9fFOtI0DVYB86lZH66alAzHY2zEwAKL/KNJqVpHTT0rAKgdODnoI/K8NqbN4hOWSnMfrrlXKpVCyet59tA/iur1esQMnJgEutSXVsANOLp0+DTESIMyeZ5ZrC4CMIwGqP+upjvdGfaZUMZUV0fPyZ4D7XY7CAQLCNp7yrIsrN3Ac6kFAsSNbK2SIJ1F8E+MJTAOYtNHBV6sVlCzkZOXLoFNQlFLARgKAyvAWDW4s7ODYnFYBs1zsYmJJq/o9dRVUACR21lgQkHAcFin00G1WsXjx4+jjs3nTZpjz+9q/ioiTmtBheHs7GyU8HNamp2dxdOnT8NY2YKfPL+c9zY3N4dmswlgmP2oOI66iXzmLMtCU1gFdZvNJqrVKubn5/H48eNgnq+trYWEIq7JwHMzUUkBZhVetv+FChHr8pz1HU+EJXDe2sESQUCVnPq/anMAI8lDfEH0fym1FZTivvxUl4I56CrN6f9ZJlINYxF4rdrjOS2mUCwWMT8//8y01rOQvX87tmoVUUAp1gKc/Z3v7++HOgILAOalattsQWC4/Dz9eYsF8FmtBWYtRF3yi0RTX10hxvx1XQZ95+o+cX/FmBSDUaVw1nc8EZbAzMzMmWPHzyKmUvJF0UKgtaC+nZr+mleujJjnd5KIA1BjdLvd0F+OiTRqddiKNp04xWIxtE3TF69uB/+obbQUVWsUVKA8qzHGYaWoCoABo0tgE8PQpCE+R7fbxfXr1/H06dMzJwzZjsRAnPCjjAvEvjXvk+9Tx0IFrxYYWWSeSsEysR1bjj8zQ1955ZVormhOAkndAB1ntaYYVTgLFhDG8kxHnxO9996pmg+dmOxgWb9R0W1FgEl88RYRBxBwBts8xKLkFCI60VSz6L3w+LzUU16DYJXmmeukXFlZQalUQqVSCUuG2+cmNnKcaIMKARVm/E0xAPvsvO5ZBQCJisOOn46HJSs4uX6gHqdCjmRdBLU8lPLWLlBBoUqG75T7qHWh79CGBLvdbhRVOCtNhCVwWemx9roq5XWCW8BJtbDiAnQV+KkaxwKApCyLV/LNA7V0X1svz/ukFuTiK8qoxA54vCYlabSi1+sFC6lYHKzKoyauYhpZlmFxcRGtViuqllMmUoCzWCyeOTSYR7Ozs9H1bBRGNbgyoYbZ+Iyq2TXtmGOvkSRtAkNh0Gg00O12sbCwgIcPHwJAcBV4X8SJSHn3yO/W6uB9EYA8L76ZCEtgEogMoC9EzTHVahZUtMithhvJfFY78RxMLyYuAMQWigXfgKE1wd+5TWPRqmk06cl24imXyyE8yWaaxeKgiInx73q9jmq1Glq/8/pbW1uBaejW8D4to2VZdu4un7oE+n7UlVOXQMOrel/a8ltdNAvKqbC1eA23EU+6detWsKyU0Xmc4gXWFVRrjfdjLZg8K+e0NBGWwCSRNfPVV0vTdERrczuAsPadmsKdTmekE7IyKCdktVpFv98PvjStCWVgYNg3j/fC/VmJqJhBkiRBoyljUKNw5WZtvmojHRavUOGo1g2fhdvUEgDOzwWwpCsYKe6hWl3dPBXQmudRr9ejrDvFfJThrRDR5+axHNeVlZXIOtJEI15PLRQgrkfR7FK+c/08L5paAgekZrECMHwZagoCcaKKAodA3GXYLnvOSQMMW0SzVkD70xMvIFkwTs17jVaoFaGWjJ5HhQXdBQoett9i7UWWDTIfm83mCODG8aBvbceH91MsFvH06dPQJ/C8iC3JFTXX8QdG03YV5LP19orYq7XF59bwod2fAt9qdY1YaGm5HqvnUCWkz8M/djw+Txd6agkckPcezjm02+2QlgsMm1ZSYwKjAJKa2uqTAvGS5RQIi4uLIS6t69BRE6lG0wmsPqhObk4WCh2bCAOMVkbStNVJS4FC7fjmm2+iXC6HAhlqeF2xhxGJ3d3diHHUggDO3xLQ+gE+bx7CnoevZNkgQUf7DdICUqDTun9a429xot3dXSwsLIyEdPV4dRlZA6IrZJP0HGoNqJVwnjS1BAxZn101vZp/lPzAaLYZj1Mfn3F9YFiGy2Pog/MauiS6njdJBouU6m8KDtq0Z96XBTwBRD4vBZJuY0RCK+24j+ICNG21eIjPWCwWcf369XNDsS3peZUxyEA6bsq8wMD8T5JBqjOtHBWSHCOOD9/h4uJihPdwPO16C/xU7EbHkQls/J9jZkPGeg7iCOfdZWpqCQgxZ4CNH8mMqtXV56SFoNpe/XAAI4yvk4z545VKJWplzRfPmnPua9eaV6tBtT4nKYEqCgIKDfXl0zSNkpkY22+1WqFSULELvT7Dm7xPBeK4z5MnT3D9+nWcN2l3IdXMKgg1LMd3p/kBJFoEeREAvmcV+GR+bfih5jvxFiv89N1rFataY7QMOJ94Pj7fONrMTYVADll0Xpd8Un8OiDva6p9aAXQfVGAUi8VIsusk4icFgDZNYfKJMpr1hRVPUPfEos/WglHGsCatChYeywIZCgA9jkzH9fjOm1iN+OTJkygywe/U/BwH1fQ6VnadAb4jDSVqXr+GjikAWF9Ai0GtL84VgriKleg8U9RfFQ+ved5goNIz3QHn3Eedc6875952zv2xc+4fHGz/bufc7znn3jn4vH6w3Tnn/pVzbvNg6fI7Y7v7MRLBsTzQTV+IAjoqAPhCqVn1GGVgYMjsms3I/YBhU0+2kFJzU8+nk8juQ+2fF9bi5LSot+6j5r/6w2piU2goaDrOTNAnT56MAGkWwFOfmvfG9SCbzWZ4pgcPHoRGMBrloMbn/1ozYAUol2RTwJSkvrxq/jwLS60Sfa5x0XEsgXcB/CPv/T3n3F8BsOac+z0APwnga977n3PO/SyAnwXwjwH8TQwWIr0F4C4Gy5jfHcfNj4voFrC7D4Bo8lstqhOCAOLq6mqE4FumpL9NAaD55NT8yswEK7XwpFAoRJWKdE84gbQvgjZEUSxAl+/mJ5nImsEUbrwv4gBaPKR5CcAwJfy860NmZmbC+7DhN7V8LKDGkt179+6NvA9gNCGKvynoyrG2FiPdAwK/is8o4/M6NpLCd2LnFJ9xXCtQHWctwpb3/t7B978EsA7gBoDPAPjVg91+FcBnD75/BsC/8QN6E0DFrGJ8pcgWH2mmmBUI3KbMByD6Tc1AJihp2TOAqOGJtifnMQQPrRsBDPMALFKfl2CkZJta8pkpJDTZSV0C3UbhoBVw4xAAQJwyrJpUtbQFdSmgWcdBUiCVodLD3p1laI0SKTirloQytSZraeUosQBgNIuw3++fuO/mSehEmIBz7iaAvwbgLQAfloVI/wzAhw++3wDwbTnsOwfb4pG/ImTNOmVuTh4tFQaA5eXlEYAsz4zXJanV3GfTCQBh5SJaFJzUKhzIiLxXZXKbo6BCI2+pLWo8novH7OzsYHFxMZi8ahGoxcLrkMYlAMhUxWIxpOJq8RJTpykIvv71r0cWio4F3ymPVZdGfwcQMbpiPMAQG2BCl1pVeddQDIHv0ioPFiONcx3KY78h59w1AP8ewD/03v83/c0PMhdOlL3gnPuic+6bzrlvnuS4iybvfTDB7UsCRjPKrNUAjNa1qyZXP5HhOEXlVQupiashJvXN+b8Ce/yzAoG/5y0VxmN4HvY/UFeA1yPx+ez286aZmZmwQrHtvKRlxGQizWHIa+HN/fMAYIuFsFEsr8fn1ufnuXU8dG7kCWu9f30O7j/OVnLHsgSccwkGAuDfeu//w8HmP3fO1bz3rQNzn8sFPQLwUTn8IwfbIvLefxnAlw/OP5kN9g+I0ph9DNWcAxBJdE0ftcAZJx9ThIHBS2ahDkNV1mLQ76o5yuVycB263e5ICFH306Ihno/CS3EBxRRUoLCOgCFNPqeNf6sPPq7IAAA8ffoUrVZrpBaClsH29naod2BiDt9Ho9EIhU8kuj7qStBc39jYiPAQYi86D9SF4BhS49uCLrXaNOckDyAEhhjVuOg40QEH4JcBrHvv/4X89BUAXzj4/gUAvyXb/+5BlOAlAF1xG64k2SQNfdn05ehP6otWcE63c/3ELMuP+yrIqGvbW1OelgMZnAtSWkuA5bIUADZ9lSE+xQMURFRmqVarkeWjeAgQr/B73mnCSlzVWAUux6ZWq2FpaSkSYtTqOgaM3PB4fueYa/IOECdY6Tk5luo2qhBdXFwM2/R+9Z7zMAfSOBrDROd/Vg6yc+4VAF8H8AAA4z3/FANc4NcBNABsA/gR7/1fHAiN/xXADwNIAfyU9/5Ik3/SLQESmY7ZZtp/DhgW9xymbdXH1glK4stn2q4iyRZd1glDDEETjHhuFqnQB1V/l+dSkFHTkvkcNseeGtK6IeVyOUpEGkfpMDBYfo7hPApB+uJJkmBraytoYApC7s/x4HPSktD8AIsH2GQpBUZ1u46pxY5sViYFKy0CFoDR5eD7SNOztw8TWvPev2g3PtMd8N6/AeAwUfRDOft7AD994tu7AqRagAyjkp2/cVKxmkwZXwWAEs/BrEEF+4A4p11BpjRNo0VA2WwiSYbdkAkuMqTI85H5bXKN+qvWrVGNqc+lZq5aDuMi63qQAdXvZ32/AqscL63gU21uk6I43gyX6jYgjizodgsI6tjaY9WK4DZ2u7qIXhvPtAQugq6KJeCcGwHGGDtmaEnBNGV6IO4wAwyrCLlyEhBrXeIEypAku021l15Xi2p4TiAWFnq8NVeBoftjC2IUNOT1arXaWNKElbj0GBdtARBpcpZvq7a2+zH/odfrjVSPAjGuoufJAzxrtRq2trai8aP1oWCjukoUVlyVShODWC04BiFwOktgSkPiCshcPMJqeP0fyG9BZf0+1QJJMgwP0iqwGkmPsX68va5qIh5LV8NqPT6PDS+qv8zrqJmsSLn6zOMkZTS1ivhc1lIi6Kdpuyp0dTz1ffBTrSR9dzr+PJZjrZ2JFAjkd1skpAL+Iq0AYCoETkzFYhHNZhP1ej1iIE46nUSqPWhSKrBXKpVCfbhdIEUZOU/A6P0AcXhJi2gUDFOhwGMtUygDKKahVgPNbiDu4qvZiOMkReKt4FHG1Gew967ZkGzwQeGriTl0K3QMiK2woSvPpwpBXcJWqxVySdjqzVoVFAykYnF8GYKWpqXEJ6Q0TaNqPxuXtua5MhHBKe7DmnZg2KJaX7xqcc3vV/NWQ1oa86erkgdAarGNMoi6MVazWotENbEVXuMmCpw81B6IMyJVYOhzqXCgEOb401VSHMYKe27T96rjbHscWMCQ96XnV5dFczrGTVNL4IREBN66AnypqgkUdeanJcs0NsSXBzrxOGp9tRQoLHhMnjvB71pQpMCfxQC0lsDiCLwGJ/G48QBeTwWYvgcgbgYCxOtD8ng9F7Wwbk+SBM1mM2rndpirpOg/z6nzoNFooNPpYG1tDf1+P4RrLaOrW/KslvDnSVNL4IRkIwLK8Hay6XfNPFMtwDyBUqmEUqkUGE8nuQJWlvFUuwGIkpVU0/FYZQS1Lqw1wO/qUlghYoVgkiRjzQ1QSpLBSkbXrl2LhIEN5en46DjpeObF+hX81Wta4cxr67nVCqOlAAALCwsBk9FOVWqN6T1cFE2FwAmJpqNlHmueKyp/lCZP0zSE7izopu3K8sxxvS6AyEylMMgzlXk/nKjK5Dwf/VaLJ+j++kzlchnXrl0bW26AEisI9/f38fTp07Bd12fQMbd4h60QBIZjpsLVAq0WMFSXQoumrKlPYgSJREtRM06ZH3CRK05PhcAJiUAeycbvuU0no/qAJE4SAlFZNshzV8tC8/vzmJ7X4nVsLQGJprxqLhU4Suom8LxaM6ACSZc/u3DtdbA8uZrdilPwWSw+ACDS/LqfulUcP2ZIsiGsHsdjmSTFcSdGoA1NKFQZkmQNglqPDAuyjP2iaCoETkF8aWQ8a4byuwWH1GxW5J2rCtP81PPQ36TA0FReNfeVAezkV9L75D2p8CITWJdAMQ2rUXnNcS8ll0dapaimNKsdgdH+fhxjjqUKEk0g4vthQRdDtvpHt01dK55PW5mpsCWDs6bgMLziomgKDJ6C+LL5ZwElvlj9JNkKtHK5HPoLKjjH31Q7AaMttKm1NUtPNT0nqO6vQsyaysAw/ZnCjWYyew5w8tv4/LgXlrXEXgV8Fs1bsE08gbjhKDWzWnEaEeAY2e7NSZIE9033V0HA8CPfA0OJrOOgxac5GwScL9qiAqaWwKmIJptOFm7XSUUNCoyaomq200zUdQx1MmizEp5LkXw9D0knqIJZFCoqACxmYbUSmUvDZGo1WGT9oohCR+8LiHssqgBULW3xlTw3Ky+ykiTDJefsewKGpb88VnEjvhPeL5dG12OnQuCKkK4IC8RmMbUAXz41qCWNMgDDfgJqPagfn7dwqVoFms1mwUpFyC3YlRdBUO1OZmI/ASa76DmttXPRxEiBmukUehbsS5IkrAilmAuA0IxEMRwVoKRSqRTqNWj2q3/Pc+UtcKoWm/c+RA4UGLxomgqBU5BFnNWE1Pp2m3ZLJlUTXyeJprkScVZflXRYjFmvqzFr63NaN0F9ZNX+FiW3QotC79q1axfuCijp0vZqYVnLhp8E7uwiLSpwNbVXhaSOh+ZZWMtCx10tKW5L0xTOuciiuCxBOsUETkHaUEIXDeE2W7LLT0XxdWIyP4DaQ019BaiAw8ONnFiqdXRyqUbT9GUFHHk9vTbPDwwm/czMDJ48eRKdcxKIQmhmZgZ7e3sjURIVfmyfRmbP0/gqCEgUHPrspVIp9IewCoDHAHFjl3q9jnfeeSf6ndblZdBUCJySVKuTMW35KE1m7qu+PTUEz6XhPdU6QL6Wz/NXbVGKzaNXraT75IGEem3VVvxtdnYWe3t7l2oBKHEdAmDQxyBJknBvVugqw9tIiuIDfC9A3HZNz6f5HWrJEUTNwxTyBCfv6zLcgakQOAV57wOI55yL2o9xkrCUVRlMXQI1Na22VgDLan4NZ5HUZFUrxWYC6n5qsuo5uK8yt64jqMlAF5EYdFzi/TJiYF0EtWyAmNmBIcPrOKgrpr47gJEVozTvn8LVdinWd6CNXC4yRTiPJkOMX1FyzsF7H3rZ9/v9YG5aBJpk/XuuRKwaQgWFHqsRAMu06qdaX5jb5+fnI5M/L1LAT2biTRKjH4dUeM3MzASrQLW9Cke1uugWaVRFk7p2d3fRbrdD4ZdlXrUG7LvR5qL67liwdJlu1dQSOANpvbf3HpVKJSpGsZV4wGhuvg01afGOzUGw5apWk6uLkecuPH36NDovf5+dnQ1ak0x03qsIXyTpisWzs7NBkNkGoWR2zSnQJcdsJmSxOEgZ73Q6IbuP24vFYetzdaWA2HXUFa0KhUJoKrO7uzv2XoKH0dQSOAcqFApwzo1EDfQPiAtQ1Adk2rD6qBbRtmaqCgcbXcjTKjbFlsU3WZZFJvRVJ00g0m00/zV0a4WxxQusEM6yLPQaYOSGroIKD2CYKqxgogppBXOzLAtW5WXQ1BI4B+r3+3DORRMMGMbZlfF1ogHD1YYsUKWLYACx1aC989W0V6vCCoWnT58G7T6pfv1JKW91I/7P56JVwHHRsVNha/EBi6EACI1kmNmni85YHMG6fRQaFPi8Do+5zDZ/V1/0TxApE5MJgThpJ08AUCPo5NGkEs081PNxotrzq7ZhZR+veZWZ3pKCf0fto5mFjNJYps0DYPV9kXE1O1GBX/6xKtAmiPE9MaSsdBkRAaWpJXAOxEw/m0Oel4VHJrfValoDYHECG1nQc9pmJXaCqSbk/+8Hs5903Geh8KM1AMRjpUzM/SzOot2Gi8XiSDdjFeTMP9DPdrsd3iE1/2VHBoCpJXBupNpZTXFuA4bawGb8qfZhroFqEptJRi2kGo3bVfDMzs4GTajJNB9kun79+gi2YvEBq/01DVz3AUaXhQfilY35++7ublhJSE3/y7YCgKkQOBdKkmQkzGMnhpLGrC2Da+hOyVoAPK9qLk1WUcRfG298UEndBtYZ2EiKNlex7y4PRMzLASDxfBoNmFSaCoFzot3d3dx4PzC0ABSdt+sJULNbzaTRBSBOFtKJpZl/xWIxavN1lcN950W0gDguMzMzkTCmm2bXAFBhYAW2Yi88hqT4EEOLk0rHWYvwo865151zbzvn/tg59w8Otv8z59wj59wfHvz9LTnmnzjnNp1zf+Kc+9Q4H2ASSGP26udbZuWks4tdKONrKEvz3/Mq9tT816pEXveDbvrnkWIjHFtNDNIWZTZCYFvG5YUbFQ9SYdLr9S4tD+BZdBxg8F0A/8h7f88591cArDnnfu/gt3/pvf9fdGfn3PMAfhTA9wKoA/i/nXP/nff+vfO88UkianFm/1ntYpN7NKfcanlbmWaTjrSdV17m2/sJ/R8HUTCqsNQ1CCjAFc9hnJ/vz64Xsb6+DiAu57Y5B0wxngQMwNIzVYX3vuW9v3fw/S8BrAO4ccQhnwHwa977Pe/9fwWwCeD7zuNmJ5W89ygWi9HiG9Z0BGJAqdfrhQmhkQRbfWbr1BVxVnOT+9D3vYxWX1eF9vb2Ai4AjFZhAgj+vEZ8gLgzMDDsOanLmvOdWBdhUulE9qJz7iaAv4bBisQA8DPOufvOuV9xzrHh/A0A35bDvoOjhcaVp/n5eQDxEmP25dviHrYY19wAi0DrcQBym1TwupquCkyjAEeRrTYks2oDEroG2q48yzI0m0202+2odbjmCQBDy1DDuKw2nEQ69kxxzl0D8O8B/EPv/X8D8IsAFgF8HEALwC+c5MLOuS86577pnDty2fKrQFxQVFeOseiybW1lEWiLC6h1oG3FbO4ABcDs7Cyy7OL7/F1lIi5A6na74c9mcZKh2X1YLTMLIqoVRzxHV46eNDrWjHHOJRgIgH/rvf8PAOC9/3Nb6W0pAAAQhklEQVTv/Xve+30Av4Shyf8IwEfl8I8cbIvIe/9l7/2LeaukXjWin6gVadyuCSeKGCvAp2CgMr/W/Su4aP1NYBoGPAmpq6TAnjZYseFBtRjUysuLENAS4LFMMrJh30mh40QHHIBfBrDuvf8Xsr0mu/1tAH908P0rAH7UOTfrnPseALcA/Jfzu+XJI0p/tQTUrCeDq5WgfQpVOCgCrYhznqbhObl9Cgoej9Ra4jtTJrWhQcvgzAlRLEFXPLbumZ5nEuk40YH/HsDfAfDAOfeHB9v+KYAfc859HIAH8BDA3wcA7/0fO+d+HcDbGEQWfvr9HBkg9fv9KFNQGRuIC4DULNTJpq6BzVzTslZbYDQNB56ObEm1CmxGDFR7J0kS+hPqQjFakqyWAhWDCo+LWmn4JOQus3op3IRzl38TZyCWEXOVGqLFxeKwwYiNHuiKNloDoD0DgLjugPuqqToNDZ6O9vb2AnNq4Q8wmqDF7ewuxEpAan8yun4CQ4yBQgO49FqBtTz3e6o+zoFUkFptr5+qOexCE4pKq/bXnAOrldSCmGICxydtOqIuleZg2BAtENcVqFBXYU5SMDFJBg1E1NWbJJoKgXMi9fMVFNLsPwWNbOMQDSGq+c9YtY05Kw6QZYOS4b29vWl+wAnIgq1pmkadormPFc56DDB8FxQYea3IJ6Fa8DCalhKfE3nvwwu3DS0Zf7YpwGoxKCptm5Lwd2DYsYaa5unTpxG4OMUGnk0cI2tNURgzQ9Au3c5PrhjMpiJA3FhEk4Z4bgCX2jjkKJrOmHMkmn+K4tPf18lmw095mWt5yUYWIKSvyvUApgLgZGTzOBR4tVma6vOzv6B14WxrN+0lMMk0nTXnTLpqkA0Bap05q8pUAACx36kmproMQLwAyqRPskmi/f39qJLQRm9sQpYNF6qZr79pslde4tck1gyQpkLgjHTz5s3wXSeE7fij3YTUHdAJx3PkTTD+BoyuPVAsFqdYwDGJTUgVTLUJWDZEW6vVwlhbZqamV+xG04i1Tdyk0lQInJE05q+TR/PQyewqCKjlVXtYK0BdB/UztXstMES5p4Lg2bS3txewE5ufoclCWjDEfZIkiRYcIfH9lcvlqJEpz6X7TiJNhcAZ6fHjx+G7am+NOwMxXsCMM6vRFRzMww7oUmh6MoXHZS8KelVIm67yfeWFBjV1m9u1oEirN/U9amaoWoJXuqnIlE5GytC6si8wWibM/cj81OjqJlhfk5Pw+vXryLIsrLIzpeMTuxSrO5WHzfA3TS3OskGdSKFQiBq/AIjcAZs0lFc8xCjDZdNUCJwjcXVh1e6abGJ7DKg1oAtYqn9qhQq37e3thdyAKZ2cKDwZWbGpwwrK6pLtap3ZFaYsyKi/5YUHbQTosmgqBM6RCBrlFY+oxuCahdoXUEOJNlKgYUONPADTdOGzkLZft+Cg4jdq3ivSr+CfhnABjCQcTTJNhcA5ktajk7ltLFpNR01G0a7BWmPA8zAeTeEybR56dlIMhWOu6P5h0Ri2F6OZr+sRaFZonvunNCnJQ1MhcI5UrVYBxMtb5f2uwKDtIdBoNABgRIuo6zA7Ozt1A86ZyLBE91VgK9irVli1Wg0hQE02ssDjpNM0bfgcif3mrMnO7+12G9VqNfpN04IpFLQrsSamAAMzk2Gu99tqQpdFXLCU4VfL/EqapMV6AAoQteb4vd1uT2xHIdJ0Bp0jaYYYJ4KamGmaot/vRw0otMrQah/tSkwTlYDWlPnPh1hSrO/OgrI2lXtubi7kCxAH0DwPUrfbnZgIwFE0nUnnSLomITAEh3SdQW1DRm3BJBMem5dRSJyBlYJTK+DstL+/H6ow1erSaAAwjMzwj/kfeZmbJPaSmPSUYeD/b+/8QuS66jj++W2YSZ1scVJ3U3bTriaxkPQh2qWUCKUPFbXNSxQU8lQfAgX/gD74EClIBV8qaEEQS6WFWsS0/sMgCFYt+GRiqsk2bYnd1mY1GZtdnV2Nwc3qHh/u+d353bMzyaZz5967O+cLy9y59+w9vzn33N/5nd/fyAQGAmtesunBQ79z++KHomfoYmy3DTFYKB+MjIykW6uVlaTUuHXyUquNfXZhDQILaypUZW+Vi44o4kzKEeFe33oJ6gut+8gwMEWZhrUe6Hl734j8oG7WYfFX9dsItfyh2zfA5ORk+n92exf6hFQZkQnkDFu+CpKJo2LhlSud+vTdHIHs/9hPu8JECSA/aD3Cbl5/Vhdjn5cyhfn5+TRDlDXxTk1NZUKPrVm3qogzKmeolhk64qQNFNJEk3ZiWYcURbcAlYh8oWbW0KFLmbKFVdxqPgGbSUjv02q10m2DnqticlGLyARygohkJpHu4S0DsMFDYfZgq2RSbbONPdgoNueNhjBLsJXiFhYWMtWfbByHtQBZRqD31PZVZwAQmUBuCMV7Xd110oS17MKJA2Ti1iGZeFYhGF2E80Xo3AMd3YxdyaHj7dloNNIScjay0EpzNgak6kpBiEygb4jIGpuxThar/W80GmnVYr1uV/3QvGRXpNHR0cgABoDR0dHUfGs/wy2BfVZ2G6cM2hYu1fuETKTKiEygT+hDDsuM2QAiXT1sG1jrVmpNiN2i1iLyh77MKytJQlir3dfnoAq+0MVbmb/1B9HnqfUmms1m5aWByAT6gNX01+v19KW2YqbVCQCZSaYTBcgktgCiZ2CBCJ+N9QfQl9uK+Lriz8/PZzxCw/ySvZSMVcN6ahHeJCInReSMiLwiIl/153eJyAkRmRWR50Sk7s9v9d9n/fX3DfYnlAu7kochwNb7z7azk86K/jZisN1uF/grhhdhnICV7GyiERsVak223ZS1VipYXFysfCLY9Swzy8D9zrkPkJQhf0BEDgCPAY87594PtIEjvv0RoO3PP+7bbUpo/cH9+/dn/McVVptsdQXW2cQyAOhoq6NJsBioxGW1/t1edusMpFaB8fHxDJPXORAGh1Ud12UCLoHWuKr5PwfcD/zIn38G+Lg/PuS/469/WKq+KeoDOiE0sagNNlERcnx8PLPK2/gCZQC657TVcCKKQ6iHscw8DCCy53VLqNB7KHPYtm1b5fU669pwisgWX5H4EvAC8Aaw6Jz7r2/yV2CnP94J/AXAX18C3pMn0VWBiNBsNoGOaA9rk4KGeedUPLSBRpYpaNvLly/HDMIDQjiuO3bsSJWEdvW30oCNC9BakioNhs/clqLbFEzAOfc/59wHgduAe4C9/XYsIg+LyCkROdXvvcqCFe9tsEm3vaI1+dnUU9pWfQTCCkZV309uVGiyUWUGNvEoZAu6WGctfdaaPVhX+tCpS+8VFp6tIm5I9eycWwReBD4ENEVEk5LcBlzwxxeA2wH89XcDf+9yryedc3d3K5W80WDNelNTU5lVA8gkq1AGYNNZ2QKWej+b5ipicLDWGOgoBHV7Z8PArbbf+nioF6hVHurzn5yc3PiSgIiMi0jTH78L+AjwGgkz+KRv9mngZ/74uP+Ov/4bV5VkajlDtwLWjdTu9/Wa9Q7UNsoUQkWh7j9VqRSdhAYHVQrasmTKFLZv357x8lxaWspYDLr5fGhbK81duXKl0jUHYH3pxSaAZ0RkCwnTeN4593MReRU4JiJfA/4IPOXbPwU8KyKzwD+AwwOguzJQz8DQPgwdpZ+uFL3MUaFpUSdaZADFIBznWq2W0Rl0cwNXxrCwsJAJCrNbBpuqrF6vVza5yHWZgHNuBriry/k3SfQD4fn/AJ/KhboNAHURtbZ/Kzpa2JXfmo+sPdrqGCKKR62WJG1pt9vp87Np4iAb6t1oNLh69Wq6bWi1WmusCfosq8oIojtanwhty9bZRLML25U/zEoTprHSNtFTsHhoyrbV1dU1btvq0WmLyVjG0Gw20wIy1kMwlA6riDjT+oDV8FulkXU20ZyCdsKEDMFOuKprkjczlPFa3YCNH2i1WszNzaXtbfanxcXFzLbBhidDtU2FMeV4n7A24lDJZ5VE1nxkQ09tpBokkW3RN6B8qILXVn6CteXk7bO1c0HrF1imrqnlqobIBPpArZaUIAfWaI2tp1ho6rMvv97HIm4FyodNDWbNuI1GI01IGor5tVotDSu2CkK9V71eR0QqU3lIEWdbH7D7Rf1urQF276/x5TYHoS1jbYtaRJQPjRy0zysMLtI/q/sJcw6EgWNVYwAQJYF3DBFJNfw2w4z1HISsPdmKkbZNKBlElIvV1dVMDIf1/VCnrrGxsZRxnz9/nmazmXp4Wq9B6026srISJYHNhMnJyfTl3bNnD2NjY2mJMehok3UCQCd9mDUD2pUibgOqgZGRkTXK2vC5WqlNA8TUecz+n/UarapyUKrAlUSkfCLWiWazyb59+5iYmGB6epparZbmnt+7d2/GdVhf6uXl5ej4s0nQbrczK73NOqTBYRcvXmRubi5NMqJ6BPUpmZ2dLUsaeKmbm35VmMA88G9g4XptC8AY5dNRBRog0hFio9PxXufceHiyEkwAQEROVSGYqAp0VIGGSMfw0BE3oRERQ47IBCIihhxVYgJPlk2ARxXoqAINEOkIsSnpqIxOICIiohxUSRKIiIgoAaUzARF5QETO+ToFRwvu+y0ReVlETmuuQxG5RUReEJHX/ef2AfT7tIhcEpGz5lzXfiXBt/z4zIjI9IDpeFRELvgxOS0iB821L3s6zonIx3Kk43YReVFEXvW1Lb7gzxc6Jtego9AxKbzWh3OutD9gC0nm4t1AHTgD3Flg/28BY8G5rwNH/fFR4LEB9HsfMA2cvV6/wEHgF4AAB4ATA6bjUeBLXdre6Z/PVmCXf25bcqJjApj2xzcDf/L9FTom16Cj0DHxv2vUH9eAE/53Pg8c9uefAD7jjz8LPOGPDwPP3Uh/ZUsC9wCzzrk3nXNXgWMkdQvKhK2bYOsp5Abn3G9JUq+tp99DwPdcgt+RJHidGCAdvXAIOOacW3bO/RmYpUtmqXdIR8s59wd//C+SHJY7KXhMrkFHLwxkTPzvKqzWR9lMIK1R4GHrFxQBB/xSRF4SkYf9uVudcy1//Dfg1oJo6dVvGWP0eS9mP222Q4XQ4UXZu0hWv9LGJKADCh4TKbDWR9lMoGzc65ybBh4EPici99mLLpGvCjeflNWvx3eAPSQl51rAN4rqWERGgR8DX3TO/dNeK3JMutBR+Ji4AdT66IWymUBao8DD1i8YOJxzF/znJeCnJIP9toqW/vNSQeT06rfQMXLOve0n4CrwXTri7UDpEJEayYv3fefcT/zpwsekGx1ljYnvO7daH71QNhP4PXCH13rWSZQax4voWES2icjNegx8FDhLtm6CracwaPTq9zjwkNeIHwCWjIicO4K99SdIxkTpOOw10buAO4CTOfUpJKnqX3POfdNcKnRMetFR9JhI0bU+8tCq9qkJPUiihX0DeKTAfneTaHbPAK9o3yR7qV8DrwO/Am4ZQN8/IBErV0j2dkd69UuiKf62H5+XgbsHTMezvp8ZP7kmTPtHPB3ngAdzpONeElF/Bjjt/w4WPSbXoKPQMQH2k9TymCFhOF8xc/YkiQLyh8BWf/4m/33WX999I/1Fj8GIiCFH2duBiIiIkhGZQETEkCMygYiIIUdkAhERQ47IBCIihhyRCUREDDkiE4iIGHJEJhARMeT4P1arbVWZtvyoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x, y = train_generator.__getitem__(0)\n",
    "plt.imshow(x[0]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9WBMpRxcDMgp"
   },
   "source": [
    "<a name='3'></a>\n",
    "## 3 Model Development\n",
    "\n",
    "Now we'll move on to model training and development. We have a few practical challenges to deal with before actually training a neural network, though. The first is class imbalance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qHBSgvxfn5yR"
   },
   "source": [
    "<a name='3-1'></a>\n",
    "### 3.1 Addressing Class Imbalance\n",
    "One of the challenges with working with medical diagnostic datasets is the large class imbalance present in such datasets. Let's plot the frequency of each of the labels in our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 365
    },
    "colab_type": "code",
    "id": "-OvyPe5en5yR",
    "outputId": "077747ad-7ab8-463d-8335-6b243cb29e63"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAFcCAYAAADGVXW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3dd5wkVbn/8c+XJS8uIKwBFndJBkCCEiRcVCQpAooi0QuKolf5iVkMVwS8ksyICiqIBAliWHElKKgIAruLBAGRZSWr5BwXnt8f5zRb29sz09NVNaH6+369+jXdVdVPn+mZevr0qRMUEZiZWXMtMtoFMDOzejnRm5k1nBO9mVnDOdGbmTWcE72ZWcM50ZuZNZwTvdkwSXqxpD9JekTS10boNadJCkmLVhTvDZLuqCKWjX1O9PY8SbdIekLSo4XbSqNdrjFof+BeYFJEfKJ9p6QfS3q67X28eqQLKWljSTMkPSjpfklXSHrPSJfDRp8TvbXbMSKWKdzuaj+gqlrlODYVuD4GH214VNv7uN5IFQ5A0qbAhcAfgTWAFYD/Ad48kuWwscGJ3oZUaDbYT9JtpASCpNdJujTXGK+W9IbCc1aV9MfcvHGBpO9IOiXvW6jZIH+b2DrfX0TSQZJulnSfpDMlvbCtLPtIuk3SvZI+X4gzQdLn8nMfkTRb0iqSjm1vZpE0XdLHBvidN5M0U9JD+edmefuPgX2AT+ea+tY9vJ9nSfp3jv0nSWsX9i0l6WuSbs37/yxpqcLT9+r0e3dwNHBSRBwZEfdGMjsi3jVAmQ4qvGfXS3p7Yd8a+W/5UH7dM/J2SfqGpLslPSzpWknrDPf9sBEQEb75RkQA3AJs3WH7NCCAnwATgaWAlYH7gLeQKgzb5MeT83P+AnwdWALYEngEOCXvewNwx0CvDRwIXAZMyc8/DvhpW1l+kMuxHvAU8Kq8/1PAtcArAOX9KwAbA3cBi+TjVgQeB17c4fd9IfAA8G5gUWCP/HiFvP/HwJcHeR+H2v9e4AX5d/smcFVh37HAH/L7OwHYLB836O/dFn9p4FngjYOUYYG/AbArsFL+W+4GPAa8NO/7KfD5vG9JYIu8fTtgNrBcfq9f1XqOb2PrNuoF8G3s3HKyfRR4MN9+mbe3ksxqhWM/A5zc9vzzSLXdlwHzgImFfacNI9HfALypsO+lwDM56bbKMqWw/wpg93z/RmDnAX6/G4Bt8v0DgBkDHPdu4Iq2bX8B9s33u0n0TxbexwdJtetOxy6Xf59lcyJ9Alivw3GD/t5tx66cj33lIGVc6G/Qtv+q1vtI+oA/vvjaeftWwD+A15E/QH0bmzc33Vi7t0XEcvn2trZ9txfuTwV2zc02D0p6ENiClJRXAh6IiMcKx986jDJMBX5RiHsDqYb64sIx/y7cfxxYJt9fBbh5gLgnAXvn+3sDJw9w3EodynsrKYF266uF93G5iNgHnm9aOiI3kzxM+oCD9A1jRVKNeaDyw8C/d9EDwHOkv0VXJP23pKsK7/k6uTwAnybV2K+QdJ2k9wJExIXAd0jfQu6WdLykSd2+po0cJ3objuLFx9tJNfpiMpsYEUcA/wKWlzSxcPzLCvcfIzUvACn5AZPbYr+5LfaSEXFnF2W8HVh9gH2nADtLWo/UzPDLAY67i/RhU/QyoJvXH8qewM7A1qRa/LS8XaSePE8ycPm7EhGPk76BvKOb4yVNJTUJHUBqnloO+FsuExHx74h4f0SsBHwA+K6kNfK+b0fEa4G1gJeTms5sjHGit16dAuwoabtcS10yX2SdEhG3ArOAQyQtLmkLYMfCc/8BLClpB0mLAV8gtUO3fB/4v5yAkDRZ0s5dluuHwGGS1swXC9eVtAJARNwBzCTV5M+OiCcGiDEDeLmkPSUtKmk3UiI7p8syDOYFpLb1+0gfdl9p7YiI54ATgK9LWim/r5tKWqJzqEF9GthX0qdav7+k9SSd3uHYiaQP8Xvyce8h1ejJj3eVNCU/fCAf+5ykjSRtkv+Gj5E+pJ7roaxWMyd660lE3E6qmX6OlCBuJ9XmWv9TewKbAPcDB5PaeVvPfQj4ECkp30lKEsVeON8CpgPnS3qEdGF2ky6L9nXgTOB84GHgR6SLly0nAa9m4GYbIuI+4K3AJ0gJ+dPAWyPi3i7LAPN75bRuref+hNQMdCdwPel3K/ok6WLyTNJ7dyQ9nKcRcSmpDX0rYK6k+0nt7DM6HHs98DXSt4D/kN6fSwqHbARcLulR0t/lwIiYC0wifRN4IP9O95F6+9gYowgvPGL1k/QlYI2I2HuoY2sux5akbyNTw//81idco7e+kZsYDgR+6CRv/cSJ3vqCpFeRujm+lNR33axvuOnGzKzhXKM3M2u4MTc51YorrhjTpk0b7WKYmY0rs2fPvjciJnfaN+YS/bRp05g1a9ZoF8PMbFyRNODoczfdmJk1nBO9mVnDOdGbmTWcE72ZWcM50ZuZNZwTvZlZwznRm5k1nBO9mVnDOdGbmTXcmBsZ22+mHfSb0jFuOWKHCkrSnSrKCyNbZrN+5xq9mVnDOdGbmTWcE72ZWcM50ZuZNZwTvZlZwznRm5k1nBO9mVnDOdGbmTWcE72ZWcM50ZuZNZwTvZlZwznRm5k1nBO9mVnDOdGbmTWcE72ZWcM50ZuZNVxXiV7S9pJulDRH0kEd9n9c0vWSrpH0e0lTC/v2kXRTvu1TZeHNzGxoQyZ6SROAY4E3A2sBe0haq+2wvwIbRsS6wM+Ao/JzXwgcDGwCbAwcLGn56opvZmZD6aZGvzEwJyLmRsTTwOnAzsUDIuKiiHg8P7wMmJLvbwdcEBH3R8QDwAXA9tUU3czMutFNol8ZuL3w+I68bSD7Ab8dznMl7S9plqRZ99xzTxdFMjOzblV6MVbS3sCGwNHDeV5EHB8RG0bEhpMnT66ySGZmfa+bRH8nsErh8ZS8bQGStgY+D+wUEU8N57lmZlafbhL9TGBNSatKWhzYHZhePEDSBsBxpCR/d2HXecC2kpbPF2G3zdvMzGyELDrUARExT9IBpAQ9ATghIq6TdCgwKyKmk5pqlgHOkgRwW0TsFBH3SzqM9GEBcGhE3F/Lb2JmZh0NmegBImIGMKNt2xcL97ce5LknACf0WkAzMyvHI2PNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOG6SvSStpd0o6Q5kg7qsH9LSVdKmifpnW37npV0Vb5Nr6rgZmbWnUWHOkDSBOBYYBvgDmCmpOkRcX3hsNuAfYFPdgjxRESsX0FZzcysB0MmemBjYE5EzAWQdDqwM/B8oo+IW/K+52ooo5mZldBN083KwO2Fx3fkbd1aUtIsSZdJelunAyTtn4+Zdc899wwjtJmZDWUkLsZOjYgNgT2Bb0pavf2AiDg+IjaMiA0nT548AkUyM+sf3ST6O4FVCo+n5G1diYg788+5wB+ADYZRPjMzK6mbRD8TWFPSqpIWB3YHuuo9I2l5SUvk+ysCm1No2zczs/oNmegjYh5wAHAecANwZkRcJ+lQSTsBSNpI0h3ArsBxkq7LT38VMEvS1cBFwBFtvXXMzKxm3fS6ISJmADPatn2xcH8mqUmn/XmXAq8uWUYzMyvBI2PNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhnOjNzBrOid7MrOGc6M3MGs6J3sys4ZzozcwazonezKzhFh3tAowX0w76TekYtxyxQwUlaaYq3l8Yufd4vJXX+ptr9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBdJXpJ20u6UdIcSQd12L+lpCslzZP0zrZ9+0i6Kd/2qargZmbWnSETvaQJwLHAm4G1gD0krdV22G3AvsBpbc99IXAwsAmwMXCwpOXLF9vMzLrVTY1+Y2BORMyNiKeB04GdiwdExC0RcQ3wXNtztwMuiIj7I+IB4AJg+wrKbWZmXeom0a8M3F54fEfe1o2unitpf0mzJM265557ugxtZmbdGBMXYyPi+IjYMCI2nDx58mgXx8ysUbpJ9HcCqxQeT8nbulHmuWZmVoFuEv1MYE1Jq0paHNgdmN5l/POAbSUtny/Cbpu3mZnZCBky0UfEPOAAUoK+ATgzIq6TdKiknQAkbSTpDmBX4DhJ1+Xn3g8cRvqwmAkcmreZmdkI6Wqa4oiYAcxo2/bFwv2ZpGaZTs89ATihRBnNzKyEMXEx1szM6uNEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcF0leknbS7pR0hxJB3XYv4SkM/L+yyVNy9unSXpC0lX59v1qi29mZkNZdKgDJE0AjgW2Ae4AZkqaHhHXFw7bD3ggItaQtDtwJLBb3ndzRKxfcbnNzKxL3dToNwbmRMTciHgaOB3Yue2YnYGT8v2fAW+SpOqKaWZmveom0a8M3F54fEfe1vGYiJgHPASskPetKumvkv4o6b86vYCk/SXNkjTrnnvuGdYvYGZmg6v7Yuy/gJdFxAbAx4HTJE1qPygijo+IDSNiw8mTJ9dcJDOz/tJNor8TWKXweEre1vEYSYsCywL3RcRTEXEfQETMBm4GXl620GZm1r1uEv1MYE1Jq0paHNgdmN52zHRgn3z/ncCFERGSJueLuUhaDVgTmFtN0c3MrBtD9rqJiHmSDgDOAyYAJ0TEdZIOBWZFxHTgR8DJkuYA95M+DAC2BA6V9AzwHPDBiLi/jl/EzMw6GzLRA0TEDGBG27YvFu4/Ceza4XlnA2eXLKOZmZXgkbFmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBO9GZmDedEb2bWcE70ZmYN50RvZtZwTvRmZg3nRG9m1nBdJXpJ20u6UdIcSQd12L+EpDPy/sslTSvs+2zefqOk7aorupmZdWPIRC9pAnAs8GZgLWAPSWu1HbYf8EBErAF8AzgyP3ctYHdgbWB74Ls5npmZjZBuavQbA3MiYm5EPA2cDuzcdszOwEn5/s+AN0lS3n56RDwVEf8E5uR4ZmY2Qhbt4piVgdsLj+8ANhnomIiYJ+khYIW8/bK2567c/gKS9gf2zw8flXRjV6XvzYrAvaMRW0eOr7glDBm3xzKPWtweNa68jjsm47ZMHWhHN4m+dhFxPHD8SLyWpFkRseF4iu24juu4jltGN003dwKrFB5Pyds6HiNpUWBZ4L4un2tmZjXqJtHPBNaUtKqkxUkXV6e3HTMd2CfffydwYURE3r577pWzKrAmcEU1RTczs24M2XST29wPAM4DJgAnRMR1kg4FZkXEdOBHwMmS5gD3kz4MyMedCVwPzAM+HBHP1vS7dKvOJqK6Yjuu4zqu4/ZMqeJtZmZN5ZGxZmYN50RvZtZwTvRmZg3XF4ne0y7YSJO0vKR1R7scZtAnF2MlzQXOBk6MiOsrjr0kaa6ftYElW9sj4r0l4y4BvAOYRqF3VEQcWiZujr08qatrsbx/Khu3DpJ2Bc6NiEckfQF4DfDliLiyZNylgU8AL4uI90taE3hFRJxTIuYfgJ1If6/ZwN3AJRHx8TJlzbHXav/flfSGiPhDybiv6bD5IeDWiJhXJnYdJB0InAg8AvwQ2AA4KCLOLxm3lvN4rOiLGj2wHvAP4IeSLpO0v6RJFcU+GXgJsB3wR9KgsEcqiPsr0lxB84DHCrdSJL0P+BOpu+wh+eeXKoi7i6SbJD0k6WFJj0h6uGxc4H9zkt8C2JrUlfd7FcQ9EXgK2DQ/vhP4csmYy0bEw8AuwE8iYhNSmatwpqTPKFlK0jHA4RXE/S5pmpLjgR8AfwHOAm6UtG2vQSW9TtJMSY9KelrSsxX9P7w3v8fbAssD7waOqCBuLedxjefF8EREX92A15NO6sdIE7GtUTLeX/PPa/LPxYDLKijn32r6/a8l1Viuyo9fCfy8grhzgFfVUN7W+3s4sGdxW8m4s9pjAVdX8N6+FDgf2Kj4f1FBeScC3yEl4r8BnwUWqSDuz4G1C4/XIk1MuFrrf6TX9xdYA/grafzNe4DDKyhv6zz7FvD2Cv8f6jqPazkvhnvrixq9pAmSdpL0C+CbwNdI/8i/BmaUDP9M/vmgpHVI0z+8qGRMgEslvbqCOO2ejIgnITUPRcTfgVdUEPc/EXFDBXHa3SnpOGA3YEZu0qri//ZpSUsBASBpdVINv4xDSd+Q5kTETEmrATeVjNnyDPAEsBTpg/qfEfFcBXFfHhHXtR5Eah56ZUTMLRs4IuYAEyLi2Yg4kTRVeVmzJZ0PvAU4T9ILgCreh7rO47rOi2EZE5OajYCbgIuAoyPi0sL2n0nasmTs43Ob9/+SpnxYBvhiyZgAWwD7SvonKQEJiIgoe4HvDknLAb8ELpD0AHBryZgAsySdkeM+nzAj4ucl476LlCC+GhEPSnop8KmSMQEOBs4FVpF0KrA5sG+ZgBFxFqnZo/V4Luk6SxVmkprzNiLNgvh9Se+IiF1Lxr1O0vdI049D+kC9Pn+gPjPw04b0eJ4y5SpJRwH/opoP6P2A9YG5EfG4pBVI3xbKqus8ruu8GJZ+uRi7TEQ8OtrlGA5JHaccjYgqknLrNV5PqrmcG2mtgTKxTuywOaL8RenVgTsi4ilJbwDWJbV/P1gmbo69AvA60ofoZRHR0xSykj4dEUfldvOFTqiI+Ei5koKkDSNiVtu2d0fEySXjLgV8iFSxALiE1G7/JLB0r+dN/v+9m9QE8jHS/9l3cy2/l3ivjIi/D3DxmCh5cb4udZ0Xwy5HkxP9QCdeS0Un4HLAf7Nw75gqYq8H/Fd+eHFEXF0i1qSIeFjSCzvtj4j7e41dJ0lXARuS3t8ZpFrt2hHxlh7jdUwULb0kDEk7RsSvJe3TaX9EnNRpey8kvYgFe4XcVlXssUzS8RGxv6SLOuyOiNiqZPzazuOxoOmJvuOJ11LFCSjpUlKvhWsptBWWjZ27kb2fdLEM4O3A8RFxTI/xzomIt+amoCDVYgvFjdVKlncKcAypCQTgYuDAiLijZNwrI+I1kj4NPBERx0j6a0Rs0GO8TomipXTCKLzOIsAykXqIVBFvR+DrwEqkmvJU4IaIWLtk3M1Jva6msmCC6+n/QdKZEfEuSdfS+dvNmBxbUON5XMt5MWyjfTV4vN+AK2uKew0wsfB4IhX14KipvBeQ2koXzbd9gQsqiHs5sAepp8mqeVstPZIqKOtpwKT8t7qetKLapyqKfTVp1bZW75A3Aj+qIO7fSetBvyjHXwFYoUS8l+afUzvdKijvrsAL8v0vkCpCG1QQt67zuJbzYri3RtfoWyRNBj5D6jpW/NpbuvYm6WPAo8A5LHixpVRTSK4RbRTze8gsCcyMiNI9cSStzMI1uFIDpiRdFRHrD7Wth7hrAR8E/hIRP1Va1+BdEdH7Aoop7n932h4RPykR86qIWF/SXqSBXQcBs6OCWqzy6kSSriYltuckXR0R65WMe3mk/v7jgqRrImLdPK7iy8DRwBfL/g41nse1nBfD1S+9bk4FzgB2ICWNfYB7Kor9NOmf7fPM/6oapO6bZZwIXJ67hAK8jTRYqBRJR5J7VgCttQGCNIiqjPsk7Q38ND/eg7TKWCmRuvt9pPD4n0CpJJ9tVLi/JPAm4Eqg50QPLCZpMdLf6jsR8YykqmpSD0pahvR3OlXS3VQwgA64SNLRpJpxMcGVHXm8C+nv9CJSM2Gr11jZgYqt/9kdSE2Zv5FUdqAb1Hce13JeDFe/1OhnR8RrW7WBvG1mRGw01HO7iD0X2Dh67LExROzXML83xMUR8dcKYt4IrBsRZfuMt8edSmqL3JR0glwKfCRKXixUmprgcBb+Nlb2BGx/neWA0yOi577ekj5C+uZ4NSkRvQw4JSL+a9Andhd7Iqkf/SLAXqReLKdGRKmkUePFzTnAjlFxH3JJ55AGPG5D+tb0BHBFBd9sajmP6zovhl2OPkn0l0XE6ySdB3wbuAv4WUSsXkHs84G3RcTjZWPleLX2jpH0W2DXGCfdTSX9mdTn/RvAjqT2zkUiooo+zsXXWYzU9l/F4LFi3EWj5JwxSpPy/S4i3lhRsWon6ZKI2HzoI4cdd2nSuIprI+KmPK7i1VF+rptKz+Oxpl+abr4saVnSJFbHkC6Yfayi2I+RBoVcxIJffXvtlnUa8FbSpFjFT2FRzVfJx0nl/T0VlHcE+pAvFRG/l6RIYwi+JGk2JQezSPo188u7COkbw5nligqSdqBtYizSiNmeRcSzkp6TtGxEPFSqgJmkvSPiFEkdJ1yLiK+XfIlaBgpFGiR1M7CdpO1I33RLJfms0vN4JMZWDEdfJPqYPyPhQ6TeClX6Zb5VIiLemn+uWlXMNtNZeHH3MlpfzWcNelTvnspdFW9SWrv4TtKoxbK+Wrg/jzRbY9muoN8Hlib9j/0QeCdwRZmYBY8C10q6gELbfImEMTH/fEHZgg1gEqlSUZwYLZjfXbgnHbodn5L72PfU7big0vOY+s+LYemXpptvd9j8EGliq19VEH8p0nS3N5aNVYi5OWlSqcfyxZzXAN8s07aXmwB+EhF7VVXOQuxdI00BMOi2HuJuRDpplgMOI7VNHxURl5WJW4dCj5DWz2WA31bURl/7YKzxQNI1wKYR8Vh+PJHUI6uKnk11nMe1nBfD1ReTmpG+Rq9PmvPmJtIw+inAfpK+WSZwHshyFWneFCStL6mKGvP3SPOFrEdqcrqZNJVqzyLiWWCq0hwkVftsl9uGJSJmRsSjEXFHRLwnInYpk+SVp4kd6FayuE/kn49LWok0V8xLS8YEnk/oPyU16c0GTqsiyUuaLOlzko6XdELrVkHcKZJ+IenufDtbafBQ6dDM73lDvq8Bju0+aH3ncS3nxXD1RdMNKbFvnhMdSpM4XUzq0XJtydhfAjYG/gAQEVcpzVpY1ryICEk7k7rq/UjSfhXEnQtckv+Ji00APbXJSnozaSbBldu+OU0iNYn0ZKiTLCJ26iVuRLwgxz+MNNHWyaREsRflk/I5uffO0aSumkFqwilNaZ6fk4BbSOVdRdI+Zcc/kKaUuBj4HQsm0LJOJF1vak26tnfetk0FcSvvdkzF53Fd50Wv+iXRL09q121dyJoIvDBf5CrbzfCZiHhIWqBSUcW0qY9I+izpBNkyt1MvVkHcm/NtEappn72L1A65E6mm2fII5S54bwrcTqrFXk4FtbY2O7V1yfue0mCkMhd5j8rdVs/O3QCXJE0OVoWvAdu2mhUkvZz03ry2ZNylI+IzZQvXweRIUxO3/FjSR8sGjYivK63k1ep2/J4quh1T/Xlc13nRk35J9EeRrqj/gZQwtgS+ktv3flcy9nWS9gQm5D7fHyH1lS1rN2BPYL+I+Lekl5FqiqVExCGQuqlV0ZUs0kRrV0s6LSLKTGvb7iWk2t8epPfhN8BPozB3ekmPKY1gPZ1U896D8gOQ/kK6lkJO+E9JurK1raTFim3HEfGP3CW0rHMkvSUiyq7L0K7ygUL5GtN1EfFK0jemKlV6Htd4XvSkLy7GAuT+thvnhzMj4q6K4i5NGk23LelD5DzgsMhTF4w1kjYlfdVdJiJelq8BfCAiPlQybm0Dm5TmRt+D9EF3SER8p4KY00irFLX6ev8Z+GhE3NJDrJcAKwOnkD6UWtXCScD3c2IqJbebP5dfA1JT04QoPw30I6RvuE/nWyUjWFXfALpfAf+v6gFHdZ3HdZ4XwypHPyR6pe9jewGrRcShuXb8koioqutb63UmkCYiKz1jYT4BW3+cxUnNNo9GxLIl415O6vY3PfIMkJL+FhHrlIxb+cCmnOB3ICX5aaRuoSdExJ1lylq13CNmX9J0yjOZn+gfBk4q23c8v8YSwIcpjJQmze9e6QjnsU7Sn0gLgl/BgteYerpmU7c6zoueytEnif57pNrQVhHxKqWVZM6PaqZAOI00f86zpJN8EvCtiCjdzFJ4DZEWCn9dRBxUMtblEbGJClP9qprJsVrTTFwbeeK11rYe4/0EWIc0B/3pEfG3MuXrEL/y6WOVVnw6u4ryjZRCJWjViDhM0iqkGShLVYKUJp/7fyw8v3uphKy0WM5CIuKPPcYrDpzrFLdseSs9L3rVL90rN4mID5MvjEXEA6RachXWyjX4twG/BVYlrUxfmUh+SVqhvqzbJW0GhKTFJH2S+YM7ylhgYJOkt1NuYNPewJrAgaT1c1tdIB+poBskpN4b00nzu69EWj+402pAw/Ha3OsGAEnLq5oJt5C0uaQLJP1D0tzWrYLQ3yU1r+yZHz8KHFtB3F+SeggdQ7qQ3LqVkhP6jaTxFJOAG3tN8tlXc7n+Seoe+4N8e5TUaaGsqs+L3sQIz4s8GjdSr40J5DmngVIxCKsAABA/SURBVMlUsHJ8jnUdqVnlLOD1edvVFcTdpXB7J3AEaWBI2bgrkmbz/A9pAYtTKDH/eCHuRqR/4CmkhHk26RvIqP/9ByjvVd1sG2bMhf6nqGiecyqeN769fMWyV/T/e3lNf7f3AbcBP2Z+d9P3VhB3Vjfbeojbfl78fDTOi37pdfNt4BfAiyT9HylxfqGi2MeR/tmuBv6UL0JVUePcsXB/Xn6NncsGjTQ7X6UjY/O1id0i4pOkmlAVizXXrY7pYydIWiJyu7nSSMslSsZseSgifltRrKJn8t8vIA2gopruwd+SdDBwPhVOf0xaGH6DyLN2Kq37eylQdpDXREmrRVrQvdX0NHGI5wwpImbmu6N6XvRFoo+IU5UmwnoT6ULZ26K66VOPjYjnB0RIuo0S8+lIOjJSv+bfRkTpSbYKcWtbPzfSeIQthj5yTHkvqVnhG8zvFVL2RDwV+L3mLwj9HlKts2eav8ZtLfPG07kS9L8lYwK8mtSEuRXzPzgiPy7jPlJf9JZHqGZ+948Bf8jNYSItzPOBXoPV3fY/XH1xMRZSeymwCgteGCrdFzf/Y/wMOLGKDw+llaXWJa1MVEX/61bc4lwph5B6Ajwvyq+N+T1SF8OzWLA3ROkeJ+OJpO2BrfPDCyLivJLxal/jVtIrmV8J+n1F/8dzSNevni4bK8drzbK5PulD5FekRLozaYnNfSt4jSWAVlfYv0eJHk2Fi8a7kMaEtLrF7gH8JyJGdNBUXyR6peHu+5Iurjy/ekxFJ8kLgN3J3aZIXyFPjx67WOZa2/tJ7XrFAU1VrdCDSiyuPUjMThcyI0r2866apMG6tUVEHFYy/lRgzYj4Xe6bPSEiHhnqeV3Efb5ZYbBtPcTdLyJ+1LbtiCjfu+uXwP4RcXeZOIV4Bw+2P/JAwB7ifjoijsr3F5hsTNJXIuJzvcQtxJgVERsOta1u/ZLobyQtTlBJ7WKQ13k9aX6P5Ui1/MMiYs4wYywREU9J+lVElG6TH+A1rqzy20KOuXlEXDLUttEm6RMdNk8E9iNd3Oy5R4Sk9wP7k6bXWD0Plvl+RLyp15iF2Av9zaropidpBmmlqlPz42OBJSOi1LxKSqPQ1yV1OS42NY2p/u7F97X9Pa7iPJF0A7BDW9v/jIh4VZm4w9UXbfTA30jJt5LaRVG+kLUDqUY/jdRV61Tgv0h9wF8+zJCtYfRVXNAdScew8FD/TttGVUQ838Uvfxs7kPS3O53y3f8+TBp9fXl+rZskvahMwNyssjawrNI6rC2TWHBxk169A5gu6TnSyk0Plk3y2aA18OGS9M2I+OhAbd8lPkA0wP1Oj3tRadt/r/ol0R8O/FXS36i+dnETcBFwdEQU58b4maQte4i3uNKcG5u1ndhA723eWnCk7dKFvuilmoSUplTYDJisBVcrmkTq0jrmKC3T+HFS76OTgNdEGltR1lMR8bTyxFiSFmWQC3JdegVpxbHlWLAn1iOkJr6eaMGlKt9H6vd+CXCIpBdGySUro1zf9k5aU3R/ddCjhi8GuN/p8fCDR5ybv9lV0vbfq35J9CeRVqS/lmq6jhWtGwOsv9pjT5YPkhJQ+4kNJVboiTw9bw0WJ11PWJQFZ8N8mNSDY0zJ10B2AY4nNedVuXbuHyV9DlhK0jbAh0gDsXoWaWGcX0naNCL+UkUhs9ZSlSr83CHfSi9Zqc5TeDxW4hrTPVDLB8h6udIj0t+tWAHq+RuTpK0i4sIOlbXVJY14J4V+aaOfGRVMdzBA7MmkmtU0FuzRU3ayqYUuko1lkqZGWtN1TMtNFE+RxiYstCZvmYvdeQTkfiw4MdYPo4KTTDVM2TBSpPJTeLS1pZ8dEe+osoxVk3RIRBw8Vjop9Eui/zrp5J5OtX2QkXQp6aSbTWHhhuhxzpO6ewFUbaz1F24qpbViT2N+E8bewF4RUWohD0kfJl2MfTA/Xh7YIyK+WybuAK/Vc28vLTg3U+W9xpquXxJ9p77IVXWvvCoi1i8bpxCv1l4AVdMAk0y11PBVe8zJYx8G+7CrYj3ThSaeq+J/r1OMKhJpW5PFIqSZPV8fEZv2GG/A82IsartetZDocUW3XvVFG31E9DxStQtVL9xQdy+ASvVDIu/CW0fgNe5V9VM2QJq2Qa3mpdyLrIoJ/6qewmOwtvRKxpdUrK5rYj3plxr9sqTuXq1eMH8EDo2IhwZ+1pAxWxebxPyFG1oryZTpxTLeavRnRsS7BqrVVlGbHU/aBkwtBSxa0YCpuhbyOJrU5e+4vOkDwO0R0Wm8gY1T/ZLozyb1pW8N8383sF5ELNR9cbRJepY0hYCApZg/OlakgSxVLB9XGUkrRcRdOREtZDxcoK1KnQOm6pIvIH+ANAUCwAWkC8ilFgpXWtP2e8CLI2IdSeuS1umtZNrm8WKsXETvl0TfqR2ysrb13B65BammdXGkueP7QutbhqSTI6LSefjHG0lXkQdMFS4cPr/gRMnYtSzkURdJfyTNNHlcVLiS2XhT10X04eqLNnrgCUlbRMSfIQ3NJy0yUJqk7wJrML/t9IOStom00Ek/qGWA1zhVx4Cpll+S1vr9NRWMBRmBJrelI+KK1nuRzSsZczyaHBHFLpY/lvTRkS5EvyT6/wFOym31Au4nTXJWha2AVxUuZp1EWoykX9QywGucqnzAVMGTUZgOuwIH5p91XUi+V9LqzJ/n/p3Av2p6rbGsjnUPhq0vmm5aJE0CiAoW7y7EPAf4cKstOrdVfyci2pNeo423AV516DRgKiJ+UFHsPUlLK1a9kEctJK1GGn28GfAAaam+vSPiltEs10hru4gOaZqJ0hfRh12OJid6SXtHxCkD9Wmtoi9rbovciLQqPfn+LOCh/Bpjsg21Dkpr0U5jwTbkn4xagUaYpAMj4ltDbesx9uGkTgQ3U1jIo+xYkNzcdiRpiUJRcXdFSROBRaroeWS9a3rTTWspsDr7tA42v3nfkHQysDpwFfNHCAfQN4ke2AdoT+r7dtjWi12B1aL6qbaPAnaM6lZcA0BpEY93kD/4W231EXFola8z1kk6Cvgy6ZrguaSpmz8WEacM+sSqy9HkGv1Iys1CxZpsqdn/xhulebfXqmJel/FG0h7AnqSeVxcXdk0C5kXE1h2fOLzXqHQhj0LcSyJi86GPHHbcc0nfatunBik7FfS40urdJ+ntpOshHwf+1D7KuW6NrtFLGvTiVZRYJ7XwGvsDhwJPkr5St2YDLDX73zj0N9KSaf14we1S0u+9IgvOaR/AbhW9xnLA3yVVspBHoYfULElnkHr1FOOWvYg+JSK2LxmjCVo5dgfgrIh4qK0n0ogWoqlm55+bA2sBZ+THuwLXV/QanwLWiYh7K4o3Xq0IXC/pCsbwikJ1yBfibwU2lbQBqXa/K+kCZE+T23VQ6UIeLNhD6nHSBeSWKnpLXSrp1RFxbck44905kv5Oarr5H6XZbp8c6UL0RdONpMuALSJiXn68GGlg0+sqiH0usEtEPD7kwQ020ORm/TAXTh4Fuke+3UuqUHwyIjqOFh4LJB0QEd+pMf71pF5Cc0kf/K2LvH01JQbQWuTloYh4Vmkd4UkR8e8RLUOfJPobgU1b7eZKU7FeFhGvqCD2BsCJpOXjijXZ0s1CNj4ozXF/MbBf5DWCJc2NiMqa71TxQh51z5vkKTHmGwu90ZredNNyBGkpwYtINYstgS9VFPs44ELqWb1qzGtLQAvsYmzOKliHXYDdgYvyN7zTqXim0SisECbNX8ijyteogqQlSYPo1iCdEz9qfZPuR2OlN1rja/T5pJhCmllyk7z58qq+OsmLIFiW+4zvTGrC2Yp0Mv8iIs6v6fXKLOQxj/kT5i2wi3Kzr55BOtcuBt4M3BoRBw7+rOYaK73RGp/oobqJpQaI/RXSXNu/ZsGmm77qXmkLys2DuwK7VTF7papfyKOWCkrxXMtz/Vwx1qbWHkmSziKNhB3V3mj90nRzpaSNImJmDbH3yD8/W9jWj90rrSAiHiBNAXB8RSGrXsijLq01GYiIeaPRlXCMGRO90fqlRv93Upvhrcyf670vewCYAUj6XER8pYvjPhsRhw8jbms9BVhwTYV+umbzvLHSG61fEn3lPQA0zhbxtvFttBbyqLt3Tj+Q9GLSHFiQmrIqHd3cjUVG+gVHQ0TcmpP6E6RmldatjN0L9z/bts8jAq1qPyD9nz0DEBHXsOD/YF36vu2lDEnvIk14uCvwLuDyPGXziOqLNnpJO5GGpq8E3E1aI/MGYO0yYQe43+mxWVmjtZBH87/y1+vzwEatWnweGfs74GcjWYi+qNEDh5H6HP8jIlYlrY95WcmYMcD9To/NyhqthTxcaSlnkbammvsYhbzbFzV64JmIuE/SIpIWiYiLJH2zZMz1JD1MvuCU75MfL1kytlm7D5N68LxS0p3khTxG4HXPGvoQG8S5ks5j/gpTuwEzRroQ/XIx9nfA24DDSd2d7iZ9ndpsVAtmNkxVLeQh6RgG+ebpKTzKkbQG6cL5JXkMxBZ514PAqRFx84iWp8mJvvVmk4YfP0H6yrQXqY3+NxExe5Cnm40Z7Qt5tLb3upCHpH0G2x8RJ/US15K8xOhn22fvlPRq4CsjvdRo0xP9mHqzzXrlhTzGF0kzI2KjAfbVNlJ/IE1vo39xp/mwI+JaSdNGvjhmPatlIY/cC+QzpPUanr+2VHYtWmO5QfYtNWKlyJre62ZMvdlmJVyav4lW7VRSV+NVgUNIUyvUMVVIv5kl6f3tGyW9j/kLIo2Ypjfd/BS4MCJ+0Lb9fcA2EVHVMm9mtaprIQ9JsyPitZKuacUarNnBupNHw/4CeJr5iX1D0loCbx/phUea3nTzUeAXkvaiw5s9aqUyG7431xS3NQnZvyTtANwFvLCm1+obEfEfYDNJbwTWyZt/ExEXjkZ5Gl2jb2l7s68brTfbbLjqXshD0ltJc8evAhwDTAIOiYjpVb2Gjb6+SPRm41WdC3lImkCaK/0bVcSzscuJ3mwMq3shD0lXRMTGVcWzsanpbfRm413dC3lcIuk7wBnMn0eeiLiy6hey0eMavdkYVvdCHpIu6rA53I++WZzozcwazk03Zn1M0hc7be91Dh0bm5zozfrbY4X7SwJvJY2UtQZx042ZPS/PknleRLxhtMti1Wn6XDdmNjxLA1NGuxBWLTfdmPUxSdcyfwGSCcBkwO3zDeOmG7M+Jmlq4eE84D9VTrFgY4Obbsz6WETcSprnZquIuBNYTtKqo1wsq5hr9GZ9TNLBpBldXxERL5e0EnBWRGw+ykWzCrlGb9bf3g7sRO5mGRF3AS8Y1RJZ5Zzozfrb05G+1geApImjXB6rgRO9WX87U9JxpLb59wO/A34wxHNsnHEbvVmfk7QNsC1porTzIuKCUS6SVcyJ3sys4TxgyqwPSXqE1C4v5g+YgoqmP7axxTV6M7OGc43erA+1LTp+DXCCR8Q2l2v0Zn2ozkXHbexxojfrQ3UvOm5ji/vRm/WnBRYdH82CWP1cozfrQ3UvOm5jixO9mVnDuenGzKzhnOjNzBrOid7MrOGc6M3MGu7/A93a7/FbUeGTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.xticks(rotation=90)\n",
    "plt.bar(x=labels, height=np.mean(train_generator.labels, axis=0))\n",
    "plt.title(\"Frequency of Each Class\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from this plot that the prevalance of positive cases varies significantly across the different pathologies. (These trends mirror the ones in the full dataset as well.) \n",
    "* The `Hernia` pathology has the greatest imbalance with the proportion of positive training cases being about 0.2%. \n",
    "* But even the `Infiltration` pathology, which has the least amount of imbalance, has only 17.5% of the training cases labelled positive.\n",
    "\n",
    "Ideally, we would train our model using an evenly balanced dataset so that the positive and negative training cases would contribute equally to the loss. \n",
    "\n",
    "If we use a normal cross-entropy loss function with a highly unbalanced dataset, as we are seeing here, then the algorithm will be incentivized to prioritize the majority class (i.e negative in our case), since it contributes more to the loss. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3nHRd9p9n5yU"
   },
   "source": [
    "#### Impact of class imbalance on loss function\n",
    "\n",
    "Let's take a closer look at this. Assume we would have used a normal cross-entropy loss for each pathology. We recall that the cross-entropy loss contribution from the $i^{th}$ training data case is:\n",
    "\n",
    "$$\\mathcal{L}_{cross-entropy}(x_i) = -(y_i \\log(f(x_i)) + (1-y_i) \\log(1-f(x_i))),$$\n",
    "\n",
    "where $x_i$ and $y_i$ are the input features and the label, and $f(x_i)$ is the output of the model, i.e. the probability that it is positive. \n",
    "\n",
    "Note that for any training case, either $y_i=0$ or else $(1-y_i)=0$, so only one of these terms contributes to the loss (the other term is multiplied by zero, and becomes zero). \n",
    "\n",
    "We can rewrite the overall average cross-entropy loss over the entire training set $\\mathcal{D}$ of size $N$ as follows: \n",
    "\n",
    "$$\\mathcal{L}_{cross-entropy}(\\mathcal{D}) = - \\frac{1}{N}\\big( \\sum_{\\text{positive examples}} \\log (f(x_i)) + \\sum_{\\text{negative examples}} \\log(1-f(x_i)) \\big).$$\n",
    "\n",
    "Using this formulation, we can see that if there is a large imbalance with very few positive training cases, for example, then the loss will be dominated by the negative class. Summing the contribution over all the training cases for each class (i.e. pathological condition), we see that the contribution of each class (i.e. positive or negative) is: \n",
    "\n",
    "$$freq_{p} = \\frac{\\text{number of positive examples}}{N} $$\n",
    "\n",
    "$$\\text{and}$$\n",
    "\n",
    "$$freq_{n} = \\frac{\\text{number of negative examples}}{N}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='Ex-2'></a>\n",
    "### Exercise 2 - Computing Class Frequencies\n",
    "Complete the function below to calculate these frequences for each label in our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li> Use numpy.sum(a, axis=), and choose the axis (0 or 1) </li>\n",
    "</ul>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TpDGeY2cChYD"
   },
   "outputs": [],
   "source": [
    "# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def compute_class_freqs(labels):\n",
    "    \"\"\"\n",
    "    Compute positive and negative frequences for each class.\n",
    "\n",
    "    Args:\n",
    "        labels (np.array): matrix of labels, size (num_examples, num_classes)\n",
    "    Returns:\n",
    "        positive_frequencies (np.array): array of positive frequences for each\n",
    "                                         class, size (num_classes)\n",
    "        negative_frequencies (np.array): array of negative frequences for each\n",
    "                                         class, size (num_classes)\n",
    "    \"\"\"\n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "    \n",
    "    # total number of patients (rows)\n",
    "    N = labels.shape[0]\n",
    "    \n",
    "    positive_frequencies = np.sum(labels, axis=0) / N\n",
    "    negative_frequencies = 1 - positive_frequencies\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "    return positive_frequencies, negative_frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "BqidQvCaD_xi",
    "outputId": "56a5905a-e61b-47a8-f444-aa89d7481c44"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "labels:\n",
      "[[1 0 0]\n",
      " [0 1 1]\n",
      " [1 0 1]\n",
      " [1 1 1]\n",
      " [1 0 1]]\n",
      "pos freqs: [0.8 0.4 0.8]\n",
      "neg freqs: [0.2 0.6 0.2]\n"
     ]
    }
   ],
   "source": [
    "# Test\n",
    "labels_matrix = np.array(\n",
    "    [[1, 0, 0],\n",
    "     [0, 1, 1],\n",
    "     [1, 0, 1],\n",
    "     [1, 1, 1],\n",
    "     [1, 0, 1]]\n",
    ")\n",
    "print(\"labels:\")\n",
    "print(labels_matrix)\n",
    "\n",
    "test_pos_freqs, test_neg_freqs = compute_class_freqs(labels_matrix)\n",
    "\n",
    "print(f\"pos freqs: {test_pos_freqs}\")\n",
    "\n",
    "print(f\"neg freqs: {test_neg_freqs}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Expected output\n",
    "\n",
    "```Python\n",
    "labels:\n",
    "[[1 0 0]\n",
    " [0 1 1]\n",
    " [1 0 1]\n",
    " [1 1 1]\n",
    " [1 0 1]]\n",
    "pos freqs: [0.8 0.4 0.8]\n",
    "neg freqs: [0.2 0.6 0.2]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Iye-sQoOFG37"
   },
   "source": [
    "Now we'll compute frequencies for our training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LoxM5jQ0E30D"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 0.11111111, 0.22222222, 0.        , 0.22222222,\n",
       "       0.11111111, 0.        , 0.11111111, 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        ])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freq_pos, freq_neg = compute_class_freqs(train_generator.labels)\n",
    "freq_pos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gsJIDPTZn5yW"
   },
   "source": [
    "Let's visualize these two contribution ratios next to each other for each of the pathologies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "colab_type": "code",
    "id": "IqnNCu4In5yW",
    "outputId": "245f1a6b-b292-4c6d-a583-c6924bc61f31",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAFaCAYAAAATl1rLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deZgcZbn+8e+dEEjCLgQVAgQVlABhS4CALKJAECWCbBFUPCD6Oy7ReDiCR1k9yOJxAREEAdlkEQRRgQAKsgaSQNiXIAgEVCBCgGCAwPP7462e9Exmksn0Wz0zXffnuubKdHXNU+90puqpeldFBGZmVl0DersAZmbWu5wIzMwqzonAzKzinAjMzCrOicDMrOKW6u0CLKlVV101RowY0dvFMDPrV6ZPn/5iRAzr7L1+lwhGjBjBtGnTersYZmb9iqSnunrPVUNmZhXnRGBmVnFOBGZmFdfv2gg689ZbbzFr1izmzZvX20XplwYPHszw4cMZNGhQbxfFzHpBSySCWbNmsfzyyzNixAgk9XZx+pWIYPbs2cyaNYt11lmnt4tjZr2gJaqG5s2bxyqrrOIk0AOSWGWVVfw0ZVZhpSUCSWdLel7SA128L0knS3pc0n2SNmvweI38eKX5szOrtjKfCH4FjFvE+7sC6xZfhwCnlVgWMzPrQmmJICJuBv61iF3GA+dFMgVYSdJ7yypPTsstt1y39z3qqKP44Q9/WFp8M7NG9WZj8RrAM3WvZxXb/t5xR0mHkJ4aWGuttZboIG8892C39ltm9Q2WKG6rqH0+81/+B08fs0+n+6x1xP3NLFKf8vQxGy12n776+XSn7NDz8jt+34/f3dj9orE4Is6IiNERMXrYsE6nyuh1v//979lyyy3ZdNNN+djHPsY///nPtvfuvfdexo4dy7rrrsuZZ57Ztv2kk05izJgxjBo1iiOPPLI3im1m1quJ4FlgzbrXw4tt/dKHP/xhpkyZwj333MN+++3HiSee2Pbefffdx5///GfuuOMOjjnmGJ577jmuu+46Zs6cyV133cWMGTOYPn06N998cy/+BmZWVb1ZNXQV8FVJFwNbAnMiYqFqof5i1qxZ7Lvvvvz973/nzTffbNcnf/z48QwZMoQhQ4bwkY98hLvuuotbb72V6667jk033RSA1157jZkzZ7Lddtv11q9gZhVVWiKQdBGwA7CqpFnAkcAggIg4Hbga+DjwOPA68IWyytIMX/va15g0aRK77747N910E0cddVTbex27Z0oiIjj88MP50pe+1OSSmpm1V2avoQkR8d6IGBQRwyPirIg4vUgCFL2FvhIR74+IjSKiX88tPWfOHNZYYw0Azj333Hbv/e53v2PevHnMnj2bm266iTFjxrDLLrtw9tln89prrwHw7LPP8vzzzze93GZmLTHFRLO9/vrrDB8+vO31pEmTOOqoo9h7771ZeeWV2XHHHXnyySfb3h81ahQf+chHePHFF/ne977H6quvzuqrr87DDz/M2LFjgdRl9IILLmC11VZr+u9jZtXmRNAD77zzTqfbx48fv9C2+iqijiZOnMjEiRMX2l57SjAza4Z+0X3UzMzK40RgZlZxTgRmZhXnRGBmVnFOBGZmFedEYGZWcS3ZfXTzQ8/rwU9N7/qdkz632J8eOHAgG220EfPnz2f99dfn3HPPZejQoUtUgoMPPphJkyYxcuRIjjvuOL7zne+0vbf11ltz++23L1E8M7Pu8BNBJkOGDGHGjBk88MADLL300px++ulLHOOXv/wlI0eOBOC4445r956TgJmVxYmgBNtuuy2PP/44AD/60Y/YcMMN2XDDDfnJT34CwNy5c9ltt93YeOON2XDDDbnkkksA2GGHHZg2bRqHHXYY//73v9lkk03Yf//9gQWL1ey333788Y9/bDvWgQceyGWXXcbbb7/NoYce2jat9S9+8Ytm/spm1o+1ZNVQb5o/fz7XXHMN48aNY/r06ZxzzjnceeedRARbbrkl22+/PU888QSrr7562wV9zpw57WIcf/zx/OxnP2PGjBkLxd9333259NJL2W233XjzzTf505/+xGmnncZZZ53FiiuuyNSpU3njjTfYZptt2HnnndvNgmpm1hk/EWRSu4MfPXo0a621FgcddBC33nore+yxB8suuyzLLbcce+65J7fccgsbbbQR119/Pd/+9re55ZZbWHHFFbt9nF133ZUbb7yRN954g2uuuYbtttuOIUOGcN1113HeeeexySabsOWWWzJ79mxmzpxZ4m9sZq3CTwSZ1NoIumO99dbj7rvv5uqrr+a73/0uH/3oRzniiCO69bODBw9mhx12YPLkyVxyySXst99+AEQEp5xyCrvsskuPfwczqyY/EZRo22235corr+T1119n7ty5XHHFFWy77bY899xzDB06lAMOOIBDDz2Uu+++e6GfHTRoEG+99Vancffdd1/OOeccbrnlFsaNGwfALrvswmmnndb2M4899hhz584t75czs5bRkk8E9d09e3Px+s0224wDDzyQLbbYAkjdQzfddFMmT57MoYceyoABAxg0aBCnnXbaQj97yCGHMGrUKDbbbDMuvPDCdu/tvPPOfPazn2X8+PEsvfTSbbH/9re/sdlmmxERDBs2jCuvvDL772RmraclE0Fv6Grq6EmTJjFp0qR223bZZZdOq3Buuummtu9POOEETjjhhE7jDxo0iH/961/tfnbAgAEcd9xxC3U7NTNbHCeCBvXmE4fB08dstNh91jri/iaUZMl1p+zQd8tvrcNtBGZmFedEYGZWcU4EZmYV50RgZlZxTgRmZhXXkr2Gutsbo7u602tj8BobMvGQz3PCkYcC8OPTz+G1ua/zvW99JWtZPD21meXmJ4JMlllmaa685gZe/NdLpR7H01ObWW5OBJksNXAgB+2/FyefsfCiOC/M/hef/vSnGTNmDGPGjOG2225L2194gZ122okNNtiAgw8+mLXXXpsXX3wRgE996lNsvvnmbLDBBpxxxhkAnp7azErhRJDRlw+cwMVX/JE5r7zabvu3jjieb37zm0ydOpXLL7+cgw8+GICjjz6aHXfckQcffJC99tqLp59+uu1nzj77bKZPn860adM4+eSTmT17Nscff3zb5HYdp52oTU8NtE1Pvdtuu7Wbnnrq1KmceeaZPPnkkyV/EmbWn7RkG0FvWWH55dh/r9059awLGTJkmbbtN94yhUef/Grb61deeYXXXnuNW2+9lSuuuAKAcePGsfLKK7ftc/LJJ7e998wzzzBz5kxWWWWVLo+96667MnHiRN544w2uvfbadtNT33fffVx22WVAWvtg5syZXqfAzNo4EWT2tYM/y1bj9uZz+36qbds777zDlClTGDx4cLdi3HTTTdxwww3ccccdDB06lB122IF58+Yt8mc8PbWZ9ZSrhjJ718orstcnd+FXF/22bdtHt9+aU045pe11bd2CbbbZpq0657rrruOll1JD85w5c1h55ZUZOnQojzzyCFOmTGn7WU9PbWa5teQTQX13z96YFG7ilz7Paedc1Pb6R8cezqRjT2HUqFHMnz+f7bbbjtNPP50jjzySCRMmcP755zN27Fje8573sPzyyzNu3DhOP/101l9/fT74wQ+y1VZbtcXy9NRmlltLJoLeMHvm1Lbv3z1sVV7667S216u+a+W2BerrrbjiikyePJmlllqKO+64g6lTp7LMMqlt4Zprrun0OJ6e2sxyKzURSBoH/BQYCPwyIo7v8P5awLnASsU+h0XE1WWWqS95+umn2WeffXjnnXdYeumlOfPMM3u7SGZWQaUlAkkDgVOBnYBZwFRJV0XEQ3W7fRe4NCJOkzQSuBoYUVaZ+pp1112Xe+65p7eLYWYVV2Zj8RbA4xHxRES8CVwMjO+wTwArFN+vCDzX04NFRE9/tPLSZ+fPz6yqykwEawDP1L2eVWyrdxRwgKRZpKeBr3UWSNIhkqZJmvbCCy8s9P7gwYOZPXu2k0EPRAQvz32Tga88s/idzawl9XZj8QTgVxHxf5LGAudL2jAi3qnfKSLOAM4AGD169EJX++HDhzNr1iw6SxLzX/5Htwqy1Jye5cSy45dt/sv/YOArzzD0HrdPmFVVmYngWWDNutfDi231DgLGAUTEHZIGA6sCzy/JgQYNGtTlSNmnj9mnWzF6ui5s2fHL1t3ym1nrKvM2dSqwrqR1JC0N7Adc1WGfp4GPAkhaHxgMLHxbb2ZmpSktEUTEfOCrwGTgYVLvoAclHSNp92K3bwFflHQvcBFwYLii38ysqUptIyjGBFzdYdsRdd8/BGxTZhn6u+4sstNXq52g/5ffrAr6ZgummZk1jROBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFlZoIJI2T9KikxyUd1sU++0h6SNKDkn5dZnnMzGxhS5UVWNJA4FRgJ2AWMFXSVRHxUN0+6wKHA9tExEuSViurPGZm1rkynwi2AB6PiCci4k3gYmB8h32+CJwaES8BRMTzJZbHzMw6UWYiWAN4pu71rGJbvfWA9STdJmmKpHGdBZJ0iKRpkqa98MILJRXXzKyaeruxeClgXWAHYAJwpqSVOu4UEWdExOiIGD1s2LAmF9HMrLWVmQieBdasez282FZvFnBVRLwVEU8Cj5ESg5mZNUmZiWAqsK6kdSQtDewHXNVhnytJTwNIWpVUVfREiWUyM7MOSksEETEf+CowGXgYuDQiHpR0jKTdi90mA7MlPQTcCBwaEbPLKpOZmS1ssd1HJb0bOA5YPSJ2lTQSGBsRZy3uZyPiauDqDtuOqPs+gEnFl5mZ9YLuPBH8inTnvnrx+jHgG2UVyMzMmqs7iWDViLgUeAfaqnzeLrVUZmbWNN1JBHMlrQIEgKStgDmllsrMzJqmO1NMTCL19nm/pNuAYcBepZbKzMyaZrGJICLulrQ98EFAwKMR8VbpJTMzs6boTq+hz3XYtJkkIuK8kspkZmZN1J2qoTF13w8GPgrcDTgRmJm1gO5UDX2t/nUxF9DFpZXIzMyaqicji+cC6+QuiJmZ9Y7utBH8nqLrKClxjAQuLbNQZmbWPN1pI/hh3ffzgaciYlZJ5TEzsybrThvBX5pREDMz6x1dJgJJr7KgSqjdW6T54lYorVRmZtY0XSaCiFi+mQUxM7Pe0Z02AgAkrUYaRwBARDxdSonMzKypFtt9VNLukmYCTwJ/Af4GXFNyuczMrEm6M47gWGAr4LGIWIc0snhKqaUyM7Om6U4ieKtYPnKApAERcSMwuuRymZlZk3SnjeBlScsBtwAXSnqeNLrYzMxaQJdPBJJOlfRhYDzwOml5ymuBvwKfbE7xzMysbIt6IngMOAl4L2lKiYsi4tymlMrMzJqmyyeCiPhpRIwFtgdmA2dLekTSEZLWa1oJzcysVIttLI6IpyLihIjYFJgA7AE8XHrJzMysKbozjmApSZ+UdCFp/MCjwJ6ll8zMzJpiUXMN7UR6Avg4cBdpMZpDIsI9hszMWsiiGosPB34NfCsiXmpSeczMrMkWNencjs0siJmZ9Y6eLFVpZmYtxInAzKzinAjMzCrOicDMrOKcCMzMKs6JwMys4kpNBJLGSXpU0uOSDlvEfp+WFJK8zoGZWZOVlggkDQROBXYFRgITJI3sZL/lgYnAnWWVxczMulbmE8EWwOMR8UREvEmaomJ8J/sdC5wAzCuxLGZm1oUyE8EawDN1r2cV29pI2gxYMyL+uKhAkg6RNE3StBdeeCF/Sc3MKqzXGoslDQB+BHxrcftGxBkRMToiRg8bNqz8wpmZVUiZieBZYM2618OLbTXLAxsCN0n6G7AVcJUbjM3MmqvMRDAVWFfSOpKWBvYDrqq9GRFzImLViBgRESOAKcDuETGtxDKZmVkHpSWCiJgPfBWYTFrR7NKIeFDSMZJ2L+u4Zma2ZBa1HkHDIuJq4OoO247oYt8dyiyLmZl1ziOLzcwqzonAzKzinAjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqzonAzKzinAjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqzonAzKzinAjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqzonAzKzinAjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqzonAzKzinAjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqbqneLkBVbX7oed3a74rlSy5ID/X38pvZAqU+EUgaJ+lRSY9LOqyT9ydJekjSfZL+JGntMstjZmYLKy0RSBoInArsCowEJkga2WG3e4DRETEKuAw4sazymJlZ58p8ItgCeDwinoiIN4GLgfH1O0TEjRHxevFyCjC8xPKYmVknykwEawDP1L2eVWzrykHANZ29IekQSdMkTXvhhRcyFtHMzPpEryFJBwCjgZM6ez8izoiI0RExetiwYc0tnJlZiyuz19CzwJp1r4cX29qR9DHgf4DtI+KNEstjZmadKPOJYCqwrqR1JC0N7AdcVb+DpE2BXwC7R8TzJZbFzMy6UFoiiIj5wFeBycDDwKUR8aCkYyTtXux2ErAc8BtJMyRd1UU4MzMrSakDyiLiauDqDtuOqPv+Y2Ue38zMFq9PNBabmVnvcSIwM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OKcyIwM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OKK3X20f5s80PP69Z+VyxfckEqqr9//t0pf18tu1WPnwjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqzonAzKzinAjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqzonAzKzinAjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqzonAzKzinAjMzCqu1EQgaZykRyU9LumwTt5fRtIlxft3ShpRZnnMzGxhpSUCSQOBU4FdgZHABEkjO+x2EPBSRHwA+DFwQlnlMTOzzpX5RLAF8HhEPBERbwIXA+M77DMeOLf4/jLgo5JUYpnMzKwDRUQ5gaW9gHERcXDx+rPAlhHx1bp9Hij2mVW8/muxz4sdYh0CHFK8/CDw6BIUZVXgxcXu1XOO7/h9MbbjO35Ha0fEsM7eWCpPecoVEWcAZ/TkZyVNi4jRmYvk+I7fp2M7vuMviTKrhp4F1qx7PbzY1uk+kpYCVgRml1gmMzProMxEMBVYV9I6kpYG9gOu6rDPVcDni+/3Av4cZdVVmZlZp0qrGoqI+ZK+CkwGBgJnR8SDko4BpkXEVcBZwPmSHgf+RUoWufWoSsnxHb8J8ftz2R2/heKX1lhsZmb9g0cWm5lVnBOBmVnFORGYmVVcSyaCYnoLsz5N0sqSRvV2OcxasrFY0hPA5cA5EfFQCfEHk+ZJ2gAYXNseEf+RKf4ywKeBEdT17IqIY3LEL46xMrAu7ct/c674ZZK0N3BtRLwq6bvAZsD3I+LuTPGHAt8C1oqIL0paF/hgRPwhQ+ybgN1J/6/TgeeB2yJiUqOxi/gjO/7NS9ohIm7KEb+It1knm+cAT0XE/FzHKYOkicA5wKvAL4FNgcMi4rpM8Uu9NpSlJZ8IgI2Bx4BfSpoi6RBJK2SMfz7wHmAX4C+kwXKvZoz/O9I8TPOBuXVfWUg6GLiZ1LX36OLfozLG31PSTElzJL0i6VVJr+SKD3yvSAIfBj5G6oZ8Wsb45wBvAGOL188C388Ue8WIeAXYEzgvIrYk/Q65XCrp20qGSDoF+EHG+AA/B6aQui+eCdwB/AZ4VNLOjQSWtJWkqZJek/SmpLcz/+38R/H57wysDHwWOD5j/NKuDaWeVxHR0l/A9qQTeS5pgrsPZIh5T/HvfcW/g4ApGcv8QMmfyf2ku5UZxesPAb/NGP9xYP0Sy1/7/H8AfKZ+W6b40zrGBO7N+Nm/F7gOGFP/d5Qp/rLAz0gX5weAw4EBmT//3wIb1L0eSZo08n21v6lGPnvgA8A9pPFHXwB+kLHstXP2p8AeJfztlHZtKPO8asknAkkDJe0u6QrgJ8D/kf5Ifw9cneEQbxX/vixpQ9LUGKtliFtzu6SNMsbraF5EzINUDRURj5Am88vlnxHxcMZ4HT0r6RfAvsDVRVVazr/lNyUNAQJA0vtJTwg5HEN6Ans8IqZKeh8wM1NsSH+b/waGkJL9kxHxTsb4AOtFxIO1F5Gqoj4UEU/kCB4RjwMDI+LtiDgHGJcjbmG6pOuAjwOTJS0P5Px8yrw2lHZe9YtJ53pgJnAjcFJE3F63/TJJ22WIf0ZRx/490jQZywFHZIhb82HgQElPki5AAiIicjUszpK0EnAlcL2kl4CnMsUGmCbpkiJ+2wU0In6bKf4+pIvDDyPiZUnvBQ7NFBvgSOBaYE1JFwLbAAfmCBwRvyFVo9ReP0FqD8plKqlqcQxpdsrTJX06IvbOeIwHJZ1GmloeUkJ+qEjIb3X9Y93yejElzQxJJwJ/J2+SPwjYBHgiIl6XtArpqSOXMq8NpZ1XrdpYvFxEvNbb5egpSWt3tj0icl6sa8fannTXcm2kdSNyxDynk80R+RrT3w/Miog3JO0AjCLVt7+cI35xjFWArUhJeEp0mBq9B/H+OyJOLOrsFzrpIuLrjcSvO87oiJjWYdtnI+L8HPGLeEOA/yTdsADcRmo3mAcMbeTcK/72nydVqXyT9Lf58+IpoZEyfygiHumioZvI1NGgTGWeVy2VCLo6yWoynmwrAZ9j4V49WeIXx9gY2LZ4eUtE3Jsh5goR8Yqkd3X2fkT8q9FjNIOkGcBo0ud/NekOeIOI+HiDcTu9SNQ0crGQ9MmI+L2kz3f2fkSc29n2Bo63Gu17rTydM35/I+mMiDhE0o2dvB0RsWOm45R+bShDqyWCTk+ymlwnm6TbSb0m7qeufjFj/InAF0mNcgB7AGdExCkNxv1DRHyiqHIK0t1uTUTE+xqJX3ec4cAppCoVgFuAiVEsQJQh/t0RsZmk/wb+HRGnSLonIjZtMG5nF4mabBeLuuMNAJaL1IslV8xPAj8CVifdWa8NPBwRG2Q8xjakXmZr0/5i1+O/H0mXRsQ+ku6n8yemfjHeosxrQ6nnVRkt0K3+Bdxdcvz7gGXrXi9Lxp4lTfh8rifVuy5VfB0IXJ8x/p3ABFKvmHWKbaX2tMpY9l8DKxT/pw8Bs4BDM8a/F1iFBb1XPgKclfl3eIS0FvlqxbFWAVZpMOZ7i3/X7uwrY9n3BpYvvv8u6WZr04zxS7s2lHletdQTQY2kYcC3Sd3a6h+Pcz3+fRN4DfgD7RttslStFHdFY2JBz57BwNSIyNaTSNIaLHxHl2VAmaQZEbHJ4rY1EH8k8GXgjoi4SNI6wD4RcUKm+J/rbHtEnJch9oyI2ETS/qSBcIcB0yPTHa+KVask3Uu6wL0j6d6I2DhH/OIYd0Ya/9DvSLovIkYVY1C+D5wEHJHr9ynz2lDmedWqvYYuBC4BdiNdMD4PvJAx/pukP6D/YcFjbJC6qOZwDnBn0f0V4FOkQVNZSDqBoqcH8HaxOUiDzHKYLekA4KLi9QQyrjwXqbvi1+tePwlkSQKFMXXfDwY+CtwNNJwIgEGSBpH+T38WEW9Jynk39rKk5Uj/lxdKep6MgxELN0o6iXQ3XX+xa7jBVdKepP/L1UhVl7Uec7kGhNb+3ncjVbf+UVKuwYJQ7rWhtPOqVZ8IpkfE5rXsX2ybGhFjFvez3Yz/BLBFNNiTZDHH2IwFvTJuiYh7MsZ+FBgVEbn6xneMvzapLnMs6SS4Hfh6ZGqwVJry4Qcs/MSXKxF3PN5KwMUR0XB/dklfJz2t3ku6GK0FXBAR2y7yB7sff1nSOIIBwP6kXjcXRkS2RFxmg6vSIlWfjJL6y0v6A2mA6U6kJ7J/A3flemIq89pQ5nnVqolgSkRsJWkycDLwHHBZRLw/U/zrgE9FxOs54tXFbUqvHknXAHtHP+1iK+lWUl//HwOfJNWbDoiInGM56o83iNQGkXPQXX38pSLDHD1Kky3eEBEfyVCsXiHptojYZvF79jj+UNIYlPsjYmYxBmWjyDfXUCnXhrK1atXQ9yWtSJo47BRS49w3M8afSxrwciPtH40b7SL2a+ATpMnI6jO0yFv19Dqp/H8iY/mb1VceGBIRf5KkSGMrjpI0nUwDdyT9ngXlH0B68rg0R+wi/m50mJSMNOK4IRHxtqR3JK0YEXMajdeRpAMi4gJJnU6QFxE/ynCYUgcjRhpE9ldgF0m7kJ62sySBQvZrQzPOq5ZMBLFglsg5pF4TuV1ZfGUVEZ8o/l0nd+wOriq+cqs9zk9b5F6Ne6PoejlTaV3sZ0kjOHP5Yd3380mzaubq+no6MJT0d/lLYC/grhyxC68B90u6nrq2gUxJeNni3+UzxOrKCqQblfrJ64IFXakb0knX7AuKMQYNdc2uU8a1ofTzqlWrhk7uZPMc0mRiv8t0jCGkaYofzRGvQ+xtSJN3zS0ahzYDfpKlLjBVH5wXEfs3GmsRx9g70lQKi9zWQPwxpJNjJeBYUj34iRExJUf8MtX1Wqn9uxxwTcY2gqYMWOuvJN0HjI2IucXrZUm9z7KNUyjr2lDmedWSk86RHrk3Ic05NJM0BcFw4CBJP2k0eDFoZwZpPhokbSIp5x32aaQ5VzYmVW/9lTS9bcMi4m1gbaX5XMpyeDe39UhETI2I1yJiVkR8ISL2zJEEVEzr29VXjrKTGich/f+uTpqb572ZYtcu+BeRqhenA7/OnQQkDZP0HUlnSDq79pUp9nBJV0h6vvi6XGkgVS5iQc8hiu/Vxb5LHrzca0Np51VLVg2RLvzbFBc9lCbIuoXUC+f+DPGPArYAbgKIiBlKs0jmMj8iQtJ4UhfDsyQdlDH+E8BtxR9offVBQ3W8knYlzeq4RoenshVIVSwNWdwJFRG7NxI/IpYvjnMsabKz80kXif3Jd7H+Q+3hJ2QAABXbSURBVNEL6SRSl9QgVRFloTT30rnA30hlX1PS53ONESn8jnQ+3UD7i2oO55DaymqT5B1QbNspY/zSumZTwrWh7PMKWjcRrEyqM641mC0LvKtoTMvRZfKtiJgjtbuRyDmV7auSDiedBNsV9eGDMsb/a/E1gLz1vc+R6jF3J92N1rxKnsb6scAzpDveO8l4J9fB7h26E56mNEArR2P0iUW33cuLroyDSZO15fJ/wM61aglJ65E+r80zHmNoRHw7Y7x6wyJNPV3zK0nfyBU8In6ktEpcrWv2F3J2zaaca0PZ51XLJoITSS33N5EuFtsBxxX1gTdkiP+gpM8AA4s+7V8n9enNZV/gM8BBEfEPSWuR7iCziIijIXWly9nNLdLEePdK+nVENDodcWfeQ7oznED6fP4IXBR1c+NnMldp5O/FpDv2CeQblHUHqc2HIiG8Ienu2rYMBtXXTUfEY0X315z+IOnjEZFjbY+Oyhs0ldrHHoyID5GexsqQ/drQhPOqNRuLAYr+wVsUL6dGxHMZYw8ljRzcmZRoJgPHRjElRF8naSzpcXi5iFiraIv4UkT8Z6b4pQ/4Upr7fgIpQR4dET/LGHsEaQWrWn/2W4FvRMTfGoj5HmAN4AJSEqvdMq4AnF5cnBpW1NW/UxwHUrXWwMi4Zq6kV0lP2W8WX9lG/6r8wYi/A76WK14n8Uu7NpR5XrVkIlB6LtsfeF9EHFPcUb8nInJ206sdayBpgricM0i+yoL+wkuTqoVei4gVM8W/k9Rt8aooZuyU9EBEbJgpfmkDvooEsBspCYwgdYM9OyKebTR2mYrePAeSps+eyoJE8Apwbq5+8sXn8xXqRqWT5vMvZRR5fyPpZtKC9XfRvn2sofalZij1vGrRRHAa6a5ox4hYX2nFoOsi3xQTvybNYfQ26aReAfhpRGSrvqk7lkgL2W8VEYdlinlnRGypuqmblXFiMi2Y4uP+KCbKq21rMO55wIakNQgujogHMhS3s+OUNt2v0mphlzcapzfV3WitExHHSlqTNHtowzdaShMIfo2F5/PPcqFWWohpIRHxlwbj1g9C7Cx+w+Uv67yC1u0+umVEfIWiES4iXiLdWecysngC+BRwDbAO8NmM8dtEciWwS8awz0jaGghJgyT9FwsGreTQbsCXpD3IM+DrAGBdYCJpXeda185XM3bvhNSz5CrSnP6rk9a67mx1qJ7YvOg1BICklZVx0jNJ20i6XtJjkp6ofeWKX/g5qermM8Xr14BTM8W+ktTj6RRSw3ftK4vigv8oaezJCsCjjSaBwg9J5XyS1EX4zOLrNVLHjBzKOq9acz0CUo+SgRRzgwPDKOZnzxT/QVJ1zW+A7Ytt92aMv2fd117A8aRBL7nir0qaofWfpMVLLqDB+eQ7xB9T/IEOJ11ALyc90fT630Y3yz+jO9t6GHuhv0MyzmFPCWsFdFXe+t8l198/cGfJ/7cHA08Dv2JBN9v/yBh/Wne29TB2x/Pqt7nOq1btNXQycAWwmqT/JV1Mv5sx/i9If0D3AjcXDVw570g/Wff9/OJY43MFjzQzYikji4s2k30j4r9Id0M5FwZvljKn0R4oaZko6uyVRqEukyk2wJyIuCZjvM68Vfw/B6QBZuTrPv1TSUcC15F5iuvCoaR1GmYDKK1NfTuQZUAcsKyk90XEE0X8dVgwNUdDImJq8W3286olE0FEXKg0CdlHSY1yn4q809qeGhFtAzskPU2GOY0knRCpf/Y1EZFtkrO6+KWv6RxprMaHF79nn/YfpKqJH7Og50quE+9C4E9asBD5F0h3pg3RgvWWS1sroE5nN1rfyxR7I1I1644sSC5RvM5hNqn/fc2rZFwrg9Sv/6aiOk6kxZ++1EjAprQ/FI8cLadoIF6T9g1OWU6G4j/5MuCcnAlGaWWyUaQVq3L1K6+PXz8PzdGkHghtIt+ay6eRukr+hvY9M7L0jOnvJI0DPla8vD4iJmeI2ez1lj/EghutP+U6D5TWIxgZEW/miFcXtzZj6iakZPM70sV1PGkZ2AMzHmsZoNYd+JFosMdWXQP3nqSxNLWuwROAf0ZEw4PKWjIRKE0RcCCpkaZtlaBcJ4Ok5YH9KLpvkR4rL44Gu5AWd3JfJNUD1g/0yr1KE8qw2PsiYnfWsBqRsS97GSQtqhteRMSxmY6zNrBuRNxQ9DsfGBGvLu7nuhm7rVpiUdsaPMZBEXFWh23HR4ZebZKuBA6JiOcbjdUh7pGLej+KQZYNxP/viDix+L7dRHCSjouI7zQSv4gzLSJGL25bj2K3aCJ4lLTYRNa7ii6OtT1pbpSVSE8Jx0bE4z2MtUxEvCHpdxGRrU2gi2PdXcZTRxF7m4i4bXHb+hpJ3+pk87LAQaQG14Z7aEj6InAIacqT9xeDhE6PiI82GruIv9D/a64uhnXxriatenZh8fpUYHBENDwfltJsAKNI3bLrq7b6dD//+s+94/9BrnNN0sPAbh3aH66OiPUbjd2SbQTAA6QLc9a7ipqioWw30hPBCFK3sQuBbUl93NfrYeja9AM5G557wyksPGVCZ9v6lIho66ZYPPVNJP0fX0y+LoxfIY14v7M45kxJqzUatKiq2QBYUWnd35oVaL8ATg6fBq6S9A5pta+XcySBwiLv3HtK0k8i4htd1bdnSDTq4vvOXvdU9vaHmlZNBD8A7pH0AOXcVcwEbgROioj6eUQuk7RdA3GXVpqnZOsOJzPQeB272o9YHlrX9z5L1ZPS1BVbA8PUfhWrFUjdefs8pWVCJ5F6VZ0LbBZpHEoub0TEmyomJZO0FItoCFwCHyStbrcS7XudvUqqbmyY2i+hejCpz/9twNGS3hUZllKNPH36O1Obxv2Hi9yr56KL7zt73bMDRFxbPEFma3+oadVEcC5wAmnK6ZyzgtaMii7W+22w582XSRegjiczZFilKYpplku0NKl9Yynaz2r6CqlnSZ9WtNHsCZxBqlosY03nv0j6DjBE0k7Af5IGrDUk0oJLv5M0NiLuaDReF2pLqKru392KryxLqarz6VXmZmgfewFKTTQbFzdWIv3f1t9kNfREJmnHiPhzJzeH75eUpRNGq7YRTI1M00l0EX8Y6S5rBO17JWVpDO2sMa4/kbR2pLWE+5WiquMN0tiNhdaMztFYX4wMPYj2k5L9MjKdiCpxeoxmk/JNr9KhDv/yiPh0jjI2g6SjI+LIMjthtGoi+BHphL6KEvpSS7qddIJNp25hjmhwDplm9DwoUzP6O9uiKa1V/GsWVIUcAOwfEbkWdkHSV0iNxS8Xr1cGJkTEz3Mdo8PxGu7hpvbzapXWY66/atVE0Fmf6pzdR2dExCY5YnWIW3rPgzKpiwm9akp8LO/zijEii0qSWdbMVSeTB+b+e+0sXq6La4fqjwGk2Vq3j4ixDcbt8tzq6zq0ty0kGlxZEFq0jSAiGh7luxhlLczRjJ4Hpanyhb4bPtGk47yo8qbHqBkoSbXqrKIXXa5JHcuaXmVRdfhZx+iUoOy2vZZ9IliR1A2t1oPnL8AxETGn65/qVtxaQ5ZYsDBHbcWgHL1u+vsTwaURsU9Xd7+57nr7uw4DyoYAS2UcUFbqwi7FMU4idV38RbHpS8AzEdHZOAzrB1o1EVxOGktQmzLhs8DGEbFQl8y+RNLbpCkZBAxhwehikQbs5F5yMCtJq0fEc8XFaCH9sQE5t7IHlDVD0eD9JdIUEwDXkxq8G17IXmmN5dOAd0fEhpJGkdaQzjZVd39VZkeAVk0EndVh5q4n3ZO0ClQAt0RaM6DSak8tks6PiFLWZ+jvJM2gGFBW13jZttBIhvilLuxSNkl/Ic0Q+osoYfW8/qzMjgAt2UYA/FvShyPiVkjTG5AWi8hC0s+BD7CgHvbLknaKtBhOlZU6IK5FlDWgrOZK0nrUvyfzGJomVf0NjYi7ap9PYX6GuK1gWETUdyH9laRv5Ajcqong/wHnFm0FAv5FmoQulx2B9esay84lLVZTdaUOiGsRpQwoqzMv6qZIz2xi8W+ZDd8vSno/C9Y62Av4e4nH609KWyejJauGaiStABAZF5Yv4v4B+EqtzruoE/9ZRHS8+FVSfx8QV6bOBpRFxJkZ43+GtJxnWQu7lErS+0gju7cGXiIt/XhARPytN8vVF3ToCABpeo8sHQFaKhFIOiAiLuiq322O/rbFcf5CWjautlj3GGAaMKc4Tr+ojy2T0prII2hfT31erxWoj5A0MSJ+urhtDcT/AalzxF+pW9gl1xia4hh7kqZwWY2UzMqYJn1ZYECu3lS2aK1WNVRbEq7sfreLmre+8iSdD7wfmMGCkdcBVD4RAJ8HOl70D+xkW0/tDbwvyp2C/UTgk5F31T8AlBZ1+TTFTUStrSAijsl9rP5G0onA90ntndeSpuv+ZkRcsMgf7E7sVnoiaLai6qn+jrfh2RdbgdK86SNzzZ/TCiRNAD5D6ml2S91bKwDzI+Jjnf7gkh+nlIVdOhzjtojYZvF79ij2taQn647Tt+SaBrzfqvV8lLQHqZ1mEnBzx5HkPdFSTwSSFtlIFhnW5C2OcwhwDDCP9Phdm42x4dkXW8QDpCX13Mi3wO2kz2NV2q9tEMC+GY+zEvCIpOwLu9T1BJsm6RJSD6X6Y+ToDDA8IsZliNOKatfr3YDfRMScDr2rGg7cKqYX/24DjAQuKV7vDTyU8TiHAhtGxIsZY7aSVYGHJN1FP1plqkxFx4KngLGSNiU9HexNagxtaLLCDkpZ2KVQ3xnidVKDd02uXmG3S9ooIu7PEKvV/EHSI6Sqof+nNAvyvByBW7JqSNIU4MMRMb94PYg06GurTPGvBfaMiNcXu3MFdTX5XJXnIipGzE4ovl4k3aT8V0R0Ogq7L5L01Yj4WcnHeIjU6+kJ0k1ErSHa05NAbXGgORHxttJ61ytExD8ajtuiieBRYGytzl5pmtwpEfHBTPE3Bc4hLTdYf8ebperJWo/SWge3AAdFsaa1pCciImt1ospb2KUp8115epJFK6s3XqtVDdUcT1qq8kbSHcV2wFEZ4/8C+DPlrYDWL3W4CLV7i74/w2PZ9gT2A24snigvpoQZZaNuFTppwcIuuY+Tm6TBpAGJHyCdV2fVnugtKbM3Xss9ERR//MNJs4JuWWy+M8fjU90xvLCF9UjRP348qYpoR9JJfEVEXFfiMXOtFTCfBRMhtnuLBhN90fj8FumpaVfgqYiYuOifqpYye+O1XCKAvJN4dRH/ONI86b+nfdWQu49atxVVlnsD++aafVQlLexSxC7tBqj+nC3mX7qrr0+73mySfkMaSZy9N16rVg3dLWlMREwtKf6E4t/D67a5+6gtkYh4iTSdwhkZw5a1sEvZaut6EBHzc3WLbDGl9cZr1SeCR0h1jU+xYH5/9zwwa4Ck70TEcd3Y7/CI+MESxq6txQHt1+Nw+1KhzN54rZoISul5oH6+uLy1vr6wsEszehdVlaR3k+Y2g1R9lmUE+YAcQfqaiHiquOj/m1RlU/tq1H513x/e4T2PhrS+4EzS3+ZbABFxH+3/bpvB9TolkLQPaaLLvYF9gDuLabob1pJtBJJ2Jw3jXx14nrS+6sPABo2G7uL7zl6b9Ya+sLBL61Uz9A3/A4ypPQUUI4tvAC5rNHBLPhEAx5L6Tj8WEeuQ1ladkiFudPF9Z6/NekNfWNjFN0XlGNChKmg2ma7hLflEALwVEbMlDZA0ICJulPSTDHE3lvQKRWNW8T3F68EZ4ps16iukXkgfkvQsxcIuTS7Dbxa/i/XAtZIms2CFsn2Bq3MEbtXG4huATwE/IHW5ep70SLV1rxbMrEnKWNhF0iks4snXU6yUQ9IHSI3/txXjRD5cvPUycGFE/LXhY7RSIqh9YKQh2P8mPTbtT2oj+GNETF/Ej5v1ex0Xdqltz7Gwi6TPL+r9iDi30WPYwoqlcQ/vOCOrpI2A43IskdtqiaD0D8ysL/PCLq1H0tSIGNPFe1lmUWi1NoJ3dzaPeUTcL2lE84tj1nSlL+xS9Fb5NmnNj7a2sZzrIls7Ky3ivSE5DtBqvYZK/8DM+rjbiyfgMl1I6o69DnA0aRqLsqZzsbQi3Bc7bpR0MAsW42pIq1UNXQT8OSLO7LD9YGCniMi5JKBZn9OMhV0kTY+IzSXdV4u7qOoLa0wxmvgK4E0WXPhHk9ab2CPHzMqtVjX0DeAKSfvTyQfWa6Uya55dm3CM2gRxf5e0G/Ac8K4mHLeSIuKfwNaSPgJsWGz+Y0T8OdcxWuqJoKbDB/Zgzg/MrC9q5sIukj5BWjdgTeAUYAXg6Ii4qozjWflaMhGYVU2zFnaRNJA0J/6Pc8e23uNEYNYCmrmwi6S7ImKLMmJb72i1NgKzqmrmwi63SfoZcAkL1hAgIu4u86BWHj8RmLWAZi7sIunGTjaHxxH0X04EZmYV56ohM1siko7obHuO+YysdzgRmNmSmlv3/WDgE6SRxtZPuWrIzBpSzHg6OSJ26O2yWM+02lxDZtZ8Q4HhvV0I6zlXDZnZEpF0PwsWqBkIDAPcPtCPuWrIzJaIpLXrXs4H/lnWdBbWHK4aMrMlEhFPkeYZ2jEingVWkrROLxfLGuAnAjNbIpKOJM3q+8GIWE/S6sBvImKbXi6a9ZCfCMxsSe0B7E7RjTQingOW79USWUOcCMxsSb0ZqSohACQt28vlsQY5EZjZkrpU0i9IbQNfBG4AzlzMz1gf5jYCM1tiknYCdiZNajc5Iq7v5SJZA5wIzMwqzgPKzKxbJL1KahcQCwaUQQlTXVtz+YnAzKzi/ERgZt0iaTDwZeADwH3A2R5R3Br8RGBm3SLpEtKSmLcAuwJPRcTE3i2V5eBEYGbdIun+iNio+H4p4K6I2KyXi2UZeByBmXXXW7VvXCXUWvxEYGbdIultFqxOJmAI8DruNdTvORGYmVWcq4bMzCrOicDMrOKcCMwWQdJ7JF0s6a+Spku6WtJ6kh7o7bKZ5eIBZWZdkCTgCuDciNiv2LYx8O5eLZhZZn4iMOvaR4C3IuL02oaIuBd4pvZa0ghJt0i6u/jautj+Xkk3S5oh6QFJ20oaKOlXxev7JX2z+b+S2cL8RGDWtQ2B6YvZ53lgp4iYJ2ld4CLSMo6fIU3P/L+SBgJDgU2ANSJiQwBJK5VXdLPucyIwa8wg4GeSNgHeBtYrtk8FzpY0CLgyImZIegJ4n6RTgD8C1/VKic06cNWQWdceBDZfzD7fBP4JbEx6ElgaICJuBrYDngV+JelzEfFSsd9NpMnbfllOsc2WjBOBWdf+DCwj6ZDaBkmjgDXr9lkR+HtEvAN8FhhY7Lc28M+IOJN0wd9M0qrAgIi4HPgu4Hl6rE9w1ZBZFyIiJO0B/ETSt4F5wN+Ab9Tt9nPgckmfA65lwRQMOwCHSnoLeA34HLAGcI6k2g3Y4aX/Embd4CkmzMwqzlVDZmYV50RgZlZxTgRmZhXnRGBmVnFOBGZmFedEYGZWcU4EZmYV9/8B86TYPHMWbOkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = pd.DataFrame({\"Class\": labels, \"Label\": \"Positive\", \"Value\": freq_pos})\n",
    "data = data.append([{\"Class\": labels[l], \"Label\": \"Negative\", \"Value\": v} for l,v in enumerate(freq_neg)], ignore_index=True)\n",
    "plt.xticks(rotation=90)\n",
    "f = sns.barplot(x=\"Class\", y=\"Value\", hue=\"Label\" ,data=data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2uvttCM8n5yY"
   },
   "source": [
    "As we see in the above plot, the contributions of positive cases is significantly lower than that of the negative ones. However, we want the contributions to be equal. One way of doing this is by multiplying each example from each class by a class-specific weight factor, $w_{pos}$ and $w_{neg}$, so that the overall contribution of each class is the same. \n",
    "\n",
    "To have this, we want \n",
    "\n",
    "$$w_{pos} \\times freq_{p} = w_{neg} \\times freq_{n},$$\n",
    "\n",
    "which we can do simply by taking \n",
    "\n",
    "$$w_{pos} = freq_{neg}$$\n",
    "$$w_{neg} = freq_{pos}$$\n",
    "\n",
    "This way, we will be balancing the contribution of positive and negative labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zs3_Rgwwn5yZ"
   },
   "outputs": [],
   "source": [
    "pos_weights = freq_neg\n",
    "neg_weights = freq_pos\n",
    "pos_contribution = freq_pos * pos_weights \n",
    "neg_contribution = freq_neg * neg_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ygNZmdyun5ya"
   },
   "source": [
    "Let's verify this by graphing the two contributions next to each other again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 380
    },
    "colab_type": "code",
    "id": "LPfSFrxjn5yb",
    "outputId": "a4b6354f-ab39-4623-d44b-90cfd9b28506",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAFaCAYAAAAJq7pFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3debxd89n//9c7gwzmIVoESYtWEFNiKGIq4taKWVQNvan2e7e9tXorerem9tYaWq2hZmqsKEXahqClNYUkGmIoiVASfhVBSAjC9fvjs3ayz845ydnZn3UG5/18PPbj7L32Wtda+5yz17XWZ1REYGZm1lrd2vsAzMysc3HiMDOzujhxmJlZXZw4zMysLk4cZmZWlx7tfQBtYbXVVosBAwa092GYmXUqEydOfD0i+tUu7xKJY8CAAUyYMKG9D8PMrFOR9K/mlruoyszM6uLEYWZmdXHiMDOzupRaxyFpOPBroDtweUT8vOb9YcCvgMHAyIi4uVi+M3Bu1aqfL96/TdJvgR2B2cV7R0bEpDI/h5l1bB9++CHTp09n3rx57X0onVLv3r3p378/PXv2bNX6pSUOSd2BC4HdgOnAeEmjI+LpqtVeAo4E/qd624i4F9isiLMKMBW4q2qV4ytJxsxs+vTpLL/88gwYMABJ7X04nUpEMGvWLKZPn87AgQNbtU2ZRVVbAVMjYlpEfADcCIyoXiEiXoyIJ4CPFxPnAOCOiHi3vEM1s85s3rx5rLrqqk4aS0ESq666al13a2UmjrWAl6teTy+W1Wsk8LuaZf8n6QlJ50rq1dxGko6RNEHShJkzZy7Fbs2sM3HSWHr1/u46dOW4pDWATYCxVYtPItV5DAVWAU5obtuIuDQihkTEkH79Fum/YmZmS6nMxDEDWLvqdf9iWT0OAm6NiA8rCyLi1UjeB64iFYmZmbWZ5ZZbrtXrnnrqqZxzzjmlxW8PZbaqGg+sL2kgKWGMBL5SZ4xDSHcYC0haIyJeVbq32gd4MsfBdgZbHn9Nq9a7dfmzl7jOOidPbvRw6taa42/NsUP7HL+ZJaXdcUTEfODbpGKmZ4CbIuIpSadL2htA0lBJ04EDgUskPVXZXtIA0h3L32pCXy9pMjAZWA34aVmfwcystf74xz+y9dZbs/nmm/PFL36Rf//73wvee/zxx9l2221Zf/31ueyyyxYsP/vssxk6dCiDBw/mlFNOaY/DXiql9uOIiDHAmJplJ1c9H08qwmpu2xdppjI9InbJe5RmZo3bfvvtGTduHJK4/PLLOeuss/jFL34BwBNPPMG4ceOYO3cum2++OXvttRdPPvkkU6ZM4dFHHyUi2Hvvvfn73//OsGHD2vmTLFmXGOTQzKxs06dP5+CDD+bVV1/lgw8+aNInYsSIEfTp04c+ffqw88478+ijj/LAAw9w1113sfnmmwMwZ84cpkyZ4sRhZtZVfOc73+G4445j77335r777uPUU09d8F5tc1dJRAQnnXQS3/jGN9r4SBvXoZvjmpl1FrNnz2attVLp+tVXX93kvdtvv5158+Yxa9Ys7rvvPoYOHcoee+zBlVdeyZw5cwCYMWMGr732Wpsf99LwHYeZWZ3effdd+vdfWD173HHHceqpp3LggQey8sors8suu/DCCy8seH/w4MHsvPPOvP766/z4xz9mzTXXZM011+SZZ55h2223BVIT3Ouuu47VV1+9zT9PvZw4zMzq9PHHzY+SNGLEiEWWVRdZ1Tr22GM59thjF1leuQvpqFxUZWZmdXHiMDOzujhxmJlZXZw4zMysLk4cZmZWFycOMzOri5vjmtknTmtHkm6tiWcfvsR1unfvziabbML8+fPZcMMNufrqq+nbt29d+zn66KM57rjjGDRoEGeccQY//OEPF7z3hS98gYceeqjuYy+D7zjMzDLo06cPkyZN4sknn2SZZZbh4osvrjvG5ZdfzqBBgwA444wzmrzXUZIGOHGYmWW3ww47MHXqVAB++ctfsvHGG7Pxxhvzq1/9CoC5c+ey1157semmm7LxxhszatQoAHbaaScmTJjAiSeeyHvvvcdmm23GoYceCiyc3GnkyJH8+c9/XrCvI488kptvvpmPPvqI448/fsEw7Zdccklpn89FVWZmGc2fP5877riD4cOHM3HiRK666ioeeeQRIoKtt96aHXfckWnTprHmmmsuSACzZ89uEuPnP/85F1xwAZMmTVok/sEHH8xNN93EXnvtxQcffMBf/vIXLrroIq644gpWXHFFxo8fz/vvv892223H7rvv3mSU3lx8x2FmlkHlDmHIkCGss846HHXUUTzwwAPsu+++LLvssiy33HLst99+3H///WyyySbcfffdnHDCCdx///2suOKKrd7Pnnvuyb333sv777/PHXfcwbBhw+jTpw933XUX11xzDZttthlbb701s2bNYsqUKaV8Vt9xmJllUKnjaI0NNtiAxx57jDFjxvCjH/2IXXfdlZNPPnnJGwK9e/dmp512YuzYsYwaNYqRI0cCEBGcf/757LHHHkv9GVrLdxxmZiXZYYcduO2223j33XeZO3cut956KzvssAOvvPIKffv25atf/SrHH388jz322CLb9uzZkw8//LDZuAcffDBXXXUV999/P8OHDwdgjz324KKLLlqwzXPPPcfcuXNL+Vy+4zCzT5zWNJ9tC1tssQVHHnkkW221FZCa226++eaMHTuW448/nm7dutGzZ08uuuiiRbY95phjGDx4MFtssQXXX399k/d23313DjvsMEaMGMEyyyyzIPaLL77IFltsQUTQr18/brvttlI+lyKilMAdyZAhQ2LChAntfRgNa23b9FuXP3uJ66xz8uRGD6durTn+1hw7tM/xW8f1zDPPsOGGG7b3YXRqzf0OJU2MiCG167qoyszM6lJq4pA0XNKzkqZKOrGZ94dJekzSfEkH1Lz3kaRJxWN01fKBkh4pYo6StEyZn8HMzJoqLXFI6g5cCOwJDAIOkTSoZrWXgCOBG5oJ8V5EbFY89q5afiZwbkSsB7wJHJX94M3MrEVl3nFsBUyNiGkR8QFwI9BkXsWIeDEingCan4exhiQBuwA3F4uuBvbJd8hmZrYkZSaOtYCXq15PL5a1Vm9JEySNk1RJDqsCb0XE/CXFlHRMsf2EmTNn1nvsZmbWgo7cHHfdiJgh6TPAXyVNBmYvaaOKiLgUuBRSq6qSjtHMrMspM3HMANauet2/WNYqETGj+DlN0n3A5sAtwEqSehR3HXXFNLOu4aXTN8karzXNvyVx3HHH8Ytf/AKAc845hzlz5nDqqadmPZaOMNx6mUVV44H1i1ZQywAjgdFL2AYASStL6lU8Xw3YDng6UqeTe4FKC6wjgNuzH7mZWZ169erFH/7wB15//fVS99MRhlsvLXEUdwTfBsYCzwA3RcRTkk6XtDeApKGSpgMHApdIeqrYfENggqTHSYni5xHxdPHeCcBxkqaS6jyuKOszmJm1Vo8ePTjmmGM499xzF3lv5syZ7L///gwdOpShQ4fy4IMPLli+2267sdFGG3H00Uez7rrrLkg8++yzD1tuuSUbbbQRl156KUCHGW691DqOiBgDjKlZdnLV8/Gk4qba7R4Cmr3XjIhppBZbZmYdyre+9S0GDx7MD37wgybLjz32WL73ve+x/fbb89JLL7HHHnvwzDPPcNppp7HLLrtw0kknceedd3LFFQuvg6+88kpWWWUV3nvvPYYOHcr+++/fYYZb78iV42ZmncoKK6zA4YcfznnnnUefPn0WLL/nnnt4+umnF7x+++23mTNnDg888AC33norAMOHD2fllVdesM5555234L2XX36ZKVOmsOqqq7a47z333JNjjz2W999/nzvvvLPJcOtPPPEEN9+cejHMnj2bKVOmOHGYmXUU3/3ud9liiy342te+tmDZxx9/zLhx4+jdu3erYtx3333cc889PPzww/Tt25eddtqJefPmLXabthxu3WNVmZlltMoqq3DQQQc1KXbafffdOf/88xe8rhQ1bbfddtx0000A3HXXXbz55ptAuitYeeWV6du3L//85z8ZN27cgm07wnDrvuMws0+c9h49+fvf/z4XXHDBgtfnnXfegvqP+fPnM2zYMC6++GJOOeUUDjnkEK699lq23XZbPv3pT7P88sszfPhwLr74YjbccEM+97nPsc022yyI1RGGW3fiMDPLYM6cOQuef+pTn+Ldd99d8Hq11VZj1KhRi2yz4oorMnbsWHr06MHDDz/M+PHj6dWrFwB33HFHs/s588wzOfPMM5vdb8+ePXnjjTearN+tWzfOOOOMRZrxNsKJw8ysnbz00kscdNBBfPzxxyyzzDJcdtll7X1IreLEYWbWTtZff33+8Y9/tPdh1M2V42b2idAVZjMtS72/OycOM+v0evfuzaxZs5w8lkJEMGvWrFY3FQYXVZnZJ0D//v2ZPn06nkJh6fTu3Zv+/RcZxKNFThxm1un17NmzoZ7QVh8XVZmZWV2cOMzMrC5OHGZmVhcnDjMzq4sTh5mZ1cWJw8zM6uLEYWZmdXHiMDOzujhxmJlZXZw4zMysLqUmDknDJT0raaqkE5t5f5ikxyTNl3RA1fLNJD0s6SlJT0g6uOq930p6QdKk4rFZmZ/BzMyaKm2sKkndgQuB3YDpwHhJoyPi6arVXgKOBP6nZvN3gcMjYoqkNYGJksZGxFvF+8dHxM1lHbuZmbWszEEOtwKmRsQ0AEk3AiOABYkjIl4s3vu4esOIeK7q+SuSXgP6AW9hZmbtqsyiqrWAl6teTy+W1UXSVsAywPNVi/+vKMI6V1KvFrY7RtIESRM81LKZWT4dunJc0hrAtcDXIqJyV3IS8HlgKLAKcEJz20bEpRExJCKG9OvXr02O18ysKygzccwA1q563b9Y1iqSVgD+DPxvRIyrLI+IVyN5H7iKVCRmZmZtpMzEMR5YX9JAScsAI4HRrdmwWP9W4JraSvDiLgRJAvYBnsx61GZmtlilJY6ImA98GxgLPAPcFBFPSTpd0t4AkoZKmg4cCFwi6ali84OAYcCRzTS7vV7SZGAysBrw07I+g5mZLarUqWMjYgwwpmbZyVXPx5OKsGq3uw64roWYu2Q+TDMzq0OHrhw3M7OOx4nDzMzq4sRhZmZ1ceIwM7O6OHGYmVldnDjMzKwuThxmZlYXJw4zM6uLE4eZmdXFicPMzOrixGFmZnVx4jAzs7o4cZiZWV2cOMzMrC5OHGZmVhcnDjMzq4sTh5mZ1cWJw8zM6uLEYWZmdXHiMDOzujhxmJlZXUpNHJKGS3pW0lRJJzbz/jBJj0maL+mAmveOkDSleBxRtXxLSZOLmOdJUpmfwczMmiotcUjqDlwI7AkMAg6RNKhmtZeAI4EbarZdBTgF2BrYCjhF0srF2xcBXwfWLx7DS/oIZmbWjDLvOLYCpkbEtIj4ALgRGFG9QkS8GBFPAB/XbLsHcHdEvBERbwJ3A8MlrQGsEBHjIiKAa4B9SvwMZmZWY4mJQ9KnJF0h6Y7i9SBJR7Ui9lrAy1WvpxfLWqOlbdcqni8xpqRjJE2QNGHmzJmt3K2ZmS1Jj1as81vgKuB/i9fPAaOAK0o6piwi4lLgUoAhQ4ZEW+xzy+OvadV6ty5/9hLXWefkyY0eTpfTmt9/a3730D6//85+/NZ1tKaoarWIuImiOCki5gMftWK7GcDaVa/7F8tao6VtZxTPlyammZll0JrEMVfSqkAASNoGmN2K7cYD60saKGkZYCQwupXHNRbYXdLKRaX47sDYiHgVeFvSNkVrqsOB21sZ08zMMmhNUdVxpBP+ZyU9CPQDDlj8JunORNK3SUmgO3BlRDwl6XRgQkSMljQUuBVYGfiypNMiYqOIeEPST0jJB+D0iHijeP5fpOKzPsAdxcPMzNrIEhNHRDwmaUfgc4CAZyPiw9YEj4gxwJiaZSdXPR9P06Kn6vWuBK5sZvkEYOPW7N/MzPJbYuKQdHjNoi0kERGtqwk2M7NPlNYUVQ2tet4b2BV4jNSHwszMupjWFFV9p/q1pJVInfnMzKwLWpqe43OBgbkPxMzMOofW1HH8kaIpLinRDAJuKvOgzMys42pNHcc5Vc/nA/+KiOktrWxmZp9sranj+FtbHIiZmXUOLSYOSe+wsIiqyVtARMQKpR2VmZl1WC0mjohYvi0PxMzMOofW1HEAIGl1Uj8OACLipVKOyMzMOrTWzMext6QpwAvA34AX8fhQZmZdVmv6cfwE2AZ4LiIGknqOjyv1qMzMrMNqTeL4MCJmAd0kdYuIe4EhJR+XmZl1UK2p43hL0nLA/cD1kl4j9R43M7MuqMU7DkkXStoeGAG8C3wXuBN4Hvhy2xyemZl1NIu743gOOBtYgzTEyO8i4uo2OSozM+uwWrzjiIhfR8S2wI7ALOBKSf+UdLKkDdrsCM3MrENZYuV4RPwrIs6MiM2BQ4B9gWdKPzIzM+uQWtOPo4ekL0u6ntR/41lgv9KPzMzMOqTFjVW1G+kO4z+AR0mTNx0TEW5RZWbWhS2ucvwk4Abg+xHxZhsdj5mZdXCLqxzfJSIubyRpSBou6VlJUyWd2Mz7vSSNKt5/RNKAYvmhkiZVPT6WtFnx3n1FzMp7qy/t8ZmZWf2WZurYVpHUHbgQ2JM0a+AhkgbVrHYU8GZErAecC5wJEBHXR8RmEbEZcBjwQkRMqtru0Mr7EfFaWZ/BzMwWVVriALYCpkbEtIj4gFRHMqJmnRFApW/IzcCuklSzziHFtmZm1gGUmTjWAl6uej29WNbsOhExH5gNrFqzzsHA72qWXVUUU/24mUQDgKRjJE2QNGHmzJlL+xnMzKxGmYmjYZK2Bt6NiCerFh8aEZsAOxSPw5rbNiIujYghETGkX79+bXC0ZmZdQ5mJYwawdtXr/sWyZteR1ANYkdRLvWIkNXcbETGj+PkOqdXXVlmP2szMFqvMxDEeWF/SQEnLkJLA6Jp1RgNHFM8PAP4aEQEgqRtwEFX1G0VnxNWK5z2BLwFPYmZmbabVU8fWKyLmS/o2MBboDlwZEU9JOh2YEBGjgSuAayVNBd4gJZeKYcDLETGtalkvYGyRNLoD9wCXlfUZzMxsUaUlDoCIGAOMqVl2ctXzecCBLWx7H2nmweplc4Etsx+omZm1WoeuHDczs47HicPMzOrixGFmZnVx4jAzs7o4cZiZWV2cOMzMrC5OHGZmVhcnDjMzq4sTh5mZ1cWJw8zM6uLEYWZmdXHiMDOzujhxmJlZXZw4zMysLk4cZmZWFycOMzOrixOHmZnVxYnDzMzq4sRhZmZ1ceIwM7O6OHGYmVldSk0ckoZLelbSVEknNvN+L0mjivcfkTSgWD5A0nuSJhWPi6u22VLS5GKb8ySpzM9gZmZNlZY4JHUHLgT2BAYBh0gaVLPaUcCbEbEecC5wZtV7z0fEZsXjm1XLLwK+DqxfPIaX9RnMzGxRZd5xbAVMjYhpEfEBcCMwomadEcDVxfObgV0XdwchaQ1ghYgYFxEBXAPsk//QzcysJWUmjrWAl6teTy+WNbtORMwHZgOrFu8NlPQPSX+TtEPV+tOXEBMAScdImiBpwsyZMxv7JGZmtkBHrRx/FVgnIjYHjgNukLRCPQEi4tKIGBIRQ/r161fKQZqZdUVlJo4ZwNpVr/sXy5pdR1IPYEVgVkS8HxGzACJiIvA8sEGxfv8lxDQzsxKVmTjGA+tLGihpGWAkMLpmndHAEcXzA4C/RkRI6ldUriPpM6RK8GkR8SrwtqRtirqQw4HbS/wMZmZWo0dZgSNivqRvA2OB7sCVEfGUpNOBCRExGrgCuFbSVOANUnIBGAacLulD4GPgmxHxRvHefwG/BfoAdxQPMzNrI6UlDoCIGAOMqVl2ctXzecCBzWx3C3BLCzEnABvnPVIzM2utjlo5bmZmHZQTh5mZ1cWJw8zM6uLEYWZmdXHiMDOzujhxmJlZXZw4zMysLk4cZmZWFycOMzOrixOHmZnVxYnDzMzq4sRhZmZ1ceIwM7O6OHGYmVldnDjMzKwuThxmZlYXJw4zM6uLE4eZmdXFicPMzOrixGFmZnVx4jAzs7qUmjgkDZf0rKSpkk5s5v1ekkYV7z8iaUCxfDdJEyVNLn7uUrXNfUXMScVj9TI/g5mZNdWjrMCSugMXArsB04HxkkZHxNNVqx0FvBkR60kaCZwJHAy8Dnw5Il6RtDEwFlirartDI2JCWcduZmYtK/OOYytgakRMi4gPgBuBETXrjACuLp7fDOwqSRHxj4h4pVj+FNBHUq8Sj9XMzFqpzMSxFvBy1evpNL1raLJORMwHZgOr1qyzP/BYRLxfteyqopjqx5LU3M4lHSNpgqQJM2fObORzmJlZlQ5dOS5pI1Lx1TeqFh8aEZsAOxSPw5rbNiIujYghETGkX79+5R+smVkXUWbimAGsXfW6f7Gs2XUk9QBWBGYVr/sDtwKHR8TzlQ0iYkbx8x3gBlKRmJmZtZEyE8d4YH1JAyUtA4wERtesMxo4onh+APDXiAhJKwF/Bk6MiAcrK0vqIWm14nlP4EvAkyV+BjMzq1Fa4ijqLL5NahH1DHBTRDwl6XRJexerXQGsKmkqcBxQabL7bWA94OSaZre9gLGSngAmke5YLivrM5iZ2aJKa44LEBFjgDE1y06uej4POLCZ7X4K/LSFsFvmPEYzM6tPh64cNzOzjseJw8zM6uLEYWZmdXHiMDOzujhxmJlZXZw4zMysLk4cZmZWFycOMzOrixOHmZnVxYnDzMzq4sRhZmZ1ceIwM7O6OHGYmVldnDjMzKwuThxmZlYXJw4zM6uLE4eZmdXFicPMzOrixGFmZnVx4jAzs7o4cZiZWV1KTRyShkt6VtJUSSc2834vSaOK9x+RNKDqvZOK5c9K2qO1Mc3MrFylJQ5J3YELgT2BQcAhkgbVrHYU8GZErAecC5xZbDsIGAlsBAwHfiOpeytjmplZicq849gKmBoR0yLiA+BGYETNOiOAq4vnNwO7SlKx/MaIeD8iXgCmFvFaE9PMzErUo8TYawEvV72eDmzd0joRMV/SbGDVYvm4mm3XKp4vKSYAko4BjilezpH0bB3Hvhrweh3r12Xd1sQ/RR02PiX+flp17LDUx192fDrC/w502ON3/E4Xf93mFpaZONpVRFwKXLo020qaEBFDMh+S47dzbMd3fMfPE7/MoqoZwNpVr/sXy5pdR1IPYEVg1mK2bU1MMzMrUZmJYzywvqSBkpYhVXaPrllnNHBE8fwA4K8REcXykUWrq4HA+sCjrYxpZmYlKq2oqqiz+DYwFugOXBkRT0k6HZgQEaOBK4BrJU0F3iAlAor1bgKeBuYD34qIjwCai1nC4S9VEZfjd/jYju/4jp+B0gW+mZlZ67jnuJmZ1cWJw8zM6uLEYWZmdXHiYMHwKGYdmqSVJQ1u7+Mwc+U4IGkacAtwVUQ8XdI+epPG5toI6F1ZHhH/mSl+L2B/YABVreUi4vQc8Yt9rExqGl19/H/PFb8skg4E7oyIdyT9CNgC+GlEPJYpfl/g+8A6EfF1SesDn4uIP2WIfR+wN+lvOhF4DXgwIo5rNHbVPgbV/t9L2iki7ssUf4tmFs8G/hUR83Pso0ySjgWuAt4BLgc2B06MiLsyxS/13FAG33EkmwLPAZdLGifpGEkrZN7HtcCngT2Av5E6L76TMf7tpHG75gNzqx5ZSDoa+DupKfRpxc9TM8XeT9IUSbMlvS3pHUlv54hd+HGRNLYHvkhqBn5RxvhXAe8D2xavZwA/zRR7xYh4G9gPuCYitiZ9hpxuknSCkj6Szgd+ljH+b0hDCF0KXAY8DPweeFbS7o0Gl7SNpPGS5kj6QNJHmf9//rP4G+wOrAwcBvw8Y/zSzg2lfbciwo+qB7Aj6Ys/lzQA43qZ4v6j+PlE8bMnMC7jcT9Z8u9lMulqaFLx+vPAHzLFngpsWOKxV373PwO+Ur0sU/wJtTGBxzP+3tcA7gKGVv8PZTz+ZYELSCf0J4GTgG4Z4/8B2Kjq9SDSoKafqfw/Nfr7B9YD/kHq3/U14GcZj7/ynf01sG8J/z+lnRvK+m75joNUxyFpb0m3Ar8CfkH6p/4jMCbTbj4sfr4laWPS8CqrZ4oN8JCkTTLGqzUvIuZBKhaLiH8Cn8sU+98R8UymWM2ZIekS4GBgTFGsl/N//wNJfYAAkPRZ0h1IDqeT7u6mRsR4SZ8BpmSKXfEh8B7Qh3Rx8EJEfJwx/gZR1VE3UrHY5yNiWq4dRMRUoHtEfBQRV5GmY8hloqS7gP8AxkpaHsj5+ynz3FDKd+sTO8hhnaYA9wJnR8RDVctvljQs0z4uLeoIfkwaJmU54ORMsQG2B46U9ALppCUgIiJXZep0SSsBtwF3S3oT+Fem2BMkjSpiLzjhRsQfMsU/iHQiOSci3pK0BnB8ptgApwB3AmtLuh7YDjgyR+CI+D2pWKfyehqpLiun8aSizqGk0VMvlrR/RByYKf5Tki4iTYMAKYE/XSTwD1verNXeLYYgmiTpLOBV8l4YHAVsBkyLiHclrUq6q8mlzHNDKd8tV44DkpaLiDntfRyNkNTs8McRkevkXr2vHUlXRXdGmhel0XhXNbM4Il/Dgc8C0yPifUk7AYNJ9QVv5Yhf7GNVYBtSwh4XEQ0NjS3pBxFxVlHfsMiXNCL+u5H4NfsaEhETapYdFhHXZorfB/gv0sUNwIOkeo95QN9Gv3vF//5rpCKe75H+N39T3IU0EvfzEfHPFir3iUyNK8pU1nerSyeOlr6UFZm/nCsBh7Noq6ec+9gU2KF4eX9EPJ4h5goR8bakVZp7PyLeaHQfZZM0CRhC+t2PIV1dbxQR/9Fg3GZPKBWNnFgkfTki/ijpiObej4irm1veCEmr07RVz0u599GZSLo0Io6RdG8zb0dE7JJpP6WfG3Lr6omj2S9lRc4vp6SHSC1LJlNVPpprH0WTwa+TKiIB9gUujYjzG4z7p4j4UlEEFqQr6oqIiM80Er/YR3/gfFIRD8D9wLERMb3R2EX8xyJiC0k/AN6LiPMl/SMiNm8wbnMnlIpsJ5aq/XUDlovUwidn3C8DvwTWJF25rws8ExEbZYq/HakF3ro0PTE29L8j6aaIOEjSZM8ZrmcAABtTSURBVJq/K+sUfV7KPDeU9t3KXdvuR4utGx4rOf4TwLJVr5clc+ubEo/9blKZcY/icSRwd8b4jwCHkFoMDSyWldoKLeOx3wCsUPw9nybNenl85n08Tpp5s9K6Z2fgiozx/wnsSarwXbXyyBB3jeLnus09Mh7/gcDyxfMfkS7ONs8Yv7RzQ1nfrS59x1EhqR9wAqmZYPWterYrRknfA+YAf6JpJVWWop7iqmtoLGz51BsYHxHZWlpJWotFrxob7gAoaVJEbLakZQ3EHwR8E3g4In6nNMfLQRFxZqb4hze3PCKuyRB7UkRsJulQUsfFE4GJkfFqWsWscJIeJ50QP5b0eERsmin+I5H6n3RKkp6IiMFFP6CfAmcDJ+f6TGWeG8r6brlVVXI9MArYi3SCOQKYmXkfH5D+4f6XhbfVQWr2m8NVwCNFk2KAfUgd3bKQdCZFaxjgo2JxkDoFNmqWpK8CvyteH0KaCTKLSM0//7vq9QtAlqRRGFr1vDewK/AY0HDiAHpK6kn6e14QER9Kyn2195ak5Uh/y+slvUbGzqPAvZLOJl2pV58Yc/Xc34/091ydVJRaaVGYqxNv5f99L1Lx758l5ergCeWeG0r5bvmOA5A0MSK2rFxZFMvGR8TQJW1bxz6mAVtFg61tlrCPLVjYcuX+iPhHxtjPAoMjIlf/hOrY65LKYbclfWEeAv47MlXOKg0B8jMWvaPMlbRr97cScGNENNyXQNJ/k+6GHyeduNYBrouIHRa7YX37WJbUj6MbcCipVdL1EZElebdB5fJU4MtRUl8gSX8idQrejXTX9x7waMY7stLODWV9t5w4AEnjImIbSWOB84BXgJsj4rMZ93EXsE9EvJsrZhG3TVo9SboDODA6YbNlSQ+Q+lqcC3yZVObbLSJy9qOp3l9PUh1Krg6StfF7RKYxnpQG+LwnInbOEa89SHowIrZb8ppLHb8vqR/Q5IiYUvQD2iTyjVVVyrmhTC6qSn4qaUXSQHXnkyojv5d5H3NJHZTupenteqNN7m4AvkQaAK/6KkDkLQp7l3T8fyHT8bdhX4U+EfEXSYrUr+VUSRPJ1MlK0h9ZePzdSHc2N+WIXcTfi5oB8Eg9yhsWER9J+ljSihExO0fMCklfjYjrJDU7IGNE/DLTrkrtQBqp09/zwB6S9iDdzWdJGoXs54ayv1tOHEAsHMV0NqlFSRluKx5ZRcSXip8Dc8euMbp45FQpWpiw2LUa937RlHWK0pz1M0i9c3M5p+r5fNKor7maEl8M9CX9X14OHAA8miN2lTnAZEl3U1W3kSFxL1v8XL7BOEuyAunCpnrAxGBh0/SGNNPU/bqij0dDTd2rlHFuKPW75aIqQNJ5zSyeTRq87vaM++lDGnr72Vwxq2JvRxowbm5RGbYF8Ksc9QRFccY1EXFoo7FaiH9gpKE1FrusgfhDSV+klYCfkMrwz4qIcTnil6mqRU/l53LAHZnrONqsk2FnJOkJYNuImFu8XpbUQi9ny7ZSzg1lfbc8yGHSmzQWzZTiMZg0tPFRkn6VYwdFJ6tJpDGNkLSZpJxX8BeRxuzZlFTk9jxpuOaGRcRHwLpK4wGV4aRWLlsqETE+IuZExPSI+FpE7JcjaagYorqlR45jJ1XEQvrbrkka22mNTLGBBQnid6TizonADTmThqR+kn4o6VJJV1YeGeP3l3SrpNeKxy1KHd+y7YKFLasonquFdesPXu65oZTvlouqksHAdsUJEqUB2e4ntVCanGkfpwJbAfcBRMQkpZFOc5kfESFpBKnZ5hWSjsoYfxrwYPEPXV2csdTl1JL2JI04ulbNXd8KpCKfhizpyxcRezcSPyKWL/bzE9LAeteSTiiHku/k/qeildbZpCa+QSqyykZp/K6rgRdJx7+2pCNy9NEp3E76Pt1D0xNwLleR6voqgzJ+tVi2W8b4pTV1p4RzQ9nfLSeOZGVSmXelcnBZYJWi4jBX89MPI2K21ORCJefQzO9IOon0pRlWlOn3zBj/+eLRjXxl1q+QymD3Jl3pVrxDnsYJ2wIvk66mHyHjVWKNvWuaZl6k1JkuR+X7WUUT6FuKZqG9SYMD5vQLYPdKMYmkDUi/sy0zxe8bESdkitWcfpGGUq/4raTv5goeEb9Umomx0tT9azmbulPOuaHU75YTR3IWqVXDfaSTyzDgjKIs855M+3hK0leA7kW/gv8mtanO5WDgK8BREfH/SVqHdJWaRUScBqlpYq5mg5EGYXxc0g0RkWN47VqfJl11HkL63fwZ+F1UzQ2RyVylnt03ku4IDiFfB7qHSfVVFAnkfUmPVZZl0rO6bD0iniuaFOfyJ0n/ERG55rapVVoH0qJ+76mI+Dzpjq8M2c8NZX+3XDleKNpmb1W8HB8Rr2SO35fUM3R3UnIaC/wkiiFCOjpJ25Juz5eLiHWKupRvRMR/ZYhdegc9pbkfDiEl09Mi4oKMsQeQZoer9CV4APhuRLzYQMxPA2sB15GSXuVydAXg4uJElkVR3/BxsS9IRW3dI9+w9u+Q7uI/KB5Ze3ar/A6ktwPfyRWvmfilnRvK+m45cQBK94iHAp+JiNOLq/VPR0TuZo+V/XUnDUiYbZTT4stZ+WMuQyqmmhMRK2aK/wipKejoKEaVlfRkRGycIXZpHfSKhLEXKWkMIDUpvjIiZjQau0xFS6cjScPBj2dh4ngbuDpXH4ViX72Ab1E16gBpPovsowR0RpL+DmxOagZdXb/XUB1ZWyjru+XEwYLK8I+BXSJiQ6XZuO6KvEOO3EAaB+sj0olgBeDXEZGtOKlqXwJGANtExImZYj4SEVurajhyZRoITwuHfJkcxaCMlWUNxr0G2Jg0B8eNEfFko8fawn5KGxZeaSa+WxqN056qLswGRsRPJK1NGtk2y4WZ0qCV32HR+SyynNiVJi5bRET8rcG41R1Hm4vf8PGX9d1yc9xk64j4FkWlY0S8Sbpqz2lQcYexD3AHMBA4LPM+gFQGEBG3AXtkDPuypC8AIamnpP9hYSejRjXpoCdpX/J00PsqsD5wLGlO9kpT2XcyNpeF1OpmNGk+izVJc9U3N/Pa0tiyaFUFgKSVlXeAPSRtJ+luSc9JmlZ5ZNzFb0jFSF8pXs8BLswY/zZSi7DzSRX9lUcWRYJ4ltT/ZwXg2UaTRuEc0nG+QGp2fVnxmENqiJJDOd+tKGkc+M70ILW46U4xLj7Qj2Jugoz7eIpUfPR7YMdi2eMZ4+9X9TgA+Dmpk1Ku+KuRRhH+N2myn+vIMKdCEXto8c/cn3TCvYV0t9Tu/xutPP5JrVm2lLEX+T8k8/wNlDRfRu3xVn+WzP/7j5T89z0aeAn4LQubLf9nxvgTWrNsKWPXfrf+kOO75VZVyXnArcDqkv6PdOL9UeZ9XEL6h3sc+HtRoZfzqvfLVc/nF/sakSt4pJE7s/ccL+p7Do6I/yFdaX0t9z7aQJnDwneX1CuK+galHsa9MsWumB0Rd2SOWe3D4u8ckDoEkrcp+q8lnQLcRQnDtgPHk+YpmQWgNL/8Q0CuTozLSvpMREwr4g9k4XAtDYmI8cXTrN8tJw4gIq5XGvRuV1Il5D6Rf4jmCyNiQUccSS+RYVwsSWdGaiN/R0RkG1ivKn6p87JH6iuz/ZLX7ND+k1RMci4LW/Xk+pJeD/xFUqXo62ukq96GaeGc6aXOl0HzF2Y/zhQbYBNSse8uLExIUbzOYRap/0PFO2ScL4bUr+K+onhQpMnSvtFIwLLrT1w5XigqxNemaeVatnbbxT/FzcBVOZOS0sx/g0mzwuVs21+JXz2O0WmkFhoLRJ55kS8iNT39PU1brWRrOdSZSRoOfLF4eXdEjM0Ut83mTJf0eRZemP0l83dgKqkO8YNcMYu4lVF9NyMlp9tJJ+MRpGmZj8y4r15ApYn1P6PBFm1VFfr7kfozVZpaHwL8OyIa6gToxMGCISOOJFVILZiBK/MXZ3lgJEVzONJt7o3RYJPc4krx66RyzOqOeblnQaO6RVVOVVfT1SIy9SMoi6TFNWmMiPhJpv2sC6wfEfcUbf67R8Q7S9qujvgLikkWt6yB+EdFxBU1y34e+Vr83QYcExGv5YhXFfeUxb0fRafYBuL/ICLOKp43GXhQ0hkR8cNG4hdxJkTEkCUtqzuuEwcozW63Se4rlsXsb0fS2Dorke5CfhIRU5cyVq+IeF/S7RGRrU6jhX09VtJdzXYR8eCSlnU0kr7fzOJlgaNIlcsNt16R9HXgGNIQOJ8tOnRdHBG7Nhq7ah+L/F1zNNmsijWGNKPg9cXrC4HeEZFlLDWlER8Gk5q5Vxe1deh+FtW/99q/Qa7vmqRngL1q6k/GRMSGjcR1HUfyJOkknvWKpVpRObgX6Y5jAKkZ3vXADqR+BhssZejKkBQ5K9rb2vksOoRGc8s6lIhY0OSzuKM8lvT3vZF8zUG/RRrR4JFin1MkrZ4jcFF8tBGwotK83RUr0HTSqEbtD4yW9DFpJr23ciWNwmLvDJaWpF9FxHdbqi/IkJjUwvPmXi+t7PUn4MRR8TPgH5KepLwrlinAvcDZEVE9Ds3NkoY1EHcZpXFuvlDz5QcarydQ0x7pfav6PzRcFKY0jMkXgH5qOkvcCqTm0R2e0pS9x5FanF0NbBGpH1Au70fEByoGwJPUg8VUetbpc6TZI1eiaau8d0jFnw1R0+mMjyb1t3gQOE3SKpFpWuPI06eiOZVpCc5Z7FpLL1p43tzrpdtBxJ3FXWq2+hNw4qi4GjiTNIR6zmaC1QZHC/N1N9gy6Zukk1btlx8yzIIWxdDhJVmGVDfTg6Yj7r5NannToRX1S/sBl5KKOsuYj/1vkn4I9JG0G/BfpA6GDYs0SdntkraNiIdzxKxRmc5YVT/3Kh7ZpjVW88PtzM1QvzcTSk1MmxYXYiL9fasvyhq645O0S0T8tZmLyc9KavyC0nUcIGl8ZBxepIV99CNdxQ2gacutXAPJLVIB2VlIWjfSXOCdSlH08j6p38wi873naJhQ9Po9iqYD4F0eGb+4KnHIlLYm5Rtup6YO4paI2D/HMbYFSadFxCllNTxx4gAk/ZJ0AhhNOe3YkfQQ6Qs5karJbKLBcYjaomVGWcpua26tozTX+A0sLJr5KnBoRGSZCEnSt0iV428Vr1cGDomI3+SI38I+G24BqKbjspXSorCzcuKgxfbsuZvjToqIzXLFq4pbesuMsqiFweMqSiwi6PCK/jmLS6o557teZLDKnP+vzcXKeSKuKY7pRhpReMeI2LbBuC1+tzq6mjrDRUQDM3eC6zgAiIiGe3C3QlmT2bRFy4xSdOXE0ApfasN9va7yhkyBNGyKKsVrRQvDnIOIljXczuLqILL2kSpBmXWTvuMAkLQiqUlfpXXT34DTI2J2y1u1Onal4k4snMymMiNXw/98nfyO46aIOKilq+ucV9WdWU0HwD5Aj8wdAMueCOlsUjPQS4pF3wBejojm+sFYJ+DEQar4IvXlqAyfcRiwaUQs0ry1o5H0EWmYDgF9WNh7XKROVjmnAM1K0poR8Upx4lpEZ6wwz60tOgCWrajg/wZpyBGAu0kV/B+1vFVd8TcALgI+FREbSxpMmgc+6/DznVFZDR+cOGixDDZ7nURRFrs96aru/khzZnRZlTsiSddGRClzk3R2kiZRdACsqqhdMClPpn2UOhFS2ST9jTSC7SWReXbKzq6shg+u40jek7R9RDwAabgL0sQq2Uj6DbAeC8uRvylpt0gTSHVVpXZe/IQoswNgxW2k+eT/SMZ+TG1YFNk3Ih6t/I4K8zPF7uz6RUR1k9zfSvpuo0GdOJL/B1xd1HUIeIM06GFOuwAbVlUQXk2a3KkrK7Xz4idEaR0Aq8yLqiH/Mzq2+Fl2Rf/rkj7Lwvk+DgBeLXmfnUUpc8W4qKqKpBUAosERa1uI/SfgW5Vy+6Jc/4KIqD1hdjmdufNi2ZrrABgRl2Xex1dIU+yWNRFSqSR9htR7/wvAm6SpWL8aES+253F1BDUNHyAN+dJww4cunTgkfTUirmupzXOjbZ1r9vU30jSOjxaLhgITgNnFvjpFeXJZlOYzH0DTMvZr2u2AOghJx0bEr5e0rMF9/IzUIOR5qiZCytWPqSiGPJM0Na0oqTmrpGWBbjlbnFnzunpRVWV6xlLbPBcWN3dDlybpWuCzwCQW9qoPoMsnDuAIoDZJHNnMskYcCHwmyptW4Czgy5F/Vk0AlCZB2p/iwqNS1xERp5exv85E0lnAT0l1tneShp//XkRct9gNlxS3K99xtIeiOKz6qjrLCKGdmdKcAYNyjr/U2Uk6BPgKqRXe/VVvrQDMj4gvNrvh0u2rlImQquI/GBHbLXnNpY5/J+nOvXY4n1xD23daldahkvYl1TUdB/y9dqSAenXpOw5Ji60QjAbn067Z1zHA6cA8UnFAZcTQLCOEdnJPkqa3dIXmQg+Rfh+r0XRujwAOzryvlYB/Sso6EVJVS7kJkkaRWm9Vx8/V+KF/RAzPFOuTpnKO3wv4fUTMrml91lDQrmpi8XM7YBAwqnh9IPB05n0dD2wcEa9njvtJsBrwtKRH6UQzuJWpaETxL2BbSZuT7j4OJFX8NjQwZjNKmQiJpi3l3iVV8FfkbDX3kKRNImJypnifJH+S9E9SUdX/Uxqle16jQV1UBUgaB2wfEfOL1z1JHfS2ybiPO4H9IuLdJa7cxbQ02GFXHsuq6A19SPF4nXRR8z8R0Wwv+45I0rcj4oI22M/TpFZh00gXHpXKdw9ZA5UJtWZHxEdKc9avEBH/X0MxnThAac7xbSv1DUrDPo+LiM9l3MfmwFWkKUCrr6qzFYfZJ4fSXB/3A0dFMR+9pGkRkb1oUyVNhNRWY6V5yJrFK6PFYlcvqqr4OWnq2HtJVyvDgFMz7+MS4K+UO8tgp1JzwmryFh1/9NGy7QeMBO4t7lZvpKTRjqNqlkdp4URIZewrJ0m9SZ1I1yN9r66olBpYUlaLxS5/x1F8UfqTRqzdulj8SKO3cs3sxxPBWN2KvgkjSEVWu5C+8LdGxF0l7zfHREjzWTjoZpO3yDMy9CjS9/Z+YE/gXxFx7OK36lrKarHY5RMH5B80roV9nEGaJ+CPNC2q6vLNca11iiLUA4GDc46Oq/ImQir1Yqn6e1uM4fVoR55GoD1I+j2pp3jWFosuqkoekzQ0IsaXuI9Dip8nVS1zc1xrtYh4kzS0xqWZQ5c1EVLZKvPaEBHzczQz/QQqpcWi7ziAornaeqTmj5W5Ldwqw6wBkn4YEWe0Yr2TIuJnSxG/MhcNNJ2PxnVkhbJaLDpxUG6rDEk/iIiziucHRsTvq947IyJ+2Og+zBrR3hMhtVXrq65K0qdIY+NBKs5reISAbo0G+CSIiH8VSeI9UvFR5ZHDyKrnJ9W8596u1hFcRvrf/BAgIp6g6f9t2VzGVBJJB5EGVj0QOAh4pBh2viGu4wAk7U0a1mFN4DXS/MjPABvlCN/C8+Zem7WH9p4IycUe5flfYGjlLqPoOX4PcHMjQX3HkfyE1G79uYgYSJobeVym2NHC8+Zem7WH9p4IyRdQ5elWUzQ1iwznfd9xJB9GxCxJ3SR1i4h7Jf0qU+xNJb1NUXlXPKd43TvTPswa8S1SS63PS5pBMRFSG+7/90texZbSnZLGsnAGwIOBMY0GdeU4IOkeYB/gZ6Tma6+Rbu++0K4HZtaGck+EJOl8FnNX7eF2yiNpPVJjhweLfjrbF2+9BVwfEc83FL8rJ47KL5fUHf890i3coaQ6jj9HxMTFbG72iVA7EVJleaMTIUk6YnHvR8TVjcS3lhVTVZ9UO2KwpE2AMxqdsrqrJ45Sf7lmnYEnQvrkkTQ+Ioa28F7DI2V09TqOTzU3hn9ETJY0oO0Px6xdlDoRUtGS5wTSnDcL6vVyzWluzVppMe/1aTR4V29VVeov16yTeKi4yy7L9aTm7QOB00hDmpQ5vI+lWRe/XrtQ0tEsnMBuqXX1oqrfAX+NiMtqlh8N7BYRuafoNOtwyp4ISdLEiNhS0hOVmIsrSrHGFb3FbwU+YGGiGEKab2XfRkf/7upFVd8FbpV0KM38ctvtqMza1p4lx68MRviqpL2AV4BVSt5nlxYR/wa+IGlnYONi8Z8j4q854nfpO46Kml/uU7l+uWYdWVtNhCTpS6Q5M9YGzgdWAE6LiNG592Vtw4nDrItqi4mQJHUnzQdxbs641r6cOMy6qLaaCEnSoxGxVe641n66eh2HWVfWVhMhPSjpAmAUC+fPICIeK2uHVi7fcZh1UW01EZKke5tZHO7H0Xk5cZiZWV1cVGVmpZJ0cnPLGx0Ly9qPE4eZlW1u1fPewJdIPcmtk3JRlZm1qWI03rERsVN7H4stna4+VpWZtb2+QP/2Pghbei6qMrNSSZrMwgmdugP9ANdvdGIuqjKzUklat+rlfODfZQxtYm3HRVVmVqqI+BdpnKpdImIGsJKkge18WNYA33GYWakknUIadfpzEbGBpDWB30fEdu18aLaUfMdhZmXbF9ibolluRLwCLN+uR2QNceIws7J9EKloIwAkLdvOx2MNcuIws7LdJOkSUt3G14F7gMuWsI11YK7jMLPSSdoN2J00gOLYiLi7nQ/JGuDEYWZmdXEHQDMrhaR3SPUaYmEHQMg8bLu1Pd9xmJlZXXzHYWalkNQb+CawHvAEcKV7jH8y+I7DzEohaRRpetr7gT2Bf0XEse17VJaDE4eZlULS5IjYpHjeA3g0IrZo58OyDNyPw8zK8mHliYuoPll8x2FmpZD0EQtn/xPQB3gXt6rq9Jw4zMysLi6qMjOzujhxmJlZXZw4zDKT9GlJN0p6XtJESWMkbSDpyfY+NrMc3AHQLCNJAm4Fro6IkcWyTYFPteuBmWXkOw6zvHYGPoyIiysLIuJx4OXKa0kDJN0v6bHi8YVi+RqS/i5pkqQnJe0gqbuk3xavJ0v6Xtt/JLOmfMdhltfGwMQlrPMasFtEzJO0PvA70tSqXyENOf5/kroDfYHNgLUiYmMASSuVd+hmrePEYdb2egIXSNoM+AjYoFg+HrhSUk/gtoiYJGka8BlJ5wN/Bu5qlyM2q+KiKrO8ngK2XMI63wP+DWxKutNYBiAi/g4MA2YAv5V0eES8Wax3H2nAwMvLOWyz1nPiMMvrr0AvScdUFkgaDKxdtc6KwKsR8TFwGNC9WG9d4N8RcRkpQWwhaTWgW0TcAvwI8FhP1u5cVGWWUUSEpH2BX0k6AZgHvAh8t2q13wC3SDocuJOFw3LsBBwv6UNgDnA4sBZwlaTKRd5JpX8IsyXwkCNmZlYXF1WZmVldnDjMzKwuThxmZlYXJw4zM6uLE4eZmdXFicPMzOrixGFmZnX5/wFufHzskjadXwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = pd.DataFrame({\"Class\": labels, \"Label\": \"Positive\", \"Value\": pos_contribution})\n",
    "data = data.append([{\"Class\": labels[l], \"Label\": \"Negative\", \"Value\": v} \n",
    "                        for l,v in enumerate(neg_contribution)], ignore_index=True)\n",
    "plt.xticks(rotation=90)\n",
    "sns.barplot(x=\"Class\", y=\"Value\", hue=\"Label\" ,data=data);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u9xgoEkpn5yc"
   },
   "source": [
    "As the above figure shows, by applying these weightings the positive and negative labels within each class would have the same aggregate contribution to the loss function. Now let's implement such a loss function. \n",
    "\n",
    "After computing the weights, our final weighted loss for each training case will be \n",
    "\n",
    "$$\\mathcal{L}_{cross-entropy}^{w}(x) = - (w_{p} y \\log(f(x)) + w_{n}(1-y) \\log( 1 - f(x) ) ).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='Ex-3'></a>\n",
    "### Exercise 3 - Weighted Loss\n",
    "Fill out the `weighted_loss` function below to return a loss function that calculates the weighted loss for each batch. Recall that for the multi-class loss, we add up the average loss for each individual class. Note that we also want to add a small value, $\\epsilon$, to the predicted values before taking their logs. This is simply to avoid a numerical error that would otherwise occur if the predicted value happens to be zero.\n",
    "\n",
    "##### Note\n",
    "Please use Keras functions to calculate the mean and the log.\n",
    "\n",
    "- [Keras.mean](https://www.tensorflow.org/api_docs/python/tf/keras/backend/mean)\n",
    "- [Keras.log](https://www.tensorflow.org/api_docs/python/tf/keras/backend/log)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pPIBVAasn5yd"
   },
   "outputs": [],
   "source": [
    "# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def get_weighted_loss(pos_weights, neg_weights, epsilon=1e-7):\n",
    "    \"\"\"\n",
    "    Return weighted loss function given negative weights and positive weights.\n",
    "\n",
    "    Args:\n",
    "      pos_weights (np.array): array of positive weights for each class, size (num_classes)\n",
    "      neg_weights (np.array): array of negative weights for each class, size (num_classes)\n",
    "    \n",
    "    Returns:\n",
    "      weighted_loss (function): weighted loss function\n",
    "    \"\"\"\n",
    "    def weighted_loss(y_true, y_pred):\n",
    "        \"\"\"\n",
    "        Return weighted loss value. \n",
    "\n",
    "        Args:\n",
    "            y_true (Tensor): Tensor of true labels, size is (num_examples, num_classes)\n",
    "            y_pred (Tensor): Tensor of predicted labels, size is (num_examples, num_classes)\n",
    "        Returns:\n",
    "            loss (Tensor): overall scalar loss summed across all classes\n",
    "        \"\"\"\n",
    "        # initialize loss to zero\n",
    "        loss = 0.0\n",
    "        \n",
    "        ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "\n",
    "        for i in range(len(pos_weights)):\n",
    "            # for each class, add average weighted loss for that class\n",
    "            loss_pos = -1 * K.mean(pos_weights[i] * y_true[:, i] * K.log(y_pred[:, i] + epsilon))\n",
    "            loss_neg = -1 * K.mean(neg_weights[i] * (1 - y_true[:, i]) * K.log(1 - y_pred[:, i] + epsilon))\n",
    "            loss += loss_pos + loss_neg\n",
    "        \n",
    "        return loss\n",
    "    \n",
    "        ### END CODE HERE ###\n",
    "    return weighted_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's test our function with some simple cases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 595
    },
    "colab_type": "code",
    "id": "CFjYda3Wulbm",
    "outputId": "87128f53-7a99-40e2-d09a-6539215879d0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test example:\n",
      "\n",
      "y_true:\n",
      "\n",
      "[[1. 1. 1.]\n",
      " [1. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [1. 0. 1.]]\n",
      "\n",
      "w_p:\n",
      "\n",
      "[0.25 0.25 0.5 ]\n",
      "\n",
      "w_n:\n",
      "\n",
      "[0.75 0.75 0.5 ]\n",
      "\n",
      "y_pred_1:\n",
      "\n",
      "[[0.7 0.7 0.7]\n",
      " [0.7 0.7 0.7]\n",
      " [0.7 0.7 0.7]\n",
      " [0.7 0.7 0.7]]\n",
      "\n",
      "y_pred_2:\n",
      "\n",
      "[[0.3 0.3 0.3]\n",
      " [0.3 0.3 0.3]\n",
      " [0.3 0.3 0.3]\n",
      " [0.3 0.3 0.3]]\n",
      "\n",
      "If we weighted them correctly, we expect the two losses to be the same.\n",
      "\n",
      "L(y_pred_1)= 0.9754045605659485, L(y_pred_2)= 0.9754045605659485\n"
     ]
    }
   ],
   "source": [
    "# Test\n",
    "sess = K.get_session()\n",
    "with sess.as_default() as sess:\n",
    "    print(\"Test example:\\n\")\n",
    "    y_true = K.constant(np.array(\n",
    "        [[1, 1, 1],\n",
    "         [1, 1, 0],\n",
    "         [0, 1, 0],\n",
    "         [1, 0, 1]]\n",
    "    ))\n",
    "    print(\"y_true:\\n\")\n",
    "    print(y_true.eval())\n",
    "\n",
    "    w_p = np.array([0.25, 0.25, 0.5])\n",
    "    w_n = np.array([0.75, 0.75, 0.5])\n",
    "    print(\"\\nw_p:\\n\")\n",
    "    print(w_p)\n",
    "\n",
    "    print(\"\\nw_n:\\n\")\n",
    "    print(w_n)\n",
    "\n",
    "    y_pred_1 = K.constant(0.7*np.ones(y_true.shape))\n",
    "    print(\"\\ny_pred_1:\\n\")\n",
    "    print(y_pred_1.eval())\n",
    "\n",
    "    y_pred_2 = K.constant(0.3*np.ones(y_true.shape))\n",
    "    print(\"\\ny_pred_2:\\n\")\n",
    "    print(y_pred_2.eval())\n",
    "\n",
    "    L = get_weighted_loss(w_p, w_n)\n",
    "\n",
    "    print(\"\\nIf we weighted them correctly, we expect the two losses to be the same.\")\n",
    "    print(\"\\nL(y_pred_1)= {}, L(y_pred_2)= {}\".format(L(y_true, y_pred_1).eval(), L(y_true, y_pred_2).eval()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yDZQMmlgn5yh"
   },
   "source": [
    "<a name='3-3'></a>\n",
    "### 3.3 DenseNet121\n",
    "\n",
    "Next, we will use a pre-trained [DenseNet121](https://www.kaggle.com/pytorch/densenet121) model which we can load directly from Keras and then add two layers on top of it:\n",
    "1. A `GlobalAveragePooling2D` layer to get the average of the last convolution layers from DenseNet121.\n",
    "2. A `Dense` layer with `sigmoid` activation to get the prediction logits for each of our classes.\n",
    "\n",
    "We can set our custom loss function for the model by specifying the `loss` parameter in the `compile()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "gZlxoCTgn5yi",
    "outputId": "7e12120b-8aab-403c-b5ca-2ff77ef978b1",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.\n",
      "\n",
      "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4074: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# create the base pre-trained model\n",
    "base_model = DenseNet121(weights='./nih/densenet.hdf5', include_top=False)\n",
    "\n",
    "x = base_model.output\n",
    "\n",
    "# add a global spatial average pooling layer\n",
    "x = GlobalAveragePooling2D()(x)\n",
    "\n",
    "# and a logistic layer\n",
    "predictions = Dense(len(labels), activation=\"sigmoid\")(x)\n",
    "\n",
    "model = Model(inputs=base_model.input, outputs=predictions)\n",
    "model.compile(optimizer='adam', loss=get_weighted_loss(pos_weights, neg_weights))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BcwhQdOAn5ym"
   },
   "source": [
    "<a name='4'></a>\n",
    "## 4 Training [optional]\n",
    "\n",
    "**Note** that we'll load a pre-trained model soon, so you don't need to run the following training cell (as it will take some time).\n",
    "\n",
    "With our model ready for training, we will use the `model.fit()` function in Keras to train our model. \n",
    "- We are training on a small subset of the dataset (~1%).  \n",
    "- So what we care about at this point is to make sure that the loss on the training set is decreasing.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 449
    },
    "colab_type": "code",
    "id": "YdZQTlGvn5ym",
    "outputId": "7d5009a0-1a00-4b45-8100-f1681691cbb8"
   },
   "outputs": [],
   "source": [
    "# \"\"\"\n",
    "# OPTIONAL: uncomment this code to practice training the model.\n",
    "# This is optional because you will load pre-trained weights after this.\n",
    "# \"\"\"\n",
    "\n",
    "# history = model.fit_generator(train_generator, \n",
    "#                               validation_data=valid_generator,\n",
    "#                               steps_per_epoch=100, \n",
    "#                               validation_steps=25, \n",
    "#                               epochs = 3)\n",
    "\n",
    "# plt.plot(history.history['loss'])\n",
    "# plt.ylabel(\"loss\")\n",
    "# plt.xlabel(\"epoch\")\n",
    "# plt.title(\"Training Loss Curve\")\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xB5nsGKrn5yp"
   },
   "source": [
    "<a name='4-1'></a>\n",
    "### 4.1 Training on the Larger Dataset\n",
    "\n",
    "Given that the original dataset is 40GB+ in size and the training process on the full dataset takes a few hours, we have trained the model on a GPU-equipped machine for you and provided the weights file from our model (with a batch size of 32 instead) to be used for the rest of this assignment. \n",
    "\n",
    "The model architecture for our pre-trained model is exactly the same, but we used a few useful Keras \"callbacks\" for this training. Do spend time to read about these callbacks at your leisure as they will be very useful for managing long-running training sessions:\n",
    "\n",
    "1. You can use `ModelCheckpoint` callback to monitor your model's `val_loss` metric and keep a snapshot of your model at the point. \n",
    "2. You can use the `TensorBoard` to use the Tensorflow Tensorboard utility to monitor your runs in real-time. \n",
    "3. You can use the `ReduceLROnPlateau` to slowly decay the learning rate for your model as it stops getting better on a metric such as `val_loss` to fine-tune the model in the final steps of training.\n",
    "4. You can use the `EarlyStopping` callback to stop the training job when your model stops getting better in it's validation loss. You can set a `patience` value which is the number of epochs the model does not improve after which the training is terminated. This callback can also conveniently restore the weights for the best metric at the end of training to your model.\n",
    "\n",
    "You can read about these callbacks and other useful Keras callbacks [here](https://keras.io/callbacks/).\n",
    "\n",
    "Let's load our pre-trained weights into the model now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "887bSajLn5yq"
   },
   "outputs": [],
   "source": [
    "#model.load_weights(\"./nih/pretrained_model.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mA90g8n6suRV"
   },
   "source": [
    "<a name='5'></a>\n",
    "## 5 Prediction and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Kz1BEwOyxFbj"
   },
   "source": [
    "Now that we have a model, let's evaluate it using our test set. We can conveniently use the `predict_generator` function to generate the predictions for the images in our test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QzNrhtf1w2bI",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#predicted_vals = model.predict_generator(test_generator, steps = len(test_generator))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wtjCtaGen5yt"
   },
   "source": [
    "<a name='5-1'></a>\n",
    "### 5.1 ROC Curve and AUROC\n",
    "We'll cover topic of model evaluation in much more detail in later weeks, but for now we'll walk through computing a metric called the AUC (Area Under the Curve) from the ROC ([Receiver Operating Characteristic](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)) curve. This is also referred to as the AUROC value, but you will see all three terms in reference to the technique, and often used almost interchangeably. \n",
    "\n",
    "For now, what you need to know in order to interpret the plot is that a curve that is more to the left and the top has more \"area\" under it, and indicates that the model is performing better.\n",
    "\n",
    "We will use the `util.get_roc_curve()` function which has been provided for you in `util.py`. Look through this function and note the use of the `sklearn` library functions to generate the ROC curves and AUROC values for our model. \n",
    "\n",
    "- [roc_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html)\n",
    "- [roc_auc_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 621
    },
    "colab_type": "code",
    "id": "6SLI8FHun5yw",
    "outputId": "4f5cc99c-4e1a-421b-fe2d-637df32d6416",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#auc_rocs = util.get_roc_curve(labels, predicted_vals, test_generator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zWZkl01ruZ7P"
   },
   "source": [
    "You can compare the performance to the AUCs reported in the original ChexNeXt paper in the table below: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GZUoShw2n5yy"
   },
   "source": [
    "For reference, here's the AUC figure from the ChexNeXt paper which includes AUC values for their model as well as radiologists on this dataset:\n",
    "\n",
    "<img src=\"https://journals.plos.org/plosmedicine/article/figure/image?size=large&id=10.1371/journal.pmed.1002686.t001\" width=\"80%\">\n",
    "\n",
    "This method does take advantage of a few other tricks such as self-training and ensembling as well, which can give a significant boost to the performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Jzy7fLgFn5yy"
   },
   "source": [
    "For details about the best performing methods and their performance on this dataset, we encourage you to read the following papers:\n",
    "- [CheXNet](https://arxiv.org/abs/1711.05225)\n",
    "- [CheXpert](https://arxiv.org/pdf/1901.07031.pdf)\n",
    "- [ChexNeXt](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002686)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G5aZAlVbn5yz"
   },
   "source": [
    "<a name='5-2'></a>\n",
    "### 5.2 Visualizing Learning with GradCAM \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gu9ExySryY6u"
   },
   "source": [
    "One of the challenges of using deep learning in medicine is that the complex architecture used for neural networks makes them much harder to interpret compared to traditional machine learning models (e.g. linear models). \n",
    "\n",
    "One of the most common approaches aimed at increasing the interpretability of models for computer vision tasks is to use Class Activation Maps (CAM). \n",
    "- Class activation maps are useful for understanding where the model is \"looking\" when classifying an image. \n",
    "\n",
    "In this section we will use a [GradCAM's](https://arxiv.org/abs/1610.02391) technique to produce a heatmap highlighting the important regions in the image for predicting the pathological condition. \n",
    "- This is done by extracting the gradients of each predicted class, flowing into our model's final convolutional layer. Look at the `util.compute_gradcam` which has been provided for you in `util.py` to see how this is done with the Keras framework. \n",
    "\n",
    "It is worth mentioning that GradCAM does not provide a full explanation of the reasoning for each classification probability. \n",
    "- However, it is still a useful tool for \"debugging\" our model and augmenting our prediction so that an expert could validate that a prediction is indeed due to the model focusing on the right regions of the image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we will load the small training set and setup to look at the 4 classes with the highest performing AUC measures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6kahoZbJn5yz",
    "outputId": "ade0a4e2-4591-4ba5-ec19-1a3487e3f972",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# df = pd.read_csv(\"nih/train-small.csv\")\n",
    "# IMAGE_DIR = \"nih/images-small/\"\n",
    "\n",
    "# # only show the lables with top 4 AUC\n",
    "# labels_to_show = np.take(labels, np.argsort(auc_rocs)[::-1])[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at a few specific images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#util.compute_gradcam(model, '00008270_015.png', IMAGE_DIR, df, labels, labels_to_show)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JC2zy1Kpn5y1",
    "outputId": "9e38a769-e19d-4143-da41-db7a3173a533"
   },
   "outputs": [],
   "source": [
    "#util.compute_gradcam(model, '00011355_002.png', IMAGE_DIR, df, labels, labels_to_show)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zCHVaLMQn5y2",
    "outputId": "57246709-2662-4590-9198-a412d2f1eea2",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#util.compute_gradcam(model, '00029855_001.png', IMAGE_DIR, df, labels, labels_to_show)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gGwL8FcFn5y4",
    "outputId": "681fb2de-194c-465e-c989-133f334b8299"
   },
   "outputs": [],
   "source": [
    "#util.compute_gradcam(model, '00005410_000.png', IMAGE_DIR, df, labels, labels_to_show)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations, you've completed the first assignment of course one! You've learned how to preprocess data, check for data leakage, train a pre-trained model, and evaluate using the AUC. Great work!"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "G5aZAlVbn5yz"
   ],
   "include_colab_link": true,
   "name": "C1M2_Assignment.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "coursera": {
   "schema_names": [
    "AI4MC1-1"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}