3_RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSE6250BDH Deep Learning Labs\n",
    "## 3. Recurrent Neural Networks\n",
    "\n",
    "In this chapter, we will learn how to implement a recurrent neural network by using PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 0. Prepare Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same dateset we used in the previous chapter of CNN will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   6   11   15 ...  -11  -17  -18]\n",
      " [ -41  -45  -45 ...  -66  -71  -57]\n",
      " [ -26  -10   28 ...   52   72   77]\n",
      " ...\n",
      " [ -51  -51  -19 ... -108  -97 -100]\n",
      " [-144 -158 -183 ...  126   82   68]\n",
      " [  23   -1  -34 ...  -17  -27  -27]]\n",
      "(7360, 178)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df = pd.read_csv('data.csv')\n",
    "labels = df['y'].values\n",
    "labels = labels - 1\n",
    "data = df.loc[:, 'X1':'X178'].values\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.2, random_state=6250)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.2, random_state=6250)\n",
    "\n",
    "print(X_train)\n",
    "print(X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Loading Datasets\n",
    "Shape of input tensor is (batch, seq_len, input_size), different from the one used for CNN in the previous chapter. In fact, the default shape of input for recurrent layer modules in PyTorch such as RNN, GRU, and LSTM is (seq_len, batch, input_size). You can choose your preferred shape, but **do not forget to set proper boolean value for `batch_first` argument for recurrent layer modules**.\n",
    "\n",
    "As our current dataset has a shape of (num_subject, seq_len) and each time step has 1D signal, we need to add a dummy axis to make it as a 3D tensor. In the following codes, you can see `unsqueeze(dim)` function on Tensor which adds a new axis (dimension)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "\n",
    "# lets fix the random seeds for reproducibility.\n",
    "torch.manual_seed(6250)\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.manual_seed(6250)\n",
    "\n",
    "trainset = TensorDataset(torch.from_numpy(X_train.astype('float32')).unsqueeze(2), torch.from_numpy(y_train.astype('long')))\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=10, shuffle=True, num_workers=2)\n",
    "\n",
    "validset = TensorDataset(torch.from_numpy(X_valid.astype('float32')).unsqueeze(2), torch.from_numpy(y_valid.astype('long')))\n",
    "validloader = torch.utils.data.DataLoader(validset, batch_size=10, shuffle=False, num_workers=2)\n",
    "\n",
    "testset = TensorDataset(torch.from_numpy(X_test.astype('float32')).unsqueeze(2), torch.from_numpy(y_test.astype('long')))\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=10, shuffle=False, num_workers=2)\n",
    "\n",
    "classes = ('Seizure', 'TumorArea', 'HealthyArea', 'EyesClosed', 'EyesOpen')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Define RNN class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class RecurNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(RecurNet, self).__init__()\n",
    "        self.rnn = nn.GRU(input_size=1, hidden_size=32, num_layers=2, batch_first=True, dropout=0.5)\n",
    "        self.fc = nn.Linear(in_features=32, out_features=5)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x, _ = self.rnn(x)\n",
    "        x = self.fc(x[:, -1, :])\n",
    "        return x\n",
    "\n",
    "\n",
    "model = RecurNet()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, let's define a class for our model. In this tutorial we use one of gated units, Gated Recurrent Unit (GRU), as our recurrent layer. `nn.GRU` is a implementatoin of (multi-layered) GRU. 'multi-layered' means you can easily define several layers of GRU by putting the number of layers as input argument for `num_layers`. On the other hand, `nn.GRUCell` can be used when you want to construct more complex architecture between each recurrent layer. We use 2-layers of GRU for this tutorial.\n",
    "\n",
    "**Note again that, we need to carefully set the value for `batch_first` for all types of RNN module.**\n",
    "\n",
    "At line 9, we retrieve the hidden unit outputs from the last recurrent layer for the last time step. To be specific, `nn.GRU` returns an tuple of `output, h_n` where `output` is the output value of hidden units from the last layer for **ALL** time steps and `h_n` is the same value for the last time step only. Therefore, `x, _ = self.rnn(x)` at line 8, we get a Tensor `x` in a shape of (batch, seq_len, hidden_size) since we set `batch_first=True`. We then extract the values for the last time step by `x[:, -1, :]`. If we set `batch_first=False`, indexing should be modified also.\n",
    "\n",
    "We add a fully-connected layer on the top of the recurrent layers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Define a Loss function and optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the same loss function and the same optimizer from the previous chapter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Train the network\n",
    "Also, we can use the same procedure for training the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "cuda = torch.cuda.is_available()\n",
    "if cuda:\n",
    "    model = model.cuda()\n",
    "\n",
    "train_losses = []\n",
    "valid_losses = []\n",
    "\n",
    "for epoch in range(15):  # loop over the dataset multiple times\n",
    "\n",
    "    # set the model as train mode\n",
    "    model.train()\n",
    "    train_loss = 0.0\n",
    "    train_counter = 0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        # get the inputs\n",
    "        inputs, targets = data\n",
    "\n",
    "        if cuda:\n",
    "            inputs, targets = inputs.cuda(), targets.cuda()\n",
    "\n",
    "        # zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # forward + backward + optimize\n",
    "        outputs = model(inputs)\n",
    "        loss = criterion(outputs, targets)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        train_loss += (loss.item() * inputs.size(0))\n",
    "        train_counter += inputs.size(0)\n",
    "\n",
    "    train_losses.append(train_loss/train_counter)\n",
    "    \n",
    "    # switch to evaluation mode\n",
    "    model.eval()\n",
    "    valid_loss = 0.0\n",
    "    valid_counter = 0\n",
    "    with torch.no_grad():\n",
    "        for i, data in enumerate(validloader, 0):\n",
    "            # get the inputs\n",
    "            inputs, targets = data\n",
    "\n",
    "            if cuda:\n",
    "                inputs, targets = inputs.cuda(), targets.cuda()\n",
    "\n",
    "            outputs = model(inputs)\n",
    "            loss = criterion(outputs, targets)\n",
    "\n",
    "            valid_loss += (loss.item() * inputs.size(0))\n",
    "            valid_counter += inputs.size(0)\n",
    "    valid_losses.append(valid_loss/valid_counter)\n",
    "    \n",
    "print('Finished Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should check the learning curve always!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f426007b908>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd81FW+//HXJ72HNFpCCUkEAiLECEKoi65gAdeOYkWxu3t1f6uru6vrXne997qu61WxoNhYvDas2FZRiiJdpBN6EiANEtLb+f3xHYYA6ZlvZpL5PB+PPJKZ+ZZP8oB5z/ec7zlHjDEopZRSAD7uLkAppZTn0FBQSinlpKGglFLKSUNBKaWUk4aCUkopJw0FpZRSThoKSimlnDQUlFJKOWkoKKWUcvJzdwGtFRsba/r37+/uMpRSqlNZs2ZNvjEmrrntOl0o9O/fn9WrV7u7DKWU6lREZG9LttPmI6WUUk4aCkoppZw0FJRSSjl1uj4FpVTXUV1dTVZWFhUVFe4upcsICgoiISEBf3//Nu2voaCUcpusrCzCw8Pp378/IuLucjo9YwwFBQVkZWWRmJjYpmPY1nwkIq+ISK6IbGzk9WtEZIPj63sROcOuWpRSnqmiooKYmBgNBBcREWJiYtp15WVnn8KrwJQmXt8NTDDGDAP+ArxoYy1KKQ+lgeBa7f172hYKxpglQGETr39vjDnseLgCSLCrFoDth47yn59spqK61s7TKKVUp+Ypdx/NAj5r7EURmS0iq0VkdV5eXptOkHW4jLnLdrNm7+HmN1ZKeYWCggKGDx/O8OHD6dmzJ/Hx8c7HVVVVLTrGjTfeyLZt22yutOO4vaNZRCZhhcLYxrYxxryIo3kpPT3dtOU8IxNj8PMRlmfmk5Ec26ZalVJdS0xMDOvXrwfgkUceISwsjN/+9rcnbGOMwRiDj0/Dn6HnzZtne50dya1XCiIyDJgLTDfGFNh5rrBAP4b36cbynbaeRinVBWRmZjJ06FBuu+020tLSOHDgALNnzyY9PZ0hQ4bw6KOPOrcdO3Ys69evp6amhm7duvHAAw9wxhlnMHr0aHJzc934W7SN264URKQv8D5wrTFme0ecc0xyLM98s4Oi8moig9t2D69Syh5//ngTm3OKXXrM1N4RPHzRkDbtu3nzZubNm8fzzz8PwOOPP050dDQ1NTVMmjSJyy67jNTU1BP2KSoqYsKECTz++OPce++9vPLKKzzwwAPt/j06kp23pC4AfgAGikiWiMwSkdtE5DbHJn8CYoDnRGS9iNg+y11GUgx1Blbs0qsFpVTTkpKSOOuss5yPFyxYQFpaGmlpaWzZsoXNmzefsk9wcDBTp04F4Mwzz2TPnj0dVa7L2HalYIyZ0czrNwM323X+hozoG0Wwvy/fZ+Zz3pCeHXlqpVQz2vqJ3i6hoaHOn3fs2ME///lPVq5cSbdu3Zg5c2aDYwECAgKcP/v6+lJTU9MhtbqSp9x91CEC/HwYmRit/QpKqVYpLi4mPDyciIgIDhw4wBdffOHukmzjVaEAkJEcQ2ZuCYeKda4VpVTLpKWlkZqaytChQ7nlllvIyMhwd0m2EWPadIen26Snp5v2LLKzMbuIC/93GU9ecQaXpNk6Xk4p1YwtW7YwePBgd5fR5TT0dxWRNcaY9Ob29borhdReEUSF+LM8U5uQlFLqZF4XCj4+wpikWL7fmU9nu0pSSim7eV0oAIxJjuFAUQW780vdXYpSSnkUrwyFjCRrmovlmflurkQppTyLV4ZCv5gQ4rsFa7+CUkqdxCtDQUTISI7hh10F1NZpv4JSSh3jlaEAkJEcS1F5tcvnWlFKdR4TJ048ZSDaU089xR133NHoPmFhYQDk5ORw2WWXNXrc5m6df+qppygrK3M+Pv/88zly5EhLS7eN14bC6KQYAJZpv4JSXmvGjBm89dZbJzz31ltvMWNGk7P0ANC7d2/efffdNp/75FBYtGgR3bp1a/PxXMVrQ6F7eBADe4Tz/U4NBaW81WWXXcYnn3xCZWUlAHv27CEnJ4fhw4czefJk0tLSOP300/nwww9P2XfPnj0MHToUgPLycq666iqGDRvGlVdeSXl5uXO722+/3Tnl9sMPPwzA008/TU5ODpMmTWLSpEkA9O/fn/x86/3oySefZOjQoQwdOpSnnnrKeb7Bgwdzyy23MGTIEH75y1+ecB5XcfsiO+40JjmGBSv3UVlTS6Cfr7vLUcq7ffYAHPzZtcfseTpMfbzRl2NiYhg5ciSff/4506dP56233uLKK68kODiYhQsXEhERQX5+PmeffTbTpk1rdP3jOXPmEBISwoYNG9iwYQNpaWnO1x577DGio6Opra1l8uTJbNiwgXvuuYcnn3ySxYsXExt74qJfa9asYd68efz4448YYxg1ahQTJkwgKiqKHTt2sGDBAl566SWuuOIK3nvvPWbOnOmav5WD114pgHVrakV1HWv3ur8dTynlHvWbkI41HRljePDBBxk2bBjnnHMO2dnZHDp0qNFjLFmyxPnmPGzYMIYNG+Z87e233yYtLY0RI0awadOmBqfcrm/ZsmX86le/IjQ0lLCwMC655BKWLl0KQGJiIsOHDwfsm5rbq68URg2IxtexROexPgallJs08YneThdffDH33nsva9eupby8nLS0NF599VXy8vJYs2YN/v7+9O/fv8Gpsutr6Cpi9+7dPPHEE6xatYqoqChuuOGGZo/T1EwLgYGBzp99fX1taT7y6iuF8CB/zkiIZLn2KyjltcLCwpg4cSI33XSTs4O5qKiI7t274+/vz+LFi9m7d2+Txxg/fjzz588HYOPGjWzYsAGwptwODQ0lMjKSQ4cO8dlnnzn3CQ8P5+jRow0e64MPPqCsrIzS0lIWLlzIuHHjXPXrNsurQwGsW1M3ZBVxtKLa3aUopdxkxowZ/PTTT1x11VUAXHPNNaxevZr09HTmz5/PoEGDmtz/9ttvp6SkhGHDhvHf//3fjBw5EoAzzjiDESNGMGTIEG666aYTptyePXs2U6dOdXY0H5OWlsYNN9zAyJEjGTVqFDfffDMjRoxw8W/cOK+bOvtkP+wsYMZLK5h7XTrnpPZw2XGVUs3TqbPtoVNnt0Nav24E+fvoeAWllEJDgUA/X87qH63jFZRSCg0FwOpX2H6ohNyjukSnUh2tszVhe7r2/j01FDg+lfYPO3XWVKU6UlBQEAUFBRoMLmKMoaCggKCgoDYfw6vHKRyT2juCyGB/lu3IZ/rweHeXo5TXSEhIICsri7y8PHeX0mUEBQWRkND29edtCwUReQW4EMg1xgxt4PVBwDwgDXjIGPOEXbU0x9dHGD0ghu93Wp9YGhvKrpRyLX9/fxITE91dhqrHzuajV4EpTbxeCNwDuC0M6stIiSX7SDl7C8qa31gppboo20LBGLME642/sddzjTGrAI8YNZbhmOZCRzcrpbxZp+hoFpHZIrJaRFbb1faYGBtKr8ggXbdZKeXVOkUoGGNeNMakG2PS4+LibDmHiDAmKZYfdhZQp0t0KqW8VKcIhY4yNiWGw2XVbD6gS3QqpbyThkI9YxzjFXR0s1LKW9kWCiKyAPgBGCgiWSIyS0RuE5HbHK/3FJEs4F7gD45tIuyqpyV6RASR3D2MZZk6iE0p5Z1sG6dgjGly5WtjzEGg7SMsbJKRFMPbq7OoqqkjwE8vpJRS3kXf9U6SkRxLeXUt6/YddncpSinV4TQUTjJqQAw+gt6aqpTyShoKJ4kM9uf0hG4s18nxlFJeSEOhARlJMfy0/wgllTXuLkUppTqUhkIDxibHUlNnWLlbrxaUUt5FQ6EBaf2iCPTzYdkODQWllHfRUGhAkL8v6f2jdBCbUsrraCg0YkxSLFsPHiW/pNLdpSilVIfRUGjE2ORjU15oE5JSyntoKDRiaHwkEUF+LN+hTUhKKe+hodAIXx/h7AExuuiOUsqraCg0ISM5lqzD5ezTJTqVUl5CQ6EJGY5+Bb1aUEp5Cw2FJiTFhdIjIpBlOg+SUspLaCg0QUTI0CU6lVJeREOhGWOSYyksrWLrwaPuLkUppWynodCMjOQYQJfoVEp5Bw2FZvSKDGZAXKj2KyilvIL3hEJ+Jnz9KNRUtXrXjKRYVu4upKqmzobClFLKc3hPKBTsgKV/hz1LWr1rRnIMZVW1/JR1xIbClFLKc3hPKAyYBAFhsPmjVu86ekAsokt0KqW8gPeEgn8QnHYebP0U6mpbtWtkiD+nx0dqKCilujzvCQWAwdOgLB/2ft/qXcckxbJu3xFKdYlOpVQXZlsoiMgrIpIrIhsbeV1E5GkRyRSRDSKSZlctTinngl8wbGl9E1JGcoy1ROeeQhsKU0opz2DnlcKrwJQmXp8KpDi+ZgNzbKzFEhAKyZNhy8dQ17o7ic7qH02Anw/faxOSUqoLsy0UjDFLgKY+Vk8HXjeWFUA3EellVz1OqdPh6AHIXt2q3YL8fTmzbxTLMnXRHaVU1+XOPoV4YH+9x1mO504hIrNFZLWIrM7Ly2vfWU87D3z8YfOHrd41IzmGLQeKKdAlOpVSXZQ7Q0EaeK7BWeeMMS8aY9KNMelxcXHtO2tQJCRNsm5NNa2b5G6MYyrtH3bp1YJSqmtyZyhkAX3qPU4AcjrkzKnToWgfHFjfqt2GxUcSHujHcm1CUkp1Ue4MhY+A6xx3IZ0NFBljDnTImQeeD+Lb6oFsfr4+jBoQo+MVlFJdlp23pC4AfgAGikiWiMwSkdtE5DbHJouAXUAm8BJwh121nCIkGhLHWbemtrIJKSM5hn2FZewv1CU6lVJdj59dBzbGzGjmdQPcadf5mzV4Gnx6L+RugR6pLd7t2BKd3+/M58rovnZVp5RSbuFdI5rrG3QhIK0eyJbSPYy48EDtV1BKdUneGwrhPaDv6Fb3K1hLdMbw/c58TCubnpRSytN5bygApE6D3E3WWgutMCY5lvySKrYd0iU6lVJdi3eHwuCLrO9bWjeQ7Vi/gjYhKaW6Gu8OhcgEiE9vdRNSfLdg+seE6DxISqkux7tDAawmpAPr4fDeVu2WkRzLil0FVNfqEp1Kqa5DQ2HwNOv7lo9btVtGciylVbVs0CU6lVJdiIZCdCL0PL3Vt6aOHhDjWKJT+xWUUl2HhgLA4Omw/0cobvnUS1GhAaT2itApL5RSXYqGAlj9CgBbPmnVbmOTY1m77zBlVbpEp1Kqa9BQAIgbCLEDW92ENCY5lupaw6o9h20qTCmlOpaGwjGp02DvcihteXPQWf2j8PcVvTVVKdVlaCgckzodTB1sbXkTUkiAHyP6RrF0h4aCUqpr0FA4psdQiEps9UC2KUN6svlAMWv2NrUctVJKdQ4aCseIWE1Iu7+D8pb3EVw1sg9RIf48u3injcUppVTH0FCob/B0qKuBbZ+3eJeQAD9uykjkm625bMopsrE4pZSyn4ZCffFpEJHQ6ruQrhvdn7BAP+Z8q1cLSqnOTUOhPhFr5tTMr6Gy5dNiR4b4M/Psfnz68wF25ZXYWKBSStlLQ+FkqdOgthK2f9Gq3WaNTSTA14cXvttlU2FKKWU/DYWT9RkFod1b3YQUFx7IlWf14f11WeQcKbepOKWUsleLQkFEkkQk0PHzRBG5R0S62Vuam/j4Wk1IO76CqrJW7Tp7/ACMgZeW6tWCUqpzaumVwntArYgkAy8DicC/bKvK3VKnQXUZ7Py6VbslRIUwfXg8C1buo6Ck0qbilFLKPi0NhTpjTA3wK+ApY8x/AL3sK8vN+o2F4OhWD2QDuH3iACpr6pi3fI/r61JKKZu1NBSqRWQGcD1wbB4I/+Z2EpEpIrJNRDJF5IEGXu8nIl+LyAYR+VZEElpeuo18/WDQ+bD9c6hp3Sf+5O7hTBnSk9d+2ENxRbU99SmllE1aGgo3AqOBx4wxu0UkEXizqR1ExBd4FpgKpAIzRCT1pM2eAF43xgwDHgX+1pribTV4OlQWw67vWr3rHROTOVpRw5srWrfEp1JKuVuLQsEYs9kYc48xZoGIRAHhxpjHm9ltJJBpjNlljKkC3gKmn7RNKnCs4X5xA6+7z4AJEBgBmz9s9a6nJ0Qy/rQ4Xl66m/KqWhuKU0ope7T07qNvRSRCRKKBn4B5IvJkM7vFA/vrPc5yPFffT8Cljp9/BYSLSEwD558tIqtFZHVeXl5LSm4/v0A4bQps+xRqW98MdOfEJApKq3h79f7mN1ZKKQ/R0uajSGNMMXAJMM8YcyZwTjP7SAPPmZMe/xaYICLrgAlANnDKMmbGmBeNMenGmPS4uLgWluwCqdOsyfH2LGv1riMTo0nvF8UL3+2kqqbOhuKUUsr1WhoKfiLSC7iC4x3NzckC+tR7nACcsAiyMSbHGHOJMWYE8JDjOc+ZVS5pMviHtHogG4CIcOekZHKKKvhwfbYNxSmllOu1NBQeBb4AdhpjVonIAGBHM/usAlJEJFFEAoCrgBPeXUUkVkSO1fB74JWWl94BAkIg5ZfW2s11re8bmDgwjsG9Ipjz3U5q606+SFJKKc/T0o7md4wxw4wxtzse7zLGXNrMPjXAXVhhsgV42xizSUQeFZFpjs0mAttEZDvQA3isjb+HfVKnQWku7P+x1btaVwtJ7Mor5YtNB20oTimlXKulHc0JIrJQRHJF5JCIvNeSMQXGmEXGmNOMMUnGmMccz/3JGPOR4+d3jTEpjm1uNsZ43jDglF+Cb2CbBrIBTB3ai8TYUJ5dnIkxerWglPJsLW0+mofV9NMb6w6ijx3PdX2B4ZA82epXqGt9h7Gvj3D7hCQ25RTz3fYOunNKKaXaqKWhEGeMmWeMqXF8vQp04G1AbjZ4GhRnQ87aNu1+8Yh4ekUG8Zwu2amU8nAtDYV8EZkpIr6Or5lAgZ2FeZSBU8DHr00D2QAC/HyYPX4AK/cUsnJ3oYuLU0op12lpKNyEdTvqQeAAcBnW1BfeITgKEidYTUht7Be46qy+RIcG8Ny3mS4uTimlXKeldx/tM8ZMM8bEGWO6G2MuxhrI5j1Sp8HhPXDw5zbtHhzgy6yxiXy7LY+N2Z4zFEMppeprz8pr97qsis5g0IUgPm0ayHbMzLP7ER7ox5xvtW9BKeWZ2hMKDU1j0XWFxkK/jDbfmgoQGezPtaP7sWjjAXbmlbiwOKWUco32hIL33XSfOh3yt0HetjYf4qaxiQT4+vC8Xi0opTxQk6EgIkdFpLiBr6NYYxa8y6ALre/tuFqIDQtkxsi+LFyXTfaRchcVppRSrtFkKBhjwo0xEQ18hRtj/DqqSI8R0Qv6jGrzranH3DJ+AAAvLdnliqqUUspl2tN85J0GT4NDP0Nh29/Q47sFc0laPAtW7iO/xPNm9lBKeS8NhdYafJH1vR1NSAC3TUiiqraOV5btdkFRSinlGhoKrRXVD3oNb9etqQAD4sI4//RevPHDXorKW7+ym1JK2UFDoS1Sp0P2GijKatdh7piYxNHKGt5csddFhSmlVPtoKLRF6nTr+5aP23WYIb0jmTQwjpeX7aa8qvWL+CillKtpKLRFTBJ0H9LufgWAOyclU1haxYKV+1xQmFJKtY+GQlulToN9P8DRQ+06THr/aEYmRvPikl1U1bR+vQallHIlDYW2GjwNMLC1fU1IYF0tHCyuYOG69vVRKKVUe2kotFX3wRCT7JImpPEpsQyNj2DOtzuprfO+2UOUUp5DQ6GtRKyrhT3LoKx9C+eICHdOTGZPQRmLfj7gogKVUqr1NBTaI3UamFrY+mm7D3XekJ4kxYXy7OJMTBsX8lFKqfbSUGiPXsOhW992z4UE4OMj3D4xma0Hj7J4W64LilNKqdbTUGgPERh2JWR+BZlft/tw04f3Jr5bMM98o1cLSin3sDUURGSKiGwTkUwReaCB1/uKyGIRWSciG0TkfDvrscW4+yBuMCy8DUry2nUof18fbp0wgLX7jvDj7vb1UyilVFvYFgoi4gs8C0wFUoEZIpJ60mZ/AN42xowArgKes6se2/gHw6VzoaIIProL2vkJ/4r0PsSGBfDs4kwXFaiUUi1n55XCSCDTGLPLGFMFvAVMP2kbA0Q4fo4Ecmysxz49h8K5j8L2z2HV3HYdKsjfl1ljB7B0Rz4bso64qECllGoZO0MhHthf73GW47n6HgFmikgWsAi4u6EDichsEVktIqvz8trXRGObUbdC8rnwxUNwaHO7DjXz7L5EBPlx39s/sbeg1EUFKqVU8+wMBWnguZPbVmYArxpjEoDzgTdE5JSajDEvGmPSjTHpcXFxNpTqAiJw8RwIioD3ZkF125faDA/yZ87MM8k9Wsm0Z5azZLuHBqFSqsuxMxSygD71HidwavPQLOBtAGPMD0AQEGtjTfYKi4OLn4fczfDVw+06VEZyLB/fNZZekUHcMG8lLy7ZqXckKaVsZ2corAJSRCRRRAKwOpJPnhNiHzAZQEQGY4VC5/5YnHIOjLodVr4A279o16H6xoTw3u1jmDK0J39dtJVfv7Vep9hWStnKtlAwxtQAdwFfAFuw7jLaJCKPisg0x2b3AbeIyE/AAuAG0xU+Dp/zCPQYCh/c0e5ZVEMD/Xj26jR+N2UgH2/I4dI537O/sMwlZSql1Mmks70Hp6enm9WrV7u7jOblboUXJ0D/sXD1O+DT/vxdvC2Xexasw89HePbqNMYkd96WNqVUxxKRNcaY9Oa20xHNduk+CM57DDL/bTUlucCkgd356K6xxIQFcu0rK3l52W7tZ1BKuZSGgp3SZ8HA8+GrP8HBn11yyMTYUD64M4PJg7rzl082c987P1FRrf0MSinX0FCwkwhMewaCo+G9m6HKNX0BYYF+PD/zTO499zTeX5vN5c//QM6Rtt8Cq5RSx2go2C00Bn41B/K2wpd/cNlhfXyEeyan8NJ16ezOL+Wi/13Gj7sKXHZ8pZR30lDoCEm/gDF3w+qXYesilx763NQefHBnBpHB/lwz90de/2GP9jMopdpMQ6Gj/OJP0HMYfHgnFLt2dbXk7mF8cFcGE06L408fbuL+9zZQWaP9DEqp1tNQ6Ch+AXDpy9b0Fx/cBnV1Lj18RJA/L12Xzj2/SObt1Vlc+cIKDhZVuPQcSqmuT0OhI8WdBlMfh13fwg/PuPzwPj7Cvb8cyPMz09hx6CgXPbOMNXt1XQalVMtpKHS0tOth0IXw9aOQs96WU0wZ2ouFd2YQGuDLVS+uYMHKfbacRynV9WgodDQRmPa/EBpnzaZaZc/U2Kf1COfDO8cyJimW37//Mw8u/JmqGtc2WSmluh4NBXcIiYZLXoCCnfD57207TWSIP6/ccBa3T0ziXz/u4+qXVpB7VPsZlFKN01Bwl8TxMPY3sPY12Hzy5LGu4+sj3D9lEM9cPYJNOcVM+9/lLPr5AHV1etuqUupUGgruNPFB6D0CProbirJtPdWFw3rz/h1jCAvy4475aznvqSV8uD6bWg0HpVQ9GgrudOw21dpqWHgr1Nk7tmBwrwi++M14np4xAhH49VvrOefJ73h79X6qa7W/QSmloeB+MUlw/n/DnqWw/J+2n87XR5h2Rm8+//V4np95JiEBvvzu3Q1MeuJb5v+4Vwe9KeXldD0FT2AMvHsjbPkYZn0J8Wd24KkNi7fl8vTXmazff4SeEUHcOmEAM0b2Jcjft8PqUErZq6XrKWgoeIrywzBnrNWkdOtSCAzr0NMbY1ieWcDT3+xg5e5CYsMCmT0+kWtG9SM00K9Da1FKuZ4ustPZBEfBpS/B4T3w2f0dfnoRYWxKLG/fOpr/m302g3uF89dFWxn7X9/wzDc7KK6o7vCalFIdT0PBk/QbA+Pug/Vvwsb33VbGqAExvDFrFO/fMYYRfaN44svtZDz+DU9+uY0jZVVuq0spZT9tPvI0tdUwbyrkbYfbl0G3vu6uiI3ZRTzzTSafbzpIaIAv147uz83jEokNC3R3aUqpFtI+hc6scDc8Pw5CY+GK16HXMHdXBMC2g0d5ZnEmn2zIIdDPh6tH9uPWCQPoERHk7tKUUs3QUOjs9v0I79wAZQXWLatp11vzJnmAnXklPLd4Jx+sz8bXR7gyvQ+3ThhAQlSIu0tTSjVCQ6ErKM2H92+Bnd/AsKvgwichINTdVTntKyhjzneZvLsmi5o6w+nxkYxNjmVcShxn9osiwE+7rJTyFB4RCiIyBfgn4AvMNcY8ftLr/wAmOR6GAN2NMd2aOqZXhQJYo5yXPAHf/g3iBsEVr0HcQHdXdYKcI+W8szqLZZl5rN13hNo6Q7C/L2cPiGZsShzjU2JJ7h6GuPtKpyTPmmtq7eswcCpM/S/31qNUB3J7KIiIL7AdOBfIAlYBM4wxmxvZ/m5ghDHmpqaO63WhcMzOxfDezdbKbRf9E4Zd7u6KGnS0opoVuwpZuiOPZTvy2ZVvTQ3eMyKIsSmxjEuJJSM5tmM7qbPWwMoXYdP7UFsFMSlQsAMufh6Gz+i4OpRyI08IhdHAI8aY8xyPfw9gjPlbI9t/DzxsjPmqqeN6bSgAFOfAuzfBvh8g/SY472/g79mdvFmHy1i2I5+lO/JZvjOfI2XWeIchvSMYmxLLeEdTk8tHT9dUwqaFVhhkr4GAMBh+NZx1C0QPgDcuhqzVcMvX0GOIa8+tlAfyhFC4DJhijLnZ8fhaYJQx5q4Gtu0HrAASjDGnTL4jIrOB2QB9+/Y9c+/evbbU3CnU1sA3j1rzJPU6Ay5/DaIT3V1Vi9TWGTZmF7EsM58l2/NYu+8w1bWGIH8fRibGMC45lnGnxTKwR3jbm5qKsmD1K7DmNSjLh9jTYORsGHYlBEUc364k17rDKyAUZn974mtKdUGeEAqXA+edFAojjTF3N7Dt/ViBcMprJ/PqK4X6ti6CD24DA1z8HAy+0N0VtVppZQ0/7i5gqeNKIjO3BIC48EDGJccyNiWWUQNi6B0Z1HRIGAN7lllXBVs/BQycNhVG3gIDJjZ+19be7+HVC2HQBdatv+7u81DKRp4QCi1uPhKRdcCdxpjvmzuuhkI9h/dYt63mrIPRd8E5j4Cvv3traocDReUs3ZHPsh35LMvMp7DUGj0dGuBLcvcwkrqHkdI9nOTuYaR0D6NPWB2+G9+BlS9B7mZrqpC0662mtah+LTvp8qfhqz/CeX+F0Xetf3ZGAAAWEklEQVTa+Nsp5V6eEAp+WB3Nk4FsrI7mq40xm07abiDwBZBoWlCMhsJJairhi4dg1UvQZxRcNg8i491dVbvV1Rk2Hyhm/f4jZOaWOL8OFlfQXw5wre+/udz3OyKkjP2BKWztO4PaIZcwoFcs/WNCW347rDHwfzNh++dww6fQ92x7fzGl3MTtoeAo4nzgKaxbUl8xxjwmIo8Cq40xHzm2eQQIMsY80JJjaig0YuN78NE94BcIl7wEyZPdXZFr1dVB5r+pWfE8fru+pk782BL1Cz4IuIDPi/qQdaSCY/+UfX2EfjEhpHQPc1xVWFcXSXFhBAc00KFdUQQvTICaCmuG2rC4jv3dlOoAHhEKdtBQaEL+Dnj7OsjdAhN+BxPuBx+b10SorYG8LZC3zTqXb6A1/bdvQBM/O778AsHHr+m2/PLDsG4+rJoLh3dDWE+reejM6yG85/HNqmrZmXf8imJH7lF25Jawt6DMueSoCMR3CyalexgDe0aQ2juC1F4RJMaG4pu7EeaeY11tXbvQ/r+bUh1MQ8FbVZXBot/C+vmQOAEunQth3V13/OIDkL0aslZZ9//nrIPq0nYcUI4HhDMs6gVJ4U6oLoO+o62O40EXWa+3UFVNHXsKSq2gOGSFxbHgqHGERbC/L4N6hTMzcCmX7v8bB8+4m8gL/tzwVYVSnZSGgrdb+4YVDkHd4PJ51rTcrVVVBgd+OjEEirOs13z8rYn64tMh4SzokWo9X1sFNVVQW1nvZ8dXTeXxnxvdrvLEfcJ7wpk3WLffulBlTS2ZuSVszilm84FiNuUUsyWnmIdqn+Mqv2+5qfr/sS9mHKm9jl9RpPaO0JlhVaeloaDg4EarOenwHpj8JxhzD/g00gFrDBTstN78j4XAoU1QV2O93q2v9eZ/LAR6nu7xA+dayxhDdl4hEf+6gICSbB7uOYelecHkFFU4t+kREVgvKCJJ7R1Bv+gQfHz0dlbl2TQUlKWiGD66GzZ/AKdNgYvnQEg0lBVaI32zHAGQvQYqjlj7BIRDfBokpDtCIN21TVCernAXvDARYgbATV9wuFLYcsC6ojh2ZbEjt8TZVxEa4MtgR1AM6R3BkN6RpPQII9BPm5+U59BQUMcZY93L/8WDEBoH/sFWWz2A+EDcYOuNP8FxFRB7mna0bvkE/u8aSJ9lzU57korqWnYcKmHzgSJnUGzOKaa0yhqQ7+8rpHQPd4REBEPiIxncK4IwXe9auYmGgjpV1hr48g/WIK9jIdB7BASGu7syz/TlH+H7p61bfIdd0ezmdXWGvYVlbMopYmN2MZtyrMAocAzCE4HEmFDHFUUkQ+Ot79GhLe84V6qtNBSUaq/aGnh9mnWH1S3fQPfBrT6EMYZDxZUnBMWmnGKyj5Q7t+kVGeRsdjp2VdHs1B6q/favssb3TP4TBHT9BaI0FJRyhaMHrYnzgiJh9mKXXVUdKatic04xGx0hsSmnmF15JTi6KYgK8XeGxOBeEfSJDqFPdDBxYYEaFq6Qtw1ePtcauDjwArjyjS7fZKqhoJSr7F5qXTGkTremEbHpTbmsqoYtB46yuV5QbDt4lKraOuc2Qf4+9IkKsUIiKpg+0SEkRFmB0Sc6hIigzjv3VYc5eghePgeqKyDtOlj6hNV3dMHfu/SkiC0NBe31Uqo5ieOsJoZ/P2INoht1qy2nCQnw48x+UZzZL8r5XHVtHXvyS9l/uIz9heXsLyxj/+Ey9hWWs2p3IUcra044RmSwvxUQ9YIjITqEPlEhJEQFu37dis6mqhT+dYW11O0Nn1p32dVVW1PRR8bDuPvcXaHbaSgo1RJjfg37frQmH+ydBn3O6pDT+vv6kNIjnJQepzZbGWMoKq+2wuJwmTMw9heWs+3QUb7emktVTd0J+3QPD3SGRe9uwcSEBRIbFkBMaCCx4db3qBB//Hy74PratTXWIlUHN8BVC6xAAJj8iLWA1dePQkQ8nHGVW8t0N20+Uqqlyg9bE+fV1cCtSyA01t0VNamuzpBXUnlCWNT/+VBxhXOqj/pEICokgJjQAGLCAogNCyQ2LNDxONDxnPV8TFggoQG+nt/PYQx8eh+sfhkueBLOmnXi6zWV8Oal1qqG17wLSZMaPk4npn0KStkhZz28/Evon2G9eXTizsm6OkNxRTX5JVUUlFRSUFpFfknl8cclVRSUWo/zSyo5WlHT4HEC/XwcAWEFSZ/oEAbEhjIgzloDo1dEkPtHfC97Cv79MGT8Bs79c8PbVBTBK1PhyD64cZE1jUsXoqGglF3WvAof/xomPACTfu/uajpMZU0thaVVFJScFB6OMCkoqSLvqHVlUr+vI8jfhwGxYQyICyUp7sTvIQEd0IK98T2r2WjopXDJ3ManegEoyrbuSqqrhZu/sqZ36SI0FJSyizHwwR3w0wLraiHlHHdX5FGMMeQdrWRnXim78kvYmev4nldC1uFy6r/l9IoMOiUokuLC6Omqq4u938Pr063pWq77wJqNtzmHNsMrU6zJGGd9YQ327AI0FJSyU1WZtf7C0QNW/0K3Pu6uqFOoqK5lT0Epu/JK2Zlbwq78UnbmlbArr5SSelcXwf6+JMaGktQ9zNEUFUq/mFASooKJCQ1oWR9G3nbrU39oHMz60przq6V2L4U3L7HC5NqFXWLyRw0FpeyWnwkvToS4gXDjZ61a50GdyBhD7tFKZ0DU/5595MSri5AAXxKigq3xGfXGaiQ4fo4M9oeSXJg7GarL4eZ/Q1T/1hf187vw3ixIvdgan9JUs1NHKD8CfkFtDigdp6CU3WKT4eJnrenJv3wIzv+f9h+zrtb6z19WcPyrosi6l76uxrqtsq7m+OO6Wqg99rPjq7bea879qh2P6+0bEQ9nzID+49z+hici9IgIokdEEGOSTryr69jVxbG7p7IOH78Fd+XuwhOuMAB6BNXwhu9f6FeXy5uDnsNvCyREHXKER3DL+zFOv8y6EvzyD/Blb5jyN1f9uq1jDGz5CBb9zhps94uHbD2dhoJS7ZE6Hc6+E1Y8ay3lefplx1+rq7OmIy8rPP4GX1544ht+/dfKCq3bXmnl1bv4WMua+vhb33396j32dTzneM355QtbF1n9It36wYiZMPxqiExw6Z/HFYL8fRnUM4JBPSNOea3+WI2sw2VkFRQzbu1/kFy8kz8FP8i7G0OoWLfphH1iwwKIr3eVEd8tmPioYPpEBRPfLeTEFfdG32V1Pq94zvrbjL7T7l/3RMU58OlvYdun0HMYDDrf9lNq85FS7VVbDa9eCAd/tm5jPPZGX14Ipq7hfXwDICTG8RVd72fHV3D08eeDIh3Lk/off0P3qf8m38ZP+dXl1hTh616H3UsAgaRfWAEx6IKWdcp6EmOs1QZXzYXzn4CRt1id3iWV1tWF4yoj69iYjcNl5Bwpp7r2xPfAmNAA4qOCSYgKJr5bMAmRAUzZ+iA9sr6gbPpcQkZcbv/vUldnjan495+tq7pJv7c+fPi2/XO89iko1ZGKc6zFjGoqT32TD44+9c0/INSz5tk5vAfWzYf1/7KWXA2OgmFXwohroedQd1fXMsufhq/+aK0w+Mu/tGiX2jpD7tEKsg+Xk32k3BEaVnBkHykn+3A5lTV1BFLFGwF/4wzZyW3yBw5GpR8PDWeAWM1T3UL82zeYL3crfHwP7P8RBkyEC/8B0QPafjwHDQWlVOvV1cKuxbDuTdj6qbVOdq/h1tXD6ZdDcDd3V9iwje/DuzfCkF/Bpa+4rI/EGEN+SRXZR8o5dCiHkd9cTVBlPo/1+AerSnuQdbjMubDSMSEBvsSFBxIa4EdYkB9hgX6EBvoRFuh7ynP1nw/3r6X3hueIXPMMBIbBeX9Fzpjhsg8PGgpKqfYpK4QNb8O6N+DQRuvOl8EXWVcPHtA57eQci3AmXPuBvbePHtln3Yrs4w83f4UJ70VRebXzCsO62iijoKSK0soaSiprKK2qoaSihpLKWkorayivrj3lsOmylcf955Lsk8PC2gz+Un0tRT6RhAb4Eh7kT2igL6GBflyalsDMs/u1qXSPCAURmQL8E/AF5hpjHm9gmyuAR7B6134yxlzd1DE1FJTqYMbAgfXW1cOGd6CyyBrpO9zROe3OMRr5O6w36baMRWirAxtg3lTrNtcbF1l9Pq1QW2ecQVFeXEjk8v8kdtu/KA+JZ83pf2Rn5GgrTBxfJZW1lFRWU1pZywXDejFjZNtGWbs9FETEF9gOnAtkAauAGcaYzfW2SQHeBn5hjDksIt2NMblNHVdDQSk3cnZOvwG7v8PqnJ5kXT10dOd0Sa4VCNVlMOsriE7suHPv/AbmXw79HHNgtWWMyuaPYNH/g9JcOPsOmPSg1ddkE08YpzASyDTG7HIU9BYwHdhcb5tbgGeNMYcBmgsEpZSb+QfDsMutr8N7rI7pdfOt9vzgKDj9Cus23YR0ewOiqhT+daUVDDd+2rGBANZdWtOegQ9ugw/vhF+90PLmtOIcKwy2fgI9T4cZ9abx9gB2hkI8sL/e4yxg1EnbnAYgIsuxmpgeMcZ8fvKBRGQ2MBugb9+uM0GVUp1aVH/r0+2E+2HXt1bz0pp5sPIF8AuGPiMhcbz11XuEdUutK9TVwruzrCatK+dbfQnuMHwGFGfDN3+xFug555Gmt6+rgzWvWLeZ1lbBOX+2xj246u/iInaGQkNd5ie3VfkBKcBEIAFYKiJDjTFHTtjJmBeBF8FqPnJ9qUqpNvPxheTJ1lf5Edi73Br3sHup9YYJEBBmrVqXON5aya7nsLZNO24MfHY/bP/MGovQAYO5mjTuPijKgmX/sEaIj7yl4e1yt1oz6+5fAYkT4KKnXHKbqR3sDIUsoH4PVAKQ08A2K4wx1cBuEdmGFRKrbKxLKWWX4G5W38KgC6zHpfmwZ+nxkMj8yno+KBL6jbUCInE8xA1uWfPLD8/AqpdgzN2NvwF3JBErnI4etJqEwnvB4AuPv15TCUufhKV/t24zvXiONbWIJ41ROYmdHc1+WB3Nk4FsrDf6q40xm+ptMwWr8/l6EYkF1gHDjTEFjR1XO5qV6sSKD9QLiSVwZK/1fEiMdZtr4jjoPx5iU05949y0EN65wXMmqKuvqgxeu8i6dfe6j6DvKNi3Aj66B/K3WWM8zvsbhMW5rUS3333kKOJ84Cms/oJXjDGPicijwGpjzEdiDfv7OzAFqAUeM8a81dQxNRSU6kKO7LOuIHYvscKiONt6Pqzn8auI/uOsT+KvT7f6Jq770DOnsi7Nt6bqLj8MAy+A9W9CZF+48ElIOdfd1XlGKNhBQ0GpLsoYKNx1PCB2L4HSPOs18bHa4Gd91TFjEdqqcBfMPdea92rU7VZHfGCYu6sCPOOWVKWUajkRiEmyvtJvtEIib5sVDgc3WJ26nhwIYAXXLd9Y4zm6D3J3NW2ioaCU8kwi1htrZ3tzjWrbNBSewoN6apRSSrmbhoJSSiknDQWllFJOGgpKKaWcNBSUUko5aSgopZRy0lBQSinlpKGglFLKqdNNcyEiecDeNu4eC+S7sBy7daZ6O1Ot0Lnq7Uy1QueqtzPVCu2rt58xptkZ+TpdKLSHiKxuydwfnqIz1duZaoXOVW9nqhU6V72dqVbomHq1+UgppZSThoJSSiknbwuFF91dQCt1pno7U63QuertTLVC56q3M9UKHVCvV/UpKKWUapq3XSkopZRqgteEgohMEZFtIpIpIg+4u57GiEgfEVksIltEZJOI/NrdNbWEiPiKyDoR+cTdtTRFRLqJyLsistXxNx7t7pqaIiL/4fh3sFFEFoiIR61DKSKviEiuiGys91y0iHwlIjsc36PcWeMxjdT6P45/CxtEZKGIdHNnjfU1VG+9134rIsaxtr1LeUUoiIgv8CwwFUgFZohIqnuralQNcJ8xZjBwNnCnB9da36+BLe4uogX+CXxujBkEnIEH1ywi8cA9QLoxZijWWudXubeqU7yKtcZ6fQ8AXxtjUoCvHY89waucWutXwFBjzDBgO/D7ji6qCa9yar2ISB/gXGCfHSf1ilAARgKZxphdxpgq4C1guptrapAx5oAxZq3j56NYb1rx7q2qaSKSAFwAzHV3LU0RkQhgPPAygDGmyhhzxL1VNcsPCBYRPyAEyHFzPScwxiwBCk96ejrwmuPn14CLO7SoRjRUqzHmS2NMjePhCiChwwtrRCN/W4B/AL8DbOkQ9pZQiAf213uchYe/0QKISH9gBPCjeytp1lNY/0jr3F1IMwYAecA8R1PXXBEJdXdRjTHGZANPYH0iPAAUGWO+dG9VLdLDGHMArA85QHc319NSNwGfubuIpojINCDbGPOTXefwllCQBp7z6NuuRCQMeA/4jTGm2N31NEZELgRyjTFr3F1LC/gBacAcY8wIoBTPado4haMtfjqQCPQGQkVkpnur6ppE5CGsptv57q6lMSISAjwE/MnO83hLKGQBfeo9TsDDLsPrExF/rECYb4x53931NCMDmCYie7Ca5X4hIm+6t6RGZQFZxphjV17vYoWEpzoH2G2MyTPGVAPvA2PcXFNLHBKRXgCO77lurqdJInI9cCFwjfHse/STsD4g/OT4/5YArBWRnq48ibeEwiogRUQSRSQAq7PuIzfX1CAREaw27y3GmCfdXU9zjDG/N8YkGGP6Y/1dvzHGeOSnWWPMQWC/iAx0PDUZ2OzGkpqzDzhbREIc/y4m48Ed4/V8BFzv+Pl64EM31tIkEZkC3A9MM8aUubuephhjfjbGdDfG9Hf8f8sC0hz/rl3GK0LB0ZF0F/AF1n+qt40xm9xbVaMygGuxPnGvd3yd7+6iupC7gfkisgEYDvzVzfU0ynFF8y6wFvgZ6/+rR43AFZEFwA/AQBHJEpFZwOPAuSKyA+sumcfdWeMxjdT6DBAOfOX4v/a8W4usp5F67T+vZ18tKaWU6khecaWglFKqZTQUlFJKOWkoKKWUctJQUEop5aShoJRSyklDQakOJCITPX0mWeXdNBSUUko5aSgo1QARmSkiKx0Dml5wrBdRIiJ/F5G1IvK1iMQ5th0uIivqzckf5Xg+WUT+LSI/OfZJchw+rN6aDvMdo5WV8ggaCkqdREQGA1cCGcaY4UAtcA0QCqw1xqQB3wEPO3Z5HbjfMSf/z/Wenw88a4w5A2vOogOO50cAv8Fa22MA1ih2pTyCn7sLUMoDTQbOBFY5PsQHY03qVgf8n2ObN4H3RSQS6GaM+c7x/GvAOyISDsQbYxYCGGMqABzHW2mMyXI8Xg/0B5bZ/2sp1TwNBaVOJcBrxpgTVuESkT+etF1Tc8Q01SRUWe/nWvT/ofIg2nyk1Km+Bi4Tke7gXHO4H9b/l8sc21wNLDPGFAGHRWSc4/lrge8ca2BkicjFjmMEOubDV8qj6ScUpU5ijNksIn8AvhQRH6AauBNrUZ4hIrIGKMLqdwBreujnHW/6u4AbHc9fC7wgIo86jnF5B/4aSrWJzpKqVAuJSIkxJszddShlJ20+Ukop5aRXCkoppZz0SkEppZSThoJSSiknDQWllFJOGgpKKaWcNBSUUko5aSgopZRy+v88YfeJUX9+uQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(np.arange(len(train_losses)), train_losses, label='Train')\n",
    "plt.plot(np.arange(len(valid_losses)), valid_losses, label='Validation')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Test the network on the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of Seizure : 95% out of 438 cases\n",
      "Accuracy of TumorArea : 67% out of 471 cases\n",
      "Accuracy of HealthyArea : 42% out of 462 cases\n",
      "Accuracy of EyesClosed : 77% out of 493 cases\n",
      "Accuracy of EyesOpen : 62% out of 436 cases\n"
     ]
    }
   ],
   "source": [
    "class_correct = list(0. for i in range(len(classes)))\n",
    "class_total = list(0. for i in range(len(classes)))\n",
    "with torch.no_grad():\n",
    "    for data in testloader:\n",
    "        # get the inputs\n",
    "        inputs, targets = data\n",
    "\n",
    "        if cuda:\n",
    "            inputs, targets = inputs.cuda(), targets.cuda()\n",
    "\n",
    "        outputs = model(inputs)\n",
    "        _, predicted = torch.max(outputs.data, 1)        \n",
    "        c = (predicted == targets).squeeze()\n",
    "        for i, label in enumerate(targets):\n",
    "            class_correct[label] += c[i].item()\n",
    "            class_total[label] += 1\n",
    "\n",
    "for i in range(len(classes)):\n",
    "    print('Accuracy of %s : %2d%% out of %d cases' % (classes[i], 100 * class_correct[i] / class_total[i], class_total[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! We have completed the example of recurrent neural network!\n",
    "\n",
    "We will try another way of using RNN in the next chapter with a slightly more complicated example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1. Try to use different type of RNN cell (e.g. LSTM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2. How do you compare the result with the one by CNN in the previous chapter? What about computation time?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}