coursera_deep_neural_network/9.4.3CNN_Small_Image_batch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Convolutional Neural Network with Batch-Normalization </h1> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Objective for this Notebook<h3>    \n",
    "<h5> 1. Learn how to compare a Convolutional Neural Network using Batch Normalization with a regular Convolutional Neural Network  to classify handwritten digits from the MNIST database..</h5>     \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<b>This lab takes a long time to run so the results are given. You can run the notebook your self but it may take a long time.</b>\n",
    "<p>In this lab, we will compare a Convolutional Neural Network using Batch Normalization with a regular Convolutional Neural Network  to classify handwritten digits from the MNIST database. We will reshape the images to make them faster to process. </p>\n",
    "\n",
    "<ul>\n",
    "    <li><a href=\"#read_me\">Read me Batch Norm for Convolution Operation  </a></li>\n",
    "<li><a href=\"#Makeup_Data\">Get Some Data</a></li>\n",
    "<li><a href=\"#CNN\">Two Types of Convolutional Neural Network</a></li>\n",
    "<li><a href=\"#Train\">Define Criterion function, Optimizer and Train the Model</a></li>\n",
    "<li><a href=\"#Result\">Analyze Results</a></li>\n",
    "</ul>\n",
    "<p>Estimated Time Needed: <strong>25 min</strong></p>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"read_me\">Read me Batch Norm for Convolution Operation  </h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like a fully connected network, we create a <code>BatchNorm2d</code> object, but we apply it to the 2D convolution object. First, we create objects <code>Conv2d</code> object; we require the number of output channels, specified by the variable <code>OUT</code>.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>self.cnn1 = nn.Conv2d(in_channels=1, out_channels=OUT, kernel_size=5, padding=2) </code>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then create a Batch Norm  object for 2D convolution as follows:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>self.conv1_bn = nn.BatchNorm2d(OUT)</code>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameter out is the number of channels in the output. We can then apply batch norm  after  the convolution operation :\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>x = self.cnn1(x)</code>\n",
    "\n",
    "<p></p>\n",
    " <code> x=self.conv1_bn(x)</code>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Preparation</h2>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:41.213246Z",
     "start_time": "2021-07-16T08:53:40.632065Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "# Import the libraries we need to use in this lab\n",
    "\n",
    "# Using the following line code to install the torchvision library\n",
    "# !conda install -y torchvision\n",
    "\n",
    "# !pip install torchvision==0.9.1 torch==1.8.1 \n",
    "import torch \n",
    "import torch.nn as nn\n",
    "import torchvision.transforms as transforms\n",
    "import torchvision.datasets as dsets\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "def show_data(data_sample):\n",
    "    plt.imshow(data_sample[0].numpy().reshape(IMAGE_SIZE, IMAGE_SIZE), cmap='gray')\n",
    "    plt.title('y = '+ str(data_sample[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Makeup_Data\">Get the Data</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we create a transform to resize the image and convert it to a tensor :\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:42.938607Z",
     "start_time": "2021-07-16T08:53:42.931098Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "IMAGE_SIZE = 16\n",
    "\n",
    "composed = transforms.Compose([transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)), transforms.ToTensor()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the training dataset by setting the parameters <code>train </code> to <code>True</code>. We use the transform defined above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:43.801009Z",
     "start_time": "2021-07-16T08:53:43.785234Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=composed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the testing dataset by setting the parameters train  <code>False</code>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:44.592841Z",
     "start_time": "2021-07-16T08:53:44.579926Z"
    }
   },
   "outputs": [],
   "source": [
    "# Make the validating \n",
    "\n",
    "validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=composed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the data type is long.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:45.408076Z",
     "start_time": "2021-07-16T08:53:45.403992Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the data type for each element in dataset\n",
    "\n",
    "type(train_dataset[0][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each element in the rectangular tensor corresponds to a number representing a pixel intensity as demonstrated by the following image.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0110EN/notebook_images%20/chapter%206/6.2.1imagenet.png\" width=\"550\" alt=\"MNIST data image\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the fourth label \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:47.676624Z",
     "start_time": "2021-07-16T08:53:47.673682Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The label for the fourth data element\n",
    "\n",
    "train_dataset[3][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the fourth sample \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:48.870909Z",
     "start_time": "2021-07-16T08:53:48.761582Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAPb0lEQVR4nO3df6zV9X3H8ecLLuqgRnAiWmFVizHZtENCxHZEm7E6ZES6pCaQyaCQ3DSbm27dDGiymu2fdt3YWjdrUNloi9jMH5W4sspsGzMzqcjPi1hBZgWGgGMTsVvw4nt/nC/N4XrP5Z7vr3vu/bweyc39nvP9fM7nzffwut/v+Z7zPR9FBGaWnlFDXYCZDQ2H3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4bfCJP2FpJ2SeiXdN9T12OA4/FaGvcDdwD8PdSE2eA7/CCbpTyU90ee+r0v6WpnjRMSaiNgAvFvm41q1HP6R7dvAHEnjASR1AQuAb/bXWNIzkv6nxc8zNdZtNega6gKsOhFxSNLzwG3AQ8Ac4O2IeLlF+3l11mdDy3v+kW8NcHu2fDvwrSGsxTqIwz/yfRf4hKRrgHnA2lYNJW2QdKLFz4baKrZa+LB/hIuI/5P0OPAo8OOIeHOAtrfkGUPSGGA0jZ1Jl6TzgPcj4lSex7N6eM+fhjXAtVR3yP8Q8L/AQuDebHlRRWNZSeQv8xj5JP0S8CpwSUQcH+p6rDN4zz/CSRoF/DHwmINvzfyafwSTNA44DPyUxtt8Zj/nw36zRPmw3yxRtR72S/JhhlnFIkKDaec9v1miHH6zRDn8ZokqFH5JcyT9RNJeScvLKsrMqpf7rT5Jo4HXgM8AB4CXgIUR8coAfXzCz6xidZzwux7YGxH7IuIk8Bgwv8DjmVmNioT/MmB/0+0D2X1nkNQtabOkzQXGMrOSVf4+f0SsAlaBD/vNOkmRPf9BYErT7cnZfWY2DBQJ/0vAVZKukHQOjS+GXF9OWWZWtdyH/RHRK+kO4Ps0vsVldUTsKq0yM6tUrVf1+TW/WfX82X4zG5DDb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfrNEOfxmifIsvdYxJk2aVFu/ffv25RrrxIkTufp1Iu/5zRLl8JslyuE3S1Tu8EuaIumHkl6RtEvSnWUWZmbVKnLCrxf4YkRskXQ+8LKkjQNN12VmnSP3nj8iDkXElmz5XWA3/czYY2adqZS3+iRdDlwHbOpnXTfQXcY4ZlaewuGX9BHgCeCuiDjed72n6zLrTIXO9ksaQyP4ayPiyXJKMrM6FDnbL+ARYHdErCyvJDOrQ5E9/68Bi4Bfl7Qt+5lbUl1mVrEic/X9GzCoaYHMrPP4E35mifJEnVaJCRMmtN3ngQceyDXWzJkz2+6zaNGiXGO98MILufrVyRN1mtmAHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5em6bECjRuXbP0yfPr3tPrNmzco11tatW9vuc/To0VxjjSTe85slyuE3S5TDb5aowuGXNFrSVknPlFGQmdWjjD3/nTRm6zGzYaTo9/ZPBn4LeLiccsysLkX3/H8L3A18UEItZlajIpN2zAOORMTLZ2nXLWmzpM15xzKz8hWdtONWSW8Aj9GYvOPbfRtFxKqImBERMwqMZWYlKzJF94qImBwRlwMLgB9ExO2lVWZmlfL7/GaJKuWz/RHxI+BHZTyWmdXDe36zRHm6LhvQ1KlTc/VbvXp1230mTpyYa6yFCxe23WfHjh25xvrgg85/V9vTdZnZgBx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXKc/UlYuzYsbn63Xbbbbn6TZs2re0+99xzT66xenp62u4zHK7Oq5r3/GaJcvjNEuXwmyWq6Iw94yU9LulVSbslfbKswsysWkVP+H0N+JeI+Jykc4B8Z5XMrHa5wy/pAuBGYAlARJwETpZTlplVrchh/xXAUeAfsim6H5Y0rm8jT9dl1pmKhL8LmA58IyKuA94Dlvdt5Om6zDpTkfAfAA5ExKbs9uM0/hiY2TBQZK6+t4D9kq7O7poNvFJKVWZWuaJn+/8AWJud6d8HfL54SWZWh0Lhj4htgF/Lmw1DvrBnGBo1qv1XazfddFOusZYuXZqr34YNG9rus27dulxj9fb25uqXOn+81yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuWr+oahqVOntt1nxYoVucaKiFz9Vq5c2XafY8eO5RrL8vGe3yxRDr9Zohx+s0QVna7rjyTtktQjaZ2k88oqzMyqlTv8ki4D/hCYERHXAKOBBWUVZmbVKnrY3wX8gqQuGvP0/WfxksysDkW+t/8g8FfAm8Ah4J2IeLZvO0/XZdaZihz2TwDm05iz76PAOEm3923n6brMOlORw/7fAP4jIo5GxPvAk8CnyinLzKpWJPxvAjdIGitJNKbr2l1OWWZWtSKv+TfRmJxzC7Aze6xVJdVlZhUrOl3Xl4AvlVSLmdXIn/AzS5Sv6htCXV35Nv+SJUva7jNz5sxcYz366KO5+vX09LTdJ+8VhJaP9/xmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5Qv7BlC06ZNy9Vv2bJlbffZvn17rrHuv//+XP3ee++9XP2sPt7zmyXK4TdLlMNvlqizhl/SaklHJPU03XehpI2S9mS/J1RbppmVbTB7/n8E5vS5bznwXERcBTyX3TazYeSs4Y+I54Fjfe6eD6zJltcAny25LjOrWN63+iZFxKFs+S1gUquGkrqB7pzjmFlFCr/PHxEhqeU3L0bEKrLv8x+onZnVK+/Z/sOSLgXIfh8pryQzq0Pe8K8HFmfLi4GnyynHzOoymLf61gH/Dlwt6YCkZcCXgc9I2kNjws4vV1ummZXtrK/5I2Jhi1WzS67FzGrkT/iZJcpX9ZXg3HPPzdVv3rx5ufpdfPHFbfd58MEHc421a9euXP2s83nPb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfrNE+cKeEkycODFXv7wX9uzZs6ftPk8/ne/7Vk6ePJmrn3U+7/nNEuXwmyXK4TdLVN7pur4q6VVJOyQ9JWl8tWWaWdnyTte1EbgmIj4BvAasKLkuM6tYrum6IuLZiOjNbr4ITK6gNjOrUBmv+ZcCG1qtlNQtabOkzSWMZWYlKfQ+v6R7gV5gbas2nq7LrDPlDr+kJcA8YHZEONRmw0yu8EuaA9wN3BQRPyu3JDOrQ97puv4OOB/YKGmbpHxfCm9mQybvdF2PVFCLmdXIn/AzS5Sv6iuBpFz9Ro8enavf/v372+6ze/fuXGP5XO7I5T2/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslSnVetTVSv8NvzJgxufpde+21ufqNGtX+3+ytW7fmGuvUqVO5+tnQiYhBXWbqPb9Zohx+s0Tlmq6rad0XJYWki6opz8yqkne6LiRNAW4G3iy5JjOrQa7pujJ/Q+Pru0fkSTyzkS7v9/bPBw5GxPazfX+dpG6gO884ZladtsMvaSxwD41D/rPydF1mnSnP2f6PA1cA2yW9QWOG3i2SLimzMDOrVtt7/ojYCVx8+nb2B2BGRLxdYl1mVrG803WZ2TCXd7qu5vWXl1aNmdXGn/AzS5Qv7DEbYXxhj5kNyOE3S5TDb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaJyfYFnAW8DP22x7qJs/VBzHWdyHWfq9Do+NtgHqPWS3oFI2hwRM1yH63Ad9dThw36zRDn8ZonqpPCvGuoCMq7jTK7jTCOmjo55zW9m9eqkPb+Z1cjhN0tUreGXNEfSTyTtlbS8n/XnSvpOtn6TpMsrqGGKpB9KekXSLkl39tPm05LekbQt+/mzsutoGusNSTuzcTb3s16Svp5tkx2Sppc8/tVN/85tko5LuqtPm8q2h6TVko5I6mm670JJGyXtyX5PaNF3cdZmj6TFFdTxVUmvZtv9KUnjW/Qd8DksoY77JB1s2v5zW/QdMF8fEhG1/ACjgdeBK4FzgO3AL/dp83vAg9nyAuA7FdRxKTA9Wz4feK2fOj4NPFPTdnkDuGiA9XOBDYCAG4BNFT9HbwEfq2t7ADcC04Gepvv+ElieLS8HvtJPvwuBfdnvCdnyhJLruBnoypa/0l8dg3kOS6jjPuBPBvHcDZivvj917vmvB/ZGxL6IOAk8Bszv02Y+sCZbfhyYrbPNAd6miDgUEVuy5XeB3cBlZY5RsvnAN6PhRWC8pEsrGms28HpEtPoUZuki4nngWJ+7m/8frAE+20/X3wQ2RsSxiPhvYCMwp8w6IuLZiOjNbr5IY1LaSrXYHoMxmHydoc7wXwbsb7p9gA+H7udtso3+DvCLVRWUvay4DtjUz+pPStouaYOkX6mqBiCAZyW9LKm7n/WD2W5lWQCsa7Guru0BMCkiDmXLbwGT+mlT53YBWErjCKw/Z3sOy3BH9vJjdYuXQW1vj2RP+En6CPAEcFdEHO+zeguNQ99fBe4HvlthKbMiYjpwC/D7km6scKyWJJ0D3Ar8Uz+r69weZ4jGMe2Qvh8t6V6gF1jboknVz+E3gI8D04BDwF+X8aB1hv8gMKXp9uTsvn7bSOoCLgD+q+xCJI2hEfy1EfFk3/URcTwiTmTL3wPGSLqo7Dqyxz+Y/T4CPEXj8K3ZYLZbGW4BtkTE4X5qrG17ZA6ffmmT/T7ST5tatoukJcA84HeyP0QfMojnsJCIOBwRpyLiA+ChFo/f9vaoM/wvAVdJuiLbyywA1vdpsx44fdb2c8APWm3wvLJzCI8AuyNiZYs2l5w+1yDpehrbqYo/QuMknX96mcYJpp4+zdYDv5ud9b8BeKfpkLhMC2lxyF/X9mjS/P9gMfB0P22+D9wsaUJ2GHxzdl9pJM0B7gZujYiftWgzmOewaB3N53h+u8XjDyZfZyrjDGUbZzLn0ji7/jpwb3bfn9PYuADn0Tjs3Av8GLiyghpm0TiM3AFsy37mAl8AvpC1uQPYReOM6YvApyraHldmY2zPxju9TZprEfD32TbbCcyooI5xNMJ8QdN9tWwPGn9wDgHv03iduozGeZ7ngD3AvwIXZm1nAA839V2a/V/ZC3y+gjr20ngdffr/yel3oj4KfG+g57DkOr6VPfc7aAT60r51tMrXQD/+eK9ZopI94WeWOoffLFEOv1miHH6zRDn8Zoly+M0S5fCbJer/AZmBYY4yDr48AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The image for the fourth data element\n",
    "show_data(train_dataset[3])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fourth sample is a \"1\".\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"CNN\">Build a Two Convolutional Neural Network Class</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a Convolutional Network class with two Convolutional layers and one fully connected layer. Pre-determine the size of the final output matrix. The parameters in the constructor are the number of output channels for the first and second layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:52.914700Z",
     "start_time": "2021-07-16T08:53:52.892293Z"
    }
   },
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    \n",
    "    # Contructor\n",
    "    def __init__(self, out_1=16, out_2=32):\n",
    "        super(CNN, self).__init__()\n",
    "        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2)\n",
    "        self.maxpool1=nn.MaxPool2d(kernel_size=2)\n",
    "\n",
    "        self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, stride=1, padding=2)\n",
    "        self.maxpool2=nn.MaxPool2d(kernel_size=2)\n",
    "        self.fc1 = nn.Linear(out_2 * 4 * 4, 10)\n",
    "    \n",
    "    # Prediction\n",
    "    def forward(self, x):\n",
    "        x = self.cnn1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.cnn2(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = self.fc1(x)\n",
    "        return x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a Convolutional Network class with two Convolutional layers and one fully connected layer. But we add Batch Norm for the convolutional layers. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:53.838948Z",
     "start_time": "2021-07-16T08:53:53.818973Z"
    }
   },
   "outputs": [],
   "source": [
    "class CNN_batch(nn.Module):\n",
    "    \n",
    "    # Contructor\n",
    "    def __init__(self, out_1=16, out_2=32,number_of_classes=10):\n",
    "        super(CNN_batch, self).__init__()\n",
    "        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2)\n",
    "        self.conv1_bn = nn.BatchNorm2d(out_1)\n",
    "\n",
    "        self.maxpool1=nn.MaxPool2d(kernel_size=2)\n",
    "        \n",
    "        self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, stride=1, padding=2)\n",
    "        self.conv2_bn = nn.BatchNorm2d(out_2)\n",
    "\n",
    "        self.maxpool2=nn.MaxPool2d(kernel_size=2)\n",
    "        self.fc1 = nn.Linear(out_2 * 4 * 4, number_of_classes)\n",
    "        self.bn_fc1 = nn.BatchNorm1d(10)\n",
    "    \n",
    "    # Prediction\n",
    "    def forward(self, x):\n",
    "        x = self.cnn1(x)\n",
    "        x=self.conv1_bn(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.cnn2(x)\n",
    "        x=self.conv2_bn(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = self.fc1(x)\n",
    "        x=self.bn_fc1(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function to train the model \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:56.914524Z",
     "start_time": "2021-07-16T08:53:56.907390Z"
    }
   },
   "outputs": [],
   "source": [
    "def train_model(model,train_loader,validation_loader,optimizer,n_epochs=4):\n",
    "    \n",
    "    #global variable \n",
    "    N_test=len(validation_dataset)\n",
    "    accuracy_list=[]\n",
    "    loss_list=[]\n",
    "    for epoch in range(n_epochs):\n",
    "        for x, y in train_loader:\n",
    "            model.train()\n",
    "            optimizer.zero_grad()\n",
    "            z = model(x)\n",
    "            loss = criterion(z, y)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            loss_list.append(loss.data)\n",
    "\n",
    "        correct=0\n",
    "        #perform a prediction on the validation  data  \n",
    "        for x_test, y_test in validation_loader:\n",
    "            model.eval()\n",
    "            z = model(x_test)\n",
    "            _, yhat = torch.max(z.data, 1)\n",
    "            correct += (yhat == y_test).sum().item()\n",
    "        accuracy = correct / N_test\n",
    "        accuracy_list.append(accuracy)\n",
    "     \n",
    "    return accuracy_list, loss_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Train\">Define the Convolutional Neural Network Classifier, Criterion function, Optimizer and Train the Model</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 16 output channels for the first layer, and 32 output channels for the second layer \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:58.072015Z",
     "start_time": "2021-07-16T08:53:58.066709Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create the model object using CNN class\n",
    "model = CNN(out_1=16, out_2=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the loss function, the optimizer and the dataset loader \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:53:58.823352Z",
     "start_time": "2021-07-16T08:53:58.812342Z"
    }
   },
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "learning_rate = 0.1\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=100)\n",
    "validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model and determine validation accuracy technically test accuracy **(This may take a long time)**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:55:33.179530Z",
     "start_time": "2021-07-16T08:53:59.597095Z"
    }
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "accuracy_list_normal, loss_list_normal=train_model(model=model,n_epochs=10,train_loader=train_loader,validation_loader=validation_loader,optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat the Process for the model with  batch norm \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:57:26.986298Z",
     "start_time": "2021-07-16T08:55:33.180804Z"
    }
   },
   "outputs": [],
   "source": [
    "model_batch=CNN_batch(out_1=16, out_2=32)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "learning_rate = 0.1\n",
    "optimizer = torch.optim.SGD(model_batch.parameters(), lr = learning_rate)\n",
    "accuracy_list_batch, loss_list_batch=train_model(model=model_batch,n_epochs=10,train_loader=train_loader,validation_loader=validation_loader,optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Result\">Analyze Results</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss with both networks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:57:27.183230Z",
     "start_time": "2021-07-16T08:57:26.987697Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f8b743ec790>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEWCAYAAAB2X2wCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd5xU1fn48c+zBZAmSldAIFKUItIUFYKxoRjsCSZRiTUqJn5tQYlGo0YTUfOzBYkSWzBGECXYiSAgSpVepLgovUmHZdl9fn+cOztlZ2Znd2d35s4+79frvuaWc+89Z3fmmTPnnnuuqCrGGGP8LyvVGTDGGJMcFtCNMSZDWEA3xpgMYQHdGGMyhAV0Y4zJEBbQjTEmQ1hAN9WGiOSJyNmpzocxlcUCujHGZAgL6MYYkyEsoJtqR0RqisjfRGSDN/1NRGp62xqJyEQR2SkiO0Rkmohkedt+LyLrRWSPiKwQkbNSWxJjwuWkOgPGpMBw4FSgG6DAe8AfgPuBO4F1QGMv7amAikgHYCjQS1U3iEhrILtqs21MfFZDN9XRL4E/qeoWVd0KPARc5W0rAJoDx6lqgapOUzfgUSFQEzhRRHJVNU9VV6ck98bEYAHdVEfHAGtDltd66wCeAFYBn4jIGhEZBqCqq4DbgQeBLSLybxE5BmPSiAV0Ux1tAI4LWW7lrUNV96jqnaraFhgE3BFoK1fVMap6hrevAn+p2mwbE58FdFMdvQn8QUQai0gj4AHgDQARuVBEjhcRAXbhmlqKRKSDiPzEu3h6EDgAFKUo/8ZEZQHdVEePAHOAhcAiYJ63DqAdMAnYC3wJvKCqk3Ht548D24BNQBPg3qrNtjHxiT3gwhhjMoPV0I0xJkNYQDfGmAxhAd0YYzKEBXRjjMkQpd76LyK1gKm4q/w5wFhV/WNEmprAa0APYDvwc1XNi3fcRo0aaevWrcuXa2OMqabmzp27TVUbR9uWyFgu+cBPVHWviOQC00XkQ1X9KiTNdcAPqnq8iAzG3XDx83gHbd26NXPmzEmwCMYYYwBEZG2sbaU2uaiz11vM9abIvo4XAa9682OBs7wbM4wxxlSRhNrQRSRbROYDW4BPVXVmRJJjge8BVPUw7g67hlGOc6OIzBGROVu3bq1Yzo0xxoRJKKCraqGqdgNaAL1FpHN5Tqaqo1S1p6r2bNw4ahOQMcaYcirTeOiqulNEJgMDgMUhm9YDLYF1IpIDHIm7OGqMqQIFBQWsW7eOgwcPpjorJklq1apFixYtyM3NTXifRHq5NAYKvGB+BHAOJUeZmwBcgxv74nLgM7UxBYypMuvWraNevXq0bt0au3zlf6rK9u3bWbduHW3atEl4v0SaXJoDk0VkITAb14Y+UUT+JCKDvDQvAw1FZBVwBzCsjPk3xlTAwYMHadiwoQXzDCEiNGzYsMy/uEqtoavqQuDkKOsfCJk/CFxRpjMbY5LKgnlmKc//0393ii5ZAg88AFu2pDonxhiTVvwX0JcuhYcfBuv2aExaqVu3bqqzUCmmTJnChRdemOpsJMR/Ad0YYxJ0+PDhVGehSvk3oFsnGmPSkqpy991307lzZ7p06cJbb70FwMaNG+nXrx/dunWjc+fOTJs2jcLCQoYMGVKc9umnny5xvCFDhvDb3/6W0047jbZt2zJ27Ni455kyZQp9+/Zl0KBBnHjiieTl5dGxY0eGDBlC+/bt+eUvf8mkSZM4/fTTadeuHbNmzQJg1qxZ9OnTh5NPPpnTTjuNFStWxC1nYWEhd911F507d6Zr1648++yzgBvW5I9//CPdu3enS5cuLF++HIAHH3yQa6+9lv79+9O2bVueeeaZ5PzBQ5SpH3pasAs/xsR1++0wf35yj9mtG/ztb4mlfeedd5g/fz4LFixg27Zt9OrVi379+jFmzBjOO+88hg8fTmFhIfv372f+/PmsX7+exYvdbS07d+6MesyNGzcyffp0li9fzqBBg7j88stjngdg3rx5LF68mDZt2pCXl8eqVat4++23GT16NL169WLMmDFMnz6dCRMm8Oc//5l3332Xjh07Mm3aNHJycpg0aRL33Xcf48aNi1nOUaNGkZeXx/z588nJyWHHjh3F2xo1asS8efN44YUXGDFiBC+99BIAy5cvZ/LkyezZs4cOHTpw8803l6mfeWn8F9CNMWlt+vTpXHnllWRnZ9O0aVN+/OMfM3v2bHr16sW1115LQUEBF198Md26daNt27asWbOG2267jYEDB3LuuedGPebFF19MVlYWJ554Ips3b457nvr169O7d++w/ttt2rShS5cuAHTq1ImzzjoLEaFLly7k5eUBsGvXLq655hpWrlyJiFBQUBC3nJMmTeI3v/kNOTkujB599NHF2y699FIAevTowTvvvFO8fuDAgdSsWZOaNWvSpEkTNm/eTIsWLcr4F47NAroxGSbRmnRV69evH1OnTuX9999nyJAh3HHHHVx99dUsWLCAjz/+mJEjR/Kf//yH0aNHl9i3Zs2axfOJ3LNYp06dmPtnZWUVL2dlZRW3s99///2ceeaZjB8/nry8PPr371+eYoadLzs7O6wdPzQfkduSwdrQjTFJ1bdvX9566y0KCwvZunUrU6dOpXfv3qxdu5amTZtyww03cP311zNv3jy2bdtGUVERl112GY888gjz5s2r8HnKa9euXRx77LEAvPLKK6WmP+ecc3jxxReLg3Jok0uq+K+Gbm3oxqS1Sy65hC+//JKTTjoJEeGvf/0rzZo149VXX+WJJ54gNzeXunXr8tprr7F+/Xp+/etfU1RUBMBjjz1W4fMELkKW1T333MM111zDI488wsCBA0tNf/311/PNN9/QtWtXcnNzueGGGxg6dGi5zp0skqohV3r27KnlesDFuHFw+eWwcCF4bWLGVHfLli3jhBNOSHU2TJJF+7+KyFxV7RktvTW5GGNMhvBfQLcmF2OMicp/Ad0YY0xU/g3o1uRijDFh/BfQrcnFGGOi8l9AN8YYE5UFdGNMUlTW8LlDhgwpHpArEXl5eYwZM6bUdK1bt2bbtm0VyVra8W9AtzZ0Y0wUiQb0TOS/gG5t6MaktWQPnwtuIKyePXvSvn17Jk6cCLjA3bdvX7p370737t2ZMWMGAMOGDWPatGl069aNp59+OuYwtwDPPvtsiWFuQ6XjELnx+O/Wf2NMfCkeP7cyhs/Ny8tj1qxZrF69mjPPPJNVq1bRpEkTPv30U2rVqsXKlSu58sormTNnDo8//jgjRowoDvx///vfyzzMbUA6DpEbj/9q6AHW5GJMWoo3fO4///lPHnzwQRYtWkS9evXChs/96KOPqF+/ftRj/uxnPyMrK4t27drRtm1bli9fTkFBATfccANdunThiiuuYOnSpVH3nTRpEjfddFOpw9wGhtGt6L6BIXIbNWpUPERuVfFfDd2aXIyJL03Hz63I8LkS8bkXEZ5++mmaNm3KggULKCoqolatWmXOU6xhbiuyb2UPkRuPf2voxpi0VBnD57799tsUFRWxevVq1qxZQ4cOHdi1axfNmzcnKyuL119/ncLCQgDq1avHnj17ivetyDC36ThEbjz+q6EHWJOLMWmpMobPbdWqFb1792b37t2MHDmSWrVqccstt3DZZZfx2muvMWDAgOKHWnTt2pXs7GxOOukkhgwZwm233VbuYW7TcYjcePw3fO6ECXDRRTB3LnTvnvyMGeNDNnxuZkr68Lki0lJEJovIUhFZIiK/i5Kmv4jsEpH53vRAuUtgjDGmXBJpcjkM3Kmq80SkHjBXRD5V1chLytNU9cLkZzEGa3IxxpgwpdbQVXWjqs7z5vcAy4BjKztjMVkvF2OiSlXzqakc5fl/lqmXi4i0Bk4GZkbZ3EdEFojIhyLSKcb+N4rIHBGZs3Xr1jJn1hgTXa1atdi+fbsF9Qyhqmzfvr3MXTET7uUiInWBccDtqro7YvM84DhV3SsiFwDvAu2iZHIUMArcRdEy5dQYE1OLFi1Yt24dVlHKHLVq1aJFixZl2iehgC4iubhg/i9VfSdye2iAV9UPROQFEWmkqpU3lJnVRIwplpubS5s2bVKdDZNiifRyEeBlYJmqPhUjTTMvHSLS2zvu9mRmNORklXJYY4zxu0Rq6KcDVwGLRCQw4s99QCsAVR0JXA7cLCKHgQPAYLXGPGOMqVKlBnRVnQ7ErRar6nPAc8nKVELs+8IYY8L4bywXa3Ixxpio/BfQjTHGROXfgG5NLsYYE8Z/Ad2aXIwxJir/BXRjjDFRWUA3xpgM4d+Abm3oxhgTxncBfcaXrg197NgUZ8QYY9KM7wJ6s2bu9YkRqc2HMcakG98F9MD4Q4Jaq4sxxoTwXUCXrGC3xc2bU5gRY4xJM74L6KGaN091DowxJn34NqAL1t5ijDGh/BfQ7U5RY4yJyn8B3RhjTFQW0I0xJkP4NqD/8kqljA/ENsaYjOa/gO61odetCwcPQkFBivNjjDFpwn8B3VO7jnvdsye1+TDGmHTh24Bet7brtrh7d4ozYowxacJ/Ad1rcsnxHm/9/fcpzIsxxqQR/wV0z1FHudf8/NTmwxhj0oVvA3rtI1yTi10UNcYYx38B3WtyyfaaXA4fTmFejDEmjfgvoHtyst2r1dCNMcYpNaCLSEsRmSwiS0VkiYj8LkoaEZFnRGSViCwUke6Vk92gbAvoxhgTJieBNIeBO1V1nojUA+aKyKequjQkzflAO286Bfi791ppcrKtDd0YY0KVWkNX1Y2qOs+b3wMsA46NSHYR8Jo6XwENRKRyRiuP6LY4c2alnMUYY3ynTG3oItIaOBmIDKPHAqE9wtdRMugnVSCgP/dcZZ7FGGP8I+GALiJ1gXHA7aparvszReRGEZkjInO2bt1ankMUCzS5GGOMcRIK6CKSiwvm/1LVd6IkWQ+0DFlu4a0Lo6qjVLWnqvZs3LhxefJb3ORiD4g2xphwifRyEeBlYJmqPhUj2QTgaq+3y6nALlXdmMR8llCnbmUe3Rhj/CeRXi6nA1cBi0RkvrfuPqAVgKqOBD4ALgBWAfuBXyc/q+Fq5iqtW0O/fpV9JmOM8YdSA7qqTgfiPshTVRW4NVmZiivkmaK5uXanqDHGBPj2TlFwPV2sH7oxxjj+DeiqVkM3xpgQ/gvoIU0uVkM3xpgg/wX0ELm5FtCNMSbA1wE9J8eaXIwxJsC/Ad1rQ7caujHGOP4L6BFt6FZDN8YYx38BPYTV0I0xJsi/AV3VaujGGBPCfwE94k5Rq6EbY4zjv4AewmroxhgT5N+Abr1cjDEmjP8CuvVyMcaYqPwX0ENYDd0YY4J8HdCthm6MMUGJPOAiPakycqSb3bcP6tRJbXaMMSbV/FdDl5LP2vjhhxTkwxhj0oz/AnoUWRlRCmOMqRj/hkLV4tkolXZjjKl2/BfQo0RvC+jGGOPHgB7i//7PvVqTizHG+Dmgq9KunZstKkptVowxJh34L6CHtK8EauYhzenGGFNt+S+ghwjEdquhG2OMzwO61dCNMSao1IAuIqNFZIuILI6xvb+I7BKR+d70QPKzGYWq1dCNMSZEIrf+vwI8B7wWJ800Vb0wKTkqjbWhG2NMVKXW0FV1KrCjCvJSZoGAbjV0Y4xJXht6HxFZICIfikinJB0zPmtyMcaYMMkYbXEecJyq7hWRC4B3gXbREorIjcCNAK1atSrf2azJxRhjoqpwDV1Vd6vqXm/+AyBXRBrFSDtKVXuqas/GjRtX9NRWQzfGmBAVDugi0kzEhVYR6e0dc3tFj1sqVauhG2NMiFKbXETkTaA/0EhE1gF/BHIBVHUkcDlws4gcBg4Ag1UrMcSGNLkEZq+4At5/H1q2rLSzGmNM2is1oKvqlaVsfw7XrbHKBWroixbBPffAm2+mIhfGGJMefH2nqDW1GGNMkH8DuipLl4YtGmNMtea/gG5PszDGmKj8F9CNMcZE5d+Abm0sxhgTxn8B3ZpcjDEmKv8F9BArV6Y6B8YYkz78G9BVadYsbNEYY6o1/wX0KHeKGmOM8WNADxEa0C24G2OqO/8G9JDx0L1FY4yp1vwX0K0qbowxUfkvoBtjjInK1wG9Y8dU58AYY9KHfwO6KpddlupMGGNM+vBfQA9pQw+9EFpQkIK8GGNMGvFfQI9h/PhU58AYY1LLvwFd1boqGmNMCP8F9BhNLsYYU935L6CHqFMn1Tkwxpj04d+ArkqtWiVWGWNMteW/gB7nTlEL6MaY6sx/AT0OC+jGmOrMAroxxmQI/wb0KNHbAroxpjrzX0C30RaNMSaqUgO6iIwWkS0isjjGdhGRZ0RklYgsFJHuyc9mYqyGboypzhKpob8CDIiz/XygnTfdCPy94tmKI1BDLyoqsckCujGmOis1oKvqVGBHnCQXAa+p8xXQQESaJyuDJeTkuNfCwhKbLKAbY6qzZLShHwt8H7K8zltXgojcKCJzRGTO1q1by3e2QEA/fLjEJgvoxpjqrEoviqrqKFXtqao9GzduXL6DWEA3xpiokhHQ1wMtQ5ZbeOsqhwV0Y4yJKhkBfQJwtdfb5VRgl6puTMJxo7OAbowxUSXSbfFN4Eugg4isE5HrROQ3IvIbL8kHwBpgFfAP4JZKyy2UCOgXXhjcZAHdGFOd5ZSWQFWvLGW7ArcmLUeliQjotWuH5qXKcmGMMWnHf3eKRgT0nJCvJAvoxpjqzPcBPTs7uMkCujGmOvN9QM8KKYEFdGNMdeb7gG41dGOMcfwX0LOy3HguXkC/4ILgJgvoxpjqzH8BHVzk3rcPgPPOC19tjDHVlT8DOsBTTwHhw6P//vewf3+K8mOMMSnm34AexejR8OSTqc6FMcakhu8DeuQDjPLzU5MPY4xJtYwL6MYYU11ZQDfGmAxR6lguaalv3+IO6NazxRhjHH/W0HNyih9BF/lo0UcfdbX2999PQb6MMSaF/BnQs7OLbyzKzY2e5JVXgvOHDkFBQeVnyxhjUsmfAT2khl6jRvQkoWO81KwJ7dpVQb6MMSaF/BvQozyxKFTkxdK1aysxP8YYkwb8GdBDmlxisd4vxpjqxp8BPaTJJRYL6MaY6saf3RbHjSs1iQV0Y0x1488aegKWL091DowxpmplbECfNw927051Lowxpur4M6D365dQsgMHKjkfxhiTRvwZ0AOdz/PyUpoNY4xJJ/4M6JMmudfJkwH4y1/KtvtXX8F331U8G/n5sHRpxY9jjDHJ4M+AHrB3LwDXXlu23fr0geOOq/jpb7kFOnWCzZsrfixjjKmohAK6iAwQkRUiskpEhkXZPkREtorIfG+6PvlZjeKuu4Dw2/yr0tSp7tUuvhpj0kGpoVBEsoHngfOBE4ErReTEKEnfUtVu3vRSkvMZ7oor3OuhQ0DsgL52bcnRGI0xJlMlUrftDaxS1TWqegj4N3BR5WarFPXrB+fff586lw/gZl4okeyUU+Dxx5Nzyvx8d7PSCyGnCYzFbjcxGWPSQSIB/Vjg+5Dldd66SJeJyEIRGSsiLaMdSERuFJE5IjJn69at5ciux6uZAzBzJrn/+5gXuDVq0ilTyn+aUDt3uteHHgquS2VALypyY77bAz6MMQHJan3+L9BaVbsCnwKvRkukqqNUtaeq9mzcuHH5zxYa0Fevjpt0z57ynyZRBw9W/jkiPf88XHgh/PvfVX9uY0x6SiSgrwdCa9wtvHXFVHW7quZ7iy8BPZKTvRjuuy84P2ZM3KRffRW+vGNH8rPTuXPyjxnqq69g9OjwdYHhgNevL5neGFM9JRLQZwPtRKSNiNQABgMTQhOISPOQxUHAsuRlMYquXcu960Wpbf0vlz594Lrrom9LRZPLz34G999f9ec1xsRX6miLqnpYRIYCHwPZwGhVXSIifwLmqOoE4LciMgg4DOwAhlRinsstWlv3wYMwZ467gJqfD3XrVn2+yiPQsycVAf3tt93rww9X/bmNMbEl1Iauqh+oantV/ZGqPuqte8AL5qjqvaraSVVPUtUzVdU3Yx3ecw/07Qtt20K9emXbN5UXJANfTpWZh8JCdxH4hx8q7xzGmOTx952iEWqQX3qiCIsWudd16+KnixY406GHSWXm4YMP4MEH4be/rbxzlMXChTBwYPg1cWNMkH8D+uDBJVbNphfn8nEKMlP1qqKGHgic+/dX3jnK4oYb3JfMvHmpzokx6cm/Af3550us6soiPmZAwof49tv4/dR37oTZs8uRtyoQGJAsU29q2rat5Bg5qSzr44+X7DFlTLrx5yPoAI4+usKHOP74kuu6dIGePd33xcCBMGNGqc+jTqnKDHKpbFIK3KaQLk1d996bunMbkyj/BvQkiDbOy+LFbnrllfB0gcC5ZYt7PXjQhmOvalXRzGSMn/m3yaUKFRWVDCLXV814klFV14BWVT173nsvff7Go0ZBixapzoXxC38H9C5doq6+nn8AyftEFhaWXPf55+HLU6e6J+MVFCTttDFVdbBJl+GBq6IN/Zln4OKL02dIhZtuSs3dwPv2ub/3qFFVf25Tfv4O6B06RF39D26kD18m7TTRmmYiuzlefTVMmxa+Pj8fPv0UJkxwTTRPPQUVGZMsEMjHjw+uq8wgF2heCjwgKtWqooYeeJLVxo2Vdw4/2LTJvSZrtNKy2L+/coboqA78HdB/+tOYm+qwL2mniVZDjxRt5MW774Zzz3XDDfTrB3feCddcE9y+a1d4jV4Vvv/edcvbsSP6eVXh8svLV46ySreHbFdFQJ87172momYqAi2jjlNa9b/KUtmjqHt3aNgwdef3M38H9KuuirlpkDfczI9YRQMqdqtjIgE9IPSDsCxkRJtA98ddu4LrGjSAyy5z84895m7nb9UKevRwb2jvgUzFVKv2gx35oR41yj1yL5OtWBH+WtVi3eBWmQ9qmTHD/a9Xriy5rbLfb59/Hv45gdT97X/4wX32vvkmNedPBn8H9DjViNt4DoBVtGMuPejEYlpSvidDBz5M5/ERRxH9t2Dgp/qmTcEbchL5MPz3v+71xRdLbhs7Nny5rAH9iiuiH7e8bropNQ/FVnX97gPNVZUZZFL1OMPSVGaZ33jDvX7ySXBdVfUo6t8fToz2/LNK9uKLroyhw1pMmOB+HT/ySOWeWyT8l3oypenbtwwiv95DKO5d2ZZvWUwXvuM48qnBYN4s0ykmTgTZ+QMfcT7vcnFx23I0p54KNWu6f1q0h0fn5bleC6HDuDduHBwON1RkcIkV0GfPLjneSlGR+0L4zW9gzRpYsCB2nr/4wjXxvPwyLFkSvNmqvB/m/fvdbfrJMnMmDBsWrLlVZUAfORKGD6+88yWqKsoc+isgU29YCwjcl/h9yKN7qrLMr71WOcf1fz/0jh3h/PPhww8TSl6DAt7kF+RTk/FcWmJ7DgWcwDIWERyi99e/hs0LXbW7AyviBvRQ0X46btjgXocODa7bti36/tECerQ3Xe/e7qfinDnBddnZwfkf/Si4P7heE3PnwqBBrjnpjDNKHjPWuRLxi1+4rn979iRn9MrInkNVGdBvvrnyzhXPsmXhNdfKbHKJHLlz/frgBfF06b6ZbIEyx7pO5Vf+r6ED/OEPZd7lHS7jBJbSmC3cwZMEujk+zjAWchKKcDwruRHXZjEhZAT4jxMcLiZe2/tHH5W+f7Sf/8tjjGMZuJiXiNNPD44LXxlv3vfec6/5CYyV9s037otj8mR4663E9omVZ1VX86rI6JDl/RKbMcP9kkuWd98NX45V5qIiN5xxRQJ+oMyBY7Ro4YaTLs2OHcnt0rplC/z+98k7XjwZ+6tEVVMy9ejRQ5MmPz/QGlHmKY9WxfN12R01TTYF2oRNYcugehHjdS4nq1CoUKRZHC5vNqJO7dq54gWWx42Ln379etVXX1VdvDj69hNOCF8uKor9p1NVffLJ8OXINPv2uXOGWrIkuH36dPc6Z47bVlSk+uCDqmvWuOUFC0oe8667ws+lqjp1ania//0v+ttg5ky3/ZJLSn/LbNyo2rCh6plnhq8/7rj4ZY6ltO2hiopUv/8+/jH+/Ofw8+7fH/1YL7/stj/7bGLnjub2290xnnwyPB+g2rJl7P1ANTe3/OcNPZeq6qWXJva3nj5dtUED1R07yn/eHj3cOWbPdsvnnKPas6db96tfxd7vnXfce7y8iorK9l6JBvcciqhxNTNq6BW4knVcyIXS05gRNU02hUjIjUrX4p4H9wa/ojtfcwEf8BLXU5ikFqyaHEQRzl35HKpwCl/xa0YX94iJ5dhj3cWWWI/Ei7zcsHJlcPjg8jjvPHdOcL8cVF3tPyBQY33jDTcuzpQpbjjeQG/Tk04qecxAk1SoyJqTxqitBnoSbd3qeg2Fjt9WUBA+amTr1rB9u/tlcOGF7uIcJOeiaLyxf/bscedo2RJmzUr8mLFq4IH+4oncfLRpk7vgF3lTXKB5Lto5Yv2tA5J5I10iv87AlWHnzooNlhZZQ//002CTZbwyX3ppxXp6laXHXHlkRkDPznadvit4B0yskRr7Mi1suTezaMJm6np93SfyU65jdLRduYZXUISj2U5NDtKObziC/WQT+1N/tNeTZjiPcvgwfEUfRhP+DLrGbIl7jER06OAGIovm5ZfDl6P9HJ0+3b0ecwyccILrVbMvSvf/v/3NDXv7k5+45XjD8Ua77hB6IxW4D9zo0e5Dfe21brz2PXuC1yVU3WNnhw51ff8vucRdJ6hTJ3iM0ODx/vvBIBdazngP//7uO/covgMHwq+BvP465Oa6C9ErV5YMDqFNd6ec4s4XraviQw+VLHNofpcsCc+vqnu87tdfx85z8+bu0YH9+4f3fgptcom8nhN63kOHol+8P3Qo2MsrUlGR+7IXcdOLL8b+ckq02bC0LxlwgXPsWFdBWLHCvS9Dv2hDA3qyr9FMnerGg4qmMq+FABnS5BJqxIjE2jPKOO2ldkLpIlfNxv22+4I++h8uL97wXwZqXz7X5qwPS9+RpdqStaqgG2nqfppFHPsI9qmCvsgNmsVhHcmN2p7lSSlqe5br2Xyixx4b3uSS6CQSnP/976OnqVEjdrNQ4KdovHN06lRy3dq1wfnTTot/7GhNPaB68KDq8ccHl195pWSa7dtV8/JUTznFLf/nP6pNmgS3//Sn7vXqqyyS0/wAABLLSURBVN3rU0+pvveeakGB6ooVqlddVfKYgeaSeOXfuTP4Fg+sKypS7dDBzd9zT3D9k0+qPvxwMP3Onapdu5Y85qOPuu2BfR9/XPXGG8PTNGwYPM6117p1u3aF5yNQpr17VT//PPzj+KMflTxvaJNGvDIHFBWpDhig+v77bvnss932iRNVH3jAvZciRXvvnnhicHufPm7dF1+o3npreLorrwym27RJddaskvndu1d1w4aS540sx/Dhqv36qR444Lbt31+yfGVFnCaXqCurYqq0gH7oUPxoUMnTVbwatmoWPUvdpxb79ae8V7y8DPcp3UQT7z/k1j/McP2SU/R6RhWv+wmTVEG/5qSSHwjQx7mnTEUIzAQCQ+T2ZmzQbsyLe4wj2KdtWF3uP6Nq2fc555zg/Kmnxk7Xv3/sbXfeqdq+fXC5UaPSzzt0aPjyoEHR04UG3NLKvGJFye0jRqiuXKnatGlw3WefBefvuqvkPrt2uSnetRdV1WHD3Pyf/6x6ww3R03z3XXB5w4ZgUAXVo492rw895F7ffTf4cYx13tWrw7dHS/vtt279li1uOScn/ON9wQXutUED1ZNPdnkP+L//i37ehx5y208/3S1Pm1by2tIJJwSPE/iyjizPySe71zVr3LkKC+OXedAgt23v3vAyl0f1CuiqqsuXl/5JrIIpl/yEAnqsaQuNtCNL46Z5jV+pEjugKy44n840HcEdqqBCodZnpx7NtpD0RVqHPWH71GK/jucibc0aPYtPNZsC3Un94u2xpk84u9Q0kZNQqLfzlLswrYnvF22KF9DjTT//ebDGW97p4ovLt195yvzYY8H5aAE9kWnuXNX77gv5P0jJNJG/aFavDl+O/OJ78knVf/7TBbtY5124MLzMzz5bMs2IES6oB5ZzcsLLfPfd7vWII8L/jqrBC72x/tZnnOHmH31UtUWL6GkGDw5fzsoqma5XL/c6b17w3PHOu3t3ybyWVfUL6Kqu+0JFPpk+mb6gT/F8iTeQN9OdOWEbXuHq4vkz+Z8OZoyO45KwNKD6C94IW/dX7op5rmjnhaLidX34QuuyW0E1l/yIfYqKv/he5AYdMEC1CZu0I0v1FL7UPnyhoNqStWHHjJxqckC/o4VewMRy/TlF3M/yI/mhzPs2Y4O2YbVecknZzwvuLVvWfY48Mjh/553lO++zz6r+7ncVexuGNjklOnXqVHqZb7pJtXPn8HWhvyACPXxCp/POS+xvfcwxZft/HDwYP/2sWapvveW+dOId84cfwpfLo3oG9L17XeNlRd6pPpsWEWxcvo5/VOhYw3lY19A6bpp/co2uoF3JN643k8VhPZeP9E/8QRV0Ihfoh7hP3GlMVyjS2uwN23kslypozGsWdzAiZpba436ZBfLUmjVakwPagB0x93kc1xbyFLcrqHbBVUeH8ow+wZ3agB3aijytyYG4f7IdNFClbL9KejGzuAvs0qWqA/hAf8p7ZTpGTQ7oI9ynR7CvXP/qW29Vvf13RcVfmmWZcjikORwq99sstAtfolP9+hV6WyuoTplS9n0iv1jKM91/v+rWrcHl8qqeAT1g2TLVLl10wWPv60ecqzd3m1Hx/4xNYdPbXKYPM7zE+lzKd3/AfxkYd3tfPtcOLNPI2vog3lUFXcIJ2pNZYfv8mMmqoB8wQP/EH3QJJ+hELghLA0V6Jf+Ked6LGB8zW8FjBNedxNfePQpuOZd8Hc0QbUWe9mB2cX4ij3EpY/UixquCXsbbcf9ct/CcKu76SqJ/4lrs19rsLV4eyjOqoOfzvoLqHYzQJ7iz1OPkk6uraVOht89VvKo/ZrKCahaH9Ze8nvT7ORKZTuFL7cSiSj/PUUeFvGfKqXoHdM+ePa4Hw4wZ6q4ugb4q15T4i3962gO6lpZV+26yqVzTJ5ytX3NS1G2DGVOuYy6gS9zt2RQU16oD0+lMK14A1b9zU/GF7WdxXSg+4lydTvTuN8N5WEFjnnMvtSOudwSn23lKFfRpfqc1OKjXM0qbsUGbs16FwuKb3r7klOKL7FtpWJxXUH2em1VBR3CHTuInxQf/BW/E/XOFljnW1JK1Op+u2owNpR7jZp5XBb2Z5+Me81LGqoLXBBc73Yks1uP4Nuq2K3hLFYqbARMpS7KmwYzRbsyLF67isoAew8GDqvmfTVd98023oqhIVd0dh8sHxrhMbpNNoKtpE3V9oMkm2dM4LtFf8roGfpXUZq8LCt72fRyhD3F/qccJfMEEphsZqX/nppjpN9M46qZsCooXQPX3PKaDGaPX8pL+iJX6LoP0HD7Wqbirj/fyqHZikSrui6IJmxQ05nkf5IGYzTnjuUgV9BLGqVAY9mV3DOuKr4GE5i9yWkjn4v9XaNozmKptWRXzT7iQzvoAD8b9M1/NK6qgTdkYdftBauifGVbuuFXhgA4MAFYAq4BhUbbXBN7yts8EWpd2zHQI6HEVFuo3c3fr4o++d51gS/mg3M1fKuWDbJNN6TJtoonOpFeJ9aU1kcWa/sWVcbcfIkeH87DWYr+2Y4X+m5/pXE4u3n4rz+o/uK54eQPNoh7nW44LW/4Z/475hRyYWvCd1uCggmoHlmkr8rQhwQZwUL2EcXoHI/RCJhSvz6OVrqKtKugU+ukpfKkK+hLX6hlMVVA9TJZroiynCgV0IBtYDbQFagALgBMj0twCjPTmBwNvlXbctA/oEQIXCJfTXvW114pr8/r116ovv6xz56oWbd6i42+forupq991v6j4n3z33aqPXDJH9d57i9f9ZUBIJ+KJE7Xwqb9p0foNuvAq+2KwyaZMnxbRqdyxqKIBvQ/wccjyvcC9EWk+Bvp48znANkDiHddvAX3ca3v1Ie7XWTMKEt8pP798JysqctOKFbpr+sLiN8HeiZP14Kef6+HDLtmMof/SPRv36K5th3TCyPX6xIVT9H8XBO+UXfreN7r220LXL+2FF7Sodm3dPWWuPnj6Jzq7xUWug/G2baqbN+v+tVt0w60Pa9Exx2jhwJ/GfCMe6tNXNzU8Qae1+Ll+9sZ63XdEwxJdHhX06QEflVj3YVbwIuS+lt4dPHfdpctbn5vyD5hNNlX5VE7xArq47bGJyOXAAFW93lu+CjhFVYeGpFnspVnnLa/20myLONaNwI0ArVq16rE22sAQpoQ5H2yh3jH16NDtiCo97/IPv2Xpzc9yyZonkayKjy2q6h4o0KpVKemKlPXZLdl895P0+OvP3Y7egCP5+e4BIhAc6GrbpsMcd/QeN4BHo0Zu44YN7skhIpCTw9Kl7rF+TZtGOWFRkTtQfj5vn/QwV6x4lMI9+9m3aQ/1j28CwNb1h8jJgaOa1qDoQD5Llwm1aip1F3xBk4aFZHVo50b8CowudvAgNG/OrvxaTP34AH3Oqk2jhuoGDqlRw6XZvNnled8+Ns76nua/Oov51z9HozM6cswVp5OVf4BDRTnMf2cNvW84CQoLWf7JWgqPasyWj7+mS+4yjmqcy+FfDSFLC8n9Yoob9WvzZor6/pipr6/lhyOOYdDFWWTv3QX161OUlYNu20722jWwZg0Fx7Ymt++p7u85ZS5bvs+nWeNC6pzQikVjFnHUT06mRfNC9izK46stbZFv19Bs3ge0KMzjyOG3ceBwLrW2rYPjj0ff/Dd7fnYd67fkMmvSbk4bUJ9jWmRR+4f1ZLU4hl1ZR3Hk11OQzW5EsRVPTKDDigmsvP15auzYSH67LrTfP5+NLXuzcNZBzrqlAzkrlzF3eR1yVq/gh0XrOGXl69S8ZCB77/kTua/+g+z2x7Nv+fdk7d9L3tnX89242ezr0J3zO35L4bwFZJ93NhQVUWv7Ovh8KjUO7eHA7gJqjx8DwNqLf8fhvHW0qL+bnB4nMeV/hTQafDZda69i18df8a20peby+Zy45n33b73rD2zffJgjt6ykoH0njnr2T3x73k1sphn7DmTxo94NqbVsHtkd2pF1dAOyD+dTe+40cqZ9RmH3XuR+5h4LtfyfX9JxyKll+wB5RGSuqkYdhalKA3qonj176pzQJzIYY4wpVbyAnshoi+uB0GeRt/DWRU0jIjnAkcD2smfVGGNMeSUS0GcD7USkjYjUwF30nBCRZgJwjTd/OfCZllb1N8YYk1SlPpFBVQ+LyFDchc9sYLSqLhGRP+Ea5ycALwOvi8gqYAcu6BtjjKlCCT1iR1U/AD6IWPdAyPxB4IrkZs0YY0xZZMYTi4wxxlhAN8aYTGEB3RhjMoQFdGOMyRCl3lhUaScW2QqU91bRRrjhBTKBlSU9ZUpZMqUcYGUJOE5VG0fbkLKAXhEiMifWnVJ+Y2VJT5lSlkwpB1hZEmFNLsYYkyEsoBtjTIbwa0AfleoMJJGVJT1lSlkypRxgZSmVL9vQjTHGlOTXGroxxpgIFtCNMSZD+C6gi8gAEVkhIqtEZFiq8xONiIwWkS3egz8C644WkU9FZKX3epS3XkTkGa88C0Wke8g+13jpV4rINdHOVcnlaCkik0VkqYgsEZHf+bgstURklogs8MrykLe+jYjM9PL8ljdENCJS01te5W1vHXKse731K0TkvKoui5eHbBH5WkQm+rwceSKySETmi8gcb53v3l9eHhqIyFgRWS4iy0SkT5WXJdaz6dJxIoEHVqfDBPQDugOLQ9b9FRjmzQ8D/uLNXwB8CAhwKjDTW380sMZ7PcqbP6qKy9Ec6O7N1wO+AU70aVkEqOvN5wIzvTz+BxjsrR8J3OzNR33wuVf+BUBNoI33fsxOwXvsDmAMMNFb9ms58oBGEet89/7y8vEqcL03XwNoUNVlqdICJ+EPVuoDq9NlAloTHtBXAM29+ebACm/+ReDKyHTAlcCLIevD0qWoTO8B5/i9LEBtYB5wCu5uvZzI9xcxHnwe+Z4LTVeF+W8B/A/4CTDRy5fvyuGdN4+SAd137y/cU9q+xetokqqy+K3J5Vjg+5Dldd46P2iqqhu9+U1A4HHFscqUVmX1fqqfjKvZ+rIsXjPFfGAL8CmuVrpTVQ9HyVdxnr3tu4CGpEdZ/gbcAxR5yw3xZzkAFPhEROaKe4g8+PP91QbYCvzTawp7SUTqUMVl8VtAzwjqvnp9019UROoC44DbVXV36DY/lUVVC1W1G66G2xvomOIslZmIXAhsUdW5qc5Lkpyhqt2B84FbRaRf6EYfvb9ycM2sf1fVk4F9uCaWYlVRFr8F9EQeWJ2uNotIcwDvdYu3PlaZ0qKsIpKLC+b/UtV3vNW+LEuAqu4EJuOaJhqIe7B5ZL5iPfg81WU5HRgkInnAv3HNLv8P/5UDAFVd771uAcbjvmj9+P5aB6xT1Zne8lhcgK/SsvgtoCfywOp0Ffog7Wtw7dGB9Vd7V71PBXZ5P9E+Bs4VkaO8K+PneuuqjIgI7nmxy1T1qZBNfixLYxFp4M0fgbsWsAwX2C/3kkWWJdqDzycAg73eI22AdsCsqikFqOq9qtpCVVvj3v+fqeov8Vk5AESkjojUC8zj3heL8eH7S1U3Ad+LSAdv1VnAUqq6LFV9ESQJFx8uwPW2WA0MT3V+YuTxTWAjUID75r4O1275P2AlMAk42ksrwPNeeRYBPUOOcy2wypt+nYJynIH7ibgQmO9NF/i0LF2Br72yLAYe8Na3xQWyVcDbQE1vfS1veZW3vW3IsYZ7ZVwBnJ/C91l/gr1cfFcOL88LvGlJ4PPsx/eXl4duwBzvPfYurpdKlZbFbv03xpgM4bcmF2OMMTFYQDfGmAxhAd0YYzKEBXRjjMkQFtCNMSZDWEA3viciM7zX1iLyiyQf+75o5zImHVm3RZMxRKQ/cJeqXliGfXI0OAZKtO17VbVuMvJnTGWzGrrxPRHZ680+DvT1xtb+P28wridEZLY35vRNXvr+IjJNRCbg7uZDRN71BohaEhgkSkQeB47wjvev0HN5d/g9ISKLxY3n/fOQY08JGRf7X94dt8ZUupzSkxjjG8MIqaF7gXmXqvYSkZrAFyLyiZe2O9BZVb/1lq9V1R3esACzRWScqg4TkaHqBvSKdCnuzsCTgEbePlO9bScDnYANwBe48VemJ7+4xoSzGrrJZOfixsuYjxv2tyFuzBKAWSHBHOC3IrIA+Ao3OFI74jsDeFPdCI6bgc+BXiHHXqeqRbjhElonpTTGlMJq6CaTCXCbqoYNbuS1te+LWD4b94CH/SIyBTcGSnnlh8wXYp8zU0Wshm4yyR7co/ICPgZu9oYARkTae6P6RToS+MEL5h1xjwQLKAjsH2Ea8HOvnb4x7rGDVTpaoTGRrOZgMslCoNBrOnkFN054a2Ced2FyK3BxlP0+An4jIstwIw9+FbJtFLBQROapG6Y2YDxuPPUFuBEp71HVTd4XgjEpYd0WjTEmQ1iTizHGZAgL6MYYkyEsoBtjTIawgG6MMRnCAroxxmQIC+jGGJMhLKAbY0yG+P/vxw0xvTvH4gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the loss and accuracy\n",
    "\n",
    "plt.plot(loss_list_normal, 'b',label='loss normal cnn ')\n",
    "plt.plot(loss_list_batch,'r',label='loss batch cnn')\n",
    "plt.xlabel('iteration')\n",
    "plt.title(\"loss\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-07-16T08:57:27.309261Z",
     "start_time": "2021-07-16T08:57:27.184302Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEWCAYAAABv+EDhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXwUVbbA8d8hhM2AKEFEwhIliqxBAioYNjdwQUFUEBeeT51xcBT3YXRm/KC4oqij88YNRhMWFZHBBRUFBHeC7CAKGCWIGpDIviSc98etTjpNQpqkQ3XS5/v59Kera+tTHbin6tate0VVMcYYE3tq+B2AMcYYf1gCMMaYGGUJwBhjYpQlAGOMiVGWAIwxJkZZAjDGmBhlCcAYY2KUJQBTrYnIXBHZIiK1/Y7FmGhjCcBUWyLSCkgHFBhwmL+75uH8PmPKwxKAqc6uBr4A/gNcE7xAROqKyOMi8oOI/C4in4hIXW/ZGSLymYjkich6ERnuzZ8rItcF7WO4iHwS9FlFZISIfAd85817ytvHVhFZKCLpQevHichfRWStiGzzljcXkWdF5PGQeGeIyK2R/oFMbLMEYKqzq4GJ3utcEWkStGws0AXoDhwN3AXsF5GWwEzgn0BjIBVYfAjfeTFwKtDW+7zA28fRwCTgdRGp4y27DRgKnAc0AK4FdgIvA0NFpAaAiCQCZ3nbGxMxlgBMtSQiZwAtgddUdSGwFrjCW1YDV9jeoqobVLVAVT9T1T3eOh+q6mRV3aeqm1X1UBLAQ6r6m6ruAlDVTG8f+ar6OFAbOMlb9zrgXlVdrc4Sb92vgN+BM731hgBzVfWXCv0oxoSwBGCqq2uAD1R1k/d5EkXVQIlAHVxSCNW8lPnhWh/8QUTuEJFVXjVTHnCk9/1lfdfLwJXe9JVARgViMqZEdqPKVDteXf5lQJyI/OzNrg00FJFOwDJgN3ACsCRk8/VAt1J2vQOoF/T52BLWKexe16vvvwt3Jr9CVfeLyBZAgr7rBGB5CfvJBJZ78Z4MTC8lJmPKza4ATHV0MVCAq4dP9V4nA/OBq1V1PzAeeEJEjvNuxp7uNRWdCJwlIpeJSE0RaSQiqd5+FwODRKSeiLQG/reMOOoD+UAuUFNE/o6r6w94EbhfRFLE6SgijQBUNQd3/yADeCNQpWRMJFkCMNXRNcAEVf1RVX8OvIBngGFeE807cFcCC4DfgEeAGqr6I+6m7O3e/MVAJ2+/44C9wC+4KpqJZcTxPvAe8C3wA+6qI7iK6AngNeADYCvwElA3aPnLQAes+sdUErEBYYyJTiLSE1cV1FLtP6qpBHYFYEwUEpF44BbgRSv8TWWxBGBMlBGRk4E8oCnwpM/hmGrMqoCMMSZG2RWAMcbEqLCeAxCRfsBTQByuTvLhkOUtcc3qGuNaTlzpNWNDRB4BzvdWvV9VX/XmJwNTgEbAQuAqVd17sDgSExO1VatW4R2ZMcYYABYuXLhJVRuHzi+zCkhE4nDN2M4GAm2Th6rqyqB1XgfeVtWXRaQv8D+qepWInA+MBPrjHsSZC5ypqltF5DVgmqpOEZF/A0tU9f8OFktaWppmZWWFf9TGGGMQkYWqmhY6P5wqoG7AGlVd552hTwEuClmnLTDbm54TtLwtMM/rB2UHsBToJyIC9AWmeuu9jHt4xxhjzGESTgJoRvGHV3K8ecGWAIO86YFAfe+JxiW4Ar+e16NhH1z/J42APFXNP8g+ARCRG0QkS0SycnNzwzkmY4wxYYjUTeA7gF4isgjoBWwAClT1A+Bd4DNgMvA57hH9sKnq86qapqppjRsfUIVljDGmnMJJABtwZ+0BSd68Qqr6k6oOUtXOwD3evDzvfYyqpqrq2bhOsL4FNuM65qpZ2j6NMcZUrnASwAIgRUSSRaQWrm/yGcEriEhiYPAKYBSuRVBgxKNG3nRHoCOui17F3SsY7G1zDfDfih6MMcaY8JWZALx6+ptwHVutwg2wsUJERotIYJzV3sBqEfkWaAKM8ebHA/NFZCXwPK55aKDe/27gNhFZg7sn8FKEjskYY0wYqtSTwNYM1BhjDl1pzUBtQBhzeO3YAd98A6tWwbp1EBcHdetCnTrFX+HMq10bRMr+TmNMiSwBmMqRm+sK+eDXN9/Ajz9G9ntq1y5/AgnMS0iAli0hOdm9161b9vcaUw1YAjDlt3+/K9CDC/jA9ObNRevVqwdt2kB6Opx8sps++WRo3dot37ULdu8u/ippXmnzDzZv27aS5+/a5eIvybHHumTQqpV7D7xatYIWLSA+vrJ/WXM4FRS4K9Nt22D79qLXtm3u30ng30PTpu6KtRqxBGDKtncvfPfdgYX86tWwc2fReomJrmAfNMi9B17Nm0ONg7Q3qFULjjyy8o8jVH6+SwZ5efDDD/D990Wv7Gz4/HN47TVXQATUqAFJSSUnh+RkOO646CwkVGHrVti0yV2dbdpUfDp4Xn5+eFdPFZkfH1++6rv9+11hHSigQwvs8nzeFeZom/HxRVeKwX//wPQxx1S5Kkm7CWyKbNtWvIAPvNauLV4ItmhRvIAPvBIT/Yu9suTnQ05O8cQQPP3TT65wDYiPd79PSckhkoXE3r3uKiu0AD9Y4b5vX8n7qlULGjd2f7/ERPc5nKuxihAp+/7O3r0HFtjBJxxlqVUL6td3VXyB16F+rlPH/Y1D/+7ff+9+12D16rm/dUnJITkZGjb0LUGUdhPYEkCsUYVffim5oN8Q9CxezZqQknJgIX/iie4/hnH27HFXDyUVECUVEnXrll5A1K9/8EI9uHD//ffSYzrqqKICPbhgD50XeE9IOPSCSdUde7jVdOWZX6vWgYVyuAX4EUe47SvT9u3ub13a3z70b9SgQcl/98DnSvx/ZQkglqi6wuK779xrzZri09u2Fa2bkFBUJx9cP3/CCVbXHQk7dhQVCsGFQ+BzXt7Bt69d2xXU4RTkiYnQqJFL3sZ/eXkH/r2Dp0OvZhITS69eatnSXY2UkyWA6kbVnS0GCvbQwn7r1qJ14+LcP6SUFPdq3bqooE9KqnL1ltVKoJDIznaJObRQP+II+/tUR4GTtNJODH74wVWBBVuyBDp2LNfX2XMAVVWgkA8+iw+8gi8xa9RwZwkpKXDaacUL++RkO5uPVg0bQufO7mVih4i7H3TMMXDqqQcu37//wHsPyckRD8MSQDTYsqX0M/ktW4rWEykq5K+4oqiQT0lx/zgqu87TGHN4BFqbJSXBGWdU2tdYAjicfvgBPvvswML+t9+K1hFxzSZTUuDyy4vO4lNS4PjjXZ2wMcZEgCWAw+XDD+Gii4pu/DRv7gr2wYOLn8kff3yFbvYYY0y4LAEcDu+8A5dc4ppQvvyyuwFr3Q0YY3xmCaCyTZ0KQ4dCaiq8955rpmeMMVEgUkNCmpK88oqrxz/1VFcFZIW/MSaKWAKoLM89B9dcA336wPvv+9PXjTHGHIQlgMowbhz88Y9w/vnw9tvuYR5jjIkylgAibcwYuO0217pn2jRr0WOMiVphJQAR6Sciq0VkjYj8pYTlLUXkIxFZKiJzRSQpaNmjIrJCRFaJyNMi7rl2b73VIrLYex0TucPygSr89a9w771w1VUwebI9mGWMiWplJgARiQOeBfoDbYGhItI2ZLWxwCuq2hEYDTzkbdsd6AF0BNoDXYFeQdsNU9VU7/VrRQ/GN6owciQ89BDccAP85z/WIZcxJuqFcwXQDVijqutUdS8wBbgoZJ22wGxvek7QcgXqALWA2kA88EtFg44qBQWu0H/6aZcE/v3vgw9+YowxUSKckqoZsD7oc443L9gSYJA3PRCoLyKNVPVzXELY6L3eV9VVQdtN8Kp//haoGgolIjeISJaIZOWG9q3ut/x8uPpqePFFV/XzxBPWc6MxpsqI1KnqHUAvEVmEq+LZABSISGvgZCAJlzT6iki6t80wVe0ApHuvq0rasao+r6ppqprWuHHjCIUbAXv3ujb+kybBgw/C/fdb4W+MqVLCqajeADQP+pzkzSukqj/hXQGISAJwiarmicj1wBequt1bNhM4HZivqhu8bbeJyCRcVdMrFTyew2PXLte1w8yZ8OSTcMstfkdkjKki9u93Pbnn5bnXli3hTc+c6UYbjaRwEsACIEVEknEF/xDgiuAVRCQR+E1V9wOjgPHeoh+B60XkIUBwVwdPikhNoKGqbhKReOAC4MNIHFCl274dBgyAuXPh+efh+uv9jsgYcxipuj4dgwvpcAvxvDw3VtPBxuGqUcMNExF4HXWUG7upMpSZAFQ1X0RuAt4H4oDxqrpCREYDWao6A+gNPCQiCswDRnibTwX6AstwN4TfU9W3ROQI4H2v8I/DFf4vRPbQKkFennu468svXTcPV17pd0TGmAjbtg0WLoQFC2DxYjcEc3BBnpcH+/YdfB8JCUWFd8OGrvPfjh2LF+qlTSckHL52JDYkZLg2bYJzz4Vly2DKFBg0qOxtjDFRbc8eN9LiggXw1Vfu/Ztvis7QW7SA444Lr+AOPmuPtlbgNiRkRfz8M5x1FqxdC9Onw3nn+R2RMeYQFRTAypWukA+8li4tOptv0gS6doUhQ9x7Wpobmrk6swRQlvXr4cwz3fic77wDffv6HZExpgyqsG5d8cL+669hxw63vEEDV8Dfdpsr7Lt2ddU0sdaQzxLAwaxb5wr8LVvggw+ge3e/IzLGlGDjxuLVOFlZRSOt1q4NnTvDtde6gr5bNzf4nj2vaQmgdN984878d++G2bOhSxe/IzLG4M7HsrKKn91v8Bqmx8VBu3buFl3gzL59e4iP9zfmaGUJoCRLl7o6/xo1XHPPDh38jsiYmLRzJyxaVLyw/+67ouUpKdCrV1Fh37kz1KvnX7xVjSWAUF99Bf36uT78P/rIjeNrjKl0qvDDDzB/PnzyiWttvXy5u3kL0KyZK+SHDy+6SXvUUb6GXOVZAgg2f75r55+Y6Kp9WrXyOyJjqi1VWLUK5s1z//Xmz3dtLsANoHfqqXDBBa7OvmtXaNrU33irI0sAAR9+6J7wbdHCnfk3C+3vzhhTEfn5rjpn/nxX6H/yCWze7JYdeyz07Al33QXp6a7ePi7O33hjgSUAcMM2Dh7sqntmzXINgo0xFbJrl6tRDZzhf/6560kF4IQT4MILXaGfnu4+x1oTzGhgCeD11+GKKyA11Q3efvTRfkdkTJX0++/w6adFZ/gLFriHrETcGf0117jCPj3dPV1r/BfbCeDll13j4O7d3VXAkUf6HZExVcbPPxfV3c+f77pUUHXdIKSlufGRevaEHj3sZm20it0E8O9/w403uuae06e7Vj/GmBKpwvffF53dz59f1ByzXj04/XT4xz/c2f2pp9p/p6oiNhPAE0/A7be7Jgavvw516vgdkTFRZf9+129OoLCfN8/1hgLubP6MM9xIqOnpcMop9qBVVRVbCUAVxoyBv/0NLr0UMjOhVi2/ozLmsNuzB3JzXSe3gffA9OLFroXOli1u3eOOK7pZ27MntG1r3ShUF7GTAFThr3+Fhx+Gq66C8eOjr89WY8ph/37XR31wYV7ae2A60BonlAi0bg0DBxYV+snJ1kKnuoqNEnD/frj1Vnj6afjDH+Bf/7JTGBO1du8uuwAPft+8uehp2VD16rkujRMT3ftJJxX/nJhYfPqoo6z9fSyp/glAFf74R3jhBZcEHn/cTmeM77ZvhxUr3PhCgdf337tCPdBlcagaNaBRo6ICu02bAwvz4EI9MdH6xTEHV/0TgIjrHvDee2H0aCv8zWGVnw/fflu8oF++3PU0HnDEEe6f6BlnuAK8tEL9qKPswtVEVlgJQET6AU/hxu99UVUfDlneEjcQfGPgN+BKVc3xlj0KnA/UAGYBt6iqikgX4D9AXeDdwPxIHNQBbrmlUnZrTICq65I4uKBftsz1dbN3r1snLs71Xtmli+vQrEMH90pOtoLd+KPMBCAiccCzwNlADrBARGao6sqg1cYCr6jqyyLSF3gIuEpEugM9gI7eep8AvYC5wP8B1wNf4hJAP2BmJA7KmMr0++/uLD70rD7QagZcV1IdOsA55xQV9G3aWItjE13CuQLoBqxR1XUAIjIFuAgITgBtgdu86TnAdG9agTpALUCAeOAXEWkKNFDVL7x9vgJcjCUAE0X27oXVqw88q//xx6J1GjRw3RxcdllRQd++vfUoYqqGcBJAM2B90Occ4NSQdZYAg3DVRAOB+iLSSFU/F5E5wEZcAnhGVVeJSJq3n+B9ltj9pojcANwA0KJFizDCNebQqLpCPbSgX726aMDwmjXdGXyPHq5NQaCwb9HCbiuZqitSN4HvAJ4RkeHAPGADUCAirYGTgSRvvVkikg7sCnfHqvo88DxAWlpa5dwjMDFnzx547z2YNMm9b91atKxlS1e4X3ihO5vv0ME1n7RnBk11E04C2AA0D/qc5M0rpKo/4a4AEJEE4BJVzROR64EvVHW7t2wmcDqQQVFSKHGfxkRaQYHr0mDSJJg61T08lZgIl1/ubswGqm8aNPA7UmMOj3ASwAIgRUSScYX0EOCK4BVEJBH4TVX3A6NwLYIAfgSuF5GHcFVAvYAnVXWjiGwVkdNwN4GvBv4ZiQMyJpgqfP21K/SnTHH92SQkuCddr7gCzjzT+rExsavMBKCq+SJyE/A+rhnoeFVdISKjgSxVnQH0Bh4SEcVVAY3wNp8K9AWW4W4Iv6eqb3nL/kRRM9CZ2A1gE0HffguTJ7uC/9tvXSF/3nmu0L/gAntAyhgAqaym95UhLS1Ns7Ky/A7DRKmffoJXX3WFflaWuznbu7cr9C+5xPqkN7FLRBaqalro/Or/JLCp1rZsgTfecGf7c+a4Kp8uXVyPH5dfbkM7G3MwlgBMlbNzpxvAbdIkePdd11QzJQX+/nd3tn/iiX5HaEzVYAnAVAn5+fDhh67Qf/NN15la06bw5z/D0KHurN/a4xtzaCwBmKilCp9/7gr9115zPWU2bAhDhrgz/Z49retiYyrCEoCJOsuXu0J/8mTIznb95wwY4Ar9fv2gdm2/IzSmerAEYKJCdrZrpz9pkuuGIS4Ozj7b9eB98cVQv77fERpT/VgCML7JzYXXX3eF/qefunndu8Mzz7ghm485xt/4jKnuLAGYw2bHDjfY+Jw5MHs2LFzoRuts1w4efNDV7Scn+x2lMbHDEoCpNLt3u5u4s2e7Qv/LL11rnpo14bTT3CBtgwe7PniMMYefJQATMXv3wldfFZ3hf/6563WzRg1IS4M77oA+fVyXykcc4Xe0xhhLAKbc8vNdR2uBM/xPPnEPaYlAaiqMGOEK/PR0OPJIv6M1xoSyBGDCtn8/LFniCvs5c1zXyoF+9Nu1g2uvhb59oVcvGxHLmKrAEoAplSqsXFlUpfPxx/Dbb25ZSop7ArdPH9fhWpMmvoZqjCkHSwCmkCqsWVNUpTNnDvz6q1vWsiVcdJEr8Pv0gaSkg+/LGBP9LAHEuOzsojP8OXNggzcu23HHuQex+vZ1Bb41zzSm+rEEEIOmT4e33nIF/vffu3mNGxed3ffp43rUtM7VjKneLAHEmAcegL/9zXWq1rs3jBzpzvLbtbMC35hYYwkghjz+uCv8r74axo+3njSNiXU1wllJRPqJyGoRWSMifylheUsR+UhElorIXBFJ8ub3EZHFQa/dInKxt+w/IvJ90LLUyB6aCfavf7kHsS67DF56yQp/Y0wYVwAiEgc8C5wN5AALRGSGqq4MWm0s8IqqviwifYGHgKtUdQ6Q6u3naGAN8EHQdneq6tTIHIopzfjx7qGsAQMgM9N1xWCMMeFcAXQD1qjqOlXdC0wBLgpZpy0w25ueU8JygMHATFXdWd5gzaGbPBmuuw7OOccNqhIf73dExphoEU4CaAasD/qc480LtgQY5E0PBOqLSKOQdYYAk0PmjfGqjcaJSInDfIjIDSKSJSJZubm5YYRrAqZNg6uuck/mvvmmDaRijCkurHsAYbgD6CUii4BewAagILBQRJoCHYD3g7YZBbQBugJHA3eXtGNVfV5V01Q1rXHjxhEKt/p7913XvXK3bq7JZ716fkdkjIk24dQGbwCaB31O8uYVUtWf8K4ARCQBuERV84JWuQx4U1X3BW2z0ZvcIyITcEnERMBHH8GgQdCxI8ycCQkJfkdkjIlG4VwBLABSRCRZRGrhqnJmBK8gIokiEtjXKGB8yD6GElL9410VICICXAwsP/TwTaj5893N3hNPhPfft144jTGlKzMBqGo+cBOu+mYV8JqqrhCR0SIywFutN7BaRL4FmgBjAtuLSCvcFcTHIbueKCLLgGVAIvBAhY7E8NVXcP750Lw5zJoFjULvwhhjTBBRVb9jCFtaWppmZWX5HUZUWrzYdeFw9NGum+ZmobfpjTExS0QWqmpa6PxI3QQ2PlqxwnXcVr++69TNCn9jTDgsAVRx330HZ53l2vfPnu26bTbGmHDYM6FVWHY2nHmmG5rx44+hdWu/IzLGVCWWAKqonBzXi+f27a5b57Zt/Y7IGFPVWAKogn75xZ35b9rk2vx36uR3RMaYqsgSQBWzaZOr88/Jce38u3b1OyJjTFVlCaAKycuDc891N37ffRfOOMPviIwxVZklgCpi2zbo3x+WLYP//tfV/xtjTEVYAqgCdu6ECy6ABQvg9dddIjDGmIqyBBDldu+Giy92ffxMmgQDB/odkTGmurAEEMX27nVDOM6aBRMmuO6djTEmUuxJ4CiVnw/Dhrm+/P/1Lxg+3O+IjDHVjSWAKLR/P/zP/8DUqfD443DjjX5HZIypjiwBRBlV+OMf3eDtDzwAt93md0TGmOrKEkAUUYWRI+GFF+Cee9zLGGMqiyWAKKEKo0bB00/DrbfC/ff7HZExprqzBBAlRo+GRx5x9f2PPw4ifkdkjKnuLAFEgUcfhfvucy19nnnGCn9jzOERVgIQkX4islpE1ojIX0pY3lJEPhKRpSIyV0SSvPl9RGRx0Gu3iFzsLUsWkS+9fb7qDTgfc/75T7j7btfG/8UXoYalZGPMYVJmcSMiccCzQH+gLTBUREJ7nx8LvKKqHYHRwEMAqjpHVVNVNRXoC+wEPvC2eQQYp6qtgS3A/0bgeKqUF16Am292T/q+8grExfkdkTEmloRzvtkNWKOq61R1LzAFuChknbbAbG96TgnLAQYDM1V1p4gILiFM9Za9DFx8qMFXZZmZ8Ic/QL9+MGWKG9LRGGMOp3ASQDNgfdDnHG9esCXAIG96IFBfRBqFrDMEmOxNNwLyVDX/IPsEQERuEJEsEcnKzc0NI9zoN3UqXHMN9OkD06ZB7dp+R2SMiUWRqnG+A+glIouAXsAGoCCwUESaAh2A9w91x6r6vKqmqWpa48aNIxSuf956C4YOhdNPd906163rd0TGmFgVTmdwG4DmQZ+TvHmFVPUnvCsAEUkALlHVvKBVLgPeVNV93ufNQEMRqeldBRywz+po1iwYPBhSU+GddyAhwe+IjDGxLJwrgAVAitdqpxauKmdG8AoikigigX2NAsaH7GMoRdU/qKri7hUM9mZdA/z30MOvOvbuhSuugJNOckM5Hnmk3xEZY2JdmQnAO0O/CVd9swp4TVVXiMhoERngrdYbWC0i3wJNgDGB7UWkFe4K4uOQXd8N3CYia3D3BF6q0JFEuZkz3Xi+Dz8MRx/tdzTGGAPiTsarhrS0NM3KyvI7jHK59FL4+GP46SeoaaMwGGMOIxFZqKppofPtsaPDIC+v6OavFf7GmGhhCeAweP112LMHrrrK70iMMaaIJYDDIDMT2rSBLl38jsQYY4pYAqhk2dkwbx5ceaV18maMiS6WACrZxInufdgwf+MwxphQlgAqkaqr/unZE1q18jsaY4wpzhJAJVq4EL75xlX/GGNMtLEEUIkyMlxHb5de6nckxhhzIEsAlWTfPpg8GS68EBo29DsaY4w5kCWASjJrFuTmWvWPMSZ6WQKoJBkZ0KgR9O/vdyTGGFMySwCVYOtWmD4dLr8casXkSMfGmKrAEkAlmDYNdu+26h9jTHSzBFAJMjKgdWs47TS/IzHGmNJZAoiwnByYM8e6fjDGRD9LABE2aZJ7Ati6fjDGRDtLABGk6qp/Tj/dVQEZY0w0swQQQUuXwvLl1u+/MaZqCCsBiEg/EVktImtE5C8lLG8pIh+JyFIRmSsiSUHLWojIByKySkRWemMEIyL/EZHvRWSx90qN1EH5JSMD4uPhssv8jsQYY8pWZgIQkTjgWaA/0BYYKiJtQ1YbC7yiqh2B0cBDQcteAR5T1ZOBbsCvQcvuVNVU77W4Asfhu4ICV/9/3nnuATBjjIl24VwBdAPWqOo6Vd0LTAEuClmnLTDbm54TWO4lipqqOgtAVber6s6IRB5lZs+GjRut+scYU3WEkwCaAeuDPud484ItAQZ50wOB+iLSCDgRyBORaSKySEQe864oAsZ41UbjRKR2SV8uIjeISJaIZOXm5oZ1UH7IyIAjj4Tzz/c7EmOMCU+kbgLfAfQSkUVAL2ADUADUBNK95V2B44Hh3jajgDbe/KOBu0vasao+r6ppqprWuHHjCIUbWTt2uKd/L7sM6tTxOxpjjAlPOAlgA9A86HOSN6+Qqv6kqoNUtTNwjzcvD3e1sNirPsoHpgOneMs3qrMHmICraqqSpk93ScCqf4wxVUk4CWABkCIiySJSCxgCzAheQUQSRSSwr1HA+KBtG4pI4NS9L7DS26ap9y7AxcDyihyInzIyoGVL6NHD70iMMSZ8ZSYA78z9JuB9YBXwmqquEJHRIjLAW603sFpEvgWaAGO8bQtw1T8ficgyQIAXvG0mevOWAYnAAxE7qsNo40bX9/+VV0INe6rCGFOF1AxnJVV9F3g3ZN7fg6anAlNL2XYW0LGE+X0PKdIoNWUK7N9vPX8aY6oeO2etoIwMSEuDNm38jsQYYw6NJYAKWLECFi2ym7/GmKrJEkAFZGZCXBwMGeJ3JMYYc+gsAZTT/v0wcSKcey4cc4zf0RhjzKGzBFBOH38M69db9Y8xpuqyBFBOmZlQvz4MGFD2usYYE40sAZTDrl0wdSpccgnUq+d3NMYYUz6WAMphxgzYutWqf4wxVZslgHLIzIRmzaBXL78jMcaY8rMEcBTV8NAAABV8SURBVIhyc+G999yg73FxZa9vjDHRyhLAIZoyBfLzrfrHGFP1WQI4RJmZ0KkTtG/vdyTGGFMxlgAOwerV8NVXdvZvjKkeLAEcgsxM1+Xz0KF+R2KMMRVnCSBMqi4BnHkmHHec39EYY0zFWQII06efQna2Vf8YY6oPSwBhysx0T/0OHOh3JMYYExmWAMKwZw+89por/BMS/I7GGGMiI6wEICL9RGS1iKwRkb+UsLyliHwkIktFZK6IJAUtayEiH4jIKhFZKSKtvPnJIvKlt89XvQHno9I778CWLVb9Y4ypXspMACISBzwL9AfaAkNFpG3IamOBV1S1IzAaeCho2SvAY6p6MtAN+NWb/wgwTlVbA1uA/63IgVSmzEw49lh3A9gYY6qLcK4AugFrVHWdqu4FpgAXhazTFpjtTc8JLPcSRU1vYHhUdbuq7hQRAfpSNJD8y8DFFTqSSvLbb/D2267pZ82afkdjjDGRE04CaAasD/qc480LtgQY5E0PBOqLSCPgRCBPRKaJyCIRecy7omgE5Klq/kH2CYCI3CAiWSKSlZubG95RRdBrr8G+fVb9Y4ypfiJ1E/gOoJeILAJ6ARuAAqAmkO4t7wocDww/lB2r6vOqmqaqaY0bN45QuOHLzIR27SA19bB/tTHGVKpwEsAGoHnQ5yRvXiFV/UlVB6lqZ+Aeb14e7sx+sVd9lA9MB04BNgMNRaRmafuMBuvWufb/V14JIn5HY4wxkRVOAlgApHitdmoBQ4AZwSuISKKIBPY1ChgftG1DEQmcuvcFVqqq4u4VDPbmXwP8t/yHUTkyM13BP2yY35EYY0zklZkAvDP3m4D3gVXAa6q6QkRGi0hgRNzewGoR+RZoAozxti3AVf98JCLLAAFe8La5G7hNRNbg7gm8FLGjioBA1w+9e0Pz5mWubowxVY64k/GqIS0tTbOysg7Ld335JZx2Grz0Elx77WH5SmOMqRQislBV00Ln25PApcjIgDp1YPDgstc1xpiqyBJACfbtcyN/XXQRNGjgdzTGGFM5LAGU4L33YPNm1/rHGGOqK0sAJcjIgMaN4dxz/Y7EGGMqjyWAEL//DjNmwJAhEB/vdzTGGFN5LAGEmDrVdf9s1T/GmOrOEkCIjAw48UTo2tXvSIwxpnJZAgjy44/w8ceu4zfr+sEYU91ZAggycaJ7t64fjDGxwHq496i66p8zzoDkZL+jMbFk37595OTksHv3br9DMVVcnTp1SEpKIj7MFiyWADyLFsGqVfDcc35HYmJNTk4O9evXp1WrVojVPZpyUlU2b95MTk4OyWGexVoVkCcjA2rVgksv9TsSE2t2795No0aNrPA3FSIiNGrU6JCuJC0BAPn5MHkyXHABHHWU39GYWGSFv4mEQ/13ZAkA+PBD+OUXG/bRGBNbLAHgqn+OOgr69/c7EmPM8OHDmTp1aonLxo4dS5s2bUhNTaVr16688sorAPTu3Zu0tKLejrOysujduzcAc+fORUR46623CpdfcMEFzJ07t9KOoaqI+QSwbRu8+SZcfjnUru13NMZUbfn5+ZW273//+9/MmjWLr776isWLF/PRRx8RPJ7Jr7/+ysyZM0vcNikpiTFjxlRabFVVzLcCevNN2LXLqn9MdBg5EhYvjuw+U1PhySfDXz8hIYFbbrmFt99+m7p16/Lf//6XJk2akJ2dzbXXXsumTZto3LgxEyZMoEWLFgwfPpw6deqwaNEievToQYMGDfj+++9Zt24dP/74I+PGjeOLL75g5syZNGvWjLfeeov4+HhGjx7NW2+9xa5du+jevTvPPffcQeuwH3zwQebOnUsDr4/2Bg0acM011xQuv/POOxkzZgz9S7iU79SpE/v27WPWrFmcffbZ4f8Y1VzMXwFkZMDxx8Ppp/sdiTHRYceOHZx22mksWbKEnj178sILbhTXP//5z1xzzTUsXbqUYcOGcfPNNxduk5OTw2effcYTTzwBwNq1a5k9ezYzZszgyiuvpE+fPixbtoy6devyzjvvAHDTTTexYMECli9fzq5du3j77bdLjWnr1q1s27aN448/vtR1Tj/9dGrVqsWcOXNKXH7PPffwwAMPHPLvUZ2FdQUgIv2Ap4A44EVVfThkeUvcQPCNgd+AK1U1x1tWACzzVv1RVQd48/8D9AJ+95YNV9UIn/sc3IYN8NFH8Le/WdcPJjocypl6ZalVqxYXXHABAF26dGHWrFkAfP7550ybNg2Aq666irvuuqtwm0svvZS4uLjCz/379yc+Pp4OHTpQUFBAv379AOjQoQPZ2dkAzJkzh0cffZSdO3fy22+/0a5dOy688MIKxX7vvffywAMP8MgjjxywrGfPngB88sknFfqO6qTMKwARiQOeBfoDbYGhItI2ZLWxwCuq2hEYDTwUtGyXqqZ6rwEh290ZtOywFv7gmn6qWs+fxgSLj48vrIqJi4sLq17/iCOOKPa5tndDrUaNGsX2V6NGDfLz89m9ezd/+tOfmDp1KsuWLeP6668/aPv1Bg0akJCQwLp16w4aR9++fdm1axdffPFFicvtKqC4cKqAugFrVHWdqu4FpgAXhazTFpjtTc8pYXlUysiAU0+FlBS/IzEm+nXv3p0pU6YAMHHiRNLT08u9r0Bhn5iYyPbt20tt9RNs1KhRjBgxgq1btwKwffv2wlZAwe69914effTREvdxzjnnsGXLFpYuXVru2KuTcBJAM2B90Occb16wJcAgb3ogUF9EGnmf64hIloh8ISIXh2w3RkSWisg4ESmxDY6I3OBtn5WbmxtGuOFZutS97OavMeH55z//yYQJE+jYsSMZGRk89dRT5d5Xw4YNuf7662nfvj3nnnsuXcPof/3GG2+kT58+dO3alfbt25Oenk6NGgcWYeeddx6NGzcudT/33HMP69evL3V5LJHgZlQlriAyGOinqtd5n68CTlXVm4LWOQ54BkgG5gGXAO1VNU9EmqnqBhE5HneVcKaqrhWRpsDPQC3geWCtqo4+WCxpaWmalZVV3mMt5q67YNw42LgREhMjsktjymXVqlWcfPLJfodhqomS/j2JyEJVTQtdN5wrgA1A86DPSd68Qqr6k6oOUtXOwD3evDzvfYP3vg6YC3T2Pm9UZw8wAVfVdFgUFLiun/v3t8LfGBO7wkkAC4AUEUkWkVrAEGBG8AoikigigX2NwrUIQkSOClTtiEgi0ANY6X1u6r0LcDGwvOKHE545c+Cnn6z6xxgT28psBqqq+SJyE/A+rhnoeFVdISKjgSxVnQH0Bh4SEcVVAY3wNj8ZeE5E9uOSzcOqutJbNlFEGgMCLAb+GMHjOqjMTGjQwHX+ZowxsSqs5wBU9V3g3ZB5fw+angoccBtfVT8DOpSyz76HFGmE7NwJb7zhun6oW9ePCIwxJjrE3JPA06fD9u1W/WOMMTGXADIzoUULqEATZmOMqRZiKgH88gt88IEb9L2E5sPGGNwDVn/4wx844YQT6NKlC7179+bLL78E3IAjt99+e+G6Y8eO5b777gPgvvvuo169evz666+FyxMSEsoVw3XXXcfKle524YMPPlg4Pzs7m/bt25e5/X333UezZs1ITU2lTZs23Hjjjezfv/+g20yfPr3wO0szd+7cwm4yDmb48OE0a9aMPXv2ALBp0yZatWpV5naHW0wVg5Mnuyag1vWDMaW77rrrOProo/nuu+9YuHAhEyZMYNOmTYDr4mHatGmFn0MlJiby+OOPVziGF198kbZtXY8zwQngUNx6660sXryYlStXsmzZMj7++OODrh9OAjgUcXFxjB8/vlzbVma32sFiKgFkZsIpp0Db0J6MjIkWI0dC796RfY0cGfbXr127li+//JIHHnig8Cnb5ORkzj//fABq1qzJDTfcwLhx40rc/tprr+XVV1/lt99+K/U7Xn/9dW677TYAnnrqqcIePtetW0ePHj0AN8BLVlYWf/nLX9i1axepqakMGzYMgIKCAq6//nratWvHOeecw65duw56THv37mX37t0c5Y33+sILL9C1a1c6derEJZdcws6dO/nss8+YMWMGd955J6mpqaxdu5Y1a9Zw1lln0alTJ0455RTWrl0LuCukwYMH06ZNG4YNG0ZpD9OOHDmScePGHVCYqyp33nkn7du3p0OHDrz66quAu7pIT09nwIABtG3bluzsbNq0acPw4cM58cQTGTZsGB9++CE9evQgJSWFr7766qDHHY6YSQCrVsHChXbz15iDWbFiBampqcV69gw1YsQIJk6cyO+//37AsoSEBK699tqDdhORnp7O/PnzAZg/fz6NGjViw4YNzJ8/v7DHzoCHH36YunXrsnjxYiZOnAjAd999x4gRI1ixYgUNGzbkjTfeKPF7xo0bR2pqKk2bNuXEE08kNTUVgEGDBrFgwQKWLFnCySefzEsvvUT37t0ZMGAAjz32GIsXL+aEE05g2LBhjBgxgiVLlvDZZ5/RtGlTABYtWsSTTz7JypUrWbduHZ9++mmJ39+iRQvOOOMMMjIyis2fNm0aixcvZsmSJXz44YfceeedbNy4EYCvv/6ap556im+//RaANWvWcPvtt/PNN9/wzTffMGnSJD755BPGjh1b7iujYDEzIExmJsTFwdChfkdizEFEQ3/QZWjQoAFXX301Tz/9NHVLaEt98803k5qayh133FHi9sceeyzbt29n27ZtrF+/niuuuIJ58+Yxf/58Bg0aVOI2wZKTkwsL8y5duhR2Lx3q1ltv5Y477mDfvn0MHjyYKVOmMGTIEJYvX869995LXl4e27dv59xzzz1g223btrFhwwYGDhwIQJ06dQqXdevWjaSkJABSU1PJzs7mjDPOKDGGUaNGcdFFFxVeQYHrjnro0KHExcXRpEkTevXqxYIFC2jQoAHdunUjOTm52LF26OBa0rdr144zzzwTESnWrXZFxMQVwP79LgGcfTY0aeJ3NMZEr3bt2rFkyRIKCgoOut7IkSN56aWX2LFjxwHLGjZsyBVXXMGzzz5b6vbdu3dnwoQJnHTSSYVXBJ9//nlhFdDB1A4auzWc7qrj4+Pp168f8+bNA9wN2meeeYZly5bxj3/846DdUFf0+1NSUkhNTeW1114La9+ldasNrivt4G62I3GfICYSwPz58OOPVv1jTFlOOOEE0tLS+Mc//lFYt52dnV04ilfA0UcfzWWXXcZLL71U4n5uu+02nnvuuVILqfT0dMaOHUvPnj3p3Lkzc+bMoXbt2hx55JEHrBsfH8++ffvKfUyqyqeffsoJJ5wAuLP7pk2bsm/fvsJqJYD69euzbdu2wumkpCSmT58OwJ49e9i5c2e5vv+ee+5h7NixhZ/T09N59dVXKSgoIDc3l3nz5tGt22HrCq2YmEgAmZmQkAAXh3ZGbYw5wIsvvsgvv/xC69atad++PcOHD+eYY445YL3bb7/9oK2BBg4cWNgMMlR6ejrr16+nZ8+exMXF0bx581KrUW644QY6duxYeBM4XIF7AO3bt6egoIA//elPANx///2ceuqp9OjRgzZt2hSuP2TIEB577DE6d+7M2rVrycjI4Omnn6Zjx450796dn3/++ZC+P6Bdu3accsophZ8HDhxIx44d6dSpE3379uXRRx/l2GOPLde+K6rM7qCjSXm7g37kEcjLg4ceKntdYw436w7aRNKhdAcdEzeB777b7wiMMSb6xEQVkDHGmANZAjAmClSlqlgTvQ7135ElAGN8VqdOHTZv3mxJwFSIqrJ58+ZizyyUJSbuARgTzZKSksjJySE3N9fvUEwVV6dOncKH1MJhCcAYn8XHxxd7+tOYw8WqgIwxJkZZAjDGmBhlCcAYY2JUlXoSWERygR/KuXkiUPJz67HJfo8i9lsUZ79HcdXh92ipqo1DZ1apBFARIpJV0qPQscp+jyL2WxRnv0dx1fn3sCogY4yJUZYAjDEmRsVSAnje7wCijP0eRey3KM5+j+Kq7e8RM/cAjDHGFBdLVwDGGGOCWAIwxpgYFRMJQET6ichqEVkjIn/xOx6/iEhzEZkjIitFZIWI3OJ3TNFAROJEZJGIvO13LH4TkYYiMlVEvhGRVSJyut8x+UVEbvX+nywXkckiEn43m1VEtU8AIhIHPAv0B9oCQ0Wkrb9R+SYfuF1V2wKnASNi+LcIdguwyu8gosRTwHuq2gboRIz+LiLSDLgZSFPV9kAcMMTfqCKv2icAoBuwRlXXqepeYApwkc8x+UJVN6rq1970Ntx/7mb+RuUvEUkCzgde9DsWv4nIkUBP4CUAVd2rqnn+RuWrmkBdEakJ1AN+8jmeiIuFBNAMWB/0OYcYL/QARKQV0Bn40t9IfPckcBew3+9AokAykAtM8KrEXhSRI/wOyg+qugEYC/wIbAR+V9UP/I0q8mIhAZgQIpIAvAGMVNWtfsfjFxG5APhVVRf6HUuUqAmcAvyfqnYGdgAxec9MRI7C1RQkA8cBR4jIlf5GFXmxkAA2AM2DPid582KSiMTjCv+JqjrN73h81gMYICLZuKrBviKS6W9IvsoBclQ1cFU4FZcQYtFZwPeqmquq+4BpQHefY4q4WEgAC4AUEUkWkVq4GzkzfI7JFyIiuPrdVar6hN/x+E1VR6lqkqq2wv27mK2q1e4sL1yq+jOwXkRO8madCaz0MSQ//QicJiL1vP83Z1INb4hX+yEhVTVfRG4C3sfdyR+vqit8DssvPYCrgGUistib91dVfdfHmEx0+TMw0TtZWgf8j8/x+EJVvxSRqcDXuNZzi6iGXUJYVxDGGBOjYqEKyBhjTAksARhjTIyyBGCMMTHKEoAxxsQoSwDGGBOjLAEYE0RECkRkcdArYk/CikgrEVkeqf0ZU1HV/jkAYw7RLlVN9TsIYw4HuwIwJgwiki0ij4rIMhH5SkRae/NbichsEVkqIh+JSAtvfhMReVNElnivQDcCcSLygtfP/AciUte3gzIxzxKAMcXVDakCujxo2e+q2gF4BteLKMA/gZdVtSMwEXjam/808LGqdsL1pxN4+jwFeFZV2wF5wCWVfDzGlMqeBDYmiIhsV9WEEuZnA31VdZ3Xod7PqtpIRDYBTVV1nzd/o6omikgukKSqe4L20QqYpaop3ue7gXhVfaDyj8yYA9kVgDHh01KmD8WeoOkC7D6c8ZElAGPCd3nQ++fe9GcUDRU4DJjvTX8E3AiFYw4febiCNCZcdvZhTHF1g3pKBTc+bqAp6FEishR3Fj/Um/dn3Ahad+JG0wr0nnkL8LyI/C/uTP9G3MhSxkQNuwdgTBi8ewBpqrrJ71iMiRSrAjLGmBhlVwDGGBOj7ArAGGNilCUAY4yJUZYAjDEmRlkCMMaYGGUJwBhjYtT/Azip+nFeJbYsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(accuracy_list_normal, 'b',label=' normal CNN')\n",
    "plt.plot(accuracy_list_batch,'r',label=' CNN with Batch Norm')\n",
    "plt.xlabel('Epoch')\n",
    "plt.title(\"Accuracy \")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see the CNN with batch norm performers better, with faster convergence.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://dataplatform.cloud.ibm.com/registration/stepone?context=cpdaas&apps=data_science_experience,watson_machine_learning\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/Watson_Studio.png\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other contributors: <a href=\"https://www.linkedin.com/in/michelleccarey/\">Michelle Carey</a>, <a href=\"www.linkedin.com/in/jiahui-mavis-zhou-a4537814a\">Mavis Zhou</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to Magnus <a href=\"http://www.hvass-labs.org/\">Erik Hvass Pedersen</a> whose tutorials helped me understand convolutional Neural Network\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description                                          |\n",
    "| ----------------- | ------- | ---------- | ----------------------------------------------------------- |\n",
    "| 2020-09-23        | 2.0     | Srishti    | Migrated Lab to Markdown and added to course repo in GitLab |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}