ICA/ICA3_MachineLearning.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href='https://fonts.googleapis.com/css?family=Passion+One' rel='stylesheet' type='text/css'><style>div.attn { font-family: 'Helvetica Neue'; font-size: 30px; line-height: 40px; color: #FFFFFF; text-align: center; margin: 30px 0; border-width: 10px 0; border-style: solid; border-color: #5AAAAA; padding: 30px 0; background-color: #DDDDFF; }hr { border: 0; background-color: #ffffff; border-top: 1px solid black; }hr.major { border-top: 10px solid #5AAA5A; }hr.minor { border: none; background-color: #ffffff; border-top: 5px dotted #CC3333; }div.bubble { width: 65%; padding: 20px; background: #DDDDDD; border-radius: 15px; margin: 0 auto; font-style: italic; color: #f00; }em { color: #AAA; }div.c1{visibility:hidden;margin:0;height:0;}div.note{color:red;}</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  Ebnable HTML/CSS \n",
    "from IPython.core.display import HTML\n",
    "HTML(\"<link href='https://fonts.googleapis.com/css?family=Passion+One' rel='stylesheet' type='text/css'><style>div.attn { font-family: 'Helvetica Neue'; font-size: 30px; line-height: 40px; color: #FFFFFF; text-align: center; margin: 30px 0; border-width: 10px 0; border-style: solid; border-color: #5AAAAA; padding: 30px 0; background-color: #DDDDFF; }hr { border: 0; background-color: #ffffff; border-top: 1px solid black; }hr.major { border-top: 10px solid #5AAA5A; }hr.minor { border: none; background-color: #ffffff; border-top: 5px dotted #CC3333; }div.bubble { width: 65%; padding: 20px; background: #DDDDDD; border-radius: 15px; margin: 0 auto; font-style: italic; color: #f00; }em { color: #AAA; }div.c1{visibility:hidden;margin:0;height:0;}div.note{color:red;}</style>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "Enter Team Member Names here (*double click to edit*):\n",
    "\n",
    "- Name 1: Will Lacey\n",
    "- Name 2: Roger Wong\n",
    "- Name 3:\n",
    "\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In Class Assignment Three\n",
    "In the following assignment you will be asked to fill in python code and derivations for a number of different problems. Please read all instructions carefully and turn in the rendered notebook (or HTML of the rendered notebook)  before the end of class (or right after class). The initial portion of this notebook is given before class and the remainder is given during class. Please answer the initial questions before class, to the best of your ability. Once class has started you may rework your answers as a team for the initial part of the assignment. \n",
    "\n",
    "<a id=\"top\"></a>\n",
    "## Contents\n",
    "* <a href=\"#Loading\">Loading the Data</a>\n",
    "* <a href=\"#ff\">Defining a Feedforward Network in Python</a>\n",
    "* <a href=\"#bp\">Back Propagation in Python</a>\n",
    "* <a href=\"#vis\">Visualizing Back Propagation</a>\n",
    "________________________________________________________________________________________________________\n",
    "\n",
    "<a id=\"Loading\"></a>\n",
    "<a href=\"#top\">Back to Top</a>\n",
    "## Loading the Data\n",
    "Please run the following code to read in the \"digits\" dataset from sklearn's data loading module. This data contains hand written digits for the characters 0-9.\n",
    "\n",
    "This will load the data into the variable `ds`. `ds` is a `bunch` object with fields like `ds.data` and `ds.target`. The field `ds.data` is a numpy matrix of the continuous features in the dataset. **The object is not a pandas dataframe. It is a numpy matrix.** Each row is a set of observed instances, each column is a different feature. It also has a field called `ds.target` that is an integer value we are trying to predict (i.e., a specific integer represents a specific person). Each entry in `ds.target` is a label for each row of the `ds.data` matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1797, 64)\n",
      "(1797,)\n",
      "-0.5 0.5\n",
      "[0 1 2 3 4 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "# lets load up the handwritten digit dataset\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "\n",
    "ds = load_digits()\n",
    "X = ds.data/16.0-0.5 # normalize the data\n",
    "y = ds.target\n",
    "\n",
    "print(X.shape)\n",
    "print(y.shape)\n",
    "print(np.min(X),np.max(X))\n",
    "print(np.unique(y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADFCAYAAABjLIjfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAC/NJREFUeJzt3U9IVVsbx/F13i6UhZVlRBldsUEWRYQWVIMCBxEFamEaDUoHRQVFNqhBTsoGQQoFNgqtwEIalIP+UEk0UCM1AgcWpRglkVezKEmoOHfwTu699Dz77OU520f6fqa/s93L9Z7zew/53LVj8XjcAQAm3v8megEAgP+jkAHACAoZAIygkAHACAoZAIygkAHACAoZAIygkAHACAoZAIz4I8yLMzMz49nZ2aFvMjo6qub9/f1iNnv2bDFbuHChmMViscB1SWsZGhpK+GLfPQnS29srZt+/fxezxYsXi9n06dO919PV1TUUj8fnJfLaVO3J2NiYmL148ULM0tPTxWzJkiXe6wmzJ87578vw8LCaa5+fadOmidny5cvFbLJ/frT/AvnNmzdiloq1OJf4eyVUIWdnZ7vOzs7Qi2lvb1fziooKMdu+fbuYVVVViZn2RtTk5+eHer3vngQpLi4Ws8HBQTE7f/68mIX93f4pFovJ7+L/SNWevHz5UszWrVsnZhs3bhSzmzdveq8nzJ44578vV65cUfO9e/eq95S0traK2WT//Gj/533w4EExq6+vT/panEv8vcI/WQCAERQyABhBIQOAERQyABhBIQOAEaGmLHxpUxTO6SNLHz9+FLO0tDQxa2trU++p/VXegoyMDDG7deuWmN27d0/MxjNlEYWBgQE1z83NFTNtv7q7u73XFJWamhoxu3Tpknrt7du3xWzr1q1i1tfXJ2baSNxk0NzcLGaWPwd8QwYAIyhkADCCQgYAIyhkADCCQgYAIyhkADAiaWNvb9++FTNtrM05fbRNG2fSrrM+9hY04qWNtmkm+vcaD21UyTnn1q9fL2a7d+8Ws0OHDnmvKSraaGjQ+levXi1m2qjgZB5t0w4Pcs65CxcuiNmpU6fE7NOnT95r0k6mTBTfkAHACAoZAIygkAHACAoZAIygkAHACAoZAIygkAHAiKTNIX/58kXMNm3apF6rzRpr1q5d63VdVJqamsTswIED6rUjIyNe98zLy/O6zoKgY1qXLl0qZiUlJWJWXl7uvaaoaJ+BoPeCNue/c+dOMdNmeX0fchqVoJn1np4eMSsoKBCz6upqMZszZ456T+3hqYniGzIAGEEhA4ARFDIAGEEhA4ARFDIAGEEhA4ARSRt7+/z5s5ht27YtWbf5F+34zaARlSiUlpaKWWFhoXqt9kRtzejoqJgl43jA8dJGrerr69VrGxsbve558eJFr+usCBoL/fbtm5ht2bLFK7t79656zyjG4jo7O8WsrKxMvbaystLrnlVVVWL28OFDr58ZBt+QAcAIChkAjKCQAcAIChkAjKCQAcAIChkAjEja2NusWbPE7OnTp94/VxuT0p4svXfvXu97TmbayV9ZWVkRruTXzp07J2bayFGQjo4OMbN+ctl4ab+fNr529OhRMaurq1PveezYseCFjVN6erqYBY0C1tbWitmTJ0+81rNhwwav68LgGzIAGEEhA4ARFDIAGEEhA4ARFDIAGEEhA4ARSRt7W7BggZi1tLSo17a3t4vZ1atXvdazZ88er+uQWtoDR4NOGNPGHNesWeN1z6AHU+bn56t5FGpqatRcO7VNO4Xxxo0bYrZ///7ghaWY9lBb7aRH55wbGBgQs5UrV4qZdkpcFOOTfEMGACMoZAAwgkIGACMoZAAwgkIGACMoZAAwgkIGACOSNoesHYcXNEtcUVEhZps2bRKzR48eBa7LqqCZRm12tqGhQczu3LkjZgUFBcELSzHtCNDW1lb1Wm22VDu6U9uvnJwc9Z4W5pAzMzPVfMeOHV4/V5s1PnPmjNfPtGLGjBliNjIyImb79u1LxXISxjdkADCCQgYAIyhkADCCQgYAIyhkADCCQgYAI2LxeDzxF8difznn3qRuOSb8GY/H5yX64t9kT5wLsS/sya/9JvvCnvxaQvsSqpABAKnDP1kAgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYQSEDgBF/hHlxZmZmPDs7O/RNent71Xzq1KlitmjRotD3G4/+/n43NDQUS/T1vnsSRNuz79+/i1lubm7S1+Kcc11dXUPxeHxeIq/13ZORkRE1//Hjh5gNDw+L2ejoqJhNmTJFveeqVavE7NmzZwnviXP++/L+/Xs11373+fPna+sRs1gs4Y/Av0T1+env71fznz9/itmSJUtC32+8Ev38hCrk7Oxs19nZGXoxxcXFap6TkyNmNTU1oe83Hvn5+aFe77snQbQ9GxwcFLPW1takr8U552Kx2JtEX+u7J01NTWquFU9jY6OYtbW1idnMmTPVe2r7mZaWlvCeOOe/L9XV1Wp++fJlMausrBSziooKMZs2bVrgun4lqs+Ptnbn9P9zv3nzZuj7jVeinx/+yQIAjKCQAcAIChkAjKCQAcCIUH/U89Xd3a3mt27dErPa2lox0/5a+vr16+CFTaCgP2Roe1JXV5fs5UwKc+fOFbP6+noxO3v2rJgFTXb4/nErmbq6uryv1T4/Dx48ELOJ+MPXf3369EnMGhoavH+uNkGyfv16MUvVH8z/iW/IAGAEhQwARlDIAGAEhQwARlDIAGAEhQwARkQy9qYdcOKcfpBORkaGmBUWForZ2NiYes+JHmc6cuSI97Xa7z2ZlZaWel978eJFMXv58qWYtbS0eN8zKnl5eWruexbMnDlzxEzbM+ecW7p0qZong3YoVJCioiIx0/arubnZ+57JwDdkADCCQgYAIyhkADCCQgYAIyhkADCCQgYAIyhkADAikjnkoJlF7Zln2vGIa9euFbOJnjMO8uHDBzXXjgHMyspK9nIik6qZ4JMnT3pdF3SkYkFBgdfPTaby8nI11x4E3NfXJ2baHHLQfzsQBe241SDXr18Xs127donZx48fve+ZDHxDBgAjKGQAMIJCBgAjKGQAMIJCBgAjKGQAMCKSsTfticDOOXf8+HExe/78uZiVlZV5r2k8Rz0mQ9B4zcqVK8WsqalJzDZv3ixms2fPDl5YimnjVON5Eremvb1dzKI4RnK8vn796n2ttmfaSKmF94o2uqqNhTrnXFpampidPn1azB4/fixm2lOwnUvOnvENGQCMoJABwAgKGQCMoJABwAgKGQCMoJABwIhIxt6CpGL06NWrV0n/mcm0bNkyNdfGlQYHB8VMGwV89+6des8oTpHTRoOCxiMbGhrErKOjQ8wmw2jbwMCAmOXm5qrX1tXViZn2RPetW7eK2e3bt9V7TvRYXNApfdp++r7PKysr1Tzo/ZsIviEDgBEUMgAYQSEDgBEUMgAYQSEDgBEUMgAYEcnYW9ApXunp6WJ24sQJr3uWlJR4XReVw4cPq7n24FdtjKunp0fMmpub1XsePHhQzVOturpazTMyMsRsxYoVyV5OpLQHemq/t3POVVRUiNnw8LCYaQ9HvXbtmnrPiX6vBNFG27T3WW1trZhppwYmC9+QAcAIChkAjKCQAcAIChkAjKCQAcAIChkAjKCQAcCISOaQ7927p+ZVVVVeP1c7Ds/6kYuFhYVqrj0ZV5uVLCoq8r7nRLt7966a379/X8y0JxRPBtr6tf9NndOfsKzNMJeXl4uZNttsQdDMeldXl5hpx9d2d3eLWRTH0/INGQCMoJABwAgKGQCMoJABwAgKGQCMoJABwIhYPB5P/MWx2F/OuTepW44Jf8bj8XmJvvg32RPnQuwLe/Jrv8m+sCe/ltC+hCpkAEDq8E8WAGAEhQwARlDIAGAEhQwARlDIAGAEhQwARlDIAGAEhQwARlDIAGDE31Qr/olk1Zx5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# reshape and print a few of the images in the digits dataset\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "for i in range(10):\n",
    "    img = X[i].reshape(8, 8)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____\n",
    "**Question 1:** For the digits dataset, what does each column in $\\mathbf{X}$ represent? What does each row in $\\mathbf{X}$ represent? What does each value in $\\mathbf{X}$ represent? What does each unique value of the target, $y$ represent?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each column in X is a pixel location\n",
    "\n",
    "Each row in X is an image.\n",
    "\n",
    "Each value in X is the darkness of a pixel.\n",
    "\n",
    "Each target in y is a class from digits '0' - '9'\n",
    "\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____\n",
    "**Question 2:** For the digits dataset, we want to train a neural network with one hidden layer. The hidden layer will have 30 neurons. What will be the size of the matrices in each layer? That is, what is the size of $\\mathbf{W}^{(1)}$ and what is the size of $\\mathbf{W}^{(2)}$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "W1 = 30 * 65\n",
    "\n",
    "W2 = 31 * 10\n",
    "\n",
    "Giving a total of 2260 weights needed to be optimized"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ff\"></a>\n",
    "<a href=\"#top\">Back to Top</a>\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/eclarson/MachineLearningNotebooks/master/PDF_Slides/MultiLayerNetwork.png\" width=\"500\">\n",
    "\n",
    "\n",
    "# Defining a Feedforward Network\n",
    "\n",
    "Below we will setup the functions for use in a feedforward neural network **with two layers**. Take a quick look at the functions defined. There are a number of convenience functions including:\n",
    "- a function for the sigmoid calculation\n",
    "- a function to one hot encode the output\n",
    "- a function to add ones (for intercept/bias) terms on the row or column\n",
    "- an initialization function for initializing the weights\n",
    "\n",
    "A few functions are not yet implemented including:\n",
    "- a `fit` function\n",
    "- a `get_gradient` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example adapted from https://github.com/rasbt/python-machine-learning-book/blob/master/code/ch12/ch12.ipynb\n",
    "# Original Author: Sebastian Raschka\n",
    "\n",
    "# This is the optional book we use in the course, excellent intuitions and straightforward programming examples\n",
    "# please note, however, that this code has been manipulated to reflect our assumptions and notation.\n",
    "import numpy as np\n",
    "from scipy.special import expit\n",
    "import sys\n",
    "import pandas as pd\n",
    "\n",
    "# start with a simple base classifier, which can't be fit or predicted\n",
    "# it only has internal classes to be used by classes that will subclass it\n",
    "class TwoLayerPerceptronBase(object):\n",
    "    def __init__(self, n_hidden=30,\n",
    "                 C=0.0, epochs=500, eta=0.001, random_state=None):\n",
    "        np.random.seed(random_state)\n",
    "        self.n_hidden = n_hidden\n",
    "        self.l2_C = C\n",
    "        self.epochs = epochs\n",
    "        self.eta = eta\n",
    "        \n",
    "    @staticmethod\n",
    "    def _encode_labels(y):\n",
    "        \"\"\"Encode labels into one-hot representation\"\"\"\n",
    "        onehot = pd.get_dummies(y).values.T\n",
    "            \n",
    "        return onehot\n",
    "\n",
    "    def _initialize_weights(self):\n",
    "        \"\"\"Initialize weights with small random numbers.\"\"\"\n",
    "        W1_num_elems = (self.n_features_ + 1)*self.n_hidden\n",
    "        W1 = np.random.uniform(-1.0, 1.0, size=W1_num_elems)\n",
    "        W1 = W1.reshape(self.n_hidden, self.n_features_ + 1) # reshape to be W\n",
    "        \n",
    "        W2_num_elems = (self.n_hidden + 1)*self.n_output_\n",
    "        W2 = np.random.uniform(-1.0, 1.0, size=W2_num_elems)\n",
    "        W2 = W2.reshape(self.n_output_, self.n_hidden + 1)\n",
    "        return W1, W2\n",
    "    \n",
    "    @staticmethod\n",
    "    def _sigmoid(z):\n",
    "        \"\"\"Use scipy.special.expit to avoid overflow\"\"\"\n",
    "        # 1.0 / (1.0 + np.exp(-z))\n",
    "        return expit(z)\n",
    "    \n",
    "    @staticmethod\n",
    "    def _add_bias_unit(X, how='column'):\n",
    "        \"\"\"Add bias unit (column or row of 1s) to array at index 0\"\"\"\n",
    "        if how == 'column':\n",
    "            ones = np.ones((X.shape[0], 1))\n",
    "            X_new = np.hstack((ones, X))\n",
    "        elif how == 'row':\n",
    "            ones = np.ones((1, X.shape[1]))\n",
    "            X_new = np.vstack((ones, X))\n",
    "        return X_new\n",
    "    \n",
    "    def _feedforward(self, X, W1, W2):\n",
    "        \"\"\"Compute feedforward step\n",
    "        -----------\n",
    "        X : Input layer with original features.\n",
    "        W1: Weight matrix for input layer -> hidden layer.\n",
    "        W2: Weight matrix for hidden layer -> output layer.\n",
    "        ----------\n",
    "        a1-a3 : activations into layer (or output layer)\n",
    "        z1-z2 : layer inputs \n",
    "\n",
    "        \"\"\"\n",
    "        A1 = self._add_bias_unit(X.T, how='row')\n",
    "        Z1 = W1 @ A1\n",
    "        A2 = self._sigmoid(Z1)\n",
    "        \n",
    "        A2 = self._add_bias_unit(A2, how='row')\n",
    "        Z2 = W2 @ A2\n",
    "        A3 = self._sigmoid(Z2)\n",
    "        return A1, Z1, A2, Z2, A3\n",
    "    \n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict class labels\"\"\"\n",
    "        _, _, _, _, A3 = self._feedforward(X, self.W1, self.W2)\n",
    "        y_pred = np.argmax(A3, axis=0)\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "<img src=\"https://raw.githubusercontent.com/eclarson/MachineLearningNotebooks/master/PDF_Slides/MultiLayerNetwork.png\" width=\"500\">\n",
    "\n",
    "Recall from the videos that all instances in $\\mathbf{X}$ can be fed into the network with a single matrix multiplication operation for each intermediate vector, $\\mathbf{a}^{(l)}$ and $\\mathbf{z}^{(l)}$. When we feed all the instances, $\\mathbf{X}$, the intermediate vectors, $\\mathbf{a}^{(l)}$ and $\\mathbf{z}^{(l)}$ get stacked together to form matrices, $\\mathbf{A}^{(l)}$ and $\\mathbf{Z}^{(l)}$. This is already done for you in the `_feedforward` function defined above.\n",
    "\n",
    "**Question 3:**\n",
    "For the digits dataset we are using and a network with 30 neurons in the hidden layer, what are the sizes of:\n",
    "- **Part A**: the intermediate vectors, $\\mathbf{a}^{(1)}$ and $\\mathbf{a}^{(2)}$\n",
    "- **Part B**: the intermediate vectors, $\\mathbf{z}^{(1)}$ and $\\mathbf{z}^{(2)}$\n",
    "- **Part C**: the intermediate matrices, $\\mathbf{A}^{(1)}$ and $\\mathbf{A}^{(2)}$\n",
    "- **Part D**: the intermediate matrices, $\\mathbf{Z}^{(1)}$ and $\\mathbf{Z}^{(2)}$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A. a1 = 65, a2 = 31\n",
    "\n",
    "B. z1 = 30, z2 = 10\n",
    "\n",
    "C. A1 = 65 x 1797; A2 = 31 x 1797\n",
    "\n",
    "D. Z1 = 30 x 1797; Z2 = 10 x 1797"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "<a id=\"bp\"></a>\n",
    "<a href=\"#top\">Back to Top</a>\n",
    "\n",
    "# Back Propagation in Python\n",
    "Now let's add in the back propagation steps from the video. First, we need to add in a `fit` function that will update all the trainable weights in the $\\mathbf{W}^{(l)}$ matrices. Because this is a two layer network we have layers $\\mathbf{W}^{(1)}$ and $\\mathbf{W}^{(2)}$. Look at the given `fit` function written below for you. \n",
    "\n",
    "The `fit` function below will update matrices using steepest descent. And the update equation is:\n",
    "$$  w_{i,j}^{(l)} \\leftarrow w_{i,j}^{(l)} - \\eta \\frac{\\partial J(\\mathbf{W})}{\\partial w_{i,j}^{(l)}}$$\n",
    "\n",
    "for each value in each matrix, $\\mathbf{W}^{(l)}$. \n",
    "\n",
    "The objective function is simply the mean squared error:\n",
    "$$ J(\\mathbf{W}) = \\sum_{k=1}^M (\\mathbf{y}^{(k)}-[\\mathbf{a}^{(L)}]^{(k)})^2 $$\n",
    "\n",
    "where $L$ is the output of the last layer. For our two layer implementation, $L=3$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TwoLayerPerceptron(TwoLayerPerceptronBase):\n",
    "    def fit(self, X, y):\n",
    "        \"\"\" Learn weights from training data.\n",
    "            Input X should be a matrix with separate instances\n",
    "            in each row, and separate features in each column.\n",
    "            The target variable, y, should be integer values \n",
    "            starting from zero, that represent the unique classes.\n",
    "        \"\"\"\n",
    "        \n",
    "        # copy over the data and one-hot-encode Y\n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        Y_enc = self._encode_labels(y)\n",
    "        \n",
    "        # init weights and setup matrices\n",
    "        self.n_features_ = X_data.shape[1]\n",
    "        self.n_output_ = Y_enc.shape[0]\n",
    "        self.W1, self.W2 = self._initialize_weights()\n",
    "\n",
    "        #------------------------------------------\n",
    "        # ENTER YOUR CODE HERE here to initialize \n",
    "        # self.grad_w1_ and self.grad_w2_\n",
    "        \n",
    "        self.grad_w1_ = np.array([])\n",
    "        self.grad_w2_ = np.array([])\n",
    "        \n",
    "        #------------------------------------------\n",
    "        \n",
    "        # run training for specified number of iterations\n",
    "        for i in range(self.epochs):\n",
    "\n",
    "            # feedforward all instances\n",
    "            A1, Z1, A2, Z2, A3 = self._feedforward(X_data,self.W1,self.W2)\n",
    "\n",
    "            # compute gradient via backpropagation\n",
    "            # WE HAVE NOT YET WRITTEN THE GRADIENT FUNCTION YET\n",
    "            grad1, grad2 = self._get_gradient(A1=A1, A2=A2, A3=A3, Z1=Z1, Z2=Z2, Y=Y_enc,\n",
    "                                              W1=self.W1, W2=self.W2)\n",
    "            \n",
    "            #------------------------------------------\n",
    "            # ENTER YOUR CODE HERE to keep track of the \n",
    "            # magnitude of gradients grad_w1_ and grad_w2_\n",
    "            \n",
    "            mag1 = np.linalg.norm(grad1)\n",
    "            mag2 = np.linalg.norm(grad2)     \n",
    "            \n",
    "            self.grad_w1_ = np.append(self.grad_w1_, np.mean(mag1))\n",
    "            self.grad_w2_ = np.append(self.grad_w2_, np.mean(mag2))\n",
    "            \n",
    "            #------------------------------------------\n",
    "\n",
    "            # update each matrix to minimize objective function\n",
    "            self.W1 -= self.eta * grad1\n",
    "            self.W2 -= self.eta * grad2\n",
    "            \n",
    "\n",
    "        return self\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1:** In the code above, add class variables, `grad_w1_` and `grad_w2_` that save the average magnitude of the gradient for each layer at every epoch. That is, if you ran the `fit` function for 50 epochs, `grad_w1_` and `grad_w2_` would be 50 element vectors when training is complete.\n",
    "\n",
    "When training is completed, `grad_w1_` and `grad_w2_` should be accessible using dot notation from the class object, as shown in the example syntax below. \n",
    "\n",
    "```\n",
    "clf = TwoLayerPerceptron()\n",
    "clf.fit(X,y)\n",
    "clf.grad_w1_ \n",
    "clf.grad_w2_\n",
    "```\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding the Gradient Calculation\n",
    "From the videos, recall that the the sensitivities, $\\mathbf{V}$, can be calculated as follows:\n",
    "\n",
    "$$ \\mathbf{V}^{(2)} = -2(\\mathbf{Y}-\\mathbf{A}^{(3)})*\\mathbf{A}^{(3)}*(1-\\mathbf{A}^{(3)}) $$\n",
    "$$ \\mathbf{V}^{(1)} =  \\mathbf{A}^{(2)} * (1-\\mathbf{A}^{(2)}) * [\\mathbf{W}^{(2)}]^T \\cdot \\mathbf{V}^{(2)}  $$\n",
    "\n",
    "Once we have these sensitivities, its easy to calculate the gradient of each layer. The gradient of the objective function with respect to the final layer $\\mathbf{W}^{(2)}$ can be calculated with:\n",
    "$$  \\frac{\\nabla J(\\mathbf{W})}{\\partial\\mathbf{W}^{(2)}} = \\mathbf{V}^{(2)}\\cdot [\\mathbf{A}^{(2)}]^T $$\n",
    "\n",
    "And the gradient of the objective function with respect to the first layer $\\mathbf{W}^{(2)}$ can be calculated with: \n",
    "$$  \\frac{\\nabla J(\\mathbf{W})}{\\partial\\mathbf{W}^{(1)}} = \\mathbf{V}^{(1)}\\cdot [\\mathbf{A}^{(1)}]^T $$\n",
    "\n",
    "___\n",
    "** Exercise 2:** In the code below, use numpy linear algebra functions to calculate the sensitivities at each layer, $\\mathbf{V}^{(1)}$ and $\\mathbf{V}^{(2)}$. This will complete the `_get_gradient` private method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TwoLayerPerceptronVect(TwoLayerPerceptron):\n",
    "    # just need a different gradient calculation\n",
    "    def _get_gradient(self, A1, A2, A3, Z1, Z2, Y, W1, W2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "        \"\"\"\n",
    "        \n",
    "        #---------------------------------\n",
    "        # ENTER YOUR CODE HERE to\n",
    "        # calculate the sensitivities\n",
    "        \n",
    "        V2 = -2 * (Y - A3) * A3 * (1 - A3)\n",
    "        V1 = A2 * (1 - A2) * ((W2.T) @ V2)\n",
    "        \n",
    "        #---------------------------------\n",
    "        \n",
    "        grad2 = V2 @ A2.T\n",
    "        grad1 = V1[1:,:] @ A1.T # don't include added bias term in recurrence relation\n",
    "\n",
    "        return grad1, grad2\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "<a id=\"vis\"></a>\n",
    "<a href=\"#top\">Back to Top</a>\n",
    "\n",
    "# Visualizing Back Propagation\n",
    "Now let's test the code you wrote above to perform gradient calculations and save the magnitude of the gradient at each epoch. Run the code in the cell below. If it runs and the accuracy is relatively high, it is likely that your sensitivity calculation above was written correctly.\n",
    "\n",
    "*If it does not run or the accuracy is low, there is probably something wrong with the sensitivity calculation. Try to fix it before moving on.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9827490261547023"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = TwoLayerPerceptronVect(n_hidden=10, epochs=1500, eta=0.001)\n",
    "clf.fit(X,y)\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "yhat = clf.predict(X)\n",
    "accuracy_score(y,yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Visualizing the gradients\n",
    "Now let's check your calculation of the average gradient magnitude. Run the code below to visualize the average gradient magnitude versus the epochs run while training. \n",
    "\n",
    "*If the code below does not run, you likely have an error in your calculation of the average gradient magnitude.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8VfX9+PHX+2bvQQaBAGEPZSMKCl9cFK3W4qizjqpYa1tra61+7fenftt+26pt7VbrqG3VVq27bsSFCoYle68AIYPsPd6/P86JJJBxSHJzk9z38/E4j3vPvO979ObN+UxRVYwxxgQvX6ADMMYYE1iWCIwxJshZIjDGmCBnicAYY4KcJQJjjAlylgiMMSbIWSIwxpggZ4nAGGOCnCUCY4wJcqGBDsCLlJQUzcrKCnQYxhjTp6xYsaJAVVM7Oq5PJIKsrCyys7MDHYYxxvQpIrLby3FWNGSMMUHOEoExxgQ5SwTGGBPk+kQdgTHG+ENdXR05OTlUV1cHOpQuiYyMJDMzk7CwsE6db4nAGBO0cnJyiIuLIysrCxEJdDidoqoUFhaSk5PD8OHDO3UNKxoyxgSt6upqBgwY0GeTAICIMGDAgC491VgiMMYEtb6cBJp09TtYIgCoq4LPHoFN/wl0JMYY0+MsEajCs9fAf34A/7wMlj0c6IiMMeYLd955J0OGDCE2NtZvn2GJYOtbsOV1OP0uGHUmvHM3VBQGOipjjAHg3HPPZfny5X79DEsEnz0KsQNh9ndg/k+hrgJWPB7oqIwxQeK+++7jd7/7HQC33HILp512GgDvvvsul19+OSeddBIZGRl+jSG4m4+WHYRtb8Mpt0BIGKSNg2GnwJp/wpwfQD+oRDLGeHPPK+vZsL+0W685YVA8d517XLvHzJkzh1/96ld897vfJTs7m5qaGurq6vjwww+ZO3dut8bTluB+Iti+GLQRJnz18LbjF0LhVsjbGLi4jDFBY/r06axYsYLS0lIiIiKYNWsW2dnZfPjhh8yZM6dHYgjuJ4Lt70JMGqQff3jbuHPhP7fCxlcgfULgYjPG9KiO/uXuL2FhYQwfPpy//vWvzJ49m0mTJrFkyRK2bdvG+PHjeySG4H0iUIUd78HIU8HX7DbEpcOgKbBjScBCM8YElzlz5nD//fczd+5c5syZw4MPPsjUqVN7rI9D8CaC4t1QkQ9DTjx634h5kPMZ1JT1dFTGmCA0Z84cDhw4wKxZs0hPTycyMvKLYqHbbruNzMxMKisryczM5O677+72zw/eoqF9K53XwdOO3jdiHnz0G9j9MYz5Uk9GZYwJQqeffjp1dXVfrG/ZsuWL9/feey/33nuvXz8/eJ8I9q+EkHBIa6VccMhJEBoJ2614yBjT/wVxIljtVBKHhh+9LywShp4EOz/o+biMMaaH+TURiEiiiDwnIptEZKOIzBKRZBF5W0S2uq9J/oyhTfmb2m8VlDUH8tZbL2NjTL/n7yeC3wJvqOo4YDKwEbgdWKyqo4HF7nrPqjzkVBSnjmv7mCy3/e7uj3omJmOMCRC/JQIRSQDmAo8CqGqtqhYD5wFPuIc9AXy19Sv4UYFbEZMytu1jBk2FsGjYZYnAGNO/+fOJYDiQDzwuIqtE5BERiQHSVfWAe0wukO7HGFqXv9l5TR3T9jGh4U49gSUCY0w/589EEApMA/6sqlOBCo4oBlJVBbS1k0VkkYhki0h2fn5+90ZWsMVpFZQwtP3jsk6BvA1QUdC9n2+MMR5UVlby5S9/mXHjxnHcccdx++3+KUn3ZyLIAXJUdZm7/hxOYjgoIhkA7mteayer6sOqOkNVZ6SmpnZvZEW7ICmrZY/i1jTVE9hTgTEmQG699VY2bdrEqlWrWLp0Ka+//nq3f4bfEoGq5gJ7RaSpIP50YAPwMnCVu+0q4CV/xdCm4j2QOJTckmruf3Mzz2bvxXk4OcKgqRAWY4nAGOM37Q1Dff3113PqqacCEB4ezrRp08jJyen2GPzds/g7wJMiEg7sAK7BST7PiMi1wG7ga36O4WjFe6hMn87CPy3lQIkz4XNJVR3XzRnR8riQMKsnMCZYvH475K7t3msOnAhn/aLdQ7wOQ11cXMwrr7zCzTff3L0x4ufmo6q62i3emaSqX1XVIlUtVNXTVXW0qp6hqof8GcNRqkugupjFByIprKjllW+fwqljU/nt4q2UVNYdfXzWKZC/Ecq7uZ7CGGPwNgx1fX09l156Kd/97ncZMWJEB1c8dsE31lDxXgDe2BfGNSdnMTEzgVvOHMNX/rCUV9fu5/ITh7U8vnl/guMW9nCwxpge08G/3P3FyzDUixYtYvTo0Xzve9/zSwzBN8RE8R4A9pH2RVHQxMEJjEqL5eXV+48+ftAUCI+14iFjjN+0Nwz1j3/8Y0pKSnjggQf89vlBlwgai3YDkDVyPKlxEQCICGdOSGfF7iIqaupbnmD1BMYYP2trGOqcnBx+9rOfsWHDBqZNm8aUKVN45JFHuv3zg65o6ND+bURrBHMnt+xVfPLIFP783nY+23WIeWPTWp6UdQq8c7dTTxDbzU1ZjTFBr71hqFtt0djNgu6JoOTgbg5oMiePbvkHffqwJMJDfHy8vZVB5r7oT/BhD0RojDE9K+gSQX1JLmVhA0iPj2yxPSo8hImZCazaU3T0SRlWT2CM6b+CKhGoKlE1+UhsWqv7J2UmsG5fKQ2NRzyKhYTC0FmWCIzph3qi6MXfuvodgioR5JXVkKzFhCZktLp/UmYCVXUNbM8vP3pn1ilQsBnKWx0RwxjTB0VGRlJYWNink4GqUlhYSGRkZMcHtyGoKos37znAXKkmLmVwq/snDk4E4POcEsakx7XcObxZPcHxF/gzTGNMD8nMzCQnJ4duH9iyh0VGRpKZmdnp8ztMBCISDfwAGKqq14vIaGCsqr7a6U8NkL17dgGQkjGs1f0jUmKICQ/h85xiLpx+xE0dOBnC45ziIUsExvQLTZ25gp2XoqHHgRpglru+D/ip3yLyo8KDTq/i6KTWi4Z8PmHCoHg2Hig9emdIKAyzegJjTP/jJRGMVNV7gToAVa0ExK9R+UltsTsfTtzANo8ZOzCOTbllrZcZDp/rzGXgDlNhjDH9gZdEUCsiUbgTyIjISJwnhD5Hyw46b2LbnhRtbHocZdX15JZWH71zzALndcsbfojOGGMCw0siuAt4AxgiIk/iTDh/m1+j8oOq2gaia/NpkBCISm7zuLED4wHYlFt29M6U0TBgFGx+zV9hGmNMj+swEajq28D5wNXA08AMVX3Pv2F1v5yiSlIpoTYipd2Zyca6rYU2t5YIAMaeDTs/dIazNsaYfqDNv4giMq1pAYYBB4D9wFB3W5+y51AlaVJMY0zrncmaJESHMTA+ki3tJYLGOti22A9RGmNMz2uv+eiv3NdIYAawBqeSeBKQzeFWRH1CTlEVM6SY0ISxHR7bVGHcqiEzIXoAbH4djj+/m6M0xpie1+YTgaqeqqqn4jwJTHNnGpsOTMVpQtqn5JZWkybFhCe03WKoydiBcWzLL6e+ofHonb4QGHMWbHkT6vtknbkxxrTgpbJ4rKp+MZGnqq4DxvsvJP/IK6kgWcqQdpqONhmbHkdtfSO7CitbP+C4hVBTYsVDxph+wUsi+FxEHhGRee7yF+BzfwfW3WqKDxJCI7Qx4FxzYwd2UGE84r+clkfr/t2dIRpjTEB4SQTXAOuBm91lg7utT2koy3XeeHgiGJUWi09gc24rPYzBmbVswnlOM9Laim6M0hhjep6X5qPVqvobVV3oLr9R1VZ6W/VuvqZRQ9vpTNYkMiyE4SkxbVcYgzPeUF2lU1dgjDF9mJdB53bi9ipuTlVH+CUiP6iuayC6rhDC8JQIAMZlxLM2p52+AsNmQ+xAp3jIWg8ZY/owL8NQz2j2PhK4CGi7a24vlFdaQxrFzorHRDB+YBz/+fwA5TX1xEa0cpt8IU6lcfZjTueyyIRujNgYY3qOl6KhwmbLPlV9APiyl4uLyC4RWSsiq0Uk292WLCJvi8hW9zWpi9+hQ3ll1aRKMXVh8RDmbfKGce5QE21WGANMvAgaamD9C90RpjHGBESHiaB5D2MRmSEi3+TYJrQ5VVWnqGrTk8XtwGJVHY0zbtHtxx72sTlYWuOpV3Fz4zKclkOb2qowBhg8DVLGwuqnuhqiMcYEjJc/6L9q9r4e2Al8rQufeR4wz33/BPAe8KMuXK9DB0urmSjF+OI7bjHUZHBiFHERoWw60M4TgQhMuQzeuQsKt8OAkd0QrTHG9CwvzUevbeplrKpnquoioNbj9RV4S0RWiMgid1u6qroTA5ALeCu074KDZdWkSzGhx5AIRIRxGXHtPxEATLoYxGdPBcaYPstLInjO47bWnKKq04CzgJtEZG7znerM/tLqrNEiskhEskUku6vzieaXOMNLeOlV3Ny4gfFsOtDGJDVN4jNg5Gmw5mlobOhSnMYYEwjtjT46TkQuABJE5Pxmy9U4rYc6pKr73Nc84AVgJnBQRDLcz8gA8to492F3fKMZqampx/SljlRWWkgktZ46kzU3LiOOspp69hVXtX/glMugdB/s/KALURpjTGC090QwFjgHSATObbZMA67v6MIiEiMicU3vgfnAOuBl4Cr3sKuAlzobvFcNJU29ilufq7gtTS2HNrZXTwAw9ssQkeA8FRhjTB/TZmWxqr4EvCQis1T1k05cOx14QUSaPucpVX1DRD4DnhGRa4HddK3i2RNfhZsIPPYhaDI+Iw6fwNqcYs6c0M65YZEw8QJY/TScfT9ExnchWmOM6VltJgIRuc2dtP4yEbn0yP2q+t32LqyqO4DJrWwvBE7vRKydUllbT2xtIYRzzE8E0eGhjEmPY3V7PYybTLnc6Vy2/nmYfnWnYjXGmEBor/noRvc1uycC8Ze80hrSpchZiTv2BkpThyby2tpcVBX36aZ1g6dD2gRY8VdLBMaYPqW9oqFX3Ncnei6c7nfQnZCmPjSG0Ii4Yz5/cmYiTy/fy67CSoanxLR9oAhMvwZe/yHsXwWDpnYhamOM6TleehaPEZGHReQtEXm3aemJ4LrDwTLnieBYehU3N2VoIgCr9xZ1fPDkiyEsGrIf79RnGWNMIHjpR/AssAr4MfDDZkufkFfqjDPkiz+2+oEmo9PiiA4PYc1eD/UEkQnOSKRrn4PqDjqiGWNML+ElEdSr6p9Vdbmqrmha/B5Zdyg9QF3+VjLkECEJnUsEIT5h4uAEVu7x8EQAMOMbUFcBa5/p1OcZY0xP85IIXhGRb4lIhjtyaLKI9I1hqF+6iXM2/5gMOYQkDOn0ZU4cnsy6fSWUVtd1fPCgaTBwklM81F6PZGOM6SW8JIKrcIqCPgZWuEvfaEkUPYCMmu2EUQ9Jwzp9mVkjU2hUWL7jUMcHi8AJ18LBdbDro05/pjHG9BQv8xEMb2XpE7OTbSgOIRR3/J+krE5fZ+rQRCJCfXy8vdDbCZMuhugU+Pj3nf5MY4zpKV6mqmxtHsYSYK07hlCvta44jAlNK4mdfyKIDAthRlYSH28v8HZCWBTMXATv/R/kbYK0cZ3+bGOM8TdPw1ADjwCXu8tfcOYPWCoiX/djbF02eNDgwytdqCMAmD0yhU25ZRwsrfZ2wgnXQWgUfGJPBcaY3s1LIggFxqvqBap6ATABZ+joE/HzhDJdNXviGOeNLwxCw7t0rTPGO72S39l40NsJMQNg6uXw+TNQktOlzzbGGH/ykgiGqGrzv3557rZDgIdmNIEjTb17Z1zT5WuNSY9l2IBo3lrvMREAnHyz8/r+L7v8+cYY4y9eEsF7IvKqiFwlIk3DRr/nDi1d7N/wuih5ONy6FRZ0/Q+xiDB/QjqfbC+kzEszUoDEoU6/glVPQsG2LsdgjDH+4CUR3AT8FZjiLn8DblLVClU91Y+xdY/YNPB5+ZodW3D8QGobGnnzWJ4K5vwAQiNgyc+6JQZjjOluXpqPqqo+p6q3uMtz2u7cjf3XtKFJZA2I5tnsvd5Pik2Dk77lDE+951P/BWeMMZ3kZdC5k0TkMxEpF5FaEWkQkaAcSEdEuGjGEJbtPMTuwgrvJ875PsQPhtdutXmNjTG9jpcykz8AlwJbgSjgOuCP/gyqN7tgWiY+gaeW7fF+UngMzP8p5K6FFTYyqTGmd/FUeK6q24AQVW1Q1ceBBf4Nq/camBDJ2RMzeGrZHu+VxgDHLYThc2HxT6DCYw9lY4zpAV4SQaWIhAOrReReEbnF43n91g1zR1JWU39sTwUicNZ9UFMGS37qv+CMMeYYefmD/nUgBPg2UAEMAS7wZ1C93cTMBE4ZlcKD72/3NiJpk7RxztATK/7qFBMZY0wv4KXV0G5VrVLVUlW9R1W/7xYVBbXbzxpHcVUdf1qy/dhOnPcjiEyE12+3YaqNMb2Cl1ZD54jIKhE5JCKlIlIWrK2Gmjt+cAILpw7msaU72VlwDC2IopLgtB/D7o9gw0v+C9AYYzzyUjT0AM6cBANUNV5V41Q13s9x9Qk/WjCOyFAfP3x2DQ2Nx/Cv++lXQ9oEePcn0FDvt/iMMcYLL4lgL7AuWDuRtSc9PpK7zj2O7N1FPPzBDu8n+kJg3h1QuA3WPee/AI0xxgMvieA24DURuUNEvt+0eP0AEQlxi5ZeddeHi8gyEdkmIv9yWyT1WedPG8zZEwdy35ub+Girx/kKAMadA+kT4f177anAGBNQXhLBz4BKIBKIa7Z4dTOwsdn6L4HfqOoooAhnvoM+S0S498LJjEyN5aanVrLxgMfqE5/PqTg+tB02vOjfII0xph1eEsEgVT1fVe9yWw3do6r3eLm4iGQCX8aZ2AYREeA0oKk85Angq52Iu1eJjQjlsatPICoshCseWeY9GYz9MiSPhE/+aC2IjDEB4yURvCYi8zt5/QdwipYa3fUBQLGqNpWF5ACDWzuxrxmSHM1T159IWIiPC/78Ma+s2d/xST4fnHQj7F8Je5f5P0hjjGmFl0RwI/CGiFQdS/NRETkHyFPVFZ0JTEQWiUi2iGTn5+d35hI9bkRqLC9/+2TGpMfxnadXcd0Tn7Ept4NbNeUyp1/BJ3/omSCNMeYIXjqUxamqT1WjjrH56MnAV0RkF/BPnCKh3wKJIhLqHpMJ7Gvjcx9W1RmqOiM1NdXTl+kN0uIjefabs/jvs8fxyfZCFjzwIVc/vpzsXYdaPyE8xmlOuuk/ULSrJ0M1xhjAj2MGqeodqpqpqlnAJcC7qno5sAS40D2sacazfiUsxMeiuSNZevtp3Dp/DJ/nlHDhg59w5WPLOVBSdfQJMxcBAsv/0uOxGmNMIAaP+xHwfRHZhlNn8GgAYugRidHhfPu00Xz0o1O58+zxZO86xDm/+4gtB8taHpgwGMafC6v+DrXH0EvZGGO6QY8kAlV9T1XPcd/vUNWZqjpKVS9S1ZqeiCGQosNDuX7uCF7+9imE+IQrHllGQfkRX/vEG6C6BNY+G5ggjTFBy8tYQ3/3ss10bFRaLE98YybFVXX86LnPadFZe+gsSD8elj1sTUmNMT3KyxPBcc1XRCQEmO6fcPq/8Rnx3PalsSzelMc7G/MO7xBx6gry1sPupYEL0BgTdNpMBO6QEmXAJLfZaKm7nkc/rODtSVfNzmJEagw/f30j9Q2Nh3dMvMhpSrr84cAFZ4wJOm0mAlX9uarGAfe5zUabmo4OUNU7ejDGficsxMdtXxrLjvwK3life3hHeDRMuxI2vgolOYEL0BgTVLz0I7hDRAaLyGwRmdu09ERw/dn8CQMZkRLDQ+/vaFlXcMK1oI2Q/VjggjPGBBUvlcW/AJYCPwZ+6C63+jmufs/nE66fO4K1+0r4ZHuzyeyTsmDsWc50lnXVgQrPGBNEvFQWLwTGqurZqnquu3zF34EFg4VTB5McE84Tn+xquWPmIqgshPUvBCIsY0yQ8ZIIdgBh/g4kGEWGhXDR9Eze2ZjHwdJm//ofMQ9SxsDyh6wpqTHG77wkgkpgtYg8JCK/a1r8HViwuHTmUBoalWc+23t4Y1NT0v2rYF+nxuwzxhjPvCSCl4GfAB8DK5otphtkpcRw8qgB/POzvS3nPZ58CYTHwbKHAhecMSYoeGk19ATwDPCpqj7RtPg/tOBx2cxh7Cuu4oMtzYbbjoiDqZc79QRlBwMXnDGm3/PSauhcYDXwhrs+RURe9ndgweTMCemkxEbw5LLdLXeccD001sFKy7vGGP/xUjR0NzATKAZQ1dXACD/GFHTCQ318bUYm727KY19xs2GqU0bByNOdPgUNdYEL0BjTr3lJBHWqWnLEtsZWjzSddunMoSjwr+V7Wu448QYoOwAb7SHMGOMfXhLBehG5DAgRkdEi8nucimPTjYYkRzNvTCr//Gwvdc3HHxp1ptPJzCatMcb4iZdE8B2cEUhrgKeBUuB7/gwqWF1x0jDyymp4Z0OzymGfz2lKuucT2Ls8cMEZY/otL62GKlX1TlU9wZ1D+E5VtbEP/GDe2DQGJ0bx5LIjioemXw3RA+D9XwYkLmNM/9beMNQPuK+viMjLRy49F2LwCPEJl84cwkfbCthZ0GzKyvAYmP0d2PYO5FgXDmNM92rviaBpFrL7gV+1shg/+NoJQwj1CU+11pQ0Khne/0VgAjPG9FvtzUewwn19v7Wl50IMLmlxkXzpuIE8uyKHytr6wzsiYmH2t2HrWzbshDGmW7VXNLRWRD5va+nJIIPNNSdnUVxZx7PZR0xOM3OR81Tw7k8DE5gxpl9qr2joHOBcnB7FbwCXu8vrwGv+Dy14zchKZvqwJP7y4Y6WU1lGxMHcH8L2d2Hb4sAFaIzpV9orGtqtqruBM1X1NlVd6y4/Aub3XIjB6Ya5I8gpquK1dbktd5xwLSQOg7fvgsaGwARnjOlXvPQjEBE5udnKbI/nmS44Y3w6I1JjeOj97S2nsgyNgDPugoNr4fNnAhegMabf8PIH/VrgTyKyS0R2A38CvtHRSSISKSLLRWSNiKwXkXvc7cNFZJmIbBORf4lIeNe+Qv/k8wk3zB3B+v2lvNd8VFKACQth0DR49ydQV9X6BYwxxiMvHcpWqOpkYDIwSVWnqOpKD9euAU5zz50CLBCRk4BfAr9R1VFAEU6iMa1YODWTzKQofvXW5pZPBT4fzP8plO6Djx4IXIDGmH7BUxGPiHwZuAG4WUT+n4j8v47OUUe5uxrmLgqcBjznbn8C+OoxRx0kwkN9fO+MMazbV8obR9YVZJ0Mx18IH/0GDu0ITIDGmH7By3wEDwIX44w5JMBFwDAvFxeREBFZDeQBbwPbgWJVbWognwMM7kTcQWPh1MGMSovlV29vaTmDGThPBSFh8MYdgQnOGNMveHkimK2qVwJFqnoPMAsY4+XiqtqgqlOATJw5DcZ5DUxEFolItohk5+fnd3xCPxXiE75/5hi25ZXz/Moj+hXEZ8C822HLG7D59cAEaIzp87wkgqYB5ipFZBBQB2Qcy4eoajGwBCeJJIpIqLsrE9jXxjkPu4PczUhNTT2Wj+t3Fhw3kMlDErn3zc2U19S33HniNyF1HLx+m1UcG2M6xUsieEVEEoH7gJXALuCpjk4SkVT3PEQkCjgT2IiTEC50D7sKeOnYww4uPp9w97kTyC+r4ffvbm25MyQMzr4PivfY6KTGmE5pNxGIiA9YrKrFqvpvnLqBcaraYWUxzlPDEnc4is+At1X1VeBHwPdFZBswAHi0S98gSEwdmsSF0zN57KOd7Mgvb7lz+FyYcgUs/R0cWBOYAI0xfZa0aJbY2gEiq1R1ag/F06oZM2ZodnZ2IEPoFfLKqjnt/veZNiyJJ645ARE5vLOqCP4w06k3uO5dCAlt+0LGmKAgIitUdUZHx3kpGlosIhdIi786JhDS4iL5wfwxfLAlnxdXH1G1EpXkFBEdWAOf/CEwARpj+iQvieAG4FmgRkRKRaRMREr9HJdpw5Wzspg2NJH/fWUDBeU1LXdOOA/GnQPv/RwKtwcmQGNMn+OlZ3GcqvpUNVxV4931+J4IzhwtxCf88oJJVNQ0cM8rG1ruFIGz74eQCHjlZuig2M8YY8Bbh7JprSwjmzUBNT1sdHocN506ilfW7Oet9Uf0OI7PgPn/C7s+hOzHAhOgMaZP8VI09CfgU+Av7vIpTlHRZhGx4agD5MZ5I5mQEc8dz68lv+yIIqJpV8GIefDW/8ChnYEIzxjTh3hJBPuBqao6XVWn4wwgtwOnX8C9/gzOtC081McDl0yhrKae2//9ectB6UTgK38AXwi8dBM0NrZ9IWNM0POSCMao6vqmFVXdgNOXwEY6C7Ax6XH8aME4Fm/K4+nle1vuTBwCC34Ou5fCsgcDE6Axpk/wkgjWi8ifReS/3OVPwAYRicAZbsIE0DWzszh51AB+8uoGdhZUtNw55XIYswAW3wMFW1u/gDEm6HlJBFcD24DvucsOd1sdcKq/AjPe+HzC/RdNJixEuOVfq1vOcSwC5/4WwqLghW9CQ33bFzLGBC0vzUerVPVXqrrQXe5X1UpVbWw234AJoIyEKP7v/Ims3lvMbxcf8S//uIFOk9J92fDx7wIToDGmV7O5h/uJcyYN4qLpmfxhyTY+3lbQcufxF8CEr8KS/4OD61u/gDEmaFki6EfuOe84hqfE8L1/raawea9jEfjyr51hKJ5fBHXVbV/EGBN0PCcCEYn2ZyCm66LDQ/nDpdMorqrj1mfX0Nh8RrOYAXDeH+HgOnjnrsAFaYzpdbz0LJ4tIhuATe76ZLflkOmFJgyK586zx7Nkcz6PLT2iM9mY+XDijU5z0i1vBiZAY0yv4+WJ4DfAl4BCAFVdA8z1Z1Cma66cNYwzJ6Tzyzc2sTanpOXOM+6G9Inw4o1Qltva6caYIOOpaEhVj+itRIMfYjHdRES478JJpMRG8J2nV7ac3jIsEi54BGornSal1uvYmKDnJRHsFZHZgIpImIjcijPlpOnFEqPD+e0lU9lzqJL/eXFdy51p42DB/8GOJTZ3gTHGUyL4JnATMBhnovkp7rrp5WYOT+bm08fwwqp9/HtFTsud069x5i5YfA/sXR6YAI0xvYKXDmUFqnq5qqarapqqXqGqhT0RnOm6b582ihOHJ/M/L61je/O5jkXgvD9AQia2p6r8AAAaDElEQVQ8cxWU5wcuSGNMQHlpNfS7VpafiMh5PRGg6ZoQn/DbS6YSGRbCt/6xksraZvUFUUnwtb9D1SF47hobgsKYIOWlaCgSpzhoq7tMAjKBa0XkAT/GZrrJwIRIfnvJFLbklXHnC+taDlmdMcnpbLbrQ3j3J4EL0hgTMF5mGZsEnKyqDQAi8mfgQ+AUYK0fYzPdaM7oVG45Ywy/fnsL04Yl8fWThh3eOfVyyFkOSx+AzBkw/tzABWqM6XFengiSgNhm6zFAspsYalo/xfRG3z51FPPGpvK/r6xn9d7iljvPuhcGT4fnb4ADnwcmQGNMQHhJBPcCq0XkcRH5K7AKuE9EYoB3/Bmc6V4+n/DAxVNIi4vkW/9YwaGK2sM7QyPgkqeceoOnL7HOZsYEES+thh4FZgMvAi8Ap6jqI6paoao/bOs8ERkiIktEZIOIrBeRm93tySLytohsdV+TuuvLmI4lRofz4BXTKSiv5TtPr6Su+fwFcQPhsn9CdYmTDGorAxeoMabHeB10rho4ABQBo0TEyxAT9cAPVHUCcBJwk4hMAG4HFqvqaGCxu2560MTMBH628HiWbivknlfWt6w8HjgRLngUDqyBFxZZz2NjgoCX5qPXAR8AbwL3uK93d3Seqh5Q1ZXu+zKc3siDgfOAJ9zDngC+2pnATddcNGMIN8wdwT8+3cMTH+9quXPsAvjS/8HGV+D1H0LzRGGM6Xe8PBHcDJwA7FbVU4GpQHH7p7QkIlnuecuAdFU94O7KBdKP5Vqm+9y2YBxnjE/nf1/dwHub81ruPOlGOPl78NkjsORngQnQGNMjvCSCalWtBhCRCFXdBIz1+gEiEgv8G/ieqpY236dOmUSr/9wUkUUiki0i2fn51uvVH5zOZlMYOzCe7zy1io0HSlsecMbdMO0q+OA++NjGJDKmv/KSCHJEJBGnsvhtEXkJ2O3l4iIShpMEnlTV593NB0Ukw92fAeS1dq6qPqyqM1R1RmpqqpePM50QExHKI1fNICYilCsfW86ewmYVxCJwzm9gwnnw1p2Q/VjgAjXG+I2XVkMLVbVYVe8G/gd4FA/l+iIi7rEbVfXXzXa9DFzlvr8KeOlYgzbda3BiFH+/diZ1DY1c8egy8sqaTWXpC4Hz/wKjvwSv3gLL/xK4QI0xftFuIhCREBHZ1LSuqu+r6suqWtveea6Tga8Dp4nIanc5G/gFcKaIbAXOcNdNgI1Oj+Pxq0+goLyGKx9dTnHlEX0MLv47jP0yvHYrfGIT1BnTn7SbCNzew5tFZOixXlhVP1JVUdVJqjrFXV5T1UJVPV1VR6vqGap6qNPRm241dWgSD319OjvyK7jsL8uO7nB20V9h/FfgzTvgw19ZayJj+gmvQ0ysF5HFIvJy0+LvwExgzBmdysNXTmdbfjmX/eVTCsqbjSISGg4XPgYTL4LF/wuv3waNNlmdMX2dl0Hn/sfvUZheZd7YNB676gSu+9tnXPrwp/zjuhNJj490doaEwcKHnV7IH/8eSvc7U1+GRQU2aGNMp3mpLH4f2AWEue8/A1b6OS4TYKeMTuHxq2eyr7iK8//0MVsPlh3e6fPB/J/Cgl/Apv/A375qE9sY04d56Vl8PfAc8JC7aTBOU1LTz80aOYBnbphFbUMj5//5Yz7ZfsTEdCfdCBc9DgdWw8PzYP+qgMRpjOkaL3UEN+G0ACoFUNWtQJo/gzK9x/GDE3jhW7NJj4/kqseW80z23pYHHLcQvvGm8/6xBfD5Mz0fpDGmS7wkgprmzUVFJJQ2egOb/ikzKZp/f3M2JwxP4rbnPueO59dSU9+sknjQFFj0HgyeAc9fD6/dBnXVbV3OGNPLeEkE74vIfwNRInIm8Czwin/DMr1NQnQYT1wzkxvnjeTp5Xv42oOfsK+46vABsalw5Ytw0rdg+UPwyBmQvyVwARtjPPOSCG4H8nGmpbwBeA34sT+DMr1TaIiPHy0Yx4NXTGd7fgVnPfABL63ed/iAkDBY8HO47Bko2w8P/xes/Jv1NzCmlxPt4EcqIucD/1HVgE1LOWPGDM3Ozg7Ux5tW7Cqo4JZnVrNqTzFfmTyIn5x3PAnRYYcPKMuF5xfBzvdhzFlwzq8hflDgAjYmCInIClWd0dFxXp4IzgW2iMjfReQct47ABLmslBievWEW3z9zDP9Ze4D5D7zPG+sOHJ7kJm4gfP1FZ16DHe/BH0+ClX+3pwNjeiEv/QiuAUbh1A1cCmwXkUf8HZjp/UJDfHz39NG88K3ZJMdE8M1/rOS6J7LJKXJHMPX5YNZNcONSZ+azl78Nf18IhdsDG7gxpoUOi4a+ONAZUnoBcA0wV1VT/BlYc1Y01PvVNzTy+NJd/Pptp4L4W/NGct2cEUSFhzgHNDbCisfg7buhocZJEHNuhYjYwAVtTD/ntWjISx3BWcDFwDzgPeAZ4C1Vre96mN5YIug7cooq+cmrG3hz/UEGxkfyg/ljOH9aJiE+cQ4oOwjv3A1rnoK4QTD/J3D8Bc7cB8aYbtWdieBp4F/A64GqMLZE0Pcs33mIn/1nA2tyShifEc/3zxzDGePTkKY/+HuXO0NaH1gDg6bBGXfBiHmBDNmYfqfbEkErFz4FuFRVb+pscMfKEkHf1NiovLr2APe/uZk9hyqZkBHPd08fxfwJA/H5xBm5dM3TsOTnUJoDI06F0/8fDJ4W6NCN6Re6NRGIyFTgMuAiYCfwvKr+vstRemSJoG+ra2jkpdX7+eOSbewsqGDcwDhunDeSs47PIDzU5/RCzn7MmRu56hCMOwfm3gqDpgY6dGP6tC4nAhEZg9NK6FKgAKd46FZVHdadgXphiaB/qG9o5NXPD/D7d7eyPb+C9PgIrpyVxaUzh5IcEw7VpfDJH+HTP0NNCYw8Deb8AIadbHUIxnRCdySCRuBD4FpV3eZu26GqI7o1Ug8sEfQvjY3K+1vzeeyjnXy4tYCIUB8Lpw7mipOGcfzgBCchZD/qJIWKfBhyIsz+jtMxLcS6sRjjVXckgq8Cl+CMPPoG8E/gEVUd3p2BemGJoP/acrCMx5fu4vmVOdTUNzIhI55LZg7hvMmDSQirh1X/gKW/g5I9kDAETrgOpl0J0cmBDt2YXq87Ww3FAOfhFBGdBvwNeEFV3+qOQL2wRND/lVTV8fLqffzzs72s319KRKiPs44fyMJpmZw8PIHQbW/Csodg14cQGulMl3nCtZAxxYqNjGmDX1oNiUgSToXxxap6ehfiOyaWCILLun0l/Ouzvby4eh9l1fWkxIZz9sQMvjJ5ENMi9uPL/gus+RfUV0H68TD1Cpj4NYgZEOjQjelV/NZ8NBAsEQSn6roG3tuczytr9vPOxoPU1DcyODGKcyZlcNaoKCYVL8a3+h+wfyX4wmDc2TDlChh5qjMSqjFBzhKB6VfKa+p5e0MuL6/ez4dbC6hvVFLjIjhjfBoLB5Uw7dB/CF33DFQWQlQyjD8Xjj8fsuaALyTQ4RsTEJYITL9VUlXHe5vzeGvDQd7blEdFbQPR4SGcPiaRS5K2ML1sCZE73oLacohJgwnnOVNqDjnRWh2ZoBLwRCAijwHnAHmqery7LRmnP0IWsAv4mqoWdXQtSwSmLTX1DXyyvZC3Nxzk7Q0HyStzRkGZlB7O1albmFv7EQP2L0Hqq5wnhdHzYexZMOp0iIgLcPTG+FdvSARzgXLgb80Swb3AIVX9hYjcDiSp6o86upYlAuNFY6OyMbeUD7YU8MGWfLJ3H6KuQUkOq+Xa9G3MD13F8KKlhNYUQ0g4ZJ3i9E0YdTokj7DWR6bfCXgicIPIAl5tlgg2A/NU9YCIZADvqerYjq5jicB0RkVNPZ/uKOSDLfl8sLWAnQUVhNDA3MjtXJywgVm1n5JQtcc5OHGYU8k84lQYPtf6KZh+obcmgmJVTXTfC1DUtN4eSwSmO+wvrmLZzkI+3X6IZTsL2VVYyXA5wBkR6zk7ehPH1awhvKECFR8yaKqTFLJOhsyZNm+C6ZN6fSJw14tUNamNcxcBiwCGDh06fffu3X6L0wSn3JJqJzHsKOTTHYfYW1DCZNnOf4Wu48yIDYyt34KPBlRCIGMSMuxkGDrLWazPgukDemsisKIh02sVlNewak8xK/cUsXJ3EdtycpnQsImZvk2cHLaFiWwlTOsAaBgwhpBhs2DwdMicAanjrJmq6XW8JoKebkv3MnAV8Av39aUe/nxj2pQSG8GZE9I5c0I64IyWuin3NFbuKeJvu4tYuzufxOJ1nOjbxAl5m5hR+BxxK58AoCE0msaBkwkbeoKTHAZPh4RMq4A2fYI/Ww09jTO9ZQpwELgLeBFnqsuhwG6c5qOHOrqWPRGY3qK4spZ1+0pZu6+EdTnFFOVsIq10HZN925ni285xvl2E48ziWhORQkP6RCIzJ+PLmAgDJ8GAkfbkYHpMryga6i6WCExvVlJZx/r9JazdV8KGnAKqcz4nrXQ9k33bGS+7GS05hEsDAPW+SKqSxhI6aBJRQybDwImQNh4iEwL8LUx/ZInAmACqqm1ga14Zm3LL2LLvEBX71hFesIEhtduZILuZ4NtNolR8cXxFRCo1iaMJGziemMzj8aWOdeodrFLadIElAmN6oYLyGjbnlrHpQCm5e7fhy1tHVPE2hjTsYZTsY5TsI0Zqvji+MiyJyvhRSNpYYgeNIyJttNP5LSkLQsMD90VMn2CJwJg+QlU5VFHL9vwKtueVkp+zg7qDG4ks3kpK1U5Gyn5GSw4JUvnFOY34KIsYSHV8FjJgBNEDxxAzcDQyYBQkDYPQiAB+I9NbWCIwph+ormtgd2El2/PKyM3dR3XuVvTQDqLLdpFcu49hkstwyT06SYSnUh09iMaEIYQNyCImfQRRKVmQONRpzWSJIihYIjCmn6upbyCnqIo9BRXk5u6nIncLWriDiNJdxFXvZxB5DJYCMigkRA7/zhsRysJSqIwaREP8EEKShxKdOpzY1CGEJAyC+MEQPcCavvYDvbUfgTGmm0SEhjAyNZaRqbEwPh2Y+sW+xkaloKKGfUVVrDlUSsnBPdTk70SL9xBWvo/46v2kVeeRWbyM1L2vESqNLa5dJ2GUh6VSHZVGQ2wGEj+IiORMYlKHEZWcCfEZEJdhEwD1E5YIjOmHfD4hLS6StLhIpg5NAoYBc1ocU1Zdx4GSaj4qLONQ3l7K8/fSWLwPKT9ARNVBYmvySKkuJL1oJRnyDpFS1+L8RoSKkASqwgdQF5VCY3Qqvrh0whIyiEoaSEzyYHzx6c6cENEDwOfruRtgjoklAmOCVFxkGHGRYYxJj4MJg4ATjzqmvKaeg6XVrCypoqgwj4qCvdQV5aCl+wmryCW8uoCYikJSKgpJYQfJUkKU1B51nQZ8VIQkUhkxgNrIVCdpxKYSHp9KVEIaUYlphMe7CSM6GSISLHH0IEsExpg2xUaEEttU/DQqFTjuqGNUlZKqOgrKa1hdWkNRcRFVh/ZRW5JLQ9lBpCKPsKoCImsKiK0oIrk8l1TZzABKj3rKaOIkjgSqwxKoCU+mITIZjU4mJCaFsLgUIhJSiU4cSERcMhKVBFFJTqc867XdKZYIjDFdIiIkRoeTGB3OqLQ4nFFlRrd6rKpSUdtAQVkN68urOVRcQnnRQWpL86kvL4CKAqTqEKE1h4isLSa6uoi4ylKSOEiSlJFMWYuK7yNVSgzVoXHUhMZRH55AQ3gCGpWIRCUSEp1IWOwAwmOTiYpPITIuGYlKtCSCJQJjTA8SEecpIyKUrJQYYAAwot1z6hsaKamqo6iyll0VNZQVF1BVnEddaT51FYdoqChCq4sJrSkhtLaUyPpSompLiakoJYEDJEgFCVS0+fTRpEqiqQ6JpjYklrrQWOrD4mgMj3WmNI2IRyLjCY1OIDQ6nvDoRCJjk4iITSQ0KgEi453jwqL7ZGsrSwTGmF4tNMTHgNgIBsRGAE1PHOM6PK++oZHymnpKqurIq6qnrKKcqpICasoOUVdxiPrKIqgsQqqL8dWUEFJbRmh9OWG1FURVVxCt+cSxm1ipIo7KFj2+29KAz00oMdSExFIXGuOMTBsWi4bFQEQMEh6HLzIGX0QcoVHxhEXFEhYdT0R0PJHRzjbCYyE8BsKieiSxWCIwxvRLoSG+L4qsHAnAYM/n1zc0UlHbQHlNPftq6imrrKG6opia8mLqKkqoqyqhoaoErS6FmlKkugxfXRmhdeWE1ZcTUV9BZE0FEZpPlO4lRqqJpYoYqo9qrtuWBnzkXvYug8dM7fjgLrBEYIwxrQgN8ZEQ5SMhqqmvRNPTyLGrrW+koqae8roG8qrrqKyqpKailNrKUmqryqivLqOhqgytKaexphxqy5Hacnx1FcxISO+279QWSwTGGONn4aE+wkPDcebljQLigYEBjak5a6hrjDFBzhKBMcYEOUsExhgT5CwRGGNMkLNEYIwxQc4SgTHGBDlLBMYYE+QsERhjTJDrE1NVikg+sLuTp6cABd0Yjj9YjN3DYuy63h4fWIzHYpiqpnZ0UJ9IBF0hItle5uwMJIuxe1iMXdfb4wOL0R+saMgYY4KcJQJjjAlywZAIHg50AB5YjN3DYuy63h4fWIzdrt/XERhjjGlfMDwRGGOMaUe/TgQiskBENovINhG5PUAxDBGRJSKyQUTWi8jN7vZkEXlbRLa6r0nudhGR37kxfy4i03ow1hARWSUir7rrw0VkmRvLv0Qk3N0e4a5vc/dn9VB8iSLynIhsEpGNIjKrt91HEbnF/e+8TkSeFpHIQN9HEXlMRPJEZF2zbcd830TkKvf4rSJyVQ/EeJ/73/pzEXlBRBKb7bvDjXGziHyp2Xa//eZbi7HZvh+IiIpIirsekPvYaaraLxcgBNiOMzN2OLAGmBCAODKAae77OGALMAG4F7jd3X478Ev3/dnA64AAJwHLejDW7wNPAa+6688Al7jvHwRudN9/C3jQfX8J8K8eiu8J4Dr3fTiQ2JvuI848iDuBqGb37+pA30dgLjANWNds2zHdNyAZ2OG+Jrnvk/wc43wg1H3/y2YxTnB/zxHAcPd3HuLv33xrMbrbhwBv4vR1Sgnkfez0dwt0AH77YjALeLPZ+h3AHb0grpeAM4HNQIa7LQPY7L5/CLi02fFfHOfnuDKBxcBpwKvu/8AFzX6IX9xP93/6We77UPc48XN8Ce4fWTlie6+5jziJYK/7Iw917+OXesN9BLKO+CN7TPcNuBR4qNn2Fsf5I8Yj9i0EnnTft/gtN93HnvjNtxYj8BwwGdjF4UQQsPvYmaU/Fw01/Sib5HAsM1f7gfvoPxVYBqSr6gF3Vy7QNDFpoOJ+ALgNaJpVewBQrKr1rcTxRYzu/hL3eH8aDuQDj7vFV4+ISAy96D6q6j7gfmAPcADnvqygd93HJsd63wL9e/oGzr+waSeWHo9RRM4D9qnqmiN29ZoYvejPiaBXEZFY4N/A91S1tPk+df5pELDmWyJyDpCnqisCFYMHoTiP5X9W1alABU6Rxhd6wX1MAs7DSVqDgBhgQaDi8SrQ960jInInUA88GehYmhORaOC/gf8X6Fi6qj8ngn04ZXdNMt1tPU5EwnCSwJOq+ry7+aCIZLj7M4A8d3sg4j4Z+IqI7AL+iVM89FsgUURCW4njixjd/QlAoZ9jzAFyVHWZu/4cTmLoTffxDGCnquarah3wPM697U33scmx3reA/J5E5GrgHOByN2H1phhH4iT9Ne5vJxNYKSIDe1GMnvTnRPAZMNptsRGOUxn3ck8HISICPApsVNVfN9v1MtDUYuAqnLqDpu1Xuq0OTgJKmj3C+4Wq3qGqmaqahXOf3lXVy4ElwIVtxNgU+4Xu8X79F6Wq5gJ7RWSsu+l0YAO96D7iFAmdJCLR7n/3phh7zX1s5ljv25vAfBFJcp985rvb/EZEFuAUV35FVSuPiP0St9XVcGA0sJwe/s2r6lpVTVPVLPe3k4PTMCSXXnQfPQl0JYU/F5ya+y04LQnuDFAMp+A8dn8OrHaXs3HKghcDW4F3gGT3eAH+6Ma8FpjRw/HO43CroRE4P7BtwLNAhLs90l3f5u4f0UOxTQGy3Xv5Ik6ri151H4F7gE3AOuDvOC1bAnofgadx6izqcP5YXduZ+4ZTTr/NXa7pgRi34ZSnN/1uHmx2/J1ujJuBs5pt99tvvrUYj9i/i8OVxQG5j51drGexMcYEuf5cNGSMMcYDSwTGGBPkLBEYY0yQs0RgjDFBzhKBMcYEOUsEJqiISLn7miUil3Xztf/7iPWPu/P6xviLJQITrLKAY0oEzXoHt6VFIlDV2ccYkzEBYYnABKtfAHNEZLU4cwiEuOPff+aOH38DgIjME5EPReRlnF7CiMiLIrJCnHkHFrnbfgFEudd70t3W9PQh7rXXichaEbm42bXfk8NzLDzp9kg2pkd19C8cY/qr24FbVfUcAPcPeomqniAiEcBSEXnLPXYacLyq7nTXv6Gqh0QkCvhMRP6tqreLyLdVdUorn3U+Tq/oyUCKe84H7r6pwHHAfmApzthEH3X/1zWmbfZEYIxjPs7YMKtxhgkfgDOGDcDyZkkA4Lsisgb4FGcAsdG07xTgaVVtUNWDwPvACc2unaOqjTjDKGR1y7cx5hjYE4ExDgG+o6otBgATkXk4Q143Xz8DZ0KZShF5D2fMoM6qafa+AftNmgCwJwITrMpwpg5t8iZwoztkOCIyxp345kgJQJGbBMbhTEPYpK7p/CN8CFzs1kOk4kx5uLxbvoUx3cD+9WGC1edAg1vE81ec+ReycMaTF5zZ0L7aynlvAN8UkY04I19+2mzfw8DnIrJSnWG8m7yAM43iGpyRaG9T1Vw3kRgTcDb6qDHGBDkrGjLGmCBnicAYY4KcJQJjjAlylgiMMSbIWSIwxpggZ4nAGGOCnCUCY4wJcpYIjDEmyP1/eqlL7AkX89UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.subplot(1,1,1)\n",
    "plt.plot(abs(clf.grad_w1_[10:]), label='w1')\n",
    "plt.plot(abs(clf.grad_w2_[10:]), label='w2')\n",
    "plt.legend()\n",
    "plt.ylabel('Average gradient magnitude')\n",
    "plt.xlabel('Iteration')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 5:** In the plot above, you should see that one layer always has a larger gradient magnitude than the other layer. \n",
    "\n",
    "- **Part A:** Which layer always has the larger magnitude? Why do you think this particular layer always has a larger magnitude? \n",
    "- **Part B:** If one layer has a larger gradient, does this also mean that the weights for that layer are training more quickly (that is, with fewer iterations)? Is that desireable for training the neural network? Explain your reasoning. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Part A: The second layer, W2, has the larger magnitude. This layer always has the larger magnitude because it builds off the weights of the first layer\n",
    "- Part B: Yes, the weights in the larger gradient layer will train more quickly. This is certainly desireable because it means quicker computation and execution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Final Exercise:** In the block of code below, come up with an adaptive scheme to mitigate the effect of unequal magnitude gradients in each layer. Save the magnitude of the resulting update for each layer. \n",
    "\n",
    "Explain your scheme below (that is, document what your adaptive algorithm does) and then implement your strategy below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For preventing unequal gradients, we decided to relatively normalize the gradients through <strong>eta</strong>. This was done by maintaining an inverse-proportional relationship. When the gradients are large, <strong>eta</strong> is small and vice versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9910962715637173\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XeYVPXVwPHvma2gSBOUmkWDDUVFNKBCLLFjiZpEJbZoMJbYX8USeycq9opikKixAYJgoRfpgvQOsvTeli0zc94/7t3Z2d0pd8vMzrLn8zz77Nw7d+49OzD3zK+LqmKMMcaU5avpAIwxxqQmSxDGGGMisgRhjDEmIksQxhhjIrIEYYwxJiJLEMYYYyKyBGGMMSYiSxDGGGMisgRhjDEmovSaDqAqDjzwQM3JyanpMIwxplaZMWPGZlVtFu+4Wp0gcnJymD59ek2HYYwxtYqIrPJynFUxGWOMicgShDHGmIgsQRhjjImoVrdBGGNMIhQVFZGbm0t+fn5Nh1Il2dnZtG7dmoyMjEq93hKEMcaUkZubS4MGDcjJyUFEajqcSlFVtmzZQm5uLu3atavUOayKyRhjysjPz6dp06a1NjkAiAhNmzatUinIEoQxxkRQm5NDsar+DXUzQQT8MHMABAM1HYkxxqSsupkgpr0HQ26D6R/UdCTGGFMpDz30EG3atGH//fdP2DXqZoLI2+L83rutZuMwxphKuvDCC5k6dWpCr1E3E4QxxqS4Pn368OqrrwJw1113ccYZZwAwatQoevbsSZcuXWjRokVCY7BursYYE8Pj38xj/tqd1XrOo1oewKMXdoh5TLdu3XjxxRe5/fbbmT59OgUFBRQVFTF+/Hi6d+9erfFEYyUIY4xJQSeccAIzZsxg586dZGVl0bVrV6ZPn8748ePp1q1bUmKwEoQxxsQQ75t+omRkZNCuXTv69+/PySefTMeOHRk9ejRLly7lyCOPTEoMVoIwxpgU1a1bN/7973/TvXt3unXrxttvv83xxx+ftDEaliCMMSZFdevWjXXr1tG1a1cOOuggsrOzQ9VL9913H61btyYvL4/WrVvz2GOPVfv1rYrJGGNS1JlnnklRUVFoe/HixaHHL7zwAi+88EJCr5/QEoSI3CUi80Rkroh8IiLZItJORKaIyFIR+UxEMt1js9ztpe7zOYmMzRhjTGwJSxAi0gq4HeisqkcDacAVwPPAy6r6W2AbcIP7khuAbe7+l93jEks14ZcwxpjaKtFtEOlAPRFJB+oD64AzgC/c5z8CLnEfX+xu4z5/puwLs2UZY0wtlbAEoaprgH8Dv+Ikhh3ADGC7qvrdw3KBVu7jVsBq97V+9/imiYoPAMs/xhgTVSKrmBrjlAraAS2B/YBzq+G8vURkuohM37RpU1VPZ4wxJopEVjH9AVihqptUtQj4CjgFaORWOQG0Bta4j9cAbQDc5xsCW8qeVFXfVdXOqtq5WbNmVYvQ2iCMMSaqRCaIX4EuIlLfbUs4E5gPjAYud4+5FhjsPh7ibuM+P0rV7uDGGFNWXl4eF1xwAUcccQQdOnSgd+/eCblOItsgpuA0Ns8E5rjXehe4H7hbRJbitDH0c1/SD2jq7r8bSMxfHM7aIIwxtdS9997LwoUL+fnnn5k4cSLDhw+v9mskdKCcqj4KPFpm93LgpAjH5gN/SmQ85VgBxRiTovr06UNWVha33347d911F7Nnz2bUqFGMGjWKfv36MXDgQAAyMzPp1KkTubm51R6DjaQ2xphYhveG9XOq95wHHwPnPRfzEK/TfW/fvp1vvvmGO+64o3pjpK7PxWRVTMaYFOVlum+/38+VV17J7bffziGHHFLtMdTtEoRVMRlj4onzTT9RvEz33atXL9q3b8+dd96ZkBjqaAnCSg7GmNQXa7rvhx9+mB07dtC3b9+EXb+OJggrORhjUl+06b5zc3N5+umnmT9/Pp06deK4447j/fffr/br1+0qJmuDMMaksFjTfSdjmFgdLUG4rA3CGGOiqqMJwkoOxhgTTx1NEFZyMMbEti/M9FPVv6GOJghjjIkuOzubLVu21Ookoaps2bKF7OzsSp+jjjZSWxWTMSa61q1bk5ubS21fUiA7O5vWrVtX+vVxE4SI1AfuAdqq6t9FpD1wuKoOrfRVa1zt/VZgjEm84kFqdZ2XKqYPgQKgq7u9BngqYRElk3VzNcaYqLwkiENV9QWgCEBV89hX6mhqcf2iMcYkmpcEUSgi9XDrZUTkUJwSRS22b+Q3Y4xJJC+N1I8CI4A2IjIQZ9nQ6xIZVOJZycEYY+KJmyBU9QcRmQl0wfnqfYeqbk54ZMYYY2pU1AQhIp3K7Frn/m4rIm1VdWbiwko0q2Iyxph4YpUgXnR/ZwOdgdk4d9aOwHRKejUZY4zZB0VtpFbV01X1dJySQydV7ayqJwDH43R1rcWsDcIYY+Lx0ovpcFUNLciqqnOBIxMXkjHGmFTgpRfTLyLyPvCxu90T+CVxISWDtUEYY0w8XhLE9cDNwB3u9jjgrYRFZIwxJiV46eaaD7zs/hhjjKkjvEzWt4IIrbqqekhCIjLGGJMSvFQxdQ57nA38CWiSmHCMMcakiri9mFR1S9jPGlXtC1yQhNgSyLq5GmNMPF6qmMJHVPtwShR1dKEhY4ypO7zc6F8Me+wHVgB/Tkw4yWLdXI0xJh4vCeIGVV0evkNEbKklY4zZx3kZSf2Fx33GGGP2IbFmcz0C6AA0FJFLw546AKc3kzHGmH1YrCqmw4EeQCPgwrD9u4C/JzIoY4wxNS9qglDVwcBgEemqqj8lMSZjjDEpIFYV032q+gJwlYhcWfZ5Vb09oZEZY4ypUbGqmBa4v6cnIxBjjDGpJVYV0zfu74+SF06SjXkG5g+GWybVdCTGGJNyvIykPgy4F8gJP15Vz0hcWEm0cV5NR2CMMSnJy0C5z4G3gfeBQEVOLiKN3NcdjTMB0t+ARcBnOAlnJfBnVd0mIgK8ApwP5AHXqerMilzPGGNM9fEyUM6vqm+p6lRVnVH84/H8rwAjVPUI4Ficdo3ewEhVbQ+MdLcBzgPauz+9sEWJjDGmRnlJEN+IyC0i0kJEmhT/xHuRiDQEugP9AFS1UFW3AxcDxe0aHwGXuI8vBv6jjslAIxFpUdE/yBhjTPXwUsV0rfv7/8L2KRBvwaB2wCbgQxE5FpiBs2zpQaq6zj1mPXCQ+7gVsDrs9bnuvnVh+xCRXjglDNq2beshfGOMMZXhZT2IdhF+vKwmlw50At5S1eOBPZRUJxWfW6ng4gyq+q6qdlbVzs2aNavIS40xxlSAl15Ml0bYvQOYo6obY7w0F8hV1Snu9hc4CWKDiLRQ1XVuFVLxOdYAbcJe39rdZ4wxpgZ4aYO4AacnUk/35z3gfmCiiFwd7UWquh5YLSKHu7vOBOYDQyiptroWGOw+HgJcI44uwI6wqihjjDFJ5qUNIh04UlU3AIjIQcB/gN8B44ABMV77T2CgiGQCy4HrcZLS/0TkBmAVJYsPfYvTxXUpTjfX6yv81xhjjKk2XhJEm+Lk4Nro7tsqIkWxXqiqs3CWKC3rzAjHKnCrh3iMMcYkgZcEMUZEhuIMmAO4zN23H7A9YZEZY4ypUV4SxK04SeEUd/s/wJfuN/7TExVYQomtSW2MMfHETRBuIviCfWmZUa1Qz1pjjKmT4vZiEpEuIjJNRHaLSKGIBERkZzKCM8YYU3O8dHN9HbgSWALUA24E3khkUMYYY2qelwSBqi4F0lQ1oKofAucmNixjjDE1zUsjdZ47jmGWiLyAMzeSp8RijDGm9vJyo78aSANuw5lPqQ1OryZjjDH7MC+9mFa5D/cCjyc2nCSxbq7GGBOXl15MPUTkZxHZKiI7RWRXre/FZN1cjTEmLi9tEH2BS3Fmb7U7qzHG1BFe2iBWA3MtORhjTN3ipQRxH/CtiIwFCop3qupLCYsq0awNwhhj4vKSIJ4GdgPZQGZiw0kSKwwZY0xcXhJES1U9OuGRGGOMSSle2iC+FZGzEx5JMlkVkzHGxOUlQdwMjBCRvdbN1Rhj6g4vA+UaJCMQY4wxqcXmVDLGGBNR3UwQ1gZhjDFx1c0EYW0QxhgTl5e5mAZ42WeMMWbf4qUE0SF8Q0TSgBMSE06SWBWTMcbEFTVBiMgDIrIL6Oh2b93pbm8EBictQmOMMTUiaoJQ1WfdLq59VPUA96eBqjZV1QeSGGP1szYIY4yJy8s4iAdEpBXwm/DjVXVcIgNLKlWrdjLGmDLiJggReQ64ApgPBNzdCtTeBFE2GViCMMaYcrxM1vdH4HBVLYh7ZG1hVUzGGBOXl15My4GMRAdSsyxhGGNMWV5KEHnALBEZSekFg25PWFSJFqmKyRhjTCleEsQQ92cfZgnCGGPK8tKL6SMRqQe0VdVFSYgp+awEYYwx5XiZauNCYBYwwt0+TkT2sRKFJQhjjCnLSyP1Y8BJwHYAVZ0FHJLAmJLPShDGGFOOlwRRpKo7yuwLJiIYY4wxqcNLI/U8EbkKSBOR9sDtwKTEhpVsVoIwxpiyvJQg/okzo2sB8AmwE7gzkUElnVUxGWNMOXEThKrmqepDqnqiqnZ2H+d7vYCIpInIzyIy1N1uJyJTRGSpiHwmIpnu/ix3e6n7fE5l/6iKswRhjDFlxZruu6/7+xsRGVL2pwLXuANYELb9PPCyqv4W2Abc4O6/Adjm7n/ZPS45rARhjDHlxGqDKF417t+VPbmItAYuAJ4G7hYRAc4ArnIP+Qinl9RbwMXuY4AvgNdFRFSTcfe2BGGMMWVFTRCqOsP9PbYK5+8L3Ac0cLebAttV1e9u5wKt3MetgNXuNf0issM9fnMVru+NlSCMMaacqAlCROYQ46u1qnaMdWIR6QFsVNUZInJapSMsf95eQC+Atm3bVtdpjTHGlBGriqmH+/tW93dxldNf8VYncwpwkYicD2QDBwCvAI1EJN0tRbQG1rjHrwHaALkikg40BLaUPamqvgu8C9C5c+dq+uofdppp78PBx0KbE6vn1MYYU0vFWnJ0laquAs5S1ftUdY77cz9wdrwTq+oDqtpaVXNwFhwapao9gdHA5e5h11KyvvUQdxv3+VHJaX+AvYWBko1h90C/PyTjssYYk9K8jIMQETklbONkj6+L5n6cBuulOG0M/dz9/YCm7v67gd5VuEaF/OGl0cm6lDHG1BpeRlLfAHwgIg0Bwema+reKXERVxwBj3MfLceZ2KntMPvCnipy3uuzO98c/yBhj6hgv033PAI51EwQR5mWq9cS6uRpjTDleShCIyAU4021ki7sam6o+kcC4ksoShDHGlOdlPYi3gb/gzMkkONVAv0lwXDXu6WHzazoEY4ypUV4am09W1WtwpsF4HOgKHJbYsJJLIux7b/yKpMdhjDGpxEuCKJ6YL09EWgJFQIvEhZR8VsVkjDHleWmD+EZEGgF9gJk4o8reS2hUSRapBGGMMXVdzBKEiPiAkaq6XVW/xGl7OEJVH0lKdEkSrQTx5ND5JGmsnjHGpJyYCUJVg8AbYdsF+2I312gzh/SbsIJ2D3zL4g27khyPMcbUPC9tECNF5DIp7t+6D4r3hw2etSbOEcYYs+/xkiBuAj4HCkRkp4jsEpGdCY4rqeI1UlstkzGmLvKy5GgDVfWpaqaqHuBuH5CM4GrS1WnfszL7KjKwaTiMMXVT3F5MItIpwu4dwKqwhX9qtUgliLvTvwBgP/YmOxxjjEkJXrq5vgl0Aua428cAc4GGInKzqn6fqOCSZZ9tXDHGmCrw0gaxFjheVU9Q1ROA44DlwFnAC4kMLlkilSBs8Jwxpq7zkiAOU9V5xRuqOh9nLMTyxIWVaFJmq3wyaCR7AFCEfbf/ljHGROelimmeiLwFfOpu/wWYLyJZONNu1EKlE0KsBGAlCWNMXeWlBHEdsBS40/1Z7u4rAk5PVGDJZUnAGGPK8rJg0F7gRfenrN3VHlEKsnEQxpi6qCprS9di8dsgjDGmrqujCaJMG0QNRWGMManMc4IQkfqJDKQmJaoEMWnpZu749GebEdYYUyt5WXL0ZBGZDyx0t48VkTcTHllCea9iejzjIzIDeyp1laven8LgWWutDcMYUyt5KUG8DJwDbAFQ1dlA90QGlXjlq5hm/rot4pEXp03ipLUfV+PVjDGmdvBUxaSqq8vsCiQglhojKJe+OSn68xqs0vmtiskYUxt5SRCrReRkQEUkQ0TuBRYkOK4Eq2izdMkNfuuewgrf8C09GGNqIy8J4h/ArUArYA3OXEy3JjKoZPPaSL1s0246PfkDAyavqtD5rQBhjKmNvKwHsVlVe6rqQaraXFX/qqpbkhFc4kS5Y8e5ky/f5DRWj120qYJXswxhjKl9vKwH8WqE3TuA6ao6uPpDSr5QCSJqgvA+d1PEV1t+MMbUQl6qmLJxqpWWuD8dgdbADSLSN4GxJVDZbq7F7E5ujDHFvMzm2hE4RVUDAO7MruOBUylZRKhWi9cGUZxAKtsbKWhFCGNMLeSlBNEY2D9sez+giZswChISVZLFr2Iq/4qKsPxgjKmNvCSIF4BZIvKhiPQHfgb6iMh+wI+JDC5Znsjo7z6KfSev7H3e8oMxpjbyMt13PxH5FjjJ3fWgqq51H/9fwiJLot/5FjoPon3V16o2UluKMMbUPl4n68sH1gHbgN+KyD411Ub8/e6zlbzPW3owxtRGXrq53gjcgdNzaRbQBfgJOCOxoaWuCo/DtgxhjKmFvJQg7gBOBFap6unA8cD2hEaVcFFu8Ym6k1uCMMbUQl4SRL6q5gOISJaqLgQOT2xYiVUYiDb5Xrw7eeXu9DaS2hhTG3lJELki0ggYBPwgIoOBuJMRiUgbERktIvNFZJ6I3OHubyIiP4jIEvd3Y3e/iMirIrJURH4RkU5V+cNimbEq8tTeFRlJvXLzHj72OCeTVTEZY2ojL3Mx/VFVt6vqY8C/gH7AJR7O7QfuUdWjcNotbhWRo4DewEhVbQ+MdLcBzgPauz+9gLcq+Ld4VpleRflFAe7532wAduwt4rR/j+HhQXPxRy2NlCgeKLdldwF/eGksKzdXbgEiY4xJppgJQkTSRGRh8baqjlXVIapaGO/EqrpOVWe6j3fhTBHeCrgY+Mg97CNKks3FwH/UMRloJCItKvwXVUnkxCEo381bT15hEXekfcnC5avivCLyWb+du56lG3fz3vjlVQ/VGGMSLGaCcEdLLxKRtlW5iIjk4DRuTwEOUtV17lPrgYPcx62A8IWJct191S5qL6QoJYvdBX4AuvvmcFfGlzyV8WHoOS/TaIQOsbomY0wt4mUupsbAPBGZCoTqRlT1Ii8XEJH9gS+BO1V1p4SNMlNVFZEK3TVFpBdOFRRt21Ypb0UQOZSlG3dzMJCOkyiyw2YY8XLPL9tIXdGBdsYYUxO8JIh/VfbkIpKBkxwGqupX7u4NItJCVde5VUgb3f1rgDZhL2/t7itFVd8F3gXo3LlzUr6Sx7qfe5qIzwoOxphayEsj9VhgJZDhPp4GzIz3OnGKCv2ABar6UthTQ4Br3cfXAoPD9l/j9mbqAuwIq4qqXhG+wgvBmMWBAzdN5rb08stfBIIeqpgqFl1Mhz00nOs+nFqNZzTGmMi8jKT+O06VThPgUJx2gbeBM+O89BTgamCOiMxy9z0IPAf8T0RuwOku+2f3uW+B84GlQB5wfYX+kio6VpYDkWcQEZRTJv4tYjr1kB9Ceac6EkVhIMiYCq5oZ4wxleGliulWnIn6pgCo6hIRaR7vRao6gei1M+WSizp9T2tsrWtB2ZVfRIMKvi5al9mNO/NLjrE6JmNMLeRloFxBeLdWEUlnH61Vf+mHRRH3a4xWiEhVTCs37+GkZ0aGtr2UMowxJtV4SRBjReRBoJ6InAV8DnyT2LCS7yjfKrJ2l2sTB2KvOBfp5r9m+95S28WljOLChlR4uj9jjEk+L1VMvYEbcJYXvQmnreD9RAZVE57O+ABWfeDp2PASxcZd+TRrkFXq+bK3fxv+YIypjbyUIC7BGeH8J1W9XFXf0zq2Ak7ZEsSBspMX0t8hi0JuGVi+Q5fEGehg4yCMMbWBlwRxIbBYRAaISA+3DaJWq+r9+XjfUv6cPpaL0iaxcWf5Zbl9ZS5QnE4XrNtZxSsbY0zyeBkHcT3wW5y2hyuBZSKyz1UxVVakqcN9ZTJEcS+mT6etLnesMcakKk9LjqpqETAc+BSYgbfZXFNYxWrIYpU4IvViKluCGL2weLC4crrvZ4b8bInCGJP64iYIETlPRPoDS4DLcBqoD05wXAlV0SqmDr6VEfdH7/5aev9j38wH4GzfdD7M7MMVRYNYsmFXBaOIIG8r7K3li/sZY1KWl/aEa4DPgJtUtXyFex3QxbegQsdHa8NvLs7NvLVsYme+v8px8UI75/djO6p+LmOMKSNuglDVK8O3ReRU4EpVrbFRz6nOy/xMxhiT6jy1QYjI8SLSR0RWAk8CC+O8JMVV/w180fpd5BU6pYJAnF7Ale1FFbTEY4xJoqglCBE5DKfX0pXAZpxqJlHV05MUW8LEGhldGcs27eacvuP4/WHNOKldE3wJGugQL/HUJv5AkPQ0T99PjDE1JFYV00JgPNBDVZcCiMhdSYmqllB1EsGZL44FYPLyLYxdHHmm1fyiQMnrKnm9faXqasmGXZz18jje6tmJ845J8qqyxhjPYn2FuxRYB4wWkfdE5EyqPsYsJVRXCeIwX+nuqgX+8mMiilXHzd3T4kS1wJw1TqP6d/PW13AkxphYoiYIVR2kqlcARwCjgTuB5iLyloicnawAU9lN6cNiPt+YnXTxOV1cq2MU9b5SgrCpRoypHbyMpN6jqv9V1QtxlgH9Gbg/4ZElULLuTwMzn+XTzKcQglz7QelV4CpzkwxGL6AYY0y1q1AroapuU9V3VTXeanIGZwpxgDSCpaqf/po+MtpLYgpvpM7LnRPxGFVl/traMefTvlEeMmbfVSe7kVRnLyYh/tf6NIL4g8pFaZNC+y59cxI39J9WoWtNX7k19DiYOyPiMYNmreH8V8endP2+rYdhTO1QJxNEdX53zSAQ9xifm0RO8pVesW5kaI4mmLBkM1/NzI15nl4DSpJCMMqkugvXO1N4LN+0J25cxhgTSx1NENUng/hTZvhiJKRlm3YD8Nd+U7j7f7M9Xzfoiz0IPt462HsLA+zML/J8vbICQWXi0s2Vfj3YQkrGpLq6mSCq8cY0+6HT6H5Ys5jHpMWohjrzxbE8N7xkYLo/EGTUwg10fOw7cnpH7yUViFKCKK6+iXfzPfPFMXR87PvYB8Xwzrhl9Hx/Smjcx39+Wum5Wqu4gX5xdUxYaIxJmLqZIKoxQ6TnbeCoFgfQnG1RSxO+iAmiJIa3xy4LPX5q2AL+1n96ucn8wgfaAQTIiBtbflGARwfPZUde+ZLC2h35cV8fS3EV1gb3PI8MnsdNAyK3i0RTXB1mjElNliCq6q2T+VvXVkzNvpUXMt7hTye05qHzj6R98/1Dh0QqQXSU5RFPN25J6ZHYu9xqoD+8NLbU/kItaejdsrtkkt2lG0tuul//vIaPflrFiz+Ubvswxhgv6mSCqO667+brxwFwbto0+vzpWP7e/RCWbNwdej5SCSI9SuN22cbluz5z2iVyt+0ttf/hr38JPe7y1Ag+m/YrAD8uKGn4LnJXu4s1AltVWbdjL5vdJPPFjFwueWNi1OONMXVHnUwQZTPEvOBvyh0yJnAsnfPfYkTO/8U/36dXAZCdmQE/fwyblzD0n6eGno7VBhHPjws2MGPVtnL7w8/ZUZZx/5dzSq1DsW7HXt4d55RS0kTYXeDnkcFzS7V3AHw2bTVdnx1F56d+BODez2cza/X2qGtaFCsuv9z35S/MWLU15rGVsWj9LvYUVMOaGWbfNuk1mPNFTUexz6qbCSKsimmTNmSLHlDuiAz8bKYhS1t6X11VCvfA4Fvh9c4c3aphaH9TqdrAtcvemlRuX3iCKHLnXLzho+mhfR9P/jVU6li5JY+jH/2O//y0qlR7B8DqbXkRrzluSeweSp/PKOmS2/vLyIP2wk1attnzKnqBoHJO33H0GjCdAn/8bsSmDvv+YfjyhpqOYp9VJxNEfnoDAN7zn0/3gpcjHpMhzrfXoC+zchfZtDj0cFjWQzx+anbEw06QRbRgi+fTDs+8n6fT+5VKEH7SABgVNq4i3LwYI6vfGF2SMI7814jQ42s/mMrgWWt4dviCcg3kZYVPb/7zryWlnUBQmbLc+duuem8KZ73sVMVJnHlGCt1R5xOXbuHwh0ewbsfemMfXlEBQQyWt7XmFKT04saxCf5CNO6vWUcHs++pkgvBLFgB9/Zexl+yII6szi3skhd3MOuW/DZe+5+0ib5xYavPa6ZeWO+TpPx7Nl1mPMyar9Czq2RTQhPI39avSRnKkbzU900eWShDxxiVv3l16pdhJUcYv7C2TCO74dBbvjF3Oe+MiN6iHrh8WwB/fnMTqrU6p5N1xy/nLu5OZEKc0UlbZUkPZ9pdU8OP8DRz64Ldc7LbX/OPjGdw0YEa59zpV3fXZLE56ZmSlF6EaMXcdOb2HsWZ76v3bxPRYQxj5ZE1HUWvUyQRRXMXkE/hN0/oRE8S7/guA0jffrRwAHf9c+sBznq1UBF9lPUaXzBUAZLmlldmPns3+Wel8nvk4M7P/Ue4196d/Enoc3vCd7mGwXrir3p9SoeNf/GExOb2HRW0TKNtdtdsLo/lh/gaWug31a8uUAMLbNyK1dRSWmTY9o4ILC23PK4zbhrJ5dwHfV+Eb/3+nOp0Cfsl1pi5fsdnpXLC3MMDcNTW7RnheoT9udd6wOesA8FcyQXw+3aliXFBL5v0qZfy/azqCWqNOJoh6mU6VzDOXdmTs/53Odwf3YmXwoNDzlx80nH/d/yBHtTiAv5zYBg46OvrJut4CJ/WqVByHDr449Dg7w0fDehmMu+90jvGtBGBw5sN8lPFcxNc2k+2hx9F6RFW3lVu8T9/x9/9MZ9Zqt7op7B6kqqWmLf9w4kqu+WBqqVJD2XU1fG6W3lsYoO+PiykKBPlx/gY6PflDueqvGau2cdwTPzBwyq8x4+v81I/0GjCDvYWR37s9Bf6YI813l0mWxfnoqWGxWbZQAAAacUlEQVTz6fHaBGas2hpKUjNWbeWhr+fETVqlz6fc+/lsJi/3Xv1Y7OaPZ3LWy+PwB+J3jijycEwkI6NUZ6Y0mw65wupkgij+o0/97YEA9DjvQi5Jf52P/c4ktT4RWjSsx7d3dKP5Adlw448cl/9O+RM95H4DrWQpIlzHVo0AaLJfSZvHsb7l/D6tpDtrQylpUP5XxsDQ4wypeILoKMs43fdzhV5zwasTQuMyvFjmdtn9z+SVoX1fzVxTakqRJ4bOZ9ziTTw1dEFoX9mqrh/mbwDgtVFL6PvjEv43fTVPDJ3P1j2FrC8z4G+DW68+ZPZa8osC5PQexjtjlzHsl3URSwxlr1WsyzMjY440L15/HGBHXlEoB05b6STFy976iU+mOgtK/fmdyQyc8itfzVwT9XxlFQWUL2bkctV7k0vt/3bOOs5/ZXz5qqHCPChySmo/LXOSSqGHm78/oBz64Lfk9B5WoX/bYolc20O1pI3nzTFLOenpH6t2wvzt8Y9JsGWbdjN7dc3H4VWdTBDFH63i/9y/O6Qpsx45m4f9N5CT/9/ylfoZ9RjzyKX88tjZ5fYDkJYO1wyG0x+Cs56AqwdVOKbXr+joPKjEt5wLO0Se6qO95LIy+yp6p/+33HNDsv7Fh5l9KnytYx77npzewxCCHCdLPb1m7pqSaohoddYDJq+i/8QVXN1vCme7jdnFXhu1lBFz14equH5atoVf3XaOn1dvY9rKrWzeXcCMVdtCJYJd+X7u/dxJRM8OX8it/51JrwEzSt3YATo9+QNTV2ylKBAstajTLvdam3cXhK4bDCr3f/ELb41Zhj9QcoO+5/NZoRJE+Lf2ycu3UBQIhkpM93w+O2L109c/55LTexjb9hSG9hXf3MuWOW4ZOJP563ZSGAjy3bz1JW0ez7SE59sRCGrotflFQVSdObPKJpTi//sbd+WH4tu2J36CWL8jv9SyuqpOgnzo6znl3tvK6PHaeF4Y4XTFPuzh4aExOS+MWMTGXQVRl/T1ZPBtVYotd1te6N/PHwjGrUosCgSZVSYZnPni2FC7VW1QJxNE8ac56rTTEWoCGtXP5IBsd3qLLrfCFZ+UPuCQ0+D398Epd8Chp5d+rvlRcUNq3rclzP0Kvnuw3HNjMu/ig4sPjPravQX5fHjdibzVsxMr72jDyuyrOFTW8Le04QD8I30oh8nqqK+vjOvSvmNQ1iOc4ovfxTXcSz8sRghyddr3ZFO6Qfexb+YzPkqD9gcTVvDRT876GkN/WRfaf9dns/nT2z/R+akfueytSeS5JYIF63aWOq7YNf2mhtpGij05dD4vfr+Y814ZH5o8sVjnp36ky7Mj+WnZFg558Fs+m76a50csLPXtfMLSzaEbdfgUKSJwWp8xpc7X47UJzF2zg40789m2p5AdeUWhwZDvjl/OvwbNZfGGXaFkUZx4duwtKjXb77a8Qm4aMIPrP5zmfstW8O/lb2FTyD8/fCEj5q6n5/tT6N5nNAvXl28vCP97t+Y51xyzaCOPDp5b7liALs+OLLf41dvjljFwyq/cOnBmueN/yd0euklu2V3Aj25pMJq5a3by5hinZ11RQJmdu6NUNWLZa1fI9pJqx9xteVE7a4T7ZvZaBk5x/t+d+vxoerw2AVXlxR8W0+O1CTHnE7v87Z+45I2JpWY3iGXdjr2h0nKqiD0l6D4qdP+PUj4+/OAGsU9w7jPxL/Kn/k6Rf+tyOP5q+KwnrI9zM/3i+oi7c3wbyPku+iqvGQQ4/XC3FPHj2wCMzCo9wO8U31w+SO/DBQXPsIOSaUCG/vNUnvhmPlPdtSaEIBkEKCSDA7LTy80JVay9ODer38hGKvp96CzfDJ7M6M+laRP4Y+ETnl4zdaW3wXjx6uynr9pWbtqSOWt2hNbJXrNtL/XdNqpiu/L9XFmmqid8xHt+UeRS3+BZayPu7/HahIj733JvjGMXbwqVkIod+3jp6q7iqrUVm/dw3/9mUlwWDP+G/dn01WxyE1futr1c8sZEFj55njsQ0jlm8+6SUsslb0ykyX6ZbHWT059PbMMB2Rm0aVIfIOKATRFCpbbRizaxfkc+Wek+GrtVpRe97vzvWPncBVzffxq/5O5g3uPnsF9W+VtPeOeE8MdvjI5fUh29cCOnH9GckQs20L55AxZv2MW/v1/EB9edSMtGbkk/WPJ/+dTnRwOw4tnzY3a7/ucnTjVsz9+VDKb9fHpuqDv3pl0FHHZQ+fvFpGWbQ1VJ2/OK2FsYiJigw130+kQ27SqIG1My1ckEUSzav8HDPY6s+sk7/LH09j8mwBtdYNOCyMdXwfm7v4DHy5c8wj2aMQCAtzP68rT/qtD+o/dO55UrujJg8q+8OWYZX2c+ynG+ZZyc/yqf/vNPdO8zOuL5Au7Yi2Y4H4IbT23H+xNWlDomkyLO901hUPAUwuvt9sepZjre562KKpozfDM51recl/2Xh/YNi1BqqIjdBX66PjuqSueoqo27Srer3DRgerlj/vimM3hyd4GfnbOHQJThOuFjY/KLglzdb0qpUtrDg0qXFLaGVXNd8KqTyFo2zGbEXd0jDth8a8wypoclji7PjqTJfpmMuLMbG3eWlBCdG6TzTXrZpt38tvn+DPtlHd3aN6PfhOX0m7AilIjAqV4q9tqo+P9P7u4/kjsv6sqjQ+aRle7DH3Q6Q5z83ChWPuf0SETLtzcV+INkZ5T+QjBp6WamrdzGgQ1K3tTzXhkfenzflyXtgks37qbn+1O47fTfcu85hzNi7jpaNqrHVe9NKXXM5W//FDHucYs3cWSLA2jWIItNu5z3a9GGXRxxsDN4d/ySTXRo2ZAm+2UyZtFGTj70QDLTk1fxUzcThPvtqWx++F27JkxZsZWs9LRyL6kWv+sFQ90xDwcdA43awKJvS57P3B/OeRq+ucPzKYMqNNtavmgfTde0+QxNe7hkx8eX0qLrbdwXyOd7OYzjfM632EnZt8OuQxl/8iwemixMCB7Dox02cshJF7BocB+uznMaDO/K+JK8jIb0Pv8l+k1YRkdZzjlp05kSPJITfQu5LX0wuwrrMSF4DBf4JjMkeHLU2A6RtfTwTeYNvZSMNF/Ub+b/Sh/AMm3JMxn9AEoliKq6JUI1SSSTs27ls8Dp/Bpszg/BTuwMK5UBNGEn16Z/xyv+ywhWsCa37N/93bzY1Q5eFq0qFq0KL5a1O/KjNthPj1Cq2LqnkJOeLr2s7pGPlAzCvOj1iVx3cg79J60sdcyqLZFH9ZeV03sYQ/95Ki0b1aOJu2941gN0GfIGUL4X3Ksjl3D9KTk0CJZ/n/KLAmSm+ZizZgcL1u2kbdP6EbuBh7dPhXt0yDwAXh+9lPpZabwwovzEmL2/ilxzUBQIcs0HU2nbpD7vXdM5tP/cvuPpf/2JdM5pwtX9ylepDbr1FI5r0yjiOaubVKTrXarp3LmzTp9e/ttVPP1HzuL9H39mxKM92T+75FvC3sIAW/YU0Lpx/RivrqK3T3WqmnqNgYM7whNNSp57dLtT6fxE45J9PV6Gtl3hzS7lTvVtu4c4MetXmi0ckLh4y2p2ZORS0B8eh6nvwc7yq+IVNjuGzE3Oh2RLi+4E251Bs0mPATAocDJ3Ft1GF998BhzwNhn5m+Hmn6Dpb5k28Qf+PDyIlrnBrsy+qtR2Tv5AQGiQnc6usCqxVmzikrSJvBG4mEcv7MDj38wHoH5mGnmFAUDpJEuYqe2JNdywGds5LW0WnwdOA5xquBXZfw09/33gBB4oupFTfHOZrYdyQ9pwrkwbRYYE+GvhA0wIHlPunFkU8kj6AF72X046fnwoa4nezhTLBb7JvJH5KgBXFT7IpGCMbtlVdI5vGlenfc//Ff0DPz420Tj+i3AGfy7Mvp4vA924p+jmaosn/P9CTn75zhiljm3xIGxbWerYf/z+0HLTz1RFffLJopBtlJ++J1yaT7jo2JZ8/XP0nm3nHX0ww+dGHqtz0bEt6fuX4/D5KlcVJSIzVLVz3ONSKUGIyLnAK0Aa8L6qRh4E4Kpsgnhn7DKeHb6Q+U+cQ/3MJBeiggGnLjTdGc2Nv9ApMXS/F5oe6uxThVn/hcz9oIM7F9TKCdDfLSqf+xws+AauG+ac6+NLYcW48tcCOOtJWDAEciu2/nUqymtyJM9s6MJTGR+W2t//qPe5bv6NvLXfzTy/pRv7k8f3Nx5By49LJkzkpnE8/vp7FJLBE384mMnjv+eUoPN/5x3/BTzr78kRBzdg4fpdtJUN7NZ6HOdbSmN282jb2Ryw/idOyX+FG9O/5fr076LGuDrYjDa+knaAl4ou58b0YeyiPvcV9eKnYAeC+LjIN4lXM19naOB39EhzvrHO6/Urb41ZxqHzXmctTUMJqRnbeSyjP8/5r0QRTvXN5dPAGaFrnO+bzJtuggC4s/AW7k7/nNMKXyaIjw6ykt7p/+UB/43kavNyMWfgpznbWENJb7jWsonH0vtzR9Ft7MGpw/cRZHlYYgQ4PL8/BWQCyj3pn/N14FSWa0t8BPmtrGGxtgHgUFkTahe7oOAZ5mlOxPfvzYy+nJ82lUPzB4SqMWOpSIKY2+he9s9fG/PYv6cN5cq0UZxX+Jz7d1XM6My7aOfbEPH8LdnMSb6FDAqGT+QZIICPSF9QWrKZxrKbeZpDGgEy8JNPVuj51648nguPbVnhGKEWJggRSQMWA2cBucA04EpVnR/tNZVNEJ9M/ZV3xy1n+B3dytU/prSvboKN85z2jGhWT4P5g2DJD3DtN9DgIAgUwZMHwn7N4P/c+twvboC5NgtmOJU0Zh18Ocev+yzi87Pb3cCxK/pV6RpfBrpxWdp4pgcPo7NvceknH1hD/sCeZP86JrQroMJ29qep7OL7tO6cHXC+CGzWA5icfiK7mhxNqw2j6B6lN9l7+93E3/eUjOHpkN+PpzP6kYmfN/0X0Vy280GmM7L4pPw3ODp7Ix2K5nFPhvN/4x3/BfTx/4Vb0oZwe/pXpEvp6pt5wd/QwbeKHwIncFaas2DU7OAhHOsrPT3LJQVPMCjrkdD2kfkfcKT8yjzNoecph9N/4jLO8P3M+5kvAvC5vzvracJabcongTMA4ZAD92P55vDBmsrK7J6hrcPyP6LQXUjr1rRBXJk+iu4FfflX+gDOSZuGX9No6ybvUwteYafW4670L5mvv2F04HjqSz7j3GlvXvFfypJgK0YGj6eIdPykc4wsJ59Mlmhr9mMvZ/um83XwVI6SVRSSThZFDMt6CID/+X9PHln8GDyBCcFjyKKQOVk3kCkBjsj/kHyyQiXR/v6zecx/HR1lGaf5ZvNqwJmWZ1lWT9JEOTL/A/pkvEOPtCmlEs+TF3fg6q45Ef/d46mNCaIr8JiqnuNuPwCgqlFHoVU2QdRJhXsgLcsZsxFJMAizBkLrEyFvMzRq6ySUJd87VWHvnwkXvOh0581uBBp0fgbdAluWwo0jYWJfGPl46fM+sAay3Pr5nWuddpaFQ2FQlGqG+gdCj5dg2D2wx/0mnp4NXW6GVZNgdcWmCTH7njUHnsL2zRvoQMU6OSwKtuZwX/kq0Jpwe+Ft3J3+OTk+p31pvTbmYClpz/nEfzpXpkfuIPJT4ChElMVn/Ydruh1WqevXxgRxOXCuqt7obl8N/E5Vo45usQSRgvyFsGsd7N+8ZCBhNHu2QL3G4N8LaZmQVmYZVX9BSVVcpOukZThd0fZscarj/Pmw+Duo3xQOaOlU2e1cCw3bONVsB7aH9XOd+bSK236uGQK7N8LamdDkEJj9KRz/V+f1aZnO6xq2dtqB+l/gJMV7lzgJ99XjQuEU7deCjD3r4PDz4aiLYeq7sKb8Eqya1QApiNAvPnN/J/5ghG7Fx/WEnFPhl//B8sg3jYh8GRAsP/gt0OZk0laX75Fkape1v+9Dy9MrN83PPpsgRKQX0Augbdu2J6xatSrpsZp9gKrz46uhsaKBIhAf+KJUcRbsgoz9yse3ZZmTVOs3ifw6cBLTwR3LJ1xw/mYRJ8Fu/9VJSAe0dJL5mpnQ/EinxCY+57hgwPm9c62TeIN+59j0LKfUGSxyEmnQDwFnypFA/m7SgwVO0k2vF2pbC6bXY9q8RZzUKhtpdQJsmAuNc5zXF+yC/B3OF4u8rVC42ynFZtSHzYuceOo1dkqV/gJo0AKWjWLKolVk/eZ31M9bw2GHHQGLR0C738OezXDw0aBBtn39f+w6tAf1t8xlwrbGXHjZtaQV7UEL91DQ6FCyG7dy/o5gAM2oR2DPVtL3bnG+MOTvYOPYd2jcpCm6aQmZh5zixHHgYbBpofMetj8bshvChvnsbtqBjB2r0J255KU1cmZl3rUe/9rZBA46lqydK6HZ4QS251Iw47/U++MrSKDQeT/rHwgrxkK9xuiqSYgI2vEKyJ3GrnWL8Tc7mob+zRTUO4j6456Cqz6DVp0q87+vViYIq2Iyxpgk8JogUmmqjWlAexFpJyKZwBXAkBqOyRhj6qyUGSinqn4RuQ34Dqeb6weqOq+GwzLGmDorZRIEgKp+C3wb90BjjDEJl0pVTMYYY1KIJQhjjDERWYIwxhgTkSUIY4wxEVmCMMYYE1HKDJSrDBHZBFR2KPWBQMUnx08ui7F6WIzVw2KsulSJ7zeqGnkx+zC1OkFUhYhM9zKSsCZZjNXDYqweFmPVpXp8ZVkVkzHGmIgsQRhjjImoLieId2s6AA8sxuphMVYPi7HqUj2+UupsG4QxxpjY6nIJwhhjTAx1MkGIyLkiskhElopI7xqKoY2IjBaR+SIyT0TucPc3EZEfRGSJ+7uxu19E5FU35l9EpHIrhVQu1jQR+VlEhrrb7URkihvLZ+707IhIlru91H0+J0nxNRKRL0RkoYgsEJGuqfY+ishd7r/zXBH5RESya/p9FJEPRGSjiMwN21fh901ErnWPXyIi1yYhxj7uv/UvIvK1iDQKe+4BN8ZFInJO2P6EfeYjxRj23D0ioiJyoLtdI+9jpalqnfrBmUp8GXAIkAnMBo6qgThaAJ3cxw2AxcBRwAtAb3d/b+B59/H5wHBAgC7AlCTGejfwX2Cou/0/4Ar38dvAze7jW4C33cdXAJ8lKb6PgBvdx5lAo1R6H4FWwAqgXtj7d11Nv49Ad6ATMDdsX4XeN6AJsNz93dh93DjBMZ4NpLuPnw+L8Sj385wFtHM/52mJ/sxHitHd3wZn+YJVwIE1+T5W+m+r6QCS/gdDV+C7sO0HgAdSIK7BwFnAIqCFu68FsMh9/A5wZdjxoeMSHFdrYCRwBjDU/Y+9OewDGno/3Q9DV/dxunucJDi+hu7NV8rsT5n3ESdBrHY//Onu+3hOKryPQE6Zm2+F3jfgSuCdsP2ljktEjGWe+yMw0H1c6rNc/D4m4zMfKUbgC+BYYCUlCaLG3sfK/NTFKqbiD2uxXHdfjXGrEI4HpgAHqeo696n1wEHu45qKuy9wHxB0t5sC21XVHyGOUIzu8zvc4xOpHbAJ+NCtBntfRPYjhd5HVV0D/Bv4FViH877MILXex2IVfd9q+vP0N5xv5MSIJekxisjFwBpVnV3mqZSJ0Yu6mCBSiojsD3wJ3KmqO8OfU+erRI11MxORHsBGVZ1RUzF4kI5TvH9LVY8H9uBUjYSkwPvYGLgYJ5m1BPYDzq2peLyq6fctHhF5CPADA2s6lnAiUh94EHikpmOpqrqYINbg1A0Wa+3uSzoRycBJDgNV9St39wYRaeE+3wLY6O6vibhPAS4SkZXApzjVTK8AjUSkeDXC8DhCMbrPNwS2JDjGXCBXVae421/gJIxUeh//AKxQ1U2qWgR8hfPeptL7WKyi71uNfJ5E5DqgB9DTTWSpFOOhOF8GZrufndbATBE5OIVi9KQuJohpQHu3B0kmTiPgkGQHISIC9AMWqOpLYU8NAYp7MFyL0zZRvP8atxdEF2BHWFVAQqjqA6raWlVzcN6nUaraExgNXB4lxuLYL3ePT+g3UFVdD6wWkcPdXWcC80mh9xGnaqmLiNR3/92LY0yZ9zFMRd+374CzRaSxW1I6292XMCJyLk6150Wqmlcm9ivcXmDtgPbAVJL8mVfVOaraXFVz3M9OLk6HlPWk0PvoSU03gtTED05PgsU4PRseqqEYTsUpvv8CzHJ/zsepax4JLAF+BJq4xwvwhhvzHKBzkuM9jZJeTIfgfPCWAp8DWe7+bHd7qfv8IUmK7ThguvteDsLpBZJS7yPwOLAQmAsMwOlpU6PvI/AJTptIEc5N7IbKvG847QBL3Z/rkxDjUpz6+uLPzdthxz/kxrgIOC9sf8I+85FiLPP8SkoaqWvkfazsj42kNsYYE1FdrGIyxhjjgSUIY4wxEVmCMMYYE5ElCGOMMRFZgjDGGBORJQhjABHZ7f7OEZGrqvncD5bZnlSd5zcmUSxBGFNaDlChBBE2GjqaUglCVU+uYEzG1AhLEMaU9hzQTURmibOGQ5q7/sA0d/7+mwBE5DQRGS8iQ3BGRSMig0RkhjjrPvRy9z0H1HPPN9DdV1xaEffcc0Vkjoj8JezcY6RkjYuB7ghsY5Iq3jcfY+qa3sC9qtoDwL3R71DVE0UkC5goIt+7x3YCjlbVFe7231R1q4jUA6aJyJeq2ltEblPV4yJc61KcUeDHAge6rxnnPnc80AFYC0zEmbtpQvX/ucZEZyUIY2I7G2funFk407E3xZnjB2BqWHIAuF1EZgOTcSZea09spwKfqGpAVTcAY4ETw86dq6pBnOkkcqrlrzGmAqwEYUxsAvxTVUtNnCYip+FMLR6+/QechX7yRGQMzpxKlVUQ9jiAfVZNDbAShDGl7cJZArbYd8DN7tTsiMhh7oJEZTUEtrnJ4Qic5SSLFRW/vozxwF/cdo5mOEtXTq2Wv8KYamDfSowp7Rcg4FYV9cdZ/yIHZz5/wVm97pIIrxsB/ENEFuDMJDo57Ll3gV9EZKY606UX+xpnOczZODP73qeq690EY0yNs9lcjTHGRGRVTMYYYyKyBGGMMSYiSxDGGGMisgRhjDEmIksQxhhjIrIEYYwxJiJLEMYYYyKyBGGMMSai/we5JOPlgHscZAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class TwoLayerPerceptronNew(TwoLayerPerceptronVect):\n",
    "    def fit(self, X, y):\n",
    "        \"\"\" Learn weights from training data.\n",
    "            Input X should be a matrix with separate instances\n",
    "            in each row, and separate features in each column.\n",
    "            The target variable, y, should be integer values \n",
    "            starting from zero, that represent the unique classes.\n",
    "        \"\"\"\n",
    "        \n",
    "        # copy over the data and one-hot-encode Y\n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        Y_enc = self._encode_labels(y)\n",
    "        \n",
    "        # init weights and setup matrices\n",
    "        self.n_features_ = X_data.shape[1]\n",
    "        self.n_output_ = Y_enc.shape[0]\n",
    "        self.W1, self.W2 = self._initialize_weights()\n",
    "\n",
    "        #------------------------------------------\n",
    "        # Enter you Code here to initialize \n",
    "        # self.grad_w1_ and self.grad_w2_\n",
    "        \n",
    "        self.grad_w1_ = np.array([])\n",
    "        self.grad_w2_ = np.array([])\n",
    "        \n",
    "        #------------------------------------------\n",
    "        \n",
    "        # run training for specified number of iterations\n",
    "        for i in range(self.epochs):\n",
    "\n",
    "            # feedforward all instances\n",
    "            A1, Z1, A2, Z2, A3 = self._feedforward(X_data,self.W1,self.W2)\n",
    "\n",
    "            # compute gradient via backpropagation\n",
    "            grad1, grad2 = self._get_gradient(A1=A1, A2=A2, A3=A3, Z1=Z1, Z2=Z2, Y=Y_enc,\n",
    "                                              W1=self.W1, W2=self.W2)\n",
    "            \n",
    "            #------------------------------------------\n",
    "            # ENTER YOUR CODE HERE to keep track of the \n",
    "            #    average magnitude of gradient of each layer \n",
    "            #    grad_w1_ and grad_w2_\n",
    "            #    and make the eta values for each adaptive \n",
    "            \n",
    "            mag1 = np.linalg.norm(grad1)\n",
    "            mag2 = np.linalg.norm(grad2)     \n",
    "            \n",
    "            avg_mag1 = np.mean(mag1)\n",
    "            avg_mag2 = np.mean(mag2)\n",
    "            \n",
    "            self.grad_w1_ = np.append(self.grad_w1_, avg_mag1)\n",
    "            self.grad_w2_ = np.append(self.grad_w2_, avg_mag2)\n",
    "            \n",
    "            self.eta = 0.001 * (8000 / (avg_mag1 + avg_mag2))\n",
    "            \n",
    "            #------------------------------------------\n",
    "\n",
    "            # update each matrix to minimize objective function\n",
    "            self.W1 -= self.eta * grad1\n",
    "            self.W2 -= self.eta * grad2\n",
    "            \n",
    "\n",
    "        return self\n",
    "    \n",
    "    \n",
    "clf2 = TwoLayerPerceptronNew(n_hidden=10, epochs=1500, eta=0.001)\n",
    "clf2.fit(X,y)\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "yhat = clf2.predict(X)\n",
    "print(accuracy_score(y,yhat))\n",
    "\n",
    "ax = plt.subplot(1,1,1)\n",
    "plt.plot(abs(clf2.grad_w1_[10:]), label='w1')\n",
    "plt.plot(abs(clf2.grad_w2_[10:]), label='w2')\n",
    "plt.legend()\n",
    "plt.ylabel('Average gradient magnitude')\n",
    "plt.xlabel('Iteration')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "________________________________________________________________________________________________________\n",
    "\n",
    "That's all! Please **save (make sure you saved!!!) and upload your rendered notebook** and please include **team member names** in the notebook submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}