03. Gradient Descent and its variants/3.06  Implementing Several Variants of Gradient Descent from Scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Implementing Several Variants of Gradient Descent from Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have understood several variants of gradient descent algorithm. In this section, we will understand how can we use them in finding the optimal parameters of simple linear regression. \n",
    "\n",
    "\n",
    "The equation of a simple linear regression can be expressed as:\n",
    "\n",
    "$$ \\hat{y} = mx + b -- (1) $$\n",
    "\n",
    "Thus, we have two parameters $m$ and $b$. We will see how can we use gradient descent and find the optimal values for these two parameters $m$ and $b$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import the Libraries\n",
    "\n",
    "Import the required libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "\n",
    "import random\n",
    "import math\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We generate some random data points with 500 rows and 2 columns (x and y) and use for training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = np.random.randn(500, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see our data has two columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.32615471,  0.67611009])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First column indicates the $x$ value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.3261547057405747"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second column indicates the $y$ value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6761100886332423"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know that equation of a simple linear regression is expressed as:\n",
    "\n",
    "$$ \\hat{y} = mx + b  $$\n",
    "\n",
    "Thus, we have two parameters $m$ and $b$.  We store both of these parameter $m$ and $b$ in an array called theta. First, we initialize theta with zeros as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0.])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "theta[0] represents the value of $m$ and theta[1] represents the value of $b$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss Function \n",
    "\n",
    "Mean Squared Error (MSE) of Regression is given as:\n",
    "\n",
    "$$J=\\frac{1}{N} \\sum_{i=1}^{N}(y-\\hat{y})^{2} -- (2) $$\n",
    "\n",
    "\n",
    "Where $N$ is the number of training samples, $y$ is the actual value and $\\hat{y}$ is the predicted value.\n",
    "\n",
    "The above loss function can be implemented as:\n",
    "\n",
    "We feed the data and the model parameter theta to the loss function which returns the MSE. Remember, data[,0] has $x$ value and data[,1] has $y$ value. Similarly, theta [0] has a value of $m$ and theta[1] has a value of $b$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def loss_function(data,theta):\n",
    "    \n",
    "    #get m and b\n",
    "    m = theta[0]\n",
    "    b = theta[1]\n",
    "    \n",
    "    loss = 0\n",
    "    \n",
    "    #on each data point \n",
    "    for i in range(0, len(data)):\n",
    "        \n",
    "        #get x and y\n",
    "        x = data[i, 0]\n",
    "        y = data[i, 1]\n",
    "        \n",
    "        #predict the value of y \n",
    "        y_hat = (m*x + b)\n",
    "        \n",
    "        #compute loss as given in quation (2)\n",
    "        loss = loss + ((y - (y_hat)) ** 2)\n",
    "        \n",
    "    #mean sqaured loss\n",
    "    mean_squared_loss = loss / float(len(data))\n",
    "        \n",
    "    return mean_squared_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we feed our randomly initialized data and model parameter theta, the loss_function returns the mean squared loss as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8814103222330404"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_function(data, theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to minimize this loss. In order to minimize the loss, we need to calculate the gradient of the loss function $J$ with respect to the model parameters $m$ and $b$ and update the parameter according to the parameter update rule. So, first, we will calculate the gradient of the loss function. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradients of Loss Function "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Gradients of loss function $J$ with respect to parameter $m$ is given as:\n",
    "\n",
    "\n",
    "$$ \\frac{d J}{d m}=\\frac{2}{N} \\sum_{i=1}^{N}-x_{i}\\left(y_{i}-\\left(m x_{i}+b\\right)\\right) -- (3)$$\n",
    "\n",
    "\n",
    "Gradients of loss function $J$ with respect to parameter $b$ is given as:\n",
    "$$ \\frac{d J}{d b}=\\frac{2}{N} \\sum_{i=1}^{N}-\\left(y_{i}-\\left(m x_{i}+b\\right)\\right) -- (4)$$\n",
    "\n",
    "We define a function called compute_gradients which takes the data and parameter theta as an input and returns the computed gradients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_gradients(data, theta):\n",
    "\n",
    "    gradients = np.zeros(2)\n",
    "    \n",
    "    #total number of data points\n",
    "    N = float(len(data))\n",
    "    \n",
    "    m = theta[0]\n",
    "    b = theta[1]\n",
    "    \n",
    "    #for each data point\n",
    "    for i in range(len(data)):\n",
    "        x = data[i, 0]\n",
    "        y = data[i, 1]\n",
    "           \n",
    "        #gradient of loss function with respect to m as given in (3)\n",
    "        gradients[0] += - (2 / N) * x * (y - (( m* x) + b))\n",
    "        \n",
    "        #gradient of loss funcction with respect to b as given in (4)\n",
    "        gradients[1] += - (2 / N) * (y - ((theta[0] * x) + b))\n",
    "    \n",
    "    #add epsilon to avoid division by zero error\n",
    "    epsilon = 1e-6 \n",
    "    gradients = np.divide(gradients, N + epsilon)\n",
    "\n",
    "    return gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.0003067 , -0.00038062])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_gradients(data,theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, when we feed the randomly initialized data and model parameter theta, the compute_gradients function returns the gradients of $m$ i.e $\\frac{d J}{d m}$ and gradients of $b$ i.e  $\\frac{d J}{d b}$. \n",
    "\n",
    "# Update Rule\n",
    "\n",
    "After computing gradients we need to update our model parameter according to our update rule as given below:\n",
    "\n",
    "Update m:\n",
    "$$m=m-\\alpha \\frac{d J}{d m} --(5) $$\n",
    "\n",
    "Update b:\n",
    "$$ b=b-\\alpha \\frac{d J}{d b} -- (6) $$\n",
    "\n",
    "\n",
    "Since we stored $m$ in theta[0] and $b$ in theta[1], we can write our update equation as: \n",
    "\n",
    "$$\\theta = \\theta - \\alpha \\frac{dJ}{d\\theta} -- (7)$$\n",
    "\n",
    "As we learned in the previous section, updating gradients for just one time will not lead us to the convergence i.e minimum of the cost function, so we need to compute gradients and the update the model parameter for several iterations:\n",
    "\n",
    "\n",
    "Set the number of iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_iterations = 50000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr = 1e-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a list called loss for storing the loss on every iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On each iteration, calculate gradients and update the gradients according to our parameter update rule (7):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "\n",
    "for t in range(num_iterations):\n",
    "    \n",
    "    #compute gradients\n",
    "    gradients = compute_gradients(data, theta)\n",
    "    \n",
    "    #update parameter\n",
    "    theta = theta - (lr*gradients)\n",
    "    \n",
    "    #store the loss\n",
    "    loss.append(loss_function(data,theta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below plot shows how the loss(cost) decreases over the training iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Gradient Descent')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8FfW9//HXOztZ2cO+g4KsBkEEEVxaXGkrRahatYq3Wvq7Vu9ttfVa6/W21la7qbXW1pVKqbWtC3UtuKJCkH2Tfd9lCUtIyOf3xwx6TBMSkpyck5PP8/E4j8x85ztzPh9I8snMd853ZGY455xzNZUU6wCcc841bF5InHPO1YoXEuecc7XihcQ551yteCFxzjlXK15InHPO1YoXEucqIGmtpHPD5e9LejTWMTkXr7yQuAZH0gRJH0g6IGl7uHyjJEXj/czsx2Z2XW2PI6mLJJOUcpw+d0oqkbQ/fK2Q9ICktrV9/2gJc+oR6zhc7HghcQ2KpFuAXwE/A9oA+cA3geFAWiX7JNdbgHXjz2aWAzQHvkyQZ2E8FxPXuHkhcQ2GpDzgLuBGM3vWzPZb4CMzu9zMisN+j0v6raTpkg4AoyVdKOkjSfskbZB0Z7ljXylpnaRdkn5Qbtudkp6OWD9d0nuS9kiaL2lUxLaZkv5X0rvhGcWrklqGm98Kv+6RVCRp2PHyNbMSM1sMXAbsAG6JeJ+LJM0LY3hPUv+Ibd+TtCl8/+WSzgnbk8PLdKvCbYWSOobbTpb0mqTd4T7jI473uKQHJb0U7veBpO7htmM5zQ9zuux4ObkEZWb+8leDeAFjgFIgpYp+jwN7Cc5SkoAMYBTQL1zvD2wDvhT27wMUASOBdOD+8H3ODbffCTwdLrcHdgEXhMc6L1xvFW6fCawCegFNwvV7wm1dADte/JHvVa79LuCDcHkQsB0YCiQDVwFrw9hPAjYA7SLes3u4/N/AwrCPgAFACyAr3OcaICU8/k6gT8S/5y5gSLh9CjA1IjYDesT6+8NfsXv5GYlrSFoCO82s9FhDxJnBIUkjI/r+w8zeNbMyMztsZjPNbGG4vgB4Bjgr7DsOeNHM3rLgrOZ/gLJKYrgCmG5m08NjvQbMISgsxzxmZivM7BAwDRhYB7lvJrjUBXA98Dsz+8DMjprZE0AxcDpwlKCg9JGUamZrzWxVuN91wO1mttwC881sF3ARsNbMHjOzUjP7CPgr8NWI9/+bmX0Y/ttPqaOcXILwQuIakl1Ay8jBajM7w8yahtsiv583RO4oaaikGZJ2SNpLMK5y7JJTu8j+ZnYgPF5FOgNfDYvXHkl7gBFA5PjF1ojlg0D2iSRZifbA7ogYbikXQ0eCs5CVwE0EZzbbJU2V1C7cryPB2VJFOQ0td7zLCcZmopmTSxBeSFxDMovgL++x1ehbflrrPwHPAx3NLA94mODyDsAWgl+yAEjKJLjkU5ENwFNm1jTilWVm99QgpmqRlARcDLwdEcP/lYsh08yeATCzP5nZCIICYcBPI/brXklOb5Y7XraZ3VCTeF3j44XENRhmtgf4EfCQpHGSciQlSRpIcJ3/eHKA3WZ2WNIQ4GsR254FLpI0QlIawXhEZT8bTwMXS/piOHidIWmUpA7VSGEHwSWzbtXoi6QUSb0JLsO1IRi7Afg98M3wLEuSssKbCXIknSTpbEnpwGHgEJ9dpnsU+F9JPcP9+ktqAbwI9ApvOEgNX6eF710d26qbk0tMXkhcg2Jm9wI3A98l+AW2Dfgd8D3gvePseiNwl6T9wB0EYxfHjrkY+BbBWcsW4BNgYyXvv4HgjOj7BIVhA8EgdpU/S2Z2EPg/4N3wEtLplXS9TFIRwQ0DzxNcZisws83hceYAk4AHwlhXAleH+6YD9xAMlm8FWgO3hdvuD/N+FdgH/AFoYmb7gS8AEwjGYrYSnMWkV5VT6E7giTCn8VV1dolHZv5gK+ecczXnZyTOOedqxQuJc865WvFC4pxzrlaiWkgkjQmnW1gp6dYKtncK7+3/SNICSReE7amSnpC0UNJSSbeF7R3D/kskLZb0n9GM3znnXNWiNtiuYKK8FQRTSGwEZgMTzWxJRJ9HgI/M7LeS+hB8YriLpK8Bl5jZhPCe/iUEU1wUA23NbK6kHKCQYJqLJRxHy5YtrUuXLjXK48CBA2RlVXVnaWLxnBuHxpZzY8sXap9zYWHhTjNrVVW/SqezrgNDgJVmthpA0lSC2yYjf+kbkBsu5xHcenisPSv8BHMT4Aiwz8x2E9yeiZntl7SU4BO/xy0kXbp0Yc6cOTVKYubMmYwaNapG+zZUnnPj0Nhybmz5Qu1zlrSuWv2ieEYyDhhj4XMcJF0JDDWzyRF92hLc096M4ANl55pZoaRU4CngHCAT+I6ZPVLu+F0IZlPta2b7Knj/6wnmJCI/P79g6tSpNcqjqKiI7OzGNRuE59w4NLacG1u+UPucR48eXWhmg6vqF80zkuqYCDxuZvcpmFL7KUl9Cc5mjhLMgdQMeFvS6xFnN9kEk8rdVFERAQgLzyMAgwcPtppWZf8rpnHwnBNfY8sX6i/naA62byJi/iKgQ9gW6VrCTxib2SyC6b5bEkxf8bIFz2PYDrwLDIZgIJ6giEwxs+eiGL9zzrlqiGYhmQ30lNQ1nL9oAsF0D5HWE1y+IpzXJ4Ng2on1wNlhexbB9NjLJIlgWoelZnY/zjnnYi5qhSR8bsFk4BVgKTDNzBZLukvSJWG3W4BJkuYTTEx3tQWDNg8C2ZIWExSkx8JnSAwHrgTOVvB0uHnHbhl2zjkXG1EdIzGz6cD0cm13RCwvISgO5fcr4vMP1TnW/g6fTf3tnHMuDvgn251zztWKF5JKlJUZUz5Yx4dbS6vu7JxzjZgXkkokJYlpszfw/Moj+FT7zjlXOS8kx/HVwR3ZWGQs3LQ31qE451zc8kJyHJcMbEdqEkybsyHWoTjnXNzyQnIcuRmpDG6TzD/mbeZwydFYh+Occ3HJC0kVRrZPZf/hUl5etDXWoTjnXFzyQlKFk5on0al5pl/ecs65SnghqUKSxFcLOvDeql2s33Uw1uE451zc8UJSDZcWdECCZwv9rMQ558rzQlIN7Zo2YWTPVjxbuJGjZf6ZEueci+SFpJrGD+7I5r2HeWflzliH4pxzccULSTWd26c1zTJTfdDdOefK8UJSTekpyXxpUHteXbyVnUXFsQ7HOefihheSE3D50E6UHDWeLdwY61Cccy5ueCE5AT1a5zC0a3P+9MF6ynzQ3TnnAC8kJ+zy0zuzfvdB3vZBd+ecA7yQnLAvnpJPi6w0pry/LtahOOdcXIhqIZE0RtJySSsl3VrB9k6SZkj6SNKCY89fl5Qq6QlJCyUtlXRbdY8ZbekpyXx1cEfeWLadLXsP1ffbO+dc3IlaIZGUDDwInA/0ASZK6lOu2+3ANDMbBEwAHgrbvwqkm1k/oAD4D0ldqnnMqPvakE6UmTH1Q78V2DnnonlGMgRYaWarzewIMBUYW66PAbnhch6wOaI9S1IK0AQ4Auyr5jGjrlOLTEb2bMXU2espPVpW32/vnHNxJSWKx24PRP7JvhEYWq7PncCrkr4NZAHnhu3PEhSILUAm8B0z2y2pOscEQNL1wPUA+fn5zJw5s0ZJFBUVVbjvgKxS3txXzK+e/RcF+dH8Z6x/leWcyDznxNfY8oX6yznWvwEnAo+b2X2ShgFPSepLcOZxFGgHNAPelvT6iRzYzB4BHgEYPHiwjRo1qkYBzpw5k4r2HXG0jGmrZjD/QA63jBpSo2PHq8pyTmSec+JrbPlC/eUczUtbm4COEesdwrZI1wLTAMxsFpABtAS+BrxsZiVmth14FxhczWPWi5TkJCYM6chbK3awdueBWITgnHNxIZqFZDbQU1JXSWkEg+nPl+uzHjgHQFJvgkKyI2w/O2zPAk4HllXzmPVm4pBOpCSJp/xWYOdcIxa1QmJmpcBk4BVgKcHdWYsl3SXpkrDbLcAkSfOBZ4CrzcwI7szKlrSYoHg8ZmYLKjtmtHKoSn5uBhf0a8u02RsoKi6NVRjOORdTUR0jMbPpwPRybXdELC8BhlewXxHBLcDVOmYsXT28C8/P38xzczfy9WFdYh2Oc87VO/9key0N6tiUAR3yePy9tT7/lnOuUfJCUkuSuGZ4V1bvOODzbznnGiUvJHXggn5taZWTzuPvrol1KM45V++8kNSBtJQkLh/aiRnLd7DGbwV2zjUyXkjqyNeGdiI1WTzx3tpYh+Kcc/XKC0kdaZ2TwcX92/GXORvYf7gk1uE451y98UJSh646owsHjhzlL3P8UbzOucbDC0kdGtCxKad1acYf313jswI75xoNLyR1bNKZ3dj4ySGmL9oa61Ccc65eeCGpY+f2zqdbqyweeWsVwWwvzjmX2LyQ1LGkJDHpzG4s2rSPWat3xToc55yLOi8kUfDlQe1pmZ3GI2+tjnUozjkXdV5IoiAjNZmrhnVh5vIdLN+6P9bhOOdcVHkhiZIrTu9Mk9RkPytxziU8LyRR0iwrjctO68jz8zexde/hWIfjnHNR44Ukiq4d0ZWjZcZj7/lkjs65xOWFJIo6Ns/kgn5tmfL+evYe9GlTnHOJyQtJlN04qgdFxaU8MWttrENxzrmo8EISZX3a5XJu79b88d01HPDnujvnElBUC4mkMZKWS1op6dYKtneSNEPSR5IWSLogbL9c0ryIV5mkgeG2iZIWhv1fltQymjnUhW+N7sGegyVM+WBdrENxzrk6F7VCIikZeBA4H+gDTJTUp1y324FpZjYImAA8BGBmU8xsoJkNBK4E1pjZPEkpwK+A0WbWH1gATI5WDnVlUKdmjOjRkkfeWsPhkqOxDsc55+pUNM9IhgArzWy1mR0BpgJjy/UxIDdczgM2V3CcieG+AApfWZIU7lvRPnFn8tk92FlUzJ9nb4h1KM45V6cUrYkFJY0DxpjZdeH6lcBQM5sc0act8CrQDMgCzjWzwnLHWQWMNbNFEcf9I3AA+Jjg7OTf/syXdD1wPUB+fn7B1KlTy3eplqKiIrKzs2u0byQz4ycfHmbnIePekU1ISVKtjxktdZVzQ+I5J77Gli/UPufRo0cXmtngKjuaWVRewDjg0Yj1K4EHyvW5GbglXB4GLAGSIrYPBRZGrKcCbwDdCc5MHgBuryqWgoICq6kZM2bUeN9/O9aybdb5ey/a1A/X1dkxo6Euc24oPOfE19jyNat9zsAcq8bv+2he2toEdIxY7xC2RboWmAZgZrOADCBy8HwC8EzE+sCw76owyWnAGXUbdvSc1asV/drn8dDMVf7gK+dcwohmIZkN9JTUVVIaQVF4vlyf9cA5AJJ6ExSSHeF6EjCez8ZHIChEfSS1CtfPA5ZGLYM6JonJZ/dg3a6D/H1egxjacc65KkWtkJhZKcEdVa8Q/LKfZmaLJd0l6ZKw2y3AJEnzCc48rg7PNABGAhvMbHXEMTcDPwLekrSA4Azlx9HKIRq+0Cefvu1z+fUbH1PiZyXOuQSQEs2Dm9l0YHq5tjsilpcAwyvZdyZwegXtDwMP12mg9UgSN5/Xi288PodnCzcycUinWIfknHO14p9sj4HRJ7VmYMem/OaNjyku9c+VOOcaNi8kMSCJW77Qi817D/vnSpxzDZ4XkhgZ0aMlQ7o054F/rfRPuzvnGjQvJDEiiZu/0Ivt+4t5+n2fg8s513B5IYmh07u1YHiPFvx25iqfGdg512B5IYmxm887iV0HjvD4e2tjHYpzztWIF5IYK+jcjHNObs3Db67ikwNHYh2Oc86dMC8kceB755/MgeJSHpixMtahOOfcCfNCEgd65ecwrqADT81ax4bdB2MdjnPOnRAvJHHiO+f1QoL7Xl0e61Ccc+6EeCGJE23zmvCNEV35+7zNLNq0N9bhOOdctXkhiSPfPKs7TTNT+enLy2IdinPOVZsXkjiS1ySVyaN78PbHO3nn452xDsc556rFC0mcuXJYZ9o3bcJP/rmUo2XReQyyc87VJS8kcSY9JZnvjjmJxZv38dfCjbEOxznnquSFJA5dMqAdBZ2bce8ry9l/uCTW4Tjn3HF5IYlDkrjjoj7sLCr2Dyk65+KeF5I4NaBjU8YVdOCxd9aybteBWIfjnHOVimohkTRG0nJJKyXdWsH2TpJmSPpI0gJJF4Ttl0uaF/EqkzQw3JYm6RFJKyQtk3RpNHOIpe9+8SRSk8X/vbQ01qE451ylolZIJCUDDwLnA32AiZL6lOt2OzDNzAYBE4CHAMxsipkNNLOBwJXAGjObF+7zA2C7mfUKj/tmtHKItda5Gdw4ugevLtnGuyv9dmDnXHyK5hnJEGClma02syPAVGBsuT4G5IbLecDmCo4zMdz3mG8APwEwszIzS+jfsNeO6ErH5k2464UllB4ti3U4zjn3b2QWnc8qSBoHjDGz68L1K4GhZjY5ok9b4FWgGZAFnGtmheWOswoYa2aLJDUFFgJ/AUYBq4DJZratgve/HrgeID8/v2Dq1Knlu1RLUVER2dnZNdq3rszZWsoD84q5vHca53VOjfr7xUPO9c1zTnyNLV+ofc6jR48uNLPBVXY0s6i8gHHAoxHrVwIPlOtzM3BLuDwMWAIkRWwfCiyMWG9JcBYzLmL/p6qKpaCgwGpqxowZNd63rpSVldkVj75vfe942bbtOxT194uHnOub55z4Glu+ZrXPGZhj1fh9H81LW5uAjhHrHcK2SNcC0wDMbBaQERaLYyYAz0Ss7wIOAs+F638BTq27kOOTJO4a25fi0jIfeHfOxZ1oFpLZQE9JXSWlERSF58v1WQ+cAyCpN0Eh2RGuJwHjiRgfCSvkCwSXtQj3XRK9FOJH15ZZfHNUd/4xbzPv+cC7cy6ORK2QmFkpMBl4BVhKcHfWYkl3Sbok7HYLMEnSfIIzj6vDYgEwEthgZqvLHfp7wJ2SFhBcLrslWjnEmxtHdadT80xu/8cijpT6wLtzLj6kRPPgZjYdmF6u7Y6I5SXA8Er2nQmcXkH7OoIi0+hkpCbzo7GncM1js/n926v51ugesQ7JOef8k+0NzeiTWjPmlDb85l8f+2N5nXNxwQtJA3THxX1Ikvjh84v57Eqgc87FhheSBqhd0ybcfF4v/rVsOy8s2BLrcJxzjZwXkgbqmuFdGdCxKXc+v5jdB47EOhznXCPmhaSBSk4S917an/2HS/jfFxvFHdDOuTjlhaQBO6lNDjeM6sHfPtrEjGXbYx2Oc66R8kLSwH1rdHd6ts7mB39b6E9TdM7FhBeSBi49JZmfjuvPln2Huffl5bEOxznXCHkhSQCndmrGNWd05an31/H+6l2xDsc518h4IUkQ//XFXnRpkcl//WW+X+JyztUrLyQJIjMthfvGD2DznkPc/aLPEOycqz/VKiSSnqpOm4utgs7N+eZZ3fnznA28vuTfnvXlnHNRUd0zklMiV8LnsRfUfTiutm46txe92+Zy63ML2FVUHOtwnHONwHELiaTbJO0H+kvaF772A9uBf9RLhO6EpKUkcf/4Aew7VMr3/7bQ5+JyzkXdcQuJmf3EzHKAn5lZbvjKMbMWZnZbPcXoTlDvtrnc/IVevLJ4G8/NLf9QSuecq1vVvbT1oqQsAElXSLpfUucoxuVqadKZ3RjcuRk/fH4x63YdiHU4zrkEVt1C8lvgoKQBBE8kXAU8GbWoXK0lJ4lfXDYQCf7fMx/5ExWdc1FT3UJSGj4CdyzwgJk9CORELyxXFzo2z+Snl/Zn/sa93Peqf+rdORcd1S0k+yXdRvCM9JckJQGpVe0kaYyk5ZJWSrq1gu2dJM2Q9JGkBZIuCNsvlzQv4lUmaWC5fZ+XtKia8TdaF/Rry8QhnfjdW6t5c8WOWIfjnEtA1S0klwHFwDfMbCvQAfjZ8XYIbxF+EDgf6ANMlNSnXLfbgWlmNgiYADwEYGZTzGygmQ0kKF5rzGxexLG/AhRVM/ZG746L+tArP5tbps1j+/7DsQ7HOZdgqlVIwuIxBciTdBFw2MyqGiMZAqw0s9VmdgSYSnBp7HOHBnLD5TxgcwXHmRjuC4CkbOBm4O7qxO6gSVoyD3ztVPYfLuWWafMpK/Nbgp1zdUfV+ZyBpPEEZyAzAQFnAv9tZs8eZ59xwBgzuy5cvxIYamaTI/q0BV4FmgFZwLlmVljuOKuAsWa2KFz/BfAW8BHwopn1reT9rweuB8jPzy+YOnVqRd2qVFRURHZ2do32jTcz1pfwxJIjXNozlYu7p1XaL5Fyri7POfE1tnyh9jmPHj260MwGV9nRzKp8AfOB1hHrrYD5VewzDng0Yv1KgoH6yD43A7eEy8OAJUBSxPahwMKI9YHA8+FyF2BRdeIvKCiwmpoxY0aN9403ZWVl9u0/zbWut75o73y8o9J+iZRzdXnOia+x5WtW+5yBOVaN37HVHSNJMrPIR/DtourLYpuAjhHrHcK2SNcC0wDMbBaQAbSM2D4BeCZifRgwWNJa4B2gl6SZ1UvBSeInX+lH91bZfPuZj9i851CsQ3LOJYDqFpKXJb0i6WpJVwMvAdOr2Gc20FNSV0lpBEXh+XJ91gPnAEjqTVBIdoTrScB4IsZHzOy3ZtbOzLoAI4AVZjaqmjk4ICs9hd9eUUBxyVFunDLXP1/inKu1quba6iFpuJn9N/A7oH/4mgU8crx9zawUmAy8AiwluDtrsaS7JF0SdrsFmCRpPsGZx9Xh6RTASGCDma2uYW6uEj1aZ/Ozrw5g3oY93P3SkliH45xr4FKq2P5L4DYAM3sOeA5AUr9w28XH29nMplPuzMXM7ohYXgIMr2TfmcDpxzn2WqDCgXZXtQv6tWXSmV35/dtrOLVTM740qH2sQ3LONVBVXdrKN7OF5RvDti5RicjVm++NOZkhXZtz63MLWLRpb6zDcc41UFUVkqbH2dakLgNx9S8lOYkHv3YqzTPTmPTkHP+wonOuRqoqJHMkTSrfKOk6oLCC/q6BaZWTziNfH8yegyX8x1OFHC45GuuQnHMNTFVjJDcBf5N0OZ8VjsFAGvDlaAbm6k/f9nncP34AN0yZy/f/tpCLW/kn351z1XfcQmJm24AzJI3ms4Htl8zsX1GPzNWr8/u15Tvn9uIXr68g5aRURsc6IOdcg1HVGQkAZjYDmBHlWFyM/b9zerBi+37+smALX1y6jXN658c6JOdcA1DdDyS6RkASPx83gM65SXz7mY9YuNHv5HLOVc0LifucJmnJ3HRqOs0y07jm8dls2H0w1iE55+KcFxL3b5pmJPHEN06j5GgZVz32IXsOHol1SM65OOaFxFWoR+scfv/1wWzcfYhJT87x24Kdc5XyQuIqNaRrc+4bP4DZaz/xB2I55ypVrbu2XON18YB2bNl7iB9PX0bL7DTuvOQUJMU6LOdcHPFC4qo06cxubN9XzKPvrCGvSSo3f+GkWIfknIsjXkhclSTxgwt7s/9wKb/+10pym6Ry3ZndYh2Wcy5OeCFx1SKJH3+lH0XFpdz90lJyMlK47LROsQ7LORcHvJC4aktOEr+4bCBFxaXc9txCstNTubB/21iH5ZyLMb9ry52QtJQkHr6igILOzbjpzx/x+pJtsQ7JORdjXkjcCWuSlswfrj6NPm1zuWFKoRcT5xq5qBYSSWMkLZe0UtKtFWzvJGmGpI8kLZB0Qdh+uaR5Ea8ySQMlZUp6SdIySYsl3RPN+F3lcjNSefLaoV5MnHPRKySSkoEHgfOBPsBESX3KdbsdmGZmg4AJwEMAZjbFzAaa2UDgSmCNmc0L9/m5mZ0MDAKGSzo/Wjm448tr4sXEORfdM5IhwEozW21mR4CpwNhyfQzIDZfzgM0VHGdiuC9mdjCc0p7wmHOBDlGI3VWTFxPnXDQLSXtgQ8T6xrAt0p3AFZI2AtOBb1dwnMuAZ8o3SmoKXAy8URfBuporX0ymL9wS65Ccc/VIZtGZP0nSOGCMmV0Xrl8JDDWzyRF9bg5juE/SMOAPQF8zKwu3DwUeNbN+5Y6dArwAvGJmv6zk/a8HrgfIz88vmDp1ao3yKCoqIjs7u0b7NlQ1zflAifHLwsOs3FPGNX3TGNkhNQrRRYf/Pye+xpYv1D7n0aNHF5rZ4Co7mllUXsAwgl/0x9ZvA24r12cx0DFifTXQOmL9F8D3Kzj2H4FfVzeWgoICq6kZM2bUeN+GqjY5HygusSsefd86f+9F+/1bq+ouqCjz/+fE19jyNat9zsAcq8bv2Ghe2poN9JTUVVIawWD68+X6rAfOAZDUG8gAdoTrScB4wvGRYyTdTTCeclMUY3c1lJmWwqNXDeb8vm24+6Wl3P/aimPF3zmXoKJWSMysFJgMvAIsJbg7a7GkuyRdEna7BZgkaT7BOMjV9tlvnZHABjNbfeyYkjoAPyC4C2xueGvwddHKwdVMekoyv5k4iK8WdODXb3zMj15Y4lPQO5fAojpFiplNJxhEj2y7I2J5CTC8kn1nAqeXa9sI+BzmDUBKchI/vbQ/uU1S+cM7a9ixv5j7xg8gIzU51qE55+qYz7XloiYpSdx+YW/a5Gbwf9OXsn3/YX7/9cE0zUyLdWjOuTrkU6S4qJLEpJHdeOBrg5i/YS9f+e17bNh9MNZhOefqkBcSVy8u6t+Op68byq6iI3z5oXdZsHFPrENyztURLySu3gzp2pznbjyDJmnJXPa79/2Di84lCC8krl51b5XNczcMp0+7XG6cMpf7X1vhd3Q518B5IXH1rlVOOn+aNPTT24NvmFLIgeLSWIflnKshLyQuJtJTkrl3XH/uuKgPry3ZxqU+CO9cg+WFxMWMJL4xoiuPXzOEzXsOcckD7/DWih2xDss5d4K8kLiYG9mrFf+YPIJWOelc9diH/PL1FRz1cRPnGgwvJC4udG2Zxd+/NZwvD2zPL1//mKsf+5BdRcWxDss5Vw1eSFzcyExL4b7xA/jJV/rxwZrdXPSbdyhc90msw3LOVcELiYsrkpg4pBPP3XAGKcnist/N4ndvrvJbhJ2LY15IXFzq2z6PF799Juf2zucn/1zGlX/8gK17D8c6LOdcBbyQuLiV1ySV315xKvd8pR9z1+1hzK/e4uVFW2MdlnOuHC8kLq5JYsKQTrz4/0bQsVkm33y6kNueW8DBI/4BRud4HGpgAAAWgUlEQVTihRcS1yB0b5XNX284g2+e1Z2pszdwwa/eZvba3bEOyzmHFxLXgKSlJHHr+Scz5bqhlJYZ4383ix+9sNjPTpyLMS8krsE5o3tLXrlpJFee3pnH3l3L+b96mw9W74p1WM41Wl5IXIOUlZ7CXWP78syk0zGDyx55nx/+YxFFPvmjc/UuqoVE0hhJyyWtlHRrBds7SZoh6SNJCyRdELZfLmlexKtM0sBwW4GkheExfy3Jn+HeiA3r3oKXbzqTa4Z34cn313He/W/y8qKtmPnnTpyrL1ErJJKSgQeB84E+wERJfcp1ux2YZmaDgAnAQwBmNsXMBprZQOBKYI2ZzQv3+S0wCegZvsZEKwfXMGSmpfDDi0/h2W+eQV6TVL75dCHXPjHHZxN2rp5E84xkCLDSzFab2RFgKjC2XB8DcsPlPGBzBceZGO6LpLZArpm9b8GfnE8CX4pG8K7hKejcjBe/PYLbL+zN+6t3cd4v3uTBGSs5UloW69CcS2iK1iUASeOAMWZ2Xbh+JTDUzCZH9GkLvAo0A7KAc82ssNxxVgFjzWyRpMHAPWZ2brjtTOB7ZnZRBe9/PXA9QH5+fsHUqVNrlEdRURHZ2dk12rehSoScdx0q40/LjlC47Shts8TEk9Po3yql0v6JkPOJamw5N7Z8ofY5jx49utDMBlfVr/KfrPoxEXjczO6TNAx4SlJfMysDkDQUOGhmi070wGb2CPAIwODBg23UqFE1CnDmzJnUdN+GKlFyvvR8+Neybdz1whLuLzzIqJNyuf3C3vRonfNvfRMl5xPR2HJubPlC/eUczUtbm4COEesdwrZI1wLTAMxsFpABtIzYPgF4ptwxO1RxTOc+dfbJ+bz6nbO4/cLeFK77hC/+8m3ufH4xnxw4EuvQnEsY0Swks4GekrpKSiMoCs+X67MeOAdAUm+CQrIjXE8CxhOOjwCY2RZgn6TTw7u1vg78I4o5uASQlpLEdWd2Y+Z/jWLikI48OWsto34+kz+8s4bDJUdjHZ5zDV7UComZlQKTgVeApQR3Zy2WdJekS8JutwCTJM0nOPO42j4btBkJbDCz1eUOfSPwKLASWAX8M1o5uMTSIjudu7/Uj3/+50j6d8jjf19cwtk/n8m02Rv8iYzO1UJUx0jMbDowvVzbHRHLS4Dhlew7Ezi9gvY5QN86DdQ1Kie1yeGpa4fyzsc7+dkry/juXxfQNkscabWFMX3b4B9Ncu7E+CfbXaM1omdL/v6t4Tx8xakIuGHKXC554F1mLt/uH2h07gTE+q4t52JKEmP6tiVtxzJ25fQInxc/mwEd8ph8dk/O7d3az1Ccq4KfkTgHJEl8dXBHZvzXKH7ylX7sPniESU/O4YJfv8NLC7b4o36dOw4vJM5FSEtJYuKQTvzrllHc99UBFJcc5Vt/mssXfvkWz83dSMlR/5S8c+V5IXGuAqnJSVxa0IHXbj6L30wcRLLEzdPmc+ZPZ/Dwm6vYe6gk1iE6Fzd8jMS540hOEhcPaMeF/dry5ood/P7t1dzzz2X8+o2PGT+4I9eO6ErH5pmxDtO5mPJC4lw1JCWJ0Se3ZvTJrVm8eS9/eHsNT7+/jidnrWVM3zZ8fVgXhnZt7gPzrlHyQuLcCTqlXR73XzaQ7445mSdmreVPH6xn+sKt9GydzeVDO/GVgg7kZqTGOkzn6o2PkThXQ23yMvjemJN5/7ZzuHdcfzLTkrnzhSUM/b83uPWvC1i0aW+sQ3SuXvgZiXO11CQtmfGDOzJ+cEcWbtzL0++v4+/zNjF19gYGdMhj3OCOXNK/HXmZfpbiEpOfkThXh/p1yOOn4/rzwW3ncsdFfSguLeN//r6I0378Ot/601xmLt/u83q5hONnJM5FQV5mKt8Y0ZVrhndh8eZ9PFu4kb/P28RLC7aQn5vOlwd14NJT29Mz/9+fjeJcQ+OFxLkokkTf9nn0bZ/HbReczIxl2/nLnI38/u3VPPzmKk5uk8NF/dtyUf92dGmZFetwnasRLyTO1ZP0lGTG9G3LmL5t2bG/mOkLt/DC/M38/NUV/PzVFfRrn8fFA9pyYf92tG/aJNbhOldtXkici4FWOelcdUYXrjqjC5v2HGL6gi28sGAzP56+jB9PX8apnZryhVPacF6ffLq3alzPGXcNjxcS52KsfdMmTBrZjUkju7F25wFeXLCZlxdv5Z5/LuOefy6jW6ssvtAnKCqDOjYlKck/9OjiixcS5+JIl5ZZTD67J5PP7smmPYd4fck2XluyjUfDMZVWOemc27s1o05qzRndW5DjH3x0ccALiXNxqn3TJp9e/tp7sISZK7bz6uJtPD9vM898uIGUJFHQuRlnndSKs3q1ok/bXJ+ixcVEVAuJpDHAr4Bk4FEzu6fc9k7AE0DTsM+t4eN5kdQf+B2QC5QBp5nZYUkTge8DBmwGrjCzndHMw7lYy8tMZezA9owd2J4jpWUUrvuEN1fs4M0VO7j35eXc+/JyWuWkM7JnK0b2asmw7i1onZMR67BdIxG1QiIpGXgQOA/YCMyW9Hz4nPZjbgemmdlvJfUheL57F0kpwNPAlWY2X1ILoCRs/xXQx8x2SroXmAzcGa08nIs3aSlJDOvegmHdW3Dr+Sezfd9h3vp4J2+u2MEby7bx17kbAejeKoth3Vtwerfg1TI7PcaRu0QVzTOSIcBKM1sNIGkqMBaILCRGcMYBkEdwhgHwBWCBmc0HMLNd4TFSAQFZknaF+66MYg7Oxb3WuRmMK+jAuIIOHC0zFm/ey6xVu3h/9S7+/tFmnn5/PQC98rM5vVsLhnVrQUGXZjGO2iUSmUVnugZJ44AxZnZduH4lMNTMJkf0aQu8CjQDsoBzzaxQ0k1AAdAaaAVMNbN7I477R+AA8DEw2syOVvD+1wPXA+Tn5xdMnTq1RnkUFRWRnd24br/0nBPH0TJj7b4ylu0+ytLdZXz8yVGKw5+WFulGr+Yp9GiWTI+mSXTITiI5ge8IS9T/4+Opbc6jR48uNLPBVfWL9WD7ROBxM7tP0jDgKUl9w7hGAKcBB4E3JBUCbwE3AIOA1cBvgNuAu8sf2MweAR4BGDx4sI0aNapGAc6cOZOa7ttQec6Jq+RoGQs37WXuuk94ec4KVhWlMGtLMQCZackM7NiUgs7NGNSpKf3aN6VVTuJcDmss/8eR6ivnaBaSTUDHiPUOYVuka4ExAGY2S1IG0JJgTOWtY4PokqYDpwL7wr6rwvZpwK1RzMG5hJKanMSpnZpxaqdm9Di6nrPOOotNew5RuO4T5q77hML1n/DQzFWfTizZJjeDvu3z6N8hj37hVC+JVFxc3YhmIZkN9JTUlaCATAC+Vq7PeuAc4HFJvYEMYAfwCvBdSZnAEeAs4BfhcfpIamVmOwgG8pdGMQfnEpokOjTLpEOzTMYObA/AgeJSFm3ay8KI1xvLtnHsKnib3Az6dcijb7s8Tm6bQ+82uXRo1sQ/KNmIRa2QmFmppMkERSEZ+KOZLZZ0FzDHzJ4HbgF+L+k7BAPvV1swaPOJpPsJipEB083sJQBJPwLeklQCrAOujlYOzjVGWekpDO3WgqHdWnzatv9wCYs37/uswGzcy2tLtn22T1oyPfNz6N02h5PyczipTS4nt8mhWVZaLFJw9SyqYyThZ0Kml2u7I2J5CTC8kn2fJrgFuHz7w8DDdRupc+54cjJSP72N+Jii4lJWbNvP8q3Ba+mWffxz0Vae+XDDp33yc9PplZ9D91bZdG+VRfdW2XRrlU1+brp/eDKBxHqw3TnXQGWnp3w63nKMmbF9fzHLtu5n2ZZ9LN+6n4+3F/GXORs4cOSzmyuz0pLp3jqbbi0/Ky7dWmXRuUUmmWn+a6mh8f8x51ydkUR+bgb5uRmc1avVp+1mxrZ9xazaUcTqHUWs2nGAVTuKmL32E/4+b/PnjtEyO51OzZvQuUUWHZtn0ql5Jp1bBF9bZaf7WEwc8kLinIs6SbTJy6BNXgbDe7T83LaDR0pZs/MAq3ccYP3ug2zYfZB1uw7y4Zrd/GPeJiKfTJyekkSn5pl0bJ5J+6ZNaNs0I/ia14R2TYMClprsTxCvb15InHMxlZmWwint8jilXd6/bTtSWsamPYdYv/sg63cFhSZ4Bbcs7z1U8rn+ErTOSadd0ya0C4vLsSKzcc9RenxykFY56aSnJNdXeo2CFxLnXNxKS0mia8ssurbMIpjk4vMOFJeyZe8hNu85zOY9h9i8N/i6Ze8hlmzZx+tLt1FcWvZp/7vfnwFA08xUWuek0zong9Y56bTK/Wy5dU46rXOD5cy0ZL8poBq8kDjnGqys9BR6tM6hR+ucCrebGZ8cLGHznkO8/u5s2nTpxfb9xWzff5jt+4rZvr+YNTsPsGN/MUeOlv3b/ukpSbTISqN5dhrNs9JpkZVGs8w0WmSn0TwreLXI+mw5NyO1UY7heCFxziUsSZ/+kt/ZOoVRQzpV2M/M2HOw5HNFZkdRMbsPHGFX0RF2Hyhm98ES1uwsYnfRkc/dgRYpOUk0y0wjr0kKTTPTyGuS+ukrt0kqTSPW8zKDr03DbRmpDfdymxcS51yjJ4lmWWk0y0rjpDYVn91EOlxylN0HjgSF5kBYaA6UhF+PsPdQCXsPlbB9/2FWbNvP3kMl7D9cetxjpqckfVpwstNTyMlIISstheyMlM/W0z9bzg6XsyLXM1JiMv7jhcQ5505QRmpyMKDftEm19zlaZuw/XPJpkdlz8LPlvYdK2Be27S8uoaj4KEWHS9i27zBFh0vZX1zKgeLSz93BVpnUZJGdnhJ8Hqf0MK+fcZQmadEtLl5InHOuHiQniaaZaTTNrNm0MWbGoZKjnxaWosOlFBWHr4jl/YdLKSou4eCRo6zbtJW0lOjfDu2FxDnnGgBJZKYFZxqtq7nPzJkz6+UZM/7JHeecc7XihcQ551yteCFxzjlXK15InHPO1YoXEuecc7XihcQ551yteCFxzjlXK15InHPO1YrMqvGZ+wZO0g5gXQ13bwnsrMNwGgLPuXFobDk3tnyh9jl3NrN/n7+/nEZRSGpD0hwzGxzrOOqT59w4NLacG1u+UH85+6Ut55xzteKFxDnnXK14IanaI7EOIAY858ahseXc2PKFesrZx0icc87Vip+ROOecqxUvJM4552rFC0klJI2RtFzSSkm3xjqeEyXpj5K2S1oU0dZc0muSPg6/NgvbJenXYa4LJJ0asc9VYf+PJV0V0V4gaWG4z68lRf/pOVWQ1FHSDElLJC2W9J9he8LmLSlD0oeS5oc5/yhs7yrpgzDOP0tKC9vTw/WV4fYuEce6LWxfLumLEe1x97MgKVnSR5JeDNcTPd+14ffdPElzwrb4+b42M3+VewHJwCqgG5AGzAf6xDquE8xhJHAqsCii7V7g1nD5VuCn4fIFwD8BAacDH4TtzYHV4ddm4XKzcNuHYV+F+54fBzm3BU4Nl3OAFUCfRM47jCM7XE4FPgjjmwZMCNsfBm4Il28EHg6XJwB/Dpf7hN/n6UDX8Ps/OV5/FoCbgT8BL4briZ7vWqBluba4+b72M5KKDQFWmtlqMzsCTAXGxjimE2JmbwG7yzWPBZ4Il58AvhTR/qQF3geaSmoLfBF4zcx2m9knwGvAmHBbrpm9b8F34ZMRx4oZM9tiZnPD5f3AUqA9CZx3GHtRuJoavgw4G3g2bC+f87F/i2eBc8K/PscCU82s2MzWACsJfg7i7mdBUgfgQuDRcF0kcL7HETff115IKtYe2BCxvjFsa+jyzWxLuLwVyA+XK8v3eO0bK2iPG+EljEEEf6EndN7hZZ55wHaCXw6rgD1mVhp2iYzz09zC7XuBFpz4v0Us/RL4LlAWrrcgsfOF4I+DVyUVSro+bIub7+uUE+nsEoeZmaSEvPdbUjbwV+AmM9sXebk3EfM2s6PAQElNgb8BJ8c4pKiRdBGw3cwKJY2KdTz1aISZbZLUGnhN0rLIjbH+vvYzkoptAjpGrHcI2xq6beFpLOHX7WF7Zfker71DBe0xJymVoIhMMbPnwuaEzxvAzPYAM4BhBJczjv2hGBnnp7mF2/OAXZz4v0WsDAcukbSW4LLT2cCvSNx8ATCzTeHX7QR/LAwhnr6vYz2IFI8vgjO11QSDcMcG3E6JdVw1yKMLnx9s/xmfH5y7N1y+kM8Pzn0YtjcH1hAMzDULl5uH28oPzl0QB/mK4PruL8u1J2zeQCugabjcBHgbuAj4C58ffL4xXP4Wnx98nhYun8LnB59XEww8x+3PAjCKzwbbEzZfIAvIiVh+DxgTT9/XMf9miNcXwZ0PKwiuN/8g1vHUIP5ngC1ACcE1z2sJrg2/AXwMvB7xTSTgwTDXhcDgiON8g2AgciVwTUT7YGBRuM8DhLMkxDjnEQTXkhcA88LXBYmcN9Af+CjMeRFwR9jeLfzlsDL8JZsetmeE6yvD7d0ijvWDMK/lRNy1E68/C3y+kCRsvmFu88PX4mMxxdP3tU+R4pxzrlZ8jMQ551yteCFxzjlXK15InHPO1YoXEuecc7XihcQ551yteCFxCUFSi3Bm1HmStkraFLGeVs1jPCbppCr6fEvS5XUU8zuSBkpKqutZZiV9Q1KbiPUqc3Oupvz2X5dwJN0JFJnZz8u1i+B7vqzCHeuZpHeAyQT37+80s6YnuH+yBdOjVHpsM5tX+0idOz4/I3EJTVIPBc8nmULwYa62kh6RNEfB8zvuiOh77AwhRdIeSfcoeM7HrHCOIyTdLemmiP73KHgeyHJJZ4TtWZL+Gr7vs+F7DTxOmPcAOeHZ05PhMa4KjztP0kPhWcuxuH4paQEwRNKPJM2WtEjSw+GzKC4DBgJ/PnZGdiy38NhXhM+eWCTpx2Hb8XKeEPadL2lGHf8XuQTghcQ1BicDvzCzPhbMWXSrmQ0GBgDnSepTwT55wJtmNgCYRfCJ4IrIzIYA/w0cK0rfBraaWR/gfwlmIT6eW4H9ZjbQzL4uqS/wZeAMMxtIMG3HhIi43jKz/mY2C/iVmZ0G9Au3jTGzPxN8qv+y8JhHPg02mIL9bmB0GNfwcCLE4+X8Q+CcsP3LVeTiGiEvJK4xWGVmcyLWJ0qaC8wFehM85Ki8Q2b2z3C5kGDesoo8V0GfEQQTCmJmx6a1OBHnAqcBc8Lp4c8CuofbjhBM2nfMOZI+JJg+4yyCOaSOZyjwLzPbaWYlBA+HGhluqyznd4EnJV2H/85wFfBp5F1jcODYgqSewH8CQ8xsj6SnCeZjKu9IxPJRKv9ZKa5GnxMl4I9m9j+fawxmrz1kxyZUkjIJ5kU61YIpxu+m4lyqq7KcJxEUoIuAuZIGWfBgJOcA/+vCNT65wH5gnz57alxdexcYDyCpHxWf8XzKwgcyRUyD/jowXlLLsL2FpE4V7NqE4OFOOyXlAJdGbNtP8Ljh8j4ARofHPHbJ7M0q8ulmwZP2/gf4hPh40JOLI35G4hqbucASYBmwjuCXfl37DcGloCXhey0heDLf8fwBWCBpTjhO8iPgdUlJBDM4fxPYHLmDme2S9ER4/C0EReKYx4BHJR0ieHbFsX02SvofYCbBmc8LZvZSRBGryC8kdQ37v2pmi6rIxTUyfvuvc3Us/KWcYmaHw0tprwI97bNHwTqXUPyMxLm6lw28ERYUAf/hRcQlMj8jcc45Vys+2O6cc65WvJA455yrFS8kzjnnasULiXPOuVrxQuKcc65W/j9Gj2lOsL4aqgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')\n",
    "plt.title('Gradient Descent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MiniBatch Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just learned how gradient descent works. In gradient descent, to perform a single\n",
    "parameter update we iterate through all the data points in our training set. Updating the parameters of the model only after iterating through all the data points in the training set makes the gradient descent very slow and it will increase the training time\n",
    "especially when we have a large dataset. \n",
    "\n",
    "\n",
    "So, we use minibatch gradient descent, in minibatch gradient descent, we update the parameters after iterating some batch of data points. Let us say the batch size is 50\n",
    "which means that we update the parameter of the model after iterating through 50 data\n",
    "points instead of updating the parameter after iterating through each individual data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def minibatch(data, theta, lr = 1e-2, minibatch_ratio = 0.01, num_iterations = 5000):\n",
    "    \n",
    "    loss = []\n",
    "    minibatch_size = int(math.ceil(len(data) * minibatch_ratio))\n",
    "    \n",
    "    for t in range(num_iterations):\n",
    "        sample_size = random.sample(range(len(data)), minibatch_size)\n",
    "        np.random.shuffle(data)\n",
    "        \n",
    "        #sample batch of data\n",
    "        sample_data = data[0:sample_size[0], :]\n",
    "        \n",
    "        #compute gradients\n",
    "        grad = compute_gradients(sample_data, theta)\n",
    "       \n",
    "        #update parameters\n",
    "        theta = theta - (lr * grad)\n",
    "        \n",
    "        loss.append(loss_function(data,theta))\n",
    "            \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform minibatch gradient descent and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = minibatch(data, theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Mini-Batch Gradient Descent')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8FdX9//HX+95sJGEPRJBVRQVRQRBQXMAVta32WxfQ2vqtVttvaau1tVqtX2ttv7bVVn9qtbS1tBZFXKqoqLiAKyqL7Iuy77ssYcn6+f0xk3CJgYQkk5vl83w88sidM2fmns/Nzf3cM2fmjMwM55xzrrpiyW6Ac865hs0TiXPOuRrxROKcc65GPJE455yrEU8kzjnnasQTiXPOuRrxROJqnaTHJP2ytutGTdJySeckux2JJF0j6f2E5TxJRySzTc6V54nEVVn4QVsgKadc+aeSTFI3ADP7npn9uir7rKxu+Jx7wg/QLyS9IqlzFdvbLWxXSlXqHypJHST9VdLasH1LJY2WdGwUzwdgZtlmtrSm+wnbeU8ldUzSrjC2LZLeknRFTZ87KpLukvTvZLejKfJE4g7VMmBE6YKk44HMiJ/zq2aWDXQANgAPRfx8lZLUFviQIPbTgebAScA7wLkH2CaShBaxE8PX/hhgNPCwpP9NbpNcfeOJxB2qJ4BvJSx/G/hXYoXEb7uShkhaLelmSRslrZP03xXVrYyZ7QWeBXolbH9R2CPaIWmVpLsSNnk3/L0t/FZ9SrjNdyUtkLRT0nxJJyVs00fSbEnbJT0tKeMAzbkJ2AFcbWZLLLDNzP5hZg+Fz1PaI7pW0krg7bD8GUnrw+d4V9JxCfG0lTQ+jOcT4Mhyr61JOip8nC7pPkkrJW0IDxM2q+x1l3Q9cBVwS/i6vFSF136zmT0BfB+4LUykSGop6e/h/tdIukdSPFx3lKR3wjg3S3o6IY7jJL0haWvY9l+E5TFJt0paEvaCxklqU+71/HYY82ZJt4frhgG/AK4IY5pVWUyu9ngicYfqI6CFpJ7hB8ZwoLLDCYcBLYHDgWuBRyS1PtQnlpQJXBG2odQugsTWCrgI+L6kS8J1Z4S/W4WHhKZIugy4K9ymBfA1YEvC/i4HhgHdgROAaw7QnHOA/5hZSRWafibQEzg/XH4V6AG0B2YAYxLqPgLsJeh9fSf8OZB7gaOBPsBRBK/vnQnrK3zdzWxU+Jy/D1+Xr1YhhlIvAinAgHB5NFAUPn9f4DzgunDdr4GJQGugE2FPUlJz4E3gNaBjuO1b4TY/BC4heM06Al8QvCaJTiPoIZ0N3Cmpp5m9BvwWeDqM6cRDiMnVkCcSVx2lvZJzgQXAmkrqFwJ3m1mhmU0A8gg+CKrqBUnbgO3hc/6hdIWZTTazOWZWYmazgacIPoQO5DqCD9CpYS9isZmtSFj//8xsrZltBV4i+JCuSA6wvnRB0tckbQt7ORPL1b3LzHaZ2Z6wzY+b2U4zyydIaieG3+zjwDeAO8P6c4F/VvTkkgRcD9xkZlvNbCfBB+nwhGo1fd2/xMwKgc1AG0m5wIXAjWF7NwJ/SmhDIdAV6Ghme82s9KSBrwDrzez+sHynmX0crvsecLuZrU54fS4td1jwV2a2x8xmAbMATxpJ5onEVccTwJUE39b/dfCqAGwxs6KE5d1AdvlKkl4ND0vkSboqYdUlZtYKyABGAu9IOizcZqCkSZI2SdpO8EGUU37fCToDSw6yfn3C4wrbWRoTQa8BADMbH7bxJiCtXN1VCTHGJd0bHrrZASwPV+UA7Qi+7a9K2DYxySVqRzA+Mz1MYNsIvuG3S2xjVV73QyEpNXyOrQRJIhVYl9CGvxD0tABuAQR8ImmepNLe1cH+Bl2B/yTsbwFQDOQm1Knq38jVEU8k7pCF3+CXEXwbfb4W93tBeFgi28zGVLC+2MyeJ/hgOS0sfhIYD3Q2s5bAYwQfXgAVTW29inLjDtX0FnCJpKr8DyW240rgYoJDYy2BbmG5gE0Eh4kSz0rrcoB9bgb2AMeZWavwp2U4MF4V1Z32++KwjZ8QvJb5QE5CG1qY2XEAZrbezL5rZh2BG4A/h+M7q4ADncK8CrggYX+tzCzDzCrr9dYkJldDnkhcdV0LnGVmu+rqCRW4mOCY+4KwuDmw1cz2ShpA8EFdahNQwv4fWn8DfiqpX7i/oyR1rUZz/hi24wlJR4b7as6BD4WVak7w4buFoEfx29IVZlZMkJjvkpQpqRfByQxfEo7N/BX4k6T2AJIOl3R+RfUrsIEDf5h/iaQ2YS/xEeB3ZrbFzNYRjIHcL6lFOFB+pKQzw20uk9Qp3MUXBB/0JcDLQAdJN4YnDDSXNDCs9xjwm9K/iaR24d+8qjF1q2Jyd7XIX3BXLeGZStPq6OlekpRHcJbUb4Bvm9m8cN3/AHdL2kkw0DwuoY27w/ofhIdKBpnZM2HZk8BO4AWgzaE2yMw2A4MIBsbfD/c1kyBRfP8gm/6L4HDVGmA++584AMGhu2yCwzejgX8cZF8/BxYDH4WHyd6k6mMgfwd6ha/LCwepNyt87RcTjC/dZGaJA/rfIjiUN58gWTzLvkN+JwMfh9uPB35sZkvD8Zxzga+GcX4ODA23eTCsOzH8m34ElCaZyjwT/t4iaUYVt3G1QH5jK+ecczXhPRLnnHM14onEOedcjXgicc45VyORJhJJwyQtkrRY0q0VrO8SXgPwqYJpKS4My1Ml/VPSHAVTWdwWlncO688Pz0v/cZTtd845V7nIBtvDq3Q/Izg7YzUwFRhhZvMT6owCPjWzR8NTHSeYWTdJVwJfM7Ph4bQY84EhBKdNdjCzGeGpltMJLlabz0Hk5ORYt27dqhXHrl27yMrKqta2DZXH3DQ0tZibWrxQ85inT5++2czaVVYvytlIBwCLS6e8ljSW4GKmxA99I5jvCIKLs9YmlGeF0yI0AwqAHeG0FesAzGynpAUE8wgdNJF069aNadOqd6bq5MmTGTJkSLW2bag85qahqcXc1OKFmscs6UAzK+xfL8IeyaXAMDO7Lly+GhhoZiMT6nRg36RuWcA5ZjY9nIbhCYJJ2TIJzl0fVW7/3Qhmd+1tZjsqeP7rCeYiIjc3t9/YsWOrFUdeXh7Z2U1rBgaPuWloajE3tXih5jEPHTp0upn1r6xesu+PMAIYbWb3K5ji+wlJvQl6M8UEs3+2Bt6T9GZC7yYbeI5gsrgvJRGAMPGMAujfv79VNyv7t5imwWNu/JpavFB3MUc52L6G/ecM6sSXZ4m9lvBKZDObQjApXw7BNBevhbOWbgQ+APpD2aRxzwFjwnmXnHPOJVGUiWQq0ENSd0lpBFNLjy9XZyXB4Ssk9SRIJJvC8rPC8iyCqSgWhlNn/x1YYGZ/jLDtzjnnqiiyRBJOXz0SeJ1ggr1xZjZP0t2SvhZWuxn4roK7mT0FXGPBoM0jQLakeQQJ6R/hvSYGA1cDZ0maGf5cGFUMzjnnKhfpGEl4M50J5cruTHg8nyA5lN8uD7isgvL32TdFuHPOuXrAr2x3zjlXI55IDuLfH61g3ubiZDfDOefqNU8kB1BYXMKTH6/kD9P20v+eN9i4c2+ym+Scc/WSJ5IDSI3HGPe9U+ifG2dzXgFX/OUjikv83i3OOVeeJ5KDyE5PYWTfDK45tRvLNu9i2eY6u6usc841GJ5IqqBvl1YAnPPHd7h29FT2FPi4iXPOlfJEUgVDj21f9vithRs5/feTeH7G6iS2yDnn6g9PJFXQIiOVT24/m4ev7MtPzzua4pISfvbsbAqKSpLdNOecS7pkT9rYYLRvnsFXTugIQFZ6Cr96aT4rt+7iqPbNk9wy55xLLu+RVEObrDQA9hZ6j8Q55zyRVENWWtCRi+hWLs4516B4IqmGWPiqlXgmcc45TyTVEcxm74nEOefAE0m1xDyROOdcGU8k1RAvSyRJbohzztUDnkiqIRbeEaXEM4lzznkiqQ55j8Q558p4IqmGsh6Jj5E455wnkuqIx3yw3TnnSkWaSCQNk7RI0mJJt1awvoukSZI+lTRb0oVheaqkf0qaI2mBpNuqus+64Ie2nHNun8gSiaQ48AhwAdALGCGpV7lqdwDjzKwvMBz4c1h+GZBuZscD/YAbJHWr4j4j54e2nHNunyh7JAOAxWa21MwKgLHAxeXqGNAifNwSWJtQniUpBWgGFAA7qrjPyJVeR7Jtd0FdP7VzztU7Uc7+eziwKmF5NTCwXJ27gImSfghkAeeE5c8SJIh1QCZwk5ltlVSVfQIg6XrgeoDc3FwmT55crSDy8vK+tO3G3cFkjTc9PYuMLZ/TLEXV2nd9VVHMjZ3H3Pg1tXih7mJO9jTyI4DRZna/pFOAJyT1Juh5FAMdgdbAe5LePJQdm9koYBRA//79bciQIdVq4OTJk6lw23aruOW52Ty3Jpu/fqt/2bhJY3DAmBsxj7nxa2rxQt3FHOWhrTVA54TlTmFZomuBcQBmNgXIAHKAK4HXzKzQzDYCHwD9q7jPOnH5yZ25on9n3lywka//+UO27y5MRjOccy7pokwkU4EekrpLSiMYTB9frs5K4GwAST0JEsmmsPyssDwLGAQsrOI+68zdlxzH2ce2Z+aqbTzw1mfJaoZzziVVZInEzIqAkcDrwAKCs7PmSbpb0tfCajcD35U0C3gKuMbMjODMrGxJ8wiSxz/MbPaB9hlVDJVJT4nz92tO5sROLfnHB8t5adbayjdyzrlGJtIxEjObAEwoV3ZnwuP5wOAKtssjOAW4SvtMtsevOZkhf5jM01NX8dUTOya7Oc45V6f8yvZa0DY7ncFH5fD+4s28Pm99spvjnHN1yhNJLbnzq8F1kau27k5yS5xzrm55IqklbbLSACgoLklyS5xzrm55IqklafHgpfz9a4v8VGDnXJPiiaSWxGL7Lkj86bOzKPKeiXOuifBEUot+dNZRALwxfwNn/mEyhZ5MnHNNgCeSWvST845h6W8v5KqBXVizbQ+TF21KdpOccy5ynkhqWSwmfnR2DwA27Nib5NY451z0PJFEoHT+Rr9biXOuKfBEEoFYWSbxVOKca/w8kUSg9PwtvxWvc64p8EQSgdJ7k5j3SJxzTYAnkgiU9kg8jTjnmgJPJBGIlfVIktwQ55yrA55IohB2SUo8kzjnmgBPJBFoRLdvd865SnkiiUDZGIl3SJxzTYAnkgiUjZH4cLtzrgmINJFIGiZpkaTFkm6tYH0XSZMkfSpptqQLw/KrJM1M+CmR1CdcN0LSnLD+a5JyooyhOkoPbS3fspuXZ/t93J1zjVtkiURSHHgEuADoBYyQ1KtctTuAcWbWFxgO/BnAzMaYWR8z6wNcDSwzs5mSUoAHgaFmdgIwGxgZVQzVpfDg1pMfr2Tkk5+yp6A4yS1yzrnoRNkjGQAsNrOlZlYAjAUuLlfHgBbh45ZARV/fR4TbQjD8ICBLwVV/LQ6wTVKVH2z3s7ecc41ZSoT7PhxYlbC8GhhYrs5dwERJPwSygHMq2M8VhAnIzAolfR+YA+wCPgd+ULvNrjk/a8s515REmUiqYgQw2szul3QK8ISk3mZWAiBpILDbzOaGy6nA94G+wFLgIeA24J7yO5Z0PXA9QG5uLpMnT65WA/Py8g5526Jyk2y99957ZKQ0nOxSnZgbOo+58Wtq8ULdxRxlIlkDdE5Y7hSWJboWGAZgZlMkZQA5wMZw/XDgqYT6fcK6SwAkjQO+NIgf1hkFjALo37+/DRkypFpBTJ48mUPdtrC4BCa+WrZ8+umnk5We7JxdddWJuaHzmBu/phYv1F3MUY6RTAV6SOouKY0gKYwvV2clcDaApJ5ABrApXI4Bl7NvfASCRNRLUrtw+VxgQWQRVFP5voePkDjnGrPIviabWZGkkcDrQBx43MzmSbobmGZm44Gbgb9Kuong8/Ya2zdl7hnAKjNbmrDPtZJ+BbwrqRBYAVwTVQzVFfNBEudcExLp8RYzmwBMKFd2Z8Lj+cDgA2w7GRhUQfljwGO12tBa5nnEOdeU+JXtEZBnEudcE+KJxDnnXI14IolIYqfE75TonGvMPJFExAfcnXNNhSeSiGSmxpPdBOecqxOeSCJy3+UnJrsJzjlXJzyRROT84w5jyDHtKq/onHMNnCeSCJ12VHCrFB9qd841Zp5InHPO1YgnEuecczXiicQ551yNeCJxzjlXI55I6oBf2O6ca8w8kUTIJ290zjUFnkicc87ViCcS55xzNeKJxDnnXI14IqkLPtjunGvEPJFEyIfanXNNQaSJRNIwSYskLZZ0awXru0iaJOlTSbMlXRiWXyVpZsJPiaQ+4bo0SaMkfSZpoaRvRBmDc865g0uJaseS4sAjwLnAamCqpPFmNj+h2h3AODN7VFIvYALQzczGAGPC/RwPvGBmM8Ntbgc2mtnRkmJAm6hicM45V7nIEgkwAFhsZksBJI0FLgYSE4kBLcLHLYG1FexnBDA2Yfk7wLEAZlYCbK7dZjvnnDsUiup+4pIuBYaZ2XXh8tXAQDMbmVCnAzARaA1kAeeY2fRy+1kCXGxmcyW1AuYAzwBDgCXASDPbUMHzXw9cD5Cbm9tv7Nix5atUSV5eHtnZ2dXaduLyQp5cWMAjZ2eSldpwRkxqEnND5TE3fk0tXqh5zEOHDp1uZv0rrWhmkfwAlwJ/S1i+Gni4XJ2fADeHj08h6K3EEtYPBOYkLOcQ9GIuTdj+icra0q9fP6uuSZMmVXvbv7+31Lr+/GX7Yld+tfeRDDWJuaHymBu/phavWc1jBqZZFT7voxxsXwN0TljuFJYluhYYB2BmU4CMMFmUGg48lbC8BdgNPB8uPwOcVHtNrl0+Q4pzrimIMpFMBXpI6i4pjSApjC9XZyVwNoCkngSJZFO4HAMuJ2F8JMyQLxEc1iLcNnHMxTnnXB2LbLDdzIokjQReB+LA42Y2T9LdBN2l8cDNwF8l3URwyOqaMFkAnAGssnCwPsHPgSckPUCQdP47qhicc85VLsqztjCzCQSn9CaW3ZnweD4w+ADbTgYGVVC+giDJOOecqwf8yvY64Pcjcc41Zp5IIuRj7c65psATiXPOuRrxROKcc65GPJE455yrEU8kdcDH2p1zjZknkgjJL213zjUBnkicc87ViCcS55xzNeKJxDnnXI1UKZFIeqIqZa5i5pe2O+casar2SI5LXAhvo9uv9pvTuPhYu3OuKThoIpF0m6SdwAmSdoQ/O4GNwIt10kLnnHP12kETiZn9n5k1B/5gZi3Cn+Zm1tbMbqujNjrnnKvHqnpo62VJWQCSvinpj5K6Rtgu55xzDURVE8mjwG5JJxLcjGoJ8K/IWtXI+FC7c64xq2oiKQrvXHgx8LCZPQI0j65ZjYOPtTvnmoKq3iFxp6TbgKuB08P7qadG1yznnHMNRVV7JFcA+cB3zGw90An4Q2Stcs4512BUKZGEyWMM0FLSV4C9ZlbpGImkYZIWSVos6dYK1neRNEnSp5JmS7owLL9K0syEnxJJfcptO17S3CpF6ZxzLjJVvbL9cuAT4DLgcuBjSZdWsk0ceAS4AOgFjJDUq1y1O4BxZtYXGA78GcDMxphZHzPrQ3A4bZmZzUzY938BeVVpe33gF7Y75xqzqo6R3A6cbGYbASS1A94Enj3INgOAxWa2NNxmLMFg/fyEOga0CB+3BNZWsJ8RwNjSBUnZwE+A64FxVWx/cvil7c65JqCqiSRWmkRCW6i8N3M4sCpheTUwsFydu4CJkn4IZAHnVLCfKwgSUKlfA/cDuw/25JKuJ0g25ObmMnny5EqaW7G8vLxqb/v5ykIAPvzwQ1qmN5ykUpOYGyqPufFravFC3cVc1UTymqTXgafC5SuACbXw/COA0WZ2v6RTgCck9TazEgBJA4HdZjY3XO4DHGlmN0nqdrAdm9koYBRA//79bciQIdVq4OTJk6nutqs+WgHz53LqqafSrnl6tfaRDDWJuaHymBu/phYv1F3MB00kko4Ccs3sZ+G4xGnhqikEg+8HswbonLDcKSxLdC0wDMDMpkjKAHII5vKCYNzkqYT6pwD9JS0P295e0mQzG1JJW5xzzkWkssNTDwA7AMzseTP7iZn9BPhPuO5gpgI9JHWXlEaQFMaXq7MSOBtAUk8gA9gULscIBvbLxkfM7FEz62hm3QiS2meeRJxzLrkqSyS5ZjanfGFY1u1gG5pZETASeB1YQHB21jxJd0v6WljtZuC7kmYR9DyusX037zgDWFU6WN+QmU+S4pxrxCobI2l1kHXNKtu5mU2g3FiKmd2Z8Hg+MPgA204GBh1k38uB3pW1IZkazvC6c85VX2U9kmmSvlu+UNJ1wPRomuScc64hqaxHciPwH0lXsS9x9AfSgK9H2TDnnHMNw0ETiZltAE6VNJR9h5FeMbO3I2+Zc865BqFK15GY2SRgUsRtabx8rN0514hVdfZfVw0+Q4pzrinwROKcc65GPJE455yrEU8kzjnnasQTSR3wsXbnXGPmiSRC8mvbnXNNgCeSCO3YG9yPpKCoJMktcc656HgiidC4acF9vYbeN5kPFm9Ocmuccy4ankgilJESB6CoxPjhU58muTXOORcNTyQR+sqJHQC4oPdh5BcWJ7k1zjkXDU8kEfr+mUcy687z6NImk2Lzc7ecc42TJ5IISaJlZirxmCgu8UTinGucPJHUgXhMFHkicc41Up5I6kA8JsygxJOJc64R8kRSB+LhNMCFJSW8Pm89c9dsp6jYry1xzjUOkSYSScMkLZK0WNKtFazvImmSpE8lzZZ0YVh+laSZCT8lkvpIypT0iqSFkuZJujfK9teWeDxIJFOXfcENT0znKw+9z9l/fId3PtvE7oKiL9XPLyrmB0/O4KlPVmI+SO+cq+eqdGOr6pAUBx4BzgVWA1MljTez+QnV7gDGmdmjknoBE4BuZjYGGBPu53jgBTObKSkTuM/MJklKA96SdIGZvRpVHLUhJRYkkuc/XQ3AcR1bMG/tDr79+CekxMQpR7blouM78F8ndSItJcaqrbt5ZfY6Xpm9jt9OWECP9tn89PxjOPXInGSG4ZxzFYoskQADgMVmthRA0ljgYiAxkRjQInzcElhbwX5GAGMBzGw34Z0azaxA0gygUyStr0WDjmhLRmqM52esAeD+y0+kbVY6C9btYNKijYybuor3Pt/M7S/M5SsndOBbp3QF4ORurTnmsOb8+6OVXPnXj7nutO6MPOsoWmWmJTMc55zbj6I6dCLpUmCYmV0XLl8NDDSzkQl1OgATgdZAFnCOmU0vt58lwMVmNrdceStgRrjN0gqe/3rgeoDc3Nx+Y8eOrVYceXl5ZGdnV2vbRHuLjMXbStiWX8Lgjiko4faJRSXG1PXFPLUwnx0F+7b58Unp9G2fwtT1RUxYVsiy7SWc2zWFy45OIy0e3YSQtRVzQ+IxN35NLV6oecxDhw6dbmb9K61oZpH8AJcCf0tYvhp4uFydnwA3h49PIeitxBLWDwTmVLDvFOBV4MaqtKVfv35WXZMmTar2todqT0GRvbNoo93/+kIb+eQM27Rz737rz7pvknX9+cvW9ecv2/eemBZZO+oy5vrCY278mlq8ZjWPGZhmVfiMjfLQ1hqgc8Jyp7As0bXAMAAzmyIpA8gBNobrhwNPVbDvUcDnZvZArbY4yTJS45xxdDvOOLpdheufvuEU3lm0id+/vpA3F2yo49Y551zFojxrayrQQ1L3cGB8ODC+XJ2VwNkAknoCGcCmcDkGXE44PlJK0j0E4yk3Rtj2eiknO51v9OvEZf06+5Xyzrl6I7JEYmZFwEjgdWABwdlZ8yTdLelrYbWbge9KmkXQ87gm7E4BnAGssoTxD0mdgNuBXsCM8NTg66KKob6Kx0SJUWenBu8pKGbZ5l118lzOuYYnykNbmNkEglN6E8vuTHg8Hxh8gG0nA4PKla0Gv+1g6enExSVGSkSD7i/NWkuJGccc1pxhD7wHwGPf7Mew3odF8nzOuYYr0kTiolF6gWNRiRHe8qRW7Smq+P4pW3bl1/6TOecaPJ8ipQFK7JFEoXT2ln5dW5OTnV5W7hfZO+cq4j2SBigeC/J/VDMKl+71ayd25FundGXDjnwG/d9bPl2Lc65C3iNpgEqHRV6aVdFEADVXmi+k4J4qqeETehpxzlXEE0kDNPCItgDc8cJcVmyp/bOpShNG6dX3pb99GnznXEU8kTRAPTu04JErTwLg8r9M4fevLeS0373NJ8u2Vmn7l2atZdS7S3hiyvIKp7MvPYQVDsWUnSbnacQ5VxEfI2mgLjqhAzNXdWfMxyv58+QlQJBUAB4c3oeL+xxe4Xartu7e74ysX744j1OPbMuY6waW9TxKE0YsXC797UMkzrmKeCJpwG6/qBe3DDuWuWu288HizezML+Iv7yzlx2NnMn7mWi7t14kLju+w3za7C4oBuKRPR77YXcg7n23iwyVb2FNYTGZa8HYoPYJV2iMp7ZKUeCZxDoCnp66kZbNUhvXuUHnlJsATSQOXGo/Rt0tr+nZpDcBXT+jIo+8s4d1Fm3hr4Uam3HYWHVo2A+DzDTs5/4F3AbjohI6c2yuXf01Zzp0vzuP3ry3CzPjOad0ZtyiYglhlPZK6j8u5+sbM+MV/5vDG/A1szgv+Ry7r14mhx7bngt6H7Tejd1PjiaSR6X14Sx658iTenL+B6/41jWWbdvHWgo08OnkJa7btKatX2rvISA2uaBz94XIAJsxdT5uUYN3pPYIbaZUNtnuPxDVhxSXGU5+s2q/smemreWb6al78wWBO7NwqSS1LPk8kjVTrrFQArvzbxxWuL/3uNLB7G45ol8XSTcHZX5t25rMJOOWItmU9mbLBds8jrhH42TOz2L6nkAHd2/DCzDX86KwenNYjh7z8Ito3zzjgdqXXbd0y7Bi+d8aRbNtTyPy1O/jm3z8mL//Lt8xuSjyRNFJ9O7fmrq/2YuXWPeQXFTP85C48POlzrjv9CAqKSjj1yOAU4q5ts3jrJ2diBmu37+G+1xfxwsy1dGrdrGxfMfl1JK7h+mTZVp6bvpqcoiKGEPQiACbOD27FcP0T+91Lj8FHtWVAt7ac1iOHI9tl0SwtTnpKnPEzg+u2iouNWEy0yUojIzU48bWpz8btiaQeKnWHAAAbwElEQVSRisXENYO771f2l6srvtGZJCTo1DqTB4b35YJ22zh36AkJ64PffmjLNUSvzF7L09NWERf89AqjeXoKl/bvxDcHdeWTZVspLC5hycY8Nu8qwMx4a8FGPli8hT+9+RkAGakxOrZqVtZrP/e43LJ9x0qnK2ri/xueSNyXpMdV9g+SqIn/r7gGrthgV0ExRSVGajzGke2yObLdl29Duyu/iFfmrOOJKSs4p2cuyzbnUVhitGqWSvecbI49rEVZ3bJ574qb9j+HJxJXqVgTPhvFNXyJH/G9//d14ODv6az0FC7v35nL+3c+YJ1SpfvxHolzlSg7tNXEjwO7hqtZapxvHhtnYzyHN+Zv4OjcL/dEqiMe8+mDwBOJqwIfbHcNXUZqjMGHpzJkSN9a3W/cx0gAn2vLVUHpQQAfbHcNUZRv27JDW028R+KJxFWq9NCW5xHXUEV11Xlpj+StBRspLjGen7GaW56dxcxV2wDYE05J9OqcdUxfUbVJVStTXGLsLqhf161EemhL0jDgQSAO/M3M7i23vgvwT6BVWOdWM5sg6SrgZwlVTwBOMrOZkvoBo4FmBPeD/7H5HZciVX4yR+dcoE1mGgDjZ61lfML9gcZNW801p3Zj9IfLyUqLs6s0ofz4dDq3ySQ7vfofvTePm8kLM9cy/OTOHJ3bnGG9DyMzLU6rsC3JEFkikRQHHgHOBVYDUyWNN7P5CdXuAMaZ2aOSehEkhm5mNgYYE+7neOAFM5sZbvMo8F3g47D+MODVqOJwAYn97pBoZtz8zCyKS4yeHVpw7GHN6dmhBe2bpzfpOYdc/WMRfgVqmZnKnLvO49U563lx1ho+WLyFmIKJT0unHbqsf+eyxxc8+B4A91zSm28O6lqt51y6Obie5fV56xk7dRV3vxx8pL7/86F0ap1Zs4CqKcoeyQBgsZktBZA0FrgYSEwkBpSelN0SqOiWfyOAseE+OgAtzOyjcPlfwCV4IomcGTz09mIKi43eh7fgtKNyeH7GGgBenLnvz9Y2K42HRvTl1KNyktVU574kyq82zTNSufzkzlzWvxMzVm7j6NxsVmzZzZJNebTNSue0HjncflFPFq3fyfItuxj55Kcs3phX7ec7rmML1m3fyye/OJvzH3iXzzYE+1q3fW/SEomiOiok6VJgmJldFy5fDQw0s5EJdToAE4HWQBZwjplNL7efJcDFZjZXUn/gXjM7J1x3OvBzM/tKBc9/PXA9QG5ubr+xY8dWK468vDyys2vnVMGGoqKYv//mLvYkHJY9u0sKb60s4ps90zilYwqrd5awamcJTy4soEWa+M1pzchKbTg9E/87N17/nJfPtA1F/N8Aqxfx/ujtXZyUm8I1x6VXa/u/z8ln7uZi/jQ0k6ISY+HWYu6bls/Jh8X5QZ/95wqr6d946NCh082s4ikxEiT79N8RwGgzu1/SKcATknqbWQmApIHAbjObe6g7NrNRwCiA/v3725AhQ6rVwMmTJ1PdbRuqimKef6axdVcBu/KLOeMPk3hrZZBVeh57NBcN3NdFj784l39OWcEvPypi6DHtOf+4XM477rC6bH61+N+58Xrjizmkf7Ge7OzUehFv9pS3yGmfw5AhJ1Zr+5c2ziJz15ayWM4oKuG+aa8Sa9aSIUNO2a9uXf2No0wka4DES0M7hWWJriUY48DMpkjKAHKAjeH64cBT5fbZqZJ9ughIom12Om2zYdwNp/Ds9FWs35HPwO5t9qv3q4t706dLK578eCXPzVjNczNWl63LbZFO68w0zji6Hb+4sGddh+BcvZCWEuPDxZvZsbeQ9JQYBUUlNM9IrXS7PQXF/GbCfJ6bsZrObfZNqpqWEuP0HjnsSuIMxFEmkqlAD0ndCT7shwNXlquzEjgbGC2pJ5ABbAKQFAMuB04vrWxm6yTtkDSIYLD9W8BDEcbgKjCgexsGlEsgib7etxNf79uJBet28PPnZjN79XYAjmyXzUdLt7Bw/U5Gf7Ccw1s349jDmvOHy04sO4tlS14+W3cV0CO3eYX7NjOmLNnC8Z1aVumfz7n6drZh5zaZvPf5Zk781cSyU+pn33UeLSp5P3+yfCv//mglAAO6td1vXWo8RkFxSSTtrYrIEomZFUkaCbxOcGrv42Y2T9LdwDQzGw/cDPxV0k0Ef+9rEk7lPQNYVTpYn+B/2Hf676v4QHu91bNDC8aPPI3iEiOmoFezfU8hj7+/jKWbd7Fg3Q5enbueOWu206VNJgO6t2HMxyvZtDOf9s3TeWhEXwYese8fZvKijVzzj6kAtGuezjs/G1J2e2DnDq7+jNc9fs3JTF/xBRPnbeDxD5YBcM797zD2+kEcUcEkkqUKi4JEMX7kYE7otP9NtNLiMXbnF+9Xtn77XhZsKWZI7Ta/QpH+F5rZBIJTdBPL7kx4PB8YfIBtJwODKiifBvSu1Ya6SMUTZhJu2SyVm849GggurPrKQ++zYN0OVn+xhw+XbCmrt3FnPjf8ezr3/tcJ9D68BZ1aZzJj5bay9Zt25nPPKwv47dePr7tAnKsFqfEYg45oy6Aj2nLHRT356sPvM2/tDv74xmc8fOVJB9yudBqWeAUzc2emxVm6eRe3PDuLEzq14rAWGdz72kIWb9zLV8/aHfnZXP51ziVNPCZe/fHp7C4o4uOlW1m4fie/e20hXz2xIx1bZfD395bxvX8HJ/Gd3iOH9z7fDMDyey/imDteZXl4Pr1zB1OfL1eOxcTLPzyN8x94lx17Kx7j2F1QxMfhfVOg4kRyw5lHsmjDTl6YuZZx01bvt64uTgn2ROKSLjMthaHHtmfose256PgOtMxMpWWzVH50Vg9mrtrGw28vLrupUKkTO7dq8vMbuaqrz9fISiK3RQbvfraJH4yZwTm92nNETjZHtMsiKy2FXne+vl/9lNiXZ7Y65rDmvPKj0ykpMTbs3MuyTbvYvqeQL1YsqJMYPJG4eqVL233fnrLSUxh8VA6Dw4sbR3+wjNLcUY8/F5w7ZD88qwfb9xTyypx1vDJnXVl5Zlq87HH/rq3p0iaTI3KyDrifWEx0aNmMDi2Ds7omb1kUXaMTeCJxDUb5Wwd7f8RVTf1/pwzo3oYXfzCYpZt3UVRsLN+yi6WbdrFkUx55e4u4ZdgxBx2ITzZPJK5BksCSd7aja2AaQg9WUtmtf485rOLT3+srn0beNUhqEB8Nrj6oz4PtjYUnEtdgRTmrq2tc6vNge2PgicQ1SMG09sluhXMOPJG4BkpqCEOorj7wLxzR80TinGv0fEwtWp5IXIMkRFT30nHOHRpPJK5B8kNbrqr8pIzoeSJxzjV6ftZWtDyRuAbLj2y5qvD3SfQ8kbgGSZIfsHBV5h2SaHkicQ1SVB8MewuL+ccHy1ixxaeod66qPJG4husQjlmUlBh/eWcJ05ZvZcqSLRSXGCUVTEP/0qy1/Oql+fzutYW12VKXRN5zjZ5P2ugapEM9a2v5ll3836v7JwcJWmSk0qFlBu1T8plb8jn3TfwMgPxCnxGyMZGPtkfKE4lrkMShDaIWhb2P7PQUBnRvQ68OLTCML3YXMnfNdt5fU8S7qz8rq98qM22/7aev+IItefm8+/kmJsxZz9Bj2nP/5SfWRijONXiRJhJJw4AHgTjwNzO7t9z6LsA/gVZhnVvD+7wj6QTgL0ALoAQ42cz2ShoB/ILgC+la4JtmtjnKOFz9c6jfMEvCrPOHS0/gguM7fGn9pEmTOPX0MyguMS548D2KSvbvkXxn9FS27yksWx4/aw3bdheQkRrnhE4tMaBNZhqX9e/k337rGT9rK3qRJRJJceAR4FxgNTBV0ngzm59Q7Q5gnJk9KqkXMAHoJikF+DdwtZnNktQWKAzLHwR6mdlmSb8HRgJ3RRWHq78qutDs4bc/Z+zUVTRLjdOhVTNOPbIt3zvzyLLb8h7oQ14S6SnB3ehSYqKoeN++zYztewrp37U13+jXicy0OGM/WcXiTXms2LJ7vzva/fqV+Ryd25xBR7Th63070aVNJmkpPhTpGrcoeyQDgMVmthRA0ljgYiAxkRhBjwOgJUEPA+A8YLaZzQIwsy3hPlIJjmpkSdoSbrs4whhcPVXRoa0Roz5iytItAJzbK5c35m/g3c82saegmEFHtAUgHqu8t2DAK3PWse7PH9AmK53WmakADDmmHSMGdAHg4j6HY2bMXr2drPQ42emp/O29pWzOy+f9xVuYvuILHpm0BIBzeubSvkU6d1zUk517i8hKTyEzNU6sCm1pyoqKS9iVX0zL8PWvLr+yPXqKar4iSZcCw8zsunD5amCgmY1MqNMBmAi0BrKAc8xsuqQbgX5Ae6AdMNbMfp+w38eBXcDnwFAzK67g+a8HrgfIzc3tN3bs2GrFkZeXR3Z2/b3FZRQaQswPTN/LzE3FXHp0Ks1SRFwwel4BALecnEGvtnGWbi/m7il799vuxpPS6dP+y9+fEmMesyCfN1YUcVimSI2LnQXG3iLjhhPT6VvBthWZv6WYt1YWsnx7CTsKjIrG7q/tncbpnWr2IVkTdfF3LjFj3S4jPQ5ZqSIjvq9XOHdzMa8sLSAukZkafDlIjYuUGBQWwwdriwDo3DzGkS1jZKWKEoIz8PKLg5MlslNFz7Zx2jUTe4th2fZi2mfGOLp1jFj4PH+dnc+iL4q5q19JvX9f17aa/o2HDh063cz6V1Yv2YPtI4DRZna/pFOAJyT1Dtt1GnAysBt4S9J04F3g+0BfYCnwEHAbcE/5HZvZKGAUQP/+/W3IkCHVauDkyZOp7rYNVUOI+dPCz5j51uc8+1nhfuX3X3Yi3+jXCYAhwH9/zfjz5CWMencp2/cUcuagfvTt0vpL+0uMuTZCHwL8T/i4sLiEx99fxvode4lLtMpM5b6Jn7GqpDVDhlT6PxqZuvg7j/1kJbe/PqdsuVlqnPOOy+XB4X15/fk5LNm+miPaZbJmbxFpKTEK9paQX1TMF7v3/V3Tm2Uyd1sh2/cUUFhspMVjtGiWipmxZVcBLy0t/NLz5mSnM+HHp9G+eQbjN85kxZ6tZGfH6v37urbV1f9ylIlkDdA5YblTWJboWmAYgJlNkZQB5BCMqbxbOoguaQJwErAjrLskLB8H3BphDK6euunco7nxnB7kF5WQl1/Ett2F7NhbSN/OrfarJ4kfDD2KHww9ik0782nXPL3O25oaj3HDmUfuV/bG/A3sLWr8pxhv3R30En998XHsKihm4rz1vDx7HVt3FbB00y46tW7Gazee8aXtzIziEiMlHjto2dZdBbzz2Uby8ovJSovTPCOV3722kMUb8xj9wXJuGXasX0hSB6JMJFOBHpK6EySQ4cCV5eqsBM4GRkvqCWQAm4DXgVskZQIFwJnAn8L99JLUzsw2EQzkL4gwBlePSSIjNU5Gapyc7MoTRDKSyIFs3JnPrNXbeeDNzxjQvQ2nHpmT7CbVyI69hbw8ax0ZqTFaZKTSJjuNo3Obsys/ODw1YkAXUuIxTu+Rw4Nvfs6GnfnEYnDG0e0q3J8kUuKqtKxNVhpf79tpv7LBR7Wl152vM2/tjoRtayNKdyCRJRIzK5I0kiApxIHHzWyepLuBaWY2HrgZ+Kukmwi+N1xjwaDNF5L+SJCMDJhgZq8ASPoV8K6kQmAFcE1UMTgXlZO7tWH8rLU88ObnADzzvVM4uVubJLeq+l6etY5f/GfOAdeXnuRwXMeWjPpWtIfzMtNS6Ne1Ne98tolR7y7hPzPX0LFls0ifs6mLdIwkvCZkQrmyOxMezwcGH2DbfxOcAly+/DHgsdptqXN16w+XncDvLz2BKUu28N+jp3LZY1O4amAXzumZy4DubYhJGEaJwZa8fDJS47TJSiM1XvVTic2M/KISMlLjEUYSyC8Kznf5z/+cSkxiyaY8FqzbQVZ6Cr07tqzza2t6dWjB9BVf8NsJwWwGOc3TgS+PpbjakezBdueapNJrVoYe254bzjiCv7y7lDEfr2TMxysPut30O86hbRUO4wHc+9pC/vLOUlpnptKhZTOapcVpnZnKxp35bNixlx2784m9/RolFlw706JZKt1zsmibnYaAj5ZupXVWGo9edRLdcrIO+lyl05Yd0S6bls1SObHcWFVd+/Ulvbk7HJfZtruANllpfPLh+0ltU2PmicS5JLvtwp789PxjWLttD9NXfMHGnflAcDqsBCmxGFOWbuGN+Rt4Zvpqvldu4H7Ntj389d2lmAUD0SlxkRITf3lnKQDnH3cY67bvZW9hMSu37iY7PYX+3dpQsH0TXTt3RoKde4v4bMPOcBA8j71FJWzdVcD6HXuZvWZ7pYmk9DKC+nRpjCSy01PITvePuaj5K+xcPZAaj9G1bRZd21b8gf3tU7tx5C8mcO+rCxn17lJS4+LwVs1olZnG2ws3ApCVFvRyCkuMgvCMsHhM3PuNEyrcZ3BqaK8DtmnDjr0M/O1b3D9xEWM+WlFW3jozja27C5i7ZjvPff9UenZoUTZzQMxHtZskTyTONQDxmLhqYJdwzCPGpp35rNu+l0078zmuYwuOOaw5f7y8T1l9M+PBtz6nR/vm1X7OdtnpfHNQF1Zu3UN+YTH5YXJasimPzzfmAXDBg+/x9PWD+HjZ1rJ2uqbHE4lzDcRvvn58letK4sZzjq7R88Vi4p5LKn5OM+OGJ6Yzcf4Grhj1UcLz1ugpXQPlicQ5d8gkMepb/flg8WZenLmGddv30q9r67KTCFzT4onEOVdtg4/KYfBRDftiSldzPr+1c865GvFE4pxzrkY8kTjnnKsRTyTOOedqxBOJc865GvFE4pxzrkY8kTjnnKsRTyTOOedqRKWzdjZmkjYR3ASrOnKAzbXYnIbAY24amlrMTS1eqHnMXc2s4ttYJmgSiaQmJE0zs2hv6VbPeMxNQ1OLuanFC3UXsx/acs45VyOeSJxzztWIJ5LKjUp2A5LAY24amlrMTS1eqKOYfYzEOedcjXiPxDnnXI14InHOOVcjnkgOQNIwSYskLZZ0a7LbUxOSHpe0UdLchLI2kt6Q9Hn4u3VYLkn/L4x7tqSTErb5dlj/c0nfTkYsVSWps6RJkuZLmifpx2F5o41bUoakTyTNCmP+VVjeXdLHYWxPS0oLy9PD5cXh+m4J+7otLF8k6fzkRFQ1kuKSPpX0crjcqOMFkLRc0hxJMyVNC8uS9942M/8p9wPEgSXAEUAaMAvolex21SCeM4CTgLkJZb8Hbg0f3wr8Lnx8IfAqIGAQ8HFY3gZYGv5uHT5unezYDhJzB+Ck8HFz4DOgV2OOO2x7dvg4Ffg4jGUcMDwsfwz4fvj4f4DHwsfDgafDx73C93w60D38X4gnO76DxP0T4Eng5XC5Uccbtnk5kFOuLGnvbe+RVGwAsNjMlppZATAWuDjJbao2M3sX2Fqu+GLgn+HjfwKXJJT/ywIfAa0kdQDOB94ws61m9gXwBjAs+tZXj5mtM7MZ4eOdwALgcBpx3GHb88LF1PDHgLOAZ8Py8jGXvhbPAmdLUlg+1szyzWwZsJjgf6LekdQJuAj4W7gsGnG8lUjae9sTScUOB1YlLK8OyxqTXDNbFz5eD+SGjw8Ue4N9TcJDGH0JvqE36rjDwzwzgY0EHwxLgG1mVhRWSWx/WWzh+u1AWxpWzA8AtwAl4XJbGne8pQyYKGm6pOvDsqS9t1Oqs5FrXMzMJDXK88AlZQPPATea2Y7gC2igMcZtZsVAH0mtgP8Axya5SZGR9BVgo5lNlzQk2e2pY6eZ2RpJ7YE3JC1MXFnX723vkVRsDdA5YblTWNaYbAi7t4S/N4blB4q9wb0mklIJksgYM3s+LG70cQOY2TZgEnAKwaGM0i+Nie0viy1c3xLYQsOJeTDwNUnLCQ4/nwU8SOONt4yZrQl/byT4wjCAJL63PZFUbCrQIzz7I41gYG58kttU28YDpWdpfBt4MaH8W+GZHoOA7WF3+XXgPEmtw7NBzgvL6qXw2PffgQVm9seEVY02bkntwp4IkpoB5xKMDU0CLg2rlY+59LW4FHjbglHY8cDw8Cyn7kAP4JO6iaLqzOw2M+tkZt0I/kffNrOraKTxlpKUJal56WOC9+RckvneTvbZB/X1h+BMh88IjjHfnuz21DCWp4B1QCHBcdBrCY4NvwV8DrwJtAnrCngkjHsO0D9hP98hGIhcDPx3suOqJObTCI4jzwZmhj8XNua4gROAT8OY5wJ3huVHEHwwLgaeAdLD8oxweXG4/oiEfd0evhaLgAuSHVsVYh/CvrO2GnW8YXyzwp95pZ9PyXxv+xQpzjnnasQPbTnnnKsRTyTOOedqxBOJc865GvFE4pxzrkY8kTjnnKsRTySuUZDUNpwJdaak9ZLWJCynVXEf/5B0TCV1fiDpqlpq8/uS+kiKqZZnmJb0HUmHJSxXGptz1eWn/7pGR9JdQJ6Z3VeuXATv+ZIKN6xjkt4HRhJc87HZzFod4vZxC6ZEOeC+zWxmzVvq3MF5j8Q1apKOUnBPkjEEF291kDRK0jQF9+y4M6FuaQ8hRdI2SfcquLfHlHBOIyTdI+nGhPr3KrgHyCJJp4blWZKeC5/32fC5+hykmfcCzcPe07/CfXw73O9MSX8Oey2l7XpA0mxggKRfSZoqaa6kx8Krl68A+gBPl/bISmML9/1NBfeymCvpt2HZwWIeHtadJWlSLf+JXCPgicQ1BccCfzKzXhbMUXSrmfUHTgTOldSrgm1aAu+Y2YnAFIIrgCsiMxsA/AwoTUo/BNabWS/g1wQzDx/MrcBOM+tjZt+S1Bv4OnCqmfUhmFx1eEK73jWzE8xsCvCgmZ0MHB+uG2ZmTxNcyX9FuM+CssYG067fAwwN2zVYweSHB4v5f4Gzw/KvVxKLa4I8kbimYImZTUtYHiFpBjAD6ElwY6Py9pjZq+Hj6UC3A+z7+QrqnEYwiSBmVjqNxaE4BzgZmKZgSvgzgSPDdQUEk/SVOlvSJwTTZZwJHFfJvgcSzDG12cwKCW4IdUa47kAxfwD8S9J1+GeGq4BPI++agl2lDyT1AH4MDDCzbZL+TTAHU3kFCY+LOfD/Sn4V6hwqAY+b2S/3KwxmrN1jpRMoSZnAwwR3glwj6R4qjqWqDhTzdwkS0FeAGZL6WnAjJOcA/3bhmp4WwE5gh/bdJa62fQBcDiDpeCru8ZSx8CZM2jf1+ZvA5ZJywvK2krpUsGkzghs6bQ5ng/1GwrqdBLcYLu9jYGi4z9JDZu9UEs8RFtxZ75fAF9T/mz65OuY9EtfUzADmAwuBFQQf+rXtIYJDQfPD55pPcDe+g/k7MFvStHCc5FfAm5JiBLM2fw9Ym7iBmW2R9M9w/+sIkkSpfwB/k7SHhNvGmtlqSb8EJhP0fF4ys1cSklhF/qRgenUBE81sbiWxuCbGT/91rpaFH8opZrY3PJQ2Eehh+27/6lyj4j0S52pfNvBWmFAE3OBJxDVm3iNxzjlXIz7Y7pxzrkY8kTjnnKsRTyTOOedqxBOJc865GvFE4pxzrkb+P979fGqoHt9/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')\n",
    "\n",
    "plt.title('Mini-Batch Gradient Descent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient descent with Momentum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer the book for the detailed explanation:\n",
    "\n",
    "We learned how momentum helps us to reduce oscillations in the gradient steps and attain the convergence faster. The update equation of momentum is given as:\n",
    "\n",
    "$$v_t = \\gamma v_{t-1} + \\eta \\nabla_\\theta J( \\theta) -- (8) $$\n",
    "\n",
    "Update theta by: \n",
    "$${\\theta = \\theta - v_t } -- (9) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def Momentum(data, theta, lr = 1e-2, gamma = 0.9, num_iterations = 5000):\n",
    "    \n",
    "    loss = []\n",
    "    \n",
    "    #Initialize vt with zeros:\n",
    "    vt = np.zeros(theta.shape[0])\n",
    "    \n",
    "    \n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients with respect to theta \n",
    "        gradients = compute_gradients(data, theta)\n",
    "        \n",
    "        #Update vt by equation (8)\n",
    "        vt = gamma * vt + lr * gradients\n",
    "        \n",
    "        #update model parameter theta by equation (9)\n",
    "        theta = theta - vt\n",
    "    \n",
    "        #store loss of every iteration\n",
    "        loss.append(loss_function(data,theta))\n",
    "        \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform gradient descent with mometum and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = Momentum(data, theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Mometum')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4FeX5//H3JztZCEsg7JuAgCAoCCIuoKhoLVhLrWhd6oLfKq1Vf1Zt/drq17a2Vm3da9Vq3dC6VKoobuAuIvuuLMq+bwkQQsj9+2MGPMZAQpKTk5xzv65rrsw888yc+4ZwbmZ7RmaGc845V1VJsQ7AOedc/eaFxDnnXLV4IXHOOVctXkicc85VixcS55xz1eKFxDnnXLV4IXHOOVctXkicC0n6SlKxpLwy7dMlmaQOtRjLYEkrauvznKsOLyTOfdtSYNTeBUm9gMzYheNc3eeFxLlvexK4IGL5QuBfexck5Ur6l6T1kr6WdJOkpHDdRZI+knS3pC2Slkg6JmxfLmmdpAsj9pUu6S+SlklaK+khSQ0kZQGvA60kFYZTK0mPS7otYvtvHbWER1TXSZolabukRyXlS3pdUoGktyU1juKfnUtQXkic+7ZPgYaSuktKBs4BnopYfy+QC3QCTiAoOj+NWD8AmAU0BZ4BxgJHAZ2BnwD3ScoO+94OdAX6hOtbAzeb2XbgNGCVmWWH06pKxv9D4ORwv98nKEi/BpoR/Hv/RSX341yleSFx7rv2HpWcDMwHVobtewvLjWZWYGZfAXcC50dsu9TM/mlme4DngLbArWa2y8zeBIqBzpIEjAauNrNNZlYA/CHcf3Xca2ZrzWwl8AEw2cymm1kR8DJwRDX379x3pMQ6AOfqoCeB94GORJzWAvKAVODriLavCY4k9lobMb8TwMzKtmUTHCFkAlODmgKACIpVdZT9rPI+27ka5UckzpVhZl8TXHQ/HXgpYtUGYDfQPqKtHd8csRyMDQRf7IeZWaNwyjWzvV/05Q3LvZ1vX/hvUYXPda7GeSFxrnyXACeG1yv22gM8D/xeUo6k9sA1fPsaSqWYWSnwD+BuSc0BJLWWdGrYZS3QVFJuxGYzgNMlNZHUAvjlQWflXBR4IXGuHGa22Mw+L2fVzwmODJYAHxJcUH+sih9zPbAI+FTSNuBt4NDw8xcAzwJLwjvAWhGccpsJfAW8SXANxrmYk7/YyjnnXHX4EYlzzrlq8ULinHOuWryQOOecq5aoFhJJwyQtlLRI0g3lrG8naWI4KN4sSaeH7amSnpA0W9J8STeG7W3D/vMkzZV0VTTjd845V7GoXWwPh5f4guDp4BXAFGCUmc2L6PMwMN3MHpTUAxhvZh0knQsMN7NzJGUC84DBwC6gpZlNk5QDTAXOjNxnefLy8qxDhw5VymP79u1kZWVVadv6ynNODImWc6LlC9XPeerUqRvMrFlF/aL5ZHt/YJGZLQGQNBYYQVAU9jKgYTifC6yKaM+SlAI0IBhWYpuZbQJWA5hZgaT5BE8VH7CQdOjQgc8/L+9OzopNmjSJwYMHV2nb+spzTgyJlnOi5QvVz1nS1xX3iu4RyUhgmJldGi6fDwwwszERfVoS3A/fGMgChprZVEmpBPfMn0TwJO/VZvZwmf13IBjGoqeZbSvn80cTjGVEfn5+37Fjx1Ypj8LCQrKzE2tUCc85MSRazomWL1Q/5yFDhkw1s34V9Yv1WFujgMfN7E5JA4EnJfUkOJrZA7QiKDIfSHo74ugmG3gR+GV5RQQgLDwPA/Tr18+qWpX9fzGJwXOOf4mWL9ReztG82L6SYOTTvdrw3TGJLiEYcgIz+wTIIBgY71zgDTPbbWbrgI+AfhBciCcoIk+b2Us455yLqWgWkilAF0kdJaURDI89rkyfZQSnr5DUnaCQrA/bTwzbs4CjgQXh0NuPAvPN7K4oxu6cc66SolZIzKwEGANMIHinw/NmNlfSrZKGh92uBS6TNJNgXKGLLLhocz+QLWkuQUH6p5nNAgYRvPvhREkzwun0aOXgnHOuYlG9RmJm44HxZdpujpifR1Acym5XCPyonPYPCd7Z4Jxzro7wJ9udc85VixeSA3jq06+Zs2FPrMNwzrk6zQvJfhSXlPL05GXcM62IlVt2xjoc55yrs7yQ7EdaShL/uKAvpQZ/f29xrMNxzrk6ywvJAbRpnEn/lim8OHUFBUW7Yx2Oc87VSV5IKnBSuxS2F+/h5elln6V0zjkHXkgq1Ck3icNaNeSZycvw1xI759x3eSGpgCTOHdCOBWsKmL58S6zDcc65OscLSSUM792KzLRknp28LNahOOdcneOFpBJyMlIZ0acV/521iq07/aK7c85F8kJSSaP6t6NodymvzPCL7s45F8kLSSUd3qYRPVv7RXfnnCvLC8lBGNXfL7o751xZXkgOwog+rclMS+YZv+junHP7eCE5CNnpKYzo04pX/aK7c87t44XkIJ3bvz1Fu0v5jz/p7pxzgBeSg9arTS49Wzfk2c/8ortzzoEXkio5t397FqwpYNoyv+junHNRLSSShklaKGmRpBvKWd9O0kRJ0yXN2vv+dUmpkp6QNFvSfEk3VnaftWF4n1ZkpSXz7Gd+0d0556JWSCQlA/cDpwE9gFGSepTpdhPwvJkdAZwDPBC2/whIN7NeQF/gckkdKrnPqMtOT2F4n9Z+0d0554juEUl/YJGZLTGzYmAsMKJMHwMahvO5wKqI9ixJKUADoBjYVsl91opzwyfd/aK7cy7RpURx362B5RHLK4ABZfr8DnhT0s+BLGBo2P4CQYFYDWQCV5vZJkmV2ScAkkYDowHy8/OZNGlSlZIoLCzc77YdGibxj3fn0W7XUiRVaf910YFyjleec/xLtHyh9nKOZiGpjFHA42Z2p6SBwJOSehIceewBWgGNgQ8kvX0wOzazh4GHAfr162eDBw+uUoCTJk1if9uObrCMX788m4ad+tC3feMq7b8uOlDO8cpzjn+Jli/UXs7RPLW1EmgbsdwmbIt0CfA8gJl9AmQAecC5wBtmttvM1gEfAf0quc9as/eiuz/p7pxLZNEsJFOALpI6SkojuJg+rkyfZcBJAJK6ExSS9WH7iWF7FnA0sKCS+6w12ekpjDgivOi+wy+6O+cSU9QKiZmVAGOACcB8gruz5kq6VdLwsNu1wGWSZgLPAhdZ8JTf/UC2pLkExeOfZjZrf/uMVg6VcW7/duwqKeWFaStiGYZzzsVMVK+RmNl4YHyZtpsj5ucBg8rZrpDgFuBK7TOWerbOpW/7xjzx8VdcdEwHkpPi56K7c85Vhj/ZXgN+OqgDyzbtYOKCdbEOxTnnap0Xkhpw6mEtaNEwg39+vDTWoTjnXK3zQlIDUpOTOH9gez5atJGFawpiHY5zztUqLyQ15Nz+7UhPSeJxPypxziUYLyQ1pHFWGj84ojUvTVvJ5u3FsQ7HOedqjReSGnTRoA7sKinl2Sn+gKJzLnF4IalB3Vo05JhDmvLkJ1+ze09prMNxzrla4YWkhv10UEdWby1iwtw1sQ7FOedqhReSGnZit+a0a5LJIx8s9VfxOucSgheSGpacJC49riMzlm9hylebYx2Oc85FnReSKPhR37Y0zkzl4fcXxzoU55yLOi8kUdAgLZkLBnbg7fnr+HKtP6DonItvXkii5IKB7UlPSeIfHyyJdSjOORdVXkiipGl2Omf3a8t/pq9i7baiWIfjnHNR44Ukii49riMlpaX886OvYh2Kc85FjReSKGrfNIthPVvw9OSvKdxVEutwnHMuKryQRNnlxx9CQVEJYz/zYVOcc/HJC0mU9W7biAEdm/DIB0spLvFhU5xz8ccLSS24Ykhn1mwr4kV/r7tzLg5FtZBIGiZpoaRFkm4oZ307SRMlTZc0S9LpYft5kmZETKWS+oTrRkmaHfZ/Q1JeNHOoCcd3yePwNrk8MGkRJT6Yo3MuzkStkEhKBu4HTgN6AKMk9SjT7SbgeTM7AjgHeADAzJ42sz5m1gc4H1hqZjMkpQB/A4aY2eHALGBMtHKoKZL4+YldWL5pJ+Nmrop1OM45V6OieUTSH1hkZkvMrBgYC4wo08eAhuF8LlDet+yocFsAhVOWJIXb1otv5qHdm9OtRQ73T1zEnlIfzNE5Fz8UrRFqJY0EhpnZpeHy+cAAMxsT0acl8CbQGMgChprZ1DL7WQyMMLM5Eft9DNgOfElwdLKnnM8fDYwGyM/P7zt27NiyXSqlsLCQ7OzsKm1b1mdrSnhgxi6u6J1O/5YpNbLPaKjJnOsLzzn+JVq+UP2chwwZMtXM+lXY0cyiMgEjgUcils8H7ivT5xrg2nB+IDAPSIpYPwCYHbGcCrwDHEJwZHIfcFNFsfTt29eqauLEiVXetqySPaV24l8m2ql3v2d79pTW2H5rWk3mXF94zvEv0fI1q37OwOdWie/7aJ7aWgm0jVhuE7ZFugR4HsDMPgEygMiL5+cAz0Ys9wn7Lg6TfB44pmbDjp7kJHHlkM4sWFPAOwvWxToc55yrEdEsJFOALpI6SkojKArjyvRZBpwEIKk7QSFZHy4nAWfzzfURCApRD0nNwuWTgflRyyAKhvduRbsmmdz77pf+4ivnXFyIWiExsxKCO6omEHzZP29mcyXdKml42O1a4DJJMwmOPC6yb75djweWm9mSiH2uAm4B3pc0i+AI5Q/RyiEaUpKTuGLwIcxasZVJX6yPdTjOOVdtUb3ia2bjgfFl2m6OmJ8HDNrPtpOAo8tpfwh4qEYDrWVnHdmG+yct4q43v2Bw12YEN6A551z95E+2x0BaShJXndSV2Su38ua8tbEOxznnqsULSYyc2acVnfKyuOvNLyj150qcc/WYF5IYSUlO4pcnd2Xh2gJenb061uE451yVeSGJoTN6teTQ/Bz++tYXPgaXc67e8kISQ0lJ4ppTurJkw3Zenl72ERvnnKsfvJDE2Ck98unVOpe/vfOlv6/EOVcveSGJMSk4KlmxeSfPfb481uE459xB80JSBwzu2ox+7RtzzztfsqPY3+3unKtfvJDUAZK48fRurC/YxSMfLI11OM45d1C8kNQRfds3YdhhLfj7e4vZULgr1uE451yleSGpQ64bdihFJaXc886XsQ7FOecqzQtJHXJIs2xG9W/LM5OXsWR9YazDcc65SvFCUsdcdVJX0lKSuGPCwliH4pxzleKFpI5plpPO5ccfwutz1jBt2eZYh+OccxXyQlIHXXpcR5rlpPPH8fP95VfOuTrPC0kdlJWewtVDuzLlq828PmdNrMNxzrkD8kJSR/34qLZ0a5HD71+bT9HuPbEOxznn9ssLSR2VnCR+N/wwVm7ZycPvL6l4A+eci5GoFhJJwyQtlLRI0g3lrG8naaKk6ZJmSTo9bD9P0oyIqVRSn3BdmqSHJX0haYGkH0Yzh1g6ulNTvterJQ9MWsSqLTtjHY5zzpUraoVEUjJwP3Aa0AMYJalHmW43Ac+b2RHAOcADAGb2tJn1MbM+wPnAUjObEW7zG2CdmXUN9/tetHKoC244rRtm8Kc3FsQ6FOecK1c0j0j6A4vMbImZFQNjgRFl+hjQMJzPBVaVs59R4bZ7XQz8EcDMSs1sQ41GXce0bZLJ5cd34pUZq/j8q02xDsc5575D0bq9VNJIYJiZXRounw8MMLMxEX1aAm8CjYEsYKiZTS2zn8XACDObI6kRMBv4NzAYWAyMMbO15Xz+aGA0QH5+ft+xY8eW7VIphYWFZGdnV2nbmrKrxLjhg500TBe/HZhBkhTVz6sLOdc2zzn+JVq+UP2chwwZMtXM+lXY0cyiMgEjgUcils8H7ivT5xrg2nB+IDAPSIpYPwCYHbGcR3AUMzJi+ycriqVv375WVRMnTqzytjXpP9NXWPvrX7VnJn8d9c+qKznXJs85/iVavmbVzxn43CrxfR/NU1srgbYRy23CtkiXAM8DmNknQEZYLPY6B3g2YnkjsAN4KVz+N3BkzYVcdw3v3Yr+HZrwpzcWsGl7cazDcc65faJZSKYAXSR1lJRGUBTGlemzDDgJQFJ3gkKyPlxOAs4m4vpIWCH/S3Bai3DbedFLoe6QxG0/6ElhUQl/HD8/1uE459w+USskZlYCjAEmAPMJ7s6aK+lWScPDbtcCl0maSXDkcVFYLACOB5abWdmHKK4HfidpFsHpsmujlUNd0zU/h0uP68S/p65gil94d87VESnR3LmZjQfGl2m7OWJ+HjBoP9tOAo4up/1rgiKTkH5xUmf+O3MVN708h1d/cSypyf5MqXMutvxbqJ7JTEvhd8MPY+HaAh770F/L65yLPS8k9dDJPfI5uUc+f337S1Zs3hHrcJxzCc4LST31u+GHBT/HzfOh5p1zMeWFpJ5q3agBV5/chbfnr+W12atjHY5zLoF5IanHLh7UkcPb5PLbV+b6syXOuZjxQlKPpSQn8eeRh7OtaDe3/ndurMNxziUoLyT1XLcWDblicGf+M2MV7y74zpBjzjkXdV5I4sCVQzrTNT+bX780h21Fu2MdjnMuwXghiQNpKUn8eWRv1hUU8cfx/t4S51zt8kISJ/q0bcQlx3bk2c+W8dGiuH5Fi3OujvFCEkeuOflQOuVlcd2/Z7J1p5/ics7VjkoVEklPVqbNxVaDtGTuPLs3awt2ccs4v4vLOVc7KntEcljkQvg+9r41H46rriPaNebKIZ15afpKXvcHFZ1zteCAhUTSjZIKgMMlbQunAmAd8EqtROgO2s9P7MzhbXL59cuzWbetKNbhOOfi3AELiZn90cxygDvMrGE45ZhZUzO7sZZidAcpNTmJu87uw47iPVz/4iwfi8s5F1WVPbX1qqQsAEk/kXSXpPZRjMtVU+fm2dx4WjcmLlzPM58ti3U4zrk4VtlC8iCwQ1JvgjcSLgb+FbWoXI24YGAHjuuSx22vzmfRuoJYh+Oci1OVLSQl4StwRwD3mdn9QE70wnI1ISlJ/OVHvclMS2bMM9Mp2r0n1iE55+JQZQtJgaQbCd6R/pqkJCC1oo0kDZO0UNIiSTeUs76dpImSpkuaJen0sP08STMiplJJfcpsO07SnErGn7DyG2Zw59m9WbCmgNtemxfrcJxzcaiyheTHwC7gYjNbA7QB7jjQBuEtwvcDpwE9gFGSepTpdhPwvJkdAZwDPABgZk+bWR8z60NQvJaa2YyIfZ8FFFYy9oQ3+NDmXH58J576dJnfEuycq3GVKiRh8XgayJV0BlBkZhVdI+kPLDKzJWZWDIwlODX2rV0DDcP5XGBVOfsZFW4LgKRs4BrgtsrE7gLXnnIovds24lcvzmL5Jn89r3Ou5qgyt4ZKOpvgCGQSIOA44Doze+EA24wEhpnZpeHy+cAAMxsT0acl8CbQGMgChprZ1DL7WQyMMLM54fLdwPvAdOBVM+u5n88fDYwGyM/P7zt27NjyulWosLCQ7OzsKm1b16zbUcpvP95Jq6wkbhyQQUqSyu0XTzlXlucc/xItX6h+zkOGDJlqZv0q7GhmFU7ATKB5xHIzYGYF24wEHolYPp/gQn1kn2uAa8P5gcA8ICli/QBgdsRyH2BcON8BmFOZ+Pv27WtVNXHixCpvWxeNm7HS2l//qt326tz99om3nCvDc45/iZavWfVzBj63SnzHVvYaSZKZrYtY3kjFp8VWAm0jltuEbZEuAZ4HMLNPgAwgL2L9OcCzEcsDgX6SvgI+BLpKmlS5FBzA93u34vyj2/OPD5Yy3q+XOOdqQGULyRuSJki6SNJFwGvA+Aq2mQJ0kdRRUhpBURhXps8y4CQASd0JCsn6cDkJOJuI6yNm9qCZtTKzDsCxwBdmNriSObjQ/57RgyPaNeK6f8/050ucc9VW0VhbnSUNMrPrgL8Dh4fTJ8DDB9rWzEqAMcAEYD7B3VlzJd0qaXjY7VrgMkkzCY48LgoPpwCOB5ab2ZIq5ub2Iy0liQfOO5IGacmMfnIqBf5WRedcNVR0RPJXYBuAmb1kZteY2TXAy+G6AzKz8WbW1cwOMbPfh203m9m4cH6emQ0ys94W3O77ZsS2k8zs6APs+yvbz4V2V7GWuQ24d9SRfL1xB796wcfjcs5VXUWFJN/MZpdtDNs6RCUiV2sGHtKU64cdyutz1vCPD/zAzzlXNRUVkkYHWNegJgNxsXHZcZ04vVcLbn99Ae9/sT7W4Tjn6qGKCsnnki4r2yjpUmBqOf1dPSOJO0b2pmt+Dlc+M41F63zAAOfcwamokPwS+KmkSZLuDKf3CG7bvSr64bnakJWewiMX9iM9JYlLn5hCYbFfL3HOVV5FL7Zaa2bHALcAX4XTLWY20IJhU1ycaNM4k7+f35dVW4q4f0YRu/eUxjok51w9Udmxtiaa2b3h9G60g3Kx0bd9E/54Vi/mbyrllv/OjXU4zrl6orIPJLoE8cO+bTi9YypPfbqMJz7+KtbhOOfqgZRYB+DqnpFdUynOaMIt/51Ly9wMTjmsRaxDcs7VYX5E4r4jSeKeUX3o1aYRvxg7nWnLNsc6JOdcHeaFxJUrMy2FRy/sR37DDC55fApL1vttwc658nkhcfuVl53OEz/tT5LEhf/8jPUFu2IdknOuDvJC4g6oQ14Wj150FBsKirn48Sls31US65Ccc3WMFxJXoT5tG3HfuUcwd9VWLn9yKkW798Q6JOdcHeKFxFXKSd3z+fPI3ny4aAO/eHY6Jf7AonMu5IXEVdrIvm24ZfhhvDlvLde9MIvSUh9KxTnnz5G4g3ThMR0o3FXCHRMWkpWezP+N6ImkWIflnIshLyTuoF0x+BAKikp46L3FZKWncMOwbl5MnEtgXkjcQZPE9cMOpXDXbv7+3hLSU5K5emgXLybOJaioXiORNEzSQkmLJN1Qzvp2kiZKmi5plqTTw/bzJM2ImEol9ZGUKek1SQskzZV0ezTjd/sniVuH9+Tsfm24550vufutL/x1vc4lqKgdkUhKBu4HTgZWAFMkjTOzeRHdbgKeN7MHJfUAxgMdzOxp4OlwP72A/5jZDEmZwF/MbKKkNOAdSaeZ2evRysPtX1KSuP2sw4MhVd5dxB4z/t8ph/qRiXMJJpqntvoDi8xsCYCkscAIILKQGNAwnM8FVpWzn1HAWAAz2wFMDOeLJU0D2kQlelcpSUniDz/ohQT3T1xMqcGvTvVi4lwiiWYhaQ0sj1heAQwo0+d3wJuSfg5kAUPL2c+PCQrQt0hqBHwf+FtNBOuqLilJ/P7MXkjiwUmLKTXzC/DOJRBF67y2pJHAMDO7NFw+HxhgZmMi+lwTxnCnpIHAo0BPMysN1w8AHjGzXmX2nQL8F5hgZn/dz+ePBkYD5Ofn9x07dmyV8igsLCQ7O7tK29ZXVc251Iyn5hXz7vISTmqXwnnd00iqJ8XE/57jX6LlC9XPeciQIVPNrF+FHc0sKhMwkOCLfu/yjcCNZfrMBdpGLC8Bmkcs3w38upx9PwbcU9lY+vbta1U1ceLEKm9bX1Un59LSUvv9a/Os/fWv2i+enWbFJXtqLrAo8r/n+Jdo+ZpVP2fgc6vEd2w0T21NAbpI6gisBM4Bzi3TZxlwEvC4pO5ABrAeQFIScDZwXOQGkm4juJ5yaRRjd1UkiRtP60Zug1TumLCQgqISHjjvSDJSk2MdmnMuSqJ2+6+ZlQBjgAnAfIK7s+ZKulXS8LDbtcBlkmYCzwIXhVUQ4HhguYUX6wEktQF+A/QApoW3BntBqWMkceWQztx2Zk8mLlzHBY99xrai3bEOyzkXJVF9INHMxhPc0hvZdnPE/Dxg0H62nQQcXaZtBVA/Tro7fnJ0e3IbpHL1czMY9fCn/POnR9E8JyPWYTnnapgP2uii6vu9W/GPC/uxZP12fnD/x3y5tiDWITnnapgXEhd1Qw5tzvOXD6R4TylnPfgxHy/eEOuQnHM1yAuJqxW92uTy8hXH0KJhBhc+9hkvT18R65CcczXEC4mrNW0aZ/LCz46hX/smXP3cTO5550sfn8u5OOCFxNWq3AapPHFxf846ojV3vfUFV42dwc5if3Wvc/WZDyPval1aShJ3nt2bzvnZ3DFhIYvXF/LwBf1o3ahBrENzzlWBH5G4mJDEFYM78+iF/Vi2cQfD7/2Qz5ZuinVYzrkq8ELiYurEbvm8fOUgchukcu4/PuXpyV/HOiTn3EHyQuJirnPzbF6+chDHdsnjNy/P4bp/z/TrJs7VI15IXJ2Q2yCVRy88ip+f2JkXpq3gzPs/YvH6wliH5ZyrBC8krs5IThLXnnIoj/+0P+sLdzH83g95ZcbKWIflnKuAFxJX55zQtRmv/eJYurdsyFVjZ/Cbl2dTtNtPdTlXV3khcXVSy9wGPDv6aC4/oRNPT17Gmfd/xMI1Pk6Xc3WRFxJXZ6UmJ3Hjad157KJ+bCjcxffv+5DHPlxKaak/De9cXeKFxNV5J3bL541fHs+xnfO49dV5XPT4FNZtK4p1WM65kBcSVy/kZafz6IX9uO3Mnny2dCOn/vV93pizJtZhOefwQuLqEUn85Oj2vPrz42jduAH/89RUfvHsdDZtL451aM4lNC8krt7p3Dybl68YxDUnd+X1Oas5+a73eHXWKh9J2LkY8ULi6qXU5CR+cVIXXv35cbRp3IAxz0znZ09NY12BXztxrrZFtZBIGiZpoaRFkm4oZ307SRMlTZc0S9LpYft5kmZETKWS+oTr+kqaHe7zHkn+DvcEdmiLHF782THceFo33l24jpPvep/npizzO7ucq0VRKySSkoH7gdOAHsAoST3KdLsJeN7MjgDOAR4AMLOnzayPmfUBzgeWmtmMcJsHgcuALuE0LFo5uPohJTmJy084hNevOo5D83O4/sXZjHzoY+at2hbr0JxLCNE8IukPLDKzJWZWDIwFRpTpY0DDcD4XWFXOfkaF2yKpJdDQzD614IT4v4AzoxG8q38OaZbNc5cfzV9+1JuvN+7g+/d9yK3/nUdB0e5Yh+ZcXFO0LlBKGgkMM7NLw+XzgQFmNiaiT0vgTaAxkAUMNbOpZfazGBhhZnMk9QNuN7Oh4brjgOvN7IxyPn80MBogPz+/79ixY6uUR2FhIdnZ2VXatr6Kh5wLi40Xvyxm0vISctPFqG5p9G+RzP7OhMZDzgcr0XJOtHyh+jkPGTL/HpINAAAUWklEQVRkqpn1q6hfrN+QOAp43MzulDQQeFJSTzMrBZA0ANhhZnMOdsdm9jDwMEC/fv1s8ODBVQpw0qRJVHXb+ipecj7jFJixfAs3/Wc2D87cxudbG3PT93rQu22j7/SNl5wPRqLlnGj5Qu3lHM1TWyuBthHLbcK2SJcAzwOY2SdABpAXsf4c4Nky+2xTwT6d26dP20a8cuWx/PGsXizdsJ0R93/E1c/NYPXWnbEOzbm4Ec1CMgXoIqmjpDSCojCuTJ9lwEkAkroTFJL14XIScDbh9REAM1sNbJN0dHi31gXAK1HMwcWB5CQxqn87Jv6/wVwx+BBem72aIX+ZxF1vLmT7rpJYh+dcvRe1QmJmJcAYYAIwn+DurLmSbpU0POx2LXCZpJkERx4X2TcXbY4HlpvZkjK7vgJ4BFgELAZej1YOLr7kZKTyq2HdeOeaExjaPZ973l3ECXdM4omPv2K33y7sXJVF9RqJmY0Hxpdpuzlifh4waD/bTgKOLqf9c6BnjQbqEkrbJpncd+6R/HTQZv70xgJ+O24uTTPElobL+cERrUlJ9ud0nTsY/i/GJay+7Rvz3Oij+dfF/clJE9e9MItT//o+r81a7Q80OncQvJC4hCaJ47s247cDM3joJ0eSJHHlM9P43r0f8tqs1ezxguJchbyQOEdQUIb1bMkbvzyeu87uza6SPVz5zDROufs9Xpy6gt17SmMdonN1lhcS5yIkJ4mzjmzDW1efwH3nHkFqchLX/nsmQ/4yiacnf82uEn93vHNleSFxrhzJSeKMw1vx+lXH8eiF/cjLTuc3L8/huD9N5P6Ji9js70Bxbp9YP9nuXJ0miZO653Nit+Z8vHgjf39/CXdMWMi9737JWUe24eJBHencPLGG3XCuLC8kzlWCJAZ1zmNQ5zy+WFvAYx8u5YWpK3hm8jKGHNqMS47txKDOTfc7lpdz8cxPbTl3kLrm53D7Dw/n4xtO5OqhXZm9cis/eXQyQ+96j0c/XMqWHX7ayyUWLyTOVVFedjpXDe3Ch9efyF9+1JuGDVL5v1fnMeAP73Dt8zOZtmyzv/7XJQQ/teVcNWWkJjOybxtG9m3D3FVbeWbyMv4zfSUvTltB95YNOXdAO4Yf3orczNRYh+pcVPgRiXM16LBWufz+B72Y/Juh/P4HwUg+//ufORz1h7e58ulpvLtgLSX+TIqLM35E4lwUZKencN6A9pzbvx1zV23jhakreGXGSl6bvZq87HTO7NOKH/ZtQ/eWDSvemXN1nBcS56JIEj1b59KzdS6/Pr07Exeu48WpK3j846945MOldG/ZkDMOb8kZh7ekfdOsWIfrXJV4IXGulqSlJHHqYS049bAWbNpezLgZK3ll5irumLCQOyYspFfrXL53eEu+16slbZtkxjpc5yrNC4lzMdAkK42LBnXkokEdWbF5B+Nnr+a1Wau5/fUF3P76Anq3bcQZvVpyco98OuT5kYqr27yQOBdjbRpnMvr4Qxh9/CEs37SDV2et5rXZq/j9+Pn8fvx8ujTPZmiPfE7ukU+fNo1ISvKHHl3d4oXEuTqkbZNMfjb4EH42OCgqb81by9vz1/Lw+0t4cNJi8rLTGdq9OUO75zOocx4N0pJjHbJzXkicq6vaNsnk4mM7cvGxHdm6YzeTvljHm/PW8uqs1Yydspy05CSO6tiY47s04/iuzejWIseHaHExEdVCImkY8DcgGXjEzG4vs74d8ATQKOxzQ/h6XiQdDvwdaAiUAkeZWZGkUcCvAQNWAT8xsw3RzMO5WMvNTGVEn9aM6NOa4pJSJi/dyHsL1/P+l+v54+sL+OPrC2iWk85xXfI4oWszju2cR9Ps9FiH7RJE1AqJpGTgfuBkYAUwRdK48D3te90EPG9mD0rqQfB+9w6SUoCngPPNbKakpsDusP1vQA8z2yDpz8AY4HfRysO5uiYtJYnjujTjuC7NAFi9dScffLmB979Yz7sL1vHStJUAdGuRw9GdmnJ0pyb079iUJllpsQzbxbFoHpH0BxaZ2RIASWOBEUBkITGCIw6AXIIjDIBTgFlmNhPAzDaG+0gFBGRJ2hhuuyiKOThX57XMbcDZ/dpydr+27Ck15qzcygdfrufTJZsYO2UZj3/8FRAUlgEdm3B0p6b079gktkG7uKJoDSonaSQwzMwuDZfPBwaY2ZiIPi2BN4HGQBYw1MymSvol0BdoDjQDxprZnyP2+xiwHfgSGGJm33ltnaTRwGiA/Pz8vmPHjq1SHoWFhWRnJ9b7Jjzn+FFSaizdWsqCTXtYuKmUL7bsoTj815LfwOjaJJVDGiXRuVEyrbJFUhxfY4nXv+MDqW7OQ4YMmWpm/SrqF+uL7aOAx83sTkkDgScl9QzjOhY4CtgBvCNpKvA+8DPgCGAJcC9wI3Bb2R2b2cPAwwD9+vWzwYMHVynASZMmUdVt6yvPOX7t3lPKrBVbmbx0IxOmLmLOZvHBymDY+5z0FHq3bcSR7RpxRPvGHNG2EY0y4+d0WKL8HUeqrZyjWUhWAm0jltuEbZEuAYYBmNknkjKAPIJrKu/vvYguaTxwJLAt7Ls4bH8euCGKOTgXV1KTk+jbvjF92zemBys44YQT+HrjDqYt2xxMX2/hvomLKA1PVLRvmknPVrkc1rohvVrn0rNVLo39WosrI5qFZArQRVJHggJyDnBumT7LgJOAxyV1BzKA9cAE4FeSMoFi4ATg7nA/PSQ1M7P1BBfy50cxB+fimiQ65GXRIS+Ls45sA8D2XSXMXLGF6cu2MHfVVmat3MJrs1fv26Z1owb0bN2Qnq2CMcR6tGpI85x0v/U4gUWtkJhZiaQxBEUhGXjMzOZKuhX43MzGAdcC/5B0NcGF94ssuGizWdJdBMXIgPFm9hqApFuA9yXtBr4GLopWDs4loqz0FI45JI9jDsnb17Z1x27mrNrKnJVbmbNqG3NWbmXC3LX71uc2SOXQ/BwObZFD1xY5wXx+jr+DJUFE9RpJ+EzI+DJtN0fMzwMG7WfbpwhuAS7b/hDwUM1G6pw7kNzM1H3vrN+roGg3c1dtY+GaAhauLWDhmgL+M30lBbtK9vVp0TCDri1y6NYih0OaZdGpWTYd87JompXmRzBxJNYX251z9VRORmr4nErTfW1mxuqtRfuKyxfhz8c/3khxyTcv9GqYkUKnZtl0apZFp7ysffMdmmaRkerDvtQ3XkicczVGEq0aNaBVowYM6dZ8X/ueUmPl5p0s3lDIkvXbWbK+kKUbtvPxoo37HqAMtodWuQ1o07gB7Zpk0q5JJm3DqV2TTPKy/UimLvJC4pyLuuQk0a5pJu2aZjLk0G+v276rhKUbtrN4fVBklm3awbJNO3jvi/WsK9j1rb4NUpNp26QBbRsHxaVN46BotczNoFWjBuRlp5PsoyPXOi8kzrmYykpP2fcWybKKdu9hxeagsCzbuIPlm3eybNMOlm/awadLNrK9+NvPIqckifyGGbTMzaBlowa0ys2gRW4GLXMbsHrrHnpsK6KpF5sa54XEOVdnZaQm07l5Dp2b53xnnZmxZcduVm8tYvXWnazaWsTqLTtZs7WIVVt3MmvFFibMLfrWtZlbPnmHJEHT7HSaZafTLCed5jnBz2A+I2I+nax0/4qsDP9Tcs7VS5JonJVG46w0erRqWG4fM2PT9mJWby3izQ+nkNeuMxsKdrGuYBfrw58L1xSwoXAXJaXfHS4qMy2ZvOx0mmSl0TT8rCZ7p8y9y6k0yUqnSWYaORkpCfniMS8kzrm4JYmm2ek0zU5nQ34Kgwd2KLdfaamxeUcx6wvDArNt17759QW72LyjmDXbipi3ehsbtxd/6ygnUnKSaJwZFJfGmWnkNkjdNzXc9zMl+Jnx7XX1+W41LyTOuYSXlPRNwenW4sB9zYydu/ewaXvxvmnzjmI2bd/Npu272LR9N5vD9q837mBb0W627tzNjuLvjC37LWkpSWFxSdlXdLLSU8hJTyErnL6ZTyYnI4WstLA9I/iZnZ5CekpSrd/Z5oXEOecOgiQy01LITEuhTePMSm+3e08p23buZltRCVt3BsVl296fRXuXS8I+u9m0vZhlm3awfVcJhUUl37mxYH9SkrSvqKikiLeO2RP1VzJ7IXHOuVqQmpy076inKkpLjR2791BYVELhrpKgwOzaz3xRCYW79vDVytWkpyTVcCbf5YXEOefqgaQkkR0eaVTWpEmba+Xif/RLlXPOubjmhcQ551y1eCFxzjlXLV5InHPOVYsXEuecc9XihcQ551y1eCFxzjlXLV5InHPOVYvMvjviZbyRtB74uoqb5wEbajCc+sBzTgyJlnOi5QvVz7m9mTWrqFNCFJLqkPS5mfWLdRy1yXNODImWc6LlC7WXs5/acs45Vy1eSJxzzlWLF5KKPRzrAGLAc04MiZZzouULtZSzXyNxzjlXLX5E4pxzrlq8kDjnnKsWLyT7IWmYpIWSFkm6IdbxVIekxyStkzQnoq2JpLckfRn+bBy2S9I9Yd6zJB0Zsc2FYf8vJV0Yi1wqS1JbSRMlzZM0V9JVYXvc5i0pQ9JnkmaGOd8StneUNDnM7TlJaWF7eri8KFzfIWJfN4btCyWdGpuMKkdSsqTpkl4Nl+M6XwBJX0maLWmGpM/Dttj9bpuZT2UmIBlYDHQC0oCZQI9Yx1WNfI4HjgTmRLT9GbghnL8B+FM4fzrwOiDgaGBy2N4EWBL+bBzON451bgfIuSVwZDifA3wB9IjnvMPYs8P5VGBymMvzwDlh+0PAz8L5K4CHwvlzgOfC+R7h73w60DH8t5Ac6/wOkPc1wDPAq+FyXOcbxvwVkFemLWa/235EUr7+wCIzW2JmxcBYYESMY6oyM3sf2FSmeQTwRDj/BHBmRPu/LPAp0EhSS+BU4C0z22Rmm4G3gGHRj75qzGy1mU0L5wuA+UBr4jjvMPbCcDE1nAw4EXghbC+b894/ixeAkyQpbB9rZrvMbCmwiODfRJ0jqQ3wPeCRcFnEcb4ViNnvtheS8rUGlkcsrwjb4km+ma0O59cA+eH8/nKvt38m4SmMIwj+hx7XeYeneWYA6wi+GBYDW8ysJOwSGf++3ML1W4Gm1K+c/wr8CigNl5sS3/nuZcCbkqZKGh22xex3u/JvkXdxy8xMUlzeBy4pG3gR+KWZbQv+AxqIx7zNbA/QR1Ij4GWgW4xDihpJZwDrzGyqpMGxjqeWHWtmKyU1B96StCByZW3/bvsRSflWAm0jltuEbfFkbXh4S/hzXdi+v9zr3Z+JpFSCIvK0mb0UNsd93gBmtgWYCAwkOJWx9z+NkfHvyy1cnwtspP7kPAgYLukrgtPPJwJ/I37z3cfMVoY/1xH8h6E/Mfzd9kJSvilAl/DujzSCC3PjYhxTTRsH7L1L40LglYj2C8I7PY4GtoaHyxOAUyQ1Du8GOSVsq5PCc9+PAvPN7K6IVXGbt6Rm4ZEIkhoAJxNcG5oIjAy7lc1575/FSOBdC67CjgPOCe9y6gh0AT6rnSwqz8xuNLM2ZtaB4N/ou2Z2HnGa716SsiTl7J0n+J2cQyx/t2N990FdnQjudPiC4Bzzb2IdTzVzeRZYDewmOA96CcG54XeAL4G3gSZhXwH3h3nPBvpF7OdigguRi4CfxjqvCnI+luA88ixgRjidHs95A4cD08Oc5wA3h+2dCL4YFwH/BtLD9oxweVG4vlPEvn4T/lksBE6LdW6VyH0w39y1Fdf5hvnNDKe5e7+fYvm77UOkOOecqxY/teWcc65avJA455yrFi8kzjnnqsULiXPOuWrxQuKcc65avJC4uCCpaTgS6gxJayStjFhOq+Q+/inp0Ar6XCnpvBqK+UNJfSQlqYZHmJZ0saQWEcsV5uZcVfntvy7uSPodUGhmfynTLoLf+dJyN6xlkj4ExhA887HBzBod5PbJFgyJst99m9mM6kfq3IH5EYmLa5I6K3gnydMED2+1lPSwpM8VvLPj5oi+e48QUiRtkXS7gnd7fBKOaYSk2yT9MqL/7QreAbJQ0jFhe5akF8PPfSH8rD4HCPN2ICc8evpXuI8Lw/3OkPRAeNSyN66/SpoF9Jd0i6QpkuZIeih8evnHQB/gub1HZHtzC/f9EwXvspgj6Q9h24FyPifsO1PSxBr+K3JxwAuJSwTdgLvNrIcFYxTdYGb9gN7AyZJ6lLNNLvCemfUGPiF4Arg8MrP+wHXA3qL0c2CNmfUA/o9g5OEDuQEoMLM+ZnaBpJ7AD4BjzKwPweCq50TE9b6ZHW5mnwB/M7OjgF7humFm9hzBk/w/DvdZvC/YYNj124AhYVyDFAx+eKCcfwucFLb/oIJcXALyQuISwWIz+zxieZSkacA0oDvBi43K2mlmr4fzU4EO+9n3S+X0OZZgEEHMbO8wFgdjKHAU8LmCIeFPAA4J1xUTDNK310mSPiMYLuME4LAK9j2AYIypDWa2m+CFUMeH6/aX80fAvyRdin9nuHL4MPIuEWzfOyOpC3AV0N/Mtkh6imAMprKKI+b3sP9/K7sq0edgCXjMzP73W43BiLU7be8ASlImcB/BmyBXSrqN8nOprP3lfBlBAToDmCbpCAtehOQc4P+7cImnIVAAbNM3b4mraR8BZwNI6kX5Rzz7WPgSJn0z9PnbwNmS8sL2ppLalbNpA4IXOm0IR4P9YcS6AoJXDJc1GRgS7nPvKbP3KsinkwVv1vtfYDN1/6VPrpb5EYlLNNOAecAC4GuCL/2adi/BqaB54WfNI3gb34E8CsyS9Hl4neQW4G1JSQSjNv8PsCpyAzPbKOmJcP+rCYrEXv8EHpG0k4jXxprZCkn/C0wiOPL5r5m9FlHEynO3guHVBbxpZnMqyMUlGL/917kaFn4pp5hZUXgq7U2gi33z+lfn4oofkThX87KBd8KCIuByLyIunvkRiXPOuWrxi+3OOeeqxQuJc865avFC4pxzrlq8kDjnnKsWLyTOOeeq5f8DFLXSN2eGRsAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')\n",
    "\n",
    "plt.title('Mometum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Nesterov Accelerated Gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer the NAG section in the book for the detailed explanation.\n",
    "\n",
    "The update equation of NAG is given as:\n",
    "\n",
    "$$v_t = \\gamma v_{t-1} + \\eta \\nabla_\\theta J( \\theta - \\gamma v_{t-1} ) -- (10) $$\n",
    "\n",
    "Update theta by:\n",
    "$${\\theta = \\theta - v_t } -- (11) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def NAG(data, theta, lr = 1e-2, gamma = 0.9, num_iterations = 5000):\n",
    "    \n",
    "    loss = []\n",
    "    \n",
    "    #Initialize vt with zeros:\n",
    "    vt = np.zeros(theta.shape[0])\n",
    "       \n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients with respet to (theta-gamma*vt-1)\n",
    "        gradients = compute_gradients(data, theta - gamma * vt)\n",
    "        \n",
    "        #update vt by equation (10)\n",
    "        vt = gamma * vt + lr * gradients\n",
    "        \n",
    "        #update model parameter theta by equation (11)\n",
    "        theta = theta - vt\n",
    "        \n",
    "        loss.append(loss_function(data,theta))\n",
    "        \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute and update gradients using NAG:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = NAG(data, theta) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8FeXZ//HP92QhkISwR/YdBUFQkEURQbRFa8U+UgUtFuvyWB9qq/ZptbXW+rPW1ketrVupdV+QuqJScQNxl33fF9k3EUiAACHX74+Z4DEmJCQ5OcnJ9X695pWZe+6Zc18nybnOzD1zj8wM55xzrrwi8W6Ac865ms0TiXPOuQrxROKcc65CPJE455yrEE8kzjnnKsQTiXPOuQrxROJcjEgaLGl9vNtREkmPS7o9zm0YI+nDqOVcSR3i2SZ39DyROCStkbRVUnpU2RWSplZwv9Xyg1TSVElfSaoT77ZUhCST1CmG+28u6Z+SNoYf8KvC5HNcrF7TzDLMbFVF91MdkmRt4onEFUoCfh7vRkSTlByDfbYDTgMMOK+y919ZYhH7Ub5+Y+BjoB7B+5UJnAS8D5xVwjZxbbOLH08krtBdwC8lNShupaTjJL0taYekpZIujFp3jqRFknIkbZD0y/Do5j9Ai/DbbK6kFpIikm6UtFLSl5ImSGoU7qdd+C37cklrgffC8vMkLZS0Mzya6BqW/1rSC0XaeZ+kvx0hzkuBT4HHgR8X2baupLslfSFpl6QPJdUN1w2U9HHYhnWSxoTldST9n6S1krZIerhwm2LewxaSXpS0TdJqSddGrbtV0guSnpa0Gxgjqa+kT8LX3CTpfkmpYf1p4aZzw/f2orD8XElzwm0+lnRC1GucKGlW+Ht6Hkg7wvt0HbAbGG1mKy2w08weM7O/h/sr6ff1b0mbw/dwmqTjo9rQWNJESbslfQ50LPIeHT7KOtJ7W3i0K+mG8Gh6k6TLwnVXAZcAvwrfm9fC8l+Hf5854d/w0CPE746GmflUyydgDXAm8BJwe1h2BTA1nE8H1gGXAcnAicB2oFu4fhNwWjjfEDgpnB8MrC/yWj8n+CBvBdQB/gE8F65rR3Ck8GT4mnWBLsAegm/BKcCvgBVAKtAW2AtkhtsnhW3pf4RYVwDXAL2Bg0B21LoHgKlAy3Bfp4RtbAvkAKPCNjQGeoXb3AtMBBoRfGt/DfhT0fgJvrTNBG4J294BWAV8N1x/a9ie88O6dcM29g/f83bAYuAXUe01oFPU8onAVqBf2P4fh7/bOuFrfkGQIFKAEeHr3V7C+/QpcGspfzff+n2F5T8J34s6wF+BOVHbjAcmhPW7AxuAD4uLqQzvbT5wWxjPOeHfQsNw/ePRsQHHEvwNt4hqe8d4/+8lyhT3BvgU/4mvE0l3YBfQlG8mkouAD4ps8w/g9+H8WuC/gfpF6hz+II0qWwwMjVpuHn6gFX5YGtAhav3vgAlRy5Hww2dwuPwhcGk4fxaw8ghxDgxfq0m4vAS4Lmq/+4CexWx3E/ByMeUiSHIdo8oGAKuLxk/w4b62mP0+Fs7fCkwr5ff0i+h28O1E8hDw/4pssxQ4HRgEbAQUte5jSk4kK4Cro5bPA3YSJNS3wrJv/b6K2U+DsE4WQXI7CBwXtf4OikkkZXxv9wHJUeu3En6J4NuJpFO4/kwgJd7/c4k2+aktd5iZLQBeB24ssqot0C88XbJT0k6CUwfHhOsvIPhG+IWk9yUNOMLLtAVejtrPYuAQkB1VZ13UfAuCb9KFbSwI17cMi54lOFIAuDhcLsmPCT4Et0dtW3h6qwnBqZ6VxWzXuoTypgR9CDOj4nkzLC+qLcFpvuj38DeUHDeSukh6PTxNtJvgQ7fJEeJrC9xQ5DVaE7yHLYANFn6qhr4obiehLwmSPABmNtHMGhAc0aQWqXu43ZKSJN0ZnrrcTfAlhbDdTQm+METHWVIbyvLefmlm+VHLe4GM4nZmZisIEvGtwFZJ4yW1KOG13VHyROKK+j1wJV9/UEPwj/++mTWImjLM7KcAZjbdzIYDzYBXCE5dQPDtsqh1wNlF9pVmZhui6kRvt5HgAxIASSL4cCys/29gsKRWwA8oIZGE59YvBE4PP5g3E3wo9pTUk+BUXR5FztlHtbm48u0E34qPj4oly8yK+zBbR/BtOjruTDM7p4S4ITjCWAJ0NrP6BIlHxcUX9Rp/LPIa9czsOYJTfi3D969QmyPs613gfEll+YyIbvfFwHCCb/5ZBEcthO3eRnA6qnUZ2nA0721pbQoKzJ41s4EEf08G/LmM+3Kl8ETiviH85vY8cG1U8etAF0mjJaWE08mSukpKlXSJpCwzO0jQQVsQbrcFaCwpK2pfDwN/lNQWQFJTScOP0KQJwPckDZWUAtwA7Cc4LYOZbSPo13iM4IN6cQn7OZ/gyKcb0CucugIfEJwaKwAeBe4JO8WTJA1QcInwM8CZki6UlBx2GPcKt/kncK+kZmE8LSV9t5jX/xzICTt864b77y7p5CPEnknwfuYquOT2p0XWbyHoayn0T+BqSf0USJf0PUmZwCcEH+LXhr+//wL6HuG17yHo73pKUsdwf5nh+3YkmQS/ny8JjijuKFxhZocI+uFulVRPUjeKXPAQVfdo3tvifOO9kXSspDPC32ceQZIqKGljd3Q8kbji3EbQGQqAmeUA3wFGEhwhbCb4Nld4H8ZoYE14KuNqgtNemNkS4DlgVXh6ogVwH0EH6luScgg6dfuV1BAzWwr8CPg7wbfU7wPfN7MDUdWeJfgGXNpprcfMbK2ZbS6cgPuBSxRcuvpLYD4wHdgRxhgxs7UEp+5uCMvnAD3D/f6aoD/h0zD+dwg6dovGcQg4l+CDeHUYyyME39pL8kuCb/g5BB+qzxdZfyvwRPjeXmhmMwiOJu8HvgrbNSZ8/QPAf4XLOwj6vV4q6YXD03/9CT50PwzbMIcgURRNaNGeJDhdtQFYRPD7jTaW4PTTZoJ+jMeOsK8yvbcl+BfQLXxvXiH4W72T4H3fTHD0fFMZ9+VKoW+eMnXOOeeOjh+ROOecqxBPJM455yrEE4lzzrkKiWkikTQsHIpghaSi9yYgqY2kKZJmS5on6ZywPEXSE5LmS1os6aawvHVYf5GCITOq1dhQzjlXG8Wss11SErCM4G7j9QRXwowys0VRdcYBs83sofBSwElm1k7SxcB5ZjZSUj2Cqz8GE1xW2NzMZoWXIs4Ezo/eZ3GaNGli7dq1K1cce/bsIT09vfSKCcRjrh1qW8y1LV6oeMwzZ87cbmbF3WD7DbEcrbMvsMLCIaEljSe4USn6Q9+A+uF8FsGlpYXl6eElmXWBA8BuM9tBcGMVZpYjaTHBjXNHTCTt2rVjxowZ5Qpi6tSpDB48uFzb1lQec+1Q22KubfFCxWOWdKTRD76uF8MjkhHAMDO7IlweDfQzs7FRdZoDbxHc+JQOnGlmM8Mbz54ChhLc1HSdmY0rsv92wDSgu5ntLub1rwKuAsjOzu49fvz4csWRm5tLRkZZb6ZNDB5z7VDbYq5t8ULFYx4yZMhMM+tTWr14Pz9gFPC4md2tYHympyR1JziaOUQwPlBD4ANJ70Qd3WQALxKMhPqtJAIQJp5xAH369LHyZmX/FlM7eMyJr7bFC1UXcyw72zfwzTF1WvH1+EiFLiccl8nMPiEYNK8Jwd28b5rZQTPbCnwE9IGgI54giTxjZiXemeucc65qxDKRTAc6S2qv4GE8IwmGxoi2luD0FQoeVpRGMLDbWuCMsDydYKiGJeGAc/8CFpvZPTFsu3POuTKKWSIJh3ceC0wmGCp8gpktlHSbpMJHnN4AXClpLsGYTGPCYa4fADIkLSRISI+Z2TzgVIJxnc5Q8BS4OYWXDDvnnIuPmPaRmNkkYFKRslui5hcRJIei2+UCPyym/EOOPIy2c865KuZ3tjvnnKsQTyRH8PSnX7Bg+6F4N8M556o1TyQlOJBfwDOfreVvs/LYsHNfvJvjnHPVlieSEqQmR/jnpb0pMPjH+8U9rts55xx4IjmiVg3r0bd5Mi/OXE9O3sF4N8c556olTySlGNommT0HDvHy7KL3UjrnnANPJKXqkBXh+Bb1efaztfhjiZ1z7ts8kZRCEhf3a8OSzTnMXrcz3s1xzrlqxxNJGZzXswX1UpN47rO18W6Kc85VO55IyiAzLYXhvVrw2ryN7Nrnne7OORfNE0kZXdy3LXkHC3h1jne6O+dcNE8kZdSjVRbdW3qnu3POFeWJ5Chc3LctSzbnMGutd7o751whTyRH4bxeLUhPTeK5z73T3TnnCnkiOQoZdZI5r1dLXvdOd+ecO8wTyVG6uG8b8g4W8Irf6e6cc4AnkqPWo1UWPVpmeae7c86FPJGUw6i+bVi6xTvdnXMOYpxIJA2TtFTSCkk3FrO+jaQpkmZLmlf4/HVJKZKekDRf0mJJN5V1n1WhsNP9Wb/T3TnnYpdIJCUBDwBnA92AUZK6Fal2MzDBzE4ERgIPhuU/BOqYWQ+gN/DfktqVcZ8x941O973e6e6cq91ieUTSF1hhZqvM7AAwHhhepI4B9cP5LGBjVHm6pGSgLnAA2F3GfVaJS/q1YX9+AS/PXh+Pl3fOuWojOYb7bgmsi1peD/QrUudW4C1JPwPSgTPD8hcIEsQmoB5wnZntkFSWfQIg6SrgKoDs7GymTp1ariByc3NL3LZ9/QiPTFlM2wNrkFSu/VdHR4o5UXnMia+2xQtVF3MsE0lZjAIeN7O7JQ0AnpLUneDI4xDQAmgIfCDpnaPZsZmNA8YB9OnTxwYPHlyuBk6dOpWStt1Uby03vTSf+h160rtto3Ltvzo6UsyJymNOfLUtXqi6mGN5amsD0DpquVVYFu1yYAKAmX0CpAFNgIuBN83soJltBT4C+pRxn1XmvJ5Bp/sz3ununKvFYplIpgOdJbWXlErQmT6xSJ21wFAASV0JEsm2sPyMsDwd6A8sKeM+q0x6nWSGn9iSN+Zt8k5351ytFbNEYmb5wFhgMrCY4OqshZJuk3ReWO0G4EpJc4HngDEW3OX3AJAhaSFB8njMzOaVtM9YxVAWF/cNOt3/PXNd6ZWdcy4BxbSPxMwmAZOKlN0SNb8IOLWY7XIJLgEu0z7jqXvLLPq0bciTn3zBZae2JymSOJ3uzjlXFn5neyUYc2o71u7Yy3tLtsa7Kc45V+U8kVSC7x5/DM2z0njso9XxbopzzlU5TySVICUpwugBbfl45Zcs2bw73s1xzrkq5Ymkkow6uQ1pKREe/2hNvJvinHNVyhNJJWmYnsoPTmzJy7M3sGPPgXg3xznnqownkko05pT27M8v8EfxOudqFU8klejYYzI5tVNjnv70Cw4eKoh3c5xzrkp4Iqlkl53Snk278pi8cHO8m+Kcc1XCE0klO+O4ZrRtXI9/frDaH8XrnKsVPJFUskhEXHFaB+au28nnq3fEuznOORdznkhi4Ie9W9EoPZVx01bFuynOORdznkhiIC0liUsHtOXdJVtZviUn3s1xzrmY8kQSI5cOaEdaSsSPSpxzCc8TSYw0Sk/lwj6teWXOBjbvyot3c5xzLmY8kcTQFQM7cKjAeOxjH8zROZe4PJHEUJvG9Ti7R3Oe/XQtOXn+BEXnXGLyRBJj/z2oAzn7833YFOdcwvJEEmMntGrAgA6N+deHq9mffyjezXHOuUrniaQKXDOkI1t27+eFmevj3RTnnKt0MU0kkoZJWipphaQbi1nfRtIUSbMlzZN0Tlh+iaQ5UVOBpF7hulGS5of135TUJJYxVIaBnZrQs3UDHpq60gdzdM4lnJglEklJwAPA2UA3YJSkbkWq3QxMMLMTgZHAgwBm9oyZ9TKzXsBoYLWZzZGUDNwHDDGzE4B5wNhYxVBZJPGzIZ1Y/9U+Xp2zMd7Ncc65ShXLI5K+wAozW2VmB4DxwPAidQyoH85nAcV9yo4KtwVQOKVLUrhtjfhkHtq1GV2b1+fBKSs4VOCDOTrnEodiNUKtpBHAMDO7IlweDfQzs7FRdZoDbwENgXTgTDObWWQ/K4HhZrYgar+PAnuA5QRHJ9/qxZZ0FXAVQHZ2du/x48cXrVImubm5ZGRklGvboqZvzueBOfu5umcd+jdPrpR9xkJlxlxTeMyJr7bFCxWPeciQITPNrE+pFc0sJhMwAngkank0cH+ROtcDN4TzA4BFQCRqfT9gftRyCvAu0JHgyOR+4ObS2tK7d28rrylTppR726IOHSqwoXdPte/c874dOlRQafutbJUZc03hMSe+2havWcVjBmZYGT7vY3lqawPQOmq5VVgW7XJgAoCZfQKkAdGd5yOB56KWe4V1V4ZBTgBOqdxmx04kIsYO6cTSLTm8vXhLvJvjnHOVIpaJZDrQWVJ7SakESWFikTprgaEAkroSJJJt4XIEuJCv+0cgSETdJDUNl88CFscsghg494TmtG1cj7+/t9wffOWcSwgxSyRmlk9wRdVkgg/7CWa2UNJtks4Lq90AXClpLsGRxxj7+tN1ELDOzFZF7XMj8AdgmqR5BEcod8QqhlhITorwP4M7sWDDbqYs3Rrv5jjnXIXFtMfXzCYBk4qU3RI1vwg4tYRtpwL9iyl/GHi4UhtaxX5wUkvun7KCu99axpBjmxFcgOacczWT39keBylJEX4+tDMLN+5m8sLN8W6Oc85ViCeSODn/xJZ0bJrOPW8v8/tKnHM1mieSOEmKiOvO6sKyLbm8Pq9G3FPpnHPF8kQSR+d0b85xx2Ry79vLyPcxuJxzNZQnkjiKRMQN3zmWNV/u5aVZRW+xcc65msETSZyd2bUZPVtlcd+7y/15Jc65GskTSZxJ4vrvHMuGnft4fvq6eDfHOeeOmieSamBQ5yb0bdeIv727gj378+PdHOecOyqeSKoBSdx4znFsz93PPz9YVfoGzjlXjXgiqSZOatOQc3ocw7hpq9iakxfv5jjnXJl5IqlG/ve7x3Egv4D73lke76Y451yZeSKpRto3SeeSfm0YP30dK7flxrs5zjlXJp5IqpmfDe1M3ZQk/vLmkng3xTnnysQTSTXTJKMOV5/egckLtzBjzY54N8c550rliaQa+snA9jTLrMOf/rPEH37lnKv2PJFUQ/VSk7nhO12Y+cVXvD5vU7yb45xzR+SJpJoa0bs1x7eoz58mLWbfAR86xTlXfXkiqaaSIuL33z+ejbvyePj9lfFujnPOlSimiUTSMElLJa2QdGMx69tImiJptqR5ks4Jyy+RNCdqKpDUK1yXKmmcpGWSlki6IJYxxFPf9o0494TmPPz+Sjbs3Bfv5jjnXLFilkgkJQEPAGcD3YBRkroVqXYzMMHMTgRGAg8CmNkzZtbLzHoBo4HVZjYn3Oa3wFYz6xLu9/1YxVAd3HROVyT406TF8W6Kc84VK5ZHJH2BFWa2yswOAOOB4UXqGFA/nM8CintU4Khw20I/Af4EYGYFZra9UltdzbRsUJerT+/I6/M28flqvxzYOVf9KFaXl0oaAQwzsyvC5dFAPzMbG1WnOfAW0BBIB840s5lF9rMSGG5mCyQ1AOYD/wYGAyuBsWa2pZjXvwq4CiA7O7v3+PHji1Ypk9zcXDIyMsq1bWXZf8i46YN9ZKSIW09JIyLF9PWqQ8xVzWNOfLUtXqh4zEOGDJlpZn1KrWhmMZmAEcAjUcujgfuL1LkeuCGcHwAsAiJR6/sB86OWmxAcxYyI2v6p0trSu3dvK68pU6aUe9vKNHHOBmv769ftqU/WxPy1qkvMVcljTny1LV6ziscMzLAyfN7H8tTWBqB11HKrsCza5cAEADP7BEgLk0WhkcBzUctfAnuBl8LlfwMnVV6Tq69zT2hO/w6N+MubS9ieuz/ezXHOucNimUimA50ltZeUSpAUJhapsxYYCiCpK0Ei2RYuR4ALieofCTPkawSntQi3XRS7EKoPSdx+fnf2HTzEHd7x7pyrRmKWSMwsHxgLTAYWE1ydtVDSbZLOC6vdAFwpaS7BkceYMFkADALWmVnRJz39GrhV0jyC02U3xCqG6qZTs0yuGtSBl2Zt4NNVX8a7Oc45B0ByLHduZpOASUXKbomaXwScWsK2U4H+xZR/QZBkaqWxQzrz6pyN3PzKAiZdexqpyX5PqXMuvvxTqIapm5rEbcOPZ8XWXB750B/L65yLP08kNdAZx2Uz7Phj+Nu7y1m3Y2+8m+Ocq+U8kdRQt3y/GxGJ309c6EPNO+fiyhNJDdWiQV2uP6sL7y3ZysS5xQ0I4JxzVcMTSQ122ant6dW6AX94bRFf+r0lzrk48URSgyVFxF9GnEBO3kFufa1W3E7jnKuGPJHUcF2yM/nZGZ15be5G3l70rSHHnHMu5jyRJICfDu7Iccdk8tuX57Nr38F4N8c5V8t4IkkAKUkR7hrRk+25+7njDR8+xTlXtTyRJIgerbK4clAHnp+xjmnLtsW7Oc65WsQTSQK57swudGqWwa9emMeuvX6KyzlXNcqUSCQ9VZYyF19pKUnce2Evtufu53evLoh3c5xztURZj0iOj14In8feu/Kb4yqqR6ssrh3amYlzN/Ka36jonKsCR0wkkm6SlAOcIGl3OOUAW4FXq6SF7qhdM7gjvVo34OZXFrB5V168m+OcS3BHTCRm9iczywTuMrP64ZRpZo3N7KYqaqM7SslJEe65sCcH8gv41YvzfCwu51xMlfXU1uuS0gEk/UjSPZLaxrBdroI6NM3gN9/ryrRl23j60y/i3RznXAIrayJ5CNgrqSfBEwlXAk/GrFWuUvyoXxtO79KU299YzLItOfFujnMuQZU1keSHj8AdDtxvZg8AmbFrlqsMkvi/H/YkMy2Fsc/OYt+BQ/FuknMuAZU1keRIuongGelvSIoAKaVtJGmYpKWSVki6sZj1bSRNkTRb0jxJ54Tll0iaEzUVSOpVZNuJkvwa11I0zazDvRf1ZPnWXG57fWG8m+OcS0BlTSQXAfuBn5jZZqAVcNeRNggvEX4AOBvoBoyS1K1ItZuBCWZ2IjASeBDAzJ4xs15m1osgea02szlR+/4vILeMba/1TuvclJ+e3pHnPl/nlwQ75ypdmRJJmDyeAbIknQvkmVlpfSR9gRVmtsrMDgDjCU6NfWPXQP1wPgso7lNuVLgtAJIygOuB28vSdhe47qwunNSmAb95aT5rv/TH8zrnKo/KcmmopAsJjkCmAgJOA/7XzF44wjYjgGFmdkW4PBroZ2Zjo+o0B94CGgLpwJlmNrPIflYCw81sQbh8LzANmA28bmbdS3j9q4CrALKzs3uPHz++uGqlys3NJSMjo1zbVjfb9hbw+4/3kZ0e4bf90kiOqNh6iRRzWXnMia+2xQsVj3nIkCEzzaxPqRXNrNQJmAs0i1puCswtZZsRwCNRy6MJOuqj61wP3BDODwAWAZGo9f2A+VHLvYCJ4Xw7YEFZ2t+7d28rrylTppR72+po0ryN1vbXr9utExeUWCfRYi4Ljznx1bZ4zSoeMzDDyvAZW9Y+koiZbY1a/pLST4ttAFpHLbcKy6JdDkwAMLNPgDSgSdT6kcBzUcsDgD6S1gAfAl0kTS1bCA7g7B7NuezUdjz20RpenVP01+Gcc0evrInkTUmTJY2RNAZ4A5hUyjbTgc6S2ktKJUgKE4vUWQsMBZDUlSCRbAuXI8CFRPWPmNlDZtbCzNoBA4FlZja4jDG40G/O6Uqftg258cX5LN3s95c45yqmtLG2Okk61cz+F/gHcEI4fQKMO9K2ZpYPjAUmA4sJrs5aKOk2SeeF1W4ArpQ0l+DIY0x4OAUwCFhnZqvKGZsrQUpShAcvOYmMtGSufnomu/N8yHnnXPmVdkTyV2A3gJm9ZGbXm9n1wMvhuiMys0lm1sXMOprZH8OyW8xsYji/yMxONbOeFlzu+1bUtlPNrP8R9r3GSuhod6VrVj+NBy85iXU79nLDhLkUFPh4XM658iktkWSb2fyihWFZu5i0yFWZk9s14jfndOXtRVt4eNrKeDfHOVdDlZZIGhxhXd3KbIiLj8tObcf3e7bgrslLmbJka+kbOOdcEaUlkhmSrixaKOkKYGYx9V0NI4k/X9CDbs3r87PnZvvgjs65o1ZaIvkFcJmkqZLuDqf3CS7b/Xnsm+eqQr3UZB75cR/qpiZx+RPTyTng/SXOubIr7cFWW8zsFOAPwJpw+oOZDbBg2BSXIJpn1WXc6N5s2b2f+2fncSC/IN5Ncs7VEGUda2uKmf09nN6LdaNcfJzYpiF3jTiBpV8VcMurC/zJis65MinrDYmulhjeqyXf75DC+Onr+NeHq+PdHOdcDZAc7wa46ucHnVM4WLcxf5y0mJYN6nJ2j+bxbpJzrhrzIxL3LRGJv47sxUltGvLz5+cwfc2OeDfJOVeNeSJxxUpLSeKRS/vQqmFdrnhiBiu2+mXBzrnieSJxJWqYnsoTl/UlJSnCjx+dztbdefFuknOuGvJE4o6odaN6PH7Zyezce4Axj00nxwd4dM4V4YnElap7yywe/FFvlm3J4aonZ5J38FC8m+Scq0Y8kbgyOb1LU+6+sCefrv6Sa56ZxcFDfsOicy7gicSV2fBeLbn9/O68t2Qr1z0/h0M+9LxzDr+PxB2lS/q1Zc/+fO6YtIT01GTuvKAHkuLdLOdcHHkicUftqkEdycnL5+/vrSAjLZmbv9fVk4lztZgnElcu15/VhZy8fP714WpSkyP86rvHejJxrpaKaR+JpGGSlkpaIenGYta3kTRF0mxJ8ySdE5ZfImlO1FQgqZekepLekLRE0kJJd8ay/a5kkrjl3G5c3K8ND01dyZ1vLvFBHp2rpWJ2RCIpCXgAOAtYD0yXNNHMFkVVuxmYYGYPSeoGTALamdkzwDPhfnoAr5jZHEn1gP8zsymSUoF3JZ1tZv+JVRyuZJGIuH14dyKCf7y/CjO46ezj/MjEuVomlqe2+gIrzGwVgKTxwHAgOpEYUD+czwI2FrOfUcB4ADPbC0wJ5w9ImgW0iknrXZlEIuL/De9ORGLctFUUFBi/9T4T52qVWCaSlsC6qOX1QL8idW4F3pL0MyAdOLOY/VxEkIC+QVID4PvAfZXRWFd+kvjDeccTkXjkw9UUGPzuXE8mztUWitV5bUkjgGFmdkW4PBqovpnKAAAZ2ElEQVToZ2Zjo+pcH7bhbkkDgH8B3c2sIFzfD3jEzHoU2Xcy8Bow2cz+WsLrXwVcBZCdnd17/Pjx5YojNzeXjIyMcm1bU5U3ZjPj2SUHePuLfE5vlcyPj08lUkOSif+eE19tixcqHvOQIUNmmlmfUiuaWUwmYADBB33h8k3ATUXqLARaRy2vAppFLd8L/KaYfT8K/K2sbendu7eV15QpU8q9bU1VkZgLCgrsrjeXWNtfv27XPD3T9h88VHkNiyH/PSe+2havWcVjBmZYGT5jY3lqazrQWVJ7YAMwEri4SJ21wFDgcUldgTRgG4CkCHAhcFr0BpJuJ+hPuSKGbXflJIlffvdYsuqm8MdJi8nZn8/DPzqJeql+pblziSpml/+aWT4wFpgMLCa4OmuhpNsknRdWuwG4UtJc4DlgTJgFAQYB6yzsrAeQ1Ar4LdANmBVeGuwJpRq6clAH/nLBCXy4fBs/euQzdu31UYOdS1Qx/ZpoZpMILumNLrslan4RcGoJ204F+hcpWw/UjJPujgtPbk39uslc+9wcLhr3CY9f1pdjstLi3SznXCXzQRtdTA3r3pxHx5zMuh17+cGDH7Fk8+54N8k5V8k8kbiYG9i5Cf+++hTMYMRDn/DB8m3xbpJzrhJ5InFVoluL+rz8P6fQqmFdLntsOhOmryt9I+dcjeCJxFWZ5ll1+ffVAxjQsTG/enEed7+11Mfnci4BeCJxVSozLYVHx5zMRX1a8/f3VnDNM7PYsz8/3s1yzlWAX9zvqlxKUoQ7L+hB5+wM7pi0mNXb9/DPS/vQulG9eDfNOVcOfkTi4kISV5zWgccv68vGnfs47/4P+XjF9ng3yzlXDp5IXFwN6tKUiWMH0jijDqMf/ZzHP1rt/SbO1TCeSFzctWuSzsvXnMKQY5tx62uLuO75Od5v4lwN4onEVQuZaSmMG92b68/qwsS5Gxn+wEcs25IT72Y558rAE4mrNiIRce3Qzjx9eT927j3I8Ps/4sWZ6+PdLOdcKTyRuGrnlE5NmHTtQE5olcUN/57Lr1+YR97BQ/FulnOuBJ5IXLXUrH4az1zRj/8Z0pHnZ6zjvPs/ZNFGH6fLuerIE4mrtpKTIvzvd4/jyZ/05au9Bzn/gY8YN20lBQV+VZdz1YknElftDerSlMm/GMTgY5tyx6QlXPLIZ2zcuS/ezXLOhTyRuBqhUXoq/xjdmz9f0IO563cy7K/TeG3uxng3yzmHJxJXg0jiopPb8Ma1p9G+aQY/e242P316Jtty9se7ac7Vap5IXI3Tvkk6L149gF8NO5Z3l2zlrHvf5+XZ6/2OeOfixBOJq5GSkyJcM7gTk64dSPsm6Vz3/FyueGIGm3flxbtpztU6MU0kkoZJWipphaQbi1nfRtIUSbMlzZN0Tlh+iaQ5UVOBpF7hut6S5of7/Jskf4Z7LdapWSYvXH0Kvzu3Gx+t3M5Z97zPU59+wSG/ssu5KhOzRCIpCXgAOBvoBoyS1K1ItZuBCWZ2IjASeBDAzJ4xs15m1gsYDaw2sznhNg8BVwKdw2lYrGJwNUNSRFw+sD1v/nwQPVpl8btXFvBfD37E/PW74t0052qFWB6R9AVWmNkqMzsAjAeGF6ljQP1wPgso7jKcUeG2SGoO1DezTy04If4kcH4sGu9qnnZN0nnmin7cN7IXG3bmcd4DH3LLqwvYte9gvJvmXEJTrDooJY0AhpnZFeHyaKCfmY2NqtMceAtoCKQDZ5rZzCL7WQkMN7MFkvoAd5rZmeG604Bfm9m5xbz+VcBVANnZ2b3Hjx9frjhyc3PJyMgo17Y1VSLEvPeg8dLyA7y7Np/MVDHyuFQGNE+ipDOhiRDz0aptMde2eKHiMQ8ZMmSmmfUprV68n5A4CnjczO6WNAB4SlJ3MysAkNQP2GtmC452x2Y2DhgH0KdPHxs8eHC5Gjh16lTKu21NlSgxn3MWLNiwi9++soBx83by+VcN+N253ejdtuG36iZKzEejtsVc2+KFqos5lqe2NgCto5ZbhWXRLgcmAJjZJ0Aa0CRq/UjguSL7bFXKPp07rHvLLF7+6SncNeIENu7cxwUPfczYZ2exbsfeeDfNuYQRy0QyHegsqb2kVIKkMLFInbXAUABJXQkSybZwOQJcSNg/AmBmm4DdkvqHV2tdCrwawxhcAohExA/7tGbKLwdz7dDOvLN4C0PveZ8/v7mEnDzvP3GuomJ2asvM8iWNBSYDScCjZrZQ0m3ADDObCNwA/FPSdQQd72Ps606bQcA6M1tVZNfXAI8DdYH/hJNzpUqvk8z1Z3Vh5MmtuWvyUh6aupLnp6/jmsEdaXPILxd2rrxi2kdiZpOASUXKbomaXwScWsK2U4H+xZTPALpXakNdrdKiQV3uvagXY05px18mL+H2NxbTKE18mbmWEb1bkZLk9+k6dzT8P8bVWj1bN+CZK/rz7BX9aFhH3PTSfM66531enbPBh6p37ih4InG13imdmnBz/zQeubQPaSlJ/Hz8HIbdN41X52wg/1BBvJvnXLXnicQ5gpGFz+yWzaRrT+Nvo07EDH4+fg5n3vM+z09fy4F8TyjOlcQTiXNRIhFxXs8WTP7FIB7+0UlkpCXz6xfnM/iuKTzx8Rp/drxzxfBE4lwxIhExrHtzXhs7kMcuO5kWDery+4kLGfjn9/jrO8vYnuvPQHGuULzvbHeuWpPEkGObMbhLUz5bvYNx01bx13eW8+DUlfygV0suP609XbIz491M5+LKE4lzZSCJ/h0a079DY1ZszeWxj1bz4qz1PD9jHad1bsLlA9szqHNTIhF/qoGrfTyROHeUOjXL4I8/6MEvv3Msz36+lic+XsOYx6bTrnE9Lu7XhhG9W9MoPTXezXSuyngfiXPl1DA9lf8Z0okPf30G943sRdPMOtwxaQn973iXX4yfzfQ1O/zxv65W8CMS5yooNTnC8F4tGd6rJUs35/DsZ1/w0qwNvDJnI12yM7i4bxuG92pJQz9KcQnKj0icq0THHpPJH4Z357PfDuXPF/SgTnISt762iL53vMN/PzWDtxZu5qDf5OgSjB+ROBcD9VKTuejkNlx0chsWbdzNi7PW8+qcDUxeuIVG6amc17MFI3q34vgW9Ut82JZzNYUnEudirFuL+nRr0Y0bzz6Oacu28eKs9Tz72Voe/3gNXbIzOPeEFpx7QnM6NK1dT+9zicMTiXNVJCUpwtCu2Qztms3OvQd4bd4mJs7ZwD1vL+Oet5fRtXl9zj2hOeee0Jy2jdPj3VznyswTiXNx0KBeKqP7t2V0/7Zs3pXHG/M38ca8jdw1eSl3TV5Kj5ZZfO+E5pzVLZuOfqTiqjlPJM7F2TFZaVw+sD2XD2zPhp37mDRvE6/P38Sd/1nCnf9ZQoem6ZzVLZuzumZzYpuGJPlNj66a8UTiXDXSskFdrhzUgSsHdWDjzn28s3gLby/awqMfruYf76+icXoqZxzXjDO7ZTOwUxPS6/i/sIs//yt0rppq0aAulw5ox6UD2rE77yDTlm3j7UVbmLxwM/+euZ6UJNG7bUMGdWnKoM5N6da8vg/R4uIipolE0jDgPoJntj9iZncWWd8GeAJoENa5MXw8L5JOAP4B1AcKgJPNLE/SKOA3BM943wj8yMy2xzIO5+KtflpKeHVXCw4eKmD6mh28v2wb05Zt5y9vLuUvby6lSUYqAzs1YVCXppzWuSlNM+vEu9mulohZIpGUBDwAnAWsB6ZLmhg+p73QzcAEM3tIUjeC57u3k5QMPA2MNrO5khoDB8Py+4BuZrZd0l+AscCtsYrDueomJSnCKR2bcErHJtx0NmzNyeODZduZtnwbHyzfzitzNgLQJTuD/h0a0699Y/p1aESTDE8sLjZieUTSF1hhZqsAJI0HhgPRicQIjjgAsgiOMAC+A8wzs7kAZvZluI8UQEC6pC/DbVfEMAbnqr1mmWlc0LsVF/RuRUGBsWjTbqYt38anq3bwwsz1PPnJFwB0bpZBvw6NDicX5yqLYjWonKQRwDAzuyJcHg30M7OxUXWaA28BDYF04EwzmynpF0BvoBnQFBhvZn+J2u+jwB5gOTDEzL712DpJVwFXAWRnZ/ceP358ueLIzc0lI6N2XX7pMSeO/AJjze4Clu44xJIdBSz/6hB54X9L0zSjS6MUOjaI0KlBhJYZkYS+IixRf8dHUtGYhwwZMtPM+pRWL96d7aOAx83sbkkDgKckdQ/bNRA4GdgLvCtpJjAN+ClwIrAK+DtwE3B70R2b2ThgHECfPn1s8ODB5Wrg1KlTKe+2NZXHnLjyDxWwYONuPl31JZNnLmfJrggfbTwAQL3UJHq2asBJbRtwUpuGnNimYUINh19bfsfRqirmWCaSDUDrqOVWYVm0y4FhAGb2iaQ0oAlBn8q0wk50SZOAk4DdYd2VYfkE4MYYxuBcQklOitCrdQN6tW7AcbaO008/nXU79jFr7VfMXvsVs9bu5OH3V3GoIDhT0aphXbq3yKJHqyyOb1GfHi2zaOx9La6IWCaS6UBnSe0JEshI4OIiddYCQ4HHJXUF0oBtwGTgV5LqAQeA04F7w/10k9TUzLYRdOQvjmEMziU0SbRpXI82jetx/oktAdh34BDz1u9k9rqdLNiwiwUbdvHmws2Ht2melcbxLbLo0TKL7i3r061FfY6pn+aDT9ZiMUskZpYvaSxBUkgCHjWzhZJuA2aY2UTgBuCfkq4j6HgfY0GnzVeS7iFIRgZMMrM3ACT9AZgm6SDwBTAmVjE4VxvVTU2iX4fG9OvwdYf87ryDLNywm4Ubg8Qyf8Mu3l2yhcIu1sy0ZI7NzuTYY4KpS3Ymxx2TSYN6iXNqzJUspn0k4T0hk4qU3RI1vwg4tYRtnya4BLho+cPAw5XbUufckdRPS2FAx8YM6Ph1ctmzP59Fm3azZHMOSzfvZtnmXF6bu5FnPss/XKdZZp0guWRn0rFZBh2apNOhaQZNMlL9CCaBxLuz3TlXQ6XXSebkdo04uV2jw2Vmxpbd+1m6JUguSzfnsmxLDk99+gX7879+oFdmWvLhpHL4Z9N02jdJJy0lKR7huArwROKcqzSSOCYrjWOy0ji9S9PD5QUFxoad+1i1fQ+rtuWyatseVm/fw2ervuTl2Ruitofm9dNo1agebcKpdaO64c96NM2o40cy1ZAnEudczEUionWYDKITDMDeA/ms3r6HVduC6Ysde1i3Yy8fLN/Glt37v1E3LSVC64bBfto0qkerhnVpnlWX5g3SaJFVl6aZdRL6XpjqyhOJcy6u6qUmc3yLLI5vkfWtdXkHD7H+q32s+2ov63bsZe2Xe1n31V7W7tjH56t3kLs//xv1kyIiO7MOzRvUpXlWWjjVpUWDNDbuOkTX3Xk0Tk8lOSlSVeHVCp5InHPVVlpKEp2aZdCp2bfvzjYzdu07yKZdeWzatY+NO4Ofm3blsWlnHgs27OLtRVu+0Tdz2yfvEhE0Sq9D08xgahb+bJpRh2b1g59NM+vQrH4a6alJfiqtDDyROOdqJEk0qJdKg3qpdG1ev9g6ZsZXew+ycec+3v5oOk1ad2Jbzn625e5nW85+tubsZ/mWHLbl7Ce/4NvDRdVNSaJJZiqN0uvQOD2VhvVSaZSeQsP01Kjlr6f6aSm1cih/TyTOuYQl6fCH/PZmyQwe0K7YegUFxs59B4Mkk7OfrTl5h+e35e5nx54DbM3JY8mm3Xy558A3jnKiRcTh5NIwPZWsuimHp/ppKWTVTSarXuF8CvWj1tdJjtTYox9PJM65Wi8S+TrhHHtMZqn19x04xI69B9iRe4Adew/w1Z4D7CicwuUv9xxg3Y69LNx3kF37DrLnwLfGlv2G1KRImFiSDyeY9DrJZNZJJj2cvp5PIjMtmfTUsDwt+JlRJzkuCckTiXPOHaW6qUm0TK1LywZ1y7zNwUMF5OTls2vfQXaHyWXXvoPszgt/7vt63e68g+zYc4C1O/ayZ38+uXn5pSaiQskRHU4qys/j7VMOUTc1tvfmeCJxzrkqkJIUOXzUUx4FBcbeg4fIzcsnd39+kGD2lzCfl0/u/kOs2bCJOsmxv0LNE4lzztUAkYjICI80ymrq1K+qpPPfL6Z2zjlXIZ5InHPOVYgnEueccxXiicQ551yFeCJxzjlXIZ5InHPOVYgnEueccxXiicQ551yFyOzbI14mGknbgC/KuXkTYHslNqcm8Jhrh9oWc22LFyoec1sza1papVqRSCpC0gwz6xPvdlQlj7l2qG0x17Z4oepi9lNbzjnnKsQTiXPOuQrxRFK6cfFuQBx4zLVDbYu5tsULVRSz95E455yrED8icc45VyGeSJxzzlWIJ5ISSBomaamkFZJujHd7KkLSo5K2SloQVdZI0tuSloc/G4blkvS3MO55kk6K2ubHYf3lkn4cj1jKSlJrSVMkLZK0UNLPw/KEjVtSmqTPJc0NY/5DWN5e0mdhbM9LSg3L64TLK8L17aL2dVNYvlTSd+MTUdlISpI0W9Lr4XJCxwsgaY2k+ZLmSJoRlsXvb9vMfCoyAUnASqADkArMBbrFu10ViGcQcBKwIKrsL8CN4fyNwJ/D+XOA/wAC+gOfheWNgFXhz4bhfMN4x3aEmJsDJ4XzmcAyoFsixx22PSOcTwE+C2OZAIwMyx8GfhrOXwM8HM6PBJ4P57uFf/N1gPbh/0JSvOM7QtzXA88Cr4fLCR1v2OY1QJMiZXH72/YjkuL1BVaY2SozOwCMB4bHuU3lZmbTgB1FiocDT4TzTwDnR5U/aYFPgQaSmgPfBd42sx1m9hXwNjAs9q0vHzPbZGazwvkcYDHQkgSOO2x7briYEk4GnAG8EJYXjbnwvXgBGCpJYfl4M9tvZquBFQT/E9WOpFbA94BHwmWRwPGWIm5/255IitcSWBe1vD4sSyTZZrYpnN8MZIfzJcVeY9+T8BTGiQTf0BM67vA0zxxgK8EHw0pgp5nlh1Wi2384tnD9LqAxNSvmvwK/AgrC5cYkdryFDHhL0kxJV4VlcfvbLvtT5F3CMjOTlJDXgUvKAF4EfmFmu4MvoIFEjNvMDgG9JDUAXgaOi3OTYkbSucBWM5spaXC821PFBprZBknNgLclLYleWdV/235EUrwNQOuo5VZhWSLZEh7eEv7cGpaXFHuNe08kpRAkkWfM7KWwOOHjBjCzncAUYADBqYzCL43R7T8cW7g+C/iSmhPzqcB5ktYQnH4+A7iPxI33MDPbEP7cSvCFoS9x/Nv2RFK86UDn8OqPVIKOuYlxblNlmwgUXqXxY+DVqPJLwys9+gO7wsPlycB3JDUMrwb5TlhWLYXnvv8FLDaze6JWJWzckpqGRyJIqgucRdA3NAUYEVYrGnPhezECeM+CXtiJwMjwKqf2QGfg86qJouzM7CYza2Vm7Qj+R98zs0tI0HgLSUqXlFk4T/A3uYB4/m3H++qD6joRXOmwjOAc82/j3Z4KxvIcsAk4SHAe9HKCc8PvAsuBd4BGYV0BD4Rxzwf6RO3nJwQdkSuAy+IdVykxDyQ4jzwPmBNO5yRy3MAJwOww5gXALWF5B4IPxhXAv4E6YXlauLwiXN8hal+/Dd+LpcDZ8Y6tDLEP5uurthI63jC+ueG0sPDzKZ5/2z5EinPOuQrxU1vOOecqxBOJc865CvFE4pxzrkI8kTjnnKsQTyTOOecqxBOJSwiSGocjoc6RtFnShqjl1DLu4zFJx5ZS538kXVJJbf5QUi9JEVXyCNOSfiLpmKjlUmNzrrz88l+XcCTdCuSa2f8VKRfB33xBsRtWMUkfAmMJ7vnYbmYNjnL7JAuGRClx32Y2p+Itde7I/IjEJTRJnRQ8k+QZgpu3mksaJ2mGgmd23BJVt/AIIVnSTkl3Kni2xyfhmEZIul3SL6Lq36ngGSBLJZ0SlqdLejF83RfC1+p1hGbeCWSGR09Phvv4cbjfOZIeDI9aCtv1V0nzgL6S/iBpuqQFkh4O716+COgFPF94RFYYW7jvHyl4lsUCSXeEZUeKeWRYd66kKZX8K3IJwBOJqw2OA+41s24WjFF0o5n1AXoCZ0nqVsw2WcD7ZtYT+ITgDuDiyMz6Av8LFCalnwGbzawb8P8IRh4+khuBHDPrZWaXSuoO/AA4xcx6EQyuOjKqXdPM7AQz+wS4z8xOBnqE64aZ2fMEd/JfFO7zwOHGBsOu3w4MCdt1qoLBD48U8++BoWH5D0qJxdVCnkhcbbDSzGZELY+SNAuYBXQleLBRUfvM7D/h/EygXQn7fqmYOgMJBhHEzAqHsTgaZwInAzMUDAl/OtAxXHeAYJC+QkMlfU4wXMbpwPGl7LsfwRhT283sIMEDoQaF60qK+SPgSUlX4J8Zrhg+jLyrDfYUzkjqDPwc6GtmOyU9TTAGU1EHouYPUfL/yv4y1DlaAh41s999ozAYsXafFQ6gJNUD7id4EuQGSbdTfCxlVVLMVxIkoHOBWZJOtOBBSM4B/u3C1T71gRxgt75+Slxl+wi4EEBSD4o/4jnMwocw6euhz98BLpTUJCxvLKlNMZvWJXig0/ZwNNgLotblEDxiuKjPgCHhPgtPmb1fSjwdLHiy3u+Ar6j+D31yVcyPSFxtMwtYBCwBviD40K9sfyc4FbQofK1FBE/jO5J/AfMkzQj7Sf4AvCMpQjBq89XAxugNzOxLSU+E+99EkCQKPQY8ImkfUY+NNbP1kn4HTCU48nnNzN6ISmLFuVfB8OoC3jKzBaXE4moZv/zXuUoWfignm1leeCrtLaCzff34V+cSih+ROFf5MoB3w4Qi4L89ibhE5kckzjnnKsQ7251zzlWIJxLnnHMV4onEOedchXgicc45VyGeSJxzzlXI/weHiBZCaFm6CAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "\n",
    "plt.title('Nesterov Accelerated Gradients')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set Learning rate adaptively using AdaGrad \n",
    "\n",
    "In a nutshell, in AdaGrad, we set the learning rate to a small value when past gradients\n",
    "value is high and to a high value when the past gradient value is less. So our learning rate\n",
    "value changes according to the past gradients updates of the parameter.(Refer the section AdaGrad in the book for a detailed explanation)\n",
    "\n",
    "The update equation of Adagrad is given as:\n",
    "\n",
    "\n",
    "$${\\theta_{t}^i = \\theta_{t-1}^i - \\dfrac{\\eta}{\\sqrt{ \\sum_{\\tau=1}^t (g_{\\tau}^i)^2   + \\epsilon}} \\cdot g_{t}^i } -- (12) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def AdaGrad(data, theta, lr = 1e-2, epsilon = 1e-8, num_iterations = 100):\n",
    "\n",
    "    loss = []\n",
    "    \n",
    "    #initialize gradients_sum for storing sum of gradients\n",
    "    gradients_sum = np.zeros(theta.shape[0])\n",
    "\n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients with respect to theta\n",
    "        gradients = compute_gradients(data, theta)\n",
    "    \n",
    "        #compute square of sum of gradients\n",
    "        gradients_sum += gradients ** 2\n",
    "    \n",
    "        #update gradients\n",
    "        gradient_update = gradients / (np.sqrt(gradients_sum + epsilon))\n",
    "    \n",
    "        #update model parameter according to equation (12)\n",
    "        theta = theta - (lr * gradient_update)\n",
    "\n",
    "        loss.append(loss_function(data,theta))\n",
    "\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update model parameter using AdaGrad and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = AdaGrad(data, theta) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYXVWd7vHve2pMDanMRUZCSBgihAAhECYTwdvgAE4NRGSwRdpWtFVu34brcNVr3x4UxwbsqEijSEzbDlFQaJGCRsaEhABBIAOZyUjmoabf/WPvSg5FVapSVScnVef9PM95ag9rr7NWDtRba6999lZEYGZm1lWZfDfAzMx6NweJmZl1i4PEzMy6xUFiZmbd4iAxM7NucZCYmVm3OEjM+jhJr0q6MN/tsL7LQWJ2EJLqJL0uqSxr252SQtKlrcp+M91+bbpeKukWSasl7Ux/oX+r1TFXSHpS0i5JG9Llj0vSYemgWQ9wkJi1Q9JY4DwggEta7X4ZuDqrbDFwGbA0q8zNwBRgKlANTAeeyTrmRuDbwNeAo4Ba4GPAOUBpO20q6nKHzHLEQWLWvquBJ4A7gWta7fsNcK6kgen6RcAi4LWsMmcAv4yItZF4NSLuApBUA3wF+HhE/DwidqRlFkTElRGxLy13p6TbJd0naRcwQ9I7JS2QtF3SKklfym6YpKskrZC0WdLnevRfxKwNDhKz9l0N3J2+/kJSbda+vcCvgSuyyt7V6vgngM+mp6pObnW6ahpQltbRkQ8C/0AyqnkU2JW+3wDgncDfSHoPgKSJwO3AVcAIYDAwqlO9NesiB4lZGySdCxwNzImI+SSnrD7YqthdwNWSBgBvBX7Vav8/Av8MXAnMA9ZIahnZDAE2RURj1ns+JmmrpD2Szs+q59cR8aeIaI6IvRFRFxHPpeuLgHvS9wf4APDbiHgkHdV8AWju3r+G2cE5SMzadg3wQERsStd/SqvTWxHxKDAU+BzJL+89rfY3RcStEXEOyejhH4A7JJ0IbAaGpHMrLeXPjogB6b7s/zdXZdcr6UxJD0naKGkbybzKkHT3iOzyEbErrc8sZxwkZq1I6kcycf5WSa9Jeg34DHCKpFNaFf8JcCNvPq31BhGxJyJuBV4HJgKPA/uASw92XMvhrdZ/CswFRkdEDfA9oOW02TpgdFZfKkhOb5nlTHHHRcwKznuAJuBkoD5r+xyyrtRKfQf4b+CR1pVI+jSwEHgSaCA5xVUNLIiIrZK+DNyWzp3cTzL3MQmo7KB91cCWiNgraSrJKbcH0n0/B55MT809RTKh7z8YLaccJGZvdg3wo4hYmb1R0r+SBMcfWrZFxBbgwXbq2Q3cAownGVW8DLw/Ipalx/6LpDXA/yIZ0ewClgF/Dzx2kPZ9HLglbc/DJAE3IK3zBUmfIBm1VALfAFZ3uudmXSA/2MrMzLrDQ14zM+sWB4mZmXWLg8TMzLrFQWJmZt1SEFdtDRkyJMaOHdulY3ft2kVlZUdXY/YthdhnKMx+F2KfoTD73ZU+z58/f1NEDO2oXEEEydixY5k3b16Xjq2rq2P69Ok926AjXCH2GQqz34XYZyjMfnelz5JWdKacT22ZmVm3OEjMzKxbHCRmZtYtDhIzM+sWB4mZmXWLg8TMzLrFQWJmZt3iIDmIf3/sVZ5Y19hxQTOzAlYQX0jsqtlPr6Ks0UFiZnYwHpEcxLDqMrbt8/NazMwOJqdBIukiSS9JWiLppjb2j5H0kKQFkhZJeke6/UpJC7NezZImp/tmSnouLf97SUNy1f6h1WVsq3eQmJkdTM6CRFIRcCtwMTARmClpYqtinwfmRMSpwBXAbQARcXdETI6IycBVwPKIWCipGPg2MCMiJgGLgBty1Yeh6YikudlhYmbWnlyOSKYCSyJiWUTUA7OBS1uVCaB/ulwDrG2jnpnpsQBKX5WSlB7b1jE9Ylh1GU0BW/c05OotzMx6vVxOto8EVmWtrwbObFXmS8ADkj4JVAIXtlHP5aQBFBENkv4GeA7YBbwCfKKtN5d0PXA9QG1tLXV1dYfcgY3pFVv3/vFRRlcXznTSzp07u/Tv1dsVYr8Lsc9QmP3OZZ/zfdXWTODOiLhF0jTgx5JOiohmAElnArsj4vl0vQT4G+BUYBnwXeBm4KutK46IWcAsgClTpkRXbhldsXwLtz37OGNPOJnzJnR4S/4+oxBvsQ2F2e9C7DMUZr9z2edc/pm9BhidtT4q3ZbtI8AcgIh4HCgHsifPrwDuyVqfnJZdGhGRHnt2zzb7gKHVZQBs2L4vV29hZtbr5TJIngYmSDpGUilJKMxtVWYlcAGApBNJgmRjup4BLuPA/AgkQTRRUsvw4O3Ai7nqwLA0SDbudJCYmbUnZ6e2IqJR0g3A/UARcEdEvCDpK8C8iJgL3Ah8X9JnSCber01HGgDnA6siYllWnWslfRl4RFIDsAK4Nld9qCwrpqzIIxIzs4PJ6RxJRNwH3Ndq2xezlhcD57RzbB1wVhvbvwd8r0cbehADyuQRiZnZQRTOpUhdVFMmNmzfm+9mmJkdsRwkHajxiMTM7KAcJB2oKRUbPUdiZtYuB0kHBpSJHfsa2VPflO+mmJkdkRwkHagpEwAbd3hUYmbWFgdJB/YHyU5PuJuZtcVB0oGWIPF3SczM2uYg6cCAsuSfyFdumZm1zUHSgepSyMgjEjOz9jhIOpCRGFJV5sl2M7N2OEg6YWh1GRt2eLLdzKwtDpJOGFZd5jkSM7N2OEg6YWh1medIzMza4SDphGHV5WzeVU9Tc3Rc2MyswDhIOmFodRlNzcGWXfX5boqZ2RHHQdIJ+5+U6Cu3zMzexEHSCfuf3e4rt8zM3sRB0gnDqssBj0jMzNriIOmEAyMSB4mZWWsOkk7oV1pEdVmxRyRmZm1wkHTS0GrfJsXMrC0Okk4a1r+M17Z7st3MrDUHSSeNHljBqi27890MM7MjjoOkk8YMqmDDjn1+druZWSsOkk4aM7gCgNWve1RiZpYtp0Ei6SJJL0laIummNvaPkfSQpAWSFkl6R7r9SkkLs17Nkian+0olzZL0sqQ/S3p/LvvQYvSgJEhW+vSWmdkbFOeqYklFwK3A24HVwNOS5kbE4qxinwfmRMTtkiYC9wFjI+Ju4O60npOBX0XEwvSYzwEbIuI4SRlgUK76kG2Mg8TMrE05CxJgKrAkIpYBSJoNXApkB0kA/dPlGmBtG/XMBGZnrf8VcAJARDQDm3q22W0bXFlKRWmRg8TMrBVF5ObW6JI+AFwUEdel61cBZ0bEDVllhgMPAAOBSuDCiJjfqp6lwKUR8bykAcBzwH8A04GlwA0Rsb6N978euB6gtrb29NmzZ7cu0ik7d+6kqqoKgM8/upuhFRn+9rTyLtXVW2T3uZAUYr8Lsc9QmP3uSp9nzJgxPyKmdFQulyOSzpgJ3BkRt0iaBvxY0knpSANJZwK7I+L5tHwxMAp4LCI+K+mzwNeBq1pXHBGzgFkAU6ZMienTp3epgXV1dbQce+LKeazcvJvp08/vUl29RXafC0kh9rsQ+wyF2e9c9jmXk+1rgNFZ66PSbdk+AswBiIjHgXJgSNb+K4B7stY3A7uBX6Tr/wGc1nNNPrjRAytYuWU3uRrFmZn1RrkMkqeBCZKOkVRKEgpzW5VZCVwAIOlEkiDZmK5ngMvImh+J5Df4b0hOa5Eemz3nklNjBvVjT0MTm3b6AVdmZi1ydmorIhol3QDcDxQBd0TEC5K+AsyLiLnAjcD3JX2GZOL92jjw5/75wKqWyfosf09yCuxbJKHz4Vz1obWW75Ks3LJ7/x2BzcwKXU7nSCLiPpJLerO3fTFreTFwTjvH1gFntbF9BUnIHHYtlwCv2rKb048emI8mmJkdcfzN9kMwaqC/S2Jm1pqD5BCUlxRR27/MQWJmlsVBcojGDPJdgM3MsjlIDtFoB4mZ2Rs4SA7RmEEVrNu+l32Nvp28mRk4SA7Z6IEVRMCa1/fkuylmZkcEB8khyv4uiZmZOUgOWfZ3SczMzEFyyIZWlVFWnPGIxMws5SA5RJmMGD2owkFiZpZykHTBmEEVrNjsIDEzAwdJl4wfVsWyTbtobGrOd1PMzPLOQdIFx9VWU9/YzAqf3jIzc5B0xXG1yeMqX35tR55bYmaWfw6SLhg/rAoJXl6/M99NMTPLOwdJF1SUFjNmUAUvr/eIxMzMQdJFE4ZVO0jMzHCQdNnxR1WxfNMu6ht95ZaZFTYHSRcdV1tNY3OwfNOufDfFzCyvHCRddFxtNQAv+fSWmRU4B0kXjRtaSVFGvOIgMbMC5yDporLiIsYOruAlf5fEzAqcg6Qbjj+qmlc2+LskZlbYHCTdMGFYNa9u3sXeBj9218wKl4OkG44/qpoIWOJRiZkVsJwGiaSLJL0kaYmkm9rYP0bSQ5IWSFok6R3p9islLcx6NUua3OrYuZKez2X7O7L/nluecDezApazIJFUBNwKXAxMBGZKmtiq2OeBORFxKnAFcBtARNwdEZMjYjJwFbA8IhZm1f0+IO/DgKMHV1JalPElwGZW0HI5IpkKLImIZRFRD8wGLm1VJoD+6XINsLaNemamxwIgqQr4LPDVHm/xISopyjBuaCWv+OaNZlbAFBG5qVj6AHBRRFyXrl8FnBkRN2SVGQ48AAwEKoELI2J+q3qWApdGxPPp+jeBR4AFwG8j4qR23v964HqA2tra02fPnt1WsQ7t3LmTqqqqdvd/79m9vPJ6M7dMr+hS/UeijvrcVxVivwuxz1CY/e5Kn2fMmDE/IqZ0VK64y63qGTOBOyPiFknTgB9LOikimgEknQnszgqRycCxEfEZSWMPVnFEzAJmAUyZMiWmT5/epQbW1dVxsGNfySzjifte5KQp0xhSVdal9zjSdNTnvqoQ+12IfYbC7Hcu+5zLU1trgNFZ66PSbdk+AswBiIjHgXJgSNb+K4B7stanAVMkvQo8Chwnqa5HW32IJo2qAWDR6q35bIaZWd7kMkieBiZIOkZSKUkozG1VZiVwAYCkE0mCZGO6ngEuI2t+JCJuj4gRETEWOBd4OSKm57APHTppZA0ZwbOrtuWzGWZmeZOzIImIRuAG4H7gRZKrs16Q9BVJl6TFbgQ+KulZkpHHtXFg0uZ8YFVELMtVG3tCZVkxx9VW86xHJGZWoHI6RxIR9wH3tdr2xazlxcA57RxbB5x1kLpfBdqcaD/cJo2q4b8WrycikJTv5piZHVb+ZnsPOGX0AF7f3cDq1/fkuylmZoedg6QHnDJqAAALV/n0lpkVHgdJDzj+qGpKizM86yAxswLkIOkBJUUZThrRn0WrfeWWmRUeB0kPmTRqAM+t2UZjU3O+m2Jmdlg5SHrI5NED2NPQ5AddmVnBcZD0kFNGJxPu/oa7mRUaB0kPGTu4gv7lxSz0N9zNrMA4SHqIJE4ZPcAjEjMrOA6SHjR59AD+/NoOdu5rzHdTzMwOGwdJDzrzmME0NQfzXt2S76aYmR02DpIedPrRAykpEo8v25zvppiZHTadChJJP+7MtkLXr7SIyaMH8MQyj0jMrHB0dkTyluwVSUXA6T3fnN7vrHGDeX7NNnbsbch3U8zMDouDBomkmyXtACZJ2p6+dgAbgF8flhb2MtPGtcyTvJ7vppiZHRYHDZKI+MeIqAa+FhH901d1RAyOiJsPUxt7ldOOHkhpUYYnPE9iZgWis6e2fiupEkDShyR9Q9LROWxXr1VeUsTkMQM84W5mBaOzQXI7sFvSKSSPx10K3JWzVvVyLfMk2z1PYmYFoLNB0pg+S/1S4F8j4lagOnfN6t2mjRtMc8DTy331lpn1fZ0Nkh2SbgauAu6VlAFKctes3u3UMQMoLfY8iZkVhs4GyeXAPuCvIuI1YBTwtZy1qpcrLyniNM+TmFmB6FSQpOFxN1Aj6V3A3ojwHMlBTBs3hBfWbuf1XfX5boqZWU519pvtlwFPAX8JXAY8KekDuWxYbzf9+KFEwMMvb8x3U8zMcqq4k+U+B5wRERsAJA0F/gD8PFcN6+1OHlnDkKoyHvzzBt5z6sh8N8fMLGc6O0eSaQmR1OZDOLYgZTJixvFDefilDTT4Oe5m1od1Ngx+L+l+SddKuha4F7ivo4MkXSTpJUlLJN3Uxv4xkh6StEDSIknvSLdfKWlh1qtZ0mRJFZLulfRnSS9I+qdD6ezhdsGJw9i+t5H5K3y7FDPruzq619Z4SedExN8B/wZMSl+PA7M6OLYIuBW4GJgIzJQ0sVWxzwNzIuJU4ArgNoCIuDsiJkfEZJJLjpdHxML0mK9HxAnAqcA5ki7ufHcPr3MnDKWkSPzxzxs6Lmxm1kt1NCL5FrAdICJ+ERGfjYjPAr9M9x3MVGBJRCyLiHpgNskXGrMF0D9drgHWtlHPzPRYImJ3RDyULtcDz5BcinxEqior5qxxg3nwxfX5boqZWc4o+cJ6OzulpyPijHb2PRcRJx/k2A8AF0XEden6VcCZEXFDVpnhwAPAQKASuDAi5reqZylwaUQ832r7AJIguTAilrXx/tcD1wPU1taePnv27Hb7eTA7d+6kqqqqS8cC/NerDdz953r++bx+1Fb2jmml7va5tyrEfhdin6Ew+92VPs+YMWN+REzpqFxHV20NOMi+fofUorbNBO6MiFskTQN+LOmkiGgGkHQmsLuNECkG7gG+01aIAETELNLTb1OmTInp06d3qYF1dXV09ViAYzbv4u4/17Gr5himn3tMl+s5nLrb596qEPtdiH2Gwux3Lvvc0Z/I8yR9tPVGSdcB89son20NMDprfVS6LdtHgDkAEfE4UA4Mydp/BUlgtDYLeCUiOjq9lndHD65k/LAqz5OYWZ/V0Yjk08AvJV3JgeCYApQC7+3g2KeBCZKOIQmQK4APtiqzErgAuFPSiSRBshEgvZ/XZcB52QdI+irJfMp1Hbz/EeOCE4Zxx5+Ws31vA/3LfYsyM+tbOnqw1fqIOBv4MvBq+vpyRExLb5tysGMbgRuA+4EXSa7OekHSVyRdkha7EfiopGdJRh7XxoFJm/OBVdmnriSNIvly5ETgmfTS4CM+UP7HW46ioSn4w2JPuptZ39Opb7anV0o9dKiVR8R9tPq+SUR8MWt5MXBOO8fWAWe12rYa0KG2I99OGzOAkQP6MffZtbzvtCP2IjMzsy7pHZcR9XKSeNcpw3n0lU1s8U0czayPcZAcJpecMoLG5uD3zx/0jKCZWa/jIDlMJg7vz7ihlcx9tvWFa2ZmvZuD5DCRxLsnjeDJ5VtYv31vvptjZtZjHCSH0btPGUEE3LtoXb6bYmbWYxwkh9H4YVVMHN6fuc+2dUsxM7PeyUFymF0yeQQLV21lxeZd+W6KmVmPcJAcZpdOHkFGMGfeqnw3xcysRzhIDrPhNf142wnDmDNvtZ+caGZ9goMkD2ZOHcPGHft48EXfyNHMej8HSR689bihHNW/nHueWpnvppiZdZuDJA+KizJcdsZoHnllI6u27M53c8zMusVBkieXn5E8quU/POluZr2cgyRPRg7ox/TjhvKzeato9KS7mfViDpI8mjl1DOu37+MPL/o5JWbWezlI8uiCE2sZPagfsx5p87HzZma9goMkj4oy4rpzx/HMyq3MX7El380xM+sSB0me/eWUUQyoKPGoxMx6LQdJnlWUFnPVWUfzwOL1LN/k+2+ZWe/jIDkCXD1tLCWZDD981KMSM+t9HCRHgKHVZbzvtJH8x7zVbN65L9/NMTM7JA6SI8R1542jvqmZHz66PN9NMTM7JA6SI8T4YVW8e9IIfvSnV9nkUYmZ9SIOkiPIpy+cwL7GJm6vW5rvppiZdZqD5AgybmgV7z9tFD95YgWvbdub7+aYmXVKToNE0kWSXpK0RNJNbewfI+khSQskLZL0jnT7lZIWZr2aJU1O950u6bm0zu9IUi77cLh96oIJNDUHtz60JN9NMTPrlJwFiaQi4FbgYmAiMFPSxFbFPg/MiYhTgSuA2wAi4u6ImBwRk4GrgOURsTA95nbgo8CE9HVRrvqQD6MHVXD5GaOZ/fRK32LezHqFXI5IpgJLImJZRNQDs4FLW5UJoH+6XAOsbaOememxSBoO9I+IJyIigLuA9+Si8fl0w9vGk5H42v0v5bspZmYdKs5h3SOB7IdtrAbObFXmS8ADkj4JVAIXtlHP5RwIoJFpPdl1jmzrzSVdD1wPUFtbS11d3aG1PrVz584uH9sdFx1dxK+fXctJ5Vs4bmDRYX3vfPU53wqx34XYZyjMfueyz7kMks6YCdwZEbdImgb8WNJJEdEMIOlMYHdEPH+oFUfELGAWwJQpU2L69OldamBdXR1dPbY7pp7dyFO3PMyvV5Uy99JzKcocvqmgfPU53wqx34XYZyjMfueyz7k8tbUGGJ21Pirdlu0jwByAiHgcKAeGZO2/ArinVZ2jOqizT6goLebmd5zIC2u3+ymKZnZEy2WQPA1MkHSMpFKSUJjbqsxK4AIASSeSBMnGdD0DXEY6PwIQEeuA7ZLOSq/Wuhr4dQ77kFfvnjScKUcP5Gv3v8T2vQ35bo6ZWZtyFiQR0QjcANwPvEhyddYLkr4i6ZK02I3ARyU9SzLyuDadRAc4H1gVEa3vZPhx4AfAEmAp8Ltc9SHfJPGlS97Clt313OKJdzM7QuV0jiQi7gPua7Xti1nLi4Fz2jm2Djirje3zgJN6tKFHsJNG1nDNtLHc+dirvOuUEZwxdlC+m2Rm9gb+Znsv8Hd/cTyjBvbj73++iL0NTflujpnZGzhIeoHKsmL+8X0ns2zTLr794Cv5bo6Z2Rs4SHqJ8yYM5S9PH8WsR5bx3Opt+W6Omdl+DpJe5PPvnMjgylI+/bMF7K5vzHdzzMwAB0mvUlNRwjcvn8yyTbv4ym8W57s5ZmaAg6TXOWf8ED721mOZ/fQqfruorVuTmZkdXg6SXuizbz+OyaMHcPMvnvMdgs0s7xwkvVBJUYbvzjwVAj5+9zO+JNjM8spB0kuNHlTBNy6fzHNrtnHzL57jwA0BzMwOLwdJL/b2ibXc+Pbj+OWCNfzgv5fnuzlmVqAcJL3cDW8bz8UnHcU//u5FHn55Y76bY2YFyEHSy0ni6395CsfVVvOJu5/h+TX+sqKZHV4Okj6gsqyYH334DPqXF3Ptj572lVxmdlg5SPqI4TX9uOsjU2loaubqO55i8859+W6SmRUIB0kfMn5YNXdcO4V12/ZwzY+eYttuPwzLzHLPQdLHnH70IG7/0Om8/NpOPvTDJx0mZpZzDpI+aMbxw/jeVafx0ms7HCZmlnMOkj7qbSfU7g+TD/7gCTZ5zsTMcsRB0oe97YRa/u3q01m6cScfuP0xVm721Vxm1vMcJH3cjOOH8dOPnsXWPQ287/bH/D0TM+txDpICcNqYgfz8Y9MoLRKX/9vj/Nfi9flukpn1IQ6SAjF+WDW/+Pg5HDusiut/PI9bH1riGz2aWY9wkBSQo2rKmfPX07j0lBF87f6X+OQ9C9i5z4/sNbPucZAUmPKSIr55+WRuuvgE7ntuHZd891EWr92e72aZWS/mIClAkvjYW4/lpx89i131jbzntj/x4ydW+FSXmXVJToNE0kWSXpK0RNJNbewfI+khSQskLZL0jqx9kyQ9LukFSc9JKk+3z0zXF0n6vaQhuexDX3bWuMHc96nzmDZuMF/41fN8+M6nWb99b76bZWa9TM6CRFIRcCtwMTARmClpYqtinwfmRMSpwBXAbemxxcBPgI9FxFuA6UBDuv3bwIyImAQsAm7IVR8KweCqMn507Rl8+ZK38MSyzbz9Gw/z2NpGj07MrNNyOSKZCiyJiGURUQ/MBi5tVSaA/ulyDbA2Xf4fwKKIeBYgIjZHRBOg9FUpSemxa7FuyWTENWeP5Xd/ez7jh1Uxa9E+rr7jKV7dtCvfTTOzXkC5+stT0geAiyLiunT9KuDMiLghq8xw4AFgIFAJXBgR8yV9GjgdGAYMBWZHxL9k1XsHsAt4hWR00tTG+18PXA9QW1t7+uzZs7vUj507d1JVVdWlY3uj5gjue2UXv10hGgPePa6Ei48pobRI+W5azhXaZw2F2WcozH53pc8zZsyYHxFTOipX3OVW9YyZwJ0RcYukacCPJZ2Ututc4AxgN/CgpPnAI8DfAKcCy4DvAjcDX21dcUTMAmYBTJkyJaZPn96lBtbV1dHVY3urjOr4uyvO4iu/XcwvF63jqU3F/P3Fx/PuScNJBoJ9UyF+1oXYZyjMfueyz7k8tbUGGJ21Pirdlu0jwByAiHgcKAeGAKuBRyJiU0TsBu4DTgMmp2WXRjKUmgOcncM+FKza/uXc+sHT+Ol1Z1LTr4RP3bOA9972GI8v3ZzvppnZESaXQfI0MEHSMZJKSSbT57YqsxK4AEDSiSRBshG4HzhZUkU6wf5WYDFJEE2UNDQ9/u3AiznsQ8E7e/wQfvPJc/mXD0xi3bY9zPz+E3zw+08wf8WWfDfNzI4QOTu1FRGNkm4gCYUi4I6IeEHSV4B5ETEXuBH4vqTPkEy8X5uONF6X9A2SMArgvoi4F0DSl4FHJDUAK4Brc9UHSxRlxGVTRnPJKSP4yRMruL1uKe+//XHOGT+Yj08fz9nHDu7Tp7zM7OByOkcSEfeRnJbK3vbFrOXFwDntHPsTkkuAW2//HvC9nm2pdUZ5SRHXnTeOmVPH8JMnVvCDR5dz5Q+eZNKoGq47bxwXn3QUJUX+jqtZocn3ZLv1QpVlxfz1W4/lmrPH8ssFa5j1yDI+dc8CavuXcdVZR3P5GWMYWl2W72aa2WHiILEuKy8pYubUMVw+ZTQPv7yRO/60nK8/8DLf+sMrXHhiLZdPHc35E4ZSlPFpL7O+zEFi3ZbJiBknDGPGCcNYunEns59ayX8+s4bfv/Aatf3LuHTySN576khOHN6/48rMrNdxkFiPOnZoFZ9750T+518czx8Wb+CXC1Zzx6PLmfXIMiYMq+IdJw/nnZOGc1xtdb6bamY9xEFiOVFWXMQ7JyWhsWVXPfcuWstvF63jO398hW8/+Arjhlby9hNruXBiLaeNGejTX2a9mIPEcm5QZSlXTRvLVdPGsmHHXu5//jUeWLyeO/60nH97ZBk1/Uo4d8IQ3nrcUM6fMJSjasrz3WSj2SRnAAAMrElEQVQzOwQOEjushlWX7w+VHXsbePjljdS9tJGHX97IvYvWATBuSCXTjh3MtGMHM3XsIIb1d7CYHckcJJY31eUlvGvSCN41aQQRwYvrdvDY0k08tnQzv1qwhrufXAnAmEEVTBk7kFPHDOTU0QM4/qhqf1/F7AjiILEjgiQmjujPxBH9ue68cTQ0NfPC2u3Me3ULT7+6hUde3sgvnklu1VZekuHE4f05eWQNJ42sYeLw/kyoraKsuCjPvTArTA4SOyKVFGWYPHoAk0cP4LrzxhERrH59DwtWbWXhyq08v2Yb/zl/NXc9vgKA4owYP6yK42qrOa42+Tl+WBVjBlVQ7NGLWU45SKxXkMToQRWMHlTBJaeMAKC5OVi+eRcvrtvO4rXbeXHdduaveJ25zx541llJkTh6cCXHDEleYwdXMnZwUs/wmnKHjFkPcJBYr5XJiGOHVnHs0CreNWnE/u079zXyyvodLN24i6Ubd7Jkw05e3bSLh1/eSH1j8/5yxRkxfEA5owZUMHJgPxq21rO+ciXDa/oxvKac2ppyqsuKfUNKsw44SKzPqSorTibmxwx8w/am5mDdtj2s3LKbVVt2s2LzbtZs3cPq1/fwyMsb2bijgV8vfe4Nx1SUFnFU/3KGVJcxrLqModVlDKlq+VnKoMoyBleWMriqlH4lRQ4dK0gOEisYRRkxamAFowZWwLFv3v+HPz7E8ZPPZO3WPby2fS/rt+/ltW37WL9jLxu37+P5NdvYsGMfu+vf9GRnAMqKMwyqLGVARSkDK0oYUFFCTb9SavqVUNOvhP79iulfXkL/fiVUlxfTv7yY6vJk2SFkvZmDxCxVnDkwD3Mwu+sb2bSjnk279rFlZz1bdtWzeVc9W3cny6/vrmfr7gZeXr+Trbsb2L6ngfqm5oPWmRFUlhZTVV5MZVkxlaVFVJQWU1l24Ge/kmL6lWaoKE2Cp7ykiH6lGfqVFFFWUkR5cRH9SosoK85QVpyhvCRdLimitChDSZEcVpYTDhKzQ1RRWsyYwcWMGXzwwGkREextaGbbngZ27G1g+96GdLmRnfsa2bG3kV3pz537Gtld38iufU3s2tfI2q0NyXp9E3vqm9hd30hzdK3dEpQWZShNg6a0KENTwz76P/MwpcUZSoqSbSXFoqQok75EcSZDcZEoaflZlKE4I4r3/xTFGVGUaXs9kxFFGcgoqatluSiT9VJLOZGRyIisZZHJQJGSIMzowPFKl1uOUTs/MzpQVoLG5qChqTlZT/9tHLJd5yAxyzFJ9CtNRgvdvf1LRLCvsZl9Dc3saUiCZW9DM3sbm9hb38Texib2NTSzr7GZvQ1N1Dc1p+tN1Dcm2/c1NlPf1Ex9YzOr165j4OAq6tNtDU3N7G1oZsfeRhqakl+2DU3NNDYFjc3Jz4amZhqbY/+2rgZb3j3wuzdtkkiDJQ0ikg37w4asQIKsfXrDsekulBZqyagD2w7UlX1MdjsOtEdvaBut3uPAQW9elMS9nzo359+xcpCY9SKSKE9Pa9VQ0u366upeZ/r007tVR3Nz0NgcNDUfCJumCJqbg4bm5GdTVpmm5qA50uXI2pa1HsH+/RHsLx8kYZrUkWxvTre/YTndn10+WU7KLV22jGPGHpOWSd6jpWyk25qD/cu8aX9ST6QhGnGgfmhdZ8u/VHb59sqk75cstCztrx+y3yPrmKwKotWC3hg3OeEgMbNuyWRE6f67N/eOuwvUaTXTp0/IdzP6DH8by8zMusVBYmZm3eIgMTOzbnGQmJlZtzhIzMysWxwkZmbWLQ4SMzPrFgeJmZl1i7K/EdlXSdoIrOji4UOATT3YnN6gEPsMhdnvQuwzFGa/u9LnoyNiaEeFCiJIukPSvIiYku92HE6F2GcozH4XYp+hMPudyz771JaZmXWLg8TMzLrFQdKxWfluQB4UYp+hMPtdiH2Gwux3zvrsORIzM+sWj0jMzKxbHCRmZtYtDpJ2SLpI0kuSlki6Kd/tyRVJoyU9JGmxpBck/W26fZCk/5L0SvpzYL7b2tMkFUlaIOm36foxkp5MP/OfSSrNdxt7mqQBkn4u6c+SXpQ0ra9/1pI+k/63/bykeySV98XPWtIdkjZIej5rW5ufrRLfSfu/SNJp3XlvB0kbJBUBtwIXAxOBmZIm5rdVOdMI3BgRE4GzgE+kfb0JeDAiJgAPput9zd8CL2at/zPwzYgYD7wOfCQvrcqtbwO/j4gTgFNI+t9nP2tJI4FPAVMi4iSSRzheQd/8rO8ELmq1rb3P9mJgQvq6Hri9O2/sIGnbVGBJRCyLiHpgNnBpntuUExGxLiKeSZd3kPxiGUnS339Pi/078J78tDA3JI0C3gn8IF0X8Dbg52mRvtjnGuB84IcAEVEfEVvp4581ySPF+0kqBiqAdfTBzzoiHgG2tNrc3md7KXBXJJ4ABkga3tX3dpC0bSSwKmt9dbqtT5M0FjgVeBKojYh16a7XgNo8NStXvgX8L6A5XR8MbI2IxnS9L37mxwAbgR+lp/R+IKmSPvxZR8Qa4OvASpIA2QbMp+9/1i3a+2x79Hecg8QAkFQF/Cfw6YjYnr0vkmvE+8x14pLeBWyIiPn5bsthVgycBtweEacCu2h1GqsPftYDSf76PgYYAVTy5tM/BSGXn62DpG1rgNFZ66PSbX2SpBKSELk7In6Rbl7fMtRNf27IV/ty4BzgEkmvkpy2fBvJ3MGA9PQH9M3PfDWwOiKeTNd/ThIsffmzvhBYHhEbI6IB+AXJ59/XP+sW7X22Pfo7zkHStqeBCemVHaUkk3Nz89ymnEjnBn4IvBgR38jaNRe4Jl2+Bvj14W5brkTEzRExKiLGkny2f4yIK4GHgA+kxfpUnwEi4jVglaTj000XAIvpw581ySmtsyRVpP+tt/S5T3/WWdr7bOcCV6dXb50FbMs6BXbI/M32dkh6B8l59CLgjoj4hzw3KScknQv8N/AcB+YL/jfJPMkcYAzJLfgvi4jWE3m9nqTpwP+MiHdJGkcyQhkELAA+FBH78tm+niZpMskFBqXAMuDDJH9Q9tnPWtKXgctJrlBcAFxHMh/Qpz5rSfcA00luF78e+D/Ar2jjs01D9V9JTvPtBj4cEfO6/N4OEjMz6w6f2jIzs25xkJiZWbc4SMzMrFscJGZm1i0OEjMz6xYHifUJkgZLWpi+XpO0Jmu9U3d2lfSjrO9YtFfmE5Ku7KE2PyppsqRMT99hWtJfSToqa73Dvpl1lS//tT5H0peAnRHx9VbbRfLffHObBx5mkh4FbgCeBzZFxIBDPL4oIpoOVndELOx+S80OziMS69MkjVfyrJW7gReA4ZJmSZqXPqPii1llW0YIxZK2SvonSc9KelzSsLTMVyV9Oqv8P0l6Ssmza85Ot1dK+s/0fX+evtfkgzTzn4DqdPR0V1rHNWm9CyXdlo5aWtr1LUmLgKmSvizpaSXP2vhe+k3ly4HJwM9aRmQtfUvr/pCk59Jj/l+67WB9viIt+6ykh3r4I7I+wEFiheAEkmdPTEzvBntTREwheR7H29X2s2ZqgIcj4hTgceCv2qlbETEV+DugJZQ+CbyWPuPl/5LcUflgbgJ2RMTkiLha0knAe4GzI2Iyyc0Wr8hq1yMRMSkiHge+HRFnACen+y6KiJ8BC4HL0zrr9zc2uX3+V4EZabvOUXITy4P1+f8AF6Tb39tBX6wAOUisECxtdfuHmZKeAZ4BTiR5eFlreyLid+nyfGBsO3X/oo0y55LcfoOIeJZkJHQoLgTOAOZJWgi8FTg23VcP/DKr7AWSngKeTcu9pYO6zyS5t9im9CaGPyV5Rgm03+c/AXdJug7/zrA2FHdcxKzX29WyIGkCyZMRp0bEVkk/AcrbOKY+a7mJ9v9f2deJModKJPd3+8IbNiZ3q92T3g4cSRUk90s6LSLWSPoqbfels9rr80dJAuhdwDOSTo2I17vxPtbH+K8LKzT9gR3AdiW31f6LHLzHn4DLACSdTNsjnv1aHrCkA7c1/wNwmaQh6fbBksa0cWg/khttbpJUDbw/a98OoLqNY54EZqR1tpwye7iD/oxLn6L3BZLH0vbVh0BZF3lEYoXmGZLbiP+Z5G6of8rBe3yX5FTQ4vS9FpM8me9gfggskjQvnSf5MvAHSRmgAfgYsDb7gIjYLOnf0/rXkYREix8BP5C0h+TR0S3HrJb0BaCOZOTzm4i4NyvE2vJNScek5R+IiOc76IsVGF/+a9bD0l/KxRGxNz2V9gAwIevRrmZ9ikckZj2vCngwDRQBf+0Qsb7MIxIzM+sWT7abmVm3OEjMzKxbHCRmZtYtDhIzM+sWB4mZmXXL/weJ7P5oUNTbXwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.title('AMSGrad')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Do away with Learning rate using Adadelta \n",
    "\n",
    "\n",
    "With AdaDelta we completely get rid of learning rate. (Refer the section AdaDelta in the book for full explanation)\n",
    "\n",
    "The update equation of AdaDelta is expressed as:\n",
    "\n",
    "Running average of gradients:\n",
    "$$E\\left[g^{2}\\right]_{t}=\\gamma E\\left[g^{2}\\right]_{t-1}+(1-\\gamma) g_{t}^{2} -- (13) $$\n",
    "\n",
    "Compute delta:\n",
    "$$ \\nabla \\theta_{t}=-\\frac{R M S[\\Delta \\theta]_{t-1}}{R M S\\left[g_{t}\\right]} \\cdot g_{t}^{i} -- (14) $$\n",
    "\n",
    "Running averge of parameter update:\n",
    "$$ E\\left[\\Delta \\theta^{2}\\right]_{t}=\\gamma E\\left[\\Delta \\theta^{2}\\right]_{t-1}+(1-\\gamma) \\Delta \\theta_{t}^{2} -- (15) $$\n",
    "\n",
    "Update theta:\n",
    "$$ \\theta_{t}^{i}=\\theta_{t-1}^{i}+\\nabla \\theta_{t} -- (16) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def AdaDelta(data, theta, gamma = 0.9, epsilon = 1e-5, num_iterations = 500):\n",
    "    \n",
    "    loss = []\n",
    "    \n",
    "    #initialize running average of gradients\n",
    "    E_grad2 = np.zeros(theta.shape[0])\n",
    "    \n",
    "    #initialize running average of parameter update\n",
    "    E_delta_theta2 = np.zeros(theta.shape[0])\n",
    "    \n",
    "    \n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients of loss with respect to theta\n",
    "        gradients = compute_gradients(data, theta)\n",
    "        \n",
    "        #compute running average of gradients as given in equation (13)\n",
    "        E_grad2 = (gamma * E_grad2) + ((1. - gamma) * (gradients ** 2))\n",
    "        \n",
    "        #compute delta_theta as given in equation (14)\n",
    "        delta_theta = - (np.sqrt(E_delta_theta2 + epsilon)) / (np.sqrt(E_grad2 + epsilon)) * gradients\n",
    "        \n",
    "        #compute running average of parameter updates as given in equation (15)\n",
    "        E_delta_theta2 = (gamma * E_delta_theta2) + ((1. - gamma) * (delta_theta ** 2))\n",
    "        \n",
    "        #update the model parameter, theta as given in equation (16)\n",
    "        theta = theta + delta_theta\n",
    "        \n",
    "        #store  the loss\n",
    "        loss.append(loss_function(data,theta))\n",
    "\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update model parameter using AdaDelta and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = AdaDelta(data, theta) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VdW5//HPN3PIyBgCCaOgoDKYCIoTOFSwVtuKCFrUOmAH2trSW/VXr62tt7eTtbbO1zpWpWi1pUoFB3BEBWQeBWSeZwIJIeT5/bE39hgDCUlOTobn/XqdF3uvvdY6z4KYx73X3mvLzHDOOedqKi7WATjnnGvcPJE455yrFU8kzjnnasUTiXPOuVrxROKcc65WPJE455yrFU8kztWCpGslvVuP37dK0vn19X3OVYcnEueOQNI0STslJddRfz+XdFDS3vCzTNJ9knJr0d9f6yI252rDE4lzlZDUBTgLMOCSOuz6b2aWAbQCvga0B2bVNJk41xB4InGuclcDHwBPANccLpTUWtJESXskfQR0j2wk6V5Ja8PjsySdVVnnZnbQzBYCVwBbgXERfVwsaY6kXZLel9SnYntJQ4H/B1whqUjS3LD8m5IWh2c8KyXdVNu/COeq4onEucpdDTwTfi6UlBOW3w+UALnAdeEn0gygH8EZx7PA85JSjvQlZnYI+CfB2Q+S+gOPATcBrYGHgYkVL6+Z2avArwjOcNLNrG94aAtwMZAJfBO4R9Ipxzx6546BJxLnKpB0JtAZmGBms4AVwJWS4oHLgDvMbJ+ZLQCejGxrZn81s+1mVmZmdwPJwPFVfOUGgsQDMAZ42Mw+NLNDZvYkcAA4rTqxm9krZrbCAm8BUwiTlHPR4onEuS+6BphiZtvC/WfDsrZAArA2ou7qyIaSfhxeWtotaReQBbSp4vs6AjvC7c7AuPCy1q6wj3ygQ3UClzRM0geSdoRtL6rG9ztXKwmxDsC5hkRSKjACiJe0KSxOBrKBHKCM4Bf7kvBYp4i2ZwE/Ac4DFppZuaSdgI7yfXHAV4DXw6K1wP+Y2f9UI9zPLd0dXv76O8FluX+a2UFJ/zja9ztXF/yMxLnP+ypwCOhNMNfRD+gFvEPwC/pF4OeSWkjqTcREPJBBkGi2AgmS7iCYq/gCSQmSegHPEdy59Yfw0P8B35I0UIE0SV+WlFFJN5uBLmEyAkgiSHpbgTJJw4Av1ehvwblj4InEuc+7BnjczNaY2abDH+A+4CpgLJAObCK4o+vxiLaTgVeBZQSXvEr4/GUwCO+yAnYDE4HtQIGZbQAws5nAjeH37QSWA9ceIdbnwz+3S/rYzPYC3wcmhG2vDL/DuaiSv9jKOedcbfgZiXPOuVqJaiKRNFTSUknLJd1ayfFOkqZKmi1pnqSLwvJESU9Kmh/eAXNbWJ4f1l8kaaGkH0Qzfuecc1WL2qWt8J77ZcAFwDqCB7VGmdmiiDqPALPN7MFw4nKSmXWRdCVwiZmNlNQCWAQMJrifPtfMPg4nH2cBX43s0znnXP2K5hnJAGC5ma00s1JgPHBphTrGf+5qySJ4MOtweZqkBCAVKAX2mNlGM/sYIJxYXExwD75zzrkYieZzJB35/B0r64CBFer8HJgi6XtAGnB4eewXCJLORqAF8EMz2xHZMFxUrz/wYWVfLmkMwVPCpKamFuTn59doEOXl5cTFNa+pJB9z8+Bjbh5qM+Zly5ZtM7O2VdWL9QOJo4AnzOxuSacDT0s6ieBs5hDB07wtgXckvW5mKwEkpRM8eHWzme2prGMzewR4BKCwsNBmzpxZowCnTZvG4MGDa9S2sfIxNw8+5uahNmOWtLrqWtG9tLWe4Angw/LCskjXE9zzjplNB1IIlnO4Eng1XCF1C/AeUAjBRDxBEnnGzF6MYvzOOeeqIZqJZAbQQ1JXSUnASL74cNQaguUkCJ/yTSF4KncNcG5YnkawYN0SSQL+Aiw2sz/gnHMu5qKWSMysjOAp4MkEk+ITzGyhpF9IOvyioHHAjeG7FJ4DrrXgNrL7gXRJCwkS0uNmNg84AxgNnBu+r2HO4VuGnXPOxUZU50jMbBIwqULZHRHbiwiSQ8V2RcDllZS/iy9A55xzDUrzun3BOedcnfNE4pxzrlY8kTjnnKsVTyRHUF5uPPPhaj7aVBbrUJxzrkHzRHIEcXFiwoy1TFxeii+175xzR+aJ5CguL8xnXZExf/3uWIfinHMNlieSo7ikXwcS4+BvMyq+5M4559xhnkiOIjMlkcL28Uycs4Hi0kOxDsc55xokTyRVOLtjInsPlPHqwo2xDsU55xokTyRVOL5VHJ1atWDCjHWxDsU55xokTyRViJMYUZjH9JXbWb19X6zDcc65BscTSTVcVpBHnOCFWX5W4pxzFXkiqYbcrFTO7tmWF2at41C5P1PinHORPJFU04jCfDbuLuGdT7bGOhTnnGtQPJFU0/m9cmiVlsSEmf5MiXPORfJEUk1JCXF8tV9HXlu0mW1FB2IdjnPONRieSI7BlQM7cfCQ8fxMn3R3zrnDPJEcg+PapXNat1Y8+9Fqyn3S3TnnAE8kx+yqgZ1Zu6OYt33S3TnngCgnEklDJS2VtFzSrZUc7yRpqqTZkuZJuigsT5T0pKT5khZLuq26fUbbhSe2p3VaEs98uKa+v9o55xqkqCUSSfHA/cAwoDcwSlLvCtVuByaYWX9gJPBAWH45kGxmJwMFwE2SulSzz6hKSohjxKn5vLF4Mxt3F9fnVzvnXIMUzTOSAcByM1tpZqXAeODSCnUMyAy3s4ANEeVpkhKAVKAU2FPNPqNu1KmdMGD8R34rsHPOJUSx745A5G/adcDACnV+DkyR9D0gDTg/LH+BIEFsBFoAPzSzHZKq0ycAksYAYwBycnKYNm1ajQZRVFRUaduTWsfz5LvL6RO/nvg41ajvhupIY27KfMzNg485OqKZSKpjFPCEmd0t6XTgaUknEZx5HAI6AC2BdyS9fiwdm9kjwCMAhYWFNnjw4BoFOG3aNCpre7DdZm58aiZlOb0478T2Neq7oTrSmJsyH3Pz4GOOjmhe2loP5Efs54Vlka4HJgCY2XQgBWgDXAm8amYHzWwL8B5QWM0+68WQ49uSm5Xik+7OuWYvmolkBtBDUldJSQST6RMr1FkDnAcgqRdBItkalp8blqcBpwFLqtlnvUiIj2PkqZ14e9lWX17eOdesRS2RmFkZMBaYDCwmuDtroaRfSLokrDYOuFHSXOA54FozM4I7s9IlLSRIHo+b2bwj9RmtMVRl1IB8EuLEU9NXxyoE55yLuajOkZjZJGBShbI7IrYXAWdU0q6I4BbgavUZK+0yU/hyn1wmzFjLDy/oSXpyrKecnHOu/vmT7bV07aAu7D1Qxosf+/pbzrnmyRNJLfXv1JK++dk88f4qX3/LOdcseSKpA98c1IWVW/f5+lvOuWbJE0kduOjkXNpmJPPE+6tiHYpzztU7TyR1ICkhjm8M7My0pVtZubUo1uE451y98kRSR64c2InEePGkn5U455oZTyR1pG1GMl/p04EXZq1jT8nBWIfjnHP1xhNJHfrmGV3ZV3qICTN8VWDnXPPhiaQOnZyXxYAurXj8vVWUHSqPdTjOOVcvPJHUsRvP7sb6XcW8Mn9jrENxzrl64Ymkjp13Qju6tU3jkbdXEiwb5pxzTZsnkjoWFyduPKsbCzfsYfqK7bEOxznnos4TSRR8rX9H2qQn8/DbK2MdinPORZ0nkihISYzn2kGdeWvZVpZu2hvrcJxzLqo8kUTJVQM7k5oYzyN+VuKca+I8kURJy7Qkrjg1n4lz17Npd0msw3HOuajxRBJF15/ZlUPlxuPvfRrrUJxzLmo8kURRfqsWXHRyLs98uIbd+33ZFOdc0xTVRCJpqKSlkpZLurWS450kTZU0W9I8SReF5VdJmhPxKZfULzw2StL8sP6rktpEcwy19d0hx1F0oMyXmHfONVlRSySS4oH7gWFAb2CUpN4Vqt0OTDCz/sBI4AEAM3vGzPqZWT9gNPCpmc2RlADcCwwxsz7APGBstMZQF3rlZnJ+r3Y89t6nFB0oi3U4zjlX56J5RjIAWG5mK82sFBgPXFqhjgGZ4XYWsKGSfkaFbQEUftIkKWxbWZsG5btDjmN38UGe+WB1rENxzrk6p2gt4yFpODDUzG4I90cDA81sbESdXGAK0BJIA843s1kV+lkBXGpmCyL6fQzYB3xCcHZyqJLvHwOMAcjJySkYP358xSrVUlRURHp6eo3aRvrdjGLW7jV+f04qSfGqdX/RVFdjbkx8zM2Dj/nYDBkyZJaZFVZZ0cyi8gGGA49G7I8G7qtQ50fAuHD7dGAREBdxfCAwP2I/EXgD6E5wZnIfcHtVsRQUFFhNTZ06tcZtI01fsc063/KyPfHep3XSXzTV1ZgbEx9z8+BjPjbATKvG7/toXtpaD+RH7OeFZZGuByYAmNl0IAWInDwfCTwXsd8vrLsiHOQEYFDdhh0dA7u2orBzSx5+awWlZb7EvHOu6YhmIpkB9JDUVVISQVKYWKHOGuA8AEm9CBLJ1nA/DhjBf+ZHIEhEvSW1DfcvABZHbQR1SBJjzz2ODbtLeGn2uliH45xzdSZqicTMygjuqJpM8Mt+gpktlPQLSZeE1cYBN0qaS3DmcW14pgFwNrDWzFZG9LkBuBN4W9I8gjOUX0VrDHXtnJ5tObljFg9MW+EvvnLONRkJ0ezczCYBkyqU3RGxvQg44whtpwGnVVL+EPBQnQZaTyTxvXOPY8zTs3hp9nouL8yvupFzzjVw/mR7Pbugdw4nd8ziT29+wkE/K3HONQGeSOqZJH50QU/W7ijm+Zk+V+Kca/w8kcTA4OPb0i8/m/ve/IQDZV94BMY55xoVTyQxIIlxX+rJht0l/G3G2liH45xzteKJJEbOPK4NA7q04r43l1Ny0M9KnHONlyeSGJHEj77Uky17D/DMh2tiHY5zztWYJ5IYOq1bawZ1b82D05azv9RXBnbONU6eSGJs3Jd6sq2olMffWxXrUJxzrkY8kcRYQedWnN8rh4emrWDHvtJYh+Occ8fME0kDcMvQ49lXWsb9U5fHOhTnnDtmnkgagB45GVxekM/T01ezdsf+WIfjnHPHxBNJA3HzBT2Q4A+vLYt1KM45d0w8kTQQuVmpfPOMrvxjznoWbtgd63Ccc67aPJE0IN8e3J3MlER+8+rSWIfinHPV5omkAclKTWTskON4e9lW3lu+LdbhOOdctXgiaWBGn96Zjtmp3PXKYg6VW9UNnHMuxjyRNDApifHcOuwEFm/cw/MzfUFH51zD54mkAbq4Ty6FnVvy+ylL2VtyMNbhOOfcUXkiaYAkccdXerOtqJT7/CFF51wDF9VEImmopKWSlku6tZLjnSRNlTRb0jxJF4XlV0maE/Epl9QvPJYk6RFJyyQtkXRZNMcQK33ysrnslDwef3cVq7fvi3U4zjl3RFFLJJLigfuBYUBvYJSk3hWq3Q5MMLP+wEjgAQAze8bM+plZP2A08KmZzQnb/BTYYmY9w37fitYYYu0nQ48nIV78atLiWIfinHNHFM0zkgHAcjNbaWalwHjg0gp1DMgMt7OADZX0Mypse9h1wP8CmFm5mTXZ+2RzMlP4zuDuTF64mfdXNNlhOucaOZlF5xZTScOBoWZ2Q7g/GhhoZmMj6uQCU4CWQBpwvpnNqtDPCuBSM1sgKRuYDzwPDAZWAGPNbHMl3z8GGAOQk5NTMH78+IpVqqWoqIj09PQata0LpYeM294pJjUB7hyUSnycov6dsR5zLPiYmwcf87EZMmTILDMrrLKimUXlAwwHHo3YHw3cV6HOj4Bx4fbpwCIgLuL4QGB+xH4bgrOY4RHtn64qloKCAqupqVOn1rhtXfn3/A3W+ZaX7dF3VtbL9zWEMdc3H3Pz4GM+NsBMq8bv+2he2loP5Efs54Vlka4HJgCY2XQgJUwWh40EnovY3w7sB14M958HTqm7kBumC09szzk923LPa8vYvKck1uE459znRDORzAB6SOoqKYkgKUysUGcNcB6ApF4EiWRruB8HjCBifiTMkP8iuKxF2HZR9IbQMEjizktOpPRQOf/zik+8O+calqglEjMrA8YCk4HFBHdnLZT0C0mXhNXGATdKmktw5nFtmCwAzgbWmtnKCl3fAvxc0jyCy2XjojWGhqRLmzS+dU53Js7dwPu+DpdzrgFJiGbnZjYJmFSh7I6I7UXAGUdoOw04rZLy1QRJptn5zuDuvDR7HXdMXMik759FUoI/T+qciz3/TdSIpCTGc+clJ7J8SxGPvfdprMNxzjnAE0mjc+4JOVzQO4d7X/+EdTv9tbzOudjzRNII/ewrvZHg9n8s4D9TSs45FxueSBqhvJYt+K8Lj2fa0q1MnFvZYgDOOVd/PJE0Ulef3oV++dnc+a9F7NhXGutwnHPNmCeSRio+Tvzmsj7sKT7IL19u8o/SOOcaME8kjdjx7TPCW4LXM23plliH45xrpjyRNHLfPfc4urdN46cvLWDfgbJYh+Oca4Y8kTRyyQnx/OayPmzYXcxvXl0S63Ccc82QJ5ImoLBLK745qCtPTV/Ne758inOunnkiaSJ+MvR4urVN4ycvzGNPycFYh+Oca0Y8kTQRKYnx3H15XzbuLuYuv4vLOVePqpVIJD1dnTIXW/07teRb53Rnwsx1vLnkCy+NdM65qKjuGcmJkTuS4oGCug/H1dYPzu/BCe0zuOXv89npDyo65+rBUROJpNsk7QX6SNoTfvYCW4B/1kuE7pgkJ8Tz+8v7snNfKbf/09fics5F31ETiZn9r5llAL8zs8zwk2Fmrc3stnqK0R2jkzpm8cMLevLKvI28MGtdrMNxzjVx1b209bKkNABJ35D0B0mdoxiXq6VvndOdgV1b8bOJC1m1bV+sw3HONWHVTSQPAvsl9SV4te0K4KmoReVqLT5O3HNFPxLj4/j++NmUlpXHOiTnXBNV3URSFr5L/VLgPjO7H8ioqpGkoZKWSlou6dZKjneSNFXSbEnzJF0Ull8laU7Ep1xSvwptJ0paUM34m6UO2an8+usnM2/dbu55fVmsw3HONVHVTSR7Jd0GjAZekRQHJB6tQXhn1/3AMKA3MEpS7wrVbgcmmFl/YCTwAICZPWNm/cysX/idn5rZnIi+vw4UVTP2Zm3YybmMPDWfh95awfv+1LtzLgqqm0iuAA4A15nZJiAP+F0VbQYAy81spZmVAuMJzmgiGZAZbmcBlb2laVTYFgBJ6cCPgLuqGXuzd8dXetO1dRo/nDCHbUUHYh2Oc66JUXVvD5WUA5wa7n5kZkddt1zScGComd0Q7o8GBprZ2Ig6ucAUoCWQBpxvZrMq9LMCuNTMFoT79wBvA7OBl83spCN8/xhgDEBOTk7B+PHjK6tWpaKiItLT02vUtiFZvecQv/yghJ4t4/hxYQpx0hHrNpUxHwsfc/PgYz42Q4YMmWVmhVVWNLMqP8AIYDXwJMEk+6fA8CraDAcejdgfTTC/ElnnR8C4cPt0YBEQF3F8IDA/Yr8fMDHc7gIsqE78BQUFVlNTp06tcduGZvxHq63zLS/b3VOWHrVeUxpzdfmYmwcf87EBZlo1fscmVDMx/RQ41cKzEEltgdeBF47SZj2QH7GfF5ZFuh4YGia06ZJSgDYEDzxCMG/yXET904FCSauABKCdpGlmNria42jWRhTm89GnO/nzm59Q2LklZ/dsG+uQnHNNQHXnSOLs85eytlej7Qygh6SukpIIksLECnXWAOcBSOoFpABbw/04gjOhz65JmdmDZtbBzLoAZwLLPIlUnyTu+upJ9GyXwc1/m8PG3cWxDsk51wRUN5G8KmmypGslXQu8Akw6WgMzKwPGApOBxQR3Zy2U9AtJl4TVxgE3SppLcOZxbXg6BXA2sNbMVh7bkNzRpCbF88A3TuHAwUOMfXY2Bw/58yXOudo56qUtSccBOWb2X+Ett2eGh6YDz1TVuZlNokLCMbM7IrYXAWccoe004LSj9L0KqHSi3R1d97bp/GZ4H8Y+O5v/nbSEO75S8a5s55yrvqrmSP4I3AZgZi8CLwJIOjk89pWoRuei5uI+HZi5aiePvfcpJ3XM5Oun5MU6JOdcI1XVpa0cM5tfsTAs6xKViFy9+emXe3F6t9bc+uJ85qzdFetwnHONVFWJJPsox1LrMhBX/xLj47j/qlNol5HMTU/PZMuekliH5JxrhKpKJDMl3VixUNINwKxK6rtGplVaEv93dSF7S8q46a+zKDl4KNYhOecamarmSG4GXpJ0Ff9JHIVAEvC1aAbm6k+v3Ezuvrwv337mY27/xwK+3MZfhuWcq76jJhIz2wwMkjSE/9wh9YqZvRn1yFy9GnZyLt8/rwd/euMTEk9IYkisA3LONRrVerLdzKYCU6Mci4uxm8/rwbJNexm/cBNDFm7iSye2j3VIzrlGoLoPJLpmIC58GVbXrOBlWH4nl3OuOjyRuM9JTYrn5lNSaJuRzA1PzmDtjv2xDsk518B5InFfkJksHr92AAcPGdc+/hG79x+MdUjOuQbME4mr1HHt0nlkdAFrdxQz5umZHCjz24Kdc5XzROKOaGC31vzu8j58+OkOxk2Yy6Fyvy3YOfdF1X0fiWumLu3Xkc17SvjVpCVkpSZy11dPQkd5u6JzrvnxROKqNObs7uzYd5CH3lpByxZJ/PjC42MdknOuAfFE4qrllqHHs7u4lPumLie7RSI3nNUt1iE55xoITySuWoK3K57M7uKD3PXKYjJTExlRmF91Q+dck+eJxFVbfPjA4t6Smdz693lkJCcw7OTcWIflnIsxv2vLHZPkhHge+kYB/fKz+d5zs3l1waZYh+ScizFPJO6YpSUn8OR1AzipYxZjn/2YKQs9mTjXnEU1kUgaKmmppOWSbq3keCdJUyXNljRP0kVh+VWS5kR8yiX1k9RC0iuSlkhaKOnX0YzfHVlGSiJPXT+AEztm8d1nP+b1RZtjHZJzLkailkgkxQP3A8OA3sAoSb0rVLsdmGBm/YGRwAMAZvaMmfUzs37AaOBTM5sTtvm9mZ0A9AfOkDQsWmNwR5eZkshT1w2gV24m33nmY6Yu2RLrkJxzMRDNM5IBwHIzW2lmpcB44NIKdQzIDLezgA2V9DMqbIuZ7Q+XtCfs82MgLwqxu2rKSk3k6esG0rN9Ojc9PYs3FvuZiXPNjcyis+yFpOHAUDO7IdwfDQw0s7ERdXKBKUBLIA0438xmVehnBXCpmS2oUJ5NkEjON7OVlXz/GGAMQE5OTsH48eNrNI6ioiLS09Nr1LaxqsmYi0qNu2eWsGZvOWP6JDMwt3HdEOj/zs2Dj/nYDBkyZJaZFVZZ0cyi8gGGA49G7I8G7qtQ50fAuHD7dGAREBdxfCAwv5K+E4B/AzdXJ5aCggKrqalTp9a4bWNV0zHvKS61yx9837rc+rKN/2h13QYVZf7v3Dz4mI8NMNOq8Ts2mpe21gORT6zlhWWRrgcmAJjZdCAFaBNxfCTwXCV9PwJ8YmZ/rLNoXa1lpCTy5HUDOKtHW275+3wee/fTWIfknKsH0UwkM4AekrpKSiJIChMr1FkDnAcgqRdBItka7scBIwjnRw6TdBfBfMrNUYzd1VBqUjz/d3UBQ09szy9eXsSf3/jk8Fmkc66JiloiMbMyYCwwGVhMcHfWQkm/kHRJWG0ccKOkuQRnHtfaf37rnA2stYj5D0l5wE8J7gL7OLw1+IZojcHVTHJCPPdd2Z+v9+/I3a8t485/LfIl6J1rwqI6I2pmk4BJFcruiNheBJxxhLbTgNMqlK0DfA3zRiAhPo7fX96XVmlJPPrup2zeU8I9V/QjJTE+1qE55+qYP9nuoiYuTtx+cW9u/3IvXl24idF/+ZBd+0tjHZZzro55InFRd8NZ3fjzqP7MXbub4Q9NZ93O/bEOyTlXhzyRuHpxcZ8OPHX9ADbvKeHrD7zP3LW7Yh2Sc66OeCJx9ea0bq35+7cHkZQQx4iHp/OvuZUtZOCca2w8kbh61TMng39+9wz65GXxvedm84cpSyn3O7qca9Q8kbh61zo9mb/eMJDLC/L405vL+e6zH7O/tCzWYTnnasgTiYuJ5IR4fju8Dz+9KLij6/KHprN2h0/CO9cYeSJxMSOJG8/uxmPXnMqaHfu5+M/vMnWpL0XvXGPjicTF3JAT2vHy986kQ3Yq1z0xgz+8tsyfhHeuEfFE4hqEzq3TeOk7g/h6/zz+9MYnXPfEDHbu84cXnWsMPJG4BiMlMZ7fX96HX33tZKav2M7Ff36XOf68iXMNnicS16BI4sqBnXj+W6cDMPzB97l/6nK/1OVcA+aJxDVIffOzmfSDs7jwpPb8bvJSvvHoh2zaXRLrsJxzlfBE4hqsrNRE7hvVn99e1oc5a3cx7N63mbJwU6zDcs5V4InENWiSGHFqPi9/P7ira8zTs7j9H/PZd8AfYHSuofBE4hqF7m3TefE7g7jxrK488+Eaht37Dh+u3B7rsJxzeCJxjUhyQjw//XJvxt8YvO/sikc+4M5/LaS49FCMI3OuefNE4hqdgd1a8+rNZ3HN6Z15/L1VDLv3bWau2hHrsJxrtqKaSCQNlbRU0nJJt1ZyvJOkqZJmS5on6aKw/KrwfeyHP+WS+oXHCiTND/v8kyR/9W4z1CIpgTsvPYlnbxxIWblx+cPT+fnEhewtORjr0JxrdqKWSCTFA/cDw4DewChJvStUux2YYGb9gZHAAwBm9oyZ9TOzfsBo4FMzmxO2eRC4EegRfoZGawyu4RvUvQ2Tbz6bq0/rzJPTV3H+H97i3/M3YubPnThXX6J5RjIAWG5mK82sFBgPXFqhjgGZ4XYWUNmbjkaFbZGUC2Sa2QcW/KZ4CvhqNIJ3jUdacnB28tJ3zqB1WjLffuZjbnhypr/S17l6omj9n5uk4cBQM7sh3B8NDDSzsRF1coEpQEsgDTjfzGZV6GcFcKmZLZBUCPzazM4Pj50F3GJmF1fy/WOAMQA5OTkF48ePr9E4ioqKSE9Pr1Hbxqoxj/lQufHa6jJeXB6s0/XV4xL5UudEEuKOfgW0MY+5pnzMzUNtxjxkyJBZZlZYVb2EGvVed0YBT5jZ3ZJOB54e5p1pAAAUHklEQVSWdJKZlQNIGgjsN7MFx9qxmT0CPAJQWFhogwcPrlGA06ZNo6ZtG6vGPubzgO/tKuZn/1zIhMWbmbE9if++uBfnnpBzxDaNfcw14WNuHupjzNG8tLUeyI/YzwvLIl0PTAAws+lACtAm4vhI4LkKfeZV0adzdMxO5dFrCnn8m6ci4LonZnLt4x+xfEtRrENzrsmJZiKZAfSQ1FVSEkFSmFihzhqC/4FEUi+CRLI13I8DRhDOjwCY2UZgj6TTwru1rgb+GcUxuEZuyPHtePXms7n9y72YtWonQ//4Nne9vIjdxX53l3N1JWqJxMzKgLHAZGAxwd1ZCyX9QtIlYbVxwI2S5hKceVxr/5m0ORtYa2YrK3T9HeBRYDmwAvh3tMbgmoakhDhuOKsbb/54MMML8vjLe59y9m+n8vBbKyg56A8zOldbUZ0jMbNJwKQKZXdEbC8CzjhC22nAaZWUzwROqtNAXbPQNiOZX1/Wh9Gnd+a3ry7lf/+9hCfeX8UPL+hJG79d2Lka8yfbXbNzYocsnrxuAM/eOJB2mSn85IV53P5eMVMWbvLnT5yrAU8krtka1L0N//jOIB686hTKy2HM07P46gPv8+aSzZ5QnDsGsb7917mYksSwk3NJ2rqErenduW/qcq57YiYnd8zi++f14Pxe7fBVeJw7Oj8jcQ6IjxMjB3Ri6o8H89vL+rC7+CA3PjWTi//8LpP9kpdzR+WJxLkIifFxjDg1nzfGncPvL+/LvgNl3PT0LC7849s8P3MtpWXlsQ7RuQbHE4lzlUiMj2N4QR6v/+gc7rmiL3ES//XCPM767Zs8OG2FP4fiXASfI3HuKBLi4/ha/zy+2q8jb3+yjUfeXsFvXl3C/VOXM/LUfL55Zlc6ZqfGOkznYsoTiXPVIIlzerblnJ5tWbB+N4+8vZLH31/FY+99yvm9crj69C6ccVxrn5h3zZInEueO0Ukds/jTqP7cMuwE/vrBav42Yy1TFm2me9s0Rp/WmcsK8shISYx1mM7VG58jca6GOmancsvQE3j/1nO5+/K+pKck8vN/LWLgr97gpy/NZ+GG3bEO0bl64WckztVSSmI8lxXkcVlBHnPX7uKp6at5ftY6nvlwDSd2yGREYT6X9utAdoukWIfqXFT4GYlzdahvfjZ3j+jLR//vPO685EQAfjZxIQN+9Qbfe24273yylfJyfybFNS1+RuJcFGS3SOKaQV24ZlAXFqzfzQuz1vHS7PX8a+4GOmSlcEm/jlzStwO9cjN8gt41ep5InIuykzpmcVLHLG4ddgKvL97M32et49F3VvLQWyvo0S6dS/p24JJ+HejcOi3WoTpXI55InKsnKYnxXNynAxf36cCOfaVMmr+RiXM3cPdry7j7tWX0zc/mkr4duOjk9uRm+bMprvHwROJcDLRKS+Ibp3XmG6d1ZsOuYv41dwMT527gly8v4pcvL6JvXhZfOrE9F57YnuPapcc6XOeOyhOJczHWITuVm87pzk3ndGfF1iImL9zE5IWb+d3kpfxu8lK6t03jwjCp9MnL8jkV1+B4InGuAeneNp3vDD6O7ww+jo27i3lt0WYmL9zEw2+v5IFpK8jJTGZwz3YMPr4tZ/RoQ6Y/+OgaAE8kzjVQuVmpXH16F64+vQs795XyxpItTF2yhUkLNvK3mWtJiBMFnVsy+PggsZzQ3u8Ac7ER1UQiaShwLxAPPGpmv65wvBPwJJAd1rk1fM87kvoADwOZQDlwqpmVSBoF/D/AgA3AN8xsWzTH4VystUxLYnhBHsML8ig7VM7Ha3YxbekWpi3dym9eXcJvXl1C+8wUzurRhkHHtWZQ9zbkZKbEOmzXTEQtkUiKB+4HLgDWATMkTTSzRRHVbgcmmNmDknoDk4AukhKAvwKjzWyupNbAwbD8XqC3mW2T9FtgLPDzaI3DuYYmIT6OAV1bMaBrK34y9AQ27ynhraVbmbZsC1MWbeb5WesA6NY2jUHdg6RyWrfWtErzJ+tddETzjGQAsNzMVgJIGg9cCkQmEiM44wDIIjjDAPgSMM/M5gKY2fawj0RAQJqk7WHb5VEcg3MNXk5mCiNOzWfEqfmUlxuLNu5h+ortvL9iGy99vJ6/frAGgF65mQzs2orCLi0p7NwqxlG7pkTReoWopOHAUDO7IdwfDQw0s7ERdXKBKUBLIA0438xmSboZKADaAW2B8Wb224h+HwP2AZ8AQ8zsUCXfPwYYA5CTk1Mwfvz4Go2jqKiI9PTmdfulj7npKCs3Vu0uZ9GOQyzefogVu8opDV/y2CrZ6NkqgR4t4+mRHUdeRhxxTXyOpan+Ox9NbcY8ZMiQWWZWWFW9WE+2jwKeMLO7JZ0OPC3ppDCuM4FTgf3AG5JmAW8D3wb6AyuBPwO3AXdV7NjMHgEeASgsLLTBgwfXKMBp06ZR07aNlY+56Tp4qJxFG/Ywc/VO/j1jKSuLEvhg4wEAMpIT6Ncpm7552fTJy6JvfnaTm2dpLv/OkepjzNFMJOuB/Ij9vLAs0vXAUAAzmy4pBWhDMKfy9uFJdEmTgFOAPWHdFWH5BODWKI7BuSYlMT6OvvnZ9M3PpnvZas455xzW7SxmxqodzFy9k49X7+TBt1ZwKFxYsl1GMn3ysumbl8XJeVn0ycv2uRb3BdFMJDOAHpK6EiSQkcCVFeqsAc4DnpDUC0gBtgKTgZ9IagGUAucA94T99JbU1sy2EkzkL47iGJxr0iSR36oF+a1a8PVT8gAoLj3Eoo27mbt2N/PX72buul28vnjzZ23yWqbSOzeTE3Iz6Z2bwQntM+nUqgVxcU37spg7sqglEjMrkzSWICnEA4+Z2UJJvwBmmtlEYBzwf5J+SDDxfq0FkzY7Jf2BIBkZMMnMXgGQdCfwtqSDwGrg2miNwbnmKDUpnoLOrSiImJDfU3KQBet3M29dkFwWb9zD64s3c3hF/BZJ8RzfPkgqvXIz6JWbSc+cDLJS/YHJ5iCqcyThMyGTKpTdEbG9CDjjCG3/SnALcMXyh4CH6jZS59zRZKYkMqh7GwZ1b/NZWXHpIT7ZspfFG/eweONelmzaw6T5G3nuozWf1WmXkUz3tukc1y74HN7OyUz2hyebkFhPtjvnGqnUpHj65GXTJy/7szIzY9OeEpZs3MuSTXtZsbWI5VuK+Mfs9ew9UPZZvfTkBLq3TaN723S6t0una5s0OrVqQefWLfx9942QJxLnXJ2RRG5WKrlZqQw5od1n5WbG1r0HWL6l6LPksmLrPt5fsZ0XZ3/+HpxWaUl0atWCLq1b0Kl1Gp3DBNOpdQvapvuZTEPkicQ5F3WSaJeZQrvMFAYd1+Zzx4oOlLF6+z7WbN/P6h37Wb19H6u372fGqp1MnLuByDcTt0iKp2N2Kh1bptIhOzXYzg62O2Sn0D4zhYR4f4N4ffNE4pyLqfTkBE7skMWJHbK+cKy0rJx1O/ezenuYYHbsZ8OuYtbvKmbeut3s2Ff6ufpxgvaZKWFiCT45mcm0y0ghJzOZrfvLKTl4iJTE+PoaXrPgicQ512AlJcTRrW063dpW/mR2cekh1u8qZkP4WR9+NuwqZvbanfx7wUYOHvr86h3/9farZKUmkpOZTE5mCm0zgj9zMpKDs6aMZFqlJdE6PZnMlAS/lFYNnkicc41WalL8Z3eEVaa83NhVfJDNe0rYvKeEd2bMpWWHLmzec4DNe0rYsvcAK7YUsWXvAcrKv7hcVGK8gqSSlkzr9CRahwmmVVoSbdKTaBWWt0lLpmVaIunJzTPxeCJxzjVZcXFBImiVlkSv3EzYmMjgwT2+UK+83Nixv5TNe0rYVlTK9qID7NhXyraiUnbsO8D2olK27ytl1fZ9bC8qZX/pF5b3AyA+TmSlJpKdmkhWi/DP1ESyWySR9dl28An2kz7bTmzEczueSJxzzV5cnGiTnkyb9ORq1S8uPcT2fUGy2V5UyraiA+zcX8ru4oPs2n+Q3cXBZ1tRKSu27mPX/lL2lJQdtc+UxDjSkxPJSEkgIyWB9OTgk5GS+Nl+RkoC6eF2ZkriZ9sZKQlkJCfSIjk+JgnJE4lzzh2j1KR48pJakNeyRbXbHCo39pYEiWZX8UF2hYlnd/FBdu8/SNGBMvaUlFF0oIyikoPsLSljzb797C0pY29JcLySq29fkJQQR1pSPC2SEkhLjmdcn+is8B7JE4lzztWD+DiR3SKJ7BY1W/TSzCg+eChMLEHC2VtykKKSMvYeCMr2HyhjX+kh9h0oY19pGfsOlJEYt7eOR/JFnkicc64RkESLpARaJCWQk1l1/cOmTZsWtZgOa7yzO8455xoETyTOOedqxROJc865WvFE4pxzrlY8kTjnnKsVTyTOOedqxROJc865WvFE4pxzrlZkFv3H52NN0lZgdQ2btwG21WE4jYGPuXnwMTcPtRlzZzNrW1WlZpFIakPSTDMrjHUc9cnH3Dz4mJuH+hizX9pyzjlXK55InHPO1Yonkqo9EusAYsDH3Dz4mJuHqI/Z50icc87Vip+ROOecqxVPJM4552rFE8kRSBoqaamk5ZJujXU8dUnSY5K2SFoQUdZK0muSPgn/bBmWS9Kfwr+HeZJOiV3kNSMpX9JUSYskLZT0g7C8KY85RdJHkuaGY74zLO8q6cNwbH+TlBSWJ4f7y8PjXWIZf21Iipc0W9LL4X6THrOkVZLmS5ojaWZYVq8/255IKiEpHrgfGAb0BkZJ6h3bqOrUE8DQCmW3Am+YWQ/gjXAfgr+DHuFnDPBgPcVYl8qAcWbWGzgN+G7479mUx3wAONfM+gL9gKGSTgN+A9xjZscBO4Hrw/rXAzvD8nvCeo3VD4DFEfvNYcxDzKxfxPMi9fuzbWb+qfABTgcmR+zfBtwW67jqeIxdgAUR+0uB3HA7F1gabj8MjKqsXmP9AP8ELmguYwZaAB8DAwmecE4Iyz/7OQcmA6eH2wlhPcU69hqMNY/gF+e5wMuAmsGYVwFtKpTV68+2n5FUriOwNmJ/XVjWlOWY2cZwexOQE243qb+L8PJFf+BDmviYw0s8c4AtwGvACmCXmZWFVSLH9dmYw+O7gdb1G3Gd+CPwE6A83G9N0x+zAVMkzZI0Jiyr15/thNp24JoeMzNJTe6+cEnpwN+Bm81sj6TPjjXFMZvZIaCfpGzgJeCEGIcUVZIuBraY2SxJg2MdTz0608zWS2oHvCZpSeTB+vjZ9jOSyq0H8iP288KypmyzpFyA8M8tYXmT+LuQlEiQRJ4xsxfD4iY95sPMbBcwleCyTrakw/8DGTmuz8YcHs8CttdzqLV1BnCJpFXAeILLW/fStMeMma0P/9xC8D8MA6jnn21PJJWbAfQI7/ZIAkYCE2McU7RNBK4Jt68hmEc4XH51eLfHacDuiFPmRkHBqcdfgMVm9oeIQ015zG3DMxEkpRLMCS0mSCjDw2oVx3z472I48KaFF9EbCzO7zczyzKwLwX+zb5rZVTThMUtKk5RxeBv4ErCA+v7ZjvVEUUP9ABcBywiuK/801vHU8dieAzYCBwmukV5PcG34DeAT4HWgVVhXBHewrQDmA4Wxjr8G4z2T4DryPGBO+LmoiY+5DzA7HPMC4I6wvBvwEbAceB5IDstTwv3l4fFusR5DLcc/GHi5qY85HNvc8LPw8O+q+v7Z9iVSnHPO1Ypf2nLOOVcrnkicc87ViicS55xzteKJxDnnXK14InHOOVcrnkhckyCpdbj66RxJmyStj9hPqmYfj0s6voo635V0VR3F/K6kfpLiVMcrTEu6TlL7iP0qx+ZcTfntv67JkfRzoMjMfl+hXAQ/8+WVNqxnkt4FxhI857HNzLKPsX28BcugHLFvM5tT+0idOzo/I3FNmqTjFLyH5BmCB7ZyJT0iaaaC93TcEVH38BlCgqRdkn6t4H0e08N1jJB0l6SbI+r/WsF7P5ZKGhSWp0n6e/i9L4Tf1e8oYf4ayAjPnp4K+7gm7HeOpAfCs5bDcf1R0jxggKQ7Jc2QtEDSQ+ETy1cQLB3/t8NnZIfHFvb9DQXvr1gg6Vdh2dHGPDKsO1fS1Dr+J3JNgCcS1xycQPA+it4WrEt0qwXvbegLXKDK3zWTBbxlwfs8pgPXHaFvmdkA4L+Aw0npe8AmC95/8kuC1YaP5lZgrwXvk7ha0knA14BBZtaPYHHVkRFxvW1mfcxsOnCvmZ0KnBweG2pmfyN4ev+KsM/Sz4KV8oC7gCFhXGcoWOzwaGP+GXBeWP61KsbimiFPJK45WGFmMyP2R0n6mOAdHb0IXl5WUbGZ/TvcnkXw/pbKvFhJnTMJFg3EzA4vXXEszgdOBWYqWAb+HKB7eKyUYGG+w86T9BHBEhnnACdW0fdAgjWltpnZQeBZ4Ozw2JHG/B7wlKQb8N8ZrhK+jLxrDvYd3pDUg+ANegPMbJekvxKsuVRRacT2IY7838qBatQ5VgIeM7P//lxhsEJtsR1eNElqAdwHnGLBMuJ3UflYqutIY76RIAFdDHwsqb+Z7azF97gmxv/vwjU3mcBeYI+C5bUvjMJ3vAeMAJB0MpWf8XzGwpcu6T9Lnb8OjJDUJixvLalTJU1TCV7gtC1cAfayiGN7gYxK2nwIDAn7PHzJ7K0qxtPNzD4A/pvgVbWN7iVfLrr8jMQ1Nx8Di4AlwGqCX/p17c8El4IWhd+1iODte0fzF2CepJnhPMmdwOuS4ghWaf4WsCGygZltl/Rk2P9GgiRx2OPAo5KKCd5PcbjNOkn/DUwjOPP5l5m9EpHEKnOPpK5h/SlmtqCKsbhmxm//da6Ohb+UE8ysJLyUNgXoYf953atzTYqfkThX99KBN8KEIuAmTyKuKfMzEuecc7Xik+3OOedqxROJc865WvFE4pxzrlY8kTjnnKsVTyTOOedq5f8DnpmYSiG6HAAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.title('AdaDelta')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overcoming limitations of AdaGrad using RMSProp\n",
    "\n",
    "We learned that in AdaGrad learning rate would become very small when the sum of gradients is high. So, we RMSProp to overcome the limitations of RMSProp. (Refer section RMSProp in the book for full explanation).\n",
    "\n",
    "The update equation of RMSProp is given as:\n",
    "\n",
    "Running average of gradients:\n",
    "$$E\\left[g^{2}\\right]_{t}=\\gamma E\\left[g^{2}\\right]_{t-1}+(1-\\gamma) g_{t}^{2} -- (17) $$\n",
    "\n",
    "Update theta:\n",
    "$$\\theta_{t}^{i}=\\theta_{t-1}^{i}-\\frac{\\eta}{\\sqrt{E\\left[g^{2}\\right]_{t}+\\epsilon}} \\cdot g_{t}^{i} -- (18)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def RMSProp(data, theta, lr = 1e-2, gamma = 0.9, epsilon = 1e-6, num_iterations = 100): \n",
    "    \n",
    "    loss= []\n",
    "    \n",
    "    #initialize running average of gradients\n",
    "    E_grad2 = np.zeros(theta.shape[0])\n",
    "    \n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients with respect to theta\n",
    "        gradients = compute_gradients(data, theta)\n",
    "        \n",
    "        #compute running average of gradients as given in equation (17)\n",
    "        E_grad2 = (gamma * E_grad2) + ((1. - gamma) * (gradients ** 2))\n",
    "        \n",
    "        #update model parameter as given in equation (18)\n",
    "        theta = theta - (lr / (np.sqrt(E_grad2 + epsilon)) * gradients)\n",
    "        \n",
    "        #store the loss\n",
    "        loss.append(loss_function(data,theta))\n",
    "\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update model parameter using RMSProp and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = RMSProp(data, theta) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VfW57/HPd2cghEyMYR4UUBA0SJjEKhSt6OmRDh4rR209tXJOW3rq0EFvrVVPj+2959hZ6+Xa1jq0iHYQKw5HS7QqWkBmcGCeZR7CFJI894+1ItttQkKSlZ3s/bxfr/3KXr/1W2s9DxvysNZv7d+SmeGcc841VizZATjnnGvbvJA455xrEi8kzjnnmsQLiXPOuSbxQuKcc65JvJA455xrEi8kzjnnmsQLiXN1kLRe0hFJ5ZK2S3pIUl647iFJJmlKwjY/DtuvC5ezJd0raXO4n/WSflLHMd6PP4ZzbYUXEudO7h/NLA8oAUYAt8Wtexf4fM2CpEzgSmBNXJ/bgFJgNJAPTADequMY54Z9b08MQgH/9+paJf+L6VwDmNl24HmCglLjaeB8SR3D5cnAUmB7XJ9RwJ/MbKsF1pvZw3UcYwvwLDAMQFKZpP+U9BpwGDhNUk9JsyXtkbRa0g0120u6U9KTkh6XdFDSW5LOaaY/Aufq5IXEuQaQ1Bu4FFgd13wUeAq4Klz+PJBYJN4Abpb0FUnDJekkx+gDXAYsimu+FphGcDazAZgJbAZ6AlcA90j6eFz/KcATQCfgd8CfJWWdQqrOnTIvJM6d3J8lHQQ2ATuA7yWsfxj4vKQi4ELgzwnrfwD8b+BqYAGwRdIXajnGPuBV4GXgnrh1D5nZCjOrBLoD44Fvm9lRM1sMPEjc5TVgoZk9aWbHgR8BOcDYxiTuXEN5IXHu5D5lZjVjG2cCXeJXmtmrQFfgO8BfzOxIwvoqM7vPzMYDRcB/Ar+WNCThGEVm1s/MvpKwj01x73sCe8zsYFzbBqBXbf3NrJoTZy/ORcYLiXMNYGYvAw8B/13L6keBW/joZa3EfRwxs/uAvcDQhh467v1WoJOk/Li2vsCWuOU+NW/Cwfne4XbORcYLiXMN9xPg4loGsH8GXAy8kriBpBslTZDUXlJmeFkrnw+PgzSImW0CXgd+IClH0tnA9QSFrMZISZ8J7yC7EThGME7jXGQykx2Ac22Fme2U9DBwB3Awrn0P8FIdmx0G7gUGEpxdvAt81szWNjKMqcADBGcZe4HvmdmLceufAj4H/JbgxoDPhOMlzkVG/mAr51KDpDuBgWZ2TbJjcenFL20555xrEi8kzjnnmsQvbTnnnGsSPyNxzjnXJGlx11aXLl2sf//+jdr20KFDdOjQoXkDauXSMWdIz7zTMWdIz7wbk/PChQt3mVnX+vqlRSHp378/CxYsaNS2ZWVlTJgwoXkDauXSMWdIz7zTMWdIz7wbk7OkDQ3p55e2nHPONYkXEuecc03ihcQ551yTeCFxzjnXJF5InHPONYkXEuecc03ihcQ551yTeCE5iYfnrefNbZXJDsM551o1LyQnMWvBJl7e7I9ycM65k4m0kEiaLOkdSasl3VrL+r6S5kpaJGmppMvC9ixJv5W0TNIqSbc1dJ/NaXivQjYcqMYntnTOubpFVkgkZQD3AZcSPJ96qqTE51TfDswysxHAVcD9Yfs/Ae3MbDgwEvhXSf0buM9mM6xXIYeOw+a9R6I6hHPOtXlRnpGMBlab2VozqwBmAlMS+hhQEL4vJHh8aE17h/C50+2BCuBAA/fZbIb1LARg2Zb9UR3COefavCgnbewFbIpb3gyMSehzJ/CCpK8BHYCLwvYnCQrENiAXuMnM9khqyD4BkDQNmAZQXFxMWVnZKSdQUWXEZMyZt4zc3e+c8vZtVXl5eaP+vNq6dMw7HXOG9Mw7ypyTPfvvVOAhM7tX0jjgEUnDCM48qoCeQEfgb5JePJUdm9kMYAZAaWmpNXamz95vPMv+jEImTKi1XqWkdJwZFdIz73TMGdIz7yhzjrKQbAH6xC33DtviXQ9MBjCzeZJygC7APwPPmdlxYIek14BSgrOR+vbZrPoXxli2ZT9mhqQoD+Wcc21SlGMk84FBkgZIyiYYTJ+d0GcjMAlA0hAgB9gZtn88bO8AjAXebuA+m1W/ghh7Dx9nyz4fcHfOudpEVkjMrBKYDjwPrCK4O2uFpLslXR52uwW4QdIS4PfAdRbca3sfkCdpBUHx+I2ZLa1rn1HlANC/IPgjWr7lQJSHcc65NivSMRIzmwPMSWi7I+79SmB8LduVE9wC3KB9RqlPfoyMmFi+ZT+Th3VvqcM651yb4d9sr0d2hhjULc9vAXbOuTp4IWmAYb0KWR4OuDvnnPswLyQNMLxXIbsPVbD9wNFkh+Kcc62OF5IGGNYr/Ib7Zr+85ZxzibyQNMDQHgXEBMt9nMQ55z7CC0kDtM/OYKAPuDvnXK28kDTQOb2LWLLZB9ydcy6RF5IGKulbxJ5DFWzcczjZoTjnXKvihaSBSvoUAbB4074kR+Kcc62LF5IGOqM4n/ZZGSza6IXEOefieSFpoMyMGMN7F/oZiXPOJfBCcgpG9Cli5dYDHKusSnYozjnXanghOQUlfYqoqKpm5VafCdg552p4ITkFI/p2BHzA3Tnn4nkhOQXdC3PoXpDjhcQ55+J4ITlFJX2KvJA451wcLySnaETfIjbsPszu8mPJDsU551oFLySnqOaLiUs2+1mJc86BF5JTNrx3IRkxsdi/mOicc4AXklOWm53J4OJ8Fvk4iXPOAREXEkmTJb0jabWkW2tZ31fSXEmLJC2VdFnYfrWkxXGvakkl4bqpkpaF/Z+T1CXKHGozsl8Rizfuo6raZwJ2zrnIComkDOA+4FJgKDBV0tCEbrcDs8xsBHAVcD+AmT1mZiVmVgJcC6wzs8WSMoGfAhPN7GxgKTA9qhzqMqp/Jw4eq+Tt7f7FROeci/KMZDSw2szWmlkFMBOYktDHgILwfSGwtZb9TA23BVD46iBJ4ba1bROp0v6dAFiwfm9LH9o551odRfWgJklXAJPN7Evh8rXAGDObHtenB/AC0BHoAFxkZgsT9rMGmGJmy+P2+2vgEPAewdnJRya/kjQNmAZQXFw8cubMmYldGqS8vJy8vLyPtN9cdpiBRTG+UpLTqP22ZnXlnOrSMe90zBnSM+/G5Dxx4sSFZlZaX7/MRkfVPKYCD5nZvZLGAY9IGmZm1QCSxgCH44pIFvBlYASwFvg5cBvw/cQdm9kMYAZAaWmpTZgwoVEBlpWVUdu2529bxJvrdnPhhRcSnByljrpyTnXpmHc65gzpmXeUOUd5aWsL0CduuXfYFu96YBaAmc0DcoD4wfOrgN/HLZeEfddYcCo1CzivecNumFH9O/L+gWNs3nskGYd3zrlWI8pCMh8YJGmApGyCojA7oc9GYBKApCEEhWRnuBwDruTE+AgEhWiopK7h8sXAqsgyOIlRA4Jxkvnr9yTj8M4512pEVkjMrJLgjqrnCX7ZzzKzFZLulnR52O0W4AZJSwjOPK6zE4M2FwCbzGxt3D63AncBr0haSnCGck9UOZzM4G755OdkMt8H3J1zaS7SMRIzmwPMSWi7I+79SmB8HduWAWNraX8AeKBZA22EWEyU9uvIAj8jcc6lOf9mexOU9u/EezvK2XuoItmhOOdc0nghaYJRNd8n2eCXt5xz6csLSROc3buQ7IyYX95yzqU1LyRNkJOVwfDehfzdC4lzLo15IWmi0QM6sWzzfg4dq0x2KM45lxReSJpo3Gmdqaw2HydxzqUtLyRNVNq/I1kZYt6a3ckOxTnnksILSRPlZmdS0qeIeWt2JTsU55xLCi8kzWDcaZ1ZtmU/B44eT3YozjnX4ryQNIOxp3em2mD+Or97yzmXfryQNINz+3YkOzPG6z5O4pxLQ15ImkFOVgYj+3b0AXfnXFryQtJMzju9Myu3HfB5t5xzaccLSTMZd3pnAN5c52clzrn04oWkmZzdu4j2WRl+ecs5l3a8kDST7MwYowZ08gF351za8ULSjMaf3pn3dpTz/oGjyQ7FOedajBeSZvSxQcGj5F9+d2eSI3HOuZbjhaQZDemRT9f8drzihcQ5l0a8kDQjSXxsUBdeXb2LqmpLdjjOOdciIi0kkiZLekfSakm31rK+r6S5khZJWirpsrD9akmL417VkkrCddmSZkh6V9Lbkj4bZQ6n6sLBXdl3+DjLtuxPdijOOdciIiskkjKA+4BLgaHAVElDE7rdDswysxHAVcD9AGb2mJmVmFkJcC2wzswWh9t8B9hhZoPD/b4cVQ6Ncf7ALkj45S3nXNqI8oxkNLDazNaaWQUwE5iS0MeAgvB9IbC1lv1MDbet8UXgBwBmVm1mrWr+9s557RjWs9ALiXMubcgsmmv5kq4AJpvZl8Lla4ExZjY9rk8P4AWgI9ABuMjMFibsZw0wxcyWSyoClgFPABOANcB0M3u/luNPA6YBFBcXj5w5c2ZilwYpLy8nLy/vlLZ58t0K5qw7zi8+nktulhp13GRqTM6pIB3zTsecIT3zbkzOEydOXGhmpfV2NLNIXsAVwINxy9cCv0joczNwS/h+HLASiMWtHwMsi1vuQnAWc0Xc9o/UF8vIkSOtsebOnXvK27yxZpf1+/Zf7NllWxt93GRqTM6pIB3zTseczdIz78bkDCywBvy+j/LS1hagT9xy77At3vXALAAzmwfkhMWixlXA7+OWdwOHgT+Gy08A5zZfyM3j3H4dyWuXycvvtqqrbs45F4koC8l8YJCkAZKyCYrC7IQ+G4FJAJKGEBSSneFyDLiSuPGRsEI+TXBZi3DbldGl0DhZGTHGnd6ZV97dWXMm5ZxzKSuyQmJmlcB04HlgFcHdWSsk3S3p8rDbLcANkpYQnHlcZyd+814AbDKztQm7/jZwp6SlBJfLbokqh6aYcEZXtuw7wuod5ckOxTnnIpUZ5c7NbA4wJ6Htjrj3K4HxdWxbBoytpX0DQZFp1T5+ZjcAXly1g0HF+UmOxjnnouPfbI9Ij8L2nNWzgJdWfeSGMuecSyleSCI0aUgxb23cyx5/aqJzLoV5IYnQRUO6UW0w9+0dyQ7FOeci44UkQsN6FtItvx0vve2Xt5xzqcsLSYRiMTFpSDdeeXcXFZXVyQ7HOeci4YUkYhcNKab8WCVvrvNH8DrnUpMXkoiNH9iFnKwYL63ycRLnXGryQhKxnKwMzh/YhRdXve/fcnfOpSQvJC3goiHFbN57hFXbDiY7FOeca3ZeSFrARUOLiQmeW74t2aE451yz80LSArrktWPMgM7MWb492aE451yz80LSQi4d3p3VO8p5732/vOWcSy1eSFrIJWd1R4Jn/azEOZdivJC0kOKCHEr7dWTOMh8ncc6lFi8kLWjysB68vf0g63YdSnYozjnXbLyQtKDJw7oD8KzfveWcSyFeSFpQr6L2lPQp4tllPk7inEsdXkha2GXDu7Nsy3427Tmc7FCcc65ZeCFpYZcN7wHA7CVbkxyJc841Dy8kLax3x1xG9e/Inxdt8bm3nHMpIdJCImmypHckrZZ0ay3r+0qaK2mRpKWSLgvbr5a0OO5VLakkYdvZkpZHGX9UppT04r0d5azcdiDZoTjnXJNFVkgkZQD3AZcCQ4GpkoYmdLsdmGVmI4CrgPsBzOwxMysxsxLgWmCdmS2O2/dngPKoYo/aZcN7kBkTsxf75S3nXNsX5RnJaGC1ma01swpgJjAloY8BBeH7QqC236xTw20BkJQH3Ax8v9kjbiGdOmRz4eCuzF6ylepqv7zlnGvb1JDr9JIeMbNr62tLWH8FMNnMvhQuXwuMMbPpcX16AC8AHYEOwEVmtjBhP2uAKWa2PFz+MfAKsAj4i5kNq+P404BpAMXFxSNnzpxZW7d6lZeXk5eX16htT+aNbZU8sOQY3x6Vw5DOGc2+/6aIKufWLh3zTsecIT3zbkzOEydOXGhmpfX1y2zg/s6KXwgvW408pYhqNxV4yMzulTQOeETSMDOrDo8zBjgcV0RKgNPN7CZJ/U+2YzObAcwAKC0ttQkTJjQqwLKyMhq77cmMrqjk4VUvsoGufHnC2c2+/6aIKufWLh3zTsecIT3zjjLnk17aknSbpIPA2ZIOhK+DwA7gqXr2vQXoE7fcO2yLdz0wC8DM5gE5QJe49VcBv49bHgeUSloPvAoMllRWTxytUm52Jpec1Z05y7ZxrLIq2eE451yjnbSQmNkPzCwf+C8zKwhf+WbW2cxuq2ff84FBkgZIyiYoCrMT+mwEJgFIGkJQSHaGyzHgSuLGR8zsl2bW08z6A+cD75rZhAbm2upMKenJgaOV/NWf5+6ca8MaOtj+F0kdACRdI+lHkvqdbAMzqwSmA88Dqwjuzloh6W5Jl4fdbgFukLSE4MzjOjsxaHMBsMnM1p5iTm3G+QO7UFzQjlkLNiU7FOeca7SGjpH8EjhH0jkEv/wfBB4GLjzZRmY2B5iT0HZH3PuVwPg6ti0Dxp5k3+uBWgfa24rMjBifPbc3D7y8hu37j9K9MCfZITnn3Clr6BlJZXimMAX4hZndB+RHF1b6uLK0D9UGf3hrc7JDcc65RmloITko6TaCLwc+E45fZEUXVvro36UDowd04okFm3zKFOdcm9TQQvI54BjwRTPbTnAH1n9FFlWa+VxpH9bvPszf1+1JdijOOXfKGlRIwuLxGFAo6ZPAUTN7ONLI0silw7uT1y6TWQv88pZzru1pUCGRdCXwd+CfCG7JfTP85rprBrnZmfzjOT2Ys2wbB48eT3Y4zjl3Shp6aes7wCgz+4KZfZ5gHq3vRhdW+rmytA9HjlfxlE/k6JxrYxpaSGJmFv+tud2nsK1rgJI+RQztUcCjb2zwQXfnXJvS0GLwnKTnJV0n6TrgGRK+H+KaRhKfH9ePt7cfZP76vckOxznnGqy+ubYGShpvZt8E/i9wdviaRzghoms+l5f0JD8nk0fe2JDsUJxzrsHqOyP5CXAAwMz+aGY3m9nNwJ/Cda4Z5WZn8k8j+/Dc8m3sOHg02eE451yD1FdIis1sWWJj2NY/kojS3DVj+3K8ypj5d59/yznXNtRXSIpOsq59cwbiAqd1zeNjg7rwuzc3UllVnexwnHOuXvUVkgWSbkhslPQlYGEt/V0zuHZsP7YfOMr/rHw/2aE451y96pv990bgT5Ku5kThKAWygU9HGVg6mzSkmD6d2vPgq+u4dHiPZIfjnHMnVd+Drd43s/OAu4D14esuMxsXTpviIpARE18cP4CFG/aycIPfCuyca90aOtfWXDP7efj6a9RBueCb7gU5mTz4t5R9rpdzLkX4t9NbqQ7tMrlmbD+eX7GdDbsPJTsc55yrkxeSVuwL5/UnIyZ+/eq6ZIfinHN18kLSihUX5DClpBezFmxm3+GKZIfjnHO18kLSyt3wsdM4cryKR+b5tCnOudYp0kIiabKkdyStlnRrLev7SporaZGkpZIuC9uvlrQ47lUtqURSrqRnJL0taYWkH0YZf2twRvd8Jp3ZjV+9to7yY5XJDsc55z4iskIiKQO4D7gUGApMlTQ0odvtwCwzGwFcBdwPYGaPmVmJmZUQPCd+nZktDrf5bzM7ExgBjJd0aVQ5tBZfmzSIfYeP86hP5uica4WiPCMZDaw2s7VmVgHMBKYk9DGgIHxfCNT2VKep4baY2WEzmxu+rwDeInh+fEor6VPExwZ14cG/reVIRVWyw3HOuQ9RVA9RCh/FO9nMvhQuXwuMMbPpcX16AC8AHYEOwEVmtjBhP2uAKWa2PKG9iKCQXGRmH/myhaRpwDSA4uLikTNnzmxUHuXl5eTl5TVq2+b07t4q7nnzKFPPzOaS/lmRHqu15NzS0jHvdMwZ0jPvxuQ8ceLEhWZWWl+/+qZIidpU4CEzu1fSOOARScPMrBpA0hjgcC1FJBP4PfCz2ooIgJnNIHxmSmlpqU2YMKFRAZaVldHYbZvTBOCvO+fx0pZDfO/qj5GTlRHZsVpLzi0tHfNOx5whPfOOMucoL21tAfrELfcO2+JdD8wCMLN5QA7QJW79VQQFI9EM4D0zS6tnovz7xwex4+AxHp/vU8w751qPKAvJfGCQpAGSsgmKwuyEPhuBSQCShhAUkp3hcgy4knB8pIak7xOMp9wYYeyt0rjTOzO6fyd+MXc1hyv8Di7nXOsQWSExs0pgOvA8sIrg7qwVku6WdHnY7RbgBklLCM48rrMTgzYXAJviL11J6g18h+AusLfCW4O/FFUOrY0kvjn5DHYePMZDr69PdjjOOQdEPEZiZnOAOQltd8S9XwmMr2PbMmBsQttmQM0eaBsyqn8nJp3ZjQfK1nD16H4U5kY78O6cc/Xxb7a3Qd+45AwOHqvkly+vSXYozjnnhaQtGtKjgCnn9OSh19fx/oGjyQ7HOZfmvJC0UTddPJjKKuMnL76X7FCcc2nOC0kb1a9zB64Z24/H529k1bYDyQ7HOZfGvJC0YTdeNIiC9lnc/fRKopqhwDnn6uOFpA0rys3m5osHM2/tbp5f8X6yw3HOpSkvJG3cP4/uy+DiPO6Zs4pjlT6ho3Ou5XkhaeMyM2J895ND2bjnML9+dX2yw3HOpSEvJCngY4O6cvHQYn720nts3ns42eE459KMF5IU8b1/DJ4ZdufsFT7w7pxrUV5IUkTvjrncdPEgXly1wwfenXMtygtJCvmX8QM4s3s+d85e4c93d861GC8kKSQrI8Y9nxnO+wePcu8L7yQ7HOdcmvBCkmLO7duRa8b046HX1zN//Z5kh+OcSwNeSFLQrZeeSa+i9nzziSX+ACznXOS8kKSgDu0y+a8rzmH97sP8n+f8EpdzLlpeSFLUuNM7c915/Xno9fXMW7M72eE451KYF5IU9q3JZ9C/cy7feGIJ+48cT3Y4zrkU5YUkheVmZ/Ljz5Xw/oGj/K8/LvMvKjrnIuGFJMWN6NuRmz8xmGeWbWPm/E3JDsc5l4IiLSSSJkt6R9JqSbfWsr6vpLmSFklaKumysP1qSYvjXtWSSsJ1IyUtC/f5M0mKModU8G8XnM75A7tw19MreO/9g8kOxzmXYiIrJJIygPuAS4GhwFRJQxO63Q7MMrMRwFXA/QBm9piZlZhZCXAtsM7MFofb/BK4ARgUviZHlUOqiMXEj648hw7ZmUz/3SK/Jdg516yiPCMZDaw2s7VmVgHMBKYk9DGgIHxfCGytZT9Tw22R1AMoMLM3LLjg/zDwqSiCTzXdCnL48edKeHfHQW79g4+XOOeaT2aE++4FxF+U3wyMSehzJ/CCpK8BHYCLatnP5zhRgHqF+4nfZ6/aDi5pGjANoLi4mLKyslOLPlReXt7obVujzwzM4g9LttLh2C4u6Z9Va59Uy7mh0jHvdMwZ0jPvKHOOspA0xFTgITO7V9I44BFJw8ysGkDSGOCwmS0/1R2b2QxgBkBpaalNmDChUQGWlZXR2G1bowsuMA4+upBZb+9gygXnMva0zh/pk2o5N1Q65p2OOUN65h1lzlFe2toC9Ilb7h22xbsemAVgZvOAHKBL3PqrgN8n7LN3Pft0JxGLiXuvPId+nXP56mNvsWmPPwjLOdc0URaS+cAgSQMkZRMUhdkJfTYCkwAkDSEoJDvD5RhwJeH4CICZbQMOSBob3q31eeCpCHNISfk5Wcy4tpTjVdV88aH5/mVF51yTRFZIzKwSmA48D6wiuDtrhaS7JV0edrsFuEHSEoIzj+vsxCjwBcAmM1ubsOuvAA8Cq4E1wLNR5ZDKBnbL44FrRrJu1yG++thbHK+qTnZIzrk2KtIxEjObA8xJaLsj7v1KYHwd25YBY2tpXwAMa9ZA09R5A7twz2eG860nl3LHU8u559PD8a/lOOdOVbIH212SXVnahw27D3Hf3DV0zc/h5osHJzsk51wb44XE8Y1PnMHOg8f42Uvv0TE3iwHJDsg516b4XFsOSdzz6eFcclYxdz29kte2+OC7c67hvJA4ADIzYvz0qhGMH9iZXy2v4C9La5tkwDnnPsoLiftATlYGM64tZWBRjK/PXMzTS7yYOOfq54XEfUiHdpncPDKHkX07cuPjXkycc/XzQuI+IidT/OZfRjGyX0e+PnMRTyzw55g45+rmhcTVqkO7TH5z3SjGD+zCN59cyoxX1iQ7JOdcK+WFxNWpQ7tMHvxCKf8wvAf3zHmbHz77tk8/75z7CP8eiTupdpkZ/GzqCIpys3jg5TVs33+EH372bHKyMpIdmnOulfBC4uqVERPf/9Qweha157+ef4dNe48w49qRdM5rl+zQnHOtgF/acg0iia9OHMh9/3wuy7fs51P3v8Y72/357845LyTuFP3D2T14/F/HcfR4NZ+67zWeWuyPg3Eu3XkhcaespE8Rz3ztfIb1KuDrMxdz19MrfBp659KYFxLXKN0KcvjdDWP54vgB/Oa19VzxwDw27D6U7LCcc0nghcQ1WlZGjDv+cSj3X30u63aW8w8/e5U/Ldqc7LCccy3MC4lrssuG9+DZGy9gSI98bnp8CdN/9xZ7DlUkOyznXAvxQuKaRa+i9vz+hrHccvFgnl+xnU/8+GWeW74t2WE551qAFxLXbDIzYnxt0iBmTz+f7oU5/Nujb/GVxxayff/RZIfmnIuQFxLX7Ib0KOBPXxnPNy85g5dW7WDSvWX8+tV1VPqdXc6lpEgLiaTJkt6RtFrSrbWs7ytprqRFkpZKuixu3dmS5klaIWmZpJywfWq4vFTSc5K6RJmDa5ysjBhfnTiQ/7npQkYN6MTdf1nJJ3/+Kq++tyvZoTnnmllkhURSBnAfcCkwFJgqaWhCt9uBWWY2ArgKuD/cNhN4FPg3MzsLmAAcD9t/Ckw0s7OBpcD0qHJwTde3cy6/uW4UD1xzLocqKrnmV29y/UPzWb2jPNmhOeeaSZRnJKOB1Wa21swqgJnAlIQ+BhSE7wuBmqcofQJYamZLAMxst5lVAQpfHSQp3NafvNTKSWLysB78z00XcuulZ/Lmuj184scv860nl7B57+Fkh+ecayJFNS24pCuAyWb2pXD5WmCMmU2P69MDeAHoCHQALjKzhZJuBEYC3YCuwEwz+z9x+/01cAh4j+DcG1pOAAAQtUlEQVTspKqW408DpgEUFxePnDlzZqPyKC8vJy8vr1HbtlVR53zgmPGXtRX8dVMlGFzYJ5PLBmTRuX1yh+z8s04f6Zh3Y3KeOHHiQjMrra9fsmf/nQo8ZGb3ShoHPCJpWBjX+cAo4DDwkqSFwCvAl4ERwFrg58BtwPcTd2xmM4AZAKWlpTZhwoRGBVhWVkZjt22rWiLny4Gt+47w87++xxMLNvPKlio+M6I3X55wOv27dIj02HXxzzp9pGPeUeYc5X8BtwB94pZ7h23xrgdmAZjZPCAH6AJsBl4xs11mdhiYA5wLlIR911hwKjULOC/CHFyEeha15wefOZuyb05g6ui+/GnxFibeW8a/PrKAv6/b4w/Rcq6NiLKQzAcGSRogKZtgMH12Qp+NwCQASUMICslO4HlguKTccID9QmAlQSEaKqlruP3FwKoIc3AtoHfHXO6eMoxXvzWRr04YyJvr9nDl/53H5b94jVnzN3Gk4iNXLp1zrUhkl7bMrFLSdIKikAH82sxWSLobWGBms4FbgP8n6SaCgffrwjONvZJ+RFCMDJhjZs8ASLoLeEXScWADcF1UObiW1a0gh29ccgZfnTiQP7y1md++vp5v/WEp339mJZ8d2ZsrS/swpEdB/TtyzrWoSMdIzGwOwWWp+LY74t6vBMbXse2jBLcAJ7Y/ADzQvJG61qR9dgbXjO3H1WP68vd1e3j0zY08+sYGfvPaeob1KuCKc3vzyXN60sWf0Ohcq5DswXbn6iSJMad1Zsxpndl76CyeWryFJxZu5s6nV/Ifz6zivNM7c/k5PfnE0O4U5mYlO1zn0pYXEtcmdOyQzXXjB3Dd+AG8s/0gs5ds4anFW/nmk0u5LbaMcad35pKzunPRkGK6F+YkO1zn0ooXEtfmnNE9n292P5NvfOIMlmzez3PLt/Pc8m3c/ufl3P7n5ZzVs4CPn9mNCwd3paRPEZkZPqWcc1HyQuLaLEmU9CmipE8R3558Bu/tKOelVTuY+/YO7pu7mp//dTX57TIZd3pnxg/swtjTOjOoWx6xmJIdunMpxQuJSwmSGFycz+DifL484XT2Hz7Oa2t28bf3dvLKu7t4YeX7AHTqkM2o/h0p7deJ0v4dOatnIdmZfsbiXFN4IXEpqTA3i8uG9+Cy4T0A2LTnMPPW7uaNtbtZsH4vz68ICkt2RoyhPQso6VNEdnklvXccZECXPDL8rMW5BvNC4tJCn0659OmUy5WlwWQLOw4cZcGGvSzetI/FG/fx+PxNHDlexYylr9A+K4MhPfI5s0cBQ3oUcGb3fAZ3y/c7w5yrgxcSl5a6FeR86IylsqqamXPKyOk5mOVb9rNy2wH+smQrv3tz44lt8tsxqDiP07rkcXrXDgzomseAzh3oWZTjA/ourXkhcY7gMcG982NMGNmbK0b2BsDM2Lr/KO9uP8i77x/k3ffLWb2znD8v3sLBo5UfbJuVIfp0zKV3p1z6dmpPn4659OrYnl5FwatLXjsf4HcpzQuJc3WQ9EExmHhmtw/azYyd5cdYt/MQG3YfZt3uQ2zYfYhNe46wZNM+9h85/qH9ZGWI4oIcehTmUFxQ82pH1/x2dM3LoWt+O7rkZVOUm+1jM65N8kLi3CmSRLf8HLrl5zDmtM4fWX/g6HG27D3C1n1H2LLvCNv2H2X7/qNs23+ElVsP8Ne3d3C4lokoYwruKot/FeVm0zE3i4652RS2z6Io/FnYPouC9pkU5GSRm51B8Jw355LDC4lzzawgJ4uCHll1TjBpZpQfq2TnwWPBq/wYuw4eY/ehCnaVH2PPoQr2HKrg7e0H2Xf4OPsOV1B9khn1M2Iir10m+TmZH/zs0C545WVnktsug7x2mbTPzqBDdvAzNzuD9lkZvLu7isKNe2mfnUFOZgY5WRnkZMXIycogOyPml+Rcg3ghca6FSSI/J4v8nCxO61r/E+uqq42DxyrZf/g4+48cZ9+RCg4cqeTA0WD54NHjlB+t5ODRSg4creTQsUr2HKpg4+7DHKqo5PCxKg5VVNZdjOa/XuexszNjtMuM0S4zI/wZI7vmlRH8zMo4sZyVIbIyYmRmxMiOe58ZE5k1yzGRGfbNiClYjsXIiJ1Yzkh8ScTCdbFwOSMmYqrpE/y5Zihoi8UIfkrEBLHYifeSOFJpHDpWSUxCIngR9v2gzYtoQ3khca6Vi8X0weWsxjIzjlVWc+hYJYcrqjh6vIrDFVW8/veFnDlsOEcrguVjldUcPV7F0coqjh2v/uDnscpqKiqrOVZZRUVlNRVVwXJFuM9jldUcr6qmstqoqKzmeJVxvCpsqzIqq6tPelaVFC8+X2+XoMB8uLiIE4VHNYUHIOyrmr58uD+caCdhHzV9448b3yc+lg/HpxNtcSsVt/6Zfz+fdpkZDfszaSQvJM6lAUnhZasM4kd19q7JYMIZ3ercrjlVVxvHq8PCUhW8D9qMyqpqqqqNqmqjMvxZ877agvfV1UaVhW3h+mAdVFtcPwuXqw2DD/pZ2F5VbaxevYbTTj+NqmowgnVmwbY1/cyC7T9YDt+faA+3gw/2D3xouw/2HS4H72r28+H1xG1D2PVE+wetH7CE9R+0J7zRR8pP8/NC4pxrEbGYaBfLoF0r+K1TVrWRCRecnuwwUoZ/i8o551yTeCFxzjnXJF5InHPONYkXEuecc00SaSGRNFnSO5JWS7q1lvV9Jc2VtEjSUkmXxa07W9I8SSskLZOUE7ZnS5oh6V1Jb0v6bJQ5OOecO7nI7p+QlAHcB1wMbAbmS5ptZivjut0OzDKzX0oaCswB+kvKBB4FrjWzJZI6AzUTGH0H2GFmgyXFgE5R5eCcc65+Ud6INxpYbWZrASTNBKYA8YXEgJp5JAqBreH7TwBLzWwJgJntjtvmi8CZYXs1sCuqBJxzztUvyktbvYBNccubw7Z4dwLXSNpMcDbytbB9MGCSnpf0lqRvAUgqCtf/R9j+hKTiyDJwzjlXr2R/NWgq8JCZ3StpHPCIpGFhXOcDo4DDwEuSFgJLgN7A62Z2s6Sbgf8Grk3csaRpwLRwsVzSO42MsQvpd9aTjjlDeuadjjlDeubdmJz7NaRTlIVkC9Anbrl32BbvemAygJnNCwfUuxCcvbxiZrsAJM0BzgX+SlBY/hhu/0S4j48wsxnAjKYmIWmBmZU2dT9tSTrmDOmZdzrmDOmZd5Q5R3lpaz4wSNIASdnAVcDshD4bgUkAkoYAOcBO4HlguKTccOD9QmClBRPKPA1MCLefxIfHXJxzzrWwyM5IzKxS0nSCopAB/NrMVki6G1hgZrOBW4D/J+kmgoH368JisVfSjwiKkQFzzOyZcNffJrgE9hOCovMvUeXgnHOufoqfNdJ9lKRp4WWytJGOOUN65p2OOUN65h1lzl5InHPONYlPkeKcc65JvJA455xrEi8kdahvnrBUIalPON/ZynBes6+H7Z0k/Y+k98KfHZMda3OTlBHO8/aXcHmApDfDz/zx8G7DlCKpSNKT4Tx1qySNS/XPWtJN4d/t5ZJ+LyknFT9rSb+WtEPS8ri2Wj9bBX4W5r9U0rlNObYXklrEzRN2KTAUmBrOBZaKKoFbzGwoMBb4apjrrcBLZjYIeClcTjVfB1bFLf9v4MdmNhDYSx3fUWrjfgo8Z2ZnAucQ5J+yn7WkXsC/A6VmNozgDtKrSM3P+iHC7+XFqeuzvRQYFL6mAb9syoG9kNTug3nCzKwCqJknLOWY2TYzeyt8f5DgF0svgnx/G3b7LfCp5EQYDUm9gX8AHgyXBXwceDLskoo5FwIXAL8CMLMKM9tHin/WBF9zaB9+Jy0X2EYKftZm9gqwJ6G5rs92CvCwBd4AiiT1aOyxvZDUriHzhKUcSf2BEcCbQLGZbQtXbQdSbU6znwDfAqrD5c7APjOrDJdT8TMfQPDdq9+El/QelNSBFP6szWwLwTRKGwkKyH5gIan/Wdeo67Nt1t9xXkgcAJLygD8AN5rZgfh14ZdEU+Y+cUmfJHgUwcJkx9LCMgmmGvqlmY0ADpFwGSsFP+uOBP/7HgD0BDrw0cs/aSHKz9YLSe0aMk9YypCURVBEHjOzmnnM3q851Q1/7khWfBEYD1wuaT3BZcuPE4wdFIWXPyA1P/PNwGYzezNcfpKgsKTyZ30RsM7MdprZcYJ5+saT+p91jbo+22b9HeeFpHYNmScsJYRjA78CVpnZj+JWzQa+EL7/AvBUS8cWFTO7zcx6m1l/gs/2r2Z2NTAXuCLsllI5A5jZdmCTpDPCppq56lL2sya4pDU2nLdPnMg5pT/rOHV9trOBz4d3b40F9sddAjtl/s32Oih47O9PODFP2H8mOaRISDof+BuwjBPjBf+LYJxkFtAX2ABcaWaJA3ltnqQJwDfM7JOSTiM4Q+kELAKuMbNjyYyvuUkqIbjBIBtYSzBXXYwU/qwl3QV8juAOxUXAlwjGA1Lqs5b0e4IJbbsA7wPfA/5MLZ9tWFR/QXCZ7zDwL2a2oNHH9kLinHOuKfzSlnPOuSbxQuKcc65JvJA455xrEi8kzjnnmsQLiXPOuSbxQuJSgqTOkhaHr+2StsQtN2hmV0m/ifuORV19virp6maK+VVJJZJizT3DtKQvSuoet1xvbs41lt/+61KOpDuBcjP774R2Efydr651wxYm6VVgOrAc2GVmRae4fYaZVZ1s32a2uOmROndyfkbiUpqkgQqetfIYsALoIWmGpAXhMyruiOtbc4aQKWmfpB9KWiJpnqRuYZ/vS7oxrv8PJf1dwbNrzgvbO0j6Q3jcJ8NjlZwkzB8C+eHZ08PhPr4Q7nexpPvDs5aauH4iaSkwWtJdkuYreNbGA+E3lT8HlACP15yR1eQW7vsaScvCbe4J206W81Vh3yWS5jbzR+RSgBcSlw7OJHj2xNBwNthbzayU4HkcF6v2Z80UAi+b2TnAPOCLdexbZjYa+CZQU5S+BmwPn/HyHwQzKp/MrcBBMysxs89LGgZ8GjjPzEoIJlu8Ki6uV8zsbDObB/zUzEYBw8N1k83scWAx8LlwnxUfBBtMn/99YGIY13gFk1ieLOfvAZPC9k/Xk4tLQ15IXDpYkzD9w1RJbwFvAUMIHl6W6IiZPRu+Xwj0r2Pff6ylz/kE029gZksIzoROxUXAKGCBpMXAhcDp4boK4E9xfSdJ+juwJOx3Vj37HkMwt9iucBLD3xE8owTqzvk14GFJX8J/Z7haZNbfxbk271DNG0mDCJ6MONrM9kl6FMipZZuKuPdV1P1v5VgD+pwqEczv9t0PNQaz1R4JpwNHUi7BfEnnmtkWSd+n9lwaqq6cbyAoQJ8E3pI0wsz2NuE4LsX4/y5cuikADgIHFEyrfUkEx3gNuBJA0nBqP+P5QM0DlnRiWvMXgSsldQnbO0vqW8um7Qkm2twlKR/4bNy6g0B+Ldu8CUwM91lzyezlevI5LXyK3ncJHkubqg+Bco3kZyQu3bxFMI342wSzob4WwTF+TnApaGV4rJUET+Y7mV8BSyUtCMdJ7gJelBQDjgP/BmyN38DMdkv6bbj/bQRFosZvgAclHSF4dHTNNpslfRcoIzjzedrMnokrYrX5saQBYf8XzGx5Pbm4NOO3/zrXzMJfyplmdjS8lPYCMCju0a7OpRQ/I3Gu+eUBL4UFRcC/ehFxqczPSJxzzjWJD7Y755xrEi8kzjnnmsQLiXPOuSbxQuKcc65JvJA455xrkv8P7BFDNOlEkHgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.title('RMSProp')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adam - Adaptive Moment Estimation\n",
    "\n",
    "\n",
    "Check the section Adam in the book for the detailed explanation. The update equation of Adam is given as:\n",
    "\n",
    "First moment estimate:\n",
    "$$m_{t}=\\beta_{1} m_{t-1}+\\left(1-\\beta_{1}\\right) g_{t} -- (19) $$\n",
    "\n",
    "Second moment estimate:\n",
    "$$v_{t}=\\beta_{2} v_{t-1}+\\left(1-\\beta_{2}\\right) g_{t}^{2} --(20)$$\n",
    "\n",
    "Bias-corrected mt:\n",
    "$$\\hat{m}_{t}=\\frac{m_{t}}{1-\\beta_{1}^{t}} --(21) $$\n",
    "\n",
    "Bias-corrected vt:\n",
    "$$\\hat{v}_{t}=\\frac{v_{t}}{1-\\beta_{2}^{t}} --(22) $$\n",
    "\n",
    "Update theta:\n",
    "\n",
    "$$\\theta_{t}=\\theta_{t-1}-\\frac{\\eta}{\\sqrt{\\hat{v}_{t}}+\\epsilon} \\hat{m}_{t} -- (23) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def Adam(data, theta, lr = 1e-2, beta1 = 0.9, beta2 = 0.9, epsilon = 1e-6, num_iterations = 1000):\n",
    "   \n",
    "    loss = []\n",
    "    \n",
    "    #initialize first moment mt\n",
    "    mt = np.zeros(theta.shape[0])\n",
    "    \n",
    "    #initialize second moment vt\n",
    "    vt = np.zeros(theta.shape[0])\n",
    "    \n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients with respect to theta\n",
    "        gradients = compute_gradients(data, theta)\n",
    "        \n",
    "        #update first moment mt as given in equation (19)\n",
    "        mt = beta1 * mt + (1. - beta1) * gradients\n",
    "        \n",
    "        #update second moment vt as given in equation (20)\n",
    "        vt = beta2 * vt + (1. - beta2) * gradients ** 2\n",
    "        \n",
    "        #compute bias-corected estimate of mt (21)\n",
    "        mt_hat = mt / (1. - beta1 ** (t+1))\n",
    "        \n",
    "        #compute bias-corrected estimate of vt (22)\n",
    "        vt_hat = vt / (1. - beta2 ** (t+1))\n",
    "        \n",
    "        #update the model parameter as given in (23)\n",
    "        theta = theta - (lr / (np.sqrt(vt_hat) + epsilon)) * mt_hat\n",
    "      \n",
    "        loss.append(loss_function(data,theta))\n",
    "\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update model parameter using Adam and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = RMSProp(data, theta) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4HNWV9/Hvr1uSZVmbV3lf8AI2NshYYBuz2DEEwyQ4C0PwAAlDgmcycSYBspB3CANMXibvzJAdwnhIIAGCMWQzwWAGYkEgBmzjfQG873hf5E2WdN4/qoQbIVmypFJL3efzPP2o69at6nNo6EPVrbolM8M555xrrFiyA3DOOde2eSFxzjnXJF5InHPONYkXEuecc03ihcQ551yTeCFxzjnXJF5InGshkm6S9Fqy43CuuXkhca4ZSCqVtE9Su2TH4lxL80LiXBNJ6g9cDBhwdVKDcS4JvJA413SfB94AHgW+UN0oqbOkWZIOSnoLGJi4kaQfS9ocrl8o6eKEdXdLelrS45IOSVomaYik70jaGW738RbKz7lT8kLiXNN9HngifF0hqShsfwA4BvQAbg5fieYDxUAn4DfA05KyE9Z/EngM6AgsAuYQ/DfbC7gX+O8oknHudMnn2nKu8SRdBMwFepjZbkmrCX7gf0JQREaY2eqw733AJWZ2UR372geMN7Mlku4GxpnZ5eG6TwJPAgVmVikpDzgIdDSz/dFm6dyp+RGJc03zBeBFM9sdLv8mbOsKZACbE/puTNxQ0jckrZJ0QNJ+oADoktDl/YT3R4HdZlaZsAyQ2zxpONd4GckOwLm2SlJ74FogLmlH2NwOKASKgAqgD7A6XNc3YduLgW8BE4EVZlYVHpGohcJ3rtl4IXGu8T4FVAIjgPKE9pkE4ya/A+6WdDPQn+BIZUPYJ4+g0OwCMiTdAeS3SNTONTM/teVc430BeMTMNpnZjuoX8DPgemAawamnHQRXdD2SsO0c4AXgXYJTXsf48Gkw59oMH2x3zjnXJH5E4pxzrkm8kDjnnGsSLyTOOeeaxAuJc865JkmLy3+7dOli/fv3b9S2hw8fpkOHDs0bUCuXjjlDeuadjjlDeubdmJwXLly428y61tcvLQpJ//79WbBgQaO2LS0tZfz48c0bUCuXjjlDeuadjjlDeubdmJwlbay/l5/acs4510ReSJxzzjWJFxLnnHNN4oXEOedck3ghcc451yReSJxzzjWJFxLnnHNN4oXkFH49bwNvbq9IdhjOOdeqeSE5hZkLNvPKlhPJDsM551q1SAuJpEmS3pG0JnwCXM31fSXNlbRI0lJJV4XtmZJ+JWlZ+Ezr7zR0n81pRK8CNh6swp/Z4pxzdYuskEiKAw8AVwLDgCmShtXodicw08xGAtcBD4btfwu0M7MRwCjgHyT1b+A+m83wXgUcPgFb9h2N6iOcc67Ni/KI5AJgjZmtM7NyYAYwuUYf4+RzqguAbQntHSRlAO0Jnod9sIH7bDbDexYAsGzrgag+wjnn2rwoJ23sxYefQb0FGF2jz93Ai5K+CnQALgvbnyEoENuBHOBWM9srqSH7BEDSVGAqQFFREaWlpaedQHmlEZMxe94ycva8c9rbt1VlZWWN+ufV1qVj3umYM6Rn3lHmnOzZf6cAj5rZ/ZLGAo9JGk5w5FEJ9AQ6An+R9NLp7NjMpgPTAUpKSqyxM332fuN5DsQLGD++1nqVktJxZlRIz7zTMWdIz7yjzDnKQrIV6JOw3DtsS/RFYBKAmc2TlA10Af4OeMHMTgA7Jb0OlBAcjdS3z2bVvyDGsq0HMDMkRflRzjnXJkU5RjIfGCxpgKQsgsH0WTX6bAImAkgaCmQDu8L2j4XtHYAxwOoG7rNZ9cuPse/ICbbu9wF355yrTWSFxMwqgGnAHGAVwdVZKyTdK+nqsNvtwC2SlgBPAjdZcK3tA0CupBUExeMRM1ta1z6jygGgf37wj2j51oNRfoxzzrVZkY6RmNlsYHaNtrsS3q8ExtWyXRnBJcAN2meU+uTFiMfE8q0HmDS8e0t9rHPOtRl+Z3s9suJicLdcvwTYOefq4IWkAYb3KmB5OODunHPuw7yQNMCIXgXsOVzOjoPHkh2Kc861Ol5IGmB4r/AO9y1+ess552ryQtIAw3rkExMs93ES55z7CC8kDdA+K84gH3B3zrlaeSFpoHN7F7Jkiw+4O+dcTV5IGqi4byF7D5ezae+RZIfinHOtiheSBiruUwjA4s37kxyJc861Ll5IGujMojzaZ8ZZtMkLiXPOJfJC0kAZ8Rgjehf4EYlzztXgheQ0jOxTyMptBzleUZnsUJxzrtXwQnIaivsUUl5ZxcptPhOwc85V80JyGkb27Qj4gLtzziXyQnIauhdk0z0/2wuJc84l8EJymor7FHohcc65BF5ITtPIvoVs3HOEPWXHkx2Kc861Cl5ITlP1jYlLtvhRiXPOgReS0zaidwHxmFjsNyY65xzgheS05WRlMKQoj0U+TuKcc0DEhUTSJEnvSFoj6Y5a1veVNFfSIklLJV0Vtl8vaXHCq0pScbhuiqRlYf8XJHWJMofajOpXyOJN+6ms8pmAnXMuskIiKQ48AFwJDAOmSBpWo9udwEwzGwlcBzwIYGZPmFmxmRUDNwLrzWyxpAzgx8AEMzsHWApMiyqHupzfvxOHjleweoffmOicc1EekVwArDGzdWZWDswAJtfoY0B++L4A2FbLfqaE2wIofHWQpHDb2raJVEn/TgAs2LCvpT/aOedaHUX1oCZJ1wCTzOxL4fKNwGgzm5bQpwfwItAR6ABcZmYLa+xnLTDZzJYn7PeXwGHgPYKjk49MfiVpKjAVoKioaNSMGTNqdmmQsrIycnNzP9J+W+kRBhXG+Kfi7EbttzWrK+dUl455p2POkJ55NybnCRMmLDSzkvr6ZTQ6quYxBXjUzO6XNBZ4TNJwM6sCkDQaOJJQRDKBLwMjgXXAT4HvAN+ruWMzmw5MBygpKbHx48c3KsDS0lJq2/ai7Yt4c/0eLr30UoKDo9RRV86pLh3zTsecIT3zjjLnKE9tbQX6JCz3DtsSfRGYCWBm84BsIHHw/DrgyYTl4rDvWgsOpWYCFzZv2A1zfv+OvH/wOFv2HU3GxzvnXKsRZSGZDwyWNEBSFkFRmFWjzyZgIoCkoQSFZFe4HAOu5eT4CASFaJikruHy5cCqyDI4hfMHBOMk8zfsTcbHO+dcqxFZITGzCoIrquYQ/NjPNLMVku6VdHXY7XbgFklLCI48brKTgzaXAJvNbF3CPrcB9wCvSlpKcIRyX1Q5nMqQbnnkZWcw3wfcnXNpLtIxEjObDcyu0XZXwvuVwLg6ti0FxtTS/hDwULMG2gixmCjp15EFfkTinEtzfmd7E5T078R7O8vYd7g82aE451zSeCFpgvOr7yfZ6Ke3nHPpywtJE5zTu4CseMxPbznn0poXkibIzowzoncBb3khcc6lMS8kTXTBgE4s23KAw8crkh2Kc84lhReSJhp7RmcqqszHSZxzacsLSROV9O9IZlzMW7sn2aE451xSeCFpopysDIr7FDJv7e5kh+Kcc0nhhaQZjD2jM8u2HuDgsRPJDsU551qcF5JmMGZgZ6oM5q/3q7ecc+nHC0kzOK9vR7IyYvzVx0mcc2nIC0kzyM6MM6pvRx9wd86lJS8kzeTCgZ1Zuf2gz7vlnEs7XkiaydiBnQF4c70flTjn0osXkmZyTu9C2mfG/fSWcy7teCFpJlkZMc4f0MkH3J1zaccLSTMaN7Az7+0s4/2Dx5IdinPOtRgvJM3o4sHBo+RfeXdXkiNxzrmW44WkGQ3tkUfXvHa86oXEOZdGvJA0I0lcPLgLr63ZTWWVJTsc55xrEZEWEkmTJL0jaY2kO2pZ31fSXEmLJC2VdFXYfr2kxQmvKknF4bosSdMlvStptaTPRpnD6bp0SFf2HznBsq0Hkh2Kc861iMgKiaQ48ABwJTAMmCJpWI1udwIzzWwkcB3wIICZPWFmxWZWDNwIrDezxeE2/wLsNLMh4X5fiSqHxrhoUBck/PSWcy5tRHlEcgGwxszWmVk5MAOYXKOPAfnh+wJgWy37mRJuW+1m4N8BzKzKzFrV/O2dc9sxvGeBFxLnXNqQWTTn8iVdA0wysy+FyzcCo81sWkKfHsCLQEegA3CZmS2ssZ+1wGQzWy6pEFgGPA2MB9YC08zs/Vo+fyowFaCoqGjUjBkzanZpkLKyMnJzc09rm2feLWf2+hP87GM55GSqUZ+bTI3JORWkY97pmDOkZ96NyXnChAkLzayk3o5mFskLuAZ4OGH5RuBnNfrcBtwevh8LrARiCetHA8sSlrsQHMVck7D9Y/XFMmrUKGusuXPnnvY2b6zdbf2+/Sd7ftm2Rn9uMjUm51SQjnmnY85m6Zl3Y3IGFlgDfu+jPLW1FeiTsNw7bEv0RWAmgJnNA7LDYlHtOuDJhOU9wBHgd+Hy08B5zRdy8zivX0dy22Xwyrut6qybc85FIspCMh8YLGmApCyCojCrRp9NwEQASUMJCsmucDkGXEvC+EhYIZ8lOK1FuO3K6FJonMx4jLEDO/Pqu7uqj6Sccy5lRVZIzKwCmAbMAVYRXJ21QtK9kq4Ou90O3CJpCcGRx0128pf3EmCzma2rsetvA3dLWkpwuuz2qHJoivFndmXr/qOs2VmW7FCccy5SGVHu3MxmA7NrtN2V8H4lMK6ObUuBMbW0byQoMq3ax87qBsBLq3YyuCgvydE451x0/M72iPQoaM/ZPfN5edVHLihzzrmU4oUkQhOHFvH2pn3s9acmOudSmBeSCF02tBtVBnNX70x2KM45FxkvJBEa3rOAbnnteHm1n95yzqUuLyQRisXExKHdePXd3ZRXVCU7HOeci4QXkohdNrSIsuMVvLneH8HrnEtNXkgiNm5QF7IzY7y8ysdJnHOpyQtJxLIz41w0qAsvrXrf73J3zqUkLyQt4LKhRWzZd5RV2w8lOxTnnGt2XkhawGXDiogJXli+PdmhOOdcs/NC0gK65LZj9IDOzF6+I9mhOOdcs/NC0kKuHNGdNTvLeO99P73lnEstXkhayBVnd0eC5/2oxDmXYryQtJCi/GxK+nVk9jIfJ3HOpRYvJC1o0vAerN5xiPW7Dyc7FOecazZeSFrQpOHdAXjer95yzqUQLyQtqFdhe4r7FPL8Mh8ncc6lDi8kLeyqEd1ZtvUAm/ceSXYozjnXLLyQtLCrRvQAYNaSbUmOxDnnmocXkhbWu2MO5/fvyB8WbfW5t5xzKSHSQiJpkqR3JK2RdEct6/tKmitpkaSlkq4K26+XtDjhVSWpuMa2syQtjzL+qEwu7sV7O8tYuf1gskNxzrkmi6yQSIoDDwBXAsOAKZKG1eh2JzDTzEYC1wEPApjZE2ZWbGbFwI3AejNbnLDvzwBlUcUetatG9CAjJmYt9tNbzrm2L8ojkguANWa2zszKgRnA5Bp9DMgP3xcAtf2yTgm3BUBSLnAb8L1mj7iFdOqQxaVDujJryTaqqvz0lnOubVNDztNLeszMbqyvrcb6a4BJZvalcPlGYLSZTUvo0wN4EegIdAAuM7OFNfazFphsZsvD5R8CrwKLgD+Z2fA6Pn8qMBWgqKho1IwZM2rrVq+ysjJyc3Mbte2pvLG9goeWHOfb52cztHO82fffFFHl3NqlY97pmDOkZ96NyXnChAkLzaykvn4ZDdzf2YkL4WmrUacVUe2mAI+a2f2SxgKPSRpuZlXh54wGjiQUkWJgoJndKqn/qXZsZtOB6QAlJSU2fvz4RgVYWlpKY7c9lQvKK/j1qpfYSFe+PP6cZt9/U0SVc2uXjnmnY86QnnlHmfMpT21J+o6kQ8A5kg6Gr0PATuCP9ex7K9AnYbl32Jboi8BMADObB2QDXRLWXwc8mbA8FiiRtAF4DRgiqbSeOFqlnKwMrji7O7OXbed4RWWyw3HOuUY7ZSExs383szzgP80sP3zlmVlnM/tOPfueDwyWNEBSFkFRmFWjzyZgIoCkoQSFZFe4HAOuJWF8xMx+bmY9zaw/cBHwrpmNb2Curc7k4p4cPFbBn/157s65Nqyhg+1/ktQBQNINkn4gqd+pNjCzCmAaMAdYRXB11gpJ90q6Oux2O3CLpCUERx432clBm0uAzWa27jRzajMuGtSFovx2zFywOdmhOOdcozV0jOTnwLmSziX48X8Y+DVw6ak2MrPZwOwabXclvF8JjKtj21JgzCn2vQGodaC9rciIx/jseb156JW17DhwjO4F2ckOyTnnTltDj0gqwiOFycDPzOwBIC+6sNLHtSV9qDL47dtbkh2Kc841SkMLySFJ3yG4OfC5cPwiM7qw0kf/Lh24YEAnnl6w2adMcc61SQ0tJJ8DjgM3m9kOgiuw/jOyqNLM50r6sGHPEd5avzfZoTjn3GlrUCEJi8cTQIGkTwDHzOzXkUaWRq4c0Z3cdhnMXOCnt5xzbU+DComka4G3gL8luCT3zfDOddcMcrIy+OS5PZi9bDuHjp1IdjjOOXdaGnpq61+A883sC2b2eYJ5tL4bXVjp59qSPhw9UckffSJH51wb09BCEjOzxLvm9pzGtq4BivsUMqxHPo+/sdEH3Z1zbUpDi8ELkuZIuknSTcBz1Lg/xDWNJD4/th+rdxxi/oZ9yQ7HOecarL65tgZJGmdm3wT+GzgnfM0jnBDRNZ+ri3uSl53BY29sTHYozjnXYPUdkfwIOAhgZr8zs9vM7Dbg9+E614xysjL421F9eGH5dnYeOpbscJxzrkHqKyRFZrasZmPY1j+SiNLcDWP6cqLSmPGWz7/lnGsb6iskhadY1745A3GBM7rmcvHgLvzmzU1UVFYlOxznnKtXfYVkgaRbajZK+hKwsJb+rhncOKYfOw4e439Xvp/sUJxzrl71zf77deD3kq7nZOEoAbKAT0cZWDqbOLSIPp3a8/Br67lyRI9kh+Occ6dU34Ot3jezC4F7gA3h6x4zGxtOm+IiEI+Jm8cNYOHGfSzc6JcCO+dat4bOtTXXzH4avv4cdVAuuNM9PzuDh/+Sss/1cs6lCL87vZXq0C6DG8b0Y86KHWzcczjZ4TjnXJ28kLRiX7iwP/GY+OVr65MdinPO1ckLSStWlJ/N5OJezFywhf1HypMdjnPO1coLSSt3y8VncPREJY/N82lTnHOtU6SFRNIkSe9IWiPpjlrW95U0V9IiSUslXRW2Xy9pccKrSlKxpBxJz0laLWmFpO9HGX9rcGb3PCae1Y1fvL6esuMVyQ7HOec+IrJCIikOPABcCQwDpkgaVqPbncBMMxsJXAc8CGBmT5hZsZkVEzwnfr2ZLQ63+S8zOwsYCYyTdGVUObQWX504mP1HTvC4T+bonGuFojwiuQBYY2brzKwcmAFMrtHHgPzwfQFQ21OdpoTbYmZHzGxu+L4ceJvg+fEprbhPIRcP7sLDf1nH0fLKZIfjnHMfoqgeohQ+ineSmX0pXL4RGG1m0xL69ABeBDoCHYDLzGxhjf2sBSab2fIa7YUEheQyM/vIzRaSpgJTAYqKikbNmDGjUXmUlZWRm5vbqG2b07v7KrnvzWNMOSuLK/pnRvpZrSXnlpaOeadjzpCeeTcm5wkTJiw0s5L6+tU3RUrUpgCPmtn9ksYCj0kabmZVAJJGA0dqKSIZwJPAT2orIgBmNp3wmSklJSU2fvz4RgVYWlpKY7dtTuOBP++ax8tbD/Ov119MdmY8ss9qLTm3tHTMOx1zhvTMO8qcozy1tRXok7DcO2xL9EVgJoCZzQOygS4J668jKBg1TQfeM7O0eibKP39sMDsPHeep+T7FvHOu9YiykMwHBksaICmLoCjMqtFnEzARQNJQgkKyK1yOAdcSjo9Uk/Q9gvGUr0cYe6s0dmBnLujfiZ/NXcORcr+CyznXOkRWSMysApgGzAFWEVydtULSvZKuDrvdDtwiaQnBkcdNdnLQ5hJgc+KpK0m9gX8huArs7fDS4C9FlUNrI4lvTjqTXYeO8+hfNyQ7HOecAyIeIzGz2cDsGm13JbxfCYyrY9tSYEyNti2Amj3QNuT8/p2YeFY3Hipdy/UX9KMgJ9qBd+ecq4/f2d4GfeOKMzl0vIKfv7I22aE455wXkrZoaI98Jp/bk0f/up73Dx5LdjjOuTTnhaSNuvXyIVRUGj966b1kh+KcS3NeSNqofp07cMOYfjw1fxOrth9MdjjOuTTmhaQN+/plg8lvn8m9z64kqhkKnHOuPl5I2rDCnCxuu3wI89btYc6K95MdjnMuTXkhaeP+7oK+DCnK5b7Zqzhe4RM6OudanheSNi4jHuO7nxjGpr1H+OVrG5IdjnMuDXkhSQEXD+7K5cOK+MnL77Fl35Fkh+OcSzNeSFLEv34yeGbY3bNW+MC7c65FeSFJEb075nDr5YN5adVOH3h3zrUoLyQp5O/HDeCs7nncPWuFP9/dOddivJCkkMx4jPs+M4L3Dx3j/hffSXY4zrk04YUkxZzXtyM3jO7Ho3/dwPwNe5MdjnMuDXghSUF3XHkWvQrb882nl/gDsJxzkfNCkoI6tMvgP685lw17jvAfL/gpLudctLyQpKixAztz04X9efSvG5i3dk+yw3HOpTAvJCnsW5POpH/nHL7x9BIOHD2R7HCccynKC0kKy8nK4IefK+b9g8f4P79b5jcqOuci4YUkxY3s25HbPj6E55ZtZ8b8zckOxzmXgiItJJImSXpH0hpJd9Syvq+kuZIWSVoq6aqw/XpJixNeVZKKw3WjJC0L9/kTSYoyh1Twj5cM5KJBXbjn2RW89/6hZIfjnEsxkRUSSXHgAeBKYBgwRdKwGt3uBGaa2UjgOuBBADN7wsyKzawYuBFYb2aLw21+DtwCDA5fk6LKIVXEYuIH155Lh6wMpv1mkV8S7JxrVlEekVwArDGzdWZWDswAJtfoY0B++L4A2FbLfqaE2yKpB5BvZm9YcML/18Cnogg+1XTLz+aHnyvm3Z2HuOO3Pl7inGs+GRHuuxeQeFJ+CzC6Rp+7gRclfRXoAFxWy34+x8kC1CvcT+I+e9X24ZKmAlMBioqKKC0tPb3oQ2VlZY3etjX6zKBMfrtkGx2O7+aK/pm19km1nBsqHfNOx5whPfOOMucoC0lDTAEeNbP7JY0FHpM03MyqACSNBo6Y2fLT3bGZTQemA5SUlNj48eMbFWBpaSmN3bY1uuQS49DjC5m5eieTLzmPMWd0/kifVMu5odIx73TMGdIz7yhzjvLU1lagT8Jy77At0ReBmQBmNg/IBrokrL8OeLLGPnvXs093CrGYuP/ac+nXOYevPPE2m/f6g7Ccc00TZSGZDwyWNEBSFkFRmFWjzyZgIoCkoQSFZFe4HAOuJRwfATCz7cBBSWPCq7U+D/wxwhxSUl52JtNvLOFEZRU3Pzrfb1Z0zjVJZIXEzCqAacAcYBXB1VkrJN0r6eqw2+3ALZKWEBx53GQnR4EvATab2boau/4n4GFgDbAWeD6qHFLZoG65PHTDKNbvPsxXnnibE5VVyQ7JOddGRTpGYmazgdk12u5KeL8SGFfHtqXAmFraFwDDmzXQNHXhoC7c95kRfOuZpdz1x+Xc9+kR+G05zrnTlezBdpdk15b0YeOewzwwdy1d87K57fIhyQ7JOdfGeCFxfOPjZ7Lr0HF+8vJ7dMzJZECyA3LOtSk+15ZDEvd9egRXnF3EPc+u5PWtPvjunGs4LyQOgIx4jB9fN5Jxgzrzi+Xl/GlpbZMMOOfcR3khcR/Izowz/cYSBhXG+NqMxTy7xIuJc65+Xkjch3Rol8Fto7IZ1bcjX3/Ki4lzrn5eSNxHZGeIR/7+fEb168jXZizi6QX+HBPnXN28kLhadWiXwSM3nc+4QV345jNLmf7q2mSH5JxrpbyQuDp1aJfBw18o4W9G9OC+2av5/vOrffp559xH+H0k7pTaZcT5yZSRFOZk8tAra9lx4Cjf/+w5ZGfGkx2ac66V8ELi6hWPie99ajg9C9vzn3PeYfO+o0y/cRSdc9slOzTnXCvgp7Zcg0jiKxMG8cDfncfyrQf41IOv884Of/67c84LiTtNf3NOD576h7EcO1HFpx54nT8u9sfBOJfuvJC401bcp5DnvnoRw3vl87UZi7nn2RU+Db1zacwLiWuUbvnZ/OaWMdw8bgCPvL6Bax6ax8Y9h5MdlnMuCbyQuEbLjMe465PDePD681i/q4y/+clr/H7RlmSH5ZxrYV5IXJNdNaIHz3/9Eob2yOPWp5Yw7Tdvs/dwebLDcs61EC8krln0KmzPk7eM4fbLhzBnxQ4+/sNXeGH59mSH5ZxrAV5IXLPJiMf46sTBzJp2Ed0LsvnHx9/mn55YyI4Dx5IdmnMuQl5IXLMb2iOf3//TOL55xZm8vGonE+8v5ZevrafCr+xyLiVFWkgkTZL0jqQ1ku6oZX1fSXMlLZK0VNJVCevOkTRP0gpJyyRlh+1TwuWlkl6Q1CXKHFzjZMZjfGXCIP731ks5f0An7v3TSj7x09d47b3dyQ7NOdfMIiskkuLAA8CVwDBgiqRhNbrdCcw0s5HAdcCD4bYZwOPAP5rZ2cB44ETY/mNggpmdAywFpkWVg2u6vp1zeOSm83nohvM4XF7BDb94ky8+Op81O8uSHZpzrplEeURyAbDGzNaZWTkwA5hco48B+eH7AqD6KUofB5aa2RIAM9tjZpWAwlcHSQq39ScvtXKSmDS8B/9766XcceVZvLl+Lx//4St865klbNl3JNnhOeeaSFFNCy7pGmCSmX0pXL4RGG1m0xL69ABeBDoCHYDLzGyhpK8Do4BuQFdghpn9R8J+fwkcBt4jODqprOXzpwJTAYqKikbNmDGjUXmUlZWRm5vbqG3bqqhzPnjc+NO6cv68uQIMLu2TwVUDMuncPrlDdv5dp490zLsxOU+YMGGhmZXU1y/Zs/9OAR41s/sljQUekzQ8jOsi4HzgCPCypIXAq8CXgZHAOuCnwHeA79XcsZlNB6YDlJSU2Pjx4xsVYGlpKY3dtq1qiZyvBrbtP8pP//weTy/YwqtbK/nMyN58efxA+nfpEOln18W/6/SRjnlHmXOU/wu4FeiTsNw7bEv0RWAmgJnNA7KBLsAW4FUz222WzOCfAAAP2UlEQVRmR4DZwHlAcdh3rQWHUjOBCyPMwUWoZ2F7/v0z51D6zfFMuaAvv1+8lQn3l/IPjy3grfV7/SFazrURURaS+cBgSQMkZREMps+q0WcTMBFA0lCCQrILmAOMkJQTDrBfCqwkKETDJHUNt78cWBVhDq4F9O6Yw72Th/PatybwlfGDeHP9Xq7973lc/bPXmTl/M0fLP3Lm0jnXikR2asvMKiRNIygKceCXZrZC0r3AAjObBdwO/I+kWwkG3m8KjzT2SfoBQTEyYLaZPQcg6R7gVUkngI3ATVHl4FpWt/xsvnHFmXxlwiB++/YWfvXXDXzrt0v53nMr+eyo3lxb0oehPfLr35FzrkVFOkZiZrMJTksltt2V8H4lMK6ObR8nuAS4ZvtDwEPNG6lrTdpnxblhTD+uH92Xt9bv5fE3N/H4Gxt55PUNDO+VzzXn9eYT5/akiz+h0blWIdmD7c7VSRKjz+jM6DM6s+/w2fxx8VaeXriFu59dyb89t4oLB3bm6nN78vFh3SnIyUx2uM6lLS8krk3o2CGLm8YN4KZxA3hnxyFmLdnKHxdv45vPLOU7sWWMHdiZK87uzmVDi+hekJ3scJ1LK15IXJtzZvc8vtn9LL7x8TNZsuUALyzfwQvLt3PnH5Zz5x+Wc3bPfD52VjcuHdKV4j6FZMR9SjnnouSFxLVZkijuU0hxn0K+PelM3ttZxsurdjJ39U4emLuGn/55DXntMhg7sDPjBnVhzBmdGdwtl1hMyQ7duZTihcSlBEkMKcpjSFEeXx4/kANHTvD62t385b1dvPrubl5c+T4AnTpkcX7/jpT060RJ/46c3bOArAw/YnGuKbyQuJRUkJPJVSN6cNWIHgBs3nuEeev28Ma6PSzYsI85K4LCkhWPMaxnPsV9Cskqq6D3zkMM6JJL3I9anGswLyQuLfTplEOfTjlcWxJMtrDz4DEWbNzH4s37WbxpP0/N38zRE5VMX/oq7TPjDO2Rx1k98hnaI5+zuucxpFueXxnmXB28kLi01C0/+0NHLBWVVcyYXUp2zyEs33qAldsP8qcl2/jNm5tObpPXjsFFuZzRJZeBXTswoGsuAzp3oGdhtg/ou7TmhcQ5gscE986LMX5Ub64Z1RsAM2PbgWO8u+MQ775/iHffL2PNrjL+sHgrh45VfLBtZlz06ZhD70459O3Unj4dc+jVsT29CoNXl9x2PsDvUpoXEufqIOmDYjDhrG4ftJsZu8qOs37XYTbuOcL6PYfZuOcwm/ceZcnm/Rw4euJD+8mMi6L8bHoUZFOUX/1qR9e8dnTNzaZrXju65GZRmJPlYzOuTfJC4txpkkS3vGy65WUz+ozOH1l/8NgJtu47yrb9R9m6/yjbDxxjx4FjbD9wlJXbDvLn1Ts5UstElDEFV5UlvgpzsuiYk0nHnCwK2mdSGP4taJ9JfvsM8rMzycmKEzznzbnk8ELiXDPLz84kv0dmnRNMmhllxyvYdeh48Co7zu5Dx9lzuJzdZcfZe7icvYfLWb3jEPuPnGD/kXKqTjGjfjwmcttlkJed8cHfDu2CV25WBjnt4uS2y6B9VpwOWcHfnKw47TPjvLunkoJN+2ifFSc7I052ZpzszBjZmXGy4jE/JecaxAuJcy1MEnnZmeRlZ3JG1/qfWFdVZRw6XsGBIyc4cPQE+4+Wc/BoBQePBcuHjp2g7FgFh45VcPBYBYePV7D3cDmb9hzhcHkFR45Xcri8ou5iNP+vdX52VkaMdhkx2mXEw78xsqpf8eBvZvzkcmZcZMZjZMRjZCW8z4iJjOrlmMgI+8ZjCpZjMeKxk8vxmi+JWLguFi7HYyKm6j7BP9e4grZYjOCvREwQi518L4mjFcbh4xXEJCSCF2HfD9q8iDaUFxLnWrlYTB+czmosM+N4RRWHj1dwpLySYycqOVJeyV/fWshZw0dwrDxYPl5RxbETlRyrqOT4iaoP/h6vqKK8oorjFZWUV1RRXhksl4f7PF5RxYnKKiqqjPKKKk5UGicqw7ZKo6Kq6pRHVUnx0px6uwQF5sPFRZwsPKouPABhX1X35cP94WQ7NfZR3TfxcxP7JMby4fh0si1hpRLWP/fPF9EuI96wfyaN5IXEuTQgKTxtFSdxVGff2jjjz+xW53bNqarKOFEVFpbK4H3QZlRUVlFZZVRWGRXh3+r3VRa8r6oyKi1sC9cH66DKEvpZuFxlGHzQz8L2yipjzZq1nDHwDCqrwAjWmQXbVvczC7b/YDl8f7I93A4+2D/woe0+2He4HLyr3s+H15OwDWHXk+0ftH7Aaqz/oL3GG32k/DQ/LyTOuRYRi4l2sTjtWsGvTmnlJsZfMjDZYaQMv4vKOedck3ghcc451yReSJxzzjWJFxLnnHNNEmkhkTRJ0juS1ki6o5b1fSXNlbRI0lJJVyWsO0fSPEkrJC2TlB22Z0maLuldSaslfTbKHJxzzp1aZNdPSIoDDwCXA1uA+ZJmmdnKhG53AjPN7OeShgGzgf6SMoDHgRvNbImkzkD1BEb/Auw0syGSYkCnqHJwzjlXvygvxLsAWGNm6wAkzQAmA4mFxIDqeSQKgG3h+48DS81sCYCZ7UnY5mbgrLC9CtgdVQLOOefqF+WprV7A5oTlLWFboruBGyRtITga+WrYPgQwSXMkvS3pWwCSCsP1/xa2Py2pKLIMnHPO1SvZtwZNAR41s/sljQUekzQ8jOsi4HzgCPCypIXAEqA38Fczu03SbcB/ATfW3LGkqcDUcLFM0juNjLEL6XfUk445Q3rmnY45Q3rm3Zic+zWkU5SFZCvQJ2G5d9iW6IvAJAAzmxcOqHchOHp51cx2A0iaDZwH/JmgsPwu3P7pcB8fYWbTgelNTULSAjMraep+2pJ0zBnSM+90zBnSM+8oc47y1NZ8YLCkAZKygOuAWTX6bAImAkgaCmQDu4A5wAhJOeHA+6XASgsmlHkWGB9uP5EPj7k455xrYZEdkZhZhaRpBEUhDvzSzFZIuhdYYGazgNuB/5F0K8HA+01hsdgn6QcExciA2Wb2XLjrbxOcAvsRQdH5+6hycM45Vz8lzhrpPkrS1PA0WdpIx5whPfNOx5whPfOOMmcvJM4555rEp0hxzjnXJF5InHPONYkXkjrUN09YqpDUJ5zvbGU4r9nXwvZOkv5X0nvh347JjrW5SYqH87z9KVweIOnN8Dt/KrzaMKVIKpT0TDhP3SpJY1P9u5Z0a/jv9nJJT0rKTsXvWtIvJe2UtDyhrdbvVoGfhPkvlXReUz7bC0ktEuYJuxIYBkwJ5wJLRRXA7WY2DBgDfCXM9Q7gZTMbDLwcLqearwGrEpb/H/BDMxsE7KOOe5TauB8DL5jZWcC5BPmn7HctqRfwz0CJmQ0nuIL0OlLzu36U8L68BHV9t1cCg8PXVODnTflgLyS1+2CeMDMrB6rnCUs5ZrbdzN4O3x8i+GHpRZDvr8JuvwI+lZwIoyGpN/A3wMPhsoCPAc+EXVIx5wLgEuAXAGZWbmb7SfHvmuA2h/bhPWk5wHZS8Ls2s1eBvTWa6/puJwO/tsAbQKGkHo39bC8ktWvIPGEpR1J/YCTwJlBkZtvDVTuAVJvT7EfAt4CqcLkzsN/MKsLlVPzOBxDce/VIeErvYUkdSOHv2sy2EkyjtImggBwAFpL633W1ur7bZv2N80LiAJCUC/wW+LqZHUxcF94kmjLXiUv6BMGjCBYmO5YWlkEw1dDPzWwkcJgap7FS8LvuSPB/3wOAnkAHPnr6Jy1E+d16IaldQ+YJSxmSMgmKyBNmVj2P2fvVh7rh353Jii8C44CrJW0gOG35MYKxg8Lw9Aek5ne+BdhiZm+Gy88QFJZU/q4vA9ab2S4zO0EwT984Uv+7rlbXd9usv3FeSGrXkHnCUkI4NvALYJWZ/SBh1SzgC+H7LwB/bOnYomJm3zGz3mbWn+C7/bOZXQ/MBa4Ju6VUzgBmtgPYLOnMsKl6rrqU/a4JTmmNCeftEydzTunvOkFd3+0s4PPh1VtjgAMJp8BOm9/ZXgcFj/39ESfnCfu/SQ4pEpIuAv4CLOPkeMH/IRgnmQn0BTYC15pZzYG8Nk/SeOAbZvYJSWcQHKF0AhYBN5jZ8WTG19wkFRNcYJAFrCOYqy5GCn/Xku4BPkdwheIi4EsE4wEp9V1LepJgQtsuwPvAvwJ/oJbvNiyqPyM4zXcE+HszW9Doz/ZC4pxzrin81JZzzrkm8ULinHOuSbyQOOecaxIvJM4555rEC4lzzrkm8ULiUoKkzpIWh68dkrYmLDdoZldJjyTcY1FXn69Iur6ZYn5NUrGkWHPPMC3pZkndE5brzc25xvLLf13KkXQ3UGZm/1WjXQT/zlfVumELk/QaMA1YDuw2s8LT3D5uZpWn2reZLW56pM6dmh+RuJQmaZCCZ608AawAekiaLmlB+IyKuxL6Vh8hZEjaL+n7kpZImiepW9jne5K+ntD/+5LeUvDsmgvD9g6Sfht+7jPhZxWfIszvA3nh0dOvw318IdzvYkkPhkct1XH9SNJS4AJJ90iar+BZGw+Fdyp/DigGnqo+IqvOLdz3DZKWhdvcF7adKufrwr5LJM1t5q/IpQAvJC4dnEXw7Ilh4Wywd5hZCcHzOC5X7c+aKQBeMbNzgXnAzXXsW2Z2AfBNoLoofRXYET7j5d8IZlQ+lTuAQ2ZWbGaflzQc+DRwoZkVE0y2eF1CXK+a2TlmNg/4sZmdD4wI100ys6eAxcDnwn2WfxBsMH3+94AJYVzjFExieaqc/xWYGLZ/up5cXBryQuLSwdoa0z9MkfQ28DYwlODhZTUdNbPnw/cLgf517Pt3tfS5iGD6DcxsCcGR0Om4DDgfWCBpMXApMDBcVw78PqHvRElvAUvCfmfXs+/RBHOL7Q4nMfwNwTNKoO6cXwd+LelL+G+Gq0VG/V2ca/MOV7+RNJjgyYgXmNl+SY8D2bVsU57wvpK6/1s53oA+p0sE87t990ONwWy1R8PpwJGUQzBf0nlmtlXS96g9l4aqK+dbCArQJ4C3JY00s31N+ByXYvz/Lly6yQcOAQcVTKt9RQSf8TpwLYCkEdR+xPOB6gcs6eS05i8B10rqErZ3ltS3lk3bE0y0uVtSHvDZhHWHgLxatnkTmBDus/qU2Sv15HNG+BS97xI8ljZVHwLlGsmPSFy6eZtgGvHVBLOhvh7BZ/yU4FTQyvCzVhI8me9UfgEslbQgHCe5B3hJUgw4AfwjsC1xAzPbI+lX4f63ExSJao8AD0s6SvDo6Opttkj6LlBKcOTzrJk9l1DEavNDSQPC/i+a2fJ6cnFpxi//da6ZhT/KGWZ2LDyV9iIwOOHRrs6lFD8ica755QIvhwVFwD94EXGpzI9InHPONYkPtjvnnGsSLyTOOeeaxAuJc865JvFC4pxzrkm8kDjnnGuS/w81p8+LFa+MfAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.title('Adam')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adamax - Adam based on infinity norm\n",
    "\n",
    "\n",
    "Check the section Adamax in the book for the detailed explanation. The update equation of Adamax is given as:\n",
    "\n",
    "First moment estimate:\n",
    "$$m_{t}=\\beta_{1} m_{t-1}+\\left(1-\\beta_{1}\\right) g_{t} -- (24) $$\n",
    "\n",
    "Second moment estimate:\n",
    "$$v_{t}=\\max \\left(\\beta_{2} \\cdot v_{t-1},\\left|g_{t}\\right|\\right) -- (25) $$\n",
    "\n",
    "Bias-corrected mt:\n",
    "$$\\hat{m}_{t}=\\frac{m_{t}}{1-\\beta_{1}^{t}} -- (26) $$\n",
    "\n",
    "Update theta:\n",
    "\n",
    "$$\\theta_{t}=\\theta_{t-1}-\\frac{\\eta}{v_{t}} \\hat{m}_{t} -- (27) $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def Adamax(data, theta, lr = 1e-2, beta1 = 0.9, beta2 = 0.999, epsilon = 1e-6, num_iterations = 500):\n",
    "    \n",
    "    loss = []\n",
    "    \n",
    "    #initialize first moment mt\n",
    "    mt = np.zeros(theta.shape[0])\n",
    "    \n",
    "    #initialize second moment vt\n",
    "    vt = np.zeros(theta.shape[0])\n",
    "    \n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients with respect to theta\n",
    "        gradients = compute_gradients(data, theta)\n",
    "        \n",
    "        #compute first moment mt as given in equation (24)\n",
    "        mt = beta1 * mt + (1. - beta1) * gradients\n",
    "        \n",
    "        #compute second moment vt as given in equation (25)\n",
    "        vt = np.maximum(beta2 * vt, np.abs(gradients))\n",
    "        \n",
    "        #compute bias-corrected estimate of mt as given in equation (26)\n",
    "        mt_hat = mt / (1. - beta1 ** (t+1))\n",
    "        \n",
    "        #update theta as give in equation (27)\n",
    "        theta = theta - ((lr / (vt + epsilon)) * mt_hat)\n",
    "        \n",
    "        #store the loss\n",
    "        loss.append(loss_function(data,theta))\n",
    "\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update model parameter using Adamax and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = Adamax(data, theta) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XuYXVV9//H355yTyT3hEkhpAgZrrMSAA4wEBOlEoA1UpRcLyQ8RKhJrxV+19gJPKT+gPNZeqGIFbbRIRUqMra2pRsFLplTLLYEQkhA0RCAEQwIGySQhmcv398fek+wczm3mzMlM5nxez3OenL332uusdRjmM2uvfVFEYGZmNlC5oW6AmZkd2hwkZmZWFweJmZnVxUFiZmZ1cZCYmVldHCRmZlYXB4nZIJF0uaQfDnU7zA42B4lZDSR1SNouafRQt8VsuHGQmFUhaQbwdiCAdw9pY8yGIQeJWXXvAx4A7gAu61sp6UhJSyW9Iukh4FeyO0m6RdKmdPtKSW/PbLte0tckfUXSDkmPS3qjpGskbU33+/VM+d+X9ERadqOkD2a2/bmkByUV0uUPSVoraUzDvhGzDAeJWXXvA+5KX78haWq6/lbgVeAY4P3pK+thoBU4AvhX4GtFv9zfBdwJHA48CtxD8v/kNOBG4J8yZbcC7wQmAb8PfErSKem2vwP2ANdKmgl8AnhvRLxaX7fNaiPfa8usPElnAcuBYyLiRUnrSX7Bf4YkRE6MiPVp2U8AZ0fEWWXq2g60R8Rjkq4HzoyI89Jt7wLuBiZHRI+kicArwOER8XKJuv4TWB4Rt6TLM4BHgBeAL0fEXw/Wd2BWjUckZpVdBtwbES+my/+arjsKKACbMmWfye4o6U/Sw1G/kPQyMBmYkinyQub9buDFiOjJLANMSOs6X9IDkn6e1nVBtq6IeJok8GaQjJTMDprCUDfAbLiSNBa4CMhL2pKuHg0cBkwFuoFjgfXptuMy+74d+DPgHGBtRPSmIxINoB2jgX8nOcT2jYjoSkckypT5TeAM4Pskh7o+WKous0bwiMSsvN8CeoBZJHMdrcAJwP+Q/FL/OnC9pHGSZpGZiAcmkgTNNqAg6TqS+Y2BaCEJsG1At6TzgexE/BTgi8AH0ja8S9IFA/wss35zkJiVdxnwpYh4NiK29L2AzwKXAFeRHHraQnJG15cy+94DfAf4Mckhr1c58DBYzSJiB/B/gSXAduD/AEszRRaRjFSWRcRLwBXAFyUdOZDPM+svT7abmVldPCIxM7O6OEjMzKwuDhIzM6uLg8TMzOrSFNeRTJkyJWbMmDGgfXfu3Mn48eMHt0HDnPvcHNzn5lBPn1euXPliRBxVrVxTBMmMGTNYsWLFgPbt6Oigvb19cBs0zLnPzcF9bg719FnSM9VL+dCWmZnVyUFiZmZ1cZCYmVldHCRmZlYXB4mZmdXFQWJmZnVxkJiZWV0cJBX8y/8+zYM/6x7qZpiZDWsNDRJJ8yQ9KWmDpKtLbD9O0nJJj0pa3fcwHkmXSFqVefVKak23LZD0eFr+O+lDfRrirgef4eEtDhIzs0oaFiSS8iTPjj6f5AlzC9KnyGVdCyyJiJOB+cBtABFxV0S0RkQrcCnw04hYJakA3ALMjYiTgNUkDxdqiJxErx/XYmZWUSNHJKcBGyJiY0TsBRYDFxaVCfY/fnQy8HyJehak+0LyjGoB4yUp3bfUPoMin3OQmJlV08h7bU3jwEeLPgfMKSpzPXCvpI8A44FzS9RzMWkARUSXpA8BjwM7gZ8AHy714ZIWAgsBpk6dSkdHR787sGvnbsbmega076Gss7PTfW4C7nNzOBh9HuqbNi4A7oiImyWdAdwpaXZE9AJImgPsiog16fIo4EPAycBG4B+Ba4CbiiuOiEUkz7Kmra0tBnLTslvW/YiunTt8k7cm4D43B/e5MRp5aGszcGxmeXq6LusKYAlARNwPjAGyk+fzgbszy61p2aciedj8EuBtg9vs/Qo50YuPbZmZVdLIIHkYmCnpeEktJKGwtKjMs8A5AJJOIAmSbelyDriI/fMjkATRLEl998c/D3iiUR3ISfT0Nqp2M7ORoWGHtiKiW9JVwD1AHrg9ItZKuhFYERFLgY8DX5D0MZKJ98vTkQbA2cCmiNiYqfN5STcA90nqAp4BLm9UH/I5eTxiZlZFQ+dIImIZsKxo3XWZ9+uAM8vs2wGcXmL954HPD2pDy8jnPCIxM6vGV7ZXkM8J54iZWWUOkgoKvo7EzKwqB0kFvrLdzKw6B0kFyZXtThIzs0ocJBXkc6LHOWJmVpGDpIJ8TnhAYmZWmYOkAo9IzMyqc5BUkPdku5lZVQ6SCnwbeTOz6hwkFfisLTOz6hwkFXhEYmZWnYOkAk+2m5lV5yCpIC+f/mtmVo2DpIJ83iMSM7NqHCQV+PRfM7PqHCQVeLLdzKw6B0kFfU9IDE+UmJmV5SCpIC8B0ONhiZlZWQ6SCvL5JEi6HSRmZmU5SCroG5H46nYzs/IcJBXkcx6RmJlV4yCpoC9Ieh0kZmZlOUgq8IjEzKw6B0kFHpGYmVXnIKlg3+m/nmw3MyvLQVJBru/Qlm+4ZWZWVkODRNI8SU9K2iDp6hLbj5O0XNKjklZLuiBdf4mkVZlXr6TWdFuLpEWSfixpvaTfbVT7Czmf/mtmVk2hURVLygO3AucBzwEPS1oaEesyxa4FlkTE5yTNApYBMyLiLuCutJ4Tgf+MiFXpPn8BbI2IN0rKAUc0qg+ebDczq65hQQKcBmyIiI0AkhYDFwLZIAlgUvp+MvB8iXoWAIszy+8H3gQQEb3Ai4Pb7P082W5mVl0jg2QasCmz/Bwwp6jM9cC9kj4CjAfOLVHPxSQBhKTD0nV/JakdeAq4KiJeKN5J0kJgIcDUqVPp6Ojodwee2NINwAMPPczmic0zndTZ2Tmg7+tQ5j43B/e5MRoZJLVYANwRETdLOgO4U9LsdKSBpDnArohYk5YvANOB/42IP5b0x8DfA5cWVxwRi4BFAG1tbdHe3t7vxu1ZuwVWreTkU05l9rTJA+jeoamjo4OBfF+HMve5ObjPjdHIP7M3A8dmlqen67KuAJYARMT9wBhgSmb7fODuzPJLwC7g6+ny14BTBq/JB/Jku5lZdY0MkoeBmZKOl9RCEgpLi8o8C5wDIOkEkiDZli7ngIvIzI9E8mCQ/wLa01XncOCcy6DKebLdzKyqhh3aiohuSVcB9wB54PaIWCvpRmBFRCwFPg58QdLHSCbeL4/9T5E6G9jUN1mf8eckh8A+TRI6v9+oPhQ82W5mVlVD50giYhnJKb3Zdddl3q8Dziyzbwdweon1z5CETMP1XdnuEYmZWXnNcyrSAOQ8IjEzq8pBUkHBcyRmZlU5SCrouyDRN200MyvPQVJBIZd8PT2+aaOZWVkOkgr232urd4hbYmY2fDlIKhiV9xyJmVk1DpIK9s2ROEjMzMpykFTQN0fiB1uZmZXnIKkgn/eIxMysGgdJBX3XkXR5st3MrCwHSQWeIzEzq85BUsEoz5GYmVXlIKnAcyRmZtU5SCrwvbbMzKpzkFSw78r2Hk+2m5mV4yCpwCMSM7PqHCQVSCInz5GYmVXiIKkiJ49IzMwqcZBUkRf0+IJEM7OyHCRVeERiZlaZg6SKvHxBoplZJQ6SKvI5eURiZlaBg6QKz5GYmVXmIKnCcyRmZpU5SKrI+zoSM7OKGhokkuZJelLSBklXl9h+nKTlkh6VtFrSBen6SyStyrx6JbUW7btU0ppGth/SEYkn283MympYkEjKA7cC5wOzgAWSZhUVuxZYEhEnA/OB2wAi4q6IaI2IVuBS4KcRsSpT9+8AnY1qe1Ze0O05EjOzsho5IjkN2BARGyNiL7AYuLCoTACT0veTgedL1LMg3RcASROAPwZuGvQWl5DPyYe2zMwqKDSw7mnApszyc8CcojLXA/dK+ggwHji3RD0Xc2AA/RVwM7Cr0odLWggsBJg6dSodHR39aHpGbw9bt7008P0PQZ2dnU3VX3Cfm4X73BiNDJJaLADuiIibJZ0B3ClpdkT0AkiaA+yKiDXpcivwKxHxMUkzKlUcEYuARQBtbW3R3t4+oAaOeuDbTD78cNrbizNw5Oro6GCg39ehyn1uDu5zYzTy0NZm4NjM8vR0XdYVwBKAiLgfGANMyWyfD9ydWT4DaJP0NPBD4I2SOga11UVygi4/j8TMrKxGBsnDwExJx0tqIQmFpUVlngXOAZB0AkmQbEuXc8BFZOZHIuJzEfHLETEDOAv4cUS0N7APPv3XzKyKhgVJRHQDVwH3AE+QnJ21VtKNkt6dFvs4cKWkx0hGHpdHRN9v7bOBTRGxsVFtrEVevkWKmVklDZ0jiYhlwLKidddl3q8DziyzbwdweoW6nwZmD0Y7K8nlPCIxM6vEV7ZX4bv/mplV5iCpIucLEs3MKnKQVJH3TRvNzCpykFThs7bMzCpzkFSRkzxHYmZWgYOkinzOcyRmZpU4SKrwWVtmZpU5SKoo5GCvb5FiZlaWg6QKj0jMzCpzkFSR3CLFIxIzs3IcJFXkc9DVE+y/BZiZmWU5SKoopN+QryUxMyvNQVJFXsm/XZ4nMTMrqaYgkXRnLetGorySJOnyPImZWUm1jkjenF2QlAdOHfzmDD/59BvymVtmZqVVDBJJ10jaAZwk6ZX0tQPYCnzjoLRwiBXSQ1vdvpbEzKykikESEX8dEROBv4uISelrYkQcGRHXHKQ2Dqm+EUmXJ9vNzEqq9dDWNyWNB5D0Xkn/IOl1DWzXsLFvsr3bIxIzs1JqDZLPAbskvYXkOetPAV9uWKuGkXwuSRJflGhmVlqtQdIdyRV5FwKfjYhbgYmNa9bwUfDpv2ZmFRVqLLdD0jXApcDbJeWAUY1r1vDhs7bMzCqrdURyMbAHeH9EbAGmA3/XsFYNI31zJL4DsJlZaTUFSRoedwGTJb0TeDUimmKOpNA3R+IgMTMrqdYr2y8CHgJ+D7gIeFDSexrZsOGib0TS7dN/zcxKqnWO5C+At0bEVgBJRwHfA/6tUQ0bLvZdR+IRiZlZSbXOkeT6QiT1Ui37Spon6UlJGyRdXWL7cZKWS3pU0mpJF6TrL5G0KvPqldQqaZykb0laL2mtpE/W2P4B239lu0ckZmal1Bok35F0j6TLJV0OfAtYVmmH9H5ctwLnA7OABZJmFRW7FlgSEScD84HbACLirohojYhWkjPFfhoRq9J9/j4i3gScDJwp6fwa+zAgfdeReERiZlZaxUNbkt4ATI2IP5X0O8BZ6ab7SSbfKzkN2BARG9O6FpNch7IuUyaASen7ycDzJepZACwGiIhdwPL0/V5Jj5CcQdYw+65s9xyJmVlJqvTkP0nfBK6JiMeL1p8IfCIi3lVh3/cA8yLiA+nypcCciLgqU+YY4F7gcGA8cG5ErCyq5yngwohYU7T+MOCRdJ+NJT5/IbAQYOrUqacuXry4bD8r+em2Tm5YKa48sYUzpzXFpTN0dnYyYcKEoW7GQeU+Nwf3uX/mzp27MiLaqpWrNtk+tThEACLicUkzBtSyAy0A7oiImyWdAdwpaXZE9AJImgPsKhEiBeBu4DOlQiRt4yJgEUBbW1u0t7cPqIEvffsHwG5mvvFNtL/12AHVcajp6OhgoN/Xocp9bg7uc2NUmyM5rMK2sVX23Qxkf/NOT9dlXQEsAYiI+4ExwJTM9vkkgVFsEfCTiPh0lTbUbf+hLc+RmJmVUi1IVki6snilpA8AK0uUz3oYmCnpeEktJKGwtKjMs8A5aZ0nkATJtnQ5R3LNygHHpCTdRDKf8tEqnz8o+i5I9N1/zcxKq3Zo66PAf0i6hP3B0Qa0AL9daceI6JZ0FXAPkAduj4i1km4EVkTEUpI7CX9B0sdIJt4vj/2TNmcDm7KHriRNJ7mmZT3wiJLH4H42Ir5Yc4/7ad+9tjzZbmZWUsUgiYgXgLdJmgvMTld/KyJ+UEvlEbGMotOEI+K6zPt1wJll9u0ATi9a9xygWj57sOR9918zs4pqurI9IpaTnnbbbPYHiQ9tmZmVUusFiU0rJ5B800Yzs3IcJFVIoiWfY4+DxMysJAdJDVoKObq6PUdiZlaKg6QGLfkce3t6hroZZmbDkoOkBi2FHHt9HYmZWUkOkhqMyud8+q+ZWRkOkhp4RGJmVp6DpAaj8jn2OEjMzEpykNSgpZDzBYlmZmU4SGowOu9DW2Zm5ThIajCqIPZ6RGJmVpKDpAYteR/aMjMrx0FSg1E+tGVmVpaDpAY+/dfMrDwHSQ1aCjnPkZiZleEgqUGLD22ZmZXlIKmBryMxMyvPQVIDT7abmZXnIKmB50jMzMpzkNSgJb37b4TvAGxmVsxBUoOWQvI1eVRiZvZaDpIatOTTIPE8iZnZazhIajAqLwA/3MrMrAQHSQ1aCnnAIxIzs1IaGiSS5kl6UtIGSVeX2H6cpOWSHpW0WtIF6fpLJK3KvHoltabbTpX0eFrnZySpkX0AGJ3Okezp7mn0R5mZHXIaFiSS8sCtwPnALGCBpFlFxa4FlkTEycB84DaAiLgrIlojohW4FPhpRKxK9/kccCUwM33Na1Qf+owZlYxIXu3yiMTMrFgjRySnARsiYmNE7AUWAxcWlQlgUvp+MvB8iXoWpPsi6RhgUkQ8EMm5uF8GfqsRjc/yiMTMrLxCA+ueBmzKLD8HzCkqcz1wr6SPAOOBc0vUczH7A2haWk+2zmmD0dhKPCIxMyuvkUFSiwXAHRFxs6QzgDslzY6IXgBJc4BdEbGmvxVLWggsBJg6dSodHR0DamBnZyfPr3kMgIdWPsquZ/IDqudQ0tnZOeDv61DlPjcH97kxGhkkm4FjM8vT03VZV5DOcUTE/ZLGAFOAren2+cDdRXVOr1InaX2LgEUAbW1t0d7ePqBOdHR0cMYbToYHf8ivzppN+6ypA6rnUNLR0cFAv69DlfvcHNznxmjkHMnDwExJx0tqIQmFpUVlngXOAZB0AjAG2JYu54CLSOdHACLiZ8Arkk5Pz9Z6H/CNBvYBgNGjPEdiZlZOw4IkIrqBq4B7gCdIzs5aK+lGSe9Oi30cuFLSYyQjj8tj/w2tzgY2RcTGoqr/EPgisAF4Cvh2o/rQZ0zBcyRmZuU0dI4kIpYBy4rWXZd5vw44s8y+HcDpJdavAGYPakOr8IjEzKw8X9leA49IzMzKc5DUwCMSM7PyHCQ16Lsg0SMSM7PXcpDUQBIthZxHJGZmJThIajSmkGOPRyRmZq/hIKnR6FF5j0jMzEpwkNRozKic50jMzEpwkNRodMEjEjOzUhwkNfKIxMysNAdJjcYU8uze6xGJmVkxB0mNxrbk2dXlIDEzK+YgqdH4lgK793YPdTPMzIYdB0mNxrXk2eVDW2Zmr+EgqdHYFs+RmJmV4iCp0biWPDt9aMvM7DUcJDUa21Lg1a5eenujemEzsybiIKnR+JbkmSS7feaWmdkBHCQ1GpcGiSfczcwO5CCp0diW5KnEuzxPYmZ2AAdJjTwiMTMrzUFSo7EOEjOzkhwkNRqfHtrytSRmZgdykNSo79CWryUxMzuQg6RG40cnI5KdexwkZmZZDpIaTRyTBMmOVx0kZmZZDpIa7Q+SriFuiZnZ8NLQIJE0T9KTkjZIurrE9uMkLZf0qKTVki7IbDtJ0v2S1kp6XNKYdP2CdHm1pO9ImtLIPvQZXcgzupDjFY9IzMwO0LAgkZQHbgXOB2YBCyTNKip2LbAkIk4G5gO3pfsWgK8AfxARbwbaga50/S3A3Ig4CVgNXNWoPhSbOGaURyRmZkUaOSI5DdgQERsjYi+wGLiwqEwAk9L3k4Hn0/e/DqyOiMcAIuKliOgBlL7GS1K67/McJJPGFDwiMTMroojG3M1W0nuAeRHxgXT5UmBORFyVKXMMcC9wODAeODciVkr6KHAqcDRwFLA4Iv42U+/twE7gJySjk9dc3CFpIbAQYOrUqacuXrx4QP3o7OxkwoQJANx4/27GjRJ/0jZmQHUdKrJ9bhbuc3Nwn/tn7ty5KyOirVq5woBqHzwLgDsi4mZJZwB3Spqdtuss4K3ALuD7klYC9wEfAk4GNgL/CFwD3FRccUQsAhYBtLW1RXt7+4Aa2NHRQd++//zUg+x4tZv29jMHVNehItvnZuE+Nwf3uTEaeWhrM3BsZnl6ui7rCmAJQETcD4wBpgDPAfdFxIsRsQtYBpwCtKZln4pkKLUEeFsD+3CAiWMKniMxMyvSyCB5GJgp6XhJLSST6UuLyjwLnAMg6QSSINkG3AOcKGlcOsH+a8A6kiCaJemodP/zgCca2IcDTBozyteRmJkVadihrYjolnQVSSjkgdsjYq2kG4EVEbEU+DjwBUkfI5l4vzwdaWyX9A8kYRTAsoj4FoCkG4D7JHUBzwCXN6oPxSaPHcXLu7uICJK5fjMza+gcSUQsIzkslV13Xeb9OqDkhENEfIXkFODi9Z8HPj+4La3NEeNb2Nvdy869PUwYPdTTS2Zmw4OvbO+HI8a3APDzzr1D3BIzs+HDQdIPR05IguSlnXuGuCVmZsOHg6Qfjhg/GoCf7/SIxMysj4OkH45MD2295ENbZmb7OEj6Yf+hLQeJmVkfB0k/jGspMHZUnm07PEdiZtbHQdJP0w4fy/Mv7x7qZpiZDRsOkn6adthYNjtIzMz2cZD00/TDx/Lc9l1D3Qwzs2HDQdJP0w8fx/ZdXezc43tumZnB0N9G/pBz7BFjAXj6pZ28+ZcnH7DtnrVbuG35Bja/vJvWYw/jyre/njmvP3IommlmdtB4RNJPs45JHui4dvMrB6xfdN9TfPDOlezu6uEdbzqaVZte5uJFD3D90rXs6X7Nc7fMzEYMj0j6acaR45kwusCa53/BRenjVn6w/gU+sWw9v3nSMXz64lZG5XO82tXD33xnPV/60dM8+ux2bnvvqUw7bOwQt97MbPB5RNJPuZyYPW0SD/305wBs+vkuPvbVx5h1zCRu/r23MCqffKVjRuX5f+96M/906als3LaTd37mf7jvx9uGsulmZg3hIBmA33jzL7F+yw4e2/Qyf3jXI/RG8Ln3nsKYUfmSZZd+5CyOnjiGy770EH/7nfWeqDezEcVBMgDvessvM3F0gQtv/RGPb/4FN//eW3jdkePLlj9+ynj+48Nv43dPmc5tHU9x1t/8gBv+ay33rt3Cc9t38WpXMocSEfT2Bl09vezt7qW7p5fkOV9mZsOXmuEXVVtbW6xYsWJA+3Z0dNDe3v6a9Y88u52vPrSJeSf+EnN/9eia63vk2e188X828t11L9DVU9t3L0FOIi+Ry5H+K/I5Ucgp2Zb5N58T9TzAcdeuXYwbN658ewZedUWNeupkLbXu3LWT8ePK/zEwErnPzeHPWns57x1zB7SvpJUR0VatnCfbB+iU4w7nlOMOH9B+t11yKrv39rDuZ7/gyS2dvLx7L7v39pCT0lcSHr0BPb1BRNATQU8v9EbQ05u8eiPo7k1GMT29SZne3mRdPbZufZWjj55UclvD/uxoUMVRY8Vbt+7m6KMnNKYRw5T73BzEK9UL1clBMkTGtuQ59XVHcOrrjhjqprxGMgo7ZaibcVAlfT51qJtxULnPzaGjo6Phn+E5EjMzq4uDxMzM6uIgMTOzujhIzMysLg4SMzOri4PEzMzq4iAxM7O6OEjMzKwuTXGLFEnbgGcGuPsU4MVBbM6hwH1uDu5zc6inz6+LiKOqFWqKIKmHpBW13GtmJHGfm4P73BwORp99aMvMzOriIDEzs7o4SKpbNNQNGALuc3Nwn5tDw/vsORIzM6uLRyRmZlYXB4mZmdXFQVKGpHmSnpS0QdLVQ92ewSTpdklbJa3JrDtC0ncl/ST99/B0vSR9Jv0eVks65J54JelYScslrZO0VtIfpetHcp/HSHpI0mNpn29I1x8v6cG0b1+V1JKuH50ub0i3zxjK9tdDUl7So5K+mS6P6D5LelrS45JWSVqRrjuoP9sOkhIk5YFbgfOBWcACSbOGtlWD6g5gXtG6q4HvR8RM4PvpMiTfwcz0tRD43EFq42DqBj4eEbOA04EPp/89R3Kf9wDviIi3AK3APEmnA38DfCoi3gBsB65Iy18BbE/Xfyotd6j6I+CJzHIz9HluRLRmrhc5uD/bEeFX0Qs4A7gns3wNcM1Qt2uQ+zgDWJNZfhI4Jn1/DPBk+v6fgAWlyh2qL+AbwHnN0mdgHPAIMIfkCudCun7fzzlwD3BG+r6QltNQt30AfZ1O8ovzHcA3ATVBn58GphStO6g/2x6RlDYN2JRZfi5dN5JNjYifpe+3AFPT9yPqu0gPX5wMPMgI73N6iGcVsBX4LvAU8HJEdKdFsv3a1+d0+y+AIw9uiwfFp4E/A3rT5SMZ+X0O4F5JKyUtTNcd1J/tQr0V2MgTESFpxJ0XLmkC8O/ARyPiFUn7to3EPkdED9Aq6TDgP4A3DXGTGkrSO4GtEbFSUvtQt+cgOisiNks6GviupPXZjQfjZ9sjktI2A8dmlqen60ayFyQdA5D+uzVdPyK+C0mjSELkroj4erp6RPe5T0S8DCwnOaxzmKS+PyCz/drX53T7ZOClg9zUep0JvFvS08BiksNbtzCy+0xEbE7/3UryB8NpHOSfbQdJaQ8DM9OzPVqA+cDSIW5Toy0FLkvfX0Yyj9C3/n3p2R6nA7/IDJkPCUqGHv8MPBER/5DZNJL7fFQ6EkHSWJI5oSdIAuU9abHiPvd9F+8BfhDpQfRDRURcExHTI2IGyf+zP4iISxjBfZY0XtLEvvfArwNrONg/20M9UTRcX8AFwI9Jjiv/xVC3Z5D7djfwM6CL5BjpFSTHhr8P/AT4HnBEWlYkZ7A9BTwOtA11+wfQ37NIjiOvBlalrwtGeJ9PAh5N+7wGuC5d/3rgIWAD8DVgdLp+TLq8Id3++qHuQ539bwe+OdL7nPbtsfS1tu931cH+2fYtUszMrC4+tGVmZnVxkJiZWV0cJGZmVhcHiZmZ1cVBYmZmdXGQ2Igg6cj07qerJG2RtDmz3FJjHV+S9KtVynxY0iWD1OYfSmqVlNMg32Fa0vsl/VJmuWrfzAbKp//aiCPpeqAzIv6+aL1IfuZ7S+54kEn6IXAVyXUeL0bEYf2Zu4TGAAADGElEQVTcPx/JbVDK1h0Rq+pvqVllHpHYiCbpDUqeQ3IXyQVbx0haJGmFkud0XJcp2zdCKEh6WdInlTzP4/70PkZIuknSRzPlP6nkuR9PSnpbun68pH9PP/ff0s9qrdDMTwIT09HTl9M6LkvrXSXptnTU0teuT0taDZwm6QZJD0taI+nz6RXLF5PcOv6rfSOyvr6ldb9XyfMr1kj6RLquUp/np2Ufk7R8kP8T2QjgILFm8CaS51HMiuS+RFdH8tyGtwDnqfSzZiYD/x3J8zzuB95fpm5FxGnAnwJ9ofQRYEskzz/5K5K7DVdyNbAjkudJvE/SbOC3gbdFRCvJzVXnZ9p1X0ScFBH3A7dExFuBE9Nt8yLiqyRX71+c1rl3X2Ol6cBNwNy0XWcqudlhpT7/P+CcdP1vV+mLNSEHiTWDpyJiRWZ5gaRHSJ7RcQLJw8uK7Y6Ib6fvV5I8v6WUr5cocxbJTQOJiL5bV/THucBbgRVKbgP/a8CvpNv2ktyYr885kh4iuUXGrwFvrlL3HJJ7Sr0YEV3AvwJnp9vK9flHwJclfQD/zrASfBt5awY7+95ImknyBL3TIuJlSV8huedSsb2Z9z2U/39lTw1l+kvA7RHxlwesTO5Quzv6bpokjQM+C5wSyW3Eb6J0X2pVrs9XkgTQO4FHJJ0cEdvr+BwbYfzXhTWbScAO4BUlt9f+jQZ8xo+AiwAknUjpEc8+kT50Sftvdf494CJJU9L1R0o6rsSuY0ke4PRiegfY381s2wFMLLHPg8DctM6+Q2b/XaU/r4+IB4C/JHlU7SH3kC9rLI9IrNk8AqwD1gPPkPzSH2z/SHIoaF36WetInr5XyT8DqyWtSOdJbgC+JylHcpfmPwCez+4QES9J+pe0/p+RhESfLwFflLSb5PkUffs8J+kvgQ6Skc9/RcS3MiFWyqckHZ+Wvzci1lTpizUZn/5rNsjSX8qFiHg1PZR2LzAz9j/u1WxE8YjEbPBNAL6fBoqADzpEbCTziMTMzOriyXYzM6uLg8TMzOriIDEzs7o4SMzMrC4OEjMzq8v/Bw+JeJSTFoCHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.title('Adamax')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adaptive Moment Estimation with AMSGrad\n",
    "\n",
    "\n",
    "Refer the section AMSGrad in the book for the detailed explanation. The update equation of AMSGrad is given as:\n",
    "\n",
    "First moment estimate:\n",
    "$$m_{t}=\\beta_{1} m_{t-1}+\\left(1-\\beta_{1}\\right) g_{t} -- (28) $$\n",
    "\n",
    "Second moment estimate:\n",
    "$$v_{t}=\\beta_{2} v_{t-1}+\\left(1-\\beta_{2}\\right) g_{t}^{2} --(29) $$\n",
    "\n",
    "Updated second moment estimate:\n",
    "$$\\hat{v}_{t}=\\max \\left(\\hat{v}_{t-1}, v_{t}\\right) -- (30) $$\n",
    "\n",
    "Bias-corrected mt:\n",
    "$$\\hat{m}_{t}=\\frac{m t}{1-\\beta_{1}^{t}} -- (31) $$\n",
    "\n",
    "Update theta:\n",
    "$$\\theta_{t}=\\theta_{t-1}-\\frac{\\eta}{\\sqrt{\\hat{v}_{t}}+\\epsilon} \\hat{m}_{t} -- (32) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def AMSGrad(data, theta, lr = 1e-2, beta1 = 0.9, beta2 = 0.9, epsilon = 1e-6, num_iterations = 50):\n",
    "    \n",
    "    loss = []\n",
    "    \n",
    "    #initialize first moment mt\n",
    "    mt = np.zeros(theta.shape[0])\n",
    "    \n",
    "    #initialize second moment vt\n",
    "    vt = np.zeros(theta.shape[0])\n",
    "    \n",
    "    #initialize vt_hat\n",
    "    vt_hat = np.zeros(theta.shape[0])\n",
    "    \n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients with respect to theta\n",
    "        gradients = compute_gradients(data, theta)\n",
    "        \n",
    "        #compute first moment as given in equation (28)\n",
    "        mt = beta1 * mt + (1. - beta1) * gradients\n",
    "        \n",
    "        #compute second moment as given in equation (29)\n",
    "        vt = beta2 * vt + (1. - beta2) * gradients ** 2\n",
    "        \n",
    "        #compute vt_hat as given in equation (30)\n",
    "        vt_hat = np.maximum(vt_hat,vt)\n",
    "        \n",
    "        #compute bias-corrected estimate of mt as shown in equation (31)\n",
    "        mt_hat = mt / (1. - beta1 ** (t+1))\n",
    "        \n",
    "        #update model parameter theta as given in (32)\n",
    "        theta = theta - (lr / (np.sqrt(vt_hat) + epsilon)) * mt_hat\n",
    "\n",
    "        #store loss\n",
    "        loss.append(loss_function(data,theta))\n",
    "\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update model parameter using AMSGrad and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = AMSGrad(data, theta) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl81dWd//HXO/cmN2SBoGBkUdGKC25YQNyqWG2LbUe7WJU6aju2Thc7dcb+WjtdpuvM1Fa7aju2tU5dap0uU61M1Sqpa1VQBMUNURHBCsgWAlk/vz++3+AlZCO5l0Dyfj7MI9/9ngMxb8453+/3KCIwMzPrq5KBLoCZme3aHCRmZtYvDhIzM+sXB4mZmfWLg8TMzPrFQWJmZv3iIDEb5CS9KOmUgS6HDV4OErNuSKqTtEZSLm/btZJC0ukdjv1uuv1D6XqZpMslLZNUn/5C/16Hc86W9JCkjZJeS5c/IUk7pIJmBeAgMeuCpAnAW4AATuuw+1ngvLxjs8CZwPN5x3wemAocBVQDM4BH8865BPg+8G1gT6AW+BhwHFDWRZkyfa6QWZE4SMy6dh7wV+Ba4PwO+24Fjpc0Ml2fCSwAXs07Zhrw+4hYHokXI+KXAJJGAF8DPhERv4mIDekxj0XEORHRmB53raQfS5otaSNwkqR3SXpM0npJL0v6Sn7BJJ0r6SVJqyV9oaB/ImadcJCYde084Ib06x2SavP2bQb+AJydd+wvO5z/V+Bf0q6qwzp0Vx0D5NJr9OSDwDdJWjX3ARvTz6sB3gV8XNJ7ACRNAn4MnAuMBXYHxveqtmZ95CAx64Sk44F9gJsjYh5Jl9UHOxz2S+A8STXAicD/dtj/H8C3gHOAucArktpbNqOAVRHRkveZD0haK2mTpBPyrvOHiLg/ItoiYnNE1EXEwnR9AfCr9PMBzgD+GBH3pK2aLwFt/fvTMOueg8Ssc+cDd0TEqnT9Rjp0b0XEfcBo4Askv7w3ddjfGhFXRsRxJK2HbwLXSDoYWA2MSsdW2o8/NiJq0n35/2++nH9dSdMlzZG0UtI6knGVUenusfnHR8TG9HpmReMgMetA0jCSgfMTJb0q6VXgn4EjJB3R4fDrgUvYtltrKxGxKSKuBNYAk4AHgUbg9O7Oaz+9w/qNwC3AXhExAvgJ0N5ttgLYK68uFSTdW2ZFk+35ELMh5z1AK3AY0JS3/Wby7tRK/QC4F7in40UkXQzMBx4Cmkm6uKqBxyJiraSvAlelYye3k4x9HA5U9lC+auD1iNgs6SiSLrc70n2/AR5Ku+YeJhnQ9z8YragcJGbbOh/4RUQszd8o6UckwfHn9m0R8TpwVxfXaQAuB/YnaVU8C7w/Ipak514m6RXgsyQtmo3AEuBzwAPdlO8TwOVpef5CEnA16TWflPRJklZLJXAFsKzXNTfrA3liKzMz6w83ec3MrF8cJGZm1i8OEjMz6xcHiZmZ9cuQuGtr1KhRMWHChD6du3HjRiore7obc/BxvYcW13to6W29582btyoiRvd03JAIkgkTJjB37tw+nVtXV8eMGTMKW6BdgOs9tLjeQ0tv6y3ppd5cz11bZmbWLw4SMzPrFweJmZn1i4PEzMz6xUFiZmb94iAxM7N+cZCYmVm/OEi6ce39L/DQipaeDzQzG8KGxAOJffWrh1+mCgeJmVl33CLpRmUuw+YWz9diZtYdB0k3KnNZNrtBYmbWLQdJNyrLsmxudYvEzKw7DpJuuEViZtYzB0k3KnMZt0jMzHrgIOmGWyRmZj1zkHSjKpelNaCppW2gi2JmttNykHSjoiwDwMZGN0vMzLriIOlGZS55XnNjk4PEzKwrDpJuVJalQdLYOsAlMTPbeTlIulGZS7q26t21ZWbWJQdJN6rSrq0Gd22ZmXXJQdKNii1dWw4SM7OuOEi60d4i8RiJmVnXHCTdqEjHSHzXlplZ1xwk3WhvkXiw3cysaw6SbuSyJZQIGty1ZWbWpaIGiaSZkp6RtFjSpZ3s31vSHEmPSVog6Z3p9nMkzc/7apM0WVKFpNskPS3pSUn/WeTyk8u4RWJm1p2iBYmkDHAlcCowCZglaVKHw74I3BwRRwJnA1cBRMQNETE5IiYD5wIvRMT89JzvRMRBwJHAcZJOLVYdAIZl5dt/zcy6UcwWyVHA4ohYEhFNwE3A6R2OCWB4ujwCWN7JdWal5xIRDRExJ11uAh4Fxheh7FvkMr5ry8ysO4ooznwbks4AZkbER9L1c4HpEXFR3jFjgDuAkUAlcEpEzOtwneeB0yPiiQ7ba0iC5JSIWNLJ518IXAhQW1s75aabbupTPf7tvnqGl2e5ZGp5n87fVdXX11NVVTXQxdjhXO+hxfXu3kknnTQvIqb2dFy2IKXqu1nAtRFxuaRjgOskHRoRbQCSpgMNnYRIFvgV8IPOQgQgIq4GrgaYOnVqzJgxo08FrHj4/yivGs6MGcf26fxdVV1dHX39M9uVud5Di+tdGMXs2noF2CtvfXy6Ld8FwM0AEfEgUA6Mytt/NklgdHQ18FxEfK9gpe1CeVbUu2vLzKxLxQySR4CJkvaVVEYSCrd0OGYpcDKApINJgmRlul4CnEk6PtJO0jdIxlMuLmLZtyjP+F1bZmbdKVqQREQLcBFwO/AUyd1ZT0r6mqTT0sMuAT4q6XGSlseH4o1BmxOAl/O7riSNB75AchfYo+mtwR8pVh0Acln5XVtmZt0o6hhJRMwGZnfY9uW85UXAcV2cWwcc3WHbMkAFL2g3yjPycyRmZt3wk+09KM/C5uY2WtuKc3ebmdmuzkHSg/JM0gDyixvNzDrnIOlBedr55/dtmZl1zkHSg/YWicdJzMw65yDpQXuLxHdumZl1zkHSg5zHSMzMuuUg6cGwLS0Sj5GYmXXGQdKD8mzSIvHT7WZmnXOQ9CCdtt2D7WZmXXCQ9KC9ReLBdjOzzjlIetDeIvEYiZlZ5xwkPSiRqCjLuEViZtYFB0kvVOaybGxyi8TMrDMOkl6odIvEzKxLDpJeqMxlHSRmZl1wkPRCZVnWT7abmXXBQdILlbmM79oyM+uCg6QXksF2t0jMzDrjIOmFyjKPkZiZdcVB0gvJYLu7tszMOuMg6YXKXIaNTS1EeN52M7OOHCS9UJnLEgGbmt0qMTPryEHSC5W5ZFISd2+ZmW3LQdILlWXJmxs94G5mtq2iBomkmZKekbRY0qWd7N9b0hxJj0laIOmd6fZzJM3P+2qTNDnd901JL0uqL2bZ87W3SDwniZnZtooWJJIywJXAqcAkYJakSR0O+yJwc0QcCZwNXAUQETdExOSImAycC7wQEfPTc24FjipWuTtTWZYESYNf3Ghmto1itkiOAhZHxJKIaAJuAk7vcEwAw9PlEcDyTq4zKz03OSHirxGxogjl7VJlzl1bZmZdyRbx2uOAl/PWlwHTOxzzFeAOSZ8CKoFTOrnOWWwbQD2SdCFwIUBtbS11dXXbewkA6uvreWXBYwA88tgC9Gox/8h2HvX19X3+M9uVud5Di+tdGAP9W3EWcG1EXC7pGOA6SYdGRBuApOlAQ0Q8sb0XjoirgasBpk6dGjNmzOhTAevq6jhy8nS472722f8AZkzbu0/X2dXU1dXR1z+zXZnrPbS43oVRzK6tV4C98tbHp9vyXQDcDBARDwLlwKi8/WcDvypiGXulqqx9sN1jJGZmHRUzSB4BJkraV1IZSSjc0uGYpcDJAJIOJgmSlel6CXAmeeMjA6UiHSNp8BiJmdk2ihYkEdECXATcDjxFcnfWk5K+Jum09LBLgI9Kepyk5fGheOM9JCcAL0fEkvzrSrpM0jKgQtIySV8pVh3alWZKKMuWUO83AJuZbaOoYyQRMRuY3WHbl/OWFwHHdXFuHXB0J9s/C3y2oAXthapclgZ3bZmZbcNPtvdShedtNzPrlIOkl6pyWT/ZbmbWCQdJL1WUZfxku5lZJxwkvVTpFomZWaccJL1UWZalwXdtmZltw0HSS55u18yscw6SXqrKZdy1ZWbWCQdJL1Xk3LVlZtYZB0kvVeWyNLcGjS3u3jIzy+cg6aWKsvb3bTlIzMzyOUh6ydPtmpl1zkHSS1VpkGz0OImZ2VYcJL3U3rXlW4DNzLbmIOmlLS0Sd22ZmW3FQdJLFeksib4F2Mxsaw6SXqrKebpdM7POOEh6qTLXPkbiFomZWT4HSS9V+q4tM7NOOUh6KZctIVMit0jMzDpwkPSSpHS6XY+RmJnlc5Bsh6pc1i0SM7MOHCTboTKX9RiJmVkHDpLtUOmuLTOzbThItkOlu7bMzLZR1CCRNFPSM5IWS7q0k/17S5oj6TFJCyS9M91+jqT5eV9tkian+6ZIWphe8weSVMw65Ksoy7KxyS0SM7N8RQsSSRngSuBUYBIwS9KkDod9Ebg5Io4EzgauAoiIGyJickRMBs4FXoiI+ek5PwY+CkxMv2YWqw4dVeUybpGYmXVQzBbJUcDiiFgSEU3ATcDpHY4JYHi6PAJY3sl1ZqXnImkMMDwi/hoRAfwSeE8xCt8Zd22ZmW0rW8RrjwNezltfBkzvcMxXgDskfQqoBE7p5Dpn8UYAjUuvk3/NcZ19uKQLgQsBamtrqaur277Sp+rr67ecu/pvTWzY1Nzna+1K8us9lLjeQ4vrXRjFDJLemAVcGxGXSzoGuE7SoRHRBiBpOtAQEU9s74Uj4mrgaoCpU6fGjBkz+lTAuro62s99vOU5/vTisxz/lhPIZgb3fQr59R5KXO+hxfUujGL+NnwF2CtvfXy6Ld8FwM0AEfEgUA6Mytt/NvCrDtcc38M1i6b9xY0NzR5wNzNrV8wgeQSYKGlfSWUkoXBLh2OWAicDSDqYJEhWpuslwJmk4yMAEbECWC/p6PRurfOAPxSxDlup9ORWZmbb6FWQSLquN9vyRUQLcBFwO/AUyd1ZT0r6mqTT0sMuAT4q6XGSlseH0kF0gBOAlyNiSYdLfwL4GbAYeB74v97UoRAcJGZm2+rtGMkh+Svprb1TejopImYDszts+3Le8iLguC7OrQOO7mT7XODQ3hS60Co9b7uZ2Ta6bZFI+rykDcDhktanXxuA19iBXUo7C7dIzMy21W2QRMR/REQ18O2IGJ5+VUfE7hHx+R1Uxp1GZVn75FZukZiZtevtYPsfJVUCSPp7SVdI2qeI5dopebpdM7Nt9TZIfgw0SDqCZID8eZKnyoeUqrRrq95BYma2RW+DpCW9m+p04EcRcSVQXbxi7Zwq0iBp8JwkZmZb9PaurQ2SPk/yAsW3pM94lBavWDunitKka6ved22ZmW3R2xbJWUAj8A8R8SrJE+XfLlqpdlIlJcm87Q3u2jIz26JXQZKGxw3ACEnvBjZHxJAbIwFPt2tm1lFvn2w/E3gY+ADJa0seknRGMQu2s/J0u2ZmW+vtGMkXgGkR8RqApNHAn4HfFKtgOyvPSWJmtrXejpGUtIdIavV2nDuoVOayvv3XzCxPb1skf5J0O2+80v0sOrxDa6ioLMuwqr5poIthZrbT6DZIJO0P1EbE/5P0PuD4dNeDJIPvQ05lLstLqxsGuhhmZjuNnlok3wM+DxARvwN+ByDpsHTf3xW1dDuhyjLftWVmlq+ncY7aiFjYcWO6bUJRSrSTSwbbfdeWmVm7noKkppt9wwpZkF1FVS7DxqYW3ph/y8xsaOspSOZK+mjHjZI+AswrTpF2bhW5LBGwyfO2m5kBPY+RXAz8XtI5vBEcU4Ey4L3FLNjOqjLvDcAVZb296c3MbPDq9jdhRPwNOFbSSbwxve1tEXF30Uu2k2qfbrehsXUIvv/YzGxbvfondUTMAeYUuSy7hErPSWJmtpUh+XR6f1R53nYzs604SLZTRXvXludtNzMDHCTbzdPtmpltzUGynTzdrpnZ1ooaJJJmSnpG0mJJl3ayf29JcyQ9JmmBpHfm7Ttc0oOSnpS0UFJ5uv2s9NgnJX2rmOXvTFVZe4vEXVtmZlDEIJGUAa4ETgUmAbMkTepw2BeBmyPiSOBs4Kr03CxwPfCxiDgEmAE0S9qdZIrfk9Pte0o6uVh16ExFLhkj8WC7mVmimC2So4DFEbEkIpqAm4DTOxwTwPB0eQSwPF1+O7AgIh4HiIjVEdEK7Ac8FxEr0+P+DLy/iHXYRmmmhLJsiV/caGaWKmaQjANezltflm7L9xXg7yUtI5nf5FPp9gOAkHS7pEclfTbdvhg4UNKEtNXyHmCvYlWgK1WeJdHMbIuBfsfHLODaiLhc0jHAdZIOTct1PDANaADukjQvIu6S9HHg10Ab8ADwps4uLOlC4EKA2tpa6urq+lTA+vr6bc4taWtmydLl1NWt7tM1dwWd1XsocL2HFte7MIoZJK+wdWthfLot3wXATICIeDAdUB9F0nq5JyJWAUiaDbwZuCsibgVuTbdfCHQ66h0RVwNXA0ydOjVmzJjRp0rU1dXR8dxR8++hqqaCGTOm9umau4LO6j0UuN5Di+tdGMXs2noEmChpX0llJIPpt3Q4ZilwMoCkg4FyYCVwO3CYpIq0C+tEYFF63B7p95HAJ4CfFbEOnarMeXIrM7N2RWuRRESLpItIQiEDXBMRT0r6GjA3Im4BLgF+KumfSQbePxTJRB9rJF1BEkYBzI6I29JLf1/SEeny1yLi2WLVoSsVZRk2bHaQmJlBkcdIImI2ySB6/rYv5y0vAo7r4tzrSW4B7rh9VoGLud2qclleXbd5oIthZrZT8JPtfVCVy7J+c/NAF8PMbKfgIOmDPUeUs3JDIy2tbQNdFDOzAecg6YOxNcNoC/jbhsaBLoqZ2YBzkPTBmBHlAKxYu2mAS2JmNvAcJH0wrmYYAK84SMzMHCR9MSYNkuVrfeeWmZmDpA+qclmGl2dZsc4tEjMzB0kfja0ZxnJ3bZmZOUj6alzNMHdtmZnhIOmzMTXlLHfXlpmZg6SvxtYMY21Ds+duN7Mhz0HSR+N855aZGeAg6bMxI9qDxN1bZja0OUj6aGxN8nS7g8TMhjoHSR/VDi9HguV+nbyZDXEOkj4qzZRQW13uFomZDXkOkn4YW+MgMTNzkPTDmJphrHDXlpkNcQ6SfhhXM4xX1m4imWbezGxocpD0w9gR5TS1tLF6Y9NAF8XMbMA4SPqh/XXyK/xQopkNYQ6SfvAEV2ZmDpJ+GVvjp9vNzBwk/TCyopRctsQTXJnZkOYg6QdJnpfEzIa8ogaJpJmSnpG0WNKlnezfW9IcSY9JWiDpnXn7Dpf0oKQnJS2UVJ5un5WuL5D0J0mjilmHnoxNbwE2MxuqihYkkjLAlcCpwCRglqRJHQ77InBzRBwJnA1clZ6bBa4HPhYRhwAzgOZ0+/eBkyLicGABcFGx6tAbY0aUu2vLzIa0YrZIjgIWR8SSiGgCbgJO73BMAMPT5RHA8nT57cCCiHgcICJWR0QroPSrUpLSc5czgMbWDOO1DY00tbQNZDG61djS6ocmzaxoVKxfMJLOAGZGxEfS9XOB6RFxUd4xY4A7gJFAJXBKRMyTdDEwBdgDGA3cFBGX5V33GmAj8BxJ66S1k8+/ELgQoLa2dspNN93Up3rU19dTVVXV5f57ljVzzRNNfPuEYYyu2LmGnF5a38r/vdDMw6+2UpMTB44s4cDdMhy0W4baCpFkced6qvdg5XoPLa5390466aR5ETG1p+OyBSlV380Cro2IyyUdA1wn6dC0XMcD04AG4C5J84B7gI8DRwJLgB8Cnwe+0fHCEXE1cDXA1KlTY8aMGX0qYF1dHd2dm3luJdc88TB7H3QE0/fbvU+fUUgRwb3PreK/7nme+xevpiqX5Zzp+/B6QxMPLVnNgyuSp/BHV+eYvu9unH/sBKZN2G2b6/RU78HK9R5aXO/CKGaQvALslbc+Pt2W7wJgJkBEPJgOqI8ClgH3RMQqAEmzgTcD69Njn0+33wxsM4i/I215lmSAx0na2oJbHl/Of92zhKdWrGeP6hyXnnoQH5y+N8PLS4EkZJ5fuZGHXljNQ0te54HnVzF74QouefuBfPzEN1FS0nULxcysK8UMkkeAiZL2JQmQs4EPdjhmKXAycK2kg4FyYCVwO/BZSRVAE3Ai8N30OpMkjY6IlcDbgKeKWIcejR2xc8zd/u+zn+Jn973AxD2quOyMwzl98lhy2cxWx0hi/z2q2H+PKs6Zvg8bNjdz6e8W8u3bn2HeS2u44swjqKkoG6AamNmuqmhBEhEtki4iCYUMcE1EPCnpa8DciLgFuAT4qaR/Jhl4/1AkgzZrJF1BEkYBzI6I2wAkfRW4R1Iz8BLwoWLVoTeGlWUYWVE6oE+3/+7RZfzsvhc475h9+MrfHdLrlkV1eSk/mnUk0/fdja//cRHv+sF9XHnOm5m8V02RS2xmg0lRx0giYjYwu8O2L+ctLwKO6+Lc60luAe64/SfATwpb0v4ZWzNswIJkwbK1XPq7hRy932586d2Ttrt7ShLnHTOBI8bX8IkbHuUDP3mAL7zzYPbxXV5m1ks7121Gu6ixAzTB1coNjfzjdfMYXZXjyg++mdJM3/86j9irhtv+6XhOmDiar9y6iJ8tbKK1zWFiZj1zkBTA2BHlO/zp9qaWNj5xwzzWNDRx9XlT2L0q1+9r1lSU8dPzpnLxKRO5f3kL//q7hbQ5TMysBw6SAhhbM4wNm1vYsLl5h33mV299kkdeXMNlZxzBIWNHFOy6JSXi4lMO4LQ3lfLruS/z9dsW+WFGM+vWQD9HMihsmeBq3Waq01tti+nGh5Zyw0NL+ccT9+O0I8YW5TPeu38po/YczzX3v0BVLsslbz+wKJ9jZrs+B0kBjKspB5IJrg6orS7qZ817aQ3/dssTnHjAaD77joOK9jmS+NK7D6ahqYUf3r2YirIsH5/xpqJ9npntuhwkBbCjJrhqawu++L9PsEd1OT84+0gyRX6AUBLffO9hNDS18q0/PU1lLsN5x0wo6mea2a7HQVIAe1SXkylR0eduv+Xx5Ty1Yj3fP3syIyqK34UGkCkRl595BA1NrXz5D09SUZbljCnjd8hnm9muwYPtBZApEXsOLy9qi6SxpZXv3PEMh4wdzt8dXpxxka6UZkr40QeP5Pj9R/G53y5gztOv7dDPN7Odm4OkQMbWFPcW4BsfWsqyNZv43MyDBuSdWOWlGf7r3CkcPKaaT974KAuXrdvhZTCznZODpEDGjCjeQ4kbNjfzw7sXc+ybductEwduQsjKXJZrzp/GyIoy/uG/H+Hl1xsGrCxmtvNwkBRI8nT7pqI8wPfTe1/g9Y1NfG7mQd3OIbIj7DG8nGs/PI3G5lY+fO0jrGvYcc/OmNnOyUFSIONqymluDVbVNxb0uis3NPKze5fwrsPGcMRO8jLFibXVXH3eVJaubuCj182lsWWbecXMbAhxkBTImPbXyRe4e+uHdz9HY0sbn3nHzvVA4NH77c63P3A4D7/wOp/5nwV+lYrZEOYgKZBiPEvy4qqN3PjQUs6ethf7jqos2HUL5fTJ4/jszAO59fHlXHb7MwNdHDMbIH6OpEDGFSFILr/zWUozJXz65IkFu2ahffzEN/HKmk385C/PM7o6xwXH7zvQRTKzHcxBUiDDh2WpKMsUbKbEhcvWcevjy7nopP3ZY3h5Qa5ZDJL46mmHsLq+ia//cRHV5VnOnLpXzyea2aDhrq0CkVTQCa4uu/1pRlaUcuGJ+xXkesWUzZTw/VmTecvEUVz62wX86YkVA10kM9uBHCQF1H4LcH/9dclq7n1uFZ+YsT/Dd8DbhAshl00eWDxy75H806/mc+9zKwe6SGa2gzhICiiZ4Kp/XVsRwRV3PMse1TnOPWafApVsx6goSx5Y3G90JRf+ch7zXloz0EUysx3AQVJAY2uGsaq+kc3NfX+u4t7nVvHwi6/zqbfuT3lppoCl2zFGVJRy3QXTqR2e48O/eJinVqwf6CKZWZE5SApo0pjhAH1+qWFE8J07nmFczTDOnLbrDliPrs5x/UemU5nLcu7PH2bxa/UDXSQzKyIHSQGddNAe7LXbMH5+3wt9Ov/ORX9jwbJ1fPrkieSyu15rJN/4kRVc/5HpQPCBnzzAo0vdzWU2WDlICihTIj587L7MfWkN819eu13ntrUFV9z5LPuOquR9bx5XpBLuWG8aXcVvP34sI4aV8sGf/pU7F/1toItkZkXgICmwM6ftRXUuu92tktsWruDpVzdw8SkTyWYGz1/LPrtX8puPH8uBtdX843VzufGhpQNdJDMrsKL+xpI0U9IzkhZLurST/XtLmiPpMUkLJL0zb9/hkh6U9KSkhZLKJVVLmp/3tUrS94pZh+1Vlcty9lF7MXvhil7PT9LS2sZ3//wsB9ZW7/BJq3aEUVU5fnXh0Zx4wGj+9fcLueLOZ4nwu7nMBouiBYmkDHAlcCowCZglaVKHw74I3BwRRwJnA1el52aB64GPRcQhwAygOSI2RMTk9i/gJeB3xapDX51/7AQAfvnAi706/vePvcKSlRv557cdMCCTVu0IFWVZrj5vKmdOHc8P7nqOS3+7kJbWtoEulpkVQDFbJEcBiyNiSUQ0ATcBp3c4JoDh6fIIYHm6/HZgQUQ8DhARqyNiq3tqJR0A7AHcW6Ty99n4kRXMPHRPbnx4KRsbW7o9tqmlje/f9RyHjhvOOw6p3UElHBilmRK+9f7D+ae37s+v577MB3/6EC+t3jjQxTKzflKxuhgknQHMjIiPpOvnAtMj4qK8Y8YAdwAjgUrglIiYJ+liYApJUIwGboqIyzpc/8vA8Ij4TBeffyFwIUBtbe2Um266qU/1qK+vp6qqarvPe35tK1//62bOObiMt+3T9dPpdy9t5peLmviXKTkOH73zvPqsr/Xurftfaeb6p5poDfjAAWWcvHeWkgGetAuKX++dles9tPS23ieddNK8iJja44ERUZQv4AzgZ3nr5wI/6nDMvwCXpMvHAItIWkmfAV4ARgEVwIPAyR3OXQRM6U1ZpkyZEn01Z86cPp/7vqvuj7d86+5oaW3rdP+zr66PKV+/M95/1f3R1tb5MQOlP/XureVrG+K8nz8U+3zuj3HmTx6IF1fVF/0ze7Ij6r0zcr2Hlt7WG5gbvfgdW8yurVeA/Kfqxqfb8l0A3AwQEQ8C5Wl4LAPuiYhVEdHE+P79AAANoklEQVQAzAbe3H6SpCOAbETMK17x+++C4/dl6esNnd72ev/iVbzvxw8A8PX3HDrgU+gOhDEjhnHth6dx2RmHs2j5emZ+716uvf8FT5JltospZpA8AkyUtK+kMpLB9Fs6HLMUOBlA0sEkQbISuB04TFJFOvB+IkkLpN0s4FdFLHtBvH1SLeNHDuOaDrcC//qRpZx/zcOMHTGM//3ksRw8ZngXVxj8JHHm1L24419O4Kh9d+Mrty7ivT9+gD8uWO7BeLNdRNGCJCJagItIQuEpkruznpT0NUmnpYddAnxU0uMkwfChtEW1BriCJIzmA49GxG15lz+TXSBIspkSPnTsBB5+8XUWLFtLW1vwrT89zed+u5Bj3rQ7//PxYxg/smKgi7lTaG+dfOcDR7BmYxMX3fgYJ1w2hx/XPc/ahqaBLp6ZdaOoo7sRMZukWyp/25fzlhcBx3Vx7vUktwB3tm/nn6Qjdda0vfjen5/jx3XPUyJx28IVfHD63nz1tEMoHUQPHhaCJM6YMp73HjmOu59+jV/c/wLf+tPTfP+uZ3nfm8dz7tH7cNCe1QPaDdjc2sbrG5tYuaGRlfWNrK5vYv2mZjZsbqG+Mfm+obGF+s0tNLW00doWtEbQ0ha0tSXfS5TcwVaWLSGXLaEsXS4vzTC8PEt1eSnDh2UZXl5KdXkpI4aVMrKylFFVOUZWlFGW9c+N7Vx2ntuEBqnq8lLOmrYXP7/vBST413cexEffst+QHBPprUyJeNukWt42qZanVqzn2vtf5DfzlnHjQ0vZrbKMN+89kin7jGTqhJEcNm5EQd6S3NYWrN3UzMsb2pjzzGu8um4zK9Zt5tV1m1ixbjN/W7+ZVfVNvL6x69bRsNIM1eVZqsqzVOey5LIZSkqgrCRDSYnIlogSiYigqbWNppY26huTwGlqaWNTc2sSRJub6W6YqLo8y6iqHLtVljG6Ksfo6hx7VKffh+cYXVXOHsNz7F5ZNqjekmA7LwfJDnDB8fuyYNlaLjh+X2YeOmagi7NLOXjMcL51xuF87tSDuHPRq8x9cQ3zXlrDn59KbmAoy5Rw8Njh1FbnqKkopaaijBHDSrd8tUWwubmVzc3JL+rNza1sam5lXUPzllbFa+sbWVXfSEv7b+/7HwFAgj2qc+w5vJwJu1cybcJujK7OMaoq+UqWk8+rymUL9ks7ItjY1LqlpbO2oYk1DU1bgmx1fSOrNzaxur6J51fW8+CS1azb1LzNdSTYvbKM0dXlW4VNUv6yLfUYVVVGm980YP3gINkBxtYM438+duxAF2OXtltlGWdN25uzpu0NwOr6Rh5dupa5L73OwmXrWPp6AwuWNbN2UxObm7sfpC/LllAzrJTR6S/WA2urtyyvXPo8Jx87hTEjkl++A9H9KImqXJaqXO//92xsaWVVfROvrd/Maxsak5Dc0Jgub2blhkae+9sGVtU30ty6bWgI2O2+OxlZWcbIilJGVpSxW2XZlmAePiybfC8vZXga0pW5DJVlWSrKMm5hD3EOEtsl7V6V29L91dHm5lbWbWpm3aZmMiWivDTDsNIM5aUllGcz3b6Gpq75JabsM7KYRS+KXDbDuJphjKsZ1u1xEcG6Tc2sqm9k5YYmVtUnrbFHn3yO4aP3ZE1D0up5aXUD819ey5qGpk6DJ58EFaUZKnNZKnPZ9M87GfMpz/tzz5VmKMuIsmwJpZmSLeNEZZkSMiWiNCOyecuZkhIyEpkSkuUSKJHIlIiMhNLlEiXhWyLSdSElx5Zo6/35+yRYvamNFes2Jeu8cVxX39sfmt3qOumfwVAOUweJDTrtv8Bqh5cPdFF2OpKoqUhaGvvv8cb2fZtfYsaMw7Y5PiLY3NzG+s1JMK/f1LxleWNjKxsbW5KvpmS5vrFlS1fi5uZW1m9uTroVm1ppbGmjuTX5amppe6MrcaD95e6CXSo/XEokSP7rEDrJ9zf2vRFW7cdCElRbQmqrdW35LLrYn3/92Z9+S9HnN3KQmFmXJDGsLMOwssIHc1tbctNBc2tyd1tza6Tfk/WWtjZa26C1LWhL73xrX25L74aLIFmP5HpbltMnrtuXW9PQSs5NXvLXFsFTTz3NAQceuOU6QRKebW2RHpOup5/V/pnt5W+/TqTHJee/ca22CIj089re2B+8cZ2IDtdJj4H2675xfP75tJ/P1p+dv06wQ1495CAxswFRUiLKSzIFueuur+rqn2fGUXsP2OcPFr430MzM+sVBYmZm/eIgMTOzfnGQmJlZvzhIzMysXxwkZmbWLw4SMzPrFweJmZn1i2IIvPVT0krgpT6ePgpYVcDi7Cpc76HF9R5aelvvfSJidE8HDYkg6Q9JcyNi6kCXY0dzvYcW13toKXS93bVlZmb94iAxM7N+cZD07OqBLsAAcb2HFtd7aClovT1GYmZm/eIWiZmZ9YuDxMzM+sVB0gVJMyU9I2mxpEsHujzFJOkaSa9JeiJv226S7pT0XPp915vIvAeS9pI0R9IiSU9K+nS6fVDXXVK5pIclPZ7W+6vp9n0lPZT+zP9aUtlAl7UYJGUkPSbpj+n6oK+3pBclLZQ0X9LcdFvBfs4dJJ2QlAGuBE4FJgGzJE0a2FIV1bXAzA7bLgXuioiJwF3p+mDTAlwSEZOAo4FPpn/Pg73ujcBbI+IIYDIwU9LRwLeA70bE/sAa4IIBLGMxfRp4Km99qNT7pIiYnPf8SMF+zh0knTsKWBwRSyKiCbgJOH2Ay1Q0EXEP8HqHzacD/50u/zfwnh1aqB0gIlZExKPp8gaSXy7jGOR1j0R9ulqafgXwVuA36fZBV28ASeOBdwE/S9fFEKh3Fwr2c+4g6dw44OW89WXptqGkNiJWpMuvArUDWZhikzQBOBJ4iCFQ97R7Zz7wGnAn8DywNiJa0kMG68/894DPAm3p+u4MjXoHcIekeZIuTLcV7Oc829/S2eAXESFp0N4nLqkK+C1wcUSsT/6RmhisdY+IVmCypBrg98BBA1ykopP0buC1iJgnacZAl2cHOz4iXpG0B3CnpKfzd/b359wtks69AuyVtz4+3TaU/E3SGID0+2sDXJ6ikFRKEiI3RMTv0s1Dou4AEbEWmAMcA9RIav/H5WD8mT8OOE3SiyTd1W8Fvs/grzcR8Ur6/TWSfzgcRQF/zh0knXsEmJjezVEGnA3cMsBl2tFuAc5Pl88H/jCAZSmKtH/858BTEXFF3q5BXXdJo9OWCJKGAW8jGR+aA5yRHjbo6h0Rn4+I8RExgeT/6bsj4hwGeb0lVUqqbl8G3g48QQF/zv1kexckvZOkPzUDXBMR3xzgIhWNpF8BM0heLf034N+A/wVuBvYmeQX/mRHRcUB+lybpeOBeYCFv9Jn/K8k4yaCtu6TDSQZXMyT/mLw5Ir4maT+Sf6nvBjwG/H1ENA5cSYsn7dr6TES8e7DXO63f79PVLHBjRHxT0u4U6OfcQWJmZv3iri0zM+sXB4mZmfWLg8TMzPrFQWJmZv3iIDEzs35xkNigIGn39M2m8yW9KumVvPVevc1V0i8kHdjDMZ+UdE6BynyfpMmSSgr9hmlJ/yBpz7z1Hutm1le+/dcGHUlfAeoj4jsdtovkZ76t0xN3MEn3AReRPBy2KiJqtvP8TPqqky6vHRHz+19Ss+65RWKDmqT90/lGbgCeBMZIulrS3HQuji/nHdveQshKWivpP9M5Ox5M31GEpG9Iujjv+P9M5/Z4RtKx6fZKSb9NP/c36WdN7qaY/wlUp62nX6bXOD+97nxJV6WtlvZyfU/SAuAoSV+V9IikJyT9RImzSF4P/+v2Fll73dJr/72SuSmekPTv6bbu6nx2euzjkuYU+K/IBgEHiQ0FB5HMNzEpfefQpemcDEcAb1Pnc82MAP6SztnxIPAPXVxbEXEU8P+A9lD6FPBqOs/J10neKtydS4EN6VwR50k6FHgvcGxETCZ5GvnsvHLdExGHR8SDwPcjYhpwWLpvZkT8GpgPnJVes2lLYZPXqH8DOCkt13FKXmbYXZ3/DTg53f7eHupiQ5CDxIaC5yNibt76LEmPAo8CB5NMXtbRpoj4v3R5HjChi2v/rpNjjid55QYR8ThJS2h7nAJMA+YqedX7icCb0n1NvPG6C4CTJT0MPJ4ed0gP155O8o6pVRHRDNwInJDu66rO9wO/lPQR/DvDOuHXyNtQsLF9QdJEkhnyjoqItZKuB8o7Oacpb7mVrv9faezFMdtLJO93+9JWG5M31G6KdGBTUgXwI+DN6SvCv0Hndemtrur8UZIAejfwqKQjI2JNPz7HBhn/68KGmuHABmB9+ursdxThM+4HzgSQdBidt3i2aJ9USW+8yvzPwJmSRqXbd5e0dyenDiN52eSq9O2u78/btwGo7uSch4CT0mu2d5n9pYf67BcRfwW+RDIV7WCc+Mn6wS0SG2oeBRYBT5O88fT+InzGD0m6ghaln7UIWNfDOT8HFkiam46TfBX4s6QSoBn4GLA8/4SIWC3pv9PrryAJiXa/AH4maRPJ3BPt5yyT9CWgjqTlc2tE3JYXYp35rqR90+PviIgneqiLDTG+/deswNJfytmI2Jx2pd0BTMybztVsUHGLxKzwqoC70kAR8I8OERvM3CIxM7N+8WC7mZn1i4PEzMz6xUFiZmb94iAxM7N+cZCYmVm//H8i11ImQqh1TgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.title('AMSGrad')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Nadam - Adding NAG to ADAM\n",
    "\n",
    "\n",
    "Refer the section Nadam in the book for the detailed explanation. The update equation of Nadam is given as:\n",
    "\n",
    "\n",
    "First moment estimate:\n",
    "$$m_{t}=\\beta_{1} m_{t-1}+\\left(1-\\beta_{1}\\right) g_{t} -- (33) $$\n",
    "\n",
    "Second moment estimate:\n",
    "$$v_{t}=\\beta_{2} v_{t-1}+\\left(1-\\beta_{2}\\right) g_{t}^{2} -- (34) $$\n",
    "\n",
    "Bias-corrected mt:\n",
    "$$\\hat{m}_{t}=\\frac{m_{t}}{1-\\prod_{i=1}^{t+1} \\beta_{1}^{i}} -- (35) $$\n",
    "\n",
    "Bias-corrected gt:\n",
    "$$\\hat{g}_{t}=\\frac{g_{t}}{1-\\prod_{i=1}^{t} \\beta_{1}^{i}} -- (36) $$\n",
    "\n",
    "Bias-corrected vt:\n",
    "$$ \\hat{v}_{t}=\\frac{v_{t}}{1-\\beta_{2}^{t}} -- (37) $$\n",
    "\n",
    "Compute m tilde:\n",
    "$$ \\tilde{m}_{t}=\\beta_{1}^{t+1} \\hat{m}_{t}+\\left(1-\\beta_{1}^{t}\\right) \\hat{g}_{t} -- (38) $$\n",
    "\n",
    "Update theta:\n",
    "$$ \\theta_{t}=\\theta_{t-1}-\\frac{\\eta}{\\sqrt{v_{t}}+\\epsilon} \\tilde{m}_{t} -- (39) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def Nadam(data, theta, lr = 1e-2, beta1 = 0.9, beta2 = 0.999, epsilon = 1e-6, num_iterations = 50):\n",
    "   \n",
    "    loss = []\n",
    "    \n",
    "    #initialize first moment mt\n",
    "    mt = np.zeros(theta.shape[0])\n",
    "    \n",
    "    #initialize second moment vt\n",
    "    vt = np.zeros(theta.shape[0])\n",
    "    beta_prod = 1\n",
    "    \n",
    "    for t in range(num_iterations):\n",
    "        \n",
    "        #compute gradients with respect to theta\n",
    "        gradients = compute_gradients(data, theta)\n",
    "        \n",
    "        #compute first moment as given in equation (33)\n",
    "        mt = beta1 * mt + (1. - beta1) * gradients\n",
    "        \n",
    "        #compute second moment as given in equation (34)\n",
    "        vt = beta2 * vt + (1. - beta2) * gradients ** 2\n",
    "        beta_prod = beta_prod * (beta1)\n",
    "        \n",
    "        #compute bias-corrected estimates of mt as shown in (35)\n",
    "        mt_hat = mt / (1. - beta_prod)\n",
    "        \n",
    "        #compute bias-corrected estimate of gt as shown in (36)\n",
    "        g_hat = gradients / (1. - beta_prod)\n",
    "        \n",
    "        #compute bias-corrected estimate of vt as shown in (37)\n",
    "        vt_hat = vt / (1. - beta2 ** (t))\n",
    "        \n",
    "        #compute mt tilde as shown in (38)\n",
    "        mt_tilde = (1-beta1**t+1) * mt_hat + ((beta1**t)* g_hat)\n",
    "        \n",
    "        #update theta as given in (39)\n",
    "        theta = theta - (lr / (np.sqrt(vt_hat) + epsilon)) * mt_hat\n",
    "        \n",
    "        #store the loss\n",
    "        loss.append(loss_function(data,theta))\n",
    "\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update model parameter using AMSGrad and compute loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "loss = Nadam(data, theta) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VXed//HX+94khOyBQAgECC20ZaclLaW0CrZVrFp0rLVY0dqFmd+IM9rOUrVW23EcZ1GrY6tTHa3WKqLjgi1KbSVdUQtl39pA2fedsGS7n98f5wSuaUhCkstN7v08H4/7yNnv9ws3953v+Z7zPTIznHPOuY6KJLsAzjnnejYPEuecc53iQeKcc65TPEicc851igeJc865TvEgcc451ykeJM4liaSpkrYnuxzOdZYHiXPnQNJmSXsl5cYtu1NSVRKL5VxSeZA4d+6iwN8nuxDOdRceJM6du/8E/kFSUfMVkr4uaZuko5KWSrombl1vSY9JOiRpLXB5s33vlbRR0jFJayW9L27dbZJekvQ1SYclbZJ0Vbh8W9hK+mgiK+3c2XiQOHfulgBVwD+0sO4VYALQB/gx8DNJ2eG6zwMXhq93AM2/+DcC1wCFwAPAjySVxa2fBKwE+obHnksQRsOBDwPflJTXybo5d848SJzrmPuBT0jqF7/QzH5kZgfMrMHMvgL0Ai4OV98M/KuZHTSzbcA3mu37MzPbaWYxM/sp8DpwRdwmb5jZ982sEfgpMBh40MxqzexpoI4gVJw7rzxInOsAM1sNPAncG79c0j9IWifpiKTDBK2LknD1QGBb3OZbmu37EUnLw1NXh4ExcfsC7ImbPhmWo/kyb5G4886DxLmO+zxwFzAIIOwP+SeClkexmRUBRwCF2+8iaEU0GdI0IWko8B1gDtA33Hd13L7OdVseJM51kJlVE5xi+rtwUT7QAOwDMiTdDxTE7TIP+LSkYknlwCfi1uUCFu6LpI8RtEic6/Y8SJzrnAcJQgBgIfA74DWC01an+MtTWQ+Ey98AngYeb1phZmuBrwCLCU5hjQVeSnDZnesS8gdbOeec6wxvkTjnnOsUDxLnnHOd4kHinHOuUxIaJJKmS9ogqVrSvS2sHyJpkaRlklZKuiFcninpB5JWhdfkfzpcPjjcfq2kNZJ8vCPnnEuyhHW2S4oSXL1yPbCdYOiImeHVKU3bPAosM7NvSRoFLDCzCkkfAm40s1sk5QBrgalALVBmZq9KygeWAu+NP2ZLSkpKrKKiokP1OH78OLm5uW1vmGK83unF651e2lvvpUuX7jezfm1tl9ElpWrZFUC1mW0CkDQXmEEQCk2MM9fZFwI745bnSsoAehMM/XDUzA4S3NSFmR2TtI7gZrBWg6SiooIlS5Z0qBJVVVVMnTq1Q/v2ZF7v9OL1Ti/trbekLW1uRGJbJDcB083sznB+FjDJzObEbVNGcD19McG1+NeZ2VJJmQTX2F8L5ACfMrNHmx2/AngeGGNmR1t4/9nAbIDS0tKJc+fO7VA9ampqyMtLv1EnvN7pxeudXtpb72nTpi01s8q2tktki6Q9ZgKPmdlXJE0GHpc0hqA100gwNlEx8IKkZ+JaN3nA/wGfbClEAMLgeRSgsrLSOvpXh//Fkl683unF6901EtnZvoO/HFeoPFwW7w6CYSMws8VANsEgdR8Cfmdm9Wa2l+AO30oIOuIJQuQJM/tFAsvvnHOuHRIZJK8AIyQNk5QF3ALMb7bNVoLTV0gaSRAk+8LlbwuX5wJXAuslCfhfYJ2ZfTWBZXfOOddOCQsSM2sgGMl0IbAOmGdmayQ9KOnGcLN7gLskrQB+AtxmQafNw0CepDUEgfR9M1sJTAFmAW8Lh9te3nTJsHPOueRIaB+JmS0AFjRbdn/c9FqCcGi+Xw3wgRaWv4gPq+2cc92K39nunHOuUzxIWvHDxZv5066GZBfDOee6NQ+SVsxbso3nttcnuxjOOdeteZC0YuygQrYcjeHPbHHOubPzIGnFmEGFHK+HbQdPJrsozjnXbXmQtGLsoEIAVu04kuSSOOdc9+VB0oqLB+QTlQeJc861xoOkFb0yopTnR1jtQeKcc2flQdKGoQURVu044h3uzjl3Fh4kbagoiHDkZD3bD3mHu3POtcSDpA0VhcE/kfeTOOdcyzxI2lCeFyEjIg8S55w7Cw+SNmRFxUWl+d7h7pxzZ+FB0g5jBxV6h7tzzp2FB0k7jCkv5PAJ73B3zrmWeJC0Q9Md7n56yznn3syDpB0uGZDvHe7OOXcWHiTtkJ0ZZURpvgeJc861IKFBImm6pA2SqiXd28L6IZIWSVomaWXT89clZUr6gaRVktZJ+nR7j5koYwcVsNo73J1z7k0SFiSSosDDwDuBUcBMSaOabXYfMM/MLgVuAR4Jl38A6GVmY4GJwF9LqmjnMRNi7KBCDp2oZ+eRU+fj7ZxzrsdIZIvkCqDazDaZWR0wF5jRbBsDCsLpQmBn3PJcSRlAb6AOONrOYybEmKYh5bf76S3nnIuXkcBjDwK2xc1vByY12+YLwNOSPgHkAteFy39OEBC7gBzgU2Z2UFJ7jgmApNnAbIDS0lKqqqo6VImamhqqqqqoazQigqdeXkn2/vUdOlZP0lTvdOP1Ti9e766RyCBpj5nAY2b2FUmTgccljSFoeTQCA4Fi4AVJz5zLgc3sUeBRgMrKSps6dWqHClhVVUXTvhetep6jGdlMnXpFh47Vk8TXO514vdOL17trJPLU1g5gcNx8ebgs3h3APAAzWwxkAyXAh4DfmVm9me0FXgIq23nMhBk7qNA73J1zrplEBskrwAhJwyRlEXSmz2+2zVbgWgBJIwmCZF+4/G3h8lzgSmB9O4+ZMGPLCzlwvI5d3uHunHOnJSxIzKwBmAMsBNYRXJ21RtKDkm4MN7sHuEvSCuAnwG0W/Ln/MJAnaQ1BeHzfzFae7ZiJqkNzY/wZ7s459yYJ7SMxswXAgmbL7o+bXgtMaWG/GoJLgNt1zPNlVFkB0YhYveMI7xg9IBlFcM65bsfvbD8H2ZlRRvTP8xaJc87F8SA5R2O8w9055/6CB8k5GjuokP01dew+6h3uzjkHHiTnbMyg4EZ8v8PdOecCHiTnaFRZIRH5s0mcc66JB8k56p0VZbh3uDvn3GkeJB0wZlAhq3YcTXYxnHOuW/Ag6YAxAwvZX1PLXu9wd845D5KOGDUw6HBfs8tbJc4550HSAU1BsnanB4lzznmQdEBBdiZD+uSwZqd3uDvnnAdJB40eWMAab5E455wHSUeNHljAlgMnOHqqPtlFcc65pPIg6aDRA4Mh5dd5q8Q5l+Y8SDpodNOVWx4kzrk050HSQf0LsinJ6+VB4pxLex4knTBqYIFfueWcS3seJJ0wemAB1XtrqG1oTHZRnHMuaRIaJJKmS9ogqVrSvS2sHyJpkaRlklZKuiFcfquk5XGvmKQJ4bqZklaF2/9OUkki69Ca0QMLaIgZr++pSVYRnHMu6RIWJJKiwMPAO4FRwExJo5ptdh8wz8wuBW4BHgEwsyfMbIKZTQBmAW+Y2XJJGcDXgWlmNg5YCcxJVB3a0nTllp/ecs6ls0S2SK4Aqs1sk5nVAXOBGc22MaAgnC4EdrZwnJnhvgAKX7mSFO7b0j7nxdA+OeT1yvAOd+dcWlOinj0u6SZgupndGc7PAiaZ2Zy4bcqAp4FiIBe4zsyWNjvORmCGma2OO+73gOPA6wStkzd1UkiaDcwGKC0tnTh37tzmm7RLTU0NeXl5Z13/pT+dJGZw35W9O3T87qqteqcqr3d68Xq3btq0aUvNrLLNDc0sIS/gJuC7cfOzgG822+Zu4J5wejKwFojErZ8ErIqbzwSeBS4kaJl8E7ivrbJMnDjROmrRokWtrv/8r1fbyM/91hoaYx1+j+6orXqnKq93evF6tw5YYu34vk/kqa0dwOC4+fJwWbw7gHkAZrYYyAbiO89vAX4SNz8h3HZjWMl5wFVdW+xzM2pgASfqGtl84Hgyi+Gcc0mTyCB5BRghaZikLIJQmN9sm63AtQCSRhIEyb5wPgLczJn+EQiCaJSkfuH89cC6hNWgHfwOd+dcuktYkJhZA8EVVQsJvuznmdkaSQ9KujHc7B7gLkkrCFoet4UtDYC3ANvMbFPcMXcCDwDPS1pJ0EL5UqLq0B4j+ueTGZVfueWcS1sZiTy4mS0AFjRbdn/c9Fpgyln2rQKubGH5t4Fvd2lBOyErI8KI/vn+kCvnXNryO9u7wOiBBazdeZQzjSnnnEsfHiRdYPTAAg4cr2PP0dpkF8U55847D5IuMHqQ3+HunEtfHiRdYGRZAZJfueWcS08eJF0gr1cGFX1zvUXinEtLHiRdJHg2ibdInHPpx4Oki4wqK2D7oZMcOVGf7KI459x55UHSRU7f4b7LT28559KLB0kXaXo2id+Y6JxLNx4kXaRffi/65/fyIHHOpR0Pki402jvcnXNpyIOkC40eWEj1vhpO1b/pOVvOOZeyPEi60JhBBTTGjHW7vFXinEsfHiRdaFx5EQCrdviVW8659OFB0oXKCrMpyctixTYPEudc+vAg6UKSGFdexKodh5NdFOecO288SLrY2EGFVO+t4XhtQ7KL4pxz54UHSRcbV15IzHwkYOdc+khokEiaLmmDpGpJ97awfoikRZKWSVop6YZw+a2Slse9YpImhOuyJD0q6TVJ6yW9P5F1OFdjy4M73Fdu99Nbzrn0kLBntkuKAg8D1wPbgVckzQ+f097kPmCemX1L0iiC57tXmNkTwBPhccYCvzKz5eE+nwX2mtlFkiJAn0TVoSP652dTVpjNyu3e4e6cSw8JCxLgCqDazDYBSJoLzADig8SAgnC6ENjZwnFmAnPj5m8HLgEwsxiwv2uL3Xnjygv9EmDnXNqQmSXmwNJNwHQzuzOcnwVMMrM5cduUAU8DxUAucJ2ZLW12nI3ADDNbLakIWAX8DJgKbATmmNmeFt5/NjAboLS0dOLcuXObb9IuNTU15OXlndM+v9lYx/+9Xs/D1+aQm6kOvW+ydaTeqcDrnV683q2bNm3aUjOrbHNDM0vIC7gJ+G7c/Czgm822uRu4J5yeTNBaicStnwSsipsvIWjF3BS3/+NtlWXixInWUYsWLTrnfZ7bsNeG/vOT9uLr+zr8vsnWkXqnAq93evF6tw5YYu34vk9kZ/sOYHDcfHm4LN4dwDwAM1sMZIdh0eQW4Cdx8weAE8AvwvmfAZd1XZG7xrjTHe5+ess5l/oSGSSvACMkDZOURRAK85ttsxW4FkDSSIIg2RfOR4CbiesfCRPyNwSntQj3je9z6RaKcrIY0ifHr9xyzqWFhHW2m1mDpDnAQiAKfM/M1kh6kKC5NB+4B/iOpE8RnLK6LQwLgLcA2yzsrI/zz8Djkh4iCJ2PJaoOnTGuvJBlWz1InHOpL5FXbWFmCwgu6Y1fdn/c9Fpgyln2rQKubGH5FoKQ6dbGlRfy5MpdHKippW9er2QXxznnEsbvbE+QsYOCkYBX+mXAzrkU50GSIGMGFSDBKu9wd86lOA+SBMnPzuSCklzvcHfOpTwPkgQaX17klwA751KeB0kCjS0vZO+xWvYcPZXsojjnXMJ4kCRQ042JK7b56S3nXOryIEmgUWWFRCPyARydcynNgySBemdFGdE/jxXeT+KcS2EeJAk2vryIVdsPc+aGfeecSy0eJAk2tryQQyfq2X7oZLKL4pxzCeFBkmA+ErBzLtV5kCTYxQPyyYpGWLnDr9xyzqWmdgWJpMfbs8y9Wa+MKJeU5bNym7dInHOpqb0tktHxM5KiwMSuL05qGldeyOodR4jFvMPdOZd6Wg0SSZ+WdAwYJ+lo+DoG7AV+fV5KmALGDSriWG0Dmw8cT3ZRnHOuy7UaJGb2b2aWD/ynmRWEr3wz62tmnz5PZezxxg0OOtyX+x3uzrkU1N5TW09KygWQ9GFJX5U0NIHlSikj+ueTn53Bki2Hkl0U55zrcu0Nkm8BJySNJ3g87kbghwkrVYqJRsRlQ4pZsvlgsovinHNdrr1B0hA+S30G8E0zexjIb2snSdMlbZBULeneFtYPkbRI0jJJKyXdEC6/VdLyuFdM0oRm+86XtLqd5U+6yqHFvLanhiMn6pNdFOec61LtDZJjkj4NzAKekhQBMlvbIbyy62HgncAoYKakUc02uw+YZ2aXArcAjwCY2RNmNsHMJoTv+YaZLY879l8BNe0se7cwsaIYgFe3+ukt51xqaW+QfBCoBW43s91AOfCfbexzBVBtZpvMrA6YS9CiiWdAQThdCOxs4Tgzw30BkJQH3A18sZ1l7xYmDC4iGhFLtvjpLedcalF7BxOUVApcHs7+2cz2trH9TcB0M7sznJ8FTDKzOXHblAFPA8VALnCdmS1tdpyNwAwzWx3Ofw14HlgGPGlmY87y/rOB2QClpaUT586d29JmbaqpqSEvL69D+zb3wOKTZEXg05N6d8nxEqkr692TeL3Ti9e7ddOmTVtqZpVtbmhmbb6Am4EtwA8IOtnfAG5qY5+bgO/Gzc8i6F+J3+Zu4J5wejKwFojErZ8ErIqbnwDMD6crgNXtKf/EiROtoxYtWtThfZt7YP4au+izC6y2vrHLjpkoXVnvnsTrnV683q0Dllg7vmPbe2rrs8DlZvZRM/sIwWmrz7Wxzw5gcNx8ebgs3h3AvDDQFgPZQEnc+luAn8TNTwYqJW0GXgQuklTVzjokXWVFMbUNMdbs9OFSnHOpo71BErG/PJV1oB37vgKMkDRMUhZBKMxvts1W4FoASSMJgmRfOB8haAmdPidlZt8ys4FmVgFcDbxmZlPbWYekqxwadLgv9ftJnHMppL1B8jtJCyXdJuk24ClgQWs7mFkDMAdYCKwjuDprjaQHJd0YbnYPcJekFQQtj9vC5hTAW4BtZrbp3KrUffUvyGZwn9684veTOOdSSEZrKyUNB0rN7B/DS26vDlctBp5o6+BmtoBmgWNm98dNrwWmnGXfKuDKVo69GWixo707u3xoH55/fR9mhqRkF8c55zqtrRbJQ8BRADP7hZndbWZ3A78M17lzNLGimP01dWw5cCLZRXHOuS7RVpCUmtmq5gvDZRUJKVGKqxzaB8DH3XLOpYy2gqSolXXd/2aIbmhE/zwKsjNY6jcmOudSRFtBskTSXc0XSroTWNrC9q4NkYi4bGgxr2z2FolzLjW02tkOfBL4paRbORMclUAW8L5EFiyVXV7Rh6oNGzh8oo6inKxkF8c55zql1SAxsz3AVZKmceYKqafM7A8JL1kKmxh3P8m1I0uTXBrnnOuctlokAJjZImBRgsuSNsaXF5EREUs8SJxzKaC9NyS6LtQ7K8roQYX+oCvnXErwIEmSyqHFrNh+hNqGxmQXxTnnOsWDJEkuryimriHG6h1Hk10U55zrFA+SJJkY3pjo95M453o6D5Ik6Zffi6F9c/x+Eudcj+dBkkQThxbz6pZDnBnw2Dnneh4PkiSqHNqHA8freGP/8WQXxTnnOsyDJIkurwhuTPQBHJ1zPZkHSRJd2C+PPrlZ/HHjgWQXxTnnOsyDJIkiETFleAkvVO/3fhLnXI+V0CCRNF3SBknVku5tYf0QSYskLZO0UtIN4fJbJS2Pe8UkTZCUI+kpSeslrZH05USW/3y4ZkQJ+47VsmHPsWQXxTnnOiRhQSIpCjwMvBMYBcyUNKrZZvcRPMv9UuAW4BEAM3vCzCaY2QRgFvCGmS0P9/kvM7sEuBSYIumdiarD+XDNiBIAXnhtf5JL4pxzHZPIFskVQLWZbTKzOmAuMKPZNgYUhNOFwM4WjjMz3BczOxEOIEl4zFeB8gSU/bwpK+zN8P55PP/6vmQXxTnnOiSRQTII2BY3vz1cFu8LwIclbQcWAJ9o4TgfBH7SfKGkIuA9wLNdUdhkumZECX9+4yCn6n3cLedcz9OuYeQTaCbwmJl9RdJk4HFJY8wsBiBpEnDCzFbH7yQpgyBcvmFmm1o6sKTZwGyA0tJSqqqqOlTAmpqaDu/bXkWnGqhtiPHdX1cxpiSa0Pdqr/NR7+7I651evN5dI5FBsgMYHDdfHi6LdwcwHcDMFkvKBkqAveH6W2ihNQI8CrxuZg+d7c3N7NFwOyorK23q1KkdqAJUVVXR0X3b64q6Br65/GmO5gxk6tSRCX2v9jof9e6OvN7pxevdNRJ5ausVYISkYZKyCEJhfrNttgLXAkgaCWQD+8L5CHAzYf9IE0lfJOhP+WQCy35e5WRlUDm0Dy+87h3uzrmeJ2FBYmYNwBxgIbCO4OqsNZIelHRjuNk9wF2SVhC0PG6zMzdUvAXYFn/qSlI58FmCq8BeDS8NvjNRdTifrrmohHW7jrL32KlkF8U5585JQvtIzGwBQSd6/LL746bXAlPOsm8VcGWzZdsBdXlBu4FrhvfjP9jAS9X7ed+lPfpCNOdcmvE727uJ0QMLKM7J9PtJnHM9jgdJNxGJiKtH9OP51324FOdcz+JB0o1cM6KE/TW1rN/tw6U453oOD5Ju5PRwKX6Xu3OuB/Eg6UbKCnszon+eXwbsnOtRPEi6mWtG9PPhUpxzPYoHSTdzzUUl1DbEeGXzwWQXxTnn2sWDpJuZNKwPWdGIn95yzvUYHiTdTE5WBhOHFvP8a97h7pzrGTxIuqFrLiph/e5j7D3qw6U457o/D5Ju6C0j+gHwYrWf3nLOdX8eJN3QqLIC+uRm+ekt51yP4EHSDUUiYtrF/Xl2/V7qGmLJLo5zzrXKg6Sbeve4Mo6dauDFam+VOOe6Nw+SbmrK8BIKsjN4cuWuZBfFOeda5UHSTWVlRHjH6AH8fs0ev8vdOdeteZB0Y+8aV8ax2ga/OdE51615kHRjU4aXUNg7k6dW7kx2UZxz7qwSGiSSpkvaIKla0r0trB8iaZGkZZJWSrohXH5r+Dz2pldM0oRw3URJq8JjfkNSSj56FyAzGmH66AH8fq2f3nLOdV8JCxJJUeBh4J3AKGCmpFHNNrsPmGdmlwK3AI8AmNkTZjbBzCYAs4A3zGx5uM+3gLuAEeFreqLq0B28e3wZx+saec7vKXHOdVOJbJFcAVSb2SYzqwPmAjOabWNAQThdCLR0DmdmuC+SyoACM/ujBc+j/SHw3kQUvruYfEFfinMyecqv3nLOdVMZCTz2IGBb3Px2YFKzbb4APC3pE0AucF0Lx/kgZwJoUHic+GMOaunNJc0GZgOUlpZSVVV1bqUP1dTUdHjfrjK+j7Fw9U6efvYwWdHzcyavO9Q7Gbze6cXr3TUSGSTtMRN4zMy+Imky8LikMWYWA5A0CThhZqvP9cBm9ijwKEBlZaVNnTq1QwWsqqqio/t2lczy/VR990/ESi9h6piy8/Ke3aHeyeD1Ti9e766RyFNbO4DBcfPl4bJ4dwDzAMxsMZANlMStvwX4SbNjlrdxzJQzaVgf+uZm+c2JzrluKZFB8gowQtIwSVkEoTC/2TZbgWsBJI0kCJJ94XwEuJmwfwTAzHYBRyVdGV6t9RHg1wmsQ7eQEY0wfcwAnl23lxN1DckujnPO/YWEBYmZNQBzgIXAOoKrs9ZIelDSjeFm9wB3SVpB0PK4LexEB3gLsM3MNjU79N8C3wWqgY3AbxNVh+7kXePKOFnfyKL1fvWWc657SWgfiZktABY0W3Z/3PRaYMpZ9q0Crmxh+RJgTJcWtAeYNKwvJXm9eGrVTt417vz0kzjnXHv4ne09RDQibhg7gD+s38vxWj+95ZzrPjxIepB3jS3jVH2MP6zfm+yiOOfcaR4kPUhlRR/65/fymxOdc92KB0kPEpzeKuMP6/eyv6Y22cVxzjnAg6THmTV5KHWNMZ7449ZkF8U55wAPkh7nwn55TLu4H4//cQu1DT4isHMu+TxIeqDbrx7G/ppanlzhfSXOueTzIOmBrh5ewkWlefzvi29w5v5N55xLDg+SHkgSt08ZxtpdR/nTGweTXRznXJrzIOmh3nvpIIpzMvnei28kuyjOuTTnQdJDZWdGuXXSUH6/bg9bDhxPdnGcc2nMg6QHmzV5KBkR8djLm5NdFOdcGvMg6cFKC7J597iB/GzJdo6dqk92cZxzacqDpIe7fcowamob+Okr29re2DnnEsCDpIcbW17IFRV9eOzlzTTG/FJg59z550GSAm6/uoLth07y+7V7kl0U51wa8iBJAdePGkB5cW+/FNg5lxQeJCkgGhG3XVXBnzcf5OXq/ckujnMuzSQ0SCRNl7RBUrWke1tYP0TSIknLJK2UdEPcunGSFktaI2mVpOxw+cxwfqWk30kqSWQdeooPXzmUoX1zuO/Xq30wR+fceZWwIJEUBR4G3gmMAmZKGtVss/uAeWZ2KXAL8Ei4bwbwI+BvzGw0MBWoD5d/HZhmZuOAlcCcRNWhJ8nOjPLgjDFs2necR5/blOziOOfSSEYCj30FUG1mmwAkzQVmAGvjtjGgIJwuBHaG028HVprZCgAzOxAeIxMQkCvpQLhvdQLr0KO89aJ+vHtcGf+9qJr3jB9IRUlusouUNA2NMbYePMGxUw3UNcaoawhetQ2N1DbEyMnKoKwwm9KCbPrmZhGJKNlFdq7HUqJGj5V0EzDdzO4M52cBk8xsTtw2ZcDTQDGQC1xnZkslfRKYCPQH+gFzzew/4o77PeA48DpB6+RN53IkzQZmA5SWlk6cO3duh+pRU1NDXl5eh/ZNhkOnYnzmxZNcWBjlnspeSB37guxJ9T50Ksa2YzG218TYfszYfizGzuMxGmLt2z8qKOolirNFn6xGLi7pxYWFEcrzI2SkScD0pP/vruT1bt20adOWmlllW9slskXSHjOBx8zsK5ImA49LGhOW62rgcuAE8KykpcDzwP8DLgU2Af8NfBr4YvMDm9mjwKMAlZWVNnXq1A4VsKqqio7umyxHCzbz+flrqOlzMe8ZP7BDx+jO9TYz1u06xsI1u1m4Zjfrd58Za2xAQTYXleUzfUA+F5XmU5yTSVZGhF4ZUbIyImRFI2RlRDhe28CuI6fYc/QUu4+eYveR4LVm+0H+vLcOgOzMCOMGFXHpkCIuG1rMlOEl5PVK9q9MYnTn/+9E8np3jUT+VuwABsfNl4fL4t0BTAcws8Vhh3oJsB143syuXWN6AAASaUlEQVT2A0haAFwGHA233Rgunwe8qRM/3X34yqH8fOl2HnxyLW+9uB8F2ZnJLlKnxWLG0q2HWLh6NwvX7mbbwZNIUDm0mM/ccAnjy4u4eEA+RTlZ7T7m+MFvXrZo0SJGTJjEsq2Hg9e2Q3z/pc38z/ObyIpGmHRBH669pD/XjixlcJ+cLqyhcz1XIoPkFWCEpGEEAXIL8KFm22wFrgUekzQSyAb2AQuBf5KUA9QBbwW+Fh5nlKR+ZrYPuB5Yl8A69EjRiPjS+8Yy4+EX+crCDTwwY0yyi9Rhx2sb+PnS7fzg5c1s2n+crGiEq4b35W+nDue6kaX0y+/Vpe8nifLiHMqLc0635mobGlm29TB/WL+XZ9bt4Qu/WcsXfrOWi0vzuXZkf24YW8bogQUdPo3oXE+XsCAxswZJcwhCIQp8z8zWSHoQWGJm84F7gO9I+hRBx/ttFnTaHJL0VYIwMmCBmT0FIOkB4HlJ9cAW4LZE1aEnG1teyEcmV/CDxZt5/8RyxpUXJbtI52TbwRP84OXN/PSVbRyrbWD84CK+evN4rh9VSv55bmH1yohy5QV9ufKCvnzmhpG8sf84z67bw7Pr9vI/z2/ikaqNXNgvl/eMH8iN4wdyQb/0O+fu0ltCT/ia2QJgQbNl98dNrwWmnGXfHxFcAtx8+beBb3dtSVPT3W+/iAWrdvGZX67i1x+/mmgP6DheuuUg//PcJp5ZtwdJ3DC2jI9NqeCyIcXJLtppw0pyufOaC7jzmgs4dLyOBat3MX/5Tr7+7Os89MzrjBlUwI3jB3Lj+EEMKMxOdnGdS7jU7Dl0ABRkZ/L594zm4z9+lX95ci2ff8+obnv6ZemWQzz0zGu88Pp+inMy+Zu3XsisyUMpK+yd7KK1qjg3i1snDeXWSUPZfeQUT67cyfwVO/nSgvX822/XM+XCEt536SCmjxlAbop21Dvnn+wUd8PYAdw+ZRjfe+kNMqPiMzeM7FZhsnzbYb72+9d47rV99M3N4jM3XMKHrxxKTlbP+2gOKMw+3VJ5Y/9xfrlsB79ctp17fraC+361mneMLuV9l5Uz5cK+ZER9dCKXOnreb6s7J5L43LtH0hCL8Z0X3iAjGuGf3nFx0sNk5fYgQBZt2EdxTib/PP0SPjJ5aMr81T6sJJe7r7+IT103giVbDvHLZTt4csVOfrV8JyV5vXj3uDJmTBjIhMFFSf+/cK6zUuO31rVKEg/cOJqGmPGtqo1kRsTdb784KWVZveMIDz3zGs+s20tRTib/+I6L+ehVFSl7f4YkLq/ow+UVffj8e0axaP1efrVsJz/+81Yee3kzQ/rkcOP4gcyYMJARpfnJLq5zHZKav73uTSTxxRljaGw0vvGHaqKRCH9/3Yjz9v5rdx7loWde4+m1eyjIzuCe6y/itikV5/0KrGTqlRFl+pgypo8p4+ipehau3s38FTt5pKqaby6q5pIB+bx99ADeMbqUUWV+ObHrOTxI0kgkIv7tr8bSEDO+9sxrZETFx6cNT+h7rt99lK8/8zq/Xb2b/OwMPnXdRXzs6oqUuEmyMwqyM/lA5WA+UDmYvcdO8dTKXfx21W7++w+v841nX6e8uDdvHxWESmVFnx5xxZ1LXx4kaSYSEf9x0zgaYzH+c+EGdhw+yd+9bUSXXqba0BjjmXV7+OHiLby88QD5vTL4u2tHcMfVwyjsnd4B0pL++dl8bMowPjZlGPtranl23R4WrtnDj/64he+99AbFOZlcNbyEKReWMGV4X4b0yfHWiutWPEjSUDQi/usD4ynKyeJHf9zCz5du50NXDOH/Tb2Q0oKOB8reY6eY++dt/PhPW9l99BSDinrzj++4mFsnDTmnoUvSWUleLz54+RA+ePkQamobeG7DPp5dv4eXqw/w1MpdAAwq6s2U4X2ZMryEy4YUU17c24PFJZUHSZrKiEb4wo2juePqYTy8qJof/XELP/7z1tOB0h6xmLFp/3Fe3XqI517bx8LVu2mIGdeMKOFf3juGt13S30/JdEJerwzeNa6Md40rw8zYuO84L2/cz0vV+/nd6t3MW7IdgKKcTMYOKmTMoELGhT89XNz55EGS5gb3yeHL7x/H304dzsOLqnk8DJTRfcTTh1ZRkptFn9ws+ub1om9uFo1mLNt6mFe3HmLZ1sMcOVkPQHFOJh+9qoJbJw3xIUISQBLD++cxvH8eH5lcQWPMWLfrKCu3H2HVjsOs2nGE776wifrG4LEQOVlRhvTJYWjfHCr65jIk/DmwqDd9crIo6J2RlKCpb4xxoq6Rk3WNnKhr4GR9MH2qPsap+kZONZyZrm2IUd8Yoz78WddoNDQG041mxCwYCdoMYuG8gIhEJBL8m0UlIgpO6WZFI2RERWY0Er7Els317PzTVrIzgxGie2VE6BVO986M0jsrQnZmMJ0dvvyPozfzIHEADOmbw7/fNI6PTxvOI1XVvLBuO1tX7+bQiTpizR5ZI8FF/fO5YewALh1SzGVDirigJM8fDnUeRSNiTNj6gCFAMLjkht3HWLn9CJv2HWfrweNs3HecRRv2Udfs4SwZEVGcm0Xf3CyKc7KoP36KX+9ZHnyRZkTolRl8qWaFN07GDIzgy5rwS7u+MUZtQ9OrMXxwWBACQVA0crI+CIwTdY2cqm88HXQdkRV++WdEg+fESEFYRBSER9OnL3Y6WOz0dGOjUR+LUd9oNDb/QK9fdW7lyIgEIZMZJScrCJfeWWfCpndWlOyMyOllvTKjZGdGyM6I0iv8mZ15JrSaHm3Q9LiDXhlB0J0JveBnUOfu+TvmQeL+wpC+QQulquogU6dOpTFmHD5Rx8HjdeyvqcPMGFNemPZXXXVHvTKijCsvetMAnbGYsfvoKTbvP87uo6c4eLzuTa+dx2Ps3XLo9BMkm1oEZ3vuXUScfr7L6dAJ53tnBV+wRTlZwXTTF23cdE5WBjlZf/kFnJ0Z/PUffNGeeYZMZlREu/BLNBY7EyqLnnuByydNflO9T9UHLaOT9Y3h9JlwPNXQyKkwJE/WxzgZBuWJugYOHq8Ltm161QXH6yoZkeDfIiMiIuHPaCRCNAJRBf9G0ciZVlhE4slPXE12ZrTLytBiuRJ6dNfjRSMKTmvl9WJEabJL4zoiEhEDi3ozsOjs45a19KAjM6MhZogzf/l317+Iz0UkInpFovTKgPwsJXxgzVjMqGs8E061cafv/vIx0GdadvWNRkMYdvWNMRrCU3uNsRgNsaCF1RALWleNYYsrZsG0GTTGgnkzzsupOA8S51yLJJEZ7fnBkWyRiMiORBPeKkgmHznOOedcp3iQOOec6xQPEuecc53iQeKcc65TEhokkqZL2iCpWtK9LawfImmRpGWSVkq6IW7dOEmLJa2RtEpSdrg8S9Kjkl6TtF7S+xNZB+ecc61L2FVbkqLAw8D1wHbgFUnzw+e0N7kPmGdm35I0iuD57hWSMgie1z7LzFZI6gvUh/t8FthrZhdJigB9ElUH55xzbUvk5b9XANVmtglA0lxgBhAfJAYUhNOFwM5w+u3ASjNbAWBmB+L2uR24JFweA/YnqgLOOefaJjvbraudPbB0EzDdzO4M52cBk8xsTtw2ZcDTQDGQC1xnZkslfRKYCPQH+gFzzew/JBUBq4CfAVOBjcAcM9vTwvvPBmYDlJaWTpw7d26H6lFTU0NeXvqNHeX1Ti9e7/TS3npPmzZtqZlVtrVdsm9InAk8ZmZfkTQZeFzSmLBcVwOXAyeAZyUtBVYA5cDLZna3pLuB/wJmNT+wmT0KPAogad+0adO2dLCMJaRnq8frnV683umlvfUe2p6DJTJIdgCD4+bLw2Xx7gCmA5jZ4rBDvYSgT+V5M9sPIGkBcBnwB4Jg+UW4/8/CY7TKzPp1tBKSlrQnkVON1zu9eL3TS1fXO5FXbb0CjJA0TFIWcAswv9k2W4FrASSNBLKBfcBCYKyknLDj/a3AWgvOw/2G4LQW4b5rcc45lzQJa5GYWYOkOQShEAW+Z2ZrJD0ILDGz+cA9wHckfYqg4/22MCwOSfoqQRgZsMDMngoP/c8Ep8AeIgidjyWqDs4559qWsM72VCFpdtjfkla83unF651eurreHiTOOec6xYdIcc451ykeJM455zrFg+Qs2honLJVI+p6kvZJWxy3rI+n3kl4PfxYns4yJIGlwONbb2nBMt78Pl6d03SVlS/qzpBVhvR8Ilw+T9KfwM//T8GrLlCMpGo7v92Q4n/L1lrQ5HLNwuaQl4bIu+5x7kLQgbpywdwKjgJnhWGCp6jHC+3ni3As8a2YjgGfD+VTTANxjZqOAK4GPh//PqV73WuBtZjYemABMl3Ql8O/A18xsOHCIdtyj1UP9PbAubj5d6j3NzCbE3T/SZZ9zD5KWnR4nzMzqgKZxwlKSmT0PHGy2eAbwg3D6B8B7z2uhzgMz22Vmr4bTxwi+XAaR4nW3QE04mxm+DHgb8PNwecrVG0BSOfAu4LvhvEiDep9Fl33OPUhaNgjYFje/PVyWTkrNbFc4vRsoTWZhEk1SBXAp8CfSoO7h6Z3lwF7g9wTj1h02s4Zwk1T9zD8E/BMQC+f7kh71NuBpSUvDcQihCz/nyR5ry/UAZmaSUvY6cUl5wP8BnzSzo8EfqYFUrbuZNQITwoFQf0k4onYqk/RugkdQLJU0NdnlOc+uNrMdkvoDv5e0Pn5lZz/n3iJpWXvGCUt1e8LRmZtGad6b5PIkhKRMghB5wsyaxnBLi7oDmNlhYBEwGSgKhySC1PzMTwFulLSZ4HT124Cvk/r1xsx2hD/3EvzhcAVd+Dn3IGlZe8YJS3XzgY+G0x8Ffp3EsiREeH78f4F1ZvbVuFUpXXdJ/cKWCJJ6Ezx8bh1BoNwUbpZy9TazT5tZuZlVEPxO/8HMbiXF6y0pV1J+0zTB855W04Wfc7+z/SwUPPb3Ic6ME/avSS5Swkj6CcFAmCXAHuDzwK+AecAQYAtws5k175Dv0SRdDbxA8IybpnPmnyHoJ0nZuksaR9C5GiX4Y3KemT0o6QKCv9T7AMuAD5tZbfJKmjjhqa1/MLN3p3q9w/r9MpzNAH5sZv+q4MmzXfI59yBxzjnXKX5qyznnXKd4kDjnnOsUDxLnnHOd4kHinHOuUzxInHPOdYoHiUsJkvqGI5sul7Rb0o64+XaN5irp+5IubmObj0u6tYvK/KKkCZIiXT3CtKTbJQ2Im2+zbs51lF/+61KOpC8ANWb2X82Wi+AzH2txx/NM0ovAHIKbw/abWdE57h8Nhzo567HNbHnnS+pc67xF4lKapOHh80aeANYAZZIelbQkfBbH/XHbNrUQMiQdlvTl8Jkdi8MxipD0RUmfjNv+y+GzPTZIuipcnivp/8L3/Xn4XhNaKeaXgfyw9fTD8BgfDY+7XNIjYaulqVwPSVoJXCHpAUmvSFot6dsKfJBgePifNrXImuoWHvvDCp5NsVrSl8JlrdX5lnDbFZIWdfF/kUsBHiQuHVxC8LyJUeGYQ/eGz2QYD1yvlp81Uwg8Fz6zYzFw+1mOLTO7AvhHoCmUPgHsDp9z8i8Eowq35l7gWPisiI9IGgO8D7jKzCYQ3I18S1y5njezcWa2GPi6mV0OjA3XTTeznwLLgQ+Gx6w7XdhgGPUvAtPCck1RMJhha3X+PHBtuPx9bdTFpSEPEpcONprZkrj5mZJeBV4FRhI8vKy5k2b223B6KVBxlmP/ooVtriYYcgMzW0HQEjoX1wGXA0sUDPX+VuDCcF0dZ4a7ALhW0p+BFeF2o9s49iSCMab2m1k98GPgLeG6s9X5JeCHku7EvzNcC3wYeZcOjjdNSBpB8IS8K8zssKQfAdkt7FMXN93I2X9XatuxzbkSwfhun/uLhcEItSct7NiUlAN8E7gsHCL8i7Rcl/Y6W53vIgigdwOvSrrUzA514n1civG/Lly6KQCOAUfDobPfkYD3eAm4GUDSWFpu8ZzW9FAlnRnK/BngZkkl4fK+koa0sGtvgsEm94eju74/bt0xIL+Fff4ETAuP2XTK7Lk26nOBmf0R+BzBo2hT8cFPrhO8ReLSzavAWmA9wYinLyXgPf6b4FTQ2vC91gJH2tjnf4GVkpaE/SQPAM9IigD1wN8AO+N3MLMDkn4QHn8XQUg0+T7wXUknCZ490bTPdkmfA6oIWj6/MbOn4kKsJV+TNCzc/mkzW91GXVya8ct/neti4ZdyhpmdCk+lPQ2MiHucq3MpxVskznW9PODZMFAE/LWHiEtl3iJxzjnXKd7Z7pxzrlM8SJxzznWKB4lzzrlO8SBxzjnXKR4kzjnnOuX/A8eQi1ppIcIfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.title('Nadam')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have learned several variants of gradient descent in the next chapter we will learn about one of the most popularly used deep learning algorithms called Recurrent Neural Network(RNN) which is widely used for the language tasks.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}