01. Introduction to Deep Learning/1.09 Building Neural Network from scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building Neural Network from Scratch\n",
    "\n",
    "Putting all the concepts we have learned so far, we will see how to build a neural network\n",
    "from scratch. We will learn how the neural network learns to perform the XOR gate\n",
    "operation. The XOR gate returns 1 only when exactly only one of its inputs is 1 else it returns 0 as shown in\n",
    "the following figure:\n",
    "\n",
    "![image](images/1.png)\n",
    "\n",
    "\n",
    "To perform the XOR gate operation, we build a simple two-layer neural network as shown\n",
    "in the following figure. As you can observe, we have an input layer with two nodes, a\n",
    "hidden layer with five nodes and an output layers which consist of 1 node:\n",
    "\n",
    "\n",
    "![image](images/2.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, import the libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare the data as shown in the above XOR table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.array([ [0, 1], [1, 0], [1, 1],[0, 0] ])\n",
    "y = np.array([ [1], [1], [0], [0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the number of nodes in each layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_input = 2\n",
    "num_hidden = 5\n",
    "num_output = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize weights and bias randomly. First, we initialize, input to hidden layer weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Wxh = np.random.randn(num_input,num_hidden)\n",
    "bh = np.zeros((1,num_hidden))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now initialize, hidden to output layer weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Why = np.random.randn (num_hidden,num_output)\n",
    "by = np.zeros((1,num_output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the sigmoid activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1 / (1+np.exp(-z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the derivative of the sigmoid function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid_derivative(z):\n",
    "    return np.exp(-z)/((1+np.exp(-z))**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the forward propagation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def forward_prop(x,Wxh,Why):\n",
    "    z1 = np.dot(x,Wxh) + bh\n",
    "    a1 = sigmoid(z1)\n",
    "    z2 = np.dot(a1,Why) + by\n",
    "    y_hat = sigmoid(z2)\n",
    "    \n",
    "    return z1,a1,z2,y_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the backward propagation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def backword_prop(y_hat, z1, a1, z2):\n",
    "    delta2 = np.multiply(-(y-y_hat),sigmoid_derivative(z2))\n",
    "    dJ_dWhy = np.dot(a1.T, delta2)\n",
    "    delta1 = np.dot(delta2,Why.T)*sigmoid_derivative(z1)\n",
    "    dJ_dWxh = np.dot(x.T, delta1) \n",
    "\n",
    "    return dJ_dWxh, dJ_dWhy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the cost function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cost_function(y, y_hat):\n",
    "    J = 0.5*sum((y-y_hat)**2)\n",
    "    \n",
    "    return J"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the learning rate and number of training iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "alpha = 0.01\n",
    "num_iterations = 5000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's start training the network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cost = []\n",
    "for i in range(num_iterations):\n",
    "    \n",
    "    #perform forward propagation and predict output\n",
    "    z1,a1,z2,y_hat = forward_prop(x,Wxh,Why)\n",
    "    \n",
    "    #perform backward propagation and calculate gradients\n",
    "    dJ_dWxh, dJ_dWhy = backword_prop(y_hat, z1, a1, z2)\n",
    "        \n",
    "    #update the weights\n",
    "    Wxh = Wxh -alpha * dJ_dWxh\n",
    "    Why = Why -alpha * dJ_dWhy\n",
    "    \n",
    "    #compute cost\n",
    "    c = cost_function(y, y_hat)\n",
    "    \n",
    "    #store the cost\n",
    "    cost.append(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the cost function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cost')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XuYXXV97/H3Z+6TmczkMpNJSLgkEEAQpDIGENB4waKnQq0exHrDVmNPS61aPQ85WmzRnnr09IitaX0QqaAV8Hi8BI1cFEaUAiZQbgkEknBJAiH3y0ySuX7PH2vNZGczt8xkzZ6Z/Xk9z372Wr912d/vEPZ3/9bltxQRmJmZDaak0AGYmdn452JhZmZDcrEwM7MhuViYmdmQXCzMzGxILhZmZjYkFwuzSUBSq6QFhY7DJi8XC5s0JP2xpFXpF+dLkn4h6YJR7vM5SW8dZPliST3pZ/a+bhvNZw4jphZJH81ti4jaiNiQ5edacSsrdABmR4OkTwNXAX8G3AF0ABcDlwK/zfjjX4yIeRl/hllBuWdhE56keuAa4C8i4kcR0RYRnRFxW0R8Nl2nUtK1kl5MX9dKqkyXNUj6maTdknZK+o2kEknfBY4Dbkt7DP/9COP6jqQv5cwvlrQpZ/45SZ+R9JikPZJulVSVs/xSSY9I2itpvaSLJf09cCHwjTSmb6TrhqSTev8ekm6StE3S85I+L6kkXXaFpN9K+t+Sdkl6VtLbR/aXt2LiYmGTwXlAFfDjQdb5HHAucBbwGmAR8Pl02V8Dm4BGoAn4H0BExAeBF4B3pod5vpJB7JeR9IDmA2cCVwBIWgTcBHwWmAa8AXguIj4H/Aa4Mo3pyn72+c9APbAAeCPwIeAjOcvPAdYCDcBXgG9L0lHPzCYVFwubDGYC2yOia5B13g9cExFbI2Ib8HfAB9NlncAc4Pi0R/KbOLJB045JeyW9r8uOYNt/iogXI2IncBtJMQP4U+CGiLgrInoiYnNEPDXUziSVApcDSyNiX0Q8B/wjh3IFeD4ivhUR3cCNJLk3HUHMVoRcLGwy2AE0SBrsHNwxwPM588+nbQBfBdYBd0raIOmqI/z8FyNiWs7rB0ew7Zac6f1AbTp9LLD+COOApLdQzitzndvfZ0bE/nSyFrNBuFjYZHA/0A784SDrvAgcnzN/XNpG+gv8ryNiAXAJ8GlJb0nXG82wzG3AlJz52Uew7UbgxAGWDRbTdpKeUn6um4/gs81ewcXCJryI2ANcDSyT9IeSpkgql/R2Sb3nGW4GPi+pUVJDuv73ACT9gaST0uP2e4BuoCfd7mWSY/8j8QjwDkkzJM0GPnkE234b+Iikt6Qn2+dKOnWomNJDSz8A/l7SVEnHA58mzdVspFwsbFKIiH8k+VL8PLCN5Jf5lcBP0lW+BKwCHgMeBx5O2wAWAr8EWkl6Kf8SEfeky/6BpMjslvSZIwzru8CjwHPAncCtR5DP70hOSn+NpID9mkO9ha8D70mvZvqnfjb/S5JezQaSy4a/D9xwhLGbHUZ++JGZmQ3FPQszMxuSi4WZmQ0p02KR3nG6VtK6gS5HlHSZpDWSVkv6ft6yOkmbeu9SNTOzwshsbKj05qBlwEUkd8eulLQ8ItbkrLMQWAqcHxG7JM3K280XgXuzitHMzIYny4EEFwHrekfClHQLyaBua3LW+RiwLCJ2AUTE1t4Fks4muav0dqB5qA9raGiIE044YcTBtrW1UVNTM+LtJ6Jiy7nY8gXnXCxGk/NDDz20PSIah1ovy2Ixl+TyxV6bSMakyXUygKT7gFLgbyPi9nTQs38EPgAMODx0rhNOOIFVq1aNONiWlhYWL1484u0nomLLudjyBedcLEaTs6Tnh16r8EOUl5Fc474YmAfcK+kMkiKxIiI2DTa+maQlwBKApqYmWlpaRhxIa2vrqLafiIot52LLF5xzsRiLnLMsFptJxrfpNY9XDjmwCXgwIjqBZyU9TVI8zgMulPTnJGPWVEhqjYjDTpJHxHXAdQDNzc0xml8T/jUy+RVbvuCci8VY5Jzl1VArgYWS5kuqIBkJc3neOj8h6VWQDsFwMrAhIt4fEcdFxAnAZ4Cb8guFmZmNncyKRTpc9JUkTy17EvhBRKyWdI2kS9LV7gB2SFoD3AN8NiJ2ZBWTmZmNTKbnLCJiBbAir+3qnOkgGc/n04Ps4zvAd7KJ0MzMhsN3cJuZ2ZBcLMzMbEhFXyz2Huzka3c9zYbd3YUOxcxs3Cr6YhE98PVfPcPTu3qGXtnMrEgVfbGoqy6jvFTs7fBzPczMBlL0xUISM2sqXSzMzAZR9MUCoGFqhYuFmdkgXCyAmTWV7Gt3sTAzG4iLBdBQW8ke9yzMzAbkYgE01CaHoZIbys3MLJ+LBTCztoKuHtjX3lXoUMzMxiUXC5LDUAA7WjsKHImZ2fjkYgHMTIvF9tb2AkdiZjY+uViQnLMA2OFiYWbWLxcLDh2G2u7DUGZm/XKxAGbUJD0LH4YyM+ufiwVQXlpCTblPcJuZDcTFIlVfIfcszMwG4GKRmloh9yzMzAbgYpGqr3TPwsxsIC4Wqak+DGVmNqBMi4WkiyWtlbRO0lUDrHOZpDWSVkv6ftp2lqT707bHJL03yzgB6irE3oNddHT5iXlmZvnKstqxpFJgGXARsAlYKWl5RKzJWWchsBQ4PyJ2SZqVLtoPfCginpF0DPCQpDsiYndW8dZXCoAdbe3Mqa/O6mPMzCakLHsWi4B1EbEhIjqAW4BL89b5GLAsInYBRMTW9P3piHgmnX4R2Ao0ZhgrUyvSYuGT3GZmr5BZzwKYC2zMmd8EnJO3zskAku4DSoG/jYjbc1eQtAioANbnf4CkJcASgKamJlpaWkYcbHn3QUDc/R8r2d6Y5Z9l/GhtbR3V32yiKbZ8wTkXi7HIudDfimXAQmAxMA+4V9IZvYebJM0Bvgt8OCJecTIhIq4DrgNobm6OxYsXjziQrSvuBg5wzIJTWXz2vBHvZyJpaWlhNH+ziabY8gXnXCzGIucsD0NtBo7NmZ+XtuXaBCyPiM6IeBZ4mqR4IKkO+DnwuYh4IMM4geQEN3jIDzOz/mRZLFYCCyXNl1QBXA4sz1vnJyS9CiQ1kByW2pCu/2Pgpoj4YYYx9qksharyEo88a2bWj8yKRUR0AVcCdwBPAj+IiNWSrpF0SbraHcAOSWuAe4DPRsQO4DLgDcAVkh5JX2dlFSuAJBpqK32C28ysH5mes4iIFcCKvLarc6YD+HT6yl3ne8D3soytPzNrK9nmnoWZ2Sv4Du4cjbUV7lmYmfXDxSLHzJpKn+A2M+uHi0WOmbUV7GzroKcnCh2Kmdm44mKRo6G2kq6eYM+BzkKHYmY2rrhY5JhZmzxedUebD0WZmeVyscjRWFsJwHaf5DYzO4yLRY6ZabHYts89CzOzXC4WOWZNdbEwM+uPi0WOaVPKKS+Vb8wzM8vjYpFDEo21lWzd62JhZpbLxSJP49RKtu47WOgwzMzGFReLPI1Tq3zOwswsj4tFnll1lS4WZmZ5XCzyNNZWsqOtg87uVzyYz8ysaLlY5JlVl1w+69FnzcwOcbHIM2tqFYBPcpuZ5XCxyNOY3pjny2fNzA5xscjTdxe3b8wzM+vjYpGnodY9CzOzfC4WeSrKSpg+pZxtrT5nYWbWy8WiH7OmVrlnYWaWI9NiIeliSWslrZN01QDrXCZpjaTVkr6f0/5hSc+krw9nGWe+WXWVbPWNeWZmfcqy2rGkUmAZcBGwCVgpaXlErMlZZyGwFDg/InZJmpW2zwC+ADQDATyUbrsrq3hzNdZWsmFb21h8lJnZhJBlz2IRsC4iNkREB3ALcGneOh8DlvUWgYjYmrb/PnBXROxMl90FXJxhrIdpTIf8iIix+kgzs3Ets54FMBfYmDO/CTgnb52TASTdB5QCfxsRtw+w7dz8D5C0BFgC0NTUREtLy4iDbW1t7dt+78uddHT38PO7Wqit0Ij3Od7l5lwMii1fcM7FYixyzrJYDPfzFwKLgXnAvZLOGO7GEXEdcB1Ac3NzLF68eMSBtLS00Lv93kdf5Oan/pNTXtPMwqapI97neJebczEotnzBOReLscg5y8NQm4Fjc+bnpW25NgHLI6IzIp4FniYpHsPZNjO9N+b5JLeZWSLLYrESWChpvqQK4HJged46PyHpVSCpgeSw1AbgDuBtkqZLmg68LW0bE31Dfnh8KDMzIMPDUBHRJelKki/5UuCGiFgt6RpgVUQs51BRWAN0A5+NiB0Akr5IUnAAromInVnFmq9vyA/3LMzMgIzPWUTECmBFXtvVOdMBfDp95W97A3BDlvENpLayjOryUt+YZ2aW8h3c/ZCUPovbxcLMDFwsBtRUV8nLe33OwswMXCwG1FRX5WJhZpZysRjA7Loqtuw96Lu4zcxwsRjQ7PoqDnb2sPdAV6FDMTMrOBeLAcyuT57F/dLeAwWOxMys8FwsBjC7LikWW/b4vIWZmYvFAHp7Fj7JbWbmYjGgWVPTw1DuWZiZuVgMpKKshIZa32thZgYuFoOaXV/pcxZmZrhYDGp2XZUPQ5mZ4WIxqNn1vovbzAxcLAY1u66KXfs7OdjZXehQzMwKysViEE11vnzWzAxcLAY1p74a8I15ZmYuFoOYXZ88MW+LexZmVuRcLAYx2z0LMzPAxWJQtZVl1FaWuWdhZkXPxWIITXW+Mc/MLNNiIeliSWslrZN0VT/Lr5C0TdIj6eujOcu+Imm1pCcl/ZMkZRnrQObUV7tnYWZFL7NiIakUWAa8HTgNeJ+k0/pZ9daIOCt9XZ9u+3rgfOBM4NXA64A3ZhXrYGbXV/HSbhcLMytuWfYsFgHrImJDRHQAtwCXDnPbAKqACqASKAdeziTKIcydVs3L+w7S0dVTiI83MxsXyjLc91xgY878JuCcftZ7t6Q3AE8Dn4qIjRFxv6R7gJcAAd+IiCfzN5S0BFgC0NTUREtLy4iDbW1t7Xf7fS93EgE/vbOFximT6xTPQDlPVsWWLzjnYjEWOWdZLIbjNuDmiGiX9HHgRuDNkk4CXgXMS9e7S9KFEfGb3I0j4jrgOoDm5uZYvHjxiANpaWmhv+3L123nhiceZN4pr+G8E2eOeP/j0UA5T1bFli8452IxFjln+VN5M3Bszvy8tK1PROyIiPZ09nrg7HT6XcADEdEaEa3AL4DzMox1QHOnJfdabN7tZ3GbWfHKslisBBZKmi+pArgcWJ67gqQ5ObOXAL2Hml4A3iipTFI5ycntVxyGGgtzpiXjQ73oYmFmRSyzw1AR0SXpSuAOoBS4ISJWS7oGWBURy4FPSLoE6AJ2Alekm/8QeDPwOMnJ7tsj4rasYh1MZVkpjVMr2bzLxcLMilem5ywiYgWwIq/t6pzppcDSfrbrBj6eZWxHYu60ah+GMrOiNrku78nI3OkuFmZW3FwshmFe2rOIiEKHYmZWEC4Ww3DMtGo6unrY3tpR6FDMzArCxWIYfPmsmRU7F4thmDs9KRa+fNbMitWwioWk7w6nbbI6prdn4ctnzaxIDbdncXruTDqi7NkDrDvp1FeXM7WyzIehzKxoDVosJC2VtA84U9Le9LUP2Ar8dEwiHCfmTq9mk3sWZlakBi0WEfEPETEV+GpE1KWvqRExM72hrmj4xjwzK2bDPQz1M0k1AJI+IOn/SDo+w7jGnXnTq9m0c7/vtTCzojTcYvGvwH5JrwH+GlgP3JRZVOPQcTNr2Nfexa79nYUOxcxszA23WHRF8pP6UpIHES0DpmYX1vhz/IwpALywc3+BIzEzG3vDLRb7JC0FPgj8XFIJyaNOi8bxM5Ni8fyOtgJHYmY29oZbLN4LtAN/EhFbSB5k9NXMohqHju3tWexwz8LMis+wikVaIP4dqJf0B8DBiCiqcxZV5aU01VXyvA9DmVkRGu4d3JcBvwP+K3AZ8KCk92QZ2Hh0/Iwa9yzMrCgN9+FHnwNeFxFbASQ1Ar8keaJd0Th2xhTuW7e90GGYmY254Z6zKOktFKkdR7DtpHH8zCls2XuQg53dhQ7FzGxMDbdncbukO4Cb0/n3kve41GLQe0XUxp37WdhUVFcOm1mRG7RYSDoJaIqIz0r6I+CCdNH9JCe8i8pxM3ovn3WxMLPiMlTP4lpgKUBE/Aj4EYCkM9Jl78w0unGmr1j4iigzKzJDnXdoiojH8xvTthOG2rmkiyWtlbRO0lX9LL9C0jZJj6Svj+YsO07SnZKelLRG0pCfl7UZNRXUVpax0cXCzIrMUD2LaYMsqx5sw/SZF8uAi4BNwEpJyyNiTd6qt0bElf3s4ibg7yPiLkm1QM8QsWZOEsfNmMJzvovbzIrMUD2LVZI+lt+Y9gAeGmLbRcC6iNgQER3ALSRjSw1J0mlAWUTcBRARrRExLn7Oz2+s4dntLhZmVlyG6ll8EvixpPdzqDg0AxXAu4bYdi6wMWd+E3BOP+u9W9IbgKeBT0XERuBkYLekHwHzSe7puCoiDrtmVdISYAlAU1MTLS0tQ4Q0sNbW1mFtX9rWwQs7Ornr7nsoL9GIP288GG7Ok0Wx5QvOuViMRc6DFouIeBl4vaQ3Aa9Om38eEXcfpc+/Dbg5ItolfRy4EXhzGteFwO8BLwC3AlcA386L7zrgOoDm5uZYvHjxiANpaWlhONvvmbaZ5esf4fjTmzl5gl8RNdycJ4tiyxecc7EYi5yHOzbUPRHxz+lruIViM3Bszvy8tC13vzsioj2dvZ5Dz/XeBDySHsLqAn4CvHaYn5upBQ21AGzY1lrgSMzMxk6Wd2GvBBZKmi+pArgcWJ67gqQ5ObOXAE/mbDstHVYEkt5G/onxgljQWAPA+m0+b2FmxWO4d3AfsYjoknQlcAdQCtwQEaslXQOsiojlwCckXQJ0ATtJDjUREd2SPgP8SpJIzpd8K6tYj0RNZRlz6qtYv9U9CzMrHpkVC4CIWEHesCARcXXO9FLSm/762fYu4Mws4xupBY01rPcVUWZWRIpuMMCj4cTGWjZsbSV50qyZ2eTnYjECCxpq2NfexbbW9qFXNjObBFwsRuDEWckVUeu3+lCUmRUHF4sROLExLRa+fNbMioSLxQjMrqtiSkUp63xFlJkVCReLESgpESc3TeWpLXsLHYqZ2ZhwsRihU2dPZe2Wfb4iysyKgovFCJ06eyq79neybZ+viDKzyc/FYoROmV0HwJNb9hU4EjOz7LlYjNCps5MRZ9f6vIWZFQEXixGaXlNBU10lT7lnYWZFwMViFE6ZXcdaFwszKwIuFqNw6uypPLO1la7ugj8e3MwsUy4Wo3BK01Q6unp4boeH/TCzyc3FYhReNSe5Imr1iz7JbWaTm4vFKCxsqqWyrITHN+0pdChmZplysRiF8tISTjumjsc2u1iY2eTmYjFKZ86tZ/XmPXT3eNgPM5u8XCxG6Yx502jr6ObZ7R6B1swmLxeLUTpzXj0Aj/m8hZlNYi4Wo3RiYy3V5aUuFmY2qWVaLCRdLGmtpHWSrupn+RWStkl6JH19NG95naRNkr6RZZyjUVoiXj23jsd9ktvMJrHMioWkUmAZ8HbgNOB9kk7rZ9VbI+Ks9HV93rIvAvdmFePRcsbcaax+cQ+dvpPbzCapLHsWi4B1EbEhIjqAW4BLh7uxpLOBJuDOjOI7as4+fjoHO3t8c56ZTVplGe57LrAxZ34TcE4/671b0huAp4FPRcRGSSXAPwIfAN460AdIWgIsAWhqaqKlpWXEwba2to54+46DSY/ill+uZPf88hHHMNZGk/NEVGz5gnMuFmORc5bFYjhuA26OiHZJHwduBN4M/DmwIiI2SRpw44i4DrgOoLm5ORYvXjziQFpaWhjN9l977B52lU1l8eLmEe9jrI0254mm2PIF51wsxiLnLIvFZuDYnPl5aVufiNiRM3s98JV0+jzgQkl/DtQCFZJaI+IVJ8nHi+YTpvPrtduICAYrcGZmE1GW5yxWAgslzZdUAVwOLM9dQdKcnNlLgCcBIuL9EXFcRJwAfAa4aTwXCoDXnTCDHW0dPLvdI9Ca2eSTWc8iIrokXQncAZQCN0TEaknXAKsiYjnwCUmXAF3ATuCKrOLJ2utOmA7Ayud2sqCxtsDRmJkdXZmes4iIFcCKvLarc6aXAkuH2Md3gO9kEN5RdWJjLdOnlPPgszt57+uOK3Q4ZmZHle/gPkok8foTG7hv3XYiPKigmU0uLhZH0YULG3h5bzvPbPWggmY2ubhYHEUXLGwA4DfPbC9wJGZmR5eLxVE0b/oUFjTU8JtnthU6FDOzo8rF4ii7YGEDD27YSXtXd6FDMTM7alwsjrILFzZyoLObVc/tKnQoZmZHjYvFUXb+STOpLCvhrjUvFzoUM7OjxsXiKJtSUcaFCxu5c/UWX0JrZpOGi0UG3nZ6Ey/uOcgTmz1kuZlNDi4WGXjrq5ooEdyxekuhQzEzOypcLDIwo6aCRfNncLsPRZnZJOFikZH/cuYxrNvaypqXfCjKzCY+F4uMvPPMOZSXih89vHnolc3MxjkXi4xMm1LBW05t4qePbKaru6fQ4ZiZjYqLRYb+6LVz2d7a4bGizGzCc7HI0OJTZjGjpoLv/+6FQodiZjYqLhYZqigr4X2LjuVXT77Mxp37Cx2OmdmIuVhk7APnHo8kvvvA84UOxcxsxFwsMjanvpqLXz2bW373Avs7ugodjpnZiLhYjIE/OX8+ew928e8P+NyFmU1MmRYLSRdLWitpnaSr+ll+haRtkh5JXx9N28+SdL+k1ZIek/TeLOPM2tnHT+fChQ1889fr3bswswkps2IhqRRYBrwdOA14n6TT+ln11og4K31dn7btBz4UEacDFwPXSpqWVaxj4ZNvXciOtg6+53MXZjYBZdmzWASsi4gNEdEB3AJcOpwNI+LpiHgmnX4R2Ao0ZhbpGDj7+BlcuLCBf2lZz+79HYUOx8zsiCirge4kvQe4OCJ6Dy19EDgnIq7MWecK4B+AbcDTwKciYmPefhYBNwKnR0RP3rIlwBKApqams2+55ZYRx9va2kptbe2Itx+Ojft6+MJ/HGDxsWV86LTKTD9rOMYi5/Gk2PIF51wsRpPzm970pocionnIFSMikxfwHuD6nPkPAt/IW2cmUJlOfxy4O2/5HGAtcO5Qn3f22WfHaNxzzz2j2n64vvDTJ2L+VT+LJzbvHpPPG8xY5TxeFFu+Ec65WIwmZ2BVDOM7PcvDUJuBY3Pm56VtuYVqR0S0p7PXA2f3LpNUB/wc+FxEPJBhnGPqU289mRk1FXz2/z5GR5fHjDKziSHLYrESWChpvqQK4HJgee4KkubkzF4CPJm2VwA/Bm6KiB9mGOOYq59Szj/80ZmseWkv1/7y6UKHY2Y2LJkVi4joAq4E7iApAj+IiNWSrpF0SbraJ9LLYx8FPgFckbZfBrwBuCLnstqzsop1rF10WhOXNc/jm79ez289yKCZTQBlWe48IlYAK/Lars6ZXgos7We77wHfyzK2Qrv6nafzyMbd/MX3H2b5ledz/MyaQodkZjYg38FdILWVZXzrQ81I8Kc3rmJnmy+nNbPxy8WigI6fWcM3P3A2G3fu5wPXP8ie/Z2FDsnMrF8uFgV27oKZXPehZtZtbeWPr3+Al/ceLHRIZmav4GIxDrzx5Ea+9eFmntvexruW3ceTL+0tdEhmZodxsRgn3nhyIz/4s/PoCfjDZffx3Qee770x0cys4FwsxpHTj6ln+V+ez7kLZvI3P3mCj3xnJc/vaCt0WGZmLhbjzaypVfzbFa/jC+88jZXP7uSir93LV+94yie/zaygXCzGoZIS8ZHz53P3ZxbzjlfPZtk96zn/f93NV25/iq0+AW5mBeBiMY411VVx7eW/xy/+6kLeeEoj//rr9Zz35btZctMq7nlqK53dHlvKzMZGpndw29Hxqjl1LPvj1/Ls9jZu+d0L/PChTdy55mXqq8t5y6tm8funz+b8kxqorfR/TjPLhr9dJpD5DTUsfcer+Ou3nULL2q3cvnoLv3pyKz96eDOlJeKMufWcd+JMzl0wk9fMq2falIpCh2xmk4SLxQRUUVbC206fzdtOn01ndw8rn93Jf6zfwf0bdvCtezfwry3rAThuxhTOmFfPGXPredWcOhbOqmVOfRWSCpyBmU00LhYTXHlpCa8/qYHXn9QAQFt7F//5wm4e37yHxzfv5tGNu/n5Yy/1rV9TUcqJs2o5aVYtJa0d7J/5EsdOn8KxM6qpry53ITGzfrlYTDI1lWVcsLCBCxY29LXtbOvg6Zf3sW5ra9/rP9btYMveTn749MN9602tLGPejCkcO72aY2dMYd70aubUV9FUV8Xs+ioaayspK/U1EWbFyMWiCMyoqeDcBcm5jFy/+OU9HHfaa9m48wCbdu1n4879bNx1gA3b27j3mW0c7Dz8aqsSQUNt5WEFpKmuitl1VTRMrWRmTQWNUyuZUVNBuYuK2aTiYlHEqsvE6cfUc/ox9a9YFhHsaOtgy56DvLz3IFv2HuTlPQd5aU8y/dyONh7YsIO9B7v63fe0KeXMrKmgobYyfVUw87DpCuqrK5g+pZz66nL3WMzGORcL65ekvi/6V899ZTHptb+ji61729nR1s62fR1sb21nR2v63tbO9n0dPPnSXra3tg9YWACmVpUxfUpaPNL36VMqqK8uT6ZrKpg2pYJp1eVMm1JOXVU5U6vKXGTMxoiLhY3KlIoyTmgo44SGoZ/0197Vzc62Drbv62BHWzt7DnSyq62DXfs7k+n96fT+Dp7b3sau/R3sG6TAAFSXl1JXXcbUtHj0FpGpVeXUVZVRV907X8bzW7uY8uzOvvnayjKmVJRRUeaCYzYUFwsbM5Vlpcypr2ZOffWwt+nq7mHPgU52H+hk9/4OdrUl0/sOdrLvYBf7Dnay90AX+9qT9937O9i4cz97D3ay92AXHV2Hn3e59uH7X/EZFaUlTKkspaaijJrKUqak78l8GVMqSvsKS/7yKZXJsuryUqrKS6muKO2bLi3xlWU2ebhY2LhWVlrCzNpKZtZWjmj7g53dfUWl5b4HWXj6mew72MXeA53s7+imrb2Lto5u9nd00drexf72bto6umhr72JHawdtHUlba3sX7V1HNrxKRWkJVeUlVFekhST2RFh1AAALlUlEQVQtIlXlJVSnhaWq/NCy6nRZbtGpLCulsqyEirKSvvdkOmmvzJmvKCtxgbLMuFjYpNb7Zdw4tZIXppVy4cLGEe+rq7uH/Z1pgWk/VGDa2rs52NnNgc7k/WBnNwc6eg6f7+zmQEfy3t7Zw/bWjrz1k2U9o3yESVmJcgpKCT2dHdQ//GsqSkuoLC9J30uT97wCVFZSQnmZqCg9NF1eUkJ5qSgrTbYtKxXlpUlbeWkJZel0Rc50srykL5ayElFeVtK3r9IS+X6eCSjTYiHpYuDrQClwfUR8OW/5FcBXgc1p0zci4vp02YeBz6ftX4qIG7OM1WwoZaUl1JWWUFdVnsn+I4LO7nhFkeno6ul7tfe9uvvmc987upNi1NHdQ3tnDy+8+CLTZ9Yetu3eA53p+t19bZ3dPXR1Bx3dyXTWz92qKD1UhHoLSFlJSfqezJeWiLJSUVpS0td2+HvaXnp4+9Yt7fxq9xOH1ivNWz9/P6X9tZf0xVBaklzwUapkvkSH2kty2gZq793PobZkZOn8/ZWIcV1EMysWkkqBZcBFwCZgpaTlEbEmb9VbI+LKvG1nAF8AmoEAHkq33ZVVvGaFJomKsuTXeH310SlILS07Wbz47CPerrsn6Ow+VEQ6u3vo7Ak608LSmbZ19fTQ0RV09eS1p4Wnb9vDlvXQ0R109bb3BD09QVdP0N33nmzbfVh7sv6BznS++1B77nr7D3bz2K6X6Oruydt+/D95skT09bxKc4rIKwvO4YWnsfQgixdnG1uWPYtFwLqI2AAg6RbgUiC/WPTn94G7ImJnuu1dwMXAzRnFamY5ki+n5BDeRNPS0sLifr45I4Ke4PDi0n14McotLp3dPfT0QE8E3ZEUtO6e3mle0RYRdPfT3pPz3hNJIe6JvOW520WyvLf9lev2fh5902rtyPzvmmWxmAtszJnfBJzTz3rvlvQG4GngUxGxcYBt5+ZvKGkJsASgqamJlpaWEQfb2to6qu0nomLLudjyBeecNZF8iY7qi7SEUT9ZqLW1I/OcC32C+zbg5ohol/Rx4EbgzcPdOCKuA64DaG5ujv5+TQzXQL9GJrNiy7nY8gXnXCzGIucs70baDBybMz+PQyeyAYiIHRHRns5eD5w93G3NzGzsZFksVgILJc2XVAFcDizPXUHSnJzZS4An0+k7gLdJmi5pOvC2tM3MzAogs8NQEdEl6UqSL/lS4IaIWC3pGmBVRCwHPiHpEqAL2AlckW67U9IXSQoOwDW9J7vNzGzsZXrOIiJWACvy2q7OmV4KLB1g2xuAG7KMz8zMhscjqJmZ2ZBcLMzMbEguFmZmNiRF1oPAjBFJ24DnR7GLBmD7UQpnoii2nIstX3DOxWI0OR8fEUOOsDlpisVoSVoVEc2FjmMsFVvOxZYvOOdiMRY5+zCUmZkNycXCzMyG5GJxyHWFDqAAii3nYssXnHOxyDxnn7MwM7MhuWdhZmZDcrEwM7MhFX2xkHSxpLWS1km6qtDxjIakGyRtlfRETtsMSXdJeiZ9n562S9I/pXk/Jum1Odt8OF3/mfRZ6OOWpGMl3SNpjaTVkv4qbZ+0eUuqkvQ7SY+mOf9d2j5f0oNpbremoz0jqTKdX5cuPyFnX0vT9rWSfr8wGQ2PpFJJ/ynpZ+n8ZM/3OUmPS3pE0qq0rXD/riN9HGAxvkhGw10PLAAqgEeB0wod1yjyeQPwWuCJnLavAFel01cB/yudfgfwC5KHfZ0LPJi2zwA2pO/T0+nphc5tkJznAK9Np6eSPHHxtMmcdxp7bTpdDjyY5vID4PK0/ZvAf0un/xz4Zjp9Oclz70n/To8ClcD89P+F0kLnN0jenwa+D/wsnZ/s+T4HNOS1FezfdbH3LPqeEx4RHUDvc8InpIi4l2So91yXkjyBkPT9D3Pab4rEA8C09Pkifc8/j4hdQO/zz8eliHgpIh5Op/eRPBNlLpM47zT21nS2PH0FyVMmf5i25+fc+7f4IfAWSUrbb4mI9oh4FlhH8v/EuCNpHvBfSB6SRhr/pM13EAX7d13sxWJYz/qe4Joi4qV0egvQlE4PlPuE/Zukhxt+j+SX9qTOOz0k8wiwleQLYD2wOyK60lVy4+/LLV2+B5jJxMr5WuC/Az3p/Ewmd76Q/AC4U9JDkpakbQX7d13oZ3DbGIqIkDQpr5WWVAv8P+CTEbE3+SGZmIx5R0Q3cJakacCPgVMLHFJmJP0BsDUiHpK0uNDxjKELImKzpFnAXZKeyl041v+ui71nUQzP+n457Y72PsZ2a9o+UO4T7m8iqZykUPx7RPwobZ70eQNExG7gHuA8kkMPvT8Ac+Pvyy1dXg/sYOLkfD5wiaTnSA4Vvxn4OpM3XwAiYnP6vpXkB8EiCvjvutiLxZDPCZ8ElgO9V0B8GPhpTvuH0qsozgX2pN3bCfX88/RY9LeBJyPi/+QsmrR5S2pMexRIqgYuIjlXcw/wnnS1/Jx7/xbvAe6O5OzncuDy9Oqh+cBC4Hdjk8XwRcTSiJgXESeQ/D96d0S8n0maL4CkGklTe6dJ/j0+QSH/XRf6jH+hXyRXETxNcsz3c4WOZ5S53Ay8BHSSHJv8U5Jjtb8CngF+CcxI1xWwLM37caA5Zz9/QnLybx3wkULnNUTOF5Ac230MeCR9vWMy5w2cCfxnmvMTwNVp+wKSL791wP8FKtP2qnR+Xbp8Qc6+Ppf+LdYCby90bsPIfTGHroaatPmmuT2avlb3fjcV8t+1h/swM7MhFfthKDMzGwYXCzMzG5KLhZmZDcnFwszMhuRiYWZmQ3KxsAlF0sx0FM5HJG2RtDlnvmKY+/g3SacMsc5fSHr/UYr5t5LOklSiozyysaQ/kTQ7Z37I3MxGwpfO2oQl6W+B1oj433ntIvm33dPvhmNM0m+BK0nuidgeEdOOcPvSSIb3GHDfEfHI6CM1G5h7FjYpSDpJyTMt/p3kJqY5kq6TtErJMx+uzlm395d+maTdkr6s5NkQ96fj8CDpS5I+mbP+l5U8Q2KtpNen7TWS/l/6uT9MP+usQcL8MjA17QXdlO7jw+l+H5H0L2nvozeuayU9BiyS9HeSVkp6QtI30zt13wucBdza27PqzS3d9weUPA/hCUn/M20bLOfL03UflXTPUf5PZBOci4VNJqcCX4uI0yIZV+eqiGgGXgNcJOm0frapB34dEa8B7ie527U/iohFwGeB3sLzl8CWiDgN+CLJiLeDuQrYFxFnRcSHJL0aeBfw+og4i2Rgz8tz4ro3Is6MiPuBr0fE64Az0mUXR8StJHesvzfdZ0dfsMmQ3l8C3pTGdb6SAfkGy/kLwFvS9ncNkYsVGRcLm0zWR8SqnPn3SXoYeBh4FcnDb/IdiIhfpNMPAScMsO8f9bPOBSQD2xERvcMyHIm3Aq8DVikZbvyNwInpsg6SweN6vUXS70iGf3gjcPoQ+z6HZEyk7RHRSfLQoDekywbK+T7gJkkfxd8NlsdDlNtk0tY7IWkh8FfAoojYLel7JGMG5evIme5m4P8n2oexzpEScENE/M1hjclIqQeid9AfaQrwDZInAm6W9CX6z2W4Bsr5YyRF5g+AhyX9XiQPzDHzrwebtOqAfcBeHXpi2NF2H3AZgKQz6L/n0ifSB/Xo0LDavwQuk9SQts+UdFw/m1aTPPRnezoS6btzlu0jeZxsvgeBN6X77D289esh8lkQyVPW/gbYxfh+MJCNMfcsbLJ6GFgDPAU8T/LFfrT9M8lhmzXpZ60heSrbYL4NPCZpVXre4u+AX0oqIRkt+M+AF3M3iIgdkm5M9/8SSSHo9W/A9ZIOkPOI0IjYJOlvgBaSHsxtEfHznELVn68pGbpbwJ0R8cQQuVgR8aWzZiOUfvGWRcTB9LDXncDCOPSoT7NJwz0Ls5GrBX6VFg0BH3ehsMnKPQszMxuST3CbmdmQXCzMzGxILhZmZjYkFwszMxuSi4WZmQ3p/wPKnDLWl/tfQQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.grid()\n",
    "plt.plot(range(num_iterations),cost)\n",
    "\n",
    "plt.title('Cost Function')\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can notice, the loss decreases over the training iterations. \n",
    "\n",
    "\n",
    "Thus, we have learned how to build a neural network from scratch in the next chapter we will one of the popularly used deep learning libraries called TensorFlow. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}