mlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# A Neural Network in just a few Lines of Python Code\n",
    "## Design a Feed Forward Neural Network with Backpropagation Step by Step with real Numbers\n",
    "\n",
    "For alot of people neural networks are kind of a black box. And alot of people feel uncomfortable with this situation. Me, too. \n",
    "\n",
    "That is, why I tried to follow the data processes inside a neural network step by step with real numbers. Thanks to the awesome tutorials of <a href=\"https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/\">Matt Mazur</a>, <a href=\"http://iamtrask.github.io/2015/07/27/python-network-part2/\">I am Trask</a> and the <a href=\"http://galaxy.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html\">Virtual Laboratory of Artificial Intelligence in Cracow</a> I got a good feeling of the functionality of feed forward neural networks with backpropagation. With their inspiration I created this tutorial and combined math and code, to comprehend each step smoothly.\n",
    "\n",
    "Today we will extend our artifical neuron, our perceptron, from the first part of this machine learning series. To solve non-linear classification problems, we need to combine this neuron to a network of neurons.\n",
    "\n",
    "<img width=\"400\" src=\"mlp_graph_latex_full.png\" />\n",
    "\n",
    "In the above picture you can see such a Multi Layer Perceptron (MLP) with one input layer, one hidden layer and one output layer. \n",
    "\n",
    "- The input layer represents the data set, each sample has three features ($x_0,x_1,x_2$)\n",
    "- The hidden layer consists of five neurons ($h_1,h_2,h_3,h_4,h_5$)\n",
    "- The output layer consists of one neuron ($o$).\n",
    "\n",
    "## Give Me the Code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "def sigmoid(x, derive=False):\n",
    "    if derive:\n",
    "        return x * (1 - x)\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "# Define the data set XOR\n",
    "X = np.array([\n",
    "    [1, 1, 1],\n",
    "    [1, 0, 1],\n",
    "    [0, 1, 1],\n",
    "    [0, 0, 1],\n",
    "])\n",
    "\n",
    "y = np.array([[0],\n",
    "              [1],\n",
    "              [1],\n",
    "              [0]\n",
    "             ])\n",
    "\n",
    "# Define a learning rate\n",
    "eta = 3\n",
    "# Define the number of epochs for learning\n",
    "epochs = 20000\n",
    "\n",
    "# Initialize the weights with random numbers\n",
    "w01 = np.random.random((len(X[0]), 5))\n",
    "w12 = np.random.random((5, 1))\n",
    "\n",
    "# Start feeding forward and backpropagate *epochs* times.\n",
    "for epoch in range(epochs):\n",
    "    # Feed forward\n",
    "    z_h = np.dot(X, w01)\n",
    "    a_h = sigmoid(z_h)\n",
    "\n",
    "    z_o = np.dot(a_h, w12)\n",
    "    a_o = sigmoid(z_o)\n",
    "\n",
    "    # Calculate the error\n",
    "    a_o_error = ((1 / 2) * (np.power((a_o - y), 2)))\n",
    "\n",
    "    # Backpropagation\n",
    "    ## Output layer\n",
    "    delta_a_o_error = a_o - y\n",
    "    delta_z_o = sigmoid(a_o,derive=True)\n",
    "    delta_w12 = a_h\n",
    "    delta_output_layer = np.dot(delta_w12.T,(delta_a_o_error * delta_z_o))\n",
    "\n",
    "    ## Hidden layer\n",
    "    delta_a_h = np.dot(delta_a_o_error * delta_z_o, w12.T)\n",
    "    delta_z_h = sigmoid(a_h,derive=True)\n",
    "    delta_w01 = X\n",
    "    delta_hidden_layer = np.dot(delta_w01.T, delta_a_h * delta_z_h)\n",
    "\n",
    "    w01 = w01 - eta * delta_hidden_layer\n",
    "    w12 = w12 - eta * delta_output_layer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Short Description of the Feed Forward Algorithm\n",
    "\n",
    "The input values $x_0-x_2$ are now emited into the network. \n",
    "\n",
    "- Each neuron in the hidden layer $h_1-h_5$ gets the weighted sum of the input values as input $ z_{h_j} = \\sum_{i=0}^3 x_i*w_{ji} $. \n",
    "- Each neuron calculates an output by using an activation function $a_h = \\sigma(z_h)$.\n",
    "- The same procedure is for the output layer neuron $O$. The output layer gets the weighted sum of the output values of the hidden layer $z_o = \\sum_{i=1}^5 h_i*w_{1i}$.\n",
    "- The output layer neuron calculates an output by using an activation function $a_o = \\sigma(z_o)$. This is our final classification result.\n",
    "\n",
    "Typical activation functions for neural networks are sigmoid, ReLU or tanh. In our case we will use <a href=\"https://en.wikipedia.org/wiki/Sigmoid_function\">sigmoid</a>.\n",
    "\n",
    "$$\n",
    "\\sigma(x) = \\frac{1}{1+e^{-x}}\n",
    "$$\n",
    "\n",
    "Let us code the sigmoid function in python using numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x, derive=False):\n",
    "    if derive:\n",
    "        return x * (1 - x)\n",
    "    return 1 / (1 + np.exp(-x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>3</b>: The derivate of the sigmoid function. This we need later in the backpropagation.<br>\n",
    "    line <b>4</b>: The sigmoid function as defined above.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Sounds tough? Let us define some toy data set, to examplify this functionality with real numbers step by step.\n",
    "\n",
    "The general structure of the tutorial is the following:\n",
    "\n",
    "- description of the mathematical process\n",
    "- translation of the math into python code\n",
    "- short description of the code in green boxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Our Ingredients \n",
    "\n",
    "First we will import numpy to easily manage linear algebra and calculus operations in python. To plot the learning progress later on, we will use matplotlib. The third line just allows matplotlib to plot the graphs directly in this jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Our Data Set\n",
    "\n",
    "This time we want to classify non-linearly seperable problems, so we need to define a new data set. Typically toy data here is the XOR function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| $x_0$        | $x_1$           | $y$  |\n",
    "| -            |                 |      |\n",
    "| $1$          | $1$             | $0$  |\n",
    "| $1$          | $0$             | $1$  |\n",
    "| $0$          | $1$             | $1$  |\n",
    "| $0$          |$0$              | $0$  |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X = np.array([\n",
    "    [1, 1, 1],\n",
    "    [1, 0, 1],\n",
    "    [0, 1, 1],\n",
    "    [0, 0, 1],\n",
    "])\n",
    "\n",
    "y = np.array([[0],\n",
    "              [1],\n",
    "              [1],\n",
    "              [0]\n",
    "             ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1-6</b>: Initialize the four samples of the XOR function.<br>\n",
    "    line <b>8-12</b> Define the labels of each sample.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This small toy data set contains two samples labeled with $0$ and two samples labeled with $+1$. This means we have a binary classification problem, as the data set contains two sample classes.\n",
    "\n",
    "Like in the last tutorials we folded a bias term of $1$ into our samples.\n",
    "\n",
    "Let us visualize the data set to clarify, that there is no way, to seperate the data with a linear hyperplane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<matplotlib.axis.YTick at 0x7fc5a24726d8>,\n",
       "  <matplotlib.axis.YTick at 0x7fc5a2472dd8>],\n",
       " <a list of 2 Text yticklabel objects>)"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWcAAAD7CAYAAAC2a1UBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAABmZJREFUeJzt2zGrHPcVxuFzhPCXcIgaYwKBlCblQgqrc6uU7t075bbp\n0wp3RrXTudrCuFFnCBZyZVsYXPkDGHNS6EqRlZV2JPbefXfneWDh7jDMPZfL/Hb2v7M9MwVAllun\nHgCA/yfOAIHEGSCQOAMEEmeAQOIMEOj2sQ7U3e7JA3hDM9P7th8tzle/5JiHuzHb7ba22+2pxwDe\nwjmfv917u1xVljUAIokzQCBxrqrNZnPqEYC3dKnnbx9rnbi751zXnAFOobtf+YGgK2eAQOIMEEic\nAQKJM0AgcQYIJM4AgcQZIJA4AwQSZ4BA4gwQSJwBAokzQCBxBggkzgCBxBkgkDgDBBJngEDiDBBI\nnAECiTNAoNXHufvpAyDJwTh39/3u/rm7v7mJgQBYduX8WVV9eN2DAPA/B+M8M19V1S83MAsAV26f\neoCb9qr15Ze3z1z/LMCbWfr50CWcv0eN83a7ff7zZrOpzWZzzMMDnLXdble73W7Rvj0LXmK6+05V\n/Xtm/vKafWbJsdI8eyU+w9GBM9fdNTN73w8svZWurx4A3IAlt9J9XlVfV9X73f1Dd398/WMBrNui\nZY1FB7KsAfBGXressbq7NV4mykCi1X99GyCROAMEEmeAQOIMEEicAQKJM0AgcQYIJM4AgcQZIJA4\nAwQSZ4BA4gwQSJwBAokzQCBxBggkzgCBxBkgkDgDBBJngEDiDBBInAECiTNAIHEGCCTOAIHEGSCQ\nOAMEEmeAQOIMEEicAQKJM0AgcQYIJM4AgcQZIJA4AwQSZ4BA4gwQSJwBAokzQCBxBggkzgCBxBkg\nkDgDBBJngEDiDBBInAECiTNAIHEGCCTOAIHEGSCQOAMEEmeAQOIMEEicAQKJM0AgcQYIJM4AgcQZ\nIJA4AwQSZ4BA4gwQSJwBAokzQCBxBggkzgCBxBkgkDgDBBJngEDiDBBInAECiTNAIHEGCCTOAIHE\nGSCQOAMEEmeAQOIMEEicAQKJM0AgcQYIJM4AgcQZIJA4AwQSZ4BA4gwQSJwBAokzQCBxBggkzgCB\nxBkgkDgDBBJngEDiDBBInAECiTNAIHEGCCTOAIHEGSCQOAMEEmeAQOIMEEicAQKJM0AgcQYIJM4A\ngcQZIJA4AwQSZ4BA4gwQSJwBAokzQCBxBggkzgCBxBkgkDgDBBJngEDiDBBInAECiTNAIHEGCCTO\nAIHEGSCQOAMEEmeAQOIMEEicAQKJM0AgcQYIJM4AgcQZIJA4AwQSZ4BA4gwQSJwBAokzQCBxBggk\nzgCBxBkgkDgDBBJngEDiDBBInAECiTNAIHEGCCTOAIHEGSCQOAMEEmeAQOIMEEicAQKJM3C2up8+\nLtGiOHf33e5+1N2Pu/vT6x4KYO16Zl6/Q/etqnpcVX+rqp+q6mFV3ZuZRy/tN4eOBXBMz66azzU9\n3V0zs/faf8mV8wdV9d3MfD8zv1bVg6r66JgDAvB7S+L8blX9+MLzJ1fbztKzNapDDyDPq87TSzx/\nby/YZ9+fuvdNxHa7ff7zZrOpzWbzVkMBXKLdble73W7RvkvWnP9aVduZuXv1/B9VNTPzz5f2s+YM\n3Ki1rzk/rKr3uvtOd79TVfeq6otjDgjA7x1c1piZ37r7k6r6sp7G/P7MfHvtkwGs2MFljcUHsqwB\n3LBLXtZY8oEgQKRzjfISvr4NEEicAQKJM0AgcQYIJM4AgcQZIJA4AwQSZ4BA4gwQSJwBAokzQCBx\nBggkzgCBxBkgkDgDBBJngEDiDBBInAECiTNAIHEGCCTOAIHEGSCQOFfVbrc79QjAW7rU81ec63L/\nubAGl3r+ijNAIHEGCNQzc5wDdR/nQAArMjO9b/vR4gzA8VjWAAgkzgCBVh/n7r7b3Y+6+3F3f3rq\neYDDuvt+d//c3d+cepbrsuo4d/etqvpXVX1YVX+uqr93959OOxWwwGf19Ly9WKuOc1V9UFXfzcz3\nM/NrVT2oqo9OPBNwwMx8VVW/nHqO67T2OL9bVT++8PzJ1TaAk1p7nPfdX+jeQuDk1h7nJ1X1xxee\n/6GqfjrRLADPrT3OD6vqve6+093vVNW9qvrixDMBy3Ttf/d7EVYd55n5rao+qaovq+o/VfVgZr49\n7VTAId39eVV9XVXvd/cP3f3xqWc6Nl/fBgi06itngFTiDBBInAECiTNAIHEGCCTOAIHEGSCQOAME\n+i8qturJplQgzAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc5a24ca710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for d, sample in enumerate(X):\n",
    "    # Plot the positive samples\n",
    "    if d % 3 == 0:\n",
    "        plt.scatter(sample[0], sample[1], s=100, marker='_', linewidths=2)\n",
    "    # Plot the negative samples\n",
    "    else:\n",
    "        plt.scatter(sample[0], sample[1], s=90, marker='+', linewidths=2)\n",
    "\n",
    "# Set the interval in axis\n",
    "plt.xticks([0,1])\n",
    "plt.yticks([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With the data set defined, we can now calculate the output using our neural network from the introduction.\n",
    "\n",
    "First we need to make some preassumptions. \n",
    "\n",
    "As initial weight values we will use $1$. Usually neural networks use random values for initial weights, but for easy calculations, here we go with $1$. For the same reason the bias term of each neuron is $0$.\n",
    "Furthermore we will set our learning rate $eta$ to $3$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Note: Before you run this code for real use, change the np.ones to np.random.random!\n",
    "# Otherwise the neural network might not converge.\n",
    "w01 = np.ones((len(X[0]), 5))\n",
    "w12 = np.ones((5, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Initialize the hidden layer weights with ones. Size of hidden layer is $5$.<br>\n",
    "    line <b>2</b> Initialize the output layer weights with ones. Size of output layer is $1$.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " \n",
    "\n",
    "For each forward pass, we will use the whole data set for calculations. This allows us, to represent the network calculations via matrix multiplications.\n",
    "\n",
    "*You will see, that the whole feed forward pass is all about matrix multiplications, so refresh your knowledge here, before you continue.*\n",
    "\n",
    "### Feed Forward Pass\n",
    "\n",
    "Let us start with the first step in the feed forward pass:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "$$\n",
    "z_h = \n",
    "\\begin{pmatrix}\n",
    "1 & 1 & 1\\\\\n",
    "1 & 0 & 1\\\\\n",
    "0 & 1 & 1\\\\\n",
    "0 & 0 & 1\n",
    "\\end{pmatrix}\n",
    "*\n",
    "\\begin{pmatrix}\n",
    "1 & 1 & 1 & 1 & 1 \\\\\n",
    "1 & 1 & 1 & 1 & 1 \\\\\n",
    "1 & 1 & 1 & 1 & 1\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The above term is a matrix representation of the feed forward pass from the input layer to the first hidden layer. Matrix multiplication is $row * column$. There is one row representing the input sample and five columns representing the neuron connections to the five hidden layer neurons.\n",
    "\n",
    "The following matrix dimensions are involved:\n",
    "\n",
    "$4x3 * 3x5 = 4x5$\n",
    "\n",
    "The result is the following matrix, representing the values of $z_{h1}-z_{h5}$ as column vectors:\n",
    "\n",
    "$$\n",
    "z_h = \n",
    "\\begin{pmatrix}\n",
    "3 & 3 & 3 & 3 & 3 \\\\\n",
    "2 & 2 & 2 & 2 & 2 \\\\\n",
    "2 & 2 & 2 & 2 & 2 \\\\\n",
    "1 & 1 & 1 & 1 & 1\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 3.  3.  3.  3.  3.]\n",
      " [ 2.  2.  2.  2.  2.]\n",
      " [ 2.  2.  2.  2.  2.]\n",
      " [ 1.  1.  1.  1.  1.]]\n"
     ]
    }
   ],
   "source": [
    "z_h = np.dot(X, w01)\n",
    "print(z_h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the input of the hidden layer $z_h$ using matrix multiplication.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we use our sigmoid activation function for the hidden layer neurons. This will result in the following matrix representing the values of $a_{h1}-a_{h5}$:\n",
    "\n",
    "$$\n",
    "a_h = \n",
    "\\begin{pmatrix}\n",
    "0.952574 & 0.952574 & 0.952574 & 0.952574 & 0.952574 \\\\\n",
    "0.880797 & 0.880797 & 0.880797 & 0.880797 & 0.880797 \\\\\n",
    "0.880797 & 0.880797 & 0.880797 & 0.880797 & 0.880797 \\\\\n",
    "0.731059 & 0.731059 & 0.731059 & 0.731059 & 0.731058\n",
    "\\end{pmatrix}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.95257413  0.95257413  0.95257413  0.95257413  0.95257413]\n",
      " [ 0.88079708  0.88079708  0.88079708  0.88079708  0.88079708]\n",
      " [ 0.88079708  0.88079708  0.88079708  0.88079708  0.88079708]\n",
      " [ 0.73105858  0.73105858  0.73105858  0.73105858  0.73105858]]\n"
     ]
    }
   ],
   "source": [
    "a_h = sigmoid(z_h)\n",
    "print(a_h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the output of the hidden layer $a_h$ using sigmoid activation function.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The above matrix represents the output of each of the five hidden layer neurons.\n",
    "As we defined in our feed forward algorithm we will emitt this output now to the next layer as input using the weighted neuron connections of the output layer.\n",
    "\n",
    "The following matrix dimensions are involved:\n",
    "\n",
    "$4x5*5x1=4x1$\n",
    "\n",
    "\n",
    "$$\n",
    "z_o =\n",
    "\\begin{pmatrix}\n",
    "0.952574 & 0.952574 & 0.952574 & 0.952574 & 0.952574 \\\\\n",
    "0.880797 & 0.880797 & 0.880797 & 0.880797 & 0.880797 \\\\\n",
    "0.880797 & 0.880797 & 0.880797 & 0.880797 & 0.880797 \\\\\n",
    "0.731059 & 0.731059 & 0.731059 & 0.731059 & 0.731058\n",
    "\\end{pmatrix}\n",
    "*\n",
    "\\begin{pmatrix}\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This results in the following matrix, representing the value $z_o$. Each row respresents $z_o$ for one input sample:\n",
    "\n",
    "$$\n",
    "z_o = \n",
    "\\begin{pmatrix}\n",
    "4.762870 \\\\\n",
    "4.403985 \\\\\n",
    "4.403985 \\\\\n",
    "3.655293\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 4.76287063]\n",
      " [ 4.40398539]\n",
      " [ 4.40398539]\n",
      " [ 3.65529289]]\n"
     ]
    }
   ],
   "source": [
    "z_o = np.dot(a_h, w12)\n",
    "print(z_o)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the input of the output layer $z_o$ using matrix multiplication.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To get the output of the output layer we give this $4x1$ matrix to our activation function. The resulting matrix represents the values of $a_o$ for each input sample."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "$$\n",
    "a_o =\n",
    "\\begin{pmatrix}\n",
    "0.991531 \\\\\n",
    "0.987919 \\\\\n",
    "0.987919 \\\\\n",
    "0.974798\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.99153128]\n",
      " [ 0.98791922]\n",
      " [ 0.98791922]\n",
      " [ 0.97479766]]\n"
     ]
    }
   ],
   "source": [
    "a_o = sigmoid(z_o)\n",
    "print(a_o)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the output of the output layer $a_o$ using sigmoid activation function.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Calculating the Error of the Output Layer\n",
    "\n",
    "As we have the final output for our whole data set, we can now calculate the error.\n",
    "\n",
    "First we need to define our error function. As we need to derive the error function later on, we choose the squared error function:\n",
    "\n",
    "$E_{a_o}(x)= \\frac{1}{2}(f(x) - y_{x})^2$\n",
    "\n",
    "Let us calculate the error for our first inpus sample $x_o$:\n",
    "\n",
    "$$\n",
    "E_{a_o}(x_0)= \\frac{1}{2}(0.991531 - 0)^2 = 0.491567 \\\\\n",
    "$$\n",
    "\n",
    "Now we can calculate the error for the rest of the samples and write it down in matrix represantation.\n",
    "\n",
    "$$\n",
    "E a_o =\n",
    "\\begin{pmatrix}\n",
    "0.491567\\\\\n",
    "0.000073 \\\\\n",
    "0.000073 \\\\\n",
    "0.475116\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  4.91567136e-01]\n",
      " [  7.29725915e-05]\n",
      " [  7.29725915e-05]\n",
      " [  4.75115235e-01]]\n"
     ]
    }
   ],
   "source": [
    "a_o_error = ((1 / 2) * (np.power((a_o - y), 2)))\n",
    "print(a_o_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the error of the output neuron $a_o$ using squared error function.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now the magic of neural networks begins. As we calculated the error for each sample now, we want to adjust the weights in all the layers to minimize this error. This procedure is called backpropagation.\n",
    "\n",
    "*Note: The error for $x_1$ and $x_2$ is almost zero. The calculated values of $a_o$ are $0.987919$, which is close to the expected value $y_{x_1}=1$ and $y_{x_2}=1$.*\n",
    "\n",
    "### Backpropagation\n",
    "\n",
    "The backward pass takes the error and passes it backward through the whole network, to find out, how much the weights have to be adapted, to minimize the error.\n",
    "\n",
    "#### Calculate the Update Matrix for the Weights of the Output Layer\n",
    "\n",
    "Let us visualize this step:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img width=\"500\" src=\"mlp_graph_latex_backprop_1.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "$$\n",
    "\\frac{\\delta E}{\\delta w12} = \\frac{\\delta E}{\\delta a_{o}} * \\frac{\\delta a_{o}}{\\delta z_{o}} * \\frac{\\delta z_{o}}{\\delta w}\n",
    "$$\n",
    "\n",
    "Lets begin with the first backward step. We want to find out, how \n",
    "much the output of the output layer $a_o$ contributes to the error. We do this by deriving the error function with respect to the output of the output neuron $a_o$.\n",
    "\n",
    "\n",
    "$\\frac{\\delta E}{\\delta a_{o}} = 2*\\frac{1}{2}(f(x) - y)^{2-1}$\n",
    "\n",
    "$\\frac{\\delta E}{\\delta a_{o}} = f(x) - y$\n",
    "\n",
    "\n",
    "Now lets calculate the derivate for our error value from the first sample:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta E(x_0)}{\\delta a_{o}}= 0.9915317 - 0 = 0.991531 \\\\\n",
    "$$\n",
    "Now we can calculate the derivates for the error values from the rest of the data set and represent them as a $4x1$ matrix:\n",
    "$$\n",
    "\\frac{\\delta E}{\\delta a_{o}} =\n",
    "\\begin{pmatrix}\n",
    "    0.991531 \\\\\n",
    "    {-}0.012081 \\\\\n",
    "    {-}0.012081\\\\\n",
    "    0.974798\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.99153128]\n",
      " [-0.01208078]\n",
      " [-0.01208078]\n",
      " [ 0.97479766]]\n"
     ]
    }
   ],
   "source": [
    "delta_a_o_error = a_o - y\n",
    "print(delta_a_o_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the derivate of the error of the output layer.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The Derivates show us the slope (steepness) of the loss function at position x. If it is close to zero, the loss is small, higher values show a higher loss.\n",
    "\n",
    "Our next step in the backward pass is from the output of the output neuron $a_o$ to the input of the output neuron $z_o$. Here we \"cross the border\" of the sigmoid activation function. So we need the derivate of the sigmoid function:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta a_{o}}{\\delta z_{o}} = a_{o} * (1 - a_{o})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Lets calculate this with our first value $x_0$ and represent the rest in matrix representation.\n",
    "\n",
    "$$\n",
    "\\frac{\\delta a_{o}}{\\delta z_{o}} = 0.991531 * (1 - 0.991531) = 0.008398\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\delta a_{o}}{\\delta z_{o}} =\n",
    "\\begin{pmatrix}\n",
    "    0.008398 \\\\\n",
    "    0.011935 \\\\\n",
    "    0.011935 \\\\\n",
    "    0.024567\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.008397  ]\n",
      " [ 0.01193483]\n",
      " [ 0.01193483]\n",
      " [ 0.02456719]]\n"
     ]
    }
   ],
   "source": [
    "delta_z_o = sigmoid(a_o,derive=True)\n",
    "print(delta_z_o)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the derivate of the sigmoid function of the output layer.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Our final step is, how much does the input of the output neuron $z_o$ change with respect to the weights $w_{11} - w_{15}$?\n",
    "\n",
    "$$\n",
    "z_{o} = w_{11}*a_{h_1} + w_{12}*a_{h_2} + w_{13}*a_{h_3} + w_{14}*a_{h_4} + w_{15}*a_{h_5}\n",
    "$$\n",
    "\n",
    "This we need to derive for all five weights:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "$\n",
    "\\frac{\\delta z_{o}}{\\delta w_{11}} = 1 * a_{h_1} * w_{11}^{1-1} + 0 + 0 + 0 + 0 = a_{h_1}\n",
    "$ \n",
    "\n",
    "$\n",
    "\\frac{\\delta z_{o}}{\\delta w_{12}} = 0 + 1 * a_{h_2} * w_{12}^{1-1} + 0 + 0 + 0 = a_{_2}\n",
    "$\n",
    "\n",
    "$\n",
    "\\frac{\\delta z_{o}}{\\delta w_{13}} = 0 + 0 + 1 * a_{h_3} * w_{13}^{1-1} + 0 + 0 = a_{h_3}\n",
    "$\n",
    "\n",
    "$\n",
    "\\frac{\\delta z_{o}}{\\delta w_{14}} = 0 + 0 + 0 + 1 * a_{h_4} * w_{14}^{1-1} + 0= a_{h_4}\n",
    "$\n",
    "\n",
    "$\n",
    "\\frac{\\delta z_{o}}{\\delta w_{15}} = 0 + 0 + 0 + 0 + 1 * a_{h_5} * w_{15}^{1-1} = a_{h_5}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The values for $a_{h}$ we already calculated in a previous step for all our samples. so we can copy them:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta z_{o}}{\\delta w} =\n",
    "\\begin{pmatrix}\n",
    "0.952574 & 0.952574 & 0.952574 & 0.952574 & 0.952574 \\\\\n",
    "0.880797 & 0.880797 & 0.880797 & 0.880797 & 0.880797 \\\\\n",
    "0.880797 & 0.880797 & 0.880797 & 0.880797 & 0.880797 \\\\\n",
    "0.731059 & 0.731059 & 0.731059 & 0.731059 & 0.731058\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.95257413  0.95257413  0.95257413  0.95257413  0.95257413]\n",
      " [ 0.88079708  0.88079708  0.88079708  0.88079708  0.88079708]\n",
      " [ 0.88079708  0.88079708  0.88079708  0.88079708  0.88079708]\n",
      " [ 0.73105858  0.73105858  0.73105858  0.73105858  0.73105858]]\n"
     ]
    }
   ],
   "source": [
    "delta_w12 = a_h\n",
    "print(a_h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the derivate of the input of the output layer $z_o$ with respect to the weights $w$.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we can combine this all and calculate the gradients for our last layer in the MLP for each weight:\n",
    "\n",
    "*Note: As we use matrix multiplication, we use transpose of matrices, to fit matrix dimensions. Furthermore we use element wise multiplication, respectively hadamard product, to calculate our update matrix. Click <a href=\"https://en.wikipedia.org/wiki/Hadamard_product_(matrices)\">here</a>, for more informations*\n",
    "\n",
    "$$\n",
    "\\frac{\\delta E}{\\delta w} = \\frac{\\delta z_{o}}{\\delta w}.T * (\\frac{\\delta E}{\\delta a_{o}} * \\frac{\\delta a_{o}}{\\delta z_{o}})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The terms $\\frac{\\delta E}{\\delta a_{o}} and \\frac{\\delta a_{o}}{\\delta z_{o}}$ are multiplied using the hadamard product. We get a $4x1$ matrice. This is the second factor of the matrix multiplication with the first term $\\frac{\\delta z_{o}}{\\delta w}$ which we transpose to get a $5x4$ matrix.\n",
    "    \n",
    "$$\n",
    "\\frac{\\delta E}{\\delta w} =\n",
    "\\begin{pmatrix}\n",
    "0.952574 & 0.952574 & 0.952574 & 0.952574 & 0.952574 \\\\\n",
    "0.880797 & 0.880797 & 0.880797 & 0.880797 & 0.880797 \\\\\n",
    "0.880797 & 0.880797 & 0.880797 & 0.880797 & 0.880797 \\\\\n",
    "0.731059 & 0.731059 & 0.731059 & 0.731059 & 0.731058\n",
    "\\end{pmatrix}.T\n",
    "*\n",
    "\\begin{pmatrix}\n",
    "\\begin{pmatrix}\n",
    "    0.991531 \\\\\n",
    "    - 0.012081 \\\\\n",
    "    - 0.012081 \\\\\n",
    "    0.974798\n",
    "\\end{pmatrix}\n",
    "\\circ\n",
    "\\begin{pmatrix}\n",
    "    0.008398 \\\\\n",
    "    0.011935 \\\\\n",
    "    0.011935 \\\\\n",
    "    0.024567\n",
    "\\end{pmatrix}\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The result is a $5x1$ matrice. If we calculate this, we will get the following gradient for the last layer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "$$\n",
    "\\frac{\\delta E}{\\delta w} = \n",
    "\\begin{pmatrix}\n",
    "0.025184 \\\\\n",
    "0.025184 \\\\\n",
    "0.025184 \\\\\n",
    "0.025184 \\\\\n",
    "0.025184 \\\\\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.02518446]\n",
      " [ 0.02518446]\n",
      " [ 0.02518446]\n",
      " [ 0.02518446]\n",
      " [ 0.02518446]]\n"
     ]
    }
   ],
   "source": [
    "delta_output_layer = np.dot(delta_w12.T,(delta_a_o_error * delta_z_o))\n",
    "print(delta_output_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the update matrix for the output layer using matrix multiplication and hadamard product.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will use this matrix, to update our weight vectors $w_{10}-w_{15}$ in the last part of this tutorial.\n",
    "\n",
    "#### Calculate the Update Matrix for the Weights of the Hidden Layer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As we calculated now the update vector for out weights $w_{11}-w_{15}$, we need to follow our backpass now further and calculate the updates for our weights $w_{01} - w_{10}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "$$\n",
    "\\frac{\\delta E}{\\delta w01} = \\frac{\\delta E}{\\delta a_{h}} * \\frac{\\delta a_h}{\\delta z_h} * \\frac{\\delta z_{h}}{\\partial w}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Ok, lets visualize this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img width=\"600\" src=\"mlp_graph_latex_backprop_2.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The first step:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta E}{\\delta a_{h}} = \\frac{\\delta E}{\\delta z_{o}} * \\frac{\\delta z_{o}}{\\delta a_{h}}\n",
    "$$\n",
    "\n",
    "The first term:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta E}{\\delta z_{o}} = \\frac{\\delta E}{\\delta a_{o}} * \\frac{\\delta a_{o}}{\\delta z_{o}}\n",
    "$$\n",
    "\n",
    "These values we know already from the last steps:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta a_{o}}{\\delta z_{o}} = \\begin{pmatrix}\n",
    "    0.008398 \\\\\n",
    "    0.011935 \\\\\n",
    "    0.011935 \\\\\n",
    "    0.024567\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "and\n",
    "\n",
    "$$\n",
    "\\frac{\\delta E}{\\delta a_{o}} = \\begin{pmatrix}\n",
    "    0.991531 \\\\\n",
    "    - 0.012081 \\\\\n",
    "    - 0.012081 \\\\\n",
    "    0.974798\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "Multiplying them with hadamard product will produce:\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "    0.008327\\\\\n",
    "    - 0.000144\\\\\n",
    "    - 0.000144\\\\\n",
    "    0.023948\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "Now we can calculate the second term:\n",
    "\n",
    "$$\n",
    "z_o = w_{11}*a_{hidden} + w_{12}*a_{hidden} + w_{13}*a_{hidden} + w_{14}*a_{hidden} + w_{15}*a_{hidden}\\\\\n",
    "\\frac{\\delta z_{o}}{\\delta a_{h}} = w\n",
    "$$\n",
    "\n",
    "This gives us the matrix of the second weight vector:\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "Now we can calculate the full equation $\\frac{\\delta E}{\\delta a_{h}}$. We have a $4x1$ matrix and a $5x1$ matrix, so we need to transpose the second term, so we get a $4x1 * 1x5 = 4x5$ matrice:\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "0.008327 \\\\\n",
    "- 0.000144 \\\\\n",
    "- 0.000144 \\\\\n",
    "0.023948\n",
    "\\end{pmatrix}\n",
    "*\n",
    "\\begin{pmatrix}\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "\\end{pmatrix}.T\n",
    "=\n",
    "\\begin{pmatrix}\n",
    "0.008326 & 0.008326 & 0.008326 & 0,008326 & 0,008326 \\\\\n",
    "- 0.000144 & - 0.000144 & - 0.000144 & - 0.000144 & - 0.000144 \\\\\n",
    "- 0.000144 & - 0.000144 & - 0.000144 & - 0.000144 & - 0.000144 \\\\\n",
    "0.023948 & 0.023948 & 0.023948 & 0.023948 & 0.023948 \n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.00832589  0.00832589  0.00832589  0.00832589  0.00832589]\n",
      " [-0.00014418 -0.00014418 -0.00014418 -0.00014418 -0.00014418]\n",
      " [-0.00014418 -0.00014418 -0.00014418 -0.00014418 -0.00014418]\n",
      " [ 0.02394804  0.02394804  0.02394804  0.02394804  0.02394804]]\n"
     ]
    }
   ],
   "source": [
    "delta_a_h = np.dot(delta_a_o_error * delta_z_o, w12.T)\n",
    "print(delta_a_h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the derivate of the Error function $E$ with respect to the output of the hidden layer $a_h$.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "\n",
    "Next we need to calculate\n",
    "\n",
    "$$\n",
    "\\frac{\\delta a_h}{\\delta z_h} = a_h * (1-a_h)\n",
    "$$\n",
    "\n",
    "Lets write this into matrix representation for all samples and all hidden nodes\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "0.045177 & 0.045177 & 0.045177 & 0.045177 & 0.045177 \\\\\n",
    "0.104993 & 0.104993 & 0.104993 & 0.104993 & 0.104993 \\\\\n",
    "0.104993 & 0.104993 & 0.104993 & 0.104993 & 0.104993 \\\\\n",
    "0.196612 & 0.196612 & 0.196612 & 0.196612 & 0.196612\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.04517666  0.04517666  0.04517666  0.04517666  0.04517666]\n",
      " [ 0.10499359  0.10499359  0.10499359  0.10499359  0.10499359]\n",
      " [ 0.10499359  0.10499359  0.10499359  0.10499359  0.10499359]\n",
      " [ 0.19661193  0.19661193  0.19661193  0.19661193  0.19661193]]\n"
     ]
    }
   ],
   "source": [
    "delta_z_h = sigmoid(a_h,derive=True)\n",
    "print(delta_z_h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the derivate of the sigmoid of the hidden layer.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally we calculate:\n",
    "$$\n",
    "z_{h} = x_0 * w + x_1 * w\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\delta z_{h}}{\\partial w} = x \n",
    "$$\n",
    "\n",
    "This is just our data set $X$:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta z_{h}}{\\partial w} = \n",
    "\\begin{pmatrix}\n",
    "1 & 1 & 1\\\\\n",
    "1 & 0 & 1\\\\\n",
    "0 & 1 & 1\\\\\n",
    "0 & 0 & 1\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 1 1]\n",
      " [1 0 1]\n",
      " [0 1 1]\n",
      " [0 0 1]]\n"
     ]
    }
   ],
   "source": [
    "delta_w01 = X\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the derivate of the input of the hidden layer $z_h$ with respect to the weight matrix $w01$.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, as we calculated all terms of $\\frac{\\delta E}{\\delta w}$ we can multiply them all to get the update matrix.\n",
    "\n",
    "As we have a $4x3 * 4x5$ structure, we need to transpose the matrix representing our input data $X$, to do matrix multiplication.\n",
    "\n",
    "$$\n",
    "\\frac{\\delta E}{\\delta w} =\n",
    "\\begin{pmatrix}\n",
    "1 & 1 & 1\\\\\n",
    "1 & 0 & 1\\\\\n",
    "0 & 1 & 1\\\\\n",
    "0 & 0 & 1\n",
    "\\end{pmatrix}.T\n",
    "*\n",
    "\\begin{pmatrix}\n",
    "\\begin{pmatrix}\n",
    "0.008327 & 0.008327 & 0.008327 & 0,008327 & 0,008327 \\\\\n",
    "- 0.000144 & - 0.000144 & - 0.000144 & - 0.000144 & - 0.000144 \\\\\n",
    "- 0.000144 & - 0.000144 & - 0.000144 & - 0.000144 & - 0.000144 \\\\\n",
    "0.023948 & 0.023948 & 0.023948 & 0.023948 & 0.023948 \n",
    "\\end{pmatrix}\n",
    "\\circ\n",
    "\\begin{pmatrix}\n",
    "0.045177 & 0.045177 & 0.045177 & 0.045177 & 0.045177 \\\\\n",
    "0.104993 & 0.104993 & 0.104993 & 0.104993 & 0.104993 \\\\\n",
    "0.104993 & 0.104993 & 0.104993 & 0.104993 & 0.104993 \\\\\n",
    "0.196612 & 0.196612 & 0.196612 & 0.196612 & 0.196612\n",
    "\\end{pmatrix}\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "After solving the hadamard product, we get:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta E}{\\delta w} = \n",
    "\\begin{pmatrix}\n",
    "1 & 1 & 1\\\\\n",
    "1 & 0 & 1\\\\\n",
    "0 & 1 & 1\\\\\n",
    "0 & 0 & 1\n",
    "\\end{pmatrix}.T\n",
    "*\n",
    "\\begin{pmatrix}\n",
    "0.000376 & 0.000376 & 0.000376 & 0.000376 & 0.000376 \\\\\n",
    "- 0.000015 & - 0.000015 & - 0.000015 & - 0.000015 & - 0.000015 \\\\\n",
    "- 0.000015 & - 0.000015 & - 0.000015 & - 0.000015 & - 0.000015 \\\\\n",
    "0.004708 & 0.004708 & 0.004708 & 0.004708 & 0.004708 \\\\\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The final $3x5$ update matrix will be:\n",
    "\n",
    "$$\n",
    "\\frac{\\delta E}{\\delta w} = \n",
    "\\begin{pmatrix}\n",
    "0.000361 & 0.000361 & 0.000361 & 0.000361 & 0.000361 \\\\\n",
    "0.000361 & 0.000361 & 0.000361 & 0.000361 & 0.000361 \\\\\n",
    "0.00505433 & 0.00505433 & 0.00505433 & 0.00505433 & 0.00505433 \\\\\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.000361    0.000361    0.000361    0.000361    0.000361  ]\n",
      " [ 0.000361    0.000361    0.000361    0.000361    0.000361  ]\n",
      " [ 0.00505433  0.00505433  0.00505433  0.00505433  0.00505433]]\n"
     ]
    }
   ],
   "source": [
    "delta_hidden_layer = np.dot(delta_w01.T, delta_a_h * delta_z_h)\n",
    "print(delta_hidden_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div style=\"background-color: lightgreen; padding: 1em\">\n",
    "    <i>\n",
    "    line <b>1</b>: Calculate the update matrix for the hidden layer using matrix multiplication and hadamard product.\n",
    "    </i>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Update the Weight Matrices\n",
    "\n",
    "Now, that we calculated the update matrices, we can update our weight matrices $w01$ and $w12$.\n",
    "\n",
    "$$\n",
    "w12 = w12 - (eta * \\frac{\\delta E}{\\delta w12})\n",
    "$$\n",
    "\n",
    "$$\n",
    "w01 = w01 - (eta * \\frac{\\delta E}{\\delta w01})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Lets first calculate the weight matrix for the output layer\n",
    "\n",
    "$$\n",
    "w12 = \n",
    "\\begin{pmatrix}\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "1 \\\\\n",
    "\\end{pmatrix}\n",
    "- (3*\n",
    "\\begin{pmatrix}\n",
    "0.025184 \\\\\n",
    "0.025184 \\\\\n",
    "0.025184 \\\\\n",
    "0.025184 \\\\\n",
    "0.025184 \\\\\n",
    "\\end{pmatrix}\n",
    ") =\n",
    "\\begin{pmatrix}\n",
    "0.92444663 \\\\\n",
    "0.92444663 \\\\\n",
    "0.92444663 \\\\\n",
    "0.92444663 \\\\\n",
    "0.92444663\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we can calculate our new weight matrix for the hidden layer:\n",
    "    \n",
    "$$\n",
    "w01 = \n",
    "\\begin{pmatrix}\n",
    "1 & 1 & 1 & 1 & 1 \\\\\n",
    "1 & 1 & 1 & 1 & 1 \\\\\n",
    "1 & 1 & 1 & 1 & 1\n",
    "\\end{pmatrix}\n",
    "- (3*\n",
    "\\begin{pmatrix}\n",
    "0.000361 & 0.000361 & 0.000361 & 0.000361 & 0.000361 \\\\\n",
    "0.000361 & 0.000361 & 0.000361 & 0.000361 & 0.000361 \\\\\n",
    "0.00505433 & 0.00505433 & 0.00505433 & 0.00505433 & 0.00505433 \\\\\n",
    "\\end{pmatrix} = \n",
    "\\begin{pmatrix}\n",
    "0.99891701 & 0.99891701 & 0.99891701 & 0.99891701 & 0.99891701 \\\\\n",
    "0.99891701 & 0.99891701 & 0.99891701 & 0.99891701 & 0.99891701 \\\\\n",
    "0.98483701 & 0.98483701 & 0.98483701 & 0.98483701 & 0.98483701\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Voila, these are our new weight matrices. \n",
    "\n",
    "### Make a Prediction with updated Weight Matrices\n",
    "\n",
    "Let us classify our data set with the new weights.\n",
    "\n",
    "$$\n",
    "a_h = \\sigma (\n",
    "\\begin{pmatrix}\n",
    "1 & 1 & 1\\\\\n",
    "1 & 0 & 1\\\\\n",
    "0 & 1 & 1\\\\\n",
    "0 & 0 & 1\n",
    "\\end{pmatrix}\n",
    "*\n",
    "\\begin{pmatrix}\n",
    "0.99891701 & 0.99891701 & 0.99891701 & 0.99891701 & 0.99891701 \\\\\n",
    "0.99891701 & 0.99891701 & 0.99891701 & 0.99891701 & 0.99891701 \\\\\n",
    "0.98483701 & 0.98483701 & 0.98483701 & 0.98483701 & 0.98483701\n",
    "\\end{pmatrix} )\n",
    "=\n",
    "\\begin{pmatrix}\n",
    "0.95178509  & 0.95178509 & 0.95178509 & 0.95178509 & 0.95178509 \\\\\n",
    "0.87908077  & 0.87908077 & 0.87908077 & 0.87908077 & 0.87908077 \\\\\n",
    "0.87908077  & 0.87908077 & 0.87908077 & 0.87908077 & 0.87908077 \\\\\n",
    "0.72806693  & 0.72806693 & 0.72806693 & 0.72806693 & 0.72806693\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "a_o = \\sigma(\n",
    "\\begin{pmatrix}\n",
    "0.95178509  & 0.95178509 & 0.95178509 & 0.95178509 & 0.95178509 \\\\\n",
    "0.87908077  & 0.87908077 & 0.87908077 & 0.87908077 & 0.87908077 \\\\\n",
    "0.87908077  & 0.87908077 & 0.87908077 & 0.87908077 & 0.87908077 \\\\\n",
    "0.72806693  & 0.72806693 & 0.72806693 & 0.72806693 & 0.72806693\n",
    "\\end{pmatrix}\n",
    "*\n",
    "\\begin{pmatrix}\n",
    "0.92444663 \\\\\n",
    "0.92444663 \\\\\n",
    "0.92444663 \\\\\n",
    "0.92444663 \\\\\n",
    "0.92444663\n",
    "\\end{pmatrix})\n",
    "=\n",
    "\\begin{pmatrix}\n",
    "0.98786405 \\\\\n",
    "0.98309866 \\\\\n",
    "0.98309866 \\\\\n",
    "0.96660214\n",
    "\\end{pmatrix})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Did we improve a bit in comparison to the first prediction? Next we compare the expected class $y$ with the predicted class $a_o$.\n",
    "\n",
    "$a_o$ after first prediction =\n",
    "\n",
    "|$x_0$|$x_1$|$y_2$|$y$| $a_o$ |\n",
    "|-|||||\n",
    "|1|1|1|0|0.991531|\n",
    "|1|0|1|1|0.987919|\n",
    "|0|1|1|1|0.987919|\n",
    "|0|0|1|0|0.974798|\n",
    "\n",
    "$a_o$ after the second (with updated weights) prediction =\n",
    "\n",
    "|$x_0$|$x_1$|$y_2$|$y$| $a_o$ |\n",
    "|-|||||\n",
    "|1|1|1|0|0.98786405|\n",
    "|1|0|1|1|0.98309866|\n",
    "|0|1|1|1|0.98309866|\n",
    "|0|0|1|0|0.96660214|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "That is not very impressive, right? But that is a typical behaviour of neural networks, they need to do these update operations very, very often. With random initialized weight matrices, a learning rate $eta=3$ and 20000 epochs of feeding forward and backpropagation, we will already have a much better result of $a_o$:\n",
    "\n",
    "|$x_0$|$x_1$|$y_2$|$y$| $a_o$ |\n",
    "|-|||||\n",
    "|1|1|1|0|0.00349888|\n",
    "|1|0|1|1|0.99565532|\n",
    "|0|1|1|1|0.99728676|\n",
    "|0|0|1|0|0.00382784|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Not bad, right?\n",
    "\n",
    "Congratulations! If you made it so far, a neural network is a bit less of a black box for you now. I hope you learnt something and could follow all this matrix multiplication. If you find any bugs in the code or other mistakes, just leave me a note in the comments.\n",
    "\n",
    "Mavicc from web.studio"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}