13. Elements of a Neural Network Expressed in General Terms.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_format = 'retina'\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "np.set_printoptions(precision=3)\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network implementation with Matrices #3: General case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we will introduce a general case of a neural network described and implemented in matrix form. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General procedure for calculating the neuron activations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naming the elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the general case, we will isolate a single neuron belonging to an arbitrary layer (just not the input layer which is not parametrised), and a number of activations of the previous layer that connect into it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With $b_j^{(l)}$ we will mark the __bias__ of the $j^{th}$ neuron belonging to the $l^{th}$ layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With $w_{jk}^{(l)}$ we will mark the __weight__ of the $j^{th}$ neuron in the $l^{th}$ layer that parametrises (multiplies) the $k^{th}$ neuron in the $(l-1)^{th}$ layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With $a_{j}^{(l)}$ we will mark the __activation__ of the $j^{th}$ neuron in the $l^{th}$ layer, and with $a_{k}^{(l-1)}$, the activation of the $k^{th}$ neuron in the $(l-1)^{th}$ layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/neural_networks_32.png\" alt=\"drawing\" width=\"450\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the activation $a_{j}^{(l)}$ we must iterate over all the neurons in the $(l-1)^{th}$ layer that connect to the neuron $n_j^{(l)}$, calculate the weighted averages between these neurons and  the $n_j^{(l)}$, sum them together, and apply the desired activation function (in our case sigmoid) to their sum. This can be described by the following rather index-heavy formula:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \n",
    "  a^{(l)}_j = \\sigma\\left( \\sum_{k=1}^{N^{(l-1)}} w^{(l)}_{jk} a^{(l-1)}_k + b^{(l)}_j \\right)\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $N^{(l-1)}$ defines the number of the neurons in the $(l-1)^{th}$ layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using matrix notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we will be using the matrices to describe and compute our neural network, we need to make adjustments to this formula. <br><br>\n",
    "First, we can take all the weights of the layer $l$ and store them in a single matrix $w^{(l)}:$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "w^{(l)}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "w_{1,1}^{(l)} & w_{1,2}^{(l)} & w_{1,3}^{(l)} & \\cdots & w_{1,k}^{(l)} & \\cdots & w_{1,N^{(l-1)}}^{(l)} \\\\\n",
    "w_{2,1}^{(l)} & w_{2,2}^{(l)} & w_{2,3}^{(l)} & \\cdots & w_{2,k}^{(l)} & \\cdots & w_{2,N^{(l-1)}}^{(l)} \\\\\n",
    "w_{3,1}^{(l)} & w_{3,2}^{(l)} & w_{3,3}^{(l)} & \\cdots & w_{3,k}^{(l)} & \\cdots & w_{3,N^{(l-1)}}^{(l)} \\\\\n",
    "\\vdots & \\vdots & \\vdots & \\ddots & \\vdots & \\vdots & \\vdots \\\\\n",
    "w_{j,1}^{(l)} & w_{j,2}^{(l)} & w_{j,3}^{(l)} & \\cdots & w_{j,k}^{(l)} & \\cdots & w_{j,N^{(l-1)}}^{(l)} \\\\\n",
    "\\vdots & \\vdots & \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "w_{N^{(l)},1}^{(l)} & w_{N^{(l)},2}^{(l)} & w_{N^{(l)},3}^{(l)} & \\cdots & w_{N^{(l)},k}^{(l)} & \\cdots & w_{N^{(l)},N^{(l-1)}}^{(l)}\\\\\n",
    "\\end{bmatrix}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row of this matrix stands for a neuron in the $l^{th}$ layer and all of its weights. The columns indicate to which neuron of the previous layer is the respective neuron of the current layer connected. For example:\n",
    "- The top left term marks the weight of the first neuron in the $l^{th}$ layer that parametrises the activation of the first neuron of the $(l-1)^{th}$ layer. \n",
    "- The top right term marks the weight of the first neuron in the $l^{th}$ layer that parametrises the activation of the last neuron of the $(l-1)^{th}$ layer. \n",
    "- The bottom left term marks the weight of the last neuron in the $l^{th}$ layer that parametrises the activation of the first neuron of the $(l-1)^{th}$ layer. \n",
    "- The bottom right term marks the weight of the last neuron in the $l^{th}$ layer that parametrises the activation of the last neuron of the $(l-1)^{th}$ layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also take all the biases of the layer $l$ and store them in a single matrix $b^{(l)}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "b^{(l)}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "b_{1}^{(l)}\\\\\n",
    "b_{2}^{(l)}\\\\\n",
    "b_{3}^{(l)}\\\\\n",
    "\\vdots \\\\\n",
    "b_{j}^{(l)} \\\\\n",
    "\\vdots \\\\\n",
    "b_{N^{(l)}}^{(l)} \\\\\n",
    "\\end{bmatrix}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same can be done with the weighted averages and activations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "z^{(l)}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "z_{1}^{(l)}\\\\\n",
    "z_{2}^{(l)}\\\\\n",
    "z_{3}^{(l)}\\\\\n",
    "\\vdots \\\\\n",
    "z_{j}^{(l)} \\\\\n",
    "\\vdots \\\\\n",
    "z_{N^{(l)}}^{(l)} \\\\\n",
    "\\end{bmatrix};\\qquad\n",
    "a^{(l)}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "a_{1}^{(l)}\\\\\n",
    "a_{2}^{(l)}\\\\\n",
    "a_{3}^{(l)}\\\\\n",
    "\\vdots \\\\\n",
    "a_{j}^{(l)} \\\\\n",
    "\\vdots \\\\\n",
    "a_{N^{(l)}}^{(l)} \\\\\n",
    "\\end{bmatrix}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating the activations of a single layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this done, the description of the computation of the $a_j^{(l)}$ can be written in much more simplified form:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "a^{(l)} = \\sigma(w^{(l)} a^{(l-1)}+b^{(l)})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "z^{(l)} = w^{(l)} a^{(l-1)}+b^{(l)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By utilising matrices, we can now compute the activations of a whole layer of neurons, instead of just a single neuron. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing the cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have been indicating the layers of the network with the symbol $l$. To indicate the __output layer__ of the network we will instead use the symbol $L$. Therefore, the activation of the output layer of our network will be marked as $a^{(L)}$. To compute the cost/loss of the network for a single training example $x$ (data point), we can use this formula, slightly modified from the previous class:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "C_x = \\frac{1}{2}\n",
    "\\sum_{i=1}^{N^{(L)}} (y_i-a_i^{(L)})^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The formula suggests that we need to substract the activation of each neuron in the output layer from the true label associated it, square the difference and sum the squared terms. This same equation can be represented in simpler terms by using vectors:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "C_x =\n",
    "\\frac{1}{2} \\|y-a^{(L)} \\|^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notation $\\|v\\|$ just denotes the magnitude or the length of a vector $v$. This is equivalent to the square root of the dot product of the vector by itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the total cost for all training examples (or their subset of length $n$) the formula becomes:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "C = \\frac{1}{2n} \\sum_{x=1}^{n} \\|y(x)-a^{(L)}(x)\\|^2\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backward pass: Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing the derivatives of the output layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding a derivative of the total cost in respect to the weighted averages of the network makes the computation of the derivatives of the weights and biases trivial. We will start by definining the derivative of the total cost/loss of the network in respect to a weighted sum of the neuron $j$ of the output layer $L$ layer with $\\delta_j^{(l)}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \n",
    "  \\delta^{(L)}_j \\equiv \\frac{\\partial C}{\\partial z^{(L)}_j}\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the derivative of the total cost in respect to the single weighted sum $\\delta^{(L)}_j$ of the output layer we can use the chain rule:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "  \\delta^{(L)}_j = \\frac{\\partial C}{\\partial a^{(L)}_j} * \\frac{\\partial a^{(L)}_j}{\\partial z^{(L)}_j}.\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The term on the left is the partial derivative of the cost in respect to the activation of the $j^{th}$ neuron of the output layer, and the term of the right is the partial derivative of the activation of the $j^{th}$ the neuron in the output layer in respect to the weighted sum of the same neuron in the same layer. Since the right side is simply a derivative of the activation function—in our case sigmoid function—the previous equation can be rewritten more simply as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \n",
    "\\delta^{(L)}_j = \\frac{\\partial C}{\\partial a^{(L)}_j} * \\sigma'(z^{(L)}_j)\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The partial derivative of the left side is easily computable. If we're using the quadratic cost function then:\n",
    "\n",
    "\n",
    "$$\\frac{\\partial C}{\\partial a^{(L)}_j} = 2*(a_j^{(L)}-y_j)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our vector implementaton, we won't be computing partial derivatives for each individual neuron $\\delta^{(L)}_j$ in $L$, but rather the activations for the whole output layer $\\delta^{(L)}$ stored in a single matrix. For that purpose we will use the Hadamard product to indicate an elementwise multiplication between two column matrices:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \n",
    "\\delta^{(L)} = (a^{(L)}-y) \\odot \\sigma'(z^{(L)})\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is in fact the same as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\\n",
    "\\delta^{(L)}\n",
    "\\equiv\n",
    "\\begin{bmatrix}\n",
    "\\delta^{(L)}_1 \\\\\n",
    "\\delta^{(L)}_2 \\\\\n",
    "\\delta^{(L)}_3 \\\\\n",
    "\\vdots \\\\\n",
    "\\delta^{(L)}_j \\\\\n",
    "\\vdots \\\\\n",
    "\\delta^{(L)}_{N^{(L)}} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "\\left(a_{1}^{(L)}-y_1\\right)*\\sigma^{\\prime}\\left(z_1^{(L)}\\right)\\\\\n",
    "\\left(a_{2}^{(L)}-y_2\\right)*\\sigma^{\\prime}\\left(z_2^{(L)}\\right)\\\\\n",
    "\\left(a_{3}^{(L)}-y_3\\right)*\\sigma^{\\prime}\\left(z_3^{(L)}\\right)\\\\\n",
    "\\vdots \\\\\n",
    "\\left(a_{j}^{(L)}-y_j\\right)*\\sigma^{\\prime}\\left(z_j^{(L)}\\right)\\\\\n",
    "\\vdots \\\\\n",
    "\\left(a_{N^{(L)}}^{(L)}-y_{N^{(L)}}\\right)*\\sigma^{\\prime}\\left(z_{N^{(L)}}^{(L)}\\right)\\\\\n",
    "\\end{bmatrix}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The term $(a^L-y)$ is a derivative specific to the quadratic cost function. A more general way to write this derivative irrespective of the cost function used would be to use the expression $\\nabla_a C$, expressing the __gradient__ of the cost function in respect to the output activations. For this general case we can rewrite the previous expression as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \n",
    "  \\delta^{(L)} = \\nabla_a C \\odot \\sigma'(z^{(L)})\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing the derivatives of the cost in respect to the weighted averages of a non-output layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the derivatives of the cost in respect to the weighted averages in an arbitrary hidden layer $l$, requires us to already have computed the derivates of the cost in respect to the weighted averages in the layer $(l+1)$ following it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The computation requires us to compute the _transpose_ matrix of the weight matrix $w^{(l+1)}$ and then use it to matrix multiply the derivatives of the cost in respect to the weighted sums in the $(l+1)^{th}$ layer. The matrix representing the product is then multiplied element-wise with the derivative matrix of the activation layer $l$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \n",
    "  \\delta^{(l)} = ((w^{(l+1)})^T \\delta^{(l+1)}) \\odot \\sigma'(z^{(l)})\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The general procedure for computing the partial derivatives of the cost in respect to the weighted averages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general procedure consists of using the first formula to compute the matrix of partial derivatives of the cost in respect to the weighted averages in the output layer $\\delta^{(L)}$. Then we can use the second formula to compute the matrix of the partials of the previous layer $\\delta^{(L-1)}$, layer then $\\delta^{(L-2)}$ in the layer before it, then $\\delta^{(L-3)}$, etc. until we reach the input layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing the partial derivative of the cost in respect to any bias in the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have the partial derivatives of the weighted sums, the computation of the partial derivative of the biases in the network is trivial. The partial derivative of the cost in respect the bias $b_j^{(l)}$ of the $l^{th}$ layer is equal to the derivative of the weighted sum $\\delta_j^{(l)}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}  \n",
    "\\frac{\\partial C}{\\partial b^{(l)}_j} = \\delta^{(l)}_j\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this follows that the entire layer of biases represented in the matrix $b^{(l)}$ is also equal to the matrix storing partial derivatives in respect to the weighted sums of the same layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "  \\frac{\\partial C}{\\partial b^{(l)}} = \\delta^{(l)}\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing the partial derivative of the cost in respect to any weight in the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the partial derivative of the total cost in respect to any particular weight in the network symbolised with $w_{jk}^{(l)}$, we simply need to find its corresponding activation. For $w_{jk}^{(l)}$, this is $a_{k}^{(l-1)}$, which is located on the same line as the weight (see the illustration). Then we need to multiply it with the partial derivative of the cost in respect to the weighted sum $\\delta_j^{(l)}$ of the same layer. The partial $\\delta_j^{(l)}$ is not depicted on the drawing, as it is considered to be \"inside\" the neuron, but you can imagine it located just before the neuron's activation $a_j^{(l)}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "\\frac{\\partial C}{\\partial w^{(l)}_{jk}} = a^{(l-1)}_k \\delta^{(l)}_j\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/neural_networks_32.png\" alt=\"drawing\" width=\"450\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, to compute the matrix of partial derivatives of the cost in respect to the weights of the $l^{th}$ layer  $w^{(l)}$, we need to transpose the matrix of activations of the previous layer $a^{(l-1)}$ and matrix multiply it by the matrix of activations of the layer $l$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "w^{(l)}\n",
    "=\n",
    "\\delta^{(l)}\\times\\left(a^{(l-1)}\\right)^T\n",
    "\\end{align*}\n",
    "$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}