Exercise 09 - 2x3x2 Neural Network.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_format = 'retina'\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "np.set_printoptions(precision=3)\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An Exercise:  2 x 3 x 2 Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the purpose of mastering the concept of gradient descent and the backpropagation algorithm,  we will extend the previous examples by programming from scratch a neural network with 2 input layers, 3 hidden layers and 2 output layers. Again, we will be using simple variables to store the activations and derivatives instead of using vectors (next lecture). The purpose of writing the code in this simplified manner is to make the concepts as clear as possible, and show how they relate with the mathematical equations. __For that reason, code will deliberately be inefficient, and shouldn't be used for any serious application__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data and labels we will be using is the same as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.array([[ 1.2, 0.7],\n",
    "                 [-0.3,-0.5],\n",
    "                 [ 3.0, 0.1],\n",
    "                 [-0.1,-1.0],\n",
    "                 [-0.0, 1.1],\n",
    "                 [ 2.1,-1.3],\n",
    "                 [ 3.1,-1.8],\n",
    "                 [ 1.1,-0.1],\n",
    "                 [ 1.5,-2.2],\n",
    "                 [ 4.0,-1.0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = np.array([  'red',\n",
    "                     'blue',\n",
    "                     'red',\n",
    "                     'blue',\n",
    "                     'blue',\n",
    "                     'red',\n",
    "                     'blue',\n",
    "                     'red',\n",
    "                     'blue',\n",
    "                     'blue'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_data(data, labels):\n",
    "    fig = plt.figure(figsize=(5,5))\n",
    "    ax = fig.add_subplot(111)\n",
    "    ax.scatter(data[:,0], data[:,1], c=labels, s=50,  cmap=plt.cm.bwr,zorder=50)\n",
    "    nudge = 0.08\n",
    "    for i, d in enumerate(data):\n",
    "        ax.annotate(f'{i}',(d[0]+nudge,d[1]+nudge))\n",
    "    ax.set_aspect('equal', 'datalim')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_data(data, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the illustration of the neural network we will be using:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/neural_networks_26.png\" alt=\"drawing\" width=\"950\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task 1:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a function `convert_label(label)`, which converts the textual labels to probabilistic`(1,0)` and `(0,1)` :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test this function with some values from `labels`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a function `forward_pass(A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5,X,Y)` that takes as an input all weights and biases and the current data point, and returns the function's output as a tuple `(O1, O2)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us first initialise the helper functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0,
     11,
     18,
     30,
     57,
     60
    ]
   },
   "outputs": [],
   "source": [
    "def create_meshgrid(data):\n",
    "    h = 0.02\n",
    "    x_min, x_max = data[:, 0].min() - 1, data[:, 0].max() + 1\n",
    "    y_min, y_max = data[:, 1].min() - 1, data[:, 1].max() + 1\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "    return (xx,yy,np.ones(xx.shape))\n",
    "\n",
    "def eval_accuracy_neural(params, data, labels):\n",
    "    A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5 = params\n",
    "    num_correct = 0;\n",
    "    \n",
    "    for (X,Y),label_old in zip(data,labels):\n",
    "        score = forward_pass(A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5,X,Y)\n",
    "        sc = np.array((int(score[0]>0.5),int(score[1]>0.5)))\n",
    "        true_label = convert_label(label_old)\n",
    "        if (sc[0]==true_label[0] and sc[1]==true_label[1]):\n",
    "            num_correct += 1\n",
    "    return num_correct / len(data)\n",
    "\n",
    "def plot_neural_simple(params, grid,data, labels, iteration, accuracy):\n",
    "    nudge = 0.06\n",
    "    A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5 = params\n",
    "    xx,yy,Z = grid\n",
    "    \n",
    "    for i in range(xx.shape[0]): # row\n",
    "        for j in range(yy.shape[1]): #column\n",
    "            X, Y = xx[i][j],yy[i][j]\n",
    "            output = forward_pass(A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5,X,Y) \n",
    "            score = 1 if np.argmax(output)==0 else -1\n",
    "            Z[i][j] = score\n",
    "    fig = plt.figure(figsize=(5,5))\n",
    "    ax = fig.add_subplot(111)\n",
    "    plt.title(f'accuracy at the iteration {iteration}: {accuracy}')\n",
    "    ax.contourf(xx, yy, Z, cmap=plt.cm.binary, alpha=0.1, zorder=15)\n",
    "    ax.scatter(data[:, 0], data[:, 1], c=labels, s=50,  cmap=plt.cm.bwr,zorder=50)\n",
    "    ax.set_aspect('equal')\n",
    "    for i, d in enumerate(data):\n",
    "        ax.annotate(f'{i}',(d[0]+nudge,d[1]+nudge))\n",
    "    plt.xlim(xx.min(), xx.max())\n",
    "    plt.ylim(yy.min(), yy.max())\n",
    "    plt.show()\n",
    "    \n",
    "def sigmoid(x):\n",
    "    return 1/(1+np.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def train_neural_network(data, labels, step_size, epochs, iter_info):\n",
    "    # number of data points\n",
    "    datalen = len(data)\n",
    "    # for visualisation purposes only\n",
    "    grid = create_meshgrid(data)\n",
    "    # first we initialise all the weights and biases as normally distributed random numbers\n",
    "    rnd = np.random.normal(size=17)\n",
    "    # hidden layer\n",
    "    # neuron 1\n",
    "    A1, B1, C1 = rnd[0], rnd[1], rnd[2]\n",
    "    # neuron 2\n",
    "    A2, B2, C2 = rnd[3], rnd[4], rnd[5]\n",
    "    # neuron 3\n",
    "    A3, B3, C3 = rnd[6], rnd[7], rnd[8]\n",
    "    # output layer\n",
    "    # neuron 1\n",
    "    A4, B4, C4, D4 = rnd[9], rnd[10], rnd[11], rnd[12]\n",
    "    # neuron 2\n",
    "    A5, B5, C5, D5 = rnd[13], rnd[14], rnd[15], rnd[16]\n",
    "\n",
    "    # in each epoch, we will go through all the data points once\n",
    "    for i in range (epochs):\n",
    "        # total cost must be restarted in every epoch\n",
    "        total_cost = 0.0\n",
    "        ddA1, ddA2, ddA3, ddA4, ddA5, ddB1, ddB2, ddB3, ddB4, ddB5, ddC1, ddC2, ddC3, ddC4, ddC5, ddD4, ddD5 = np.zeros(17)\n",
    "        # here we loop trough all the data points, and get X, Y and the corresponding label\n",
    "        for index, ((X,Y), label_text) in enumerate(zip(data, labels)): \n",
    "            # forward pass\n",
    "            N1 = sigmoid(A1*X + B1*Y + C1) # 1st neuron activation\n",
    "            N2 = sigmoid(A2*X + B2*Y + C2) # 2nd neuron activation\n",
    "            N3 = sigmoid(A3*X + B3*Y + C3) # 3rd neuron activation\n",
    "            z1 = A4*N1 + B4*N2 + C4*N3 + D4 #intermediate step 1\n",
    "            z2 = A5*N1 + B5*N2 + C5*N3 + D5 #intermediate step 2\n",
    "            O1 = sigmoid(z1) # final activation 1\n",
    "            O2 = sigmoid(z2) # final activation 2\n",
    "            # convert the old labels -1 and 1 to tuples (1,0) and (0,1)\n",
    "            label = convert_label(label_text)\n",
    "            # This is the implementation of the quadratic cost for the single data point\n",
    "            cost = (O1-label[0])**2 + (O2-label[1])**2\n",
    "            # This value is then accumulated into total_cost variable, so that the cost is expressed\n",
    "            # for all data points\n",
    "            total_cost += cost\n",
    "\n",
    "            # backpropagating through the network\n",
    "            # here we compute the derivative of the total cost TC in respect to the intermediate steps z1 and z2\n",
    "            dz1 = 2*(O1-label[0])*O1*(1-O1)\n",
    "            dz2 = 2*(O2-label[1])*O2*(1-O2)\n",
    "            # computing the derivatives of the weights and biases of the first neuron in the output layer\n",
    "            dA4, dB4, dC4, dD4 = dz1*N1, dz1*N2, dz1*N3, dz1\n",
    "            # computing the derivatives of the weights and biases of the second neuron in the output layer\n",
    "            dA5, dB5, dC5, dD5 = dz2*N1, dz2*N2, dz2*N3, dz2\n",
    "\n",
    "            # computing the partials of the intermediate steps dz1, dz2, dz3 in respect to the activations N1, N2, N3\n",
    "            dN1 = dz1*A4 + dz2*A5\n",
    "            dN2 = dz1*B4 + dz2*B5\n",
    "            dN3 = dz1*C4 + dz2*C5\n",
    "            # computing the partials of the intermediate steps p1, p2 and p3\n",
    "            dp1 = dN1*N1*(1-N1)\n",
    "            dp2 = dN2*N2*(1-N2)\n",
    "            dp3 = dN3*N3*(1-N3)\n",
    "            \n",
    "            # computing the partials of the total cost in respect to weights and biases in the neuron n1\n",
    "            dA1, dB1, dC1 = dp1*X, dp1*Y, dp1\n",
    "            # computing the partials of the total cost in respect to weights and biases in the neuron n2\n",
    "            dA2, dB2, dC2 = dp2*X, dp2*Y, dp2\n",
    "            # computing the partials of the total cost in respect to weights and biases in the neuron n3\n",
    "            dA3, dB3, dC3 = dp3*X, dp3*Y, dp3\n",
    "            \n",
    "            # since we have multiple data points, and we can compute only a single partial derivative at the time\n",
    "            # we accumulate their values in the variables ddA1...ddD4\n",
    "            ddA1 += dA1\n",
    "            ddA2 += dA2\n",
    "            ddA3 += dA3\n",
    "            ddA4 += dA4\n",
    "            ddA5 += dA5\n",
    "            ddB1 += dB1\n",
    "            ddB2 += dB2\n",
    "            ddB3 += dB3\n",
    "            ddB4 += dB4\n",
    "            ddB5 += dB5\n",
    "            ddC1 += dC1\n",
    "            ddC2 += dC2\n",
    "            ddC3 += dC3\n",
    "            ddC4 += dC4\n",
    "            ddC5 += dC5\n",
    "            ddD4 += dD4\n",
    "            ddD5 += dD5\n",
    "            \n",
    "        # Now we make an average of all the accumulated derivatives by dividing with the number of data points\n",
    "        ddA1 /= datalen\n",
    "        ddA2 /= datalen\n",
    "        ddA3 /= datalen\n",
    "        ddA4 /= datalen\n",
    "        ddA5 /= datalen\n",
    "        ddB1 /= datalen\n",
    "        ddB2 /= datalen\n",
    "        ddB3 /= datalen\n",
    "        ddB4 /= datalen\n",
    "        ddB5 /= datalen\n",
    "        ddC1 /= datalen\n",
    "        ddC2 /= datalen\n",
    "        ddC3 /= datalen\n",
    "        ddC4 /= datalen\n",
    "        ddC5 /= datalen\n",
    "        ddD4 /= datalen\n",
    "        ddD5 /= datalen\n",
    "\n",
    "        # finally, we do the parameter update with the averaged values ddA1...ddD4\n",
    "        A1 -= ddA1 * step_size;\n",
    "        B1 -= ddB1 * step_size; \n",
    "        C1 -= ddC1 * step_size;\n",
    "        A2 -= ddA2 * step_size; \n",
    "        B2 -= ddB2 * step_size;\n",
    "        C2 -= ddC2 * step_size;\n",
    "        A3 -= ddA3 * step_size; \n",
    "        B3 -= ddB3 * step_size; \n",
    "        C3 -= ddC3 * step_size;\n",
    "        A4 -= ddA4 * step_size; \n",
    "        B4 -= ddB4 * step_size; \n",
    "        C4 -= ddC4 * step_size;\n",
    "        A5 -= ddA5 * step_size;\n",
    "        B5 -= ddB5 * step_size;\n",
    "        C5 -= ddC5 * step_size;\n",
    "        D4 -= ddD4 * step_size;\n",
    "        D5 -= ddD5 * step_size;\n",
    "\n",
    "        if (i%iter_info==0):\n",
    "            accuracy = eval_accuracy_neural((A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5),data,labels)\n",
    "            plot_neural_simple((A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5),grid, data, labels, i, accuracy)\n",
    "            print (f'total cost: {total_cost}\\n')\n",
    "            \n",
    "    return (A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "params = train_neural_network(data, labels, 2, 301, 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect each data point's predicted vs true label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5 = params\n",
    "for i, ((X,Y),label_text) in enumerate(zip(data,labels)):\n",
    "    score = forward_pass(A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5,X,Y)\n",
    "    label = convert_label(label_text)\n",
    "    sc = (int(score[0]>=0.5),int(score[1]>=0.5))\n",
    "    print (f'data point {i}: real label : {label}, pred. label: {sc}, {sc==label}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}