exercise_750.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Backpropagation in a Multi-Layer Neural Network\n",
    "\n",
    "We explore backpropagation assuming the batch has two data points with features $\\begin{bmatrix}-10 & 20\\end{bmatrix}$ and $\\begin{bmatrix}8 & -2\\end{bmatrix}$ and the correct categories $1$ and $0$ which are one-hot encoded as $\\begin{bmatrix}0 & 1\\end{bmatrix}$ and $\\begin{bmatrix}1 & 0\\end{bmatrix}$.\n",
    "\n",
    "The parameters of the multi-layer neural network (the weights and biases) which minimize the loss are discovered by descending the loss gradient.\n",
    "\n",
    "In this exercise, we compute partial derivatives over batches of training data for a multi-layer neural network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters (weights) 1st layer: \n",
      " 1  1\n",
      " 2  2\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "Parameters (weights) 2nd layer: \n",
      " 2  1\n",
      " 2  1\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "Parameters (bias) 1st layer: \n",
      "-1 -1\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "Parameters (bias) 2nd layer: \n",
      " 5  5\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "weights1 = torch.nn.Parameter(torch.Tensor([[1,1],[2,2]]))\n",
    "print(\"Parameters (weights) 1st layer: \"+str(weights1.data))\n",
    "\n",
    "weights2 = torch.nn.Parameter(torch.Tensor([[2, 1],[2,1]]))\n",
    "print(\"Parameters (weights) 2nd layer: \"+str(weights2.data))\n",
    "\n",
    "bias1 = torch.nn.Parameter(torch.Tensor([[-1,-1]]))\n",
    "print(\"Parameters (bias) 1st layer: \"+str(bias1.data))\n",
    "\n",
    "bias2 = torch.nn.Parameter(torch.Tensor([[5,5]]))\n",
    "print(\"Parameters (bias) 2nd layer: \"+str(bias2.data))\n",
    "\n",
    "inputs = torch.autograd.Variable(torch.Tensor([[-10, 20],[8, -2]]))\n",
    "\n",
    "target = torch.autograd.Variable(torch.LongTensor([1, 0]))\n",
    "#print(target)\n",
    "\n",
    "one_hot_target = torch.autograd.Variable(torch.Tensor([[0, 1],[1, 0]]))\n",
    "#print(one_hot_target)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume the weights are $\\begin{bmatrix}1 & 1 \\\\ 2 & 2\\end{bmatrix}$ for the lower layer and $\\begin{bmatrix}2 & 1 \\\\ 2 & 1\\end{bmatrix}$ for the higher layer at the start.\n",
    "\n",
    "The training is assumed to proceed one data point at a time (batch size of 1).\n",
    "\n",
    "For this iteration of training, the features are $\\begin{bmatrix}-10 & 20\\end{bmatrix}$ and $\\begin{bmatrix}8 & -2\\end{bmatrix}$ and the correct categories are $1$ and $0$ which are one-hot encoded as $\\begin{bmatrix}0 & 1\\end{bmatrix}$ and $\\begin{bmatrix}1 & 0\\end{bmatrix}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a = Variable containing:\n",
      " 29  29\n",
      "  3   3\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "h = relu(a) = Variable containing:\n",
      " 29  29\n",
      "  3   3\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "c = Variable containing:\n",
      " 121   63\n",
      "  17   11\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "softmax(c) = Variable containing:\n",
      " 1.0000e+00  6.4702e-26\n",
      " 9.9753e-01  2.4726e-03\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "Cross entropy loss: Variable containing:\n",
      " 29.0012\n",
      "[torch.FloatTensor of size 1]\n",
      "\n",
      "MANUALLY COMPUTED PARTIAL DERIVATIVES\n",
      "\tThe manually computed gradient of the loss with respect to the weights of the second layer is \n",
      " 14.4963 -14.4963\n",
      " 14.4963 -14.4963\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\tThe manually computed gradient of the loss with respect to the weights of the first layer is \n",
      " -5.0099  -5.0099\n",
      " 10.0025  10.0025\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\tThe manually computed gradient of the loss with respect to the biases of the second layer is \n",
      " 0.4988 -0.4988\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "\tThe manually computed gradient of the loss with respect to the biases of the first layer is \n",
      " 0.4988  0.4988\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "AUTOMATICALLY COMPUTED PARTIAL DERIVATIVES\n",
      "\tThe automatically computed gradient of the loss with respect to the weights of the second layer is \n",
      " 14.4963 -14.4963\n",
      " 14.4963 -14.4963\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\tThe automatically computed gradient of the loss with respect to the weights of the first layer is \n",
      " -5.0099  -5.0099\n",
      " 10.0025  10.0025\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\tThe automatically computed gradient of the loss with respect to the biases of the second layer is \n",
      " 0.4988 -0.4988\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "\tThe automatically computed gradient of the loss with respect to the biases of the first layer is \n",
      " 0.4988  0.4988\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass\n",
    "\n",
    "hidden = torch.mm(inputs, weights1) + bias1\n",
    "print(\"a = \"+str(hidden))\n",
    "\n",
    "hidden_relu = F.relu(hidden)\n",
    "print(\"h = relu(a) = \"+str(hidden_relu))\n",
    "\n",
    "result = torch.mm(hidden_relu, weights2) + bias2\n",
    "print(\"c = \"+str(result))\n",
    "\n",
    "softmax_result = F.softmax(result, dim=1)\n",
    "print(\"softmax(c) = \"+str(softmax_result))\n",
    "\n",
    "loss = F.cross_entropy(result, target)\n",
    "print(\"Cross entropy loss: \"+str(loss))\n",
    "\n",
    "# Backward Pass\n",
    "\n",
    "batch_size = inputs.size()[0]\n",
    "grad_softmax = (softmax_result.data - one_hot_target.data)\n",
    "grad_hidden_activation = grad_softmax.mm(weights2.data.t())\n",
    "grad_hidden_pre_activation = grad_hidden_activation.clone()\n",
    "grad_hidden_pre_activation[hidden.data < 0] = 0\n",
    "grad_weight2 = hidden.data.t().mm(grad_softmax)\n",
    "grad_weight2 = grad_weight2/batch_size\n",
    "grad_bias2 = grad_softmax\n",
    "grad_bias2 = torch.unsqueeze(torch.sum(grad_bias2, 0), 0)\n",
    "grad_bias2 = grad_bias2/batch_size\n",
    "grad_weight1 = inputs.data.t().mm(grad_hidden_pre_activation)\n",
    "grad_weight1 = grad_weight1/batch_size\n",
    "grad_bias1 = grad_hidden_pre_activation\n",
    "grad_bias1 = torch.unsqueeze(torch.sum(grad_bias1, 0), 0)\n",
    "grad_bias1 = grad_bias1/batch_size\n",
    "\n",
    "print(\"MANUALLY COMPUTED PARTIAL DERIVATIVES\")\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to the weights of the second layer is \"+str(grad_weight2))\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to the weights of the first layer is \"+str(grad_weight1))\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to the biases of the second layer is \"+str(grad_bias2))\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to the biases of the first layer is \"+str(grad_bias1))\n",
    "\n",
    "# Autograd\n",
    "\n",
    "loss.backward()\n",
    "\n",
    "print(\"AUTOMATICALLY COMPUTED PARTIAL DERIVATIVES\")\n",
    "\n",
    "gradient = weights2.grad\n",
    "\n",
    "print(\"\\tThe automatically computed gradient of the loss with respect to the weights of the second layer is \"+str(gradient.data))\n",
    "\n",
    "gradient = weights1.grad\n",
    "\n",
    "print(\"\\tThe automatically computed gradient of the loss with respect to the weights of the first layer is \"+str(gradient.data))\n",
    "\n",
    "gradient = bias2.grad\n",
    "\n",
    "print(\"\\tThe automatically computed gradient of the loss with respect to the biases of the second layer is \"+str(gradient.data))\n",
    "\n",
    "gradient = bias1.grad\n",
    "\n",
    "print(\"\\tThe automatically computed gradient of the loss with respect to the biases of the first layer is \"+str(gradient.data))\n",
    "\n",
    "if weights1.grad is not None:\n",
    "    weights1.grad.data.zero_()\n",
    "if weights2.grad is not None:\n",
    "    weights2.grad.data.zero_()\n",
    "if bias1.grad is not None:\n",
    "    bias1.grad.data.zero_()\n",
    "if bias2.grad is not None:\n",
    "    bias2.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}