PyTorch101 - Part 3 - Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch101 - Part 3 - Neural Networks\n",
    "In this part of the tutorial, we are going to deal with neural networks, for which we have all the basic ingredients: tensors and autograd. In this part we are only going to describe basic ingredients for neural networks, while in the following part we will effectively train the network on the MNIST dataset. \n",
    "Our architecture will be the following:\n",
    "- image input 28x28\n",
    "- 32 3x3 conv filters\n",
    "- 64 3x3 conv filters\n",
    "- max pooling size 2\n",
    "- dropout 0.25\n",
    "- flatten\n",
    "- dense layer with 128 neurons\n",
    "- dropout 0.5\n",
    "- softmax for the 10 classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4.0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network definition and forward pass\n",
    "\n",
    "Neural net structure are usually defined inside classes for simplicity, we will see incrementally how this is implemented. \n",
    "\n",
    "Relevant modules we will use in the init phase and relative definition:\n",
    "- 2D convolution: *Conv2d(input_filters, output_filters, filter_size)*\n",
    "- Dense: *Linear(input_size, output_size)*\n",
    "- Softmax: *Softmax()*\n",
    "\n",
    "We will also use other function inside the forward pass, namely:\n",
    "- Relu: *F.relu(x)*\n",
    "- Maxpool2D: *F.max_pool2d(x)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        # Define the network modules\n",
    "        self.conv1 = nn.Conv2d(1, 32, 3) # input_filters=1 because MNIST is gray-scale\n",
    "        self.conv2 = nn.Conv2d(32, 64, 3)\n",
    "        self.fc1 = nn.Linear(9216, 128) # 9216 is the size of the flattened layer\n",
    "        self.fc2 = nn.Linear(128, 10)\n",
    "        self.softmax = nn.Softmax(dim=1)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # Defines what happens in the forward pass\n",
    "        x = F.relu(self.conv1(x))\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)),(2,2))\n",
    "        x = x.view(-1, self.num_flat_features(x))\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        #x = self.softmax(x)\n",
    "        return x\n",
    "        \n",
    "    def num_flat_features(self, x):\n",
    "        size = x.size()[1:]  # all dimensions except the batch dimension\n",
    "        num_features = 1\n",
    "        for s in size:\n",
    "            num_features *= s\n",
    "        return num_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now instantiate the network. Having defined the forward pass, pytorch automatically defines the backward pass based on the operations we perform. We can also access the parameters of this network directly.\n",
    "\n",
    "[SIDENOTE] This is very similar to what we do in Tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of params: 8\n",
      "First conv size torch.Size([32, 1, 3, 3])\n",
      "Value of the first filter: tensor([[[ 0.3201,  0.2941,  0.0749],\n",
      "         [-0.0144,  0.1617,  0.0281],\n",
      "         [ 0.0413, -0.2278,  0.1221]]])\n"
     ]
    }
   ],
   "source": [
    "net = Net()\n",
    "params = list(net.parameters())\n",
    "print(\"Number of params:\", len(params))\n",
    "print(\"First conv size\", params[0].size())\n",
    "print(\"Value of the first filter:\", params[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that it is very \"transparent\" w.r.t. the parameters of the network, making them very accessible. We can throw some random input at it if we want, just to verify that it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0000,  0.0110,  0.0000,  0.0355,  0.0000,  0.1435,  0.1409,\n",
      "          0.2006,  0.0000,  0.0000]])\n"
     ]
    }
   ],
   "source": [
    "random_input = torch.randn(1, 1, 28, 28)\n",
    "out = net(random_input)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss function and optimization\n",
    "We still miss the loss function and the optimization which will minimize it.\n",
    "For this task, we choose a simple cross-entropy computed on the softmax, while we will use vanilla gradient descent as an optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2.2150)\n"
     ]
    }
   ],
   "source": [
    "from torch.autograd import Variable\n",
    "\n",
    "target = Variable(torch.LongTensor([5]))\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "loss = criterion(out, target)\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CrossEntropyLoss is not so transparent instead, it surely incorporates the one hot encoding (which is required in TF), and I am quite sure that it performs the SoftMax on the outputs automatically (I will do a test later on).\n",
    "\n",
    "Having defined the loss, we can backprop directly on it using *.backward()* as usual, calculating the gradients (remember also to clean them each time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "net.zero_grad() # Clear the gradients\n",
    "loss.backward() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also follow the computation backward, identifying which functions are called."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<NllLossBackward object at 0x10edf0f98>\n",
      "<LogSoftmaxBackward object at 0x10edf07f0>\n",
      "<ReluBackward object at 0x10edf0f98>\n"
     ]
    }
   ],
   "source": [
    "print(loss.grad_fn)\n",
    "print(loss.grad_fn.next_functions[0][0])\n",
    "print(loss.grad_fn.next_functions[0][0].next_functions[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that, as I suspected, the CrossEntropyLoss automatically performs a LogSoftmax operation.\n",
    "\n",
    "Given that PyTorch allows a great control over the gradients, we can inspect them directly (not possible in TF):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.00000e-02 *\n",
      "       [ 0.6627,  0.4830, -0.9244,  0.8536,  0.3521,  0.7296,  1.8234,\n",
      "         2.5440,  0.5693,  0.9661,  0.7899, -1.4543,  0.8642, -0.4157,\n",
      "        -2.1407, -2.1223,  0.4236, -1.0479,  1.6950,  2.3712, -2.3934,\n",
      "        -1.7147, -2.8960,  1.3432, -2.8582, -0.8453, -0.7836, -0.2788,\n",
      "        -1.1999, -0.0280, -1.2987,  0.1399])\n"
     ]
    }
   ],
   "source": [
    "print(net.conv1.bias.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last missing piece is the optimizer: we could use the same strategy as in the previous part with Gradient Descent, just by subtracting the gradients scaled of a given learning rate. But this time, instead, we will use the built-in optim package, which includes not only SGD but also other optimizers. \n",
    "\n",
    "How optimizer are handled in PyTorch is quite similar to TF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01)\n",
    "\n",
    "# This is one pass of the optimizer\n",
    "optimizer.zero_grad()                 # Clear the gradients\n",
    "output = net(random_input)            # Compute outputs\n",
    "loss = criterion(output, target)      # Compute loss\n",
    "loss.backward()                       # Backward pass\n",
    "optimizer.step()                      # Optimization step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it, all we need to know to implement basic neural networks is here. In the next part we will train this architecture on the MNIST dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}