GettingStartedWithPyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started With PyTorch\n",
    "\n",
    "In this tutorial, we're going to dive into the basics of running [PyTorch](http://pytorch.org/) on Linux, including installation, creating and training a simple neural network that can recognize digits, and finally a more complicated example that uses convolutional neural networks (CNNs) to improve accuracy. This won't be a full introduction to neural networks, but I'll be explaining concepts as they crop up in our code.\n",
    "\n",
    "While a computer with a GPU is not entirely necessary for this tutorial, it's recommended. If you'd prefer following along in a Jupyter notebook, you can find this article on [GitHub](https://github.com/falloutdurham).\n",
    "\n",
    "## Installing PyTorch\n",
    "\n",
    "The easiest way to install PyTorch is to use the Anaconda Python distribution. If you have it installed, getting the latest PyTorch is just entering this on the command line:\n",
    "\n",
    "\t\tconda install pytorch torchvision -c pytorch\n",
    "\t\t\n",
    "If you'd rather use Python's PIP, then for Python 2.6, it's \n",
    "\n",
    "\t\tpip install http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl\n",
    "\t\tpip install torchvision\n",
    "\t\t\n",
    "And for Python 3.6, you'd use:\n",
    "\n",
    "\t\tpip3 install http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl\n",
    "\t\tpip3 install torchvision\t\t\n",
    "\n",
    "Note that if you're wanting to use GPU-accelerated calculations, you'll need to have the [CUDA](https://developer.nvidia.com/cuda-zone) libraries installed as well (and consequently, an [NVIDIA](https://nvidia.com) graphics card). \n",
    "\n",
    "## Our First Model\n",
    "\n",
    "Having got PyTorch installed, we're going to do the \"Hello world\" of deep learning, which is creating a neural network that can look at the images of handwritten digits from a dataset called MNIST and output which number it's looking at. Here's what some of the digits look like:\n",
    "\n",
    "![](mnist.png)\n",
    "\n",
    "Firstly, we're going to need to get our hands on the dataset. While we could download these directly from the [MNIST website](http://yann.lecun.com/exdb/mnist/) and build scaffolding to load them into PyTorch, the framework allows us to download standard reference datasets like MNIST, CIFAR-10, COCO, and others without much fuss.\n",
    "\n",
    "\t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms\n",
    "from torch.autograd import Variable\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "transforms = transforms.Compose([\n",
    "                               transforms.ToTensor(),\n",
    "                               transforms.Normalize((0.1307,), (0.3081,))])\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(\n",
    "        datasets.MNIST('../data', train=True, download=True,\n",
    "               transform=transforms),\n",
    "        batch_size=64, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(\n",
    "        datasets.MNIST('../data', train=False, transform=transforms),\n",
    "        batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code will create two DataLoader objects that will download the MNIST dataset (if not present, so the `test_loader` DataLoader will simply use the images that were downloaded by 'train_loader') and serve up random batches of 64 images from MNIST's collection of 60000. You can also see a `transforms` argument applied to both loaders. PyTorch's `torchvision` package allows you to create a complex pipeline of transformations  for data augmentation that are applied to images as they get pulled out of the DataLoader, including random cropping, rotation, reflection, and scaling. In our example, we're not doing any of that, but we are taking advantage of the pipeline to transform the image data into a tensor (in MNIST's case, this tensor is an array of 1x28x28, as the images are all grayscale 28x28 pixels) and normalizing that tensor to the standard deviation and mean of the MNIST dataset as a whole. This takes us from an array of pixels going form 0…255 to a tensor of values from -1 to 1. We do this because neural network training does a lot better within this smaller range rather than the full integer pixel values.\n",
    " \n",
    "Next, let's create our first neural network by creating a new Python class that inherits from PyTorch's nn.Module:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FirstNet(nn.Module):\n",
    "    def __init__(self,image_size):\n",
    "        super(FirstNet, self).__init__()\n",
    "        self.image_size = image_size\n",
    "        self.fc0 = nn.Linear(image_size, 1000)\n",
    "        self.fc1 = nn.Linear(1000, 50)\n",
    "        self.fc2 = nn.Linear(50, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.view(-1, self.image_size)\n",
    "        x = F.relu(self.fc0(x))\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        return F.log_softmax(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general convention for these network classes is that you create all your layers in the constructor, and then lay out their relationship in the `forward()` method. Here, we're creating a very simple network where all our layers are `Linear`, the classic 'fully-connected' neural network, which applies a linear translation to all input (the values in the layer are initialized randomly). We start with `image_size`, the size of our MNIST images, and the network ends with 10 outputs, corresponding to the 10 digits (zero to nine) that we're attempting to recognize.\n",
    "\n",
    "`forward()` then shows us how an image flows through the network. Firstly, we have to convert the image tensor (1x28x28 once it comes through the transformation pipeline) into a shape that the first layer can understand. We do this via the `view()` method, which in this case _flattens_ the tensor into a shape of 1x784, the shape for the fist linear layer.\n",
    "\n",
    "The next three lines of code all apply the layers to the incoming data in turn, but there's also a 'F.relu()' call happening at each level. What is this? Well, it's an example of an _activation function_. These functions can be applied to outputs of each layer and insert non-linearity into the system. Without them, we'd just essentially have a linear regression model, but with them neural networks gain their power as universal function approximators. There are many different types of activation function, but most modern deep learning architectures will use the ReLU, or Rectified Linear Unit. While this sounds intimidating, it's literally just a function f(x) where f(x) = max(x,0), i.e. the function returns zero if the output is less than zero, or the original output if it's greater than zero. \n",
    "\n",
    "Finally, we use a different activation, softmax, on the output of the final layer which squashes the output in the final layer to be in the range of 0…1 for each of the ten output classes. These will become probability estimates for each class, so to determine the predicted class of an image, we find the class with the probability closest to 1.\n",
    "\n",
    "Creating an instance of the network is done in the traditional Python way of calling the constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = FirstNet(image_size=28*28)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a GPU-enabled machine, you can copy this model to the GPU by calling the `cuda()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FirstNet(\n",
       "  (fc0): Linear(in_features=784, out_features=1000)\n",
       "  (fc1): Linear(in_features=1000, out_features=50)\n",
       "  (fc2): Linear(in_features=50, out_features=10)\n",
       ")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training And Testing\n",
    "\n",
    "Having created our model, we now need to train it. In some frameworks, like Keras, most of this will be handled for you behind the scenes, but PyTorch requires that you write an explicit training procedure. Here's an example, taken from the PyTorch examples:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.SGD(model.parameters(), lr=0.001)\n",
    "\n",
    "def train(epoch):\n",
    "    model.train()\n",
    "    for batch_idx, (data, labels) in enumerate(train_loader):\n",
    "        if torch.cuda.is_available():\n",
    "            data, labels = data.cuda(), labels.cuda()\n",
    "        data, labels = Variable(data), Variable(labels)\n",
    "        optimizer.zero_grad()\n",
    "        output = model(data)\n",
    "        loss = F.nll_loss(output, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        if batch_idx % 100 == 0:\n",
    "            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
    "                epoch, batch_idx * len(data), len(train_loader.dataset),\n",
    "                100. * batch_idx / len(train_loader), loss.data[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While there's a lot going on here, it's fairly straightforward if we take it a line at a time. Firstly, before we create the `train()` method itself, we instantiate our optimizer which will update the values of the layers of the neural network at each step through each batch from the DataLoader, exploring values which will hopefully get more accurate as training continues. \n",
    "\n",
    "There are various different optimizers that you can choose from, including [RMSProp](http://ruder.io/optimizing-gradient-descent/index.html#rmsprop), [AdaGrad](http://ruder.io/optimizing-gradient-descent/index.html#adagrad), and the one most commonly used today, [ADAM](http://ruder.io/optimizing-gradient-descent/index.html#adam). But here we're simply going to use the classic vanilla [Stochastic Gradient Descent](http://ruder.io/optimizing-gradient-descent/index.html#stochasticgradientdescent) with a learning rate of 0.001. The learning rate tells the optimizer how much to move the values in the layers during each pass; too high and your network may bounce around high and low accuracy, while too low may see training take a very long time. 0.001 is a decent starting choice.\n",
    "\n",
    "In the `train()` method itself, we first put the model in training mode and then loop through all the batches in the dataset. For each batch, we copy the image data and the labels (i.e. what digit an image represents) to the GPU if available and reset the optimizer for this batch.\n",
    "\n",
    "The images in this batch are then passed through the model to generate the `output` tensor, our predictions. This is then compared to what they should have been (the `labels`) via a _loss function_. We're using the negative log likelihood loss function here, which is commonly used in classification architectures. \n",
    "\n",
    "We then invoke PyTorch magic. The call to `loss.backward()` calculates the backpropagation, working out the gradient of the loss with respect to the values in the layers (or 'weights'). Then by calling optimizer.step() we adjust the layers using this gradient and the optimizer function.  You can think of this as a ball rolling through a landscape with each step, and we're trying to get to the bottom. Each step, we nudge the network in the direction we think is down. \n",
    "\n",
    "Finally, we print out some debugging information on some batch indices - the current epoch and the loss on the training set.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `test()` method then switches the model into evaluation mode, makes predictions and reports the accuracy of the model. If we run the train/test cycle for 10 iterations (also known as _epochs_), we'll get an accuracy in the area of 80%. Not bad, but we can do better without much effort."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test():\n",
    "    model.eval()\n",
    "    test_loss = 0\n",
    "    correct = 0\n",
    "    for data, target in test_loader:\n",
    "        if torch.cuda.is_available():\n",
    "            data, target = data.cuda(), target.cuda()\n",
    "        data, target = Variable(data, volatile=True), Variable(target)\n",
    "        output = model(data)\n",
    "        test_loss += F.nll_loss(output, target, size_average=False).data[0] # sum up batch loss\n",
    "        pred = output.data.max(1, keepdim=True)[1] # get the index of the max log-probability\n",
    "        correct += pred.eq(target.data.view_as(pred)).cpu().sum()\n",
    "\n",
    "    test_loss /= len(test_loader.dataset)\n",
    "    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
    "        test_loss, correct, len(test_loader.dataset),\n",
    "        100. * correct / len(test_loader.dataset)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutions For The Win!\n",
    "\n",
    "Most computer vision deep learning architectures these days are made up of stacks of _convolutional neural networks (CNNs)_ instead of the fully-connected layers shown above. These networks can be thought of as a group of small filters that pass over the image that are each trained to look for certain things, so one filter might end up recognizing eyes, another might seek out noses, and so on. Here's a very basic CNN network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class CNNNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CNNNet, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n",
    "        self.conv2_drop = nn.Dropout2d()\n",
    "        self.fc1 = nn.Linear(320, 50)\n",
    "        self.fc2 = nn.Linear(50, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(F.max_pool2d(self.conv1(x), 2))\n",
    "        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n",
    "        x = x.view(-1, 320)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.dropout(x, training=self.training)\n",
    "        x = self.fc2(x)\n",
    "        return F.log_softmax(x, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we re-initialize the optimizer and create a new `model` with this network and run again for 10 epochs, we're already at an accuracy above 90%. Aside from the convolutional layers (`conv2d`), the other new concepts introduced here are MaxPooling, which is form of downsampling, and Dropout, which forces the network to randomly discount a number of activations when it's in training mode. This helps the model to train in a more generalizable fashion, i.e learning to discern the structure of what makes a 1 instead of just learning to recognize exact pixel values from the training images.        \n",
    "\n",
    "## Where To Go Next\n",
    "\n",
    "That's it for this tutorial - if you're eager to learn more about the framework, then the [PyTorch tutorials site](http://pytorch.org/tutorials/) has all sorts of examples, from image classification to translating text between different languages. If you're looking to explore deep learning in general using PyTorch, I recommending having a look at the [fast.ai](https://fast.ai) course. It'll take you through all the theory of deep learning while staying focussed on its applications in a very accessible manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FirstNet(\n",
      "  (fc0): Linear(in_features=784, out_features=1000)\n",
      "  (fc1): Linear(in_features=1000, out_features=50)\n",
      "  (fc2): Linear(in_features=50, out_features=10)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ian/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:14: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 2.308621\n",
      "Train Epoch: 1 [6400/60000 (11%)]\tLoss: 2.285088\n",
      "Train Epoch: 1 [12800/60000 (21%)]\tLoss: 2.263519\n",
      "Train Epoch: 1 [19200/60000 (32%)]\tLoss: 2.210358\n",
      "Train Epoch: 1 [25600/60000 (43%)]\tLoss: 2.217742\n",
      "Train Epoch: 1 [32000/60000 (53%)]\tLoss: 2.140791\n",
      "Train Epoch: 1 [38400/60000 (64%)]\tLoss: 2.171224\n",
      "Train Epoch: 1 [44800/60000 (75%)]\tLoss: 2.134961\n",
      "Train Epoch: 1 [51200/60000 (85%)]\tLoss: 2.020994\n",
      "Train Epoch: 1 [57600/60000 (96%)]\tLoss: 1.994740\n",
      "\n",
      "Test set: Average loss: 2.0090, Accuracy: 4920/10000 (49%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 2.060917\n",
      "Train Epoch: 2 [6400/60000 (11%)]\tLoss: 1.953271\n",
      "Train Epoch: 2 [12800/60000 (21%)]\tLoss: 1.864485\n",
      "Train Epoch: 2 [19200/60000 (32%)]\tLoss: 1.877632\n",
      "Train Epoch: 2 [25600/60000 (43%)]\tLoss: 1.686222\n",
      "Train Epoch: 2 [32000/60000 (53%)]\tLoss: 1.716414\n",
      "Train Epoch: 2 [38400/60000 (64%)]\tLoss: 1.759521\n",
      "Train Epoch: 2 [44800/60000 (75%)]\tLoss: 1.707442\n",
      "Train Epoch: 2 [51200/60000 (85%)]\tLoss: 1.568280\n",
      "Train Epoch: 2 [57600/60000 (96%)]\tLoss: 1.506321\n",
      "\n",
      "Test set: Average loss: 1.5291, Accuracy: 6129/10000 (61%)\n",
      "\n",
      "Train Epoch: 3 [0/60000 (0%)]\tLoss: 1.511923\n",
      "Train Epoch: 3 [6400/60000 (11%)]\tLoss: 1.533344\n",
      "Train Epoch: 3 [12800/60000 (21%)]\tLoss: 1.375491\n",
      "Train Epoch: 3 [19200/60000 (32%)]\tLoss: 1.615774\n",
      "Train Epoch: 3 [25600/60000 (43%)]\tLoss: 1.315576\n",
      "Train Epoch: 3 [32000/60000 (53%)]\tLoss: 1.253645\n",
      "Train Epoch: 3 [38400/60000 (64%)]\tLoss: 1.379005\n",
      "Train Epoch: 3 [44800/60000 (75%)]\tLoss: 1.233217\n",
      "Train Epoch: 3 [51200/60000 (85%)]\tLoss: 1.431975\n",
      "Train Epoch: 3 [57600/60000 (96%)]\tLoss: 1.301563\n",
      "\n",
      "Test set: Average loss: 1.2115, Accuracy: 6621/10000 (66%)\n",
      "\n",
      "Train Epoch: 4 [0/60000 (0%)]\tLoss: 1.217490\n",
      "Train Epoch: 4 [6400/60000 (11%)]\tLoss: 1.305362\n",
      "Train Epoch: 4 [12800/60000 (21%)]\tLoss: 1.430542\n",
      "Train Epoch: 4 [19200/60000 (32%)]\tLoss: 1.017697\n",
      "Train Epoch: 4 [25600/60000 (43%)]\tLoss: 1.174440\n",
      "Train Epoch: 4 [32000/60000 (53%)]\tLoss: 0.946163\n",
      "Train Epoch: 4 [38400/60000 (64%)]\tLoss: 0.986592\n",
      "Train Epoch: 4 [44800/60000 (75%)]\tLoss: 1.179967\n",
      "Train Epoch: 4 [51200/60000 (85%)]\tLoss: 1.131385\n",
      "Train Epoch: 4 [57600/60000 (96%)]\tLoss: 1.218150\n",
      "\n",
      "Test set: Average loss: 1.0563, Accuracy: 6810/10000 (68%)\n",
      "\n",
      "Train Epoch: 5 [0/60000 (0%)]\tLoss: 1.237385\n",
      "Train Epoch: 5 [6400/60000 (11%)]\tLoss: 1.014336\n",
      "Train Epoch: 5 [12800/60000 (21%)]\tLoss: 1.123598\n",
      "Train Epoch: 5 [19200/60000 (32%)]\tLoss: 1.042482\n",
      "Train Epoch: 5 [25600/60000 (43%)]\tLoss: 0.982507\n",
      "Train Epoch: 5 [32000/60000 (53%)]\tLoss: 1.164286\n",
      "Train Epoch: 5 [38400/60000 (64%)]\tLoss: 0.956896\n",
      "Train Epoch: 5 [44800/60000 (75%)]\tLoss: 1.185416\n",
      "Train Epoch: 5 [51200/60000 (85%)]\tLoss: 0.766228\n",
      "Train Epoch: 5 [57600/60000 (96%)]\tLoss: 1.084051\n",
      "\n",
      "Test set: Average loss: 0.9694, Accuracy: 6914/10000 (69%)\n",
      "\n",
      "Train Epoch: 6 [0/60000 (0%)]\tLoss: 0.857197\n",
      "Train Epoch: 6 [6400/60000 (11%)]\tLoss: 0.883530\n",
      "Train Epoch: 6 [12800/60000 (21%)]\tLoss: 0.764402\n",
      "Train Epoch: 6 [19200/60000 (32%)]\tLoss: 1.020829\n",
      "Train Epoch: 6 [25600/60000 (43%)]\tLoss: 0.763314\n",
      "Train Epoch: 6 [32000/60000 (53%)]\tLoss: 1.057998\n",
      "Train Epoch: 6 [38400/60000 (64%)]\tLoss: 0.757057\n",
      "Train Epoch: 6 [44800/60000 (75%)]\tLoss: 0.854571\n",
      "Train Epoch: 6 [51200/60000 (85%)]\tLoss: 1.021454\n",
      "Train Epoch: 6 [57600/60000 (96%)]\tLoss: 0.828457\n",
      "\n",
      "Test set: Average loss: 0.9144, Accuracy: 6974/10000 (70%)\n",
      "\n",
      "Train Epoch: 7 [0/60000 (0%)]\tLoss: 0.991422\n",
      "Train Epoch: 7 [6400/60000 (11%)]\tLoss: 0.890321\n",
      "Train Epoch: 7 [12800/60000 (21%)]\tLoss: 1.007158\n",
      "Train Epoch: 7 [19200/60000 (32%)]\tLoss: 0.735566\n",
      "Train Epoch: 7 [25600/60000 (43%)]\tLoss: 1.147203\n",
      "Train Epoch: 7 [32000/60000 (53%)]\tLoss: 1.130080\n",
      "Train Epoch: 7 [38400/60000 (64%)]\tLoss: 0.861833\n",
      "Train Epoch: 7 [44800/60000 (75%)]\tLoss: 0.842657\n",
      "Train Epoch: 7 [51200/60000 (85%)]\tLoss: 1.244288\n",
      "Train Epoch: 7 [57600/60000 (96%)]\tLoss: 0.961094\n",
      "\n",
      "Test set: Average loss: 0.8767, Accuracy: 7017/10000 (70%)\n",
      "\n",
      "Train Epoch: 8 [0/60000 (0%)]\tLoss: 1.019881\n",
      "Train Epoch: 8 [6400/60000 (11%)]\tLoss: 0.934166\n",
      "Train Epoch: 8 [12800/60000 (21%)]\tLoss: 0.776214\n",
      "Train Epoch: 8 [19200/60000 (32%)]\tLoss: 1.003691\n",
      "Train Epoch: 8 [25600/60000 (43%)]\tLoss: 0.841994\n",
      "Train Epoch: 8 [32000/60000 (53%)]\tLoss: 0.864001\n",
      "Train Epoch: 8 [38400/60000 (64%)]\tLoss: 0.617983\n",
      "Train Epoch: 8 [44800/60000 (75%)]\tLoss: 0.811713\n",
      "Train Epoch: 8 [51200/60000 (85%)]\tLoss: 0.950729\n",
      "Train Epoch: 8 [57600/60000 (96%)]\tLoss: 0.866750\n",
      "\n",
      "Test set: Average loss: 0.8487, Accuracy: 7079/10000 (71%)\n",
      "\n",
      "Train Epoch: 9 [0/60000 (0%)]\tLoss: 0.857568\n",
      "Train Epoch: 9 [6400/60000 (11%)]\tLoss: 0.996886\n",
      "Train Epoch: 9 [12800/60000 (21%)]\tLoss: 0.757119\n",
      "Train Epoch: 9 [19200/60000 (32%)]\tLoss: 0.792250\n",
      "Train Epoch: 9 [25600/60000 (43%)]\tLoss: 0.895727\n",
      "Train Epoch: 9 [32000/60000 (53%)]\tLoss: 0.935252\n",
      "Train Epoch: 9 [38400/60000 (64%)]\tLoss: 0.939990\n",
      "Train Epoch: 9 [44800/60000 (75%)]\tLoss: 0.739427\n",
      "Train Epoch: 9 [51200/60000 (85%)]\tLoss: 0.566500\n",
      "Train Epoch: 9 [57600/60000 (96%)]\tLoss: 0.847343\n",
      "\n",
      "Test set: Average loss: 0.8283, Accuracy: 7112/10000 (71%)\n",
      "\n",
      "Train Epoch: 10 [0/60000 (0%)]\tLoss: 1.076067\n",
      "Train Epoch: 10 [6400/60000 (11%)]\tLoss: 0.643881\n",
      "Train Epoch: 10 [12800/60000 (21%)]\tLoss: 0.917296\n",
      "Train Epoch: 10 [19200/60000 (32%)]\tLoss: 0.781996\n",
      "Train Epoch: 10 [25600/60000 (43%)]\tLoss: 0.938146\n",
      "Train Epoch: 10 [32000/60000 (53%)]\tLoss: 0.786229\n",
      "Train Epoch: 10 [38400/60000 (64%)]\tLoss: 0.774557\n",
      "Train Epoch: 10 [44800/60000 (75%)]\tLoss: 0.747954\n",
      "Train Epoch: 10 [51200/60000 (85%)]\tLoss: 0.900513\n",
      "Train Epoch: 10 [57600/60000 (96%)]\tLoss: 0.947276\n",
      "\n",
      "Test set: Average loss: 0.8114, Accuracy: 7134/10000 (71%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(1, 10 + 1):\n",
    "    train(epoch)\n",
    "    test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = CNNNet()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "if torch.cuda.is_available():\n",
    "    model.cuda()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 2.287821\n",
      "Train Epoch: 1 [6400/60000 (11%)]\tLoss: 2.331887\n",
      "Train Epoch: 1 [12800/60000 (21%)]\tLoss: 2.321481\n",
      "Train Epoch: 1 [19200/60000 (32%)]\tLoss: 2.285847\n",
      "Train Epoch: 1 [25600/60000 (43%)]\tLoss: 2.289074\n",
      "Train Epoch: 1 [32000/60000 (53%)]\tLoss: 2.276981\n",
      "Train Epoch: 1 [38400/60000 (64%)]\tLoss: 2.277756\n",
      "Train Epoch: 1 [44800/60000 (75%)]\tLoss: 2.269778\n",
      "Train Epoch: 1 [51200/60000 (85%)]\tLoss: 2.249616\n",
      "Train Epoch: 1 [57600/60000 (96%)]\tLoss: 2.257169\n",
      "\n",
      "Test set: Average loss: 2.2180, Accuracy: 3538/10000 (35%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 2.285938\n",
      "Train Epoch: 2 [6400/60000 (11%)]\tLoss: 2.208218\n",
      "Train Epoch: 2 [12800/60000 (21%)]\tLoss: 2.224085\n",
      "Train Epoch: 2 [19200/60000 (32%)]\tLoss: 2.203454\n",
      "Train Epoch: 2 [25600/60000 (43%)]\tLoss: 2.139344\n",
      "Train Epoch: 2 [32000/60000 (53%)]\tLoss: 2.147911\n",
      "Train Epoch: 2 [38400/60000 (64%)]\tLoss: 2.087396\n",
      "Train Epoch: 2 [44800/60000 (75%)]\tLoss: 2.062924\n",
      "Train Epoch: 2 [51200/60000 (85%)]\tLoss: 2.036571\n",
      "Train Epoch: 2 [57600/60000 (96%)]\tLoss: 2.049504\n",
      "\n",
      "Test set: Average loss: 1.8973, Accuracy: 7246/10000 (72%)\n",
      "\n",
      "Train Epoch: 3 [0/60000 (0%)]\tLoss: 1.997772\n",
      "Train Epoch: 3 [6400/60000 (11%)]\tLoss: 2.003385\n",
      "Train Epoch: 3 [12800/60000 (21%)]\tLoss: 1.898295\n",
      "Train Epoch: 3 [19200/60000 (32%)]\tLoss: 1.839159\n",
      "Train Epoch: 3 [25600/60000 (43%)]\tLoss: 1.850581\n",
      "Train Epoch: 3 [32000/60000 (53%)]\tLoss: 1.762830\n",
      "Train Epoch: 3 [38400/60000 (64%)]\tLoss: 1.647081\n",
      "Train Epoch: 3 [44800/60000 (75%)]\tLoss: 1.573870\n",
      "Train Epoch: 3 [51200/60000 (85%)]\tLoss: 1.535875\n",
      "Train Epoch: 3 [57600/60000 (96%)]\tLoss: 1.556666\n",
      "\n",
      "Test set: Average loss: 1.0909, Accuracy: 7957/10000 (80%)\n",
      "\n",
      "Train Epoch: 4 [0/60000 (0%)]\tLoss: 1.490186\n",
      "Train Epoch: 4 [6400/60000 (11%)]\tLoss: 1.270658\n",
      "Train Epoch: 4 [12800/60000 (21%)]\tLoss: 1.226860\n",
      "Train Epoch: 4 [19200/60000 (32%)]\tLoss: 1.261400\n",
      "Train Epoch: 4 [25600/60000 (43%)]\tLoss: 0.964228\n",
      "Train Epoch: 4 [32000/60000 (53%)]\tLoss: 1.381001\n",
      "Train Epoch: 4 [38400/60000 (64%)]\tLoss: 1.077304\n",
      "Train Epoch: 4 [44800/60000 (75%)]\tLoss: 1.283929\n",
      "Train Epoch: 4 [51200/60000 (85%)]\tLoss: 1.224038\n",
      "Train Epoch: 4 [57600/60000 (96%)]\tLoss: 1.137127\n",
      "\n",
      "Test set: Average loss: 0.6365, Accuracy: 8592/10000 (86%)\n",
      "\n",
      "Train Epoch: 5 [0/60000 (0%)]\tLoss: 1.089653\n",
      "Train Epoch: 5 [6400/60000 (11%)]\tLoss: 0.940471\n",
      "Train Epoch: 5 [12800/60000 (21%)]\tLoss: 0.923951\n",
      "Train Epoch: 5 [19200/60000 (32%)]\tLoss: 0.692761\n",
      "Train Epoch: 5 [25600/60000 (43%)]\tLoss: 1.129929\n",
      "Train Epoch: 5 [32000/60000 (53%)]\tLoss: 1.023723\n",
      "Train Epoch: 5 [38400/60000 (64%)]\tLoss: 0.747783\n",
      "Train Epoch: 5 [44800/60000 (75%)]\tLoss: 0.943883\n",
      "Train Epoch: 5 [51200/60000 (85%)]\tLoss: 0.906017\n",
      "Train Epoch: 5 [57600/60000 (96%)]\tLoss: 0.708689\n",
      "\n",
      "Test set: Average loss: 0.4749, Accuracy: 8886/10000 (89%)\n",
      "\n",
      "Train Epoch: 6 [0/60000 (0%)]\tLoss: 0.905593\n",
      "Train Epoch: 6 [6400/60000 (11%)]\tLoss: 0.957780\n",
      "Train Epoch: 6 [12800/60000 (21%)]\tLoss: 1.313062\n",
      "Train Epoch: 6 [19200/60000 (32%)]\tLoss: 0.734315\n",
      "Train Epoch: 6 [25600/60000 (43%)]\tLoss: 0.767096\n",
      "Train Epoch: 6 [32000/60000 (53%)]\tLoss: 0.876026\n",
      "Train Epoch: 6 [38400/60000 (64%)]\tLoss: 0.865832\n",
      "Train Epoch: 6 [44800/60000 (75%)]\tLoss: 1.011665\n",
      "Train Epoch: 6 [51200/60000 (85%)]\tLoss: 0.819896\n",
      "Train Epoch: 6 [57600/60000 (96%)]\tLoss: 0.809723\n",
      "\n",
      "Test set: Average loss: 0.3969, Accuracy: 8991/10000 (90%)\n",
      "\n",
      "Train Epoch: 7 [0/60000 (0%)]\tLoss: 0.549480\n",
      "Train Epoch: 7 [6400/60000 (11%)]\tLoss: 0.625268\n",
      "Train Epoch: 7 [12800/60000 (21%)]\tLoss: 0.780716\n",
      "Train Epoch: 7 [19200/60000 (32%)]\tLoss: 0.672696\n",
      "Train Epoch: 7 [25600/60000 (43%)]\tLoss: 0.921079\n",
      "Train Epoch: 7 [32000/60000 (53%)]\tLoss: 0.686154\n",
      "Train Epoch: 7 [38400/60000 (64%)]\tLoss: 0.700540\n",
      "Train Epoch: 7 [44800/60000 (75%)]\tLoss: 0.515900\n",
      "Train Epoch: 7 [51200/60000 (85%)]\tLoss: 0.746904\n",
      "Train Epoch: 7 [57600/60000 (96%)]\tLoss: 0.901216\n",
      "\n",
      "Test set: Average loss: 0.3472, Accuracy: 9085/10000 (91%)\n",
      "\n",
      "Train Epoch: 8 [0/60000 (0%)]\tLoss: 0.641703\n",
      "Train Epoch: 8 [6400/60000 (11%)]\tLoss: 0.726754\n",
      "Train Epoch: 8 [12800/60000 (21%)]\tLoss: 0.664848\n",
      "Train Epoch: 8 [19200/60000 (32%)]\tLoss: 0.700821\n",
      "Train Epoch: 8 [25600/60000 (43%)]\tLoss: 0.448349\n",
      "Train Epoch: 8 [32000/60000 (53%)]\tLoss: 0.495703\n",
      "Train Epoch: 8 [38400/60000 (64%)]\tLoss: 0.953885\n",
      "Train Epoch: 8 [44800/60000 (75%)]\tLoss: 0.602276\n",
      "Train Epoch: 8 [51200/60000 (85%)]\tLoss: 0.588009\n",
      "Train Epoch: 8 [57600/60000 (96%)]\tLoss: 0.441581\n",
      "\n",
      "Test set: Average loss: 0.3121, Accuracy: 9132/10000 (91%)\n",
      "\n",
      "Train Epoch: 9 [0/60000 (0%)]\tLoss: 0.751948\n",
      "Train Epoch: 9 [6400/60000 (11%)]\tLoss: 0.516822\n",
      "Train Epoch: 9 [12800/60000 (21%)]\tLoss: 0.662870\n",
      "Train Epoch: 9 [19200/60000 (32%)]\tLoss: 0.467419\n",
      "Train Epoch: 9 [25600/60000 (43%)]\tLoss: 0.676435\n",
      "Train Epoch: 9 [32000/60000 (53%)]\tLoss: 0.764132\n",
      "Train Epoch: 9 [38400/60000 (64%)]\tLoss: 0.584641\n",
      "Train Epoch: 9 [44800/60000 (75%)]\tLoss: 0.596789\n",
      "Train Epoch: 9 [51200/60000 (85%)]\tLoss: 0.736869\n",
      "Train Epoch: 9 [57600/60000 (96%)]\tLoss: 0.704933\n",
      "\n",
      "Test set: Average loss: 0.2860, Accuracy: 9202/10000 (92%)\n",
      "\n",
      "Train Epoch: 10 [0/60000 (0%)]\tLoss: 0.573813\n",
      "Train Epoch: 10 [6400/60000 (11%)]\tLoss: 0.524586\n",
      "Train Epoch: 10 [12800/60000 (21%)]\tLoss: 0.697951\n",
      "Train Epoch: 10 [19200/60000 (32%)]\tLoss: 0.541513\n",
      "Train Epoch: 10 [25600/60000 (43%)]\tLoss: 0.429471\n",
      "Train Epoch: 10 [32000/60000 (53%)]\tLoss: 0.450556\n",
      "Train Epoch: 10 [38400/60000 (64%)]\tLoss: 0.582792\n",
      "Train Epoch: 10 [44800/60000 (75%)]\tLoss: 0.630293\n",
      "Train Epoch: 10 [51200/60000 (85%)]\tLoss: 0.452451\n",
      "Train Epoch: 10 [57600/60000 (96%)]\tLoss: 0.524293\n",
      "\n",
      "Test set: Average loss: 0.2683, Accuracy: 9231/10000 (92%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(1, 10 + 1):\n",
    "    train(epoch)\n",
    "    test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}