LeNet-5 Implementation Using Pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Network (CNN) for MNIST classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import time\n",
    "import platform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pytorch version: 1.0.0\n",
      "cuda version: 9.0.176\n",
      "Python version: 3.6.8\n"
     ]
    }
   ],
   "source": [
    "print(f'Pytorch version: {torch.__version__}')\n",
    "print(f'cuda version: {torch.version.cuda}')\n",
    "print(f'Python version: {platform.python_version()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load MNIST data and normalize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from MNISTtools import load, show"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "xtrain, ltrain = load(dataset='training', path='dataset/')\n",
    "xtest, ltest = load(dataset='testing', path='dataset/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize_MNIST_images(x):\n",
    "    '''\n",
    "    Args:\n",
    "        x: data\n",
    "    '''\n",
    "    x_norm = x.astype(np.float32)\n",
    "    return x_norm*2/255-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalization\n",
    "xtrain = normalize_MNIST_images(xtrain)\n",
    "xtest = normalize_MNIST_images(xtest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshape\n",
    "Torch expects that the input of a convolutional layer is stored in the following format\n",
    "`Batch size × Number of input channels × Image width × Image height`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of xtrain after reshape is (28, 28, 1, 60000).\n",
      "shape of xtest after reshape is (28, 28, 1, 10000).\n"
     ]
    }
   ],
   "source": [
    "# reshape to 3d\n",
    "xtrain = xtrain.reshape([28,28,-1])[:,:,None,:]\n",
    "xtest = xtest.reshape([28,28,-1])[:,:,None,:]\n",
    "print(f'shape of xtrain after reshape is {xtrain.shape}.')\n",
    "print(f'shape of xtest after reshape is {xtest.shape}.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of xtrain after moveaxis is (60000, 1, 28, 28).\n",
      "shape of xtest after moveaxis is (10000, 1, 28, 28).\n"
     ]
    }
   ],
   "source": [
    "# moveaxis\n",
    "xtrain = np.moveaxis(xtrain, (2,3), (1,0))\n",
    "xtest = np.moveaxis(xtest, (2,3), (1,0))\n",
    "print(f'shape of xtrain after moveaxis is {xtrain.shape}.')\n",
    "print(f'shape of xtest after moveaxis is {xtest.shape}.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wrap all the data into torch Tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "xtrain = torch.from_numpy(xtrain)\n",
    "ltrain = torch.from_numpy(ltrain)\n",
    "xtest = torch.from_numpy(xtest)\n",
    "ltest = torch.from_numpy(ltest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "xtrain_gpu = xtrain.to(device)\n",
    "ltrain_gpu = ltrain.to(device)\n",
    "xtest_gpu = xtest.to(device)\n",
    "ltest_gpu = ltest.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## LeNet -5 network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Convolutional layers can be created as `nn.Conv2d(N, C, K)`. For input images of size `W×H`, the output feature maps have size `[W−K+1]x[H−K+1]`.  \n",
    "\n",
    "* Maxpooling is implemented like any other non-linear function (such as ReLU or softmax). For input images of size `W×H`, the output feature maps have size `[W/L]×[H/L]`.  \n",
    "\n",
    "* A fully connected layer can be created as `nn.Linear(M, N)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Architecture:  \n",
    "\n",
    "(a) a convolutional layer connecting the input image to `6` feature maps with `5×5` convolutions (`K=5`) and followed by ReLU and maxpooling (`L=2`)  \n",
    "\n",
    "(b) a convolutional layer connecting the `6` input channels to `16` output channels with `5×5` convolutions and followed by ReLU and maxpooling (`L=2`)  \n",
    "\n",
    "(c) a fully-connected layer connecting `16` feature maps to `120` output units and followed by ReLU  \n",
    "\n",
    "(d) a fully-connected layer connecting `120` inputs to `84` output units and followed by ReLU  \n",
    "\n",
    "(e) a final linear layer connecting `84` inputs to `10` linear outputs (one for each of our digits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First layer  \n",
    "* input: `(28, 28, 1)`  \n",
    "* after *padding*: `(32, 32, 1)`\n",
    "* after convolution(kernel=`5x5`): `(28, 28, 6)` where `28=32-5+1`  \n",
    "* after ReLU: `(28, 28, 6)`  \n",
    "* after maxpooling(stride=`2x2`): `(14, 14, 6)` $\\Rightarrow$ **OUTPUT**  \n",
    "\n",
    "\n",
    "Second layer\n",
    "* input: `(14, 14, 6)`\n",
    "* after convolution(kernel=`5x5`): `(10, 10, 16)`\n",
    "* after ReLU: `(10, 10, 16)`  \n",
    "* after maxpooling(stride=`2x2`): `(5, 5, 16)` $\\Rightarrow$ **OUTPUT**  \n",
    "\n",
    "\n",
    "Third layer\n",
    "* input: `(5, 5, 16)` $\\Rightarrow$ `5x5x16=400`  \n",
    "* after fully-connected: `(120, 1)`\n",
    "* after ReLU: `(120, 1)` $\\Rightarrow$ **OUTPUT**  \n",
    "\n",
    "\n",
    "Fourth layer\n",
    "* input: `(120, 1)`\n",
    "* after fully-connected: `(84, 1)`\n",
    "* after ReLU: `(84, 1)` $\\Rightarrow$ **OUTPUT**  \n",
    "\n",
    "\n",
    "Fifth layer\n",
    "* input: `(84, 1)`\n",
    "* after fully-connected: `(10, 1)`\n",
    "* after ReLU: `(10, 1)` $\\Rightarrow$ **OUTPUT**  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNet(nn.Module):\n",
    "\n",
    "    # network structure\n",
    "    def __init__(self):\n",
    "        super(LeNet, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5, padding=2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.fc1   = nn.Linear(16*5*5, 120)\n",
    "        self.fc2   = nn.Linear(120, 84)\n",
    "        self.fc3   = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        '''\n",
    "        One forward pass through the network.\n",
    "        \n",
    "        Args:\n",
    "            x: input\n",
    "        '''\n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))\n",
    "        x = x.view(-1, self.num_flat_features(x))\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "    def num_flat_features(self, x):\n",
    "        '''\n",
    "        Get the number of features in a batch of tensors `x`.\n",
    "        '''\n",
    "        size = x.size()[1:]\n",
    "        return np.prod(size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the network structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LeNet(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "net = LeNet()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the network parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv1.weight torch.Size([6, 1, 5, 5]) True\n",
      "conv1.bias torch.Size([6]) True\n",
      "conv2.weight torch.Size([16, 6, 5, 5]) True\n",
      "conv2.bias torch.Size([16]) True\n",
      "fc1.weight torch.Size([120, 400]) True\n",
      "fc1.bias torch.Size([120]) True\n",
      "fc2.weight torch.Size([84, 120]) True\n",
      "fc2.bias torch.Size([84]) True\n",
      "fc3.weight torch.Size([10, 84]) True\n",
      "fc3.bias torch.Size([10]) True\n"
     ]
    }
   ],
   "source": [
    "for name, param in net.named_parameters():\n",
    "    print(name, param.size(), param.requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The accuracy without backprop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# avoid tracking for gradient during testing and then save some computation time\n",
    "with torch.no_grad():\n",
    "    yinit = net(xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(8.9300)\n"
     ]
    }
   ],
   "source": [
    "_, lpred = yinit.max(1)\n",
    "print(100 * (ltest == lpred).float().mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ltest == lpred` generates a tensor with values of `0` and `1`, where `0` means inequal and `1` means equal. Therefore, `(ltest == lpred).float().mean()` implies the accuracy which is **11.55%**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Mini-Batch) Stochastic Gradient Descent (SGD) with cross-entropy and momentum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: PyTorch’s CrossEntropyLoss is the composition of a softmax activation with the standard cross-entropy loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backprop_deep(xtrain, ltrain, net, T, B=100, gamma=.001, rho=.9):\n",
    "    '''\n",
    "    Backprop.\n",
    "    \n",
    "    Args:\n",
    "        xtrain: training samples\n",
    "        ltrain: testing samples\n",
    "        net: neural network\n",
    "        T: number of epochs\n",
    "        B: minibatch size\n",
    "        gamma: step size\n",
    "        rho: momentum\n",
    "    '''\n",
    "    N = xtrain.size()[0]     # Training set size\n",
    "    NB = N//B                # Number of minibatches\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    optimizer = torch.optim.SGD(net.parameters(), lr=gamma, momentum=rho)\n",
    "    \n",
    "    for epoch in range(T):\n",
    "        running_loss = 0.0\n",
    "        shuffled_indices = np.random.permutation(NB)\n",
    "        for k in range(NB):\n",
    "            # Extract k-th minibatch from xtrain and ltrain\n",
    "            minibatch_indices = range(shuffled_indices[k]*B, (shuffled_indices[k]+1)*B)\n",
    "            inputs = xtrain[minibatch_indices]\n",
    "            labels = ltrain[minibatch_indices]\n",
    "\n",
    "            # Initialize the gradients to zero\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            # Forward propagation\n",
    "            outputs = net(inputs)\n",
    "\n",
    "            # Error evaluation\n",
    "            loss = criterion(outputs, labels)\n",
    "\n",
    "            # Back propagation\n",
    "            loss.backward()\n",
    "\n",
    "            # Parameter update\n",
    "            optimizer.step()\n",
    "\n",
    "            # Print averaged loss per minibatch every 100 mini-batches\n",
    "            # Compute and print statistics\n",
    "            with torch.no_grad():\n",
    "                running_loss += loss.item()\n",
    "            if k % 100 == 99:\n",
    "                print('[%d, %5d] loss: %.3f' %\n",
    "                      (epoch + 1, k + 1, running_loss / 100))\n",
    "                running_loss = 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = LeNet()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,   100] loss: 2.301\n",
      "[1,   200] loss: 2.291\n",
      "[1,   300] loss: 2.277\n",
      "[1,   400] loss: 2.250\n",
      "[1,   500] loss: 2.174\n",
      "[1,   600] loss: 1.864\n",
      "[2,   100] loss: 1.022\n",
      "[2,   200] loss: 0.640\n",
      "[2,   300] loss: 0.535\n",
      "[2,   400] loss: 0.435\n",
      "[2,   500] loss: 0.395\n",
      "[2,   600] loss: 0.341\n",
      "[3,   100] loss: 0.293\n",
      "[3,   200] loss: 0.293\n",
      "[3,   300] loss: 0.260\n",
      "[3,   400] loss: 0.242\n",
      "[3,   500] loss: 0.227\n",
      "[3,   600] loss: 0.210\n",
      "It takes 36.121122 seconds.\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "backprop_deep(xtrain, ltrain, net, T=3)\n",
    "end = time.time()\n",
    "print(f'It takes {end-start:.6f} seconds.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate on the testing dataset (CPU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,   100] loss: 0.182\n",
      "[2,   100] loss: 0.166\n",
      "[3,   100] loss: 0.151\n",
      "It takes 5.754438 seconds.\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "backprop_deep(xtest, ltest, net, T=3)\n",
    "end = time.time()\n",
    "print(f'It takes {end-start:.6f} seconds.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = net(xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(95.4200)\n"
     ]
    }
   ],
   "source": [
    "print(100 * (ltest==y.max(1)[1]).float().mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy for 3 epochs is **95.42%**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network on GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "net_gpu = LeNet().to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,   100] loss: 2.298\n",
      "[1,   200] loss: 2.288\n",
      "[1,   300] loss: 2.274\n",
      "[1,   400] loss: 2.253\n",
      "[1,   500] loss: 2.201\n",
      "[1,   600] loss: 2.031\n",
      "[2,   100] loss: 1.437\n",
      "[2,   200] loss: 0.852\n",
      "[2,   300] loss: 0.606\n",
      "[2,   400] loss: 0.470\n",
      "[2,   500] loss: 0.402\n",
      "[2,   600] loss: 0.357\n",
      "[3,   100] loss: 0.311\n",
      "[3,   200] loss: 0.294\n",
      "[3,   300] loss: 0.251\n",
      "[3,   400] loss: 0.219\n",
      "[3,   500] loss: 0.227\n",
      "[3,   600] loss: 0.210\n",
      "[4,   100] loss: 0.194\n",
      "[4,   200] loss: 0.181\n",
      "[4,   300] loss: 0.164\n",
      "[4,   400] loss: 0.168\n",
      "[4,   500] loss: 0.173\n",
      "[4,   600] loss: 0.151\n",
      "[5,   100] loss: 0.143\n",
      "[5,   200] loss: 0.142\n",
      "[5,   300] loss: 0.131\n",
      "[5,   400] loss: 0.136\n",
      "[5,   500] loss: 0.127\n",
      "[5,   600] loss: 0.115\n",
      "[6,   100] loss: 0.116\n",
      "[6,   200] loss: 0.122\n",
      "[6,   300] loss: 0.110\n",
      "[6,   400] loss: 0.105\n",
      "[6,   500] loss: 0.106\n",
      "[6,   600] loss: 0.104\n",
      "[7,   100] loss: 0.111\n",
      "[7,   200] loss: 0.094\n",
      "[7,   300] loss: 0.097\n",
      "[7,   400] loss: 0.092\n",
      "[7,   500] loss: 0.097\n",
      "[7,   600] loss: 0.087\n",
      "[8,   100] loss: 0.081\n",
      "[8,   200] loss: 0.091\n",
      "[8,   300] loss: 0.089\n",
      "[8,   400] loss: 0.086\n",
      "[8,   500] loss: 0.083\n",
      "[8,   600] loss: 0.089\n",
      "[9,   100] loss: 0.071\n",
      "[9,   200] loss: 0.091\n",
      "[9,   300] loss: 0.087\n",
      "[9,   400] loss: 0.077\n",
      "[9,   500] loss: 0.074\n",
      "[9,   600] loss: 0.080\n",
      "[10,   100] loss: 0.068\n",
      "[10,   200] loss: 0.072\n",
      "[10,   300] loss: 0.072\n",
      "[10,   400] loss: 0.078\n",
      "[10,   500] loss: 0.078\n",
      "[10,   600] loss: 0.076\n",
      "It takes 13.838324 seconds.\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "backprop_deep(xtrain_gpu, ltrain_gpu, net_gpu, T=10)\n",
    "end = time.time()\n",
    "print(f'It takes {end-start:.6f} seconds.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Re-evaluate on the testing dataset (GPU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = net_gpu(xtest_gpu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(97.9800)\n"
     ]
    }
   ],
   "source": [
    "print(100 * (ltest==y.max(1)[1].cpu()).float().mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy for 10 epochs is **97.98%** which is higher than that for 3 epochs, **95.42%**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,   100] loss: 0.063\n",
      "[2,   100] loss: 0.059\n",
      "[3,   100] loss: 0.052\n",
      "[4,   100] loss: 0.050\n",
      "[5,   100] loss: 0.049\n",
      "[6,   100] loss: 0.045\n",
      "[7,   100] loss: 0.046\n",
      "[8,   100] loss: 0.043\n",
      "[9,   100] loss: 0.042\n",
      "[10,   100] loss: 0.038\n",
      "It takes 2.311700 seconds.\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "backprop_deep(xtest_gpu, ltest_gpu, net_gpu, T=10)\n",
    "end = time.time()\n",
    "print(f'It takes {end-start:.6f} seconds.')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}