intro-to-pytorch/5-model.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a neural network\n",
    "\n",
    "Neural networks comprise of layers/modules that perform operations on data. \n",
    "The [torch.nn](https://pytorch.org/docs/stable/nn.html) namespace provides all the building blocks you need to \n",
    "build your own neural network. Every module in PyTorch subclasses the [nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html). \n",
    "A neural network is a module itself that consists of other modules (layers). This nested structure allows for\n",
    "building and managing complex architectures easily.\n",
    "\n",
    "In the following sections, we'll build a neural network to classify images in the FashionMNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a hardware device for training\n",
    "\n",
    "We want to be able to train our model on a hardware accelerator like the GPU, \n",
    "if it is available. Let's check to see if \n",
    "[torch.cuda](https://pytorch.org/docs/stable/notes/cuda.html) is available, else we \n",
    "continue to use the CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cuda device\n"
     ]
    }
   ],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print('Using {} device'.format(device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the class\n",
    "\n",
    "We define our neural network by subclassing `nn.Module`, and \n",
    "initialize the neural network layers in `__init__`. Every `nn.Module` subclass implements\n",
    "the operations on input data in the `forward` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.linear_relu_stack = nn.Sequential(\n",
    "            nn.Linear(28*28, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 10),\n",
    "            nn.ReLU()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.flatten(x)\n",
    "        logits = self.linear_relu_stack(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create an instance of `NeuralNetwork`, and move it to the `device`, and print \n",
    "it's structure.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NeuralNetwork(\n",
      "  (flatten): Flatten()\n",
      "  (linear_relu_stack): Sequential(\n",
      "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "    (3): ReLU()\n",
      "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
      "    (5): ReLU()\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "model = NeuralNetwork().to(device)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the model, we pass it the input data. This executes the model's `forward`,\n",
    "along with some [background operations](https://github.com/pytorch/pytorch/blob/270111b7b611d174967ed204776985cefca9c144/torch/nn/modules/module.py#L866). \n",
    "Do not call `model.forward()` directly!\n",
    "\n",
    "Calling the model on the input returns a 10-dimensional tensor with raw predicted values for each class.\n",
    "We get the prediction densities by passing it through an instance of the `nn.Softmax` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted class: tensor([2], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "X = torch.rand(1, 28, 28, device=device)\n",
    "logits = model(X) \n",
    "pred_probab = nn.Softmax(dim=1)(logits)\n",
    "y_pred = pred_probab.argmax(1)\n",
    "print(f\"Predicted class: {y_pred}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model layers\n",
    "\n",
    "Let's break down the layers in the FashionMNIST model. To illustrate it, we \n",
    "will take a sample minibatch of 3 images of size **28x28** and see what happens to it as \n",
    "we pass it through the network. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 28, 28])\n"
     ]
    }
   ],
   "source": [
    "input_image = torch.rand(3,28,28)\n",
    "print(input_image.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Flatten\n",
    "\n",
    "We initialize the [nn.Flatten](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html) \n",
    "layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values (the minibatch dimension (at dim=0) is maintained).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 784])\n"
     ]
    }
   ],
   "source": [
    "flatten = nn.Flatten()\n",
    "flat_image = flatten(input_image)\n",
    "print(flat_image.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Linear \n",
    "\n",
    "The [linear layer](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)\n",
    "is a module that applies a linear transformation on the input using it's stored weights and biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 20])\n"
     ]
    }
   ],
   "source": [
    "layer1 = nn.Linear(in_features=28*28, out_features=20)\n",
    "hidden1 = layer1(flat_image)\n",
    "print(hidden1.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.ReLU\n",
    "\n",
    "Non-linear activations are what create the complex mappings between the model's inputs and outputs.\n",
    "They are applied after linear transformations to introduce *nonlinearity*, helping neural networks\n",
    "learn a wide variety of phenomena.\n",
    "\n",
    "In this model, we use [nn.ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) between our\n",
    "linear layers, but there's other activations to introduce non-linearity in your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before ReLU: tensor([[ 0.2190,  0.1448, -0.5783,  0.1782, -0.4481, -0.2782, -0.5680,  0.1347,\n",
      "          0.1092, -0.7941, -0.2273, -0.4437,  0.0661,  0.2095,  0.1291, -0.4690,\n",
      "          0.0358,  0.3173, -0.0259, -0.4028],\n",
      "        [-0.3531,  0.2385, -0.3172, -0.4717, -0.0382, -0.2066, -0.3859,  0.2607,\n",
      "          0.3626, -0.4838, -0.2132, -0.7623, -0.2285,  0.2409, -0.2195, -0.4452,\n",
      "         -0.0609,  0.4035, -0.4889, -0.4500],\n",
      "        [-0.3651, -0.1240, -0.3222, -0.1072, -0.0112, -0.0397, -0.4105, -0.0233,\n",
      "         -0.0342, -0.5680, -0.4816, -0.8085, -0.3945, -0.0472,  0.0247, -0.3605,\n",
      "         -0.0347,  0.1192, -0.2763,  0.1447]], grad_fn=<AddmmBackward>)\n",
      "\n",
      "\n",
      "After ReLU: tensor([[0.2190, 0.1448, 0.0000, 0.1782, 0.0000, 0.0000, 0.0000, 0.1347, 0.1092,\n",
      "         0.0000, 0.0000, 0.0000, 0.0661, 0.2095, 0.1291, 0.0000, 0.0358, 0.3173,\n",
      "         0.0000, 0.0000],\n",
      "        [0.0000, 0.2385, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.2607, 0.3626,\n",
      "         0.0000, 0.0000, 0.0000, 0.0000, 0.2409, 0.0000, 0.0000, 0.0000, 0.4035,\n",
      "         0.0000, 0.0000],\n",
      "        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0247, 0.0000, 0.0000, 0.1192,\n",
      "         0.0000, 0.1447]], grad_fn=<ReluBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print(f\"Before ReLU: {hidden1}\\n\\n\")\n",
    "hidden1 = nn.ReLU()(hidden1)\n",
    "print(f\"After ReLU: {hidden1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Sequential\n",
    "\n",
    "[nn.Sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) is an ordered \n",
    "container of modules. The data is passed through all the modules in the same order as defined. You can use\n",
    "sequential containers to put together a quick network like `seq_modules`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "seq_modules = nn.Sequential(\n",
    "    flatten,\n",
    "    layer1,\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(20, 10)\n",
    ")\n",
    "input_image = torch.rand(3,28,28)\n",
    "logits = seq_modules(input_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Softmax\n",
    "\n",
    "The last linear layer of the neural network returns `logits` - raw values in \\[`-infty`, `infty`] - which are passed to the\n",
    "[nn.Softmax](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html) module. The logits are scaled to values \\[0, 1\\] representing the model's predicted densities for each class. `dim` parameter indicates the dimension along which the values must sum to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax = nn.Softmax(dim=1)\n",
    "pred_probab = softmax(logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model parameters\n",
    "\n",
    "Many layers inside a neural network are *parameterized*, i.e. have associated weights \n",
    "and biases that are optimized during training. Subclassing `nn.Module` automatically \n",
    "tracks all fields defined inside your model object, and makes all parameters \n",
    "accessible using your model's `parameters()` or `named_parameters()` methods.\n",
    "\n",
    "In this example, we iterate over each parameter, and print its size and a preview of its values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model structure:  NeuralNetwork(\n",
      "  (flatten): Flatten()\n",
      "  (linear_relu_stack): Sequential(\n",
      "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "    (3): ReLU()\n",
      "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
      "    (5): ReLU()\n",
      "  )\n",
      ") \n",
      "\n",
      "\n",
      "Layer: linear_relu_stack.0.weight | Size: torch.Size([512, 784]) | Values : tensor([[-0.0320,  0.0326, -0.0032,  ..., -0.0236, -0.0025, -0.0175],\n",
      "        [ 0.0180,  0.0271, -0.0314,  ..., -0.0094, -0.0170, -0.0257]],\n",
      "       device='cuda:0', grad_fn=<SliceBackward>) \n",
      "\n",
      "Layer: linear_relu_stack.0.bias | Size: torch.Size([512]) | Values : tensor([-0.0134,  0.0036], device='cuda:0', grad_fn=<SliceBackward>) \n",
      "\n",
      "Layer: linear_relu_stack.2.weight | Size: torch.Size([512, 512]) | Values : tensor([[-0.0262,  0.0072, -0.0348,  ..., -0.0374,  0.0345,  0.0374],\n",
      "        [ 0.0439, -0.0101,  0.0218,  ..., -0.0419,  0.0212, -0.0081]],\n",
      "       device='cuda:0', grad_fn=<SliceBackward>) \n",
      "\n",
      "Layer: linear_relu_stack.2.bias | Size: torch.Size([512]) | Values : tensor([ 0.0131, -0.0289], device='cuda:0', grad_fn=<SliceBackward>) \n",
      "\n",
      "Layer: linear_relu_stack.4.weight | Size: torch.Size([10, 512]) | Values : tensor([[ 0.0376, -0.0359, -0.0329,  ..., -0.0057,  0.0040,  0.0307],\n",
      "        [-0.0196, -0.0440,  0.0250,  ...,  0.0335,  0.0024, -0.0207]],\n",
      "       device='cuda:0', grad_fn=<SliceBackward>) \n",
      "\n",
      "Layer: linear_relu_stack.4.bias | Size: torch.Size([10]) | Values : tensor([-0.0287,  0.0321], device='cuda:0', grad_fn=<SliceBackward>) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Model structure: \", model, \"\\n\\n\")\n",
    "\n",
    "for name, param in model.named_parameters():\n",
    "    print(f\"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  }
 ],
  "metadata": {
    "kernelspec": {
      "name": "conda-env-py38_pytorch-py",
      "language": "python",
      "display_name": "py38_pytorch"
    },
    "language_info": {
      "name": "python",
      "version": "3.8.12",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernel_info": {
      "name": "conda-env-py38_pytorch-py"
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}