PyTorch-Techniques/52-Input-Shape-of-Conv-Layer-and-Linear-Layer-of-PyTorch-Neural-Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input Shape of Tensor for PyTorch Neural-Network - Conv-Layer and-Linear-Layer\n",
    "\n",
    "### [Link to my Youtube Video Explaining this whole Notebook](https://www.youtube.com/watch?v=qQ6xbv5kPxE&list=PLxqBkZuBynVQqJTE9nRM2p7Tb12TDPlnq&index=8)\n",
    "\n",
    "[![Imgur](https://imgur.com/NJDJmrT.png)](https://www.youtube.com/watch?v=qQ6xbv5kPxE&list=PLxqBkZuBynVQqJTE9nRM2p7Tb12TDPlnq&index=8)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch import Tensor\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc = nn.Linear(in_features=4, out_features=3, bias=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([32, 1, 5, 15, 15])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.Size([16])\n",
    "    # 1d Tensor : [batch_size]\n",
    "    # used for target labels or predictions.\n",
    "torch.Size([16, 256])\n",
    "    # 2D- Tensor : [batch_size, num_features (aka: C * H * W)]\n",
    "    # use for nn.Linear() input.\n",
    "torch.Size([10, 1, 2048])\n",
    "    # 3-D Tensor : [batch_size, channels, num_features (aka: H * W)]\n",
    "    # when used as nn.Conv1d() input.\n",
    "    # (but [seq_len, batch_size, num_features]\n",
    "    # if feeding an RNN).\n",
    "torch.Size([16, 3, 28, 28])\n",
    "    # 4-D Tensor : [batch_size, channels, height, width]\n",
    "    # use for nn.Conv2d() input.\n",
    "torch.Size([32, 1, 5, 15, 15])\n",
    "    # 5D-Tensor: [batch_size, channels, depth, height, width]\n",
    "    # use for nn.Conv3d() input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I want to pass in a 28 x 28 image into a linear layer\n",
    "\n",
    "Lets say my `image` vector is of size 28 * 28\n",
    "\n",
    "But this 28 x 28 pixel image can’t be input as a [28, 28] tensor and nn.Linear() will accept only following form of Tensor\n",
    "\n",
    "#### [batch_size, num_features]\n",
    "\n",
    "Hence, in nn.Linear() layer the [28, 28] tensor will be interpreted as 28 batches and 28 num_features.\n",
    "\n",
    "So to correct this, I need to reshape the input vector of [28, 28] to the form of [batch_size, num_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 28, 28])\n",
      "torch.Size([1, 784])\n"
     ]
    }
   ],
   "source": [
    "batch_size = 1\n",
    "\n",
    "input_image = torch.randn(1, 28, 28)\n",
    "\n",
    "print(input_image.size())\n",
    "\n",
    "input_image = input_image.view(batch_size, -1)\n",
    "\n",
    "print(input_image.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 10])\n"
     ]
    }
   ],
   "source": [
    "fc = torch.nn.Linear(784, 10)\n",
    "\n",
    "# Pass in the simulated image to the layer.\n",
    "\n",
    "output = fc(input_image)\n",
    "print(output.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example from PyTorch [Official Documentation](https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([32, 10])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Net(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        # 1 input image channel, 6 output channels, 5x5 square convolution\n",
    "        # kernel\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        # an affine operation: y = Wx + b\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension. And 16*5*5 gives me 400\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Max pooling over a (2, 2) window\n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
    "        # If the size is a square, you can specify with a single number\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n",
    "        # print('4-D Tensor Shape of x BEFORE Linear layer & before reshaping ', x.shape)  \n",
    "        \n",
    "        # x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension\n",
    "        x = x.view(x.shape[0], -1)\n",
    "        \n",
    "        # print('Shape of x AFTER re-shaping before feeding to first Linear layer ', x.shape)\n",
    "        \n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "net = Net()\n",
    "\n",
    "input = torch.randn(32, 1, 32, 32)\n",
    "\n",
    "output = net(input)\n",
    "output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Another example of a Random Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of x BEFORE Linear layer  torch.Size([32, 32, 7, 7])\n",
      "Shape of x AFTER the first Linear layer  torch.Size([32, 10])\n"
     ]
    }
   ],
   "source": [
    "class CNNClassifier(torch.nn.Module):\n",
    "    def __init__(self, C=10):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 16, 3)\n",
    "        self.maxpool = nn.MaxPool2d(kernel_size=3, padding=1)\n",
    "        self.conv2 = nn.Conv2d(16, 32, 3)\n",
    "        \n",
    "        self.linear1 = nn.Linear(32 * 7 * 7, C) #1568\n",
    "        self.softmax1 = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.maxpool(F.leaky_relu(x))\n",
    "        x = self.conv2(x)\n",
    "        x = self.maxpool(F.leaky_relu(x))\n",
    "        print('Shape of x BEFORE Linear layer ', x.shape)\n",
    "        \n",
    "        fc1 = self.linear1(x.view(x.shape[0], -1))\n",
    "        \n",
    "        print('Shape of x AFTER the first Linear layer ', fc1.shape)\n",
    "        \n",
    "        final_layer = self.softmax1(fc1)\n",
    "        return final_layer\n",
    "    \n",
    "net = CNNClassifier()\n",
    "input = torch.randn(32, 3, 64, 64)\n",
    "out = net(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summarize - The rule\n",
    "\n",
    "## For transitioning from a convolutional layer output to a linear layer input - I must resize Conv Layer output which is a 4d-Tensor to a 2d-Tensor using view.\n",
    "\n",
    "#### An example, a conv output of [32, 21, 50, 50] should be “flattened” to become a [32, 21 * 50 * 50] tensor. \n",
    "\n",
    "#### And the in_features of the linear layer should also be set to [21 * 50 * 50]."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "f9f85f796d01129d0dd105a088854619f454435301f6ffec2fea96ecbd9be4ac"
  },
  "kernelspec": {
   "display_name": "Python 3.9.10 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}