Pytorch_Variable_and_Autograd.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Pytorch Variable and Autograd.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOSc32KW3sTGFphr7rSBO/U",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ashishpatel26/Pytorch-Learning/blob/main/Pytorch_Variable_and_Autograd.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iErdTCa97l_r"
      },
      "source": [
        "### Variable\r\n",
        "* The difference between pytorch and numpy is that it provides automatic derivation, which can automatically give you the gradient of the parameters you want. This operation is provided by another basic element, Variable.\r\n",
        "![](https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/dynamic_graph.gif)\r\n",
        "\r\n",
        "* A Variable wraps a Tensor. It supports nearly all the API’s defined by a Tensor. \r\n",
        "* Variable also provides a backward method to perform backpropagation. For example, to backpropagate a loss function to train model parameter x, we use a variable loss to store the value computed by a loss function. \r\n",
        "* Then, we call loss.backward which computes the gradients ∂loss∂x for all trainable parameters. PyTorch will store the gradient results back in the corresponding variable x.\r\n",
        "* Variable in torch is to build a computational graph, but this graph is dynamic compared with a static graph in Tensorflow or Theano.So torch does not have placeholder, torch can just pass variable to the computational graph."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tLaavaRK7SvT"
      },
      "source": [
        "import torch\r\n",
        "from torch.autograd import Variable"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "14JXSllJ8O_-"
      },
      "source": [
        "* Build a **tensor**\r\n",
        "* Build a **Variable, usually for compute Gradient**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EwZkKdOF7lMO"
      },
      "source": [
        "tensor = torch.FloatTensor([[5,2],[6, 8]])\r\n",
        "variable = Variable(tensor, requires_grad=True)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nWhDW2z18wlM",
        "outputId": "13c32cd0-6608-4a05-b9d7-64f03ef70104"
      },
      "source": [
        "print(tensor)       # [torch.FloatTensor of size 2x2]\r\n",
        "print(variable)     # [torch.FloatTensor of size 2x2]"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[5., 2.],\n",
            "        [6., 8.]])\n",
            "tensor([[5., 2.],\n",
            "        [6., 8.]], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oybLGR-688YQ"
      },
      "source": [
        "* Till now the **tensor and variable** seem the same.However, the **variable is a part of the graph**, it's a part of the **auto-gradient**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4n1FhSk482E3"
      },
      "source": [
        "t_out = torch.mean(tensor * tensor)\r\n",
        "v_out = torch.mean(variable * variable)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H8fOulGb9PDO",
        "outputId": "5afaf8d2-739d-4795-9735-07341a61e14f"
      },
      "source": [
        "print(t_out)\r\n",
        "print(v_out)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(32.2500)\n",
            "tensor(32.2500, grad_fn=<MeanBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ttlgGwt-90Rj"
      },
      "source": [
        "* Backpropagation from v_out\r\n",
        "```\r\n",
        "v_out = 1 / 4 * sum(variable * variable)\r\n",
        "```\r\n",
        "* the gradients w.r.t the variable, \r\n",
        "```\r\n",
        "d(v_out)/d(variable) = 1/4*2*variable = variable/2\r\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9-oQb6jJ9TPJ",
        "outputId": "70f62db1-f618-4942-9d62-ea8b9be75d7c"
      },
      "source": [
        "v_out.backward()\r\n",
        "print(variable.grad)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[2.5000, 1.0000],\n",
            "        [3.0000, 4.0000]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RkTmcCTH-Jhw"
      },
      "source": [
        "* This is data in **variable format**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1x_Lo-Fz9lxS",
        "outputId": "aab321d0-acb1-4c59-f0fe-84ba753e4b85"
      },
      "source": [
        "print(variable) # variable with require gradient format"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[5., 2.],\n",
            "        [6., 8.]], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GwXdFXF6-W-e"
      },
      "source": [
        "* This is data in **tensor format**.\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KW4sbunB-P1t",
        "outputId": "3b5406a4-86d8-41d2-e2e1-07dd97596776"
      },
      "source": [
        "print(variable.data) # Variable with tensor format on original data"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[5., 2.],\n",
            "        [6., 8.]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ntS0TuM_-u3F"
      },
      "source": [
        "* This is in **numpy format**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aD8W6KqD-esQ",
        "outputId": "74cf8a8c-f371-43eb-e619-b5d3f11b1850"
      },
      "source": [
        "print(variable.data.numpy())"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[5. 2.]\n",
            " [6. 8.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MmZR369F-7Fu"
      },
      "source": [
        "### Torch.AutoGrad\r\n",
        "* `torch.autograd` is PyTorch’s automatic differentiation engine that powers neural network training. In this section, you will get a conceptual understanding of how autograd helps a neural network train.\r\n",
        "\r\n",
        "### Background\r\n",
        "Neural networks (NNs) are a collection of nested functions that are executed on some input data. These functions are defined by parameters (consisting of weights and biases), which in PyTorch are stored in tensors.\r\n",
        "\r\n",
        "Training a NN happens in two steps:\r\n",
        "\r\n",
        "**Forward Propagation**: In forward prop, the NN makes its best guess about the correct output. It runs the input data through each of its functions to make this guess.\r\n",
        "\r\n",
        "**Backward Propagation**: In backprop, the NN adjusts its parameters proportionate to the error in its guess. It does this by traversing backwards from the output, collecting the derivatives of the error with respect to the parameters of the functions (gradients), and optimizing the parameters using gradient descent. For a more detailed walkthrough of backprop, check out this"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eL3z6ygJ-0pO"
      },
      "source": [
        "import torch, torchvision\r\n",
        "model = torchvision.models.resnet18(pretrained=True)\r\n",
        "data = torch.rand(1, 3, 64, 64)\r\n",
        "labels = torch.rand(1, 1000)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jPyH-RPI_ns4"
      },
      "source": [
        "* Next step run the input data through model through each of its layers to make a prediction. This is the **forward pass**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5CsezQOf_XhR"
      },
      "source": [
        "prediction = model(data) # forward pass"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2-bWCMrmAHco"
      },
      "source": [
        "* Now, Model's prediction and corresponding label to calculate the error.\r\n",
        "* Next, Step is to backpropagate this error through network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lYJs8jkX_3mb"
      },
      "source": [
        "loss = (prediction - labels).sum()\r\n",
        "loss.backward() # backward pass"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cib7wgSOAwQe"
      },
      "source": [
        "* Next step is to load an optimiser and this case we are applying SGD with learning rate 0.01 and momentum of 0.9"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fl2pmwMgAfTe"
      },
      "source": [
        "optim = torch.optim.SGD(model.parameters(), lr = 1e-02, momentum=0.9)"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zGlDk2U5BE7k"
      },
      "source": [
        "* Finally, we call `.step()` to initiate gradient descent. The optimizer adjusts each parameter by its gradient stored in `.grad.`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9-05EgrXBCOo"
      },
      "source": [
        "optim.step() #gradient descent"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S53XVeD6BdFt"
      },
      "source": [
        "* _At this point, you have everything you need to train your neural network. The below sections detail the workings of autograd - feel free to skip them._"
      ]
    }
  ]
}