Chapter-wise code/Code - PyTorch/1. Intro to PyTorch/02. Autograd_Automatic_Differentiation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "02. Autograd: Automatic Differentiation.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyO6nwQnQT5Ts5p+5E5mDbuV",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/purvasingh96/pytorch-examples/blob/master/Basics/02.%20Autograd_Automatic_Differentiation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M4CBwliKi3lO",
        "colab_type": "text"
      },
      "source": [
        "# Autograd : Automatic Differentiation\n",
        "The `autograd` package provides automatic differentiation for all operations on Tensors. \n",
        "\n",
        "### Tensor\n",
        "\n",
        "* `torch.Tensor` is the central class of the package\n",
        "* Setting `requires_grad` as `True`, tensor will start to track all operations on it.\n",
        "* Call `.backward()` to compute gradients automatically.\n",
        "* While training neural network, to prevent tracking of history, use `torch.no_grad()`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aH-X8bPSnkGD",
        "colab_type": "code",
        "outputId": "1502f6d5-fb83-413d-97e0-59339e06f894",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "import torch\n",
        "\n",
        "x = torch.ones(2,3, requires_grad=True)\n",
        "print(x)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1., 1., 1.],\n",
            "        [1., 1., 1.]], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cznW0FfeomML",
        "colab_type": "code",
        "outputId": "d3adff40-9649-4f83-e443-f7aa134f0256",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# y = 2x\n",
        "y = x*2\n",
        "print('operation 1 : ', y)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "operation 1 :  tensor([[2., 2., 2.],\n",
            "        [2., 2., 2.]], grad_fn=<MulBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RnYbniy7o39q",
        "colab_type": "code",
        "outputId": "05a4d6e7-a537-49cf-d71a-8a00d8811b22",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# y = 2x\n",
        "# z = 32x^2\n",
        "z = y*y*8\n",
        "print('operation 2 : ', z)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "operation 2 :  tensor([[32., 32., 32.],\n",
            "        [32., 32., 32.]], grad_fn=<MulBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I3x_pNSZpIVg",
        "colab_type": "code",
        "outputId": "f6960f51-e428-4ba5-804d-645fca4c2fb6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "tensor_1 = torch.rand(2,2)\n",
        "tensor_1 = ((tensor_1*3)/(tensor_1 -1))\n",
        "print(tensor_1.requires_grad)\n",
        "tensor_1.requires_grad_(True)\n",
        "print(tensor_1.requires_grad)\n",
        "tensor_2 = tensor_1+6\n",
        "# mean tensor = (32x^2)/6\n",
        "mean_tensor = z.mean()\n",
        "\n",
        "print(z.mean())\n",
        "print(tensor_2.grad_fn)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "False\n",
            "True\n",
            "tensor(32., grad_fn=<MeanBackward0>)\n",
            "<AddBackward0 object at 0x7ff940cff588>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tATg3ssQq4Kb",
        "colab_type": "text"
      },
      "source": [
        "### Backward() and requires_grad()\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ap0M4H8Vq-f6",
        "colab_type": "code",
        "outputId": "e364da1c-4e3a-4944-cdac-e782a7c1f8a0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 326
        }
      },
      "source": [
        "mean_tensor.backward()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-28-feedfd663480>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmean_tensor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m    164\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \"\"\"\n\u001b[0;32m--> 166\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m     97\u001b[0m     Variable._execution_engine.run_backward(\n\u001b[1;32m     98\u001b[0m         \u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 99\u001b[0;31m         allow_unreachable=True)  # allow_unreachable flag\n\u001b[0m\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mRuntimeError\u001b[0m: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time."
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mfhotacNrEBZ",
        "colab_type": "code",
        "outputId": "468b936d-13fa-41bc-8758-682377113dd9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "'''\n",
        "Calculating d(x)/dx\n",
        "x = 1/6(32*x^2)\n",
        "d(x)/dx = 1/6(64*x); x= 1; 64/6; 10.6667\n",
        "'''\n",
        "x.grad"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[10.6667, 10.6667, 10.6667],\n",
              "        [10.6667, 10.6667, 10.6667]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wBqm1P-l0KmI",
        "colab_type": "text"
      },
      "source": [
        "# Autograd\n",
        "Main idea behind differentiation is to calculate the vector-jacobian product.<br>\n",
        "<img src=\"https://github.com/purvasingh96/pytorch-examples/blob/master/Basics/images/jacobian_product.png?raw=1\"></img><br><br>\n",
        "This characteristic of vector-Jacobian product makes it very convenient to feed external gradients into a model that has non-scalar output.<br>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XMghJmwW1k2I",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f0680089-2e12-4cf7-b43c-b193f507432c"
      },
      "source": [
        "import torch\n",
        "v = torch.tensor([0.1, 1.0, 0.001], dtype=torch.float)\n",
        "x = torch.rand(3, requires_grad=True)\n",
        "# y has a non scalar value\n",
        "y = x*3\n",
        "print(y)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([0.0686, 0.9892, 2.8836], grad_fn=<MulBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eDfEmmFF2Ldn",
        "colab_type": "text"
      },
      "source": [
        "In case y is no longer a scalar. torch.autograd could not compute the full Jacobian directly, but if we just want the vector-Jacobian product, simply pass the vector to backward as argument."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VtlMGp0d2NHN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        },
        "outputId": "d768ab8f-2b34-4cf6-f240-f7d0d1af1909"
      },
      "source": [
        "y.backward(v)\n",
        "print(x.grad)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-5-3a37ff73f036>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mv\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m    164\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \"\"\"\n\u001b[0;32m--> 166\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m     97\u001b[0m     Variable._execution_engine.run_backward(\n\u001b[1;32m     98\u001b[0m         \u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 99\u001b[0;31m         allow_unreachable=True)  # allow_unreachable flag\n\u001b[0m\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mRuntimeError\u001b[0m: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time."
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wkPL4VXx2asA",
        "colab_type": "text"
      },
      "source": [
        "### torch.no_grad() and detach()\n",
        "To stop autograd from tracking history on Tensors with `.requires_grad=True` either by wrapping the code block in with `torch.no_grad()`.<br>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PGno4TxY2hia",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "7a1c7dc4-6dd9-43f1-955a-0ca0f2882e27"
      },
      "source": [
        "print(x.requires_grad)\n",
        "print((x ** 2).requires_grad)\n",
        "\n",
        "with torch.no_grad():\n",
        "  print((x ** 2).requires_grad)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "True\n",
            "True\n",
            "False\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ujcNucjR3QuL",
        "colab_type": "text"
      },
      "source": [
        "Use `.detach()` to get a new Tensor with the same content but that does not require gradients:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vfEcbSBq2_8w",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "fcf7b486-684e-4aae-dd0c-f20046c9e6ce"
      },
      "source": [
        "print(x.requires_grad)\n",
        "y = x.detach()\n",
        "print(y.requires_grad)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "True\n",
            "False\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}