2.Pytorch_Basic of Autograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "### What is AutoGrad?\r\n",
    "- This is automatic Differentiation Engine that helps the neural network to optimize its performance.\r\n",
    "- `torch.autograd()` provides Classes and functions implementing automatic differentiation of arbitary scaler value function.\r\n",
    "- To compute the gradient we are using `.backward()` function that form acyclic graph, which stores the history of computation."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import torch\r\n",
    "\r\n",
    "# Tensor wihtout gradient\r\n",
    "x = torch.rand(3,3)\r\n",
    "print(\"3x3 Tensor:\\n\", x)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "3x3 Tensor:\n",
      " tensor([[0.1670, 0.8011, 0.6163],\n",
      "        [0.0406, 0.3943, 0.6229],\n",
      "        [0.4281, 0.6103, 0.2381]])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "- `required_grad=True` track all the operation on the tensor will be tracked.\r\n",
    "- we will get benefits and usage in the later."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "# Tensor with gradient\r\n",
    "x = torch.rand(3,3, requires_grad=True)\r\n",
    "print(x)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[0.4968, 0.1123, 0.9465],\n",
      "        [0.2396, 0.6592, 0.2948],\n",
      "        [0.4272, 0.5338, 0.1749]], requires_grad=True)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "y = x + 5\r\n",
    "print(\"y Tensor after adding Grad True with x:\\n\", y)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "y Tensor after adding Grad True with x:\n",
      " tensor([[5.4968, 5.1123, 5.9465],\n",
      "        [5.2396, 5.6592, 5.2948],\n",
      "        [5.4272, 5.5338, 5.1749]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "print(y.grad_fn)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<AddBackward0 object at 0x000001CB13B00070>\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "print(y.requires_grad)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "True\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "# tracking a multiplicative gradient\r\n",
    "x = torch.rand(3,3, requires_grad=True)\r\n",
    "y = x * 2\r\n",
    "print(y) # automatically operation are tracked\r\n",
    "print(y.grad_fn)\r\n",
    "\r\n",
    "# Confirm requires_grad\r\n",
    "print(y.requires_grad)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[1.8927, 1.9646, 1.2404],\n",
      "        [0.5084, 1.5763, 1.8005],\n",
      "        [1.6676, 1.8217, 1.4911]], grad_fn=<MulBackward0>)\n",
      "<MulBackward0 object at 0x000001CB13AD3580>\n",
      "True\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "x = torch.ones(2,2, requires_grad=True)\r\n",
    "y = x + 3\r\n",
    "z = y ** 2\r\n",
    "res = z.mean()\r\n",
    "print(\"Z:\\n\",z)\r\n",
    "print(\"Result:\\n\", res)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Z:\n",
      " tensor([[16., 16.],\n",
      "        [16., 16.]], grad_fn=<PowBackward0>)\n",
      "Result:\n",
      " tensor(16., grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "- Two important method `.backward()` => used for backward propogation and `.grad()` used for calculate the gradient."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "res.backward()\r\n",
    "print(x.grad)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[2., 2.],\n",
      "        [2., 2.]])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Computational Graph\r\n",
    "- **Leaf Nodes** : A tensor is leaf tensor if it has `required_grad = False`\r\n",
    "- We populate the gradients of a leaf tensor only using `backward()` and `requires_grad` for that tensor should be `True`. Even if a tensor has `requires_grad=True` but it is created by a user, then also it is a leaf tensor by default.\r\n",
    "\r\n",
    "So, to sum it up:\r\n",
    "1. We can use `backward()` on a tensor, if it is created because of operations performed on tensors which have `requires_grad=True`.\r\n",
    "2. A leaf node with no `grad_fn` cannot have the gradients populated backward.\r\n",
    "3. To populate the gradients, we will need `grad_fn` and the tensor should be a leaf tensor.\r\n",
    "\r\n",
    "First, let’s try some tensor operations with requires_grad=False."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "x = torch.ones(1,1)\r\n",
    "print(x.requires_grad, x.grad_fn, x.is_leaf)\r\n",
    "y = torch.ones(1,1)\r\n",
    "print(y.requires_grad, y.grad_fn, y.is_leaf)\r\n",
    "z = x + y\r\n",
    "print(z)\r\n",
    "print(z.requires_grad, z.grad_fn, z.is_leaf)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "False None True\n",
      "False None True\n",
      "tensor([[2.]])\n",
      "False None True\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "# z.backward() # it will give error like below has"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "```python\r\n",
    "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn\r\n",
    "---------------------------------------------------------------------------\r\n",
    "RuntimeError                              Traceback (most recent call last)\r\n",
    "~\\AppData\\Local\\Temp/ipykernel_12056/148511205.py in <module>\r\n",
    "----> 1 z.backward()\r\n",
    "\r\n",
    "~\\miniconda3\\envs\\pytorch19\\lib\\site-packages\\torch\\_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)\r\n",
    "    253                 create_graph=create_graph,\r\n",
    "    254                 inputs=inputs)\r\n",
    "--> 255         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)\r\n",
    "    256 \r\n",
    "    257     def register_hook(self, hook):\r\n",
    "\r\n",
    "~\\miniconda3\\envs\\pytorch19\\lib\\site-packages\\torch\\autograd\\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)\r\n",
    "    145         retain_graph = create_graph\r\n",
    "    146 \r\n",
    "--> 147     Variable._execution_engine.run_backward(\r\n",
    "    148         tensors, grad_tensors_, retain_graph, create_graph, inputs,\r\n",
    "    149         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag\r\n",
    "\r\n",
    "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn\r\n",
    "```\r\n",
    "\r\n",
    "![](https://debuggercafe.com/wp-content/uploads/2019/11/no_grad.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "`required_grad = True` approach"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "source": [
    "x = torch.ones(1,1, requires_grad=True)\r\n",
    "print(x.requires_grad, x.grad_fn, x.is_leaf)\r\n",
    "y = torch.ones(1,1, requires_grad=True)\r\n",
    "print(y.requires_grad, y.grad_fn, y.is_leaf)\r\n",
    "z = x + y\r\n",
    "print(z) \r\n",
    "print(z.requires_grad, z.grad_fn, z.is_leaf)\r\n",
    "z.backward()\r\n",
    "print(z)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "True None True\n",
      "True None True\n",
      "tensor([[2.]], grad_fn=<AddBackward0>)\n",
      "True <AddBackward0 object at 0x000001CB13AD9340> False\n",
      "tensor([[2.]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "![](https://debuggercafe.com/wp-content/uploads/2019/11/grad_fn.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "orig_nbformat": 4,
  "language_info": {
   "name": "python",
   "version": "3.8.0",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.0 64-bit ('pytorch19': conda)"
  },
  "interpreter": {
   "hash": "884a8f81666a19c0851426c83cd6eaa7b212468ad852fb3caa21591c98d4369f"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}