Numerical Computation/PyTorch/PyTorch.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "PyTorch.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i2q-I72Kn3YG",
        "colab_type": "text"
      },
      "source": [
        "## What is PyTorch?\n",
        "It's a Python-based scientific computing package targeted at two sets of audiences : \n",
        "- A replacement for NumPy to use the power of GPUs\n",
        "- A deep learning research platform that provides maximum flexibility and speed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q-1rutbFoNjq",
        "colab_type": "text"
      },
      "source": [
        "## Getting Started\n",
        "Tensors : \n",
        "Tensors are similar to the NumPy's ndarrays, with the addition being that tensors can also be used on a GPU to accelerate computing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IRZl0H5DnwsG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z6kzj2C1ohek",
        "colab_type": "text"
      },
      "source": [
        "Construct a 5x3 matrix, uninitialized."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MCKZ89BKomd-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "9cde82a6-2d2d-47cc-fec2-06bc5fe6afd2"
      },
      "source": [
        "x = torch.empty(5,3)\n",
        "print(x)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[5.4652e-36, 0.0000e+00, 3.3631e-44],\n",
            "        [0.0000e+00,        nan, 0.0000e+00],\n",
            "        [1.1578e+27, 1.1362e+30, 7.1547e+22],\n",
            "        [4.5828e+30, 1.2121e+04, 7.1846e+22],\n",
            "        [9.2198e-39, 7.0374e+22, 0.0000e+00]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rdGOnk2SorWd",
        "colab_type": "text"
      },
      "source": [
        "Construct a randomly initialized matrix"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vxuyhIkBo3u0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "437f655c-9b23-4a8a-845f-315017643af2"
      },
      "source": [
        "x = torch.rand(5,3)\n",
        "print(x)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0.9991, 0.8260, 0.8563],\n",
            "        [0.7040, 0.5143, 0.2630],\n",
            "        [0.4226, 0.8316, 0.6310],\n",
            "        [0.1072, 0.0822, 0.1412],\n",
            "        [0.1886, 0.4350, 0.1597]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sercFJbbo_J5",
        "colab_type": "text"
      },
      "source": [
        "Construct a matrix filled zeros and of dtype long"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Tq1QOdWpDDY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "7f2d6e24-3dc0-4ada-ebf2-e0eb374f7e80"
      },
      "source": [
        "x = torch.zeros(5, 3, dtype = torch.long)\n",
        "print(x)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0, 0, 0],\n",
            "        [0, 0, 0],\n",
            "        [0, 0, 0],\n",
            "        [0, 0, 0],\n",
            "        [0, 0, 0]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rARQT6IvpPTd",
        "colab_type": "text"
      },
      "source": [
        "Construct a tensor directly from data :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VYnbXeAhpSCx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "061b7d9d-5621-4902-c720-e9e5e77cc0ad"
      },
      "source": [
        "x = torch.tensor([5.5, 3])\n",
        "print(x)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([5.5000, 3.0000])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PjSld4aTpcuK",
        "colab_type": "text"
      },
      "source": [
        "We can also create a tensor based on an existing tensor. These methods will reuse properties of the input tensor, e.g. dtype, unless new values are provided by the user."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mXkfSI-jplq1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "750d432d-0557-4880-df05-aa877b67dc3e"
      },
      "source": [
        "x = x.new_ones(5, 3, dtype=torch.double)            # new_* methods take in size\n",
        "print(x)\n",
        "\n",
        "y = torch.randn_like(x, dtype = torch.float)       # Override dtype\n",
        "print(y)                                            # Result has the same size"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1., 1., 1.],\n",
            "        [1., 1., 1.],\n",
            "        [1., 1., 1.],\n",
            "        [1., 1., 1.],\n",
            "        [1., 1., 1.]], dtype=torch.float64)\n",
            "tensor([[-0.2649, -0.1494, -0.3615],\n",
            "        [ 1.2624,  0.2052, -0.2961],\n",
            "        [-3.0455, -1.2726, -0.1877],\n",
            "        [-0.8213, -0.3532, -0.7591],\n",
            "        [ 0.5191, -0.9640,  1.3605]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZS3R3AeVp6-K",
        "colab_type": "text"
      },
      "source": [
        "Getting its size : "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NLwfrq28p85E",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "4acbb687-3323-4f48-d871-61eab806692e"
      },
      "source": [
        "print(x.size())\n",
        "print(y.size())"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 3])\n",
            "torch.Size([5, 3])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DScp4durqBVq",
        "colab_type": "text"
      },
      "source": [
        "### Note :\n",
        "\n",
        "`torch.Size` is in fact a tuple, so it supports all tuple operations.\n",
        "Operations :\n",
        "There are multiple syntaxes for operations.\n",
        "\n",
        "Addition operation : "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0sURqqA3q_wP",
        "colab_type": "text"
      },
      "source": [
        "Addition : Syntax 1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OBHDa78_rCoK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "ecc5baf0-b8fc-4480-b671-3fec75fbb817"
      },
      "source": [
        "y = torch.rand(5,3)\n",
        "print(x + y)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1.4421, 1.6687, 1.9086],\n",
            "        [1.5954, 1.6225, 1.3454],\n",
            "        [1.4600, 1.1715, 1.0370],\n",
            "        [1.4059, 1.8558, 1.3631],\n",
            "        [1.6533, 1.6914, 1.0692]], dtype=torch.float64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "twie9HkurMz6",
        "colab_type": "text"
      },
      "source": [
        "Addition : Syntax 2 "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_lmrmKlmrMIl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "69b0fbbf-7ea3-4f3b-e69a-6971e73d88cf"
      },
      "source": [
        "print(torch.add(x,y))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1.4421, 1.6687, 1.9086],\n",
            "        [1.5954, 1.6225, 1.3454],\n",
            "        [1.4600, 1.1715, 1.0370],\n",
            "        [1.4059, 1.8558, 1.3631],\n",
            "        [1.6533, 1.6914, 1.0692]], dtype=torch.float64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5DuZvrcprWGG",
        "colab_type": "text"
      },
      "source": [
        "Addition : Providing an output tensor as argument"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0OihZ_NDrZen",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "2b217417-4b92-4df3-e0d9-d9efc41741ae"
      },
      "source": [
        "result = torch.empty(5, 3)\n",
        "torch.add(x, y, out = result)   # Adds x and y and stores it inside result\n",
        "print(result)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1.4421, 1.6687, 1.9086],\n",
            "        [1.5954, 1.6225, 1.3454],\n",
            "        [1.4600, 1.1715, 1.0370],\n",
            "        [1.4059, 1.8558, 1.3631],\n",
            "        [1.6533, 1.6914, 1.0692]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i4ZA7-wgrvEg",
        "colab_type": "text"
      },
      "source": [
        "Addition using In-place operations : "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8-yQ3jW-rx75",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "ddaad9f5-2cbe-484d-8eb6-568bdce5529e"
      },
      "source": [
        "y.add_(x)   # Adds x to y\n",
        "print(y)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1.4421, 1.6687, 1.9086],\n",
            "        [1.5954, 1.6225, 1.3454],\n",
            "        [1.4600, 1.1715, 1.0370],\n",
            "        [1.4059, 1.8558, 1.3631],\n",
            "        [1.6533, 1.6914, 1.0692]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lvLjVDEXr8it",
        "colab_type": "text"
      },
      "source": [
        "### Note : \n",
        "Any operation that mutates a tensor in-place is post-fixed with an `_`. For example `x.copy_(y)`, `x.t_()`, will change `x`.\n",
        "\n",
        "You can use standard NumPy-like indexing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hL5jqtnMsZYe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "1da62411-57cc-495f-ac83-f8ae16f5783b"
      },
      "source": [
        "print(y)\n",
        "print(y[:, 1])  # Slicing X to get all 1st column values."
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1., 1., 1.],\n",
            "        [1., 1., 1.],\n",
            "        [1., 1., 1.],\n",
            "        [1., 1., 1.],\n",
            "        [1., 1., 1.]], dtype=torch.float64)\n",
            "tensor([1., 1., 1., 1., 1.], dtype=torch.float64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M-qFjx6EszXK",
        "colab_type": "text"
      },
      "source": [
        "Resizing : If you want to resize/reshape tensor, you can use `torch.view`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1x8hza56s4xH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5d6f896a-c958-4c1f-d7d1-f5f3199ec0b6"
      },
      "source": [
        "x = torch.randn(4, 4)   # 4x4 = 16 elements\n",
        "y = x.view(16)          # A flat single dimentional array.\n",
        "z = x.view(-1, 8)       # If -1, it automatically calculates size of first dimension according to the other dimensions.\n",
        "p = x.view(-1, 2, 2)\n",
        "print(x.size(), y.size(), z.size(), p.size())"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8]) torch.Size([4, 2, 2])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5LKPePgetwtZ",
        "colab_type": "text"
      },
      "source": [
        "If you have a one element tensor, use `.item()` to get the value as a Python number."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4UQiO8z-t2mn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "f1ce87f3-047b-42ff-e08c-20d6276d0f73"
      },
      "source": [
        "x = torch.randn(1)\n",
        "print(x)\n",
        "print(x.item()) # returns a normal primitive datatype -> double or float"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([-0.2894])\n",
            "-0.2894309461116791\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1x7o1Qi8uD3C",
        "colab_type": "text"
      },
      "source": [
        "## NumPy Bridge\n",
        "\n",
        "Converting a Torch Tensor to a NumPy array and vice versa : \n",
        "The Torch Tensor and NumPy array will share their underlying memory locations, and changing one will change the other.\n",
        "Converting a Torch Tensor to a NumPy array."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JCxNGYOwuDLd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "79b6294f-6383-42e1-c351-57fbfb29ccfb"
      },
      "source": [
        "a = torch.ones(5)\n",
        "print(a)\n",
        "print(type(a))"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([1., 1., 1., 1., 1.])\n",
            "<class 'torch.Tensor'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "79FAaVrRuXRe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "94f972a8-f3f1-4850-c605-72518a4f308c"
      },
      "source": [
        "b = a.numpy()\n",
        "print(b)\n",
        "print(type(b))"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1. 1. 1. 1. 1.]\n",
            "<class 'numpy.ndarray'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kep15UuKuiYT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "1715a717-213a-4e14-df7a-e026245070fb"
      },
      "source": [
        "a.add_(1)\n",
        "print(a)\n",
        "print(b)    # Even though converted to a NumPy array, they still share the same memory location."
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([2., 2., 2., 2., 2.])\n",
            "[2. 2. 2. 2. 2.]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mrX9z7EJu6sA",
        "colab_type": "text"
      },
      "source": [
        "Converting NumPy Array to Torch Tensor : "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vBqFopKJu9N4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "ce24adaf-5443-470a-b656-26d95641886b"
      },
      "source": [
        "import numpy as np\n",
        "a = np.ones(5)\n",
        "b = torch.from_numpy(a)\n",
        "np.add(a, 1, out = a)\n",
        "print(a)\n",
        "print(b)        # Modifying NumPy array also affects Tensors."
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[2. 2. 2. 2. 2.]\n",
            "tensor([2., 2., 2., 2., 2.], dtype=torch.float64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6eOCisLCvnnN",
        "colab_type": "text"
      },
      "source": [
        "## CUDA Tensors : \n",
        "To check if you have an available GPU, use : "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eBlk7dHgv1hN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e4b1e518-f217-430f-e969-5fc195f31536"
      },
      "source": [
        "torch.cuda.is_available()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ud62y_-VwKzc",
        "colab_type": "text"
      },
      "source": [
        "To check number of GPU's available : "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dHNk0xjmwOct",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8a0b8763-76be-4fb6-91bb-b4e12f999001"
      },
      "source": [
        "torch.cuda.device_count()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IkKKKPX1wnCr",
        "colab_type": "text"
      },
      "source": [
        "To know the type of GPU allocated "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UO86vjfAwrsT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "91887c2e-307f-4306-c883-4907e0cb8c5f"
      },
      "source": [
        "device = torch.device('cuda')\n",
        "torch.cuda.get_device_properties(device)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "_CudaDeviceProperties(name='Tesla P4', major=6, minor=1, total_memory=7611MB, multi_processor_count=20)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xUu2QZZzxUxS",
        "colab_type": "text"
      },
      "source": [
        "Mounting model to and from GPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PZ-CM8MsxWz1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "80d22ef0-2c1b-4f3e-e22f-6488a5387924"
      },
      "source": [
        "if torch.cuda.is_available():\n",
        "    device = torch.device(\"cuda\")\n",
        "    y = torch.ones_like(x, device = device)\n",
        "    x = x.to(device)\n",
        "    z = x + y\n",
        "    print(z)\n",
        "    print(z.to(\"cpu\", torch.double))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1.0000e+00, 1.0000e+00, 1.0000e+00],\n",
            "        [1.0000e+00,        nan, 1.0000e+00],\n",
            "        [1.1578e+27, 1.1362e+30, 7.1547e+22],\n",
            "        [4.5828e+30, 1.2122e+04, 7.1846e+22],\n",
            "        [1.0000e+00, 7.0374e+22, 1.0000e+00]], device='cuda:0')\n",
            "tensor([[1.0000e+00, 1.0000e+00, 1.0000e+00],\n",
            "        [1.0000e+00,        nan, 1.0000e+00],\n",
            "        [1.1578e+27, 1.1362e+30, 7.1547e+22],\n",
            "        [4.5828e+30, 1.2122e+04, 7.1846e+22],\n",
            "        [1.0000e+00, 7.0374e+22, 1.0000e+00]], dtype=torch.float64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hmQsRcVYzN_T",
        "colab_type": "text"
      },
      "source": [
        "To read more about pytorch , click [here](https://pytorch.org/docs/torch)"
      ]
    }
  ]
}