Build Basic Generative Adversarial Networks (GANs)/Week 1 - Intro to GANs/Intro_to_PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j1k6Y9afpxL6"
   },
   "source": [
    "# Intro\n",
    "\n",
    "[PyTorch](https://pytorch.org/) is a very powerful machine learning framework. Central to PyTorch are [tensors](https://pytorch.org/docs/stable/tensors.html), a generalization of matrices to higher ranks. One intuitive example of a tensor is an image with three color channels: A 3-channel (red, green, blue) image which is 64 pixels wide and 64 pixels tall is a $3\\times64\\times64$ tensor. You can access the PyTorch framework by writing `import torch` near the top of your code, along with all of your other import statements.\n",
    "\n",
    "This guide will help introduce you to the functionality of PyTorch, but don't worry too much about memorizing it: the assignments will link to relevant documentation where necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fwp6T5ZMteDC"
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IvXp0rlPBqdQ"
   },
   "source": [
    "# Why PyTorch?\n",
    "\n",
    "One important question worth asking is, why is PyTorch being used for this course? There is a great breakdown by [the Gradient](https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry/) looking at the state of machine learning frameworks today. In part, as highlighted by the article, PyTorch is generally more pythonic than alternative frameworks, easier to debug, and is the most-used language in machine learning research by a large and growing margin. While PyTorch's primary alternative, Tensorflow, has attempted to integrate many of PyTorch's features, Tensorflow's implementations come with some inherent limitations highlighted in the article.\n",
    "\n",
    "Notably, while PyTorch's industry usage has grown, Tensorflow is still (for now) a slight favorite in industry. In practice, the features that make PyTorch attractive for research also make it attractive for education, and the general trend of machine learning research and practice to PyTorch makes it the more proactive choice. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MCgwdP20r1yX"
   },
   "source": [
    "# Tensor Properties\n",
    "One way to create tensors from a list or an array is to use `torch.Tensor`. It'll be used to set up examples in this notebook, but you'll never need to use it in the course - in fact, if you find yourself needing it, that's probably not the correct answer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B0hgYekGsxlB"
   },
   "outputs": [],
   "source": [
    "example_tensor = torch.Tensor(\n",
    "    [\n",
    "     [[1, 2], [3, 4]], \n",
    "     [[5, 6], [7, 8]], \n",
    "     [[9, 0], [1, 2]]\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9dO4C2oft7zq"
   },
   "source": [
    "You can view the tensor in the notebook by simple printing it out (though some larger tensors will be cut off)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "U2FKEzeYuEOX",
    "outputId": "dfa12ff7-afd1-4737-a669-54f36b4209dd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1., 2.],\n",
       "         [3., 4.]],\n",
       "\n",
       "        [[5., 6.],\n",
       "         [7., 8.]],\n",
       "\n",
       "        [[9., 0.],\n",
       "         [1., 2.]]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VUwlmUngw-VR"
   },
   "source": [
    "## Tensor Properties: Device\n",
    "\n",
    "One important property is the device of the tensor - throughout this notebook you'll be sticking to tensors which are on the CPU. However, throughout the course you'll also be using tensors on GPU (that is, a graphics card which will be provided for you to use for the course). To view the device of the tensor, all you need to write is `example_tensor.device`. To move a tensor to a new device, you can write `new_tensor = example_tensor.to(device)` where device will be either `cpu` or `cuda`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "R7SF44_Vw9h0",
    "outputId": "57f90e38-f9e1-4115-8f27-ebe651d5b2fa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_tensor.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FkfySyFduHQi"
   },
   "source": [
    "## Tensor Properties: Shape\n",
    "\n",
    "And you can get the number of elements in each dimension by printing out the tensor's shape, using `example_tensor.shape`, something you're likely familiar with if you've used numpy. For example, this tensor is a $3\\times2\\times2$ tensor, since it has 3 elements, each of which are $2\\times2$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "DKmfzpOBun0t",
    "outputId": "883009b6-7300-4329-f9ec-df99cc36d846"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 2, 2])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_tensor.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aL954xmAuq4b"
   },
   "source": [
    "You can also get the size of a particular dimension $n$ using `example_tensor.shape[n]` or equivalently `example_tensor.size(n)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "7IKy3BB8uqBo",
    "outputId": "7fac1275-132f-4d2b-bf63-73065a2aea6a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape[0] = 3\n",
      "size(1) = 2\n"
     ]
    }
   ],
   "source": [
    "print(\"shape[0] =\", example_tensor.shape[0])\n",
    "print(\"size(1) =\", example_tensor.size(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3pzzG8bav5rl"
   },
   "source": [
    "Finally, it is sometimes useful to get the number of dimensions (rank) or the number of elements, which you can do as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "l_j9qTwyv41-",
    "outputId": "5921cbd1-19a2-4543-9488-3f72c0cb4970"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rank = 3\n",
      "Number of elements = 12\n"
     ]
    }
   ],
   "source": [
    "print(\"Rank =\", len(example_tensor.shape))\n",
    "print(\"Number of elements =\", example_tensor.numel())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gibyKQJQzLkm"
   },
   "source": [
    "# Indexing Tensors\n",
    "\n",
    "As with numpy, you can access specific elements or subsets of elements of a tensor. To access the $n$-th element, you can simply write `example_tensor[n]` - as with Python in general, these dimensions are 0-indexed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "F87bFA5SzNz7",
    "outputId": "1b0a8381-6fd8-40b4-a5c8-88cc80029f8e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[5., 6.],\n",
       "        [7., 8.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_tensor[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1CegGw5wzpGa"
   },
   "source": [
    "In addition, if you want to access the $j$-th dimension of the $i$-th example, you can write `example_tensor[i, j]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "bl1JSZcRz0xn",
    "outputId": "7f98e47b-66cb-4927-b784-7e4bcb9eb687"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(7.)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_tensor[1, 1, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dyQRCRIa4NaY"
   },
   "source": [
    "Note that if you'd like to get a Python scalar value from a tensor, you can use `example_scalar.item()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "e56KSJOq4YOE",
    "outputId": "29e1fd13-32df-40c5-e558-3193fa5da629"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_scalar = example_tensor[1, 1, 0]\n",
    "example_scalar.item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wZdMEQfu0A7h"
   },
   "source": [
    "In addition, you can index into the ith element of a column by using `x[:, i]`. For example, if you want the top-left element of each element in `example_tensor`, which is the `0, 0` element of each matrix, you can write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "x2cFxJx50eGH",
    "outputId": "e66eade9-4b4b-4c7a-ea99-a83195d10541"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 5., 9.])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_tensor[:, 0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "w-rTBP-1whd2"
   },
   "source": [
    "# Initializing Tensors\n",
    "\n",
    "There are many ways to create new tensors in PyTorch, but in this course, the most important ones are: \n",
    "\n",
    "[`torch.ones_like`](https://pytorch.org/docs/master/generated/torch.ones_like.html): creates a tensor of all ones with the same shape and device as `example_tensor`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "g7gbs4AnwlIo",
    "outputId": "b0c67ed9-e33f-47d6-d95c-e53bc4f90dec"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1., 1.],\n",
       "         [1., 1.]],\n",
       "\n",
       "        [[1., 1.],\n",
       "         [1., 1.]],\n",
       "\n",
       "        [[1., 1.],\n",
       "         [1., 1.]]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.ones_like(example_tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_aIbSlaJy9Z0"
   },
   "source": [
    "[`torch.zeros_like`](https://pytorch.org/docs/master/generated/torch.zeros_like.html): creates a tensor of all zeros with the same shape and device as `example_tensor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "X4cWQduzzCd8",
    "outputId": "dbc8a5fa-8db1-4f6d-e38e-d1deb982ff36"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[0., 0.],\n",
       "         [0., 0.]],\n",
       "\n",
       "        [[0., 0.],\n",
       "         [0., 0.]],\n",
       "\n",
       "        [[0., 0.],\n",
       "         [0., 0.]]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.zeros_like(example_tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wsOmgS1izDS_"
   },
   "source": [
    "[`torch.randn_like`](https://pytorch.org/docs/stable/generated/torch.randn_like.html): creates a tensor with every element sampled from a [Normal (or Gaussian) distribution](https://en.wikipedia.org/wiki/Normal_distribution) with the same shape and device as `example_tensor`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "2hto51IazDow",
    "outputId": "cb62a68a-6171-4d1e-eb9b-f31784464aac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.3675,  0.2242],\n",
       "         [-0.3378, -1.0944]],\n",
       "\n",
       "        [[ 1.5371,  0.7701],\n",
       "         [-0.1490, -0.0928]],\n",
       "\n",
       "        [[ 0.3270,  0.4642],\n",
       "         [ 0.1494,  0.1283]]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.randn_like(example_tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HXp0i5Cf6AGj"
   },
   "source": [
    "Sometimes (though less often than you'd expect), you might need to initialize a tensor knowing only the shape and device, without a tensor for reference for `ones_like` or `randn_like`. In this case, you can create a $2x2$ tensor as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "RZRqt3-S6cUZ",
    "outputId": "7bef97cc-a303-4200-c0f8-ef9bf3cb4996"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.2235, -1.8912],\n",
       "        [-1.2873,  0.7405]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.randn(2, 2, device='cpu') # Alternatively, for a GPU tensor, you'd use device='cuda'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JTkmDwVsrM6R"
   },
   "source": [
    "# Basic Functions\n",
    "\n",
    "There are a number of basic functions that you should know to use PyTorch - if you're familiar with numpy, all commonly-used functions exist in PyTorch, usually with the same name. You can perform element-wise multiplication / division by a scalar $c$ by simply writing `c * example_tensor`, and element-wise addition / subtraction by a scalar by writing `example_tensor + c`\n",
    "\n",
    "Note that most operations are not in-place in PyTorch, which means that they don't change the original variable's data (However, you can reassign the same variable name to the changed data if you'd like, such as `example_tensor = example_tensor + 1`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "FpfwOUdopsF_",
    "outputId": "32347400-2e6a-40c6-e6f1-21e6aacde795"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ -8.,  -6.],\n",
       "         [ -4.,  -2.]],\n",
       "\n",
       "        [[  0.,   2.],\n",
       "         [  4.,   6.]],\n",
       "\n",
       "        [[  8., -10.],\n",
       "         [ -8.,  -6.]]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(example_tensor - 5) * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uciZnx4b3UjX"
   },
   "source": [
    "You can calculate the mean or standard deviation of a tensor using [`example_tensor.mean()`](https://pytorch.org/docs/stable/generated/torch.mean.html) or [`example_tensor.std()`](https://pytorch.org/docs/stable/generated/torch.std.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "0ELXUKG7329z",
    "outputId": "720dd190-7dd4-43f1-e53c-cba4263eb2be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean: tensor(4.)\n",
      "Stdev: tensor(2.9848)\n"
     ]
    }
   ],
   "source": [
    "print(\"Mean:\", example_tensor.mean())\n",
    "print(\"Stdev:\", example_tensor.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_QsyTRym32SX"
   },
   "source": [
    "You might also want to find the mean or standard deviation along a particular dimension. To do this you can simple pass the number corresponding to that dimension to the function. For example, if you want to get the average $2\\times2$ matrix of the $3\\times2\\times2$ `example_tensor` you can write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "eCJl3Im25B9k",
    "outputId": "4bd9decd-579e-462c-bde1-ee8d9d1b2061"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[5.0000, 2.6667],\n",
       "        [3.6667, 4.6667]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_tensor.mean(0)\n",
    "\n",
    "# Equivalently, you could also write:\n",
    "# example_tensor.mean(dim=0)\n",
    "# example_tensor.mean(axis=0)\n",
    "# torch.mean(example_tensor, 0)\n",
    "# torch.mean(example_tensor, dim=0)\n",
    "# torch.mean(example_tensor, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Vb-_5ubc8t97"
   },
   "source": [
    "PyTorch has many other powerful functions but these should be all of PyTorch functions you need for this course outside of its neural network module (`torch.nn`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RtWjExD69JEs"
   },
   "source": [
    "# PyTorch Neural Network Module (`torch.nn`)\n",
    "\n",
    "PyTorch has a lot of powerful classes in its `torch.nn` module (Usually, imported as simply `nn`). These classes allow you to create a new function which transforms a tensor in specific way, often retaining information when called multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UYrgloYo_slC"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uyCPVmTD_kkl"
   },
   "source": [
    "## `nn.Linear`\n",
    "\n",
    "To create a linear layer, you need to pass it the number of input dimensions and the number of output dimensions. The linear object initialized as `nn.Linear(10, 2)` will take in a $n\\times10$ matrix and return an $n\\times2$ matrix, where all $n$ elements have had the same linear transformation performed. For example, you can initialize a linear layer which performs the operation $Ax + b$, where $A$ and $b$ are initialized randomly when you generate the [`nn.Linear()`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "pNPaHPo89VrN",
    "outputId": "c14dc316-ae68-49d3-a8eb-8ad6e1464f01"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.2900, -0.5352],\n",
       "        [ 0.4298,  0.4173],\n",
       "        [ 0.4861, -0.3332]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linear = nn.Linear(10, 2)\n",
    "example_input = torch.randn(3, 10)\n",
    "example_output = linear(example_input)\n",
    "example_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YGNULkJR_mzn"
   },
   "source": [
    "## `nn.ReLU`\n",
    "\n",
    "[`nn.ReLU()`](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) will create an object that, when receiving a tensor, will perform a ReLU activation function. This will be reviewed further in lecture, but in essence, a ReLU non-linearity sets all negative numbers in a tensor to zero. In general, the simplest neural networks are composed of series of linear transformations, each followed by activation functions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "nGxVFS3nBASc",
    "outputId": "d5f57584-1bad-4803-ba8c-b69881db4a1f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.2900, 0.0000],\n",
       "        [0.4298, 0.4173],\n",
       "        [0.4861, 0.0000]], grad_fn=<ReluBackward0>)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relu = nn.ReLU()\n",
    "relu_output = relu(example_output)\n",
    "relu_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KzfOEZ03AJzA"
   },
   "source": [
    "## `nn.BatchNorm1d`\n",
    "\n",
    "[`nn.BatchNorm1d`](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html) is a normalization technique that will rescale a batch of $n$ inputs to have a consistent mean and standard deviation between batches.  \n",
    "\n",
    "As indicated by the `1d` in its name, this is for situations where you expects a set of inputs, where each of them is a flat list of numbers. In other words, each input is a vector, not a matrix or higher-dimensional tensor. For a set of images, each of which is a higher-dimensional tensor, you'd use [`nn.BatchNorm2d`](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html), discussed later on this page.\n",
    "\n",
    "`nn.BatchNorm1d` takes an argument of the number of input dimensions of each object in the batch (the size of each example vector)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "O4tYsi9-G9vM",
    "outputId": "ba61d37c-a8af-4663-fcc2-1691c6d241de"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.3570, -0.7070],\n",
       "        [ 0.3368,  1.4140],\n",
       "        [ 1.0202, -0.7070]], grad_fn=<NativeBatchNormBackward>)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batchnorm = nn.BatchNorm1d(2)\n",
    "batchnorm_output = batchnorm(relu_output)\n",
    "batchnorm_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EMZewDz9Idr1"
   },
   "source": [
    "## `nn.Sequential`\n",
    "\n",
    "[`nn.Sequential`](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) creates a single operation that performs a sequence of operations. For example, you can write a neural network layer with a batch normalization as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "R3GhASjyJt3N",
    "outputId": "3ef779ca-a17b-42fd-f2e5-fbb5fdc60b13"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: \n",
      "tensor([[ 1.7690,  0.2864,  0.7925,  2.2849,  1.5226],\n",
      "        [ 0.1877,  0.1367, -0.2833,  2.0905,  0.0454],\n",
      "        [ 0.7825,  2.2969,  1.2144,  0.2526,  2.5709],\n",
      "        [-0.4878,  1.9587,  1.6849,  0.5284,  1.9027],\n",
      "        [ 0.5384,  1.1787,  0.4961, -1.6326,  1.4192]])\n",
      "output: \n",
      "tensor([[0.0000, 1.1865],\n",
      "        [1.5208, 0.0000],\n",
      "        [0.0000, 1.1601],\n",
      "        [0.0000, 0.0000],\n",
      "        [0.7246, 0.0000]], grad_fn=<ReluBackward0>)\n"
     ]
    }
   ],
   "source": [
    "mlp_layer = nn.Sequential(\n",
    "    nn.Linear(5, 2),\n",
    "    nn.BatchNorm1d(2),\n",
    "    nn.ReLU()\n",
    ")\n",
    "\n",
    "test_example = torch.randn(5,5) + 1\n",
    "print(\"input: \")\n",
    "print(test_example)\n",
    "print(\"output: \")\n",
    "print(mlp_layer(test_example))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SToQiSv5K5Yb"
   },
   "source": [
    "# Optimization\n",
    "\n",
    "One of the most important aspects of essentially any machine learning framework is its automatic differentiation library. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r4GZFCZ0QqI1"
   },
   "source": [
    "## Optimizers\n",
    "\n",
    "To create an optimizer in PyTorch, you'll need to use the `torch.optim` module, often imported as `optim`. [`optim.Adam`](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam) corresponds to the Adam optimizer. To create an optimizer object, you'll need to pass it the parameters to be optimized and the learning rate, `lr`, as well as any other parameters specific to the optimizer.\n",
    "\n",
    "For all `nn` objects, you can access their parameters as a list using their `parameters()` method, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AIcCbs35K4wY"
   },
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "adam_opt = optim.Adam(mlp_layer.parameters(), lr=1e-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-BsPFZu2M0Xx"
   },
   "source": [
    "## Training Loop\n",
    "\n",
    "A (basic) training step in PyTorch consists of four basic parts:\n",
    "\n",
    "\n",
    "1.   Set all of the gradients to zero using `opt.zero_grad()`\n",
    "2.   Calculate the loss, `loss`\n",
    "3.   Calculate the gradients with respect to the loss using `loss.backward()`\n",
    "4.   Update the parameters being optimized using `opt.step()`\n",
    "\n",
    "That might look like the following code (and you'll notice that if you run it several times, the loss goes down):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "zm6lPx4sOJht",
    "outputId": "c21672bd-a306-42ab-face-9a299511a059"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.7719, grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "source": [
    "train_example = torch.randn(100,5) + 1\n",
    "adam_opt.zero_grad()\n",
    "\n",
    "# We'll use a simple loss function of mean distance from 1\n",
    "# torch.abs takes the absolute value of a tensor\n",
    "cur_loss = torch.abs(1 - mlp_layer(train_example)).mean()\n",
    "\n",
    "cur_loss.backward()\n",
    "adam_opt.step()\n",
    "print(cur_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wDjhZBCeTc6o"
   },
   "source": [
    "## `requires_grad_()`\n",
    "\n",
    "You can also tell PyTorch that it needs to calculate the gradient with respect to a tensor that you created by saying `example_tensor.requires_grad_()`, which will change it in-place. This means that even if PyTorch wouldn't normally store a grad for that particular tensor, it will for that specified tensor. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mB22ovHyUEvH"
   },
   "source": [
    "## `with torch.no_grad():`\n",
    "\n",
    "PyTorch will usually calculate the gradients as it proceeds through a set of operations on tensors. This can often take up unnecessary computations and memory, especially if you're performing an evaluation. However, you can wrap a piece of code with `with torch.no_grad()` to prevent the gradients from being calculated in a piece of code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kowb1M425CE_"
   },
   "source": [
    "\n",
    "## `detach():`\n",
    "\n",
    "Sometimes, you want to calculate and use a tensor's value without calculating its gradients. For example, if you have two models, A and B, and you want to directly optimize the parameters of A with respect to the output of B, without calculating the gradients through B, then you could feed the detached output of B to A. There are many reasons you might want to do this, including efficiency or cyclical dependencies (i.e. A depends on B depends on A)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-9HY2wgKLOr-"
   },
   "source": [
    "# New `nn` Classes\n",
    "\n",
    "You can also create new classes which extend the `nn` module. For these classes, all class attributes, as in `self.layer` or `self.param` will automatically treated as parameters if they are themselves `nn` objects or if they are tensors wrapped in `nn.Parameter` which are initialized with the class. \n",
    "\n",
    "The `__init__` function defines what will happen when the object is created. The first line of the init function of a class, for example, `WellNamedClass`, needs to be `super(WellNamedClass, self).__init__()`. \n",
    "\n",
    "The `forward` function defines what runs if you create that object `model` and pass it a tensor `x`, as in `model(x)`. If you choose the function signature, `(self, x)`, then each call of the forward function, gets two pieces of information: `self`, which is a reference to the object with which you can access all of its parameters, and `x`, which is the current tensor for which you'd like to return `y`.\n",
    "\n",
    "One class might look like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WOip473tQs-d"
   },
   "outputs": [],
   "source": [
    "class ExampleModule(nn.Module):\n",
    "    def __init__(self, input_dims, output_dims):\n",
    "        super(ExampleModule, self).__init__()\n",
    "        self.linear = nn.Linear(input_dims, output_dims)\n",
    "        self.exponent = nn.Parameter(torch.tensor(1.))\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.linear(x)\n",
    "\n",
    "        # This is the notation for element-wise exponentiation, \n",
    "        # which matches python in general\n",
    "        x = x ** self.exponent \n",
    "        \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x4CUFH_GS5UY"
   },
   "source": [
    "And you can view its parameters as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "id": "YuelIiE4S3KR",
    "outputId": "27a52620-ca40-4dc8-dff5-4f3a56ba0e5b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parameter containing:\n",
       " tensor(1., requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([[ 0.2789,  0.2618, -0.0678,  0.2766,  0.1436,  0.0917, -0.1669, -0.1887,\n",
       "           0.0913, -0.1998],\n",
       "         [-0.1757,  0.0361,  0.1140,  0.2152, -0.1200,  0.1712,  0.0944, -0.0447,\n",
       "           0.1548,  0.2383]], requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([ 0.1881, -0.0834], requires_grad=True)]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_model = ExampleModule(10, 2)\n",
    "list(example_model.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1F7E1wKN5tez"
   },
   "source": [
    "And you can print out their names too, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "dYTuTDsQ5pnY",
    "outputId": "6635a493-7318-4688-bd18-bfba41d43e9d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('exponent',\n",
       "  Parameter containing:\n",
       "  tensor(1., requires_grad=True)),\n",
       " ('linear.weight',\n",
       "  Parameter containing:\n",
       "  tensor([[ 0.2789,  0.2618, -0.0678,  0.2766,  0.1436,  0.0917, -0.1669, -0.1887,\n",
       "            0.0913, -0.1998],\n",
       "          [-0.1757,  0.0361,  0.1140,  0.2152, -0.1200,  0.1712,  0.0944, -0.0447,\n",
       "            0.1548,  0.2383]], requires_grad=True)),\n",
       " ('linear.bias',\n",
       "  Parameter containing:\n",
       "  tensor([ 0.1881, -0.0834], requires_grad=True))]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(example_model.named_parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iWPoIqX2UsaH"
   },
   "source": [
    "And here's an example of the class in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "7NXwbg5tUroC",
    "outputId": "0836e447-7c37-464e-b196-048ae0a0cc73"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.0567,  0.4562],\n",
       "        [ 0.3780,  0.3452]], grad_fn=<PowBackward1>)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = torch.randn(2, 10)\n",
    "example_model(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6Ocol8DABScy"
   },
   "source": [
    "# 2D Operations\n",
    "\n",
    "You won't need these for the first lesson, and the theory behind each of these will be reviewed more in later lectures, but here is a quick reference: \n",
    "\n",
    "\n",
    "*   2D convolutions: [`nn.Conv2d`](https://pytorch.org/docs/master/generated/torch.nn.Conv2d.html) requires the number of input and output channels, as well as the kernel size.\n",
    "*   2D transposed convolutions (aka deconvolutions): [`nn.ConvTranspose2d`](https://pytorch.org/docs/master/generated/torch.nn.ConvTranspose2d.html) also requires the number of input and output channels, as well as the kernel size\n",
    "*   2D batch normalization: [`nn.BatchNorm2d`](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html) requires the number of input dimensions\n",
    "*   Resizing images: [`nn.Upsample`](https://pytorch.org/docs/master/generated/torch.nn.Upsample.html) requires the final size or a scale factor. Alternatively, [`nn.functional.interpolate`](https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.interpolate) takes the same arguments. \n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Intro to PyTorch.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}