chapter1/2_autograd_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Autograd: 自动求导机制\n",
    "===================================\n",
    "\n",
    "PyTorch 中所有神经网络的核心是 ``autograd`` 包。\n",
    "我们先简单介绍一下这个包，然后训练第一个简单的神经网络。\n",
    "\n",
    "``autograd``包为张量上的所有操作提供了自动求导。\n",
    "它是一个在运行时定义的框架，这意味着反向传播是根据你的代码来确定如何运行，并且每次迭代可以是不同的。\n",
    "\n",
    "\n",
    "示例\n",
    "\n",
    "张量（Tensor）\n",
    "--------\n",
    "\n",
    "``torch.Tensor``是这个包的核心类。如果设置\n",
    "``.requires_grad`` 为 ``True``，那么将会追踪所有对于该张量的操作。 \n",
    "当完成计算后通过调用 ``.backward()``，自动计算所有的梯度，\n",
    "这个张量的所有梯度将会自动积累到 ``.grad`` 属性。\n",
    "\n",
    "要阻止张量跟踪历史记录，可以调用``.detach()``方法将其与计算历史记录分离，并禁止跟踪它将来的计算记录。\n",
    "\n",
    "为了防止跟踪历史记录（和使用内存），可以将代码块包装在``with torch.no_grad()：``中。\n",
    "在评估模型时特别有用，因为模型可能具有`requires_grad = True`的可训练参数，但是我们不需要梯度计算。\n",
    "\n",
    "在自动梯度计算中还有另外一个重要的类``Function``.\n",
    "\n",
    "``Tensor`` and ``Function`` are interconnected and build up an acyclic\n",
    "graph, that encodes a complete history of computation. Each tensor has\n",
    "a ``.grad_fn`` attribute that references a ``Function`` that has created\n",
    "the ``Tensor`` (except for Tensors created by the user - their\n",
    "``grad_fn is None``).\n",
    "\n",
    "``Tensor`` 和 ``Function``互相连接并生成一个非循环图，它表示和存储了完整的计算历史。\n",
    "每个张量都有一个``.grad_fn``属性，这个属性引用了一个创建了``Tensor``的``Function``（除非这个张量是用户手动创建的，即，这个张量的\n",
    "``grad_fn`` 是 ``None``）。\n",
    "\n",
    "如果需要计算导数，你可以在``Tensor``上调用``.backward()``。 \n",
    "如果``Tensor``是一个标量（即它包含一个元素数据）则不需要为``backward()``指定任何参数，\n",
    "但是如果它有更多的元素，你需要指定一个``gradient`` 参数来匹配张量的形状。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***译者注：在其他的文章中你可能会看到说将Tensor包裹到Variable中提供自动梯度计算，Variable 这个在0.41版中已经被标注为过期了，现在可以直接使用Tensor，官方文档在这里：***\n",
    "(https://pytorch.org/docs/stable/autograd.html#variable-deprecated) \n",
    "\n",
    "具体的后面会有详细说明"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建一个张量并设置 requires_grad=True 用来追踪他的计算历史\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(2, 2, requires_grad=True)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对张量进行操作:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[3., 3.],\n",
      "        [3., 3.]], grad_fn=<AddBackward>)\n"
     ]
    }
   ],
   "source": [
    "y = x + 2\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果``y``已经被计算出来了，所以，``grad_fn``已经被自动生成了。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<AddBackward object at 0x00000232535FD860>\n"
     ]
    }
   ],
   "source": [
    "print(y.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对y进行一个操作\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[27., 27.],\n",
      "        [27., 27.]], grad_fn=<MulBackward>) tensor(27., grad_fn=<MeanBackward1>)\n"
     ]
    }
   ],
   "source": [
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "print(z, out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``.requires_grad_( ... )`` 可以改变现有张量的 ``requires_grad``属性。\n",
    "如果没有指定的话，默认输入的flag是 ``False``。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n",
      "<SumBackward0 object at 0x000002325360B438>\n"
     ]
    }
   ],
   "source": [
    "a = torch.randn(2, 2)\n",
    "a = ((a * 3) / (a - 1))\n",
    "print(a.requires_grad)\n",
    "a.requires_grad_(True)\n",
    "print(a.requires_grad)\n",
    "b = (a * a).sum()\n",
    "print(b.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "梯度\n",
    "---------\n",
    "反向传播\n",
    "因为 ``out``是一个纯量（scalar），``out.backward()`` 等于``out.backward(torch.tensor(1))``。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "out.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print gradients d(out)/dx\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4.5000, 4.5000],\n",
      "        [4.5000, 4.5000]])\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到矩阵 ``4.5``.调用 ``out``\n",
    "*Tensor* “$o$”.\n",
    "\n",
    "得到 $o = \\frac{1}{4}\\sum_i z_i$,\n",
    "$z_i = 3(x_i+2)^2$ and $z_i\\bigr\\rvert_{x_i=1} = 27$.\n",
    "\n",
    "因此,\n",
    "$\\frac{\\partial o}{\\partial x_i} = \\frac{3}{2}(x_i+2)$, hence\n",
    "$\\frac{\\partial o}{\\partial x_i}\\bigr\\rvert_{x_i=1} = \\frac{9}{2} = 4.5$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以使用 autograd 做更多的操作\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-920.6895, -115.7301, -867.6995], grad_fn=<MulBackward>)\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(3, requires_grad=True)\n",
    "\n",
    "y = x * 2\n",
    "while y.data.norm() < 1000:\n",
    "    y = y * 2\n",
    "\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 51.2000, 512.0000,   0.0512])\n"
     ]
    }
   ],
   "source": [
    "gradients = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)\n",
    "y.backward(gradients)\n",
    "\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果``.requires_grad=True``但是你又不希望进行autograd的计算，\n",
    "那么可以将变量包裹在 ``with torch.no_grad()``中:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print(x.requires_grad)\n",
    "print((x ** 2).requires_grad)\n",
    "\n",
    "with torch.no_grad():\n",
    "\tprint((x ** 2).requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**稍后阅读:**\n",
    "\n",
    " ``autograd`` 和 ``Function`` 的官方文档 https://pytorch.org/docs/autograd\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Pytorch for Deeplearning",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}