Chapter3/chapter3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# 第三章 Tensor\n",
    "\n",
    "几乎所有的深度学习框架背后的设计核心都是张量和计算图，PyTorch也不例外，本章我们将学习PyTorch中的张量系统（Tensor）和自动微分系统（autograd）。\n",
    "\n",
    "## 3.1 Tensor基础\n",
    "\n",
    "Tensor，又名张量，读者可能对这个名词似曾相识，因为它不仅在PyTorch中出现过，它也是Theano、TensorFlow、Torch和MXNet中重要的数据结构。关于张量的本质不乏深度剖析的文章，但从工程角度讲，可简单地认为它就是一个数组，且支持高效的科学计算。它可以是一个数（标量）、一维数组（向量）、二维数组（矩阵）或更高维的数组（高阶数据）。Tensor和Numpy的ndarrays类似，但前者支持GPU加速。\n",
    "\n",
    "本节将系统讲解Tensor的使用，力求面面俱到，但不会涉及每个函数。对于更多函数及其用法，读者可通过在IPython/Notebook中使用函数名加`?`查看帮助文档，或查阅PyTorch官方文档。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "# 查看PyTorch的版本号\r\n",
    "import torch as t\r\n",
    "t.__version__"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'1.6.0'"
      ]
     },
     "metadata": {},
     "execution_count": 1
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.1.1 Tensor 的基本操作\n",
    "\n",
    "学习过NumPy的读者会对本节内容感到非常熟悉，因为Tensor的接口设计得与Numpy类似，以方便用户使用。当然，若读者不熟悉Numpy也没关系，本节内容并不要求读者事先掌握Numpy。\n",
    "\n",
    "从接口的角度来讲，对Tensor的操作可分为两类：\n",
    "\n",
    "1. `torch.function`，如`torch.save`等。\n",
    "2. `tensor.function`，如`tensor.view`等。\n",
    "\n",
    "为方便使用，对Tensor的大部分操作同时支持这两类接口，在本书中不做具体区分，如`torch.sum(a, b)`与`a.sum(b)`功能等价。\n",
    "\n",
    "而从存储的角度来讲，对Tensor的操作又可分为两类：\n",
    "\n",
    "1. 不会修改自身的数据，如 `a.add(b)`， 加法的结果会返回一个新的Tensor。\n",
    "2. 会修改自身的数据，如 `a.add_(b)`， 加法的结果仍存储在a中，a被修改了。\n",
    "\n",
    "函数名以`_`结尾的都是inplace方式，即会修改调用者自己的数据，在实际应用中需加以区分。\n",
    "\n",
    "#### 创建Tensor\n",
    "\n",
    "在PyTorch中创建Tensor的方法有很多，具体如表3-1所示：\n",
    "\n",
    "表3-1: 常见创建Tensor的方法\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|Tensor(\\*sizes)|基础构造函数|\n",
    "|tensor(data,)|类似np.array的构造函数|\n",
    "|ones(\\*sizes)|全1的Tensor|\n",
    "|zeros(\\*sizes)|全0的Tensor|\n",
    "|eye(\\*sizes)|对角线为1，其他为0|\n",
    "|arange(s,e,step)|从s到e，步长为step|\n",
    "|linspace(s,e,steps)|从s到e，均匀切分成steps份|\n",
    "|rand / randn(\\*sizes)|均匀/标准分布|\n",
    "|normal(mean,std) / uniform(from,to)|正态分布/均匀分布|\n",
    "|randperm(m)|随机排列|\n",
    "|tensor.new_\\* / torch.\\*_like | 创建一个相同size大小，用*类型去填充（如tensor.new_ones就是使用全一的数据去填充）的张量，具有相同的torch.dtype和torch.device|\n",
    "\n",
    "这些创建方法都可以在创建的时候指定数据类型dtype和存放device(CPU/GPU)。\n",
    "\n",
    "使用`Tensor`函数新建Tensor可以接收一个list，并根据list的数据新建Tensor，同时也能根据指定的形状新建Tensor，还能传入其他的Tensor，下面将举几个例子说明。\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "# 指定Tensor的形状\r\n",
    "a = t.Tensor(2, 3)\r\n",
    "a # 数值取决于内存空间的状态，print时候可能overflow"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[-8.9209e-11,  4.5846e-41, -8.9209e-11],\n",
       "        [ 4.5846e-41,  4.4400e-29,  3.0956e-41]])"
      ]
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "# 用list的数据创建Tensor\r\n",
    "b = t.Tensor([[1,2,3],[4,5,6]])\r\n",
    "b"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 2., 3.],\n",
       "        [4., 5., 6.]])"
      ]
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "b.tolist() # 把Tensor转为list"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "`tensor.size()`返回`torch.Size`对象，它是tuple的子类，但其使用方式与tuple略有区别。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "b_size = b.size()\r\n",
    "b_size"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "# 统计b中元素总个数，2*3，两种方式等价\r\n",
    "b.numel(), b.nelement()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(6, 6)"
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "# 创建一个和b形状一样的Tensor c\r\n",
    "c = t.Tensor(b_size)\r\n",
    "# 创建一个元素为2和3的Tensor d\r\n",
    "d = t.Tensor((2, 3))\r\n",
    "c, d"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(tensor([[0., 0., 0.],\n",
       "         [0., 0., 0.]]), tensor([2., 3.]))"
      ]
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "需要注意的是，`t.Tensor(*sizes)`创建Tensor时，系统不会马上分配空间，只是会计算剩余的内存是否足够使用，在真正使用到Tensor时才会分配，而其它操作都是在创建完Tensor之后马上进行空间分配。在使用中，我们更推荐使用`torch.tensor`而非`torch.Tensor`。接下来比较一下两者的区别。\n",
    "\n",
    "`torch.Tensor`是Python类，并且默认是torch.FloatTensor()，运行`torch.Tensor([2,3])`会直接调用Tensor类的构造函数`__init__()`，生成结果是单精度浮点类型的Tensor（FloatTensor）。\n",
    "而`torch.tensor()`是一个Python函数，函数的原型为：`torch.tensor(data, dtype=None, device=None, requires_grad=False)`。其中`data`支持list、tuple、array、scalar等类型，`torch.tensor()`直接从`data`中做数据拷贝，并根据原数据类型生成相应的Tensor。\n",
    "\n",
    "由于`torch.tensor()`能够根据数据类型生成对应的Tensor，因此在实际应用中我们更加推荐使用`torch.tensor()`方法来创建一个新的Tensor。下面将举例说明二者的不同之处。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "# torch.Tensor能直接创建空的张量\r\n",
    "t.Tensor()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([])"
      ]
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "# torch.tensor不能直接创建空的张量，需要传入一个空的data\r\n",
    "# t.tensor() # TypeError: tensor() missing 1 required positional arguments: \"data\"\r\n",
    "t.tensor(())"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([])"
      ]
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "a = t.tensor([2,3])\r\n",
    "print(a.type())\r\n",
    "b = t.Tensor([2,3])\r\n",
    "print(b.type())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "torch.LongTensor\n",
      "torch.FloatTensor\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "import numpy as np\r\n",
    "arr = np.ones((2, 3), dtype=np.float64)\r\n",
    "a = t.tensor(arr)\r\n",
    "a"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], dtype=torch.float64)"
      ]
     },
     "metadata": {},
     "execution_count": 11
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "其它常用的创建Tensor方法举例如下。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "# 创建一个形状是(2,3)值全为1的Tensor\r\n",
    "t.ones(2, 3)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "metadata": {},
     "execution_count": 12
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "与`torch.ones()`类似的函数还有`torch.ones_like(input)`，输入`input`是一个Tensor，返回一个与之大小相同的全部填充为1的新Tensor。这意味着，`torch.ones_like(input)`等价于`torch.ones(input.size(), dtype=input.dtype, layout=input.layout, device=input.device)`。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "input_tensor = t.tensor([[1, 2, 3], [4, 5, 6]])\r\n",
    "t.ones_like(input_tensor)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1, 1, 1],\n",
       "        [1, 1, 1]])"
      ]
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "# 创建一个形状是(2,3)的全0的Tensor\r\n",
    "t.zeros(2, 3)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]])"
      ]
     },
     "metadata": {},
     "execution_count": 14
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "# 创建一个对角线为1，其余为0的Tensor,不要求行列数一致\r\n",
    "t.eye(2, 3, dtype=t.int)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1, 0, 0],\n",
       "        [0, 1, 0]], dtype=torch.int32)"
      ]
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "# 创建一个起始值为1，上限为6，步长为2的Tensor\r\n",
    "t.arange(1, 6, 2)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([1, 3, 5])"
      ]
     },
     "metadata": {},
     "execution_count": 16
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "# 创建一个均匀间距的Tensor，将1到10的数分为3份\r\n",
    "t.linspace(1, 10, 3)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([ 1.0000,  5.5000, 10.0000])"
      ]
     },
     "metadata": {},
     "execution_count": 17
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "# 创建一个形状是(2,3)的Tensor，其取值为从标准正态分布中抽取的随机数\r\n",
    "t.randn(2, 3)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 1.3969, -1.5042, -0.8430],\n",
       "        [-0.8707, -1.0794, -1.3357]])"
      ]
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "# 创建一个长度为5的随机排列的Tensor\r\n",
    "t.randperm(5) "
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([2, 4, 0, 3, 1])"
      ]
     },
     "metadata": {},
     "execution_count": 19
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "source": [
    "# 创建一个大小为(2，3)值全1的Tensor，保留原始的torch.dtype和torch.device\r\n",
    "tensor3 = t.tensor((), dtype=t.int32)\r\n",
    "tensor3.new_ones((2, 3))"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1, 1, 1],\n",
       "        [1, 1, 1]], dtype=torch.int32)"
      ]
     },
     "metadata": {},
     "execution_count": 20
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 命名张量\n",
    "\n",
    "命名张量（Named Tensors）旨在允许用户将显式名称与张量维度关联起来，使张量更易于使用。在大多数情况下，使用维度名称来进行维度操作，从而避免了按位置确定维度的需要。此外，命名张量使用名称来自动检查API是否在运行时正确使用，从而提供额外的安全性。名称还可用于重新排列维度，例如，支持“按名称广播”而不是“按位置广播”等等。现在允许张量拥有命名维度的工厂函数（factory functions）有：tensor，empty，ones，zeros，randn等等。\n",
    "\n",
    "下面将以深度学习中常用的维度来举例说明命名张量的使用，其中`N`代表batch_size，`C`代表通道数，`H`代表高度，`W`代表宽度。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "# 命名张量API在后续还有可能还有变化，所以会系统会提示warning，在此忽略\r\n",
    "import warnings\r\n",
    "warnings.filterwarnings(\"ignore\")\r\n",
    "# 直接使用names参数创建命名张量\r\n",
    "imgs = t.randn(1, 2, 2, 3, names=('N', 'C', 'H', 'W'))\r\n",
    "imgs.names"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "('N', 'C', 'H', 'W')"
      ]
     },
     "metadata": {},
     "execution_count": 21
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "# 查看旋转操作造成的维度变换\r\n",
    "imgs_rotate = imgs.transpose(2, 3)\r\n",
    "imgs_rotate.names"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "('N', 'C', 'W', 'H')"
      ]
     },
     "metadata": {},
     "execution_count": 22
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "source": [
    "# 通过refine_names()函数对未命名的张量命名，其中不需要名字的可以用None表示\r\n",
    "another_imgs = t.rand(1, 3, 2, 2)\r\n",
    "another_imgs = another_imgs.refine_names('N', None, 'H', 'W')\r\n",
    "another_imgs.names"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "('N', None, 'H', 'W')"
      ]
     },
     "metadata": {},
     "execution_count": 23
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "source": [
    "# 修改部分维度的名称\r\n",
    "renamed_imgs = imgs.rename(H='height', W='width')\r\n",
    "renamed_imgs.names"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "('N', 'C', 'height', 'width')"
      ]
     },
     "metadata": {},
     "execution_count": 24
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "source": [
    "# 通过维度的名称做维度转换\r\n",
    "convert_imgs = renamed_imgs.align_to('N', 'height', 'width', 'C')\r\n",
    "convert_imgs.names"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "('N', 'height', 'width', 'C')"
      ]
     },
     "metadata": {},
     "execution_count": 25
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "命名张量还可以直观的用于说明Tensor的形状改变和维度增减，这一部分将在3.2节进行详细介绍。\n",
    "\n",
    "#### Tensor的类型\n",
    "\n",
    " Tensor有不同的数据类型，如表3-2所示，每种类型分别对应有CPU和GPU版本(HalfTensor除外)。默认的Tensor是FloatTensor，可通过`torch.set_default_tensor_type` 来修改默认Tensor类型(如果默认类型为GPU tensor，则所有操作都将在GPU上进行)。Tensor的类型对分析内存占用很有帮助。例如，一个size为(1000, 1000, 1000)的FloatTensor，它有`1000*1000*1000=10^9`个元素，每个元素占32bit/8 = 4Byte内存，所以共占大约4GB内存/显存。HalfTensor是专门为GPU版本设计的，同样的元素个数，显存占用只有FloatTensor的一半，所以可以极大缓解GPU显存不足的问题，但由于HalfTensor所能表示的数值大小和精度有限，所以可能出现溢出等问题。\n",
    "\n",
    "\n",
    "表3-2: Tensor数据类型\n",
    "\n",
    "| Data type | dtype | CPU tensor   | GPU tensor   |\n",
    "| ------------------------ | --------------------------------- | ----------------------------------------- | ------------------------- |\n",
    "| 32-bit 浮点型   | `torch.float32` or `torch.float`  | `torch.FloatTensor`  | `torch.cuda.FloatTensor`  |\n",
    "| 64-bit 浮点型   | `torch.float64` or `torch.double` | `torch.DoubleTensor` | `torch.cuda.DoubleTensor` |\n",
    "| 16-bit 半精度浮点型    | `torch.float16` or `torch.half`  | `torch.HalfTensor`| `torch.cuda.HalfTensor`|\n",
    "| 8-bit无符号整型 | `torch.uint8`  | `torch.ByteTensor`| `torch.cuda.ByteTensor`|\n",
    "| 8-bit有符号整型  | `torch.int8`  | `torch.CharTensor`| `torch.cuda.CharTensor`|\n",
    "| 16-bit有符号整型  | `torch.int16` or `torch.short`   | `torch.ShortTensor` | `torch.cuda.ShortTensor`  |\n",
    "| 32-bit有符号整型 | `torch.int32` or `torch.int`  | `torch.IntTensor` | `torch.cuda.IntTensor` |\n",
    "| 64-bit有符号整型  | `torch.int64` or `torch.long`  | `torch.LongTensor` | `torch.cuda.LongTensor`|\n",
    "\n",
    "\n",
    "不同Tensor类型之间是可以互相转换的，下面是常用的三种转换方法：\n",
    "\n",
    "1. 各种数据类型之间可以互相转换，`type(new_type)`是通用的做法，同时还有`float`、`long`、`half`等快捷方法；\n",
    "2. CPU Tensor与GPU Tensor之间的互相转换通过`tensor.cuda`和`tensor.cpu`方法实现，此外还可以使用`tensor.to(device)`；\n",
    "3. 创建同种类型的张量:`torch.*_like` 和 `tensor.new_*`。Tensor还有一个`new`方法，用法与`torch.Tensor`一样，会调用该Tensor对应类型的构造函数，生成与当前Tensor类型一致的Tensor。如`torch.*_like(tensorA)` 可以生成和`tensorA`拥有同样属性(类型，形状，CPU/GPU)的新Tensor，例如`torch.ones_like()`，而`tensor.new_*(new_shape)` 可以新建一个不同形状的Tensor，例如`tensor.new_zeros()`。\n",
    "\n",
    "下面将举例说明Tensor之间的转换："
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "source": [
    "# 更改默认Tensor的类型\r\n",
    "a = t.rand(2, 3)\r\n",
    "print(a.dtype)\r\n",
    "# 设置默认类型为DoubleTensor\r\n",
    "t.set_default_tensor_type('torch.DoubleTensor')\r\n",
    "a = t.rand(2, 3)\r\n",
    "print(a.dtype)\r\n",
    "# 恢复之前的默认设置\r\n",
    "t.set_default_tensor_type('torch.FloatTensor')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "torch.float32\n",
      "torch.float64\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "source": [
    "# 通过type方法和快捷方法修改Tensor的类型\r\n",
    "b1 = a.type(t.FloatTensor)\r\n",
    "b2 = a.float()\r\n",
    "a.dtype, b1.dtype, b2.dtype"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(torch.float64, torch.float32, torch.float32)"
      ]
     },
     "metadata": {},
     "execution_count": 27
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "source": [
    "# new方法相当于利用DoubleTensor的构造函数，因为此时a是torch.float64类型\r\n",
    "a.new(2,3)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[6.9427e-310, 6.9427e-310, 4.6877e-310],\n",
       "        [6.9427e-310, 4.6877e-310, 4.6877e-310]], dtype=torch.float64)"
      ]
     },
     "metadata": {},
     "execution_count": 28
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 索引操作\n",
    "\n",
    "Tensor支持与numpy.ndarray类似的索引操作，语法上也类似，下面将通过一些例子讲解常用的索引操作。其中，能通过修改张量stride/start/dim等属性实现的索引操作与原Tensor共享内存，也就是说如果修改其中一个，另一个会跟着修改。关于索引操作更详细的内容将在本书第六章中进行讲解。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "source": [
    "a = t.randn(3, 4)\r\n",
    "a"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[-0.0317,  1.7469, -1.4530, -0.4462],\n",
       "        [ 2.5300, -1.0586, -1.0968,  0.0187],\n",
       "        [-0.5891,  0.1420,  0.3084, -0.5744]])"
      ]
     },
     "metadata": {},
     "execution_count": 29
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "source": [
    "print(\"查看第1行结果：\", a[0])\r\n",
    "print(\"查看第2列结果：\", a[:,1])\r\n",
    "print(\"查看第2行最后两个元素：\", a[1, -2:])"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "查看第1行结果： tensor([-0.0317,  1.7469, -1.4530, -0.4462])\n",
      "查看第2列结果： tensor([ 1.7469, -1.0586,  0.1420])\n",
      "查看第2行最后两个元素： tensor([-1.0968,  0.0187])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "source": [
    "# 返回一个ByteTensor\r\n",
    "print(a > 0) # bool型\r\n",
    "print((a > 0).int()) # 整型"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[False,  True, False, False],\n",
      "        [ True, False, False,  True],\n",
      "        [False,  True,  True, False]])\n",
      "tensor([[0, 1, 0, 0],\n",
      "        [1, 0, 0, 1],\n",
      "        [0, 1, 1, 0]], dtype=torch.int32)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "source": [
    "# 返回Tensor中满足条件的结果，下面两种写法等价\r\n",
    "# 选择返回的结果与原Tensor不共享内存空间\r\n",
    "print(a[a > 0])\r\n",
    "print(a.masked_select(a>0)) \r\n",
    "# 用torch.where保留原始的索引位置，不满足条件的位置置0\r\n",
    "print(t.where(a > 0, a, t.zeros_like(a)))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([1.7469, 2.5300, 0.0187, 0.1420, 0.3084])\n",
      "tensor([1.7469, 2.5300, 0.0187, 0.1420, 0.3084])\n",
      "tensor([[0.0000, 1.7469, 0.0000, 0.0000],\n",
      "        [2.5300, 0.0000, 0.0000, 0.0187],\n",
      "        [0.0000, 0.1420, 0.3084, 0.0000]])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "其它常用的选择函数如表3-3所示。\n",
    "\n",
    "表3-3常用的选择函数\n",
    "\n",
    "函数|功能|\n",
    ":---:|:---:|\n",
    "index_select(input, dim, index)|在指定维度dim上选取，比如选取某些行、某些列\n",
    "masked_select(input, mask)|例子如上，a[a>0]，使用ByteTensor进行选取\n",
    "non_zero(input)|非0元素的下标\n",
    "gather(input, dim, index)|根据index，在dim维度上选取数据，输出的size与index一样\n",
    "\n",
    "\n",
    "`gather`是一个比较复杂的操作，对一个2维Tensor，输出的每个元素如下：\n",
    "\n",
    "```python\n",
    "out[i][j] = input[index[i][j]][j]  # dim=0\n",
    "out[i][j] = input[i][index[i][j]]  # dim=1\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "source": [
    "a = t.arange(0, 16).view(4, 4)\r\n",
    "a"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 0,  1,  2,  3],\n",
       "        [ 4,  5,  6,  7],\n",
       "        [ 8,  9, 10, 11],\n",
       "        [12, 13, 14, 15]])"
      ]
     },
     "metadata": {},
     "execution_count": 33
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "source": [
    "# 选取对角线的元素\r\n",
    "index = t.tensor([[0,1,2,3]])\r\n",
    "a.gather(0, index)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 0,  5, 10, 15]])"
      ]
     },
     "metadata": {},
     "execution_count": 34
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "source": [
    "# 选取反对角线上的元素\r\n",
    "index = t.tensor([[3,2,1,0]]).t()\r\n",
    "a.gather(1, index)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 3],\n",
       "        [ 6],\n",
       "        [ 9],\n",
       "        [12]])"
      ]
     },
     "metadata": {},
     "execution_count": 35
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "source": [
    "# 选取两个对角线上的元素\r\n",
    "index = t.tensor([[0,1,2,3],[3,2,1,0]]).t()\r\n",
    "b = a.gather(1, index)\r\n",
    "b"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 0,  3],\n",
       "        [ 5,  6],\n",
       "        [10,  9],\n",
       "        [15, 12]])"
      ]
     },
     "metadata": {},
     "execution_count": 36
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "与`gather`相对应的逆操作是`scatter_`，`gather`把数据从input中按index取出，而`scatter_`是把取出的数据再放回去。注意`scatter_`函数是inplace操作，会直接对当前数据进行修改。\n",
    "\n",
    "```python\n",
    "out = input.gather(dim, index)\n",
    "-->近似逆操作\n",
    "out = Tensor()\n",
    "out.scatter_(dim, index)\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "source": [
    "# 把两个对角线元素放回去到指定位置\r\n",
    "c = t.zeros(4,4).long()\r\n",
    "c.scatter_(1, index, b)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 0,  0,  0,  3],\n",
       "        [ 0,  5,  6,  0],\n",
       "        [ 0,  9, 10,  0],\n",
       "        [12,  0,  0, 15]])"
      ]
     },
     "metadata": {},
     "execution_count": 37
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "对Tensor的任何索引操作得到的结果仍是一个Tensor，若想要获取标准的Python对象数值，需要调用`tensor.item()`，需要注意的是这个方法只对包含一个元素的Tensor适用。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "source": [
    "t.Tensor([1.]).item()\r\n",
    "# t.Tensor([1, 2]).item()  ->\r\n",
    "# raise ValueError: only one element tensors can be converted to Python scalars"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "metadata": {},
     "execution_count": 38
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 高级索引\n",
    "\n",
    "目前的PyTorch已经支持绝大多数NumPy风格的高级索引。高级索引可以看成是普通索引操作的扩展，但是高级索引操作的结果一般不和原始的Tensor共享内存。关于更多高级索引的内容可以参考本书第六章第三节。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "source": [
    "x = t.arange(0,16).view(2,2,4)\r\n",
    "x"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[[ 0,  1,  2,  3],\n",
       "         [ 4,  5,  6,  7]],\n",
       "\n",
       "        [[ 8,  9, 10, 11],\n",
       "         [12, 13, 14, 15]]])"
      ]
     },
     "metadata": {},
     "execution_count": 39
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "source": [
    "x[[1, 0], [1, 1], [2, 0]] # x[1,1,2]和x[0,1,0]"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([14,  4])"
      ]
     },
     "metadata": {},
     "execution_count": 40
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "source": [
    "x[[1, 0], [0], [1]] # x[1,0,1],x[0,0,1]"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([9, 1])"
      ]
     },
     "metadata": {},
     "execution_count": 41
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 逐元素操作\n",
    "\n",
    "逐元素操作会对Tensor的每一个元素(point-wise，又名element-wise)进行操作，此类操作的输入与输出形状一致。常用的操作如表3-4所示。\n",
    "\n",
    "表3-4: 常见的逐元素操作\n",
    "\n",
    "|函数|功能|\n",
    "|:--:|:--:|\n",
    "|abs/sqrt/div/exp/fmod/log/pow..|绝对值/平方根/除法/指数/求余/对数/求幂..|\n",
    "|cos/sin/asin/atan2/cosh..|三角函数|\n",
    "|ceil/round/floor/trunc| 上取整/四舍五入/下取整/只保留整数部分|\n",
    "|clamp(input,min,max)|超过min和max部分截断|\n",
    "|sigmod/tanh..|激活函数\n",
    "\n",
    "对于很多操作，例如div、mul、pow、fmod等，PyTorch中都实现了运算符重载，因此读者可以很方便的直接使用。如`a ** 2` 等价于`torch.pow(a,2)`, `a * 2`等价于`torch.mul(a,2)`。\n",
    "\n",
    "其中`clamp(x, min, max)`的输出满足以下公式：\n",
    "$$\n",
    "y_i =\n",
    "\\begin{cases}\n",
    "min,  & \\text{if  } x_i \\lt min \\\\\n",
    "x_i,  & \\text{if  } min \\le x_i \\le max  \\\\\n",
    "max,  & \\text{if  } x_i \\gt max\\\\\n",
    "\\end{cases}\n",
    "$$\n",
    "`clamp`常用在某些需要比较大小的地方，如取一个Tensor的每个元素与另一个数的较大值。下面将举例说明一些常见的逐元素操作："
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "source": [
    "a = t.arange(0, 6).float().view(2, 3)\r\n",
    "t.cos(a)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 1.0000,  0.5403, -0.4161],\n",
       "        [-0.9900, -0.6536,  0.2837]])"
      ]
     },
     "metadata": {},
     "execution_count": 42
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "source": [
    "# 取模运算的运算符重载，二者等价\r\n",
    "print(a % 3)\r\n",
    "print(t.fmod(a, 3))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[0., 1., 2.],\n",
      "        [0., 1., 2.]])\n",
      "tensor([[0., 1., 2.],\n",
      "        [0., 1., 2.]])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "source": [
    "# 将a中的值进行上下限截断\r\n",
    "print(a)\r\n",
    "print(t.clamp(a, min=2, max=4))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[0., 1., 2.],\n",
      "        [3., 4., 5.]])\n",
      "tensor([[2., 2., 2.],\n",
      "        [3., 4., 4.]])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "####  归并操作 \n",
    "\n",
    "归并操作会使用Tensor中的部分元素进行计算，其输出形状可能小于输入形状。同时我们可以沿着某一维度进行指定的归并操作。如加法`sum`，既可以计算整个Tensor的和，也可以计算Tensor中每一行或每一列的和。常用的归并操作如表3-5所示。\n",
    "\n",
    "表3-5: 常用归并操作\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|mean/sum/median/mode|均值/求和/中位数/众数|\n",
    "|norm/dist|范数/距离|\n",
    "|std/var|标准差/方差|\n",
    "|cumsum/cumprod|累加/累乘|\n",
    "\n",
    "以上大多数函数都有一个维度参数`dim`，用来指定这些操作是在哪个维度上执行的。关于`dim`(对应于Numpy中的axis)的解释众说纷纭，这里提供一个简单的记忆方式：\n",
    "\n",
    "假设输入的形状是(m, n, k)\n",
    "\n",
    "- 如果指定dim=0，输出的形状就是(1, n, k)或者(n, k)\n",
    "- 如果指定dim=1，输出的形状就是(m, 1, k)或者(m, k)\n",
    "- 如果指定dim=2，输出的形状就是(m, n, 1)或者(m, n)\n",
    "\n",
    "其中size中是否有\"1\"，取决于参数`keepdim`，若指定`keepdim=True`则会保留维度`1`。注意，以上只是经验总结，并非所有函数都符合这种形状变化方式，如`cumsum`。下面将举例说明："
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "source": [
    "# 注意对比是否保留维度1的区别\r\n",
    "b = t.ones(2, 3)\r\n",
    "print(b.sum(dim=0, keepdim=True ),b.sum(dim=0, keepdim=True ).shape)\r\n",
    "print(b.sum(dim=0, keepdim=False), b.sum(dim=0, keepdim=False).shape)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[2., 2., 2.]]) torch.Size([1, 3])\n",
      "tensor([2., 2., 2.]) torch.Size([3])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "source": [
    "a = t.arange(2, 8).view(2, 3)\r\n",
    "print(a)\r\n",
    "print(a.cumsum(dim=1)) # 沿着行累加"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[2, 3, 4],\n",
      "        [5, 6, 7]])\n",
      "tensor([[ 2,  5,  9],\n",
      "        [ 5, 11, 18]])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 比较\n",
    "\n",
    "比较函数中有一些是逐元素比较，操作类似于逐元素操作，还有一些则类似于归并操作。常用比较函数如表3-6所示。\n",
    "\n",
    "表3-6: 常用比较函数\n",
    "\n",
    "|函数|功能|\n",
    "|:--:|:--:|\n",
    "|gt/lt/ge/le/eq/ne|大于/小于/大于等于/小于等于/等于/不等于|\n",
    "|topk|最大的k个数|\n",
    "|sort|排序|\n",
    "|max/min|比较两个Tensor的最大、最小值|\n",
    "\n",
    "表中第一行的比较操作已经实现了运算符重载，因此可以使用`a>=b`、`a>b`、`a!=b`、`a==b`，其返回结果是一个`ByteTensor`，可用来选取元素。max/min这两个操作比较特殊，以max来说，它有以下三种使用情况：\n",
    "- t.max(tensor)：返回Tensor中最大的一个数\n",
    "- t.max(tensor,dim)：指定维上最大的数，返回Tensor和下标\n",
    "- t.max(tensor1, tensor2): 返回两个Tensor相比较大的元素\n",
    "\n",
    "而若要比较一个Tensor和一个数，可以使用clamp函数。下面举例说明。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "source": [
    "a = t.linspace(0, 15, 6).view(2, 3)\r\n",
    "b = t.linspace(15, 0, 6).view(2, 3)\r\n",
    "print(a > b)\r\n",
    "print(\"a中大于b的元素: \", a[a > b]) # 返回a中大于b的元素\r\n",
    "print(\"a中最大的元素: \", t.max(a)) # 返回a中最大的元素"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[False, False, False],\n",
      "        [ True,  True,  True]])\n",
      "a中大于b的元素:  tensor([ 9., 12., 15.])\n",
      "a中最大的元素:  tensor(15.)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "source": [
    "t.max(b, dim=1)\r\n",
    "# 第一个返回值15和6分别表示第0行和第1行最大的元素\r\n",
    "# 第二个返回值的0和0表示上述最大的数是该行第0个元素"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "torch.return_types.max(\n",
       "values=tensor([15.,  6.]),\n",
       "indices=tensor([0, 0]))"
      ]
     },
     "metadata": {},
     "execution_count": 48
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "source": [
    "t.max(a, b) # 两个Tensor对应位置上较大的元素"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[15., 12.,  9.],\n",
       "        [ 9., 12., 15.]])"
      ]
     },
     "metadata": {},
     "execution_count": 49
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 线性代数\n",
    "\n",
    "PyTorch的线性函数主要封装了Blas和Lapack，其用法和接口都与之类似。常用的线性代数函数如表3-7所示。\n",
    "\n",
    "表3-7: 常用的线性代数函数\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|trace|对角线元素之和(矩阵的迹)|\n",
    "|diag|对角线元素|\n",
    "|triu/tril|矩阵的上三角/下三角，可指定偏移量|\n",
    "|mm/bmm|矩阵乘法，batch的矩阵乘法|\n",
    "|addmm/addbmm/addmv|矩阵运算\n",
    "|t|转置|\n",
    "|dot/cross|内积/外积\n",
    "|inverse|求逆矩阵\n",
    "|svd|奇异值分解\n",
    "\n",
    "具体使用说明请参见官方文档，需要注意的是，矩阵的转置会导致存储空间不连续，因此需调用它的`.contiguous`方法将其转为连续。下面将举例说明：\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "source": [
    "a = t.linspace(0, 15, 6).view(2, 3)\r\n",
    "b = a.t()\r\n",
    "b.is_contiguous()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "metadata": {},
     "execution_count": 50
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "source": [
    "b = b.contiguous()\r\n",
    "b.is_contiguous()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 51
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.1.2 Tensor与NumPy\n",
    "\n",
    "Tensor和NumPy数组之间具有很高的相似性，彼此之间的互相操作也非常简单高效。需要注意的是，NumPy和Tensor共享内存。由于NumPy中已经封装了很多常用操作，所以当遇到Tensor不支持的操作时，可先将其转换成NumPy数组，进行相应处理后再转回Tensor。由于这样操作的转换开销很小，因此在实际应用中我们将经常进行两者的相互转换，下面将举例说明。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "source": [
    "import numpy as np\r\n",
    "a = np.ones([2, 3],dtype=np.float32)\r\n",
    "a"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "array([[1., 1., 1.],\n",
       "       [1., 1., 1.]], dtype=float32)"
      ]
     },
     "metadata": {},
     "execution_count": 52
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "source": [
    "# 从NumPy转化为Tensor，下面两种方式等价\r\n",
    "b = t.from_numpy(a)\r\n",
    "# 注意此时是大写的Tensor，torch.tensor()只进行数据拷贝，不会共享内存。\r\n",
    "# b = t.Tensor(a)\r\n",
    "b"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "metadata": {},
     "execution_count": 53
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "source": [
    "# 此时，NumPy和Tensor是共享内存的\r\n",
    "a[0, 1] = -1\r\n",
    "b # 修改a的值，b的值也会被修改"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 1., -1.,  1.],\n",
       "        [ 1.,  1.,  1.]])"
      ]
     },
     "metadata": {},
     "execution_count": 54
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**注意**： 当NumPy的数据类型和Tensor的类型不一样的时候，数据会被复制，不会共享内存。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "source": [
    "a = np.ones([2, 3])\r\n",
    "# 注意和上面的a的区别（dtype不是float32）\r\n",
    "a.dtype"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "metadata": {},
     "execution_count": 55
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "source": [
    "b = t.Tensor(a) # 此处进行拷贝，不共享内存\r\n",
    "b.dtype"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "metadata": {},
     "execution_count": 56
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "source": [
    "c = t.from_numpy(a) # 注意c的类型（DoubleTensor）\r\n",
    "c"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], dtype=torch.float64)"
      ]
     },
     "metadata": {},
     "execution_count": 57
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "source": [
    "a[0, 1] = -1\r\n",
    "print(b) # b与a不共享内存，所以即使a改变了，b也不变\r\n",
    "print(c) # c与a共享内存"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]])\n",
      "tensor([[ 1., -1.,  1.],\n",
      "        [ 1.,  1.,  1.]], dtype=torch.float64)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**注意：** 不论输入的类型是什么，torch.tensor()都只进行进行数据拷贝，不会共享内存。读者需要注意区分torch.Tensor(),torch.from_numpy()与torch.tensor()在内存共享方面的区别。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "source": [
    "a_tensor = t.tensor(a)\r\n",
    "a_tensor[0, 1] = 1\r\n",
    "a # a和a_tensor不共享内存"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "array([[ 1., -1.,  1.],\n",
       "       [ 1.,  1.,  1.]])"
      ]
     },
     "metadata": {},
     "execution_count": 59
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.1.3 Tensor的基本结构\r\n",
    "\r\n",
    "Tensor的数据结构如图3-1所示。Tensor分为头信息区(Tensor)和存储区(Storage)，信息区主要保存着Tensor的形状（size）、步长（stride）、数据类型（type）等信息，而真正的数据则保存成连续数组。由于数据动辄成千上万，因此信息区元素占用内存较少，主要内存占用取决于Tensor中元素的数目，也即存储区的大小。\r\n",
    "\r\n",
    "一般来说一个Tensor有与之对应的Storage，Storage是在data之上封装的接口，便于使用，而Tensor的内存地址是指向Tensor的头（head）。不同Tensor的head信息一般不同，但却可能使用相同的Storage。很多关于Tensor的操作只是创建了一个新的head，但是它们仍共享同一个Storage，下面将举例说明。\r\n",
    "\r\n",
    "![图3-1: Tensor的数据结构](imgs/tensor_data_structure.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "source": [
    "a = t.arange(0, 6).float()\r\n",
    "b = a.view(2, 3)\r\n",
    "# id(变量)可以查看它在内存中的地址\r\n",
    "# Storage的内存地址一样，即它们是同一个Storage\r\n",
    "id(b.storage()) == id(a.storage())"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 60
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "source": [
    "# a改变，b也随之改变，因为他们共享Storage\r\n",
    "a[1] = 100\r\n",
    "b"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[  0., 100.,   2.],\n",
       "        [  3.,   4.,   5.]])"
      ]
     },
     "metadata": {},
     "execution_count": 61
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "source": [
    "# c取得a的部分索引，只改变了head信息，Storage相同\r\n",
    "c = a[2:] \r\n",
    "id(c.storage()) == id(a.storage())"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 62
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "source": [
    "c.data_ptr(), a.data_ptr() # data_ptr返回Tensor首元素的内存地址\r\n",
    "# 可以看出两个内存地址相差8，这是因为2*4=8：相差两个元素，每个元素占4个字节(float)\r\n",
    "# 如果差值不是8（如16），可以用a.type()查看一下数据类型是不是torch.FloatTensor"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(94880397551496, 94880397551488)"
      ]
     },
     "metadata": {},
     "execution_count": 63
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "source": [
    "c[0] = -100 # c[0]的内存地址对应a[2]的内存地址\r\n",
    "a"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([   0.,  100., -100.,    3.,    4.,    5.])"
      ]
     },
     "metadata": {},
     "execution_count": 64
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "source": [
    "d = t.Tensor(c.storage()) # 此处拷贝共享了内存\r\n",
    "d[0] = 6666\r\n",
    "b"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 6.6660e+03,  1.0000e+02, -1.0000e+02],\n",
       "        [ 3.0000e+00,  4.0000e+00,  5.0000e+00]])"
      ]
     },
     "metadata": {},
     "execution_count": 65
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "source": [
    "# 下面４个Tensor共享Storage\r\n",
    "id(a.storage()) == id(b.storage()) == id(c.storage()) == id(d.storage())"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 66
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "source": [
    "# c取得a的部分索引，改变了偏移量\r\n",
    "a.storage_offset(), c.storage_offset(), d.storage_offset()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(0, 2, 0)"
      ]
     },
     "metadata": {},
     "execution_count": 67
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "source": [
    "e = b[::2, ::2] # 隔2行/列取一个元素\r\n",
    "print(id(e.storage()) == id(a.storage())) # 共享内存\r\n",
    "print(e.is_contiguous()) # e的存储空间是不连续的"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "True\n",
      "False\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "可见绝大多数操作并不修改Tensor的数据，而只是修改了Tensor的头信息。这种做法更节省内存，同时提升了处理速度。在使用中需要注意：有些操作会导致Tensor不连续，这时需要调用`tensor.contiguous`方法将它们变成连续的数据，该方法会使数据复制一份，不再与原来的数据共享Storage。\n",
    "另外读者可以思考一下，之前说过的高级索引一般不共享Storage，而普通索引则共享Storage，这是为什么？（提示：普通索引可以通过修改Tensor的offset、stride和size实现，不修改Storage的数据，高级索引则不行）。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3.2 变形记：N种改变Tensor形状的方法\n",
    "\n",
    "Tensor作为PyTorch的基本数据对象，使用过程中常需要改变Tensor的形状。在PyTorch中有很多用于改变形状的函数，在这一小节中我们将详细介绍一些常用的方法。\n",
    "\n",
    "本小节介绍的所有函数都可以用`tensor.shape` 和`tensor.reshape(*new_shape)`来实现，这里笔者将对常见的Tensor形状相关的操作进行总结，以方便读者选择最灵活便捷的函数。\n",
    "\n",
    "\n",
    "### 查看Tensor的维度\n",
    "\n",
    "关于Tensor的形状信息，除了`tensor.shape`， 还有以下三个常用函数：\n",
    "- `tensor.size()` 等价于`tensor.shape`；\n",
    "- `tensor.dim()` 能查看Tensor的维度，等效于`len(tensor.shape)`,对应于NumPy中的`array.ndim`；\n",
    "- `tensor.numel()` 用来查看tensor中元素的数量，等价于`tensor.shape[0]*tensor.shape[1]*...`或者`np.prod(tensor.shape)`, 对应于NumPy中的`array.size`。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "source": [
    "import torch as t\r\n",
    "tensor = t.arange(24).reshape(2, 3, 4)\r\n",
    "# tensor.shape 和tensor.size() 等价\r\n",
    "print(f\"a.shape={tensor.shape}. a.size()={tensor.size()}\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "a.shape=torch.Size([2, 3, 4]). a.size()=torch.Size([2, 3, 4])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "source": [
    "f'这是个{tensor.dim()}维Tensor, 总共{tensor.numel()}个元素'"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'这是个3维Tensor, 总共24个元素'"
      ]
     },
     "metadata": {},
     "execution_count": 70
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 改变Tensor的维度\n",
    "所有改变Tensor形状的操作都可以通过`tensor.reshape`实现。`tensor.reshape(new_shape)`会把不连续的Tensor变成连续的再进行形状变化，这一操作等价于`tensor.contiguous().view(new_shape)`。关于`view`和`reshape`的选用可参考下面的建议:\n",
    "\n",
    "1. 对于`reshape`而言:其接口更便捷，会自动把不连续的Tensor变为连续的，能避免很多报错。同时它的函数名和NumPy一样，便于使用。\n",
    "2. 对于`view`而言:其函数名更短，而且Tensor经过`view`操作之后仍然共享存储空间。当我们并不希望改变形状之后的Tensor和原Tensor共享存储时可以使用`view`方法。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "`reshape`和`view`二者之间有一些区别：如果对Tensor使用了`transpose`、`permute`等操作，会造成Tensor的内存变得不连续，而`view`方法只能改变连续的张量，所以需要先调用`contiguous()`方法，但`reshape`方法则不受此限制。具体来说，`view`方法返回的Tensor和原Tensor共享Storage（注意：不是共享内存地址），而`reshape`方法可能返回原Tensor的copy或者共享Storage的view。如果当前满足连续性条件，则结果与`view`相同；否则返回的就是copy（此时等价于`tensor.contiguous().view()`）。"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "source": [
    "a = t.arange(1, 13)\r\n",
    "b = a.view(2, 6)\r\n",
    "c = a.reshape(2, 6) # 此时view和reshape等价，因为Tensor是contiguous\r\n",
    "# a,b,c三个对象的内存地址是不一样的，其中保存的是Tensor的形状(size)、步长(stride)、数据类型(type)等信息\r\n",
    "id(a)==id(b)==id(c)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "metadata": {},
     "execution_count": 71
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "source": [
    "# view和reshape存储在与原始对象不同的地址内存中，但是它们共享存储器Storage，也就意味着它们共享基础数据\r\n",
    "id(a.storage())==id(b.storage())==id(c.storage())"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 72
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "source": [
    "b = b.t() # b 不再连续\r\n",
    "b.reshape(-1, 4) # reshape 可以\r\n",
    "\r\n",
    "# 下面会报错，view无法在改变数据存储的情况下进行形状变化\r\n",
    "# b.view(-1, 4)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 1,  7,  2,  8],\n",
       "        [ 3,  9,  4, 10],\n",
       "        [ 5, 11,  6, 12]])"
      ]
     },
     "metadata": {},
     "execution_count": 73
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "reshape操作的难点在于如何快速灵活地指定形状，常用的快捷变形方法有：\n",
    "\n",
    "1. `tensor.view(dim1,-1,dimN)`: 在调整Tensor形状的时候，我们不需要指定每一维的形状，可以把其中一个维度指定为-1，PyTorch会自动计算对应的形状\n",
    "2. `tensor.view_as(other)`: 将Tensor的形状变为和other 一样，等价于`tensor.view(other.shape)`\n",
    "3. `tensor.squeeze()`: 将Tensor形状中为`1`的维度去掉，比如(1,3,1,4)变为(3,4)\n",
    "4. `tensor.flatten(start_dim=0, end_dim=-1)`: 将Tensor形状中某些连续维度合并为一个维度. 比如形状(2,3,4,5) ->(2,12,5)。\n",
    "5. `tensor[None]` 和`tensor.unsqueeze(dim)`: 为Tensor新建一个维度，并把形状设为1, 比如形状(2,3) -> (2,1,3)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "source": [
    "# 创建一张噪声图像, 并计算RGB每一个通道的噪声均值\r\n",
    "img_3xHxW = t.randn(3,128,256) \r\n",
    "\r\n",
    "# 将img_3xHxW的后两维合并\r\n",
    "img_3xHW = img_3xHxW.view(3, -1)\r\n",
    "# 写法等价于img_3xHxW.view(3,128*256)\r\n",
    "# 也等价于img_3xHxW.flatten(1,2)\r\n",
    "img_3xHW.mean(dim=1)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([-1.6643e-03, -3.8993e-03,  8.6497e-07])"
      ]
     },
     "metadata": {},
     "execution_count": 74
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "source": [
    "# 图片除了RGB通道，还可以有alpha通道用来表示透明度\r\n",
    "alpha_HxW = t.rand(128,256)\r\n",
    "alpha_1xHxW = alpha_HxW[None] # 等价于 alpha.unsqueeze(0)\r\n",
    "rgba_img = t.cat([alpha_1xHxW, img_3xHxW], dim=0)\r\n",
    "rgba_img.shape "
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "torch.Size([4, 128, 256])"
      ]
     },
     "metadata": {},
     "execution_count": 75
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "source": [
    "# 去掉第一维的1\r\n",
    "# 也可以使用alpha_1xHxW.squeeze()：去掉所有为1的维度\r\n",
    "# 也可以使用alpha_1xHxW.flatten(0,1): 1和128合并\r\n",
    "# 也可以使用alpha_1xHxW[0]： 通过索引取出第一维的数据\r\n",
    "alpha_HxW = alpha_1xHxW.view(128, 256)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 转置`torch.transpose/tensor.permute`\n",
    "\n",
    "注意，Tensor的转置（transpose）和改变形状（reshape）是两个不一样的概念。比如将一张图片旋转90度，就属于向量的转置，无法通过单纯的改变向量形状实现。\n",
    "\n",
    "`transpose`和`permute`函数都可以用于高维矩阵的转置，但用法上稍有区别：`transpose`只能用于两个维度的转置，意思就是只能改变两个维度的信息，而`permute`可以对任意高维矩阵进行转置，直接输入转置后维度index即可。我们可以通过多次`transpose`变换达到和`permute`相同的效果。常用的转置操作还有`tensor.t()`和`tensor.T`，它们和`tensor.transpose()`一样都属于`permute`的特例。\n",
    "\n",
    "另外，向量的转置操作大多数情况下和输入的Tensor共享存储，但是会使得向量变得不连续, 此时最好通过`tensor.contiguous()`将它变成连续。但是有些操作(比如`tensor.sum/max`)支持不连续的Tensor，那么你可以不用将它变成连续, 这样可以节省内存/显存。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "source": [
    "mask = t.arange(6).view(2,3) # 一张高为2，宽为3的图片\r\n",
    "mask"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[0, 1, 2],\n",
       "        [3, 4, 5]])"
      ]
     },
     "metadata": {},
     "execution_count": 77
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "source": [
    "# 将图片旋转90度, 也就是第一个维度(0)和第二个维度交换\r\n",
    "# 等价于mask.transpose(1,0)\r\n",
    "# 等价于mask.t() 或 img.T\r\n",
    "# 等价于mask.permute(1,0), 不等价于img.permute(0,1)\r\n",
    "mask.transpose(0,1)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[0, 3],\n",
       "        [1, 4],\n",
       "        [2, 5]])"
      ]
     },
     "metadata": {},
     "execution_count": 78
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "source": [
    "# 单纯地改变图片的形状\r\n",
    "# 注意和上面的区别，结果仍然是连续的\r\n",
    "mask.view(3,2)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[0, 1],\n",
       "        [2, 3],\n",
       "        [4, 5]])"
      ]
     },
     "metadata": {},
     "execution_count": 79
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "source": [
    "# PyTorch等深度学习框架中图片一般存储为3xHxW\r\n",
    "img_3xHxW = t.randn(3,128,256) \r\n",
    "\r\n",
    "# 而在OpenCV/NumPy/skimage 中，图片一般存储为HxWx3\r\n",
    "# img_3xHxW的形状为shape=[3,H,W],经过permute(1,2,0)得到的形状为:\r\n",
    "# [shape[1], shape[2], shape[0]] = [H, W, 3]\r\n",
    "img_HxWx3 = img_3xHxW.permute(1,2,0)\r\n",
    "\r\n",
    "img_HxWx3.is_contiguous()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "metadata": {},
     "execution_count": 80
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "source": [
    "img_HxWx3.reshape(-1) # .view 会报错，因为img_HxWx3不连续"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([ 0.4553,  1.0848, -1.9221,  ..., -0.6087,  0.2638, -0.0149])"
      ]
     },
     "metadata": {},
     "execution_count": 81
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "一个用来区分什么时候用`tensor.reshape`还是`tensor.transpose`的小技巧：如果输出Tensor的数据排列和输入一样就是用`tensor.reshape`，否则应该使用`tensor.transpose`。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "source": [
    "H, W = 4, 5\r\n",
    "img_3xHxW = t.randn(3, H, W)\r\n",
    "\r\n",
    "# 目标数据排列和输入一样，直接使用reshape\r\n",
    "img_3xHW = img_3xHxW.reshape(3, -1)\r\n",
    "\r\n",
    "# 目标数据排列和输入不一样，先通过transpose变成(3,W,H), 再变成(3,WH)\r\n",
    "img_3xWH = img_3xHxW.transpose(1, 2).reshape(3, -1)\r\n",
    "\r\n",
    "# 再变形为3xWxH的形式\r\n",
    "img_3xWxH = img_3xWH.reshape(3, W, H)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3.3 小试牛刀：线性回归\n",
    "\n",
    "线性回归是机器学习的入门知识，应用十分广泛。线性回归利用数理统计中的回归分析来确定两种或两种以上变量间相互依赖的定量关系，其表达形式为$y = wx+b+e$，误差$e$服从均值为0的正态分布。线性回归的损失函数是：\n",
    "$$\n",
    "loss = \\sum_i^N \\frac 1 2 ({y_i-(wx_i+b)})^2\n",
    "$$\n",
    "我们可以利用随机梯度下降法更新参数$\\textbf{w}$和$\\textbf{b}$来最小化损失函数，最终学得$\\textbf{w}$和$\\textbf{b}$的数值。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "source": [
    "import torch as t\r\n",
    "%matplotlib inline\r\n",
    "from matplotlib import pyplot as plt\r\n",
    "from IPython import display\r\n",
    "\r\n",
    "device = t.device('cpu') #如果你想用GPU，改成t.device('cuda:0')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "source": [
    "# 设置随机数种子，保证在不同电脑上运行时下面的输出一致\r\n",
    "t.manual_seed(1000) \r\n",
    "\r\n",
    "def get_fake_data(batch_size=8):\r\n",
    "    ''' 产生随机数据：y=x*2+3，加上了一些噪声'''\r\n",
    "    x = t.rand(batch_size, 1, device=device) * 5\r\n",
    "    y = x * 2 + 3 +  t.randn(batch_size, 1, device=device)\r\n",
    "    return x, y"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "source": [
    "# 来看看产生的x-y分布\r\n",
    "x, y = get_fake_data(batch_size=16)\r\n",
    "plt.scatter(x.squeeze().cpu().numpy(), y.squeeze().cpu().numpy())"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7fcd24179c88>"
      ]
     },
     "metadata": {},
     "execution_count": 85
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ],
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAAD4CAYAAADIH9xYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAP+ElEQVR4nO3dX2xk513G8efBcZRJ2sooO0DsJGwrVebPrhKvRquUiCiQgNM0apZVL4LUIirEAgqQVMKoywVVr/bCCPHnArRqi4ooQSXdWFXU1OkFoepFUmbjpLvbjSENpM24kAnFTUNGzcb8uPDYsSe258zOnJnzHn8/krXjM++Of2ff3WfOvud953VECABQfD8y6gIAANkQ2ACQCAIbABJBYANAIghsAEjEFXm86IEDB+LgwYN5vDQAlNLZs2dfiYjqXm1yCeyDBw+qXq/n8dIAUEq2X+zWhiERAEgEgQ0AiSCwASARBDYAJILABoBE5DJLBAD2k4WlhuYXl7Wy2tLkREVzs9M6NjM18J9DYANAHxaWGjp55pxal9YkSY3Vlk6eOSdJAw9thkQAoA/zi8ubYb2hdWlN84vLA/9ZBDYA9GFltdXT8X4Q2ADQh8mJSk/H+0FgA0Af5manVRkf23asMj6mudnpgf8sbjoCQB82biwySwQAEnBsZiqXgO7EkAgAJILABoBEENgAkAjGsAEU2rCWfaeAwAZQWMNc9p0ChkQAFNYwl32nIFNg237A9nnbF2w/mHNNACBpuMu+U9A1sG0fkvSbko5KuknSPbbfm3dhADDMZd8pyHKF/dOSnoyI1yPiTUn/LOlX8i0LAIa77DsFWQL7vKTbbF9r+2pJd0u6obOR7RO267brzWZz0HUC2IeOzUzp1PHDmpqoyJKmJio6dfzwvrzhKEmOiO6N7N+QdL+k1yR9U1IrIj62W/tarRb1en1gRQLAbsoy7c/22Yio7dUm003HiPh0RByJiNskfU/Svw2iQADox8a0v8ZqS6G3pv0tLDVGXVouss4S+bH2rzdKOi7poTyLAoAs9tu0v6wLZ75g+1pJlyTdHxH/k2NNAJDJfpv2lymwI+Ln8y4EAHo1OVFRY4dwLuu0P1Y6AkjWfpv2x2eJAEjWMHd7KQICG0DShrXbSxEwJAIAieAKG0AplWVBzVYENoDSKevnaDMkAqB0yrqghsAGUDplXVBDYAMonbJ+jjaBDaB0yrqghpuOAEqnrAtqCGwApVTGBTUMiQBAIghsAEgEgQ0AiSCwASARBDYAJILABoBEENgAkAgCGwASkSmwbX/M9gXb520/ZPuqvAsDAGzXNbBtT0n6fUm1iDgkaUzSfXkXBgDYLuuQyBWSKravkHS1pJX8SgIA7KTrZ4lERMP2n0j6tqSWpMcj4vHOdrZPSDohSTfeeOOg6wSwgzJug4XdZRkS+VFJ90p6t6RJSdfY/nBnu4g4HRG1iKhVq9XBVwpgm41tsBqrLYXe2gZrYakx6tKQkyxDIndK+veIaEbEJUlnJP1cvmUB6Kas22Bhd1kC+9uSbrF9tW1LukPSxXzLAtBNWbfBwu66BnZEPCXpYUlPSzrX/j2nc64LQBdl3QYLu8s0SyQiPhERPxURhyLiIxHxw7wLA7C3sm6Dhd2x4wyQqLJug4XdEdhAwsq4DRZ2x2eJAEAiCGwASARDIsCQsToRl4vABoZoY3XixoKXjdWJkghtdEVgoxRSuWrda3ViEetFsRDYSF5KV62sTkQ/uOmI5KX0mRqsTkQ/CGwkL6WrVlYnoh8MiSB5kxMVNXYI52FftWYZR2d1IvpBYCN5c7PT28awpeFftfYyjs7qRFwuhkSQvGMzUzp1/LCmJiqypKmJik4dPzzUUExpHB3p4gobpTDqq9aUxtGRLq6wgQFg9geGgcAGBoDZHxgGhkSAAWD2B4aBwAYGZNTj6Cg/hkQAIBEENgAkomtg2562/cyWr1dtPziE2gAAW3Qdw46IZUk3S5LtMUkNSY/kWxYAoFOvQyJ3SPpWRLyYRzEAgN31Gtj3SXpopydsn7Bdt11vNpv9VwYA2CZzYNu+UtIHJf3jTs9HxOmIqEVErVqtDqo+AEBbL1fY75f0dET8V17FAAB210tg/6p2GQ4BAOQv00pH21dL+iVJv5VvOUD5pbJhMIonU2BHxOuSrs25FqD0UtowGMXDZ4mg1Ip2NbvXRgcENrohsFFaRbyaZaMD9IPPEkFpFXHbLjY6QD8IbJRWEa9m2egA/SCwUVpFvJotwobBSBdj2CitudnpbWPYUjGuZtnoAJeLwEZpsW0XyobARqlxNYsyYQwbABJBYANAIghsAEgEgQ0AiSCwASARBDYAJILABoBEENgAkAgCGwASQWADQCIIbABIBIENAInIFNi2J2w/bPs52xdtvy/vwgAA22X9tL4/l/TliPiQ7SslXZ1jTQCAHXQNbNvvknSbpF+XpIh4Q9Ib+ZYFAOiUZUjkPZKakv7G9pLtT9m+Jue6AAAdsgT2FZKOSPqriJiR9L+SPt7ZyPYJ23Xb9WazOeAyAQBZAvslSS9FxFPt7x/WeoBvExGnI6IWEbVqtTrIGgEAyhDYEfGfkr5je2Pn0jskfTPXqgAAb5N1lsjvSfpce4bIC5I+ml9JAICdZArsiHhGUi3fUgAAe2GlIwAkgsAGgERkHcMGtLDU0PzislZWW5qcqGhudlrHZqZGXRawbxDYyGRhqaGTZ86pdWlNktRYbenkmXOSVKjQ5k0FZcaQCDKZX1zeDOsNrUtrml9cHlFFb7fxptJYbSn01pvKwlJj1KUBA0FgI5OV1VZPx0chhTcVoB8ENjKZnKj0dHwUUnhTAfpBYCOTudlpVcbHth2rjI9pbnZ6l98xfCm8qQD9ILCRybGZKZ06flhTExVZ0tRERaeOHy7UDb0U3lSAfjBLBJkdm5kqVEB32qiNWSIoKwIb26Q+La7obypAPwhsbEplrjWwXzGGjU1MiwOKjcDGJqbFAcVGYGMT0+KAYiOwsYlpcUCxcdMRm5gWBxQbgY1tmBYHFBdDIgCQCAIbABJBYANAIjKNYdv+D0k/kLQm6c2IYAd1DEzqy+GBYenlpuMvRMQruVWCfYnl8EB2DIlgpFgOD2SXNbBD0uO2z9o+sVMD2yds123Xm83m4CpEqbEcHsgua2DfGhFHJL1f0v22b+tsEBGnI6IWEbVqtTrQIlFeLIcHsssU2BGx0v71ZUmPSDqaZ1HYP1gOD2TXNbBtX2P7nRuPJf2ypPN5F4b9IYWtx4CiyDJL5MclPWJ7o/3fR8SXc60K+wrL4YFsugZ2RLwg6aYh1AIA2APT+gAgEQQ2ACSCwAaARBDYAJAIAhsAEkFgA0AiCGwASASBDQCJILABIBEENgAkgsAGgEQQ2ACQCAIbABJBYANAIghsAEgEgQ0AiSCwASARBDYAJILABoBEZNmEdygWlhqaX1zWympLkxMVzc1OszErAGyRObBtj0mqS2pExD2DLGJhqaGTZ86pdWlNktRYbenkmXOSRGgDQFsvQyIPSLqYRxHzi8ubYb2hdWlN84vLefw4AEhSpsC2fb2kD0j6VB5FrKy2ejoOAPtR1ivsP5P0h5L+b7cGtk/YrtuuN5vNnoqYnKj0dBwA9qOugW37HkkvR8TZvdpFxOmIqEVErVqt9lTE3Oy0KuNj245Vxsc0Nzvd0+sAQJlluel4q6QP2r5b0lWS3mX77yLiw4MqYuPGIrNEAGB3jojsje3bJf1Bt1kitVot6vV6f5UBwD5i+2xE1PZqw8IZAEhETwtnIuIJSU/kUgkAYE9cYQNAIghsAEgEgQ0AiSCwASARBDYAJILABoBEENgAkAgCGwASQWADQCIIbABIBIENAIkgsAEgEQQ2ACSCwAaARBDYAJAIAhsAEkFgA0AiCGwASASBDQCJILABIBFdA9v2Vba/bvtZ2xdsf3IYhQEAtsuya/oPJf1iRLxme1zS12w/FhFP5lwbAGCLroEdESHptfa34+2vyLMoAMDbZRrDtj1m+xlJL0v6SkQ8lWtVAIC3yRTYEbEWETdLul7SUduHOtvYPmG7brvebDYHXCYAoKdZIhGxKukJSXft8NzpiKhFRK1arQ6mOgDApiyzRKq2J9qPK5LulPRcznUBADpkmSVynaTP2h7TesB/PiIezbcsAECnLLNEviFpZgi1AAD2kOUKe99YWGpofnFZK6stTU5UNDc7rWMzU6MuCwAkEdibFpYaOnnmnFqX1iRJjdWWTp45J0mENoBC4LNE2uYXlzfDekPr0prmF5dHVBEAbEdgt62stno6DgDDRmC3TU5UejoOAMNGYLfNzU6rMj627VhlfExzs9MjqggAtuOmY9vGjUVmiQAoKgJ7i2MzUwQ0gMJiSAQAEkFgA0AiCGwASASBDQCJILABIBFe37JxwC9qNyW9OPAXvnwHJL0y6iL6kHr9EudQFKmfQ+r1S7ufw09GxJ67v+QS2EVjux4RtVHXcblSr1/iHIoi9XNIvX6pv3NgSAQAEkFgA0Ai9ktgnx51AX1KvX6JcyiK1M8h9fqlPs5hX4xhA0AZ7JcrbABIHoENAIkoTWDbvsv2su3nbX98h+dvt/1928+0v/54FHXuxfZnbL9s+/wuz9v2X7TP8Ru2jwy7xr1kqD+FPrjB9j/Zvmj7gu0HdmhT2H7IWH+h+8H2Vba/bvvZ9jl8coc2he0DKfM59N4PEZH8l6QxSd+S9B5JV0p6VtLPdLS5XdKjo661y3ncJumIpPO7PH+3pMckWdItkp4adc091p9CH1wn6Uj78Tsl/esOf5cK2w8Z6y90P7T/XN/Rfjwu6SlJt6TSBz2cQ8/9UJYr7KOSno+IFyLiDUn/IOneEdfUs4j4qqTv7dHkXkl/G+uelDRh+7rhVNddhvoLLyK+GxFPtx//QNJFSZ0fkl7YfshYf6G1/1xfa3873v7qnB1R2D6QMp9Dz8oS2FOSvrPl+5e081/S97X/i/KY7Z8dTmkDlfU8iyyZPrB9UNKM1q+OtkqiH/aoXyp4P9ges/2MpJclfSUikuuDDOcg9dgPZQls73Cs893saa2v1b9J0l9KWsi7qBxkOc8iS6YPbL9D0hckPRgRr3Y+vcNvKVQ/dKm/8P0QEWsRcbOk6yUdtX2oo0nh+yDDOfTcD2UJ7Jck3bDl++slrWxtEBGvbvwXJSK+JGnc9oHhlTgQXc+zyFLpA9vjWg+7z0XEmR2aFLofutWfSj9IUkSsSnpC0l0dTxW6D7ba7Rwupx/KEtj/Ium9tt9t+0pJ90n64tYGtn/CttuPj2r93P976JX254uSfq19h/wWSd+PiO+OuqisUuiDdn2flnQxIv50l2aF7Ycs9Re9H2xXbU+0H1ck3SnpuY5mhe0DKds5XE4/lGIT3oh40/bvSlrU+oyRz0TEBdu/3X7+ryV9SNLv2H5TUkvSfdG+VVsUth/S+p3jA7ZfkvQJrd+s2DiHL2n97vjzkl6X9NHRVLqzDPUXvg8k3SrpI5LOtccfJemPJN0oJdEPWeovej9cJ+mztse0HmKfj4hHO/49F7kPpGzn0HM/sDQdABJRliERACg9AhsAEkFgA0AiCGwASASBDQCJILABIBEENgAk4v8BcE8sn0KhT7EAAAAASUVORK5CYII="
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "source": [
    "# 随机初始化参数\r\n",
    "w = t.rand(1, 1).to(device)\r\n",
    "b = t.zeros(1, 1).to(device)\r\n",
    "\r\n",
    "lr =0.02 # 学习率learning rate\r\n",
    "\r\n",
    "for ii in range(500):\r\n",
    "    x, y = get_fake_data(batch_size=4)\r\n",
    "    \r\n",
    "    # forward：计算loss\r\n",
    "    y_pred = x.mm(w) + b.expand_as(y) #expand_as用到了广播法则，将在第六章详析介绍\r\n",
    "    loss = 0.5 * (y_pred - y) ** 2 # 均方误差\r\n",
    "    loss = loss.mean()\r\n",
    "    \r\n",
    "    # backward：手动计算梯度\r\n",
    "    dloss = 1\r\n",
    "    dy_pred = dloss * (y_pred - y)\r\n",
    "    \r\n",
    "    dw = x.t().mm(dy_pred)\r\n",
    "    db = dy_pred.sum()\r\n",
    "    \r\n",
    "    # 更新参数\r\n",
    "    w.sub_(lr * dw) # inplace修改\r\n",
    "    b.sub_(lr * db)\r\n",
    "    \r\n",
    "    if ii%50 ==0:\r\n",
    "        # 画图\r\n",
    "        display.clear_output(wait=True)\r\n",
    "        x = t.arange(0, 6).float().view(-1, 1)\r\n",
    "        print(x.type(), w.type())\r\n",
    "        y = x.mm(w) + b.expand_as(x)\r\n",
    "        plt.plot(x.cpu().numpy(), y.cpu().numpy()) # predicted\r\n",
    "        \r\n",
    "        x2, y2 = get_fake_data(batch_size=32) \r\n",
    "        plt.scatter(x2.numpy(), y2.numpy()) # true data\r\n",
    "        \r\n",
    "        plt.xlim(0, 5)\r\n",
    "        plt.ylim(0, 13)\r\n",
    "        plt.show()\r\n",
    "        plt.pause(0.5)\r\n",
    "        \r\n",
    "print('w: ', w.item(), 'b: ', b.item())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "torch.FloatTensor torch.FloatTensor\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ],
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAiw0lEQVR4nO3dd3zV5d3/8dcFmWQQIGEFQpgJyDAYXFRERZZacdWtVVu6tFa9UWh73+2v96+A4l61uFqrrbWursQAAuICBRFRMwgjJGEkjCRkj3PdfxAKxEBOzs457+fjwQNy+J7z/fgV3vlyfT/XdRlrLSIi0vV183cBIiLiGQp0EZEgoUAXEQkSCnQRkSChQBcRCRJhvjxZYmKiTU1N9eUpRUTc5rCwr7qB8kMNOKyld0wE/eKjCOtmfHL+DRs27LPWJnV0nE8DPTU1lfXr1/vylCIiLmtxWN7YUMJDy/PpVtXAjWP6cd+sdIYnxfq0DmNMkTPH+TTQRUS6AmstqwvKWZyVR/7eQ2SkJPDkdROZlNrb36WdlAJdROQYX5ZWsjArl4+27mdInx48ff1EZo3tjzG+GV5xhwJdRAQoOVjLQ8sKeGtjKb16hPOrS8Zw/RlDiAjrOr0jCnQRCWmVtU08vbqQFz/agQF+NHU4P5o6nPiocH+X1mkKdBEJSQ3NLfzp4yKeWFlIVX0Tl2cM4p7poxiYEO3v0lymQBeRLuntjaUsyclnV0UdAxOimTcjjTkZyR2+z+Gw/Gvzbpbk5FF8oI5zRiayYNZoxgyM90HV3qVAF5Eu5+2NpSx4czN1TS0AlFbUseDNzQAnDfW12/azMCuXL0oqGT0gnpduHceUUR22d3cZCnQR6XKW5OT/J8yPqGtqYUlOfruBvmXvIRZn5/FuXhkDekbx0FUTmJORTHcfTQzyFQW6iHQ5uyrqnHq9rKqeR1YU8NdPi4mJCOPemWncOnkoUeHdfVGmzynQRaTLGZgQTWk7oX7kgWZNQzO/X7ONZ9dso9nh4OazU7nj/JH0jonwdak+pUAXkS5n3oy048bQAaLDu3P3haN4eW0Rj67Ywr7qBi4aN4B7Z6YxpE+MH6v1HQW6iHQ5R8bJj3S5DOgZxaxxA3hqdSHbymuYlNqLZ286jYyUXn6u1Lc6DHRjzAvAxUCZtXZs62tLgEuARmArcIu1tsKLdYqIHGdORjJzMpLZuPMgi7LyeP6D7QxLimHpjadx4Zh+XWKqvqc5M6f1D8DMNq8tB8Zaa8cDBcACD9clInJSRftr+MmfP+Oypz9i275q/v+cseT8bArTT+ka6654Q4d36NbaNcaY1DavLTvmy7XAlR6uS0SkXQdqGnli5RZeXltEWLdu/PSCkcydMozYSI0ge+IK3Ar89US/aYyZC8wFSElJ8cDpRCQU1Te18OKHO3h6dSE1Dc1cPWkwP5s2in7xUf4uLWC4FejGmF8AzcArJzrGWrsUWAqQmZlp3TmfiIQeh8Py1sZSHlqWz67Kei5I78t9s9IZ1S/O36UFHJcD3RhzM4cfll5grVVQi4jHvb+lnIVZeeTurmJwr2gSYyJYmVdG3p5DTq/dEkpcCnRjzEzgPuBca22tZ0sSkVCXu7uKRdl5rCkoZ1CvaG48cwh/W19MfbMDcH7tllDTYZeLMeYvwMdAmjGmxBhzG/AkEAcsN8Z8box5xst1ikgI2F1Zxz2vbWL24++zqbiCX140mnfvOZeVeWX/CfMjjqzdIkc50+VybTsvP++FWkQkRFXVN/HM6q08/8F2rIXvnzOMH08dTkKPw1P1nV27JdSpz0dE/Kax2cGf1xXx+MpCDtQ0MufUgdwzPY3BvXscd1xHa7fIYQp0EfE5ay3ZX+7hgXfy2LG/lrOG9eHns0czblDPdo8/0dot82ak+arkLkGBLiI+9emOAyzMymXjzgpG9Yvlxe9OYmpa0klnd7Zdu6UzOxSFEgW6iPjE1vJq7s/OY9nXe+kbF8n9V4zjytMGO73JxJG1W+TEFOgi4lXlhxp47N0C/vJJMVFh3bjnwlHcds5QekQofjxNV1REvKK2sZnn3t/O79/bSn2zg+tOT+HOaSNJjI30d2lBS4EuIh7V4rD8bX0xDy8voOxQAzNO6ce9M9MZnhTr79KCngJdRDzCWsuq/DIWZ+dRsLeaiSkJPH39RDJTe/u7tJChQBcRt20uqWRhVi4fb9tPap8e/O76icwcG7rrkvuLAl1EXFZ8oJYHl+Xz98930Tsmgl9fMobrzhhCRJgze+eIpynQRaTTKmobeWpVIX/8qAhj4CfnDecH5w4nPirc36WFNAW6iDitobmFlz4q4slVhVTVN3HFxEHcM30UA3pqCn4gUKCLSIccDss/v9jFkpx8Sg7Wce6oJObPSmf0gPh2j397Y6lmdfqBAl1ETuqjrftYlJXH5tJKxgyI5+XbxvOtkYknPP7tjaXHrbuitct9R4EuIu0q2HuIxdl5rMwrY2DPKB7+zgTmnJpMtw6m6i/JyT9uES04una5At27FOgicpy9VfU8sryA19YXExMZxvxZ6Xz37FSiwrs79X6tXe4/CnQRAaC6oZml723l2fe30+xw8N2zh3LH+SPoFRPRqc/R2uX+o0AXCXFNLQ5e/bSYx1YUsK+6kYvHD2DejDSG9Ilx6fO0drn/KNBFQpS1lmVf7+X+7Dy27avh9NTePHfzaE4dnODW52rtcv9RoIuEoM92HmRRVi6f7jjI8KQYnr0pk2mj+3psqn4gr13ubktlILdkKtBFQsiOfTU8kJNH1uY9JMZG8tvLxnJ15mDCuofGVH13WyoDvSVTgS4SAg7UNPL4u1t4eW0R4d27cecFI5k7ZRgxkaEVAe62VAZ6S2Zo/d8UCTH1TS08/8F2nlm9lZrGZq6elMJd00bSNz7K36X5hbstlYHekqlAFwlCLQ7Lm5+V8PDyAnZX1jNtdF/mz0pnRN84n9cSSGPO7rZUBnpLZocDZ8aYF4wxZcaYL495rbcxZrkxZkvrz728W6aIOOu9gnIuevx95r3+BX3jInl17pk8d/Mkv4X5gjc3U1pRh+XomPPbG0t9XgscbqmMbjNBqjMtle6+39uceRLyB2Bmm9fmA+9aa0cC77Z+LSJ+9NWuSm58fh03v/AJNY3NPHFtBm/9eDJnDuvjt5pONubsD3Mykll0+TiSE6IxQHJCNIsuH+f0vxjcfb+3dTjkYq1dY4xJbfPypcDU1l//EVgN3OfJwkTEOaUVdTy0LJ+3NpbSMzqc/754DDecmUJkmHNT9b0pEMec3W2pDOSWTFfH0PtZa3cDWGt3G2P6erAmEXFCZV0Tv1u9lRc+3A7A3CnD+PG5I+jZI3A2mQj0Medg4/WHosaYucBcgJSUFG+fTiToNTY7eHltEU+s3MLB2iYuz0jm7umjGNSrh8fP5e4DTS0D4FuuBvpeY8yA1rvzAUDZiQ601i4FlgJkZmZaF88nEvKstfx7824eeCefnQdqmTyiDwtmjWZsck+vnM8Tk2i0DIBvuRro/wBuBha3/vx3j1UkIt/wyfYD/DYrl03FFaT3j+MPt0zi3FFJHpuq3x5PTaIJ5DHnYNNhoBtj/sLhB6CJxpgS4FccDvLXjDG3ATuBq7xZpEioKiyrZnF2Hity99IvPpIHrhzPFRMH0b2DTSY8IRAfaMrJOdPlcu0JfusCD9ciIq3KDtXz2IotvPpp8X/GnG+dPJToCN91ruiBZtcTGivyiHQRNQ3NPLqigG/dv4pX1u2kxWGJjQwjOSHap2EOgT+JRr5JU/9FAkBzi4O/bTg8Vb/8UAPHjqjsqap3e0U/V7pV9ECz61Ggi/iRtZaVeWUszs5jS1k1pw3phbWWfdWNxx3nzop+7nSrBMsDzUBaT8abNOQi4idflFRwzdK13PbH9TQ7LM/cMJHXf3gW+9uE+RGuPowMtOn3vhZo68l4k+7QRXys+EAtD+Tk889Nu+gTE8FvLj2Fa09PIbx1kwlPP4wM9W6VQF/D3JMU6CI+UlHbyBMrC3np4x1072a4/bwR/ODcYcRFHT9V39OzK0O9WyWUvqEp0EW8rL6phT9+tIOnVhVyqKGZq04bxN0XptG/Z/ubTHj6YWSoT78PpW9oCnQRL3E4LH/fVMqDOQWUVtQxNS2J+bPSSe8f3+F7Pfkw0tfdKoH2ADKUvqEp0EW84MPCfSzMyuWrXVWcMjCeB64cz+QRiX6rx1fdKoG4iXIotV8q0EU8KG9PFYuz81idX05yQjSPXn0q354wkG5enKofSHfEgfoAMljaLzuiQBfxgD2V9Ty8PJ/XN5QQGxnGz2enc9NZqUSFe3d2Z6DdEYfSA8hApEAXccOh+iZ+/942nvtgGy0Oyy2Th3L7eSPoFRPhk/MH2h1xKD2ADEQKdBEXNLU4+MsnO3lsxRb21zTy7QkDmTcjjcG9Pb/JxMkE2h1xKD2ADEQKdJFOsNaS89Ue7n8nn+37ajhjaG9emD2aCYMT/FJPoN0Rh9IDyECkQBdx0oaiAyzMymND0UFG9I3l+ZszOT+9r1c3mehIIN4Rh8oDyECkQBfpwPZ9Ndyfncc7X+0hKS6SRZeP46rTBhHW3f9LIemOWI6lQBc5gf3VDTz+7hZeWbeTiLBu3DVtFN87ZygxkYH110Z3xHJEYP3JFAkAdY0tvPDhdn63eit1TS1cM2kwd04bSd+49qfqiwQKBbpIqxaH5Y0NJTy0PJ+9VQ1cOKYf981MZ0TfWH+XJuIUBboEJF/OfrTW8l5BOYuz88jbc4gJgxN4/JoMzhjWxyvnE/EWBboEHF/OfvyytJJF2bl8WLiflN49ePK6DC4aN8CvnSsirlKgS8DxxezHkoO1PLSsgLc2lpLQI5z/uXgMN5w5hIgw/3euiLhKgS4Bx5uzHyvrmnh6VSEvfrQDgB+eO5wfTR1Oz+jwk79RpAtQoEvA8cbsx4bmFv70cRFPriqksq6JyzKSuWd6GslaY0SCiAJdAo4nZz86HJZ/bd7Nkpw8ig/Ucc7IRObPSueUgT09WbJIQHAr0I0xdwHfAyywGbjFWlvvicIktBzpaimtqMNw+A/UEQnR4fz626d0evx87bb9LMrKZVNJJen943jp1tOZMirJrfo0G1MCmcuBboxJBn4KjLHW1hljXgOuAf7godokRLTtarFtfr+msblTn7dl7yHufyePFbllDOgZxYNXTeCyjGS6u7jJRKCtOS5yIu4OuYQB0caYJqAHsMv9kiTUtNfVcqymFutUh0tZVT2PrNjCXz/dSUxEGPfOTOPWyUPd3mQi0NYcFzkRlwPdWltqjHkQ2AnUAcustcvaHmeMmQvMBUhJSXH1dBLg3BmScKZ75WTH1DQ0s3TNNp59fxuNzQ5uOiuVO84fQZ/YSKfrd+Xc2oVHAo07Qy69gEuBoUAF8DdjzA3W2pePPc5auxRYCpCZmdn2X9MSBNwdkjhRV0vbY9pqbnHw1/XFPLJ8C/uqG7ho3ADmzUgjNTHGhf+KztfnrTXHNV4vrnJnFsU0YLu1ttxa2wS8CZztmbKkKznZkIQz5s1II/okwyLh3c1xHS7WWpZ/vZcZj67hF299SWqfHrz547N56vqJHg/zE9XnrTXHj3xzLK2ow3L0m+PbG0s9fi4JPu6Moe8EzjTG9ODwkMsFwHqPVCVdirtDEseu6d22y6VXj3B+dcnRDpfPiytYmJXLJ9sPMCwxht/feBrTx/Tz6lR9X645rvF6cYc7Y+jrjDGvA58BzcBGWodWJLR4YkiiozW9i/bXsCQnn399sZvE2Aj+d85Yrpk0mHAfbTLhqzXHNV4v7nCry8Va+yvgVx6qRboob26DdrCmkcdXbuHltUWEdevGT88fwdxzhxMbYJtMtMeVsfBA2yNUupbA/1shAc8bQxL1TS28+OEOnl5dSE1DM9/JHMxdF46iX3zX2GTC1QfFgbhHqHQdCnTxCE8NSTgclrc2lvLQsnx2VdZzfnpf5s9KZ1S/OA9U6TuujoVrj1BxhwJdAsYHW/axMCuXr3dXMTY5nge/M4Gzhyf6uyyXuDMWrj1CxVUKdPG73N1VLMrOY01BOckJ0Tx2zalcMn4g3Vycqh8INBYu/qBAF7e5OhFmd2UdDy0r4I3PSoiLDOMXs0dz09lDiAxzb6p+INBYuPiDsdZ3kzczMzPt+vVqVQ8mbR/+AYR3M8RGhVFR29RuwFfVN/HM6q08/8F2rIWbzx7CT84bQUKPCH/8J3iNZnyKpxhjNlhrMzs6Tnfo4pb2Hv41OSwHa5uA47s7Zo8bwJ/XFfH4ykIO1DRy6akD+a/paQzu3cPndfuCxsLF1xTo4hZnHvLVNbXwm39+zaMrCtixv5azhvXh57NHM26QNpkQ8SQFurjFmYW1AA7UNpIYF8GL353E1LQkr07VFwlV2uJc3NLRwlpHJESHk/XTczgvva/CXMRLdIcubmk7ESYuKozqhmYcxzxrjwrrxq+/fQphbdZd0UNDEc/SHbq4bU5GMsvvnsLdF46ipTXJYyIO37UnJ0Sz+Irx3whqLRMr4nm6Qxe3tDgsr28o5uHlBeytamD6mH7cNyud4UmxJ32flokV8TwFurjEWsvq/HIWZedSsLeajJQEnrxuIpNSezv1fi0TK+J5CnTptM0llSzMyuXjbfsZ0qcHT18/kVlj+3fqYaemxot4ngJdnFZ8oJYHl+Xz98930Tsmgl9fMobrzhhCRFjnH8VoaryI5ynQpUOVtU08tbqQP3y4A2Pgx1OH88Opw4mPCnf5M7VMrIjnKdDlhBqaW3jpoyKeXFVIVX0TV0wcxD3TRzGgp2eGRdyZGq+WR5FvUqDLNzgcln9+sYslOfmUHKxjyqgk5s9MZ8zAeH+XBri+G5BIsFOgy3E+3rqfRdm5fFFSyZgB8fzptnGcMzLJ32UdRy2PIu1ToAsABXsPsTg7j5V5ZQzsGcXD35nAnFOTA3KTCbU8irRPgR7iyqrqeXh5Aa+tLyYmMoz5s9L57tmpRDmxPou/qOVRpH0K9BBV3dDM0jXbeHbNNpodDm4+O5U7zh9J75jA32RCLY8i7VOgh5imFgevflrMYysK2FfdyEXjB3DvjDSG9Ik54XsCraNELY8i7XMr0I0xCcBzwFjAArdaaz/2QF3iYdZaln29l/vfyWNbeQ2np/bm2ZvSyUjpddL3BWpHiXYDEvkmd+/QHwPesdZeaYyJAIJzL7Eu7rOdB1mUlcunOw4yPCmGZ2/KZNpo59YlV0eJSNfhcqAbY+KBKcB3Aay1jUCjZ8oSTyjaX8MD7+Tz7827SYyN5LeXjeXqzMHfWJf8ZNRRItJ1uHOHPgwoB140xkwANgB3Wmtrjj3IGDMXmAuQkpLixunEWQdqGnn83S28sq6IsG7duPOCkXx/yjBiIzv/v1sdJSJdhzuBHgZMBO6w1q4zxjwGzAf++9iDrLVLgaUAmZmZ9hufIh5T39TCCx9u53ertlLT2MzVk1K4a9pI+sZHufyZ7XWUGOC8dPcmGwXag1aRYOBOoJcAJdbada1fv87hQBcfa3FY3tpYykPL8tldWc+00X25b2Y6I/vFuf3ZczKSWV90gFfW7uTId2MLvLGhlMwhvV0K4UB90CrS1bkc6NbaPcaYYmNMmrU2H7gA+NpzpYkz1hSUsyg7j9zdVYwf1JOHv3MqZw3v0+6xrt4Vr8orp+0/rdx5MKoHrSLe4W6Xyx3AK60dLtuAW9wvSZzx9a4qFmXn8v6WfQzuHc3j12Zw8bgBJ5yq785dsacfjOpBq4h3uBXo1trPgUzPlCLO2FVRx0PLCnhzYwnxUeH88qLR3HjWECLDTj5V3527Yk8/GNWDVhHv6PxWM+IXVfVN3P9OHuc9uJp/frGLuecMY8288/jeOcM6DHNw76543ow0otus7eLOVHtPf56IHKap/wGusdnBK+uKePzdLRysbeKyjGTumT6KQb06N4fLnbtiT0+119R9Ee8w1vqukzAzM9OuX7/eZ+fryqy1ZG3ewwM5eRTtr2XyiD4smDWasck9Xfq8tmPocPiueNHl4xSkIgHOGLPBWtvh8Lbu0APQpzsO8Nt/5/J5cQVp/eL4wy2TOHdUklNT9U9Ed8UiwU+BHkC2llezODuP5V/vpV98JA9cMZ4rThtEdw9tMqEFrUSCmwI9AJQfauCxdwv4yyfF/3k4eOvkoURHBO4mEyISeBToflTb2Myza7azdM1WGpodXH9GCj+9YCSJsZH+Lk1EuiAFuh80tzh4fUMJDy8voOxQA7PG9mfejDSGJcX6uzQR6cIU6E7yxGJS1lpW5ZexKCuPLWXVTExJ4Hc3TOS0Ib29VLWIhBIFuhM8sZjUFyUVLMzKZe22AwxNjOF3109k5tj+bnWuiIgcS4HuBHemzRcfqGVJTj7/2LSLPjER/ObSU7j29BTCO7HJhIiIMxToTnBl2nxFbSNPrizkpY+L6NYNbj9vBD84dxhxUeHeKlNEQpwC3QmdmTZf39TCSx/v4MmVhRxqaOaq0wZx14WjGNBTC0+JiHcp0J3Q3q49bReTcjgs/9i0iyU5+ZRW1DE1LYn5s9JJ7x/vj5JFJAQp0J3Q0bT5jwr3sTA7ly9LqzhlYDwPXDmeySMS/VmyiIQgBbqT2ps2n7/nEIuzc1mVX05yQjSPXD2BSyckn3CTCRERb1Kgu2BPZT2PLC/gbxuKiYkMY8GsdG4+O5WocE3VFxH/UaB3QnVDM79/byvPvr+NFofllslDuf28EfSKifB3aSIiCnRnNLU4ePWTnTy6Ygv7axq5ZMJA5k1PI6VP5zaZEBHxJgX6SVhryflqLw+8k8e2fTWcMbQ3L8wezYTBCf4uTUTkGxToJ7Ch6CCLsnJZX3SQEX1jee6mTC4Y3VdT9UUkYCnQ29i+r4YlOXlkbd5DUlwkCy8bx3cyBxGmqfoiEuAU6K32VzfwxMpCXl5bRERYN342bSTfP2cYMZG6RCLSNYR8WtU3tfD8B9t5ZvVWaptauHrSYH42bSR946L8XZqISKeEbKC3OCxvfnZ4k4ndlfVMG92P+bPSGNE3zt+liYi4xO1AN8Z0B9YDpdbai90vyfveKyhnUVYueXsOMWFwAo9efSpnDOvj77JERNziiTv0O4FcIOBXofpqVyWLs/N4f8s+Unr34MnrMrho3AB1rohIUHAr0I0xg4CLgN8Cd3ukIi8orajjoZx83vq8lJ7R4fzPxWO4/swUIsM0VV9Egoe7d+iPAvcCJxx4NsbMBeYCpKSkAJ7Zn9MZlXVNPL26kBc/3AHA3CnD+PHUEfSM1iYTIhJ8XA50Y8zFQJm1doMxZuqJjrPWLgWWAmRmZlpP7M/ZkcZmBy+vLeKJlVuoqGvisoxk7pmeRnI7G1KIiAQLd+7QJwPfNsbMBqKAeGPMy9baG072Jnf25+yItZZ/fbGbJTn57DxQy7dGJDJ/Vjpjk3u69bkiIl2By4FurV0ALABovUP/r47CHFzbn9MZn2w/wG+zctlUXEF6/zj+eOvpTBmZqAeeIhIyfN6H3pn9OZ1RWFbN4uw8VuTupX98FEuuHM/lEwfRXZtMiEiI8UigW2tXA6udOdaZ/TmdUXaonkdXbOGvnxb/5/23Th5KdIQ6V0QkNPn8Dr2j/Tk7UtPQzLPvb+Pp1VtpbHYAEBsZRnJCtMJcREKaX6b+t7c/Z0eaWxy8tr6ER1YUUH6ogWNHVPZU1Xu8U0ZEpKsJ+DVhrbWs+HovMx97n5+/tZmU3j1IjI3AYY8/7kinjIhIqAroxbk2FVewMCuXddsPMCwxhmduOI0Zp/Rj2IKsdo93t1NGRKQrC8hA37m/liXL8vnnpl30iYngfy89hWtOTyG8dZMJT3fKiIgEg4AK9IraRp5YWchLH++gezfDHeePYO6UYcRFHT9V31OdMiIiwSQgAr2+qYU/frSDp1YVUt3QzFWnDeauC0fRv2f7m0y42ykjIhKM/BroDofl75tKeTCngNKKOs5LS2L+rNGk9e94kwlXOmVERIKZ3wL9w8J9LMzK5atdVYxNjmfJleM5e0Siv8oREenyfB7oeXuqWJydx+r8cpITonnsmlO5ZPxAummqvoiIW3wa6CUH65j92PvERobx89np3HRWKlHhmt0pIuIJPg30itpGbp88lNvPH0FCjwhfnlpEJOj5NNBH9YvjlxeP8eUpRURChk+n/keEBfxKAyIiXZYSVkQkSCjQRUSChAJdRCRIKNBFRIKEAl1EJEgo0EVEgoQCXUQkSCjQRUSCRECsh+4rb28s1RrqIhK0QibQ395YetwuR6UVdSx4czOAQl1EgoLLQy7GmMHGmFXGmFxjzFfGmDs9WZinLcnJP27LOoC6phaW5OT7qSIREc9y5w69GbjHWvuZMSYO2GCMWW6t/dpDtXnUrnY2lT7Z6yIiXY3Ld+jW2t3W2s9af30IyAUCduxiYEJ0p14XEelqPNLlYoxJBTKAde383lxjzHpjzPry8nJPnM4l82akEd1mM43o8O7Mm5Hmp4pERDzL7UA3xsQCbwA/s9ZWtf19a+1Sa22mtTYzKSnJ3dO5bE5GMosuH0dyQjQGSE6IZtHl4/RAVESChltdLsaYcA6H+SvW2jc9U5L3zMlIVoCLSNByp8vFAM8Dudbahz1XkoiIuMKdIZfJwI3A+caYz1t/zPZQXSIi0kkuD7lYaz8AjAdrERERN2gtFxGRIKFAFxEJEgp0EZEgoUAXEQkSCnQRkSChQBcRCRIKdBGRIKFAFxEJEgp0EZEgoUAXEQkSCnQRkSChQBcRCRIKdBGRIKFAFxEJEgp0EZEgoUAXEQkSCnQRkSChQBcRCRIKdBGRIKFAFxEJEgp0EZEgoUAXEQkSCnQRkSChQBcRCRIKdBGRIOFWoBtjZhpj8o0xhcaY+Z4qSkREOs/lQDfGdAeeAmYBY4BrjTFjPFWYiIh0jjt36KcDhdbabdbaRuBV4FLPlCUiIp0V5sZ7k4HiY74uAc5oe5AxZi4wt/XLBmPMl26cM5gkAvv8XUSA0LU4StfiKF2Lo9KcOcidQDftvGa/8YK1S4GlAMaY9dbaTDfOGTR0LY7StThK1+IoXYujjDHrnTnOnSGXEmDwMV8PAna58XkiIuIGdwL9U2CkMWaoMSYCuAb4h2fKEhGRznJ5yMVa22yMuR3IAboDL1hrv+rgbUtdPV8Q0rU4StfiKF2Lo3QtjnLqWhhrvzHsLSIiXZBmioqIBAkFuohIkPBJoGuJgKOMMS8YY8pCvR/fGDPYGLPKGJNrjPnKGHOnv2vyF2NMlDHmE2PMptZr8f/8XZO/GWO6G2M2GmP+5e9a/MkYs8MYs9kY87kzrYteH0NvXSKgALiQw62OnwLXWmu/9uqJA5QxZgpQDbxkrR3r73r8xRgzABhgrf3MGBMHbADmhOKfC2OMAWKstdXGmHDgA+BOa+1aP5fmN8aYu4FMIN5ae7G/6/EXY8wOINNa69QEK1/coWuJgGNYa9cAB/xdh79Za3dbaz9r/fUhIJfDs49Djj2suvXL8NYfIdutYIwZBFwEPOfvWroaXwR6e0sEhORfXGmfMSYVyADW+bkUv2kdYvgcKAOWW2tD9loAjwL3Ag4/1xEILLDMGLOhdRmVk/JFoDu1RICEJmNMLPAG8DNrbZW/6/EXa22LtfZUDs+4Pt0YE5LDccaYi4Eya+0Gf9cSICZbaydyeFXbn7QO2Z6QLwJdSwRIu1rHi98AXrHWvunvegKBtbYCWA3M9G8lfjMZ+Hbr2PGrwPnGmJf9W5L/WGt3tf5cBrzF4SHsE/JFoGuJAPmG1geBzwO51tqH/V2PPxljkowxCa2/jgamAXl+LcpPrLULrLWDrLWpHM6KldbaG/xcll8YY2JaGwYwxsQA04GTdsd5PdCttc3AkSUCcoHXnFgiIGgZY/4CfAykGWNKjDG3+bsmP5kM3MjhO7DPW3/M9ndRfjIAWGWM+YLDN0DLrbUh3a4nAPQDPjDGbAI+Af5trX3nZG/Q1H8RkSChmaIiIkFCgS4iEiQU6CIiQUKBLiISJBToIiJBQoEuIhIkFOgiIkHi/wC23jT5Q1yahQAAAABJRU5ErkJggg=="
     },
     "metadata": {
      "needs_background": "light"
     }
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "w:  1.9115010499954224 b:  3.044184446334839\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "可见程序已经基本学出w=2、b=3，并且图中直线和数据已经实现较好的拟合。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3.4 autograd和计算图基础\n",
    "\n",
    "在训练网络时使用Tensor非常方便，但从3.3节线性回归的例子来看，反向传播过程需要手动实现。这对于像线性回归等较为简单的模型，还比较容易，但在实际使用中经常出现非常复杂的网络结构，此时如果手动实现反向传播，不仅费时费力，而且容易出错，难以检查。`torch.autograd`就是为了方便用户使用，而专门开发的一套自动求导引擎，它能够根据输入和前向传播过程自动构建计算图，并执行反向传播。\n",
    "\n",
    "计算图(Computation Graph)是包括PyTorch和TensorFlow在内的许多现代深度学习框架的核心，它为自动求导算法——反向传播(Back Propogation)提供了计算基础，了解计算图在实际写程序过程中会有极大的帮助。本节将涉及一些基础的计算图知识，但并不要求读者事先对此有深入的了解。\n",
    "\n",
    "### 3.4.1 requires_grad\n",
    "\n",
    "PyTorch在autograd模块中实现了计算图的相关功能，autograd中的核心数据结构便是Tensor。当用户定义网络模型时，autograd会记录与网络相关的Tensor的所有操作，从而形成一个前向传播的有向无环图（DAG）。在这个图中，输入网络的Tensor会成为叶子节点，而网络输出的Tensor会成为根节点，autograd便可从根节点开始遍历，并对其中所有`requires_grad=True`的Tensor进行求导操作，这样逐层遍历至叶子节点时，便可通过链式操作计算梯度，从而自动完成了反向传播操作。autograd中核心的反向传播函数如下：\n",
    "\n",
    "`torch.autograd.backward(tensors,grad_tensors=None,retain_graph=None,create_graph=False)`\n",
    "\n",
    "其中：\n",
    "- tensors：用于计算梯度的Tensor，如torch.autograd.backward(y)（等价于y.backward()）；\n",
    "- grad_tensors：形状与Tensor一致，对于`y.backward()`，grad_variables相当于链式法则${dz \\over dx}={dz \\over dy} \\times {dy \\over dx}$中的$\\textbf {dz} \\over \\textbf {dy}$。grad_variables也可以是tensor或序列。\n",
    "- retain_graph：反向传播需要缓存一些中间结果，反向传播之后，这些缓存就被清空，可通过指定这个参数不清空缓存，用来多次反向传播。\n",
    "- create_graph：对反向传播过程再次构建计算图，可通过`backward of backward`实现求高阶导数。\n",
    "\n",
    "下面举几个简单的例子来说明autograd如何使用。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "source": [
    "from __future__ import print_function\r\n",
    "import torch as t\r\n",
    "# 下面两种写法等价\r\n",
    "a = t.randn(3,4, requires_grad=True)\r\n",
    "# a = t.randn(3, 4).requires_grad_()\r\n",
    "a.requires_grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 87
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "source": [
    "# 也可以单独设置requires_grad\r\n",
    "a = t.randn(3, 4)\r\n",
    "a.requires_grad = True"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "source": [
    "b = t.zeros(3,4).requires_grad_()\r\n",
    "c = (a + b).sum()\r\n",
    "c.backward()\r\n",
    "c"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor(-1.6152, grad_fn=<SumBackward0>)"
      ]
     },
     "metadata": {},
     "execution_count": 89
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "source": [
    "a.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]])"
      ]
     },
     "metadata": {},
     "execution_count": 90
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "source": [
    "# 此处虽然没有指定c需要求导，但c依赖于a，而a需要求导，\r\n",
    "# 因此c的requires_grad属性会自动设为True\r\n",
    "a.requires_grad, b.requires_grad, c.requires_grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "metadata": {},
     "execution_count": 91
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Tensor的is_leaf属性\n",
    "\n",
    "对于计算图中的Tensor而言，只有当`is_leaf=True`时，其导数结果才会被保留下来，我们称这样的Tensor为Leaf Tensor，也就是计算图中的叶子节点。设计Leaf Tensor的初衷是为了节省内存/显存，因为通常情况下我们不会直接使用非叶子节点的梯度信息。下面给出Leaf Tensor的判断准则：\n",
    "\n",
    "- 当Tensor的requires_grad为False的时候，它就是Leaf Tensor；\n",
    "- 当Tensor的requires_grad为True，但是是由用户创建的时候，它也是Leaf Tensor。\n",
    "\n",
    "需要注意的是，Leaf Tensor的grad_fn属性为None。下面将举例说明Leaf Tensor的判断准则。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "source": [
    "a.is_leaf, b.is_leaf, c.is_leaf"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(True, True, False)"
      ]
     },
     "metadata": {},
     "execution_count": 92
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "source": [
    "a = t.rand(10, requires_grad=True)\r\n",
    "a.is_leaf"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 93
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "source": [
    "# 接下来几个测试是在有GPU的环境下进行的\r\n",
    "b = t.rand(10, requires_grad=True).cuda(0)\r\n",
    "b.is_leaf # b是由CPU上的Tensor转换为cuda上的Tensor创建的，所以不是Leaf Tensor"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "metadata": {},
     "execution_count": 94
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "source": [
    "c = t.rand(10, requires_grad=True) + 2\r\n",
    "c.is_leaf"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "metadata": {},
     "execution_count": 95
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "source": [
    "d = t.rand(10).cuda(0)\r\n",
    "print(d.requires_grad) # False\r\n",
    "print(d.is_leaf) # 除了创建外没有其他操作（由autograd实现的）"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "False\n",
      "True\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "source": [
    "e = t.rand(10).cuda(0).requires_grad_()\r\n",
    "e.is_leaf # 同样的，在创建e的时候没有额外的操作"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 97
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "接着我们来看看autograd计算的导数和我们手动推导的导数的区别：\n",
    "$$\n",
    "y = x^2\\bullet e^x\n",
    "$$\n",
    "它的导函数是：\n",
    "$$\n",
    "{dy \\over dx} = 2x\\bullet e^x + x^2 \\bullet e^x\n",
    "$$"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "source": [
    "def f(x):\r\n",
    "    '''计算y'''\r\n",
    "    y = x**2 * t.exp(x)\r\n",
    "    return y\r\n",
    "\r\n",
    "def gradf(x):\r\n",
    "    '''手动求导函数'''\r\n",
    "    dx = 2*x*t.exp(x) + x**2*t.exp(x)\r\n",
    "    return dx"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "source": [
    "x = t.randn(3,4, requires_grad = True)\r\n",
    "y = f(x)\r\n",
    "y"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[0.0109, 0.2316, 0.8111, 7.1278],\n",
       "        [0.4126, 0.5035, 0.5146, 0.9632],\n",
       "        [0.5159, 1.0523, 0.0118, 0.3755]], grad_fn=<MulBackward0>)"
      ]
     },
     "metadata": {},
     "execution_count": 99
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "source": [
    "y.backward(t.ones(y.size())) # gradient形状与y一致\r\n",
    "x.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 0.2308, -0.4555,  3.3047, 17.6441],\n",
       "        [-0.3174, -0.1639, -0.1357,  3.7401],\n",
       "        [-0.1319,  3.9884,  0.2404, -0.3601]])"
      ]
     },
     "metadata": {},
     "execution_count": 100
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "source": [
    "# autograd的计算结果与利用公式手动计算的结果一致\r\n",
    "gradf(x) "
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 0.2308, -0.4555,  3.3047, 17.6441],\n",
       "        [-0.3174, -0.1639, -0.1357,  3.7401],\n",
       "        [-0.1319,  3.9884,  0.2404, -0.3601]], grad_fn=<AddBackward0>)"
      ]
     },
     "metadata": {},
     "execution_count": 101
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.4.2 计算图\r\n",
    "\r\n",
    "PyTorch中`autograd`的底层采用了计算图，计算图是一种特殊的有向无环图（DAG），用于记录算子与变量之间的关系。一般用矩形表示算子，椭圆形表示变量。如表达式$ \\textbf {z = wx + b}$可分解为$\\textbf{y = wx}$和$\\textbf{z = y + b}$，其计算图如图3-2所示，图中`MUL`，`ADD`都是算子，$\\textbf{w}$，$\\textbf{x}$，$\\textbf{b}$即变量。\r\n",
    "\r\n",
    "![图3-2:computation graph](imgs/com_graph.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "如上有向无环图中，$\\textbf{X}$和$\\textbf{b}$是叶子节点，这些节点通常由用户自己创建，不依赖于其他变量。$\\textbf{z}$称为根节点，是计算图的最终目标。利用链式法则很容易求得各个叶子节点的梯度。\r\n",
    "$${\\partial z \\over \\partial b} = 1,\\space {\\partial z \\over \\partial y} = 1\\\\\r\n",
    "{\\partial y \\over \\partial w }= x,{\\partial y \\over \\partial x}= w\\\\\r\n",
    "{\\partial z \\over \\partial x}= {\\partial z \\over \\partial y} {\\partial y \\over \\partial x}=1 * w\\\\\r\n",
    "{\\partial z \\over \\partial w}= {\\partial z \\over \\partial y} {\\partial y \\over \\partial w}=1 * x\\\\\r\n",
    "$$\r\n",
    "而有了计算图，上述链式求导即可利用计算图的反向传播自动完成，其过程如图3-3所示。\r\n",
    "\r\n",
    "![图3-3：计算图的反向传播](imgs/com_graph_backward.png)\r\n",
    "\r\n",
    "\r\n",
    "在PyTorch实现中，autograd会随着用户的操作，记录生成当前Tensor的所有操作，并由此建立一个有向无环图。用户每进行一个操作，相应的计算图就会发生改变。更底层的实现中，图中记录了操作`Function`，每一个变量在图中的位置可通过其`grad_fn`属性在图中的位置推测得到。在反向传播过程中，autograd沿着这个图从当前变量（根节点$\\textbf{z}$）溯源，可以利用链式求导法则计算所有叶子节点的梯度。每一个前向传播操作的函数都有与之对应的反向传播函数用来计算输入的各个Tensor的梯度，这些函数的函数名通常以`Backward`结尾。下面结合代码学习autograd的实现细节。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "source": [
    "x = t.ones(1)\r\n",
    "b = t.rand(1, requires_grad = True)\r\n",
    "w = t.rand(1, requires_grad = True)\r\n",
    "y = w * x # 等价于y=w.mul(x)\r\n",
    "z = y + b # 等价于z=y.add(b)\r\n",
    "\r\n",
    "x.requires_grad, b.requires_grad, w.requires_grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(False, True, True)"
      ]
     },
     "metadata": {},
     "execution_count": 102
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "source": [
    "# 虽然未指定y.requires_grad为True，但由于y依赖于需要求导的w\r\n",
    "# 故而y.requires_grad为True\r\n",
    "y.requires_grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 103
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "source": [
    "# grad_fn可以查看这个Tensor的反向传播函数，\r\n",
    "# z是add函数的输出，所以它的反向传播函数是AddBackward\r\n",
    "z.grad_fn "
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<AddBackward0 at 0x7fcd240d3e80>"
      ]
     },
     "metadata": {},
     "execution_count": 104
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "source": [
    "# next_functions保存grad_fn的输入，是一个tuple，tuple的元素也是Function\r\n",
    "# 第一个是y，它是乘法(mul)的输出，所以对应的反向传播函数y.grad_fn是MulBackward\r\n",
    "# 第二个是b，它是叶子节点，由用户创建，grad_fn为None，但是有\r\n",
    "z.grad_fn.next_functions "
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "((<MulBackward0 at 0x7fcd240d3f28>, 0),\n",
       " (<AccumulateGrad at 0x7fcd240d3f60>, 0))"
      ]
     },
     "metadata": {},
     "execution_count": 105
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "source": [
    "# 第一个是w，叶子节点，需要求导，梯度是累加的\r\n",
    "# 第二个是x，叶子节点，不需要求导，所以为None\r\n",
    "y.grad_fn.next_functions"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "((<AccumulateGrad at 0x7fcd240d3da0>, 0), (None, 0))"
      ]
     },
     "metadata": {},
     "execution_count": 106
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "source": [
    "# 叶子节点的grad_fn是None\r\n",
    "w.grad_fn,x.grad_fn"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(None, None)"
      ]
     },
     "metadata": {},
     "execution_count": 107
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "计算w的梯度的时候，需要用到x的数值(${\\partial y\\over \\partial w} = x $)，这些数值在前向过程中会保存成buffer，在计算完梯度之后会自动清空。为了能够多次反向传播需要指定`retain_graph`来保留这些buffer。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "source": [
    "# 使用retain_graph来保存buffer\r\n",
    "z.backward(retain_graph=True)\r\n",
    "w.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([1.])"
      ]
     },
     "metadata": {},
     "execution_count": 108
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "source": [
    "# 多次反向传播，梯度累加，这也就是w中AccumulateGrad标识的含义\n",
    "z.backward()\n",
    "w.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([2.])"
      ]
     },
     "metadata": {},
     "execution_count": 109
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "PyTorch使用的是动态图，它的计算图在每次前向传播时都是从头开始构建，所以它能够使用Python控制语句（如for、if等）根据需求创建计算图。这点在自然语言处理领域中很有用，它意味着你不需要事先构建所有可能用到的图的路径，图在运行时才构建。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "source": [
    "def abs(x):\n",
    "    if x.data[0]>0: return x\n",
    "    else: return -x\n",
    "x = t.ones(1,requires_grad=True)\n",
    "y = abs(x)\n",
    "y.backward()\n",
    "x.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([1.])"
      ]
     },
     "metadata": {},
     "execution_count": 110
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "source": [
    "def f(x):\n",
    "    result = 1\n",
    "    for ii in x:\n",
    "        if ii.item()>0: result=ii*result\n",
    "    return result\n",
    "x = t.arange(-2,4).float().requires_grad_()\n",
    "y = f(x) # y = x[3]*x[4]*x[5]\n",
    "y.backward()\n",
    "x.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 6., 3., 2.])"
      ]
     },
     "metadata": {},
     "execution_count": 111
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "变量的`requires_grad`属性默认为False，如果某一个节点`requires_grad`被设置为True，那么所有依赖它的节点`requires_grad`都是True。这其实很好理解，对于$ \\textbf{x}\\to \\textbf{y} \\to \\textbf{z}$，x.requires_grad = True，当需要计算$\\partial z \\over \\partial x$时，根据链式法则，$\\frac{\\partial z}{\\partial x} = \\frac{\\partial z}{\\partial y} \\frac{\\partial y}{\\partial x}$，自然也需要求$ \\frac{\\partial z}{\\partial y}$，所以`y.requires_grad`会被自动标为True. \n",
    "\n",
    "\n",
    "\n",
    "有些时候我们可能不希望autograd对Tensor求导。认为求导需要缓存许多中间结构，增加额外的内存/显存开销，那么我们可以关闭自动求导。对于不需要反向传播的情景（如inference，即测试推理时），关闭自动求导可实现一定程度的速度提升，并节省约一半显存，因其不需要分配空间计算梯度。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "source": [
    "x = t.ones(1, requires_grad=True)\n",
    "w = t.rand(1, requires_grad=True)\n",
    "y = x * w\n",
    "# y依赖于w，而w.requires_grad = True\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "metadata": {},
     "execution_count": 112
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "source": [
    "with t.no_grad():\n",
    "    x = t.ones(1)\n",
    "    w = t.rand(1, requires_grad = True)\n",
    "    y = x * w\n",
    "# y依赖于w和x，虽然w.requires_grad = True，但是y的requires_grad依旧为False\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(False, True, False)"
      ]
     },
     "metadata": {},
     "execution_count": 113
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "source": [
    "t.set_grad_enabled(False) # 更改了默认设置\n",
    "x = t.ones(1)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "y = x * w\n",
    "# y依赖于w和x，虽然w.requires_grad = True，但是y的requires_grad依旧为False\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(False, True, False)"
      ]
     },
     "metadata": {},
     "execution_count": 114
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "source": [
    "# 恢复默认配置\n",
    "t.set_grad_enabled(True)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<torch.autograd.grad_mode.set_grad_enabled at 0x7fcd240d62e8>"
      ]
     },
     "metadata": {},
     "execution_count": 115
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "如果我们想要修改Tensor的数值，但是又不希望被autograd记录，那么我们可以对tensor.data进行操作。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "source": [
    "a = t.ones(3,4,requires_grad=True)\n",
    "b = t.ones(3,4,requires_grad=True)\n",
    "c = a * b\n",
    "\n",
    "a.data # 还是一个Tensor"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]])"
      ]
     },
     "metadata": {},
     "execution_count": 116
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "source": [
    "a.data.requires_grad # 已经独立于计算图之外了"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "metadata": {},
     "execution_count": 117
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "在反向传播过程中非叶子节点的导数计算完之后即被清空。若想查看这些变量的梯度，有两种方法：\n",
    "- 使用autograd.grad函数；\n",
    "- 使用hook。\n",
    "\n",
    "`autograd.grad`和`hook`方法都是很强大的工具，更详细的用法参考官方api文档，这里仅举例说明基础的使用方法。笔者推荐使用`hook`方法，但是在实际使用中应尽量避免修改grad的值。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "source": [
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "# y依赖于w，而w.requires_grad = True\n",
    "z = y.sum()\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "metadata": {},
     "execution_count": 118
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "source": [
    "# 非叶子节点grad计算完之后自动清空，y.grad是None\n",
    "z.backward()\n",
    "(x.grad, w.grad, y.grad)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(tensor([0.8637, 0.1238, 0.0123]), tensor([1., 1., 1.]), None)"
      ]
     },
     "metadata": {},
     "execution_count": 119
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "source": [
    "# 第一种方法：使用grad获取中间变量的梯度\n",
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "z = y.sum()\n",
    "# z对y的梯度，隐式调用backward()\n",
    "t.autograd.grad(z, y)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(tensor([1., 1., 1.]),)"
      ]
     },
     "metadata": {},
     "execution_count": 120
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "source": [
    "# 第二种方法：使用hook\n",
    "# hook是一个函数，输入是梯度，不应该有返回值\n",
    "def variable_hook(grad):\n",
    "    print('y的梯度：',grad)\n",
    "\n",
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "# 注册hook\n",
    "hook_handle = y.register_hook(variable_hook)\n",
    "z = y.sum()\n",
    "z.backward()\n",
    "\n",
    "# 除非你每次都要用hook，否则用完之后记得移除hook\n",
    "hook_handle.remove()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "y的梯度： tensor([1., 1., 1.])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "在PyTorch中计算图的特点可总结如下：\n",
    "\n",
    "- autograd根据用户对Tesnor的操作构建计算图。对Tensor的操作抽象为`Function`；\n",
    "- 由用户创建的节点称为叶子节点，叶子节点的`grad_fn`为None。叶子节点中需要求导的Tesnor，具有`AccumulateGrad`标识，因其梯度是累加的；\n",
    "- Tensor默认是不需要求导的，即`requires_grad`属性默认为False，如果某一个节点requires_grad被设置为True，那么所有依赖它的节点`requires_grad`都为True；\n",
    "- 多次反向传播时，梯度是累加的。反向传播的中间缓存会被清空，为进行多次反向传播需指定`retain_graph=True`来保存这些缓存；\n",
    "- 非叶子节点的梯度计算完之后即被清空，可以使用`autograd.grad`或`hook`技术获取非叶子节点梯度的值；\n",
    "- Tensor的grad与data形状一致，应避免直接修改tensor.data，因为对data的直接操作无法利用autograd进行反向传播；\n",
    "- PyTorch采用动态图设计，可以很方便地查看中间层的输出，动态地设计计算图结构。；\n",
    "\n",
    "如果读者还并不太明白以上内容，也不用担心，因为在大多数情况下这并不影响读者对PyTorch的一般使用，但是掌握这些内容有助于更好地理解PyTorch，并有效地避开很多陷阱。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.4.3 扩展autograd\n",
    "\n",
    "\n",
    "目前，绝大多数函数都可以使用autograd进行反向求导，但如果需要自己写一个复杂的函数，但并不支持自动反向求导怎么办? 答案是写一个`Function`，实现它的前向传播和反向传播代码，`Function`对应于计算图中的矩形，它接收参数，计算并返回结果。下面给出一个例子。\n",
    "\n",
    "```python\n",
    "\n",
    "class Mul(Function):                                                     \n",
    "    @staticmethod\n",
    "    def forward(ctx, weight, x, bias, x_requires_grad = True):\n",
    "        ctx.x_requires_grad = x_requires_grad\n",
    "        ctx.save_for_backward(weight, x)\n",
    "        output = weight * x + bias\n",
    "        return output\n",
    "        \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):\n",
    "        weight, x = ctx.saved_tensors\n",
    "        grad_weight = grad_output * x\n",
    "        if ctx.x_requires_grad:\n",
    "            grad_x = grad_output * weight\n",
    "        else:\n",
    "            grad_x = None\n",
    "        grad_bias = grad_output * 1\n",
    "        return grad_weight, grad_x, grad_bias, None\n",
    "```\n",
    "\n",
    "分析如下：\n",
    "\n",
    "- 自定义的Function需要继承autograd.Function，没有构造函数`__init__`，forward和backward函数都是静态方法；\n",
    "- backward函数的输出和forward函数的输入一一对应，backward函数的输入和forward函数的输出一一对应；\n",
    "- 如果某一个输入不需要求导，直接返回None，如forward中的输入参数x_requires_grad显然无法对它求导，直接返回None即可；\n",
    "- 反向传播可能需要利用前向传播的某些中间结果，需要进行保存，否则前向传播结束后这些对象即被释放。\n",
    "\n",
    "读者可通过Function.apply(tensor)调用已经实现的Function。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "source": [
    "from torch.autograd import Function\n",
    "class MultiplyAdd(Function):\n",
    "                                                            \n",
    "    @staticmethod\n",
    "    def forward(ctx, w, x, b):                              \n",
    "        ctx.save_for_backward(w,x)\n",
    "        output = w * x + b\n",
    "        return output\n",
    "        \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):                         \n",
    "        w,x = ctx.saved_tensors\n",
    "        grad_w = grad_output * x\n",
    "        grad_x = grad_output * w\n",
    "        grad_b = grad_output * 1\n",
    "        return grad_w, grad_x, grad_b  "
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "source": [
    "x = t.ones(1)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "# 开始前向传播\n",
    "z = MultiplyAdd.apply(w, x, b)\n",
    "# 开始反向传播\n",
    "z.backward()\n",
    "\n",
    "# x不需要求导，中间过程还是会计算它的导数，但随后被清空\n",
    "x.grad, w.grad, b.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(None, tensor([1.]), tensor([1.]))"
      ]
     },
     "metadata": {},
     "execution_count": 123
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "source": [
    "x = t.ones(1)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "#print('开始前向传播')\n",
    "z = MultiplyAdd.apply(w,x,b)\n",
    "#print('开始反向传播')\n",
    "\n",
    "# 调用MultiplyAdd.backward\n",
    "# 输出grad_w, grad_x, grad_b\n",
    "print(z.grad_fn)\n",
    "z.grad_fn.apply(t.ones(1))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<torch.autograd.function.MultiplyAddBackward object at 0x7fcd1fe14668>\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(tensor([1.]), tensor([0.3763], grad_fn=<MulBackward0>), tensor([1.]))"
      ]
     },
     "metadata": {},
     "execution_count": 124
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.4.4 小试牛刀: 用autograd实现线性回归\n",
    "\n",
    "在3.3节中我们讲解了如何利用Tensor来实现线性回归，本节将讲解如何利用autograd实现线性回归，读者可以从中体会autograd的便捷之处。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "source": [
    "import torch as t\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display \n",
    "import numpy as np"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "source": [
    "# 设置随机数种子，以保证结果可复现\n",
    "t.manual_seed(1000) \n",
    "\n",
    "def get_fake_data(batch_size=8):\n",
    "    ''' 产生随机数据：y = x * 2 + 3，加上了一些噪声'''\n",
    "    x = t.rand(batch_size,1) * 5\n",
    "    y = x * 2 + 3 + t.randn(batch_size, 1)\n",
    "    return x, y"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "source": [
    "# 来看看产生x-y分布是什么样的\n",
    "x, y = get_fake_data()\n",
    "plt.scatter(x.squeeze().numpy(), y.squeeze().numpy())"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7fcd1c1ed3c8>"
      ]
     },
     "metadata": {},
     "execution_count": 127
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ],
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPrUlEQVR4nO3dX2xk5X3G8eep1yheQmvCDpQ1bDeRkNWGFHZrIf60KCmlBkLCBuUC1FRpFNVqlbbQC1dsL4J6lVbuRdqLtlpRWqImRCnxbiuUYFBSmqoUIi+G7NKNC6FAsGnWlDiEMBK77q8XMyazE9vz5xzPOe/M9yONPH7P8Zyf3n39+Jz3/FlHhAAA6fmpogsAAHSHAAeARBHgAJAoAhwAEkWAA0CidvRyY7t27Yq9e/f2cpMAkLyjR4++GhGV5vaeBvjevXs1Pz/fy00CQPJsv7hRe8spFNv32j5p+3hD27tsP2L72frXc/MsFgDQWjtz4H8v6YamtrskfS0iLpH0tfr3AIAeahngEfENSa81Nd8i6b76+/skHci3LABAK91ehXJBRLwiSfWv52+2ou0p2/O251dWVrrcHACg2bZfRhgRhyJiIiImKpWfOIkKAOhSt1ehfM/2hRHxiu0LJZ3MsygA6BdHFpY0M7eo5dWqdo+OaHpyXAf2jeXy2d3ugf+zpI/X339c0j/lUg0A9JEjC0s6OHtMS6tVhaSl1aoOzh7TkYWlXD6/ncsI75f0H5LGbb9s+5OS/lTS9baflXR9/XsAQIOZuUVVT62d0VY9taaZucVcPr/lFEpE3L7JoutyqQAA+tTyarWj9k7xLBQA2Ca7R0c6au8UAQ4A22R6clwjw0NntI0MD2l6cjyXz+/ps1AAYJCsX22yXVehEOAAsI0O7BvLLbCbMYUCAIkiwAEgUQQ4ACSKAAeARBHgAJAoAhwAEkWAA0CiCHAASBQBDgCJIsABIFEEOAAkigAHgEQR4ACQKAIcABJFgANAoghwAEgUAQ4AiSLAASBRBDgAJCpTgNu+w/Zx28/YvjOnmgAAbeg6wG1fKum3JV0h6TJJN9u+JK/CAABby7IH/vOSHo+INyPitKR/lfSRfMoCALSSJcCPS7rW9nm2d0q6SdLF+ZQFAGhlR7c/GBEnbP+ZpEckvSHpaUmnm9ezPSVpSpL27NnT7eYAAE0yncSMiL+NiP0Rca2k1yQ9u8E6hyJiIiImKpVKls0BABp0vQcuSbbPj4iTtvdIulXSVfmUBQBoJVOAS/qy7fMknZL0qYj4fg41AQDakCnAI+JX8ioEANAZ7sQEgEQR4ACQKAIcABJFgANAoghwAEgUAQ4AiSLAASBRBDgAJIoAB4BEZb2VHkjWkYUlzcwtanm1qt2jI5qeHNeBfWNFlwW0jQDHQDqysKSDs8dUPbUmSVparerg7DFJIsSRDKZQMJBm5hbfDu911VNrmplbLKgioHMEOAbS8mq1o3agjAhwDKTdoyMdtQNlRIBjIE1PjmtkeOiMtpHhIU1PjhdUEdA5TmJiIK2fqOQqFKSMAMfAOrBvjMBG0phCAYBEEeAAkCgCHAASRYADQKIIcABIFAEOAIkiwAEgUZkC3PYf2n7G9nHb99t+R16FAQC21nWA2x6T9AeSJiLiUklDkm7LqzAAwNayTqHskDRie4eknZKWs5cEAGhH1wEeEUuS/lzSS5JekfSDiHi4eT3bU7bnbc+vrKx0XykA4AxZplDOlXSLpHdL2i3pbNsfa14vIg5FxERETFQqle4rBQCcIcsUyq9J+u+IWImIU5JmJV2dT1kAgFayBPhLkq60vdO2JV0n6UQ+ZQEAWskyB/6EpAckPSnpWP2zDuVUFwCghUzPA4+IuyXdnVMtAIAOcCcmACSKAAeARBHgAJAoAhwAEkWAA0CiCHAASBQBDgCJIsABIFGZbuRBfziysKSZuUUtr1a1e3RE05PjOrBvrOiyALRAgA+4IwtLOjh7TNVTa5KkpdWqDs4ekyRCHCg5AnzAzcwtvh3e66qn1jQzt0iAA5soy1ErAT7gllerHbUDg65MR62cxBxwu0dHOmoHBt1WR629RoAPuOnJcY0MD53RNjI8pOnJ8YIqAsqtTEetBPiAO7BvTJ+59X0aGx2RJY2Njugzt76P+W9gE2U6amUOHDqwb4zABto0PTl+xhy4VNxRKwEOAB1Y39nhKhQASFBZjlqZAweARLEH3kNlufgfQH8gwHukTBf/A+gPTKH0SJku/gfQHwjwHinTxf8A+kPXAW573PZTDa/Xbd+ZY219pUwX/wPoD10HeEQsRsTlEXG5pF+S9Kakw3kV1m+4ZR1A3vI6iXmdpO9ExIs5fV7fKdPF/wD6Q14Bfpuk+zdaYHtK0pQk7dmzJ6fNpaksF/8D6A+ZT2LaPkvShyX940bLI+JQRExExESlUsm6OQBAXR5Xodwo6cmI+F4OnwUAaFMeAX67Npk+AQBsn0wBbnunpOslzeZTDgCgXZlOYkbEm5LOy6kWAEAHuBMTABJFgANAoghwAEgUj5PFQOMZ7UgZAY6BxTPakTqmUDCweEY7UkeAY2DxjHakjgDHwOIZ7UgdAY6BxTPakTpOYmJg8Yx2pI4Ax0DjGe1IGVMoAJCo0u+Bc6MFAGys1AHOjRYAsLlST6FwowUAbK7UAc6NFgCwuVIHODdaAMDmSh3g3GgBAJsr9UlMbrQAgM2VOsAlbrQAgM2UegoFALA5AhwAEkWAA0CiMgW47VHbD9j+tu0Ttq/KqzAAwNaynsT8C0kPRcRHbZ8laWcONQEA2tB1gNv+aUnXSvotSYqItyS9lU9ZAIBWskyhvEfSiqS/s71g+x7bZzevZHvK9rzt+ZWVlQybAwA0yhLgOyTtl/TXEbFP0o8k3dW8UkQcioiJiJioVCoZNgcAaJQlwF+W9HJEPFH//gHVAh0A0ANdB3hE/I+k79pefzDJdZL+M5eqAAAtZb0K5fclfb5+Bcrzkj6RvSQAQDsyBXhEPCVpIp9SAACd4E5MAEgUAQ4AiSLAASBRBDgAJIoAB4BEEeAAkCgCHAASRYADQKIIcABIFAEOAIkiwAEgUQQ4ACSKAAeARBHgAJAoAhwAEkWAA0CiCHAASBQBDgCJIsABIFEEOAAkigAHgEQR4ACQKAIcABJFgANAonZk+WHbL0j6oaQ1SacjYiKPogAArWUK8LoPRMSrOXwOAKADTKEAQKKyBnhIetj2UdtTG61ge8r2vO35lZWVjJsDAKzLGuDXRMR+STdK+pTta5tXiIhDETEREROVSiXj5gAA6zIFeEQs17+elHRY0hV5FAUAaK3rALd9tu1z1t9L+nVJx/MqDACwtSxXoVwg6bDt9c/5QkQ8lEtVAICWug7wiHhe0mU51gIA6ACXEQJAoghwAEgUAQ4AiSLAASBRBDgAJIoAB4BEEeAAkCgCHAASRYADQKIIcABIFAEOAIkiwAEgUQQ4ACSKAAeARBHgAJAoAhwAEkWAA0CiCHAASBQBDgCJIsABIFEEOAAkigAHgEQR4ACQqMwBbnvI9oLtB/MoCADQnjz2wO+QdCKHzwEAdCBTgNu+SNIHJd2TTzkAgHZl3QP/rKQ/kvR/m61ge8r2vO35lZWVjJsDAKzrOsBt3yzpZEQc3Wq9iDgUERMRMVGpVLrdHACgSZY98Gskfdj2C5K+KOlXbf9DLlUBAFrqOsAj4mBEXBQReyXdJunrEfGx3CoDAGyJ68ABIFE78viQiHhU0qN5fBYAoD25BHiZHFlY0szcopZXq9o9OqLpyXEd2DdWdFkAkLu+CvAjC0s6OHtM1VNrkqSl1aoOzh6TJEIcQN/pqznwmbnFt8N7XfXUmmbmFguqCAC2T18F+PJqtaN2AEhZXwX47tGRjtoBIGV9FeDTk+MaGR46o21keEjTk+MFVQQA26evTmKun6jkKhQAg6CvAlyqhTiBDWAQ9NUUCgAMEgIcABJFgANAoghwAEgUAQ4AiXJE9G5j9oqkF1ustkvSqz0oJwtqzE8KdVJjflKos4w1/lxE/MR/adbTAG+H7fmImCi6jq1QY35SqJMa85NCnSnUuI4pFABIFAEOAIkqY4AfKrqANlBjflKokxrzk0KdKdQoqYRz4ACA9pRxDxwA0AYCHAAS1ZMAt32v7ZO2j2+y/Ddsf6v+esz2ZQ3LXrB9zPZTtucLrvP9tn9Qr+Up259uWHaD7UXbz9m+q8AapxvqO257zfa76st60pe2L7b9L7ZP2H7G9h0brGPbf1nvr2/Z3t+wbNv7ss0aCx+XbdZZ6Lhss8ZCx6Xtd9j+pu2n6zX+yQbrFDomuxIR2/6SdK2k/ZKOb7L8aknn1t/fKOmJhmUvSNpVkjrfL+nBDdqHJH1H0nsknSXpaUm/UESNTet+SNLXe92Xki6UtL/+/hxJ/9XcH5JukvRVSZZ05fq/ea/6ss0aCx+XbdZZ6Lhsp8aix2V9nL2z/n5Y0hOSrizTmOzm1ZM98Ij4hqTXtlj+WER8v/7t45Iu6kVdG9SxZZ1buELScxHxfES8JemLkm7Jtbi6Dmu8XdL921HHViLilYh4sv7+h5JOSGp+SPstkj4XNY9LGrV9oXrUl+3UWIZx2WZfbqY0fdmk5+OyPs7eqH87XH81X8FR6JjsRhnnwD+p2l/BdSHpYdtHbU8VVFOjq+qHYV+1/d5625ik7zas87La/yXbFrZ3SrpB0pcbmnvel7b3Stqn2h5Po836rOd9uUWNjQofly3qLMW4bNWXRY5L20O2n5J0UtIjEVHaMdmuUv2PPLY/oNovyi83NF8TEcu2z5f0iO1v1/dCi/Ckas8keMP2TZKOSLpEtUOuZkVfn/khSf8eEY176z3tS9vvVO0X9c6IeL158QY/Elu0b4sWNa6vU/i4bFFnKcZlO32pAsdlRKxJutz2qKTDti+NiMZzSaUYk50ozR647V+UdI+kWyLif9fbI2K5/vWkpMOqHc4UIiJeXz8Mi4ivSBq2vUu1v8gXN6x6kaTlAkpsdJuaDlN72Ze2h1X7Zf58RMxusMpmfdazvmyjxlKMy1Z1lmFcttOXdYWOy/p2ViU9qtqRQKPCx2THejXZLmmvNj85uEfSc5Kubmo/W9I5De8fk3RDgXX+rH5889MVkl5S7a/zDknPS3q3fnyS471F1Fhf/jOqzZOfXURf1vvkc5I+u8U6H9SZJ4y+WW/vSV+2WWPh47LNOgsdl+3UWPS4lFSRNFp/PyLp3yTdXKYx2c2rJ1Motu9X7Uz5LtsvS7pbtZMIioi/kfRpSedJ+ivbknQ6ak8Du0C1Q531TvxCRDxUYJ0flfS7tk9Lqkq6LWr/wqdt/56kOdXOWN8bEc8UVKMkfUTSwxHxo4Yf7WVfXiPpNyUdq885StIfqxaI63V+RbWz/s9JelPSJ+rLetWX7dRYhnHZTp1Fj8t2apSKHZcXSrrP9pBqMw9fiogHbf9OQ41Fj8mOcSs9ACSqNHPgAIDOEOAAkCgCHAASRYADQKIIcABIFAEOAIkiwAEgUf8PdoBTdmOFwaUAAAAASUVORK5CYII="
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "source": [
    "# 随机初始化参数\n",
    "w = t.rand(1,1, requires_grad=True)\n",
    "b = t.zeros(1,1, requires_grad=True)\n",
    "losses = np.zeros(500)\n",
    "\n",
    "lr =0.005 # 学习率\n",
    "\n",
    "for ii in range(500):\n",
    "    x, y = get_fake_data(batch_size=32)\n",
    "    \n",
    "    # forward：计算loss\n",
    "    y_pred = x.mm(w) + b.expand_as(y)\n",
    "    loss = 0.5 * (y_pred - y) ** 2\n",
    "    loss = loss.sum()\n",
    "    losses[ii] = loss.item()\n",
    "    \n",
    "    # backward：自动计算梯度\n",
    "    loss.backward()\n",
    "    \n",
    "    # 更新参数\n",
    "    w.data.sub_(lr * w.grad.data)\n",
    "    b.data.sub_(lr * b.grad.data)\n",
    "    \n",
    "    # 梯度清零\n",
    "    w.grad.data.zero_()\n",
    "    b.grad.data.zero_()\n",
    "    \n",
    "    if ii%50 ==0:\n",
    "        # 画图\n",
    "        display.clear_output(wait=True)\n",
    "        x = t.arange(0, 6).float().view(-1, 1)\n",
    "        y = x.mm(w.data) + b.data.expand_as(x)\n",
    "        plt.plot(x.numpy(), y.numpy()) # predicted\n",
    "        \n",
    "        x2, y2 = get_fake_data(batch_size=20) \n",
    "        plt.scatter(x2.numpy(), y2.numpy()) # true data\n",
    "        \n",
    "        plt.xlim(0,5)\n",
    "        plt.ylim(0,13)   \n",
    "        plt.show()\n",
    "        plt.pause(0.5)\n",
    "        \n",
    "print(w.item(), b.item())"
   ],
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ],
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcJ0lEQVR4nO3deXSV9b3v8feXOczKpAQiAhJAQMHYqrQOOICztVZrq7Xalp4OVmulwupdt/ecdc8FxalOVep8tK22Uk9PlUkRUVRqEJVqEgjIFJAwGMKQkOl7/9gBYkhgJ3t49n7257UWy2RnJ883W/isZz379/w+5u6IiEj6axP0ACIiEh8KdBGRkFCgi4iEhAJdRCQkFOgiIiHRLpkH6927tw8aNCiZhxQRiTsHduyporS8kpo6p0dWe/p170THdrGfI5ftraakrIK6BisQa8q2eG1F+RF/eFIDfdCgQeTn5yfzkCIicePuvLric2bOK2T79r1ccvzRTLtoBCcP7Bm3Y4yfsZCasoovPbb5mVstmu9NaqCLiKSr99ZsZ/qcQj7aUMawfl158vt5nJPbF7OosjZqmxqFeUso0EVEDqPo813cObeQhYWlHNujE3ddNYZvjhtA2zbxDfL9+vfMoqSVoa5AFxFpwuadFdw7fyUvfbCRLh3bccek4dw4fhCd2rdNyPFeXl7CzHlFlJRVYESu0x/gXhfNz1Cgi4g0sLOimt8vWs1TSz7DHW4afzw/O2coR3XpkLBjvry8hGmzV1BRXQtEwnx/qGf3zGJD+dZ10fwcBbqICLCvppb/encdD71RzM6Kaq44OZvbzh/GwKM7J/zYM+cVHQjz/faH+ZKpE7Bp5Tui+TkKdBHJaHV1zn9/VMLd81ZSUlbBmcP6cMekXE7s3yNpMzT3RmhL3yBVoItIxlq8cisz5hTy6eZyRmV3566rxjB+aO+kz9HcG6H9e2a16Oco0EUk4/yrZCcz5hTydvE2Bh6dxe++fTKXjulPmwStXDmSKRNzv3QNHSCrfVumTMxt0c9RoItIxtiwYy8z5xXx9482cVTn9vzvS0by3dNy6NguMStXonXF2Gwgci19U1kF/XtmMWVi7oHHo6VAF5HQ27GnigcXruK599bRto3xs3OG8OOzhtC9U/ugRzvgirHZLQ7wxhToIhJaFVW1PLnkMx5dtJo9VTVcnTeQW88bxjE9OgU9WkIo0EUkdGpq6/jrso3c99pKtpTv47wR/bhjUi4n9OsW9GgJdcRAN7MngUuAUncfVf/YTOBSoApYDdzo7mUJnFNE5IjcndcKSrlrbiGrSnczNqcnD31nHKcOOjro0ZIimr0enwYmNXpsATDK3ccAK4FpcZ5LRKRFPlj/BVc/9i4/ejaf2jrn0evGMfsnZ2RMmEMUZ+juvtjMBjV6bH6DT98DrorzXCIiUVm9dTcz5xYx95PP6d21I//3ilFcc+pA2rfNvP6eeFxDvwl4obkvmtlkYDJATk5OHA4nIgKluyr53Wur+PP7G+jUrg23nT+MH3zteLp0zNy3BmP6zc3sN0AN8Hxzz3H3WcAsgLy8PG/ueSIi0di9r4ZZi9fw+FtrqKqp47qv5nDzuSfQu2vHoEcLXKsD3cxuIPJm6bnurqAWkYSqrq3jT/9czwOvr2Lb7iouHnMsUy7IZVDvLkGPljJaFehmNgm4AzjL3ffGdyQRkYMa1r6t3b6X0wYfzRM3jOCkONa+hUU0yxb/BJwN9DazjcBviaxq6QgsqK9fes/d/y2Bc4pIBmpY+5bbrxtPff9Uzs7tE/fat7CIZpXLtU08/EQCZhERAQ6tfZt51RiuTGDtW1hk7tvBIpJyNpVVcO+CSO1b147tmHrhcL5/RuJq38JGgS4igdtZUc0ji4p5esla3OGHX4vUvvXsnLjatzBSoItIYCqrD9a+lVdW842Ts7ntgmEMOCrxtW9hpEAXkaSrq3Ne/rCEe+YfrH2bOmk4I/t3D3q0tKZAF5GkcXcWr9rGjDmFFARc+xZGCnQRSYoVG3cyY24BS4q3p0TtWxgp0EUkoVK19i2MFOgikhCNa99+fs5QJp81OKVq38JGgS4icdW49u2aUyO1b/26h7P2LZUo0EUkLvbXvt27YCWlu/Zx/shI7dvQvsHVvr28vISZ84rYVFZB/55ZTJmYG3MRcypToItITPbXvt05t5Di0t2My+nJw98Nvvbt5eUlTJu9gorqWgBKyiqYNnsFQGhDXYEuIq22bN0XzJhTwPtrv2Bwny48et0pTDyxX0psnjVzXtGBMN+vorqWmfOKFOgiIvut3rqbu+YWMu+TLfTp1pH//MYorskbSLsUqn3bVFbRosfDQIEuIlErLa/k/tdX8UKD2rcffv14OndIvSjp3zOLkibCu3/PrACmSY7U+78gIiln974aZr25mj+89RnVtelR+zZlYu6XrqEDZLVvy5SJuQFOlVgKdBFpVlXNwdq37XvSq/Zt/3VyrXIRkYzm7ryyYjMz5xWxrr727ckL06/27Yqx2aEO8MYU6CLyJe+u3s6MOQV8tHGnat/SjAJdRAAo/LycO+cU8kbRVtW+pSkFukiGa1j71k21b2lNgS6SoXbujdS+PfXOWgB+9PXB/PTsIap9S2MKdJEMU1ldy7PvruXhN1ZHat/GZnPb+ap9CwMFukiGqK1zXl5ewr0LIrVvZw3rwx2qfQsVBbpIyLk7b67cyow5hRR+vovR2T2YedUYzlDtW+gcMdDN7EngEqDU3UfVP3Y08AIwCFgLXO3uXyRuTBFpjRUbdzJ9TgHvrI7Uvj1w7VguGX1sWtW+ZdoWuLGI5gz9aeAh4NkGj00FXnf3GWY2tf7zO+I/noi0xvrte5k5v4j/+WgTR3fpwG8vHcl3v3ocHdoFs3lWa0M5E7fAjcURA93dF5vZoEYPXw6cXf/xM8AiFOgigdu+ex8PLizm+aUHa99+fNZgugVY+xZLKGfiFrixaO019H7uvhnA3TebWd84ziQiLbS3qoYn3/6MR99cw94Uq32LJZQzcQvcWCT8TVEzmwxMBsjJyUn04UQySk1tHX9ZtpH7DlP7FvQ16FhCORO3wI1Fay+obTGzYwHq/1va3BPdfZa757l7Xp8+fVp5OBFpyN2Z98nnTLx/MdNmr2DAUVn89d9O5w/fyzskzKfNXkFJWQXOwcsdLy8vSdqszYVvNKE8ZWIuWY3uWA37FrixaG2g/x24of7jG4D/js84InIky9bt4FuPvsuP/2sZDjx63Sm89JMzyGuiw/NwlzuSJZZQvmJsNtOvHE12zywMyO6ZxfQrR+v6eTOiWbb4JyJvgPY2s43Ab4EZwItm9gNgPfCtRA4pIlBcupuZ81pW+5YK16Bj3Zc807bAjUU0q1yubeZL58Z5FhFpQml5Jfe9tooX81te+5Yq16AVysmhO0VFUtSuympmLV7D4/W1b9efdhw/nzC0RbVvmVjDlskU6CIppqqmjj8uXceDC4vZvqeKS8Ycy5SJuRzXq+W1b5lYw5bJFOgiKcLd+cfHm7l7fqT27fTBvZh64fCYa990uSNzKNBFUsA7q7cxY04hH2/cyfBjuvHUjady9jDVvknLKNBFAlSwuZw75xaySLVvEgcKdJEAlJRVcO/8lcxeHql9m3bhcG5Q7ZvESIEukkSqfZNEUqCLJIFq3yQZFOgiCbS/9u2e+UVs2lmp2jdJKAW6SAK4O4tWbuXOBrVvd3/rJNW+SUIp0EXi7OONZUx/tZB312wn5+jOaVn7JulJgS4SJ+u27+Hu+SsP1L79n0tH8p0Aa99STdD7smcCBbpIjBrXvt08YSiTzwy29i3VqBs0ORToIq20t6qGJ976jMcWr6Giupar8wZy63knpETtW6pRN2hyKNBFWqimto4X8zdy/2uR2rcLRvbj15OGM7Rv16BHS1mpsC97JlCgi0TJ3Zn/6RbumlvI6q17OOW4o3jku+OabAqSL0uVfdnDToEuEoVl63Yw/dVC8td9weA+XXjs+lO4YGQ/bZ4VJe3LnhwKdJHDKC7dzV1zC5n/aaT27f99YzRX5w04bO2bHEr7sieHAl2kCQ1r37Lat+VX5w/jB1HWvknTtC974ulvp2SsptZFnzui74Hat5q6SO3bzROG0qsFtW8iQVGgS0Zqal30lL98RIf2bdizrzam2jeRoCjQJSM1tS66us6xWufvPx/PmAE9gxlMJAYKdElZibxVvLn1z1U1dQpzSVsKdElJibxVvGBzOR3atWFfTd0hX8vWumhJY1p7JSnpcLeKt1ZJWQW/evEjLnrgLdqY0a7R7odaFy3pLqYzdDP7JfBDwIEVwI3uXhmPwSSzxfNW8Z17q3l4UTFP19e+Tf76YH569lDeKCrVumgJlVYHupllA78ARrp7hZm9CHwbeDpOs0kGi8et4pXVtTzzzloefqOYXftquHLsAG67YNiByyqpsi5a28pKvMR6Db0dkGVm1UBnYFPsI4nEdqt4bZ3zt+Ul3Ftf+3Z2bqT2bcSxqVf7pm1lJZ5aHejuXmJmdwPrgQpgvrvPb/w8M5sMTAbIyclp7eEkw7TmVvHGtW9jBvTg7qtP4owhqVv7pm1lJZ5iueRyFHA5cDxQBvzFzK5z9+caPs/dZwGzAPLy8rz1o0qmacklkca1bw9eO5aL06D2TdvKSjzFcsnlPOAzd98KYGazgTOA5w77XSJxtG77HmbOK+IfH29Oy9o3bSsr8RRLoK8HTjOzzkQuuZwL5MdlKpEjaFj71q5Nm7StfdO2shJPsVxDX2pmfwU+AGqA5dRfWhE5ktau7Ghc+3bNqQO59dwT6JumtW/aVlbiydyTd1k7Ly/P8/N1Ep/pGq/sgMhZ6fQrRzcbZI1r3yae2I8pE1X7JpnBzJa5e96Rnqdb/yXpWrKyo3HtW95xR/H768ZxynGqfRNpTIEuSRftyo78tTuYPqeQZeu+YEifLsy6/hTOV+2bSLMU6JJ0R1rZ0bD2rW+3jky/cjTfOkW1byJHokCXpGtuZcfkMwczbfaKA7Vvt18wjJu+lvzaN92KL+lKgZ5iMiFMGq/sOKZHJ0Zl92DGnMLAa990K76kM61ySSGtWf2Rzqpq6vjj0nU8sLCYHXuquPSk/ky5IJecXp0Dm2n8jIVNXg7K7pnFkqkTAphIRKtc0lKm7OtRV+f8Y8Vm7p5XxPodezljSC+mXjg8JZqCdCu+pDMFegrJhDB5p3gb0+cUsqJkJ8OP6cbTN57KWcP6pMzKFd2KL+lMgZ5CwhwmBZvLmTGnkDdXbiW7Zxb3Xn0SV5ycnXKbZ+lWfElnCvQUEsYwKSmr4J75RfxteQndO7XnNxeN4PrTj6NT+7ZBj9Yk3Yov6UyBnkLCFCaH1L6dOZifnjWUHp1Tf/OsVGkyEmkpBXqKSfcwaVz79s1xA/jl+Qdr30QkcRToEheNa9/Oye3DHRcOZ/gxqVf7JhJWCnSJSePat5PSoPZNJKwU6NJqDWvfjuvVmYe+E6l9S5UliCKZRoEuLdaw9q1Xlw78+2Uncu1XctKm9k0krBToErXGtW+/mDCUH6Vh7ZtIWCnQ5YjCVvsmElYKdGmWat9E0osCXQ7h7sz7ZAt3zStkjWrfRNKGAl2+RLVvIulLgS4AFJfu4s65RSxQ7ZtI2lKgZ7gt5ZXc/9pKXnh/A507tAus9k1EYqd/tRlqV2U1j725hsffXkNtnfO90wclvfYtE+r2RJIppkA3s57A48AowIGb3P3dOMwlCVJVU8fzS9fxYMC1b+ruFIm/WM/QfwfMdferzKwDEFwZpBxWqtW+ZUrdnkgytTrQzaw7cCbwfQB3rwKq4jOWxFMq1r5lQt2eSLLFcoY+GNgKPGVmJwHLgFvcfU/DJ5nZZGAyQE5OTgyHk5ZK5dq3MNftiQQlljVp7YBxwO/dfSywB5ja+EnuPsvd89w9r0+fPjEcTqJVUlbBbS9+yEUPvMWHG8r4zUUjeP1XZ3HluAEpEeYQqdvLalRDl+51eyJBi+UMfSOw0d2X1n/+V5oIdEmesr1VPLJodVrUvoWpbk8kVbQ60N39czPbYGa57l4EnAt8Gr/RJFrpWvuW7nV7Iqkm1lUuNwPP169wWQPcGPtIEi3VvolIQzEFurt/COTFZxSJlruzqGgrd849WPt2z9Unc/qQXkGPJiIB0p2iaeajDWVMn1PAe2t2qPZNRL5EgZ4m1m3fw13zinhFtW8i0gwFejNSZZ+Rbbv38eDrq3h+6Xrat1Xtm4g0T4HehFTYZ2RvVQ2Pv/UZj725msqaOtW+icgRKdCbEOQ+IzW1dbyQv4H7X1vFVtW+iUgLKNCbEMQ+I03Vvj2q2jcRaQEFehOSvc+Iat9EJB4U6E2YMjH3S9fQITH7jKj2TUTiSYHehETvM6LaNxFJBHP3pB0sLy/P8/Pzk3a8VNO49u30wb1YuWU3W8ortTmViDTLzJa5+xHvytcpYRI0rn277KT+jBnQg3vmr1QFm4jEjQI9gfbXvs2cV8iGHRWcMaQX0y4cwegBPRg/Y6Eq2EQkrhToCbKkeBszDlP7pgo2EYk3BXqcfbqpnBlzC1l8hNo3VbCJSLwp0ONk4xd7uXf+Sv72YQndO7XnNxeN4PrTj6NTo5q1/ZK1NFJEMocCPUZle6t4+I1innl3HdB07dvhNvpKhQ3A4iFVNjMTyWQK9FaqrK7l6XfW8kiD2rfbzh92yCWTI230FYbQS4XNzEQkJIGezLPD2jpn9gcbuW/Byqhq34Lc6CtZMuF3FEkHaR/oyTo7bG3tWyasZsmE31EkHaR9oCfj7DCW2rdMWM2SCb+jSDpI+12gEnl2uHbbHn72xw+4/OEl/POzHQBU19RRU+tR74Q4ZWIuWY1WuoRtNUsm/I4i6SDtz9ATcXbYsPatjRnt2hg1dZE9bzbtrGzRJZ2wrWZpSib8jiLpIO0352p8DR0iZ4fTrxzd4kBpqvZtYUEpn5dXHvLc7J5ZLJk6Ieb5RUSOJGM254rH2WF1bR0vNqh9m3TiMUyZlMuQPl05fukrTX6P3vATkVST9oEOrV/PHal9+5y75haxZtseTh10FI9edwqnHHfUgefoDT8RSRcxB7qZtQXygRJ3vyT2kZLj/bU7mP5qAR+sL2No36784Xt5nDei7yFvduoWfRFJF/E4Q78FKACavrMmxRSX7mLGnCJeK9hCv+4dmXHlaK46TO2b3vATkXQRU6Cb2QDgYuA/gdviMlGCbCmv5L4FK3kxfwNdOrRjysRcbhp/PFkdmt48q6Gw3KIvIuEW6xn6/cCvgW7NPcHMJgOTAXJycmI8XMuVV1bz2JureeLtz6itc244YxA3TziBo7t0SPosIiKJ1OpAN7NLgFJ3X2ZmZzf3PHefBcyCyLLF1h6vpfbV1PL8e+t5cOEqvthbzWUn9ef2C3LJ6dU5WSOIiCRVLGfo44HLzOwioBPQ3cyec/fr4jNa69TVOf/z8Sbunl/Ehh0VjB/ai6mTIrVvIiJh1upAd/dpwDSA+jP024MO8yXF25g+p4B/lZQz4tjuPHPTaM48oXfUt+mLiKSzUKxDj7b2TUQkzOIS6O6+CFgUj5/VEhu/2Ms981fycn3t2/+6eATXndZ87ZuISJil5Rl62d4qHlpYzLPvrsMMfnzmEH5y9hB6ZLU/8jeLiIRUWgV6ZXUtTy1ZyyOLitm9r4arxg3gl03UvomIZKK0CPTaOuel+tq3zTsrmTC8L3dMGk7uMc0ufxcRyTgpHejuzhtFpdw5p4iiLZHat/uuOZnTBh++9k1EJBOlbKB/uKGM6a8WsPSzHQzq1ZmHvzOOi0YfoyWIIiLNSLlAX7ttDzPnFfHKis306tKB/7j8RK79Sg7tm9k8S0REIlIm0Lft3scDr6/ij0vX06FdG35x7glMPnMwXTumzIgiIikt8LTcsy9S+zZrcaT27dunDuSW806gb7dOQY8mIpJWAgv06to6/vz+Bn732iq27d7HhaOO4faJkdo3ERFpuaQHelO1b49d/+XaNxERabmkBvqefTVc+ft3WL6+jBP6duXx7+VxbhO1byIi0nJJDfQ12/bQvayCO785mm+Oa772TUREWi6pgd6veycW3X5OVLVvIiLSMkk9Re7braPCXEQkQXTNQ0QkJBToIiIhoUAXEQkJBbqISEgo0EVEQkKBLiISEgp0EZGQUKCLiISEAl1EJCQU6CIiIdHqQDezgWb2hpkVmNknZnZLPAcTEZGWiWVzrhrgV+7+gZl1A5aZ2QJ3/zROs4mISAu0+gzd3Te7+wf1H+8CCoDseA0mIiItE5dr6GY2CBgLLG3ia5PNLN/M8rdu3RqPw4mISBNiDnQz6wq8BNzq7uWNv+7us9w9z93z+vTpE+vhRESkGTEFupm1JxLmz7v77PiMJCIirRHLKhcDngAK3P3e+I0kIiKtEcsZ+njgemCCmX1Y/+eiOM0lIiIt1Opli+7+NmBxnEVERGKgO0VFREJCgS4iEhIKdBGRkFCgi4iEhAJdRCQkFOgiIiGhQBcRCQkFuohISCjQRURCQoEuIhISCnQRkZBQoIuIhIQCXUQkJBToIiIhoUAXEQkJBbqISEgo0EVEQkKBLiISEgp0EZGQUKCLiISEAl1EJCQU6CIiIaFAFxEJCQW6iEhIKNBFREIipkA3s0lmVmRmxWY2NV5DiYhIy7U60M2sLfAwcCEwErjWzEbGazAREWmZWM7QvwIUu/sad68C/gxcHp+xRESkpdrF8L3ZwIYGn28Evtr4SWY2GZhc/+k+M/tXDMcMk97AtqCHSBF6LQ7Sa3GQXouDcqN5UiyBbk085oc84D4LmAVgZvnunhfDMUNDr8VBei0O0mtxkF6Lg8wsP5rnxXLJZSMwsMHnA4BNMfw8ERGJQSyB/j5wgpkdb2YdgG8Df4/PWCIi0lKtvuTi7jVm9nNgHtAWeNLdPznCt81q7fFCSK/FQXotDtJrcZBei4Oiei3M/ZDL3iIikoZ0p6iISEgo0EVEQiIpga4tAg4ysyfNrDTT1+Ob2UAze8PMCszsEzO7JeiZgmJmnczsn2b2Uf1r8e9BzxQ0M2trZsvN7B9BzxIkM1trZivM7MNoli4m/Bp6/RYBK4HziSx1fB+41t0/TeiBU5SZnQnsBp5191FBzxMUMzsWONbdPzCzbsAy4IpM/HthZgZ0cffdZtYeeBu4xd3fC3i0wJjZbUAe0N3dLwl6nqCY2Vogz92jusEqGWfo2iKgAXdfDOwIeo6guftmd/+g/uNdQAGRu48zjkfsrv+0ff2fjF2tYGYDgIuBx4OeJd0kI9Cb2iIgI//hStPMbBAwFlga8CiBqb/E8CFQCixw94x9LYD7gV8DdQHPkQocmG9my+q3UTmsZAR6VFsESGYys67AS8Ct7l4e9DxBcfdadz+ZyB3XXzGzjLwcZ2aXAKXuvizoWVLEeHcfR2RX25/VX7JtVjICXVsESJPqrxe/BDzv7rODnicVuHsZsAiYFOwkgRkPXFZ/7fjPwAQzey7YkYLj7pvq/1sK/I3IJexmJSPQtUWAHKL+jcAngAJ3vzfoeYJkZn3MrGf9x1nAeUBhoEMFxN2nufsAdx9EJCsWuvt1AY8VCDPrUr9gADPrAlwAHHZ1XMID3d1rgP1bBBQAL0axRUBomdmfgHeBXDPbaGY/CHqmgIwHridyBvZh/Z+Lgh4qIMcCb5jZx0ROgBa4e0Yv1xMA+gFvm9lHwD+BV9x97uG+Qbf+i4iEhO4UFREJCQW6iEhIKNBFREJCgS4iEhIKdBGRkFCgi4iEhAJdRCQk/j/Pmm3/Gvml7QAAAABJRU5ErkJggg=="
     },
     "metadata": {
      "needs_background": "light"
     }
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "2.026895761489868 2.9732823371887207\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "source": [
    "plt.plot(losses)\n",
    "plt.ylim(5,50)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(5.0, 50.0)"
      ]
     },
     "metadata": {},
     "execution_count": 129
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ],
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD8CAYAAABuHP8oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABb2ElEQVR4nO19eZgkRZn++2VWVd89Z8/NMBwDA8MxwHDJITPch466HuiK/FwULxSPVUF3Bd2VxQPQ1RVFPEAFRAVBQK6BAREEG+ZghhmYE+bunrPv6qrK+P2RGZmRkZFZWVdXV3e8z9NPV2XlEREZ8cUbb3zxBTHGoKGhoaFRezCqnQANDQ0NjeKgDbiGhoZGjUIbcA0NDY0ahTbgGhoaGjUKbcA1NDQ0ahTagGtoaGjUKBJxTiKiTQC6AeQAZBlj84loPIDfA5gFYBOA9zPG9lYmmRoaGhoaMgph4AsYY/MYY/Od79cAWMwYmw1gsfNdQ0NDQ2OIUIqEsgjAHc7nOwC8q+TUaGhoaGjEBsVZiUlEGwHsBcAA/IwxdhsR7WOMjRXO2csYG6e49koAVwJAU1PTCXPmzCkpwes6epAwCLMmNpV0Hw0NDY1awcsvv7yLMdYmH4+lgQM4jTG2jYgmAXiCiNbEfTBj7DYAtwHA/PnzWXt7e9xLlVj04+cwtjGFO/7tpJLuo6GhoVErIKI3VcdjSSiMsW3O/w4A9wM4CcBOIprq3HwqgI7yJDUPiKCjt2hoaGjEMOBE1ERELfwzgPMArATwIIDLndMuB/BApRLpSw8AHYBLQ0NDI56EMhnA/UTEz7+LMfYoEf0TwL1EdAWAtwC8r3LJ9GAnQ0NDQ0MjrwFnjG0AcKzi+G4AZ1ciUVGwGfhQP1VDQ0Nj+KHmVmISEZhWwTU0NDRq0IBDM3ANDQ0NoBYNuNbANTQ0NADUoAEHNAPX0NDQAGrQgBO0Bq6hoaEB1KABB2kGrqGhoQHUoAEnQPNvDQ0NDdSiAdcWXENDQwNALRpwrYFraGhoAKhFA641cA0NDQ0AtWrAq50IDQ0NjWGA2jPg0Ct5NDQ0NIAaNOCADieroaGhAdSgAdcSioaGhoaNmjPggJ7E1NDQ0AAKMOBEZBLRUiJ6yPl+PRFtJaJlzt9FlUumLx2agWtoaGgg/qbGAHA1gNUAWoVjtzDGvl/eJEWDAE3BNTQ0NBCTgRPRDAAXA7i9ssmJkxatgWtoaGgA8SWUHwD4CgBLOn4VEa0gol8S0biypiwEekMHDQ0NDRtxdqW/BEAHY+xl6adbARwCYB6A7QBuCrn+SiJqJ6L2zs7OEpOrt1TT0NDQ4IjDwE8D8E4i2gTgHgALiei3jLGdjLEcY8wC8HMAJ6kuZozdxhibzxib39bWVnKC9TIeDQ0NDRt5DThj7FrG2AzG2CwAlwJ4ijH2YSKaKpz2bgArK5RGRZqG6kkaGhoawxeFeKHI+C4RzYM9p7gJwCfKkaB80MGsNDQ0NGwUZMAZY0sALHE+X1aB9MSA9gPX0NDQAGpwJabNwLUJ19DQ0Kg9A17tBGhoaGgME9SeAdcauIaGhgaAWjTgeks1DQ0NDQC1aMC1hqKhoaEBoAYNOKAlFA0NDQ2gBg24DmaloaGhYaP2DDhIuxFqaGhooAYNODQD19DQ0ABQgwbc3tCh2qnQ0NDQqD5qz4DrLdU0NDQ0ANSiAYdeSq+hoaEB1KIBJ2DT7j5s3ddf7aRoaGhoVBW1Z8Cd/wu+t6SaydDQ0NCoOmrOgHMM5uTtOTU0NDRGF2rOgJNeS6+hoaEBoAADTkQmES0looec7+OJ6AkiWuv8H7Jd6VXYsX8Av/77xqFIgoaGhsawQCEM/GoAq4Xv1wBYzBibDWCx873iCPM/+did/8T1f3lNT25qaGiMGsQy4EQ0A8DFAG4XDi8CcIfz+Q4A7yprykKQs9QmfF9fxv49p10MNTQ0RgfiMvAfAPgKAHHmcDJjbDsAOP8nqS4koiuJqJ2I2js7O0tJK4BwA86lcR0rXENDY7QgrwEnoksAdDDGXi7mAYyx2xhj8xlj89va2oq5hQ9ZS+19Qo46rtf4aGhojBbE2ZX+NADvJKKLANQDaCWi3wLYSURTGWPbiWgqgI5KJpQjjIFraGhojDbkZeCMsWsZYzMYY7MAXArgKcbYhwE8COBy57TLATxQsVQKyIRo3J6EoqGhoTE6UIof+I0AziWitQDOdb5XHKEauPNfx0nR0NAYLYgjobhgjC0BsMT5vBvA2eVPUjRCNXC9wEdDQ2OUoeZWYubTwDX/1tDQGC2oOQOezSuhDF1aNDQ0NKqJmjPgoQxcKygaGhqjDDVnwLOCF4p6wlJTcA0NjdGBmjPgIgMXXQq1hKKhoTHaUHMGXPRCEWOCcy8Ubb81NDRGC2rQgHsmejArGHDnv6UpuIaGxihB7RlwQTZJZ3OB30PcxDU0NDRGHGrOgIsa+MbOXvczX8ejGbiGhsZoQc0ZcFFC+dDtL7qfdTRCDQ2N0YaaM+C5PBqJZuAaGhqjBQXFQhkO4Ax8+tgGtDYk3eNaQtHQ0BhtqEEGbhvoiS11yoU8Oly4hobGaEHNGXDOwOtMQ8m2dThZDQ2N0YLaM+DO4p1UwvCxbb6QRzNwDQ2N0YI4e2LWE9FLRLSciFYR0Ted49cT0VYiWub8XVT55HoGui5hwLKCS+m1Bq6hoTFaEGcSMw1gIWOsh4iSAJ4jor86v93CGPt+5ZIXDpuBCwZcT2JqaGiMMuQ14MwWlXucr0nnr+pWUpZQOLT91tDQGC2IpYETkUlEy2DvPP8EY4yvoLmKiFYQ0S+JaFzItVcSUTsRtXd2dpYn1QBMgzQD19DQGNWIZcAZYznG2DwAMwCcRERHAbgVwCEA5gHYDuCmkGtvY4zNZ4zNb2trKznBJ86y+wmDyMe2+UpMPYmpoaExWlCQFwpjbB/sTY0vYIztdAy7BeDnAE4qf/KC+PVHT8KzX14Ag9RsWzNwDQ2N0YI4XihtRDTW+dwA4BwAa4hoqnDauwGsrEgKJTTVJTBzQiMMUkso2g9co1Yw71uP4wM/e6HaydCoYcTxQpkK4A4iMmEb/HsZYw8R0W+IaB7sCc1NAD5RsVQqQEQQ9nPw3Ah1OFmNGsG+vgxe3Lin2snQqGHE8UJZAeA4xfHLKpKimDANNdvWEoqGhsZoQc2txOSQJRTolZgaGhqjDDVuwL3v3qbG2oJraGiMDtSsASfJC8XzA69SgjQ0NDSGGDVrwGU/cA6tgWtoaIwW1LABlxi4818bcA0NjdGCGjbg5NvgmIeT1fZbQ0NjtKB2DbihJRQNDY3Rjdo14KESSnXSo6GhoTHUqGEDrqMRatQutLurRjlQswacJD9wDt0wNGoBOT1U1CgDataAG1LwKh1OVqOWkNUVVaMMqGEDbhvsxas77ANaQtGoIWgGrlEO1LABt/9/7M52DGRyehJTo6agGbhGOVC7BpxbcPgbg9bANWoB2ZyOe6xROmrXgJNgwHOW54WimY1GDUBLKBrlQJwdeeqJ6CUiWk5Eq4jom87x8UT0BBGtdf4rNzWuFAQCjsGcpScxNWoKWkLRKAfiMPA0gIWMsWNhb2B8ARGdAuAaAIsZY7MBLHa+DxlEBj6Y9YajehJToxagGbhGOZDXgDMbPc7XpPPHACwCcIdz/A4A76pEAsNAggHP5JiwJyaws2sA33tsTdFySm86i427esuRTA0NJTJaA9coA2Jp4ERkEtEyAB0AnmCMvQhgMmNsOwA4/yeFXHslEbUTUXtnZ2eZki1JKBID/+K9y/B/T6/H0s17i7r3R375EhZ8f0mJKdTQCIdm4BrlQCwDzhjLMcbmAZgB4CQiOiruAxhjtzHG5jPG5re1tRWZzCAMHwMXDTiQztjfiyU5L79ZnOEHgPNueQb/88jqoq/XGB3QGnh18Mpbe5HO5qqdjLKhIC8Uxtg+AEsAXABgJxFNBQDnf0e5ExcF0Y1wUPRCYcw17tVwKXxjZw9+9uyGIX+uRm1BM/Chx8ZdvXjPT57H9Q++Vu2klA1xvFDaiGis87kBwDkA1gB4EMDlzmmXA3igQmlUQpZQuBcKY0xYlTmUKdLQiA+tgQ899vUNAgBe295V5ZSUD4kY50wFcAcRmbAN/r2MsYeI6AUA9xLRFQDeAvC+CqYzAFlCEffEdDc4RmkWnDHmmyzV0CgXNAMfeozEEs9rwBljKwAcpzi+G8DZlUhUHERNYpJnwUuCxQBT2+9hg47uAfSlc5g1sanaSSkZ5dTAH3l1OxYcPgkNKbNs9xzJGElNumZXYlLEJKYrp8S4z/7+DB5cvk35m/YpH164+u5lOOv7S7Blb1+1k1IyysXAV27dj0//7hX8x59XluV+xWBv7yDOu+UZbOjsyX+yRllRswbct5An54+FYhj8c/77fOne5fjc3UuxriNY+SzG8PKbezGQGTmz1rWM3b1pAMBjq3aW7Z73vPQWXt2yv2z3i4tyaeD9Tt3ctLt66xYeW7UDb+zswU+fWV+1NMTBSORjNWzAvc+DWctlyxZjwrL6/G9s275+AFAa6a17+/Evtz6Pr933ahlSrFEq2lrqAJQ3ENQ1972Kd/z4ubLdLy7KxcCTpt2ERRlxqMFzQjUiToykaa2aNeCm4ZdQuK22mPeCSm0i3QNZAMCrW4eeoRWCHy1ei/f99PlqJ6PiEN9xraNcGnjKMeDDwatl+BvGEVBxJMTxQhmWICkWite4CwstG6fSDXct/KYn3qh2EoYUw/19xEG5GDiXCweraMBHwOuoWdQsAxcllEzOcl0GGfOMe6kVixsKXUGHB9xOukzGr5qhh8vFmC2rvPcrBrztDX8GbqNGkhkLNWzA/SsxLaFxc+NeKsvJ5DxdXaM6eH79Ljy9xl7kyw1Fuexuxqqe0SsXA+d1s5oa+FCAMYa/LN9WUj5HYjOuYQPufR7MWq68JS7kKURnVL1cXlni3kXvBlR+/PSZDfjBk7ZEpJLJSkEmV733VS4NnJdFNfPioXLc9tm1u/DZu5fipideBwA8unIHPn5ne1H3GkmL80aEBi5KKJawerJUljOYy7n3jIORMLk23JDJWshJUlbZDHgVWWv5GLj9v5p5GQrewsnU2p22u+8nf/ty5R9aA6hhBu6fxOQVmTEmMPD4lVq17H4w63QKMW+jpZbyI2tZblRJbsjLZsCrKKGUyxWSdwRVncR0/leS2LbW21yzqz9T9D1GYuusWQNuCinP5JgrX4huhIWwHNW5vFHElUZ0fIvyY1B4t7x8y2WrRoKEwsummgaco5LCRMJp8Ny1txjwZjxyBJQaNuAkTWLy5lCohBJl7NPO4p74GnjMEzViI5vzL9ICyjfXUE3ZgXuNlMpaeb2tat0bgofzd9894GfghdSFkThCrlkDLkoofems54UiTGKWj4HHvEeVK0g13eIqhWyOCcy7zBJKFVkrZ/9Jo7QmOBxe+VBIKPzdywy8kDY+EttHDRtw7/NLG/e4PTETohEWMkxVGV8+cRJ/ErO6FaTaHUglkLG8RVojSULhnUeJ9rvqdQ4QpYnKWXBufLvTfgNeiE3m544gJ5RaNuDeW9i2fwBv7OwG4I+FUljvHDzmGfC496iyAR+BDCOTswKTlyODgZdpIc8wMOAcFWXgIfksJP+VIDhv/97T+N2Lb5b9vnERZ0eeA4joaSJaTUSriOhq5/j1RLSViJY5fxdVPrliuvzfB5x9MLM55jLvOAycG3uVx4rrB16EG2E1fMJHogHP5hh6BrJYs6NL0HtHggEvz6Kk4fDOh6Kuh2WzkEdzY1/OkcKbu/vw9furF8o3jh94FsCXGGOvEFELgJeJ6Annt1sYY9+vXPLCYYR09/ct3ep+LoQRq3pyVwMv4h45iyExxLtBjEgJJcewu3cQF/zgb5g5vhFA+fJZTc8Nd0VhiVkZDq/c1cAr+IywtlwIAy+lo+noGsALG3Zj0bzpZblfuRBnR57tALY7n7uJaDWA6dFXVR5iNMIwFKSBR0ooMRm48LysxZAY4g1ScsNiNV55IY6MCpW08t67qm6EhdWtMAwHBs5RyRWOYj7FdlaQhFJCf/2vt7+ItR09WLy6A0s378XfvrJwWJR9QRo4Ec2Cvb3ai86hq4hoBRH9kojGhVxzJRG1E1F7Z2dnaakVEMN+I1fAQg3VuWluMGK+KPG0agzPRyQDF1z90lnHrXMkSCh8kViJeRkOGvhQJEGs2/1C7P5C1mJZ3mxrwVjv7Db04PJt2LynP5CmaiG2ASeiZgB/AvB5xlgXgFsBHAJgHmyGfpPqOsbYbYyx+Yyx+W1tbaWn2EtP3nNKZuAluBFWg90NB0bw+o5ubHU2ySgHMkKe+DxHufI5HCYxS81JoeGTK4GheKpIovoGBQM+RBKKqspVcSGvi1gGnIiSsI337xhj9wEAY2wnYyzHGLMA/BzASZVLZhBhGriIqIb+1u4+dHanvXMVLzedKV5CqcYy7eFgwM//wbM47canynY/cck5Z+DlyuZgmTrZ0258Cp+7e2mBz/bIQbkMS7VeP0//UHmh9A16roRDJaGoUEiojkohjhcKAfgFgNWMsZuF41OF094NYEinYuNIKFEM/MzvPY2TbnjS/a6SSUqZxBytDLycyFlMaaDK5a6ZVayGXNfRXfB9tu7rD90YOwwi+y+FOIvvvNrvv5J+4GLeetMiA1efv6snHTjmeaGUB8PAfsdi4KcBuAzAQsll8LtE9CoRrQCwAMAXKplQGXEYeL6Gbm/+YH9WGftBh/HFbRjiadqAF4Z72zfjydf8mxWHSRzl9gNnzN4T9aEV23DOzc/i8VU7ynL/6GcXNxEnQ7y2lPvs6knj/qVbir4+Lu54fhMeXbm94OvErIkMXDV6WdfRjfn//SR+88ImvL6j2+2Uyz1fMBw08DheKM9B3Wk9Uv7kxAc3vEThDKYQDVxl7Hkji6uV5kqUULoGMugfzGFya33B1wLlC5BULEphxl/54woAwKYbL3aPheWnEhLKnP98FJ8/ZzYAYOXW/Thv7hQ7DTkL6zt7cfiUlvI81IGPgZdwH9l1tVh87I52LNu8D6cf2uZuHh0HO7sG3N3o40go1z24CoD/PceBj4EPRjPwXT2DAIBv/uU1tw5tuvHiog14mMRVExLKcAV3IxRjSZx75GRMG+MZv4JioUQupY8X/pOVKKEs/P4zOPmGxQVfx7G3bxDn3fKMuyp1qNEneAeUA2FlXi7mIwezakjafp+il8N3Hl2D83/wLDbt6i36ORs6e3DhD/+GfX2D3rOFvJXEwIUslNKBb9lrTzwXqsd/9q6lrsHMh1K0fp8GLiynV9UF3vbk8uBlVahW/4Hb/qE8Pgzsd+0acC6hiItl6pMmLn/bLPd7nB4yavcePmkGxIubUaoboUq3KwRPvrYTb+zswf89va6k+xSL3nTxoT5VCCtzxhiuvLMds655uMT7SwY8FTTg7W/uBQDs7i3+3fz4qXVYvb0Li1d3CM8WvUe8c9/a3YfLfvFi7LIUDVgpI6BiO5GugfjxufeXEMvbCmPgedqtiGI7/pc27om8X5z5uEqhhg24/T8pBAYn+Bf4FBOprF+oHOL+e3H24hOfVw05gxuelFAm7Zv2YMveviF5fk8eo7N2ZzfueH5T7PuFauAW8Likl4dhy94+nHzDk9i8J1gG8juqdxg4d1NbuXV/7LRGgbtCimQjjIF/57E1+NvaXVi8xjP2URBZbSkjEz7ayRRYb8W5qHx2TPT6kpHJWfjC75dhY8hIJ8wLJWorRBmut0yZpjH5wrlqbtFWswacIyk0CiIgIRjwQmSMnMVw90tv4YhvPOoeS4uLSHL55QGxIVbDx5h3PsmE91rf+9MXcPp3nh6S5/elo8vo639eieseXIXXtnXFul/Y+yvEUN33ylbs7ErjD+2bA7/JDZ1LKAOZHJ5+vQOX/Og5LH1rH4DSPEW4cUwIcp/47NK8ULzPxTLwnMVcj6tCdwoSoynms2MdEQb8lTf34v6lW/GVPy5X/u5j4OloP/B0iAEvllOF5SvndgjVQ80acP7exEZhEMEU2GchDd1iDPe/stV3bM0OT0sO69XXdXTjvle2uPfId345sbsn7eso+hQMfCiRj4HPmmDHMnlg+dbI8zjCJoIL0VJ547p/2VbMuuZh34YAcifLG+pAxgpo3tc9uKpoiYvLJSIDF9l/ubxQugayBUkaHO+59XkvGFwJDDzfpZyBj2lIBn5z46OH1F1xdJvPDzxUQinSgvMYPGH3i+MRVynUrAHn70JsFAQgGUNCURmArMXQlwk3QGEG+Zybn8UX713uSxMQXonKhYFMDm//3hL88WXP9WvAYeB1ieq8Vq7bhj2/uc5uuHH3NQxl4AU0RN62+PJnPlkHBI1VTpDR5Ca5alsX/u3X/yzKQPIwDUmTkM1Z+EP7Zgxkoj0p4poE0YCdc/MzOOb6xwtO3/LN+9zPhU6+k8+AR1+7p9ee7GxtCDq/8c46EWLAxTLKV3YqBj6YtYpecHTAuGgDHvayBjI5zLrmYfzM8dKpBGrWgHP4emzya+BhbEJlAHIWi5QA4kSuE+/LGU2l0DWQQU86i+37B9xjrgauMKD7+zN4ft2uiqap12FGfDJQBm/gcQ1wuB94/DTJ+qRvnkIyVjx9/SHeNCu27Mf7f/pC/Ifz5zjPNA0Dd77wJr78xxW+jkRFKOJmsdwx6AuV/sTSzTeQ4E4FKg066+5QpLaG/N2YBvnIlNxpdHanfUvtOfoHc0VLKKqOybK8naLC+gM+aXv7cxuLe3AM1KwB57vIi5q3QeRj5GHR+VSG3bKY8sVzxJFExIZYaQbO9e60YGx4+lUSyid/8zI+dPuLroSwobNHGbPEshi+/fBreGt34ROfXJtsSJpo37QH7//pC75y4xU+7jA9zkKeQofF4rPlAGb8cQOZXOjElCirqbBiy76AG2fG1cBJ6c0i2odCB+PlXi9WaHmK9jbftbx8maJ74tp7PgmlPmH4yNR3/rrGLd+BTA4nfvtJ3PjXNYHrewezRUsoKgOeY8w9bhBh7c5u3KuYZwEqq5HXrAHnbc80yFvUA5vlcIQZCpVhyFrMp63JKNQLJV0CA4/DqrixFoeT3KirGPjqHV2+NC686RllzJLXtnfh53/biM/es7TgdPPyq0+auOqupXhp0x5s3x+ULKLy19E1gO88ugY5i0Us5Cl+slg02kE/YY+Bxx1my+z5nT/+O8675Vnfsay7eQPz6aXc+EVJD/96+z8w15lYV7lOljuAVaGLU4wCJBR3R6WIwHFhMfT5fFZDyvRJJIvXdOCBZXYYg7DJS8BuL1aREoqqGor1kwi48Id/cxejcQzFQs2aNeC8Fycil4UTyRq4+oVmfcuY+X81A//ue48BkN+AM8YkDbx4Ax6HofK0is953WF+pkF4YNlWnz7OjVM+FlLKJgd8IsogYEeXLe2Iw2UrhIGLBv2rf1qBW5esx4sbd0e6EXrPjE6vPMEkvnu5LLiRUGngYYjD6rJC2Yv3rXMCxkfd4u/rdrt+zyrXSdXze9JZ5QhwQ2dP3gVJhWrghUxiRu2olM0zicnrSH3SDKwPiBNmuG8wW7RBZYwF9h+wmF9CySral2ejintuHNSsAT/AmRn+5NsPdguXQEoNfOlbe/HPTZ4zvujdIO54rjKc3LUsncdQWCxaQtmxfwCzrnnYl44wxGFB/QoGzpGzGK6+Zxn+/Q+eSxZPWr7Oga9OTBWxm5CKgfnKOkQDF8/x3NlY6EKeQoKGyY0nKviTTwOP2erieDrx98mYP0F8pKTUwCPu+18PvYbvPLoGu3rSSqN51HWP4Z0/+nvg+MKbnsFZ31+SJ63ho9a9vcEVl2Ix5RsN8PJWPYKXUSJEA+fNryFpujGKOMKIgYjedK5oP3mLBdNlB1rzJBQOsd17Br5yFrxmDXhrfRKbbrwYi+ZNd10JDcM/BOMTfO/+yfN4nzD55NvdgxuVkJfLF3fkY+A5i/nukc5YyFkM97z0FjI5Cy9u3A0AuPOF/BugxmPgtlyhmixV5YXnMx9jzefOFQWXYQnHvnTvctd9LKdgKYCaFecsFuqTXIiEIjedbJQB54Ygx+J7gcQYsPD8BRm4Y8DF9MboOH7x3EbcumQ95v/3k6GyxetFhlMIK88v/H4ZjvuvJwLHReOVbzTC06pe9RzthcLrdH3SDLTFsHolom8wG7knZs5iysVePN3yvJJlCfVWuJ3YHl0Drhl4NDzWTT4NfNOuXrXerTIYIUyuIaYBtxQSyr3tm3HNfa/iV3/f6DbMOD6/cYax3FNCNVRW5YUfyXfvjGIyaUNnj0/LFmFZzB0NqBrSss378D9/XR36OyD5RDvF3J3OurugBJ4pXK5aOZjO5vDbf7wJy2KRDDzMjTDHgteFIc5oKRty37qkXcalxUIpr9AaZgQfWmFHEJQ7VXEhT76keDJDsMwG3DUM6oJnTtmlpElM8bliW5eJfN9gLrKsfvTUWpzx3addI84Yw/89vQ67e9KwrOD+tuIkpviLOCJ2NfLQp5aOOJsaD3uIGnhCklDeVHhTiC86Jw6bFWhI2TU0DgMXK0g6m8NORwfuGciCxtjH40w6xTEK3iRm8FxVXjz2E31v3iGIBnzhTc8AUEeQu+GR1bj9uY1Y9+0L3WfIZcVZWi5ktCMaBZ6+qA0SfFEfFe/lJ0+vxw8Xr0VjygywLd+7D2jg3vG4w954DJxLKP77xtHA86HcIU3DZCvTIOQshnTW8rFkkYHnq9viCEcGr8ehDNxiMImQNMndjs69r0KaG9uYcv3O+Tn8Z1Xn3L5pLwBgXWcPpo1twG9e2ITvPfY6lr61FxYLjkhzghuhYZBbPqIB9xi4llAiwRm4QUGtaoOCxfkZn/158141w2xI2n1cvsk9sUcGbAbOh4V1SdOt6IUMuaPgGnAFA1ctWeZZHsxG39vzJY9X6f6ywvYA2Ly33y1XedTDX0kuF2xogCRrxLBHoqE46/tLfA0V8AIs7ekdDDTWQUVnwSFO9MZtc3EMaMbNt/94c13C91zffSMmen3Hy2C//fNG6grKQ1ZETc7nKwueJ9UKW9VcjnxvwyCkEmZgPkrlniqv9hQ1axXGNtrn7+4ZxI+eWovr//IaAHtC2GIsYMDlSUxehioJpZKIsyPPAUT0NBGtJqJVRHS1c3w8ET1BRGud/+MqntoQJIRJTHmoo2KjPgnFealv7lbPzvNFKXm9UCx5IU/OdSWsSxjg71/lAxuVvjD0Oxq4yl1RZcBZTAbOO4a4GviB45sA2B0lNzJyWfHKHTqJKQZ2ilHp5VO2Sf7sPO1ZiwU8D6IZuPc9LmfKKibE5c/8nBxjvue31IcPgGV5J4xAlENCqRfcTsOMDg/bLEt2BXmh8DoYwcDDnm85DDxlGkENXFGvWqWytZhXVio73uoY/C17+/CKE/8GsG2KzcCDk5giw+bebwOKScxKIk4rzQL4EmPsCACnAPgMER0J4BoAixljswEsdr5XBabpSSiiBg6oK4uv0eW4AVdPYMTVwHMKDZxX9rqEIWjgkbcJpC8MkQy8ayBwjFdaeYh8xa//6XM37I9YDKQC9wba0NnrMSzpGd7ow8+Ueh13t0KjOMoNQ3bx4h16JmsFWKH4HsM0cCD+xNOrW/Yr7y2uKeB18M9Lt+LHQqjf1nrbaISt9BMRHqCpDAY86a2cDZNQeIA0mTCIRZ/XD1x4/7LcwutxWN3PWfZ7rksYAS8Ur25719YnTdQnvTpsWV77VKWTv7vNe/p9spy9YUwIA3e9UEQGrtDAqzmJyRjbzhh7xfncDWA1gOkAFgG4wzntDgDvqlAa88L1QhF8wjnECpHN2Z4hGQUDD4NrwPNJKNIQLZ2xfI2Op4pX3Dtf2IT5/x2c1bfTHF9CkRtUY8rEvr5gvA5XA5fysXhNh8/dUBXRMApNdXb5rO/scctSLiteuWVf2bnXPYZFP/67/33E6LzkBij7evPGlhFYEodqZah7X+F7nA2Pd/WkccUd7e53kZ2KYYl5vp+SQsS6EoriUXIdKHeAJhG+IFsh9TxMQilEAxfbmpxubvhUj2eMoWsgAyNkElM1OV6XNN22C3ANnLmfZfQM2B1uZ0/aZzOI7PNlbd7yjbjJrXNpn4RiufeoFArSwIloFoDjALwIYDJjbDtgG3kAk0KuuZKI2omovbOzs8TkqiEyMJmNicbh0K//FR/99T99lVSsC/K1gOcpEM8LxX5WQ9JEOptzK3s6awmVxz7/Gw+swq6eQeUQOJ6EovYDN4lCJjGde+dp8PxaXhL5DAQv39e2d4UuFjIkDxzx9zU7uv2dbAyDJLc/mbVxY5PNWYGyFA1zMBaK93kgIqwChxw7x8/ABQMeYhSb67kBD0o36zt7cNeLbynvLSLORiP50Jjy5Iaw8k+ESChUgBuhWAyBDsqVUIL5vPWZ9fjjy1vQNZBVTmKqNPCUafgMeE5gzKpkdqdt0pPNWYHO22JB75icqIGT1wmK5cPr1+Y9/dixPzgqLgdiG3AiagbwJwCfZ4zFC+gMgDF2G2NsPmNsfltbWzFpzAvfSkypoGUD9+wbnUo/cMBmr6p7J02KxcD5fflyXx6nZDBnecvI5SG9cll/DAklozbghkGRcVgyOUvJlPh9uOHhlS/f5BI3Tmt2dIeWkeuFEmLgfStj40goUvplI+Yy8JwVKMsoBi7eN8wrCbD9sGd//ZEAs0qHGPCwTRK4Bq4irr/6+yZ87f5XlfcWUY648y31CbfuF87Avc95NfCIjprXM1UH8sir3ibIajfCYL1qa0mhPiUycK+cVfWfM/CsFVx/oGLg4oib4HVwYZOYp/xP8VslRiGWASeiJGzj/TvG2H3O4Z1ENNX5fSqAeFuIVAD+lZj+LKliVIsNSixklQE3DfXEiYycMGFmM3ALA841g1mPCcquTGoDHoeBOwt5pHQlDIqMhJjNqVec8olAuSFFGTLxvMGshbU71X7bOcvC0693KJcbi/cA8jNKg4KdoGzEeH3IKPIqnpu1LMw/cJzrsSAamKh8/9dDryGTC06Q+uLSCKGJw5hpi6OBx3EtveHh1crjUQY87gSnxRiOmjYGALBTMQHeN5hVSgQy8q/E9D7LRpK3A15WK7fux1V3vYI3d/fCFFevmsGFPKpFalNaG/wSiuW5+aoZeNZNl3gfAoGxIDG0GHPbtBhET6WBVxJxvFAIwC8ArGaM3Sz89CCAy53PlwN4oPzJi4dEhBsh71lFiJXH8hnwoFcAEdm9ft5YKF5Fshl4zo2PPZj1KgWv5HxWX3XfQtwIA3JFng36spalNCi8AbkM3DFm+Ri4WNn3hcT5vuOFN/HRX/0TLzv7SwYZuHcPVYREEQnTiPQsEe+fyQXzKjPwproErjj9ICcdAgOPI6FI8eNF1t0/mN+drKU+XAOXEbbFWlSHp5rgVsGygKTjNnrrkvV4fNUO97c/vrwFR37jMWx2tuWTR3eFRIb0r6BVj8L4PZ5e04GHVmzH9x573VenVW2Rv37x+VPH1PslFEuUUMIZuL0CWJC0HMKg9APnDFywOz4/8DL76KsQh4GfBuAyAAuJaJnzdxGAGwGcS0RrAZzrfK8KTEFCET8DXoxqEWLlERmwOBsvIo4BFzWxxpSJdMZyQ7eKhkSMawyEGfDgsbU7u/G2/1nsepiEhb4188yYZHJMydp4OvolBp4vrrlY2eOMUux7+88rRMdNOgsmoq7nac/mWKAz9DNwhoThxc8R60KccMAPr9ju+y7WtajIlhxNKYUGXuCEVxQDjxuT3mLMXVQE2KtnOZ50Amjxe8kSil+OjH6O7Gb5wLKteHTldt9vsp49kLH8DDxCQhEZb1trnS8uvcW8EZMqnXyknskx5SSpyoBz8ueTUESCUO5YvwrE8UJ5jjFGjLFjGGPznL9HGGO7GWNnM8ZmO//3VDy1IeCFR8JQhuuu3QoGfv9Sz21ObACphBEYKgH2irl8DVolofAFJoNZy5VtuO3iPbZK21QNvX77jzexbf8AHlxuL5wJY4iqiVgRz63dhd8Jk2Mc3Pjy8uCdSF4NXDDGcfVYOXv8uoMmNuW9NpkwghJK1sJHf/USHli21Xc/u+P0pyktMXDTILeuiOmPw8B/9NQ633eV50kU+LsSs1MoaYuaL8n37jhshunVG5HIyPVJbgdhW8Pt7BrAZ373ijsKBfyMNJOzcPU9y/DJ377i+82V5Hg9tCw/A1e0T5e9CwZzXGPKlw9x4Y0s9WRzlm/kKXf6oRKKO4lJkW6ElcSIWEovsm7XmMOWVHoVGvhjq7ywnGIZp0xCwjCQkTYwbqpLoCfPhr2iX2hDykTfYM4z4DnLNYguAzcjGLiiUU4d2wDAC9AVtv1bPgP++/bNSh9vng6ZCeWXUPyNMg7k/PGVk/LiCxUShhHQdjM5C0+/3omnX+/ExUdPdd0o01kLDREaeM6yY1zwIvNJKDGNn4jeAg24Kh54ocPuqJW1cfKwatt+bN83gNmTWtxj4pZ4AQOeUbNfwDZ0OYvht/94Ey+s341HV+3AGbMn4tKTZtrnCmUil488PzIozB/JDFwGr09ivTqkrUlyIxTikUtlLG6SnLX8I1QiCpVQxPji/POwW4lZC+CsWwwnazGGhGHk3WhXRCphKAPKt9QlfJvhqiC+0MaU6Qv1mc5aQu/vpLlADXxCUwoAXHckmSEe3NaEX/2/E/MacEA9ccqXJ6uGsFHIWpY77I8bA92y/CyIhykNk7BEJE0KMHgxP//+h+X49fObANgGLJ8GbhqGb9KTo7+IDTn6fYt3nA47ohFzFzzRnhQ67I6WUPIb8Iv/9zl0p7P+IFtRBlx6x3JguPte2YLrHlyFRx0dXTxfNNpy2mQXVM70B7OWLw0q8uGOGp1rH//CmWipTwY0cP54mR9xF0KeH/8kploD9y2lJ7G9iJ1B6R5C+TAiDLifgTuNArZhlyWUj59xUOh9kqahXELeUp9QdgSypicGnRdhT2I6BtxZSs+HZIO5YCNTsTfeyLft78d7fvJ37Orxx/+4/SPzsWDOpFgGXAVu2ET9GMjP4jI5hnpHPy2EgYtlt6fXbkBxDLjhMCIxmyKD+rOzOwtg69BRS9K5Bs4Nqdjg4soPIvoUDFwV9wMAll93XlkYeCkauNiZiQty6oT3IC+SCvMA4Z+fXG2PbhPCHM9n7noFd734lq8zk8M9iLH7Z13zMN5wPJoGc5KEkgjWkYw0AcqfLWrgjIleKPb/Pb2D+Our29223VyXCCz0s89XxQP3nmeQNy8jEivNwGPCi4Xi1xUTBgUM7+TW+tD7pEwjEIYSsBdcqLR0sfFc8qPn3OX4sjui7UbIJRT7mFmgBs4Nwc79A75YDRyc0eebxAwDb5jecueYGnjOcoe1cScjc5Y/j3v7OAPPXx35oiixo1LJZIC9vF82cLKEYhoEPugSfyvZgDtlEVYmYxqSrnEUzyi00WdzwVjVHGF54KOfnULIBbE8RaMsGy5ZAxfTyxiwapu9RIQvUupJZ/Hwiu342v2v+jqnHVJ4Yj5X0eW0s5c22lNqtoTinaeSULx5G27A7XPqfQzcyxdPxtX3LMWnfveK6/46piEZWDvAl9Kr/MDFYFY8DeJEtjbgMeFFIyQfg06YQQlFtePH4ZNt/S8pxCwREcbAZUPLYyaLQ7eGpOlbyCOzhLheKK42GMK4uKaez40wDGEMPP8kJnOH3HErbM6yfGXHWUtDDAbeP5iFxfwrAMNkso7utLtfIkc6Y2FdRze6BzLIWhYSBrll5pNQBGP83hNmxMiV3/OEGyRVuFtucEnBwAud+BrMWcqJd8Bft0T2yz+KQcDEai/KInJ9ivJCyTHmdqa8fr8ubAItnrttn9d5MBYMeSDmQexcVHnNSPIfbwtifepJZ/AHJ+YPL2++0chbTgzwcU1JeyGPkBZbQgnaDXHOiwQG7pu01QY8HjwvFG9i6IQDxyFhELok32RT6kknt9Zh7vRWAECdaSgZbHNdEt0DmcDstdw4+T6QYsWZOqbe16vzyhbtRqhg4Dm/Ni2DV7CwLanyYdAdIcgauGfINnT24Iv3LvMx1UyOueEG4iJrMZ/Wy1ldHAmlL2NvTmsqDLgsH01sTgWu7xrI4Jybn8XR1z+OnV3pUC8U0Ye6raUuTrawZrtnrDIuA/e/388tPBTPfXUBAFEDF4xroQzcskLj1vg8bnydhH18h8DA5W3BuD9+kIFb6OgacD1+cpKEwidyeb1+det+AMD0sQ0BLxWOQYW/vvibT95R5JVP5PJRqieheOf+/G8bfekEvAiEbzkj53GNKWelsndv5pwvN6uHVmzDM6/boUFEDbzQiexSMSIMOB9WEexG8fDnTsevPnoiEkZwwitpEP74yVPd73bUMttwJE1DqSG31CeQybEA+wjTNxuEBUETm+tsCcXyN2h3ElMx2aV68ZzRicZdHDq7o5ASGbjoq/38+l24+6XN7jn/9ut/4r5Xtvo2xs1ZlquBx4VlMV/Z8XKNMuBJkzCxOYUL5k6BHK9bZn0cJx88IXAf0cjya7iB8C/k8dIXN7Tu4jUdONV5ZtiI6YiprZjkyHg8uaLBUE18fWbBIbjslAOVz8xkmVuXZKj2ZxQ/b98vGnDvutue3YDTbnwKa3d2K7xQcvjSH5bj6nuWYcvePncRje2J4dUjbsh4RzBzfKMvDaIk2RexX2VgElNhwJ9cvRPrOnqCGnhIfeKP4tEgNzqhpMc2pgIjTj63JY/M735pM150ZB6CN2rui5BQwsIUlIIRYcD5ijZeyHOnjUFrfVK5u4dpEObPGo+b338sALtH5wYolTCgagstgp4nIkzfbBAYaUPK9C2lf2NnD558bafr7eIaTgVDUj1L/E1kI7zSFrEXsT8dgoTyoZ+/iNe2e2FvNjlMJWkaGMjk8OelW5EtloELlXswhgE/pK0Z7f9xLqaNbXBXvX70tFkAvPciG9q501oD9+mW3qHtheKkI2cpV9Q118XvoE471DbgGXfE5a8jojHygnx5xlbFREVPGRmZnBW6DVlY3BdOEMRVyuL9+QT5A8u2BSYx+wZzbl22jaaFC+ZOwVmHtfm8cIIrhG3vD046RK24dzAb6n0jM/CUqX4X3/zLqoAGPnVMg3J+gDNwHkJho0NIxjUmlStEGQtO5srwQiSHM/C4XlqFYEQYcB6WU4ZKTuCNnFdYMW5wMlRCse8vRxQL61HFJfmphIG0MIkJAB+7sz2ggfsamKIyi4aVQ2R3piuhFPdK5VgUUcO/HGP47qOv4/O/X4YNu3p9q/jiwGJ+Vy3u6SJO/n7i7Qcrr+VbV1mWx8Y4A5c72HGNQQlFhu0H7kkovBPp9xnwpPJaFTi7zuYY9vdnAhKK2Mnwqnb73zbg8P94FHt6B5XlbgoLRWQM5qzQbcjCXPi4sRQZusgweX3/8dPr8Pz6Xb57dg9kcXCbveBq9fZueyLYKcO9ijDGHJksQ9ayXNIhRnLsG8yF1jd79OrlQ8XAAftdyxr42UdMwvPXLgxMjm/a3Yd1HT3urlOd3WkQqTfYyFp2TPmogS33fwdkDTx8EVm5MDIMuFPw8vBH5dPNj3FDV5cw3EZrGuqetsmp0Jf86Dnf8d6QxT1iFDS+9FesoBOb6wJeKKI++Mpbe7Guwz/U5x2AeB+xUbsx0Yt8o2ELeVSwLIYdXd4EWBzvEREyA+eVnjegtpY6fPhktWTAF03kGHMjRYa9h6gdbzhMSULheRHT1xzjPhxtzbZefucLb+LYbz6ON6Td4cV3xvfHfNxZrr6vTx1eOGGGG/BsjhU8icnrkGhQDAJ+eOk8AH7dXt6pvTudcYNwfefRNVjfaQebMgzCvj6/a6svLTkLOeaFZxYXovWms5AX14h5EElLWF7HN6UCGjgRYWJznbJNn3PzM778N9cllFJZJscCXk8yxMBWHd1p/OI5W2+X+V2c8AyFYkQY8BbHwMrDYxUbdaUG5yeRgVtMrSGLS7z/vHQrnlpjN7iwndobBSmgzolkKFbC8U3JwEIe0WA8tGI7zrn5Wd89VQzlQ84KNzs/5PtfKFwGrojsJuO+pVvxyKtewKNCGThj/vv3pLMwyNMsbbctdT5MIpfxmESRi7W4xhkFMRaKyMBFxOkIOMY3pWAahF09tofDakGCAvykQn5VquiJgL+TkZHJWaEaPTdQPekstgh7vqpW2RpEWDRvuh3HRzBssqTQ1Z9VRn80KNqNNGtZsCwv5kpvTAaetfyxScIYeG/ak2HkNhDmWit2cK31SeWIPWtZAa8nGRbzS5v/9ZC9n2aAgRexOCwfRsRSetfnVPLVVjJw16jbv9UlTDcEbc5Z2CHjsMkt+JfjZ+C5dZ34/O+XAQDWfvvCwF6MHI0SA8/kLGQsC811CRw5tRXrO3sw0WFq3iRmdB7lCv6VCw73LdTwOqYiJRTuRhiy8bCIW5es931XeQbkg5j2vsEcGpKmKwXYsSf89+QNyCByG4vhMPBQA96Q34AbRK6UkbWYMqRwS4hEp8L4phQShuhWJo0KhfcjkwVV9EQATkdVuAEfzFo45YbFPm8TwDM2fgau9mKSJ2G7BzIB6TCqg3HTmWUwjaDsxT9H1TfRpVNV1w6b3IzuAW/RlpyHsKSJ+R/ToJ4zszXwaAmFr/JsdEJoADYJ4R44queVCyOCgXONMo7PtxuDxKmEdUnDnfizGAvVTeuTho9FX3vfq9i6L7jLRkPSdFeyGeRFMsxZDBOaUzjt0InY3TvoDhnTWQsPLt+GPSHDzxfW78aLG3YHdPGkYfjyx41BqZOYqshu+RDH/U+GWJl70lk0pEx3eGwx5obblWGQsBjKCfUrzvxPG+Mt1IrDnBngY+AJwwhMfBUioXADzhGokxEMPJ21whm4dPL5cyc7aQ4f3vdncgHjDXhkQWSE3ADn87jpHsgGNqjgEkoUXt/Zjde2d3kauBh6VxHyQES7E4YYUE9ittQn0Z3OuAuzZLYcOn8g1MGxjWoGnsnZDDyqg+K2RFSBHly+zRdzCdASSihawhi4wghww8CHj/UJ0618FmMYr/AdBuyKvVcwso+u3KGMXd2QMt1KmjANdzOIbM5m95Nbbea9s8seYm/Z24fP3b0UH7+zPXAvAPjgz/+BD9z2j4BnimmQcoSRT0JRuVYRqRbyxGcLYQz84c+djtmTmpXpExtPbzqLuoTpvi+G4OiJfzOkTispLdYSl0/HkVCygpdD1jGGsldN2CS5Co0p08fk5FWifiPhz+PH72x3PSIAoMnJS8KkgAwg7jqkeuUG2duyqeCuss3mfOeL9w1D90A2sP7BCGHgh01uDhzzDLhXLgN5DLgIlYTS4qyUzliWsv6HGd84BjxrMaUfuAguKfX71kz0Bs7TDDwEzWEaeMQkJt/urD7prb7MWV7QqMB1gk/57EnN6ElnsXlPXyCCnmmQ59ViEJLOJCYf6vJhPV9gxF221nWoGxuHzMySJilZqueNEsz7CQeOw9lHBLcubU4lAv7ohTDwMANenzQDxpCzW5GNWMw2vPzdqCQUDrEx8p1QRDlG9ACKw8AHs54B53E35BFFSwFeKCTJHb2DERKK9Ir29A76DNnkMdxfnCAXBy/HrOVn4D+8dB5ufM/RGNOQxKtb/EN4DjdglMjAeb3JM4QbFEKvenkipYE7YFwjXr3+PLx/vreSlWvgFvM6KDHYm4gmSc761f87UTmJ2VxnG/BcTi2BhunXYh0c05BSSiiZrL2wh4hw0MQmnHzQePe3K888GAeMb3A7gpOE3xoUUlwlNPC8BpyIfklEHUS0Ujh2PRFtlTZ4qBq4Ae9JS6suFS/T8/P1fI+9BRXhEor4cg+bYi+937av352RF+/PKyln4HwTBb49G+D11s++Eb7Rs2jUg7qjOnY5z7OKad/y/nn41FmHBI431pkKBl66hGJPEEoGPMENuD8/DUnTfTdyfGrA0zHFV2oaQUMvNpw40g5/L+5nCnrVxJFQ/vjJU3Hfp98GwG8EAwxc+K0pD7PnHi1ineIQmaj428zxjbj0pJmoS5hK+QQAlrzeiY/d8U+8sGG3e4yXLy/PsPgqALC7N+0rI1kD5++uPmmipT7pT6twHSczAxn1JKZY7uceORkL5kwKYeBJJzSC2oBzfOGcw7BwjkdgRH2/LmEor+XnGER4+t/Pwk3O+hEAOHRSM04/tM1l4BceNQUfPmUmGlOmci6ltaH8U45xGPivAVygOH6LuMFDeZNVGHiAqktPnOk7ziujWJjcEJ9x2EQAwMXHTHWHp1ynVkE0KHOc2Ckd3enAizIcXZZfwz/3Z3K2QU/4DXgUPvTzf7if5QqeMINMkT8f8EeU46hLGkq23JRKoDudRWd3OnRn+SiEMXBDMfkWacC50YiY9fdJKEQBQ6NqOFEYzHnDYy6hyCtL43j2HDihCcfPHAfAz7LlUA5ieeTbwIKXlWkQPnzKgT4mK3ZcYj2oExalheHbj6zGk6v9W7SZrgbuMfGwe+zpHfSNdGQDzsuvzm0HYmcjGHCH/ITtLiWSIx5uQJWmVkdCseO7q9JsX5tMkM+DZuVWz0PI3lk+eC2vp/y1iXWhLmH4vG8SpoHW+iQGs0Fvpp9++HgcM2OsMp+lIK8BZ4w9C2BP2Z9cRjSkTGz8n4vwmQWH+o7zxiIOpfmxOVNasenGi3H8zHG+FXHjQyQUsRIeIui68lDJFNzSZo5v9E3aJA1yvzOWP3iTGHJTZsQJxVBfzJ8YB4KjPmEqXf6a6hJ49o1OnPjtJ/OGQVVB7CzEjk4sCw5XQpE6sLqkgaRzrqrr8Bi4//7ykF824OLzZ45vDNx3MGsJwawsEKnL9aRZ4wPHwp4jpkneJ1Q07vn0ZtE1tCFl4r/fdbTyWjHPXHcu1DPI80JxOg0i1IWkb09vxld3uRshB68P/L/YyYr5b0iZMA0KGHBe1mIMfm5IVSOD1oYk0llb2lF1tl4MfgrVoQnkq7t/+OSpeO8JM9yRqeckIHRUSdNXH/lIKWuxgD9/sd5h+VDKXa8iohWOxDIu7CQiupKI2omovbMzXC4oFSrGxstQDEak0vjOnzsFxx4wFlctPBSnHjxBLb0I100RPB3kxm4a9uKBH146D7d9ZL7LGHjlEhlEk2KJ9ucWHho4BgQZccI0lAtoeEVTxSepSxpKBiM2Rj4yKCQstdioRMNih2r1l2VdDAYetqgD8Dcg06BAJyhPXK+/4SJcc+EcAPbcxaYbL8bR08e4v2fESUzHt5yX68zxjfjbVxYAAO684iR8zNn8WAW/tON92S8bcKn+HTdzbOBeYxuTuP0j8wXXUD87BrxNiAH/++NlGMXAo9LPn2GaFAiSxe/Zk874Og2DJAbulJ+4wlmFVMJAfcIIyEw8+qNIYHh9IcWoa2yjzdQ7e9JKGcTbhzZ8b1vT8L+38U0pjG9KCc918qpg4N49vPYdnPvIP4orBsUa8FsBHAJgHoDtAG4KO5ExdhtjbD5jbH5bW1uRjysOfPg6qcUzuCrPlDGNSTzwmdNw0MQmTGiuw/obgpK+OGEoxhQPSij2/0XzpmNic52neQ/mkDQNHwNWaaCzJ7fgPy85MnA8k7P8DdUgpUzCDZxqEiVlqiUUubGHSQZhumjSpICGyu8jG6xQCcVhY4DX4O658hT87LITfOeNafSG1QZRwNMkat7D9SUXzrF1b09CM4WRzdjGJA5wWHt90sS4kNEZ4O/wxLoiGwy5Id/98VPw5BfP9B07bHILzjlycmBxlkhSxHfRkDLxp0+diheuXegeK5SBkyuhGG46w+SpgYyFxjr/qFYsU15+vK6LdYIJ4yu+Clpm4AuPmIQfXjoPVwkjarG+yPMjfN6qszutJGj8iSYF38fZcybhncdOw6fOOtRnG1KmXxPndURm4OI7SZreCDtq7qOcKMqAM8Z2MsZyjDELwM8BnFTeZJUH+/ptDw/uugcU3xOKL0AMUyozwIBk4LzQ7oGMTxOXr+WLRVSsErAD5TdJjUbVSBPCBJIMw1DrmvKxMLe5sKBVCSGGjHgvVQwP3phln9j6hOcHzo3hKQdPwFRntMOXnZ8tTEKZBrkBiTgMRTq8+DdOeiVXRDGJDN5qwXzbiXE01yV8E25Rmrmss9YnTXdRFwD85yVH4jv/coyTTk9ukyEa1/qkiRMOHI+pYxq83ws04K73kul1GvI9xBXGYsA2sQyJvLS5DFwYLeQsj81yAy77yo9vTGHRvOlYNG+ae0w0vHK6fAZcQdC4nGGawR3tiQj/+8HjML4p5esYUkKIDcCrV0EG7tf+VQuVgOJXSOdDUQaciKYKX98NYGXYudUE39zWx8CL7AnFhlcnvCiZ6co+p/y8roEsmuuTPqMrGuSxTUn3etVE3NZ9/X4tP2QSk4fpnBQSw7o+YeJth0zAjz90nHtMZjRhBjzMq6Mu4UXLEw2LoZBQXAaeCTJw0Q+cQzZeE5rrXNdNkyiw2tI0gPs+/TYs+fez3GOiUeLXcXzznXN9jbInnXMNj5x20YiI4V1XfvN8H8sMi9chpkGEaJDeNW+aO7nJz1VNKIvyhqq+FBreQPYDTxhBuU2s60nTcA2x2FEnBe+oeoUGnrUsd4SSStgjQm7svnHJkVj9rQvc93HopBb86VN26Gexw5fTxeetdvWklaSGl17CoAADFw2tvPemSKTcvEojDbE/HtuYEiSU/GtSyoG8fi1EdDeAswBMJKItAK4DcBYRzYNdNpsAfKIiqSsRfAJJlDyKLcik1PDqnRWWAQ1Warxiw26pT4Qa8HGNKWze04+sZYWyJ3/4WEOpcy91tls75eAJ7g5BIgyDcNfHTwFgM52lm/dhjbBrCk+nCmGTrq0NSaVOGzWJKe+JWJ8UGbhnsPj8xZmO15CdviS6BrIwFAzcNMj1BuHgBoMPd3marlpwKCY21/kWXXT3Z1DvTFLLqwt54/+Pi4/Ax844GL/5x5uq4ohkWyrjLrNpjkSEARdHEar3UqqEYhoU0K59UTZNm33mnLg1KeE6nn+eBlFm4J4+yDmSXtLE8+ttd8ZkwggQIt5GIhl4k1cHVLIir06myoALhlbMbyph+ILSGQoJRWbg45qS6OjmDNw/wqwUA89rwBljH1Qc/kUF0lJ28Jflk1DKwMAB72XLjefCo6b6vouVraUu4fsuDkm5IeofzEUOf4m8/T5Vk5jjmpLYuq8fJxw4zn1+2MTN2w6diLcdOhFfune573g4A1enq7U+KbihiZ1MUAPnFfnul97yHW9Iehq4yLqnjW3AC9cu9I2iml25CaESigiXgUsG3PUsELLVNZAJZ+DOBtT5jGNYeFfx2WHnq3aEVxlwsY7Uq+Y7ivZC8f5HMfCE6cV7qU+a7oggYZKbH94ZiR4ZdvhbAjL2KEHs0FTyZpPTafg0cImEjW3wJE0lA2dejBQuoVx09BQ88uqOcAae8DNw/pOv3CUNfGxDyn0+l4UObmvChs7eyFFZKagMrx8mOGKqHdBf9CcNi7GRD/IL4I1frNSv/Oe5+LS0UKYuwMD9ga44xjo6Xn8mF2pw01kvnrLthRJsuL+4/ET84ZOnug1y+tiGwDkyAhp4CAMPk1BaGxJujBnRgNsrCP33PqRN7fucShjutbK5mjqmwde4uPeOQWoGLoMbFFGnBTyDIRr9/f0Zd2QT3I3GctMaBX7fI6e24oZ3H40PneytT8hX/0Rjzjse1SIXsZzVDLwwCYXfTmTgshuhaj0FPy5OfiYkBi5uViIG37Lj2HhMVRU1kD9TdDtVzTPxOSRVHbUUDPzL59ueSWce5jlWyJ2Jz4ALE8rvONbW5lvqE775k7GNSbfc+waz9v2EZ1cCIyIaYRju/vjJ2NE14BvOmMUycKnhJRQGXOVD7mPg0qo08Z2Ob/QWNXD2PG1MPbYJm0ikMznUJ00MZCyft4SIya31mNxaj950FhOb6/CtRXOxp3fQ3cBVBZm1hDLwEKPgY+ASg5SL+6jpY/Chk2firhf9DDxpemw9yo0Q8IbVphFcoahi4EmJcYvpk6/J5JhbrgEJhQdAy2McuXFLJgzXePP8FrLlHU+XHJYUkLxQFPWgaAYuzBeIk4+AFGVTMuA+eU9i4PJmJbztpBJG6K5A3r3tdz1JlEEVI5yxTUl0p7MhGjhn4IZvVP7i1872hc4Q2zgR+dZSiEz7fy+dh69dNAct9Um33BqcrRk9V0vb64wp7l1OjGgGPrYxhTlTWv2LLIrsCQMM3HlxjSGslENsSM11Cdu1TmFQTnAWLxw4vhFzp43Bym+ej3+T/I4HBM1ddFlSoakugfb/OAdnzG7DonnT8bEzDo6VRiBcAw/zQmmtT7p5aUr5h52mFPtjzpRWZQdhM7dwrwsR/HomxWEGohm4zPDCYqhzCUXexPqQNlsbnzEuelTD3688b1Io+H2UDDzvJGb+pj1nSgvOOcKOahhwIzSDboS27u1PG2AbLy+SpPebanOMjLBtXUryAVe9u4aUif/94HH4zRWeo5tKjuBSiyrfIgM/6/A25zwTk1vrfZ1BsB4EJRTALivu8cOPj3MImOiFkjDIrUPDygul1iA23OINuMTAzSADV8HPwBO+Y2K63nnsNDzyuTNw4dG2ht5cl8DbD/P7zXMGDiCUgReDQPjUEAZ+imKTYMCWXHi5jhX8tMXATsfNHIt1374IR88Y4zY2EQlTHYtC+TwnfX2DWcydNsZ5lv1b1CIsmZ3z8pdJO2f48qjlMwsOxZ8+dSrmx1yVGXcz5DBc5NQFVbmL+2Cq6gE3ZAdPbMInzjzYNfLXXjgHRzrS4r8cP8ONzyFrvKbCC0UM9OWXULzdbCzGBAklyMD5alfAHtH1DEYbcMBuGz4XSUW58vwp24SwEvN/P3gcnvvqAuWzDhjXiKOnj8EtH7Djnfg1cHXaeF7GOBKo6AeeSngMXBvwEiAWXrEFGZiMcxfMRKtQYmXj2nKd20j89zxS2oR39uQWfO7s2e73AUEDT5rhG90WiqAfuF9XntCUwvPXLHRXyMkQY2HIwcDEJcj8s0pjFyWUfGhyg5flcNDEJmy44SJceNQU+3lKCcXwpUVMt/ifg7NSORyraRBOONAz3l+9YI5P3xbzAiCwkrFQnHzwBGy68WJ3LgcIBp0C1CSC15OGlIlrLzrC9eZ5++FtLjOuSxquf73sOWJSsAMyhDADYgfSkPKkAzBxLUKQgWctbz/UxpTpG23F7cBV+W2MZODcD9xO/4xxwZAKgL1I7C+fPR3vPm5G4Dlhu/rw+tYqkbOedBYJw3DzVxnzPcI1cA6xgUZtjRQFWcOKivonQjSOfNUgPxZHDxWH4TmL+Rh4uRBnEnPa2IZIHZ0Xj2zAVXKRapf3hGHEZqycgfPhtxiPWnULj4H7j6s08Ac+cxoOGN+I7733GEzLMwGsiuxo39eZpKuA50HCIGRyLO8kpvxOxRg8CTd9ni83bxduJE3DcMMA1CcNDGTsuOPiJDqHOIlp7x9p+O519AwhdEHWcg14YA1FzDqtYtmcgavmJ0Q/8EKg8gOXwW8p+7z3prMY05B09fcizU5ejBIGXtx1c5ywsYDHquSher4NfUUGLksoceqTPOnqxRov36sLSij+RhBnGMiZnGw4RO8EDlUIgYQZvm2YjEMm2Z4s4qSxyk/XS4OaacsGPGEQjj1gLADgffMPwGmHTkQx4J1uJSau3AVTIjFQhCkV424DnoFJZy23s00lDG+jDOLXcXIBd33APKdMRAYujpYaU6Zbhxi8/PO6ev7cKfjLVacDsP36efS+xpSJhz57unuf2Aw8woCr2qNVpA4dR0IxpJEL90O3mN2uPAZeGQs+Shh44Q3pxa+d7dOCPVc0vzHIp0OLDY0PY92l2jG6ZblSuwy8jOwuyNbUeVKll8sqYRq061PtY+CqScz4ktC75k3H2IaUb45AFWxIvDeg0MAlCaVQz40wuBp4me4nws6Lfx/MCU3BVbc8L9wHm7tbZp1t4wD7PctRHvlEtWUBA46L3zEzxuIfG/aABAYux2Jxnyfs4C7Wo6NnjMEPL52HUw+ZgNO/87R9XdLEUdPH4KCJTdi4qzc2A49ym1QycFcDL+x9+BbyhFzKy4/nf1xj0lurYRL4HG2lGPioMOC8XogxJ/JBXL0JiD7D9nc3dkSeSudbuOPodLzyGwbZk3sRu/HIlc71QqmghBJmSFWV+PvvO9Z/Dqk7HFHr5AacV3TArux8GP/l8w+PTC8RYcGc4M5CgLqTkScxxZV59nH7e7kMuOtGWEAne+8nTo3VgalCFqjSXScYVAC46f3H4ld/34TjZ45zy0O8jtzr7PeVtSzc9fFTsHTzXux3QlIw5l0j1kv/JKaXf9lradG86QAQkFASMdsSB7/urMPbcP075gLwCE3UiLgiDFySnhpTCcyaYHdISdPAde84HN94YBUmtca3PYVgVBhwPow8cda4ou/BK6g86ZPPZ5k3tHGN4nJfzwvl/k+fFnm9OFRdOGdSQAN/7PNnKsPSFgJ54keu6HyCJg6DkeUq3gjEIEJcQmmpS6DL8QPmxm7TjRcXkHIP/DWoWJwczMpNK/kNe6HLz8PgTjQXwPjE7biiIIeYDU1D0m/AJ7XU46sX2ItXTLcjY16nJqU9ZzEcPWMMjp4xBj9Zss69lxz0CvC7EcLnRqiul/xdcULjvp+YNJXf96SDxmMWjxujWAkso1ANXLxX6AYjXHoSOo45U1qwcVcvEqaB8+ZOwXlzpxT03EIwKgz4QROb8NMPH+9bdVUoAhq4u8gi+rqEaeAn/3q8Lz6HGDciH/g5h09uwc8uOwHfeGCle18AOFzQ6YuFrIGbBuG/Fs3FgROasLajB+cdaXtlyPboG4qwt0EGbl8k7oTCG2CzYMDLpReryjQpyV8c3AhxL4lyMXC+2nR3b/ikb7GQA3OFge/eruIXIvngRcINPTfgou85N3yMefVerDNi5ELLiY0ChC/84uC6NZea4jJkTgoGBoOrM6PoVCkT/2GXckIndv6HTW7BX1fuKCyofrHpqvgThgkuOGqqLxhPoZCNwKcX2B4Iqp23ZVx09FTlJhBxPGI4i2uuT/jiiZczQLzMWkwiXHbqLJx5WBuuOP0gNya2yJBufv+xvoVGfEm0vAiIT+qI4QF4pMSvOhstAOWLl6yUUBSeMID3LuOusIyLw6fYbn/rFTuTlwp3x5w8LVeWUHz3cMrankz0y0r8fYmuf3wOicHbzEQOYSBOYiYMOz58PgmJSyFcDozrIcZXSIrbErq7akVsBVhKHQvd2V5Rd7j82jWQVV5TTowKBl4OJCQGvnDOZN9w/wPzD4h9L8484njHyDqtygugVMjkN2yCVGQwstHnLn2t9Un85arTsWVvHwCPLWUkCYWX3dX3LANQOgPnzTZSQpEZuHMud318xzHTUA4c7uyZKu7G84m3H4yOrtIZuew5o9rRBxAmMRX2jG+WrJKMohi4xRi+995jccXpB2HutDH4yp9WuOeIboQnHDgOb+3py2uQXQbuXCtv3B0GXqf6ffFR7P+5CNZbCukJu5QTE3H0xvcLkHdjqgS0AY+JME8GoHDdllfAWF4ovONwmNKkljo01yXK6qImuziFpUtskLK+yRn4mIakq52K5+Xb5b58DDz83nIj5Ma+raUOy79xXmgIgUIxpjGJL59/OM6Y7bkhXnvhEWW5d0KQCp776oLQPVxFLVvGVy+cg4PbmnHOEZOx5I1O935h14lRIhtSJo5z5MC/fWUB9vbZm6ZwA8aYPdq9QIrKqUJj0plbMf0joXzgdap/0Dufv8uozbjDWHQchHVGPEqi2BlOdEaY+5yyqSS0AY8JWQMvBXyYGsdtymW9zr8PnTwT5wrbbVUC+SZmgfAJP3mDBa6Bh+1mw1GqX3tUzImwlZgiIxO3aisH5A22ywUxxGzYikJANKjBd9mYSuDyt80CIKwQdDVwzwuFw9PA/fc6YHyjK68VEzaASyhcfsnk6eTl6waETR7C0iiiGNKTMMjdK1UFvtGEz4A7bp0RfUnZkDdHzqbFHUS0Ujg2noieIKK1zv/i3TtqDKX04hzuYolYfuDc+8VGfdJ0G02lkI8t83SoIGujKglFhXIxcKUfuBQPnKOSnWClwNOc7x3JC3nC4GrHrgbO5YwgA4+6V9h+qVHgpEiuM/nAJ0fFScwrzzgE75o3DZedOiv0umLWTogLm1RwGbjQHia2hO+dWm7EKfVfA7hAOnYNgMWMsdkAFjvfRzRa65N493HT8cv/d2LJ96pTLG4Jg7chb8mPDYd0bznCnwphDFxmsvUxDXipAe+9jWuD90klDDTXJTBWkhvK0RkPNXhgq3wjBs7AozRhIFiv+HsV2XCcML/FePBwWeK6d8zF586ejYUhvv0yuO7/EWcUAdjl8YNLj4vsDIrRwOvyOBxwDVyMnV6Ks0ShiLMjz7NENEs6vAj2NmsAcAeAJQC+Ws6EDTcYBuGWD8wry704O8pn1ACPNVRqKS4AnHzQeJw4axz29WWwtqMn1lA2LLRss1R5uYQStkkFR7k0/bBJzMe/cGZgIVcN2m98/eIj8OFTZubdqKMuQkJRgUkSihiDXPRCCUMhHfCsCY3YtLvP/T6mMYkvnntY7OsnNNcVtV6gmBFXvtGyx8D99bcxZeLio/PPA5SKYruKyYyx7QDAGNtORPG6Tg0AXqXIZ9QA0QWuculpTCXwh0++DVffsxRrO3piMfAwH1/ZgHoMPNqQlEvOCNMq8wWmqhUkTQOHTsrv+18X4YUigpeWPIkpeqGIC39C01UAA3/ws6f7NnIYKhTFwPPELeI7Bckj0te+JYsWlUHF/cCJ6Eoiaiei9s7Ozko/ribAX3a+iT1AWGE4BHSR7wQUNTnGEcbAZdQrVmKqUGrs7EpvXVVrSAhufVEg1zjb3+sUIWB5kUb164Vo4K31yap0qMUxcG/7PhU8L5TyrCEoFMUy8J1ENNVh31MBdISdyBi7DcBtADB//vwhmJcd/vAMeC7PmdHD1nLjslMOxKkHT8DsyfkZnszAf/3RE5XhZuNuOlHJSczRCC5dnZRn8wkOj4FzLxSv5smrNVUouQMeAhQj0/EOLYw/LZo3Dc+80VmWFdHFoFgD/iCAywHc6Px/oGwpGgXgjYRvkhsFPmwtNo55ISCiWMYbCDLwsw5Xq2j1MYfWJbsROiZI228bjakEHvv8mZiZx2OJVysmLaXP+Qy453seBs5uw3ZzGg4oRQMPm4N6z/Ez8O7jpg9J+1Qhb2kT0d2wJywnEtEWANfBNtz3EtEVAN4C8L5KJnKkgc/Yx1m4UOkdPYpF3CEjH8rna9jlYuCVnOytNcRhhXJ5cYN12SkHuseMGBo4ANzygWNx3AHD16O4GA085XrlhLfVahlvIJ4XygdDfjq7zGkZNXAllDgMvMI7ehSLQtjM/33oeBw1vTXynFK16yGIGzQi4TFw/p3wxn9f6PMq4Z/yTYjyrciGK4qR19zRcoz5qmpg+I53RjDqkgVo4MOUgReCi4/J705VCxrqSISqXgU3M7b/x3VJHEkoxGOsGtCtpgoY6wRPmhBjg4nDHE2aB8MfqSiX98hwG6kMd7gMPELh5jLLUCwNH24oxOGgGtAGvAo4fuY4/OAD83D9O+fmPfeA8Y3YcMNFeNdxI9uAl4pCyOEVZxwEADhyarSsMxpwobPY5IzZ4bHy43ihDGd8a9Fc3/62hYCHhk0M0xGillCqhEIMsnaNi484JbXg8ElF7/wz0nD8zHF5y6Kak3TlwEdOnYWPRMRIicIXzj0MU8bU45IhWFVZDLQB1xgRiJIANEqDIU10jibUJ0189LSDqp2MUAzPcYGGRpGocbI4LOFFLByFFnyYQxtwDQ2NSHhuhNqADzdoCUWjIHz5/MOLiv1caWjbUjnwkKqt9eXd9EKjdGgDrlEQyr3TzLjGJPb2lXPvQK2hlBvHzxyL/7zkSLxHe0INO2gDrlFVLPnyAvQPlu5jqwl45UBEuOL04TuRN5qhDbhGVTGmIVnwllpR0JOYGqMJw0/M1NAoAloD1xiN0AZcY0Tg3CPtcLaHTmqucko0NIYOWkLRGBF4//wDcPEx04Z1PGoNjXJDM3CNEQEi0sZbY9RBG3ANDQ2NGkVJlIWINgHoBpADkGWMzS9HojQ0NDQ08qMcY84FjLFdZbiPhoaGhkYB0BKKhoaGRo2iVAbOADxORAzAzxhjt8knENGVAK50vvYQ0etFPmsigNHG9HWeRwd0nkcHSsnzgaqDVMo+d0Q0jTG2jYgmAXgCwGcZY88WfcPoZ7WPNo1d53l0QOd5dKASeS5JQmGMbXP+dwC4H8BJ5UiUhoaGhkZ+FG3AiaiJiFr4ZwDnAVhZroRpaGhoaESjFA18MoD7nf3yEgDuYow9WpZUqRHQ10cBdJ5HB3SeRwfKnueSNHANDQ0NjepBuxFqaGho1Ci0AdfQ0NCoUdSEASeiC4jodSJaR0TXVDs95QIR/ZKIOohopXBsPBE9QURrnf/jhN+udcrgdSI6vzqpLh5EdAARPU1Eq4loFRFd7RwfyXmuJ6KXiGi5k+dvOsdHbJ45iMgkoqVE9JDzfUTnmYg2EdGrRLSMiNqdY5XNM2NsWP8BMAGsB3AwgBSA5QCOrHa6ypS3MwEcD2ClcOy7AK5xPl8D4DvO5yOdvNcBOMgpE7PaeSgwv1MBHO98bgHwhpOvkZxnAtDsfE4CeBHAKSM5z0LevwjgLgAPOd9HdJ4BbAIwUTpW0TzXAgM/CcA6xtgGxtgggHsALKpymsoCZi962iMdXgTgDufzHQDeJRy/hzGWZoxtBLAONeZ3zxjbzhh7xfncDWA1gOkY2XlmjLEe52vS+WMYwXkGACKaAeBiALcLh0d0nkNQ0TzXggGfDmCz8H2Lc2ykYjJjbDtgGzwAk5zjI6ociGgWgONgM9IRnWdHSlgGoAPAE4yxEZ9nAD8A8BUAlnBspOeZhxZ52QkhAlQ4z7UQAV+1Te1o9H0cMeVARM0A/gTg84yxLgrfiXhE5JkxlgMwj4jGwl47cVTE6TWfZyK6BEAHY+xlIjorziWKYzWVZwenMSG0CBGtiTi3LHmuBQa+BcABwvcZALZVKS1DgZ1ENBUAnP8dzvERUQ5ElIRtvH/HGLvPOTyi88zBGNsHYAmACzCy83wagHc6+wXcA2AhEf0WIzvPYOrQIhXNcy0Y8H8CmE1EBxFRCsClAB6scpoqiQcBXO58vhzAA8LxS4mojogOAjAbwEtVSF/RIJtq/wLAasbYzcJPIznPbQ7zBhE1ADgHwBqM4Dwzxq5ljM1gjM2C3V6fYox9GCM4zxGhRSqb52rP3Mac3b0ItsfCegBfr3Z6ypivuwFsB5CB3SNfAWACgMUA1jr/xwvnf90pg9cBXFjt9BeR39NhDxNXAFjm/F00wvN8DIClTp5XAviGc3zE5lnK/1nwvFBGbJ5he8ktd/5WcTtV6TzrpfQaGhoaNYpakFA0NDQ0NBTQBlxDQ0OjRqENuIaGhkaNQhtwDQ0NjRqFNuAaGhoaNQptwDU0NDRqFNqAa2hoaNQo/j+gr+5N3uFe7AAAAABJRU5ErkJggg=="
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "用autograd实现的线性回归最大的不同点就在于autograd不需要计算反向传播，可以自动计算微分。这点不单是在深度学习，在许多机器学习的问题中都很有用。另外需要注意的是在每次反向传播之前要记得先把梯度清零。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 本章小结\n",
    "\n",
    "本章主要介绍了PyTorch中的基础底层的数据结构Tensor和autograd方法。Tensor是一个类似Numpy数组的高效多维数值运算数据结构，有着和Numpy相类似的接口，并提供简单易用的GPU加速。`autograd`是PyTorch的自动微分引擎，采用动态计算图技术，能够快速高效的计算导数。"
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}