chapter2/2.3-deep-learning-neural-network-introduction.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.3 神经网络简介\n",
    "目前最广泛使用的定义是Kohonen于1988年的描述，神经网络是由具有适应性的简单单元组成的广泛并行互连的网络，它的组织能够模拟生物神经系统对真实世界物体所做出的交互反应。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概述\n",
    "在生物神经网络中，每个神经元与其他神经元相连，当它兴奋时，就会向相连的神经元发送化学物质，从而改变这些神经元内的电位；如果某神经元的电位超过了一个阈值，那么它就会激活，即兴奋起来并向其他神经元发送化学物质。\n",
    "\n",
    "在深度学习中也借鉴了这样的结构,每一个神经元（上面说到的简单单元）接受输入x，通过带权重w的连接进行传递，将总输入信号与神经元的阈值进行比较，最后通过激活函数处理确定是否激活，并将激活后的计算结果y输出,而我们所说的训练，所训练的就是这里面的权重w。\n",
    "\n",
    "[参考](http://www.dkriesel.com/en/science/neural_networks)\n",
    "\n",
    "每一个神经元的结构如下：\n",
    "![](6.png)\n",
    "\n",
    "[来源](https://becominghuman.ai/from-perceptron-to-deep-neural-nets-504b8ff616e)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 神经网络的表示\n",
    "我们可以将神经元拼接起来，两层神经元，即输入层+输出层(M-P神经元),构成感知机。\n",
    "而多层功能神经元相连构成神经网络，输入层与输出层之间的所有层神经元，称为隐藏层：\n",
    "![](7.png)\n",
    "如上图所示，输入层和输出层只有一个，中间的隐藏层可以有很多层（输出层也可以多个，例如经典的GoogleNet，后面会详细介绍）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 激活函数\n",
    "介绍神经网络的时候已经说到，神经元会对化学物质的刺激进行，当达到一定程度的时候，神经元才会兴奋，并向其他神经元发送信息。神经网络中的激活函数就是用来判断我们所计算的信息是否达到了往后面传输的条件。\n",
    "\n",
    "### 为什么激活函数都是非线性的\n",
    "在神经网络的计算过程中，每层都相当于矩阵相乘，无论神经网络有多少层输出都是输入的线性组合，就算我们有几千层的计算，无非还是个矩阵相乘，和一层矩阵相乘所获得的信息差距不大，所以需要激活函数来引入非线性因素，使得神经网络可以任意逼近任何非线性函数，这样神经网络就可以应用到众多的非线性模型中，增加了神经网络模型泛化的特性。\n",
    "\n",
    "早期研究神经网络主要采用sigmoid函数或者tanh函数，输出有界，很容易充当下一层的输入。 \n",
    "近些年Relu函数及其改进型（如Leaky-ReLU、P-ReLU、R-ReLU等），由于计算简单、效果好所以在多层神经网络中应用比较多。\n",
    "\n",
    "下面来总结下较常见的激活函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 初始化一些信息\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "x= torch.linspace(-10,10,60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sigmod 函数\n",
    "$a=\\frac{1}{1+e^{-z}}$ 导数 ：$a^\\prime =a(1 - a)$\n",
    "\n",
    "在sigmod函数中我们可以看到，其输出是在(0,1)这个开区间,它能够把输入的连续实值变换为0和1之间的输出，如果是非常大的负数，那么输出就是0；如果是非常大的正数输出就是1，起到了抑制的作用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2ad8aa2b9b0>]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAAD8CAYAAABAWd66AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xt8VPWd//HXN5dJCJcQwjWEWwwEEkTQQVGrQlVAdIP2oWxst9Z13fSCfbg+tlW73eVnb4/Sde3+1vJrd+PS2m67oGurZLuAoAJau4JBrgFykSBJICSREBJyn/n+/pgxDZBgApk5c3k/H488Zs4535nzyZmTd858z81YaxEREefFOF2AiIj4KJBFREKEAllEJEQokEVEQoQCWUQkRCiQRURChAJZgsoY83NjTK0x5mAf040x5nljTLkxZr8x5tpg1yjiFAWyBNuLwNJLTL8LmO7/yQd+FoSaREKCAlmCylr7NnD6Ek2WA7+yPu8BI40xE4JTnYiz4gbYXqf1yRWrqKjgnnvugV7Wp7vvvpunn346H/hPgM9+9rP86Ec/OtHb+xQUFFBQUABAa2srxcXFgSs6ArV0dPFxcwf1ze2caemksbWTs22dnG3t5GxbF01tXbR0dNHS4el+bO3w0N7lpa3T99juf+zyRn40HPneUhLjYy/35aY/jQYayCIB1dup/Mb0vi7n5+eTn58PgNvtDmhd4cbjtRw/3cLx0y1UN7RS1dBC9ZlWqhpaqW1q4+PmDlo6PH2+PjE+hmEJ8QxNiCXJFUeSK5ZhCXGMGZZAYnwsifExJMTFkhAXg8v/Ex8bgys2hrhYQ1xsDK5YQ2xMDHExhrhYQ6wxxMb4fmJi/jRsDMQa37gY4/u8DRBjDDHGN90YMPTyHN/wJ3n3p3H+Yf/v88lr6DF8Kb1NT4gLfIeCAllCSnp6OpWVld3DVVVVpKWlOVhR6Gts6eSDygZKapoorWmi5FQT5bXNtHd5u9vExRgmjExk4sghXDc5hdRhCYwelkDqMBejh7lISXKRPCSeEUPiGZ4YR0LcZW8JyhVQIEtIyc3NZc2aNeTl5bFz506Sk5OZMEFdyD2dbGxlV8Vp3j92mvcrGig51dQ9bUJyItPHDeemq1KZPm4400YPZeLIIYwbkUhsTL++NYuDFMgSVA8++CDbt2+nvr6e9PR0vvOd79DZ2QnAV77yFZYtW8bGjRvJzMwkKSmJX/ziFw5XHBqO1jWz8cBJNh6o4dDJswAMS4jj2ikp3DNnAu6po8hOG0HykHiHK5UrYQZ4+c3I77mXsOR2uykqKnK6jEFVfaaVV4qq2HTwJEdqfFvB104eydLZ47npqtHMHD+cuFgdKBUmtFNPJBwdqGrkhXeO8j8HTuK1lvlTRvF//iybpbPHMyF5iNPlSQApkEVCgNdr2V5aS8HbR3nv6GmGJcTxyM1TefjmaUwcqRCOFgpkEYcdrG7k268eYF9VIxOSE/n2sln8+fWTGJGo/uBoo0AWcUhzexc/3lLKi3+sYNTQBJ69fw73zptIvPqFo5YCWSTIrLVsPljDd/77EKea2vjCDZP55pKZOkJCFMgiwdTc3sU3Xt7H5uIasieM4Gd/cS3zJqc4XZaECAWySJBUNbTw6C+LKKtt5um7ZvLoZ6bpsDU5jwJZJAh2f9TAl/+jiPYuL794eD63zhjjdEkSghTIIgH26p4qnnrlABNGJrI+fz6ZY4c5XZKEKAWySIBYa/nnraU8/1Y5CzJG8bMvXEfKUJfTZUkIUyCLBMhPt3/I82+Vs8KdzvfvvRpXEC7fKOFNgSwSAK/sruLZ10tYPjeN1Z+bQ4yutCb9oH/ZIoNse0ktT/12P5/JHM2z91+jMJZ+UyCLDKJ9lWf42m8+IGvccH72F9eqm0IGRGuLyCA5Vn+OR158n1FDXbz4yHyG61oUMkAKZJFB0NjayZd+sQuvtfzqkesZOzzR6ZIkDGmnnsgg+E5hMVUNrbz85QVkjNFxxnJ5tIUscoU2H6zhd3uqWbkok+umjHK6HAljCmSRK1Df3M63Xz3A7Ikj+PpnM50uR8KcuixELpO1lr/73QGa2rtYt2KurmMsV0xrkMhlenVPNVsOneIbi2cwY9xwp8uRCKBAFrkMJ8608n8Ki5k/NYW/+kyG0+VIhFAgiwyQtZanfrsfj9fyTw9cQ6zOxJNBokAWGaDX9lbzTlk931o2iympQ50uRyKIAllkANq7PDy3pZSctBF84frJTpcjEUaBLDIA63Yep6qhlSeXztRFg2TQKZBF+qm5vYufvFXOjRmp3Dp9tNPlSARSIIv009p3Kvj4XAdPLs3CGG0dy+BTIIv0w8fN7RS8/SFLc8Yzb3KK0+VIhFIgi/TDmm3ltHZ6+MaSGU6XIhFMgSzyKSpPt/Cb947zwHWTyByrM/IkcBTIIp/in98oBQOP3zHd6VIkwimQRS6hpKaJV/dU8/BNU0kbOcTpciTCKZBFLuHf3v6QpPhYvnrbVU6XIlFAgSzSh7qmdn6/7yT3X5dOylCX0+VIFFAgS9Bt3ryZrKwsMjMzWb169UXTjx8/zqJFi5g3bx5z5sxh48aNDlQJ63Ydp8Pj5aGbpjoyf4k+CmQJKo/Hw8qVK9m0aROHDh1i3bp1HDp06Lw23//+91mxYgV79uxh/fr1fO1rXwt6nR1dXn793kfcNmMMV+keeRIkCmQJql27dpGZmUlGRgYul4u8vDw2bNhwXhtjDGfPngWgsbGRtLS0oNe56eBJapvaefjmqUGft0Qv3cJJgqq6uppJkyZ1D6enp7Nz587z2jzzzDMsXryYn/zkJ5w7d4433nij1/cqKCigoKAAgLq6ukGt88U/HmPa6KHcNn3MoL6vyKVoC1mCylp70bgLrwuxbt06Hn74Yaqqqti4cSNf/OIX8Xq9F70uPz+foqIiioqKGDNm8IJzb+UZ9hw/w0M3TtEV3SSoFMgSVOnp6VRWVnYPV1VVXdQlsXbtWlasWAHAjTfeSFtbG/X19UGr8Zd/PMZQVyz3X5cetHmKgAJZgmz+/PmUlZVRUVFBR0cH69evJzc397w2kydP5s033wTg8OHDtLW1DeoW8KXUNrXx+/0neMA9ieGJ8UGZp8gnFMgSVHFxcaxZs4YlS5Ywa9YsVqxYQU5ODqtWraKwsBCA5557jhdeeIFrrrmGBx98kBdffDFol7v8z53H6fRYHrpxSlDmJ9KT6a1P7xIG1FgkWNxuN0VFRVf0Hh1dXm5a/RazJ47gxb+8fpAqEwGgX1sU2kIW8dt44CT1ze08rBNBxCEKZBG/9e8fZ2pqErfqUDdxiAJZBKg+08p7R09z37x0HeomjlEgiwCFe08AcO+84J8VKPIJBbJEPWstr+6p4trJI5mSOtTpciSKKZAl6h0+2UTpqWbumzfR6VIkyimQJeq9treauBjD3XPUXSHOUiBLVPN4LYV7T7AwawyjdBF6cZgCWaLazqMfU3O2jXvVXSEhQIEsUe3VPdUMS4jjjlnjnC5FRIEs0aut08OmgzXcNXs8ifGxTpcjokCW6PXG4VM0t3fp6AoJGQpkiVqv7alm/IhEbshIdboUEUCBLFHq9LkOtpfUsXxuGrE6VVpChAJZotL/7D9Bl9fq6AoJKQpkiUob9p4ga9xwZk0Y4XQpIt0UyBJ16pra2X28gWVXT3C6FJHzKJAl6rxx+BTWwuIcHXssoUWBLFFnS3ENk0YNYeb44U6XInIeBbJEleb2Lt4t/5gl2eODduNUkf5SIEtU2V5SS4fHy+Kc8U6XInIRBbJElS3Fp0gd6uK6KSlOlyJyEQWyRI2OLi/bjtRyx6xxOhlEQpICWaLG/x79mKb2Lh1dISFLgSxRY0txDUmuWG7OHO10KSK9UiBLVPB6LVsPnWJh1hhdalNClgJZosLeqjPUNrWzOFtHV0joUiBLVNhSfIq4GMOimWOdLkWkTwpkiQpbDtVw41WpJA+Jd7oUkT4pkCXildc2cbTunE4GkZCnQJaI93rxKQDu1I1MJcQpkCXibTl0imsmjWR8cqLTpYhckgJZIlpdUzv7Ks9w5yztzJPQp0CWiLajtA5AR1dIWFAgS9Bt3ryZrKwsMjMzWb16da9tXn75ZbKzs8nJyeHzn//8Zc9rW0ktY4cnkK1bNUkYiHO6AIkuHo+HlStXsnXrVtLT05k/fz65ublkZ2d3tykrK+OHP/wh7777LikpKdTW1l7WvLo8Xt4urWPZ7Am69rGEBW0hS1Dt2rWLzMxMMjIycLlc5OXlsWHDhvPavPDCC6xcuZKUFN8lMseOvbzuhg+On6GprYuFWWOuuG6RYFAgS1BVV1czadKk7uH09HSqq6vPa1NaWkppaSk333wzCxYsYPPmzb2+V0FBAW63G7fbTV1d3UXTt5XUEhdjuHm6LiYk4UFdFhJU1tqLxl3YndDV1UVZWRnbt2+nqqqKW265hYMHDzJy5Mjz2uXn55Ofnw+A2+2+6H23HanFPTWFEYk6O0/Cg7aQJajS09OprKzsHq6qqiItLe2iNsuXLyc+Pp5p06aRlZVFWVnZgOZzsrGVIzVNLMrS0RUSPhTIElTz58+nrKyMiooKOjo6WL9+Pbm5uee1uffee9m2bRsA9fX1lJaWkpGRMaD57CjR4W4SfhTIElRxcXGsWbOGJUuWMGvWLFasWEFOTg6rVq2isLAQgCVLlpCamkp2djaLFi3i2WefJTU1dUDz2VZSy8SRQ5g+dlggfg2RgDC99eldwoAaiwSL2+2mqKgI8N07b953t3DvvIn84L6rHa5MBIB+HXepLWSJOEXHTnOuw8NC9R9LmFEgS8TZVlKLKzaGm64aWDeHiNMUyBJxtpXUcUPGKIYm6KhOCS8KZIkoladbKK9tVneFhCUFskSU7SW+614s0unSEoYUyBJRtpXUMSU1iWmjhzpdisiAKZAlYrR1evjjh/Usyhqrq7tJWFIgS8TYVXGatk4vt81Qd4WEJwWyRIwdpXW44mJYkKHD3SQ8KZAlYuworeOGaaMY4op1uhSRy6JAlojQ6fFSXtus7goJawpkiQhNbV0ACmQJawpkiQhNbV2kJSeSqau7SRhTIEvY6/R4aW7v4rasMTrcTcKaAlnC3p7jZ/Baq+4KCXsKZAl7O0p9p0vflKmbmUp4UyBL2Hu7tJ6hrljdzFTCngJZwlp9czsHqhsZpjCWCKBAlrD2TpnvZqbDde1jiQBaiyWs7SipI3WoC6Oz8yQCaAtZwpbXa3m7rJ5bdXSFRAgFsoStgycaOX2uQ4e7ScRQIEvYeru0DmPgluk63E0igwJZwtaO0jpmpyWTOizB6VJEBoUCWcJSY2snHxw/o+4KiSgKZAlL75bX4/FabtPNTCWCKJAlLG07UsuIxDjmTRrpdCkig0aBLGHHWsv20jpunTGGuFitwhI5tDZL2Ck+cZa6pnYWZY11uhSRQaVAlrCzvcR3dTf1H0ukUSBL2NlWUsc16cmM1uFuEmEUyBJWGs51sOd4A7epu0IikAJZwsrbZXV4LSxSd4VEIAWyhJXtJXWMGupiTroOd5PIo0CWsOHxWnaU1nHbjDHExuhmphJ5FMgSdJs3byYrK4vMzExWr17dZ7tXXnkFYwxFRUUA7K86w+lzHSxUd4VEKAWyBJXH42HlypVs2rSJQ4cOsW7dOg4dOnRRu6amJp5//nluuOGG7nHbSuqIMXDrdAWyRCYFsgTVrl27yMzMJCMjA5fLRV5eHhs2bLio3T/8wz/w5JNPkpiY2D1ue0kt8yankDLUFcySRYJGgSxBVV1dzaRJk7qH09PTqa6uPq/Nnj17qKys5J577ukeV9fUzv6qxvOOrigoKMDtduN2u6mrqwt88SIBpkCWoLLWXjTOmD/toPN6vTzxxBM899xz57XZUeoL3IU9jj/Oz8+nqKiIoqIixoxRN4aEP93kVIIqPT2dysrK7uGqqirS0tK6h5uamjh48CALFy4EoKamhtzcXBb+3a8YMzyBnLQRwS5ZJGi0hSxBNX/+fMrKyqioqKCjo4P169eTm5vbPT05OZn6+nqOHTvGsWPHWLBgAa++toGD9R4Wzhhz3ta0SKRRIEtQxcXFsWbNGpYsWcKsWbNYsWIFOTk5rFq1isLCwl5fU3q6k7NtXSyaqdOlJbKZ3vr0LmFAjUUGww83HmbtHyr4YNWdjEiM77WN2+3uPl5ZJAT166udtpAlpFlreb24hhuvSu0zjEUihQJZQlpZbTPHPm5hSc54p0sRCTgFsoS0LcU1ANyZPc7hSkQCT4EsIe314lPMnTSScSMSP72xSJhTIEvIOnGmlQPVjequkKihQJaQtfXQKQAW56i7QqKDAllC1uvFNWSOHcZVY4Y5XYpIUCiQJSSdaelgZ8VpFmtnnkQRBbKEpLeO1OLxWhar/1iiiAJZQtLrxTWMH5HInInJTpciEjQKZAk5rR0edpTWcWf2OGJ07zyJIgpkCTl/KK+nrdOrw90k6iiQJeS8XlzD8MQ4bsgY5XQpIkGlQJaQ0uXx8ubhU9w+cyzxsVo9JbpojZeQ8v6xBhpaOtVdIVFJgSwhZfPBk7jiYrh1hu6RJ9FHgSwho9Pj5ff7T3LnrHEMTdDtHiX6KJAlZPyhvJ6Pz3Vw77yJTpci4ggFsoSM1/ZUMzIpntvUXSFRSoEsIaG5vYvXi2u4++oJuOK0Wkp00povIWFLcQ1tnV7uU3eFRDEFsoSEV/dUk54yhOumpDhdiohjFMjiuNqzbbxbXs998yZijK5dIdFLgSyOK9x3Aq+F5XPVXSHRTYEsjnttbzVz0pPJHKs7g0h0UyCLo8prmzhYfZZ7tXUsokAWZ7225wSxMYY/uybN6VJEHKdAFsd4vZbX9lbzmczRjBme4HQ5Io5TIItjdh9voKqhVccei/gpkMUxv/ugmiRXLItzdGdpEVAgi0MaWzvZsLeau6+eQJJLV3YTAQWyOOS/iipp6fDwpZumOl2KSMhQIEvQebyWX/3vR1w/dRSzJyY7XY5IyFAgS9BtO1LL8dMtPHzzVKdLEQkpCmQJun/67yJoaeDrn7uN1atXXzT9xz/+MdnZ2cyZM4fbb7+djz76yIEqRYJPgSxBdeRkI0ca4NHbZnC4+CDr1q3j0KFD57WZN28eRUVF7N+/n/vvv58nn3zSoWpFgkuBLEH1Txvex3i7+NrSubhcLvLy8tiwYcN5bRYtWkRSUhIACxYsoKqqyolSRYJOgSxB09jSyY7jbUzoqGLUUBcA6enpVFdX9/matWvXctddd/U6raCgALfbjdvtpq6uLiA1iwSTDgCVoHm5qJJOr2FS69Hzxvd1DeRf//rXFBUVsWPHjl6n5+fnk5+fD4Db7R7cYkUcoECWoPB4Lb/832PMTI3jzJ6D3eOrqqpIS7v4wkJvvPEGP/jBD9ixYwcJCbrOhUQHdVlIULx5+BRVDa08tjiHsrIyKioq6OjoYP369eTm5p7Xds+ePXz5y1+msLCQsWPHOlSxSPApkCUofv5uBWnJiSydncaaNWtYsmQJs2bNYsWKFeTk5LBq1SoKCwsB+OY3v0lzczMPPPAAc+fOvSiwRSKVsdYOpP2AGosAvFtezxf+fSd/f/csHr0lIyDzcLvdFBUVBeS9RQZBv24WqS1kCShrLf+4+QhpyYn8xYIpTpcjEtIUyBJQmw/WsK+qkSfunEFifKzT5YiENAWyBEyXx8uzW0qYPnYYn7s23elyREKeAlkC5pXdVRytO8c3l2QRG9OvLjSRqKZAloBo6/Twf98o49rJI7kzW3cEEekPBbIExC//eIyas208tXRmn2fiicj5FMgy6BpbO/np9g9ZmDWGGzJSnS5HJGwokGXQ/duOD2ls7eTJJTOdLkUkrCiQZVAdqz/Hz9+tYPncNLLTRjhdjkhYUSDLoPF4LX/7X/twxcbwrbtmOV2OSNjR1d5k0BS8fZTdHzXwL3lzGZ+c6HQ5ImFHW8gyKI7UnOWft5ay7Orx5F5z8eU0ReTTKZDlinV0eXnipX2MGBLH9++9Woe5iVwmdVnIFXv+zTIOnzzLCw+5u2/NJCIDpy1kuSJ7jjfw0+3l3H9dus7IE7lCCmS5bOfau/jbl/cxIXkIq/4s2+lyRMKeuizksnR6vHz1Nx9w7ONz/PqvbmBEYrzTJYmEPQWyDJi1lqd+u5+3S+tY/bmruSlztNMliUQEdVnIgD37egm/+6Cav7ljOnnXT3a6HJGIoUCWAfnlH4/x0+0f8uD1k3j89ulOlyMSURTI0m+bDpzkmf8u5o5Z4/je8tk63lhkkCmQpV/eOnKKx1/ay7xJI/nJg/OIi9WqIzLYtFNPLslay9o/VPCDjYfJSRvB2i/NZ4hLNysVCQQFsvSpo8vLqg0HWf9+JXfNHs9zK64hyaVVRiRQ9NclvWo418FXfr2bnRWn+fpnM3nijhnE6EalIgGlQJaLHDpxlq/+ZjcnG9v4l7y5LJ870emSRKKCAlm6nWvv4l/eLGPtHypISXKx7q8XcN2UFKfLEokaCmQBYEtxDc8UFnOisY0Hr5/EU0tnMjJJV24TCSYFcpSrPN3Cd39/iK2HTpE1bji//fw8rpsyyumyRKKSAjlK7as8wwvvHGXjgZMkxMXyrbtm8shnphGv44tFHKNAjiJer+WtI7UUvHOUXRWnGZ4Qx1/fksFf3jxN98ATCQEK5AhnrWVfVSObDpzkfw6cpKqhlYkjh/D3d8/iz+dPYrgumykSMhTIEai9y8O+yka2FNew6WAN1WdaiYsx3Jw5mm8uyWLZ1RPUNSESghTIEaCprZPdHzXw/rHTvH+sgX2VZ2jv8uKKjeGW6aN54s4Z3DlrHMlJ2hoWCWUK5DDS1NbJ8dMtlJ1qpuRUE6U1TZTWNlF5uhWA2BjD7LQRfHHBFOZPG8WNV6XqTh4iYUSB7DBrLa2dHhpaOvm4uZ365nbqmzuob26n9mw71WdaqW5opaqhhbNtXd2vi4sxXDVmGHMnpfDn7knMm5zCvMkjw+JaE5s3b+bxxx/H4/Hw6KOP8vTTT583vb29nYceeojdu3eTmprKSy+9xNSpU50pViSIQv+vt5+stVgL9pPn4B/2j/c/91rf9J6PXmt9P17wWIvXa/F4bffzLv+w79FLp8fS5bF0erx0eLx0+n/aO720d3lp7/J0P2/p8NDa2cW5dg8tHR5aOrpobu/ibGsnZ9t8j11e2+vvNNQVy8SUIaSnJHHdlBTS/c+njxvG1NShuOLCrx/Y4/GwcuVKtm7dSnp6OvPnzyc3N5fs7D/dJHXt2rWkpKRQXl7O+vXreeqpp3jppZccrFokOIISyHc//w7ltc0Xje81huwnD3+aau2fJln/gO0xPlTFxhiS4mMZ4oolyRVLkiuOJFcsKUkupqYOZcSQOEYkxjNiSDwjh8QzelgCqcNc3Y/hsLU7ULt27SIzM5OMjAwA8vLy2LBhw3mBvGHDBp555hkA7r//fh577DGstbogvkQ8YweQasaYzcDl3tFyNFB/ma8NJNU1MFdaVwowAvjIPzwKGAYc79EmBygFOv3Ds4EjQBfnGw2M8T+PB/ZfQV2BEqmfY6BEal311tqln9ZoQIF8JYwxRdZad1BmNgCqa2CutC5jzAPAEmvto/7hLwLXW2u/3qNNsb9NlX/4Q3+bjwNVV6CoroGJ9rrCrxNSwl0VMKnHcDpwoq82xpg4IBk4HZTqRBykQJZgex+YboyZZoxxAXlA4QVtCoEv+Z/fD7xlg/VVTsRBwdxrVBDEeQ2E6hqYK6rLWttljHkMeB2IBX5urS02xnwXKLLWFgJrgf8wxpTj2zLOC3RdAaS6Biaq6wpaH7KIiFyauixEREKEAllEJEQMaiAbYx4wxhQbY7zGGPcF075ljCk3xpQYY5b08fppxpidxpgyY8xL/p0+g8r/vnv9P8eMMXv7aHfMGHPA365osOvoZX7PGGOqe9S2rI92S/3LsNwY83RvbQa5rmeNMUeMMfuNMa8aY0b20S4oy+vTfn9jTIL/My73r0tTA1VLj3lOMsZsM8Yc9q//j/fSZqExprHH57sq0HX553vJz8X4PO9fXvuNMdcGoaasHsthrzHmrDHmby5oE5TlZYz5uTGm1hhzsMe4UcaYrf4c2mqM6fXGksaYL/nblBljvtRbmwHznXI8OD/ALCAL2A64e4zPBvYBCcA04EMgtpfXvwzk+Z//K/DVwayvl/k9B6zqY9oxYHQg53/B/J4BvvEpbWL9yy4DcPmXaXaA61oMxPmf/wj4kVPLqz+/P/A14F/9z/OAl4Lw2U0ArvU/H47vpJYL61oI/D5Y61N/PxdgGbAJMMACYGeQ64sFaoApTiwv4FbgWuBgj3H/CDztf/50b+s8vhOajvofU/zPU660nkHdQrbWHrbWlvQyaTmw3lrbbq2tAMqB63s2ML7zYj8LvOIf9Uvg3sGsr5f5rQDWBWoeAXA9UG6tPWqt7QDW41u2AWOt3WKt/eQMuffwHTfslP78/svxrTvgW5duNwE+59pae9Ja+4H/eRNwGJgYyHkOouXAr6zPe8BIY8yEIM7/duBDa+1Hn9oyAKy1b3PxMe4916G+cmgJsNVae9pa2wBsBT71TLxPE6w+5IlAZY/hKi5eYVOBMz3++HtrM5huAU5Za8v6mG6BLcaY3caY/ADW0dNj/q+NP+/ja1J/lmMgPYJva6o3wVhe/fn9u9v416VGfOtWUPi7SOYBO3uZfKMxZp8xZpMxJidIJX3a5+L0OpVH3xtFTiwvgHHW2pPg+2cLjO2lTUCW24CPQzbGvAGM72XSt621G/p6WS/jLjzerj9t+qWfNT7IpbeOb7bWnjDGjAW2GmOO+P+bXrZL1QX8DPgevt/5e/i6Ux658C16ee0VH7fYn+VljPk2vmtJ/KaPtxn05dVbqb2MC9h6NFDGmGHAb4G/sdaevWDyB/i+ljf79w+8BkwPQlmf9rk4ubxcQC7wrV4mO7W8+isgy23AgWytveMy5tOf02Xr8X1divNv2fTWZlBqNL7TcT8HXHeJ9zjhf6w1xryK7+vyFQVMf5edMeYF4Pe9TOrPchz0uvw7LO4Bbrf+DrRe3mPQl1cvBnLadZUJ4mnXxph4fGGie0abAAAB8klEQVT8G2vt7y6c3jOgrbUbjTE/NcaMttYG9EI6/fhcArJO9dNdwAfW2lMXTnBqefmdMsZMsNae9Hff1PbSpgpfP/cn0vHtO7siweqyKATy/HvAp+H7T7erZwP/H/o2fKfKgu/U2b62uK/UHcAR6794zYWMMUONMcM/eY5vx9bB3toOlgv67e7rY379Oe14sOtaCjwF5FprW/poE6zlFZKnXfv7qNcCh621P+6jzfhP+rKNMdfj+9vr82JJg1RXfz6XQuAh/9EWC4DGT76uB0Gf31KdWF499FyH+sqh14HFxpgUf/fiYv+4KzPIeyzvw/efox04BbzeY9q38e0hLwHu6jF+I5Dmf56BL6jLgf8CEgazvh7zfBH4ygXj0oCNPerY5/8pxvfVPdB7e/8DOIDvEpKFwIQL6/IPL8O3F//DINVVjq+vbK//518vrCuYy6u33x/4Lr5/GACJ/nWn3L8uZQRhGX0G39fV/T2W0zLgK5+sZ8Bj/mWzD9/O0ZuCUFevn8sFdRng//mX5wF6HB0V4NqS8AVsco9xQV9e+P4hnMR3qdcq4K/w7XN4EyjzP47yt3UD/97jtY/417Ny4C8Hox6dOi0iEiJ0pp6ISIhQIIuIhAgFsohIiFAgi4iECAWyiEiIUCCLiIQIBbKISIj4/+0UO9hTJCLjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.spines['right'].set_color('none')\n",
    "ax.spines['top'].set_color('none')\n",
    "ax.xaxis.set_ticks_position('bottom')\n",
    "ax.spines['bottom'].set_position(('data', 0))\n",
    "ax.yaxis.set_ticks_position('left')\n",
    "ax.spines['left'].set_position(('data', 0))\n",
    "plt.ylim((0, 1))\n",
    "sigmod=torch.sigmoid(x)\n",
    "plt.plot(x.numpy(),sigmod.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是sigmod由于需要进行指数运算（这个对于计算机来说是比较慢，相比relu），再加上函数输出不是以0为中心的（这样会使权重更新效率降低），当输入稍微远离了坐标原点，函数的梯度就变得很小了（几乎为零）。在神经网络反向传播的过程中不利于权重的优化，这个问题叫做梯度饱和，也可以叫梯度弥散。这些不足，所以现在使用到sigmod基本很少了，基本上只有在做二元分类（0，1）时的输出层才会使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tanh 函数\n",
    "$a=\\frac{e^z-e^{-z}}{e^z+e^{-z}}$ 导数：$a^\\prime =1 - a^2$ \n",
    "\n",
    "tanh是双曲正切函数,输出区间是在(-1,1)之间，而且整个函数是以0为中心的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2ad8ab67cc0>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADxCAYAAAD8x81kAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X1YVOedN/DvgREEXxAUDDj4MsyIgIxEh2h2+6SKEjSaSZOYSGoTWmMx1T67NbtJ2KbaJBtXfJKrm7TaXMsmJphchSQ2FWoMilHbrJsExzo1QWIBQQecyMsMKsjrzP38gU5FBh1w3uf7uS4uM+fcM+d3DpOvx/vc5z6SEAJEROR5QZ4ugIiI+jGQiYi8BAOZiMhLMJCJiLwEA5mIyEswkImIvAQDmVxCkqSdkiQ1SZL09RDrJUmSfi1JUo0kSSclSZp73bocSZKqr/7kuK9qIs9iIJOrvANg6U3WLwOguvqTC+ANAJAkKQrALwHMB3AXgF9KkhTp0kqJvAQDmVxCCPFnAKabNHkAwC7R7wsAEyRJigWQBaBcCGESQpgBlOPmwU7kN2TDbM/b+shhdXV1WLFiBWDne7N8+XLk5eXlAvgdAGRkZGDbtm3njxw5gq6uLqD/rBkvvfQSwsLCbO2uV1BQgIKCAgBAZ2cnKisrXbYvI9XdZ0GDuRMG05X+P839f7Zc7sbFzl5c7urDxc5etHf3ebpUuoVv/n0pRo8KHunbJUcaDTeQiZzC3i37kiQNudye3Nxc5ObmAgA0Go1zCxyhzh4L/nLOjC/OtOLLMyboDW3osVht60cFS5gyIQwx40ZDHhmO8WEyRISNQkTYKIwNlSF0VDBCZUEYffXPUFkQRgUHIThIgixIuvpnEIKCgCBJuvoDSFL/cZLQ/2eQBEiQcOOhu9bOHocS4+pnBKJQmes7FBjI5BFyuRwGg8H2uqGhAXFxcZDL5Thy5MiA5QsXLnR/gcNgtQqUV13A20frcPysGb0WgSAJSJ0SgR/+43TMumMc5JHhiI8Kw+RxoxEUFKCJRrfEQCaP0Gq12L59O7Kzs/Hll18iIiICsbGxyMrKws9//nOYzWYAwIEDB7B161YPV2tfn8WKj78yYsfhGvztQjumRoVjzXdmYIFiIjTTIjFu9ChPl0g+hoFMLvHYY4/hyJEjaGlpgVwux4svvoje3l4AwFNPPYX77rsP+/btg1KpRHh4ON5++20AQFRUFDZt2oT09HQAwObNmxEVFeWx/bDHYhXYfdyA3x6pxdnWK1DFjMXr2WlYnhoLWTCvk9PIScOcfpMX9cgraTQa6HQ6l2+nu8+Cp9//Kz7+yojUKRHYsEiJe5MnsxuCboUX9Yicqb27D0+9exz/U9OCf1s2C7n3KIa8QEY0EgxkIgeYOnrwo7cr8PX5S3hlpRqPaOI9XRL5IQYy0S00tnXi8be+RKO5E//1g3lYkjzZ0yWRn2IgE91EfUsHsgu+QEdPH95bOx/p073rAiP5FwYy0RCEEHju9yfR2WvBB+vuRlLseE+XRH6OY3SIhrD3pBFf1pnwTFYiw5jcgoFMZEdHdx/+Y18VUuLG47G7pnq6HAoQ7LIgsmPH4RoYL3Zh+/fvRDDHGJOb8AyZ6AZ1LR1487M6PDR3CuZN40U8ch8GMtENXvpjJUJkQchbNsvTpVCAYSATXefTqgs4fLoZP1uiQsy40Z4uhwIMA5noqq5eC17aewrKmLHI+Yfpni6HAhAv6hFdVfi/9TjbegXvPTkfozhrG3kAv3VE6L8J5H2dAfNnROE7qkmeLocCFAOZCECV8TLONHdAmxbn6VIogDGQiQDsPXkewUESlqbc4elSKIAxkCngCSGw96QR/5AwERPHhnq6HApgDGQKeF81XsQ50xWsUMd6uhQKcAxkcomysjIkJiZCqVQiPz9/0PqNGzciLS0NaWlpmDlzJiZMmGBbFxwcbFun1WpdXuvek0bIgiRksbuCPIzD3sjpLBYLNmzYgPLycsjlcqSnp0Or1SI5OdnW5j//8z9t//2b3/wGJ06csL0OCwuDXq93S61CCHx80oj/o5qECeEhbtkm0VB4hkxOV1FRAaVSCYVCgZCQEGRnZ6OkpGTI9kVFRXjsscfcWOHf/eVcGxrbOrFCzdEV5HkMZHK6xsZGxMf//ZlzcrkcjY2NdtuePXsWdXV1yMjIsC3r6uqCRqPBggULsGfPHpfWuvfkeYQEByEzhY9lIs9jlwU5nRBi0LKhns5cXFyMlStXIjg42Lbs3LlziIuLw5kzZ5CRkYHU1FQkJCQMem9BQQEKCgoAAM3NzcOu02oV2PeVEd9NjMb40aOG/X4iZ+MZMjmdXC6HwWCwvW5oaEBcnP0ugeLi4kHdFdfaKhQKLFy4cED/8vVyc3Oh0+mg0+kQHR097DqP1Ztw4VI3R1eQ12Agk9Olp6ejuroadXV16OnpQXFxsd3REqdPn4bZbMbdd99tW2Y2m9Hd3Q0AaGlpwdGjRwdcDHSmvSeNCJUFYUkSuyvIO7DLgpxOJpNh+/btyMrKgsViwZo1a5CSkoLNmzdDo9HYwrmoqAjZ2dkDujOqqqqwbt06BAUFwWq1Ii8vzyWBbLEKfPK1ERmzYjAmlP8bkHeQ7PX33cSwGhO5i0ajgU6nc7j9/9a04Ptvfokd35+L5eyyINdz6Dlg7LKggPTHk0aEhwQjY1aMp0shsmEgU0A6WtOC7ygnISwk+NaNidyEgUwBp7W9G+dMV3Dn1EhPl0I0AAOZAs5fG9oAAGnxE27Rksi9GMgUcPTn2hAkAWp5hKdLIRqAgUwB54ShDTMnj+NwN/I6DGQKKFarwF8NbeyuIK/EQKaAUtfagUtdfQxk8koMZAoo+nNXL+hNZSCT92EgU0DRG9owJiQYqphxni6FaBAGMgWUvza0IVUegeAgh+5kJXIrBjIFjK5eC6qMl5AWzxtCyDsxkClgVJ6/hF6L4AU98loMZAoYekP/Bb07eUGPvBQDmQKG3tCG2IjRmDx+tKdLIbKLgUwBQ28ws7uCvBoDmQJCa3s3DKZOBjJ5NQYyBYRr/ccMZPJmDGQKCHpDG4KDJKRyhjfyYgxkcpmysjIkJiZCqVQiPz9/0Pp33nkH0dHRSEtLQ1paGt58803busLCQqhUKqhUKhQWFt52LXpDGxInj0N4CGd4I+/Fbye5hMViwYYNG1BeXg65XI709HRotdpBT5BetWoVtm/fPmCZyWTCiy++CJ1OB0mSMG/ePGi1WkRGjuyGDqtVQG9ow/1z4ka8P0TuwDNkcomKigoolUooFAqEhIQgOzsbJSUlDr13//79yMzMRFRUFCIjI5GZmYmysrIR13KmpQOXOcMb+QAGMrlEY2Mj4uPjba/lcjkaGxsHtfv9738PtVqNlStXwmAwDOu9BQUF0Gg00Gg0aG5uHrIW2w0hDGTycgxkcgkhxKBlkjRwQp/7778f9fX1OHnyJJYsWYKcnByH3wsAubm50Ol00Ol0iI6OHrIWvcGMcaEyJESPHe5uELkVA5lcQi6X2854AaChoQFxcQP7cCdOnIjQ0FAAwI9//GMcP37c4fcOh97QBnV8BII4wxt5OQYyuUR6ejqqq6tRV1eHnp4eFBcXQ6vVDmhjNBpt/11aWoqkpCQAQFZWFg4cOACz2Qyz2YwDBw4gKytrRHX0Waz427ftmB3H4W7k/TjKglxCJpNh+/btyMrKgsViwZo1a5CSkoLNmzdDo9FAq9Xi17/+NUpLSyGTyRAVFYV33nkHABAVFYVNmzYhPT0dALB582ZERUWNqI4Gcyd6LFYoY9hdQd5PstdfdxPDakzkLhqNBjqdbtDyT6su4MlCHT5a/w+YO5XzIJPHONRfxi4L8ms1Te0AwAt65BMYyOTXapvbET0uFBFhozxdCtEtMZDJr9U0tSMheoynyyByCAOZ/JYQArXNHeyuIJ/BQCa/1drRg4udvRxhQT6DgUx+ixf0yNcwkMlv1TZfDWSeIZOPYCCT36ppakd4SDBi+VBT8hEMZPJbtc0dUESP4RwW5DMYyOS3apvaoWT/MfkQBjL5pSs9fWhs6+QFPfIpDGTyS2eaOwDwgh75FgYy+aVrIyw4Bpl8CQOZ/FJtUzuCJGDaxHBPl0LkMAYy+aXa5g5MjQpHqCzY06UQOYyBTH6ppqmd3RXkcxjI5HcsVoG6Fk4qRL6HgUx+p8F8BT0WKwOZfA4DmVyirKwMiYmJUCqVyM/PH7T+V7/6FZKTk6FWq7F48WKcPXvWti44OBhpaWlIS0sb9GBUR9gmFWKXBfkYPuSUnM5isWDDhg0oLy+HXC5Heno6tFotkpOTbW3uvPNO6HQ6hIeH44033sCzzz6L999/HwAQFhYGvV4/4u3bJhXixPTkY3iGTE5XUVEBpVIJhUKBkJAQZGdno6SkZECbRYsWITy8f0jaggUL0NDQ4LTt1zS1Y9LYEEwID3HaZxK5AwOZnK6xsRHx8fG213K5HI2NjUO2f+utt7Bs2TLb666uLmg0GixYsAB79uwZ8n0FBQXQaDTQaDRobm62LedTQshXscuCnE4IMWiZJNmfce29996DTqfDn/70J9uyc+fOIS4uDmfOnEFGRgZSU1ORkJAw6L25ubnIzc0FAGg0Gtu2a5rasVwd64xdIXIrniGT08nlchgMBtvrhoYGxMXFDWp38OBBbNmyBaWlpQgNDbUtv9ZWoVBg4cKFOHHihMPbtj22iWfI5IMYyOR06enpqK6uRl1dHXp6elBcXDxotMSJEyewbt06lJaWIiYmxrbcbDaju7sbANDS0oKjR48OuBh4K7UcYUE+jF0W5HQymQzbt29HVlYWLBYL1qxZg5SUFGzevBkajQZarRbPPPMM2tvb8cgjjwAApk6ditLSUlRVVWHdunUICgqC1WpFXl7e8AL52ixvHGFBPkiy1993E8NqTOQuGo0GOp0OL/3xFIoqzqHyxSw+KYS8iUNfRnZZkF+pbW7nY5vIZzGQya/UNLVzyBv5LAYy+Y3OHgsf20Q+jYFMfuOsqf+CnoIX9MhHMZDJb9S3XAEATJ/IQCbfxEAmv3G2tf8MeSof20Q+ioFMfqO+tQNRY0IQETbK06UQjQgDmfxGfcsVPtSUfBoDmfzG2dYOzGD/MfkwBjL5BSGA8xe7MI2BTD6MgUx+oafPCgCYPoldFuS7GMjkF7otFgDgGTL5NAYy+QXbGTIv6pEPYyCTX+jus2JC+Cg+R498GgOZ/EJPn5XdFeTzGMjkF7r7rOyuIJ/HQCaf191nQa+FZ8jk+xjI5PMMpk4AvKBHvo+BTC5TVlaGxMREKJVK5OfnD1rf3d2NVatWQalUYv78+aivr7et27p1K5RKJRITE7F///6bbufapELTJ/EMmXwbA5lcwmKxYMOGDfjkk09w6tQpFBUV4dSpUwPavPXWW4iMjERNTQ02btyI5557DgBw6tQpFBcXo7KyEmVlZVi/fj0sV8cZ21Pfymk3yT8wkMklKioqoFQqoVAoEBISguzsbJSUlAxoU1JSgpycHADAypUr8emnn0IIgZKSEmRnZyM0NBQzZsyAUqlERUXFkNs629qBIElCZDhneSPfNqynTi9dulS0tLSMaEPNzc2Ijo4e0XtdiXUNj6N1mc1mXLp0CdOmTQMAtLa2oqOjA1OnTrW1qayshEqlQkhI/9jhr776CklJSTh//jzGjh2LqKgoAEB9fT0iIiIQGRk5qJaWlhZYwqPQ09aEuXNSnbWbTuPrv0d389e6jh8/vl8IsfSWDYUQw/kZsXnz5t3O212GdQ2Po3V98MEH4sknn7S93rVrl/jpT386oE1ycrIwGAy21wqFQrS0tIj169eLd99917Z8zZo1Yvfu3UNu657/d0iETpzi6C64la//Ht3Nj+tyKGPZZUEuIZfLYTAYbK8bGhoQFxc3ZJu+vj5cvHgRUVFRDr33ml6LFQ3mTkjWPhfsBZF7MZDJJdLT01FdXY26ujr09PSguLgYWq12QButVovCwkIAwO7du5GRkQFJkqDValFcXIzu7m7U1dWhuroad911l93tNJg7YbEKgIFMfkDmrg3l5ua6a1PDwrqGx9G6ZDIZtm/fjqysLFgsFqxZswYpKSnYvHkzNBoNtFotnnzySTz++ONQKpWIiopCcXExACAlJQWPPvookpOTIZPJsGPHDgQHB9vdTv3VIW+R48c5ZwedzNd/j+4W6HUN66IegGE1JnK1d47W4YU/nsKEA5uh/8txT5dDNBTJkUbssiCfVt96BWNCgiELcuj7TuTVGMjk0+pbOziHBfkNpwbyhx9+iJSUFAQFBUGn0w1Y58itsHV1dZg/fz5UKhVWrVqFnp4eZ5YHAFi1ahXS0tKQlpaG6dOnIy0tzW676dOnIzU1FWlpadBoNE6v40YvvPACpkyZYqtt3759dtvd6nZkZ3vmmWcwa9YsqNVqPPjgg2hra7Pbzl3H68b9P9t6BTOuu2X6Zrdju4rBYMCiRYuQlJSElJQUvP7664PaHDlyBBEREbbf70svveTyuoBb/16EEPinf/onKJVKqNVq/OUvf3F5TadPn7Ydh7S0NIwfPx6vvfbagDbuOl5r1qxBTEwMZs+ebVtmMpmQmZkJlUqFzMxMmM1mu+8tLCyESqWCSqWyXZy+bY6OjxMOjEM+deqU+Oabb8R3v/tdcezYMdvyyspKoVarRVdXlzhz5oxQKBSir69v0PsfeeQRUVRUJIQQYt26deK3v/3tiAb8Oerpp58WL774ot1106ZNE83NzS7d/vV++ctfildeeeWmbfr6+oRCoRC1tbWiu7tbqNVqUVlZ6dK69u/fL3p7e4UQQjz77LPi2WeftdvOHcfrxv1PnZMmFHl7xbZPqmzjRHfs2CHWrVsnhBCiqKhIPProoy6tSQghzp8/L44fPy6EEOLSpUtCpVIN+r0cPnxYLF++3OW13OhWv5ePP/5YLF26VFitVvH555+Lu+66y43V9f9OJ0+eLOrr6wcsd9fx+tOf/iSOHz8uUlJSbMueeeYZsXXrViGEEFu3brX7nW9tbRUzZswQra2twmQyiRkzZgiTyXSzTbl/HHJSUhISExMHLXfkVlghBA4dOoSVK1cCAHJycrBnzx5nljdoex988AEee+wxl23D2Ry5HdnZ7r33Xshk/YNxFixYgIaGBpdu72Zu3P/lj/wAFjFwDouhbsd2pdjYWMydOxcAMG7cOCQlJaGxsdGl23SWkpISPPHEE5AkCQsWLEBbWxuMRqPbtv/pp58iISHBdkenu91zzz22O0Kvuf47NFQO7d+/H5mZmYiKikJkZCQyMzNRVlZ22/W4pQ+5sbER8fHxttdyuXzQF7a1tRUTJkyw/c9vr40zffbZZ5g8eTJUKpXd9ZIk4d5778W8efNQUFDgsjqut337dqjVaqxZs8buP5McOY6utHPnTixbtszuOnccrxv3PySq/2aRaddNu3l9G5lMhoiICLS2trqkHnvq6+tx4sQJzJ8/f9C6zz//HHPmzMGyZctQWVnplnpu9Xvx9HequLh4yJMiTxwvALhw4QJiY2MB9P9l29TUNKiNq47bsMchL1myBN9+++2g5Vu2bMEDDzxg9z32zlAkSRp2G2fWWFRUdNOz46NHjyIuLg5NTU3IzMzErFmzcM8994yoHkfq+slPfoJNmzZBkiRs2rQJ//Iv/4KdO3cOaOfMY+RoXdeO15YtWyCTybB69Wq7n+GK43WjG/e/tbt/bPL102666hg5or29HQ8//DBee+01jB8/fsC6uXPn4uzZsxg7diz27duH733ve6iurnZ5Tbf6vXjyePX09KC0tBRbt24dtM5Tx8tRrjpuww7kgwcPDnsjjtwKO2nSJLS1taGvrw8ymeymt8vebo19fX346KOPcPz40ONWr207JiYGDz74ICoqKm47YBw9dj/+8Y+xYsWKQcuHc0uxM+sqLCzE3r178emnnw75pXPF8brRjft/1nQFMoQhZlzooDZyuXzA7diu1tvbi4cffhirV6/GQw89NGj99QF93333Yf369WhpacGkSZNcWtetfi+u+k454pNPPsHcuXMxefLkQes8dbwAYPLkyTAajYiNjYXRaERMTMygNnK5HEeOHLG9bmhowMKFC297227psnDkVlhJkrBo0SLs3r0bQH8IDHXGfbsOHjyIWbNmQS6X213f0dGBy5cv2/77wIEDA67CusL1/XZ/+MMf7G7PkduRna2srAzbtm1DaWkpwsPtP5HDXcfrxv2vamhBfGTogL8khrod25WEEHjyySeRlJSEp59+2m6bb7/91nZWVVFRAavViokTJ7q0Lkd+L1qtFrt27YIQAl988QUiIiJs/1x3tZv9K9UTx+ua679DQ+VQVlYWDhw4ALPZDLPZjAMHDiArK+v2N+7o1T/hwCiLjz76SEyZMkWEhISImJgYce+999rWvfzyy0KhUIiZM2eKffv22ZYvW7ZMNDY2CiGEqK2tFenp6SIhIUGsXLlSdHV13WqTI5KTkyPeeOONAcsaGxvFsmXLbHWo1WqhVqtFcnKyePnll11Sx/V+8IMfiNmzZ4vU1FRx//33i/Pnzw+qS4j+q+IqlUooFAq31JWQkCDkcrmYM2eOmDNnjm0Eg6eO1/X7P+ffdot1u3Ri06ZNIiEhQQghRGdnp1i5cqVISEgQ6enpora21mW1XPPZZ58JACI1NdV2nD7++GPxxhtv2L5nv/nNb0RycrJQq9Vi/vz54ujRoy6va6jfy/V1Wa1WsX79eqFQKMTs2bMHjI5ypY6ODhEVFSXa2tpsyzxxvLKzs8Udd9whZDKZmDJlinjzzTdFS0uLyMjIEEqlUmRkZIjW1lYhhBDHjh0bMIPhW2+9JRISEkRCQoLYuXPnrTblUMby1mnySRarQNKmMvzoO9Pxb8uSoNFoBo19J/IivHWa/JfxYid6LFY+ton8CgOZfFJdS/8sb9P4pGnyIwxk8km1Te0AAGXMWA9XQuQ8DGTySTXN7Rg3WobosaG3bkzkIxjI5JNqmzqgjBnrtpsYiNyBgUw+qba5HQnR7K4g/8JAJp9zqasXTZe7GcjkdxjI5HSOzCer1+tx9913IyUlBWq1Gu+//75t3Q9/+EPMmDHDNheuXq8f8F5e0CN/xUAmp8vPz8fixYtRXV2NxYsX251IPzw8HLt27UJlZSXKysrws5/9bMDk96+88gr0ej30ev2ghwjUNvcPeUuI5hhk8i8MZHI6R+aTnTlzpm3q07i4OMTExKC5udmhz69paseoYAlTozgGmfwLA5mczpH5ZK9XUVGBnp4eJCQk2JY9//zzUKvV2LhxI7q7uwe0r21ux/SJY7DzrTeh0Wig0WgcDnMib8ZAphFZsmQJZs+ePehnuE8wMRqNePzxx/H2228jKKj/67h161Z88803OHbsGEwmE7Zt2zbgPddGWOTm5kKn00Gn0yE6Otpp+0bkKcOeD5kIuPkcyo7MJwsAly5dwvLly/Hyyy9jwYIFtuXXzq5DQ0Pxox/9CK+++qptXU+fFWdbr+C+2e6ZIpLInXiGTE7nyHyyPT09ePDBB/HEE0/gkUceGbDu2tzQQgjs2bNnwBy+50wdsFgFEmJ4QY/8DwOZnC4vLw/l5eVQqVQoLy9HXl4eAECn02Ht2rUAgA8++AB//vOf8c477wwa3rZ69WqkpqYiNTUVLS0t+MUvfmH77JprQ96ix7l5r4hcj/Mhk0/ZcbgGr+w/jcoXszAm9O89bpwPmbwc50Mm/1Pb1I7YiNEDwpjIXzCQyafUNLfzDj3yWwxk8hlCCNQ2cVIh8l8MZPIZFy51o6PHwlumyW8xkMlnXBthkcAuC/JTDGTyGbXN14a8MZDJPzGQyWfUXnts0zg+ton8EwOZfEbN1Qt6fGwT+SsGMvkMPraJ/B0DmXzC5a5eXLjUzTHI5NcYyOQT+JQQCgQMZPIJfI4eBQIGMvmEmub+xzbF87FN5McYyOQTapvaMW3iGIwK5leW/Be/3eQTapvbeUMI+T0GMnm9Xkv/Y5v4lBDydwxk8npnW6+gzyo4Bpn8HgOZvJ5tDguOsCA/x0AmpzOZTMjMzIRKpUJmZibMZrPddsHBwbbn6Wm1Wtvyuro6zJ8/HyqVCqtWrcLJcyYESQxk8n8MZHK6/Px8LF68GNXV1Vi8eDHy8/PttgsLC4Ner4der0dpaalt+XPPPYeNGzeiuroakZGR+PjLU0i8YzzCQ/jYJvJvDGRyupKSEuTk5AAAcnJysGfPHoffK4TAoUOHsHLlSgDA4088gXPtEtLiI1xSK5E3YSCT0124cAGxsbEAgNjYWDQ1Ndlt19XVBY1GgwULFthCu7W1FRMmTIBMdvVseGw0rLLRSIufMOj9BQUF0Gg00Gg0aG5uds3OELkR/w1II7JkyRJ8++23g5Zv2bLF4c84d+4c4uLicObMGWRkZCA1NRXjx48f0OZUUycAIC0+ctD7c3NzkZubCwDQaDTDKZ/IKzGQaUQOHjw45LrJkyfDaDQiNjYWRqMRMTExdtvFxcUBABQKBRYuXIgTJ07g4YcfRltbG/r6+iCTyXCstgmSpYcX9CggsMuCnE6r1aKwsBAAUFhYiAceeGBQG7PZjO7ubgBAS0sLjh49iuTkZEiShEWLFmH37t0AgC+qLyA+3IrgIE5KT/6PgUxOl5eXh/LycqhUKpSXlyMvLw8AoNPpsHbtWgBAVVUVNBoN5syZg0WLFiEvLw/JyckAgG3btuFXv/oVEmbOwuXgcViaPstj+0LkTpIQYjjth9WY6HYcP2vGw2/8L/7r8XnISrnjpm01Gg10Op2bKiMaNof+icczZPJaekMbAOBOOyMsiPwRA5m8lt7QhriI0YgZP9rTpRC5BQOZvJbeYEbaVJ4dU+BgIJNXam3vhsHUafeGECJ/xUAmr3St/9jeDSFE/oqBTF5Jb2hDcJCE1Cmcw4ICBwOZvJLe0IbEyeMQFhLs6VKI3IaBTF7HahXQG9p4QY8CDgOZvM6Zlg5c7urjBT0KOAxk8jp/v6DHQKbAwkAmr6M3mDE2VMaHmlLAYSCT19Eb2qCWR3CGNwo4DGTyKl29FnxjvMzuCgpIDGTyKl83XkSfVTAv+k0XAAAGu0lEQVSQKSAxkMmr2C7occgbBSAGMnmVQ980QRE9BjHjOMMbBR4GMnmN5svd+OJMK1akxnq6FCKPYCCT1/jkayOsAlgxJ87TpRB5BAOZnM5kMiEzMxMqlQqZmZkwm82D2hw+fBhpaWm2n9GjR+OdQ19h5uSx+I/n/i9mzJhhW6fX6z2wF0Tux0Amp8vPz8fixYtRXV2NxYsXIz8/f1CbRYsWQa/XQ6/X49ChQxgzaQrqLgdheWr/2fErr7xiW5+WlubuXSDyCAYyOV1JSQlycnIAADk5OdizZ89N2+/evRvq5TkQAFbMYf8xBS4GMjndhQsXEBvbH6yxsbFoamq6afvi4mKIqXORFDvedrv0888/D7VajY0bN6K7u9vlNRN5AwYyjciSJUswe/bsQT8lJSXD+hyj0Yiv686j/rKEFer+EN+6dSu++eYbHDt2DCaTCdu2bbP73oKCAmg0Gmg0GjQ3N9/2PhF5miSEGE77YTWmwJSYmIgjR44gNjYWRqMRCxcuxOnTp+22ff3117G3thvV4Sn48zOLMHVi+ID1R44cwauvvoq9e/fedJsajQY6nc5p+0DkZA5NzMIzZHI6rVaLwsJCAEBhYSEeeOCBIdsWFRWh945UqOURtjA2Go0AACEE9uzZg9mzZ7u+aCIvwEAmp8vLy0N5eTlUKhXKy8uRl5cHANDpdFi7dq2tXX19PQzmLtRfstq6KwBg9erVSE1NRWpqKlpaWvCLX/zC7ftA5AnssiCP2nG4Bq/sP42jeRmYMiFsxJ/DLgvycuyyIO+396QRc6dOuK0wJvIXDGTymNrmdlQZL2GFmrdKEwEMZPKgvX81QpKA5WreDEIEMJDJQy529mLX5/X4x4RJmDyeU20SAQxk8pDXDv4Npis9yFs2y9OlEHkNBjK53elvL2PX52fx/bumYvaUCE+XQ+Q1GMjkVkII/LL0a4wbLcO/3pvo6XKIvAoDmdxq70kjvjhjwr/em4jIMSGeLofIqzCQyW06uvuw5eMqpMSNx2N3TfV0OUReR+bpAihw7Dhcg28vdWHH6jsRHOTQjUtEAYVnyOQWdS0d+O/PzuChuVMwb1qUp8sh8koMZHK5rl4Lfv7RVwiVBXOYG9FNMJDJpS519SJnZwU+P9OKzfcnI2YcbwIhGgr7kMllmi93I2dnBf524TJez07DA2lTPF0SkVdjIJNLnGu9gsd3fommS914M0eDhYkxni6JyOsxkMnpqoyX8MTOCvT0WfHe2vmYNy3S0yUR+QQGMjlN06Uu/PdnZ/DeF+cQETYKHz51N2ZOHufpsoh8BgOZbpvBdAX/9edafKBrQJ/FCu2cODy7dBbiOOk80bAwkGlEWtq7UVFnwsFTF1D61/OQJGDlPDme+m4Cpk0c4+nyiHwSA5lu6XJXLwymTtQ0t6OirhVfnjGhuqkdADAmJBiP3z0NufcoEBvRf0b84Ycf4oUXXkBVVRUqKiqg0Wjsfm5ZWRn++Z//GRaLBWvXrrU9DLWurg7Z2dkwmUyYO3cu3n33XYSEcN4L8n98yKmfEELAYhWwCsAqBIQA+qxWWKwCfVaBPotAn9WKnj4ruvus6Oq12P680mPBxc5eXOrsxcWrP6aOHhjMV2AwdeJiZ69tO2NCgqGZHoX5iigsUExE6pQIjAoeOJy9qqoKQUFBWLduHV599VW7gWyxWDBz5kyUl5dDLpcjPT0dRUVFSE5OxqOPPoqHHnoI2dnZeOqppzBnzhz85Cc/uen+8yGn5OUcmivALWfI2u3/g9qrZ1T+ytG/qYb6+0/Y+YRrbcV1GxDoD1uB/hC2OvmvyOAgCRFhozAhfBTiI8MxRz4B8VHhkEeGYfrEMZh1xzjIgm9+P1FSUtItt1NRUQGlUgmFQgEAyM7ORklJCZKSknDo0CH87ne/AwDk5OTghRdeuGUgE/mD4Z4hEzlMkqQjAP5VCDHo1FWSpJUAlgoh1l59/TiA+QBeAPCFEEJ5dXk8gE+EELPtfEYugNyrL0fba0PkS9iHTCMiSdJBAHfYWfW8EKLEkY+ws0zcZPnghUIUAChwYFtEPoGBTCMihFhymx/RACD+utdyAOcBtACYIEmSTAjRd91yIr/HyYXIU44BUEmSNEOSpBAA2QBKRX8f2mEAK6+2ywHgyBk3kc9jIJPTSZL0oCRJDQDuBvCxJEn7ry6PkyRpHwBcPfv9KYD9AKoAfCCEqLz6Ec8BeFqSpBoAEwG85e59IPIEXtQjIvISPEMmIvISDGQiIi/BQCYi8hIMZCIiL8FAJiLyEgxkIiIvwUAmIvIS/x9ZVrp4cyCNdgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.spines['right'].set_color('none')\n",
    "ax.spines['top'].set_color('none')\n",
    "ax.xaxis.set_ticks_position('bottom')\n",
    "ax.spines['bottom'].set_position(('data', 0))\n",
    "ax.yaxis.set_ticks_position('left')\n",
    "ax.spines['left'].set_position(('data', 0))\n",
    "plt.ylim((-1, 1))\n",
    "tanh=torch.tanh(x)\n",
    "plt.plot(x.numpy(),tanh.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与sigmoid函数类似，当输入稍微远离了坐标原点，梯度还是会很小，但是好在tanh是以0为中心点，如果使用tanh作为激活函数，还能起到归一化（均值为0）的效果。\n",
    "\n",
    "一般二分类问题中，隐藏层用tanh函数，输出层用sigmod函数，但是随着Relu的出现所有的隐藏层基本上都使用relu来作为激活函数了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ReLU 函数\n",
    "Relu（Rectified Linear Units）修正线性单元\n",
    "\n",
    "$a=max(0,z)$ 导数大于0时1，小于0时0。\n",
    "\n",
    "也就是说：\n",
    "z>0时，梯度始终为1，从而提高神经网络基于梯度算法的运算速度。然而当\n",
    "z<0时，梯度一直为0。\n",
    "ReLU函数只有线性关系（只需要判断输入是否大于0）不管是前向传播还是反向传播，都比sigmod和tanh要快很多，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2ad8b097470>]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADvCAYAAADFG66PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAH15JREFUeJzt3XlclPW+B/DPKFKuKAqI4MKm7CIMoJ0W9QoucTCVElJRwTA93RZNT917PC1XM+t0cyuNEkUqyGzBo4YLpplaMOCKuSEubMoqi2zD/O4fJFcFdUBmnoeZz/v16lUMv3E+PDN9/PHwfBmFEAJERCS9DlIHICKiBixkIiKZYCETEckEC5mISCZYyEREMsFCJiKSCRYySUqhUMQoFIrrCoXi1G23mSsUij0KheL8n//uJWVGIn1hIZPUNgEYd9dtbwBIFkI4AUj+82Mig6fgYAhJTaFQDAKwXQjh/ufHZwGMFELkKRQKawD7hRBDJIxIpBfcIZMcWQkh8gDgz39bSpyHSC9MWrie22lqc1lZWQgKCgL+fH2ZmZk1/jcA9OzZ846Pb4mOjkZ0dDQAoKqqChkZGXpIS8Zk58k8/O3rdIweYonPZvjApGOr97AKbRZxh0yyY2Vlhby8PABAXl4eLC2b3yBHRUVBpVJBpVKhc+fO+oxIRuDwhUK8mnAMPgN6Ye3z3g9TxlpjIZPsBAcHIzY2FgAQGxuLiRMnSpyIjM2pnBuIikvDoD5dsGGmLzqbdtTL47KQSVJhYWEYMWIEzp49C1tbW2zYsAFvvPEG9uzZAycnJ+zZswdvvMGLLEh/LhVWYtbGFJh17oTNEf4w69JJb4/d0qsseA6ZZEmpVEKlUkkdg9q56+XVCFl3BOXVdfj2xcfgaNmtrf5orc4ht/SHekREBqmsug6zYlJRUF6D+KjhbVnGWuMpCyIyetV19XghVoVz18qxbro3vPr3lCQHd8hEZNTqNQKvJhzD71nFWDnVCyOHSHfZO3fIRGS0hBD4x4+nkJSRj38GueKZYTaS5mEhE5HR+njPOcSnXMH8kQ6IeNxO6jgsZCIyTpuPXMLqfRcwVdkfi8bK41elsJCJyOhsP5GLt7ZlIMDVCssmuUOh0OqqNJ1jIRORUfn1fCFe++YYfAeaY03YML2MRGtLPkmIiHTsRHYp5sap4GDRDZ/PVOLRTvoZidYWC5mIjEJWYSVmb0xFzy6miI3wg1ln/Y1Ea4uFTEQG71pZNWZs+B0CQFykH6x6PCp1pGaxkInIoN2oqsPMmBQUV9Zi4yxf2FvofyRaWyxkIjJYt0aiMwsq8NkMHwyVaCRaWxydJiKDpK7X4OX4o0i9XIxVocPwhJOF1JEeiDtkIjI4t0aid5++hreCXBE8tJ/UkbTCQiYig/PR7nNISL2Kl0Y5YtZfpB+J1hYLmYgMyqZDWVj78wWE+fXHwsDBUsdpERYyERmMfx/PxTvbTyPQ1Qr/M1E+I9HaYiETkUE4eL4AC7Y0jESvltlItLbaX2Iiorscv1qKuXFpsh2J1hYLmYjatcyCCszelArzrqbYLNORaG2xkImo3bpWVo3wDSlQAIiL9IelTEeitcVCJqJ26cbNOoRvSEHpzVpsmu0Huz5dpY700DipR0TtTnVdPeZsTsXFwgpsnOUHD1szqSO1CRYyEbUr6noNXvr6KFSXS7AmbBged+ojdaQ2w1MWJEsff/wx3Nzc4O7ujrCwMFRXV0sdiWRACIH/+uEk9v5xDe8EuyHIs32MRGuLhUyyk5OTg9WrV0OlUuHUqVOor69HQkKC1LFIBj7cdRZbVNl4ebQjwkcMkjpOm2Mhkyyp1WpUVVVBrVbj5s2b6NfPsHZC1HIxv2bh0/2ZCPMbgNcC2tdItLZYyCQ7NjY2eP311zFgwABYW1vDzMwMgYGBTdZFR0dDqVRCqVSioKBAgqSkL4nHcvDu9tMY62aFpc+0v5FobbGQSXZKSkqQmJiIrKws5ObmorKyEl9++WWTdVFRUVCpVFCpVLCwkP/vuqXWOXCuAAu3HIe/nTlWhQ5Dxw6GWcYAC5lkaO/evbCzs4OFhQU6deqEyZMn4/Dhw1LHIgkcu1qKeV+mwcmqe7seidYWC5lkZ8CAAfjtt99w8+ZNCCGQnJwMFxcXqWORnmUWVGD2xhT07maK2Nm+6PFo+x2J1hYLmWTH398fISEh8Pb2hoeHBzQaDaKioqSORXqUf6NhJLpjBwXiItr/SLS2FEKIlqxv0WIifVEqlVCpVFLHoDZw42YdnvvsCHJKq5AQNRzuNgYxhafViW/ukIlINqpq6xEZm4qswkpEz/AxlDLWGkeniUgWGkai05F2pQRrw7zxmKPhjERriztkIpKcEAJvfn8SyWeu492J7nja01rqSJJgIROR5FYkncW3adl45T+cMGP4QKnjSIaFTESS+uLgRaw/kIlp/gPw6hgnqeNIioVMRJL58WgOlu74AxM8+uLddvgu0W2NhUxEkvj57HW8/u1xjLDvjY+nehn0SLS2WMhEpHdHr5Rg/pfpGNK3O6LDffCIiWGPRGuLhUxEenXhejlmb0qFZY9HsGm2H7obwUi0tljIRKQ3eTeqEL4hBSYdOiAuwh8W3R+ROpKssJCJSC9Kb9YifEMKyqrViI3wxYDeXaSOJDssZCLSuYaRaBUuF91EdLgP3PoZ10i0tjg6TUQ6VVevwd++Tkf6lRJ8+rw3HnMwvpFobXGHTEQ6I4TAG9+dxL4z17H0GXeM9zDOkWhtsZCJSGfe/+kMvkvPxmtjBmOav/GORGuLhUxEOvH5Lxfx2S8XET5iIF7+D0ep47QLLGQianPfpWVj2c4/8LSHNd76q5vRj0Rri4VMRG3q5zPX8ffvTuAvjr3xv1OHciS6BVjIRNRm0i6XYN5XaXC27o710zkS3VIsZCJqE+evlSNiUyr69niUI9GtxEImooeWW1qF8JgUmJp0QFykP/p040h0a7CQieihlFTWYsaG31FRrUbsbD/0N+dIdGtxUo+IWu1mrRqzN6XiakkVNkf4wbVfD6kjtWvcIRNRq9TVazD/q3ScyC7F6tBhGG7fW+pI7R4LmWSptLQUISEhcHZ2houLC44cOSJ1JLqNRiOweOsJ7D9bgGWTPDDOva/UkQwCT1mQLL3yyisYN24ctm7ditraWty8eVPqSHSb5T/9gR+O5mBhwGCE+Q2QOo7BYCGT7JSVleGXX37Bpk2bAACmpqYwNTWVNhQ1+uxAJj4/mIWZIwbipdEciW5LPGVBsnPx4kVYWFhg9uzZGDZsGObMmYPKykqpYxGArWnZWP7TGQR5ciRaF1jIJDtqtRrp6emYN28ejh49iq5du+L9999vsi46OhpKpRJKpRIFBQUSJDUu+85cw9+/O4HHHfvgo+eGogNHotscC5lkx9bWFra2tvD39wcAhISEID09vcm6qKgoqFQqqFQqWFhY6DumUUm7XIz5X6XD1boH1s/gSLSusJBJdvr27Yv+/fvj7NmzAIDk5GS4urpKnMp4nbtWjohNKlibdcam2b7o9gh/9KQrPLIkS2vWrMG0adNQW1sLe3t7bNy4UepIRimntOFdoh8x6YDNEX7ozZFonWIhkyx5eXlBpVJJHcOoFf85El1Zq8aWuSM4Eq0HLGQiaqKypmEkOrukCnERfnCx5ki0PvAcMhHdoVatwbyv0nEyuxRrw4bBnyPResMdMhE10mgEFm09jl/OFeD9yR4IdONItD5xh0xEAAAhBJbu+AOJx3KxaOwQhHIkWu9YyEQEAFh3IBMxh7Iw67FBmD/SQeo4RomFTETYorqKD5LOInhoP/wzyJUj0RJhIRMZub2nr+HN70/iCac++NezHImWEguZyIilXirG375Oh3u/Hlg/3QemJqwEKfHoExmpM/lliNyUCpuenREzyxddORItORYykRHKLrmJmTEp6GzaEZsjORItFyxkIiNTVFGD8A0pqKqtx+YIf9j24ki0XPB7FCIjUlmjRsSmVOSUVuHLOf4Y0re71JHoNixkIiNRq9bgxS/TcCq3DOun+8B3kLnUkeguPGVBZAQ0GoHXvz2Og+cLsXySBwJcraSORM1gIRMZOCEE3t1+GtuO52LxuCF4zre/1JHoHljIRAbu0/2Z2HT4EiL+Yod5T3EkWs5YyEQG7JvUK/hw11k849UP/3jahSPRMsdCJjJQuzPy8eb3J/HkYAt8EMKR6PaAhUxkgFKyivGf8UfhYdsT66Z5cyS6neCzRGRg/sgrQ2RsKmx6dcZGjkS3KyxkIgNytbhhJLqrqQniIv1h3tVU6kjUAvyrk8hAFFbUIDwmBTVqDb59cQRsenaWOhK1EHfIRAagokaN2RtTkXejCjGzlBhsxZHo9og7ZKJ2rkZdjxfj0nA6rwyfh/vAZyBHotsr7pCJ2jGNRmDhluP49UIhVkzxxGhnjkS3Zyxkkq36+noMGzYMQUFBUkeRJSEE3vl3BrafyMOb450R4mMrdSR6SCxkkq1Vq1bBxcVF6hiytXbfBcQeuYwXnrDDXI5EGwQWMslSdnY2duzYgTlz5kgdRZbiU67goz3nMHmYDd4cz7+0DAULmWTp1VdfxQcffIAOHe79Eo2OjoZSqYRSqURBQYEe00kr6VQ+/vuHkxg5xAIrQjw5Em1AWMgkO9u3b4elpSV8fHzuuy4qKgoqlQoqlQoWFhZ6Siet3y8W4eWEoxjavyc+neaNTh35v7Ah4bNJsnPo0CFs27YNgwYNQmhoKPbt24fp06dLHUtyp3PLMCdWhf69OiNmpi+6mPKqVUOjEEK0ZH2LFhM9rP379+Nf//oXtm/fft91SqUSKpVKT6n070rRTUxZfxgmHRT4bt5j6McpvPZGq/NK3CETyVzDSPTvqKvXYHOEH8vYgHGHTAbBUHfI5dV1CPv8N1y4XoGv5gyHz8BeUkei1tFqh8yTUEQyVaOux9y4NPyRV44vwpUsYyPAUxZEMlSvEVjwzXEczizChyGeGOVsKXUk0gMWMpHMCCHw9rYM7DiZh/+e4ILJ3hyJNhYsZCKZWZ18AXG/XcbcJ+3xwpP2UschPWIhE8nIV79fxsd7z2Gytw3+Ps5Z6jikZyxkIplIOpWHJT+ewmhnS6yYwpFoY8RCJpKBI5lFeDn+GLz698Qnz3Mk2ljxWSeSWEbuDURtVmFg7y6ImeWLzqYdpY5EEmEhE0noclElZsakovujJtgc6YeeXfgu0caMhUwkkYLyhneJVms02BzpB2szjkQbOxYykQTKq+swa2MKrpfVYOMsXzha8l2iiYVMpHfVdfWI2pyGs/nlWDfdG8MGcCSaGvB3WRDpUb1G4LVvjuHIxSKsnOqFkUM4Ek3/jztkIj0RQuCfiafw06l8/ONpFzwzzEbqSCQzLGQiPVm59zy++v0KXnzKAXOe4Eg0NcVCJtKDuN8uY1XyeTzrY4u/jxsidRySKRYykY7tOJGHfyaewhgXSyyf7AGFgiPR1DwWMpEOHb5QiNe+OQafAb2wJswbJhyJpvvgq4NIR07l3EBUXBrs+nTFhpkciaYHYyET6cClwkrM2pgCs86dEBvhB7MunaSORO0AC5mojV0vr0Z4TArqNQKbI/3Q1+xRqSNRO8HBEKI2VFZdh5kxqSisqMHXLwyHg0U3qSNRO8IdMlEbqa6rxwuxKpy/Vo71033g1b+n1JGoneEOmagN1GsEXk04ht+zirEq1AtPDraQOhK1Q9whk+xcvXoVo0aNgouLC9zc3LBq1SqpI92XEAL/+PEUkjLysSTIFRO9OBJNrcMdMsmOiYkJPvroI3h7e6O8vBw+Pj4ICAiAq6ur1NGa9fGec4hPuYJ5Ix0Q+bid1HGoHeMOmWTH2toa3t7eAIDu3bvDxcUFOTk5EqdqXuzhS1i97wKeU9pi8ViORNPDYSGTrF26dAlHjx6Fv7+/1FGa2H4iF2//OwNjXKzw3iSORNPD4ykLkq2KigpMmTIFK1euRI8ePZp8Pjo6GtHR0QCAgoICvWb79XzDSLRyYC+sfX4YR6KpTSiEEC1Z36LFRK1VV1eHoKAgjB07FgsWLHjgeqVSCZVKpYdkwMnsGwiNPoL+5l3wzdwRMOvMKTx6IK2+feJf6yQ7QghERkbCxcVFqzLWp6w/R6J7djFtGIlmGVMbYiGT7Bw6dAhxcXHYt28fvLy84OXlhZ07d0odC9fKqjFjw+8QAOIi/WDVgyPR1LZ4Dplk5/HHH0cLT6Xp3I2qOsyMSUFxZS0SoobDniPRpAPcIRM9wK2R6MyCCnw2wweethyJJt3gDpnoPtT1Gvxn/FGkXi7GqtBheMKJI9GkO9whE93DrZHoPaev4a0gVwQP7Sd1JDJwLGSie/ho9zkkpF7FS6McMesvHIkm3WMhEzVj06EsrP35AsL8+mNh4GCp45CRYCET3WXb8Vy8s/00Al2t8D8T3TkSTXrDQia6zcHzBVi45Rh8B5ljdRhHokm/+Goj+tOJ7FLMjUuDg0U3fB6uxKOd+C7RpF8sZCIAFwsqMGtjKsy7mmIzR6JJIixkMnoNI9EpUACIi/SHJUeiSSIsZDJqt0aiS2/WYtNsP9j16Sp1JDJinNQjo1VdV485sanILKjAxll+8LA1kzoSGTkWMhkldb0GL32dDtXlEqwJG4bHnfpIHYmIpyzI+Agh8F8/nMTeP67jnWA3BHlyJJrkgYVMRufDXWexRZWNl0c7InzEIKnjEDViIZNRifk1C5/uz0SY3wC8FsCRaJIXFjIZjcRjOXh3+2mMc+uLpc9wJJrkh4VMRuGXcwVYuOU4htubY2WoFzp2YBmT/LCQyeAdu1qKF79Mg5NVd0RzJJpkjIVMBi2zoAIRm1LRu5spYmf7osejHIkm+WIhk8HKv1GN8A0p6KAA4iI4Ek3yx8EQMkg3bjaMRN+oqkNC1HAM4kg0tQPcIZPBqaqtR2RsKrIKKxE9wwfuNhyJpvaBO2QyKLdGotOulOCT573xmCNHoqn94A6ZZCkpKQlDhgyBo6Mj3n//fa3uU11Xj8XfnUDymet4d6I7JnhY6zglUdtSCCFasr5Fi4lao76+HoMHD8aePXtga2sLX19fxMfHw9XVtfn1GoHB7kNhOeNj5N2oxmtjBuOVMU56Tk10X1pd+M4dMslOSkoKHB0dYW9vD1NTU4SGhiIxMbHZtQfOFeDp1QeRXVIFyx6P4puo4Sxjarf0skP+IOkM8suqW3NXMkKXL19Gbk4uRjw2AgBwMfMiCgsL4Ofvf8e67JIqpGQVo795Z+THvoZzp45xHJrkSqsXZosKedy4caKwsLDFSbIKK1FVU4cOHeU3IaWpr2euFtBHLo1GA41GAxMTkz8/rofQCHQ0ufNn0EJTD01VGTrUVqKmpgZeXl46zdUaBQUFsLCwkDpGE8zVMg+bKy0tbZcQYtwDFwohWvJPq/n4+DzM3XWGuVpGH7kOHz4sAgMDGz9+7733xHvvvXff+3Tp0kXXsVrFmJ/H1jDgXFp1LM8hk+z4+vri/PnzyMrKQm1tLRISEhAcHCx1LCKd43XIJDsmJiZYu3Ytxo4di/r6ekRERMDNzU3qWEQ6p7dCjoqK0tdDtQhztYy+ck2YMAETJkzQen2fPvIcADH257GljD0Xr0Mmg6BUKqFSqaSOQXQvvA6ZiKg9adNC/vbbb+Hm5oYOHTo02a0sX74cjo6OGDJkCHbt2tXs/bOysuDv7w8nJydMnToVtbW1bRkPADB16lR4eXnBy8sLgwYNuuelUoMGDYKHhwe8vLygVCrbPMfd3n77bdjY2DRm27lzZ7PrWjNS/DAWLVoEZ2dneHp6YtKkSSgtLW12nb6O14O+/pqaGkydOhWOjo7w9/fHpUuXdJbllqtXr2LUqFFwcXGBm5sbVq1a1WTN/v37YWZm1vj8vvvuuzrPBTz4eRFC4OWXX4ajoyM8PT2Rnp6u80xnz55tPA5eXl7o0aMHVq5ceccafR2viIgIWFpawt3dvfG24uJiBAQEwMnJCQEBASgpKWn2vrGxsXBycoKTkxNiY2PbJpC2l2MILS57O336tDhz5ox46qmnRGpqauPtGRkZwtPTU1RXV4uLFy8Ke3t7oVarm9z/2WefFfHx8UIIIebOnSs+/fTTVl1foq0FCxaId955p9nPDRw4UBQUFOj08W/31ltviQ8//PC+a9RqtbC3txeZmZmipqZGeHp6ioyMDJ3m2rVrl6irqxNCCLF48WKxePHiZtfp43jd7+u/dVnSJ598IubOnSuEECI+Pl4899xzOs0khBC5ubkiLS1NCCFEWVmZcHJyavK8/Pzzz+Lpp5/WeZa7Peh52bFjhxg3bpzQaDTiyJEjws/PT4/pGp5TKysrcenSpTtu19fxOnDggEhLSxNubm6Nty1atEgsX75cCCHE8uXLm33NFxUVCTs7O1FUVCSKi4uFnZ2dKC4uvt9D6f+yNxcXFwwZMqTJ7YmJiQgNDcUjjzwCOzs7ODo6IiUlpclfDPv27UNISAgAYObMmfjxxx/bMl6Tx9uyZQvCwsJ09hhtrSUjxW0lMDCwcUBj+PDhyM7O1unj3Y82X39iYiJmzpwJAAgJCUFycjJEy35O0mLW1tbw9vYGAHTv3h0uLi7IycnR6WO2lcTERISHh0OhUGD48OEoLS1FXl6e3h4/OTkZDg4OGDhwoN4e83ZPPvkkzM3N77jt9tfQvXpo165dCAgIgLm5OXr16oWAgAAkJSU9dB69nEPOyclB//79Gz+2tbVt8oItKipCz549G//nb25NWzp48CCsrKzg5NT87z1QKBQIDAyEj48PoqOjdZbjdmvXroWnpyciIiKa/TZJm+OoSzExMRg/fnyzn9PH8dLm6799jYmJCczMzFBUVKSTPM25dOkSjh49Cv+7xrwB4MiRIxg6dCjGjx+PjIwMveR50PMi9WsqISHhnpsiKY4XAFy7dg3W1g2/KdDa2hrXr19vskZXx63Fl72NGTMG+fn5TW5ftmwZJk6c2Ox9mtuh3P07B7RZ05YZ4+Pj77s7PnToEPr164fr168jICAAzs7OePLJJ1uVR5tc8+bNw5IlS6BQKLBkyRIsXLgQMTExd6xry2Okba5bx2vZsmUwMTHBtGnTmv0zdHG87qbv11FLVVRUYMqUKVi5ciV69Ohxx+e8vb1x+fJldOvWDTt37sQzzzyD8+fP6zzTg54XKY9XbW0ttm3bhuXLlzf5nFTHS1u6Om4tLuS9e/e2+EFsbW1x9erVxo+zs7PRr1+/O9b06dMHpaWlUKvVMDExaXZNW2VUq9X4/vvvkZaWds81tx7b0tISkyZNQkpKykMXjLbH7oUXXkBQUFCT27U5jrrIFRsbi+3btyM5OfmeLzpdHK+7afP131pja2sLtVqNGzduNPmWVBfq6uowZcoUTJs2DZMnT27y+dsLesKECZg/fz4KCwt1fv30g54XXb2mtPHTTz/B29sbVlZWTT4n1fECACsrK+Tl5cHa2hp5eXmwtLRsssbW1hb79+9v/Dg7OxsjR4586MfWyymL4OBgJCQkoKamBllZWTh//jz8/PzuWKNQKDBq1Chs3boVQEMJ3GvH/bD27t0LZ2dn2NraNvv5yspKlJeXN/737t277/gprC7cft7uhx9+aPbxpBgpTkpKwooVK7Bt2zZ06dKl2TX6Ol7afP3BwcGNP/HeunUrRo8erfMdnxACkZGRcHFxwYIFC5pdk5+f37irSklJgUajQe/evXWaS5vnJTg4GJs3b4YQAr/99hvMzMwav13Xtft9lyrF8brl9tfQvXpo7Nix2L17N0pKSlBSUoLdu3dj7NixD//g2v70T2hxlcX3338vbGxshKmpqbC0tLzjF8QsXbpU2Nvbi8GDB4udO3c23j5+/HiRk5MjhBAiMzNT+Pr6CgcHBxESEiKqq6sf9JCtMnPmTLFu3bo7bsvJyRHjx49vzOHp6Sk8PT2Fq6urWLp0qU5y3G769OnC3d1deHh4iL/+9a8iNze3SS4hGn4q7uTkJOzt7fWSy8HBQdja2oqhQ4eKoUOHNl7BINXxau7rX7JkiXBwcBBCCFFVVSVCQkKEg4OD8PX1FZmZmTrLcsvBgwcFAOHh4dF4nHbs2CHWrVvX+Dpbs2aNcHV1FZ6ensLf318cOnRI57nu9bzcnkuj0Yj58+cLe3t74e7ufsfVUbpUWVkpzM3NRWlpaeNtUhyv0NBQ0bdvX2FiYiJsbGzEF198IQoLC8Xo0aOFo6OjGD16tCgqKhJCCJGamioiIyMb77thwwbh4OAgHBwcRExMzIMeSquO5aQeGQRO6pHMcVKPiKg9YSETEckEC5mISCZYyEREMsFCJiKSCRYyEZFMsJCJiGSChUxEJBMsZCIimWAhExHJBAuZiEgmWMhERDLBQiYikgkWMhGRTLCQSVYWLVoEZ2dneHp6YtKkSSgtLZU6EpHesJBJVgICAnDq1CmcOHECgwcPbvb91ogMFQuZZCUwMLDxnceHDx+O7OxsiRMR6Q8LmWQrJiYG48ePlzoGkd60+F2niR7WmDFjkJ+f3+T2ZcuWNb6h5LJly2BiYoJp06bd88+Jjo5GdHQ0AKCgoEA3YYn0iO+pR7ITGxuL9evXIzk5+Z7vdH03vqceyZxW76nHHTLJSlJSElasWIEDBw5oXcZEhoI7ZJIVR0dH1NTUoHfv3gAafrC3fv36B96PO2SSOe6Qqf25cOGC1BGIJMOrLIiIZIKFTEQkEyxkIiKZYCETEckEC5mISCZYyEREMsFCJiKSCRYyEZFMsJCJiGSChUxEJBMsZCIimWjpLxcikiWFQpEkhBgndQ6ih8FCJiKSCZ6yICKSCRYyEZFMsJCJiGSChUxEJBMsZCIimWAhExHJBAuZiEgmWMhERDLBQiYikon/A/Hf49bOO3V3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.spines['right'].set_color('none')\n",
    "ax.spines['top'].set_color('none')\n",
    "ax.xaxis.set_ticks_position('bottom')\n",
    "ax.spines['bottom'].set_position(('data', 0))\n",
    "ax.yaxis.set_ticks_position('left')\n",
    "ax.spines['left'].set_position(('data', 0))\n",
    "plt.ylim((-3, 10))\n",
    "relu=F.relu(x)\n",
    "plt.plot(x.numpy(),relu.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当输入是负数的时候，ReLU是完全不被激活的，这就表明一旦输入到了负数，ReLU就会死掉。但是到了反向传播过程中，输入负数，梯度就会完全到0，这个和sigmod函数、tanh函数有一样的问题。 但是实际的运用中，该缺陷的影响不是很大。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Leaky Relu 函数\n",
    "为了解决relu函数z<0时的问题出现了 Leaky ReLU函数，该函数保证在z<0的时候，梯度仍然不为0。\n",
    "ReLU的前半段设为αz而非0，通常α=0.01 $ a=max(\\alpha z,z)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2ad8b0bd3c8>]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADvCAYAAADFG66PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XlclOXeP/DPCCKoLC7sqKwqiIis2mkxH/dMTS0hDRUM09OTZsfynCfLevRndvKpbNEoEaTEzOpgaJhpmpkJg/uuCCabsgjIzsxcvz9QEkEdhJn7Zubzfr164cxcN/Ode6YP11xzf+dWCCFARETS6yB1AUREVI+BTEQkEwxkIiKZYCATEckEA5mISCYYyEREMsFAJkkpFIpYhUJxTaFQnLztuu4KhWKXQqG4cPNnNylrJNIXBjJJLQ7AmDuuWwJgtxDCC8Dum5eJDJ6CjSEkNYVC4QogWQjhe/PyOQDDhBB5CoXCEcBeIUQ/CUsk0gvOkEmO7IUQeQBw86edxPUQ6YVpC8dzOk1tLjMzE+PHjwduvr6sra0b/g0ANjY2jS7fEhMTg5iYGABAVVUVTp06pYdqyZjsOJGHv286jOH97PDZc4EwNXngOaxCm0GcIZPs2NvbIy8vDwCQl5cHO7vmJ8jR0dFQKpVQKpWwsLDQZ4lkBH6/WIiFm48isHc3fPxsQGvCWGsMZJKdCRMmID4+HgAQHx+PiRMnSlwRGZuTOaWITkiHa8/OWD8zGBZmJnq5XwYySSo8PBxDhw7FuXPn4OLigvXr12PJkiXYtWsXvLy8sGvXLixZwoMsSH+yCiswa0MqrC06YmNkKKw7d9Tbfbf0KAuuIZMsBQUFQalUSl0GtXPXblRj6tqDuFFdh29eeAiedl3b6ldrtYbc0g/1iIgMUll1HWbFpqHgRg0So4e0ZRhrjUsWRGT0quvUeD5eifNXb2DtjAD497KRpA7OkInIqKk1Ags3H8WhzGJ8MM0fw/pJd9g7Z8hEZLSEEHj9PyeRciofb4z3waTBzpLWw0AmIqP1/q7zSEz9E/OHeSDyYTepy2EgE5Fx2ngwC2v2XMS0oF5YPFoeX5XCQCYio5N8PBdvbjuFkT72WPGULxQKrY5K0zkGMhEZld8uFOLlr48iuE93fBQ+WC8t0dqSTyVERDp2PLsEcxOU8LDtis9nBsG8o35aorXFQCYio5BZWIHZG9Jg09kM8ZEhsLbQX0u0thjIRGTwrpZV47n1hyAAJESFwN7KXOqSmsVAJiKDVlpVh5mxqSiuqMWGWcFwt9V/S7S2GMhEZLButURnFJTjs+cCMUiilmhtsXWaiAySSq3BS4lHkHa5GB+GDcYjXrZSl3RfnCETkcG51RL90+mreHO8DyYMcpK6JK0wkInI4Kz+6Tw2p13Bi497YtbfpG+J1hYDmYgMStyBTHz8y0WEh/TCK6P6Sl1OizCQichg/HAsF28ln8YoH3v870T5tERri4FMRAZh/4UCLNpS3xK9RmYt0dpqfxUTEd3h2JUSzE1Il21LtLYYyETUrmUUlGN2XBq6dzHDRpm2RGuLgUxE7dbVsmpErE+FAkBCVCjsZNoSrS0GMhG1S6WVdYhYn4qSylrEzQ6BW88uUpfUauzUI6J2p7pOjTkb03CpsBwbZoVgoIu11CW1CQYyEbUrKrUGL246AuXl6/gofDAe9uopdUlthksWJEvvv/8+BgwYAF9fX4SHh6O6ulrqkkgGhBD41/cn8POZq3hrwgCM92sfLdHaYiCT7OTk5GDNmjVQKpU4efIk1Go1Nm/eLHVZJAP/3nkOW5TZeGm4JyKGukpdTptjIJMsqVQqVFVVQaVSobKyEk5OhjUTopaL/S0Tn+7NQHhIb7w8sn21RGuLgUyy4+zsjH/84x/o3bs3HB0dYW1tjVGjRjUZFxMTg6CgIAQFBaGgoECCSklfko7m4O3k0xg9wB7LJ7W/lmhtMZBJdq5fv46kpCRkZmYiNzcXFRUV+PLLL5uMi46OhlKphFKphK2t/L/rlh7MvvMFeGXLMYS6dceHYYNh0sEwwxhgIJMM/fzzz3Bzc4OtrS06duyIyZMn4/fff5e6LJLA0SslmPdlOrzsLdt1S7S2GMgkO71798Yff/yByspKCCGwe/dueHt7S10W6VlGQTlmb0hFj65miJ8dDCvz9tsSrS0GMslOaGgopk6dioCAAAwcOBAajQbR0dFSl0V6lF9a3xJt0kGBhMj23xKtLYUQoiXjWzSYSF+CgoKgVCqlLoPaQGllHZ757CBySqqwOXoIfJ0NogtPq4VvzpCJSDaqatWIik9DZmEFYp4LNJQw1hpbp4lIFupbog8j/c/r+Dg8AA95Gk5LtLY4QyYiyQkh8M/vTmD32Wt4e6IvnvBzlLokSTCQiUhyq1LO4Zv0bCz4Ly88N6SP1OVIhoFMRJL6Yv8lrNuXgemhvbFwhJfU5UiKgUxEkvnPkRws334G4wY64O12eJbotsZAJiJJ/HLuGv7xzTEMde+B96f5G3RLtLYYyESkd0f+vI75Xx5GPwdLxEQEopOpYbdEa4uBTER6dfHaDcyOS4OdVSfEzQ6BpRG0RGuLgUxEepNXWoWI9akw7dABCZGhsLXsJHVJssJAJiK9KKmsRcT6VJRVqxAfGYzePTpLXZLsMJCJSOfqW6KVuFxUiZiIQAxwMq6WaG2xdZqIdKpOrcHfNx3G4T+v49NnA/CQh/G1RGuLM2Qi0hkhBJZ8ewJ7zl7D8km+GDvQOFuitcVAJiKdeefHs/j2cDZeHtEX00ONtyVaWwxkItKJz3+9hM9+vYSIoX3w0n95Sl1Ou8BAJqI29216NlbsOIMnBjrizScHGH1LtLYYyETUpn45ew2vfXscf/Psgf+bNogt0S3AQCaiNpN++TrmfZWO/o6WWDeDLdEtxUAmojZx4eoNRMalwcHKnC3RD4iBTEStlltShYjYVJiZdkBCVCh6dmVL9INgIBNRq1yvqMVz6w+hvFqF+Nkh6NWdLdEPip16RPTAKmtVmB2XhivXq7AxMgQ+TlZSl9SucYZMRA+kTq3B/K8O43h2CdaEDcYQ9x5Sl9TuMZBJlkpKSjB16lT0798f3t7eOHjwoNQl0W00GoFXtx7H3nMFWPHUQIzxdZC6JIPAJQuSpQULFmDMmDHYunUramtrUVlZKXVJdJuVP57B90dy8MrIvggP6S11OQaDgUyyU1ZWhl9//RVxcXEAADMzM5iZmUlbFDX4bF8GPt+fiZlD++DF4WyJbktcsiDZuXTpEmxtbTF79mwMHjwYc+bMQUVFhdRlEYCt6dlY+eNZjPdjS7QuMJBJdlQqFQ4fPox58+bhyJEj6NKlC955550m42JiYhAUFISgoCAUFBRIUKlx2XP2Kl779jge9uyJ1c8MQge2RLc5BjLJjouLC1xcXBAaGgoAmDp1Kg4fPtxkXHR0NJRKJZRKJWxtbfVdplFJv1yM+V8dho+jFdY9x5ZoXWEgk+w4ODigV69eOHfuHABg9+7d8PHxkbgq43X+6g1ExinhaG2BuNnB6NqJHz3pCvcsydJHH32E6dOno7a2Fu7u7tiwYYPUJRmlnJL6s0R3Mu2AjZEh6MGWaJ1iIJMs+fv7Q6lUSl2GUSu+2RJdUavClrlD2RKtBwxkImqioqa+JTr7ehUSIkPg7ciWaH3gGjIRNVKr0mDeV4dxIrsEH4cPRihbovWGM2QiaqDRCCzeegy/ni/AO5MHYtQAtkTrE2fIRAQAEEJg+fYzSDqai8Wj+yGMLdF6x0AmIgDA2n0ZiD2QiVkPuWL+MA+pyzFKDGQiwhblFbybcg4TBjnhjfE+bImWCAOZyMj9fPoq/vndCTzi1RPvPc2WaCkxkImMWFpWMf6+6TB8naywbkYgzEwZCVLi3icyUmfzyxAVlwZnGwvEzgpGF7ZES46BTGSEsq9XYmZsKizMTLAxii3RcsFAJjIyReU1iFifiqpaNTZGhsKlG1ui5YLvUYiMSEWNCpFxacgpqcKXc0LRz8FS6pLoNgxkIiNRq9LghS/TcTK3DOtmBCLYtbvUJdEduGRBZAQ0GoF/fHMM+y8UYuVTAzHSx17qkqgZDGQiAyeEwNvJp7HtWC5eHdMPzwT3krokugsGMpGB+3RvBuJ+z0Lk39ww7zG2RMsZA5nIgH2d9if+vfMcJvk74fUnvNkSLXMMZCID9dOpfPzzuxN4tK8t3p3Kluj2gIFMZIBSM4vx34lHMNDFBmunB7Alup3gs0RkYM7klSEqPg3O3SywgS3R7QoDmciAXCmub4nuYmaKhKhQdO9iJnVJ1AL800lkIArLaxARm4oalQbfvDAUzjYWUpdELcQZMpEBKK9RYfaGNOSVViF2VhD62rMluj3iDJmonatRqfFCQjpO55Xh84hABPZhS3R7xRkyUTum0Qi8suUYfrtYiFVT/DC8P1ui2zMGMsmWWq3G4MGDMX78eKlLkSUhBN764RSSj+fhn2P7Y2qgi9QlUSsxkEm2PvzwQ3h7e0tdhmx9vOci4g9exvOPuGEuW6INAgOZZCk7Oxvbt2/HnDlzpC5FlhJT/8TqXecxebAz/jmWf7QMBQOZZGnhwoV499130aHD3V+iMTExCAoKQlBQEAoKCvRYnbRSTubjf74/gWH9bLFqqh9bog0IA5lkJzk5GXZ2dggMDLznuOjoaCiVSiiVStja2uqpOmkdulSElzYfwaBeNvh0egA6mvB/YUPCZ5Nk58CBA9i2bRtcXV0RFhaGPXv2YMaMGVKXJbnTuWWYE69Er24WiJ0ZjM5mPGrV0CiEEC0Z36LBRK21d+9evPfee0hOTr7nuKCgICiVSj1VpX9/FlViyrrfYdpBgW/nPQQnduG1N1qtK3GGTCRz9S3Rh1Cn1mBjZAjD2IBxhkwGwVBnyDeq6xD++R+4eK0cX80ZgsA+3aQuiR6MVjNkLkIRyVSNSo25Cek4k3cDX0QEMYyNAJcsiGRIrRFY9PUx/J5RhH9P9cPj/e2kLon0gIFMJDNCCCzbdgrbT+Thf8Z5Y3IAW6KNBQOZSGbW7L6IhD8uY+6j7nj+UXepyyE9YiATychXhy7j/Z/PY3KAM14b01/qckjPGMhEMpFyMg9L/3MSw/vbYdUUtkQbIwYykQwczCjCS4lH4d/LBp88y5ZoY8VnnUhip3JLEb1RiT49OiN2VjAszEykLokkwkAmktDlogrMjE2DpbkpNkaFwKYzzxJtzBjIRBIpuFF/lmiVRoONUSFwtGZLtLFjIBNJ4EZ1HWZtSMW1shpsmBUMTzueJZoYyER6V12nRvTGdJzLv4G1MwIwuDdboqkev8uCSI/UGoGXvz6Kg5eK8ME0fwzrx5Zo+gtnyER6IoTAG0kn8ePJfLz+hDcmDXaWuiSSGQYykZ588PMFfHXoT7zwmAfmPMKWaGqKgUykBwl/XMaHuy/g6UAXvDamn9TlkEwxkIl0bPvxPLyRdBIjvO2wcvJAKBRsiabmMZCJdOj3i4V4+eujCOzdDR+FB8CULdF0D3x1EOnIyZxSRCekw61nF6yfyZZouj8GMpEOZBVWYNaGVFhbdER8ZAisO3eUuiRqBxjIRG3s2o1qRMSmQq0R2BgVAgdrc6lLonaCjSFEbaisug4zY9NQWF6DTc8PgYdtV6lLonaEM2SiNlJdp8bz8UpcuHoD62YEwr+XjdQlUTvDGTJRG1BrBBZuPopDmcX4MMwfj/a1lbokaoc4QybZuXLlCh5//HF4e3tjwIAB+PDDD6Uu6Z6EEHj9PyeRciofS8f7YKI/W6LpwXCGTLJjamqK1atXIyAgADdu3EBgYCBGjhwJHx8fqUtr1vu7ziMx9U/MG+aBqIfdpC6H2jHOkEl2HB0dERAQAACwtLSEt7c3cnJyJK6qefG/Z2HNnot4JsgFr45mSzS1DgOZZC0rKwtHjhxBaGio1KU0kXw8F8t+OIUR3vb4f0+xJZpaj0sWJFvl5eWYMmUKPvjgA1hZWTW5PSYmBjExMQCAgoICvdb224X6luigPt3w8bOD2RJNbUIhhGjJ+BYNJnpQdXV1GD9+PEaPHo1Fixbdd3xQUBCUSqUeKgNOZJciLOYgenXvjK/nDoW1Bbvw6L60evvEP+skO0IIREVFwdvbW6sw1qfMmy3RNp3N6luiGcbUhhjIJDsHDhxAQkIC9uzZA39/f/j7+2PHjh1Sl4WrZdV4bv0hCAAJUSGwt2JLNLUtriGT7Dz88MNo4VKazpVW1WFmbCqKK2qxOXoI3NkSTTrAGTLRfdxqic4oKMdnzwXCz4Ut0aQbnCET3YNKrcF/Jx5B2uVifBg2GI94sSWadIczZKK7uNUSvev0Vbw53gcTBjlJXRIZOAYy0V2s/uk8NqddwYuPe2LW39gSTbrHQCZqRtyBTHz8y0WEh/TCK6P6Sl0OGQkGMtEdth3LxVvJpzHKxx7/O9GXLdGkNwxkotvsv1CAV7YcRbBrd6wJZ0s06RdfbUQ3Hc8uwdyEdHjYdsXnEUEw78izRJN+MZCJAFwqKMesDWno3sUMG9kSTRJhIJPRq2+JToUCQEJUKOzYEk0SYSCTUbvVEl1SWYu42SFw69lF6pLIiLFTj4xWdZ0ac+LTkFFQjg2zQjDQxVrqksjIMZDJKKnUGry46TCUl6/jo/DBeNirp9QlEXHJgoyPEAL/+v4Efj5zDW9NGIDxfmyJJnlgIJPR+ffOc9iizMZLwz0RMdRV6nKIGjCQyajE/paJT/dmIDykN14eyZZokhcGMhmNpKM5eDv5NMYMcMDySWyJJvlhIJNR+PV8AV7ZcgxD3LvjgzB/mHRgGJP8MJDJ4B29UoIXvkyHl70lYtgSTTLGQCaDllFQjsi4NPToaob42cGwMmdLNMkXA5kMVn5pNSLWp6KDAkiIZEs0yR8bQ8gglVbWt0SXVtVhc/QQuLIlmtoBzpDJ4FTVqhEVn4bMwgrEPBcIX2e2RFP7wBkyGZRbLdHpf17HJ88G4CFPtkRT+8EZMslSSkoK+vXrB09PT7zzzjtabVOjUuO1b09g99lreHuiL8YNdNRxlURtSyGEaMn4Fg0mehBqtRp9+/bFrl274OLiguDgYCQmJsLHx+eu2/Tz9YfTrA+QWViBhSO8sHAEu/BIVrQ68J0zZJKd1NRUeHp6wt3dHWZmZggLC0NSUlKzY3NLqjDvy3RkFVUAADZGhjCMqd3iGjLJTk5ODnr16tVw2cXFBYcOHQJQvyxxtbQGeaVVSM0sxqd7MyAgYG9ljpSFj6CTKZs+qP1q0ZLFmDFjRGFh4QPdUUFBAWxtbR9oW11iXS2jy7qEAOrUGhSXlOJGZRVsuvVAnVqDiqoa1Kk1UJiYQqVp/HpV1FWjQ3Upaqsr4e/vr5O6WsMYn8fWMNS60tPTdwohxtxvnN7WkIOCgqBUKh90c51hXS3zoHVV1qqQV1qN/NLqmz+rkHvH5euVdU22s7boCJOaMnTpUIeHAwbAwcoCjtbmcLA2R6/unRtOudSlSxdUVFS0+vG1NUN7HnXNgOvSag2ZSxbUauU1KuSXViHvZrjmlVQjv6yqIYBzS6pQVq1qsl33LmawtzKHk7U5Anrb3AxaC9h17YiZT0/A9q2b4OnaC8HBwVi3aRMGDBggwaMj0h8GMt2VEAJl1aqbs9gq5JdWo8J9GF7deqwhbPNLq3GjpmnY9uzaCY43Z7Ehbt3rg9fGHI7W9TNceyvze37JzyfvvIlJ48dCrVYjMjKSYUxGQW+BHB0dra+7ahFjrUsIgZLKuvpgvTmbzSupbnQ5v7QalbXqxhu6D8PecwVwtLGAh21X/M2zZ8MSgpONBRys6sPWzLR1B/CMGzcO48aN03p8z57ybAAx1tfXgzL2ungcsgHSaASKK2vvsl5b1XB9jUrTaLsOCsDeqj5cHaz+ms3Wh2190NpbmaOjifyOlpTr2iPRTVxDNkQajUBhRc1tYVuN3NtCNq+0CldLa1Crbhy2ph0UDWHr62yNkT72cLg9cK0t0LOrGUxlGLZExqJNA/mbb77BsmXLcObMGaSmpiIoKKjhtpUrV2L9+vUwMTHBmjVrMHr06CbbZ2ZmIiwsDMXFxQgICEBCQgLMzMzaskRMmzYN586dAwCUlJTAxsYGR48ebTLO1dUVlpaWMDExgampqc5nX8uWLcPnX3yBHs7u0JhbITxyHhw9fJBXWtXk6AT1He9TzEw61M9qrc0R0LsbHKzN4WhlDkebvwK3Z5dO6PAAZ8lYvHgxfvjhB5iZmcHDwwMbNmyAjY1Nk3H62l8pKSlYsGAB1Go15syZgyVLljS6vaamBhEREUhPT0ePHj3w9ddfw9XVVSe13HLlyhVEREQgPz8fHTp0QHR0NBYsWNBozN69ezFx4kS4ubkBACZPnow33nhDp3UB939ehBBYsGABduzYgc6dOyMuLg4BAQE6rencuXOYNm1aw+VLly7h7bffxsKFCxuu09f+ioyMRHJyMuzs7HDy5EkAQHFxMaZNm4asrCy4urpiy5Yt6NatW5Nt4+PjsXz5cgDA66+/jpkzZ7a+ICFES/67p9OnT4uzZ8+Kxx57TKSlpTVcf+rUKeHn5yeqq6vFpUuXhLu7u1CpVE22f/rpp0ViYqIQQoi5c+eKTz/99H532SqLFi0Sb731VrO39enTRxQUFLTZfdWq1CL7eqVQZhWJH47liJh9GeKtbafEvC+VYtInvwnv174Trq/9IPq8ltzov77/s0M89u4eEfbZQbEg8bBwnbBA/N+2NPHjsWwx4OHR4vf040Kj0bRZnXfauXOnqKurE0II8eqrr4pXX3212XFtvb+ao1KphLu7u8jIyBA1NTXCz89PnDp1SgghRGBgoBBCiE8++UTMnTtXCCFEYmKieOaZZ3RakxBC5ObmivT0dCGEEGVlZcLLy6uhrlt++eUX8cQTT+i8ljvd73nZvn27GDNmjNBoNOLgwYMiJCREj9XVP6f29vYiKyur0fX62l/79u0T6enpYsCAAQ3XLV68WKxcuVIIIcTKlSubfc0XFRUJNzc3UVRUJIqLi4Wbm5soLi6+111plbFtOkP29vZu9vqkpCSEhYWhU6dOcHNzg6enJ1JTUzF06NBGfxj27NmDTZs2AQBmzpyJZcuWYd68eW1ZYqP727JlC/bs2dPq31Wr0uBq2V9LBrcvH9z6d0F5De5crrfoaAJHG3M4WpvDsUMZunUCJo0eBicbczhYWcDB2hzdOndsOBnnwYMHcab6DF5+sv6dx5Fxj2HvzmQMDRjY6sdwN6NGjWr495AhQ7B161ad3df93N5SDaChpfr277hISkrCsmXLAABTp07Fiy++CCGETk9o6ujoCEfH+i8ysrS0hLe3N3Jycu753RtykZSUhIiICCgUCgwZMgQlJSXIy8treDy6tnv3bnh4eKBPnz56ub87Pfroo8jKymp0XVJSEvbu3QugPoeGDRuGVatWNRqzc+dOjBw5Et27dwcAjBw5EikpKQgPD29VPXpZQ87JycGQIUMaLru4uCAnJ6fRmKKiItjY2MDU1PSuY9rS/v37YW9vDy8vr2ZvVygU9WFk0hHTZkXjsbGTmlk+qP9ZWF7TZHvLTqawt64P234Olg3rtbeWEBytLWBlbtoQFMuWpSBuQxzOf2uFoKAgrF69Gt26NF6uuVdLsT7ExsY2eqt5u1v7S6FQYO7cuTr5VFqbx3/7GFNTU1hbW6OoqEhvR2FkZWXhyJEjCA0NbXLbwYMHMWjQIDg5OeG9997Ty6F893temtunOTk5egvkzZs33zXEpNhfAHD16tWGx+/o6Ihr1641GXO3/dZaLQ7kESNGID8/v8n1K1aswMSJE5vdRjRzJMedMxZtxrRljYmJiZga9iwuFZTf/GCsuqG5Ib+0Gp5/X4+iKjWKK2qx9gqwNuaPht9jZW5af4iXtTl8na0ausduzXbtrcxh2cy52+5V17x587B06VIoFAosXboUr7zyCmJjYxuNa8t9pG1dt/bXihUrYGpqiunTpzf7Ow4cOAAnJydcu3YNI0eORP/+/fHoo4+2urbb6ft11FLl5eWYMmUKPvjgA1hZWTW6LSAgAJcvX0bXrl2xY8cOTJo0CRcuXNB5Tfd7XqTcX7W1tdi2bRtWrlzZ5Dap9pe2dLXfWhzIP//8c4vvxMXFBVeuXGm4nJ2dDScnp0ZjevbsiZKSEqhUKpiamjY7piU1Nuoeu3l87R9lVfh+QyrySqpw1nwYFIVdsW71vkbbduvcEQ7WFujV0xLBNz8cO7Q3BT0sTPDS8xFwtDZHZ7MHe2Oh7b57/vnnMX78+CbXa7MfdVFXfHw8kpOTsXv37ru+6G7VYWdnh6eeegqpqaltHsjaPP5bY1xcXKBSqVBaWtrwtlKX6urqMGXKFEyfPh2TJ09ucvvtAT1u3DjMnz8fhYWFOp+53+950dVrShs//vgjAgICYG9v3+Q2qfYXANjb2zcs2+Tl5cHOzq7JGBcXl4ZlDaB+vw0bNqzV962XJYsJEybg2WefxaJFi5Cbm4sLFy4gJCSk0RiFQoHHH38cW7duRVhYGOLj45udcQshcKNG1dCSe/t67f27x8zgYG2OTqpyWF4/j/mzwuuXEG77fgTzjiaoqKiARqOBpaUlKioqkLg0ETPeeAMetl11to9uX7f7/vvv4evr22RMcHAwLly4gMzMTDg7O2Pz5s0Na+66kpKSglWrVmHfvn3o3Llzs2Pu3F8//fSTTj4R1+bxT5gwAfHx8Rg6dCi2bt2K4cOH63zGJ4RAVFQUvL29sWjRombH5Ofnw97eHgqFAqmpqdBoNOjRo4dO69LmeZkwYQI+/vhjhIWF4dChQ7C2ttbbckViYuJdlyuk2F+33HoNLVmy5K45NHr0aPzrX//C9evXAQA//fRTszP9FtP20z+hxVEW3333nXB2dhZmZmbCzs5OjBo1quG25cuXC3d3d9G3b1+xY8eOhuvHjh0rcnJyhBBCXLx4UQQOfUS4BzwiRkQsFHH7L4r3dp4Vr2w5KqZ//ocY/t4vwmfpj02ORHBdkiyCl+8ST360X0RvTBNvJp0U6/ZeFP85ki3+yCgUfxZViOq6v47qmDlzpli7dm2j2nNycsTYsWMMcQ56AAADPUlEQVSFEEJkZGQIPz8/4efnJ3x8fMTy5cvv99BbbcaMGcLX11cMHDhQPPnkkyI3N7dJXULUfyru5eUl3N3d9VKXh4eHcHFxEYMGDRKDBg1qOIJBqv3V3ONfunSp8PDwEEIIUVVVJaZOnSo8PDxEcHCwyMjI0Fktt+zfv18AEAMHDmzYT9u3bxdr165teJ199NFHwsfHR/j5+YnQ0FBx4MABndd1t+fl9ro0Go2YP3++cHd3F76+vo2OjtKliooK0b17d1FSUtJwnRT7KywsTDg4OAhTU1Ph7OwsvvjiC1FYWCiGDx8uPD09xfDhw0VRUZEQQoi0tDQRFRXVsO369euFh4eH8PDwELGxsfe7K60yVm+dekIIFFfUNnwBTf6dH5CV1c9yq+uado/ZWf61PutgZdHQNeZkc/PLaCw7ybJ7jPSHnXokc/Lp1Htm3UEcvVJyz+4xHycrjPC2uxm0FjePRDCHbddO7B4jIqOgl0Ae6tEDg2/7esVbh3/16NoJJg/QPUZEZIj0Esgvj+Q5zoiI7odrAUREMsFAJiKSCQYyEZFMMJCJiGSCgUxEJBMMZCIimWAgExHJBAOZiEgmGMhERDLBQCYikgkGMhGRTDCQSVYWL16M/v37w8/PD0899RRKSkqkLolIbxjIJCsjR47EyZMncfz4cfTt27dtzsJA1E4wkElWRo0a1XDm8SFDhiA7O1viioj0h4FMshUbG4uxY8dKXQaR3ujl+5CJbjdixAjk5+c3uX7FihUNJ5RcsWIFTE1NMX369Lv+npiYGMTExAAACgoKdFMskR7p7Zx6RNqKj4/HunXrsHv37rue6fpOPKceyZx8zqlHpK2UlBSsWrUK+/bt0zqMiQwFZ8gkK56enqipqUGPHj0A1H+wt27duvtuxxkyyRxnyNT+XLx4UeoSiCTDoyyIiGSCgUxEJBMMZCIimWAgExHJBAOZiEgmGMhERDLBQCYikgkGMhGRTDCQiYhkgoFMRCQTDGQiIplo6ZcLEcmSQqFIEUKMkboOotZgIBMRyQSXLIiIZIKBTEQkEwxkIiKZYCATEckEA5mISCYYyEREMsFAJiKSCQYyEZFMMJCJiGTi/wOL6RpYunAGYwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.spines['right'].set_color('none')\n",
    "ax.spines['top'].set_color('none')\n",
    "ax.xaxis.set_ticks_position('bottom')\n",
    "ax.spines['bottom'].set_position(('data', 0))\n",
    "ax.yaxis.set_ticks_position('left')\n",
    "ax.spines['left'].set_position(('data', 0))\n",
    "plt.ylim((-3, 10))\n",
    "l_relu=F.leaky_relu(x,0.1) # 这里的0.1是为了方便展示，理论上应为0.01甚至更小的值\n",
    "plt.plot(x.numpy(),l_relu.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "理论上来讲，Leaky ReLU有ReLU的所有优点，但是在实际操作当中，并没有完全证明Leaky ReLU总是好于ReLU。\n",
    "\n",
    "ReLU目前仍是最常用的activation function，在隐藏层中推荐优先尝试！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 深入理解前向传播和反向传播\n",
    "在最后我们再详细说下神经网络中的前向传播和反向传播，这里继续使用吴恩达老师的板书\n",
    "![](8.png)\n",
    "### 正向传播\n",
    "对于一个神经网络来说，把输入特征$a^{[0]}$这个输入值就是我们的输入$x$，放入第一层并计算第一层的激活函数，用$a^{[1]}$表示，本层中训练的结果用$W^{[1]}$和$b^{[l]}$来表示，这两个值与，计算的结果$z^{[1]}$值都需要进行缓存，而计算的结果还需要通过激活函数生成激活后的$a^{[1]}$,即第一层的输出值，这个值会作为第二层的输入传到第二层，第二层里，需要用到$W^{[2]}$和$b^{[2]}$，计算结果为$z^{[2]}$，第二层的激活函数$a^{[2]}$。\n",
    "后面几层以此类推，直到最后算出了$a^{[L]}$，第$L$层的最终输出值$\\hat{y}$,即我们网络的预测值。正向传播其实就是我们的输入$x$通过一系列的网络计算，得到$\\hat{y}$的过程。\n",
    "\n",
    "在这个过程里我们缓存的值,会在后面的反向传播中用到。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 反向传播\n",
    "对反向传播的步骤而言，就是对正向传播的一系列的反向迭代，通过反向计算梯度，来优化我们需要训练的$W$和$b$。\n",
    "把${\\delta}a^{[l]}$值进行求导得到${\\delta}a^{[l-1]}$，以此类推，直到我们得到${\\delta}a^{[2]}$和${\\delta}a^{[1]}$。反向传播步骤中也会输出 ${\\delta}W^{[l]}$和${\\delta}b^{[l]}$。这一步我们已经得到了权重的变化量，下面我们要通过学习率来对训练的$W$和$b$进行更新，\n",
    "\n",
    "$W=W-\\alpha{\\delta}W $\n",
    "\n",
    "$b=b-\\alpha{\\delta}b $\n",
    "\n",
    "这样反向传播就就算是完成了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}