2-2,自动微分机制.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "201a1b8f",
   "metadata": {},
   "source": [
    "# 2-2,自动微分机制"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63102fcc",
   "metadata": {},
   "source": [
    "神经网络通常依赖反向传播求梯度来更新网络参数，求梯度过程通常是一件非常复杂而容易出错的事情。\n",
    "\n",
    "而深度学习框架可以帮助我们自动地完成这种求梯度运算。\n",
    "\n",
    "Pytorch一般通过反向传播 backward 方法 实现这种求梯度计算。该方法求得的梯度将存在对应自变量张量的grad属性下。\n",
    "\n",
    "除此之外，也能够调用torch.autograd.grad 函数来实现求梯度计算。\n",
    "\n",
    "这就是Pytorch的自动微分机制。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ca9b970",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "print(\"torch.__version__=\"+torch.__version__) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191847a6",
   "metadata": {},
   "source": [
    "```\n",
    "torch.__version__=1.10.0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58d2c348",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "36e66065",
   "metadata": {},
   "source": [
    "### 一，利用backward方法求导数"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac1d6e6c",
   "metadata": {},
   "source": [
    "backward 方法通常在一个标量张量上调用，该方法求得的梯度将存在对应自变量张量的grad属性下。\n",
    "\n",
    "如果调用的张量非标量，则要传入一个和它同形状 的gradient参数张量。\n",
    "\n",
    "相当于用该gradient参数张量与调用张量作向量点乘，得到的标量结果再反向传播。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2baf6e72",
   "metadata": {},
   "source": [
    "**1, 标量的反向传播**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f007e5ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import torch \n",
    "\n",
    "# f(x) = a*x**2 + b*x + c的导数\n",
    "\n",
    "x = torch.tensor(0.0,requires_grad = True) # x需要被求导\n",
    "a = torch.tensor(1.0)\n",
    "b = torch.tensor(-2.0)\n",
    "c = torch.tensor(1.0)\n",
    "y = a*torch.pow(x,2) + b*x + c \n",
    "\n",
    "y.backward()\n",
    "dy_dx = x.grad\n",
    "print(dy_dx)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "234280b7",
   "metadata": {},
   "source": [
    "```\n",
    "tensor(-2.)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "171c6ddf",
   "metadata": {},
   "source": [
    "**2, 非标量的反向传播**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "830a8071",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import torch \n",
    "\n",
    "# f(x) = a*x**2 + b*x + c\n",
    "\n",
    "x = torch.tensor([[0.0,0.0],[1.0,2.0]],requires_grad = True) # x需要被求导\n",
    "a = torch.tensor(1.0)\n",
    "b = torch.tensor(-2.0)\n",
    "c = torch.tensor(1.0)\n",
    "y = a*torch.pow(x,2) + b*x + c \n",
    "\n",
    "gradient = torch.tensor([[1.0,1.0],[1.0,1.0]])\n",
    "\n",
    "print(\"x:\\n\",x)\n",
    "print(\"y:\\n\",y)\n",
    "y.backward(gradient = gradient)\n",
    "x_grad = x.grad\n",
    "print(\"x_grad:\\n\",x_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae429fd4",
   "metadata": {},
   "source": [
    "```\n",
    "x:\n",
    " tensor([[0., 0.],\n",
    "        [1., 2.]], requires_grad=True)\n",
    "y:\n",
    " tensor([[1., 1.],\n",
    "        [0., 1.]], grad_fn=<AddBackward0>)\n",
    "x_grad:\n",
    " tensor([[-2., -2.],\n",
    "        [ 0.,  2.]])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a962975d",
   "metadata": {},
   "source": [
    "**3, 非标量的反向传播可以用标量的反向传播实现**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61e9640d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import torch \n",
    "\n",
    "# f(x) = a*x**2 + b*x + c\n",
    "\n",
    "x = torch.tensor([[0.0,0.0],[1.0,2.0]],requires_grad = True) # x需要被求导\n",
    "a = torch.tensor(1.0)\n",
    "b = torch.tensor(-2.0)\n",
    "c = torch.tensor(1.0)\n",
    "y = a*torch.pow(x,2) + b*x + c \n",
    "\n",
    "gradient = torch.tensor([[1.0,1.0],[1.0,1.0]])\n",
    "z = torch.sum(y*gradient)\n",
    "\n",
    "print(\"x:\",x)\n",
    "print(\"y:\",y)\n",
    "z.backward()\n",
    "x_grad = x.grad\n",
    "print(\"x_grad:\\n\",x_grad)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d81bdd26",
   "metadata": {},
   "source": [
    "```\n",
    "x: tensor([[0., 0.],\n",
    "        [1., 2.]], requires_grad=True)\n",
    "y: tensor([[1., 1.],\n",
    "        [0., 1.]], grad_fn=<AddBackward0>)\n",
    "x_grad:\n",
    " tensor([[-2., -2.],\n",
    "        [ 0.,  2.]])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ba89750",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b37e32f4",
   "metadata": {},
   "source": [
    "### 二，利用autograd.grad方法求导数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "474468ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import torch \n",
    "\n",
    "# f(x) = a*x**2 + b*x + c的导数\n",
    "\n",
    "x = torch.tensor(0.0,requires_grad = True) # x需要被求导\n",
    "a = torch.tensor(1.0)\n",
    "b = torch.tensor(-2.0)\n",
    "c = torch.tensor(1.0)\n",
    "y = a*torch.pow(x,2) + b*x + c\n",
    "\n",
    "\n",
    "# create_graph 设置为 True 将允许创建更高阶的导数 \n",
    "dy_dx = torch.autograd.grad(y,x,create_graph=True)[0]\n",
    "print(dy_dx.data)\n",
    "\n",
    "# 求二阶导数\n",
    "dy2_dx2 = torch.autograd.grad(dy_dx,x)[0] \n",
    "\n",
    "print(dy2_dx2.data)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "005fad86",
   "metadata": {},
   "source": [
    "```\n",
    "tensor(-2.)\n",
    "tensor(2.)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41abfc48",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import torch \n",
    "\n",
    "x1 = torch.tensor(1.0,requires_grad = True) # x需要被求导\n",
    "x2 = torch.tensor(2.0,requires_grad = True)\n",
    "\n",
    "y1 = x1*x2\n",
    "y2 = x1+x2\n",
    "\n",
    "\n",
    "# 允许同时对多个自变量求导数\n",
    "(dy1_dx1,dy1_dx2) = torch.autograd.grad(outputs=y1,inputs = [x1,x2],retain_graph = True)\n",
    "print(dy1_dx1,dy1_dx2)\n",
    "\n",
    "# 如果有多个因变量，相当于把多个因变量的梯度结果求和\n",
    "(dy12_dx1,dy12_dx2) = torch.autograd.grad(outputs=[y1,y2],inputs = [x1,x2])\n",
    "print(dy12_dx1,dy12_dx2)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d30abba9",
   "metadata": {},
   "source": [
    "```\n",
    "tensor(2.) tensor(1.)\n",
    "tensor(3.) tensor(2.)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "132689e9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "19acc361",
   "metadata": {},
   "source": [
    "### 三，利用自动微分和优化器求最小值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc700c05",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import torch \n",
    "\n",
    "# f(x) = a*x**2 + b*x + c的最小值\n",
    "\n",
    "x = torch.tensor(0.0,requires_grad = True) # x需要被求导\n",
    "a = torch.tensor(1.0)\n",
    "b = torch.tensor(-2.0)\n",
    "c = torch.tensor(1.0)\n",
    "\n",
    "optimizer = torch.optim.SGD(params=[x],lr = 0.01)\n",
    "\n",
    "\n",
    "def f(x):\n",
    "    result = a*torch.pow(x,2) + b*x + c \n",
    "    return(result)\n",
    "\n",
    "for i in range(500):\n",
    "    optimizer.zero_grad()\n",
    "    y = f(x)\n",
    "    y.backward()\n",
    "    optimizer.step()\n",
    "   \n",
    "    \n",
    "print(\"y=\",f(x).data,\";\",\"x=\",x.data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72e8fd55",
   "metadata": {},
   "source": [
    "```\n",
    "y= tensor(0.) ; x= tensor(1.0000)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3009819c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b0602053",
   "metadata": {},
   "source": [
    "**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号\"算法美食屋\"下留言。作者时间和精力有限，会酌情予以回复。\n",
    "\n",
    "也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。\n",
    "\n",
    "![算法美食屋logo.png](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}