notebooks/From_Expressions_to_Machine_Learning_chapter.ipynb

{"nbformat": 4, "nbformat_minor": 0, "metadata": {"colab": {"name": "From_Expressions_to_Machine_Learning_chapter.ipynb", "provenance": [], "collapsed_sections": []}, "kernelspec": {"name": "python3", "display_name": "Python 3"}}, "cells": [{"cell_type": "markdown", "metadata": {"id": "sw9GOMU5PEAp"}, "source": ["# From Expressions to Machine Learning\n", "\n", "Copyright Luca de Alfaro, 2019-21.\n", "License: [CC-BY-NC-ND](https://creativecommons.org/licenses/by-nc-nd/4.0/).\n"]}, {"cell_type": "markdown", "metadata": {"id": "9295404ae500fbd2", "colab_type": "text"}, "source": "\nPrepared on: Thu Oct 28 11:35:32 2021\n\nThis is a book chapter; it is not a homework assignment.  \nDo not submit it as a solution to a homework assignment; you would receive no credit.\n"}, {"cell_type": "markdown", "metadata": {"id": "gezmsMI_TSwo"}, "source": ["## ML in a nutshell\n", "\n", "Optimization, and machine learning, are intimately connected.  At a very coarse level, ML works as follows. \n", "\n", "First, you come up somehow with a very complicated model $\\hat{y} = M(x, \\theta)$, which computes an output $\\hat{y}$ as a function of an input $x$ and of a vector of parameters $\\theta$.   In general, $x$, $y$, and $\\theta$ are vectors, as the model has multiple inputs, multiple outputs, and several parameters.  The model $M$ needs to be complicated, because only complicated models can represent complicated phenomena; for instance, $M$ can be a multi-layer neural net with parameters $\\theta = [\\theta_1, \\ldots, \\theta_k]$, where $k$ is the number of parameters of the model. \n", "\n", "Second, you come up with a notion of _loss_ $L$, that is, how badly the model is doing.  For instance, if you have a list of inputs $x_1, \\ldots, x_n$, and a set of desired outputs $y_1, \\ldots, y_m$, you can use as loss: \n", "\n", "$$\n", "L(\\theta) = \\sum_{i=1}^n |\\!|y_i - \\hat{y}_i |\\!| = \\sum_{i=1}^n |\\!| y_i - M({x}_i, \\theta)|\\!| \\; .\n", "$$\n", "\n", "Here, we wrote $L(\\theta)$ because, once the inputs $x_1, \\ldots, x_n$ and the desired outputs $y_1, \\ldots, y_n$ are chosen, the loss $L$ depends only on $\\theta$. \n", "\n", "Once the loss is chosen, you decrease it, by computing its _gradient_ with respect to ${\\theta}$.  Remembering that ${\\theta} = [\\theta_1, \\ldots, \\theta_k]$,\n", "\n", "$$\n", "\\nabla_{\\theta} L = \\left[ \\frac{\\partial L}{\\partial \\theta_1}, \\ldots, \n", "    \\frac{\\partial L}{\\partial \\theta_k} \\right] \\; .\n", "$$\n", "\n", "The gradient is a vector that indicates how to tweak ${\\theta}$ to decrease the loss.  You then choose a small _step size_ $\\delta$, and you update ${\\theta}$ via ${\\theta} := {\\theta} - \\delta \\nabla_{\\theta} L$.  This makes the loss a little bit smaller, and the model a little bit better.  If you repeat this step many times, the model will hopefully get (a good bit) better. \n", "\n", "## Autogradient\n", "\n", "The key to _pleasant_ ML is to focus on building the model $M$ in a way that is sufficiently expressive, and on choosing a loss $L$ that is helpful in guiding the optimization.  The computation of the gradient is done automatically for you.  This capability, called _autogradient_, is implemented in ML frameworks such as [Tensorflow](https://www.tensorflow.com), [Keras](https://keras.io), and [PyTorch](https://pytorch.org).  \n", "\n", "It is possible to use these advanced ML libraries without ever knowing what is under the hood, and how autogradient works.  Here, we will insted dive in, and implement autogradient.  \n", "\n", "Building a model $M$ corresponds to building an expression with inputs ${x}$, ${\\theta}$.  We will provide a representaton for expressions that enables both the calculation of the expression value, and the differentiation with respect to any of the inputs.  This will enable us to implement autogradient.  On the basis of this, we will be able to implement a simple ML framework. \n", "\n", "We say we, but we mean you.  _You_ will implement it; we will just provide guidance."]}, {"cell_type": "markdown", "metadata": {"id": "FEWinmYlKCDQ"}, "source": ["## Overview\n", "\n", "Let us give you a preview of how our ML framework works with an example. \n", "\n", "Assume for simplicity that our ML problem has only one input $x$, and one output $y$.  We are given many pairs $x_1, y_1, \\ldots, x_n, y_n$ of training input/output pairs, and we want to approximate the input-output relation via a parabola, whose coefficient we need to determine.  In particular, the predicted $\\hat{y}$ is given by: \n", "\n", "$$\n", "\\hat{y} = a x^2 + bx + c\n", "$$\n", "\n", "and the coefficients of the model $M$ are ${\\theta} = [a, b, c]$.  We build our model $M$ as an expression giving $\\hat{y}$ as an expression with variables $x, a, b, c$.  We write: \n", "\n", "    x = V()\n", "    a = V()\n", "    b = V()\n", "    c = V()\n", "\n", "    oy = a * x * x + b * x + c\n", "\n", "where `oy` is our _predicted_ output $\\hat{y}$.  We can then compute the loss, for instance using the quadratic distance of the output to the true $y$: \n", "\n", "    y = V() \n", "    loss = (y - oy) * (y - oy)\n", "\n", "In the code above, `oy` and `loss` are objects of class `Expr`.  If we have a particular input $x_0$, and desired output $y_0$, where both $x_0$ and $y_0$ are numbers, we can do: \n", "\n", "    x.assign(x0)\n", "    y.assign(v0)\n", "    loss.eval()\n", "\n", "and obtain the loss.  Then, we can backpropagate the gradient of the loss to the model variables, that is, compute $\\partial L / \\partial a$, $\\partial L / \\partial b$, $\\partial L / \\partial c$, via: \n", "\n", "    loss.zero_gradient()\n", "    loss.compute_gradient()\n", "\n", "We can then find the gradient $\\partial L / \\partial a$ in `a.gradient`, and similarly for the other variables. \n", "We can update the value of `a` simply via: \n", "\n", "    a.assign(a.eval() - delta * a.gradient)\n", "\n", "where `delta` is the learning step. To make all this work, what we need to do is endow our expressions with the methods for gradient propagation, namely, `zero_gradient` and `compute_gradient`. \n", "We will guide you through this. \n"]}, {"cell_type": "markdown", "metadata": {"id": "Hu206mH0Uli7"}, "source": ["## Our Expression Class\n", "\n", "We repeat here our definition of the `Expr` class from the last chapter. \n"]}, {"cell_type": "code", "metadata": {"id": "7kaIWA4jyj_k"}, "source": ["class Expr(object):\n", "    \"\"\"Abstract class representing expressions\"\"\"\n", "\n", "    def __init__(self, *args):\n", "        \"\"\"An object is created by passing to the constructor the children\"\"\"\n", "        self.children = args\n", "        self.value = None # The value of the expression\n", "        self.child_values = None # The values of the children; useful to have\n", "        self.gradient = None # This is where we will accummulate the gradient.\n", "\n", "    def eval(self):\n", "        \"\"\"Evaluates the expression.\"\"\"\n", "        # First, we evaluate the children.\n", "        self.child_values = [c.eval() if isinstance(c, Expr) else c\n", "                             for c in self.children]\n", "        # Then, we evaluate the expression itself.\n", "        self.value = self.op(*self.child_values)\n", "        return self.value\n", "\n", "    def op(self):\n", "        \"\"\"This operator must be implemented in subclasses; it should\n", "        compute self.value from self.values, thus implementing the\n", "        operator at the expression node.\"\"\"\n", "        raise NotImplementedError()\n", "\n", "    def __repr__(self):\n", "        \"\"\"Represents the expression as the name of the class, followed by the\n", "        children, and teh value.\"\"\"\n", "        return \"{}({})\".format(self.__class__.__name__,\n", "                                  ', '.join(repr(c) for c in self.children))\n", "\n", "    # Expression constructors\n", "\n", "    def __add__(self, other):\n", "        return Plus(self, other)\n", "\n", "    def __radd__(self, other):\n", "        return Plus(self, other)\n", "\n", "    def __sub__(self, other):\n", "        return Minus(self, other)\n", "\n", "    def __rsub__(self, other):\n", "        return Minus(other, self)\n", "\n", "    def __mul__(self, other):\n", "        return Multiply(self, other)\n", "\n", "    def __rmul__(self, other):\n", "        return Multiply(other, self)\n", "\n", "    def __truediv__(self, other):\n", "        return Divide(self, other)\n", "\n", "    def __rtruediv__(self, other):\n", "        return Divide(other, self)\n", "\n", "    def __neg__(self):\n", "        return Negative(self)\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "odSg1CMCy7rn"}, "source": ["import random\n", "import string\n", "\n", "class V(Expr):\n", "    \"\"\"Variable.\"\"\"\n", "\n", "    def __init__(self, value=None):\n", "        super().__init__()\n", "        self.children = []\n", "        self.value = random.gauss(0, 1) if value is None else value\n", "        self.name = ''.join(\n", "            random.choices(string.ascii_letters + string.digits, k=8))\n", "\n", "    def eval(self):\n", "        return self.value\n", "\n", "    def assign(self, value):\n", "        self.value = value\n", "\n", "    def __repr__(self):\n", "        return \"V(name={}, value={})\".format(self.name, self.value)\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "1LeV-zCkzABB"}, "source": ["class Plus(Expr):\n", "    def op(self, x, y):\n", "        return x + y\n", "\n", "class Minus(Expr):\n", "    def op(self, x, y):\n", "        return x - y\n", "\n", "class Multiply(Expr):\n", "    def op(self, x, y):\n", "        return x * y\n", "\n", "class Divide(Expr):\n", "    def op(self, x, y):\n", "        return x / y\n", "\n", "class Negative(Expr):\n", "    def op(self, x):\n", "        return -x\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "MsT06U-t-ppc"}, "source": ["vx = V(value=3)\n", "vy = V(value=2)\n", "e = vx - vy\n", "assert e.eval() == 1.\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "fGPO4Hhcd3BG"}, "source": ["## Implementing autogradient\n", "\n", "The next step consists in implementing autogradient.  Consider an expression $e = E(x_0, \\ldots, x_n)$, computed as function of its children expressions $x_0, \\ldots, x_n$.  \n", "\n", "The goal of the autogradient computation is to accumulate, in each node of the expression, the gradient of the loss with respect to the node's value.  For instance, if the gradient is $2$, we know that if we increase the value of the expression by $\\Delta$, then the value of the loss is increased by $2 \\Delta$. We accumulate the gradient in the field `self.gradient` of the expression. \n", "\n", "We say _accumulate_ the gradient, because we don't really do:\n", "\n", "    self.gradient = ...\n", "\n", "Rather, we have a method `e.zero_gradient()` that sets all gradients to 0, and we then _add_ the gradient to this initial value of 0: \n", "\n", "    self.gradient += ... \n", "\n", "We will explain later in detail why we do so; for the moment, just accept it. \n", "\n", "### Computaton of the gradient\n", "\n", "In the computation of the autogradient, the expression will receive as input the value $\\partial L / \\partial e$, where $L$ is the loss, and $e$ the value of the expression.  The quantity $\\partial L / \\partial e$ is the gradient of the loss with respect to the expression value. \n", "\n", "With this input, the method `compute_gradient` of an Expr $e$ must do the following:\n", "\n", "* It must _add_ $\\partial L / \\partial e$ to the gradient `self.gradient` of the expression. \n", "* It must call the method `operator_gradient`.  If the expression node has children $c_1, \\ldots, c_n$, the method `operator_gradient` must return the list of derivatives of $e$ with respect to its children $x_1, \\ldots, x_n$:\n", "\n", "  $$\n", "  \\left[ \\frac{\\partial e}{\\partial x_1}, \\ldots, \n", "    \\frac{\\partial e}{\\partial x_n} \\right]\n", "  $$\n", "\n", "  The method `operator_gradient` is implemented not directly in the class `Expr`, but for each specific operator, such as `Plus`, `Multiply`, etc: each operator knows how to compute the partial derivative of its output with respect to its children (its arguments).\n", "* It must propagate to each child $x_i$ the gradient \n", "\n", "  $$\n", "  \\frac{\\partial L}{\\partial e} \\cdot \\frac{\\partial e}{\\partial x_i} \\; , \n", "  $$\n", "\n", "  by calling the method `compute_gradient` of the child with argument $\\frac{\\partial L}{\\partial e} \\cdot \\frac{\\partial e}{\\partial x_i}$. "]}, {"cell_type": "code", "metadata": {"id": "9JkTBgU1FOQC"}, "source": ["def expr_operator_gradient(self):\n", "    \"\"\"This method computes the derivative of the operator at the expression\n", "    node.  It needs to be implemented in derived classes, such as Plus,\n", "    Multiply, etc.\"\"\"\n", "    raise NotImplementedError()\n", "\n", "Expr.operator_gradient = expr_operator_gradient\n", "\n", "def expr_zero_gradient(self):\n", "    \"\"\"Sets the gradient to 0, recursively for this expression\n", "    and all its children.\"\"\"\n", "    self.gradient = 0\n", "    for e in self.children:\n", "        if isinstance(e, Expr):\n", "            e.zero_gradient()\n", "\n", "Expr.zero_gradient = expr_zero_gradient\n", "\n", "def expr_compute_gradient(self, de_loss_over_de_e=1):\n", "    \"\"\"Computes the gradient.\n", "    de_loss_over_de_e is the gradient of the output.\n", "    de_loss_over_de_e will be added to the gradient, and then\n", "    we call for each child the method compute_gradient,\n", "    with argument de_loss_over_de_e * d expression / d child.\n", "    The value d expression / d child is computed by self.derivate. \"\"\"\n", "    pass # We will write this later.\n", "\n", "Expr.compute_gradient = expr_compute_gradient\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "tZz0DJpLrl-E"}, "source": ["Let us endow our classes `Plus` and `Multiply` with the `operator_gradient` method, so you can see how it works in practice."]}, {"cell_type": "code", "metadata": {"id": "0-rlK2Y4pEfY"}, "source": ["def plus_operator_gradient(self):\n", "    # If e = x + y, de / dx = 1, and de / dy = 1\n", "    return 1, 1\n", "\n", "Plus.operator_gradient = plus_operator_gradient\n", "\n", "def multiply_operator_gradient(self):\n", "    # If e = x * y, de / dx = y, and de / dy = x\n", "    x, y = self.child_values\n", "    return y, x\n", "\n", "Multiply.operator_gradient = multiply_operator_gradient\n", "\n", "def variable_operator_gradient(self):\n", "    # This is not really used, but it needs to be here for completeness.\n", "    return None\n", "\n", "V.operator_gradient = variable_operator_gradient\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "nNqqUC93qwXw"}, "source": ["Let us comment on some subtle points, before you get to work at implementing `compute_gradient`.  \n", "\n", "**`zero_gradient`:** First, notice how in the implementation of `zero_gradient`, when we loop over the children, we check whether each children is an `Expr` via isinstance(e, Expr).  In general, we have to remember that children can be either `Expr`, or simply numbers, and of course numbers do not have methods such as `zero_gradient` or `compute_gradient` implemented for them. \n", "\n", "**`operator_gradient`:** Second, notice how `operator_gradient` is not implemented in `Expr` directly, but rather, only in the derived classes such as `Plus`.  The derivative of the expression with respect to its arguments depends on which function it is, obviously.  \n", "\n", "For `Plus`, we have $e = x_0 + x_1$, and so: \n", "\n", "$$\n", "\\frac{\\partial e}{\\partial x_0} = 1 \\qquad \\frac{\\partial e}{\\partial x_1} = 1 \\; ,\n", "$$\n", "\n", "because $d(x+y)/dx = 1$.  Hence, the `operator_gradient` method of `Plus` returns\n", "\n", "$$\n", "\\left[ \\frac{\\partial e}{\\partial x_0}, \\: \\frac{\\partial e}{\\partial x_1}\\right] \\; = \\; [1, 1] \\; .\n", "$$\n", "\n", "For `Multiply`, we have $e = x_0 \\cdot x_1$, and so:\n", "\n", "$$\n", "\\frac{\\partial e}{\\partial x_0} = x_1 \\qquad \\frac{\\partial e}{\\partial x_1} = x_0 \\; ,\n", "$$\n", "\n", "because $d(xy)/dx = y$.  Hence, the `operator_gradient` method of `Plus` returns\n", "\n", "$$\n", "\\left[ \\frac{\\partial e}{\\partial x_0}, \\: \\frac{\\partial e}{\\partial x_1}\\right] \\; = \\; [x_1, x_0] \\; .\n", "$$\n", "\n", "\n", "**Calling `compute` before `compute_gradient`:** Lastly, a very important point: when calling `compute_gradient`, we will assume that `eval` has _already_ been called.  In this way, the value of the expression, and its children, are available for the computation of the gradient.  Note how in `Multiply.compute_gradient` we use these values in order to compute the partial derivatives. "]}, {"cell_type": "markdown", "metadata": {"id": "4bkCxtAvxoP2"}, "source": ["With these clarifications, we ask you to implement the `compute_gradient` method, which again must:\n", "\n", "* _add_ $\\partial L / \\partial e$ to the gradient `self.gradient` of the expression; \n", "* compute $\\frac{\\partial e}{\\partial x_i}$ by calling the method `operator_gradient` of itself; \n", "* propagate to each child $x_i$ the gradient $\\frac{\\partial L}{\\partial e} \\cdot \\frac{\\partial e}{\\partial x_i}$, by calling the method `compute_gradient` of the child $i$ with argument $\\frac{\\partial L}{\\partial e} \\cdot \\frac{\\partial e}{\\partial x_i}$."]}, {"cell_type": "code", "metadata": {"id": "for4fPAwrJgq"}, "source": ["### Exercise: Implementation of `compute_gradient`\n", "\n", "def expr_compute_gradient(self, de_loss_over_de_e=1):\n", "    \"\"\"Computes the gradient.\n", "    de_loss_over_de_e is the gradient of the output.\n", "    de_loss_over_de_e will be added to the gradient, and then\n", "    we call for each child the method compute_gradient,\n", "    with argument de_loss_over_de_e * d expression / d child.\n", "    The value d expression / d child is computed by self.compute_gradient. \"\"\"\n", "    ### BEGIN SOLUTION\n", "    raise NotImplementedError()\n", "    ### END SOLUTION\n", "\n", "Expr.compute_gradient = expr_compute_gradient\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "QRvO2iJ2rNEj"}, "source": ["Below are some tests."]}, {"cell_type": "code", "metadata": {"id": "NzZQq2lH5aA-"}, "source": ["## Tests for `compute_gradient`\n", "\n", "# First, the gradient of a sum.\n", "vx = V(value=3)\n", "vz = V(value=4)\n", "y = vx + vz\n", "assert y.eval() == 7\n", "y.zero_gradient()\n", "y.compute_gradient()\n", "assert vx.gradient == 1\n", "\n", "# Second, the gradient of a product.\n", "vx = V(value=3)\n", "vz = V(value=4)\n", "y = vx * vz\n", "assert y.eval() == 12\n", "y.zero_gradient()\n", "y.compute_gradient()\n", "assert vx.gradient == 4\n", "assert vz.gradient == 3\n", "\n", "# Finally, the gradient of the product of sums.\n", "\n", "vx = V(value=1)\n", "vw = V(value=3)\n", "vz = V(value=4)\n", "y = (vx + vw) * (vz + 3)\n", "assert y.eval() == 28\n", "y.zero_gradient()\n", "y.compute_gradient()\n", "assert vx.gradient == 7\n", "assert vz.gradient == 4\n", "\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "vpzAoZpC4GE0"}, "source": ["### Hidden tests for compute gradient\n", "\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "LFFby8YgFrNh"}, "source": ["## Why do we accumulate gradients?\n", "\n", "We are now in the position of answering the question of why we accumulate gradients.  There are two reasons. \n", "\n", "### Multiple variable occurrence\n", "\n", "The most important reason why we need to _add_ to the gradient of each node is that nodes, and in particular, variable nodes, can occur in multiple places in an expression tree.  To compute the total influence of the variable on the expression, we need to _sum_ the influence of each occurrence.  Let's see this with a simple example.  Consider the expression $y = x \\cdot x$.  We can code it as follows: \n"]}, {"cell_type": "code", "metadata": {"id": "bqohh6aU0l-3"}, "source": ["vx = V(2.) # Creates a variable vx and initializes it to 2.\n", "y = vx * vx\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "fSB5PnCjFAYg"}, "source": ["For $y = x^2$, we have $dy / dx = 2x = 4$, given that $x=2$.  How is this reflected in our code? \n", "\n", "Our code considers separately the left and right occurrences of `vx` in the expression; let us denote them with $vx_l$ and $vx_r$.  The expression can be written as $y = vx_l \\cdot vx_r$, and we have that $\\partial y / \\partial \\, vx_l = vx_r = 2$, as $vx_r = 2$.  Similarly, $\\partial y / \\partial \\, vx_r = 2$.  These two gradients are added to `vx.gradient` by the method `compute_gradient`, and we get that the total gradient is 4, as desired."]}, {"cell_type": "code", "metadata": {"id": "-GAPGN4uMOSS"}, "source": ["y.eval() # We have to call eval() before compute_gradient()\n", "y.zero_gradient()\n", "y.compute_gradient()\n", "print(\"gradient of vx:\", vx.gradient)\n", "assert vx.gradient == 4\n"], "execution_count": null, "outputs": [{"output_type": "stream", "text": ["gradient of vx: 4.0\n"], "name": "stdout"}]}, {"cell_type": "markdown", "metadata": {"id": "yTQG7bP7LTjT"}, "source": ["### Multiple data to fit\n", "\n", "The other reason why we need to tally up the gradient is that in general, we need to fit a function to more than one data point.  Assume that we are given a set of inputs $x_1, x_2, \\ldots, x_n$ and desired outputs $y_1, y_2, \\ldots, y_n$. Our goal is to approximate the desired ouputs via an expression \n", "\n", "$$\n", "y = e(x, \\theta)\n", "$$\n", "\n", "of the input, according to some parameters $\\theta$.  Our goal is to choose the parameters $\\theta$ to minimize the sum of the square errors for the points: \n", "\n", "$$\n", "L_{tot}(\\theta)\n", "= \\sum_{i=1}^n L_i(\\theta) \n", "= \\sum_{i=1}^n (e(x_i, \\theta) - y_i)^2 \n", "\\; ,\n", "$$\n", "\n", "where $L_i(\\theta) = (e(x_i, \\theta) - y_i)^2$ is the loss for a single data point.  The gradient $L_{tot}(\\theta)$ with respect to $\\theta$ can be computed by adding up the gradients for the individual points:\n", "\n", "$$\n", "\\frac{\\partial}{\\partial \\theta} L_{tot}(\\theta) \\;=\\;\n", "\\sum_{i=1}^n \\frac{\\partial}{\\partial \\theta} L_i(\\theta) \\; .\n", "$$\n", "\n", "To translate this into code, we will build an expression $e(x, \\theta)$ involving the input $x$ and the parameters $\\theta$, and an expression \n", "\n", "$$\n", "L = (e(x, \\theta) - y)^2\n", "$$\n", "\n", "for the loss, involving $x, y$ and $\\theta$.  We will then zero all gradients via `zero_gradient`. Once this is done, we compute the loss $L$ for each point $(x_i, y_i)$, and then the gradient $\\partial L / \\partial \\theta$ via a call to `compute_gradient`.  \n", "The gradients for all the points will be added, yielding the gradient for minimizing the total loss over all the points. "]}, {"cell_type": "markdown", "metadata": {"id": "ccBTGbcYS9lJ"}, "source": ["## Rounding up the implementation\n", "\n", "Now that we have implemented autogradient, as well as the operators `Plus` and `Multiply`, it is time to implement the remaining operators: \n", "\n", "* `Minus`\n", "* `Divide` (no need to worry about division by zero)\n", "* and the unary minus `Negative`. "]}, {"cell_type": "code", "metadata": {"id": "naOrgtVWgquM"}, "source": ["### Exercise: Implementation of `Minus`, `Divide`, and `Negative`\n", "\n", "def minus_operator_gradient(self):\n", "    # If e = x - y, de / dx = ..., and de / dy = ...\n", "    ### BEGIN SOLUTION\n", "    raise NotImplementedError()\n", "    ### END SOLUTION\n", "\n", "Minus.operator_gradient = minus_operator_gradient\n", "\n", "def divide_operator_gradient(self):\n", "    # If e = x / y, de / dx = ..., and de / dy = ...\n", "    ### BEGIN SOLUTION\n", "    raise NotImplementedError()\n", "    ### END SOLUTION\n", "\n", "Divide.operator_gradient = divide_operator_gradient\n", "\n", "def negative_operator_gradient(self):\n", "    # If e = -x, de / dx = ...\n", "    ### BEGIN SOLUTION\n", "    raise NotImplementedError()\n", "    ### END SOLUTION\n", "\n", "Negative.operator_gradient = negative_operator_gradient\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "DyVaBxC0h20V"}, "source": ["Here are some tests."]}, {"cell_type": "code", "metadata": {"id": "3rR5bBNH45pR"}, "source": ["### Tests for `Minus`\n", "\n", "# Minus.\n", "vx = V(value=3)\n", "vy = V(value=2)\n", "e = vx - vy\n", "assert e.eval() == 1.\n", "e.zero_gradient()\n", "e.compute_gradient()\n", "assert vx.gradient == 1\n", "assert vy.gradient == -1\n", "\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "uIzE3qmI91HL"}, "source": ["### Hidden tests for Minus\n", "\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "w9N0DYIxjdDe"}, "source": ["# Tests for `Divide`\n", "\n", "# Divide.\n", "vx = V(6)\n", "vy = V(2)\n", "e = vx / vy\n", "assert e.eval() == 3\n", "e.zero_gradient()\n", "e.compute_gradient()\n", "assert vx.gradient == 0.5\n", "assert vy.gradient == -1.5\n", "\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "lkx4SUGV_JLC"}, "source": ["### Hidden tests for Divide\n", "\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "BPDbGazX-JWl"}, "source": ["## Tests for `Negative`\n", "\n", "# Negative\n", "vx = V(6)\n", "e = - vx\n", "assert e.eval() == -6\n", "e.zero_gradient()\n", "e.compute_gradient()\n", "assert vx.gradient == -1\n", "\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "leiUDEca_oKX"}, "source": ["### Hidden tests for Negative\n", "\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "IN0NQijlCYHV"}, "source": ["## Optimization\n", "\n", "Let us use our ML framework to fit a parabola to a given set of points.  Here is our set of points:"]}, {"cell_type": "code", "metadata": {"id": "SHqT-fHZDtgI"}, "source": ["points = [\n", "    (-2, 2.7),\n", "    (-1, 3),\n", "    (0, 1.3),\n", "    (1, 2.4),\n", "    (3, 5.5),\n", "    (4, 6.2),\n", "    (5, 9.1),\n", "]\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "eqFIpL1lEg98"}, "source": ["Let us display these points."]}, {"cell_type": "code", "metadata": {"id": "ojGwBf5bEjHN"}, "source": ["import matplotlib\n", "import matplotlib.pyplot as plt\n", "\n", "matplotlib.rcParams['figure.figsize'] = (8.0, 3.)\n", "params = {'legend.fontsize': 'large',\n", "          'axes.labelsize': 'large',\n", "          'axes.titlesize':'large',\n", "          'xtick.labelsize':'large',\n", "          'ytick.labelsize':'large'}\n", "matplotlib.rcParams.update(params)\n", "\n", "def plot_points(points):\n", "    fig, ax = plt.subplots()\n", "    xs, ys = zip(*points)\n", "    ax.plot(xs, ys, 'r+')\n", "    plt.xlabel('x')\n", "    plt.ylabel('y')\n", "    plt.show()\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "NUwylufEF5kn"}, "source": ["plot_points(points)\n"], "execution_count": null, "outputs": [{"output_type": "display_data", "data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAewAAADWCAYAAAATvLEfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANJUlEQVR4nO3db4xldX3H8feHHaKW7Ypmx41iylgDDdqmK94mTRoKD2gQktambSqW/nnQFgPBphATfUCzs2BjTExtIojZdqtETBtikFDqn6TGxmii9W6CbYmU2iC6LRsHRWGBXal+++AO7Didhdntvefc37nvV3LCnXsu8/2e3Dn57O93/qWqkCRJ8+2MvhuQJEkvzMCWJKkBBrYkSQ0wsCVJaoCBLUlSAwxsSZIasNR3A89n9+7dtbKy0ncbkiR14tChQ49W1fJW6+Y6sFdWVhiPx323IUlSJ5I8fLJ1TolLktQAA1uSpAYY2JIkna7V1c5KGdiSJJ2u/fs7K2VgS5LUAANbkqRTsboKyWSBE69nPD2eeX685mg0Ki/rkiTNrQSmmKNJDlXVaKt1jrAlSWqAgS1J0unat6+zUga2JEmny8u6JEnSRga2JEkNMLAlSWqAgS1JUgMMbEmSGmBgS5LUAANbkqQGGNiSJDXAwJYkqQEGtiRJDTCwJUlqgIEtSVIDDGxJkhrQWWAnWUnyySSPJTmS5JYkS13VlySpZV2OsD8IfBt4JbAXuBi4tsP6kiQ1q8vAfg1wZ1Udq6ojwKeB13dYX5KkZnUZ2H8JXJnkJ5KcA1zOJLR/TJKrk4yTjNfW1jpsT5Kk+dVlYH+eyYj6ceAwMAbu3vyhqjpQVaOqGi0vL3fYniRJ86uTwE5yBpPR9F3AWcBu4GXAe7uoL0lS67oaYb8c+Cnglqo6XlXfAT4MXNFRfUmSmtZJYFfVo8BDwDVJlpKcDfwB8C9d1JckqXVdHsP+DeBNwBrwdeAZ4PoO60uS1KzOblxSVfcBl3RVT5KkIfHWpJIkNcDAliSpAQa2JEkNMLAlSWqAgS1JUgMMbEmSGmBgS5LUAANbkqQGGNiSJDXAwJYkqQEGtiRJDTCwJUlqgIEtSVIDDGxJkhpgYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYkSQ0wsCVJaoCBLUlSAwxsSZIaYGBLktQAA1uSpAYY2JKk6Vtd7buDwTGwJUnTt39/3x0MjoEtSVIDDGxJ0nSsrkIyWeDEa6fHp6LTwE5yZZKvJXkyyX8muajL+pKkGVpdharJAideG9hTsdRVoSS/ArwXeAvwz8Aru6otSVLrOgtsYD9wU1V9af3n/+qwtiSpS/v29d3B4HQyJZ5kBzAClpN8PcnhJLckeckWn706yTjJeG1trYv2JEnT5jT41HV1DHsPcCbwW8BFwF7gDcCNmz9YVQeqalRVo+Xl5Y7akyRpvnUV2E+v//cDVfVIVT0K/AVwRUf1JUlqWieBXVWPAYeB2vh2F7UlSRqCLi/r+jDw9iSvSPIy4Hrg3g7rS5LUrC7PEr8Z2A08CBwD7gT+vMP6kiQ1q7PArqpngGvXF0mSdAq8NakkdcnLnXSaDGxJ6pJPsdJpMrAlSWqAgS1Js+ZTrDQFqZrfy6FHo1GNx+O+25Ck6UlOPM1K2iTJoaoabbXOEbYkSQ3YdmAneX+SvbNsRpIGz6dY6TSdygh7B/CZJP+W5J1JXj2rpiRpsDxurdO07cCuqj8BXgW8i8nTtr6W5B+T/H6SnbNqUJIkneIx7Kr6YVXdW1VvBX4RWAY+AhxJ8tdJzplBj5IkLbxTCuwku5L8YZLPAZ8Hvszk+dYXAEeBT02/RUmStO17iSf5OHAZk6D+EHB3VR3fsP4G4PtT71CSJJ3Swz++BFxXVUe2WllVP0qyZzptSZKkjbYd2FX1vm185qn/XzuSJGkr3jhFkqQGGNiSJDXAwJYkqQEGtiRJDTCwJUlqgIEtSVIDDGxJkhpgYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYkSQ0wsCVJaoCBLUlSAzoP7CTnJTmW5I6ua0uS1Ko+Rti3Al/poa4kSc3qNLCTXAl8D/hsl3UlSWpdZ4GdZBdwE3DDC3zu6iTjJOO1tbVumpMkac51OcK+GThYVYef70NVdaCqRlU1Wl5e7qg1SZLm21IXRZLsBS4F3tBFPUmShqaTwAYuAVaAbyYB2AnsSPK6qrqwox4kSWpWV4F9APi7DT+/g0mAX9NRfUmSmtZJYFfVU8BTz/6c5ChwrKo8q0ySpG3oaoT9Y6pqtY+6kiS1yluTSpLUAANbkqQGGNiSJDXAwJYkqQEGtiRJDTCwJUlqgIEtSVIDDGxJkhpgYEuS1AADe4hWV/vuQJI0ZQb2EO3f33cHkqQpM7AlSWqAgT0Uq6uQTBY48drpcUkahFRV3z2c1Gg0qvF43Hcb7Ulgjr9XSdLWkhyqqtFW6xxhS5LUgMUK7EWZHt63r+8OJElTtlhT4k4VS5LmmFPikiQ1bviB7dnTkqQBcEpckqQ54ZS41DJngySxaIHt2dNqkbealcSiBbYjFUlSoxYrsKVWeLKkpE0W66QzqUWeLCktDE86kySpcQa2NO88WVISBrY0/zxuLYmOAjvJi5IcTPJwkieS3Jfk8i5qS5I0BF2NsJeAbwEXAy8FbgTuTLLSUX1Jkpq21EWRqnoSWN3w1r1JHgLeCHyjix4kSWpZL8ewk+wBzgfu32Ld1UnGScZra2vdNydJ0hzqPLCTnAl8DLi9qh7YvL6qDlTVqKpGy8vLXbcnSdJc6jSwk5wBfBT4AXBdl7UlSWpZJ8ewAZIEOAjsAa6oqme6qi1JUus6C2zgNuAC4NKqerrDupIkNa+r67DPBd4G7AWOJDm6vlzVRX1JklrX1WVdDwPpopYkSUPkrUklSWqAgS1JUgMMbEmSGmBgq10+xUrSAjGw1a79+/vuQJI6Y2BLktQAA1ttWV2FZLLAiddOj0sauFRV3z2c1Gg0qvF43HcbmlcJzPHfrySdqiSHqmq01TpH2JIkNcDAVrv27eu7A0nqjIGtdnncWtICMbAlSWqAgS1JUgMMbEmSGjDXl3UlWQMenuKv3A08OsXfN6/czmFxO4fF7RyWaW/nuVW1vNWKuQ7saUsyPtn1bUPidg6L2zksbuewdLmdTolLktQAA1uSpAYsWmAf6LuBjridw+J2DovbOSydbedCHcOWJKlVizbCliSpSQa2JEkNWLjATvKiJAeTPJzkiST3Jbm8775mIcl1ScZJjif5SN/9TEuSlyf5RJIn17/H3+m7p1kY6ve30YLtj3ckeSTJ40keTPJHffc0S0nOS3IsyR199zILSf5pffuOri//PuuaCxfYwBLwLeBi4KXAjcCdSVZ67GlW/ht4N/A3fTcyZbcCPwD2AFcBtyV5fb8tzcRQv7+NFml/fA+wUlW7gF8D3p3kjT33NEu3Al/pu4kZu66qdq4vPzPrYgsX2FX1ZFWtVtU3qupHVXUv8BAwuB2nqu6qqruB7/Tdy7QkOQv4TeDPqupoVX0BuAf4vX47m74hfn+bLdj+eH9VHX/2x/XltT22NDNJrgS+B3y2716GZOECe7Mke4Dzgfv77kXbcj7wP1X14Ib3vgoMcYS9cIa+Pyb5YJKngAeAR4BP9tzS1CXZBdwE3NB3Lx14T5JHk3wxySWzLrbQgZ3kTOBjwO1V9UDf/WhbdgKPb3rv+8BP9tCLpmgR9sequpbJ3+pFwF3A8ef/P5p0M3Cwqg733ciMvRP4aeAcJtdi/32Smc6YDC6w108EqJMsX9jwuTOAjzI5Fnpdbw2fpu1u5wAdBXZtem8X8EQPvWhKWt8fT0VV/XD9UM6rgWv67meakuwFLgXe33cvs1ZVX66qJ6rqeFXdDnwRuGKWNZdm+cv7UFWXvNBnkgQ4yOSkpSuq6plZ9zVt29nOgXoQWEpyXlX9x/p7P89Ap1AXwRD2x9O0xPCOYV8CrADfnHyt7AR2JHldVV3YY19dKCCzLDC4EfY23QZcAPxqVT3ddzOzkmQpyYuBHUx2mhcnafofaVX1JJOpxJuSnJXkl4A3MxmdDcoQv7+TGPz+mOQVSa5MsjPJjiSXAW9leCdlHWDyj5C968uHgH8ALuuzqWlLcnaSy57dJ5NcBfwy8OlZ1l24wE5yLvA2Jn9MRzZcQ3dVz63Nwo3A08C7gN9df31jrx1Nx7XAS4BvA38LXFNVQxxhD/X7e84C7Y/FZPr7MPAY8D7gT6vqnl67mrKqeqqqjjy7MDmEdayq1vrubcrOZHLJ5RqTZ2G/Hfj1TSfDTp33EpckqQELN8KWJKlFBrYkSQ0wsCVJaoCBLUlSAwxsSZIaYGBLktQAA1uSpAYY2JIkNcDAliSpAQa2JACSvDbJd5NcuP7zq5KsdfGcX0kvzFuTSnpOkj8GrgdGwCeAf62qd/TblSQwsCVtkuQe4DVMHljxC1V1vOeWJOGUuKT/66+AnwU+YFhL88MRtqTnJNkJfBX4HHA58HNV9d1+u5IEBrakDZIcBHZW1VuSHADOrqrf7rsvSU6JS1qX5M3Am4Br1t+6AbgwyVX9dSXpWY6wJUlqgCNsSZIaYGBLktQAA1uSpAYY2JIkNcDAliSpAQa2JEkNMLAlSWqAgS1JUgMMbEmSGvC/6lNqKQtODBAAAAAASUVORK5CYII=\n", "text/plain": ["<Figure size 576x216 with 1 Axes>"]}, "metadata": {"tags": [], "needs_background": "light"}}]}, {"cell_type": "markdown", "metadata": {"id": "W2F5nQ8CGNpK"}, "source": ["To fit a parabola to these points, we will build an `Expr` that represents the equation $\\hat{y} = ax^2 + bx + c$, where $\\hat{y}$ is the value of $y$ predicted by our parabola. \n", "If $\\hat{y}$ is the predicted value, and $y$ is the observed value, to obtain a better prediction of the observations, we minimize the loss $L = (\\hat{y} - y)^2$, that is, the square prediction error. \n", "Written out in detail, our loss is:\n", "\n", "$$\n", "    L \\;=\\; \\left( y - \\hat{y}\\right)^ 2 \\;=\\; \\left( y - (ax^2 + bx + c) \\right)^2 \\; .\n", "$$\n", "\n", "Here, $a, b, c$ are parameters that we need to tune to minimize the loss, and obtain a good fit between the parabola and the points. \n", "This tuning, or training, is done by repeating the following process many times:\n", "\n", "* Zero the gradient\n", "* For each point:\n", "    * Set the values of x, y to the value of the point.\n", "    * Compute the expression giving the loss.\n", "    * Backpropagate.  This computes all gradients with respect to the loss, and in particular, the gradients of the coefficients $a, b, c$. \n", "* Update the coefficients $a, b, c$ by taking a small step in the direction of the negative gradient (negative, so that the loss decreases)."]}, {"cell_type": "code", "metadata": {"id": "ye0Vy0IPF706"}, "source": ["va = V(value=0.)\n", "vb = V(value=0.)\n", "vc = V(value=0.)\n", "vx = V(value=0.)\n", "vy = V(value=0.)\n", "\n", "oy = va * vx * vx + vb * vx + vc\n", "\n", "loss = (vy - oy) * (vy - oy)\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "eHuR9hs-B_xi"}, "source": ["## Implementing learning: the `fit` function.\n", "\n", "The `fit` function minimizes the loss of fitting a series of $(x, y)$ points with a function.  Precisely, the `fit` function takes as input: \n", "\n", "* An expression `loss`, which gives the loss as a function of the values of two variables, `vx` and `vy`, as well as a list of parameters `params`.  In the above example, `params = [va, vb, vc]`. \n", "* A list `points` of $(x, y)$-pairs, such as `[(2, 3), (3.2, 4.1)]`. \n", "* A list `params` of parameters to be optimized.  In the above example, `params = [va, vb, vc]`. \n", "* A learning-step size `delta`. \n", "* A number of iterations `num_iterations`.\n", "\n", "Below, implement the \"for each point\" part of the above informal description.  Hint: this takes about 4-5 lines of code.  As a reminder, these are the operations you need to use: \n", "\n", "* For a variable `v`, you can assign it a value `x` by saying `v.assign(x)`.  If you want to assign it a `4`, you do `v.assign(4)`, and so forth (look also at the example at line 13). \n", "* Forward propagation: for the loss `loss`, you can compute its value via `loss.compute()`. \n", "* Backward propagation: for the loss `loss`, you can perform gradient backpropagation via `loss.compute_gradient()`. "]}, {"cell_type": "code", "metadata": {"id": "r0SwTd9v_uXu"}, "source": ["def fit(loss, points, params, delta=0.0001, num_iterations=4000):\n", "    \"\"\"\n", "    @param loss: expression giving the loss as a function of variables and parameters.\n", "    @param points: list of (x, y) values to which we have to fit the expression.\n", "    @param params: list of parameters whose value we can tune.\n", "    @param delta: learning step size.\n", "    @param num_iterations: number of learning iterations.\n", "    \"\"\"\n", "\n", "    for iteration_idx in range(num_iterations):\n", "        loss.zero_gradient()\n", "        total_loss = 0.\n", "        for x, y in points:\n", "            ### You need to implement here the computaton of the\n", "            ### loss gradient for the point (x, y).\n", "            total_loss += loss.value\n", "        if (iteration_idx + 1) % 100 == 0:\n", "            print(\"Loss:\", total_loss)\n", "        for vv in params:\n", "            vv.assign(vv.value - delta * vv.gradient)\n", "    return total_loss\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "Y2MPSUxKuMXk"}, "source": ["### Exercise: Implementation of `fit`\n", "\n", "def fit(loss, points, params, delta=0.0001, num_iterations=4000):\n", "    \"\"\"\n", "    @param loss: expression giving the loss as a function of variables and parameters.\n", "    @param points: list of (x, y) values to which we have to fit the expression.\n", "    @param params: list of parameters whose value we can tune.\n", "    @param delta: learning step size.\n", "    @param num_iterations: number of learning iterations.\n", "    \"\"\"\n", "\n", "    for iteration_idx in range(num_iterations):\n", "        loss.zero_gradient()\n", "        total_loss = 0.\n", "        for x, y in points:\n", "            ### BEGIN SOLUTION\n", "            raise NotImplementedError()\n", "            ### END SOLUTION\n", "            total_loss += loss.value\n", "        if (iteration_idx + 1) % 100 == 0:\n", "            print(\"Loss:\", total_loss)\n", "        for vv in params:\n", "            vv.assign(vv.value - delta * vv.gradient)\n", "    return total_loss\n"], "execution_count": null, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "VHF8lbVaBWV6"}, "source": ["Let's train the coefficients `va`, `vb`, `vc`: "]}, {"cell_type": "code", "metadata": {"id": "YpAougBKF96w"}, "source": ["lv = fit(loss, points, [va, vb, vc])\n", "assert lv < 2.5\n"], "execution_count": null, "outputs": [{"output_type": "stream", "text": ["Loss: 15.691480263172831\n", "Loss: 13.628854145973717\n", "Loss: 11.95710461470721\n", "Loss: 10.577288435238483\n", "Loss: 9.421709915417068\n", "Loss: 8.442894724380599\n", "Loss: 7.606627717852742\n", "Loss: 6.887533665385938\n", "Loss: 6.266252079473012\n", "Loss: 5.727613949776414\n", "Loss: 5.259450432876111\n", "Loss: 4.851802101057622\n", "Loss: 4.4963837663728965\n", "Loss: 4.18621382152429\n", "Loss: 3.9153507183966223\n", "Loss: 3.6787002636071753\n", "Loss: 3.471870598609928\n", "Loss: 3.2910600093895632\n", "Loss: 3.13296792199636\n", "Loss: 2.9947227347465053\n", "Loss: 2.8738222326925422\n", "Loss: 2.768083672161589\n", "Loss: 2.6756014919475786\n", "Loss: 2.594711177586694\n", "Loss: 2.523958185206276\n", "Loss: 2.462071090153853\n", "Loss: 2.4079383059994615\n", "Loss: 2.360587848696134\n", "Loss: 2.3191697158813342\n", "Loss: 2.28294052348259\n", "Loss: 2.251250098021605\n", "Loss: 2.2235297678961032\n", "Loss: 2.199282133501638\n", "Loss: 2.1780721263873506\n", "Loss: 2.1595191931327227\n", "Loss: 2.1432904612844434\n", "Loss: 2.1290947632296184\n", "Loss: 2.1166774098449848\n", "Loss: 2.105815619569577\n", "Loss: 2.096314520528721\n"], "name": "stdout"}]}, {"cell_type": "markdown", "metadata": {"id": "AFyhpqOKFLMm"}, "source": ["Let's display the parameter values after the training:"]}, {"cell_type": "code", "metadata": {"id": "QcaWvOl7H_Wd"}, "source": ["print(\"a:\", va.value, \"b:\", vb.value, \"c:\", vc.value)\n"], "execution_count": null, "outputs": [{"output_type": "stream", "text": ["a: 0.2676799989525098 b: 0.09448237346204012 c: 1.9726742011455856\n"], "name": "stdout"}]}, {"cell_type": "markdown", "metadata": {"id": "cyQrkGGhFfXX"}, "source": ["Let's display the points, along with the fitted parabola."]}, {"cell_type": "code", "metadata": {"id": "9KDgM_6NGMgH"}, "source": ["import numpy as np\n", "\n", "def plot_points_and_y(points, vx, oy):\n", "    fig, ax = plt.subplots()\n", "    xs, ys = zip(*points)\n", "    ax.plot(xs, ys, 'r+')\n", "    x_min, x_max = np.min(xs), np.max(xs)\n", "    step = (x_max - x_min) / 100\n", "    x_list = list(np.arange(x_min, x_max + step, step))\n", "    y_list = []\n", "    for x in x_list:\n", "        vx.assign(x)\n", "        oy.eval()\n", "        y_list.append(oy.value)\n", "    ax.plot(x_list, y_list)\n", "    plt.xlabel('x')\n", "    plt.ylabel('y')\n", "    plt.show()\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "cllVWssOH1zD"}, "source": ["plot_points_and_y(points, vx, oy)\n"], "execution_count": null, "outputs": [{"output_type": "display_data", "data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAewAAADWCAYAAAATvLEfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV9Z3/8dcn+56QhRCWEJawQxABW0SxLnWpWqfutTPV6thqbasz7bT+pgpa2452sZvasaXoSFtH0aq17lZbQVFAEAhrwr5kJ3tucnPv9/dHAkUGEDC55y7v5+NxHyS5J/l+Djcn7/v9nu/5HnPOISIiIuEtzusCRERE5KMpsEVERCKAAltERCQCKLBFREQigAJbREQkAiiwRUREIkCC1wUcTX5+vispKfG6DBERkZBYsWJFnXOu4HDPhXVgl5SUsHz5cq/LEBERCQkz236k5zQkLiIiEgEU2CIiIhFAgS0iIhIBFNgiIiInat68kDWlwBYRETkBTyzfyS9e3Riy9sJ6lriIiEi4cc7xwBsV/PiVTZw2dALdgSAJ8f3f/1UPW0RE5BgFgo47v7uAH7+yiUvK32D+ortJSIgHs34fHlcPW0RE5Bj4/AFufXwVLwUK+fLpI/n2Dy4gLv4n4FxI2ldgi4iIfISmdj//+thy3tvawB0XTuD62SNCXoMCW0RE5Ch2N3bwxd+9x476dn5x9UlcXDb4H0/OnRuyOhTYIiIiR7BuTzPXLniPDn+AR780k0+OyvvwBiG8rEuBLSIichiLN9fxlYUryExJYNFXZjF2UKan9SiwRUREDrFoxS6+89RqRhVk8MiXZlCUnep1SQpsERGR/Zxz/OL1Cu5/bROnjs7joS+cTFZKotdlAQpsERERAPyBILc/vYZFK3Zx6bSh/PBzk0lKCJ/lShTYIiIS85p9fr76+/d5a3Md3zirlFvPLsXMvC7rQxTYIiIS03Y3dvClBcuorG3lvsumcMX0YV6XdFgKbBERiVmrdzVy/aPL8fkD/M+XZjJrdL7XJR2RAltERGLSK+VVfP3xleRnJPOHG06htNDby7Y+igJbRERiinOO3761lR+8uJ4pQ3P47b9MpyAz2euyPpICW0REYoY/EOTOZ8v543s7uGDyIH5y+VRSk+K9LuuYKLBFRCQmNHX0zARfXFHHVz81in8/ZyxxceE1E/xoFNgiIhL1tte3cf2jy9le38aPLpvC5WE6E/xoQnZFuJmVmNkLZrbPzKrM7FdmpjcMIiLSr5ZuqeezDyyhrrWTx64/JSLDGkIY2MCDQA1QBEwF5gA3h7B9ERGJMf+7bAf/PP9d8tKTeParp/KJkXkf/U1hKpQ93BHAr5xzPqDKzF4CJoawfRERiRGBoOO/XlzPb97aymml+fzq89PITg2PNcFPVCh72D8DrjKzNDMbApwPvBTC9kVEJAY0+/xc/+gyfvPWVr74yeEsuHZGxIc1hLaH/XfgRqAZiAceBZ45dCMzu7F3O4qLi0NYnoiIRLqtdW3c8Ogytte38/1/msQ1pwz3uqQ+E5IetpnF0dObfhpIB/KBAcC9h27rnHvYOTfdOTe9oKAgFOWJiEgUWLy5jkseWEJDWxcLbzglqsIaQjckngsU03MOu9M5Vw8sAC4IUfsiIhKlnHMsWLKVLy54j0FZKTx3y+yInlx2JCEJbOdcHbAVuMnMEswsB/gisDoU7YuISHTq7A7wH4tWc9ef13HWuIE8dfMshuWmeV1WvwjlpLPPAecBtUAF4AduC2H7IiISRWqafVz18FKeXLGLb5xVyq+/cDIZydG7vEfI9sw5two4I1TtiYhI9Fq5Yx9fWbiCFl83v/7CNM6bVOR1Sf0uet+KiIhIVPrfZTu445lyCrOTefrmWYwblOV1SSGhwBYRkYjQ1R3k7ufLWbh0B6eV5vPLq08iJy3J67JCRoEtIiJhr6bFx80L32f59n18ec5I/uPcccRH0J22+oICW0REwtqK7Q3ctPB9Wnzd/PLqk7iobLDXJXlCgS0iImHJOcfCpdu5+/l1DM5J5X+unxkz56sPR4EtIiJhx+cP8J9/WstT7+/izHEDuf/KqVGxHvjHocAWEZGwsr2+ja8sfJ/1e5u59exSvn5mKXExdr76cBTYIiISNl5bV81tT6wizowF187gU+MGel1S2FBgi4iI5wJBx09f3cgDb1QyaUgWD11zctQuMXqiFNgiIuKp2pZOvvH4St6urOfqmcOYe9FEUhLjvS4r7CiwRUTEM+9tbeCWP7xPU4ef+y6bwhXTh3ldUthSYIuISMg553j471u47+WNFOem8eiXZjK+KHYv2ToWCmwREQmpxvYuvvnkB7y2voYLJg/i3kunkJkS25dsHQsFtoiIhMz7O/bxtT+spKbFx9yLJnDtrBLMdMnWsVBgi4hIv3POMX/xVv7rxQ0U5aSw6CuzKBuW43VZEUWBLSIi/WpfWxffWtQzBH7uxELuu6ws5lctOxEKbBER6TfvbW3gG4+vpL61S0PgH5MCW0RE+lwg6HjwjQruf20TxblpPH3zLCYNyfa6rIimwBYRkT5V3ezjtv9dxduV9VxcNpjv/9MkzQLvAwpsERHpM6+vr+abT36Azx/k3ksnc8X0YRoC7yMKbBER+dg6uwP88IUNPPL2NsYXZfHLq09i9MAMr8uKKgpsERH5WDZXt/D1x1exfm8z151awrfPG6e1wPuBAltERE6Ic46F7+7gnufXkZGcwPwvTues8YVelxW1FNgiInLc6ls7+fZTa3htfTVzxhTwo8unMDAzxeuyopoCW0REjssbG2v41pOrae7wc8eFE7huVglxcZpY1t/ivC5AREQiQ0dXgDufXct1C5aRm57IM189letnjzh8WM+bF/L6op0557yu4YimT5/uli9f7nUZIiIxb+3uJr7x+Eoqa9u4fvYIvnXu2KNPLDODMM6XcGVmK5xz0w/3nIbERUTkiLoDQR56s5Kfv76ZvIwkFl5/CrNL870uKyaFdEjczK4ys/Vm1mZmlWZ2WijbFxGRY7eltpXLfv0OP3l1ExdMLuKVW+ccPaznzevpWe9fKGX/xxoe7xMhGxI3s3OA3wJXAu8BRQDOud1H+h4NiYuIhF4w6Pj9u9v5wQsbSEqI455LJnFR2eDj+yEaEj8h4TIkfhdwt3Nuae/nRwxqERHxxu7GDr69aDWLK+qYM6aA+y6bQmGWLtcKByEJbDOLB6YDz5lZBZACPAN8yznXcci2NwI3AhQXF4eiPBGRmOec48nlu/je8+sIOscPPzeZq2Z8jHXA587t2wIlNEPiZjaYnh71CuAiwA88C7zpnPvPI32fhsRFRPpfdbOP259ew1831HDKiFx+fHkZw3LTvC4rJoXDkPj+XvQvnXN7e4v6KfBd4IiBLSIi/cc5x1Pv7+buP5fTFQhy54UTuFaLoIStkAS2c26fme0CDu7OazaCiIhHDu5VzygZwH2XlTEiP93rsuQoQjnpbAHwNTN7iZ4h8duA50PYvohIzHPO8cTyndzzl/X41auOKKEM7O8B+cAmwAc8AXw/hO2LiMS0nQ3t3P70GhZX1HHKiFzuvXQKJepVR4yQBbZzzg/c3PsQEZEQCQQdj72zjfte3ogB91wyic/PLFavOsJoaVIRkVCaNy+kK39tqm7hO0+t5v0djcwZU8APPjeZITmpIWtf+o5u/iEiEkohWgGsszvAA29U8tCbFWQkJ3DnRRO4ZOqQE7+uWkIiHC7rEhGREFm2rYHbn15DRU0rn506mDsvnEBeRrLXZcnHpPthi4j0txDdFKOp3c/tT6/m8l+/Q0dXgAXXzuDnV52ksI4SGhIXEQmlfhgSd87x/Oq93PXndTS0dXL97BHcds4Y0pI0iBppNCQuIhKltte3ccez5fx9Uy2Th2TzyHUzmDQk2+uypB8cc2Cb2f3Ao865Vf1Yj4hIdOujm2J0dgf4779t4VdvVJAUH8e8iybwz58sIV6XakWt4+lhxwMvm1kt8Bjwe+fcrv4pS0QkSvXBeeslFXXc8exattS28ZkpRdx54QTdAjMGHPOkM+fc14HBwHeAqcB6M3vNzP7FzDL6q0AREelR1eTjq394n2t++y6BoOOR62bwwOenKaxjxHGdw3bOBehZ//t5M5sI/AF4BHjQzB4H5jrndvd5lSIiMcwfCPLIkm387LVNdAcdt509hi/PGUlKYrzXpUkIHVdgm1kWcDnwBWAK8BQ9S43uAP4deLH36yIi0gcWb65j3p/Lqahp5cxxA5l30USK83Sv6lh0PJPOFgHnAn8Hfg0845zrPOj5fwOa+rxCEZEYtLuxg+//ZR0vrKmiODeN3/7LdM6eUOh1WeKh4+lhLwVucc5VHe5J51zQzPTbJCLyMfj8PbO/H/pbBQD/fs4Y/vV0DX/LcQS2c+7Hx7BN+8crR0QkNjnneGFNFT94YT27Gzv4zOQi/t9nxutGHXKAFk4REfFY+Z4m7v7zOt7d2sD4oix+ckUZnxiZ53VZEmYU2CIiHqlp8fGTlzfxxIqd5KQmcs8lk7h6ZrEWP5HDUmCLiISYzx9g/uKtPPhGBV2BINefOoKvnVVKdmqi16VJGFNgi4iESDDoeO6DPfzo5Y3sbuzg0xMKuf2C8YzIT/e6NIkACmwRkRBYuqWe7/9lPWt2NzFpSBY/unwKs0ble12WRBAFtohIP9pc3cK9L23gtfU1DM5O4f4ry/hs2RDidJ5ajpMCW0SkH1Q1+bj/1U08uWIn6UkJfOvcsVw/e4Sup5YTpsAWEelDTR1+/vtvlfxuyVYCQce1s0Zwy5mjyU1P8ro0iXAKbBGRPtDRFeDRd7bx0JuVNPv8XFw2mG9+eizDcrXut/QNBbaIyMfQ1R3kieU7+eVfN1Pd3MmnxhbwrXPHMWFwltelSZRRYIuInIBA0PGnlbv5+eub2NnQwcnDB/CLq07iFK1QJv1EgS0ichyCQccLa/dy/6ubqKxtY9KQLO6+bhJnjCnATDO/pf8osEVEjkEw6HhlXRX3v7qZjdUtjB6YwUPXTOO8SYMU1BISCmwRkaNwzvHqump+9tpm1u1tZmRBOj+/aioXThmsNb8lpEIe2GZWCqwBFjnnvhDq9kVEjsX+HvXPX69g/d5mSvLSuP/KMi4uG6KgFk940cN+AFjmQbsiIh8pEHS8uHYvv/prBRuqWhiZn85Pryjj4rLBJMTHeV2exLCQBraZXQU0Am8Do0PZtojI0fgDQZ5dtYcH36xgS20bIwvS+dmVU7moTEPfEh5CFthmlgXcDZwJ3HCU7W4EbgQoLi4OTXEiErN8/gBPrtjFf/+tkl37OhhflMUDn++ZTKaglnASyh7294D5zrldR5tR6Zx7GHgYYPr06S5EtYlIjGn2+Vm4dDu/W7yNutZOpg7L4a6LJ3LmuIGa9S1hKSSBbWZTgbOBk0LRnojIkdQ0+1jw9jYWvrOdls5uTh9TwE1zRvGJkbkKaglroephnwGUADt6D4gMIN7MJjjnpoWoBhGJYRU1LTz89y08s3IP3cEg508q4qYzRjFpSLbXpYkck1AF9sPA4wd9/k16AvymELUvIjHIOcc7W+qZ/9ZWXt9QQ0piHFfOGMYNp41geF661+WJHJeQBLZzrh1o3/+5mbUCPudcbSjaF5HY0tUd5C9r9vDbt7ZSvqeZvPQkvnFWKV+cVaLbXErE8mSlM+fcPC/aFZHoVt/ayR/e3cFjS7dT09LJ6IEZ/NfnJnPJSUNISYz3ujyRj0VLk4pIxFu3p5kFS7by7Ad76OoOclppPvdeNoU5pQXE6dIsiRIKbBGJSP5AkJfWVvE/72xj2bZ9pCbGc/nJQ7nu1BJGD8z0ujyRPqfAFpGIUtXk44/v7eCP7+2gpqWT4tw0/vOC8VwxfRjZaYlelyfSbxTYIhL2gkHHkso6Fi7dzmvrawgEHXPGFHDvpSXMGaNhb4kNCmwRCVt1rZ0sWrGLx9/bwbb6dnLTk7jhtBFcM3M4xXlpXpcnElIKbBEJK8GgY3FFHY8v28Er5dV0Bx0zS3K59ewxnD95EMkJmu0tsUmBLSJhYWdDO4tW7GLRil3sbuxgQFoi184q4aqZxYwemOF1eSKeU2CLiGc6ugK8XF7FohW7WFJZB8Ds0fl85/xxfHpioXrTIgdRYItISAWDjuXb97FoxU5eWFNFa2c3QwekcutZY7j05CEMHaBz0yKHo8CORvPm9TxEwkhFTSvPrNzNM6t2s2tfB2lJ8XxmchGXnjyUmSW5mukt8hHMufC95fT06dPd8uXLvS4j8phBGL+uEjuqmnw8v3oPz67aw5rdTcQZzC4t4JKpgzlv0iDSktRnEDmYma1wzk0/3HM6WkSkTzW0dfHi2r08t2oP721rwDmYPCSbOy6cwEVlRQzMTPG6RJGIpMCOFvPmwV13/eNz6x1enDtXw+PS7/a1dfFyeRV/WbOXtyvrCQQdIwvSufWsMVxUVsTIAs3yFvm4NCQejTQkLiFQ29LJK+uqeGltFe9U1tMddAzPS+Mzk4u4YHIREwdnYabz0iLHQ0PiwJPLd/LcB3v49IRCzpkwiEHZGpYTOV47G9p5ZV01L5dXsax3uLskL40bThvJhVMU0iL9KWYCG2BXxS7u2FzHHc+WUzY0m3MmFHL2hELGFmZG1x+ZuXO9rkCiRDDoKN/TzKvrq3mlvIoNVS0AjC3M5OtnlnL+5EHRd/yIhKmYGhJ3ZlRWN/NyeTWvrKvmg52NAAzJSeXs8QM5a3whM0fk6kb3EtPau7pZUlHPXzdU8/r6GmpaOokzmD48l09PLOScCYUMz0v3ukyRqHS0IfGYCuxDz+1WN/v464YaXl9fzeKKOnz+IKmJ8cwalccZ4wZyxpgChuVqEQeJbs45tta18ebGWt7YWMO7Wxvo6g6SkZzA6WPyOWtcIWeMLSAvI9nrUkWiXmwH9qGzp/c7ZPa0zx/g7cq6A3+0djZ0ADCyIJ3TSws4fUw+nxiZp+tGJSo0+/y8XVHPW5treWtzHTsa2gEYVZDOGWMH8qmxA5k5IpekhDiPKxWJLbEd2Ac7xtnTzjm21LXxxoYa3tpcx7tb6/H5gyTGGycVD2D26HxOHZ3HlKE5JMbrD5qEv87uAO9vb+TtyjqWVNTxwa4mAkFHRnICnxyVx+ml+ZwxdqBGlEQ8psDe7wQvd/L5Ayzfto+3NteypLKO8j3NOAcZyQnMKBnAJ0bm8clReUwoyiJBAS5hoKs7yJrdjSzd0sDSLfUs29aAzx8kzmDK0Bxmj87n9DEFnFSsN50i4USXde13grOnUxLjmV2az+zSfKBnkYh3ttSzpKKOpVvqeWNjLQCZyQmcXDKAGSW5zByRy5Sh2brbkHx8x7A2fEdXgJU79rFs2z6WbWtgxfZ9dPgDAIwblMlVM4o5dXQ+p4zMJSslsf9rFpE+F1s97H5S0+xj6dYG3qns6clU1LQCkJQQR9nQbKYNH8DJxQOYNnwA+Zq4I8frMCND1c0+Vmzfd+CxdncT3UGHGYwblMUpI3L5xMhcZo7IIzc9yaPCReR4aUg8xBrauli2rYFlWxtYsaPnj6k/0PP/XJybxtRhOZQNy2HqsBwmDs7SZWRyVO1JKazduIdVO/examcjH+xsYndjz6TI5IQ4yobmcHLJAGaW5DJt+ACyU9WDFolUCmyP+fwB1u5uYvn2fXyws5FVOxvZ2+QDICHOGFOYyeQh2Uwems2kIdmMG5SpEI9RbZ3dbKhqYe1v/sjqlZtZM6iUiryhBON6fh+GWidTJ5cwrXfEZkJRlmZyi0QRBXYYqm72sXJHI2t2N7J6VxNrdjfR2O4HIM5gVEEGEwdnMb4oi3FFWYwflElBZrJWlIoSzjn2NvnYUNXMhqoW1u1pZt3eZrbWtR0Y/c7PSGbK0Gwmzf8ZZQ//lLJhOTqlIhLlFNgRwDnHrn0dlO9pYt2eZsp7H1XNvgPbDEhLZOygTEoHZjKmMIPRAzMZPTCD/IwkBXmYcs5R09LJ5upWNte0sLmmlc3VLWyoaqHF131gu6EDUpk4OIsJRdlMGJzF5CHZFGb1vkHTzVxEYoZmiUcAM2NYbhrDctM4b1LRga/va+tiQ1ULG3t7YpuqW3hm5W5aOv/xxz4zJYFRBRmMLEhnZH46JfnplOSlU5yXphnBIdLY3sW2+na217exra6dLXWtbKltY2tdG60HvVbZqYmMKczgs1MHM3ZQFuMGZTKmMPPo5521NryIEKIetpklAw8CZwO5QCVwu3PuxaN9Xyz1sI+Hc47q5k42VbdQWdtKZW1POFTWtlLd3PmhbQekJVKcm8bQ3DSGDUhj6IBUhuSkMjgnlcE5KWQq0D+Sc47mjm52N3awp7GDPU0d7GxoZ9e+Dnbua2dnQwdNHf4D25vB4OxURhakH3gjNXpgBqUDMzUaIiJHFQ497ARgJzAH2AFcADxhZpOdc9tCVEPUMDMGZacwKDuF08cUfOi59q5utu/v6dW3s72+nV372inf3cQr5VUHZqvvl5mcQGF2CoVZyRRmpVCYlUJBRjIFmT2P/Ixk8tKTyE5NJC4uuoImEHQ0tnfR0NZFbUsnta2dB/6tae6kqslHdYuPqiYf7V2BD31vUkIcQwekMmxAGmVDcxiRn87wvHSG56VRnJumSYMi0uc8O4dtZquBu5xzTx1pG/Ww+1Yg6Khp8bGn0ceexg72NnWwp9FHdbOPqmYf1U0+alo66Q7+39+JOIMBaUkM6A3v7NREclITyUpNJCslgYyUBDKSE8lISSA9KZ7UpHjSkhJIS4onJSGepIQ4khPiSE6MIyEujsR4O+6epnOO7qCjqztIV3eQzu4gnd0BfP4g7V3ddHQFaO8K0NbVTWtnN62+nn9bfN00tnfR1OGnqcNPY4effW1dNHb4D3tqODHeGJjZ84Zo/xuZITmpFGX3jEoMzkmlICM56t7AiIj3wqGH/SFmVgiMAcoP89yNwI0AxcXFIa4susXHGUXZPcFz8vABh90mGHQ0dfipbe2krre32dDW9aFHU4ef6mYfG6taaO7w09rVfUJzouLjjIQ4Iz7OMCDOrGd+FYCDoHM4oDvoCPQ+jpdZzyhCdlrigTcaRdmp5Kb3vPnI6/23Z1QhifyMZLJTEzVsLSJhJ+Q9bDNLBF4EKp1zXz7atuphR4Zg0NHuD/T2aP09vdzOAB3+btq7AnT6/9Eb7uwO0h0I4g84uoNBugOOoHMEXc9E6GDv7+P+8DYgPt5IjIs7EPD7e+tJvT331MR40np79amJ8aQnx5OZkkhGck8PX+ErIpEibHrYZhYHPAZ0AbeEsm3pP3FxRkZyAhnJCUCK1+WIiESlkAW29XRz5gOFwAXOOf9HfIuIiIj0CmUP+yFgPHC2c64jhO2KiIhEvJAsQmxmw4EvA1OBKjNr7X1cE4r2RUREIl1IetjOue30zB8SERGRE6Db/IiIiEQABbaIiEgEUGCLiIhEAAW2RK5587yuQEQkZBTYErnuusvrCkREQkaBLSIiEgEU2BJZ5s3ruaPH/vXB93+s4XERiXKe3V7zWOjmH3JUZpzQbcJERMLU0W7+oR62iIhIBFBgS+SaO9frCkREQkaBLZFL561FJIYosEVERCKAAltERCQCKLBFREQiQFhf1mVmtcD2PvyR+UBdH/68cKX9jC7az+ii/Ywufb2fw51zBYd7IqwDu6+Z2fIjXd8WTbSf0UX7GV20n9EllPupIXEREZEIoMAWERGJALEW2A97XUCIaD+ji/Yzumg/o0vI9jOmzmGLiIhEqljrYYuIiEQkBbaIiEgEiLnANrNkM5tvZtvNrMXMVpnZ+V7X1R/M7BYzW25mnWb2iNf19BUzyzWzP5lZW+/r+Hmva+oP0fr6HSzGjseFZrbXzJrNbJOZ3eB1Tf3JzErNzGdmC72upT+Y2Zu9+9fa+9jY323GXGADCcBOYA6QDXwXeMLMSjysqb/sAe4Bfud1IX3sAaALKASuAR4ys4neltQvovX1O1gsHY8/BEqcc1nAxcA9ZnayxzX1pweAZV4X0c9ucc5l9D7G9ndjMRfYzrk259w859w251zQOfc8sBWIugPHOfe0c+4ZoN7rWvqKmaUDlwJ3OOdanXOLgeeAf/a2sr4Xja/foWLseCx3znXu/7T3McrDkvqNmV0FNAKve11LNIm5wD6UmRUCY4Byr2uRYzIG6HbObTroax8A0djDjjnRfjya2YNm1g5sAPYCL3hcUp8zsyzgbuDfvK4lBH5oZnVmtsTMzujvxmI6sM0sEfg98KhzboPX9cgxyQCaD/laE5DpQS3Sh2LheHTO3UzP7+ppwNNA59G/IyJ9D5jvnNvldSH97NvASGAIPddi/9nM+nXEJOoCu3cigDvCY/FB28UBj9FzLvQWzwo+Qce6n1GoFcg65GtZQIsHtUgfifTj8Xg45wK9p3KGAjd5XU9fMrOpwNnA/V7X0t+cc+8651qcc53OuUeBJcAF/dlmQn/+cC845874qG3MzID59ExausA55+/vuvrasexnlNoEJJhZqXNuc+/XyojSIdRYEA3H4wlKIPrOYZ8BlAA7el5WMoB4M5vgnJvmYV2h4ADrzwairod9jB4CxgMXOec6vC6mv5hZgpmlAPH0HDQpZhbRb9Kcc230DCXebWbpZnYq8Fl6emdRJRpfvyOI+uPRzAaa2VVmlmFm8WZ2LnA10Tcp62F63oRM7X38GvgLcK6XRfU1M8sxs3P3H5Nmdg1wOvBSf7Ybc4FtZsOBL9Pzy1R10DV013hcWn/4LtABfAf4Qu/H3/W0or5xM5AK1AB/BG5yzkVjDztaX78DYuh4dPQMf+8C9gE/Bm51zj3naVV9zDnX7pyr2v+g5xSWzzlX63VtfSyRnksua+m5F/bXgEsOmQzb57SWuIiISASIuR62iIhIJFJgi4iIRAAFtoiISARQYIuIiEQABbaIiEgEUGCLiIhEAAW2iIhIBFBgi4iIRAAFtoiISARQYIsIAGY2yswazGxa7+eDzaw2FPf5FZGPpqVJReQAM/tX4DZgOvAnYI1z7pveViUioMAWkUOY2XPACHpuWDHDOdfpcUkigobEReT/+g0wCfilwlokfKiHLSIHmDWxDx8AAACKSURBVFkG8AHwBnA+MNk51+BtVSICCmwROYiZzQcynHNXmtnDQI5z7gqv6xIRDYmLSC8z+yxwHnBT75f+DZhmZtd4V5WI7KcetoiISARQD1tERCQCKLBFREQigAJbREQkAiiwRUREIoACW0REJAIosEVERCKAAltERCQCKLBFREQigAJbREQkAvx/k4DLuT2JtQYAAAAASUVORK5CYII=\n", "text/plain": ["<Figure size 576x216 with 1 Axes>"]}, "metadata": {"tags": [], "needs_background": "light"}}]}, {"cell_type": "markdown", "metadata": {"id": "Uw0ECo8VFrMI"}, "source": ["This looks like a good fit!\n", "\n", "Note that if we chose too large a learning step, we would not converge to a solution.  A large step causes the parameter values to zoom all over the place, possibly missing by large amounts the (local) minima where you want to converge.  In the limit where the step size goes to 0, and the number of steps to infinity, you are guaranteed (if the function is differentiable, and some other hypotheses) converge to the minimum; the problem is that it would take infinitely long.  You will learn in a more in-depth ML class how to tune the step size."]}, {"cell_type": "code", "metadata": {"id": "eIDOa62EvYe2"}, "source": ["# Let us reinitialize the variables.\n", "va.assign(0)\n", "vb.assign(0)\n", "vc.assign(0)\n", "# ... and let's use a big step size.\n", "fit(loss, points, [va, vb, vc], delta=0.01, num_iterations=1000)\n"], "execution_count": null, "outputs": [{"output_type": "stream", "text": ["Loss: 7.445536528760376e+257\n", "Loss: inf\n", "Loss: nan\n", "Loss: nan\n", "Loss: nan\n", "Loss: nan\n", "Loss: nan\n", "Loss: nan\n", "Loss: nan\n", "Loss: nan\n"], "name": "stdout"}, {"output_type": "execute_result", "data": {"text/plain": ["nan"]}, "metadata": {"tags": []}, "execution_count": 30}]}, {"cell_type": "markdown", "metadata": {"id": "skvCcj0-vvP3"}, "source": ["A step size of 0.01 was enough to take us to infinity and beyond."]}, {"cell_type": "markdown", "metadata": {"id": "2SgARd2_wWZx"}, "source": ["Let us now show you how to fit a simple linear regression: $y = ax + b$, so $L = (y - (ax + b))^2$. "]}, {"cell_type": "code", "metadata": {"id": "J0Ic7OdBH6VT"}, "source": ["# Parameters\n", "# Sometimes you have to be careful about initial values.\n", "va = V(value=1.)\n", "vb = V(value=1.)\n", "\n", "# x and y\n", "vx = V(value=0.)\n", "vy = V(value=0.)\n", "\n", "# Predicted y\n", "oy = va * vx + vb\n", "\n", "# Loss\n", "loss = (vy - oy) * (vy - oy)\n"], "execution_count": null, "outputs": []}, {"cell_type": "code", "metadata": {"id": "DzmeetE2R88z"}, "source": ["fit(loss, points, [va, vb])\n"], "execution_count": null, "outputs": [{"output_type": "stream", "text": ["Loss: 28.04911931725314\n", "Loss: 24.56807140584221\n", "Loss: 22.04737547646302\n", "Loss: 20.018461554483544\n", "Loss: 18.363064879989466\n", "Loss: 17.010221159926804\n", "Loss: 15.904419410376471\n", "Loss: 15.000526654509262\n", "Loss: 14.261674191105309\n", "Loss: 13.657727388420048\n", "Loss: 13.164054060381849\n", "Loss: 12.760519585409948\n", "Loss: 12.43066568540981\n", "Loss: 12.161039167477448\n", "Loss: 11.940643231055738\n", "Loss: 11.760488960015408\n", "Loss: 11.613228706940784\n", "Loss: 11.492856417067383\n", "Loss: 11.394462669488892\n", "Loss: 11.314034444913947\n", "Loss: 11.248291453438622\n", "Loss: 11.194552346912685\n", "Loss: 11.150625359329684\n", "Loss: 11.114718914975501\n", "Loss: 11.085368558461209\n", "Loss: 11.061377226459106\n", "Loss: 11.041766425106331\n", "Loss: 11.025736321831346\n", "Loss: 11.012633123936265\n", "Loss: 11.00192241346032\n", "Loss: 10.99316735077883\n", "Loss: 10.986010857964564\n", "Loss: 10.980161055254529\n", "Loss: 10.975379356643991\n", "Loss: 10.971470739082786\n", "Loss: 10.968275788400177\n", "Loss: 10.965664197548818\n", "Loss: 10.963529451991706\n", "Loss: 10.96178448547398\n", "Loss: 10.960358128998926\n"], "name": "stdout"}, {"output_type": "execute_result", "data": {"text/plain": ["10.960358128998926"]}, "metadata": {"tags": []}, "execution_count": 32}]}, {"cell_type": "code", "metadata": {"id": "nECsieinSBYj"}, "source": ["plot_points_and_y(points, vx, oy)\n"], "execution_count": null, "outputs": [{"output_type": "display_data", "data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAewAAADWCAYAAAATvLEfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAeC0lEQVR4nO3deXDc533f8fcXNwEQ5+4PokiCIIhjSVkSJYGmZIoHVnQkccZ2J3FrKZYbeyrLlSN3LNceu7EbUnY6mcxkms44tjPqqImbOM24qeM6buL+wUuiJNMiLVmqTAA8RRIiubu4iBuL3ad/LLTi0BQvLfa3x+c1szPaQ3i+y8XuB8+z39/zM+ccIiIikttK/C5ARERErk2BLSIikgcU2CIiInlAgS0iIpIHFNgiIiJ5QIEtIiKSB8r8LuBqAoGAa2tr87sMERGRrDh8+HDMORe80n05HdhtbW0cOnTI7zJERESywszeeq/7tCQuIiKSBxTYIiIieUCBLSIicrN27craUApsERGRm/XMM1kbSoEtIiKSBxTYIiIiN2LXLjBLXeDd/17k5XHL5dNr9vT0OB3WJSIiOcsMMpijZnbYOddzpfs0wxYREckDCmwREZGbtXNn1oZSYIuIiNwsHdYlIiIil1Jgi4iI5AEFtoiISB5QYIuIiOQBBbaIiEgeUGCLiIjkAQW2iIhIHlBgi4iI5AEFtoiISB5QYIuIiOQBBbaIiEgeUGCLiIjkgawFtpm1mdk/mdmImZ03sz83s7JsjS8iIpLPsjnD/i4QAZYB64GtwOezOL6IiEjeymZgrwZ+6Jybcc6dB34G3JbF8UVERPJWNgP7vwCPmFm1mS0HHiYV2iIiInIN2Qzs50nNqC8CZ4FDwI8vf5CZPWFmh8zsUDQazWJ5IiIiuSsrgW1mJaRm0z8CaoAA0Aj8yeWPdc4965zrcc71BIPBbJQnIiKS87I1w24CWoE/d87NOueGgL8EdmRpfBERkbyWlcB2zsWAk8CTZlZmZg3A7wGvZ2N8ERGRfJfN77B/G3gIiALHgDjwdBbHFxERyVtZ27jEOfcasC1b44mIiBQSbU0qIiKSBxTYIiIiN2gmnmBff4R//NXbWRtTe3mLiIhch3Nj0+zti7Kn7wIvHhtiOp6gPVDDR+68NSvjK7BFRESuIJF0vHZmhD19Efb0RTly7iIAyxuW8PF7VhAOedy3pjlr9SiwRUREFoxOzbF/IMrevgj7B6KMTMUpLTHuWdXI1x4OEQ55dHq1mFnWa1Ngi4hI0XLOMXBhgj19Efb2RTh8eoRE0tFUU0Fvt0dvyGNLZ5D66nK/S1Vgi4hIcZmJJ3j5+NDCUneEwdFpANYtq+PJrWvoDXmsX9lAaUn2Z9FXo8AWEZGCd3Zkir0LAf3S8SFm55NUV5SyqSPAU+EOers9bqmv8rvMq1Jgi4hIwZlPJPnl6dH0Unf/hXEAWpuqefSDrYRDHhvbm6gsK/W50uunwBYRkYIwMjnHvoFUR/fzA1HGpuOUlRgb2pr4+o619IY81gRrfGkYywQFtoiI5CXnHEfOjbO3P7XU/erpEZIOArUVfHhdC73dHpu7AtRV+d8wlgkKbBERyRtTc/O8eCzVMLavP8K5sRkA7lhRzxfCnYRDHrcvr6ckxxrGMkGBLSIiOe300BR7+i6wpz/Kz08MMTefpLayjPs7Ajy93WNbKIi3NLcbxjJBgS0iIjklnkjyyqnhdFf38egkAO2BGj517yrCIY8NbU1UlBXX6TAU2CIi4rvYxCz7+lM7jD0/EGV8dp7yUmPj6mZ+d2MqpFcHavwu01cKbBERybpk0vHm2xdTm5f0R3j97CjOgbe0kh23L6M35HF/Z4DaSsXUO/QvISIiWTExO8+Bo9HUsdH9UaLjs5jBnSsaeHp7F+GQx2231uXtYVeLTYEtIiKL5mRsMr15ycGTQ8QTjqVVZWzpChLu9tjaHSRQW+l3mXlBgS0iIhkzN5/kFyeHF2bREU7GUg1jHV4tn9m0mt5uj562RspLi6thLBMU2CIi8r5ExmfY15da6j5wLMbE7DwVZSXc297Mpz/URjjksbKp2u8y854CW0REbkgy6Xh9cCy91P3G4BgAy+qr+Oj6W+nt9tjU0Ux1hSImk/SvKSIi13RxJs4LAzH29EXYPxAhNjFHicFdrY185cFuwiGP0C1L1TC2iBTYIiLyG5xzHI9OpM8ZfejUCPNJR/2ScrZ2BQmHPLZ2BWmsqfC71KKhwBYREQBm4gkOnnx3h7HTw1MAdLcs5fHN7YRDHne3NlCmhjFfKLBFRIrY+bGZ9Cz6xWMxpuMJKstK2NQR4LNb2untDrKi8SYaxnbtSl0kY8w553cN76mnp8cdOnTI7zJERApGIul47cxoehb963MXAVjesIRwyCMc8rhvTTNV5aXvbyAzyOF8yVVmdtg513Ol+zTDFhEpcGNTcfYfTe3TvX8gyvDkHKUlxj2tjXzt4RDhkEenV6uGsRynwBYRKTDOOQYuTKQPuzp8eoRE0tFYXc62bo/ekMfWziD11eWZHXjXLnjmmXevv/MHwM6dWh7PgKwuiZvZI8BOoBU4D3zaOffCez1eS+IiItdnJp7g5eND7O67wN6+KIOj0wCsW1ZHOJQK6fUrGygtydIsWkviNyUnlsTN7MPAnwCfAH4BLMvW2CIihWhwdDo9i37peIyZeJLqilI2dQT4/d4OwiGPW+qr/C5TMiSbS+LPAN90zv184fpgFscWEcl784kkvzw9mg7p/gvjALQ2VfPIhlbCIY+N7U1Ulr3PhrFM2LnT7woKTlaWxM2sFJgG/hB4HKgCfgx8xTk3fdljnwCeAGhtbb3nrbfeWvT6RERy1fDkHPsHIuzpi/L8QJSx6ThlJcaGtqb0UveaYI0axgpELiyJtwDlwMeBzUAc+N/AN4CvX/pA59yzwLOQ+g47S/WJiOQE5xxHzo2ztz912NWrp0dIOgjUVvDhdS2EQx73dwaoq8pww5jkvGwF9juz6G87584BmNl/5gqBLSJSbKbm5nnx2BB7+iLs649wbmwGgNuX1/NUuJNwyOOO5fWUZKthTHJSVgLbOTdiZmeBS2fMmj2LSNE6MzyV3mHs5RNDzM0nqa0s4/6OAE9v99jWHcSrU8OYvCubTWd/CXzBzH5Gakn8aeCnWRxfRMQ38USSQ6dG0kvdxyITALQHavjUvasIhzw2tDVRUaZ9uuXKshnY3wICwAAwA/wQ+E9ZHF9EJKtiE7Ps60/tMPb8QJTx2XnKS42Nq5t59IOtPBDyaAvU+F2m5ImsBbZzLg58fuEiIlJwkknHm29fTC1190d4/ewozoG3tJIdty+jN+Rx//98ltrH/9DvUiUP6eQfIiLvw8TsPAeOxlIn0+iPEB2fxQzuWNHAAwsn01i3rO7dhjHtACZXkQuHdYmIFIyTscn05iUHTw4RTziWVpaxpStIbyjVMBaorfS7TCkwCmwRkWuYm0/yi5PDqZDuj3AyNglAh1fLZzatprfbo6etkfLS92gY00kxJAO0JC4icgWR8Rn29UXZ3XeBA0djTM4lqCgr4b725vR5o1c2Vd/4D9aSuFyFlsRFRK4hmXS8PjiWXup+Y3AMgFvqqvjo+uWEQx6bOpqprtDHpvhDv3kiUrQuzsR5YSDGnr4I+wcixCbmKDG4q7WRrzzYTW+3x9plSzO7T7dOiiE3SYEtIkXDOcfx6ER6h7FDp0aYTzrql5SztStIOOSxpStIU03F4hWh76zlJl13YJvZnwHfd869toj1iIhk1Ew8wcGTw6nDrvoinB6eAqC7ZSmf3dJOOORx18oGyt6rYUwkR9zIDLsU+L9mFgX+GviBc+7s4pQlInLzzo/NpGfRLx6LMR1PUFlWwqaOQDqklzcs8btMkRty3YHtnPt3ZvY08DDwSeAbZnYQ+O/Aj5xzE4tUo4jIVSWSjtfOjLK3L8LuvghHzl0EYHnDEj5+zwrCIY/71jRTVV7qc6UiN++GvsN2ziVInbDjp2Z2G/C3wF8B3zWzvwN2OucGM16liMhlxqbi7D+a2qd7/0CU4ck5SkuMe1ob+epDIcIhj66W2sw2jIn46IYC28zqgH8JPAbcAfwvUnuDnwb+PfDPC7eLiGSUc46BCxPpw64Onx4hkXQ0VpezrdujN+SxtTNIfXW536WKLIobaTr7e+BB4HngL4AfO+dmL7n/S8BYxisUkaI1E0/w8vEhdvddYG9flMHRaQDWLavjya1r6A15rF/ZQGmJZtFS+G5khv1z4Cnn3Pkr3emcS5pZS2bKEpFiNTg6nZ5Fv3Q8xkw8yZLyUu7vDPBUuIPebo9b6qv8LlMk626k6exPr+MxU++vHBEpNvOJJK+eGWX3kVRI918YB6C1qZpHNrTSG/LYuLpJDWNS9LRxiohk3cjkHPsHogs7jEUZm45TVmJsaGvi6zvW0hvyWBOsUcOYyCUU2CKy6JxzHDk3zt7+1LHRr54eIekgUFvB9rUthEMem7sC1FWpYUzkvSiwRWRRTM3N8+KxIfb0RdjXH+Hc2AwAty+v56lwJ+GQxx3L6ylRw5jIdVFgi0jGnB6aYk/fBfb0R/n5iSHm5pPUVJSyuTPI09s9tnUH8erUMCZyMxTYInLT4okkh06NpJe6j0VSGx6uDtTw2MZVPLDWY0NbExVl2qdb5P1SYIvIDYlNzLKvP7XD2PNHo4zPzFNeamxc3cyjH2wlHPJYHajxu0yRgqPAFpGrcs7x5tsX0yfT+NXZUZwDb2klOz6wjN6Qx/2dAWor9XEispj0DhOR3zAxO8+BozH29kXY2x8hMj6LGdyxooGnt3cRDnmsW1anhjGRLFJgiwgAJ2OT6R3GDp4cIp5wLK0sY0tXMLVPd1eQ4NJKv8sUKVoKbJEiNTef5JVTw+ml7pOxSQA6vFo+s2k1vd0ePW2NlJeqYUwkFyiwRYpIZHyGfX2pHcYOHIsxMTtPRVkJ97U38+kPtdHb7dHaXO13mSJyBQpskQKWTDpeHxxLL3W/MZg6od4tdVV85M5beSDk8aGOZqor9FEgkuuy/i41s07gDeDvnXOPZXt8kUJ3cSbOCwOxhX26I8Qm5igxuKu1ka882E1vt8faZUu1T7dInvHjz+rvAK/4MK5IQXLOcTw6yd6F76JfOTXMfNJRV1XG1m6PcCjI1i6PppoKv0sVkfchq4FtZo8Ao8BLQEc2xxYpJLPzCQ6eeLdh7PRw6sy23S1LeXxzO+GQx92tDZSpYUykYGQtsM2sDvgmEAYev8rjngCeAGhtbc1OcSJ54PzYTHoL0BePxZiaS1BVXsKH1gT47JZ2eruDrGhUw5hIocrmDPtbwHPOubNX++7MOfcs8CxAT0+Py1JtIjknkXS8dmZkYRYd5ci5iwAsb1jC79y9gnDI4741zVSVl/pcqYhkQ1YC28zWA9uBu7Ixnki+GpuKs/9oap/uff0RRqbilJYY97Q28tWHQjyw1qPTq1XDmEgRytYMexvQBpxe+KCpBUrNbJ1z7u4s1SCSc5xzHI1MsPtI6rCrw6dHSCQdjdXlbOv2UjuMdQapry73u1QR8Vm2AvtZ4O8uuf5lUgH+ZJbGF8kZM/EELx8fSjeMDY5OA7BuWR1Pbl1DbyjI+pWNlGqfbhG5RFYC2zk3BUy9c93MJoAZ51w0G+OL+G1wdDp92NVLx2PMxJNUV5SyqSPAU+EOers9bqmv8rtMEclhvmxv5Jzb5ce4Itkyn0jy6pnR9A5jfefHAWhtquaRDalzRm9sb6KyTA1jInJ9tB+hSIaMTM6xfyC6sMNYlLHpOGUlxoa2Jr6+Yy29IY81wRo1jInITVFgi9wk5xxHzo2nj41+9fQISQfNNRVsX9tCOOSxuStAXZUaxkTk/VNgi9yAqbl5Xjo2xO6Fw67Ojc0AcPvyep4KdxIOedyxvJ4SNYyJSIYpsEWu4czwVLqj++UTQ8zNJ6mpKGVzZ5Cnt3ts6w7i1alhTEQWlwJb5DLxRJJDp0bSS93HIhMArA7U8NjGVTyw1mNDWxMVZdqnW0SyR4EtAsQmZtnXn9ph7PmjUcZn5ikvNTaububRD6a6ulcHavwuU0SKmAJbilIy6Xjz7Yuppe7+CK+fHcU5CC6t5OEP3EI45HF/Z5DaSr1FRCQ36NNIisbE7DwHjsbY2xdhb3+EyPgsZnDHiga++EAX4ZDHbbfWqWFMRHKSAlsK2qnYZGrzkv4IB08MM5dIsrSyjC1dQXpDqYaxQG2l32WKiFyTArsQ7dqVuhShufkkr5waTu8wdiI2CcCaYA2/96FVhEMt9LQ1Ul6qhjERyS/mXO6ecrqnp8cdOnTI7zLyjxnk8OuaaZHxGfb1pXYYO3AsxsTsPBVlJdzb3ky4O0g41EJrc7XfZYqIXJOZHXbO9VzpPs2wJe8kk443BsfYvTCLfmNwDIBb6qr4yJ23Eg55bOpoprpCv94iUjj0iVYodu2CZ5559/o7+1Xv3FkQy+PjM3FeOBpjT1+Eff1RYhOphrG7Vjbw5d/qojfksW5ZnfbpFpGCpSXxQlQAS+LOOU7EJtOnpPzFyWHmk466qjK2dnuEQ0G2dnk01VT4XaqISMZoSVzywkw8wcGTw+mQPj2cOoV6d8tSHt/cTjjkcXdrA2VqGBORIlRcgV0s3dM7d/pdwXU7PzaT3gL0wNEY0/EElWUlbOoI8Nkt7fR2B1nRqIYxEZHiWhIvgKXifJdIOl47M5qeRf/63EUAljcsoTcUJBzyuK89wJKKUp8rFRHJPi2Ji6/GpuM8P5Dap3vfQJThyTlKS4x7Whv56kMhwiGPrpZaNYyJiFxF4Qd2gXdP5yLnHEcjE+lTUh5+a4RE0tFYXc62bo/ekMfWziD11eV+lyoikje0JC4ZMRNP8PLxoXRID45OA7B2WR0PhFIhvX5lA6Xap1tE5D1pSVwWxeDodHoL0JeOx5iJJ1lSXsqmjgC/39tBbyjIsvolfpeZ/4qlWVJErqq4AjuPuqdz0XwiyatnRtMh3Xd+HICVTUv4RM9Kwmtb2Li6iapyNYxl1DPPKLBFpMgCWx96N2x0ao79A1F2H4mwfyDK2HScshKjp62RP9iRahhbE1TDmIjIYiuuwJZrcs5x5Nw4e/tTs+hfnh4h6SBQW8H2tS2EQx6buwLUValhbFGpWVJELlNcTWdyRVNz87x0bIjdfRH29Uc4NzYDwO3L6+kNeYRDHncsr6dEDWP+ULOkSNFQ05n8hjPDU+mO7pdPDDE3n6SmopT7OwN8cXsnvd0eXl2V32WKiMgCBXaRiCeSHH5rhL19EXb3RTgWmQCgrbmaxzauIhzy2LC6kcoyNYzlHDVLighZCmwzqwS+C2wHmoDjwH9wzv1zNsYvVkMTs+zrj7KnP8LzA1HGZ+YpLzU2rm7m0Q+2Eg55rA7U+F2mXIu+sxYRsjfDLgPOAFuB08AO4Idmdrtz7lSWaih4zjnefPtieqn7V2dHcQ6CSyvZ8YFl9IY87u8MUFuphRURkXyTlU9u59wksOuSm35qZieBe4BT2aihUE3MznPgaIy9fRH29keIjM9iBnesaOCLD3QRDnncdmudGsZERPKcL1MtM2sBuoA3/Rg/352KTbJ7YfOSgyeHiCccSyvL2NIVpDfksa07SKC20u8yRUQkg7Ie2GZWDvwA+L5zru8K9z8BPAHQ2tqa5epy09x8kldODad3GDsRmwRgTbCGT3+ojd6Qx4a2JspLS3yuVEREFktWj8M2sxLgb4E64GPOufjVHl/Mx2FHxmfY1586JeULR2NMzM5TUVbCve3NhLuDhEMttDZX+12miIhkUE4ch22pvSufA1qAHdcK62KTTDreGBxLL3W/MTgGwC11VXzkzlsJhzw2dTRTXaGGMRGRYpTNT//vAWuB7c656SyOm7PGZ+K8cDTGnr4I+/qjxCZSDWN3rWzgy7/VRW/IY92yOu3TLSIiWTsOexXwOWAWOH9JAH3OOfeDbNSQC5xznIhNsudI6rCrV04NM5901FWVsbXbIxwKsrXLo6mmwu9SRUQkx2TrsK63gKKcJs7OJzh4YqFhrD/CW0NTAHS3LOXfbF5NuNvjnlWNlKlhTERErkJfiC6C82Mz7O1PzaJfPBZjai5BZVkJmzoCPL65nd7uICsa1TAmIiLXT4GdAYmk47Uzo+xd2GHs1+cuArC8YQm/ffdywiGP+9oDLKnQPt0iInJzFNg3aWw6zvMDUfb0Rdg/EGV4co4Sg55VTXz1oRDhkEdXS60axkREJCMU2NfJOcfRyER6n+7Db42QSDoaqsvZtrDD2NauIA3VahgTEZHMU2BfxUw8wcsnhtJd3YOjqaPR1i6r499ubScc8li/spFS7dPtj127dCYrESkaWd3p7Eb5sdPZ4Oh06kQafRFePB5jJp5kSXkpmzoChEMevaEgy+qXZLUmeQ9mkMO/vyIiNyondjrLVfOJJK+eGU0tdR+J0H9hHICVTUv4RM9Kwmtb2Li6iapyNYyJiIh/ijKwRybn2H9Jw9jYdJyyEqOnrZE/2JFqGFsTVMNYTtq1C5555t3r77xGO3dqeVxEClrRLIkfj07ws/93nj19EV49PULSQXNNBVu7gzwQamFzV4C6qvKMjCVZoiVxESkwWhIHfvzqIN/ec4zbl9fzVLiT3u4gd65ooEQNYyIikgeKJrD/9X1tfOreVXh1VX6XIpmyc6ffFYiIZE3RBHZwaaXfJUim6TtrESkiOuOEiIhIHlBgi4iI5AEFtoiISB5QYIuIiOQBBbaIiEgeyOmNU8wsCryVwR8ZAGIZ/Hm5Ss+zsOh5FhY9z8KS6ee5yjkXvNIdOR3YmWZmh95rB5lCoudZWPQ8C4ueZ2HJ5vPUkriIiEgeUGCLiIjkgWIL7Gf9LiBL9DwLi55nYdHzLCxZe55F9R22iIhIviq2GbaIiEheUmCLiIjkgaILbDOrNLPnzOwtMxs3s9fM7GG/61oMZvaUmR0ys1kz+yu/68kUM2sys38ws8mF1/F3/a5pMRTq63epIns//o2ZnTOzi2Y2YGaP+13TYjKzTjObMbO/8buWxWBm+xae38TCpX+xxyy6wCZ1StEzwFagHvgG8EMza/OxpsXyNvBHwH/zu5AM+w4wB7QAnwS+Z2a3+VvSoijU1+9SxfR+/GOgzTlXB3wU+CMzu8fnmhbTd4BX/C5ikT3lnKtduHQv9mBFF9jOuUnn3C7n3CnnXNI591PgJFBwbxzn3I+ccz8GhvyuJVPMrAb4HeA/OucmnHMHgJ8An/K3sswrxNfvckX2fnzTOTf7ztWFyxofS1o0ZvYIMArs9ruWQlJ0gX05M2sBuoA3/a5FrksXMO+cG7jktl8BhTjDLjqF/n40s++a2RTQB5wD/snnkjLOzOqAbwJf8ruWLPhjM4uZ2Ytmtm2xByvqwDazcuAHwPedc31+1yPXpRa4eNltY8BSH2qRDCqG96Nz7vOkflc3Az8CZq/+f+SlbwHPOefO+l3IIvsq0A4sJ3Us9j+a2aKumBRcYC80Arj3uBy45HElwF+T+i70Kd8KvknX+zwL0ARQd9ltdcC4D7VIhuT7+/FGOOcSC1/lrACe9LueTDKz9cB24M/8rmWxOecOOufGnXOzzrnvAy8COxZzzLLF/OF+cM5tu9ZjzMyA50g1Le1wzsUXu65Mu57nWaAGgDIz63TOHV247U4KdAm1GBTC+/EmlVF432FvA9qA06mXlVqg1MzWOefu9rGubHCALeYABTfDvk7fA9YCH3HOTftdzGIxszIzqwJKSb1pqswsr/9Ic85NklpK/KaZ1ZjZJuBjpGZnBaUQX7/3UPDvRzPzzOwRM6s1s1IzexB4lMJrynqW1B8h6xcufwH8H+BBP4vKNDNrMLMH33lPmtkngS3AzxZz3KILbDNbBXyO1C/T+UuOofukz6Uthm8A08DXgMcW/vsbvlaUGZ8HlgAR4H8ATzrnCnGGXaivX1oRvR8dqeXvs8AI8KfAF51zP/G1qgxzzk05586/cyH1FdaMcy7qd20ZVk7qkMsoqXNhfwH4F5c1w2ac9hIXERHJA0U3wxYREclHCmwREZE8oMAWERHJAwpsERGRPKDAFhERyQMKbBERkTygwBYREckDCmwREZE8oMAWERHJAwpsEQHAzNaY2bCZ3b1w/VYzi2bjPL8icm3amlRE0szss8DTQA/wD8Abzrkv+1uViIACW0QuY2Y/AVaTOmHFBufcrM8liQhaEheR3/RfgQ8A31ZYi+QOzbBFJM3MaoFfAXuBh4HbnXPD/lYlIqDAFpFLmNlzQK1z7hNm9izQ4Jz7V37XJSJaEheRBWb2MeAh4MmFm74E3G1mn/SvKhF5h2bYIiIieUAzbBERkTygwBYREckDCmwREZE8oMAWERHJAwpsERGRPKDAFhERyQMKbBERkTygwBYREckDCmwREZE88P8BoKgciB6BIEUAAAAASUVORK5CYII=\n", "text/plain": ["<Figure size 576x216 with 1 Axes>"]}, "metadata": {"tags": [], "needs_background": "light"}}]}, {"cell_type": "markdown", "metadata": {"id": "DbR2_iazxQt7"}, "source": ["## Exercises\n", "\n", "Using the method illustrated above, fit the following equations to our set of points.  Use `vx`, `xy` for $x$, $y$, and `va`, `vb`, `vc`, etc for the parameters.  This is important, or the tests won't pass.\n", "\n", "$$\n", "y = a^x + bx + c\n", "$$"]}, {"cell_type": "markdown", "metadata": {"id": "ibNLDGNLW-8r"}, "source": ["## Conclusion\n", "\n", "Congratulations, you have now implemented the core of [PyTorch](https://pytorch.org)!  Pytorch is build on the ability to write expressions, of which the gradient is computed and automatically propagated; the core constructs, such as `zero_gradient`, `compute_gradient`, and `compute`, are essentially similar.  It is true; Pytorch does add a few things:\n", "\n", "* Ability to deal with $n$-dimensional tensors, to represent vectors of inputs and outputs. \n", "* Optimized backends, using both parallelism and GPUs. \n", "* Sophisticated mechanisms to deal with large datasets.\n", "* A wide array of operations available on tensors, of which autograd can be computed. \n", "* Many implemented heuristics to deal with step size.\n", "* Implemented neural layers (sigmoids, ReLU, etc).\n", "* Convolutional nets for images.\n", "* Recurrent nets and LSTMs of streams of data.\n", "\n", "And more! But [aside from tensors, backends, GPUs, parallelism, operations, heuristics, neural layers including convolutions and LSTMs, and good output](https://www.youtube.com/watch?v=Y7tvauOJMHo), PyTorch is really nothing more than what you have already implemented."]}]}