Introduction_to_Neural_Networks_with_TensorFlow_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Introduction to Neural Networks with TensorFlow 2.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/n2cholas/dsc-workshops/blob/master/Introduction_to_Neural_Networks_with_TensorFlow_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g2kP92FhDK-Z",
        "colab_type": "text"
      },
      "source": [
        "# Introduction to Neural Networks in TensorFlow 2.0\n",
        "By Nicholas Vadivelu (nicholas.vadivelu@gmail.com)\n",
        "\n",
        "## Workshop Overview\n",
        "\n",
        "1.   What is TensorFlow?\n",
        "2.   Linear Regression \n",
        "3.   Gradient Descent\n",
        "3.   Multivariate Logistic Regression\n",
        "4.   Neural Networks\n",
        "5.   Intro to tf.keras\n",
        "\n",
        "## What is Machine Learning?\n",
        "\n",
        "*  **Supervised Learning**: given some input $X$ (called features), predict $y$ (called the target)\n",
        "*  Given data, we train a Machine Learning Model\n",
        "  *  A simple model:  $y = mx + b$\n",
        "*  Training involves finding the best $m$ and $b$ for the data\n",
        "*  Applications include classifying images, generating text, predicting stocks, etc.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vbPZZeyhVzxh",
        "colab_type": "text"
      },
      "source": [
        "# Introduction to TensorFlow\n",
        "\n",
        "TensorFlow is a open source numerical computation library designed for machine learning. Its key advantages over libraries like numpy and scipy are its support for hardware acceleration (GPUs, TPUs, etc), automatic differentiation, and end-to-end ecosystem for designing and deploying machine learning models.\n",
        "\n",
        "The recent 2.0 update revamped the library for ease of use, making the API feel similar to numpy and PyTorch (another deep learning library). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AN0ViEM7V1tG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "45ed75ed-ef5b-47b1-c352-50426c2722e0"
      },
      "source": [
        "%tensorflow_version 2.x # Enables version 2.0\n",
        "import tensorflow as tf"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "`%tensorflow_version` only switches the major version: `1.x` or `2.x`.\n",
            "You set: `2.x # Enables version 2.0`. This will be interpreted as: `2.x`.\n",
            "\n",
            "\n",
            "TensorFlow 2.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-eNjZf2Wpgg",
        "colab_type": "text"
      },
      "source": [
        "TensorFlow is a library for manipulation **tensors**, which are essentialy multidimensional arrays."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68-cAlWhWl_J",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d1751aac-8681-40b4-83c5-19c438901b98"
      },
      "source": [
        "x = tf.convert_to_tensor([1, 2, 3, 4])\n",
        "x"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=0, shape=(4,), dtype=int32, numpy=array([1, 2, 3, 4], dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o4oLymOaYFZZ",
        "colab_type": "text"
      },
      "source": [
        "You can read all about Tensors in the [documentation](https://https://www.tensorflow.org/api_docs/python/tf/Tensor). The key ideas are they have a data type (dtype), a shape, and a numeric value (numpy array).\n",
        "\n",
        "Unlike Python lists, you can do **vectorized operations**:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ONmhIrL2X6iD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ad096103-a725-4e78-f2c9-3164ccc69b91"
      },
      "source": [
        "2*x + 1"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=4, shape=(4,), dtype=int32, numpy=array([3, 5, 7, 9], dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hl2tLvqlYet8",
        "colab_type": "text"
      },
      "source": [
        "Just like numpy, TensorFlow supports **broadcasting**. Here's a brief example. Suppose you want to add a vector to every row of a matrix:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xVp5dBgOYWjv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "vec = tf.convert_to_tensor([1, 2, 3, 4])\n",
        "matrix = tf.convert_to_tensor([[1, 1, 1, 1],\n",
        "                               [4, 4, 4, 4],\n",
        "                               [5, 5, 5, 5]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-0P-xxvtZJoQ",
        "colab_type": "text"
      },
      "source": [
        "With normal lists, you might be inclined to do something like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fK0aPna3clUc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a5ce9bb6-2946-492d-9c77-4936c727c250"
      },
      "source": [
        "matrix[1, :] # second row of the matrix"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=10, shape=(4,), dtype=int32, numpy=array([4, 4, 4, 4], dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7uHMzaT8ZIes",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        },
        "outputId": "7552438c-cf47-4a42-f98f-25a21600d93f"
      },
      "source": [
        "for i in range(3):\n",
        "  matrix[i, :] += vec"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-7-10dabb925653>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m   \u001b[0mmatrix\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mvec\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m: 'tensorflow.python.framework.ops.EagerTensor' object does not support item assignment"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q3EyMFtOZxii",
        "colab_type": "text"
      },
      "source": [
        "Oops, you can't assign values to a Tensor! In general, computations create a new Tensor instead of modifying an existing one. If you need something that is modifable, use a variable. Read more about those [here](https://www.tensorflow.org/api_docs/python/tf/Variable)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hdl4qjYlZ6g0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "3cd9c909-3fa5-4c32-ac20-bd32bf37d563"
      },
      "source": [
        "matrix = tf.Variable([[1, 1, 1, 1],\n",
        "                      [4, 4, 4, 4],\n",
        "                      [5, 5, 5, 5]])\n",
        "matrix"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Variable 'Variable:0' shape=(3, 4) dtype=int32, numpy=\n",
              "array([[1, 1, 1, 1],\n",
              "       [4, 4, 4, 4],\n",
              "       [5, 5, 5, 5]], dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G3mVDzhpaOy1",
        "colab_type": "text"
      },
      "source": [
        "Now, back to adding this vector to the matrix. If you add them directly, broadcasting will see that the all the dimensions of `vec` and `matrix` match except for one, so it will duplicate vec along that dimension so the addition works. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GS-AVgWhZ-5r",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "6d644ea0-4755-4b38-8a18-c6f11b3109c6"
      },
      "source": [
        "matrix + vec"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=26, shape=(3, 4), dtype=int32, numpy=\n",
              "array([[2, 3, 4, 5],\n",
              "       [5, 6, 7, 8],\n",
              "       [6, 7, 8, 9]], dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CDswIcHKabf8",
        "colab_type": "text"
      },
      "source": [
        "Now we can assign this new value to matrix:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nAOwiKSSabA7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "09362ddb-e779-4201-ac56-579a53e6270d"
      },
      "source": [
        "matrix.assign(matrix + vec)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Variable 'UnreadVariable' shape=(3, 4) dtype=int32, numpy=\n",
              "array([[2, 3, 4, 5],\n",
              "       [5, 6, 7, 8],\n",
              "       [6, 7, 8, 9]], dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4WxhdR22a-Ij",
        "colab_type": "text"
      },
      "source": [
        "[Here's](https://scipy-lectures.org/intro/numpy/operations.html#broadcasting) a more detailed overview of broadcasting.\n",
        "\n",
        "We can create scalar tensors and variables:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BNrdti5KbAUJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7e3f6ffd-84cb-4fa2-bdf8-58943bcef0cd"
      },
      "source": [
        "x = tf.Variable(0)\n",
        "v = tf.convert_to_tensor(1)\n",
        "\n",
        "x + v"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=39, shape=(), dtype=int32, numpy=1>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbhovveKaf1C",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "Take a few minutes to explore Tensors and Variables. Create some, play with them, and use the documentation to figure out what they can and can't do!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m1dCyY90FIp1",
        "colab_type": "text"
      },
      "source": [
        "## Introduction to Linear Regression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fqiy2NzUFL0q",
        "colab_type": "text"
      },
      "source": [
        "We're going to start by fitting a line ($y = mx + b$) to some simulated data in TensorFlow. Suppose have some data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yY8wAFnEGFxt",
        "colab_type": "code",
        "outputId": "ad3567bb-340f-496e-df81-12c221fdb845",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        }
      },
      "source": [
        "x = tf.range(1., 5., 0.1) # ensure all the inputs are floats!\n",
        "x"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=43, shape=(40,), dtype=float32, numpy=\n",
              "array([1.       , 1.1      , 1.2      , 1.3000001, 1.4000001, 1.5000001,\n",
              "       1.6000001, 1.7000002, 1.8000002, 1.9000002, 2.0000002, 2.1000001,\n",
              "       2.2      , 2.3      , 2.3999999, 2.4999998, 2.5999997, 2.6999996,\n",
              "       2.7999995, 2.8999994, 2.9999993, 3.0999992, 3.199999 , 3.299999 ,\n",
              "       3.399999 , 3.4999988, 3.5999987, 3.6999986, 3.7999985, 3.8999984,\n",
              "       3.9999983, 4.0999985, 4.1999984, 4.2999983, 4.399998 , 4.499998 ,\n",
              "       4.599998 , 4.699998 , 4.799998 , 4.8999977], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fEfzTROMbwVC",
        "colab_type": "text"
      },
      "source": [
        "Above we've created our x values, which is our **explanatory variable**. We will simulate a **response variable** y, which has the relationship $y = 3.5x - 1$, plus some noise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wjm8dVdPG_r0",
        "colab_type": "code",
        "outputId": "da149ebf-b6e4-4c2b-cb89-6b2384ea9fb9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        }
      },
      "source": [
        "y = 3.5*x - 1 + tf.random.uniform(shape=(40,), minval=-1, maxval=1, dtype=tf.float32)\n",
        "y"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=55, shape=(40,), dtype=float32, numpy=\n",
              "array([ 1.5081716,  3.4402354,  3.565304 ,  2.9521132,  4.642995 ,\n",
              "        5.1984725,  4.033718 ,  4.244085 ,  5.6146545,  6.6339474,\n",
              "        6.4838257,  6.9260926,  6.266905 ,  6.4089746,  6.5717545,\n",
              "        6.8176184,  7.19166  ,  9.247049 ,  7.9582567, 10.14625  ,\n",
              "        8.848628 , 10.280046 ,  9.729949 , 10.760101 , 11.438638 ,\n",
              "       11.2754   , 11.156017 , 11.16319  , 12.735874 , 13.418887 ,\n",
              "       12.87348  , 13.070494 , 13.1910925, 13.565499 , 14.655623 ,\n",
              "       14.032754 , 15.643642 , 16.414732 , 16.290714 , 16.515282 ],\n",
              "      dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMG48EIlcLwY",
        "colab_type": "text"
      },
      "source": [
        "Let's see how our data looks:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q9gK37UcHPQC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "0bd01c6f-7b2c-4f0a-b23e-0fd8250ddcb8"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline\n",
        "\n",
        "plt.scatter(x, y)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.collections.PathCollection at 0x7f2120205d30>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAUBElEQVR4nO3df4xlZX3H8fe3sOqgxNHuVNlZtkub\nuomy6uLE0G5rUVRoJbDZ+gckWKg2G61RbHUJ2ETTJu1ugrH2R1KzAQKtiBrELRV/EVdD2ih2lkVB\n8FdS0B2xO0hAq1tl8ds/5o47e7lzz517zr33nHvfr4Rw59yTe745MJ/7zPM853kiM5EkNc+vjLoA\nSVJ/DHBJaigDXJIaygCXpIYywCWpoU4e5sXWr1+fmzdvHuYlJanxDh48+EhmzrQfH2qAb968mfn5\n+WFeUpIaLyIe6nTcLhRJaigDXJIaygCXpIYywCWpoQxwSWqooc5CkaRJsv/QAtd89pt8/7GjbJie\nYvd5W9ixbbayzzfAJWkA9h9a4Opb7+XoE08CsPDYUa6+9V6AykLcAJekVZRpQV/z2W/+MryXHX3i\nSa757DcNcEkapF5a0N0C/vuPHe34uasd74eDmJLUQbcWNBwP+IXHjpIcD/j9hxYA2DA91fFzVzve\nDwNckjooakEXBfzu87Ywte6kE96fWncSu8/bUlmNBrgkdVDUgi4K+B3bZtmzcyuz01MEMDs9xZ6d\nW4c7CyUirgcuAI5k5pkrjr8NeCvwJHB7Zl5ZWVWSNGK7z9tyQh84nNiC3jA9xUKHEF8Z/Du2zVYa\n2O16aYHfAJy/8kBEvBK4CHhJZr4IeF/1pUnS6BS1oIfRRVKksAWemXdGxOa2w28B9mbmz1rnHKm+\nNEkarW4t6OXjg3xQp0i/0whfAPxeRPwN8H/AuzLzvzqdGBG7gF0AmzZt6vNyklQ/g+4iKdLvIObJ\nwHOBs4HdwMciIjqdmJn7MnMuM+dmZp6yoYQkqU/9tsAPA7dmZgJfiYhfAOuBxcoqk6QhGPR6JYPU\nbwt8P/BKgIh4AfA04JGqipKkYSh6GKfuCgM8Im4GvgRsiYjDEfEm4HrgNyLiPuAjwGWt1rgkNUbR\nwzh118sslEtWeevSimuRpKEaxnolg+STmJIm1jDWKxkkA1zSxKrDwzhluJyspIlVh4dxyjDAJU20\nUT+MU4ZdKJLUUAa4JDWUAS5JDWWAS1JDGeCS1FDOQpHUaE1ejKosA1xSYy0vRrW8nsnyYlTARIS4\nXSiSGqvpi1GVZYBLaqymL0ZVlgEuqbGavhhVWQa4pMZq+mJUZTmIKamxmr4YVVkGuKRGa/JiVGX1\nsqXa9RFxpLV9Wvt774yIjIj1gylPksrZf2iB7XsPcMZVt7N974HG7HfZi176wG8Azm8/GBGnA68F\nvltxTZJUiaZvWlykMMAz807g0Q5v/R1wJeBmxpJqadznifc1CyUiLgIWMvOrPZy7KyLmI2J+cXGx\nn8tJUl/GfZ74mgM8Ik4B3g28p5fzM3NfZs5l5tzMzMxaLydJfRv3eeL9tMB/EzgD+GpEPAhsBO6O\niOdXWZgklTXu88TXPI0wM+8Ffm3551aIz2XmIxXWJUlAudUGx32eeGGAR8TNwDnA+og4DLw3M68b\ndGGSVMVqg+M8T7wwwDPzkoL3N1dWjaRaGtWa291mkYxrKK+FT2JK6mqUa26P+yySslzMSlJXo5xL\nPe6zSMoywCV1NehWcLdH3cd9FklZdqFI6mrD9BQLHcK6ilZwUffMuM8iKcsAl9TV7vO2nBCysPZW\n8GqDoL0MUo7zLJKyDHBJXZVtBXdrZTtIWY4BLqlQUSu42zTDbq3sQXbPTAIHMSWVUrRka7dWtoOU\n5RjgkkptelA0zbDbVMAd22bZs3Mrs9NTBDA7PcWenVvt8+6RXSjShCv7oE5RP3bRIKiDlP2zBS5N\nuLIP6hQ9bGMre3BsgUsTruxMkF6mGdrKHgxb4NKEK/u4ui3s0bEFLk24Kh7UsYU9Gga4NOF8XL25\nDHBJtqAbyj5wSWooA1ySGqowwCPi+og4EhH3rTh2TUR8IyK+FhGfiIjpwZYpSWrXSwv8BuD8tmN3\nAGdm5ouBbwFXV1yXJKlAYYBn5p3Ao23HPpeZx1o/fhnYOIDaJEldVNEH/kbg06u9GRG7ImI+IuYX\nFxcruJwkCUoGeET8JXAMuGm1czJzX2bOZebczMxMmctJklboex54RFwOXACcm5lZWUWSpJ70FeAR\ncT5wJfD7mfnTakuSJPWiMMAj4mbgHGB9RBwG3svSrJOnA3dEBMCXM/PNA6xTGnvdtiWTOikM8My8\npMPh6wZQizSxym6qoMnkWihSDXTbVGE5wG2hq50BLtVA0aYKttDViQEu1cCG6SkWOoT48qYKVbTQ\nbcGPHxezkmpg93lbmFp30gnHVm6q0GsLfeGxoyTHW+jLu8sXva9mMsClGijalqxo27OijYnLblys\nerILRaqJbpsqFG17VtRCL7txserJFrjUAGVb6GU3LlY92QKXhqjMQGKZFnoVGxerfgxwaUgGORWw\naGNiNy4eTzHMdajm5uZyfn5+aNeT6mT73gMdpwrOTk/xn1e9agQVqSki4mBmzrUftw9cGhIHElU1\nA1waEgcSVTUDXBqSood1pLVyEFNag7KzSMCBRFXHAJd6VMUskm5TAaW1MsA1Ucq0oHtZUEoaJgNc\nE6NsC9pZJKqbwkHMiLg+Io5ExH0rjj03Iu6IiG+3/v2cwZYplVd2QSdnkahuepmFcgNwftuxq4DP\nZ+ZvAZ9v/SzVWtkWtLNIVDeFAZ6ZdwKPth2+CLix9fpGYEfFdUmVK9uCLlpQShq2fvvAn5eZD7de\n/wB43monRsQuYBfApk2b+rycVF4VCzo5i0R1UnoQMzMzIlZdUCUz9wH7YGktlLLXk/o1jHnYblum\nYeo3wP8nIk7LzIcj4jTgSJVFSYMyyBa0Gw9r2Pp9lP424LLW68uAf6umHKm53LZMw9bLNMKbgS8B\nWyLicES8CdgLvCYivg28uvWzNNGcJ65hK+xCycxLVnnr3IprkRptw/RUx/W+nSeuQXE1QqkizhPX\nsPkovRqnrjM9XG1Qw2aAq1HqPtPDeeIaJrtQ1CjO9JCOM8DVKM70kI4zwNUorggoHWeAq1EGPdNj\n/6EFtu89wBlX3c72vQfYf2ihks+VBsFBTDXKIGd61H2AVGpngKtxBjXTwy3T1DR2oUgtDpCqaQxw\nqcUBUjWNAS61+Ci8msY+cKnFR+HVNAa4tIKPwqtJ7EKRpIYywCWpoQxwSWqoUgEeEX8eEV+PiPsi\n4uaIeEZVhUmSuut7EDMiZoG3Ay/MzKMR8THgYuCGimqT+lLXDR+kqpWdhXIyMBURTwCnAN8vX5LU\nP9cz0STpuwslMxeA9wHfBR4GHs/Mz7WfFxG7ImI+IuYXFxf7r1TqgRs+aJL0HeAR8RzgIuAMYAPw\nzIi4tP28zNyXmXOZOTczM9N/pVIPXM9Ek6TMIOargf/OzMXMfAK4FfidasqS+uN6JpokZQL8u8DZ\nEXFKRARwLvBANWVJ/XE9E02SvgcxM/OuiLgFuBs4BhwC9lVVmNQP1zPRJInMHNrF5ubmcn5+fmjX\nk6RxEBEHM3Ou/bhPYkpSQxngktRQLier2vFJSqk3BrhqxScppd7ZhaJa8UlKqXcGuGrFJyml3tmF\nMqHq2s+8YXqKhQ5h7ZOU0lPZAp9Ay/3MC48dJTnez7z/0MLQrr997wHOuOp2tu89cMJ1fZJS6p0B\nPoFG2c9c9OWxY9sse3ZuZXZ6igBmp6fYs3NrLf46kOrGLpQG67cbZJT9zN2+PJZrd2d4qTcGeEOV\nmW43jH7m1b5cHKSUqmMXSkP10g2yWl/zoPuZu3WTuNyrVB1b4A1V1JLtpYVeZhZKt+6bbl8uu8/b\nckJd4CCl1C8DvKGKukGK+prL9DMXfTl0+3JxuVepOgZ4QxW1ZMv2Nffbwt6xbbbwy8VBSqka9oE3\nVNF0uzJ9zUVT/Yq+HJzLLQ2HLfAG69aSLdPXXEULe/lz7CaRBscAH1NlQrSXFnbRl4PdJNLglQrw\niJgGrgXOBBJ4Y2Z+qYrCVF6/IWoLW2qGsi3wvwc+k5mvj4inAadUUJNGzBa21Ax9B3hEPBt4BXA5\nQGb+HPh5NWVplGxhS83Q9670EfFSYB9wP/AS4CBwRWb+pO28XcAugE2bNr3soYceKlWwJE2aQexK\nfzJwFvDPmbkN+AlwVftJmbkvM+cyc25mZqbE5SRJK5XpAz8MHM7Mu1o/30KHAFf/6rrpgqR66LsF\nnpk/AL4XEcsjW+ey1J2iCox60wVJ9Vf2Scy3ATdFxNeAlwJ/W74kgZv7SipWahphZt4DPKVjXeW5\nbrakIq6FUlOumy2piAFeUrcNestwQShJRVwLpYQy25oV8WEaSUUM8BJ62aC3DB9Xl9SNXSglONAo\naZQM8BIcaJQ0SgZ4CQ40Shol+8BLcKBR0igZ4CU50ChpVAzwAi4oJamuDPAuBjnPW5LKchCzCxeU\nklRnBngXzvOWVGcGeBfO85ZUZwZ4F4Oe5z2ohbAkTQYHMbsY5DxvB0gllWWAFxjUPO9BL4QlafyV\nDvCIOAmYBxYy84LyJY2PbnPIHSCVVFYVfeBXAA9U8DljpWhTYgdIJZVVKsAjYiPwOuDaasoZH0Vz\nyF0IS1JZZbtQPgBcCZy62gkRsQvYBbBp06aSl2uOoi4SF8KSVFbfAR4RFwBHMvNgRJyz2nmZuQ/Y\nBzA3N5f9Xq9pNkxPsdAhxFd2kbgQlqQyynShbAcujIgHgY8Ar4qID1VS1Riwi0TSoPXdAs/Mq4Gr\nAVot8Hdl5qUV1VWZUa0maBeJpEEbi3ngq4X0qB+WsYtE0iBVEuCZ+UXgi1V81lp1C2kflpE0zhq/\nFkq3kPZhGUnjrPEB3i2kfVhG0jhrfIB3C2lngkgaZ40P8G4hvWPbLHt2bmV2eooAZqen2LNzq/3f\nksZC42ehFE3XcyaIpHHV+ACHciHtrvOSmmosArxfo54nLkllNL4PvAx3nZfUZBMd4M4Tl9RkEx3g\nzhOX1GQTHeDOE5fUZBM9iOmKgZKabKIDHJwnLqm5JroLRZKazACXpIYywCWpoQxwSWqovgM8Ik6P\niC9ExP0R8fWIuKLKwiRJ3ZWZhXIMeGdm3h0RpwIHI+KOzLy/otokSV303QLPzIcz8+7W6x8DDwDO\nx5OkIamkDzwiNgPbgLs6vLcrIuYjYn5xcbGKy0mSqCDAI+JZwMeBd2Tmj9rfz8x9mTmXmXMzMzNl\nLydJaikV4BGxjqXwvikzb62mJElSL8rMQgngOuCBzHx/dSVJknpRZhbKduANwL0RcU/r2Lsz81Pl\nyzqR255J0lP1HeCZ+R9AVFhLR257Jkmd1f5JTLc9k6TOah/gbnsmSZ3VPsDd9kySOqt9gLvtmSR1\nVvsdedz2TJI6q32Ag9ueSVInte9CkSR1ZoBLUkMZ4JLUUAa4JDWUAS5JDRWZObyLRSwCD5X4iPXA\nIxWVUzVrW7u61gXW1i9r609Rbb+emU/ZUGGoAV5WRMxn5tyo6+jE2taurnWBtfXL2vrTb212oUhS\nQxngktRQTQvwfaMuoAtrW7u61gXW1i9r609ftTWqD1ySdFzTWuCSpBYDXJIaqnYBHhHXR8SRiLhv\nlfcjIv4hIr4TEV+LiLNqVNs5EfF4RNzT+uc9Q6rr9Ij4QkTcHxFfj4grOpwzkvvWY22jum/PiIiv\nRMRXW7X9VYdznh4RH23dt7siYnONars8IhZX3Lc/HUZtrWufFBGHIuKTHd4byT3rsbZR3rMHI+Le\n1nXnO7y/9t/RzKzVP8ArgLOA+1Z5/w+BT7O0ofLZwF01qu0c4JMjuGenAWe1Xp8KfAt4YR3uW4+1\njeq+BfCs1ut1wF3A2W3n/Bnwwdbri4GP1qi2y4F/GvZ9a137L4APd/rvNqp71mNto7xnDwLru7y/\n5t/R2rXAM/NO4NEup1wE/Esu+TIwHRGn1aS2kcjMhzPz7tbrHwMPAO0LqI/kvvVY20i07sX/tn5c\n1/qnfVT/IuDG1utbgHMjImpS20hExEbgdcC1q5wyknvWY211tubf0doFeA9mge+t+PkwNQmElt9u\n/dn76Yh40bAv3vpzdRtLLbaVRn7futQGI7pvrT+37wGOAHdk5qr3LTOPAY8Dv1qT2gD+qPXn9i0R\ncfow6gI+AFwJ/GKV90d2zyiuDUZzz2DpC/hzEXEwInZ1eH/Nv6NNDPA6u5ulNQteAvwjsH+YF4+I\nZwEfB96RmT8a5rWLFNQ2svuWmU9m5kuBjcDLI+LMYV27SA+1/TuwOTNfDNzB8VbvwETEBcCRzDw4\n6GutVY+1Df2erfC7mXkW8AfAWyPiFWU/sIkBvgCs/Nbc2Do2cpn5o+U/ezPzU8C6iFg/jGtHxDqW\nAvKmzLy1wykju29FtY3yvq2o4THgC8D5bW/98r5FxMnAs4Ef1qG2zPxhZv6s9eO1wMuGUM524MKI\neBD4CPCqiPhQ2zmjumeFtY3oni1fe6H17yPAJ4CXt52y5t/RJgb4bcAft0ZszwYez8yHR10UQEQ8\nf7mvLyJeztL9Hfj/uK1rXgc8kJnvX+W0kdy3Xmob4X2biYjp1usp4DXAN9pOuw24rPX69cCBbI04\njbq2tv7RC1kaXxiozLw6Mzdm5maWBigPZOalbaeN5J71Utso7lnrus+MiFOXXwOvBdpns635d7R2\nmxpHxM0szUpYHxGHgfeyNIBDZn4Q+BRLo7XfAX4K/EmNans98JaIOAYcBS4exv+4LLU83gDc2+oz\nBXg3sGlFbaO6b73UNqr7dhpwY0ScxNKXxscy85MR8dfAfGbextKXz79GxHdYGsC+eAh19Vrb2yPi\nQuBYq7bLh1TbU9TknvVS26ju2fOAT7TaKScDH87Mz0TEm6H/31EfpZekhmpiF4okCQNckhrLAJek\nhjLAJamhDHBJaigDXJIaygCXpIb6fwZHyKPO+giWAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cCiDdzrxcsBt",
        "colab_type": "text"
      },
      "source": [
        "When we fit a linear model to data, we want to find the values of **parameters** $m$ and $b$ to make our line best represent our data. Let's createt these variables first:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2KEFL520HmyV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "m = tf.Variable(1, dtype=tf.float32) # explicitly set the data type, because the initial value is an integer\n",
        "b = tf.Variable(0, dtype=tf.float32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lL41na5WdQHL",
        "colab_type": "text"
      },
      "source": [
        "Let's see our line looks before fitting it to data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "505Jh2vMH2d6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "outputId": "8b49937c-08e4-4c9b-8e30-924520b30a98"
      },
      "source": [
        "y_pred = x*m + b # notice y is a vector, we are doing a vectorized operation here on all the entries of x\n",
        "y_pred"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=97, shape=(40,), dtype=float32, numpy=\n",
              "array([1.       , 1.1      , 1.2      , 1.3000001, 1.4000001, 1.5000001,\n",
              "       1.6000001, 1.7000002, 1.8000002, 1.9000002, 2.0000002, 2.1000001,\n",
              "       2.2      , 2.3      , 2.3999999, 2.4999998, 2.5999997, 2.6999996,\n",
              "       2.7999995, 2.8999994, 2.9999993, 3.0999992, 3.199999 , 3.299999 ,\n",
              "       3.399999 , 3.4999988, 3.5999987, 3.6999986, 3.7999985, 3.8999984,\n",
              "       3.9999983, 4.0999985, 4.1999984, 4.2999983, 4.399998 , 4.499998 ,\n",
              "       4.599998 , 4.699998 , 4.799998 , 4.8999977], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FURqcdZwH5HW",
        "colab_type": "code",
        "outputId": "30238c0d-1d15-40e7-e55f-ecde2eb7d314",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "plt.scatter(x, y)\n",
        "plt.plot(x, y_pred)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f212013c7b8>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcBklEQVR4nO3de3Bc53nf8e9DghcACxIkuLsiwasI\nkKot2iKNynLp2IoVW6qtSLKstFLq1MplOEkzuTSJNLI7jaadccUZZ5K4zR8Zjq3YaRwlGUdWXF9i\na8JkNO3YSkHRtlQrDimLkgnRBHjHhTcAT//Ys4vFCtiz2LOXc3Z/nxmNFrtHOI+OhAcvn/d93tfc\nHRERSZ5lzQ5ARESqowQuIpJQSuAiIgmlBC4iklBK4CIiCdXRyJtt2LDBt2/f3shbiogk3pEjR864\ne7r0/YYm8O3btzM8PNzIW4qIJJ6ZvbbQ+yqhiIgklBK4iEhCKYGLiCSUEriISEIpgYuIJFRDV6GI\niLSTZ46O8Klv/IA3LlxmU28nj9y5m/v29tfs+yuBi4jUwTNHR/j40y9y+foMACMXLvPxp18EqFkS\nVwIXEVlElBH0p77xg0Lyzrt8fYZPfeMHSuAiIvVUyQi6XIJ/48LlBb/vYu9XQ5OYIiILKDeChrkE\nP3LhMs5cgn/m6AgAm3o7F/y+i71fDSVwEZEFhI2gwxL8I3fupnPF8nmfd65YziN37q5ZjErgIiIL\nCBtBhyX4+/b288T9e+jv7cSA/t5Onrh/T2NXoZjZk8DdwKi731z0/q8BvwrMAF9190drFpWISJM9\ncufueTVwmD+C3tTbycgCSbw48d+3t7+mCbtUJSPwzwF3Fb9hZj8J3Au83d3fCvxe7UMTEWmesBF0\nI0okYUJH4O7+nJltL3n7V4CD7n41uGa09qGJiDRXuRF0/v16NuqEqXYZ4S7gJ8zsk8AV4Hfc/f8u\ndKGZHQAOAGzdurXK24mIxE+9SyRhqp3E7ADWA7cBjwB/ZWa20IXufsjdh9x9KJ1+04ESIiJSpWpH\n4CeBp93dgX80s1lgAzBWs8hERBqg3vuV1FO1I/BngJ8EMLNdwErgTK2CEhFphLBmnLgLTeBm9hTw\nLWC3mZ00s18EngRuNLOXgL8APhaMxkVEEiOsGSfuKlmF8tAiH320xrGIiDRUI/YrqSd1YopI22rE\nfiX1pAQuIm0rDs04UWg7WRFpW3FoxolCCVxE2lqzm3GiUAlFRCShlMBFRBJKCVxEJKGUwEVEEkqT\nmCKSaEneyyQqJXARSaxKTo5vZSqhiEhiJX0vk6iUwEUksZK+l0lUSuAiklhJ38skKiVwEUmspO9l\nEpUmMUUksZK+l0lUSuAikmhJ3sskqkpO5HnSzEaD03dKP/ttM3Mz21Cf8EREonnm6Aj7Dx5mx2Nf\nZf/Bw4k5Lq0SldTAPwfcVfqmmW0BPgC8XuOYRERqIulnXoYJTeDu/hxwboGP/gB4FNBZmCISS62+\nTryqVShmdi8w4u7freDaA2Y2bGbDY2Nj1dxORKQqrb5OfMkJ3My6gE8Av1vJ9e5+yN2H3H0onU4v\n9XYiIlVr9XXi1YzAdwI7gO+a2QlgM/CCmd1Qy8BERKJq9XXiS15G6O4vApn810ESH3L3MzWMS0QE\niLbbYKuvEw9N4Gb2FHA7sMHMTgKPu/tn6x2YiEgtdhts5XXioQnc3R8K+Xx7zaIRkVhq1p7b5VaR\ntGpSXgp1YopIWc3cc7vVV5FEpc2sRKSsZq6lbvVVJFEpgYtIWfUeBZdrdW/1VSRRqYQiImVt6u1k\nZIFkvZRR8GI19LDyTKuvIolKCVxEynrkzt3zkiwsbRRcLklXMknZyqtIolICF5Gyoo6CyyVpTVJG\nowQuIqHCRsHllhmWS9K1KM+0M01iikgkYVu2lltJoknKaJTARSTSoQdhywzLJen79vbzxP176O/t\nxID+3k6euH+Pat4VUglFpM1FbdQJq2OH1dA1SVk9JXCRNhe1Xb2SOraSdH2ohCLS5qKuBFEdu3mU\nwEXaXNR2ddWxm0clFJE2F7VRB1QiaRYlcJE2p3b15FICFxGNoBNKNXARkYQKTeBm9qSZjZrZS0Xv\nfcrM/snMvmdmXzKz3vqGKSIipSoZgX8OuKvkvWeBm939bcA/Ax+vcVwiIhIiNIG7+3PAuZL3vunu\n08GX3wY21yE2EREpoxY18F8Avr7Yh2Z2wMyGzWx4bGysBrcTERGImMDN7D8B08AXFrvG3Q+5+5C7\nD6XT6Si3ExGRIlUvIzSzh4G7gTvc3WsWkYiIVKSqBG5mdwGPAu9196nahiQiIpUITeBm9hRwO7DB\nzE4Cj5NbdbIKeNbMAL7t7r9cxzhFWl65U21EFhKawN39oQXe/mwdYhFpW1H35Jb2pFZ6kRioZE9u\njdCllBK4SAyE7cmtEbosRAlcJAbCTrWpxQhdI/jWo82sRGIg7FSbSkfoi50MH/a5JJMSuEgMhJ1q\nE3ZqTtjJ8GGfSzKphCISE+X25A47NSdshB713EuJJ43ARRIg6gg96rmXEk8agYs0UJSJxCgj9Fqc\neynxowQu0iD1XAoYdq6lzr1sTdbIfaiGhoZ8eHi4YfcTiZP9Bw8vuFSwv7eT//PY+5oQkSSFmR1x\n96HS91UDF2kQTSRKrSmBizSIJhKl1pTARRokrFlHZKk0iSmyBFFXkYAmEqV2lMClrURJwLVYRVJu\nKaDIUqmEIm0j6n4gakeXuAlN4Gb2pJmNmtlLRe+tN7NnzexY8Pd19Q1TJLqoCVirSCRuKhmBfw64\nq+S9x4C/c/dB4O+Cr0ViLWoC1ioSiZvQBO7uzwHnSt6+F/h88PrzwH01jkuk5qImYK0ikbiptgae\ndfdTwesfA9nFLjSzA2Y2bGbDY2NjVd5OJLqoCThsQymRRquold7MtgNfcfebg68vuHtv0efn3T20\nDq5Wemm2ep9Ko1NvpB4Wa6WvdhnhaTPb6O6nzGwjMBotPJHGqOcyPp1bKY1WbQnly8DHgtcfA/6m\nNuGIJJeWGUqjVbKM8CngW8BuMztpZr8IHATeb2bHgJ8KvhZpa1pmKI0WWkJx94cW+eiOGscikmhh\nJ8uL1Jo6MUVqRMsMpdG0F4okTlxXemizKmk0JXBJlLiv9NBmVdJIKqFIomilh8gcJXBJFK30EJmj\nBC6Jog2lROYogUui1HulxzNHR9h/8DA7Hvsq+w8ernivcJFm0CSmJEo9V3rEfYJUpJQSuCROvVZ6\nlJsgVQKXOFIJRSSgCVJJGiVwkYAmSCVplMBFAmqFl6RRDVwkoFZ4SRolcJEiaoWXJFEJRUQkoZTA\nRUQSSglcRCShItXAzew/Ar8EOPAi8PPufqUWgYlUK677hYvUWtUjcDPrB34dGHL3m4HlwIO1Ckyk\nGvl2+JELl3Hm2uG1p4m0oqgllA6g08w6gC7gjeghiVRP+4VLO6k6gbv7CPB7wOvAKeCiu3+z9Doz\nO2Bmw2Y2PDY2Vn2kIhVQO7y0kygllHXAvcAOYBPQbWYfLb3O3Q+5+5C7D6XT6eojFamA2uGlnUQp\nofwU8Kq7j7n7deBp4F/VJiyR6qgdXtpJlFUorwO3mVkXcBm4AxiuSVQiVVI7vLSTqhO4uz9vZl8E\nXgCmgaPAoVoFJlIttcNLu4i0DtzdHwcer1EsIiKyBOrEFBFJKO1GKLGjTkqRyiiBS6zoYGGRyqmE\nIrGiTkqRymkE3qbiWqZQJ6VI5TQCb0Nx3vBJnZQilVMCb0PNLlM8c3SE/QcPs+Oxr7L/4OF5vzjU\nSSlSOZVQ2lAzyxRhk5TqpBSpnBJ4glVbx97U28nIAsm6EWWKcqP/fOzqpBSpjEooCRWljt2IMsVi\nZRJNUorUjkbgCVXJSHaxEXq9yxTlyiTNHP2LtBol8IQKG8lWUmuOkrDLlW/K/XJ55M7d8+ICTVKK\nVEsllIQKW25Xz5UmYeWbcr9c7tvbzxP376G/txMD+ns7eeL+Pap5i1RBI/CEChvJRq01VzvCvm9v\nf2iZRJOUIrWhEXhChY1kozTERBlhg9ZyizSKRuAJVm4kG6XWXIsRdv77aC23SP1ESuBm1gt8BrgZ\ncOAX3P1btQhMoomSRCsZYYf9clCZRKT+oo7APw38rbs/YGYrga4axCQ1Um0S1QhbJBmqTuBmthZ4\nD/AwgLtfA67VJixpJo2wRZIhyiTmDmAM+BMzO2pmnzGz7tKLzOyAmQ2b2fDY2FiE20mjaKmfSDKY\nu1f3D5oNAd8G9gcn1H8auOTu/3mxf2ZoaMiHh4eri1REpE2Z2RF3Hyp9P0oN/CRw0t2fD77+IvBY\nhO8nJeJ66IKIxEPVJRR3/zHwIzPLF0bvAL5fk6gk1ocuiEg8RG3k+TXgC2b2PeAW4L9FD0mg+Ycu\niEj8RVpG6O7fAd5Ul5HotO2qiIRRK31M6WxIEQmjBB5RufMdo9B+IiISRnuhRBC253YU6nYUkTBK\n4BFUcipOFOp2FJFyVEKJQBONItJMSuARaKJRRJpJCTwCTTSKSDOpBh6BJhpFZCHuzo8vXeHY6QmO\njU6wM93N7bszNb+PEnhEmmgUaV+zs87IhcscH53g2Oh4IWEfH51g4up04bqffedWJfBm0IZSIjIz\n6/zo3BTHgkR9vChRF69ES/esYjCT4v59/QxmUgxkehjMpujrXlmXuJTAy6jnOm8RiZ/rM7O8dnaK\n40Wj6WOjE7wyNsG16dnCdRvXrmYgk+KhW7cymE0xkEkxmEnR21WfRL0YJfAy6r3OW0Sa4+r0DCfO\nTBXKHvkSyKtnJrk+M3dGwuZ1nQxmUrx7oI/BbA+DmRQ7MynWrF7RxOjnKIGXoXXeIsl2+doMr4xN\nzKtRHx+d4LVzU8zM5hL1MoMt67sYzPRwx7/IMphJMZjpYWemm66V8U6R8Y6uycIO9xWReJi8Oh0k\n6fk16h+dnyJ/6NjyZcb2vi52ZXv40Ns2BmWPHm5Md7O6ZDlwUiiBl1HJ4b5RaIJUZGkuTl3n+FhJ\nfXp0Yt5Aa+XyZdyY7uZtm9fykX2bGcjkatQ7NnSzsqO1Wl+UwMuo5zpvTZCKLO7c5DWOnR4vrPTI\nlz9Gx68WrlnVsYyBTIp/uX0dP5vdys50il3ZFFvXd9GxvLUS9WKUwENEWeddboStCVJpd+7O2PjV\nNyXp46MTnJ28Vriue+VyBrI9/MRgml3ZVG7VR7qH/nWdLF9mTfw3aL7ICdzMlgPDwIi73x09pNYQ\nNsLWBKm0C3fn1MUruZLH6fG5WvXpcS5dmWt2WbO6g8FsD+9/SzZXnw5WfWxcuxqz9k7Ui6nFCPw3\ngJeBNTX4Xi0jbIStCVJpNfmuxGOla6hLuhLXd69kIJPip9++KbfiI0jU6Z5VStRLFCmBm9lm4EPA\nJ4HfqklELSJshF3vCVKRepmZdV4/N/WmGvXx0QmuXJ9rdsn0rGIwm+KBd8xNJA5mUvSlVjUx+tYS\ndQT+h8CjQM9iF5jZAeAAwNatWyPeLjnCRtjaCEviLteVODlvNH3s9Dg/PDM5rytx09rVDGR7+Hfv\n7GMgk5tIHEj3sLYrHs0urazqBG5mdwOj7n7EzG5f7Dp3PwQcAhgaGvLFrms1lYywtRGWxMGV6zO8\nemYyN5oORtXHRic4cWaS6dm5H9kt6zsZSKd47650oUa9M91NT0y6EttRlBH4fuAeM/sgsBpYY2Z/\n5u4frU1otdGstdYaYUvc5LsSS3fNe+3sJPk8vcxgW183A5kUH3hLlsHsXLNL3LsS25G5Rx8UByPw\n3wlbhTI0NOTDw8OR71ep0pUgkBsFP3H/HiVSaVkT+a7E4hUfo+OcPH+50JXYsczYsaE7WJKXYiCY\nSNyxIbldia3MzI64+1Dp+y3xK3WxUbbWWksruzh1PTeaHp3IrZ8ey5VA3rh4pXBNvivxli3r+Jl3\nbCnUqLf1dbOiTZpdWllNEri7/wPwD7X4XktVbr211lpLKzg7cbVQly6uUY8VdSWuXpHrSrx1x/rC\nsrzBbA9b1nW2TVdiO0r8CLzcKFtrrSUp3J3R8atBbTpYnhe8Pj91vXBdalUHOzMpbt+VLtSnBzIp\n+ns7WdbmXYntKPEJvNwo+w/+7S1aay2x4u68cfHKXH26KGGPF3Ulru1cwWAmxV0335A71SWTayG/\nYY26EmVO4hN4uVG2VoJIs8zOOifPX55fow6aXSavzQ0oNqRyXYn33dJfNKGYIp1SV6KES3wCD1tv\nrbXWUk/TM7O5rsR8R2JQo35lbH5XYnbNKgYzPfzMUH4iMVf6WF+nsxKlPSQ+gUcdZWtPbqnEtelZ\nTgRdicWt4z8cm+TazFyi7u/tZGcmxbtu7AvOSswl6rWdanaR2kt8AofqR9nak1tKXbk+ww/HJgsJ\nOl+jPnF27ggus9xZibsyPbx3d5rBzNxZialVLfEjJQnR1v+3aZ14+5q6Ns0ro5NvqlG/fm6q0JW4\nfJmxbX0XA8FkYn7Fx850is6VanaR5mvrBK514q1v/Mr1QjdicY365Pm5/8Yrlue6Et+6aS335icT\ng67EVR1K1BJfbZ3AtU68dVyYulYYSefLH8dHJzhV3JXYsYyd6RT7tq7j3wxtye2al+lhW1+XuhIl\nkdo6gWtP7mRxd85MXJtXn86Prs9MzHUldq1czs50inft7CvUpwcyKbas72r7I7iktbR1Atc68Xhy\nd05fulqya16u9HGhqCuxZ1UHA9kU77spN5E4kM0dGLBprboSpT20dQIHrRNvptlZ542Ll+e1jedf\njxcdwdXbtYJdmR4+uGcjA+lUoYU8u0bNLtLe2j6BS/3NzDonz08VnewyV6OemteVuIrBTIoP7+sv\nLMvble2hr3ulErXIApTApWZyR3BNBcl5bnneK2MTXC06guuGNasZzKaCicSeQgv5OnUliiyJErgs\n2dXpGU6cmSrUqPOdia+emeT6zNwBIf29nQxmU+wf6CvUqAcyKdboCC6RmlACl0VduZ47gqt017zX\nSroS880u77spW9g1b2c6Rbe6EkXqKsqhxluAPwWygAOH3P3TtQpMGmfy6nTurMSSFR+vn5sqHMG1\nfJmxva+LXZkePrRnIwPB0ryd6ZSO4BJpkihDpGngt939BTPrAY6Y2bPu/v0axSY1dvHy9bn6dHAE\n17HTE/OamVYsN27ckOLm/rV8eG9/Yee87X3drOxQs4tInFSdwN39FHAqeD1uZi8D/YASeJOdn7xW\nWO1RXKM+fWmu2WVV0JX4jm3reOjWLblDA7Iptq3v0hFcIglRkyKlmW0H9gLPL/DZAeAAwNatW2tx\nO2HhrsT86zMT1wrXda1czkAmxbsH8kdw5Uofm9epK1Ek6SIncDNLAX8N/Ka7Xyr93N0PAYcAhoaG\nvPRzKc/d+fGlK/Pr00H5Y15X4uoOBjMp7rgpW9iMaTDbw8Y1q9WVKNKiIiVwM1tBLnl/wd2frk1I\n7Wl21hm5cLlkRJ0rf0wUdSWu61rBYHZuIjG/xam6EkXaT5RVKAZ8FnjZ3X+/diG9WSudmjMz67kj\nuIJtTfP16VdGJ+dtqpXuyXUlfmRff2E0PZhJ0Zda1cToRSROoozA9wM/B7xoZt8J3vuEu38telhz\nknpqTq4rcbKofTy3F/UPz0xyragrcePa1QxkUjx0a9+8GnVvl7oSRaS8KKtQ/jdQ9z+zx/3UnKvT\nM7x6ZvJNNepXz0wyPTu/K3FXNsV7dqWD0kcuUfeoK1FEqhT7Vrm4nJpz+VpRV2LR8rwTZycLR3At\nM9jW181AJsX73xJMJqZ72Jnppmtl7B+1iCRM7LNKo0/Nmbg6XTh6K39YwLHRcU6ev1zoSuxYljuC\n66aNPdz99k25rsR0ihvT3epKFJGGiX0Cr9epORenrnN8bHxejfr46XHeKD6Ca/kybkx3c8uWdTyw\nL3cE12A2xba+bh3BJSJNF/sEHvXUnLMTV+cl6PzKj9Hxua7E1StyXYnvvLGvUJ8ezPawZV2nuhJF\nJLZin8Ah/NQcd2ds/GphpUchYY9OcG5yrisxtaqDnZncRGJ+17zBTA/9vTqCS0SSJxEJPM/dOXXx\nSiFRHy9annfpylyzy5rVHezK9nDnW7O5PT6CFR8b165Ws4uItIzEJPAHD32LF09eZLLoCK6+7pUM\nZFLcc8smBtK5XfMGsinSKXUlikjrS0wC35lOcdMNa+atoVZXooi0s8Qk8E9+eE+zQxARiRUtsRAR\nSSglcBGRhFICFxFJKCVwEZGEUgIXEUkoJXARkYRSAhcRSSglcBGRhDL3xh0Ub2ZjwGsRvsUG4EyN\nwqk1xbZ0cY0LFFu1FFt1wmLb5u7p0jcbmsCjMrNhdx9qdhwLUWxLF9e4QLFVS7FVp9rYVEIREUko\nJXARkYRKWgI/1OwAylBsSxfXuECxVUuxVaeq2BJVAxcRkTlJG4GLiEhACVxEJKFil8DN7EkzGzWz\nlxb53Mzsv5vZcTP7npnti1Fst5vZRTP7TvDX7zYori1m9vdm9n0z+39m9hsLXNOU51ZhbM16bqvN\n7B/N7LtBbP9lgWtWmdlfBs/teTPbHqPYHjazsaLn9kuNiC2493IzO2pmX1ngs6Y8swpja+YzO2Fm\nLwb3HV7g86X/jLp7rP4C3gPsA15a5PMPAl8HDLgNeD5Gsd0OfKUJz2wjsC943QP8M/CWODy3CmNr\n1nMzIBW8XgE8D9xWcs1/AP44eP0g8Jcxiu1h4I8a/dyCe/8W8OcL/Xdr1jOrMLZmPrMTwIYyny/5\nZzR2I3B3fw44V+aSe4E/9ZxvA71mtjEmsTWFu59y9xeC1+PAy0B/yWVNeW4VxtYUwbOYCL5cEfxV\nOqt/L/D54PUXgTusASdmVxhbU5jZZuBDwGcWuaQpz6zC2OJsyT+jsUvgFegHflT09UlikhAC7wr+\n2Pt1M3tro28e/HF1L7kRW7GmP7cysUGTnlvwx+3vAKPAs+6+6HNz92ngItAXk9gAPhL8cfuLZral\nEXEBfwg8Cswu8nnTnhnhsUFznhnkfgF/08yOmNmBBT5f8s9oEhN4nL1Abs+CtwP/A3imkTc3sxTw\n18BvuvulRt47TEhsTXtu7j7j7rcAm4FbzezmRt07TAWx/S9gu7u/DXiWuVFv3ZjZ3cCoux+p972W\nqsLYGv7Mirzb3fcB/xr4VTN7T9RvmMQEPgIU/9bcHLzXdO5+Kf/HXnf/GrDCzDY04t5mtoJcgvyC\nuz+9wCVNe25hsTXzuRXFcAH4e+Cuko8Kz83MOoC1wNk4xObuZ939avDlZ4B3NCCc/cA9ZnYC+Avg\nfWb2ZyXXNOuZhcbWpGeWv/dI8PdR4EvArSWXLPlnNIkJ/MvAvw9mbG8DLrr7qWYHBWBmN+RrfWZ2\nK7nnW/f/cYN7fhZ42d1/f5HLmvLcKomtic8tbWa9wetO4P3AP5Vc9mXgY8HrB4DDHsw4NTu2kvro\nPeTmF+rK3T/u7pvdfTu5CcrD7v7Rksua8swqia0Zzyy4b7eZ9eRfAx8ASlezLflntKMu0UZgZk+R\nW5WwwcxOAo+Tm8DB3f8Y+Bq52drjwBTw8zGK7QHgV8xsGrgMPNiI/3HJjTx+DngxqJkCfALYWhRb\ns55bJbE167ltBD5vZsvJ/dL4K3f/ipn9V2DY3b9M7pfP/zSz4+QmsB9sQFyVxvbrZnYPMB3E9nCD\nYnuTmDyzSmJr1jPLAl8KxikdwJ+7+9+a2S9D9T+jaqUXEUmoJJZQREQEJXARkcRSAhcRSSglcBGR\nhFICFxFJKCVwEZGEUgIXEUmo/w88XJMtpi2sFwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cgi9VI9Ydgv4",
        "colab_type": "text"
      },
      "source": [
        "Our line doesn't look great. But how do we know when our model is doing well? We will define a numeric measure of how poorly our model is doing. In machine learning, this is called a **loss function**. The larger our loss, the worse our model is. \n",
        "\n",
        "For regression, a common loss function is **mean squared error** (MSE):\n",
        "\n",
        "MSE = $\\frac{1}{N}\\sum^N_{i=1} (y_{i, true} - y_{i, predicted})^2$\n",
        "\n",
        "Our goal is to minimize this loss function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4DsEqmoTaU9n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def mse(y_true, y_pred):\n",
        "  # TensorFlow has this function built in: tf.keras.losses.MSE(y_true, y_pred)\n",
        "  return tf.reduce_mean((y_true - y_pred)**2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yOBsPIiUeuuz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d3f38f77-a299-447c-9e94-3fa469209102"
      },
      "source": [
        "mse(y, y_pred) # want to minimize this value!"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=182, shape=(), dtype=float32, numpy=49.60046>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gzaG4KcUexoV",
        "colab_type": "text"
      },
      "source": [
        "For linear regression, we know there's a closed form solution. let's just compute that:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u5K54xGfIKLI",
        "colab_type": "code",
        "outputId": "9f6b5860-43fb-478f-fad7-c23059e0cc74",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "sxx = tf.reduce_mean((x - tf.reduce_mean(x))**2)\n",
        "sxy = tf.reduce_mean((x - tf.reduce_mean(x))*(y - tf.reduce_mean(y)))\n",
        "\n",
        "m.assign(sxy/sxx)\n",
        "b.assign(tf.reduce_mean(y) - m*tf.reduce_mean(x))\n",
        "\n",
        "y_pred = x*m + b\n",
        "\n",
        "print(f'Loss: {mse(y, y_pred).numpy()}')"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 0.4125025272369385\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GKk6RCsGe3Wc",
        "colab_type": "text"
      },
      "source": [
        "Our loss went down (and is minimized, because this is a closed form solution)! Let's check out how this looks graphically:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n12V7C-hIkCs",
        "colab_type": "code",
        "outputId": "3c63c7c9-742a-4a70-fce2-28cf8968d552",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "plt.scatter(x, y)\n",
        "plt.plot(x, y_pred)\n",
        "\n",
        "print(f'm: {m.numpy()}   b: {b.numpy()}')"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "m: 3.5368335247039795   b: -1.110854148864746\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyU1dn/8c8FARL2JayBEEB2EAIR\nUFwQVFwBUau21K0+tNWfSx+Lj9YKYrWK+9LFUrVqXWqrVhYRxQVRq9iACwkkgKxhDQhhSUK28/sj\nQSFMZiYzk2TuzPf9evEyzEzmvrw13zm57nOfY845RETEexrUdQEiIhIaBbiIiEcpwEVEPEoBLiLi\nUQpwERGPiqvNgyUmJrqUlJTaPKSIiOctW7Zsl3OufeXHazXAU1JSSE9Pr81Dioh4nplt9PW4Wigi\nIh6lABcR8SgFuIiIRynARUQ8SgEuIuJRtToLRUQklrz55RYefCebrXsL6NI6gWnj+zIpNSli768A\nFxGpAW9+uYXb31hBQXEpAFv2FnD7GysAIhbiCnARkSqEM4J+8J3s78P7sILiUh58J1sBLiJSk4IZ\nQfsL+K17C3y+b1WPh0IXMUVEfPA3goYfAn7L3gIcPwT8m19uAaBL6wSf71vV46FQgIuI+BBoBB0o\n4KeN70tCo4ZHPZ/QqCHTxveNWI0KcBERHwKNoAMF/KTUJH5/4SDaNG0EQFLrBO6bPDiis1ACBriZ\nPWtmO80so9LjN5hZlpllmtkDEatIRCQKBBpBBwr4lVv38fIXm9iTX8yP0rry6W1jIxreENwI/Dng\n7CMfMLPTgYnAEOfcQOChiFYlIlLHJqUmcd/kwSS1TsA4dgRdVcBff3ov7pqbyflPfsy3uQeZddFg\n7p98fI3UGHAWinNuiZmlVHr4l8D9zrlDFa/ZGfnSRETq1qTUpCpHzYcfPzwLpXOreMb07cAji1az\n+2ARU0Z255az+tC6aeMaqy/UaYR9gFPM7F6gEPi1c+6/vl5oZlOBqQDJyckhHk5EJPocDvjMrXlM\nn5PJy19sIjW5Nc9dPYJBSa1q/PihBngc0BYYBZwA/NPMejrnXOUXOudmA7MB0tLSjnleRMSr8gqK\neeTdbP7++UbaNG3MAxcfz8XDutKggdXK8UMN8BzgjYrA/sLMyoBEIDdilYmI1IJQ7rYsK3O8tjyH\nWW9nsSe/iCmjunPLmX1pVTHjpLaEGuBvAqcDH5pZH6AxsCtiVYmI1IJQ1ivJ2JLH9DkZLN+0l+Hd\n2/DCxBEM7FLz7RJfAga4mb0CjAESzSwHmAE8CzxbMbWwCLjSV/tERCSaVWe9krz8Yh56N5uXlpa3\nSx66ZAiTU5NqrV3iSzCzUC6v4qkpEa5FRKRWBbNeSVmZ41/LNjNrYTZ784u44sQUfnVmH1ol1G67\nxBctZiUiMatL6wS2+AjxwzfjrMjJ4845GXy1eS9p3dtw98SRDOjSsrbLrJICXERi1rTxfY/qgUP5\nzTjXjenFHf9ewctfbKJdsyY8fMkQJg9Lwqzu2iW+KMBFJGb5uhnnlN7teejdbPYVlnDVSeXtkpbx\ndd8u8UUBLiIx7fDNON/k7OXOOZm8mr6ZESltmTlxIP07R0+7xBcFuIjEtD0Hi3jw3Wxe+WITic2b\n8NilQ5k4tEvUtUt8UYCLSEwqLXP847+bePCdbPYXlnDN6B7cfEZvWkRpu8QXBbiIxJyvNu9l+pwM\nvsnJY2SPttw9cRB9O7Wo67KqTQEuIjHju4NFPLAwi1fTN9O+eRMev2woE4Z4o13iiwJcROq90jLH\nK1+Ut0sOHirh2pN7cOM4b7VLfFGAi4inBVqMavmmPcyYk8mKLXmc2LMdd08cSO+O3muX+KIAFxHP\n8rcY1Sm9E5m1MIt/pufQsWUTnrg8lQuO7+zZdokvCnAR8ayqFqOaMTcT5xz5RaX8/NSe3DCuN82b\n1L+4q3//RiISM6pajCqvoJiTerVj5oT60y7xRQEuIp5V1WJUbZo24qVrR9ardokvwexKLyISlf73\nzD40qrQed3xcA2ZcMLDehzdoBC4iHpW+4Tue/mQ9xWWOJnENOFRSRlKQW6LVFwpwEfGU3P2HuP/t\nLF5fnkPnVvH86SfDOGdQp5gYcVcWzJZqzwLnAzudc4MqPXcL8BDQ3jmnPTFFpMaUlJbx98838sii\n1RQWl/LLMb24YexxNG3sP8ZC2bTYK4IZgT8H/AF44cgHzawbcBawKfJliYj84L8bvuPONzPI2r6f\nU3oncteEgfRq3zzg94WyabGXBLMn5hIzS/Hx1KPArcCcCNckIgLAzv2F3L8gize+3EKXVvE8NWUY\n4wcG3y6pzqbFXhRSD9zMJgJbnHNfBzqRZjYVmAqQnJwcyuFEJMaUlJbx/GcbeWzRag6VlHH96b24\n/vTA7ZLKgtm02MuqHeBm1hT4DeXtk4Ccc7OB2QBpaWmuuscTkdiydN1uZszNJGv7fk7t056ZEwbS\nI7FZSO8VaNNirwtlHngvoAfwtZltALoCy82sUyQLE5HYsnNfITf/40sunf05+wtLeGrKcJ6/+oSQ\nwxvKNy1OaNTwqMcSGjVk2vi+4ZYbFao9AnfOrQA6HP57RYinaRaKiISiuLSM5/+zgcfeW0NRSRk3\njD2O68YcR0Lj8uANZxZJ5U2LY24Wipm9AowBEs0sB5jhnHumpgsTkfrvs293M2NuBqt3HGBM3/bM\nuODodkkkZpEc3rS4PgpmFsrlAZ5PiVg1IhKVIj2Xese+Qu59axVzv95K1zYJzP7pcM4c0PGY2SX1\nfRZJuHQnpoj4Fcm51MWlZTz36QYee281xWWOG8f15roxvYiv1Kc+rL7PIgmXAlxE/IrUKPg/3+5i\nxpxM1uw8wNh+HZhxwQC6t/N/gbK+zyIJlwJcRPwKdxS8Pa+QexesYt7XW+nWNoGnr0jjjAEdv3/e\nX3tm2vi+R43+oX7NIgmXAlxE/Ap1FFxUUsbfPl3PE++vobjMcdO43vyyUrskUHumvs8iCZcCXET8\nCmUU/OnaXUyfk8G3uQc5o38HRvVsx98+3cAT7685KoSDac/U51kk4VKAi4hf1RkFb8sr4J63VvHW\nN9tIbtuUZ65MY39hSZWjbF2kDI8CXEQCCjQKfi19M3fPX8m+whIAzh7UiccuHUp8o4aMvv+DKkfZ\nukgZHm2pJiJhmfV2FtNe++b78Ab4KDuXhRnbAf+j7Pp+q3tN0whcREK6UWfr3gLueWslC1ZsP+a5\nI/vY/kbZukgZHgW4SIyr7o06RSVlPP3JOp58fy2OqhcYPTzyDnQRVBcpQ6cWikiM8zcTpLIlq3M5\n+7ElPLAwm1N6J7LoV6eRVEW/+nAfe1JqEvdNHkxS6wQMSGqdwH2TByu0I0AjcJEYF8xMkC17C7hn\n/krezthOSrumPHf1CYzpW74oaTDTDDXKrhkKcJEY569HfaiklKc/Xs+TH6wBysP62lN60CTuhwuP\n6mPXHXOu9jbJSUtLc+np6bV2PBEJrHIPHMpH0Fec2J13V+5g/a6DnDOoE789f0CV7RKpWWa2zDmX\nVvlxjcBFYlzlEXSHFk3o0DKevyxZR8/EZrxwzQhO7dO+jqsUXxTgIsKk1CTOHtSJvy5Zxx8Xr2Vf\nYQm3nt2Xn518dLtEoosCXET4MGsnd83LZOPufM4b3Jk7zuuvuyE9QAEuEsM2f5fPzHkreW/VDnq2\nb8aLPxvJyb0T67osCVIwe2I+C5wP7HTODap47EHgAqAI+Ba42jm3tyYLFZHIKSwu5S8freNPi9fS\nsIFx2zn9uGZ0DxrH6dYQLwnmv9ZzwNmVHlsEDHLOHQ+sBm6PcF0iUkPeX7WDsx5dwqPvreaMAR15\n/5bT+MVpvRTeHhTMpsZLzCyl0mPvHvHXz4GLI1uWiETapt353D0/k/dW7eS4Ds156dqRjD5O7RIv\ni0QP/Brg1aqeNLOpwFSA5OTkCBxORKqjsLiUPy/+lj9/9C1xDYzbz+nH1WqX1AthBbiZ3QGUAC9V\n9Rrn3GxgNpTfyBPO8USket5buYOZ8zPZ/F0BFwzpwh3n9qdTq/i6LksiJOQAN7OrKL+4Oc7V5u2c\nIhLQxt0HmTlvJR9k7aR3h+a8/D8jOamX2iX1TUgBbmZnA7cCpznn8iNbkoiEqqColD8vXstTS9bR\nqIFxx7n9uWp0Co0aql1SHwUzjfAVYAyQaGY5wAzKZ500ARaZGcDnzrlf1GCdIvVeKJsqHOacY9HK\nHdw9fyU5ewqYOLQLvzm3Px1bql1SnwUzC+VyHw8/UwO1iMSs6m6qcKQNuw5y17xMFmfn0qdjc175\nn1Gc2KtdjdcsdU93YopEAX+bKhwO8Moj9JvG9Wbznnz+8tE6Gsc14Lfn9efKk9QuiSUKcJEoEGhT\nBV8j9P97/RsccGFqEref048OapfEHAW4SBTwt6kC+B6hOyCxeWMevXQoELiHHk6PXaKTftcSiQLT\nxvclodHRy7YeuS2Zr3AH2H2gCPhhhL5lbwGOH3rob365JajnxZsU4CJRoKqNfycO7cLbK7bRsHy2\n1zH8jdCP3Ji4OhsXi3eohSISJSpv/Ptt7gGuePYLPl6ziy6t4tl1oIii0rLvnz9yhB6ohx7MxsXi\nPQpwkSiTX1TCkx+s5emP1xEf15C7LhjAlFHdmf/Ntip72IF66IGeF29SgIvUIn8XEp1zvJ2xnXvm\nr2RrXiEXDevKbef0o32LJsCxI/QjTRvf1+fGxIdH6IGeF29SgIvUEn836wxKasVdczP5ZO0u+ndu\nyROXp5KW0jbo9668MXHlD4dAz4s3WW2uQ5WWlubS09Nr7Xgi0WT0/R/4bGM0bxLHoZJS4hs15Ndn\n9eUnI5OJ0804cgQzW+acS6v8uEbgIrWkqguGBw6VcMnwrvzfOf1IbN6klqsSL9PHvEgtqeqCYWLz\nxjx4yRCFt1SbAlykltww9jjiGhw9nzs+rgG/PW9AHVUkXqcWikg1hHI7unOOed9s49H3VlNS5mja\nuCH5RaUk6UKihEkBLhKkUJZ8Xb1jPzPmZPLZut0MSmrJn6cMZ1hym1qrWeo3BbjElHAWdApmydfD\nDhwq4fH3VvO3TzfQrEkcv5s0iB+PSKZhA9+3xIuEQgEuMSOcTRMguNvRnXPM/Xor9761itwDh7g0\nrRu3nt2Pts0aR+DfQORoAS9imtmzZrbTzDKOeKytmS0yszUV/9TvhBL1wl3QqapZJIcfz96+n8tm\nf85N//iKTq3i+fd1o7n/ouMV3lJjgpmF8hxwdqXHbgPed871Bt6v+LtIVAt3Qaeqlny9Yexx/G7+\nSs594mOyd+zn9xcO5t/XjWZot9Zh1yziTzB7Yi4xs5RKD0+kfKNjgOeBxcD/RbAukYgLd0Gnyrej\nd24Vz9h+HXh40Wp2HTjEZSckc+v4vrTRiFtqSajzwDs657ZVfL0d6FjVC81sqpmlm1l6bm5uiIcT\nCV+gTROCMSk1iU9vG8uCm06ha5umvLh0E11axfPmdaO5b/JghbfUqrAvYjrnnJlVuaCKc242MBvK\n10IJ93gioYrEgk77Cot5dNFqXvhsIy3j47h/8mB+lNaNBhWzS7RtmdSmUAN8h5l1ds5tM7POwM5I\nFiVSU/wtyeqPc443lm/hvrez2H3wEJePSGbaWUe3S8Kd5SJSXaEG+FzgSuD+in/OiVhFIlFm5dZ9\nTJ+TQfrGPQzt1ppnr0rj+K7HXqCszjxxkUgIGOBm9grlFywTzSwHmEF5cP/TzH4GbAR+VJNFitSF\nvILD7ZINtG7amFkXDeaS4T+0SyrTtmVS24KZhXJ5FU+Ni3AtIlGhrMzx+vIcZi3MYvfBIqaM7M4t\nZ/WhdVP/Fyi1bZnUNt2JKXKEjC15TJ+TwfJNe0lNbs1zV49gUFKroL5X25ZJbVOAi+fUxEyPvPxi\nHl6UzYufb6R108Y8cNHxXDy8a5XtEl+0bZnUNgW4eEqkZ3qUlTleW57DrLez2JNfxJRR3bnlzL60\natoopPpCneUiEgoFuHhKJGd6ZGzJ4845GXy5aS/Du7fhhYkjGNgluHaJSDRQgIunRGKmR15+MQ+9\nm81LSzfStlljHrpkCJNTk6rVLhGJBgpw8ZRwZnqUlTn+tWwzsxZmsze/iCtOTOFXZ/ahVUJo7RKR\nuqYAF08JdabHipzydslXm/dyQkobZk4YyYAuLY95nW6FFy9RgIunVHemx56DRTz4bjavfLGJds2a\n8MiPhnBhahJmx7ZLdCu8eI0CXDwnmJkeZWWOV9M388DCLPYVlnDVSeXtkpbxVbdLdCu8eI0CXOqd\nrzfvZfqcDL7OyWNESltmThxI/87Htksq063w4jUKcKk39hws4oF3svnHfzeR2LwJj146hElDfbdL\nfNGt8OI1CnDxvNIyxz/+u4kH38lmf2EJ14zuwc1n9KaFn3aJL7oVXrxGAS6e9lVFu+SbnDxG9mjL\n3RMH0bdTi5DeS7fCi9cowMWTvjtYxAMLs3g1fTPtmzfh8cuGMmFIl6DbJVXRrfDiJQpw8ZTSMscr\nX5S3Sw4eKuHak3tw47jqt0tE6gMFuHjG8k17mD4ng4wt+zixZztmThxIn46htUtE6gMFuES93QcO\nMWthFv9Mz6FjyyY8eXkq5x/fOex2iYjXhRXgZvYr4FrAASuAq51zhZEoTKS0zPHS0o089E42+UWl\n/PzUntwwrjfNm2jcIQJhBLiZJQE3AgOccwVm9k/gMuC5CNUmMWzZxvJ2SebWfZzUqx0zJwykd5Dt\nEq1nIrEi3KFMHJBgZsVAU2Br+CVJLNt14BD3v53Fa8ty6NQynj/8OJXzBgffLtF6JhJLQg5w59wW\nM3sI2AQUAO86596t/DozmwpMBUhOTg71cFLPlZSW8dLSTTz0bjYFRaX8/LSe3Di2N82q2S7ReiYS\nS8JpobQBJgI9gL3Av8xsinPuxSNf55ybDcwGSEtLc2HUKvVU+obvuHNOJqu27ePk4xK5a8JAjuvQ\nPKT30nomEkvCaaGcAax3zuUCmNkbwEnAi36/S6RC7v7ydsnry3Po3CqeP/1kGOcM6hTW7BKtZyKx\nJJwA3wSMMrOmlLdQxgHpEalK6rWS0jJe/HwjDy9aTWFxKb8c04sbxh5H08bhzy7ReiYSS8LpgS81\ns9eA5UAJ8CUVrRKRqvx3w3fc+WYGWdv3c0rv8nZJr/ahtUt80XomEkvMudprS6elpbn0dA3SY9HO\n/YXcvyCLN77cQpdW8Uy/YADjB4bXLhGJFWa2zDmXVvlx3REhNaqktIznP9vIY4tWc6ikjOtP78X1\np0emXSIS6/RTJDVm6brdzJibSdb2/Zzapz13XTCAnhFsl4jEOgW4RNzOfYX8fsEq3vxqK0mtE3hq\nynDGD+xYrZtx1MMWCUwBLhFTXFrG8//ZwGPvraGopIz/d/pxXH/6cSQ0bhj0e+hOSpHgKcAlIj5f\nt5vpczJYveMAY/q2Z8YFA+mR2Kza76M7KUWCpwCXsOyoaJfM+WorXdskMPunwzlzQPDtksp0J6VI\n8BTgMSrcPnNxaRnPfbqBx95bTXGZ48ZxvbluTC/iGwXfLvFFd1KKBE8BHoPC7TP/59tdzJiTyZqd\nBxjbrwMzLhhA93bBt0v8fXjoTkqR4CnAY1CofebteYXcu2AV877eSre2CTx9RRpnDOhYrWMH+vDQ\nnZQiwVOAe1iobZDq9pmLSsr426freeL9NZSUOW4+oze/OC20dkkwHx7aGV4kOApwjwqnDVKdPvOn\na3cxfU4G3+Ye5Iz+HZh+/kCS2zUNqj5fHy66SCkSOQpwjwpmJFtViAbTZ96WV8A981fx1optJLdt\nyrNXpTG2X3DtEn8fLrpIKRI5CnCPCjSSDWaE7ivci0rKeOaT9Tz5wRpKyxy/OqMPPz+t5zHtEn/t\nG38fLrpIKRI5CnCPCjSSDTRC99Vn/nhNLjPmZrIu9yBnDujI9PMH0K3tse2SQB8O/j5cdJFSJHIU\n4B4VaCRbnV7z1r0F3PPWShas2E73dk3521UnkFdQzGWzP6/2CHtSalLADxddpBSJjAZ1XYCEZlJq\nEvdNHkxS6wQMSGqdwH2TB38fjFX1lI98/FBJKX/8cC3jHv6ID7J2csuZfXjn5lPJKyjm9jdWsGVv\nAY4fRthvfrkFCPzhMG18XxIqtVzUJhGJPI3APczfSDbQCP2j1bncNTeT9bsOMn5gR+48fwBd25S3\nSyIxwj78PmqTiNQcBXg9VVWIntCjLb/4+zIWZm4npV1Tnrv6BMb07XDU9wYzwg50IVJtEpGaF1aA\nm1lr4GlgEOCAa5xzn0WiMAnfkSF6qKSUpz9ez20PLwbKQ/jaU3rQJO7Ym3E0whbxhnBH4I8DC51z\nF5tZYyDwHR5S6xZn72TmvJWs33WQcwZ14rfnDyDJz7xrjbBFvCHkADezVsCpwFUAzrkioCgyZUkk\n5OzJ53fzV/JO5g56JjbjhWtGcGqf9gG/TyNsEW8IeVd6MxsKzAZWAkOAZcBNzrmDlV43FZgKkJyc\nPHzjxo1hFSyBFRaX8tcl6/jj4rUYxg3jjuNnJ/tul4hI9KtqV/pwAjwN+BwY7ZxbamaPA/ucc3dW\n9T1paWkuPT09pONJcD7M2sld8zLZuDuf8wZ35o7z+us2dRGPqyrAw+mB5wA5zrmlFX9/DbgtjPeT\nSqqz2uDm7/KZOW8l763aQc/2zXjxZyM5uXdiLVcsIrUp5AB3zm03s81m1tc5lw2Mo7ydIhEQ7GqD\nhcWlPPXRt/x58bc0bGDcdk4/rhndg8ZxukdLpL4LdxbKDcBLFTNQ1gFXh1+SQHCrDb6/agcz561k\n03f5nHd8Z357Xn86t1K7RCRWhBXgzrmvgGP6MhI+fzfTbNqdz8x5mbyftZNeapeIxCzdiRmlqrqZ\npnmTOM549CPiGhi3n9OPq9UuEYlZCvAwhbu7e1V83UxjwP5DJVwwpAt3nNufTq3iwz6OiHiXAjwM\n4e7u7s/h779vwSp27D8EQMeW8Txy6RBO6qV2iYgowMMS6u7uwSgoKmVd7gH2FBTTrHFDbj6jD1eN\nTqFRQ7VLRKScAjwMNbFBr3OORSt3cPf8leTsKWDCkC7ccV5/OrZUu0REjqYAD0OkN+jdsOsgd83L\nZHF2Ln06NueV/xnFib3ahVumiNRTCvAwRGqD3oKiUv60eC1/+WgdjeMa8Nvz+nPlSWqXiIh/CvAw\nhLtqn3OOdzJ38Lv5K9myt4BJQ7vwm3P700HtEhEJggI8TKGui71+10FmzM1kyepc+nZswatTRzGy\np9olIhI8BXgAkZ7nnV9Uwh8/XMtfl6ynSVwD7jx/AFec2F3tEhGpNgW4H5Gc5+2cY2HGdn43fyVb\n8wqZnJrEbef2o0MLtUtEJDQKcD8iNc97Xe4BZszN5OM1u+jXqQWPX57KCSltI12uiMQYBbgf4c7z\nzi8q4ckP1vL0x+uIj2vIXRcMYMqo7sSpXSIiEaAA9yPUed7OORas2M49b61kW14hFw3rym3n9KN9\niyY1VaqIxCAFuB+hzPNeu/MAd83N5JO1uxjQuSV/+HEqw7v7bpfU1EJYIhIbFOB+VGee98FDJTzx\nwRqe/WQ98Y0aMnPCQH4yMrnKdklNLoQlIrFBAR5AoHnezjneWrGNe+avYvu+Qi4eXt4uSWzuv11S\nkwthiUhsCDvAzawhkA5scc6dH35J3rF2536mz8nkP9/uZkDnlvzxJ0e3S/y1SGpiISwRiS2RGIHf\nBKwCWkbgvTzhwKESnni/vF3StHFD7p44kJ+M7E7DBvb9awK1SCK9EJaIxJ6w5rOZWVfgPODpyJQT\n3ZxzzP16K+MeXszsJeuYPCyJD389hitOTDkqvMF/iwTKL5AmNGp41POhLIQlIrEr3BH4Y8CtQIuq\nXmBmU4GpAMnJyWEeru6s3rGfGXMy+WzdbgYlteTPU4YzLLlNla8P1CIJdyEsEZGQA9zMzgd2OueW\nmdmYql7nnJsNzAZIS0tzoR6vruwvLObx99bw3H820KxJHL+bNIgfj0g+ZsRdWTAtklAXwhIRgfBG\n4KOBCWZ2LhAPtDSzF51zUyJTWt063C65961V5B44xKVp3bj17H60bdY4qO+P1FrhIiJVCTnAnXO3\nA7cDVIzAfx2N4R3KzTLZ2/czfU4GS9d/x/FdWzH7ijSGdmtdreOqRSIiNa1ezAOvKqSre7PM/sJi\nHqtol7SIj+P3Fw7m0hO6BWyXVEUtEhGpSREJcOfcYmBxJN6ruvyFdLA3yzjnePOrLfx+QRa7Dhzi\n8hHJTDurL22CbJeIiNQFz4/A/YV0MDfLrNq2jxlzMvliw3cM6daaZ65M4/iu1WuXiIjUBc8HuL+Q\n9jcTZF9hMY8uWs0Ln22kZXwc908ezI/SutEgxHaJiEht83yA+wtpXzNB4uMacFqf9ox96CN2HzzE\nj0ckM218X1o3VbtERLzF8zsL+LujcVJqEvdNHkxS6wQM6NCiCZ1bJ/DyF5vo2iaBudefzL0XDlZ4\ni4gneX4EHmi63qTUJE7v16GiXbKBkjLHrIsGc8lwtUtExNs8H+BQ9XS9sjLH68tzmLUwi90Hi5gy\nsju3nNXnqBG3NlUQEa+qFwHuS8aWPKbPyWD5pr2kJrfmuatHMCip1VGv0aYKIuJl9S7A8/KLeXhR\nNi9+vpE2TRvzwMXHc/Gwrj7bJdpUQUS8rN4EeFmZ47XlOcx6O4s9+UX8dFR3/vfMvrRq2qjK79Gm\nCiLiZfUiwI9slwzv3oYXJo5gYJdWAb9PmyqIiJd5OsDz8ot56N1sXly6kXbNGvPQJUOYnJoU9OwS\nrRgoIl7myQAvK3P8a9lmZi3MZm9+EVeemMKvzuxDq4Sq2yW+aMVAEfEyzwX4ipw87pyTwVeb93JC\nShtmThjJgC6hb8epFQNFxKs8E+B784t48J1sXv5iE+2aNeGRHw3hwtQkzHQzjojEJk8EeGFxKWc/\n9jG5Bw5x9Uk9uPnM3rSMr167RESkvvFEgMc3asgtZ/VhcNdW9OsUertERKQ+8USAA1yS1q2uSxAR\niSohr0ZoZt3M7EMzW2lmmWZ2UyQLExER/8IZgZcAtzjnlptZC2CZmS1yzq2MUG0iIuJHyCNw59w2\n59zyiq/3A6sAzccTEaklEZcKvTwAAAVcSURBVNnQwcxSgFRgqY/npppZupml5+bmRuJwIiJCBALc\nzJoDrwM3O+f2VX7eOTfbOZfmnEtr3759uIcTEZEKYQW4mTWiPLxfcs69EZmSREQkGOHMQjHgGWCV\nc+6RyJUkIiLBCGcWymjgp8AKM/uq4rHfOOcWhF/W0bTtmYjIsUIOcOfcJ0CNL0Sibc9ERHyLyCyU\nmuRv2zMRkVgW9QGubc9ERHyL+gCvanszbXsmIrEu6gN82vi+JDRqeNRj2vZMRMQDqxFq2zMREd+i\nPsBB256JiPgS9S0UERHxTQEuIuJRCnAREY9SgIuIeJQCXETEo8w5V3sHM8sFNobxFonArgiVE2mq\nrfqitS5QbaFSbaEJVFt359wxGyrUaoCHy8zSnXNpdV2HL6qt+qK1LlBtoVJtoQm1NrVQREQ8SgEu\nIuJRXgvw2XVdgB+qrfqitS5QbaFSbaEJqTZP9cBFROQHXhuBi4hIBQW4iIhHRV2Am9mzZrbTzDKq\neN7M7AkzW2tm35jZsCiqbYyZ5ZnZVxV/ptdSXd3M7EMzW2lmmWZ2k4/X1Ml5C7K2ujpv8Wb2hZl9\nXVHbTB+vaWJmr1act6VmlhJFtV1lZrlHnLdra6O2imM3NLMvzWy+j+fq5JwFWVtdnrMNZrai4rjp\nPp6v/s+ocy6q/gCnAsOAjCqePxd4m/INlUcBS6OotjHA/Do4Z52BYRVftwBWAwOi4bwFWVtdnTcD\nmld83QhYCoyq9JrrgKcqvr4MeDWKarsK+ENtn7eKY/8v8LKv/251dc6CrK0uz9kGINHP89X+GY26\nEbhzbgnwnZ+XTARecOU+B1qbWecoqa1OOOe2OeeWV3y9H1gFVF5AvU7OW5C11YmKc3Gg4q+NKv5U\nvqo/EXi+4uvXgHFmZlFSW50ws67AecDTVbykTs5ZkLVFs2r/jEZdgAchCdh8xN9ziJJAqHBixa+9\nb5vZwNo+eMWvq6mUj9iOVOfnzU9tUEfnreLX7a+AncAi51yV5805VwLkAe2ipDaAiyp+3X7NzLrV\nRl3AY8CtQFkVz9fZOSNwbVA35wzKP4DfNbNlZjbVx/PV/hn1YoBHs+WUr1kwBHgSeLM2D25mzYHX\ngZudc/tq89iBBKitzs6bc67UOTcU6AqMMLNBtXXsQIKobR6Q4pw7HljED6PeGmNm5wM7nXPLavpY\n1RVkbbV+zo5wsnNuGHAOcL2ZnRruG3oxwLcAR35qdq14rM455/Yd/rXXObcAaGRmibVxbDNrRHlA\nvuSce8PHS+rsvAWqrS7P2xE17AU+BM6u9NT3583M4oBWwO5oqM05t9s5d6jir08Dw2uhnNHABDPb\nAPwDGGtmL1Z6TV2ds4C11dE5O3zsLRX/3An8GxhR6SXV/hn1YoDPBa6ouGI7Cshzzm2r66IAzKzT\n4V6fmY2g/PzW+P+4Fcd8BljlnHukipfVyXkLprY6PG/tzax1xdcJwJlAVqWXzQWurPj6YuADV3HF\nqa5rq9QfnUD59YUa5Zy73TnX1TmXQvkFyg+cc1MqvaxOzlkwtdXFOas4bjMza3H4a+AsoPJstmr/\njEbdpsZm9grlsxISzSwHmEH5BRycc08BCyi/WrsWyAeujqLaLgZ+aWYlQAFwWW38j0v5yOOnwIqK\nninAb4DkI2qrq/MWTG11dd46A8+bWUPKPzT+6Zybb2Z3A+nOubmUf/j83czWUn4B+7JaqCvY2m40\nswlASUVtV9VSbceIknMWTG11dc46Av+uGKfEAS875xaa2S8g9J9R3UovIuJRXmyhiIgICnAREc9S\ngIuIeJQCXETEoxTgIiIepQAXEfEoBbiIiEf9fyHYb0HZP+QSAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5-tt-dRCfXSJ",
        "colab_type": "text"
      },
      "source": [
        "Much better! Our parameter values are also very close to the true relationship between $x$ and $y$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IqigNebSZv2m",
        "colab_type": "text"
      },
      "source": [
        "## What if we didn't have a closed form? Gradient Descent!\n",
        "\n",
        "We have a function $L$ (our loss function, MSE) that we want to minimize by changing our parameters $m$ and $b$. The partial derivative $\\frac{\\partial L}{\\partial m}$ represents the direction of steepest ascent with respect to $m$. If we travel in the direction of the $-\\frac{\\partial L}{\\partial m}$, we are travelling in a direction where our loss function gets smaller. By taking small steps and computing the direction of the derivative at each step, we can slowly make progress towards a local minima. In particular, at every step, here is the update rule:\n",
        "\n",
        "$m_{k+1} = m_k - \\alpha\\frac{\\partial L}{\\partial m}$\n",
        "\n",
        "$b_{k+1} = b_k - \\alpha\\frac{\\partial L}{\\partial b}$\n",
        "\n",
        "$\\alpha$ is our **learning rate**. Typically, this is a value < 1 so we don't take steps that are too large. If our steps are too large, we may never reach the minimum (might jump over the minimum, or diverge altogether). A **gradient** is a vector of partial derivatives of our parameters. Concretely, let $\\theta = [m, b]^T$. we can rewrite the above expression concisely as follows.\n",
        "\n",
        "$\\theta_{k+1} = \\theta_k - \\alpha\\nabla_\\theta L = \\begin{bmatrix} m_k \\\\ b_k \\end{bmatrix} - \\alpha \\begin{bmatrix} \\frac{\\partial L}{\\partial m} \\\\ \\frac{\\partial L}{\\partial b} \\end{bmatrix}$\n",
        "\n",
        "Gradient Descent has some drawbacks. It's much slower than a closed form solution. Also, if we have a loss function that isn't **convex**, it could have multiple minima, so our algorithm may converge to a minimum that isn't the global minimum. \n",
        "\n",
        "The image below shows the value of an example loss function $J(\\theta_0, \\theta_1)$ (which could be MSE) plotted against $\\theta_0$ and $\\theta_1$ (which could be $m$ and $b$).\n",
        "\n",
        "\n",
        "![alt text](https://hackernoon.com/hn-images/1*f9a162GhpMbiTVTAua_lLQ.png)\n",
        "\n",
        "For linear regression, we know that the loss function is convex (unlike the picture above), though for machine learning models in general, this is not true.\n",
        "\n",
        "Let's use gradient descent to fit our linear model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QhHwHvvEZkOh",
        "colab_type": "code",
        "outputId": "259c8151-2259-4487-e38b-0b4c2b0f82eb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "m = tf.Variable(1, dtype=tf.float32)\n",
        "b = tf.Variable(0, dtype=tf.float32)\n",
        "m, b"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=1.0>,\n",
              " <tf.Variable 'Variable:0' shape=() dtype=float32, numpy=0.0>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xUnGoq94jXa6",
        "colab_type": "text"
      },
      "source": [
        "We want to iteratively update our parameters in the direction of the negative gradient. TensorFlow supports automatic differentiation, so let's take advantage of that. We use something called a `GradientTape` to track computations in TensorFlow. After tracking them, we are able to differentiate wirth respect to them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-x2aot6ejlua",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "4bd7ac73-696c-4404-dafe-9230a07d5d12"
      },
      "source": [
        "with tf.GradientTape() as tape:\n",
        "  y_pred = x*m + b\n",
        "  loss = mse(y_true=y, y_pred=y_pred)\n",
        "\n",
        "tape.gradient(loss, [m, b])"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<tf.Tensor: id=354, shape=(), dtype=float32, numpy=-44.360188>,\n",
              " <tf.Tensor: id=347, shape=(), dtype=float32, numpy=-12.745608>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N7FieBzWjrF_",
        "colab_type": "text"
      },
      "source": [
        "The result is the value of the partials with respect to the variables. With GradientTape, computatiosn with Variables are automatically tracked. If you want to differentiate with respect to Tensors, you have to use `tape.watch(tensor)`. We won't cover that in this workshop, but you can check that out [here](https://www.tensorflow.org/tutorials/customization/autodiff).\n",
        "\n",
        "Now we can write our training loop. We will take 100 steps with a learning rate of 0.05. Here's how that looks:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OJgZkgyQcFnO",
        "colab_type": "code",
        "outputId": "c49fb6c9-e748-4604-b5b3-62163f2489e3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        }
      },
      "source": [
        "lr = 0.05 # learning rate\n",
        "\n",
        "for step in range(100):\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = x*m + b\n",
        "    loss = mse(y_true=y, y_pred=y_pred)\n",
        "\n",
        "  gradient = tape.gradient(loss, [m, b])\n",
        "  m.assign_sub(lr*gradient[0])\n",
        "  b.assign_sub(lr*gradient[1])\n",
        "\n",
        "  if step % 10 == 0:\n",
        "    print(f'Loss: {loss.numpy()}')"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 49.600460052490234\n",
            "Loss: 0.717918336391449\n",
            "Loss: 0.6513835191726685\n",
            "Loss: 0.5993430018424988\n",
            "Loss: 0.558639645576477\n",
            "Loss: 0.5268036127090454\n",
            "Loss: 0.5019029378890991\n",
            "Loss: 0.482427179813385\n",
            "Loss: 0.46719399094581604\n",
            "Loss: 0.4552793502807617\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YTvRxcHzl9ka",
        "colab_type": "text"
      },
      "source": [
        "Our loss went down, nice! Let's check out the parameter values and the a graph:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pkK_Uo_DdgLI",
        "colab_type": "code",
        "outputId": "6c7d3bb4-ae08-4021-fce4-219ab819c09d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "plt.scatter(x, y_pred)\n",
        "plt.scatter(x, y)\n",
        "\n",
        "print(f'm: {m.numpy()}   b: {b.numpy()}')"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "m: 3.387526035308838   b: -0.6091350317001343\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcEElEQVR4nO3df3Bd9Xnn8fdj/cDyhlgbZEJs2TWl\n1DO7xsEgMmzkzaSoqUkB42GpQ3ZoQxuP0tIxodu1Y7oZwXo6a2+8U4iT6SYeh026EMBLQDihlDA4\nLYNLiWWb2I4J3cwujSXhWqa1061lZEnP/nHvlXWv7z3n6J5zfxzdz2vGI91zj+55OIkef/2c7/f7\nmLsjIiLpM6fWAYiISHmUwEVEUkoJXEQkpZTARURSSglcRCSlmqt5sY6ODl+6dGk1LykiknoHDhw4\n5e4LCo9XNYEvXbqUgYGBal5SRCT1zOzvih1XCUVEJKWUwEVEUkoJXEQkpZTARURSSglcRCSllMBF\nRFJKCVxEJKWUwEVEUkoJXEQkaYd3w8PL4aH2zNfDuytymaquxBQRSYXDu+HlLXBmEOZ3Qk8frFgX\n/We/ex+cH828PnM88xqif0ZEGoGLiEyXS8BnjgN+IQFHHUW/vOVC8s45P5o5njAlcBGR6aIk4KAS\nyZnB4p9b6ngMSuAiItOFJeCwEfr8zuI/X+p4DErgIiLThSXgsBF6Tx+0tOW/39KWOZ6w0ARuZo+a\n2UkzO1pwfIOZ/cTMfmxmX0o8MhGRWghLwGEj9BXr4LYdMH8xYJmvt+1I/AEmRJuF8k3gq8Cf5Q6Y\n2a8AtwMfdvf3zOzyxCMTEamFXKItNQtlfme2fFJg2si9f6Kb7e/tYPjcKAvntrFxYhlrKxBqaAJ3\n91fMbGnB4d8Dtrn7e9lzTiYfmohIjaxYV3rE3NOXP00Q8kbo/YeGeOCZI4yenwBg6PQoDzxzBIC1\nKxclGma5NfBfBv6tmb1uZn9lZjeUOtHMes1swMwGRkZGyryciEidCCmRbH/xranknTN6foLtL76V\neCjlLuRpBj4A3AjcAOw2s190dy880d13AjsBurq6LnpfRCR1Akbow6dHZ3Q8jnIT+CDwTDZh/9DM\nJoEOQENsEUmHGKst+w8Nsf3Ftxg+PcrC9jY2rl42VR5Z2N7GUJFkvbC97aJjcZVbQukHfgXAzH4Z\naAVOJRWUiEhFxVhtmatxD50exblQ4+4/NATAxtXLaGtpyvuZtpYmNq5elvh/RpRphE8ArwHLzGzQ\nzD4LPAr8YnZq4ZPAZ4qVT0RE6lKM5e5hNe61Kxex9Y5rWNTehgGL2tvYesc1iT/AhGizUD5d4q27\nE45FRKQ6Yix3j1LjXrtyUUUSdiHtRigijSdkLne91LjDaCm9iDSegNWW9VTjDqMELiKNJ2Audz3V\nuMOohCIijanEXO56qnGH0QhcRGSaUrXsWtS4wyiBi0hD6j80RPe2vVy5+Xm6t+2tyxp3GJVQRKTh\nRNlwqtQslHqiBC4iDSfoQWWuvl2PCbuQSigi0nCqueFUJWkELiLpFLIZVVoW48ShEbiIpE/IZlRp\nWowThxK4iKRPyGZUaVqME4dKKCKSPiGbUaVpMU4cGoGLSPpMayBc7HiaFuPEoQQuIqmz/6oNjHpr\n3rFRb2X/VRuA2VPjDqMELiKpc/+xq/nC+fUMTnYw6cbgZAdfOL+e+49dDcyeGncY1cBFJHWGT48y\nxCr2jK3KO26zrMYdJkpLtUfN7GS2fVrhe39oZm5mHZUJT0TkYpFq3Id3w8PL4aH2zNcI/S7TJkoJ\n5ZvAzYUHzWwx8GvAzxKOSUQEiLHhVIymxWkSpSfmK2a2tMhbDwObgOcSjklEJN6GU0HzxIvsAZ5W\nZdXAzex2YMjdf2RmYef2Ar0AS5YsKedyItKAYm04FaNpcZrMeBaKmc0D/gjoi3K+u+909y5371qw\nYMFMLyciDSrWhlMh88Rni3KmEV4FXAn8yMzeBjqBg2Z2RZKBicjsV6rGDTEX4wQ0LZ5NZlxCcfcj\nwOW519kk3uXupxKMS0RmubAa98bVy3j12T/lfp5koZ1i2Dt4hLtYtfre8A/P1bkDdiucDUITuJk9\nAXwc6DCzQeBBd/9GpQMTkdkttMbdtI9bW3bRPHEOgE47xbamXTQ3fRiIkIhLNC2eTaLMQvl0yPtL\nE4tGROpPyL7b5Qqtcb+8ZSp55zRPnJt1M0ni0EpMESktN586NyUvN58aIiXRWE0VGmQmSRzaC0VE\nSgvZdztIlKYKd7b+Na+23sf/ueTf82rrfdzZ+tcXFuM0yEySOJTARaS0GKPg0KYKTfvY1rKLzjmn\nmGPQOecU21p2sbZpX+bkBplJEocSuIiUFmMUPHx6lDVzXs0bYa+Z82q0GjdkSjS37YD5iwHLfL1t\nh+rf06gGLiKl9fTl18Ah8ij4M+/7IZvO72KejQHZWSQtu/hASytwS7TRfQPMJIlDI3ARKS3CKLjU\nYpxNLU9NJe+ceTbGppanMi9U445NI3ARCRYwCu4/NMSrz/4pT/EkCy85xfDZDh559i7gXtaOnij6\nM/Nyx2OM7iVDI3CRRhdj3+w3nt/JFtuZ9yByi+3kjed3ho+wVeOOTSNwkUYWc573+rHHmDfn4jLJ\n+rHHYM3W8BG2atyxKIGLNLII+2YHLsaZ827Rj104592G2Y+klpTARRpZyEyQsA2nzrVdwbzRdy76\n8XNtVzAPNMKuMNXARRpZSJ06bDHOvE9uYbxpbt77401zmffJ8JWaEp8SuEgjC1ntGLrh1Ip1NN/+\nlbwHkc23f0Wj7ipRCUWkka1Yx/63/5HFB7dzuZ/ipHVw/JqN3JBNwKEbTmU/Qwm7NpTARRpY/6Eh\nHtj/C4ye//LUsbb9TWxdPDTVVGF6DRwKur9LTamEItLAQjecWrmIrXdcw6L2NgxY1N7G1juuKd1M\nWKpKI3CRBhalcXBg93epqdARuJk9amYnzezotGPbzewnZnbYzJ41s/bKhikilRCrcbDUXJQSyjeB\nmwuOvQQsd/cVwN8CDyQcl4gkqNSGUxtXL6OtpSnvXNW40yNKT8xXzGxpwbHvT3v5N8CdyYYlIkkJ\nW4wDlFxpKfUtiRr47wBPlXrTzHqBXoAlS5YkcDkRmYnQ7u+qcadWrFkoZvafgHHg8VLnuPtOd+9y\n964FCxbEuZyIlCHKg0pJp7JH4GZ2D3Ar0OPunlhEIjJjsbq/S2qVNQI3s5uBTcAadz+bbEgiMhNR\nur/rQeXsFGUa4RPAa8AyMxs0s88CXwUuBV4yszfM7GsVjlNk9iuzsYIW4zSuKLNQPl3k8DcqEItI\n44rRWEGLcRqXltKL1IOgxgrA/j1f58RDv8Tkg/M58dAvsX/P16dO02KcxqUELlIPAhor7N/zdZYf\n+CJXMMIcgysYYfmBL04l8Ug17hh9L6V+aS8UkXowvzNTNilyfPHB7bRZft/JNhtj8cHtsOZzrF25\niEXHv5fdEnaEk7aA49dt5IaV2QXUMfteSv3SCFykHgQ0VrjcR4r+yOV+KvPN4d3ccOTBvBH6DUce\nvDDKDinPSHopgYvUgxXr4LYdeZ1tuG0HrFjHSSu+AO6kdWS+CUvQIX0vJb2UwEXqRP9EN93v7eDK\nc4/T/d4O+ie6ATh+3UZGvTXv3FFv5fh1GzMvwhJ0SN9LSS8lcJE6ELQY54Y1n+Po9X/MCRYw6cYJ\nFnD0+j/mhjWfy/xwWIIO6Xsp6WXVXAXf1dXlAwMDVbueSF05vDtT1jgzmEmuPX1TDxG7t+0tutx9\nUXsb+zbfFP650x9SQiZBZ0swYdeW+mdmB9y9q/C4ZqGIVEPITJBYG07lEnFQglbj4VlJCVykGoIe\nNK5YF3/DKSXohqQauEgVeIkHjbnj2nBKyqEELlIFf09H4HFtOCXlUAlFpAq2jv0GW1t2MW/aisqz\n3srW87/Bl7OvteGUzJQSuMhMBMzmCGqqMPD+T7D557CpeTcL7V2G/TK+NL6OA+//RC3/ayTllMBF\nogqYSdI/0R3YOHjj6mU88MwYe8ZWTX1cW0sTW1XjlhhUA5fGU+7OfAEzSdRUQWohdARuZo+S6X15\n0t2XZ499gEwn+qXA28A6d//HyoUpkpA4O/MFLFkfPqemClJ9UUbg3wRuLji2GXjZ3a8GXs6+Fql/\ncXbmC1iyrqYKUguhCdzdXwH+oeDw7cC3st9/C1ibcFwilRFnZ76ePsab5uYdGm+aCz19msctNVFu\nDfyD7v5O9vsTwAdLnWhmvWY2YGYDIyPF9zUWqZoYO/P1T3Sz+fx6Bic7mHRjcLKDzefX0z/RrRq3\n1ETsWSju7mZWckcsd98J7ITMZlZxrycSS09f8Y2fIuzMt/3Ftxga+yhP89G846+9+Famvt20j7WX\nbIG5g3BJJzT1AVreLpVT7gj8783sQwDZryeTC0mkggIaJ4QJ3HAq93D0zHHALzwcVe9JqaByR+B7\ngM8A27Jfn0ssIpFKC9j4KWgxTuCGUy9/IXCzKpFKCB2Bm9kTwGvAMjMbNLPPkkncnzCz/w38ava1\nSKoFNVWAkA2n1LZMaiB0BO7uny7xVk/CsYjUVNBinOlzuIuO0P+ydFd5kUrRUnqRrChNFUouxonx\ncFSkXErgkj4x24OVqnPHaqoQpSuOSMKUwCVd4iyF50Kdu9imU5kNp47klVFmtBhHXXGkyrSZlaRL\nnKXwhNe5tRhH0kQjcEmXmLM9wurc2nBK0kQjcEmXGEvhoXQ9W5tOSRopgUu69PRlZndMVzDbo//Q\nEN3b9nLl5ufp3rZ3ah43RGgeXO5e4SI1oBKKpEvIbI+gh5Shc7ljPiAVqTZzr97+Ul1dXT4wMFC1\n60nj6d62t+hUwEXtbezbfFPwDz+8vMRinMXwB0cTilBk5szsgLt3FR5XCUVmlSiLcUrScnhJGZVQ\nJHXK3nAqzHwth5d00QhcUiXWhlNhIjwgFaknSuCSKhXt/h5jr3CRWlAJRVIl1oZTUWg5vKSIErjU\npYpsOCUyy6iEInUnqM6t7u8iFyiBS93RhlMi0cQqoZjZHwDrAQeOAL/t7ueSCEwalzacEomm7BG4\nmS0C7gO63H050ATclVRg0rhibzil/UykQcQtoTQDbWbWDMwDhuOHJI0g1oZTQXL7mZw5DviF/UyU\nxGUWKjuBu/sQ8N+AnwHvAGfc/fuF55lZr5kNmNnAyMhI+ZHKrBG2GCdWnTtmwweRNCl7Mysz+5fA\nd4BPAaeB/wU87e6PlfoZbWYlEHPDqTAPtZN5JFPI4KHT8T5bpEYqsZnVrwL/191H3P088Azw0Rif\nJw0i1oZTYWI2fBBJkzgJ/GfAjWY2z8wM6AHeTCYsSbugGndFu+JoPxNpIHFq4K8DTwMHyUwhnAPs\nTCguSbGKbjgVRvuZSANRQwdJXJQad9CWsCKSr1QNXHuhSOIqvuGUiABaSi8VkEiNW4txREIpgUvZ\nSj2ojF3j1mIckUhUQpGyhHV/hxKd36MIWoyjh5EiU5TApSxhOwbGqnGrubBIJCqhNLIYdWYtxhGp\nPSXwRhWhzlzRxThBf3loMY5IJErgjSpk06eKLsYJ+8tDi3FEIlENvFGF1Jmj1Lhz5834QWWUh5Rq\nLiwSSgk87Q7vziS+M4OZGnFPX7TEN78zOwIucpwEFuMExaWHlCKJUAklzeLMl+7pY7xpbt6h8aa5\nU3XmWDXusLj0kFIkEUrgaRaleUGJh4X9E91sPr+ewckOJt0YnOxg8/n19E90AxFr3KUeRIbFpYeU\nIolQCSXNwkoRuZFwLpnmRsLA9hc7GBr7KE8XbOH+WtQad8Bnh8aVK6WUU/oRkSlK4GkWUscOGgkP\nn/6vRT8yr8bdtI+1l2yBuYNwSSc09QHTkm+pUXZYXKCHlCIJUAklzcJKEQEj4dAad1gdO2iUrRKJ\nSFUogadZ2HzpgIeFoTXusDp20INIzeMWqQqVUNIuoBSx/6oNLD/wRdpsbOrYqLdy9KoN4TXusDp2\nT19+DRzyR9kqkYhUXKwEbmbtwC5gOZlW4L/j7q8lEZjEd/+xq7n+/Ho2Ne9mob3LsF/Gl8bXceDY\n1exbEzKPO6yOrQeRIjUXdwT+ZeAv3P1OM2sF5iUQkyRk+PQoQ6xiz9iqvOMWZcOpsBE2aJQtUmNl\nJ3Azmw98DLgHwN3HgLGgn5HkBfWWXNjeVrQ3ZaTFOBphi9S9OCPwK4ER4H+Y2YeBA8Dn3f2fp59k\nZr1AL8CSJUtiXE4KhTVV2Lh6Wd77MMPOOBphi9S1OLNQmoHrgP/u7iuBfwY2F57k7jvdvcvduxYs\nWBDjclIoaMMpyCTxrXdcw6L2NoxMV/itd1yjZsIis0ScEfggMOjur2dfP02RBC4xBWwKpe7vIo2t\n7BG4u58AjptZ7t/jPcCxRKKSjMO7GX9uQ95imvHnNkwtpkmk+7uIpFbchTwbgMfN7DBwLfBf4ock\nOWdf6KN54lzeseaJc5x9ITMTJHb3dxFJtVjTCN39DaAroVikwNzRE4HHY3d/F5FU00rMOjY8eRmd\nc04VP579XjVukcalvVCSEKO7e5BdrXdz1lvzjp31Vna13p3I54tIumkEHlfQvtgR51CXWoxz7S29\n9D07zv3+5NRS+Ee4i1W39FboP0ZE0kQJPK4oDXoDhC3GgXv51Is9qnGLyEWUwOOK2aA3Svd3JWwR\nKUY18LhiNuiNshhHRKQYJfC4InSf6T80RPe2vVy5+Xm6t+2l/9DQ1HtajCMi5VICjyuk+0yuxj10\nehTnQo07l8S1GEdEyqUaeFQBe5IE7doXpcadO08PKkVkJpTAo4gxVVAbTolIpaiEEkVIg1/VuEWk\nFpTAowiYKqgat4jUihJ4FAFTBdVUQURqRTXwKAIa/A5/O4Ead9ADUhGREjQCjyJgqmDsGnfuAem0\npg18977ENsQSkdlLCTyi/oluut/bwZXnHqf7vR30T3QDCdS4Qx6QioiUEruEYmZNwAAw5O63xg+p\n/oRvOBUyjzuoRBJzLxURaVxJ1MA/D7wJvD+Bz6pLsTacCptDPr8zWz4pEHEvFRFpXLFKKGbWCdwC\n7EomnPoUa8OpsBJJhL1URESKiTsCfwTYBFxa6gQz6wV6AZYsWRLzcpVTqqkCZB5IDhVJ1pEeVIaV\nSHKlFM1CEZEZKjuBm9mtwEl3P2BmHy91nrvvBHYCdHV1ebnXq6SwGvfG1cvy3ocZPKiMUiIJ2EtF\nRKSUOCWUbmCNmb0NPAncZGaPJRJVlVV0MY5KJCJSIWWPwN39AeABgOwI/D+6e/122w2YCVLRDadU\nIhGRCmmMlZiHdzP+3AaaJ85lXp85nnkNU4txyq5xR6ESiYhUQCILedz9L+tiDvjh3fDwcnioPfM1\nu5rx7At9F5J3VvPEOc6+kCljaMMpEUmj2TMCD5hvPXf0RNEfyR1XUwURSaPZk8AD5lsPT15G55xT\nF/3I8ORl5OaCqKmCiKTN7NkLJWC+9a7WuznrrXmHz3oru1rr95mriEiY2ZPAA/bsvvaWXvq8l8HJ\nDibdGJzsoM97ufaW3urGKCKSoFlTQtl/1QaWH/gibTY2dWzUWzl61YZsaeRePvVij2rcIjJrzJoE\nfv+xq7n+/Ho2Ne9mob3LsF/Gl8bXceDY1exbo6YKIjL7zJoEPnx6lCFWsWdsVd5xi7LhVIyu8yIi\ntZKqGnjFur+rqYKIpFBqEnhFu7+rqYKIpFBqEnhFN5wKmMEiIlKvUlMDr+iGUwFd50VE6lVqRuCx\nu78HCeg6LyJSr1IzAo/VVCEK7RgoIimTmgSuDadERPKlJoGDNpwSEZkuNTVwERHJpwQuIpJSZSdw\nM1tsZj8ws2Nm9mMz+3ySgYmISLA4NfBx4A/d/aCZXQocMLOX3P1YQrGJiEiAskfg7v6Oux/Mfv9P\nwJuAnjCKiFRJIjVwM1sKrAReL/Jer5kNmNnAyMhIEpcTERESSOBm9j7gO8D97v7zwvfdfae7d7l7\n14IFC+JeTkREsmIlcDNrIZO8H3f3Z5IJSUREoogzC8WAbwBvuvufJBdSgMO74eHl8FB75uvh3VW5\nrIhIPYozAu8GfhO4yczeyP759YTiuliua86Z44Bf6JqjJC4iDarsaYTu/ipgCcYSLKhrjjahEpEG\nlJ6VmOqaIyKSJz0JXF1zRETypCeB9/RluuRMp645ItLA0pPA1TVHRCRPqvYDV9ccEZEL0jMCFxGR\nPErgIiIppQQuIpJSSuAiIimlBC4iklJK4CIiKaUELiKSUkrgIiIpZe5evYuZjQB/l8BHdQCnEvic\npNVrXFC/sSmumanXuKB+Y5sNcf2Cu1/U0qyqCTwpZjbg7l21jqNQvcYF9Rub4pqZeo0L6je22RyX\nSigiIimlBC4iklJpTeA7ax1ACfUaF9RvbIprZuo1Lqjf2GZtXKmsgYuISHpH4CIiDU8JXEQkpeo2\ngZvZo2Z20syOlnjfzGyHmf3UzA6b2XV1FNvHzeyMmb2R/VPxvm9mttjMfmBmx8zsx2b2+SLn1OSe\nRYytFvdsrpn90Mx+lI3rPxc55xIzeyp7z143s6V1Etc9ZjYy7X6tr3Rc067dZGaHzOx7Rd6r+v2K\nGFct79fbZnYke92BIu+X/3vp7nX5B/gYcB1wtMT7vw68ABhwI/B6HcX2ceB7Vb5fHwKuy35/KfC3\nwL+qh3sWMbZa3DMD3pf9vgV4Hbix4Jx7ga9lv78LeKpO4roH+Go179e0a/8H4NvF/veqxf2KGFct\n79fbQEfA+2X/XtbtCNzdXwH+IeCU24E/84y/AdrN7EN1ElvVufs77n4w+/0/AW8CiwpOq8k9ixhb\n1WXvw//LvmzJ/il8qn878K3s908DPWZmdRBXTZhZJ3ALsKvEKVW/XxHjqmdl/17WbQKPYBFwfNrr\nQeogKUzzb7L/BH7BzP51NS+c/WfrSjIjt+lqfs8CYoMa3LPsP7vfAE4CL7l7yXvm7uPAGeCyOogL\n4N9l/8n9tJktrnRMWY8Am4DJEu/X5H5FiAtqc78g85fv983sgJn1Fnm/7N/LNCfwenaQzN4FHwa+\nAvRX68Jm9j7gO8D97v7zal03ipDYanLP3H3C3a8FOoGPmNnyalw3TIS4vgssdfcVwEtcGPVWjJnd\nCpx09wOVvtZMRIyr6vdrmlXufh3wSeD3zexjSX1wmhP4EDD9b9HO7LGac/ef5/4J7O5/DrSYWUel\nr2tmLWQS5OPu/kyRU2p2z8Jiq9U9m3b908APgJsL3pq6Z2bWDMwH3q11XO7+rru/l325C7i+CuF0\nA2vM7G3gSeAmM3us4Jxa3K/QuGp0v3LXHsp+PQk8C3yk4JSyfy/TnMD3AL+VfYJ7I3DG3d+pdVAA\nZnZFru5nZh8hc58r+n/i7PW+Abzp7n9S4rSa3LMosdXoni0ws/bs923AJ4CfFJy2B/hM9vs7gb2e\nffJUy7gKaqRryDxXqCh3f8DdO919KZkHlHvd/e6C06p+v6LEVYv7lb3uvzCzS3PfA78GFM5eK/v3\nsjnRaBNkZk+QmZnQYWaDwINkHubg7l8D/pzM09ufAmeB366j2O4Efs/MxoFR4K5K/5+YzCjkN4Ej\n2dopwB8BS6bFVat7FiW2WtyzDwHfMrMmMn9h7Hb375nZFmDA3feQ+Yvnf5rZT8k8uL6rwjFFjes+\nM1sDjGfjuqcKcRVVB/crSly1ul8fBJ7Njk2agW+7+1+Y2e9C/N9LLaUXEUmpNJdQREQamhK4iEhK\nKYGLiKSUEriISEopgYuIpJQSuIhISimBi4ik1P8H+8WW3lS5rNIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1I-h1-k0ocTm",
        "colab_type": "text"
      },
      "source": [
        "Our code is clean and simple for the simple linear regression case, but it does not scale well. Let's use some classes and objects to clean things up a bit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M5xi18xto2Qp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Linear(object):\n",
        "  def __init__(self):\n",
        "    self.m = tf.Variable(1, dtype=tf.float32)\n",
        "    self.b = tf.Variable(0, dtype=tf.float32)\n",
        "    self.trainable_variables = [self.m, self.b]\n",
        "  \n",
        "  def __call__(self, x): # assume x is a row vector\n",
        "    return self.m * x + self.b"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jypQs0aXo_YB",
        "colab_type": "text"
      },
      "source": [
        "The class above is a callable. When you call it with some input, it'll apply the linear function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uscRSRa5o9YL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "d2e85cb4-89a4-40c3-a1fa-a0553759fa16"
      },
      "source": [
        "lr = 0.05 # learning rate\n",
        "\n",
        "linear_model = Linear()\n",
        "\n",
        "for step in range(100):\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = linear_model(x)\n",
        "    loss = mse(y_true=y, y_pred=y_pred)\n",
        "\n",
        "  gradient = tape.gradient(loss, linear_model.trainable_variables)\n",
        "  \n",
        "  for partial, variable in zip(gradient, linear_model.trainable_variables):\n",
        "    variable.assign_sub(lr*partial)\n",
        "  \n",
        "  if step % 10 == 0:\n",
        "    print(f'Loss: {loss.numpy()}')"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 49.600460052490234\n",
            "Loss: 0.717918336391449\n",
            "Loss: 0.6513835191726685\n",
            "Loss: 0.5993430018424988\n",
            "Loss: 0.558639645576477\n",
            "Loss: 0.5268036127090454\n",
            "Loss: 0.5019029378890991\n",
            "Loss: 0.482427179813385\n",
            "Loss: 0.46719399094581604\n",
            "Loss: 0.4552793502807617\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pF73sgPKpgWl",
        "colab_type": "text"
      },
      "source": [
        "Let's also encapsulate our gradient update into an object too, to make things even cleaner:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G-5npzcmpfH5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class GradientDescentOptimizer(object):\n",
        "  def __init__(self, lr=0.05):\n",
        "    self.lr = 0.05\n",
        "  \n",
        "  def apply_gradients(self, grads_and_vars):\n",
        "    for grad, var in zip(gradient, linear_model.trainable_variables):\n",
        "      var.assign_sub(self.lr*grad)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jaRG4EtNp8Xp",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "eb153a4b-8d69-4c3d-9e32-d9f6d9340907"
      },
      "source": [
        "linear_model = Linear()\n",
        "optimizer = GradientDescentOptimizer(lr=0.05)\n",
        "\n",
        "for step in range(100):\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = linear_model(x)\n",
        "    loss = mse(y_true=y, y_pred=y_pred)\n",
        "\n",
        "  gradient = tape.gradient(loss, linear_model.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(gradient, linear_model.trainable_variables))\n",
        "  \n",
        "  if step % 10 == 0:\n",
        "    print(f'Loss: {loss.numpy()}')"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 49.600460052490234\n",
            "Loss: 0.717918336391449\n",
            "Loss: 0.6513835191726685\n",
            "Loss: 0.5993430018424988\n",
            "Loss: 0.558639645576477\n",
            "Loss: 0.5268036127090454\n",
            "Loss: 0.5019029378890991\n",
            "Loss: 0.482427179813385\n",
            "Loss: 0.46719399094581604\n",
            "Loss: 0.4552793502807617\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ezfQ6mAqIZC",
        "colab_type": "text"
      },
      "source": [
        "TensorFlow actually gives us exactly this optimizer (with some more bells and whistles, of course): "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3BrRrwj3vjxK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.optimizers import SGD"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zPwzLKEUqMCA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "200de7ba-756c-44af-f732-1a455e50dfaf"
      },
      "source": [
        "linear_model = Linear()\n",
        "optimizer = SGD(lr=0.05)\n",
        "\n",
        "for step in range(100):\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = linear_model(x)\n",
        "    loss = mse(y_true=y, y_pred=y_pred)\n",
        "\n",
        "  gradient = tape.gradient(loss, linear_model.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(gradient, linear_model.trainable_variables))\n",
        "  \n",
        "  if step % 10 == 0:\n",
        "    print(f'Loss: {loss.numpy()}')"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 49.600460052490234\n",
            "Loss: 0.717918336391449\n",
            "Loss: 0.6513835191726685\n",
            "Loss: 0.5993430018424988\n",
            "Loss: 0.558639645576477\n",
            "Loss: 0.5268036127090454\n",
            "Loss: 0.5019029378890991\n",
            "Loss: 0.482427179813385\n",
            "Loss: 0.46719399094581604\n",
            "Loss: 0.4552793502807617\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hQ6iAOixqSSK",
        "colab_type": "text"
      },
      "source": [
        "SGD stands for Stochastic Gradient Descent. It essentially exactly the same thing as our optimizer. We'll talk about it in more detail later."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mi-gVBTie265",
        "colab_type": "text"
      },
      "source": [
        "## What if we have multiple inputs?\n",
        "\n",
        "$y = \\vec x^T \\vec w + b $\n",
        "\n",
        "Expanded:\n",
        "\n",
        "$y =  x_1w_1 + x_2w_2 + .... + x_mw_m + b$\n",
        "\n",
        "<br>\n",
        "\n",
        "## What if we have multiple outputs as well?\n",
        "\n",
        "$\\vec y = x^T \\vec W + \\vec b$\n",
        "\n",
        "Expanded:\n",
        "\n",
        "$y_1 =  x_1W_{1,1} + x_2W_{2,1} + ... + x_kW_{1,m} + b_1$\n",
        "\n",
        "$\\dots$\n",
        "\n",
        "$y_n =  x_1W_{k,1} + x_2W_{k,2} + ... + x_kW_{k,m} + b_k$\n",
        "\n",
        "<br>\n",
        "\n",
        "Let's update our linear class to support multiple inputs and ouputs.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RhSTGJJ2nGJ7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Linear(object):\n",
        "  def __init__(self, input_length, output_length):\n",
        "    self.W = tf.Variable(tf.ones([input_length, output_length]))\n",
        "    self.b = tf.Variable(tf.zeros(output_length))\n",
        "    self.trainable_variables = [self.W, self.b]\n",
        "  \n",
        "  def __call__(self, x): # assume x is a row vector\n",
        "    return x @ self.W + self.b # @ is the matrix multiplication operator"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2L2jakmoBcU",
        "colab_type": "text"
      },
      "source": [
        "TensorFlow gives us almost exactly this in `tf.keras.layers.Dense`. For now, we will use our class for more transparency."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q4TnhNq7mRZV",
        "colab_type": "text"
      },
      "source": [
        "## MNIST\n",
        "\n",
        "MNIST is a dataset of handwritten digits from 1 to 9, originally published by Yann Lecun, Corinna Cortes, Christopher J.C. Burges. Let's check it out:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D9AbH9J6e7Kk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "(x, y), _ = tf.keras.datasets.mnist.load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z-XCZ9gUuS1O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x, y = tf.convert_to_tensor(x), tf.convert_to_tensor(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZOoHhUnFfH0v",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "9ad2bb57-4306-4230-f3d7-8cf3d7e6335e"
      },
      "source": [
        "plt.imshow(x[3])\n",
        "y[3]"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16386, shape=(), dtype=uint8, numpy=1>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAMtklEQVR4nO3da4xcdRnH8d/Pum2lqGlBa1OqKAEN\nklh0rTdEFDVI1MILkRpNNcTVKCpGEwm+gBcaGy8oiUazSKXeMEZufYFCaVRiUGTBCr2oXGylzbaF\n1AtoWrbt44s9kAV2zmznnDNn2uf7STYzc545c56c9Ndznfk7IgTgyPesthsA0B+EHUiCsANJEHYg\nCcIOJPHsfi5stufEXM3r5yKBVPbqv3o89nm6WqWw2z5L0hWSZkn6fkSsKnv/XM3T63xmlUUCKHFH\nrO9Y63k33vYsSd+R9C5JJ0taYfvkXj8PQLOqHLMvk3R/RDwYEY9L+pmk5fW0BaBuVcK+WNJDU15v\nL6Y9he0R22O2xya0r8LiAFTR+Nn4iBiNiOGIGB7SnKYXB6CDKmHfIWnJlNfHFdMADKAqYb9T0om2\nX2p7tqTzJa2tpy0Adev50ltE7Ld9oaSbNXnpbXVEbKqtMwC1qnSdPSJuknRTTb0AaBC3ywJJEHYg\nCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2\nIAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5BEpVFcgSY98LU3lNa3fODb\npfUhz+pYO/0TI6XzPueGP5bWD0eVwm57q6RHJR2QtD8ihutoCkD96tiyvzUiHqnhcwA0iGN2IImq\nYQ9Jt9i+y/a0B0G2R2yP2R6b0L6KiwPQq6q78adFxA7bL5S0zvZfIuK2qW+IiFFJo5L0PC+IissD\n0KNKW/aI2FE87pZ0vaRldTQFoH49h932PNvPfeK5pHdK2lhXYwDqVWU3fqGk620/8Tk/jYhf1dIV\nUtj52TeW1n/z/q+W1ididu8LT3hA2XPYI+JBSa+qsRcADeLSG5AEYQeSIOxAEoQdSIKwA0nwFVe0\n5rElB0vrC55V4dIanoEtO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwXV2NOqx972uY+3ac6/oMrdL\nq9/71ytK67ee1/nHjudt21Q6b/kdAIcntuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATX2VHJ3neX\njwty6VdWd6ydNFR+Hb2bNVeeVVp/0ebbK33+kYYtO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwXV2\nVDL+wb2l9bc+p6w+q3TelVvfXlp/0RVcRz8UXbfstlfb3m1745RpC2yvs31f8Ti/2TYBVDWT3fir\nJT39VqWLJa2PiBMlrS9eAxhgXcMeEbdJ2vO0ycslrSmer5F0Ts19AahZr8fsCyNivHi+U9LCTm+0\nPSJpRJLm6qgeFwegqspn4yMiJEVJfTQihiNieEhzqi4OQI96Dfsu24skqXjcXV9LAJrQa9jXSlpZ\nPF8p6cZ62gHQlK7H7LavkXSGpGNtb5d0qaRVkn5u+wJJ2ySd12STaM+zj1tcWt/05h+U1ifiQMfa\nlonyZf/j8pNK6/N0R/kH4Cm6hj0iVnQonVlzLwAaxO2yQBKEHUiCsANJEHYgCcIOJMFXXJOb9cqX\nl9aHf7qxtF7F+6/7dGn9hGv/0NiyM2LLDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJcJ09uW3vPaa0\n/otj/tTlE8p/DvoDD7ynY+2kVQ+Uztv5y7HoBVt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiC6+xH\nuD0feUNp/fqPf63LJwyVVj/+0FtK6xMrO48CdODhf3RZNurElh1IgrADSRB2IAnCDiRB2IEkCDuQ\nBGEHkuA6+xGg7Lffb//St7vMPbfSsn+//fjS+pKtzf3uPA5N1y277dW2d9veOGXaZbZ32N5Q/J3d\nbJsAqprJbvzVks6aZvo3I2Jp8XdTvW0BqFvXsEfEbZL29KEXAA2qcoLuQtv3FLv58zu9yfaI7THb\nYxPaV2FxAKroNezflXSCpKWSxiV9o9MbI2I0IoYjYnhInb8UAaBZPYU9InZFxIGIOCjpSknL6m0L\nQN16CrvtRVNeniuJ6yvAgOt6nd32NZLOkHSs7e2SLpV0hu2lkkLSVkkfa7BHdPG3S47qWJuIZn99\n/cWryuvR6NJxKLqGPSJWTDP5qgZ6AdAgbpcFkiDsQBKEHUiCsANJEHYgCb7iehg4+JZTS+tfGr6h\nsWW/Y+P5pfWjx7jF4nDBlh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkuA6+2Hgy1ePltZPGer9i6Sf\nHz+9tP78Ff8srTf7BVrUiS07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBdfbDwKmzy/9PrvJz0b//\nwatL6y/85+09fzYGC1t2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiC6+wD4KFfnFJaH/KGxpa96DeP\nlNb5vvqRo+uW3fYS27+2vdn2JtufKaYvsL3O9n3F4/zm2wXQq5nsxu+X9LmIOFnS6yV90vbJki6W\ntD4iTpS0vngNYEB1DXtEjEfE3cXzRyVtkbRY0nJJa4q3rZF0TlNNAqjukI7ZbR8v6VRJd0haGBHj\nRWmnpIUd5hmRNCJJc3VUr30CqGjGZ+NtHy3pWkkXRcR/ptYiIiRN+6uHETEaEcMRMTykOZWaBdC7\nGYXd9pAmg/6TiLiumLzL9qKivkjS7mZaBFCHrrvxti3pKklbIuLyKaW1klZKWlU83thIh0eAbkMu\nf2vpj0vr3b7C+u+DezvWXvvLi0rnfcW2zaV1HDlmcsz+JkkfknSv/eQF30s0GfKf275A0jZJ5zXT\nIoA6dA17RPxOkjuUz6y3HQBN4XZZIAnCDiRB2IEkCDuQBGEHkuArrn2wd8Hs0vppc//b5RNmlVZv\n/t+LO9ZOGrmzdN6DXZaMIwdbdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEk\nCDuQBGEHkiDsQBKEHUiC77P3wfM27Cytf2r720rr31vy2zrbQVJs2YEkCDuQBGEHkiDsQBKEHUiC\nsANJEHYgiZmMz75E0g8lLZQUkkYj4grbl0n6qKSHi7deEhE3NdXo4Wz/37eV1re/vnz+d+s1NXaD\nrGZyU81+SZ+LiLttP1fSXbbXFbVvRsTXm2sPQF1mMj77uKTx4vmjtrdIWtx0YwDqdUjH7LaPl3Sq\npDuKSRfavsf2atvzO8wzYnvM9tiE9lVqFkDvZhx220dLulbSRRHxH0nflXSCpKWa3PJ/Y7r5ImI0\nIoYjYnhIc2poGUAvZhR220OaDPpPIuI6SYqIXRFxICIOSrpS0rLm2gRQVdew27akqyRtiYjLp0xf\nNOVt50raWH97AOoyk7Pxb5L0IUn32t5QTLtE0grbSzV5OW6rpI810iGAWszkbPzvJHmaEtfUgcMI\nd9ABSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeScET0b2H2\nw5Km/q7ysZIe6VsDh2ZQexvUviR661Wdvb0kIl4wXaGvYX/Gwu2xiBhurYESg9rboPYl0Vuv+tUb\nu/FAEoQdSKLtsI+2vPwyg9rboPYl0Vuv+tJbq8fsAPqn7S07gD4h7EASrYTd9lm2/2r7ftsXt9FD\nJ7a32r7X9gbbYy33str2btsbp0xbYHud7fuKx2nH2Gupt8ts7yjW3QbbZ7fU2xLbv7a92fYm258p\npre67kr66st66/sxu+1Zkv4m6R2Stku6U9KKiNjc10Y6sL1V0nBEtH4Dhu3TJT0m6YcRcUox7auS\n9kTEquI/yvkR8YUB6e0ySY+1PYx3MVrRoqnDjEs6R9KH1eK6K+nrPPVhvbWxZV8m6f6IeDAiHpf0\nM0nLW+hj4EXEbZL2PG3ycklriudrNPmPpe869DYQImI8Iu4unj8q6YlhxltddyV99UUbYV8s6aEp\nr7drsMZ7D0m32L7L9kjbzUxjYUSMF893SlrYZjPT6DqMdz89bZjxgVl3vQx/XhUn6J7ptIh4taR3\nSfpksbs6kGLyGGyQrp3OaBjvfplmmPEntbnueh3+vKo2wr5D0pIpr48rpg2EiNhRPO6WdL0Gbyjq\nXU+MoFs87m65nycN0jDe0w0zrgFYd20Of95G2O+UdKLtl9qeLel8SWtb6OMZbM8rTpzI9jxJ79Tg\nDUW9VtLK4vlKSTe22MtTDMow3p2GGVfL66714c8jou9/ks7W5Bn5ByR9sY0eOvT1Mkl/Lv42td2b\npGs0uVs3oclzGxdIOkbSekn3SbpV0oIB6u1Hku6VdI8mg7Wopd5O0+Qu+j2SNhR/Z7e97kr66st6\n43ZZIAlO0AFJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEv8H5d3EV+oCzLMAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CyMeQyilsvfp",
        "colab_type": "text"
      },
      "source": [
        "#### Exercise:\n",
        "\n",
        "1.   What's the shape of each image?\n",
        "2.   How many images do we have in this dataset?\n",
        "3.   How many of each class do we have?\n",
        "4.   What's are the datatypes?\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kl0i9cA_uLnX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "9ee73867-d036-401f-832c-6bb8c5f5e7bb"
      },
      "source": [
        "x.shape, y.shape # 28*28 images, 60 000 images"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([60000, 28, 28]), TensorShape([60000]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fc2Qb-CtuYlw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "6354b15b-8584-41ff-e35f-0dc3bbc62fd7"
      },
      "source": [
        "for i in range(10):\n",
        "  print(f'Number of {i}: {tf.reduce_sum(tf.cast(y == i, tf.int32)).numpy()}')"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of 0: 5923\n",
            "Number of 1: 6742\n",
            "Number of 2: 5958\n",
            "Number of 3: 6131\n",
            "Number of 4: 5842\n",
            "Number of 5: 5421\n",
            "Number of 6: 5918\n",
            "Number of 7: 6265\n",
            "Number of 8: 5851\n",
            "Number of 9: 5949\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eRdqQal3uufN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "0adf7f14-d2d8-475a-f656-36848610cd63"
      },
      "source": [
        "x.dtype, y.dtype"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tf.uint8, tf.uint8)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "95VaMVtYfuFg",
        "colab_type": "text"
      },
      "source": [
        "#### Preprocessing\n",
        "\n",
        "We'll work with just 500 examples, so our code runs faster. We also want to normalize our image values so they're between 0 and 1. This makes makes training our models easier, since the scale of our gradients and parameters are more predictable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kAW2pDSIf4Sz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x, y = x[:500], y[:500]\n",
        "x = tf.cast(x, tf.float32) / 255.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0o_35LYrvWyq",
        "colab_type": "text"
      },
      "source": [
        "When we did regression, we assumed our response was a continuous variable. Now, our response is one of 9 categories. It doesn't make sense to fit a line to discrete values from 1 to 9, so we will do something called **one hot encoding**.\n",
        "\n",
        "Essentially, instead of 1 number representing the category, we have a vector of size 10. This vector has 0s everywhere, except a 1 indicating which category the example corresponds to. [Here](https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/)'s a great article about why we one hot encode.\n",
        "\n",
        "Intuitively, this vector represents a probability distribution. In our training data, if our example is a 1, there is a 100% chance it is a 1. So, vec[1] = 100%, while the rest = 0. We will create a model that will predict probabilities for each digit. Our model's prediction is whichever digit it assigned the highest probability."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5VM2T38iwO0_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "outputId": "4f0d01e6-d5f3-4147-b66e-42f1366b7be5"
      },
      "source": [
        "y = tf.one_hot(y, depth=10)\n",
        "y"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16451, shape=(500, 10), dtype=float32, numpy=\n",
              "array([[0., 0., 0., ..., 0., 0., 0.],\n",
              "       [1., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       ...,\n",
              "       [0., 0., 0., ..., 1., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 1., 0.]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meFT8Mtewf8T",
        "colab_type": "text"
      },
      "source": [
        "We will also reshape our explanatory variable from a 28x28 image to a 28*28 long vector, so we can continue to use the techniques we have developed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZDaixNqFyYzo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "outputId": "714469a8-ab4a-4bf1-f7f0-81ff90dc8819"
      },
      "source": [
        "x = tf.reshape(x, (-1, 28*28))\n",
        "x"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16453, shape=(500, 784), dtype=float32, numpy=\n",
              "array([[0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       ...,\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zb2qefkAwrc3",
        "colab_type": "text"
      },
      "source": [
        "Now our dataset contains 60,000 vectors of length 784.\n",
        "\n",
        "For easy reuse, we'll define a function that does all this for us, in case we need it again:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xfNo5p-Q7Fgd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def preprocess(x, y):\n",
        "  # Converts numpy arrays to preprocessing tensors\n",
        "  x, y = tf.convert_to_tensor(x), tf.convert_to_tensor(y)\n",
        "  x = tf.reshape(x, (-1, 28*28))\n",
        "  x = tf.cast(x, tf.float32) / 255.0\n",
        "  y = tf.one_hot(y, depth=10)\n",
        "  return x, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7guKhz2UxXXc",
        "colab_type": "text"
      },
      "source": [
        "Let's jump right into it with our linear model!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "03QQTMTfxmIY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "outputId": "b566c5c7-f6cc-47f7-eb0b-0cf7b1b6b225"
      },
      "source": [
        "x"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16453, shape=(500, 784), dtype=float32, numpy=\n",
              "array([[0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       ...,\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.],\n",
              "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jxa9xySEx4u8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "linear_model = Linear(input_length=784, output_length=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sD_oQ8Nuz54C",
        "colab_type": "text"
      },
      "source": [
        "Broadcasting is what keeps our code so concise. Even though there are 500 training examples, broadcasting will do the matrix multplication on each row of the input matrix (i.e. each example). The output has one output vector per row:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XhYvU8O4xd7y",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 263
        },
        "outputId": "562914e8-7012-44b5-86d0-0908062f706e"
      },
      "source": [
        "linear_model(x)"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16475, shape=(500, 10), dtype=float32, numpy=\n",
              "array([[107.94119 , 107.94119 , 107.94119 , ..., 107.94119 , 107.94119 ,\n",
              "        107.94119 ],\n",
              "       [121.94121 , 121.94121 , 121.94121 , ..., 121.94121 , 121.94121 ,\n",
              "        121.94121 ],\n",
              "       [ 76.247055,  76.247055,  76.247055, ...,  76.247055,  76.247055,\n",
              "         76.247055],\n",
              "       ...,\n",
              "       [ 81.074524,  81.074524,  81.074524, ...,  81.074524,  81.074524,\n",
              "         81.074524],\n",
              "       [ 71.69804 ,  71.69804 ,  71.69804 , ...,  71.69804 ,  71.69804 ,\n",
              "         71.69804 ],\n",
              "       [108.50588 , 108.50588 , 108.50588 , ..., 108.50588 , 108.50588 ,\n",
              "        108.50588 ]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YBeFzQMDxvD8",
        "colab_type": "text"
      },
      "source": [
        "Great...but we wanted our outputs to represent a probability distribution. We need to normalize the output of model so the vector represents a probability distribution.\n",
        "\n",
        "We are doing what's called **multiclass logistic regression**. The current outputs of our linear model are called **logits**. To get the probabilities, we normalize our output using a function called a **softmax**. Let $\\vec y = [y_1, y_2, .... y_k]$ be our output vector:\n",
        "\n",
        "$softmax(\\vec y) = \\left\\{\\frac{\\exp(y_i)}{\\sum_{j=1}^k \\exp(y_j)}\\right\\}_{i=1...k}$\n",
        "\n",
        "This forces the sum of the elements to equal 1 (like a probability). You'll notice we don't just normalize--we exponentiate then normalize. Intuitively, the exponent makes large grow faster than small ones. If the model outputs small differences between the classes, this function will make one entry in the vector large and close to one, and the rest close to 0. This will make it easier for our model's output to match the one_hot encoded ground truth."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e9T_AKjh0zpz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def softmax(x):\n",
        "  return tf.math.exp(x) / tf.reduce_sum(tf.math.exp(x))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fdG2l3K91leM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "outputId": "0ebdac62-0550-47ad-e4f5-03d3d55a0323"
      },
      "source": [
        "softmax(linear_model(x))"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16484, shape=(500, 10), dtype=float32, numpy=\n",
              "array([[nan, nan, nan, ..., nan, nan, nan],\n",
              "       [nan, nan, nan, ..., nan, nan, nan],\n",
              "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
              "       ...,\n",
              "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
              "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
              "       [nan, nan, nan, ..., nan, nan, nan]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t8B6nTQUzPCp",
        "colab_type": "text"
      },
      "source": [
        "Yikes, looks like there's some numerical overflow in our softmax. We'll use the built in one to avoid this (there are some tricks to make it stable):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qTve2Cfx14lu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "outputId": "f3d38292-152d-462b-dd08-e13bd114faf0"
      },
      "source": [
        "y_pred = tf.nn.softmax(linear_model(x))\n",
        "y_pred"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16489, shape=(500, 10), dtype=float32, numpy=\n",
              "array([[0.1, 0.1, 0.1, ..., 0.1, 0.1, 0.1],\n",
              "       [0.1, 0.1, 0.1, ..., 0.1, 0.1, 0.1],\n",
              "       [0.1, 0.1, 0.1, ..., 0.1, 0.1, 0.1],\n",
              "       ...,\n",
              "       [0.1, 0.1, 0.1, ..., 0.1, 0.1, 0.1],\n",
              "       [0.1, 0.1, 0.1, ..., 0.1, 0.1, 0.1],\n",
              "       [0.1, 0.1, 0.1, ..., 0.1, 0.1, 0.1]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LCZe-gYgzc8a",
        "colab_type": "text"
      },
      "source": [
        "Great, let's check that our probability vectors add up to 1, just like we want."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HnQPQPGV2BY0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 562
        },
        "outputId": "cd3fcfcd-f29e-45a5-ff17-51c72e53b888"
      },
      "source": [
        "tf.reduce_sum(y_pred, axis=1)"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16491, shape=(500,), dtype=float32, numpy=\n",
              "array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
              "       1., 1., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1tMJh0RizioN",
        "colab_type": "text"
      },
      "source": [
        "Since we are doing classification, we can compute accuracy! Let's do that:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LQZeJzkr3-wb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def accuracy(y_true, y_pred):\n",
        "  equal_preds = tf.math.argmax(y_pred, axis=1) == tf.math.argmax(y_true, axis=1)\n",
        "  return tf.reduce_mean(tf.cast(equal_preds, tf.float32))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rB7DJ28dzt3X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b83ba2aa-5b33-45c3-874a-a9e04a93941f"
      },
      "source": [
        "accuracy(y, y_pred)"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16499, shape=(), dtype=float32, numpy=0.1>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "riAGx1YizxH-",
        "colab_type": "text"
      },
      "source": [
        "As we'd expect, our untrained model has a 10% accuracy, which is as accurate as one would be when randomly guessing.\n",
        "\n",
        "At this point, you might think we're going to maximize accuracy via gradient descent. Unfortunately, accuracy is not a great measure to do this with. It simple doesn't have enough fine grain information about the discrepancy between our model's predictions and the true y values. Instead, we are going to go back to thinking about our outputs as a probability distribution. We use a loss called **categorical crossentropy**. This essentially measures how far our output vector is from the target vector (our one hot encoded labels). Suppose our prediction/true values are of the form $\\vec y = [y_1, y_2, ..., y_k]$ (i.e. we have k classes).\n",
        "\n",
        "$L(\\vec y_{true}, \\vec y_{pred}) = - \\sum^k_{i=1} y_{i, true}\\log(y_{i, prediction})$\n",
        "\n",
        "Notice that $y_{i, true} = 0$ for all values except for $i = $true class. Also, $\\log(y_{i, prediction}) = 0$ when $y_{i, prediction} = 1$. When it's smaller, the value is negative, so our loss is positive. Essentially, this loss 0 when our model output exactly matches the training data, which is what we want."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-AbjZ39U2Hvs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def categorical_crossentropy(y_true, y_pred):\n",
        "  return -tf.reduce_sum(y_true*tf.math.log(y_pred), axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5NZIHf2aAJLH",
        "colab_type": "text"
      },
      "source": [
        "TensorFlow gives us this function too: `tf.keras.losses.categorical_crossentropy`. We can use ours and tensorflow's interchangeably."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "73cKyKXO96eI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "019f4743-8b22-413d-f9c0-7830106a83d8"
      },
      "source": [
        "categorical_crossentropy(y, y_pred) # this outputs the cross entropy per example"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=16504, shape=(500,), dtype=float32, numpy=\n",
              "array([2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851, 2.3025851,\n",
              "       2.3025851, 2.3025851], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4nVpQcm13Puu",
        "colab_type": "text"
      },
      "source": [
        "Now you have all the tools to write the training loop for logistic regression. Try it before look at the solution below!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZxseGyBB3U7M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "f2173baa-d338-4e33-c904-4ce130989973"
      },
      "source": [
        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)\n",
        "\n",
        "linear_model = Linear(28*28, 10)\n",
        "\n",
        "for i in range(1000):\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = tf.nn.softmax(linear_model(x))\n",
        "    loss_per_example = categorical_crossentropy(y_true=y, y_pred=y_pred)\n",
        "    loss = tf.reduce_mean(loss_per_example)\n",
        "\n",
        "  gradient = tape.gradient(loss, linear_model.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(gradient, linear_model.trainable_variables))\n",
        "  if i % 100 == 0:\n",
        "    print(f'Loss: {loss.numpy()}  Accuracy: {accuracy(y_pred, y)}')"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 2.3025851249694824  Accuracy: 0.10000000149011612\n",
            "Loss: 0.4364985227584839  Accuracy: 0.9039999842643738\n",
            "Loss: 0.29362964630126953  Accuracy: 0.9440000057220459\n",
            "Loss: 0.22477126121520996  Accuracy: 0.9660000205039978\n",
            "Loss: 0.18162311613559723  Accuracy: 0.9819999933242798\n",
            "Loss: 0.1515568494796753  Accuracy: 0.9860000014305115\n",
            "Loss: 0.12939059734344482  Accuracy: 0.9919999837875366\n",
            "Loss: 0.11246717721223831  Accuracy: 0.9980000257492065\n",
            "Loss: 0.099213607609272  Accuracy: 1.0\n",
            "Loss: 0.08861198276281357  Accuracy: 1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PQ6rtSV220VQ",
        "colab_type": "text"
      },
      "source": [
        "Wow, our model's doing really well! Let's take a look at a prediction:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pNYA1D2A22wa",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "9bb09f66-c3fa-4a11-d15a-e9609388050d"
      },
      "source": [
        "y_pred[0].numpy()"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([7.2486359e-03, 8.7252660e-03, 1.6632091e-02, 6.6710129e-02,\n",
              "       2.4541716e-05, 8.8565266e-01, 1.4892247e-03, 4.6367473e-03,\n",
              "       7.0132478e-03, 1.8675643e-03], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BkIeAdpI23gq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5d7fad42-9419-426b-8c8b-e17e7d7c28dd"
      },
      "source": [
        "y[0].numpy()"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MGiAsojj3EY5",
        "colab_type": "text"
      },
      "source": [
        "Nice, for this example, our model's outputting a probability of 75% for the most likely class, which is correct. \n",
        "\n",
        "But our results are misleading. We are evaluating our model on the same data we trained it on. This is the equivalent of teaching a student some concepts, then testing them on it, BUT giving them the test as prep material. We want to evaluate our model on new examples.\n",
        "\n",
        "In machine learning, we call this a **test set**. We do not train our model on this data, we only evaluate our model. Typically, we do a random split of our dataset to get a train/test split. MNIST already has this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1jbrxXh15lbO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H298j1bK5s08",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_test, y_test = preprocess(x_test, y_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qg6i3kNm5zz8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "712faaee-d019-44bb-a3eb-f41e2e03028f"
      },
      "source": [
        "y_pred = tf.nn.softmax(linear_model(x_test))\n",
        "loss = tf.reduce_mean(categorical_crossentropy(y_true=y_test, y_pred=y_pred))\n",
        "print(f'Loss: {loss.numpy()}  Accuracy: {accuracy(y_pred, y_test)}')"
      ],
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 0.5398113131523132  Accuracy: 0.8337000012397766\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ytZYnud4P_q",
        "colab_type": "text"
      },
      "source": [
        "Not great. Our model has **overfit** to our training set, i.e., it has memorized the input data. So, it doesn't generalize well to data it has not seen before. There are many ways to combat this (which are out of the scope of this workshop), but one way is to just give our model more training data. Now, we'll train with the entire train set instead of just 500 examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GhE8YVv07ck3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_train, y_train = preprocess(x_train, y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jzv9S38c70dh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "da33d779-f461-41b6-b107-708ea9fab0e6"
      },
      "source": [
        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)\n",
        "linear_model = Linear(28*28, 10)\n",
        "\n",
        "for i in range(1000):\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = tf.nn.softmax(linear_model(x_train))\n",
        "    loss_per_example = categorical_crossentropy(y_true=y_train, y_pred=y_pred)\n",
        "    loss = tf.reduce_mean(loss_per_example)\n",
        "\n",
        "  gradient = tape.gradient(loss, linear_model.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(gradient, linear_model.trainable_variables))\n",
        "  if i%100 == 0:\n",
        "    print(f'Loss: {loss.numpy()}  Accuracy: {accuracy(y_pred, y_train)}')"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 2.3025853633880615  Accuracy: 0.09871666878461838\n",
            "Loss: 0.6087093949317932  Accuracy: 0.8613333106040955\n",
            "Loss: 0.48937270045280457  Accuracy: 0.877216637134552\n",
            "Loss: 0.44037213921546936  Accuracy: 0.885450005531311\n",
            "Loss: 0.41215795278549194  Accuracy: 0.8906000256538391\n",
            "Loss: 0.39325031638145447  Accuracy: 0.8945833444595337\n",
            "Loss: 0.37943124771118164  Accuracy: 0.8974999785423279\n",
            "Loss: 0.3687483072280884  Accuracy: 0.8997833132743835\n",
            "Loss: 0.3601595163345337  Accuracy: 0.9016333222389221\n",
            "Loss: 0.3530517518520355  Accuracy: 0.9029833078384399\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y8QO5JiI8Oyt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "152bb376-095b-4a9a-e327-44b634281a59"
      },
      "source": [
        "y_pred = tf.nn.softmax(linear_model(x_test))\n",
        "loss_per_example = categorical_crossentropy(y_true=y_test, y_pred=y_pred)\n",
        "loss = tf.reduce_mean(loss_per_example)\n",
        "print(f'Loss: {loss.numpy()}  Accuracy: {accuracy(y_pred, y_test)}')"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 0.33095377683639526  Accuracy: 0.909600019454956\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "P1-zkT216FGN"
      },
      "source": [
        "Now our model doesn't overfit as much, but we can definitely do better. Our logistic regression model has some drawbacks:\n",
        "\n",
        "1.   It still only models linear relationships between the explanatory and response variables. In particular, the **decision boundary** it models is a plane. This boundary is what determines if the model chooses one class over another.\n",
        "2.   It assumes the x variables are all independent, i.e., it doesn't model any iteractions between the explanatory variables. Especially in an image like this, it's clear there are are relationships between the variables.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CNicXXUyoSHQ",
        "colab_type": "text"
      },
      "source": [
        "## Introduction to Artificial Neural Networks\n",
        "\n",
        "From Wikipedia: \n",
        "\n",
        "\"Artificial neural networks (ANN) or connectionist systems are computing systems that are inspired by, but not identical to, biological neural networks that constitute animal brains. Such systems \"learn\" to perform tasks by considering examples, generally without being programmed with task-specific rules. \"\n",
        "\n",
        "How can we make our linear or logistic regression more expressive? What if we added more weight matrices?\n",
        "\n",
        "$\\vec y = ((\\vec xW_1 + b_1)W_2 + b_2)....W_l + \\vec b_l$.\n",
        "\n",
        "Well this doesn't help, since we can define $W = W_1W_2....W_l$ (and similarly collapse $\\vec b$. and it'll just be another linear model. What if we add some nonlinear function $a$ between each linear function?\n",
        "\n",
        "$\\vec y = (a(a(\\vec xW_1 + b_1)W_2 + b_2)....)W_l + \\vec b_l$\n",
        "\n",
        "This is called a **fully connected neural network**! $a$ is called the **activation function**. There are many other types of neural networks, such as Convolutional Neural Networks (which work well for images) and Recurrent Neural Networks (which work well for sequences), but they all boil down to this fundamental structure.\n",
        "\n",
        "Common activation functions include the sigmoid function, tanh, and relu (rectified linear). This one is a common choice, so we will stick to it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pI5M28ifABCT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def relu(x):\n",
        "  return tf.maximum(x, 0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uMnD4rhS_Hav",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://miro.medium.com/max/1283/1*DfMRHwxY1gyyDmrIAd-gjQ.png)\n",
        "\n",
        "Though this function is technically non-linear, even though it doesn't look like it. It's only linear when x > 0. We will not talk about this too much, and just use it.\n",
        "\n",
        "We need to make a slight modification to our Linear class, then we'll make our neural network class:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xvkU4hCeEXTk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Linear(object):\n",
        "  def __init__(self, input_length, output_length):\n",
        "    self.W = tf.Variable(tf.random.normal([input_length, output_length]))\n",
        "    self.b = tf.Variable(tf.zeros(output_length))\n",
        "    self.trainable_variables = [self.W, self.b]\n",
        "  \n",
        "  def __call__(self, x): # assume x is a row vector\n",
        "    return x @ self.W + self.b # @ is the matrix multiplication operator"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rD1PNK88Eeo6",
        "colab_type": "text"
      },
      "source": [
        "We made our initializer random. Unlike linear/logistic regression, the loss is not convex, i.e. there are multiple local minima. We initialize randomly, so we don't bias towards any particular minimum."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QeokwYJK87Ye",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class FullyConnectedNetwork(object):\n",
        "  def __init__(self, input_length, output_length):\n",
        "    self.l1 = Linear(input_length, 50)\n",
        "    self.l2 = Linear(50, output_length)\n",
        "\n",
        "    self.trainable_variables = self.l1.trainable_variables + self.l2.trainable_variables\n",
        "  \n",
        "  def __call__(self, x):\n",
        "    return tf.nn.softmax(self.l2(tf.nn.relu(self.l1(x))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QHYcTJk1_eIn",
        "colab_type": "text"
      },
      "source": [
        "This model has 1 **hidden layer** of size 50. Machine learning with neural networks is called **deep learning**, which comes from the fact that a neural network typically has several hidden layers.\n",
        "\n",
        "Now, write the training loop for our neural network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oXz7_0PJBVpM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "525a0f35-2467-4cf7-e58d-09dd567f82c0"
      },
      "source": [
        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)\n",
        "neural_network = FullyConnectedNetwork(28*28, 10)\n",
        "\n",
        "for i in range(1000):\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = neural_network(x_train)\n",
        "    loss_per_example = tf.keras.losses.categorical_crossentropy(y_true=y_train, y_pred=y_pred)\n",
        "    loss = tf.reduce_mean(loss_per_example)\n",
        "\n",
        "  gradient = tape.gradient(loss, neural_network.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(gradient, neural_network.trainable_variables))\n",
        "  if i % 100 == 0:\n",
        "    print(f'Loss: {loss.numpy()}  Accuracy: {accuracy(y_pred, y_train)}')"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 13.893878936767578  Accuracy: 0.10978333652019501\n",
            "Loss: 7.0014495849609375  Accuracy: 0.5102999806404114\n",
            "Loss: 5.423182964324951  Accuracy: 0.6128833293914795\n",
            "Loss: 4.862724781036377  Accuracy: 0.6526166796684265\n",
            "Loss: 4.611066818237305  Accuracy: 0.6717333197593689\n",
            "Loss: 4.456557273864746  Accuracy: 0.683650016784668\n",
            "Loss: 4.345137596130371  Accuracy: 0.6911166906356812\n",
            "Loss: 4.257636547088623  Accuracy: 0.6973999738693237\n",
            "Loss: 4.185241222381592  Accuracy: 0.701533317565918\n",
            "Loss: 4.123439788818359  Accuracy: 0.7055833339691162\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pHCrU2blQJp4",
        "colab_type": "text"
      },
      "source": [
        "We wrote our neural network from scratch, but of course TensorFlow has APIs to make this much easier for you."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BFEFcug1ITj4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.layers import Dense"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vzj1AXpUQPBE",
        "colab_type": "text"
      },
      "source": [
        "TensorFlow layers function similarly to the ones we created: they have `trainable_variables` you can access, and do computation through a `call` method. They have some other nice properties as well, for example, you don't need to specify an input shape -- this will be computer upon your first use of the model.\n",
        "\n",
        "You can build a model almost exactly like we did by subclassing tf.keras.Model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gAWqULruRoj1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Model(tf.keras.Model):\n",
        "  def __init__(self, input_length, output_length):\n",
        "    super(Model, self).__init__()\n",
        "    self.layer1 = Dense(50, input_shape=(input_length,), activation='relu')\n",
        "    self.layer2 = Dense(output_length, input_shape=(output_length,))\n",
        "  \n",
        "  def call(self, x):\n",
        "    output1 = self.layer1(x)\n",
        "    output2 = self.layer2(output1)\n",
        "    return tf.nn.softmax(output2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_lmImOhwQh4F",
        "colab_type": "text"
      },
      "source": [
        "If you're building a simple model like ours that just stacks computation, you can use the sequential API (notice we don't specify an input shape):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2MuVYerCQjK4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(50),\n",
        "  tf.keras.layers.ReLU(),\n",
        "  tf.keras.layers.Dense(10, activation='softmax')\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Tf7x6-kSPwg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "8aa82d54-8ee4-412d-af70-994dfe7c5f8c"
      },
      "source": [
        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.05)\n",
        "\n",
        "for i in range(1000):\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = model(x_train)\n",
        "    loss_per_example = tf.keras.losses.categorical_crossentropy(y_true=y_train, y_pred=y_pred)\n",
        "    loss = tf.reduce_mean(loss_per_example)\n",
        "\n",
        "  gradient = tape.gradient(loss, model.trainable_variables)\n",
        "  if i % 100 == 0:\n",
        "    print(f'Loss: {loss.numpy()}  Accuracy: {accuracy(y_pred, y_train)}')\n",
        "  optimizer.apply_gradients(zip(gradient, model.trainable_variables))"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss: 0.2992660701274872  Accuracy: 0.9152833223342896\n",
            "Loss: 0.2906898856163025  Accuracy: 0.9175333380699158\n",
            "Loss: 0.2830638587474823  Accuracy: 0.919866681098938\n",
            "Loss: 0.27615323662757874  Accuracy: 0.9215666651725769\n",
            "Loss: 0.26982128620147705  Accuracy: 0.9236000180244446\n",
            "Loss: 0.2639724612236023  Accuracy: 0.9250166416168213\n",
            "Loss: 0.2585376501083374  Accuracy: 0.926716685295105\n",
            "Loss: 0.25347232818603516  Accuracy: 0.9281833171844482\n",
            "Loss: 0.24871543049812317  Accuracy: 0.9294000267982483\n",
            "Loss: 0.24421274662017822  Accuracy: 0.9306333065032959\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHVwY9O1Qw8I",
        "colab_type": "text"
      },
      "source": [
        "I encourage you to read the documentation to learn more about the layers, as there are some tricks that will make these perform slightly better than ours (e.g. initialization)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3oszaY51MK7N",
        "colab_type": "text"
      },
      "source": [
        "### Using tf.data\n",
        "\n",
        "Our model's training well now, but each step is pretty slow. Our update step takes time proportional to the size of our dataset, since our loss/gradients are calculated with respect to the entire training set. Also, when working with larger datasets, this may not even be possible.\n",
        "\n",
        "Instead, we can compute our loss and gradients with respect to a subset of our data called a **minibatch**. This is great, since our update no longer takes time proportional to our dataset size. The drawback is our update is now **stochastic**, since we are updating our parameters on a random subset of our data. Typically, we split our dataset into minibatches of size 32 - 256.\n",
        "\n",
        "Let's do this with TensorFlow dataset API, tf.data. It will be much more performant than raw tensors for minibatch gradient descent."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y-ipP4IXML9y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train, test = tf.keras.datasets.mnist.load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjF19zYcRLR1",
        "colab_type": "text"
      },
      "source": [
        "Given numpy arrays as input, we can simply turn them into datasets with `from_tensor_slices`. There's the `tensorflow_datasets` library that has many datasets built in"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "adQMHxBaNfM2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_dataset = tf.data.Dataset.from_tensor_slices(train)\n",
        "test_dataset = tf.data.Dataset.from_tensor_slices(test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vNQoPQ-uRUa4",
        "colab_type": "text"
      },
      "source": [
        "We need to do some similar preprocessing of the data. Notice we aren't one_hot encoding y. TensorFlow has a loss function called `sparse_categorical_crossentropy`, which is the same as `categorical_crossentropy` except it allows the `y_true` to be an index instead of a one hot tensor (which is faster and more memory efficient). We need to change our accuracy function to reflect that."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i9575fBMNyG-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def preprocess_tfdata(x, y):\n",
        "  x = tf.reshape(x, (1, 28*28))\n",
        "  x = tf.cast(tf.squeeze(x), tf.float32) / 255.0\n",
        "  return x, y\n",
        "\n",
        "def sparse_accuracy(y_true, y_pred):\n",
        "  equal_preds = tf.math.argmax(y_pred, axis=1) == tf.cast(y_true, tf.int64)\n",
        "  return tf.reduce_mean(tf.cast(equal_preds, tf.float32))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dll9cDe2Rin5",
        "colab_type": "text"
      },
      "source": [
        "We apply this preprocessing to our dataset using `map`. Then, we shuffle our data. Since we are batching our data, we don't want to use the same minibatches for every pass of the dataset, because that would introduce bias. Then, we batch our data into batches of size 256. Finally, we prefetch, which means the next batch will be loaded and ready to go before the previous one finishes. There are tons of optimizations you can do to get more performance, which you can read about [here](https://www.tensorflow.org/guide/data_performance)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qtQPzZnhNkFJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_dataset = train_dataset.map(preprocess_tfdata).shuffle(buffer_size=5000)\n",
        "train_dataset = train_dataset.batch(256).prefetch(buffer_size=tf.data.experimental.AUTOTUNE)\n",
        "\n",
        "test_dataset = test_dataset.map(preprocess_tfdata).batch(256)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7pwvXsuTREtv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(50),\n",
        "  tf.keras.layers.ReLU(),\n",
        "  tf.keras.layers.Dense(10, activation='softmax')\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5h-3vCMSGdW",
        "colab_type": "text"
      },
      "source": [
        "Now we can train. This time, our loop looks a bit different. We call every pass of our dataset an **epoch**. Then, each minibatch process is called a **training step**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e4aBkD-zNTRO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "0e999c13-4ad0-4ada-9dd2-8aa0f3ae605d"
      },
      "source": [
        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.05)\n",
        "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)\n",
        "\n",
        "for epoch in range(2):\n",
        "  for step, (x, y) in enumerate(train_dataset):\n",
        "    with tf.GradientTape() as tape:\n",
        "      y_pred = model(x)\n",
        "      loss_per_example = loss_fn(y_true=y, y_pred=y_pred)\n",
        "      loss = tf.reduce_mean(loss_per_example)\n",
        "\n",
        "    gradient = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(zip(gradient, model.trainable_variables))\n",
        "    if step % 100 == 0:\n",
        "      print(f'Epoch {epoch} Step {step} Loss: {loss.numpy()} Accuracy: {sparse_accuracy(y, y_pred)}')"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0 Step 0 Loss: 2.420623302459717 Accuracy: 0.0703125\n",
            "Epoch 0 Step 100 Loss: 0.6510080099105835 Accuracy: 0.8515625\n",
            "Epoch 0 Step 200 Loss: 0.5177682638168335 Accuracy: 0.86328125\n",
            "Epoch 1 Step 0 Loss: 0.415408730506897 Accuracy: 0.87890625\n",
            "Epoch 1 Step 100 Loss: 0.4126032888889313 Accuracy: 0.8984375\n",
            "Epoch 1 Step 200 Loss: 0.365292489528656 Accuracy: 0.89453125\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zg0cLYJrTL5J",
        "colab_type": "text"
      },
      "source": [
        "### Metrics\n",
        "\n",
        "Now our model's training really quickly. But, we're reporting statistics on individual minibatches. Ideally, we can report statistics on the entire training set at the end of each epoch. Or, report statistics on the data we've seen since the previous report. Let's implement this."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i5uL9rO1TU1i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Mean(object):\n",
        "  def __init__(self):\n",
        "    self.vals = []\n",
        "\n",
        "  def add_value(self, x):\n",
        "    self.vals.append(x)\n",
        "  \n",
        "  def result(self):\n",
        "    return sum(self.vals)/len(self.vals)\n",
        "  \n",
        "  def reset_states(self):\n",
        "    self.vals = []"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RsJyDuCkUY_7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(50),\n",
        "  tf.keras.layers.ReLU(),\n",
        "  tf.keras.layers.Dense(10, activation='softmax')\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vSwlA1r0T2jX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "2b57447f-aeab-4949-c7a4-f303a2b14a07"
      },
      "source": [
        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.05)\n",
        "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)\n",
        "loss_mean = Mean()\n",
        "accuracy_mean = Mean()\n",
        "\n",
        "for epoch in range(2):\n",
        "  for step, (x, y) in enumerate(train_dataset):\n",
        "    with tf.GradientTape() as tape:\n",
        "      y_pred = model(x)\n",
        "      loss_per_example = loss_fn(y_true=y, y_pred=y_pred)\n",
        "      loss = tf.reduce_mean(loss_per_example)\n",
        "\n",
        "    gradient = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(zip(gradient, model.trainable_variables))\n",
        "    loss_mean.add_value(loss.numpy())\n",
        "    accuracy_mean.add_value(sparse_accuracy(y, y_pred))\n",
        "  \n",
        "    if step % 100 == 0:\n",
        "      print(f'Epoch {epoch} Step {step} Loss: {loss_mean.result()} Accuracy: {accuracy_mean.result()}')\n",
        "      loss_mean.reset_states()\n",
        "      accuracy_mean.reset_states()"
      ],
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0 Step 0 Loss: 2.346635580062866 Accuracy: 0.15234375\n",
            "Epoch 0 Step 100 Loss: 1.2027271282672882 Accuracy: 0.6998828053474426\n",
            "Epoch 0 Step 200 Loss: 0.5806717258691788 Accuracy: 0.8575390577316284\n",
            "Epoch 1 Step 0 Loss: 0.43301645602498734 Accuracy: 0.8878348469734192\n",
            "Epoch 1 Step 100 Loss: 0.42787124931812287 Accuracy: 0.8848046660423279\n",
            "Epoch 1 Step 200 Loss: 0.4001629453897476 Accuracy: 0.8896093964576721\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7r7ZelV8VYjp",
        "colab_type": "text"
      },
      "source": [
        "Of course, TensorFlow already has a way to do this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TFhKlgteVN-7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "6b8c2f18-ef34-4e32-fbe3-a0b784868729"
      },
      "source": [
        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.05)\n",
        "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)\n",
        "loss_metric = tf.keras.metrics.Mean()\n",
        "accuracy_metric = tf.keras.metrics.SparseCategoricalAccuracy()\n",
        "\n",
        "\n",
        "for epoch in range(2):\n",
        "  for step, (x, y) in enumerate(train_dataset):\n",
        "    with tf.GradientTape() as tape:\n",
        "      y_pred = model(x)\n",
        "      loss_per_example = loss_fn(y_true=y, y_pred=y_pred)\n",
        "      loss = tf.reduce_mean(loss_per_example)\n",
        "\n",
        "    gradient = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(zip(gradient, model.trainable_variables))\n",
        "    loss_metric(loss)\n",
        "    accuracy_metric(y_true=y, y_pred=y_pred)\n",
        "  \n",
        "    if step % 100 == 0:\n",
        "      print(f'Epoch {epoch} Step {step} Loss: {loss_metric.result()} Accuracy: {accuracy_metric.result()}')\n",
        "      loss_metric.reset_states()\n",
        "      accuracy_metric.reset_states()"
      ],
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0 Step 0 Loss: 0.30616647005081177 Accuracy: 0.9140625\n",
            "Epoch 0 Step 100 Loss: 0.3538685739040375 Accuracy: 0.9021875262260437\n",
            "Epoch 0 Step 200 Loss: 0.34719112515449524 Accuracy: 0.9018359184265137\n",
            "Epoch 1 Step 0 Loss: 0.28928661346435547 Accuracy: 0.9163636565208435\n",
            "Epoch 1 Step 100 Loss: 0.3155575692653656 Accuracy: 0.9126172065734863\n",
            "Epoch 1 Step 200 Loss: 0.32115286588668823 Accuracy: 0.908007800579071\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ti3MkPJGA-7W",
        "colab_type": "text"
      },
      "source": [
        "## We did a bunch of heavy lifting. We don't have to.\n",
        "\n",
        "Keras has really high level APIs that take care of all the work we did for us."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yPrfbB9oWR0W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
        "x_train = x_train/255.0\n",
        "x_test = x_test/255.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MpYQFZf1WOZ5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Flatten(),\n",
        "  tf.keras.layers.Dense(50),\n",
        "  tf.keras.layers.ReLU(),\n",
        "  tf.keras.layers.Dense(10, activation='softmax')\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0SDopNqVbhv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 386
        },
        "outputId": "5399a406-1fd9-4ac3-aec4-f136b1d19c1a"
      },
      "source": [
        "model.compile(optimizer='sgd', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
        "history = model.fit(x=x_train, y=y_train, batch_size=256, epochs=10, validation_data=(x_test, y_test))"
      ],
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 60000 samples, validate on 10000 samples\n",
            "Epoch 1/10\n",
            "60000/60000 [==============================] - 2s 28us/sample - loss: 1.6077 - accuracy: 0.5817 - val_loss: 1.0738 - val_accuracy: 0.7891\n",
            "Epoch 2/10\n",
            "60000/60000 [==============================] - 1s 19us/sample - loss: 0.8704 - accuracy: 0.8139 - val_loss: 0.6882 - val_accuracy: 0.8476\n",
            "Epoch 3/10\n",
            "60000/60000 [==============================] - 1s 19us/sample - loss: 0.6321 - accuracy: 0.8508 - val_loss: 0.5457 - val_accuracy: 0.8697\n",
            "Epoch 4/10\n",
            "60000/60000 [==============================] - 1s 19us/sample - loss: 0.5306 - accuracy: 0.8660 - val_loss: 0.4755 - val_accuracy: 0.8798\n",
            "Epoch 5/10\n",
            "60000/60000 [==============================] - 1s 19us/sample - loss: 0.4746 - accuracy: 0.8761 - val_loss: 0.4336 - val_accuracy: 0.8860\n",
            "Epoch 6/10\n",
            "60000/60000 [==============================] - 1s 19us/sample - loss: 0.4389 - accuracy: 0.8819 - val_loss: 0.4050 - val_accuracy: 0.8900\n",
            "Epoch 7/10\n",
            "60000/60000 [==============================] - 1s 19us/sample - loss: 0.4140 - accuracy: 0.8877 - val_loss: 0.3852 - val_accuracy: 0.8946\n",
            "Epoch 8/10\n",
            "60000/60000 [==============================] - 1s 20us/sample - loss: 0.3954 - accuracy: 0.8910 - val_loss: 0.3695 - val_accuracy: 0.8980\n",
            "Epoch 9/10\n",
            "60000/60000 [==============================] - 1s 19us/sample - loss: 0.3807 - accuracy: 0.8942 - val_loss: 0.3573 - val_accuracy: 0.9001\n",
            "Epoch 10/10\n",
            "60000/60000 [==============================] - 1s 20us/sample - loss: 0.3686 - accuracy: 0.8968 - val_loss: 0.3470 - val_accuracy: 0.9029\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "43tcKfh3WicU",
        "colab_type": "text"
      },
      "source": [
        "These APIs are highly performant and robust, so you don't have to worry about messing things up. Use them whenever you can!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g78nNj35WpRc",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7kQ_IsLaXKfL",
        "colab_type": "text"
      },
      "source": [
        "We covered the fundamentals of neural networks and of TensorFlow 2.0. This workshop emphasized the fundamentals and programming aspects instead of best pratices (necessarily). \n",
        "\n",
        "Here are some things I'd recommend checking out next for Neural Networks:\n",
        "*   Backpropogation (how are the gradients computed? spoiler: it's basically chain rule)\n",
        "*   Convolutional and Recurrent Neural Networks \n",
        "*   Regularization (my model's still overfitting! how do I fix that?)\n",
        "\n",
        "Here are some things I'd recommend checking out for TensorFlow 2.0:\n",
        "*   The various layers, metrics, losses, optimizers, and activation functions offered.\n",
        "*   Functional API for building models\n",
        "*   Using `@tf.function` to speed up your code\n",
        "*   Distributed Training with tf.distribute\n",
        "\n",
        "\n"
      ]
    }
  ]
}