lessons/3-NeuralNetworks/05-Frameworks/IntroKerasTF.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "En2vX4FuwHlu"
      },
      "source": [
        "## Introduction to Tensorflow and Keras\n",
        "\n",
        "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n",
        "\n",
        "### Neural Frameworks\n",
        "\n",
        "We have learnt that to train neural networks you need:\n",
        "* Quickly multiply matrices (tensors)\n",
        "* Compute gradients to perform gradient descent optimization\n",
        "\n",
        "What neural network frameworks allow you to do:\n",
        "* Operate with tensors on whatever compute is available, CPU or GPU, or even TPU\n",
        "* Automatically compute gradients (they are explicitly programmed for all built-in tensor functions)\n",
        "\n",
        "Optionally:\n",
        "* Neural Network constructor / higher level API (describe network as a sequence of layers)\n",
        "* Simple training functions (`fit`, as in Scikit Learn)\n",
        "* A number of optimization algorithms in addition to gradient descent\n",
        "* Data handling abstractions (that will ideally work on GPU, too)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cACQoFMwHl3"
      },
      "source": [
        "### Most Popular Frameworks\n",
        "\n",
        "* Tensorflow 1.x - first widely available framework (Google). Allowed to define static computation graph, push it to GPU, and explicitly evaluate it\n",
        "* PyTorch - a framework from Facebook that is growing in popularity\n",
        "* Keras - higher level API on top of Tensorflow/PyTorch to unify and simplify using neural networks (Francois Chollet)\n",
        "* Tensorflow 2.x + Keras - new version of Tensorflow with integrated Keras functionality, which supports **dynamic computation graph**, allowing to perform tensor operations very similar to numpy (and PyTorch)\n",
        "\n",
        "We will consider Tensorflow 2.x and Keras. Make sure you have version 2.x.x of Tensorflow installed:\n",
        "```\n",
        "pip install tensorflow\n",
        "```\n",
        "or\n",
        "```\n",
        "conda install tensorflow\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xwqVx9-bwHl3",
        "outputId": "2aa591b4-b647-441f-9c8e-4e0da2d517a0",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2.7.0\n"
          ]
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "print(tf.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6tp2xGV7wHl4"
      },
      "source": [
        "## Basic Concepts: Tensor\n",
        "\n",
        "**Tensor** is a multi-dimensional array. It is very convenient to use tensors to represent different types of data:\n",
        "* 400x400 - black-and-white picture\n",
        "* 400x400x3 - color picture \n",
        "* 16x400x400x3 - minibatch of 16 color pictures\n",
        "* 25x400x400x3 - one second of 25-fps video\n",
        "* 8x25x400x400x3 - minibatch of 8 1-second videos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qG2bsaR7wHl4"
      },
      "source": [
        "### Simple Tensors\n",
        "\n",
        "You can easily create simple tensors from lists of np-arrays, or generate random ones:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ybpnk08HwHl4",
        "outputId": "fad9ed4a-df82-44a0-84ea-324bc71ea46f",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[1 2]\n",
            " [3 4]], shape=(2, 2), dtype=int32)\n",
            "tf.Tensor(\n",
            "[[-0.33552304 -1.8252622  -1.8532339 ]\n",
            " [ 1.0871267  -1.2779568   0.5240014 ]\n",
            " [-0.12793781 -1.8618349  -0.9020286 ]\n",
            " [ 0.5948797   0.11144501 -2.0396452 ]\n",
            " [ 0.47620854  1.1726047  -0.4405675 ]\n",
            " [-0.27211484 -0.08985762 -0.03376012]\n",
            " [ 0.64274263  0.53368104 -0.9006528 ]\n",
            " [-0.43745974 -1.0081122  -0.13442488]\n",
            " [ 0.36497566  1.3221073  -1.8739727 ]\n",
            " [ 0.94821155 -0.02817811  1.3563292 ]], shape=(10, 3), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "a = tf.constant([[1,2],[3,4]])\n",
        "print(a)\n",
        "a = tf.random.normal(shape=(10,3))\n",
        "print(a)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXFMsV3r09Ux"
      },
      "source": [
        "You can use arithmetic operations on tensors, which are performed element-wise, as in numpy. Tensors are automatically expanded to required dimension, if needed. To extract numpy-array from tensor, use `.numpy()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e5Nu5Xgj1DnQ",
        "outputId": "0dfc8758-4ffd-4968-c7bf-6ba8d435df2e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[ 0.          0.          0.        ]\n",
            " [ 1.4226497   0.54730535  2.3772354 ]\n",
            " [ 0.20758523 -0.03657269  0.9512053 ]\n",
            " [ 0.93040276  1.9367073  -0.18641126]\n",
            " [ 0.8117316   2.9978669   1.4126664 ]\n",
            " [ 0.0634082   1.7354046   1.8194739 ]\n",
            " [ 0.97826564  2.3589432   0.9525811 ]\n",
            " [-0.1019367   0.81715     1.718809  ]\n",
            " [ 0.7004987   3.1473694  -0.02073872]\n",
            " [ 1.2837346   1.7970841   3.2095633 ]], shape=(10, 3), dtype=float32)\n",
            "[0.71496403 0.16117539 0.15672949]\n"
          ]
        }
      ],
      "source": [
        "print(a-a[0])\n",
        "print(tf.exp(a)[0].numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uQ5zN6cVyrG7"
      },
      "source": [
        "## Variables\n",
        "\n",
        "Variables are useful to represent tensor values that can be modified using `assign` and `assign_add`. They are often used to represent neural network weights.\n",
        "\n",
        "As an example, here is a silly way to get a sum of all rows of tensor `a`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7pu0UZ-_yqfB",
        "outputId": "6708c83e-02e6-4442-8757-45918eb1fbc2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([ 2.9411097, -2.9513645, -6.2979555], dtype=float32)>\n"
          ]
        }
      ],
      "source": [
        "s = tf.Variable(tf.zeros_like(a[0]))\n",
        "for i in a:\n",
        "  s.assign_add(i)\n",
        "\n",
        "print(s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rIh1EHcezlNo"
      },
      "source": [
        "Much better way to do it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aQIdWZ1kzn6P",
        "outputId": "1c123d9a-ecd2-4f2e-828e-5ade85ac8f63"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(3,), dtype=float32, numpy=array([ 2.9411097, -2.9513645, -6.2979555], dtype=float32)>"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tf.reduce_sum(a,axis=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-auwezDwHl6"
      },
      "source": [
        "## Computing Gradients\n",
        "\n",
        "For back propagation, you need to compute gradients. This is done using `tf.GradientTape()` idiom:\n",
        " * Add `with tf.GradientTape` block around our computations\n",
        " * Mark those tensors with respect to which we need to compute gradients by calling `tape.watch` (all variables are watched automatically)\n",
        " * Compute whatever we need (build computational graph)\n",
        " * Obtain gradients using `tape.gradient` "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m8vFOXr7wHl6",
        "outputId": "860ac72e-50c7-4ff2-f258-747f27194f90",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[ 0.40935674 -0.3495818 ]\n",
            " [ 0.94165146 -0.33209163]], shape=(2, 2), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "a = tf.random.normal(shape=(2, 2))\n",
        "b = tf.random.normal(shape=(2, 2))\n",
        "\n",
        "with tf.GradientTape() as tape:\n",
        "  tape.watch(a)  # Start recording the history of operations applied to `a`\n",
        "  c = tf.sqrt(tf.square(a) + tf.square(b))  # Do some math using `a`\n",
        "  # What's the gradient of `c` with respect to `a`?\n",
        "  dc_da = tape.gradient(c, a)\n",
        "  print(dc_da)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8sfjBMBu59B5"
      },
      "source": [
        "## Example 1: Linear Regression\n",
        "\n",
        "Now we know enough to solve the classical problem of **Linear regression**. Let's generate small synthetic dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "j723455WwHl7",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.datasets import make_classification, make_regression\n",
        "from sklearn.model_selection import train_test_split\n",
        "import random"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "WJNK_J6v6I-Z",
        "outputId": "eb4a66a6-6b9a-4c8a-bc24-d81eeb2d3f27"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.collections.PathCollection at 0x12892776880>"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbVklEQVR4nO3df4ylVXkH8O+zw0XuAjJaphVGx8XULKkQdtmJoZnGyGrFij82YNTGtKltsmmTNkIo7egfRf9iEmPVxqbNxNJqSnVpFzYGVDRZDHWj2Bl2EZClUXErs7aMZQfEmbqzw9M/7n2XO++85z3nve857z3n3u8n2bC795075903PPfMc57zHFFVEBFRvLYNegBERFSOgZqIKHIM1EREkWOgJiKKHAM1EVHkzgnxphdffLHu2LEjxFsTEQ2lxcXFn6nqRNFrQQL1jh07sLCwEOKtiYiGkoicML3G1AcRUeQYqImIIsdATUQUOQZqIqLIMVATEUUuSNUHEdEoOXR0CZ+4/0mcXFnDpeNt3HrdTuzbPent/RmoiYhqOHR0CR+5+1GsrW8AAJZW1vCRux8FAG/BmqkPIqIaPnH/k2eDdGZtfQOfuP9Jb9+DgZqIqIaTK2uV/r4fDNRERDVcOt6u9Pf9YKAmIkIn1zwzdxiXzd6HmbnDOHR0yenrbr1uJ9qtsU1/126N4dbrdnobGxcTiWjk1VkQzF5n1QcRUUBlC4IuAXff7kmvgTmPqQ8iGnlNLAjWYQ3UIrJTRI71/HpeRG5qYGxERI1oYkGwDmugVtUnVXWXqu4CsAfAKoB7Qg+MiKgpTSwI1lE1R/0WAD9UVWODayKi1FRZEOzdLn5RuwURYGV1PcgiYkZU1f1ikTsAPKyqny14bT+A/QAwNTW158QJxnIiGi756pC8dmsMt99wZV/BWkQWVXW66DXnxUQRORfAuwH8a9HrqjqvqtOqOj0xUXjsFxFR9MrqqYuqQ3r53jqeqZL6+B10ZtP/430UREQRsNVTu1SBhKgUqVKe97sAvuh9BEREkbA1WHKpAglRKeIUqEVkO4DfBnC39xEQEUXCVk9dVB3SK1SliFPqQ1VXAfyK9+9ORBSRS8fbWCoI1tksOV8d0lTVB7eQExF13Xrdzi1VHflZcujt4kUYqImIupposNQPBmoioh6DmDHbsCkTEVHkGKiJiCLHQE1EFDnmqIloJPQ2U4plkdAVAzURDYWyQNzvUVvZey6trGFMBBuqmBxAkGegJqJKYpyZlgViALjlrkewkesUajtqK/+e2ddXOU/Rl0ptTl1NT0/rwsKC9/closEqavNZp7WnLzNzhwt3FI63W/jlmRdLO95NjrcLP3RM79n7dUdm99YffJeXNqdERLamRYNi6tGxsrZeGqQFnRmy4qWZctbW1NYFr8nzFBmoichZrIfA9tOxTgDk8wlVOuU1eZ4iAzUROTMFp4vaLWOz/SaYzjx8xfZW4fVjIluCdMalU17T5ykyUBORs6Lg1dom+MXpM8YUQgj5U1gA4PYbrsTkeBuCTv749huuxG3vekNhAP/k+67CpOXk8X27J8++J9AJ7uh5b1Z9EFGUipoWrZ4+g1Or65uus1VU1GGq8Lj9hiuNi3umKpUYO+UVYdUHEdVy2ex9hWkEAfDU3PXev5+pGqOfKoyYSg3Lqj44oyaiWmzN9n3zuaAZy4zZhjlqIqrFtJAXarHN9AHQZBVG0xioiaiW3kW33oW8UDPVpj8YYsDUBxHV1mQKIdZTWEJyCtQiMg7gcwCuQKdG/A9V9dsBx0VEZJRKbtkX1xn1ZwB8TVXfKyLnAtgecExERIViqtJokjVQi8jLAbwJwB8AgKqeBnA67LCIiDbrt1XpMHBZTHwdgGUA/ygiR0XkcyJyfv4iEdkvIgsisrC8vOx9oEQ0WPndgE1vE4+1IVQTXAL1OQCuBvB3qrobwC8AzOYvUtV5VZ1W1emJiQnPwySiQcpms01uE+/93mUtRwfdEKoJLjnqpwE8raoPdf/8bygI1EQ0vEyz2Y99+fHSU1Xq5pOL+l/nDXP9dMYaqFX1v0XkJyKyU1WfBPAWAN8PPzQiikVZv+eVtU6fj/ypKj7yyUUfEL2GvX4641r18WcA7uxWfPwIwIfCDYmIYmPaJp7XmzM25ZOrBOqytMYgzi4cFKdArarHABQ2CyGi4XfrdTutKYhMWXCtmk82fUD4PgYrdtxCTkRWRdvETU35Lx1vO/XjcKkiGcXt4kW4hZyInOR3A5oOus2CaNlrLjXR2WLk2voGxkSwoTpS6Y5eDNREI6xOZYZLzw3Ta2U10ft2T24J5BuqZwP9qAVpgAcHEI0s04y4iWOmTIcNAJ20imnhsig3PSzbyssODmCOmmhEDXKnnymHLUBpdUl+MXKQG3GaxEBNNKJ8npRSVdEioQDGWXYmH+BHZVs5AzXRkDNVV5hmtQoE7+WRryIZb7esQbqo2mOQHzZN4mIi0RArq64oq4323ZnOlEfuXTgsY6r2aPq8xkFhoCYaQllgLApiWWogW5SzXVc3UNtK8cq2idsWN4s+bNqtMVx7+QRm5g4nv8CYYeqDyINBtwDNjyVbYDPJUgP7dk/iyOxeiOW6Omx55LLvYatAKdqIc+OeSRxcXBqqBUbOqIlqiq2hva2REbA1NeAzhZBPc9jak5ZtE3f598tvxJmZO+ylz0hMOKMmqmkQlQdlM3jbLLhoUc7XVu2icjnTbD37EPC9TdxlgTGmn4BccEZNVFPTlQe2GXzZLHZMZNOHSDbD9HWyd9GHlqK49G719BkcOrrk/VRx208Hsf0E5IKBmqimpisPbNuvixbYWtsEEGB9oxMui4KTj5O9TR9Oik4JXta7GgBOra5vGoOvIGlaYMxm6LZ/vxgx9UFUU9Md3mwz+KIFtgvOO+dskM6Y0jN10gKmD6fJ8TbOf9nWeWGIFFHR/fcuSqZYe80ZNVFNvn90t3GZwednqJfN3lf4XqYt2f2mBcpmszcfOOY0Bh/KZugp1l5zRk3kQVbm9tTc9Tgyuzfoj9D9zOBd+kMD9RdGy2azrmMILcUe15xREyWmnxm8LW+b8ZEWMM1mXccQWtM/AfnAQE2UoKqLb67BKWRaIKYA6XPxsgnsR00UqUH0WR5kj+pRV9aPmjNqoggNqtY3plkvvcQpUIvIjwH8HMAGgDOmqE9Efgyy1je1tMAoqDKjvlZVfxZsJER0Voq1vhQOUx9EEcjno8e3t3BqdX3LdTHX+lI4rnXUCuDrIrIoIvuLLhCR/SKyICILy8vL/kZINOSKGhm98H9n0Brb3M4o9lpfCsd1Rj2jqidF5FcBfENEjqvqg70XqOo8gHmgU/XheZxEQyM/e149fWZLPnr9RcV4u4XzX3YOF/XILVCr6snuf58RkXsAvBHAg+VfRUR5RdUcJs+trePYbW9ramgUMWvqQ0TOF5ELs98DeBuAx0IPjGgYuTT1zzAfTRmXGfWvAbhHRLLr/0VVvxZ0VERDyrVqg/lo6mUN1Kr6IwBXNTAWoqFn2qLNfDSVYXkeJWkQ26t9MDUm+ti732Acf++9XtRuQQRYWV1P6r6pHgZqSk6T26ttHwhVPzCqbtHO32vvCSlF953qBxiVY1MmSs7M3GHjqdVHZvd6+z62BkVNNDAy3Wuv7L7ZUCltZU2ZeHAAJaep7dW2JvpNnD7uck/ZNYM4DZ2awUBNyWnqpBDbB0ITHxgu95Rdw/4gw4uBmpLT1FFKtg+EJj4wiu61V+99x3LUFfnHQE3JsZ0y7YvtA6GJD4z8vY63W3jF9lbhfYcYT50TyckfLiYSlfBd9THo8VZ9Ly5ONqdsMZGBmihxoT4smqquoQ4exUWUiKpBN2RNORcn48EcNVHOoPKyRX2pP3L3o6XfP2RJHhcn48FATUkJHUT7CZau72sbdz9BN+Sst6nqGrJjoKZkhAqivULMUF3H3U/QDTnrbaq6huyYo6ZkNHEyd4gZquu4TZ31yoKuqcmTr1kvTySPAwM1JaOpnYBVg6WNaXxLK2uYmTt8duHw2ssncHBxqVLQrdrkidLEQE3JCBFE80LMUE3jFrx0FNfSyhoOLi7hxj2TeOD4cqWgy1nv8GOgpmSE/jEfKJ+h9luvXDRuAZDfwbC2voEHji83UqMc20YdKscNL5SUQQWYol16APCK7S3c9q7ipv9lDf9NrUsFwFNz14e4hU3j4o7D+HDDCw0N24/5oQK56VDaU6vrhRtMihr+t1tj+NT7d2Hf7knjrr8mapSbWJQlv1ieR17E0LwnZPle2YJlUfmercxvkDXK3HGYHudALSJjInJURO4NOSBKTxP1zS5MwfGWux6pPRbbTDcf5GzBcJA1ytxxmJ4qqY8PA3gCwMsDjYUSFcuP0qbguKFau/9F0YJgr94gd+joEraJYKNg/af3ukFVazSxKEt+Oc2oReTVAK4H8Lmww6EUxfKjdNmMsO7uwmwGPN5ubXmtN8hlP10UBelYgiF3HKbHdUb9aQB/AeBC0wUish/AfgCYmpqqPTBKRxP1zS5ss966HxzZDLhswdK06DgmElUwZO11WqyBWkTeCeAZVV0UkTebrlPVeQDzQKc8z9cAKX6x/CidBZ5b7nrEmnao+31MQc70YfCiKgMj9c0l9TED4N0i8mMAXwKwV0T+OeioKCkx/Si9b/ckPvm+qwZWUcGFOgqh0oaX7oz6z1X1nWXXccMLDVpMG2O4mYRccMMLjZxB5WDZJIlC4BZyGphB9ptgrwuKDWfUFB2fZ/35Omdw4cSzlTvXETWBM2oaCFOvC6CzGOkaJKvkhLOAXtYQqff/BuaWqUllM2r2+qCBKKtprrIF3eXorENHl7Dr41/HTQeOGYM0UNx21MchsUR1MVDTQNjK1VyDpG1XZDbjXllbrz7IkvcnahIDNTnz2SGvqHtcnkuQtNUtm3YK5knF9ydqEgM1OfHdIa93k4yJS5C0tQt1CfaT42188JqpgW2SIbJh1Qc5CdEhr7d3Rr9b0G11y2WnqeQXC6df+0qW7FGUGKjJScgOeXU3iZRtbjE1aio6QouNiihWDNRDoInNG6E75IUKktwpSMOAgTpxPjeOlImlQ14/OFOm1HExMXEudcQ+xNQhj2jUcEaduCZPV2liZtqbxrmo3YJI56Tvse7RVlV2LRINCwbqxMVyuoqNSx49n8bp3aSSHQQQKrVDFDOmPhJnqyOOgWsNtuvmFG7tplHDQJ24FHLHpjz6TQeObdrhWCVdw63dNEqY+hgCsVc1uDRgAso3p+TFltohComBmrzL56PHt7dwatXcFClLZdhOEc/EltohCo2BmrwqqutubRO0xgTrG+be5ydX1rZsTmHVB1EHAzV5VZSPXn9RMd5u4fyXnWNMbWSpjKbTODySi1LAxUTyypSPfm5tHUdm9+LT798VTZWK746ARKFYA7WInCci3xWRR0TkcRH5eBMDo5f47AMdmq0/dExVKk3t6iSqyyX18UsAe1X1BRFpAfiWiHxVVb8TeGyE5np5+OLSEySWKpUmd3US1WEN1No5/faF7h9b3V/+T8SlQj77QDeRj02pW10quzqJnE4hF5ExAIsAfh3A36rqXxZcsx/AfgCYmprac+LECc9DHU2Xzd5X+KkoAJ6au975fYqa82enbhdVUozCIluVE8yJQis7hdyp6kNVNwDsEpFxAPeIyBWq+ljumnkA8wAwPT3NGbcnvmZ9RTPz7CHl0ym2dEtR46SV1fXkAnpKs38abZXK81R1RUS+CeDtAB6zXD60mpxt+uoDbcu79qZTbItspsZJsefPi8SSLycq41L1MdGdSUNE2gDeCuB44HFFq+mSLl9VEi4z8CyYly2y2RonsWqCyD+XGfUlAD7fzVNvA3CXqt4bdljxCnHIq42PWZ/L9uwsmJvSLdtEnHpxsGqCyC+Xqo/vAdjdwFiSkGJJV5aqWVvfOLsVO1tIzPSmU0xBvejrimQBfxQWJImawJ2JFdk2dMSmN1UDdIJtuzWGD14zZUynZOmWMZEt76foVIuYZAGfu/6I/GGvj4piOuTVZcZqStU8cHwZR2b3Gt973+5J3HzgWOFrWUlfWdXHzNzhxlNERMOKgbqiWEq6XHcs1knVmHLVk+Pt0iBf9/sS0WYM1H2IoaTLdVGzTh12nZ8euOuPyB/mqAML1VDJdcZa50zFOqWBKZzlSJQKzqgDCtlQyXXGWjdV0+9PD7GkiIiGgVOvj6qmp6d1YWHB+/umZmbucGEwHRPBJ993Va2gxT4VRMOldq8P6o8pPbGhWntmzRkr0ehgoA6o7FRtH6Vqvhc1uUGFKE5cTAyoaEGtV0ylatygQhQvBuqAynb4AXGVqvFYKqJ4MfURWJY6aHo3Y9U0BjeoEMWLgboBLgt/PvPD/ZQFcoMKUbwYqBtStvDnu966n1asMfUwIaLNmKOOgM/88KGjS8ZKk7I0hq8DCojIP86oI+ArP5zNzE1saYwYepgQ0VacUUfAV4/rsmOymMYgShdn1DX5WAQ05YevvXwCM3OHa1duAGAagyhhDNQ1+FoELKoKufbyCRxcXPJSuTE53maQJkrY0AbqujPdOqenuG4NL/se/ZyQwsoNouFkDdQi8hoAXwDwKgAvAphX1c+EHlgddWe6TZyeYvse/bx3P42a2N+DKH4uM+ozAG5R1YdF5EIAiyLyDVX9fuCx9a3uTLeJ01Ns36Pf965SuRGyXzYR+WOt+lDVn6rqw93f/xzAEwCi/r+4brlbE6en2L5HEyeksL8HURoqleeJyA4AuwE8VPDafhFZEJGF5eVlT8PrT91yN9evr7NJxPY9mtiAwv4eRGlwXkwUkQsAHARwk6o+n39dVecBzAOdE168jdBBPs+ar5gAqs1GqyzK9btJxOV7hN6Awv4eRGlwmlGLSAudIH2nqt4ddkjVFPVRPri4hBv3TPY9G+1nNlv1ENsYtmzzAFqiNFjPTBQRAfB5AM+q6k0ub9rkmYmmcwknx9s4Mru39vu7VEWkfH4hqz6I4lB2ZqJLoP4tAP8O4FF0yvMA4KOq+hXT14QM1PnAYmpAJACemru+8vv1BqqiACwAFJ0Pguza0B8WRDT8ah1uq6rfQic+DVxROVkWOPNc8qy28rSiqojse/Vey0U5IgopqaZMpsCZ/xRxzbPaytNsgTa71ldTJSKiIkkFalPgzFIRAmC83cJ5rW24+cAx66KebSbsEmhPrqxxUY6IgkoqUJsCZ5YL/tT7d+GXZ17EqdV1p5O0bTNh2yni2bUxVHAQ0fBKqimTrfa46tZx2/v19s4oyofnr2VgJqIQkgrUtqZDVRf1XJoY9QbgGErZYhgDETXLWp7XjybrqHsNe5lcyvXaRFSurDwvqRy1TSqLelV3MWbYRIloNCWV+rDppx9z0+q0FmW9NtFoSjpQm/K1/QZmX/nfsvep0yubTZSIRlOygdp303tf71f35JayIM+jtohGU7I5at/5Wl/vZ3ufstrtok6AvXXgrNcmGk3Rzahd0w++87W+3s/l5BbTrNglLcJ6baLRE1WgNqUNFk48iweOL28K3r7ztb7ez/Y+ZQueNx84VvieXCwkGm1RBWrTjPLO7/zXlq51N+6ZrHWKS56v/G+dk1u4WEhERaLKUZc1Xeq1tr6BB44ve83X+sr/1nmfVOrAiahZUe1MNO0sLOJ6MEBquEWcaDTVOjigSUVpgzoHA6SIi4VElBdV6qMobfDBa6aYDiCikRbVjBoonlFOv/aVTAcQ0ciKLlAXYTqAiEaZNVCLyB0A3gngGVW9IvyQwuFCHRGlyGVG/U8APgvgC2GHspXPwOq7NwgRUVOsi4mq+iCAZxsYyya2vhdVsZczEaUqqqqPXr4DK3s5E1GqvAVqEdkvIgsisrC8vFz7/XwHVtuJ40REsfIWqFV1XlWnVXV6YmKi9vv5Dqzcnk1EqYo29eEzsGaLkmvrGxgTAQCMt1s4r7UNNx84VuncQiKiplkDtYh8EcC3AewUkadF5I/CD8tfk6TeRUkA2FBFa5vgF6fP4NTqupeFSiKikKJqyhRClUZPk+NtHJndG3hERERblTVlijb14UuVxUdWgBBRjIY+UFdZfGQFCBHFaOgDddGiZGuboDUmm/6OFSBEFKskmjLVYTqjsOjvuJWciGI09IuJREQpSOKEF3a2IyIqFkWgZmc7IiKzKBYT2dmOiMgsikDNznZERGZRBGp2tiMiMosiULOzHRGRWRSLiaZaZy4kEhFFEqgBnjRORGQSReqDiIjMGKiJiCLHQE1EFDkGaiKiyDFQExFFLkj3PBFZBnCizy+/GMDPPA5nkIblXoblPgDeS4yG5T6AevfyWlWdKHohSKCuQ0QWTK3+UjMs9zIs9wHwXmI0LPcBhLsXpj6IiCLHQE1EFLkYA/X8oAfg0bDcy7DcB8B7idGw3AcQ6F6iy1ETEdFmMc6oiYioBwM1EVHkBhKoReTtIvKkiPxARGYLXhcR+Zvu698TkasHMU4XDvfyZhF5TkSOdX/91SDGaSMid4jIMyLymOH1lJ6J7V5SeSavEZEHROQJEXlcRD5ccE0Sz8XxXlJ5LueJyHdF5JHuvXy84Bq/z0VVG/0FYAzADwG8DsC5AB4B8Bu5a94B4KsABMA1AB5qepwe7+XNAO4d9Fgd7uVNAK4G8Jjh9SSeieO9pPJMLgFwdff3FwL4z4T/X3G5l1SeiwC4oPv7FoCHAFwT8rkMYkb9RgA/UNUfqeppAF8C8J7cNe8B8AXt+A6AcRG5pOmBOnC5lySo6oMAni25JJVn4nIvSVDVn6rqw93f/xzAEwDyTduTeC6O95KE7r/1C90/trq/8lUZXp/LIAL1JICf9Pz5aWx9YC7XxMB1nL/Z/THpqyLyhmaG5l0qz8RVUs9ERHYA2I3O7K1Xcs+l5F6ARJ6LiIyJyDEAzwD4hqoGfS6DOOFFCv4u/2nkck0MXMb5MDp7+F8QkXcAOATg9aEHFkAqz8RFUs9ERC4AcBDATar6fP7lgi+J9rlY7iWZ56KqGwB2icg4gHtE5ApV7V0T8fpcBjGjfhrAa3r+/GoAJ/u4JgbWcarq89mPSar6FQAtEbm4uSF6k8ozsUrpmYhIC53Adqeq3l1wSTLPxXYvKT2XjKquAPgmgLfnXvL6XAYRqP8DwOtF5DIRORfABwB8OXfNlwH8fnfl9BoAz6nqT5seqAPrvYjIq0REur9/Izr/5v/b+EjrS+WZWKXyTLpj/AcAT6jqXxsuS+K5uNxLQs9lojuThoi0AbwVwPHcZV6fS+OpD1U9IyJ/CuB+dKom7lDVx0Xkj7uv/z2Ar6CzavoDAKsAPtT0OF043st7AfyJiJwBsAbgA9pdFo6JiHwRnVX3i0XkaQC3obNIktQzAZzuJYlnAmAGwO8BeLSbDwWAjwKYApJ7Li73kspzuQTA50VkDJ0Pk7tU9d6QMYxbyImIIsediUREkWOgJiKKHAM1EVHkGKiJiCLHQE1EFDkGaiKiyDFQExFF7v8B4SC4LI9GLoEAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "np.random.seed(13) # pick the seed for reproducability - change it to explore the effects of random variations\n",
        "\n",
        "train_x = np.linspace(0, 3, 120)\n",
        "train_labels = 2 * train_x + 0.9 + np.random.randn(*train_x.shape) * 0.5\n",
        "\n",
        "plt.scatter(train_x,train_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ng4rZmGc6oxk"
      },
      "source": [
        "Linear regression is defined by a straight line $f_{W,b}(x) = Wx+b$, where $W, b$ are model parameters that we need to find. An error on our dataset $\\{x_i,y_u\\}_{i=1}^N$ (also called **loss function**) can be defined as mean square error:\n",
        "$$\n",
        "\\mathcal{L}(W,b) = {1\\over N}\\sum_{i=1}^N (f_{W,b}(x_i)-y_i)^2\n",
        "$$\n",
        "\n",
        "Let's define our model and loss function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "QxhI4GlB6aiH"
      },
      "outputs": [],
      "source": [
        "input_dim = 1\n",
        "output_dim = 1\n",
        "learning_rate = 0.1\n",
        "\n",
        "# This is our weight matrix\n",
        "w = tf.Variable([[100.0]])\n",
        "# This is our bias vector\n",
        "b = tf.Variable(tf.zeros(shape=(output_dim,)))\n",
        "\n",
        "def f(x):\n",
        "  return tf.matmul(x,w) + b\n",
        "\n",
        "def compute_loss(labels, predictions):\n",
        "  return tf.reduce_mean(tf.square(labels - predictions))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JUxwj3367gD2"
      },
      "source": [
        "We will train the model on a series of minibatches. We will use gradient descent, adjusting model parameters using the following formulae:\n",
        "$$\n",
        "\\begin{array}{l}\n",
        "W^{(n+1)}=W^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial W} \\\\\n",
        "b^{(n+1)}=b^{(n)}-\\eta\\frac{\\partial\\mathcal{L}}{\\partial b} \\\\\n",
        "\\end{array}\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "-991PErM7fJU"
      },
      "outputs": [],
      "source": [
        "def train_on_batch(x, y):\n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions = f(x)\n",
        "    loss = compute_loss(y, predictions)\n",
        "    # Note that `tape.gradient` works with a list as well (w, b).\n",
        "    dloss_dw, dloss_db = tape.gradient(loss, [w, b])\n",
        "  w.assign_sub(learning_rate * dloss_dw)\n",
        "  b.assign_sub(learning_rate * dloss_db)\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idr2VEWb9rr0"
      },
      "source": [
        "Let's do the training. We will do several passes through the dataset (so-called **epochs**), divide it into minibatches and call the function defined above:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "nOuu0qpx-wAp"
      },
      "outputs": [],
      "source": [
        "# Shuffle the data.\n",
        "indices = np.random.permutation(len(train_x))\n",
        "features = tf.constant(train_x[indices],dtype=tf.float32)\n",
        "labels = tf.constant(train_labels[indices],dtype=tf.float32)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3zdIf6c_85Ht",
        "outputId": "43b04684-8b90-4c65-d5ff-20ebac61c73c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 94.5247\n",
            "Epoch 1: last batch loss = 9.3428\n",
            "Epoch 2: last batch loss = 1.4166\n",
            "Epoch 3: last batch loss = 0.5224\n",
            "Epoch 4: last batch loss = 0.3807\n",
            "Epoch 5: last batch loss = 0.3495\n",
            "Epoch 6: last batch loss = 0.3413\n",
            "Epoch 7: last batch loss = 0.3390\n",
            "Epoch 8: last batch loss = 0.3384\n",
            "Epoch 9: last batch loss = 0.3382\n"
          ]
        }
      ],
      "source": [
        "batch_size = 4\n",
        "for epoch in range(10):\n",
        "  for i in range(0,len(features),batch_size):\n",
        "    loss = train_on_batch(tf.reshape(features[i:i+batch_size],(-1,1)),tf.reshape(labels[i:i+batch_size],(-1,1)))\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We now have obtained optimized parameters $W$ and $b$. Note that their values are similar to the original values used when generating the dataset ($W=2, b=1$)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "US6q0nCBD-LL",
        "outputId": "65a79620-a3eb-445b-aafb-60a60575ab0e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(<tf.Variable 'Variable:0' shape=(1, 1) dtype=float32, numpy=array([[1.8616779]], dtype=float32)>,\n",
              " <tf.Variable 'Variable:0' shape=(1,) dtype=float32, numpy=array([1.0710956], dtype=float32)>)"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "w,b"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "_e6xRMZFDnyI",
        "outputId": "d202b7fe-4383-4d82-b98e-a20f3180093e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x12892ae5eb0>]"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAArG0lEQVR4nO3deXhU1f3H8fdJiBBBCSKLBBGUxQUEBK02rmhdQePSuvxcaq1YxSouSLQu4BpFpXWpihbBrW5IQEGpNoqIoobFFVFBUYMKFMMaIAnn98fNxGEyy53Mncm94fN6njwNmTt3zvU+/d4z53zP9xhrLSIi4l9Zjd0AERGJT4FaRMTnFKhFRHxOgVpExOcUqEVEfK5ZOk668847265du6bj1CIiTdLcuXNXWmvbRXstLYG6a9eulJWVpePUIiJNkjFmaazXNPQhIuJzCtQiIj6nQC0i4nMK1CIiPqdALSLic2nJ+hAR2ZaUzC9nzIxFLKuopFNeLiOO6UVh/3zPzq9ALSKSgpL55Vz70idUVtUAUF5RybUvfQLgWbDW0IeISArGzFhUF6RDKqtqGDNjkWefoUAtIpKCZRWVSf29IRSoRURS0CkvN6m/N4QCtYgIzlhzQXEp3YqmUVBcSsn8clfvG3FML3Jzsrf6W25ONiOO6eVZ2zSZKCLbvFQmBEOvK+tDRCSN4k0Iugm4hf3zPQ3MkTT0ISLbvExMCKYiYaA2xvQyxiwI+1ljjBmegbaJiGREJiYEU5EwUFtrF1lr+1lr+wEDgA3A5HQ3TEQkUzIxIZiKZMeojwQWW2tjFrgWEQmaZCYEw5eLt87NwRio2FCVlknEEGOtdX+wMeOBedbaB6K8NhQYCtClS5cBS5cqlotI0xKZHRIpNyebO07p06BgbYyZa60dGO0115OJxpjtgBOBF6K9bq0dZ60daK0d2K5d1G2/RER8L14+dbTskHBeLx0PSWbo4zic3vTPnrdCRMQHEuVTu8kCSUemSDLpeWcC//a8BSIiPpGowJKbLJB0ZIq4CtTGmO2B3wEved4CERGfSJRPHS07JFy6MkVcDX1YazcAbT3/dBERH+mUl0t5lGAd6iVHZodkKutDS8hFRGqNOKZXvayOyF5yupeLR6NALSJSKxMFlhpCgVpEJExj9JgTUVEmEREvzJsHjzySllMrUIuIpOLrr+GMM2DAABg9GjZs8PwjFKhFRBrip59g2DDYay94+WW4/npYuBC2397zj9IYtYhsE8KLKaU0SbhmDYwZA2PHwqZNcOGFcOON0LGj942upUAtIk1CvEDc0K22Qucsr6gkt6aaM+dN46/vP0+b9avh9NPh1luhe/e0X5sCtYgkxbOeqcdtihWIAa56/iNqIiqFJtpqK3TOTZs2c/LnM7lq1lN0XrOcWbv14+9n/Ilzhp1CYffMXHdSZU7dGjhwoC0rK/P8vCLSuKKV+UyltKdXCopLo64ozMvNYVP1lrgV7/LzcqM+dAru+C89583impkT2WvFt3zcsTt3HvZHZnftV/e+2UWDPLuGeGVO1aMWEddS3QQ2XWLV6KiorIr7PgN1AX6r4ZBN33PvQ8P5zfef8k2bXbj0xGuYtufBWPNr/kUm91NUoBYR1/y6CWysGh3xGCByPKHTj9/S5uxb4fPZdG/VhuuPvoRn9z2a6uz6oTKT+ykqUIuIa7ECYuvcHAqKSxtt3DpWjY4WOVn8sqF+rzrbmK3GrDuuWcnw2c/w+0/eYENOc7jlFt494g9Mem0J1VGGTTK9n6LyqEXEtWhlPnOyDOs3V1NeUYnl1yGE8J1RvBa5CwvAHaf0IT8vF4MzfnzHKX24acg+UTetvecPfcnPy6V15VqK3hzPW48O5eTPSnl8wBBOv+ZpuP56hhT0rDsnOMGdsHNn8kGkHrWIuBataNGGzdX1eq3pHLeOleFxxyl9Yk7u1ctS6dWGHhPeoPO4+9lh43om9z6Cew8+m1U778Idp/TZ6nobO6MFlPUhIinqVjSt3lgvOGPA3xSf4PnnxcrwcJWFUV0Njz8Oo0bBsmX8dMhRjNjvdN5psUujpxoq60NE0iZRsX2vNWhC01qYPBmuuw4WLYKDDoJnn6XjIYfwZFpa6S2NUYtISqKNW6dzsi3WAyDmg+Gtt5zAfOqpkJ0NJSUwezYcckha2pcOCtQikpLC/vlRJ/LSNYTg+sHw0Udw3HFwxBFQXg7jx8PHH8NJJ0HtxGBQaOhDRFKWyUm3hLuwfPMN3HADPPMM5OU5BZSGDYPczOU9e81VoDbG5AGPAb1xcsT/ZK19L43tEhGJKeqDYflyp0jSww9Ds2YwcqTzk5fXKG30ktse9T+A16y1pxljtgO8L7gqIpJA1IJQ3XeEe++Fu++Gykq44AK46Sbo1Kmxm+uZhIHaGLMjcCjwRwBr7WZgc3qbJSKytcj86eX/W8On197KcR+8QPNf/udMFt52G/TK3IrBTHHTo94dWAE8bozpC8wFLrfWrg8/yBgzFBgK0KVLF6/bKSKNrLHLm4YKQhm7hSEL3+aqWU+xW8VPzN29HwNemw4HHJCxtmSam6yPZsB+wEPW2v7AeqAo8iBr7Thr7UBr7cB27dp53EwRaUyh3mwml4mHf3ZBcSnlv2zgsCVzeWXCcO57+W7Wb5fLub8fzWmn3dKkgzS461H/APxgrX2/9t8vEiVQi0jTFau86aipn8XdVSXVHnjoAdFz6eeMmTmR3373Md+17sBlQ67m5b0OxZqsulocTVnCQG2t/ckY870xppe1dhFwJPB5+psmIn4Rr95zqOZz5K4qDdn6KtKzT73B3dMe5YRFs1m5fWtuOuoinul3LFXZOUDmq9g1FrdZH38Fnq7N+FgCnJ++JomI37it9xwqxhT6PdprrgL1smUwejRPPfoYG3OaM7bgLB7bv5D1zX9NOMv3yTZgmeAqUFtrFwBRi4WISNMXrd5zLPFqbiTcYKCiAu66C/7+d6iuZvKBJ1E84FT+1zJvq8O83gbL77QyUUQSclveFH6tuZGoUFP4GHbXltncv2o2vSc+6ATrs86Cm28mZ3VzNrz0CURsCLAtDHeEU6AWEVciVwPG2ug2FETjvRa+w/dpn5ZyxTtP02ntSn7+7eF0eHAs9OtXF8grq2rqdmTZloY7wilQi2zDUsnMSFhzI85rY177goM/n82ImU/Q83/fsWCXnlw5+Eq+3/c3zK4N0uGBvsbaukC/rQVp0MYBItusWD3itG8zNWsWZWdexMDyhSzeqTN3HXouM3oeVFfRLj/OxGW0senGXojjFW0cICL1xMqNTtcWWnzyiVO4/5VX2G3HthQdcykv7Ps7arJ+LVlqiD62HRI5GRlrWy5ILg3Q71SPWmQb1aCdUhpi6VI47zzo2xdmzYLiYua8Oocp+59QL0gn+n4fuTlAvIdNU6JALdLERe7YHVr2HWtHFAtbHddgK1fClVdCz57w3HNw9dWwZAmMHMmQ33bfarOBvNychEE6WrZHxh42jUyBWqQJi1ejI9pOKSEp1fJYv96pC73HHvCPf8DZZzNj8iwKdjqWbne9V/cQKOyfz+yiQYw9vR+bqrfEPWWsXWOS3pYroDRGLdIEhSbYoo33hoYGQpNyiY5zPdZbVQWPPQY33ww//QSFhXDbbZRsah13HDna8EVIosnNaAtxcnOyOWLPdhQUlwZ+gjFEPWoRD8QaXmistoR60bGEhgZCvdpYOwi6GkLYssUZ2th7b7jkEujRA95919n1e++9E44jx/uMRBko0fZrPHVAPpPmljdKpb90UY9aJEV+yzyI10MNiRwaiFXLI+EQwhtvQFERzJ0LffrAK69QsktfxvznS5ZNmRa3RkgoQMc6Jj8v19V/v8iFOAXFpZnNZskA9ahFUtQYmQfxevCJesHRJuVc7+wdMncu/O53zs/KlfDEEzB/PiWd+nHt5E+36s3G6q2HHgJJf3YCbiYY/fQNyA31qEVSlOnMg0Q9+Hi92GxjtnqIhHqYblYZAvD113D99c5QR9u2MHYsXHwxNG9e9/7Ih5Yleurdhs3VdZOKrj7bpUTfDvz2DcgNBWqRFDV42KCBEi1UiTbBlpNlwEBVjRMuowWnqDt7h/z0kzNJ+OijsN12TrC++mpo3Xqrw2I9nCxOCl6odjXALxuqtmqDV0Ey1gRjqIee8YU+HtDQh0iKvP7qnkiiHny0CbZWLZrVBemQWMMz4cMCvxv1MosuHO6k2j36KAwdCosXwy231AvSEPvhlJ+XS8vm9fuF6Rgiinb94ZOSQcy9Vo9aJEVef3VPxE0PPrKH2q1oWtRzxVqSXVO5kfPnT+fS955jp8o1/HD0iXR+8B7o3j1u2+L1Zq94boGrNnghXg8909+AvKBALeIBL7+6J5Loq300boPTPa9+znHz/8OVs56m85rlvN21P3cddh6/7NmH2QmCNMR/aMXK1850gGzIf7/GpkAtEjAN6cEnDE7WwvTpPDp2GHuuXMrHHbsz8rjLmN21HwAmiV5vrIeWXwJkpr8BeUGBWiSAku3Bxw1O770HI0fCrFm0bJvPsBNHMn3PAqz5dQrLi16vnwJkJr8BeUGBWsSnvK6zXC84ff45FA6DKVOgY0d46CHm7XcspS9/gU1TrzdoAdIvFKhFfCitub7ffw+jRsGECdCqlVNAafhwaNmSkwCbk+OLXq/8ylWgNsZ8C6wFaoDqWLsQiIg30pLru2oV3HEH3H+/MyY9fDhcey3svPNWh6nX6z/J9KiPsNauTFtLRKSOp7m+GzbAffdBcTGsWQPnngujR8Nuu6XYSskUDX2I+EDkeHTe9jn8sqGq3nFJTepVV8P48U5QXrYMBg+G2293iidJoLhdmWiB/xhj5hpjhkY7wBgz1BhTZowpW7FihXctFGniohX3X7exmpzsrcsZuZ7UsxYmTYLeveGii6BrV2cLrJdfVpAOKLc96gJr7TJjTHvgdWPMF9bat8MPsNaOA8aBswu5x+0UaTIie88bNlfXG4+u2mLJy82hZfNmyU3qvfmmU3b0gw+c+tBTpsCQIXU7fEswuQrU1tpltf+73BgzGTgAeDv+u0QkUrRsjlhWV1ax4Kaj3Z14wQJnYvC112DXXZ0hj3PPhezoW21JsCQc+jDGtDTG7BD6HTga+DTdDRNpitwU9Q9xNR69ZAn83/9B//7w/vtw993w5Zdw/vkK0k2Imx51B2Cycb46NQOesda+ltZWiTRRbrM2Eo5HL1/u5D8//DA0a+b0pq+5BvLyvGmo+ErCQG2tXQL0zUBbRJq8WMWRXI9Hr10L99zj/FRWwgUXwE03QadOGWi9NBal50kgeb28OlNiFSYadeI+MdtfMr+csdM+5YiZk7ns3WfZacNqpvcq4MkTLuT0s4+iUEG6yVOglsDJ5FZKiR4IyT4wki1MVDL3e9659QGefHMiXVb/zLtd9uXOw87jo07OsMiCiOsO6gNM4jPWep9JN3DgQFtWVub5eUXA2WU61q7Vs4sGefY5kQ8EcHq/od1CEr2eEmthxgy+/NNf6fnj13zWfnfuPOw83u62X71Uu9B1p7U9knbGmLmxynNoKy4JnExtpZRod/G07T7+wQcwaBAcdxzNK9dx2ZARDP7j33l79wFR86FD190Yu6FLZmjoQwInU1spJXogeP7AWLQI/vY3Z1Vh+/Zw//2cW9GDpeuq474tdN1B3AtQ3FGPWgInU5vJxgr8ob8net218nJn09h99oEZM5zaHF9/DZdeyhUn9K53reHCr9uz9ojvKFBL4CTaZdoriR4IKT8wKiqc/OcePZza0MOGOTt833gj7LADUP9a83JzaLN9TtTrTscDLHxH8oLiUkrmlzf4XNJwmkwUicPrrA/AyX9+4AGnNnRFhbOy8OaboVu3tLc32XNpcjJz4k0mKlCLZEp1NTzxhLNA5Ycf4LjjnGDdN7X1ZOlKyctUdo044gVqTSaKpJu1ThW7666DhQvhgAPgySfh8MPrHZps0E1nTrkmJ/1DY9QiETwdl501CwoK4OSTYcsWJ6NjzpyYQTqyLvW1L30S9/PTmZKnyUn/UKCWQEn35FZDgmVUn3zi7Khy6KGwdCnzb7iLQ8+5n24fNKfgzjejnq8hQTedvd5MZddIYgrUEhieBdE4Uu6hLl0K553njDvPng3Fxbz8wkzOsn34bu3muO1uSNBNZ683U9k1kpjGqCUw0rIzd4QG91BXroTbboN//hOysmDECGenlTZtKC4uddXuhizkiVXkyater3Yk9wcFagmMTExuJR0s16+HsWPhrruc388/H0aNgs6dE7avvKKSguLSuonDI/Zsx6S55UkF3WSLPEkwKVBLYGRi6bjrHmpVFTz6qJP//PPPUFjo7PC9116u2234dSuu8opKJs0t59QB+bz5xYqkgq56vU2fArUERrq/5kP8HmrJ/HLufnUh/d5/naJ3nqLzqmXOZOHkyXDQQUm12wCRKxgqq2p484sVGclRVjnUYFGglsDI1Nf8aD3UkvnlvHz3RB7673j6/LyYhe268sfTbuKj3gdxU4suFEY5T3gwbJ2bQ4ucLCo2VMXsYUNmcpQzWc9bvKFALYGS6Gt+WnqKZWXkn3ER/1o8jx92bM8VJ1zJlL0PY0tWNlRWRw1ykcGworKK3Jxsxp7ej8L++TFX/WUiRzkTk7LiLaXniSf8ULzH8/S9r76C00+H/fdn92VfM/rICxl04SNM7j3ICdK1oqXvJUrza8wcZa04DB7XPWpjTDZQBpRbawenr0kSNH75Kh0rOF71/EfJteWnn5xJwkcfhebN4YYbOCNrf77aGLtfExnkEgXDxszWyFQ9b/FOMkMflwMLgR3T1BYJKL98lY4VHGusdffgWL0axoxx0u02b4aLLoIbboAOHRgWpZJcuPAgVzK/nCxjqIlS8Cz8uMbK1sjEpKx4y9XQhzGmM3AC8Fh6myNB5Jev0vF6hHFXF27c6ATnPfZwFq2ceKJTPOmBB6BDB+DXVXp5uTn13h4e5ELfLqIFab8EQ604DB63Peq/A9cAO8Q6wBgzFBgK0KVLl5QbJsHhl6/S0XqK4eo9OGpq4KmnnEL9330HRx/tlB3db7+o7w/1gONNWEb7dgGQbYyvgqFyr4MlYaA2xgwGlltr5xpjDo91nLV2HDAOnHrUXjVQ/M8vX6VDgeeq5z+KP+xgLUyb5uyu8umnMHAgjB8PRx7p+nNiBblY3yK2WKvAKA3mZuijADjRGPMt8CwwyBjzVFpbJYHip6/Shf3zuecPfWNnVLz7rrNIZcgQZ8jj+eedXb9dBulEVBpU0iGpHV5qe9RXJ8r60A4v0tgihydu7g5HPvkPmDoVOnZ0dlm54ALIqT/mnOrnavsqaQjt8CLbnLrhie+/d4Ly3yZCq1bOZOHll0PLlmn7XFCRJPGW9kyURpPWehOrVjkTg/ff74xJX3qpsxVW27bp/2yRBlCPWnzHy0Uy4UG32/aG+1fMYp8nH4I1a5wi/qNHQ1gmUqzPLlu6KunKdSKZoB61NIpYtS7AmYx0GyRDQXfzps384ePXGT77GTqsW8WPh/6OXR68F3r33urYMTMWxfzcyIp2GluWTFKPWnwn3mKYZHrXY177gsM/mcnVs55kj1XllOXvxbCTRvJj74HMrg3SJfPLGTX1Myoqq+KeK1rZURUqEj9QoJZGEa/UJ7gMkm++yT/vv5i+P37Fl2278OdTbuCN7geAMZjac0fLwkiGChWJHyhQi2teTsAlWkUIcYLk/PnOYpUZM+jQuj1XHz+cl/Y5YquKdqG85VgrBSNFK+Qffh6RxqQyp+KK1yVEwxfJxFIvSC5ZAmed5Szx/vBDuPtuPnjtXabtd8xWQTp8VaSbHnF+Xi7/d2CXRis7KpKIetTiSjoq5IXXzoi7BP3nn+HWW+GRR6BZMyfNbsQIyMvjRGBL8xYxe/rxhlgiJwsH7raTUvbElxSoxZV0VsiLuUik+47OYpV77nGWe//5z04BpU6d6r0/VkCNNcTSZvscbhqyz1bvU6Ei8SsF6iYgE4s30l0hb6sguWmT03s+5lZYsQJ+/3unR92zZ4POC1opKMGmQB1wmdpdJSMV8rZsgX//G66/Hr79FgYNguJi2H//lE6rnrIEnSYTAy7R3nxeSWuFPGvh1VedScKzz4a8PJgxA954I+UgLdIUqEcdcJncXSUtPdP334eRI2HmTNh9dz68/QGusL0oL91E6/dexxj4ZUMV2bVbWyWzalGkqVCgDji/7K6SSOQ4+uie2Rz19H3w0kvQvj088ABT9j+eopcXUVm1CWCrlYShjQAaa+NckcakoY+AG3FML9/n/4bnYLdfu5JL/30nh582iKrXZjgFkxYvhmHDuKv0G1eLU9IxtCPiZ+pRB1wQshrGzFhEztrVXDbnRc6fO5WsLVt4Yr/BPPDb08ndriMjvlpNYf9WSQ3XaGm3bEsUqJsAX2c1VFYyZMaT/GXOi+y4cT0l+xzOvQf/Hz/kdXReDxvKSFT/I5zfhnZE0kmBWjxXMr+ce6d/zkHvvMJV7z5D0ZqVlO4+kLsOO48v2nerd3xoKMNN/Q/w39COSLopUIunSub9wBvF4/hX6QR6/O975u/SiyuGXM0Hu/WhqiZ27fNlFZX1hnFa5+Yo60MEBWrx0ttvs8c5F1P43ed8vVNnLjr5Omb0OAiMIW+7ZrRs3izm0EZoKCPTwzjakkuCQIFaUvfxx07Z0enT2blVW0Ye+1de7HMUNWEV7VZXVrHgpqMTF2DKoEyt6hRJVcJAbYxpAbwNNK89/kVr7U3pbpj8yre9vm+/dYokPfUUtG4Nd97JWZv68M36LfUODe8xgz+yVNJREVAkHdz0qDcBg6y164wxOcA7xphXrbVz0tw2wae9vhUr4Lbb4KGHICvLKTlaVARt2nC5ix6zX7JUMrmqUyQVCQO1dXa/XVf7z5zaH+93xJWovOz1pdwzX7cOxo6FMWNg/Xr405+cMqSdO9cd4qcecyJBWdUp4moXcmNMNjAX6A48aK0dGeWYocBQgC5dugxYunSpx03dNnUrmhb1qWiAb4pPcH2eaGPDoe2nomVShAf1Lq2acd+6MvpOuN8p4n/yyU6Peq+9GnxdfhBrvFw7j0tjSHkXcmttDdDPGJMHTDbG9LbWfhpxzDhgHMDAgQPV4/aIV72+aD3z0E2KHE4JBbCNm6sYvHAWV816iq4VP7JywIHsXFJCSfNdGTNlEcsmLqlLoavYUOXr3nM0Qer9y7YtqawPa22FMeYt4Fjg0wSHN1mZnNzzqg50onHX8OGUMTMWMeDLMopmTqD3z4tZ2K4rfzxtFF/tdzAjmu+6VXvCCyf5Yvw8SX4ZLxeJx03WRzugqjZI5wJHAXemvWU+lenJPa96fW6WZy+rqISyMu58+EoOXvoRP+zYnuGDr2LK3odhTRZm9caEu3ora0LEe2561LsAE2vHqbOA5621r6S3Wf7VGCldXvT6Ei3P7rqqnBvmPAN3zmSflq0ZfeSFPN3veDY3y6k7JssYV7U4lDUh4i03WR8fA/0z0JZACGJKV2ioprKqpm4pdmgisd26VVw++9+c8dEMaNECbryR2UedybP/+ZbNEUE9/H3xhMbPfZv/LRIwWpmYpKCldEUO1dRYS25ONmftuSNdHnuQ38+exHY11Xz3+3PY/b47oUMHBgPVrXbgquc/qivYH2IhbrAOjZ/7Mv9bJKC0cUCS/FSov2R+OQXFpXQrmkZBcSkl88vrHRM5VNO8ejNnzX6Ryy48lvPeeobtTzuFZl8uYvfnJkCHDnXHFfbPZ0uM1M1QSp8B8nJzaLN9Tr19FDO1l6PItkA96iT5JaXLbY81NCSTtaWGUz57k+HvPE3nNSt4u2t/Dp30mLOhbAyxvj3k5+Uyu2hQ3PYFcYhIxK8UqBvADyldbic1O7VuwZ5zZ3LNzIn0WvkdH3XswYjjh/Nd3wOZHSdIQ2qpgUEbIhLxMwXqNEvXhJqrHuvs2Ux9/lraLviQJW06cclJRUzvVUDuds24w0WwTeXbg1f53yKiQJ1W6ZxQi9tj/ewzuO46mDqVtrvswoLrirls+/58v7Yq6cL7Df324JchIpGmwFWtj2QNHDjQlpWVeX7eoCkoLo0aTLON4Z4/9E0paEWrU7H7hv/x+LfT2G3ai9CqFYwcCZdfDi1bNvhzRCQzUq71IQ0Ta3iixtqUe9bhPdYNP/7MiPmTOf39qWQb4IornEL+bds26Nwi4i8K1GkUb9m2F6sZC3vmUfjqHJhwp1OC9NxzYfRo6NKlQefTAhURf1IedRpFy7kO1+BUtaoqeOQR6NED/vY3OPxwZzusxx9PKUhf+9InlFdUYvl1PD1abraIZJYCdRoV9s/njlP6kG1M1NeTTlWzFl54AfbZB/7yF9h9d3jnHZgyxflbCrRARcS/NPSRZqGhg5RT1UpLne2uPvzQCcpTp8LgwRDjIZDsMIYWqIj4lwJ1BrhJVYsZWOfPdwL0f/4Du+4KEybA2WdDduwhlYakBWqBioh/KT3PB6Kl2vVc+zP/WjKVXV+bAjvt5IxFX3KJU+EugVhpgfGWfmtbKpHGpfQ8nwsfH955/S/89d1nOWvBa9RkN3MWrlxzDbRu7epcJfPLY2aaxBvG0AIVEf9SoPaBZRWVtNq0gQs/mMyfP5xM8+rNPNv3GO4rOJMPbjvH9XlCveJYEg1j+KGGiYjUp0Dd2DZt4vLPpnPOf5+ibeUaXul1MPcceg7f7JRPvgcb2IaozoZIcClQp6jBi0S2bIFnnoEbbmD4t9/yXte+3H7oH/lklx6AE1iP2LMdBcWlKWduABprFgkw5VGnoEGLRKyF6dOhf3845xxo0wZmzODnSa+waq996wrwnzogn0lzy5M6d6yhjfy8XAVpkQBrsj3qVJdDu3l/0hvdzpnjpNrNnAl77MGHdzzIFVt6Ul66iU55X271GQXFpUlvoqvSoiJNU8JAbYzZFXgC6AhsAcZZa/+R7oalItXyosnunhKp3t+/+MLJ3pg8Gdq3hwceYMr+x1P08iIqqzZF/YyGLEBpSOaG6nuI+J+bHnU1cJW1dp4xZgdgrjHmdWvt52luW4Ml3dNt4PsTLhIpL4dRo2D8eKfU6M03O5XtWrXirgQ95oYuQEkmc0Mb0IoEQ8Ixamvtj9baebW/rwUWAr7+f3Gqy6Hdvj/WRrfXHdTBqQXdvTtMnAh//SssXgw33ODUiXbxGZnYRFf1PUSCIakxamNMV6A/8H6U14YCQwG6NLCCm1dSXQ7t9v2RQw1dW2Zx/8pZ9C78J6xe7Sz1vvlm6No16c/IxAIU1fcQCQbXgdoY0wqYBAy31q6JfN1aOw4YB84Scs9a6ELkOOsRe7Zj0tzyBk+qJTMpV9g/n8I+HZwaHKNGOcMdxx8Pd9wB++6b0mekewGK6nuIBIOr9DxjTA5OkH7aWvtSepuUnGgpcpPmlnPqAGfBSCjdLZk84lB50oTvt9aZIOzTBy68kE+zduT0s4opOOQqSmri767i+jPSKBPDKyKSuoRFmYwxBpgIrLLWDndz0kwWZWpIAaJkxMyKmDnTSbWbM4e1Xbtz7f5n8kq3A+rKjgaloJGyPkT8IV5RJjeB+mBgFvAJTnoewHXW2umx3pPOQB0ZWGIVIDLAN8UnJH2+8EAVraLcXsu/YcTMiQxaUkZl+47k3n4rh/7Uhe/Wbq53bq8eFiLS9KVUPc9a+w5O3Gt00dLJDBDtUeNmnDVRelp4VkTn1T9z5aynKPzsLdY2357bDz+f539zEqP2G8j3zy2Ien5NyomIFwK1MjFaOpmFesHa7ThronzpZRWV7LRhNZe++xxnz5/OlqwsHvnNqTx04GmsadGq7hyalBORdApUrY9YPVQLdZNyebk5tMjJ4ornFlBQXBq3Nkbc9LR16/jb3BeY+cifOW/eK0zqPYjDLxzHnYf/sS5Ih47VpJyIpFOgetSxeq6hseBkV9pFO19OTRUXLyqFPf7En5cv5z97FnDnwWezuO2uMdukovsikk6BCtSJco+TXToefj5jtzBk4SyunvUkXSp+gsMOgylT2NB8VzbOWARRxsPDP1tF90UkXQIVqBP1XJNdaVfYPx+sZeYDz3DB9HH0/nkxq3vsBc+Mh2OPBWMoDPtcP6Sy+aENIpJZgQrUEL/nmvSk3ocfUlhURGFpqbPM+8knaX3WWZAVfei+sXvNKqIksm0K1GRiIq4n9b78Ev7wBzjgAPj4Y/jHP5xSpGefHTNIe6lkfjkFxaV0K5qWcMIznIooiWybAtejjifhpN6PP8Lo0fDYY9CiBdx4I1x1Fey4Y8bamEqvWEWURLZNgQ7UscZr6wW81avhrrtg7FioroaLL4brr4cOHVydz6t2QWq1spWvLbJtCmygdtUz3bgRHnwQbr8dVq2CM8+EW26BPfZo2Pk8aFeiXnG8IK+ttkS2TYEdo447XltT45Qd7dkTrr4a9t8f5s1zdv2OEqQTns+rdhG799spLzfhZrl+qLgnIpnnux612+GHqD1Ta9n7w7eg72Xw2WdOgJ4wAQYlLozk1fivm51bYvWK3QyLNHbmiYhknq8Cdaxhg7Klq3jzixVbBe/I8dqBP3zGyLcmsn/5505P+oUX4NRT68qOJuLV+G8qO7dcoeJOIhKFrwJ1rB7l03O+q1sRGArepw7IZ9LccnZdtpgRbz/B777+gOWtdmL+34rpf9OVkJOT1Gd7Nf6bys4tmiwUkWh8FajjFV0KV1lVw+dzPuW1z0vY9ZUXWLfd9jx8zAXk31jEkN92b9Bne1WvI5XzaLJQRKJJuHFAQzR044BYu7WEy6tcw7D3nufcedNo3izL2eG7qAjaxt/6Kii0RFxk25TSxgGZFK1HGSqElLt5I38qm8JF70+iZdVGXt3vaAa/9Ag08o7nXtNkoYhE8lWgjjZscGT3PLLGP84lbz9N+/W/8Hr333DfkedzwUWDoYsCmog0fb4K1BDWo7TWydy47i/w1Vd81LU3lxRey4+9B2g4QES2Kb4L1AD897/OuHNZGeyzD0ydSt/Bg3nRZaqdiEhTkjBQG2PGA4OB5dba3mltzbx5ToB+/XVn7HnCBKeiXXZ2wre6oYk6EQkiN0vIJwDHprkdUFEBhxwCc+fCvffCokWU7HsUBWNmJl0ONJpEy7NFRPwqYY/aWvu2MaZr2luSlweTJ8NvfgOtW3teJD+VqnUiIo3JX0WZjj4aWrcGvC+Sr1rOIhJUngVqY8xQY0yZMaZsxYoVKZ/P68Aar2qdiIifeRaorbXjrLUDrbUD27Vrl/L5vA6srrfpEhHxGX8NfYTxMrCGsj0qq2rIrk3xy8vNoUVOFlc8tyDliUoRkXRKGKiNMf8G3gN6GWN+MMZckP5meVckPzzbA6DGWnKyDOs3V/PLhiplgIiI7/mqKFM6uCn0FJKfl8vsosSbDIiIeC1eUSbfDn14JZnJR2WAiIgfNflAnczkozJARMSPmnygjjYpmZNlyMneum6IMkBExK/8WZTJQ7F2XIn2N61QFBE/avKTiSIiQRCIHV5U2U5EJDpfBGqvCzCJiDQlvphM9LoAk4hIU+KLQK3KdiIisfkiUKuynYhIbL4I1KpsJyISmy8mE2PlOmsiUUTEJ4EanGCtwCwiUp8vhj5ERCQ2BWoREZ9ToBYR8TkFahERn1OgFhHxubRUzzPGrACWNvDtOwMrPWxOY2oq19JUrgN0LX7UVK4DUruW3ay17aK9kJZAnQpjTFmsUn9B01SupalcB+ha/KipXAek71o09CEi4nMK1CIiPufHQD2usRvgoaZyLU3lOkDX4kdN5TogTdfiuzFqERHZmh971CIiEkaBWkTE5xolUBtjjjXGLDLGfG2MKYryujHG3Ff7+sfGmP0ao51uuLiWw40xq40xC2p/bmyMdiZijBlvjFlujPk0xutBuieJriUo92RXY8ybxpiFxpjPjDGXRzkmEPfF5bUE5b60MMZ8YIz5qPZaRkc5xtv7Yq3N6A+QDSwGdge2Az4C9o445njgVcAABwLvZ7qdHl7L4cArjd1WF9dyKLAf8GmM1wNxT1xeS1DuyS7AfrW/7wB8GeD/r7i5lqDcFwO0qv09B3gfODCd96UxetQHAF9ba5dYazcDzwInRRxzEvCEdcwB8owxu2S6oS64uZZAsNa+DayKc0hQ7ombawkEa+2P1tp5tb+vBRYCkUXbA3FfXF5LINT+t15X+8+c2p/IrAxP70tjBOp84Puwf/9A/Rvm5hg/cNvOg2q/Jr1qjNknM03zXFDuiVuBuifGmK5Af5zeW7jA3Zc41wIBuS/GmGxjzAJgOfC6tTat96UxdngxUf4W+TRyc4wfuGnnPJw1/OuMMccDJUCPdDcsDYJyT9wI1D0xxrQCJgHDrbVrIl+O8hbf3pcE1xKY+2KtrQH6GWPygMnGmN7W2vA5EU/vS2P0qH8Adg37d2dgWQOO8YOE7bTWrgl9TbLWTgdyjDE7Z66JngnKPUkoSPfEGJODE9ietta+FOWQwNyXRNcSpPsSYq2tAN4Cjo14ydP70hiB+kOghzGmmzFmO+AMYGrEMVOBc2tnTg8EVltrf8x0Q11IeC3GmI7GGFP7+wE4/83/l/GWpi4o9yShoNyT2jb+C1horb03xmGBuC9uriVA96VdbU8aY0wucBTwRcRhnt6XjA99WGurjTGXAjNwsibGW2s/M8b8pfb1h4HpOLOmXwMbgPMz3U43XF7LacDFxphqoBI4w9ZOC/uJMebfOLPuOxtjfgBuwpkkCdQ9AVfXEoh7AhQA5wCf1I6HAlwHdIHA3Rc31xKU+7ILMNEYk43zMHneWvtKOmOYlpCLiPicViaKiPicArWIiM8pUIuI+JwCtYiIzylQi4j4nAK1iIjPKVCLiPjc/wPCCvjKMZGi9wAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.scatter(train_x,train_labels)\n",
        "x = np.array([min(train_x),max(train_x)])\n",
        "y = w.numpy()[0,0]*x+b.numpy()[0]\n",
        "plt.plot(x,y,color='red')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0giuwC9GHzi8"
      },
      "source": [
        "## Computational Graph and GPU Computations\n",
        "\n",
        "Whenever we compute tensor expression, Tensorflow builds a computational graph that can be computed on the available computing device, e.g. CPU or GPU. Since we were using arbitrary Python function in our code, they cannot be included as part of computational graph, and thus when running our code on GPU we would need to pass the data between CPU and GPU back and forth, and compute custom function on CPU.\n",
        "\n",
        "Tensorflow allows us to mark our Python function using `@tf.function` decorator, which will make this function a part of the same computational graph. This decorator can be applied to functions that use standard Tensorflow tensor operations. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "HK7HPLz3Hyrl"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def train_on_batch(x, y):\n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions = f(x)\n",
        "    loss = compute_loss(y, predictions)\n",
        "    # Note that `tape.gradient` works with a list as well (w, b).\n",
        "    dloss_dw, dloss_db = tape.gradient(loss, [w, b])\n",
        "  w.assign_sub(learning_rate * dloss_dw)\n",
        "  b.assign_sub(learning_rate * dloss_db)\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J7HusxWkGjLX"
      },
      "source": [
        "The code has not changed, but if you were running this code on GPU and on larger dataset - you would have noticed the difference in speed. \n",
        "\n",
        "## Dataset API\n",
        "\n",
        "Tensorflow contains a convenient API to work with data. Let's try to use it. We will also train our model from scratch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oYro9Lbr8q0M",
        "outputId": "78c0a6de-71bd-4eef-8819-439495b28672"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 173.4585\n",
            "Epoch 1: last batch loss = 13.8459\n",
            "Epoch 2: last batch loss = 4.5407\n",
            "Epoch 3: last batch loss = 3.7364\n",
            "Epoch 4: last batch loss = 3.4334\n",
            "Epoch 5: last batch loss = 3.1790\n",
            "Epoch 6: last batch loss = 2.9458\n",
            "Epoch 7: last batch loss = 2.7311\n",
            "Epoch 8: last batch loss = 2.5332\n",
            "Epoch 9: last batch loss = 2.3508\n"
          ]
        }
      ],
      "source": [
        "w.assign([[10.0]])\n",
        "b.assign([0.0])\n",
        "\n",
        "# Create a tf.data.Dataset object for easy batched iteration\n",
        "dataset = tf.data.Dataset.from_tensor_slices((train_x.astype(np.float32), train_labels.astype(np.float32)))\n",
        "dataset = dataset.shuffle(buffer_size=1024).batch(256)\n",
        "\n",
        "for epoch in range(10):\n",
        "  for step, (x, y) in enumerate(dataset):\n",
        "    loss = train_on_batch(tf.reshape(x,(-1,1)), tf.reshape(y,(-1,1)))\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A10prCPowHl7"
      },
      "source": [
        "## Example 2: Classification\n",
        "\n",
        "Now we will consider binary classification problem. A good example of such a problem would be a tumour classification between malignant and benign based on it's size and age.\n",
        "\n",
        "The core model is similar to regression, but we need to use different loss function. Let's start by generating sample data:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "j0OTPkGpwHl7",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "np.random.seed(0) # pick the seed for reproducibility - change it to explore the effects of random variations\n",
        "\n",
        "n = 100\n",
        "X, Y = make_classification(n_samples = n, n_features=2,\n",
        "                           n_redundant=0, n_informative=2, flip_y=0.05,class_sep=1.5)\n",
        "X = X.astype(np.float32)\n",
        "Y = Y.astype(np.int32)\n",
        "\n",
        "split = [ 70*n//100, (15+70)*n//100 ]\n",
        "train_x, valid_x, test_x = np.split(X, split)\n",
        "train_labels, valid_labels, test_labels = np.split(Y, split)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "id": "c-_BjSHPwHl8",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def plot_dataset(features, labels, W=None, b=None):\n",
        "    # prepare the plot\n",
        "    fig, ax = plt.subplots(1, 1)\n",
        "    ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
        "    ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
        "    colors = ['r' if l else 'b' for l in labels]\n",
        "    ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
        "    if W is not None:\n",
        "        min_x = min(features[:,0])\n",
        "        max_x = max(features[:,1])\n",
        "        min_y = min(features[:,1])*(1-.1)\n",
        "        max_y = max(features[:,1])*(1+.1)\n",
        "        cx = np.array([min_x,max_x],dtype=np.float32)\n",
        "        cy = (0.5-W[0]*cx-b)/W[1]\n",
        "        ax.plot(cx,cy,'g')\n",
        "        ax.set_ylim(min_y,max_y)\n",
        "    fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "tq0vFchQwHl8",
        "outputId": "9a5aa6a0-c92f-4d72-9e78-c0f615804bff",
        "scrolled": false,
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_66184/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABOsElEQVR4nO2dd3zddfX/Xyc3u0maNLM7pZRSSmkLoYBlz7ItoCBDv4LwU0EFVETBAYii4EJBRBTEgYLsMovMQoEW6KQFOtLdJk2bZs97fn+87od7k9zZ3J3zfDzuI7n3fu7nnrve5322qCoMwzAMIyPRAhiGYRjJgSkEwzAMA4ApBMMwDMODKQTDMAwDgCkEwzAMw4MpBMMwDAMAkJloAQZDWVmZVldXJ1oMwzCMlOK9997bqarl/W9PaYVQXV2NxYsXJ1oMwzCMlEJENvi73VxGhmEYBgBTCIZhGKlFby/Q2QnEoMtESruMDMMwhgzr1gEvvQS88w7gdgPDhgEnnwwcdRRQUhKVpzCFYBiGkey88grwt78B2dnAqFGAywV0dACPPw7Mnw9cdx0wduygn8ZcRoZhGMnMxx8D999PReAoAwDIzQXGj+f/v/kNFcQgMYVgGIaRzDz7LFBQQOvAH6WlwK5dwJIlg34qUwiGYRjJSmsrsGwZUFYW/LiiIuCNNwb9dKYQDMMwkhXHDZQRYqnOzgaamgb9dEmjEEQkV0TeFZGlIrJSRG5KtEyGYRgJZdgwQISppsHo6KDraJAkjUIA0AngeFWdDmAGgDkicnhiRTIMw0ggubnA4YcDdXXBj2tpAY49dtBPlzQKQUmL52qW52LzPQ3DGNrMmQP09DCe4I/t24ExY4CpUwf9VEmjEABARFwisgRAHYD5qvqOn2OuEJHFIrK4vr4+7jIahmHElbFjgW99C9izB9i4kYqhqwvYvRtYvx4YMQK45hogK2vQTyUag/LnwSIixQAeB/ANVV0R6Liamhq15naGYQwJGhqAhQuZTdTeDlRUACedBMyYAeTkRHQqEXlPVWv6356Ulcqq2igirwKYAyCgQjAMwxgylJYCZ5zBS4xIGpeRiJR7LAOISB6AEwGsTqhQhmEYQ4hkshBGAvibiLhARfWwqs5LsEyGYRhDhqRRCKq6DMDMRMthGFGnrg5YsABYtAjo7mb/mRNOACZP9valMYwkIGkUgmGkJa+9xi6VIswGycwEVq0CFi9mMPCrXwXy8hItpWEAMIVgGLFj2TLgL38BRo/umwWSm8vhJkuXAn/9K3DllYmTEWAV7McfA9u20WIZNw6orqYSSzZ6e6lQP/mEMwHGjwemTYs4y8bwjykEIyBuNyviRbiGJeP6kLSoAo89xsEl/hYrES5mixYBW7ZQaSSClSuplHbt6juBa9w44PLLo9JjP2p88gnwxz8y/z4jg5fubiA/H/i//wNmzUq0hCmPKQRjAO3twFtvsevurl28bfx44LTTgEMOMbd3WGzfDmzYwIU1ECJ8M999F5g7N36yOaxYAdxxB11ZTl99gIqhoQG49VbgxhtZBZto1q0DbruNXT19ZQWAtjbgD38ArrrKlMIgSZq0UyM5aGoCfv5z4MEHvZvYceO4Kfv974F772UVvRGC5mYu9qHMqpwcIBEV9z09wH33Mbe9qKjvfSJstywC/POf8ZetP6rAP/7BWMvw4QPvz88HqqoYq+nqir98aYQpBKMP990HbN0KTJjAmRwA14WSEmCffYC336blYIQgJ4c+t1B0dw9ckOPBqlVAYyNQWBj4mIoKHrdtW/Sfv60NWL4ceP997v6DvVebN/OYYN088/PZ0mHlyujLOoQwl5HxKVu2MA7a3yJ3EOEEv+efB045xeJ4QRkzhlq0pcWrWfujyh3tIYfEVzaAPXFC9dgX4THbtgEjR0bneTs6gCeeAF5+mVaKCJVBZSVw/vnATD+Z53V1lCOUteVy8Uvs7xxGWJiFYHzK8uX8zQX73eXk8De9bt3A+3p6uP6Z1Q4uTmeeCezYEXj3u2MHza59942vbJEQzV5nXV3A737HHUV5OX2RY8dyB9LVxbnA/qZ+ZWSEJ4fbbQGuQWIWgvEpLS1Mkw+FSN953vX1TKh57DHGIFwu4KijgC9+EZgyJXbyJj1HHw1s2gTMn0/f94gRfPPa2rjrraxkIDQR6VsTJoR2abndXIijlQH16qt06UyYMPA1FxVx6tcDDzCNtLjYe58TmHe7g1s1qlSwxl5jFoLxKSNG0KUdCrfb63r+6CNa+nfcwcSa3l5mKT32GHDuucDf/x5bmZOajAzg4ouBb3+bLqQNG3jp7OSb9sMfRmXK1V4xeTKfe8+ewMfU1XFxrqwc/PP19gLPPcdzBVKAubn8cr3Tr+t9aSndasFiGbt20a01adLgZR3CmIVgfMrMmUzm6O0NbHm3tFBx7LMPY3hXXsnf6ahRfX/nhYW8/5ZbWON01FFxeQnJhwgwfTovnZ30q+XlhfbfxxqXC7jiCuCXv+QHXlLi/QBVqQwyM4GLLorO8+3aRfMxVF1DURGwZAmDVL5ceCF7/2/ezIXf+YI6srrdwLXXJv59TXHs3TM+paQEOP54bmL9eRO6u+n2Pvdc/u6efx6orWUyir9N37Bh/N3+4Q8xFz01yMnhm5Isi9bkycD113MRdqwX51JdDdxwA9M5o0E4GVcA3xt/x44YQXkOOYSB440b6Y7buJG7kxtvpMzGoDALwejD+edzZ//WW7Tgi4u5Cdu1i5vbL3wBOOIIHvvf/3KNC+YCLytjZmF9PeOIRpIxaRLNuPXrqe1FuIuPduV0SQknenV2Bk9Pa2oCPvMZ//eNGAF87WvABRdQGajSBRUtpWWYQjD6kpVFT8LxxzMzcM0a7vJPPJEx0lGjvMfW1YVOPXVqs3bsMIWQtIhwlx3LgGx2Nqd7zZsXOK+5t5eXUP7FkhJejKhjCsEYgAg3jqHic0VFVArB6O3l+QKl4htJjir9gmvX0kSsquIw972Z33viiaxs3LaN5/E1Lbu76f4580zb8ScQUwjGXjN3LnDzzcGPaWykazeZeqQZYbJ1K/CnPzGmAHj9+wUFzJ46/PDIzjd8OGMWf/oTu6s6vZx6ehjAPu88KgQjYZhCMPaauXOBO+9kHzR/2ZMdHbx8+ctWL5Ry7NgB/OxntBDGj++7m29rA+6+m8ohkL8/EKWlwPe/zxjAypXMUS4vZ4rbYM3I9nZ+GTMyeM69sWKGOKYQjLDZs4c923JyGCwuLATuugv4f/+PXoCiIt7X28vjurqAz3+eGz8jxXjkEX6AvkEjh/x83v63v3Ehj3TAjwiLzYJ1go2E3bvZYOu11/jlc2Q85RS6qXJzo/M8Q4CkUQgiMhbAgwCqALgB3Kuqv0usVAbANhVPPcU+R47XYOxYWvezZjHb6J57WIi6Zw+tgf32Ay65hMeEU/1sJBG7djE1LFjb69xcZgx98EHkVkI0qatje949e1if4FgF7e3Aww9zCNG119pUujBJpp9qD4Bvq+r7IlII4D0Rma+qHyZasKHMe++xjiAvj+uD01Zmzx62pZk7l5c77uBGrbGRCmDkSFMEKcv27d7GdsHIyWG6aqIUgioH5rS3D7Q28vLYImPNGpbNR6vALs1Jmp+sqm4DsM3zf7OIrAIwGoAphASxaxd3/uXltMAdRFifUFDAxpX77QcceKBlA36KKs2qxYvpOystpSmVqKlokSISXjM51cQW2dXWUiGFas/76qvAZz/LokAjKEmjEHwRkWoAMwG84+e+KwBcAQDjouWDNPzy1lt0yfoqA18yMxlHeP55KgQDNJPuuos708xM5t93dABPPgnU1ACXXZb87ovRo7mYOtk/gejqYrVzoli1KnR73qwsfonXrWNfJiMoSacQRKQAwKMArlbVpv73q+q9AO4FgJqamij25jUAJmm8+y5bxjz9NC2BYE0mS0s5iTFUAeqQoLWVvYEaGgZm5qjS/9bRAVxzTXKnXRUVAbNnA2++GThf2JnzkMidQEdH+H5JG/MXFkmlEEQkC1QG/1TVxxItz1Cit5eu1uee4/XcXKafr1vH2eazZvl3B/nOOh/yCmHhQu+4uf44mTXLl3Nnm+wm1TnnUM4tWxgQcnYEqgwUNTUxWJudnTgZKytDt+dV5cV8mWGRJF22ABERAH8BsEpVf51oeYYa//0vM4lGj+a6VVHBdSAvjxbCm2/SHd6fjg4ek+xekJijSm1aURH4GKdke/78+Mm1txQXAz/4ATBjhreJ3MaN3CUUFrLALNEumBkz6BIKphSamhhHCBRnMPqQTBbCbACXAFguIks8t/1AVW2Cb4ypr2ccoLq6rydjwgTWFxQX00Pw0Ud0g/tSVwecdVZye0DiQmcno/ChFh6ns2gqUFLC/uYNDQzg9vYyw6C6OjFDffozbBiDxf/+N9/3/u6j9nZ+JpdemhzypgBJoxBUdQEA+9QSwNtve7sI+FJezgK0Xbu4sd2yhZ4Op85n505uFo85Jv4yx5yeHrp/uru5MI4YEfx43/78wRaf3t6Qfm+3mzHpJUtYFFxZCRx6KD+LhFBamrhBPqE49VQGt596ymuBqXIHk5VFhZZoSyaFSBqFYCSODRv8Z+RlZACHHQYsWkQroqODHQ3y8rhQVVQAV1+dZu7Znh7gpZfo/mlu5pvQ2wscdBALLgL13M/KYtO32trgK3dDA3DyyQHvrqtjktKGDdQbWVl83//zH3ag/cIXrCNDHzIy+LkcdRTT4pz2vAcdRC1qXRUjwhSC8Wlmnj+ys1l3tHs3i1JHjmSM4YgjgAMOSLPFqaeHhU7vvssX6mg6t5vN2G65BfjOdwIPij7lFOD22/k4fz60ri6e6+ij/T68sRG47TZ6OvrrHbebeqqri5mr5gHpR1kZfZfGoEiaoLKROGbO5I4/ECK0CmbMAG66CfjqVzkRMq2UAQC8/jqVwT779C2+yMig36a4GPj977li++PAA6kUamvpsnBwMnM2bWKX0JEj/T78f//jYf5GGGdkUEksWJA6IQgj9TCFYGD6dFrWTQOqPogqg8tz5qShEnDo7eXwlmBD4AsLqTk/+MD//SL06XzlK7y+YYM3Q6ewEPj2t4ETTvD70K4uJh8Fm2efkcH3//XXI3hdhhEB5jIykJMDfOMbrKlyuhE7aecdHVQG06Zx4FUgVBl03raN6+Lo0QE3wslJfT2356Gq3wsK2JIiUP+ejAxG2Y88ktV9HR18zKhRQf08TnfYULUcRUXs1mAYscAUggGAHQh++EMWpy1f7m1il5cHnHsurYNANUgbNwJ//zvd7BkZXPfcbsYYLrkkRRRDT094fXlcLq7c4RwXQe67836HSlJyu61poBE77KtlfEp1NYtPd+5kEDkzkzv9YMWoGzZwjkpmZt/0dKe/209/Ctxwg/+2+klFcTGFDtW/p7U1JkVOxcWcHNncTCsgEI2NDFMYRiywGIIxgLIyzlOeMCG4MlAF7ruPx5SX993ZinCBc7uBBx+MvcyDpqCAbqAdOwIf43ZTYcyeHfWnFwFOP52eq0CNRtvbaXhEOrnSMMLFFIKx16xbx5hpsJqtigpWOG/ZEj+59pozzmDUdvfugfe53cweOu64mJk7RxzBxX79+r6JTKosDty+nfHq4uKYPL1hmMvI2Hs2buTfYD5v575Nm1JgHEBlJfC973FQdG0tS7IzM5lZ5HZzHOOFF8bs6V0ujiOtrmYrkfp6bzymupr3TZ0as6c3DFMIxkDcbu5Sm5roDpo40f9YWn+uDbeb1bZbtrDrQ2Ghd1FLCcaPZ3XYhx9621WPGcOS7WA5oVEiM5Ouo5NPpsLt6gKGD2dg3orRjFhjCsH4FFXWZT3yCDssOAuQ2801MS+Pt02cyE4B/bOHmprYF6mtjQuby0WXfHMzq2wPPjhF5p1nZbE4Y/r0hIowcWLCnt4YophCMD7lpZeYPlpe7k2k2bCBdVgLFlApTJ8OrF3LVj8nn8yeZ3v2cAFbsICP6e/jHjWKj/nzn4GrrrKdrmEkKxZUNgDQzfPQQ1z0Cwt527ZtwPvv8/ro0QxsNjdziNbYsVQK48czFXLFCrqIfDs+dHTwthkz6AN/7z0qBsMwkhNTCAYA7u5FvGmmqlzk8/Pp/hHh/598wvucuqtly9hsbfNmZmQ2NnovLhcLdkeM4ONzcoDXXkvgi4wmqizY2LaNtQnJQlMTte769dTIhhEB5jIyALD3vq+rp7GR65zvbTk5XG+cFguZmVQCu3cDhxzCGIPTD6m4mO4kX/dQQQGzjVIat5uBknnzqAyc0uzPfIa9+RNVll1fDzz5JMd4OmRmsnfS6af7729uGP0whWAA4Drn27mho2Ogr9+3CtkhM5PKA2DsIdgEyd7exI7gHTRuN/C3vwEvv8wXO24c35SeHi7E77wDXHdd/KPB27axXLy9nQEbp/V2Vxf9esuXM53WZgMYITCXkQGAa9iePd7r/tr5d3dzQfdd1N1uWgLjxvV9vD/27EnxKtuFC6kMJkxgfwnfNCwRmko/+lHf1texRhW45x5q2zFj+n5w2dn0623ZAjz8cPxkMlKWpFIIIvJXEakTkRWJlmWoceyxHAvs1As4s2F276Y3oqGBQeUJE7yu84YGrkMHHkivREND4HqD1lauT4ceGpeXE31Ugaef7tsKtqeHQZTnn2e+7scf00r48pc53CBQD4posm4dU8HKywMfM2oU8OabgfubG4aHiF1GIjIMQIeqBpixNSgeAPAHAKnQ/SatqK6mUnjlFe72N2/mjr6xkSmlPT287NrFXkfZ2d4xmlu2ALNm0V39v//xNidTye2mAmlvB665xnt7ylFfz1SssWN5vbeXSmDHDlaOOUoiO5ta9IEHuADPnRtbuT76iNZJsFzezExvtWECayuM5CekQhCRDAAXALgIwKEAOgHkiEg9gGcB3Kuqn0RDGFV9XUSqo3EuIzJE2Ko6Px+4/35mEw0bxlhCe7s3A0mESsGZuz5tGnD33UxH/eIXOWxs3jxW2YpwkzxtGnD22bwvZenu7rvwbt7M5kIlJX0XYyfIPH48g7yHHkpXTqzo6vLv3+uPSOA5qYbhIRwL4RUALwH4PoAVquoGABEZAeA4ALeJyOOq+o/YiWnEg8xM4LTTgGee4WLvjAjYts07IwHg7Tk5nAOTk8OMooceYr3B0Ucz1XT7dm/bBcf9lNIMH86/vb18IxyN2X9n3tVFMygzk5c33uAUtVgxciQ/kGCoeoM9hhGEcBTCiara3f9GVd0F4FEAj4pI3AYrisgVAK4AgHGhplsZEbN4MTeczm5elZvhSZN4u1OD0NLidZFnZ3ONXLAAOOcc/p/08w8ipaCA/YwWL6bPrKXFqyQcnIXX+V6WlFCTxlIhHHQQtXJnZ+Bxa42NdHXZ78UIQcigsqp2i8j+InKCiPTJWxOROc4xsRLQjzz3qmqNqtaUBwukGXvFunWsJ3Do6uLal5PDWEJ2NhWCCGMMDQ2MEeTksJYhrTn9dC76TU0D/faqXHhHj+6rKGIdWM7LAz7/eWptf5PcWlv5QV14ofUMMUISTgzhmwCuBLAKwF9E5Fuq+qTn7p8BeD6G8hlxJiOjb6aQs575riW9vYwjvPEGlQRAF/vIkSw8c+Kuacfo0cB3vwv87ndcfPfsoSbs6uKbNno0MHOm981qbKT/LNYcfzw/lP/8h3Lk5/ODa2tjN8GrrwamTIm9HEbKE47L6HIAh6hqiyfg+18RqVbV3wGI6pZDRB4CcCyAMhHZDODHqvqXaD6HEZypU/u2l3DqDrq7ufj39jJg3NJCt5BTk7BnD4+55RbWQKVtp8799gPuuAO49152Ahw2jJpw/HhaBo4y6O2lojj22NjLJMJOg4cfDixaBKxZQzNu6lQqqJRoMWskA+EoBJeqtgCAqtaKyLGgUhiPKCsEVY2hs9UIh+nTuca1tNBtnpEB7LsvsHIlg8cNDcwoqqzs2/fI7WZQGQD+8Afg9tvTeBh8Xh6n1TQ308c2ZkzfMu/ubmrNOXPi67cvKmLu7wknxO85jbQinMK07SIyw7niUQ5nACgDMC1GchkJIicH+NrXvIVoqtz8FhXx+o4dVBhlZTze7aZnZMIEbpCLi3l95coEvoh4kJvLwopZs+gnq61lgVhtLWsWPvc5BpPNb2+kEKIhgl4iMgZAj6pu93PfbFV9M1bChaKmpkYXL16cqKdPaz7+GPjXv7i+ZWTQ+7FsGRVCZaW31kmEFsSUKd5N8tatTEm96KKEvoT4oMrisJUr+QaMGUNXjW8fcMNIMkTkPVWt6X97SKNeVTcHuS9hysCILfvtB/z4x0xe2bmTLumuLrrPMzPpFcnLA6qqBrqoMzJ4f1rjjJd7+mmWartc1JDTpzPfP6Wr8IyhSrp6eY0oIOIdhgNQMeTl8XpGEGdjZ6d34lpaosqMnmeeoe/M6XrqdgOrVzP/9lvf8gZVDCNFSKrmdkZyU1bGNhT19YGP6e6msqgZYIymEcuWAc8+ywZQvplFGRn0p5WXA3fd5e0LHglud3ya4hmGH8K2EEREwH5G+6jqzSIyDkCVqr4bM+mMpOO884Cf/pRrXXEx16/6emY61tczO+nooxl7mDo1uCWRsjz7LBVBoB5C+fl8M955BzjllNDna27m0J0XXvBW+X3mM6wvSNuijhiiykaEXV3MhuhfUW4EJGRQ+dMDRf4IwA3geFWdIiIlAF5U1YQ1NLagcmJYu5appQ0NzLqsr+cmOTeXweWSEiqG2bM5XjOt0k/b2oCvf50+sWAZRE1NXIx+8pPg59u+HfjlL9khtayMyqSnhwtaTw9baR99dFRfQtqiSsX69NNswOVysR5kxgzgrLOYCmcAGERQ2YfDVPVgEfkAAFR1t4ik8vwrYy+ZOJF1Br/6FTsq778/rYVRo7ztdMrK2IK/rAw499yEihtduru9vTuCkZnJYEowurqAX/+aLWV9gy5ZWax67uwE/vIXFr5NmjR42dMZVabFPf984LjONdew95MRkEgUQreIuAAoAIhIOWgxGGnExo3Aq68ygaa7m+vSKaew4NV3UlpHBxt+HnOMfwsgI4Pejuef56jhtMnCHDaMb0SwZnIA3UBTpwY/17JltASqq/3fn5PDN+7ZZxmkHiz19ZQrJ4dKJp38ee+/zy9bdXVfV54T12ltZVzn9ttpuRl+iUQh3AngcQAVInIrgPMA3BgTqYyE8OKL3GRlZnLITUYG3UJ3380symuu8f6WPvyQ1ngwd5AzWGfVKuCQQ+LzGmJOZiZw4onMMApUhez0ETr++ODneuON0HOOy8uBpUu9peN7w+rVwGOPUYM7zaoqKoAzz6RfL9WL51SpNEtKAsd1hg1jfObdd/n5GX4Ja4vgCSi/DuA6AD8HsA3AZ1X1kRjKZsSRJUvYmmfUKNZWZWdz7Sspoet10ybgj3/0JsC0tfH/nh5aC8Fa8re1xeUlxI/jjvMuMP1RpZl14IHA5MnBz+M0xwuGM3CnvX3vZF24EPj5z+lTHzeOZtv48TT//vQnDrJI9aymlhYGs0IN3iguZozBCEhYFoKqqog8oaqHAFgdY5mMOKMKPP44MGJEX7eQL6NH0ypYv57Wwp49rGZeupT3i1CRTJw4MKkjbdxFDiNGANdfD/z2t0ynys+nOdTWxoV22jT2/wg1yaykhG6c/nNFe3rokhKhwlDduzdx507gvvvoHupfPVhURKX2/PNUXqnsW+/pCT1GFODnESquM8SJxGX0togcqqqLYiaNkRB27OCmNlgfNmeE5rvvcp155BHvfGWnjcWWLbQkZs3iGuR0SE3LzsujRwM/+xlbVrz9NneplZV0wUyYEJ4b5phjgA8+oFsIoEJZu5ZKxrcH+bHH7l3H0jffpDIJ9FiXi4rhuedSWyEUFPDL2dUVeEcD8DPad9/4yZWCRKIQjgPw/0RkA4BWsNOpqmoKf5MMgPE2xzMRjJwcWubz53M9dALLxcV8fGEhlcCiRVzDGhqY7Zd2FoJDVhZTGve2InnqVProtm/nbn3BAi5qBQVcrLu7WfCxfj3bbV9xRXjzkx3eeYfWTDBKSxlj6OhI3TbZWVl0473wQvC4Tns7jzMCEkmawakAJgI4HsCZYMfTM2MhlBFf8vNDF8i2t7Np3ZIl3rkrU6bQTdTYSKXidnO9am/npMkjjwTOPjteryIFycpipL6wkBH9zk7+r0qfXFsbcMQR7I+0cCHw0kuRnb+rK7QC8Z3fkMqceCK/yA0NA+9z4jrTp1v6bgjCthBUdUMsBTESR1UVd/xNTQP9/x0dwIoVbHLnzFHOz+emdto0Zg+NG0dPhxNjHTmSl8suS6/MxphQXg5ccAFdTy0t1KyZmXRtjB/vzSyqqmImzQknhF/pN3YsO7H6zkTtT3s7nyPYMalAaSnjOr/5jf+4zowZtLDsCxmUSFpX/Mjf7ap6c/TEMRKBCHfyd95Jz4Wz3nR0MDOyvd3b2bmxkW7n1lbeN3s2XeeVld7zud20Juy3FyYffsg3d9SowMfk51Pjbt4cuG6hPyeeyPx81cD+wLo6Vg6mw4c1Zgxw223cwbzzDr+kTlynujr102vjQCQxhFaf/3NBl9Gq6IpjxBpnrsH8+cxEdNrmzJ7NWe3//S/dQRUV3LQ2NlJBDB/OCY2vv86kjvx8ejgWL+b0Rt/1pLXVO0DHCIO2Nu9w6mBE2ld8//1pxq1c6a3c9WXHDsYY0qk1RlYWqyhnzky0JClJJC6jX/leF5E7ADwVdYmMT3FcyT093JUHS6AIh6YmWtTr1nGBLyjguZ9+Gpg3D7j8cuCHPwT+9z8mqKxeTcWw337cvGZlMeV0xQrKkpNDhVFXR4+GQ0MD8MUvDk7WIcXIkUzfCoYq/fzFxfx/3Tq+8S4Xd78VFQMf43Kx79Kf/8xspsxMavvubmrzMWOAb37TKneNTxlM27F8AFGdAiIicwD8DoALwH2qels0z58quN3M1Jk3j6mcABfg44+nFyBU4kigc/7+90wL9e3xlZ3NzWN7OwvPfvhDjgs+8kgOw+k/12DsWHY2bWujleBUMzsKob6e8s2a1fdxe/bQtdvbS+th7Fiz4D/lsMNYCOJ2B3bdNDRwx19fz/5H27b1fQOnTwcuuYQL/vr11PQjRvCN/uY3+cEvXEi3U0EBP6D99ossa8lIeyKJISyHp48RuGCXA7glWoJ4+iTdBeAkAJsBLBKRp1T1w2g9RyrgdgP3389+QqWl3oWzq4s1RAsWMHYWzN3sj08+4SXQ4Jq8PMYPnn6aiS+q/tcmx8W0cCGtA6dSedcutskpLQW+/W1vLLS5GXj4YVocvq9x/HjgwgtDF/MOCSor6bZ55RXu9vu/8a2tDDhPncrOqMXFfbutut30A158MbVtVhbv6+3lcRdcwJSwYIUmhoHILIQzfP7vAbBDVYM0LIiYWQDWqOo6ABCRfwM4G8CQUgivvUZlMGFC33XB2cnv3An87nesiYpkc7dgARfzYLvysjJg+XLu5isruc7427QWFTHZZds2eiKcTqfHHceNqpPO3tLCGN+2bcxicuRVZbfn224Drr2Wbu4hz8UXcwFfsICunaIiXm9q4od/1VXA3/7GD6l/T6PeXloFjjl52GH8q0qt/YtfAFdfbRPcjJBEklrwdVXd4LlsUdUeEflFFGUZDWCTz/XNntv6ICJXiMhiEVlcH2x0VwrS28sdelVVYM9BWRldx6siDOc74y+DkZHBS0sLsyGnTeNz+SMzk8cceihd1Ndfz3XIt7bpqaeYbTRuXF/lJUJvRmkpcM891k0AAHf1l10G3HwzK5hLS6lFL7mEvruMDJpb/hrcrVlDE62qivnAHR28XYTtMcrL+UbvbT8kY8gQiUI4yc9tp0ZLELDyuT8DSqVU9V5VrVHVmnKn5D9N2LqVG7phw4Ifl53N7J5IKCqi2ykYTtzSWdTPP59//dX6tLdzQ3rxxf4LXNvaaOkEc20VFNAb4vRDGvKIUHtecglwww3AddcxcFRYSH+fv6yC3l4GmAsLvbuI5ua+xwwbRiXx3nuxfw1GShNSIYjI1zzxg8kissznsh7A8ijKshmA77zAMQC2RvH8SU9XV3jp4FlZXEgj4TOfCd11dPduNqdzgtajRwPf/z4X7tpaFntu3sz/m5uBr36VwWd/bN3KtSpUNmVuLtPwjRD49jbypaWFgRzfYjV/JefDhtEfaBhBCCeG8C8Az4Ftr6/3ub1ZVXdFUZZFACaJyAQAWwBcAODCKJ4/6Rk+nItosDoigLvzkSMjO/fUqfQo1NX5z1B02uZcdlnf5x4/Hrj1Vm9QureXu/6DDgreudlpjb1xI93gLhc9F6Wlfc/vDLUyQrDPPuzV0x/fL4ujCPp3TwXsjTbCIqRCUNU9APYA+IJnjvIksDANIgJVfT0agnhiElcBeAHMYvqrqq6MxrlThbIyJoNs2OBtgNkfVf6uDz88snNnZjKu+Mtfes/v9DCqq6OSOf98/00vMzKYDRRuRpAqN6Nvv00LwOmG+tFHtDYOPdTbIqOjw0bdhoUTrXfyfR3y871fitZW7hT8BYva2qhUDCMIYccQROQr4JCcFwDc5Pn7k2gKo6rPqup+qjpRVW+N5rlThXPO4e/an3tHle6aI46IPO0UoIXwk5+wIrmzk4ph82YGj2+4ATj99OjUBrzwAtPqJ0ygZVBYSAVQXEy32Btv0OXk9F479NDBP2fak5sLXHopg8a+/sLsbOYmO0VqBxww8LFOdXOkuwhjyCEa5rQkTxzhUABvq+oMEdkfwE2qen4sBQxGTU2NLo40upoCvP8+k0K6u7mQulzeBXTWLLp1Qg3aCoUqz5eZGd3apKYm1jFUVvL8r7/ubYjnKBunXqGqCvjSl5jCaoTJ4sUcbdfU5HURdXQw7XTKlL71CQA/hE2buAs405oTG0RE3lPVmv63R1KH0KGqHSICEclR1dUiYmVFMeDgg5lp+M473mH3Bx7IbMRwZ6+EwhnGFW0WL6b3IjublyOP5G179vD+jAzGITZsYGp9qLHDRj9qalhPsGoVexG5XPxSZGdzF7FxIz9cZ55CVhYrAOfMSbTkRgoQiULYLCLFAJ4AMF9EdmOIZQHFk6Ii4KSTeEkl1q/v68IuKmLB2u7drIVwuxlH6O7mhtbaV0RAczMb1e3ZwzexpoZ+OIebbmIf8lWr6BOsqmKTt1B5zIbhIZLmdnM9//5ERF4BMBzA8zGRykgYTlp7UxMtiIkTI2uV73INTGZxCtF8ezBt3JgeHZfjQnc38NhjbFHrzA92eoscfTTwhS94y9D33dfGRBp7TSS9jATARQD2UdWbRWQcgBkAQrRpNFIBVWYF/fe/LHp1du5ZWbRSzj47vA7NBx7IgrRgdHXxXHsTGB9yuN3AX//KZlDjxvWtN+jtZa+TXbvYwC7cwTmGEYBIvkF3A3CDIzRvBtAM4FEw0GykOPPnA//4B2sUfBvgdXWxncaWLcCVV4Zecw46iN6M5uaB6fBODHTrVuDUU1N/SFfUaWykVl6zhhp56lS6e95803/TO5eLSmLJEjaVsnQtY5BEohAOU9WDReQDAFDV3SIyyA79RjKwYwfw0ENsj9+/O0J2Ntei995jkHv27ODnys6m4rj9drqxi4uZKem022lv5/mqq4N3ex5SqHKm8n/+w/8LCvh38WIOpRg5MvAb5fQreu45UwjGoInk59jtaVGtACAi5aDFYKQ4CxZwvQk0gEeEaaLPPOO/K0J/pkxhXUNZGVt2v/qqN01+6lQmxdx1F9t8p/ps96jw6qs0z6qqaJ6VlvLNGz+eKaWffMKWsYEoLmY03yqRjUESiYVwJ4DHAVSIyK0AzgNwY0ykMuLK0qXcZAajqIjp7C0t/jsj9GfiRMYIJk7kuV0u1lQ4Ssftpvu7qooFcUOWzk5aBqNH+9fIWVm8LF/ON8vSsowYEk5zu797/i0DcB3Y02gbgM+q6iMxlM2IE2536HWmf7ucUOzcySE6kydzHSsv77veZWRwDXzmmb7tr5ubWY29aVNk44NTlhUraAX4axkL8I1zu1m6vitA67Bdu/hGm//NGCThWAiHiMh4AJcCeBDAQ84dIjIiyg3ujASw777AW28FT1dva6OVEG5K+/LlgaeuOeTkcC1cs4YekqeeYkzVyarMywNOOQU4+eTYFNElBXV1wd+kffZhfxHAO+fAF2fw9mWXxUY+Y0gRjkK4B6w32AeAb0N1AeMJQ6ZjVkcHg6I5OX37i6U6xx5LN3awIO+OHSx4DbfNhdPhNBzWrwfuvpsZTaNGeR/X0cE02JUr2Zgv0CY6pcnODu77HzGCSmHZMtYg+NLVxfSvI46wsXNGVAin2+mdAO4UkT+q6tfiIFPSsXkzG7YtXMjfrioLQE89FZg0KdHSDZ7x46kUXn55YKq7KtNEx44NPPvAH04r71CoAk88QWtgdL/5eLm5zEZatQqYNw8477zwnz9l2G8//g3U89xJP21p4f8bNnjvy8oCzjqLRSLRbEhlDFlCKgQRESUBlYFzTHRFSw6WLwd++1v+3qqqvK2cV61iKuaXv8zFNJUR4ZCu/Hwqvt5evk5nNsO0acBXvhJZB4Tp071z3gOtVe3t3o6nY8YElm30aNZJnHFGGloJY8bQ/79hA79g/ti2jc3pvvhF9hBvaeGHNXly+hZzuN3Axx8zoOR2c0cyZYoV38WYcN7dV0TkUQBPqupG50ZPDcKRAL4E4BUAD8REwgSyaxfw+9/TavddDDMy2M2zs5Nzz8ePT/2e/i4X15w5c1jntHMn0+GnTh24cw+HkhKv1VFdPXDz29tLy6O6OvDcZofsbAaYN270bqjTBhHg8suBn/2ML3DkSG9JeGcnlcHYsd72FP4GVqQba9YA997LL4aId7hPcTFbgA+F9yBBhKMQ5oAB5Yc808wawQE5LgAvAviNqi6JlYCJ5K236LYNtDPOyeFi9dJL/E33x+3motfZyVRNf5PKko2iIrbHiQZf+AJjCYsW8fWXlPA9aWhgfOC00/je+JvZ3B+RgS70tKGsDPjhD5ly9dprXr9kdjZbVs+ZM3Qa1K1fD9x2G19vdXXf+5qbgV//Gvj2ty1mEiPCiSF0gG0r7haRLDD9tF1VG2MsW8J54w3WCAWjooKZMZde6nWNuN3sNvD009xpOy2fJ03iAJz994+97MlAdjbw9a9zZvKLL3Ljl5HB+MsJJzC76ZVXODMhGM5AMN/meGlHSQkwdy6/UB9+SNfI4Yez1fVQcZOoAg8+SDeYv8IYpwDmr39lf3iLm0SdiL5pqtoN1iAMCTo6Qm/MnO6ePT38XxX45z+5AFZUMEgL8PZt24Cf/xz42teGzvAql4ubuUAbukMOYZFu/znxvjQ00FUUyMWe8qjSt/bvf9M3lpPjbV1RXMxeIOmQvRCKTZtoIfg20+pPYSHjLatX059pRJWkqGQRkc+JyEoRcYvIgCk+iaK8vO+0Qn90dDC+5xRdLV3KAOiECX0rep32D1VVwH33hecmGQoMH85EmQ0b/LuEWlr4GXz+8/GXLW688gr7eJSV0U0yciTzb53gyy9+weBqurNtmzdmEIotW2IvzxAkKRQCgBUAzgFnNicNJ57onfQViLo6Fk45xVTPPMNFLlA+f14eLYqFC6Mvb6py1ln0lmzZwrhqQwNQX881sK2NLuOJExMtZYxoa6NlMHas/+q74cN5+3/+E3/Z4k24bTlCVTwae81eOSdFpEpVt0dLCFVd5TlvtE4ZFWbO5EZt+3b/7opdu2gdOEHY9nb6yR03USBKStg59Iwzoi9zKpKRwdjKMcdQUdbWMtHmoIP4GaRdqqkv77/vdRMFoqyMec6BvojpwrhxXOwD1WQA3t4pwdxKxl6zt9GqZwEcHE1BkpGcHO5Of/1rLlIFBdzhd3XRciguBr7zHW/8yxlmFUqvuVw8h9GX0tIhqCQ3bw49eciZkbxzZ/gKYds2lp8vWMCdSlkZTdnDD+cXORmpquKEpTVrAr/O3bu5S7OpcDFhbxVCxFt5EXkJgL9P+QZVfTKC81wB4AoAGBdqKx4FSkuBn/yEPcheeYVWQWkpcP753L361gUNG0aLIVivMoDZc5Y1ZwDwVjqGIhI3yTvvAH/6E48vL+cXtq2N0ftnngGuu45ximTkkkuAn/6UCs23u6sqFWJ3N3Dttdb1NUbsrUL4c6QPUNUT9/K5+p/nXgD3AkBNTU1cqqOzsrj4z5wZ/DiXi5uwxx8PbNGq8rd5/PHRl9NIQaZMYV+OYPT0cHEPZwO0fj1wzz1McfPdrRQU8FJfD/zqV8CttyZnx8DKSuDGG6m8VqzgbU6Abt99qTDisBEcquytQngsqlKkEcccQ0t9xw5+t31RZTbNjBlpWHFrDKCnhyUFL71E939uLvvQHXEE3Y0AWJRSVkZXSKChFFu3suw7HFfPCy8w5S1QS4vycvo/ly4FZs2K/EXFg8pK+mp37KBLTdWbeWWWQUzZ21D9s9EUQkTmishmAEcAeEZEXojm+ePJ8OHA9dfTSq+tpeW7cye/1xs2ADU1wFe/akkS6U5zMwtuf/UrYO1armOtrcDDD9Njs2yZ50CXi3UGHR1cAH3dR93d/NKMGsWoeyg6OoB33w1dEl9UxF1LslNZyUKVmhr2TzFlEHPiFkMIhqo+Dk5jSwsqKoCbb2YfsnffZS59ZSXjefa9Tn/cbvbAWr9+YB+nwkIqht/+Fvjxjz2uxQkT6CZ55BF2U3R2CxkZ9C3OnRueddDe7g1AByMnB2hs3LsXZ6Q1cYshDDUyMugenjIl0ZIY8eaTFZ34+M09GN+7HrKxhzvyceP4VwTDhtGCeOYZtvYAwDqEa6+lj7++nl+gsWMj62GUl0f3SrAWswAbSKVCYy0j7uyVQlDVu6MtiGGkBZ98gtevXoTcLSMhw9u5sDc00G80ZgwzE1wulJezM0VLS7/Nf3k5L3tDbi5w2GGsbQiWRdTUxGBXuLS0UEmJMPMnrQtDhjZDpGuWYcSBLVuAX/4S9Z2fRV5JLpDr8RXl5nLnvmkTrx9yCFwugYgfhTBYTjmF3Rbb2/0HluvqqHCmTw99rp07Odf0rbe8BWEuF/sqjR7N1zVhAmsHQtVSGCmBKQTDiBZPP02XUKFg665+LhsRphZt3gxMmgQtGg63OwaZn9XV9EPdcw+vV1RwsW5ro6VSUkLXVKgn3rGDMxpaW72ToerqGBR7+WVqMWcKUkEBJyjNmBHlF2PEm3AmpoXTdNg9FNphG0ZAmpu5WI4ejc/oZnywfSTK+s/dFqELadMm7B41HBMn+qSfRpNDD6V76rXXvJXKpaXAxRfTpRTKJFFlYVtXl3eU3c6dtBTy8njbnj1MlT34YCqN3/yGqaI2vCalCcdC2Oq5BMuNcQGwahFj6LJnz6cZPtOrdmBEXjsa2vJQmt/e97isLPQ0NqMxnzM0YpZxNnIkcMEFvERKbS2wbp23ulKVY/Ryc72WRWEhXWAHHMDAd1kZ8MADwO23Bw5o797NmgqnriDUsBEj7oSjEFapatAaXRH5IEryGEZqkpn5qZ8929WLaw5/G794czY27ilC5bBW5GT2wq3AztZ8tGgJzrkqPDd+Qvj4Y/51tNXu3Qx2DB/uPSYjg693924u7s6cgo8+opLwZedOptS++673caoMsJ9//sAKTiNhhKMQjojSMYaRvlRUcJfc3AwUFmLs8CbcfNyreLW2GvPXTkBnbxYUwLS81ZhzbQmmnJ3E9ShdXX13+c5QkP4COy0lHFRZku2rEOrr2SajtZWuJt+xgitWUPnccEPy9lYaYoQ7QtMvIvJlVb0/2DGG0Z/OTnZOWLKExbjjxrGdQ1lZoiUbBBkZwOmnc/rRsGFARgZG5LXjnCmrcPbk1WjrzkJWy27kSicw95wol3ZGmaoq1jI4+NNczi6/fyZTf3fRgw8yhjF6dN/bMzJYgV1Xx/fsxhuTWEMOHQabZXQTgPujIUi86OmhleskfQyVcbXJwurVwB/+QA9Efj7XhffeAx59FDj1VMZD162joqis5JTERPdga2qit2PhQq5tI0eygHj//futf0ceCXzyCYO5FRWfjsxzaQ8KGzwTvr73vcS/oFBMm0YZOzv513EV+c4p6Ozk63Oi4o6lUF3tPc/27ay8Dja7oLycNRobN9qMgyQgnCyjZYHuApAyzr+2NmbLvfAC/weYbDFnDvuGBeoFZkSP2lrGHIcP77tuAFx0f/5zri/jx3tnVefnMy561FGJ2UCuXMk2FJ2d3g3EqlUsKps8GfjmN32Sdlwu4MtfZufCZ56hT90ZtH344bQgRo2K/4uIlNxc+vbvv5/V0oWFXLh37eL/3d3UjDNnej+U+npgn336diKtrQ09IMS5r7bWFEISEM7+uBLAKQB297tdALwVdYliQHMz8MtfchNSVeVNbmhrAx56CFi0iBlzkXQJSEdaWrjQLV7sdeUcdRTXhGgsxo8+yg1nUVHf29vamNGYkcEWO4cd5lXQHR30KPT0xL9l+KZNHI5UUtJ3XsuwYVzj164F7roL+O53fZoVulwcoXfkkVwke3qoAZNlKI3jr3vrLW+NwVFHsbW0b8fF447jl+Dhh6mZx47ljn/rViqFww6jCed20+3jcrEWwfeLEs6cBwdfF5WRMMJRCPMAFKjqkv53iMir0RYoFjz4ILuOTpjQ9/b8fG5qamupGL7ylYSIlxQsWwbcfbfXE+By0XUzfz7Xti99aXDFqDt3MoY4duzA+z76iHHMoiIqhC1bvAOxcnPpfv7Xv+hO8nhh4sIzz9Ai8LeWizBGumoVB3wNaGeekZF82TMbNrBeoLGRLyori7ukN95g060rr/S+WBFWPR9+OP1la9fSf7d5Mz/Mnh4+1u1mutQFFwwMDFdU9A06B0Ik+d6rIUo4QeXLgtx3YXTFiT719dzx+luIHEaP5obpvPNiVCiU5Kxdy3WirKzvTnj4cP7eX3/d6w3pT2sr3T2ZmXx8IEti1y6ukf3bfnd2cifurENZWTyfLzk53EAuWhQ/K6G1lc8XzMMjQtlefz0F5lvU19NMdrn6+uuGD+ei/dFHDO5897t9AyPDhwMnncSLQ0MDtTZAJRCo99I++/ALtWdP35RVX5x01v33H9TLM6JD2odUP/449PRBx8378cfJOzMkljz+OF00/lxmGRlcP15/nUFfR2Fs3Qo8+ywDrQDfv6oqzkQ+/PCB77dPmn4fnIxG53i323+gPz+fO/F4KYTmZv4N1Ul62DB2eUh6/vc/+t/87YxEePuqVf7rCPpTWhpeUVlGBvDFL3qL1fqbWm1tVFRXXx36jTbiQsgxLSLyfjSOSRSdneEdpzo0B9/v3MnAabAGm87O/p13eH3tWuCmm+hJGDWKa8nYsVxv/vhHuuj6u4/HjKH7p6NfgnL/VPbeXv/eg0hGCkeDnBy+hlAej66uFIg9dXdzIHgwt4wIte7LL0f3uQ84gL2TOjvpm92yhbuJ2lpaB9/4RujZtEbcCMdCmBIk0whgcDmAPZh4Ak0l7I+ThjrUaGzkQhsqaJyby99xRweHu+Tn9/UCiDAGUFDAzeikScDs2d77s7M5b/qxxxjLcZ6voIDP39vLxTUvz79yam+P72yJ4mJ6PHbuDP4damkBPvOZuIm1d7S2UilkZwc/rqCAwbZoM20ao/MrVjAtF+CX4KCDkj8Fd4gRjkIIx7mXtCkCBxzARcZJqfZHezt/C5Mnx1e2ZCA7O7y4X08PlcDSpXSn9E8bdcjI4II+bx4XSl9Fc+qpXA+WL6frOS+PMYNx42ilFBWxQK2/JdDWxs/u4IP3+mVGjAizRH/7Wway/bmxdu+m4kjaFhQOWVlecyeY5u/pGZgCFk0ZZs40ayDJCWmEq+oG3wuAywBcAWAWgGzP7ZtjLejekpMDnHsukyN6egbe393Nne/nPjc0W7qPGsU1wKnNCERXFxfkt98OnUFZWMgMxfr6vrfn5ADf+hbw+c/z+TZu9AaU99mHgdn+WUR79tBH/5WvxL9WZOZM4LOfpZw7d3oVZ1cX5e7pAa65JgU2ucOGMWi7a1fw4xobU8DcMWJJxEFlVf2RiFQCmAngXBGZqKqXD0YIEbkdwJkAugCsBfDlaLbTPuEELkCPP87dp7MJ2rOHfy+6iKnYiaK1lbtNl4uZevGMr2Vmcif84INclP1tIBsamEE0ZQoDyaGqu50uz93dA+/Lzmbg+ZRT6J3o7fXGJx96iHEKX4ulshL4znfodYg3IhxnPGkSX/eqVXxdWVl0f51wQgq12zjtNOCOO2jSOF+wzk5qt8xMfhA5OcztNYYsouH4CwCIyG8BXKPhPiASIUROBvCyqvaIyC8AQFW/F+pxNTU1unjx4rCfp66O7eFXr+b1qVPp507Uj7qujrnub77J6243ldVpp7EuKF4WS28vcO+9TL316biA7m7u9PPygOuvZ2D4H/9gZ4b+rWn6n2/rVuB3v4u8Hmv3btY/OIpiwoT4BpOD0dbmDSKnnDWpygDOk09S023fzi8gwBdWWMh+Qp/9bMxE6O7mU2VnW2eARCMi76lqTf/bI7EQWgA8JSIXqGqrZxH/sarODvXAUKjqiz5X3wZw3mDP6Y+KCuCcc2Jx5sjZvJmtGjo7vQOpAFoL//gH429XXRU6DhgNXC7giisYb3n2WbpInF3+ccdxN+8Eeo88EnjppeDu6B07wpvD4o+SEuCQQ/b+tcSS/HxeUhIRfvlbWhgYaWvjqizCKsDycpaS5+ayn0sU2bmT35lXXqFSUOVm7NRT+Z2znnbJQ9gKQVVvFJELAbwqIp0AWgFcHwOZLgXwn0B3isgVYAwD43z7pqQQvb3AnXdywXUGUjkMG8Zd8dKl7Lt05pnxkcnl4tz1o46iq9npuNB/Jzd+PFBTw6Kt6uqBP2anqOz00+MithEJ9fUsKDnpJH5wvb3ccTi7ju5u4N//po9s4sSoPGVtLevhOjvp/svOpiW8fj3wi1+wGPTMM00pJAthKwQROQHA5aAiGAngMlX9KILHvwSgys9dN6jqk55jbgDQA+Cfgc6jqvcCuBegyyjc508mVq/mbzNQLy8RZuE8/zx35/GwEhwyMoK70ES8LT4WL6brZNgwri3NzbQKvvvd4C4lI0EsWMC/gcycrCzGEV56KSoKoaODFfBZWbTOHZxMtJIS4L//5QYoETEiYyCRuIxuAPBDVV0gItMA/EdErlXVsCpZVPXEYPeLyJcAnAHghFjEKZKJpUtD+6Bzc+ni3bQpapu1qJGby7Y3GzawDc6WLbztsMM4Z938w0nKm2+GrjCuqGBk//LLBx28WbKEiRuBUpQzM2mFzpuX5gqhqYltEJyGXZMnJ20QKhKX0fE+/y8XkVMBPApg0HlqIjIHwPcAHKOqIRIgU5+OjvAyiUT8p8omAyL8oQf6sRtJSGdn4J5CDk7f8Z6eQZumCxaELmsYMYK1KcHaHaUsHR3sFvvaa31L9wsKmAt/zDFJ5yvb615GqrrN40aKBn8AkANgvvANeltVvxqlcycdo0aFbqmhyu/QUKyeNmJEVRV9lcFKr9va+KWLwg62tTX0aZxxCWnXNqari8H71asZKPTN1W5vB/76V/pY4xUkDJNB2YSq2h4NIVR1X1Udq6ozPJe0VQaAt4FesBbwDQ2sJbKuwEbUOPlkFp8Fo76egaso7FwrKkIXPPb28qmSvh9UpLz5Jsvvx48fWLiTl8fmX489xvTfJCJJMryHFiNG8Le5YYN/pdDSwt3VeTFJvjWGLNOnc4FyWlf3p66O1sPsQWeSA6BHpK0teGsUJ0U5ZdN5/eF2s8CosjKwYs3KYozmjTfiK1sITCEkiM99junemzbxsns387U3bKAyuPba5AsmGylOTg6/WGPHMh90+3Y67+vqeL2wELjuuqj1M5o8me1ItmzxrxRaWrh2nnpqVJ4ueWhupokfqhCnuJgFR0lE2s9DSFZcLuDCC9nf/623+HvMzGS/oIMPTrMdk5E8FBcDN9zArJc33uAuZPhwWgVTp0Y1+8XlYnfrO+9k4LiggK6h7m56rnJy2Auqfy1OyhOqiaBD/97vSYAphARTVZU81dPGEMHlYmOqOPQTLypi25NVq9gWfft26qTTT2csLVbNVRNKYSGVbGtr8OBIY2PSleWbQjAMI6ZkZrLOIK1rDXxxuegH++c/Bw5yd+jtZWrv0UfHV7YQWAzBMAwj2hx1FJXBxo0Dxwd2ddFHfNppwYd2JwCzEAzDMKJNXh77tj/4IBt/ud3euEJODnDBBbQi0qUwzTAMwwhCQQHw9a8zcL96NSuXi4sZvE/S/i6mEAzDMGJJWRn7xqcAFkMwDMMwAJhCMAzDMDyYQjAMwzAAWAzBSFHcbmDtWtb2ZGayzUdaFjkZRhwxhWCkHO+/z0mP9fW87rRQPvJI4Pzzre2HYewtphCMlOLNN4E//YkjGH1HkPb0cFzwpk0c4ZmkWX2GkdRYDMFIGZqagPvvZ3FnYWHf+zIzqSDWrmXPHMMwIscsBCNlWLSILWBycwMfM3Ik8PzznPGSpGNrU5OmJmDZMmDXLvrkDjyQnRmNtMIUghE2W7YA69dzUS4rY7/7/sOgYsmyZQMtg/7k5jK2UF+fdG1iUpPeXuDRR4EXXuD/Lpd3qtPMmcCll4b+UIyUISkUgojcAuBsAG4AdQD+T1W3JlYqw6G+niNgV63idVUOeyoqAi65BKipiY8cKdxmPjVRBf7+d+Dll4Fx4/pqf1Vq6DvuYH9rC9qkBckSQ7hdVQ9S1RkA5gH4UYLlMTw0NAC33gqsW0cffXU1mziOH0+XzJ13AgsXxkeWSZM4ZSsYXV3cxJaWxkemtGb9euDVV/mh9zcFRbyT1956KwHCGbEgKRSCqjb5XB0GwPZ3ScITT3ARHjly4O68oIC3P/AA0N4ee1mOOIIb0+7uwMds28YpdMHiDEaYvPoqkJ1NczAQFRWcH9y/xbORkiSFQgAAEblVRDYBuAhBLAQRuUJEFovI4nonEd2ICU1N3PwFix3m5QGdnawNiDVlZcDcuWwx39nZ9z5VKoPSUs6qNqLAxx+zO2cwhg1jdWBbWzwkMmJM3BSCiLwkIiv8XM4GAFW9QVXHAvgngKsCnUdV71XVGlWtKS8vj5f4Q5Lt2/k3VOA4L48zc+PBGWcAF1/MZJfaWiqH2lpgwwa6sr7/fU4vNKKAyxU6GKMafnDHSHriFlRW1RPDPPRfAJ4B8OMYimNEkXgGcEWYUnr00cDy5Qx45+QA++8PjB5t61JUOeggYP784HOBm5r4xlt5eFqQLFlGk1TV2WOeBWB1IuUxyMiR/NvTE9xK6OgA9tsvPjI55OVxSLsRQ44+GnjuOQZt/BV1qHL4y+c+Z5o4TUiWGMJtHvfRMgAnA/hWogUymF4+ezZ984Foa2MA9+CD4yeXESdGjuRiv3Ej0Nra9z5nLnBNjWnmNCIpLARVPTfRMhj+mTsXWLkS2LqVwWXfhJPmZrpsvvENy+pJW047jQUnjz3GQI1T5JGdDZx5JnD22fGtTjRiimgKV/DU1NTo4sWLEy1G2tPQwFnhS5d6bxMBRoxgYdqMGQkTzYgXvb3AmjXcBeTkAPvua8VoKYyIvKeqA0pKTbUbISktBa65BtixgwVqTuuKSZOYiGIMAVwu9iox0hpTCEbYVFbyYhhGepIsQWXDMAwjwZhCMAzDMACYQjAMwzA8mEIwDMMwAJhCMAzDMDyYQjAMwzAAmEIwDMMwPJhCMAzDMACYQjAMwzA8mEIwDMMwAJhCMAzDMDyYQjAMwzAAmEIwDMMwPJhCMAzDMACYQjAMwzA8JJVCEJHviIiKSFmiZUl2VDnWcskSYNkyoLEx0RIZhpHqJM2AHBEZC+AkABsTLUuyU1sL/OtfwCefcMaxMwX1sMOA888HiosTKZ1hGKlK0igEAL8BcB2AJxMtSDKzZg1w220caztuHGcbAxxruWgRsHYt8IMfmFIwDCNyksJlJCJnAdiiqktDHjyE6e0F/vhHoKAAKC/3KgOAI2/HjAEaGoBHHkmcjIZhpC5xsxBE5CUAVX7uugHADwCcHOZ5rgBwBQCMGzcuavKlAqtWccGvrg58zKhRwMKFwOc/DwwfHjfRDMNIA+KmEFT1RH+3i8g0ABMALBVueccAeF9EZqnqdj/nuRfAvQBQU1OjsZM4+fj4YyAzxCfmctFy2LTJFIJhGJGR8BiCqi4HUOFcF5FaADWqujNhQiUpvb193USBUAXc7tjLYxhGepEUMQQjPMaNA7q7gx/jKIPKyvjIZBhG+pBwC6E/qlqdaBmSlenTgdxcoL0dyMvzf0x9PTB1qikEwzAixyyEFCI3F7jkEhaktbcPvH/3bqCnh7UIhmEYkZJ0FoIRnNmz+ffvfwd27GCQWZWKoLISuPpqupYMwzAixRRCCjJ7NlBTw5YVmzZRKUyaBEyezMplwzCMvcEUQoqSkwMceigvhmEY0cD2k4ZhGAYAUwiGYRiGB1MIhmEYBgBAVFO3+4OI1APY4HNTGYBkq3BORpkAkytSklGuZJQJMLkiJRFyjVfV8v43prRC6I+ILFbVmkTL4UsyygSYXJGSjHIlo0yAyRUpySSXuYwMwzAMAKYQDMMwDA/pphDuTbQAfkhGmQCTK1KSUa5klAkwuSIlaeRKqxiCYRiGsfekm4VgGIZh7CWmEAzDMAwAaaoQROQ7IqIiUpZoWQBARG4RkWUiskREXhSRUYmWCQBE5HYRWe2R7XERKU60TAAgIp8TkZUi4haRhKbjicgcEflIRNaIyPWJlMVBRP4qInUisiLRsvgiImNF5BURWeX5/L6VBDLlisi7IrLUI9NNiZbJFxFxicgHIjIv0bIAaagQRGQsgJMAbEy0LD7crqoHqeoMAPMA/CjB8jjMB3Cgqh4E4GMA30+wPA4rAJwD4PVECiEiLgB3ATgVwAEAviAiByRSJg8PAJiTaCH80APg26o6BcDhAK5MgverE8DxqjodwAwAc0Tk8MSK1IdvAViVaCEc0k4hAPgNgOsAJE20XFWbfK4OQ5LIpqovqmqP5+rbAMYkUh4HVV2lqh8lWg4AswCsUdV1qtoF4N8Azk6wTFDV1wHsSrQc/VHVbar6vuf/ZnChG51gmVRVWzxXszyXpPj9icgYAKcDuC/RsjiklUIQkbMAbFHVpYmWpT8icquIbAJwEZLHQvDlUgDPJVqIJGM0gE0+1zcjwQtcqiAi1QBmAngnwaI4bpklAOoAzFfVhMvk4bfg5tWdYDk+JeXmIYjISwCq/Nx1A4AfADg5vhKRYHKp6pOqegOAG0Tk+wCuAvDjZJDLc8wNoLn/z3jIFK5cSYD4uS0pdpfJjIgUAHgUwNX9rOOEoKq9AGZ4YmSPi8iBqprQ+IuInAGgTlXfE5FjEymLLymnEFT1RH+3i8g0ABMALBURgO6P90VklqpuT5RcfvgXgGcQJ4UQSi4R+RKAMwCcoHEsSong/UokmwGM9bk+BsDWBMmSEohIFqgM/qmqjyVaHl9UtVFEXgXjL4kOyM8GcJaInAYgF0CRiPxDVS9OpFBp4zJS1eWqWqGq1apaDf6YD46HMgiFiEzyuXoWgNWJksUXEZkD4HsAzlLVtkTLk4QsAjBJRCaISDaACwA8lWCZkhbhTuwvAFap6q8TLQ8AiEi5kz0nInkATkQS/P5U9fuqOsazVl0A4OVEKwMgjRRCknObiKwQkWWgSyvh6Xge/gCgEMB8T0rsPYkWCABEZK6IbAZwBIBnROSFRMjhCbhfBeAFMED6sKquTIQsvojIQwAWApgsIptF5LJEy+RhNoBLABzv+T4t8eyAE8lIAK94fnuLwBhCUqR4JiPWusIwDMMAYBaCYRiG4cEUgmEYhgHAFIJhGIbhwRSCYRiGAcAUgmEYhuHBFIJhGIYBwBSCkSaISLWItHt61ji3DWhdLSJ5nvz4rsG2R/ec6zVPV1SIyDc9rZ8jagEiIsUi8vXByBLGcwxomS0i2SLyuoikXMcCIzaYQjDSibWeFuMBW1erarvnmGi0oLgUwGOeXjkA8HUAp6nqRRGep9jz2IgQEu5v+AH0a5nt6eD6PwDnR/rcRnpiCsFIGTzDV07y/P9TEbkzyOHxaF19EQCnQeA9APYB8JSIXCMiF3sGsywRkT/5WBFPiMh7nmEtV3jOcxuAiZ5jb/dYO747+e+IyE88/1d7rJC7AbwPYGyg5/IlSMvsJzyvwzBMIRgpxY/BjrEXga2VrwlybExbV3t6G+2jqrUAoKpfBa2O4wA8D+66Z3uskV54F91LVfUQADUAvikipQCuh8e6UdXvhvH0kwE8qKozAeQHea5wWAHg0AiON9IY8x0aKYOqvu5poHYtgGMdV42I3AI2VfMl1q2rywA0BrjvBACHAFjk6bybB/biB6gE5nr+HwtgEoBIGzBuUNW3w3iukKhqryeeUugZamMMYUwhGCmDp8X5SAA7ncVLRKrg/3sccetqEbkSwOWeq6cBmOt7XVV9H98Oti32eyoAf1PVPiNJPX3vTwRwhKq2eVox+ztHD/pa7/2PaQ31XBGSA6BjEI830gRzGRkpgYiMBAf4nA2gVURO8dw1E8ASPw+JuHW1qt7lcdvMUNWt/a/3O3Y3AJeI+FvQ/wfgPBGp8Mg+QkTGAxgOYLdHGewPzh0GgGaw66zDDgAVIlIqIjngvIpABHqusPC4rOpVtTvcxxjpiykEI+kRkXwAj4ED3FcBuAXATzx3z4AfhRCn1tUvAjjSz3N/COBGAC962i7PBy2b5wFkem67BZxjDVVtAPCmp0X67Z7F+WZw/OQ8BOnfH+S5+hCkZfZxAJ7dmxdvpB/W/tpIaUTkL6BbZxyAeap6YJiPqwVQo6o7B/HcMwFcq6qX7O05Eo2IPAbg+6r6UaJlMRKPWQhGSqOql6mqG8yuGe5bmOYPpzANQBYGOdxcVT8Ah68MSPNMBTyutCdMGRgOZiEYhmEYAMxCMAzDMDyYQjAMwzAAmEIwDMMwPJhCMAzDMACYQjAMwzA8mEIwDMMwAJhCMAzDMDyYQjAMwzAAAP8fjkYFTXjbEJ0AAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x, train_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Normalizing Data\n",
        "\n",
        "Before training, it is common to bring our input features to the standard range of [0,1] (or [-1,1]). The exact reasons for that we will discuss later in the course, but in short the reason is the following. We want to avoid values that flow through our network getting too big or too small, and we normally agree to keep all values in the small range close to 0. Thus we initialize the weights with small random numbers, and we keep signals in the same range.\n",
        "\n",
        "When normalizing data, we need to subtract min value and divide by range. We compute min value and range using training data, and then normalize test/validation dataset using the same min/range values from the training set. This is because in real life we will only know the training set, and not all incoming new values that the network would be asked to predict. Occasionally, the new value may fall out of the [0,1] range, but that's not crucial.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {},
      "outputs": [],
      "source": [
        "train_x_norm = (train_x-np.min(train_x)) / (np.max(train_x)-np.min(train_x))\n",
        "valid_x_norm = (valid_x-np.min(train_x)) / (np.max(train_x)-np.min(train_x))\n",
        "test_x_norm = (test_x-np.min(train_x)) / (np.max(train_x)-np.min(train_x))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjPlpf2-wHl8"
      },
      "source": [
        "## Training One-Layer Perceptron\n",
        "\n",
        "Let's use Tensorflow gradient computing machinery to train one-layer perceptron.\n",
        "\n",
        "Our neural network will have 2 inputs and 1 output. The weight matrix $W$ will have size $2\\times1$, and bias vector $b$ -- $1$.\n",
        "\n",
        "Core model will be the same as in previous example, but loss function will be a logistic loss. To apply logistic loss, we need to get the value of **probability** as the output of our network, i.e. we need to bring the output $z$ to the range [0,1] using `sigmoid` activation function: $p=\\sigma(z)$.\n",
        "\n",
        "If we get the probability $p_i$ for the i-th input value corresponding to the actual class $y_i\\in\\{0,1\\}$, we compute the loss as $\\mathcal{L_i}=-(y_i\\log p_i + (1-y_i)log(1-p_i))$. \n",
        "\n",
        "In Tensorflow, both those steps (applying sigmoid and then logistic loss) can be done using one call to `sigmoid_cross_entropy_with_logits` function. Since we are training our network in minibatches, we need to average out the loss across all elements of a minibatch using `reduce_mean`: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "id": "kdDxWeCqwHl8",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "W = tf.Variable(tf.random.normal(shape=(2,1)),dtype=tf.float32)\n",
        "b = tf.Variable(tf.zeros(shape=(1,),dtype=tf.float32))\n",
        "\n",
        "learning_rate = 0.1\n",
        "\n",
        "@tf.function\n",
        "def train_on_batch(x, y):\n",
        "  with tf.GradientTape() as tape:\n",
        "    z = tf.matmul(x, W) + b\n",
        "    loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=y,logits=z))\n",
        "  dloss_dw, dloss_db = tape.gradient(loss, [W, b])\n",
        "  W.assign_sub(learning_rate * dloss_dw)\n",
        "  b.assign_sub(learning_rate * dloss_db)\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zAAgw0h6KzUd"
      },
      "source": [
        "We will use minibatches of 16 elements, and do a few epochs of training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PfyqjVb2wHl8",
        "outputId": "308850b8-fe17-4cda-ac27-8bcda210f113",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 0.3823\n",
            "Epoch 1: last batch loss = 0.5243\n",
            "Epoch 2: last batch loss = 0.4510\n",
            "Epoch 3: last batch loss = 0.3261\n",
            "Epoch 4: last batch loss = 0.4177\n",
            "Epoch 5: last batch loss = 0.3323\n",
            "Epoch 6: last batch loss = 0.6294\n",
            "Epoch 7: last batch loss = 0.6334\n",
            "Epoch 8: last batch loss = 0.2571\n",
            "Epoch 9: last batch loss = 0.3425\n"
          ]
        }
      ],
      "source": [
        "# Create a tf.data.Dataset object for easy batched iteration\n",
        "dataset = tf.data.Dataset.from_tensor_slices((train_x_norm.astype(np.float32), train_labels.astype(np.float32)))\n",
        "dataset = dataset.shuffle(128).batch(2)\n",
        "\n",
        "for epoch in range(10):\n",
        "  for step, (x, y) in enumerate(dataset):\n",
        "    loss = train_on_batch(x, tf.expand_dims(y,1))\n",
        "  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4_Atvn5K4K9"
      },
      "source": [
        "To make sure our training worked, let's plot the line that separates two classes. Separation line is defined by the equation $W\\times x + b = 0.5$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "PgRTHttLwHl9",
        "outputId": "e4407e1b-edf5-48e5-fdc2-da28120a3c6b",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "C:\\Users\\dmitryso\\AppData\\Local\\Temp/ipykernel_66184/2721537645.py:17: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
            "  fig.show()\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABSWklEQVR4nO2dd3yb5bXHf0e25CF5ZNhOnDhxSAIBAtmMhBFWGWWvsinQUlp6oZNCd0snLbSlF24vhRZ6oUDLKA2kYSbMABkEskkgdhIncZbjGduy9dw/fnoj2da0NV7J5/v56JNIev3qaLzPec4WYwwURVEUxZFuARRFURR7oApBURRFAaAKQVEURfGjCkFRFEUBoApBURRF8aMKQVEURQEA5KZbgIEwfPhwU11dnW4xFCVmVuxYgaEFQzGmZEy6RVEGMcuWLdttjCnr/XhGK4Tq6mosXbo03WIoSsxU/74aJ1afiEfOfyTdoiiDGBGpDfW4uowUJYV4XB60dLakWwxFCUlGWwiKkmm4XW5VCEr/aW8HPv0U6OgASkqA6mrAkbh9vSoERUkhHpcHrZ2t6RZDyTS8XmDePODFF/l/ADAGKC8HLrkEmDkzIS+jLiNFSSHqMlLipqsLuO8+4LnngGHDgDFjeBs7lsrhD38AFi5MyEupQlCUFOJ2qstIiZN33gGWL6d7yOXq+VxxMTB6NPDoo8Du3QN+KVUIipJCPC4PWr3qMlJixBhg/nxg+HBAJPQxeXk87p13BvxyqhAUJYWoy0iJi8ZGoL6elkAkSkuBFSsG/HKqEBQlhVhBZZ/xpVsUJRMwJrxlEIzDAXR3D/jlbKMQRCRfRN4XkQ9FZLWI/CTdMilKonE73TAw2O/dn25RlEygqAhwu4H9UX4v+/YB48cP+OVsoxAAdAA42RgzBcBUAGeIyDHpFUlREovH5QEAjSMosZGbC5x+Ot1G4fD5mIl04okDfjnbKARDLOeq03/T+Z5KVmEpBI0jKDEzdy4wYgSwbRtdSMF0dQE1NcBJJzEVdYDYRiEAgIjkiMgKADsBvGyMeS/EMTeKyFIRWbpr166Uy6goA8HtcgNQhaDEgccDfOc7wEEHAbW1wObNQF0dFcG2bcBnPwtceWVssYYo2KpS2RjTDWCqiJQCeFZEJhtjVvU65gEADwDAzJkz1YJQMooDLiOtVlbiYcgQKoWtW4GPPgLa2piKOn06W1gkCFspBAtjzD4RWQTgDACrohyuKBmDuoyUfiMCVFXxliRs4zISkTK/ZQARKQBwKoB1aRVKURKMKgTFztjJQhgJ4BERyQEV1T+MMc+nWSZFSShuJ2MImmWk2BHbKARjzEcApqVbDkVJJmohKHbGNgpBUbIWY4CNG4H16+FpbwIAtLQ2pFkoRemLKgRFSSZ1dcCf/gRs2QKIoDBHgCqgZd5TQPfRzB9PQLqgoiQCVQiKkizq64Ff/pIWwtixgAhyABSYXLQW5AJ//Sv7z5x2WrolzTx8Pn6uOTnpliSrUIWgKMniX//iqMNRo3o87IELLTndTB988kngmGPYsyYd+HzA2rXAyy/TreVwAEccAZx8Mguh7GS9+HzAqlXAggWU2VK0Z57JfPzeswKUuFGFoIRl507eHA7O4IjWgVcJorEReO+9PsoA8CsEeLmAdXcDS5ZwAU41nZ3An/9MOT0etlA2Bli6FHjrLVbAXnKJPZRCdzctqjfe4A/RatPQ2Ajcfz9w+OHALbcABQXplTPDUYWg9GHrVuAf/wBWruw5v3vOHOCiixJaGJm97NzJhTSES8MNF1rRyTsFBcCmTSkWzs8TTwDvvw+MG9dz0R85kgvwvHnA0KHAqaemR75g5s+nMug9VL60lD/IdeuAv/0N+NKX0iVhVmCbwjTFHtTUAHfeCWzYECiKrKriGvH223SJNzamW8oMwBH+0qKF4FcIPl/EY5PGvn3AokXcaYeyAHJygMpKzvG1hrqni44OKoRRo0J/VlYF77vvAtrfbECoQlAO4PMB//M/nMhXUdHz2svN5TW3ezetByUKlZVcVEMspj0UQns73R2p5sMP+YVHCsoWFAAtLYwtJJqODuDjj4HVq2mS9u7iGcyGDTw+Ly/8MQ4Hz7FyZeJlHUSoy0g5wMcfMzGmujr8MSNHAosXA5deqq6jiBQUMKX0pZcY+AzCDSfq0cLF1u0GpkxJvXyNjbFl6IhQzkTR2cnd/ksvURmK0D1VXc14RSjl2N4e27lzcoDm5sTJOghRC0E5wMcfR18jrOdra5MvT8Zzzjl0c2zezL71fjxwodXXDuzZA9x0U+Sdb7IoKop95GKiArVeL3DffcCzz3I3MWYMzc6xY4GGBuCuuxhg743bHdv5u7vZFVTpN2ohKAfo6ootoUSE3gYLn49W/bx5XPs8HmYCHn30IM8EtPrY//OfDMD43SKeIR1oKeoG7rgDOPjg9Mg2eXLgiwwXw+jo4Bc4cWJiXvPNN4Hly/ums4pwIc/PBx54ADjkkJ4pbRMmBMZIhlNO3d2BlFml36hCUA4wenSPjWxIjOG1V1bG+y0twI9/DLzyCq/X3FyuMU89xXXkd79L35pnC4qKgOuvBy6+mBH77m54PslFy8ePpfeDKSujxn7vvdCBZZ+PVdYXX5wYC8bno6uooiL8rqOggD/A99/vmdnkdALnnQc88ghdS73NWGO4EznlFLUQBogqBOUARx7JTVp7O/8NxZ49XOgrK6kYbr8dePFFXoclJYFrvbsb+PRT4JprgGee4fGDmuJifsAA3PtGos3bBp/xwSFp9Npeey2zjdasYXqpFRTaswdoagKOP561CIlgzx5g797oYx6Li4EPPuib6nrKKTzH/Pn8cQ4bxh9bQwN3JUcdBVx2WWJkHcSoQlAOkJ/PSXwPPEDXd2+l0NhIZXHFFbwWly9nbHD48L6byJwcoLycm8x77wV+9avUvQ+7Y3U8bfO2Hfh/WigoAL7xDX6RL7zAwJAIMGkScMYZdL8kKiXW54vNH+lwhE5zFWEmw4wZwGuvMZvI5+Pu5LTTgEMP1TYWCUAVgtKD446jBf7YY0wIcTp5v6uLNUC33cY6JgD4v//j9RvOoyBCZfH888CPfqRFpBbBLbDTqhAAxgiOOYa37m5+acmoiygtpT+xszNyYKm5ma6sUIgwnjBhQuLlUwCoQlB6IQKccAIwaxYt99pabrwmTQIOO4zXtMXatdEX+fx8Wvo1NdzEKYDb5R+SY7e5ysncYeflBdJww7mNuru58zj++OTJoUREFYISkoICYPZs3sIRy/phTNgODoOWQTsk5/TTGcSur6c/MdiF1N3N3cdnPhOy/5OSGrQOQek3xx4bvWaptZXeAr3GA2ScQti2DXjnHaaNrlsXe/1Cb4YMYRZCeTkX/y1bgO3b+f+6OgawL788sbIrcaEWgtJvrroK+Pvfw2cl+XxMYrnuuthriwYD1lxl2yuEXbuAhx9me4lghg1jhpI/ayouKiqYp7xxIwPDbW3AiBEMFmvKaNpRhaD0m/Hj2VzyvvuYLVhUFGgp097OjMCDDwZuvDHdktoLy0Jo9doshhDMnj3Az3/O4hL/cJ8DNDcDd9/NdtMzZsR/bhFmByWq4E1JGKoQlAHxta9RGfzlL7T+HQ5aBh4PY4M//WmgiE0hGeEyevpp+vtC+foszf/QQ6x4TkfrjWCsUvlXXwU++YSyTZ0KnHgiqy2VmLGNQhCRKgB/AzACgA/AA8aYP6RXKgXg9bZxI13Iu3ZxsZ89m2nqTifwhS8wRfytt5hN5HYzi3HixPR0drY7tlcI+/axlXSkwI/bzda3H37IorB04fVSMS1ezEyI0lL+YBcu5BS4iy9mbMIOQ34yANsoBABdAL5pjFkuIkUAlonIy8aYNekWbDDT3EyX0Lp13AgWFjIWuHQpawy+/nWuG8XFwFlnpVvazMC2aacW27fHlhrmcnFnnk6F8OSTVAbV1T0X/cJCKosnn2QFtqayxoRt9m/GmO3GmOX+/zcDWAtAc1PSSFcXq4w3bKAbeeRIXltlZbz+2ttZgdzQkG5JM4tCZyEAG1sImUJDA6uWww35cToZsH722f5nRg0y7GQhHEBEqgFMA/BemkUZ1KxZw5bYvTdfFsOHs6fYokXABRekWjobs28f0zRff515udYYypkzAbcbDnHA7XTbVyGMHBnoYhjJSujoSG9geMWK6EN+3G6mtX76qQaxY8A2FoKFiHgAPA3ga8aYphDP3ygiS0Vk6S4dl5dUXnmF8YJI7teKCrpqdQPmZ/165tr/85/8UEpLGZz961+BH/yAOf2g28i2CqG0lG6g7dvDH9PWRp99Oob7WDQ00AqIRqKH/GQxtlIIIuIElcFjxphnQh1jjHnAGDPTGDOzTNNXEs7OncC//w38938DCxYwhhBpsc/Lo+to//7UyWhbdu5kOmZhIX1sHg8XrOJimlkdHXy+jU3tbJ12evHFlH/79r7jLZubWW18ww3pzTDyeKL3a7dIdyZUhmAbl5GICICHAKw1xtyTbnkGG11dnJX88svcUBUWMhW9ro6b3qOPpuejNz4fb7Fs1LKe116j9gwe7hJMWRndF8uXw+Py2NdCAOgP/O53mU+8bh1/FJZiGDqU2QRTp6ZVRBxxBCsjrf4ooejoYNXk+PGplS1DsY1CADAHwNUAVorICv9j3zXGzE+fSIOHJ5/kXIOxYwMu2UmTGEcQ4cCvE0/su9bt2cPrctBvwKxUx4qKyMeVlgKvvALPGJsrBIAtJm6/nbsC/3AflJWx2tAOzalGjqRSWrUqdL2BMYkd8jMIsI1CMMa8BUCThdPAzp20DIKVAcBrbN26QC3B+vXsgmrR1UXvwRlnpFZeW9LZGb21M8DdakMD3OPdaO7MkIHwo0bZtxnVDTfQDffpp1RgHg8VQUMDb7Nnaz50HNgqhqCkh3feoSLovekrKKACsOIDdXWMF1jXW20ts4sOOyz1MtsOp5MfYDSfdmcnUFQU1WVkDBX1li0cXqaEoaiIVsx113HnUlvL2/DhwK23sm9Krm32vbZHPykFW7cyZhCKESM4H2HDBlYrb9rEY8eNAz7/eVrsWVcE2toKLFnC2b4dHTSVjj+efuhwbzYnh9OF3n478m56717gqqvgadocUiEYwyLhefOAHTsCrvvp04FzzqEVp/QiP5+zFubO5Y7F4aCllnU/zOSjCkGByxU5k2jIEGYhDh3KTdf48XwsK6+3NWuAP/6RZlFJCRf6LVtYUzB9Onec4QZOn3IK8MYb/NtQk4MaG6lNZ82C+42n+lQqGwM8+ihnyJSVAVVV/Ix9PrrJP/iAsdzJk5PwvrMBER3LN0DUZaRg+nSmlUeirY1dj6dNo2LISmWweTNwzz1ctKurqfWKi4HKSm7Nly8H/vznvmmYFlVVwE03seFTXV1gNnBHB8+9fz9nGBcXh3QZLV1KZTBuHF/W+owdDlpqQ4awjYim1CvJQi0EBUceyQWosZGb4t4YQ/fFFVdkuTv2+ef5BouK+j4nQqWwbBkX93C+m6OOogJ57TV2A/R6qWDOOYe+t2HDALDB3f6u/ej2dSPHkQNjOOd+6NDwDQE9HmZ1LVlCD4miJJpsvryVGHG52Nr+rrtoCVRUBBaltjYqg2nT6BHJWpqauEWP1C5ZhArjrbciO/NHjwauuQa4+moGmXNz+5hUwTMRivOK0dTEWGi4ccMWxcWcQqkKQUkGqhAUAMCECcAPfwg89xw3wVYws6iIlsEpp0QvPtuxI9DtYPToDJuD0NTENx2tX7fbHbmlQzAiYT+04I6nxXnF8Hr50tFccbm59EApSjJQhaAcYPRo4Oab2ZutoYGLz8iR0d1EdXXAY48xHmtNTAPoirriiui1WrbA6WT0NlLVK8AdfwICl71nIhQVMX7t9UZWvC0twKGHDvjlFSUkGlRW+lBaysBmVVV0ZbBlC3DnnUxHHTuWLo+xY/m369YBP/sZLQfbU17OyG1zlGKx1lb28RggvRVCXh4rwevrw/+NMbQOTjxxwC+vKCFRhaD0G2OABx/kzraioufG2uGgdeH1Ao88kj4ZY0YEOPtsVoP5fKGPaWhgqk9/hsv3wu30u4yCGtx95jNUDHv39j3eGMYYpkzRtjxK8lCFoPSbmppAUWg4KiqAtWsPdH22N8ceC5x2Gs2dffsCvi+vl9V7nZ0sxIjWniIGQo3RLCsDvvMdWmU1NcxebWigS662lunBN92kY0mV5KExBKXf1Nby30gudxHeNm9mNqatcTiYGTRpEnNAN2/mYw4HU0bPOIOupQQQbq7ymDGcQrdyJbOJ2troyTruOLrisrL+Q7ENqhCUfhPOswJwc+3zBXaz4Wq5bIfDwRjBUUfRSujqYsQ3XHVyPwmnEAAaIDNm8KYoqUQVgtKH+noWP9XXsxhq2jSmpVq7U+vfESP6/m1DAxtP1tVRIbhcXE/D9UqyLSKMFySJ4LRTRbELqhCUA3R2An/7G+uuHA5uijs6gMcfZ9eFYcMY9KyuBs48k7HVoUOZwl9cTNf7hx8yyOzx8BzNzcDu3Zyzcttt9u2inGoiWQiKki40PKUAoEvnz39mt4UxY3grK2PGy6ZNjKlu2sQg8d69HLH5hz8Al1/O+5s3c+Z5URFvDgcbT/p8dL93d7NNkBZVkYLcAghEFYJiK+JWCCLiFhEbjEtSEsnGjez2XF0d8Pt/8glvpaVUDvv3M5A8ZAjrFNatY1XzN77B9tgdHSycamyk+10EmDOHlsXw4VQcH36YxjdpI0QEbpfb3nOVlUFHVJeRiDgAXAbgSgCzAHQAyBORXQDmA3jAGLMhqVIqSWfRIrqIrPhAdzfw8cc9u256PFQcEydSaVRVMRPm9NOZG3/YYYEOEKWlVALBKZIeDy2Qo45K9btLItEqmyOQlLnKxlBzi/T8QhUlBmKJISwE8AqAOwCsMsb4AEBEhgI4CcCvRORZY8yjyRNTSTY1NT3nJe/dy5hCcDA4N5eFup2dXGusxX7ZMsYNRo7kLRwuV5ZM/6qt5fzkd9+lWTRiBLXiUUfFFT1PqELYvx9YvBiYP58tUQHmqZ51FtOV7DADWbE9sSiEU40x3t4PGmP2AngawNMiEqXtmWJ3nE76/C26uvpuLo3hLXjXn5sbaPvf3R153Wlvz4KJX6+9xsi708mahNxc+skefhhYsAD49rcPtLiOhtuZIJdRczPnCm/aRN/e2LGBOad//CNwzDHAl76U5b3LlUQQNYZgjPGKyCQROUVEPMHPicgZ1jHJElBJDTNncv2wcDr71g60t9MVFNx8rauL6+LMmez6EInW1gzvw7N6NRf+ykqmSzmd1JpFRQy+7NsH/P73kcfPBZEwC+Evf2FTqXHj6JcDAmmzBx3E4NC8eQN/HSXriaoQROQWAM8B+C8Aq0TkvKCnf5EswZTUcuyxXEMsK2HoULp4rN2/5ZoOrkewCtOmTGEaqtcbfvLa9u2MORx+eHLfR1L59785QSi4dUVbG7B+PUedLVvGhfexx/hhRSEhCmH7ds7WDDfHQYQK7MUXe5qAihKCWLKMvghghjHmfABzAfxARG71P5fQiJWI/EVEdorIqkSeV4nOsGHA9dez51BDAxf74cMZRF6zhjN9AXpHlizhLPmFC4GDD+bfVlcDX/0q/3bzZq6TnZ3cNNfU8JivfS2DvRZ79zLKPnRo4LH6euDVV5luBTCw4vMBf/oTh0tEMZkSohCsLyZS8Dgvj1/GJ58M7LWUrCcWhZBjjGkBAGNMDagUzhSRe5BghQDgYQBnJPicSozMmcPiMY+HscmPP+YC7vMF5sK8+CLXoC1baBGsWQP87ndUANOnA7/8JXDuufybpiaunzfdxPUxRte6PWlrY4DEWnibmhhUzssLWA05OQwqFxZSc959d8TCC7fLPfBK5ZaW2LWsWghKFGL5Je0QkanGmBUAYIxpEZGzAfwFwBGJFMYY84aIVCfynEp8HH44Xc+HHMLFPCeHa85rr3EttDabM2YAkyfzsZUrOfz9m9+kVXH++bxlFW43YwNWmumnn/Lf3p1PrQE6FRWB0u0webYeZwIshKFD+ZrRsMbfKUoEYrEQrgHQY8SJMabLGHMNgBOSIlUERORGEVkqIkt37dqV6pfPejZvpgVw8MFMWBk6lEksQ4awzmDSJCax7NoV6GQ6Zgz/Zv36dEufRIYM4Zvfs4eKYfPmQAA3mK6uwGDkkhL61cKQEJfRlClM+4oUyG5tZTaADlJQohBLltFWY0zImVfGmLcTL1JUeR4wxsw0xswsy6ihvZnBkiU9PSPd3dzoBq99eXlUEtZwMRF6SSKsfdnBuefSVdTa2jf/FuAHUloa8I3l54eeduPH7XKjo7sDXb4YdvjhKC3lDIfa2tDtZ71ejqy78EKtRVCior2MlB7s3duz07PX27e+wLIMOjr4fGcnPSpbt6Ze3pRy6KGMvO/ezZiC18tFuL2d0fPCQrbOthRFZ2dEN43V4G7AcYRLLmHDqNpaBnr276d8W7ey7eznPseBCooShUzN+VCSxJAhPeOgDkegIM2yGozhJnn58kCXBIAupZaW0J6UrGHuXObefv/7NKfy8viGDzuMZdrBRRr79gEXXxz2VMEdT0vyS/ovU24ucMMNlO2119hYyuEATjmFikJbzCoxErNCEBEB+xkdZIz5qYiMATDCGPN+ooQRkcfBLKbhIrIVwI+MMQ8l6vxKdGbN4rAwSwG4XPSANDdzA2wMN55NTfRWlJTwuN27mWV5550cAxmcnZl1jB4N/OhHwE9+wrYVoYbn7N3LD2f69LCnOWAhJKJaWYSKasKEgZ9LGbTE4zK6H8CxAC73328GcF8ihTHGXG6MGWmMcRpjRqsySD1jx9IzUlcXeGziRFoNPh/XuT17uCYWFnId6uzkRnnGDG6K77svgyak9Zdx44AvfpG1CHV1gUyf9nYGnH0+toGN0NvI7eSQHG2BrdiFeFxGRxtjpovIBwBgjGkQkYFPG1dshQjw5S9zdsGmTdzpl5ezA8LKlVQIFRVshGe5jrq62C4nP58b5k8+YVZm1ie1zJnD8uvXXuNUoe5uuo/OPx84/vioE9d0SI5iN+JRCF7/HAQDACJSBiDCVF0lUykuBm6/nZ0YFixgEdqwYXRJv/46F/6mJiqEkSOZomqtfSJ0aX/wwSBQCABTTD//eeDaa6kZc3NjbjmtCkGxG/EohHsBPAugXER+DuBiAN9PilRK2snP5wZ4zpyA+2fdOibWDBvGzbDLFdp97nQyuDwoaGsLTAcqLqYPP8bKYZ2rrNiNmH65/oDyGwCWATgFbFlxvjFmbRJlU2yCteEtKqJysGIH4ejoYMVyVtPZCTz7LPDKK4Fe4cZQKVx0EV1GUSwFtRAUuxGTQjDGGBH5lzFmBoB1SZZJsSmjRjGY3NjIDKNQ+Hy8zZqVUtFSi9fLodIffRRog23R1sbh1E1NwNlnRzyNKgTFbsSTZfSuiGTzZa5EQYRp9Xv3hu7ZZgwTbObMYeA5a3n3XWDFCqZkBSsDgObTmDHA00+zSCwCCU07VZQEEE8M4SQAXxKRWgCtoNvIGGOOTIpkii2ZMoXFun/7G+8PHcritI8/Zi1CZSW9JvX1WaoUjGEr2OHDw7uEnE4Whr35JnDppWFPlZeTB4c40NLexOE7CxeyzURBATB7Ns2srK7yU+xGPArhzKRJoWQUc+eyMPfNN4HHH2ewubiYymLIEM6KWbCArvTPfjbL5ry3tHDRthrYhWPIELqUIigEEWHH08WvAxvr2P/D7eZrPPII8I9/cIjEIYck9j1kM/X1/GG++Sbdd+Xl7PUU57zrwUrMCsEYU5tMQZTMorycSqCggGn3wYk1JSV0sz/xBDe4c+emS8okYJVwR9NyVpA5Et3d8HQatLTsBqpP7XnO0lLGIX77W1ZEV1YOWPSsZ+lSDicCaMEVF7NQ5uGHWX7/7W/zh6uEJZ7WFT8M9bgx5qeJE0exA3v3srCsq4sppuPH923s2dEBPPMMg8yhsiydTsZbn3qKMYXervaMxeOhxmtt5W4+HPv2RW8ot3493F5Bq9uJkLOmiovZM+TFF4HrrhuI1NlPTQ1L5MvLuUuxKC4O+DDvuQf46U/7zrBQDhCPyyg48pUP4GwAmnaaRTQ1cRzw+/7uVNYmt6wMuOIKYOrUwLFr1rBLQ15e+PMVFLC/0dq1wJHZEmlyODhA+rHH2L4iFN3dNJFOPDHyuV5/HR640AJv+GMqKjiv9LLLei508bJ9O5vx7dpFpTZjBjV9tvjzFizgQh/uM6qooNL46CNg5syUipZJxOMyujv4voj8FsC/Ey6Rkhaam4Ff/YoL+OjRPdtdNzVxTObNNweGfzU2xtavyBgem1UcfzwX6a1baQYFL6pdXWxD/ZnPMAspEvX18EgeWtAZ/pjc3ECPkP4ohPZ2ukzefZfKLD+fNRT/+Q/7kdx8c4bPNgXf45Il0d1qRUUstVeFEJaBtL8uBHBQogRR0ssLL3ATGWoNKy6my+fBBzk2s7CQmzFjqEi6uvi82x16w5l1FnpBAfCtbwF//St7dBhDDdrdzQX8gguA886LvvsuLISnMRe7cyIoBGNY2NGfD9HnA/73fynj2LE95TGGTfnuugv4wQ8yO5uprS3wHUQiPx9oaEiNTBlKPDGElfD3MQKQA6AMwJ3JEEpJLfv3sz/byJHhj7HcP8uW0TXe3s5UfCAwM6G0lAkx1nm6u/lcVibJFBUBt9xC3/Tq1VyUhg6lbyzWxXXOHLifux81zl4KobOTH7AI/z34YGrleFm/nkMrqqv7KicRflE1NczIOTODkwgty6n3JKfetLdrUDkK8VgIwWWXXQDqjTEDmP2n2IX6eu7yo21CCwsZO2hqAp58kp4Gq+2/MbzeFi/mmjhhAjegs2eHr2rOCioq+l9wMX06PM8XoMXsYUy5pYV9kbZsCUwl6uxk6qnVOC8eXn01ep+R8nIGrU8/vW/mQKZQUMCYyMqVbLcbjubm6HGdQU48v4CvGGNq/bc6Y0yXiPw6aZIpKSN4Glo0du4E/vlPdn0+5hgqg4YGrlf5+by/YgWvzepqBqOVMBQUwDNlFlrhZTHHwoVUBm43fXDG0Dp4803g/vsZqI6H2trolkVhITX8/v39fx924IwzuCNpbw/9/K5dTEWdMiW1cmUY8SiE00I8lsF2pmJRXk6F0BXB3rOCwzt2cK1yOmlRHHcc4wpdXXze6nI6bhxw221aCxQN9/BKtOT6gzHt7YHATHExzaspUxj8XboUePnl+E6em8s4QiQsSySa/93ujB/PgUU7d9I09XoDwfjaWv5gv/GNyGlxSnSXkYh8GcBXABwkIh8FPVUE4J1kCaakDrebiTNvvMGdfzA+H7BtG1NHd+0KxAra2hindLk4UW38+EB/IxGuaaFaYys98bg86PR1orN8GFzTp9NFlJPT1383ciQzg047LfaiDmseaqR6ib17+QVmw0I5Zw5/lAsXcmBRZyfjOpdfTnO2P3GYQUYsTsm/A/gPgF8CuD3o8WZjzN6kSKWknHPOAT78kJlGI0ZwUff5GESuqeH/jzoKWLWK69Hq1dx4zZlDF67D0TO21xkhcUYJcKDBXZ4Drpyc8KmlBQXUyHV19MXFwnHHAc8/T00dasH3+VhAd8MN2VOPMHo0cPXVvPl8mRsXSRNRPy1jTKMxpsYYczmAJgAVAMYCmCwiJyRbQCXxGENX9cqVdF23t3MjdccdXGtqa6kE3nmHzxcUcIM1fnxgJkJpKd3OS5f2rUdobtZOC7FizVVudcZQ1OFwxBdHKC/nJLetW7nwB39Rra2ckXrqqdnrV1dlEDfxpJ1+AcCtAEYDWAHgGACLAZycFMmUpLB6NXum1dYGrhenk56Ic87h6My6OmYT/fGPwAkn9CxUmziRVkNeHrMr9+zpOx+hoYGFtUp0DsxE6GiOfKAxNL2izGnuw4kn8m+efpq9ya0c4ZIStsOYOzd7rANlwMSTx3YrgFkA3jXGnCQikwD8JJHCiMgZAP4A1jk8aIz5VSLPn2k0N3MTZ/UUGjNmYNfukiWc61Ja2rNOqbMTmDePr3XrrVQAbW1sWdG7qWdlJbBxIxNTiop4jh07eE5jGG8YNw6YNq3/cg4mDigE8UbOo9+1Czj8cGbK7NnD/N6aGh4/ZQo/8HDupiOPBI44gl+OFdypqsr8QLKScOJRCO3GmHYRgYjkGWPWiUjCSo5EJAfAfWA201YAS0Tk38aYNYl6jUyhtZWpnW+9RTeoCNeKqirGxw47LP5zNjdzkNeIEX3XDZeLrqKVK4FFi9h1oaMjtPLJzWXyy/vvMx7Z3s5/rcSOQw5hN4Rgl7UxtEg2buQxI0Zwbcu6CuZ+cCCGMGsK8EYtv4jero6WFn7QF1zAsZ3z5vFxt5s/kPfeYzrXl79M39+GDfyghw8PfNAibLOhKBGIRyFsFZFSAP8C8LKINADYlkBZjgKw0RjzKQCIyBMAzgMwqBRCayu7CWzZwt24VYtkDN3Ad93FOqXgRnOxsGQJ14hwm0gRLtTz5wMnn8yEDJ8vdI1Cfj6zkhoaWHNw8MFUEkcfzQzJ4ON37AAeeIDdUwGudT4f168rrmBQejB7LNwuxhBa5hwFuA5myXhODv1x3d1UBoWFwDe/SX/fM8/QvAsuUisrowVx3XU06YqKAq6hwkL672KY8awo8TS3u8D/3x+LyEIAJQAWJFCWUQC2BN3fCuDoBJ4/I5g/n7vp3okkInQFu1xs+f6738XX62zFCq4TkSgspDdi926uKyNG0DVUUtL3WBE+PmECOwoPHdr3mF27gF/8gi6v3q109u9nm52uriyblxAnB1xG3fuBa65hMOftt+kOcrlYgTt9OrX5Pffwi+ldsbx/P1tU7N/PXcMRRwQ+7P37aRp6vcApp6T0vSmZR8xheCFXicgPjTGvg4HlqQmUJdT2pU/qhYjcKCJLRWTprl27Evjy6ae9nd0GImXouN087oMP4ju31VcoGla6qTU/ec+e0POTfT4qrrlzQysDgN4Na2hV781pQQFjFY89RqtosHJAIXT6K/pGjuQH/61vsVeSlde7bBk/9FA1CBs28EcxfDj9d1Z1IMC/raoC/v53andFiUA8eVn3AzgWwOX++82gzz9RbAUQXBY1GiFcUsaYB4wxM40xM8vKyhL48ulnxw5u5GLpKbRqVXznHj+ecYRIeL1UGlYiy4wZ9ELU19OF1dzMxXv7diasHH88YxqhaGxkx+VIDfPy8viaS5fG916yiQNpp94oWnHbttA/DK+X1oTHE5jk1rsNhcsViDUoSgTiUQhHG2NuBtAOAMaYBgCJDAsuATBRRMaJiAvAZRhk8xaidRmwsILM8TB7Nv8m0t9t3w6cdFJPV9RJJwG//jVw9tmBFjuzZrFj8he+EL5odudOyhktkaWgIBBfGIz0sRDCkZcX+strbe3beiKUKeh2Ax9/PABJlcFAPEFlrz8TyACAiJQBiHEJi46/Wd5XAbwIpp3+xRizOlHnzwQsgydaY8u2Nu7442HECPb/euGFvjFJY+jvLy5m08tQcl1wAW+xYk1bM4abWBG+Zm/XUTyN9bIRV44LuY7c6Aph8mRWHUfC8guGCvoM9g9aiYl4FMK9AJ4FUC4iPwdwMYDvJ1IYY8x8APMTec5MoqiIO/nFi+lfD4Xl1jm6H+H2Sy+l9+CFFwLuaMtqGDsW+MpXwscD4sXjYSrqypVUcMbw3BMnBlpjAIxPTJqUmNfMREQEHpcHrZ1RXEYHH0z/2+7djBVYeDz8QXR1MXYwcWJos62trX/5ysqgIpbmdv9njLkawHAAtwE4BQwAn2+M0ZnKCebcc5kRtHMnd+bBm7qODnYhuOKK0JvAaDgcwIUXMtlk+XLGLPLzmZSSyPG6u3cDd9/NNai9nUV1xnC9WryYqalTpvD5goL4U2izDbfTHd1CcDiAr34V+OUv+SMYMYImV24udw+rV1Orh5pG1N7O43R0pBKFWCyEGSIyFsD1AP4G4HHrCREZqg3uEktZGfDd7zK11GovYW0AXS5mJp566sBeo6SEsYFk4POxGrq5mb3V3nyTAebiYgbDCwqATz7h+uTxsKZisHdF9bg8aPFGUQgAC8t++EMWpr3zTsAvV1HBH4fb3Td+0NxMf+BNN2X2mEwlJcSiEP4E1hscBGBZ0OMCxhN0rnKCGTkS+PGPGWxdv56WwciR3FX3Z856KvnkEya9WHUUxx1Ht1FdXcACMYZZSg8+yK4Kg52YXEYW5eXsTnrppcwJdjj449i/nzm877/f8/hhw9iPZMaMxAuuZB1RFYIx5l4A94rI/xhjvpwCmRRw8Rw/Pv7gcbpZtqynCzs/n1lJkyezstkqng1X8DYYcbticBn1pqioZ6Wh08nWFZ/7HLVyVxeDNhMmaM8iJWZiiSGIIWGVgXVMYkVTMpHm5tAxzYKCntZNS0vogrfBiMflQX1LffQDOzuZOtrayg/z4IP7+tuGDk1cZoAy6IjFZbRQRJ4G8JwxZrP1oL9W4DgA1wJYCODhpEiopJy2Ni7YeXn0/ccTbC4rCz/W1sIYxhqitdIYLHhcHnzS+Un4A3w+4KWXgH//m64hK3bgcgFnnskikUh5yooSI7H8is4AA8qPi8g4APsA5IO1Ai8B+J0xZkWyBFRSx9atnNL43nuBRXviROCzn6WvPxbFcNRRwL/+FTntfd8+xhhGjEig8BmMx+kJX6lsDPD448CCBQwql5cHnuvsZLO7nTtZJagDYZQBEksMoR1sW3G/iDjB9NP9xph9SZZNSSFr1rB3mhWjzMnhWrR9O1NIL7wQOO+86EqhspI1Eu+917ehHUA3UUMD1y+tkyI9YgiNjewVIsIPc9s2WgfV1X1jAVbf8rfe4oeerZPPlJQRl51pjPEC2J4kWZQ00dwM3Hsvg7zBbhwRJqmUlHAjOmECg8PRuO46bl6XL2cA2eNhbcL27TzntdfGdp7BgsfloUJ44AE2gLI0pTUy0+EIHxh2OOjXW7BAFYIyYNTGVPD++9y5h/Pp5+byuf/8J7bz5ecD//VfnNE8dCiH7ixbRiVRUsK164c/ZOqpAng6DLp8Xeh8fzHdQlVVvJWX09SyBl+HY9gwHhNrMyxFCYNGohS8807PmcihGD6cbqW2Nu76o2Ftbuvq2BW1uDjwnDFMof/FL4Dvfz98m45BgTFwL1kBAGgZXY6hCLIEnE6aV/v3s5hj1qz0yKgMGvplIYiIhgOziLa26EkqVmflzs7YztnVBTz0EDevwcrAOtfw4fz3scf6J3PWUFsLzy7OKWhBiA936FB+OXV1fdtaW1hReg0qKwOkv7+gQduALhsZOZJKIRKdndywut2xnXPtWsZHI6WWlpfT07EtaOrF7t38248/ji5TVrBxIzyG2jikQhg/nqYWwGh8b4zh42eemUQhlcFCf11Gmh+SRZx8MgPA1q49FPX1nO4Ybv5Bb2pro2cRWVbH9u3suPrPf9IzYo0Dzs1lz6Vzz41dEWUcXV3w+MeKtIZSCGVl9KmtX0+zKxifj5OLjjgCmDYtBcIq2U5/FcKfEyqFklYOPZT1Bp9+yrWn90Le0MAMx3hG8saTUlpXx/nKOTmMpVqeD6+XGZfr1gG33ZalSqGiAu5uxg1CWggiXOybm3mrqeEHZM05Pe444MorY9fUihKB/iqEZxIqhc1pbmYmzssvc3F0uzlLeM4c+sgznZwcju/94x+5ES0o4Hvs7OR7LynhghzPxNJx47jLj4TPR8tg/nwGqnsHtp1O1jLU1rJIN9y4zozm8MPhyaNfLaRCAFg2PmcO8J3vsM11YyODzZMna5sKJaH0VyHMBzA9kYLYla1bgd/8hs3Yhg2j37ujA3juOQ6aufXW7Jg7UlQE3H47FcLrr9NF5PFwHZo6Nf4W1YccwrWqsTF8E7v6eiqZPXsiVy1XVgILF7IwLpYMp4zC5YLnrPOBd59Cq7cN6L3Rb23lLuSLX6TGnDMnDUKmia4u/jis6UrRho0rA0ZjCBFobQV++1vuZMeODTxeWMhbczPw+98Dd97JlvSZjsNB99Ghhw78XDk5wJe+xHnM3d3AkCE921/X1/P6njiRyjYSTifXhi1bQs9/yXTcRx8PvAu0tOwFGmtoohnDnUd+PvD1rwOHH55uMVPH/v3Aq68CL74YyCzIy+MgkNNO0yZYSURjCBFYsiSQ0ReKoiLugBcuBC67LJWSZQYHH8zitIcf7hlkNoZK59prgVdeiS1bUiT0jPls4IDL6KJzAHM0gzki/JCmTuViOFhoa+Mu7JNPaDZaPtmODvoN33+fpmy0whmlX/RLIRhj7k+0IHbk1Veju2jLy6kQLr2058JmDK/r5cvpAi4rY11RNlgS8TBhAi2omhqO7HQ4GDiurOTzo0dHb4NtNdoLHiWcTbhdjJa3wguceCJvg5UnngA2bWIQKpi8PO7M6uqAv/wF+MY30iJetqOVyhGwYneRcLkYfO3sDPjZ9+zhGMlNm5g66XRy0Xv6abqAr756cG36RHh9977GAY75ffRRfn7hXMS7dtFjEtzoM5tw5bjgdDjRsvVT4K67Aj+cadOYd1tdPTg6ATY1AW+/Hbl0vbIS+Ogj5iqPHJk62QYJqhAiUFxMd2akWJbXywXfOqa5mX7zpqa+3T59Ps4Y7ujgcCstLKXCvfhiVixXVfX9rBsb+Rlfeml65EsJ7e3w+HLRsvRtoLOAZqkxbHT3xhvA+efH1mo209mwgRdJpAlv1iyI9etVISSBqEuSiAyN4VY6ECFE5BIRWS0iPhGZOZBzJZKTTgL27o18zM6dwAknBBb3N97gYyNG9L1+HQ5u9t5/H9i4MSkiZySnn85U+p07A66lbdv4f4DZlmPGpFPCJPPII/B0OdBSnE+/mLXDqKzkbvnpp9niOtvxemNTeg5H9ClMSr+IxULY5r9F+qZyAAzkkl0F4EIA/zuAcyQca9hLQwOzZHrT1sbNyskn8353NxMjIsUJRJhE8tprDLoq/EzOOIPutGXLGIB2OukmOvzwLB8GVl8PLF4MT3UBYwi9yc3lD+q554DZs7N7PvKQIdGLVwAek60BpTQTy6W21hgTsS5eRD4YiBDGmLX+8wzkNAmnqIixq9/+lq2ay8oYJ/B6AzNMbr45ECC1Rk9GC0QXFwd2v0qAoiIW/A0qPvgAcDjghit8YZrHwx9gbS1w0EGplS+VTJhApdDSEj5419HBAJwO1EgKsSiEYxN0TEIQkRsB3AgAY1LgRxg3DvjZzxjreuklKoKCAuAzn2EySHBBlTVlLNL4SIBuUq2xUQDQJ+l0whNJIVi0tMR+3s5OBl/ffZcFNRUVbHMxfrx9YxE5OcDnPseMDKezb+ZFVxcrRa+6Kv5KSSUmYh2hGRIRuc4Y89dIxwQd+wqAUPWo3zPGPBft74PkeQDAAwAwc+bMGOzLgTNkCOeYn302F/NwweCCAiqQPXtCu5gs9u5lwzZFQXEx4PXCAxe2oTn8ccbwBxYLmzezYrKhgRWUTifz+hct4s76y1+2b2Ooo49mJsf//R8vtuJiKrCmJn4Gl1zC3ZiSFAbqnf0JgL/GcqAx5tQBvpYtiJQZJAKcdVZgHGWoY62c+8HUgUCJwLRpwFNPwQ1neAuhrY0LY6i83d7s3s3UVYejZ3l9aSkX1DVruAP/1rfsG4+YO5fjQN99F1i1iorBah4WT0MtJW6iKgQR+SjcUwAGWZlVdKZPpytp0SJmxVm9d4xhCuXevRwwr7/rwcOuXax4dzqZNNQjSD5qFDB1Kjx1i9HiDqEQuruZc//5z8cWXX/1Ve6wq6r6PifCx9es4cCJRPQoSRZDhnDGg855SCmxWAgVAE4H0Hs6hwB4JxFCiMgFAP4IoAzACyKywhhzeiLOnWocDl67Y8cCzz/PDZsINzljxgDXXw8ceWS6pVRSwcaNzBhdty4w46GoiGvcqacGre9f+AI8dz+JFt8moKmR1oAx/PG0tDAv96STor+g18uy+WhpboWFPM7OCkFJC7EohOcBeIwxK3o/ISKLEiGEMeZZAM8m4lx2ICeHF/zcuXTndnTwGq+stG88T0ksH35IN77bzY2A9b23tQF//ztd+jfd5PfaeDzwzD4RrYuXwLgLIbW11CBHHEF/+WGHxfbDaW2NXPJt4fGwBYSi9CKWoPINEZ67IrHiZBe5udmdJaiEpqUF+J//Yap879htYSFDAe++y/iu1bbInV+CbtONjh//APnipEKId/fgdNIUjZbm1tXVd9C1oqD/M5UVRQnDktfb0L5pG9zL36RrZtmyQF9/cK0uL+c8DZ+Pf+NxMe++pbOFZkN/TEm3m9WO0crr9+0Djjkm/vMrWU8srSuWJ+IYRRkUrFqFZT+bj+LNq+jC8XrZh+PNN2kWeFmNXFTEEIG1dlsKobWzdWCv/9nPMnshXK/wlha6lI4+OvZzWoNq9u4NaDAlK4klhnBohEwjgMHlMDOxFGUQUVMD3HMPOnMvRk5JEZDvzzF2uQJTgZYvZ08UEYhwrQUCLbBbOuMoPgvFkUeyYGbePM4SKCkJZDXs3MkYw9e/HpvLqKWFPVZeeikwqGbIEFoXY8bQJ1pZqU3msohYFMKkGI7J0tElihIH8+YBLheqy9uwaVMFSiyFAHBRLilhCmljI7zuUuTkBOa89HAZDQQRFm+NG8c0ty1bAh1CZ8ygsgiuTwjHvn1s27t9O8vxhw+ngliyhA2+hgxhsDsnh2PsrroqcttqJSOIJahcG3xfRH4KNrNbAWCFMWZDckRTlAyisZF9iUaPxnEFW/DSJxPgM4AjOBQgwmDx5s3YMaQUJ50U6MBwwGXkHaDLyHqdWbM4bGLPHloFHk98geSHHqJPyxoX2NxMt5fPx9qJpia6xCZPZo+ln/0M+N73Qtc/KBlD3EFlY8wPAdwLoBnARSIyKMZpKkpEmpoOLPhVxY04fkwtavaVwmd6BYedTuyu70JhITu8WiTMQghGhDv7ysr4lEFdHSuEra6NAPNofT4qFoeDQZBNmxgTqaig++jBB2PrVqrYlphbV4jI7wF83ZB6AAv8N0VRXK4DKZ8igmumfAinoxsLa8ZBxKAgtwtdPgc6WvNQMQa45faeHZzdzgTFEBLBmjX818p0amqitVASFCrMyeH73bs3MPu4tpZxlFAtNnbvBt55h4oGACZNYiuKwTZT1ubE08uoBcC/ReQyY0yriHwGwI+MMdqVR1HKyxlcbW4GiovhzPHhmqkf4YyJn+C9raOwvcWDgtwuTPctxaTvX4ycUT3/PCkWQn9pb+/ZiKupif/2ToUVCWQzWc9t3txTIRjDoPSTT/K+pVQ++YQxl/POGxzT4DKEmBWCMeb7InIFgEUi0gGgFcDtSZNMUTIJEeCcc4D77gu4VQCUu1txziEf8xh/q2tM6dvLP2Fpp4lg2LCeaavh3EDG9G1D3Tst9a23ODS793zUkhKmWD39NOsnTjstMbIrAyIel9EpAL4IKoKRAG4wxqxPlmBK9rJnD7B4MRNWvF5uKOfOZav+jJ4zfcwxbGD00kucklRaSkXh9XIuaH4+u4yGaC2RsLTTRHDkkYwJWAPDrWE1wRXQnZ0su+49DSp4QEhXF/CPf9ByCtVOIzc3MCL0hBP6zj9QUk48LqPvAfiBMeYtETkCwJMi8g1jzGtJki0p+HyB+F9RUYYvQBnIO+8wgcUYZi46HMDSpdxIzprF1j21tYFY5eTJ6V8n9u6l3IsX05syYgQ3tIcfzvXyACJMvzz0UJYh19TwMaeTza1OPTVsm9tcRy7ycvLsoRA8Hlo7Tz3FLKPSUgal29s5k6G7mymo/noKALyohg9nCqrF+vU8btiw8K+Vl8dmX6tXs1WwklbicRmdHPT/lSJyJoCnAcxOhmCJpr2dWXP/+Q9TrAFubs46i7GtdC86g4HVq4E//YnJK8GeBrebrvcHH2Tjt4MPDnQHLSjgEK0TTkiPm/mDD9iXqKuLvxenkwrrd79jn6o+NV4iTPecMYOLZFcXF9gYfmAelycxaaeJ4Jxz2Eb7xRcZQJ4wAXjvvUCl89SpgbqD5mZqzW9/u+cOy7rQomEMh/koaaffA3KMMdv9biTb09oK3H0341jl5SyyBPjbfvhhdhT4+tdjH0iVrRjDz2j9em7aKis5pyQRn4sx9AyUlvZ1O7e10ULIyeH/hw8PeCna26kouruBk0/uc9qkUlMD/PGP3OAGN6nLz6eMW7bw+TvuCGFpWoVoceBxeZJvIRjDtNLVq3lhDBvGIT2901IdDuCyy4Djj+dOasMGvp9PP+UPIjeXAWSfj1/qt75FkymYeObE6o7MFgxoYpoxZn+iBEkmjz7KXV3vbDiPhxf6hg3AE08A112XHvnswPbt3Alv3sy1LCeHm1uXi4Wvp546sB36jh1MWw81Bnv9erqIiou5qayrC3ge8vMZj/z73+lSKirqvwzx8sILfP/hpk2OGsXfzvr1iRkt4Ha5k6sQ9u0D/vxnKgMRLupdXcDf/saZCxde2HcIz6hRVAwW3d18wzU1VC5jxvDNhxreM3FiIBMp3HQ2n4/HBLualLQx0BGatmfPHloA4QooRWj5vvUWcNFFg7Mr8K5dwC9+wet27NieC39HB9eL7u6ehVTx0tQUuqNzRweVkLXQO519Z8m7XHz9JUtSZyW0tLBJaaRuDCJUWG++mRiFkFSXUWsr8JvfsJ9R7y+5q4ttLtragGuvjaz5c3LYsuKww6K/Zmkp/bFvvx16JwCw8d/06TpC0CZkfUh1/XpuZCIFj60am/WDNGfqX/+iu7i8vO9akJfHa/mf/2R3hmB27ACefZaZlg89BKxYcaCZZx/y8kJnL7a2Bjo6AFz4Q3kaCguZwJMqWlp6yhWOwkIq1ESQVJfR66/T9Bo1qu+XnJvL4PGiRTSlE8nll/Pcn34aaJAH8Ae3aRMj9Ndem9jXVPpN1lsI7e2xHWcMd6uDjaYmWlDBXQp6Y81def99Ztd4vXTDvf46F8zCQm4y33yTLulbbunbP62qitZXa2tPF4zVd83C5wstSzSlnmjy82ObNWO1CUoEbqcbe/dHmWXQH7q7GRwuLw9/jMPBL3rRIs6ATRSFhcBtt3EuxIIFgZmyhYX0RZ50UnifnJJysl4hWN1/o9GPGGBWsHs3/w3n4rUoLOQmzxjgkUeAN97gxq/3Ir13L/CrXwE//nHPrgQ5OUxcefhhxnKsv7NquLq7uWksKemb2g5wc5nKEcAlJXSBb98eOWuypYVekUSQNAuhpYWZQEOGRD6upCQ5Zlh+PgdJn3Ya4xhWznGouIOSVrLeZXT44XRBRNr9t7dzwZsUS6PvLKP3Dj0cxnBRr6tjvCWUMgC4mHu9wPz5fZ+bO5e3TZuYZWgMN6WVlVx4nc6eqe0WbW1cU1KZpi7CTtGNjYGZBb3Zs4fK4sgjE/OaHpcnOZXK1pcc7Yu2vuRkkZvL9KyyMlUGNiXrFUJ+PnD++cDWraEv7K4uLnIXXtiryGiQMHIk33dnZ+Tj2tqoXN96i9dyJPfNiBGMI7b2WttycpjJdcstVBybNzN1c8QIZj5OmNAzJdUYLsj19cAXvpD6tOAjjgAuvZQy1tcHujK0t1N2gOnK8WRXRsLtTFKWUVERP+Tm5sjHNTQwz1gZtNhCTYvIbwCcA6ATwCcArjPG7EvU+U8/nYvTCy/gwFASY2i9dnfTlZnqHHe7kJ/PlNIXXgi0vu9NaysX42nT6CqK5jPPzQ0s5r3dww5HoFV/ays/f7ebCufxx1n7ZPntjeE69u1vs2I51YhwIuWECXTBr1jBxwsLgXPPBU48MboXJh4sl5Hxd0xNGNYbeeABKodQ57Z2BMcdl7jXVTIOWygEAC8DuMMY0yUivwZwB4DvJOrkDgdw8cXAscdyQVu3jo8ffTTrbtI9AdAYWioOR3It9nCcdRa7Em/eTPeNZc1bSrOpCfja16g88vLCj+sNJpr3QaSnYikuBr70Je7IN23i5zFsGKuB09kIU4SuxEmTuGZ6vVSOyQhwe1weGBi0d7WjwJlgc+iYY9gj5IMPmGlkmTXGUDPv3Alcc03kwLOS9dhCIRhjXgq6+y6Ai5PxOqNGMQvOLuzfzwyf//wnkLp45JHs53PYYalbCN1u7sKfeoqZQpar2edjyulNNwUCujNnclZKpEBrc3PAVRwvQ4YkdtedSFyuxLmHQhHcAjvhCiE3F7j5Zracfu45+lC3bw+kSV1ySXrMMMVW2EIh9OJ6AE+mW4hks28f22ls3syFc+xYLsAbNnATd+aZLBBNlVLweJhteNFF3KF3d9PPP2ZMTxmmT2fVcHNz6KphY7jZvOEGbRwYL8EdT8vcSSjUcjrZg2jBAppkVVX8Et1uppD94AfAf/1X4qLkfqzf9RtvsA6toACYPZu/pcLChL6UMkBSphBE5BUAI0I89T1jzHP+Y74HoAvAYxHOcyOAGwFgTLjqR5tjDNtE7NjRs52Gw0GLfdgwWg2VlfRTp5KiosjrQWEhN5r33EO/f1lZYOFvbeV7OvZYuuKU+Ej6kJymJu5CCgv7Bo+rqvgF3nsvcOedCfOjtrYC999Pl2RBAXXPnj0sZHziCeDWW5neq9iDlO3hjDGnGmMmh7hZyuBaAGcDuNKY8PlxxpgHjDEzjTEzyzK03L2mhlXR4YrBcnKoGObN6ztvxA4cfjg3k4ccQs/D1q20dLxeuqFvvDE9sZBM58CQnGS1r3jvPWrx0tLQz1sZAIsWJeTlfD4qg7VrmbAwYgQ3HEOH0iJ2OtlNY/v2hLyckgBs4TISkTPAIPKJxpi2aMdnOsuW0aUbyR3k8QTSMntX/dqB6moGmvfupfvLqidQRdB/kj5X+bXXIgd/AFYTLlqUEH/lhg20DKqrQ5+qpIQ1cy++mNjiaKX/2EIhAPhvAHkAXvan271rjLkpvSIlj8bG2IKTDgcDz3Zm6NDQlcVK/CTdZdTcHL0c3ypK8XoHHEF//XW6iSLplYoK1qx87nNZ2n6+qwtYsyZgnVVUMIBSVWXLOdK2UAjGmAnpliGVDBsWvW+SMQzsJqpPjmJ/kj5XubSUO4xIFZidncwtTkCV5vbt0dsU5ebStdTSkoUKoa4O+MMfmEKYn8/PdNUqBvVnzEhPtWUUNA8kDcyaxcU+UieBpiamyY4alTq5lPSSdAvh5JPp44tEfT1wyikJ2b0WFITvfmthddTIui4BVlOvtjb6fCsqaEqPGsX7y5ZxfKDNgoSqENLAqFHM59+8ObRS6OxkJsYFF9jSqlSSRHDaaVI46iimm4ZTCs3NDAIlKLVt9uzYumUcdFAWNpZ89dXA+L/eiFAprFiR2p7uMaAKIU1cfz2Lz2pq2HG0s5M9cqx6oauu0pnjgw0rqJy0LCOPh6MuAc49aGmhqdrWxt1JSwubMyWoWnn6dLqMes/RsOjuZkLCWWdl2cbH66VCCG732xsRmlALF6ZOrhiwRQxhMFJYyGtvzRrgpZd4febmsj38iSdGntSlZCc5jhwU5BYkd4xmVRXw85+zRP7ll+ki8njY3XH27IRmCBQWMhPtrruoayoqAn2uGhp4O+cc9sjKKlpaYgvKFxUxjdBGqEJII7m5LAJLcGGoksEkfa4ywIXotNN4SzITJgA/+QlTS99+OzB06KCDmGo6bVqWWQdAIFIebbpSV5ftZvaqQlAUG5HUMZppYuRILv6f+xw3zy4X18GsUwQWHg813u7dkRtz7dvHHjU2QmMIimIjPC5P8mIIaaaggK1OYp1imLGIMDDS0BA+i6itjZbE0UenVrYoqEJQFBuRjRbCoGT6dLrkNm3qmWrl87H7465dwFe+Yrv0KnUZKYqNSNrUNCW1iABXXsnulc8/z6wRh4MK4YgjgPPOA8aPT7eUfVCFoCg2wuPyYHfb7nSLoSQCh4MT6GbPZjZXRweDJzbu9aIKQVFshLqMshCHI/1jGWNEYwiKYiPUZaSkE1UIimIj1EJQ0okqBEWxER6XB23eNviMvZqeKYMDVQiKYiM8Lg8MDPZ7bT4IQ8lKVCEoio1IesdTRYmAZhkpGUd3N2dSv/YaZ5Dk5wPHHMOiz3DjgjOFpM9VVpQIqEJQMorWVuC++4DVq9lNs6iI7ZWfeAJ46ingq18FpkxJt5T9J+lDchQlAuoyUjIGYzhkat06Dm6vqKBSKC7mvJHSUuD3v2e3gEzFmomgCkFJB6oQlIxh0yZg5crw88ndbrqP5s1LvWyJIulzlRUlAuoyUjKGt95i6+RInTLLyoAPPqAbyWZ9w2LCti6j5mbgo4/YwbOwEDj88MgTwZSMRBWCEjMtLRz52d3NUbGVlaltY1xfzxbKkXA4eGtuzkyFYLsso+5u4NlngQULONAlJ4ePAezoed11DOQoWYEtFIKI3AngPAA+ADsBfN4Ysy29UikWLS3A008Db75JP751Gz8euPzy1DVt9Hg4mTASlmzRphfaFVtZCMYAjz7K+cBjxrB/v4XPB3z4IXD33cB3vhNdUysZgV1iCL8xxhxpjJkK4HkAP0yzPIqflhbg178GXn8dGDGC/vsxY3irr+d43rVrUyPLMcdwrkgkGhuBUaPoOspEbJV2WlPDIfDV1T2VAUAzrKqKgZ133kmHdEoSsIVCMMY0Bd11AzDpkkXpybx5wNatfTeIIsCwYZwQeN99QGdn8mWZPJmuqt1hukN3dwN79nBwe6ZO5Cp0FgKwiYWwaBFNLUeEZaKsDJg/P/xkMCWjsIVCAAAR+bmIbAFwJSJYCCJyo4gsFZGlu3btSp2Ag5C2Nm4QKyvDH1NURCvio4+SL4/TCXzta1zsN29me3mAno3duzmD5LOfBWbNSr4sycIhDhQ6C+2hENavj17p5/Ew0BzNdFMygpQpBBF5RURWhbidBwDGmO8ZY6oAPAbgq+HOY4x5wBgz0xgzsyxT/QIZwrZtjCM6nZGPy8sD1qxJjUyjRwM/+QkX/n37gC1bqBxGjwa++U0Ocs9U68DC4/LYI+1UhNo2ElbQJtM/dAVACoPKxphTYzz07wBeAPCjJIqjxEB3d2zXucMRSDxJBUOHAhddxCmEra10ZbndqXv9ZONxedDitYGFMGUK8MorkT/cpiaakIWFqZNLSRq2cBmJyMSgu+cCWJcuWZQA5eXc/EVzD+/fz7hjqsnNZWppNikDwEZDck44gSZiuNQuYxi0OesstRCyBFsoBAC/8ruPPgLwGQC3plsghQHj6dOBHTvCH9PZyYV55szUyZXt2GZITmUlcPHF9Mn1jhF4vcxCmjaNXQWVrMAWdQjGmIvSLYMSmosuYiO5PXuYVRRMRwczkK6+WmuTEoltFALAYE1xMQtRgpM4nE4+d/75fVNSlYxFv0klIiNHAt/9LnD//dwQ5uQwZmAFm6+9FjjllHRLmV14XB7saIlglqUSEbqOZs8GNm5kzCAvD5g4UeMGWYgqBCUqVVUsQNuwgUVonZ3M6pk6VdeEZOB22SSGEExuLjBpUrqlUJKMKgQlJhwO4JBDeFOSi8fpsUelsjLosEtQWVEUP7aKISiDClUIimIz3C432rxt8BltB6GkFlUIimIzrAZ3bV5tB6GkFlUIimIzbNUCWxlUqEJQFJuhCkFJF6oQFMVmuJ02m5qmDBpUISiKzTgwJMcOHU+VQYUqBEWxGeoyUtKFKgRFsRlul7qMlPSgCkFRbIat5iorgwpVCIpiM9RlpKQLVQiKYjNUISjpQky0mak2RkR2AagNemg4gN1pEiccdpQJULnixY5y2VEmQOWKl3TINdYY02cofUYrhN6IyFJjjK1md9lRJkDlihc7ymVHmQCVK17sJJe6jBRFURQAqhAURVEUP9mmEB5ItwAhsKNMgMoVL3aUy44yASpXvNhGrqyKISiKoij9J9ssBEVRFKWfqEJQFEVRAGSpQhCRb4mIEZHh6ZYFAETkThH5SERWiMhLIlKZbpkAQER+IyLr/LI9KyKl6ZYJAETkEhFZLSI+EUlrOp6InCEi60Vko4jcnk5ZLETkLyKyU0RWpVuWYESkSkQWisha//d3qw1kyheR90XkQ79MP0m3TMGISI6IfCAiz6dbFiALFYKIVAE4DcDmdMsSxG+MMUcaY6YCeB7AD9Msj8XLACYbY44E8DGAO9Isj8UqABcCeCOdQohIDoD7AJwJ4DAAl4vIYemUyc/DAM5ItxAh6ALwTWPMoQCOAXCzDT6vDgAnG2OmAJgK4AwROSa9IvXgVgBr0y2ERdYpBAC/A3AbANtEy40xTUF33bCJbMaYl4wxXf677wIYnU55LIwxa40x69MtB4CjAGw0xnxqjOkE8ASA89IsE4wxbwDYm245emOM2W6MWe7/fzO40I1Ks0zGGGP1AHH6b7a4/kRkNIDPAngw3bJYZJVCEJFzAdQZYz5Mtyy9EZGfi8gWAFfCPhZCMNcD+E+6hbAZowBsCbq/FWle4DIFEakGMA3Ae2kWxXLLrACwE8DLxpi0y+Tn9+Dm1ZdmOQ6Qm24B4kVEXgEwIsRT3wPwXQCfSa1EJJJcxpjnjDHfA/A9EbkDwFcB/MgOcvmP+R5o7j+WCplilcsGSIjHbLG7tDMi4gHwNICv9bKO04IxphvAVH+M7FkRmWyMSWv8RUTOBrDTGLNMROamU5ZgMk4hGGNODfW4iBwBYByAD0UEoPtjuYgcZYzZkS65QvB3AC8gRQohmlwici2AswGcYlJYlBLH55VOtgKoCro/GsC2NMmSEYiIE1QGjxljnkm3PMEYY/aJyCIw/pLugPwcAOeKyFkA8gEUi8ijxpir0ilU1riMjDErjTHlxphqY0w1eDFPT4UyiIaITAy6ey6AdemSJRgROQPAdwCca4xpS7c8NmQJgIkiMk5EXAAuA/DvNMtkW4Q7sYcArDXG3JNueQBARMqs7DkRKQBwKmxw/Rlj7jDGjPavVZcBeC3dygDIIoVgc34lIqtE5CPQpZX2dDw//w2gCMDL/pTYP6VbIAAQkQtEZCuAYwG8ICIvpkMOf8D9qwBeBAOk/zDGrE6HLMGIyOMAFgM4RES2isgN6ZbJzxwAVwM42f97WuHfAaeTkQAW+q+9JWAMwRYpnnZEW1coiqIoANRCUBRFUfyoQlAURVEAqEJQFEVR/KhCUBRFUQCoQlAURVH8qEJQFEVRAKhCULIEEakWkf3+njXWY31aV4tIgT8/vnOg7dH953rd3xUVInKLv/VzXC1ARKRURL4yEFlieI0+LbNFxCUib4hIxnUsUJKDKgQlm/jE32I8bOtqY8x+/zGJaEFxPYBn/L1yAOArAM4yxlwZ53lK/X8bF0JivYYfRq+W2f4Orq8C+Fy8r61kJ6oQlIzBP3zlNP//fyYi90Y4PBWtq68EYDUI/BOAgwD8W0S+LiJX+QezrBCR/w2yIv4lIsv8w1pu9J/nVwDG+4/9jd/aCd7Jf0tEfuz/f7XfCrkfwHIAVeFeK5gILbP/5X8fiqIKQckofgR2jL0SbK389QjHJrV1tb+30UHGmBoAMMbcBFodJwFYAO665/itkW4EFt3rjTEzAMwEcIuIDANwO/zWjTHm2zG8/CEA/maMmQagMMJrxcIqALPiOF7JYtR3qGQMxpg3/A3UvgFgruWqEZE7waZqwSS7dfVwAPvCPHcKgBkAlvg77xaAvfgBKoEL/P+vAjARQLwNGGuNMe/G8FpRMcZ0++MpRf6hNsogRhWCkjH4W5yPBLDbWrxEZARC/47jbl0tIjcD+KL/7lkALgi+b4wJ/vv9YNvikKcC8IgxpsdIUn/f+1MBHGuMafO3Yg51ji70tN57H9Ma7bXiJA9A+wD+XskS1GWkZAQiMhIc4HMegFYROd3/1DQAK0L8Sdytq40x9/ndNlONMdt63+91bAOAHBEJtaC/CuBiESn3yz5URMYCKAHQ4FcGk8C5wwDQDHadtagHUC4iw0QkD5xXEY5wrxUTfpfVLmOMN9a/UbIXVQiK7RGRQgDPgAPc1wK4E8CP/U9PRQiFkKLW1S8BOC7Ea68B8H0AL/nbLr8MWjYLAOT6H7sTnGMNY8weAG/7W6T/xr84/xQcP/k8IvTvj/BaPYjQMvskAPP78+aV7EPbXysZjYg8BLp1xgB43hgzOca/qwEw0xizewCvPQ3AN4wxV/f3HOlGRJ4BcIcxZn26ZVHSj1oISkZjjLnBGOMDs2tKggvTQmEVpgFwYoDDzY0xH4DDV/qkeWYCflfav1QZKBZqISiKoigA1EJQFEVR/KhCUBRFUQCoQlAURVH8qEJQFEVRAKhCUBRFUfyoQlAURVEAqEJQFEVR/KhCUBRFUQAA/w/5hPYSWTcjQgAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_dataset(train_x,train_labels,W.numpy(),b.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's see how our model behaves on the validation data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "oEQswfCGrmHw",
        "outputId": "3cf61882-60e1-4baa-8e51-0c31ea80875c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.collections.PathCollection at 0x12892a01460>"
            ]
          },
          "execution_count": 61,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAebklEQVR4nO3dd5hcZf3+8fdnZqdsTUIIEmoQkBaiQJBqaKEjTRALTQyo/ARRkI6Aij8BRRRBjIqoVOlIEVBAmoAJUpKQQCACoaVn++zMzuf7x0ZMshuy7JyZM8/mfl3XXuyeyZ65Z69nb84+c855zN0REZFwJeIOICIipVGRi4gETkUuIhI4FbmISOBU5CIigauJ40lXX311HzVqVBxPLauAyZMnz3P3EXE8t8a2lNOKxnYsRT5q1CgmTZoUx1PLKsDM3ojruTW2pZxWNLY1tSIiEjgVuYhI4FTkIiKBU5GLiAQuyCLv7u5mxr9mMv3ZV+nu7o47jkhkmhe08OJj03h31vtxR5GAxHLWSimmPjWDCz53Kbn2HADpbJrv3XIqY8ZtHnMykYFzd3575nXcecX9pDIp8rk8W+y0Keffdhr1TXVxx5MqF9QReeuiNs7a94csen8xHS2ddLR0snhuM+cc8P9pnt8SdzyRAXvw2ke5+8oH6OrM07a4na7OPFOemM5Pjrsq7mgSgKCK/LFbn8aLvW+7691FHr35qRgSiUTjlsvupnPJX5n/lc/leebeybQ1t8eUSkIRVJE3z2smn8v32t7VmWfxvOYYEolEo2V+a5/bLZGgvbmjwmkkNEEV+ZhdtyCVSfXanqlL86ndRseQSCQaW+85hkSy969j49B6hq81LIZEEpKginyz7TZmm70+SbY+88G2bH2GT+42mtE7bxpjMpHSHPv9L1A/pI6adM/5B5YwMnVpvnX1CSQSQf2aSgyCOmvFzDjvz9/h4Rue4K/XPIy7s/exuzH+yHGYWdzxRAbsY+uP4DcvXcZtP/sLL/5jGmttuCaHn3YgG2/98bijSQCCKnKAZDLJnkftwp5H7RJ3FJFIDR85jBMuOTruGBIg/c0mIhI4FbmISOBU5CIigVORi4gETkUuIhK4kovczNY1s0fM7GUzm2pm34oimEjcNLYlFFGcflgATnX358ysEZhsZg+5+7QI9i0SJ41tCULJR+Tu/q67P7fk8xbgZWDtUvcrEjeNbQlFpHPkZjYK2Ap4po/HTjCzSWY2ae7cuVE+rUjZaWxLNYusyM2sAbgNOMXde92K0N0nuvtYdx87YsSIqJ5WpOw0tqXaRVLkZpaiZ6Bf7+63R7FPkWqgsS0hiOKsFQN+B7zs7peVHkmkOmhsSyiiOCLfCTgK2N3Mnl/ysV8E+xWJm8a2BKHk0w/d/QlA95CVQUdjW0KhKztFRAKnIhcRCZyKXEQkcCpyEZHAqchFRAKnIhcRCZyKXEQkcCpyEZHAqchFRAKnIhcRCZyKXEQkcCpyEZHAqchFRAKnIhcRCZyKXEQkcCXfj7xS3J3Xnv8PC99fxCbbbkTT8Ma4I4lEorM9x7SnZpCuTbPZ9huTTCbjjiSBCaLI586ez1n7/JD335hLsiZJV2eeL5x5MEef//m4o4mU5OEbH+dnJ/yaRDKBO2TrM1x0z1lsvPXH444mAQliauX8gy/mrRnv0NmWo21xO/lcnlt+cjf//MukuKOJDNhbM97msglX09mWo725g46WDha+t4gz9voBXbl83PEkIFVf5G/PfJc3X36bYndxme2dbTlu//m9MaUSKd39v/s7hXx3r+3d+W4mPfB85QNJsKq+yFsXtZNM9T1n2Dy/pcJpRKLTPK+F7kLvIncv0rqwLYZEEqqqL/INtlwPvPf2dDbFzodsV/lAIhHZ7oCxZOszvbYX8kU+tdsWMSSSUFV9kaczKU66agKZujSW6FnQPF2bZrWRwzjk5P1iTicycDseOJZPbLPhMmWerc/wuVP2Z431RsSYTEITxFkr4788jvU2XZs7rrifeW/N59P7bcV+x4+nvqku7mgiA5asSXLxQ+fx9+sf55GbnqS2IcN+x+/Jtnt/Ku5oEhhz72PeoszGjh3rkybpjBMpDzOb7O5j43hujW0ppxWN7aqfWhERkQ+nIhcRCZyKXEQkcCpyEZHAqchFRAKnIhcRCVwkRW5m15jZHDObEsX+RKqBxrWEIqoj8muBfSLal0i1uBaNawlAJEXu7o8BC6LYl0i10LiWUGiOXEQkcBUrcjM7wcwmmdmkuXPnVuppZRDywht4/kXcc3FHATS2JRruXXj+Jbww6yN/b8WK3N0nuvtYdx87YkTpd3ab/uyrnLLzuexX+yW+sM4J3Hb5PRSLxZV/owTLu9+nOO8QfN5n8QXH4nO2p9h+e9yxIh3b7kWKbb+nOGdniu+Npjj/C3jXCxEllWpV7Lgfn7MDvuBofN5BFOcdgBdm9/v7g5xamTXlTb67+4VMfWoG+Vye+e8s5Npzb+K3Z14XdzQpE3fHF34VCtOBTvBW8DZovmBQFZ23/ARaLofiHKAL8s/1/HLnX4k7mpSJ52fA4jPAW3rGNJ1QmIkvPIb+3tQwqtMPbwT+CWxiZrPN7KtR7HdFrv/hreQ6u5bZ1tme465fPkB7S0c5n1riUpgB3W8By6+ok8Pb/1CWp6z0uPZiK7T/CVh+DOfw1l+W86klRt5+A7D8Gq1FKC6A/L/7tY9I7kfu7l+MYj/9NfPfs/Bi7/9T1aSTvDdrDh8fs34l40glFOcDfS3559D9XlmestLjmu63wVLQa+6/CIVpFY0iFVR8j94HKAAGxXn92kWQUyvrbbYOZr23F7oKrLHe6pUPJOWXGg3e18ryGcjsWuk05ZEcCd7VxwMGNRtVPI5USHoXoLb3du+C1Fb92kWQRf7lcw8jXZteZlumLs34o3ahYWh9TKmknCwxBBq+Abb0gE9DcnWsrrIHzuViiSaoPQTILvdIBms4MY5IUgFWdwgk1wSWXr+1FuqOwZL9e/M8yCLfZOyGfP/OM1hnk7WwhFHbkOWQk/bj5CsnxB1NyijRcCI25HJI7wg1m0H917Dhd2KJxrijRcaazoe6o8HqgAQkN8CG/QpLjYk7mpSJWS02/DZoOBFqNofUdtjQS7HG0/q/j9CXest35alJ1WB9zbXIKmkwLPXW83uZxyy90n8rq44Vje0gFl/+MKl0Ku4IIpHrOTBRiUv/BDm1IiIi/6MiFxEJnIpcRCRwKnIRkcCpyEVEAqciFxEJnIpcRCRwKnIRkcCpyEVEAqciFxEJnIpcRCRwKnIRkcCpyEVEAqcil4pw78LzU/HCm3FHEYmUd8/H8y/hxZbYMgR/G1upfsWOe6H5e0ARvIDXbNSzWEJyzbijiQyYexe++AzofAgsDZ7H647EGk+v+PoIOiKXsvL8NFh8FngLeBuQg8J0fMFXiGNRE5GoePOPoPNvQBd4K5CDjhvw9hsqnkVFLmXl7X8Cll9QuBuK70JhahyRRErmnoeO24Dccg90QPvvKp5HRS7l1f0uUOzjgQR0z610GpFoeA7o7vux4sKKRgEVuZRbZhd6rwoPeB7Sn6x4HJFIWD30+R6PQWqbisdRkUtZWe3hkFydZdefrIX647DEanHFEimJmWFNF9JzkPLfNzaTYLVY4+kVz6OzVqSsLNEAw+/E266F3ENgQ7D6oyGzZ9zRREpimc/A8Ovx1olQeB3SY7D6r2M161c8i4pcys4STVjjydB4ctxRRCJlqS2xYVfEHUNTKyIioVORr4K88CbFRd+hOGcHivP2p9h+u87plkHh8duf4RvbnM7ha36V7x10MbOmrBpXEmtqZRXj3e/i8w9ZcnFOEYrzoflCvPs/WON34o4nMmB3XXk/vznjenLtPed2P33PZP79yBSu+OePGLXFujGnKy8dka9ivPW3PRctLHNudwe0/R4vNscVS6QkhXyBa8658YMSB3B3cm05/njBzTEmq4xIitzM9jGzGWY208zOjGKfUib5fwGF3tstBYXXKh6n2mlsh2HuW/Mpdve+8MzdmfbPV2NIVFklF7mZJYErgX2BzYEvmtnmpe5XyiS5Pv8773UpnofkyIrHqWYa2+EYMqKJ7j6KHOBj669e4TSVF8UR+aeBme7+urt3ATcBB0WwXykDazgeyCy3NQ3pHXQ3wt40tgNR11jL+C9/hkxtepntmbo0R553WEypKieKIl8beGupr2cv2bYMMzvBzCaZ2aS5c3WPjbhYagwM+SkkRtBT6GnI7okN/Vnc0aqRxnZATrpyAuOPGkc6myJTm6ZpeAMn/XIC2+6zVdzRyi6Ks1b6uvFur3PZ3H0iMBFg7NixOtctRonaPfHsHlCcA9aIJerjjlStNLYDkkqnOOXqr/H1y46ldWErw9YcSjKZjDtWRURR5LOBpc/tWQd4J4L9ShmZJVZw0x9ZisZ2gLJ1GbJ1y08fDm5RTK38C9jYzDYwszTwBeDuCPYrEjeNbQlCyUfk7l4ws28CDwBJ4Bp314oBEjyNbQlFJFd2uvt9wH1R7EukmmhsSwh0ZaeISOBU5CIigVORi4gETkUuIhI4FbmISOBU5CIigVORi4gETkUuIhI4FbmISOBU5CIigVORi4gETkUuIhI4FbmISOBU5CIigVORi4gETkUuIhI4FbmISOAiWSFIVg3uDvnnoXsW1GyKpTaPO5JIJLzYDl2Pg+chsxOWGBZ3pI9ERS794sVmfMHR0P2fJRscT43BVvsNZtlYs4mUwnNP4ov+Hz0TFA5ewJvOJVF3RNzR+k1TK9Iv3nwBFF4Fb+/5oAPyz+MtP4s5mcjAebEVX3TiknHdCt4G5KD5IrzwWtzx+k1FLivl3g2dDwD55R7JQcdtcUQSiUbuYfquwTzecVel0wyYilz6oQh09/2Qd1U0iUikvAO82McDRSi2VzzOQKnIZaXMUpD6FGDLPZKAzLgYEolEJDOOPg9SLItlx1c8zkCpyKuMu7No7mI623NxR1mGDfkhWAPw3zc2s2BDsaaz44wlAenK5Vn4/iK6u1fw110MLDkSGk4EavlfHdZCZg9Ibxdjso+mas5aaZ7fwh/Ov5nHb3uGVLqGfSfszhFnHEwqnYo7WsVMfugFfnbCr1nw3kIAdjr403x74tepa6yNORlYzUYw4iG8/RYovAKpLbHaQ7FEU9zRqt6UJ17mmnNu5I1ps1l7ozU59gdfYOvxY+KOVTHdhW5+c+Z13HP1gxSLTm19hgkXH8m+x+0RdzQAEg0n4umd8I47gC4suy+kd8Zs+b9Aq5e5e8WfdOzYsT5p0qQPvs515Jgw+jvMe3sBha4CAJnaNKM/sxk//uu5Fc8Xh1lT3uSk7c8mt9SReCpTw5af2ZyLHzwvxmThMbPJ7j42judefmz/++GXOO+zPybX8b/3EjJ1ac667lvsdPCn44hYcVef9gfuufpBcu1L/wwynH3Dt9jxwG1jTBaeFY3tqphaefTmp1g0Z/EHJQ6Q6+hiyhPTeWVyOKcAleK2y/5CPrfsWSH5XIEpT07nndfeiymVlOrXp/1xmRIHyLV38evT/hhTosrqyuW551fLljhArj3Hny68JaZUg09VFPnUp2bQ2dbXnLAz87lZFc8Th7dmvEOxu/e756l0DXPenBdDIonCG1Nn97n9vVnv012onrnicmlZ0MqK/ubXuI5OVRT5Op8YSbo23Wt7IplgjfVHxJCo8kbvvBmpdO+3LLpyedbfYt0PvvbCG3jb7/C2a/BC3yUh1WPYmkP73F4/tJ5Esip+/cpq6BpNZPr43QbYeJuPf/C5ezeeexRvvQrvuBv3zkpFHBSqYiTtdcyu1NQkl9mWSCYYOmIIW+0xOqZUlXXoKfuTqc9gif+9wZKpy7DfhD0YtsYQAIqtE/F5B+Atl/V8zNuXYtt1cUWWfvjSOYeSrc8ssy1Tl+GI0w8O6s20gUomk0z48ZfJ1PX+GRx30ReBJVdXzj8EX3QK3voLvPl8fO5ueOHNOCIHqaQiN7PDzWyqmRXNbMBvLg0dMYSfPnoho0avR026hpp0DWPGbc5l/7iQZDK58h0MAsNHDuOqSRezy+E70DS8kbU2WpPjLzmSEy//CgBeeB1arwBy9Fxh2dXzecvFePc7MSYfnKIa2/sfP56jzv88dU21pGvTZBuyHH7agRxx+kFRxq1q+00Yz5l/OokNP7k+jcMb2Hr8GH766IV8YpsNAfDWK6Dw+pJbPxR7LpMvLsQXnx5v8ICUdNaKmW1Gz2V/vwZOc/dJK/kWoPc7+0tbPK+ZmlSS+iH1A841GBVbfwWtv6D3xQsZrPG7WP3RccSqSlGctRL12C7kCyye10LT8IZV6pTa/ijO2RGKfc2X12BrPIslGiqeqVqtaGyXdB65u7+8ZOel7GYZQ1bXeckSv6jHdk2qhuEjw7o1qoSjYnPkZnaCmU0ys0lz586t1NMOGpbdC1jBkVxAlxIPRhrbJcoeACz/hmii56IzHY33y0qL3Mz+ZmZT+vj4SJN87j7R3ce6+9gRI1aNM1GiZDUbQsM3gQw9f0ilej5vPB1LrhVvuEBpbFcHazgZajYAqwOs5782FBtySdzRgrHSqRV31+FelUg0nIBn94LOh8ASkNkLq1l35d8ofdLYrg6WaIDhd0LuMSi8DMm1Ibu3Fiz5CKrmXivSP1YzChqOjzuGSKTMkpDdDdgt7ihBKvX0w0PMbDawA3CvmT0QTSyReGlsS0hKPWvlDuCOiLKIVA2NbQlJVVzZ+V8dbZ105ZZfTkwkbO7FnqsX+1yJRqR0VTFHPvP5WVw24Ve8/uIbmBk7HDiWU379NZpWa4w7msiAuTvefi20XtVztaI14A0nk6g/Mu5oMsjEfkQ+/92FnLrL+bz63Cy6C0UK+W7++ZfJnLHnD4jjXukiUfH266HlcvDFQAF8EbRcSrH91piTyWATe5HfO/Eh8kvdhxyg0FVg9ivvMP3ZmTGlEolA25VAx3IbO5bcM0ckOrEX+RvTZvdaUAHAEsa7H2FBBXdn1ktv8NoL/6FY1FykxMvdoTi/7weLcz7SvtpbOpj+7KvMe3sF+5NVXuxz5JtvvzHP3Du51woixUKRDcas3699vPrc61xw6KU0z2/BzKhtyHLeLacyeqdNyxFZZKXMDE+uA9193DM+Oapf+3B3rvvhrdz84ztJppLkcwU+tftozr3p21WxjqtUj9iPyPf+yu7UNtQuc5P9dDbFmF23YIPR6630+ztaO/juHhcy5815dLbl6GjtZMF7izh734tont9SzugiH67hDGD5qxOzWGP/bs/66M1P8edL7iLX0UV7cwf5XJ7nH36Jnxx3ZeRRJWyxF3nD0PoP7sNd11TLsI8N4fBTP8sFt3+3X9//xO3P0t3HEmnF7iIP3/hE1HFF+i1Ruzc29OdQs0nP/UNqNsOGXYll+3f14p8vvbPXEoj5XIGn75lM66K2ckSWQMU+tQIwYp3hnH3DKQP63oXvL+pzjj3X0cXC9xaVFkykRJbdrd/FvbxFc5r73J5IJmlZ2ErDUN2zX3rEfkReqi3Hbd7nWpfZhiyf3HWLGBKJRONTu4/uc13PbF2aNdZbPYZEUq2CL/JNP70RW48fs8yagJm6DJtuuxFb7bFljMlESnPMhUdQ11RLTapnuUMzyNSl+eYVX11llkCU/qmKqZVSmBnfu/VUHrz2Ue777d8pdhfZ69hd2W/CHqvE4rYyeK05ag0mvvBT/nzJXbzwj6mM/PjH+Px3D2KLHTeJO5pUmZLW7ByoD1uzc8a/ZnLXVX9l8dxmdjxwW8YfNY5MbabPfyvSlyjW7ByoFY3tfFeeh294gsdufZrGYfV89ht7q5DlIyvLmp1Ru++3f+OqU35PV2ceLzovPjqNu3/1AD9/8iKydSpzCVO+K893djmf/0x5k862HGbwxB3PcuwPjuCwb3827ngyCFTNHHlHawdXfev35Nq78GLPXwmd7TnefvU9Hrj2kZjTiQzcIzc++UGJA7hDrj3H78+5keYFutZBSlc1Rf7yMzNJpnq/gZNrz/H4rU/HkEgkGk/e8Wyv88EBatI1THl8egyJZLCpmiKvH1JHsdj3fH3jalpJW8LVuFo9luj9xrt7z7gXKVXVFPkntvk4w9Zo6nWmSaYuw4En7h1TKpHS7f+1vUhnU722Z+szjP6M7gckpauaIjczfnTfOYxYdzi1jVnqmmpJZVMced5hbLW7zgeXcG223cZM+PGRpLNp6ppqqWusZbWRw7j4gXN1PrhEoqrOWlnnE2vxp9evZNo/X6FlQStb7LgJTcO1SpCE7+Bv7sv4I8cx5Ynp1DXVssVOm6jEJTJVVeQAiURCt5+VQalhaD3bH7BN3DFkEKqaqRURERkYFbmISOBU5CIigVORi4gETkUuIhI4FbmISOBU5CIigVORi4gETkUuIhK4korczC41s+lm9qKZ3WFmQyPKJRIrjW0JSalH5A8Bo919DPAKcFbpkUSqgsa2BKOkInf3B929sOTLp4F1So8kEj+NbQlJlHPkxwH3r+hBMzvBzCaZ2aS5c+dG+LQiZaexLVVtpXc/NLO/AWv28dA57n7Xkn9zDlAArl/Rftx9IjARelYaH1BakQhpbMtgsdIid/fxH/a4mR0DHADs4e4axBIMjW0ZLEq6H7mZ7QOcAezi7u3RRBKJn8a2hKTUOfJfAo3AQ2b2vJldHUEmkWqgsS3BKOmI3N03iiqISDXR2JaQ6MpOEZHAqchFRAKnIhcRCZyKXEQkcCpyEZHAqchFRAKnIhcRCZyKXEQkcCpyEZHAlXRlp1QfL7bhHX+BwgwstRlk98cS9XHHEimZF17HO+4G78Cy4yE1FjOLO1ZVUJEPIt79Nj7/MCi2Ax14Rx20Xg7Db8OSI+OOJzJgxfabofkieu4o3I133ASZvWHIxSpzNLUyqPjiC6C4EOhYsqUdigvw5u/HmEqkNF5cAM0/ADrpKXIH74DcA9D1VMzpqoOKfJBwd+h6Aigu90gRco/FEUkkGrkngVTv7d6Bd65w4aZViop8UEl+xO0iAbAa6HP2JAGkKxymOqnIBwkzg+y+9D5ySUHtvnFEEolGehy9/9IESGN1B1c4THVSkQ8i1nQu1IwCqwMyPf+t2RBrPCfuaCIDZol6bOgVQC1QB2SBDDR8HUuNiTdcldBZK4OIJYbA8L9A19NQeB1qNoT09npXX4JnmXGwxhOQ+zt4J2TGYcm14o5VNVTkg4xZAjI79nyIDCKWaITag+OOUZVU5FIx3vUC3nkvYFjt/vqzWAYF907ouA/P/xuSo7C6Q7DEahXNoCKXiig2XwLt1wM5ALz9Rrz+KyQavx1vMJESeHEhPv9zUFwA3g5k8bYrYbXrsNTmFcuhNzul7Dw/A9qvo+dCpeKSj05ouwYvvBZvOJESeMsvoPv9JSUO0Aneii8+o6I5VORSfrmHgXwfDxQh90il04hEJ/dX+hzbhdfx4sKKxVCRS/lZmr4vSkqiCzokbB82fvu4GrVMVORSftl96fvSPIfs3pVOIxKd2sPoOa99aUlIb4MlGioWQ0UuZWfJtaDpQj64SIklFyw1XYQlPxZzOpGBs4avQXprsFogC1YPyZHYkEsqmkNnrUhFJOoOxbO7Qu5RwCCzK5YYFnMqkdKYpbHVrsXzL0F+KiTXhvSOmFX2/kYqcqkYS6wGtYfGHUMkcpbaElJbxvb8mloREQmcilxEJHAqchGRwKnIRUQCpyIXEQmcuXvln9RsLvBGHw+tDsyrcJxKGcyvDarr9a3v7iPieOIVjO1q+tmUw2B+fdX22voc27EU+YqY2SR3Hxt3jnIYzK8NBv/rK8Vg/9kM5tcXymvT1IqISOBU5CIigau2Ip8Yd4AyGsyvDQb/6yvFYP/ZDObXF8Rrq6o5chER+eiq7YhcREQ+IhW5iEjgqqrIzexSM5tuZi+a2R1mNjTuTKUys33MbIaZzTSzM+POEyUzW9fMHjGzl81sqpl9K+5M1UpjOyyhje2qmiM3s72Ah929YGYXA7h7ZVcxjZD13JT4FWBPYDbwL+CL7j4t1mARMbORwEh3f87MGoHJwMGD5fVFSWM7LKGN7ao6Inf3B929sOTLp4F14swTgU8DM939dXfvAm4CDoo5U2Tc/V13f27J5y3Ay8Da8aaqThrbYQltbFdVkS/nOOD+uEOUaG3graW+nk0VD4ZSmNkoYCvgmZijhEBjOyAhjO2KrxBkZn8D1uzjoXPc/a4l/+YcoABcX8lsZbCCFYcHFzNrAG4DTnH35rjzxEVjW2M7LhUvcncf/2GPm9kxwAHAHl5NE/gDMxtYd6mv1wHeiSlLWZhZip6Bfr273x53njhpbGtsx6Xa3uzcB7gM2MXd58adp1RmVkPPG0J7AG/T84bQl9x9aqzBImJmBvwBWODup8Qcp6ppbIcltLFdbUU+E8gA85dsetrdvx5jpJKZ2X7A5UASuMbdL4o3UXTMbGfgceAloLhk89nufl98qaqTxnZYQhvbVVXkIiLy0VXzWSsiItIPKnIRkcCpyEVEAqciFxEJnIpcRCRwKnIRkcCpyEVEAvd/OWSI7uIq5ngAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "pred = tf.matmul(test_x,W)+b\n",
        "fig,ax = plt.subplots(1,2)\n",
        "ax[0].scatter(test_x[:,0],test_x[:,1],c=pred[:,0]>0.5)\n",
        "ax[1].scatter(test_x[:,0],test_x[:,1],c=valid_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To compute the accuracy on the validation data, we can cast boolean type to float, and compute the mean:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HUjdeIefsIsg",
        "outputId": "f267f505-8ba4-43ef-9ebe-df124c3c05a1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(), dtype=float32, numpy=0.46666667>"
            ]
          },
          "execution_count": 62,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tf.reduce_mean(tf.cast(((pred[0]>0.5)==test_labels),tf.float32))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's explain what goes on here:\n",
        "* `pred` is the values predicted by the network. They are not quite probabilities, because we have not used an activation function, but values greater than 0.5 correspond to class 1, and smaller - to class 0.\n",
        "*  `pred[0]>0.5` creates a boolean tensor of results, where `True` corresponds to class 1, and `False` - to class 0\n",
        "* We compare that tensor to expected labels `valid_labels`, getting the boolean vector or correct predictions, where `True` corresponds to the correct prediction, and `False` - to incorrect one.\n",
        "* We convert that tensor to floating point using `tf.cast`\n",
        "* We then compute the mean value using `tf.reduce_mean` - that is exactly our desired accuracy  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_95qF9lY2kHp"
      },
      "source": [
        "## Using TensorFlow/Keras Optimizers\n",
        "\n",
        "Tensorflow is closely integrated with Keras, which contains a lot of useful functionality. For example, we can use different **optimization algorithms**. Let's do that, and also print obtained accuracy during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ups7nlV22ofp",
        "outputId": "aa4dff06-82b9-4b2f-ca00-33970ea2b989"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: last batch loss = 4.7787, acc = 1.0000\n",
            "Epoch 1: last batch loss = 8.4343, acc = 0.5000\n",
            "Epoch 2: last batch loss = 8.3255, acc = 0.5000\n",
            "Epoch 3: last batch loss = 7.5579, acc = 0.5000\n",
            "Epoch 4: last batch loss = 6.5254, acc = 0.5000\n",
            "Epoch 5: last batch loss = 7.3800, acc = 0.5000\n",
            "Epoch 6: last batch loss = 7.7586, acc = 0.5000\n",
            "Epoch 7: last batch loss = 10.4724, acc = 0.0000\n",
            "Epoch 8: last batch loss = 9.4423, acc = 0.5000\n",
            "Epoch 9: last batch loss = 4.1888, acc = 1.0000\n",
            "Epoch 10: last batch loss = 11.2127, acc = 0.0000\n",
            "Epoch 11: last batch loss = 9.0417, acc = 0.5000\n",
            "Epoch 12: last batch loss = 7.9847, acc = 0.5000\n",
            "Epoch 13: last batch loss = 3.7879, acc = 1.0000\n",
            "Epoch 14: last batch loss = 6.8455, acc = 0.5000\n",
            "Epoch 15: last batch loss = 6.5204, acc = 0.5000\n",
            "Epoch 16: last batch loss = 9.2386, acc = 0.5000\n",
            "Epoch 17: last batch loss = 6.2447, acc = 0.5000\n",
            "Epoch 18: last batch loss = 3.9107, acc = 1.0000\n",
            "Epoch 19: last batch loss = 5.7645, acc = 1.0000\n"
          ]
        }
      ],
      "source": [
        "optimizer = tf.keras.optimizers.Adam(0.01)\n",
        "\n",
        "W = tf.Variable(tf.random.normal(shape=(2,1)))\n",
        "b = tf.Variable(tf.zeros(shape=(1,),dtype=tf.float32))\n",
        "\n",
        "@tf.function\n",
        "def train_on_batch(x, y):\n",
        "  vars = [W, b]\n",
        "  with tf.GradientTape() as tape:\n",
        "    z = tf.sigmoid(tf.matmul(x, W) + b)\n",
        "    loss = tf.reduce_mean(tf.keras.losses.binary_crossentropy(z,y))\n",
        "    correct_prediction = tf.equal(tf.round(y), tf.round(z))\n",
        "    acc = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
        "    grads = tape.gradient(loss, vars)\n",
        "    optimizer.apply_gradients(zip(grads,vars))\n",
        "  return loss,acc\n",
        "\n",
        "for epoch in range(20):\n",
        "  for step, (x, y) in enumerate(dataset):\n",
        "    loss,acc = train_on_batch(tf.reshape(x,(-1,2)), tf.reshape(y,(-1,1)))\n",
        "  print('Epoch %d: last batch loss = %.4f, acc = %.4f' % (epoch, float(loss),acc))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvAiaj_JndyP"
      },
      "source": [
        "**Task 1**: Plot the graphs of loss function and accuracy on training and validation data during training\n",
        "\n",
        "**Task 2**: Try to solve MNIST classificiation problem using this code. Hint: use `softmax_crossentropy_with_logits` or `sparse_softmax_cross_entropy_with_logits` as loss function. In the first case you need to feed expected output values in *one hot encoding*, and in the second case - as integer class number."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "995iCprDrgYQ"
      },
      "source": [
        "## Keras\n",
        "### Deep Learning for Humans\n",
        "\n",
        "* Keras is a library originally developed by Francois Chollet to work on top of Tensorflow, CNTK and Theano, to unify all lower-level frameworks. You can still install Keras as a separate library, but it is not advised to do so. \n",
        "* Now Keras is included as part of Tensorflow library\n",
        "* You can easily construct neural networks from layers\n",
        "* Contains `fit` function to do all training, plus a lot of functions to work with typical data (pictures, text, etc.)\n",
        "* A lot of samples\n",
        "* Functional API vs. Sequential API\n",
        "\n",
        "Keras provides higher level abstractions for neural networks, allowing us to operate in terms of layers, models and optimizers, and not in terms of tensors and gradients.  \n",
        "\n",
        "Classical Deep Learning book from the creator of Keras: [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python)\n",
        "\n",
        "### Functional API\n",
        "\n",
        "When using functional API, we define the **input** to the network as `keras.Input`, and then compute the **output** by passing it through a series of computations. Finally, we define **model** as an object that transforms input into output.\n",
        "\n",
        "Once we obtained **model** object, we need to:\n",
        "* **Compile it**, by specifying loss function and the optimizer that we want to use with our model\n",
        "* **Train it** by calling `fit` function with the training (and possibly validation) data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QJWplVfy34Eo",
        "outputId": "9be976f2-4f9a-495c-bddc-a7f9ec30989a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 2)]               0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 1)                 3         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 3\n",
            "Trainable params: 3\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/15\n",
            "9/9 [==============================] - 1s 2ms/step - loss: 0.7812 - accuracy: 0.2857\n",
            "Epoch 2/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.7142 - accuracy: 0.4000\n",
            "Epoch 3/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.6683 - accuracy: 0.6143\n",
            "Epoch 4/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.6221 - accuracy: 0.8429\n",
            "Epoch 5/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.5843 - accuracy: 0.8857\n",
            "Epoch 6/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.5447 - accuracy: 0.9429\n",
            "Epoch 7/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.5135 - accuracy: 0.9286\n",
            "Epoch 8/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4878 - accuracy: 0.9429\n",
            "Epoch 9/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4679 - accuracy: 0.9429\n",
            "Epoch 10/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4446 - accuracy: 0.9429\n",
            "Epoch 11/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4349 - accuracy: 0.8714\n",
            "Epoch 12/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4156 - accuracy: 0.9286\n",
            "Epoch 13/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.4019 - accuracy: 0.9429\n",
            "Epoch 14/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.3908 - accuracy: 0.9286\n",
            "Epoch 15/15\n",
            "9/9 [==============================] - 0s 2ms/step - loss: 0.3777 - accuracy: 0.9286\n"
          ]
        }
      ],
      "source": [
        "inputs = tf.keras.Input(shape=(2,))\n",
        "z = tf.keras.layers.Dense(1,kernel_initializer='glorot_uniform',activation='sigmoid')(inputs)\n",
        "model = tf.keras.models.Model(inputs,z)\n",
        "\n",
        "model.compile(tf.keras.optimizers.Adam(0.1),'binary_crossentropy',['accuracy'])\n",
        "model.summary()\n",
        "h = model.fit(train_x_norm,train_labels,batch_size=8,epochs=15)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "K2Kf60IrZcqs",
        "outputId": "b60b868d-3562-4715-f5d5-1f9764e45f09"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x12894b95250>]"
            ]
          },
          "execution_count": 65,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAiPElEQVR4nO3deXxV9Z3/8dcnG3vCFrYEZAv7ZqVYtS4VQVQqtvbXol2ctj7QTq12ZtoprdO9U7V2av1VO9TpOE6tI9OxOiKgQe3UWmsdQCHJZQ0gELIQtiRs2e5n/siVhpDlAjc59968n49HHuSce3Lv+0Hufefc7/2ec8zdERGRxJcSdAAREYkNFbqISJJQoYuIJAkVuohIklChi4gkibSgHnjw4ME+evTooB5eRCQhrV+//oC7Z7d2W2CFPnr0aNatWxfUw4uIJCQz293WbRpyERFJEip0EZEkoUIXEUkSKnQRkSShQhcRSRIqdBGRJKFCFxFJEoHNQ5fuY8PeI7y9+zCTh2cyNSeTzJ7pQUc6g7tTXn2SwpIqtlXUUNcQDjpS1GaO7M/cyUODjtHlGsPOjsqjFJZUUXm0lonD+jFtRBbZ/XoEHS0wKnTpNO7Ov73xLv+4ejON4b+cd3/0oN5My8liWk4W03OymDoik/69M7o0174jJyjaV0XRvmoK91URKq3iwNG6U9uYdVmc8/Le5Qy+v2gqn75kdKBZOlNDY5jt+482/a72VVG4r4pNZdWcrD/zD++wzJ5My8k89fyalpPF0MyeAaTueip06RQn6hpZ+mwBz28oZd6UoXzzhinsPHD0VIm+s+cIKwvKTm0/cmCvSLn/5UU4sM/5l7y7s/fQCQr3VVFUWhV5/CoOH68HIDXFyBvSlw9NHHLqj8yU4Zn0ykg978fuCrUNjXzxqXf45vMhwg63XTo66Ejnra4hzLaKmqbfVWkVhfuq2VJWTW3kXVOfjFSmjsjiljmjmB4p7SH9erK5vPrU77eotJpXt+w/9Qcvu18Ppo3IbHqORX5meFZPLFH+ckfJgrpi0ezZs12H/ienPQePs+TJdWytqOEr8yfyhSvHkZJy5gvn8LG6yAu2ilBkT3nPoeOnbs/p36tpT2tEFtNym16Eg/u2/XY6HHZ2Hzp+6kVdGPm3+mQDAOmpxoSh/U57UU8a1o+e6YlR3m2pawjzpaffJj9UwbcWTuFzHxwTdKSo1TY0srW8JvK7airkreU11DU2lXe/HmlMjTwHpuc2/cEdM6hPq8+nlo7VNrCprPq050Lx/qO892ZxUJ+MyPMg8hzLySJ3QK+4L3kzW+/us1u9TYUusfT7rfu5Z/kGAP7/LRdy5YRWzyHUpqrj9YQiJV9U2vRi3HXg2Knbm95OZ50q+mN1DRSWRN6Cl1ZTU9tU3hmpKUwa3u/UHv/0nCwmDOtLj7TELu+21DeGufvpd3ixqJx/uGEyt18+NuhIrSraV8U7ew6fGuraVlFDQ6RhM3umNZX2iL8Mx40a2Duq8o7WibpGNpVVNz3HIs+b7fuPnhoS7N87/dTjT8vJJKd/5xT8kH49GNG/1zn9rApdOl047Pz898X808vbmDQsk1986iJGDeodk/uuOVlPqLT6tD3vnQeOnXo73SMthcnDM5mWk3lquCZvSD8y0rrXJK76xjBfXr6BVYVlfP26Sdxx5bigI51S1xDmB6s28as3m84rNaB3+mmfo0wPcO/4ZH0jW8prTnt+bauoob6x87rxzivHsfS6Sef0s+0VusbQ5bxVn6zn736zkZc3VXDTrBHc99EZMR2D7tcznQ+MHcQHxg46te5YbQNbyqvpnZHG+CF9SU/tXuXdmvTUFB5ePIuUFOO+F7fQ6M5fXzU+6Fjsrz7JXz/1Nut2H+b2D47hry4b3Wl7vueiZ3oqs0b2Z9bI/qfW1TY0sq38KAeO1nbKY44cGJudnZZU6HJetlfUcMeT69l96Djf/vAU/urS0V3yQu3TI42LLhjY6Y+TaNJSU3jo4zNJMfjRS1sJh527rs4LLM+6dw/xhafe5lhtA4/ceiELZ4wILMvZ6JGWyvTcrKBjnDUVupyzFwvL+Mp/baRXRir/cfvFXNxsD1qCk5aawk8+PotUM368ZhuNYbjnmq4tdXfnyT/v5nsvbCJ3QC9+/fmLmTisX5dm6I5U6HLWGsPOg/lbWfbaDi4c1Z9//uRFDMvqHvN8E0VqivHg/5tJSorx0CvbCLvz5WvyuuTd08n6Ru59rojfvl3C3ElD+MknZpHVK/4OJktGKnQ5K4eP1XH38nd4ffsBPnnxKL714SlJO3Mk0aWmGD+6eQYpBg+/up2wO387b0KnlvreQ8e589frCZVW8+Vr8rj76ryYzlKR9kVV6Ga2AHgYSAV+6e73t7h9APA4MA44CXzO3YtinFUCVrSvijueXE9lTS0/unkGH3//yKAjSQdSUoz7PzqD1BTjZ78rpjHsfPXaiZ1S6n/cfoAvPf02DWHnX2+b3S1PRxC0DgvdzFKBR4F5QAmw1sxWuPumZpt9A9jg7h8xs0mR7ed2RmAJxm/Xl/CN5woZ2CeD/7rzEmY2mxEg8S0lxfjHm6aTYsbPf7+DRneWLpgUs1J3d37xh5386KUt5A3pxy8+fRGjB/eJyX3L2YlmD30OUOzuOwHMbDmwCGhe6FOA+wDcfYuZjTazoe5eEevA0rWazx++ZOwgfnbrhe0erSnxKSXF+MFN00gx4xev7SQcdr5x/eTzLvWjtQ189b828mJROQtnDOeBm2fQp4dGcoMSzf98DrC32XIJcHGLbTYCHwX+aGZzgAuAXOC0QjezJcASgFGjRp1jZOkqzecPL7liLH9/7UTSNN87YZkZ31s0ldQU419e30VjGL658NxLfUflUe54cj07K49y7/WTuf3yMXEzt7y7iqbQW/sNtTyE6n7gYTPbABQC7wANZ/yQ+2PAY9B0pOhZJZUu9d784aMnG/jZLRfy4ZmJMX9Y2mdmfPvDU0gx4/E3dhF259sfnnLWRbwmVM7f/WYj6Wkp/PrzF3Pp+MGdlFjORjSFXgI0//QrFyhtvoG7VwOfBbCmZ8auyJckGHfn13/ezXc1fzhpmRnfXDiZ1BT4l9ebSv27N06NqtQbw85PX9nGz35XzIzcLP75UxeRc47nJJHYi6bQ1wJ5ZjYG2AcsBm5tvoGZ9QeOu3sdcDvwh0jJSwI5XtfAt54P8cz6Eq6eNISHNH84aZkZ37h+MikpTWPqjWHn+4umtTvFsOp4PXcvf4fXtlXy8dm5fG/RtIQ/U2Wy6bDQ3b3BzO4C8mmatvi4u4fM7M7I7cuAycCvzKyRpg9LP9+JmSXG3J0Xi8r5wcpNlFad5J65edwzV/OHk52ZsXTBJFIjs1/C7k2zYVr5vW8uq+aOJ9dTVnWCH35kOrfMGanx8jgU1cfR7r4aWN1i3bJm378JBHfCCDln2ytq+M4LId4oPsikYf346eILmTNG50jpLsyMr1478dQ89XAY7vvo6aX+/IZ9fO23BWT1Suc/77iE940aEGBiaY/mF3VTNSfrefiV7Tzxp3fpnZHK9xZN5dY5ozSLpRsyM/523gRSzHj41e00uvPAzTMIu3Pf6i08/sYu5oweyKOffF+3vl5nIlChdzPuznPv7OO+F7dw4Ggti98/kq/Mn8ggzS3v1syMv4mU+kOvbKO+MUx51Une2nWIz142mm9cP1mnKE4AKvRupGhfFd9ZEWLd7sPMHNmfX35mto74lNPcc00eqSnw4zXb6JmewkOfmMlHLswNOpZESYXeDRw5XseP12zlP97aw4DeGfzo5hl87KJcfegprbrr6jzGD+nH2Ow+TBiqKauJRIWexBrDzn+u3cuD+VuoOlHPZy4Zzd/Mm6CpiNKhBdOGBR1BzoEKPUm9vecw334+ROG+KuaMGch3b5zK5OGZQccSkU6kQk8ylTW1PPDSFp5ZX8LQzB48vHgWN84coTnDIt2ACj1JNDSG+dWbu3no5W2cbGjkjivH8qWr8+irM9+JdBt6tSeBN3cc5DsrQmytqOHyvMF858apjMvuG3QsEeliKvQEVlZ1gh+u3sILG0vJ6d+LX3z6IuZPGarhFZFuSoWeoH75+k5+8vI2GsPOPXPz+MJV43SiJJFuToWegApKjvCDVZv50MRsvrdoGiMH9g46kojEARV6AsoPlZOaYjz0iVn0750RdBwRiRM6OUMCyg9VcPGYgSpzETmNCj3B7Kg8SvH+o1w7VUfyicjpVOgJJj9UDsD8qUMDTiIi8UaFnmDyQxXMzM1ieJau4ygip1OhJ5DyqpNs3HuE+RpuEZFWqNATyJpNTcMt12q4RURaoUJPIPmhcsZm92H8EJ2jWkTOpEJPEEeO1/HnnYc0u0VE2hRVoZvZAjPbambFZra0lduzzOwFM9toZiEz+2zso3Zvr27eT2PYVegi0qYOC93MUoFHgeuAKcAtZjalxWZfBDa5+0zgKuCfzExHvcRQfqicYZk9mZGTFXQUEYlT0eyhzwGK3X2nu9cBy4FFLbZxoJ81neavL3AIaIhp0m7sRF0jf9heyfypQ3UdUBFpUzSFngPsbbZcElnX3CPAZKAUKATucfdwyzsysyVmts7M1lVWVp5j5O7ntW2VnKwPa7hFRNoVTaG3tkvoLZavBTYAI4BZwCNmdsYFLN39MXef7e6zs7OzzzJq97UmVE5Wr3TmjBkYdBQRiWPRFHoJMLLZci5Ne+LNfRZ41psUA7uASbGJ2L3VN4Z5ZXMFcycPIT1Vk5JEpG3RNMRaIM/MxkQ+6FwMrGixzR5gLoCZDQUmAjtjGbS7emvnIapPNmi4RUQ61OH50N29wczuAvKBVOBxdw+Z2Z2R25cB3weeMLNCmoZovubuBzoxd7eRHyqnZ3oKV+RpiEpE2hfVBS7cfTWwusW6Zc2+LwXmxzaahMPOmk3lXDkhm14ZuryciLRPg7JxbGPJESqqazXcIiJRUaHHsTWbKkhLMeZO0sm4RKRjKvQ4lh8q5wNjB5HVOz3oKCKSAFTocap4fw07K4/pVLkiEjUVepzKD1UAMG+Kxs9FJDoq9DiVHypn1sj+DMvqGXQUEUkQKvQ4VHrkBAUlVboQtIicFRV6HFoTeu9ScxpuEZHoqdDjUH6ogvFD+jIuu2/QUUQkgajQ48zhY3X877uHNLtFRM6aCj3OvLK5QpeaE5FzokKPM/mhCkZk9WS6LjUnImdJhR5Hjtc18Pr2SuZPHUbT1fxERKKnQo8jr22tpLYhrOmKInJOVOhxJD9UzoDe6cwZrUvNicjZU6HHibqGMK9u2c/cyUNJ06XmROQcqDnixJ93HqRGl5oTkfOgQo8T+aFyemekcnne4KCjiEiCUqHHgXDYeXlTBVdOyKZnui41JyLnRoUeBzaUHGF/jS41JyLnR4UeB/JD5aSlGB+aNCToKCKSwKIqdDNbYGZbzazYzJa2cvtXzWxD5KvIzBrNTHPvouDurAlVcMm4QWT10qXmROTcdVjoZpYKPApcB0wBbjGzKc23cfcH3X2Wu88Cvg685u6HOiFv0tm+/yi7DhzTcIuInLdo9tDnAMXuvtPd64DlwKJ2tr8FeDoW4bqD/KJyzGD+FB0dKiLnJ5pCzwH2Nlsuiaw7g5n1BhYAv23j9iVmts7M1lVWVp5t1qSUv6mcC0f2Z0imLjUnIucnmkJv7SxR3sa2HwbeaGu4xd0fc/fZ7j47Ozs72oxJq+TwcYr2VTNfwy0iEgPRFHoJMLLZci5Q2sa2i9FwS9TWhCoAXWpORGIjmkJfC+SZ2Rgzy6CptFe03MjMsoArgedjGzF55YfKmTC0L2MG9wk6iogkgQ4L3d0bgLuAfGAz8Bt3D5nZnWZ2Z7NNPwKscfdjnRM1uRw8Wsvadw9p71xEYiYtmo3cfTWwusW6ZS2WnwCeiFWwZPfq5v2EXcMtIhI7OlI0IPmhcnL692LqiMygo4hIklChB+BobQOvFx9g/tShutSciMSMCj0Ar22tpK4hrOEWEYkpFXoA8kPlDOyTwft1qTkRiSEVeherawjzP1v2c83kIaSmaLhFRGJHhd7F/rTjADW1utSciMSeCr2L5Ycq6JORymXjdak5EYktFXoXeu9Sc1dNHKJLzYlIzKnQu9A7ew9z4Ggt86fqVLkiEnsq9C6UH6ogPVWXmhORzqFC7yLuTn6onEvHDSazpy41JyKxp0LvIlsrath98Lhmt4hIp1Ghd5H8ogrMYJ4uNScinUSF3kXyQ+VcNGoA2f16BB1FRJKUCr0L7D10nE1l1RpuEZFOpULvAvmhcgBNVxSRTqVC7wJrQhVMGtaPCwbpUnMi0nlU6J3swNFa1u4+xHwNt4hIJ1Ohd7JXNlXgDtdquEVEOpkKvZPlh8rJHdCLKcN1qTkR6Vwq9E5UdaKePxYfYMHUYbrUnIh0uqgK3cwWmNlWMys2s6VtbHOVmW0ws5CZvRbbmInp5U0V1Dc6C2eOCDqKiHQDaR1tYGapwKPAPKAEWGtmK9x9U7Nt+gM/Bxa4+x4z09mngJUFpeQO6MXM3Kygo4hINxDNHvocoNjdd7p7HbAcWNRim1uBZ919D4C7749tzMRz5Hgdf9x+gBtmDNdwi4h0iWgKPQfY22y5JLKuuQnAADP7vZmtN7PPtHZHZrbEzNaZ2brKyspzS5wg1oQqaAg7C6druEVEukY0hd7a7qW3WE4DLgJuAK4FvmlmE874IffH3H22u8/Ozs4+67CJ5IWCUkYN7M20HM1uEZGuEU2hlwAjmy3nAqWtbPOSux9z9wPAH4CZsYmYeA4dq+NPOw6yUMMtItKFoin0tUCemY0xswxgMbCixTbPA5ebWZqZ9QYuBjbHNmriyA+V0xh2bpgxPOgoItKNdDjLxd0bzOwuIB9IBR5395CZ3Rm5fZm7bzazl4ACIAz80t2LOjN4PFtZUMqYwX10MJGIdKkOCx3A3VcDq1usW9Zi+UHgwdhFS0wHjtby5o6DfPFD4zXcIiJdSkeKxthLReWEHQ23iEiXU6HH2MqCUsZl92Hi0H5BRxGRbkaFHkP7a07y1q5DLJwxQsMtItLlVOgx9GJhOe6wUMMtIhIAFXoMrSooY+LQfuRpuEVEAqBCj5HyqpOs3X1IH4aKSGBU6DGyurAM1+wWEQmQCj1GVhWWMXl4JuOy+wYdRUS6KRV6DJQeOcH63Yf1YaiIBEqFHgOrC8sAuGG6Cl1EgqNCj4GVBWVMy8lk9OA+QUcRkW5MhX6e9h46zoa9R7hBF7IQkYCp0M+ThltEJF6o0M/TqsIyZuZmMWpQ76CjiEg3p0I/D3sOHqegpEpzz0UkLqjQz8PKwqYr8V2v4RYRiQMq9POwqqCMC0f1J3eAhltEJHgq9HO068AxQqXV+jBUROKGCv0crSpoGm7R+LmIxAsV+jlaWVDG7AsGMDyrV9BRREQAFfo5Kd5/lC3lNdo7F5G4ElWhm9kCM9tqZsVmtrSV268ysyoz2xD5+lbso8aPVQVlmGl2i4jEl7SONjCzVOBRYB5QAqw1sxXuvqnFpq+7+8JOyBh3VhWW8v7RAxma2TPoKCIip0Szhz4HKHb3ne5eBywHFnVurPi1raKGbRVHdapcEYk70RR6DrC32XJJZF1Ll5jZRjN70cymtnZHZrbEzNaZ2brKyspziBu8lQVlpBgsmDYs6CgiIqeJptCtlXXeYvlt4AJ3nwn8DPjv1u7I3R9z99nuPjs7O/usgsYDd2dVQSkXjxnEkH4abhGR+BJNoZcAI5st5wKlzTdw92p3Pxr5fjWQbmaDY5YyTmwpr2FH5THNbhGRuBRNoa8F8sxsjJllAIuBFc03MLNhZmaR7+dE7vdgrMMGbVVkuOU6DbeISBzqcJaLuzeY2V1APpAKPO7uITO7M3L7MuBjwBfMrAE4ASx295bDMgnN3VlVWMal4wYzqG+PoOOIiJyhw0KHU8Moq1usW9bs+0eAR2IbLb6ESqvZdeAYS64YG3QUEZFW6UjRKK0qLCM1xVgwVcMtIhKfVOhRaJrdUsZl4wczoE9G0HFERFqlQo9C4b4q9hw6roOJRCSuqdCjsKqgjPRU49opGm4RkfilQu+Au7OyoIwPjh9MVu/0oOOIiLRJhd6BDXuPsO/ICRbOGBF0FBGRdqnQO7CqoIyM1BSumTI06CgiIu1SobcjHHZWF5ZxxYTBZPXScIuIxDcVejve2XuY0qqTGm4RkYSgQm/HyoIyMtJSmDt5SNBRREQ6pEJvw3vDLVdNyKZfTw23iEj8U6G3Yd3uw1RU17JwpoZbRCQxqNDbsKqglB5pKcydpOEWEUkMKvRWNIad1UXlXD1pCH16RHVCShGRwKnQW/G/uw5RWVOr2S0iklBU6K1YVVhKr/RUPjQp8a57KiLdlwq9hYbGMC8WljN38hB6Z2i4RUQShwq9hbd2HeLgsTqdKldEEo4KvYWVBWX0yUjlqoma3SIiiUWF3kx9Y5iXisq4ZspQeqanBh1HROSsqNCbeXPHQQ4fr+eG6RpuEZHEE1Whm9kCM9tqZsVmtrSd7d5vZo1m9rHYRew6qwrK6NcjjSsmaHaLiCSeDgvdzFKBR4HrgCnALWY2pY3tHgDyYx2yK9Q1hHkpVM48DbeISIKKZg99DlDs7jvdvQ5YDixqZbsvAb8F9scwX5d5Y8cBqk7Uc4Nmt4hIgoqm0HOAvc2WSyLrTjGzHOAjwLLYRes67s4Tb7xLZs80Ppg3OOg4IiLnJJpCt1bWeYvlnwJfc/fGdu/IbImZrTOzdZWVlVFG7HwrC8p4bVslX75mAj3SNNwiIokpmkMhS4CRzZZzgdIW28wGlpsZwGDgejNrcPf/br6Ruz8GPAYwe/bsln8UAlF1vJ7vvhBiRm4Wt106Oug4IiLnLJpCXwvkmdkYYB+wGLi1+QbuPua9783sCWBlyzKPV/e/tJnDx+t54rNzSE1p7c2IiEhi6LDQ3b3BzO6iafZKKvC4u4fM7M7I7Qk5bg7w1s6DPP2/e7njirFMy8kKOo6IyHmJ6uxT7r4aWN1iXatF7u5/df6xOl9tQyNff66QkQN7cc81eUHHERE5b932dII//58d7Kw8xr9/bo7OqigiSaFbHvpfvL+Gn/++mJtmjeBKHRUqIkmi2xV6OOx8/dlC+vRI4x8WnnHAq4hIwup2hb587V7WvnuYe6+fzOC+PYKOIyISM92q0PdXn+S+FzdzydhBfOyi3KDjiIjEVLcq9O++sInahjA//Oh0IgdBiYgkjW5T6K9sqmBVYRl3Xz2eMYP7BB1HRCTmukWhH6tt4FvPFzFhaF+WXDEu6DgiIp2iW0zA/qc12yirPskzt15KRlq3+BsmIt1Q0rfbxr1HeOJPu/jUxRdw0QUDgo4jItJpkrrQ6xvDLH22kOx+PfjqgolBxxER6VRJPeTy+B93sbmsmmWfuojMnulBxxER6VRJu4e+5+BxHnplG/OnDGXBtGFBxxER6XRJWejuzr3/XUhaSgrfXTQ16DgiIl0iKQv9+Q2lvL79AF+9diLDs3oFHUdEpEskXaEfPlbH91ZuYtbI/nzqAxcEHUdEpMskXaH/4+rNVJ+o576PTtcl5USkW0mqQv9T8QGeWV/CkivGMnl4ZtBxRES6VNIU+sn6Rr7xXCEXDOrN3XN1STkR6X6SZh76I78r5t2Dx3nq9ovpmZ4adBwRkS6XFHvoW8trWPbaDm5+Xy6XjR8cdBwRkUBEVehmtsDMtppZsZktbeX2RWZWYGYbzGydmX0w9lFbFw47S58tILNXOvfeMLmrHlZEJO50OORiZqnAo8A8oARYa2Yr3H1Ts81eBVa4u5vZDOA3wKTOCNzSU2/t5p09R3joEzMZ2CejKx5SRCQuRbOHPgcodved7l4HLAcWNd/A3Y+6u0cW+wBOFyivOskDL23l8rzB3DQrpyseUkQkbkVT6DnA3mbLJZF1pzGzj5jZFmAV8LnYxGvft1cUUd8Y5gc3TdMl5USk24um0FtryjP2wN39OXefBNwEfL/VOzJbEhljX1dZWXlWQVt6qaic/FAFX75mAhcM0iXlRESiKfQSYGSz5VygtK2N3f0PwDgzO2O6ibs/5u6z3X12dnb2WYd9T83Jer69oohJw/px++Vjzvl+RESSSTSFvhbIM7MxZpYBLAZWNN/AzMZbZMzDzN4HZAAHYx32PQ/mb2V/TS333zyD9NSkmHkpInLeOpzl4u4NZnYXkA+kAo+7e8jM7ozcvgy4GfiMmdUDJ4BPNPuQNKbW7z7Mk3/ezW2XjGbWyP6d8RAiIgkpqiNF3X01sLrFumXNvn8AeCC20VqXlmJcnpfNV67VJeVERJpLuEP/Z47sz68+NyfoGCIicUcD0CIiSUKFLiKSJFToIiJJQoUuIpIkVOgiIklChS4ikiRU6CIiSUKFLiKSJKyTjtDv+IHNKoHd5/jjg4EDMYzT2RIpbyJlhcTKm0hZIbHyJlJWOL+8F7h7q2c3DKzQz4eZrXP32UHniFYi5U2krJBYeRMpKyRW3kTKCp2XV0MuIiJJQoUuIpIkErXQHws6wFlKpLyJlBUSK28iZYXEyptIWaGT8ibkGLqIiJwpUffQRUSkBRW6iEiSSLhCN7MFZrbVzIrNbGnQedpiZiPN7H/MbLOZhczsnqAzRcPMUs3sHTNbGXSW9phZfzN7xsy2RP6PLwk6U3vM7G8iz4MiM3vazHoGnak5M3vczPabWVGzdQPN7GUz2x75d0CQGd/TRtYHI8+FAjN7zsz6BxjxNK3lbXbbV8zMzWxwLB4roQrdzFKBR4HrgCnALWY2JdhUbWoA/s7dJwMfAL4Yx1mbuwfYHHSIKDwMvOTuk4CZxHFmM8sB7gZmu/s0mq7NuzjYVGd4AljQYt1S4FV3zwNejSzHgyc4M+vLwDR3nwFsA77e1aHa8QRn5sXMRgLzgD2xeqCEKnRgDlDs7jvdvQ5YDiwKOFOr3L3M3d+OfF9DU+HkBJuqfWaWC9wA/DLoLO0xs0zgCuBfAdy9zt2PBBqqY2lALzNLA3oDpQHnOY27/wE41GL1IuDfI9//O3BTV2ZqS2tZ3X2NuzdEFv8M5HZ5sDa08X8L8BDw90DMZqYkWqHnAHubLZcQ5yUJYGajgQuBtwKO0pGf0vQECwecoyNjgUrg3yLDQ780sz5Bh2qLu+8DfkzTnlgZUOXua4JNFZWh7l4GTTsowJCA80Trc8CLQYdoj5ndCOxz942xvN9EK3RrZV1cz7s0s77Ab4Evu3t10HnaYmYLgf3uvj7oLFFIA94H/LO7XwgcI36GA84QGXteBIwBRgB9zOxTwaZKTmZ2L03DnU8FnaUtZtYbuBf4VqzvO9EKvQQY2Ww5lzh769qcmaXTVOZPufuzQefpwGXAjWb2Lk1DWVeb2a+DjdSmEqDE3d97x/MMTQUfr64Bdrl7pbvXA88ClwacKRoVZjYcIPLv/oDztMvMbgMWAp/0+D7AZhxNf9w3Rl5vucDbZjbsfO840Qp9LZBnZmPMLIOmD5ZWBJypVWZmNI3xbnb3nwSdpyPu/nV3z3X30TT9v/7O3eNyL9Ldy4G9ZjYxsmousCnASB3ZA3zAzHpHnhdzieMPcZtZAdwW+f424PkAs7TLzBYAXwNudPfjQedpj7sXuvsQdx8deb2VAO+LPK/PS0IVeuRDj7uAfJpeEL9x91Cwqdp0GfBpmvZ0N0S+rg86VBL5EvCUmRUAs4AfBhunbZF3Es8AbwOFNL3u4upQdTN7GngTmGhmJWb2eeB+YJ6ZbadpNsb9QWZ8TxtZHwH6AS9HXmvLAg3ZTBt5O+ex4vudiYiIRCuh9tBFRKRtKnQRkSShQhcRSRIqdBGRJKFCFxFJEip0EZEkoUIXEUkS/wfJZUdiftjMZQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.plot(h.history['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iJruFXmb_dur"
      },
      "source": [
        "### Sequential API\n",
        "\n",
        "Alternatively, we can start thinking of a model as of a **sequence of layers**, and just specify those layers by adding them to the `model` object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iWc_kSr8_YXt",
        "outputId": "345dbe65-629d-468f-ed75-1d412c966340"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense_1 (Dense)             (None, 5)                 15        \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 1)                 6         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 21\n",
            "Trainable params: 21\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/15\n",
            "9/9 [==============================] - 1s 64ms/step - loss: 0.6994 - accuracy: 0.5000 - val_loss: 0.6719 - val_accuracy: 0.4667\n",
            "Epoch 2/15\n",
            "9/9 [==============================] - 0s 6ms/step - loss: 0.6635 - accuracy: 0.5429 - val_loss: 0.6531 - val_accuracy: 0.4667\n",
            "Epoch 3/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.6469 - accuracy: 0.5857 - val_loss: 0.5775 - val_accuracy: 1.0000\n",
            "Epoch 4/15\n",
            "9/9 [==============================] - 0s 4ms/step - loss: 0.5639 - accuracy: 0.9143 - val_loss: 0.5395 - val_accuracy: 0.7333\n",
            "Epoch 5/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.5236 - accuracy: 0.7143 - val_loss: 0.4498 - val_accuracy: 0.9333\n",
            "Epoch 6/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.4573 - accuracy: 0.8714 - val_loss: 0.3584 - val_accuracy: 1.0000\n",
            "Epoch 7/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.3867 - accuracy: 0.8714 - val_loss: 0.2989 - val_accuracy: 0.9333\n",
            "Epoch 8/15\n",
            "9/9 [==============================] - 0s 7ms/step - loss: 0.3388 - accuracy: 0.8857 - val_loss: 0.2204 - val_accuracy: 1.0000\n",
            "Epoch 9/15\n",
            "9/9 [==============================] - 0s 6ms/step - loss: 0.2815 - accuracy: 0.9429 - val_loss: 0.1957 - val_accuracy: 1.0000\n",
            "Epoch 10/15\n",
            "9/9 [==============================] - 0s 6ms/step - loss: 0.2692 - accuracy: 0.8857 - val_loss: 0.1323 - val_accuracy: 1.0000\n",
            "Epoch 11/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.2591 - accuracy: 0.9429 - val_loss: 0.1105 - val_accuracy: 1.0000\n",
            "Epoch 12/15\n",
            "9/9 [==============================] - 0s 6ms/step - loss: 0.2229 - accuracy: 0.9286 - val_loss: 0.1051 - val_accuracy: 1.0000\n",
            "Epoch 13/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.2146 - accuracy: 0.9143 - val_loss: 0.0919 - val_accuracy: 1.0000\n",
            "Epoch 14/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.2031 - accuracy: 0.9429 - val_loss: 0.0859 - val_accuracy: 1.0000\n",
            "Epoch 15/15\n",
            "9/9 [==============================] - 0s 5ms/step - loss: 0.1997 - accuracy: 0.9429 - val_loss: 0.0829 - val_accuracy: 1.0000\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x12894cfba30>"
            ]
          },
          "execution_count": 66,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Dense(5,activation='sigmoid',input_shape=(2,)))\n",
        "model.add(tf.keras.layers.Dense(1,activation='sigmoid'))\n",
        "\n",
        "model.compile(tf.keras.optimizers.Adam(0.1),'binary_crossentropy',['accuracy'])\n",
        "model.summary()\n",
        "model.fit(train_x_norm,train_labels,validation_data=(test_x_norm,test_labels),batch_size=8,epochs=15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BmHNhUU8bqEX"
      },
      "source": [
        "## Classification Loss Functions\n",
        "\n",
        "It is important to correctly specify loss function and activation function on the last layer of the network. The main rules are the following:\n",
        "* If the network has one output (**binary classification**), we use **sigmoid** activation function, for **multiclass classification** - **softmax**\n",
        "* If the output class is represented as one-hot-encoding, the loss function will be **cross entropy loss** (categorical cross-entropy), if the output contains class number - **sparse categorical cross-entropy**.  For **binary classification** - use **binary cross-entropy** (same as **log loss**)\n",
        "* **Multi-label classification** is when we can have an object belonging to several classes at the same time. In this case, we need to encode labels using one-hot encoding, and use **sigmoid** as activation function, so that each class probability is between 0 and 1.\n",
        "\n",
        "| Classification | Label Format | Activation Function | Loss |\n",
        "|---------------|-----------------------|-----------------|----------|\n",
        "| Binary      | Probability of 1st class | sigmoid | binary crossentropy |\n",
        "| Binary      | One-hot encoding (2 outputs) | softmax | categorical crossentropy |\n",
        "| Multiclass |  One-hot encoding | softmax | categorical crossentropy |\n",
        "| Multiclass | Class Number | softmax | sparse categorical crossentropy |\n",
        "| Multilabel | One-hot encoding | sigmoid | categorical crossentropy |\n",
        "\n",
        "> Binary classification can also be handled as a special case of multi-class classification with two outputs. In this case, we need to use **softmax**.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gZ-kWx84bMDH"
      },
      "source": [
        "**Task 3**: \n",
        "Use Keras to train MNIST classifier:\n",
        "* Notice that Keras contains some standard datasets, including MNIST. To use MNIST from Keras, you only need a couple of lines of code (more information [here](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist))\n",
        "* Try several network configuration, with different number of layers/neurons, activation functions.\n",
        "\n",
        "What is the best accuracy you were able to achieve?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yX6hqiafwHl9"
      },
      "source": [
        "## Takeaways\n",
        "\n",
        "* Tensorflow allows you to operate on tensors at low level, you have most flexibility.\n",
        "* There are convenient tools to work with data (`td.Data`) and layers (`tf.layers`)\n",
        "* For beginners/typical tasks, it is recommended to use **Keras**, which allows to construct networks from layers\n",
        "* If non-standard architecture is needed, you can implement your own Keras layer, and then use it in Keras models\n",
        "* It is a good idea to look at PyTorch as well and compare approaches. \n",
        "\n",
        "A good sample notebook from the creator of Keras on Keras and Tensorflow 2.0 can be found [here](https://t.co/k694J95PI8)."
      ]
    }
  ],
  "metadata": {
    "celltoolbar": "Slideshow",
    "colab": {
      "collapsed_sections": [],
      "name": "IntroKerasTF.ipynb",
      "provenance": []
    },
    "interpreter": {
      "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
    },
    "kernelspec": {
      "display_name": "Python 3.8.12 64-bit (conda)",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "livereveal": {
      "start_slideshow_at": "selected"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}