Multivariate_Linear_Regression.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Multivariate Linear_Regression.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p9YUP3dGMeQE"
      },
      "source": [
        "# **Multivariate Linear Regression**\n",
        "## Note : The Code takes about 12 seconds to execute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4wb_jiU2OFEm"
      },
      "source": [
        "The goal of this notebook is to demonstrate a linear regression model from the ground up using numpy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wEKtDT3sH-tm"
      },
      "source": [
        "%matplotlib inline\n",
        "from IPython.display import HTML\n",
        "from numpy import *\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VI4RgCenNoMR"
      },
      "source": [
        "Downloaded data set to colab from UCI machine learning repository \"**QSAR fish toxicity Data Set**\"\n",
        "\n",
        "Number of attributes : 07\n",
        "Number of instances : 908"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YepdM38uNZJe",
        "outputId": "84ba1d67-303c-476d-a448-ebb2345f1751"
      },
      "source": [
        "!ls"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "sample_data\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DRmnQm96N8KN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "93689fda-db96-4c3f-eb74-91ea4ed306ef"
      },
      "source": [
        "import urllib.request\n",
        "urllib.request.urlretrieve(\"https://archive.ics.uci.edu/ml/machine-learning-databases/00504/qsar_fish_toxicity.csv\",\"qsar_fish_toxicity.csv\")\n",
        "#x1 = array(qsar_fish_toxicity[:,0])\n",
        "#x2 = array(qsar_fish_toxicity[:,1])\n",
        "#x3 = array(qsar_fish_toxicity[:,2])\n",
        "#x4 = array(qsar_fish_toxicity[:,3])\n",
        "#x5 = array(qsar_fish_toxicity[:,4])\n",
        "#x6 = array(qsar_fish_toxicity[:,5])\n",
        "#y = qsar_fish_toxicity[:, 6]\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('qsar_fish_toxicity.csv', <http.client.HTTPMessage at 0x7fe62ad9b510>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BIGfGuZ-Muk2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "36098763-da8b-4f7d-8e96-acf746e117d4"
      },
      "source": [
        "!ls"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "qsar_fish_toxicity.csv\tsample_data\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-spV44xRTZTA"
      },
      "source": [
        "## Import the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N3KBcQavTff8"
      },
      "source": [
        "Here, This is a dataset with **seven(7)** columns was used to develop quantitative regression QSAR models to predict acute aquatic toxicity towards the fish Pimephales promelas (fathead minnow) on a set of 908 chemicals. LC50 data, which is the concentration that causes death in 50% of test fish over a test duration of 96 hours, was used as model response. The model comprised 6 molecular descriptors: MLOGP (molecular properties), CIC0 (information indices), GATS1i (2D autocorrelations), NdssC (atom-type counts), NdsCH ((atom-type counts), SM1_Dz(Z) (2D matrix-based descriptors). respectively."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SiK1ePJITUy7"
      },
      "source": [
        "qsar_fish_toxicity = genfromtxt('qsar_fish_toxicity.csv', delimiter=';')\n",
        "#print(qsar_fish_toxicity)\n",
        "#Extract coulmns, we have 7 columns\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BgmrqUSrZBbc"
      },
      "source": [
        "## Defining the hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HVgUJVKqY_TI"
      },
      "source": [
        "#define and declaration of hyperparamters\n",
        "learning_rate = 0.0001\n",
        "initial_t0 = 0\n",
        "initial_t1 = 0\n",
        "initial_t2 = 0\n",
        "initial_t3 = 0\n",
        "initial_t4 = 0\n",
        "initial_t5 = 0\n",
        "initial_t6 = 0\n",
        "num_iterations = 1000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D6bBJPkSZ1qc"
      },
      "source": [
        "## Define the Cost Function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wOocPbC9Z08p"
      },
      "source": [
        "def compute_cost( t0, t1, t2, t3, t4, t5, t6, qsar_fish_toxicity):\n",
        "    total_cost = 0\n",
        "    \n",
        "    # number of datapoints in training data ,In this scenario N = 908\n",
        "    N = float(len(qsar_fish_toxicity))\n",
        "    \n",
        "    # Compute sum of squared errors\n",
        "    for i in range(0, len(qsar_fish_toxicity)):\n",
        "        x1 = qsar_fish_toxicity[i, 0]\n",
        "        x2 = qsar_fish_toxicity[i, 1]\n",
        "        x3 = qsar_fish_toxicity[i, 2]\n",
        "        x4 = qsar_fish_toxicity[i, 3]\n",
        "        x5 = qsar_fish_toxicity[i, 4]\n",
        "        x6 = qsar_fish_toxicity[i, 5]\n",
        "        y = qsar_fish_toxicity[i, 6]\n",
        "        \n",
        "        total_cost += (y - (t0 + t1 * x1 + t2 * x2 + t3 * x3 + t4 * x4 + t5 * x5 + t6 * x6)) ** 2\n",
        "        \n",
        "    # Return average of squared error\n",
        "    return total_cost/(2*N)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZnNr4T3Q-9TE"
      },
      "source": [
        "## Define Gradient Descent functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z7dQ8n5m-ep5"
      },
      "source": [
        "def step_gradient(t0_current, t1_current, t2_current, t3_current, t4_current, t5_current, t6_current, qsar_fish_toxicity, alpha):\n",
        "    \"\"\"takes one step down towards the minima\n",
        "    \n",
        "    Args:\n",
        "        t0_current (float): current value of t0\n",
        "        t1_current (float): current value of t1\n",
        "        t2_current (float): current value of t2\n",
        "        t3_current (float): current value of t3\n",
        "        t4_current (float): current value of t4\n",
        "        t5_current (float): current value of t5\n",
        "        t6_current (float): current value of t6\n",
        "        qsar_fish_toxicity (np.array): array containing the training data (t0,t1,t2,t3,t4,t5,t6,y)\n",
        "        alpha (float): learning rate / step size\n",
        "    \n",
        "    Returns:\n",
        "        tuple: (t0,t1,t2,t3,t4,t5,t6) new values of t0,t1,t2,t3,t4,t5,t6\n",
        "    \"\"\"\n",
        "    \n",
        "    t0_gradient = 0\n",
        "    t1_gradient = 0\n",
        "    t2_gradient = 0\n",
        "    t3_gradient = 0\n",
        "    t4_gradient = 0\n",
        "    t5_gradient = 0\n",
        "    t6_gradient = 0\n",
        "\n",
        "    N = float(len(qsar_fish_toxicity))\n",
        "\n",
        "    # Calculate Gradient Function\n",
        "    for i in range(0, len(qsar_fish_toxicity)):\n",
        "        x1 = qsar_fish_toxicity[i, 0]\n",
        "        x2 = qsar_fish_toxicity[i, 1]\n",
        "        x3 = qsar_fish_toxicity[i, 2]\n",
        "        x4 = qsar_fish_toxicity[i, 3]\n",
        "        x5 = qsar_fish_toxicity[i, 4]\n",
        "        x6 = qsar_fish_toxicity[i, 5]\n",
        "        y = qsar_fish_toxicity[i, 6]\n",
        "        \n",
        "        t0_gradient += - (2/N) * (y - (t0_current + t1_current * x1 + t2_current * x2 + t3_current * x3 + t4_current * x4 + t5_current * x5 + t6_current * x6))\n",
        "        t1_gradient += - (2/N) * x1 * (y - (t0_current + t1_current * x1 + t2_current * x2 + t3_current * x3 + t4_current * x4 + t5_current * x5 + t6_current * x6))\n",
        "        t2_gradient += - (2/N) * x2 * (y - (t0_current + t1_current * x1 + t2_current * x2 + t3_current * x3 + t4_current * x4 + t5_current * x5 + t6_current * x6))\n",
        "        t3_gradient += - (2/N) * x3 * (y - (t0_current + t1_current * x1 + t2_current * x2 + t3_current * x3 + t4_current * x4 + t5_current * x5 + t6_current * x6))\n",
        "        t4_gradient += - (2/N) * x4 * (y - (t0_current + t1_current * x1 + t2_current * x2 + t3_current * x3 + t4_current * x4 + t5_current * x5 + t6_current * x6))\n",
        "        t5_gradient += - (2/N) * x5 * (y - (t0_current + t1_current * x1 + t2_current * x2 + t3_current * x3 + t4_current * x4 + t5_current * x5 + t6_current * x6))\n",
        "        t6_gradient += - (2/N) * x6 * (y - (t0_current + t1_current * x1 + t2_current * x2 + t3_current * x3 + t4_current * x4 + t5_current * x5 + t6_current * x6))\n",
        "    \n",
        "    #(y - (t0_current + t1_current * x1 + t2_current * x2 + t3_current * x3 + t4_current * x4 + t5_current * x5 + t6_current * x6))\n",
        "    # Update current t0,t1,t2,t3,t4,t5 and t6\n",
        "    t0_updated = t0_current - alpha * t0_gradient\n",
        "    t1_updated = t1_current - alpha * t1_gradient\n",
        "    t2_updated = t2_current - alpha * t2_gradient\n",
        "    t3_updated = t3_current - alpha * t3_gradient\n",
        "    t4_updated = t4_current - alpha * t4_gradient\n",
        "    t5_updated = t5_current - alpha * t5_gradient\n",
        "    t6_updated = t6_current - alpha * t6_gradient\n",
        "\n",
        "\n",
        "\n",
        "    #Return updated parameters\n",
        "    return t0_updated, t1_updated, t2_updated, t3_updated, t4_updated, t5_updated, t6_updated\n",
        "\n",
        "\n",
        "def gradient_descent(qsar_fish_toxicity, starting_t0, starting_t1, starting_t2, starting_t3, starting_t4, starting_t5, starting_t6, learning_rate, num_iterations):\n",
        "\n",
        "    \"\"\"runs gradient descent\n",
        "    \n",
        "    Args:\n",
        "        qsar_fish_toxicity (np.array): training data, containing t0,t1,t2,t3,t4,t5,t6,y\n",
        "        starting_t0 (float): initial value of t0 (random)\n",
        "        starting_t1 (float): initial value of t1 (random)\n",
        "        starting_t2 (float): initial value of t2 (random)\n",
        "        starting_t3 (float): initial value of t3 (random)\n",
        "        starting_t4 (float): initial value of t4 (random)\n",
        "        starting_t5 (float): initial value of t5 (random)\n",
        "        starting_t6 (float): initial value of t6 (random)\n",
        "        learning_rate (float): hyperparameter to adjust the step size during descent\n",
        "        num_iterations (int): hyperparameter, decides the number of iterations for which gradient descent would run\n",
        "    \n",
        "    Returns:\n",
        "        list : the first and second item are t0,t1,t2,t3,t4,t5,t6 respectively at which the best fit curve is obtained, the third and fourth items are two lists, which store the value of t0,t1,t2,t3,t4,t5 and t6  as gradient descent proceeded.\n",
        "    \"\"\"\n",
        "\n",
        "    # initial values\n",
        "    \n",
        "    t0 = starting_t0\n",
        "    t1 = starting_t1\n",
        "    t2 = starting_t2\n",
        "    t3 = starting_t3\n",
        "    t4 = starting_t4\n",
        "    t5 = starting_t5\n",
        "    t6 = starting_t6\n",
        "    \n",
        "    # to store the cost after each iteration\n",
        "    cost_graph = []\n",
        "    \n",
        "    \n",
        "    t0_progress = []\n",
        "    t1_progress = []\n",
        "    t2_progress = []\n",
        "    t3_progress = []\n",
        "    t4_progress = []\n",
        "    t5_progress = []\n",
        "    t6_progress = []\n",
        "    \n",
        "    # For every iteration, optimize t0,t1,t2,t3,t4,t5,t6  and compute its cost\n",
        "    for i in range(num_iterations):\n",
        "        cost_graph.append(compute_cost( t0, t1, t2, t3, t4, t5, t6, qsar_fish_toxicity))\n",
        "        \n",
        "        t0, t1, t2, t3, t4, t5, t6 = step_gradient(t0, t1, t2, t3, t4, t5, t6, array(qsar_fish_toxicity), learning_rate)\n",
        "       \n",
        "        t0_progress.append(t0)\n",
        "        t1_progress.append(t1)\n",
        "        t2_progress.append(t2)\n",
        "        t3_progress.append(t3)\n",
        "        t4_progress.append(t4)\n",
        "        t5_progress.append(t5)\n",
        "        t6_progress.append(t6)\n",
        "        \n",
        "        \n",
        "    \n",
        "    return [t0, t1, t2, t3, t4, t5, t6, cost_graph, t0_progress, t1_progress, t2_progress, t3_progress, t4_progress, t5_progress, t6_progress]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8xNWc1uQOHFb"
      },
      "source": [
        "## Run gradient_descent() to get optimized parameters t0, t1, t2, t3, t4, t5 and t6"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FbmElJbuOEIm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "36f1b4ab-e7ca-4d6d-cfef-33d996204e9d"
      },
      "source": [
        "\n",
        "t0, t1, t2, t3, t4, t5, t6, cost_graph, t0_progress, t1_progress, t2_progress, t3_progress, t4_progress, t5_progress, t6_progress = gradient_descent(qsar_fish_toxicity, initial_t0, initial_t1, initial_t2, initial_t3, initial_t4, initial_t5, initial_t6, learning_rate, num_iterations)\n",
        "#Print optimized parameters\n",
        "\n",
        "print ('Optimized t0:', t0)\n",
        "print ('Optimized t1:', t1)\n",
        "print ('Optimized t2:', t2)\n",
        "print ('Optimized t3:', t3)\n",
        "print ('Optimized t4:', t4)\n",
        "print ('Optimized t5:', t5)\n",
        "print ('Optimized t6:', t6)\n",
        "\n",
        "\n",
        "\n",
        "#Print error with optimized parameters\n",
        "\n",
        "print ('Minimized cost:', compute_cost(t0, t1, t2, t3, t4, t5, t6, qsar_fish_toxicity))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Optimized t0: 0.2377149844428744\n",
            "Optimized t1: 0.6520488852918811\n",
            "Optimized t2: 0.1933557478938609\n",
            "Optimized t3: 0.27235048317118216\n",
            "Optimized t4: 0.07145504153924973\n",
            "Optimized t5: 0.12261957647400726\n",
            "Optimized t6: 0.5598174130857656\n",
            "Minimized cost: 0.7211288099651326\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NaAbq9-FRmXF"
      },
      "source": [
        "## Plotting the cost per iterations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z-oJEVy9RkQn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "01ef72ad-b1b9-41da-a7c9-899c77c09b5b"
      },
      "source": [
        "\n",
        "plt.plot(cost_graph)\n",
        "plt.xlabel('No. of iterations')\n",
        "plt.ylabel('Cost')\n",
        "plt.title('Cost per iteration')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEWCAYAAABsY4yMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxcdb3/8ddnZrI0aZqkTbovoZRCN9rSCi0FReUiArLcHyIKFHDl53Ld7k/F5Sr+XO5VUXBB4AqKiLigIBdBtiK7LSmttLSlC93XdE26ZP/cP85JGUraJjTTM3Pm/Xw8zmNmzvo5OXm858z3nPmOuTsiIhI/iagLEBGRzFDAi4jElAJeRCSmFPAiIjGlgBcRiSkFvIhITCngRbrBzC4zs0ciruFmM/talDVIblDAS8aY2QfMrNbMdpvZRjN7yMxOO8J1rjKzM3uqxu5y97vc/ay0etzMRmVqe2Z2lZk9c0AN17j7/8/UNiU+FPCSEWb2OeAG4DvAAGA4cBNwQZR1dYeZpXJ5/SK4uwYNPToA5cBu4L2HmKeI4A1gQzjcABSF06qAB4CdwHbgaYKTkTuBdmBfuP4vdLLeM4B1wJeBrcAq4LIDtvsDYA2wGbgZ6HXAsl8ENgF3drL+q4BnwudPAQ7sCet5Xzj+PGB+WP9zwIlpy68K1/8S0ASkgC8BK4AGYBFwUTjvGKARaAvXvzMc/yvgW2nr/AiwPPxb3Q8MTpvmwDXAsrCenwEW9f+IhqMz6AxeMmE6UAzce4h5vgJMAyYBE4GTga+G0z5PELTVBGf/Xwbc3a8gCOb3uHtvd//eQdY9kOBNYghwJXCrmR0fTvtPYHS43VHhPP9xwLJ9gRHARw+1k+7+1vDpxLCe35vZZOB24GNAP+AW4H4zK0pb9P3AuUCFu7cShPvpBG+M1wG/MbNB7r6YIJyfD9dfcWANZvYO4LvAJcAgYDXwuwNmOw94C3BiON+7DrVfEh8KeMmEfsDWMLwO5jLgm+6+xd3rCILtinBaC0FYjXD3Fnd/2t2722nS19y9yd2fBP4KXGJmRhDan3X37e7eQNCEdGnacu3A18Nl93Vzm4Trv8XdZ7t7m7vfQXCmPi1tnh+7+9qO9bv7H919g7u3u/vvCc62T+7i9i4Dbnf3F929CbgWmG5mNWnz/Ke773T3NcATBG9ukgcU8JIJ24Cqw7QxDyY42+ywOhwH8H2CJodHzOxVM/tSN7e/w933dLLuaqAEmGtmO81sJ/C3cHyHOndv7Ob20o0APt+x/nAbw3ht3wDWpi9gZjPNbH7a/OMJPoF0xev+ju6+m+DvPyRtnk1pz/cCvbu8N5LTFPCSCc8TnLVeeIh5NhCEYYfh4TjcvcHdP+/uI4Hzgc+Z2TvD+bpyJl9pZqWdrHsrQfv9OHevCIdyd08PvCPtXnUt8O209Ve4e4m7393ZNsxsBPDfwCeBfmEzzELAuljP6/6O4X73A9Yf4X5IDCjgpce5+y6Cdu2fmdmFZlZiZgVm9m4z62g3vxv4qplVm1lVOP9vAMzsPDMbFTap7CK4yNgeLrcZGNmFMq4zs0IzO52gDfqP7t5OEKY/MrP+4baGmNmRtEkfWM9/A9eY2SkWKDWzc82s7CDLlxKEeF1Yz9UEZ/Dp6x9qZoUHWf5u4GozmxS2838HmO3uq978LklcKOAlI9z9euBzBBdO6wjObD8J3BfO8i2gluBukgXAi+E4gOOAxwjuHHkeuMndnwinfZfgjWGnmf37QTa/CdhBcHZ7F3CNuy8Jp32RoPnnH2ZWH27n+E7X0jXfAO4I67nE3WsJ7mr5aVjDcoI7bzrl7ouA6wn2czMwAXg2bZZZwMvAJjPb2snyjwFfA/4EbASO5fXXFCSPWfevXYlkLzM7A/iNuw+NuhaRqOkMXkQkphTwIiIxpSYaEZGY0hm8iEhMZVVnR1VVVV5TUxN1GSIiOWPu3Llb3b26s2lZFfA1NTXU1tZGXYaISM4ws9UHm6YmGhGRmFLAi4jElAJeRCSmFPAiIjGlgBcRiSkFvIhITCngRURiKucDvqm1jZufXMHTy+qiLkVEJKvkfMAXJhPc8uQK/jJ/Q9SliIhklZwPeDNjyohKXly9I+pSRESySs4HPMCUEX15desetu9pjroUEZGsEZOArwRgrs7iRUT2i0XAnzi0nIKkKeBFRNLEIuCLC5KMG1yudngRkTSxCHgImmn+uW4nza3tUZciIpIVYhXwTa3tvLxhV9SliIhkhVgFPOhCq4hIh9gE/IA+xQyt7MWLaxTwIiIQo4CH4Cy+dtUO3D3qUkREIhergJ86opItDU2s27Ev6lJERCIXq4A/KWyHVzONiEjMAv74AWWUFiZ5YdX2qEsREYlcrAI+lUxw0ohKXlipM3gRkVgFPMC0kf14ZXODOh4TkbwXu4A/5Zi+AGqmEZG8F7uAnzC0nKJUgtmvKuBFJL/FLuCLUklOGl7J7JXboi5FRCRSsQt4gJOP6cuijfXUN7ZEXYqISGRiGfCnjOyLO9SqHV5E8lgsA37ysEoKksbslQp4EclfsQz4XoVJJg6t0IVWEclrsQx4CNrhF67fxZ6m1qhLERGJRGwD/pSR/Whtd/VLIyJ5K7YBP2VEJcmEqZlGRPJWbAO+d1GK8UPKef5V3Q8vIvkpowFvZp81s5fNbKGZ3W1mxZnc3oFmHNuPf67dyW61w4tIHspYwJvZEODfgKnuPh5IApdmanudmTGqitZ2Z46+1SoieSjTTTQpoJeZpYASYEOGt/c6U0ZUUphK8OxyBbyI5J+MBby7rwd+AKwBNgK73P2RTG2vM8UFSaaOqOTZ5VuP5mZFRLJCJptoKoELgGOAwUCpmV3eyXwfNbNaM6utq6vr8TpmjKpiyaYGtu5u6vF1i4hks0w20ZwJrHT3OndvAf4MnHrgTO5+q7tPdfep1dXVPV7EjFFVADy/Qs00IpJfMhnwa4BpZlZiZga8E1icwe11asKQcsqKUzy3Qs00IpJfMtkGPxu4B3gRWBBu69ZMbe9gkglj2sh+utAqInkno3fRuPvX3f0Edx/v7le4eyQN4TOO7cea7XtZu31vFJsXEYlEbL/Jmq6jHV7NNCKST/Ii4Ef1703/siKeUTONiOSRvAh4M+O046p4Zlkdbe0edTkiIkdFXgQ8wBnH92fH3hYWrN8VdSkiIkdF3gT86aOqMIMnX+n5L1OJiGSjvAn4ytJCJg6t4MmlW6IuRUTkqMibgAd42+hq5q/dyc69zVGXIiKScfkV8MdX0+7w9DLdLiki8ZdXAT9xaAUVJQU8uVTt8CISf3kV8MmEcfpx1Ty5tA533S4pIvGWVwEPQTt8XUMTizbWR12KiEhG5V3Av/W4oNsCNdOISNzlXcD371PM2EF9dD+8iMRe3gU8wBnHVzN39Q7qG1uiLkVEJGPyMuDfOaY/re2us3gRibW8DPhJwyrpV1rIY4s3R12KiEjG5GXAJxPGO07ozxNLttDS1h51OSIiGZGXAQ9w5tgB1De28sKq7VGXIiKSEXkb8KcfV0VhKsFji9T5mIjEU94GfElhitNGVfHo4k36VquIxFLeBjzAmWMGsHb7PpZt2R11KSIiPS6vA/6dY/oD8Ogi3U0jIvGT1wE/oE8xE4eW63ZJEYmlvA54CJpp5q/dyZaGxqhLERHpUQr4sQNwh8cX624aEYmXvA/4EwaWMaJfCQ8t3BR1KSIiPSrvA97MePf4QTy3fKt+q1VEYiXvAx7gnAkDaW133U0jIrGigAcmDClnaGUvHlywMepSRER6jAKeoJnmnAmDeGb5VnbtUx/xIhIPCvjQu8cPpKXNeVz3xItITCjgQ5OGVTC4vJgHF+huGhGJBwV8yMw4e/wgnlpWR4N+yk9EYkABn+acCQNpbm1n1hJ96UlEcp8CPs1JwysZ0KeIv76ku2lEJPcp4NMkEsa5Ewbz91fq2LVXzTQiktsU8Ae4cPJgmtvaeWihzuJFJLdlNODNrMLM7jGzJWa22MymZ3J7PWHCkHJGVpVy3/z1UZciInJEMn0GfyPwN3c/AZgILM7w9o6YmXHBpCHMXrmdjbv2RV2OiMiblrGAN7Ny4K3AbQDu3uzuOzO1vZ50waTBuMP98zdEXYqIyJuWyTP4Y4A64JdmNs/MfmFmpRncXo+pqSpl0rAK7lPAi0gOy2TAp4CTgJ+7+2RgD/ClA2cys4+aWa2Z1dbV1WWwnO65cNJgFm+sZ+nmhqhLERF5UzIZ8OuAde4+O3x9D0Hgv4673+ruU919anV1dQbL6Z7zJg4mmTDum6eLrSKSmzIW8O6+CVhrZseHo94JLMrU9npaVe8iThtVxV/mb6C93aMuR0Sk2zJ9F82ngLvM7CVgEvCdDG+vR100eQjrd+5j9srtUZciItJtGQ14d58fNr+c6O4XuvuOTG6vp509fiBlxSn+WLs26lJERLpN32Q9hOKCJOdPHMyDCzdSrx4mRSTHKOAP45Kpw2hsaed//qlbJkUktyjgD+PEoeWcMLCMP9Sui7oUEZFuUcAfhpnx3qnD+OfanbyySffEi0juUMB3wYWTBlOQNP6gi60ikkMU8F3Qr3cRZ44ZwL3z1tPc2h51OSIiXaKA76JLpg5j+55mHl+8OepSRES6RAHfRW8dXc3g8mLumr0m6lJERLpEAd9FyYTxgVOG88zyrayo2x11OSIih9WlgDezO7syLu4uecswCpLGXf/QWbyIZL+unsGPS39hZklgSs+Xk936lxVz9vhB/HHuWvY2t0ZdjojIIR0y4M3sWjNrAE40s/pwaAC2AH85KhVmmSumjaChsVW/9iQiWe+QAe/u33X3MuD77t4nHMrcvZ+7X3uUaswqb6mp5ISBZfz6+dW4qxthEcleXW2ieaDj5/bM7HIz+6GZjchgXVnLzLh82ggWbaxn3tqc+IlZEclTXQ34nwN7zWwi8HlgBfDrjFWV5S6cPITeRSnufH511KWIiBxUVwO+1YP2iAuAn7r7z4CyzJWV3XoXpbh4ylAeeGkDm+sboy5HRKRTXQ34BjO7FrgC+KuZJYCCzJWV/a6eUUNru3PHc6uiLkVEpFNdDfj3AU3AB8PfWh0KfD9jVeWAEf1KedfYgdw1e41umRSRrNSlgA9D/S6g3MzOAxrdPW/b4Dt8+PRj2LWvhXvmqq94Eck+Xf0m6yXAHOC9wCXAbDO7OJOF5YIpIyqZNKyC255ZSVu7bpkUkezS1SaarwBvcfcr3X0mcDLwtcyVlRvMjI+cPpLV2/by6CL1Miki2aWrAZ9w9y1pr7d1Y9lYe9e4AQyp6MUvnn416lJERF6nqyH9NzN72MyuMrOrgL8CD2aurNyRSib44GnHULt6B3NXb4+6HBGR/Q7XF80oM5vh7v8PuAU4MRyeB249CvXlhEvfMozKkgJ+Omt51KWIiOx3uDP4G4B6AHf/s7t/zt0/B9wbThOgtCjFh047hideqWPh+l1RlyMiAhw+4Ae4+4IDR4bjajJSUY6aeWoNZcUpfvaEzuJFJDscLuArDjGtV08Wkuv6FBdw1ak1PLRwE0s3N0RdjojIYQO+1sw+cuBIM/swMDczJeWuq2ccQ0lhkpt0Fi8iWSB1mOmfAe41s8t4LdCnAoXARZksLBf1LS3k8mkj+MXTr/KZM0dTU1UadUkikscO94Mfm939VOA6YFU4XOfu08PuC+QAHz79GAqSCX48a1nUpYhInutqXzRPuPtPwmFWpovKZf3Lipk5fQT3zVvPMrXFi0iE9G3UDPi/Z4yiV0GSHz66NOpSRCSPKeAzoG9pIR8+fSQPLdzEgnW6L15EoqGAz5APn34MFSUF/OCRV6IuRUTylAI+Q8qKC/j4Gcfy5NI6Zr+6LepyRCQPKeAzaOb0Ggb0KeL7D79C8JO2IiJHjwI+g4oLknzmzNHUrt7BQwt1V6mIHF0ZD3gzS5rZPDN7INPbykaXTB3GCQPL+O5Di2lqbYu6HBHJI0fjDP7TwOKjsJ2slEwYXz13LGu37+NXz66KuhwRySMZDXgzGwqcC/wik9vJdqcdV8U7TujPT2ctZ9vupqjLEZE8kekz+BuALwDtGd5O1vvyOWPY29LGjx7Tl59E5OjIWMCb2XnAFnc/ZK+TZvZRM6s1s9q6urpMlRO5Uf17c/kpw/nt7DUs2VQfdTkikgcyeQY/AzjfzFYBvwPeYWa/OXAmd7/V3ae6+9Tq6uoMlhO9z5w5mvJeBXztvoW6bVJEMi5jAe/u17r7UHevAS4FZrn75ZnaXi6oLC3k2neP4YVVO/jTi+ujLkdEYk73wR9lF08ZypQRlXznwcXs3NscdTkiEmNHJeDd/e/uft7R2Fa2SySMb104nl37Wvjew+qnRkQyR2fwERgzqA9XnVrD3XPWMG/NjqjLEZGYUsBH5LP/MpoBZcVc++cFNLfm/V2kIpIBCviI9C5K8e2LxrNkUwM/1Y90i0gGKOAj9M4xA7ho8hBuemI5L2/QD4OISM9SwEfs6+8ZS0VJIV+45yVa2tRUIyI9RwEfsYqSQr514The3lDPLU+uiLocEYkRBXwWOHv8IM49cRA3Pr6MhevVVCMiPUMBnyW+feF4+pUW8W+/m8e+ZvUbLyJHTgGfJSpKCvnhJRNZuXUP3/rroqjLEZEYUMBnkVNHVfHR00dy1+w1PLpoc9TliEiOU8Bnmc+fdTzjBvfhi396ic31jVGXIyI5TAGfZQpTCW68dDKNLW188rcv6tZJEXnTFPBZaFT/3nz3XyfwwqodfO9vS6IuR0RylAI+S10waQhXTh/Bfz+9kgcXbIy6HBHJQQr4LPaVc8cyaVgFX7jnJV6t2x11OSKSYxTwWawwleCmy06iMJXgmt/MpaGxJeqSRCSHKOCz3OCKXvzk/ZNZUbeHT/9uPm3t+i1XEekaBXwOmDGqiuvOH8esJVv4zoOLoy5HRHJEKuoCpGsunzaC5Vt2c9szKxnVvzfvP3l41CWJSJbTGXwO+eq5Y3jb6Gq+dt9Cnl2+NepyRCTLKeBzSCqZ4CcfmMzI6lI+dudc9TwpIoekgM8xfYoLuOODJ9OnOMVVv5zDqq17oi5JRLKUAj4HDSrvxa8/dApt7c7M2+ewpUF91ojIGyngc9So/r355dUns3V3E1fe/gK79uoeeRF5PQV8Dps0rIKbL5/C8i0NzLx9Nrv2KeRF5DUK+Bz31tHV/PyyKSzaWM/M2+dQr2+7ikhIAR8DZ44dwE2XTWHRhl3MvG2OujQQEUABHxv/MnYAP/3ASSxcv4srbpvDzr3NUZckIhFTwMfIu8YN5KbLTmLRhnred8s/9ItQInlOAR8zZ40byC+vfgvrduzl4pufY/U23Scvkq8U8DE0Y1QVv/3INHY3tnLxzc+zaEN91CWJSAQU8DE1cVgFf7xmOqmE8d6bn2PWks1RlyQiR5kCPsZG9S/j3o/P4JjqUj58Ry23P7MSd/UnL5IvFPAxN7C8mD98bDpnjhnANx9YxH/85WVa29qjLktEjgIFfB4oKUxx8+VT+NjbRnLnP1ZzxW1zqGtoirosEckwBXyeSCSMa989huvfO5EX1+zgvJ88Te2q7VGXJSIZpIDPM/9nylDu/fgMiguSXHrrP9QuLxJjGQt4MxtmZk+Y2SIze9nMPp2pbUn3jB3ch/s/eRpnHN+fbz6wiI/dOZfte/TNV5G4yeQZfCvweXcfC0wDPmFmYzO4PemG8l4F3HrFFL5yzhieeGULZ9/wFE8trYu6LBHpQRkLeHff6O4vhs8bgMXAkExtT7ovkTA+8taR3PeJGfTpVcDM2+fwzf9ZRGNLW9SliUgPOCpt8GZWA0wGZh+N7Un3jBtczgOfOo0rp4/g9mdXcs6NTzP71W1RlyUiRyjjAW9mvYE/AZ9x9zd8Z97MPmpmtWZWW1enJoKoFBckue6C8fzmQ6fQ0t7O+279B1+5d4H6lxfJYZbJOyjMrAB4AHjY3X94uPmnTp3qtbW1GatHumZvcys/fGQptz+7kv5lxXzj/LG8a9xAzCzq0kTkAGY2192ndjYtk3fRGHAbsLgr4S7Zo6QwxVfPG8u9H59BRUkB1/zmRa64bQ7LNjdEXZqIdEMmm2hmAFcA7zCz+eFwTga3Jz1s4rAKHvjUaVx3/jheWreTs298mm/c/7J+4FskR2S0iaa71ESTvbbvaeb6R17ht3PWUN6rgI+fcSwzp9dQXJCMujSRvBZJE43ES9/SQr590QQe+NRpTBhSznceXMIZ3/87v529hhZ1XiaSlRTw0i3jBpdz54dO4e6PTGNQRTFfvncBZ/3oKf784joFvUiWURONvGnuzmOLt3D9I6+wZFMDQyp68bG3jeSSqcPUdCNylByqiUYBL0esvd2ZtWQLN/19OS+u2UlV70KunnEMl58ygvKSgqjLE4k1BbwcFe7O7JXbuenvK3hqaR3FBQkumjyEmdNrGDOoT9TlicTSoQI+dbSLkfgyM6aN7Me0kf1YtKGeXz+/invnrefuOWs5uaYvM08dwVljB1KY0qUfkaNBZ/CSUTv3NvOH2rX8+vnVrNuxj8qSAi6YNIT3Th3KuMHlUZcnkvPURCORa2t3nlpWxz2163h00Waa29oZM6gPF08ZynsmDqJ/WXHUJYrkJAW8ZJWde5u5/58buGfuOl5atwszOLmmL+edOIh3jR+osBfpBgW8ZK1lmxt44KWN/HXBRpZv2U3C4ORj+nLOhEG8/fj+DOtbEnWJIllNAS85YWlH2L+0gRV1ewAYPaA3bz+hP28/vj9TRlRSkNQFWpF0CnjJOa/W7WbWki3MWrKFOSu309rulBWnOG1UFace24/px1ZxbHWpujCWvKfbJCXnjKzuzcjq3nz49JE0NLbwzLKtzFqyhWeXb+WhhZsAqC4rCsJ+ZD9OGdmPmn4lCnyRNDqDl5zi7qzetpfnX93Gcyu28fyKbWzd3QQEHaJNHlbB5OEVnDS8khOHVdC7SOcwEm86g5fYMDNqqkqpqSrl/ScPx91ZvmU3L6zawbw1O3hxzQ4eX7IFgITB6AFlnDi0nLGD+jB2cDljBpVRVqzuEyQ/6AxeYmfX3hbmrd3BvDU7mbd2Jy+v38W2Pc37p4/oVxIE/qA+nDCoD6P692ZYZS9SuoArOUhn8JJXyksKOOP4/pxxfH8gaNbZ0tDEog31vLxhF4s21rNoQ/3+tnyAwmSCmqoSjq3uzaj+vfc/1lSVqplHcpb+cyX2zIwBfYoZ0KeYt5/Qf//4hsYWlm/ZzfItu1lRt4flW3azZFMDD7+8ifa0D7b9SgsZ2reE4X1LGN63F8Mqg+fD+pYwqLxYZ/6StRTwkrfKiguYPLySycMrXze+qbWN1dv2snzLblZv28ua7XtZu30vL63byUMLNtKalv7JhNG/rIiB5cUM7FPc6eOAPsXqH18ioYAXOUBRKsnoAWWMHlD2hmmtbe1s3NXI2h1B6K/dvo+NuxrZXN/I0s0NPLW0jj3NbW9Yrk9xiqreRfTrXUi/0vCxdxFV4eu+pYXB895FlPcqIJnQ7Z5y5BTwIt2QSiYYFjbPcGzn8zQ0trC5vpGNuxrZFA51u5vYtruZbXuaWFG3mzmrmtmxt5mD3eNQVpSiT68Cyg8cSoLH9Gm9i5KUFqUoLUwFj0VJilL6xCAKeJEeV1ZcQFlxAaP6v/ETQLrWtnZ27G1h+55mtu1uYmv4uHNvC7v2tVC/L3jcta+FFXW79z9vaj38b98WJI2SwhS9w8BPf15alKKkMElxKkmvwiTFBUmKUgmKC5LhkKA4FTzvVZigKJU2vmOeVELXHnKAAl4kIqlkguqyIqrLioBDvxmka2xpo35fC/WNQeDvbmpjT1Pra0NzG7ubWtnb1PratOZWdje1UtfQFExrbqWxpZ3G1raDfoo4nIRBYSpBQTJBYTJ4LEjZ/tcd0wqSnY1LUBjO+9pgJBNGKmEkEwmSCUgmEuFr2/+Y3P868dr4pJG01+ZJJW3/sgnreB3Mk0wYZpAwC4fgQnwibZwlINnxfP949r/OlW9MK+BFckzHWXT/PkferbK709zWTmNLO00tbTS2tLOvpY3GjqG1ff/zpvANobGljX3N7bS0BUNz+NjSGqyrua2dltaO6cG4hsbW/fO3tDnNrWnLtgaPbe3+uruXsll66Ke/OSQ73hASh3jz6OQNo19pEX+4ZnqP16mAF8ljZkZRKmyz7xX9N3zb2502d9randZ2p60teN3aHrwBtLYF0/bP09YxbzC9Y2h93WM7be3sn6fdod0d99eet3vwZtfenj4ufV5eN62zZQ+77gPXFy7jBNdcMkEBLyJZI5EwEhi6q7Rn6CqJiEhMKeBFRGJKAS8iElMKeBGRmFLAi4jElAJeRCSmFPAiIjGlgBcRiams+sk+M6sDVr/JxauArT1YTi7QPucH7XP8Hcn+jnD36s4mZFXAHwkzqz3Y7xLGlfY5P2if4y9T+6smGhGRmFLAi4jEVJwC/taoC4iA9jk/aJ/jLyP7G5s2eBEReb04ncGLiEgaBbyISEzlfMCb2dlm9oqZLTezL0VdT08xs2Fm9oSZLTKzl83s0+H4vmb2qJktCx8rw/FmZj8O/w4vmdlJ0e7Bm2dmSTObZ2YPhK+PMbPZ4b793swKw/FF4evl4fSaKOt+s8yswszuMbMlZrbYzKbH/Tib2WfD/+uFZna3mRXH7Tib2e1mtsXMFqaN6/ZxNbMrw/mXmdmV3akhpwPezJLAz4B3A2OB95vZ2Gir6jGtwOfdfSwwDfhEuG9fAh539+OAx8PXEPwNjguHjwI/P/ol95hPA4vTXv8X8CN3HwXsAD4Ujv8QsCMc/6Nwvlx0I/A3dz8BmEiw77E9zmY2BPg3YKq7jweSwKXE7zj/Cjj7gHHdOq5m1hf4OnAKcDLw9Y43hS7x8PcDc3EApgMPp72+Frg26roytK9/Af4FeAUYFI4bBLwSPr8FeH/a/Pvny6UBGBr+478DeAAwgm/4pQ485sDDwPTweSqcz6Leh27ubzmw8sC643ycgSHAWqBveNweAN4Vx+MM1AAL3+xxBd4P3JI2/nXzHW7I6TN4XvtH6bAuHBcr4UfSycBsYIC7bwwnbQIGhM/j8re4AfgC0MSpzXEAAAVUSURBVB6+7gfsdPfW8HX6fu3f53D6rnD+XHIMUAf8MmyW+oWZlRLj4+zu64EfAGuAjQTHbS7xPs4duntcj+h453rAx56Z9Qb+BHzG3evTp3nwlh6b+1zN7Dxgi7vPjbqWoygFnAT83N0nA3t47WM7EMvjXAlcQPDmNhgo5Y1NGbF3NI5rrgf8emBY2uuh4bhYMLMCgnC/y93/HI7ebGaDwumDgC3h+Dj8LWYA55vZKuB3BM00NwIVZpYK50nfr/37HE4vB7YdzYJ7wDpgnbvPDl/fQxD4cT7OZwIr3b3O3VuAPxMc+zgf5w7dPa5HdLxzPeBfAI4Lr74XElyouT/imnqEmRlwG7DY3X+YNul+oONK+pUEbfMd42eGV+OnAbvSPgrmBHe/1t2HunsNwbGc5e6XAU8AF4ezHbjPHX+Li8P5c+pM1903AWvN7Phw1DuBRcT4OBM0zUwzs5Lw/7xjn2N7nNN097g+DJxlZpXhJ5+zwnFdE/VFiB64iHEOsBRYAXwl6np6cL9OI/j49hIwPxzOIWh7fBxYBjwG9A3nN4I7ilYACwjuUIh8P45g/88AHgifjwTmAMuBPwJF4fji8PXycPrIqOt+k/s6CagNj/V9QGXcjzNwHbAEWAjcCRTF7TgDdxNcY2gh+KT2oTdzXIEPhvu+HLi6OzWoqwIRkZjK9SYaERE5CAW8iEhMKeBFRGJKAS8iElMKeBGRmFLAS9YwMzez69Ne/7uZfSMD27k77LHvsweMv8bMZobPrzKzwT24zTPM7NTOtiWSKanDzyJy1DQB/2pm33X3rZnYgJkNBN7iQc+Er+PuN6e9vIrgHu0N3Vh3yl/rS+VAZwC7gec62ZZIRugMXrJJK8FvU372wAlmVmNms8Iz78fNbPihVhT2L/5LM1sQduL19nDSI8AQM5tvZqcfsMw3wk8NFwNTgbvC+XqZ2RQze9LM5prZw2lfN/+7md1gZrXAp83sPWGf5fPM7DEzGxB2FncN8NmO7XZsK1zHJDP7R7hv96b1Ef53M/svM5tjZks76jWzceG4+eEyx73pv7jEmgJess3PgMvMrPyA8T8B7nD3E4G7gB8fZj2fIOjPaQJBl6t3mFkxcD6wwt0nufvTnS3o7vcQfLP0MnefRPDG8xPgYnefAtwOfDttkUJ3n+ru1wPPANM86Djsd8AX3H0VcDNBX+edbffXwBfDfVtA0P93h5S7nwx8Jm38NcCNYW1TCb4lKfIGaqKRrOLu9Wb2a4IfhNiXNmk68K/h8zuB7x1mVacRhDLuvsTMVgOjgfpDLtW544HxwKNB1ykkCb6C3uH3ac+HAr8Pz/ALCfp6P6jwjazC3Z8MR91B8LX8Dh2dzM0l6Fsc4HngK2Y2FPizuy/r7g5JftAZvGSjGwj67SiNupCQAS+HZ9+T3H2Cu5+VNn1P2vOfAD8NPzl8jKAflSPRFD62EZ6QuftvCT6J7AMeNLN3HOE2JKYU8JJ13H078Ade+8k2CC5OXho+vwzotHklzdPhfJjZaGA4wa/kdFUDUBY+fwWoNrPp4foKzGzcQZYr57XuXNN/PzN9ffu5+y5gR9r1gCuAJw+cL52ZjQRedfcfE/RGeOLhd0fykQJestX1QFXa608BV5vZSwQh2PEj5NeY2TWdLH8TkDCzBQRNKFe5e1Mn8x3Mr4CbzWw+QZPMxcB/mdk/CXr2PPUgy30D+KOZzSX4abkO/wNc1NnFXYI3gu+H+zYJ+OZharsEWBjWNp6gDV/kDdSbpIhITOkMXkQkphTwIiIxpYAXEYkpBbyISEwp4EVEYkoBLyISUwp4EZGY+l8OF8sN8AcJxgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    }
  ]
}