change_fit_function_draft2(cos_similarity).ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nicholassung97/DecomposeFit/blob/main/change_fit_function_draft2(cos_similarity).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "14494ebe",
      "metadata": {
        "id": "14494ebe"
      },
      "source": [
        "### Code-1.0: ml-PINN for 1D Convection-Diffusion Equation\n",
        "\n",
        "Tasks:\n",
        "    \n",
        "    1. PDE-NN for 1D convection-diffusion equation:\n",
        "        1.0. Data\n",
        "        1.1. Geometry & BC\n",
        "        1.2. Sampling plan\n",
        "        1.3. PDE-NN \n",
        "    2. Optimize PDE-NN: [SGD]\n",
        "    3. Visualize training dynamic\n",
        "\n",
        "Purpose:\n",
        "\n",
        "    1. Study PDE-NN flexibility "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e84a150c",
      "metadata": {
        "id": "e84a150c"
      },
      "source": [
        "#1. Import Libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "505f0da5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "505f0da5",
        "outputId": "1ca319eb-5380-4311-a910-620353b60e07"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (3.2.2)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.4.4)\n",
            "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.21.6)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.8.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (0.11.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (3.0.9)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib) (4.1.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib) (1.15.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install matplotlib\n",
        "\n",
        "%matplotlib inline\n",
        "import matplotlib.pylab as plt \n",
        "from matplotlib import cm\n",
        "\n",
        "import os\n",
        "import math\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import models, layers\n",
        "from tensorflow.keras.callbacks import LearningRateScheduler, ReduceLROnPlateau, ModelCheckpoint, Callback\n",
        "import tensorflow.keras.backend as K\n",
        "\n",
        "from IPython.display import clear_output\n",
        "from time import time, gmtime, strftime, localtime\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3d0bc77e",
      "metadata": {
        "id": "3d0bc77e"
      },
      "outputs": [],
      "source": [
        "from scipy import pi\n",
        "\n",
        "import seaborn as sns\n",
        "\n",
        "## run on CPU...\n",
        "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"-1\"\n",
        "\n",
        "# disable eager excution\n",
        "# tf.compat.v1.disable_eager_execution()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e9d37e4a",
      "metadata": {
        "id": "e9d37e4a"
      },
      "source": [
        " **PDE-NN for 1D convection-diffusion equation**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cbfabdb2",
      "metadata": {
        "id": "cbfabdb2"
      },
      "source": [
        "      \n",
        "**Prior**\n",
        "\n",
        "    1. Convection-Diffusion Equation: toy problem for a one-dimensional steady-state case \n",
        "        - v*u_x = k*u_xx\n",
        "    2. BC \n",
        "        - u = 0, x = 0\n",
        "        - u = 1, x = L\n",
        "    3. General solution\n",
        "        - u(x) = C1 + D1 exp(ux/k)\n",
        "    4. Given BCs specified in 2, the solution become\n",
        "        - u(x) = [1 - exp(Pe x/L)] / [1 - exp(Pe)]\n",
        "        - Pe = uL/k\n",
        "        "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4bce8bb7",
      "metadata": {
        "id": "4bce8bb7"
      },
      "source": [
        "# 2. Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9bab3fa0",
      "metadata": {
        "id": "9bab3fa0"
      },
      "outputs": [],
      "source": [
        "# function to generate analytical solution\n",
        "def eval_u(x, Pe, L):\n",
        "    x = tf.convert_to_tensor(x, dtype=tf.float32)\n",
        "    u = tf.convert_to_tensor((1. - tf.math.exp(Pe*x/L)) / (1. - tf.math.exp(Pe)))\n",
        "    return u#.eval(session=tf.compat.v1.Session()) \n",
        "\n",
        "def eval_Pe(u, k, L):\n",
        "    return u*L/k"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e0863f8b",
      "metadata": {
        "id": "e0863f8b"
      },
      "source": [
        "#### 2.1. Geometry & BC"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f7626c4f",
      "metadata": {
        "id": "f7626c4f"
      },
      "outputs": [],
      "source": [
        "# choose v (PDE parameter)\n",
        "v = 10.\n",
        "\n",
        "# fixed PDE & BC parameters k & L\n",
        "k = 1.\n",
        "L = 1.\n",
        "Pe = eval_Pe(v, k, L)\n",
        "\n",
        "# domain boundary\n",
        "x_l, x_u = 0, L"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bb245c9e",
      "metadata": {
        "id": "bb245c9e"
      },
      "source": [
        "#### 2.2. Sampling plan"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f6401cd5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "f6401cd5",
        "outputId": "d209546c-174e-4de7-bfa4-c66f535440dc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "# training sample = 5000\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV1bn/8c9zTk4SIMyBCDJPAgIKRHCgGsQBrIJTK6i1tlqvtvbXwdtbe3trrbbX2tbeDtqqrRPaiqitRkVxIuIEAjKPhjnMMwRIcob1+2OfQIxBImSfIef7fr14efba65z9rCTu56y99l7LnHOIiEjmCiQ7ABERSS4lAhGRDKdEICKS4ZQIREQynBKBiEiGy0p2AF9Ufn6+69at2zG9d//+/TRr1qxhA0pxanNmUJszw/G0ec6cOdudc+3q2pd2iaBbt27Mnj37mN5bUlJCUVFRwwaU4tTmzKA2Z4bjabOZrT3SPl0aEhHJcEoEIiIZTolARCTDKRGIiGQ4JQIRkQznWyIws0fNbKuZLTrCfjOzP5lZqZktMLMhfsUiIiJH5meP4HFg9OfsHwP0jv+7Cfirj7GIiKQ15xw4hx8zRvuWCJxz04Gdn1NlHDDReWYArcysg1/xiIikq1gsRmTGdNzunURnf9DgySCZD5SdCKyvsV0WL9tUu6KZ3YTXa6CgoICSkpJjOmB5efkxvzddqc2ZQW1u3Nz+fbw9dSpDRpzNu6vXYeVVYNZgn58WTxY75x4GHgYoLCx0x/pknZ5EzAxqc2bIlDZHKw6y+767uPjBhygePowRZSsIjLmIYPOWDXaMZN41tAHoXGO7U7xMRETiYhUHWbF2HX3atiEQCEDnbliz5g16jGQmgmLguvjdQ6cDe5xzn7ksJCKS0aqqWLptJyflt/a2z73ISwgNyLdLQ2b2NFAE5JtZGfBzIATgnHsQmAJcBJQCB4Bv+BWLiEjaysri7G6dGFiQTzlAdnbDH6LBPzHOOTfhKPsd8B2/ji8i0iiEw2QFjJPbt2VmfLuh6cliEZFUFosxZuLzLNv+eXfjHx8lAhGRFBWLxQhPfZHVu/bQu018jCAnp8GPo0QgIpKiYuV7WTX3Yzo2zyM3lAWBAIHcJg1+HCUCEZEU5aJRQjh+cOZQr6BZHoGc3AY/jhKBiEiqqqqiR5tW3DzsVG87twnWgE8UV1MiEBFJYTcXv86UFat8PYYSgYhIqgoG+ahsMx3ymnnbPvQGQIlARCQlxWIxIi9PpnTnLk7Kb+MVNvwM1IASgYhISoru28OOxYu4oFc3mmaHIJSNBf05ZSsRiIikospKCvKa8dz4cd72sBFgSgQiIpkjK4un5i+heFmptz14mG+HUiIQEUlF4TBTVqxiX2WVtx2J+HYoJQIRkVQUibBk6w76t8/3tn1Yq7iaEoGISIqJRqNEn32C9Xv20bd6HYJg0LfjpcVSlSIimSS2awfBPbvY8uNvkxUMQJt8Am3yfTueegQiIqkmEGD+5m1MX1vmbY//JkEfewRKBCIiqSYcpnhZKdNWr/O2YzFfD6dEICKSapzzBorbtT207SclAhGRVBONsnDLNk5u79+4QE1KBCIiKSQajcLzE3ni8jH0axefY8jH8QFQIhARSSmxndvZXraBFrk5hIJB3+8YAiUCEZHUEo1SvKyUe6bP9LYvv9bXO4ZAiUBEJOUs2LKNQQXtvA2fkwAoEYiIpJZYjAWbtzHohHYJO6QSgYhIiojFYvBmMdcPGcDgDu29woD/p2lNMSEikiKi+/YQKf2Eawb1IxgIQFYIa97C9+OqRyAikioqKnj1k9V85ZmXvO1hIwjmNvH9sEoEIiKpwjkWbNlG3+o1igcMxnxasL4mJQIRkRTyqYHiBCQBUCIQEUkd0Sgn5behsGNBQg+rwWIRkRRQPbXEXaPOOlyYgGcIwOcegZmNNrPlZlZqZrfXsb+LmU0zs7lmtsDMLvIzHhGRVBXbuZ25i5dyc/HrXkECppao5lsiMLMg8AAwBugPTDCz/rWq/Q8w2Tk3GBgP/MWveEREUlo0yuyNW4jG4lNOJ2BqiWp+9giGAaXOuVXOuSpgEjCuVh0HVN8k2xLY6GM8IiIp7eONWxhSPT6QoCQAYM6nBQ/M7EpgtHPuxvj214Dhzrlba9TpALwOtAaaAec55+bU8Vk3ATcBFBQUDJ00adIxxVReXk5eXt4xvTddqc2ZQW1Ofy4c5u6f38GVYy6kX69ekF+AhUKfqnM8bR45cuQc51xhXfuSPVg8AXjcOXefmZ0BPGlmA5xzn1qXzTn3MPAwQGFhoSsqKjqmg5WUlHCs701XanNmUJvTWzQaJXb/PUy98EyI7YMVc+Hs7xM6sfOn6vnVZj8vDW0AaraiU7ysphuAyQDOuQ+BXCAxoyMiIikitnM7yz5ZyR8+iF8QadIUK+iQsOP7mQhmAb3NrLuZZeMNBhfXqrMOGAVgZv3wEsE2H2MSEUk90SjvrS1j4Zb46e/qb5GVlbgLNr4lAudcBLgVmAosxbs7aLGZ3WVmY+PVbgO+ZWbzgaeB651fgxYiIins4001BooTmATA5zEC59wUYEqtsjtqvF4CnFX7fSIiGSUSZd6mrXztlJO97QR/H072YLGISEbznih+gjev/yrZ1beMxmKf/6YGprmGRESSKLZzO+vWrGX62jKygoGEDxSDEoGISHJFIryxcg3PLFzmbU+4MaEDxaBEICKSXJEwH2/cenigmMTfL6NEICKSVAHvjqEO1VNLJH7oVoPFIiJJEo1G4V8T+eNF5zKwIP4sbRLuoFePQEQkSWI7t7NtwwYK8prRJBRKykAxKBGIiCRPJMIzC5dx77szve0kDBSDEoGISPKEq/how2aGd6ruBSRnYgUlAhGRJHDOwZIFfFS2idNOjCeCrNDnv8knSgQiIkkQrTiI++g9/t/pQ+ib38YrTMJlIdBdQyIiSeH27qGyqopbhp2KmSV0jeLa1CMQEUmGygrueXcmv3pnhrd98VcStkZxbUoEIiLJEIsxq2wzgzskfo3i2pQIREQSLBaLEXtrCrM2bGZYpxO8wuycpMWjRCAikmDRvbspL13Odaf2p12zpl5hkyZJi0eJQEQk0cr30SI3h/vGjPS2u/Yk2Lxl0sJRIhARSTQLcPvr7/DC0k+87fMuJhBI3ulYiUBEJIGcczBvFlNWrKZrq3gvwCV2RbLalAhERBIocvAA2995k437yhnYPv7cQE5uUmNSIhARSaRdOyjbu48rT+7jLU3Zsg2Btu2SGpISgYhIIlmAUzu058GxF3jbV1ybtAfJqikRiIgkUmUl35/yNmt37/W2kzw+AEoEIiIJE41GOTD5MR6fu4h2TePPDVjyT8PJj0BEJEPEdmxj1tIVDCxoR9Ps+JTTbZMz0VxNSgQiIolScZCl23YwouuJ3vZpZ5HVpGlyY0LTUIuIJE4kys3DTiUWi69E1v8UbwrqJFOPQEQkAaLRKJF/Pckdb71/uDCYGt/FlQhERBIgtmMbCz5ZyQtLPyEQiPcCUmB8AJQIREQSo+Ig01avY2T3zt52iowPgM+JwMxGm9lyMys1s9uPUOerZrbEzBab2T/9jEdEJGkiUaavKaOoexdvO0XGB8DHwWIzCwIPAOcDZcAsMyt2zi2pUac38BPgLOfcLjNr71c8IiLJEo1G4cV/8tSVXyZUPctoiowPgL89gmFAqXNulXOuCpgEjKtV51vAA865XQDOua0+xiMikhSx7VspXb2GORs3kxuKJ4AUGR8Af28fPRFYX2O7DBheq04fADN7HwgCdzrnXqv9QWZ2E3ATQEFBASUlJccUUHl5+TG/N12pzZlBbU5trrKSJzfvpfzAAbIuHAxNm2EfzfrCn+NXm5PdN8kCegNFQCdgupkNdM7trlnJOfcw8DBAYWGhKyoqOqaDlZSUcKzvTVdqc2ZQm1NbeOUn3D37I753xlBGrJgL191CqHuvL/w5frXZz0tDG4DONbY7xctqKgOKnXNh59xqYAVeYhARaRSi0SgVz09k1obNjOgSf6I4K5TcoGrxMxHMAnqbWXczywbGA8W16ryA1xvAzPLxLhWt8jEmEZGEim3fSlZ5OW9e/1Va5OZ4hSk0PgA+JgLnXAS4FZgKLAUmO+cWm9ldZjY2Xm0qsMPMlgDTgB8553b4FZOISMLt28PibTvo2qqFtz3k9JR5fqCar88ROOemOOf6OOd6Oud+FS+7wzlXHH/tnHM/dM71d84NdM5N8jMeEZFEcs7BkoXc+vKbzNsUvylywOCUeX6gmp4sFhHxSeTgAXZ9UMLirds5q3p8IJSd1JjqokQgIuKXHdt4a9U6vtS1k/f8QHYudkLHZEf1Gcm+fVREpPE6eIAhHdrTvXp84LIJZGWl3mlXPQIRER/EYjFcyetEnWNIxwKvMMUGiaspEYiI+CC6dzeL583l4if/dbhQiUBEJIPs3cPrpWu4oFc37y6hlq0JtG2X7KjqpEQgIuKH/fsPJQIARl9KMBhMakhHokQgItLAwuEwTH6MrwzoQ1H1QjTZOckN6nOk3vC1iEi627CWvRWVfGPwwPiylAHo3C3ZUR2RegQiIg0t6vjJG+/ywEdzve0J3yQUSq2J5mpSIhARaUCxWAz35su8smIlF/bq7hUGU/tUm9rRiYikmeieXcz9eA552dn0yW/tFea1SG5QR6FEICLSkHZsIysQ4KfnnO5tN29JID+1l2Ov12Cxmd1RV7lz7q6GDUdEJH0552DhXPrmt2HQCfFnBi66PGVvG61W3x7B/hr/osAYoJtPMYmIpKXIgf2snf42A+9/3EsKAM2aJzeoeqhXj8A5d1/NbTP7Hd6iMiIiUm3zBl5ZvpIvdevkPU0cysE6nJjsqI7qWMcImuKtQSwiIsQvCy34mJeWr+Tik3p4hZeOT8nZRmur7xjBQiDezyEItAM0PiAiEhc5sB83fxYD2udzXo+uXmFuk+QGVU/1TVUX13gdAbbE1yQWERGALRtxDn47uihekNpPE9dUr0tDzrm1Nf5tUBIQEanlYAVXPlPMWyvXettXXJPSTxPXpOcIRESOUzgcZveTDzF9zXqGdergFaZJEgAlAhGR41e2huLlKynq3pnmOfHF6bv3Tm5MX4ASgYjI8dp/gFAgwI1DB3nbl19LKDs7uTF9Aal/X5OISAoLV1XhnnuCCYP6HS5MoyQA6hGIiByfNaU8OX8Jt7067XBZGl0WAiUCEZHjU1nFc4uWU3jiCd72Zdek1WUhUCIQETlm4aoqNj/xIDPKNjG2by+vMCd1l6Q8EiUCEZFjtXIFW/cf4LazTqNZdvx20TS7LAQaLBYROSaxWAzeepUB7fMPTzk9dnzaXRYC9QhERI5JdPdOFi9dxDmPTjpc2LZd8gI6DkoEIiLHYtMG/jF/KWd1iU8zHczCOqbnpMy+JgIzG21my82s1Mxu/5x6V5iZM7NCP+MREWkI0WiUWPGzTFq4lGtOiT8/MPaqtJhyui6+JQIzCwIP4K1m1h+YYGb966jXHPgeMNOvWEREGlJs2xZ27d7J5f37MLAgfjmo/QnJDeo4+NkjGAaUOudWOeeqgEnAuDrq3Q3cC1T4GIuISMPZvIkWOdn8rnrK6aZ5BNoVJDWk4+FnP+ZEYH2N7TJgeM0KZjYE6Oyce8XMfnSkDzKzm4CbAAoKCigpKTmmgMrLy4/5velKbc4ManPiOOfY88ky/uOvTzHxvt+SHQpBqzbYu+/6fmy/2py0C1pmFgB+D1x/tLrOuYeBhwEKCwtdUVHRMR2zpKSEY31vulKbM4PanDjhJQv5473PcF7H9py7epFXePv/EkrAg2R+tdnPS0MbgM41tjvFy6o1BwYAJWa2BjgdKNaAsYikqmg0intxEn+fs5Abhw70CsdckZAk4Cc/E8EsoLeZdTezbGA8UFy90zm3xzmX75zr5pzrBswAxjrnZvsYk4jIMYtt3cKB8n1c1q/34dtGO3dNblANwLdEEF/O8lZgKrAUmOycW2xmd5nZWL+OKyLimw3rcM5x16izMDPIbUYgje8WqubrGIFzbgowpVbZHUeoW+RnLCIixyNcVcWOZycy6P7HWf69G8jLyYYLLiEYDCY7tOOmJ4tFROrjk2U8Mmcho3t395IAQP9ByY2pgSgRiIgcRSQSIfzMozw4ax63nj7EK2wEg8TVlAhERI7Cla2lMhLlJ2efzuAO7b3CRjBIXE2JQETkc8RiMXjxWTaV7+dbhfFLQblNGsUgcTUlAhGRzxHduYOPlyxizMTniMZiXuG4CY1ikLiaEoGIyBE45+CtKfzxwzncfNqpBAPxU2bPPskNrIEpEYiIHEGkfB9rPnyXqaVrDj9JfPFXCYVCyQ2sgSkRiIgcyYqltGvWlH9NGEerJrle2YBTkxuTD9JzFQUREZ+FKyvZOulxFm/dzsgeXbzC88Y2mltGa1IiEBGpy+L5/HHGHPZVVh1OBAMbX28AdGlIROQzwpWV7Hl2Io/OWcgPz4xPiNyrP1nNWyQ3MJ8oEYiI1LZ4PhPnLWZ07+50a93SK7v4Cm+iuUZIl4ZERGoIV1bCS89wy2mnsq+qyivs1Z+sFi2TG5iP1CMQEalp8Twe/GgeczdtoXX1nUKNuDcASgQiIoeEKyvZ+vTj3DntAwrymnmF3U9q1L0BUCIQETls4Vx+895HTBjYly6t4gPDY7/SqHsDoEQgIgJA+MABYi9NZsnWHfzXl4Z5hd37kNWyVXIDSwANFotIxnPOwb//QdTFmHLdFYd3XPm1Rt8bAPUIRESI7NjBrJJpnPvYZC8pAIz8MqGmTZMbWIIoEYhIRotEIrj7/5f/nPoONw4deLgHMOzM5AaWQEoEIpLR3KpSnl28goPhCF875WSv8NJrCOXmJjewBFIiEJGMFa6shKf/Ru82rfjrJecRCMR7A/0HJjewBNNgsYhkrvmzmbJiFWd37UReTrZXduP3G916A0ejHoGIZKTw7t0sfPwhvvXCVPaHw15h81YEO5yY3MCSQIlARDJOOBwm9n938Z2X3uTOc886/BTxTT8gEMi802LmtVhEZOkiZpZtIhgIcMOQ+HjAJeMJ5eUlN64kUSIQkYwS3rOH6PMTOaNLR974+lcODxA30kVn6kOJQEQyRjgcJvb7X3DJU/+mZPU6soLxU+At/5VxA8Q1KRGISOZYupAHZ81jX1UVI7p08sp6DyCrXfvkxpVkSgQikhHCO3fyyd/v5+6SD3nk0tGHewNXXJ0R8wl9Hl8TgZmNNrPlZlZqZrfXsf+HZrbEzBaY2Vtm1tXPeEQkM4UPHIA//4oOec14+qsX0ye/tbfj5h8RyslJbnApwLdEYGZB4AFgDNAfmGBm/WtVmwsUOucGAc8Bv/ErHhHJTNFoFP50D/e9P4tN5fsp6t7F29F3EFntC5IbXIrws0cwDCh1zq1yzlUBk4BxNSs456Y55w7EN2cAnXyMR0QyUGzlJ7wwbz4PzZpPftMmh3dcNiHjLwlVs0NTrjb0B5tdCYx2zt0Y3/4aMNw5d+sR6t8PbHbO/bKOfTcBNwEUFBQMnTRp0jHFVF5eTl6G3SesNmcGtbluLhpl06L5fOeOO/nVf95Gv149vR0FHbE0fHDseH7PI0eOnOOcK6xrX0rMNWRm1wKFwDl17XfOPQw8DFBYWOiKioqO6TglJSUc63vTldqcGdTmzwrv2we/v5OZ6zdyzznDuSG2F1bMheu+Tah7z8QF2oD8+j37mQg2AJ1rbHeKl32KmZ0H/BQ4xzlX6WM8IpIhwhUVxH73c/65cCkTBvZleOeO3o427Qh27Z7c4FKQn32jWUBvM+tuZtnAeKC4ZgUzGww8BIx1zm31MRYRyRDRaBT+8EvuKvmAv81eQCQWO7wzQ+cSOhrffiLOuQhwKzAVWApMds4tNrO7zGxsvNpvgTzgWTObZ2bFR/g4EZF6iS1dwtOzPuap+UuYfNUl5GTFL3z84Oe6VfQIfB0jcM5NAabUKrujxuvz/Dy+iGSW8Pr18PzjzN20lX9ffenhWUW/8xNCLVokN7gUlhKDxSIixytcVsacu28nKxjkNxfWuO/kpv8klJ+fvMDSgC6WiUjaC2/YwPJ7f8ZlT7/Iut17D++49hZCHTokL7A0oUQgImktvGEDy3/9P4x+4jl+OWoEl/SN3xo6YChZPdLzNtFE06UhEUlb4bIyeOT/OBCO8MvzRnDNKfFZbFq3I3DpVXpyuJ7UIxCRtOSqqlj66//hv994l8Ed2h9OAoEQ3HIbwWAwuQGmESUCEUkrzjnCK5azZMYHXPD4s/Rv3/bwzkAIfnxXRi8ycyx0aUhE0ka4qgqefpxFM9/np0/9m4njLmBMnx7ezkA2/PgXhLKzkxtkGlIiEJG0UD130OZ9++nfri1//PnPGFO+2dvZph3cfJt6AsdIl4ZEJOWFt20j/Nuf8f0pb3PV5Jcwgy4d47eFnlxI4Ns/UhI4DuoRiEjKisViRJcvZdtj9zNh8ss0yw7x4tWXHr4b6JqbyerZS3cHHSclAhFJSeGDB+HPv8Yd2Meug5Wc070z/332cILVk8blFxDq1Tu5QTYSSgQiknLCW7cSfeAe7pk+k637D/CnL4/iZ0VnHK7wnZ9gixYlL8BGRolARFJGuLISSt5k7Wsv8M1/v0pWIMBjl435dKXbfkEow1Zj85sSgYgknXOOyPp1uEf/iJlRvKyU0b2788MzCw9fCrpkAjboVLKydNpqaPqJikhShffuhYf/wJLVq7i5+HXuOf9svnv6kE9X+o8fETrhhOQEmAGUCEQkKcIHD8IbUzgw8x3unPYBT85bzJ3nnsUZ1ctKAnTuAeO/Qahp0+QFmgGUCEQkocIVFTD9baLvv8G6Pfvo1CKP3Kws5n3n64cXkgG4/rtkdemqW0MTQIlARBIiXFEB704j9t4bPL9kBXeXfMiwTh34+6UXcteosw5XPHMUjBhJqEmT5AWbYZQIRMRX4fJyeK0YFs8B4IYXXmPFjl38bnQR5/fs+unK3/0poTZtkhBlZlMiEJEG55wjsnUL/OtpDm5YzdMLljFp4TJeuvYyfje6iDZNcj99yWf8jQR69dHU0UmiRCAiDSZ84AC89zbMeBdchGcXLecHr06jsGMBPzl7ONnBIG2b1jjtXHApDBlGKCcneUGLEoGIHJ9wVRUsnAsffUBk0zpeK13Nk/OW8NDY8xlY0I43r/8qfdvVutxz7sVw2hmEcnOTE7R8ihKBiHxh4YoKmD0Dli7CbViFmTFlxSpuKX6Drq1a8I0hA8jJCn42AZw/FoYMVwJIMUoEIlIv4T17YOqLsGMnbss65m3exvOLV/CvJSt47borGVjQjlevu/LTK4YBdO8LhadD776aKjpFKRGISJ3C5eVQ8jps3wo7drB/5xbeXVvGhb268bfZC7jv/dlcfnJvnrryy3Ru2fyz9/sPLIRRFxFq2TI5DZB6UyIQEW/e/41l3om/ogIOHCS2YxOBgPHK8lXcP/NjZpZtorDjCQw78QSuHzyAbxUO+uzJv0c/OGUI9B2gJSPTiBKBSIYJh8OwbDHMmwWVlYDBhlUAzFy/kamla3hv7QY+2bGLlT/4Fq1yc7j5tFOZfNVYmufUcXLv3BN69oHTztRUEGlKiUCkkQpXVMCsD6F0OTgHOO/Ev7UMgJLV65i9YQsfb9pCblYWj142mnfWlFEZjfKDM4dyRueOBALGWV1P/OyH9zoZuveAwcP1BHAjoEQgkqZqfrN3TVoRfvSvgINYDCqrcNvKcA72h8P8Y/4Slm3fybJtOynq3pnbzx7OE3MX06ZJLhf16UFhR29mz//60rC6D3ZCV2jbFs44h2CHjgQCWu68MVEiEEkx4fJymDYVtm0BB7gYWMD7byBIzW/2+6vCrNm9hw+b5nN6kxhzNm7mt+/PYs2uvazZvYeHxl7Ahb26MX/zNvq1a8tFfXowqKAdAI9dPqbuAPI7QpOmkNcMBgzW3T4ZQIlApIGFd++GV/8NBw96386rT96xGif02if26n379hLZuYVdFRXsOFBBXnaITi2b8/Cs+azatZtN+/bTMjeHP315FN975S0em7uILi1b0KpTZ24cPYL2ec0YP7Af3Vu1oFvrloemcvjr2PPrDrb9iZDTBAIGbdrAORfqLp8M5GsiMLPRwB+BIPB359yva+3PASYCQ4EdwFXOuTV+xiSNS3jfPnjrVdi1E6IRsACuZTvCj/yl7hPtkcoaat/OnVTt2c6+qir2VlYRCgTo1LI5M9dvZN2efZRXVVEZiXLzsFN5c+VaJs5bzM4DB9l5sII/fXkUzXOyOfWBJ2iVm0Obpk245bRTuPX0Iew8WEHbpk04uX07erT2TtS/ufAc/nDRuZgZ7/UZTOsVc2ndJJfures4kXfoBllBL/bsHOjQAc4o0pKPAviYCMwsCDwAnA+UAbPMrNg5t6RGtRuAXc65XmY2HrgXuMqPeCKRCK6igvBTf/duj0vESSFRJ5/P2edaHOWkmM4/h/L9xHZtIRyLEo7GCAaMJqEQuyOwbtlcwrEYAD3btGLD3n1s2FtOOBojHIsytOMJRGIx3l1bFi+LMaB9PgMK8vnb7AXsOlhBRSRCQV4z/uO0U3hi7iJKVq/nYCRCRSTKxMvH8PGmLfz49elURKIcDIf55XlfYlzfnrT+3/tpnh2ieU42Y/v24vdjRvLyilWs2L6L5jnZtMr15tUpyGvKBb260bZJLm2bNuGk/DY0C4XY/7PvEwh8+rbM288e/pm/6ZyaSzZmhaBzr0//jHKyoXdfOKVQT/LK5/KzRzAMKHXOrQIws0nAOKBmIhgH3Bl//Rxwv5mZc841ZCCRSAT3qx/z4todPLNwHg6HYfz54lG8u6aMpxcuPVT3lmGn0qlFc37yxvRDZSO6duLaU/pzd8mHrN+zF4DWubnce+E5vLD0E15evorqkO8YeSbRWIxfvjPjUNml/Xozrl8vvvfKW+yuqASgW+uW/OLcs3hkzgJKVq8/dKw/f3kUK3fu5vcfzDlUduPQgZzTrTPXPPcKDu8zh3Ys4EcjhvHr6TOZu2nroTY9c9UlvF66hr/NXoDDsSOvFb8bfBJdW7XkpuLXD33mBdpcfFYAAAiySURBVD27cvOwU/nhq9NYtWsPAG2b5PLIZaP5x/wlTF60/FDd348ZSTTmuO21aYfadPUp/bl6UD+ufvZldh6oIOYcvdu25oFLzuM3737ES8tXEnOOmHO8fO3lLNyyjdteKyEa88p+PvJMLu3XiwF/foyYg5hznNujCw+Nu4CvPz+Ft1atIxY/1sb/uoXHPl7ID18tOfSZz44fy4D2+fT4v78RCgQIBQPcOHQQvxtdxK/u/wtb160jFAzQsXkeb3/zKl5evoon5i4iFAwSCgT469jzcQ4e+3gRoWCAUCBIblaQAQX5rN+zj3AsSm5WFk3iJ9serVsRsAC5WUFys7LIyQoyqKAdf7n4PHJDXr0T8pqRk5XF/p997zP31989asRn/i4HFrRjYPx6/RF17A5G3Qmxxjd7mz2b0Fd8+Q4lGcAa+Jx7+IPNrgRGO+dujG9/DRjunLu1Rp1F8Tpl8e2V8Trba33WTcBNAAUFBUMnTZr0xYKJhHHbtjBt4RL2bt54qHjc+eexuqyMRctXHCo7bdAgWjbP460PPjxU1rVjRwb2PYl3Z81mX3k5ADk5OYw68wxWrF7DyrXr4nHCGUMGg4MZ8+Z5ZRg9u3amZ9eulMyYSVU4jJnRIi+P4aeewtLSUjZu2XroWCNOK2TvvnIWLj98Ij6pZw86tm/POzM/OlSW37o1A07qw8Jly9m5Z0/1z4mzh53Gxi1bKV27FoDK7FyGdutMs6ZNmb1g4aH3n9Aun55duzJ/6VL2HzgIQHZ2iMKBA1m3cRMbt2w5VHfASX3AORZ/UnroOCcWFHDiCQXMXbyEaCxKIBCgWZMmnNSjBxs2b2H33r0EAoZZgF5du1BRVcWWbdswC2ABI79Va/KaNWXjlq2YGYGAkZOdTeuWLdlbXk4kGiVghmG0bNGccCRCOBwmEAgQMCOYlUUgfrKtfdItz2lKXuWBev5x+MkglO2dyAHq+l+trn3BIDRtiuXU/1t8eXk5eRl2mUdt/mJGjhw5xzlXWNe+tEgENRUWFrrZs2d/oViqewTv9RnMiBVzj6E16Suj2tyhGwSDvNeyHSP2bEveZbrWbeBL55HVrl3CllksKSmhqKgoIcdKFWrzF2NmR0wEfl4a2gB0rrHdKV5WV50yM8sCWuINGjeorKwsIj+9F958E3r2y6gxAkI50Kln4xwjsIB3i+OwEQS6dDu0qImVlBC68isN/Wck0mj5mQhmAb3NrDveCX88cHWtOsXA14EPgSuBtxt6fKBaVlYWlptL6Nob/fj4lKWToogcjW+JwDkXMbNbgal4t48+6pxbbGZ3AbOdc8XAI8CTZlYK7MRLFiIikkC+PkfgnJsCTKlVdkeN1xWAvq6KiCSRJgwREclwSgQiIhlOiUBEJMMpEYiIZDjfHijzi5ltA9Ye49vzgSM+rNZIqc2ZQW3ODMfT5q7OuTrnNEm7RHA8zGz2kZ6sa6zU5sygNmcGv9qsS0MiIhlOiUBEJMNlWiJ4ONkBJIHanBnU5szgS5szaoxAREQ+K9N6BCIiUosSgYhIhmuUicDMRpvZcjMrNbPb69ifY2bPxPfPNLNuiY+yYdWjzT80syVmtsDM3jKzrsmIsyEdrc016l1hZs7M0v5Ww/q02cy+Gv9dLzazfyY6xoZWj7/tLmY2zczmxv++L0pGnA3FzB41s63xhbvq2m9m9qf4z2OBmQ057oM65xrVP7wpr1cCPYBsYD7Qv1adbwMPxl+PB55JdtwJaPNIoGn89S2Z0OZ4vebAdGAGUJjsuBPwe+4NzAVax7fbJzvuBLT5YeCW+Ov+wJpkx32cbT4bGAIsOsL+i4BX8RY6PR2YebzHbIw9gmFAqXNulXOuCpgEjKtVZxzwRPz1c8AoS9Sagv44apudc9Occ9UL+c7AWzEundXn9wxwN3AvUJHI4HxSnzZ/C3jAObcLwDm3lfRWnzY7oEX8dUtgI2nMOTcdb32WIxkHTHSeGUArM+twPMdsjIngRGB9je2yeFmddZxzEWAP0DYh0fmjPm2u6Qa8bxTp7KhtjneZOzvnXklkYD6qz++5D9DHzN43sxlmNjph0fmjPm2+E7jWzMrw1j/5bmJCS5ov+v/7Ufm6MI2kHjO7FigEzkl2LH4yswDwe+D6JIeSaFl4l4eK8Hp9081soHNud1Kj8tcE4HHn3H1mdgbeqocDnHOxZAeWLhpjj2AD0LnGdqd4WZ11zCwLrzu5IyHR+aM+bcbMzgN+Cox1zlUmKDa/HK3NzYEBQImZrcG7llqc5gPG9fk9lwHFzrmwc241sAIvMaSr+rT5BmAygHPuQyAXb3K2xqpe/79/EY0xEcwCeptZdzPLxhsMLq5Vpxj4evz1lcDbLj4Kk6aO2mYzGww8hJcE0v26MRylzc65Pc65fOdcN+dcN7xxkbHOudnJCbdB1Odv+wW83gBmlo93qWhVIoNsYPVp8zpgFICZ9cNLBNsSGmViFQPXxe8eOh3Y45zbdDwf2OguDTnnImZ2KzAV746DR51zi83sLmC2c64YeASv+1iKNygzPnkRH796tvm3QB7wbHxcfJ1zbmzSgj5O9Wxzo1LPNk8FLjCzJUAU+JFzLm17u/Vs823A38zsB3gDx9en8xc7M3saL5nnx8c9fg6EAJxzD+KNg1wElAIHgG8c9zHT+OclIiINoDFeGhIRkS9AiUBEJMMpEYiIZDglAhGRDKdEICKS4ZQIREQynBKBiEiGUyIQOU5mdlp8XvhcM2sWXwdgQLLjEqkvPVAm0gDM7Jd4Uxs0Acqcc/ckOSSRelMiEGkA8XlwZuGte3Cmcy6a5JBE6k2XhkQaRlu8uZya4/UMRNKGegQiDcDMivFWz+oOdHDO3ZrkkETqrdHNPiqSaGZ2HRB2zv3TzILAB2Z2rnPu7WTHJlIf6hGIiGQ4jRGIiGQ4JQIRkQynRCAikuGUCEREMpwSgYhIhlMiEBHJcEoEIiIZ7v8DqqwV8d8filgAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# training data\n",
        "n = 5000\n",
        "x = np.linspace(x_l, x_u, n)\n",
        "X_train = np.hstack([x.reshape(-1, 1), np.ones((len(x), 1))])\n",
        "y_train = np.reshape(eval_u(x, Pe, L),(-1, 1))\n",
        "\n",
        "print ('# training sample = %d' %len(y_train))\n",
        "\n",
        "# viz training data\n",
        "plt.plot(x, eval_u(x, Pe, L), \"k--\", linewidth=1); plt.scatter(X_train[:, 0], y_train, c='salmon', alpha=.5, s=5);\n",
        "plt.xlabel('x'); plt.ylabel('u'); plt.grid();"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ff66fa06",
      "metadata": {
        "id": "ff66fa06"
      },
      "outputs": [],
      "source": [
        "# BC data\n",
        "bc = np.where((X_train[:,0] == x_l) | (X_train[:,0] == x_u))[0]\n",
        "X_bc, y_bc = X_train[bc], y_train[bc]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d2a06ced",
      "metadata": {
        "id": "d2a06ced"
      },
      "outputs": [],
      "source": [
        "# evaluation data\n",
        "X_eval, y_eval = X_train, y_train\n",
        "\n",
        "# PDE data\n",
        "X_pde, y_pde = X_train, y_train\n",
        "X_ic, y_ic = X_bc, y_bc\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "2.3 Preparation of training data"
      ],
      "metadata": {
        "id": "YljstXTrKyxC"
      },
      "id": "YljstXTrKyxC"
    },
    {
      "cell_type": "code",
      "source": [
        "# create multiple batches\n",
        "\n",
        "# taking a random sample of 498 PDE points and 2 BC points\n",
        "batch_pde, batch_ic, batch_bc = (498, 0, 2)\n",
        "batch_per_epoch = 100\n",
        "n_pde, n_ic, n_bc = len(X_pde), len(X_ic), len(X_bc)\n",
        "# i = 0\n",
        "# while i < batch_per_epoch:\n",
        "# pick pde sample\n",
        "idxs = np.random.choice(n_pde, batch_pde)\n",
        "batch_X_pde, batch_y_pde = X_pde[idxs], y_pde[idxs]\n",
        "# pick ic sample\n",
        "idxs = np.random.choice(n_ic, batch_ic)\n",
        "batch_X_ic, batch_y_ic = X_ic[idxs], y_ic[idxs]  \n",
        "# pick bc sample \n",
        "idxs = np.random.choice(n_bc, batch_bc)\n",
        "batch_X_bc, batch_y_bc = X_bc[idxs], y_bc[idxs]\n",
        "# combine all sample\n",
        "batch_X, batch_y = np.vstack([batch_X_pde, batch_X_ic, batch_X_bc]), np.vstack([batch_y_pde, batch_y_ic, batch_y_bc])  \n",
        "# train_dataset = np.hstack([batch_X, batch_y])\n",
        "    # i += 1\n",
        "#batch_X = tf.convert_to_tensor(np.reshape(batch_X, (-1, 2)))\n",
        "batch_X = np.reshape(batch_X, (-1, 2))\n",
        "batch_y = np.reshape(batch_y, (-1, 1))\n"
      ],
      "metadata": {
        "id": "ca9CsizSjseG"
      },
      "id": "ca9CsizSjseG",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# \"TypeError: 'PrefetchDataset' object is not subscriptable\"\n",
        "# # Sampling Plan is able to continuous generate \n",
        "\n",
        "# # define sampling plan\n",
        "# def SamplingPlan_B1(batch_size=( ), batch_per_epoch=1):\n",
        "#     batch_pde, batch_ic, batch_bc = batch_size\n",
        "#     n_pde, n_ic, n_bc = len(X_pde), len(X_ic), len(X_bc)\n",
        "#     i = 0\n",
        "#     while i < batch_per_epoch:\n",
        "#         # pick pde sample\n",
        "#         idxs = np.random.choice(n_pde, batch_pde)\n",
        "#         batch_X_pde, batch_y_pde = X_pde[idxs], y_pde[idxs]\n",
        "#         # pick ic sample\n",
        "#         idxs = np.random.choice(n_ic, batch_ic)\n",
        "#         batch_X_ic, batch_y_ic = X_ic[idxs], y_ic[idxs]  \n",
        "#         # pick bc sample \n",
        "#         idxs = np.random.choice(n_bc, batch_bc)\n",
        "#         batch_X_bc, batch_y_bc = X_bc[idxs], y_bc[idxs]\n",
        "#         # combine all sample\n",
        "#         batch_X, batch_y = np.vstack([batch_X_pde, batch_X_ic, batch_X_bc]), np.vstack([batch_y_pde, batch_y_ic, batch_y_bc])  \n",
        "#         yield batch_X\n",
        "#         i += 1\n",
        "# BPE =100\n",
        "# batch_X = tf.data.Dataset.from_generator(SamplingPlan_B1, args=[(499, 0, 1), BPE],\n",
        "#                                       output_signature=(tf.TensorSpec(shape=(500, 2), dtype=tf.float32)))\n",
        "# batch_X = batch_X.repeat()\n",
        "# batch_X = batch_X.prefetch(5)\n",
        "# batch_X"
      ],
      "metadata": {
        "id": "3PksHBx0Zszn"
      },
      "id": "3PksHBx0Zszn",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "ce1c1371",
      "metadata": {
        "id": "ce1c1371"
      },
      "source": [
        "# 3.0 Set up the neural network architecture"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "350df4e6",
      "metadata": {
        "id": "350df4e6"
      },
      "outputs": [],
      "source": [
        "# specify FF-PINN\n",
        "def create_nn(scheme, sigma, lmbda, n_ffs, n_nodes, lr_int):\n",
        "    # input layers -> split into (x, w)\n",
        "    inputs = layers.Input(shape=(2,))\n",
        "    x, w = layers.Lambda( lambda k: tf.split(k, num_or_size_splits=2, axis=1))(inputs)\n",
        "\n",
        "    # initializer\n",
        "    if (scheme == 'He'):\n",
        "        initializer = tf.keras.initializers.HeUniform()  # hidden layers initializer\n",
        "    if (scheme == 'Xavier'):\n",
        "        initializer = tf.keras.initializers.GlorotUniform()  # hidden layers initializer\n",
        "\n",
        "    if (sigma > 0):\n",
        "        initializer_ff = tf.keras.initializers.TruncatedNormal(stddev=sigma)  # features initializer\n",
        "    else:\n",
        "        initializer_ff = initializer  # features initializer\n",
        "\n",
        "    # features mapping\n",
        "    hidden_f = layers.Dense(n_ffs*2, activation='linear', kernel_initializer=initializer_ff)(x)\n",
        "    hidden_f = tf.math.sin(2*tf.constant(pi)*hidden_f)\n",
        "\n",
        "    # hidden layers\n",
        "    hidden_1 = layers.Dense(n_nodes, activation=use_this_activation, kernel_initializer=initializer)(hidden_f)\n",
        "    hidden_2 = layers.Dense(n_nodes, activation=use_this_activation, kernel_initializer=initializer)(hidden_1)\n",
        "    hidden_l = layers.Dense(n_nodes, activation=use_this_activation, kernel_initializer=initializer)(hidden_2)\n",
        "\n",
        "    # output layers\n",
        "    u = layers.Dense(1, use_bias=False, kernel_initializer=initializer)(hidden_l)\n",
        "    \n",
        "    # initiate model\n",
        "    nn = models.Model(inputs=inputs, outputs=u)      \n",
        "  \n",
        "    return nn\n",
        "\n",
        "# activation\n",
        "def use_this_activation(y):\n",
        "    if use_this_acf == 'tanh':\n",
        "        return tf.math.tanh(y)\n",
        "    if use_this_acf == 'sin':\n",
        "        return tf.math.sin(y)\n",
        "    if use_this_acf == 'sigmoid':\n",
        "        return 1. / (1. + tf.math.exp(-y))\n",
        "\n",
        "# f: IC/BC\n",
        "def tf_bc(x):\n",
        "    u = tf.convert_to_tensor((1. - tf.math.exp(Pe*x/L)) / (1. - tf.math.exp(Pe)), dtype=tf.float32)\n",
        "    return u"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "89c9352e",
      "metadata": {
        "scrolled": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "89c9352e",
        "outputId": "4bd811c2-d4e4-40ec-ed2d-8e997e5121df"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 2)]               0         \n",
            "                                                                 \n",
            " lambda (Lambda)             [(None, 1),               0         \n",
            "                              (None, 1)]                         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 32)                64        \n",
            "                                                                 \n",
            " tf.math.multiply (TFOpLambd  (None, 32)               0         \n",
            " a)                                                              \n",
            "                                                                 \n",
            " tf.math.sin (TFOpLambda)    (None, 32)                0         \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 10)                330       \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 10)                110       \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 10)                110       \n",
            "                                                                 \n",
            " dense_4 (Dense)             (None, 1)                 10        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 624\n",
            "Trainable params: 624\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# initiate NN model (& pathway to internal values)\n",
        "scheme = 'Xavier'\n",
        "use_this_acf = 'tanh'\n",
        "\n",
        "sigma = .5\n",
        "lmbda = 500.\n",
        "# Initialise the model\n",
        "model = create_nn(scheme, sigma, lmbda, n_ffs = 16, n_nodes = 10, lr_int = 0.001)\n",
        "\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 4. Define the routines to determine the loss and gradients"
      ],
      "metadata": {
        "id": "vRlRSIQ92HLl"
      },
      "id": "vRlRSIQ92HLl"
    },
    {
      "cell_type": "code",
      "source": [
        "def get_deriviatives(model, batch_X):\n",
        "    # A tf.GradientTape is used to compute derivatives in TensorFlow\n",
        "    with tf.GradientTape(persistent=True) as tape:\n",
        "      # split to commute partial deriviatives\n",
        "      x, w = tf.convert_to_tensor(batch_X[:,0:1]), tf.convert_to_tensor(batch_X[:,1:2])\n",
        "      # Variable x is watched during tape\n",
        "      # to compute derivatives u_x and u_xx\n",
        "      tape.watch(x)\n",
        "      # Determine residual \n",
        "      u = model(tf.stack([x[:,0], w[:,0]], axis=1))\n",
        "      # u = model(tf.stack(batch_X))\n",
        "      # Compute gradient u_x within the GradientTape\n",
        "      # since we need second derivatives\n",
        "      u_x = tape.gradient(u, x)\n",
        "    u_xx = tape.gradient(u_x, x)\n",
        "    x = tf.cast(x, tf.float32)\n",
        "    u_x = tf.cast(u_x, tf.float32)\n",
        "    u_xx = tf.cast(u_xx, tf.float32)\n",
        "    del tape\n",
        "    \n",
        "    return x, u, u_x, u_xx"
      ],
      "metadata": {
        "id": "7So21Nin2TEy"
      },
      "id": "7So21Nin2TEy",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def compute_loss(model, batch_X):\n",
        "    x, u, u_x, u_xx = get_deriviatives(model, batch_X)    \n",
        "    # BCs (which define the problem)\n",
        "    t_u = tf_bc(x)  # ground truth\n",
        "    _bc = ( tf.equal(x, x_l) | tf.equal(x, x_u) )\n",
        "    bc_mse = tf.compat.v1.losses.mean_squared_error(labels=tf.boolean_mask(t_u, _bc), predictions=tf.boolean_mask(u, _bc))    \n",
        "      \n",
        "    # PDE (physics laws): v*u_x = k*u_xx   \n",
        "    pde = v*u_x - k*u_xx \n",
        "    \n",
        "    # exclude BC points \n",
        "    pde = tf.boolean_mask(pde, ~_bc)\n",
        "    pde_mse = tf.compat.v1.losses.mean_squared_error(labels=tf.zeros_like(pde), predictions=pde)\n",
        "    pde_mse = pde_mse / lmbda    \n",
        "    \n",
        "    # # u mse (for validation)\n",
        "    # u_mse = tf.compat.v1.losses.mean_squared_error(labels=t_u, predictions=u)    \n",
        "\n",
        "    loss = compute_physics_loss(pde_mse, bc_mse)\n",
        "\n",
        "    return loss, bc_mse, pde_mse"
      ],
      "metadata": {
        "id": "Dn8tj8nW1OHC"
      },
      "id": "Dn8tj8nW1OHC",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0db7866b",
      "metadata": {
        "id": "0db7866b"
      },
      "outputs": [],
      "source": [
        "# define loss function (Physics: PDE + BC loss)\n",
        "def compute_physics_loss(pde_mse, bc_mse):\n",
        "  return (pde_mse + bc_mse)\n",
        "    # def physics_loss(y_true, y_pred): \n",
        "    # return physics_loss\n",
        "\n",
        "# define loss function (BC loss)\n",
        "def compute_bc_loss(bc_mse):\n",
        "    def bc_loss(y_true, y_pred): return bc_mse\n",
        "    return bc_loss # return a function \n",
        "\n",
        "# define loss function (PDE loss)\n",
        "def compute_pde_loss(pde_mse):\n",
        "    def pde_loss(y_true, y_pred): return pde_mse\n",
        "    return pde_loss # return a function\n",
        "\n",
        "# define loss function (u loss)\n",
        "def compute_u_loss(u_mse):\n",
        "    def u_loss(y_true, y_pred): return u_mse\n",
        "    return u_loss # return a function"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def get_grad(model, batch_X):\n",
        "    \n",
        "    with tf.GradientTape(persistent=True) as tape:\n",
        "        # This tape is for derivatives with\n",
        "        # respect to trainable variables\n",
        "        tape.watch(model.trainable_variables)\n",
        "        loss, bc_mse, pde_mse = compute_loss(model, batch_X)\n",
        "\n",
        "    gradient_loss = tape.gradient(loss, model.trainable_variables)\n",
        "    gradient_bc_mse = tape.gradient(bc_mse, model.trainable_variables)\n",
        "    gradient_pde_mse = tape.gradient(pde_mse, model.trainable_variables)\n",
        "    del tape\n",
        "\n",
        "    return loss, bc_mse, pde_mse, gradient_loss, gradient_bc_mse, gradient_pde_mse\n",
        "# compute the gradient for the indivdual loss\n",
        "# cosine similarity between the between the different gradients - see if the gredients are heading  in the same direction\n",
        "# gradient surgery paper??"
      ],
      "metadata": {
        "id": "oK2Cb3Xh9B7E"
      },
      "id": "oK2Cb3Xh9B7E",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "346624ca",
      "metadata": {
        "id": "346624ca"
      },
      "source": [
        "# 5. Train the model"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# optimizer\n",
        "optimizer = tf.keras.optimizers.Adam(lr=0.001)\n",
        "# list to collect all of the gradients\n",
        "grad_bc_ls = []\n",
        "grad_pde_ls = []"
      ],
      "metadata": {
        "id": "h0-EmPGaRqGm"
      },
      "id": "h0-EmPGaRqGm",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from time import time\n",
        "\n",
        "# Define one training step as a TensorFlow function to increase speed of training\n",
        "@tf.function\n",
        "def train_step():\n",
        "    # Compute current loss and gradient w.r.t. parameters\n",
        "    loss, bc_mse, pde_mse, grad_theta, grad_bc, grad_pde = get_grad(model, batch_X)\n",
        "\n",
        "    # Perform gradient descent step\n",
        "    optimizer.apply_gradients(zip(grad_theta, model.trainable_variables))\n",
        "    return loss, bc_mse, pde_mse, grad_bc, grad_pde\n",
        "\n",
        "# Number of training epochs\n",
        "N = 5000\n",
        "hist = []\n",
        "\n",
        "# Start timer\n",
        "t0 = time()\n",
        "\n",
        "for i in range(N+1):\n",
        "    loss, bc_mse, pde_mse, grad_bc, grad_pde = train_step()\n",
        "    \n",
        "    # Append current loss to hist\n",
        "    hist.append(loss.numpy())\n",
        "    grad_bc_ls.append(tf.concat([tf.reshape(tensor, [-1,1]) for tensor in grad_bc], 0))\n",
        "    grad_pde_ls.append(tf.concat([tf.reshape(tensor, [-1,1]) for tensor in grad_pde], 0))\n",
        "      \n",
        "    # Output current loss after 50 iterates\n",
        "    if i%50 == 0:\n",
        "        print('It {:05d}: loss = {:10.8e} bc_loss = {:10.8e} pde_loss = {:10.8e}'.format(i,loss, bc_mse, pde_mse))\n",
        "        \n",
        "# Print computation time\n",
        "print('\\nComputation time: {} seconds'.format(time()-t0))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "spxyeXiL9oil",
        "outputId": "db538d17-695c-486f-d6ea-0dbcbe051bab"
      },
      "id": "spxyeXiL9oil",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:Calling GradientTape.gradient on a persistent tape inside its context is significantly less efficient than calling it outside the context (it causes the gradient ops to be recorded on the tape, leading to increased CPU and memory usage). Only call GradientTape.gradient inside the context if you actually want to trace the gradient in order to compute higher order derivatives.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "It 00000: loss = 3.93157601e-01 bc_loss = 0.00000000e+00 pde_loss = 3.93157601e-01\n",
            "It 00050: loss = 8.62991204e-04 bc_loss = 1.00976686e-05 pde_loss = 8.52893514e-04\n",
            "It 00100: loss = 3.08193230e-05 bc_loss = 5.77459893e-08 pde_loss = 3.07615774e-05\n",
            "It 00150: loss = 2.48099241e-05 bc_loss = 6.62995214e-10 pde_loss = 2.48092620e-05\n",
            "It 00200: loss = 2.11854076e-05 bc_loss = 1.43322659e-10 pde_loss = 2.11852639e-05\n",
            "It 00250: loss = 1.81142623e-05 bc_loss = 8.41426581e-11 pde_loss = 1.81141786e-05\n",
            "It 00300: loss = 1.54916797e-05 bc_loss = 2.73332225e-11 pde_loss = 1.54916524e-05\n",
            "It 00350: loss = 1.32931573e-05 bc_loss = 3.60964188e-12 pde_loss = 1.32931536e-05\n",
            "It 00400: loss = 1.14815666e-05 bc_loss = 1.56390569e-12 pde_loss = 1.14815648e-05\n",
            "It 00450: loss = 1.00028437e-05 bc_loss = 1.58296345e-11 pde_loss = 1.00028283e-05\n",
            "It 00500: loss = 8.79696290e-06 bc_loss = 3.90089974e-11 pde_loss = 8.79692379e-06\n",
            "It 00550: loss = 7.80819573e-06 bc_loss = 6.61406277e-11 pde_loss = 7.80812934e-06\n",
            "It 00600: loss = 6.98846679e-06 bc_loss = 9.33321129e-11 pde_loss = 6.98837357e-06\n",
            "It 00650: loss = 6.29967235e-06 bc_loss = 1.14357523e-10 pde_loss = 6.29955821e-06\n",
            "It 00700: loss = 5.71224609e-06 bc_loss = 1.30220695e-10 pde_loss = 5.71211604e-06\n",
            "It 00750: loss = 5.20467620e-06 bc_loss = 1.41987463e-10 pde_loss = 5.20453432e-06\n",
            "It 00800: loss = 4.76118294e-06 bc_loss = 1.49394844e-10 pde_loss = 4.76103332e-06\n",
            "It 00850: loss = 4.37032031e-06 bc_loss = 1.53133436e-10 pde_loss = 4.37016706e-06\n",
            "It 00900: loss = 4.02394608e-06 bc_loss = 1.51470544e-10 pde_loss = 4.02379465e-06\n",
            "It 00950: loss = 3.71583360e-06 bc_loss = 1.46510942e-10 pde_loss = 3.71568717e-06\n",
            "It 01000: loss = 3.44114210e-06 bc_loss = 1.41261572e-10 pde_loss = 3.44100090e-06\n",
            "It 01050: loss = 3.19609126e-06 bc_loss = 1.32705777e-10 pde_loss = 3.19595847e-06\n",
            "It 01100: loss = 2.97751967e-06 bc_loss = 1.25083402e-10 pde_loss = 2.97739462e-06\n",
            "It 01150: loss = 2.78253697e-06 bc_loss = 1.16332159e-10 pde_loss = 2.78242055e-06\n",
            "It 01200: loss = 2.60874367e-06 bc_loss = 1.08154707e-10 pde_loss = 2.60863544e-06\n",
            "It 01250: loss = 2.45382557e-06 bc_loss = 9.84237691e-11 pde_loss = 2.45372712e-06\n",
            "It 01300: loss = 2.31576996e-06 bc_loss = 9.07313044e-11 pde_loss = 2.31567924e-06\n",
            "It 01350: loss = 2.19266622e-06 bc_loss = 8.34576852e-11 pde_loss = 2.19258277e-06\n",
            "It 01400: loss = 2.08279675e-06 bc_loss = 7.58531640e-11 pde_loss = 2.08272081e-06\n",
            "It 01450: loss = 1.98450948e-06 bc_loss = 6.75090123e-11 pde_loss = 1.98444195e-06\n",
            "It 01500: loss = 1.89649677e-06 bc_loss = 6.17963528e-11 pde_loss = 1.89643492e-06\n",
            "It 01550: loss = 1.81739335e-06 bc_loss = 5.62817744e-11 pde_loss = 1.81733708e-06\n",
            "It 01600: loss = 1.74603792e-06 bc_loss = 5.06837836e-11 pde_loss = 1.74598722e-06\n",
            "It 01650: loss = 1.68137501e-06 bc_loss = 4.61673651e-11 pde_loss = 1.68132885e-06\n",
            "It 01700: loss = 1.62250217e-06 bc_loss = 4.15938776e-11 pde_loss = 1.62246056e-06\n",
            "It 01750: loss = 1.56857027e-06 bc_loss = 3.74794674e-11 pde_loss = 1.56853275e-06\n",
            "It 01800: loss = 1.51881159e-06 bc_loss = 3.53004015e-11 pde_loss = 1.51877623e-06\n",
            "It 01850: loss = 1.47265746e-06 bc_loss = 3.18680984e-11 pde_loss = 1.47262563e-06\n",
            "It 01900: loss = 1.42946385e-06 bc_loss = 2.80027373e-11 pde_loss = 1.42943588e-06\n",
            "It 01950: loss = 1.38875839e-06 bc_loss = 2.67568797e-11 pde_loss = 1.38873168e-06\n",
            "It 02000: loss = 1.35014784e-06 bc_loss = 2.43104217e-11 pde_loss = 1.35012351e-06\n",
            "It 02050: loss = 1.31332433e-06 bc_loss = 2.13003139e-11 pde_loss = 1.31330307e-06\n",
            "It 02100: loss = 1.27791623e-06 bc_loss = 1.97890714e-11 pde_loss = 1.27789644e-06\n",
            "It 02150: loss = 1.24363669e-06 bc_loss = 1.79564870e-11 pde_loss = 1.24361873e-06\n",
            "It 02200: loss = 1.21036874e-06 bc_loss = 1.70758772e-11 pde_loss = 1.21035168e-06\n",
            "It 02250: loss = 1.17787317e-06 bc_loss = 1.58162528e-11 pde_loss = 1.17785737e-06\n",
            "It 02300: loss = 1.14599391e-06 bc_loss = 1.42490568e-11 pde_loss = 1.14597970e-06\n",
            "It 02350: loss = 1.11462850e-06 bc_loss = 1.40122766e-11 pde_loss = 1.11461452e-06\n",
            "It 02400: loss = 1.08367703e-06 bc_loss = 1.22044415e-11 pde_loss = 1.08366487e-06\n",
            "It 02450: loss = 1.05306833e-06 bc_loss = 1.05062617e-11 pde_loss = 1.05305787e-06\n",
            "It 02500: loss = 1.02275419e-06 bc_loss = 1.02049280e-11 pde_loss = 1.02274396e-06\n",
            "It 02550: loss = 9.92667879e-07 bc_loss = 9.68352828e-12 pde_loss = 9.92658215e-07\n",
            "It 02600: loss = 9.62810759e-07 bc_loss = 1.24205793e-11 pde_loss = 9.62798367e-07\n",
            "It 02650: loss = 9.38000028e-07 bc_loss = 3.04857628e-09 pde_loss = 9.34951458e-07\n",
            "It 02700: loss = 1.00119416e-06 bc_loss = 5.36843316e-08 pde_loss = 9.47509818e-07\n",
            "It 02750: loss = 8.77968262e-07 bc_loss = 9.11305309e-10 pde_loss = 8.77056948e-07\n",
            "It 02800: loss = 8.47666911e-07 bc_loss = 1.48474887e-13 pde_loss = 8.47666740e-07\n",
            "It 02850: loss = 8.19478259e-07 bc_loss = 1.91532310e-13 pde_loss = 8.19478089e-07\n",
            "It 02900: loss = 2.32466846e-06 bc_loss = 8.75039404e-07 pde_loss = 1.44962894e-06\n",
            "It 02950: loss = 7.65372363e-07 bc_loss = 1.04354234e-11 pde_loss = 7.65361904e-07\n",
            "It 03000: loss = 7.38567053e-07 bc_loss = 5.13401925e-11 pde_loss = 7.38515723e-07\n",
            "It 03050: loss = 7.11717291e-07 bc_loss = 2.19124111e-12 pde_loss = 7.11715074e-07\n",
            "It 03100: loss = 1.26469570e-06 bc_loss = 3.25550928e-07 pde_loss = 9.39144797e-07\n",
            "It 03150: loss = 6.86694023e-07 bc_loss = 1.49989532e-08 pde_loss = 6.71695091e-07\n",
            "It 03200: loss = 6.37309029e-07 bc_loss = 7.47866158e-10 pde_loss = 6.36561140e-07\n",
            "It 03250: loss = 6.10851259e-07 bc_loss = 1.29758712e-11 pde_loss = 6.10838299e-07\n",
            "It 03300: loss = 5.86345209e-07 bc_loss = 7.71325642e-11 pde_loss = 5.86268072e-07\n",
            "It 03350: loss = 2.39122937e-06 bc_loss = 1.03673187e-06 pde_loss = 1.35449761e-06\n",
            "It 03400: loss = 5.52015990e-07 bc_loss = 5.95444094e-09 pde_loss = 5.46061528e-07\n",
            "It 03450: loss = 5.18161812e-07 bc_loss = 1.09722093e-10 pde_loss = 5.18052104e-07\n",
            "It 03500: loss = 4.95534096e-07 bc_loss = 6.29777090e-12 pde_loss = 4.95527786e-07\n",
            "It 03550: loss = 5.68450446e-07 bc_loss = 5.31845608e-08 pde_loss = 5.15265867e-07\n",
            "It 03600: loss = 7.74614023e-07 bc_loss = 1.81277841e-07 pde_loss = 5.93336154e-07\n",
            "It 03650: loss = 4.34041510e-07 bc_loss = 1.64765063e-10 pde_loss = 4.33876750e-07\n",
            "It 03700: loss = 4.13562560e-07 bc_loss = 2.38451429e-11 pde_loss = 4.13538714e-07\n",
            "It 03750: loss = 3.93732989e-07 bc_loss = 1.06207361e-11 pde_loss = 3.93722360e-07\n",
            "It 03800: loss = 3.27484940e-06 bc_loss = 1.64533685e-06 pde_loss = 1.62951267e-06\n",
            "It 03850: loss = 3.71230186e-07 bc_loss = 7.01787117e-09 pde_loss = 3.64212326e-07\n",
            "It 03900: loss = 3.41722284e-07 bc_loss = 2.44168796e-10 pde_loss = 3.41478113e-07\n",
            "It 03950: loss = 3.23938366e-07 bc_loss = 1.00560463e-11 pde_loss = 3.23928305e-07\n",
            "It 04000: loss = 3.06969270e-07 bc_loss = 7.29056052e-12 pde_loss = 3.06961965e-07\n",
            "It 04050: loss = 3.78731283e-06 bc_loss = 1.95720440e-06 pde_loss = 1.83010832e-06\n",
            "It 04100: loss = 2.81546363e-07 bc_loss = 1.75498771e-09 pde_loss = 2.79791379e-07\n",
            "It 04150: loss = 2.64093558e-07 bc_loss = 3.38246875e-10 pde_loss = 2.63755311e-07\n",
            "It 04200: loss = 2.48968433e-07 bc_loss = 7.05248490e-12 pde_loss = 2.48961385e-07\n",
            "It 04250: loss = 2.34829898e-07 bc_loss = 5.46644327e-12 pde_loss = 2.34824427e-07\n",
            "It 04300: loss = 4.79215998e-07 bc_loss = 1.47929242e-07 pde_loss = 3.31286770e-07\n",
            "It 04350: loss = 6.19060586e-07 bc_loss = 2.28094891e-07 pde_loss = 3.90965710e-07\n",
            "It 04400: loss = 1.99820420e-07 bc_loss = 1.17259926e-11 pde_loss = 1.99808696e-07\n",
            "It 04450: loss = 1.87841479e-07 bc_loss = 1.97398535e-13 pde_loss = 1.87841280e-07\n",
            "It 04500: loss = 1.76379402e-07 bc_loss = 1.70562848e-12 pde_loss = 1.76377696e-07\n",
            "It 04550: loss = 1.65639420e-07 bc_loss = 1.72332357e-10 pde_loss = 1.65467085e-07\n",
            "It 04600: loss = 1.90591481e-06 bc_loss = 9.69156872e-07 pde_loss = 9.36757885e-07\n",
            "It 04650: loss = 1.69186265e-07 bc_loss = 1.08954472e-08 pde_loss = 1.58290817e-07\n",
            "It 04700: loss = 1.39670064e-07 bc_loss = 3.76693295e-11 pde_loss = 1.39632391e-07\n",
            "It 04750: loss = 1.30532271e-07 bc_loss = 5.14896827e-13 pde_loss = 1.30531760e-07\n",
            "It 04800: loss = 1.21967261e-07 bc_loss = 7.22751557e-13 pde_loss = 1.21966536e-07\n",
            "It 04850: loss = 1.02289159e-05 bc_loss = 5.62197511e-06 pde_loss = 4.60694127e-06\n",
            "It 04900: loss = 2.23301484e-07 bc_loss = 6.25259986e-08 pde_loss = 1.60775485e-07\n",
            "It 04950: loss = 1.03195028e-07 bc_loss = 4.90331442e-10 pde_loss = 1.02704696e-07\n",
            "It 05000: loss = 9.54720818e-08 bc_loss = 5.96895146e-12 pde_loss = 9.54661132e-08\n",
            "\n",
            "Computation time: 34.85872197151184 seconds\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fig = plt.figure(figsize=(9,6))\n",
        "plt.title(\"Loss against Epoch\")\n",
        "ax = fig.add_subplot(111)\n",
        "ax.semilogy(range(len(hist)), hist,'k-')\n",
        "ax.set_xlabel('$n_{epoch}$')\n",
        "ax.set_ylabel('Loss');"
      ],
      "metadata": {
        "id": "5y0heftaZtVi",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 406
        },
        "outputId": "dec376f2-917e-4c85-c85e-a582ca2f024a"
      },
      "id": "5y0heftaZtVi",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 648x432 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGFCAYAAADn6xkJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXxU1f3/8dcnCRAQRHZlC8gSwQXEgIh1KUjBhVItValV+3XBjVqrLUX0Z9GWotWKG4pUKC4tCq6siuKCgiKgIrui7IKshiWscn5/zEw6hElIMvfOnZm8n4/HPMjcmfncMxeYeeecc8815xwiIiIiqSgj6AaIiIiIlJeCjIiIiKQsBRkRERFJWQoyIiIikrIUZERERCRlKciIiIhIylKQEZGUYWY7zez4oNvhNTNrZmbOzLKCbotIqlGQEUlTZrbSzM4Luh1ecs5Vd859G08NMxtjZn87wnOcme0KB6fIbUA8+xURfyj9i4jE1s45tzzoRohIydQjI1LBmFkVM3vEzL4L3x4xsyrhx+qa2SQz+8HMtprZh2aWEX7sz2a2zsx2mNkyM+tWTP0LzexzM9tuZmvMbHCRx68ys1VmtsXM/l90z5GZdTKzj8P7X29mT5hZ5ajXOjNrGf55jJkNN7PJ4TbNNrMW4cfMzIaZ2cZwOxaY2Ulm1g+4AhgQ7mWZWI7jN9jMXjazl8L7/czM2kU93sbM3g+/h0Vm9vOox6qa2T/D7z/fzD4ys6pR5a8ws9VmttnM7ipr20QqIgUZkYrnLqAz0B5oB3QC7g4/dgewFqgHNAAGAc7McoH+QEfnXA2gB7CymPq7gKuAY4ALgZvM7BcAZtYWeJJQmDgOqAk0inrtj8AfgLrAGUA34OYS3svlwL1ALWA5MCS8/WfA2UDr8D4uBbY450YC/wH+ER6m6lVC7ZL0BsYDtYH/Aq+bWSUzqwRMBKYB9YHfAf8JHz+Ah4DTgC7h1w4ADkbV/QmQG37f95hZm3K2T6TCUJARqXiuAO5zzm10zm0iFASuDD+2n1DAyHHO7XfOfehCF2T7EagCtDWzSs65lc65b2IVd86975xb4Jw76Jz7EhgLnBN+uA8w0Tn3kXNuH3AP4KJeO88594lz7oBzbiXwdNRrY3nNOfepc+4AoYDSPup91ABOAMw5t8Q5t75MRwk+C/eqRG49oh6b55x72Tm3H3gYyCYUDjsD1YH7nXP7nHPvApOAvuGerWuA3zvn1jnnfnTOzXLO7Y2qe69zbrdzbj4wn1DQFJESKMiIVDwNgVVR91eFtwE8SKhnY5qZfWtmAwHCc0VuAwYDG83sRTNrSAxmdrqZvWdmm8wsH7iRUA9LZN9rIs91zhUAW6Je2zo8tLXBzLYDf496bSwbon4uIBQiCAeIJ4Dh4faONLOjS6gTSwfn3DFRt7eiHot+DwcJ9WI1jLy/8LaIVYR6neoSCjwxA2BJ70dEiqcgI1LxfAfkRN1vGt6Gc26Hc+4O59zxwM+B2yNzYZxz/3XO/ST8Wgc8UEz9/wITgCbOuZrACMDCj60HGkeeGJ4fUifqtU8BS4FWzrmjCQ1tGeXgnHvMOXca0JbQENOfIg+Vp14RTSI/hHtaGhM6ht8BTSLzisKaAuuAzcAeoIUH+xeRMAUZkfRWycyyo25ZhIZ67jazemZWl9DwzgsAZnaRmbU0MwPyCQ0pHTSzXDPrGp4UvAfYzaFzO6LVALY65/aYWSfg11GPvQz0MrMu4Um8gzk0qNQAtgM7zewE4KbyvGkz6xjuGapEaM7Onqj2fg/EuxbNaWZ2Sfh43gbsBT4BZhPqSRkQnjNzLtALeDHcSzMaeNjMGppZppmdET6mIlJOCjIi6W0KodARuQ0G/gbMBb4EFgCfhbcBtALeAXYCHwNPOufeIzQ/5n5CvQobCE1kvbOYfd4M3GdmOwiFpHGRB5xziwhNgH2RUO/MTmAjoSAA8EdCwWcH8C/gpXK+76PDr99GaGhnC6FhM4BRhOb6/GBmr5dQY74duo7MI1GPvQFcFq5/JXBJeE7RPkLB5XxCx+pJ4Crn3NKo97cAmANsJdSrpc9hkThYaB6fiEjimVl14AdCQ0krgm5PaYRPJ2/pnPtN0G0REf0mICIJZma9zKyamR1F6HTkBRR/KreISIkUZEQk0Xrzv4mxrYDLnbqGRaScNLQkIiIiKUs9MiIiIpKy0vKikXXr1nXNmjULuhkiIiLigXnz5m12ztWL9VhaBRkz6wX0atmyJXPnzg26OSIiIuIBM1tV3GNpNbTknJvonOtXs2bNoJsiIiIiCZBWQUZEREQqlrQKMuH1KUbm5+cH3RQRERFJgLQKMhpaEhERqVjSKsiIiIhIxZJWQUZDSyIiIhVLWgUZDS2JiIhULGkVZERERKRiSasgo6ElERGRiiWtgoyGlkRERCqWtAoyIiIiUrEoyIiIiEjKUpAppQMHDnDxxRczbty4oJsiIiIiYWkVZPyc7JuRkcHrr7/OkiVLPK8tIiIi5ZNWQcbPyb4ZGRmYGQcOHPC8toiIiJRPWgUZv1WqVElBRkREJIkoyJRBVlaWgoyIiEgSUZApAwUZERGR5JJWQcbvlX0VZERERJJLWgUZv1f2zcrKYv/+/b7UFhERkbJLqyDjN032FRERSS4KMmWgoSUREZHkoiBTBgoyIiIiyUVBpgwUZERERJKLgkwZaLKviIhIclGQKQP1yIiIiCSXtAoyfq8jo7OWREREkktaBZlErCOjICMiIpI80irI+E1BRkREJLkoyJSBgoyIiEhyUZApA521JCIiklwUZMpAPTIiIiLJRUGmDHTWkoiISHJRkCkD9ciIiIgkFwWZMlCQERERSS4KMmWgICMiIpJcFGTKQGctiYiIJJekDzJmdryZjTKzl4Nuiyb7ioiIJBdfg4yZjTazjWa2sMj2nma2zMyWm9nAkmo45751zl3rZztLS0NLIiIiySXL5/pjgCeA5yIbzCwTGA50B9YCc8xsApAJDC3y+muccxt9bmOpKciIiIgkF1+DjHNuhpk1K7K5E7DcOfctgJm9CPR2zg0FLirvvsysH9APoGnTpuUtUyIFGRERkeQSxByZRsCaqPtrw9tiMrM6ZjYCONXM7izuec65kc65POdcXr169bxrbRRN9hUREUkufg8txc05twW4sTTPNbNeQK+WLVv60pasrCx+/PFHX2qLiIhI2QXRI7MOaBJ1v3F4W9yccxOdc/1q1qzpRbnDaGhJREQkuQQRZOYArcysuZlVBi4HJnhR2Mx6mdnI/Px8L8odJhJknHO+1BcREZGy8fv067HAx0Cuma01s2udcweA/sBbwBJgnHNukRf7S0SPDMDBgwd9qS8iIiJl4/dZS32L2T4FmOLnvv0QCTIHDhwgMzMz4NaIiIhI0q/sWxaJGFoCNE9GREQkSaRVkEnU0JKCjIiISHJIqyDjNwUZERGR5JJWQUZDSyIiIhVLWgUZDS2JiIhULGkVZPymICMiIpJc0irIaGhJRESkYkmrIKOhJRERkYolrYKM3xRkREREkouCTBkoyIiIiCQXBZkyUJARERFJLmkVZDTZV0REpGJJqyCjyb4iIiIVS1oFGb8pyIiIiCQXBZkyUJARERFJLgoyZZCZmQnAAw88EHBLREREBNIsyCRqsu/bb7/tS30REREpm7QKMn5P9q1UqZIvdUVERKR80irI+K1+/fpBN0FERESiKMiUQaNGjbjssssK58qIiIhIsBRkyig3N5cff/wR51zQTREREanwFGTKqHLlygDs378/4JaIiIiIgkwZRYLMvn37Am6JiIiIKMiUUZUqVQAFGRERkWSQVkHG73VkQD0yIiIiySStgozf68jA/4LM3r17fduHiIiIlE5aBZlEUI+MiIhI8lCQKSPNkREREUkeCjJlpB4ZERGR5KEgU0aaIyMiIpI8FGTKKHJ5gh9//DHgloiIiIiCTBkpyIiIiCQPBZkyUpARERFJHgoyZaQgIyIikjyygm5AaZjZL4ALgaOBUc65aUG1RUFGREQkefjeI2Nmo81so5ktLLK9p5ktM7PlZjawpBrOudedc9cDNwKX+dneI1GQERERSR6J6JEZAzwBPBfZYGaZwHCgO7AWmGNmE4BMYGiR11/jnNsY/vnu8OsCoyAjIiKSPHwPMs65GWbWrMjmTsBy59y3AGb2ItDbOTcUuKhoDTMz4H5gqnPuM39bXDIFGRERkeQR1GTfRsCaqPtrw9uK8zvgPKCPmd0Y6wlm1s/M5prZ3E2bNnnX0iIUZERERJJHSkz2dc49Bjx2hOeMBEYC5OXlOb/aoiAjIiKSPILqkVkHNIm63zi8LS5m1svMRubn58dbqlgKMiIiIskjqCAzB2hlZs3NrDJwOTAh3qLOuYnOuX41a9aMu4HFUZARERFJHok4/Xos8DGQa2Zrzexa59wBoD/wFrAEGOecW+TBvtQjIyIiUoEk4qylvsVsnwJM8XhfE4GJeXl513tZN5qCjIiISPJIq0sUqEdGRESkYkmrIKM5MiIiIhVLWgWZRFCQERERSR5pFWQ0tCQiIlKxpFWQScTQUlZWaH60goyIiEjw0irIJIJ6ZERERJKHgkwZKciIiIgkj7QKMpojIyIiUrGkVZDR6dciIiIVS1oFmUTIyAgdMgUZERGR4CnIlJGZkZGRwdq1azl48GDQzREREanQ0irIJGKODMDBgwcZNWoUQ4YM8XU/IiIiUrK0CjKJmCMT7aWXXkrIfkRERCS2tAoyibZ///6gmyAiIlKhKcjEQUFGREQkWAoycdi3b1/QTRAREanQFGTi4JwLugkiIiIVWloFmUSdtRS1v4TsR0RERGJLqyCT6LOWFGRERESClVZBJtEUZERERIKlIBMHBRkREZFgKcjEQUFGREQkWAoycYhcQFJERESCoW/iOKhHRkREJFgKMnHQ1a9FRESClVZBJtHryOgSBSIiIsFKqyCT6HVkFGRERESClVZBJtF0rSUREZFgKcjEQUFGREQkWAoycdDQkoiISLAUZOKwf/9+XQFbREQkQAoy5bBixQquvvpqQKdgi4iIBElBphyaNWtGbm4uoOElERGRICnIlFOlSpUABRkREZEgKciUk4KMiIhI8JI+yJhZGzMbYWYvm9lNQbcnQkFGREQkeL4GGTMbbWYbzWxhke09zWyZmS03s4El1XDOLXHO3QhcCpzpZ3vLQkFGREQkeH73yIwBekZvMLNMYDhwPtAW6Gtmbc3sZDObVORWP/yanwOTgSk+t7fUFGRERESC52uQcc7NALYW2dwJWO6c+9Y5tw94EejtnFvgnLuoyG1juM4E59z5wBXF7cvM+pnZXDObu2nTJr/eUqFIkJk2bZrv+xIREZHYgpgj0whYE3V/bXhbTGZ2rpk9ZmZPU0KPjHNupHMuzzmXV69ePe9aW4xIkLnxxht935eIiIjElhV0A47EOfc+8H5pnmtmvYBeLVu29LNJwP+CDIQWxcvISPp50yIiImkniG/fdUCTqPuNw9vi5pyb6JzrV7NmTS/KlSg6yOzevdv3/YmIiMjhgggyc4BWZtbczCoDlwMTAmhHXKKDzK5duwJsiYiISMXl9+nXY4GPgVwzW2tm1zrnDgD9gbeAJcA459wij/bXy8xG5ufne1GuRNFBZufOnb7vT0RERA7n6xwZ51zfYrZPwYdTqZ1zE4GJeXl513tdu6joILN3716/dyciIiIxpNUM1aB6ZLSWjIiISDDSKsgkcrLvwYMHC38+cOCA7/sTERGRw6VVkEmkGjVqFP6sHhkREZFgpFWQSeTQ0sknn8ydd94JKMiIiIgEJa2CTCKHlgB+9rOfAQoyIiIiQUmrIJNounCkiIhIsBRk4qAgIyIiEqy0CjKJnCMDCjIiIiJBS6sgk+g5MgoyIiIiwUqrIJNoCjIiIiLBUpCJg4KMiIhIsNIqyGiOjIiISMWSVkFGc2REREQqlrQKMokWCTK61pKIiEgwFGTikJWVBSjIiIiIBEVBJg6RIKOhJRERkWAoyMRBQ0siIiLBKlWQMbOjzCwj/HNrM/u5mVXyt2lll+izljS0JCIiEqzS9sjMALLNrBEwDbgSGONXo8or0WctZWRkYGYKMiIiIgEpbZAx51wBcAnwpHPuV8CJ/jUrdWRlZWmOjIiISEBKHWTM7AzgCmByeFumP01KLZUqVVKPjIiISEBKG2RuA+4EXnPOLTKz44H3/GtW6sjKylKQERERCUhWaZ7knPsA+AAgPOl3s3PuVj8blio0tCQiIhKc0p619F8zO9rMjgIWAovN7E/+Ni01ZGZm8tRTT1FQUBB0U0RERCqc0g4ttXXObQd+AUwFmhM6c6nC27RpEwcPHuTWW9VBJSIikmilDTKVwuvG/AKY4JzbDzj/mlU+iV5HJtrKlSsTvk8REZGKrrRB5mlgJXAUMMPMcoDtfjWqvBK9joyIiIgEq7STfR8DHovatMrMfupPk0RERERKp7STfWua2cNmNjd8+yeh3hkJcy7pRtpERETSXmmHlkYDO4BLw7ftwL/9alQqyszU+oAiIiKJVqqhJaCFc+6XUffvNbMv/GhQqlKQERERSbzS9sjsNrOfRO6Y2ZnAbn+alJoUZERERBKvtD0yNwLPmVnkdKBtwNX+NCk17du3L+gmiIiIVDilPWtpPtDOzI4O399uZrcBX/rZuFSyZ8+eoJsgIiJS4ZR2aAkIBZjwCr8At/vQnpSlICMiIpJ4ZQoyRZhnrTjSjsyOCp/2fVGi9llaU6ZMARRkREREghBPkDniwilmNtrMNprZwiLbe5rZMjNbbmYDS7GvPwPjyttQP51//vlcfvnlCjIiIiIBKHGOjJntIHZgMaBqKeqPAZ4AnouqmQkMB7oDa4E5ZjYByASGFnn9NUA7YDGQXYr9BaJq1aoKMiIiIgEoMcg452rEU9w5N8PMmhXZ3AlY7pz7FsDMXgR6O+eGAocNHZnZuYRWEW5L6DTwKc65gzGe1w/oB9C0adN4ml1m2dnZCjIiIiIBKO3p115qBKyJur8WOL24Jzvn7gIws98Cm2OFmPDzRgIjAfLy8hJ6vYDs7Gx279ayOiIiIokWRJApF+fcmCM9x8x6Ab1atmzpf4OiqEdGREQkGPFM9i2vdUCTqPuNw9vi5pyb6JzrV7NmzSM/2UPZ2dkcOHCAAwcOJHS/IiIiFV0QQWYO0MrMmptZZeByYEIA7fBMdnZoHvLevXsDbomIiEjF4muQMbOxwMdArpmtNbNrnXMHgP7AW8ASYJxzbpFH++tlZiPz8/O9KFdqVauGTuDS8JKIiEhi+TpHxjnXt5jtU4ApPuxvIjAxLy/veq9rlyTSI7Nr1y7q1KmTyF2LiIhUaEEMLfkmqB6ZSJC54IILErpfERGRii6tgkxQk30rVaoEwKJFnoyQiYiISCmlVZAJyrZt24JugoiISIWUVkEmqKGl/fv3J3R/IiIiEmLOJXQR3ITIy8tzc+fOTdj+9uzZQ9WqValevTo7duxI2H5FREQqAjOb55zLi/VYWvXIBCU7O5sBAwaoZ0ZERCTBFGQ8UqNGDfbu3aswIyIikkBpFWSCmiMDoSADsHPnzoTvW0REpKJKqyAT1OnXANWrVwfQHBkREZEESqsgE6RIj4yCjIiI+GXy5MmsX78+6GYkFQUZjyjIiIiInw4cOMBFF13ET3/606CbklTSKsgkwxyZHj16kI6ntIuISMm2bNnCddddR0FBgS/1d+/eDcCyZct8qQ/w7LPPYma+vQc/pFWQCXKOTCTIbN++nY0bNyZ8/yIiUrKlS5fy0EMP+VZ/8ODBjBo1itGjR/tSf8+ePb7UjTZkyBAAVq9e7fu+vJJWQSZIkcm+EPrPIiIiyeXss8/mT3/6U2HPhl8OHjzoS91ELO+RlZUFwI8//uj7vryiIOORSI8MwBdffBFgS0REUtf27dt9Cxq7du0CQnNN/JCREfpK9SvIJCJcRI5NKk2RUJDxSP369XnuuecAuO222/jqq68CbpGISOqpWbMmp556qi+1MzMzAdi3b58v9dMhyEQCjHpkKqgrr7ySCy64AIAXXngh4NaIiKQmvyazRoLM3r17famfDkEmwq9eKz+kVZAJ8qyliMmTJ9O1a1cefvhhPv7448DaISIisfkVNCK9GWbmS/1IkPGrfqx9pYK0CjJBnrUU7ZFHHqF+/fpceOGFvP7664G2RUREQiJBw+8gE+mZ8VoiwkUkJCnIVHAnn3wyb7/9Ns2bN+fiiy+mf//+hZPMREQkGJEA41eQidRNhx4ZDS0JLVq0YNasWdx2220MHz6cU045hQ8//DDoZolIBbV+/XrP17h68cUX6d+/v6c1/ZSoHhm/g4yfNNlXDlGlShWGDRvG+++/j3OOc845h9tvvz2lVkwUkfTQsGFDGjRo4GnNvn37Mnz4cE9r+snvIBOpm8pDS5FjpNOv5RDnnHMOX375JTfddBPDhg3j1FNP1URgEZEES5cg41d9+N97UJCRw1SvXp3hw4fzzjvvsGfPHn7yk58wYMCAhCw5LSIi6TO05GeQUY+MHFG3bt1YsGAB1113HQ8++CAdOnTg008/DbpZIpJkJkyY4PtS+l7xKxh4ze/JvukUZFLl7xTSLMgkwzoypXH00Ufz9NNP8+abb7J9+3bOOOMM7rrrLt8WaRKR1PL555/Tu3dvbr311qCbUiqp8tt7qp9+7ffQVfQ+UuXvFNIsyCTLOjKl1aNHDxYuXMjVV1/N3//+d/Ly8vjss8+CbpaIBCxyQsDixYsDbknpePWl53cvQLrMkSkoKGDTpk2+7ENBRsrsmGOOYfTo0UycOJEtW7bQqVMn/vKXv/h2LRARSX6VK1cG/LsmkNe8CgZ+n5WTqCCTiNOvP//8c1/2oaElKbeLLrqIhQsX0rdvX+677z5OP/105s+fH3SzRKSUxo0bx9atWz2p5fdcC6959dt7ooKMX/uJ1J87d64v9aPb7ffw1bp163yp7wcFmSRSu3Ztnn/+eV577TW+++478vLyuPvuu3Vmk0iSW758OZdddhn9+vXzpF6qBZlU6ZGJ1L/77rt9qR/5e3v66ad9qZ/IIHP99df7Ut8PCjJJ6Be/+AWLFy+mb9++DBkyhFNPPZWPPvoo6GaJSDEiv2wsXbrUk3qpFmS86pFJ1LL4b775pi91ExXEwL9/G6k0NyZCQSZJ1alTh+eee46pU6dSUFDAWWedxS233ML27duDbpqIFMPrIZZUCTJ+9Mh89dVXntRMJL/nlSQiyKTS3JgIBZkk17NnTxYtWsStt97KU089xYknnsjkyZODbpZISlu3bh2vvPKKZ/UiXzCp1jPhFT/myIwYMcKTmomUyCDj94TlVKIgkwKqV6/Oo48+ysyZMzn66KO56KKLuOKKK9iwYUPQTRNJSZdccgl9+vTBqzWnvA4eFbVHJvo4+rlWil8SGWT8GsZSkBFfnXHGGXz22Wf85S9/Yfz48eTm5vLoo4+m3G9vIkFbtWoVADt27PCkXuT/oHpk4hP95ZyZmelJzURKZJDxay6L5sj4wMzONbMPzWyEmZ0bdHuCVqVKFQYPHsyCBQvo3Lkzt912Gx06dGDGjBlBN00kZUQ+rL0KDJE6Xn35el3Pb37MkUnFHplETvb1i3pkijCz0Wa20cwWFtne08yWmdlyMxt4hDIO2AlkA2v9amuqyc3N5c033+SVV14hPz+fc845h9/85jesX78+6KaJpAwFGW9U1J6oohIZZPzqOVGQOdwYoGf0BjPLBIYD5wNtgb5m1tbMTjazSUVu9YEPnXPnA38G7vW5vSnFzLjkkktYsmQJd911F+PHj6d169b89a9/ZdeuXUE3TyTpefXFk+xBJvpLz4+5dbfffrsndRIxB8RPiRxa8ouGlopwzs0Aii512QlY7pz71jm3D3gR6O2cW+Ccu6jIbaNzLvIvYxtQpbh9mVk/M5trZnP9ugZFsqpWrRp/+9vfWLhwId27d+eee+6hdevWjBo1KiU/DEQSxeseGa+GQ7wOMtFfsFdddZUnNaM999xzntRRkClZIo5JKh73IAYhGwFrou6vDW+LycwuMbOngeeBJ4p7nnNupHMuzzmXV69ePc8am0patWrFq6++ykcffUROTg7XXXcd7dq1Y8qUKSmZskX85lWQ2b9/P5C8PTLR73Pz5s2e1PRDqg8tJTLI+HUJGw0t+cA596pz7gbn3GXOufdLeq6Z9TKzkV6dUpmqzjzzTGbOnMnLL7/M3r17ufDCCzn77LOZPn26Ao0I3l9zx6/g4VUPT6r0dCRz20ojuv2zZ8/2vH50yPjzn//seX1Izb+DIILMOqBJ1P3G4W1xc85NdM71q1mzphflUpqZ8ctf/pLFixczfPhwVqxYwXnnncc555zDu+++q0AjFZpfC9h5HTymTJniSb3ono5k/r+fiJVrI/w4DtHtf/LJJ32t75fo47JixQrf9+eFIILMHKCVmTU3s8rA5cAELwqrR+ZwlSpV4uabb2b58uU8/vjjfPPNN3Tr1o1zzz1XgUYqLL+CjB9DQV5cliRVfstO5NDSrFmzPK8Z3WOyd+9ez+sn+u9x1KhRCd1fefl9+vVY4GMg18zWmtm1zrkDQH/gLWAJMM45t8iL/alHpnjZ2dn079+fb775hscee4yvv/6abt260alTJ8aPH58yH3QiXvAryHg9eRhg586dntZL5l9eoj+Hvv32W1/3tW3bNs9rpluQ8eM9+MHvs5b6OueOc85Vcs41ds6NCm+f4pxr7Zxr4Zwb4mcb5FDZ2dn87ne/45tvvmHEiBH88MMPXHrppbRu3Zonn3ySgoKCoJso4rvIF47XQWbWrFls3LjRs3rgTRtTZRJt9Bf1a6+9lrB9eSU6yPhRX0EmtqSf7FsWGloqvapVq3LDDTewdOlSXnnlFerVq8ctt9xCTk4OgwcP1nWcJK1FvhD8uEaQF4tSeh1k/P7S9koiA5ffc2T8kOggk8y9d9HSKshoaKnsMjMzueSSS/j444+ZMWMGnTt35t5776Vp06b8+te/ZtasWSnzj1mktPy8WnWlSpU8redFYPBjaCkdehy8Fv13lYpBqahU+exPqyAj5WdmnHXWWX/ZYxIAACAASURBVEycOJGvvvqKW265hSlTpnDmmWdy2mmnMXr0aHbv3h10M0U84WePjBcTflOhRyZVahYnFYNGqgc9v6RVkNHQkjdatWrFsGHDWLt2LSNGjGD//v1ce+21NG7cmDvuuIPFixcH3USRuES+xHr27HmEZ5aOnz0oydoj48cwUKoPLaXDyr7R1CMTAA0teat69erccMMNfPnll7z//vt069aNxx9/nBNPPJEzzzyTf//737qmk6Q0L84Igv+t7AvefNn4OdnXq19E1CNzOL8v6lj0+KTKJG6/pVWQEX+YGeeccw7jxo1j3bp1/POf/2Tr1q1cc801HHfccdxwww3MmTMnZdK7iNeiv1CiQ015RX9hefFbvh8BoeiX6Lp18a9rmupDJ4nukXn77bd93V+qUJCRMqlXrx633347ixcvZubMmfTp04cXXniBTp060a5dO/75z396ctaGSCrxswfFj9Ov/Zh3s3Tp0rhrJrKHYc2aNUd+UhlFB5nJkyd7Xr/oMfc7OKXKL6dpFWQ0RyZxzIwuXbowevRo1q9fz9NPP03VqlX54x//SOPGjenRowcvvPCChp6kQvAzePjRI+P1vBuAffv2xV0zkT0yf/jDHzyvmegeGb/359VVzf2WVkFGc2SCcfTRR9OvXz9mz57N0qVLGTRoEF999RVXXnklDRo04Morr2TatGkp320sUhw/J+eOGTPG03pQsc6ESiS/21/035bfPSap8otoWgUZCV5ubi5//etf+eabb5gxYwa//vWvmThxIj169KBJkybccccdfPrppynTZSlSGtFBwYvhkega999/v6f1IHmHq4rW9HJ4xu/eC0j86deJeE+pQEFGfJGRkcFZZ53FyJEj2bBhA+PHj6djx448/vjjnH766TRv3pwBAwYwb948hRpJedFfMF26dIm7ntdzRfz4AkxEj4yXQSYRvT2JHlry+rMzVYNRWgUZzZFJTtnZ2fTp04c33niD77//nn//+9+0bduWYcOGkZeXR8uWLbnzzjv5/PPPFWokJXkdPPyul6w9Mn6eXpyIicSpHmRSdWgvrYKM5sgkv1q1avHb3/6WKVOm8P333zNq1ChatmzJgw8+SIcOHcjNzeXuu+9m/vz5CjWSMpI9yPjxBehHzaLv28vjkA49MkXre70/BRmRMqpduzbXXHMNb731Fhs2bGDkyJHk5OQwdOhQ2rdvT4sWLbj99tv54IMPtPCT+GblypVx1/A7eMQrET0yXij6vr1Yk6e42n5Qj0wwFGQkKdStW5frr7+et99+u/B07jZt2jB8+HDOPfdcjj32WH7729/y+uuvU1BQEHRzJY288MILcddI9h6ZovVGjBgRd81UO2spEV/SiZ7s69XK1MXVTxUKMpJ06tevT79+/Zg8eTKbN29m/Pjx9OzZkzfeeIOLL76YOnXq0Lt378I1bETi4ceaKslWr+gX1IABA+KuWbSNc+fO9byml9IxyFxzzTWe1tdk3ySgyb7pp0aNGoWrB2/cuJHp06dz/fXX88UXX3DttdfSsGFDTj31VAYNGsSHH36oISgps2QcZvG7nhdfuEVr3HvvvZ7XNLO4a0bEOqZef3EXrTdp0iRf63tNPTJJQJN901ulSpXo2rUrjz32GCtXruTzzz/n73//O0cffTT/+Mc/OPvss6lbty6/+tWvGD16NN99913QTZYKItmDTCKuteSFRA8tffLJJ57uo2jQGD9+vKf1E93jkyqygm6ASHmYGe3bt6d9+/bceeed5Ofn88477zB16lTefPNNXn75ZQBOPvlkunXrRteuXTn77LNRyJWiUqFHJj8/P65/u6kSOhI9tOTlZOJY+0i1s4pSNcikVY+MVFw1a9bkl7/8Jc888wxr1qzhyy+/5IEHHqBBgwaMGDGCn//859SuXZvOnTszaNAg3nnnHU0aFsCfU5FXrFgRV72iX+gffvihp/W8kIhw5GWPSawv6VQ4OyxaEENLY8eO9XWfXlCQkbRjZpx88skMGDCAt99+mx9++IH333+fu+66i6ysLB588EG6d+9OrVq1OPfcc7nnnnt466232L59e9BNlwQo+mWwe/fuuGsW/QJ75513PK0X7xdiqpxhVLTm119/7Vvt4rZ5uY906JGZN2+er/v0goKMpL0qVapwzjnncN999/HRRx+xbds2pk6dyq233sqOHTsYMmQIPXv2pFatWrRv357+/fszduxYVq9eHXTTxQdFhxMefPDBuGt6PZnW6yCTKj0yfg4txaqtIHPk+l4Efb9pjoxUONWrV6dnz5707NkTgB07djB79mxmzpzJRx99xLPPPsvw4cMBaNKkCV26dKFjx4507NiRDh06UL169SCbL3FKxBdwvF9gXi98lipzZBI92VdB5sj1U+GUbAUZqfBq1KjBeeedx3nnnQeEPvS//PLLwmDzySef8NJLLwGhi2G2adOmMNh06tSJU045hcqVKwf5FqQMvJ7gCd4HmYraI5PoIOPl6d3OOd8vIaDTr2NLqyBjZr2AXi1btgy6KZLCsrKy6NChAx06dOB3v/sdABs3bmTOnDmFt8mTJzNmzBgAKleuzIknnki7du0OudWuXTvAdyHFScUgE69Y9fbt2xdXAE+Hs5ZSqX4i9hGrfipc8y6tgoxzbiIwMS8v7/qg2yLppX79+lx44YVceOGFQOg/9+rVq/n000+ZM2cOX3zxBVOmTCkMNwCNGzc+JNi0bduWVq1aUaVKlYDehUBigky8v+kXrbdv3z5P64E/vTzvvvsuXbt2LXdNP7+o/V4sM6gg45zzrGdJQUakAjEzcnJyyMnJ4Ve/+lXh9g0bNjB//vxDbm+++WbhB0RmZiYtWrSgbdu2tGnTpvDPE044gaOOOiqot1OhxAoy8X4ZFP2S/OKLL8pdK1a9yy67jEsvvbTc9WK953i/eGO9funSpUkbZGLVnjVrFhdccIFv9V955RVPape0jxUrVnD88cf7Vj8VKMiIeOjYY4/l2GOPpUePHoXb9uzZw5IlS1iyZAmLFy8u/HnSpEmHfGHl5OTQpk0b2rRpQ8uWLQtvTZs2JStL/1W9Eus38zFjxvB///d/ntUcPXo0o0aN8qxevPw4Y6e44So/anoxBy3W+x0yZAh/+9vf4q5dXH2vxRqy9PLCkeqREZGYsrOzOfXUUzn11FMP2b5//36WL19+WMCZMWPGIYv1ZWVl0bx580PCTeTWrFkzTTQuo1i9Ex999FFcQcbvhdX8qOdHj0y8w3ax2vn3v/+dwYMHx1UX0neOjJdDpbGCUipcu1BBRiQglSpVKuyBueSSSwq3O+fYsGEDy5cvP+z20UcfsWPHjsLnZmRk0LRpU1q0aEHz5s1p1qxZ4Z/NmjXj2GOPJSNDy0VFi/XBv3fv3rhqeh084u3ZKCpRPTLxilVz48aNntT2O2gk4oK1QYSxcePGFZ61mawUZESSjJlx3HHHcdxxx3HWWWcd8phzjs2bNx8Sbr7++mu++eYbJkyYcNiHfuXKlcnJySkMNkWDToMGDSpc0IkVZLyenBsvr4NMoubIxHsc/VzrJV0n+6ZSfb8oyIikEDOjXr161KtXjzPOOOOwxwsKCli1ahUrV6487PbGG2/EDDpNmjQhJyeHpk2bHvZnkyZNyM7OTtTbS4hYX+ovvPACzz//fLlreh084u0hKirWl/hnn31WuCikVzXjFaumV2unpEMIiHUsvFwLR0FGRAJXrVq1wuGqWHbt2nVI0Fm1ahWrV69m1apVvP3223z33XeHTe5r0KBBYbCJFXZq167t6Yep37wOHRCa0F3U2rVrady4cbnqxWrj1q1by702UayAcP7558c1kTPWl14yniYekQ5BJh3egx8UZEQqkKOOOoq2bdvStm3bmI/v27ePdevWHRJwVq9ezerVq1mwYAGTJ08+7NorRx11VMyAE/mzUaNGSXXW1a5duzyvGasHZejQoYWXuiirWIFg7Nix3HLLLeWql6jek7vuuotBgwZ5WtOrL9d0CAHp8B78kDyfLiISuMqVK9O8eXOaN28e8/HIHJ3okBP957x589i0adMhr8nIyKBRo0bFBp2cnJyEXr+quCCzc+fOcrcjVo9MeXupDhw4EHMIIZ4A5scigKm2sq/fc2TSdbJvKkj6IGNmGcBfgaOBuc65ZwNukkiFFT1H57TTTov5nIKCAtasWRMz6Hz88ceMGzfusA/9WrVqlRh06tev79mk5OICwc0338xzzz1X5nrOOU+DjB9DX6lyraUggsz3339PgwYN4q4f1ByZ+++/n3HjxnlSX0EmBjMbDVwEbHTOnRS1vSfwKJAJPOOcu7+EMr2BxsAWYK2PzRURD1SrVo3c3Fxyc3NjPv7jjz+yYcOGmEFnxYoVvP/++2zfvv2Q11SuXLlwjk68k5KL1o54/vnnyxVkigseTzzxBI8//rhn9eKRKhd4jFVzzJgx/Pvf/467dnHH4KuvvvI1yMyaNYsuXbrEXb+4fYwfP96T2sXVTwV+98iMAZ4ACj8dzCwTGA50JxRM5pjZBEKhZmiR118D5AKznHNPm9nLwHSf2ywiPsrMzKRRo0Y0atSo2A/4/Pz8YoevSpqUHB12mjRpcsjPkV6d7777jqysLA4ePHjYb7j79++nUqVKZXo/Xs+5KS7I/PnPf2bAgAHlqlncl3g8q+amSuA6Um0/r1ME8Oqrr/oaZLykIBODc26GmTUrsrkTsNw59y2Amb0I9HbODSXUe3MIM1sLRP7HFHuUzawf0A+gadOmcbddRIJTs2ZNTj75ZE4++eSYj8ealLxq1SrWrFnD4sWLefPNNw8LGJUrV6Zx48Z8++235ObmcuONN/KHP/zhkOdkZ2eX+cP8hx9+KPax8gQFr0+9huLnyOzcubPcZ0L50c5UDjJ+1wcFmeIEMUemEbAm6v5a4PQSnv8q8LiZnQXMKO5JzrmRwEiAvLy85L84hIiUW2kmJW/bto01a9YUnnUV+TknJ4f+/ftzySWXcNNNNx0yJHXw4EGOOeYY7r77bjp06EDr1q1p2LBhifNztm3bBsDPfvYzpk2bdshjf/zjH3nsscfK9N78CAix5vBAqDfJ6yBTnl6tiFSe7BvUHBkvFfce3nvvPX7605/6uu94JP1kX+dcAXBtaZ5rZr2AXi1btvS3USKS1MyM2rVrU7t2bdq1a1fs86pUqcJHH31E3759WbMm9PtVfn4+f/rTnwqfU7VqVVq0aEGrVq1o2bIlxx9/fOHKyDk5OYWv69at22FB5vHHHy9zkInM4bnpppt46qmnDnls165d5bpKenFBJp6zmYoLMkOHDuWee+4pV81U7pFJh9OviwtKU6dOVZApYh3QJOp+4/C2uDnnJgIT8/LyrveinoikvzPPPJPVq1cX3nfOsX79epYsWcJXX31VeBmIpUuXMnny5JhzQ6pUqUL79u1j1n/mmWe47rrrSt2eSJCJ1VMyaNAgHn300VLXiti9eze5ubksW7bskO2TJk3i1ltvLXM9KH6OzLp15f84DyLIRF+gNR5+nOJeVFBDS8UF4WQRRJCZA7Qys+aEAszlwK+9KKweGRGJl5nRsGFDGjZsSLdu3Q557ODBg6xfv/6QSz+sWrWKnj170r17d1q2bMny5csPec3111/Pk08+yS233EL79u3Jzc0tcb2ayJybunXrHvbYjBnFjq6XaM+ePbRp04azzz6bf/3rX4Xbf//735c7yBTXIxNPGPHzzJ/i2tW9e3dPVg9O53Vk/JjY7SW/T78eC5wL1A1P2v2Lc26UmfUH3iJ0ptJo59wiL/anHhkR8VNkcb9GjRpx5plnHvb4119/zfz583nttdcYM2YMq1atAuDzzz8/pFemSZMmtG7dmtzc3MI/c3Nzadq0aeFr+vbte9hk5C+++ALnXJmHQ/bs2UN2dna558PEksjJvi+88IJvQcYrkfoZGRmHDNE89NBDPPjgg57so7ihn+3bt3P00UfHXV+TfWNwzvUtZvsUYIqf+xYRCUK7du1o164dgwcPLtzmnGPZsmUsXryYpUuXFg5b/ec//yE/P7/weVWqVAGgYcOG1K9fnzlz5tCxY8dD6vfu3ZsJEyaUqU3bt2+nevXqDB48mAceeOCQxw4ePFiuxQaLCzIzZ84sc62IVJ7sG6mflZXlWw9GcUFj69atngaZ6667jmeeeSbueomS9JN9y0JDSyKSjMyME044gRNOOOGQ7c45Nm3axLJly1i2bBlfffUVq1at4tprr8XMyMvLO+wsoIkTJ5KXl8cjjzxCXl7eERcCdM6xdetW6tSpE/O5c+fOpVOnTmV+T8UFmaLzcMrCzyGMRAWZQYMGHRJivVRckPH6wpr33HPPIUEm2S8Km1ZBRkNLIpJKzIz69etTv359zjrrrJjPycrKYtu2bdx+++2FK9zOmzePs846i8zMTHJzcznppJM46aSTOPHEE2nTpg0tWrQoXL9mx44d7Nu3jzp16sSsP2HCBE+DTDyKm1SaCnNYIvV79+6d8CDjdf3MzExf9+O1tAoyIiLp6JhjjmH06NGMHj0a5xwbN27kww8/5IsvvmDhwoXMmzeP8ePHF37hZ2Zm0qJFC9q0aVPYm9OqVauYtYcMGcJ9991X5uElr872iVZcOJozZ07ctRM5tOSX4ubIeN0jk0xXqy+N1GrtEWhoSUTSnZnRoEED+vTpQ58+fQq3FxQUsGTJksI5OJE/v/76a1q1anXYGVjRmjRpwurVq8v0m3hJKxqPHTuWvn1jTpEsUXE9MvPmzStzraJKCjLlmUBdXH0/Q0CihpaK/jsYMWLEYWsaJZO0CjIaWhKRiqpatWqcdtpph12V/MCBA2RkZBT2uLz66qssWbKEu+66q/A5ketP3XzzzXTp0qVwVeOSgk1kReNY7r333jIHGedcYHNkvv/+e4499lhP6pd3VeMjcc4lbGXfVOuRKftUdRERSRlZWVmHDBtdfPHFDBo0iL179zJ9+nTuuOMOatWqRW5uLmPGjOE3v/kNbdu2pUaNGnTu3JmbbrqJp59+mk8//bRwOGnXrl1s27at8HTuogv+LVu2rMy9BH6vVVJSkNm5c6dn9WOFgE2bNsVdv6QQU1LvWFmk6hyZtAoyZtbLzEZGn84oIiKHq1y5Ml27duWhhx5i69atLF26lPz8fObPn8+zzz7LDTfcQNWqVRk7diw33ngjp59+OjVq1KBt27b07NkTgA4dOhTWKmrWrFllas+R5ty8/PLLZapX1L59+6hatWrMx7zo6SgpyEQvQlheJa0cfO6558ZdH/53HBRkAuScm+ic61ezZs2gmyIiknKysrI45ZRTuOqqqxg2bBjvvfce27Zt49tvv+XVV1/lrrvuolWrVmzZsoULL7yQV155hUGDBnH//fcfVusnP/kJK1asKPW+j/QL6MSJE8v8fqLt2bOHBg0axAxYXkwEjgSNWEFm9+7dcdf3u0cJUrdHJrUGwkREJKHMrPBK4xdffPFhjw8ZMgSARYsWceKJJx7y2PHHH89ll11Gz5496dSpE7m5ucV+SR5peOT1118v5zsIiaxuHGvIq0+fPixevDiu+iX1yHgZlPyUqnNkUqu1IiKSlNq2bVsYEsaOHcuYMWNYvHgx77zzDi+99BIA1atXJy8vj44dO9KxY0c6depE06ZNMTM2btxYYv3IxTTLq6Qgs2TJkrhqg/9nLSUyyJRnpecgKciIiIin+vbtW3jW0sGDB1m6dClz5swpvD366KOFk3vr1atHx44dSxVUPvzww2IXDjySSJBp165duV5/JOkSZFJtWAnSbI6MJvuKiCSXjIwM2rZty9VXX80TTzzB7Nmz2b59O3PmzGH48OFceOGFrFq1ik8++YQrrriCP/3pT8XWOvvss8s932TPnj1UqVKF6tWrl3kicmmUFGRizSEqKwWZ4qVVj4zWkRERSX5VqlQhLy+PvLy8wm3Ri9K1bNmSadOmUadOHUaOHHnIa6tVq8ZTTz3FL37xizKt/ZKfn0/Tpk2B2NcOWrt2LY0bNy7P2wH+t5hfrDO4vOD3ysSRfRTXozR//nzferPilVY9MiIikpqiw0W/fv14+eWXefrpp3HOMW3atMI1awBuuukmjjvuOFq2bMlVV13FU089xeeff15ir8W2bduoVasWEHsl3O+++y6u9keGrvy6wOKRemS8WN13//79xQYxrewrIiJSTt27d2fLli1AaM7NZ599xvvvv8+sWbOYNm0azz//PABVq1alQ4cOdOrUidNPP51OnTrRrFkzADZv3lzshTMBTj/99LjCQCTI+OVIQSZ6gcLy2rdvX7ErEyeiR6i8FGRERCRlZGRkHDIs5Zxj5cqVzJ49m08//ZTZs2fz1FNPMWzYMCA0mbh169YUFBTQunXrEmvv37+/3JcYOFKQ2bx5M3Xr1i1XbThykPHiytjxvP8gpdXQkib7iohULJF1bi6//HIefvhhZs6cyfbt25k3bx5PPfUUF110Edu2beOEE04oXAen6PWoIm6//fZyt+NIQebxxx8vd204co+IV2vVKMgETCv7iohIpUqV6NChAzfeeCOjR49m0aJFLFmyhAYNGgChCbkzZ84svB/xxBNP0KVLl3JdG2nXrl3FXgIBYO/evWWuGe1IPTJenNWkICMiIpIiunTpwoYNG/jhhx/o06dP4faPP/6Y4447js6dO3PnnXfy1ltvleoSANGTiWN54IEH4mrvkYLKNddcE1f9yD6KCzJr1qyJu75fFGRERKTCqlmzJuPHj8c5x/79+5k2bRqDBg0iMzOThx56iJ49e3LMMcfQuXNnBg4cyNSpU9mxY8dhdY4UZOJ1pCAzffp0T/ZRXJCZNm1a3PX9oiAjIiJCaDG77t27c9999zFz5kx++OEHpk2bxsCBA8nKyuLhhx/mggsuoFatWpx++ukMGDCAKVOmkJ+fz+rVq2nYsCEAW7ZsoXr16ofV//7778vdtniHpkojVYeWdNaSiIhIDEcddRTdu3ene/fuABQUFPDxxx/z/vvv88EHH/DII4/w4IMPFj6/Y8eOANSuXZvs7OzDhqTOO+88FixYUK62FBQUlPNdlJ6CjIiISBqrVq0a3bp1o1u3bkAoXHzyySd8+umnAFx99dWFz61duzabN28+5PULFy5k9erVhSsMl0Vpgkz06sjlkapBRkNLIiIi5VCtWjW6du3KwIEDGThw4CGr4k6bNo0nnnjisMso5OTkMHjw4DKvJFyaIBNZNLC8ooPM8ccfH1etREqrIKN1ZEREJBnk5ORwyy23sH79erZs2XLIxTDvvfdeGjVqRPv27fnzn//MO++8c8SLYZbmzKnSXEG8JEfqkSlNG4KQVkFG68iIiEiyqV27Nv/4xz9wznHw4EHmzZvH0KFDqVWrFsOGDaN79+7UqlWLrl27MmTIEGbPnn3YAncbN26kWrVqJe7n5z//eVztPFKQ+d3vfhdXfb+kVZARERFJZmZGhw4dGDhwIO+99x5bt25lypQp9O/fn23btnH33XfTuXNn6tSpQ+/evXnsscdYtGgRa9euPeLVvhctWhRX2/bs2VO4qF/Rq44DrF+/Pq76ftFkXxERkYBUr16d888/n/PPPx+ATZs28d577zF9+nSmT5/OhAkTCp972WWXccwxx/DZZ58VW2/Dhg1HDDzFKSgo4KijjgIonNAcza8re8dLQUZERCRJ1KtXj0svvZRLL70UgJUrVzJ9+nQWL17MrbfeSk5ODs899xz16tXjggsuOOz1xx13XLmv4r1r164Sh6+2bdtWrrp+U5ARERFJUs2aNePaa689ZNtVV10FwMGDB/ntb3/Lc889d8jjlStXZsWKFTRq1KhM+yooKCgxyMyePbtM9RJFc2RERERSkJnx7LPPUlBQwOTJkwu379+/n5ycHLp27cojjzzC119/fcRaP/744yFDS6lEQUZERCSFVa1alQsuuKDwrKjZs2fzxz/+ke+//54//OEPtG7dmlatWnHrrbcyderUmKd6R9agqVu3buG2f/3rX4c970iniQfByjuWlszy8vLc3Llzg26GiIhIoL799lumTp3K1KlTeffdd9m9ezeVK1fmjDPOoGvXrnTt2pVOnTqxZMkS2rdvz0svvVQ4PwcOn+B78cUX8+qrryb6bWBm85xzeTEfU5ARERFJf7t372bGjBlMnz6dd999l88++wznHNWqVaN69eps3LiRBQsWcNJJJxW+JtaZSkHkhpKCjCb7ioiIVABVq1alR48e9OjRAwidhfTBBx/w7rvv8t5773Huuedy4oknHrHOpk2bqFevnt/NLbWk75Exs7OAKwiFrrbOuS5Heo16ZEREROJ34MAB2rdvf9hie7t37yY7Ozth7SipR8bXyb5mNtrMNprZwiLbe5rZMjNbbmYDS6rhnPvQOXcjMAl41s/2ioiIyP9kZWXxwQcfHLa9atWqPP3000mxtozfZy2NAXpGbzCzTGA4cD7QFuhrZm3N7GQzm1TkVj/qpb8G/utze0VERCRKnTp1cM6xd+9eJk2aRIsWLahSpQo33ngj9evXp2vXrgwbNqxUp3n7wfehJTNrBkxyzp0Uvn8GMNg51yN8/04A59zQEmo0Bf6fc+76Ep7TD+gH0LRp09NWrVrl1VsQERGRKJGLX7766qtMmjSJhQtDAy/ff/899evXP8Kryy6woaViNALWRN1fG95WkmuBf5f0BOfcSOdcnnMuL5kmIYmIiKSbjIwMOnbsyNChQ1mwYAErVqzgv//9ry8h5khS4qwl59xfSvM8M+sF9GrZsqXPLRIREZGIZs2a0axZs0D2HUSPzDqgSdT9xuFtcXPOTXTO9atZs6YX5URERCTJBRFk5gCtzKy5mVUGLgcmHOE1pWJmvcxsZH5+vhflREREJMn5ffr1WOBjINfM1prZtc65A0B/4C1gCTDOObeopDqlNENX/AAABvJJREFUpR4ZERGRisXXOTLOub7FbJ8CTPFz3yIiIpL+0urq1xpaEhERqVjSKshoaElERKRiSasgIyIiIhVLWgUZDS2JiIhULGkVZDS0JCIiUrGkVZARERGRiiWtgoyGlkRERCqWtAoyGloSERGpWMw5F3QbPGdmm4BVPpWvC2z2qbYcSsc6cXSsE0vHO3F0rBPHz2Od45yrF+uBtAwyfjKzuc65vKDbURHoWCeOjnVi6Xgnjo514gR1rNNqaElEREQqFgUZERERSVkKMmU3MugGVCA61omjY51YOt6Jo2OdOIEca82RERERkZSlHhkRERFJWQoyIiIikrIUZErJzHqa2TIzW25mA4NuT6oys9FmttHMFkZtq21mb5vZ1+E/a4W3m5k9Fj7mX5pZh6jXXB1+/tdmdnUQ7yXZmVkTM3vPzBab2SIz+314u463x8ws28w+NbP54WN9b3h7czObHT6mL5lZ5fD2KuH7y8OPN4uqdWd4+zIz6xHMO0p+ZpZpZp+b2aTwfR1rn5jZSjNbYGZfmNnc8Lbk+Rxxzul2hBuQCXwDHA9UBuYDbYNuVyregLOBDsDCqG3/AAaGfx4IPBD++QJgKmBAZ2B2eHtt4Nvwn7XCP9cK+r0l2w04DugQ/rkG8BXQVsfbl2NtQPXwz5WA2eFjOA64PLx9BHBT+OebgRHhny8HXgr/3Db8+VIFaB7+3MkM+v0l4w24HfgvMCl8X8fav2O9EqhbZFvSfI6oR6Z0OgHLnXPfOuf2AS8CvQNuU0pyzs0AthbZ3Bt4Nvzzs8AvorY/50I+AY4xs+OAHsDbzrmtzrltwNtAT/9bn1qcc+udc5+Ff94BLAEaoePtufAx2xm+Wyl8c0BX4OXw9qLHOvJ38DLQzcwsvP1F59xe59wKYDmhzx+JYmaNgQuBZ8L3DR3rREuazxEFmdJpBKyJur82vE280cA5tz788wagQfjn4o67/j7KKNydfiqhngIdbx+Ehzq+ADYS+pD+BvjBOXcg/JTo41Z4TMOP5wN10LEurUeAAcDB8P066Fj7yQHTzGyemfULb0uaz5EsL4qIeMU558xMawJ4yMyqA68Atznntod+GQ3R8faOc+5HoL2ZHQO8BpwQcJPSkpldBGx0zs0zs3ODbk8F8RPn3Dozqw+8bWZLox8M+nNEPTKlsw5oEnW/cXibeOP7cNcj4T83hrcXd9z191FKZlaJUIj5j3Pu1fBmHW8fOed+AN4DziDUrR75hTH6uBUe0/DjNYEt6FiXxpnAz81sJaFh/q7Ao+hY+8Y5ty7850ZCIb0TSfQ5oiBTOnOAVuFZ8ZUJTRibEHCb0skEIDKD/WrgjajtV4VnwXcG8sNdmW8BPzOzWuGZ8j8Lb5Mo4XkAo4AlzrmHox7S8faYmdUL98RgZlWB7oTmJL0H9Ak/reixjvwd9AHedaEZkROAy8Nn2jQHWgGfJuZdpAbn3J3OucbOuWaEPovfdc5dgY61L8zsKDOrEfmZ0P//hSTT50jQs6FT5UZoJvZXhMa97wq6Pal6A8YC64H9hMZIryU0Xj0d+Bp4B6gdfq4Bw8PHfAGQF1XnGkKT85YD/xf0+0rGG/ATQmPbXwJfhG8X6Hj7cqxPAT4PH+uFwD3h7ccT+nJcDowHqoS3Z4fvLw8/fnxUrbvCfwfLgPODfm/JfAPO5X9nLelY+3OMjyd0dtd8YFHk+y+ZPkd0iQIRERFJWRpaEhERkZSlICMiIiIpS0FGREREUpaCjIiIiKQsBRkRERFJWQoyIiIikrIUZERERCRlKciISFIws1fN7G9mNsPMVpvZeUG3SUSSn4KMiCSLkwldwfhs4PfAFQG3R0RSgIKMiATOzKoRupjfsPCmSsAPHtZ/xqtaIpJcso78FBER37UF5jnnfgzfPwVYGL56971ANUK/eP0NeJPQxeZyCV0EMBt4ENgLFAB/Cd93wCrgGaClmQ0B2jrnLk7UmxIR/6lHRkSSwcmELmoZcQqhCzD2A6oS6p2pCXQExjrn7gQ2Erpw3S3AGOfc7cAJwE3AG865O5xzjwEdgFecc3cBuxL0fkQkQRRkRCQZFA0yJxG6ivSpwEDn3GDn3NWEgsz88HNqOuc2AScC88ysMqEemdOAmVG1Ov7/9u7QpqIgCgLoTAE4CiB0RAG0g0OgCI4E+RUN/Ga+RFABWcT7AomBxyXnqF131012Ntlsv/QmyUeAf0W1BOzufJvydX+dJG1fkzy3PSU5ZquTLtreZKuMkuSQ5PG8vk9yleSp7XuSu2y11UPbyyRvP3wU4Jd1rbX3DADf0vZlrXW79xzA3yHIAABjeSMDAIwlyAAAYwkyAMBYggwAMJYgAwCMJcgAAGMJMgDAWIIMADDWJ1lj8Brb8U5kAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#6. Finding the cosine similarity"
      ],
      "metadata": {
        "id": "PZ4qil-zpF1g"
      },
      "id": "PZ4qil-zpF1g"
    },
    {
      "cell_type": "code",
      "source": [
        "cos_sim_ls = []\n",
        "# Cosine values closer to -1 indicate greater similarity\n",
        "# Cosine values closer to 0 indicates orthogonal\n",
        "# and values closer to 1 indicate greater dissimilarity.\n",
        "for grad_bc, grad_pde in zip(grad_bc_ls, grad_pde_ls):\n",
        "  cosine_loss = tf.keras.losses.CosineSimilarity(axis=1)\n",
        "  cos_sim_ls.append(cosine_loss(grad_bc, grad_pde).numpy())\n"
      ],
      "metadata": {
        "id": "0Yi7-Yr_eViJ"
      },
      "id": "0Yi7-Yr_eViJ",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "fig = plt.figure(figsize=(9,6))\n",
        "plt.title(\"Cosine Similarity against Epoch\")\n",
        "ax = fig.add_subplot(111)\n",
        "ax.semilogy(range(len(cos_sim_ls)), cos_sim_ls,'k-')\n",
        "ax.set_xlabel('$n_{epoch}$')\n",
        "ax.set_ylabel('Cosine similarty');"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 406
        },
        "id": "5qnL3NsnoWXp",
        "outputId": "fc6e29f6-202a-4b15-c44e-268f05891c6d"
      },
      "id": "5qnL3NsnoWXp",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 648x432 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGFCAYAAADn6xkJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xU5dn/8c/Fsiy9I0pZQTHYNboSo8ZHk6iYQHyCJZYYn0hCNGo0MYka9Wd8YsQSOxofbNhih9gI9h5FwApiASQUkQ7SYXev3x/nzDq77OzO7s6Zcub7fr3mxZx7ztznmjPLzDV3Obe5OyIiIiKFqFWuAxARERFpLiUyIiIiUrCUyIiIiEjBUiIjIiIiBUuJjIiIiBQsJTIiIiJSsJTIiOQxM5thZodk+ZjlZrbWzEqa+fy1ZrZDeH+cmV3Wglj+ZWanNPf52ZL8muPEzAaYmZtZ61zHIpKKEhmRDDGzE81saviltij8Ej6oJXW6+27u/nKGQqxhZv3M7DEzW2Zmq81supn9T3jMee7e0d2rmlN3+Nw5mYjT3Y9097vDmP/HzF7PRL2ZlonXnE7SFyYV68K/scTtjy05rkihU5YtkgFm9jvgfOA04BlgMzAUOArIxy/fe4H3ge2BTcAewLY5jSiJmRlg7l6d61jy0F7uPivXQYjkC7XIiLSQmXUB/hc4w93Hu/s6d9/i7k+6+x/CfcrM7Hoz+yK8XW9mZeFjPc3sKTNbZWYrzOw1M2sVPjbXzL4f3v+zmT1sZveY2Zqw26kiKY4+YSvLUjP73Mx+00DY+wHjwlgr3f1dd/9XWE+t7gQze9nMLjOzf4ctAE+aWQ8zu9/MvjKzKWY2ICkON7NB9ZynbuHrXGpmK8P7/ZIef9nM/mpmbwDrgR3Csl+Y2S7ArcC3wxhWmdl+ZrY4uQvMzEaY2fsp3qcfmtm7YczzzezPdR7/mZn9x8yWm9nFdc79EDN7MzzuIjMbY2Zt6nvNYcvKzWb2dPg+TTazHcPHzMyuM7MlYRwfmtnuZjYKOAn4Y+IcN/De1Sv8+3jUzB4Kj/uOme2V9Pgu4flcFf7t/CjpsXZmdk34+leb2etm1i6p+pPMbJ4FLXgXNjU2kSgpkRFpuW8DbYEJDexzIbA/sDewFzAEuCh87FxgAdAL6A38CUi1dsiPgAeBrsATwBiAMPF5kqCVpS/wPeAcMzsiRT1vATeb2fFmVt74S+R44OSw7h2BN4G7gO7ATOCSNOpoFT5ne6Ac2JCIP8nJwCigE/CfRKG7zyRo7Xoz7Mbp6u5TgOXA4XWef0+K468DfkZw7n4InG5m/w1gZrsCtxAkE9sBXcLXmlAF/BboSfB+fw/4dQOv9XjgUqAbMAv4a1h+OHAw8I3wGMcBy919LHA/cFX4+oY3UHdDjgIeIXhf/gH808xKzayU4O/jWWAb4CzgfjMbHD7vb8C+wAHhc/8IJLeGHQQMDl/3/wsTS5G8oERGpOV6AMvcvbKBfU4C/tfdl7j7UoIvuZPDx7YQfHluH7bkvOapF0F73d0nhuNX7iVIiiBoYenl7v/r7pvD8Rq3EXyh1udY4DXgYuBzM3vPzPZrIP673H22u68G/gXMdvfnw9f8CPDNBp4LgLsvd/fH3H29u68h+HL/rzq7jXP3GWEr0ZbG6gTuBn4KYGbdgSMIvsDrO/7L7v6hu1e7+wfAA0nHPwZ40t1fd/fNwP8jKZl092nu/lYY11zg/+qJPdkEd387PD/3EySwELzXnYCdCbrOZrr7ojReZ7J3wlaVxC05WZ3m7o+G5+5aggR7//DWEbgi/Pt4EXgKOCFMgk8Fznb3he5e5e7/dvdNSfVe6u4b3P19gmR5L0TyhBIZkZZbDvS0hmd29CGphSG83ye8fzXBr/ZnzWyOmZ3fQD1fJt1fD7QNj7s90Cf5C46gZad3fZW4+0p3P9/ddwv3eY/g17ulOO7ipPsb6tnu2EDMAJhZezP7v7D74ivgVaCr1Z4dNb+xeuq4DxhuZh0IWjdeS5UYmNm3zOylsGtrNUELT8/w4T7Jx3b39QTva+K53wi7wr4MY7886bn1qfs+dQzrfZGgFepmYImZjTWzzk17yewTtkglbs8kPZb8GqoJWvr6JF5fnTFH/yFodepJkPDMburrEckHSmREWu5NggGz/93APl8QJBsJ5WEZ7r7G3c919x0Iuo5+Z2bfa2IM84HP63zBdXL3HzT2RHdfRtC10IegWyEq5xJ0T3zL3TsTdLEAJCdPqVqi6n3M3RcSnP8RBC1c9zbw/H8QdMf1d/cuBGNuEsdeBCSP12lH0NKW8HfgY2CnMPY/1Yk7be5+o7vvC+xK0MX0h8RDzamvjv6JO2FLSz+Cv7MvgP5hWUI5sBBYBmwk6DIUKThKZERaKOxu+X8EY07+O2x5KDWzI83sqnC3B4CLzKyXmfUM978PwMyGmdmgsDVkNcF4jKbO1nkbWGNm54UDN0vCQaT1dheZ2ZXh463NrBNwOjDL3ZfXt3+GdCJovVkVdgOlM64m2WKgX/Ig29A9BGM69gDGN3L8Fe6+0cyGACcmPfYoQcvOAWH9f6Z2otIJ+ApYa2Y7E5yvJrNggPK3wjEr6wgSiMR7vRho6bVo9rVgwHNr4ByCBPstYDJBS8ofw7/NQ4DhwINhK82dwLUWDBgvMbNvWzgYXSTfKZERyQB3vwb4HcEA3qUELSRnAv8Md7kMmAp8AHwIvBOWAewEPA+sJWhduMXdX2ri8auAYQRjMT4n+JV9O8GA0vq0JxicvAqYQ9Ba9KMU+2bK9UC7MLa3gElNfP6LwAzgSzNbllQ+gSD+CWGXUCq/Bv7XzNYQJJIPJx5w9xkEA2AfJGidWQssIUgEAH5PkPisIRh79FATY0/oHD5/JUHXznKCrkWAO4Bdw67Bf6Z4PsD7Vvs6MtcnPfY48JOw/pOBEeG4q80EicuRBOf/FuBn7v5x0uv7EJgCrACuRN8PUiAs9ZhCEZHCYGazgV+5+/MZqq8jQZK3k7t/nok6o2bBdPJB7v7TXMcikk3KuEWkoJnZ0QTjS15sYT3Dw27BDgRjhj4E5rY8QhGJkq7sKyIFy8xeJhg0e3IGrgJ8FMFgYSPoBjy+gWnwIpIn1LUkIiIiBUtdSyIiIlKwYtm11LNnTx8wYECuwxAREZEMmDZt2jJ371XfY7FMZAYMGMDUqVNzHYaIiIhkgJn9J9Vj6loSERGRghWrRCacPjl29erVuQ5FREREsiBWiYy7P+nuo7p0SXUxUxEREYmTWCUyIiIiUlyUyIiIiEjBUiIjIiIiBStWiYwG+4qIiBSXWCUyGuwrIiJSXGKVyIiIiEhxUSIjIiIiBUuJjIiIiBQsJTIiIiJSsJTINMGUKVNo27YtmzdvznUoIiIiQswSmainXx988MFs2rSJ8ePHN7jfpEmTeOCBByKJQURERL4Wq0Qm6unXw4cPT2u/I488khNPPDGSGERERORrsUpkota6dWsAqqurcxyJiIiIgBKZJmnVKjhd9SUyGzZs4JhjjmHu3LlZjkpERKR4tc51AIUkkcg8/PDD9O3bl0MPPbTmsYkTJ/LYY4+ptUZERCSL1CLTBIlE5sknn+S73/1uo/u//PLLEUckIiJS3NQi0wSJRCbhjjvu4Nhjj6Vz586Y2Vb7H3roocydO5ftt98+WyFKyN1Zt24dK1asYM2aNaxdu5YNGzawYcMG1q5dy9q1a1m3bl3N/cT25s2bWb9+PRC832ZGaWkpHTt2pLKyEgAzo1WrVlRXV9OpUyfKysro3LkzlZWVtG/fng4dOtCqVStat25N165d6dSpE507d6Zr165UVlZSVlbGdtttR/v27XN5ikREYkGJTBPUTWR+8YtfMHnyZMaOHVtTNmHChFr7aCXuzKqsrGThwoXMmzePefPmsWDBAubPn8+CBQtYunQpK1euZMWKFaxYsYItW7akVWerVq3o2LEjHTp0oKysjHbt2gHBWCh3Z/PmzaxevZo2bdrQqlUr3J3q6mo2btxIZWUlmzdvrklymqJbt25UV1fTv39/unTpQpcuXejZsyc9e/akR48ebLPNNpgZu+22G1VVVey66660bt2ajh071ps4i4gUo7xPZMxsB+BCoIu7H5PjWLYq+/jjj1M+BlBVVRVpTIUskSRUV1fXtJps3LiRBQsWsGzZMpYuXcq8efP44osvmD9/PvPmzWPhwoVbjUPq2rUr/fv3Z5tttqFv375069aN7t270717d7p160bnzp3p0KED7dq1o3379nTs2LHm1qFDB9q2bduixKC6upp169bVul9VVcXGjRtrkqrq6mpWrVrFxo0bKSkpYeHChcydO5fNmzezZs0aVq9ezeLFi/nwww9Zvnx5TatQfTp16kTXrl3p27cvZWVl9OzZk7KyMhYvXsyRRx6Ju7PffvvRvn17Bg4cSI8ePZT4iEhsRZrImNmdwDBgibvvnlQ+FLgBKAFud/crUtXh7nOAkWb2aJSxpqNuiwxQ86tfiUxqK1as4N133+Xhhx/mvffe4/PPP6eqqoqvvvqq0ZaMdu3a0adPH/r378+hhx5KeXk522+/PeXl5ZSXl9OvXz86duyYpVdSv1atWtGpU6ea7Uxcx2jjxo3Mnz+fqqoqPvvsM6qrq/n4448xM+bNm8dXX33FwoUL2bJlCzNmzGDevHmsX7+eF154ocF6R4wYQVlZGZ9++ik9evTg5JNP5pNPPmH//ffnsMMOo3Xr1vX+nYuI5KuoW2TGAWOAexIFZlYC3AwcBiwAppjZEwRJzeg6zz/V3ZdEHGPalMikx935/PPPmThxIs8++ywTJ06sOQ9Dhgxh6NChdOrUiS5dutCxY0dKSkpo3749nTt3pqysjL59+9KrVy969OhBz549i7I1oW3btuy0004A7LzzzgAcddRRDT6nqqqKBQsWMHv2bObPn8/ixYtZu3YtM2fO5O2332bBggV8/PHHfPrppzUJ5LPPPltvXfvttx+77LIL3bp1Y9ttt6VXr15069aNfv360aNHj5qxP6WlpUX5/ohI/og0kXH3V81sQJ3iIcCssKUFM3sQOMrdRxO03jSLmY0CRgGUl5c3t5oG1ZfINNai0JyxE4Vq5cqV3HPPPdx555188MEHAAwcOJAzzjiDAw88kMMOO4xu3brlOMr4KikpYfvtt09rcPmKFStYt24dU6ZM4bTTTmPp0qW1Hv/ggw9YuHAhS5cuTXusUV0HHXQQX3zxBQceeCBDhgxh2LBhlJeXq8VHRDIqF2Nk+gLzk7YXAN9KtbOZ9QD+CnzTzC4IE56tuPtYYCxARUWFZy7crzXnA7gYWmSqqqq4/fbbufDCC1m+fDkVFRXccMMNHHHEEQwePDjX4Uk9EmOI+vfvz4gRI1Lu5+5s2LCBJUuWsHz5chYuXMiqVatYtGgR559/foPHeP311wGYM2cO9957L2eddVbKfTt16oS74+7sv//+7LPPPhxzzDHss88+lJSUqNVHRFLK+8G+7r4cOC2dfc1sODB80KBBkcRSXyKT+IBNlbBUVVUxc+ZMjj32WN59911KS0sjiS1XXn/9dc466yzee+89/uu//otrrrmGfffdN9dhSYaYGe3bt2fAgAEMGDCg1nt73nnnbbX/pk2bWL9+PStWrOCdd97h7LPPZtGiRY0eZ82aNTX3X3jhBV544QWuvvrqRmNzD36zXHvttbzzzjucc8453HTTTYwZM4aOHTtSXV3NfffdR+/evenbty8777xzzVIjIhIPufgfvRDon7TdLyxrMXd/EniyoqLil5mor676fhWaGX//+9/59a9/Xe9zEtNmAX77298yZsyYKELLumXLlnHOOedw//33069fPx566CGOPfZY/XIucmVlZZSVldGtWzd23HFHjj322JT7rlu3jgcffJA333yTwYMHc9lll/HVV1+lfaxEEgPwu9/9DoD77rsPgLvvvjut56Vr5cqVlJSUUFJSAkCHDh2aXIeIRCMXicwUYCczG0iQwBwPZGSp6KhbZFK58sorUz6W3FKzYcOGbIQTuWXLlnHIIYfw2WefcdFFF3H++efrg12arEOHDowcOZKRI0cC8Ic//KHB/Tdu3Mh9991H69at+fTTTxkzZkytlpxM22WXXfj888/ZtGlTyn0WLVrEtttum3ady5YtY82aNfTp04c2bdqwbt06Zs2aRZ8+fdhmm20yEbZI0Yl6+vUDwCFATzNbAFzi7neY2ZnAMwQzle509xmZOF6uWmQaGjuT3DxeUVERRVhZtWTJEoYOHcrs2bOZNGlSrfWmRKLUtm1bfvGLX9RsX3755VvtM2fOHA477DCee+45tt1223oT7KqqqpqWlYYkrhHVkO222y6tFp6NGzfSrVs3Nm7c2OB+55xzDtddd12j9dXl7kybNo3S0lKWLFlCu3btOOigg5pcj0ghinrW0gkpyicCEzN9vFy0yDSWyCRf16O5sz/yxdSpUznqqKNYsWIFEyZMUBIjeWeHHXZg9uzZNduVlZVbjYmZMGECxxyT3Wtrfvzxx40mMQDXX399sxKZCRMmcPTRR29VvmXLlmaPCXrvvfeYN28e/fv3Z/To0fzhD39g+fLlDB06tFn1iUQlVqPe8rFFJtnkyZMzHVLWvPbaaxx++OH07t2bt956i7322ivXIYk0qqSkpKbFJPH/t6GrJheqhQvrH2ZYWlrarDFBP/jBD/jXv/5Vq+yRRx6pud+cOiGY1j9nzhxefPFFXnzxRQ444ABWrVrFBRdcwDe/+c1m1SkSq0QmF5qSyLz00ksRRxONNWvWMHLkSPr06cObb76pvnwpaI888gg/+9nPch1GxpgZPXv2zFh9t99++1ZJTKbU/QE0Y0YwquCRRx5pcnKU2P+DDz5gzz335JNPPuG5555jzJgxXHbZZVRUVDB+/HhOOeWUjJ4fyT+xSmRyNdg33URm0aJFLF68mN69e0ccUeZUV1dzyimnMHv2bJ5//nklMVLwnnrqqYzWN336dHbffffGd8ygpUuXsvfee/PMM88AwSDiTLnooosyVleyyy67rMHH165d26TlRr7//e/z4osv1vvYcccdV3P/97//PdXV1c2eUblixQrWrFnDRx99RNeuXfn444/Zb7/92HbbbencuTNt2rRpVr2SObG6xKa7P+nuozKx1k19UnUtNeU/SFS/dKJy1VVXMWHCBP72t79pTIxIPRJXsc6mp59+mi+++IK//e1vje775ZdfZiGi1F555RXMjIsvvrjB/b744otG6zriiCM45JBDAFImMfWZNm1a2vtWVlZyxx13YGacc8459OjRgwEDBvCDH/yAAw44gFNPPZU99tiDXr16UVZWxrXXXpt23Q1JLKKb6/erEMUqkcmF5Itypbt/oXjllVe46KKLOO644zjnnHNyHY5IRmRiiYTkad+rVq1qcX1RWrlyZU6P/+STT6a1XzozyZ599lleeeWVJsdw1VVXpb3vjTfeWDM77oYbbmh0/3PPPbfZY4ZmzpzJjTfeyJ577kmrVq0oKytju+22q/mBnLjF5dIdUYlVImNmw81s7OrVq6Oqv96y6urqtOto1aoV7s5VV12V1vTOXFmzZg0nn3wyO+64I7fddltBJWAi9Tn11FMBturaraqqqjXTqTElJSW1ukDOOOOMFsc2evTXK6989tlnLa4v2bhx49Le9/7772fx4sUZPX66Mn3F5eTkoqFrASW75pprOPfcc5t8rPfee6/JzwHYd999Ofvss/nwww8b3K+hC0vWNX/+fB577LGa7blz53LLLbfwzjvvMGLEiK2SpN///vfNij2fxCqRibprqYHjNmnfu+66i/POO4/9998/wqha5sILL2TBggWMGzeOzp075zockRZL9Yv/4osvZtCgQcyZMyetevr379/4Tk1w0kkn1Vq3qikLzaaarZRs1qxZadf305/+NOVj//nPf2ruL1myJO0605XJBK7ushjz5s1L63nXXHNNs47XnFYiSH2R1COOOILPP/+8Zvvpp59Ou87999+/1uUFEgv37rvvvkyYMGGr/a+55pqCb/GJVSITtUy0yNx00001VzJdvXo1UbUetcQ777zDmDFjOOOMM/j2t7+d63BEIvXyyy8D6Y8lefTRRyOMpmmef/75RvdJt2unMeXl5TX3E+csk+6///60921szEvdqy03t8UklS1bttC3b9+a7d/+9rcZq/u6665j0qRJDBgwoFnjEtMZa1TX22+/3eTnJLvooos4+uijI0lw0xGrRCbqrqUUx0y72RLg008/rbWdzi+qbLvwwgvp1q1bo7MMRApROotYNqTQFkWN4kKcmfqMTW7NTmeMTEIur5I+efJkWrdu3eIWpKVLl9Zbnjwe8ZJLLmnRMVI56KCDap37lnbr/fWvf2X8+PE5mxASq0QmV7OW5s+fn3YdnTp1qrW9YsWKFseVSVdeeSWTJk3ivPPOI9tddCISLKbZUrfffnsGIkktE+OC6nr44YczXmcUhgwZAkC7du1aVE86F0gtLS1t0TFSefXVVwHo1asXAA8++GCz6qmqqqr1/ffRRx+1PLhmiFUikwtNHQRbd8ZEPiUyCxcu5NJLL+VHP/pRzWrCIpJdyeuzNVfUv4yjaOXJ1AKgu+yyS839liYbcZX43kpcTXnq1KmNPmfVqlVbjaX585//XKvLMVeUyDRBS2bunH322QAsX768Vvmll17KzTff3KK4MuXiiy+mqqqK66+/PuMzCEQkPZs3b25xHcU8y/AnP/lJrkNIacWKFYwYMSLllPjmrLPVEk2ZqNKtWzf23nvvWmXPPfdcpkNqFiUyLZTuB8avfvUr4Ot1XhKDz9555x3OPPPMnI8af++99xg3bhxnnXUWAwcOzGksIpJ92f4SzSfjx4+vNQU+Ktdffz0TJkzgpptuqvfx5l6PprkSx0v38gN1x3jmi1glMrm6jkzbtm0bfW7dFo4999yz1vZTTz2V9T/iZJdffjldu3blwgsvzFkMIpIZffr0adL+VVVVselOTu5WS/cz9eijj+ZPf/pTVCGlLdvfAYkZt6kGHheKWCUyubiOTLotMsmJzNVXX73V84477jjOO++8jMaWrkWLFvHPf/6TU045hW7duuUkBhHJnLKysibtn7yydaE7+OCDcx1Cs9VNZJKneGfjeIUqVolM1FIlLekkM8mJTN0ri3bt2hUIEpy77rqrBRE2zx133MGWLVs47bTTsn5sEcm9RJe35FbdxCJ5IG1zxk7NnDmzxTEVAiUyLdScFpm6K7zOmDGDL7/8kt13351TTz21SYuhZcIbb7zBnnvuyeDBg7N6XBGRfNLSawy1VN2lCJK/X5qTbJ500kkNPh5Fi0xVVVXG62yMEpkWak4ik5gSmJiKve2229K7d2/eeustvvGNb3D88cdndT7++++/v9VodBEpTN/4xjcyXuePf/zjtPfduHEjv/rVr7aaoSmNS2e8Zb6LYmp+Y5TINEGqwb7pSE5kEv3XH330EY8++mhNQtOhQweeeuopWrduzfe+9z3Wrl2bgagbtnjxYhYtWsRee+0V+bFEJHoXX3xxxutsyti5e++9l7Fjx9YMnk31Gamu7Nzr3r17rkPICCUyLdScRKZNmzYADB48mKOPPrrWfjvttBN33XUXX375ZSTrmdQ1ZcoUAPbbb7/IjyVS6K6//vpch9CoI444IqfHT3RXNNZt0dDMqpUrV2bkejodOnTYKq5CFUVLxx//+MeM15kLsUpkcjH9Ol31JTKpJFbFbmxp95aqrKzkiiuuoHv37gW3foxILmy33Xa5DqFRjX2+NEe2L7DXvXt3hg0b1uJ6MnGV5Hxxxx13ZLzO5L+VxYsXZ7z+bIlVIpOr6deN/Sfv379/rX0a+6Dp0qULO++8M0888USkvyIuv/xy3njjDW644Qbat28f2XFERJoqE1eNjdN6cZlooWrI3XffHWn9UYpVIhO1+hKW6urqRkeTf+9736u1nc5CYGeddRZvvfUWr7zyStOCTNMLL7zApZdeynHHHcdPf/rTSI4hIiKFoZAviqdEpoXSmSrt7k1qkQH4+c9/TpcuXbjvvvtaFF99vvjiC0488UQGDx4cSXOlSFwV8xpG0rAZM2Y0aX/9LWWOEpkmaMkfXlMTmXbt2nHQQQfx73//u9nHTDZnzhwuueQSli5dyoknnsjatWt59NFHt7qmjYjkVj4OSs3HmPJNNmaZ5pt8Sca0xHEWNKdFBuCAAw7g6aefZuXKlS1aOmDdunXsvvvubNiwgcsvv5zKykpuv/12dt1112bXKVKMsvHB/fjjj0d+DJE4UYtMEzT3Q+zpp5+utZ1uIvOtb30L+HqKdHNNnjy5ZnXtyspKBg8ezM9//vMW1SkixSNffnmL1EeJTBb07t27WS0yQ4YMobS0tMVLFiSWXj/llFOA4LLViYvwiYiIFLKC+DYzs/82s9vM7CEzOzyHcTTref369av13HRmLQF06tSJAw88kPHjx9cst94cn332GW3btuXKK6/k9NNP54wzzmh2XSLFTC0TkqwYP0vfeuutXIewlcgTGTO708yWmNn0OuVDzewTM5tlZuc3VIe7/9PdfwmcBvwkynijkvwBmHxxvMb88pe/5LPPPmPSpEnNPvZnn33GoEGD6N27N7fccktsLkstItkRxUX24iCx3EwcpPtjedasWRFH0nTZaJEZBwxNLjCzEuBm4EhgV+AEM9vVzPYws6fq3LZJeupF4fNyIhe/xo455hi22WabZl+syN15++232X333TMcmYgUi0K4orFsrSnfWddee22EkUQr8kTG3V8FVtQpHgLMcvc57r4ZeBA4yt0/dPdhdW5LLHAl8C93f6e+45jZKDObamZT8/HCPs1Ngtq0acOIESN46qmnmrWM+9y5c1m8eDHf+c53mnV8EfmaupYkE6644opchxAruRoj0xeYn7S9ICxL5Szg+8AxZlbvkqnuPtbdK9y9olevXpmLNENa8gF47LHHsn79+iZ3L1VVVXHCCScAuV9ITkSKk5K/rSVmkUpmFMRgX3e/0d33dffT3P3WVPvl86KRLfGd73wHM+P999+vKXvzzTfZb7/9WLVqVcrnPc453SAAAB99SURBVPPMM0yePJlhw4ax4447ZiNUkVjTl7JI/slVIrMQ6J+03S8sa5FcLBqZDaWlpWy77bYsXPj1KfrLX/7C1KlTGxxBfuutt9K7d2/Gjx+fjTBFRCRPnH766WzatCnXYWRFrhKZKcBOZjbQzNoAxwNPtLTSuLbIAPTp06dWIpO4ZHiqJsrbbruNJ598kpEjR6Y93VtEROLh1ltvbfL6T4UqG9OvHwDeBAab2QIzG+nulcCZwDPATOBhd2/xGY9riwwEy9GvWbOmZjsxhXvJkiW19nvkkUcwM0aNGkV5eTm/+c1vshqnSJypaylzfvjDH+Y6hJxq6t9SZWVlRJEUvmzMWjrB3bdz91J37+fud4TlE939G+6+o7v/NRPHinOLTKdOnepNZBYtWlRrv5tuugmA3XbbjYkTJ9K7d+/sBSkikuTwww/n6quvrvexffbZJ8vRFLa//OUvuQ4hbxXEYN90xblFplOnTrVWV00kNfPnfz35a9OmTbz++utccMEFTJ8+nd122y3rcYqIJDz33HO5DkHqyGQCmS8tlLFKZKKWyzetY8eOtVpkvvrqK6B2IrNkyRLcnYEDB2Y9PpFikC8f3CLN1a5du1yHkHGxSmSi7lrKpbpdS4lBvl988UVN2bJlywDIx+voiIiIRCFWiUzUXUu5HiOzcePGmgFfiUQmeSbTunXrAOjQoUP2AxQpAnFskXH3opmmK/EUq0SmUDTnw7Bjx44ANeNkEonMqlWrapYu2LhxIxDPpkMRicYtt9zCr3/961yHURA0eSI/xSqRifusJfh6kO+GDRvo0aMHALNnz64pAyUyIpK+hx56KNchFIxp06blOgSpR6wSmbjPWoKvW2TWr19PRUUFANOnTwe+TmTatm2bgwhFROKtb9+GlgSUXIlVIhNnia6lNWvWUFVVxaZNm9hnn30oKSnhgw8+ANS1JCIixSdWiUwxdC2tXbu2ZlBvjx492H333XnttdcAdS2JiEjxiVUiE+eupURysnHjxlqzkw455BDeffddqqur1bUkIiJFJ1aJTJwlkpONGzfWzFJq164dO+64I+vXr2f58uXqWhIRSSHqtYqqq6sjrV9SUyJTIJITmc2bNwNQVlZWMx1w8eLFNS0yZWVluQlSRCRPPfLII5HWf8MNN0Rav6TWOtcBxN3AgQO58sorW1xPciKzZcsWAEpLS+nZsycA//jHP6isrKRt27axvGiXiEhLJD43o5K4DEa69DmdObFqkcnHJQqmTZvG3nvv3eJ6Eq0sdROZ/fffH4DHHnuM5cuX07Vr1xYfS0RE4imTCVS+JGOxSmTiPNg3VYtMIsH59NNPeeONN9hll11yFqOIiOQ3d891CBkXq0QmzuobI9OmTZta+3zyyScMGDAg26GJiIjkjBKZAlFaWoqZbdUiAzBp0qSa/RJjZkRERIqBEplmOu6445r93Ob0K5oZZWVl9SYy5eXlNfspkRERkWKiRKaAlJaWUllZuVUi06tXr5p9lMiIiEgq+TJAN5OUyBSQ1q1bU1lZWXNhp9atg9nz3bt3r9lHiYyIiBSTWCUy+Tj9OpMSLTKJUeetWrWq9S9A//79cxKbiIikr6qqKtchxEasEpk4T7+GoAUm1UWdErOaNGtJRCT/XXHFFbkOITZ0Zd9mykU/Y6JFpj6TJ0/m9ddfp1u3blmOSkREmur999/PdQixoUQmYplMeBpqkdlzzz3Zc889M3YsERGJztq1a3MdQmzEqmsp7hKDfUVERCSgRKaAlJaWRr7wmYiISCFRIlNA1CIjIiItkVhoOE6UyBQQJTIiIoXnmmuuwcxYv359rkNh9OjRGasrXy6ul/eJjJntYma3mtmjZnZ6ruPJJXUtiYgUnuuuuw6AFStW5DiSry+kGieRJjJmdqeZLTGz6XXKh5rZJ2Y2y8zOb6gOd5/p7qcBxwEHRhlvtjQ3i1WLjIiISG1Rt8iMA4YmF5hZCXAzcCSwK3CCme1qZnuY2VN1btuEz/kR8DQwMeJ485paZERERGqLtI3J3V81swF1iocAs9x9DoCZPQgc5e6jgWEp6nkCeMLMngb+Ud8+ZjYKGAW1V4OOk9atW7N+/fqaJQpEJHN0yXiRwpSLzrK+wPyk7QXAt1LtbGaHACOAMhpokXH3scBYgIqKilh+09ftWsqXgVYicXDwwQfn/Y+E6dOnN76TSJHJ+1E/7v4y8HI6+5rZcGD4oEGDogwpZ9S1JFLc9thjDwDuuuuuHEcikj9yMWtpIZC8RHO/sKzFimHRSA32FRER+VouEpkpwE5mNtDM2gDHA09komIzG25mY1evXp2J6vJOQ2stiYiIFKOop18/ALwJDDazBWY20t0rgTOBZ4CZwMPuPiMTx4t7i0xDq1+LiIgUo6hnLZ2QonwiEUylzuYYmVwMtFXXkoiISG15f2XfpsjHFplMJjwa7CsiIlJbrBKZuFOLjIiISG2xSmQ02FdERCQ78uVaZrFKZPKxaymTNNhXRESktlglMnGX6FrK96uPioiIZEusEpli61rKl2Y9ERGRXIlVIhP3rqWSkhItbCciIpIkVolM3KkFRkREpLZYJTJx71pKJDIaIyMiIhKIVSIT964lJTIiIiK1xSqRiTslMiJSKJYsWZLrEKRIKJEpIEpkRKRQPP7447kOQYqEEpkCokRGRESktlglMnEf7NuqVfB2KZEREREJxCqRyeZg33SnQmdyynSirurq6ozVKSIiUshilcjEnbqWREREams0kTGz8Wb2QzNT0pNjSmRERCRf5MtFWtNJTm4BTgQ+M7MrzGxwxDFJCnW7lvLlj0hERCRXGk1k3P15dz8J2AeYCzxvZv82s5+bWWnUAcrX1CIjIiJSW1rdRWbWA/gf4BfAu8ANBInNc5FFJltRIiMiIlJb68Z2MLMJwGDgXmC4uy8KH3rIzKZGGVxTmdlwYPigQYNyHUoklMiIiIjUlk6LzG3uvqu7j04kMWZWBuDuFZFG10RxX2tJ15ERERGpLZ1E5rJ6yt7MdCDSOF1HRkREpLaUXUtmti3QF2hnZt8EElNkOgPtsxCb1KGuJRERkdoaGiNzBMEA337ANXydyHwF/CnasKQ+SmRERERqS5nIuPvdZnYvcIK735/FmCQFJTIiIiK1NThGxt2rgd9mKRZphBIZERGR2tIZ7Pu8mf3ezPqbWffELfLIZCtKZERERGpr9DoywE/Cf89IKnNgh8yHUz8z6wC8AvzZ3Z/K1nEbkovlAZTIiIiI1JbOEgUD67mllcSY2Z1mtsTMptcpH2pmn5jZLDM7P42qzgMeTueY+SbKhEdrLYmISK7ky3dQOi0ymNnuwK5A20SZu9+TxlPHAWOAmn3NrAS4GTgMWABMMbMngBJgdJ3nnwrsBXyUfGwRERERSG+JgkuAQwgSmYnAkcDrJCUnqbj7q2Y2oE7xEGCWu88J638QOMrdRwPD6jn+IUCH8PgbzGxiOAi57n6jgFEA5eXljYUmIiIiMZBOi8wxBK0i77r7z82sN3BfC47ZF5iftL0A+Faqnd39QgAz+x9gWX1JTLjfWGAsQEVFhQaRiIiIFIF0EpkN7l5tZpVm1hlYAvSPOK6tuPu4xvaJ+6KRIiIiUls606+nmllX4DZgGvAOLVtraSG1E6F+YVmLxX3RSBEREamt0RYZd/91ePdWM5sEdHb3D1pwzCnATmY2kCCBOR44sQX11VCLjIiISHFJ2SJjZvvUvQHdgdbh/UaZ2QMErTeDzWyBmY1090rgTOAZYCbwsLvPaPlLUYuMiIhIsWmoReaaBh5z4LuNVe7uJ6Qon0gwAyqj1CIjIiJSXBpaNPLQbAaSCe7+JPBkRUXFL3Mdi4iIiEQvZSJjZt919xfNbER9j7v7+OjCEhEREWlcQ11L/wW8CAyv5zEH8i6RKZauJa21JCIiEmioa+mS8N+fZy+clim2rqV8WedCREQkV9JZoqAr8DNgQPL+7v6b6MISERERaVw6V/adCLwFfAjUuzxAvshm11K6rSFqNRERkTjKl++3dBKZtu7+u8gjyYBi61oSEREpduksUXCvmf3SzLYzs+6JW+SRiYiIiDQinRaZzcDVwIUEs5UI/90hqqCaq1hmLYmIiEggnRaZc4FB7j7A3QeGt7xLYkBLFIiIiBSbdBKZWcD6qAMRERERaap0upbWAe+Z2UvApkShpl+LiIhIrqWTyPwzvImIiIjklUYTGXe/OxuBZIIG+4qIiBSXlGNkzOzh8N8PzeyDpNuHZvZB9kJMX7EM9tVaSyIiIoGGWmTODv8dlo1ApOny5aqKIiIiuZKyRcbdF4V3lwHz3f0/QBmwF/BFFmITERERaVA6069fBdqaWV/gWeBkYFyUQYmIiIikI51Extx9PTACuMXdjwV2izYsERERyWf5MrwhrUTGzL4NnAQ8HZaVRBdSYciXN1BERKSYpZPInA1cAExw9xlmtgPwUrRhNY+ZDTezsatXr851KDWU8IiIiESn0UTG3V919x+5+5Xh9px8vapvsUy/FhERkUA6LTIiIiIieUmJjIiIiBQsJTIiIiJSsBpNZMzsG2b2gplND7f3NLOLog9NUtESBSIiIoF0WmRuI5i1tAXA3T8Ajo8yKEmPZkSJiEixSyeRae/ub9cpq4wiGBEREZGmSCeRWWZmOwIOYGbHAIsafkrmmNkhZvaamd1qZodk67giIiKS/9JJZM4A/g/Y2cwWAucAp6dTuZndaWZLEuNrksqHmtknZjbLzM5vpBoH1gJtgQXpHFdERESKQ+vGdnD3OcD3zawD0Mrd1zSh/nHAGOCeRIGZlQA3A4cRJCZTzOwJgmUPRtd5/qnAa+7+ipn1Bq4lWCpBREREpPFExszKgKOBAUDrxABTd//fxp7r7q+a2YA6xUOAWWGChJk9CBzl7qOBYQ1UtxIoayDOUcAogPLy8sZCExERkRhoNJEBHgdWA9OATRk4Zl9gftL2AuBbqXY2sxHAEUBXgtadern7WGAsQEVFheYni4iIRChfZs6mk8j0c/ehkUeSgruPB8ans6+ZDQeGDxo0KNqgREREJC+kM9j332a2RwaPuRDon7TdLyxrsWwuGpluJpovGauIiEgcpZPIHARMC2cZfWBmH5rZBy045hRgJzMbaGZtCC6u90QL6qthZsPNbOzq1aszUZ2IiIjkuXS6lo5sbuVm9gBwCNDTzBYAl7j7HWZ2JvAMwUylO919RnOPkczdnwSerKio+GUm6hMREZH8ljKRMbPO7v4V0JTp1rW4+wkpyicCE5tbbyrFMkZGay2JiIgEGupa+kf47zRgavjvtKTtvJPNMTL5QONvRESk2KVskXH3YeG/A7MXjoiIiEj6Gh3sa2YHhlf1xcx+ambXmlleXnFOg31FRESKSzqzlv4OrDezvYBzgdnAvZFG1UzF1rUkIiJS7NJJZCo9GF16FDDG3W8GOkUbloiIiEjj0klk1pjZBcDJwNNm1goojTas5lHXkoiISHFJJ5H5CcEaS6e6+5cEV+K9OtKomkldSyIiIsWl0UQmTF7uB7qY2TBgo7vfE3lkIiIikrfy5RIg6cxaOg54GzgWOA6YbGbHRB1Yc6hrSUREpLik07V0IbCfu5/i7j8DhgAXRxtW86hrSUREpLikk8i0cvclSdvL03xerOVLk5qIiEgxS2fRyElm9gzwQLj9E+Bf0YUUL1EkPFprSUREJNBoIuPufzCzEcBBYdFYd58QbViSDrUKiYhIsWto9etBQG93f8PdxwPjw/KDzGxHd5+drSDTVSyrX4uIiEigobEu1wNf1VO+Onws72iwr4iISHFpKJHp7e4f1i0MywZEFpGIiIhImhpKZLo28Fi7TAciIiIi0lQNJTJTzeyXdQvN7BfAtOhCEhEREUlPQ7OWzgEmmNlJfJ24VABtgB9HHZiIiIhIY1ImMu6+GDjAzA4Fdg+Ln3b3F7MSmYiIiEgj0rmOzEvAS1mIpcU0/VpERCQ78uVaZrFaakDTr0VERIpLrBIZERERKS5KZAqQ1loSEREJKJEpYPnSPykiIpIrSmSaKd0kQsmGiIhIdJTIiIiISMFSIiMiIiIFq9HryOSambUC/gJ0Bqa6+905DklERETyRKQtMmZ2p5ktMbPpdcqHmtknZjbLzM5vpJqjgH7AFmBBVLGKiIhI4Ym6RWYcMAa4J1FgZiXAzcBhBInJFDN7AigBRtd5/qnAYODf7v5/ZvYo8ELEMYuIiEiBiDSRcfdXzWxAneIhwCx3nwNgZg8CR7n7aGBY3TrMbAGwOdysSnUsMxsFjAIoLy9vcewiIiKS/3Ix2LcvMD9pe0FYlsp44Agzuwl4NdVO7j7W3SvcvaJXr16ZiVRERETyWt4P9nX39cDIdPbVopEiIiLFJRctMguB/knb/cKyFiuWRSO1RIGIiORavlzwNReJzBRgJzMbaGZtgOOBJzJRsZkNN7Oxq1evzkR1eS9f/ohERERyJerp1w8AbwKDzWyBmY1090rgTOAZYCbwsLvPyMTxiqVFRkRERAJRz1o6IUX5RGBipo+nMTIiIiLFJVZLFKhFRkREpLjEKpERERGR4hKrRCabg33THWirAbkiIiLRiVUio64lERGR4hKrREZERESKS6wSmWK7joyIiEixi1Uio64lERGR4hKrREZERESKS6wSmWLpWtJaSyIiIoFYJTLF1rWkqd0iIpIr+fIdFKtERkRERIqLEhkREREpWEpkREREpGDFKpEplsG+IiIiEohVIlNsg31FRESKXawSGRERESkuSmSaSatfi4iI5J4SGRERESlYSmRERESkYCmRERERkYIVq0SmWKZfa60lERGRQKwSmWKbfq2BxCIiUuxilciIiIhIcVEiIyIiIk2WL70CSmRERESkYCmRERERkYKlREZEREQKlhIZERERKVitcx1AY8zsO8BJBLHu6u4H5DgkERERyRORtsiY2Z1mtsTMptcpH2pmn5jZLDM7v6E63P01dz8NeAq4O8p4RUREpLBE3SIzDhgD3JMoMLMS4GbgMGABMMXMngBKgNF1nn+quy8J758IjIw4XhERESkgkSYy7v6qmQ2oUzwEmOXucwDM7EHgKHcfDQyrrx4zKwdWu/uaVMcys1HAKIDy8vKWB9+IdOfP58s8exERkTjKxWDfvsD8pO0FYVlDRgJ3NbSDu4919wp3r+jVq1cLQ8xvWmtJREQkkPeDfQHc/ZJ09jOz4cDwQYMGRRxRflBrj4iIFLtctMgsBPonbfcLy1qs2BaNFBERKXa5SGSmADuZ2UAzawMcDzyRiYrNbLiZjV29enUmqhMREZE8F/X06weAN4HBZrbAzEa6eyVwJvAMMBN42N1nZOJ4apERERHJjnwZ3hD1rKUTUpRPBCZm+njFNkZGRESk2MVqiQK1yIiIiBSXWCUyGiMjIiJSXGKVyKhFRkREpLjEKpERERGR4hKrREZdSyIiIsUlVomMupZERESKS6wSmWKhtZZEREQCSmSaKR9Wv86XixGJiIjkSqwSGY2RERERKS6xSmQ0RkZERKS4xCqRERERkeKiREZEREQKlhIZERERabJ8mXASq0RGg31FRESKS6wSGQ32FRERKS6xSmRERESkuCiRERERkYKlRKYAaYkCERGRgBKZApYvI8ZFRERyRYmMiIiIFKxYJTKafi0iIlJcYpXIaPq1iIhIcYlVIpNN6Y5P0TgWERGR6CiRERERkYKlREZEREQKlhIZERERabJ8GTqhREZEREQKlhIZERERKVitcx1AY8ysHLgRWAF86u5X5DgkERERyRORtsiY2Z1mtsTMptcpH2pmn5jZLDM7v5Fq9gAedfdTgW9GFmwB0VpLIiIigahbZMYBY4B7EgVmVgLcDBwGLACmmNkTQAkwus7zTwXeAh41s1OBeyOOt6Dky0ArERGRXIk0kXH3V81sQJ3iIcAsd58DYGYPAke5+2hgWN06zOz3wCVhXY8Cd9V3LDMbBYwCKC8vz9hrEBERkfyVi8G+fYH5SdsLwrJUJgG/MbNbgbmpdnL3se5e4e4VvXr1ykigIiIikt/yfrCvu08HjklnXzMbDgwfNGhQtEGJiIhIXshFi8xCoH/Sdr+wrMW0aKSIiEhxyUUiMwXYycwGmlkb4HjgiUxUbGbDzWzs6tWrM1GdiIiI5Lmop18/ALwJDDazBWY20t0rgTOBZ4CZwMPuPiMTx8tmi4xWvxYREcm9qGctnZCifCIwMdPH0xgZERGR4hKrJQo0RkZERKS4xCqR0RgZERGR7MiXoROxSmTUIiMiIlJcYpXIFAuttSQiIhKIVSJTbF1L+dKsJyIikiuxSmTUtSQiIlJcYpXIiIiISHFRIiMiIiIFK1aJTLGNkRERESl2sUpkNEZGRESkuMQqkREREZHiokRGRERECpYSGRERESlYsUpksjnYN92L0emidSIiItGJVSKjwb4iIiLZkS8/1GOVyBQLrbUkIiISUCJTwPIlGxYREckVJTIiIiJSsJTIiIiISMFSIiMiIiIFK1aJjNZaEhERKS6xSmQ0/VpERKS4xCqRERERkeKiREZEREQKlhIZERERKVhKZERERKRgKZEREckhLTmSHVGfZ72PuWNxPPlmthT4T0TV9wSWRVS31KZznT0619ml8509OtfZE+W53t7de9X3QCwTmSiZ2VR3r8h1HMVA5zp7dK6zS+c7e3SusydX51pdSyIiIlKwlMiIiIhIwVIi03Rjcx1AEdG5zh6d6+zS+c4enevsycm51hgZERERKVhqkREREZGCpURGRERECpYSmTSZ2VAz+8TMZpnZ+bmOp1CZ2Z1mtsTMpieVdTez58zss/DfbmG5mdmN4Tn/wMz2SXrOKeH+n5nZKbl4LfnOzPqb2Utm9pGZzTCzs8Nyne8MM7O2Zva2mb0fnutLw/KBZjY5PKcPmVmbsLws3J4VPj4gqa4LwvJPzOyI3Lyi/GdmJWb2rpk9FW7rXEfEzOaa2Ydm9p6ZTQ3L8udzxN11a+QGlACzgR2ANsD7wK65jqsQb8DBwD7A9KSyq4Dzw/vnA1eG938A/AswYH9gcljeHZgT/tstvN8t168t327AdsA+4f1OwKfArjrfkZxrAzqG90uByeE5fBg4Piy/FTg9vP9r4Nbw/vHAQ+H9XcPPlzJgYPi5U5Lr15ePN+B3wD+Ap8JtnevozvVcoGedsrz5HFGLTHqGALPcfY67bwYeBI7KcUwFyd1fBVbUKT4KuDu8fzfw30nl93jgLaCrmW0HHAE85+4r3H0l8BwwNProC4u7L3L3d8L7a4CZQF90vjMuPGdrw83S8ObAd4FHw/K65zrxHjwKfM/MLCx/0N03ufvnwCyCzx9JYmb9gB8Ct4fbhs51tuXN54gSmfT0BeYnbS8IyyQzerv7ovD+l0Dv8H6q8673o4nC5vRvErQU6HxHIOzqeA9YQvAhPRtY5e6V4S7J563mnIaPrwZ6oHOdruuBPwLV4XYPdK6j5MCzZjbNzEaFZXnzOdI6E5WIZIq7u5npmgAZZGYdgceAc9z9q+DHaEDnO3PcvQrY28y6AhOAnXMcUiyZ2TBgibtPM7NDch1PkTjI3Rea2TbAc2b2cfKDuf4cUYtMehYC/ZO2+4VlkhmLw6ZHwn+XhOWpzrvejzSZWSlBEnO/u48Pi3W+I+Tuq4CXgG8TNKsnfjAmn7eacxo+3gVYjs51Og4EfmRmcwm6+b8L3IDOdWTcfWH47xKCJH0IefQ5okQmPVOAncJR8W0IBow9keOY4uQJIDGC/RTg8aTyn4Wj4PcHVodNmc8Ah5tZt3Ck/OFhmSQJxwHcAcx092uTHtL5zjAz6xW2xGBm7YDDCMYkvQQcE+5W91wn3oNjgBc9GBH5BHB8ONNmILAT8HZ2XkVhcPcL3L2fuw8g+Cx+0d1PQuc6EmbWwcw6Je4T/P+fTj59juR6NHSh3AhGYn9K0O99Ya7jKdQb8ACwCNhC0Ec6kqC/+gXgM+B5oHu4rwE3h+f8Q6AiqZ5TCQbnzQJ+nuvXlY834CCCvu0PgPfC2w90viM513sC74bnejrw/8LyHQi+HGcBjwBlYXnbcHtW+PgOSXVdGL4HnwBH5vq15fMNOISvZy3pXEdzjncgmN31PjAj8f2XT58jWqJARERECpa6lkRERKRgKZERERGRgqVERkRERAqWEhkREREpWEpkREREpGApkREREZGCpURGRERECpYSGRHJC2Y23swuM7NXzWyemX0/1zGJSP5TIiMi+WIPghWMDwbOBk7KcTwiUgCUyIhIzplZe4LF/K4Li0qBVRms//ZM1SUi+aV147uIiERuV2Cau1eF23sC08PVuy8F2hP88LoMmESw2NxggkUA2wJXA5uA9cAl4bYD/wFuBwaZ2V+BXd39x9l6USISPbXIiEg+2INgUcuEPQkWYBwFtCNonekC7Ac84O4XAEsIFq47Axjn7r8DdgZOBx5393Pd/UZgH+Axd78QWJel1yMiWaJERkTyQd1EZneCVaS/CZzv7n9291MIEpn3w326uPtSYDdgmpm1IWiR2Rd4I6mu/QhW6QWoQkRiRV1LIpJzYWtK8vYOAGb2ODDOzOYDLxJ0J3Uysx8TdBkBPAL8Pbx/DTAA+D8zWwlcTtBtdYOZ9QSWRvxSRCTLzN1zHYOISFrM7F53PznXcYhI/lAiIyIiIgVLY2RERESkYCmRERERkYKlREZEREQKlhIZERERKVhKZERERKRgKZERERGRgqVERkRERAqWEhkREREpWP8fgaIOus6Ofb4AAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3.9 (tensorflow)",
      "language": "python",
      "name": "tensorflow"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 5
}