genai/transformers_101_nmt_eng_hin.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sriksmachi/sriksml/blob/main/genai/transformers_101_nmt_eng_hin.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XQ9FiUiN6slr"
      },
      "source": [
        "## Neural Machine Translation using Transformers.\n",
        "==========================================================================================================\n",
        "\n",
        "In this notebook we will create a transformer model using multi-headed attention for neural machine translation task.\n",
        "\n",
        "A little bit about transformers, before we get started. RNNs were traditionally used for modelling seq2seq tasks, however they have few inherent challenges\n",
        "- They fail to learn as the length of the sequence of input increases. In a typical Encoder-Decoder network architecture, the encoder is responsible for encoding the entire input sentence into latent vector space, which is then used by the decoder network to predict each token.\n",
        "- The learning process in RNN is also sequential due to the auto regressive nature. The prediction of the next token in RNN models depends on the previous token.\n",
        "\n",
        "The breakthough technique that made transformers so popular is the attention mechanism. Let us now understand how transformers solve this problem\n",
        "- Transformers learn using self attention mechanism. Intuitevly, this means they learn the relative weights of every word with respect to other words in the sentence.\n",
        "- For every next token that should be predicted the network learns the relative importance that should be given to the input tokens, that can be learned in parallel.\n",
        "\n",
        " Note: Attention technique can be used with RNNs as well. Using attention mechanism RNNs learn hidden state for each word so that it is easy to pay more attention or discard words rather than learning the hidden state for the entire sentence. However this does not solve 2 problems\n",
        "1.   The entire process is still sequential, it cannot be parallelized\n",
        "2.   For a large sentence the time taken for translating the sentence to encodings is large.\n",
        "\n",
        "Transformers introduced a completely new concept, with some learnings from how CNNs learn embeddings in parallel. Transformers use encoder, decoder architecture to learn attention weights in parallel and they can be used for a variety of tasks like neural language translation, question answering, next word prediction etc.\n",
        "\n",
        "Complete understanding of how transformers work is not explained in this notebook but you may refer to the following blogs for better understanding.\n",
        "- https://jalammar.github.io/illustrated-transformer/\n",
        "- https://mchromiak.github.io/articles/2017/Sep/12/Transformer-Attention-is-all-you-need/#.XIWlzBNKjOR\n",
        "\n",
        "This notebook builds a transformer model following the explanation from the original paper (https://arxiv.org/pdf/1706.03762.pdf).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:42:40.524883Z",
          "iopub.status.busy": "2023-10-22T04:42:40.524538Z",
          "iopub.status.idle": "2023-10-22T04:42:49.170759Z",
          "shell.execute_reply": "2023-10-22T04:42:49.169842Z",
          "shell.execute_reply.started": "2023-10-22T04:42:40.524856Z"
        },
        "id": "_fGjw2YZ_BLF",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "## Importing required libraries\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import matmul, cast, float32,sqrt, math, transpose\n",
        "from tensorflow import convert_to_tensor, string\n",
        "from keras.backend import softmax\n",
        "from numpy import random\n",
        "from tensorflow.keras.layers import TextVectorization, Embedding, Layer, Dropout, Input\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.data import Dataset\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import pathlib\n",
        "import pickle\n",
        "import random\n",
        "import re\n",
        "import tqdm\n",
        "import unicodedata\n",
        "import tensorflow as tf\n",
        "from tensorflow import convert_to_tensor, int64\n",
        "from pickle import load\n",
        "from numpy.random import shuffle\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ig9WSiTY2yc0"
      },
      "source": [
        "## Transformers Architecture.\n",
        "\n",
        "The following images shows the transformers architecture.\n",
        "\n",
        "![image.png](https://github.com/sriksmachi/sriksml/blob/main/genai/images/trans-arch.png?raw=1)\n",
        "\n",
        "Image Source :  https://arxiv.org/pdf/1706.03762.pdf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lVEmW-et8LmZ"
      },
      "source": [
        "We will use the bottom-up approach, i.e. build the transformer from the bottom most layer and then walk upwards.\n",
        "This notebook builds each layer as a separate model so that they can be plugged into a full fledged model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZSO1L1TzKjUy"
      },
      "source": [
        "## Download and Preprocess"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sQRP4wpuuIS_",
        "outputId": "1bcc9bdf-efee-452f-c21b-f1c9582d10c6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1vyR40sftEJ1Jexoat17JzzB2cdcMVI0T\n",
            "To: /content/Hindi_English_Truncated_Corpus.csv\n",
            "100% 41.4M/41.4M [00:00<00:00, 64.0MB/s]\n"
          ]
        }
      ],
      "source": [
        "# for this notebook I'm using a hindi-english Truncated Corpus from OPUS, it can be downloaded from the link below\n",
        "!gdown 1vyR40sftEJ1Jexoat17JzzB2cdcMVI0T"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 225
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:42:49.172859Z",
          "iopub.status.busy": "2023-10-22T04:42:49.172324Z",
          "iopub.status.idle": "2023-10-22T04:42:50.979430Z",
          "shell.execute_reply": "2023-10-22T04:42:50.978266Z",
          "shell.execute_reply.started": "2023-10-22T04:42:49.172818Z"
        },
        "id": "L6LNUZqOtqn5",
        "outputId": "e2c0842c-ed9d-4069-a283-982cbd60a05f",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of examples = 127607\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      source                                   english_sentence  \\\n",
              "0        ted  politicians do not have permission to do what ...   \n",
              "1        ted         I'd like to tell you about one such child,   \n",
              "2  indic2012  This percentage is even greater than the perce...   \n",
              "3        ted  what we really mean is that they're bad at not...   \n",
              "4  indic2012  .The ending portion of these Vedas is called U...   \n",
              "\n",
              "                                      hindi_sentence  \n",
              "0  राजनीतिज्ञों के पास जो कार्य करना चाहिए, वह कर...  \n",
              "1  मई आपको ऐसे ही एक बच्चे के बारे में बताना चाहू...  \n",
              "2   यह प्रतिशत भारत में हिन्दुओं प्रतिशत से अधिक है।  \n",
              "3     हम ये नहीं कहना चाहते कि वो ध्यान नहीं दे पाते  \n",
              "4        इन्हीं वेदों का अंतिम भाग उपनिषद कहलाता है।  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-23f50e94-c685-4b44-874f-7b5f345e62b2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>english_sentence</th>\n",
              "      <th>hindi_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ted</td>\n",
              "      <td>politicians do not have permission to do what ...</td>\n",
              "      <td>राजनीतिज्ञों के पास जो कार्य करना चाहिए, वह कर...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ted</td>\n",
              "      <td>I'd like to tell you about one such child,</td>\n",
              "      <td>मई आपको ऐसे ही एक बच्चे के बारे में बताना चाहू...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>indic2012</td>\n",
              "      <td>This percentage is even greater than the perce...</td>\n",
              "      <td>यह प्रतिशत भारत में हिन्दुओं प्रतिशत से अधिक है।</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ted</td>\n",
              "      <td>what we really mean is that they're bad at not...</td>\n",
              "      <td>हम ये नहीं कहना चाहते कि वो ध्यान नहीं दे पाते</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>indic2012</td>\n",
              "      <td>.The ending portion of these Vedas is called U...</td>\n",
              "      <td>इन्हीं वेदों का अंतिम भाग उपनिषद कहलाता है।</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-23f50e94-c685-4b44-874f-7b5f345e62b2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-23f50e94-c685-4b44-874f-7b5f345e62b2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-23f50e94-c685-4b44-874f-7b5f345e62b2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-5bc1820e-97ab-40a6-948b-e8de477c7051\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-5bc1820e-97ab-40a6-948b-e8de477c7051')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-5bc1820e-97ab-40a6-948b-e8de477c7051 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "ds = pd.read_csv('/content/Hindi_English_Truncated_Corpus.csv')\n",
        "print(f'Total number of examples = {len(ds)}')\n",
        "ds.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "rcP035IKJmOY",
        "outputId": "a13ce818-e59b-4128-ca2c-274f4dc88134"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      source                                   english_sentence  \\\n",
              "0        ted  politicians do not have permission to do what ...   \n",
              "1        ted         I'd like to tell you about one such child,   \n",
              "2  indic2012  This percentage is even greater than the perce...   \n",
              "3        ted  what we really mean is that they're bad at not...   \n",
              "4  indic2012  .The ending portion of these Vedas is called U...   \n",
              "\n",
              "                                      hindi_sentence  eng_len  hin_len  \n",
              "0  राजनीतिज्ञों के पास जो कार्य करना चाहिए, वह कर...       12       14  \n",
              "1  मई आपको ऐसे ही एक बच्चे के बारे में बताना चाहू...        9       11  \n",
              "2   यह प्रतिशत भारत में हिन्दुओं प्रतिशत से अधिक है।       10        9  \n",
              "3     हम ये नहीं कहना चाहते कि वो ध्यान नहीं दे पाते       12       11  \n",
              "4        इन्हीं वेदों का अंतिम भाग उपनिषद कहलाता है।        9        8  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4e78fe04-a933-4c0d-b940-8a731333fc61\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>english_sentence</th>\n",
              "      <th>hindi_sentence</th>\n",
              "      <th>eng_len</th>\n",
              "      <th>hin_len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ted</td>\n",
              "      <td>politicians do not have permission to do what ...</td>\n",
              "      <td>राजनीतिज्ञों के पास जो कार्य करना चाहिए, वह कर...</td>\n",
              "      <td>12</td>\n",
              "      <td>14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ted</td>\n",
              "      <td>I'd like to tell you about one such child,</td>\n",
              "      <td>मई आपको ऐसे ही एक बच्चे के बारे में बताना चाहू...</td>\n",
              "      <td>9</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>indic2012</td>\n",
              "      <td>This percentage is even greater than the perce...</td>\n",
              "      <td>यह प्रतिशत भारत में हिन्दुओं प्रतिशत से अधिक है।</td>\n",
              "      <td>10</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ted</td>\n",
              "      <td>what we really mean is that they're bad at not...</td>\n",
              "      <td>हम ये नहीं कहना चाहते कि वो ध्यान नहीं दे पाते</td>\n",
              "      <td>12</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>indic2012</td>\n",
              "      <td>.The ending portion of these Vedas is called U...</td>\n",
              "      <td>इन्हीं वेदों का अंतिम भाग उपनिषद कहलाता है।</td>\n",
              "      <td>9</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4e78fe04-a933-4c0d-b940-8a731333fc61')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-4e78fe04-a933-4c0d-b940-8a731333fc61 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-4e78fe04-a933-4c0d-b940-8a731333fc61');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6c645bf0-6913-4b7c-8f3f-217b58f4354e\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6c645bf0-6913-4b7c-8f3f-217b58f4354e')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6c645bf0-6913-4b7c-8f3f-217b58f4354e button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "ds['eng_len']=ds['english_sentence'].apply(lambda x:len(str(x).split(\" \")))\n",
        "ds['hin_len']=ds['hindi_sentence'].apply(lambda x:len(str(x).split(\" \")))\n",
        "ds.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MjXsgwyJwAgA"
      },
      "source": [
        "Let us take only sentences less than 15 words to reduce training time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:42:51.760692Z",
          "iopub.status.busy": "2023-10-22T04:42:51.760345Z",
          "iopub.status.idle": "2023-10-22T04:42:51.799146Z",
          "shell.execute_reply": "2023-10-22T04:42:51.798143Z",
          "shell.execute_reply.started": "2023-10-22T04:42:51.760663Z"
        },
        "id": "n9XzwKoItqn5",
        "outputId": "7ef9a9f0-db06-488d-9bc3-62403a5da1ee",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of examples = 66074\n"
          ]
        }
      ],
      "source": [
        "ds = ds[(ds['eng_len'] < 15)]\n",
        "ds = ds[(ds['hin_len'] < 15)]\n",
        "print(f'Total number of examples = {len(ds)}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "t2q97ePcEKmn"
      },
      "outputs": [],
      "source": [
        "# removing the punctuations\n",
        "filters = '!\"#$%&()*+,-./:;=?@[\\\\]^_`{|}~\\t\\n'\n",
        "ds['english_sentence']=ds['english_sentence'].apply(lambda x: ''.join(ch for ch in x if ch not in filters))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YxmVZv7Dv2jK",
        "outputId": "ea78e651-aee6-405d-c2e0-722a962792fb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of examples = 66074\n"
          ]
        }
      ],
      "source": [
        "# we can further reduce the number of examples by setting n_sentences to lower number, this can be useful for debugging\n",
        "n_sentences = None\n",
        "batch_size=32\n",
        "if n_sentences is not None:\n",
        "    ds = ds.head(n_sentences)\n",
        "print(f'Total number of examples = {len(ds)}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t0HUxyyBKNPF",
        "outputId": "2603f122-6bbc-4b65-ed47-0fbad8055833"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "66074\n"
          ]
        }
      ],
      "source": [
        "# creating a list of all the english and hindi sentences\n",
        "def normalize(eng, hin):\n",
        "    \"\"\"Normalize a line of text and split into two at the tab character\"\"\"\n",
        "    eng = str(eng.strip().lower())\n",
        "    hin = str(hin.strip())\n",
        "    eng = \"[start] \" + eng + \" [end]\"\n",
        "    hin = \"[start] \" + hin + \" [end]\"\n",
        "    return eng, hin\n",
        "\n",
        "text_pairs = []\n",
        "for index in ds.index:\n",
        "    text_pairs.append(normalize(ds['english_sentence'][index], ds['hindi_sentence'][index]))\n",
        "print(len(text_pairs))"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "text_pairs[0]"
      ],
      "metadata": {
        "id": "N9dSDVFOy7YO",
        "outputId": "59ca675e-e595-49be-f967-7d3136ee68aa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('[start] politicians do not have permission to do what needs to be done [end]',\n",
              " '[start] राजनीतिज्ञों के पास जो कार्य करना चाहिए, वह करने कि अनुमति नहीं है . [end]')"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKSTu2wW4uDh"
      },
      "source": [
        "## Prepare dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GNsjfotokWRn",
        "outputId": "aa7c8e30-f07a-405c-e68d-ec327e808404"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train: 46252\n",
            "Validation: 19822\n"
          ]
        }
      ],
      "source": [
        "# Create train, val and testing datasets\n",
        "val_split = 0.30\n",
        "\n",
        "len_text_pairs = len(text_pairs)\n",
        "n_val_index = int(val_split * len_text_pairs)\n",
        "n_train_index = len_text_pairs - n_val_index\n",
        "\n",
        "train_tp = text_pairs[:n_train_index]\n",
        "val_tp = text_pairs[n_train_index:]\n",
        "print(f\"Train: {len(train_tp)}\")\n",
        "print(f\"Validation: {len(val_tp)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ftsUg6HRYi7q",
        "outputId": "53d2c75e-19b3-4529-d009-a8ec2efe17fe"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('[start] politicians do not have permission to do what needs to be done [end]',\n",
              "  '[start] राजनीतिज्ञों के पास जो कार्य करना चाहिए, वह करने कि अनुमति नहीं है . [end]')]"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "train_tp[:1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 452
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:15.479987Z",
          "iopub.status.busy": "2023-10-22T04:43:15.479579Z",
          "iopub.status.idle": "2023-10-22T04:43:17.993364Z",
          "shell.execute_reply": "2023-10-22T04:43:17.992391Z",
          "shell.execute_reply.started": "2023-10-22T04:43:15.479930Z"
        },
        "id": "agg-7Oc2ONKB",
        "outputId": "c565003c-1e0e-4f9f-daa2-d1879db27621",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj4AAAGzCAYAAAAv9B03AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8OUlEQVR4nO3de1xVVf7/8fcBBAQBxQuIKHhXRA8malp+vVEOFZM6ppMZSPPNR4Wlknabr9oV09JxmiHJLmr1dWRy1MpMM1Ipo7xi9qVMjczRBJm8AY9EOev3R3F+IqCCwFH36/l48Hh01l5nr8/eYLxZe+2zbcYYIwAAAAtwc3UBAAAA9YXgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgA1zlNm7cKJvNpo0bN7q6FNShH374QTabTS+++KKrS5EkhYeHa/z48a4u46LGjx+vRo0auboMXEEIPrimLV68WDabrcqvL774wtUlog4cPnxYTz75pLKzs11aR1lYuZSvH374waW1Xs2Ki4v15JNPEv5xSTxcXQBQH55++mm1bdu2QnuHDh1cUA3q2uHDh/XUU08pPDxcUVFRLqujefPmeuutt8q1zZ07V//+97/1l7/8pUJf1ExxcbGeeuopSdKgQYNcWwyueAQfWEJsbKyio6NdXQYsxtfXV+PGjSvXtmzZMh07dqxCO4D6waUuQNLMmTPl5uamjIyMcu0TJkyQp6endu3aJUkqKSnRjBkz1KtXLwUEBMjX11cDBgzQhg0byr3v3PUYqampateunXx8fHTzzTfr4MGDMsbomWeeUWhoqBo2bKjbb79dP//8c7l9hIeH67bbbtNHH32kqKgoeXt7KyIiQitWrLikY/ryyy/1u9/9TgEBAfLx8dHAgQO1efPmcn1OnTqlyZMnKzw8XF5eXmrRooVuuukm7dix46L7P3TokP70pz8pJCREXl5eatu2re6//36VlJQ4+3z//fe64447FBgYKB8fH11//fX64IMPyu2n7HLk+Zd6Klu7NGjQIEVGRionJ0eDBw+Wj4+PWrVqpTlz5pR7X+/evSVJiYmJzktJixcvrvQ4li9fLpvNpk2bNlXY9sorr8hms+nrr7+WJB05ckSJiYkKDQ2Vl5eXWrZsqdtvv/2yL1Pl5+frT3/6k4KCguTt7S273a4lS5Zc9H3GGOfP6Lk/F2+//bZ69eqlhg0bKjAwUH/84x918ODBcu+9lHNZXcePH9fkyZPVunVreXl5qUOHDpo9e7YcDoezz7n/NhYuXKj27dvLy8tLvXv31tatWyvs85133lFERIS8vb0VGRmplStXavz48QoPD3fur2y27KmnnnJ+v5988sly+zl06JCGDx+uRo0aqXnz5po6dapKS0trfKy4ihngGrZo0SIjyXz88cfm6NGj5b4KCgqc/UpKSkzPnj1NWFiYOXnypDHGmLVr1xpJ5plnnnH2O3r0qGnZsqVJTk42CxYsMHPmzDGdO3c2DRo0MDt37nT2y83NNZJMVFSUiYiIMPPmzTP/8z//Yzw9Pc31119vnnjiCdO/f3/z0ksvmYceesjYbDaTmJhYrvawsDDTqVMn07hxY/PYY4+ZefPmme7duxs3Nzfz0UcfOftt2LDBSDIbNmxwtmVkZBhPT0/Tr18/M3fuXPOXv/zF9OjRw3h6epovv/zS2W/s2LHG09PTJCcnm9dee83Mnj3bxMXFmbfffvuC5/XQoUMmJCTE+Pj4mMmTJ5u0tDQzffp007VrV3Ps2DFjjDFHjhwxQUFBxs/Pz/z5z3828+bNM3a73bi5uZkVK1ZU+B7l5uaWG6Oy4xo4cKAJCQkxrVu3NpMmTTIvv/yyGTJkiJFk1qxZ4xz36aefNpLMhAkTzFtvvWXeeusts3///kqPpbi42DRq1Mg88MADFbYNHjzYdOvWzfm6f//+JiAgwPzP//yPee2110xKSooZPHiw2bRp0wXP17luvfVWExYWVm78rl27mgYNGpgpU6aYl156yQwYMMBIMvPnz3f2K/uZeuGFF4wxxpw9e9bEx8cbLy8vs3r1ame/Z5991thsNjNmzBjz8ssvm6eeeso0a9bMhIeHO783l3ouLyQsLMwkJCQ4XxcVFZkePXqYpk2bmieeeMKkpaWZ+Ph4Y7PZzKRJkyocR8+ePU2HDh3M7NmzzZw5c0yzZs1MaGioKSkpcfZdvXq1sdlspkePHmbevHlm+vTppkmTJiYyMtJ5DgsLC82CBQuMJDNixAjn93vXrl3GGGMSEhKMt7e36datm7nnnnvMggULzB/+8Acjybz88suX8i3DNYbgg2ta2S/Vyr68vLzK9d29e7fx9PQ0//3f/22OHTtmWrVqZaKjo82ZM2ecfc6ePWtOnz5d7n3Hjh0zQUFB5p577nG2lf3PvXnz5ub48ePO9scff9xIMna7vdx+77zzTuPp6Wl++eUXZ1tYWJiRZP71r385206cOGFatmxpevbs6Ww7PyA4HA7TsWNHM2zYMONwOJz9iouLTdu2bc1NN93kbAsICDBJSUmXfD7LxMfHGzc3N7N169YK28rGnDx5spFkPv30U+e2U6dOmbZt25rw8HBTWlpqjKl+8JFk3nzzTWfb6dOnTXBwsPnDH/7gbNu6dauRZBYtWnRJx3PnnXeaFi1amLNnzzrbfvrpJ+Pm5maefvppY8yv3+dzg0dNnR985s+fbySVC5slJSWmX79+plGjRs4gfm7wOXPmjBkzZoxp2LChWbdunfN9P/zwg3F3dzfPPfdcuTF3795tPDw8yrVf6rmsyvnB55lnnjG+vr7mu+++K9fvscceM+7u7ubHH38sdxxNmzY1P//8s7Pfu+++aySZ999/39nWvXt3Exoaak6dOuVs27hxo5FU7hwePXrUSDIzZ86sUGdCQoKR5Pw+lunZs6fp1avXRY8T1x4udcESUlNTtX79+nJfH374Ybk+kZGReuqpp/Taa69p2LBhKigo0JIlS+Th8f+Xwrm7u8vT01OS5HA49PPPP+vs2bOKjo6u9PLQHXfcoYCAAOfrvn37SpLGjRtXbr99+/ZVSUmJDh06VO79ISEhGjFihPO1v7+/4uPjtXPnTh05cqTSY83OztbevXs1duxY/ec//1FBQYEKCgpUVFSkoUOHKjMz03npoXHjxvryyy91+PDhSzqPZce9atUqxcXFVbpuymazSZLWrFmjPn366MYbb3Rua9SokSZMmKAffvhBOTk5lzzmuRo1alRufYynp6f69Omj77//vkb7k6QxY8YoPz+/3GW15cuXy+FwaMyYMZKkhg0bytPTUxs3btSxY8dqPNb51qxZo+DgYN15553OtgYNGuihhx5SYWFhhUtwJSUluuOOO7R69WqtWbNGN998s3PbihUr5HA4NHr0aOf3vaCgQMHBwerYsWOFS7K1eS7feecdDRgwQE2aNCk3dkxMjEpLS5WZmVmu/5gxY9SkSRPn6wEDBkiSc+zDhw9r9+7dio+PL3c7+sCBA9W9e/dq13ffffeVez1gwIDL+pnB1YvFzbCEPn36XNLi5mnTpmnZsmXasmWLUlJSFBERUaHPkiVLNHfuXH377bc6c+aMs72yu8batGlT7nVZCGrdunWl7ef/Qu3QoYMzSJTp1KmTpF/XNgQHB1cYc+/evZKkhISEyg9S0okTJ9SkSRPNmTNHCQkJat26tXr16qVbbrlF8fHxateuXZXvPXr0qE6ePKnIyMgq+0jSgQMHnEHvXF27dnVuv9g+KhMaGlrhnDRp0kRfffVVtfdVpmwtVHp6uoYOHSpJSk9PV1RUlPN8e3l5afbs2Xr44YcVFBSk66+/Xrfddpvi4+Mr/T5cqgMHDqhjx45ycyv/d+i55+lcs2bNUmFhoT788MMKdzDt3btXxhh17Nix0rEaNGhQ7nVtnsu9e/fqq6++qvLutPz8/HKvz/+3URaCyv4NlB13ZXdedujQ4ZLWoZXx9vauUFeTJk1qNcDi6kHwAc7x/fffO4PD7t27K2x/++23NX78eA0fPlzTpk1TixYt5O7urlmzZmn//v0V+ru7u1c6TlXtxpjLqP5XZbM5L7zwQpW3cpf9BT169GgNGDBAK1eu1EcffaQXXnhBs2fP1ooVKxQbG3vZtVyK83/xlqlq4WldnDsvLy8NHz5cK1eu1Msvv6y8vDxt3rxZKSkp5fpNnjxZcXFxWrVqldatW6fp06dr1qxZ+uSTT9SzZ88aj18dw4YN09q1azVnzhwNGjRI3t7ezm0Oh0M2m00ffvhhpefp/A/yq81z6XA4dNNNN+mRRx6pdHtZgKyLsS+mqrFgTQQf4DcOh0Pjx4+Xv7+/Jk+erJSUFI0aNUojR4509lm+fLnatWunFStWlPuFPXPmzDqpad++fTLGlBvru+++kyTnXS3na9++vaRfL4vFxMRcdIyWLVvqgQce0AMPPKD8/Hxdd911eu6556oMPs2bN5e/v7/zTqeqhIWFac+ePRXav/32W+d26f//pX/8+PFy/c6f6aiOqsLUhYwZM0ZLlixRRkaGvvnmGxljnJe5ztW+fXs9/PDDevjhh7V3715FRUVp7ty5evvtt2tUa1hYmL766is5HI5ysz7nn6cy119/ve677z7ddtttuuOOO7Ry5UrnZdP27dvLGKO2bdtWCBp1rX379iosLLykn7lLUXbc+/btq7Dt/LaafL9hXazxAX4zb948ff7551q4cKGeeeYZ9e/fX/fff78KCgqcfcr+cjz3r9Ivv/xSWVlZdVLT4cOHtXLlSufrkydP6s0331RUVFSVl1d69eql9u3b68UXX1RhYWGF7UePHpX064zKiRMnym1r0aKFQkJCdPr06SprcnNz0/Dhw/X+++9r27ZtFbaXnZtbbrlFW7ZsKXduioqKtHDhQoWHhzsvI5YFtXPXgJSWlmrhwoVV1nAxvr6+kiqGqQuJiYlRYGCg0tPTlZ6erj59+pS7fFlcXKxffvml3Hvat28vPz+/C56vi7nlllt05MgRpaenO9vOnj2rv/3tb2rUqJEGDhxYaa3Lli3T2rVrdffddztn+UaOHCl3d3c99dRTFWZOjDH6z3/+U+M6L2b06NHKysrSunXrKmw7fvy4zp49W639hYSEKDIyUm+++Wa5n+NNmzZVmI318fFxjgNcDDM+sIQPP/zQ+Rf0ufr376927drpm2++0fTp0zV+/HjFxcVJ+vXzZaKiovTAAw/on//8pyTptttu04oVKzRixAjdeuutys3NVVpamiIiIioNGZerU6dO+tOf/qStW7cqKChIb7zxhvLy8rRo0aIq3+Pm5qbXXntNsbGx6tatmxITE9WqVSsdOnRIGzZskL+/v95//32dOnVKoaGhGjVqlOx2uxo1aqSPP/5YW7du1dy5cy9YV0pKij766CMNHDhQEyZMUNeuXfXTTz/pnXfe0WeffabGjRvrscce0z/+8Q/FxsbqoYceUmBgoJYsWaLc3Fz961//cs5udOvWTddff70ef/xx/fzzzwoMDNSyZcuq/YvyXO3bt1fjxo2VlpYmPz8/+fr6qm/fvpWuwyrToEEDjRw5UsuWLVNRUVGFZ2J99913Gjp0qEaPHq2IiAh5eHho5cqVysvL0x//+Mca1zphwgS98sorGj9+vLZv367w8HAtX75cmzdv1vz58+Xn51fp+4YPH65FixYpPj5e/v7+euWVV9S+fXs9++yzevzxx/XDDz9o+PDh8vPzU25urlauXKkJEyZo6tSpNa71QqZNm6b33ntPt912m8aPH69evXqpqKhIu3fv1vLly/XDDz+oWbNm1dpnSkqKbr/9dt1www1KTEzUsWPH9Pe//12RkZHl/r01bNhQERERSk9PV6dOnRQYGKjIyMgarSGDBbjmZjKgflzodnb9drvz2bNnTe/evU1oaGi5W8+NMeavf/2rkWTS09ONMb/eqp2SkmLCwsKMl5eX6dmzp1m9erVJSEgod3vt+Z+5UqbsFu133nmn0jrPvT08LCzM3HrrrWbdunWmR48exsvLy3Tp0qXCeyu77dsYY3bu3GlGjhxpmjZtary8vExYWJgZPXq0ycjIMMb8euvytGnTjN1uN35+fsbX19fY7fZL/myTAwcOmPj4eNO8eXPj5eVl2rVrZ5KSksrd7r9//34zatQo07hxY+Pt7W369OlT7jNnzu0XExNjvLy8TFBQkHniiSfM+vXrK72d/dzP1Slz/vk35tfboyMiIoyHh8cl39peNqbNZjMHDx4st62goMAkJSWZLl26GF9fXxMQEGD69u1r/vnPf150v+c6/3Z2Y4zJy8sziYmJplmzZsbT09N07969Qr1V/Uy9/PLLRpKZOnWqs+1f//qXufHGG42vr6/x9fU1Xbp0MUlJSWbPnj3OPtU5l5U5/3Z2Y379uILHH3/cdOjQwXh6eppmzZqZ/v37mxdffNH5+TxVHYcxptJb0pctW2a6dOlivLy8TGRkpHnvvffMH/7wB9OlS5dy/T7//HPTq1cv4+npWW4/CQkJxtfXt8JYM2fONPwKtCabMXWwkgzAZQsPD1dkZKRWr17t6lKAK0pUVJSaN2+u9evXu7oUXIVY4wMAuCKdOXOmwiXPjRs3ateuXTyMFDXGGh8AwBXp0KFDiomJ0bhx4xQSEqJvv/1WaWlpCg4OrvCBhMClIvgAAK5ITZo0Ua9evfTaa6/p6NGj8vX11a233qrnn39eTZs2dXV5uEqxxgcAAFgGa3wAAIBlEHwAAIBlsMbnPA6HQ4cPH5afnx8fgw4AwFXCGKNTp04pJCSkwkN/z0XwOc/hw4crPDkbAABcHQ4ePKjQ0NAqtxN8zlP28fAHDx6Uv7+/i6sBAODaUHS0SCEdfn2O3uF9RfJt7lur+z958qRat25d5WNeyhB8zlN2ecvf35/gAwBALXH/xV3Sr2HH389dvv61G3zKXGyZCoubAQCAZRB8AACAZRB8AACAZbDGp5qMMTp79qxKS0tdXYrLuLu7y8PDg9v9AQBXHYJPNZSUlOinn35ScXGxq0txOR8fH7Vs2VKenp6uLgUAgEtG8LlEDodDubm5cnd3V0hIiDw9PS0542GMUUlJiY4eParc3Fx17Njxgh8UBQDAlYTgc4lKSkrkcDjUunVr+fj4uLocl2rYsKEaNGigAwcOqKSkRN7e3q4uCQCAS8Kf6tXE7MavOA8AgKsRv70AAIBlXJPBp7i4WGFhYZo6daqrSwEAAFeQa3KNz3PPPafrr7++/gZcvrz+xpKkUaPqdzwAAK4R19yMz969e/Xtt98qNjbW1aUAAIArzBUVfDIzMxUXF6eQkBDZbDatWrWqQp/U1FSFh4fL29tbffv21ZYtW8ptnzp1qmbNmlVPFQMAgKvJFXWpq6ioSHa7Xffcc49GjhxZYXt6erqSk5OVlpamvn37av78+Ro2bJj27NmjFi1a6N1331WnTp3UqVMnff755y44giuXw+HQ7NmztXDhQh05ckSdOnXS9OnTNWrUKG3cuFGDBw/Wxx9/rEcffVQ5OTmKiorSokWL1LlzZ1eXDtSu+r40XRu4vA3Umisq+MTGxl7wEtW8efN07733KjExUZKUlpamDz74QG+88YYee+wxffHFF1q2bJneeecdFRYW6syZM/L399eMGTOq3Ofp06d1+vRp5+uTJ0/W3gFdQWbNmqW3335baWlp6tixozIzMzVu3Dg1b97c2efPf/6z5s6dq+bNm+u+++7TPffco82bN7uwagAAatcVdanrQkpKSrR9+3bFxMQ429zc3BQTE6OsrCxJv/5yP3jwoH744Qe9+OKLuvfeey8YesreExAQ4Pxq3bp1nR6HK5w+fVopKSl64403NGzYMLVr107jx4/XuHHj9Morrzj7Pffccxo4cKAiIiL02GOP6fPPP9cvv/ziwsoBAKhdV03wKSgoUGlpqYKCgsq1BwUF6ciRIzXe7+OPP64TJ044vw4ePHi5pV5x9u3bp+LiYt10001q1KiR8+vNN9/U/v37nf169Ojh/O+WLVtKkvLz8+u9XgAA6soVdamrNo0fP/6S+nl5ecnLy6tui3GxwsJCSdIHH3ygVq1aldvm5eXlDD8NGjRwtpc9h8zhcNRTlQAA1L2rJvg0a9ZM7u7uysvLK9eel5en4OBgF1V1dYiIiJCXl5d+/PFHDRw4sML2c2d9AAC4ll01wcfT01O9evVSRkaGhg8fLunX2YiMjAxNnDjRtcVd4fz8/DR16lRNmTJFDodDN954o06cOKHNmzfL399fYWFhri4RAIB6cUUFn8LCQu3bt8/5Ojc3V9nZ2QoMDFSbNm2UnJyshIQERUdHq0+fPpo/f76Kioqcd3m5zFVwq+kzzzyj5s2ba9asWfr+++/VuHFjXXfddXriiSe4nAUAsIwrKvhs27ZNgwcPdr5OTk6WJCUkJGjx4sUaM2aMjh49qhkzZujIkSOKiorS2rVrKyx4RkU2m02TJk3SpEmTKt1ujCn3OioqqkIbAABXuysq+AwaNOiiv2wnTpzIpS0AAFAjV83t7AAAAJeL4AMAACyD4AMAACyD4AMAACyD4AMAACyD4AMAACyD4AMAACyD4AMAACzjivoAw6vV8uX1O151n5AxaNAgRUVFaf78+XVSDwAAVwtmfAAAgGUQfAAAgGUQfCzC4XDokUceUWBgoIKDg/Xkk086t9lsNr322msaMWKEfHx81LFjR7333nuuKxYAgDpC8PlNamqqIiIi1Lt3b1eXUieWLFkiX19fffnll5ozZ46efvpprV+/3rn9qaee0ujRo/XVV1/plltu0V133aWff/7ZhRUDAFD7CD6/SUpKUk5OjrZu3erqUupEjx49NHPmTHXs2FHx8fGKjo5WRkaGc/v48eN15513qkOHDkpJSVFhYaG2bNniwooBAKh9BB+L6NGjR7nXLVu2VH5+fqXbfX195e/vX247AADXAoKPRTRo0KDca5vNJofDccnbAQC4FhB8AACAZRB8AACAZfDJzbWgup+kDAAAXIPgYwEbN26s0LZq1SrnfxtjKmw/fvx43RUEAICLcKkLAABYBjM+AHANqu+HJ9cEywTgCsz4AAAAyyD4AAAAyyD4VFNlC4GtiPMAALgaEXwuUdknGxcXF7u4kitD2Xk4/xOfAQC4krG4+RK5u7urcePGzudX+fj4yGazubiq+meMUXFxsfLz89W4cWO5u7u7uiQAAC4ZwacagoODJYmHd0pq3Lix83wAAHC1IPhUg81mU8uWLdWiRQudOXPG1eW4TIMGDZjpAQBclQg+NeDu7s4vfgAArkIsbgYAAJZB8AEAAJZB8AEAAJZB8AEAAJZB8AEAAJZB8PlNamqqIiIi1Lt3b1eXAgAA6gjB5zdJSUnKycnR1q1bXV0KAACoIwQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGR6uLgBA3Vq+3NUVXJpRo1xdAQArYMYHAABYBsEHAABYBsEHAABYBsEHAABYBsHnN6mpqYqIiFDv3r1dXQoAAKgjBJ/fJCUlKScnR1u3bnV1KQAAoI4QfAAAgGUQfAAAgGXwAYYAgNqXlXUJnQ7VeRnVwqdoWgIzPgAAwDIIPgAAwDIIPgAAwDIIPgAAwDIIPgAAwDIIPgAAwDIIPgAAwDIIPgAAwDIIPgAAwDIIPgAAwDIIPgAAwDJ4VheuPcuXu7qC6uMZQQBQL5jxAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlnHNBp/jx48rOjpaUVFRioyM1KuvvurqkgAAgItdsx9g6Ofnp8zMTPn4+KioqEiRkZEaOXKkmjZt6urSAACAi1yzMz7u7u7y8fGRJJ0+fVrGGBljXFwVAABwpRoFn0OHDmncuHFq2rSpGjZsqO7du2vbtm21VlRmZqbi4uIUEhIim82mVatWVdovNTVV4eHh8vb2Vt++fbVly5Zy248fPy673a7Q0FBNmzZNzZo1q7UaAQDA1afawefYsWO64YYb1KBBA3344YfKycnR3Llz1aRJk0r7b968WWfOnKnQnpOTo7y8vErfU1RUJLvdrtTU1CrrSE9PV3JysmbOnKkdO3bIbrdr2LBhys/Pd/Zp3Lixdu3apdzcXC1durTK8QAAgDVUO/jMnj1brVu31qJFi9SnTx+1bdtWN998s9q3b1+hr8PhUFJSksaOHavS0lJn+549ezRkyBAtWbKk0jFiY2P17LPPasSIEVXWMW/ePN17771KTExURESE0tLS5OPjozfeeKNC36CgINntdn366adV7i81NVURERHq3bv3hQ4fAABcxaodfN577z1FR0frjjvuUIsWLdSzZ88q75hyc3PTmjVrtHPnTsXHx8vhcGj//v0aMmSIhg8frkceeaRGRZeUlGj79u2KiYkpN1ZMTIyysrIkSXl5eTp16pQk6cSJE8rMzFTnzp2r3GdSUpJycnK0devWGtUEAACufNUOPt9//70WLFigjh07at26dbr//vv10EMPVTl7ExISok8++USfffaZxo4dqyFDhigmJkYLFiyocdEFBQUqLS1VUFBQufagoCAdOXJEknTgwAENGDBAdrtdAwYM0IMPPqju3bvXeEwAAHD1q/bt7A6HQ9HR0UpJSZEk9ezZU19//bXS0tKUkJBQ6XvatGmjt956SwMHDlS7du30+uuvy2azXV7lF9GnTx9lZ2fX6RgAAODqUu0Zn5YtWyoiIqJcW9euXfXjjz9W+Z68vDxNmDBBcXFxKi4u1pQpU6pf6TmaNWsmd3f3CouV8/LyFBwcfFn7BgAA165qB58bbrhBe/bsKdf23XffKSwsrNL+BQUFGjp0qLp27aoVK1YoIyND6enpmjp1as0qluTp6alevXopIyPD2eZwOJSRkaF+/frVeL8AAODaVu1LXVOmTFH//v2VkpKi0aNHa8uWLVq4cKEWLlxYoa/D4VBsbKzCwsKUnp4uDw8PRUREaP369RoyZIhatWpV6exPYWGh9u3b53ydm5ur7OxsBQYGqk2bNpKk5ORkJSQkKDo6Wn369NH8+fNVVFSkxMTE6h4SAACwiGoHn969e2vlypV6/PHH9fTTT6tt27aaP3++7rrrrgp93dzclJKSogEDBsjT09PZbrfb9fHHH6t58+aVjrFt2zYNHjzY+To5OVmSlJCQoMWLF0uSxowZo6NHj2rGjBk6cuSIoqKitHbt2goLngEAAMrU6Fldt912m2677bZL6nvTTTdV2t6zZ88q3zNo0KBLerzExIkTNXHixEuqAwAA4Jp9VhcAAMD5CD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyPFxdAAAAuHzLl7u6ggs7c9zVFfyK4POb1NRUpaamqrS01NWlAFeHrKxa3uGhWt4fAFTEpa7fJCUlKScnR1u3bnV1KQAAoI4w42MhV/o0aJlRo1xdAQDgWsWMDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAyCDwAAsAwPVxcAALiI5cur/56sVrVfB3ANYMYHAABYBsEHAABYBsEHAABYBmt8gCtBTdZwXCrWegCAEzM+AADAMpjxAXBFWM7MFIB6wIwPAACwDIIPAACwDIIPAACwDIIPAACwDIIPAACwDIIPAACwDIIPAACwDIIPAACwDIIPAACwDIIPAACwjGs2+Bw/flzR0dGKiopSZGSkXn31VVeXBAAAXOyafVaXn5+fMjMz5ePjo6KiIkVGRmrkyJFq2rSpq0sDAAAucs3O+Li7u8vHx0eSdPr0aRljZIxxcVUAAMCVLiv4PP/887LZbJo8eXItlfOrzMxMxcXFKSQkRDabTatWraq0X2pqqsLDw+Xt7a2+fftqy5Yt5bYfP35cdrtdoaGhmjZtmpo1a1ardQIAgKtLjYPP1q1b9corr6hHjx4X7Ld582adOXOmQntOTo7y8vIqfU9RUZHsdrtSU1Or3G96erqSk5M1c+ZM7dixQ3a7XcOGDVN+fr6zT+PGjbVr1y7l5uZq6dKlVY4HAACsoUbBp7CwUHfddZdeffVVNWnSpMp+DodDSUlJGjt2rEpLS53te/bs0ZAhQ7RkyZJK3xcbG6tnn31WI0aMqHLf8+bN07333qvExERFREQoLS1NPj4+euONNyr0DQoKkt1u16efflqNowQAANeaGgWfpKQk3XrrrYqJibnwzt3ctGbNGu3cuVPx8fFyOBzav3+/hgwZouHDh+uRRx6pUdElJSXavn17ufHd3NwUExOjrKwsSVJeXp5OnTolSTpx4oQyMzPVuXPnKveZmpqqiIgI9e7du0Y1AQCAK1+17+patmyZduzYoa1bt15S/5CQEH3yyScaMGCAxo4dq6ysLMXExGjBggXVLrZMQUGBSktLFRQUVK49KChI3377rSTpwIEDmjBhgnNR84MPPqju3btXuc+kpCQlJSXp5MmTCggIqHFtAADgylWt4HPw4EFNmjRJ69evl7e39yW/r02bNnrrrbc0cOBAtWvXTq+//rpsNlu1i62OPn36KDs7u07HAAAAV5dqXeravn278vPzdd1118nDw0MeHh7atGmTXnrpJXl4eJRbx3OuvLw8TZgwQXFxcSouLtaUKVMuq+hmzZrJ3d29wmLlvLw8BQcHX9a+AQDAtatawWfo0KHavXu3srOznV/R0dG66667lJ2dLXd39wrvKSgo0NChQ9W1a1etWLFCGRkZSk9P19SpU2tctKenp3r16qWMjAxnm8PhUEZGhvr161fj/QIAgGtbtS51+fn5KTIyslybr6+vmjZtWqFd+jWMxMbGKiwsTOnp6fLw8FBERITWr1+vIUOGqFWrVpXO/hQWFmrfvn3O17m5ucrOzlZgYKDatGkjSUpOTlZCQoKio6PVp08fzZ8/X0VFRUpMTKzOIQEAAAup00dWuLm5KSUlRQMGDJCnp6ez3W636+OPP1bz5s0rfd+2bds0ePBg5+vk5GRJUkJCghYvXixJGjNmjI4ePaoZM2boyJEjioqK0tq1aysseAYAAChz2cFn48aNF9x+0003Vdres2fPKt8zaNCgS3q8xMSJEzVx4sSL9gMAAJCu4Wd1AQAAnI/gAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALKNOH1kBAEBVlme1cnUJl2TUKFdXgNrEjA8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMgg8AALAMns4OAMAFLF/u6gpQm5jxAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlkHwAQAAlnHNBp/jx48rOjpaUVFRioyM1KuvvurqkgAAgIt5uLqAuuLn56fMzEz5+PioqKhIkZGRGjlypJo2berq0gAAgItcszM+7u7u8vHxkSSdPn1axhgZY1xcFQAAcKVqB58FCxaoR48e8vf3l7+/v/r166cPP/ywVovKzMxUXFycQkJCZLPZtGrVqkr7paamKjw8XN7e3urbt6+2bNlSbvvx48dlt9sVGhqqadOmqVmzZrVaJwAAuLpUO/iEhobq+eef1/bt27Vt2zYNGTJEt99+u/7v//6v0v6bN2/WmTNnKrTn5OQoLy+v0vcUFRXJbrcrNTW1yjrS09OVnJysmTNnaseOHbLb7Ro2bJjy8/OdfRo3bqxdu3YpNzdXS5curXI8AABgDdUOPnFxcbrlllvUsWNHderUSc8995waNWqkL774okJfh8OhpKQkjR07VqWlpc72PXv2aMiQIVqyZEmlY8TGxurZZ5/ViBEjqqxj3rx5uvfee5WYmKiIiAilpaXJx8dHb7zxRoW+QUFBstvt+vTTT6vcX2pqqiIiItS7d+8LHT4AALiKXdYan9LSUi1btkxFRUXq169fxZ27uWnNmjXauXOn4uPj5XA4tH//fg0ZMkTDhw/XI488UqNxS0pKtH37dsXExJQbKyYmRllZWZKkvLw8nTp1SpJ04sQJZWZmqnPnzlXuMykpSTk5Odq6dWuNagIAAFe+Gt3VtXv3bvXr10+//PKLGjVqpJUrVyoiIqLSviEhIfrkk080YMAAjR07VllZWYqJidGCBQtqXHRBQYFKS0sVFBRUrj0oKEjffvutJOnAgQOaMGGCc1Hzgw8+qO7du9d4TAAAcPWrUfDp3LmzsrOzdeLECS1fvlwJCQnatGlTleGnTZs2euuttzRw4EC1a9dOr7/+umw222UVfjF9+vRRdnZ2nY4BAACuLjW61OXp6akOHTqoV69emjVrlux2u/76179W2T8vL08TJkxQXFyciouLNWXKlBoXLEnNmjWTu7t7hcXKeXl5Cg4Ovqx9AwCAa1etfI6Pw+HQ6dOnK91WUFCgoUOHqmvXrlqxYoUyMjKUnp6uqVOn1ng8T09P9erVSxkZGeVqyMjIqHStEQAAgFSDS12PP/64YmNj1aZNG506dUpLly7Vxo0btW7dugp9HQ6HYmNjFRYWpvT0dHl4eCgiIkLr16/XkCFD1KpVq0pnfwoLC7Vv3z7n69zcXGVnZyswMFBt2rSRJCUnJyshIUHR0dHq06eP5s+fr6KiIiUmJlb3kAAAgEVUO/jk5+crPj5eP/30kwICAtSjRw+tW7dON910U4W+bm5uSklJ0YABA+Tp6elst9vt+vjjj9W8efNKx9i2bZsGDx7sfJ2cnCxJSkhI0OLFiyVJY8aM0dGjRzVjxgwdOXJEUVFRWrt2bYUFzwAAAGVshuc4lHPy5EkFBAToxIkT8vf3d3U5tWr5cldXcGlGjbrMHVwtB1pPlme1cnUJwNWBpRJ16szxIo2911eSVJhXJN8WvrW6/0v9/X3NPqsLAADgfAQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGQQfAABgGdV+VhcsJivLBYMecsGYAAArYMYHAABYBsEHAABYBsEHAABYBsEHAABYBsEHAABYBsEHAABYBsEHAABYBsEHAABYBh9gCNTQ8qxWri4BAFBNzPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADL8HB1AZayfLlrx89q5drxAQBwMWZ8AACAZRB8AACAZRB8AACAZRB8AACAZRB8AACAZRB8AACAZRB8AACAZRB8AACAZRB8AACAZfDJzQAASFJWlqsrqL5+/VxdwVWHGR8AAGAZzPjgirOcZ4oBAOoIMz4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyCD4AAMAyrtngc/z4cUVHRysqKkqRkZF69dVXXV0SAABwsWv2WV1+fn7KzMyUj4+PioqKFBkZqZEjR6pp06auLg0AALjINTvj4+7uLh8fH0nS6dOnZYyRMcbFVQEAAFeqdvCZNWuWevfuLT8/P7Vo0ULDhw/Xnj17arWozMxMxcXFKSQkRDabTatWraq0X2pqqsLDw+Xt7a2+fftqy5Yt5bYfP35cdrtdoaGhmjZtmpo1a1ardQIAgKtLtYPPpk2blJSUpC+++ELr16/XmTNndPPNN6uoqKjS/ps3b9aZM2cqtOfk5CgvL6/S9xQVFclutys1NbXKOtLT05WcnKyZM2dqx44dstvtGjZsmPLz8519GjdurF27dik3N1dLly6tcjwAAGAN1Q4+a9eu1fjx49WtWzfZ7XYtXrxYP/74o7Zv316hr8PhUFJSksaOHavS0lJn+549ezRkyBAtWbKk0jFiY2P17LPPasSIEVXWMW/ePN17771KTExURESE0tLS5OPjozfeeKNC36CgINntdn366adV7i81NVURERHq3bv3hQ4fAABcxS57jc+JEyckSYGBgRV37uamNWvWaOfOnYqPj5fD4dD+/fs1ZMgQDR8+XI888kiNxiwpKdH27dsVExNTbqyYmBhlZWVJkvLy8nTq1ClnjZmZmercuXOV+0xKSlJOTo62bt1ao5oAAMCV77Lu6nI4HJo8ebJuuOEGRUZGVtonJCREn3zyiQYMGKCxY8cqKytLMTExWrBgQY3HLSgoUGlpqYKCgsq1BwUF6dtvv5UkHThwQBMmTHAuan7wwQfVvXv3Go8JAACufpcVfJKSkvT111/rs88+u2C/Nm3a6K233tLAgQPVrl07vf7667LZbJcz9EX16dNH2dnZdToGAAC4utT4UtfEiRO1evVqbdiwQaGhoRfsm5eXpwkTJiguLk7FxcWaMmVKTYeVJDVr1kzu7u4VFivn5eUpODj4svYNAACuXdUOPsYYTZw4UStXrtQnn3yitm3bXrB/QUGBhg4dqq5du2rFihXKyMhQenq6pk6dWuOiPT091atXL2VkZDjbHA6HMjIy1K9fvxrvFwAAXNuqfakrKSlJS5cu1bvvvis/Pz8dOXJEkhQQEKCGDRuW6+twOBQbG6uwsDClp6fLw8NDERERWr9+vYYMGaJWrVpVOvtTWFioffv2OV/n5uYqOztbgYGBatOmjSQpOTlZCQkJio6OVp8+fTR//nwVFRUpMTGxuocEAAAsotrBp2xR8qBBg8q1L1q0SOPHjy/X5ubmppSUFA0YMECenp7Odrvdro8//ljNmzevdIxt27Zp8ODBztfJycmSpISEBC1evFiSNGbMGB09elQzZszQkSNHFBUVpbVr11ZY8AwAAFDGZniOQzknT55UQECATpw4IX9//9rd+fLltbu/6g6f1cql4wMAatlVtLzjzPEijb3XV5JUmFck3xa+tbr/S/39fc0+qwsAAOB8BB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZBB8AAGAZHq4uAAAA1FBWlqsruHTFpyUNcnUVzPgAAADrIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADLIPgAAADL8HB1AVcaY4wk6eTJk7W/8+Li2t9ndYY/XeTS8QEA1nWm5LSkX3+3njxVpFLv0lrdf9nv7bLf41Uh+Jzn1KlTkqTWrVu7uBIAAK5NIR3qbt+nTp1SQEBAldtt5mLRyGIcDocOHz4sPz8/2Wy2WtvvyZMn1bp1ax08eFD+/v61tt+rhdWPX+IcWP34Jc6B1Y9f4hzU5fEbY3Tq1CmFhITIza3qlTzM+JzHzc1NoaGhdbZ/f39/S/6wl7H68UucA6sfv8Q5sPrxS5yDujr+C830lGFxMwAAsAyCDwAAsAyCTz3x8vLSzJkz5eXl5epSXMLqxy9xDqx+/BLnwOrHL3EOroTjZ3EzAACwDGZ8AACAZRB8AACAZRB8AACAZRB8AACAZRB8AACAZRB86tisWbPUu3dv+fn5qUWLFho+fLj27Nnj6rJc5vnnn5fNZtPkyZNdXUq9OXTokMaNG6emTZuqYcOG6t69u7Zt2+bqsupNaWmppk+frrZt26phw4Zq3769nnnmmYs+SPBqlZmZqbi4OIWEhMhms2nVqlXlthtjNGPGDLVs2VINGzZUTEyM9u7d65pi68iFzsGZM2f06KOPqnv37vL19VVISIji4+N1+PBh1xVcyy72M3Cu++67TzabTfPnz6+3+urDpZyDb775Rr///e8VEBAgX19f9e7dWz/++GOd10bwqWObNm1SUlKSvvjiC61fv15nzpzRzTffrKIi6z0pfevWrXrllVfUo0cPV5dSb44dO6YbbrhBDRo00IcffqicnBzNnTtXTZo0cXVp9Wb27NlasGCB/v73v+ubb77R7NmzNWfOHP3tb39zdWl1oqioSHa7XampqZVunzNnjl566SWlpaXpyy+/lK+vr4YNG6ZffvmlniutOxc6B8XFxdqxY4emT5+uHTt2aMWKFdqzZ49+//vfu6DSunGxn4EyK1eu1BdffKGQkJB6qqz+XOwc7N+/XzfeeKO6dOmijRs36quvvtL06dPl7e1d98UZ1Kv8/HwjyWzatMnVpdSrU6dOmY4dO5r169ebgQMHmkmTJrm6pHrx6KOPmhtvvNHVZbjUrbfeau65555ybSNHjjR33XWXiyqqP5LMypUrna8dDocJDg42L7zwgrPt+PHjxsvLy/zjH/9wQYV17/xzUJktW7YYSebAgQP1U1Q9qur4//3vf5tWrVqZr7/+2oSFhZm//OUv9V5bfansHIwZM8aMGzfOJfUw41PPTpw4IUkKDAx0cSX1KykpSbfeeqtiYmJcXUq9eu+99xQdHa077rhDLVq0UM+ePfXqq6+6uqx61b9/f2VkZOi7776TJO3atUufffaZYmNjXVxZ/cvNzdWRI0fK/TsICAhQ3759lZWV5cLKXOvEiROy2Wxq3Lixq0upFw6HQ3fffbemTZumbt26ubqceudwOPTBBx+oU6dOGjZsmFq0aKG+ffte8JJgbSL41COHw6HJkyfrhhtuUGRkpKvLqTfLli3Tjh07NGvWLFeXUu++//57LViwQB07dtS6det0//3366GHHtKSJUtcXVq9eeyxx/THP/5RXbp0UYMGDdSzZ09NnjxZd911l6tLq3dHjhyRJAUFBZVrDwoKcm6zml9++UWPPvqo7rzzTss8rXz27Nny8PDQQw895OpSXCI/P1+FhYV6/vnn9bvf/U4fffSRRowYoZEjR2rTpk11Pr5HnY8Ap6SkJH399df67LPPXF1KvTl48KAmTZqk9evX18+12yuMw+FQdHS0UlJSJEk9e/bU119/rbS0NCUkJLi4uvrxz3/+U//7v/+rpUuXqlu3bsrOztbkyZMVEhJimXOAyp05c0ajR4+WMUYLFixwdTn1Yvv27frrX/+qHTt2yGazubocl3A4HJKk22+/XVOmTJEkRUVF6fPPP1daWpoGDhxYp+Mz41NPJk6cqNWrV2vDhg0KDQ11dTn1Zvv27crPz9d1110nDw8PeXh4aNOmTXrppZfk4eGh0tJSV5dYp1q2bKmIiIhybV27dq2XOxeuFNOmTXPO+nTv3l133323pkyZYskZwODgYElSXl5eufa8vDznNqsoCz0HDhzQ+vXrLTPb8+mnnyo/P19t2rRx/j/xwIEDevjhhxUeHu7q8upFs2bN5OHh4bL/NzLjU8eMMXrwwQe1cuVKbdy4UW3btnV1SfVq6NCh2r17d7m2xMREdenSRY8++qjc3d1dVFn9uOGGGyp8fMF3332nsLAwF1VU/4qLi+XmVv5vLHd3d+dffVbStm1bBQcHKyMjQ1FRUZKkkydP6ssvv9T999/v2uLqUVno2bt3rzZs2KCmTZu6uqR6c/fdd1dY6zhs2DDdfffdSkxMdFFV9cvT01O9e/d22f8bCT51LCkpSUuXLtW7774rPz8/53X8gIAANWzY0MXV1T0/P78K65l8fX3VtGlTS6xzmjJlivr376+UlBSNHj1aW7Zs0cKFC7Vw4UJXl1Zv4uLi9Nxzz6lNmzbq1q2bdu7cqXnz5umee+5xdWl1orCwUPv27XO+zs3NVXZ2tgIDA9WmTRtNnjxZzz77rDp27Ki2bdtq+vTpCgkJ0fDhw11XdC270Dlo2bKlRo0apR07dmj16tUqLS11/n8xMDBQnp6eriq71lzsZ+D8oNegQQMFBwerc+fO9V1qnbnYOZg2bZrGjBmj//qv/9LgwYO1du1avf/++9q4cWPdF+eSe8ksRFKlX4sWLXJ1aS5jpdvZjTHm/fffN5GRkcbLy8t06dLFLFy40NUl1auTJ0+aSZMmmTZt2hhvb2/Trl078+c//9mcPn3a1aXViQ0bNlT6bz4hIcEY8+st7dOnTzdBQUHGy8vLDB061OzZs8e1RdeyC52D3NzcKv+/uGHDBleXXisu9jNwvmvxdvZLOQevv/666dChg/H29jZ2u92sWrWqXmqzGXONfnwqAADAeVjcDAAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALIPgAwAALOP/AaR+iheZ3wVOAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# histogram of sentence length in tokens\n",
        "en_lengths = [len(e.split()) for e, h in train_tp]\n",
        "hn_lengths = [len(h.split()) for e, h in train_tp]\n",
        "\n",
        "plt.hist(en_lengths, label=\"en\", color=\"red\", alpha=0.33)\n",
        "plt.hist(hn_lengths, label=\"hn\", color=\"blue\", alpha=0.33)\n",
        "plt.yscale(\"log\")     # sentence length fits Benford\"s law\n",
        "plt.ylim(plt.ylim())  # make y-axis consistent for both plots\n",
        "plt.plot([max(en_lengths), max(en_lengths)], plt.ylim(), color=\"red\")\n",
        "plt.plot([max(hn_lengths), max(hn_lengths)], plt.ylim(), color=\"blue\")\n",
        "plt.legend()\n",
        "plt.title(\"Examples count vs Token length\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cOv3zcar3Whm",
        "outputId": "d25af759-fe03-45e7-eefe-d585439ec533"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total English tokens: 32403\n",
            "Total Hindi tokens: 37674\n",
            "Max English length: 16\n",
            "Max Hindi length: 16\n",
            "66074 total pairs\n"
          ]
        }
      ],
      "source": [
        " # count tokens\n",
        "eng_tokens, hin_tokens = set(), set()\n",
        "eng_maxlen, hin_maxlen = 0, 0\n",
        "for eng, hin in text_pairs:\n",
        "    eng_tok, hin_tok = eng.split(), hin.split()\n",
        "    eng_maxlen = max(eng_maxlen, len(eng_tok))\n",
        "    fra_maxlen = max(hin_maxlen, len(hin_tok))\n",
        "    eng_tokens.update(eng_tok)\n",
        "    hin_tokens.update(hin_tok)\n",
        "print(f\"Total English tokens: {len(eng_tokens)}\")\n",
        "print(f\"Total Hindi tokens: {len(hin_tokens)}\")\n",
        "print(f\"Max English length: {eng_maxlen}\")\n",
        "print(f\"Max Hindi length: {fra_maxlen}\")\n",
        "print(f\"{len(text_pairs)} total pairs\")\n",
        "vocab_size_en = len(eng_tokens)\n",
        "vocab_size_hn = len(hin_tokens)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:03.951559Z",
          "iopub.status.busy": "2023-10-22T04:43:03.951199Z",
          "iopub.status.idle": "2023-10-22T04:43:14.624167Z",
          "shell.execute_reply": "2023-10-22T04:43:14.623233Z",
          "shell.execute_reply.started": "2023-10-22T04:43:03.951533Z"
        },
        "id": "BXVcdS4gkNoH",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "seq_length = 15\n",
        "\n",
        "# Create vectorizer for english sentences\n",
        "eng_vectorizer = TextVectorization(\n",
        "    max_tokens=vocab_size_en,\n",
        "    standardize=None,\n",
        "    split=\"whitespace\",\n",
        "    output_mode=\"int\",\n",
        "    output_sequence_length=seq_length,\n",
        ")\n",
        "\n",
        "# Create vectorizer for hindi sentences\n",
        "# For hindi we need to use a language specific tokenizer, since the default one in tensorflow doesn't support hindi\n",
        "# one such tokenizer is indic-nlp-library, which can be installed using the command below\n",
        "# !pip install indic-nlp-library\n",
        "hin_vectorizer = TextVectorization(\n",
        "    max_tokens=vocab_size_hn,\n",
        "    standardize=None,\n",
        "    split=\"whitespace\",\n",
        "    output_mode=\"int\",\n",
        "    output_sequence_length=seq_length + 1\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:17.995282Z",
          "iopub.status.busy": "2023-10-22T04:43:17.994970Z",
          "iopub.status.idle": "2023-10-22T04:43:18.011061Z",
          "shell.execute_reply": "2023-10-22T04:43:18.010011Z",
          "shell.execute_reply.started": "2023-10-22T04:43:17.995254Z"
        },
        "id": "tMubKI2Mqkw8",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "# adapt the vectorizers to the training data\n",
        "train_eng_texts = [pairs[0] for pairs in train_tp]\n",
        "train_hin_texts = [pairs[1] for pairs in train_tp]\n",
        "eng_vectorizer.adapt(train_eng_texts)\n",
        "hin_vectorizer.adapt(train_hin_texts)\n",
        "\n",
        "# set up Dataset object\n",
        "def format_dataset(eng, hin):\n",
        "    eng = eng_vectorizer(eng)\n",
        "    hin = hin_vectorizer(hin)\n",
        "    # split input and target\n",
        "    # Notice that hindi is shifted by 1 to the right, as you will see later that the input to the decoder is the target sequence, right shifted by 1\n",
        "    source = (eng, hin[:, :-1])\n",
        "    # Notice that the target is shifted by `shift` tokens\n",
        "    target = hin[:, 1:]\n",
        "    return (source, target)\n",
        "\n",
        "def make_dataset(pairs, batch_size=64):\n",
        "    \"\"\"Create TensorFlow Dataset for the sentence pairs\"\"\"\n",
        "    # aggregate sentences using zip(*pairs)\n",
        "    eng_texts, hin_texts = zip(*pairs)\n",
        "    # convert them into list, and then create tensors\n",
        "    dataset = tf.data.Dataset.from_tensor_slices((list(eng_texts), list(hin_texts)))\n",
        "    return dataset.shuffle(2048) \\\n",
        "                  .batch(batch_size).map(format_dataset) \\\n",
        "                  .prefetch(16).cache()\n",
        "\n",
        "train_ds = make_dataset(train_tp)\n",
        "val_ds = make_dataset(val_tp)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:18.013207Z",
          "iopub.status.busy": "2023-10-22T04:43:18.012286Z",
          "iopub.status.idle": "2023-10-22T04:43:22.254969Z",
          "shell.execute_reply": "2023-10-22T04:43:22.253799Z",
          "shell.execute_reply.started": "2023-10-22T04:43:18.013177Z"
        },
        "id": "p7t9PFD0y00l",
        "outputId": "843155d5-0767-494a-8b2a-bd7fb362db7d",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "inputs[\"encoder_inputs\"].shape: (64, 15)\n",
            "inputs[\"encoder_inputs\"][0]: [   2   12   10  131   77 2220 1466   53 1711   32 2421  405    3    0\n",
            "    0]\n",
            "inputs[\"decoder_inputs\"].shape: (64, 15)\n",
            "inputs[\"decoder_inputs\"][0]: [   2   21  116   38 4547 4490 4280    9 4295  208   30    3    0    0\n",
            "    0]\n",
            "targets.shape: (64, 15)\n",
            "targets[0]: [  21  116   38 4547 4490 4280    9 4295  208   30    3    0    0    0\n",
            "    0]\n"
          ]
        }
      ],
      "source": [
        "# test the dataset\n",
        "for inputs, targets in train_ds.take(1):\n",
        "    print(f'inputs[\"encoder_inputs\"].shape: {inputs[0].shape}')\n",
        "    print(f'inputs[\"encoder_inputs\"][0]: {inputs[0][0]}')\n",
        "    print(f'inputs[\"decoder_inputs\"].shape: {inputs[1].shape}')\n",
        "    print(f'inputs[\"decoder_inputs\"][0]: {inputs[1][0]}')\n",
        "    print(f\"targets.shape: {targets.shape}\")\n",
        "    print(f\"targets[0]: {targets[0]}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:22.256553Z",
          "iopub.status.busy": "2023-10-22T04:43:22.256183Z",
          "iopub.status.idle": "2023-10-22T04:43:22.320250Z",
          "shell.execute_reply": "2023-10-22T04:43:22.319260Z",
          "shell.execute_reply.started": "2023-10-22T04:43:22.256525Z"
        },
        "id": "HKfvkKeaKKty",
        "outputId": "86451b41-7a8a-4046-9504-0e6853bb7b12",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15)\n",
            "source: [start] this is why some ladies cover their faces from unknown men [end]   \n",
            "hin source: [start] इस लिये कुछ स्त्रियाँ अजनबी पुर्षों से पर्दा करती हैं। [end]    \n",
            "target: इस लिये कुछ स्त्रियाँ अजनबी पुर्षों से पर्दा करती हैं। [end]     \n"
          ]
        }
      ],
      "source": [
        "eng_lookup = list(eng_vectorizer.get_vocabulary())\n",
        "hin_lookup = list(hin_vectorizer.get_vocabulary())\n",
        "eng_sen, hin_tar, hin_source = \"\", \"\",\"\"\n",
        "print(inputs[1].shape)\n",
        "for index in inputs[0][0]:\n",
        "  eng_sen += eng_lookup[index.numpy()] + \" \"\n",
        "for index in inputs[1][0]:\n",
        "  hin_source += hin_lookup[index.numpy()] + \" \"\n",
        "for index in targets[0]:\n",
        "  hin_tar += hin_lookup[index.numpy()] + \" \"\n",
        "print(f\"source: {eng_sen}\")\n",
        "print(f\"hin source: {hin_source}\")\n",
        "print(f\"target: {hin_tar}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UQOyjE95EJ7y"
      },
      "source": [
        "## Vocab size"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:22.322598Z",
          "iopub.status.busy": "2023-10-22T04:43:22.321757Z",
          "iopub.status.idle": "2023-10-22T04:43:22.345887Z",
          "shell.execute_reply": "2023-10-22T04:43:22.344881Z",
          "shell.execute_reply.started": "2023-10-22T04:43:22.322560Z"
        },
        "id": "iiiVhpnJRdRp",
        "outputId": "7d6070ad-4c58-4e74-d654-d818a86dafed",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total English tokens = 32403.\n",
            "Total Hindi tokens = 37674.\n"
          ]
        }
      ],
      "source": [
        "# find out how many distinct tokens exist\n",
        "print(f\"Total English tokens = {vocab_size_en}.\")\n",
        "print(f\"Total Hindi tokens = {vocab_size_hn}.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2g9r7GokBBaB",
        "outputId": "b38ece89-5a7b-41c7-9096-86b1f51ab33e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15)\n",
            "(64, 15)\n",
            "(64, 15)\n",
            "tf.Tensor([   2   84 1330  420  528 1943  694    3    0    0], shape=(10,), dtype=int64)\n",
            "tf.Tensor([  84 1330  420  528 1943  694    3    0    0    0], shape=(10,), dtype=int64)\n"
          ]
        }
      ],
      "source": [
        "for (eng, hin), hn_labels in train_ds.take(1):\n",
        "  break\n",
        "print(eng.shape)\n",
        "print(hin.shape)\n",
        "print(hn_labels.shape)\n",
        "\n",
        "print(hin[0][:10])\n",
        "print(hn_labels[0][:10])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oSr-WkRA_33s"
      },
      "source": [
        "## Positional Embedding Layer\n",
        "\n",
        "Transformers should have the context have relative position of words in a sentence. Hence in positional encoding layer, we create encoding matrix that represent the position of the word in the sentence. Position encodings are then added to the input embeddings.\n",
        "\n",
        "Position encodings are calculated using the below equation\n",
        "\n",
        "![image](https://github.com/sriksmachi/sriksml/blob/main/genai/images/trans-pos-enc.png?raw=1)\n",
        "\n",
        "- d_model -> Size of the embedding <br/>\n",
        "- Pos -> position of the work in the sentence <br/>\n",
        "- i -> i is used to make alternative sequences for even and odd sequences. If this is not considered then ith word on every sentence will have same position encoding\n",
        "\n",
        "Positional encoding is a means of translating the location of objects in a sequence into information that a neural network (or other model) can understand and use.\n",
        "\n",
        "For better understanding of how positional encoding works\n",
        "- Read: https://towardsdatascience.com/master-positional-encoding-part-i-63c05d90a0c3\n",
        "- Video link: https://www.youtube.com/watch?v=dichIcUZfOw\n",
        "\n",
        "The below two cells form the input to the first encoding layer, as described in the below image.\n",
        "\n",
        "![image.png](https://github.com/sriksmachi/sriksml/blob/main/genai/images/trans-PositionalEmbedding.png?raw=1)\n",
        "\n",
        "Intuitively the position encoding matrix represents an overlapping sine and cosine waves, with variations for each sentence such that when all the ith positions have similar representations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:22.347604Z",
          "iopub.status.busy": "2023-10-22T04:43:22.347214Z",
          "iopub.status.idle": "2023-10-22T04:43:22.361671Z",
          "shell.execute_reply": "2023-10-22T04:43:22.360576Z",
          "shell.execute_reply.started": "2023-10-22T04:43:22.347568Z"
        },
        "id": "SWNd4soU1dTv",
        "outputId": "ef098d6f-f10b-42da-8b14-5bb0bf9f683c",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(4, 4)\n",
            "tf.Tensor(\n",
            "[[ 0.          0.          1.          1.        ]\n",
            " [ 0.84147096  0.00999983  0.5403023   0.99995   ]\n",
            " [ 0.9092974   0.01999867 -0.41614684  0.9998    ]\n",
            " [ 0.14112     0.0299955  -0.9899925   0.99955004]], shape=(4, 4), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "# Positional embedding matrix - This matrix represents the position of the word in the sentence.\n",
        "# It takes input as length of the sentence and size of the vector embeddings\n",
        "# The output is a matrix where every row is a word in the sentence, each row represents the position encoding of that word\n",
        "# The length of each row is equal to the length of word embeddings.\n",
        "def positional_encoding(length, depth):\n",
        "  depth = depth/2\n",
        "  positions = np.arange(length)[:, np.newaxis]     # (seq, 1)\n",
        "  depths = np.arange(depth)[np.newaxis, :]/depth   # (1, depth)\n",
        "  angle_rates = 1 / (10000**depths)         # (1, depth)\n",
        "  angle_rads = positions * angle_rates      # (pos, depth)\n",
        "  pos_encoding = np.concatenate(\n",
        "      [np.sin(angle_rads), np.cos(angle_rads)],\n",
        "      axis=-1)\n",
        "  return tf.cast(pos_encoding, dtype=tf.float32)\n",
        "\n",
        "pos_matrix = positional_encoding(4, 4)\n",
        "assert pos_matrix.shape == (4,4)\n",
        "print(pos_matrix.shape)\n",
        "print(pos_matrix)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:22.363213Z",
          "iopub.status.busy": "2023-10-22T04:43:22.362842Z",
          "iopub.status.idle": "2023-10-22T04:43:22.371065Z",
          "shell.execute_reply": "2023-10-22T04:43:22.370169Z",
          "shell.execute_reply.started": "2023-10-22T04:43:22.363183Z"
        },
        "id": "OUmDv5MuIAO2",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "# Positional Embedding\n",
        "class PositionalEmbedding(tf.keras.layers.Layer):\n",
        "  def __init__(self, vocab_size, d_model):\n",
        "    super().__init__()\n",
        "    self.d_model = d_model\n",
        "    # Create a matrix of shape (vocab_size, d_model)\n",
        "    # Embedding matrix maps word index to a vector of size d_model\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, d_model, mask_zero=True)\n",
        "    self.pos_encoding = positional_encoding(length=vocab_size, depth=d_model)\n",
        "\n",
        "  def compute_mask(self, *args, **kwargs):\n",
        "    return self.embedding.compute_mask(*args, **kwargs)\n",
        "\n",
        "  def call(self, x):\n",
        "    length = tf.shape(x)[1]\n",
        "    x = self.embedding(x)\n",
        "    # This factor sets the relative scale of the embedding and positonal_encoding.\n",
        "    x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
        "    x = x + self.pos_encoding[tf.newaxis, :length, :]\n",
        "    return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:22.372551Z",
          "iopub.status.busy": "2023-10-22T04:43:22.372204Z",
          "iopub.status.idle": "2023-10-22T04:43:22.482002Z",
          "shell.execute_reply": "2023-10-22T04:43:22.481059Z",
          "shell.execute_reply.started": "2023-10-22T04:43:22.372507Z"
        },
        "id": "_gZ0CIdtDkHZ",
        "outputId": "63457c1d-4226-45c5-999c-217ee5cedef9",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15, 512)\n",
            "(64, 15, 512)\n"
          ]
        }
      ],
      "source": [
        "eng_pos = PositionalEmbedding(vocab_size_en, 512)\n",
        "hin_pos = PositionalEmbedding(vocab_size_hn, 512)\n",
        "print(eng_pos(eng).shape)\n",
        "print(hin_pos(hin).shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HyiMqSKn__kH"
      },
      "source": [
        "## Self Attention Layer\n",
        "\n",
        "Before understanding the input to the attention layer, let us understand how attention mechanism works.\n",
        "\n",
        "#### Attention Mechanism\n",
        "Attention mechanism contains 3 matrices : query, key and value. If you consider key, value as a dictionary, query matches one of those keys and relates to values correspondingly.\n",
        "The objective here is to find the right values for these attention matrices, so that the attention scores are ideal. Because end of the day, attention scores determine what tokens to focus on.\n",
        "So, how are the attention scores calculated.\n",
        "\n",
        "- Let us take a simple sentence -> \"I can see\". If I consider a 4 dim embedding vector the dimension of the input will be (3 * 4). Each row represents a word in the sentence.\n",
        "- We initialize the weight matrices query, key and value with a 4 * 3 matrix. Every word gets multiplied with its weight matrices resulting in its own query, key and value matrix. So, we have 3 paris of query, key and value matrices, each of size 3 * 3.  Since we are going to multiply Query and Key matrices they should be of same shape, the Value can have a different shape. You will notice that the final outcome dimension is same as dimension of value matrix.\n",
        "- The input (each word has position embedding of size 1 * 4) with dim 3 * 4 gets multiplied by the weight matrix 4 * 3. This results in 3 * 3 Query matrix.\n",
        "- The same operation is applied to keys, values as well.  That is, input gets multiplied by Keys, Value matrices each resulting in  3 * 3 Matrices.\n",
        "- The values within attention filters are scaled by square root of the Key dimension in our case 4.\n",
        "- Softmax function is applied on scores to standardize the values between 0-1\n",
        "- The final matrix of size 3 * 3, is multiplied with Value filter resulting in a final out of size 3 * 3.\n",
        "\n",
        "The steps explained above are visually represented in this rought hand sketch.\n",
        "\n",
        "![image.png](https://github.com/sriksmachi/sriksml/blob/main/genai/images/trans-self.png?raw=1)\n",
        "\n",
        "In the nutshell we have implemented the below mathematical operation\n",
        "\n",
        "![image.png](https://github.com/sriksmachi/sriksml/blob/main/genai/images/trans-atten-math.png?raw=1)\n",
        "\n",
        "#### Multi headed attention\n",
        "\n",
        "The final attention scores are a result of one attention head. In multi-headed attention, there are multiple attention heads resulting in multiple attention matrices.\n",
        "Each attention matrix learns different representations of the sentences.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:22.483351Z",
          "iopub.status.busy": "2023-10-22T04:43:22.483092Z",
          "iopub.status.idle": "2023-10-22T04:43:22.490128Z",
          "shell.execute_reply": "2023-10-22T04:43:22.489165Z",
          "shell.execute_reply.started": "2023-10-22T04:43:22.483330Z"
        },
        "id": "IrefrQsA_iWE",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class BaseAttention(tf.keras.layers.Layer):\n",
        "  def __init__(self, **kwargs):\n",
        "    super().__init__()\n",
        "    self.mha = tf.keras.layers.MultiHeadAttention(**kwargs)\n",
        "    self.layernorm = tf.keras.layers.LayerNormalization()\n",
        "    self.add = tf.keras.layers.Add()\n",
        "\n",
        "class GlobalSelfAttention(BaseAttention):\n",
        "  def call(self, x):\n",
        "    attn_output = self.mha(\n",
        "        query=x,\n",
        "        value=x,\n",
        "        key=x)\n",
        "    x = self.add([x, attn_output])\n",
        "    x = self.layernorm(x)\n",
        "    return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:22.491634Z",
          "iopub.status.busy": "2023-10-22T04:43:22.491298Z",
          "iopub.status.idle": "2023-10-22T04:43:25.040081Z",
          "shell.execute_reply": "2023-10-22T04:43:25.038858Z",
          "shell.execute_reply.started": "2023-10-22T04:43:22.491602Z"
        },
        "id": "GqqGTnJ2E9wv",
        "outputId": "721d50f6-7d57-4fac-c3cd-680e37ff7e26",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15, 512)\n",
            "(64, 15, 512)\n"
          ]
        }
      ],
      "source": [
        "sample_gsa = GlobalSelfAttention(num_heads=2, key_dim=512)\n",
        "enc_output = eng_pos(eng)\n",
        "print(enc_output.shape)\n",
        "print(sample_gsa(enc_output).shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LKFWcmfmJexs"
      },
      "source": [
        "## Cross Attention Layer\n",
        "\n",
        "The decoder network uses a cross attention layer. In cross attention layer, along with the input we pass the context vector to the key and value parameters.\n",
        "The context vector is the final output from the encoder network, that represents the final attention scores of the input sentences across all heads. The following image represents the Cross attention part of the decoder network.\n",
        "\n",
        "![image.png](https://github.com/sriksmachi/sriksml/blob/main/genai/images/trans-crossatten.png?raw=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.041834Z",
          "iopub.status.busy": "2023-10-22T04:43:25.041493Z",
          "iopub.status.idle": "2023-10-22T04:43:25.048186Z",
          "shell.execute_reply": "2023-10-22T04:43:25.047063Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.041804Z"
        },
        "id": "XArWRqWdJh-t",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class CrossAttention(BaseAttention):\n",
        "  def call(self, x, context):\n",
        "    attn_output, attn_scores = self.mha(\n",
        "        query=x,\n",
        "        key=context,\n",
        "        value=context,\n",
        "        return_attention_scores=True)\n",
        "    # Cache the attention scores for plotting later.\n",
        "    self.last_attn_scores = attn_scores\n",
        "    x = self.add([x, attn_output])\n",
        "    x = self.layernorm(x)\n",
        "    return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.049854Z",
          "iopub.status.busy": "2023-10-22T04:43:25.049529Z",
          "iopub.status.idle": "2023-10-22T04:43:25.108484Z",
          "shell.execute_reply": "2023-10-22T04:43:25.107447Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.049807Z"
        },
        "id": "Ge-jOyeqJfm8",
        "outputId": "bd779dba-e44a-4de9-e586-08ddc0ef62e9",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15, 512)\n",
            "(64, 15, 512)\n",
            "(64, 15, 512)\n"
          ]
        }
      ],
      "source": [
        "sample_ca = CrossAttention(num_heads=2, key_dim=512)\n",
        "emb_output_eng = eng_pos(eng)\n",
        "emb_output_hin = hin_pos(hin)\n",
        "print(emb_output_eng.shape)\n",
        "print(emb_output_hin.shape)\n",
        "print(sample_ca(emb_output_eng, emb_output_hin).shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tXsQFv7pKY-E"
      },
      "source": [
        "## Causal Self Attention\n",
        "\n",
        "The first layer of the decoder network has different inputs compared to the other cross attention layers mentioned above.\n",
        "During training phase the input to the decoder network is same as that of the encoder network, except that certain areas of the input are masked.\n",
        "The causal mask ensures that for predicting a token the network has access only to the tokens that come before it.\n",
        "\n",
        "The below images shows the part of the architecture related to causal self attention.\n",
        "\n",
        "![image.png](https://github.com/sriksmachi/sriksml/blob/main/genai/images/trans-dec-input.png?raw=1)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.109995Z",
          "iopub.status.busy": "2023-10-22T04:43:25.109673Z",
          "iopub.status.idle": "2023-10-22T04:43:25.115625Z",
          "shell.execute_reply": "2023-10-22T04:43:25.114687Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.109971Z"
        },
        "id": "XegPUftHKbxO",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class CausalSelfAttention(BaseAttention):\n",
        "  def call(self, x):\n",
        "    attn_output = self.mha(\n",
        "        query=x,\n",
        "        value=x,\n",
        "        key=x,\n",
        "        use_causal_mask = True)\n",
        "    x = self.add([x, attn_output])\n",
        "    x = self.layernorm(x)\n",
        "    return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.117206Z",
          "iopub.status.busy": "2023-10-22T04:43:25.116939Z",
          "iopub.status.idle": "2023-10-22T04:43:25.174222Z",
          "shell.execute_reply": "2023-10-22T04:43:25.173253Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.117183Z"
        },
        "id": "eDQnHr9sKdIV",
        "outputId": "e1f23df6-11c4-4987-dd62-6a4caa155d61",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15, 512)\n",
            "(64, 15, 512)\n"
          ]
        }
      ],
      "source": [
        "sample_csa = CausalSelfAttention(num_heads=2, key_dim=512)\n",
        "emb_output_eng = eng_pos(eng)\n",
        "print(emb_output_eng.shape)\n",
        "print(sample_csa(emb_output_eng).shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CXsPLh9hN4xL"
      },
      "source": [
        "## Feedforward Layer\n",
        "\n",
        "Feedforward layer is a common layer that is added on top of every Multi head attention block in both encoder and decoder networks.\n",
        "The feedforward layer has 2 key components - 1. Feed forward 2. Add & Norm\n",
        "The feedfoward layer is composed using a sequential model consists of 2 dense layers that perform point-wise feed forward and a drop out layer.\n",
        "The outcome of the feedforward layer is added to the input (skip connection) followed by normalization layer.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.176207Z",
          "iopub.status.busy": "2023-10-22T04:43:25.175801Z",
          "iopub.status.idle": "2023-10-22T04:43:25.184325Z",
          "shell.execute_reply": "2023-10-22T04:43:25.183350Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.176168Z"
        },
        "id": "XAmJ_GS0W3TW",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class FeedForward(tf.keras.layers.Layer):\n",
        "  def __init__(self, d_model, dff, dropout_rate=0.1):\n",
        "    super().__init__()\n",
        "    self.seq = tf.keras.Sequential([\n",
        "      tf.keras.layers.Dense(dff, activation='relu'),\n",
        "      tf.keras.layers.Dense(d_model),\n",
        "      tf.keras.layers.Dropout(dropout_rate)\n",
        "    ])\n",
        "    self.add = tf.keras.layers.Add()\n",
        "    self.layer_norm = tf.keras.layers.LayerNormalization()\n",
        "\n",
        "  def call(self, x):\n",
        "    x = self.add([x, self.seq(x)])\n",
        "    x = self.layer_norm(x)\n",
        "    return x\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.186022Z",
          "iopub.status.busy": "2023-10-22T04:43:25.185699Z",
          "iopub.status.idle": "2023-10-22T04:43:25.269065Z",
          "shell.execute_reply": "2023-10-22T04:43:25.267963Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.185995Z"
        },
        "id": "IA0vBi3sQLoN",
        "outputId": "1f7b8911-84ce-45e6-890d-a5343ca78380",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15, 512)\n"
          ]
        }
      ],
      "source": [
        "sample_ffn = FeedForward(512, 2048)\n",
        "print(sample_ffn(emb_output_eng).shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3bHMYBxqYfyX"
      },
      "source": [
        "## Encoder\n",
        "\n",
        "The encoder layer is built by stacking one global attention block followed by feedforward layer, this forms one multi-headed attention block. This notebook uses 8 headed attention as configured in the paper.\n",
        "You will later see that layer we stack the encoder layers, there can be any number of encoder layers. The below picture represents one encoder layer.\n",
        "\n",
        "![image.png](https://github.com/sriksmachi/sriksml/blob/main/genai/images/trans-enc-layer.png?raw=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.270544Z",
          "iopub.status.busy": "2023-10-22T04:43:25.270227Z",
          "iopub.status.idle": "2023-10-22T04:43:25.277258Z",
          "shell.execute_reply": "2023-10-22T04:43:25.276221Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.270517Z"
        },
        "id": "Mv2dLw3uZ5LZ",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class EncoderLayer(tf.keras.layers.Layer):\n",
        "  def __init__(self,*, d_model, num_heads, dff, dropout_rate=0.1):\n",
        "    super().__init__()\n",
        "    self.self_attention = GlobalSelfAttention(\n",
        "        num_heads=num_heads,\n",
        "        key_dim=d_model,\n",
        "        dropout=dropout_rate)\n",
        "    self.ffn = FeedForward(d_model, dff)\n",
        "\n",
        "  def call(self, x):\n",
        "    x = self.self_attention(x)\n",
        "    x = self.ffn(x)\n",
        "    return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.285074Z",
          "iopub.status.busy": "2023-10-22T04:43:25.284773Z",
          "iopub.status.idle": "2023-10-22T04:43:25.385018Z",
          "shell.execute_reply": "2023-10-22T04:43:25.383948Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.285048Z"
        },
        "id": "YxYYSA2uSh6n",
        "outputId": "edc5770a-1d02-4ae4-d623-ed11902d7dad",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15, 512)\n",
            "(64, 15, 512)\n"
          ]
        }
      ],
      "source": [
        "sample_encoder_layer = EncoderLayer(d_model=512, num_heads=8, dff=2048)\n",
        "print(emb_output_eng.shape)\n",
        "print(sample_encoder_layer(emb_output_eng).shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.386808Z",
          "iopub.status.busy": "2023-10-22T04:43:25.386362Z",
          "iopub.status.idle": "2023-10-22T04:43:25.396551Z",
          "shell.execute_reply": "2023-10-22T04:43:25.394919Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.386765Z"
        },
        "id": "g2UW3YR7S0Z-",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class Encoder(tf.keras.layers.Layer):\n",
        "  def __init__(self, *, num_layers, d_model, num_heads,\n",
        "               dff, vocab_size, dropout_rate=0.1):\n",
        "    super().__init__()\n",
        "    self.d_model = d_model\n",
        "    self.num_layers = num_layers\n",
        "    self.pos_embedding = PositionalEmbedding(\n",
        "        vocab_size=vocab_size, d_model=d_model)\n",
        "    self.enc_layers = [\n",
        "        EncoderLayer(d_model=d_model,\n",
        "                     num_heads=num_heads,\n",
        "                     dff=dff,\n",
        "                     dropout_rate=dropout_rate)\n",
        "        for _ in range(num_layers)]\n",
        "    self.dropout = tf.keras.layers.Dropout(dropout_rate)\n",
        "\n",
        "  def call(self, x):\n",
        "    # `x` is token-IDs shape: (batch, seq_len)\n",
        "    x = self.pos_embedding(x)  # Shape `(batch_size, seq_len, d_model)`.\n",
        "    # Add dropout.\n",
        "    x = self.dropout(x)\n",
        "    for i in range(self.num_layers):\n",
        "      x = self.enc_layers[i](x)\n",
        "    return x  # Shape `(batch_size, seq_len, d_model)`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.398521Z",
          "iopub.status.busy": "2023-10-22T04:43:25.398120Z",
          "iopub.status.idle": "2023-10-22T04:43:25.765480Z",
          "shell.execute_reply": "2023-10-22T04:43:25.764291Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.398483Z"
        },
        "id": "7C_MFnxcS7gd",
        "outputId": "94cbe8c6-f986-42e5-a25d-f84246b5e9ce",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15)\n",
            "(64, 15, 512)\n"
          ]
        }
      ],
      "source": [
        "# Instantiate the encoder.\n",
        "sample_encoder = Encoder(num_layers=4,\n",
        "                         d_model=512,\n",
        "                         num_heads=8,\n",
        "                         dff=2048,\n",
        "                         vocab_size=vocab_size_en)\n",
        "sample_encoder_output = sample_encoder(eng, training=False)\n",
        "print(eng.shape)\n",
        "print(sample_encoder_output.shape)  # Shape `(batch_size, input_seq_len, d_model)`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LJ7uvGogY2Qy"
      },
      "source": [
        "## Decoder\n",
        "\n",
        "The decoder layer is same as encoder except that each layer contains two attention layers - causal self attention (masked inputs) and cross attention (context vector from encoder and inputs from causal attention layer).\n",
        "It outcome of the causal attention layer passes through the feedforward layer as in the encoder network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.767057Z",
          "iopub.status.busy": "2023-10-22T04:43:25.766759Z",
          "iopub.status.idle": "2023-10-22T04:43:25.775392Z",
          "shell.execute_reply": "2023-10-22T04:43:25.774158Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.767032Z"
        },
        "id": "Y6QcpGeXcvsD",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class DecoderLayer(tf.keras.layers.Layer):\n",
        "  def __init__(self,\n",
        "               *,\n",
        "               d_model,\n",
        "               num_heads,\n",
        "               dff,\n",
        "               dropout_rate=0.1):\n",
        "    super(DecoderLayer, self).__init__()\n",
        "    self.causal_self_attention = CausalSelfAttention(\n",
        "        num_heads=num_heads,\n",
        "        key_dim=d_model,\n",
        "        dropout=dropout_rate)\n",
        "    self.cross_attention = CrossAttention(\n",
        "        num_heads=num_heads,\n",
        "        key_dim=d_model,\n",
        "        dropout=dropout_rate)\n",
        "    self.ffn = FeedForward(d_model, dff)\n",
        "\n",
        "  def call(self, x, context):\n",
        "    x = self.causal_self_attention(x=x)\n",
        "    x = self.cross_attention(x=x, context=context)\n",
        "    # Cache the last attention scores for plotting later\n",
        "    self.last_attn_scores = self.cross_attention.last_attn_scores\n",
        "    x = self.ffn(x)  # Shape `(batch_size, seq_len, d_model)`.\n",
        "    return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CmX7H225xpiw"
      },
      "source": [
        "The decoder network can also be stacked using any number of decoder layers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.909632Z",
          "iopub.status.busy": "2023-10-22T04:43:25.909208Z",
          "iopub.status.idle": "2023-10-22T04:43:25.918906Z",
          "shell.execute_reply": "2023-10-22T04:43:25.917903Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.909595Z"
        },
        "id": "huNZYkKMUeS0",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class Decoder(tf.keras.layers.Layer):\n",
        "  def __init__(self, *, num_layers, d_model, num_heads, dff, vocab_size,\n",
        "               dropout_rate=0.1):\n",
        "    super(Decoder, self).__init__()\n",
        "    self.d_model = d_model\n",
        "    self.num_layers = num_layers\n",
        "    self.pos_embedding = PositionalEmbedding(vocab_size=vocab_size,\n",
        "                                             d_model=d_model)\n",
        "    self.dropout = tf.keras.layers.Dropout(dropout_rate)\n",
        "    self.dec_layers = [\n",
        "        DecoderLayer(d_model=d_model, num_heads=num_heads,\n",
        "                     dff=dff, dropout_rate=dropout_rate)\n",
        "        for _ in range(num_layers)]\n",
        "    self.last_attn_scores = None\n",
        "\n",
        "  def call(self, x, context):\n",
        "    # `x` is token-IDs shape (batch, target_seq_len)\n",
        "    x = self.pos_embedding(x)  # (batch_size, target_seq_len, d_model)\n",
        "    x = self.dropout(x)\n",
        "    for i in range(self.num_layers):\n",
        "      x  = self.dec_layers[i](x, context)\n",
        "    self.last_attn_scores = self.dec_layers[-1].last_attn_scores\n",
        "    # The shape of x is (batch_size, target_seq_len, d_model).\n",
        "    return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:25.920313Z",
          "iopub.status.busy": "2023-10-22T04:43:25.920026Z",
          "iopub.status.idle": "2023-10-22T04:43:26.598556Z",
          "shell.execute_reply": "2023-10-22T04:43:26.597457Z",
          "shell.execute_reply.started": "2023-10-22T04:43:25.920287Z"
        },
        "id": "83Vikt41UhQD",
        "outputId": "393761d5-6581-4f14-d1a4-b39e79378877",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 15)\n",
            "(64, 15, 512)\n",
            "(64, 15, 512)\n"
          ]
        }
      ],
      "source": [
        "# Instantiate the decoder.\n",
        "sample_decoder = Decoder(num_layers=4,\n",
        "                         d_model=512,\n",
        "                         num_heads=8,\n",
        "                         dff=2048,\n",
        "                         vocab_size=vocab_size_hn)\n",
        "output = sample_decoder(x=hin,context=emb_output_eng)\n",
        "# Print the shapes.\n",
        "print(hin.shape)\n",
        "print(emb_output_hin.shape)\n",
        "print(output.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:26.600462Z",
          "iopub.status.busy": "2023-10-22T04:43:26.600061Z",
          "iopub.status.idle": "2023-10-22T04:43:26.607256Z",
          "shell.execute_reply": "2023-10-22T04:43:26.606238Z",
          "shell.execute_reply.started": "2023-10-22T04:43:26.600427Z"
        },
        "id": "e1VgXYNQVMOH",
        "outputId": "24918377-624c-4dba-9cfe-2248b00bbfc5",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([64, 8, 15, 15])"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ],
      "source": [
        "sample_decoder.last_attn_scores.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CRbKuSF8bRIv"
      },
      "source": [
        "## Transformer Model\n",
        "\n",
        "The final transformer model contains of three key components\n",
        "1. Encoder Network with multiple layers of Encoders\n",
        "2. Decoder Network with multiple layers of Decoders\n",
        "3. The output of the decoder passes through a Dense layer that reduces the output to a dense vector of target vocab size. This is specific to the downstream task, neural machine translation in this case."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:26.609229Z",
          "iopub.status.busy": "2023-10-22T04:43:26.608782Z",
          "iopub.status.idle": "2023-10-22T04:43:26.618875Z",
          "shell.execute_reply": "2023-10-22T04:43:26.617728Z",
          "shell.execute_reply.started": "2023-10-22T04:43:26.609201Z"
        },
        "id": "f2LpORoYtQJE",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "class Transformer(tf.keras.Model):\n",
        "  def __init__(self, *, num_layers, d_model, num_heads, dff,\n",
        "               input_vocab_size, target_vocab_size, dropout_rate=0.1):\n",
        "    super().__init__()\n",
        "    self.encoder = Encoder(num_layers=num_layers, d_model=d_model,\n",
        "                           num_heads=num_heads, dff=dff,\n",
        "                           vocab_size=input_vocab_size,\n",
        "                           dropout_rate=dropout_rate)\n",
        "\n",
        "    self.decoder = Decoder(num_layers=num_layers, d_model=d_model,\n",
        "                           num_heads=num_heads, dff=dff,\n",
        "                           vocab_size=target_vocab_size,\n",
        "                           dropout_rate=dropout_rate)\n",
        "\n",
        "    self.final_layer = tf.keras.layers.Dense(target_vocab_size)\n",
        "\n",
        "  def call(self, inputs):\n",
        "    # To use a Keras model with `.fit` you must pass all your inputs in the\n",
        "    # first argument.\n",
        "    context, x  = inputs\n",
        "    context = self.encoder(context)  # (batch_size, context_len, d_model)\n",
        "    x = self.decoder(x, context)  # (batch_size, target_len, d_model)\n",
        "    # Final linear layer output.\n",
        "    logits = self.final_layer(x)  # (batch_size, target_len, target_vocab_size)\n",
        "    try:\n",
        "      # Drop the keras mask, so it doesn't scale the losses/metrics.\n",
        "      # b/250038731\n",
        "      del logits._keras_mask\n",
        "    except AttributeError:\n",
        "      pass\n",
        "\n",
        "    # Return the final output and the attention weights.\n",
        "    return logits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sHEBhPdXibco"
      },
      "source": [
        "## Training Utilities\n",
        "\n",
        "Here we are using a custom training schedule which is also used in the paper."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:26.620641Z",
          "iopub.status.busy": "2023-10-22T04:43:26.620271Z",
          "iopub.status.idle": "2023-10-22T04:43:26.930907Z",
          "shell.execute_reply": "2023-10-22T04:43:26.929922Z",
          "shell.execute_reply.started": "2023-10-22T04:43:26.620614Z"
        },
        "id": "LpJUx205idZx",
        "outputId": "7c0ff380-a027-4c98-f2d3-3a9af446a61d",
        "trusted": true
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlEAAAGwCAYAAACJjDBkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABqB0lEQVR4nO3de1xUdf4/8NcMw8xwmwFEZkBRMPF+DXTENLeVwrI22lovy6ZrbtSuVmal6aa25f4wL7utZZnbxdr1ln3LLVOK0DITURHzgndRFBkQkRlArjOf3x84RydRAZkZZng9H495IOe8Z857Ttm8+pzPfI5MCCFARERERE0id3UDRERERO6IIYqIiIioGRiiiIiIiJqBIYqIiIioGRiiiIiIiJqBIYqIiIioGRiiiIiIiJpB4eoGPJnVasX58+cREBAAmUzm6naIiIioEYQQKCsrQ3h4OOTyG483MUQ50Pnz5xEREeHqNoiIiKgZzp49i44dO95wP0OUAwUEBACo/4eg0Whc3A0RERE1htlsRkREhPQ5fiMMUQ5ku4Sn0WgYooiIiNzMrabicGI5ERERUTMwRBERERE1A0MUERERUTMwRBERERE1A0MUERERUTMwRBERERE1A0MUERERUTMwRBERERE1A0MUERERUTMwRBERERE1Q6sIUcuWLUNkZCTUajUMBgN27dp10/r169ejR48eUKvV6Nu3LzZt2mS3XwiBuXPnIiwsDD4+PoiPj8fx48ftakpKSpCUlASNRoPAwEBMnjwZ5eXl0v5XX30VMpnsuoefn1/LvXEiIiJyWy4PUevWrcP06dMxb9487N27F/3790dCQgKKiooarN+xYwfGjx+PyZMnIzs7G4mJiUhMTMTBgwelmoULF2Lp0qVYvnw5MjMz4efnh4SEBFRVVUk1SUlJOHToENLS0rBx40Zs27YNycnJ0v4XX3wRBQUFdo9evXrhd7/7neNOBhEREbkP4WKDBw8WU6ZMkX63WCwiPDxcpKSkNFg/ZswYMXr0aLttBoNBPPXUU0IIIaxWq9Dr9WLRokXS/tLSUqFSqcSaNWuEEELk5OQIAGL37t1SzebNm4VMJhP5+fkNHnffvn0CgNi2bdsN30tVVZUwmUzS4+zZswKAMJlMtzgL7s1isYrKmjpXt0FERNQiTCZToz6/XToSVVNTg6ysLMTHx0vb5HI54uPjkZGR0eBzMjIy7OoBICEhQarPzc2F0Wi0q9FqtTAYDFJNRkYGAgMDERsbK9XEx8dDLpcjMzOzweO+//776NatG4YPH37D95OSkgKtVis9IiIibnEGPMPkj3fD8P/ScbG82tWtEBEROY1LQ1RxcTEsFgt0Op3ddp1OB6PR2OBzjEbjTettP29VExoaardfoVAgODi4weNWVVVh1apVmDx58k3fz6xZs2AymaTH2bNnb1rvCYQQ2Hr0AkyVtfh0zzlXt0NEROQ0Clc34A6++OILlJWVYeLEiTetU6lUUKlUTuqqdSi9XCv9+eylyy7shIiIyLlcOhIVEhICLy8vFBYW2m0vLCyEXq9v8Dl6vf6m9baft6r55cT1uro6lJSUNHjc999/Hw8++OB1o1sEFJiuTtY/lG9yYSdERETO5dIQpVQqERMTg/T0dGmb1WpFeno64uLiGnxOXFycXT0ApKWlSfVRUVHQ6/V2NWazGZmZmVJNXFwcSktLkZWVJdVs2bIFVqsVBoPB7rVzc3OxdevWW17Ka6sKzdeEqPNmVNZYXNgNERGR87j8ct706dMxceJExMbGYvDgwXjzzTdRUVGBSZMmAQAmTJiADh06ICUlBQDw3HPPYcSIEViyZAlGjx6NtWvXYs+ePVixYgUAQCaTYdq0aZg/fz6io6MRFRWFOXPmIDw8HImJiQCAnj17YtSoUXjyySexfPly1NbWYurUqRg3bhzCw8Pt+vvwww8RFhaG+++/33knxY1cOxJVZxXYf64Uhi7tXNgRERGRc7g8RI0dOxYXLlzA3LlzYTQaMWDAAKSmpkqXzvLy8iCXXx0wGzp0KFavXo1XXnkFs2fPRnR0NDZs2IA+ffpINTNmzEBFRQWSk5NRWlqKYcOGITU1FWq1WqpZtWoVpk6dipEjR0Iul+PRRx/F0qVL7XqzWq1YuXIl/vjHP8LLy8vBZ8I9Ga8ZiQKArLxLDFFERNQmyIQQwtVNeCqz2QytVguTyQSNRuPqdhxixmc/49M956D18YapshbxPUPx/sRBrm6LiIio2Rr7+e3yFcvJvRnN9WtDPdA3DACQdeYSmMuJiKgtYIii22I0VQIA4nuGQqmQ49LlWuQWV7i4KyIiIsdjiKLbYrwysTwi2Bf9O2oB1I9GEREReTqGKGq2yzV1MFfVAQD0WjXu7BwEgCGKiIjaBoYoajbbKJSv0gsBKgViOwcDAHadLnFlW0RERE7BEEXNZlveQK9VQyaTYXBkMGQy4NSFChT9YukDIiIiT8MQRc1mG4nSa+rX39L6eqNXWP1XQXfmcjSKiIg8G0MUNdu1I1E2hqj6hTYzT110SU9ERETOwhBFzfbLkSgAGNKlfl7UToYoIiLycAxR1Gy2EBV2zUjU4Kj6eVEnL1SgqIzzooiIyHMxRFGz2S7n6a4ZiQr0VaKHvn5e1C7OiyIiIg/GEEXNdnUkysduOy/pERFRW8AQRc1Sa7HiQnn9ffN0WpXdviFd6ieX7zzFkSgiIvJcDFHULBfKqiEEoJDLEOJnH6IMV+ZFnSgqx4Wyahd1SERE5FgMUdQsBaar86HkcpndvkBfJXpemRe142Sx03sjIiJyBoYoapbCBtaIutbwbiEAgG3HGKKIiMgzMURRsxQ0sEbUte6Obg8A+PH4BQghnNYXERGRszBEUbPcaiQqpnMQ1N5yFJVV43hRuTNbIyIicgqGKGqWW41Eqb29MPjKLWC2HbvgtL6IiIichSGKmqXQdPORKAC4O7p+XtSPxzkvioiIPA9DFDVLgbkSwM1D1PAr86Iycy+iqtbilL6IiIichSGKmkwIgUJT/fpPN7qcBwDddP4IDVChqtaKrDOXnNUeERGRUzBEUZOVVNSgxmIFYH/fvF+SyWTSaNS245wXRUREnoUhiprMduPhEH8llIqb/yt095X1on44yhBFRESehSGKmsx4zWrlt3J3dHvIZcARYxnOXbrs6NaIiIichiGKmsw2EhV2k0nlNkF+SsR0DgIAbDlS5NC+iIiInIkhipqsKSNRADCypw4AkH6YIYqIiDwHQxQ1mS1ENWYkCgBG9ggFAGScvIiK6jqH9UVERORMDFHUZLbLeY0dieoa6o9Owb6osVix/QQX3iQiIs/AEEVNdnUkyqdR9TKZDL++Mhq1hZf0iIjIQzBEUZMZpVu+qBr9nHjbvKgjRbBahUP6IiIiciaGKGqS8uo6lF2Z16Rv5EgUAAyOCoa/SoHi8mocyDc5qj0iIiKnYYiiJrGNQvmrFPBXKRr9PKVCLi28+c0ho0N6IyIiciaGKGqSQmlSeeMv5dkk9NYDAFIPGiEEL+kREZF7Y4iiJmnqpPJr/bpHKJQKOU4VV+BYYXlLt0ZERORUDFHUJE1d3uBaAWpv3B1df0lv04GCFu2LiIjI2RiiqEmautDmL93fJwxA/SU9IiIid+byELVs2TJERkZCrVbDYDBg165dN61fv349evToAbVajb59+2LTpk12+4UQmDt3LsLCwuDj44P4+HgcP37crqakpARJSUnQaDQIDAzE5MmTUV5eft3rLF68GN26dYNKpUKHDh3w97//vWXetBsrsN3ypZkhKr6nDgq5DEcLy3DyAi/pERGR+3JpiFq3bh2mT5+OefPmYe/evejfvz8SEhJQVNTwgow7duzA+PHjMXnyZGRnZyMxMRGJiYk4ePCgVLNw4UIsXboUy5cvR2ZmJvz8/JCQkICqqiqpJikpCYcOHUJaWho2btyIbdu2ITk52e5Yzz33HN5//30sXrwYR44cwZdffonBgwc75kS4EdvE8rBmXM4DAK2vN4Z2rb+kx9EoIiJya8KFBg8eLKZMmSL9brFYRHh4uEhJSWmwfsyYMWL06NF22wwGg3jqqaeEEEJYrVah1+vFokWLpP2lpaVCpVKJNWvWCCGEyMnJEQDE7t27pZrNmzcLmUwm8vPzpRqFQiGOHDnSpPdTVVUlTCaT9Dh79qwAIEwmU5NepzWLeT1NdJ65URw4V9rs11iTeUZ0nrlRjF66rQU7IyIiahkmk6lRn98uG4mqqalBVlYW4uPjpW1yuRzx8fHIyMho8DkZGRl29QCQkJAg1efm5sJoNNrVaLVaGAwGqSYjIwOBgYGIjY2VauLj4yGXy5GZmQkA+Oqrr9ClSxds3LgRUVFRiIyMxJ/+9CeUlJTc9D2lpKRAq9VKj4iIiCackdavps6KixXVAAB9My/nAcC9vXSQy4CD+WacuVjRUu0RERE5lctCVHFxMSwWC3Q6nd12nU4Ho7HhyzxGo/Gm9baft6oJDQ21269QKBAcHCzVnDp1CmfOnMH69evxySefYOXKlcjKysJjjz120/c0a9YsmEwm6XH27Nmb1ruborIqCAF4e8kQ7Kts9uu081fhriuX9P6373xLtUdERORUjV9yug2xWq2orq7GJ598gm7dugEAPvjgA8TExODo0aPo3r17g89TqVRQqZq+CKW7KLxmeQO5XHZbr/XwgA748XgxNuzLxzO/7gqZ7PZej4iIyNlcNhIVEhICLy8vFBYW2m0vLCyEXq9v8Dl6vf6m9baft6r55cT1uro6lJSUSDVhYWFQKBRSgAKAnj17AgDy8vKa9D49ie2befpmTiq/VkJvHVQKOU5dqMCh8+bbfj0iIiJnc1mIUiqViImJQXp6urTNarUiPT0dcXFxDT4nLi7Orh4A0tLSpPqoqCjo9Xq7GrPZjMzMTKkmLi4OpaWlyMrKkmq2bNkCq9UKg8EAALjrrrtQV1eHkydPSjXHjh0DAHTu3Pl23rZbs60RdTvzoWwC1N6I71l/2XVDdv5tvx4REZGzuXSJg+nTp+Pf//43Pv74Yxw+fBh//vOfUVFRgUmTJgEAJkyYgFmzZkn1zz33HFJTU7FkyRIcOXIEr776Kvbs2YOpU6cCAGQyGaZNm4b58+fjyy+/xIEDBzBhwgSEh4cjMTERQP2I0qhRo/Dkk09i165d+OmnnzB16lSMGzcO4eHhAOonmt9555144oknkJ2djaysLDz11FO499577Uan2hpjC45EAcDDA+rP95c/n4fFynvpERGRe3HpnKixY8fiwoULmDt3LoxGIwYMGIDU1FRpYnheXh7k8qs5b+jQoVi9ejVeeeUVzJ49G9HR0diwYQP69Okj1cyYMQMVFRVITk5GaWkphg0bhtTUVKjVVz/4V61ahalTp2LkyJGQy+V49NFHsXTpUmm/XC7HV199hWeeeQZ33303/Pz8cP/992PJkiVOOCutl+2WLy0xEgUAv+oeCq2PN4rKqrHz1EVpsjkREZE7kAkhOATgIGazGVqtFiaTCRqNxtXt3LbH3t2BPWcu4e3fD8SD/cJb5DVnfX4Aa3bl4XcxHbHod/1b5DWJiIhuR2M/v11+2xdyH7aRqObeN68hiVcu6aUeNKKyxtJir0tERORoDFHUKFarsFvioKUMigxGxyAflFXXIfVQQYu9LhERkaMxRFGjlFyuQa1FQCYDQgNaLkTJ5TL8LqZ+Zfd1uz1rcVIiIvJsDFHUKLZv5rXzU0GpaNl/bR6L7QiZDNh5qoS3gSEiIrfBEEWNYgtRLTkfyqZDoA+GR7cHAHy6h6NRRETkHhiiqFEKHDAf6lpjY+sv6X2WdQ51FqtDjkFERNSSGKKoUQodOBIFAPG9QhHk641CczW2Hb/gkGMQERG1JIYoapSCFrzlS0NUCi88MrAjAE4wJyIi98AQRY1iW96gpW750pCxg+ov6aUfLpKOR0RE1FoxRFGjFJgqAThuJAoAuusDMCgyCHVWgdWZeQ47DhERUUtgiKJGKTRXA3BsiAKAx+MiAQCrd+Whpo4TzImIqPViiKJbKquqRXl1HQDHXs4DgFG99QjxV+FCWTW+OWR06LGIiIhuB0MU3ZJtflKAWgE/lcKhx1Iq5Pi9oRMA4D8ZZxx6LCIiotvBEEW3JH0zz8GjUDa/H9wJXnIZdp0uweECs1OOSURE1FQMUXRLRgcvb/BLeq0aCb11AIBPOBpFREStFEMU3ZLRySNRADDhygTzDdn5KL1c47TjEhERNRZDFN2S0ezY1cobYogKRs8wDSprLVjF5Q6IiKgVYoiiW7JNLNc5MUTJZDIk3x0FAPjop9OorrM47dhERESNwRBFt1Tg4Pvm3ciD/cIRplWjuLwa/8s+79RjExER3QpDFN2SNBLlxDlRAODtJccTd9WPRq348RSsVuHU4xMREd0MQxTdVHWdBcXl9RO7w7Q+Tj/+uMERCFApcKKoHN8fK3L68YmIiG6EIYpuqujK7V6UCjmCfL2dfvwAtTfGX1l8c8W2U04/PhER0Y0wRNFNGaVLeSrIZDKX9PDHoZFQyGXYeaoE2XmXXNIDERHRLzFE0U3Z1ogK0zj/Up5NeKAPEgd2AAC8teWEy/ogIiK6FkMU3ZQtRDlzeYOGTLmnK+QyYMuRIhw4Z3JpL0RERABDFN2CKxbabEhUiB8eHlA/GrV0y3GX9kJERAQwRNEtSCNRTl7eoCFT7ukKmQxIyynEofMcjSIiItdiiKKbai0jUQDQNdQfD/YLBwC8zblRRETkYgxRdFOtaSQKAJ75dVcAwOaDRhwxml3cDRERtWUMUXRDVquQVitvDSNRANBNF4AH+uoBAP/49piLuyEioraMIYpuqLiiGnVWAZkMaB+gcnU7kun3doNcBnybU8h1o4iIyGUYouiGCk31q5W391fB26v1/KvSNTQAj97ZEQDwRuoRCMF76hERkfO1nk9GanUKTJUAAH0ruZR3rWn3doPSS46dp0rw4/FiV7dDRERtEEMU3ZBtPpS+lUwqv1aHQB88HtcZALDwmyOwWjkaRUREzsUQRTdUcOWbea1xJAoA/vKrO+Cn9MLBfDM2HSxwdTtERNTGMETRDdnWiGqtIaqdvwpP3t0FALAw9Siq6ywu7oiIiNoShii6IdsaUa3xcp7Nk8O7IDRAhbySy1j502lXt0NERG1IqwhRy5YtQ2RkJNRqNQwGA3bt2nXT+vXr16NHjx5Qq9Xo27cvNm3aZLdfCIG5c+ciLCwMPj4+iI+Px/Hj9vdbKykpQVJSEjQaDQIDAzF58mSUl5dL+0+fPg2ZTHbdY+fOnS33xlu51j4SBQB+KgVmjOoBAHhrywlcKKt2cUdERNRWuDxErVu3DtOnT8e8efOwd+9e9O/fHwkJCSgqKmqwfseOHRg/fjwmT56M7OxsJCYmIjExEQcPHpRqFi5ciKVLl2L58uXIzMyEn58fEhISUFVVJdUkJSXh0KFDSEtLw8aNG7Ft2zYkJydfd7zvvvsOBQUF0iMmJqblT0IrJIRwi5EoAPjtwA7o20GL8uo6/CPtqKvbISKitkK42ODBg8WUKVOk3y0WiwgPDxcpKSkN1o8ZM0aMHj3abpvBYBBPPfWUEEIIq9Uq9Hq9WLRokbS/tLRUqFQqsWbNGiGEEDk5OQKA2L17t1SzefNmIZPJRH5+vhBCiNzcXAFAZGdnN/u9mUwmAUCYTKZmv4armCprROeZG0XnmRtFRXWtq9u5pV25F0XnmRtF5MsbxcH8Ule3Q0REbqyxn98uHYmqqalBVlYW4uPjpW1yuRzx8fHIyMho8DkZGRl29QCQkJAg1efm5sJoNNrVaLVaGAwGqSYjIwOBgYGIjY2VauLj4yGXy5GZmWn32r/5zW8QGhqKYcOG4csvv7zp+6murobZbLZ7uCvbKJRGrYCvUuHibm5tUGQwHuwXBiGA177K4QKcRETkcC4NUcXFxbBYLNDpdHbbdTodjEZjg88xGo03rbf9vFVNaGio3X6FQoHg4GCpxt/fH0uWLMH69evx9ddfY9iwYUhMTLxpkEpJSYFWq5UeERERtzoFrZYtRIVpfVzcSeO9fH8PKBVyZOaWYON+LnlARESO5fI5Ua1VSEgIpk+fDoPBgEGDBmHBggX4wx/+gEWLFt3wObNmzYLJZJIeZ8+edWLHLcsWonSteFL5L3UM8sWUX3UFALy2MQfmqloXd0RERJ7MpSEqJCQEXl5eKCwstNteWFgIvV7f4HP0ev1N620/b1Xzy4nrdXV1KCkpueFxAcBgMODEiRM33K9SqaDRaOwe7sr2zbywVj6p/Jee/lUXRIX44UJZNf7x7TFXt0NERB7MpSFKqVQiJiYG6enp0jar1Yr09HTExcU1+Jy4uDi7egBIS0uT6qOioqDX6+1qzGYzMjMzpZq4uDiUlpYiKytLqtmyZQusVisMBsMN+923bx/CwsKa/kbdUIEbjkQBgErhhdcf7gMA+CTjNPafK3VtQ0RE5LFcPmN4+vTpmDhxImJjYzF48GC8+eabqKiowKRJkwAAEyZMQIcOHZCSkgIAeO655zBixAgsWbIEo0ePxtq1a7Fnzx6sWLECACCTyTBt2jTMnz8f0dHRiIqKwpw5cxAeHo7ExEQAQM+ePTFq1Cg8+eSTWL58OWprazF16lSMGzcO4eHhAICPP/4YSqUSAwcOBAB8/vnn+PDDD/H+++87+Qy5hu2+eWFuFqIAYFh0CB4eEI7/7TuPv35xEBum3AUvuczVbRERkYdxeYgaO3YsLly4gLlz58JoNGLAgAFITU2VJobn5eVBLr86YDZ06FCsXr0ar7zyCmbPno3o6Ghs2LABffr0kWpmzJiBiooKJCcno7S0FMOGDUNqairU6quBYNWqVZg6dSpGjhwJuVyORx99FEuXLrXr7fXXX8eZM2egUCjQo0cPrFu3Do899piDz0jrUOAma0TdyF9H98SWI0U4kG/Cf3eewcShka5uiYiIPIxM8LvgDmM2m6HVamEymdxuftSdr6ehpKIGm58bjp5h7tW7zX92nsGcDQfhp/TCN8/fjY5Bvq5uiYiI3EBjP7/57Ty6TlWtBSUVNQDcdyQKAJIGd8LgyGBU1Fjw8v8d4NpRRETUohii6DpF5vr7z6kUcgT6eru4m+aTy2V447F+UCnk2H6iGGt3u++SE0RE1PowRNF1rr3xsEzm3hOyo0L88FJCdwDA378+jPzSShd3REREnoIhiq5TYKoPGu58Ke9ak+6Kwp2dAlFeXYdZn/OyHhERtQyGKLpO4TUjUZ7ASy7Dwsf6Q6mQY9uxC1izi5f1iIjo9jFE0XWk5Q08JEQBQNdQf7x0X/1lvdc35uDkhXIXd0RERO6OIYquI41EecjlPJvJw6JwV9d2qKy1YNrafaips7q6JSIicmMMUXQd20iUO65WfjNyuQxLfjcAgb7eOJBvwj/SeG89IiJqPoYouk6h7b55HjYSBdRfolzw234AgPe2ncSOk8Uu7oiIiNwVQxTZsVgFCsvq14kK0/q4uBvHGNVHj3GDIiAEMH3dz7h0ZWFRIiKipmCIIjsXy6thsQrIZUCIv9LV7TjMnAd7ISrED0ZzFZ7/dB+sVi57QERETcMQRXZs86FCA9RQeHnuvx5+KgXe/v1AqBRyfH/0At794aSrWyIiIjfjuZ+S1Cy21cp1HjapvCG9w7V4/eE+AIAl3x7l/CgiImoShiiyY7StEaVRubgT5xgzKAKPxXSEVQDPrtmHoishkoiI6FYYosiObSTKUyeVN+T1h/ughz4AxeXVmLomG7UWrh9FRES3xhBFdowevLzBjfgovfBO0p3wVymwK7cE8zfmuLolIiJyAwxRZMfooQtt3kqX9v74x5j+AICPM85gza48F3dEREStHUMU2ZEmlrehkSib+3rr8eJ93QAAc/93ELtyS1zcERERtWa3FaKqqjgJ15MIIdrsSJTNlHu6YnS/MNRaBP783yycu3TZ1S0REVEr1eQQZbVa8frrr6NDhw7w9/fHqVOnAABz5szBBx980OINkvOYK+tQWWsBUH97lLZIJpNh0WP90Dtcg4sVNXjykyxUVNe5ui0iImqFmhyi5s+fj5UrV2LhwoVQKq+uaN2nTx+8//77LdocOZftUl6grzfU3l4u7sZ1fJUKrJgQixB/JQ4XmPHMmmzU8Rt7RET0C00OUZ988glWrFiBpKQkeHld/aDt378/jhw50qLNkXMVmCoBAPo2OB/qlzoE+mDFhFioFHJsOVKEuV8eghC8NQwREV3V5BCVn5+Prl27XrfdarWitra2RZoi1yi8MhLVVi/l/dKdnYLwr3EDIZMBqzPz8M73vDUMERFd1eQQ1atXL/z444/Xbf/ss88wcODAFmmKXKNAWq2cIcpmVB89Xn2oNwBg0TdH8UX2ORd3RERErYWiqU+YO3cuJk6ciPz8fFitVnz++ec4evQoPvnkE2zcuNERPZKTcCSqYROHRiK/tBIrtp3CjM/2IzRAjbu6hri6LSIicrEmj0Q9/PDD+Oqrr/Ddd9/Bz88Pc+fOxeHDh/HVV1/h3nvvdUSP5CQcibqxl0f1wINXlj548pM9yM675OqWiIjIxZo8EgUAw4cPR1paWkv3Qi4m3XyYI1HXkctlWDKmP0ov12L7iWL88aPdWJs8BD3DNK5ujYiIXKTJI1FdunTBxYsXr9teWlqKLl26tEhT5BpGXs67KZXCCysmxCCmcxBMlbV4/INdyC2ucHVbRETkIk0OUadPn4bFYrlue3V1NfLz81ukKXK+qloLSi/Xf7syTOPj4m5aL1+lAh/+cRB6hWlQXF6NP7yfifzSSle3RURELtDoy3lffvml9OdvvvkGWq1W+t1isSA9PR2RkZEt2hw5j+1SntpbDo1Ps67ythlaH298MnkwxryXgVMXKvCH9zOxNnlIm7zfIBFRW9boT8vExEQA9bfFmDhxot0+b29vREZGYsmSJS3aHDmP7VJemNYHMpnMxd20fiH+Kqz6kwGPvZuB3OIKjFuxE2ueHMJLoUREbUijL+dZrVZYrVZ06tQJRUVF0u9WqxXV1dU4evQoHnzwQUf2Sg5kW95Ap1G5uBP3Eab1wdrkIegY5IPc4gqMXZGB87y0R0TUZjR5TlRubi5CQrhGjqexLW8QpuV8qKaICPbF2uQhiAj2wZmLlzFuxU7OkSIiaiOaNfmloqICP/zwA/Ly8lBTU2O379lnn22Rxsi5bHOiOK+n6ToG+WJtchzGr9iJvJLLGPteBtY8OQQRwb6ubo2IiByoySEqOzsbDzzwAC5fvoyKigoEBwejuLgYvr6+CA0NZYhyU0ZpJIohqjk6BPpg3VNDMH7FTpy+eBlj3svAfyYb0DXU39WtERGRgzT5ct7zzz+Phx56CJcuXYKPjw927tyJM2fOICYmBosXL3ZEj+QERjNHom5X/RypONzR3g8Fpir8bvkO/Hy21NVtERGRgzQ5RO3btw8vvPAC5HI5vLy8UF1djYiICCxcuBCzZ892RI/kBByJahl6rRrrnx6Kfh21uHS5Fr//9078dKLY1W0REZEDNDlEeXt7Qy6vf1poaCjy8vIAAFqtFmfPnm1WE8uWLUNkZCTUajUMBgN27dp10/r169ejR48eUKvV6Nu3LzZt2mS3XwiBuXPnIiwsDD4+PoiPj8fx48ftakpKSpCUlASNRoPAwEBMnjwZ5eXlDR7vxIkTCAgIQGBgYLPeX2tXZ7HiQnk1AK5W3hKC/ZRY/eQQDL2jHSpqLJj00W6kHixwdVtERNTCmhyiBg4ciN27dwMARowYgblz52LVqlWYNm0a+vTp0+QG1q1bh+nTp2PevHnYu3cv+vfvj4SEBBQVFTVYv2PHDowfPx6TJ09GdnY2EhMTkZiYiIMHD0o1CxcuxNKlS7F8+XJkZmbCz88PCQkJqKqqkmqSkpJw6NAhpKWlYePGjdi2bRuSk5OvO15tbS3Gjx+P4cOHN/m9uYvi8hpYrAJechlC/LnEQUvwVynw0aRBGNVbjxqLFX9ZtRerM/Nc3RYREbUk0US7d+8WW7ZsEUIIUVhYKBISEkRAQIC48847RXZ2dlNfTgwePFhMmTJF+t1isYjw8HCRkpLSYP2YMWPE6NGj7bYZDAbx1FNPCSGEsFqtQq/Xi0WLFkn7S0tLhUqlEmvWrBFCCJGTkyMAiN27d0s1mzdvFjKZTOTn59u99owZM8Qf/vAH8dFHHwmtVtuk92YymQQAYTKZmvQ8Z8vOuyQ6z9wohvy/71zdiseps1jFzM9+Fp1nbhSdZ24UKZsOC4vF6uq2iIjoJhr7+d3kkajY2Fjcc889AOov56WmpsJsNiMrKwsDBgxo0mvV1NQgKysL8fHx0ja5XI74+HhkZGQ0+JyMjAy7egBISEiQ6nNzc2E0Gu1qtFotDAaDVJORkYHAwEDExsZKNfHx8ZDL5cjMzJS2bdmyBevXr8eyZcsa9X6qq6thNpvtHu7AaKpf14iX8lqel1yGlN/2xbT4aADA8h9O4pk12aiqvf7+k0RE5F6aHKJuZO/evU1esby4uBgWiwU6nc5uu06ng9FobPA5RqPxpvW2n7eqCQ0NtduvUCgQHBws1Vy8eBF//OMfsXLlSmg0mka9n5SUFGi1WukRERHRqOe5mm1SuZ7fzHMImUyGafHd8I8x/eHtJcPXBwow/t87UXxlHhoREbmnJoWob775Bi+++CJmz56NU6dOAQCOHDmCxMREDBo0CFar1SFNusKTTz6J3//+97j77rsb/ZxZs2bBZDJJj+ZOtHe2givLG3AkyrF+e2dHfPKEARq1Atl5pXjknZ9woqjhLzMQEVHr1+gQ9cEHH+D+++/HypUr8cYbb2DIkCH473//i7i4OOj1ehw8ePC6b8ndSkhICLy8vFBYWGi3vbCwEHq9vsHn6PX6m9bbft6q5pcT1+vq6lBSUiLVbNmyBYsXL4ZCoYBCocDkyZNhMpmgUCjw4YcfNtibSqWCRqOxe7iDQo5EOU3cHe3w+V/uQqdgX5wtqcQjy37CliOFt34iERG1Oo0OUf/617/wxhtvoLi4GJ9++imKi4vxzjvv4MCBA1i+fDl69uzZ5IMrlUrExMQgPT1d2ma1WpGeno64uLgGnxMXF2dXDwBpaWlSfVRUFPR6vV2N2WxGZmamVBMXF4fS0lJkZWVJNVu2bIHVaoXBYABQP29q37590uO1115DQEAA9u3bh0ceeaTJ77U1s903jyNRztE11B9f/GUoBkUGoay6DpM/3oO3txyHEMLVrRERUVM0dqa6r6+vyM3NFULUfwPO29tbbN++/XYmvwshhFi7dq1QqVRi5cqVIicnRyQnJ4vAwEBhNBqFEEI8/vjj4uWXX5bqf/rpJ6FQKMTixYvF4cOHxbx584S3t7c4cOCAVLNgwQIRGBgo/ve//4n9+/eLhx9+WERFRYnKykqpZtSoUWLgwIEiMzNTbN++XURHR4vx48ffsE9P/nbeiIVbROeZG8XOk8WubqVNqa61iL9+sV/65t7T/9kjyqtqXd0WEVGb19jP70bfO6+yshK+vvU3VJXJZFCpVAgLC7vtEDd27FhcuHABc+fOhdFoxIABA5CamipNDM/Ly5MW9wSAoUOHYvXq1XjllVcwe/ZsREdHY8OGDXZrVM2YMQMVFRVITk5GaWkphg0bhtTUVKjVV0daVq1ahalTp2LkyJGQy+V49NFHsXTp0tt+P+5GCCGNRIVpfVzcTduiVMgxP7EveodrMfd/B7H5oBGnLlRgxYQYdG7n5+r2iIjoFmRCNO4aglwux/z58+HvX39D1ZkzZ+Kll15CSEiIXR1vQHyV2WyGVquFyWRqtfOjSi/XYMBraQCAI6+Pgtrby8UdtU1ZZ0rw9H/34kJZNTRqBRb/rj/u693wvEAiInKsxn5+NzpERUZGQiaT3bRGJpNJ39oj9whRhwvMuP9fPyLYT4m9c+51dTttWqG5Ck//NwvZeaUAgMnDojBzVA8oFS22EgkRETVCYz+/G3057/Tp0y3RF7UyxivLG+j4zTyX02nUWJcch4WpR/D+9lx8sD0XWWcu4e3fD0THIF9Xt0dERL/A/8Vt464utMl75rUGSoUcrzzYCysej4FGrcC+s6UYvXQ7vsvhMghERK0NQ1QbJ4UoTipvVe7rrcfXzw5H/45amCpr8adP9uDVLw/xdjFERK0IQ1Qbx1u+tF4Rwb5Y//RQPHFXFABg5Y7TeOit7Th03uTizoiICGCIavNsc6LCuNBmq6RUyDH3oV74aNIghPircLyoHInLfsKKbSdhtXJxTiIiV2KIauNsI1E6hqhW7Z7uofhm2nDc20uHWovA/9t0BEnvZ+J8aaWrWyMiarOaHKLMZnODj7KyMtTU1DiiR3IgjkS5j3b+Kqx4PAYpv+0LH28vZJy6iIR/bsO63Xm8ZQwRkQs0OUQFBgYiKCjoukdgYCB8fHzQuXNnzJs3D1ar1RH9UguqrLHAVFkLgEscuAuZTIbxgzth03PDMSAiEGXVdZj5fwcw4cNdOHfpsqvbIyJqU5ocolauXInw8HDMnj0bGzZswIYNGzB79mx06NAB7777LpKTk7F06VIsWLDAEf1SC7KNQvkqvaBRN3rJMGoFokL88NnTcZj9QA+oFHL8eLwYCf/chv/sPMO5UkRETtLkT86PP/4YS5YswZgxY6RtDz30EPr27Yv33nsP6enp6NSpE/7+979j9uzZLdostawCU/18Gr1GfcvV6Kn1UXjJkXz3HYjvqcOMz/Zjz5lLmLPhIL7efx5vPNqP998jInKwJo9E7dixAwMHDrxu+8CBA5GRkQEAGDZsGPLy8m6/O3KoQrNtjSheynNnXdr749On4jDvoV7w8fbCzlMluO+f27Bs6wnU1PGyOhGRozQ5REVEROCDDz64bvsHH3yAiIgIAMDFixcRFBR0+92RQxVwjSiPIZfLMOmuKKROG46hd7RDdZ0Vi745igeW/oidpy66uj0iIo/U5Mt5ixcvxu9+9zts3rwZgwYNAgDs2bMHR44cwWeffQYA2L17N8aOHduynVKLKzRxJMrTdG7nh1V/MmDDvnzM33gYJ4rKMW7FTvz2zg6Y/UBPhPjz9j5ERC1FJprx3ejc3Fy89957OHbsGACge/fueOqppxAZGdnS/bm1xt4F2lWe+s8efHOoEK893BsT4iJd3Q61MNPlWiz85ghW78qDEIDWxxszR/XAuEERkMs5B46I6EYa+/ndrBBFjdPaQ9TDb2/Hz+dMWPF4DO7rrXd1O+Qge/Mu4a9fHMThAjMAoE8HDeY+2BuDo4Jd3BkRUevU2M/vZn2vvbS0FLt27UJRUdF160FNmDChOS9JLmDkxPI24c5OQfhq6l34OOMM3kw7hoP5Zox5LwOj+4Vh1v090DHI19UtEhG5pSaPRH311VdISkpCeXk5NBqN3VfjZTIZSkpKWrxJd9WaR6LqLFZ0e2UzrALY9deRCA1gkGoLisurseTbY1i7u/4Sn0ohR/LdXfDnX90BXyXXCiMiAhr/+d3kb+e98MILeOKJJ1BeXo7S0lJcunRJejBAuY8L5dWwCkAhlyHEj5ON24oQfxVSftsXG58ZBkNUMKrrrHhrywncs/h7/F/WOVi4UCcRUaM1OUTl5+fj2Wefha8vLwG4M9vyBjqNmpOM26De4VqsTR6Cd5PuRMcgHxSaq/HC+p8xeumP2Hq0iPfiIyJqhCaHqISEBOzZs8cRvZATFUohiqNQbZVMJsP9fcPw3fQRmDGqOwLUChwxlmHSR7sxbsVOZOddcnWLREStWpMnQYwePRovvfQScnJy0LdvX3h7e9vt/81vftNizZHj2EaiwrQ+Lu6EXE3t7YW//Korxg/qhHd/OImVO04jM7cEj7yzA/f30ePFhO64o72/q9skImp1mjyxXC6/8eCVTCaDxWK57aY8RWueWJ6y6TDe23YKT9wVhbkP9XJ1O9SK5JdW4p9px/D53nOwCsBLLsNvB3bAM7+ORqd2vIxPRJ7PYRPLrVbrDR8MUO7j6kgUv5VH9joE+mDx7/pj83N3I75nKCxWgfVZ5/DrJd9j5mf7cbbksqtbJCJqFZocosgz2NaI0jFE0Q101wfg/YmD8H9/Horh0SGoswqs23MW9yz+HrM+349zlximiKhta9ScqKVLlyI5ORlqtRpLly69ae2zzz7bIo2RYxk5EkWNFNM5CP+ZbEDWmRK8+d1x/Hi8GGt2ncVnWefwu9gI/OVXd3DBTiJqkxo1JyoqKgp79uxBu3btEBUVdeMXk8lw6tSpFm3QnbXWOVFCCHSfk4qaOit+nHEPIoL5AUiNt/t0Cd787hh+OnERQP2cqYf7h+PpX92BbroAF3dHRHT7eO+8VqC1hqiSihrc+XoaAODo/FFQKbxc3BG5o8xTF/HWlhPYfqJY2hbfU4e/3HMH7uwU5MLOiIhuj0PvnUfuzXYpr52fkgGKms3QpR0MXdrh57OlWP7DSaQeMuK7w4X47nAhDFHB+Ms9XXF3dIjdraGIiDxJk0OUxWLBypUrkZ6e3uANiLds2dJizZFjGM2VAHjjYWoZ/SMC8e4fYnDyQjne++EkvsjOR2ZuCTJzd6FXmAaTh0Xhwf5hDOxE5HGaHKKee+45rFy5EqNHj0afPn34f5luyGiqBgDoNQxR1HLuaO+PhY/1x/P3dsP7P+Ziza485BSY8cL6n7Eg9QgeH9IZSYZOaOfPVfKJyDM0OUStXbsWn376KR544AFH9ENOYDRxJIocJ0zrgzkP9sIzv+6K1bvy8MmOMzCaq/CPtGN4e+sJPDKgA54YFoXuek5CJyL31uR1opRKJbp27eqIXshJbGtEcSSKHCnQV4m//Korfpx5D/41bgD6ddSips6KdXvOIuHNbXj8g0ykHy6ExcrvthCRe2pyiHrhhRfwr3/9i3d5d2O21co5EkXO4O0lx8MDOuB/U+7CZ0/H4f4+eshlwI/HizH54z24e+FWLNt6AsXl1a5ulYioSZp8OW/79u3YunUrNm/ejN69e193A+LPP/+8xZojxyg0M0SR88lkMsRGBiM2MhhnSy7jk4zTWJ91DvmllVj0zVG8+d0x3N8nDI/HdUZs5yDOtySiVq/JISowMBCPPPKII3ohJ+F988jVIoJ98dfRvfDCfd2xcX8B/rvzDPadLcWXP5/Hlz+fR3ddAP4Q1xmPDOwAfxVXYiGi1qlJi23W1dVh9erVuO+++6DX6x3Zl0dojYttVlTXofe8bwAAB169DwFq71s8g8g5Duab8N+dZ7BhXz6qauuXTvHx9sLofmEYOyiCo1NE5DSN/fxu0pwohUKBp59+GtXVLTt3YdmyZYiMjIRarYbBYMCuXbtuWr9+/Xr06NEDarUaffv2xaZNm+z2CyEwd+5chIWFwcfHB/Hx8Th+/LhdTUlJCZKSkqDRaBAYGIjJkyejvLxc2n/06FHcc8890Ol0UKvV6NKlC1555RXU1ta23Bt3Adukcn+VggGKWpU+HbRY8Gg/ZM6Ox7yHeuGO9n6orLXU36NveQZGLvkB735/EkVX/h0mInK1Jk8sHzx4MLKzs1usgXXr1mH69OmYN28e9u7di/79+yMhIQFFRUUN1u/YsQPjx4/H5MmTkZ2djcTERCQmJuLgwYNSzcKFC7F06VIsX74cmZmZ8PPzQ0JCAqqqrv7HNykpCYcOHUJaWho2btyIbdu2ITk5Wdrv7e2NCRMm4Ntvv8XRo0fx5ptv4t///jfmzZvXYu/dFQqvXMrTabhWD7VOWh9vTLorCt9NH4H/+3McxsR2hK/SC6eKK/BG6hHELdiCP328G98eMqLWYr31CxIROUiT75336aefYtasWXj++ecRExMDPz8/u/39+vVrUgMGgwGDBg3C22+/DQCwWq2IiIjAM888g5dffvm6+rFjx6KiogIbN26Utg0ZMgQDBgzA8uXLIYRAeHg4XnjhBbz44osAAJPJBJ1Oh5UrV2LcuHE4fPgwevXqhd27dyM2NhYAkJqaigceeADnzp1DeHh4g71Onz4du3fvxo8//tio99YaL+f9X9Y5vLD+ZwzrGoL//sng6naIGqW8ug6b9hdg3Z6zyDpzSdoe4q/CwwPC8cjADugdruHlPiJqEQ67d964ceMAAM8++6y0TSaTQQgBmUwGi8XS6NeqqalBVlYWZs2aJW2Ty+WIj49HRkZGg8/JyMjA9OnT7bYlJCRgw4YNAIDc3FwYjUbEx8dL+7VaLQwGAzIyMjBu3DhkZGQgMDBQClAAEB8fD7lcjszMzAYnzp84cQKpqan47W9/e8P3U11dbXep02w23/wEuIDtcp6Oa0SRG/FXKTBmUATGDIrAiaIyrN9zDv+39xyKy6vxwfZcfLA9F11D/fHIwA74Tf9wRAT7urplImoDmhyicnNzW+zgxcXFsFgs0Ol0dtt1Oh2OHDnS4HOMRmOD9UajUdpv23azmtDQULv9CoUCwcHBUo3N0KFDsXfvXlRXVyM5ORmvvfbaDd9PSkoK/va3v91wf2tglNaI4uU8ck9dQwMw64GeeDGhO74/egEb9uXju5xCnCgqx6JvjmLRN0cxODIYiQM7YHTfMGh9OfePiByjySGqc+fOjuij1Vq3bh3Kysrw888/46WXXsLixYsxY8aMBmtnzZplN0pmNpsRERHhrFYbRVqtXOvj4k6Ibo+3lxz39tLh3l46mKtqkXrQiA3Z+cg4dRG7Tpdg1+kSvPrlIdzToz1+078Dft0jFD5K3gSZiFpOsxdgycnJQV5eHmpqauy2/+Y3v2n0a4SEhMDLywuFhYV22wsLC2+4hIJer79pve1nYWEhwsLC7GoGDBgg1fxy4npdXR1KSkquO64tBPXq1QsWiwXJycl44YUX4OV1/X+MVSoVVKrWPcIjjUTxch55EI3aG2NiIzAmNgIFpkp8ue88vsjOxxFjGb45VIhvDhXCx9sLv+4Zigf7huFX3RmoiOj2NTlEnTp1Co888ggOHDggzYUCIE3obMqcKKVSiZiYGKSnpyMxMRFA/cTy9PR0TJ06tcHnxMXFIT09HdOmTZO2paWlIS4uDgAQFRUFvV6P9PR0KTSZzWZkZmbiz3/+s/QapaWlyMrKQkxMDABgy5YtsFqtMBhuPNnaarWitrYWVqu1wRDlDmwjUVxokzxVmNYHT424A0+NuANHjGZsyD6PTQcKkFdyGV/vL8DX+wvgq/TCyJ46jO6rx6+6h0Lt7Z5/n4nItZocop577jlERUUhPT0dUVFR2LVrFy5evIgXXngBixcvbnID06dPx8SJExEbG4vBgwfjzTffREVFBSZNmgQAmDBhAjp06ICUlBTp+CNGjMCSJUswevRorF27Fnv27MGKFSsA1Ie5adOmYf78+YiOjkZUVBTmzJmD8PBwKaj17NkTo0aNwpNPPonly5ejtrYWU6dOxbhx46Rv5q1atQre3t7o27cvVCoV9uzZg1mzZmHs2LHX3erGXdRarNL9yTixnNqCHnoNXr5fg5mjuuNgvhkbD5zH1/sLcO5SJb76+Ty++vk8/K4Eqgf6hmFEt/YcoSKiRmtyiMrIyMCWLVsQEhICuVwOuVyOYcOGISUlBc8++2yT15AaO3YsLly4gLlz58JoNGLAgAFITU2VJobn5eVBLr+6nNXQoUOxevVqvPLKK5g9ezaio6OxYcMG9OnTR6qZMWMGKioqkJycjNLSUgwbNgypqalQq68Gh1WrVmHq1KkYOXIk5HI5Hn30USxduvTqiVEo8MYbb+DYsWMQQqBz586YOnUqnn/++aaeslajqKwaQgDeXjK081O6uh0ip5HJZOjbUYu+HbV4eVQP7D9nwtcH6kel8ksrpdvNqL3lGB7dHvf10mFkTx2C+feEiG6iyetEBQUFYe/evYiKisIdd9yB999/H/fccw9OnjyJvn374vLly47q1e20tnWiss5cwqPv7kCHQB/89PKvXd0OkcsJIbDvbCm+3l+AzQeNyC+tlPbJZcCgyGDc11uP+3rpuGwCURvisHWi+vTpg59//hlRUVEwGAxYuHAhlEolVqxYgS5dutxW0+RYRt54mMiOTCbDwE5BGNgpCH8d3RM5BWZ8e6gQ3+YU4nCBGZm5JcjMLcHrG3PQM0yDe3vpcF8vHRf2JCIAzQhRr7zyCioqKgAAr732Gh588EEMHz4c7dq1w7p161q8QWo50kKbDFFE15HJZOgdrkXvcC2ev7cbzpZcRlpOIb7NMWJXbgkOF5hxuMCMpenH0SHQB7/q3h6/7hGKuDvawVfZ7C86E5Eba/LlvIaUlJQgKIh3WP+l1nY57+9f5+DfP+biT8Oi8MqDvVzdDpHbKKmowZYjRfj2kBHbjl9AVe3Ve/YpFXIM6dIO93Rvj3u6hyIyxO8mr0RE7sBhl/NsTpw4gZMnT+Luu+9GcHAwWiCLkYMZzfXfzNNzJIqoSYL9lHgspiMei+mIyhoLMk4VY+uRC9hypAj5pZXYduwCth27gL99lYOoED/86kqgGhwVzOUTiDxYk0PUxYsXMWbMGGzduhUymQzHjx9Hly5dMHnyZAQFBWHJkiWO6JNagNFUP2mWIYqo+XyUXvh1Dx1+3UOH14TAyQvlUqDafboEucUVyC2uwEc/nYaPtxfu6toOI7q1x7Do9ohs58sReyIP0uQQ9fzzz8Pb2xt5eXno2bOntH3s2LGYPn06Q1QrJt3yhWtEEbUImUyGrqEB6BoagCfv7oKyqlr8dKJ+lGrr0SIUlVXju8NF+O5w/R0SOgT6YFjXEAyLDsFdXUO4hAKRm2tyiPr222/xzTffoGPHjnbbo6OjcebMmRZrjFqWEAKFJl7OI3KkALU3RvUJw6g+YRBCIKfAjO+PXsCPxy8g68wl5JdWYt2es1i35ywAoHe4BsOiQzC8a3vERgbx0h+Rm2lyiKqoqICv7/XrpZSUlLT6+8a1ZSUVNaix1E+GDQ1giCJytGu/7Tflnq64XFOHzNwSbD9ejJ9OFOOIsQyHzptx6LwZ7/1wCiqFHIMigzEsOgTDuoagZ5gGXnJe+iNqzZocooYPH45PPvkEr7/+OoD6/1BYrVYsXLgQ99xzT4s3SC2j4MoaUSH+KigV8ltUE1FL81UqcE/3UNzTPRQAUFRWhZ9OFGP78YvYfuICCs3V2H6iGNtPFAMAAtQKDI4MxpAu7TCkSzv0CmeoImptmhyiFi5ciJEjR2LPnj2oqanBjBkzcOjQIZSUlOCnn35yRI/UAgpt86G0HC0kag1CA9R4ZGBHPDKwI4QQOFFUXh+ijhcjM7cEZVV1SD9ShPQj9fOpAlQKDIoKxpAu9cGqV5gGCi/+DxGRKzVrxfJjx47h7bffRkBAAMrLy/Hb3/4WU6ZMQVhYmCN6pBZgG4nSa3xc3AkR/ZJMJkO0LgDRugBMuisKdRYrcgrM2HnqIjJPlWBXbgnKquuw5UgRtlwJVf4qBQZFBmFIl3YwdGmHPuEMVUTO1qx1orRaLf7617/abTt37hySk5OxYsWKFmmMWhZHoojch8JLjn4dA9GvYyCS774DFqtAznkzMnMv1gerKyNVW49ewNajFwAAvkovDOwUiJjOwYjtHISBnQIRoPZ28Tsh8mwtdq+Cixcv4oMPPmCIaqUKpPvmcSSKyN14yWXo21GLvh21+NPwLrBYBQ5fGanaeaoEu3IvwlxVh59OXMRPJy4CqL+Bcg+9BrGRQYjpHITYyGB0COTff6KWxBs+tRG2kSgd14gicntechn6dNCiT4f6UGW1ChwrKsOe05eQdeYSdp8uwblLlcgpMCOnwIxPMuqXnwnTqhHTOQiDIoMR0zkIPfQBvARIdBsYotqIqyNRDFFEnkYul6GHXoMeeg3+MKQzgPr/cdpz+hL2nClB1plLOHTejAJTFTbuL8DG/QUAALW3HH07aDEgIhADOwVhQEQgwrRqrqpO1EgMUW1EoYkjUURtiU6jxuh+YRjdr/4LPxXVdfj5bCn2nLmEPWcuIfvMJZRV12H36UvYffoSgFwAQGiACgMiAjGgUyAGRNTPy/JX8aOCqCGN/pvx29/+9qb7S0tLb7cXcpDy6jqUVdcB4GrlRG2Vn0qBoV1DMLRrCADAahU4VVyO7LxS7Dtb/zhiLENRWTW+zSnEtzmFAOrnVkWHBtgFq+hQf14GJEITQpRWq73l/gkTJtx2Q9TyjFdGoQJUCv4fJREBqL8EaLvv3+9iIwAAlTUWHDxvwr5rglV+aSWOFpbhaGGZdLsalUKOXuEa9L0yL6tvBy2DFbVJjf5E/eijjxzZBzmQLURxFIqIbsZH6YVBkcEYFBksbSsyVyH7SqDal1eKA/kmlFfXITuvFNl5pVIdgxW1RRyWaAOMZoYoImqeUI0aCb31SOitB1B/GfD0xQocyDfhYL4J+8+ZcOi8+YbBqmeYBv061gerXmEaROv8oVLwRsvkGRii2gCjqRIAoOekciK6TXK5DF3a+6NLe388PKADgJsHK9tlQRuFXIauof7oFaZBzzANeoXX/wz2U7roHRE1H0NUG8CRKCJypFsFqwPnTDh43oTDBWUwVdbiiLEMR4xlQHa+9Bp6jfpKoApArzAteoYFILKdH+S86TK1YgxRbQDnRBGRszUUrIQQOG+qwuHz9YuAHr6yGOiZi5dhNFfBaK6S7g0I1N/Kpoc+AD3DNOgRpkF3XQC66wKg9eXtbKh1YIhqA6SRKF7OIyIXkslk6BDogw6BPojvpZO2l1XV4qix7GqwOm/GEWMZLtdYsDevFHuvmWcF1P+3rJs+AN11/uimC0B3fQCiQwPgo+RcK3Iuhqg2gCNRRNSaBai9ERsZjNhrvhVYZ7Hi9MUK5BSUIee8GccKy3DUWIb80kpp1GrbsQtSvUwGdA72lUJVd339qFVkiB+8+Q1BchCGKA9XU2dFcXkNAI5EEZH7UHjJpXWsftM/XNpurqrF8cJyKVQdNdavYVVSUYPTFy/j9MXL0kKhAODtJcMd7f1xR6g/urb3R9fQ+kdUiB/U3hy5otvDEOXhbDceVnrJ+e0XInJ7GrU3YjoHIaZzkN324vJqHLsyYf3YlcVBjxnLUFFjuTqR/RpyGRAR7CsFqztCrwYsjZpzrqhxGKI8nC1E6bQq3lSUiDxWiL8KIV1V0m1tgPqJ7PmllThWWIYTReV2D3NVHc5cvIwzFy8j/ZrJ7ED9/QO7hvojOtQ+YLX3539HyR5DlIezTSoP0/i4uBMiIueSyWToGOSLjkG++HWPqxPZhRC4UF6NE0XlOGkLVhfqfxaaq1FUVv/YcfKi3esFqBSIau+HqJCrjy4h/ogM8UUAR6/aJIYoD2ebVK7jpHIiIgD14So0QI3QADWG3hFit89cVYuTReU4/ouAdbbkMsqq67D/XP1ior8U4q9CF1u4am8LWH7o1M6XK7R7MIYoDyd9M0+jcnEnREStn0btjYGdgjCwk/2cq6paC86WXMap4grkFlcg90L9z1PFFSgur5Yeu06X2D1PLgM6BPkgKsQfXUL8ENnOF1Ht6/8cHugDLy4m6tYYojxcgbRaOS/nERE1l9rbC9G6AETrAq7bZ66qxekr4erUlXB1+mL9n8ur63C2pBJnSyrtlmQA6r852DHIF52CfdG5ne2nn/Rnfnuw9WOI8nCFJi60SUTkSBq1N/p1DES/joF224UQKC6vqR+5Ki6vH8W6ErLOXLyMGov1yr6KBl9Xp1Ghc3D9JcHOwb71P9v5oXOwLwJ9vTnJvRVgiPJwBVxok4jIJWQyGdoHqNA+QIXBUcF2+yxWAaO5CmcuViDv4mWcKbl85Wd9wCqrqkOhuRqF5usvEQJAgFqBzu18pZAVEeSLiGAfdAzyRXigmvOwnIQhyoNZrQJFZQxRREStjZf86i1wht5hv08IgdLLtThTcrnBkFVorkZZVR0O5ptxMN983WvLZIAuQC2Fqoig+p8dg30QEeSLMK0aCq7i3iIYojzYxYoa1FoEZLL6dU+IiKj1k8lkCPJTIshPiQERgdftr6yx4Oyly1fWuarA2ZLLOHupEucuXcbZkkpU1lqkW+PsPn3puud7yWXQa9ToGOSDiGDf+p9BvtLvOo2aE94biSHKg9kW2gzxV/HeUUREHsJH6YVuugB0a2CSuxACJRU1OHupEmdLLuPcpUqcvVT/81zJZZwrrURNnRX5pZXIL61EZu71lwoVchn0WjXCA33QMdAH4dJDjQ5X/uynYnwAWkmIWrZsGRYtWgSj0Yj+/fvjrbfewuDBg29Yv379esyZMwenT59GdHQ03njjDTzwwAPSfiEE5s2bh3//+98oLS3FXXfdhXfffRfR0dFSTUlJCZ555hl89dVXkMvlePTRR/Gvf/0L/v7+AIDvv/8e//znP7Fr1y6YzWZER0fjpZdeQlJSkuNORAuzzYcK46U8IqI2QSaToZ2/Cu38VQ2OYlmt9QuN2katbD9tQet8aSXqrKI+dF2qxK4bHEfr4y0Fqg6B6muCVv0lyvYBqjYxmuXyELVu3TpMnz4dy5cvh8FgwJtvvomEhAQcPXoUoaGh19Xv2LED48ePR0pKCh588EGsXr0aiYmJ2Lt3L/r06QMAWLhwIZYuXYqPP/4YUVFRmDNnDhISEpCTkwO1uj5QJCUloaCgAGlpaaitrcWkSZOQnJyM1atXS8fp168fZs6cCZ1Oh40bN2LChAnQarV48MEHnXeCboNttXIdv5lHREQA5HIZdBo1dBo1Yjpfv7/OYkVhWTUKroxUnS+twvnS+nBlG70qq6qDqbIWpspa5BRcPycLqF++Qa9VI1zrI4Wt8EAfhGnV0GvVCNOqofVx/28YyoQQwpUNGAwGDBo0CG+//TYAwGq1IiIiAs888wxefvnl6+rHjh2LiooKbNy4Udo2ZMgQDBgwAMuXL4cQAuHh4XjhhRfw4osvAgBMJhN0Oh1WrlyJcePG4fDhw+jVqxd2796N2NhYAEBqaioeeOABnDt3DuHh4dcdFwBGjx4NnU6HDz/8sFHvzWw2Q6vVwmQyQaPRNOm8tIRF3xzBsq0nMSGuM157uI/Tj09ERJ7HXFWLgivhKv9KwDp/Tegymqtgsd46Wqi95QjT+kCvUduFK732atgK9lVC7oIRrcZ+frt0JKqmpgZZWVmYNWuWtE0ulyM+Ph4ZGRkNPicjIwPTp0+325aQkIANGzYAAHJzc2E0GhEfHy/t12q1MBgMyMjIwLhx45CRkYHAwEApQAFAfHw85HI5MjMz8cgjjzR4bJPJhJ49e97w/VRXV6O6ulr63WxuOKE7i9FU3wtHooiIqKVo1N7Q6L3RXX/9nCygfjSrqKzaLljZ/mw01YeskooaVNXefJ0sAFB6yRGqUdmHK7vQ5YPQAJVLghbg4hBVXFwMi8UCnU5nt12n0+HIkSMNPsdoNDZYbzQapf22bTer+eWlQoVCgeDgYKnmlz799FPs3r0b77333g3fT0pKCv72t7/dcL+zGc2VADgnioiInEfhJZcu38XeoKaq1oJCcxUKTFVSsDKaqlBgqrzyswoXyqtRY7FK87OA679pCAD75t6LQF+lw97Pzbh8TpQ72Lp1KyZNmoR///vf6N279w3rZs2aZTdKZjabERER4YwWG2TkauVERNQKqb29rtzixu+GNbVXRrSMpkopbP0ydJkra6H18XZi5/ZcGqJCQkLg5eWFwsJCu+2FhYXQ6/UNPkev19+03vazsLAQYWFhdjUDBgyQaoqKiuxeo66uDiUlJdcd94cffsBDDz2Ef/7zn5gwYcJN349KpYJK1XrWYzJytXIiInJT3l5yaUHSG7FahUsnp7t08SClUomYmBikp6dL26xWK9LT0xEXF9fgc+Li4uzqASAtLU2qj4qKgl6vt6sxm83IzMyUauLi4lBaWoqsrCypZsuWLbBarTAYDNK277//HqNHj8Ybb7yB5OTk23/DTlRWVYuKGgsAhigiIvJMrpoLZePyy3nTp0/HxIkTERsbi8GDB+PNN99ERUUFJk2aBACYMGECOnTogJSUFADAc889hxEjRmDJkiUYPXo01q5diz179mDFihUA6tfImDZtGubPn4/o6GhpiYPw8HAkJiYCAHr27IlRo0bhySefxPLly1FbW4upU6di3Lhx0jfztm7digcffBDPPfccHn30UWmulFKpRHBwMFo72yiURq2Ar9Ll/5iJiIg8j2gF3nrrLdGpUyehVCrF4MGDxc6dO6V9I0aMEBMnTrSr//TTT0W3bt2EUqkUvXv3Fl9//bXdfqvVKubMmSN0Op1QqVRi5MiR4ujRo3Y1Fy9eFOPHjxf+/v5Co9GISZMmibKyMmn/xIkTBYDrHiNGjGj0+zKZTAKAMJlMjT8ZLWTbsSLReeZGce8/vnf6sYmIiNxZYz+/Xb5OlCdz5TpRn+45ixmf7cfd3drjkyduvPo7ERER2Wvs5zdvqOahCqVv5rWeie5ERESehCHKQxWYbd/Mu/G3GoiIiKj5GKI8VCHXiCIiInIohigPVXAlRHG1ciIiIsdgiPJQhVcu5/G+eURERI7BEOWBqussuFhRA4AjUURERI7CEOWBiszVAAClQo5AX9fdU4iIiMiTMUR5oGvnQ7nynkJERESejCHKAxk5H4qIiMjhGKI8kNFUCYDzoYiIiByJIcoDGU31c6K4RhQREZHjMER5oEJptXKGKCIiIkdhiPJABVcu53EkioiIyHEYojxQ4ZUlDjgSRURE5DgMUR7GahW8nEdEROQEDFEepriiGnVWAbkMaO+vcnU7REREHoshysMYryy02T5ABYUX//ESERE5Cj9lPYwtROm1Pi7uhIiIyLMxRHkY22rleg0v5RERETkSQ5SHkUaiuLwBERGRQzFEeRheziMiInIOhigPI13O0/JyHhERkSMxRHmYq5fzOBJFRETkSAxRHkQIcc1IFOdEERERORJDlAcxV9Xhco0FACeWExERORpDlAex3e5F6+MNH6WXi7shIiLybAxRHqTgynyoMF7KIyIicjiGKA9SeCVE6Xgpj4iIyOEYojwIR6KIiIichyHKg9i+mceRKCIiIsdjiPIgRlMlAI5EEREROQNDlAcxmqsBADqGKCIiIodjiPIgHIkiIiJyHoYoD1FVa8Gly7UAuNAmERGRMzBEeQjbQptqbzm0Pt4u7oaIiMjzMUR5iKs3HlZDJpO5uBsiIiLPxxDlIXjjYSIiIudyeYhatmwZIiMjoVarYTAYsGvXrpvWr1+/Hj169IBarUbfvn2xadMmu/1CCMydOxdhYWHw8fFBfHw8jh8/bldTUlKCpKQkaDQaBAYGYvLkySgvL5f2V1VV4Y9//CP69u0LhUKBxMTEFnu/jnLtSBQRERE5nktD1Lp16zB9+nTMmzcPe/fuRf/+/ZGQkICioqIG63fs2IHx48dj8uTJyM7ORmJiIhITE3Hw4EGpZuHChVi6dCmWL1+OzMxM+Pn5ISEhAVVVVVJNUlISDh06hLS0NGzcuBHbtm1DcnKytN9iscDHxwfPPvss4uPjHXcCWpBttXK91sfFnRAREbURwoUGDx4spkyZIv1usVhEeHi4SElJabB+zJgxYvTo0XbbDAaDeOqpp4QQQlitVqHX68WiRYuk/aWlpUKlUok1a9YIIYTIyckRAMTu3bulms2bNwuZTCby8/OvO+bEiRPFww8/3Kz3ZzKZBABhMpma9fymePo/e0TnmRvFR9tPOfxYREREnqyxn98uG4mqqalBVlaW3UiPXC5HfHw8MjIyGnxORkbGdSNDCQkJUn1ubi6MRqNdjVarhcFgkGoyMjIQGBiI2NhYqSY+Ph5yuRyZmZm39Z6qq6thNpvtHs5ydU4UR6KIiIicwWUhqri4GBaLBTqdzm67TqeD0Whs8DlGo/Gm9baft6oJDQ21269QKBAcHHzD4zZWSkoKtFqt9IiIiLit12sKaU4UJ5YTERE5hcsnlnuSWbNmwWQySY+zZ8865bgWq0BRWf0tX7haORERkXO4LESFhITAy8sLhYWFdtsLCwuh1+sbfI5er79pve3nrWp+OXG9rq4OJSUlNzxuY6lUKmg0GruHMxSXV8NiFfCSyxDir3LKMYmIiNo6l4UopVKJmJgYpKenS9usVivS09MRFxfX4HPi4uLs6gEgLS1Nqo+KioJer7erMZvNyMzMlGri4uJQWlqKrKwsqWbLli2wWq0wGAwt9v6cyXYpLzRABS85F9okIiJyBoUrDz59+nRMnDgRsbGxGDx4MN58801UVFRg0qRJAIAJEyagQ4cOSElJAQA899xzGDFiBJYsWYLRo0dj7dq12LNnD1asWAEAkMlkmDZtGubPn4/o6GhERUVhzpw5CA8Pl9Z66tmzJ0aNGoUnn3wSy5cvR21tLaZOnYpx48YhPDxc6i0nJwc1NTUoKSlBWVkZ9u3bBwAYMGCA085PY9mWN9BxjSgiIiKncWmIGjt2LC5cuIC5c+fCaDRiwIABSE1NlSaG5+XlQS6/Olg2dOhQrF69Gq+88gpmz56N6OhobNiwAX369JFqZsyYgYqKCiQnJ6O0tBTDhg1Damoq1OqrAWPVqlWYOnUqRo4cCblcjkcffRRLly616+2BBx7AmTNnpN8HDhwIoH4xz9bGdt88zociIiJyHplojanAQ5jNZmi1WphMJofOj1qw+QiW/3ASfxwaiVd/09thxyEiImoLGvv5zW/neQCORBERETkfQ5QHKDBVAuAaUURERM7EEOUBCs31a0Tx5sNERETOwxDl5oQQHIkiIiJyAYYoN2eurENVrRUAlzggIiJyJoYoN1dgrh+FCvL1htrby8XdEBERtR0MUW7u6o2HfVzcCRERUdvCEOXmpBCl4T3ziIiInIkhys0ZzRyJIiIicgWGKDd3dSSKk8qJiIiciSHKzV0dieLlPCIiImdiiHJznFhORETkGgxRbk4aieLlPCIiIqdiiHJjVbUWlF6uBcDVyomIiJyNIcqN2S7l+Xh7QaNWuLgbIiKitoUhyo0VXAlRYVo1ZDKZi7shIiJqWxii3FjhlflQvGceERGR8zFEubFrR6KIiIjIuRii3Jg0EsUQRURE5HQMUW6swFQJgCNRRERErsAQ5caM5moAnBNFRETkCgxRbszIkSgiIiKXYYhyU3UWKy6U1Y9EcbVyIiIi52OIclPF5TWwCkAhl6GdP28+TERE5GwMUW7KNqk8NEAFLzkX2iQiInI2hig3ZVvegPfMIyIicg2GKDdlW2iTIYqIiMg1GKLclNE2EqXxcXEnREREbRNDlJsySiNRnFRORETkCgxRbupqiOJIFBERkSswRLmpq5fzOCeKiIjIFRii3JAQQhqJ4mrlRERErsEQ5YZKL9eius4KAAjVcE4UERGRKzBEuSHbpbx2fkqoFF4u7oaIiKhtYohyQ7ZLeTrOhyIiInIZhig3ZBuJ4nwoIiIi12GIckO21cp1DFFEREQu0ypC1LJlyxAZGQm1Wg2DwYBdu3bdtH79+vXo0aMH1Go1+vbti02bNtntF0Jg7ty5CAsLg4+PD+Lj43H8+HG7mpKSEiQlJUGj0SAwMBCTJ09GeXm5Xc3+/fsxfPhwqNVqREREYOHChS3zhm9Toe2bebycR0RE5DIuD1Hr1q3D9OnTMW/ePOzduxf9+/dHQkICioqKGqzfsWMHxo8fj8mTJyM7OxuJiYlITEzEwYMHpZqFCxdi6dKlWL58OTIzM+Hn54eEhARUVVVJNUlJSTh06BDS0tKwceNGbNu2DcnJydJ+s9mM++67D507d0ZWVhYWLVqEV199FStWrHDcyWikAjNHooiIiFxOuNjgwYPFlClTpN8tFosIDw8XKSkpDdaPGTNGjB492m6bwWAQTz31lBBCCKvVKvR6vVi0aJG0v7S0VKhUKrFmzRohhBA5OTkCgNi9e7dUs3nzZiGTyUR+fr4QQoh33nlHBAUFierqaqlm5syZonv37o1+byaTSQAQJpOp0c9pjPv+8YPoPHOj2HasqEVfl4iIiBr/+e3SkaiamhpkZWUhPj5e2iaXyxEfH4+MjIwGn5ORkWFXDwAJCQlSfW5uLoxGo12NVquFwWCQajIyMhAYGIjY2FipJj4+HnK5HJmZmVLN3XffDaVSaXeco0eP4tKlSw32Vl1dDbPZbPdwhAJTJQCuVk5ERORKLg1RxcXFsFgs0Ol0dtt1Oh2MRmODzzEajTett/28VU1oaKjdfoVCgeDgYLuahl7j2mP8UkpKCrRarfSIiIho+I3fhuo6C5SK+n9sel7OIyIichmXz4nyJLNmzYLJZJIeZ8+ebfFjqBRe2PPKvTg2/34EqL1b/PWJiIiocVwaokJCQuDl5YXCwkK77YWFhdDr9Q0+R6/X37Te9vNWNb+cuF5XV4eSkhK7moZe49pj/JJKpYJGo7F7OIptNIqIiIhcw6WfxEqlEjExMUhPT5e2Wa1WpKenIy4ursHnxMXF2dUDQFpamlQfFRUFvV5vV2M2m5GZmSnVxMXFobS0FFlZWVLNli1bYLVaYTAYpJpt27ahtrbW7jjdu3dHUFDQbb5zIiIicntOmuh+Q2vXrhUqlUqsXLlS5OTkiOTkZBEYGCiMRqMQQojHH39cvPzyy1L9Tz/9JBQKhVi8eLE4fPiwmDdvnvD29hYHDhyQahYsWCACAwPF//73P7F//37x8MMPi6ioKFFZWSnVjBo1SgwcOFBkZmaK7du3i+joaDF+/Hhpf2lpqdDpdOLxxx8XBw8eFGvXrhW+vr7ivffea/R7c9S384iIiMhxGvv57fIQJYQQb731lujUqZNQKpVi8ODBYufOndK+ESNGiIkTJ9rVf/rpp6Jbt25CqVSK3r17i6+//tpuv9VqFXPmzBE6nU6oVCoxcuRIcfToUbuaixcvivHjxwt/f3+h0WjEpEmTRFlZmV3Nzz//LIYNGyZUKpXo0KGDWLBgQZPeF0MUERGR+2ns57dMCCFcOxbmucxmM7RaLUwmk0PnRxEREVHLaeznN2cnExERETUDQxQRERFRMzBEERERETUDQxQRERFRMzBEERERETUDQxQRERFRMzBEERERETUDQxQRERFRMzBEERERETWDwtUNeDLbYvBms9nFnRAREVFj2T63b3VTF4YoByorKwMAREREuLgTIiIiaqqysjJotdob7ue98xzIarXi/PnzCAgIgEwma7HXNZvNiIiIwNmzZ3lPPgfieXYOnmfn4bl2Dp5n53DkeRZCoKysDOHh4ZDLbzzziSNRDiSXy9GxY0eHvb5Go+FfUCfgeXYOnmfn4bl2Dp5n53DUeb7ZCJQNJ5YTERERNQNDFBEREVEzMES5IZVKhXnz5kGlUrm6FY/G8+wcPM/Ow3PtHDzPztEazjMnlhMRERE1A0eiiIiIiJqBIYqIiIioGRiiiIiIiJqBIYqIiIioGRii3NCyZcsQGRkJtVoNg8GAXbt2ubqlVmPbtm146KGHEB4eDplMhg0bNtjtF0Jg7ty5CAsLg4+PD+Lj43H8+HG7mpKSEiQlJUGj0SAwMBCTJ09GeXm5Xc3+/fsxfPhwqNVqREREYOHChdf1sn79evTo0QNqtRp9+/bFpk2bWvz9ukpKSgoGDRqEgIAAhIaGIjExEUePHrWrqaqqwpQpU9CuXTv4+/vj0UcfRWFhoV1NXl4eRo8eDV9fX4SGhuKll15CXV2dXc3333+PO++8EyqVCl27dsXKlSuv68dT/068++676Nevn7SYYFxcHDZv3izt5zl2jAULFkAmk2HatGnSNp7r2/fqq69CJpPZPXr06CHtd8tzLMitrF27ViiVSvHhhx+KQ4cOiSeffFIEBgaKwsJCV7fWKmzatEn89a9/FZ9//rkAIL744gu7/QsWLBBarVZs2LBB/Pzzz+I3v/mNiIqKEpWVlVLNqFGjRP/+/cXOnTvFjz/+KLp27SrGjx8v7TeZTEKn04mkpCRx8OBBsWbNGuHj4yPee+89qeann34SXl5eYuHChSInJ0e88sorwtvbWxw4cMDh58AZEhISxEcffSQOHjwo9u3bJx544AHRqVMnUV5eLtU8/fTTIiIiQqSnp4s9e/aIIUOGiKFDh0r76+rqRJ8+fUR8fLzIzs4WmzZtEiEhIWLWrFlSzalTp4Svr6+YPn26yMnJEW+99Zbw8vISqampUo0n/5348ssvxddffy2OHTsmjh49KmbPni28vb3FwYMHhRA8x46wa9cuERkZKfr16yeee+45aTvP9e2bN2+e6N27tygoKJAeFy5ckPa74zlmiHIzgwcPFlOmTJF+t1gsIjw8XKSkpLiwq9bplyHKarUKvV4vFi1aJG0rLS0VKpVKrFmzRgghRE5OjgAgdu/eLdVs3rxZyGQykZ+fL4QQ4p133hFBQUGiurpaqpk5c6bo3r279PuYMWPE6NGj7foxGAziqaeeatH32FoUFRUJAOKHH34QQtSfV29vb7F+/Xqp5vDhwwKAyMjIEELUB165XC6MRqNU8+677wqNRiOd2xkzZojevXvbHWvs2LEiISFB+r2t/Z0ICgoS77//Ps+xA5SVlYno6GiRlpYmRowYIYUonuuWMW/ePNG/f/8G97nrOeblPDdSU1ODrKwsxMfHS9vkcjni4+ORkZHhws7cQ25uLoxGo93502q1MBgM0vnLyMhAYGAgYmNjpZr4+HjI5XJkZmZKNXfffTeUSqVUk5CQgKNHj+LSpUtSzbXHsdV46j8nk8kEAAgODgYAZGVloba21u4c9OjRA506dbI713379oVOp5NqEhISYDabcejQIanmZuexLf2dsFgsWLt2LSoqKhAXF8dz7ABTpkzB6NGjrzsfPNct5/jx4wgPD0eXLl2QlJSEvLw8AO57jhmi3EhxcTEsFovdv0AAoNPpYDQaXdSV+7Cdo5udP6PRiNDQULv9CoUCwcHBdjUNvca1x7hRjSf+c7JarZg2bRruuusu9OnTB0D9+1cqlQgMDLSr/eW5bu55NJvNqKysbBN/Jw4cOAB/f3+oVCo8/fTT+OKLL9CrVy+e4xa2du1a7N27FykpKdft47luGQaDAStXrkRqaireffdd5ObmYvjw4SgrK3Pbc6xo8jOIiK4xZcoUHDx4ENu3b3d1Kx6pe/fu2LdvH0wmEz777DNMnDgRP/zwg6vb8ihnz57Fc889h7S0NKjVale347Huv/9+6c/9+vWDwWBA586d8emnn8LHx8eFnTUfR6LcSEhICLy8vK77tkJhYSH0er2LunIftnN0s/On1+tRVFRkt7+urg4lJSV2NQ29xrXHuFGNp/1zmjp1KjZu3IitW7eiY8eO0na9Xo+amhqUlpba1f/yXDf3PGo0Gvj4+LSJvxNKpRJdu3ZFTEwMUlJS0L9/f/zrX//iOW5BWVlZKCoqwp133gmFQgGFQoEffvgBS5cuhUKhgE6n47l2gMDAQHTr1g0nTpxw23+fGaLciFKpRExMDNLT06VtVqsV6enpiIuLc2Fn7iEqKgp6vd7u/JnNZmRmZkrnLy4uDqWlpcjKypJqtmzZAqvVCoPBINVs27YNtbW1Uk1aWhq6d++OoKAgqeba49hqPOWfkxACU6dOxRdffIEtW7YgKirKbn9MTAy8vb3tzsHRo0eRl5dnd64PHDhgF1rT0tKg0WjQq1cvqeZm57Et/p2wWq2orq7mOW5BI0eOxIEDB7Bv3z7pERsbi6SkJOnPPNctr7y8HCdPnkRYWJj7/vvc5Kno5FJr164VKpVKrFy5UuTk5Ijk5GQRGBho922FtqysrExkZ2eL7OxsAUD84x//ENnZ2eLMmTNCiPolDgIDA8X//vc/sX//fvHwww83uMTBwIEDRWZmpti+fbuIjo62W+KgtLRU6HQ68fjjj4uDBw+KtWvXCl9f3+uWOFAoFGLx4sXi8OHDYt68eR61xMGf//xnodVqxffff2/3deXLly9LNU8//bTo1KmT2LJli9izZ4+Ii4sTcXFx0n7b15Xvu+8+sW/fPpGamirat2/f4NeVX3rpJXH48GGxbNmyBr+u7Kl/J15++WXxww8/iNzcXLF//37x8ssvC5lMJr799lshBM+xI1377TwheK5bwgsvvCC+//57kZubK3766ScRHx8vQkJCRFFRkRDCPc8xQ5Qbeuutt0SnTp2EUqkUgwcPFjt37nR1S63G1q1bBYDrHhMnThRC1C9zMGfOHKHT6YRKpRIjR44UR48etXuNixcvivHjxwt/f3+h0WjEpEmTRFlZmV3Nzz//LIYNGyZUKpXo0KGDWLBgwXW9fPrpp6Jbt25CqVSK3r17i6+//tph79vZGjrHAMRHH30k1VRWVoq//OUvIigoSPj6+opHHnlEFBQU2L3O6dOnxf333y98fHxESEiIeOGFF0Rtba1dzdatW8WAAQOEUqkUXbp0sTuGjaf+nXjiiSdE586dhVKpFO3btxcjR46UApQQPMeO9MsQxXN9+8aOHSvCwsKEUqkUHTp0EGPHjhUnTpyQ9rvjOZYJIUTTx6+IiIiI2jbOiSIiIiJqBoYoIiIiomZgiCIiIiJqBoYoIiIiomZgiCIiIiJqBoYoIiIiomZgiCIiIiJqBoYoIiIiomZgiCIiAhAZGYk333zT1W0QkRthiCIityKTyW76ePXVV5v1urt370ZycvJt9Zabm4vf//73CA8Ph1qtRseOHfHwww/jyJEjAIDTp09DJpNh3759t3UcImodFK5ugIioKQoKCqQ/r1u3DnPnzsXRo0elbf7+/tKfhRCwWCxQKG79n7r27dvfVl+1tbW499570b17d3z++ecICwvDuXPnsHnzZpSWlt7WaxNR68SRKCJyK3q9XnpotVrIZDLp9yNHjiAgIACbN29GTEwMVCoVtm/fjpMnT+Lhhx+GTqeDv78/Bg0ahO+++87udX95OU8mk+H999/HI488Al9fX0RHR+PLL7+8YV+HDh3CyZMn8c4772DIkCHo3Lkz7rrrLsyfPx9DhgwBAERFRQEABg4cCJlMhl/96lfS899//3307NkTarUaPXr0wDvvvCPts41grV27FkOHDoVarUafPn3www8/tMAZJaLmYogiIo/z8ssvY8GCBTh8+DD69euH8vJyPPDAA0hPT0d2djZGjRqFhx56CHl5eTd9nb/97W8YM2YM9u/fjwceeABJSUkoKSlpsLZ9+/aQy+X47LPPYLFYGqzZtWsXAOC7775DQUEBPv/8cwDAqlWrMHfuXPz973/H4cOH8f/+3//DnDlz8PHHH9s9/6WXXsILL7yA7OxsxMXF4aGHHsLFixebenqIqKUIIiI39dFHHwmtViv9vnXrVgFAbNiw4ZbP7d27t3jrrbek3zt37iz++c9/Sr8DEK+88or0e3l5uQAgNm/efMPXfPvtt4Wvr68ICAgQ99xzj3jttdfEyZMnpf25ubkCgMjOzrZ73h133CFWr15tt+31118XcXFxds9bsGCBtL+2tlZ07NhRvPHGG7d8r0TkGByJIiKPExsba/d7eXk5XnzxRfTs2ROBgYHw9/fH4cOHbzkS1a9fP+nPfn5+0Gg0KCoqumH9lClTYDQasWrVKsTFxWH9+vXo3bs30tLSbviciooKnDx5EpMnT4a/v7/0mD9/Pk6ePGlXGxcXJ/1ZoVAgNjYWhw8fvul7ICLH4cRyIvI4fn5+dr+/+OKLSEtLw+LFi9G1a1f4+PjgscceQ01NzU1fx9vb2+53mUwGq9V60+cEBATgoYcewkMPPYT58+cjISEB8+fPx7333ttgfXl5OQDg3//+NwwGg90+Ly+vmx6LiFyLI1FE5PF++ukn/PGPf8QjjzyCvn37Qq/X4/Tp0w4/rkwmQ48ePVBRUQEAUCqVAGA3Z0qn0yE8PBynTp1C165d7R62ieg2O3fulP5cV1eHrKws9OzZ0+Hvg4gaxpEoIvJ40dHR+Pzzz/HQQw9BJpNhzpw5txxRaqp9+/Zh3rx5ePzxx9GrVy8olUr88MMP+PDDDzFz5kwAQGhoKHx8fJCamoqOHTtCrVZDq9Xib3/7G5599llotVqMGjUK1dXV2LNnDy5duoTp06dLx1i2bBmio6PRs2dP/POf/8SlS5fwxBNPtOj7IKLGY4giIo/3j3/8A0888QSGDh2KkJAQzJw5E2azuUWP0bFjR0RGRuJvf/ubtCSB7ffnn38eQP08pqVLl+K1117D3LlzMXz4cHz//ff405/+BF9fXyxatAgvvfQS/Pz80LdvX0ybNs3uGAsWLMCCBQuwb98+dO3aFV9++SVCQkJa9H0QUePJhBDC1U0QEdGNnT59GlFRUcjOzsaAAQNc3Q4RXcE5UURERETNwBBFRERE1Ay8nEdERETUDByJIiIiImoGhigiIiKiZmCIIiIiImoGhigiIiKiZmCIIiIiImoGhigiIiKiZmCIIiIiImoGhigiIiKiZvj/5/gVUB0JXLcAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
        "  def __init__(self, key_dim, warmup_steps=4000):\n",
        "    super().__init__()\n",
        "    self.key_dim=key_dim\n",
        "    self.warmup_steps=warmup_steps\n",
        "    self.d=tf.cast(self.key_dim, tf.float32)\n",
        "\n",
        "  def __call__(self, step):\n",
        "    step = tf.cast(step, tf.float32)\n",
        "    arg1 = tf.math.rsqrt(step)\n",
        "    arg2 = step * (self.warmup_steps ** -1.5)\n",
        "    return tf.math.rsqrt(self.d) * tf.math.minimum(arg1, arg2)\n",
        "\n",
        "  def get_config(self):\n",
        "    config = {\n",
        "        \"key_dim\": self.key_dim,\n",
        "        \"warmup_steps\": self.warmup_steps\n",
        "    }\n",
        "    return config\n",
        "\n",
        "key_dim = 512\n",
        "lr = CustomSchedule(key_dim)\n",
        "optimizer = tf.keras.optimizers.Adam(lr, beta_1=0.9, beta_2=0.98, epsilon=1e-9)\n",
        "plt.plot(lr(tf.range(50000, dtype=tf.float32)))\n",
        "plt.ylabel('Learning Rate')\n",
        "plt.xlabel('Train Step')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:26.932649Z",
          "iopub.status.busy": "2023-10-22T04:43:26.932256Z",
          "iopub.status.idle": "2023-10-22T04:43:26.940582Z",
          "shell.execute_reply": "2023-10-22T04:43:26.939591Z",
          "shell.execute_reply.started": "2023-10-22T04:43:26.932614Z"
        },
        "id": "t0qrwEIX14E8",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "# Compute masked loss\n",
        "# Since we are masking the inputs that are after the current token\n",
        "# the loss and accuracy should be updated to consider only the unmasked values.\n",
        "def masked_loss(label, pred):\n",
        "  # pred shape is (batch_size, seq_len, vocab_size)\n",
        "  mask = label != 0\n",
        "  # shape of mask is (batch_size, seq_len)\n",
        "  loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction='none')\n",
        "  loss = loss_object(label, pred)\n",
        "  mask = tf.cast(mask, dtype=loss.dtype)\n",
        "  loss *= mask\n",
        "  loss = tf.reduce_sum(loss)/tf.reduce_sum(mask)\n",
        "  return loss\n",
        "\n",
        "def masked_accuracy(label, pred):\n",
        "  pred = tf.argmax(pred, axis=2)\n",
        "  label = tf.cast(label, dtype=pred.dtype)\n",
        "  match = pred==label\n",
        "  mask = label !=0\n",
        "  match = match & mask\n",
        "  match = tf.cast(match, dtype=tf.float32)\n",
        "  mask = tf.cast(mask, dtype=tf.float32)\n",
        "  return tf.reduce_sum(match) / tf.reduce_sum(mask)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YUwhaPsp3Qur"
      },
      "source": [
        "## Final Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:26.942239Z",
          "iopub.status.busy": "2023-10-22T04:43:26.941902Z",
          "iopub.status.idle": "2023-10-22T04:43:37.961616Z",
          "shell.execute_reply": "2023-10-22T04:43:37.960551Z",
          "shell.execute_reply.started": "2023-10-22T04:43:26.942213Z"
        },
        "id": "AvEUKOmqfx4y",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "# Parameters\n",
        "n_sentences = None\n",
        "num_layers = 4\n",
        "d_model = 512\n",
        "dff = 512\n",
        "# run with 8 heads\n",
        "num_heads = 8\n",
        "dropout_rate = 0.2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:37.963351Z",
          "iopub.status.busy": "2023-10-22T04:43:37.963032Z",
          "iopub.status.idle": "2023-10-22T04:43:38.131246Z",
          "shell.execute_reply": "2023-10-22T04:43:38.130301Z",
          "shell.execute_reply.started": "2023-10-22T04:43:37.963324Z"
        },
        "id": "2p2LASBd5SYN",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "model = Transformer(\n",
        "    num_layers=num_layers,\n",
        "    d_model=d_model,\n",
        "    num_heads=num_heads,\n",
        "    dff=dff,\n",
        "    input_vocab_size=vocab_size_en,\n",
        "    target_vocab_size=vocab_size_hn,\n",
        "    dropout_rate=dropout_rate)\n",
        "lr = CustomSchedule(dff)\n",
        "optimizer = tf.keras.optimizers.Adam(lr, beta_1=0.9, beta_2=0.98, epsilon=1e-9)\n",
        "model.compile(loss=masked_loss, optimizer=optimizer, metrics=[masked_accuracy])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iw0Z-2Us3vGw"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "execution": {
          "iopub.execute_input": "2023-10-22T04:43:38.876041Z",
          "iopub.status.busy": "2023-10-22T04:43:38.875671Z"
        },
        "id": "7MrIMDm34B7c",
        "outputId": "1fece4ed-5bf6-4a56-c842-8dd0b81cd29c",
        "trusted": true
      },
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/15\n",
            "723/723 [==============================] - 157s 178ms/step - loss: 7.7071 - masked_accuracy: 0.1326 - val_loss: 6.3243 - val_masked_accuracy: 0.1883\n",
            "Epoch 2/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 5.8406 - masked_accuracy: 0.2208 - val_loss: 5.6201 - val_masked_accuracy: 0.2483\n",
            "Epoch 3/15\n",
            "723/723 [==============================] - 117s 161ms/step - loss: 5.1839 - masked_accuracy: 0.2659 - val_loss: 5.2808 - val_masked_accuracy: 0.2771\n",
            "Epoch 4/15\n",
            "723/723 [==============================] - 117s 161ms/step - loss: 4.7151 - masked_accuracy: 0.2970 - val_loss: 5.0321 - val_masked_accuracy: 0.2975\n",
            "Epoch 5/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 4.2952 - masked_accuracy: 0.3252 - val_loss: 4.9344 - val_masked_accuracy: 0.3047\n",
            "Epoch 6/15\n",
            "723/723 [==============================] - 117s 161ms/step - loss: 3.9244 - masked_accuracy: 0.3539 - val_loss: 4.7861 - val_masked_accuracy: 0.3212\n",
            "Epoch 7/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 3.4648 - masked_accuracy: 0.3990 - val_loss: 4.6848 - val_masked_accuracy: 0.3417\n",
            "Epoch 8/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 3.0670 - masked_accuracy: 0.4443 - val_loss: 4.6026 - val_masked_accuracy: 0.3597\n",
            "Epoch 9/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 2.7213 - masked_accuracy: 0.4869 - val_loss: 4.7119 - val_masked_accuracy: 0.3636\n",
            "Epoch 10/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 2.4557 - masked_accuracy: 0.5240 - val_loss: 4.6326 - val_masked_accuracy: 0.3752\n",
            "Epoch 11/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 2.1949 - masked_accuracy: 0.5622 - val_loss: 4.6839 - val_masked_accuracy: 0.3787\n",
            "Epoch 12/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 1.9959 - masked_accuracy: 0.5938 - val_loss: 4.6696 - val_masked_accuracy: 0.3863\n",
            "Epoch 13/15\n",
            "723/723 [==============================] - 116s 160ms/step - loss: 1.8303 - masked_accuracy: 0.6224 - val_loss: 4.7590 - val_masked_accuracy: 0.3843\n",
            "Epoch 14/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 1.6654 - masked_accuracy: 0.6504 - val_loss: 4.8869 - val_masked_accuracy: 0.3914\n",
            "Epoch 15/15\n",
            "723/723 [==============================] - 116s 161ms/step - loss: 1.5246 - masked_accuracy: 0.6759 - val_loss: 4.9079 - val_masked_accuracy: 0.3947\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAL3CAYAAADIljxHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACZJ0lEQVR4nOzdd3hc1Z0+8PdOn9FUadQ1Km6yLPeKTQ82htBhCcXBxiRhA6Yk3vwWnGxMS3AICevdhUDIgkmo3hBq6BiMwRhsXOQquan3Nk3S9Pv740ojyZZlyRppZqT38zzzyHPn3pkzsq15dc73nCOIoiiCiIiIKIJk0W4AERERjT4MGERERBRxDBhEREQUcQwYREREFHEMGERERBRxDBhEREQUcQwYREREFHEMGERERBRxDBhEREQUcQwYRKPQrbfeitzc3DO69sEHH4QgCGd07QsvvABBEFBWVnZG15+orKwMgiDgD3/4w2nPHUq7iSjyGDCIRpAgCAO6bd68OdpNHRNqamrw4IMPYs+ePdFuCtGoI3AvEqKR89JLL/W6/7e//Q2ffPIJXnzxxV7HlyxZgtTU1DN+Hb/fj1AoBLVaPehrA4EAAoEANBrNoK8NBoPw+/1Qq9UR6U0oKytDXl4eHn/8cfziF7/o99wzafd3332HefPmYcOGDbj11luH2Foi6kkR7QYQjSU//OEPe93/5ptv8Mknn5x0/ETt7e3Q6XQDfh2lUnlG7QMAhUIBheLMfjTI5XLI5fIzfu2hGEq7I62trQ0JCQnRbgZRVHGIhCjGXHDBBZg6dSp27tyJ8847DzqdDr/85S8BAG+//TYuu+wyZGRkQK1WY/z48XjkkUcQDAZ7PceJNRg9axmeffZZjB8/Hmq1GvPmzcOOHTt6XdtXLYMgCLjrrrvw1ltvYerUqVCr1SgsLMSHH37Y67y+ajBCoRAefPBBZGRkQKfT4cILL8TBgweRm5s7qF6DM2n3J598gnPOOQdmsxl6vR75+fnh7+XmzZsxb948AMDKlSvDw1MvvPBC+Pq///3vmDNnDrRaLaxWK374wx+iurq612vceuut0Ov1OHbsGL7//e/DYDBg2bJleOCBB6BUKtHY2HjSe7n99tthNpvh8XgG/P6J4k1sxH0i6qW5uRmXXnopbrzxRvzwhz8MD5e88MIL0Ov1WL16NfR6PT777DOsXbsWTqcTjz/++Gmf95VXXoHL5cK//uu/QhAE/P73v8e1116L48ePn7bX46uvvsIbb7yBO++8EwaDAf/93/+N6667DhUVFUhKSjrldWvWrMHvf/97XHHFFVi6dCmKioqwdOnSQX24nkm7Dxw4gMsvvxzTp0/Hww8/DLVajaNHj2Lr1q0AgIKCAjz88MNYu3Ytbr/9dpx77rkAgEWLFgGQvtcrV67EvHnzsG7dOtTX1+O//uu/sHXrVuzevRtmszn8WoFAAEuXLsU555yDP/zhD9DpdFi4cCEefvhhbNy4EXfddVf4XJ/Ph9dffx3XXXfdGQ1DEcUNkYiiZtWqVeKJ/w3PP/98EYD4zDPPnHR+e3v7Scf+9V//VdTpdKLH4wkfW7FihZiTkxO+X1paKgIQk5KSxJaWlvDxt99+WwQgvvvuu+FjDzzwwEltAiCqVCrx6NGj4WNFRUUiAPF//ud/wsc2bNggAhBLS0tFURTFuro6UaFQiFdffXWv53vwwQdFAOKKFSv6+K50G0q7//M//1MEIDY2Np7y+Xfs2CECEDds2NDruM/nE1NSUsSpU6eKHR0d4eP//Oc/RQDi2rVrw8dWrFghAhDvv//+k55/4cKF4oIFC3ode+ONN0QA4ueff97veyeKdxwiIYpBarUaK1euPOm4VqsN/9nlcqGpqQnnnnsu2tvbUVxcfNrnveGGG2CxWML3u35rP378+GmvXbx4McaPHx++P336dBiNxn6v3bRpEwKBAO68885ex+++++7Tvt5Q293Vw/D2228jFAoN6vW+++47NDQ04M477+zVy3DZZZdh8uTJeO+990665o477jjp2PLly/Htt9/i2LFj4WMvv/wybDYbzj///EG1iSjeMGAQxaDMzEyoVKqTjh84cADXXHMNTCYTjEYjkpOTwwWiDofjtM+bnZ3d637Xh3Zra+ugr+26vr9ry8vLAQATJkzodTwxMbFXYBjsaw+k3TfccAPOPvts/PjHP0ZqaipuvPFG/N///d+AwkZXu/Pz8096bPLkyeHHuygUCmRlZfXZBrVajZdffhmA9Hf0z3/+E8uWLeOaHTTqMWAQxaCePRVd7HY7zj//fBQVFeHhhx/Gu+++i08++QSPPfYYAAzog/NUMzzEAcxWH8q1Q3Umr63VarFlyxZ8+umnuOWWW7B3717ccMMNWLJkyUlFsUOlVqshk53849RiseDyyy8PB4zXX38dXq/3tLOGiEYDBgyiOLF582Y0NzfjhRdewL333ovLL78cixcvHlRPwEjLyckBABw9erTX8ebm5gH1mgyVTCbDRRddhCeeeAIHDx7Eb3/7W3z22Wf4/PPPAeCUvQhd7S4pKTnpsZKSkvDjA7F8+XIcPnwYO3bswMsvv4xZs2ahsLDwDN4NUXxhwCCKE12/xff8rd3n8+FPf/pTtJp0WhdddBEUCgWefvrpXseffPLJYX/tlpaWk47NnDkTAOD1egEgvFaF3W7vdd7cuXORkpKCZ555JnwuAHzwwQc4dOgQLrvssgG349JLL4XVasVjjz2GL774gr0XNGZwmipRnFi0aBEsFgtWrFiBe+65B4Ig4MUXXxyRIYozlZqainvvvRd//OMfceWVV+KSSy5BUVERPvjgA1it1mGtQ3j44YexZcsWXHbZZcjJyUFDQwP+9Kc/ISsrC+eccw4AYPz48TCbzXjmmWdgMBiQkJCABQsWIC8vD4899hhWrlyJ888/HzfddFN4mmpubi5+/vOfD7gdSqUSN954I5588knI5XLcdNNNw/WWiWIKezCI4kRSUhL++c9/Ij09Hf/xH/+BP/zhD1iyZAl+//vfR7tp/Xrsscfw61//Gjt27MAvfvELHD16FB9//DFEURzWdSCuvPJKZGdn4/nnn8eqVavw1FNP4bzzzsNnn30Gk8kEQPrw/+tf/wq5XI6f/vSnuOmmm/DFF18AkBbQ2rhxI3w+H+677z78+c9/xjXXXIOvvvqq1xoYA7F8+XIAUo9Oenp6RN8nUaziXiRENOLsdjssFgt+85vf4Fe/+lW0mzPsioqKMHPmTPztb3/DLbfcEu3mEI0I9mAQ0bDq6Og46dj69esBSMuijwV/+ctfoNfrce2110a7KUQjhjUYRDSsNm7ciBdeeAHf//73odfr8dVXX+HVV1/FxRdfjLPPPjvazRtW7777Lg4ePIhnn30Wd911FzdAozGFQyRENKx27dqFf//3f8eePXvgdDqRmpqK6667Dr/5zW+g1+uj3bxhlZubi/r6eixduhQvvvgiDAZDtJtENGIYMIiIiCjiWINBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRx4BBREREEceAQURERBHHgEFEREQRF3cBIxgM4te//jXy8vKg1Woxfvx4PPLIIxBFMdpNIyIiok6KaDdgsB577DE8/fTT+Otf/4rCwkJ89913WLlyJUwmE+65555oN4+IiIgACGKc/ep/+eWXIzU1Fc8991z42HXXXQetVouXXnopii0jIiKiLnHXg7Fo0SI8++yzOHz4MCZNmoSioiJ89dVXeOKJJ/o83+v1wuv1hu+HQiG0tLQgKSkJgiCMVLOJiIjiniiKcLlcyMjIgEx2mioLMc4Eg0HxvvvuEwVBEBUKhSgIgvjoo4+e8vwHHnhABMAbb7zxxhtvvEXoVllZedrP67gbInnttdfw//7f/8Pjjz+OwsJC7NmzBz/72c/wxBNPYMWKFSedf2IPhsPhQHZ2NiorK2E0Gkey6URERHHN6XTCZrPBbrfDZDL1e27cBQybzYb7778fq1atCh/7zW9+g5deegnFxcWnvd7pdMJkMsHhcDBgEBERDcJgPkPjbppqe3v7SeM+crkcoVAoSi0iIiKiE8VdkecVV1yB3/72t8jOzkZhYSF2796NJ554Arfddlu0m0ZERESd4m6IxOVy4de//jXefPNNNDQ0ICMjAzfddBPWrl0LlUp12us5REJERHRmBvMZGncBY6gYMIiIYoMoiggEAggGg9FuCvWgVCohl8v7fGwwn6FxN0RCRETxz+fzoba2Fu3t7dFuCp1AEARkZWVBr9cP6XkYMIiIaESFQiGUlpZCLpcjIyMDKpWKCx/GCFEU0djYiKqqKkycOPGUPRkDwYBBREQjyufzIRQKwWazQafTRbs5dILk5GSUlZXB7/cPKWDE3TRVIiIaHU671DRFRaR6k/i3GwHHGt14a3c1mt3e059MREQ0BjBgRMCBGidKm9qwv8YZ7aYQEdEwuuCCC/Czn/0s2s2ICwwYETA1Q5qqc6jWiUCQK4oSERExYERAblICDBoFOnxBHGtsi3ZziIiIoo4BIwJkMgFT0qVejP3Vjii3hoiIRkJrayuWL18Oi8UCnU6HSy+9FEeOHAk/Xl5ejiuuuAIWiwUJCQkoLCzE+++/H7522bJlSE5OhlarxcSJE7Fhw4ZovZVhwWmqEVKYacL2shZUtLTD0e6HSaeMdpOIiOKGKIrwB0d+YWmlXDjjWRO33norjhw5gnfeeQdGoxH33Xcfvv/97+PgwYNQKpVYtWoVfD4ftmzZgoSEBBw8eDC8eNWvf/1rHDx4EB988AGsViuOHj2Kjo6OSL61qGPAiBCTVonsRB3Km9uxv8aBsydYo90kIqK44Q+KeOrzoyP+uqsunACVYvABoytYbN26FYsWLQIAvPzyy7DZbHjrrbdw/fXXo6KiAtdddx2mTZsGABg3blz4+oqKCsyaNQtz584FAOTm5g79zcQYDpFE0NRMEwDgYI0TodCY2uKFiGhMOXToEBQKBRYsWBA+lpSUhPz8fBw6dAgAcM899+A3v/kNzj77bDzwwAPYu3dv+Nw77rgDr732GmbOnIl///d/x9dffz3i72G4sQcjgsYn66FVyeH2BlDa3IbxyUNbx52IaKxQygWsunBCVF53uPz4xz/G0qVL8d577+Hjjz/GunXr8Mc//hF33303Lr30UpSXl+P999/HJ598gosuugirVq3CH/7wh2Frz0hjD0YEyVnsSUR0RgRBgEohG/HbmdZfFBQUIBAI4Ntvvw0fa25uRklJCaZMmRI+ZrPZ8NOf/hRvvPEG/u3f/g1/+ctfwo8lJydjxYoVeOmll7B+/Xo8++yzZ/4NjEHswYiwqZkm7CxvRWlTG1wePwwaFnsSEY02EydOxFVXXYWf/OQn+POf/wyDwYD7778fmZmZuOqqqwAAP/vZz3DppZdi0qRJaG1txeeff46CggIAwNq1azFnzhwUFhbC6/Xin//8Z/ix0YI9GBGWmKBCpkULUZRqMYiIaHTasGED5syZg8svvxwLFy6EKIp4//33oVRKv1gGg0GsWrUKBQUFuOSSSzBp0iT86U9/AgCoVCqsWbMG06dPx3nnnQe5XI7XXnstmm8n4gRRFMdUNaLT6YTJZILD4YDRaByW1zhY48RHB+pg1Cpx29m53IaYiKgHj8eD0tJS5OXlQaPRRLs5dIL+/n4G8xnKHoxhMDFVD7VSBmeHHxUt7dFuDhER0YhjwBgGSrkMk9MMAID91RwmISKisYcBY5h0rYlxrNGNdl8gyq0hIiIaWQwYwyTFoEGqUYNgSMShWvZiEBHR2MKAMYymZnatieHEGKulJSKiMY4BYxjlpxmglAtoafOhxuGJdnOIiIhGDAPGMFIr5JiU2lXsyZU9iYho7GDAGGZdxZ5H6l3w+INRbg0REdHIYMAYZukmDax6FfxBESV1rmg3h4iIaEQwYAwzQRBQ2NmLsb+GwyRERDQ2MGCMgII0I+QyAQ1OL+qdLPYkIhqrcnNzsX79+gGdKwgC3nrrrWFtz3BiwBgBWpUcE1L0AFjsSUREYwMDxgiZ1jlMUlzngi8QinJriIiIhlfcBYzcXGl30hNvq1atinbT+pVl0cKsU8IXCOFIA4s9iYh6EUUg4Bv52yAWQXz22WeRkZGBUKj3L4lXXXUVbrvtNhw7dgxXXXUVUlNTodfrMW/ePHz66acR+xbt27cP3/ve96DVapGUlITbb78dbrc7/PjmzZsxf/58JCQkwGw24+yzz0Z5eTkAoKioCBdeeCEMBgOMRiPmzJmD7777LmJt64tiWJ99GOzYsQPBYPd0z/3792PJkiW4/vrro9iq0xMEAYUZJmw92oT91Q4UZpii3SQiotgR9ANf/nHkX/fcfwMUqgGdev311+Puu+/G559/josuuggA0NLSgg8//BDvv/8+3G43vv/97+O3v/0t1Go1/va3v+GKK65ASUkJsrOzh9TMtrY2LF26FAsXLsSOHTvQ0NCAH//4x7jrrrvwwgsvIBAI4Oqrr8ZPfvITvPrqq/D5fNi+fTsEQQAALFu2DLNmzcLTTz8NuVyOPXv2QKlUDqlNpxN3ASM5ObnX/d/97ncYP348zj///Ci1aOCmZBix7VgzauweNLm9sOrV0W4SERENkMViwaWXXopXXnklHDBef/11WK1WXHjhhZDJZJgxY0b4/EceeQRvvvkm3nnnHdx1111Deu1XXnkFHo8Hf/vb35CQkAAAePLJJ3HFFVfgscceg1KphMPhwOWXX47x48cDAAoKCsLXV1RU4P/9v/+HyZMnAwAmTpw4pPYMRNwFjJ58Ph9eeuklrF69OpzSYplerUBecgKONbhxoMaJ8ycln/4iIqKxQK6UehOi8bqDsGzZMvzkJz/Bn/70J6jVarz88su48cYbIZPJ4Ha78eCDD+K9995DbW0tAoEAOjo6UFFRMeRmHjp0CDNmzAiHCwA4++yzEQqFUFJSgvPOOw+33norli5diiVLlmDx4sX4wQ9+gPT0dADA6tWr8eMf/xgvvvgiFi9ejOuvvz4cRIZL3NVg9PTWW2/Bbrfj1ltvPeU5Xq8XTqez1y2apmZIG6AdqnUiEGSxJxERAEAQpKGKkb4N8pfTK664AqIo4r333kNlZSW+/PJLLFu2DADwi1/8Am+++SYeffRRfPnll9izZw+mTZsGn883HN+xk2zYsAHbtm3DokWLsHHjRkyaNAnffPMNAODBBx/EgQMHcNlll+Gzzz7DlClT8Oabbw5re+I6YDz33HO49NJLkZGRccpz1q1bB5PJFL7ZbLYRbOHJcpMSYNAo0OEL4lhjW1TbQkREg6PRaHDttdfi5Zdfxquvvor8/HzMnj0bALB161bceuutuOaaazBt2jSkpaWhrKwsIq9bUFCAoqIitLV1f25s3boVMpkM+fn54WOzZs3CmjVr8PXXX2Pq1Kl45ZVXwo9NmjQJP//5z/Hxxx/j2muvxYYNGyLStlOJ24BRXl6OTz/9FD/+8Y/7PW/NmjVwOBzhW2Vl5Qi1sG8ymYApGV3buHNNDCKieLNs2TK89957eP7558O9F4BU1/DGG29gz549KCoqws0333zSjJOhvKZGo8GKFSuwf/9+fP7557j77rtxyy23IDU1FaWlpVizZg22bduG8vJyfPzxxzhy5AgKCgrQ0dGBu+66C5s3b0Z5eTm2bt2KHTt29KrRGA5xW4OxYcMGpKSk4LLLLuv3PLVaDbU6toopCzNM2F7agoqWdjja/TDphreSl4iIIud73/seEhMTUVJSgptvvjl8/IknnsBtt92GRYsWwWq14r777ovYsLxOp8NHH32Ee++9F/PmzYNOp8N1112HJ554Ivx4cXEx/vrXv6K5uRnp6elYtWoV/vVf/xWBQADNzc1Yvnw56uvrYbVace211+Khhx6KSNtORRDFQUwCjhGhUAh5eXm46aab8Lvf/W5Q1zqdTphMJjgcDhiNxmFq4em9sasK5c3tmJ+XiLMnWKPWDiKikebxeFBaWoq8vDxoNJpoN4dO0N/fz2A+Q+NyiOTTTz9FRUUFbrvttmg35Yx1rex5sMaJUCjuMh4REVG/4jJgXHzxxRBFEZMmTYp2U87YuGQ9dCo53N4ASptZ7ElENJa8/PLL0Ov1fd4KCwuj3byIiNsajHgnlwkoSDdiZ3kr9lc7MD5ZH+0mERHRCLnyyiuxYMGCPh8b7hU2RwoDRhRNzTRhZ3krSpva4PL4YdCMjn9URETUP4PBAIPBEO1mDKu4HCIZLRITVMi0aCGKUi0GEdFYEodzDMaESP29MGBE2dTOTc/21zj5n42IxoSuIYD29vYot4T60rXyqFwuH9LzcIgkyiam6rH5sAzODj8qWtqRk5Rw+ouIiOKYXC6H2WxGQ0MDAGkNh3jYT2osCIVCaGxshE6ng0IxtIjAgBFlSrkMBWlG7Km0Y3+1kwGDiMaEtLQ0AAiHDIodMpkM2dnZQw59DBgxoDBTChjHGt1o9wWgU/GvhYhGN0EQkJ6ejpSUFPj9/mg3h3pQqVSQyYZeQcFPshiQYtAg1ahBvdODQ7VOzMlJjHaTiIhGhFwuH/JYP8UmFnnGiK6VPfdXs9iTiIjiHwNGjJiUpodSLqClzYcahyfazSEiIhoSBowYoVbIMSlVWnRlXxW3cSciovjGgBFDpmVJwyRHG1zw+INRbg0REdGZY8CIIWlGDax6FfxBESV1rmg3h4iI6IwxYMQQQRBQ2FXsWcNhEiIiil8MGDGmIM0IuUxAg9OLeieLPYmIKD4xYMQYrUqOiSnS1u37q9mLQURE8YkBIwZN7RwmKa5zwRcIRbk1REREg8eAEYOyLFqYdUr4AiEcrmexJxERxR8GjBgkCEK4F+MAiz2JiCgOMWBESiiyQxkF6UbIBAE1dg+a3N6IPjcREdFwY8CIBHsl8N1zQEdrxJ5Sr1YgL1nauv1AjTNiz0tERDQSGDCGShSBo58CbU3A7peBtuaIPXXXBmiHap0IBFnsSURE8YMBY6gEAZj2L0CCFfC6gD0vAe6GiDx1TqIOBo0CHb4gjjW2ReQ5iYiIRgIDRiSoDcDMmwF9CuBrB/a8DDhrh/y0MpmAKRlGAFwTg4iI4gsDRqSoEqSQYcwA/B6g6BXAUTXkpy3MMEEQgIqWdtjbfRFoKBER0fBjwIgkpRaYcSNgtgEBH1D0GtBaNqSnNGmVyEnSAWCxJxERxQ8GjEhTqIHpNwCJeUDQD+z9O9B8bEhPOTVDKvY8WONEKCRGopVERETDigFjOMiVwNR/AawTgVAA2P8PoLHkjJ9uXLIeOpUcbm8Apc0s9iQiotjHgDFc5Aqg8BogZTIQCgIH3gLqD5zZU7HYk4iI4gwDxnCSyYGCq4C0aYAYAg69C9QWndFTFXYOk5Q2tcHl8UeylURERBHHgDHcZDJg8mVAxixpUa7i94GqnYN+msQEFTItWoiiVItBREQUy+IyYFRXV+OHP/whkpKSoNVqMW3aNHz33XfRbtapCQIwaSlgmyfdP/IxUPHNoJ+ma2XP/TVOiCKLPYmIKHYpot2AwWptbcXZZ5+NCy+8EB988AGSk5Nx5MgRWCyWaDetf4IAjL8IkCmB8q+BY59Ls0xyz5EeG4AJKXqolTI4O/yoaGlHTlLCMDeaiIjozMRdwHjsscdgs9mwYcOG8LG8vLwotmgQBAEYd740y+T4F0DZV0DID4y7cEAhQymXoSDNiD2VduyvdjJgEBFRzIq7IZJ33nkHc+fOxfXXX4+UlBTMmjULf/nLX055vtfrhdPp7HWLupxFwITF0p8rvgWOfCLVZwxAYaY0m+RYoxvtvsBwtZCIiGhI4i5gHD9+HE8//TQmTpyIjz76CHfccQfuuece/PWvf+3z/HXr1sFkMoVvNptthFt8CrZ5QP4lUs9F9U6g5AMgdPodU1MMGqSZNAiGRByqjYGwRERE1AdBjLNqQZVKhblz5+Lrr78OH7vnnnuwY8cObNu27aTzvV4vvF5v+L7T6YTNZoPD4YDRaByRNverbh9Q/J7Ug5E6BZh8uTS9tR/7qhz49FA9EhNUWL4wB8IAaziIiIiGwul0wmQyDegzNO56MNLT0zFlypRexwoKClBRUdHn+Wq1GkajsdctpqRNA6ZcBQgyoP4gcPAtaWGufkxK00OlkKGlzYcah2dk2klERDQIcRcwzj77bJSU9F52+/Dhw8jJyYlSiyIgpQCYep3Uc9F4WFpaPHjqxbTUCjkmpRoASL0ZREREsSbuAsbPf/5zfPPNN3j00Udx9OhRvPLKK3j22WexatWqaDdtaKwTgGn/Ii0x3nwM2Pd3aUfWU5jaWex5tMEFj7//Hg8iIqKRFncBY968eXjzzTfx6quvYurUqXjkkUewfv16LFu2LNpNG7rEcdJOrHIl0FoO7H0N8Pc9BJJm1MCqV8EfFHGghr0YREQUW+KuyHOoBlOgEjXOGqDoNSDgBQxpUuhQ6U46rajSjs+KGyAIwIX5KZhhM498W4mIaMwY1UWeY4IxA5i5DFBqAVcdUPQK4HWfdNq0TBOmZZogisBnxQ34+mgTlxAnIqKYwIARqwypwKwfAmo94G4E9rwCeHqveyGTCbioIAVnjUsCAHxb2oJPDtYjGGLIICKi6GLAiGUJVqknQ2ME2puBPS8DHa29ThEEAQvHJ2FxQSoEAThQ48S7RTXwBU6/aBcREdFwYcCIdbpEKWRoLUCHHdj9MtDectJp07JMuGJGBpRyAaVNbfjHriouJU5ERFHDgBEPtGZg1jKpR8PrAna/JA2bnGB8sh7Xzs6CRilHncOD/9tRCUf7qdfTICIiGi4MGPFCbQBm3gzoUwBfmzRc4qo76bQMsxY3zLPBqFWitd2Pjd9VoMHJ1T6JiGhkMWDEE1WCFDKM6YC/Qyr8dFSddFpiggo3zLMh2aBGmzeIv++sQnlzWxQaTEREYxUDRrxRaoEZNwFmm7RORtFrQO3ek/Yv0asV+Jc5WbAl6uALhPDW7hruvkpERCOGASMeKdTAtB8Allxpz5Li94BvnwEqd/RaXlyjlOPqmRnITzMgJIr4cH8ddpa3cK0MIiIadgwY8UqhAqZdD4y7QBo68TiBo58C3zwFlG6R6jQAKOQyXDo1DbNzLACALYeb8MXhRoYMIiIaVlwqfDQIBoD6fUDl9u4prDIFkD4dyJonTXUFsLO8BVsONwEA8tMMuHhKKhRyZkwiIhqYwXyGKkaoTTSc5AogYxaQNgNoOgxUfgM4a4HqXUDNbiA5H7CdhTk56dCpFPjkYD1K6lxo9wVx+fR0aJTyaL8DIiIaZdiDMRqJImCvACq/lbZ+72LJBbIXoCKUinf31cIXCMFqUOOaWZnQq5k1iYiof4P5DGXAGO3cDUDFN0DDIUDsXD7ckIrmxFn4R6UebT4RBo0C187OQmKCKrptJSKimMaA0Y8xFzC6dNiBqu+A2t1SzQaADrkem9w5OCafAJVajatmZiDDrI1uO4mIKGYxYPRjzAaMLr52oGaXFDb8HfAHQzjQ6McRZT6aTVOxZGYexifro91KIiKKQSzypFNT6YDccwDbAqBuL5SV2zFNaIWmYS+aK4uwrzUfnjkXoXB8drRbSkREcYwBY6ySK4HMOUD6LMgbizGp4huUlh6DzHUAjs0HUVI+A5PmLYZgSIt2S4mIKA4xYIx1MhmQOgWylAKMG1+G0M5P0VRRgpbSPShtOoTcSdMgyzkLMOcAghDt1hIRUZxgwCCJIEBIzMPEJT+B/3AJju/6DHAdh+9QESa2HIfcmA5knwVY86VQQkRE1A8GDDrJlEn5UJkz8dmuEiQ79sJXdwwFoRooXW8BWgtgmw+kTZOGWYiIiPrAWSR0StX2DryzpwYBjxuTAiU4L6ESWnRupqbSAekzgJQpQEIyh0+IiMYATlPtBwPG4DS7vXhzdzVcngAMyhCuzWhGYvMeaXO1LrokIKVAuiVYo9ZWIiIaXgwY/WDAGDyXx4+39tSgyeWFSiHDldNSYQtVAfUHgJZSIBToPjnBKvVqpBSEN1kjIqLRgQGjHwwYZ8bjD+LdohpUtXZALhOwtDAN+WkGIOAFmo5IS5G3lgKhYPdF+pTOsDFZqt0gIqK4xoDRDwaMMxcIhvDRgXocrncBABaNT8K83ETIZJ31F/4OaTfXhmKgtax77xMAMKYDyQVS2NCYRr7xREQ0ZAwY/WDAGBpRFLH5cCP2VNgBAClGNZYUpCLFqOl9oq8daCqRwoa9XNrhtYspsztsqA0j13giIhoSBox+MGAMnSiKOFjrxJbDTfD4g5AJAmbnmHHWuCQo5X2skeF1d4cNR2V32BAEwJQlhY3kfEDNPVCIiGIZA0Y/GDAip80bwBeHG1FSJw2ZmLRKLC5IRXaS7tQXeV1AYwnQcBBwVHcfFwTAnC0Vh1rzpWmwREQUUxgw+sGAEXnHG934rLgBLo80m2RKhhHnTUyGViXv/0KPQ+rVaDwEOGu7jwsywJLTGTYmAUpuIU9EFAtGdcB48MEH8dBDD/U6lp+fj+Li4gFdz4AxPLyBIL4+2oyiKjtEEdCp5LggPwWTUvUQBrIIV0drd9hw1Xcfl8kBS55Ur5E0EVBqTv0cREQ0rEb9du2FhYX49NNPw/cVirh8G6OKWiHHhZNTkJ9mwKeH6tHs9uH9fbUorkvAhZNTYNScZllxrQXIWSjd2lukaa8NB4G2JqD5qHSTKYDEPCBxHKA1S9eojVIIISKimBKXn8wKhQJpadxGPBZlmLVYtiAHO8pasL20Bccb21DVWo5F45MwI8vcPaW1P7pEIPds6dbW1Bk2DgHtzdKaG01Hus8VZIDGCGjMUug48atCw2XMiYiiIC4DxpEjR5CRkQGNRoOFCxdi3bp1yM7O7vNcr9cLr9cbvu90Ovs8jyJHLhNw1rgkTEzRY9OhBlTbO7C5RCoGXTwlFVa9euBPlmAF8s4Fcs8B2hqloOFukIZUPA5pFdEOu3Rr7eN6hbrv4KExS+txsPeDiGhYxF0NxgcffAC32438/HzU1tbioYceQnV1Nfbv3w+D4eQ1Ffqq2QDAGowRIooi9lU78OWRJvgCIchlAubmWjA/NxGKvqa0Du7JpVkpHrsUME786mvr/3pBkIZYTgogFvZ+EBH1YVQXeZ7IbrcjJycHTzzxBH70ox+d9HhfPRg2m40BY4S5PH58XtKIYw1uAEBiggqLp6Qi0zyMM0QCPqmX41QBpOceKn05sfdDawEM6dLusez5IKIxaNQXefZkNpsxadIkHD16tM/H1Wo11OpBdMnTsDBolLhiejqONrjxeUkDWtp8+L8dlZieZcLZE6zQKIfhA1uhAvTJ0u1Eogj43H0Hj45Wqfcj4JVmtPSc1QJI4UKfIoUNfWpn6LAydBDR8AuFADEo/YIUCkj7P4U67/c6Hur+c9KEqMzAi/uA4Xa7cezYMdxyyy3RbgqdhiAImJhqgC1Rhy+PNGF/tQN7qxw43tiGCyenYELKCK7kKQjSMuVqAwDbyY8H/SeHj7YmwFUrBQ9nbe+1O2SK7tBhSJO+6pIA2RCHgYgofvk7pNqxtkZp+4RwAOgRCkIBad+mXoGh82tfQaLnHk8DNe9Hoz9gVFZWQhAEZGVlAQC2b9+OV155BVOmTMHtt98+oOf4xS9+gSuuuAI5OTmoqanBAw88ALlcjptuumk4m04RpFHKsWRKKianGbDpUD1a2/14t6gGE1P1uCA/BXp1DOReubLv3g9RlHo4XHVS2HDXd4YOH+CskW7h51AA+s6wYejs6dAmMnQQjTZBvzTLzd3QGSiagLYGaZuE4SQIUs+pTAEInV9lCulnTNefu45HwYi+6s0334zbb78dt9xyC+rq6rBkyRIUFhbi5ZdfRl1dHdauXXva56iqqsJNN92E5uZmJCcn45xzzsE333yD5OQ+usEpptkSdVh2Vg62l7bgu7JWHKl3o6KlHedOSMbUTOPAFugaaYIgTaPVJQKpU6Rj4dBR23mrk25BP+Cokm5d5MrOHo6u4JEu1XbE4nslot5CIak3MxwkOm8drb03dOxJY5LqttSGU3/4y+TdQUGmkKbfh8NCj3N6hQi5dF4M/+wY0SJPi8WCb775Bvn5+fjv//5vbNy4EVu3bsXHH3+Mn/70pzh+/Piwt4ErecamBpcHnx5sQL3TAwDIsmixuCAVlgRVlFt2hkIhoKOlO2y4agF3HRDso7BUoepdz2FIY+ggiqauGWo9eyPaGoG25lMXhyu1UpDQp0g1WQnJ0k0xumoAY7bI0+/3hwsuP/30U1x55ZUAgMmTJ6O2tra/S2mUSzFocOM8G3ZX2rHtWBOqWjvw0jflWDAuCXNyLJAPZIGuWCKTdf6QsQJpU6VjoZDUjRru5aiVfhMK+IDWcunWRaHuUc+RJg2taM2j7ocVUdT5Pb17I7pufk/f58sVgM7aGSSSO/+fpwCqBP5ScIIRDRiFhYV45plncNlll+GTTz7BI488AgCoqalBUlLSSDaFYpBMJmBOjgUTUvT4rLgeZU3t2Hq0CSX1LiwpSEWaKc73IZHJuus60qdLx0LBU4QOL9BaJt16Uuk61+mw9Fizo3PdDqWOP+CI+iKKQMAjTVsP90g0SUHCc4rFFwWZNBTaszciIVn6f8c6qgEZ0SGSzZs345prroHT6cSKFSvw/PPPAwB++ctfori4GG+88cawt4FDJPFBFEUU17nwxeFGdPiCEARgps2MReOtUClG+X/uUFD6wdcVOrpWLvV39H+dQtV38Ojas4Xhg0YrUZSmlnscgNfZuf6NQwoPHrv056D/1NdrjL17IxKSpVlg8hgoOI8xMb3QVjAYhNPphMViCR8rKyuDTqdDSkrKsL8+A0Z8afcFsOVwIw7VugAARq0S35ucgjxrQpRbFgV+T+e02dbOW48/e139XytTSMVm4eDRI3xwyXQpvLW3ABClYjyVnt+TWBIKScEhHB6cJ4QJ5+kXzgOkHkBdUu8eiYRk7tI8CDEbMDo6OiCKInQ6HQCgvLwcb775JgoKCrB06dIRaQMDRnwqa2rDpuIGODuk30LyrAk4b1IyEuO1CDTSgn7pB+2JwaNrHY/+5s6Hl0w/MXiYpfChUI+O3o+utU3am6UC3PaW7q8n9g4JgjSm3hU21MbudVN63uSn2SU4VomiNAwX9HbORJBJMxQEWXRmJwQDvXseTuyF8LpOv/5DeG0bo/TvVtP11QSoO+/H699XDInZgHHxxRfj2muvxU9/+lPY7XZMnjwZSqUSTU1NeOKJJ3DHHXcMexsYMOKXLxDCN8ebsbvCjpAoQiYImGEz4axxScOzEuhoEQoBXkcfwaPzz33NbOlJruz8od3zQ9bY44e5MXaKT7vea3uL9N56hgiv89RTCQFArZc+ZH1uaZhqIJSaE74fXYGkx7HhDGihoFRb4PcAgY6Bfw14+/9eAN3BQ3aqACI74f5AH+8c4vS6u8PEQNaLkMm7/71pTD2CRGd4UBvZ6zQCYjZgWK1WfPHFFygsLMT//u//4n/+53+we/du/OMf/8DatWtx6NChYW8DA0b8a23zYcuRRhxvlDYz06rkWDguCdMyTQPbDp66dY1d9xU8Ouynr/voolB1fqB2fqj2CiOmzg/aCPU2iSLgb+8RHpq7A0VHa//hQKGWCve0iSd/7Wpf1/fE65LChrfzN+heN+fpg1mXroCm1vfu/ej6Xqn0Urv8HZ1h4YSv/QWIgG9o30uZXHq/Z7I6ZKTJFVKv2YnBoStMqA2joyctzsXsNNX29vbwjqcff/wxrr32WshkMpx11lkoLy8/zdVEEkuCClfNzER5cxu2HG5Ek9uHz4obsLfKjvMmJSMnaQzWZ5wpQej84NMD5lMsmd71gepx9v6A7ToW8EofdIEmqTL/VBTq7t80T/ytX2M6ecgh4Dt5KKPra8B76teRyaUhnr6CxEBm2vT8npxK1xBD+HvRI3z0DCV+T/cqj+3N/b/uUCjU0joMCs3gvnb9xt8VMrqWohaDJ9wP9fO42Mf5wc49M/p6rs5rThzO4CyoUWdEA8aECRPw1ltv4ZprrsFHH32En//85wCAhoYG9ibQoOUkJWDZAh32VTuw7Xgzmtw+vLGrGuOSE3DexOT4XaQrlsiV3SuXnkrA1/nh6uj+kA2Hkc4gEvBJH8juRul2Kl1DDv6O/rvNu+pGeoWIzlChNg3/NEJBkNqq1PS9mV6XngHN6z4hoHX+2d8ufeAKMun5FNrTfO0RErpuQ32/gtC5SiSHGChyRnSI5PXXX8fNN9+MYDCI733ve/jkk08AAOvWrcOWLVvwwQcfDHsbOEQyOnn8QXxzvBlFlY5wfcbMbDMW5CWyPiMWdP2273H0/SF7qmmEKl3fwxla8+gp2Ova3Equ4m/wFPNitgYDAOrq6lBbW4sZM2ZA1pm6t2/fDqPRiMmTJw/76zNgjG4tbT58eUJ9xqLxSZiawfqMmBYecugMHQpN55CGNtotI6IeYjpgdKmqkjaA6tpZdaQwYIwNZU1t2HKkEc1uqQjOqlfh/EkpyE7SRbllRETxazCfoSO6JGIoFMLDDz8Mk8mEnJwc5OTkwGw245FHHkEoFANVzDRq5FoTsGxBDi6cnAKNUo4mtw//2FWFt/dUo7VtiJX3RER0WiNa5PmrX/0Kzz33HH73u9/h7LPPBgB89dVXePDBB+HxePDb3/52JJtDo5xcJmCmzYzJaQZsO96MvZUOHG9sQ3lzO2bazJjP+gwiomEzokMkGRkZeOaZZ8K7qHZ5++23ceedd6K6unrY28AhkrGr2e3Fl0eaUNok1WfoVHIsGm9FYYaR9RlERAMQs0MkLS0tfRZyTp48GS0tLSPZFBqDkvRqXD0rE1fPykRiggrtviA+PVSPl7dXoLKlPdrNIyIaVUY0YMyYMQNPPvnkSceffPJJTJ8+fSSbQmNYnjUBPzwrBxfkJ0v1GS4vXt9ZhXeLamBvZ30GEVEkjOgQyRdffIHLLrsM2dnZWLhwIQBg27ZtqKysxPvvv49zzz132NvAIRLqqcMnrZ+xt0paP0MuEzArW6rPUCtYn0FE1FPMDpGcf/75OHz4MK655hrY7XbY7XZce+21OHDgAF588cWRbAoRAGmdjAsnp2DZWdnISdIhGBLxXVkrXthahv3VDoRCUZnFTUQU96K2DkZPRUVFmD17NoLBAe5gOATswaBTEUURpU3S/iat7dKqkskGNc6flAxbItfPICKK2c3OiGKZIAgYl6xHTlICiqrs+OZ4Mxo76zMmpOhx3sRkmHSjZHlqIqJhxoBBdAK5TMDsbAsK0ozYdrwJe6scONrgRmlTG6ZmGjEvNxEGDYMGEVF/GDCITkGrkuN7k1MxPcuMLYcbUd7cjqJKB/ZXOzEty4R5uYnQq/lfiIioLyPy0/Haa6/t93G73T4SzSA6I1a9GtfOzkJlSzu2HW9GdWsH9lTYsb/KEQ4aCQwaRES9jMhPRZPJdNrHly9fPhJNITpjtkQdsixaVLV2YNuxZlTbO7C7wo791Q5MyzJjbo6FQYOIqFNMzCIZSZxFQpEgiiIqWtrxzfFm1Ng9AAClXMAMmxlzcizQqRg0iGj04SwSomEmCAJykhKQnahDebM0dFLn8OC7slbsrXJgRpYUNLQqLtZFRGMTAwbREAiCgFxrAnKSdChrbse2Y82od3qwo6wFRVV2zLKZMTvHwl1biWjMYcAgigBBEJBnTUBukg7Hm9rwzfFmNDi9+La0Bbsr7ZiVbcbsbAYNIho7GDCIIkgQBIxP1mOcNQHHGtvCi3V9e7wFeyrtmGWzYFa2mUGDiEa9Ed2LJNJ+97vfQRAE/OxnP4t2U4h6EQQBE1L0WLYgG5dPT4dVr4LXH8I3x5vx/NZSfHu8Gd7A8C+NT0QULXHbg7Fjxw78+c9/5jbvFNMEQcDEVAMmpOhxpMGNb443o9ntw9fHmrGrwo45ORbMsJm4cysRjTpx2YPhdruxbNky/OUvf4HFYol2c4hOSxAETEo14IcLcnDptDQkJqjg8Qex9WgTNmwtw3dlLfAFQtFuJhFRxMRlwFi1ahUuu+wyLF68+LTner1eOJ3OXjeiaJHJBExOM+KWs3JwydQ0WHRKdPiC+PJIEzZsLcXO8hb4gwwaRBT/4m6I5LXXXsOuXbuwY8eOAZ2/bt06PPTQQ8PcKqLBkckEFKQbkZ9qQHGdC9+WNsPe7seWw03YWd6KubmJmJZpglIel78DEBHF10qelZWVmDt3Lj755JNw7cUFF1yAmTNnYv369X1e4/V64fV6w/edTidsNhtX8qSYEgyJOFTrxLelLXB2+AEAerUCc3MtmJZpgoJBg4hiwGBW8oyrgPHWW2/hmmuugVzeXRAXDAYhCAJkMhm8Xm+vx/rCpcIplnUFjW+ON8PlCQAADBoFzp2YjEmpegiCEOUWEtFYNmoDhsvlQnl5ea9jK1euxOTJk3Hfffdh6tSpp30OBgyKB8GQiAM1DmwvbQkHjVyrDt/LT4VJp4xy64horBq1e5EYDIaTQkRCQgKSkpIGFC6I4oVcJmB6lhlT0o34rrwV20tbUNbUjhdby3DWuCTMyrZALmNvBhHFLg7sEsUwhVyGs8Yl4ZazcmBL1MEfFPHlkSa8sr0CtY6OaDePiOiU4mqIJBI4RELxShRFHKx14ssjTejwBSEIwPQsExaNt3LpcSIaEYP5DGUPBlGcEAQBhRkmrFiYiykZRogiUFTpwIvbynGk3oUx9rsCEcU4BgyiOKNVybG0MA3/MicLFp0Sbm8A/9xbi3eKauDonOJKRBRtDBhEccqWqMMPz8rBgnGJkMsEHG9sw4vbyrCzvAWhEHsziCi6GDCI4phCLsOi8VYsW5CNTIsW/qCILYelItA6hyfazSOiMYwBg2gUSNKrcf2cLCyZkgqNUo5Glxev7ajA5yUN3BaeiKKCAYNolBAEAVMzTVixKAcF6QaIIrCnwo4Xt5XjaIMr2s0jojGGAYNolNGpFLhkajqunZ0Js04JlyeAd4ukIlCnh0WgRDQyGDCIRqmcpASpCDQvETJBwLEGN17cVo5dFa0sAiWiYceAQTSKKeUyLJpgxbKzspFh1sAXCOGLkka8tqMSDU4WgRLR8GHAIBoDrHo1fjDXhsUFqVArZah3evDK9gp8cbgRvkAo2s0jolGIAYNojBAEAdOypJVA89OkItBd5a3427YyHGt0R7t5RDTKMGAQjTEJagW+Py0d18zKhEkrFYG+s6cG7xbVwMUiUCKKEAYMojEq15qAWxbmYF6uVAR6tMGNv20rx55KO4tAiWjIGDCIxjClXIZzJlpx84JspJukItDPixuw8btKNLhYBEpEZ44Bg4iQbJCKQL83OQUqhQx1Dg9e/bYSnxc3oN0XiHbziCgOKaLdACKKDTKZgBk2M8an6PFFSSMO17uwp9KOg7VOzMmxYHa2BSoFfychooHhTwsi6kWvVuCy6en4lzlZSDVKwybbjjXjha9LsbfKjiDrM4hoAARRFMfUTwun0wmTyQSHwwGj0Rjt5hDFNFEUcbjeja1Hm+DokGaYWHRKnDPRivHJegiCEOUWEtFIGsxnKIdIiOiUBEFAfpoBE1L02Ftlx7elLWht9+PdolqkmzQ4Z6IVWRZdtJtJRDGIPRhENGDeQBA7y1qxq6IV/qD0o2NccgLOmWBFkl4d5dYR0XBjDwYRDQu1Qo5FE6yYbjPj2+PN2F/txPHGNpQ2taEww4SzxiXCoFFGu5lEFAPYg0FEZ6ylzYetR5twtEFaalwpFzAr24I5ORZolPIot46IIm0wn6EMGEQ0ZDX2Dnx1pAnV9g4AgEYpx/y8RMzIMkEh52Q1otGCAaMfDBhEw0MURRxrbMPWo01oafMBAIxaJRaNT8LkNANnnBCNAqzBIKIRJwgCJqToMc6agIO1Tmw71gxnhx8f7q/DropWnDPBipykhGg3k4hGCAMGEUWUTCZgaqYJ+WkG7K6wY0dZCxqcXryxqxo5STqcM8GKFKMm2s0komHGgEFEw0Ipl2F+XiKmZZrwbWkz9lY5UN7cjvLmCkxOM2DReCtMOs44IRqtGDCIaFhpVXJckJ+CWTYLvj7WhOI6F4rrXDjS4Mb0LBMW5CVBq+KME6LRhkWeRDSiGpwefHmkCRUt7QAAlUKGebmJmJVthpIzTohiGmeR9IMBgyg2lDe34csjTWh0eQFIm6wtHJ+EKelGyGSccUIUiwbzGRp3vy48/fTTmD59OoxGI4xGIxYuXIgPPvgg2s0iokHKSUrAsgXZuGRqGoxaJdzeAD45WI+Xvi3HkXoXxtjvPkSjTtz1YLz77ruQy+WYOHEiRFHEX//6Vzz++OPYvXs3CgsLT3s9ezCIYk8gGEJRlQPbS1vg8QcBAIkJKszJsaAg3Qg5ezSIYsKYGyJJTEzE448/jh/96EenPZcBgyh2efxB7CpvxZ4qO7z+EADAoFFgdo4FUzNMUCnirtOVaFQZMwttBYNB/P3vf0dbWxsWLlzY5zlerxderzd83+l0jlTziGiQNEppM7U5uRbsq3JgV0UrXJ4AvihpxPbSFsy0mTHTZuY+J0RxIC4Dxr59+7Bw4UJ4PB7o9Xq8+eabmDJlSp/nrlu3Dg899NAIt5CIhkKtkGNubiJm2sw4VOvCd+UtsLf7se1YM3aWt2Jqpgmzs83cuZUohsXlEInP50NFRQUcDgdef/11/O///i+++OKLPkNGXz0YNpuNQyREcSQUEnGkwY0dZS3hWSdymYCCdCPm5lhgSVBFuYVEY8OYq8FYvHgxxo8fjz//+c+nPZc1GETxSxRFlDe3Y3tZC6pbpZ1bBQGYmGLAvFwLlyAnGmZjpgajSygU6tVLQUSjkyAIyLUmINeagBp7B3aUteB4YxsO17twuN6FXKsOc3MSkWXRcvdWoiiLu4CxZs0aXHrppcjOzobL5cIrr7yCzZs346OPPop204hoBGWYtbhqZiYaXV7sLG9BSZ0bZU3tKGtqR7pJg3l5iRhnTWDQIIqSuAsYDQ0NWL58OWpra2EymTB9+nR89NFHWLJkSbSbRkRRkGxQ45Kp6Vg4zo+dFS04UO1ErcODd/bUwKpXYU5OIvLTDFxLg2iEjYoajMFgDQbR6NbmDWB3hR1FVXb4AtJaGkatEnNyLCjMMHK/E6IhGHNFnoPBgEE0Nnj8QeyrdmBXeSvafdLqoDqVHLOyLZieZeJaGkRngAGjHwwYRGOLPxjCwRonvitvhbPDD0DawXV6lgmzsy1IUMfdSDFR1DBg9IMBg2hsCoVElNS78F1ZC5rcPgCAQiZgSoYRc3MSYdJx0S6i0xlz01SJiE5H1rkw1+Q0A0qb2rCjrAU1dg/2Vjmwr9qB/FQDZudYkGJQc+YJUQQwYBDRmCIIAsYl65FnTUC1vQPflbWitKkNxXUuFNe5YNWrMDndiPw0A4xcipzojHGIhIjGvAaXBzvLWnG0wY1ASPqRKAhAlkWHyWkGTEzVQ61gUSgRazD6wYBBRKfi8QdxtMGNQ7VOVHUuRQ5ItRrjU/SYnGZATlIC19SgMYs1GEREZ0CjlGNqpglTM01wdPhRUufCoVonWtp8KKlzoaTOBZ1KjklpBhSkGZFqZL0G0amwB4OIqB+iKKLB5cWhWidK6lzhNTUAIDFBhclpBkxON8KkZb0GjX4cIukHAwYRnalQSER5SzuKa5041uiGP9j94zPTokVBmhETU/VcxItGLQaMfjBgEFEkeANSvUZxrQuVre3o+kkqlwkYl5yAyWlG5FlZr0GjC2swiIiGmVohR2GGCYUZJrg83fUaTW4fjtS7caTeDY1Sjvw0PSanGZFu0rBeg8YU9mAQEUVQg8uD4lqpINTtDYSPm3VKTE4zoiDdALNOFcUWEp05DpH0gwGDiEZCKCSisrUdh2pdONboDu/sCgAZZg0mpxkxKdUArYr1GhQ/OERCRBRlMpmAnKQE5CQlwBcI4VijG8V1TpQ3t6PG7kGN3YMvDjci15qAgjQD8qwJUHAreRpFGDCIiIaZSiFDQboRBelGuL0BlNS5UFznRIPTi2MNbhxrcEOtlGFCsh4F6UZkmrWQsTiU4hyHSIiIoqTJ7UVxrRQ2XJ7ueg29WoH8NAMmpxmQzM3XKIawBqMfDBhEFGtEUURVawdK6lw43OCC199drxFezCvNyC3lKeoYMPrBgEFEsSwQDKGsuR3FdU6UNraFN18DpOLQ/DQjJqXqoVNxhJtGHgNGPxgwiChedG2+VlLXezEvmSAgJ0mHyekGjLPqoVKwOJRGBmeREBGNAj03XzuxOLS0qQ2lTW1QKWQY37lyaHaijsWhFDPYg0FEFGda2nwornWiuM4FR4c/fLxrp9fJaQakGblyKEUeh0j6wYBBRKOFKIqodXikreTrXejosdOrWadEfue28pYErhxKkcGA0Q8GDCIajYIhERWn2Ok11ajB5HQDJqUaoFdzZJzOHGswiIjGGLlMQJ41AXnW7pVDS+pcKG9uR73Tg3qnB1sON8JmkYpDJ6TooVZwmXIaPuzBICIaxdp9ARyud6O41olahyd8XCETkJcsBZLsRB0MGq6xQafHHgwiIgIA6FQKzLSZMdNmhr3dh+I6aafXlrbubeUBIEmvQnaiDjlJCcg0azn1lYaMPRhERGOMKIpocEn7oJS3SEMoPT8J5DIB6SZN52ZtOiTr1Zz+SgBY5NkvBgwiot48/iAqW9pR3tyO8pZ2OHtMfQWk9Tik3g0dbIk6mLQcThmrOERCREQDplHKMTHVgImpBoiiCHu7HxUtUtiobGmHxx/E4XoXDte7AAAWnRI5SQnITtIhy6JlsSj1Ke4Cxrp16/DGG2+guLgYWq0WixYtwmOPPYb8/PxoN42IKO4JggBLggqWBBVm2MwIhkTUOT2oaG5HRUsb6hxetLb70dpux55KO2SCNJySnST1cKQaNBxOIQBxOERyySWX4MYbb8S8efMQCATwy1/+Evv378fBgweRkJBw2us5REJEdOY8/iCqWjtQ0dKG8uZ22Nt7D6eolTLYLFLYyElM4A6wo8yYqsFobGxESkoKvvjiC5x33nmnPZ8Bg4gochzh4ZQ2VLS099pqHpBWFO2q38iy6KBRcjglno2pGgyHwwEASExM7PNxr9cLr9cbvu90OkekXUREY4FJp8Q0nQnTskwIhUTUu6ThlPKWdtTaPbC3+2Fvd2BvlQOCAKQZpeGULLMOqSY16zdGsbjuwQiFQrjyyitht9vx1Vdf9XnOgw8+iIceeuik4+zBICIaXt5AENWtHShvaUdFczta2ny9HhcEwKpXI8OsQbpJiwyTFkatgpu0xbAxM0Ryxx134IMPPsBXX32FrKysPs/pqwfDZrMxYBARjTCnx4+KZmlmSo3Dc9J0WEDaETbdrEWGSYN0sxapBjUUci76FSvGRMC466678Pbbb2PLli3Iy8sb8HWswSAiig1ubwB1jg7U2D2odXSg3ulFMNT7I0kuE5BiUPcKHdywLXpGdQ2GKIq4++678eabb2Lz5s2DChdERBQ79GoFJqQYMCHFAAAIBENocHlR2yN0tHmDqHV4UOvwYFfndQaNAhlmLdJNGmSYtbDq1ZBzamzMibsejDvvvBOvvPIK3n777V5rX5hMJmi12tNezx4MIqL4IIoinB0B1Dg6UOvoQK3Dg0aXFyd+ainlAlKNmnDoSDdpoVWxeHQ4jOohklMV/2zYsAG33nrraa9nwCAiil++QAj1Tg9q7FLgqHF0nDQ1FgASE1ThsJFu1iApQcXi0QgY9UMkREQ0NqkUMtgSpT1RAOkzoaXNJ4WNztDR0uYL3w7USEsTqJWycODIMGk5RXYExF3AICIi6iIIApL0aiTp1ZiaaQIgrTZa6/Cg1t6BGocHdZ29HGVN7Shrau+8DkjSq6XCUZMWGWYNTFolezkiiAGDiIhGFY1SjjxrAvKs0vYRoZCIJrcXNT1Ch7PDjyaXF00uL/ZWSQs2alXycOFoukmDVKMGSk6RPWMMGERENKrJZAJSjBqkGDWYaTMDOHmKbIPTiw5fEMcb23C8sU26ThCQbFAj3axBRmcth0HNhcAGigGDiIjGnL6myDa6veHAUWv3wO0NoN7pQb3Tgz2wh69LN3cPqyTruRDYqTBgEBHRmKeQy6QZJyYtAAtEUYTLG0CtXZqpUmuXpsi6vQEcqXfjSL1buk4mIMWoDgeONBMXAuvC7wIREdEJBEGAUaOEMU2J/DSpl8MflKbI9pyx0uELosbuQY3dg53l0rVGrTK86miGSQOrXg3ZGFwIjAGDiIhoAJRyGbIs0rbzgDRF1tHhDw+r1Dg8aHZ74ezww9nhR3Gdq/M6qQYk2aBGikGNZIMaSQmjf/VRBgwiIqIzIAgCzDoVzDoVpmRIi055A0HUO7y9Vh/1+kOobu1AdWtH+Fq5TIBVr+4VOqx6NVSK0VPPwYBBREQUIWqFHNlJOmQn9V4IrN7pRYNLquNodHvh9YfCBaRdBAGw6FRIMaiRYlQjWa9BilENjTI+FwRjwCAiIhomPRcCmwKpl6Nrj5UGlwcNLi8aXVL4aPMGwyuQdg2vANLmbilGacZKilHq7YiH6bIMGERERCNIEASYdEqYdEpMTDWEj7d5A70CR4PTC0eHHy5PAC6PG8ca3OFztSp5eGglxSDVd1h0sbUSKQMGERFRDEhQK5CnVoRXIAWkZc+7hlUanNLXFrcPHb4gypvbUd7cHj5XpZDBqleFA0eKQeo5iVYxKQMGERFRjNIo5b02dwOkRcGa3L5wT0ejy4smtxe+QCg8ZbaLXCbgmlmZva4fKQwYREREcUQhlyHNpEGaSQNA2uAtFBLR2u5Dg8vba5jF6w/BrFNGp51ReVUiIiKKGJmsu5i0IF06JooinJ5A1FYWZcAgIiIahQRBgEkbnd4LABg9K3oQERFRzGDAICIioohjwCAiIqKIY8AgIiKiiGPAICIioohjwCAiIqKIY8AgIiKiiGPAICIioohjwCAiIqKIG3MreYqiCABwOp1RbgkREVF86frs7Pos7c+YCxgulwsAYLPZotwSIiKi+ORyuWAymfo9RxAHEkNGkVAohJqaGhgMBgiCEO3mDJnT6YTNZkNlZSWMRmO0mzPs+H5Hv7H2nsfa+wXG3nseTe9XFEW4XC5kZGRAJuu/ymLM9WDIZDJkZWVFuxkRZzQa4/4f7mDw/Y5+Y+09j7X3C4y99zxa3u/pei66sMiTiIiIIo4Bg4iIiCKOASPOqdVqPPDAA1Cr1dFuyojg+x39xtp7HmvvFxh773msvd8uY67Ik4iIiIYfezCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4hgwiIiIKOIYMIiIiCjiGDCIiIgo4mIiYDz11FPIzc2FRqPBggULsH379lOee8EFF0AQhJNul1122Qi2mIiIiPoT9YCxceNGrF69Gg888AB27dqFGTNmYOnSpWhoaOjz/DfeeAO1tbXh2/79+yGXy3H99dePcMuJiIjoVARRFMVoNmDBggWYN28ennzySQBAKBSCzWbD3Xffjfvvv/+0169fvx5r165FbW0tEhISTnt+KBRCTU0NDAYDBEEYcvuJiIjGClEU4XK5kJGRAZms/z4KxQi1qU8+nw87d+7EmjVrwsdkMhkWL16Mbdu2Deg5nnvuOdx4442nDBderxderzd8v7q6GlOmTBlaw4mIiMawyspKZGVl9XtOVANGU1MTgsEgUlNTex1PTU1FcXHxaa/fvn079u/fj+eee+6U56xbtw4PPfTQSccrKythNBoH32giIqIxyul0wmazwWAwnPbcqAaMoXruuecwbdo0zJ8//5TnrFmzBqtXrw7f7/rmGI1GBgwiIqIzMJASg6gGDKvVCrlcjvr6+l7H6+vrkZaW1u+1bW1teO211/Dwww/3e55arYZarR5yW4mIiGjgojqLRKVSYc6cOdi0aVP4WCgUwqZNm7Bw4cJ+r/373/8Or9eLH/7wh8PdTCIiIhqkqA+RrF69GitWrMDcuXMxf/58rF+/Hm1tbVi5ciUAYPny5cjMzMS6det6Xffcc8/h6quvRlJSUjSaTURERP2IesC44YYb0NjYiLVr16Kurg4zZ87Ehx9+GC78rKioOGkqTElJCb766it8/PHH0WgyERERnUbU18EYaU6nEyaTCQ6Hg0WeREREgzCYz9Cor+RJREREow8DBhEREUUcAwYRERFFHAMGERHRKOPxB7Gn0o63dlcjFIpOqWXUZ5EQERHR0ImiiBqHB/urHThS74I/KAWLipZ25FpPvxlopDFgEBERxbEOXxCH6pzYX+1As9sXPm7VqzA104Q0kyYq7WLAICIiijOiKKKqtQP7qx042uBGoHMYRCkXMCnVgGlZJqQZNQPaM2S4MGAQERHFiXZfAIdqndhX5UBruz98PNmgxrRME/LTDNAo5VFsYTcGDCIiohgmiiIqWzqwr9qBY41uBDt7K1QKGfI7eytSDOqo9lb0hQGDiIgoBrV5AzjY2Vvh6OjurUgzaTA1w4RJaXqoFbHRW9EXBgwiIqIYEQqJqGhpx75qB443tiEkdvdWFKQbMDXThBRDdIo2B4sBg4iIKMpcHj8O1DhxoMYJZ4/eigyzBlMzTZiYYoBKEV9LVzFgEBERRUEoJKKsuQ37qh0obWpD19ajGqUck9MNmJZpglWvjm4jh4ABg4iIaAQ5PX7sr3bgYI0TLk8gfDzTosW0TBMmpOihlMdXb0VfGDCIiIiGWTAkorSpDfurHShr7u6t0KrkmJJuxNRMExITVNFtZIQxYBAREQ0Tl8ePfVUO7K9xoM0bDB+3JeowLdOE8ckJUIyC3oq+MGAQERFFWK2jA7sr7DhS7w7PBNGp5CjMMGFqphFm3ejqregLAwYREVEEBEMijjS4sLvCjjqHJ3w806LFLJsZ45L1kMtiazGs4cSAQURENATtvgD2VTmwt8oBt1cq2pTLBOSnGTDLZkaKMT7WrYg0BgwiIqIz0ODyYE+FHSV1rvBmYwlqOaZnmTE9ywSdamx/xI7td09ERDQIoZCI401t2F3RiqrWjvDxNJMGM21mTEo1jKlhkP4wYBAREZ2Gxx/EgRoniirt4X1BZIKAial6zLSZkW6K7tbosYgBg4iI6BRa2nwoqrTjYK0TvkAIgLTS5vQsE6ZnmWDQKKPcwtjFgEFERNSDKIoob27Hnko7SpvawsetehVm2iyYnG4YFSttDjcGDCIiIgC+QAiHap3YU2lHS5sPACAIQJ41AbNsFtgStRwGGQQGDCIiGtMcHX4UVdqxv8YBr18aBlEpZCjMMGKmzTwmFsUaDgwYREQ05oiiiGq7tNrmsUZ3eG8Qs06JmTYzpmQYoVbIo9vIOMeAQUREY0YgGEJJvbTaZqPLGz6enajDrGwz8qwJHAaJEAYMIiIa9dq8ARRV2bGvyoF2n7TpmFIuYHKaETOzzbDq1VFu4ejDgEFERKNWo8uLXRWtKKlzIdi52qZBo8AMmxlTM0zQqjgMMlwYMIiIaFQRRRFlze3YVd6Kipb28PEMswazsi2YkKyHjKttDjsGDCIiGhX8QWma6e6K3tNMJ6YYMDvHjHSTNsotHFsYMIiIKK65vQHsrbRjb7UDHZ31FSqFDNMyTZhhM8Ok5Wqb0cCAQUREcanB5cHuzt1Mu+orjFolZmWbUchpplHHgEFERHFDFEWUNrVhV4UdlT3qKzLNWszKNmM86ytiBgMGERHFvL7qK7p2M52dbUGaSRPlFtKJor5by1NPPYXc3FxoNBosWLAA27dv7/d8u92OVatWIT09HWq1GpMmTcL7778/Qq0lIqKR5PYGsPVoE/73y1JsOtSAljYf1EoZ5uRYsPKcXHx/WjrDRYyKag/Gxo0bsXr1ajzzzDNYsGAB1q9fj6VLl6KkpAQpKSknne/z+bBkyRKkpKTg9ddfR2ZmJsrLy2E2m0e+8URENGwanB7sqrDjcH13fYWps76Cy3jHB0EUu1ZgH3kLFizAvHnz8OSTTwIAQqEQbDYb7r77btx///0nnf/MM8/g8ccfR3FxMZTKM6sKdjqdMJlMcDgcMBqNQ2o/ERFFjiiKON7Uht0n1ldYtJidbcY4K+srom0wn6FR68Hw+XzYuXMn1qxZEz4mk8mwePFibNu2rc9r3nnnHSxcuBCrVq3C22+/jeTkZNx888247777IJf3nWa9Xi+83u715p1OZ2TfCBERDUnXNum7K1rR2u4HINVXTErVYxbrK+JW1AJGU1MTgsEgUlNTex1PTU1FcXFxn9ccP34cn332GZYtW4b3338fR48exZ133gm/348HHnigz2vWrVuHhx56KOLtJyKioXF5/Nhb5cDeKgc8fmn9CrWye/0Ko4brV8SzuJpFEgqFkJKSgmeffRZyuRxz5sxBdXU1Hn/88VMGjDVr1mD16tXh+06nEzabbaSaTEREJ5DqK1pRUudGqHOU3qxTYla2BVPSjVApoj7/gCIgagHDarVCLpejvr6+1/H6+nqkpaX1eU16ejqUSmWv4ZCCggLU1dXB5/NBpVKddI1arYZazV3yiIiirdrege2lzShrOrG+woJx1gTWV4wyUYuJKpUKc+bMwaZNm8LHQqEQNm3ahIULF/Z5zdlnn42jR48iFAqFjx0+fBjp6el9hgsiIoq+ypZ2vL6zCv+3oxJlTe2QCQIK0g24eUE2fjDXhgkpLN4cjaI6RLJ69WqsWLECc+fOxfz587F+/Xq0tbVh5cqVAIDly5cjMzMT69atAwDccccdePLJJ3Hvvffi7rvvxpEjR/Doo4/innvuiebbICKiE4iiiMqWDnxT2ozq1g4AgFwmYEq6EfNyE2HSsb5itItqwLjhhhvQ2NiItWvXoq6uDjNnzsSHH34YLvysqKiATNbdyWKz2fDRRx/h5z//OaZPn47MzEzce++9uO+++6L1FoiIqAdRFFHe3I5vS5tRY/cAkILF1Ewj5uYmsnBzDInqOhjRwHUwiIgir2sNi2+Pt6DeKQULhUzA1CwT5uZYYGCwGBXiYh0MIiKKf6Io4lijG9+WtqDBKa05pJQLmJ5lxpwcCxLU/JgZq/g3T0REgxYKiTja6Ma3x5vR5JY2H1MpZJiRZcbsHDN0Kn68jHX8F0BERAMWCokoqXdhR1kLmnsEi1k2M2ZlW6BVcY8QkjBgEBHRaYVCIg7VObGjtCW8nLdaKcMsmwWzss3QKBksqDcGDCIiOqVgSMShWie2l7bA0SEFC41Sjjk5FsywmbirKZ0SAwYREZ0kEAzhYGewcHkCAACdSgoW07IYLOj0GDCIiCjMHwxhf7UDO8tbw8EiQS3HnJxETMs0cZ8QGjAGDCIigj8Ywt4qB3aWt6DNK+1satAoMCfHgqmZJijlDBY0OAwYRERjmC8Qwt4qO3aWt6Ld1x0s5uclYkq6EQoGCzpDDBhERGOQNxBEUaUDuypa0dEZLExaJebnJaIg3Qg5Nx+jIWLAICIaQ0RRRHGdC18daYLbK9VYmHVSsJicxmBBkcOAQUQ0RjS4PNhc3Ihqu7S7qVmnxFnjkpCfauB26RRxDBhERKOcxx/E18easLfKAVGU9gqZn5eE2dlm1ljQsGHAICIapUIhEQdqnNh6rClcZzEp1YBzJ1m5bToNOwYMIqJRqNbRgc+LG8Nbp1v1KlyQnwJboi7KLaOxggGDiGgUafMG8NXRJhyscQKQNiJbOD4JM7LMLOCkEcWAQUQ0CgRDIoqq7Nh2rBm+QAgAUJhhxNkTrEhQ80c9jTz+qyMiinOVLe3YXNKAps7t01ONGlw4ORnpJm2UW0ZjGQMGEVGccnr8+PJwEw7XuwAAWpUcZ4+3ojDDyGmnFHUMGEREcSYQDGFXhR3bS5vhD4oQBGBGlhkLxydBo+QupxQbGDCIiOLI8UY3vjjcCHu7HwCQadbigsnJSDFootwyot4YMIiI4oC93YcvDjfieGMbAECvVuCciVZMTjNAEDgcQrGHAYOIKIb5AiF8V9aC78pbEQyJkAkCZueYMT8vEWoFh0ModjFgEBHFIFEUcaTBjS2HG+HySJuS5STpcEF+ChITVFFuHdHpMWAQEcWYZrcXn5c0orKlHQBg1Cpx/iQrxifrORxCcYMBg4goRnj8QXxb2oI9FXaERBEKmYC5uYmYm2uBkpuSUZxhwCAiijJRFHGw1omtR5vQ5pU2JRufosf5E5Nh0nFTMopPDBhERFFU7/Rgc0kDauzSpmQWnRIX5Kcg15oQ5ZYRDQ0DBhFRFLR5A9h6tAkHa50QRWlTsgV5iZiVbeGmZDQqMGAQEY2gQDCEPZV2fFvaEt6UbHKaAedMtMKg4XAIjR4MGEREI0AURRxrbMOXR7pX4Uw1anBBfjIyzNyUjEYfBgwiomHW5Pbii5JGVHROO01Qy3H2BCumpBs57ZRGLQYMIqJh0uELYtvxJuytckAUAYVMwOwcC+bmWrgKJ416DBhERBEWDInYW2XHN8db4PFL004npOhxHqed0hjCgEFEFEFlTW3YcqQRzW4fAMBqUOOCScmwJeqi3DKikcWAQUQUAa1tPmw50r3bqVYlx6LxSZiaYYKM005pDIqJtWefeuop5ObmQqPRYMGCBdi+ffspz33hhRcgCEKvm0ajGcHWEhF18/iD+OJwI178phzHG9s6dzu14NZFuZieZWa4oDEr6j0YGzduxOrVq/HMM89gwYIFWL9+PZYuXYqSkhKkpKT0eY3RaERJSUn4PquwiWikhUIiDtQ48fWxJrT7pDqLPGsCzpuUzN1OiRADAeOJJ57AT37yE6xcuRIA8Mwzz+C9997D888/j/vvv7/PawRBQFpa2kg2k4gorLKlHV8cbkSjywsASExQ4fxJyVzem6iHqAYMn8+HnTt3Ys2aNeFjMpkMixcvxrZt2055ndvtRk5ODkKhEGbPno1HH30UhYWFfZ7r9Xrh9XrD951OZ+TeABGNKY4OP7480ogj9W4AgFopw1njkjAjy8zlvYlOENUajKamJgSDQaSmpvY6npqairq6uj6vyc/Px/PPP4+3334bL730EkKhEBYtWoSqqqo+z1+3bh1MJlP4ZrPZIv4+iGh08wVC2Hq0CX/7ugxH6t0QBGCGzYSVi/Iwm3uHEPUp6kMkg7Vw4UIsXLgwfH/RokUoKCjAn//8ZzzyyCMnnb9mzRqsXr06fN/pdDJkENGAiKKIQ7UubD3aBLc3AACwJepw/qRkJBvUUW4dUWyLasCwWq2Qy+Wor6/vdby+vn7ANRZKpRKzZs3C0aNH+3xcrVZDreYPAiIanFpHBzaXNKLOIW2jbtIqcd6kZIxPTmBhOdEARHWIRKVSYc6cOdi0aVP4WCgUwqZNm3r1UvQnGAxi3759SE9PH65mEtEY4vL48eH+Wry2vRJ1Dg9UChnOmWjF8oU5mJCiZ7ggGqCoD5GsXr0aK1aswNy5czF//nysX78ebW1t4Vkly5cvR2ZmJtatWwcAePjhh3HWWWdhwoQJsNvtePzxx1FeXo4f//jH0XwbRBTn/MEQdpa34ruyFviDIgQBmJJuxNkTrEhQR/1HJVHcifr/mhtuuAGNjY1Yu3Yt6urqMHPmTHz44Yfhws+KigrIZN0dLa2trfjJT36Curo6WCwWzJkzB19//TWmTJkSrbdARHGuorkdnx6qh6ND2kY906zF+fnJSDVyET+iMyWIoihGuxEjyel0wmQyweFwwGg0Rrs5RBRF3kAQXx5uwr5qBwDAoFHg3InJmJTKoRCivgzmMzTqPRhERNFQ1tSGTw/Vw+WRZofMsJlw9gQrt1EnihAGDCIaUzz+ILYcbsSBGmnRPZNWiSVTUrnbKVGEMWAQ0ZhxvNGNTYca4PYGIAjATJsZi8ZboVLExL6PRKMKAwYRjXoefxCbSxpwqNYFALDolFhSmIZMszbKLSMavRgwiGhUO9rgwmfFDWjzBiEIwOxsCxaOT4JSzl4LouHEgEFEo1K7L4DPixtxuF7qtUjSq7BkSirSTey1IBoJDBhENKqIoojD9W58XtKADl8QMkHA3FwLFuQlQsFeC6IRw4BBRKNGmzeAz4obcLRB2k7dalDj4impXDCLKAoYMIgo7omiiOI6FzaXNMLjl3ot5uclYn5eIrdSJ4qSQQeM3Nxc3Hbbbbj11luRnZ09HG0iIhowtzeATYfqcbyxDQCQYlRjyZRUpBjYa0EUTYMekPzZz36GN954A+PGjcOSJUvw2muvwev1DkfbiIhOSRRF7K924G/bynC8sQ1ymYBF45Nw47xshguiGHDGe5Hs2rULL7zwAl599VUEg0HcfPPNuO222zB79uxItzGiuBcJUfxzevzYdKgeZU3tAIA0kwZLpqTCqldHuWVEo9tgPkOHvNmZ3+/Hn/70J9x3333w+/2YNm0a7rnnHqxcuTImNwtiwCCKX1KvhRNbjjTCFwhBIROwcHwSZmdbIGOtBdGwG5HNzvx+P958801s2LABn3zyCc466yz86Ec/QlVVFX75y1/i008/xSuvvHKmT09E1Iujw49PD9ajokXqtcgwa7BkShoSE1RRbhkR9WXQAWPXrl3YsGEDXn31VchkMixfvhz/+Z//icmTJ4fPueaaazBv3ryINpSIxiZRFFFU5cDWo03wBUJQygUsHG/FLJuZvRZEMWzQAWPevHlYsmQJnn76aVx99dVQKpUnnZOXl4cbb7wxIg0korHL3u7DJwfrUdXaAQDItGhx8ZRUmHXstSCKdYMOGMePH0dOTk6/5yQkJGDDhg1n3CgiGttCIRF7quz4+mgT/EERKoUMZ0+wYkaWKSZru4joZIMOGA0NDairq8OCBQt6Hf/2228hl8sxd+7ciDWOiMaeljYfPjlYhxq7BwBgS9RhSUEqTLqTe0uJKHYNeh2MVatWobKy8qTj1dXVWLVqVUQaRURjjyiK2F3Ripe/KUeN3QOVQoaLClJw3exMhguiODToHoyDBw/2udbFrFmzcPDgwYg0iojGFrc3gE8O1oXXtchJ0mHxlFQYNQwWRPFq0AFDrVajvr4e48aN63W8trYWCgW3NiGiwTna4MKnh6SdTxUyAedNSsZ01loQxb1BD5FcfPHFWLNmDRwOR/iY3W7HL3/5SyxZsiSijSOi0csbCOLjA3V4t6gWHb4gUoxq3LwgGzNsZoYLolFg0F0Of/jDH3DeeechJycHs2bNAgDs2bMHqampePHFFyPeQCIafWrsHfhwfx0cHX4IAjA3JxELxydx51OiUWTQASMzMxN79+7Fyy+/jKKiImi1WqxcuRI33XRTn2tiEBF1CYZEfFvajO2lLRBFwKBR4JKpaciy6KLdNCKKsDMqmkhISMDtt98e6bYQ0SjW2ubDRwfqUOuQpp8WpBtwQX4KNEp5lFtGRMPhjKsyDx48iIqKCvh8vl7Hr7zyyiE3iohGD1EUcaDGiS8OSxuUqZUyXDQ5Fflphmg3jYiG0Rmt5HnNNddg3759EAQBXZuxdhVlBYPByLaQiOJWuy+ATw814FiDG4C0aNbFhZx+SjQWDHoWyb333ou8vDw0NDRAp9PhwIED2LJlC+bOnYvNmzcPQxOJKB6VNrXhpW/KcazBDblMwHmTrLhudibDBdEYMegejG3btuGzzz6D1WqFTCaDTCbDOeecg3Xr1uGee+7B7t27h6OdRBQn/MEQvjrShD2VdgCAVa/C0qlpSDFootswIhpRgw4YwWAQBoM0dmq1WlFTU4P8/Hzk5OSgpKQk4g0kovjR4PTgg/11aGmTarNmZZtx9gQrlPJBd5YSUZwbdMCYOnUqioqKkJeXhwULFuD3v/89VCoVnn322ZNW9ySisSEUErGzohVfH21GSBShVytwcWEqcpISot00IoqSQQeM//iP/0BbWxsA4OGHH8bll1+Oc889F0lJSdi4cWPEG0hEsc3R4cdHB+pQ3doBAJiQosfiglRoVZx+SjSWCWLXNJAhaGlpgcViiYvlfZ1OJ0wmExwOB4xGY7SbQxS3RFFEcZ0LnxU3wBcIQaWQ4YL8ZExJN8bFzwIiGrzBfIYOqgfD7/dDq9Viz549mDp1avh4YmLimbWUiOKSxx/EZ8UNKKlzAQAyzBpcUpjObdWJKGxQAUOpVCI7O5trXRCNYZUt7fjoQB1cngBkgoCzxiViXm4iZNxHhIh6GHRp969+9Sv88pe/REtLS8Qa8dRTTyE3NxcajQYLFizA9u3bB3Tda6+9BkEQcPXVV0esLUTUt0AwhC2HG/GPXVVweQKw6JS4YZ4NC8YlMVwQ0UkGXeT55JNP4ujRo8jIyEBOTg4SEnpXie/atWtQz7dx40asXr0azzzzDBYsWID169dj6dKlKCkpQUpKyimvKysrwy9+8Quce+65g30LRDRITW4vPthfhyaXFwAwPcuEcycmQ6Xg9FMi6tugA0akewueeOIJ/OQnP8HKlSsBAM888wzee+89PP/887j//vv7vCYYDGLZsmV46KGH8OWXX8Jut0e0TUQkEUURuyvt2HqkCYGQCJ1KjsVTUjE+WR/tphFRjBt0wHjggQci9uI+nw87d+7EmjVrwsdkMhkWL16Mbdu2nfK6hx9+GCkpKfjRj36EL7/8MmLtIaJubm8AHx+oQ3lzOwBgXHICFhekIkF9xnskEtEYEtWfFE1NTQgGg0hNTe11PDU1FcXFxX1e89VXX+G5557Dnj17BvQaXq8XXq83fN/pdJ5xe4nGiiP1Lnx6qAEefxBKuYDzJiVjWqaJ00+JaMAGHTBkMlm/P2SGc4aJy+XCLbfcgr/85S+wWq0DumbdunV46KGHhq1NRKOJNxDE5pJGHKyRgniqUYNLpqYhMUEV5ZYRUbwZdMB48803e933+/3YvXs3/vrXvw76g9xqtUIul6O+vr7X8fr6eqSlpZ10/rFjx1BWVoYrrrgifCwUCgEAFAoFSkpKMH78+F7XrFmzBqtXrw7fdzqdsNlsg2on0VhQ1dqOjw7Uw9nhhyAA83MTsWBcEuScIUIUHaIIBH2dNz8Q8Pa47+u87weC3hPO6XEs4ANm3ABoLSPe/EEHjKuuuuqkY//yL/+CwsJCbNy4ET/60Y8G/FwqlQpz5szBpk2bwsWjoVAImzZtwl133XXS+ZMnT8a+fft6HfuP//gPuFwu/Nd//VefwUGtVkOtVg+4TURjTTAkYtuxZnxX3gJRBExaJZZOTUOmWRvtphHFt4AX8LoArxPwezo/9PsICqcKD6GAFDKGyu8BovDfOWI1GGeddRZuv/32QV+3evVqrFixAnPnzsX8+fOxfv16tLW1hWeVLF++HJmZmVi3bh00Gk2vFUQBwGw2A8BJx4no9JrcXny4vw6NndNPCzOMOD8/GWoF9xEhOiVR7B0evK4Tbk7pFvBF5vUEAZArAbkakKsAhUr62vMWPqaWzlWoux/TRWe17YgEjI6ODvz3f/83MjMzB33tDTfcgMbGRqxduxZ1dXWYOXMmPvzww3DhZ0VFBWQyzrUniqQTp59qVXIsLkjBhBRDtJtGFF2iCAQ8J4SFEwKExyH1OgyEQg2oDYBS1/mhr+wjBJwmPMiVUsiIM4Pe7OzETc1EUYTL5YJOp8NLL72EK6+8MuKNjCRudkZjncvjxycH68PTT/OsCVg8JRV6Tj+lSBFFwN8hfTh7On+b9zh6fGC7pQ9MmQKQyQFB3vnnzvvhP3fd7+uYYvDXCbLOdvUTHrwOIBgY2PtUaqTwoDZ2fjX0uG8E1HopRIwiw7bZGQD853/+Z6+AIZPJkJycjAULFsBiGfkiEiIauMP1LmzqMf303InJmJ7F6ac0SMFA9zCAp6+vg/iQjlVK7cnhQWPsPqbSS70NdEqDDhi33nrrMDSDiIaTxx/E5pIGHKqVdj/l9FM6JVEE/O0nB4ae931tA3suVULnh7Kx+2vXb/YAEApKhYzhr503MdT7fq9zTnFNn9cFpdtJ7dL10fNwQi+EnDsDD9WgA8aGDRug1+tx/fXX9zr+97//He3t7VixYkXEGkdEQ1fV2o4P90u7n3L66RgkitKHbdcshYBXms0Q8AGBjr57IEID6H2QKwC16eQA0TNIyGNg2K3r/XcFEoUmNto1Bgz6u7xu3Tr8+c9/Pul4SkoKbr/9dgYMohgRCIaw7Xgzdpa3hqefXjI1DRmcfhofRLF7PYOAt3cw6PW1Z3Dw9X2eGBr866v1JwQHU+/7Sm18FB6GZ2CwR2KkDTpgVFRUIC8v76TjOTk5qKioiEijiGhoTpx+OjXThPMmWTn9NFb4PUBbI9DWALgbO2cl9BEWIrEGQpeeUx27Zi8oNH33QKgM/C2fhmzQ/4JSUlKwd+9e5Obm9jpeVFSEpKSkSLWLiM6AKIrYVWHH10d7Tj9NxYQU7n4aFaEQ0NHaGSQapFDhbpACxUAJss6pi+oeX7sCQs+v6tOfFw89DjRqDDpg3HTTTbjnnntgMBhw3nnnAQC++OIL3Hvvvbjxxhsj3kAiGhiXx4+PD9SjoqV7+umSKdz9dMT42rt7JMKBounU9QwaI5CQAuiTpWWcFZoeQaFHMJApGAwoLg36J88jjzyCsrIyXHTRRVAopMtDoRCWL1+ORx99NOINJKLTK6lzYVNxPbz+EHc/HW6hINDecnKvhNfV9/lyBZCQ3BkmOm8JyVINA9EoNuiFtrocOXIEe/bsgVarxbRp05CTkxPptg0LLrRFo4nHH8TnxQ0orpM+3NJMGlxSmAYLp59Ghtd9cq9Ee3PfUx8BQGuWwoM+pTtQaMwAVyOmUWJYF9rqMnHiREycOPFMLyeiIapsacdHB6TppzJBwLw8CxbkcfrpgHXtVBnwdE7Z9PSol+gMFL72vq9VqLoDRDhQJI+6VRuJhmLQAeO6667D/Pnzcd999/U6/vvf/x47duzA3//+94g1johOFgiG8PWxZuyqkKafmnXS9NN00xjrcg8Gekzh9PQOCuGpmt7u+z3/3PXY6TpwBUGqj+jVK5Es9Upw+ImoX4MOGFu2bMGDDz540vFLL70Uf/zjHyPRJiI6hUaXFx8eqENT5/TTaZkmnDcpGSrFKOmC77AD9nJpaKLfoOAd2GJQAyGTdxdXqg0n90pw/QSiMzLogOF2u6FSnTy+q1Qq4XQ6I9IoIupNmn7aiq1HmxEMidCp5Fg8JRXjk+N8+mnQD9grgJZSoOW4VN8wWArVCTMwOv8cvp0wO0Oh6bx1XsdZGkTDYtABY9q0adi4cSPWrl3b6/hrr72GKVOmRKxhRCRxdk4/reycfjouOQGLC+J0+qkoSiGi5bh0s1f27okQZIAxHdBZe4eDrj/Le4SJrvssoCSKSYP+CfXrX/8a1157LY4dO4bvfe97AIBNmzbhlVdeweuvvx7xBhKNVd5AEPurnfi2tDk8/fT8SSmYmmmMr+mnfg/QWga0dvZSeE7o6dQYAUsekDgOsORKW2ATUdwbdMC44oor8NZbb+HRRx/F66+/Dq1WixkzZuCzzz5DYmLicLSRaExxdPixp9KO/dUO+ALSHhLpJg2Wxsv0U1EEXLXdwx7Omt57YcgUgNkmBYrEcYAuiUMURKPQGa+D0cXpdOLVV1/Fc889h507dyIYPMX88BjBdTAoVtU6OrCr3I6jDW6EOv9bJiaoMDvbgikZxtiefup1d/dQtJQC/o7ej+uSOgNFHmDOZuEkUZwakXUwtmzZgueeew7/+Mc/kJGRgWuvvRZPPfXUmT4d0ZgUCok41ujGropW1Ng94ePZiTrMzrEgN0kXm8MhoSDgqOqupXA39H5coZKGOxLHScMfWnM0WklEUTSogFFXV4cXXngBzz33HJxOJ37wgx/A6/XirbfeYoEn0SB4A0EcrHFid4Udjg4/AEAuE5CfZsCsbDNSDDFYh9DR2t1D0VomzQDpyZAm9VAkjgOMmdL0TyIaswYcMK644gps2bIFl112GdavX49LLrkEcrkczzzzzHC2j2hUcXr8KKq0Y1+1A16/VJegUcoxI8uE6TYz9LE0MyTgk6aQdg19tLf0flyl6y7OTMwDVAnRaScRxaQB/zT74IMPcM899+COO+7gEuFEg1Tv9GBXeSsO13fXV1h0SszOsaAg3QilPEamWvrageYjQONhqZfixCmkpszu4kx9KoszieiUBhwwvvrqKzz33HOYM2cOCgoKcMstt3B7dqJ+hEIijje1YVdFK6pbu4sebYk6zM42I8+aEBv1FR4H0HQEaCwBHJW9l8/WmLoDhSWHe20Q0YANehZJW1sbNm7ciOeffx7bt29HMBjEE088gdtuuw0Gg2G42hkxnEVCw80XCOFgrRO7K1phb5fqFGSCVF8xO9uMFGMM1Fe0NQFNh6VQ4arr/ZghFbDmA9ZJQIKVvRREFDaYz9AhTVMtKSnBc889hxdffBF2ux1LlizBO++8c6ZPNyIYMGi4uDx+FFU6sK/aAY9fmq6tVsowPdOMGTYTDJooTs0URSlINJVIwx89l+QWBMCU1RkqJnLGBxGd0ogFjC7BYBDvvvsunn/+eQYMGnManB7sqrCjpM4Vrq8w65SYlW3BlHRj9DYiC4WkIY+mw9Kt5wqaMrk0jdQ6EUiaCKjjfE8TIhoRIx4w4gkDBkWCKIoobWrDrgp7eI8QAMi0aDE724Jx1gTIorEwVjAgFWc2lUh1FT0XvJIrpVqK5HwgcTyX5CaiQRuRhbaIxiJ/MIRDtU7sKm9Fa4/6ikmpeszOsSA1GvUVAS/QfEwKFc3Heq9PodRIPRTJ+VKPBVfQJKIRwoBBNABubwB7K+0oqupdXzEt04QZNjOMI11f4WuTeiiajkjrVIR6LNGvNkgFmsmTAFM2dxsloqhgwCDqR73Tg90VdhyudyEYkkYTTVolZmWbMSXDCLViBFer9DikAs2mwydPJ9UldoaKfMCQzpkfRBR1DBhEJ5DWr3BjV4W91/oVGWYN5uRYMM6qH5n6ilAIcNdLPRT9TSdNzueOpEQUcxgwiDp5/EEcqHFgT6UDzo6e61foMdNmQZppmOsrRBFoawRaywF7ubRMd8Db/TinkxJRHGHAoDGvtc2HPZV2HKx1wheQ9gfRquSYnjnM+4OIorS/h72sM1RUnLzNuUIl1VFYJ3A6KRHFFQYMGpNEUURlSwd2V7aitKktXM5g1aswK9uC/DRD5PcHEUVpR1J7RXcPhdfd+xy5QgoU5mxpaW59Gos0iSguMWDQmOIPhlBS58LuilY0uX3h4+OSEzDLZoEtURvZ/UE8ju7eCXt578WuAECmkDYQM+dIocKYwW3OiWhUYMCgMaFrmuneagc6fNKUTpVChinpRsy0mWFJUEXmhbxuKUh0hYqO1t6Py+TSLA9LjhQqjJlSrwUR0SjDn2w0qknTTKVt0rummRq1Ssy0mVCYYYJGOcTeAl97d+9Ea3nvPT4AaYtzQ1r3kIcxS6qrICIa5WIiYDz11FN4/PHHUVdXhxkzZuB//ud/MH/+/D7PfeONN/Doo4/i6NGj8Pv9mDhxIv7t3/4Nt9xyywi3mmJVKCTiWKMbuyvsqLZ3F01mWrSYZTNjfPIQppn6PZ2BokIqznQ39n5cEAB9SueQRw5gtnGLcyIak6IeMDZu3IjVq1fjmWeewYIFC7B+/XosXboUJSUlSElJOen8xMRE/OpXv8LkyZOhUqnwz3/+EytXrkRKSgqWLl0ahXdAsaJrmunuCjtcngAAQC4TMCnVgFnZ5oEv4x3wSrUSXifgdXV/dTdI61KcuH2PPrl3oFBqI/zOiIjiT9Q3O1uwYAHmzZuHJ598EgAQCoVgs9lw99134/777x/Qc8yePRuXXXYZHnnkkdOey83ORp9TTjPNMmF61gnTTAPe7tDg6QoQrt5hIuA7xSt10iV11lB0zvZQJQzjuyMiih1xs9mZz+fDzp07sWbNmvAxmUyGxYsXY9u2bae9XhRFfPbZZygpKcFjjz02nE2lGNNzmunxxjYAgCzkR6bWh5nJMowziFAEaoHSnr0QAwgPXZQaaU8PtbHzZgC0FqmHQm0YxndGRDQ6RDVgNDU1IRgMIjU1tdfx1NRUFBcXn/I6h8OBzMxMeL1eyOVy/OlPf8KSJUv6PNfr9cLr7V4N0el09nkexQFRhN9Rg7LychyvqoHHbYcq0IYZITfS1H5k6mUwBhUQ6gSgrp/nUailkKAxdYYIQ3eI6PrKQkwioiGJeg3GmTAYDNizZw/cbjc2bdqE1atXY9y4cbjgggtOOnfdunV46KGHRr6RFBmhEOCogL++GLVH96KhqRH+oAgjAItMQLJBjXSjBhplZyFlV3joCgoa48khgkWXRETDLqoBw2q1Qi6Xo76+vtfx+vp6pKWlnfI6mUyGCRMmAABmzpyJQ4cOYd26dX0GjDVr1mD16tXh+06nEzabLTJvgIZHMBDe4MvfcAR1zS2oc3gQCIkICkoEjRnITEtFdnoqVDpTZ4hgeCAiiiVRDRgqlQpz5szBpk2bcPXVVwOQijw3bdqEu+66a8DPEwqFeg2D9KRWq6FW80Mn5vk9QMsxadfQluPw+byotXeg3uWFF2q0aCchZJ2IyQXTkZ9uGZndTImI6IxFfYhk9erVWLFiBebOnYv58+dj/fr1aGtrw8qVKwEAy5cvR2ZmJtatWwdAGvKYO3cuxo8fD6/Xi/fffx8vvvginn766Wi+DToTXjfQfARoPCwtVBUKwhsIosbuQVWHAk2a8Wix5kGTnIP5eVZMSNFHdhlvIiIaNlEPGDfccAMaGxuxdu1a1NXVYebMmfjwww/DhZ8VFRWQ9djsqa2tDXfeeSeqqqqg1WoxefJkvPTSS7jhhhui9RZoMDpagaYjUk+Fszq8pkSHP4iyDg0O+bPRpM1Fm8mKDIsW38tLQm6SjsGCiCjORH0djJHGdTBGmCgCbY1A02EpVLgbej3sUiVjvy8VRZ5UdCjMAIDsRB3m5yUiyxLhjceIiGhI4mYdDBqlRFHqnWg6LA1/9NzwS5ABZhuaNTn41m1FSWtngFBIO5rOz0tEuokrYRIRxTsGDIqMUFCqo2g8LNVVeN3dj8kUQGIeYJ2Iankmvq3yoLyyHYC0dcfEFAPm5VmQYhjgUt5ERBTzGDDozAV84emkaD4qLcPdRaECkiYA1kkQE8eh3B7A9rIWVLe2AABkgoD8NAPm5yUiMVJbpRMRUcxgwKDBEUXAUQlUfSdNKw0Guh9T6QDrJOlmzoEok+NYYxu276xHvdMDQNp8rDDDiLk5iTDplFF6E0RENNwYMGhgQkGgsRio3A64eqzDrTEByZ2hwpgFyGQIhUQcbnBhR2kLmtzS3h9KuYBpWWbMzjbDoGGwICIa7RgwqH/+DqC2SOqx8LqkYzIFkDYNyJgJ6FOlQgoAwZCIQ9UO7Chrgb3dDwBQKWSYaTNjVrYZOhX/uRERjRX8iU9962iVQkVtERCUwgJUCUDmHCBjljQc0skfDOFAjRPflbXA5ZGGTDRKOWZnmzHDZoZGKY/GOyAioihiwKBuogg4qoCq7dJiWF1LpOiTgaz5QMoUQN79T8YXCGFftR07y1vR5g0CABLUcszJsWBaphkqhayvVyEiojGAAYOkHUsbi6Vg4aztPp40HsiaB1hyw8MgABAIhrCzvBW7Kuzw+KVgYdAoMC83EYUZRijkDBZERGMdA8ZY5vcAdXuBqh2AxykdkymA1EIpWOiTT7qkye3Fh/vr0OiSpqRadErMzU1EQboRcm5ARkREnRgwxqIOO1DdWV8RkGZ5QKUDMmYDmbOlWosTiKKI3ZV2bD3ShEBIhFYlx/mTkpGfauDOpkREdBIGjLHEUS0NgzSWdNdXJFil3orUqb3qK3pyefz4+EA9Klqk1TfzrAlYMiUVCWr+8yEior7xE2K0C4WkPUGqtksBo0tinhQsEsf1qq84UUmdC5uK6+H1h6CUCzhvUjKmZZq4CRkREfWLAWO0CniB2r3SUEiHXTomk/eor0jp93KPP4jNJQ04VCutfZFm0uCSwjRYuKw3ERENAAPGaONxdK5fsae7vkKpldauyJwDqPWnfYrKlnZ8dKAOLk8AggDMz0vEgrwkFnESEdGAMWCMFs5aaRikoRgQQ9IxXRKQNVdadVN++uW5A8EQth1vxs7yVogiYNYpccnUNG6fTkREg8aAEc9EUVoQq2o7YK/sPm7JkRbGShrfb31FTydOP52aacL5k5K5WBYREZ0RBox4FQoBhz+UppoCgCADUqdI9RWGtAE/jSiK2FVhx9dHu6efLi5IxYSU0w+lEBERnQoDRjwKBoBD70jTTQUBsM0HMucCGuOgnubE6afjkhOwuIDTT4mIaOj4SRJvAj7gwBtAS6k0K2TKVUBy/qCfhtNPiYhoODFgxBN/B7D3/wBnjVS0OfU6aT2LQfD4g/i8uAHFdZx+SkREw4cBI154XUDRa0BbE6DUANN+AJgyB/UUnH5KREQjhQEjHnS0SuGiwy6tYzH9xj43IjuVQDCEr481Y1cFp58SEdHIYMCIde5GYO9rgNcNaM3AjBsBrWXAlze6vPjwQB2aOP2UiIhGEANGLHNUA/v+T9pWXZ8MTL8BUBsGdGnX9NOtR5sQDInQqeRYPCUV45M5/ZSIiIYfA0asaikF9v8DCPoBYwYw/QfSkt8D4OycflrJ6adERBQl/MSJRY0lwMG3gVBQmiVSeC2gGNgsD04/JSKiWMCAEWtqi4CSD6RlwJPzgYIrAfnp/5o4/ZSIiGIJA0YsqdwOHN0k/Tl9BjDpEkB2+mLMntNPZYKAeXkWTj8lIqKoYsCIBaIIlG4Byr+W7mcvAMZdeNqNyoIhEVuPNnH6KRERxRwGjGgTReDIx0D1Lun+uAuAnIUDunTLkUbsqbADAKZlmnAep58SEVGMYMCIplAQKP4nUH9Q6q2YeDGQOXtAlx5rdIfDxaXT0jA5bXAbnRERxZJgMAi/3x/tZhAApVIJuVw+5OdhwIiWoB848CbQfEzaar3gCmm79QHo2gUVAGbnWBguiCiuud1uVFVVQRTFaDeFAAiCgKysLOj1Q1s3iQEjGvweYP/rgL1SmiFSeC2QNH5Al4ZCIj7cXwePP4hUowbnTLAOc2OJiIZPMBhEVVUVdDodkpOTOaU+ykRRRGNjI6qqqjBx4sQh9WQwYIw0X5u0r4i7AVCogWnXA2bbgC/fXtaCqtYOqBQyfH9aGmeKEFFc8/v9EEURycnJ0GpZoB4LkpOTUVZWBr/fz4ARNzwOKVy0twAqnbRpmSF1wJdXtbbjm+PNAIDvTU6BWcc1LohodGDPReyI1N9FTEw5eOqpp5CbmwuNRoMFCxZg+/btpzz3L3/5C84991xYLBZYLBYsXry43/NjRlszsOtFKVxoTMCsWwYVLjz+ID7cXwdRBArSjShIZ90FERHFrqgHjI0bN2L16tV44IEHsGvXLsyYMQNLly5FQ0NDn+dv3rwZN910Ez7//HNs27YNNpsNF198Maqrq0e45YPgrAV2vwh4XUCCFZj1Q0CXOODLRVHExwfr4fIEYNEp8b3JKcPYWCIioqETxCiX7S5YsADz5s3Dk08+CQAIhUKw2Wy4++67cf/995/2+mAwCIvFgieffBLLly8/7flOpxMmkwkOhwNG4wj0ArSWSwWdAR9gSJN2RFXpBvUURZV2fFbcALlMwI3zbEgxaoapsUREI8vj8aC0tBR5eXnQaPizLRb093cymM/QqPZg+Hw+7Ny5E4sXLw4fk8lkWLx4MbZt2zag52hvb4ff70diYt89Al6vF06ns9dtxDQdAfb+nxQuLDnAzJsHHS4aXV5sOdwIADhnopXhgoiI4kJUA0ZTUxOCwSBSU3vXIqSmpqKurm5Az3HfffchIyOjV0jpad26dTCZTOGbzTbwGRtDUrcf2P8GEAoA1onAtB9Is0YGwRcI4YP9tQiERIxLTsAsm3l42kpERIP24Ycf4pxzzoHZbEZSUhIuv/xyHDt2LPx4VVUVbrrpJiQmJiIhIQFz587Ft99+G3783Xffxbx586DRaGC1WnHNNddE420Mm6jXYAzF7373O7z22mt48803T9m1tmbNGjgcjvCtsrJy+BtWtRM49C4ghoC0qdI6FwPYEfVEXxxuRLPbB71agSVTUlllTUSjniiK8AVCUbkNtmKgra0Nq1evxnfffYdNmzZBJpPhmmuuQSgUgtvtxvnnn4/q6mq88847KCoqwr//+78jFAoBAN577z1cc801+P73v4/du3dj06ZNmD9//nB8S6MmqtNUrVYr5HI56uvrex2vr69HWlpav9f+4Q9/wO9+9zt8+umnmD59+inPU6vVUKsH13NwxkQRKN8KlH4p3c+aC0xYfNpNy/pSUufC/moHBAG4ZGoadCrOKCai0c8fFPHU50ej8tqrLpwAlWLgP6+vu+66Xveff/55JCcn4+DBg/j666/R2NiIHTt2hIfwJ0yYED73t7/9LW688UY89NBD4WMzZswY4juILVHtwVCpVJgzZw42bdoUPhYKhbBp0yYsXHjqDb9+//vf45FHHsGHH36IuXPnjkRTT08Upa3Wu8JF3rlnHC4c7X58ekgKXfNzE2FLHFzdBhERDb8jR47gpptuwrhx42A0GpGbmwsAqKiowJ49ezBr1qxT1gfu2bMHF1100Qi2duRF/dfi1atXY8WKFZg7dy7mz5+P9evXo62tDStXrgQALF++HJmZmVi3bh0A4LHHHsPatWvxyiuvIDc3N1yrodfrh7xu+hkLhYCS94G6fdL9iUuk3oszEAyJ+GB/LXyBEDLMGpw1LimCDSUiim1KuYBVF044/YnD9NqDccUVVyAnJ+f/t3fvQVGddx/Av7vshQVWbirsGlASCHL1MqJRfMckaEhMNqEZ70hstJ0x0UaoGpI6aNJU8VKMJTYakvSSGYxNTcxFE0ekSIWqURBfUV68IRhvIKKAyEXO8/5B3XSrUYGze2T5fmZ2Bs45LN/HRfbHc54LPvzwQ5jNZkiShMjISLS2tt5zVdLesGqp4gXG1KlTUVNTg6VLl+LixYsYOnQoduzYYR34WVVVBbX6x46WDRs2oLW1FZMmTbJ5nmXLluGtt95yZPQOQgDHtgI1xzs2LRs8EfCP6vLT7T1ViwvXmqHXqvF0pAlqLgVORL2ISqXq1G0KpdTW1qK8vNy6+CMAFBQUWM9HR0fjo48+wpUrV+7YixEdHY3c3FzrH9POSPECAwDmz5+P+fPn3/Hc7t27bT4/c+aM/QN1hkoFeAYCtaeBiISOGSNdVFXbhIOVVwAAE8L84GnQyhSSiIjk5O3tDV9fX2RlZcFkMqGqqspm7abp06djxYoVSEhIQHp6OkwmEw4dOgSz2YzRo0dj2bJliIuLwyOPPIJp06bh5s2b+Pbbb5Gamqpgq+TVo2eRPDACYoCRv+xWcdHUehM7jl6AEED0Q54I8TPKGJCIiOSkVquxefNmFBUVITIyEikpKVizZo31vE6nw86dO9G/f39MnDgRUVFRWLlypXXzsMcffxx///vf8fXXX2Po0KF48skne8a2F52g+EqejubwlTzvgxACX5acw5nLTejrocO0kYHQurD2IyLnx5U8HzxOsZIndSiuqsOZy03QqFV4JsrE4oKIiHo8vpMp7FJ9MwpPdmzBPi60H/p6OGjNDiIiIjtigaGglpvt+PbIBbRLAiF+Hoga4Kl0JCIiIlmwwFCIEAJ5/1eNq01tMLpqMD6MS4ETEZHzYIGhkGMX6lF2oQFqlQoTo0xw1booHYmIiEg2LDAUUHe9FbvLO7Zgf+xhH5i9nH9FNyIi6l1YYDjYzXYJ3/57KfAAHzfEDLrzOvVEREQ9GQsMBys4eRnV9S0w6FzwdKQ/lwInIiKnxALDgU7XNOJQ1VUAwFPhfvDQPxArtRMREcmOBYaDNDS3Yeexji3YhwV64eF+Cu38SkRED4RBgwZh3bp1SsewGxYYDiBJAjtKL+JGazv699FjbHBfpSMRERHZFQsMBzhw5gp+qLsBnUaNiZEmaLgUOBEROTm+09nZuas3sO90xxbsT4T2h7e7TuFERETUXVlZWTCbzZAkyeb4Cy+8gNmzZ+PUqVN44YUX4OfnBw8PD8TExGDXrl1d/n5r165FVFQU3N3dERAQgFdffRWNjY021xQWFuLxxx+Hm5sbvL29ER8fj7q6OgCAJElYvXo1goODodfrERgYiOXLl3c5z/1ggWFHzW3t+O7IBUhCIMxkRLj5wdi9lYjogSUEcLNVmUcnNhefPHkyamtrkZeXZz125coV7NixA4mJiWhsbMTEiRORm5uLQ4cO4emnn4bFYkFVVVWX/lnUajUyMzNx9OhR/PWvf8U//vEPvP7669bzJSUliIuLQ3h4OPbu3YuCggJYLBa0t7cDAN58802sXLkSaWlpOHbsGDZt2gQ/P78uZblf3K7dToQQ2Pa/F3CyuhFeblrMGBUIvYardRIR/afbtga/2QrsyVAmzP8sBDT338uckJAAX19ffPzxxwA6ejXefvttnD17Fmr17X+/R0ZGYu7cuZg/fz6AjkGeycnJSE5O7nTULVu2YO7cubh8+TIAYMaMGaiqqkJBQcFt1zY0NKBfv35Yv349fvGLX9zzubld+wPuyLlrOFndCBd1x1LgLC6IiJxLYmIiPv/8c7S0tAAAsrOzMW3aNKjVajQ2NmLRokUICwuDl5cXPDw8UFZW1uUejF27diEuLg4DBgyA0WhEUlISamtr0dTUBODHHow7KSsrQ0tLy0+etxcuxGAHlxtbkP/vpcBjg/vCr4/rPb6CiIgAAC7ajp4Epb53J1gsFgghsH37dsTExGDPnj149913AQCLFi1CTk4Ofv/73yM4OBgGgwGTJk1Ca2trp2OdOXMGzz33HF555RUsX74cPj4+KCgowJw5c9Da2go3NzcYDD+95cTdztkTCwyZtbVL+PbIBdyUBIL6umN4oJfSkYiIeg6VqlO3KZTk6uqKF198EdnZ2Th58iRCQ0MxfPhwAB0DLn/+85/jZz/7GQCgsbERZ86c6dL3KSoqgiRJyMjIsN56+eyzz2yuiY6ORm5uLt5+++3bvj4kJAQGgwG5ubn3dYtELiwwZJZfXoPaxla4613wVAS3YCcicmaJiYl47rnncPToUcycOdN6PCQkBF988QUsFgtUKhXS0tJum3Fyv4KDg9HW1ob33nsPFosFhYWF2Lhxo801b775JqKiovDqq69i7ty50Ol0yMvLw+TJk9G3b1+kpqbi9ddfh06nQ2xsLGpqanD06FHMmTOnW+2/G47BkNHxSw04cu4aVCrg6QgT3HSs34iInNmTTz4JHx8flJeXY8aMGdbja9euhbe3N8aMGQOLxYL4+Hhr70ZnDRkyBGvXrsWqVasQGRmJ7OxspKen21zz6KOPYufOnTh8+DBGjhyJ0aNH46uvvoJG0/E+lJaWhoULF2Lp0qUICwvD1KlTUV1d3fWG3wfOIpHJtRttyN5fiZY2CSODfBDL1TqJiO7pbjMWSBmcRfIAaZcEdpReQEubBJOnKx572FfpSERERIpigSGDfadrcf5qM/RaNZ6JMsGFW7ATEdF9ys7OhoeHxx0fERERSsfrMg4S6CYhBJpaO1ZKGx/mB09D56Y5ERFR7/b8889j1KhRdzyn1fbc9xQWGN2kUqkwIdwPkQP6wOSpzFxjIiLquYxGI4xGo9IxZMdbJDJhcUFERPQjFhhERKS4Xjah8YEm12vBAoOIiBTj4tKxT1NXltAm+7j1Wtx6bbqKYzCIiEgxGo0Gbm5uqKmpgVarveMupOQ4kiShpqYGbm5u1kW6uooFBhERKUalUsFkMqGiogKVlZVKxyEAarUagYGB3d7qggUGEREpSqfTISQkhLdJHhA6nU6WniQWGEREpDi1Ws2lwp0Mb3YRERGR7FhgEBERkexYYBAREZHset0YjFsLiNTX1yuchIiIqGe59d55P4tx9boCo6GhAQAQEBCgcBIiIqKeqaGhAZ6enne9RiV62fqskiTh/PnzMBqN3Z7j+yCor69HQEAAzp49iz59+igdx+7YXufX29rc29oL9L42O1N7hRBoaGiA2Wy+51TWXteDoVar8dBDDykdQ3Z9+vTp8T+4ncH2Or/e1ube1l6g97XZWdp7r56LWzjIk4iIiGTHAoOIiIhkxwKjh9Pr9Vi2bBn0er3SURyC7XV+va3Nva29QO9rc29r7y29bpAnERER2R97MIiIiEh2LDCIiIhIdiwwiIiISHYsMIiIiEh2LDB6oPT0dMTExMBoNKJ///5ISEhAeXm50rEcZuXKlVCpVEhOTlY6il2dO3cOM2fOhK+vLwwGA6KionDw4EGlY9lFe3s70tLSEBQUBIPBgEceeQTvvPPOfe130FP885//hMVigdlshkqlwpdffmlzXgiBpUuXwmQywWAwYPz48Thx4oQyYWVwt/a2tbUhNTUVUVFRcHd3h9lsxksvvYTz588rF1gG93qN/9PcuXOhUqmwbt06h+VzNBYYPVB+fj7mzZuHffv2IScnB21tbXjqqadw/fp1paPZ3YEDB/DBBx8gOjpa6Sh2VVdXh9jYWGi1Wnz33Xc4duwYMjIy4O3trXQ0u1i1ahU2bNiA9evXo6ysDKtWrcLq1avx3nvvKR1NNtevX8eQIUPwxz/+8Y7nV69ejczMTGzcuBH79++Hu7s74uPj0dzc7OCk8rhbe5uamlBcXIy0tDQUFxfjiy++QHl5OZ5//nkFksrnXq/xLVu3bsW+fftgNpsdlEwhgnq86upqAUDk5+crHcWuGhoaREhIiMjJyRHjxo0TCxYsUDqS3aSmpoqxY8cqHcNhnn32WTF79mybYy+++KJITExUKJF9ARBbt261fi5JkvD39xdr1qyxHrt69arQ6/Xi008/VSChvP67vXfy/fffCwCisrLSMaHs7Kfa/MMPP4gBAwaI0tJSMXDgQPHuu+86PJujsAfDCVy7dg0A4OPjo3AS+5o3bx6effZZjB8/Xukodvf1119jxIgRmDx5Mvr3749hw4bhww8/VDqW3YwZMwa5ubk4fvw4AODw4cMoKCjAM888o3Ayx6ioqMDFixdtfrY9PT0xatQo7N27V8FkjnPt2jWoVCp4eXkpHcVuJElCUlISFi9ejIiICKXj2F2v2+zM2UiShOTkZMTGxiIyMlLpOHazefNmFBcX48CBA0pHcYjTp09jw4YN+PWvf43f/OY3OHDgAF577TXodDrMmjVL6Xiye+ONN1BfX4/BgwfDxcUF7e3tWL58ORITE5WO5hAXL14EAPj5+dkc9/Pzs55zZs3NzUhNTcX06dOdYjOwn7Jq1SpoNBq89tprSkdxCBYYPdy8efNQWlqKgoICpaPYzdmzZ7FgwQLk5OTA1dVV6TgOIUkSRowYgRUrVgAAhg0bhtLSUmzcuNEpC4zPPvsM2dnZ2LRpEyIiIlBSUoLk5GSYzWanbC/9qK2tDVOmTIEQAhs2bFA6jt0UFRXhD3/4A4qLi6FSqZSO4xC8RdKDzZ8/H9u2bUNeXp5TbkF/S1FREaqrqzF8+HBoNBpoNBrk5+cjMzMTGo0G7e3tSkeUnclkQnh4uM2xsLAwVFVVKZTIvhYvXow33ngD06ZNQ1RUFJKSkpCSkoL09HSlozmEv78/AODSpUs2xy9dumQ954xuFReVlZXIyclx6t6LPXv2oLq6GoGBgdbfY5WVlVi4cCEGDRqkdDy7YA9GDySEwK9+9Sts3boVu3fvRlBQkNKR7CouLg5HjhyxOfbyyy9j8ODBSE1NhYuLi0LJ7Cc2Nva2qcfHjx/HwIEDFUpkX01NTVCrbf/ecXFxgSRJCiVyrKCgIPj7+yM3NxdDhw4FANTX12P//v145ZVXlA1nJ7eKixMnTiAvLw++vr5KR7KrpKSk28aPxcfHIykpCS+//LJCqeyLBUYPNG/ePGzatAlfffUVjEaj9R6tp6cnDAaDwunkZzQabxtf4u7uDl9fX6cdd5KSkoIxY8ZgxYoVmDJlCr7//ntkZWUhKytL6Wh2YbFYsHz5cgQGBiIiIgKHDh3C2rVrMXv2bKWjyaaxsREnT560fl5RUYGSkhL4+PggMDAQycnJ+N3vfoeQkBAEBQUhLS0NZrMZCQkJyoXuhru112QyYdKkSSguLsa2bdvQ3t5u/T3m4+MDnU6nVOxuuddr/N9FlFarhb+/P0JDQx0d1TGUnsZCnQfgjo8///nPSkdzGGefpiqEEN98842IjIwUer1eDB48WGRlZSkdyW7q6+vFggULRGBgoHB1dRUPP/ywWLJkiWhpaVE6mmzy8vLu+P921qxZQoiOqappaWnCz89P6PV6ERcXJ8rLy5UN3Q13a29FRcVP/h7Ly8tTOnqX3es1/m/OPk2V27UTERGR7DjIk4iIiGTHAoOIiIhkxwKDiIiIZMcCg4iIiGTHAoOIiIhkxwKDiIiIZMcCg4iIiGTHAoOIerzdu3dDpVLh6tWrSkchon9jgUFERESyY4FBREREsmOBQUTdJkkS0tPTERQUBIPBgCFDhmDLli0Afrx9sX37dkRHR8PV1RWPPfYYSktLbZ7j888/R0REBPR6PQYNGoSMjAyb8y0tLUhNTUVAQAD0ej2Cg4Px8ccf21xTVFSEESNGwM3NDWPGjLltR1oichwWGETUbenp6fjkk0+wceNGHD16FCkpKZg5cyby8/Ot1yxevBgZGRk4cOAA+vXrB4vFgra2NgAdhcGUKVMwbdo0HDlyBG+99RbS0tLwl7/8xfr1L730Ej799FNkZmairKwMH3zwATw8PGxyLFmyBBkZGTh48CA0Go1T7cZK1OMovdsaEfVszc3Nws3NTfzrX/+yOT5nzhwxffp06w6Tmzdvtp6rra0VBoNB/O1vfxNCCDFjxgwxYcIEm69fvHixCA8PF0IIUV5eLgCInJycO2a49T127dplPbZ9+3YBQNy4cUOWdhJR57AHg4i65eTJk2hqasKECRPg4eFhfXzyySc4deqU9brRo0dbP/bx8UFoaCjKysoAAGVlZYiNjbV53tjYWJw4cQLt7e0oKSmBi4sLxo0bd9cs0dHR1o9NJhMAoLq6utttJKLO0ygdgIh6tsbGRgDA9u3bMWDAAJtzer3epsjoKoPBcF/XabVa68cqlQpAx/gQInI89mAQUbeEh4dDr9ejqqoKwcHBNo+AgADrdfv27bN+XFdXh+PHjyMsLAwAEBYWhsLCQpvnLSwsxKOPPgoXFxdERUVBkiSbMR1E9GBjDwYRdYvRaMSiRYuQkpICSZIwduxYXLt2DYWFhejTpw8GDhwIAPjtb38LX19f+Pn5YcmSJejbty8SEhIAAAsXLkRMTAzeeecdTJ06FXv37sX69evx/vvvAwAGDRqEWbNmYfbs2cjMzMSQIUNQWVmJ6upqTJkyRammE9HdKD0IhIh6PkmSxLp160RoaKjQarWiX79+Ij4+XuTn51sHYH7zzTciIiJC6HQ6MXLkSHH48GGb59iyZYsIDw8XWq1WBAYGijVr1ticv3HjhkhJSREmk0nodDoRHBws/vSnPwkhfhzkWVdXZ73+0KFDAoCoqKiwd/OJ6A5UQgihcI1DRE5s9+7deOKJJ1BXVwcvLy+l4xCRg3AMBhEREcmOBQYRERHJjrdIiIiISHbswSAiIiLZscAgIiIi2bHAICIiItmxwCAiIiLZscAgIiIi2bHAICIiItmxwCAiIiLZscAgIiIi2bHAICIiItn9P3Nb5VFHcZbpAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 600x800 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "epochs = 15\n",
        "history = model.fit(train_ds, epochs=epochs, validation_data=val_ds)\n",
        "\n",
        "# Plot the loss and accuracy history\n",
        "fig, axs = plt.subplots(2, figsize=(6, 8), sharex=True)\n",
        "fig.suptitle('Traininig history')\n",
        "x = list(range(1, epochs+1))\n",
        "axs[0].plot(x, history.history[\"loss\"], alpha=0.5, label=\"loss\")\n",
        "axs[0].plot(x, history.history[\"val_loss\"], alpha=0.5, label=\"val_loss\")\n",
        "axs[0].set_ylabel(\"Loss\")\n",
        "axs[0].legend(loc=\"upper right\")\n",
        "axs[1].plot(x, history.history[\"masked_accuracy\"], alpha=0.5, label=\"acc\")\n",
        "axs[1].plot(x, history.history[\"val_masked_accuracy\"], alpha=0.5, label=\"val_acc\")\n",
        "axs[1].set_ylabel(\"Accuracy\")\n",
        "axs[1].set_xlabel(\"epoch\")\n",
        "axs[1].legend(loc=\"lower right\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MQgnnk9NSZTQ",
        "outputId": "3fbbfd8f-18c0-41e0-e1a5-4177d82e68fe",
        "trusted": true
      },
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Test 0:\n",
            "[start] but before we start we set this golden rule [end]\n",
            "== [start] लेकिन शुरू करने से पहले हम ये नियम बनाते है | [end]\n",
            "-> [start] लेकिन इससे पहले हमने इसे विदेशी शासक में शुरु किया । [end]\n",
            "\n",
            "Test 1:\n",
            "[start] hindi editor and spell checker [end]\n",
            "== [start] हिन्दी संपादक तथा वर्तनी जाँचकर्ता [end]\n",
            "-> [start] हिन्दी संपादक तथा वर्तनी जाँचकर्ता [end]\n",
            "\n",
            "Test 2:\n",
            "[start] every single day [end]\n",
            "== [start] हर एक दिन. [end]\n",
            "-> [start] हर दिन [end]\n",
            "\n",
            "Test 3:\n",
            "[start] four vedas [end]\n",
            "== [start] चार वेद [end]\n",
            "-> [start] चार वेद [end]\n",
            "\n",
            "Test 4:\n",
            "[start] very efficient process [end]\n",
            "== [start] हमारी कार्यप्रणाली अत्यंत सुचारु है, [end]\n",
            "-> [start] बहुत ही मजबूत प्रक्रिया से तुलना [end]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# training parameters used\n",
        "seq_len = 20\n",
        "def translate(sentence):\n",
        "    \"\"\"Create the translated sentence\"\"\"\n",
        "    enc_tokens = eng_vectorizer([sentence])\n",
        "    lookup = list(hin_vectorizer.get_vocabulary())\n",
        "    start_sentinel, end_sentinel = \"[start]\", \"[end]\"\n",
        "    output_sentence = [start_sentinel]\n",
        "    # generate the translated sentence word by word\n",
        "    for i in range(seq_len):\n",
        "        vector = hin_vectorizer([\" \".join(output_sentence)])\n",
        "        dec_tokens = vector[:, :-1]\n",
        "        pred = model([enc_tokens, dec_tokens])\n",
        "        word = lookup[np.argmax(pred[0, i, :])]\n",
        "        output_sentence.append(word)\n",
        "        if word == end_sentinel:\n",
        "            break\n",
        "    return output_sentence\n",
        "\n",
        "test_count = 5\n",
        "for n in range(test_count):\n",
        "    english_sentence, Hindi_sentence = random.choice(val_tp)\n",
        "    translated = translate(english_sentence)\n",
        "    print(f\"Test {n}:\")\n",
        "    print(f\"{english_sentence}\")\n",
        "    print(f\"== {Hindi_sentence}\")\n",
        "    print(f\"-> {' '.join(translated)}\")\n",
        "    print()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The model has overfit. since this notebook is only to understand how transformers architecture works I choose to ignore this. However, there are few ways the accuracy can be improved.\n",
        "- First of all, the vectorization can be replace with language toolkits that understand \"hindi\".\n",
        "- The english text can be tokenized better using lemmatization or sub word tokenization.\n",
        "- Use more data, ofcourse the training time increases as you use more data but transformers have proven to learn better with more data."
      ],
      "metadata": {
        "id": "aNe3S5YfwblK"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "id": "RyXCibz7DdGT"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "V100",
      "machine_shape": "hm",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}