Neural_Machine_Translation_tf_4L_4H_BigData.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Neural_Machine_Translation_tf-4L-4H-BigData.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard",
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#### Mounting Google Drive"
      ],
      "metadata": {
        "id": "0dqNGaFLKKAJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "bVenoImvJZs9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c10e328-30a1-4729-b497-c560b91397eb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "cd \"/content/drive/MyDrive/IASNLP\""
      ],
      "metadata": {
        "id": "iNpRDc1-KMj8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9d1846a1-2e55-44f9-eece-762a249fbe7c"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/IASNLP\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Importing Necessary Libraries"
      ],
      "metadata": {
        "id": "OlMuBUdIApMX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentencepiece\n",
        "!pip install sacrebleu\n",
        "!pip install tensorflow-gpu # ---> For GPU"
      ],
      "metadata": {
        "id": "Iwi6TDKsB0St"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Lambda, Layer, Embedding, LayerNormalization\n",
        "import sentencepiece as spm\n",
        "from sacrebleu.metrics import BLEU, CHRF, TER\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from collections import Counter"
      ],
      "metadata": {
        "id": "DXSYYy2b-jMM"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Loading Data"
      ],
      "metadata": {
        "id": "ToLMSNAHBnvJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We load the `train_data`, `train_dev_data`, `test_val_data` and `test_data` as well as the byte-pair encoder tokenizer for English and Bengali i.e. `sp_en_bpe` and `s_ben_bpe`."
      ],
      "metadata": {
        "id": "zXIf-FL7IG7f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data, train_dev_data = pd.read_csv(\"train_data_big.csv\")[['src', 'tgt']], pd.read_csv(\"train_dev.csv\")[['src', 'tgt']] \n",
        "test_val_data, test_data = pd.read_csv(\"test_val.csv\")[['src', 'tgt']], pd.read_csv(\"test_data.csv\")[['src', 'tgt']]\n",
        "sp_en_bpe, sp_ben_bpe = spm.SentencePieceProcessor(), spm.SentencePieceProcessor()\n",
        "sp_en_bpe.load('eng_big_bpe.model'); sp_ben_bpe.load('ben_big_bpe.model');"
      ],
      "metadata": {
        "id": "IXsCg4mBBeoo"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Preprocessing"
      ],
      "metadata": {
        "id": "auf4TlPFhNtn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We decide on some formats of the data and batches. We take the maximum tokens we can have per tokenized English and Bengali sentence to be `MAX_TOKENS = 60`. We use the vocabulary size of `32000`(i.e. `ENCODER_VOCAB = DECODER_VOCAB = 32000`). A batch size of `256` was choosen for our Mini-Batch Gradient Descent which is used to train our Model."
      ],
      "metadata": {
        "id": "3z7BUnn_G3WL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MAX_TOKENS = 60\n",
        "VOCAB_SIZE = 32000\n",
        "BATCH_SIZE = 256\n",
        "BUFFER_SIZE = BATCH_SIZE*4"
      ],
      "metadata": {
        "id": "7jTuQSqOAybO"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following two functions help in tokenizing an detokenizing the data based on our sentencepiece byte-pair encoding model."
      ],
      "metadata": {
        "id": "NlEW7GndbCfw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def tokenize(sentence, sp_model, add_bos = True, add_eos = True):\n",
        "    bos = [sp_model.bos_id()] if add_bos else []\n",
        "    eos = [sp_model.eos_id()] if add_eos else []\n",
        "    inputs = bos + sp_model.encode_as_ids(sentence) + eos\n",
        "    return np.reshape(np.array(inputs), [1, -1])\n",
        "def detokenize(tokenized, sp_model, is_bos = True, is_eos = True):\n",
        "    integers = np.squeeze(tokenized).tolist()\n",
        "    if is_eos:\n",
        "        return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.eos_id())])\n",
        "    else:\n",
        "        if sp_model.pad_id() in tokenized:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.pad_id())])\n",
        "        else:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):])"
      ],
      "metadata": {
        "id": "VC-6hav2BrJJ"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have tokenized the data."
      ],
      "metadata": {
        "id": "5-76mwBnbuzp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Tokenization\n",
        "src_train_data_enc = [np.squeeze(tokenize(train_data['src'].iloc[i], sp_en_bpe)) for i in range(train_data.shape[0])]\n",
        "tgt_train_data_enc = [np.squeeze(tokenize(train_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_data.shape[0])]\n",
        "src_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['src'].iloc[i], sp_en_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "tgt_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "src_test_val_data_enc = [np.squeeze(tokenize(test_val_data['src'].iloc[i], sp_en_bpe)) for i in range(test_val_data.shape[0])]\n",
        "tgt_test_val_data_enc = [np.squeeze(tokenize(test_val_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(test_val_data.shape[0])]"
      ],
      "metadata": {
        "id": "nT0QpepuDoFF"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have padded the data with post-truncation and post padding upto MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "wdyEjbm5hWXg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Padding\n",
        "# The reason why we add MAX_TOKENS+1 after-pad length for Bengali is that when we form the tf.Dataset later, we will shift the target sentence once and will truncate the last token once\n",
        "# Hence, it will get adjusted\n",
        "train_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_src = tf.keras.preprocessing.sequence.pad_sequences(src_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')"
      ],
      "metadata": {
        "id": "NT63vC8PcK-2"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we discard all those pairs where the tokenized source or target sentences are larger than MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "FvxtnfKniMRA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Removing all pairs whose either the target sentence or source setence exceeds MAX_TOKENS number of tokens\n",
        "same_idx = list(set([i for i in range(train_src.shape[0]) if train_src[i][-1] == 1 or train_src[i][-1] == 0]).intersection(set([i for i in range(train_tgt.shape[0]) if train_tgt[i][-1] == 1 or train_tgt[i][-1] == 0])))\n",
        "train_src = train_src[same_idx]\n",
        "train_tgt = train_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(train_dev_src.shape[0]) if train_dev_src[i][-1] == 1 or train_dev_src[i][-1] == 0]).intersection(set([i for i in range(train_dev_tgt.shape[0]) if train_dev_tgt[i][-1] == 1 or train_dev_tgt[i][-1] == 0])))\n",
        "train_dev_src = train_dev_src[same_idx]\n",
        "train_dev_tgt = train_dev_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(test_val_src.shape[0]) if test_val_src[i][-1] == 1 or test_val_src[i][-1] == 0]).intersection(set([i for i in range(test_val_tgt.shape[0]) if test_val_tgt[i][-1] == 1 or test_val_tgt[i][-1] == 0])))\n",
        "test_val_src = test_val_src[same_idx]\n",
        "test_val_tgt = test_val_tgt[same_idx]"
      ],
      "metadata": {
        "id": "L6-cV6zpcO5e"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_labels = np.zeros(train_tgt.shape)\n",
        "train_labels[:,0:train_tgt.shape[1] -1] = train_tgt[:,1:]\n",
        "train_dev_labels = np.zeros(train_dev_tgt.shape)\n",
        "train_dev_labels[:,0:train_dev_tgt.shape[1] -1] = train_dev_tgt[:,1:]\n",
        "test_val_labels = np.zeros(test_val_tgt.shape)\n",
        "test_val_labels[:,0:test_val_tgt.shape[1] -1] = test_val_tgt[:,1:]"
      ],
      "metadata": {
        "id": "JGMvPvIxOx-0"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"*** Preprocessed Data ***\")\n",
        "print(\"Train Data Shape(Source): \", train_src.shape)\n",
        "print(\"Train Data Shape(Target): \", train_tgt.shape)\n",
        "print(\"Train Labels Shape(Target): \", train_labels.shape)\n",
        "print(\"Train Dev Data Shape(Source): \", train_dev_src.shape)\n",
        "print(\"Train Dev Data Shape(Target): \", train_dev_tgt.shape)\n",
        "print(\"Train Dev Labels Shape(Target): \", train_dev_labels.shape)\n",
        "print(\"Test Val Data Shape(Source): \", test_val_src.shape)\n",
        "print(\"Test Val Data Shape(Target): \", test_val_tgt.shape)\n",
        "print(\"Test Labels Shape(Target): \", test_val_labels.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rO5HD33XOGxf",
        "outputId": "9efa03b0-fa0b-4f6b-f42b-7ae7b29b93f7"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "*** Preprocessed Data ***\n",
            "Train Data Shape(Source):  (728047, 60)\n",
            "Train Data Shape(Target):  (728047, 60)\n",
            "Train Labels Shape(Target):  (728047, 60)\n",
            "Train Dev Data Shape(Source):  (3649, 60)\n",
            "Train Dev Data Shape(Target):  (3649, 60)\n",
            "Train Dev Labels Shape(Target):  (3649, 60)\n",
            "Test Val Data Shape(Source):  (1194, 60)\n",
            "Test Val Data Shape(Target):  (1194, 60)\n",
            "Test Labels Shape(Target):  (1194, 60)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we prepare the data in the form ready to feed to the Model."
      ],
      "metadata": {
        "id": "xtsMxbAYzePz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data_prep = tf.data.Dataset.from_tensor_slices((train_src, train_tgt, train_labels))\n",
        "train_dev_data_prep = tf.data.Dataset.from_tensor_slices((train_dev_src, train_dev_tgt, train_dev_labels))\n",
        "test_val_data_prep = tf.data.Dataset.from_tensor_slices((test_val_src, test_val_tgt, test_val_labels))"
      ],
      "metadata": {
        "id": "_OkR5tbtyBQj"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we can see how our data looks"
      ],
      "metadata": {
        "id": "jSGBKDhc2sc-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for encoder_inputs, decoder_inputs, labels in train_data_prep.take(1):\n",
        "    print(f'encoder_inputs.shape: {encoder_inputs.shape}')\n",
        "    print(f'decoder_inputs.shape: {decoder_inputs.shape}')\n",
        "    print(f\"labels.shape: {labels.shape}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ao1TX6GHzWn7",
        "outputId": "6f09238e-0284-45ac-a24f-f2eca75d16a8"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "encoder_inputs.shape: (60,)\n",
            "decoder_inputs.shape: (60,)\n",
            "labels.shape: (60,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model"
      ],
      "metadata": {
        "id": "zjXk6pt-28kL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use the Transformer Architechture with the following hyperparameters."
      ],
      "metadata": {
        "id": "a9UFHuFO3dHe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Transformer parameters\n",
        "d_model = 256 # 512 in the original paper\n",
        "d_k = 32 # 64 in the original paper\n",
        "d_v = 32 # 64 in the original paper\n",
        "n_heads = 4 # 8 in the original paper\n",
        "n_encoder_layers = 4 # 6 in the original paper\n",
        "n_decoder_layers = 4 # 6 in the original paper\n",
        "\n",
        "max_token_length = MAX_TOKENS # 512 in the original paper"
      ],
      "metadata": {
        "id": "v9ULcpgbJ9WP"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class SingleHeadAttention(Layer):\n",
        "  def __init__(self, input_shape=(3, -1, d_model), dropout=.0, masked=None):\n",
        "    super(SingleHeadAttention, self).__init__()\n",
        "    self.q = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.normalize_q = Lambda(lambda x: x / np.sqrt(d_k))\n",
        "    self.k = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.v = Dense(d_v, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.dropout = dropout\n",
        "    self.masked = masked\n",
        "  \n",
        "  # Inputs: [query, key, value]\n",
        "  def call(self, inputs, training=None):\n",
        "    assert len(inputs) == 3\n",
        "    # We use a lambda layer to divide vector q by sqrt(d_k) according to the equation\n",
        "    q = self.normalize_q(self.q(inputs[0]))\n",
        "    k = self.k(inputs[1])\n",
        "    # The dimensionality of q is (batch_size, query_length, d_k) and that of k is (batch_size, key_length, d_k)\n",
        "    # So we will do a matrix multication by batch after transposing last 2 dimensions of k\n",
        "    # tf.shape(attn_weights) = (batch_size, query_length, key_length)\n",
        "    attn_weights = tf.matmul(q, tf.transpose(k, perm=[0,2,1]))\n",
        "    if self.masked: # Prevent future attentions in decoding self-attention\n",
        "      # Create a matrix where the strict upper triangle (not including main diagonal) is filled with -inf and 0 elsewhere\n",
        "      length = tf.shape(attn_weights)[-1]\n",
        "      #attn_mask = np.triu(tf.fill((length, length), -np.inf), k=1) # We need to use tensorflow functions instead of numpy\n",
        "      attn_mask = tf.fill((length, length), -np.inf)\n",
        "      attn_mask = tf.linalg.band_part(attn_mask, 0, -1) # Get upper triangle\n",
        "      attn_mask = tf.linalg.set_diag(attn_mask, tf.zeros((length))) # Set diagonal to zeros to avoid operations with infinity\n",
        "      # This matrix is added to the attention weights so all future attention will have -inf logits (0 after softmax)\n",
        "      attn_weights += attn_mask\n",
        "    # Softmax along the last dimension\n",
        "    attn_weights = tf.nn.softmax(attn_weights, axis=-1)\n",
        "    if training: # Attention dropout included in the original paper. This is possibly to encourage multihead diversity.\n",
        "      attn_weights = tf.nn.dropout(attn_weights, rate=self.dropout)\n",
        "    v = self.v(inputs[2])\n",
        "    return tf.matmul(attn_weights, v)\n",
        "\n",
        "class MultiHeadAttention_my(Layer):\n",
        "  def __init__(self, dropout=.0, masked=None):\n",
        "    super(MultiHeadAttention_my, self).__init__()\n",
        "    self.attn_heads = list()\n",
        "    for i in range(n_heads): \n",
        "      self.attn_heads.append(SingleHeadAttention(dropout=dropout, masked=masked))\n",
        "    self.linear = Dense(d_model, input_shape=(-1, n_heads * d_v), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    \n",
        "  def call(self, x, training=None):\n",
        "    attentions = [self.attn_heads[i](x, training=training) for i in range(n_heads)]\n",
        "    concatenated_attentions = tf.concat(attentions, axis=-1)\n",
        "    return self.linear(concatenated_attentions)\n",
        "\n",
        "class TransformerEncoder(Layer):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerEncoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerEncoder, self).build(input_shape)\n",
        "  def call(self, x, training=None):\n",
        "    sublayer1 = self.multihead_attention((x, x, x), training=training)\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer1 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    return layernorm2\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class TransformerDecoder(Layer):\n",
        "  def __init__(self, dropout=.0, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerDecoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_self_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate, masked=True)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.multihead_encoder_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout3 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization3 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerDecoder, self).build(input_shape)\n",
        "  def call(self, x, hidden, training=None):\n",
        "    sublayer1 = self.multihead_self_attention((x, x, x))\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.multihead_encoder_attention((x, hidden, hidden))\n",
        "    sublayer2 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    \n",
        "    sublayer3 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer3 = self.dropout3(sublayer3, training=training)\n",
        "    layernorm3 = self.layer_normalization2(layernorm2 + sublayer3)\n",
        "    return layernorm3\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class SinusoidalPositionalEncoding(Layer): # This is a TensorFlow implementation of https://github.com/graykode/nlp-tutorial/blob/master/5-1.Transformer/Transformer_Torch.ipynb\n",
        "  def __init__(self):\n",
        "    super(SinusoidalPositionalEncoding, self).__init__()\n",
        "    self.sinusoidal_encoding = np.array([self.get_positional_angle(pos) for pos in range(max_token_length)], dtype=np.float32)\n",
        "    self.sinusoidal_encoding[:, 0::2] = np.sin(self.sinusoidal_encoding[:, 0::2])\n",
        "    self.sinusoidal_encoding[:, 1::2] = np.cos(self.sinusoidal_encoding[:, 1::2])\n",
        "    self.sinusoidal_encoding = tf.cast(self.sinusoidal_encoding, dtype=tf.float32) # Casting the array to Tensor for slicing\n",
        "  def call(self, x):\n",
        "    return x + self.sinusoidal_encoding[:tf.shape(x)[1]]\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "  def get_angle(self, pos, dim):\n",
        "    return pos / np.power(10000, 2 * (dim // 2) / d_model)\n",
        "  def get_positional_angle(self, pos):\n",
        "    return [self.get_angle(pos, dim) for dim in range(d_model)]\n",
        "\n",
        "class Transformer(Model):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(Transformer, self).__init__(**kwargs)\n",
        "    self.encoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.decoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.pos_encoding = SinusoidalPositionalEncoding()\n",
        "    self.encoder = [TransformerEncoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_encoder_layers)]\n",
        "    self.decoder = [TransformerDecoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_decoder_layers)]\n",
        "    self.decoder_final = Dense(VOCAB_SIZE, input_shape=(None, d_model))\n",
        "  def call(self, inputs, training=None): # Source_sentence and decoder_input\n",
        "    source_sentence, decoder_input = inputs\n",
        "    embedded_source = self.encoding_embedding(source_sentence)\n",
        "    encoder_output = self.pos_encoding(embedded_source)\n",
        "    for encoder_unit in self.encoder:\n",
        "      encoder_output = encoder_unit(encoder_output, training=training)\n",
        "    \n",
        "    embedded_target = self.decoding_embedding(decoder_input)\n",
        "    decoder_output = self.pos_encoding(embedded_target)\n",
        "    for decoder_unit in self.decoder:\n",
        "      decoder_output = decoder_unit(decoder_output, encoder_output, training=training)\n",
        "    if training:\n",
        "      decoder_output = self.decoder_final(decoder_output)\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    else:\n",
        "      decoder_output = self.decoder_final(decoder_output[:, -1:, :])\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    return decoder_output"
      ],
      "metadata": {
        "id": "KvsIOzPnzt4F"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer = Transformer(dropout=0.3) # Instantiating a new transformer model\n",
        "train = [tf.cast(train_src, dtype=tf.float32), tf.cast(train_tgt, dtype=tf.float32)] # Cast the tuples to tensors\n",
        "validation = [tf.cast(train_dev_src, dtype=tf.float32), tf.cast(train_dev_tgt, dtype=tf.float32)]"
      ],
      "metadata": {
        "id": "eRsGRYTR6XQW"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.build([train_src.shape, train_tgt.shape])\n",
        "transformer.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IHXUpv-0eh-b",
        "outputId": "038eaaa2-b36a-46b1-874e-8414b508ef9f"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"transformer\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       multiple                  8192000   \n",
            "                                                                 \n",
            " embedding_1 (Embedding)     multiple                  8192000   \n",
            "                                                                 \n",
            " sinusoidal_positional_encod  multiple                 0         \n",
            " ing (SinusoidalPositionalEn                                     \n",
            " coding)                                                         \n",
            "                                                                 \n",
            " transformer_encoder (Transf  multiple                 658304    \n",
            " ormerEncoder)                                                   \n",
            "                                                                 \n",
            " transformer_encoder_1 (Tran  multiple                 658304    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " transformer_encoder_2 (Tran  multiple                 658304    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " transformer_encoder_3 (Tran  multiple                 658304    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " transformer_decoder (Transf  multiple                 790016    \n",
            " ormerDecoder)                                                   \n",
            "                                                                 \n",
            " transformer_decoder_1 (Tran  multiple                 790016    \n",
            " sformerDecoder)                                                 \n",
            "                                                                 \n",
            " transformer_decoder_2 (Tran  multiple                 790016    \n",
            " sformerDecoder)                                                 \n",
            "                                                                 \n",
            " transformer_decoder_3 (Tran  multiple                 790016    \n",
            " sformerDecoder)                                                 \n",
            "                                                                 \n",
            " dense (Dense)               multiple                  8224000   \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 30,401,280\n",
            "Trainable params: 30,401,280\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training"
      ],
      "metadata": {
        "id": "0JkChx5Ndp1s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])"
      ],
      "metadata": {
        "id": "InXkWf-LawJi"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "EPOCHS = 10\n",
        "checkpoint_filepath = '/Models/checkpoint'\n",
        "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n",
        "    filepath=checkpoint_filepath,\n",
        "    save_weights_only=True,\n",
        "    monitor = \"accuracy\",\n",
        "    mode='max',\n",
        "    save_best_only=True)"
      ],
      "metadata": {
        "id": "LMttctIwaNxG"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with tf.device('/device:GPU:0'):\n",
        "    history = transformer.fit(train, tf.cast(train_labels, dtype=tf.float32), batch_size=256, epochs=EPOCHS, callbacks=[model_checkpoint_callback])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bzdsT8kmBEMo",
        "outputId": "91b7dcda-a919-47e3-b9db-54318c5fcc6e"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "2844/2844 [==============================] - 2184s 759ms/step - loss: 1.8921 - accuracy: 0.7886\n",
            "Epoch 2/10\n",
            "2844/2844 [==============================] - 2163s 761ms/step - loss: 1.7719 - accuracy: 0.7932\n",
            "Epoch 3/10\n",
            "2844/2844 [==============================] - 2160s 759ms/step - loss: 1.8478 - accuracy: 0.7880\n",
            "Epoch 4/10\n",
            "2844/2844 [==============================] - 2157s 759ms/step - loss: 1.7938 - accuracy: 0.7908\n",
            "Epoch 5/10\n",
            "2844/2844 [==============================] - 2161s 760ms/step - loss: 1.7107 - accuracy: 0.7995\n",
            "Epoch 6/10\n",
            "2844/2844 [==============================] - 2160s 760ms/step - loss: 1.7082 - accuracy: 0.7997\n",
            "Epoch 7/10\n",
            "2844/2844 [==============================] - 2161s 760ms/step - loss: 1.6932 - accuracy: 0.8002\n",
            "Epoch 8/10\n",
            "2844/2844 [==============================] - 2161s 760ms/step - loss: 1.6768 - accuracy: 0.8005\n",
            "Epoch 9/10\n",
            "2844/2844 [==============================] - 2162s 760ms/step - loss: 1.6709 - accuracy: 0.8007\n",
            "Epoch 10/10\n",
            "2844/2844 [==============================] - 2160s 760ms/step - loss: 1.6626 - accuracy: 0.8009\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.save_weights(\"model_weights\")\n",
        "# transformer.save(\"./Models/\")"
      ],
      "metadata": {
        "id": "6tmeafiFdZ_U"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 10))\n",
        "plt.plot(history.history['accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "JbSCoj0cft6O",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 621
        },
        "outputId": "22bf9c19-b78b-46fa-f19c-4a2ac8f33369"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAJcCAYAAABAGii1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde3jc5X3n/c9X56MlWRr5IMuWD5KMIWCDIYQQBCUEGppA220C3XQ3bVO6TdOm2abdbNsrzeZ6sk/y7Paw7TZtkzz0XGg3aYCmJDI04ZBgEkxtCAI0YxsfJFujszyj88zc+8eMZNmW7bGtmd/8Zt6v69LlOfxm5jskyB/u+/7etznnBAAAgNxX5HUBAAAASA/BDQAAwCcIbgAAAD5BcAMAAPAJghsAAIBPENwAAAB8guAGIG+Z2V+a2f+T5rVHzOzdma4JAK4EwQ0AAMAnCG4AkOPMrMTrGgDkBoIbAE+lpih/w8xeNbNJM/v/zWyNmX3TzCJm9rSZNSy5/v1m1mNm42b2jJldteS5XWb2b6nX/YOkirM+68fM7EDqtS+Y2bVp1nivme03s1NmdtzMPnPW87em3m889fyHU49XmtnvmdlRM5sws++mHrvdzPqW+efw7tTtz5jZV83sb83slKQPm9lNZrY39Rknzex/m1nZktdfbWZPmdmomYXN7LfMbK2ZTZlZ45LrrjezITMrTee7A8gtBDcAueAnJd0lqUPS+yR9U9JvSQoo+XvqVyXJzDokPSLp11LPPSnpn82sLBViHpP0N5JWS/o/qfdV6rW7JD0s6RclNUr6c0lPmFl5GvVNSvoPkuol3Svpl8zs/tT7bkrV+8epmnZKOpB63f+UdIOkW1I1/aakRJr/TO6T9NXUZ/6dpLikT0hqkvQOSXdK+miqhlpJT0v6lqT1krZJ+lfn3ICkZyR9YMn7/oykR51z82nWASCHENwA5II/ds6FnXP9kp6X9H3n3H7n3Iykr0valbrug5L+xTn3VCp4/E9JlUoGo5sllUr6Q+fcvHPuq5JeWvIZD0n6c+fc951zcefcX0maTb3ugpxzzzjnfuicSzjnXlUyPHalnv5pSU875x5Jfe6Ic+6AmRVJ+jlJH3fO9ac+8wXn3Gya/0z2OuceS33mtHPuZefci865mHPuiJLBc6GGH5M04Jz7PefcjHMu4pz7fuq5v5L0IUkys2JJDyoZbgH4EMENQC4IL7k9vcz9mtTt9ZKOLjzhnEtIOi6pJfVcv3POLXnt0SW3N0n69dRU47iZjUtqTb3ugszs7Wb2ndQU44Sk/6TkyJdS73FomZc1KTlVu9xz6Th+Vg0dZvYNMxtITZ/+9zRqkKTHJe0ws81KjmpOOOd+cJk1AfAYwQ2An5xQMoBJkszMlAwt/ZJOSmpJPbZg45LbxyV9zjlXv+Snyjn3SBqf+/eSnpDU6pyrk/RnkhY+57ikrcu8ZljSzHmem5RUteR7FCs5zbqUO+v+n0p6U1K7c26VklPJS2vYslzhqVHLf1Ry1O1nxGgb4GsENwB+8o+S7jWzO1OL639dyenOFyTtlRST9KtmVmpmPyHppiWv/bKk/5QaPTMzq041HdSm8bm1kkadczNmdpOS06ML/k7Su83sA2ZWYmaNZrYzNRr4sKTfN7P1ZlZsZu9IrakLSqpIfX6ppN+RdLG1drWSTkmKmtl2Sb+05LlvSFpnZr9mZuVmVmtmb1/y/F9L+rCk94vgBvgawQ2AbzjnepUcOfpjJUe03ifpfc65OefcnKSfUDKgjCq5Hu6flrx2n6RfkPS/JY1JOpi6Nh0flfRZM4tI+rSSAXLhfY9Jeq+SIXJUycaE61JPf1LSD5Vcazcq6QuSipxzE6n3/IqSo4WTks7oMl3GJ5UMjBElQ+g/LKkhouQ06PskDUgKSbpjyfPfU7Ip4t+cc0unjwH4jJ25HAQAkI/M7NuS/t459xWvawFw+QhuAJDnzOxGSU8puUYv4nU9AC4fU6UAkMfM7K+U3OPt1whtgP8x4gYAAOATjLgBAAD4REEcXNzU1OTa2tq8LgMAAOCiXn755WHn3Nl7O0oqkODW1tamffv2eV0GAADARZnZebftYaoUAADAJwhuAAAAPkFwAwAA8ImCWOO2nPn5efX19WlmZsbrUjKqoqJCGzZsUGlpqdelAACAK1Swwa2vr0+1tbVqa2uTmXldTkY45zQyMqK+vj5t3rzZ63IAAMAVKtip0pmZGTU2NuZtaJMkM1NjY2PejyoCAFAoCja4Scrr0LagEL4jAACFoqCDGwAAgJ8Q3DwyPj6uL37xi5f8uve+970aHx/PQEUAACDXEdw8cr7gFovFLvi6J598UvX19ZkqCwAA5LCC7Sr12qc+9SkdOnRIO3fuVGlpqSoqKtTQ0KA333xTwWBQ999/v44fP66ZmRl9/OMf10MPPSTp9PFd0WhUP/qjP6pbb71VL7zwglpaWvT444+rsrLS428GAAAyheAm6b/9c49eP3FqRd9zx/pV+t33XX3e5z//+c/rtdde04EDB/TMM8/o3nvv1Wuvvba4bcfDDz+s1atXa3p6WjfeeKN+8id/Uo2NjWe8RygU0iOPPKIvf/nL+sAHPqCvfe1r+tCHPrSi3wMAAOQOgluOuOmmm87Ya+2P/uiP9PWvf12SdPz4cYVCoXOC2+bNm7Vz505J0g033KAjR45krV4AAJB9BDfpgiNj2VJdXb14+5lnntHTTz+tvXv3qqqqSrfffvuye7GVl5cv3i4uLtb09HRWagUAAN6gOcEjtbW1ikQiyz43MTGhhoYGVVVV6c0339SLL76Y5eoAAEAuYsTNI42NjXrnO9+pa665RpWVlVqzZs3ic/fcc4/+7M/+TFdddZU6Ozt18803e1gpAADIFeac87qGjNu9e7fbt2/fGY+98cYbuuqqqzyqKLsK6bsCAOB3Zvayc273cs8xVQoAAOATBDcAAACfKOjgVgjTxIXwHQEAKBQFG9wqKio0MjKS18HGOaeRkRFVVFR4XQoAAFgBBdtVumHDBvX19WloaMjrUjKqoqJCGzZs8LoMAACwAgo2uJWWlp5xUgEAAIAkzcbi6hub1tGRSR0Znkr+OZL88yv/8UZta67xrLaCDW4AAKBwzczHdWx0SkeGJ3V0ZEpHRk7/eWJ8WoklK6lqy0u0qalKV7fUeVdwCsENAADkpehsTEeXBLKjw8k/j41O6eTEmUdJNlSValNjtXZvatCm6zeoralKmxqr1dZYrYaqUpmZR9/iTAQ3AADgWxPT86enModPT2keGZnScHT2jGubasrV1lilW7Y2qa2xSpuaqpN/rq5WXVWpR9/g0hDcAABAznLOaWxqPjWVee6as7Gp+TOuX7uqQpsaq3Tn9mZtaqpSW2O1NjUmR89qyv0fe/z/DQAAgK855zQUmV0MY2evOYvMxBavNZNa6ivV1lit975t3WIwa2uqVmtDlSrLij38JplHcAMAABmXSDgNnJo5I5AtXXM2NRdfvLa4yNTaUKlNjdW6fmN9cq1Zas3ZhoZKlZfkdzi7EIIbAABYEbF4QicnkuHs7DVnR0enNBdLLF5bVlyk1tXJkbNbtjYtaQao0vr6SpUWF+wZARdEcAMAAOeVSDiNT89rODqb+pnTcGRWI5OzGo7MnfH4YGRG8/HT+2hUlBaprbFaWwLV+pHtzdq0uN6sSuvqKlVclBudmn5CcAMAoMDMxxManZzTUCQZukaiZwawpX+OTs4pnjj3eMiSIlNjTZkaq8vVVFuurc01WrOqQpuXrDlrri3PmW008gXBDQCAPDA9Fz8nfI2kbg8tuT0cndX4WZ2YCypKi9RUU66mmnK11Ffoug11aqopV2NN2eLjTanbdZWlKmLELOsIbgAA5CDnnE7NxJJBLDKrkcm5xdvDk8npyuFo6vHIrCaXLO5fqraiRIFU6OpYU6N3bGlMBrDa5GhZoLYsFc7KVV1WzAhZjiO4AQCQJfGE09jUQgA7d3py5Izbc5qLJ855DzOpsXphirJMO1fXL95uqilXYMkIWWNNWUF3YOYjghsAABkwNRfTD/smdOD4uPYfG9crfeMKn5rRMsvFVFpsi1ORjTVl6lxbe8a05NIRstXVZSzqL2AENwAArpBzTm8NT2r/sXHtPz6m/cfG9eZAZHFR/6bGKt20ebU2ra5SU215coSspkxNteVqqi7XqsoSpiiRFoIbAACXaGJ6Xq+kRtL2Hx/TgePjiwv+a8pLdF1rnX6pa6t2bazXztZ6NdaUe1wx8gXBDQCAC4gnnILhSDKkHRvT/uPjOjgYlZRcb9beXKO7d6zVro312rWxQduaa5jKRMYQ3AAAWGIoMptal5ac8ny1b3yxY3N1dZl2tdbr/p3rtbO1Qde21mlVRanHFaOQENwAAAVrLpbQ6ydPLYa0/cfHdHx0WlJyg9kd61fp392wQbs2NmjXxnptXF3FWjR4iuAGACgIzjmdmJg5HdKOjem1E6cWz89cV1ehXRvr9R9ubtOujfW6pqVOFaVspYHcQnADAOSlhe049i+Z9hyMzEqSykuKdO2GOn34ljbtaq3Xzo31WldX6XHFwMUR3AAAvnex7TjaGqv0zm1NyQaC1gZtX1er0uIij6sGLh3BDQDgO2dvx7H/2LgmppPbcdSWl2jnxnr98u1btXNjvXa2Nmh1dZnHFQMrg+AGAMhpsXhCwXB0MaDtPzamQ0OTkpLbcXSuqdV737ZWu1qTDQRbAzUcfo68RXADAOSUocjs4n5p+4+N6dW+CU2ltuNorC7Tro31+onrN2hXa73etqFOtWzHgQJCcAMAeGp6Lq6n3gjrqdfD2n9sTH1jp7fjuHr9Kn1gd+vi2rTW1ZVsx4GCRnADAGRdLJ7Q9w6N6PH9/eruGdDkXFyB2nLd1LY62em5sV5Xr2c7DuBsBDcAQFY45/RK34Qe29+vb7x6QsPROa2qKNH7rluv+3a26O2bV7M2DbgIghsAIKPeGp7UY/v79fiBfh0ZmVJZSZHu3N6s+3a26I7tAZWXMKoGpIvgBgBYcYORGX3jlZN6/EC/XumbkJn0ji2N+ujt23T3NWtVV0lDAXA5CG4AgBURnY2p+7UBPXagX987OKyEk65ev0q//d6r9L7r1mttXYXXJQK+R3ADAFy2uVhCzwWH9NiBfj39Rlgz8wltaKjUR2/fpvt3rde25lqvSwTySkaDm5ndI+l/SSqW9BXn3OfPen6jpL+SVJ+65lPOuSdTz/1XST8vKS7pV51z3em8JwAgsxIJp5ePjemx/f36lx+e1PjUvBqqSvVTN7Tq/l3rdf3GBrbsADIkY8HNzIol/YmkuyT1SXrJzJ5wzr2+5LLfkfSPzrk/NbMdkp6U1Ja6/YCkqyWtl/S0mXWkXnOx9wQAZEAwHEk1GZxQ//i0KkqL9J4da3X/rvV6V3uAsz+BLMjkiNtNkg465w5Lkpk9Kuk+SUtDlpO0KnW7TtKJ1O37JD3qnJuV9JaZHUy9n9J4TwDACjk5Ma0nDpzQYwdO6I2Tp1Rk0q3tAX3y7g69Z8daVZez4gbIpkz+G9ci6fiS+32S3n7WNZ+RtMfMfkVStaR3L3nti2e9tiV1+2LvKUkys4ckPSRJGzduvPTqAaBATUzP65s/PKnHDvTr+2+NyjnputZ6/e77dujHrl2vQG251yUCBcvr/1R6UNJfOud+z8zeIelvzOyalXhj59yXJH1Jknbv3u1W4j0BIF/NzMf1nTcH9diBfn3nzSHNxRPa3FStj9/Zrvt2tmhzU7XXJQJQZoNbv6TWJfc3pB5b6ucl3SNJzrm9ZlYhqekir73YewIA0hBPOH3/8IgeO9Cvb742oMhMTE015fr3N2/U/TtbdO2GOpoMgByTyeD2kqR2M9usZLh6QNJPn3XNMUl3SvpLM7tKUoWkIUlPSPp7M/t9JZsT2iX9QJKl8Z4AgPNwzqnnxCk9fqBf//zKSQ2cmlF1WbHuvmat7t/Zolu2NqqEJgMgZ2UsuDnnYmb2MUndSm7d8bBzrsfMPitpn3PuCUm/LunLZvYJJRsVPuycc5J6zOwflWw6iEn6ZedcXJKWe89MfQcAyBfHR6f0+IF+PXbghA4ORlVSZLq9M6DfvvcqvfuqNaos49gpwA8smZPy2+7du92+ffu8LgMAsmp0ck7/8mqyI/Tlo2OSpBvbGnTfzhbd+7Z1aqgu87hCAMsxs5edc7uXe87r5gQAwAqanovrqTfCenx/v54NDimWcOpYU6PfuLtT779uvVpXV3ldIoArQHADAJ+LxRP63qERPb6/X909A5qci2vtqgr9/K2bdd/OFl21rpYmAyBPENwAwIecc3qlb0KP7e/XN149oeHonGorSvS+69brvp0tevvm1SoqIqwB+YbgBgA+8tbwZOrYqX4dGZlSWUmR7tzerPt2tuiO7QGVl9BkAOQzghsA5KjpubgOD0d1aGhShwajeqZ3UK/0TchMunlzoz56+zbdfc1a1VWWel0qgCwhuAGAh5xzCp+a1eGhqA4NpULaUFSHhybVPz69eJ2ZtGPdKv3We7fr/de1aG1dhYdVA/AKwQ0AsmBmPq4jI5M6NLgQzJIh7fBQVJNz8cXrqsqKtTVQo91tDfpgoFVbAzXaEqjW5qZqVZQyDQoUOoIbAKwQ55yGorNLwlnqz+Go+samtXTbzJb6Sm0JVOundrdqa6BaWwI12hqo0ZpV5XSAAjgvghsAXKLZWFxHR6YWR80ODUZ1aHhShwejiszGFq+rLC3WlkC1drY26Cev35AKZ8nRs6oyfv0CuHT85gCAZTjnNDI5tzhqdmgwqsPDydvHR6eUWDJ6tq6uQlsC1frx61u0palaW5trtCVQo3WrKtiSA8CKIrgBKGhzsYSOjU6lGgOWTG8OTWpien7xuvKSIm1uqtY1LXW677r1yXDWVKPNgWrVlPOrFEB28NsGQEEYm5w7I5gt3D46OqX4kuGz5tpybQ3U6MeuXbfYGLA1UKOW+kpGzwB4juAGIG/E4snRs7PD2aGhqMamTo+elRUnR88619bqvW9bp63N1drSlAxptRXsiQYgdxHcAPje9FxcD375RfWcmNB8/PToWVNNubYEqnXPNeu0NTVytjVQo5aGShUzegbAhwhuAHzv9ZOndOD4uH5iV4tu2da0uL0GJwoAyDcENwC+FwxHJEm/9u4ObWys8rgaAMicIq8LAIArFQxHVFlarA0NlV6XAgAZRXAD4HuhcFTbmmvo+gSQ9whuAHwvGI6oY02t12UAQMYR3AD42vjUnAYjs+pYU+N1KQCQcQQ3AL4WDEcliRE3AAWB4AbA1xY6SjvWEtwA5D+CGwBfC4Yjqikv0fq6Cq9LAYCMI7gB8LVgOKJtzTUyo6MUQP4juAHwtVA4SmMCgIJBcAPgW8PRWY1MztGYAKBgENwA+NZiYwLBDUCBILgB8K0QW4EAKDAENwC+FQxHtKqiRGtWlXtdCgBkBcENgG8tHHVFRymAQkFwA+BLzjkFw1G1M00KoIAQ3AD40lBkVhPT82wFAqCgENwA+NLCGaWdjLgBKCAENwC+1JvaCoSpUgCFhOAGwJdC4YgaqkrVVFPmdSkAkDUENwC+REcpgEJEcAPgO8651BmlTJMCKCwENwC+c3JiRpHZGB2lAAoOwQ2A7wRpTABQoAhuAHyHM0oBFCqCGwDf6Q1H1FRTrtXVdJQCKCwENwC+EwpHWN8GoCAR3AD4SiLhFBqkoxRAYSK4AfCV/vFpTc3FCW4AChLBDYCvLHSUMlUKoBAR3AD4ysLh8mwFAqAQEdwA+EooHNHaVRWqqyz1uhQAyDqCGwBf6Q1H1M40KYACRXAD4BvxhNNBOkoBFDCCGwDfOD46pdlYgsYEAAWL4AbAN053lDLiBqAwEdwA+AaHywModAQ3AL4RDEfVUl+pmvISr0sBAE8Q3AD4RpAzSgEUOIIbAF+IxRM6PDTJ+jYABY3gBsAXjoxMaS6eYH0bgIJGcAPgCyHOKAUAghsAfwiGozKTtjUT3AAULoIbAF8IhiNqbahSVRkdpQAKF8ENgC/QUQoABDcAPjAXS+itYTpKAYDgBiDnvTU8qVjCEdwAFDyCG4Ccd/qoK6ZKARQ2ghuAnBcKR1Rk0tYAwQ1AYSO4Ach5wXBUbY3Vqigt9roUAPAUwQ1AzguGI0yTAoAIbgBy3Mx8XEdG6CgFACnDwc3M7jGzXjM7aGafWub5PzCzA6mfoJmNL3nuC2b2Wurng0sev9PM/i31mu+a2bZMfgcA3jo8NKmEE8ENACRlbAtyMyuW9CeS7pLUJ+klM3vCOff6wjXOuU8suf5XJO1K3b5X0vWSdkoql/SMmX3TOXdK0p9Kus8594aZfVTS70j6cKa+BwBvBRfPKCW4AUAmR9xuknTQOXfYOTcn6VFJ913g+gclPZK6vUPSc865mHNuUtKrku5JPeckrUrdrpN0YsUrB5AzguGISopMm5uqvS4FADyXyeDWIun4kvt9qcfOYWabJG2W9O3UQ69IusfMqsysSdIdklpTz31E0pNm1ifpZyR9/jzv+ZCZ7TOzfUNDQ1f8ZQB4IxiOqq2pWmUlLMkFgFz5TfiApK865+KS5JzbI+lJSS8oOQq3V1I8de0nJL3XObdB0l9I+v3l3tA59yXn3G7n3O5AIJDp+gFkSGgwok6mSQFAUmaDW79Oj5JJ0obUY8t5QKenSSVJzrnPOed2OufukmSSgmYWkHSdc+77qcv+QdItK1s2gFwxPRfXsdEptgIBgJRMBreXJLWb2WYzK1MynD1x9kVmtl1Sg5KjaguPFZtZY+r2tZKulbRH0pikOjPrSF16l6Q3MvgdAHjo4GBUjo5SAFiUsa5S51zMzD4mqVtSsaSHnXM9ZvZZSfuccwsh7gFJjzrn3JKXl0p63swk6ZSkDznnYpJkZr8g6WtmllAyyP1cpr4DAG/RUQoAZ8pYcJMk59yTSq5VW/rYp8+6/5llXjejZGfpcu/5dUlfX7kqAeSq4GBEZcVFamus8roUAMgJudKcAADnCA5EtCVQrZJiflUBgERwA5DDguGo2pkmBYBFBDcAOSk6G1P/+LQ66SgFgEUENwA5KZRqTGDEDQBOI7gByEmhcFQSHaUAsBTBDUBOCoYjKi8p0sbVdJQCwAKCG4CcFByMaltzjYqLzOtSACBnENwA5KTgQIRpUgA4C8ENQM6ZmJ7XwKkZzigFgLMQ3ADknIODyY7STkbcAOAMBDcAOad3gI5SAFgOwQ1AzgmGI6osLVZLfaXXpQBATiG4Acg5ocGI2tfUqIiOUgA4A8ENQM4JhqNMkwLAMghuAHLK2OSchiKz6qCjFADOQXADkFOCnFEKAOdFcAOQU4KDyY5StgIBgHMR3ADklOBARLXlJVpXV+F1KQCQcwhuAHJKMBzRtjU1MqOjFADORnADkFNCg1F1NDNNCgDLIbgByBnD0VmNTs6pYy3BDQCWQ3ADkDOCA8mOUrYCAYDlEdwA5IyFrUDYfBcAlkdwA5AzgoNR1VWWqrm23OtSACAnEdwA5IzgQEQddJQCwHkR3ADkBOecguEIJyYAwAUQ3ADkhMHIrE7NxNTRTGMCAJwPwQ1ATlhsTGArEAA4L4IbgJzQO0BHKQBcDMENQE4IhaNaXV2mpho6SgHgfAhuAHJCcDDCxrsAcBEENwCec84pFI4yTQoAF0FwA+C5ExMzis7G2AoEAC6C4AbAc4sdpWwFAgAXRHAD4LkQZ5QCQFoIbgA81zsQVaC2XA3VZV6XAgA5jeAGwHMhOkoBIC0ENwCeSiToKAWAdBHcAHiqf3xa0/NxghsApIHgBsBTp4+6YqoUAC6G4AbAU8HBZHDb1syIGwBcDMENgKdC4ajW1VWorrLU61IAIOcR3AB4qncgwokJAJAmghsAz8QTToeGopyYAABpIrgB8Myx0SnNxhLqWMuIGwCkg+AGwDNBjroCgEtCcAPgmWBqK5B2pkoBIC0ENwCeCQ5G1VJfqeryEq9LAQBfILgB8EwoHFEn69sAIG0ENwCemI8ndGgoqnZOTACAtBHcAHji6Mik5uNOHZyYAABpI7gB8EQwHJVERykAXAqCGwBPBMMRmUnb6CgFgLQR3AB4IhiOaOPqKlWWFXtdCgD4BsENgCeC4ajaWd8GAJeE4AYg62ZjcR0ZnlTnWqZJAeBSENwAZN1bw5OKJRyNCQBwiQhuALJuoaOUqVIAuDQENwBZFwpHVGTSlkC116UAgK8Q3ABkXTAcUVtTtSpK6SgFgEtBcAOQdcFwlBMTAOAyENwAZNXMfFxHRybVwRmlAHDJCG4AsurQUFQJJ3WsZcQNAC4VwQ1AVgXDEUmcUQoAl4PgBiCrguGoSopMbY10lALApSK4AciqUDiizU3VKivh1w8AXCp+cwLIqmA4yvo2ALhMBDcAWTM1F9Ox0Sm2AgGAy5TR4GZm95hZr5kdNLNPLfP8H5jZgdRP0MzGlzz3BTN7LfXzwSWPm5l9LnX9G2b2q5n8DgBWzsHB5FFXbAUCAJenJFNvbGbFkv5E0l2S+iS9ZGZPOOdeX7jGOfeJJdf/iqRdqdv3Srpe0k5J5ZKeMbNvOudOSfqwpFZJ251zCTNrztR3ALCyFs4oZaoUAC5PJkfcbpJ00Dl32Dk3J+lRSfdd4PoHJT2Sur1D0nPOuZhzblLSq5LuST33S5I+65xLSJJzbjAj1QNYccFwRGXFRdq0usrrUgDAlzIZ3FokHV9yvy/12DnMbJOkzZK+nXroFUn3mFmVmTVJukPJUTZJ2irpg2a2z8y+aWbt53nPh1LX7BsaGlqBrwPgSgXDEW0JVKukmOW1AHA5cuW35wOSvuqci0uSc26PpCclvaDkKNxeSfHUteWSZpxzuyV9WdLDy72hc+5LzrndzrndgUAg0/UDSEMoHGXjXQC4ApkMbv06PUomSRtSjy3nAZ2eJpUkOec+55zb6Zy7S5JJCqae6pP0T6nbX5d07YpVDCBjIjPz6h+fVifr2wDgsmUyuL0kqd3MNptZmZLh7ImzLzKz7ZIalBxVW3is2MwaU7evVTKc7Uk9/ZiSU6eS1KXTgQ5ADgulOkrbm+koBYDLlbGuUudczMw+JqlbUrGkh1F6cpMAACAASURBVJ1zPWb2WUn7nHMLIe4BSY8659ySl5dKet7MJOmUpA8552Kp5z4v6e/M7BOSopI+kqnvAGDlhDijFACuWMaCmyQ5555Ucq3a0sc+fdb9zyzzuhklO0uXe89xSfeuXJUAsiEYjqqitEitdJQCwGXLleYEAHkuGI5oW3ONiovM61IAwLcIbgCyIhiOcNQVAFwhghuAjJuYnlf41KzaWd8GAFeE4AYg4xYaEzrX0lEKAFeC4AYg43pTwa2dqVIAuCIENwAZFwpHVVVWrJb6Sq9LAQBfI7gByLhgOKL2NbUqoqMUAK4IwQ1AxgXDUXVwYgIAXDGCG4CMGp2c03B0lhMTAGAFENwAZFRwoTFhDSNuAHClCG4AMur0ViCMuAHAlSK4Acio3nBEteUlWruqwutSAMD3CG4AMioYjqp9TY3M6CgFgCtFcAOQMc45hcIRGhMAYIUQ3ABkzHB0TmNT8wQ3AFghBDcAGbPQUUpwA4CVQXADkDGngxtbgQDASiC4AciYYDiq+qpSBWrLvS4FAPICwQ1AxgTDEXU019JRCgArhOAGICOcc6nD5ZkmBYCVQnADkBHhU7OKzMRoTACAFURwA5ARdJQCwMojuAHICDpKAWDlEdwAZEQwHFFjdZkaa+goBYCVQnADkBHBcJRpUgBYYQQ3ACvu9BmlTJMCwEoiuAFYcf3j05qci6udETcAWFEENwArLhSOSqKjFABWGsENwIqjoxQAMoPgBmDF9YYjaq4tV31VmdelAEBeIbgBWHEhOkoBICMIbgBWVCLhdHCQ4AYAmUBwA7Ci+samNT0fZ30bAGQAwQ3AiupNNSawFQgArDyCG4AVFVwMboy4AcBKSyu4mdk/mdm9ZkbQA3BBoXBE6+sqtKqi1OtSACDvpBvEvijppyWFzOzzZtaZwZoA+FhvOMo0KQBkSFrBzTn3tHPu30u6XtIRSU+b2Qtm9rNmxn9WA5AkxRNOh4aiNCYAQIakPfVpZo2SPizpI5L2S/pfSga5pzJSGQDfOToyqblYgq1AACBDStK5yMy+LqlT0t9Iep9z7mTqqX8ws32ZKg6AvwQ5oxQAMiqt4Cbpj5xz31nuCefc7hWsB4CPLXSUbmtmqhQAMiHdqdIdZla/cMfMGszsoxmqCYBPBcMRbWioVHV5uv9NCAC4FOkGt19wzo0v3HHOjUn6hcyUBMCvQuGoOpkmBYCMSTe4FZuZLdwxs2JJZZkpCYAfzccTOjzMViAAkEnpzmd8S8lGhD9P3f/F1GMAIEk6Mjyp+bhjKxAAyKB0g9t/UTKs/VLq/lOSvpKRigD4Eh2lAJB5aQU351xC0p+mfgDgHMFwREVGRykAZFK6+7i1S/p/Je2QVLHwuHNuS4bqAuAzwXBEG1dXqaK02OtSACBvpduc8BdKjrbFJN0h6a8l/W2migLgP8FwhMYEAMiwdINbpXPuXyWZc+6oc+4zku7NXFkA/GQ2FteRkSm2AgGADEu3OWHWzIokhczsY5L6JbGQBYAk6fDQpOIJp3Y6SgEgo9Idcfu4pCpJvyrpBkkfkvQfM1UUAH9ZOOqKjlIAyKyLjrilNtv9oHPuk5Kikn4241UB8JVQOKriItOWQLXXpQBAXrvoiJtzLi7p1izUAsCnguGI2hqrVF5CRykAZFK6a9z2m9kTkv6PpMmFB51z/5SRqgD4SjAc0VXrVnldBgDkvXSDW4WkEUk/suQxJ4ngBhS4mfm4jo5O6f07W7wuBQDyXronJ7CuDcCyDg5G5ZzYCgQAsiDdkxP+QskRtjM4535uxSsC4CunO0rZCgQAMi3dqdJvLLldIenHJZ1Y+XIA+E0wHFVpsamtiY5SAMi0dKdKv7b0vpk9Ium7GakIgK+EwhFtbqpWaXG620ICAC7X5f6mbZfUvJKFAPCn4GCEjXcBIEvSXeMW0Zlr3AYk/ZeMVATANyZnYzo+Oq2fuqHV61IAoCCkO1XKf04DOMfBwagkGhMAIFvSmio1sx83s7ol9+vN7P7MlQXADzijFACyK901br/rnJtYuOOcG5f0u5kpCYBfBMMRlZUUaVMjHaUAkA3pBrflrkt3KxEAeSoYjmproEbFReZ1KQBQENINbvvM7PfNbGvq5/clvZzJwgDkvlA4wvo2AMiidIPbr0iak/QPkh6VNCPply/2IjO7x8x6zeygmX1qmef/wMwOpH6CZja+5LkvmNlrqZ8PLvPaPzKzaJr1A1hhkZl5nZiYYX0bAGRRul2lk5LOCV4XYmbFkv5E0l2S+iS9ZGZPOOdeX/K+n1hy/a9I2pW6fa+k6yXtlFQu6Rkz+6Zz7lTq+d2SGi6lHgArKxhe6CgluAFAtqTbVfqUmdUvud9gZt0XedlNkg465w475+aUHKm77wLXPyjpkdTtHZKec87FUqHxVUn3pD67WNL/kPSb6dQOIDNCnFEKAFmX7lRpU6qTVJLknBvTxU9OaJF0fMn9vtRj5zCzTZI2S/p26qFXJN1jZlVm1iTpDkkLO3x+TNITzrmTF/pwM3vIzPaZ2b6hoaGLlArgUgXDUVWUFqm1ocrrUgCgYKTbGZows43OuWOSZGZtOvMkhSv1gKSvOufikuSc22NmN0p6QdKQpL2S4ma2XtJPSbr9Ym/onPuSpC9J0u7du1eyVgCSQoMRtTfXqoiOUgDImnRH3H5b0nfN7G/M7G8lPSvpv17kNf06PUomSRtSjy3nAZ2eJpUkOec+55zb6Zy7S5JJCiq5Bm6bpINmdkRSlZkdTPM7AFhBvQMRtTNNCgBZlW5zwrdSDQEPSdov6TFJ0xd52UuS2s1ss5KB7QFJP332RWa2XclGg71LHiuWVO+cGzGzayVdK2mPcy4mae2S66LOuW3pfAcAK2dial6DkVkaEwAgy9I9ZP4jkj6u5KjZAUk3Kxm0fuR8r3HOxczsY5K6JRVLetg512Nmn5W0zzn3ROrSByQ96pxbOp1ZKul5M5OkU5I+lAptAHJAcDDZmNBJcAOArEp3jdvHJd0o6UXn3B2pUbL/frEXOeeelPTkWY99+qz7n1nmdTNKdpZe7P2ZpwE80DuQDG5MlQJAdqW7xm0mFaZkZuXOuTcldWauLAC5LBSOqLqsWC31lV6XAgAFJd0Rt77UPm6PSXrKzMYkHc1cWQByWTAcVfuaWqWWMwAAsiTd5oQfT938jJl9R1KdpG9lrCoAOS00GNGPbL/YVo4AgJWW7ojbIufcs5koBIA/jERnNRydo6MUADyQ7ho3AJB0+ozSdoIbAGQdwQ3AJQmxFQgAeIbgBuCS9A5EVFtRojWryr0uBQAKDsENwCUJhaPqoKMUADxBcAOQNuecgoMRGhMAwCMENwBpG4rOanxqXh2cmAAAniC4AUhbcCDZUcqIGwB4g+AGIG3BMGeUAoCXCG4A0hYajKihqlSBGjpKAcALBDcAaesdiHBGKQB4iOAGIC3OudRWIEyTAoBXCG4A0jJwakaR2RiNCQDgIYIbgLQsnFFKcAMA7xDcAKQlOJDsKCW4AYB3CG4A0hIMR9RUU6bV1WVelwIABYvgBiAtwcEoo20A4DGCG4CLSiScQmHOKAUArxHcAFxU//i0pubinJgAAB4juAG4qNAgjQkAkAsIbrioocisHvzSi3preNLrUuCRxa1AmgluAOAlghsu6huvntDewyN6/EC/16XAI8GBiNasKlddVanXpQBAQSO44aK6ewYkSc8GhzyuBF4JDtKYAAC5gOCGCxqdnNMP3hrVqooSvXJ8XGOTc16XhCyLJ5wOshUIAOQEghsu6Ok3wko46ZN3dyrhpO8eHPa6JGTZ8dEpzcwnOFweAHIAwQ0XtKdnQC31lfrpmzaqrrJUzzFdWnCC4WRHaTsjbgDgOYIbzmtyNqbnQsO6a8calRQX6db2Jj0bHJJzzuvSkEWhwWRHaXszI24A4DWCG87r2eCQ5mIJ3X31WklSV0dAg5FZvZk6bByFIRiOqKW+UrUVdJQCgNcIbjiv7p4BNVSV6sa2BknSbe0BSWK6tMD0DkQ4MQEAcgTBDcuaiyX07TcH9e6rktOkkrS2rkLb19ayLUgBicUTOjw0SUcpAOQIghuWtffwiCIzscVp0gVdHQG9dGRUk7MxjypDNh0dndJcPEFwA4AcQXDDsrp7BlRVVqxb25vOePy2joDm4057D414VBmyKRReOKOUqVIAyAUEN5wjkXB66vWwbu8MqKK0+Izndrc1qLK0WM+FmC4tBL0DyY7SbXSUAkBOILjhHPuPj2koMnvONKkklZcU65atjaxzKxDBwYhaV1eqqqzE61IAACK4YRndPWGVFpvu2N687PNdnQEdHZnSkeHJLFeGbAuFI+pkfRsA5AyCG87gnFN3z4DesbVJq86zb9fitiBMl+a1uViyo5QTEwAgdxDccIbecERHR6Z099VrzntNW1O1NjVW6dlegls+OzIyqVjC0ZgAADmE4IYzdL8Wlpl0147zBzcpuS3IC4dGNBuLZ6kyZFtwsaOUETcAyBUEN5yhu2dA129sUHNtxQWvu609oOn5uPYdGctSZci2YDiqIpO2BhhxA4BcQXDDouOjU3r95KkLTpMueMfWRpUWG8df5bHgQESbGqvP2RIGAOAdghsWdfcMSNKy24Ccrbq8RDe2rWZbkDwWHIyonf3bACCnENywaE9PWNvX1mpTY3Va13d1BPTmQEQDEzMZrgzZNjMf19GRKXWuZX0bAOQSghskScPRWb10dFTvSWO0bcFtHWwLkq8OD00qnnBsBQIAOYbgBknS06+H5ZzSWt+2YPvaWjXXljNdmodCg5xRCgC5iOAGScn1bRsaKrVj3aq0X2Nm6uoI6LuhYcXiiQxWh2wLhiMqKTJtaSK4AUAuIbhBkZl5fe/giO6+eq3M7JJe29UZ0MT0vF7pm8hQdfBCMBxVW1O1ykr4FQEAuYTfytAzvUOaiyfS6iY9263bmlRkYluQPBMMR5gmBYAcRHCDunsG1Fhdphs2NVzya+urynRdaz3r3PLI9Fxcx0an1N5MYwIA5BqCW4GbjcX1TO+Q7tqxRsVFlzZNuqCrI6BX+sY1Njm3wtXBC4eGonJObAUCADmI4FbgXjg4ouhs7LKmSRfc1hGQc9J3Dw6vYGXwSu8AHaUAkKsIbgWuu2dANeUlumVb42W/x3Ub6lVXWcp0aZ4IDkZUWmxpb8QMAMgeglsBiyecnno9rNs7AyovufzzKIuLTO9qb9KzwSE551awQnghFI5qS1ONSov59QAAuYbfzAXs5aNjGpmcu6Jp0gVdHQENRWb1xsnIClQGLwXDEXWwvg0AchLBrYB19wyorLhIt3cGrvi9OP4qP0zOxtQ3Nq0ODpcHgJxEcCtQzjl19wzondsaVVtResXvt2ZVhbavrdWzvQQ3PwsNRiWJM0oBIEcR3ArU6ydPqW9sekWmSRd0dQa07+ioJmdjK/aeyK5gODnVzVYgAJCbCG4FqrsnrCKT3r0j/UPlL6arPaD5uNPeQyMr9p7IruBAROUlRdq4usrrUgAAyyC4Fag9PQPavWm1mmrKV+w9b2hrUFVZMduC+FhwMKqtgZrL3owZAJBZBLcCdHRkUm8ORPSeq1dutE2SykuKdcvWRj0THGRbEJ8KcUYpAOQ0glsB6u4ZkKQVXd+2oKsjoOOj0zoyMrXi743MOjUzr5MTM2wFAgA5jOBWgLp7wrp6/Sq1ZmAd0+K2IEyX+k4o1ZjQweHyAJCzCG4FZjAyo387NpaR0TZJ2tRYrbbGKta5+VAwnNwKpIOtQAAgZxHcCsxTr4flXGamSRd0dQS099CIZmPxjH0GVl4wHFFlabE2NFR6XQoA4DwIbgWmuyestsaqjC5A7+oMaHo+rn1HxjL2GVh5oXBU7WtqVERHKQDkLIJbATk1M6+9h4Z199VrZZa5v5xv3tKosuIipkt9pjccUTvr2wAgp2U0uJnZPWbWa2YHzexTyzz/B2Z2IPUTNLPxJc99wcxeS/18cMnjf5d6z9fM7GEzu/LzmgrEd94c1Hzc6T0ZnCaVpKqyEt24uYHjr3xkfGpOQ5FZtgIBgByXseBmZsWS/kTSj0raIelBM9ux9Brn3Cecczudczsl/bGkf0q99l5J10vaKentkj5pZqtSL/s7SdslvU1SpaSPZOo75JvungE115ZrV2t9xj+rqyOg3nBEJyemM/5ZuHKLjQlsBQIAOS2TI243STronDvsnJuT9Kik+y5w/YOSHknd3iHpOedczDk3KelVSfdIknPuSZci6QeSNmTsG+SRmfm4nukd0l071mRlDdPCtiDPB4cz/lm4cr0LW4HQUQoAOS2Twa1F0vEl9/tSj53DzDZJ2izp26mHXpF0j5lVmVmTpDsktZ71mlJJPyPpW+d5z4fMbJ+Z7RsaYsru+dCwpubiGe0mXapzTa3WrCpnnZtPhMIR1ZSXaH1dhdelAAAuIFeaEx6Q9FXnXFySnHN7JD0p6QUlR+H2Sjp7b4kvKjkq9/xyb+ic+5JzbrdzbncgEMhc5T7R3TOg2ooS3bylMSufZ2bq6gjo+dCQYvFEVj4Tly8Yjqh9TU1Gm1YAAFcuk8GtX2eOkm1IPbacB3R6mlSS5Jz7XGr9212STFJw4Tkz+11JAUn/eUUrzlOxeEL/+kZYd25vVllJ9rJ6V0ezTs3E9ErfRNY+E5cnFI5yYgIA+EAm/xZ/SVK7mW02szIlw9kTZ19kZtslNSg5qrbwWLGZNaZuXyvpWkl7Uvc/IuluSQ865xjKScMPjoxqbGo+a9OkC27d1qQiE9OlOW44OquRyTm101EKADkvY8HNOReT9DFJ3ZLekPSPzrkeM/usmb1/yaUPSHo01WywoFTS82b2uqQvSfpQ6v0k6c8krZG0N7WNyKcz9R3yxZ6esMpLitTVmd0p47qqUu1srSe45bggjQkA4BslmXxz59yTSq5VW/rYp8+6/5llXjejZGfpcu+Z0ZrzjXNOe3oG9K72gKrKsv+PrqujWX/4r0GNTs5pdXVZ1j8fFxdKbQXSyVYgAJDzcqU5ARnyw/4JnZiY0d1Xr/Hk82/raJJz0ncPsi1IruoNR7SqokTNteVelwIAuAiCW57r7hlQcZHp3Vd5E9yu3VCv+qpSTlHIYaFwRB1raukoBQAfILjlue6esG5qW60Gj6Ypi4tM72oP6LnQkM5cxohc4JxTMBzlxAQA8AmCWx47NBTVwcGoZ9OkC7o6AhqKzOqNkxFP68C5hiKzmpieV0czHaUA4AcEtzzW3TMgSRk/VP5ibmtvksS2ILmIo64AwF8IbnmsuyesazfUaX19pad1NK+q0FXrVunZ4KCndeBcC4fLtxPcAMAXCG55amBiRq8cH8/6prvn09UR0L4jY4rOxi5+MbImFI5odXWZmmrYqgUA/IDglqf2vJ6cJvV6fduCro6AYgmnvYdGvC4FS/SGI2pv5oxSAPALglue6u4Z0JZAtbblyPmTN2xqUHVZMdOlOcQ5p4PhKOvbAMBHCG55aHxqTi8eHs2ZaVJJKisp0ju2NunZINuC5IqTEzOKzMbYCgQAfITglof+9Y1BxRMup4KbJHV1BnR8dFpHRqa8LgVackYpW4EAgG8Q3PJQd8+A1q6q0LUtdV6Xcoau9uQh98/2Ml2aCzhcHgD8h+CWZ6bn4nouNKT3XL1GRUW5teB8Y2OVNjdVs59bjgiGo2qqKffsVA0AwKUjuOWZZ4NDmplP5Nw06YKujoD2Hh7RzHzc61IKXigcUedapkkBwE8IbnlmT8+A6ipLddPm1V6XsqyujoBm5hPad2TM61IKWiKRPKO0PUe6jgEA6SG45ZH5eEJPvxHWnVc1q7Q4N/+nffuW1SorLmJbEI/1j09rej7O+jYA8Jnc/Nsdl+X7h0d1aiaWs9OkklRVVqKbNq9mnZvHTjcmMFUKAH5CcMsj3T0Dqigt0m2p7s1c1dURUDAc1cmJaa9LKVicUQoA/kRwyxOJhNOe1wfU1RFQZVmx1+Vc0G0dyWD5HKNungmGI1q7qkJ1laVelwIAuAQEtzzxSt+4wqdmc3qadEHHmhqtXVXBdKmHguGI2pkmBQDfIbjlie6esEqKTHduz41D5S/EzNTVEdDzoWHF4gmvyyk48YTTwcGoOpkmBQDfIbjlAeec9vQM6OYtjaqr8sfUV1dnQJGZmF7pG/e6lIJzbHRKs7EEHaUA4EMEtzxwcDCqw8OTuvvq3B9tW/DOrU0qMunZXqZLs22ho5SpUgDwH4JbHujuGZAk3bUj99e3LairKtWujQ2sc/NAaDG4MeIGAH5DcMsD3T1h7Wyt19q6Cq9LuSRdHQG92j+h0ck5r0spKMFwVC31laopL/G6FADAJSK4+Vz/+LR+2D/hi27Ss93WEZBz0vMhRt2yKRiOsPEuAPgUwc3n9qSmSf20vm3B21rq1FBVynRpFsXiCR0emqQxAQB8iuDmc909A2pvrtGWgP9GUIqLTO9qD+i54LASCed1OQXhyMiU5uJ0lAKAXxHcfGx0ck4/eGvUl9OkC7o6AhqOzuqNgVNel1IQQotnlBLcAMCPCG4+9vQbYSWcfB3c3tXRJElMl2ZJbzgiM2lbs/9GaAEABDdf29MzoJb6Sl3TssrrUi5bc22FdqxbxX5uWRIKR9XaUJXz59kCAJZHcPOpydmYngsN664da2RmXpdzRbo6A3r56JgiM/Nel5L3kh2lTJMCgF8R3Hzq2eCQ5mIJX0+TLujqCCiWcNp7aMTrUvLaXCyht4Yn2QoEAHyM4OZT3T0Daqgq1Y1tDV6XcsWu39ig6rJi1rll2FvDk4olHCNuAOBjBDcfmosl9O03B/Xuq9aopNj//xOWlRTplm1NejY4JOfYFiRTgnSUAoDv+f9v/QK09/CIIjOxvJgmXdDVEVDf2LTeGp70upS8FQpHVGTSlkC116UAAC4Twc2HunsGVFVWrFvbm7wuZcV0dQQksS1IJvWGI2prrFZFKR2lAOBXBDefSSScnno9rNs7A3n1F3Dr6iptaaomuGVQKBxVO40JAOBrBDef2X98TEOR2byaJl1wW0dALx4e0cx83OtS8s7MfFxHRibVyfo2APA1gpvPdPeEVVpsumN7s9elrLiuzoBm5hN66cio16XknUNDUSWc1E5wAwBfI7j5iHNO3T0DesfWJq2qKPW6nBV38+ZGlZUUcYpCBoTCUUl0lAKA3xHcfKQ3HNHRkSndffUar0vJiMqyYr1982rWuWXAN149qdqKEm1uoqMUAPyM4OYj3a+FZSbdtSM/g5uU7C4NDUZ1Ynza61LyxstHx/T0G2H94m1bVFbCv/IA4Gf8FveR7p4BXb+xQc21FV6XkjG3pbYFeY5RtxXhnNP/96031VRTpp9952avywEAXCGCm08cH53S6ydP5e006YL25hqtq6tgunSFPBca1vffGtXH7tim6vISr8sBAFwhgptPdPcMSFJebgOylJmpqyOg7x4cViye8LocX0sknP5H95tqqa/Ug2/f6HU5AIAVQHDziT09YW1fW6tNjfm/uLyrI6DITEwHjo97XYqvffO1Ab3Wf0r/+a4OlZfkz2bNAFDICG4+MByd1UtHR/WePB9tW3DLtiYVFxnTpVcgFk/o957qVceaGt2/q8XrcgAAK4Tg5gNPvx6Wc8r79W0L6ipLtau1nuB2Bb72b306PDSpX39Pp4qLzOtyAAArhODmA909A9rQUKkd61Z5XUrWdHUE9MP+CY1EZ70uxXdm5uP6w6dD2tlar/fk8dYxAFCICG45LjIzr+8dHNHdV6+VWeGMnHR1BuSc9N2Dw16X4jt/++JRnZyY0W/e3VlQ/58BgEJAcMtxz/QOaS6eyPtu0rNds75Oq6vLOP7qEkVnY/riM4d067Ym3bKtyetyAAArjOCW47p7BtRYXaYbNjV4XUpWFRWZ3tXepOdCQ0oknNfl+MZXnj+s0ck5/cbdnV6XAgDIAIJbDpuNxfVM75Du2rGmIBeYd3UENByd0+snT3ldii+MTs7pK8+/pXuuXqvrWuu9LgcAkAEEtxz2wsERRWdjBTdNuuBd7cnjr+guTc8Xv3NQU3MxffLuDq9LAQBkCMEth3X3DKimvES3bGv0uhRPBGrLdfX6VQS3NJwYn9Zfv3hUP/F/27v/6KjrO9/jz3d+QwIEkglKQAgkE4wWwSIqUkZBwN7tzz3dVvtj3bXbntur11artt1zf/ac/WPV1equp63duuue9da6rj3bdrsOiApaFEW0yAAJIciPQEIIEEIgP+d9/8hwb5TfITPfmczrcU7OId+Z+c5rHElefD7f+Xyunkp1xbig44iISJKouKWpgbizaksrN9aGsnrV+0g4xMZdh+ns7gs6Slp77KXt4PCdm2uCjiIiIkmk4pam3tl1mPau3qydJj0pEg7RH3fW7WgPOkra2tF2jH95Zw9fvvYypk4cG3QcERFJIhW3NBWNtVCQm8ONtaGgowTq6ukTKSnM03TpWTyysoGi/FzuWlIddBQREUkyFbc05O5EYy3cUF3GuKL8oOMEKj83h4WzylhT34a7lgX5qM3NHfz7+/v5i0VVlJcUBh1HRESSTMUtDW3Zf5S9h09k/TTpSZHaEM1HTtB0sCvoKGnnwWg9pWPz+YvFM4OOIiIiKaDiloaisVZyDG7WPpMALD65LIh2UfiQN5vaWdvQxn+5cRbjs3xkVkQkW6i4paGVsRbmT5+kqa+EaZPGMjNUrOvchnB3HnxxG5eML+JPr58RdBwREUkRFbc0s6u9i20tnSy/QqNtQ0XCIdbvbKe7byDoKGlh9dYDbNx9hLuX1lCUn73LxYiIZBsVtzQTjbUA6Pq2j4iEQ3T3xXlr56GgowQuHnceXlnPjLKx/Mn8qUHHERGRFFJxSzPRWCt1l45n2iStxzXUtVVlFOTlaLoU+PUf9rGtpZN7l9eSn6u/wiIi2UQ/9dPIgc5uNu4+LJjvIwAAFttJREFUrNG20xhTkMu1VZOyvrj19sd5ZFUDdZeO51MfuzToOCIikmJJLW5mdouZ1ZtZo5l9/zS3P2pm7yW+GszsyJDb/trMNie+vjTkeJWZrU+c85dmVpDM15BKq7a04g4rrtT1bacTCYdoPHCM5iMngo4SmF9u2MPuQ8e5f0UtOTkWdBwREUmxpBU3M8sFngA+CdQBt5lZ3dD7uPs97j7X3ecCfwu8kHjsHwFXA3OBa4H7zGx84mF/DTzq7tXAYeDryXoNqRaNtTK9bCy1k7VJ+Omc3EVibZaOuh3v7efx1du5ZsbErN9RQ0QkWyVzxG0B0OjuTe7eCzwLfPYs978N+EXiz3XAWnfvd/cuYBNwi5kZsAR4PnG/p4HPJSV9ih3t7uONHQdZccUlDL5M+ahZoRKmTCjK2vXc/nHdB7R19vDALbP1/4iISJZKZnGrBPYM+X5v4tgpzGw6UAW8nDj0BwaL2lgzKwduAqYBZcARd+8/j3N+08w2mNmGtrb0/0X/yrYD9A04K7QMyBmZGZHaEL9vPEjfQDzoOCnVcbyPn7y6gyWzK7hmxqSg44iISEDS5cMJtwLPu/sAgLuvBH4HrGNwFO4N4IIW8HL3J919vrvPD4XSf1opGmshNK6QedMmBh0lrUXCITp7+nlvz5Fz33kU+enaHRzt7ue+5bVBRxERkQAls7g1MzhKdtLUxLHTuZX/P00KgLv/VeL6t2WAAQ1AO1BqZnnncc6M0d03wKv1bSyrm6wLzs9hYXU5uTmWVdOlBzq7+Yfff8BnrppC3ZTx536AiIiMWsksbm8DNYlPgRYwWM5+/dE7mdlsYCKDo2onj+WaWVniz3OAOcBKd3fgFeALibveDvxbEl9DSry+/SDHewe0DMh5GF+Uz9WXlWbVsiB/93IjfQNx7l0WDjqKiIgELGnFLXEd2l1AFNgKPOfuMTP7oZl9ZshdbwWeTZSyk/KB18xsC/Ak8NUh17V9D7jXzBoZvObt58l6DakSjbUwriiP62eWBR0lI0TCId5v7uDgsZ6goyTdnkPH+cVbu/niNdOYUV4cdBwREQlY3rnvMnzu/jsGr1Ubeux/fOT7/3Wax3Uz+MnS052zicFPrI4K/QNxXtraypLZFRTkpcslh+ktEq7g4ZUNvL79IJ+bd9rPpowaj65qIMeMu5fUBB1FRETSgJpCwN7+4DCHj/dpmvQCXDFlPGXFBaN+urS+pZNfvdfMny2cwSUTioKOIyIiaUDFLWDRWAsFeTlEwun/ydd0kZNjfKKmnNe2txGP+7kfkKEeXllPSUEe/zkyK+goIiKSJlTcAuTurNrSyuKacooLkzprPepEakMcPNbLlv1Hg46SFBt3H2bVlla+uXgmE4tHza5uIiJykVTcArS5+SjNR06wXNOkF+wTNYMjlKNxutTdeejFespLCrhjUVXQcUREJI2ouAUoGmshN8e4+XLtlnChyksKubJy/Khcz+31xoO80dTOnTdVayRWREQ+RMUtQNFYCwtmTGKSpsKGJRIOsXH3YY529wUdZcS4Ow++WE9l6Ri+fO1lQccREZE0o+IWkKa2Y2w/cEx7k16ESLiC/rizrrE96Cgj5j82t/B+cwf3LAtTmJcbdBwREUkzKm4BicZaAXR920WYd1kpJYV5o+Y6t/6BOA+vrKemooTPj/L16UREZHhU3AISjbUwZ+oEppSOCTpKxsrPzeGG6jLWNrTx4Y03MtMLG5tpauviu8trydWetSIichoqbgFo6ejmvT1HtOjuCIiEK2g+coIdbV1BR7ko3X0D/OilBq6aVqrpcxEROSMVtwCs2tICoF/QI2BxuBzI/GVBnlm/m30d3TywohYzjbaJiMjpqbgFIBprZWaomOqKcUFHyXhTJ45lVqg4o4vbsZ5+nnilkRuqy7ihujzoOCIiksZU3FKs43gfbza1a5p0BEXCFaxvaqe7byDoKMPy89d2cqirl/tXzA46ioiIpDkVtxRbva2V/riruI2gSG2Inv4463ceCjrKBTvU1cvPXmtixRWTmTutNOg4IiKS5lTcUiwaa+GS8UXMqZwQdJRR49qqSRTm5WTkLgo/frWR47393Le8NugoIiKSAVTcUuhE7wBrGtpYfsVkcrTcw4gpys/l2pllrGk4EHSUC7K/4wRPv7GLz8+bSs1kXe8oIiLnpuKWQmu3t9HdF9c0aRJEwiF2tHWx9/DxoKOct8dXb8fd+c7NNUFHERGRDKHilkLRWAsTxuSzoGpS0FFGnUg4BMDahoMBJzk/TW3HeG7DXr5y7XSmTRobdBwREckQKm4p0jcQZ/XWAyy9vIL8XP1nH2mzQsVUlo7JmOnSv1nVQGFeDnfeVB10FBERySBqECny1s5DdJzo0zRpkpgZi8Mh1jW20zcQDzrOWW1u7uDfN+3n64uqCI0rDDqOiIhkEBW3FInGWijKz2FxTSjoKKNWJByis6efd3cfCTrKWT0Urad0bD7fWDwz6CgiIpJhVNxSIB53VsZaiYRDjCnIDTrOqLWwuozcHEvr6dL1Te2saWjjW5FZjC/KDzqOiIhkGBW3FNjU3EHL0W5NkybZ+KJ8Pn7ZxLTd/srdeTBaz+Txhdy+cEbQcUREJAOpuKVANNZCXo6xdLY2lU+2SG2Izc1HOXisJ+gop3h52wHe2XWYu5fWUJSvkVcREblwKm4pEI21cN3MMiaM1dRYsp1cFuS17ek16haPOw9F65lRNpYvzp8WdBwREclQKm5J1nigk6a2LlZcodG2VKi7dDxlxQVpt/3VbzbtY1tLJ/csC2s5GBERGTb9BkmyaKwVgGV1ur4tFXJyBpcFeW37QeJxDzoOMLiG3yOrGrj80vF8es6UoOOIiEgGU3FLsmishbnTSrlkQlHQUbJGJByivauX2L6jQUcB4Jdv72FX+3HuXxHWHrUiInJRVNySaN+RE2za26FPk6bYoppygLRYFuRE7wCPr97O/OkTuam2Iug4IiKS4VTckmhlrAVA17elWHlJIR+rnJAWy4L847oPONDZwwO3zMZMo20iInJxVNyS6MVYCzUVJcwMlQQdJetEwiE27j7C0e6+wDJ0nOjjJ2t2cFNtiAVVkwLLISIio4eKW5Ic6urlrZ2HNE0akEhtiIG4s67xYGAZnly7g44Tfdy3ojawDCIiMrqouCXJS1tbiTsqbgGZO62UcYV5gU2XHujs5qnXP+DTV03hiikTAskgIiKjj4pbkqyMtVBZOoYrK8cHHSUr5efmcEN1OWsbDuKe+mVBnni5kd6BOPcuC6f8uUVEZPRScUuCrp5+1m4/yLK6ybogPUCR2hDNR06wo+1YSp93z6Hj/J+3dvPF+dOoKi9O6XOLiMjopuKWBGsa2ujtj2uaNGCLE9tfvZriXRQefamBHDO+vbQmpc8rIiKjn4pbEkRjLUwcm881MyYGHSWrVZaOobqiJKXXuTW0dvKrd5u5feEMLbosIiIjTsVthPX2x3l52wFuvnwyedqTMnCRcIi3dh6iu28gJc/3cLSekoI8vhWZlZLnExGR7KJmMcLeaGqns7tf06RpIhIO0dMf582m9qQ/17u7D7NySyvfWDyTicUFSX8+ERHJPipuIywaa2FsQe7/23ZJgrWgahKFeTkpmS59KFpPWXEBdyyqSvpziYhIdlJxG0HxuLNqSys31oYoys8NOo4ARfm5XDezjLVJLm6vbz/Iuh3t3HlTNSWFeUl9LhERyV4qbiPo3T2Haevs0TRpmomEQ+xo62LPoeNJOb+782B0G5WlY/jKdZcl5TlERERAxW1ERWOt5OcaN82uCDqKDBGpHVwWZO325Iy6vbi5hU17O/j2zTUU5mmkVUREkkfFbYS4O9FYC9fPKmd8UX7QcWSImeXFVJaOYU0S1nPrH4jz8Mp6qitK+ON5lSN+fhERkaFU3EZIfWsnu9qPs+KKyUFHkY8wMyK1IdbtaKdvID6i537h3WZ2tHVx3/Kwln8REZGk02+aERLd3IoZLKtTcUtHkXCIYz39bNx1eMTO2dM/wGMvbeeqqRN0XaOIiKSEitsIicZauPqyiVSM02r56WjhrDLycmxElwV55s3dNB85wf0rZmtPWhERSQkVtxGw59Bxtuw/qmnSNDauKJ+rp08csQ8oHOvp54lXGlk4q0xr9omISMqouI2AaKwFQNNlaS4SDrG5+ShtnT0Xfa6nXt9Je1cv96+oHYFkIiIi50fFbQTkmLE4HGJ6WXHQUeQsIuHBZUFeu8hRt8NdvfxsbRPL6yYz77KJIxFNRETkvKi4jYA7FlXxT3csCDqGnEPdpeMpLym46OvcfrxmB8d6+7lPo20iIpJiKm6SNXJyjMU1IV7bfpB43Id1jpaObp5e9wGfn1dJePK4EU4oIiJydipuklUitSEOdfWyeV/HsB7/2OrtxN255+bwCCcTERE5NxU3ySqLqssxY1i7KDS1HeO5DXv48oLLmDZpbBLSiYiInJ2Km2SVspJCPlY5YVjLgjyyqoGC3BzuWlKThGQiIiLnpuImWScSDrFx9xE6TvSd92M2N3fw2037+fqiKkLjCpOYTkRE5MxU3CTrRMIhBuLOusaD5/2Yh1fWM2FMPt9YPDOJyURERM5OxU2yztxppYwryjvvZUHe2nmIV+vb+NaNs5gwJj/J6URERM5MxU2yTl5uDouqy1nb0Ib72ZcFcXcefHEbFeMKuf36GakJKCIicgYqbpKVIuEQ+zq6aTxw7Kz3e6X+ABt2HebupTWMKchNUToREZHTU3GTrLQ4sf3V2aZL43HnoWgD08vG8qVrpqUqmoiIyBmpuElWmlI6hpqKkrMWt99s2sfW/Ue5d1mY/Fz9VRERkeDpt5FkrUg4xPqdhzjRO3DKbX0DcR5Z1cDsS8bx6TlTAkgnIiJyKhU3yVqR2hC9/XHe3Nl+ym3PbdjDrvbj3L+ilpwcCyCdiIjIqVTcJGtdM2MSRfk5p2x/daJ3gMdXb+fj0yeyZHZFQOlEREROpeImWasoP5frZpadsv3V0298QOvRHh5YUYuZRttERCR9qLhJVouEQzS1dbHn0HEAOk708eNXdxAJh7h2ZlnA6URERD5MxU2yWuQjy4L8bG0THSf6uH9FbZCxRERETiupxc3MbjGzejNrNLPvn+b2R83svcRXg5kdGXLbg2YWM7OtZva4JeaszOw2M3vfzDaZ2YtmVp7M1yCjW1V5MVMnjmFtQxttnT089fudfGrOpVxZOSHoaCIiIqdIWnEzs1zgCeCTQB1wm5nVDb2Pu9/j7nPdfS7wt8ALiccuBG4A5gBXAtcAETPLAx4DbnL3OcAm4K5kvQYZ/cyMSDjEuh3t/OilBnr643x3uUbbREQkPSVzxG0B0OjuTe7eCzwLfPYs978N+EXizw4UAQVAIZAPtAKW+CpOjMCNB/YlJ75ki0g4xLGefp5Zv5svzp9KVXlx0JFEREROK5nFrRLYM+T7vYljpzCz6UAV8DKAu78BvALsT3xF3X2ru/cB3wLeZ7Cw1QE/P8M5v2lmG8xsQ1vbmVfHF7l+Vhl5OUZBXg53L60JOo6IiMgZpcuHE24Fnnf3AQAzqwYuB6YyWPaWmNknzCyfweI2D5jC4FTpD053Qnd/0t3nu/v8UCiUitcgGWpcUT53LKrie7fM5tIJY4KOIyIickZ5STx3MzB0Z+6piWOncytw55DvPw+86e7HAMzsP4DrgW4Ad9+ROP4ccMqHHkQu1F/+p8uDjiAiInJOyRxxexuoMbMqMytgsJz9+qN3MrPZwETgjSGHd5P4MEJilC0CbGWw+NWZ2ckhtGWJ4yIiIiKjXtJG3Ny938zuAqJALvCUu8fM7IfABnc/WeJuBZ51dx/y8OeBJQxey+bAi+7+GwAz+9/AWjPrA3YBf5as1yAiIiKSTuzDfWl0mj9/vm/YsCHoGCIiIiLnZGbvuPv8092WLh9OEBEREZFzUHETERERyRAqbiIiIiIZQsVNREREJEOouImIiIhkCBU3ERERkQyh4iYiIiKSIVTcRERERDKEipuIiIhIhlBxExEREckQKm4iIiIiGULFTURERCRDqLiJiIiIZAgVNxEREZEMoeImIiIikiFU3EREREQyhIqbiIiISIZQcRMRERHJECpuIiIiIhlCxU1EREQkQ5i7B50h6cysDdiV5KcpBw4m+TkkufQeZj69h5lN71/m03s4Mqa7e+h0N2RFcUsFM9vg7vODziHDp/cw8+k9zGx6/zKf3sPk01SpiIiISIZQcRMRERHJECpuI+fJoAPIRdN7mPn0HmY2vX+ZT+9hkukaNxEREZEMoRE3ERERkQyh4iYiIiKSIVTcRoCZ3WJm9WbWaGbfDzqPnD8zm2Zmr5jZFjOLmdm3g84kw2NmuWb2rpn9NugscuHMrNTMnjezbWa21cyuDzqTXBgzuyfxc3Szmf3CzIqCzjQaqbhdJDPLBZ4APgnUAbeZWV2wqeQC9APfdfc64DrgTr1/GevbwNagQ8iwPQa86O6zgavQe5lRzKwSuBuY7+5XArnArcGmGp1U3C7eAqDR3ZvcvRd4FvhswJnkPLn7fnffmPhzJ4O/LCqDTSUXysymAn8E/H3QWeTCmdkEYDHwcwB373X3I8GmkmHIA8aYWR4wFtgXcJ5RScXt4lUCe4Z8vxf94s9IZjYDmAesDzaJDMOPgAeAeNBBZFiqgDbgHxLT3X9vZsVBh5Lz5+7NwMPAbmA/0OHuK4NNNTqpuIkAZlYC/CvwHXc/GnQeOX9m9inggLu/E3QWGbY84Grgx+4+D+gCdL1wBjGziQzONlUBU4BiM/tqsKlGJxW3i9cMTBvy/dTEMckQZpbPYGl7xt1fCDqPXLAbgM+Y2QcMXqqwxMz+OdhIcoH2Anvd/eRo9/MMFjnJHDcDO929zd37gBeAhQFnGpVU3C7e20CNmVWZWQGDF2P+OuBMcp7MzBi8rmaruz8SdB65cO7+A3ef6u4zGPz797K761/6GcTdW4A9ZlabOLQU2BJgJLlwu4HrzGxs4ufqUvQBk6TICzpApnP3fjO7C4gy+Cmap9w9FnAsOX83AF8D3jez9xLH/tLdfxdgJpFs9F+BZxL/AG4C/jzgPHIB3H29mT0PbGTw0/rvou2vkkJbXomIiIhkCE2VioiIiGQIFTcRERGRDKHiJiIiIpIhVNxEREREMoSKm4iIiEiGUHETEUkSM7vRzH4bdA4RGT1U3EREREQyhIqbiGQ9M/uqmb1lZu+Z2U/NLNfMjpnZo2YWM7PVZhZK3Heumb1pZpvM7FeJPRoxs2oze8nM/mBmG81sVuL0JWb2vJltM7NnEqvKi4gMi4qbiGQ1M7sc+BJwg7vPBQaArwDFwAZ3vwJYA/zPxEP+Cfieu88B3h9y/BngCXe/isE9Gvcnjs8DvgPUATMZ3K1DRGRYtOWViGS7pcDHgbcTg2FjgANAHPhl4j7/DLxgZhOAUndfkzj+NPAvZjYOqHT3XwG4ezdA4nxvufvexPfvATOA15P/skRkNFJxE5FsZ8DT7v6DDx00++8fud9w9wfsGfLnAfRzV0QugqZKRSTbrQa+YGYVAGY2ycymM/jz8QuJ+3wZeN3dO4DDZvaJxPGvAWvcvRPYa2afS5yj0MzGpvRViEhW0L/8RCSrufsWM/tvwEozywH6gDuBLmBB4rYDDF4HB3A78JNEMWsC/jxx/GvAT83sh4lz/EkKX4aIZAlzH+7ov4jI6GVmx9y9JOgcIiJDaapUREREJENoxE1EREQkQ2jETURERCRDqLiJiIiIZAgVNxEREZEMoeImIiIikiFU3EREREQyxP8Fo06QXi7+7WYAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 10))\n",
        "plt.plot(history.history['loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "9dIpZk6AK8Zl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 621
        },
        "outputId": "9b7d4553-6ee0-4daa-a9b7-249d3b722b7a"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAJcCAYAAAC8DwN/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXjU12Hv/88ZabQvIwlpBEhjVgMSiybBjncwtiUvwXWaJq3bNIlvGjdN7aa9bZ6mvff++vvd3l9v7tP2d9vE2dzETdPFvW1TN5bj2HjDODaODQaxgwED2jfQvs7M+f2hgWAjQMvMfL8z8349jx7EfGf5YHjg43O+5xxjrRUAAADcweN0AAAAAPwc5QwAAMBFKGcAAAAuQjkDAABwEcoZAACAi1DOAAAAXIRyBiCtGWO+b4z5HzN87iljzJ3zfR8AuBLKGQAAgItQzgAAAFyEcgbA9aLTiV82xuwzxgwbY75njPEbY35ijBk0xrxojCm56Pn3G2MOGmP6jDHbjTFrLroWNMa8E33d/5GU84HP+qgxZm/0tW8YY9bPMfPnjTHHjTFnjTFPG2MWRR83xpj/bYzpMsYMGGP2G2PWRq/da4w5FM3Waoz5gzn9BwOQ1ChnAJLFxyXdJelaSVsl/UTSH0sq19TfZb8jScaYayU9Kel3o9eeldRojMkyxmRJ+g9Jfy+pVNK/Rt9X0dcGJT0h6TcllUn6jqSnjTHZswlqjNki6X9K+qSkhZJOS/rn6OV6SbdFfx3F0ef0Rq99T9JvWmsLJa2V9PJsPhdAaqCcAUgWX7fWdlprWyW9Juln1to91toxSU9JCkaf98uSfmytfcFaOynpLyTlSrpJ0g2SvJL+ylo7aa39N0lvX/QZD0v6jrX2Z9basLX27ySNR183G78m6Qlr7TvW2nFJfyTpRmPMEkmTkgolrZZkrLWHrbXt0ddNSqoxxhRZa89Za9+Z5ecCSAGUMwDJovOi70en+XlB9PtFmhqpkiRZayOSmiUtjl5rtdbai157+qLvr5H0+9EpzT5jTJ+k6ujrZuODGYY0NTq22Fr7sqTHJH1DUpcx5nFjTFH0qR+XdK+k08aYV40xN87ycwGkAMoZgFTTpqmSJWnqHi9NFaxWSe2SFkcfOy9w0ffNkv5fa63voq88a+2T88yQr6lp0lZJstZ+zVr7YUk1mpre/HL08bettb8gqUJT06//MsvPBZACKGcAUs2/SLrPGHOHMcYr6fc1NTX5hqSdkkKSfscY4zXG/KKk6y967d9I+oIx5iPRG/fzjTH3GWMKZ5nhSUkPGWPqover/ZmmpmFPGWOui76/V9KwpDFJkeg9cb9mjCmOTscOSIrM478DgCRFOQOQUqy1RyV9StLXJfVoavHAVmvthLV2QtIvSvqspLOauj/t3y967S5Jn9fUtOM5Scejz51thhcl/TdJP9TUaN1ySb8SvVykqRJ4TlNTn72S/jx67dclnTLGDEj6gqbuXQOQZsz7b70AAACAkxg5AwAAcBHKGQAAgItQzgAAAFyEcgYAAOAimU4HiKUFCxbYJUuWOB0DAADgqnbv3t1jrS3/4OMpVc6WLFmiXbt2OR0DAADgqowxp6d7nGlNAAAAF6GcAQAAuAjlDAAAwEXids+ZMeYJSR+V1GWtXTvN9RJJT2jqWJMxSf/JWnsgeu1uSX8tKUPSd621X51rjsnJSbW0tGhsbGyub5EUcnJyVFVVJa/X63QUAAAwD/FcEPB9TZ1P94PLXP9jSXuttR8zxqyW9A1JdxhjMqLf3yWpRdLbxpinrbWH5hKipaVFhYWFWrJkiYwxc3kL17PWqre3Vy0tLVq6dKnTcQAAwDzEbVrTWrtDUwcLX06NpJejzz0iaYkxxi/peknHrbUno4cU/7OkX5hrjrGxMZWVlaVsMZMkY4zKyspSfnQQAIB04OQ9Z02SflGSjDHXS7pGUpWkxZKaL3peS/SxaRljHjbG7DLG7Oru7r7cc2KV2bXS4dcIAEA6cLKcfVWSzxizV9KjkvZICs/2Tay1j1trN1prN5aXX7KPGwAAQFJxrJxZawestQ9Za+skfVpSuaSTklolVV/01KroY0mpr69P3/zmN2f9unvvvVd9fX1xSAQAANzMsXJmjPEZY7KiP/0NSTustQOS3pa00hizNHr9VyQ97VTO+bpcOQuFQld83bPPPiufzxevWAAAwKXiuZXGk5I2S1pgjGmR9CeSvJJkrf22pDWS/s4YYyUdlPS56LWQMeYRSc9raiuNJ6y1B+OVM96+8pWv6MSJE6qrq5PX61VOTo5KSkp05MgRHTt2TA888ICam5s1NjamL33pS3r44Ycl/fwoqqGhId1zzz265ZZb9MYbb2jx4sX60Y9+pNzcXId/ZQAAIB7iVs6stQ9e5fpOSdde5tqzkp6Ndab/p/GgDrUNxPQ9axYV6U+21l72+le/+lUdOHBAe/fu1fbt23XffffpwIEDF7a8eOKJJ1RaWqrR0VFdd911+vjHP66ysrL3vce7776rJ598Un/zN3+jT37yk/rhD3+oT33qUzH9dQAAAHdIqYPPk8H111//vr3Ivva1r+mpp56SJDU3N+vdd9+9pJwtXbpUdXV1kqQPf/jDOnXqVMLyAgCAxEqrcnalEa5Eyc/Pv/D99u3b9eKLL2rnzp3Ky8vT5s2bp92rLDs7+8L3GRkZGh0dTUhWAACQeJytGWeFhYUaHByc9lp/f79KSkqUl5enI0eO6M0330xwOgAA4DZpNXLmhLKyMt18881au3atcnNz5ff7L1y7++679e1vf1tr1qzRqlWrdMMNNziYFAAAuIGx1jqdIWY2btxod+3a9b7HDh8+rDVr1jiUKLHS6dcKAECyM8bsttZu/ODjTGsCAAC4COUMAADARdKinKXS1O3lpMOvEQCAdJDy5SwnJ0e9vb0pXV6stert7VVOTo7TUQAAwDyl/GrNqqoqtbS0qLu72+kocZWTk6OqqiqnYwAAgHlK+XLm9XrftyP/fHz+B7tUWZSjP31gbUzeDwAA4INSflozlsIRqzdP9jodAwAApDDK2SwEq316t2tI/aOTTkcBAAApinI2C8FAiSRpX0ufw0kAAECqopzNwvrqYhkj7TlDOQMAAPFBOZuFohyvVpQXaG8z5QwAAMQH5WyWggGf9pw5l9L7pgEAAOdQzmYpGCjRuZFJne4dcToKAABIQZSzWQoGfJKkPc3nHE4CAABSEeVsllZWFCo/K0N7WRQAAADigHI2Sxkeo/VVPu1hUQAAAIgDytkcBAM+HWob0Nhk2OkoAAAgxVDO5iAYKFEoYnWgtd/pKAAAIMVQzuagrjq6KID7zgAAQIxRzuagvDBbVSW5bEYLAABijnI2R8FAifacYTsNAAAQW5SzOQpW+9TWP6aO/jGnowAAgBRCOZuj85vR7mUzWgAAEEOUszmqWVSkrAwP+50BAICYopzNUXZmhmoWFbFiEwAAxBTlbB6CAZ/2tfQpFI44HQUAAKQIytk8BAMlGpuM6EjHoNNRAABAiqCczUOw+vyiAKY2AQBAbFDO5qGqJFcLCrK47wwAAMQM5WwejDGqqy7RHrbTAAAAMUI5m6dgwKeT3cPqH5l0OgoAAEgBlLN5unDfWQtTmwAAYP4oZ/O0vtonY8Q5mwAAICYoZ/NUkJ2pVf5CFgUAAICYoJzFQDDg097mPkUi1ukoAAAgyVHOYqCu2qf+0Um91zvsdBQAAJDkKGcxEAyUSJL2MrUJAADmiXIWAyvKC1SYncl+ZwAAYN4oZzHg8RhtqPaxKAAAAMwb5SxGggGfjnQManQi7HQUAACQxChnMVJX7VM4YrW/td/pKAAAIIlRzmKkLnpSAJvRAgCA+aCcxUhZQbauKcvjvjMAADAvlLMYClb7WLEJAADmhXIWQ3XVPnUOjKu9f9TpKAAAIElRzmLo/Ga0TG0CAIC5opzF0JqFRcrK9LAoAAAAzBnlLIayMj1at7iYkTMAADBnlLMYq6v2aX9rvybDEaejAACAJEQ5i7FgwKfxUERH2gedjgIAAJIQ5SzGLiwKYEsNAAAwB5SzGFtUnKOKwmzuOwMAAHNCOYsxY4zqqn2s2AQAAHNCOYuDYKBEp3pHdG54wukoAAAgyVDO4iAYmDoEfW8zU5sAAGB2KGdxsL6qWB4jpjYBAMCsUc7iIC8rU6sri7SHkTMAADBLlLM4qQv4tLe5T5GIdToKAABIIpSzOAlW+zQ4FtLJniGnowAAgCRCOYuT85vRvsN+ZwAAYBYoZ3GybEG+inIy2YwWAADMCuUsTjweow3VPrbTAAAAs0I5i6NgoERHOwY0PB5yOgoAAEgSlLM4CgZ8ilhpX0u/01EAAECSoJzFUV3V1EkBe5rZjBYAAMwM5SyOSvKztHRBvvayKAAAAMwQ5SzOgtU+7Wnuk7VsRgsAAK6OchZnwYBP3YPjau0bdToKAABIApSzODu/GS37nQEAgJmgnMXZqspCZWd62O8MAADMCOUszrwZHq2vKtaeM6zYBAAAV0c5S4BgoEQH2gY0Hgo7HQUAALgc5SwBgtU+TYQiOtw+6HQUAADgcpSzBPj5ogCmNgEAwJVRzhKgsjhHlUU5LAoAAABXRTlLkGDAx3YaAADgqihnCRIM+HTm7Ih6hsadjgIAAFyMcpYg5+8745xNAABwJZSzBFm7qFgZHsN9ZwAA4IooZwmSm5WhNQsLtaeZFZsAAODyKGcJFKwuUVNzv8IR63QUAADgUpSzBAoGfBoaD+l415DTUQAAgEtRzhKortonSdrL1CYAALgMylkCLV2Qr+JcL/udAQCAy6KcJZAxhs1oAQDAFVHOEixYXaJjXYMaHJt0OgoAAHAhylmC1QV8slba39LvdBQAAOBClLMEq6uaWhSwh81oAQDANChnCVac59Xy8nztOcOKTQAAcCnKmQOCgRLtOdMna9mMFgAAvB/lzAHBgE+9wxNqPjvqdBQAAOAylDMHnN+MlnM2AQDAB1HOHLDKX6hcbwb7nQEAgEtQzhyQmeHR+qpiVmwCAIBLxK2cGWOeMMZ0GWMOXOZ6sTGm0RjTZIw5aIx56KJrYWPM3ujX0/HK6KRgoESH2vo1Nhl2OgoAAHCReI6cfV/S3Ve4/tuSDllrN0jaLOkvjTFZ0Wuj1tq66Nf9cczomLpqnybDVofaB5yOAgAAXCRu5cxau0PS2Ss9RVKhMcZIKog+NxSvPG4TDEQXBXDfGQAAuIiT95w9JmmNpDZJ+yV9yVobiV7LMcbsMsa8aYx54EpvYox5OPrcXd3d3XGOHDv+ohwt9uWyGS0AAHgfJ8tZg6S9khZJqpP0mDGmKHrtGmvtRkm/KumvjDHLL/cm1trHrbUbrbUby8vL4x46luoCPkbOAADA+zhZzh6S9O92ynFJ70laLUnW2tbojyclbZcUdCpkPAWrfWrtG1XX4JjTUQAAgEs4Wc7OSLpDkowxfkmrJJ00xpQYY7Kjjy+QdLOkQ46ljKPz953tZfQsISIRq+/99D119FOGAQDuFc+tNJ6UtFPSKmNMizHmc8aYLxhjvhB9yp9KuskYs1/SS5L+0Frbo6n70HYZY5okvSLpq9balCxntYuK5c0w7HeWILtOn9OfPnNIf/bsYaejAABwWZnxemNr7YNXud4mqX6ax9+QtC5eudwkx5uhmoVFLApIkMamNknSM/va9Lt3rtSy8gKHEwEAcClOCHBYXbVP+1r6FY5Yp6OktFA4omf3t+um5WXKyvToW9tPOB0JAIBpUc4cFgyUaGQirGOdg05HSWlvnOhV7/CEPnPTEj14fUBP7WlV89kRp2MBAHAJypnD2Iw2MRqb2lSYnalN15br4duWyWOMvv0qo2cAAPehnDksUJqn0vws7juLo/FQWM8d7FDD2krleDO0sDhXv7SxSv+6q4WVmwAA16GcOcwYo2C1T3tZsRk3rx7t1uBYSFs3LLrw2G9tWq6wtXp8x0kHkwEAcCnKmQvUVfv0bteQ+kcnnY6Skhr3tas0P0s3LS+78Fh1aZ4eqFusf3rrtHqGxh1MBwDA+1HOXCAYKJEk7Wth9CzWRiZCevFQp+5ZWylvxvv/uH/x9uUaD0X0vZ++51A6AAAuRTlzgfXVxTKGRQHx8NLhLo1OhnX/RVOa5y0vL9B96xbqB2+cUt/IhAPpAAC4FOXMBYpyvFpZUcCigDh4uqlN/qJsXbekdNrrj2xZoeGJsL7/xqnEBgMA4DIoZy5RF10UYC2b0cZK/+ikXj3arY+uXySPx0z7nNWVRaqv8etvXz+lwTHu+QMAOI9y5hLBQInOjUzqdC8bo8bKtoMdmghH3rdKczqPbFmh/tFJ/cObZxKUDACAy6OcucSFzWibmdqMlcZ97QqU5mlDVfEVn7e+yqdN15bru6+d1MhEKEHpAACYHuXMJVZWFCo/K4NFATHSOzSu14/3aOuGhTJm+inNiz26ZYV6hyf05FvNCUgHAMDlUc5cIsNjtL6KzWhj5dkDHQpH7FWnNM/buKRUNywr1eM7TmhsMhzndAAAXB7lzEWCAZ8OtQ1QDmKgsalNKysKtMpfOOPXPLplpToHxvVvu1vimAwAgCujnLlIMFCiUMTqQGu/01GSWnv/qN4+dVb3b1g0oynN825aXqZgwKdvbT+hyXAkjgkBALg8ypmL1FVHFwVw39m8/Hhfu6yVPjrDKc3zjDF6dMsKtfaN6j/2tMYpHQAAV0Y5c5HywmxVleRy39k8NTa1ad3iYi1dkD/r196+qkK1i4r0ze0nFI6w5xwAIPEoZy4TDJRwUsA8nO4dVlNLv7ZuWDin158fPXuvZ1g/3t8e43QAAFwd5cxlgtU+tfWPqaN/zOkoSamxqU2S9NH1s5vSvFh9TaVWVhToGy8fV4TRMwBAglHOXOb8ZrR72Yx2Thqb2nXdkhIt8uXO+T08HqNHtqzQ0c5BbTvUGcN0AABcHeXMZWoWFSkrw6M93Hc2a0c7BnW0c3DGe5tdyX3rFmpJWZ4ee+VdzjsFACQU5cxlsjMzVLOoiBWbc/DMvjZ5jHTP2rndb3axzAyPvrh5hQ60Dmj7se4YpAMAYGYoZy4UDPi0r6VPIfbamjFrrZ5uatPNKxaovDA7Ju/5QHCxFvty9fWXGD0DACQO5cyFgoESjU1GdKRj0OkoSWN/a79O945o6zwWAnxQVqZHX9i0TO+c6dPOk70xe18AAK6EcuZCwfOb0XLf2Yw1NrXJm2HUUFsZ0/f9xMZqVRRm67GXj8f0fQEAuBzKmQtVleRqQUGW9nLf2YxEIlbP7GvXpmsrVJznjel753gz9PBty/TGiV7tPn02pu8NAMB0KGcuZIxRXXWJ9rCdxozsOn1O7f1jc9549mp+9SMBleZnMXoGAEgIyplLBQM+neweVt/IhNNRXK+xqU05Xo/uXOOPy/vnZWXqc7cs1StHuzmUHgAQd5Qzl/r5ZrRMbV5JKBzRs/vbdccav/KzM+P2OZ++8RoV5WTq6y+/G7fPAABAopy51voqn4yhnF3NzpO96h2e0P0x2Hj2SgpzvPrszUv1/MFOHWUVLQAgjihnLlWQnalV/kI2o72Kp/e2qTA7U5uuLY/7Zz100xLlZ2XoG69w7xkAIH4oZy4WDPi0t7mPw7cvYzwU1nMHO1RfW6kcb0bcP68kP0ufuvEaPbOvTSe7h+L+eQCA9EQ5c7FgdYn6Ryf1Xu+w01FcacexHg2OheK2SnM6n791mbIyPfrW9hMJ+0wAQHqhnLlY3flFAUxtTuvppjaV5mfp5hULEvaZCwqy9eD1AT21p1XNZ0cS9rkAgPRBOXOxFeUFKszOZL+zaYxMhPTioU7ds7ZS3ozE/jF++LZl8hij7+xg9AwAEHuUMxfzeIw2VPtYFDCNlw53aXQyrK1xXqU5nYXFufqljVX6l7db1DkwlvDPBwCkNsqZywUDPh3pGNToRNjpKK7S2NQmf1G2rltS6sjn/9am5Qpbq8d3nHTk8wEAqYty5nJ11T6FI1b72Zn+gv7RSW0/2q2Prl+kDI9xJEN1aZ4eqFusf/zZafUMjTuSAQCQmihnLldXPbUoYM8Z7js7b9vBDk2EI45MaV7si7cv13goou/99D1HcwAAUgvlzOXKCrJ1TVke951dpHFfu6pLc7WhqtjRHMvLC3TfuoX6wRunOAMVABAzlLMkEKz26Z0z52Qtm9H2Do3r9eM92rp+kYxxZkrzYo9sWaHhibC+/8Ypp6MAAFIE5SwJBAMl6hocV3s/KwOfPdChcMTq/jpnpzTPW11ZpPoav/729VMaHJt0Og4AIAVQzpLA+fvOOAR9apXmyooCrfIXOh3lgke2rFD/6KT+4c0zTkcBAKQAylkSWLOwSFmZnrRfFNDeP6q3T53V1g3umNI8b32VT5uuLdd3XzvJlicAgHmjnCWBrEyP1i0uTvtFAT/e1y5r5fgqzek8umWFeocn9ORbjJ4BAOaHcpYkgtU+7W/t12Q44nQUxzQ2tWnd4mItXZDvdJRLbFxSqhuWleo7O05oPMToGQBg7ihnSaIu4NN4KKIj7YNOR3HE6d5hNbX0a+uGhU5HuaxHt6xU58C4/nVXi9NRAABJjHKWJIKBEklK20PQn9nXLkm6b737pjTPu2l5mYIBn761/URaj3ACAOaHcpYkFhXnqKIwO23vO3t6b5uuW1Kixb5cp6NcljFGj25Zoda+Uf3Hnlan4wAAkhTlLEkYYxQM+NJyxebRjkEd7Rx05UKAD7p9VYVqFxXpm9tPKBxh02AAwOxRzpJIXXWJTvWO6Nxweh0V9My+NnmMdM9a995vdt750bP3eob14/3tTscBACQhylkSCQbSbzNaa60am9p00/IFKi/MdjrOjNTXVGplRYG+8fJxRRg9AwDMEuUsiayvKpbHKK2mNve39utU74juT4IpzfM8HqNHtqzQ0c5BvXC40+k4AIAkQzlLInlZmVpdWaQ9aTRy1tjUJm+GUUNtpdNRZuW+dQu1pCxPj718nAPrAQCzQjlLMnUBn/Y296XFdFkkYvXMvnZturZcxXlep+PMSmaGR1/cvEL7W/v16rFup+MAAJII5SzJBKt9GhwL6WTPkNNR4m73mXNq7x9LilWa03kguFiLfbn6OqNnAIBZoJwlmfOb0b6TBvudPb23TTlej+5c43c6ypxkZXr0hU3LtPv0Oe082et0HABAkqCcJZllC/JVlJOZ8pvRhsIRPbu/XXes8Ss/O9PpOHP2iY3VqijM1mMvH3c6CgAgSVDOkozHY1QXKEn57TR2nuxV7/CEtrr4uKaZyPFm6OHblumNE73affqs03EAAEmAcpaE6qp9OtoxoOHxkNNR4qaxqU2F2ZnavKrc6Sjz9qsfCag0P4vRMwDAjFDOklAw4FPESvta+p2OEhfjobB+cqBD9bWVyvFmOB1n3vKyMvW5W5bqlaPdOtCamr9nAIDYoZwlobqqqZMC9jSn5ma0O471aHAspK0b3H9c00x9+sZrVJSTyegZAOCqKGdJqCQ/S8sW5KfsooDGpjaV5Hl184oFTkeJmcIcrz5781I9d7BDxzoHnY4DAHAxylmSqque2ow21fbPGpkI6YVDnbpn3UJ5M1Lrj+dDNy1RflaGvvEKo2cAgMtLrX/90kgw4FP34Lha+0adjhJTLx3u0uhkOKnO0pypkvwsferGa9TY1Kb3eoadjgMAcCnKWZI6vxltqk1tNja1yV+UreuWlDodJS4+f+syZWV69E1GzwAAl0E5S1KrKguV4/WkVDkbGJvU9qPdum/dImV4jNNx4mJBQbYevD6gp/a0qvnsiNNxAAAuRDlLUt4Mj9YtLtbeFFqx+fyBDk2EI7q/LvWmNC/28G3L5DFG39lxwukoAAAXopwlsWCgRAfaBjQeCjsdJSYa97WrujRXG6qKnY4SVwuLc/VLG6v0L2+3qHNgzOk4AACXoZwlsWC1TxOhiA63J//WDL1D43r9eI+2rl8kY1JzSvNiv7VpucLW6vEdJ52OAgBwGcpZEvv5ooDkn9r8yYEOhSNWW1NwleZ0qkvz9EDdYv3jz06rd2jc6TgAABehnCWxyuIcVRblpMQh6E83tWllRYFWVxY6HSVhvnj7co2HIvreT99zOgoAwEUoZ0kuGPAl/YrN9v5RvX3qrLZuSI8pzfOWlxfovnUL9YOdp9U/Mul0HACAS1DOklww4NOZsyPqSeKpsR/va5e10kfXp85ZmjP1yJYVGhoP6W/fYPQMADCFcpbkzt93tjeJR88a97Vr7eIiLSsvcDpKwq2uLFJ9jV9/+/opDY4xegYAoJwlvbWLipXpMUl739np3mE1Nfel5HFNM/XIlhXqH53UP7x5xukoAAAXoJwludysDK1eWKg9SboZ7TP72iVJ961P33K2vsqnTdeW67uvndToRGrsWQcAmDvKWQoIVpeoqblf4Yh1OsqsNTa1aeM1JVrsy3U6iqMe3bJCvcMTevItRs8AIN1RzlJAMODT0HhIx7uGnI4yK8c6B3WkYzBt9ja7ko1LSnXDslJ9Z8eJlDnxAQAwN5SzFJCsm9E2NrXJY6R716XfKs3pPLplpToHxvVvu1ucjgIAcBDlLAUsKcuTL8+bVIsCrLVqbGrTTcsXqLww2+k4rnDT8jIFAz59a/sJTYYjTscBADiEcpYCjDGqq06uzWgPtA7oVO+Itm5g1Ow8Y4we3bJCLedG9aO9bU7HAQA4hHKWIoLVJTrWNZg0e2U93dQqb4bR3bWUs4vdvqpCtYuK9M1XjiflAg8AwPxRzlJEMOCTtdK+ln6no1xVJGL1zL52bbq2XMV5XqfjuMr50bOTPcP68f52p+MAABxAOUsRG6p9kpQU953tPnNO7f1jrNK8jPqaSq2sKNA3Xj6uCKNnAJB2KGcpojjXq+Xl+UmxYrOxqU05Xo/uXON3OooreTxGj2xZoaOdg3rhcKfTcQAACUY5SyHBQIn2nOmTte4dbQmFI3p2f7vuWONXfnam03Fc6751C7WkLE+PvXzc1b+fAIDYo5ylkGDAp97hCTWfHXU6ymXtPNmrnqEJbU3j45pmIjPDoy9uXqH9rf169Vi303EAAAlEOUshddH7ztx8zmZjU5sKsjO1ecCbzLgAACAASURBVFW501Fc74HgYi325errjJ4BQFqhnKWQVf5C5XozXLvf2XgorOcOdKi+1q8cb4bTcVwvK9OjL2xapt2nz+nNk2edjgMASBDKWQrJzPBofVWx9rh0xeaOYz0aGAvpflZpztgnNlarojBbj73yrtNRAAAJQjlLMcFAiQ619Wts0n2HZzc2takkz6ubVyxwOkrSyPFm6OHblun1473afdq909UAgNihnKWYYMCnybDVofYBp6O8z8hESC8c6tQ96xbKm8Efu9n41Y8EVJqfpcdeZvQMANIB/0qmmOD5RQEuu+/s5SNdGp0Ms0pzDvKyMvW5W5bqlaPdOtDq/hMgAADzQzlLMRVFOVrsy3XdZrRP722Tvyhb1y8tdTpKUvr0jdeoKCdTj7183OkoAIA4o5yloLqAz1UjZwNjk9p+tFv3rVukDI9xOk5SKszx6rM3L9VzBzt0rHPQ6TgAgDiinKWgYLVPrX2j6hocczqKJGnbwU5NhCPaumGh01GS2kM3LVF+Voa+8QqjZwCQyuJazowxTxhjuowxBy5zvdgY02iMaTLGHDTGPHTRtc8YY96Nfn0mnjlTTTAQPQTdJaNnTze1qbo098ImuZibkvwsferGa9TY1Kb3eoadjgMAiJN4j5x9X9LdV7j+25IOWWs3SNos6S+NMVnGmFJJfyLpI5Kul/QnxpiSOGdNGbWLiuXNMK7Y76x3aFyvH+/R1vWLZAxTmvP1G7csU1amR9/azugZAKSquJYza+0OSVfa2txKKjRT/2oXRJ8bktQg6QVr7Vlr7TlJL+jKJQ8XyfFmqGZhkSsWBfzkQIfCEautbDwbE+WF2Xrw+oD+/Z1WtZwbcToOACAOnL7n7DFJayS1Sdov6UvW2oikxZKaL3peS/SxSxhjHjbG7DLG7Oru5oDo84KBEu1r6Vc44uyZjI1NbVpRUaDVlYWO5kglD9+2TB5j9O1XTzgdBQAQB06XswZJeyUtklQn6TFjTNFs3sBa+7i1dqO1dmN5OYdpn1dX7dPIRNjRlX0d/WN669RZpjRjbGFxrn5pY5X+5e0WdQ64Y9EHACB2nC5nD0n6dzvluKT3JK2W1Cqp+qLnVUUfwwydXxTg5JYaz+xrk7VilWYc/Nam5Qpbq8d3nHQ6CgAgxpwuZ2ck3SFJxhi/pFWSTkp6XlK9MaYkuhCgPvoYZihQmqfS/CxH7ztr3NeutYuLtKy8wLEMqaq6NE8P1C3WP/7stHqHxp2OAwCIoXhvpfGkpJ2SVhljWowxnzPGfMEY84XoU/5U0k3GmP2SXpL0h9baHmvt2ei1t6Nf/z36GGbIGKNgtc+xFZtnekfU1NzHcU1x9MXbl2s8FNH3fvqe01EAADGUGc83t9Y+eJXrbZoaFZvu2hOSnohHrnRRV+3TS0e61D86qeJcb0I/u3FfmyTpo6zSjJvl5QW6b91C/WDnaf3mbctVnJfY32MAQHw4Pa2JOAoGpraG29eS+NGzxqY2bbymRIt9uQn/7HTyyJYVGhoP6ftvnHI6CgAgRihnKWx9dbGMSfyigGOdgzrSMcjeZgmwurJI9TV+PfH6exoaDzkdBwAQA5SzFFaU49XKioKELwp4pqlNHiPdu45VmonwyJYV6h+d1N/vPO10FABADFDOUlywukR7m/tkbWI2o7XW6ummNt20fIHKC7MT8pnpbn2VT5uuLdd3Xzup0Ymw03EAAPNEOUtxdQGfzo1M6nRvYo76OdA6oFO9I+xtlmCPblmh3uEJPfnWGaejAADmiXKW4i5sRtucmKnNxn1t8mYYNdRWJuTzMGXjklLdsKxU39lxQuMhRs8AIJlRzlLcyopC5WdlJGRRQCRi1djUpttWlsuXlxX3z8P7PbplpToHxvVvu1ucjgIAmAfKWYrL8BhtqPZpbwI2o9195pza+8d0fx2rNJ1w0/Iybaj26fuvn3I6CgBgHihnaaCu2qdDbQMam4zvdFdjU5tyvB7ducYf18/B9Iwx+ljdIr3bNaST3UNOxwEAzBHlLA0EAyUKRawOtPbH7TNC4Yie3d+uO1b7lZ8d14MncAX10Xv9nj/Y6XASAMBcUc7SQF11dFFAHO8723myVz1DE6zSdNgiX67WLS7WtkMdTkcBAMwR5SwNlBdmq7o0N64rNhub2lSQnanNqyri9hmYmYZav/ac6VPnwJjTUQAAc0A5SxN11SXaG6eRs/FQWM8d6FB9rV853oy4fAZm7vw2JtsOMbUJAMmIcpYmgtU+tfWPqaM/9qMprx3r0cBYiLM0XWJFRYGWLsjXtoNMbQJAMqKcpYnzm9HujcPU5tNNbSrJ8+qWFQti/t6YPWOM6mv92nmiV/2jk07HAQDMEuUsTdQsKlJWhifmiwJGJkJ64VCn7lm3UN4M/ji5RUNtpUIRq1eOdDkdBQAwS/xrmiayMzNUs6hIe2K8Ge3LR7o0OhnW1vVMabpJXZVPFYXZrNoEgCREOUsjwYBP+1r6FApHYvaejU1tqijM1vVLS2P2npg/j8forhq/th/tjvvmwwCA2KKcpZFgoERjkxEd6RiMyfsNjE3qlaPdum/9QmV4TEzeE7HTUFupkYmwfvpuj9NRAACzQDlLI8Hzm9HGaGpz28FOTYQiup9Vmq50w7IyFeZkMrUJAEmGcpZGqkpytaAgO2b7nTU2tamqJPfCCQRwl6xMj7asrtCLh7tiOpUNAIgvylkaMcaortoXk5MCeofG9dPjPdq6YZGMYUrTreprKnV2eEK7T8fvdAgAQGxRztJMMODTye5h9Y1MzOt9fnKgQ+GIZZWmy21eVa6sTA8HoQNAEqGcpZmfb0Y7v6nNxqY2rago0JqFhbGIhTjJz87UrSsW6PmDHbLWOh0HADADlLM0s77KJ4+ZXznr6B/TW6fOaut6pjSTQX2tX619ozrUPuB0FADADFDO0kxBdqau9RfO66SAZ/a1yVpp64aFMUyGeLlzjV8eI6Y2ASBJUM7SUDDg097mPkUic5vmatzXrrWLi7SsvCDGyRAPZQXZ2riklIPQASBJUM7SULC6RP2jk3qvd3jWrz3TO6Km5j4WAiSZ+hq/jnQM6vQcfs8BAIlFOUtDFxYFzGFqs3FfmyTpvvVMaSaThtpKSVMbBwMA3I1yloaWlxeoMDtzTvudNTa16cPXlKiqJC8OyRAv1aV5qllYpOeZ2gQA16OcpSGPx2hDtW/WiwKOdQ7qSMcgxzUlqfpav3afOafuwXGnowAAroBylqaCAZ+OdAxqZCI049c809Qmj5HuWVcZx2SIl4baSlkrvXiYqU0AcDPKWZoKBnwKR6z2t/TP6PnWWjXua9eNy8tUUZgT53SIh9WVhaouzWXVJgC4HOUsTW2omt1JAQdaB/RezzCrNJOYMUYNNZV6/XivBscmnY4DALgMylmaKivI1jVleTO+76xxX5u8GUZ3r2VKM5k1rK3URDii7Ue7nY4CALgMylkaC1b79M6Zc1c9czESsXqmqU23rSyXLy8rQekQDx8KlKgsP0vbDnHfGQC4FeUsjQUDJeoaHFd7/9gVn7f7zDm19Y9pK6s0k16Gx+iuGr9eOdKl8VDY6TgAgGnMqJwZY75kjCkyU75njHnHGFMf73CIrwub0V7lvrPGpjZlZ3p0Z40/EbEQZw21lRoaD+mNE71ORwEATGOmI2f/yVo7IKleUomkX5f01bilQkKsrixSVqZHe85cfjPaUDiiZ/e36841fhVkZyYwHeLlxuVlys/K4LQAAHCpmZYzE/3xXkl/b609eNFjSFJZmR6tW1x8xUUBb548q56hCW3dwHFNqSLHm6HNqyv0wqFOhSNXvt8QAJB4My1nu40x2zRVzp43xhRKisQvFhIlWO3T/tZ+TYan/+18uqlVBdmZ2ryqIsHJEE8NtZXqGRq/4qgpAMAZMy1nn5P0FUnXWWtHJHklPRS3VEiYYKBE46GIjrQPXnJtPBTWcwc6VF/jV443w4F0iJfNq8rlzTCs2gQAF5ppObtR0lFrbZ8x5lOS/qukmW0tD1eriy4KmO4Q9NeO9WhgLKStdazSTDVFOV7dtHyBnj/YcdWtVAAAiTXTcvYtSSPGmA2Sfl/SCUk/iFsqJMyi4hxVFGZPe99Z4742+fK8umXFAgeSId7qa/063TuiY51DTkcBAFxkpuUsZKf+9/oXJD1mrf2GpML4xUKiGGMUDPguufdodCKsFw516p61C+XNYDu8VHRXjV/GSM9z1iYAuMpM/9UdNMb8kaa20PixMcajqfvOkAKCgRKd6h3RueGJC4+9dKRTIxNh3c/GsymrojBHHwqUUM4AwGVmWs5+WdK4pvY765BUJenP45YKCVVXfelmtI1NbaoozNb1S0udioUEqK/x62DbgFrOjTgdBQAQNaNyFi1k/yip2BjzUUlj1lruOUsR66uK5TG6MLU5MDapV4526771C5XhYTu7VNZQO3WQPRvSAoB7zPT4pk9KekvSJyR9UtLPjDG/FM9gSJy8rEytrizSnujI2baDnZoIRThLMw0sWZCvVf5CpjYBwEVmOq35XzS1x9lnrLWflnS9pP8Wv1hItGDAp71n+hSJWDU2tamqJFfB6HQnUlt9rV9vnzqrsxfdcwgAcM5My5nHWtt10c97Z/FaJIG6ap8Gx0Padfqcfnq8R1s3LJIxTGmmg4baSkWs9OJhpjYBwA1mWrCeM8Y8b4z5rDHms5J+LOnZ+MVCogUDJZKkP3v2sMIRq63rmdJMF7WLirTYl6ttTG0CgCtkzuRJ1tovG2M+Lunm6EOPW2ufil8sJNqyBfkqysnU3uY+LS/P15qFbGOXLowxuqvGr39664yGx0PKz57RXwsAgDiZ8dSktfaH1tr/HP2imKUYj8eoLjp6dv+GxUxpppmG2kpNhCLacazb6SgAkPauWM6MMYPGmIFpvgaNMQOJConE+HC0nH10w0KHkyDRrltSopI8LwehA4ALXHH+wlrL3FYaeeiWJbpuSYmWlxc4HQUJlpnh0R1r/Np2sEOT4QhHdgGAg/gbGBcU5Xh1E4ecp62G2koNjIX05slep6MAQFqjnAGQJN26coFyvRmcFgAADqOcAZAk5XgztOnacm071KFIxDodBwDSFuUMwAUNa/3qHBhXU0uf01EAIG1RzgBcsGWVX5kew6pNAHAQ5QzABcV5Xt2wrIyD0AHAQZQzAO/TUOvXye5hHe8adDoKAKQlyhmA97mrplKS9DyrNgHAEZQzAO9TWZyjDdU+DkIHAIdQzgBcor7Gr6aWfrX3jzodBQDSDuUMwCUaaqemNl9g1SYAJBzlDMAlVlQUaHl5Pqs2AcABlDMA06qvrdSbJ8+qf2TS6SgAkFYoZwCm1VBbqXDE6qUjTG0CQCJRzgBMa/3iYlUW5TC1CQAJRjkDMC2Px+iuGr9ePdat0Ymw03EAIG1QzgBcVkNtpcYmI3rt3W6nowBA2qCcAbisjywrVVFOJqcFAEACUc4AXJY3w6M71vj10pFOhcIRp+MAQFqgnAG4ooZav/pGJvXWqbNORwGAtEA5A3BFt11bruxMj7YxtQkACUE5A3BFeVmZunVlubYd7JC11uk4AJDyKGcArqqh1q+2/jEdaB1wOgoApDzKGYCrumONXx4jbTvEhrQAEG+UMwBXVZqfpeuXlnJaAAAkAOUMwIw01FbqWOeQ3usZdjoKAKQ0yhmAGbmrxi9J2sboGQDEFeUMwIxUleRp7eIipjYBIM4oZwBmrKGmUu+c6VPXwJjTUQAgZVHOAMxYfW2lJOmFw2xICwDxQjkDMGPX+gu0pCyPg9ABII4oZwBmzBij+tpK7TzRo4GxSafjAEBKopwBmJWGWr8mw1avHOlyOgoApCTKGYBZCVaXaEFBNgehA0CcUM4AzIrHY3RXjV/bj3ZpbDLsdBwASDmUMwCz1lDr1/BEWG+c6HE6CgCkHMoZgFm7afkCFWZn6vkDTG0CQKxRzgDMWlamR5tXV+jFw50KR6zTcQAgpVDOAMxJQ61fvcMT2n36nNNRACClUM4AzMmma8uVleHhrE0AiDHKGYA5Kczx6uYVZdp2qEPWMrUJALFCOQMwZw21lWo+O6rD7YNORwGAlEE5AzBnd6zxyxgxtQkAMRS3cmaMecIY02WMOXCZ6182xuyNfh0wxoSNMaXRa6eMMfuj13bFKyOA+SkvzNbGa0q07RBbagBArMRz5Oz7ku6+3EVr7Z9ba+ustXWS/kjSq9basxc95fbo9Y1xzAhgnhpqK3W4fUDNZ0ecjgIAKSFu5cxau0PS2as+ccqDkp6MVxYA8VNfUymJqU0AiBXH7zkzxuRpaoTthxc9bCVtM8bsNsY8fJXXP2yM2WWM2dXd3R3PqACmESjL0+rKQg5CB4AYcbycSdoq6fUPTGneYq39kKR7JP22Mea2y73YWvu4tXajtXZjeXl5vLMCmEZDbaXePn1WPUPjTkcBgKTnhnL2K/rAlKa1tjX6Y5ekpyRd70AuADNUX+uXtdJLhxk9A4D5crScGWOKJW2S9KOLHss3xhSe/15SvaRpV3wCcIeahUWqKsnV80xtAsC8ZcbrjY0xT0raLGmBMaZF0p9I8kqStfbb0ad9TNI2a+3wRS/1S3rKGHM+3z9Za5+LV04A82eMUX1Npf7hzdMaGg+pIDtuf7UAQMqL29+g1toHZ/Cc72tqy42LHzspaUN8UgGIl4Zav554/T29erRb961f6HQcAEhabrjnDEAK2LikVKX5WWypAQDzRDkDEBMZHqM711TolSNdmghFnI4DAEmLcgYgZhpqKzU4HtLOk71ORwGApEU5AxAzN69YoLysDKY2AWAeKGcAYibHm6HNq8r1wqFORSLW6TgAkJQoZwBiqqG2Ut2D49rT3Od0FABISpQzADF1++oKeTOMtjG1CQBzQjkDEFNFOV7dsKxMzx/skLVMbQLAbFHOAMRcQ22lTvWO6N2uIaejAEDSoZwBiLm7avySpOcPMLUJALNFOQMQc/6iHAUDPm07xEHoADBblDMAcdFQW6n9rf1q7Rt1OgoAJBXKGYC4qI9ObbJqEwBmh3IGIC6WlRdoZUWBth1kahMAZoNyBiBuGmor9dapszo3POF0FABIGpQzAHFTX+tXOGL10pEup6MAQNKgnAGIm3WLi7WoOIeD0AFgFihnAOLGGKP62krtONatkYmQ03EAIClQzgDEVX2NX+OhiHYc63E6CgAkBcoZgLi6fmmpinO9bKkBADNEOQMQV5kZHt2xpkIvHu7UZDjidBwAcD3KGYC4a6it1MBYSG+9d9bpKADgepQzAHF328py5Xg9rNoEgBmgnAGIu9ysDN22slzbDnYqErFOxwEAV6OcAUiIhtpKdQyMaX9rv9NRAMDVKGcAEuKONRXK8BimNgHgKihnABLCl5eljywt1bZDHIQOAFdCOQOQMA21lTreNaQT3UNORwEA16KcAUiYu2r8ksTUJgBcAeUMQMIs8uVqfVWxth1kahMALodyBiChGmortbe5Tx39Y05HAQBXopwBSKj66NTmC4eY2gSA6VDOACTUiooCLVuQz6pNALgMyhmAhDLGqL62UjtP9Kp/ZNLpOADgOpQzAAlXX+tXKGL18lFGzwDggyhnABKursqnisJsVm0CwDQoZwASzuMxqq/1a/vRbo1Nhp2OAwCuQjkD4Ij6mkqNTob12rs9TkcBAFehnAFwxA3LylSYk6ltnBYAAO9DOQPgiKxMj+5YXaEXD3cqFI44HQcAXINyBsAx9bWVOjcyqV2nzzkdBQBcg3IGwDGbri1XVqaHg9AB4CKUMwCOyc/O1K0rFmjbwU5Za52OAwCuQDkD4KiG2kq19o3qYNuA01EAwBUoZwAcdceaCnmMWLUJAFGUMwCOKivI1sYlpXqe0wIAQBLlDIALNNRW6mjnoE71DDsdBQAcRzkD4Lj6Gr8kadshpjYBgHIGwHHVpXmqWVjE1CYAiHIGwCUaaiv1zplz6hocczoKADiKcgbAFRrW+mWt9OKhLqejAICjKGcAXGGVv1CB0jzuOwOQ9ihnAFzBGKOGWr/eON6rwbFJp+MAgGMoZwBco762UhPhiF452u10FABwDOUMgGt8KFCiBQVZnBYAIK1RzgC4RobH6K4av7Yf7dZ4KOx0HABwBOUMgKvU11RqaDykN473Oh0FABxBOQPgKjetKFNBdiarNgGkLcoZAFfJzszQ5lXleuFQp8IR63QcAEg4yhkA16mvrVTP0ITeOXPO6SgAkHCUMwCuc/uqcmVleFi1CSAtUc4AuE5hjlc3rSjT8wc7ZS1TmwDSC+UMgCvV11TqzNkRHe0cdDoKACQU5QyAK91ZUyFjpOcPdDodBQASinIGwJUqCnP0oUCJnue+MwBphnIGwLUaav061D6g5rMjTkcBgIShnAFwrfqaSknStkNMbQJIH5QzAK61ZEG+VvkLmdoEkFYoZwBcraHWr12nzqp3aNzpKACQEJQzAK5WX1upiJVeOtzldBQASAjKGQBXq11UpMW+XKY2AaQNyhkAVzPGqL7Wr9eO92h4POR0HACIO8oZANdrqK3URCiiV491Ox0FAOKOcgbA9TZeU6KSPC8HoQNIC5QzAK6XmeHRnWv8eulIlyZCEafjAEBcUc4AJIX62koNjoX05slep6MAQFxRzgAkhVtXLlBeVoa2HWJqE0Bqo5wBSAo53gxturZczx/s1Gvvdutox6DODk/IWut0NACIqUynAwDATG3dsEg/OdChX//eWxce82YYLSjIVkVhtsoLs1VemKPywp///OePZys7M8PB9AAwM5QzAEnj3nULtePLt6u9f1TdQ+PqGhh/348t50a1t7lPvcMTmm5ArTjXe0lpq7ikzOWoKDdTxpjE/wIBQJQzAEkmUJanQFneFZ8zGY7o7PBEtLSNqXvw0iK3+8w5dQ2Ma3ya1Z9ZmR6VF1ylxBVlqyw/W1mZ3B0CILYoZwBSjjfDI39RjvxFOZKKL/s8a60Gx0MfKG9j6h4aV3f056d7R7Tr9DmdHZ6Y9j1K8ryXFLfyDxa6omwVZjMaB2BmKGcA0pYxRkU5XhXleLW8vOCKz50IRdQ7HC1xg+PqGjz/49iFn7/XM6zuwXFNhC8djcvxeqZKW8H0o3DlBTnR0bgsZWYwGgekM8oZAMxAVqZHC4tztbA494rPs9ZqYDSk7qGxS6ZSz4/Knege0pvv9apvZPKS13uMdM/ahfrKPatVXXrl6VsAqYlyBgAxZIxRcZ5XxXleragovOJzx0Nh9QxNTJW26Ojbye5h/dNbp/XC4U59/tal+uLmFcrP5q9qIJ2YVNojaOPGjXbXrl1OxwCAeWnvH9X/+skR/cfeNlUUZusP716tjwUXy+PhnjUglRhjdltrN37wcW5sAACXWVicq7/6laB++Fs3aWFxjn7/X5v0sW+9od2nzzkdDUACUM4AwKU+fE2JnvrizfrLT2xQe9+oPv6tN/S7/7xH7f2jTkcDEEeUMwBwMY/H6OMfrtIrf7BZj9y+Qs8e6NCWv3hVf/3iuxqdCDsdD0AcUM4AIAnkZ2fqDxpW6aX/vEm3ry7X/37xmO74y+16uqmN80WBFEM5A4AkUl2ap2/+2of1zw/fIF9eln7nyT36xLd3an9Lv9PRAMQI5QwAktANy8rU+Ogt+p+/uE7v9Qzr/m/8VF/+1yZ1DY45HQ3APFHOACBJZXiMHrw+oFe+vFmfv3WZ/mNvq27/8+361vYTGg9xPxqQrChnAJDkinK8+uN712jb723SjcvL9L+eO6K7/r8deu5AB/ejAUmIcgYAKWLpgnx99zPX6e8/d71yvB594R9261f/5mc63D7gdDQAs0A5A4AUc+vKcj37O7fqv/9CrQ53DOi+r72mP35qv3qHxp2OBmAGKGcAkIIyMzz69I1LtP0PNuvTNy7R/3m7WZv/Yru++9pJTYQiTscDcAWUMwBIYb68LP3f99fquS/dqmCgRP/jx4d191/t0CtHupyOBuAy4lbOjDFPGGO6jDEHLnP9y8aYvdGvA8aYsDGmNHrtbmPMUWPMcWPMV+KVEQDSxUp/of7uoev0xGenzlh+6Ptv6zNPvKXjXYMOJwPwQSZeK3mMMbdJGpL0A2vt2qs8d6uk37PWbjHGZEg6JukuSS2S3pb0oLX20NU+c+PGjXbXrl3zDw8AKWwiFNEPdp7SX7/0rkYmwvr1G67R7915rYrzvE5HA9KKMWa3tXbjBx+P28iZtXaHpLMzfPqDkp6Mfn+9pOPW2pPW2glJ/yzpF+IQEQDSUlamR79x6zJt/4PN+uXrqvWDnae06S9e0Q92nlIozP1ogNMcv+fMGJMn6W5JP4w+tFhS80VPaYk+drnXP2yM2WWM2dXd3R2/oACQYsoKsvVnH1unZx69VasrC/V//eig7v3aa/rpuz1ORwPSmuPlTNJWSa9ba2c6yvY+1trHrbUbrbUby8vLYxwNAFJfzaIiPfn5G/TtT31Io5Nhfep7P9Nv/N0u/f/t3Wt0VXV+xvHnl3sIuQpJIAkXBbmEOxlHBQUEIS5GplptZ6Y6jnW61lS0ozhjnVZrV1+MfYHazmgRx05rK2qnyuClSLiIYTkqColcQkARSQjkApKECITc/n1xDjEyzqwk5mTvc87384psDuF31obkyX/v578Pnzjt9WhAVPJDOPuOvrikKUlHJRX0+Dg/eAwAECJmpuIpI7Tp3nm6v3iC3v3khK59vFSPrK9US2u71+MBUcXTcGZm6ZLmSXqlx+EPJI03s7FmlqBAeHvVi/kAINokxcfqzvnjtPUn8/XtGXlave2QFqx8Sy++X63OLh4FBQyGUG6l8YKkdyVNMLMaM7vDzH5kZj/q8bIbJG10znWvnTvnOiTdJalEUqWk3zjnKkI1JwDg92WnJWnlzdP16l1zNPqiFD2wdo+WPfG2th/6zOvRgIgXsq00vMBWGgAw8Jxzem13WdAPpQAAD5ZJREFUrR5ZX6na5lYtnTpCD1w3UQVZQ7weDQhrg76VBgAgMpiZlk0fqTfvm697Fo3Xlv31WvhYqVaWHNDpcx1ejwdEHMIZAKBXkhNidc+iS/XmffNVXJirJ7Ye1DWPvqW1ZTXq4n40YMAQzgAAfTIyI1m/+O5MvfzXVygnLUkrfrNLN6x6R2XVjV6PBkQEwhkAoF9mj87SujvnaOXN01XbdFY3/ts7uufFctU2n/V6NCCsEc4AAP0WE2O6aXa+tv5kvpYvuETr99bpmpWl+sWWj9Xa3un1eEBYIpwBAL62lMQ4/XTJRG1ZMU/zJwzXY5s+0sJHS/XarmOKpF0BgMFAOAMADJiCrCFadctsvfBXlystOV53v1CuP1v9rvbUNHs9GhA2CGcAgAF3xSUX6fW75+rnN0zVoeOntezJt3X/S7vU0NLq9WiA7xHOAAAhERtj+t43R2nrT+frh3PHam3ZUV2zslSrSz9Re2eX1+MBvkU4AwCEVFpSvP5+6WRtvPdqXTY2S4+8sV/X//Jt7axi6w3gqxDOAACD4uLhQ/XrH3xDq2+dreaz7brpqXf04Lo9aj7b7vVogK8QzgAAg2pJYa42rZinH1w5Rs9vr9aix0r1+m5ancB5hDMAwKAbmhinh68v1Lrlc5STlqi7ni/X7f/5gY6cPOP1aIDnCGcAAM9My8/Qujvn6KFvTdb7n57UtY+X6ikKA4hyhDMAgKfiYmN0x9yx2rxinq4aP1z/TGEAUY5wBgDwhZEZyfrV94u0+tbZajpDYQDRi3AGAPCVJYW52nwfhQFEL8IZAMB3KAwgmhHOAAC+RWEA0YhwBgDwtZ6FgbnjvigMlFVTGEBkIpwBAMLCyIxkPXPbF4WBP10VKAycaqUwgMhCOAMAhJULCwMLHy3V/+2upTCAiEE4AwCEnQsLA8ufL9NfUhhAhCCcAQDCVs/CwPZgYWA1hQGEOcIZACCsXVgYeITCAMIc4QwAEBG+qjDw0Lq9FAYQdghnAICI0rMwsGZ7FYUBhB3CGQAg4lAYQDgjnAEAIhaFAYQjwhkAIKKdLwxsojCAMEE4AwBEhTwKAwgThDMAQFS5sDCwiMIAfIZwBgCIOj0LA8NTKQzAXwhnAICoNS0/Q68sn6MHl07S9k9PavHj2ygMwHOEMwBAVIuLjdEPr7pYm1bM05xxF3UXBsopDMAjhDMAABQoDPzq+0V66pZAYeBGCgPwCOEMAIAgM1PxlFxtWnG1brtijJ4LFgbW76EwgMFDOAMA4AKpSfH6x2WFWndnoDBw55oy3fHsDgoDGBSEMwAA/oDpBV8UBt479JkWP75NT2+jMIDQIpwBAPBHXFgY+Pn6/Vr2xO8oDCBkCGcAAPRCz8JA4+k23bjqHf3DKxQGMPAIZwAA9NKFhYH/fo/CAAYe4QwAgD6iMIBQIpwBANBPX1UYWPXWJ6ppPMNKGvrNIukfT1FRkduxY4fXYwAAotDRprN6+JW92lzZIEnKSUvUrFGZmj06U7NGZ6pwZJoS42I9nhJ+YmY7nXNFFx6P82IYAAAizfnCQGVti3ZUnVRZVaN2Vjfqjb11kqSE2BhNzU/XrFEZgcA2KlPZaUkeTw0/YuUMAIAQamhpVVlVk8qqG1VW1ajdR5vV1hHYJy0/M7l7dW326ExNzE1VXCx3HEWLP7RyRjgDAGAQnevo1L5jp7SzqlHl1U3aUXVS9afOSZKS42M1vSC9O7DNHJWprJQEjydGqHBZEwAAH0iMi9XMUYHgJUnOOR1rbg1cBq1qVHl1o57edkgdXYHFk4uHpWhmj9W18dlDFRNjXr4FhBjhDAAAD5mZ8jKSlZeRrOunj5QknW3r1J6jzdoZDGxvHWjQy2U1kqTUxDjNGJXRvbo2Y1SG0pLivXwLGGCEMwAAfCY5IVaXjc3SZWOzJAVW16o+O6Oy6kBYK6tu0i/f/FhdTjKTLs1O1azRmd1lg7HDUmTG6lq44p4zAADCUEtru3Ydae4ObOXVjTrV2iFJyhwSr1mjMoOBLVPTC9I1JIH1GL/hnjMAACJIalK85o4fprnjh0mSurqcPjn+eXdY21nVqC37A3uuxcaYJo1I1ewegS0/M5nVNZ9i5QwAgAjVdKZN5dVN3YHtwyNNOtPWKUkanpoYDGuBS6GFI9OVFM8muYOJlTMAAKJMxpAELZiYrQUTsyVJHZ1dOlDforLqpu526IaKLzbJLcxL615dmz06UzlskusJVs4AAIhix1vOBTbIDW6Su6vmi01y8zKSv1Q0mDQiTfFskjtgWDkDAAC/Z3hqopYU5mpJYa4kqa2jS/tqTwVboY3acfikXtt1TJKUFB+jafkZumrcMBVPydW47KHctxYCrJwBAIA/6ljT2eDKWuCJBrtrmiUFNshdMiVXxYW5mpafTlDrIx7fBAAABkT9qVZt3Fevkr11evfQZ+rschqRntS9AveNMZk8I7QXCGcAAGDANZ1p05bKBm2oqNO2j47rXEeXslISdO2kHC2ZkqM544YpMY4W6FchnAEAgJA609ah0gPHtaGiTm9WNqjlXIeGJsZpwcRsLSnM0fwJ2RqayO3u51EIAAAAITUkIU7XTR2h66aO0LmOTr37yWcqqajTxop6vbbrmBLiYnT1+GFaUpirRZNylJmS4PXIvsTKGQAACKnOLqcdh0+qpKJeJRV1Otp0VrExpm+OzVLxlFwtnpyr3PTo21ONy5oAAMBzzjntPXpKGypqVVJRr4MNn0uSZhRkqDjY/BwzLMXjKQcH4QwAAPjOwYYWlVTUa8PeOu05GtiiY2JuqhYXBoLapBGpEbtFB+EMAAD4Wk3jGW2sqNeGijrtOHxSXU4alTVExVNytaQwRzMLMhUTEzlBjXAGAADCxonPz2nzvkBQ+93BE2rvdMpOTdTiwhwtKczV5RdfFPaPkiKcAQCAsHSqtV1b9zeopKJOW/cf19n2TqUnx2vhpGwVF+bq6kuHKyk+/PZSI5wBAICw19reqW0fBfZS21LZoOaz7UqOj9X8CcNVPCVXCyZmKy0p3usxe4V9zgAAQNhLio/V4sJcLS7MVXtnl7YfOqkNFbXaWFGvN/bWKT7WdOUlgQezXzs5R8OGJno9cp+xcgYAAMJeV5dT+ZEmlVTUacPeOlWfPKMYk4rGZAWf+Zmj/MwhXo/5JVzWBAAAUcE5p8raFpVU1Kmkok7761okSVPz0rubn+OyUz2eknAGAACi1OETpwMrahV1Kq9ukiRdMjwluOntCE3JS/NkLzXCGQAAiHp1za3auC+wovbeoZPq7HLKy0jW4sIcFRfmqmhMlmIHaS81whkAAEAPjafbtLmyXiUV9dr28XG1dXTpopQEXTs5Rw9fX6jkhNBuz0FbEwAAoIfMlATdXFSgm4sKdPpch946ENiiY1dNs5LivdvglnAGAACiXkpinJZOG6Gl00bIOefp8zzD+7kHAAAAA8zrB60TzgAAAHyEcAYAAOAjhDMAAAAfIZwBAAD4COEMAADARwhnAAAAPkI4AwAA8BHCGQAAgI8QzgAAAHyEcAYAAOAjhDMAAAAfIZwBAAD4COEMAADARwhnAAAAPkI4AwAA8BHCGQAAgI8QzgAAAHyEcAYAAOAjhDMAAAAfIZwBAAD4COEMAADARwhnAAAAPmLOOa9nGDBmdlxSVYj/mmGSToT470BocQ7DG+cv/HEOwx/ncGCMds4Nv/BgRIWzwWBmO5xzRV7Pgf7jHIY3zl/44xyGP85haHFZEwAAwEcIZwAAAD5COOu7p70eAF8b5zC8cf7CH+cw/HEOQ4h7zgAAAHyElTMAAAAfIZwBAAD4COGsl8ys2MwOmNlBM3vA63nQN2ZWYGZbzWyfmVWY2Y+9ngn9Y2axZlZuZq97PQv6zswyzOwlM9tvZpVmdoXXM6H3zOze4NfQvWb2gpkleT1TJCKc9YKZxUp6UtJ1kiZL+q6ZTfZ2KvRRh6T7nHOTJV0uaTnnMGz9WFKl10Og3/5V0gbn3ERJ08W5DBtmlifpbyQVOeemSIqV9B1vp4pMhLPeuUzSQefcIedcm6QXJX3b45nQB865WudcWfDXLQp8Q8jzdir0lZnlS1oq6RmvZ0HfmVm6pKsl/bskOefanHNN3k6FPoqTlGxmcZKGSDrm8TwRiXDWO3mSjvT4uEZ8Yw9bZjZG0kxJ272dBP3wL5Lul9Tl9SDol7GSjkv6j+Cl6WfMLMXrodA7zrmjklZKqpZUK6nZObfR26kiE+EMUcXMhkp6WdI9zrlTXs+D3jOzb0lqcM7t9HoW9FucpFmSVjnnZko6LYl7eMOEmWUqcNVorKSRklLM7BZvp4pMhLPeOSqpoMfH+cFjCCNmFq9AMFvjnFvr9TzoszmSlpnZYQVuLbjGzJ7zdiT0UY2kGufc+VXrlxQIawgPiyR96pw77pxrl7RW0pUezxSRCGe984Gk8WY21swSFLgB8lWPZ0IfmJkpcJ9LpXPuMa/nQd85537mnMt3zo1R4P/gm845fmoPI865OklHzGxC8NBCSfs8HAl9Uy3pcjMbEvyaulAUOkIizusBwoFzrsPM7pJUokA75dfOuQqPx0LfzJF0q6Q9ZvZh8NjfOefWezgTEI3ulrQm+IPuIUm3ezwPesk5t93MXpJUpkADvlw8xikkeHwTAACAj3BZEwAAwEcIZwAAAD5COAMAAPARwhkAAICPEM4AAAB8hHAGAF+Tmc03s9e9ngNAZCCcAQAA+AjhDEDUMLNbzOx9M/vQzFabWayZfW5mj5tZhZltMbPhwdfOMLP3zGy3mf02+FxBmdk4M9tsZrvMrMzMLgl++qFm9pKZ7TezNcEd1AGgzwhnAKKCmU2S9OeS5jjnZkjqlPQXklIk7XDOFUoqlfRw8I/8l6S/dc5Nk7Snx/E1kp50zk1X4LmCtcHjMyXdI2mypIsVeCoFAPQZj28CEC0WSpot6YPgolaypAZJXZL+J/ia5yStNbN0SRnOudLg8Wcl/a+ZpUrKc879VpKcc62SFPx87zvnaoIffyhpjKS3Q/+2AEQawhmAaGGSnnXO/exLB80euuB1/X2m3bkev+4UX18B9BOXNQFEiy2SbjKzbEkysywzG63A18Gbgq/5nqS3nXPNkhrN7Krg8VsllTrnWiTVmNmfBD9HopkNGdR3ASDi8ZMdgKjgnNtnZg9K2mhmMZLaJS2XdFrSZcHfa1DgvjRJuk3SU8HwdUjS7cHjt0pabWb/FPwcNw/i2wAQBcy5/q7gA0D4M7PPnXNDvZ4DAM7jsiYAAICPsHIGAADgI6ycAQAA+AjhDAAAwEcIZwAAAD5COAMAAPARwhkAAICP/D8LP18rC4l4hQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# dot_img_file = \"d_mod-\"+str(d_model)+\"-d_k-\"+str(d_k)+\"-d_v-\"+str(d_v)+\"-n_heads-\"+str(n_heads)+\"-enc_lays-\"+str(n_encoder_layers)+\"-dec_lays-\"+str(n_decoder_layers)+'-transformer.png'\n",
        "# tf.keras.utils.plot_model(transformer, to_file=dot_img_file, show_shapes=True)"
      ],
      "metadata": {
        "id": "RSK1eufbeQPL"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#transformer = keras.models.load_model('./Models/')"
      ],
      "metadata": {
        "id": "Oly3kYeU3FnV"
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Testing"
      ],
      "metadata": {
        "id": "KJ_Gx7RVeNla"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def greedy_translate(model, source_sentence, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    predict_label = tf.argmax(predict_output, axis=-1) # Pick the label with highest softmax score\n",
        "    predict_seq = tf.concat([predict_seq, predict_label], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label[0][0].numpy())\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False)\n",
        "  return detokenize(predict_sentence, sp_ben_bpe)"
      ],
      "metadata": {
        "id": "zfg1AaKDnPQM"
      },
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow_probability as tfp"
      ],
      "metadata": {
        "id": "pLrtBO__iPPl"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "KtWeJl1JoapJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "955f3547-c6dd-4638-f9af-d0a3b8e4e5e8"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  কিন্তু,।\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = list(test_val_data['src'][:250])\n",
        "refs = [list(test_val_data['tgt'][:250])]\n",
        "sys = [greedy_translate(transformer, sentence) for sentence in sentences]"
      ],
      "metadata": {
        "id": "9XNckV-wBgiN"
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bleu = BLEU()\n",
        "bleu.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "otRhPJC1VKyj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e767a95c-1144-4390-b53d-38daf6447d47"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BLEU = 0.00 13.2/0.1/0.1/0.0 (BP = 0.023 ratio = 0.209 hyp_len = 750 ref_len = 3583)"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "chrf = CHRF()\n",
        "chrf.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "mlhToSrvYNqd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7c9286c2-1b9c-4bca-821b-f0101db52ddf"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "chrF2 = 2.18"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ter = TER()\n",
        "ter.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "DtIadLMJYe1w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "89522385-f2ca-487b-ce56-677224a603a1"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TER = 100.00"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def sampling_translate(model, source_sentence, temperature=0.6, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  logprob = 0\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    probs = predict_output[-1].numpy()[0]\n",
        "    log_probs = np.log(probs)\n",
        "    u = np.random.uniform(low=1e-6, high=1.0 - 1e-6, size=log_probs.shape)\n",
        "    g = -np.log(-np.log(u))\n",
        "    sample_idx = np.argmax(log_probs + g * temperature, axis=-1)\n",
        "    sample_prob = predict_output[-1].numpy()[0][sample_idx]\n",
        "    logprob += np.log(sample_prob)\n",
        "    predict_label = np.arange(predict_output.shape[-1])[sample_idx] # Sample the label softmax score\n",
        "    predict_seq = tf.concat([predict_seq, tf.cast([[predict_label]], dtype=tf.int32)], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label)\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False), logprob\n",
        "  return detokenize(predict_sentence, sp_ben_bpe), logprob"
      ],
      "metadata": {
        "id": "2dmVo_eQhE8v"
      },
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", sampling_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "CB5iHHlxh1cC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a69bc655-c527-45be-a2d0-dd114817f438"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  ('তাঁর এমন রকম?', -23.182364129461348)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def generate_samples(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = [], []\n",
        "    for _ in range(n_samples):\n",
        "        sample, logp = sampling_translate(model, sentence, temperature)\n",
        "        samples.append(sample)\n",
        "        log_probs.append(logp)\n",
        "    return samples, log_probs"
      ],
      "metadata": {
        "id": "xIxJ9HqTtxHY"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def rouge1_similarity(system, reference):\n",
        "    sys_counter = Counter(system)\n",
        "    ref_counter = Counter(reference)\n",
        "    overlap = 0\n",
        "    for token in sys_counter:\n",
        "        token_count_sys = sys_counter[token]\n",
        "        token_count_ref = ref_counter[token]\n",
        "        overlap += min(token_count_ref, token_count_sys)\n",
        "    precision = overlap / sum(sys_counter.values())\n",
        "    recall = overlap / sum(ref_counter.values())\n",
        "    if precision + recall != 0:\n",
        "        rouge1_score = 2 * ((precision * recall)/(precision + recall))\n",
        "    else:\n",
        "        rouge1_score = 0     \n",
        "    return rouge1_score"
      ],
      "metadata": {
        "id": "uhIlVoVwt7YC"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def weighted_avg_overlap(samples, log_probs):\n",
        "    scores = {}\n",
        "    for index_candidate, candidate in enumerate(samples):    \n",
        "        overlap, weight_sum = 0.0, 0.0\n",
        "        for index_sample, (sample, logp) in enumerate(zip(samples, log_probs)):           \n",
        "            if index_candidate == index_sample:\n",
        "                continue\n",
        "            sample_p = float(np.exp(logp))\n",
        "            weight_sum += sample_p\n",
        "            sample_overlap = rouge1_similarity(candidate, sample)\n",
        "            overlap += sample_p * sample_overlap\n",
        "        score = overlap / weight_sum\n",
        "        scores[index_candidate] = score\n",
        "    return scores"
      ],
      "metadata": {
        "id": "7KxiaRENt-C_"
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def mbr_decode(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = generate_samples(sentence, n_samples, model, temperature)\n",
        "    scores = weighted_avg_overlap(samples, log_probs)\n",
        "    max_score_key = max(scores, key=scores.get)\n",
        "    return {sample:score for sample, score in zip(samples, scores.values())}"
      ],
      "metadata": {
        "id": "ny5EYLFWt_5u"
      },
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "translated_sentence = mbr_decode(sentence, 2, transformer, 1)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "lzrxKRMAaOgr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a4694ceb-f969-4eb4-ccb2-4726e19591a3"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I love you.\n",
            "Bengali:  {'দেশের ছোট সরকারি ওঁকে সামনে।': 0.4380952380952381, 'রাস্তায় কোন তেমনটা কোন পেয়েছে পারে কারো এখানে এ গ্রেপ্তারবেন হয় করে সঙ্গে।': 0.4380952380952381}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "3ACvLvfouZz0"
      },
      "execution_count": 40,
      "outputs": []
    }
  ]
}