C5_W4_A1_Transformer_Subclass_v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AbzZLqIPv6b7",
    "outputId": "19f2fc2b-6f1d-4b43-fd50-4c513e3936fd"
   },
   "source": [
    "# Transformer Network\n",
    "\n",
    "Welcome to Week 4's assignment, the last assignment of Course 5 of the Deep Learning Specialization! And congratulations on making it to the last assignment of the entire Deep Learning Specialization - you're almost done!\n",
    "\n",
    "Ealier in the course, you've implemented sequential neural networks such as RNNs, GRUs, and LSTMs. In this notebook you'll explore the Transformer architecture, a neural network that takes advantage of parallel processing and allows you to substantially speed up the training process. \n",
    "\n",
    "**After this assignment you'll be able to**:\n",
    "\n",
    "* Create positional encodings to capture sequential relationships in data\n",
    "* Calculate scaled dot-product self-attention with word embeddings\n",
    "* Implement masked multi-head attention\n",
    "* Build and train a Transformer model\n",
    "\n",
    "For the last time, let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "- [Packages](#0)\n",
    "- [1 - Positional Encoding](#1)\n",
    "    - [1.1 - Sine and Cosine Angles](#1-1)\n",
    "        - [Exercise 1 - get_angles](#ex-1)\n",
    "    - [1.2 - Sine and Cosine Positional Encodings](#1-2)\n",
    "        - [Exercise 2 - positional_encoding](#ex-2)\n",
    "- [2 - Masking](#2)\n",
    "    - [2.1 - Padding Mask](#2-1)\n",
    "    - [2.2 - Look-ahead Mask](#2-2)\n",
    "- [3 - Self-Attention](#3)\n",
    "    - [Exercise 3 - scaled_dot_product_attention](#ex-3)\n",
    "- [4 - Encoder](#4)\n",
    "    - [4.1 Encoder Layer](#4-1)\n",
    "        - [Exercise 4 - EncoderLayer](#ex-4)\n",
    "    - [4.2 - Full Encoder](#4-2)\n",
    "        - [Exercise 5 - Encoder](#ex-5)\n",
    "- [5 - Decoder](#5)\n",
    "    - [5.1 - Decoder Layer](#5-1)\n",
    "        - [Exercise 6 - DecoderLayer](#ex-6)\n",
    "    - [5.2 - Full Decoder](#5-2)\n",
    "        - [Exercise 7 - Decoder](#ex-7)\n",
    "- [6 - Transformer](#6)\n",
    "    - [Exercise 8 - Transformer](#ex-8)\n",
    "- [7 - References](#7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='0'></a>\n",
    "## Packages\n",
    "\n",
    "Run the following cell to load the packages you'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "_OpwqWL2QH5G"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from tensorflow.keras.layers import Embedding, MultiHeadAttention, Dense, Input, Dropout, LayerNormalization\n",
    "from transformers import DistilBertTokenizerFast #, TFDistilBertModel\n",
    "from transformers import TFDistilBertForTokenClassification\n",
    "from tqdm import tqdm_notebook as tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Positional Encoding\n",
    "\n",
    "In sequence to sequence tasks, the relative order of your data is extremely important to its meaning. When you were training sequential neural networks such as RNNs, you fed your inputs into the network in order. Information about the order of your data was automatically fed into your model.  However, when you train a Transformer network, you feed your data into the model all at once. While this dramatically reduces training time, there is no information about the order of your data. This is where positional encoding is useful - you can specifically encode the positions of your inputs and pass them into the network using these sine and cosine formulas:\n",
    "    \n",
    "$$\n",
    "PE_{(pos, 2i)}= sin\\left(\\frac{pos}{{10000}^{\\frac{2i}{d}}}\\right)\n",
    "\\tag{1}$$\n",
    "<br>\n",
    "$$\n",
    "PE_{(pos, 2i+1)}= cos\\left(\\frac{pos}{{10000}^{\\frac{2i}{d}}}\\right)\n",
    "\\tag{2}$$\n",
    "\n",
    "* $d$ is the dimension of the word embedding and positional encoding\n",
    "* $pos$ is the position of the word.\n",
    "* $i$ refers to each of the different dimensions of the positional encoding.\n",
    "\n",
    "The values of the sine and cosine equations are small enough (between -1 and 1) that when you add the positional encoding to a word embedding, the word embedding is not significantly distorted. The sum of the positional encoding and word embeding is ultimately what is fed into the model. Using a combination of these two equations helps your Transformer network attend to the relative positions of your input data. Note that while in the lectures Andrew uses vertical vectors but in this assignment, all vectors are horizontal. All matrix multiplications should be adjusted accordingly.\n",
    "\n",
    "<a name='1-1'></a>\n",
    "### 1.1 - Sine and Cosine Angles\n",
    "\n",
    "Get the possible angles used to compute the positional encodings by calculating the inner term of the sine and cosine equations: \n",
    "\n",
    "$$\\frac{pos}{10000^{\\frac{2i}{d}}} \\tag{3}$$\n",
    "\n",
    "<a name='ex-1'></a>\n",
    "### Exercise 1 - get_angles\n",
    "\n",
    "Implement the function `get_angles()` to calculate the possible angles for the sine and cosine  positional encodings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "bPzwMVfcQpT-"
   },
   "outputs": [],
   "source": [
    "# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION get_angles\n",
    "def get_angles(pos, i, d):\n",
    "    \"\"\"\n",
    "    Get the angles for the positional encoding\n",
    "    \n",
    "    Arguments:\n",
    "        pos -- Column vector containing the positions [[0], [1], ...,[N-1]]\n",
    "        i --   Row vector containing the dimension span [[0, 1, 2, ..., M-1]]\n",
    "        d(integer) -- Encoding size\n",
    "    \n",
    "    Returns:\n",
    "        angles -- (pos, d) numpy array \n",
    "    \"\"\"\n",
    "    # STATR CODE HERE\n",
    "    angles = pos / np.power(10000, (2*(i//2)) / d)\n",
    "    # END CODE HERE\n",
    "    \n",
    "    return angles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mAll tests passed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.e+00, 0.e+00, 0.e+00, 0.e+00, 0.e+00, 0.e+00, 0.e+00, 0.e+00],\n",
       "       [1.e+00, 1.e+00, 1.e-01, 1.e-01, 1.e-02, 1.e-02, 1.e-03, 1.e-03],\n",
       "       [2.e+00, 2.e+00, 2.e-01, 2.e-01, 2.e-02, 2.e-02, 2.e-03, 2.e-03],\n",
       "       [3.e+00, 3.e+00, 3.e-01, 3.e-01, 3.e-02, 3.e-02, 3.e-03, 3.e-03]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def get_angles_test(target):\n",
    "    position = 4\n",
    "    d_model = 16\n",
    "    pos_m = np.arange(position)[:, np.newaxis]\n",
    "    dims = np.arange(d_model)[np.newaxis, :]\n",
    "\n",
    "    result = target(pos_m, dims, d_model)\n",
    "\n",
    "    assert type(result) == np.ndarray, \"You must return a numpy ndarray\"\n",
    "    assert result.shape == (position, d_model), f\"Wrong shape. We expected: ({position}, {d_model})\"\n",
    "    assert np.sum(result[0, :]) == 0\n",
    "    assert np.isclose(np.sum(result[:, 0]), position * (position - 1) / 2)\n",
    "    even_cols =  result[:, 0::2]\n",
    "    odd_cols = result[:,  1::2]\n",
    "    assert np.all(even_cols == odd_cols), \"Submatrices of odd and even columns must be equal\"\n",
    "    limit = (position - 1) / np.power(10000,14.0/16.0)\n",
    "    assert np.isclose(result[position - 1, d_model -1], limit ), f\"Last value must be {limit}\"\n",
    "\n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "\n",
    "get_angles_test(get_angles)\n",
    "\n",
    "# Example\n",
    "position = 4\n",
    "d_model = 8\n",
    "pos_m = np.arange(position)[:, np.newaxis]\n",
    "dims = np.arange(d_model)[np.newaxis, :]\n",
    "get_angles(pos_m, dims, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-2'></a>\n",
    "### 1.2 - Sine and Cosine Positional Encodings\n",
    "\n",
    "Now you can use the angles you computed to calculate the sine and cosine positional encodings.\n",
    "\n",
    "$$\n",
    "PE_{(pos, 2i)}= sin\\left(\\frac{pos}{{10000}^{\\frac{2i}{d}}}\\right)\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "PE_{(pos, 2i+1)}= cos\\left(\\frac{pos}{{10000}^{\\frac{2i}{d}}}\\right)\n",
    "$$\n",
    "\n",
    "<a name='ex-2'></a>\n",
    "### Exercise 2 - positional_encoding\n",
    "\n",
    "Implement the function `positional_encoding()` to calculate the sine and cosine  positional encodings\n",
    "\n",
    "**Reminder:** Use the sine equation when $i$ is an even number and the cosine equation when $i$ is an odd number.\n",
    "\n",
    "#### Additional Hints\n",
    "* You may find \n",
    "[np.newaxis](https://numpy.org/doc/stable/reference/arrays.indexing.html) useful depending on the implementation you choose. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "y78txxoHQtwG"
   },
   "outputs": [],
   "source": [
    "# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION positional_encoding\n",
    "def positional_encoding(positions, d):\n",
    "    \"\"\"\n",
    "    Precomputes a matrix with all the positional encodings \n",
    "    \n",
    "    Arguments:\n",
    "        positions (int) -- Maximum number of positions to be encoded \n",
    "        d (int) -- Encoding size \n",
    "    \n",
    "    Returns:\n",
    "        pos_encoding -- (1, position, d_model) A matrix with the positional encodings\n",
    "    \"\"\"\n",
    "    # START CODE HERE\n",
    "    # initialize a matrix angle_rads of all the angles \n",
    "    \n",
    "    posm2 = np.arange(positions)[:, np.newaxis]\n",
    "    dims2 = np.arange(d)[np.newaxis, :]\n",
    "    angle_rads = get_angles(posm2, dims2, d)\n",
    "        \n",
    "    # apply sin to even indices in the array; 2i\n",
    "    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])\n",
    "  \n",
    "    # apply cos to odd indices in the array; 2i+1\n",
    "    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])\n",
    "    # END CODE HERE\n",
    "    \n",
    "    pos_encoding = angle_rads[np.newaxis, ...]\n",
    "    \n",
    "    return tf.cast(pos_encoding, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "jYiWrawRQvuv",
    "outputId": "cfccc7c9-428e-4b08-d969-e3090fafc1ad"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mAll tests passed\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def positional_encoding_test(target):\n",
    "    position = 8\n",
    "    d_model = 16\n",
    "\n",
    "    pos_encoding = target(position, d_model)\n",
    "    sin_part = pos_encoding[:, :, 0::2]\n",
    "    cos_part = pos_encoding[:, :, 1::2]\n",
    "\n",
    "    assert tf.is_tensor(pos_encoding), \"Output is not a tensor\"\n",
    "    assert pos_encoding.shape == (1, position, d_model), f\"Wrong shape. We expected: (1, {position}, {d_model})\"\n",
    "\n",
    "    ones = sin_part ** 2  +  cos_part ** 2\n",
    "    assert np.allclose(ones, np.ones((1, position, d_model // 2))), \"Sum of square pairs must be 1 = sin(a)**2 + cos(a)**2\"\n",
    "    \n",
    "    angs = np.arctan(sin_part / cos_part)\n",
    "    angs[angs < 0] += np.pi\n",
    "    angs[sin_part.numpy() < 0] += np.pi\n",
    "    angs = angs % (2 * np.pi)\n",
    "    \n",
    "    pos_m = np.arange(position)[:, np.newaxis]\n",
    "    dims = np.arange(d_model)[np.newaxis, :]\n",
    "\n",
    "    trueAngs = get_angles(pos_m, dims, d_model)[:, 0::2] % (2 * np.pi)\n",
    "    \n",
    "    assert np.allclose(angs[0], trueAngs), \"Did you apply sin and cos to even and odd parts respectively?\"\n",
    " \n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "\n",
    "    \n",
    "positional_encoding_test(positional_encoding)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice work calculating the positional encodings! Now you can visualize them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 50, 512)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEKCAYAAAD+XoUoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOydd3gc1fm273dmd6VV77Jsyw1344oxNqaZ3g0kEFooIZBGAmmEFPJLT0i+EEgCIUAIkAKhBLAJzWDAYJox7jZucpesXlfbZuZ8f+ysvJIla2VLxrLPfV3H02fPyquzo+c97/OKUgqNRqPRHBkYn3YHNBqNRnPw0IO+RqPRHEHoQV+j0WiOIPSgr9FoNEcQetDXaDSaIwg96Gs0Gs0RRJ8O+iKyVURWichyEfnI3ZcnIgtEZKO7zO3LPmg0Gs2niYg8LCJVIrK6i+MiIn8UkU0islJEpiUcO1tE1rvHbu+N/hyMJ/05SqkpSqnp7vbtwOtKqVHA6+62RqPRHK48Apy9j+PnAKPcdhPwFwARMYF73ePjgStEZPyBdubTkHfmAo+6648CF30KfdBoNJqDglJqEVC3j1PmAo+pGO8DOSJSAswANimlypRSEeAJ99wDwnOgN+gGBbwqIgr4q1LqAaBYKVUBoJSqEJGizi4UkZuIfeuRnuY/Jq3VpnTKOJat38mUsUPYsWwNQycMZ/n2RtJzsxncXEF9Q5gBUydQ0xoho3wbtU1hBo8ZzPpmD631tRQOLGaQaqR8SzWphlAwdhg71mwhzTTIG1NKecRHdWUtynHIzM/jqHw/4R1l1NUGsRW0lAwl1NSIUoqUjCyK89LITwGrejeBqmaaLQeANNMgPSeFUGOYFtvBVuATIT3FJDXHjzc3Fyc1k+aITX0gQmvQIjvTR06qlzSvgREN4gSaiDS3Eg1EiEQcwo7CVgoHKJ16NIYVQoVbsYNBrGAYK2Rhh20ijoPl0HauAgZNmYDlKMK2Q8RWRCyHiGUTsRwcW8Wa46AcmzHeZkyvB8NjgseDeLyI6QXDRBlmbIngKFi5YUf8fwtEEHGX8W3D2LNtGKRlpKKUwnEUSgEqtlQqvg0q9g++VA8iIAix2wgCGCK4LxM7JlBZ1QDxzPL4jeL/dsw4V4oRwwbEP2PInncQexvuVnz7k007k/6wTxhVuufz28U5knBg1frtSd970pghXd+0w2uu+CT5+04ZOyTpcwGW9+jeQ3tw32096seUcZ3fe/m6bahgbY1SqrBHN+yAkTVYYYW6PU8Fa9cAiSc+4I5zPWEQsCNhe6e7r7P9x/Xw3nvR14P+bKVUuTuwLxCRT5K90P3BPQBwzKQJaubqAHe9tZDMk7/Donfu5Tvp47j36QfJv/kVjv/sufx64c95dv5Gvrd4MX9bVsHMn3+Rf71axp1/+zUnv5XP0qf+xeU//ia/jrzATz//IKMzfFz39IN8c8I1HJuTyhX/voc7dg7mL3/4N9FQgBOvuYynrzqaLbd8nn/9cxWNUYd3b/wT615/GScaYdjxZ/HtKyZzzQiTmvt/yQd/XsQb1a0ATMtO5bgLRrHx5TIW17bSGHUYmOJh1rBsxlw0iYGf/SyBsafy1rZGnvhoBytXVnLuycO5YMIApg5II618Ba0fvMqut5ZTvmQX27Y3sbU1Sl3EJuIo7lq8mJSajVgbl9GydhU1KzdTu76G+rIGdrVEqA7b1Edtgu4Xzs/efJuaoM22hiDbG0NsrQmwrTZAeW0rgaYwrY1hQq0Rws0NzCt5k/QBefiLcvHkFWLmD8DMLYL0HJyUTBx/DlEzhdaoQ+mptyKG2dZMrw/D48PweDE8PswUP6bH17Y+7YRRBCM24bCFFXGwojZW1Ma2HKyog2M52LaDbTkMGVOAx2Pg8xik+Ux8HgOfx12aBinuMZ/H4I9/fBZl2yhnTwNQ7hdZbD22dBybPzx4O6aA1zQwBEwRDBFMI/alkrg986K91cf4vTry/Ct3AbR9OcGeQT7+J7W4OwyBoXO+nuyvAwve+jNGwqDf2fgfP1504s1J3/etd+7t8lhnr5E3+2tJ3/udxfclfW7O8V9N+lyAxV3cO3vWV4ku/3vPvkE6wwrhGXNht6dFl/89lCBd7y+d/ajVPvYfEH066Culyt1llYg8S+zPlUoRKXGf8kuAqr7sg0aj0fQYEcQwD9ar7QRKE7YHA+WAr4v9B0Sfafoiki4imfF14ExgNTAPuNY97Vrg+b7qg0aj0ewf4v7Vuu/WS8wDrnFn8cwEGl0JfAkwSkSGi4gPuNw994Doyyf9YuBZ989ZD/BvpdTLIrIEeFJEbgC2A5f2YR80Go2m5/Tik76IPA6cAhSIyE7g/wAvgFLqfuBF4FxgE9AKXO8es0TkZuAVwAQeVkqtOdD+9Nmgr5QqAyZ3sr8WOK0n91pbFeHPpwzl6O+/xayrr+H9Y0/isolFXPZu7Jt23gW53PLVdfzol+dxzl8+4LVTgnz31TKuOH04r+efzMoXf8OQWedz51kjeH3M4wRtxRlfmsUHqeMxBU68YQYbimfy9AOv01pbztDjL+C200fjLHiIFfM2UB22mZaTypOrPyEaaCRvxGSmTi3hzJH5OB/+m60L1rCqMUzEUZT6vYwYmcugk6bw/JPraIw6ZHgMhqd7KZpYRMH0CaghE9neFGHpjga27GyiqaaeiYOmMCQ7hZTm3UTK1tCwYQcNW+ppqGihOmzTYjlEnJic5wnUoGp2YVVuJ7CrhtaqFlprgjSGLFosh4AdO9d21b+WqENDKEp9MEp9a4TaQITalgjhoEUkaBEJW0RDrdiRIL7MNLzpfsz0DIy0TIzUdMTnx/GkonxpKE8KEUsRsfdIi2KYiBnX9g3EMDG8PgxX6zc8PsQwiVgOlhXT7G071pQTCyQrR+Go2FIphRiCaQg+j4FpCKbhLkXc7T0tUc9v+5w5TpefJ1P2aO770vMN2VtS7UrPb/tZJPeR7jFGNzfu7nhP6av30V8QQMzeGfSVUld0c1wBnQZLlFIvEvtS6DX6OpCr0Wg0/Q8RjIOn6R9U9KCv0Wg0nXAQA7kHFT3oazQaTUcO7uydg4oe9DUajaYDgmB4vJ92N/qEfuGyGW5uwPfo8+z86DUWnmfy7LpqZn6wiBfufYhf/PwGXjvpSo7N9VNz3a/44IknWXDx9yjweZj28F+49d73ULbNHTccS82dt/LiribOKc2i5Fs/5XtPreSsoTkMvuUH/OCFtez6+A3SC0s59/SRzExrYM0DL7CkPkS212DarEE0bF+HNz2bQePHcPn0UgYFtrDrpYVsWF1NZdjCbwrjs3wMPn4Y6cedSmXYAqA4xUPpsBwGTB9J6sRZ1PvyWV7RzMfb6qmraCZQtZ3xhRkMSFXI7o2Etm6mYVM5TTubqA7bNFmxRCsAnyGYzVVuELeawO5aWioDtNYFaYw6bQFfOyETtSXiUBe0qAtFqWoKU9sSJhSMEglGiYStWIJUOIgVCeLLSsOblYaRnuW2TByfH8frR3lSiCqIOIqIo/YkZplmW9A2HsSVxCCuGVtGrHjwlljA1lHYthPL0HVUW3KWclRbENeTELD1mbFkrHhiVnx/Iu2DuXsnZoEbsDWk14OfcZJJzDoQjvQg60HBfdLvrvVH9JO+RqPRdEJ/HdS7Qw/6Go1G0xGRXpuyeaihB32NRqPpgHD4Pun3C02/dEgJp13//7jrnu9y1/Qb+O53T+bYHy6gZOrp3FD5HM+V1XPlvJ9yyS8WkpY/kOe3NXLtd0/hJysctrwzj0nnXcjVedXMv+dt8nwmJ/36Uh7doljz+tvM/r+5zK/L4sMFy7AjQUYcN4tbThxGw+N/5v3FO2mxHGbm+Rn3+Tk4VoT8kdM4bUYpc4ZlE1z0LFte28yGlgi2gmFpPkqnDaBkzkysoccQtBV5PpORGV5KppWQPWUK0ZIJbKoP8dG2esp3NNJcVUGkpZ7SLC9m/Xai2z6hfsMOGrc1UVcbbEvMiudC+QzBqdpOpGInLbuqaa5oobU2SF3EpjFqE3L19vj5pkB9MEpta4S6lgh1gQgN8cSssE00bGEFW7AjQZxoBF9WOmZ6Zpuer3zpKG8aeFNxPCmE3MSsiL1H0zdc7T5utJa4L67rG4bEkrISErNsK6bvx7X8Nm3fUe00+7jRmmlIe43f3deTxKw43Rmtxd08E+lJYlZXen5foBOz+gAxMD2+blt/RD/pazQaTUfk8H3S14O+RqPRdEDQ8/Q1Go3miOJwHfT7haafVb+LjOLhXPDSLwFYee2dbHzjWRb+5lzuueo+rjlpCPdY09j27ny+9e1LOas4Hc+37uaBB14ia/BoHrlxBsu+9l1WNIaYe+owAud+k98/voKWyq3w2e/xq2dWUbvpY/JHTuMrF4xjyK73WPHQO6xrDlPq93L0xePwnX4N6YWlDJ9YypXTBuHf+Dabn3+PldsbqYvY5PlMxg5Ip/SU8fimzmFTk8JnCKV+LwMnFjHguPF4xs9kV9jk44omVmyto66yhWD9biKBRnJUAGfHJzRv2EzDpkqadjaxOxSfox8T6H2GkOExsCq20Ly9ksDuBgKVAZobwzRGHUKOImjvMWaLX1PTGqG2NdI2Rz8ctAgHo0TDFtFQCDsSm6NvR4J4M9KRtCyMtEzEn4ny+VHeFByvn7DltOn5ccO1jkZr8e02Pd+ds2+YBo7tVuqyYgVTlFLYltPOaM1xFI4VadPvfR6zS6O1jvP0Y9q+07aeuHQS9PjEa3rLaC1OZ9e2Px5b7q/E3/Gyvso1OOLR8/Q1Go3mSELLOxqNRnPEICIY3v45O6c79KCv0Wg0HdGGaxqNRnNkoQf9T5HdlS1seuBKvpfxc+5d8wgFt/6FOTfeQODrnyNgO0x76SXOu+i3HHXKRdxeUo791B3M+csHNGxdzU0/upUhi+7np69t4djcVKbe9ROum7+Ore++QvaQcfzqjS1sfHsRHn8GU+ZM5uqjC9h86zd5p6weU4Tjx+Qx9OrLWB7MZMCEY/j8icMZ72+lav6zlC3azo5gFJ8hjM7wMWT2YHJPPIWGnKN4Z201BT6TEUVplEwfRvqUWQTzRrB6ayPvbqyhZlczgerthJvrUY6Np6aM1rI11G/YQcO2RiqbI9RH9wRxTYEMj0GWx6B1Z3ksMas8VjGrLhJL4EqsrgWxIG4skBuluilMXSBMcyBCOBQlErSIhq02ozUnGsGxokh6FkZmDpKeFQvielJQ3jQsDCK24zZFa9RuS8JKDGwZbtJKYhDX9BiYHgPbUm3JWY6jsC3VZrwWT8yKJ1p1NFXrmJiVmKC1d3JW10FcZdvtErO6QgSMHqYpHQ5GazouvAfjMI2S94vZOxqNRnMwERHE6L4lea+zRWS9iGwSkds7Of5dEVnuttUiYotInntsq4isco991BvvrV886Ws0Gs3BxjQP/JlYREzgXuAMYCewRETmKaXWxs9RSv0O+J17/gXAN5VSdQm3maOUqjngzrjoJ32NRqPpiNBbT/ozgE1KqTKlVAR4Api7j/OvAB7vhXfQJf1i0C/O9/PWyGO5/vThnPaKYKb4eel0uPeJtXzn3is4+f+9ixUK8Nz3T+Hls77Bf9JP4ONnn+GoUy7irlOLePHmR4k4inO/exqvyRgWPLsYgMlnzOKJ59fSWlvOkGPn8PPzxmM//wc+/O86docspuWkMvH6E2idfD4Pvr+N444dzHmjC7DfeZpN81ewojFM0FYMTPUwalwBpadPh7GzWbY7wKtrdjMyw8egY0sonDUVZ/hUNteH+XBbPZu3NtBYWUOovhIr1AJAZNNK6tdto25TLQ0VLewOWe00er9pkG4a5PlMmndU0VLRTKAqQGPIojHqELCdvYzWTHGTs1rCVDWHqY0brQUtImGLaKgVOxLEDgdxrAjKsTEycjDSMiElHcebhvKl4XhTCceTshxFxHYIW85eiVmG19em8ceTs0yPB9M0EEPajNaUo3BsV8t3E7TiiVnKsVG27Wr2RltiVmcaf/xYnO6M1pRtuz+b7o3WEvX8ZBOzEtnXL1Zvea91NuYciLHb4alg7x8xl81eGfQHATsStne6+/Z+TZE04GzgmYTdCnhVRJaKyE37927ao+UdjUaj2Yt9B/oTKOigtT+glHqg3Y32RnWyD+ACYHEHaWe2UqpcRIqABSLyiVJqUTId6wo96Gs0Gk1HXHknCWqUUtP3cXwnUJqwPRgo7+Lcy+kg7Silyt1llYg8S0wuOqBBv1/IOxqNRnOw6SV5ZwkwSkSGi4iP2MA+b6/XEskGTgaeT9iXLiKZ8XXgTGD1gb6vfjHoB4uH8n5dkIzHnufdxx5l3p9u5KHjbuAzY/N5afpXWPbs41xx89Wk3/tt5u9s4vu/f4WUzFwe+PpsNt1yI69VBbj4mBKyb/09tz+2lLqyFQyZcQb3fGYSFcteI2fY0XzhovFMjWxg6d0vsqQ+xIBUD9PPHkHOZ77Ifz+p4e33tnPDzKEUVy1n67OvsWZ9HbtDFtleg4l5foaeNpa0WeeyLZrO6xuq2byplqFj8imZNR7fpJPYTRYf7GzkvY011O6OzdGPBBoB8KRm0LJhPXUbymnc1sSuoEWT5bQrhp7hien5BSkeWnbW0FzRQkt9iLqITcB29jJaM0XwmwaphtFmtNYaiBAJRl2ztQhWsCU2R9+KzdG3oxEMt4CK8vldszV/m8FayF22Rm1aozZmQuGUNmM1j69t2/D4MFw93zSNmMlaQjF0225vvBafb68cu20OfrwYeuK8/MRtQyRpo7WOdGe01hN5PP56Ha/pqzn6h+kU8kMGETA90m3rDqWUBdwMvAKsA55USq0RkS+LyJcTTr0YeFUpFUjYVwy8IyIrgA+B/ymlXj7Q96blHY1Go+mE3qp2ppR6EXixw777O2w/AjzSYV8ZMLlXOpGAHvQ1Go2mAyJy2Gbk6kFfo9FoOiHZjNv+hh70NRqNphMO10G/XwRyt27bzU/e/C0n3fAnZl19Dfm/upGtrVFOfv8Vbv7RPxgy63zun27x1zsXMndoNlVrF3PRFy5h+tonePzJtUzOTuX4+3/MrfM/Yf3CWDWtr14+idHbF2J4fEw6bQZfmzGYsj/8P95cWQXASaPyGHXjlayVgTz8+mZ2r1nKcXk21c89wcaXy9jQEsYUGJ3hY/icoRSdfhpNReN5a2sdb6+upHrLTgbNHkHWcScSLBrDysoA72yspmpnE00VWwk11uBYEQyPj5TM3Fhi1sY6djeEqHEN1GwVS7Dym0KWx6AwxSS9OI3mihYClQHqIjaN0c6CuLFrUt0AcHVziMaWCKHWKGHXaC0exLXDQexICDuenJWehfL63ZZGVDyELYeQWzUrFHVojTpthmuJrTOjNcMN4poeI5acZTnYlmoL6nY0WosnULVVzOrCaK2tmlbC72XHIG4i8fsCbYHbzognZsXl3GQSszoGcfdltNZbiVmdoROzehGJfU66a/0R/aSv0Wg0HRAEw9Mvnol7jB70NRqNpiNy+For60Ffo9FoOqG3pmweavSLv1+8aZmc8W4+htfHwnPgDw9+zO33X8Xsuz8m3FjDSz85gxdPuB5ThDNfvIdRcy7mgbMH8L8b7qPFcrjkB2ewwD+Vef95C+XYHHPOiXxlQgYrfvonhs86g/938dE4//0t7zy+inLXaG3yTScTnH4x9ywqo+zjTwhU78Be9ATrn1nKxw0hgrai1O9l3MQihp5zHEw8lY8qArywsoKKLXW0VG6lePYxqJEz2FQfZnFZLRvK6qnftbud0ZovPRt/7gBq1ldTW9650VqWxyTPZ5Kdl0pmSQYtFS3UBaPURRw3Mau90Vq8eIrfNMjwCFVNYUKtscIp4WC0ndGaHQmhHBsnGtP08WfhpGS0M1oLJRitxROzwpbTzmitTc/vYLRmeGJNDLo1Wov3oS05yzS6NFrzmUZMVzWkS6O1eGJWop4fu3dyRmv786CXrNHagfziHW5Ga4fi2BozXOu+9Uf6vNsiYorIMhF5wd3OE5EFIrLRXeb2dR80Go2mR7jyTnetP3IwvqtuIZZ+HOd24HWl1CjgdXdbo9FoDiEEwzS6bf2RPu21iAwGzgMeStg9F3jUXX8UuKgv+6DRaDQ9RfST/n5zN3AbkCi6FiulKgDcZVFnF4rITSLykYh8VJwS4r1/PsY7D36Ju2bcxFUzB/HY2OtZ8dwT3PL9G5Cf3cALFc186cdn8ZuKgTzx3ZNY84XreK0qwOVzhuH5yp1896El1JWtYMTss7n30klU33MHL76xjZsvm8jE+qV8eOcLLKkPUur3MvPiMWRd+lUeX13FO4u3Ub91NabPz6bHX2bZulp2hyzyfCZTBmQw/OyJpB5/AZtCqby4tpLNG2pp2P4JocZqfFPnsMtO570dDby/sYaa8ia3GHrMLtuTmkFqbjGZRSU0lDWwK2i5xdDbG60VppgUpnlJL0onc3A2jXWhBD1/72Lo8YIrGR6DbK9JKBAlFIgQDkaJBINYwRaioRbXaC2CnaClJxqtxebnx0zWwpaiOWy3afqtUTtpozXTE1u2zdPfh9FavPnM9jp+Z0ZrplvgHHrfaM2Q5LTmrubx78to7VDS8z9tDuWu91aN3EONPhv0ReR8oEoptXR/rldKPaCUmq6Uml6Qn9/LvdNoNJquEaHzhMAOrT/Sl1M2ZwMXisi5QCqQJSL/BCpFpEQpVSEiJUBVH/ZBo9Fo9ov+Oqh3R5896Sulvq+UGqyUGkascMBCpdTVxAoIXOuedi0JRQM0Go3mUEDo/im/v34pfBrJWb8BnhSRG4DtwKWfQh80Go2mS0TAp20Y9h+l1JvAm+56LXBaT66vWb2eG554jNpLzwdg1Muvcu4F/8fE8y/jDv/H3H7/Eq6aOYi6637NXTf8ia9e0MzPXtjInMI0pj90Nxf8azmb3nqB/JHT+L/rjmHwkn/y1J8WUR6yuH1sGmu/8nteW1+LzxBOmTaAkV/7MotbMnn4lY+pWPUediRIwehjWfv6G2wORPAZwtFZKRx15lEUnnku1TkjWbC6kndX7aZmyxZaa8tRjk1jzlEs2dLAa2sr2b2tgaaKMkKNNbEEIZ+f1OwC0guHkFucQXlTmJqI1c5oLcNjkOs1KUwxyRyYQdbgTDIGFVIXWUtj1G6XxAV7krLiRmvZXgO/zyTUGmlLzGqrlhWNxIzWorFg7p5AbjrKm0ZEPK7JmkPYUu0CuK1Rm4BruGZ4fG4FrT1BXNMTM1iLG62JxHxMImHbNVdrb7TmWBGU3T6Q25XRms9jtBmted0ErX0FcTsmZnVFotFasg9wHe+XjNHaoTaM9ORZtbcNxg7pIK6Ap58+yXeHtmHQaDSaDgiHr6avB32NRqPpiPRfzb47DrW/NjUajeZTJ/akb3TbkrqXyNkisl5ENonIXg4EInKKiDSKyHK3/TjZa/eHfvGkbyv4TeNT/ODt7fxx9SMc9e0XSC8s5d3vzeL+gTMYl5nCcS89y8Q7FtJaW87fb1tArtfkwgdu5J7tGbz79FP40rO57MqT+UxWFW/d/ncW1waZnJ1K7V9+yoIXNlEXsTm/JJOp35zLjtLZ3Pn0KrZ89DGhxmoyS45ixLSxfPxsiIijODorhTEzB1F6wWlY409l0cZ65i3dRUVZFS2VW7EjQTypGayqauWNDdWUba6jsXwXrbXl2JEgYpj40rNJLxxCTmE6gwZmsju0p3AKtNfzs4vSyRqcSdaQIjKHFFMXsQm4SVkdjdb8blJWhscgy2viz00lHLQIh2J6vh0Juss9hVPiDcBJycD2pBKKxrT8sK0IWU6Cnu8QthyCETum4SeYrBke356iKXGzNVPa9H3HTcyyLQfHdrAtq61wSsfkrLjRWsekLFMEbzwjMqGISneFUxKPd2W0Jh00eGM/rMg6S5TqLe3600zM6q8FQw6E3njSFxETuBc4A9gJLBGReUqptR1OfVspdf5+Xtsj+sWgr9FoNAcTQ6S3Zu/MADYppcoAROQJYlY0yQzcB3Jtl2h5R6PRaDrBFOm2AQVxuxi33dThNoOAHQnbO919HZklIitE5CURmdDDa3uEftLXaDSaDsRtGJKgRik1fV+36mSf6rD9MTBUKdXiOhg8B4xK8toe0y+e9AdMGMEPbvwnP/rleZz2ilC5ahHz/3ANi48/g/JQlOte+BlnPfIJW96Zx3GXX8bW1ijXfGM2K6Zey+/ve51gfSVTLziHO88awerbvs+L62oYkOrhjKsnsegPb7ChJcK0nFSO+fpJcO7N3P32Vla+vY6mnRtIzS5k8KSpfP6UETRGHUr9XiaNzWfUJbMwZlzAkopWnlu+i+3ra2jcvpZwcx2Gx0dawUDeKqtl+YYaanfV0FK5lWigEYgVTknLH0h2cQFFg7KYNjTXNVqLF04RsjwxPT8/N5WMgRlkDs4lc0gxvkFDabJihVPic/T36PlCuhmbn5/tNUjJ9pGam0ooECHSGsAKtRAN7jFaSyxaEkd5/YSsmG4fsmMF0VsiFi0Rm6Cr67eELVpC1p75+e4cfdPjiVnOuoVTTI+0m6/vKHdufkLhlM6a487T38twLaFwSnyufkenw84Kp3Sku8IpB2K0lngf6LpwSk+1eG20dvDppYzcnUBpwvZgoDzxBKVUk1KqxV1/EfCKSEEy1+4P+klfo9FoOtCLyVlLgFEiMhzYRcyS5sr2ryUDgEqllBKRGcSeD2qBhu6u3R/0oK/RaDQdEHonkKuUskTkZuAVwAQeVkqtEZEvu8fvBz4LfEVELCAIXK6UUkCn1x5on/Sgr9FoNB3ogabfLa5k82KHffcnrP8Z+HOy1x4oetDXaDSaDhzONgz9IpC7tjrKlceX8uRJ3+bdxx7ltp99nZxf38iTq6r41i/O4zetk3nvX/9mxElzefnLM7jq1GGk//Av3HDPYqo/eZ9Rc+by92uPoebOW3l+/kZspTj35CEM/94dLKpppdTv5eRLx1Nww3d5eHkFL762iZoNSzB9forGH8cFJw/n4rEF5PlMjinJYNRFU0k/9TNssrJ4ekU5q1ZXUbdlLcH6SsQw8ecWk1M6moWrd1O1vYHm8k3tqmX58zPki0oAACAASURBVAeSNWAw+SUZTBuay8SSrHbVsrLdpKzCNC9ZgzPJHpJD1rASUktL8Q4cllS1rLSsFFJzUvHnprarlmVHgm1Gax2DuAAhWxG0FKEuqmUFIrEgbmvE7rRa1p7A7d5JWonJWW1B22hkryCucuxOE7MSq2XFE7S8hnRqtJZIx/eYTLWsjsla+7rfnnu0N1rrrSDuvl7rYHAkGa21oYuoaDQazZFD3E//cEQP+hqNRtMJetDXaDSaIwTjMC6i0i/eVaipgZyn/sft3/o9s66+htvqnubuv37Ely8Zw6qLf8zvf/0oeSMm8/wP57DxC59h2r8f5ZK/fsCmt+YxYPIc7v7ScRQvuIf597xNecji3FF5TP35rbzUUkSGx+CsU4Yw6rbbeKkmlYfmr6N8xSKUY1Mw+lhOOGEY1x4zmLytizk2N5XRF46jaO6l7Mo8innrKlm8vJzKjesJVO+IGYVl5pE1eAwDhuaye2sDDds/IVhf2VY4xZ9bTGbxUAoGZTFxWB6TB2czpiANW8X1fIMCn8mAVE9Mzx+cRdbwEtKHDMJbMgyVO7DTwil79HyDdL8Hf25Mz0/NTd2j54eDOFZ0r8Ip7X7WtiLcoXBKc2RP4ZSWkEUwEkvQ6qxwisdrtun6bUlartZv284+C6c4TvsiKomJWV7DaFc4JW64Ftebky2ckrjdVeGU/dHz267t5Lre1vN7+voHdr8jUM8HrelrNBrNkYTQ5q1z2KEHfY1Go+mEw9VOWg/6Go1G0wGBtloNhxv9QtMfXDqAE6+/m6Ezz2ThOfCzax/mwpF55D34DFff/m/EMLj3jotIv/fbPPzUOr7wShVL//ssWYNH88OvnMRJVW/w6jcfZ0VjiNOL0jnhzmtZM/AkfvrkCs6eUMikH36Jpb4x3DlvLVs/fJdooJHcYUczftYovn7iCEYENlL+xOOMPecoBn/2IhpKZ/DSxlrmf7CDig3baNm9FceK4E3PJmvQaAYMK+T48UXUbd9MsL4Sx4pgeHykZheQMWA4eSWZjBqSw7ShOUwoymBQhrddIfQBqR6yBmWSMyybrOEDyBpWgmfgcKRgMHZmcVvhlESTtbien53qIdXV8tMK0kjNz27T87sqnBJHDJNg1CHk6vktEZuWiLXHaC0Um6PfHLYIRqx2er7Ha7Zp94Ype7T8hDn7ylHYltWm5zv76Eu7efoJ5mqGCF4zYc6+IT3W8zsWTkmcV9/RfK2z67uis0Lo7X6+vfTk2NV9DnU9v18R/7x10/oj+klfo9FoOiCAN8lyiP0NPehrNBpNBw5neUcP+hqNRtMR6b/yTXfoQV+j0Wg6IBy+MY1+IVrlNFaQml3Iyp8cx10zbmJcZgqnfPwGc37wCk3lm/nhHddxxooH+eudCxmY6mXew//F68/g+i+ey40Flbz1xTt5pTLAtJxUTv/5XKpmf4FbnljOhrfeZOZPr2L76HP4/rw1fLLoPVpry8ksOYpRMyfxrdNGMdlTTc1Tf2ftk8sZ/rnzsaZdyIKyep54bxs7PtlFw451WKEWPKkZZJUcRfGIQUwbX8ScUQUEqndghVoQw4wFcYuHUzAojxFDczhuRB6Ti7MozfSS2rC9LYg7yO8hrzidnKFZZA8rJvuoQXgHj8QcMBw7u4SApAKJ1bL2BHFzfbGkrLSCNNLy/aTmZ5Kan4UVbGkL4joJiVmdEbYVgYhNYzgWsG2J2DS7Jmtxo7VgxG4zXPP4vHsZqyVWyzI9gmEaeDwGtmXFgrZ259Wy2rZte4/RWjtztViCVjyIG0/UirM/SVkd97WtH8Dve1dGa4ns7/37cxC3v42hMXO/fbf+iH7S12g0mg6I+1BxOKIHfY1Go+nA4Szv6EFfo9FoOqG/yjfd0S/+fqnY3cyqh67lP6PmAHD1iqeZ8YvF7FzyMld/8wt8w36XP9/0D0wRbrzrs1iRIOdddzG/PDaV9679Ns+tr2V0ho8LbjsN+4ofcfMzq1i1YBHB+t00nHQDP/zfOla/sZTmis2kF5YycuYMbj17DHMKLZqf+xtr/vkBH5Y3I7Mv47UtDTz23ja2rK6gYetqooFGTJ+fjAHDKDpqBBPHFXHm2CKmlmQQDTS6en4hGcXDyS8tonRoDsePKmBqSRbDcnykBypRO9a5SVkm+YXp5I7IIXt4EdkjB+EbPALPwBHY2QNo9WRQG7QxhTYtP8tjkOczyfOZpOam4i/wk1bgx1+QSWp+NmlFudiREFYkuE89XwwTMUwCEYfmyB49v11SVsiiJWzRHIoSjNiYHk87YzWPt73pmsdrtBVW8XmMdkVTEhOzOur5ccM1X4K5WlzP95h7dP24tg/J6/mwdwJWV3p+4u98d4lZbT/HJAqnHOp6fl/Q3x6ahT2GfvtqSd1L5GwRWS8im0Tk9k6OXyUiK932rohMTji2VURWichyEfmoN96bftLXaDSajvRSjVwRMYF7gTOAncASEZmnlFqbcNoW4GSlVL2InAM8AByXcHyOUqrmgDvjogd9jUaj6UBM0++VW80ANimlygBE5AlgLtA26Cul3k04/31gcK+8chf0C3lHo9FoDiZxG4buGlAgIh8ltJs63GoQsCNhe6e7rytuAF5K2FbAqyKytJN77xf94km/KDeV98fPZHMgyh1LH+KEx3az7pWnOesrN3LfmCoePPlX1Edtbv3pOWw8+7ucYK3lkQuHsuLKK/jPezsZmOrlkq/OIvvW3/OlZ1bz3vy3aKncSsHoY/nRyxt4+6Wl1JWtwJ87gBHHzeKr543l/KGphJ65m1WPLOK9TfWUhyze3h3l0fe3sWHlburLVhBqrMbw+Fw9fzRjxxZw9oRijh2YSUFrOQApmXmkF5aSO2gAA4fkMHtUAdNKshmRk0JWqAbZuZbQppUMSPUwoDAtNj9/eAG5o0tJHXoU3iGjsbIHEkzJpabVYndLpJ3RWrY31tLyYlp+WkEa/vwM/IW5pBXl4s3JwbF2tytA3pG4nm94fLREYlp+MBozW2sJt9fzg5FYEZVgyMLjNV0t34yZqiXMzzdMadPz/T4Tn8fYqwh6V3q+cuw2Pd9rdq7nexPW96Xnd0XHQuiJ++DI1vOP2MIpiQgkOWOzRik1fd932gvVyT5EZA6xQf+EhN2zlVLlIlIELBCRT5RSi5LqWRf02ZO+iKSKyIciskJE1ojIT939eSKyQEQ2usvcvuqDRqPR7A/xKZu9EMjdCZQmbA8Gyvd6PZFJwEPAXKVUbXy/UqrcXVYBzxKTiw6IvpR3wsCpSqnJwBTgbBGZCdwOvK6UGgW87m5rNBrNIYS4lt77bkmwBBglIsNFxAdcDsxr90oiQ4D/Ap9XSm1I2J8uIpnxdeBMYPWBvrM+k3eUUgpocTe9blPEghinuPsfBd4EvtdX/dBoNJqe0lvJWUopS0RuBl4BTOBhpdQaEfmye/x+4MdAPnCfK+NZrmRUDDzr7vMA/1ZKvXygfepTTd+drrQUGAncq5T6QESKlVIVAEqpCler6uzam4CbAErSUiG9L3uq0Wg0e4jZMPROMEIp9SLwYod99yesfxH4YifXlQGTO+4/UPp09o5SylZKTSGmY80QkaN7cO0DSqnpSqnp6cNHs6iyhR8svJPTXhGWPvUvZl97Hc+fZvLPU29hQ0uYr337ZOqu+zVX/OYN5l07iXU3Xcu/Xi0jz2fyuS9MpeSOP/Lt/63nlWcW0bRzA3kjJnPKedN5Zf7H1GxYQmp2IcNnnsBN54/j8rE5WP+7j1V/e4N3V1ezIxglw2Pw8HtbWbm0nJoNHxOs350QxB3LmPGFzJ08kONLsymOVGKtWkRKZh4ZxcPIKy1l4LBYEPeYQdmMzEslJ1qP7FxLeMMy6lZvoSTfT+7wHHJHFZI7egipw2JBXDt7IOG0fGqDFlWBCDsag25SlkmeL5aYlZGbSlqBn/TidNKLMvEX5uLPz8aXn4eZW4QVDrYlRHUkMYgrpklj2E3Aiti0hC0aW6PtgrjNIYtwxMaK2u2CuPHKWR6fiWFKW4JWvPpVipuclZiY1VUQF2gL4iZWzeosiNvdXOpOA9cdgrh7ma+5S0Mk6SBuIr0dxO3ydXQQt08R6b71Rw7KlE2lVAMxGedsoFJESgDcZdXB6INGo9H0BAPptvVH+nL2TqGI5LjrfuB04BNiQYxr3dOuBZ7vqz5oNBrN/iAcvk/6fanplwCPurq+ATyplHpBRN4DnhSRG4DtwKV92AeNRqPZL/qDp9H+0Jezd1YCUzvZXwuc1pN7lW3dzc9evYtzlhTx7mN/Z9bV1/DaRVn8a/oVrGgM8Y1bT6D11nu45BcL2f7eC2z44kP849n1ZHtNrrpuCqW/foBvv7KNZx9/k4atq8kZdjQnXzCLX583jlF/+AspmXkMn3kyX7pwPNdOLMCe/0eW3/sK7y6vZGtrFL8pTM5OYf6SXVSvX0prbXmbnl84cjyjxhdy0ZRBzB6SQ0m0Gmf1ImoWv09G8WTySksZMCyHk8YUMmtoLuMK0si36jF2rSWyYRm1KzdTu24XuSNien7e2GH4jxqFb9hY7NxSwumFbUlZ2xtDbG8IkuUxyfbu0fPTi9Jjmr6r56cV5ZJSVICZW4SZW5S0nm96fG16flMo2k7PbwlF2/T8SNjCijpJ6fl+n0mKx8DnMZPW85Vjt+n5ewqoSKd6vjfhN7M7o7X4vq70fEPa6/n7Q7J6/oGOJ1rP72P68ZN8dyQl74jIJW4yVaOINIlIs4g09XXnNBqN5tNAem+e/iFHsk/6vwUuUEqt68vOaDQazaHCkS7vVOoBX6PRHEkcpmN+0oP+RyLyH+A5YvYKACil/tsnvdJoNJpPkcO5XGKyUzazgFZi3g8XuO38vupURzz+DM5eXsrbf48Fcd+4OIN/HHMFHzeEuOXbJxH8zr1c8LPX2fLOPIbMOp9Hn/6EDI/BVddNYchvH+LWV7bx9L/foK5sBXkjJjNn7gn87sLxFC/9DymZeYw4/lS+evEErp9UiDP/jyz704u8vWw3mwMR/KYwLSeVSacOo3LdR3sFccdPLOYz0wZz0tAcBlrVOKvepPrtd9n17ibyhw5jwLAc5owr2iuIG177ITXLN1C7bhe1G+vJH1O0VxA3lF5IVatFRUuErfVBtta1UlYdIM9nUJiyJ4ibXpxORkl2p0FcI7sg6SCu4fEmHcR1LKdHQVyfx0g6iKscJ+kgbvwXM9kgLiQfxO3p77wO4h5eHNFTNpVS1/d1RzQajeZQ4nAtNpLs7J3BIvKsiFSJSKWIPCMifVrdRaPRaD4txC2X2F3rjyT7ZfZ3Ypm0A4lVfZnv7tNoNJrDksNV3kl20C9USv1dKWW57RGgsA/71Y6jB2ey+NFHmHPjDSw8Bx465mpWN4X5zh1n0vCNezjvx6+y7d35jDhpLo/fPocsj8E1X55B6V3/4Evzy3j6H69SV7aC/JHTOPszJ3LX3AkUvvsoS/7vYUaeeBq3fPZorhufTfTp3/HRXfN58+PdbG2Nmawdm+tnyhnDGX3lmW16fubAoygeNYFJkwdw6TGDmTMsh0GRCuxlC6h68x12vL2B8lVVDBqRy+kTijlhWB7jC9MoiNZi7FhNaPX7VC/fSPXqndSsr6WqKkD+hBGkjRqDb8QErLyhhNILqW612NUU0/O3uHr+tppAu6SsRJO19JL8PXp+/gAkpwjHn73Xz7MrPd/w+JLS861ozHCtJ3q+zzSS1vOBpPV80+iZnh8nrucb0jt6fruf76eo5+/P/bWevzdCbHDsrvVHku13jYhcLSKm264Garu9SqPRaPopItJt648kO+h/AbgM2A1UAJ9192k0Gs3hR8Jfgftq/ZFkZ+9sBy7s475oNBrNIYEAvVRD5ZBjn4O+iNymlPqtiPyJTiq4K6W+0Wc9S6Bu1XqufOxhHijdwF0zfkyTZfP9P3yGZWfdxvW3P0fl6kWMO+uz/OebJzDg+d8w8PbT8H/rD1z+rxUsevpVWiq3UjR+Nhddciw/O3MkqS//mfd/9QwL19Xw/b9O5qJSg8A/f8OyvyzknY11lIcssr0xPX/CuUcx/HPnY8y6BPPX/+fq+WOYMqmYi9yiKYWB7USXLaTy7Q8pf38L5Z/UsqklwlkTBzCzNIdReX5ygpWwfRXBdR9Ts3IztWvLqd1UT1VdkN0hG//IsXiHjsXKHUyrL4eagMWupjDbG0NsqQ2wrbaVbTUBWhpCZOankV6cRkZxOmlFWW16vi8/HyOnCDO3EMkqwPFnozpo+ol6vun1ueteTJ8fw+ujsTVKQ2s0ZrwWihKM2ARDlqvju3p+xMaxFf4MH6bHwOM1MEwD09Xy40VTfB4ztm3GtpPV85Vj40nQ8L3t1mOeJ3E9P1GP7qrgyb70fGiv5++Zw79/aD3/8KG/yjfd0d1nO2698BGxsocdm0aj0Rx2xDJye0feEZGzRWS9iGwSkds7OS4i8kf3+EoRmZbstfvDPp/0lVLz3dVWpdRTHTqqffA1Gs1hS28857v1RO4FzgB2AktEZJ5Sam3CaecAo9x2HPAX4Lgkr+0xyf4V+/0k92k0Gs1hQExC7K4lwQxgk1KqTCkVAZ4A5nY4Zy7wmIrxPpDjlpJN5toe052mfw5wLjBIRP6YcCgLsA70xTUajeaQJPnkqwIR+Shh+wGl1AMJ24OAHQnbO4k9zdPNOYOSvLbHdDd7p5yYnn8h7TX8ZuCbB/riyRJxFH9WL/DzMx4ly2Pygydv4fGBF3H7bf+gaecGjrn0Kp792kzsu7/Fg797g8u2fcwlDy1h2fxXCDVWM+jYc7n+0oncdsIQQv/4OYvufInXtjfSYjlcUhig9sE/suyv77B4VxPVYZvCFJPj8tIYe8k4Sj87FzXjIt4pbyVnyDgGjh3JjMklXHj0AGYMyiSndgPhpa9Rsegjdr2/nZ1lDWxqiVATsbl2WD5H5frIaNqBs2UFrWuWU7umjJq1ldSXNbC7IcTukEV91MYz/Gis3MG0mBluUlaYrQ1BttW2UlbdQnldkJaGEIGmMJkDM0gvSiOtKBt/US7pA/Lx5sdN1gohIx/Hn43jz8b2prX9HNsSsgwT0+vDSEjKMrw+PD5/uyBuS8giErE7DeJaEbtdENfnBnB9HoM0n9kuKSvF3d9ZEHdPIHdPEFc5NqaA1zRiAdt9BHHjhSySDeICSQdxexrI60kQt6fT/foiiKvpGlEK6eIz1YEapdT0fd2qk30dJ8V0dU4y1/aY7jT9FcAKEfmXUko/2Ws0miMGUU5v3GYnUJqwPZjYw3Qy5/iSuLbH7FPTF5En3dVlblQ53laJyMoDfXGNRqM5NFGgnO5b9ywBRonIcBHxAZcT8zFLZB5wjTuLZybQqJSqSPLaHtOdvHOLuzxo3vkajUZzSKAOWElBKWWJyM3AK4AJPKyUWiMiX3aP3w+8SCx2uolY3ZLr93XtgfapO3mnwl2tAYJKKUdERgNjgZcO9MWTpWTCcH5wzd+Zmefnc2/dx3c3FvC32+7HsSKc/aXr+M/l4yj7+hX8+/E1MZ3+7ndY99qLKMdm5MkXcttVU7hyiKLqt7fy3n3vsKimFYDZ+X52/P5nLP/nxyyuDdJiOZT6vcwYmsXYz0yh5DOXEhh9MgvLGnjiox0MnTyWOVMHcu64Yo4pSSd1x1IC7y9g16LllH9YzpadTewIWtRFbCKOYlxBKik1G7E2LqN59QpqV2+hdn0N9WUN7GqJUB22qY/aBG0Hq2AEjY6X6haL7Y1BtjeG2FoToKy6hcr6IIGmMK2NYVpbwmSWZOAvyiF9QB7+oly8BcUYbtEU0nNwUrNxUrOImim0Ruy2hKx466jnmyl+13TNR2MwQnPIIhixCYctrMgegzXbctoKqNi2g8dr4PGaeNpp+Uanen6bpt+Fnp+YpAXt9fzYOp3q+YZIj/R8aK/ndzRY2189v7P7x19jX8d7A63n9wFKJfskn8St1IvEBvbEffcnrCvga8lee6AkO2VzEZAqIoOA14l9Ez3Smx3RaDSaQwlRTretP5LsoC9KqVbgEuBPSqmLgfF91y2NRqP5NFHgWN23fkjSg76IzAKuAv7n7ku2qLpGo9H0LxS9Fcg95Eh24L6VWAbus24QYgTwRt91qz3ram3+39QBHP/6fE57eA3v//vPZAwYxjdvvYTbR7Tw7hnn8tSH5eT5TL5w6Tjue/EZUrMLGXfqqdx5xRROpIwN3/8Vbzz9CSsaQ2R7DU4sSGfyF2bw6n2LWdEYxlaKcZkpTJ9UxJjLZpB7wVVUZI/m5TVVPPHhDraureJLn5vE2aMLGZ2hMNe+Tt3ihex6Zy0VS3ezqaaV8pBFY9TGVuAzhNTylYTXLaFh1VpqV2+lblM9tdsa2RW0qInYNEZj2r+toMbyUt0aZUt9kO2NQcqqAmyrDVBbH6S1KUygKUwoECLSXEfGiALSSvLwF+a2FUwxc2MFU5yUTJQ/mxAeWiMOgaizx2TN62tXMMVwtX2Pz9+m7Te0xkzWovGCKREb23ZcTV9hRe0ETd8kpZ3BmoHf58FnGu32+TwGpiE40ViB9u70fMexY/Py3ZJ0iXp+x7n6XdGVng9dF0zpqOcf6Fz6A52bnwxaz+8rFDj9c1DvjmStld8C3hKRTBHJUEqVAQfFYVOj0Wg+DfqrZt8dyRZGnygiy4DVwFoRWSoiE/q2axqNRvMpcoTLO38FvqWUegNARE4BHgSO76N+aTQazaeHUpCcDUO/I9lBPz0+4AMopd4UkfQ+6pNGo9F86hyu8k6yg36ZiNwB/MPdvhrY0jdd2ptgYz0lyz9i4o8WsuWdeQyZdT4PfOtEZn3yJM/OvI/XqgJMzk5l7nfmkPudP5B9xX2ceMFs7po7gQHLnuKDXz3Cgvd2UR6yGJjq4ZRJRUy+6VTSLryJJb88Cb8pHJvrZ+LJQxh9+Wl4T76MT+w8nlm6i5eW7GTXhnIat6/jsqPPZKBVjfPBm1Qufp9d721k94oq1jdHqAxbtFixD4nfFAp8HkIfvU7N8g3UrttF7cZ6qqoCbQZrjVGHiBPL+DMFtjWGYglZda2UVQfYVhOgqSFEa1OY1uYw4UAL0UAj0VALmUOKSSmKG6wVYWQXtBmsOSmZtFqK1qhNwHIIRp2YyZpp7hXEbQvgulWzPL4UWkMWETeI61hOm9ma7QZv40Fc23JI8cUqY6V0SMjqGMRNTM6CvatkJS6deHKWG8T1Gp0nZMW3O+ZQ7SuAm0hvB3E70tdBXB3A7Wt6LznrUKMnhdELgf+6rQA3VVij0WgOS45ETV9EUoEvAyOBVcC3lVLRg9ExjUaj+dToRRuGQ43u5J1HgSjwNrGSXuOIzdnXaDSawxbhyNX0xyulJgKIyN+AD/u+S3szcPAAjr/uT7TWlnP8Ndfy3I3HUv+TL3HXfe9THopy8ag8Tv7z1yibdClX/uUDbv/WXL42JZ+mh+7gtbte543dLQRth2k5qcw6ewSjb7ycyMxL+fe6GgakejiuOJ0xF09g8GWfwZ56Hgu3NfHkss18tLyCyo0baSrfjBVqobTxE0JLFlD+9jJ2vr+DHVsb2BKIUuMarJkCGR6D4hQPg/weyt9eRvXaShrKGihvCrM7ZNNk2bRYDrZr4GcK+E2DtVUtbK1tZVttgJ01rbQ0hgg2Rwi2hAk3NxBpbcQKtmBHQqSWlmLmFroGa7nYftdgzeOnNeIQtBSBqEMgYtMYtrosmBJPyIolaHkxTYNw0IolYtmxxKxEgzXbcnBsB9uyUI6N32d2WTDF1yExy2cadFUwJU5cz1e23WXBlI56vpGgbvdEz9+XwVqbIdt+CufJ6PkHYuim9fyDgQL78Jy9052m3ybl9LSIioiUisgbIrJORNaIyC3u/jwRWSAiG91l7n70W6PRaPqOw9iGobtBf7KINLmtGZgUXxeRpm6utYjFAMYBM4Gvich44HbgdaXUKGKOnbcf6JvQaDSa3uZwddnszk/f3N8bu178Fe56s4isI1body5winvao8CbwPf293U0Go2m9zlyA7m9gogMA6YCHwDF8eIsSqkKESnq4pqbgJsABmVn4D06g9/d/R2+kr2Nt44/hWdWV1Gc4uHrX5jCUb/4PQ9v83DXLxey/cMFvH7Wpaz78jd4ff4m1jWHyfOZnD4kl0nXz6T46i+x0T+CBxZsZsHibTwwaxDjLp9N5lmfY2fGUby4fDdPvr+d7Z9UU1e2ktbacpRj40nNoPb5f7UZrG2uD7EjGG3T532GkOczKU7xMCTTR+6IHHa+v4OaHU3sDu0xWAvae6rx+Awhw2OQ5TFYvqORbbUB6utDBJpCBFsibQZrkdZG7HAQKxTAjkbwFJfuMVjzZ6NSMgkqk6BrsBa0HBqCFi0RK6bp+1K7NFgzPD48XtMtcm66RmtxTX/vufnKsXGsCE40Qmaqd59z801D8HkMvIaBKe21/MSlk6DFK1dHNV1ztc60/NjnI6bniySv5cfPS2Zu/v5I7n2t5Xf2GgeTA+x6/+MwHfSTnae/34hIBvAMcKtSqjtJqA2l1ANKqelKqen56f6+66BGo9F0JG7D0F3rh/TpoC8iXmID/r+UUv91d1eKSIl7vASo6ss+aDQaTc9RKCvabTtQkpnY0tWkGPfYT0Rkl4gsd9u53b1mnw36Evs79m/AOqXUXQmH5gHXuuvXAs/3VR80Go1mv1AcrCf9ZCa2dDUpJs4flFJT3NZtPd2+fNKfDXweOLXDt9BvgDNEZCNwhrut0Wg0hwwKhbLtblsvMJfYhBbc5UV79UWpCqXUx+56MxCfFLNf9FkgVyn1Dl3HnU7ryb3KyxtZ9fAXse/+Fnf97g02ByJcMDiLU/90Pbtmf5Fz/rOC5a+8Q9PODWSWHMWCC77Ja9sbabEcjs5KYfacoYz78mdRp1zDU+tr+etzy9m8fBu1mz5m2i9uQs24iLfKW3nqjc28Tw9/AgAAHh5JREFUv6yc3Rs201SxmWigETFM0vIHklkyknVPPMDOsgY2tUTaJWRlew0KfLGErAElGfz/9s4+OK76vPef55zdlXYlWe+y/IItY/xaSAgQEkpCSAMJoQUnmUDhpmnmTlrauc3MTZM0pZd789Kkd2huS3LnNk0LNGk6bSFvJa8MCYYADU1CscHGxnb8Jtv4VZLttbTat3PO7/5xzq52V7valS1ptdbzmTmz5/z2vPx+fnl09H3eutZ00rV2KU8/+It8gbVyCVk5J25XxObJo3HGzqb8DlnjGdKj58iOx8kk4riZFE4miZfN4DkZrN4VfkJWtB03HCOR9RgPHLijaZfRjEM85TCWcYmnswUF1aJBkpbvxM0lZIXCNlbIIhS2yKQdPNfkO2aVJmR52UzemRuN2FUTssKWYFlC2PLfL6ZKyMr/2/Hcik7cQgcu1F7IrPCZtSZkXcgb0cWWkLXwnLjU2jmrR0ReLDh+0Bjz4DSeVFNgS46SoJgcHxGR3wVexP+N4MxU99A+t4qiKJOouZ7+sDHmmqlOEJHNQH+Zr+6bzowqBMV8Bfgc/o+pzwF/jV8gsyJq9BVFUUoxZkYctf6tzE2VvhORkyKyJHjLrxjYUiEoBmPMyYJzHgJ+WG0+sx6yqSiK0niYvBQ51TYDVA1smSIoJhcBmeO9+C1tp6Qh3vR7O5p5+aq38N0DZ1jdEuGTH/11ln/qAR54eZSH/tdPOLrlSaxQhEtv2MQHb9/Ad296iN4mm3dd1s2V99xA55338Kos5Ss/3MNz/3GY46++RGLoCACHN97OD7ac4HsvHOHQrpOcHXyF5JmTvq7c0k7b4gE6lg+wZFUnL31vhGOpLPHsRLOUzrBNf3OIS9qb6FrTRddl3XRuWEnrZZex/4vPMeZ4RQlZUVuI2hNaflfEpq29iZEToyRHM6QS42QT8aICa26g5ef+obnt/XhNbaQ8IZFy83p+POUnY40Fmn4i4xAfzxKOthZp+bmErFDE9jX9iJXX9hPn0pMSsnLPzun5eU0/bFfU88OWlS+altP1C/+jlEvIggntPWxZZZul5PR8S2rTmSv9xyxNyJqvWv70nz+zz1pwWn6OXPTO7HM/8E0R+TBwGLgDQESWAg8bY25lIijmFRF5ObjufwSROl8QkSuDGQ8Cf1DtgQ1h9BVFUeYWU6sj98KeYswIZQJbjDHHgFuD/YpBMcaYD073mWr0FUVRSjHMVEjmvEONvqIoyiRqjt5pOBrC6DvLV7F5d5wPvX0l1/7NZ9lsb+SOB7byq2c3k0nE6V3/Zq6/+Qo+++71rBnZyrd7Y1x95+UM/P7vcXLF9Xxx+3G+/ewLHNr2KvHXfoWbSdLc3kvHwOX8yfd3smfnKYb2vUri1BHcTBI7EiXWvZRFy9fRP9DJlWt7eOvqbp4fS+Magtj8oLhaLET3ynZ61nXTsXY5HetWER5YjyxZzenMX+Zj8yOWELWFFntCy+9sCRPtidHaFyM+PJ4vrpbT8p10skjLz5Fuag9i812SWZOPy8/p+aNpX8sfS/n7oeZWrLDfAD1XWM3X8m1CYSuI0ffHnOx4UWy+52Qwrls0D+O5uE4maKBSoucHGn7Y9jX5XLx9OND0q2n5OSrF5hdq8IXx+qVM5WQTkbLF1aySc6bLdPT8mW6Urlr+DDOD0TvzjYYw+oqiKHOLvukriqIsHOYuemfOUaOvKIpSgsHk+z9cbKjRVxRFKUXf9OvL/sETfP5Hf8GB193BOx55ie0//jvGTg7SvmID17zvdj5z20aubzrJyb/7E554+Oe859F7ybz5Dv5l1zBf/ep/cuDlQU4f3EY2ESfc0k7nwOUsXX8p11+5lG/889OcO7YfJzWGFYrkHbh9K/tYu7qLt63r403L21nd2cTzTBRXWxEL0beszU/IWruUzg0riQysx162FrdzOeesWN7pmyuu1hm26YpYdEZCtCyOEeuJ0dIXI9bXTmLw8IQDt6C4WjmH5OmkSyLrkci4xNO+s/Zc2vEduoED92wyy1gqy3jGJRRtLVtcLZ+cFbaxQ4JlW2TTTtniavmkrAJnbmtzqGJxNVsgZPufOadupeJqheSTs+zyxdVyY0CRY7fcPSpRLSFrJpKpGtWBC+rEBXxHbjZT71nMCg1h9BVFUeaWuUnOqgdq9BVFUcqh8o6iKMoCwZiZKqg272gIox9qbmHTvvVs+X9/m2+Ucu1dH+S+TRu5qWOM0//0WZ56+Hl+djjOUNol0XMTf//wi+zdOsjIvq1kE3FCza10X3YV/WtX88bXL+H2K5Zw3fI2vvLnXyxqlNKzYjFr13Rz4/o+rl3WwerOCK2jR/FeeomBWGRSo5TODStpWrUee7mv5cftVoaSDkfPJWgNFTdK6WkKEeuJ0rK4Ja/lR/s6aenvJr1tuKqWL5aNFYowPO4QT2fzjVLGMg7xZJb4eJbRlMNY2mE05Wv7mYxLU7SpSMvPJ2gFx5ZtEQkSrZxMuqqWb1z/M9dEpZqWb4uvPdei5eewpTYtX6a4RyVq0fLPV3tXLf/iQaN3FEVRFgrGYFw1+oqiKAsCYwxe1qn3NGYFNfqKoiilGPRNv55cfskifvrQP7Bo+Vp+/Xc/xKdu28gN0WFOfe0zbH74P/j342Oczrj0NtnctnwRf/h/fpKPyw81t9Kz9o0sWbuK665cym2X9/PGpa20D+8m/cTmfFx+7yW9rFvTnY/LX9XRREv8MN5LLzG2azvD2/dxzZrOfFx+x9pLaFq9MR+Xf9aKMTTu8Nq5BIfjSQ4MJVjaHKqo5bcs6Sba20m4uwe7u59M4tmqWr5YNnY4wuF4clJc/mjKIZ7MMJ5x81p+Nu3iZF0i0XDFuPxIQdG0WMTGLSnyVk7Lz20tYbuqlu/v+xo9VNfy82uW2rR8S+S8HG4zreWX3mcm7lcO1fLnDjX6iqIoCwRjDJ7W01cURVk4aPSOoijKQmGOondEpAv4BjCA3+P2TmPMmTLnDQKjgAs4xphrpnN9IRfSA1pRFOWiJBe9U22bAe4FnjLGrAGeCo4r8XZjzJU5g38e1wMN8qZ/evtu3vvw3+c7Yx380h/xnW/u4BenkyRdw0AszE3rull/59Usft9vc/ID/0Rzey/dr387l6xfxk1vWMpvbVjMFb3NhA/+krFvbmbPs9s4tuUEGz5wP1dc1s2Na3q4aukiBhaFCZ/cQ/b5LZzZuYORnQcZ2T3CmQNnueq/vaWoM5bbsZwhN8TQuMOhs6MciSc5OJTg0EiCEyPj3NsdzXfGalncEiRidRHt6yTU2YvV2Ueoux8v1oGbeaJozWLZ+c0KR7ACZ64VCnM4nizqjDWWCpKyUg5O1sXJeP5nsDW3hAu6ZVn+Z8giGrFpyne98h26biaZ74yVc94CRQ5c/9ijKWQXdcayrdJ934Gb64JV6HCt5HzNjdvW5GJr4Dtwc87M83VAWkx2uhZ10jq/21a8Xzmm+4zZcOCCOnGnwpsbR+4m4MZg/+vAM8Cfzub1+qavKIpSShCyWW0DekTkxYLtnmk+abEx5jhA8NlXeUb8RES2lDyj1uvzNMSbvqIoypxSu6Y/XCK3TEJENgP9Zb66bxozut4Yc0xE+oAnRWS3Mea5aVyfR42+oihKCYaZi94xxtxU6TsROSkiS4wxx0VkCXCqwj2OBZ+nROQx4FrgOaCm6wtpCKOf8Qxfa3uOl+/8BA9sOcH+RIaoLby+vZnXv/US1t39dsI33sVe083Xdp7g0hs2sebX+nj/1cu5YWUHy70RvB0/YPhrz3P053s5se0U+8YyHEs5fO6O17GhJ0aviWMd+QWZZ7Zw7JV9DO84wsjeM4wMJTiadDiTdXn3+/8LXtclpFsXMzTucHIky+DZMQZPj3NgKMFrp8eJn02ROJciOZphydX9tPS1EevvJtbXSVNPF3Z3P3ZnH1Z7D160Hae5Da+5Pb/WvI4fiiC2jR3o+FYoghWOEIpEOXAqwViBlp/O5PR7D6dg33U8XNejrTOaL7AWKdLyg8Qs2x9vClk4gaZfmIgFOU3fy+8DxMJ+ElY4SMrytXtfyw9blq/Li+R1/cJrCyk3lkvmsqQ4EQsmdOjz1Sal4N5F4yXnTTexaqZ1fKWOGIOXmZMyDN8HPgTcH3x+r/QEEWkBLGPMaLD/TuDPa72+FNX0FUVRSjHgeV7VbQa4H7hZRPYCNwfHiMhSEXk8OGcx8DMR2Qa8APzIGPPEVNdPRUO86SuKoswlhrmJ0zfGjADvKDN+DLg12D8AvH4610+FGn1FUZRSTHEv54uJhjD6Szau5L7f/jJJ17C6JcJdVy9hw53X0Pu+D3Cq9wq+s/8Mjzx2mP07tzG8fydPPfRHbOiwsff9nNFvPc2en73C8a0n2Hc8wbFUltMZF9dAxBJ+wz5E5pdbObtjJyM7DzK8e4Qzg3GOJh2G0g7nHI+k6+EaONV/FUPjDoODcb+o2ik/Jn/4TJKxsynGxzKkEhkyo6fJjMdZduNGPyY/0PHtzl68WAdecztucxsZK0Ii6zGecPIF1Upj8u2mKFYogh2KYEeiWOEIh0YSFWPyXcfgOf6Y63oYzxBriUyKyY+G7byOn29uHrLyDVRKY/ILtX0Az3P9OP0KMfmFWn7uuNZiazCh5VfS8Svp8rVQLSZ/poukqZbfiJiLtgzDrGn6IvJVETklIjsKxrpE5EkR2Rt8ds7W8xVFUc6b2uP0G47ZdOT+I3BLydi0U4YVRVHmGmMMbsapujUis2b0g8SB0yXDm/BThQk+3zNbz1cURTl/TCBrTr01InOt6RelDAfZZWUJUo3vAVixZDEQnZsZKoqiaOesuccY8yDwIEDLsrXmto0d+YJqZy+5licPnOHRZ46wZ+dTDO17lcSpI7iZJHYkyqon/ooDP9vO0ReOc+DYKEeSE85bW6A9bLO4KcSKWIhf/e/P5wuqHRnPTnLegu/wbQ0J3909VFRQLXEuTeJcush56yTHcDMpnHSS9je/jVB3Pya6CK+5nWy0fcJ5m/JIZrN+96uUQzjamnfeWqEIdlO0yHlrR6LYIb/r1fBIsqzz1nX9hCzP9XAdx++A5br0LVpZlIg14dAt3mwR3EzS//Ov4LzN//24LrGwVdV5m+t8lXPE1tLlyngutkhNztvzKRhWq/O2XCesC3mG0kAYMDkDcJEx10Z/2inDiqIoc43BzFWVzTlnrjNycynDUGPKsKIoypxjwHim6taIzNqbvog8gl/nuUdEXgM+jZ8i/E0R+TBwGLhjtp6vKIpyvhgDbkaTs6aFMebuCl9NK2UYIHn2DAPbtvBve4f59o8Pc2jX45wdfIXxkWMYzyXc0k7b0tV0rVhN/0AH//jH93AslSWe9X89i1hCb1OIpc0hlrVG6FrTSddl3XRtWMm/fvpxzmRd4lmXZIGGF7WFqG2xKGTRHrbpbbL50rMH/WJqYxmSowmyiXiRju9mM76OnktsWncdmeZ2Up6QyBqSSY/xbIZ4yiGedhjLOIylHc6lHZrae7BCfkG1nKZvhSKEwjahSHEDlLF4Eifja/hFWn7w7MIEK8/J0NvWPEnHzyVjhS2LsO1r8WFL8Jys//dXQcfP73susbA9ScMHinR8S6hJzy/9zs41TSnR8Qtl9gv5NXWmNXyYno4/001RtBnKDGOMavqKoigLCU+NvqIoygJBQzYVRVEWDgbwGtRRWw01+oqiKKUYo47cetK/bDHX/tcvFyVgRTsXs/Tqd7F4RQevW9vDWy/r4Y3LFjGwKMzHP56mPWyzoa2JFbEQ3Svb6bqsk64NK+hYt4rwwHpkyWrcjuXs+thjgO/sbQ9btNgWXRGbrohNZ0uYaE+M1r4YLYtbOPjyXrKpMbKJeD4Bq8hxW4BYNq95bSTjbj4BayzjO3BH0w7x8SxjKX9/LJUl1r2sKAHLd9zahMIWVsGYHRKOHTgzKQGrcB7Gc3Fzx65L36KmogSssOV3u/K7XvmO2Fy1TNfJ5NdQ6rgtxHguzSFrUgJWocPVosCxW+JorJakZRdcUK5T1oU4XYuTu8rfZ6YrbarjtrEwmpylKIqygFCjryiKspDQjFxFUZSFwxxl5NbSY0RE1onIywXbORH5aPDdZ0TkaMF3t1Z7ZkO86fclhzgWirDyze+kf6CD69b2cv2l3VzR18KSUAr7+C4yu3/K6R/sZu/uI/zOjSvzyVetay4jMrAer2cAp2MZQ+MOQwmHwTPjHDo4zEAsnE++amtvItYdpXVxC7G+VmJ9ncT6u4j2dmF19nHm09um1PBzmxX2O129cPRcPvkqPp5lNOUnY42l/P3xXPerrMeins588lUobAc6vpXX+HOafFPIYv/W/UXJV16Blm/cwo5X/n5fW1Ney7es4FN8Db9w35IJHb+WLlcR2ypKviosrJbrfOXvS8V7VML3CRQeT4jYF6q3l9PxVcNXCjHMWZx+rsfI/SJyb3D8p0VzMWYPcCWAiNjAUeCxglO+aIz5q1of2BBGX1EUZU4xBm9uonc24ZerAb/HyDOUGP0S3gHsN8YcOt8HqryjKIpSgjH+m361bQYo6jECVOwxEnAX8EjJ2EdEZHvQorZqC1o1+oqiKGWosXNWj4i8WLDdU3ofEdksIjvKbJumMx8RiQC3A98qGP4KsBpf/jkO/HW1+zSEvHP0tbNs2XYP/dY49rFXSe96nNOP7mFk12vs3z3C0IkxTqRchjMOY47HF44+TXbREobGHQYTWQ6eTXJozzgHhvZwaDhB/GzKL5w2muEbt1xKS18b0b5Oor0dRBf3Ynf2Ynf2YXX04kXb/a2pDSf1PODr91YoUqTf55qfWOGJomlP7jrFWCrLeMYt0u+dTND8xPXyhdN6lrZN0u9jEbuo+UlO03967HRF/T7Xwq1wvLM5XFa/D1vWpOYn5fwVhfcrJGJPFEMr1e8rNUCpFbtMwxSYXNTsfLT4atecr3w+0zq+UkdMzW/yw8aYa6a+lbmp0nciMp0eI+8GthpjThbcO78vIg8BP6w2YX3TVxRFKSWI06+2zQDT6TFyNyXSTvCDIsd7gR3VHtgQb/qKoihziWHOCq6V7TEiIkuBh40xtwbHMeBm4A9Krv+CiFwZTHmwzPeTUKOvKIpSijG4mdk3+saYEcr0GDHGHANuLTgeB7rLnPfB6T5Tjb6iKEoJxoBntAxD3ehd1MSe69/Gs0PjnEi5nMm6jDkemSAjzha/YFpryGJpc5g//OlZDg0fJXEuzfi5NOOjadIJv1BaJhHHSSXwnAxOOsnlD34cWdSDF23HNLfhNi9iLOuRyHokHY9k1iN+2iGeHqW5vTfvsLWbooEDN4IdiQYO3KaCxCqb7XuG8BwPJ+t3tvIdty7GmHynq1yXqze9cTmRkEU0bOe7XOW6W+W3oEhaNhEv67CFiU5XhcXSemKRSQ7b3HFpYTSvoODaVBjPJWxJxSSqcp2upoNdct1Md7qqt8tVfb7zH1eNvqIoysLAABdpvTU1+oqiKOXQN31FUZQFgmfIy8cXGw1h9L0Vl/L4rtO0hiwWhWxWt4Tpiti0dceI9URpWdxCS18bsf5uYn2dDDz8nXyTk1xRslJyxdF29FxLPOUQP+0wls4QT59grKBAWjyZJZlxGE059K6/tkizt0NS1PDEsoPjkEU0YrP9Fwfzmn1uHsZzyxZIe8PKt+c1+7AtfuKUQMj2P/1xfz+bSkzZ4KR0rCsa9tccNDMpLZCW198rXF+JiC1F2vRMFkjz5zk7DU7KXa4F0pRSVN5RFEVZIBiMyjuKoigLBXXkKoqiLDDU6NeRvYdO8tJ3/2e+EBotnX4RtOZFZENRxrMeScdwNutxNOOS+dqnscMRIi3tZQuh2U3+ZygS5o+/tZ1surAAml8UzQvi6l3Hyzchv+LaFWULoTUVxtIXxNj/7NtPFMTRT8TVF+rlubj6X+trwxImxdGXi6t308n89bVo760RX22fqghaTiefTqOTSEEw/UwUQivELrnBTErks1UYTXX8iwdjNHpHURRlwWDQ6B1FUZQFg2r6iqIoCwyVdxRFURYIvqZf71nMDg1h9O1IM3efuJqxwaD7VOY0TnYo6ETl4jomKGzmO2Ovu/sOQkGC1IST1SYadKUqLGj2f7/4baCw89SE47W0mNnvf+xtfpKUNdF9airHa/LMyUlrqeQovbSzGfAdltW6T9VaFC1HLGwVOVbLJydN65ZAsSO3lAv1adqz6BVVh6tSC/qmryiKskAwwJy0UKkDavQVRVFKMBiN3lEURVko+NE7avTrxuUru3j8yw/WfP4rD/xtzed+/pP7az735ks7aj4Xpqe9L2kNT+ve0yGXnDXThC40A2sKVHdX6spF7MidHWtQBRG5RUT2iMg+Ebm3HnNQFEWpRO5Nv9p2oYjIHSKyU0Q8EblmivPK2kwR6RKRJ0Vkb/DZWe2Zc270RcQGvgy8G9gI3C0iG+d6HoqiKFPhmurbDLADeB/wXKUTqtjMe4GnjDFrgKeC4ympx5v+tcA+Y8wBY0wGeBTYVId5KIqilMXDL8NQbbtQjDG7jDF7qpw2lc3cBHw92P868J5qzxQzx84KEXk/cIsx5veC4w8CbzLGfKTkvHuAe4LDy/F/Il4s9ADD9Z7EDHOxrUnXM/+ptKaVxpjeC7mxiDwR3L8azUCq4PhBY0ztDsiJ5z0DfMIY82KZ7yraTBE5a4zpKDj3jDFmSomnHo7cci66ST95gj+4BwFE5EVjTEW9q9G42NYDF9+adD3zn9lckzHmlpm6l4hsBvrLfHWfMeZ7tdyizNh5v63Xw+i/BlxScLwcOFaHeSiKosw6xpibLvAWU9nMkyKyxBhzXESWAKeq3awemv5/AmtEZJWIRIC7gO/XYR6KoiiNwFQ28/vAh4L9DwFVf3OYc6NvjHGAjwA/BnYB3zTG7Kxy2bQ1snnOxbYeuPjWpOuZ/zT8mkTkvSLyGnAd8CMR+XEwvlREHoeqNvN+4GYR2QvcHBxP/cy5duQqiqIo9aMuyVmKoihKfVCjryiKsoCY10a/Ucs1iMhXReSUiOwoGKuYLi0ifxascY+IvKs+s66MiFwiIj8VkV1Byvh/D8Ybck0i0iwiL4jItmA9nw3GG3I9OUTEFpGXROSHwXGjr2dQRF4RkZdF5MVgrKHXNC8wxszLDbCB/cClQATYBmys97xqnPsNwFXAjoKxLwD3Bvv3An8Z7G8M1tYErArWbNd7DSXrWQJcFey3Ab8K5t2Qa8KPe24N9sPAL4E3N+p6Ctb1MeBfgR82+r+5YJ6DQE/JWEOvaT5s8/lNv2HLNRhjngNOlwxXSpfeBDxqjEkbYw4C+/DXPm8wxhw3xmwN9kfxIwiW0aBrMj5jwWE42AwNuh4AEVkO/CbwcMFww65nCi7GNc0p89noLwOOFBy/Fow1KouNMcfBN6JAXzDeUOsUkQHgDfhvxw27pkAKeRk/meVJY0xDrwf4EvBJihs+NfJ6wP9B/BMR2RKUZYHGX1Pdmc/19Gc09Xge0zDrFJFW4DvAR40x56Ry0ft5vyZjjAtcKSIdwGMicvkUp8/r9YjIbwGnjDFbROTGWi4pMzZv1lPA9caYYyLSBzwpIrunOLdR1lR35vOb/sVWruFkkCZNSbp0Q6xTRML4Bv9fjDH/Fgw39JoAjDFngWeAW2jc9VwP3C4ig/gy6G+IyD/TuOsBwBhzLPg8BTyGL9c09JrmA/PZ6F9s5RoqpUt/H7hLRJpEZBWwBnihDvOriPiv9P8A7DLGPFDwVUOuSUR6gzd8RCQK3ATspkHXY4z5M2PMcmPMAP7/k6eNMb9Dg64HQERaRKQttw+8E7/SbsOuad5Qb0/yVBtwK36kyH78inR1n1ON834EOA5k8d9APgx04zc52Bt8dhWcf1+wxj3Au+s9/zLreQv+r8rbgZeD7dZGXRPwOuClYD07gE8F4w25npK13chE9E7Drgc/am9bsO3M/f9v5DXNl03LMCiKoiwg5rO8oyiKoswwavQVRVEWEGr0FUVRFhBq9BVFURYQavQVRVEWEGr0lYZGRD4jIp+o9zwUpVFQo68oirKAUKOvNBwicl9QM30zsK7e81GURmI+F1xTlEmIyNX4pQbegP/vdyuwpa6TUpQGQo2+0mi8FXjMGDMOICKNXI9JUeYclXeURkRrhyjKeaJGX2k0ngPeKyLRoArjbfWekKI0EirvKA2FMWariHwDv9LnIeDf6zwlRWkotMqmoijKAkLlHUVRlAWEGn1FUZQFhBp9RVGUBYQafUVRlAWEGn1FUZQFhBp9RVGUBYQafUVRlAXE/wde/fYLBVoQLQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos_encoding = positional_encoding(50, 512)\n",
    "\n",
    "print (pos_encoding.shape)\n",
    "\n",
    "plt.pcolormesh(pos_encoding[0], cmap='RdBu')\n",
    "plt.xlabel('d')\n",
    "plt.xlim((0, 512))\n",
    "plt.ylabel('Position')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row represents a positional encoding - notice how none of the rows are identical! You have created a unique positional encoding for each of the words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## 2 - Masking\n",
    "\n",
    "There are two types of masks that are useful when building your Transformer network: the *padding mask* and the *look-ahead mask*. Both help the softmax computation give the appropriate weights to the words in your input sentence. \n",
    "\n",
    "<a name='2-1'></a>\n",
    "### 2.1 - Padding Mask\n",
    "\n",
    "Oftentimes your input sequence will exceed the maximum length of a sequence your network can process. In this case, your sequence will be cut off, and the shorter sequence will have zeros appended onto the end. \n",
    "However, these zeros will affect the softmax calculation - this is when a padding mask comes in handy! By multiplying a padding mask by -1e9 and adding it to your sequence, you mask out the zeros by setting them to close to \n",
    "negative infinity. We'll implement this for you so you can get to the fun of building the Transformer network! 😇Just make sure you go through the code so you can correctly implement padding when building your model. \n",
    "\n",
    "After masking, your input should go from `[1, 2, 3, 0, 0]` to `[1, 2, 3, -1e9, -1e9]`, so that when you take the softmax, the zeros don't affect the score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "JOL9XWsFQxxo"
   },
   "outputs": [],
   "source": [
    "def create_padding_mask(seq):\n",
    "    \"\"\"\n",
    "    Creates a matrix mask for the padding cells\n",
    "    \n",
    "    Arguments:\n",
    "        seq -- (n, m) matrix\n",
    "    \n",
    "    Returns:\n",
    "        mask -- (n, 1, 1, m) binary tensor\n",
    "    \"\"\"\n",
    "    seq = tf.cast(tf.math.equal(seq, 0), tf.float32)\n",
    "  \n",
    "    # add extra dimensions to add the padding\n",
    "    # to the attention logits.\n",
    "    return seq[:, tf.newaxis, tf.newaxis, :] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5J5FFjklQ1Fz",
    "outputId": "8319446f-3ed4-406a-cf38-ca2b08142ff4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[[0. 0. 1. 1. 0.]]]\n",
      "\n",
      "\n",
      " [[[0. 0. 0. 1. 1.]]]\n",
      "\n",
      "\n",
      " [[[1. 1. 1. 0. 0.]]]], shape=(3, 1, 1, 5), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "x = tf.constant([[7., 6., 0., 0., 1.], [1., 2., 3., 0., 0.], [0., 0., 0., 4., 5.]])\n",
    "print(create_padding_mask(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we multiply this mask by -1e9 and add it to the sample input sequences, the zeros are essentially set to negative infinity. Notice the difference when taking the softmax of the original sequence and the masked sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[7.2876644e-01 2.6809821e-01 6.6454901e-04 6.6454901e-04 1.8064314e-03]\n",
      " [8.4437378e-02 2.2952460e-01 6.2391251e-01 3.1062774e-02 3.1062774e-02]\n",
      " [4.8541026e-03 4.8541026e-03 4.8541026e-03 2.6502505e-01 7.2041273e-01]], shape=(3, 5), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[[[7.2973627e-01 2.6845497e-01 0.0000000e+00 0.0000000e+00\n",
      "    1.8088354e-03]\n",
      "   [2.4472848e-01 6.6524094e-01 0.0000000e+00 0.0000000e+00\n",
      "    9.0030573e-02]\n",
      "   [6.6483547e-03 6.6483547e-03 0.0000000e+00 0.0000000e+00\n",
      "    9.8670328e-01]]]\n",
      "\n",
      "\n",
      " [[[7.3057163e-01 2.6876229e-01 6.6619506e-04 0.0000000e+00\n",
      "    0.0000000e+00]\n",
      "   [9.0030573e-02 2.4472848e-01 6.6524094e-01 0.0000000e+00\n",
      "    0.0000000e+00]\n",
      "   [3.3333334e-01 3.3333334e-01 3.3333334e-01 0.0000000e+00\n",
      "    0.0000000e+00]]]\n",
      "\n",
      "\n",
      " [[[0.0000000e+00 0.0000000e+00 0.0000000e+00 2.6894143e-01\n",
      "    7.3105860e-01]\n",
      "   [0.0000000e+00 0.0000000e+00 0.0000000e+00 5.0000000e-01\n",
      "    5.0000000e-01]\n",
      "   [0.0000000e+00 0.0000000e+00 0.0000000e+00 2.6894143e-01\n",
      "    7.3105860e-01]]]], shape=(3, 1, 3, 5), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(tf.keras.activations.softmax(x))\n",
    "print(tf.keras.activations.softmax(x + create_padding_mask(x) * -1.0e9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2-2'></a>\n",
    "### 2.2 - Look-ahead Mask\n",
    "\n",
    "The look-ahead mask follows similar intuition. In training, you will have access to the complete correct output of your training example. The look-ahead mask helps your model pretend that it correctly predicted a part of the output and see if, *without looking ahead*, it can correctly predict the next output. \n",
    "\n",
    "For example, if the expected correct output is `[1, 2, 3]` and you wanted to see if given that the model correctly predicted the first value it could predict the second value, you would mask out the second and third values. So you would input the masked sequence `[1, -1e9, -1e9]` and see if it could generate `[1, 2, -1e9]`.\n",
    "\n",
    "Just because you've worked so hard, we'll also implement this mask for you 😇😇. Again, take a close look at the code so you can effictively implement it later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "9O9UbM31Q3hK"
   },
   "outputs": [],
   "source": [
    "def create_look_ahead_mask(size):\n",
    "    \"\"\"\n",
    "    Returns an upper triangular matrix filled with ones\n",
    "    \n",
    "    Arguments:\n",
    "        size -- matrix size\n",
    "    \n",
    "    Returns:\n",
    "        mask -- (size, size) tensor\n",
    "    \"\"\"\n",
    "    mask = 1 - tf.linalg.band_part(tf.ones((size, size)), -1, 0)\n",
    "    return mask "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nfzHoVj9Q5nG",
    "outputId": "300e76ec-77d0-460a-b6df-71e40de86606"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 3), dtype=float32, numpy=\n",
       "array([[0., 1., 1.],\n",
       "       [0., 0., 1.],\n",
       "       [0., 0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.random.uniform((1, 3))\n",
    "temp = create_look_ahead_mask(x.shape[1])\n",
    "temp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VG0gPyv0oDBi"
   },
   "source": [
    "<a name='3'></a>\n",
    "## 3 - Self-Attention\n",
    "\n",
    "As the authors of the Transformers paper state, \"Attention is All You Need\". \n",
    "\n",
    "<img src=\"self-attention.png\" alt=\"Encoder\" width=\"600\"/>\n",
    "<caption><center><font color='purple'><b>Figure 1: Self-Attention calculation visualization</font></center></caption>\n",
    "    \n",
    "The use of self-attention paired with traditional convolutional networks allows for the parallization which speeds up training. You will implement **scaled dot product attention** which takes in a query, key, value, and a mask as inputs to returns rich, attention-based vector representations of the words in your sequence. This type of self-attention can be mathematically expressed as:\n",
    "$$\n",
    "\\text { Attention }(Q, K, V)=\\operatorname{softmax}\\left(\\frac{Q K^{T}}{\\sqrt{d_{k}}}+{M}\\right) V\\tag{4}\\\n",
    "$$\n",
    "\n",
    "* $Q$ is the matrix of queries \n",
    "* $K$ is the matrix of keys\n",
    "* $V$ is the matrix of values\n",
    "* $M$ is the optional mask you choose to apply \n",
    "* ${d_k}$ is the dimension of the keys, which is used to scale everything down so the softmax doesn't explode\n",
    "\n",
    "<a name='ex-3'></a>\n",
    "### Exercise 3 - scaled_dot_product_attention \n",
    "\n",
    "    Implement the function `scaled_dot_product_attention()` to create attention-based representations\n",
    "**Reminder**: The boolean mask parameter can be passed in as `none` or as either padding or look-ahead. Multiply it by -1e9 before applying the softmax. \n",
    "\n",
    "**Additional Hints**\n",
    "* You may find [tf.matmul](https://www.tensorflow.org/api_docs/python/tf/linalg/matmul) useful for matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "CSysk_rjQ7lp"
   },
   "outputs": [],
   "source": [
    "# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION scaled_dot_product_attention\n",
    "def scaled_dot_product_attention(q, k, v, mask):\n",
    "    \"\"\"\n",
    "    Calculate the attention weights.\n",
    "      q, k, v must have matching leading dimensions.\n",
    "      k, v must have matching penultimate dimension, i.e.: seq_len_k = seq_len_v.\n",
    "      The mask has different shapes depending on its type(padding or look ahead) \n",
    "      but it must be broadcastable for addition.\n",
    "\n",
    "    Arguments:\n",
    "        q -- query shape == (..., seq_len_q, depth)\n",
    "        k -- key shape == (..., seq_len_k, depth)\n",
    "        v -- value shape == (..., seq_len_v, depth_v)\n",
    "        mask: Float tensor with shape broadcastable \n",
    "              to (..., seq_len_q, seq_len_k). Defaults to None.\n",
    "\n",
    "    Returns:\n",
    "        output -- attention_weights\n",
    "    \"\"\"\n",
    "    # START CODE HERE\n",
    "    \n",
    "    # Q*K'\n",
    "    matmul_qk = tf.matmul(q, k.T)\n",
    "    \n",
    "    # scale matmul_qk\n",
    "    #dk = len(k)\n",
    "    dk = tf.cast(tf.shape(k)[-1], tf.float32)\n",
    "    scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)\n",
    "\n",
    "    # add the mask to the scaled tensor.\n",
    "    if mask is not None:\n",
    "        scaled_attention_logits += (mask * -1.0e9)\n",
    "\n",
    "    # softmax is normalized on the last axis (seq_len_k) so that the scores\n",
    "    # add up to 1.\n",
    "    attention_weights = tf.keras.activations.softmax(scaled_attention_logits, axis=-1)  # (..., seq_len_q, seq_len_k)\n",
    "    print('attention_weights: ', attention_weights)\n",
    "    # attention_weights * V\n",
    "    output = tf.matmul(attention_weights, v) # (..., seq_len_q, depth_v)\n",
    "    print('output: ', output)\n",
    "    \n",
    "    # END CODE HERE\n",
    "\n",
    "    return output, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "attention_weights:  tf.Tensor(\n",
      "[[0.2589478  0.42693272 0.15705977 0.15705977]\n",
      " [0.2772748  0.2772748  0.2772748  0.16817567]\n",
      " [0.33620113 0.33620113 0.12368149 0.2039163 ]], shape=(3, 4), dtype=float32)\n",
      "output:  tf.Tensor(\n",
      "[[0.74105227 0.15705977]\n",
      " [0.7227253  0.16817567]\n",
      " [0.6637989  0.2039163 ]], shape=(3, 2), dtype=float32)\n",
      "attention_weights:  tf.Tensor(\n",
      "[[0.3071959  0.5064804  0.         0.18632373]\n",
      " [0.38365173 0.38365173 0.         0.23269655]\n",
      " [0.38365173 0.38365173 0.         0.23269655]], shape=(3, 4), dtype=float32)\n",
      "output:  tf.Tensor(\n",
      "[[0.6928041  0.18632373]\n",
      " [0.61634827 0.23269655]\n",
      " [0.61634827 0.23269655]], shape=(3, 2), dtype=float32)\n",
      "\u001b[92mAll tests passed\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def scaled_dot_product_attention_test(target):\n",
    "    q = np.array([[1, 0, 1, 1], [0, 1, 1, 1], [1, 0, 0, 1]]).astype(np.float32)\n",
    "    k = np.array([[1, 1, 0, 1], [1, 0, 1, 1 ], [0, 1, 1, 0], [0, 0, 0, 1]]).astype(np.float32)\n",
    "    v = np.array([[0, 0], [1, 0], [1, 0], [1, 1]]).astype(np.float32)\n",
    "\n",
    "    attention, weights = target(q, k, v, None)\n",
    "    assert tf.is_tensor(weights), \"Weights must be a tensor\"\n",
    "    assert tuple(tf.shape(weights).numpy()) == (q.shape[0], k.shape[1]), f\"Wrong shape. We expected ({q.shape[0]}, {k.shape[1]})\"\n",
    "    assert np.allclose(weights, [[0.2589478,  0.42693272, 0.15705977, 0.15705977],\n",
    "                                   [0.2772748,  0.2772748,  0.2772748,  0.16817567],\n",
    "                                   [0.33620113, 0.33620113, 0.12368149, 0.2039163 ]])\n",
    "\n",
    "    assert tf.is_tensor(attention), \"Output must be a tensor\"\n",
    "    assert tuple(tf.shape(attention).numpy()) == (q.shape[0], v.shape[1]), f\"Wrong shape. We expected ({q.shape[0]}, {v.shape[1]})\"\n",
    "    assert np.allclose(attention, [[0.74105227, 0.15705977],\n",
    "                                   [0.7227253,  0.16817567],\n",
    "                                   [0.6637989,  0.2039163 ]])\n",
    "\n",
    "    mask = np.array([0, 0, 1, 0])\n",
    "    attention, weights = target(q, k, v, mask)\n",
    "\n",
    "    #assert np.allclose(weights, [[0.30719590187072754, 0.5064803957939148, 0.0, 0.18632373213768005],\n",
    "    #                             [0.3836517333984375, 0.3836517333984375, 0.0, 0.2326965481042862],\n",
    "    #                             [0.3836517333984375, 0.3836517333984375, 0.0, 0.2326965481042862]]), \"Wrong masked weights\"\n",
    "    #assert np.allclose(attention, [[0.6928040981292725, 0.18632373213768005],\n",
    "    #                               [0.6163482666015625, 0.2326965481042862], \n",
    "    #                               [0.6163482666015625, 0.2326965481042862]]), \"Wrong masked attention\"\n",
    "    \n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "    \n",
    "scaled_dot_product_attention_test(scaled_dot_product_attention)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent work! You can now implement self-attention. With that, you can start building the encoder block! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "blS0pEpTqRVI"
   },
   "source": [
    "<a name='4'></a>\n",
    "## 4 - Encoder\n",
    "\n",
    "The Transformer Encoder layer pairs self-attention and convolutional neural network layers to improve the speed of training and passes K and V matrices to the Decoder, which you'll build later in the assignment. In this section of the assignment, you will implement the Encoder by pairing multi-head attention and a feed forward neural network (Figure 2a). \n",
    "<img src=\"encoder_layer.png\" alt=\"Encoder\" width=\"250\"/>\n",
    "<caption><center><font color='purple'><b>Figure 2a: Transformer encoder layer</font></center></caption>\n",
    "\n",
    "* `MultiHeadAttention` you can think of as computing the self-attention several times to detect different features. \n",
    "* Feed forward neural network contains two Dense layers which we'll implement as the function `FullyConnected`\n",
    "\n",
    "Your input sentence first passes through a *multi-head attention layer*, where the encoder looks at other words in the input sentence as it encodes a specific word. The outputs of the multi-head attention layer are then fed to a *feed forward neural network*. The exact same feed forward network is independently applied to each position.\n",
    "   \n",
    "* For the `MultiHeadAttention` layer, you will use the [Keras implementation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention). If you're curious about how to split the query matrix Q, key matrix K, and value matrix V into different heads, you can look through the implementation. \n",
    "* You will also use the [Sequential API](https://keras.io/api/models/sequential/) with two dense layers to built the feed forward neural network layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "sC5vJhz29vZR"
   },
   "outputs": [],
   "source": [
    "def FullyConnected(embedding_dim, fully_connected_dim):\n",
    "    return tf.keras.Sequential([\n",
    "        tf.keras.layers.Dense(fully_connected_dim, activation='relu'),  # (batch_size, seq_len, dff)\n",
    "        tf.keras.layers.Dense(embedding_dim)  # (batch_size, seq_len, d_model)\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R65WbX5wqYYH"
   },
   "source": [
    "<a name='4-1'></a>\n",
    "### 4.1 Encoder Layer\n",
    "\n",
    "Now you can pair multi-head attention and feed forward neural network together in an encoder layer! You will also use residual connections and layer normalization to help speed up training (Figure 2a).\n",
    "\n",
    "<a name='ex-4'></a>\n",
    "### Exercise 4 - EncoderLayer\n",
    "\n",
    "Implement `EncoderLayer()` using the `call()` method\n",
    "\n",
    "In this exercise, you will implement one encoder block (Figure 2) using the `call()` method. The function should perform the following steps: \n",
    "1. You will pass the Q, V, K matrices and a boolean mask to a multi-head attention layer. Remember that to compute *self*-attention Q, V and K should be the same.\n",
    "2. Next, you will pass the output of the multi-head attention layer to a dropout layer. Don't forget to use the `training` parameter to set the mode of your model. \n",
    "3. Now add a skip connection by adding your original input `x` and the output of the dropout layer. \n",
    "4. After adding the skip connection, pass the output through the first layer normalization.\n",
    "5. Finally, repeat steps 1-4 but with the feed forward neural network instead of the multi-head attention layer.\n",
    "\n",
    "**Additional Hints**:\n",
    "* The `__init__` method creates all the layers that will be accesed by the the `call` method. Wherever you want to use a layer defined inside  the `__init__`  method you will have to use the syntax `self.[insert layer name]`. \n",
    "* You will find the documentation of [MultiHeadAttention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention) helpful. *Note that if query, key and value are the same, then this function performs self-attention.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "tIufbrc-9_2u"
   },
   "outputs": [],
   "source": [
    "# UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION EncoderLayer\n",
    "class EncoderLayer(tf.keras.layers.Layer):\n",
    "    \"\"\"\n",
    "    The encoder layer is composed by a multi-head self-attention mechanism,\n",
    "    followed by a simple, positionwise fully connected feed-forward network. \n",
    "    This archirecture includes a residual connection around each of the two \n",
    "    sub-layers, followed by layer normalization.\n",
    "    \"\"\"\n",
    "    def __init__(self, embedding_dim, num_heads, fully_connected_dim, dropout_rate=0.1, layernorm_eps=1e-6):\n",
    "        super(EncoderLayer, self).__init__()\n",
    "\n",
    "        self.mha = MultiHeadAttention(num_heads=num_heads,\n",
    "                                      key_dim=embedding_dim)\n",
    "\n",
    "        self.ffn = FullyConnected(embedding_dim=embedding_dim,\n",
    "                                  fully_connected_dim=fully_connected_dim)\n",
    "\n",
    "        self.layernorm1 = LayerNormalization(epsilon=layernorm_eps)\n",
    "        self.layernorm2 = LayerNormalization(epsilon=layernorm_eps)\n",
    "\n",
    "        self.dropout1 = Dropout(dropout_rate)\n",
    "        self.dropout2 = Dropout(dropout_rate)\n",
    "    \n",
    "    def call(self, x, training, mask):\n",
    "        \"\"\"\n",
    "        Forward pass for the Encoder Layer\n",
    "        \n",
    "        Arguments:\n",
    "            x -- Tensor of shape (batch_size, input_seq_len, fully_connected_dim)\n",
    "            training -- Boolean, set to true to activate\n",
    "                        the training mode for dropout layers\n",
    "            mask -- Boolean mask to ensure that the padding is not \n",
    "                    treated as part of the input\n",
    "        Returns:\n",
    "            out2 -- Tensor of shape (batch_size, input_seq_len, fully_connected_dim)\n",
    "        \"\"\"\n",
    "        # START CODE HERE\n",
    "        # calculate self-attention using mha(~1 line)\n",
    "        attn_output = self.mha(x, x, x, mask)  # Self attention (batch_size, input_seq_len, fully_connected_dim)\n",
    "        \n",
    "        # apply dropout layer to the self-attention output (~1 line)\n",
    "        attn_output = self.dropout1(attn_output, training=training)\n",
    "        \n",
    "        # apply layer normalization on sum of the input and the attention output to get the  \n",
    "        # output of the multi-head attention layer (~1 line)\n",
    "        out1 = self.layernorm1(x + attn_output)  # (batch_size, input_seq_len, fully_connected_dim)\n",
    "\n",
    "        # pass the output of the multi-head attention layer through a ffn (~1 line)\n",
    "        ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, fully_connected_dim)\n",
    "        \n",
    "        # apply dropout layer to ffn output (~1 line)\n",
    "        ffn_output = self.dropout2(ffn_output, training=training)\n",
    "        \n",
    "        # apply layer normalization on sum of the output from multi-head attention and ffn output to get the\n",
    "        # output of the encoder layer (~1 line)\n",
    "        out2 = self.layernorm2(out1 + ffn_output)  # (batch_size, input_seq_len, fully_connected_dim)\n",
    "        # END CODE HERE\n",
    "        \n",
    "        return out2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mAll tests passed\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def EncoderLayer_test(target):\n",
    "    q = np.array([[[1, 0, 1, 1], [0, 1, 1, 1], [1, 0, 0, 1]]]).astype(np.float32)\n",
    "    encoder_layer1 = EncoderLayer(4, 2, 8)\n",
    "    tf.random.set_seed(10)\n",
    "    encoded = encoder_layer1(q, True, np.array([[1, 0, 1]]))\n",
    "    \n",
    "    assert tf.is_tensor(encoded), \"Wrong type. Output must be a tensor\"\n",
    "    assert tuple(tf.shape(encoded).numpy()) == (1, q.shape[1], q.shape[2]), f\"Wrong shape. We expected ((1, {q.shape[1]}, {q.shape[2]}))\"\n",
    "\n",
    "    assert np.allclose(encoded.numpy(), \n",
    "                       [[-0.5214877 , -1.001476  , -0.12321664,  1.6461804 ],\n",
    "                       [-1.3114998 ,  1.2167752 , -0.5830886 ,  0.6778133 ],\n",
    "                       [ 0.25485858,  0.3776546 , -1.6564771 ,  1.023964  ]],), \"Wrong values\"\n",
    "    \n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "    \n",
    "\n",
    "EncoderLayer_test(EncoderLayer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='4-2'></a>\n",
    "### 4.2 - Full Encoder\n",
    "\n",
    "Awesome job! You have now successfully implemented positional encoding, self-attention, and an encoder layer - give yourself a pat on the back. Now you're ready to build the full Transformer Encoder (Figure 2b), where you will embedd your input and add the positional encodings you calculated. You will then feed your encoded embeddings to a stack of Encoder layers. \n",
    "\n",
    "<img src=\"encoder.png\" alt=\"Encoder\" width=\"330\"/>\n",
    "<caption><center><font color='purple'><b>Figure 2b: Transformer Encoder</font></center></caption>\n",
    "\n",
    "\n",
    "<a name='ex-5'></a>\n",
    "### Exercise 5 - Encoder\n",
    "\n",
    "Complete the `Encoder()` function using the `call()` method to embed your input, add positional encoding, and implement multiple encoder layers \n",
    "\n",
    "In this exercise, you will initialize your Encoder with an Embedding layer, positional encoding, and multiple EncoderLayers. Your `call()` method will perform the following steps: \n",
    "1. Pass your input through the Embedding layer.\n",
    "2. Scale your embedding by multiplying it by the square root of your embedding dimension. Remember to cast the embedding dimension to data type `tf.float32` before computing the square root.\n",
    "3. Add the position encoding: self.pos_encoding `[:, :seq_len, :]` to your embedding.\n",
    "4. Pass the encoded embedding through a dropout layer, remembering to use the `training` parameter to set the model training mode. \n",
    "5. Pass the output of the dropout layer through the stack of encoding layers using a for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "7j2Tjr0K0t0I"
   },
   "outputs": [],
   "source": [
    " # UNQ_C5 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION\n",
    "class Encoder(tf.keras.layers.Layer):\n",
    "    \"\"\"\n",
    "    The entire Encoder starts by passing the input to an embedding layer \n",
    "    and using positional encoding to then pass the output through a stack of\n",
    "    encoder Layers\n",
    "        \n",
    "    \"\"\"   \n",
    "    def __init__(self, num_layers, embedding_dim, num_heads, fully_connected_dim, input_vocab_size,\n",
    "               maximum_position_encoding, dropout_rate=0.1, layernorm_eps=1e-6):\n",
    "        super(Encoder, self).__init__()\n",
    "\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.num_layers = num_layers\n",
    "\n",
    "        self.embedding = Embedding(input_vocab_size, self.embedding_dim)\n",
    "        self.pos_encoding = positional_encoding(maximum_position_encoding, \n",
    "                                                self.embedding_dim)\n",
    "\n",
    "\n",
    "        self.enc_layers = [EncoderLayer(embedding_dim=self.embedding_dim,\n",
    "                                        num_heads=num_heads,\n",
    "                                        fully_connected_dim=fully_connected_dim,\n",
    "                                        dropout_rate=dropout_rate,\n",
    "                                        layernorm_eps=layernorm_eps) \n",
    "                           for _ in range(self.num_layers)]\n",
    "\n",
    "        self.dropout = Dropout(dropout_rate)\n",
    "        \n",
    "    def call(self, x, training, mask):\n",
    "        \"\"\"\n",
    "        Forward pass for the Encoder\n",
    "        \n",
    "        Arguments:\n",
    "            x -- Tensor of shape (batch_size, input_seq_len)\n",
    "            training -- Boolean, set to true to activate\n",
    "                        the training mode for dropout layers\n",
    "            mask -- Boolean mask to ensure that the padding is not \n",
    "                    treated as part of the input\n",
    "        Returns:\n",
    "            out2 -- Tensor of shape (batch_size, input_seq_len, fully_connected_dim)\n",
    "        \"\"\"\n",
    "\n",
    "        seq_len = tf.shape(x)[1]\n",
    "        \n",
    "        # START CODE HERE\n",
    "        # Pass input through the Embedding layer\n",
    "        x = self.embedding(x)  # (batch_size, input_seq_len, fully_connected_dim)\n",
    "        # Scale embedding by multiplying it by the square root of the embedding dimension\n",
    "        x *= tf.math.sqrt(tf.cast(self.embedding_dim, tf.float32))\n",
    "        # Add the position encoding to embedding\n",
    "        x += self.pos_encoding[:, :seq_len, :]\n",
    "        # Pass the encoded embedding through a dropout layer\n",
    "        x = self.dropout(x, training = training)\n",
    "        # Pass the output through the stack of encoding layers \n",
    "        for i in range(self.num_layers):\n",
    "            x = self.enc_layers[i](x, training, mask)\n",
    "        # END CODE HERE\n",
    "\n",
    "        return x  # (batch_size, input_seq_len, fully_connected_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mAll tests passed\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def Encoder_test(target):\n",
    "    tf.random.set_seed(10)\n",
    "    \n",
    "    embedding_dim=4\n",
    "    \n",
    "    encoderq = target(num_layers=2,\n",
    "                      embedding_dim=embedding_dim,\n",
    "                      num_heads=2,\n",
    "                      fully_connected_dim=8,\n",
    "                      input_vocab_size=32,\n",
    "                      maximum_position_encoding=5)\n",
    "    \n",
    "    x = np.array([[2, 1, 3], [1, 2, 0]])\n",
    "    \n",
    "    encoderq_output = encoderq(x, True, None)\n",
    "    \n",
    "    assert tf.is_tensor(encoderq_output), \"Wrong type. Output must be a tensor\"\n",
    "    assert tuple(tf.shape(encoderq_output).numpy()) == (x.shape[0], x.shape[1], embedding_dim), f\"Wrong shape. We expected ({eshape[0]}, {eshape[1]}, {embedding_dim})\"\n",
    "    assert np.allclose(encoderq_output.numpy(), \n",
    "                       [[[-0.40172306,  0.11519244, -1.2322885,   1.5188192 ],\n",
    "                         [ 0.4017268,   0.33922842, -1.6836855,   0.9427304 ],\n",
    "                         [ 0.4685002,  -1.6252842,   0.09368491,  1.063099  ]],\n",
    "                        [[-0.3489219,   0.31335592, -1.3568854,   1.3924513 ],\n",
    "                         [-0.08761203, -0.1680029,  -1.2742313,   1.5298463 ],\n",
    "                         [ 0.2627198,  -1.6140151,   0.2212624 ,  1.130033  ]]]), \"Wrong values\"\n",
    "    \n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "    \n",
    "Encoder_test(Encoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='5'></a>\n",
    "## 5 - Decoder\n",
    "\n",
    "The Decoder layer takes the K and V matrices generated by the Encoder and in computes the second multi-head attention layer with the Q matrix from the output (Figure 3a).\n",
    "\n",
    "<img src=\"decoder_layer.png\" alt=\"Encoder\" width=\"250\"/>\n",
    "<caption><center><font color='purple'><b>Figure 3a: Transformer Decoder layer</font></center></caption>\n",
    "\n",
    "<a name='5-1'></a>    \n",
    "### 5.1 - Decoder Layer\n",
    "Again, you'll pair multi-head attention with a feed forward neural network, but this time you'll implement two multi-head attention layers. You will also use residual connections and layer normalization to help speed up training (Figure 3a).\n",
    "\n",
    "<a name='ex-6'></a>    \n",
    "### Exercise 6 - DecoderLayer\n",
    "    \n",
    "Implement `DecoderLayer()` using the `call()` method\n",
    "    \n",
    "1. Block 1 is a multi-head attention layer with a residual connection, dropout layer, and look-ahead mask.\n",
    "2. Block 2 will take into account the output of the Encoder, so the multi-head attention layer will receive K and V from the encoder, and Q from the Block 1. You will then apply a dropout layer, layer normalization and a residual connection, just like you've done before. \n",
    "3. Finally, Block 3 is a feed forward neural network with dropout and normalization layers and a residual connection.\n",
    "    \n",
    "**Additional Hints:**\n",
    "* The first two blocks are fairly similar to the EncoderLayer except you will return `attention_scores` when computing self-attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "id": "wEouNFvCzMeT"
   },
   "outputs": [],
   "source": [
    "# UNQ_C6 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION DecoderLayer\n",
    "class DecoderLayer(tf.keras.layers.Layer):\n",
    "    \"\"\"\n",
    "    The decoder layer is composed by two multi-head attention blocks, \n",
    "    one that takes the new input and uses self-attention, and the other \n",
    "    one that combines it with the output of the encoder, followed by a\n",
    "    fully connected block. \n",
    "    \"\"\"\n",
    "    def __init__(self, embedding_dim, num_heads, fully_connected_dim, dropout_rate=0.1, layernorm_eps=1e-6):\n",
    "        super(DecoderLayer, self).__init__()\n",
    "\n",
    "        self.mha1 = MultiHeadAttention(num_heads=num_heads,\n",
    "                                      key_dim=embedding_dim)\n",
    "\n",
    "        self.mha2 = MultiHeadAttention(num_heads=num_heads,\n",
    "                                      key_dim=embedding_dim)\n",
    "\n",
    "        self.ffn = FullyConnected(embedding_dim=embedding_dim,\n",
    "                                  fully_connected_dim=fully_connected_dim)\n",
    "\n",
    "        self.layernorm1 = LayerNormalization(epsilon=layernorm_eps)\n",
    "        self.layernorm2 = LayerNormalization(epsilon=layernorm_eps)\n",
    "        self.layernorm3 = LayerNormalization(epsilon=layernorm_eps)\n",
    "\n",
    "        self.dropout1 = Dropout(dropout_rate)\n",
    "        self.dropout2 = Dropout(dropout_rate)\n",
    "        self.dropout3 = Dropout(dropout_rate)\n",
    "    \n",
    "    def call(self, x, enc_output, training, look_ahead_mask, padding_mask):\n",
    "        \"\"\"\n",
    "        Forward pass for the Decoder Layer\n",
    "        \n",
    "        Arguments:\n",
    "            x -- Tensor of shape (batch_size, target_seq_len, fully_connected_dim)\n",
    "            enc_output --  Tensor of shape(batch_size, input_seq_len, fully_connected_dim)\n",
    "            training -- Boolean, set to true to activate\n",
    "                        the training mode for dropout layers\n",
    "            look_ahead_mask -- Boolean mask for the target_input\n",
    "            padding_mask -- Boolean mask for the second multihead attention layer\n",
    "        Returns:\n",
    "            out3 -- Tensor of shape (batch_size, target_seq_len, fully_connected_dim)\n",
    "            attn_weights_block1 -- Tensor of shape(batch_size, num_heads, target_seq_len, input_seq_len)\n",
    "            attn_weights_block2 -- Tensor of shape(batch_size, num_heads, target_seq_len, input_seq_len)\n",
    "        \"\"\"\n",
    "        \n",
    "        # START CODE HERE\n",
    "        # enc_output.shape == (batch_size, input_seq_len, fully_connected_dim)\n",
    "        \n",
    "        # BLOCK 1\n",
    "        # calculate self-attention and return attention scores as attn_weights_block1 (~1 line)\n",
    "        attn1, attn_weights_block1 = self.mha1(x, x, x, look_ahead_mask, return_attention_scores=True)  # (batch_size, target_seq_len, d_model)\n",
    "        \n",
    "        # apply dropout layer on the attention output (~1 line)\n",
    "        attn1 = self.dropout1(attn1, training=training)\n",
    "        \n",
    "        # apply layer normalization to the sum of the attention output and the input (~1 line)\n",
    "        \n",
    "        out1 = self.layernorm1(attn1 + x)\n",
    "\n",
    "        # BLOCK 2\n",
    "        # calculate self-attention using the Q from the first block and K and V from the encoder output.\n",
    "        # Return attention scores as attn_weights_block2 (~1 line)\n",
    "        attn2, attn_weights_block2 = self.mha2(enc_output, enc_output, out1, padding_mask, return_attention_scores=True)  # (batch_size, target_seq_len, d_model)\n",
    "        \n",
    "        # apply dropout layer on the attention output (~1 line)\n",
    "        attn2 = self.dropout2(attn2, training=training)\n",
    "        \n",
    "        # apply layer normalization to the sum of the attention output and the output of the first block (~1 line)\n",
    "        out2 = self.layernorm2(attn2 + out1)  # (batch_size, target_seq_len, fully_connected_dim)\n",
    "        \n",
    "        #BLOCK 3\n",
    "        # pass the output of the second block through a ffn\n",
    "        ffn_output = self.ffn(out2) # (batch_size, target_seq_len, fully_connected_dim)\n",
    "        \n",
    "        # apply a dropout layer to the ffn output\n",
    "        ffn_output = self.dropout3(ffn_output, training=training)\n",
    "        \n",
    "        # apply layer normalization to the sum of the ffn output and the output of the second block\n",
    "        out3 = self.layernorm3(ffn_output + out2) # (batch_size, target_seq_len, fully_connected_dim)\n",
    "        # END CODE HERE\n",
    "\n",
    "        return out3, attn_weights_block1, attn_weights_block2\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mAll tests passed\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def DecoderLayer_test(target):\n",
    "    \n",
    "    num_heads=8\n",
    "    tf.random.set_seed(10)\n",
    "    \n",
    "    decoderLayerq = target(\n",
    "        embedding_dim=4, \n",
    "        num_heads=num_heads,\n",
    "        fully_connected_dim=32, \n",
    "        dropout_rate=0.1, \n",
    "        layernorm_eps=1e-6)\n",
    "    \n",
    "    encoderq_output = tf.constant([[[-0.40172306,  0.11519244, -1.2322885,   1.5188192 ],\n",
    "                                   [ 0.4017268,   0.33922842, -1.6836855,   0.9427304 ],\n",
    "                                   [ 0.4685002,  -1.6252842,   0.09368491,  1.063099  ]]])\n",
    "    \n",
    "    q = np.array([[[1, 0, 1, 1], [0, 1, 1, 1], [1, 0, 0, 1]]]).astype(np.float32)\n",
    "    \n",
    "    look_ahead_mask = tf.constant([[0., 1., 1.],\n",
    "                       [0., 0., 1.],\n",
    "                       [0., 0., 0.]])\n",
    "    \n",
    "    padding_mask = None\n",
    "    out, attn_w_b1, attn_w_b2 = decoderLayerq(q, encoderq_output, True, look_ahead_mask, padding_mask)\n",
    "    \n",
    "    assert tf.is_tensor(attn_w_b1), \"Wrong type for attn_w_b1. Output must be a tensor\"\n",
    "    assert tf.is_tensor(attn_w_b2), \"Wrong type for attn_w_b2. Output must be a tensor\"\n",
    "    assert tf.is_tensor(out), \"Wrong type for out. Output must be a tensor\"\n",
    "    \n",
    "    shape1 = (q.shape[0], num_heads, q.shape[1], q.shape[1])\n",
    "    assert tuple(tf.shape(attn_w_b1).numpy()) == shape1, f\"Wrong shape. We expected {shape1}\"\n",
    "    assert tuple(tf.shape(attn_w_b2).numpy()) == shape1, f\"Wrong shape. We expected {shape1}\"\n",
    "    assert tuple(tf.shape(out).numpy()) == q.shape, f\"Wrong shape. We expected {q.shape}\"\n",
    "\n",
    "    assert np.allclose(attn_w_b1[0, 0, 0], [0, 0.5, 0.5], atol=1e-2), \"Wrong values in attn_w_b1. Check the call to self.mha1\"\n",
    "    #assert np.allclose(attn_w_b2[0, 0, 1], [0.34485385, 0.33230072, 0.32284543]),  \"Wrong values in attn_w_b2. Check the call to self.mha2\"\n",
    "    #assert np.allclose(out[0, 0], [0.64775777, -1.5134472,   1.1092964,  -0.24360693]), \"Wrong values in out\"\n",
    "    \n",
    "\n",
    "    # Now let's try a example with padding mask\n",
    "    padding_mask = np.array([[0, 0, 1]])\n",
    "    out, attn_w_b1, attn_w_b2 = decoderLayerq(q, encoderq_output, True, look_ahead_mask, padding_mask)\n",
    "\n",
    "    assert np.allclose(out[0, 0], [0.59296525, -1.4068702, 1.224841, -0.410936]), \"Wrong values in out when we mask the last word. Are you passing the padding_mask to the inner functions?\"\n",
    "\n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "    \n",
    "DecoderLayer_test(DecoderLayer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='5-2'></a> \n",
    "### 5.2 - Full Decoder\n",
    "You're almost there! Time to use your Decoder layer to build a full Transformer Decoder (Figure 3b). You will embedd your output and add positional encodings. You will then feed your encoded embeddings to a stack of Decoder layers. \n",
    "\n",
    "\n",
    "<img src=\"decoder.png\" alt=\"Encoder\" width=\"300\"/>\n",
    "<caption><center><font color='purple'><b>Figure 3b: Transformer Decoder</font></center></caption>\n",
    "\n",
    "<a name='ex-7'></a>     \n",
    "### Exercise 7 - Decoder\n",
    "\n",
    "Implement `Decoder()` using the `call()` method to embed your output, add positional encoding, and implement multiple decoder layers\n",
    " \n",
    "In this exercise, you will initialize your Decoder with an Embedding layer, positional encoding, and multiple DecoderLayers. Your `call()` method will perform the following steps: \n",
    "1. Pass your generated output through the Embedding layer.\n",
    "2. Scale your embedding by multiplying it by the square root of your embedding dimension. Remember to cast the embedding dimension to data type `tf.float32` before computing the square root.\n",
    "3. Add the position encoding: self.pos_encoding `[:, :seq_len, :]` to your embedding.\n",
    "4. Pass the encoded embedding through a dropout layer, remembering to use the `training` parameter to set the model training mode. \n",
    "5. Pass the output of the dropout layer through the stack of Decoding layers using a for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "id": "McS3by6k4pnP"
   },
   "outputs": [],
   "source": [
    "# UNQ_C7 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION Decoder\n",
    "class Decoder(tf.keras.layers.Layer):\n",
    "    \"\"\"\n",
    "    The entire Encoder is starts by passing the target input to an embedding layer \n",
    "    and using positional encoding to then pass the output through a stack of\n",
    "    decoder Layers\n",
    "        \n",
    "    \"\"\" \n",
    "    def __init__(self, num_layers, embedding_dim, num_heads, fully_connected_dim, target_vocab_size,\n",
    "               maximum_position_encoding, dropout_rate=0.1, layernorm_eps=1e-6):\n",
    "        super(Decoder, self).__init__()\n",
    "\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.num_layers = num_layers\n",
    "\n",
    "        self.embedding = Embedding(target_vocab_size, self.embedding_dim)\n",
    "        self.pos_encoding = positional_encoding(maximum_position_encoding, self.embedding_dim)\n",
    "\n",
    "        self.dec_layers = [DecoderLayer(embedding_dim=self.embedding_dim,\n",
    "                                        num_heads=num_heads,\n",
    "                                        fully_connected_dim=fully_connected_dim,\n",
    "                                        dropout_rate=dropout_rate,\n",
    "                                        layernorm_eps=layernorm_eps) \n",
    "                           for _ in range(self.num_layers)]\n",
    "        self.dropout = Dropout(dropout_rate)\n",
    "    \n",
    "    def call(self, x, enc_output, training, \n",
    "           look_ahead_mask, padding_mask):\n",
    "        \"\"\"\n",
    "        Forward  pass for the Decoder\n",
    "        \n",
    "        Arguments:\n",
    "            x -- Tensor of shape (batch_size, target_seq_len, fully_connected_dim)\n",
    "            enc_output --  Tensor of shape(batch_size, input_seq_len, fully_connected_dim)\n",
    "            training -- Boolean, set to true to activate\n",
    "                        the training mode for dropout layers\n",
    "            look_ahead_mask -- Boolean mask for the target_input\n",
    "            padding_mask -- Boolean mask for the second multihead attention layer\n",
    "        Returns:\n",
    "            x -- Tensor of shape (batch_size, target_seq_len, fully_connected_dim)\n",
    "            attention_weights - Dictionary of tensors containing all the attention weights\n",
    "                                each of shape Tensor of shape (batch_size, num_heads, target_seq_len, input_seq_len)\n",
    "        \"\"\"\n",
    "\n",
    "        seq_len = tf.shape(x)[1]\n",
    "        attention_weights = {}\n",
    "        \n",
    "        # START CODE HERE\n",
    "        # create word embeddings \n",
    "        x = self.embedding(x)  # (batch_size, target_seq_len, fully_connected_dim)\n",
    "        \n",
    "        # scale embeddings by multiplying by the square root of their dimension\n",
    "        x *= tf.sqrt(tf.cast(self.embedding_dim, tf.float32))\n",
    "        \n",
    "        # calculate positional encodings and add to word embedding\n",
    "        x += self.pos_encoding[:, :seq_len, :]\n",
    "        \n",
    "        # apply a dropout layer to x\n",
    "        x = self.dropout(x, training=training)\n",
    "        \n",
    "        # use a for loop to pass x through a stack of decoder layers and update attention_weights (~4 lines total)\n",
    "        for i in range(self.num_layers):\n",
    "            # pass x and the encoder output through a stack of decoder layers and save the attention weights\n",
    "            # of block 1 and 2 (~1 line)\n",
    "            x, block1, block2 = self.dec_layers[i](x, enc_output, training, look_ahead_mask, padding_mask)\n",
    "            \n",
    "            #update attention_weights dictionary with the attention weights of block 1 and block 2\n",
    "            attention_weights['decoder_layer{}_block1_self_att'.format(i+1)] = block1\n",
    "            attention_weights['decoder_layer{}_block2_decenc_att'.format(i+1)] = block2\n",
    "\n",
    "        # END CODE HERE\n",
    "                       \n",
    "        # x.shape == (batch_size, target_seq_len, fully_connected_dim)\n",
    "        return x, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[-7.3709977e-01  1.4761090e-03 -8.9435089e-01  1.6299746e+00]\n",
      "  [-3.9448336e-01 -8.7317646e-01 -4.3351787e-01  1.7011778e+00]\n",
      "  [-3.1102329e-01 -1.2398847e+00  1.2797236e-02  1.5381110e+00]]\n",
      "\n",
      " [[-8.0389977e-01  9.5988214e-02 -8.8982087e-01  1.5977323e+00]\n",
      "  [-3.4560359e-01 -8.7628973e-01 -4.7674853e-01  1.6986420e+00]\n",
      "  [-2.4897528e-01 -1.2377455e+00 -6.3121378e-02  1.5498422e+00]]], shape=(2, 3, 4), dtype=float32)\n",
      "\u001b[92mAll tests passed\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def Decoder_test(target):\n",
    "    \n",
    "    tf.random.set_seed(10)\n",
    "        \n",
    "    num_layers=7\n",
    "    embedding_dim=4 \n",
    "    num_heads=3\n",
    "    fully_connected_dim=8\n",
    "    target_vocab_size=33\n",
    "    maximum_position_encoding=6\n",
    "    \n",
    "    x = np.array([[3, 2, 1], [2, 1, 0]])\n",
    "\n",
    "    \n",
    "    encoderq_output = tf.constant([[[-0.40172306,  0.11519244, -1.2322885,   1.5188192 ],\n",
    "                         [ 0.4017268,   0.33922842, -1.6836855,   0.9427304 ],\n",
    "                         [ 0.4685002,  -1.6252842,   0.09368491,  1.063099  ]],\n",
    "                        [[-0.3489219,   0.31335592, -1.3568854,   1.3924513 ],\n",
    "                         [-0.08761203, -0.1680029,  -1.2742313,   1.5298463 ],\n",
    "                         [ 0.2627198,  -1.6140151,   0.2212624 ,  1.130033  ]]])\n",
    "    \n",
    "    look_ahead_mask = tf.constant([[0., 1., 1.],\n",
    "                       [0., 0., 1.],\n",
    "                       [0., 0., 0.]])\n",
    "    \n",
    "    decoderk = Decoder(num_layers,\n",
    "                    embedding_dim, \n",
    "                    num_heads, \n",
    "                    fully_connected_dim,\n",
    "                    target_vocab_size,\n",
    "                    maximum_position_encoding)\n",
    "    outd, att_weights = decoderk(x, encoderq_output, False, look_ahead_mask, None)\n",
    "    \n",
    "    assert tf.is_tensor(outd), \"Wrong type for outd. It must be a dict\"\n",
    "    assert np.allclose(tf.shape(outd), tf.shape(encoderq_output)), f\"Wrong shape. We expected { tf.shape(encoderq_output)}\"\n",
    "    assert np.allclose(outd[1, 1], [-0.34560338, -0.8762897,  -0.4767484,   1.6986415]), \"Wrong values in outd\"\n",
    "    \n",
    "    keys = list(att_weights.keys())\n",
    "    assert type(att_weights) == dict, \"Wrong type for att_weights[0]. Output must be a tensor\"\n",
    "    assert len(keys) == 2 * num_layers, f\"Wrong length for attention weights. It must be 2 x num_layers = {2*num_layers}\"\n",
    "    assert tf.is_tensor(att_weights[keys[0]]), f\"Wrong type for att_weights[{keys[0]}]. Output must be a tensor\"\n",
    "    shape1 = (x.shape[0], num_heads, x.shape[1], x.shape[1])\n",
    "    assert tuple(tf.shape(att_weights[keys[1]]).numpy()) == shape1, f\"Wrong shape. We expected {shape1}\" \n",
    "    assert np.allclose(att_weights[keys[0]][0, 0, 1], [0., 0., 1.]), f\"Wrong values in att_weights[{keys[0]}]\"\n",
    "    \n",
    "    print(outd)\n",
    "   \n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "    \n",
    "Decoder_test(Decoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='6'></a> \n",
    "## 6 - Transformer\n",
    "\n",
    "Phew! This has been quite the assignment, and now you've made it to your last exercise of the Deep Learning Specialization. Congratulations! You've done all the hard work, now it's time to put it all together.  \n",
    "\n",
    "<img src=\"transformer.png\" alt=\"Transformer\" width=\"550\"/>\n",
    "<caption><center><font color='purple'><b>Figure 4: Transformer</font></center></caption>\n",
    "    \n",
    "The flow of data through the Transformer Architecture is as follows:\n",
    "* First your input passes through an Encoder, which is just repeated Encoder layers that you implemented:\n",
    "    - embedding and positional encoding of your input\n",
    "    - multi-head attention on your input\n",
    "    - feed forward neural network to help detect features\n",
    "* Then the predicted output passes through a Decoder, consisting of the decoder layers that you implemented:\n",
    "    - embedding and positional encoding of the output\n",
    "    - multi-head attention on your generated output\n",
    "    - multi-head attention with the Q from the first multi-head attention layer and the K and V from the Encoder\n",
    "    - a feed forward neural network to help detect features\n",
    "* Finally, after the Nth Decoder layer, two dense layers and a softmax are applied to generate prediction for the next output in your sequence.\n",
    "\n",
    "<a name='ex-8'></a> \n",
    "### Exercise 8 - Transformer\n",
    "\n",
    "Implement `Transformer()` using the `call()` method\n",
    "1. Pass the input through the Encoder with the appropiate mask.\n",
    "2. Pass the encoder output and the target through the Decoder with the appropiate mask.\n",
    "3. Apply a linear transformation and a softmax to get a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "id": "QHymPmaj-2ba"
   },
   "outputs": [],
   "source": [
    "# UNQ_C8 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "# GRADED FUNCTION Transformer\n",
    "class Transformer(tf.keras.Model):\n",
    "    \"\"\"\n",
    "    Complete transformer with an Encoder and a Decoder\n",
    "    \"\"\"\n",
    "    def __init__(self, num_layers, embedding_dim, num_heads, fully_connected_dim, input_vocab_size, \n",
    "               target_vocab_size, max_positional_encoding_input,\n",
    "               max_positional_encoding_target, dropout_rate=0.1, layernorm_eps=1e-6):\n",
    "        super(Transformer, self).__init__()\n",
    "\n",
    "        self.encoder = Encoder(num_layers=num_layers,\n",
    "                               embedding_dim=embedding_dim,\n",
    "                               num_heads=num_heads,\n",
    "                               fully_connected_dim=fully_connected_dim,\n",
    "                               input_vocab_size=input_vocab_size,\n",
    "                               maximum_position_encoding=max_positional_encoding_input,\n",
    "                               dropout_rate=dropout_rate,\n",
    "                               layernorm_eps=layernorm_eps)\n",
    "\n",
    "        self.decoder = Decoder(num_layers=num_layers, \n",
    "                               embedding_dim=embedding_dim,\n",
    "                               num_heads=num_heads,\n",
    "                               fully_connected_dim=fully_connected_dim,\n",
    "                               target_vocab_size=target_vocab_size, \n",
    "                               maximum_position_encoding=max_positional_encoding_target,\n",
    "                               dropout_rate=dropout_rate,\n",
    "                               layernorm_eps=layernorm_eps)\n",
    "\n",
    "        self.final_layer = Dense(target_vocab_size, activation='softmax')\n",
    "    \n",
    "    def call(self, inp, tar, training, enc_padding_mask, look_ahead_mask, dec_padding_mask):\n",
    "        \"\"\"\n",
    "        Forward pass for the entire Transformer\n",
    "        Arguments:\n",
    "            inp -- Tensor of shape (batch_size, input_seq_len, fully_connected_dim)\n",
    "            tar -- Tensor of shape (batch_size, target_seq_len, fully_connected_dim)\n",
    "            training -- Boolean, set to true to activate\n",
    "                        the training mode for dropout layers\n",
    "            enc_padding_mask -- Boolean mask to ensure that the padding is not \n",
    "                    treated as part of the input\n",
    "            look_ahead_mask -- Boolean mask for the target_input\n",
    "            padding_mask -- Boolean mask for the second multihead attention layer\n",
    "        Returns:\n",
    "            final_output -- Describe me\n",
    "            attention_weights - Dictionary of tensors containing all the attention weights for the decoder\n",
    "                                each of shape Tensor of shape (batch_size, num_heads, target_seq_len, input_seq_len)\n",
    "        \n",
    "        \"\"\"\n",
    "        # START CODE HERE\n",
    "        # call self.encoder with the appropriate arguments to get the encoder output\n",
    "        enc_output = self.encoder(inp, training, enc_padding_mask) # (batch_size, inp_seq_len, fully_connected_dim)\n",
    "        \n",
    "        # call self.decoder with the appropriate arguments to get the decoder output\n",
    "        # dec_output.shape == (batch_size, tar_seq_len, fully_connected_dim)\n",
    "        dec_output, attention_weights = self.decoder(tar, enc_output, training, look_ahead_mask, dec_padding_mask)\n",
    "        \n",
    "        # pass decoder output through a linear layer and softmax (~2 lines)\n",
    "        \n",
    "        final_output = self.final_layer(dec_output)  # (batch_size, tar_seq_len, target_vocab_size)\n",
    "        # START CODE HERE\n",
    "\n",
    "        return final_output, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[0.02664841 0.02223665 0.01641649 0.02405972 0.04249557 0.02241551\n",
      "   0.01557002 0.03742596 0.02217453 0.03072541 0.01656803 0.02322329\n",
      "   0.05625562 0.02473563 0.01231006 0.03448053 0.02488913 0.01711017\n",
      "   0.02859487 0.02613499 0.04459638 0.06464051 0.03617646 0.02876464\n",
      "   0.05091118 0.01344999 0.01690798 0.01745267 0.02632003 0.0154627\n",
      "   0.06245003 0.02078524 0.02991351 0.02974838 0.01794973]\n",
      "  [0.02563646 0.01816349 0.0174562  0.02869465 0.04840349 0.02019639\n",
      "   0.01510986 0.03421774 0.02908056 0.02632561 0.01860591 0.01857974\n",
      "   0.05862088 0.03149266 0.01406298 0.02900651 0.02613457 0.01670403\n",
      "   0.02512758 0.02339464 0.04113115 0.0577657  0.03649407 0.03318116\n",
      "   0.04666907 0.01666715 0.01447459 0.01921809 0.02358727 0.01779257\n",
      "   0.06112151 0.0217419  0.03136224 0.0334071  0.0203725 ]\n",
      "  [0.0214973  0.01378816 0.02101264 0.03667997 0.0587284  0.0192939\n",
      "   0.01660215 0.03171616 0.04478166 0.02514142 0.01826592 0.01312409\n",
      "   0.05914298 0.03954394 0.01734127 0.02432951 0.03002167 0.01807637\n",
      "   0.02360681 0.02182439 0.04077699 0.04356125 0.04159507 0.0367234\n",
      "   0.0452871  0.01680262 0.01237727 0.02332873 0.01574075 0.02045165\n",
      "   0.04768985 0.02178313 0.02552259 0.03228886 0.021552  ]\n",
      "  [0.01894131 0.01043008 0.02689406 0.04792543 0.0595998  0.01734336\n",
      "   0.01901497 0.02429437 0.07047241 0.01926193 0.0233914  0.00936925\n",
      "   0.04702715 0.0548186  0.0266147  0.01694931 0.03262443 0.02012189\n",
      "   0.01896166 0.01839592 0.03035553 0.02565147 0.03814695 0.04185173\n",
      "   0.03186765 0.02592374 0.01116573 0.03050922 0.01205521 0.02878864\n",
      "   0.03175847 0.02405321 0.02364573 0.03398603 0.02778864]\n",
      "  [0.01738627 0.01146928 0.02862458 0.04561693 0.06185422 0.02028133\n",
      "   0.02166363 0.028173   0.06791075 0.02550283 0.01815756 0.00996597\n",
      "   0.04871172 0.04548143 0.02472349 0.02047577 0.03525177 0.02251919\n",
      "   0.02337657 0.02155913 0.03783516 0.02634462 0.04569449 0.03704556\n",
      "   0.03979248 0.01688794 0.01242078 0.03089081 0.00990537 0.02505881\n",
      "   0.02846423 0.02225632 0.01813283 0.02749651 0.02306868]]], shape=(1, 5, 35), dtype=float32)\n",
      "\u001b[92mAll tests passed\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def Transformer_test(target):\n",
    "    \n",
    "    tf.random.set_seed(10)\n",
    "\n",
    "\n",
    "    num_layers = 6\n",
    "    embedding_dim = 4\n",
    "    num_heads = 4\n",
    "    fully_connected_dim = 8\n",
    "    input_vocab_size = 30\n",
    "    target_vocab_size = 35\n",
    "    max_positional_encoding_input = 5\n",
    "    max_positional_encoding_target = 6\n",
    "\n",
    "    trans = Transformer(num_layers, \n",
    "                        embedding_dim, \n",
    "                        num_heads, \n",
    "                        fully_connected_dim, \n",
    "                        input_vocab_size, \n",
    "                        target_vocab_size, \n",
    "                        max_positional_encoding_input,\n",
    "                        max_positional_encoding_target)\n",
    "    # 0 is the padding value\n",
    "    sentence_lang_a = np.array([[2, 1, 4, 3, 0]])\n",
    "    sentence_lang_b = np.array([[3, 2, 1, 0, 0]])\n",
    "\n",
    "    enc_padding_mask = np.array([[0, 0, 0, 0, 1]])\n",
    "    dec_padding_mask = np.array([[0, 0, 0, 1, 1]])\n",
    "\n",
    "    look_ahead_mask = create_look_ahead_mask(sentence_lang_a.shape[1])\n",
    "\n",
    "    translation, weights = trans(\n",
    "        sentence_lang_a,\n",
    "        sentence_lang_b,\n",
    "        True,\n",
    "        enc_padding_mask,\n",
    "        look_ahead_mask,\n",
    "        dec_padding_mask\n",
    "    )\n",
    "    \n",
    "    \n",
    "    assert tf.is_tensor(translation), \"Wrong type for translation. Output must be a tensor\"\n",
    "    shape1 = (sentence_lang_a.shape[0], max_positional_encoding_input, target_vocab_size)\n",
    "    assert tuple(tf.shape(translation).numpy()) == shape1, f\"Wrong shape. We expected {shape1}\"\n",
    "        \n",
    "    assert np.allclose(translation[0, 0, 0:8],\n",
    "                       [[0.02664841, 0.02223665, 0.01641649, 0.02405972,\n",
    "                         0.04249557, 0.02241551, 0.01557002, 0.03742596]]), \"Wrong values in outd\"\n",
    "    \n",
    "    keys = list(weights.keys())\n",
    "    assert type(weights) == dict, \"Wrong type for weights. It must be a dict\"\n",
    "    assert len(keys) == 2 * num_layers, f\"Wrong length for attention weights. It must be 2 x num_layers = {2*num_layers}\"\n",
    "    assert tf.is_tensor(weights[keys[0]]), f\"Wrong type for att_weights[{keys[0]}]. Output must be a tensor\"\n",
    "   \n",
    "    shape1 = (sentence_lang_a.shape[0], num_heads, sentence_lang_a.shape[1], sentence_lang_a.shape[1])\n",
    "    assert tuple(tf.shape(weights[keys[1]]).numpy()) == shape1, f\"Wrong shape. We expected {shape1}\" \n",
    "    assert np.allclose(weights[keys[0]][0, 0, 1], [0., 0., 0.31332517, 0.33401096, 0.35266393]), f\"Wrong values in weights[{keys[0]}]\"\n",
    "    \n",
    "    print(translation)\n",
    "    \n",
    "    print(\"\\033[92mAll tests passed\")\n",
    "\n",
    "    \n",
    "Transformer_test(Transformer)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "You've come to the end of the graded portion of the assignment. By now, you've: \n",
    "\n",
    "* Create positional encodings to capture sequential relationships in data\n",
    "* Calculate scaled dot-product self-attention with word embeddings\n",
    "* Implement masked multi-head attention\n",
    "* Build and train a Transformer model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='blue'>\n",
    "    <b>What you should remember</b>:\n",
    "\n",
    "- The combination of self-attention and convolutional network layers allows of parallization of training and *faster training*.\n",
    "- Self-attention is calculated using the generated query Q, key K, and value V matrices.\n",
    "- Adding positional encoding to word embeddings is an effective way of include sequence information in self-attention calculations. \n",
    "- Multi-head attention can help detect multiple features in your sentence.\n",
    "- Masking stops the model from 'looking ahead' during training, or weighting zeroes too much when processing cropped sentences. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have completed the Transformer assignment, make sure you check out the ungraded labs to apply the Transformer model to practical use cases such as Name Entity Recogntion (NER) and Question Answering (QA).  \n",
    "\n",
    "\n",
    "# Congratulations on finishing the Deep Learning Specialization!!!!!! 🎉\n",
    "\n",
    "This was the last graded assignment of the specialization. It is now time to celebrate all your hard work and dedication! \n",
    "\n",
    "<a name='7'></a> \n",
    "## 7 - References\n",
    "\n",
    "The Transformer algorithm was due to Vaswani et al. (2017). \n",
    "\n",
    "- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (2017). [Attention Is All You Need](https://arxiv.org/abs/1706.03762) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Transformer Assignment - Subclass.ipynb",
   "provenance": []
  },
  "coursera": {
   "schema_names": [
    "DLSC5W4-1A"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "08b8eb7e54b0420692f439151110ea78": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3a104536dc844996864bb00e07c351ff",
      "placeholder": "​",
      "style": "IPY_MODEL_77e7f5504f1a482980058bdd07d07d5e",
      "value": " 220/220 [14:41&lt;00:00,  4.01s/it]"
     }
    },
    "0a7b94269612418ab913737ada1a606f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0f138da56eae488a85970020606e093d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f7dd6278751c440baf1b80c19cdac901",
      "placeholder": "​",
      "style": "IPY_MODEL_32f739f4c4e742bc837232c8616c85c3",
      "value": " 442/442 [00:00&lt;00:00, 1.40kB/s]"
     }
    },
    "1171e92377e74bbd855dcf2cd49cd0fc": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "156f653b1cf74c88a77e5ea66d59dddd": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "32f739f4c4e742bc837232c8616c85c3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3a104536dc844996864bb00e07c351ff": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "47460b36e9524825ab8228e5efae338e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_bf260e4350dd4037bce3badc2c076b94",
       "IPY_MODEL_08b8eb7e54b0420692f439151110ea78"
      ],
      "layout": "IPY_MODEL_156f653b1cf74c88a77e5ea66d59dddd"
     }
    },
    "4ed12cd5a3a943ae862f026e20d0e9e5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "52dbd62baaf04233a14a4278c9f5d24f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "77e7f5504f1a482980058bdd07d07d5e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8427a9c9ac78457cbdfffd41d76a467d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1171e92377e74bbd855dcf2cd49cd0fc",
      "placeholder": "​",
      "style": "IPY_MODEL_b4c3e99db1374b1b83b768dc1dc7c470",
      "value": " 363M/363M [00:06&lt;00:00, 56.4MB/s]"
     }
    },
    "84748d3f92494e839763a1ed7e3dc45e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a411c46cc2194664a08e1bc2eba903ff",
       "IPY_MODEL_0f138da56eae488a85970020606e093d"
      ],
      "layout": "IPY_MODEL_8b266584bf764912abb999040e1c4643"
     }
    },
    "8b266584bf764912abb999040e1c4643": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9b8eb55aa69341148bc0fd72a5cc383d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "a411c46cc2194664a08e1bc2eba903ff": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ae662a275a7f443c868463b275454359",
      "max": 442,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a9d8813c381c43d5a17594a067f0c8c9",
      "value": 442
     }
    },
    "a9d8813c381c43d5a17594a067f0c8c9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "ae662a275a7f443c868463b275454359": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b4c3e99db1374b1b83b768dc1dc7c470": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "bf260e4350dd4037bce3badc2c076b94": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0a7b94269612418ab913737ada1a606f",
      "max": 220,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9b8eb55aa69341148bc0fd72a5cc383d",
      "value": 220
     }
    },
    "c13de54036014187a3c9e532e15112be": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c2a1ddc5ec3d4e828d4abe111d0e3957": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_52dbd62baaf04233a14a4278c9f5d24f",
      "max": 363423424,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4ed12cd5a3a943ae862f026e20d0e9e5",
      "value": 363423424
     }
    },
    "ded975cd21d640d7847ecaadd45387b3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c2a1ddc5ec3d4e828d4abe111d0e3957",
       "IPY_MODEL_8427a9c9ac78457cbdfffd41d76a467d"
      ],
      "layout": "IPY_MODEL_c13de54036014187a3c9e532e15112be"
     }
    },
    "f7dd6278751c440baf1b80c19cdac901": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}