Transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformer for Language Translation in PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook I will go trough an PyTorch implementation of transformer based on [Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf) paper.\n",
    "\n",
    "In [another notebook](https://github.com/mf1024/transformers) I implemented Transforer Encoder model and trained it on Language Modeling task. \n",
    "\n",
    "In this notebook I will continue and add the Decoder component to the implementation and train the whole on Language Translation task.\n",
    "\n",
    "Troughout this notebook as well I will **use some of the illustrations from the incredible [Illustrated transformer](https://jalammar.github.io/illustrated-transformer) blog post by *Jay Alammar***. I highly recommend you to take a look at it. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "For the translation task I will use a small english-french [dataset](https://raw.githubusercontent.com/mf1024/transformers/master/fra-eng/fra.txt). The dataset consists of just 170k sentences but that is enough to train and demonstrate the Transformer.\n",
    "\n",
    "I will use the same PyTorch [Dataset](https://github.com/mf1024/transformers/blob/master/fra_eng_dataset.py) implementation to process the sentences and prepare the batches.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer architecture:\n",
    "\n",
    "<img src=\"imgs/transformer.png\">\n",
    "\n",
    "The key idea of Transformer is to avoid using [reccurence](https://arxiv.org/abs/1409.3215) at all for encoding and decoding variable-length sequences. That solves issues with long-range dependencies and the amount of computation that can be parallelized.\n",
    "\n",
    "The transformer consists of two parts - the Encoder and the Decoder. They are both very similar in their structure but different in a few aspects. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Encoder and Decoder\n",
    "\n",
    "<img src=\"imgs/encoder-decoder.png\">\n",
    "\n",
    "In the full Transformer implementation, first, the sentence in the source language is encoded. One the encoding is done, the output of the last encoder layer is used in every layer of decoder, where the last layer generates the sentence in target language. \n",
    "\n",
    "\n",
    "<img src=\"imgs/transformer_decoding.gif\">\n",
    "\n",
    "The Decoder layer consists of same sublayers as Encoder(Self Attention and Feed Forward) with one additional sublayer - Memory Attention. The Memory Attention sublayer is similar to Self Attention except it uses **Key** and **Value** matrices from the Encoder.\n",
    "\n",
    "The Decoder will use these **Key** and **Value** matrices but will learn to calculate it's own **Query** to gather the information from the encoded sentences.\n",
    "\n",
    "When decoding the sentence, each word is prediced one at a time, meaning that for Decoder to predict the word x(i) the input of the Decoder should be all the words in target langugae from 1 to i-1 . During the traing the decoding can be paralelized using masking, but during translation it is going to take as many Decoder runs as there are words in the target sentence as shown in the GIF. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "import torch.functional as F\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from fra_eng_dataset import FraEngDataset, fra_eng_dataset_collate\n",
    "from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence, pad_packed_sequence\n",
    "\n",
    "device = 'cpu'\n",
    "if torch.cuda.is_available():\n",
    "    device = 'cuda'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SelfAttentionHead(nn.Module):\n",
    "    def __init__(self, d_model):\n",
    "        super().__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.K = nn.Linear(d_model, d_model)\n",
    "        self.V = nn.Linear(d_model, d_model)\n",
    "        self.Q = nn.Linear(d_model, d_model)\n",
    "\n",
    "    def forward(self, x, padding_mask = None, subsq_mask = None):\n",
    "        # x shape: [N, SEQ, D_MODEL]\n",
    "\n",
    "        keys = self.K.forward(x)\n",
    "        values = self.V.forward(x)\n",
    "        queries = self.Q.forward(x)\n",
    "\n",
    "        sqrt_d = self.d_model ** 0.5\n",
    "\n",
    "        att = torch.matmul(queries, keys.transpose(1,2)) / sqrt_d\n",
    "        # att shape: [N, SEQ, SEQ]\n",
    "        # Broadcast padding mask to word attentions so that word attention does not attend to positions outside the sentence\n",
    "        if padding_mask is not None:\n",
    "            att = att + padding_mask.transpose(1,2)\n",
    "        # Add subsequent mask so that each position can attend only itself and the previous elements\n",
    "        if subsq_mask is not None:\n",
    "            att = att + subsq_mask.unsqueeze(0)\n",
    "\n",
    "        att_softmax = torch.softmax(att, dim=2)\n",
    "        # shape: [N, SEQ, SEQ]\n",
    "        att_out = torch.matmul(att_softmax, values)\n",
    "        # shape: [N, SEQ, D_MODEL]\n",
    "\n",
    "        return att_out, keys, values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Memory Attention\n",
    "Implementation is the same as Self Attention only the keys and values matrices are not calculated from weights and input but are passed to the forward function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MemAttentionHead(nn.Module):\n",
    "    def __init__(self, d_model):\n",
    "        super().__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.Q = nn.Linear(d_model, d_model)\n",
    "\n",
    "    def forward(self, x, mem_padding_mask, keys = None, values = None):\n",
    "\n",
    "        # X shape: [N, SEQ, D_MODEL]\n",
    "        queries = self.Q.forward(x)\n",
    "        sqrt_d = self.d_model ** 0.5\n",
    "\n",
    "        att = torch.matmul(queries, keys.transpose(1,2)) / sqrt_d\n",
    "        # att shape: [N, SEQ_TGT, SEQ_SRC]\n",
    "\n",
    "        # Broadcast padding mask to word attentions so that word attention does not attend to positions outside the source sentence\n",
    "        if mem_padding_mask is not None:\n",
    "            att = att + mem_padding_mask.transpose(1,2)\n",
    "\n",
    "        att_softmax = torch.softmax(att, dim=2)\n",
    "        # shape: [N, SEQ_TGT, SEQ_SRC]\n",
    "        att_out = torch.matmul(att_softmax, values)\n",
    "        # shape: [N, SEQ_TGT, D_MODEL]\n",
    "\n",
    "        return att_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-head attentions for Self Attention and Memory Attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadSelfAttention(nn.Module):\n",
    "    def __init__(self, d_model, num_heads):\n",
    "        super().__init__()\n",
    "\n",
    "        self.num_heads = num_heads\n",
    "        self.heads = nn.ModuleList([SelfAttentionHead(d_model) for i in range(num_heads)])\n",
    "        self.linear = nn.Linear(num_heads * d_model, d_model)\n",
    "\n",
    "    def forward(self, src, src_padding_mask, src_subsq_mask):\n",
    "\n",
    "        out_cat = None\n",
    "        keys = None\n",
    "        values = None\n",
    "\n",
    "        for i in range(self.num_heads):\n",
    "            head_outp, keys, values = self.heads[i].forward(src, src_padding_mask, src_subsq_mask)\n",
    "\n",
    "            if i == 0:\n",
    "                out_cat = head_outp\n",
    "            else:\n",
    "                out_cat = torch.cat([out_cat, head_outp], dim=2)\n",
    "\n",
    "        ret = self.linear.forward(out_cat)\n",
    "\n",
    "        return ret, keys, values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadMemAttention(nn.Module):\n",
    "    def __init__(self, d_model, num_heads):\n",
    "        super().__init__()\n",
    "\n",
    "        self.num_heads = num_heads\n",
    "        self.heads = nn.ModuleList([MemAttentionHead(d_model) for i in range(num_heads)])\n",
    "        self.linear = nn.Linear(num_heads * d_model, d_model)\n",
    "\n",
    "    def forward(self, src, src_padding_mask, keys, values):\n",
    "\n",
    "        out_cat = None\n",
    "        for i in range(self.num_heads):\n",
    "            head_outp = self.heads[i].forward(src, src_padding_mask, keys = keys, values = values)\n",
    "\n",
    "            if i == 0:\n",
    "                out_cat = head_outp\n",
    "            else:\n",
    "                out_cat = torch.cat([out_cat, head_outp], dim=2)\n",
    "\n",
    "        ret = self.linear.forward(out_cat)\n",
    "\n",
    "        return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderLayer(nn.Module):\n",
    "    def __init__(self, d_model, num_att_heads, ff_dim = 2048, dropout = 0.1):\n",
    "        super().__init__()\n",
    "\n",
    "        self.multihead_attention = MultiHeadSelfAttention(d_model, num_att_heads)\n",
    "        self.dropout1 = nn.Dropout(dropout)\n",
    "        self.att_sublayer_norm = torch.nn.LayerNorm(d_model)\n",
    "\n",
    "        self.linear1 = nn.Linear(d_model, ff_dim)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.dropout_lin = nn.Dropout(dropout)\n",
    "        self.linear2 = nn.Linear(ff_dim, d_model)\n",
    "\n",
    "        self.dropout2 = nn.Dropout(dropout)\n",
    "        self.lin_sublayer_norm = torch.nn.LayerNorm(d_model)\n",
    "\n",
    "    def forward(self, src, src_padding_mask, src_subsq_mask):\n",
    "\n",
    "        residual_1 = src\n",
    "        x, keys, values = self.multihead_attention.forward(src, src_padding_mask, src_subsq_mask)\n",
    "        x = self.att_sublayer_norm.forward(residual_1 + self.dropout1(x))\n",
    "\n",
    "        residual_2 = x\n",
    "        x = self.linear2(self.dropout_lin(self.relu(self.linear1.forward(x))))\n",
    "        x = self.lin_sublayer_norm(residual_2 + self.dropout2(x))\n",
    "\n",
    "        return x, keys, values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoder layer\n",
    "\n",
    "The only difference between Encoder and Decoder layer is the additional MultiHeadMemAttention() sublayer in the decoder layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DecoderLayer(nn.Module):\n",
    "    def __init__(self, d_model, num_att_heads, ff_dim = 2048, dropout = 0.1):\n",
    "        super().__init__()\n",
    "\n",
    "        self.multihead_self_attention = MultiHeadSelfAttention(d_model, num_att_heads)\n",
    "        self.self_att_sublayer_norm = torch.nn.LayerNorm(d_model)\n",
    "        self.dropout1 = nn.Dropout(dropout)\n",
    "\n",
    "        self.multihead_mem_attention = MultiHeadMemAttention(d_model, num_att_heads)\n",
    "        self.mem_att_sublayer_norm = torch.nn.LayerNorm(d_model)\n",
    "        self.dropout2 = nn.Dropout(dropout)\n",
    "\n",
    "        self.linear1 = nn.Linear(d_model, ff_dim)\n",
    "        self.dropout_lin = nn.Dropout(dropout)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.linear2 = nn.Linear(ff_dim, d_model)\n",
    "        self.lin_sublayer_norm = torch.nn.LayerNorm(d_model)\n",
    "        self.dropout3 = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x, src_padding_mask, tgt_padding_mask, tgt_subsq_mask, mem_keys, mem_values):\n",
    "\n",
    "        residual_1 = x\n",
    "        x, keys, values = self.multihead_self_attention.forward(x, tgt_padding_mask, tgt_subsq_mask)\n",
    "        x = self.self_att_sublayer_norm.forward(residual_1 + self.dropout1(x))\n",
    "\n",
    "        residual_2 = x\n",
    "        x = self.multihead_mem_attention.forward(x, src_padding_mask, keys = mem_keys, values = mem_values)\n",
    "        x = self.mem_att_sublayer_norm.forward(residual_2 + self.dropout2(x))\n",
    "\n",
    "        residual_3 = x\n",
    "        x = self.linear2(self.dropout_lin(self.relu(self.linear1.forward(x))))\n",
    "        x = self.lin_sublayer_norm(residual_3 + self.dropout3(x))\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, num_layers, d_model, num_att_heads):\n",
    "        super().__init__()\n",
    "        self.layers = nn.ModuleList([EncoderLayer(d_model, num_att_heads) for i in range(num_layers)])\n",
    "        self.norm = nn.LayerNorm(d_model)\n",
    "\n",
    "    def forward(self,src, src_padding_mask, src_subsq_mask):\n",
    "        x = src\n",
    "\n",
    "        keys = None\n",
    "        values = None\n",
    "        for layer in self.layers:\n",
    "            x, keys, values = layer.forward(x, src_padding_mask, src_subsq_mask)\n",
    "\n",
    "        x = self.norm.forward(x)\n",
    "\n",
    "        return keys, values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, num_layers, d_model, num_att_heads):\n",
    "        super().__init__()\n",
    "        self.layers = nn.ModuleList([DecoderLayer(d_model, num_att_heads) for i in range(num_layers)])\n",
    "        self.norm = nn.LayerNorm(d_model)\n",
    "\n",
    "    def forward(self, tgt, src_padding_mask, tgt_padding_mask, tgt_subsq_mask, mem_keys, mem_values):\n",
    "        x = tgt\n",
    "        for layer in self.layers:\n",
    "            x = layer.forward(x, src_padding_mask, tgt_padding_mask, tgt_subsq_mask, mem_keys, mem_values)\n",
    "\n",
    "        x = self.norm.forward(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Positional Encoding\n",
    "\n",
    "The attention module itself have no information about the position of other word embeddings in the sentence. In NLP and other sequential data tasks the order of the sequence is critical. To solve this issue the authors of the paper introduce the Positional Encoding method. First they generate a vector containing information about the position in the same size as the embedding and then simply add this vector to the word embedding and hope that the model will learn to recognize it.\n",
    "\n",
    "They use the following function to generate the positional information: \n",
    "<img src=\"imgs/pos_enc.png\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(nn.Module):\n",
    "    def __init__(self, d_model):\n",
    "        super().__init__()\n",
    "        self.d_model = d_model\n",
    "\n",
    "        self.sin_args = torch.zeros(1, self.d_model).to(device)\n",
    "        self.cos_args = torch.zeros(1, self.d_model).to(device)\n",
    "        for i in range(self.d_model//2):\n",
    "            self.sin_args[0,i * 2] = 10000**(2.*i/self.d_model)\n",
    "            self.cos_args[0,i * 2 + 1] = 10000**(2.*i/self.d_model)\n",
    "\n",
    "        self.sin_args_mask = (self.sin_args > 1e-10).float()\n",
    "        self.sin_args = self.sin_args + (self.sin_args < 1e-10).float()\n",
    "\n",
    "        self.cos_args_mask = (self.cos_args > 1e-10).float()\n",
    "        self.cos_args = self.cos_args + (self.cos_args < 1e-10).float()\n",
    "\n",
    "    def forward(self, x):\n",
    "        for pos in range(x.size()[-2]):\n",
    "            x[:,pos,:] = x[:,pos,:] + \\\n",
    "                         torch.sin(pos / self.sin_args) * self.sin_args_mask + \\\n",
    "                         torch.cos(pos / self.cos_args) * self.cos_args_mask\n",
    "\n",
    "        return x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a vizualization of the positional encoding vectors. Each row is an encoding for a word. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlIAAAD8CAYAAAC8VkrEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztnXl4XVd57t91Bs2SJXmQBzmW4ymOQ0aTkSEkgSQEwpBchpJboIRACy1woQXKLaX3cm8ZcoEUaEMeQhMKJAwJxYRAyciQ0UNGZ7AT24llW7IsWePRmdf9Q8L2933bOtvbx8eS/P6eh8f+zllr73W2tsPS1k/v57z3IIQQQgghh07saC+AEEIIIWSqwo0UIYQQQkhEuJEihBBCCIkIN1KEEEIIIRHhRooQQgghJCLcSBFCCCGERIQbKUIIIYSQiHAjRQghhBASkcPaSDnnLnHOPe+ce8E595lyLYoQQgghZCrgoiabO+fiADYBeD2ATgBrAbzbe//MwebEG+p9oqV1X/2K1h4z5qm+2aLWY/T7YcZEOc9kWst0O89kWstkPs9kWst0O89kWstkPs9kWstkPs9kWst0O8/RWsu27Tns6Ss4MyiAw9lInQPgC977i8frzwKA9/6fDzaneuFCP/+TH99Xv/iu682YJbd+WNR6jH4/zJgo55lMa5lu55lMa5nM55lMa5lu55lMa5nM55lMa5nM55lMa5lu5zlaaznz4u1Y90Q61EbqcH60twDA9gPqzvHXCCGEEEKOCY64bO6cu8Y5t845t64wMnKkT0cIIYQQUjEShzF3B4CFB9Tt468JvPc3ALgBAFaeXO2//7Z/2ffeR3e82hz062/+vqi/2rdE1H97yS/NnFuHWkR91YV/EPU9o3Ez5+JXPy7qxzMZM+aMMzeL+sXcsKiXnbodms68HDPvxN2i3lOwm8nmZX2iHiiOirpu8aCZkypmRV21UB4343NmTnyePG7OF+SAOfYa6DF+pjxvwRfNnGJLrvSYprx5TbzfUJjwfQAo1ocYU2fPLd6vnfh9ACjWlB7jqyf+EbmvKv0jdJ8s05hEibWUeB8AvP0nc+hjwnybVo4x5TpPmIf4pcaU4xiT6TyTaS3T7TyTaS3T7TzlWktIDueJ1FoAy5xzi51zVQDeBWBNeZZFCCGEEDL5ifxEynufd859FMB/AYgD+J73fmPZVkYIIYQQMsk5nB/twXt/J4A7y7QWQgghhJApBZPNCSGEEEIiclhPpA6VJIpoi++Xldd94zQz5rqvPCTqz9x0saif+ci/mjmL11wj6o1v+paoz1r7PjPnV2fcIOqPbr3SjPn7hb8S9Tf3nC/qD7ZLqR0Afj60StTvXLhe1L8bnWfmXNz+nKifyNaK+uz5L5k5L+SlAL1q7i5Rd+atOL54Tq+o9xSkfD531oCZM1BMi7q5RUrtw96ep36GnJPxViyvbsqoMVJQTzRYWV6L77F6edwgqd3VlhhTbeeYMVUhZPPkxGNCCd5lEMlDjQkhkiMeIl8uVkJqL/F+6DElpNBS74cdM+VkWkLIpIBPpAghhBBCIsKNFCGEEEJIRLiRIoQQQgiJSEUdqeeH2vCaez+6r172o0fMmA//jQzpXPy9LaL+zfurzZylP5Q+zfBlsq5ZM8PMOe6sBlFvuv94M+bsD0qZ5N2PnSrqL17yRzPnNRsuFfVPT7lR1J9++a1mzicW/FbUawZOF/WFzbYP9IMpGVR6bou8Tk9l55o5J7fIvNQt+TpRn9Asw0MBoLsg99odzTI8tK9gQzHbmoZEPaDCQwGgpTEl6lRRfs0a6qVnBViPqqY2q963LlayRr6Wh1xvvMbOKUJ6O7FqOSfQxVIeVRjPyowJ4yWFcpcmftuHOEYYd6liQZml1uJCrDXEmHK4VqFcrHJAF4uQSQOfSBFCCCGERIQbKUIIIYSQiHAjRQghhBASkYo6UjVdBZzwlf3+TP6ck82Y9f8uvZ22vY+J+q/+eJWZs+yPG0T9uZ1vEPWc32w1c37zOelaLbx71IzZ8xcyM2nWA0lRN7yxxsxJPTZT1EtWSxdr/fMdZs4pi6Xr89fbV4j6mlOsi6Vdq1KeFQCcUb9N1E+lF4p6VcNOM2dzbraolzTsEfXOgvx6AUBHg/Koija4aF69bMTcX5S+UGu9dKgAYKgofaamutJ5VXU18trqLKrq6tJ5VYkqWWvPCgBiaozxrJKlGyy7gCwq41GpjKggX6ukR1UO/wmlXatQnlU5PJ6yNSQuj2tVirLlXlWKCjZ/JWQqwidShBBCCCER4UaKEEIIISQi3EgRQgghhESEGylCCCGEkIhUVDb3mSyKL+5vwNv9YxuCOf+/y2DJPe+WjY07fmTl4MRx7aL+w3/JetEu2QgZAD7//OWinrn2WTPm+wOvEPXsh6Vo/WTWhkbO2SCF55QKo2x6usrM0dL64OYWUR93hmxiDABPbJefcVmHlOUf3bPIzHnPsodF/c3uC0V9xcx1Zs6GVIeoT6iVzZFfzM4xcxbXyeu0I99kxsyvlbJ5X1Fel7ZaGeoJAEPKwG2pkZ95JEC8bqyRzZHTSiSvC5LNMbGQrmV0AEgmtWwu1xJP2LVZIb30GBdwHEOiRDhouYI/S4nXYYT1cjQ2rmQ4ZUnxmsJ6IJNpLYSUGT6RIoQQQgiJCDdShBBCCCER4UaKEEIIISQiFXWk8rPr0PWuM/bVD6z+mhnzjurLRL34A5tEPfCafjPn5U+eJeqOX0q/xp18gpmTvrdV1D5vQzuvf/pVoj5+s2we/B9955g5jU90ifqxrLzEM5+Wzg4ADBSl6zNjkxQKks4GWrpt0pua+VpZv7Rjlpmz8ATpyjzVO0/UfztXBmkCwPeHzhX1uXM3i/q+4ZVmzpLqblFvz800YxbWyHN1FaRHNa9mwMzpK0iXbFbNsKiHivb7gmblUaW89FMaqu3XI6Ocoroq7UhZT6kqmVdjpDOVrLJhoXpMLG7dK+1axZT/pB0qAHAl/Kag941HVY7QzlC+UOkhpcb4Cp0nFOVygcrgUYVhSnlUk2mthBwAn0gRQgghhESEGylCCCGEkIhwI0UIIYQQEhFupAghhBBCIlJR2Xzm7AG878N37qvvHrVC9Pb3rRD1usXXifrtbW82c8684klRd147Io/5WSuFH3enlNb9KSvMmNqHGuQLTu47f/7cSWbOkpefEvWP+6QIX/u8FLEB4LFMvaibN8sQz72FlJnTpNz4uFpbstMGfzbFpKzd3T1D1G0n23315r2zRT2/XYr8zw+3mTmva5DhpncPrTJjVtbuFPWOnJT/51Zb2Xx3oVHUs6ukbD5QrDZzZlbLe2GoKMX9GVU2VDWthPT6Kvn1yAUI3rUlhPSqRGmRXId6AkBBrSUe17K5Fd9LCekuXjrUUwvpRkYHSgvpYb5NK4coHiqcskJCesWk9srI6MAkE9JLMZXWSqYNfCJFCCGEEBIRbqQIIYQQQiLCjRQhhBBCSEQq6kjNiWfw1837mxK/4vqPmjFvv+oPon4oLb2XHVfaRsc/bpfBnu+YKUM9V73xeTNn4EvSkdqpQj0BYMH9KthzpTx33fo6M8fF5A/pf/OCDKw8fpcM9QSAXw2cIuqaLT2ifiYn3SYAaNoqvR0d6tmw3UyxHtUu6VE1OOsY9e5RXpL6fNsGpNsEWI9qy4h14S5q3ChqHeypQz0BoEc1P55TJc/TW5SuGQC0JrUjJT9zU5W8bgAwooI965PyWmuHCgBqk8qRUl5SdTIokFN6R4kAd0k7UAnlWmmHCgBisYk9qqBAThPsGSKQs6RHVY5QT6C0D1QmL6lsHlU5mEIe1ZRyqAB6VKTs8IkUIYQQQkhEuJEihBBCCIkIN1KEEEIIIRGpqCO1Od2My57fnwPV8c2NZswX/1LmMC3+z2tEfdp/k01zAaCnIP2TvW9YJurrj7vWzPlQzRtE3XCBdXLcN3eIevd7TxP1nA02gyjWsVDUiY0yi8oXbFbQ3dtlhlXbLhkS9buAxsA1L0vHa2tO7okbO62TkypK16dul5QFtEMFALEe5VHFpEe1Z690qACgVR2mc7jZjJkdl+7Sy6PStTqvXjarBoBHU0tEPS8pr0FvXuV+AWhJygyu/qL02pqT1pEa8klRNyXl11k7VEBpj6omEeBIhfCoCjj0HKlEYuKcKO1QBY5R/lNQc+TSOVLTqzkyEMKjYnPkQKacR0XIIcAnUoQQQgghEeFGihBCCCEkIiU3Us657znndjvnnj7gtVbn3F3Ouc3jf7Yc2WUSQgghhEw+wjyRugnAJeq1zwC4x3u/DMA94zUhhBBCyDFFSdnce/9751yHevktAM4f//vNAO4H8OmSx+pOIn3t/H11fcNOM+aGgfmiPuE7MnTxxjt+Yea8aeOfi7r3zVIgnhevNXNyZ54g6i8uv9mM+UrmZFHvPS8j6rbbberl4GtlaOfMjVIuj8+RTYABYGizlLHnZOR57utZbuYku2Ro5/r0IlHX7pAyNwDsKkghur5Lri3nrQhf2yMt0aSTTX+LvbY5cmNMvtYzaCXw1pg8184R2UB5Zptdf1dGjjm59mVRP5+ZZ+bMSsj7p78gZXMtowPAUFEGoGrZPOPlNQCAuoRqbKwc3pqEDOwEgKwS0qvi9vrr0M6kGqNldMAK6bbxcVAgp5wTRkh3sYmFdP1+ILESoZ5AmQI5KyOkV6w5chimmJA+aaAYTw6RqI5Um/d+1/jfuwC0lWk9hBBCCCFThsOWzb33Hgj63egxnHPXOOfWOefW5bL2KQMhhBBCyFQl6kaq2zk3DwDG/9x9sIHe+xu896u996uTVbYXGiGEEELIVCVqIOcaAO8F8KXxP624FIAbSKH6zrX76ue+frYZ88LPjhN1xxMPiVqHGAJA7hb5k8Vr/6f0nb65VwZ0AsD2C2Ww5IW11k/52mLpHf3ZyWtFvbbHujK7T18q6qU3S5cpv1Q6YAAwY5P8oXysTno8m16yPzldPtQp6gcH5HljXb1mznM52Ty4bpd0sfYWbcBo7e6J/YjqPnsNqp0MtEz326bLjTF56+0elh5Va0w6RwDQlZbhnzPjw6Lek7PhoIvq94haNz6eEQ8I5CxKp64xoQI5VWDn2Bh5LdPKo9IOFWA9Kt34eGyM8qhU0+JiQNPiRImmxdqhAqxHZf0nO6eUAxXGkXJhfBT17V6kUM+y+UJlOs7hMlnWUUamVGjnVForOeKEiT+4BcBDAFY45zqdcx/A2Abq9c65zQAuGq8JIYQQQo4pwvzW3rsP8taFZV4LIYQQQsiUgsnmhBBCCCERqWjTYj+jDplXv3JffdtbrzNjPnf+laJOXbxa1H++eYGZM+sXz4j68n+W2UCL77rIzDnttS+Iemtu2IzpO3uuqK9p/YGo19fJxscAMPM06d37/yUbH/eeLxsfA0DLJunXuPnSiardarOaNBu620XdtnerGfNESjpfyd0yY2ln3t4OdT2yka5ufFxtVSxDvN8eV3tUQ8PKS4pZCWF3SjpQzTF53XZnrCPV3CjvhU15mTWlGx8DwGBBOl2NcelIaYcKAOqVIzXi5WeuC8iRSnv5fUx1PKixMdQYlf0V4AwmExNnTQVlRIXxqDSlsqYCemDb5seh/KYKNQouR15SORofhzxOWShLXhUbH5NjGz6RIoQQQgiJCDdShBBCCCER4UaKEEIIISQi3EgRQgghhESkorJ5rC2Huk/tl69nxa2AW9jZLeqer8sAxaqfyMBOAGhLPybqRzPyuB1rrAz5hTf+UtT/t+tiM6b7PDnvuIRqvru8w8y5uuNeUf8kLYX1vSdZQXfub2RoZ+oEKZs3bbVz4jPkdenfIWvd+BgANgwslC/07hXlptwcM6e6R4rWfUo2r+kLCneUr1X1W0tUNz8uDEr5vM7Z0Mu9KS2kS6m6N2OT85ticv17cvJruLLGNs7emWsRtQ7tHCnKMFcAaIjrQE65/tqAez2nvo+pCRqjbl0tpOsgTcA2NtaNj3VgJxAkpOtmwvY8eowW1l0ICTlUaGepMQESsgntrJTgXSbxvaSQXqnGxyQ6vP7HDHwiRQghhBASEW6kCCGEEEIiwo0UIYQQQkhEKupILa3pxy+X37G//u3HzJi5/026JXeuvlbUH/7Q28ycwctOEfVHn5Eu0MzfbzRzTq6SoYv3/eEVZszrXvW0qDdmpSuz57QZZs6bG14U9W1Ny0W9eOUuM6fQJUM8+98s1z/rcRkqCQBomy3Kuu3qSxmzzYSf65HuVfvgZlFvHJWhngAQ75NBpTsL0g+q6bPNnke9Cu20mZeGxKBcb7Wzt2ZqWJ67TnW87R2VzZ4BYIYK7ezLyjGNMdu0uC8vr8Oiatn4eKhomzBrRyqlPCod2AnYxsY1AYGcWRXaafwnMwNImqbFkkSIsM14icbHgHWkNK5EYCdgmxabwE6gtGtSLv+pUqGdUw2Gdh45ptp6SSB8IkUIIYQQEhFupAghhBBCIsKNFCGEEEJIRCrqSO0u1OC6vUv31Su/MmjGFL8tfSC90yv2D5g5e/9MejzVd8xSk7abOTprqv0e6/p87Mq7Rf2Nbtn8uPcM64DMicssI79YNll+54LfmTm3ZWR+08ByuZb5v7SdgXXWVEOndBDiDTZTaaRLrS0nXaaNg7KhLwCgX36NtuXkta3ulTlNANBflK5PdX/prKnkoJQF4gEdb4sjE2dNDY5ad6leeTr9ypHSOVMAsDcvx5xUK8fonCnANjbWWVN1MXmtgfJkTemcKaB01pT2rMaOo5oWq+umc6aA0llTQQ7VkciaCpNFNd2ypsrW+JiODiGHDZ9IEUIIIYREhBspQgghhJCIcCNFCCGEEBIRbqQIIYQQQiJSUdm8t6cJ/3H9Jfvqti3rzZifLZcy9rnrPijq1jeoxsEAvn/6t0X9+f9xhahTrz3JzPnCtlZR1z20yYzRoZ13r18l6tNO3mLmbM1J8b1/lQztvKTenufndR2iblsmAyCL3bKpMQAMXiJDI1s2qcDHOTPNnNpd6sut0hBf7FOSPoC2oa2ifkE1YY7327DQPQUpUdf0W7k546UQXTVkhhjiQzLAUjc+Hk3ZZsI6tLM/rRsfWwl8IKfHyNDOgcJ8M2deUqaOjhSr5Dri9jw6tLM2YEyp0E4d2AmUDu3UgZ1AtNDOWInQzlKBnUCZQjvLFaRZDjmbgZ2HcRyGdpKpCZ9IEUIIIYREhBspQgghhJCIcCNFCCGEEBKRijpSiZ4U2r6zbl/d9eHVZswzuT+KuvU70onqvMo2dj25Snok+ZdkAOdLf2udlur7OkR93MBDZkxnXvpObQ/Kfec1b/i9mXPLwBmi7j1J/kD+uIR1vNwC6R1dMv9ZUT+Ylr4NAAwtlnXbfTKoNDev2cyp61KBibXSBerfY9c2JyPdq80pGR7qBuU1AoAdBemFVfXboMlhL1+rGtThjtadSQ7La6kdqcKIvZ1r1JihtG58bP2tgax04+pV4+OBvLxuALC8Rjaj7sk3ibohboM/QwVyKkeqOibvfx3YCQBVMR3IKd8PCuTUYZqmabG3/kqihN8U5Ejp81j/KeCYJdyZUKGeEfynoHuwpMdTNl+oTMcpB5NpLccavPZTAj6RIoQQQgiJCDdShBBCCCER4UaKEEIIISQiFXWkXHUVYks69tXv+/CdZsw77/1LUS//9VpRf/dbT5o5/9RzqqjjK5aK+kOvvdfMuefD58k5y5eYMTfulW5M68Pdon5NjQ0/+tST0vuqPVHmCw0XrSszukRmPl3a9ISoH4ydbeagY0TWPX3yPKtnmyn1u6QbE2uRHlWyRzo7Qbw4ILOm6od2mzHbsvLc8QH7mXsL8of/1YPSR8nDejxJq2MJYqm4eU17VCnlm9UEOAhDOeVIOekuDeaDmiPLe2VrQY5pSaivF6I1NtYNiXMBOVJVJkdKN4QOaCKtap01FdS0OK6bFkM3LQ6TRVXab4pF8JJs1lSYHKkKZUCV4zwh3JmyNTYuB2Vp9nwMZnSRKQGfSBFCCCGERIQbKUIIIYSQiHAjRQghhBASEW6kCCGEEEIiUlHZPD03juc+vT/08ZfNtunvr78jpd3YSSeI+vzax82cD//g1aJOvlm+//HWZ8yc+9dLwbvr/aeZMd9/8ixRL31Rnrva2ctXXC/DKK94h2zC/Eim3szZu1wKxSclVTjiTNlgGQBOX9gpjzEwKOrhdrtHXnCvtLWLs+Raa3ZbI9Ql5Gfs2tso6sWjL5s529JSSI8NWdG6p1gn6uSglKjT3gavJocnlk3jKbv+BKRsnhtVDZWdvU5DGS2By7VoGR0A6p0UxVOqafE8t9fMSRfVWmI2kNOEdiohXQd2AkGBnPK6aBkdsE2LTSCnmVE6tDPIL7aNjZXUHhD8qSVjfQwXQlgPIztHCe2Mch4yuZlyTY2n2nqnIXwiRQghhBASEW6kCCGEEEIiUnIj5Zxb6Jy7zzn3jHNuo3PuY+Ovtzrn7nLObR7/s+XIL5cQQgghZPIQxpHKA/ik936Dc64RwHrn3F0A3gfgHu/9l5xznwHwGQCfnuhAKxq7ccfr/mVffdnz77SDHn1KlM9/S3pKX+s73kw5/tYeURe/nRJ1T0F6VwDgC9KzSF9owzVn3CN9oFi1dGcez1qPZ84G6blc8cH1ov5m94VmzuBy6azUxaRf4+dL5wgALmiV7tVtRdlMeKTdejDxHtnYOHVCm6hre6xrEquTLlOuT/lBRXuerSNyvX4kZcZ05aWflRyU7s9QwHG1I6WbyiZGrCwQVw6UT6smwAGe20hGh3bK8w7n5X0wNkZ+3XVj4/paG7a5Myd9ucBATuVa6abFWVhHqjqmAznlNdAOFQDk1Je+Sh0jyF3SjpQO5NTvB40J4yWVCu0MpTaF8qgO3bUyjY0rFYJZDp8LZQrtpKNDjnFKPpHy3u/y3m8Y//sQgGcBLADwFgA3jw+7GcBbj9QiCSGEEEImI4fkSDnnOgCcBuARAG3e+13jb3UBaDvINEIIIYSQaUnojZRzrgHAbQA+7r0Xv2vvvfdAQDOusXnXOOfWOefW9fWV7r1FCCGEEDJVCLWRcs4lMbaJ+qH3/vbxl7udc/PG358HwHavBeC9v8F7v9p7v7q1lb8kSAghhJDpQ0nZ3DnnANwI4Fnv/dcOeGsNgPcC+NL4n78odawcYugu7Jdn01+dbxd0vvwJ4fWXfE/UH7ntajPn+GcfEvU3lzwg6r/brhI6AeDUdlF+/hV3mCE3f/FSUftVS0R94x572Pqndop6ZVIGKt73wnIzZ95SKcvvKcgAy5HFUnoHgPNqXxT17Un5eeraZfgmAPi9UjYfmSvn1HdbCdnNaBJ1VZ+VmzWdQ82ibkntNGN25uQvecaH5S8EDBQDgiZH5BPNPKQQnbROuyGWkpv5pLPnyaR1aKe0aYeyVjavU2GaI0pIr4/ZX3jQoZ0tCRtcqgM5tWyuQz2DxuS8ls2tyF9QxrAVyS3xEqJyPITgrQM5gyglpLsQwnTQMYr6IXoogbvEucokgYeS1qcbZZHwK3PdplxoJzmihPmtvfMA/HcATznn/hTt/fcY20D9xDn3AQAvAXjHkVkiIYQQQsjkpORGynv/Rxz8ewX7u/yEEEIIIccIlJYIIYQQQiJS0abFW/bOwTtv/5t99ZJfP2zH/OhUUb+uNi3qJbfK5rwAgNNWyTGJDaJ+/NcrzZSYepZ2RYMVnm56VnpIXdecIepNG+V5AWDZjg3mtQOpfrrWvHbpu6XjtSEjHaOBDvtlWpyQbk+sWQZcntxmvaTeYengpObJB42tT1vJqDhTOlI1vXKObmoMAH0DMmhyRtr6QdvTshGzGxmVay3a65QckqZOqii9pMRIaT8iMSrXHwt42JrPyM+UVGNGstJtAoAaJ9c2ktehnrYh8VBBhpu2V/WZMSNF1UA5Lq9lLjCQU55LNzZOBDhS2qNKRGlaXOJ9wDY21p6VbkgMhGhsHODFmOOE8qiiuVYkBLxuRxde/yMKn0gRQgghhESEGylCCCGEkIhwI0UIIYQQEpGKOlI13Vms+NrL++qRy15pxvzqvG+I+r3b3iRq/9hGM+fFr54j6psGZT7Vojv6zZyhL0n3aqCYNmN8TubxpM6VjlHTQ9IFAgCXkLk+z+akrzLraZvVdGnjk6K+ufc8UQ93WG/ENDZumynq81pklhYArCnKMaNz5XHjvbZx8+jS2aKu7pMeiW5qDAC5AZWzFNCAePuoXItubNyTl24WACRGVFaTahibTFnHRTeVjStHSjc1Bko3Nk7n7D+bUo2Ngxyp0YLOq7Jj+ooNaozOkbK+VqnGxrqpMWBzpHRjY93UGLAelXaXghwpTTm8pFJNjYEyKiIRcqSmemPjihyDkCkMn0gRQgghhESEGylCCCGEkIhwI0UIIYQQEhFupAghhBBCIlJR2RzFInxqf/Biw992miGtamu36XsniHrOib1mzkff+GtRf/FBKagvf3ydmfPFZVLw/kbv2WZMfNliUX/gpAdFfc+3pBQOALHjjxP1zwdOF3X9s7JBMQCsVH1n731ZNjZu6JDNhgFgWMnx6XbZ2PiVtVvMnDWxOaKOz5WCtx8IkM1nS3G/tleKyq5RytAAkOwv3di4a0TK5PWju+X7eRkwCgCx4ayoh4qqAXGAbK4bGydGzRB7nrQ8blzZzpmMbRSsXxnNa5Hc/pLBcEEJ6bGsGVOqsbFuagwENS2WX49khEBOLaMDQNxN3Ng4qKlxAVpI9xO+D5RubBwmJDOM1K4/omlqHIZKNRuuYFNjP1kaKJdLap8sn4dMG/hEihBCCCEkItxIEUIIIYREhBspQgghhJCIVNSRysypwdYP7W8g/PSyb5kx5z3xXlHPvuUJUW/5zClmzsdbton69l8q32Num5lzfq30Lt5/1zlmzOxzpFvy/ubHRf27Z9rNnP6LV4j6P7edLOq27VvNHB2umdkk/aHXX/iYmfNCTgoDQ+3yS7ksYcMd403SZzp+jvTNiqqpMQCk5shr2fiSlIx8kw0lrepXMkPMOlO9wzLIsy4r17srKxs3A0AsJb2wIeUHJVLWpcl5ae7EQzlScv0JFWiZz9rPU6OCPVM5ubZqZ710TdPXAAAgAElEQVSkMIGc6aIcU+Um9p8AIKnOlfPy3tBhm4B1oBLaf/JWUKmKy/OEalqsau0u6abGgFVjdEPicP5TmRoST6WgTHJUCfgnM3mZSmudhPCJFCGEEEJIRLiRIoQQQgiJCDdShBBCCCERqagj1T6zF1++6qZ99df2LjNjqm5oFXWsSWYbvfMtvzdz7h+V+8H6e54Vdc/bTzJzXswNi3r+PfaHxDtfL12SOXHpAxX27jVzek6Xx3HPtIjaZ54zc/YUpJvUvEm+f+HbnzFz/pCSWVMj7fK8LXHbTNi1SO/o9Bbpa63P2X11erZ0SxJ7ZfZUvsWep6pfNTausllHqcEaUfu8cqTSNkcKo9KR6itI5ysxYt2flNc5UqVdmUR64sbGxbT9Z5NUY0az2m2yvlAqL924IEdK50jpMf0Fe/1rYnKMzprSOVOAda30mGKARJFQLlZBXdpYUI5UhMbGQcc5kCBHSp8nyH/SrlU4j6rUWkoewvgopqlxmLWULVOpTMcpB5NpLYQcAnwiRQghhBASEW6kCCGEEEIiwo0UIYQQQkhEuJEihBBCCIlIRWXzxlgeF9T27av/8RvvM2PmrHlE1Ns+fZao18ySDYoBYPl9HxD1svRGUY9cPmjmfHHXpaJu/sM2M+aMT/eJ+tGMlHhjjbJRMAC0n7ZT1KM3zRN1vNlK1OsyUrBv3iyl6tOr5TEB4D92yQDR9ELb8FaTny2DPk+tf0nU6yGbNANAdraUjt2gFOMzS6RMDwA1SjZ39QHi+5C69ZQc3J2219an5XXpVbJ5fDQg0FIdN6lkcx3YCQCxjHlJ4LL2+4+Y+p4kk5OfrzpApNWyuQ7SBAJCO7VIng9qWizHZHXT4oDz6NDOREyHetrPrBsba2Vah3oGoRsbh2larMfEYocviQMhXedSg6ZjQ9wSnzlUU2OK5GQawydShBBCCCER4UaKEEIIISQi3EgRQgghhESkoo7UcyOzcM7av9hXL7hhnRkTW9oh6ve8+x5RP561YYILfyQ/RuHcVaL++ik/MHM+cvvVoj6+6yEz5qOzHxD1FzrfJGq/YoGZc/Vx0uH6wcaLRV1cYhsd37H3VFFXbd0t6vZErZnzdOd8Uc9fIH0uHfIJAKPz5HFWVXWJ2iVlw2UAqJ8tAzj9kAwyHZ1pm+bWdyuvqs46UsnBiffwe1K2GXJLZkDUPXnpUcVS1hMbKsr1JdLa67EeTyJtXhK4jBU+kk6eJ5dVjYIDkhpH87ppcZAjNXEgZ6ZoHam6hLwO2n8KcqSsR6W9pKBAzonHBIVt6jOXCtscGzPx+6H8pxCOTpTjFLXTFSqQswxOUbnOM90oS1Ppyl23KdXYmBwUPpEihBBCCIkIN1KEEEIIIRHhRooQQgghJCIVdaSSXQ7zvrLf6Ygtt7lFm94vc4nunPW8qFc+IDOjAGDRbx8T9QtfWi3qN9TZfKH2e+RriUULzZjlSenpPLJWOkTNp9sfcL+xbruof7ilU9R9b5X+FgDc9/JSUS/s3ixq7d8AQHyrbPr7yhOeEvUWlVEEACNz5XHa1Vc/1mC9pONn9oo6m5LOVKbVXoPmTdLR8U32uFWDal5Mrm1gxHphzVl53N1ZmYvlRm0A1JBq+hsfld5O2lvnLp6e2JGIZ+1njik5I59TzlGAvJHOK3cpwNfSOVJJJ9ermxoDwDwnm2mnixNnUQFADsolMzlS9h7UYwpK+NA5U4DNmtIeVVDylBmjGxIHzNHuW5jGxtqNCfLnykGoxsaETDZ43x4UPpEihBBCCIkIN1KEEEIIIREpuZFyztU45x51zj3hnNvonPun8dcXO+cecc694Jz7sXPO/oyBEEIIIWQaE+aJVAbABd77UwCcCuAS59zZAL4M4Ove+6UA9gKw8hIhhBBCyDSmpGzuvfcA/pTCmBz/nwdwAYA/G3/9ZgBfAPBvEx5sZBTu4Sf3lc/ddJoZ8tVzfiTqb+ztEPWsW2y4Y2ymbPr7jgsfFPXvAwIWax/aJOo9ARL41pwMn5wrD4udF9hgw5a4XF9hUDZM7rOngdsspWmfk1J1ULhm41ZZn9soBfW1o8ebOal50hacEZNCt2uW6wCAlU2ysfHjys1Oz7QSb2JgVNT5ZiuOJwdV49kqKURnRuwDTl+Q17snqxobp61s3l+UX49ESh4j461QHC8RyBkLCOSMO/k9iVeNjZPOfs+iGxtXBTT5TRfkGB3IqUXysePIz5gqVqu1lG5arMcEBXLq0M6iGpMIOE9B3S46kNMI4CgdlBkm1LNcoZ2lwhrDnCcUai2FgPv0UI8RmckiGU+WdRCiCOVIOefizrnHAewGcBeAFwH0e7/vV546AdiYb0IIIYSQaUyojZT3vuC9PxVAO4AzAZwQ9gTOuWucc+ucc+tysE8MCCGEEEKmKof0W3ve+34A9wE4B0Czc+5PPw9oB7DjIHNu8N6v9t6vTqI6aAghhBBCyJSkpCPlnJsNIOe973fO1QJ4PcZE8/sAXAngVgDvBfCLUscqzKxH/2Vn76sfeN1XzZg5yjF6xfVXiXrRnRvMnK73S9fq72evEfV5695v5swdek7UPRfap2X/2vtqUTc/ulPUiz8u/ScAeDYrAytjNTI4s3mVDLgEANw2U85plO7PMzkbaDljm/SoTq2Wa/tCz+lmTnquDZ88kEJrg3ltVa0MFH0csulyttV6MG5QOl3ZjhlmTLVypFyt8qiGA25N5c/0pOV6fdY2Le4vyPspnpbXIB3g5CQy2tuRfko8zINV5UjFAr5nyepAzgAHJG0COZXjVbTXSQdupvMTHwOwTYurY/I6aYcKCAjkVBJLkLukTR/d+DgI41GpRsGxwObI6v4qV9Piwx6A8jTFnUwNiUN8Zl+ORs2ETFLCJJvPA3Czcy6OsSdYP/He3+GcewbArc65LwJ4DMCNR3CdhBBCCCGTjjC/tfckAPPrdd77LRjzpQghhBBCjkmYbE4IIYQQEhFupAghhBBCIhLGkSob9XNSOOtj6/bV2wv2t/j+d/e5ol78vW2i9jV2ztx3yNDItJcSbPKXzWZOfOUyUf/N6nvNmOsevkjUy7etE/XVbU9C88P+s0TtFkk5+8pFj5k5d28+T86Z3ybq+4ZONHOqX94r6va4FIqf7J5v5jTNHRL1cFEmT2Zm27DTE6p3yRdii0SZbLXplX5ECveZGXEzpmpQCsKuTsrmieHSe/y+tFxvfXq3GdNbkEK6G5Ui9kjRnieRlmJsHvJ+CiObx5RsHg+wnXM5eV1stCaQUYGcVUrXHi3YWUknRfGMCu1sSdiA1yiBnHElEOe8/MyJQAlcHkcL3laDt+fRxEJIykFCuqZcQrqmiAhieKVk8skkrVeKckjtx+J1IweFT6QIIYQQQiLCjRQhhBBCSES4kSKEEEIIiUhFHanjkiP4l/lr99VLb/lrM6bxRbm3a9uzXtR977ZBk3csvVbUn++6UB7jN9KhAoCdb+sQ9dUznjNjfvDgpaKO1ctgzHNqrCzzoXUyKWLOidL9ubzxCTPn91ulrzV8mvSq7t8t3weAmq4eUdfFZJPfVKcN1zzzDNnYeKdqApyabW+HhUoIitXKgNG5LdK7AgA/KpsWZ5qtlNCwU7pKvk4eNzEUIDIoiaV/VM6py9vA0b15+TVzGRnamQoImoynpU+TU85dzOZ+GmJZFU4Z8D1LXjtSAZLOaE6Hacq1ZQMCOZPKNMqpsM0qZ6+THqMdqcBAzhIeVVDYZsGrMTHd+NiiAzn1mKhBmkV1pFD+U8mmxaUPUR5Hp0LnIUcVP9W+hlNtvWWCT6QIIYQQQiLCjRQhhBBCSES4kSKEEEIIiUhFHamXcvX4UOc5++oVX3vZjPFDw6Lufad0oqre1W3m6N3gPf8lPaWOHQ+ZOfkLZbZUkMMy5yHZYDh/8lI15/dmTnKDdJN6V8n3lyel1wMAhW6Zf7R3eYeoh16abeYsH5Lel/Z46jttdtMZF8g5z2XlcUdn2x9wt8ZlblesUX6+jibbhLk7K/2nIEcqPihFo2KDdMmSNuoILiF9odGUXJvPWfdnT07lSKXleYeK9uthHSndtLi0S6P6BiMWIA8Uc/KeS7qAxsYF7S6VzpGqUu5Suli6afFIUV7LpGpaHJQjpY9TVDlSsQBHqhjCo9KUcpcCmyOrZtRBY0qdpxDQ0Fo7UNqzCpMvFMrpOkZdk8OG140cBfhEihBCCCEkItxIEUIIIYREhBspQgghhJCIcCNFCCGEEBKRisrmqe46PPmNU/bVLann7aCkXNKCq18Q9f9b9HMz5eotV4i6Y40U1mMnnWDm/MOqX4n63weXmDHFTVtE3fWxM0X9aMaajXPWywDLzquldRzUwNSrIMmh5bKu3SrDNgEYG3VXQYZgNmy3Eu8pNVLufzQlP3N6jl1btZOism+S8vYJ9dvMnO6iFMezzfa4sWG53tz8GaJODgeIvlVyLfmUun2LVqLuyzaJ2melbD4YJJtnVBil+poFNS0uKCE9pu6NeIBIDiWbxwNM2UxONRNWa9EyOhDUtFg3JA4K5KxTY0IEcqowzSwmFuMBK61rIb0Q4GHr82gJPJy8XRnBu2Kuc8CJ9D0YqrFupcJBCZnG8IkUIYQQQkhEuJEihBBCCIkIN1KEEEIIIRGpqCMV7xtB0y2P7Ku3fv4cMyapeuCuXXKdqINcjZd+Kl2fOeseEfXLnz7LzLmifq+ol9/3djNmmdso6vw5g6L+7u7Xmjm1T3eK+pKlfaJ+Mms9nlhjo6g7lsjQ0dE/zDNz4jOk+/NMdqaoGzqtyLMsKT/zv+5dJOrcbJUiGUChWbo0S2u6zJjfYbGo883WyXGptKizjbNEXRXkSNXI0EiXsn6Qpi8jmxYjK887pHwuAIhl5HrTyslJBARyavdNB3IG4ZQjFdjYuKBDO+X76YBATu03aUdKB3YCpZsWZ32QizVxIGciZs9TjNK0OMArPJB4gAtUQGmPKswYTajGxiUPEmZMGc4zmQjxmX2pz0wXi0xS+ESKEEIIISQi3EgRQgghhESEGylCCCGEkIhU1JFCQy38qftzpL581U1myAPDy0X9SEY6IP+++wIzZ/7PXhS1b5GZRCsu3Wzm9BZljtHMu22ekFtxvKg/cuLvRH3tA5eYOcu71or6ytbHRf2TfplFBQCufa6o3zhvg6jv2iobLAMA5s0R5UMjsqFycle/mdIWl3lUm/dIL6l5lszfAoDhonSKMjPldepI7jFzXGKZqKuara/lU/L6Z5vknr5q0NoyrkaeO5Eq/X1Af1o6UI1Z6bn1F6TzBQAuLR2pkaLKe8oGOVKqsXHWDLHnyemsKSuB5HLSTdKmUqZg/wlXqbWEyZHKqMbGdQn5AYLcRO1I6YyocDlSfsL3ASCuc6TU+2EaEsdC+DXlyJqK4lAFZcuVXkgFHarp5muVolwu1rF23Y5h+ESKEEIIISQi3EgRQgghhESEGylCCCGEkIhwI0UIIYQQEpGKyub5uR49f7dfXr6gts+MubxeytmL11wj6oYX7JIX7HlU1INXrhb1Dxdda+b8vz2vEvXs+3eYMd2vXyDqP2uUTZZv3PAmMyemhOjVVVLa/cstq8ycWUtlaOTFDTII9P7tK82c1CoZ0vnwHhmCGd/da+bUxaRsPtolGxCvPFlK+wDQXVDhlK1Sd54f0MHXVcnzzJphJXY/KmXzTJM0POt3WiHa18pAzsSwskIDTOChtJzToBpE782rwE4ALiO/ZhkVRhnPBEjUKrQzFkI2j2WVeB3wfU2xoIMy5ZxcYNNi1UxYy+ZG17aBnFVKSNfvj51n4sbGuiExYGXyhGlabL+GWibXRw2SzfWYaBJ4wC88lAyNLFNz5Eo1E2bI5ZQn4J8MqTB8IkUIIYQQEhFupAghhBBCIsKNFCGEEEJIRCrqSJ1Q14vfn/H9ffVZa//CjPnkyrtFveI7I6KO98lARQBIXXiqqHdfLkMk2xO2Me1PHpLBmMu3PWrG9J03W9RNMek/zV5v3R8s7RBlXexhURc2ymbDALB3ufwh94qk9FGKPTb0cmDxQlnvkmtdOmSdr4KXzkfNLnmeVa/aZeZsy8tw03Sr3Hu3Ku8KAFydvN4LG2046N6s7OqbU45UYthKRr5eBXJKzQoubj2e0bRq6luQXs9A3t4bLqcCOb08RizAkcop7ygotNOcR2lgsQBhpagaG8fVmGwIR0qHdoZpWhxTflBQIGdc+00mkDOoabFq1BzgUWlKBW6G8Z8CPSrltYVSinSAqC99buNalSGoMZRnRchkY5ret3wiRQghhBASEW6kCCGEEEIiEnoj5ZyLO+cec87dMV4vds494px7wTn3Y+ec/RkPIYQQQsg05lAcqY8BeBbAnySfLwP4uvf+Vufc9QA+AODfJjrAQDGBX6f2N8qd/2V7+n9+8xWi7njsIVHnA+SAl/6X9IO+uvp2UV/fL5sPA8CCe1TD2NmzzZh3nSIbED+QUb7Kcy+ZOb1vkTlRW3PSo2rdaP2IXa+TLkm1k05OMS2dLwAY6pB1vFM1XQ5wN/aqRs11u+SYVbWdZs5zmfmizsyU7+tsKgBwDTKbaVHdywFrkXW2SeUwpawjlW+WPlNiRDkuCXs/5ZUj5ZUj1Z+zTYuhcqSGivK8QTlSOeWfhcqRMk2L7fc1Pq+9Izkmm7eOVFw1wc0WdRaVzehKq6bF2qNKFWUe19hxSuVI2XuwVGPjYoBEkdDNkdVhA8+j/aeIHtWhUi53KUrulT2ILLUjGfU4R+0Y5WIyrYVMC0I9kXLOtQO4DMB3x2sH4AIAPxsfcjOAtx6JBRJCCCGETFbC/mjvGwD+DvsDg2cC6Pfe/+lb204AC4ImOueucc6tc86tG+yz3wkTQgghhExVSm6knHNvArDbe78+ygm89zd471d771c3tVY0bYEQQggh5IgSZmdzHoDLnXNvBFCDMUfqOgDNzrnE+FOpdgA2uIgQQgghZBpTciPlvf8sgM8CgHPufACf8t6/xzn3UwBXArgVwHsB/KLUsXb0tuIfbr5qX73wkYfMmKXdx4l69KIzRB3PWWHy+vP+Q9RvqJNhj4vXvMvMWfnHraIeerUV0j/Y+nNRf2KbFOELg91mzp4zpCR6x7CUz2dstOGUtR9MiXpXXgrqQRJ1fLEcU/u7Rjmn2srBnXl5nIYu+aPW5cndZs79A7JhcqbVhixqik1S4F5c3WPGPA4p9+calRycsoJ9vl2GmSblZTPNkgHAjyoZW0nIAzkl6QPwOXn/DBblmFjOXoO0Om48p4Mb7X0bRkhHXgVYqofIwU2LZa2bFmsZHQDyRXncUiJ50JiCn/gYAFCADuRU1ynABC7VtFg3Pg4ijEhuwjYDrlMpmTyMJB7KdS41qBwyejmPUw6mm9ROjhkOJ0fq0wD+h3PuBYw5UzeWZ0mEEEIIIVODQ5KWvPf3A7h//O9bAJw50XhCCCGEkOkMk80JIYQQQiJS0V+jq949io5vPr2v7r3qbDOm5RYZgrnrK9KLSffaJrPaiVqvAhUXrQkI7OuWPlDnRR1mzOJkg6g3PiI9qmXN1uNZcYoMn/xp5+mirt+63cy5bK5sFvxIZq6oYzNbzZzV7fI4L3cuF3W8pdnMeTIjEypquqRkND9h4yme7W9Ta5HXNuetB5OfIb9GHVW26TLcHFH6Jvk19KP22uYaVLBkSrkxAV5YPDXx9woDGXs/xbOyMfZQQY6Jpe11yqhbLK78p2KAbxMLkQbiTGinrPMBgZz6E+vGxkHuUkYFcurQTt3UGACqY7kJxyRjpZsW6zFFX9qR0gR5SfrMQeqMdqBC+U1HySkKun8mDRW6Jn66+VzA5HLUSGT4RIoQQgghJCLcSBFCCCGERIQbKUIIIYSQiFQ2ajweh5ux33k65+NrzZBHs6tFfeeZ14p6bcZ2ovl2/0JR/+jlV4q66f6NZk6sQ+ZVveOcR82YZ7PSIZr7kHRy8is7zJy/ar9N1J+84ypRLx2R+VUAcFHDM6K+rusiUft5s6B5Xcu9ov5pp7wuxbmquzCADSNyvfHdA6JuidlMpR17pGs1q2VI1MPFjJmTbZa+zfzEgBnjEnJMXZNyogIcqWy93PfX7pEej6sJcKTSE8sMg1k7pzkvjzukcqRcYI6U9INiOZ19FJAjlTMvGZxqWqxzpAp5+71QlfKocqZpcVDTZdWQW40JcqQanPwamYyoAK8nq68TDj1HSo8JkxEVbkzJISUdqTBNi8O5WKUGlD5PKP8m1HFCjCGTmgD1kJQRPpEihBBCCIkIN1KEEEIIIRHhRooQQgghJCLcSBFCCCGERKSisnm6rQrPfqp9X/2reXeYMa+/Zp6oa5Qk944GKy4vveXdom7cIveHjYUdZk7P+VLO/uSsW8yYL3RdKOqmR2UI5s63dpg5F9XK8MmWp5UYW19v5qxISgH3jy/J4M85i21o5Lm1W0R9e5cM/hw51Ur5T/SpQM7+PlEnnRWK8z1StO44Tl6DnqIVWtPN8jizA5InXZWUzWc2SLHfZ21H35zMR0XTy/K4vsY2LY6n1A2kLN6RjJ0zoyBl8uGCkvCz1hLXsnk8o2VtK6iHkc1juYkt0ULBfi+kX8nkSzct1o2Nk9BNi+29UVUitDMo+LNomhbL6xQkm+umxAVfWjbXOn2YMVEaDutfIggV2MkQRkIOzhQU4/lEihBCCCEkItxIEUIIIYREhBspQgghhJCIVNSRWtLcjR9cft2++kOd55sxP1/xU1G/ZsMHRP3JFXebOUtvGRZ1vFeGRo6++iQzp/dCGSY4M2Y9pF+vP1nUy3fI0M7+M+abObVOOjetz0j3x3W0Q1PtpC+EzdKjGlhsf2h8fFLOKfZK32lw4SIzZ3C3bH68ZKRT1AVvgxpreqT3sqKhW9Q7841mTqZZrrcxZm8zVyu9o7n1slHwQICHlGtUDXxTypGqtb5TQuV6urj8POm0uvYAoBypgby8N1zOOl9pLz9jLKvdH+vFxLMhnBx1qpgSCHxAIGdcjckXVaPggEDOTEGuv8qVdqRi0B6YcrECzqMdKO1R6abGgPWo7PvlaTasj1P0h+4yBTZQLnGcoLDWcnhUYcJBCSGHD59IEUIIIYREhBspQgghhJCIcCNFCCGEEBIRbqQIIYQQQiJSUdk8AY/W+H6J+Mmvn2LGvPB/7xd103eaRP0/L7vCzFm+TkrgWgXe9om5Zs4nTv+NqG8baTFj5jyoQhab5FouWvWsmbMxJ4MkE5ul0D3wuqVmzq68lOWbN8n3e1bbYEMtqBfT0qoesU47sKta1kUVPOkzZkrtbim9LqmWsvmL2TlmTnaGOoazErirrxP1wloZmjpQtAJuTmWZxlLyWhca1ecDEB9V503IWz6fsf8EvJLNB/MqkDNANh8pynPHc1rEtp8nVCCnOlXcye99fN4axTFlGedUaGdQIGfeTxyUmSsGBXLK65RS1yAokLNgziPXEhTIGdcSuBoTC/g8BfVSkJCuJfAoQnoUyiGBhwr+DHUgWQb9wsmhHiMyk0WOnyzrIFMGPpEihBBCCIkIN1KEEEIIIRHhRooQQgghJCIVdaSeH2zDa+/663318lseNmPe9oa/EvXyO9eJeumA9ariK5R3pNyHq8+/38z5wIzNoj5r7fvMmIUP7xZ17rQl8hizv2vm3Nx7rqgLe3pF3XvSCjPnwbQM9pyxWYZ45q5UqZIAhos6aVJ5MQut71T/lHR9tC/UXbB+RO0e+dqSKnlN/rP/DDMn2yznaK8HAHy9DLlcUN0v6qchfTQAyDeoBrGj8jPm21RXYwCJUeWSqCBTn7buj75/hnLyuvmAsNCUl36Qy2pHyjot2pEK8lNKNS1GiEDOXEF+xqDvnrI6kFM1Lc4U7X8qjEelQ0kDgjT1GO1RFQJWZ5oWa0eqRGDn2Jjy+E+l3KQw/lOU5shHlcnSZHm6uVhk2sAnUoQQQgghEeFGihBCCCEkItxIEUIIIYREpKKOVE13ASu/st+Fyb3qVDNm2Q0yOCd+4nI54IEnzJyt/3COqKtkz2J8cuatZo7O9Unc02zGFF54RNQ7rzhL1GfY2CK851npcC1LPCXq2CrZnBcA7uiTc5Jbu0R97jzpDwHA8zmVx9Mg/aBF86WbBQCjv52n5shgpi052dQYAGp3y6ym+XHpb704PMvMKbTYnCUzpkFevPYqvV7rSBUaVC5RRq4tX2d9p6RypFyVdKRcpvT3EoPKkULeOms6RyqWU01/A44bz+l8JOui6KbF5v1CQI6U+v6oUNBNi+1xsionyvpPpXOkSjUkDhpjGwWXblpc9PIYiZh1pPQrQVlTGn1ZghpNa79JjylbvlMpwng+k8VtqiD+GPzM5OjDJ1KEEEIIIRHhRooQQgghJCLcSBFCCCGERIQbKUIIIYSQiFRUNveZLIrbtu+re7/aYcbMvvx5UT933dmiXvn1hWbO5W97UNQb+uSYvoINp/zx0EminnfvHjOmmFQhhWf3yfeN0grUb5BBk/F2GbZ52fEbzZxfbHqFqBfvlmMuapbXBAD+kJISfmyWFMXPnPWSmbNulxTDXats1PxMeoGZk+wdEXVrXErH2/oDBPUZUsZOFbNmTK5JNjKemxiQA2JWbo43SPPaq0bNufqAMMdRFeKpAjnjo6W/lxjJybXWqCbTADBUUGGnWbnWtA+QwvNaNg8I5Czl7Qc0LY6rVEgtmwdEkJqmxFVqLfkA2TyGiYX0GmcVey2Tx0uEbY6NmVggDmxIrI4TJIFbFb40sbI0HA4z5tCl6aBfVih9okrJ8cegBF6O4M9j8bpNQfhEihBCCCEkItxIEUIIIYREhBspQgghhJCIOB/QTPWIncy5HgAvAZgFwEpJpBzw2h45eG2PHLy2Rw5e2yMHr+2R42hf20Xe+9lhBlZ0I7XvpM6t896vrviJjwF4bY8cvLZHDl7bIwev7ZGD1/bIMZWuLa8uw/wAAAUuSURBVH+0RwghhBASEW6kCCGEEEIicrQ2UjccpfMeC/DaHjl4bY8cvLZHDl7bIwev7ZFjylzbo+JIEUIIIYRMB/ijPUIIIYSQiFR0I+Wcu8Q597xz7gXn3Gcqee7phnNuoXPuPufcM865jc65j42/3uqcu8s5t3n8z5ZSxyLBOOfizrnHnHN3jNeLnXOPjN+/P3bOVZU6BrE455qdcz9zzj3nnHvWOXcO79vy4Jz7xPh/D552zt3inKvhfRsd59z3nHO7nXNPH/Ba4L3qxviX8ev8pHPu9KO38snNQa7rV8f/m/Ckc+7nzrnmA9777Ph1fd45d/HRWfXBqdhGyjkXB/BtAJcCOBHAu51zJ1bq/NOQPIBPeu9PBHA2gI+MX8/PALjHe78MwD3jNYnGxwA8e0D9ZQBf994vBbAXwAeOyqqmPtcB+I33/gQAp2DsGvO+PUyccwsA/A2A1d77kzDWVvFd4H17ONwE4BL12sHu1UsBLBv/3zUA/q1Ca5yK3AR7Xe8CcJL3/mQAmwB8FgDG/3/tXQBWjc/51/H9xKShkk+kzgTwgvd+i/c+C+BWAG+p4PmnFd77Xd77DeN/H8LY/xktwNg1vXl82M0A3np0Vji1cc61A7gMwHfHawfgAgA/Gx/CaxsB59wMAK8BcCMAeO+z3vt+8L4tFwkAtc65BIA6ALvA+zYy3vvfA+hTLx/sXn0LgO/7MR4G0Oycm1eZlU4tgq6r9/633vs/tWl/GED7+N/fAuBW733Ge78VwAsY209MGiq5kVoAYPsBdef4a+Qwcc51ADgNwCMA2rz3u8bf6gLQdpSWNdX5BoC/A1Acr2cC6D/gHzrv32gsBtAD4N/Hf2z6XedcPXjfHjbe+x0ArgXwMsY2UAMA1oP3bbk52L3K/48rH38B4Nfjf5/015Wy+RTHOdcA4DYAH/feDx74nh/7lUz+WuYh4px7E4Dd3vv1R3st05AEgNMB/Jv3/jQAI1A/xuN9G41xV+ctGNuszgdQD/vjE1JGeK+WH+fc5zCmrvzwaK8lLJXcSO0AsPCAun38NRIR51wSY5uoH3rvbx9/uftPj5PH/9x9tNY3hTkPwOXOuW0Y+xH0BRjzeprHf2QC8P6NSieATu/9I+P1zzC2seJ9e/hcBGCr977He58DcDvG7mXet+XlYPcq/z/uMHHOvQ/AmwC8x+/PZpr017WSG6m1AJaN/wZJFcbksTUVPP+0YtzZuRHAs977rx3w1hoA7x3/+3sB/KLSa5vqeO8/671v9953YOw+vdd7/x4A9wG4cnwYr20EvPddALY751aMv3QhgGfA+7YcvAzgbOdc3fh/H/50bXnflpeD3atrAPz5+G/vnQ1g4IAfAZISOOcuwZhOcbn3PnXAW2sAvMs5V+2cW4wxmf/Ro7HGg1HRQE7n3Bsx5p7EAXzPe/9/KnbyaYZz7lUA/gDgKez3eP4eY57UTwAcB+AlAO/w3mtZkoTEOXc+gE9579/knDseY0+oWgE8BuAq733maK5vKuKcOxVjEn8VgC0A3o+xb+p43x4mzrl/AvBOjP1o5DEAV2PMJ+F9GwHn3C0AzgcwC0A3gH8E8J8IuFfHN6/fwtiPU1MA3u+9X3c01j3ZOch1/SyAagC948Me9t5/eHz85zDmTeUxprH8Wh/zaMJkc0IIIYSQiFA2J4QQQgiJCDdShBBCCCER4UaKEEIIISQi3EgRQgghhESEGylCCCGEkIhwI0UIIYQQEhFupAghhBBCIsKNFCGEEEJIRP4/WtI4353zLVoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "positional_enc = PositionalEncoding(128).to(device)\n",
    "data = torch.zeros(1, 50, 128).to(device)\n",
    "data_pos_enc = positional_enc.forward(data)\n",
    "\n",
    "enc_np = data_pos_enc.squeeze(dim=0).to('cpu').numpy()\n",
    "fig, ax = plt.subplots(figsize=(12, 4))\n",
    "ax.imshow(enc_np)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Transformer model\n",
    "\n",
    "The biggest addition to the Transformer module is the translate() function. The translate function reads the source sentence, encodes it with one encoder forward run and then using the **Value** and **Key** matrices from encoder it runs the decoder as many times as needed, generating one additional word per run, until the < end-of-sentence > token is generated or max sentence length is reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class Transformer(nn.Module):\n",
    "    def __init__(self, num_layers, d_model, num_att_heads, input_dict_size, output_dict_size):\n",
    "        super().__init__()\n",
    "\n",
    "        self.max_sent_len = 50\n",
    "\n",
    "        self.input_emb = nn.Embedding(input_dict_size, d_model)\n",
    "        self.outp_emb = nn.Embedding(output_dict_size, d_model)\n",
    "\n",
    "        self.positional_encoder = PositionalEncoding(d_model)\n",
    "        self.encoder = Encoder(num_layers, d_model, num_att_heads)\n",
    "        self.decoder = Decoder(num_layers, d_model, num_att_heads)\n",
    "\n",
    "        self.outp_logits = nn.Linear(d_model, output_dict_size)\n",
    "        self.softmax = nn.Softmax(dim=2)\n",
    "\n",
    "    def translate(self, src, tgt_start_code, tgt_eos_code, src_padding_mask, src_subsq_mask):\n",
    "\n",
    "        # TODO: Beam search\n",
    "\n",
    "        enc_x = self.input_emb.forward(src.squeeze(dim=2))\n",
    "        enc_x = self.positional_encoder.forward(enc_x)\n",
    "        enc_keys, enc_values = self.encoder.forward(enc_x, src_padding_mask, src_subsq_mask)\n",
    "\n",
    "        snt = torch.ones((1,1,1)) * tgt_start_code\n",
    "        snt = snt.long()\n",
    "        snt = snt.to(device)\n",
    "\n",
    "        translation_idxes = []\n",
    "\n",
    "        for idx in range(self.max_sent_len):\n",
    "\n",
    "            dec_x = self.outp_emb.forward(snt.squeeze(dim=2))\n",
    "            dec_x = self.positional_encoder.forward(dec_x)\n",
    "            dec_x = self.decoder.forward(\n",
    "                dec_x,\n",
    "                src_padding_mask = src_padding_mask,\n",
    "                tgt_padding_mask = torch.zeros_like(snt).float().to(device),\n",
    "                tgt_subsq_mask = get_square_subsequent_mask(snt.size()[1]),\n",
    "                mem_keys = enc_keys,\n",
    "                mem_values = enc_values\n",
    "            )\n",
    "            dec_x = self.outp_logits.forward(dec_x)\n",
    "            dec_x = self.softmax(dec_x)\n",
    "            next_word_softmax = dec_x[0,idx,:].to('cpu').detach()\n",
    "            next_word_idx = torch.argmax(next_word_softmax)\n",
    "            snt = torch.cat([snt, torch.ones((1,1,1)).long().to(device) * next_word_idx], dim=1)\n",
    "\n",
    "            translation_idxes.append(next_word_idx)\n",
    "\n",
    "            if next_word_idx == tgt_eos_code:\n",
    "                break\n",
    "\n",
    "        return translation_idxes\n",
    "\n",
    "\n",
    "    def forward(self, src, tgt, src_padding_mask, src_subsq_mask, tgt_padding_mask, tgt_subsq_mask):\n",
    "\n",
    "        enc_x = self.input_emb.forward(src.squeeze(dim=2))\n",
    "        enc_x = self.positional_encoder.forward(enc_x)\n",
    "        enc_keys, enc_values = self.encoder.forward(enc_x, src_padding_mask, src_subsq_mask)\n",
    "\n",
    "        dec_x = self.outp_emb.forward(tgt.squeeze(dim=2))\n",
    "        dec_x = self.positional_encoder.forward(dec_x)\n",
    "        dec_x = self.decoder.forward(dec_x, src_padding_mask, tgt_padding_mask, tgt_subsq_mask, enc_keys, enc_values)\n",
    "        dec_x = self.outp_logits.forward(dec_x)\n",
    "        dec_x = self.softmax(dec_x)\n",
    "\n",
    "        return dec_x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some helper functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_square_subsequent_mask(seq_len):\n",
    "    mask = (torch.triu(torch.ones(seq_len, seq_len).to(device)) == 1).transpose(0, 1)\n",
    "    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n",
    "    return mask\n",
    "\n",
    "def get_padding_mask(input, val1 = float('-inf'), val2 = float(0.0)):\n",
    "    mask = torch.ones(input.size()).to(device)\n",
    "    mask = mask.float().masked_fill(input == 0, val1).masked_fill(input > 0, val2)\n",
    "    return mask\n",
    "\n",
    "\n",
    "def get_one_hot(x, out_dim, mask):\n",
    "\n",
    "    tens = x.view(-1)\n",
    "    tens_one_hot = torch.zeros(list(tens.size()) + [out_dim]).to(device)\n",
    "    for i in range(len(tens)):\n",
    "        tens_one_hot[i,tens[i]] = 1\n",
    "\n",
    "    tens_one_hot = tens_one_hot.view(list(x.size()) + [out_dim])\n",
    "    tens_one_hot = tens_one_hot * mask\n",
    "    return tens_one_hot.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Prepare the sentence for model translate function\n",
    "def translate_sentences(src_sentences, tgt_sentences, max_sent_num = 15):\n",
    "\n",
    "    transformer_model.eval()\n",
    "\n",
    "    with torch.no_grad():\n",
    "\n",
    "        for snt_idx in range(len(src_sentences)):\n",
    "\n",
    "            if snt_idx > max_sent_num:\n",
    "                break\n",
    "\n",
    "            src = src_sentences[snt_idx:snt_idx+1]\n",
    "\n",
    "            padded_src = pad_sequence(src, padding_value=0, batch_first=True).to(device)\n",
    "            src_padding_mask = get_padding_mask(padded_src)\n",
    "            src_subsq_mask = get_square_subsequent_mask(padded_src.size()[1])\n",
    "\n",
    "            snt_translation = transformer_model.translate(\n",
    "                src = padded_src,\n",
    "                tgt_start_code = dataset.get_eng_start_code(),\n",
    "                tgt_eos_code = dataset.get_eng_eos_code(),\n",
    "                src_padding_mask = src_padding_mask,\n",
    "                src_subsq_mask = src_subsq_mask\n",
    "            )\n",
    "\n",
    "            src_sent = ''\n",
    "            for word_idx in src_sentences[snt_idx]:\n",
    "                src_sent = f\"{src_sent} {dataset.fra_token_to_text[word_idx]}\"\n",
    "\n",
    "            tgt_sent = ''\n",
    "            for word_idx in tgt_sentences[snt_idx]:\n",
    "                tgt_sent = f\"{tgt_sent} {dataset.eng_token_to_text[word_idx]}\"\n",
    "\n",
    "            translated_sent = ''\n",
    "            for word_idx in snt_translation:\n",
    "                translated_sent = f\"{translated_sent} {dataset.eng_token_to_text[word_idx]}\"\n",
    "\n",
    "            print(f\"Source sentence is: {src_sent}\")\n",
    "            print(f\"Target sentence is: {tgt_sent}\")\n",
    "            print(f\"Model translation is: {translated_sent}\")\n",
    "\n",
    "    transformer_model.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparams, model definition and dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "170190\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE = 64\n",
    "LEARNING_RATE = 1e-4\n",
    "EPOCHS = 10\n",
    "STORE_MODELS = True\n",
    "models_path = 'models'\n",
    "\n",
    "if not os.path.exists(models_path):\n",
    "    os.mkdir(models_path)\n",
    "\n",
    "dataset = FraEngDataset()\n",
    "sentences_loader = DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True, drop_last=True, collate_fn=fra_eng_dataset_collate)\n",
    "\n",
    "in_dict_size = dataset.get_fra_dict_size()\n",
    "out_dict_size = dataset.get_eng_dict_size()\n",
    "\n",
    "transformer_model = Transformer(\n",
    "    num_layers=6,\n",
    "    d_model=512,\n",
    "    num_att_heads=8,\n",
    "    input_dict_size=in_dict_size,\n",
    "    output_dict_size=out_dict_size\n",
    ").to(device)\n",
    "\n",
    "optimizer = torch.optim.Adam(transformer_model.parameters(), lr = 1e-4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training and sentence translation\n",
    "To follow the progress of the model I will run sentence translation at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 ============================================================\n",
      "Total loss per word: 2.284416913986206\n",
      "Some translated sentences:\n",
      "Source sentence is:  J'ai de nombreux vieux livres . Quelques-uns d'entre eux sont d'assez grande valeur . <EOS>\n",
      "Target sentence is:  <START> I have a lot of old books . A couple of them are quite valuable .\n",
      "Model translation is:  I have many books books . The teacher is a great lot of them . <EOS>\n",
      "Source sentence is:  J'ai seulement mangé une tartine et bu un café ce matin . <EOS>\n",
      "Target sentence is:  <START> I only had a piece of toast and a cup of coffee this morning .\n",
      "Model translation is:  I only ate a new and drank this morning . <EOS>\n",
      "Source sentence is:  Si deux personnes sont toujours du même avis , alors l'une d'elle est inutile . <EOS>\n",
      "Target sentence is:  <START> If two people are in agreement , one of them is unnecessary .\n",
      "Model translation is:  If two people are always the same opinion , then one one of her . <EOS>\n",
      "Source sentence is:  Je ne sais pas comment nous allons faire ça . <EOS>\n",
      "Target sentence is:  <START> I do n't know how we 're going to do that .\n",
      "Model translation is:  I do n't know how we 're going to do this . <EOS>\n",
      "Source sentence is:  Il ne permettrait à personne de se mêler de ses affaires . <EOS>\n",
      "Target sentence is:  <START> He would n't let anyone interfere in his personal affairs .\n",
      "Model translation is:  He does n't tell anyone about his business . <EOS>\n",
      "Source sentence is:  Il y a une chanson que je souhaite vous chanter . <EOS>\n",
      "Target sentence is:  <START> There 's a song I want to sing for you .\n",
      "Model translation is:  There is a song I want to sing . <EOS>\n",
      "Source sentence is:  Si tu prends ce remède , tu te sentiras mieux . <EOS>\n",
      "Target sentence is:  <START> If you take this medicine , you 'll feel better .\n",
      "Model translation is:  If you take this medicine , you 'll have better . <EOS>\n",
      "Source sentence is:  Il apporta de la nourriture à son invité et lui procura un abri . <EOS>\n",
      "Target sentence is:  <START> He brought food to his guest and provided him shelter .\n",
      "Model translation is:  He is taking food and invited him to a lie . <EOS>\n",
      "Source sentence is:  Je pensais t'avoir dit de rester en arrière . <EOS>\n",
      "Target sentence is:  <START> I thought that I told you to stay behind .\n",
      "Model translation is:  I thought I told you to stay . <EOS>\n",
      "Source sentence is:  Je devrais sans doute rentrer à la maison voir bobonne . <EOS>\n",
      "Target sentence is:  <START> I guess I should get home to the missus .\n",
      "Model translation is:  I should see without going home . <EOS>\n",
      "Source sentence is:  Je n'ai personne d'autre vers qui me tourner . <EOS>\n",
      "Target sentence is:  <START> I do n't have anybody else to turn to .\n",
      "Model translation is:  I have no one else to do . <EOS>\n",
      "Source sentence is:  Il vient voir mon fils de temps en temps . <EOS>\n",
      "Target sentence is:  <START> He comes to see my son now and then .\n",
      "Model translation is:  He just saw my son to time . <EOS>\n",
      "Source sentence is:  Tu es trop jeune pour y aller tout seul . <EOS>\n",
      "Target sentence is:  <START> You 're too young to go there alone .\n",
      "Model translation is:  You 're too young to go alone . <EOS>\n",
      "Source sentence is:  J'ai trouvé la photo que tu cherchais . <EOS>\n",
      "Target sentence is:  <START> I found the picture you were looking for .\n",
      "Model translation is:  I found the picture you were looking for . <EOS>\n",
      "Source sentence is:  Le cancer peut être guéri s'il est découvert à temps . <EOS>\n",
      "Target sentence is:  <START> Cancer can be cured if discovered in time .\n",
      "Model translation is:  The cancer can be able to take out of time . <EOS>\n",
      "Source sentence is:  J'ai joué avec Tom et Marie aujourd'hui . <EOS>\n",
      "Target sentence is:  <START> I played with Tom and Mary today .\n",
      "Model translation is:  I played with Tom and Mary today . <EOS>\n",
      "Epoch 1 ============================================================\n",
      "Total loss per word: 1.2289352416992188\n",
      "Some translated sentences:\n",
      "Source sentence is:  Tu n'es pas obligée d'en parler si tu ne veux pas . <EOS>\n",
      "Target sentence is:  <START> You do n't have to talk about it if you do n't want to .\n",
      "Model translation is:  You do n't have to talk about it if you do n't want to . <EOS>\n",
      "Source sentence is:  Je ne te blâme pas pour l'accident . Ce n'était pas ta faute . <EOS>\n",
      "Target sentence is:  <START> I do n't blame you for the accident . It was n't your fault .\n",
      "Model translation is:  I do n't deserve you for the accident . It was n't your fault . <EOS>\n",
      "Source sentence is:  C'est en 1912 que le Titanic a coulé lors de son premier voyage . <EOS>\n",
      "Target sentence is:  <START> It was in 1912 that the Titanic sank during her first voyage .\n",
      "Model translation is:  It 's in a few days that the Titanic broke out his first trip . <EOS>\n",
      "Source sentence is:  As-tu entendu dire qu'un cambrioleur a forcé la maison de mon voisin ? <EOS>\n",
      "Target sentence is:  <START> Have you heard that a burglar broke into my neighbor 's house ?\n",
      "Model translation is:  Did you hear a burglar that a burglar broke out my neighbor 's house ? <EOS>\n",
      "Source sentence is:  S'il connaissait son numéro de téléphone , il pourrait l'appeler . <EOS>\n",
      "Target sentence is:  <START> If he knew her phone number , he could call her .\n",
      "Model translation is:  If he knew her phone number , he might call him . <EOS>\n",
      "Source sentence is:  Je vais arrêter d'essayer d'être amical à votre égard . <EOS>\n",
      "Target sentence is:  <START> I 'm going to stop trying to be friendly with you .\n",
      "Model translation is:  I 'll stop trying to be friendly with you . <EOS>\n",
      "Source sentence is:  Je veux séjourner en Amérique pendant quelques années . <EOS>\n",
      "Target sentence is:  <START> I want to stay in America for a few years .\n",
      "Model translation is:  I want to stay in America for a few years . <EOS>\n",
      "Source sentence is:  Nous mangeons habituellement avec un couteau , une fourchette et une cuillère . <EOS>\n",
      "Target sentence is:  <START> We usually eat with a knife , fork and spoon .\n",
      "Model translation is:  We usually eat with a knife , and a spoon . <EOS>\n",
      "Source sentence is:  Mon professeur d'anglais a recommandé que je lise ces ouvrages . <EOS>\n",
      "Target sentence is:  <START> My English teacher recommended that I read these books .\n",
      "Model translation is:  My English teacher recommended that I read these books . <EOS>\n",
      "Source sentence is:  C'est la chose la plus étrange que j'ai jamais vue . <EOS>\n",
      "Target sentence is:  <START> That 's the strangest thing I 've ever seen .\n",
      "Model translation is:  This is the most strange thing I 've ever seen . <EOS>\n",
      "Source sentence is:  Si nous partons maintenant , nous devrions y parvenir . <EOS>\n",
      "Target sentence is:  <START> If we leave now , we should make it .\n",
      "Model translation is:  If we leave now , we should go there . <EOS>\n",
      "Source sentence is:  Je suis heureux de voir tant de visages amicaux . <EOS>\n",
      "Target sentence is:  <START> I 'm happy to see so many friendly faces .\n",
      "Model translation is:  I 'm happy to see so many glasses . <EOS>\n",
      "Source sentence is:  J'ai pensé que vous aimeriez peut-être du café chaud . <EOS>\n",
      "Target sentence is:  <START> I thought you might like some hot coffee .\n",
      "Model translation is:  I thought you might like some hot coffee . <EOS>\n",
      "Source sentence is:  Je pense qu'il se pourrait que vous ayez un problème d'alcool . <EOS>\n",
      "Target sentence is:  <START> I think you might have a drinking problem .\n",
      "Model translation is:  I think you might have a problem . <EOS>\n",
      "Source sentence is:  Tom ne devrait pas être autorisé à faire cela . <EOS>\n",
      "Target sentence is:  <START> Tom should n't be allowed to do that .\n",
      "Model translation is:  Tom should n't be allowed to do that . <EOS>\n",
      "Source sentence is:  Je pensais à acheter un nouvel appareil photo . <EOS>\n",
      "Target sentence is:  <START> I was thinking about buying a new camera .\n",
      "Model translation is:  I thought I 'd buy a new camera . <EOS>\n",
      "Epoch 2 ============================================================\n",
      "Total loss per word: 0.8988859057426453\n",
      "Some translated sentences:\n",
      "Source sentence is:  Si je ne le fais pas maintenant , je ne le ferai jamais . <EOS>\n",
      "Target sentence is:  <START> If I do n't do that now , I 'll never do it .\n",
      "Model translation is:  If I do n't do that now , I 'll never do it . <EOS>\n",
      "Source sentence is:  Maintenant , pourquoi irais-tu commettre une telle bêtise ? <EOS>\n",
      "Target sentence is:  <START> Now why would you go and do a stupid thing like that ?\n",
      "Model translation is:  Now why do you think you 're looking for a stupid thing ? <EOS>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source sentence is:  Après qu'il se fut arrêté de pleuvoir , il sortit faire une promenade . <EOS>\n",
      "Target sentence is:  <START> Since it stopped raining , he went out for a walk .\n",
      "Model translation is:  After it stopped raining , he went out for a walk . <EOS>\n",
      "Source sentence is:  Cela a couté bien plus que je le pensais . <EOS>\n",
      "Target sentence is:  <START> It cost a lot more than I thought it would .\n",
      "Model translation is:  It has been a lot more than I thought . <EOS>\n",
      "Source sentence is:  Ne fourre pas le nez dans un vase étranger . <EOS>\n",
      "Target sentence is:  <START> Do n't stick your nose where it does n't belong .\n",
      "Model translation is:  Do n't make me in a vase vase . <EOS>\n",
      "Source sentence is:  Tom n ' a pas eu le temps hier de regarder la télévision . <EOS>\n",
      "Target sentence is:  <START> Tom did n't have time to watch TV yesterday .\n",
      "Model translation is:  Tom did n't have time to watch TV yesterday . <EOS>\n",
      "Source sentence is:  Un agent des douanes me demanda d'ouvrir ma valise . <EOS>\n",
      "Target sentence is:  <START> A customs official asked me to open my suitcase .\n",
      "Model translation is:  A member of mine asked me to open my suitcase . <EOS>\n",
      "Source sentence is:  Faites attention , il y a des couguars dans ce coin ! <EOS>\n",
      "Target sentence is:  <START> Be careful , there are cougars in this area .\n",
      "Model translation is:  Be careful , there are cougars in this area . <EOS>\n",
      "Source sentence is:  La musique m ' a ramenée à mon enfance . <EOS>\n",
      "Target sentence is:  <START> The music carried me back to my childhood .\n",
      "Model translation is:  The music gave me up to my childhood . <EOS>\n",
      "Source sentence is:  J'ai été au Japon pendant deux mois . <EOS>\n",
      "Target sentence is:  <START> I have been in Japan for two months .\n",
      "Model translation is:  I 've been in Japan for two months . <EOS>\n",
      "Source sentence is:  N'êtes-vous pas trop jeune pour ce travail ? <EOS>\n",
      "Target sentence is:  <START> Are n't you too young for that job ?\n",
      "Model translation is:  Are n't you too young for this work ? <EOS>\n",
      "Source sentence is:  Tom ignorait qu'il ferait du mal à quelqu'un . <EOS>\n",
      "Target sentence is:  <START> Tom did n't know he 'd hurt anybody .\n",
      "Model translation is:  Tom did n't know he would hurt someone . <EOS>\n",
      "Source sentence is:  Mère était très occupée la plupart du temps . <EOS>\n",
      "Target sentence is:  <START> Mother was very busy most of the time .\n",
      "Model translation is:  Mother was very busy most of the time . <EOS>\n",
      "Source sentence is:  Le gratte-ciel a été érigé sur de solides fondations . <EOS>\n",
      "Target sentence is:  <START> The skyscraper was built on a solid foundation .\n",
      "Model translation is:  The coach was built in order . <EOS>\n",
      "Source sentence is:  Peux-tu me dire pourquoi tu l'aimes ? <EOS>\n",
      "Target sentence is:  <START> Can you tell me why you like him ?\n",
      "Model translation is:  Can you tell me why you like her ? <EOS>\n",
      "Source sentence is:  Je ne fus pas invité à la cérémonie d'ouverture . <EOS>\n",
      "Target sentence is:  <START> I was n't invited to the opening ceremony .\n",
      "Model translation is:  I was n't invited to the opening ceremony . <EOS>\n",
      "Epoch 3 ============================================================\n",
      "Total loss per word: 0.7014974355697632\n",
      "Some translated sentences:\n",
      "Source sentence is:  Pourquoi ne restes-tu pas un moment après que tout le monde soit parti de manière à ce que nous puissions discuter ? <EOS>\n",
      "Target sentence is:  <START> Why do n't you hang around a while after everyone else leaves so we can talk ?\n",
      "Model translation is:  Why do n't you stay for everybody to talk ? <EOS>\n",
      "Source sentence is:  Il adopte l'attitude que la résistance est une perte d'énergie . <EOS>\n",
      "Target sentence is:  <START> He takes the attitude that resistance is a waste of energy .\n",
      "Model translation is:  He says that the world is a waste of energy . <EOS>\n",
      "Source sentence is:  Je ne vois pas en quoi ce sont vos affaires . <EOS>\n",
      "Target sentence is:  <START> I do n't see how that is any of your business .\n",
      "Model translation is:  I do n't see how you are . <EOS>\n",
      "Source sentence is:  Je suis peut-être ivre mais je ne suis pas fou . <EOS>\n",
      "Target sentence is:  <START> I might be drunk , but I 'm not crazy .\n",
      "Model translation is:  I may be drunk , but I 'm not crazy . <EOS>\n",
      "Source sentence is:  Si tu as la moindre difficulté , n'hésites pas à me demander de l'aide . <EOS>\n",
      "Target sentence is:  <START> If you have any difficulty , ask me for help .\n",
      "Model translation is:  If you have any difficulty , help me with help . <EOS>\n",
      "Source sentence is:  Je n'aime pas porter les vêtements de quelqu'un d'autre . <EOS>\n",
      "Target sentence is:  <START> I do not like wearing anybody else 's clothes .\n",
      "Model translation is:  I do n't like to wear someone else 's clothes . <EOS>\n",
      "Source sentence is:  Je n'ai pas prétendu être votre amie . <EOS>\n",
      "Target sentence is:  <START> I did n't pretend to be your friend .\n",
      "Model translation is:  I did n't pretend to be your friend . <EOS>\n",
      "Source sentence is:  J'ai fait ce que Tom m ' a dit . <EOS>\n",
      "Target sentence is:  <START> I did what Tom told me to do .\n",
      "Model translation is:  I did what Tom told me . <EOS>\n",
      "Source sentence is:  Toutes les filles sont folles d'un homme bien habillé . <EOS>\n",
      "Target sentence is:  <START> Every girl 's crazy about a sharp-dressed man .\n",
      "Model translation is:  All the girls are crazy of a man . <EOS>\n",
      "Source sentence is:  Le résultat dépend entièrement de tes propres efforts . <EOS>\n",
      "Target sentence is:  <START> The outcome depends entirely on your own efforts .\n",
      "Model translation is:  The result is all up to your own efforts . <EOS>\n",
      "Source sentence is:  J'ai été impressionnée par le travail de Tom . <EOS>\n",
      "Target sentence is:  <START> I 've been impressed with Tom 's work .\n",
      "Model translation is:  I was impressed with Tom 's work . <EOS>\n",
      "Source sentence is:  Pourquoi n'allez-vous pas discuter avec lui ? <EOS>\n",
      "Target sentence is:  <START> Why do n't you go talk to him ?\n",
      "Model translation is:  Why do n't you go talk to him ? <EOS>\n",
      "Source sentence is:  Nous allâmes faire une promenade sur la plage . <EOS>\n",
      "Target sentence is:  <START> We went for a walk on the beach .\n",
      "Model translation is:  We went for a walk on the beach . <EOS>\n",
      "Source sentence is:  Il contra leur proposition par une suggestion surprenante . <EOS>\n",
      "Target sentence is:  <START> He countered their proposal with a surprising suggestion .\n",
      "Model translation is:  He would accept their proposal by a surprising suggestion . <EOS>\n",
      "Source sentence is:  Nous ne sommes pas supposées faire ça . <EOS>\n",
      "Target sentence is:  <START> We 're not supposed to do that .\n",
      "Model translation is:  We 're not supposed to do that . <EOS>\n",
      "Source sentence is:  C'est tout ce que j'ai besoin de dire . <EOS>\n",
      "Target sentence is:  <START> That 's all I need to say .\n",
      "Model translation is:  That 's all I need to say . <EOS>\n",
      "Epoch 4 ============================================================\n",
      "Total loss per word: 0.563602089881897\n",
      "Some translated sentences:\n",
      "Source sentence is:  On a dépensé un paquet d'argent sur des choses dont on ne se sert pas vraiment . <EOS>\n",
      "Target sentence is:  <START> We spend piles of money on the things we do n't really use .\n",
      "Model translation is:  A package was spent a package on things you do n't really use . <EOS>\n",
      "Source sentence is:  Tu t ’ es trompée sur la date quand tu as rempli le chèque . <EOS>\n",
      "Target sentence is:  <START> You got the date wrong when you were filling in the check .\n",
      "Model translation is:  You were on the date when you were filled in the check . <EOS>\n",
      "Source sentence is:  Je ne vois pas de quoi vous êtes tellement contentes , toutes les deux . <EOS>\n",
      "Target sentence is:  <START> I do n't see what you two are so happy about .\n",
      "Model translation is:  I do n't see what you two are so happy about . <EOS>\n",
      "Source sentence is:  Je ne veux pas entendre un mot de plus à ce sujet ! <EOS>\n",
      "Target sentence is:  <START> I do n't want to hear another word about it !\n",
      "Model translation is:  I do n't want to hear another word about it . <EOS>\n",
      "Source sentence is:  Tu dois trouver une autre issue à cette situation . <EOS>\n",
      "Target sentence is:  <START> You need to find another way out of this situation .\n",
      "Model translation is:  You must find another way out to this situation . <EOS>\n",
      "Source sentence is:  Elle fit de son mieux pour ne jamais songer à lui . <EOS>\n",
      "Target sentence is:  <START> She did her best never to think of him .\n",
      "Model translation is:  She did her best never to think of him . <EOS>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source sentence is:  Les fortes chutes de neige leur ont fait repousser leur départ . <EOS>\n",
      "Target sentence is:  <START> The heavy snow made them put off their departure .\n",
      "Model translation is:  The heavy snowfall made their snow set . <EOS>\n",
      "Source sentence is:  Pouvez-vous me dire pourquoi Tom n'est pas là ? <EOS>\n",
      "Target sentence is:  <START> Can you tell me why Tom is n't here ?\n",
      "Model translation is:  Can you tell me why Tom is n't here ? <EOS>\n",
      "Source sentence is:  Je ne suis pas la seule à prévoir de le faire . <EOS>\n",
      "Target sentence is:  <START> I 'm the only one planning to do that .\n",
      "Model translation is:  I 'm not the only one planning to do that . <EOS>\n",
      "Source sentence is:  Quel est le fruit le plus délicieux au Japon ? <EOS>\n",
      "Target sentence is:  <START> What 's the most delicious fruit in Japan ?\n",
      "Model translation is:  What 's the most delicious fruit in Japan ? <EOS>\n",
      "Source sentence is:  Ce ne sera pas difficile de faire ça . <EOS>\n",
      "Target sentence is:  <START> It wo n't be difficult to do that .\n",
      "Model translation is:  It wo n't be difficult to do that . <EOS>\n",
      "Source sentence is:  Veux-tu plus de lait dans ton café ? <EOS>\n",
      "Target sentence is:  <START> Do you want more milk in your coffee ?\n",
      "Model translation is:  Do you want more milk in your coffee ? <EOS>\n",
      "Source sentence is:  Vous allez vouloir voir ça . <EOS>\n",
      "Target sentence is:  <START> You 're going to want to see this .\n",
      "Model translation is:  You 're going to want to see this . <EOS>\n",
      "Source sentence is:  Nous allons nous marier en octobre . <EOS>\n",
      "Target sentence is:  <START> We 're going to get married in October .\n",
      "Model translation is:  We 're going to get married in October . <EOS>\n",
      "Source sentence is:  Je veux seulement ce qui est le mieux pour tout le monde . <EOS>\n",
      "Target sentence is:  <START> I only want what 's best for everyone .\n",
      "Model translation is:  I just want what 's best for everyone . <EOS>\n",
      "Source sentence is:  Vous êtes deux fois plus fort que moi . <EOS>\n",
      "Target sentence is:  <START> You are twice as strong as me .\n",
      "Model translation is:  You are twice as strong as I am . <EOS>\n",
      "Epoch 5 ============================================================\n",
      "Total loss per word: 0.4647580087184906\n",
      "Some translated sentences:\n",
      "Source sentence is:  Ça ne fait pas une grosse différence que tu y ailles en taxi ou à pied . <EOS>\n",
      "Target sentence is:  <START> It will make little difference whether you go there by taxi or on foot .\n",
      "Model translation is:  It does n't make a big difference whether you go by taxi or on foot . <EOS>\n",
      "Source sentence is:  Nous avons deux chiens . L'un est noir et l'autre est blanc . <EOS>\n",
      "Target sentence is:  <START> We have two dogs . One is black and the other is white .\n",
      "Model translation is:  We have two dogs . One is black black , and the other is white . <EOS>\n",
      "Source sentence is:  Je dois lui dire que le téléphone a sonné pendant qu'elle était sortie . <EOS>\n",
      "Target sentence is:  <START> I have to tell her that the phone rang while she was gone .\n",
      "Model translation is:  I have to tell her that the phone rang while she was gone . <EOS>\n",
      "Source sentence is:  Lorsque j'étudiais à l'université , j'ai toujours passé des nuits blanches avant les examens . <EOS>\n",
      "Target sentence is:  <START> When I was a college student , I always pulled all-nighters before tests .\n",
      "Model translation is:  When I was a college , I always spent white before tests the tests . <EOS>\n",
      "Source sentence is:  Mon père est parti en Chine . Il ne se trouve pas là pour l'instant . <EOS>\n",
      "Target sentence is:  <START> My father has gone to China . He is n't here now .\n",
      "Model translation is:  My father has gone to China . He is n't here now . <EOS>\n",
      "Source sentence is:  La seule chose qui compte est que vous n'avez pas été blessé . <EOS>\n",
      "Target sentence is:  <START> The only thing that matters is that you were n't injured .\n",
      "Model translation is:  The only thing that matters is that you were n't injured . <EOS>\n",
      "Source sentence is:  Je connais beaucoup de gens qui parlent français . <EOS>\n",
      "Target sentence is:  <START> I know a lot of people who can speak French .\n",
      "Model translation is:  I know a lot of people who can speak French . <EOS>\n",
      "Source sentence is:  Elle lui a demandé d'aider son père à nettoyer le garage . <EOS>\n",
      "Target sentence is:  <START> She asked him to help her father clean the garage .\n",
      "Model translation is:  She asked him to help her father clean the garage . <EOS>\n",
      "Source sentence is:  Tom et Mary ont décidé de se marier en octobre . <EOS>\n",
      "Target sentence is:  <START> Tom and Mary have decided to get married in October .\n",
      "Model translation is:  Tom and Mary decided to marry himself in October . <EOS>\n",
      "Source sentence is:  Tom lit un magazine , assis sur le canapé . <EOS>\n",
      "Target sentence is:  <START> Tom is sitting on the sofa , reading a magazine .\n",
      "Model translation is:  Tom is sitting on the couch , sitting on the couch . <EOS>\n",
      "Source sentence is:  C'est exactement comme je l'espérais . <EOS>\n",
      "Target sentence is:  <START> It 's just as I hoped it would be .\n",
      "Model translation is:  It 's exactly as I expected . <EOS>\n",
      "Source sentence is:  Je n'ai rien à te dire . <EOS>\n",
      "Target sentence is:  <START> I do n't have anything to say to you .\n",
      "Model translation is:  I have nothing to tell you . <EOS>\n",
      "Source sentence is:  Autant pécher par excès de prudence . <EOS>\n",
      "Target sentence is:  <START> Better to err on the side of caution .\n",
      "Model translation is:  It is constantly by caution . <EOS>\n",
      "Source sentence is:  Eh ! Qu'avez-vous derrière la tête , tous les deux ? <EOS>\n",
      "Target sentence is:  <START> Hey , what are you two up to ?\n",
      "Model translation is:  Hey , what are you two up to ? <EOS>\n",
      "Source sentence is:  Il m ' a dit que son père était mort . <EOS>\n",
      "Target sentence is:  <START> He told me that his father was dead .\n",
      "Model translation is:  He told me that his father died . <EOS>\n",
      "Source sentence is:  Je me lève à six heures presque tous les jours . <EOS>\n",
      "Target sentence is:  <START> I get up at six almost every day .\n",
      "Model translation is:  I wake up at six almost every day . <EOS>\n",
      "Epoch 6 ============================================================\n",
      "Total loss per word: 0.3884657025337219\n",
      "Some translated sentences:\n",
      "Source sentence is:  La floraison des cerisiers ne dure que quelques jours , une semaine tout au plus . <EOS>\n",
      "Target sentence is:  <START> Cherry blossoms last only for a few days , a week at the most .\n",
      "Model translation is:  Cherry blossoms last only a few days , a week at the most days . <EOS>\n",
      "Source sentence is:  Ça m'est égal que tu sois occupé . Aide-moi maintenant , je te prie ! <EOS>\n",
      "Target sentence is:  <START> I do n't care if you 're busy . Please help me now .\n",
      "Model translation is:  I do n't care if you 're busy . Please help me now . <EOS>\n",
      "Source sentence is:  Récemment , il y a eu beaucoup de manifestations dans la ville . <EOS>\n",
      "Target sentence is:  <START> Recently there have been a lot of protests in the city .\n",
      "Model translation is:  Recently there have been a lot of protests in town . <EOS>\n",
      "Source sentence is:  Si j'étais la patronne , je vous virerais . <EOS>\n",
      "Target sentence is:  <START> If I were the boss , I 'd fire you .\n",
      "Model translation is:  If I were the boss , I 'd fire you . <EOS>\n",
      "Source sentence is:  Pourquoi voudrais-tu faire une telle chose ? <EOS>\n",
      "Target sentence is:  <START> Why would you want to do a thing like that ?\n",
      "Model translation is:  Why would you want to do something like that ? <EOS>\n",
      "Source sentence is:  Un trajet en bateau dure plus longtemps qu'en voiture . <EOS>\n",
      "Target sentence is:  <START> A trip by boat takes longer than by car .\n",
      "Model translation is:  A party 's office is further by boat . <EOS>\n",
      "Source sentence is:  Pensez-vous vraiment que ce soit une bonne idée ? <EOS>\n",
      "Target sentence is:  <START> Do you really think this is a good idea ?\n",
      "Model translation is:  Do you really think it 's a good idea ? <EOS>\n",
      "Source sentence is:  Es-tu sûre qu'il n ' y a rien d'autre à boire ? <EOS>\n",
      "Target sentence is:  <START> Are you sure there 's nothing else to drink ?\n",
      "Model translation is:  Are you sure there 's nothing else to drink ? <EOS>\n",
      "Source sentence is:  Malgré tous ses revers , il reste optimiste . <EOS>\n",
      "Target sentence is:  <START> Despite all his setbacks , he remains an optimist .\n",
      "Model translation is:  Despite all his setbacks , he remains an optimist . <EOS>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source sentence is:  Il y a une librairie de l'autre côté de ma rue . <EOS>\n",
      "Target sentence is:  <START> There is a bookstore across from my house .\n",
      "Model translation is:  There is a bookstore across from my street . <EOS>\n",
      "Source sentence is:  J'ai dit à Tom comment Mary avait été blessée . <EOS>\n",
      "Target sentence is:  <START> I told Tom how Mary had been injured .\n",
      "Model translation is:  I told Tom how Mary had been injured . <EOS>\n",
      "Source sentence is:  Vous n'êtes pas censée être là-dedans . <EOS>\n",
      "Target sentence is:  <START> You 're not supposed to be in here .\n",
      "Model translation is:  You 're not supposed to be in here . <EOS>\n",
      "Source sentence is:  J'allai faire un tour en voiture dans la campagne . <EOS>\n",
      "Target sentence is:  <START> I went for a drive in the country .\n",
      "Model translation is:  A ride for a drive in the country . <EOS>\n",
      "Source sentence is:  Vous êtes apparu au bon moment . <EOS>\n",
      "Target sentence is:  <START> You 've turned up at the right moment .\n",
      "Model translation is:  You 've turned up at the right moment . <EOS>\n",
      "Source sentence is:  Il utilise un crayon avec une pointe fine . <EOS>\n",
      "Target sentence is:  <START> He uses a pencil with a fine point .\n",
      "Model translation is:  He uses a pencil with a fine point . <EOS>\n",
      "Source sentence is:  Elle croit que son fils est encore vivant . <EOS>\n",
      "Target sentence is:  <START> She believes her son is still alive .\n",
      "Model translation is:  She believes that her son is still alive . <EOS>\n",
      "Epoch 7 ============================================================\n",
      "Total loss per word: 0.33213016390800476\n",
      "Some translated sentences:\n",
      "Source sentence is:  Si ça vous convient , j'aimerais rester encore un moment . <EOS>\n",
      "Target sentence is:  <START> If it 's all right with you , I 'd like to stay for a while longer .\n",
      "Model translation is:  If it 's all right with you , I 'd like to stay for a while longer . <EOS>\n",
      "Source sentence is:  Ne pensez-vous pas que ça pourrait être un peu trop cher pour que nous l'achetions ? <EOS>\n",
      "Target sentence is:  <START> Do n't you think it might be a bit too expensive for us to buy ?\n",
      "Model translation is:  Do n't you think it could be a little too expensive for us to buy ? <EOS>\n",
      "Source sentence is:  N'oubliez pas que les emplois sont difficiles à trouver de nos jours . <EOS>\n",
      "Target sentence is:  <START> Do n't forget that good jobs are very hard to come by these days .\n",
      "Model translation is:  Do n't forget that jobs are very hard to come by these days . <EOS>\n",
      "Source sentence is:  Tom m ' a dit que j'étais celui à qui il voulait parler . <EOS>\n",
      "Target sentence is:  <START> Tom told me I was the one he wanted to talk to .\n",
      "Model translation is:  Tom told me I was the one he wanted to talk to . <EOS>\n",
      "Source sentence is:  Qui est cette mignonne fille avec laquelle je vous ai vues à la galerie commerciale ? <EOS>\n",
      "Target sentence is:  <START> Who 's that cute girl I saw you with at the mall ?\n",
      "Model translation is:  Who 's that cute girl I saw you with at the mall ? <EOS>\n",
      "Source sentence is:  Je ne pense pas que ça me dérangerait de manger chinois tous les jours . <EOS>\n",
      "Target sentence is:  <START> I do n't think I 'd mind eating Chinese food every day .\n",
      "Model translation is:  I do n't think I would mind eating Chinese food every day . <EOS>\n",
      "Source sentence is:  Si je te le disais , je devrais te tuer . <EOS>\n",
      "Target sentence is:  <START> If I told you , I 'd have to kill you .\n",
      "Model translation is:  If I told you , I should kill you . <EOS>\n",
      "Source sentence is:  Si je l'avais su , je te l'aurais dit . <EOS>\n",
      "Target sentence is:  <START> If I had known it , I would have told you .\n",
      "Model translation is:  If I had known it , I would have told you . <EOS>\n",
      "Source sentence is:  Tom ne m ' a pas dit qu'il ne pouvait pas parler français . <EOS>\n",
      "Target sentence is:  <START> Tom did n't tell me he could n't speak French .\n",
      "Model translation is:  Tom did n't tell me he could n't speak French . <EOS>\n",
      "Source sentence is:  En mars , ça fera un an que nous habitons ici . <EOS>\n",
      "Target sentence is:  <START> We will have lived here for a year next March .\n",
      "Model translation is:  In March , we will do a year here for us to call here . <EOS>\n",
      "Source sentence is:  Ferme les yeux et dis-moi ce que tu entends ! <EOS>\n",
      "Target sentence is:  <START> Close your eyes and tell me what you hear .\n",
      "Model translation is:  Close your eyes and tell me what you hear . <EOS>\n",
      "Source sentence is:  Ils disent que je suis une vieille femme . <EOS>\n",
      "Target sentence is:  <START> They say that I 'm an old woman .\n",
      "Model translation is:  They say I 'm an old woman . <EOS>\n",
      "Source sentence is:  Ce n'est pas quelque chose que je veuille faire . <EOS>\n",
      "Target sentence is:  <START> That 's not something I want to do .\n",
      "Model translation is:  That 's not something I want to do . <EOS>\n",
      "Source sentence is:  Je pense que je devrais le faire pour vous . <EOS>\n",
      "Target sentence is:  <START> I think I should do it for you .\n",
      "Model translation is:  I think I should do that for you . <EOS>\n",
      "Source sentence is:  Je suis tombé amoureuse dans un endroit improbable . <EOS>\n",
      "Target sentence is:  <START> I fell in love in an unlikely place .\n",
      "Model translation is:  I fell in love in an unlikely place . <EOS>\n",
      "Source sentence is:  Il fit ce que je lui dis de faire . <EOS>\n",
      "Target sentence is:  <START> He did what I told him to do .\n",
      "Model translation is:  He did what I told him to do . <EOS>\n",
      "Epoch 8 ============================================================\n",
      "Total loss per word: 0.2886980175971985\n",
      "Some translated sentences:\n",
      "Source sentence is:  Si tu passes trop de temps au soleil sans mettre de crème solaire , il est probable que tu attrapes un coup de soleil . <EOS>\n",
      "Target sentence is:  <START> If you spend too much time in the sun without putting on sunscreen , you are likely to get a sunburn .\n",
      "Model translation is:  If you spend too much time in the sun without putting on sunscreen , you 're likely to get a sunburn . <EOS>\n",
      "Source sentence is:  Pensez-vous que vos parents ont passé suffisamment de temps avec vous lorsque vous étiez adolescente ? <EOS>\n",
      "Target sentence is:  <START> Do you think your parents spent enough time with you when you were in your teens ?\n",
      "Model translation is:  Do you think your parents spent enough time with you when you were in your teens ? <EOS>\n",
      "Source sentence is:  Elle consacre l'essentiel de son temps à s'occuper de ses enfants . <EOS>\n",
      "Target sentence is:  <START> She spends a majority of her time taking care of her children .\n",
      "Model translation is:  She spends most of her time taking care of her children . <EOS>\n",
      "Source sentence is:  Le nombre d'élèves dans cette classe est limité à 15 . <EOS>\n",
      "Target sentence is:  <START> The number of students in the class is limited to fifteen .\n",
      "Model translation is:  The number of students in this class is limited to fifteen . <EOS>\n",
      "Source sentence is:  J'aimerais savoir où Tom a caché sa clé . <EOS>\n",
      "Target sentence is:  <START> I 'd like to know where Tom hid his key .\n",
      "Model translation is:  I 'd like to know where Tom hid his key . <EOS>\n",
      "Source sentence is:  Je n'aime pas les poissons qui ont trop d'arêtes . <EOS>\n",
      "Target sentence is:  <START> I do n't like to eat fish with many bones .\n",
      "Model translation is:  I do n't like fish who has too much famous . <EOS>\n",
      "Source sentence is:  Vous pouvez être assez serviable lorsque vous le voulez . <EOS>\n",
      "Target sentence is:  <START> You can be pretty helpful when you want to be .\n",
      "Model translation is:  You can be pretty helpful when you want to be . <EOS>\n",
      "Source sentence is:  C'était la première fois que je visitais le musée . <EOS>\n",
      "Target sentence is:  <START> It was the first time that I visited the museum .\n",
      "Model translation is:  This was the first time I visited the museum . <EOS>\n",
      "Source sentence is:  Je veux que vous sachiez combien je suis désolée . <EOS>\n",
      "Target sentence is:  <START> I want you to know how sorry I am .\n",
      "Model translation is:  I want you to know how sorry I am . <EOS>\n",
      "Source sentence is:  Je n'ai pas reçu ne serait-ce qu'une lettre d'elle . <EOS>\n",
      "Target sentence is:  <START> I did n't receive even one letter from her .\n",
      "Model translation is:  I did n't receive even one letter from her . <EOS>\n",
      "Source sentence is:  Comment as-tu fini par être le manager de Tom ? <EOS>\n",
      "Target sentence is:  <START> How did you end up being Tom 's manager ?\n",
      "Model translation is:  How did you wind up Tom 's manager ? <EOS>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source sentence is:  Comment sais-tu quelle profondeur a le lac ? <EOS>\n",
      "Target sentence is:  <START> How do you know how deep the lake is ?\n",
      "Model translation is:  How do you know how deep the lake is ? <EOS>\n",
      "Source sentence is:  Il y a un arrêt de bus à proximité de notre école . <EOS>\n",
      "Target sentence is:  <START> There 's a bus stop close to our school .\n",
      "Model translation is:  There is a bus stop near our school . <EOS>\n",
      "Source sentence is:  Je souhaite seulement vous poser quelques questions . <EOS>\n",
      "Target sentence is:  <START> I just want to ask you some questions .\n",
      "Model translation is:  I just want to ask you a few questions . <EOS>\n",
      "Source sentence is:  De qui es-tu le plus proche dans ta famille ? <EOS>\n",
      "Target sentence is:  <START> Who are you closest to in your family ?\n",
      "Model translation is:  Whose closest is the closest in your family ? <EOS>\n",
      "Source sentence is:  Vous pouvez vous rendre où bon vous semble . <EOS>\n",
      "Target sentence is:  <START> You can go wherever you want to go .\n",
      "Model translation is:  You can go wherever you want to go . <EOS>\n",
      "Epoch 9 ============================================================\n",
      "Total loss per word: 0.25465041399002075\n",
      "Some translated sentences:\n",
      "Source sentence is:  Mon Dieu ! Tu es la dernière personne que je m'attendais à voir dans une telle situation . <EOS>\n",
      "Target sentence is:  <START> Oh my gosh ! You 're the last person I expected to meet in a situation like this .\n",
      "Model translation is:  My God ! You 're the last person I expected to see in a situation like this . <EOS>\n",
      "Source sentence is:  Tom est bon en français mais est bien meilleur en anglais . <EOS>\n",
      "Target sentence is:  <START> Tom is good at French , but he 's a lot better at English .\n",
      "Model translation is:  Tom is good at French , but he 's a lot better at English . <EOS>\n",
      "Source sentence is:  Si on veut connaitre un pays , il faut apprendre son histoire . <EOS>\n",
      "Target sentence is:  <START> If you are to know a nation , you must learn its history .\n",
      "Model translation is:  If you can know a nation , you must learn its history . <EOS>\n",
      "Source sentence is:  Pendant que j'attendais le bus , j'ai vu un accident de la circulation . <EOS>\n",
      "Target sentence is:  <START> While I was waiting for the bus , I saw a traffic accident .\n",
      "Model translation is:  While I was waiting for the bus , I saw a traffic accident . <EOS>\n",
      "Source sentence is:  Je suis désolé . Je ne voulais pas te faire pleurer . <EOS>\n",
      "Target sentence is:  <START> I 'm sorry . I did n't mean to make you cry .\n",
      "Model translation is:  I 'm sorry . I did n't mean to make you cry . <EOS>\n",
      "Source sentence is:  J'aimerais que cette voiture soit réparée aussi vite que possible . <EOS>\n",
      "Target sentence is:  <START> I would like to have this car repaired as soon as possible .\n",
      "Model translation is:  I 'd like this car repaired as soon as possible . <EOS>\n",
      "Source sentence is:  Sur vingt étudiantes , une seule a lu le livre . <EOS>\n",
      "Target sentence is:  <START> Out of twenty students , only one has read the book .\n",
      "Model translation is:  Out of twenty students , only one has read the book . <EOS>\n",
      "Source sentence is:  Plusieurs voitures sont garées devant chez moi . <EOS>\n",
      "Target sentence is:  <START> A number of cars are parked in front of my house .\n",
      "Model translation is:  Several cars are parked in front of my house . <EOS>\n",
      "Source sentence is:  Le Canada et le Mexique sont situés le long des frontières américaines . <EOS>\n",
      "Target sentence is:  <START> Canada and Mexico both share a border with the USA .\n",
      "Model translation is:  Canada and Mexico both share a border with the USA . <EOS>\n",
      "Source sentence is:  La locomotive tractait une longue file de wagons de marchandises . <EOS>\n",
      "Target sentence is:  <START> The locomotive was pulling a long line of freight cars .\n",
      "Model translation is:  The locomotive was carrying a long line of freight cars . <EOS>\n",
      "Source sentence is:  J'aimerais que vous rencontriez mon frère , Tom . <EOS>\n",
      "Target sentence is:  <START> I 'd like you to meet my brother , Tom .\n",
      "Model translation is:  I 'd like you to meet my brother , Tom . <EOS>\n",
      "Source sentence is:  Je suis désolé , je ne voulais pas dire ça . <EOS>\n",
      "Target sentence is:  <START> I 'm sorry , I did n't mean that .\n",
      "Model translation is:  I 'm sorry , I did n't mean that . <EOS>\n",
      "Source sentence is:  Je viens d'apprendre six nouvelles choses à propos des wombats . <EOS>\n",
      "Target sentence is:  <START> I just learned six new facts about wombats .\n",
      "Model translation is:  I just learned six new shoes about wombats . <EOS>\n",
      "Source sentence is:  Elles veulent vous parler , un moment . <EOS>\n",
      "Target sentence is:  <START> They want to talk to you a moment .\n",
      "Model translation is:  They want to talk to you a moment . <EOS>\n",
      "Source sentence is:  Quels jeux avez-vous sur votre téléphone ? <EOS>\n",
      "Target sentence is:  <START> What games do you have on your phone ?\n",
      "Model translation is:  What games do you have on your phone ? <EOS>\n",
      "Source sentence is:  Je veux aller où que tu ailles . <EOS>\n",
      "Target sentence is:  <START> I want to go wherever you 're going .\n",
      "Model translation is:  I want to go wherever you 're going . <EOS>\n",
      "Traing done! Generating some more sentences.\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(EPOCHS):\n",
    "    \n",
    "    src_sentences = None\n",
    "    tgt_sentences = None\n",
    "\n",
    "    train_loss_sum = 0.0\n",
    "    total_word_count = 0.0\n",
    "\n",
    "    for sentences in sentences_loader:\n",
    "\n",
    "        src_sentences = sentences['fra_sentences']\n",
    "        tgt_sentences = sentences['eng_sentences']\n",
    "\n",
    "        tgt_sentences_out = []\n",
    "        for idx in range(len(tgt_sentences)):\n",
    "            tgt_sentences_out.append(tgt_sentences[idx][1:])\n",
    "            tgt_sentences[idx] = tgt_sentences[idx][:-1]\n",
    "\n",
    "        # Create tensors from token lists\n",
    "        padded_src = pad_sequence(src_sentences, padding_value=0, batch_first=True).to(device)\n",
    "        padded_tgt = pad_sequence(tgt_sentences, padding_value=0, batch_first=True).to(device)\n",
    "        padded_tgt_out = pad_sequence(tgt_sentences_out, padding_value=0, batch_first=True).to(device)\n",
    "\n",
    "        src_padding_mask = get_padding_mask(padded_src)\n",
    "        src_subsq_mask = get_square_subsequent_mask(padded_src.size()[1])\n",
    "\n",
    "        tgt_padding_mask = get_padding_mask(padded_tgt)\n",
    "        tgt_subsq_mask = get_square_subsequent_mask(padded_tgt.size()[1])\n",
    "\n",
    "\n",
    "        pred = transformer_model.forward(\n",
    "            src=padded_src,\n",
    "            tgt=padded_tgt,\n",
    "            src_padding_mask=src_padding_mask,\n",
    "            src_subsq_mask=src_subsq_mask,\n",
    "            tgt_padding_mask=tgt_padding_mask,\n",
    "            tgt_subsq_mask=tgt_subsq_mask\n",
    "        )\n",
    "\n",
    "        # Mask to zero one hot vectors corresponding to padded elements\n",
    "        one_hot_mask = get_padding_mask(padded_tgt_out, val1=float(0.0), val2=float(1.0))\n",
    "        y_one_hot = get_one_hot(padded_tgt_out.squeeze(dim=2), out_dict_size, mask=one_hot_mask)\n",
    "\n",
    "        loss = - torch.sum(torch.log(pred) * y_one_hot)\n",
    "\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        train_loss_sum += loss.detach().to('cpu').data\n",
    "        total_word_count += torch.sum(y_one_hot).to('cpu').data\n",
    "\n",
    "    print(f\"Epoch {epoch} \" + '=' * 60)\n",
    "    print(f\"Total loss per word: {train_loss_sum / total_word_count}\")\n",
    "    print(f\"Some translated sentences:\")\n",
    "    \n",
    "    translate_sentences(src_sentences,tgt_sentences) # Take the last batch and translate some sentences.\n",
    "\n",
    "    \n",
    "    if STORE_MODELS == True:\n",
    "        model_path = os.path.join(models_path, f'Epoch_{epoch}_model.pt')\n",
    "        torch.save(transformer_model, model_path)\n",
    "\n",
    "print(\"Traing done! Generating some more sentences.\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}