05_transformer/transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Transformer Architecture and Machine Translation with the Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "In this notebook, you will understand the Transformer architecture introduced in [Vaswani et al., 2017], learn how to load a pretrained Transformer model in GluonNLP and translate a few sentences youself with the `BeamSearchTranslator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:18:58.390004Z",
     "start_time": "2019-08-06T01:18:57.117533Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import math\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import numpy as np\n",
    "import mxnet as mx\n",
    "from mxnet import gluon, nd\n",
    "from mxnet.gluon import nn\n",
    "import gluonnlp as nlp\n",
    "from utils import print_side_by_side\n",
    "\n",
    "# Switch on Numpy compatible shape handling (eg. for arrays with shape 0; will be default in MXNet 2)\n",
    "mx.set_np_shape(True)\n",
    "\n",
    "ctx = mx.context.gpu(0) if mx.context.num_gpus() else mx.context.cpu()\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Attention Mechanism"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\n",
    "In :numref:`chapter_seq2seq`, we encode the source sequence input information in the recurrent unit state and then pass it to the decoder to generate the target sequence. A token in the target sequence may closely relate to some tokens in the source sequence instead of the whole source sequence. For example, when translating \"Hello world.\" to \"Bonjour le monde.\", \"Bonjour\" maps to \"Hello\" and \"monde\" maps to \"world\". In the seq2seq model, the decoder may implicitly select the corresponding information from the state passed by the decoder. The attention mechanism, however, makes this selection explicit.\n",
    "\n",
    "Attention is a generalized pooling method with bias alignment over inputs. The core component in the attention mechanism is the attention layer, or called attention for simplicity. An input of the attention layer is called a query. For a query, the attention layer returns the output based on its memory, which is a set of key-value pairs. To be more specific, assume a query $\\mathbf{q}\\in\\mathbb R^{d_q}$, and the memory contains $n$ key-value pairs, $(\\mathbf{k}_1, \\mathbf{v}_1), \\ldots, (\\mathbf{k}_n, \\mathbf{v}_n)$, with $\\mathbf{k}_i\\in\\mathbb R^{d_k}$, $\\mathbf{v}_i\\in\\mathbb R^{d_v}$. The attention layer then returns an output $\\mathbf o\\in\\mathbb R^{d_v}$ with the same shape as a value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<center>\n",
    "<img src=\"../img/attention.svg\" width=\"33%\"/>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\n",
    "\n",
    "To compute the output, we first assume there is a score function $\\alpha$ which measures the similarity between the query and a key. Then we compute all $n$ scores $a_1, \\ldots, a_n$ by\n",
    "\n",
    "$$a_i = \\alpha(\\mathbf q, \\mathbf k_i).$$\n",
    "\n",
    "Next we use softmax to obtain the attention weights\n",
    "\n",
    "$$b_1, \\ldots, b_n = \\textrm{softmax}(a_1, \\ldots, a_n).$$\n",
    "\n",
    "The output is then a weighted sum of the values\n",
    "\n",
    "$$\\mathbf o = \\sum_{i=1}^n b_i \\mathbf v_i.$$\n",
    "\n",
    "Different choices of the score function lead to different attention layers. We will discuss two commonly used attention layers in the rest of this section. Before diving into the implementation, we first introduce a masked version of the softmax operator and explain a specialized dot operator `nd.batched_dot`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Enforcing Causality: Masked Softmax\n",
    "\n",
    "<center>\n",
    "<img src=\"../img/causal-attention.png\" width=\"40%\"/>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The masked softmax enables enforcing causality when computing attention weights.\n",
    "It takes the attention scores and a mask as input and filters out masked scores when computing the attention weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:07.112462Z",
     "start_time": "2019-08-06T01:19:07.072350Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[[0.488994   0.511006   0.        ]\n",
       "  [0.56008387 0.43991613 0.        ]\n",
       "  [0.42497858 0.57502145 0.        ]]\n",
       "\n",
       " [[0.35589045 0.36392704 0.28018245]\n",
       "  [0.29034293 0.25239876 0.45725834]\n",
       "  [0.21196374 0.5249825  0.26305377]]]\n",
       "<NDArray 2x3x3 @cpu(0)>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attention_scores = nd.random.uniform(shape=(2,3,3))\n",
    "mask = nd.ones(shape=(2,3,3))\n",
    "mask[0, :, 2:] = 0\n",
    "nlp.model.attention_cell._masked_softmax(nd, attention_scores, mask, attention_scores.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Now we create two batches, and each batch has one query and 10 key-value pairs. \n",
    "We specify through `mask` that for the first batch, we will only pay attention to the first key-value pair, while for the second batch, we will check the first 6 key-value pairs. Therefore, though both batches have the same query and key-value pairs, we obtain different outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Computing Attention Weights: Dot Product Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The dot product assumes the query has the same dimension as the keys, namely $\\mathbf q, \\mathbf k_i \\in\\mathbb R^d$ for all $i$. It computes the score by an inner product between the query and a key, often then divided by $\\sqrt{d}$ to make the scores less sensitive to the dimension $d$. In other words,\n",
    "\n",
    "$$\\alpha(\\mathbf q, \\mathbf k) = \\langle \\mathbf q, \\mathbf k \\rangle /\\sqrt{d}.$$\n",
    "\n",
    "Assume $\\mathbf Q\\in\\mathbb R^{m\\times d}$ contains $m$ queries and $\\mathbf K\\in\\mathbb R^{n\\times d}$ has all $n$ keys. We can compute all $mn$ scores by\n",
    "\n",
    "$$\\alpha(\\mathbf Q, \\mathbf K) = \\mathbf Q \\mathbf K^T /\\sqrt{d}.$$\n",
    "\n",
    "Now let's implement this layer that supports a batch of queries and key-value pairs. In addition, it supports randomly dropping some attention weights as a regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:13.111709Z",
     "start_time": "2019-08-06T01:19:13.106976Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class DotProductAttention(nn.Block): \n",
    "    def __init__(self, dropout, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, query, key, value, mask=None):\n",
    "        d = query.shape[-1]\n",
    "        scores = nd.batch_dot(query, key, transpose_b=True) / math.sqrt(d)\n",
    "        attention_weights = nlp.model.attention_cell._masked_softmax(nd, scores, mask, scores.dtype)\n",
    "        attention_weights = self.dropout(attention_weights)\n",
    "        return nd.batch_dot(attention_weights, value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:15.485150Z",
     "start_time": "2019-08-06T01:19:15.477547Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[[[0.5      ]\n",
      "  [0.7310586]\n",
      "  [0.880797 ]]\n",
      "\n",
      " [[1.       ]\n",
      "  [1.5752103]\n",
      "  [1.8509371]]]\n",
      "<NDArray 2x3x1 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "atten = DotProductAttention(dropout=0.5)\n",
    "atten.initialize()\n",
    "X = nd.array([[[0], [1] , [2]],[[0], [1] , [2]]])  # batch_size=2, number of queries / keys / values = 3, dim = 1\n",
    "print(atten(X, X, X, mask))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Multi-Head Attention\n",
    "\n",
    "<center>\n",
    "<img src=\"../img/multi-head-attention.svg\" width=\"50%\"/>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from utils import MultiHeadAttention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Transformer Architecture\n",
    "\n",
    "<center>\n",
    "<img src=\"../img/transformer.png\" width=\"33%\"/>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The Transformer model is also based on the encoder-decoder architecture. It,\n",
    "however, differs to the seq2seq model that the transformer replaces the\n",
    "recurrent layers in seq2seq with attention layers. To deal with sequential\n",
    "inputs, each item in the sequential is copied as the query, the key and the\n",
    "value as illustrated in :numref:`fig_self_attention`. It therefore outputs a same length\n",
    "sequential output. We call such an attention layer as a self-attention layer.\n",
    "\n",
    "\n",
    "<!-- Compared to a recurrent layer, output items of a self-attention layer can be computed in parallel and, therefore, it is easy to obtain a high-efficient implementation. -->\n",
    "\n",
    "The transformer architecture, with a comparison to the seq2seq model with\n",
    "attention, is shown in :numref:`fig_transformer`. These two models are similar to\n",
    "each other in overall: the source sequence embeddings are fed into $n$ repeated\n",
    "blocks. The outputs of the last block are then used as attention memory for the\n",
    "decoder.  The target sequence embeddings is similarly fed into $n$ repeated\n",
    "blocks in the decoder, and the final outputs are obtained by applying a dense\n",
    "layer with vocabulary size to the last block's outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "It can also be seen that the transformer differs to the seq2seq with attention model in three major places:\n",
    "\n",
    "1. A recurrent layer in seq2seq is replaced with a transformer block. This block contains a self-attention layer (multi-head attention) and a network with two dense layers (position-wise FFN) for the encoder. For the decoder, another multi-head attention layer is used to take the encoder state.\n",
    "1. The encoder state is passed to every transformer block in the decoder, instead of using as an additional input of the first recurrent layer in seq2seq.\n",
    "1. Since the self-attention layer does not distinguish the item order in a sequence, a positional encoding layer is used to add sequential information into each sequence item.\n",
    "\n",
    "In the rest of this section, we will explain every new layer introduced by the transformer, and construct a model to train on the machine translation dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Position-wise Feed-Forward Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The position-wise feed-forward network accepts a 3-dim input with shape (batch size, sequence length, feature size). It consists of two dense layers that applies to the last dimension, which means the same dense layers are used for each position item in the sequence, so called position-wise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:33.405267Z",
     "start_time": "2019-08-06T01:19:33.401301Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class PositionWiseFFN(nn.Block):\n",
    "    def __init__(self, units, hidden_size, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        # Dense layers is used for each position item in the sequence, so called position-wise. \n",
    "        self.ffn_1 = nn.Dense(hidden_size, flatten=False, activation='relu')\n",
    "        self.ffn_2 = nn.Dense(units, flatten=False)\n",
    "\n",
    "    def forward(self, X):\n",
    "        # shape of X: (batch size, sequence length, feature size)\n",
    "        return self.ffn_2(self.ffn_1(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Similar to the multi-head attention, the position-wise feed-forward network will only change the last dimension size of the input. In addition, if two items in the input sequence are identical, the according outputs will be identical as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:34.733340Z",
     "start_time": "2019-08-06T01:19:34.725505Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "6"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[[-0.00432132  0.0002378  -0.00044518 -0.0015296 ]\n",
       "  [-0.00432132  0.0002378  -0.00044518 -0.0015296 ]]\n",
       "\n",
       " [[-0.00432132  0.0002378  -0.00044518 -0.0015296 ]\n",
       "  [-0.00432132  0.0002378  -0.00044518 -0.0015296 ]]]\n",
       "<NDArray 2x2x4 @cpu(0)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ffn = PositionWiseFFN(units=4, hidden_size=8)\n",
    "ffn.initialize()\n",
    "ffn(nd.ones((2, 2, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Add and Norm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The input and the output of a multi-head attention layer or a position-wise feed-forward network are combined by a block that contains a residual structure and a layer normalization layer.\n",
    "\n",
    "Layer normalization is similar batch normalization, but the mean and variances are calculated along the last dimension, e.g `X.mean(axis=-1)` instead of the first batch dimension, e.g. `X.mean(axis=0)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:36.701220Z",
     "start_time": "2019-08-06T01:19:36.689630Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "7"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\t\n",
      "[[-0.99998  0.99998]\t\t[[-0.99998 -0.99998]\n",
      " [-0.99998  0.99998]]\t\t [ 0.99998  0.99998]]\n",
      "<NDArray 2x2 @cpu(0)>\t\t<NDArray 2x2 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "layer = nn.LayerNorm()  # Normalize along channel-dimension\n",
    "layer.initialize()\n",
    "batch = nn.BatchNorm()  # Normalize along batch-dimension\n",
    "batch.initialize()\n",
    "X = nd.array([[1,2],[2,3]])\n",
    "with mx.autograd.record():  # compute mean and variance from X in the training mode.\n",
    "    print_side_by_side(layer(X), batch(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The connection block accepts two inputs $X$ and $Y$, the input and output of an other block. Within this connection block, we apply dropout on $Y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:37.869396Z",
     "start_time": "2019-08-06T01:19:37.865671Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "8"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3, 4)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class AddNorm(nn.Block):\n",
    "    def __init__(self, dropout, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.norm = nn.LayerNorm()\n",
    "\n",
    "    def forward(self, X, Y):\n",
    "        return self.norm(self.dropout(Y) + X)\n",
    "    \n",
    "add_norm = AddNorm(0.5)\n",
    "add_norm.initialize()\n",
    "add_norm(nd.ones((2,3,4)), nd.ones((2,3,4))).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Due to the residual connection, $X$ and $Y$ should have the same shape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Positional Encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Unlike the recurrent layer, both the multi-head attention layer and the position-wise feed-forward network compute the output of each item in the sequence independently. This property allows us to parallel the computation but is inefficient to model the sequence information. The transformer model therefore adds positional information into the input sequence.\n",
    "\n",
    "Assume $X\\in\\mathbb R^{l\\times d}$ is the embedding of an example, where $l$ is the sequence length and $d$ is the embedding size. This layer will create a positional encoding $P\\in\\mathbb R^{l\\times d}$ and output $P+X$, with $P$ defined as following:\n",
    "\n",
    "$$P_{i,2j} = \\sin(i/10000^{2j/d}),\\quad P_{i,2j+1} = \\cos(i/10000^{2j/d}),$$\n",
    "\n",
    "for $i=0,\\ldots,l-1$ and $j=0,\\ldots,\\lfloor(d-1)/2\\rfloor$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:44.447059Z",
     "start_time": "2019-08-06T01:19:44.440950Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "10"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from utils import PositionalEncoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Now we visualize the position values for 4 dimensions. As can be seen, the 4th dimension has the same frequency as the 5th but with different offset. The 5th and 6th dimension have a lower frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:46.047460Z",
     "start_time": "2019-08-06T01:19:45.868882Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "11"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOydd3hUVf643zOTSU8mvTcSQg0QIHTEihXL6s+CBQUVV+yu61oW/drW1V3rqggiYlnXgq6dRVCw0FsSQiCUENJ7r5PMnN8fdyYESJkkM3cCmfd55kly77n33MmduZ/z6UJKiRMnTpw4GbxoHH0BTpw4ceLEsTgFgRMnTpwMcpyCwIkTJ04GOU5B4MSJEyeDHKcgcOLEiZNBjoujL6AvBAUFybi4OEdfhhMnTpycUuzcubNcShl84vZTUhDExcWxY8cOR1+GEydOnJxSCCGOdrbdaRpy4sSJk0GOUxA4ceLEySDHKQicOHHiZJDjFAROnDhxMshxCgInTpw4GeTYRBAIIVYIIUqFEBld7BdCiNeFEIeEEOlCiAkd9t0shDhoft1si+tx4sSJEyfWYyuNYCVwYTf7LwISza+FwBIAIUQA8CQwBZgMPCmE8LfRNTlx4sSJEyuwSR6BlPJXIURcN0MuBz6QSs3rLUIIPyFEOHAWsFZKWQkghFiLIlD+Y4vrOpGaNx6ntbQCzdApaP2DcQkJwX3EcLR6vT2ma6eguokNWaUMDfZm8pAAhBB2na+vSCkpbCjkYNVBKpsraWhtoKG1AZ1Gh7+7P35ufkT5RBGvj8dFMwBTUKSE5mqoKYCafAgYAsHDVZm6prGV0rpmGgxGWlqNjI3yw8NVq8rcg5LcrdBQCiGjwD8ONOr8r5tbjXyfXkSgtyspcQF4u9n/e2Csb6A1LxdDbh6tebn4z52LxsvLpnOo9W2OBPI6/J1v3tbV9pMQQixE0SaIiYnp00XUrllL/cE64JfjtusiI/EYNw7vc8/B+8wz0Xp79+n8HZFS8tHWXFbtzCctr7p9e0qsP4vOTuDs4SEDQiDk1OSwPm89v+b/yr7KfTS0NvR4jIeLByMCRjAhZAKz42YzKmCU499LfRl8Mhfytx/bJrRw5sNwxp9Aq7Pb1F/tLuDhL9IxtJnat8UHe/HWDRMYEeZrt3kHJfVlsOZR2PP5sW0u7pB0FVz6mt3us8kk+Ta9kBdW76ewphkAjYCkSD1/vmA4ZySelKzb97laWmjcsoWGTZtp2LyZlgMHjtvvNXMm7iNG2Gw+AGGrxjRmjeA7KWVSJ/u+A/4upfzd/PdPwF9QNAJ3KeWz5u2LgSYp5T+7myslJUX2KbNYSkyHfsW0aQXGjDW0NrrRnLCQ5rxyGrdtx1hRgdDp8DrjDAJuvAHPadP6/IB7bd1BXll3gFHhvswZF87skaFszq5g6S/ZFFQ38Yfxkbx8zTiHPEAbWhv4+tDXfJb1GYdrDgMw3H8440PGMyxgGIl+iYR6huKp88RT50mrsZWalhoqWyo5UnOEveV7ySjPYE/5HozSSJR3FBfHX8zcEXMJ8ghS/f1QdRQ+vAJqi2DWnyBwKPiEw/blygMjYgJcuQyCEm06rZSSV9Ye4PWfDzE1PoAbpsTi5aalvsXIM99lUtfcyjOXJ3F1SrRN5x207FkFPzwELfWKcB92PpTuh7ytsOt9GH4xXL0SXNxsOm1Vg4EF729nd241SZG+PHLhSAC2Hangu/QiCqqbeG/+JKYn9O+z35x1gOpVq6j55htMNTUIV1c8UybiOWkSrkOG4BoTgy4mpl8LVSHETillyknbVRIES4ENUsr/mP/OQhECZwFnSSnv6GxcV/RZEHSkOheWn6esJm5fj3T3oyk1lbq166j59luMFRW4DRtGwPz56C+7FKG1XvX8bHseD3+RzlUTovjn1WOPe9i3Gk28uu4Ab64/zOI5o7h15pD+vY9eUNFUwbsZ7/Lfg/+lvrWeMUFjmBM/h7OjzybcO7zX56turmZ93nrW5KxhU+EmXDQuXJZwGTePvpkhepXeV0kmfPgHaGuC6z+DmKnH78/4Er5/EDQusGgreAXaZNo2o4kHPkvj27RCrkmJ4tkrxuDqcszlVlbXwn2f7GbT4QoenD2Me8+1rRAadORugfcugsgUuPyNk01+295RhMTQ8+Daj0DnYZNppZTc9v4OfjtYznN/SOKqCVFoNMe+z1UNBq5btoW8qkY+vHUyE2MDej1H4+7dlL3+Oo2btyB0Onxmz0b/hyvwnDQJjbu7Td6Hha4EAVJKm7yAOCCji32XAKsBAUwFtpm3BwBHAH/z6wgQ0NNcEydOlDYhb7uUTwdL+e6FUra2tG82NjfLqlVfyMOXXiYzh4+Qhy+9TNZv2mTVKX/eXyLjH/1e3rh8izS0GTsdYzKZ5MIPtsv4R7+Xmw+X2+StdEdja6NclrZMTvn3FJn8frJ8+JeHZVppmk3nOFpzVD6z+Rk58cOJcuz7Y+Wzm5+V1c3VNp3jJJpqpPzHMOVVnNH1uKI9Uj4VKOWnN0lpMtlk6uW/ZcvYv3wn//XTAWnq4pxtRpO85+NdMv7R72VGgZ3/F6czTTVSvjJGylfHStlc2/W4HSulfFIv5cfX2fw+r/g9u8sxJbVN8ux/rJdJT/xP7sm3/j43ZWXJ3IV3yMzhI2TW9BmyfPm7srWy0haX3SXADtnZM7qzjb19oTh3i4BWFDv/rcAfgT+a9wvgTeAwsAdI6XDsAuCQ+TXfmvlsJgiklDL9cymf9JXy63tO2mUymWTN6tXy4DnnyszhI2TunYukoaioy1MVVDXKkYtXy4tf+1XWNbd2O21tk0Ge/c/1cuIzP8qi6qZ+v42u+CXvF3nuZ+fKpJVJ8t6f7pVHqo/YbS4ppSxvLJfPbH5Gjn1/rJzxnxny0/2fSqOpc4HYb/73mPLFz9vR89hf/6nc5/TP+z1tUXWTHLV4tZz37tYuhYCFqoYWOfGZtfKS13+VrV0sDJz0wH/vlPL//KQ8uqXnsb+/ptzn/av7PW1aXpUc+tj38rb3t/d4nwuqGuWU59bJi1/7VRqN3Y81NjfLkpdfkZmjk+T+yVNk2dJl0lhf3+/rtQa7CgK1XzYVBFJKueavyocnf2enu43NzbJs2TK5L3m83J8ySVZ/802nH4yHP0+TiY/9IPMqG6ya9kBxrRy1eLW8ecXWfl1+Z9S21MrFvy+WSSuT5BVfXSG3FW2z+RzdkVWZJRf8b4FMWpkkb19zuyyuL7btBKX7pXwqQMqv7rJufFurlMvOlvLvsVLWdi3MrWHRRzvlsMd/kDnl1n15f0gvlLF/+U4u2XCoX/MOSjL+q3w3f3rWuvFtBilfnyjla+OP0/J7S11zq5z14s9y2t/WyaoG687z1e58GfuX7+QXO/O6HNOwY4c8dMGFMnP4CFnwyKN21wBOpCtB4MwsBiWyxMMf1v+t090aNzeCbr+d+K/+i1tCAoV/fpiCBx7EWFfXPuZwWT2f78zjxqmxRPl7WjVtYqgPd5+TyIasMlI7RBb1l9TSVK765iq+Pvw1tybdyqdzPmVS2CSbnd8ahvkPY/n5y3li2hOklqVy5TdX8mPOj7Y5uZSw+i+g84Jzn7TuGK0LXPE2tDbBdw/2eepfDpTx/Z4i7j57KLGB1oXwXTQmnAtHh/HK2gMcKe85KsuJmZZ6+O4Bxdl/5sPWHaPVwQV/g8rDsP2dPk/9weYcjlY08sq1yfh5ulp1zKVjIxgTqeefa7JobjUet0+aTJQve4ejN81DtrURs+JdIp7/Gy7+AyNtyikIANx8YMZ9cGgt5G3rcphrbCyxH31I8AMPULduHTlXX0NLdjYAL/94AA+dlrvOTujV1DdNi8XPU8e/fjrYr7cAinb3WdZnzF8zH63Q8sFFH3D/xPtx1Vr3QbY1QgiuHnY1n835jBifGP70y594acdLGE3Gng/ujv3fQ/Z6OPsx8O5F2F7wMJj1EGR9D0VpvZ62udXIE19nEB/sxcIz43t17NOXj8bNRcNz32f2et5By64PoKkSLnqxd2Ghw85XnMYbXoCG8l5P29xqZMXvRzgjMYgp8dYHF2g0gscuHklhTTPvbcxp395WVUXenXdS9vLL+F54AUO++gqv6dN7fV32xCkILExeCJ5BXWoFFoSLC0F3LCT2vRUYa2vJueZaMlZ9z/d7irj1jHgCvXsXuubt5sJtM4fw0/5SMgpq+nz5BqOBpzY/xTNbnmFK+BQ+mfMJ44LH9fl8tiROH8cHF3/AdcOvY+Xeldz9893UGep6PrAzjG3w4+MQPBIm3dr74yfdDq7esOmNXh/6dWoBRysa+b9LR+Pm0rsEphBfd26Zodxnp1ZgBcZW2PIWxEyH6D5os+c/B4Z6WP9crw/9fEce5fUGFp01tNfHTksI5NwRIby1/hCVDQZaso+Qc/U1NG7aTOgTi4l46SW03rZNBrMFTkFgwdULZj6grDSPbupxuOekSQxZ9TmuMTFo/vpnrs3bxG1n9C1kct70OHzdXXi9j1pBvaGeResW8cXBL7h9zO28ec6b6N3smy3dW3QaHY9PfZzFUxezpXALN/xwAwX1Bb0/0cE1UJWjaAN9SR7y8IPxN8HeL5UMZCuRUvL+pqOMCPPhjMS+xYvfODUGF43g/U05fTp+ULH3v1CTp2jqfSFkBEy8BXZ9qCShWUmb0cTSX7OZEOPH1Pjeh4ICPHLRCBoMbXy24luOzp2LqbGR2I8+JOD66x2feNkFTkHQkZQF4B3ao1ZgQRcRQe2Lb7IpfDS37PySlmVvWyKheoWvu45bZ8bzY2YJmYW1vTq2oqmCBWsWsKNkB3+b+TfunXAvWpXS7fvCNcOvYdn5yyhvKmfe6nlkV2f37gTb3wWfCCV5qK9M/SNIE2x92+pDdh6tIrOolpunx/X5yxzi486l4yL4bEceNU2tfTrHoEBK2Pg6BA2HxPP7fp4pd4CpFVI/svqQb9MLya9qYtFZQ/t8nxNDfbhHm8u0Jf+Hxt+fuE8/wWPcwNDOu8IpCDri6gnT74Wc36Bkr1WHfLqnjJen34znFX+g/K23KHnmWaTJ1POBJ3DLjDh83Fx4c8Mhq48prC9k3up5HKk5wuvnvM6lCZf2el5HMClsEu9d8B5Gk5Fb/ncLmRVW2s0rs+HwTzDxZsX521f842DkZbDzfWixzkT1/uaj+Lq7cHlyRN/nBRbMGEKjwcjnO/J6HjxYOfwzlOyBGfeCph+PqODhEDsDdq4EK76TJpPkrfWHGR7qwzkjQvo8bc0333DBl29w0C+KfY+/hGv0wM8sdwqCExk3FzQ6SP24x6FNBiPfphZy4dgoYp5/joAFC6j6+GOKnnii18JA76HjmknR/Li3mOpGQ4/jixuKWbBmAVUtVbxz/jvMiprVq/kczfCA4bx/0fu4u7izYM0C0sqscN7uWKHUD5owr/8XMP0eaKlRTAc9UFrbzOo9RVyTEo2na//KcyVF6pkcF8DKTTkYTbbJ6j/t2PgaeIfBmKv7f66J8xVT4pENPQ795WAZB0vrufOshOOyh3tD9X+/ovAvj+A1aRJL5tzPv/f13e+nJk5BcCJegTDsAkj/THFMdsOavcXUtbRxdUo0QghCH/4zgXf+kZpVX1Dy7HO9NhP9YXwkrUbJd+lF3Y4rbSzl1jW3UtNSwzuz3yE5JLlX8wwUYn1j+eCiDwhwD+DOdXeyv3J/14Nbm2H3v2HEJeDbv1U5AFEpEDMNti7pcbX48bZcjFJy49TY/s8LLJgZR35VE2szS2xyvtOK8kNw5BeYstA2NYNGXQYeAbDjvR6H/ndXAX6eOi4e0/tyKwDVX/6Xoscew2vaVKLfXsKV0xLYkl15SgQHOAVBZyRfr5S4PfxTt8M+25FHTIAnU4YccyoF33svAfPnU/Xxx5T+45+9EgajI3wZHurDl7vyuxxT3lTOrWtupbypnLdnv83ooNFWn38gEuYVxvLzl+Ol82Lhjwu79hlkfqWEEvYlUqgrJt2m1JzK29LlEEObiX9vzeWsYcHEBdkm2mP2qDAi/Tz4cEuOTc53WpGxChCKZm4LXNxg/A2Q9QPUdS14G1raWJtZwiVjwo+rGWUttWt+pOjxx/GaNo2ot95C4+HB1ROj0GoEn24f+GZApyDojKGzwTMQUv/d5ZC8ykY2Ha7g/008vgiVEIKQh/+M//XXU7liBRVLl1o9rRCCP0yIZFduNTmdrCIaWhtYtG4RJY0lLDlvyYAJD+0vEd4RvDP7HTRCw20/3kZ+XSeCcPu7SlXRIWfabuJhF4KLB2R80eWQn/eXUlbXwrxpcTabVqsRXDUhks2HKyira7HZeU95pFQqjMbOsI3WZ2HifDC1we6uzYA/ZhbT1GrkivGdVsHvloYtWyl86CE8kpOJevON9kJxIb7unDMihFU782k19t5vqCZOQdAZLq4w5hrIWg2NlZ0OWbUzHyHgqolRJ+0TQhD618fxvexSyl59jZpvvrF66suTIxACvtx9fGhjq6mVP234EweqDvDSmS8xIXRCF2c4NYnTx/HO+e/QYmxh0U+LqGnpYFutOAz525RwQFuG37l5w/ALIfPrLs2AqzOK8PfU9TlktCsuHhuOScL/9hbb9LynNMXpUHEQxlxl2/MGJsCQWUqp6i409K92FxLp58HEmN5l+jZnZpJ/1124xsUSvUTRBDoyd3I05fUt/LSvtM+XrwZOQdAVyXPBaFDizU/AZJKs2pnPzKFBRPp1Xu5WaDREPPssnlOmUPj4X2nY0rX5oSPheg+mJwTy1e6CdrOSlJKnNz/NxsKNPDHtCc6IOqPv72sAk+ifyKtnv0peXR4PbHiAVqM5xHKfWZCOusL2k46+EhrKlEixE2huNfLTvlIuGB2Gi9a2X5XhoT4kBHvxfXqhTc97SrNnlVIu3B73edz1ihmwcNdJu8rrW/j9UDmXJ0f0ykncWlhI7sI70Pj6Ev3OO2j9/E4aMysxmDBfd1btHNjmIacg6IqwsRCa1Gn00NYjlRRUN/XYcES4uhL1r9dxi4sl/+57aD6h01BXXDk+itzKRnYerQJgafpSvjr0FXeOu5MrE6/s/Xs5hZgUNolnZjzD9uLtPLnpSUUY7vsWIsaDnx3C8BJng6tPp+ah3w+WU9/SxkV9dB52hxCCS8ZGsPVIJaV1zTY//ymHyaT0jkg4Bzz7lsjVLcMuUCLO9n130q7v0goxmmSvzEKmhgbyFt2FbG4m5p1l6MLCOh3notVw8Zhwfj1YTkNL98EnjsQpCLpCmB1WBTuV+PUO/JhZjKuLhvNG9hxrrPX1JXrpUoSHO/l33Y2xuufichcmheGh0/LFrgJ+zv2ZN1Pf5NL4S7lz3J19fjunEnPi53BX8l18m/0ty7e/pNyDkXbKkdB5KJFI+76FtuPDdn/IKELvoWN6gm2a2ZzInLHhSAn/y3Cah8jbCrX5tgkZ7QzPAIibCftPFgRfpRYyMtyXYaE+Vp1KmkwUPvIoLQcOEPnKy7gN7b4UxfmjQzG0mfj1gPUZzmrjFATdMeIS5WfW/9o3SSlZt6+EmUODrI4p10VEEPX667QWF1Pwp4eQxu6Lrnm5uTB7VCirs3bz6G+PkhSYxJPTnxyw6en24I6xd3DRkIv41773+d3DHUZebr/Jkq5Umt5nr2/fZGgzsTazhNmjQtHZ2CxkYVioD4kh3nzfQ7jwoCBjleK470/GeE+MvBTKD0BZVvumnPIGUvOquaIXiYLlb7xB3dq1hP7lYbzP6NlMmxLrj7+njh8HcLiwUxB0R8AQCB4BB1a3bzpQUk9eZROzR4X26lSe48cTtvivNGzcSNmrr/Y4fsYwT1qDVuAi3Hnl7Fdw09q2D+tARwjBU9OfYpjU8XBoCHmudnz/8WeDu99x5qGNh8upa27j4jGdq/y24pKx4WzLqaS0dhCbh4xtsPcrxXHv1vd+vD1iWdjt+7Z902qzNnbpOOsEQd26dZS/tQT9/7sK/3nWJTa6aDWcOzKUn/aVDNjoIZsIAiHEhUKILCHEISHEI53sf0UIkWp+HRBCVHfYZ+ywz/rwGrUYdqFShK5ZiWJZt0+R6uf2IQXd/5pr8Lv2WireWU7t6tVdjjNJEz9XvIrQVTPd5wHCvOz7MBqoeDTX80pBLkLryn0b7qOxtdE+E7m4KolH+79XEteA1XuK8HFzYcZQ20YLncglYxTz0OrBbB7K2wKN5TD6D/adxzdC6XncwTy0PquUUeG+RHQR9NERQ24uhY88ivuYMYQ98USvNPTZo0KpbW5j25HOoxAdTb8FgRBCi9KG8iJgFDBXCDGq4xgp5QNSymQpZTLwL6BjKE6TZZ+U8rL+Xo/NGX6REoN8aB0AP2aWMC7ajxDfvjWVDnv8MTySkyl6/K8YcnI6HfP+3vfZVPQ74W1Xs+ewHRxnpwpZ3xPd2so/xj/I4erDPLvlWfvNNeJSpWxx7iZajSZ+zCzh3JEhvS433VsSQ30YHurD93sGsXno4FolWij+bPvPNXIOFO6GaqXw386jVVbVFTI1N5N/3/2g1RL16itoXHvX42NWYjDuOg0/DtBwYVtoBJOBQ1LKbCmlAfgE6M6gOxelx/GpQdQkJbnswBpKa5tJy6tmthVO4q4Qrq5EvvwS6HTkP/ggJsPxDsrU0lRe2/Uas2Nnc9XQa9lfXEdBdVN/38WpSeY34B/H9NHXc8fYO/g2+1u+PvS1feaKmwlaNzj0E1uzK6lubLVLtFBnnDcqhJ1Hq6htHqQVSQ+tU8p9uPvaf64R5qCD/d/z+8FyjCbJ2SN6bm5U/OyztOzbR+SLL6CL7H3SmYerljMSg/kxs6RPFYrtjS0EQSTQMUg237ztJIQQscAQ4OcOm92FEDuEEFuEEF0GEAshFprH7SgrU9H7rtEqpXAP/sjPmUrM93m99A+ciC4igojn/0ZL5j5KX/xH+/bq5moe+uUhwr3CeWr6U+3zrN8/sJNR7EJzjVJzZuRlIAR3jL2DlNAUntv6HEdqjth+PldPiJsBB9eyIasUVxcNsxJ70f2sH5w5LASjSbLpUO+7aZ3y1BZCSYbSUUwNgoYqfr/93/Hz/lL8PHUkR3efRFbz7XfUrPqCwDvuwPvMvme2nz8qlKKaZjIKeldqXg3UdhZfB6ySUnYMm4mVUqYA1wOvCiE67fUopVwmpUyRUqYEB6vzBW1n2IXQVEVO6gai/D0YbmWYWXf4nHMOATffTNVHH1G7di1SShZvXExlcyX/POuf+Lj6kBDsTUyAJz8PRkFw5FfFJDfsAgC0Gi1/P+PvuGvdeeiXh2gx2qE0w9DzoDyLrKxMJscF4OGqTl+H8TF+eLu58MuBQSgIzCZXEmerN+eIOcijG9mddZhZicFou0kiM+QXUPzUU3hMmEDwPXf3a9pzR4aiEbA2c+CZh2whCAqAjpk+UeZtnXEdJ5iFpJQF5p/ZwAZgvA2uybYknIPU6Agq/JnZo0JtFsYZ8qcHcU9Kovivi/lqywo25G/g/gn3MzpQKSQnhOCcESFsPFROk6GffX5PNQ6vV5rTR01u3xTqFcqzM5/lQNUBXt7xsu3nNK9Koyo32bykRHfotBqmJwTy64GyAWk2sCsH1yqNhkJG9TzWVgw9DyFNDGtK69YsJNvaKHz4YQAiXnwR4dK/EuQBXq6kxAawdgCWm7CFINgOJAohhgghXFEe9idF/wghRgD+wOYO2/yFEG7m34OAGcDA6+7t7ktl8CTOYifnjeyfWagjwtWViH+8iLG5GcOzrzAlbDI3jrrxuDHnjAihpc3E5uxBtlrMXq/Y7V2Od8rNiprFDSNv4OP9H7OpsOeWor0iaBgNHuGcpUnjDJXMQhZmDQumoLqJ7FOgZLHNMLZC9gZIPM+2NaR6IioFg9aTmdo93Zr/ypcupWnXLsKefALXqN77BTpj1rAg9hXVUlE/sIoN9lsQSCnbgLuBNcA+4DMp5V4hxNNCiI5RQNcBn8jjlzwjgR1CiDRgPfB3KeXAEwTANt1khmoKmeRr20YT2thoVl8SwtjDRv5aOAmNOP6WTIkPwNNVO+CLVtmUqqNKNndC51Ek90+4nyH6ISzeuPj44nT9RQhS3VKYod3LiGB18zbOHKY8kH7JGrjZpzYnbxu01CrVftVEqyNVm8Q5rvsI9O78PjelpVH+1hJ8L70U/aW2y2q3hCNvOlxhs3PaApv4CKSUP0gph0kpE6SUz5m3PSGl/KbDmP+TUj5ywnGbpJRjpJTjzD/ftcX12IPPq4YB4Jp3cnGy/vBO+ju8N6yQpokjaH5tKS3ZxztC3Vy0zBwaxPr9pYPHbGDJ8O0inNDdxZ3nZz5PZVMlz2973mbTmkySL2tH4k0TmoLtNjuvNUQHeBIf5MWvBweRIDhkCRs9S9Vpy+tbWN04gghjobLoOAFTSwuFjz6GS0gIYU8stuncYyL1+Li5sOnwwNLwnZnFVlBa18zPFXoaXIMUJ6aN2F+5n2Xpy7gkYQ5jXl6Kxs2NwkceOakExRmJQRTWNHO0wk4JVQONw+vBJ1zpOdsFo4NGs3DcQr7P/p41OWtsMm1mUS1rGodhEi6K7VplZg0LZkt2Bc2tg8QfdHAdRE9VJ2y0A78eKOM3Y5LyR/aGk/aXvf46huxswp99Bq1P/wNDOuKi1TA1IZDfB1iEmFMQWMHmwxWAwBA9E4781mVN897Qampl8cbF+Ln78ejkR9GFhhC6eDHN6elUvv/BcWOnmYuebc4eWOqkXTAZlbDR+LN7tBvfNuY2kgKT+NvWv1HVXNXvqX87WE49nrRFToZD3XenswdnDgumudXEjpz+v5cBT22R0qA+UaWw0Q5sOlxBhXsc0if8uPpSAI27dlO54j38rr0W7xkz7DL/zKFB5FU2kTuAFnZOQWAFmw5V4Ovugn7UuUoLyw5Fq/rKij0r2F+5n79O/St6Nz0AvpdcjPc551D22mvHZR0nBHsT7OPGlsEgCIrSoKmqS/9AR3QaHU/PeJpaQy1/3/b3fk/928EyRoT54Dp8tvKQqlM3zG9KfACuWs3gMA9Z+j8knKP61FuyK5gSH4SIPwuyf2nvWW1qbqboscfQhYcT8uc/223+GUOVhd3GAWQecgoCK9iUXdIvnNcAACAASURBVM7U+EA08bOUDf00Dx2sOsjb6W9zYdyFnBtzbvt2IQRhTz6JcHWl6K+LkeYPqBCCqfGBbD5ccfr7CQ6bcw3jz7JqeKJ/IgvHLOSHIz/wS94vfZ620dDGjpwqZg0Lhnhz0lDO730+X1/wdHVh0hD/weEwzvkN3PVKzw8VyatsJL+qianxAcpnrKlSEfooVUUNOTmE/+05tN626U/dGQnB3oT6urFxAJmHnIKgB/IqG8mrbFK8/f5x4BejmC76iNFk5ImNT+Dr6sujUx49ab8uNITQRx6hcccOqj75pH37tPhASutaTv/wwuwNysPB2/oyHreNuY1E/0Se3vI0dYa6Pk27NbsSg9Gk5A+EjVOa1Rzd2Kdz9YfpCUFkldRR2WDoefCpTM5GiJmuZO6ryFZz0bepCYHHFhuH19OcmUnFeyvxu/r/4TV1ql2vQQjBjIQgNh2uwGQaGAs7pyDoAYt3v705yZBZykrR1Ldysp9kfUJGRQZ/mfQXAtw7Lyinv/IPeM2YQdlLL9NaolQ7bfcTDLCwM5tiaIDcLb2OItFpdTw9/WnKm8p5eWffEs02Z1fgqtUwKS4AtC4QM0V5WKnMlCHKZ2J7zsCsUmkTaoug8rBS0kNltmRXEODlyrAQH/AJg+CRyEPrKVr8BFp/f0IeekiV65gxNIjKBgP7i/u2cLE1TkHQAxsPVRDs48bQEHOd9CFnKk1MzOpkbyhuKOZfu//FjIgZXDTkoi7HCSEI+78nkW1tlDz3NwDiAj0J83U/vR3GeVvB1NqnKpRJQUncOPJGVh1YRWppaq+P33qkknHRetx15hVq7Awoz4J6dc00Y6L0uLloBmy5Yptg0bTiZqo+9ZbsCqYMCTjWmzj+LCrX7qZ5717CHnsUrV6vynVY8gkGinnIKQi6QUrJpsMVTE8IPFZWIs7ckagPfoIXtr1Am6mNx6c+3mOZCtfoaIIWLaLuxx+pW78eIQTTEgLZmn0a+wmObgahgejJPY/thLuS7yLUM5SntzxNq8n6Sp6Nhjb2FtQweUgHDc3ykMq1cfZyD7i5aJkQ4396C4Kc38HNV+kLriLH/APHWo8aPJMoS/XAe/JYfC7qenFma8L07iQEew2YMFKnIOiGg6X1lNe3MCOhQ90Z33AITOy1IFifu551uev447g/Eu1jXRP2wPm34JY4lOJnnsHU2Mi0+EDK6w0cLK3v1dynDLmbFf9AH+PKPXWePDrlUQ5WHeSjzI+sPm53bjVtJqmYhSyEJ4PO0yHmoclDAthbWEPd6VqWOud3iJnqOP9AB0FQ8u8NAIT9YYTqrWCnJwSxI6eStgHQtcwpCLrBYo+fdmLz8iGzlK5lRuu+qI2tjTy/7XmG+g3l5tE3Wz2/cHUl7KmnaCssouyNN09vP0GbAfJ3QOz0fp3m3JhzOSv6LJakLaGwvtCqY7YeqUQjYGJsh3LELq5KLwoHOIynDAnAJGHH0dMwn6CuBCoOOswsFODlSqLZzFv383rqf91I8BQPdA0Zql9PSpw/DQbjgPATOAVBN+w4WkW43p0o/xPa2A2ZpXSzKrTOFr18z3KKGopYPHUxOo2uV9fgOWECflf/Pyrff5+QigIi/TxOT0FQlAZtTUqDkn7y6GQlGsva8hPbj1QyKsIXH/cT7k3cTCjZC43qmmnGx/jjohGnp3nIIlhjHesfMDU1UfLcc7gmJBBw6Syl7pFJ3Yxuiwa6YwAEBjgFQTfsyKlkYqz/ySqj5WGVt6XHcxytPcrKvSu5LOEyJoRO6NN1BD/4IBpvb4qffY5p8QFsPTJwws5shsUW30+NACDCO4I7xt7BhrwN/JbffW0oQ5uJ3XlVx5uFLMTOAKQSyaQiHq5axkbpT09BkPM7uHpD+DhVpz3RP1C+bBmtBQVK7+H4GUrxu1J1611G+HkQ6efB9gGg+TkFQRcUVDdRVNNMSmwn3Yt8QpWcgh4eEFJKnt/2PG5aNx6Y+ECfr8XF35+Q+++jcetWZpdmUNXYevrlExzdDAEJvcof6I6bRt1EnG8cL2x/AYOx65j8PQU1NLea2sM2jyNyotK+0gHmoclDAknPrz79+lAc3aj4B7T9q+3fWzr6Bww5OVQufxffOXPwmjJZuR5QPoMqkxLnz46cSocHgDgFQRdY1LWUzlaKoBTLytvWbd2hn/N+ZmPBRhYlLyLIo3+NTvyuuQa3USOJ+ewd3Nta2DUAVhE2w2RSHMU2MAtZcNW68sjkRzhae5QPMj/octz27u6zzl3xE6icYQyKn6DVKNmddxrd5/oyKNtv1rTUZduRCvw8dSSGeFPy/N8Rrq6EPGwuI+EXA76RymdQZVLiAiipbSG/yrF9yZ2CoAt2Hq3C01XLiLAuqg/GTFHqDlV13j+3ua2ZF7e9yFC/ocwdMbff1yO0WsL+uhhKS7klez07jp5GZoPyLCU3I9Z2ggBgRuQMzo4+m2Xpyyhu6Lxu0LYjlSQEexHURV164mZAcbrSQ1lFJsb5IwSnl3nI8qB1gCDYebSKiTH+NP7+G/W//ELQokXoQszapxCKVpC72SYFJXuDxeLg6O+zUxB0wY6cKsbH+OGi7eJfFD1F+Zm7tdPd7+99n8KGQh6b8hguGtuowZ4TxqO/4grm7F/P0fT+F74bMBw1+wdsqBFYeHjSwxhNxk5bW5pMkh05lcfnD5xI7HSQJshTtz+Br7uOUeG+bM0+jQRB/jbQukJEsqrTVjcaOFzWwMRIRRtwjY0l4KbjOwESMw3qiqD65P4E9mRYqA8+7i5sd3DFWZsIAiHEhUKILCHEISHEI53sv0UIUSaESDW/buuw72YhxEHzy/rYSjtS19zK/uJaJsZ284AIHglueiUb9gRKGkp4N+NdZsfOZlLYJJteW/CDDyB1Os7/7XOqTpd6NLmbwTsUAuJtfuoonyjmJ81ndc7qkzKOs0rqqG1u69xRbCFyopLklq+uIAAln2B3XhWtAyDO3CbkbVPyM1zU7f62O7cagKlpP2M4coSQRx9BuB7fArV9EaJyYIBWI5gY6+/wyKF+CwIhhBZ4E7gIGAXMFUJ01on6Uyllsvm13HxsAPAkMAWYDDwphOjEO6suu3OrMUmYFNfNpWg0ED2pU0Hw2q7XaDO19ctB3BW6kBDarpvH9KIM9n6vfs18u3DU7B+wU0LPgqQFhHiE8MK2FzDJYw9Vi9mlW43AzUdprJ6/zS7X1h0TY/1pbjWxv8jxceb9ps2ghFv3MWu8P+zKrSLAUI/nJyvxmnUGPmeddfKgEPPC7qi6meSghJEeKKmnutFxCztbaASTgUNSymwppQH4BLjcymMvANZKKSullFXAWuBCG1xTv9hxtAqNUOK5uyV6CpTug6bq9k3pZel8m/0t80bNszqDuLcMu/sOSjz90b39+kndzE45qvOgNt8mYaNd4anz5L6J95FRkcH32d+3b9+VW0WYrztR/p7dnyBqEuTv7HOhwb5i+fztyj0NHMbF6WBsUf6XKrPzaBX35PyEbG4m9JGTK/4CSpZzzBTVNQI45idw5H22hSCIBPI6/J1v3nYiVwkh0oUQq4QQliektccihFgohNghhNhRVmbfQmA7j1YyIswXb7cebPvRUwCpZMSihIu+sP0FgjyCuH3s7Xa7Pi8fT9bNugbfwhyqv/jCbvOogkWjsvhc7MSc+DmMDhzNq7tepbFV6Qy1O7ea8TF+PR8cPRlaahSntopE6N0J9XVj9+kgCPLMGpXKGkGb0URFxj4m7/sd/+vn4hY/pOvBMVOVe9ygbsLmuGg/dFrhUD+BWs7ib4E4KeVYlFX/+709gZRymZQyRUqZEhwcbPMLtNBmNLE7t5qU7sxCFiIngtC2J5atyVlDelk6946/Fy+d/RpbAOjOOY/MoCGUvfoaxvpTuPZQ/g5w8bB7gxKN0PDwpIcpbSxl5d6VlNe3kFvZaJ0gsKxi89Q1DwkhGB/tz+686p4HD3Tyt4E+GnwjVJ02q6SOG3d/jfT0InjRou4HR5vzCVT2B7nrtCRF6h3qJ7CFICgAOtpAoszb2pFSVkgpW8x/LgcmWnus2uwvrqPRYDy+7kxXuHlDWBLkbsFgNPDqrlcZ7j+cyxIus/t1TowL4O3Rl2GsrKTineV2n89u5G+HiPGqJBhNCJ3ABXEX8F7Ge2w4dAiwwvwHEDgUPPwd4icYH+PH0YpGyutbeh48kMnb7hCz0KHv1pJSmoXHrQvR+vUg9COSlYVdwQ51Lq4DKbH+pOXXYGhzTGCALQTBdiBRCDFECOEKXAd803GAECK8w5+XAfvMv68BzhdC+JudxOebtzmMHhPJTiR6KhTs5D+ZH1FQX8CDKQ+iVaGq4sRYfw76R1M+5SwqV66ktVjd/ro2oa1FsR1HTex5rI24b8J9tMk2/n1gKS4aQVKEFfXnhTD7CdR/QEwwL0hSc09hraCmQPEDqWwWkm1tBH20lBKfIIbcOq/nA1y9IHSUQyLEkqP9MbSZ2F9cq/rcYANBIKVsA+5GeYDvAz6TUu4VQjwthLAsje8VQuwVQqQB9wK3mI+tBJ5BESbbgafN2xzG7rxqQn3diPTz6HkwQPRkaozNLE1fyozIGUyPsJ/TsyPheg8i9O78MOUKMJkoe+11Vea1KcV7wGhQdaUY7RPN3BFzOdS4gYTIOjxcrRTaUZOVrNgmdR/IYyL1uGjEqe0wtmhSUeoKguovviSgNJ8d51+Pxs3KkNXIFCjYpXpgQLLZRJnqIDOgTXwEUsofpJTDpJQJUsrnzNuekFJ+Y/79USnlaCnlOCnl2VLK/R2OXSGlHGp+vWeL6+kPaXnVJEdbYTe2ED2FpX6+NLQ18aeJf7LfhXXChFh/fqlxwf+mm6j56iua9+/v+aCBhGWFrbLJ4LakhUiTO0a/76w/KNp8jSqbDdx1WkZF+LbHwp+S5G0HF3cIG6PalKbGRkpe/xd7A+LQn3++9QdGpSgF6CoO2u/iOiFC706wj5vDND9nZnEHqhoM5FQ0Mq4XgiBPA//x9eEPuhAS/RPteHUnMzHWn8KaZtrmzkPr60vpi/9Qdf5+k78dfCJUdyCW1mhoKT+bkrZUthRZGS4YMQEQqmcYA4yP9iMtv3pANDDpE/mWRDLXnsfaiIqVK5EV5bw7eg4TrAn8sGBZlKhsHhJCkBztd2prBKcLafnKTeiNRvCv1H/hIjQsqlL/Blquc0+NJGjRnTRs2kT97+pXyuwz+duVFZjK7M6tprVqGiEeYby046Xjksy6xN3XYYllE2L9aTQYOVByCkaHtbUovSai1dP62ioqqFz+LoVJkzkcEs9oa/xAFgITlcQyB/iDkqP9yC5voKZR/c50TkHQgdS8aoSAsVHWCYJ9FftYfWQ1N+mTCCk7qHphspHhvui0grT8avzmzkUXGUnpyy8hVbZv9on6MqWui0MEQRX+Hp48MPE+9lfuZ/WR1dYdGO2gxLJoZUV7SlYiLUpT/EB2zhPpSPmStzG1tPDlxMsZGeGLu64XwRsaDUSOd0jk0Hjzwi41X/1FpVMQdCAtr5rEEO+eE8nMvLrrVfRueuaPvEHZULDLjld3Mu46LSPDfUnLq0bj6krwvffQkrmP2tVWPtgcSYFj/AOgaATJ0X5cHH8xw/2H88buN2i1pu1olGMSy6IDPAjydmXX0VPQT2Axsah0nw25uVR9+in6K69kfZMnyVG90AYsRE2CkkwwqNvzY0yUHiEcEyHmFARmpJSk9sJRvKVoC5sKN3H7mNvxiTFHChXstOMVds7YKD3p+TWYTBLfOXNwGz6csldfQxoGeEG6/B1KzHa4upUoa5paOVhaz/gYfzRCw30T7iO/Pp/PD3ze88GWh5nK91mxH/ufmhpBwS7FD+QTpsp0Za++hnBxoW7ufBoNxl75+9qJTAFptLoVra3wcVf6JaQ64D47BYGZvMomqhpbrfrgmKSJV3a+QrhXONeNuE5JNgocqrpGADAuyo/6ljayy+sRWi0hDz5Aa14eVZ9b8WBzJPnbIXQ0uPZQ58fGpJvVbktG8czImaSEprA0fWl76YkuCRwKbr4Ouc/jY/zILnOM/bhfFO6CyL61aO0tzZmZ1P7wAwHz5pHWqGj11pp5j8NirnSAecjiMFa7Y5lTEJixrLas0QjWHl1LZkUmdyXfhZvWHJ8cOVH54Kh8Ay3Xm5qn+Ce8Zs3CMyWF8reWYGoYoO0sTUblYeogs5AQtAt8IQT3T7yfyubKbjuZAYr9OHyc8nBTGct9Ti84hcxDjZVQma2aICh99VU0ej2Bty4gLa8aHzcX4oP6UOrFKwj8Yh3kMPanqrGV3MoeFiU2xikIzKTmVeOu0zA8tIuOZGbaTG28sfsNEvQJzImfc2xHZArUl0BtoZ2v9HjigxWfRpo57EwIQchDf8JYUUHlhx+qei1WU34ADHUOcRSn5VUTH+SFr7uufdu44HGcG3MuK/eupKq5B7U8cgIUZyjRMCqSFKnYutPz1Q1I6BeFu5WfEfYXBI3bt9Pw628E3X4bWl9f0vNrGBOlR6PpY2lzB2WSj4tW7rPaYaROQWAmLa9ayeLsqiOZmW8Pf0tObQ73jL/n+FISkeYyCSrbj7UawZhIfXvoK4BHcjLe55xDxbsrMFYPwBWk5X8Uqa4gkFKSXlDDuE7MBfeMv4emtiZWZKzo/iQRE8DUqggDFdF76BgS5NUu8E8JLJpTxHi7TiOlpPTlV3AJDsb/hhtobjWyr6i2b/4BC1EpUFeo+sJueKgPHjqt6gmETkEAGNpMZBTWdvqAOG6c0cCStCUkBSZxTsw5x+8MSwKNziF2xXHRfuwrqqWl7VhvguD77sVUX0/Fuz082BxBwS7F1h44VNVpi2ubKatrYWwnkSQJfoqG95/9/6G0sbTrk1jMHA4wD1kCA04ZCnabC/b144FsBfUbNtC0ezdBdy1C4+HBvqJa2kyScX2JGLJgWaSorBW4aDWMidQ7NQJHkFVch6HN1F7voys+P/A5RQ1F3DvhXsSJ3bRc3JQUegc4EpOj9bQaJfs6dLJyHz4c34svpvKjj2izc/+GXlO4W7G1a9T9+FkeomO6EPh/HPdHjCYjy9KXdX0SfTR4BjnkPo+N8qO4tpnS2mbV5+4ThbvsbhaSJhNlr76GLiYGv6uuAmjXmvqlEYQlgcYFitSNHAKl7lBmYa2qlUidggDaw7W6cxQ3tjayLH0Zk8MmMzV8aueDIicqDzmTul3DLB/4E80GwffcjTQYKF/azYNNbdoMUJJhd3NBZ6TnV+OiEYyO8O10f7RPNFcNu4ovDnxBXl1ep2MQwnyfHREhdgr5CWoLlWbwkfatLFv3v//RkpVF8D13I3SK3yctv4ZgHzfCfN37fmKdh9K+0uLnUJGxUXoMRhMHStRrUeoUBCgfnEAv124rjn68/2Mqmyu5Z/w9J2sDFiIngqFecYaqSJivOyE+bicJAte4OPyuvJKqTz+ltcChbR6OUbpXyTR1iCCoYVioT7eZpgvHLkSr0fJ22ttdnyhyApRlQYu6vYRHRfiiEcdCYAc0Fo3JjhFDsq2Nsn+9gVviUHwvvrh9e1p+NeOi/Lr+nlpLxHhFEKgcCTg20hwhpqLAdwoCYE9+DWOj9F1+cOoMdbyX8R6zomaRHNJNAlSUY+yKQgjGRft1mpoedNcihBCULVmi6jV1SXskibqCQErJnoKaTv0DHQnxDGHuiLl8l/0d2dXZnQ+KmABIpXyCini6ujAs1Ie0U0EjKNylmFbsWHG05tvvMBw5QtA99yC0inCvaWolu6yhf/4BCxHjoakKqnL6f65eEB3ggd5Dxx4VQ4UHvSBoNLRxsLSuS7sxwIeZH1JrqOWu5Lu6P1lAArj6OMauGG1OOGo6PuFIFxaG37XXUvPfrzAcPar6dZ1E4W4lAc8/TtVp8yqbqG5stSrBaEHSAty17ryV9lbnAyyrXIf4CfSk56ufcNRrCnYpRfp0Vvb16CXSYKD8zTdxHzUKn9mz27dnFChCsl/+AQuWxYrK5iEhhOqBATYRBEKIC4UQWUKIQ0KIRzrZ/6AQItPcvP4nIURsh31GIUSq+fXNicfam31FtZik0gCkM6qbq/kg8wNmx85mVOCo7k/WnnDkGLsiHPsidCRo4e0InY7yt7p4sKlJwW7lC9Zftb2XWMJre9IIAPzd/blh5A2syVlDVmUndYW8gkAf46DIIT+qGlvJr2pSfW6rkdLuGcXVX35Ja34+wfffd5wmb4m2seY+90jIaNC6OuT7PCZST1ZxHc2t6vgb+y0IhBBa4E3gImAUMFcIceITczeQYm5evwp4scO+Jillsvll/2a/J2CRul19cFbuXUljayOLxvXQ+NpCRLISY25NETMbYmm5uKcTQaDEV19PzTff0nL4sKrXdRytTVCa6RD/wJ6CGlxdNAzrIWHQws2jb8ZH58NbqV1pBeMdVlIEOC5vZMBRma1U4rVTxJCppYXyJW/jMX48Xmeccdy+9PxqYgM98fO0Qe8DF1cITXLYwq7NJNlfrI4fyhYawWTgkJQyW0ppAD4BLu84QEq5XkppyZnegtKkfkCwJ7+GEB83QjuJMChvKufj/R9z0ZCLGOpvZcx7xHgwtkDpvp7H2hB/L1ei/D06FQQAgbfdhsbDg7I33lD1uo6jOEMp5qVCpumJpOVVMzLcF1cX6z7yejc9N42+iZ/zfmZvxd6TB0RMUMpoN1TY+Eq7Z3iYD65azcCOHGp3FNsnYqj6s89pKykh+L6Tw7gVf58N8xYixiu+IJVLj1tM1XtUEvi2EASRQMdYu3zztq64FehYJ9ldCLFDCLFFCHFFVwcJIRaax+0os2FcfHcOxPcy3qPF2MKd4+60/oSW1a4D/ARjo/Ts6eIB4eLvj/+8m6hb/T+as9Qto9yOSpmmJ2IySTIKanrtQLxp5E3o3fS8ufvNk3e2J5apu1p0ddEwMtxnYGcYF+5WWlMGj7D5qU3NzZQvW4rnpEl4Tjm+x0FFfQuFNc2Miew8PLhPRIxXWldWdhE4YCci9O4EermqJvBVdRYLIW4EUoCOPRVjpZQpwPXAq0KIhM6OlVIuk1KmSClTgoODbXI9DS1tHCqrb6/j0pGyxjI+zfqUOfFziNPHWX9S/yFKhyMHqJNJkXpyKxu7rFAZOH8+Gh8fyh2lFRTuBq8Q1VtTZpfX02Aw9nql6O3qzS2jb+G3gt9ILT1BsFvKZzvEbOBHRoFSenxAUpSqRAtprevr0RuqPvkEY1k5QffcfbI2YNaGO/s+9xkHOozHROm71PBtjS0EQQEQ3eHvKPO24xBCnAc8DlwmpWyv2CWlLDD/zAY2AKotF/cW1iJl5/6BFRkraDO18cexf+zdSTUaCB/rmAeEOf64qw+PVq8nYN486tauo3mfuqYrQPmfOMBR3JMfqDuuH3E9/m7+LEk7IfzW3Vwiw0GaX4PBSHb5AGxdaTJBUbpd+kyYGhupeGc5ntOm4jV58kn7M+whCIJHKNqNIwIDIvUcKKmjyWB/h7EtBMF2IFEIMUQI4QpcBxwX/SOEGA8sRRECpR22+wsh3My/BwEzgEwbXJNVWBJzTvzglDaW8lnWZ1yWcBnRvtGdHdo9EeOhZK+SRasiSWaVuLtVRMDN89D4+FD2RifmDnvSUqckYalUkrgj6fk1eLpqSQj27vWxnjpP5ifNZ1Phps61ApWbl4DSyQq6v88Oo/KwUlk2wvaCoOo//8FYUUHwPfd0uj89v4YhJ1SW7TdaFwhzzMJuTJQfJgmZRfa/z/0WBFLKNuBuYA2wD/hMSrlXCPG0EMISBfQPwBv4/IQw0ZHADiFEGrAe+LuUUjVBkFFQQ7jenRCf4x3Fy/csxyRNLBy7sG8njhivZM+WqvZWAPDzdCUmwLPTEFILWl9fAm65mfqffqIpoxMnqL0oSgekwyKGRkf4ou1jSeJrh19LgHsAb6aeIDwjkqE2HxrKbXCV1jM02Bs3Fw0ZBbWqzmsVFsFoY43A1NBAxfJ38ZoxA88JnS8mMgpqbKsNWGh3GKtbOmasiiVFbOIjkFL+IKUcJqVMkFI+Z972hJTyG/Pv50kpQ08ME5VSbpJSjpFSjjP/fNcW12Mt6Z18cIobill1YBWXD72cKJ8+BjdFOM5+PCZS32PzkoB589D4+qrrK7D8L1RuTWk0STILa/v1gPDUebIgaQFbirawq6SDiaDdT6CuVuCi1TAy3HdgagRFqXZxFFd+/DHGqiqC77m70/12cRRbiBgPrY2ql44JNZeO6SoAxJYM2sziumYlFX3sCQ+Id/e8i5SS28fe3veT+w8Bd71D7MdJkXpzFm3XZimtjw+BC+Yr5Xv37FHnwopSzb1rQ9WZz8zhsnqaWo1dJgxayzXDryHQPfD4vILwscrPIscI/MzC2oHnMC5MVWLvbegoNjU0UPnuCrzOOAOP5M4XEnZxFFtwkMMYzJnkKgj8QSsILGr1mA4OxOKGYr44+AVXJF5BpHd3EbA9IITZfuy4DOOeVov+N96IVq+nXC1fQVGaXezGPWFZTfVXEHi4eLAgaQFbi7eyvXi7stFdr5QVcYSfIFJPfUsbRyoGUDtSk8l8n21r/qv8+GOM1dUE39V1UqddHMUWghJB5+Wg++zH4bJ66lva7DrPIBYEJz8g2rWBMf3QBixEjIeSTPVbGnaTYdwRrbc3AfPnU//LL/bXClrqoPygUn5DZfYU1OCh0xLfB0fxiVwz/BqCPIKOr0wakax68Tk49sDrzh+kOnZwFBvre9YGwE6OYgsarRIO64D7PCbKFykhs9C+/qBBKwjSC2qI9PMg0FtpPt9RG4jwtkGce8R4paVhiYoOWUDvqSM2sHuHsQX/G29Ao9dT/qadaxAVZwBSdf8AKA/KUf1wFHfE3cWd+aPns614GzuKzRVmw5OhJk/1DOPEUG9cXTSq2I+txg6O4iqLNnB39wUf7eYothA+DorTVXcYdYwcKAAAIABJREFUqyXwB60g2FtQ0x5uCTbWBsChDuOkSOsqF2q9vQmcf4vZV2DHHrwWX4nKpiGjSZJZVNtvs1BHrh5+NYHugce0Ast7UtlPoBuIDmMbO4pNDQ1UrliB16wz8BjXtTZpV0exhYhkxWFccch+c3RCiI/iMHYKAjtQ19xKdnlDuxnF5toAgF+sUm7ZAerk2Eg9+VVNVDX0nMfgf+ONZq3Ajr6CwlTwDgWfMPvN0QlHyutpNBhtulL0cPFgftJ8thZvZWfJzmPmLofYj33ZO5AcxjZ2FB/zDXSvDexpN/PasTeyQ++z/TOMB6UgsNjbksyOVYs2cNuY22w3iRDKh8cBkUOWFbA1Hx5VtIKiNIeYhfZ04geyBdcMv4YA9wAl29hdDwHxDrvP9S1tHK1s7HmwvWl3FNvmPivawHt4zZzZrTYAx8wmo+2pEQQNBxcPh9zn0ZF6DpfV02iwn8N4UAqC9lCzCD0lDSV8cfALLh96ef8ihTojfJzZYaxuhvFos6az10oHU7tWYI9+BYYGKM9yUMRQLe46DQnBXjY9b3sEUdFWJa8gPBkKHecwHhDmocpsxVFsI4Ff9cknGKuqCOomUsiCXR3FFrQuSkN7RziMI/WYpNI7xV4MSkGwt7CWMF93gn3ceG/ve7bXBiyEj1McxmXq1vXRe+p6zDDuiNbbm4Cb51G/fj3NmTbOhi7OAGlySMRQRkENI8N9cdHa/mNu0QreTntbEXI1udBYafN5uiMxRClJPSAih2zoBzI1NlLx7gq8pk/Hc3zPoah7+5kwaDXh45QMeZVLUlt8mfbMJB+UgmCP2VFc1ljGqgOruDTh0r5nEXeHgzJPQfnwZBRa/4AIuOkmpQaRrbUCywpKZdOQySTZW1hjc7OQBQ8XD+aPns/mos2keplNEg4oST0i3GdgRA4V7gatm00cxVWffIqxspKgHiKFACobDBRUN9nXUWwhPFnRelQuSR3m606Qt6tdNb9BJwgaDW0cNpeetlQY7VcWcXf4DwE3X4eok6Mj9BytaDyph3FXaH18CJg3j/p1P9G8f7/tLqQoFbyCVS89faSigQYbO4pP5Jrh1+Dv5s/bJRuVDQ7KJM8orHF8D+OiNMV0ou2fecbU1ETFihV4TZ/WZU2hjmR0MPPanfYIMXXvsxCC0RF6u2p+g04Q7CtSSk/HBBv5/MDnzImfQ7RPHyqMWoOlh7EDHcZ7e6MVzLsJjbc35W8t6XmwtRSmKv8DlUtPq/GA8NR5cvPom9lYvJU9QbEOsx/XNbdxtMKBDmMpzaWn+2/+q/7sM4zl5QQtsq41rEXrHa2GIAgeoWg9Dvo+Hyytt1sP40EnCCxqdEb917SaWu2nDVgIH+eYHsYWQdALu6LSr+Am6n78keYsGxTYam2Csv2OiRjKV3oUJ4b2P6O4O+aOmIufmx9L9D4OEwRAr8yANqfqCLTU9Ps+m5qbKV++HM8pU/BMSbHqmL0FtUQHeKD3tKOj2IJWB6GjHWTq1WM0Sbs5jAedIMgorCXQ18C3OV9wyZBLiPWNte+E4eOUHsYqVy4M8HIl0s+j1w+IgHnz0Hh5Ub7EBlpByV5zj2IHZBQXKo5inR0cxR3x1Hkyb9Q8fjPVsrehEJqq7DrfiSSGeqPTCseWpG73A/VPI6j+fJXSfcyKSCELewrs5wfqFIvDWGVTXLvD2E6lJgafICioISBiEwajwf7aADjUYTw6oveZp1o/P/xvvJG6NWtoOdTPLEqLCq1yxJDJJNlbUKuOAxFFK/B18eRtP18oVqmaqxk3Fy3Dw3x6ZQK0OUVpoNFByMg+n8LU0kLFO+/gmZLSafexzqhpbCW3slEds5CFiGRF+6k6ot6cQKSfB/6eOjLsFBgwqARBc6uRgxUllGs2cGHchQzRD7H/pIEJSuVCBxUmO1Le0OvKhQG33Izw8KB8yds9D+6OwlTwCAC9nXwwXZBb2UhdS5s6DkSU3sbzhl/HBi9P9h1Zp8qcHUmKUDJPHeYwLkyF0FHg4tbnU1R/8QVtpaW90gb2Ftmx4mhXOGhhJ4RoDwywBzYRBEKIC4UQWUKIQ0KIRzrZ7yaE+NS8f6sQIq7DvkfN27OEEBfY4nq6Yl9RLVq/3zDKlr53H+stGq1St95BDibZh0QUF39/Am64ntoffqAlux+hckVpjnEUF6r/gLh+7G34mCRLCzeoNqeF0ZF6qhtbKahuUn1uxVGc1i+tTxoMVLyzHI8JE/CcOtXq444FBKij+QEQMkrRfhy0sDtQUkdLm+0dxv0WBEIILfAmcBEwCpgrhBh1wrBbgSop5VDgFeAF87GjUHocjwYuBN4yn88ubM8rwNV/E2dEnEuCX4K9pjmZ8HGKyUDlyoWWlPu+xJkH3HILwt2diqVL+zZ5WwuU7nNMRnFBDTqtYFioj2pz+rj6cKMulJ+MVWRVZqk2Lxx7EDrET1CTD02V/RIE1f/9iraiIoIWLUL0YtGQUVBLhN69vYKwKri4KtqPIwRBhJ5Wo+RgSb3Nz20LjWAycEhKmS2lNACfAJefMOZy+P/tnXd4VFXegN8zM5n03iuhhJpA6E0s2MWCq7i4uiKCCOiqq+uuru639nXFVXFVxLKWXRsqCqKIDVRUOiSEmoQE0ntvk8yc7487EwKkTJ8JzPs8eWbmzr33nMmZub/767xtfP4xcL5QVvwq4AMpZZuUMg/IMZ7PIaw/ugqhbuOu8UscNUT3xI5xbeVCK9RJTXg4oXPnUvf5OnRHj1o+ePl+JavaBRnF+4rqGRYTiFbjXMvnDXHnEWAw8OoeB5f1PokRsUqZbZdkGNuYMCh1OqpWrsR3zBj8p0+z6Nis4jpGOdMsZMIUEu5kU9y5wyLZ9tfzHaLp2uOXEg8UdHldaNzW7T7GZvd1QLiZxwIghFgkhNghhNhRUVFh1UTbqSdCTGRo2FCrjrcaFzqM0+KtT0QJX3ALwsuLyles0ArsFEliKVJKsorrnOYf6EpwwiR+V9fANwUbyalxntD38VKTEhXgmhDSkj0g1EpYpRXUrV1Le3ExEbdbpg00tnWQV9nk3IghE7FjlOiwuoK+97Uj/t4aooJ8HHLufuMsllK+KqWcIKWcEBkZadU51v3ueb698TU7z8wMIoa6tHJhTnkjLTrLzVKaiAhC5/6WurVr0RVY+KUvyQDvYCW72okU1bZQ29zusjvF39c34Cs0rMy00qRmJabMU6c7jEsylEQrL1+LD5Xt7VS+shKf1FT8Z8yw6Nj9xUpiaKqTIsNOINbUw9j5v2dHYQ9BUAR0DQtJMG7rdh8hhAYIBqrMPNauqFUOc0H0MqjrKhemxgVhkLDfykSUsFsWINRqql591bIDi/coTnIXZRS75E4xMJoQvyiu94piQ/4GjtQ5ryZNWnwQlY06yhuc2xrVFkdx3bovaC8stNg3AE4uLXEy0SMVLcjJv2eDNFDWVOaQc9tDEGwHUoQQA4UQWhTn79qT9lkLzDM+vxb4Xiq3LmuBucaoooFACrDNDnNyP1xUuTAtwfJSE13xio4i5LrrqP30M3SFZspovbFFp0sqjtajVgmGxzjPUXwCcencVF2Dj8aHVzMtFJ420FmS2pkF6OpLoLHMqoAA2dFB5Ssr8B45goDzzrX4+KyiOqICvR1mKukVL19FC3KyIPj26LdcuvpS9lXav/2tzYLAaPO/A9gAHABWSSn3CSEeFUJcadztDSBcCJED3APcbzx2H7AK2A98BdwupXRuaI2zcGHlwnB/rU0XiPCFCxBCUPWamWa1ikNKNrUrehQX15ESFYCPlws0P4DYMYRVHmbukGtYn7ee/Lp8pww7IjYIIZxcasIGP1D9l1/SfvQYEUuWWKwNgNFR7Myw0ZOJS3eqw9ggDbyS+QoJgQkMD7NPK9Cu2MVHIKX8Uko5VEo5WEr5hHHb/0kp1xqft0op50gph0gpJ0kpj3Q59gnjccOklOvtMR+3xPRjcUHlQiURxfrQQq+YGIKvvYba1atpLy7u+wAX9SiWUjq+iXlfxI4BaeCmiPFoVVpe2+scn5S/t4bBkQHODSEtyQCE0p7SAqReT+WKV/AeNozA88+3eNhmXQc55Y2uMf+ZiB0DTRXQUOKU4TYe20h2TTaLRi9yiHm73ziL+z1RI0CtdVGp4iCyyxpsqlwYcatSjqPSHK2gJAO0ARDmxFwNoKy+jcpGnXMTjE7GqAVFVOUxZ9gcvjjyBQX1zokuSY0Lcm4IackeiEgBb8sK+9Wv/wpdXp7iG1BZfgk6UNKAQTo5o/hkOm/sHG8eklLySuYrDAgawCXJlzhkDI8gcBamyoUuqlDZYZAcKm2w+hxecXGEXH01dR9/Qntpae87l2RATJpShtuJdDoQXXmBCIoDvwgo3sMtqbegUWmcphWkxgdTWt9KhbMcxsV7LDb/KdrACrxTUgi88AKrhu0MCEhw4TrHpAHCKZFDmwo2cbD6ILem3YpGpXHIGB5B4Exi05WLpNMrF9qnt234okVIKal67fWedzLolSxqFzWrFwJGulIjEMJoP84gwjeCa4dey+e5n1PYUOjwoU3F15ziJ2gsh4Zii81/DRs2oMvNJWLpEqu0AVDWOdxfS4wrHMUmtP5KWLiDb+yklKzIWEFCQAKzBs1y2DgeQeBMYsdAax3U5Dt12PgQX0L8vGw2G2gT4gmefRW1H31Ee1l59ztVZitZ1K7IKC6uY3BkAH5ax9w1mU3sGKVPdXsrt6TegkqoeH1vL8LTTphKijiqQuUJWJFRLA0GKlesQDt4MIEXXWT10CY/kDVOZrsSO8bhguDHwh85UH2ARaMXOUwbAI8gcC4ubHWXZqfKhRG33YbU66l6vYcLm4syisHYi9qV2oCJ2HQwdEDZPqL8orhm6DWsyVlDUaNDU2QI8vFiYIS/Q3vbdmIyicSONvuQhq+/oS07h4jFixFq6xyere16sl3tKDYRl65oRY093BTZiEkbiA+I5/LBlztkDBMeQeBMXFi5cFRcMIdKba9cqE1MJPiqq6hdtYr28m5+ACV7lCzqCOeW8ShvaKWsvs21/gETnQJfaWZ/S+otCCGcohWk2lBSxCJK9kD4EPA2L19DGgxUvvQS2oEDCbrsUquHPVBSj94gXZNRfDIOdhj/VPQT+6r2cWvarXipHNuBzSMInInGW4keclHNIXtVLoxYfBuyo4PqN/5z6pvFexRHmtq55hmXZhSfTHCi0ofBuM4x/jH8JuU3fJbzGSWNjg03TIsPoriulapGBzuMLXQUN3zzLW3Z2YpvwEptANwkIMBETJry6IDfs5SSVzJeIT4gnisHX9n3ATbiEQTOxmRXdFGrO3uYDbRJSQRfcQU1H3xAR9cCgAa98tlcUXq6sB4hcE2NoZPpdBgfv0AsSF0A4PAIInsFBvRKUyXUF5q9ztJgoPLll9EmJxN02WU2DZ1VVE+onxfxIZbXNrI7PsFKiLQDTL2bizazt3IvC9MW4qV2fD9mjyBwNnHpSv12J1cuTArzI8hHYzezQcSSxciODqpef+P4xqocaG9yWcTQoAh/Arxd7Cg2EZuu9GNob1VeBsTymyG/4dOcTx2qFZgEgUPNQ50tSM1b54Zvv6Xt0CGbtQEw+oHcwVFsIm6s3TUCkzYQ6x/LVYNPrujvGDyCwNm4utWdnS4Q2gEDTtUKTJ8pbqxdxrCEvUW17mEWMhFndBiX7+/ctDBtIYBDfQVBPl4kh/s5ViOwwFGsaAMr0A4YYLM20Nqu53BZg3uYhUzEpSvaUaN1pfG74+fin8mszHSaNgAeQeB8okcZKxe6IsM4mAOlDbTr7VP47hStoHi3x1FsopuSIiatYHXOaodrBQ4tNVGyB8IGKaaRPmj47jvaDh5UtAGNbdra4bIGOgzSzQS+8abHTr9nKSUr9qwgxj+Gq4dcbZdzmoNHEDgbL1/FYeyinqe6DoPdWt2dohWUeBzFnYQMAJ+QUzS/W0crpTocqRWkxQdTVNtCdZPOMQMUZ5hlFpIGA5UvvqRoA7NsT4ba68rS0z0RY9SKinfb5XQmbeDWtFudpg2ARxC4hrh05YvjbIdxnMlhXGu3cx7XCl5Xymy7wizkTo5iE904jEGJILom5RqHagVpjnQYN1dD3TGzHMUn+AZs1AZAEfjBvl4khrmBo9iETxCEp9jF1Cul5OU9LxPnH+dUbQA8gsA1xI2F5iqnO4yTw/0J9NbY9QLRqRW8/wHt9S0ua1Y/0J0cxSZi06FsP3ScGMpp8hU4KoJolCMdxp2O4t4TBju1geRku2gDoEQMpcYHuY+j2EQ3At8aTJFCt452rjYAHkHgGkx3zXZSJ81FpRKkJQTbvXlJxNIlyI52qg4EuKYHQVGde5mFTMSlg6H9BIcxHNcKPs351CHZxsG+XiSF+TmmSU2xeYKg4etvaDt8WOlFbAdtoK1Dz8HSevcyC5mITYf6IpsyjE3aQHxAvNMihbriEQSuIDpVyTB2siAApWLjgZIGdB3265SmTUoieEIitTn+tBtC7HZecyhvaKW0vtU9BUEvEWIL0xYiELyW6RitIC0+2DGmoZI9EJoMvqE97tKZRTxokM2RQiYOlTbQrpeMTnDu98ssOm/srNcKfir6iayqLKf7BkzYJAiEEGFCiG+EENnGx1O+HUKIdCHEr0KIfUKITCHEb7u895YQIk8Iscf45/zbSVeg8Vb6nrpAEIyOD0GnN9hUkro7ItI7kAiqXu8m29iBuKWj2ERosuIw7sZsEOMfw5yhc/gs5zMKGuxvIkw1Ooxr7O0wLtrdpx+o4euvjVnES23OGzCRYdRuRruy9HRPxI5GKUlt3e+5qzZw5RDHZxF3h60awf3Ad1LKFOA74+uTaQZuklKOAi4BnhdCdBXr90kp041/zo+pdBVxY13iMDb9kDLt6DDGoEfbsp+QiUlKDaK++hXYEbd0FJswOYyLdnX79oK0BWhUGof0NnaIw7ip0ugoHtfjLlKvp+LfL6IdPJigS+3XRGVvYS2hfl4khLqRo9iEd6DSoMdKP8HGgo3sq9rH4jGLHV5TqCdsFQRXAW8bn78NzD55BynlYSlltvF5MVAORNo4bv8nbqyxJHWeU4dNCFVKUtvVfmwsPR1+w1VIoHLlSvuduw/c1lFsIm6c4iMwZhh3JcovijlD5/B57uccrT9q12EdIghMd7zxPQuC+i+/RJebS+Qf7rCbNgCQWVjH6IQQ93MUm4hNt8o0ZJAGXt7zMkmBSVw+yLEVRnvDVkEQLaU0xcCVAtG97SyEmARogdwum58wmoyeE0J493LsIiHEDiHEjooK+2XxuQwXOYxNJakz7SkIjHdC2vTzCLnmN9R+/AntRY4tuWzCbR3FJuLHGUtSZ3X79oK0BXipvHgl4xW7Dhvsp2QYZxTYUfMr2gWIHh3FsqODyhdfUnoR29Bv4GRadEpGsVuahUzEjVVKUjeUWXTYt0e/5VDNIRaPWezQfgN90acgEEJ8K4TI6ubvBNe2lFICPdo5hBCxwH+B+VJKk6fyAWA4MBEIA/7S0/FSylellBOklBMiI08DhSJyBKi9XeMnSAjmsI09jE+geA94+UHEUKXWvBBUrFhhn3P3Qlm94ih2SweiCZPA78E8FOEbwdzhc/ky70uO1B6x69CjE0LsK/CLdytZ4z2Unq5bsxbd0aOKNmDHNqX7S+owSNx8nS3vNaI36Hl5z8sMDB7IZQPt41S3lj5XS0p5gZQytZu/NUCZ8QJvutB3Gz8lhAgCvgAelFJu6XLuEqnQBrwJTLLHh+oXaLQQk+qiktQhdBgkB0rsVIageJeSYalS4xUTQ8jc31L36Wfo8vPtc/4eMN3tjnHnO8WgePCPUv5HPTA/dT4+ah9e2vOSXYcenaD0MC6vP9UsZTFSKp+hB0exbG+n8uWX8Rk1ioDzz7d9vC5kFLixo9hEjOUO4w35G8ity2XpmKWoVfYzo1mDrWJ7LTDP+HwesObkHYQQWuBT4B0p5ccnvWcSIgLFv9C9/ny6YrIrGuwXymkOph+UXezH+nalXEb8+M5NEbfeivDyouKll20/fy9kFtahVonOXr1uiRCKeagHjQAgzCeMG0feyNdHv+Zg9UG7DZ2eqNxBZ9hDK2gogcayHv0Dtas/pb2oiMg7/2B3O/7eojqiAr2JdmWP4r7wDoDIYb2uc1c6DB2syFjBkJAhXJRsPzOatdgqCJ4CLhRCZAMXGF8jhJgghDAVU7kOOBu4uZsw0XeFEHuBvUAE8LiN8+lfxI0FXQNU5/a9rx2JDfYhIkDbeadlE+UHoKP1hAuEJjKSsBtvoH7dOtqys20fowcyCmsZGh2Ir9a1d1N9EjcOKg9DW88hu/NGzSNQG8hLu+2nFYyKC0atEmQW2sFPYLrAdRMxZGhro/Lll/EdMwb/s8+2fayTyCisdW+zkIn48YrWZEYk4Oe5n5Nfn88dY+9AJVyfzmXTDKSUVVLK86WUKUYTUrVx+w4p5ULj8/9JKb26hIh2holKKWdKKdOMpqYbpZT2qYbWX3Chw3h0Qoh9ag4V7VQeT7pTDFuwAJWfHxX/ftH2MbpBSklmYZ17m4VMxI0FZK+FBoO0Qdw86mY2FW4isyLTLsP6atWkRAXYRyMo3gUqjWLOPIma99+no6yMyD/ebXdtoKG1nSMVTe5tFjIRPw6aKvosHaPT61iRsYLU8FRmJs500uR6x/Wi6EwmcjhofFyTYRwfTE55I01tHbadqGinkmUaOvCEzZrQUMLmzaPh669pydpn2xjdcLSqmbqWdsYk9oc7RaOQ7MNscMOIGwj1DuXfu/9tt6HHJISQWViLtDVfpXi3UjXX68Q4fkNTE1Wvvobf1Cn4T5li2xjdYCqn3T8EgdE8aro56oGPD39MSVMJfxhrfzOatXgEgStRa5RQPDPtivZkdEIwBgn7bXUYF+9WzAXdfKHD5t+MOjiYiuXLbRujGzKM5o5+cYHwj4DgpF4dxgD+Xv4sSFvAlpItbCvZZpehRycGU9vcTkF1i/UnkdK4zqc6iqv/+1/01dVE3X23DbPsGZNZy61DhE1EjQK1tldB0NLRwmt7X2N89Himxk114uR6xyMIXE38eCXkTN/u1GHTjBdQm+LMdU1KslQXR3FX1IGBhC+6laaffqJ5+3brx+mGjII6vDUqhkZ3H8rodsSPNUvg/3bYb4nyi2L57uW238WjaARwXHBaRU0etNSc4h/Q19VR9cZ/CDjvPHzH9F6Ezloyi+qID/ElPKDHFCP3QaNVood6Wef3D75PZUsld4690220AfAIAtcTP15xtpbZ33zSG1GBPsQF+7DHFkFQkgnS0KMgAAj93e/QREZS/tzzdrmwmcgsrCU1PhgvdT/5CseNhdqjSj3/XvDR+LBkzBIyKzLZVLDJ5mGHxQSi1ahscxibTJcnaQRVb/wHQ0MDkXffZcMMeyezsJYxif1AGzARP94YCXhqjk69rp439r7B9PjpjIvuOTvbFfSTX9FpTMIE5bFoh9OHTk8KsU0Q9OAo7orK15eI25fSsmsXjT/8YP1YXejQG8gqrusfZiETprvpPsxDALOHzGZA0ABe2P0C+m4uKJbgpVYxMjbINodx0S4l+TF6VOem9vJyqt95h6BZs/AZNsymOfZEVWMbBdUt/SNiyET8eGhvgopDp7z1VtZb1OvquXucY8xotuARBK4mZAD4RbjETzA2MZTCmhYqG9v63rk7indBcCIERPW6W8g11+CVmEjF88uRdsiZOFzWSGu7oTNOvl9gyjwt6jswQKPScEf6HeTU5vBl3pc2Dz0mIZisojr0Bis1suLdxhakxwuiVb78MrKjg8i77rR5fj1hMmeN7U/r3IPDuKK5gv8d+B+XDryU4WHDXTCx3vEIAlcjhPLlKXSNRgCw55iVWkHRTrNaUwovLyLv/ANtBw9S/4XtF7bMTkdxP7pA+AQrLQ37iCgxcVHyRQwPG85Le16i3Ub/0eiEEJp1enIrrIjO1ncoNykJEzs36fLzqf3oY0Kvm4M2KcmmufXG7mO1qI3NlPoNYYPAO/iUdV6ZuZJ2fTt3pN/hoon1jkcQuAMJE5SEo1YHNBLphVRjwtHughrLD26qgpr8Xv0DXQmaNQvv4cOpeP55DDrbauRnFNYR5KMhOdzPpvM4nYSJULjdrIQjlVBx17i7KGosYtXhVTYNa7KxW2UGLN8HHS3HTZhAxQsvILRaIpYssWlefbH7WC3DogPx07ppZdnuUKmMgQHHBUFBfQGfHP6Ea4ZeQ1KQ4wSnLXgEgTsQPx6QTjcP+WrVDI8JtO4C0VmS2DxBIFQqou69l/aiImo/+NDy8bqgOBDduCRxTyRMgOZKRYCawfS46UyOmczKjJU06qzPtRwUEUCgj8a6dS40RnsZNYKWffuo/3I9YfNuQuPA4o8GgySjoJaxSf1I6zMRP95YelwJ2f33nn+jUWm4bfRtLp5Yz3gEgTvQaVd0vnlobFIImQV1GCy1HxftBIRFzer9z5qO35QpVK5Ygb7Rugtbi07PoVI3L0ncEybziplmQCEEd4+/m5q2Gt7a95bVw6pUgvTEEHYdtULzK9yhFM0LUe5kK/71LOrgYMIXLLB6PuaQW9FIQ1tH//IDmYgfr5QeL93Lvqp9rM9bz+9H/p5IP/etmuwRBO6Ab4jRfux8h3F6YigNbR2W24+LdvZakrg7hBBE3Xsv+poaqv9jXUvLzMJaOgyScUk998x1W6JGgpc/FJqfLJYakcrFyRfzzv53qGyptHrocUmhHC5roNHSTPLC7YoAE4LGzT/T9MsvRCxdgjrQsfkbu43ay9j+uM7GGztZuINndzxLqHco81Pnu3hSveMRBO6CyWHs5NaVpjuu3ZaYDaRULmaJllcN901LJeiyS6l68y3ay7utWt4rO48pd7X98gKh1iihtoWWJdfdOfZO2vXtrNhjfY+HcQNCMUgLEwibq6EqBxL//UzbAAAgAElEQVQmIPV6ypctwyshgZDrr7d6Huay+1gtQT4aBkX4O3wsuxMYA0HxbD76HdtKt3HbmNsI1Lp34qNHELgLCROgqbzPglX2ZlCEv+X246ocJdPUCkEAEHnXXUo3KysK0u06WsugCH/C/LVWje1yEiZA6d5O+7E5JAUlce3Qa/kk+xOO1FnXvKZT4B+zwDxkcngmTKRu7ee0HTpE5B/vRqV1/P9+97EaxiSGoFL1Mz+QEX38BJ5tPkxiYCLXDb3O1dPpE48gcBdMfgInh5Ga7Me7LQkhLdiqPCZOtmpM7YABhP3uemo/+YTWw4fNPk5Kye5jNf1TGzCRMEmxH/dSibQ7Fo9ZjI/Gh+d2PGfVsMG+XgyJCmCXJetcuB2ECkPYCCqWL8cnLY2gSy+1anxLaGrr4HBZQ79e57XBIeSo4a4R8/BSu6YhvSV4BIG7EJ2qZG+aGWduT8YmhnCotJ5mnZn244Kt4GP0a1hJ+OLFqAICKF/2jNnHHKtupqpJx/gB/fcC0RmGWWBZUblw33AWpi1kU+EmqwvSjUsKYfexGvNLfRRuh6hRVH/4KR2lpUT96U92bUHZE5mFSmvKfhkxBDS3N/Ni7R7SWtu4SN8/NFebVlUIESaE+EYIkW187PYXKoTQd2lKs7bL9oFCiK1CiBwhxIfGbmZnJhqtUonUQvuxPUhPCsEgYa+5ZQgKjA5EGy4KmtBQIhYvpumnn2jc/LNZx+w0Rr2MG9A/LxCAkoUdMsCqdb5xxI3E+sfyzI5nMEjLM7THJYVS09xOXmVT3zsbDFC4k47gNKpWriTg3HPxn+ycTrImM2V6f0oY7MLb+96mvK2GP9U1I1zwe7YGW8X7/cB3UsoU4Dvj6+5o6dKU5sou2/8JPCelHALUAI6NSXN3kiYr8fntdugxawHpiYr8Nsth3FILFQesNgt1JfTGG/CKj6d82TKkvu+aOruO1RDgrSElyr0db32SOMkqE6CPxoc7x93JgeoDfHHkC4uPH2fUpMwyA1ZlQ1sdFT/XYWhrI+rPf7Z4PGvZfayGgRH+hPZDP1BZUxlv7nuTiwZcxLiItONmVDfHVkFwFfC28fnbKH2HzcLYp3gmYOpjbNHxpyWJU0CvU8pSO5Ewfy0DI/w777h7xXQBs9JR3BWVVkvUvffQdugQtatX97n/rqO1pCeGoO6nDsROEiZCQzHUFVp86GUDL2NU+CiW71pOS4dlPQaGRAYQ6K1hlzkO48LttNZqqN2YQejvrsd70MC+j7EDUkp2F9T2r/pCXXhh9wt0GDq4e/zdym+kJMOiwABXYasgiJZSlhiflwLRPeznI4TYIYTYIoQwXezDgVoppckwXQjE2zif/k2SscPTsV+dPvTE5FB25Ff3nVhWsBWEyuyM4r4IvPRSfMeNo+L55egbeu7p29TWwcHS+s672n6NyU9ghdlAJVTcN/E+yprLeCvrLcuOVQnSk0LMchjLgm2UZ4SjCgggculSi+dpLYU1LVQ0tPVL/8C+qn2szV3LjSNvJDEwUdGaDe1KWWo3p09BIIT4VgiR1c3fVV33k4oHqqeryAAp5QTgd8DzQojBlk5UCLHIKEx2VFRUWHp4/8A/QnHAHnO+OjkhOYya5naOVPaRWFa4TSlH7B1gl3GFEET/9a/oq6upXPFKj/tlFNRikIrDs98Tnaa0KLUyQmx89HguSb6E/2T9h5LGkr4P6MLYpFAOldb3mVjW+NMvNJWoibzjDtQhzvufb89X+jVMSA5z2pj2QErJ09ueJswnjFvTblU2mrTmfmAe6lMQGJvSp3bztwYoE0LEAhgfu80QklIWGR+PAJuAsUAVECKEMFWUSgCKepnHq1LKCVLKCZEOrHHicpImQ8EWxVnnRCYaf3jb83sxGxj0ysXLDv6BrvimjiL46qup/u9/0eXnd7uPyZwxNvE00Ag0WqVq67EtVp/invH3IJE8u/NZi44bZwwMyOzFHyTryijfVIc2OpjQ6+daPUdr2J5fTZCPhmH9pfOckQ35G9hVvovb028/njzmHwFhgy2OEHMFtpqG1gLzjM/nAWtO3kEIESqE8DY+jwCmA/uNGsRG4Nrejj/jSJqqJGtVmh9fbw+Sw/2ICNB23pF1S/l+0DXaXRAARP3xblReXpQ9vazb93cdq2VIVADBfu4fk20WSVMVX5DOjAiebogNiOWW1Fv4Kv8rdpaZH3JsEqS9+YOqX34GXYOG6DtuRng59/+9La+aCclh/SqRrLm9mWd2PMOIsBFck3LNiW8mTlY0AidXDLAUWwXBU8CFQohs4ALja4QQE4QQrxv3GQHsEEJkoFz4n5JS7je+9xfgHiFEDorP4A0b59P/STI2tHayn0AIwcTksN4FgUnF7VKb3l5oIiMJX7KYxu+/PyWc1JRINr4fJxidwoDpSmKZDeGF81PnE+0XzT+3/dPsTmbBfl4Miw5kWw/r3F5WTuUHXxEQ30bA7Jutnps1VDW2kVvR1Kmd9hfeyHqDsuYyHpj8AGqV+sQ3EycpFWerrcsIdxY2CQIpZZWU8nwpZYrRhFRt3L5DSrnQ+PwXKWWalHKM8fGNLscfkVJOklIOkVLOkVJa2SrrNCJskNKxzAV2xQnJYRRUt1Ba10P4asF2pRJlaLJDxg+bNw/tgAGUPf74CT0LcsobqWluZ3zyaSQIEicpTvejv1h9Cl+NL/dOuJcD1Qf4JPsTs4+bPCiMnUdraNefan4sf+YZZEcH0ZcNBC8fq+dmDSaz5KSB/WedCxoKeCvrLWYNmsXYqG6aNJm0Zzf3E3gyi90NIZToIRdFDgHsONqDVnDsF8WH4aA+ACqtluiHHkSXn0/1W293bt9ypAqAKQPDHTKuS/AJgpjRNgkCgEuSL2FizESW71pOdWsv2lwXJg8Mp1mnJ6voxATC5p07qf/8c8KGN6FNP8emeVnD9vxqvDUq0uL7T0DAsu3LUKvU/HHcH7vfIXK40rHMBn+QM/AIAnckaarSvKSh1KnDjowNwk+rZkd3DuOao1B7DJJnOHQOATNmEHjhBVSuWEF7cTEAW/KqiQv2ITHM16FjO50B0xXTUIf1irAQgr9O+ivN7c0s37XcrGMmGu+4t+UdFxyyo4PSxx5HExlKxIh6SJpm9ZysZXt+NemJIWg1/eOy9GPhj2ws2Mii0YuI9u8hcl6lUrQ/F9zYWUL/+I+faXTmEzj3LkKjVjE2KeSEC0QnR412++SzHD6P6PvvBykp++fTSCnZeqSKyYPC+19Hsr4YMA06Wo93e7OSIaFDuHHkjazOXk1GRd/F7KICfRgU6c/WLutc8+67tB08SPTsNFRewi4Jg5bQ1NbBvuJ6Jg3sH/6Blo4Wntz6JIOCBzFv5Lzedx44Qwn+cPKNnSV4BIE7EjMaNL4uUScnJodxsLSe+taTGqbn/QS+YRA5wuFz8IqPJ2LxbTRs2ED2F99S2ahjyqD+cYGwCFNgwFHzai31xuIxi4nyjeKJLU+Y5TiePDCc7fnV6A2S9tJSKpa/gP/ZMwgMK4SYNMV05UR2HatBb5D9xlH8WuZrFDUW8dCUh/quLmq6ecrf7PiJWYlHELgjGq2SfXrMNvuxNUxMDsMgu6lHk79Z+UI7ofokQNgtt6BNTqb5n//Au0PH5NPJP2DCP1wRrDb6CQD8vfy5b9J9HKg+wPsH3+9z/8kDw2ho7eBAST1l/3gKqdcT88D9iKIdisnKyWzPq0Yl6BeZ40fqjvDmvje5YtAVTIwxI4IuZgx4B0H+T46fnJV4BIG7knwWlGQqXaKciKmWz/au5qGao1DneP9AV1RaLTGPPIJ3RQm35m9kQLif08Z2KgOmKZnkegtbSHbDxQMuZnr8dF7Y/UKfGceTjRrWoTUbaNiwgYglS9CqKxRTlUlTcSLb8qsZFRdMgLem751diJSSJ7c82RmxZRZqjbLOeR5B4MFSBp4DSKerk/7eGkYnBPNLbpf+uKY5OME/0BW/SRP5cfAULtn/PW2Hs506ttMYMA10DVC21+ZTCSF4aPJDADy59cle+w7EBvsyJFBN/Dsvoh00iPBb5h/XQAc411Gs6zCw+1htvzALrcldw9bSrdw97m7CfS3QUpNnQHUu1Bc7bnI24BEE7kr8eKXRed4PTh/6rCERZBTWHfcT5G8Gv3CIcrx/oCt5lU28OOxSpH8Apf/3f0gnl91wCqaLrh3MQwAJgQncnn47mwo38e2xb3vdd1HutwTXVhD98MMIrVaZQ8RQpTSCE9lTUEtbh8Ht8wcqWypZtn0Z46LGce3Qa/s+oCsDjdq0m/oJPILAXdFoIXk6HNnk9KGnD4lAb5BsPWI0D5n8A06O2tlypJoGrT8+d95DS0YGNe/1bfvudwTFQehAyLfdYWzihhE3MCJsBE9ufZJ6XX23+7RkZJD663rWDZxKUdJw0LfD0V9d4h/YnF2BSsDUwc4VQJby1LanaOlo4e/T/o5KWHjpjE4Fn2DI+9Exk7MRjyBwZwaeozSKr+uxFp9DGJcUiq+Xms3ZFUo+g5P9Aya2HKkiKtCbwTdci//06ZQ/+yy6Qstr+Ls9yWcpwtYOfgIAjUrD36f9nerWap7ZfmorUKnTUfLQQ6giI3lz5Cy25lUpxQR1DTB4pl3mYAk/5VQyJjGEYF/3rSO18dhGNuRvYPGYxQwKHmT5CVRqGHCW2zqMPYLAnRlkzO50snlIq1ExeVAYm3MqXeYfkFKyNU/JH1CpVMQ+9ihCCEoe+pv5PXf7C0POh7Y6u/arHhU+ivmj5vNpzqdsLjrRHFH56mu0ZecQ/8jDhEaG8nNOJeR+r5S8GHi23eZgDnXN7WQU1DJjiPtqAw26Bh7f8jgpoSnMT51v/YkGzlBurGoL7DY3e+ERBO5M1Cil7tAR1/gJciuaaD68SZlD5HCnjp9b0URZfVtn/oBXXBxRf/4zzVu2UPvhKqfOxeEMOle5COd+Z9fTLk1fyuDgwTz8y8M06JSmP60HD1K5ciVBs2YReN65nD00kp9zqjDkfg/xE8DXueUdfj1SiUHCjKHuW1r+qW1PUdVaxaPTHsVLZYPWkuy+fgKPIHBnVMY7tCObnF7GdvqQCEAqNk0X+Ac2HVJaW5ydcvwCEXLdHPynTaX86adpL3Kuucyh+IYqwQE59hUEWrWWx6Y/RkVLBcu2L8Og01H857+gDgkm+qEHAThnaCTqtlpE8S7XmIWyKwnw1pDupq0pvz/2PWtz17IgbQGpEam2nSxqpJKU6YbmIY8gcHcGnQONpU7vTzAsOpAp/iX4tZbBkAucOjbAD4crGBIVQGLY8fwBIQSxjz0GQPEDfz29oogGnw/Fu+yeN5IWmdZpItr95F9oO3yY2MceQxOqROhMGxLO2ep9CGlwmSCYMigML7X7XYqqW6t55NdHGB42nMWjF9t+QpVKMQ/lbnS7/gTu99/3cCKDzlUenWweUqkEN4QdAkA6WRA06zrYeqSac7sxF3jFxxP94F9p3raN6jffcuq8HMqQ80EaHBIltjR9KTNr4/D58Ct8rr6CwHPP7XwvyMeL2UGHaBT+dutDbS7Hqpo5Vt3MjBT3MwtJKXl8y+M06Bp44qwn+i4jYS4pF0FDMZTanjdiTzyCwN0JTYaQAS4JI52i38leQzKHm+3Tn9hcfsmpQqc3cO6wqG7fD/7Nbwi88ELKn3+e1gMHnDo3hxE3TgkvtLOfAEDd0s7itToqgwT/PqvhRGe7lEwy7OGnjpGUN9snaslcfspReo+fleJ+juLPj3zON0e/YWn6UoaGDrXfiVMuUh6zN9jvnHbAJkEghAgTQnwjhMg2Pp6SESKEOE8IsafLX6sQYrbxvbeEEHld3ku3ZT6nLYPOVeyKHbq+9rQfzdVE1Ozhe8NYJXrIiWw6XI6fVt1ZLvlkhBDEPPoImpAQiu67D0NrD410+hNqjbLOOd/b3WxQ9thjUFJOzX038k3lZj46/NHxNyuzCWwr5SfDaH467Nx1/ulwJXHBPgyK8HfquH2RX5fP41seZ3z0eOaPsiFKqDsCohShf/g0EgTA/cB3UsoU4Dvj6xOQUm6UUqZLKdOBmUAz8HWXXe4zvS+l3GPjfE5Phl0KbfXOdTLlfo+QBg4HTe103DoDKSUbD1YwbXAE3hp1j/tpQkOJffJJdDm5lD/9tNPm51AGn6+YDSoO2u2UdWvWULdmDRFLl3LlNfczLW4ay7Yv40idsXVi7vcAZPmM54fDFXYbty/0BskvuZXMSIl0q/LiOr2OP//4Z7RqLU/NeOrU1pP2YOjFSt5Gk3MFb2/YKgiuAkytpN4GZvex/7XAeills43jnlkMOhe8/ODQl84bM/tr8AsncdR0fs2toq6lve9j7EBuRSNFtS2cO6xvu3HAjLMImzePmvfep379eifMzsEMOV95tFP0kC4/n5JHHsVvwgQilixGJVQ8Pv1xfDW+/OmHP9HS0aIIgrDBDBk2ip+yK9AbnOPE3FNQQ31rh9uZhZbvWs6B6gM8Ou1RYvxjHDPI0IsBCTm9lwBxJrYKgmgppanMYSnQQ5ueTuYCJ9cJeEIIkSmEeE4I4d3TgUKIRUKIHUKIHRUVzrtzcQu8fJWIjoNfOifawKBXvqRDLuDC1Hg6DNJpWsGmQ8ramiMIAKLuvQefMaMpeehv6PLzHTgzJxCcABHD7OInMOh0FN1zLyovL+KeWYZQK3e2kX6RPDXjKXJqcnjyl0cVLXPwTM4ZGklNczt7T2pf6Si+yirFSy04x8x1dgY/FPzAO/vfYe6wucxMcmAEVcwYCIiGw185bgwL6VMQCCG+FUJkdfN3Vdf9pOKB6vEqJYSIBdKArsaxB4DhwEQgDPhLT8dLKV+VUk6QUk6IjHSfL4/TGH65YjawsZuVWRTtguYqSLmIsYkhRAV6s2Gfc7orbTxUTkpUAAmh5pWdFlotCc8+CxoNhX+8B0Ob9W0f3YKUC5WEo1bbLshljz9B6/79xP7jSbxiTryznRY/jUWjF/FZ3jo+9RYw4nKjiQanCHwpJeuzSjlrSARBPu5RVuJY/TEe+OkBRoSNML+8tLWoVMo653yv1HhyA/oUBFLKC6SUqd38rQHKjBd404W+t2/RdcCnUsrOTy6lLJEKbcCbgHP74/Unhl4MQg0Hv3D8WNkblEzXwTNRqQQXjoxm06EKWtv77nxlC01tHWzPqzFbGzDhFR9P3FP/oO3AAcoef7x/l6AYORv0Ojhkvamr5qOPqF21ivBFiwic2f2d7ZIxS5isCuKJiDAOBUUR5q9lXFIoG/aVWT2uuewrrqewpoVLUh1kerGQlo4W/rjpjwghePbcZ/HR+Dh+0JSLlbIiBVsdP5YZ2GoaWguYGnbOA9b0su/1nGQW6iJEBIp/IcvG+Zy++IUpJYud4Sc4vAESJiljAhePiqFZp2dztmOdW98fLEenNzBzeF8WxlMJPO88whctovajj6l5vx9XKU2YAMGJsO9Tqw5vycyk7NHH8J82jci77uxxP7Vex1NFRwlSeXPXD/dQ01rDrLRYDpTUk1PeaO3szeKrrFLUKsGFI10vCKSUPPrro2TXZPPUjKdICExwzsCDzwOVl9uYh2wVBE8BFwohsoELjK8RQkwQQrxu2kkIkQwkAidnRb0rhNgL7AUigMdtnM/pzbDLoHw/VB9x3BhVuVCaqUQqGZkyKJxAH43DzUNr9hQTHeRtdQPzyLvuJOCccyh78h80bd1m59k5CSFg5FWKE7eltu/9u9BRVUXhnXehiYwk7l/PdPoFuiX3eyJaG3g+bSkVzRXc+8O9XJyqmIfWZTq2ecr6rBImDwwjzF/r0HHM4d0D77LuyDqWpC9hRoITK+x6Bypl5g+td4ssY5sEgZSySkp5vpQyxWhCqjZu3yGlXNhlv3wpZbyU0nDS8TOllGlGU9ONUkrH3or0d4ZfpjwedKBWsPcjQEDa8cYbWo2K84dH8e2BMjr0jinrUNfczg+Hy7l8dBxqlXXhhEKtJu6ZZWiTkii6+250hf20HtGoqy02DxlaWylYuhR9bS3x/36hs4REj+xfA76hjB59Ew9Pe5jtpdt58+DzTEwOY11micPMa9llDeRWNHGpG5iFfiz8kWU7ljEzcSa3jb7N+RMYOVspM+8Mv18feDKL+xOhyUqDC0f5CaSEzFVKkbngE1Xki0bFUNPczvb8GocMvT6rhHa95Kr0OJvOow4MJPHll5B6PYVLlqCv774xi1sTP94i85A0GCj+y/20Zu4lbtnT+I4a1fsBHW2KkBk+C9ReXDH4CuanzufDQx8Sn7SLnPJGDpU12OGDnMpXWYpWedEo1wqCQ9WHuO+H+xgWOox/zPiH5Y1m7MGo2aDWQuaHzh/7JDyCoL8x/HI49ivUOaBBS/Eupa9q2pxT3jpnaCRajcph5qG1GcUkh/uRFh9s87m0yckkLH+etvx8Cm+/A4POiRnZ9sBC81DFc8/TsGEDUffdR9CFF/Z9/tyNSoLiyONpP3eNvYtzE87lu/KVeAVm8XmGY8xD67NKGT8glOggJzhke6CiuYLbv7udAG0AL57/In5e5kWo2R3fUBh6Cez92OXRQx5B0N9Ivx6QsPtd+587c5VyhzLyqlPe8vfWMHNYFJ9nFKPrsK95qKy+lV+PVHFlerzdskz9p04l7sknad6+neK//KX/VSod9RswtPcZHFDz/vtUvfYaIb/9LWHzbzbv3PvXKHWNBp7TuUmtUvP0OU+TFpmGb/wHfHpgs93NQ8eqmtlfUu9Ss1BdWx23fXsb9bp6Xpz5IlF+3dezchpj5kJzZWeGt6vwCIL+Rmiykmm8+79K4pe90HdA1idKmGoPzUnmTkqkqknH1/vtqxUoNmm4coxtZqGTCb7icqLuu4+G9V9R9o+n+ldYafw4CE6CfZ/1uEvdmjWUPvIoAeedR8xDD5onRNtb4dAXMGyW0he7C74aX16a+RJh3jHUB61k/SH7VnxZtaMAlYBL02Ltel5zaW5vZul3S8mvy2f5ecsZET7CJfM4gSEXKj0KMj5w6TQ8gqA/Mu4mqCuwb0XSI5ugqQJG/7bHXWakRBIf4st7W4/Zb1xg7Z4iRsYGMSTK/lVOw26ZT9i8m6j573+p+Ne/+o8wEEKxIed+B/Ulp7xd/803FP/1QfymTCH++ecQXmYmZmV9oiSrjZnb7dshPiG8csErYNDy921/4EitfSLUdB0GPth+jJnDo4kP8bXLOS0aX6/j7o13k1WZxdNnP83UuKlOn0O3aLSQ+htF82t1nT/LIwj6I8MvV+yLu96x3zn3rlLMBaYyud2gVgmun5TIL7lV5FU22WXYvMomMgrrbHYS94QQgqj77yfk+rlUvf4GFc8v7z/CYMJ8Revb8cYJmxt/+IHie+7FNzWVxJdeROXdY2WWE5EStq1U2o720pt4eOQA0r3+QmuHgfkbbrGLMPhqXymVjTp+P3WAzeeylDZ9G/dsuodfS37lkWmPcMEA5zda6pXRc6GjFQ6sddkUPIKgP6LxhjHXK9FD9qhg2FIDB9YpvgFN7xeVORMSUasEH2yzj1bw5s95eKkFs8fG2+V83SGEIOZvfyPkuuuoWrmSiuX9RBiEDVKciTveVEw6QP1XX1Fwxx/wTkkh8dWVqPwtKOFcuB1KMmDSrX22Hr39rOk05d9KW7ueW+wgDP7361EGhPs5vUl9c3szd3x3Bz8U/sDfpvyN2UP6qovpAhImQNhgl5qHPIKgvzLuJsWZaI8vz7bXob0JJvUdSx0d5MMFI6L4aGchbR22+SiqGttYtaOA2enxDo8iESoVMQ//nZA511L1ykpKH30UqXdsyQy7MGWx4kzM+pja1Z9SdM+9+KalkfT2W6iDLYyw2roSvIOVO9C+hh0UxsjIIXhX3g7AzV/dzN4K67pqHSytZ1t+NTdMTkJlZY6INTTqGlny7RK2lW7j8emPc92w65w2tkUIAWNvUAoAlmS4ZAoeQdBfiRqhlIHY9bZtmYm6JtjysnLnGWNec+7fTR5AdZOOr22sS/POr0dpbTdw2zmDbDqPuQiViphHHyX81oXUvv8BRXf/0f2L1A08Bxk5gqqXnqPkr3/Ff8oUkl5/DXVgoGXnaSiF/Z8pFxzvvn0xQggWnjWIo6WBLBn6L/y8/Fjw9QJ+KLC8Zeq7W46h1aiYMz7R4mOtpbSplJu/upnMikz+efY/uWrIqZFwbsXEhYqQ/nGZS4b3CIL+zIRblKb2ttgWd74NLdUww/yKizOGRJAQ6svbv+RbbWJp1nXw9q/5XDAimiFRFl7UbEAIQdS99xL91wdo+OYbjt2ygI5K92kQcjKG9nZKMhIp39xM4IzxJLyyApWfFXHvO95U/A0TF/a9r5HL0mKJCfJh7c52/nfZ/xgYPJA7N97JqkOrzD5HY1sHq3cVcsXoOEKdVFIiqzKL67+4nsLGQv59/r+5JPkSp4xrEz7BMPk2OPA5lO13+vAeQdCfSZujOP6+fcS6hJSONvjl35A8AxLNL/yqUgluO3sQO47WWF2tctX2Amqb21nsJG3gZMJuuon4556ldd8+8q65lpY97tccr6OqimPzb6Huxywi0tuJnylRaa24mLa3ws43ldLH4YPNPkyrUTFvWjI/51RRXqPlzYvfZFrcNB7b8hh//+XvtHb03SL07V/yadLpneYkXp+3nvlfzcdb7c1/L/0vZ8Wf5ZRx7cKUJaANgJ+ecfrQHkHQn1Fr4IJHlGzgnW9ZfnzGB0qPgxn3WHzo9ZOSSIkK4B/rD1jsK+jQG3jtpzwmDAhlQrJ1BebsQdCll5L8/nsIrZb8399E9XvvuY0TuXHzzxyZPZvWrCzin/0XkQt/jzj8JRTttPxkm5+FxjKY1nM10p743aQk/LRqVv6Yi5+XHy/OfJFb025ldfZqblp/EwUNBT0eW1zbwovf53DxqGjSE7vPTbEXze3NPPzLw/z5xz8zPGw47172LimhKSwv1+4AAA1HSURBVA4d0+74hSkaW9ZqqMx26tAeQdDfGXoxDDgLNj0FbRbUh+nQwebnIG4sDDrP4mE1ahUPzhrB0apm3vnlqEXHvrv1GEW1Ldx2jvl3p47CZ8QIBn78Ef7TplL26GMU3HYb7aXOacLTHQadjrJ/PEXBwoWog4NJXvUhQZddBmfdrXS1WnunZdpfZbayzmlzYKDl1TWD/by4eVoya/YU8+PhCtQqNXeOu5OXzn+JosYi5nw+h1WHVmGQp2ZuP/HFAQxS8tCskRaPawkHqw8y94u5rM5ezcK0hfznkv8Q7hvu0DEdxtQ7QOMDP/3LqcN6BEF/Rwi48FElsuTnF8w/7pv/g5o8OO/BPkMJe+LcYVGcMzSSF77PpqrRPKdrdlkDT355gHOGRnLBCBen9xtRBweTuGIF0Q89RPP2HRy5/ApqPvrI6WUpGjf/TN5Vs6l++21Cb7iBgR9/jM+wYcqbPsEw619QlgW/mLnOUsIX94DGFy56wup53Xl+CoMj/bn/k0zqWxUhdHbC2ay6YhWp4ak8tuUxFn69kIL649rBzzmVfLG3hNvPG0JimGNq+TTqGnl6+9PMXTeXRl0jr170KneNuwsvlXt0PbOKgEiYuEApRJe70WnDegTB6UDCeKU2za8vQtm+vvc/sA62roDJixW7sQ08NGsEzTo9//rmcJ/7tnXoufODPQR4a1g2Z7Td6grZA6FSEXbjDQxauwafkSMp/dv/kXfttTT+/LPDx9YVFFD4hzspWLgQadCT+NprxPztIVQ+J4XUDp+l5Hps+idU5vR94swPIe9HuODvEGh5sx8TPl5qnpkzhtL6Vp5Yd6Bze3xAPK9d9BoPT32YA1UHmL1mNsu2L6O8qYq/r91HUpgfi862vw9Ib9Dzee7nXPnZlfxv//+4OuVqVl+5mimxU+w+lks49wGIGAqfLHBMccluEO5iE7WECRMmyB07drh6Gu5FXSG8foESGXLLVz07BWuOwsoZSrLSLRv6TCAzh0c/389/fs7jTxcN5Y6ZPdtlH1+3n9c35/HGvAmcP8L6C5OjkQYD9V98QcVzz9NeXIz/tKmE3bIA/2lTESr73Tu1Hj5M1WuvU//llwitlojFiwmbf3PvDuGGMnhpolKO/PefnVIvqJOqXHjjIggbCLd8rfTJtZF/fnWQFZtyeXP+RM4bdqI2V9ZUxkt7XmJN7hpUUktTxXSeuvA2fjPGfvV82vXtrDuyjjey3uBo/VFGhI3goSkPMTpytN3GcBsqs+HV8yByGMz/0i6/UwAhxE4p5YRTttsiCIQQc4CHgRHAJCllt1dnIcQlwHJADbwupTR1MhsIfACEAzuB30sp+6wZ7BEEPVB+EN66DLz8FGFwUk8BGsrg/blKM4zbflQuEnagQ2/gTx9l8NmeYv54wVDuuuBEYaA3SN7+JZ9H1+3n91MG8Nhs8/IVXI1Bp6PmvfeoevU19NXVaJOTCb1+LoEXX3xKQ3hz0dfX0/DNt9R/8QVNv/yC8PMj9LrrCJt/M17RZgrHPe/BZ0uUPJI5b0HwSVnZeT/CqpuU5zd/CdH2sdG3dei5/IXNlNa18tjs1FOywVvb9Sx87wt21L+HV9A+NELDeUnnce3Qa5kYM9Eqk42UkkM1h1iXu44v876koqWCEWEjWDR6ETOTZrqmj4Cz2L9GWcfxN8Nl/1KCQ2zEUYJgBGAAVgJ/6k4QCCHUwGHgQqAQ2A5cL6XcL4RYBayWUn4ghHgFyJBSruhrXI8g6IXiPfD2FeAXDlOWwqBzICgOfnlRCRXVtykXjxFX2HVYvUFy38cZrN5VxE1TB3DxqBhGxgZRWNPCg5/tJbOwjrOHRrLyxvH4antpoeiGGHQ6Gr76ipp336MlQ8n89B42jICzz8YnLRXvIUPQJiaeUvhN6vXoq6tpPXiIlowMWnbvpnnbNmR7O16JiQRfPZvQ66/vu5tYd+z7FNYYHYtXvwIRKUoAwJFNsOEBpWTB7z5QND87UljTzF0f7GHn0RquHBPHg7NG0KLTU97QxrINB9lxtIa/Xz6Sc1MFHx/+mM9yP6OurQ5/L38mx0xmatxUUkJTGBA0gHCf8FPMg83tzZQ0lbC3ci97yvews2wn+fX5aISGsxLOYs7QOcyIn+FWZkWH8s3/wc/LlfU8+0+K019tvQ/EIYKgy8k30bMgmAo8LKW82Pj6AeNbTwEVQIyUsuPk/XrDIwj64NgW+PQ2qMlXXqs0YOhQGpGc/38WxZJbgt4geeizLN4/qQ5RZKA3f7t8JFeMju33P+C2nBwaN22i8Ycfad61C0xlKjQa1AEBCF9fVFot+qYm9NXVYHI4C4F3Sgr+U6cQNGsWPmlptv8vKg7DhzdC5aETt6dcBNe8rjiYHUCH3sCKTbk8/102esPx64dWreLZ347h8tHHCwi26dvYXLiZn4t/5ueinyluOt7wxt/LHz+NH14qL9QqNTWtNTS2H+9WG6gNJD0ynXMSzuHi5IsJ8XFsCKpbIiUcXAc/PK30Eg8ZANe/D9F9dKHrAVcKgmuBS0w9jIUQvwcmo5iUtkgphxi3JwLrpZTd2g2EEIuARQBJSUnjjx61LGTxjKQ6D/J+UDIVR1+nFLdyAlWNbRwoaWB/SR26DgO/n5pMsG8/juToAUNzM21H8mjLyUZ3JA9DYwOG1jZkaysqf3/UEeFowiPwHjIYn9RU1AH2L7NNW6OSWS6lYkf2CYbBM0HleK0rq6iOzTmVRAR4ExXoTUp0ALHBPZeYllJS3FRMXl0e+XX5FDYW0trRSruhnXZDO2E+YUT6RhLlF8WIsBEMChl0ept+LEFKOLwBtr8O170DWusisawWBEKIb4HuDKIPSinXGPfZhIMFQVc8GoEHDx48WE5PgqBP74OU0tbi3UVA12pTCcZtVUCIEEIjpezost2DBw8ePDgRZ+hd24EUIcRAIYQWmAuslYoqshG41rjfPGCNE+bjwYMHDx66YJMgEEJcLYQoBKYCXwghNhi3xwkhvgQw3u3fAWwADgCrpJSmrKe/APcIIXJQQkjfOHkMDx48ePDgWDwJZR48ePBwhtCTj8DjkvfgwYOHMxyPIPDgwYOHMxyPIPDgwYOHMxyPIPDgwYOHM5x+6SwWQlQA1qYWRwDu26TWcZyJn/tM/MxwZn5uz2c2jwFSysiTN/ZLQWALQogd3XnNT3fOxM99Jn5mODM/t+cz24bHNOTBgwcPZzgeQeDBgwcPZzhnoiB41dUTcBFn4uc+Ez8znJmf2/OZbeCM8xF48ODBg4cTORM1Ag8ePHjw0AWPIPDgwYOHM5wzShAIIS4RQhwSQuQIIe539XwcgRAiUQixUQixXwixTwhxl3F7mBDiGyFEtvHRika57o0QQi2E2C2EWGd8PVAIsdW43h8ay6CfVgghQoQQHwshDgohDgghpp7uay2E+KPxu50lhHhfCOFzOq61EOI/QohyIURWl23drq1QeMH4+TOFEOMsGeuMEQRCCDXwEnApMBK4Xggx0rWzcggdwL1SypHAFOB24+e8H/hOSpkCfGd8fbpxF0qpcxP/BJ4zdsGrARa4ZFaOZTnwlZRyODAG5fOftmsthIgH7gQmGLsZqlF6nJyOa/0WcMlJ23pa20uBFOPfImCFJQOdMYIAmATkSCmPSCl1wAfAVS6ek92RUpZIKXcZnzegXBjiUT7r28bd3gZmu2aGjkEIkQDMAl43vhbATOBj4y6n42cOBs7G2MdDSqmTUtZymq81SmdFXyGEBvADSjgN11pK+SNQfdLmntb2KuAdqbAFpftjrLljnUmCIB4o6PK60LjttEUIkQyMBbYC0VLKEuNbpUC0i6blKJ4H/gwYjK/DgVpjYyQ4Pdd7IFABvGk0ib0uhPDnNF5rKWUR8AxwDEUA1AE7Of3X2kRPa2vT9e1MEgRnFEKIAOAT4G4pZX3X94xtQk+buGEhxOVAuZRyp6vn4mQ0wDhghZRyLNDESWag03CtQ1HufgcCcYA/p5pPzgjsubZnkiAoAhK7vE4wbjvtEEJ4oQiBd6WUq42by0yqovGx3FXzcwDTgSuFEPkoJr+ZKLbzEKP5AE7P9S4ECqWUW42vP0YRDKfzWl8A5EkpK6SU7cBqlPU/3dfaRE9ra9P17UwSBNuBFGN0gRbFwbTWxXOyO0bb+BvAASnls13eWgvMMz6fB6xx9twchZTyASllgpQyGWVdv5dS3gBsBK417nZafWYAKWUpUCCEGGbcdD6wn9N4rVFMQlOEEH7G77rpM5/Wa92FntZ2LXCTMXpoClDXxYTUN1LKM+YPuAw4DOQCD7p6Pg76jGehqIuZwB7j32UoNvPvgGzgWyDM1XN10Oc/F1hnfD4I2AbkAB8B3q6enwM+bzqww7jenwGhp/taA48AB4Es4L+A9+m41sD7KH6QdhTtb0FPawsIlKjIXGAvSlSV2WN5Skx48ODBwxnOmWQa8uDBgwcP3eARBB48ePBwhuMRBB48ePBwhuMRBB48ePBwhuMRBB48ePBwhuMRBB48ePBwhuMRBB48ePBwhvP/eytfs+HAgbkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pe = PositionalEncoding(20)\n",
    "pe.initialize()\n",
    "X = nd.zeros((1, 100, 20))\n",
    "Y = pe(X)\n",
    "_ = plt.plot(np.arange(100), Y.asnumpy()[0, :, 4:8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformer Encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<img src=\"../img/self-attention.svg\" width=\"33%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Now we define the transformer block for the encoder, which contains a multi-head attention layer, a position-wise feed-forward network, and two connection blocks.\n",
    "\n",
    "Due to the residual connections, this block will not change the input shape. It means the `units` argument should be equal to the input's last dimension size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The encoder stacks $n$ blocks. Due to the residual connection again, the embedding layer size $d$ is same as the transformer block output size. Also note that we multiple the embedding output by $\\sqrt{d}$ to avoid its values are too small compared to positional encodings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:51.588232Z",
     "start_time": "2019-08-06T01:19:51.561649Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "14"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3, 24)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class EncoderBlock(nn.Block):\n",
    "    def __init__(self, units, hidden_size, num_heads, dropout, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.attention = MultiHeadAttention(units, num_heads, dropout)\n",
    "        self.add_1 = AddNorm(dropout)\n",
    "        self.ffn = PositionWiseFFN(units, hidden_size)\n",
    "        self.add_2 = AddNorm(dropout)\n",
    "        \n",
    "    def forward(self, X, mask):\n",
    "        Y = self.add_1(X, self.attention(X, X, X, mask))\n",
    "        return self.add_2(Y, self.ffn(Y))\n",
    "    \n",
    "class TransformerEncoder(nn.Block):\n",
    "    def __init__(self, vocab_size, units, hidden_size, num_heads, num_layers, dropout, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.units = units\n",
    "        self.embed = nn.Embedding(vocab_size, units)\n",
    "        self.pos_encoding = PositionalEncoding(units, dropout)\n",
    "        self.blks = nn.Sequential()\n",
    "        for i in range(num_layers): self.blks.add(EncoderBlock(units, hidden_size, num_heads, dropout))\n",
    "\n",
    "    def forward(self, X, mask, *args):\n",
    "        X = self.pos_encoding(self.embed(X) * math.sqrt(self.units))\n",
    "        for blk in self.blks: X = blk(X, mask)\n",
    "        return X\n",
    "    \n",
    "encoder = TransformerEncoder(vocab_size=200, units=24, hidden_size=48, num_heads=8, num_layers=2, dropout=0.5)\n",
    "encoder.initialize()\n",
    "encoder(nd.ones((2, 3)), mask).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformer-Decoder\n",
    "\n",
    "<center>\n",
    "<img src=\"../img/self-attention-predict.svg\" width=\"50%\"/>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Let first look at how a decoder behaviors during predicting. Similar to the seq2seq model, we call $T$ forwards to generate a $T$ length sequence. At time step $t$, assume $\\mathbf x_t$ is the current input, i.e. the query. Then keys and values of the self-attention layer consist of the current query with all past queries $\\mathbf x_1, \\ldots, \\mathbf x_{t-1}$.\n",
    "\n",
    "During training, because the output for the $t$-query could depend all $T$ key-value pairs, which results in an inconsistent behavior than prediction. We can eliminate it by specifying the valid length to be $t$ for the $t$-th query.\n",
    "\n",
    "Another difference compared to the encoder transformer block is that the decoder block has an additional multi-head attention layer that accepts the encoder outputs as keys and values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:53.525380Z",
     "start_time": "2019-08-06T01:19:53.500161Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "16"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class DecoderBlock(nn.Block):\n",
    "    def __init__(self, units, hidden_size, num_heads, dropout, i, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.i = i  # i means it's the i-th block in the decoder\n",
    "        self.attn1, self.attn2 = [MultiHeadAttention(units, num_heads, dropout) for _ in range(2)]\n",
    "        self.addnorm1, self.addnorm2, self.addnorm3 = [AddNorm(dropout) for _ in range(3)]\n",
    "        self.ffn = PositionWiseFFN(units, hidden_size)\n",
    "\n",
    "    def forward(self, X, state, dec_mask=None):  # key_values contains the past queries for this block\n",
    "        enc_outputs, enc_mask, key_values = state\n",
    "        key_values[self.i] = nd.concat(key_values[self.i], X, dim=1)  # update key_values\n",
    "        X2 = self.attn1(X, key_values[self.i], key_values[self.i], dec_mask)\n",
    "        Y = self.addnorm1(X, X2)\n",
    "        Y2 = self.attn2(Y, enc_outputs, enc_outputs, enc_mask)\n",
    "        Z = self.addnorm2(Y, Y2)\n",
    "        return self.addnorm3(Z, self.ffn(Z)), [enc_outputs, enc_mask, key_values]\n",
    "    \n",
    "class TransformerDecoder(nn.Block):\n",
    "    def __init__(self, vocab_size, units, hidden_size, num_heads, num_layers, dropout, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.units = units\n",
    "        self.embed = nn.Embedding(vocab_size, units)\n",
    "        self.pos_encoding = PositionalEncoding(units, dropout)\n",
    "        self.blks = [DecoderBlock(units, hidden_size, num_heads, dropout, i) for i in range(num_layers)]\n",
    "        self.dense = nn.Dense(vocab_size, flatten=False)\n",
    "\n",
    "    def forward(self, X, state):\n",
    "        X = self.pos_encoding(self.embed(X) * math.sqrt(self.units))\n",
    "        for blk in self.blks: X, state = blk(X, state)\n",
    "        return self.dense(X), state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Use the Pretrained Transformer model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:19:58.598851Z",
     "start_time": "2019-08-06T01:19:56.864366Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#Source Vocab: 36794 , #Target Vocab: 36794\n",
      "Sample BPE tokens: ('At least one h@@ ac@@ ktiv@@ ist group online claimed that they were responsible for bringing down the N@@ SA site with a D@@ Do@@ S attack .', 'Min@@ destens eine Gruppe von H@@ ac@@ kti@@ visten nahm für sich in Anspruch , die N@@ SA@@ -@@ Website mit einem D@@ Do@@ S-@@ Angriff la@@ h@@ m@@ gelegt zu haben .')\n"
     ]
    }
   ],
   "source": [
    "with mx.np_shape(False):\n",
    "    wmt_transformer_model, wmt_src_vocab, wmt_tgt_vocab = \\\n",
    "        nlp.model.get_model('transformer_en_de_512',\n",
    "                            dataset_name='WMT2014',\n",
    "                            pretrained=True,\n",
    "                            ctx=ctx)\n",
    "\n",
    "# This model is based on a mixed EN-DE vocabulary, thus the src and tgt vocabs are of equal size\n",
    "print('#Source Vocab:', len(wmt_src_vocab), ', #Target Vocab:', len(wmt_tgt_vocab))\n",
    "\n",
    "wmt_data_test = nlp.data.WMT2014BPE('newstest2014', src_lang='en', tgt_lang='de')\n",
    "print('Sample BPE tokens:', wmt_data_test[1337])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T01:20:00.162682Z",
     "start_time": "2019-08-06T01:20:00.156501Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NMTModel(\n",
      "  (encoder): TransformerEncoder(\n",
      "    (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "    (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "    (transformer_cells): HybridSequential(\n",
      "      (0): TransformerEncoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (1): TransformerEncoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (2): TransformerEncoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (3): TransformerEncoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (4): TransformerEncoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (5): TransformerEncoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (decoder): TransformerDecoder(\n",
      "    (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "    (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "    (transformer_cells): HybridSequential(\n",
      "      (0): TransformerDecoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell_in): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (attention_cell_inter): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj_in): Dense(512 -> 512, linear)\n",
      "        (proj_inter): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm_in): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        (layer_norm_inter): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (1): TransformerDecoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell_in): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (attention_cell_inter): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj_in): Dense(512 -> 512, linear)\n",
      "        (proj_inter): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm_in): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        (layer_norm_inter): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (2): TransformerDecoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell_in): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (attention_cell_inter): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj_in): Dense(512 -> 512, linear)\n",
      "        (proj_inter): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm_in): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        (layer_norm_inter): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (3): TransformerDecoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell_in): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (attention_cell_inter): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj_in): Dense(512 -> 512, linear)\n",
      "        (proj_inter): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm_in): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        (layer_norm_inter): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (4): TransformerDecoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell_in): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (attention_cell_inter): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj_in): Dense(512 -> 512, linear)\n",
      "        (proj_inter): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm_in): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        (layer_norm_inter): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "      (5): TransformerDecoderCell(\n",
      "        (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "        (attention_cell_in): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (attention_cell_inter): MultiHeadAttentionCell(\n",
      "          (_base_cell): DotProductAttentionCell(\n",
      "            (_dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          )\n",
      "          (proj_query): Dense(512 -> 512, linear)\n",
      "          (proj_key): Dense(512 -> 512, linear)\n",
      "          (proj_value): Dense(512 -> 512, linear)\n",
      "        )\n",
      "        (proj_in): Dense(512 -> 512, linear)\n",
      "        (proj_inter): Dense(512 -> 512, linear)\n",
      "        (ffn): PositionwiseFFN(\n",
      "          (ffn_1): Dense(512 -> 2048, linear)\n",
      "          (activation): Activation(relu)\n",
      "          (ffn_2): Dense(2048 -> 512, linear)\n",
      "          (dropout_layer): Dropout(p = 0.1, axes=())\n",
      "          (layer_norm): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        )\n",
      "        (layer_norm_in): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "        (layer_norm_inter): LayerNorm(eps=1e-05, axis=-1, center=True, scale=True, in_channels=512)\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (src_embed): HybridSequential(\n",
      "    (0): Embedding(36794 -> 512, float32)\n",
      "    (1): Dropout(p = 0.0, axes=())\n",
      "  )\n",
      "  (tgt_embed): HybridSequential(\n",
      "    (0): Embedding(36794 -> 512, float32)\n",
      "    (1): Dropout(p = 0.0, axes=())\n",
      "  )\n",
      "  (tgt_proj): Dense(512 -> 36794, linear)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(wmt_transformer_model) # Print the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Translation Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Translating English to German:\n",
      " ['We', 'love', 'natural', 'language', 'processing', '.', '<eos>']\n",
      "The German translation is:\n",
      " ['<bos>', 'Wir', 'lieben', 'die', 'natürliche', 'Sprach@@', 'verarbeitung', '.', '<eos>']\n"
     ]
    }
   ],
   "source": [
    "from gluonnlp.model import BeamSearchScorer, BeamSearchSampler\n",
    "import nmt\n",
    "\n",
    "def decode_logprob(step_input, states):\n",
    "    out, states, _ = wmt_transformer_model.decode_step(step_input, states)\n",
    "    return nd.log_softmax(out), states\n",
    "\n",
    "sampler = BeamSearchSampler(decoder=decode_logprob, beam_size=4, max_length=200,\n",
    "    eos_id=wmt_tgt_vocab.token_to_idx[wmt_tgt_vocab.eos_token],\n",
    "    scorer=nlp.model.BeamSearchScorer(alpha=0.6, K=5))\n",
    "\n",
    "def translate(src_sentence, bos_idx):\n",
    "    encoder_outputs, _ = wmt_transformer_model.encode(nd.array([src_sentence], dtype=np.int32, ctx=ctx))\n",
    "    decoder_states = wmt_transformer_model.decoder.init_state_from_encoder(encoder_outputs)\n",
    "    samples, scores, valid_length = sampler(nd.array([bos_idx], ctx=ctx, dtype=np.float32), decoder_states)\n",
    "    return samples[0, 0, :valid_length[0, 0].asscalar()]  # sample with max score\n",
    "\n",
    "sample_sentence = 'We love natural language processing .'.split() + [wmt_src_vocab.eos_token]\n",
    "print('Translating English to German:\\n', sample_sentence)\n",
    "sample_src_seq = wmt_src_vocab[sample_sentence]\n",
    "sample_tgt_seq = translate(sample_src_seq, wmt_tgt_vocab[wmt_tgt_vocab.bos_token])\n",
    "sample_translation = [wmt_tgt_vocab.idx_to_token[ele] for ele in sample_tgt_seq.asnumpy()]\n",
    "print('The German translation is:\\n', sample_translation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you'd like to train your own machine translation models, check the GluonNLP [Model Zoo](http://gluon-nlp.mxnet.io/model_zoo/machine_translation/index.html) or the source on Github: [github.com/dmlc/gluon-nlp](https://github.com/dmlc/gluon-nlp/tree/master/scripts/machine_translation)\n",
    "\n",
    "## References\n",
    "\n",
    "[1] Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in Neural Information Processing Systems. 2017."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}