Notebooks/Texts/transformer_tutorial/transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Transformer Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prelims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import math, copy, time\n",
    "from torch.autograd import Variable\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn\n",
    "seaborn.set_context(context=\"talk\")\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background\n",
    "\n",
    "The goal of reducing sequential computation also forms the foundation of the Extended Neural GPU, ByteNet and ConvS2S, all of which use convolutional neural networks as basic building block, computing hidden representations in parallel for all input and output positions.\n",
    "\n",
    "In these models, the number of operations required to relate signals from two arbitrary input or output positions grows in the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes it more difficult to learn dependencies between distant positions. In the Transformer this is reduced to a constant number of operations, albeit at the cost of reduced effective resolution due to averaging attention-weighted positions, an effect we counteract with Multi-Head Attention.\n",
    "\n",
    "Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations. End- to-end memory networks are based on a recurrent attention mechanism instead of sequencealigned recurrence and have been shown to perform well on simple- language question answering and language modeling tasks.\n",
    "\n",
    "To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence aligned RNNs or convolution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most competitive neural sequence transduction models have an encoder-decoder structure. Here, the encoder maps an input sequence of symbol representations $z=(z_1, ..., z_n)$. Given $z$, the deconder then generates an output sequence $(y_1, ..., y_m)$ of symbols one element at a time. At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class EncoderDecoder(nn.Module):\n",
    "    \"\"\"\n",
    "    A standard Encoder-Decoder architecture.\n",
    "    Base for this and many other models.\n",
    "    \"\"\"\n",
    "    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):\n",
    "        super(EncoderDecoder, self).__init__()\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.src_embed = src_embed\n",
    "        self.tgt_embed = tgt_embed\n",
    "        self.generator = generator\n",
    "        \n",
    "    def forward(self, src, tgt, src_mask, tgt_mask):\n",
    "        \"Take in and process masked src and target sequences.\"\n",
    "        return self.decoder(self.encoder(src, src_mask), src_mask, tgt, tgt_mask)\n",
    "    \n",
    "    def encode(self, src, src_mask):\n",
    "        return self.encoder(self.src_embed(src), src_mask)\n",
    "    \n",
    "    def decode(self, memory, src_mask, tgt, tgt_mask):\n",
    "        return self.decoder(self.tgt_embed(tgt), memory, src_mask, tgt_mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Generator(nn.Module):\n",
    "    \"Define standard linear + softmax generation step.\"\n",
    "    def __init__(self, d_model, vocab):\n",
    "        super(Generator, self).__init__()\n",
    "        self.proj = nn.Linear(d_model, vocab)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return F.log_softmax(self.proj(x), dim=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder and Decoder Stacks\n",
    "\n",
    "### Encoder\n",
    "\n",
    "The Encoder is composed of a stack of $N=6$ identical layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clones(module, N):\n",
    "    \"Produce N identical layers.\"\n",
    "    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    \"Core Encoder is a stack of N layers.\"\n",
    "    def __init__(self, layer, N):\n",
    "        super(Encoder, self).__init__()\n",
    "        self.layers = clones(layer, N)\n",
    "        self.norm = LayerNorm(layer.size)\n",
    "        \n",
    "    def forward(self, x, mask):\n",
    "        \"Pass the input (and mask) through each layer in turn.\"\n",
    "        for layer in self.layers:\n",
    "            x = layer(x, mask)\n",
    "        return self.norm(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We employ a residual connection around each of the two sub-layers, followed by layer normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LayerNorm(nn.Module):\n",
    "    \"Construct a layernorm module.\"\n",
    "    def __init__(self, features, eps=1e-6):\n",
    "        super(LayerNorm, self).__init__()\n",
    "        self.a_2 = nn.Parameter(torch.ones(features))\n",
    "        self.b_2 = nn.Parameter(torch.zeros(features))\n",
    "        self.eps = eps\n",
    "        \n",
    "    def forward(self, x):\n",
    "        mean = x.mean(-1, keepdim=True)\n",
    "        std = x.std(-1, keepdim=True)\n",
    "        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is, the output of each sub-layer is `LayerNorm(x + Sublayer(x))`, where `Sublayer(x)` is the function implemented by the sub-layer itself. We apply dropout to the output of each sub-layer, before it is added to the sub-layer input and normalized.\n",
    "\n",
    "To faciliate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension $d_{model} = 512$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class SublayerConnection(nn.Module):\n",
    "    \"\"\"\n",
    "    A residual connection followed by a layer norm.\n",
    "    Note for code simplicity the norm is first as opposed to last.\n",
    "    \"\"\"\n",
    "    def __init__(self, size, dropout):\n",
    "        super(SublayerConnection, self).__init__()\n",
    "        self.norm = LayerNorm(size)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, x, sublayer):\n",
    "        \"Apply residual connection to any sublayer with the same size.\"\n",
    "        return x + self.dropout(sublayer(self.norm(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class EncoderLayer(nn.Module):\n",
    "    \"Encoder is made up of self-attention and feed-forward (defined below).\"\n",
    "    def __init__(self, size, self_attn, feed_forward, dropout):\n",
    "        super(EncoderLayer, self).__init__()\n",
    "        self.self_attn = self_attn\n",
    "        self.feed_forward = feed_forward\n",
    "        self.sublayer = clones(SublayerConnection(size, dropout), 2)\n",
    "        self.size = size\n",
    "        \n",
    "    def forward(self, x, mask):\n",
    "        x = self.sublayer[0](x, lambda x: self_attn(x, x, x, mask))\n",
    "        return self.sublayer[1](x, self.feed_forward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder\n",
    "\n",
    "The decoder is also composed of a stack of $N=6$ identical layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    \"Generic N layer decoder with masking.\"\n",
    "    def __init__(self, layer, N):\n",
    "        super(Decoder, self).__init__()\n",
    "        self.layers = clones(layer, N)\n",
    "        self.norm = LayerNorm(layer.size)\n",
    "        \n",
    "    def forward(self, x, memory, src_mask, tgt_mask):\n",
    "        for layer in self.layers:\n",
    "            x = layer(x, memory, src_mask, tgt_mask)\n",
    "        return self.norm(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. Similar to the encoder, we employ residual connections around each of the sub-layers, followed by layer normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class DecoderLayer(nn.Module):\n",
    "    \"Decoder is made of self-attn, src-attn, and feed forward (defined below)\"\n",
    "    def __init__(self, size, self_attn, src_attn, feed_forward, dropout):\n",
    "        super(DecoderLayer, self).__init__()\n",
    "        self.size = size\n",
    "        self.self_attn = self_attn\n",
    "        self.src_attn = src_attn\n",
    "        self.feed_forward = feed_forward\n",
    "        self.sublayer = clones(SublayerConnection(size, dropout), 3)\n",
    " \n",
    "    def forward(self, x, memory, src_mask, tgt_mask):\n",
    "        \"Follow Figure 1 (right) for connections.\"\n",
    "        m = memory\n",
    "        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, tgt_mask))\n",
    "        x = self.sublayer[1](x, lambda x: self.src_attn(x, m, m, src_mask))\n",
    "        return self.sublayer[2](x, self.feed_forward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position $i$ can depend only on the known outputs at positions less than $i$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def subsequent_mask(size):\n",
    "    \"Mask out subsequent positions.\"\n",
    "    attn_shape = (1, size, size)\n",
    "    subsequent_mask = np.triu(np.ones(attn_shape), k=1).astype('uint8')\n",
    "    return torch.from_numpy(subsequent_mask) == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Below the attention mask shows the position each tgt word (row) is allowed to look at (column). Words are blocked for attending to fucture words during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAE7CAYAAABUlaNfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE8xJREFUeJzt3X+s3fV93/HnK8cmzmZsFjuOlpFgGPWN1jUEwhRWKYKl\nTUqmdi0klUCAaKYtRNWYiNSWEllL1mlLTdZGmzdWkj9mKA6J1MLaLgrO8qMTXUYb2YoZnuI74dIy\nIuIf4F+RU+Lr9/64X0eHu3uvP+f4e39w/XxIX51zPt/P9/t9n6PLi8/3e77n41QVkqT5vW6pC5Ck\n1wLDUpIaGJaS1MCwlKQGhqUkNTAsJamBYSlJDQxLSWpgWEpSg1VLXUAfNr5xUJvfunrk7Saf/msL\nUI2k15ITvHy4qt50rn4rIiw3v3U1f7brrSNv9zNveecCVCPpteSr9Xt/0dLP03BJatBbWCYZJPl0\nkkNJTiT5/SQb5+l/Y5J9SU4leSbJ+/uqRZL61ufI8teBnwfeDVzatf3ubB2TXAE8BnwKWN89Pp5k\nc4/1SFJv+gzLjwDbqupAVR0Dfg24cY4AvBPYXVWPVNUrVbUT2NO1S9Ky00tYJlkPvA3Yfbatqp4F\njgPvmGWTq4b7dvZ07a3H3JBkS5Itp6eck1PSwuprZLmuezw2o/3o0LphF4/Qdy53A/uB/QcPT42w\nmSSNrq+wPNE9rp/RfgnTo8vZ+rf2nct2YAKY2LRxMMJmkjS6XsKyqo4Cfwlcc7at+xJnHfD0LJvs\nHe7bubprbz3mkaqarKrJVYOMXrQkjaDPL3g+C9yb5PIk64BtwK6qem6Wvg8D1ya5NcnqJLcC7wIe\n6rEeSepNn2H5m8AfAd8CXgAGwO0ASW5LcvJsx+7Ln5uBrUyfem8FbpojWCVpyfX2c8eqmgJ+pVtm\nrtsJ7JzR9gTwRF/Hl6SF5M8dJanBiphIY1y7vvvtkbdx8g3pwuTIUpIaGJaS1MCwlKQGhqUkNTAs\nJamBYSlJDQxLSWpgWEpSA8NSkhoYlpLUwLCUpAaGpSQ1uKAn0hjHOJNvgBNwSK91jiwlqYFhKUkN\nDEtJamBYSlIDw1KSGhiWktTAsJSkBoalJDUwLCWpQS9hmWRbkn1Jjif5bpLPJXnjPP1vSFJJTg4t\n3+yjFklaCH2NLKeA24ENwFXApcB/Ptc2VbV2aPnJnmqRpN718tvwqvr40MtDSf4D8Pk+9i1Jy8FC\nXbP8KeDpc/QZJHk+yYtJvpTkqlEOkGRDki1JtpyeqvErlaQGvc86lOSDwD8Frp+n23eAdwL7gLXA\nvcDXk/xEVX238VB3A58AOHh4avyCF8k4sxU5U5G0fPQ6skzyi8DngH9UVXvm6ldVL1bV3qo6XVVH\nq+o+4CXgAyMcbjswAUxs2jg4r7ol6Vx6C8skHwYeBH6uqr4xxi7OAGntXFVHqmqyqiZXDZo3k6Sx\n9HXr0D8H/i3wM1X1Pxr6vzfJlUlel2Rtkk8CbwZ29VGPJPWtr5HlvwPWAd8Yvnfy7Moktw2/Zvr2\noq8BJ4ADwHXA+6rq+Z7qkaRe9XXr0LznwVW1E9g59PozwGf6OLYkLQZ/7ihJDQxLSWpgWEpSA8NS\nkhoYlpLUwLCUpAaGpSQ16H0iDfVnnMk3wAk4pIXgyFKSGhiWktTAsJSkBoalJDUwLCWpgWEpSQ0M\nS0lqYFhKUgPDUpIaGJaS1MCwlKQGhqUkNTAsJamBsw6tQM5WJPXPkaUkNTAsJalBL2GZZEeSHyY5\nObT88jm2uTHJviSnkjyT5P191CJJC6HPkeVDVbV2aHlgro5JrgAeAz4FrO8eH0+yucd6JKk3S3Ua\nfiewu6oeqapXqmonsKdrl6Rlp8+w/GCSl5JMJvl0krXz9L0K2D2jbU/X3iTJhiRbkmw5PVXj1CtJ\nzfoKy+3A24GNwE3A9cDn5ul/MXBsRttRYN0Ix7wb2A/sP3h4aoTNJGl0vYRlVe2uqu9V1Zmq2gd8\nDPhQktfPsckJpq9VDrsEOD7CYbcDE8DEpo2DkWuWpFEs1DXLM91j5li/F7hmRtvVXXuTqjpSVZNV\nNblqMNdhJKkffd06dEuSS7rnPwb8FvCHVfWDOTZ5GLg2ya1JVie5FXgX8FAf9UhS3/oaWX4UOJDk\n+8BXgKeAD59dmeS2JCfPvq6qZ4Gbga1Mn3pvBW6qqud6qkeSetXLb8Or6oZzrN8J7JzR9gTwRB/H\nl6SF5s8dJamBsw7pR8aZrciZinShcGQpSQ0MS0lqYFhKUgPDUpIaGJaS1MCwlKQGhqUkNTAsJamB\nYSlJDQxLSWpgWEpSA8NSkho4kYbOyziTb4ATcOi1x5GlJDUwLCWpgWEpSQ0MS0lqYFhKUgPDUpIa\nGJaS1MCwlKQGhqUkNeglLJPsS3JyaDmVpJJcM0vfzd267w/1/7991CFJC6WXnztW1Y8Pv07yr4Ff\nqKo982w2UVWGpKTXhN5Pw5OsAv4x8GDf+5akpbIQ1yx/AVgPPHyOfn+a5FCSP05yw6gHSbIhyZYk\nW05P1Th1SlKzhZh16C7gi1V1dI71h4G/D+wBVjM9Cv1ykndX1dMjHOdu4BMABw9PnUe5WgrjzFbk\nTEVaSr2OLJP8beCngN+Zq09Vnayqp6rqlar6flVtB/4E+MURD7cdmAAmNm0cjF2zJLXo+zT8LmBv\nVf3piNudATLKBlV1pKomq2py1WCkTSVpZL2FZZKLgF9inlFl1++6JH83yaoka5J8BLgeeLyvWiSp\nb32OLG8G3gDsHG5M8p7uXsq3dU2XA/8FOAa8ANwB/FxV7e6xFknqVW9f8FTVF4AvzNL+JLB26PWj\nwKN9HVeSFoM/d5SkBoalJDUwLCWpgWEpSQ0MS0lqYFhKUgPDUpIaLMREGtKCGGfyDXACDvXDkaUk\nNTAsJamBYSlJDQxLSWpgWEpSA8NSkhoYlpLUwLCUpAaGpSQ1MCwlqYFhKUkNDEtJamBYSlIDZx3S\niudsReqDI0tJamBYSlKDprBMckuSJ5McT3J6lvU3JtmX5FSSZ5K8/xz725TksSQnkhxKsi2JwS1p\n2WoNqJeBB4B7Zq5IcgXwGPApYH33+HiSzfPsb2f3eCnwbuAm4Fcba5GkRdcUllW1q6oeBQ7MsvpO\nYHdVPVJVr1TVTmBP1/7/SXI58NPAr1bVsao6AGwDPjrWO5CkRdDHqe9VwO4ZbXu69rn6H6uqZ2f0\n35xkXetBk2xIsiXJltNTNVLBkjSqPsLyYuDYjLajwFzBN1d/5tlmNncD+4H9Bw9PjbCZJI2uj7A8\nwfS1ymGXAMdH7H92XavtwAQwsWnjYITNJGl0fYTlXuCaGW1Xd+1z9V/ffTE03P+5qpo54pxTVR2p\nqsmqmlw1yEgFS9KoWm8dGiRZA1zUvV7TLQEeBq5NcmuS1UluBd4FPDTbvqrqz4GvAvcnWdd94XMv\n8GAP70eSFkTryPIO4BSwCxh0z08Bl3Vf1NwMbGX61HsrcFNVPXd24yQnk9w2tL/bumO/AHwL+APg\n/vN6J5K0gJp+G15VO4Ad86x/AnhinvVrZ7w+yHTAStJrgr+akaQGzjokzWGc2YqcqWjlcmQpSQ0M\nS0lqYFhKUgPDUpIaGJaS1MCwlKQGhqUkNTAsJamBYSlJDQxLSWpgWEpSA8NSkho4kYbUo3Em3wAn\n4HgtcGQpSQ0MS0lqYFhKUgPDUpIaGJaS1MCwlKQGhqUkNTAsJamBYSlJDZrCMsktSZ5McjzJ6Rnr\n/mGSryc5nOTlrt97zrG/55L8IMnJoeUnzueNSNJCah1Zvgw8ANwzy7q/AWwHrgTeBHwe+HKSt55j\nn/+kqtYOLf+rtWhJWmxNvw2vql0ASW6YZd3OGU3/KclvANcCz59vgZK0HPR+zTLJO4ANwDPn6Prb\nSV5K8u0kd41xnA1JtiTZcnqqxqpVklr1OutQkk3A7wH3V9X/mafrncBu4K+AG4AvJKGqHhzhcHcD\nnwA4eHhqvIKlZWKc2YqcqWhx9TayTPIW4BvAV4D75utbVf+9qk5W1Q+r6r8Bvw3cPuIhtwMTwMSm\njYNxSpakZr2EZZLNwJPAl6vqn1XVqOfFZ4CMskFVHamqyaqaXDUYaVNJGlnrrUODJGuAi7rXa7ol\nSd4O/AnwaFX9SsO+LkvyD7rtB0muBz4GfPE83ockLajWkeUdwClgFzDonp8CLgPuBf4WcM+M+yZv\nO7vxjNd/nenT7kNM35L0H4HfqKrtfbwhSVoIrbcO7QB2zLH6w90y3/Zrh57/b+DqtvIkaXnw546S\n1MCwlKQGhqUkNTAsJamBYSlJDQxLSWpgWEpSg14n0pC0eMaZfAOcgGNcjiwlqYFhKUkNDEtJamBY\nSlIDw1KSGhiWktTAsJSkBoalJDUwLCWpgWEpSQ0MS0lqYFhKUgPDUpIaOOuQdIFxtqLxOLKUpAaG\npSQ1aArLJLckeTLJ8SSnZ6y7IUklOTm0fPMc+9uU5LEkJ5IcSrIticEtadlqvWb5MvAA8Abgs7Os\nn6qqtSMcdydwArgU2AA8AbwEbBthH5K0aJrCsqp2wfQo8nwPmORy4KeBK6vqGHAsyTZgK4alpGWq\nr1PfQZLnk7yY5EtJrpqn71XAsap6dqhtD7A5ybrWAybZkGRLki2np2rcuiWpSR9h+R3gncDlwNuB\np4GvJ3nLHP0vBo7NaDvaPTaHJXA3sB/Yf/Dw1AibSdLozjssq+rFqtpbVaer6mhV3cf09ccPzLHJ\nCWD9jLZLhta12g5MABObNg5GqlmSRrVQ30CfATLHur3A+iRXDLVdDTzXXcNsUlVHqmqyqiZXDeY6\nlCT1o/XWoUGSNcBF3es13ZIk701yZZLXJVmb5JPAm4Fds+2rqv4c+Cpwf5J13Rc+9wIP9vGGJGkh\ntI4s7wBOMR2Ag+75KeAypr+w+RrTp9AHgOuA91XV82c37u69vG1of7d1x34B+BbwB8D95/VOJGkB\ntd46tAPYMcfqz3TLfNuvnfH6IHBzy7ElaTnwVzOS1MBZhyQ1GWe2opU0U5EjS0lqYFhKUgPDUpIa\nGJaS1MCwlKQGhqUkNTAsJamBYSlJDQxLSWpgWEpSA8NSkhoYlpLUwIk0JC2YcSbfgOU5AYcjS0lq\nYFhKUgPDUpIaGJaS1MCwlKQGhqUkNTAsJamBYSlJDQxLSWrQFJZJbknyZJLjSU7PWPfxJCdnLJXk\n38+zvz9O8lcztvnZ830zkrRQWkeWLwMPAPfMXFFV/6aq1p5dgKuBAh45xz7/1fB2VfVfR6pckhZR\n02/Dq2oXQJIbGrrfBXy7qv7sPOqSpGWl12uWSV4P/BLwOw3d70nyUpJ9Se5LsnrEY21IsiXJltNT\nNU65ktSs71mHPgRcBHz+HP3uA74DHAf+HrATWNe1t7ob+ATAwcNTIxcqafkaZ7aihZ6pqO9vw+8C\ndlbVyfk6VdX/rKqXq2qqqp4C/gVw+4jH2g5MABObNg7Gq1aSGvUWlkn+DvAe2k7BZzoDZJQNqupI\nVU1W1eSqwUibStLIWm8dGiRZw/QpNknWdMtwSt0FPFVVe8+xr0uS/GyStZl2NfBJ4IvjvQVJWnit\nI8s7gFPALmDQPT8FXAaQ5A1dn1lHld2XOB/vXq4GtgIvMH3N8otMX+Mc5XqlJC2q1luHdgA75ll/\nCnjjPOt/fOj5IeC65golaRnw546S1MCwlKQGhqUkNTAsJamBYSlJDQxLSWpgWEpSg74n0pCkJTHO\n5BsAg7/Z1s+RpSQ1MCwlqYFhKUkNDEtJamBYSlIDw1KSGhiWktTAsJSkBoalJDUwLCWpgWEpSQ0M\nS0lqYFhKUoNU1VLXcN6SHAL+YpZVA+DNwPeAqUUtanny83g1P49Xu1A/j8uq6k3n6rQiwnIuSbYA\n+4GJqppc6nqWmp/Hq/l5vJqfx/w8DZekBoalJDVY6WF5BPiX3aP8PGby83g1P495rOhrlpLUl5U+\nspSkXhiWktTAsJSkBoalJDUwLCWpgWEpSQ0MS0lqYFhKUoMVGZZJBkk+neRQkhNJfj/JxqWua6kk\n2ZHkh0lODi2/vNR1LZYktyR5MsnxJKdnWX9jkn1JTiV5Jsn7l6LOxTLf55HkhiQ142/lm0tV63Ky\nIsMS+HXg54F3A5d2bb+7dOUsCw9V1dqh5YGlLmgRvQw8ANwzc0WSK4DHgE8B67vHx5NsXsT6Ftuc\nn0dnasbfyk8uYm3L1koNy48A26rqQFUdA34NuHGF/wegOVTVrqp6FDgwy+o7gd1V9UhVvVJVO4E9\nXfuKdI7PQ3NYcWGZZD3wNmD32baqehY4DrxjqepaBj6Y5KUkk90lirVLXdAycRVDfyudPV37hWqQ\n5PkkLyb5UpIL+bP4kRUXlsC67vHYjPajQ+suNNuBtwMbgZuA64HPLWlFy8fF+Lcy7DvAO4HLmf6b\neRr4epK3LGlVy8BKDMsT3eP6Ge2XMD26vOBU1e6q+l5VnamqfcDHgA8lef1S17YMnMC/lR+pqher\nam9Vna6qo1V1H/AS8IGlrm2prbiwrKqjwF8C15xt6y7ir2P6/5KCM91jlrSK5WEvQ38rnau7dk07\ng38rKy8sO58F7k1yeZJ1wDZgV1U9t7RlLY3uVpFLuuc/BvwW8IdV9YOlrWxxdLeSrQEu6l6v6ZYA\nDwPXJrk1yeoktwLvAh5awpIX1HyfR5L3JrkyyeuSrE3ySab/EbNdS1nzcrBSw/I3gT8CvgW8wPS/\nWnf7kla0tD4KHEjyfeArwFPAh5e2pEV1B3CK6f/gB93zU0z/q37PAjcDW5k+9d4K3LTC/8c65+fB\n9BdbX2P68sQB4DrgfVX1/NKUunw4U7okNVipI0tJ6pVhKUkNDEtJamBYSlIDw1KSGhiWktTAsJSk\nBoalJDUwLCWpwf8Dm2VeIxBIu/wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb2625def60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 5))\n",
    "plt.imshow(subsequent_mask(20)[0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention\n",
    "\n",
    "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.\n",
    "\n",
    "The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.\n",
    "\n",
    "We call our particular attention \"Scaled Dot-Product Attention\". The input consists of queries and keys of dimension $d_k$, and values of dimension $d_v$. We compute the dot products of the query with all keys, divide each by $\\sqrt(d_k)$, and apply a softmax function to obtain the weights on the values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix $Q$. The keys and values are also packed together into matrices $K$ and $V$. We compute the matrix of outputs as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "Attention(Q, K, V) = softmax(\\frac{QK^{T}}{\\sqrt(d_k)})V\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def attention(query, key, value, mask=None, dropout=None):\n",
    "    \"compute the 'Scaled Dot-Product Attention'.\"\n",
    "    d_k = query.size(-1)\n",
    "    scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(d_k)\n",
    "    if mask is not None:\n",
    "        scores = scores.masked_fill(mask == 0, -1e9)\n",
    "    p_attn = F.softmax(scores, dim=-1)\n",
    "    if dropout is not None:\n",
    "        p_attn = dropout(p_attn)\n",
    "    return torch.matmul(p_attn, value), p_attn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two most commonly used attention functions are additive attention (cite), and dot-product (multiplicative) attention. Dot-product attention is identical to our algorithm, except for the scaling factor of $\\frac{1}{\\sqrt(d_k)}$. Additive attention computes the compatibility function using a feed-forward network with a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.\n",
    "\n",
    "While for small values of $d_k$ the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of $d_k$. We suspect that for large values of $d_k$, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients (To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean $0$ and variance $1$. Then their dot product, $q·k = \\sum^{d_k}_{i=1}q_ik_i$, has mean $0$ and variance $d_k$. To counteract this effect, we scale the dot products by $\\frac{1}{\\sqrt(d_k)}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.\n",
    "\n",
    "    MultiHead(Q, K, V) = Concat(head_1, ..., head_h)W^{o}\n",
    "    where head_i = Attention(QW^{Q}_{i}, KW^{k}_{i}, VW^{V}_{i})\n",
    "    \n",
    "Where the projections are parameter matrices $W^Q_i \\in \\mathbb{R}^{d_{model}\\times d_k}$, $W^K_i \\in \\mathbb{R}^{d_{model}\\times d_k}$, $W^V_i \\in \\mathbb{R}^{d_{model}\\times d_v}$, and $W^O \\in \\mathbb{R}^{hd_{v}\\times d_{model}}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this work we employ $h = 8$ parallel attention layers, or heads. For each of these we use $d_k = d_v = d_{model}/h = 64$. Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadedAttention(nn.Module):\n",
    "    \n",
    "    def __init__(self, h, d_model, dropout=0.1):\n",
    "        \"Take in model size and number of heads.\"\n",
    "        super(MultiHeadedAttention, self).__init__()\n",
    "        assert d_model % h == 0\n",
    "        # We assume d_v always equals d_k\n",
    "        self.d_k = d_model // h\n",
    "        self.h = h\n",
    "        self.linears = clones(nn.Linear(d_model, d_model), 4)\n",
    "        self.attn = None\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "    \n",
    "    def forward(self, query, key, value, mask=None):\n",
    "        if mask is not None:\n",
    "            # Same mask applied to all h heads.\n",
    "            mask = mask.unsqueeze(1)\n",
    "        nbatches = query.size(0)\n",
    "        \n",
    "        # 1) Do all the linear projections in batch from d_model => h x d_k \n",
    "        query, key, value = \\\n",
    "            [l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)\n",
    "             for l, x in zip(self.linears, (query, key, value))]\n",
    "        \n",
    "        # 2) Apply attention on all the projected vectors in batch. \n",
    "        x, self.attn = attention(query, key, value, mask=mask, \n",
    "                                 dropout=self.dropout)\n",
    "        \n",
    "        # 3) \"Concat\" using a view and apply a final linear. \n",
    "        x = x.transpose(1, 2).contiguous() \\\n",
    "             .view(nbatches, -1, self.h * self.d_k)\n",
    "        return self.linears[-1](x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applications of Attention in our Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Transformer uses multi-head attention in three different ways: \n",
    "\n",
    "1) In “encoder-decoder attention” layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models.\n",
    "\n",
    "2) The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder.\n",
    "\n",
    "3) Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position. We need to prevent leftward information flow in the decoder to preserve the auto-regressive property. We implement this inside of scaled dot-product attention by masking out (setting to $−∞$) all values in the input of the softmax which correspond to illegal connections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Position-wise Feed-Forward Networks\n",
    "\n",
    "In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between.\n",
    "\n",
    "$$ FFN(x) = max(0, xW_1 + b_1)W_2 + b_2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1. The dimensionality of input and output is $d_{model} = 512, and the inner-layer has dimensionality $d_{ff} = 2048$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class PositionwiseFeedForward(nn.Module):\n",
    "    \"Implements FFN equation.\"\n",
    "    def __init__(self, d_model, d_ff, dropout=0.1):\n",
    "        super(PositionwiseFeedForward, self).__init__()\n",
    "        self.w_1 = nn.Linear(d_model, d_ff)\n",
    "        self.w_2 = nn.Linear(d_ff, d_model)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.w_2(self.dropout(F.relu(self.w_1(x))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings and Softmax\n",
    "\n",
    "Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{model}$. We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation. n the embedding layers, we multiply those weights by $\\sqrt(d_{model})$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Embeddings(nn.Module):\n",
    "    def __init__(self, d_model, vocab):\n",
    "        super(Embeddings, self).__init__()\n",
    "        self.lut = nn.Embedding(vocab, d_model)\n",
    "        self.d_model = d_model\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.lut(x) * math.sqrt(self.d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Positional Encoding\n",
    "\n",
    "Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add “positional encodings” to the input embeddings at the bottoms of the encoder and decoder stacks.\n",
    "\n",
    "The positional encodings have the same dimension $d_{model}$ as the embeddings, so that the two can be summed. There are many choices of positional encodings, learned and fixed.\n",
    "\n",
    "In this work, we use sine and cosine functions of different frequencies:\n",
    "$$ PE_{pos, 2i} = \\sin(pos / 10000^{2i/d_{model}}) $$\n",
    "$$ PE_{pos, 2i+1} = \\cos(pos / 10000^{2i/d_{model}}) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $pos$ is the position and $i$ is the dimension. That is, each dimension of the positional corresponds to a sinusoid. The wavelengths form a geometric progression from $2π$ to $10000·2π$. We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, \n",
    "$PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$.\n",
    "\n",
    "n addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks. For the base model, we use a rate of $P_{drop} = 0.1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(nn.Module):\n",
    "    \"Implement the PE function.\"\n",
    "    def __init__(self, d_model, dropout, max_len=5000):\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "        \n",
    "        # Compute the positional encodings once in log space.\n",
    "        pe = torch.zeros(max_len, d_model)\n",
    "        position = torch.arange(0, max_len, dtype=torch.float32).unsqueeze(1)\n",
    "        div_term = torch.exp(torch.arange(0, d_model, 2, dtype=torch.float32) *\n",
    "                             - (math.log(10000.0) / d_model))\n",
    "        pe[:, 0::2] = torch.sin(position * div_term)\n",
    "        pe[:, 1::2] = torch.cos(position * div_term)\n",
    "        pe = pe.unsqueeze(0)\n",
    "        self.register_buffer('pe', pe)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = x + Variable(self.pe[:, :x.size(1)], \n",
    "                         requires_grad=False)\n",
    "        return self.dropout(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Below the positional encoding will add in a sine wave based on position. The frequency and offset of the wave is different for each dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4sAAAE7CAYAAAB0RhQ/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FFX3wPHv7G6ymx7SaAlpEKqh9yZK7yAdaTawN1BR\nUV/9YXvlVVQUARUpSi/Sm3SQ3gk1CSShJCG9bJ/fHxNChxB2syn38zw8TnZnZ28iWebce+45kizL\nCIIgCIIgCIIgCMLNVI4egCAIgiAIgiAIglD8iGBREARBEARBEARBuIMIFgVBEARBEARBEIQ7iGBR\nEARBEARBEARBuIMIFgVBEARBEARBEIQ7iGBREARBEARBEARBuIMIFgVBEARBEARBEIQ7iGBREARB\nEARBEARBuIMIFgVBEARBEARBEIQ7aBw9gKLm5+cnh4SEOHoYgiAIgiAIgiAIDnHgwIFkWZb9H3Re\nmQsWQ0JC2L9/v6OHIQiCIAiCIAiC4BCSJF0oyHkiDVUQBEEQBEEQBEG4gwgWBUEQBEEQBEEQhDuI\nYFEQBEEQBEEQBEG4gwgWBUEQBEEQBEEQhDuIYFEQBEEQBEEQBEG4gwgWBUEQBEEQBEEQhDuIYFEQ\nBEEQBEEQBEG4g82DRUmSBkmStF2SpAxJkswFOL+RJEl7JUnKkSTpvCRJT9/2fIAkSUskScqUJClJ\nkqSvJEkSQa4gCIIgCIIgCIId2SPoSgV+At540ImSJHkBa4DFQDlgDDBVkqTmN502N++/gUBToA8w\nzpYDFgRBEARBEARBEG6lsfUFZVleByBJ0uMFOL0vkAt8LcuyDGyQJGkp8AKwW5KkUKA9UFWW5XQg\nXZKkr4APga9sPXZHMBzcQs66eagCwpECqiK5e6PSapG0WiRnLZLWGZWrKxofH1Suro4ersNlGcyc\nS8wiPjWHUD83alTwRK2SHD0s4R5yTDlc018j05iJ0WLEZDVhspiU/1pN+Y+pVWp0ah06jQ6dWoeL\nxgWtWotOoxx7a71Rq9SO/naEu5FlMOtBnwGGDHByBa/Kjh5VsSbLMiaLjNlqxWSWcdWqcVKLhBlB\nKDKyDEmnwZgNbn7g5g/O4h7rfnKMZg5dTEOtkqjs7UJFLx0a8bl1B9lkwpKRgSU9HUtaOpb0NFRu\nbrg1aeLooRWazYPFh1QXOJgXKF53EBh20/Ppsiyfv+35EEmSPGVZzijIm0iS5Av4AtStW/fRR21D\nORuWcuWPrcDWB56rcnND4++Pxs8PTYA/aj8/NP7+OFWshDYsFOfQUFQuLvYfdBHIMZo5fSWTs1ez\nOJuYyZmrWZxLzCIhLfeW8zy0GhqGlKNxiA+NQ3yIDPRC5ySCiqKQacwkNj2W2IxYLmZeJDk3mWu5\n17imv0ZKbgrX9NfINec++EIFoJJU+On8CHANwN/VnwDXAOXYxZ8KbhUI8wojwDUASRITB3ZjzIHD\nc+HUSshJUQLD6wGi9bYdB5XqQ52noHYf8Ap0zHgdTJZlDlxI5Y/dF/g3+hoGkwWTRcZksWK2yrec\n66HV0L1uJfo3CqR+kLf4eywI9mC1QvxeiFqhfI6lxt76vJPbjcDRzQ/cA6BGD6jWAcrg76Qsy5xL\nzGLrmSS2nkliT0wKRrM1/3mVBBU8dVQu50Jlbxcql3Mh2MeNTnUq4OXi5MCR248lMxNTXBzGuHhM\n8XEY4+IwxSdgTk7GkpGONS0da07OHa9zbdIEt1klN1iUbo3TbHhhZWVxoyzL9wxIJUn6FdDIsjzi\npsdGAR/IslxVkqRhwP/Jshx80/OhQDQQJMtyfAHH8gnwMUDFihW5dOlSIb4j+0ib9BZJf61BNlmQ\nLRJWiwRy4T+UnCpVwjk8HG1YGM5hYWjDw9DWqIHa3d2Go7YfvcnCjO3R/LzlPNlGyz3Pc1arMFqs\nd328bpAXzcP9eKZlCN6uzvYcbpmQqk/lePJxotOjiUmPITYjltj0WK7prxX6mhpJg5PaCSeV8scq\nW9Fb9I8UXHo4eRDmHUZV76qEeYUR7h1OuHc45V3Li5vvR5F9DfZNh73TIKcQ/8+rNFcCx1q9wd3f\n9uMrZvQmC38fvsTMXbGcvFyg+cxbVA1wp3/DQPo0qEyAh84OIxSEMsRshJhtcGoFnFoN2YkPf42K\ndaHNOKjeDVSleyUt22Bm+1klONx6OolL6fqHvoanTsPotuGMahmCq7Oj16QKx5qdjf7UKfQnTqI/\neRLDuXOY4uKwpKc//MXUalwbNCB49izbD/QRSZJ0QJblRg88z8HB4ndAiCzLvW967E1gmCzLDSRJ\n6g3MlGXZ+6bn66OsLnrnpaYWZCw3ryyePnz4cGG+JfvKSYFLh+DyYeS4g8jxR7CmJiBbwGpWYdar\nMLtWxxzcDYvBCXNSEuakZMxJSRgTEpDvMpORT5LQVq2KS716yp/69XAOCUEqRh96VqvMiqOX+GrN\nqVs+nLxdnYgI8KBqeXeqBbgTUd6DagHu+HtoiU7OZl9MCntjU9gfm8rFlFt/Bn7uznzcozbdIyuK\nYKGAjBYjp1NOczT5KEeTjnIs+RhxmXH3fY2HkwfBnsH4u/rj6+KLj84HX53vjWMXX7ycvXBWO+Os\ndsZJ5YTqHjWqZFnGYDGgN+vzg8frqayJOYn5f5Jyk0jKSSIxJ/GBQau31pt6/vWoF1CP+gH1qe1X\nG61aW+ifUZmREgO7p8ChOXA9iFdplKAvoCbovEDrCTrPW/+bch6OL1Zm7/U3fURLKghtC01egBpd\nHfM92VFcSg5z9lxg/r440nJM+Y/XqODBgEZB+HtocVJLOKlVOKlVaNQSzmoVKpXE3pgUFu6P43xS\ndv7r1CqJxyP86d8okCdrlhdpqoLwMEx62DwRDvwBhttuFf0ioEZ3qNkdvEMgJxmykyD7tv9eOQrx\n+268zr8mtBmrZEyUwm0Ra45d5sNlx7mWbbzlcV83Z9pE+NM2wp/W1fzQOqlJSM0lIS2HhNRc4tNy\n877O5eSlDAx5q49+7s683K4qQ5pWQaspvj8vq15P7pGj6E+cQH9SCQ6NMTFKqvI9SM7OOAUF4RwY\niFNQEBp/f9Te3qi9PFF7eaH28kLl6YXa2wuVm1uxvQctKcHiKOCT21YOZwEWWZZH3bSKGC7LcnTe\n888AE2RZDi3MuBo1aiTv37+/MC8tetnJcOkwHPhdSZkA5Yar8fPQ7n1wUWJoWZYxX7mC4Xw0xuho\nDNHnMZ6PxhATgyU5+a6XVnl54RIZiUv9erg1b45LZCSS2jG/zPtjU/hsVRRH4tIA0Kgknm4WzOi2\nYVTw1BX4l+xKup59sSnsibnGogPx6E3KB1a76v581rsOgeXEfoTbGSwGDl49yK5Luzh49SBRKVGY\nrKY7zpOQCPIIIsQrhBDPEEK8Qgj1DCXEKwRfna9DPwhzTDnEpMdwPv0859LOEZ0Wzbm0cyRkJdz1\nfI1KQy3fWtTzV4LHxhUa46X1KuJRF2OXDsPOyXByGch5q/fO7tBgBDR7EbyDCnYdswHO/6MEjqdW\ng+lGIESDEdD5y1KxRyguJYfPVp5kY9RVrmeXqlUSnWtXYHjzYJqE+hTo90OWZQ5eTGPRgThWHLlM\nluFGam+tip78/HQDgn3d7PVtCELpcfUkLH4OEk/ceKxSfajZQ0kr9Y8o+LVid8K2ryF6y43HfMKh\n9dsQOQDUJT/dMi3HyEfLT/D3ESXrTq2SaFDFm7YR/rSNCKB2JU9UBawNkZihZ8rmc/y59yImi/KB\nWMlLx2tPVqNfw8BiscdRNpnIPXaM7H//JeffPeQeOoRsuvO+B0BTvjy6WrXQ1qiOc5VgnKsE4RQY\nhMbfr1gtuBSWw4JFSZLUgBPQBlgNXM9/NNy2NxFJkryBs8DXwGSgNbAc6CDL8u68czYA6cAzKKuD\na4HfZVn+sjDjK1HB4s3OboDV4yA1RvnazR86fAqRg+6bFmFOTUV/7Bi5hw8rf44cxZqdfcd5ai8v\n3Fq2xK11a9xbt0Lj52ev7yRfXEoOX649xaqjl/Mfa18zgPFdaxLu/2hps3EpOXyw7DjbziQB4OKk\n5u2OEYxqGVqmC+LIskxsRiy7Lu1iR8IO9l/Zj95yZ5qJn4sfj/k9RqR/JJF+kdT2q42bU8m6Ub0e\nRB5LPsahxEMcSTpy1wBSJamo51+P1oGtaV25NRHlIortLKBdyTJs/wb++b8bj7mXh6ZjoNEocClX\n+Gsbc+DsOvh3KsT9qzzmXxP6z4SAGo80bEfadT6Zl+ceJDVvJdHP3ZnBTaowpGkVKnoVfv94rtHC\nmuOXWbg/nt3Rysq5h07DpP516Vi7gk3GLgiljiwr6fLrJ4DFAEjQ8jUlm+FR907H7VM+H8+svfGY\ndzD0+w0CH3ivXWxtirrKe0uOkZRpAKBZmA//7VeXIJ9Hm8iLS8nh+01nWXwwPn8SLcTXlbGdqtPt\nsaLN9pJlGcOpU2Tv2k32nn/J2X/grtl4TpUqoatdG13tWuhqKX+K4l7YkRwZLI4Efr/LU6FAEEqr\njFqyLF/MO78xMAV4DLgMfCTL8pybrhcATAU6AAbgN+A9WZbv3LBWACU2WAQlrWLXD7B90o20sKCm\n0PUbqBhZoEvIFguGc+fzg8ecffswxd+59VNXq5YSOLZti0u9ujadQZFlmZ+2nGfyxrP5+w5rVPBg\nQvdatKxqu19MWZZZfvgSn648SUpeWkVkoBdf9H2M2pXKzkqSyWpiz+U9bLq4iV0Ju7iUfeee3are\nVWlWsRl1A+oS6RdJRbfSmbqbmJPI4cTD+cFj1LUozPKtxVkCXANoXbk1rQNb07xic1ydSv7q1wOZ\nDbDidTjyl/K1b1Vo+TpEDgSNDVN2rRbY9g1s/VJZtdS4QNevof6wElVAQpZlZu2+wKcrT2Kxynjq\nNEzoXoue9SrZPN1q25kkXp93KD8gHdM2nLEdI4rFDL0gFBtZibDsJTi3QfnaMxD6ToOQlrZ9n8tH\nlM+wqL+VrzUu8NQMJaW1BMnQm/hsxUkWHlDu/3ROKt7tXIMRzUMKvIpYEOeTsvh2wxlW3rQo8Fyr\nUN7vWtOm73M72Wol9/BhMtdvIHPDBkwJd04UOwUF4dasKa5Nm+HWtAka/9K/p/52Dk9DLa5KdLB4\nXeoFWPf+jdRUlUaZ3arV66EvJcsyxthYsrfvIGv7dnL27kU2GG45R1OxIp5duuDZtSu62rUeKYiw\nWmU+XXmSmbtiAfBz1zK2YwT9GwXZbcUvNdvIxNVRLMr7UFSrJF5sG85bHSLs+mHlSGarmb1X9rI+\ndj0bL24k/bY9G15aL5pXbE6LSi1oUakF5d3KO2ikjpVlzGLP5T1sT9jO9vjtJObeWvzASeVEm8A2\ndA3tSpvANug0pbDgSPY1mP80XNylfF27L/T+CZzsWFk5doeSJpaZdwNRpx90/1bZ91jMGc1WPlp+\nnHn7lL28VQPcmTG8ESF+9lt5T0jL5aW5B/NT9ZuF+fDD4Ab4e4i9t4LAmXVKoJiTt+2mdl/o/r9H\ny4Z4kMtHYN5QSI8DJCWtvtkY+72fDe04m8w7i47k14doUMWbb/rXJewRM7ru58SldCauimLXeSVT\none9Snzdry7OGhsuRBiNZO/dR+aGDWRu2nTHNiyNvz+uzZrlB4jOgaLFkwgW76FUBIvXnd0Aq96G\ntAsgqZXZrTp9H+mSVr2enH37yNq2next2zBeuHDL887BwXh264pn165oq1Z9qGtbrDLvLj6aH7R1\ne6wiX/WLxF1bNNWydp5L5v2lx7hwTUk/GNYsmE971S41K2gWq4UDVw+wNnYtGy9sJNWQesvzkX6R\ntAlsQ8vKLanpU1P0LbyNLMucTj3N9vjtbIvfxtHko1hvSmBwc3LjySpP0i20G00qNkGjKplV3m6R\nfBbm9r+R3t7mHXh8fNFU/MtOhmUvwtn1ytflQqH/78reomIqKdPAi3MOsP+C8rvVvmYA3w6sh4fO\n/vuWDGYLE1dFMWu38pkc4KFlytAGNA7xsft7C0KxZMpVUk73TVe+dvaAbt8oGRFF8e965hXl8/PK\nUeXrZi9Dx/8r1hVTv91whsmbzgJK9fi3OkbwfOuwItmeY7ZYeX/pMRbsV+4B20T48/PQBrg9wj2g\nLMvkHjxI2pIlZG7YiDXj1grUzuHheLRvj0eHDo+82FEaiWDxHkpVsAhK6sUfPSEpSil+02caRPa3\nyaVlWcZw5iwZq1eTsXo1prhbq2Jqq1fHq2cPvPr0QeNz/xsWo9nKm/MPs+qYspIwoFEgX/SNLPL9\ng3qThfeXHmPJQSUlYWSLED7uUbI/QOIy41hydgnLzy0nKTfpludq+daic0hnOoZ0pLK7mEV7GGn6\nNDbHbWZVzCr2Xt6LzI3PSh+dD51COtEjrAd1/OqUzL8/0VthwTClYqnaGXr+CHUHFu0YrFb49yfY\n+AlYTaByUlY1IwcU7TgK4Fh8Oi/M3s/lvNn4V9pVdUh2wvLDCby3+Bi5JgtqlcT4LjV4tlVoyfw7\nKAiFZTbCX4Pg/Cbl68DG0Hc6+BSq9mHhGbJg4cgb6a81eyjjsGdmRiFN2XyO/647DcBjlb2YNKAu\nEeU9inQMsizzzfrTTNmstE+vG+jFbyMb4+v+cFkSpitXSF+2nPSlS+9Y1NDVro1Hhw54dGiPNjzc\nZmMvjUSweA+lLlgEZYZ+Vi+4elwJGHv9BPUG2/QtZFlGf+wYGatWk7FmDebEG+l6kpMTHh074j1w\nAK6NG99x06I3WXhp7kH+OaW8ZmSLED7qXsthKaAWq8y4hUdYckgJGJ9pGcqE7jVL1M2WyWJic9xm\nFp1ZxO7Lu295rnq56nQK6USnkE5U8azioBGWLkk5SayNXcuamDUcSz52y3M1fWoyoPoAuoZ2LTn7\nGw/8AaveAqsZXH1h4FwIbu648SQcgIWjbmRJDJmvNMIuJpYfTuCdRUcxmK3onFR8078u3SMrOWw8\nZ65mMmbOAaLzWm2MbhPG+K41HTYeQShSVisseU6ptAxKZdLH3we1g7I9LGZY/TYcmKl8HdgEBv8F\nbsWnOMofu2L5+G+lOuwTNQKY+nRDm6aAPqzfd8bwnxUnAQjzc2PWs00eWLHeajCQtWkTaUuWkr1r\nl/L3II82IgKv3r3x7NQRp8piYrygRLB4D6UyWASlT+OsnnDlGCBBzx+gwTC7vJVssZBz4AAZK1aS\nsWoV1puqSjmHhVFu4AC8evVC7e1NlsHM83/sz6/o90q7qrzd0fHVJi1WmbcWHGb5YaXYywttwhjf\npYbDx/UgFzMusvjsYpadW0aKPiX/cR+dD72r9qZX1V6EeYU5cISl38WMi6yOWc2q6FXEZsTmP+7u\n5E6P8B4MrD6QcO9iOpspy7DpU9jxP+VrvwgYsqDoZ+PvJicFfu+qZEk4ucKIlRDY0NGjYvWxy7w0\n9yAAlb1dmDa8YbEokJVlMPP2gsOsO3EVgM961WZY8xDHDkoQ7E2Wlcrw11NPW4+FJyc4dkygjGvH\nt7DpP8rXPmEwdBH4Ov7fgkUH4hm78AgAzcN8+X1UY3ROjt+G8veRS7y94DAmi0x5Ty1/PNOEGhXu\n3LduSkggZe6fpC1ejDX9Rv0FlZcXXt264dW3r0gxLSQRLN5DqQ0WQbnZmt0HLh9Wvu7+nVLy3o4s\nWdlkrFxJ6vz5GKKi8h+XtFp0HTrytVs9VhuUG6t3O9fgxccd/8F5ndli5Y35h/OrdL34eDjvdKpe\n7D5wZFlm96XdzDwx845VxBaVWtAvoh+PBz6OUyno91SSyLLM/qv7mX96PpsubLqlqmrD8g0ZWH0g\n7au0L17/X/ZOh9VjlePQtjBgVn6/1mIhPR5+7QgZCcqK5zPrwe/h9kbb0pG4NAZO243eZCUyL13K\n7yHTpezJZLHyzMx9bD+bjEqCacMa0b5W2SxWJZQRW76ELV8oxw1HKYWxitO/2UcXwvKXwGJUWpw9\n/w94Oy7DZ/Wxy7zy50GsMtQL8mbOc02LrE5EQWw/m8To2QfIMVrw0Gn4dURjmoT6KHsRDx0i5Y9Z\nZG7YcGMVUZJwa9kS76f64v7EE6i0xefzuCQSweI9lOpgESA3Deb0VdK6QGmr0eR5u7/t9TTV1Hnz\nyVi9Gll/o3ffQf9quI8YSc9n+xS7QMxssfLavEOsPnYFgNeeqMpbHas7eFQKk9XE2pi1zDwxkzOp\nZ/If93Pxo0/VPvSp1ocgjwI2SBfsKjk3mSVnl7DozCIuZ98oEV7etTzDaw2nX0Q/x6eoRm+B2X1B\ntkBEZxg4p3g2lE6Mgt86KXspvavAsxvBo+gDoEtpufSaspOkTAOVvV1Y+nILAjyKXzXcTL2J/lN3\nc+pKJi5OauaPbkZkYDGaABAEW7l5sqtWL+j3OxTHQm2xO+CvwWDIgAqR8Mw6cC76z//NpxN5YdZ+\nTBaZGhU8mP9Cc7xci99n/tH4NEb+vo+UbCPeGlhQMxenpfPRHz+ef47ax4dygwbhPaA/ThVEr1lb\nEcHiPZT6YBGUm6w5/SB+r/J1EQWM+W+fmsYP7/1AvQMbCMy6UXBFW6smvs8+i2enTkia4jOzZbJY\neeXPg/npXG+2j+D19tUcNp4sYxaLzy5m9snZXM25mv94pH8kI2qNoF2Vdjipit8HvqBUpN2esJ35\np+ezM2FnflEcT2dPhtQcwpAaQyins2M593tJiYZp7UCfBv414dn1xbtNxYXdMLs3mPVQ4TEYubpI\nx5ttMNN/6m5OXs7AzVnNohdbULNi8f15XUnX0+ennVxO1+Pn7szSl1o+clNtQShWji1S2u0gQ9jj\nSvq8LXvA2tqZ9fDnAEBWWgM9NaNIV0D/jb7GiN/2YjBbCfNzY/7o5sW61c75mCvMGT+JJ05tw09/\no6Kptnp1fIYPx7N7N7GKaAciWLyHMhEsAhgyYe4ApXeapIIRKyCkVZG89cRVJ5m+PQaVbGV6eA5h\nm5agP3I0/3mnwEB8Ro3Eu29fVC7Fo2KY0WzlpbkH2BilFOF5v2sNXmhTtCmzSTlJzD45m4VnFpJl\nygJAQuLxoMcZVWcU9QOKb0sB4U6x6bHMPDGT5eeXY7YqKao6tY6+1foyovYIKrkXUYEUfQbMaA/J\np5W+Y8//o+ynKe6iVirVWmWrkjI7dGGR3BxarDKjZx9gY9RVVBLMGNGIJ2oU/9TOU1cy6P/zbjIN\nZsL93Vj8Ygu8XZ0dPSxBeHTnNsKfg5SKyZUawIi/QVu0VTwLZfskZY84QIfPoOVrRfK2R+LSGDpj\nD1kGM5W9XVg4pjmVvIvHvdbtLBkZpMyaTcqsWfltL6xIxFarR9sPXsOtadNil5FWmohg8R7KTLAI\nSknnGU9C0inwqAhjdti9Ote6E1cYPVtJgX22VSgTutdScs8PHODajF/J2rIl/1x1uXL4Pvcc5YYO\nQaVzfHqXwWzhxTlK1VaVBAtGN6dREfQwS9Gn8Nux35h3eh4GiwEAZ5UzPcJ7MKL2CEK9ikHxEaHQ\nrmZfZU7UHBacXkCOWSkGpZbUdA3tyguRLxDiFWK/N7dalHSos+uUKqPDlkJYW/u9n63t+1Wp2gpQ\n5ynoO8PuPcy+WB3FL9uiAZjQvRbPtio5v387ziYz8ve9mK0yTUJ9mP1sE7SaYpimJwgFFbdPKd5n\nygHfakpKp5uvo0dVMLKstNU4uUyZtB+6EKq2t+tbnk/K4qmfd5GWY8LfQ8vC0c0J8XOz63sWhiUt\njZRZs0iZNRtrVt7kuFbLpRYd+FBdm0vu/ozvUoPRbYtPnYvSSASL91CmgkVQ9v9MawfmXAh/UqnO\nZaebrQvXsun+ww4y9WYaVPFm/ujmOKlvfS/9mTOk/PY76StXgllZbdH4++P74hjK9euH5OzYmfBc\no4UeP+7gXGIWlb1dWP1aa7vl+Kcb0pl5YiZzo+aSa84FwMPZg8E1BjO4xmD8XIpP2W3h0aUb0llw\negFzoubkV7JVS2p6V+3NmLpjqOBmh30YGz6Gnd8px0Wcjm4zmz+HrV8px81egk6f2y2da/6+i7y7\nWGmNMqRpFSb2Lnk9NBfuj2PcIiWTo0fdSkweWM9hbYoE4ZGkxMD0dpCbCp6VlUDRu4Tt0zdmw4wO\nkHgCdF7w/Ga7VUg1mq30+WknJy5l4O3qxPwXmlO9QvFagTWnppIy8w9S58zBmq20/pF0OsoNGoTv\ns8+g9vPjtXmHWXHkEioJ5jzblBZVxb2QvYhg8R7KXLAIcGgOLH9ZOX7yY2j9ls3fQm+y8NTPuzhx\nKYNyrk6seq31fdMeTJcvkzz1F9IWL84PGp0qVcLv5Zfx6tXToXsaT13JoOePOzGarXSuXYGfn25g\n0xvGTGMms0/OZvbJ2fnppm5ObgyvNZxhtYbh4Vy8PtwF29Kb9Sw7t4wZx2bk70l1VjkzsMZAnnvs\nOXx0NlrNProAluQFh8WxamBByTKseA0OzlK+7vUT1B9q87fZff4aw37dg9kq06qqH7+PanzHZFdJ\n8e2GM0zedBZQqjy/27mGg0ckCA/JalFa6cT9q6TPP7MO/ItH8bmHlhoL0x5Xgl7/mvDcRtC62/xt\nvlgTxS9blayIOc82pVW14hNkWTIzufbrr6TOmp3fbk1ycaHckMH4PvMMGt8bq8U5RjN9puzi9NVM\nfNycWflqq2KbRlvSiWDxHspksCjLsOxFOPKXkoo2ajVUaWbTtxi/5Bh/7b2IJMHMUU1oG+FfoNcZ\nL14kecoU0v9eoYwTcA4Jwe/VV/Ds0gXJziln9zJ7dywTlisNbCf2qcPQpsGPfM1ccy5zo+by+/Hf\nyTAqufkuGheG1BjCyNoj8daJCoZlicFiYP6p+cw4NoNUQyoArhpXhtUaxojaIx5t0iDhAPzWBSwG\nCG4Jw5aBpgTvX7OYlWIR5zeB1hNe3GnTcvQxydn0nrKT9FwT4f5uLHmpJV4uJbeIlCzLjF14lMUH\n4wGYMVy01BBKmO3/u9GzcNBfUKOrY8fzqKK3KK3NZCvU7AEDZtt08m7XuWSG/roHWYbnW4fyQbda\nNrv2o5BNJlLnLyB5yhQsqcq/c5KrKz5Dh+AzahQan7tPjsYkZ9Pzhx1kGszUDfRi/ujmxaI3ZGkj\ngsV7KJOzIJ2bAAAgAElEQVTBIij7F6e3g+QzSjrHmB3gapsVjCUH43lrgdLw9bUnq/FWh4iHH965\ncyT98COZ69blP6atWZPy772HW9MmNhnnw5BlpcjF+pNX0WpU/P1Kq0Knc1hlK6uiVzH54OT8lSSt\nWsvA6gN5ps4z+LqUkP0Xgl1km7KZfXI2f5z4I3+l2UvrxXN1nmNIzSE4qx8yyMu4rMxiZ11RAqrn\nN9t9r3KRyLwKPzWD3BQIaQ3D/7ZJSr3BbKHHDzs4czWLcq5OLHu5JcG+xW+Pz8Mymq0Mmf4v+y+k\n4u+hZcObbUTBG6FkuHwEpj+pFLRpMBx6/uDoEdnG7imw7n3luN2H0HacTS6blmOk83fbuZKhp2ZF\nT5a93MLhe5VlWSZz/QaS/vc/jBcuAMqeRJ/hw/B55hk05R5cFXzjyas8N0u5Xx/cJIgv+kbadcxl\nUUGDxZKZYyM8PK079J8JGp3S8HrZi/kreY/izNVMPliq9MJpVdWP158sXMsJbdWqBE7+jtAli3Fv\nqxTgMERFcXHECOJffQ1jXNwjj/VhSJLE1/0iqeilw2C28upfB8k1Wh76OocTDzN01VDe3/E+V3Ou\nopE0DKo+iNV9VzOu8TgRKAq4Obkxpu4Y1vRdw6jao9CqtaQb0pl0YBK9l/fmn4v/UOBJPatVST3N\nugJObsqMfGkIFEHptdj9f8px7HbYO80ml/3xn3OcuZqFJMFPQxuWikARwFmj4pv+ddE5qUjKNPDJ\n3yccPSRBeDCTHpa8oASK5UKUPcqlRbOXIHKQcrz5/+D0mke+pCzLjF9yjCsZerQaFd8PqufwQDHn\n4CEuDBlKwuuvK4GiJOHVqxfha1YT8PbbBQoUAdrXKs9rT1QF4K+9cczbe9GewxbuQwSLZUn52tDl\na+X4zFrY/eMjXS7bYObFOQfINVko76nlu0H1UD9iIQVdrVoE/TKV4Llz0NWuDUDmhg1Ed+1G4qRJ\nWPKqZhUFb1dnJg+qj0qCM1ez+GzVyQK/NiErgbFbxzJszTCOX1OC6ceDHmdpr6V80OwDAlwD7DVs\noYTy1nnzVqO3WN13Nf0j+qOSVMRlxvH65td5YcMLnE09++CLHJqlBFIAvadAhTr2HXRRq91H6VkG\nsPFjSC7Az+Q+TlxK5+ct5wF4pmUozcNL1+RNiJ8b7+XtV1x2+BJrj19x8IgE4QE2fapUcJdU0OeX\nktEio6AkCXp8B5Xy2mAtfwVyUh7pkgsPxLMm7/f6w241qVbecT8v48WLxL/2OheGDCH30CEA3Fq0\nIHTJYip99SVOlR6+XdTr7SPytzV9tPwER+LSbDpmoWBEsFjWNBgOj/VXjjd+opSlLgRZlnlvyTHO\nJ2WjVkn8OKQBfu6264Hm2rAhIQsXUPHzz1H7+yGbTFybPoPznbuQtmgRsuXhV/kKo0moD6/lrZb+\nuecia45dvu/5WcYsvjvwHT2X9mRdrJJSG1Eugukdp/PDEz/Yt02CUCoEuAbwUfOPWNB9AU0qKCnY\n/17+l/4r+jPx34mk6e/xj2XGZVj/kXJcq5cSWJVGXf+rtAIy62HpaGU/YyGYLFbeWXQUs1Um2NeV\nsR1LaPGMBxjePISmocqWgw+XHSMl2+jgEQnCPURvgX+nKMet3rR5bYViwckF+v8BTq6QkwzrPyz0\npWKTs/lPXsbAEzUCeLrZo9dWKAyrXk/SDz8S3b0HmevXA6CNiCBo+nSq/PYrupo1C31ttUpi8qB6\nBPm4YLRYeXvhEYxmq62GLhSQCBbLGklSqiL6VgWrGRY9o1Toekh/H7nEiiOXAHivcw0a26EfoaRS\n4d23D+Fr1uI7ejSSszOW5GQufziBmP79yTl40ObveTevtKtKk7zv793FR4lPzbnjHFmWWROzhh7L\nevDr8V8xWo346Hz4pPknLOi+gGYVS+E/eoJdVfepzoyOM/j28W+p7F4Zi2xh3ul5dFvajT+j/sRs\nvSlIkmVYPRYM6Up59i7/ddzA7c3VB3rmZUUkHIAd3xbqMtO2RXPiklJo6su+kbg4l87iCSqVxH/7\n1cXVWU1ylpGPlh939JAE4U65abDsJeW4QiS0fc+x47GncsHwRF6QeHiuEiQ/JJPFyhvzD5NttODn\n7sxXT0U6pM1P5ubNRHfvQfKUKchGI2p/PypOnEjo0iW4t25lk/fwdnXm+0H1kSQ4l5jFtG3nbXJd\noeBEsFgWaT2U/YtqLaRfhDUP96GcoTfx2cooANpG+PNca/s2rVa7uxHw5huErV6NR+fOABhORnFh\nyFAuT/gIS5p90xI0ahXfDaqHl4sTGXozb8w7jNlyY2brYsZFxmwcwzvb3iE5NxlnlTPP1nmWVX1W\n8VTEU6hVpfMmVLA/SZJoH9ye5b2X83qD13HRuJBhzOCLvV8wcOVAjiYp/fSI+htOrVSOO05U9veV\nZtXaK+1AALZ+qRTEeAhnr2YyeaOSwvp0syqlLv30dlV8XRnfRUlHXXn0MquO3j9DQhCK3OpxSj0F\ntRb6Ti/Z1ZsLoumYG+moK94AU+5DvfyHTWc5nJeS+d9+dfH3sF1mV0EY4+OJe+ll4l98CVN8PKjV\n+IwYQfiaNXg/1RdJbdv7nvpVyjEsb+X0+3/OEZucbdPrC/cngsWyqsJj0OFT5fjoPLiwq8Av/d/6\nMyRnGdBqVHzWq+iaVjsHVibwu28Jnj0LbXUlZSxt4ULOd+tO+oqVBS8CUgiVvF34up9SiWv/hVSm\nb4/BaDEy9chU+izvw65Lys+vZeWWLOu1jDcavoG7s+37KAllk1at5bnHnmNln5X0DO8JwJnUMzy9\n+mkm7vyIrDV5VfVC20L9px040iLU8f+UAhhWMywdoxTGKACLVWbcoqMYLVYqe7vwXpfCp0iVJEOb\nBtMiLyiesPw4yVkGB49IEPIcXwLHFijHHf4DAWWgL6hKDT2+V9qZpcbA1q8K/NJ9sSn8uPkcACOa\nB9OuRtHVQLAaDCT99BPR3bqT9c8/ALg0bEjoksWUH/8eanf73feM7VSdAA8tRrOVCcuP2/WeT7iV\nCBbLsibPK0EjwKqxBdr7czwhnVm7YwElPbOKr6v9xncPro0bE7poIQHjxiLpdFiuXePSuHHEPfts\nfolme+hUuwKDmwQB8MPuNfRe1pcph6dgtBrxd/Hnm7bf8POTPxPkGWS3MQhlW4BrABNbTeSPzn8Q\n7hWOjMy8c0vp5a1mo4cXcvfvbNq7q1jTukPvqYAEiSdh88QCvez3nTH5M/Kf930Md63GjoMsPlQq\nia+eisTNWU1KtpEJy8TNllAMZFyClW8qx6Ftoclox46nKFWMhBavKMc7v4crD04RzzGaeXP+Yawy\nVAtwZ3zXopvsyv53D9E9e5L8/Q/IBgNqX18qffUlwXNmo6tu/z3fnjonPu6hFD7cfjaZv/O2Qgn2\nJ4LFskylhq6TlOPEE7Bv+n1Pt1hlPlh6DKsMYX5uvNA2rAgGeXeSkxO+zz5L2MqVuLVtA0D2rt1E\n9+hJ8tSpyEb7FHF4vm0AnkELUVf6hbisC6gkFUNqDOHv3n/TKaSTQ/YMCGVPg/INWNhjIa8Ed8PZ\nKpOo0fCmnxevHfmWK9llqOJlcHNo8apyvOsHuLD7vqfHJmfzzfrTAPRvGJhfZa+sCPJxzW/Wveb4\nFVaIdFTBkWQZ/n4V9GnKXuveP9ukd2qJ0vY9JUNCtig/C+v9i/dN3XKe+NRcnNQSkwfVL5JG9ZbM\nTC5/9DEXR47EdOEiqFSUe/ppwtesxqtXryK97+n6WAUer658bn+28iTpOaYie++yzOa/lZIkqSVJ\n+q8kSUmSJGVKkrRYkqS7NvqSJOl9SZKybvsjS5L0/U3nxEqSpL/tnMdsPe4yq0pTqJeXtrb5c6Xx\n9T3M23eRI/HpAHzaq47De/mAkpoaNHUqlb/7Do2/P7LRSNJ3k4nu05fcIw+3j+lB1sauZcSG/sju\nBwCw5FZmfN2fGN90vEg5FYqck8XE6CNrWJxwmSYWZXVsS9wWei3rxZyTc7A84Kaj1Gj3AQTUAmSl\nf+w90lGtVpl3Fx9Fb7Li76Hlw7ygqawZ3CSI1tWUf5I/Wn6cxMyCpe8Kgs2dWgXnNirHXSeBV2XH\njscRnF2VooMAlw7C3ntP2sel5PDLtmgARrUMpVYlT7sPL3PLFqK79yBtgZImrK1Vk9BFC6nw4Qeo\nPe3//reTJInPetVB56QiOcvIl2tPFfkYyiJ7TOG8B/QCmgKBeY/NvtuJsix/Lsuy+/U/QH1ABubc\ndupzN58ny/IxO4y77Gr/iTKrZ8iADR/d9ZTkLANfrVF+KXvUrUSrasWn0bckSXh27kTY6lWUGzIE\nJAnj+fPEDh5C4qRJWA2PtjfnWu413tryFuO2jiPNkIarxhXvnP7kxL7MzM2WW4rdCEKR2fI5pMYS\nYoUZXf7g/1r+H95ab3LMOXy17ytGrh3JhQz7pWUXG0466DP1xt6fPT/f9bS5ey+yJ0bpaTaxdx28\nXJ2KcpTFhiRJfPlUJO5aDWk5JiYsE9VRBQcwG260jQhtA4/1c+x4HCn8CYgcpBxv+hTS4u562hdr\nojCYrfi5O/NKXrN6ezGnppIw7h3ix7yI+epVJGdn/N98k9D589HVcuxEW5CPK2+0jwDgr70X2R/7\naL0qhQezR7D4AvCVLMvRsiynA+8AnSVJCinAa0cDh2VZ3muHcQn34u4PT0xQjo/Og9idd5zyxepT\nZOjNuGs1TOhWPAtCqD08qPDRBELm/YVzeDhYrVybPoOYp54i91jhbojWxa6jz/I+bLiwAYBmFZux\ntNdSvu3yCqDi1JVM5u65aMPvQhAK4NIh2J3Xj6zl60gVI+lVtRd/9/47vwDO4aTD9Pu7H39G/YlV\nLuUTGhXrQsORyvG2SZCVdMvT8ak5fLlaqeDco24lOtauUMQDLF4qe7vwYd7n+LoTV9l1PtnBIxLK\nnD2/KJM7kgo6fVF29lrfS6fPwcUHTNmw6m0lRfcmu89fY/UxZYvBuE7V8dTZZ7JLlmUy1qwhult3\nMlasAMClfn1Cly3Fb/QLSE7FY5Lt2Vah1KjgAcAHS49jEpP2dmXTYFGSJC+gCnDg+mOyLJ8HMoDI\nB7xWC4wEpt7l6f9JkpQiSdJhSZIeevezJEm+kiRFSJIUYTYXroFzqdfoGaW3ESj92iw38sD3RF9j\n8cF4AN7uGEGAp84RIywwl7p1CV2yGJ9nn1FWGc+dJ3bQIBInTy7wXsYUfQpvb3mbsVvHkmpIxVXj\nyoRmE5jWYRqV3CvRKMSHPvWVlJlJ609zTVQWFIqKxQTLXwXZCr7VoM07+U+V05VjYquJ/PjEj/i5\n+KG36Pli7xc8v/55LmWV8mIA7d4HrScYM+8odvP56iiyjRZ83Jz5pEfZTD+93YBGQUQGegEwcVUU\nVqsodiMUkawk2JbXC7bBCKhQx7HjKQ7cfKHzF8rx2XVwYmn+U2aLlf+sOAHAY5W96N/QPkX0zCkp\nJLz2OglvvoUlJQXJxYXy779P8JzZaMMcV6PibpzUKib2UXaknb6ayfTt0Q4eUelm65XF6wnM6bc9\nnnbTc/fSD3AG/rzt8RFAGFAeGAd8XoiA8VXgNHA6MTHxIV9aRqjU0O16sZuT+XnzJotSohigVkXP\n/D43xZ1Kq6X8uHEEz52Lc3AwWCxc+3kqMf0HoD958r6vXR+7nj7L+7D+wnoAmlZsytJeSxlQfcAt\nG7nHd6mBm7OaDL05v2iGINjd3ulwNS8Tv+f3ShrmbdoGtWVpz6V0Ce2ivOTKXvr+3ZfFZxaX3gqY\nbn7QZqxyfPAPuKr8nh+8mJo/I/9elxr4uhdtP7LiSqWSeD+vkuKJSxksPZTg4BEJZcbmicq2F62n\nsudYUEQOhLB2yvGadyE3FYB5++I4dSUTgI971EKlsv0qbNa2bUT37EXmBiWLyrV5M8JW/I3P8GE2\n75loKw2DyzG0aRUAJm88y8VrOQ4eUell62AxM++/Xrc97o2yung/o4G5sixn3fygLMtbZVnOkmXZ\nJMvyBuB/wMM2EvsBqA5UDwgoun40JU5QkxvFbrZ8AZlX+G1HDGeuZiFJMLFPHTTqklWpzLWBkj7h\nM2I4SBKG06eJGTCQpClTkG9bZc4yZjF++3je3vo2KfqU/NXE6R2mU8m90h3XDvDU8Xr7aoDyYX40\nPq1IviehDNOn35iRr/80BLe456neOm++bvM1k9pOopy2HNmmbD7Z/QkvbXqJxJxSOmnWdAx4Byur\nrus/QLZa+SIv/bRGBQ+eahD4gAuULc3CfOlQqzwA36w/Ta6xjBRFEhznynFlMgegzThlG4ygkCSl\n2I3GBbITYctXpOeYmJQ3Gd2rnpLVZEvW3FyufPopcS+MxpKcjOTiQoVPPqHKb7/hHFj8Py/f6VwD\nP3ctBrOVD0XvRbux6Z2/LMtpwEWgwfXHJEkKQ1lVPHqv10mSVAtozd1TUG9nBR5qWkWW5WuyLJ+R\nZfmMRlM2emoVWvtP8ovd5Kz6gO82ngVgUOMq1K9SzqFDKyyViwvlx48neNYfOAUFgdlM8g8/cmHY\ncIzxymz64cTD9FvRj5XRKwFoWqEpS3otuWM18XYjW4QS5u+GLMNHy0+IVC7BvnZ+D7kpoNEVeEa+\nY0hHlvRawhNBTwCwI2HHLftwSxWNFjp8qhyf/4dD/yxkX6wyO/9elxqo7TAjX9KN71IDjUricrqe\n33bGOHo4Qmkmy7BuvDKZUy4UmpahnooF5RMKrfL6Tu7/ld9XbSU1x4SLk5r3utSw6VvlHjtOTN+n\nSP3zLwB0dSMJW7qEcoMGlpg2YF4uTnyUt7Vg25kk1p24d0V/ofDssUw0DXhXkqRQSZI8ga+AdbIs\nx97nNaOBf2VZvqXXgSRJwZIktZMkSZfXkqMt8CYw3w7jFuCWYjeupxbxmPk4Pm7OvNvZ/g1X7c21\ncWPCli/De9BAAHIPHSKmd28W//QmI9eOJCErAY1Kw9hGY5nWcRqV3R9cxttZo+KTvCaxh+PS8vd2\nCoLNZV65UdSm2Yvgeedq9734ufjxXbvv+LzV53g4eZBhzOCtLW/xn93/Ideca6cBO0itXlBFWXH1\n3fUpGsy0rOpb5noqFlSYv3t+KtdPm8+RlCn2Xwt2cno1xGxTjjv+nzK5I9yp+cvg5g8WI1WOfgfA\ni4+HU9HLxSaXl81mkqdOJXbwYIwxMaBW4/fKK4TMnYtzSIhN3qMo9YisSItwX0CpIWERk/Y2Z49g\n8UtgBbAPSADU5KWNSpI0VJKkW9JMJUlyAYZx91VFN5S00yQgFZgCfCrL8g92GLdwXaNnyPFRAqBP\nnWYyvlM43q7Ojh2TjahcXan4yScETvkRycsDa1YWtb5fywsrjFTXBfNXt78YUXsEKqngvxptIvzp\nVFtJ5fpq7Sky9KJJrGAHW74Ecy7ovKHlGw/9ckmS6BHegyW9ltCwfEMAFp1ZxKCVgzidUor23EoS\ndFIK3ARb4xms/ofxXWqWmJlyR3i9fQQeWg3ZRgvfbTzj6OEIpZHZAOvysiFCWkONbo4dT3GmdUdu\nMw6A3qodtPZM5IU2tikwY4yP58Kw4SR9NxnMZpyCqxDy51z8X3kZqYRm3kmSxLudlVXXs4lZYv+1\nHdg8WJRl2SLL8lhZlv1kWfaQZbmvLMvJec/NzeunePP5ubIs+8iy/MddrnVSluX6edfxlGW5jizL\nP9p6zMJtVGq+VD0HQA1VHE9JWx08INvbFqrnzVESx4KVG8h2x2S++NVIcELhAr0Pu9VCq1GaxH63\n4awthyoIkHwWDs5SjtuMBRfvQl+qglsFfu34Ky/XexmVpCI6PZohq4YwN2puqdnvke0XySqpLQDv\n6pZQx6d0fF/24uPmzMt5fdv+2nuRs1czH/AKQXhIe6fdaJXR+UvRKuMBNrl25YI1AJUk843PMnRO\nj15kJmPtOmJ69yH30CEAvAcMIGzJElzq1n3kazta3SBvOue1RPp2wxkMZrH/2pZKVrUSoUjsPJfM\nrPjyLLcoqVyq7d8os4KlQI4ph/e3v897298j3iWHH0f4kPFcb9BoMF+MI3bwEK7NmIFsfbiePUE+\nrrz4eDgAf+yOJSY52w6jF8qsTf8B2QJeQdD4+Ue+nFqlZkzdMczsPJOKbhUxWo18ufdLXv3nVVL0\nJb/B8fTt0XyW249c2Rl3S8aNokDCPY1sEUJlbxesMnyx5pSjhyOUJtnJsPVr5bjBcNEq4wEMZguf\nrT3HJHN/AMpf2QIXdhf6elaDgSuffkrCG29gzcpCXa4cgT/9RMVP/4PKzc1Go3a8sZ0iUEmQkJbL\nn6L/tU2JYFG4hSzL/HedkpK2zm8ksqSCjHg4cMfCb4lzNvUsg1YNYkW00mi2ecXmLOq9hKZjvyDk\nrz9xCq4CZjOJ30wi7rnnMac83E3zmLbhVPDUYbHK/LBJrC4KNhK3D6KUv7O0++CurTIKq35AfRb1\nXESnkE4AbI3fSr+/+7Hn8h6bvUdRS8zUM21bNFfwZX/lvOrOe36BFNGH6350Tmreydub/s+pRHae\nS3bwiIRS43qrDGcPaPeho0dT7P2+M5YL13JYJTcn11fZEsTGj5UCQQ/JGBtL7ODB+UVsXBo1JHTZ\nUjyeaGfLIRcLVQM86JtX8frHf86RbRB91W1FBIvCLTZFJXI4TmkBMbTbk0h1BytPbP8GjCW3h83y\nc8sZsmoIMekxqCU1bzV8i6kdphLgqrRScXnsMUIXL8Grd28AsnftIqZPX3IOHizwe+ic1PmpXMsO\nJ3AuMesBrxCEB5Bl2PCRchxQGyIH2PwtPJ09+W+b//Jpi09x0biQlJvE8+uf55cjv2CVH26FvTiY\nvPEsOUYLnjoNjw34CNwrgNV04+co3FPPupWoG6SkOP/fqihRKEJ4dFeOw4GZynFb0SrjQdJzTfy0\n+RwAA5uE4NI5r7pz3B44s/bhrrVyFTF9n8JwMgokCd8XxxA8cyZO5cvbetjFxhvtq+GsVnEt28hv\nO0R1Z1sRwaKQz2qV85vLNw/zpWVVP2j7Dqg0kHUV9v/q4BE+PL1Zz8e7PubDnR+it+gJcA3g986/\nM6rOqDuK2Kjd3aj05RdU/PILJJ0O89WrXBg+gmu/zyzwXq4BjQLzU7m+F6uLwqM6ux4u7lKO238C\nKvs0R5YkiT7V+jCv+zwiykUgI/Pj4R95edPLpOlLTv/Qc4lZzNsXB8ArT1TF27scPJkXJEatgNid\nDhxd8SdJEh92qwlA1OUMlojqzsKj2vSfvFYZIUofVOG+Zu6MJUNvRqtR8Ub7alD1SQhupTy58T9g\nffBePKtez+UJH3Fp7FisOTmofX0JmjGdgNdfL7FFbAoqsJwrQ/KqO0/bFk1qttHBIyodRLAo5Ft1\n7DKnriiFDcZ2ymuVUS4E6g9Tjnd8C4aSU/ggNj2WoauHsuTsEgBaVGrBwh4LqR9Q/76v8+7dm5AF\n83EODVXSUr/6ioTXXsOSkfHA99Rq1LySt7q44uglzohCEUJhWS2w8RPlOLgVVOtg97cM8wpjbte5\n9K6qrLDvSNjBgJUDOJZ0zO7vbQtfrz2FxSpT2duF4c1DlAfrDoaKeQUcCpnKVZY0DvHJLxTxzfrT\n5BpFoQihkBIOKBNeoLTkEq0y7is918SvO5R0+aebBRPgoVMKAbX/RDkhKQqO3r9znCEmhtgBA0lb\nuBAA12bNCF26BPeWLe048uLllSeq4uqsJtNgZurW844eTqkggkUBALPFyrcblJLpT9QIoGFwuRtP\nthkLamfIuabs/SkB1sauZdCqQZxJPYNKUvFyvZf56cmf8NH5FOj1uogIQhYuxLNrFwAyN2wkpl9/\n9FFRD3xtv4aBBPm4IMtKSpwgFMqReZB4Ujnu8J8iqx6o0+j4rOVnfNriU7RqLZezLzN87XDmnZpX\nrKul7otNYf1JpSHz2E4RN6oHqlQ3brbi90H0FkcMr0R5t0sNNCqJqxkGftspUrmEQtqaV1jKrzrU\n7uPYsZQAN68qjm57U6uMoMZQo7tyvPlzMOnv+vrMTZuI7T8Aw5kzoFLh9+orVPl1Bk4BAUUw+uLD\nz13Ls61CAZi5K5Yr6Xf/eQkFJ4JFAYAlBxOIzqvg+XbHiFuf9AqERs8ox7u+h9zim5Zmspr4cu+X\njNs6jmxTNj46H6Z1mMaYumNQP2QKn9rdjUqTJlF+wofg5ITp4kViBw4idcGC+940O6lVvPpENUBZ\nrY26/OAVSUG4hSlXKQoBULMnBDYq8iH0qdaHOV3nEOgeiNlqZuKeiby3/T1yTMVv77Isy3y+WpnI\nqVXRk151K996Qlg7CGysHG/9SqwuPkConxtPNwsGYMb2aFEoQnh4lw7DmTXKcdt37JZCX1rcdVXx\nZk9+pLQdSY+7Y0uQbLGQOHky8S+/olQ79fGhym+/4f/yy0jqsvlzf75NGN6uThjMVr7/R0zaPyoR\nLAoYzBYm5+2v6/ZYRWpX8rrzpFZvgcYF9Onw709FPMKCSc5N5vn1zzM3ai4Ajco3YlGPRTSt2LTQ\n15QkCZ+hQwmZOwdNpYrIRiNXPvqYyxMmYDXeOxe+b/3KBPu6Aogm18LD2zsdMhJAUt/Yc+cANXxq\nML/HfNoFKZXzVsesZvCqwUSnF6/KopuiEjl0UZnEer9rTVSq21ZhJQnavqscX9wNsTuKeIQlz5i2\n4TirVaTmmJi754KjhyOUNNdbZfhWE6uKBXDPVcXr/KtDvSHK8bZvQK9MQlvS0ogbPYZrP08FQBcZ\nSeiSxbg1K/x9T2ngqXPixbZKO7MF++KIFe3MHokIFgXm7Y0jIS0XlQRvdoi4+0ke5aHJc8rx7p8g\np3j1YjuefJxBKwdx4OoBAEbWHsn0jtPxd7VN5TWXyEjClizBrW0bANIXLebisOGYribe9XyNWsXr\nTyqri+tOXOV4QrpNxiGUAfp02D5JOW4wHPyqOXQ4ns6eTG43mbcavoVaUhOdHs3QVUPZGrfVoeO6\nTiFuHn4AACAASURBVJZlfsyrHtgszIdW1fzufmLV9lApb7/ytq+LaHQlVwUvHf0bKWXop22LQW8S\nexeFArp8FE6vUo7bjBOrig/wwFXF6x4fD2ot5KbArh/QnzxJTL/+ZO9QJr+8Bw4keM5snCpUKKqh\nF2sjWoRQ3lOL2Srzvw1i0v5RiGCxjMs1WvJvtPrUD6RqgPu9T275Bji7gzETdk4uohE+2LJzyxix\nZgRXc66iU+v4us3XvN3obTQq21b9Unt7E/Tzz/i99CIAuUeOENPvKXIOHbrr+T3rViLMX2l4K1YX\nhQLb9yvo00Cjg8ffc/RoAGWFfVSdUczoOAMfnQ9Zpixe/edVfjnyi8P3Me4+fy2/3c8r7e4TWEsS\ntHlHOY7Z9khNrsuKMW3DUaskkrMMzNsrmlwLBXR9MsYnHOo85dixlAC/74whQ29G53SPVcXrvAKh\n6QsApM2ZRuzgwZji/5+9846Oqsz7+OeZkt5IJaTTewlVBbuAva2L0kEUdNW1rbu+6+6677vFXXV1\ndVWkV8GCrl2siPTeAtLTeyAhPVPu+8czk2AkmSTMzJ2Z3M85OfOcmVt+B8dn7q99f7kIPz/i//J/\nxP/5WXR+fm6y2vMJMOr59TUyAfLR/nwO52stQR1FcxY7Ocu3ZlJSWY9RL6RMc2sERzdJX+9YAFUX\nzqq5C5PVxN+2/40/bP4DDdYGEkISWHXDKq5Pu95l9xQ6HTGPPELCq6+gCwrCUlJK1vQZnH3nnZ8d\ne3528esjxezP8dxeTw0PwVTbVOY9bBqEelaEeETXEbx909v0j+rfOF7jie+fULWP0R7sGpIUwWU9\no1o/uM/1EDdIrrXsokOSIoO4fZjs/3xz4ynqzVp2UcMBhYfkmBqwieP59qiGi0VmFaWI1NTRrWQV\nbSijH6FwbzQFW4JQ6hswxMeTsno1Eb/4hTvM9TruGpFIqq0lyD4aTqP9aM5iJ+ZcnalRVnjSyCSS\nIoMcn3TpQ+AfDqYaOUpDJUprS5mzfg5rflwDwJj4May9cS19Ivu45f5h111H6jtvY0xJBpNJ9jE+\n+yxKsz7GmwZ3a8zWatlFDYfsXQXVJXK26WWPqG3NBeka3JXlE5dzU3epzvdV1ldM/XwqOZU5brdl\nT/ZZtpwsA+Chq3oiHCnGCiEHgwOc/BZyd7nYQu/nwSt7oBNQUFHHut15apuj4elstCmgdkmDQb9U\n1xYvYOnm01Tasor3t5ZVBMxnz5L9yFOcPSqzh0HxFtJWLyVw0EB3mOqVGPU6Hh8vnwu//bGYg7la\nS1BH0JzFTsySTacprzHhb2hS73RIYBfpMIIslzuX7zoDWyCjNINJn0xiT/EeAGYNmMUb175BRECE\nW+3w79mTtHffJfjycQCUr32brJmzMJeUNB6j1zVlbL87WsKe7LNutVHDi7CYmsq7B90FEcnq2tMK\nAYYA/jb2bzw54kl0Qsfxs8e5+5O72Zrv3tLO176VWcW+XUO5pm8b5eH73gwxcvB8owiHRot0jwnh\npsHdAHh9wwlMFqvKFml4LMVH4PCHcq1lFR3Snqxi/YkTZP5yEjXbtwMQ2beW5HFFGE7/1y22ejM3\nDYpvbAmav1Gbu9gRNGexk1LTYGbZlkwApo1JIS6s9dKHnzB6nnQaLfVNQhxuYn3memZ+MZPimuLG\n/sTHRzzu9P7EtqIPCyPpjTeIul/2EdTu2cPpX9xFbUZG4zE3DIynT1woQOMsSw2Nn3HwPSmLjoCx\nj6ltjUOEEMwYMIP5184nzC+Mcw3nmPf1PJZnLHdLH+Ph/HN886MshX/gyh4/V0BtCZ2uKbt4fD3k\nX7jnWKOJX13VE4Dcs7V8uM/9AUINL+H7fwIKRKTA4ElqW+PxtDWrWLlhA5mT7saUk4MwGon/+9+J\nm3M7Qgdsfa3FuYsaEp1OMPdy+e/7+cECTRm1A2jOYifl7Z05lNeYMOoFc8a1XvrwMwLCpNgNwO7l\ncK7A+QY2Q1EU5u+fz5PfP0mdpY6uwV1ZecNKl/YnthWh1xP7+GMkvPwyIigIc1ERWVOnce7LLwG5\nUT12ncwu/nC8lJ2ZnqUkq+EBWK1NZd39bpIy6V7CJd0uYe1Na+kZ0ROrYuWFXS/wpy1/wmQxufS+\nr2+QWcXUqKDGzFeb6X+blPSHpsHhGi3Sp2soEwfI/tnXvzuBxarNqdRoRslRyPhArsc9AXqjuvZ4\nOG3JKiqKQtnixeQ+8CDW6mr0UVEkL19OxO23waUPy7mLVUWw/y13m+913DYsgbgwf6wKLPjBs0Y/\neQOas9gJMVmsLPpBblK3D0uga3g7sop2Rs6BgAiwmmD7fCdb+FPqzHX89off8tq+1wAYHDOYNTeu\noW9kX5fet72ETZxA6lurMcTHo9TWkvfIryl9cwGKojC+f1f6x4cBWu+ixgU4+imU2prvxz6uri0d\nICk0idU3rOba5GsB+ODEB8z9ei4V9a7pDzlVUsWnB2WQ6oErpWJnu9DppaQ/yH/7woNOttD3eOhq\nmV08VVrNZwddHyDU8DI2Pg8oEJ4MQ+5R2xqPx1FW0VpfT8Hvfkfx8y+AouDfrx9p775DULpt/E9k\ndxhwh1xv/jdYzG603vvwN+iZM1b+O7+3K5fic1o2tj1ozmIn5NMDBeSV1wJw/+XtzCra8Q+RDiPA\nrqVQX+kk635KaW0p966/l89Pfw7Ajd1vZMmEJUQHtjBLTWUC+vYl7Z23CRg8GICSl16i4HdPg9nE\nr229i5tPlGlzFzWaUJSmcu7uV0FCurr2dJAgYxAvXvkicwbJfWFn4U6mfDaFzIpMp9/rjQ0nURTo\nFh7A7cMSO3aRgXfKBy5oEuXQaJGBCeFcbesL/c+3J7Bq2UUNO6XH4dA6uR73GBi08Q2t4SiraC4p\nIXv6DCo+/AiA0AkTSF29CmO3ZhUU9naFs5lwWOtddMQ9o5MJCzDQYLGyZHOm2uZ4FZqz2MlQFKVR\nAfXafnH0jA3t+MVGz5UDYusrZDmqkzl65ij3fHoPB0oPAPDwsIf5+9i/46/3d/q9nIkhJoaUFcsJ\nvX4iABUffkj2zFlcFWeke7Rssl6olUFo2Dn1XVPf3Lgn1LXlItEJHb9O/zV/uewvGHQGss5lMeWz\nKewo2OG0e+SereGDvVKV8/7Lu+Nn6ODPmN7Q9O99+EMpzqHRKvbexaNFlXx1pEhlazQ8ho0vgGKF\nsEQYOlVtazyeFVsyW8wq1h07xulJk6jdvx+A6IceIuGlf6ELuoBafdeB0GuCXG96WQYeNVokxN/A\n9EtSAVi9LYtzda5tlfAlNGexk7HxeCk/Fsos4ANXdjCraCckFobayk22vS7VHJ3Et9nfMu3zaRRW\nFxKgD+DFK17k/sH3O5bG9xB0AQEkvPgi0Q8+CEjhm+y77+ahNPm/3CcHCsg9q95sOg0P4od/ydfE\nkZA6Vl1bnMStPW9l0fhFRPhHcK7hHHO/msu6Y+uccu0FG09htipEh/hx96iLVIwdPKlJdXbjCxdv\nnI8zPKVL4yzL/3x7wi1CRhoeTtlJOGibM6xlFR1SZ7KwfGsmAHePTP5JVrFq02ayJk/BnF+ACAgg\n4eWXiHnoVwhdK4/q42xtC0UH4fhXrjPcR5h5WSr+Bh2V9WZWb8tW2xyvQXMWOxnzN8is4oiULgxP\nibz4C17yMCDgXF5TGcpFoCgKKzJW8Oh3j1JrriU2MJZl1y9jfOr4i7fVzQidjphHHqbbCy8g/Pww\n5ebS/+9PcFWFFIhYqpVBaOTsgMwf5HrcE3IOoI8wPG44b93wFmnhaZgVM89ufZYXd72Ixdrxwe7F\nlXWs3SnnOc4em0aAUX9xRuqNTT2ih9bJcjqNVrGPWTqYV8GGYyUOjtbweba+JrOKofEwbJra1ng8\nH+7Lo7SqAZ2Ae8emNb5/9p13yJk7F2tVFfqoKFJWLCds4kTHF0weA8mXyPWmf7nIat8hOsSfSSOT\nAFi86TR1po7/HnUmNGexE7E/p5ytp+QA67lX9HDORaN7Qt8b5XrzKxdVBmGxWnhux3M8v+t5FBT6\nR/VnzU1rGBA1wDm2qkT4TTeSsmI5+qgolKoqnvx+ARMyt7N2RzYVtVoZRKfGnlWMHdBUTuRDJIUl\nseqGVYyJHwPAsoxlPLbhMWrNtR263uJNp2kwWwkLMDBtTIpzjBw6RZbPocCWV51zTR9mdFokI1O7\nAPDqN8e17GJnpuYM7LMpcY6eBwbPbhFRG0VRGsUFrx8YT1JkEIrVSvGLL1L4xz+BxYJfzx6kvv02\ngTbdgzZhD3hlb4Us98669UbuG9cdvU5QWlXP+3vy1DbHK3C6syiE0AshnhdClAghKoUQ64QQF1Qj\nEUJcKYRQhBBV5/1taXZMTyHE10KIaiFErhDCu5t6VGTBRtkn1zM2pO0DrNvCpY/I1+IMOPFNhy5R\na67lsQ2P8daP8ofnysQrWTphKbFBTrRTRQKHDiXtnbfx79UTndXKo/ve5c59n7Bme5bapmmoRVEG\nHJPCTYx9TM7/80HC/MJ4/drXuav3XQB8l/Md966/l7LasnZdp7ymgVVb5f8vMy9NJTTASdL8Bj8Y\nM0+uD7wN1aXOua6PIoRozC7uyS5n+2ltFFCnZddiMNeCMRiGz1TbGo9nw7ESjhdXATBnXBrWujry\nHn+CsoWLAAi6ZAypb72FX2JC+y7c6zqIGyTX9hFMGi2SFBnETYPjAViw8aQ2CqgNuOLp5HfArcBo\nwC5Tt7KV4y2KooSc93ep/QMhhB74GDgCxAC3AL8VQmjTXttJZmk1nx+Scuf3X9697QOs20LyaEiS\nmQO2/Lvdp5fWljL7i9l8l/MdAJP6TOLlq14myHiBhm4vxpiQQMrq1QSNHg3APce+wfjP/6W+VpNw\n7pTYf9S7pMKA21U1xdUYdUb+MOYPPDniSQAOlh5k6mdT26WUumxLJtUNFoL89My6LM3xCe0hfTr4\nhYC5DnYtce61fZBxvaIZmCBHAdlVHTU6GeZ62LFQrtOnQWCEuvZ4AQttAfuRqV0YFGwle8ZMKr/4\nAoDwO+8g+c030YeFtf/CQsBY2+zr4+uh8JCzTPZZ5tmq6zLLavjiUKHK1ng+rnAW7wf+oSjKKUVR\nKoCngIlCiNQOXOtyIAV4WlGUGkVR9gBvAvOcZWxnYeEPp7AqEBfmz61D2znAui1cZssunt7YpOzY\nBk5VnGLqZ1M5VCY3tyeGP8HvR/8eve4ie5E8FH1YGMkLF6CfeAMAl57ayYEps7CcO6eyZRpu5cyp\nph7fyx6Vypw+jhCCGQNm8PwVz2PUGcmtymXa59PYV7zP4bm1DRaWbckEYMroZLoEO1lEIyBcOowg\nH4DN9c69vo8hhGjst/r6SBGZpdUqW6Thdg6tkwPhhU6WoGq0SkZ+BVtOymqKed0NZE66u1HxNOax\nx4j/y18Qfhexr/W/DbrYgmhadtEh/eLDuLJPDABvfK+JdTnCqc6iECIcSAZ2299TFOUkcA5oqQBb\nL4TIEUIUCiE+FUIMOe+zIcAxRVGqzntvj+399tgVJYToLYTobTZ3vsGlJZX1vLs7F5AN1f4GFzhi\nva+HKFmaxOZX2nTKrsJdTPtsGnlVefjp/HjhiheYOXCm1yiedhTh50evl15gx9hbAQg5vI/MyVMw\n5eerbJmG29j8ihSFCOkKQyerbY1bmZg6kYXjFxLmF0Z5fTlzvpzD11lft3rOB3vzKK8xYdAJ7h17\nkSrOLTF6rnzwrS6Gg++55h4+xI2DuhEb6o+i0OjIa3QSFAW2/Eeu+94EkU7O9Psg9l7Fq8yFJP7h\nEUy5uQg/P7q9+ALRc52g9K43NAXtM96XAUmNVnnAll08lHeOTSe09oPWcHZm0Z4/bz5xvPy8z87n\nR2AokAb0BQ4A3woh7Kmv0HZcqzUeBo4CR4uLi9t5qvezYmsmDWYrof4G7rlYqfmW0Ong0ofl+vB/\n5ZDYVvj89Ofc/9X9nGs4R4R/BIsmLGJCqu8JfLSEEIJBf/wtLw+9C4vQ0XDiBJmT7qbuiDbrzeep\nLoP9a+T6kl91SlGI4XHDWXn9ShJCEqi31PP4hsdZdXjVBY9VFIWlm+WD1o2D4+kaHnDB4y6aLqny\nwRdsCo9apLk1/Aw6ZlyaCsA7u3I0sa7OxKkNUqMA4JKHVDXFGyioqOXj/fmMLsjgiS9fxVpRgS48\nnOSlSwi/8Ubn3WjIZAiJk4HINgbtOzOj0iJJT5bl0/b54xoXxtnOYqXtNbzZ+xHI7OJPUBSlUFGU\n/YqimBVFKVcU5WngDHD9eddr07Uc8CrQB+gTG+sbgiltpbrezAqbKMSUMSnOE4W4EIMnQXCs3Ki2\nvt7iYSsyVvDUxqcwWU0khUq1xGGxw1xnl4eSntyFM1dO5E9jZlNvDMBcUkLWlKlUbdqstmkarmTP\nMtkbZwxuKn3shHSP6M6qG1bRP6o/Cgr/2PkP/rnzn1gV60+O23SitFEUwum9is2xP/gWZ8Dp7117\nLx/gnlHJ+Bt01DRYeHunNrOs07D1NfmaMAKSRqlrixewbEsm157ayh93LEPfUI8hPp7U1asIGj7c\nuTcyBsgAJMiAZI0mPtUaQojG3sXNJ8rYn1OuskWei1OdRUVRyoFsIN3+nhCiOzITeKCNl7EC9nz8\nfqC3ECL4vM+H2d5vj11liqIcUxTlmMHg+71B57N2p4z4+ul1zL4s1bU3MwbIUi6AvSt/tlFZFSv/\n2vUvnt/1PACDogex6oZVpIQ5SQLfC7lvXHd2x/XlicseQImKxlpTQ868eVR8/Inapmm4AosJdkjl\nO4ZO7vSiENGB0SydsJRxCeMAWHl4JU9tfIoGS0PjMUtsAirpyREMTXLxv1fSKEiwPcDZH4g1WiQy\n2I870qWO3fItWZgtVgdnaHg9xT/CCdvw90t+5VOzYV1BZZ0J06I3+fW+99ApCv69e5O6dg3+PXu6\n5obpM2Qg0lwHu5e55h4+xLX94ugZGwI0TQzQ+DmuELhZgFQsTRNChAH/ANYripLZ/EAhxNW20Rg6\nIUSIEOJZIA5YbztkI5AF/E0IESiEGArMRYrcaDjAZLGy+Af55b8jPYHYMBeVb53PyHvlRmWqgZ2L\nmmyxmnhm0zMszVgKwNiEsSwav4jIgEjX2+TBXNsvjrToYE5GJPDWtD/gl5YGZjP5v/kNZ5YvV9s8\nDWdz5COotPWm2gMrnZwgYxCvXP0Kv+j9CwDWZ67nwW8epNpUzcmSKr47Kge/zx7rhr4oIZoi88e/\nhJKjrr+nl3Pv2FQA8sprWZ9RpK4xGq5nmy2IEp4M/W5R1xYPRzGb2f3wb7grQz7SGoaPIGX1Koxx\nca67aWAEDL1HrncuAkvn0+loDzqd4P5xsg/+i4xC8ss7NgPY13GFs/gcctzFTiAP0ANTAYQQU4QQ\n54vVDAG+QZabngLGANcpipIDoCiKBbgZGAiUAZ8BzyuKstYFdvscnx0sIL+iDiHgvstdJArRnMAu\nMHyGXG9/E0y11JhqePjbh/n41McA3NLjFl65+hWfG43REXQ6wZxx8iF4TbYZvzcWETBEakEV/f05\nil98UVPp8iW2zZevPa+D6F7q2uJBGHQG/jjmjzw45EEAthdsZ9YXs3jzB1lE0i08gIkDurrHmH63\nQniSXG97wz339GJ6xoZyRW+pKrh4kxaZ92mqSmD/23I9Zl6nUHHuKNaaGnJ+9RBxP0hH8fTAMfRY\nuhh9aKjrbz7KFog8lwc/fuz6+3k5twztRmSwHxarwqpt2uzrC+F0Z1FRFIuiKE8qihKtKEqooih3\nKIpSavtstaIoIecd+5KiKCmKogQrihKrKMpERVF2NrveCUVRrlEUJUhRlG6KorzgbJt9laWbMwG4\npm8sPWJCWj/YmYx5AIQeako5u2sxc76cw+Y82Yc3e+Bs/nLZXzDqXNg76WXcmZ5IlG2jWn6onJSl\nSwm+XJbllS1cRMHT/4Ni0sQjvJ683ZC7Q67HaFLzzRFC8MDQB3hm9DMIBEfOHOGzsmcQxjKmX5qK\nQe+K2OYF0Btg1P1yvX+NFCTSaBX7GI092eXszT6rsjUaLmPnIrDUg38YDJumtjUei/nsWbJmzaL6\ne9n3/EGPcfR49WV0FzMaoz3E9IYe18j1dq0QzxEBRj13j5QBwjU7sqkzWVS2yPNw06+vhrvZn1PO\nPluz7sxL3SxrHZEMA+8gz6Bn+o8LOFh6EIDfjvwtjw1/zOdHY7SXAKOe6ZekAnKjqhRGkl57jfBb\n5WiNiv/+l5yHHsJaU6OilRoXjT2rGN2n6Ydc42dM6juJF654AR0GhF8pwanzGd7Lzd/99OngFyL7\nfnYtce+9vZBxvaLpZev7WWzrMdXwMUy1Ta0l6dMhoAPD4zsBpsJCsqZOo26/lOlYOOAmjt45hz7x\nzbUaXcyYB+Rr9lbIdzzLtrMzdUwKep3gbI2Jj/ZpY8yaozmLPspy29yrHjHBXNYzyu33PzbgJqbF\nx5GpUzAIPf+8/J9M7T/V7XZ4C9MuSSHAqKO6wcLaHdkIo5H45/5O5L2zAaj+fiPZs2ZjPqtF7b2S\nykLI+ECuR8/VRCEccHXStfiXzkWx+CMMlTyy4X52Fu50fKKzCIxoypzsXAjmevfd2wsRQjT2lH5+\nqJA8re/H9zjwDtSUyqohrd/6gtSfPk3m5Mk0nDyJotPzz+H38H6vK7nfXW1A59PjGoiyieho2UWH\ndIsIbGx1WLolU2v/aYbmLPogpVX1fHKgAIAZl6a6PZO3r3gfM/f+kxKDgSCrldf1yVyfdr3jEzsx\nkcF+3GlTFVy5LQuLVUEIQdxvfkPsb38LQO3+/WRNmYqpoEBNUzU6ws7FYDVBQDgMuVttazyeLw8X\nUVySRE3W/UT4RVJlqmLeV/P4Outr9xkxei4goKoIDq1z3329lNuHJTT2/aywBSs1fASrtUkduP+t\nsnpI4yfUZmSQNWUq5vwChL8/H/7iUb5LGk6/+DAu7eH+gD06XVPv4qH3oKrzzRhvLzNtEwOOFJxj\nZ6YWmD8fzVn0QdZsz6bBYiXE39Aoa+4uNudt5r4v76OyoZJIQxBLCoq45MQPcFZrGnaEvRQ192wt\n3/3YtLFHzZpJt+f/CQYDDadOkTllCg2ZmeoYqdF+TOeVMqbPAL/g1o/XaByXMS5lKG/duIrEkEQa\nrA088f0TrDvmJsctMg363STXW18DLdLcKgFGPVNGSydizY5squs1FUaf4eQ3UGpTBrbPItVopHr7\nDrKnz8By5gy60FD8XnqNBaZ4AO4bl6Ze683Qe2R/qaVBG6PRBkakdGFAN1levWyLVk5/Ppqz6GOY\nLFZWbZeO2S+GJxLi7z61si8yv+Chbx+izlJH1+CuLLt+BQMM4aBYtb6fNtCnayhjustRIiuaKXKF\n33wzSa+/hggIwJxfQOaUqdQdOaKGmRrt5dB7tvItHYy6T21rPJ79OeXsypJR3dlj00gKS2LlDSvp\nG9kXq2Ll2a3PsjzDTWNl7A/GRYfg9Eb33NOLmTYmBaNecK7OzLo9uWqbo+EstrwqX5MvgUQnD5L3\nciq//pqc++7DWl2NPiaalFUrWVPbBUWBmFB/bhrcTT3j/ENhmK39Z+ciMDe0fnwnRwjBjEtTAVif\nUaSV05+H5iz6GOszCik6J/trpl/ivmH37x17j6e+fwqz1UxqWCorr19JWmQfGD5THrBnuWyQ12iV\nGbbs4sZjJZwqqfrJZyGXX07y4kXoQkKwlJWRNX0GNXv2qGClRptRlCZhm743aeVbbWDpZhnR7Rkb\nwuW9ogGIDoxmyYQlpMemA/DCrhd4Zc8rru8rSRoN3eQ9G8vwNFokNiyAm20Px0s3Z2K1atlYr6f4\nCJyWqp6NM0g1AChf9z65j/wapaEBY1ISqatXo6T15O2dOQBMHpWMn0Hlx+xR99FYTn/4Q3Vt8QJu\nGaKN0bgQmrPoY9iFba7oHUN3N43LWHxwMX/e+mcUFPpF9mP59cvpGmybiTZitmyIrz2r9f20gev6\nxxEfHgDI3sXmBA0fTsqK5egjI7FWVpI9+16qfvjB3WZqtJWszVAk1YAb1ek0WqToXF1jv/Wsy37a\nbx3qF8r86+YzNmEsAAsPLuRv2/+GVbG6ziAhmh6Qj6+HspOuu5ePYBe6OV1azbc/an1SXs/OxfI1\nPAn63KCuLR5E2dJlFPz+92C14t+7NymrV+GXnMxH+/M4V2fGoBNMHu0BwcHI7tB7olxv1+bGOiLA\nqOeeUdoYjeZozqIPkZFf0diUO9OWSncliqLwr93/4uU9LwMwPG44SyYsITIgsumg8ISmvp/tb2p9\nPw4w6HVMHiV/YN7blXvBvp+A/v1JWb0KQ7d4lLo6ch78Fec+/9zdpmq0BftQ966DZQmXRqus3JqF\n2aoQHmjkjmE/77cONATyylWvMDFVPvysPbqW32/6PSarC+eQ9r8VQmX/kVZO75iBCeGMTpO/AdoY\nDS+nvhL2r5XrEbNAp1fXHg9AURRKXnmV4n/8A4DA9HRSVq7AGBuLoiis2CqDvBMGdCUuLEBNU5uw\nq9fm7YbcXera4gXYx2iU15j4cF+e2uZ4BJqz6EPYs4qpUUFc0TvGpfeyWC38eeufWXpoKQBXJF7B\n/GvnE+J3gWymfcB14QHI2eFSu3yBu0clY9QLKuvNfLD3whuVf1oaqatX45eWBiYTeY8/wdl33nGz\npRqtcjYLjn4m12Me0MZlOKDOZGG1rd/6nlHJBPpd+MHUqDfy3LjnuLPXnQB8cuoTHt/wOPUWF423\n0Bubyun3roIGbd6pI+61ZRe3nirjeFGlytZodJgDb0NDJej9YNh0ta1RHUVRKH7uOUpffx2A4HHj\nSF68CH24nKG4J7ucjPxzgByH5TF0vxJi+sr1Ni276Ij48EAmDrSN0disjdEAzVn0Gc5WN/ChbZDo\ntEtS0elc92Bqspp4+oenWXdclpXe2P1GXrrqJQIMLUTRUi6D2AFyvWOBy+zyFWJC/blxkMxkqOXH\nSQAAIABJREFUrNja8kZljI8nZfUqAvr3B0Wh8I9/omzxYjdaqtEqOxZIcafgGBh4p9rWeDwf7svj\nbI0JvU447LfW6/T86ZI/MWvgLAA25Gzgwa8fpNpU7Rrj0mfIcvq6cq2cvg1c0y+OhIhAAK3vx1tR\nFNixSK773wYhrg1AezqKxULhH//ImeUrAAgdP57E1/6DLjCw8ZiVWzMB6B0X0phd9wiEaMouHv4v\nnNPGbznCXp33Y2El20+fUdcYD0BzFn2EtTtzqDdbCfLTc9cI143LqLfU8/iGx/k8U5Y9Tuozib+N\n/RtGnbHlk4RoUoE8/F85oFyjVabbNqpjRVVsO9XyRmWIjCR5+TKCRowAoPj5Fyh55VUtEqY29VWw\nZ6Vcj5gNBn917fFwFEVh+RbpVEwc2JVuEYEOzpDKdY8Pf5xfp/8agB2FO5izfg4V9RXONzAsvqmc\nfudCrZzeAfrz+rXW7cnTxmh4I1lboMSmuN3JVZwVk4n83/yG8nffAyD8tttI+NeL6Pz8Go8prarn\ns4Py2WbaJe6fb+2QwZMgIAKsZtilBZUdcf4YjeXa3FjNWfQFzBZrY/T2jvQEwgJacdwughpTDQ9/\n8zAbcjYAMHvgbH4/+vfoRBu+RoN/KQeSW83avJ82MCwpgkEJsrRl5bbMVo/Vh4aStGghwVdcDkDp\n669T/M/nNYdRTQ6shfoK0BlhxL1qW+Px7M0p53CBLN+aPqZ95VtzBs3hmdHPIBAcKjvE7PWzKast\nc76RI+fI14L9kKepEDvilyOSMOoFVfXmxqoXDS9i50L52nUQJI5U1xYVsdbVkfvwI5z7TAbIu0ye\nTPzf/oow/HQs2ds7cxrnW98+LEENU1vHLxjSbaXEu5bK+b8aLSKEaMwurs8o7PRjNDRn0Qf4+khx\n4xfZPnrB2VQ1VPHA1w+wtWArAA8NfYhH0x9te/TMLxiGTZPrXUu0eT8OEEI09jyszyiioKL1jUoX\nEEDSq68Set11AJxZupTCP/8ZxepCpUiNC6MosNMmhDLgNgiNU9ceL8Ae7OodF8KoDpRvTeo7ib+O\n/St6oefY2WPMWj+Louoi5xqZOg6i+8j1zkXOvbYPEhPqz8SBspx+5bYsLXjlTVQWwpGP5XrkfZ22\n39paXU3O3HlUbdgAQNT99xP3h2cQup8+OpstVlbb9rA70xPcOt+6XYy6T877rSnVyunbwM22MRpW\nRYqvdWY0Z9EHsKfIL+sZRa+4UKdfv7yunDlfzmFPsYym/2bEb5g7ZG77yyxG3kvjvJ8jHzndTl/j\nliHdiAgyYrEqvLU92+Hxws+PhJf+RdgtNwNQvvZtCp5+GsWslYC5lZztUJwh1/ZslEaLnK1uaByX\nMWV0SofLt27ucTPPX/E8Bp2B0xWnmfnFTPKqnKhkJ0TTf89D66BG62NxxDRblvhIwTn2ZJerbI1G\nm9m9XFYB+YfDoF+obY0qWCoqyJ59LzXbtwMQ89hjxD7+2AX3p29+LCa/QmbqPErYpjkRydD3RrnW\nSlEdEmDUN6rTr92ZTW1D5x2joTmLXs7Rwkq2npIlV67IKpbWljJr/SwyyjIQCP54yR+ZPqCDqmiR\n3aHXeLnesdB5RvooAUY9k0Y2zfupNzveqITBQLfnniPil78EoOLDj8h74kmUBi2T6zbsc8liB8ih\n7hqt8t7uXBrMVgKNem5Pv7jyretSruPfV/0bP50fuVW5zPh8BpkVmc4xFGDIJDAGg6VeKqNqtMrI\n1C70sQUwNaEbL8Figt1S5ZxhU2RVUCfDfPYsWTNnUbt/PwBxf3iG6Ln3t3i8Pet0aY8oesY6P2Dv\nVOxtEXm7IX+furZ4AVPGJDeO0fhof+cdo6E5i17O8q2ZACR2CeSafs4tdyusLmTmFzM5UX4CvdDz\n17F/5a7ed13cRe1jNHK2yd4fjVaZOjoFIaC0qoHPD7ZNGEjodHT987NEzpBOfeX69eQ8/DDWOq1H\nweVUl0kRJ5BzyTpp+VZbsVoVVtnGZdw2rJtT+q0vT7yc1659jUBDIEU1Rcz8YibHzx6/6OsCsu96\nsAzEsGsxaGXerSKEYOoYGZn/9EABZVUuGm+i4TyOfgaVNrXMTthvbS4pIXv6dOqPHAGdjvi//Y3I\nKVNaPP5EcRWbTpQCMN1FbUBOJe0KiOwh19rcWIfEhwcyYYB8tl7dhgovX0VzFr2YiloTH+yRkY5p\ntiGiziK3MpeZX8wk61wWBp2BF694kZt73HzxF+5xddNGpY3RcEhSZBDX9JUblT0w0BaEEMT+7ndE\nzZNy2dXfbyRn3gNYq100WkBDsm8VWBpk9mnwJLWt8Xg2nSglq0zOLZwy2nnlW2Pix/DmdW8SYgyh\nrK6M2etnk1GW4ZyL20tRz2bCyW+cc00f5rZhCQT76WmwWHl3d67a5mg4wt6P2/0qiO6pri1uxlRU\nRNa06dQfPwF6Pd2e/ycRd9ze6jn2jHm38ACu7RfrDjMvDp1OBjIBDr4HdS5Qj/Yx7L9NB3IrOJDb\nOcvpNWfRi/nv3jxqTRb8DLrGckVnkHUuq7Hfx1/vz6tXv8o1Kdc45+I6XZMM98H3tL6fNjDjUrlR\n7c0u52Bu2zd2IQSxjz5KzGOPAVCzbRvZ992PparKJXZ2eqxWqTIHNvXfMHXt8QLsD1rDkiMYaFP/\ndRbDYoexaPwiwv3DKa8vZ876OewrdkLZVdeBkHyJXGtCNw4JDTA2lhev3p6FxaoJ3XgsJUfh9Ea5\n7mT91qa8PLKmTqMhMxOMRhJefonwG29s9ZzqejPrbAGQyaOTMei95JF66BTQ+4OpGg68o7Y1Hs8l\n3aNIi5bl2G3Rj/BFvOSbrdEcRWkSPblpcDwRQX4Ozmgbp8pPMeuLWRTVFBFoCOS1a15jbMJYp1y7\nkaGTZebFXKf1/bSBy3pE0922Ua3Ymtnu86Pn3k/c//wPALV79pB9771Yzp1zooUaAJz6Ds6elusR\ns9W1xQsoqKjl6yNSsXSqE7OK5zMgegBLJiwhKiCKKlMVc7+ay67CXRd/YfuD9LH1cFbrxXPEVJvQ\nTc6ZWjYeK1HZGo0WsfdbhyVC74nq2uJGGrKyyJw2DVNODsLPj8RXXyHMpizeGh/szaOy3oxRL5g0\nMtkNljqJoEgYYMuY7lqizY11gE4nGoVuPtyXz7k6k8oWuR/NWfRSdmed5WhRJQBTRjtnkzp+9jiz\n1s+ipLaEIEMQb1z7BqPjXSDQEXCewtruZdpG5QCdrmmMxof78zlT3X6xmsjp0+j67LMA1O0/QPas\n2VjKO2c5hcuw938kjoT4wera4gWs2ZGDVYGIICM3Do532X16d+nN0olLiQ2MpcZcw4PfPMi2gm0X\nd9F+N0NwDKA0iYFotEjfrmGMTO0CaEI3Hkt9FexfI9cjZoHeQ8c/OJn6U6fImjoNc34BIiCApPlv\nEHrllQ7PUxSlUdjmhkHxxIT6u9hSJ2MPaBYflgreGq1y5/BE/Aw6ak0WPtzb+YRuNGfRS7FnFft2\nDSU9uctFX+9I2RFmr5/NmbozhBpDWTB+AcPjhl/0dVvEXjN/5mRT2YtGi9w5PJEgPz0NZivv7+lY\n30+XuycR/9e/gBDUZWSQNXMW5jNaGbBTqMiDo3Joc2cUhWgvJouVtTvkHnbX8EQCjHqX3i8tPI2l\nE5fSNbgrteZaHvrmITbnbe74BQ3+TQOu96wAsybc4gh7dvHbo8XknKlR2RqNn3HwHag/Bzpj03fb\nx6k7eoysadMxl5SgCwoieeECgi+9tE3n7jh9pjFgP92Tx2W0RNIoqdgNTRlljRaJDPbjxkEyqLl6\ne3anmxvrdGdRCKEXQjwvhCgRQlQKIdYJIaJbOPYGIcS3QohSIcRZIcQPQohxzY5RhBA1Qoiq8/6c\n29ziZZytbuCTg1KtbPLo5A7PJbNzqPQQ9355L+X15YT5hbFwwkKGxAxxhqkt022Y/AMtMt8GwgKM\n3DKkGwBv7ej4RhVx5510+8dzoNNR/+OPZM+Ygbm01Jmmdk72rADFAgERMOA2ta3xeL4+XERxpXSw\nJruoBLU5yWHJLJ2wlISQBOot9Tz87cN8n/N9xy84fJZtwHUZZPzXeYb6KBMHdiU6xA9FkXuYhgeh\nKLDD1n874DYI8QKhlouk7vBhsmfMwFJWhi4khKTFiwgaObLN56+wZcj7x4c5JWDvdoSAkbbs4uH/\nSiVvjVaxV/H9WFjJnuyzKlvjXlyRWfwdcCswGki0vbeyhWO7AK8CPYEY4C3gcyFEc7WW8YqihJz3\n16nlm9btaZpLdtuwi5tLtq94H/d9eR+VDZV08e/CkglLGBA1wEmWOmC4Lbt45BOo0vpYHDHZtlGd\nKqlm26mOZwTDb7mFhBeeB72e+uMnyJo+A1NRsbPM7HxYTLBnuVwPmwrGQHXt8QLs4zLG9YpuFA5w\nB4mhiSydsJSk0CRMVhOPbniUb7I7qGgakQS9r5drTejGIf4GPb8cIX/a39mZ06a5sRpuInsbFNvU\ngjuBsE3toQyybK0YuvBwkpctI2jYsDafX1JZz/pDcpTV9EtSLjpgrxqDfmmbG9sA+1arbY3HMzyl\naW7s6m2dK+DlCmfxfuAfiqKcsjl1TwEThRCpzQ9UFGW1oigfKIpSriiKWVGUN4BaYIQzDRJCRAkh\negshepvNZmde2u0oitIYlb1lyMXNJdtVuIu5X82lylRFVEAUSyYsoU9kH2eZ6piBd4JfKFhNcuSA\nRqsMSghnQDepsHmxkfmwG24g4eWXwGik4dQpsqZPw1RQ4AwzOx/HvmiaS2YPgGi0yMmSKjafkFFs\nZ47LaCvxIfEsnbCU1LBUzFYzT254kvWZ6zt2sZG2kuPcHdrc2DYgK2GgrLqBLw61bW6shhuwBzvi\nBkGSC3QKPIjaAwfInjULa0UF+ogIUpYvI3Bg+wLk7+3OxWxVCPE3cMvQbi6y1A0EhMFg2+zsXUu0\nubEOEEIwxTY39pODBZztgH6Et+JUZ9FWHpoM7La/pyjKSeAc4FDxQQgxGIgCDjX76F1bqep2IcQd\nHTDtYeAocLS42LszKNtOneFUiZyVZ//SdoSdhTt58JsHqTHXEBsYy9KJS+nZxc0zlfxDmgZc716m\nbVQOEEI0Zhe/OHTxA67DrruOxFf+jTAaMWVlkzVtOg25na9x+6Kx93ukXdHp5pJ1BHtEtmuYenPJ\n4oLjWDpxKT3Ce2BWzDy18Sk+PfVp+y/U/SqI7C7XWnbRIYldgrimr/xvbhcH0VCZ6jI48pFcj5wt\nyxN9lNp9+8iefS/Wykr0kZEkL19OQN++7bqG1aqwxhasvW1YN4L8vFwIyC50c/Y0nN6gqinewG3D\nEgg0Sv2IdR3Uj/BGnJ1ZtA8Wa14mWn7eZxdECBELvAf8U1GU4+d9dC2Qhixp/RewWgjRXk3nV4E+\nQJ/YWO+uxV9tK98alBDO4MSIDl1je8F2Hvz6QWrNtcQFyYemtPA0Z5rZduxCN2cz5egBjVa5dagc\ncG2yKE7ZqEKvuorE119H+Ptjys0le7rmMLaLspNN39uRmrCNI2obLLy3OweAe0apO5csOjCaxRMW\n06tLL6yKlad/eJqPTn7UvovodE2CRgfXQZ02ksYRU2xCN7uyznKkQPv3Up39b8kyRGMwDLpLbWtc\nRs2ePWTfOwdrVRX66GhSViwnoE/vdl9ny8kysm0CTZNHeaGwTXPih0CCrZhPE7pxSFiAkVtt2eTO\nJHTj7F/qSttrcwGaCGR28YIIIboB3wFfAk+f/5miKN8oilJn+3sbWAVMaY9RiqKUKYpyTFGUYwaD\n90aBSqvqWZ8hS3cmd3Bcxtb8rfzqm19RZ6kjPjiepROXkhym4nygroOaNipN6MYhsuxF9qmu2ZHj\nlI0qZNxYkua/gQgIwJSfT9b0aTTk5Fz0dTsF9u9sSFfoc4O6tngBHx/I51ydGb1OcPeo5q3p7icq\nMIol45fQL7IfCgrPbHqGD0982L6LDJ0Mej854PrQe64x1Ie4olcMSZGyr1cbo6EyiiKrekCOs/IP\nVdUcV1GzcyfZc+7DWl2NISaGlBXL8e/ZsSqQt3bI7+yQpAj6d2s1B+I92LOLRz+Hc/nq2uIF2Nsn\nTpdWs/Vk5xAGcqqzqChKOZANpNvfE0J0R2YVD1zoHFsv4w/A54qiPKQ4fvq1Ar5bJ9EK7+7KxWSx\n1ckPaX+d/Ja8LTz87cPUW+rpFtyNJROWkBSq/gNbY3bxx8+gUutjcYRdkcuZG1XwJZc0Oozm/AKy\nps+gIbtzNXC3G1Md7LWJAqRPB33H+4c7C6ttzsH4/nHEhQWobI0kIiCCheMX0j+qPwoKf9j8Bz44\n/kHbLxAUCf1vlWv7g7dGi+h0ovFh68N9+dQ0eLeOgFeTtRnKTsj18JmqmuIqqrfvIPv+uSg1NRhi\nY0lesRz/7t07dK2Synq+zCgCYMooFYPszmbgHXL+tWKBPS3pUWrYGZQYzuBEmRNbvb1zPCe5ogZo\nAfBbIUSaECIM+AewXlGUzOYHCiH6ApuANYqiPHmBzwcKIUYJIfyEEEYhxG3ANOAdF9jt0TSvkw/2\nb1+GdFPepkZHMSEkgSUTl5AYmuj4RHcw4A7wt21Ue7WNyhEDE8IZlGDbqJwoQR88ZgxJb76JCAzE\nXGBzGLO0yH+LHP4v1J6R4xOGz1DbGo/nYG4F+3Nlh4J95p6nEO4fzoLrFjAgagAKCn/a8qf2OYz2\nB+2C/ZC/1yU2+hK/GJ6IUS+oqjfzyX5NWEs1dtkqI+KHQEJ668d6IdVbt5Izdy5KbS2Grl1JWbkC\n/7SOt9y8uzunUdjmpiHxTrRUZYyBMNRWsLd7GVi0AI4j7EH79RmFFFfWqWyN63GFs/gc8DGwE8gD\n9MBUACHEFCFE1XnH/hZIAB5tNkfRXmYaAywFzgLFwDPAbEVR2tlY4v1sOlHa4Tr5jbkbeeTbR2iw\nNpAYktg4a8xj8AuCIXfL9e4VYNUk1R1hL0P+MqOQ0osUujmf4NGjSHpzPiIoCHNhoXQYMzOddn2f\nYtcS+dr7egj3kMCLB2NX8E2LDubSHlEqW/Nzwv3DWTB+AQOjBqKg8Mctf2TdsXVtOznlMoiylbVp\n2UWHRIf4c13/OECbuaga5wvb+GBWsXrLFnLmPYBSV4ehWzwpK1fgl9LxIJXVqrB2h2zP8Alhm+bY\nlbwr8+F4B9WhOxE3D+lGqL8Bs1Xh3V2+L3TjdGdRURSLoihPKooSrShKqKIodyiKUmr7bLWiKCHn\nHTtLURTRbIZiiKIoq22ff6coygBFUYIVRemiKMoIRVHWOttmb+AtW6p7WHL76uS/z/meR797FJPV\nRFJoEksnLiU+xAMjYvZS1IpsOPmturZ4AbcM6UaIvwGTReG93c7dqIJHjSJ5wZvSYSwqImvadOpP\nnXbqPbyeogzI2S7X9n4PjRaprjfz0T4pnHT3yCSPnUsW5hfGgvELGBwtxbuf3fos7x571/GJQjQ9\ncB98D+orWz1cQwocAezLKedwviZ043b2r2kSthn4C7WtcSrVW7aQ88CDKPX1GBMSSFmxEr+ki2u5\n2Xyy4wF7ryCmN6SOk2tN6MYhQX4G7kiXSZe3tmdjsfq20I16UnQababoXB1fHbHVybdjLtn3Od/z\n6AbpKKaEpbBkwhK6Bnd1lZkXR2w/SBoj1/aMjUaLBPsbGhW51uzIxurkjSpoxAiSFy1EFxSEuaSE\n7BkzqD91yqn38Gp2L5evEcnQ42p1bfECPt6fT3WDBaNecOdwz87ChvqFMv+6+QyOkQ7j/279X945\n2obOhyE2oZuGKjjUxoxkJ+ayHtGNQjdrd2rZRbfyE2GbO+W8PR+havPmnziKycuX45d48ZVU9jYg\nnxK2aY498HnyGzijBYgdMdn2PJ5XXsvGYyUqW+NaNGfRC3hnZw4Wq0JYgIGbBrctK2h3FM1WM6lh\nqZ7tKNqxZxePfQEV2vgGR9hLUbPKatjiAkWuoPR0khYtQhccjLmkhKzpmsMIgKkWDtgKHNKny/EJ\nGq2yZqcs3xrfvyvRIf4qW+OYUL9Q3rz2TYbEDAHg/7b9H2//+HbrJwVHQb+b5VorRXWITie4e6Tc\nwz7Ym0dtg9Z+4DaytkCZbUKZvfzQB6javJncB391XkbROY5icWWdbwrbNKfvTRAcI9d7VqhrixfQ\np2soI1O7AE1j7XwV7SnHw7GcJ2xz5/BEAox6h+c0dxQXT1hMbJAXzJfsfysERIBi1YRu2sCAbuEM\nSZKzNu1y3s4mKH0YyYulw2gpLSVLyzDC4Q+hrgKEHoZOVdsaj+dw/jn255QDTaWH3kCIXwhvXvcm\nw2KHAfCX7X9xnGG0l6Lm74X8fa410Ae4a3giep2gss7Mpwc1oRu3YR/503UwdBumri1OomrTzx1F\nY4JztBne252L2aoQ6mvCNs0x+MlRQAD7VoPFpK49XoC92u/bH4vJL69V2RrXoTmLHs73x4rJr5BK\nS1PaMFvRax1FsCly2TaqPSs0Ra42MNk2q+7LjCKXKXIFDh3a5DCW2B3GTlyiYi9B7T0Rwnz4wcFJ\n2EsMkyIDPVLYpjWCjcG8ce0bjQ7j/237v9YdxtRxEGmT5d+z3A0WejexYQFc20/+Pq3RhG7cQ80Z\nGfACGdzw0P7h9lC1aTO5v7I5iomJTnUUfypsk+B7wjbNSbcpe1cVwTFN6MYREwd2JSLIiFXBp4Vu\nNGfRw7EL24xKi6RnbOsDc73aUbRjj8yfy4MTX6lqijdwviKXs4VuzufnDmMnFb0pOQrZW+RaG5fh\nkNoGCx/stQvbJKPTed+DabscxvOFbg68C/VVFz5OoxF7tnl31lmOFWnCQC6nUdgmCAbdpbY1F83P\nHMXly5zmKMJPhW28qTKiw0T1aBK60QJeDgkw6rl9mPy+vbMrx2eFbjRn0YMpqKjl2x+LAZjsYJP6\nPud7HtvwGGarmZSwFO90FAFi+kDKWLm2z4DSaJEgPwO32TaqtTtynC50cz6BQ4eStGhho8OY3Rkz\njPY+jrAE6HmturZ4AZ8eLKCyzoxeJ7jLw4VtWsPuMA6NGQpIh7FFldQhk0FnhIZKyHjfjVZ6J+N6\nxZAQIYVutOyiizlf2Gag9wvbuDKjaMcesB/qy8I2zbEHvE58DeU5qpriDdiDCHnltfxw3DeFbjRn\n0YN5d1cuVgXCA41MHNiyOI3dUTxf9dQrHUU7dqGb419qG1UbsAvdZJ+pYdOJUpfeK2jYMJIWLmwU\nvelUDqO5Hva9JdfDpoHOcf9wZ8f+8H9tv1hiwwJUtubisDuMdtGb/936vxd2GENioN9Ncq0J3ThE\nrxNMGinL6d/fk0edSRO6cRlZW6D0mFyP8G5hm6rNF3AUu3Vz6j2KK+v46rAUtnEUsPcp+t4EgV1s\n+hGr1LbG4+kdF0p6stSPsJcs+xqas+ihWK0Kb9sUBG8fltCisM3G3I0/cRQXj/fSjOL59LtZblQo\nsslao1X6xYcxzLZR2aOgriQo3eYwnj9W43QncBiPfAy1ZwABwzRhG0ccK6pkd9ZZAO72kQetEL8Q\n5l87/ycO43vH3vv5gfbIfN5uKDjgPgO9lF+OSEInoKLWxOeHNKEbl2EPXnQdBN3SVTXlYqjeurVJ\nzMZFjiLIgH2nELZpjjEAhtwj13tXgVUL4DjC/hv39RHX6UeoieYseiibTpSSZ1NWaqlOfmPuRh79\n7tGfOIpxwXHuNNM1GPybNqo9K7WNqg3cc95GVVJZ7/L7BaUPk2M17A7j9E7gMNr7N3peCxEXN+C5\nM2DPKiZEBHJ5rxiVrXEedofRPofxz1v/zLpjzeYqpl4OXdLkWssuOqRreABX95W/XWu2+2ZkXnV8\nRNimetu2n8xRTFm+zCWOotWqNIpzdQphm+bYhW7O5cKJb9S1xQu4aXC8W/Qj1EJzFj0U+yY1LDmC\nPl1/LmyzKW9To6OYHJrsO46infM3qpPfqmuLF6DGRiUdxvMzjDNpyMx0y73dTtlJOL1Rru1ZI40W\nqTNZeH+PFLaZNDIJvRcK27RGc4fx2a3P8v7x8/oTdbomAaQD70BDtQpWeheTR8sAzI7MM5wo1oRu\nnM7+tWCp92phm+rtO8iZ9wBKXR3Gbt1IXu78HkU7m0+WknOm9YC9TxPbF5LGyLUmdOOQID8DtwyV\nQYu3d7pWP0INNGfRAymtqm+sk79n5M83qS15W/j1t7/GZDWRFJrE4gk+5iiCbaMaLddaZN4hP92o\nslEU92xUQenpTQ5jcTFZM2bSkOWDw2ntwjYhcdB7grq2eAFfHCqkotaETsBdI7xX2KY1Qv1CpcMY\nbXMYtzzLB8c/aDpg6BTQGaTQzSFN6MYRV/SOJT5c9rX6at+PavxM2CZcVXM6QvWOHeTMm4dSV4eh\nWzzJK1bgl+gaRxGaKiM6lbBNc+wBr6OfQ2WhurZ4AfagQlZZDdtOlalsjXPRnEUPZN3uXEwWhZAL\n1Mlvyd/CI989QoO1gcSQRJZMWELX4JbFb7wae3bx2BdQWaSuLV6AfaPKLKth26kzbrtvUHo6SQve\nRAQFYS4qkg5jjg897FlMTcI2Q6eA3qiuPV6A/UHr6r6xxIcHqmyN6wj1C2X+dfMZFD0IBYU/bfkT\nH56wlfqFxEKfG+RaC3g5RK8T/HKEzC6u25OrCd04k+ytUHpUrod7n7BNzc6d5Mydh1JbiyE+nhQX\nO4rnB+w7lbBNc/rfBv7hoFg0oZs2MDAhnEEJMhCzZqcPPQOhOYseh6I0CdvcMrTbT+rktxVs45Fv\nH6HeUk9CSIJvO4oAA24D/zCwmmH/W2pb4/EMTAhnYIKMgNrLmN1F0IgRJM1/AxEYiLmwkKwZM2jI\n9ZG6/aOfQ7UcYUP6dHVt8QJOllSx/bQMVtx9gcoIX8PuMA6IGoCCwh82/4GPT34sP2wUutkFhQdV\ns9Fb+OVIKXRztsbE+gwtk+E0dtvKCOMGQYJ3CdvU7NpFtt1R7NqVlBXL8Ut0bbXC+3sN5sxmAAAg\nAElEQVRkwD7YT8+NgzuRsE1z/IJgsK1kec8KsFrVtccLuHuUDHitP1TImeoGla1xHpqz6GFsP32G\nU6Wyv+XukU0iGjsKdvDwNw//xFGMD/HxTcwvuKm3Ys8KWUqj0Sr2h/PPDxVSXuPejSp41CiS5s9H\nBARgzi8ge/oMGnLz3GqDS7BnhbpfCZFpKhriHdiDXV3DAriyj+8I27RGmF8Yb173Jv0i+6Gg8Mzm\nZ/j01KfQ/SqISJEH7db6fhyREBHIlX2kmrc2c9FJ1JbD4f/K9fAZXiVsU7NnD9n3z0WpqcEQFycd\nxSTXiospisLaxoB9AsH+nUzYpjn2gFd5Fpz+XlVTvIFbhnQj0KinwWLl/T0+EjBHcxY9jrW2H8j+\n8WGN6eydhTv51Te/os5SR3xwPIsnLKZbiPPVvzwSe838mVOQ+YO6tngBtwy1bVRmKx/sdb+jFjx6\nlMww+vtjys8ne8YMTPn5brfDaZzNahJYspdFa7RIvdnSKLD0yxGJGPSd5ycm3D+cheMX0jeyL1bF\nyv9s+h8+z1oP6dPkAQffAVOtukZ6AfYg6bZTZzhVUqWyNT7AwXfBXAeGABj0C7WtaTM1e/eSM+c+\n6SjGxkpHMdn1lQo7Tp/hVMnPA/adlvPHrGhCNw4JDTBys619bO3OHLfpR7iazvNL7gWU1zTw2SFZ\nenPPqCSEEOwu2t3oKHYN7sriCYtJCHFdrb7HET9E/oEWmW8DYQHGxrKZtTvU2aiCx4wh6Y3XpcOY\nl0fWjJmYCrx0dtreVYACQdFyULFGq3x1uIgz1Q0IIUsKOxvh/uEsvG4hfbr0wapYefqHp1kfkwxC\nB3UVcPgjtU30eK7uG0tcmD/QlKXW6CCK0vS72f822/xiz6d2/35y5tyHtaYGQ0yMdBRTUtxyb/t3\nrl98GIMTvU8IyCXYg/ZHPoHqUnVt8QLsMxdPFFc1zhr2djRn0YP4YG8eDWYrAUYdtw5LYG/xXh74\n+gFqzbXEBcWxZPwSkkI73wNYY0bnyEdyVpRGq9xjq5k/WlTJ3pxyVWwIvvRSEl97DeHnhyknRzqM\nhV7Wg2Qxw96Vcj30HjD4qWuPF2AvHby8VwyJXYJUtkYdIgIiWDh+Ib269MKiWPjtzr/zVY9L5Ida\nZN4hBr2Ou4Y3Cd00mLU+qQ6TvxeKbL2yXtJvXXvwINn3zsFaXY0+Oprk5cvxS011y70rakx8elAG\nNu0Bew2kgq4xGKznib1ptMiwpAj6xMmRd2t8RNlZcxY9BEVRGuXCbxzUjVPnMpj31TxqzbXEBsay\nZMISksI6oaMIsm/RGASWBjjwttrWeDzpyV3oGRsCNJU1q0HI2MtIfO0/CKMRU3Y22TNmYioqVs2e\ndnPiK6i0ZUS1ElSHZJVVs/mElAu3Byw6K10CurBo/CJ6RvTEolh4yprHN0GBkLUZSk+obZ7HY1dF\nLa1q4JsjmhJ2h7EHJ6J6Qcql6trSBmoPZUhHsaoKfVQUKcuX4d/dfX3i/92XR73Zir9Bx61DOlEF\nlyP8Q2HQnXKt6Uc4RAjRKHTz6cF8KmpNKlt08WjOooewN6eco0VyEPGovpXM+3oeNeYaYgJjWDxh\nMclhvq8q2CIBYTDgDrnevVzbqBwghGjstfh4fwGVdeptVCHjxpH4n1cRRiMNWVlkz5yJqdhLHEZ7\n+VbKWIjupa4tXsA7u2SwKzrEn2v6+djc1w4QGRDJovGL6BHeA7Ni5cnYaL4LCtSyi20gOSqIsT2j\nARrFRjTaSX0VHHxPrtOne7ywTd3hw2Tfey/Wc+fQR0aSsmwp/j16uO3+iqI0VkbcOCie8CBtRNJP\nsAvdlB2HrC2qmuIN3D4sAT+DjjqTlY/2eb/Qn+Yseghv27KKKfFl/PvQU1SbqokKiGLRhEWkhqeq\na5wnYK+ZLzkCuTvVtcULuCM9ET+9jlqThY/3q9svGHLFFST8+99gNNJw+jTZs2ZjLvXwvodz+XB8\nvVwP17KKjjBbrLy7Swrb/GJ4IsZOJGzTGlGBcg9PC0/DLASPx0bz/eG1YPYdSXVXMckW8Np4vITc\nszUqW+OFZHwADVWgM8KQe9S2plXqfvyR7FmzsVZUoI+IIHnpUvx7uTdAdyC3gh8LZcD+7s48W7El\nuqXL0SugBbzaQESQHzcMlKPt1qikH+FMtF90D6Cq3szHB/LRBeRRHfk6laZKIgMiWTxhMd3Du6tt\nnmeQOBJi+sm1JnTjkMhgPybYNip3z1y8EKFXX0Xiyy+BwUDDyZNkz5qFuaxMbbNaZu9qUKwQEAH9\nblHbGo/n2x+LKa6sB5oe8jUk0YHRLB6/mNTgBMxC8FiEPz/s+LfaZnk84wfE0SXIiKLQGIjQaAf2\nB/q+N0CI546wqTt6jOyZs7BUVKAPDyd52VIC+vR2ux3238nuMcGMTPUOISC3IkRT4PTwh1DrG8It\nrsQedDhccI6DeRUqW3NxON1ZFELohRDPCyFKhBCVQoh1QojoVo6fKITIEELUCiEOCSHGN/u8pxDi\nayFEtRAiVwjxhLNtVpuP9uVTJ3IISl5EvbWaLv6y36VHhPtKMDweIZoa9DPeh7pz6trjBdhLUQ/k\nVpCRr/5GFXrNNST860XQ66k/fkJmGM964A+O1Qp7V8j14ElgDFDXHi/AriA4pnskadHBKlvjecQE\nxbDo+uUkY8AkBI8eX8mWPK2UqzX8DXruTJfD19/dlYPF6t2RebdSdLipAseDhW3qjx8ne9YsLOXl\n6MLDSV66hIC+fd1uR3W9mY/2yRFPd4/UhG1aZNBdYAiUo1gOvKO2NR7P6LRIutt+D71d6MYVmcXf\nAbcCo4FE23srL3SgEKI78D7wdyDc9vqBECLV9rke+Bg4AsQAtwC/FUJMcoHdqrFi9xYCkxch9LVE\n+Dcp6Wk0Y8jdoPcDU42cHaXRKpd0jyI5UipSeooEfdj48SS8+IJ0GI8d80yH8fQGKLdlY7USVIcU\nVNTy3VHZh3qPVr7VInHBcSwe9AiJJhMNKDzy7cNszd+qtlkejV0kIr+ijo3H/p+98w6Pqkr/+OfM\nZNILqSQhjd57V1GKIIrYFgtIb4oF1rrqT3d1Veyirq4ovdvXLqCoNJUeeodUSC+kZ8r5/XFnkoiQ\nAjNzZ5L7eZ55cpi5954vKXfmPed9v2+2ymrciN3Wxa6gOGg1VF0tF6Hi5EmSJ0/BnJeHLiCAuEWL\n8O7USRUt3+47Q0mlGYNecFuvmLpPaKr4NIPOtyhjzT+iToQQVZk2Px7KwGh2X2dnRwSLM4FXpJSn\npJSFwOPASFsAeB6TgF1SypVSykop5Spgt/V5gKuBeOBJKWWplHI38AFwb0MECSFChRDthBDtTCbT\npf2vHMTaY3tI9XoLnUcpfh6BLBixgPYh7dWW5Zr4hlSnBNreDDUuik5XfaP63550yirNKitSCBw5\nkuhXXwGdjoojR0iZNg1zgTotPi6ILc25RR9o3lldLW7AZzvTsEgI8jFwXedIteW4NJHdxrG4wEgL\no4kKSyWzf57N9rPb1ZblsrSJCKBPvJIS6Arp9G6BsRz2faSMe44HnetVG1WcOk3y5MmYc3PR+fsT\nt3gRPl3Uu9faTJSGd2pOmL+XajrcAttOddZBSN+trhY34G+9Y3j+5s5seGSwW9fy21W5ECIIiAN2\n2Z6TUp4EzgHdLnBK95rHWtltfd72+jEpZfFFXq8vDwJHgaNZLubE+O6uJeg8ShAWHxaN+JAOIc5P\nwXArbDs9ZxPh7F51tbgBt/eOQa8TFJWb+H6/ukY3NQkaNYroV15WAsZDh0mZNh3zORdILS7JgSPf\nKWMXTt9yFSwWycdWF9Rbe7bA26BXWZGL4+FFVNe7WJSRSZRZUm4u54GfH2BHhmbadTFsdT8bDmeR\nVVSusho34Mi3Sj2Z0CnBootRmZREyqRJmLNz0Pn5EbdwAT5du6qm52hGEXtSlMXKu/pqmRF1EjcQ\nwqw1pbuXqirFHQjz92LCwASCfNzbXdfeYW6g9ev5BVIFNV6rSUAdx9b1en35D9AeaB8REdHAUx3L\n4htfolPAtUxt8xKdw7VdjDpJGAQhVtMfzeimTiICvRnaQfmdd5VUVBtBo0cT/dJcEILyg0p/LXNR\nkbqi9q5RGg97+iuNiDVqZcuJHNLyy4DqlEGNOug1kRYmM4vOnCHSsxllpjLu33A/uzLPXzfVALih\nayQBXh6YLJLPd7m/Bb3D2bVU+dpmOAS5Vq/AypQUkidNxpSdjc7Xl9gFC/Dp0UNVTbZ2GS2a+VS1\na9GohZr+Efs/hwqV37M1nIK9g0Xbb03Qec83Q9ldvNDxtR1b1+v1QkqZK6U8JqU85uHh0ZBTHU5E\ngB8f3zaPvw8aprYU9+BPN6pPobJEXT1ugK1B+vakPE5kFddxtHMJuvlmol54QQkY9+8ndfoMzMUq\naZSyegGiy23g5a+ODjfCtgDRI7YZHSIbuobXRAlvD7EDiDWZWUxzInwjKDOVcd9P95GYlai2OpfD\n19ODm3tGA/DxjhS3t6B3KLknIWmzMnaxzIjKtDQlUMzMRPj6ErvgQ3x79VRVU7nRzP/2KAsQd/aN\nRafTjG3qRfexSksWYwkc+EJtNRpOwK7BopSyAEgBetmes5rYBAL7LnDK3prHWulpfd72ejshhN9F\nXtdoivS4G3QeUHEODn6pthqX55p2EUQFKY6eH7tg3U+zv91G1PP/BqBs715SZ8zEXKzCIkDKH0rD\nYYBek50/v5uRW1zB+kMZQPWChEY9sabTxx7/hcVXvUq4TzilplLu/ele9mZrb2/nY0sPTMot5Y9T\neSqrcWH2WL0E/ZtDu+vU1VKDyrR0UiZOwnT2LMLHh9j57+Pbu7faslh3MIPCMiM6Abf30Yxt6o1f\nGHQYpYy1nou1YraYeXfPu+SXu5iRXwNxRLXlhyiOpS2FEIHAK8A6KWXSBY5dDvQRQowVQhiEEGOB\n3oDtt28TkAzMFUL4CCF6APegmNxoNFX8I6D99cpYu1HViV4nuL2P8mH+893pVJhcw+imJs3GjCHy\nuecAKNuzh9R77sFS4uSA0fa71LwLtDh/DUvjfD7fnYbRLPHz1HNjt2i15bgXnW4Br0CwmIg/sZFF\n1y0izCeMEmMJ9/54L/uz96ut0KXo0iKILi2UnWvN6OYimI1Kf1iAHuNA7xo1UsazZ0mZPBnjmTMI\nb29i338fv3791JYFVKegDm4fQVSQj8pq3Aybf0T6Lsg4oK4WF8VsMfPP3/7JB/s+YNr6aRRXulZm\nV0NwRLD4Mkq7ix1AOqAHxgMIIe4WQlR9t6zmN7cBT6Oklj4N3GoLLKWUZmA00AXIBb4HXpNSfuQA\n3RruhG3nJ3UbZB1WVYo7cEefGISAvJJKfjyUqbacCxJ85x00/+czAJTt2kXqvbOwlJY6Z/Kygupd\n6l4TlXRnjYsipaxyELypRzR+Xq6V3u/yePoqPcsAdi+nZWACi0YsItQ7lGJjMff8eA8Hcw+qq9HF\nsO0u/nAgg4LSSpXVuCDH1kGJ1cCv5wR1tVgxZmSQPHESxrQ0hJcXse//F78B/dWWBcDpnJKqXWpb\nT2KNBtByMDSzGgJp7vR/wSIt/Ou3f/H1ya8B6NO8D34G9+1BbPdgUUppllI+KqUMk1IGSClvk1Lm\nWF9bJaX0P+/4tVLKzlJKH+vX9ee9fkJKOUxK6SuljJZSvm5vzRpuSOshEGS9wWs3qjqJCfbl6rbh\nAHzkws1hQ8aNo/nTTwNQumMHqbPuw1JW5viJ938KpjLw8IZudzh+PjdnR1I+p7KVnV/NQfASsa3M\n55+GpM20ataKhSMWEuIdQpGxiJnrZ3I4V1sIs3FTj2h8DHoqTZaqOjONGtgyIxIGQWhrdbUAxsxM\nkidNwpiaivD0JObdd/EbOFBtWVXYdqjDA7yqTOA0GoBOBz2tdbH7PgKjE96n3QSLtPDc78/x1cmv\nALiz/Z082e9JhBsvQrtv0w+Npo1OX716uneN0ltKo1ZsdWVbTuSQnOu6xkAh4++m+ZNPAFC6bRtp\n99+PpdyBP18pqz9odboZfIIdN1cj4SNr+laHyAC6xZzvQaZRL6K6Kw+oWvBqE9yGBSMWEOwVzLnK\nc8z4cQZH846qKNJ1CPQ2MKpbFKAseGlGNzUoTIMTPynj3pNVlQJgzMwiZdJkjMkpCIOBmPfexX/Q\nVWrLqqLSZOHzXWmAknXj4cb971Sl591Ki5byQjj0tdpqXAKLtPD8H8/zxXHF+GdMuzE81f8ptw4U\nQQsWNdyZnuOVG1VZvtJbSqNWhnWsbjjsam00zidk0iQiHn8cgJLffift/gewVFQ4ZrIzeyDDWiPm\nYg6CrkhhmZHvrD07x/aLc/s3QVXpZd1dPPQ1lCopce2C27FgxAKCvIIorChk+vrpWsBoxbbgdTSz\niMTUApXVuBB7VoK0KAtdHW5UVYoxK4uUyZOpTEoCg4EW/3kH/0GDVNV0PhsOZ5JTrKQy39lHy4y4\nZAKjoa3VSEnL8EJKydxtc/ns2GcA/K3t33hmwDPohPuHWu7/P9BougS1UHpJQXVvKY2LYtDrqhzf\nPt2VhtFsUVlR7YROnULEo48AULJ1K2kPPOiYgNG2qxjSGuKvtP/1GxlfJaZTYbLg5aHjlh6u1cfN\n7eh6Oxh8wVwB+z6uerp9SHsWDF9AoGcgBRUFzFg/g+P5x1UU6hr0igumTYRSyeLK6fROxWKG3VYX\n1O7jwOCtmhRTTg4pk6dQefo0GAzEvP02AYMHq6bnYqy2ZkYMahtGXKivymrcHNsCa/IWyDmhrhYV\nsQWKHx9V7uO3tLmFfw78Z6MIFEELFjXcHVvdT9JmpceURq3YCvmziyr4+UiWymrqJnT6dMIffhiA\nks2bSZs9G0ulHc0tKophv7IKqBnb1I2UkjXWD+k3dI0iyNc1HBfdFu9ApacnKAteNVIrO4Z25MMR\nHxLgGUB+RT7T10/nRH7T/TAGIISouod9s+8MxRUmlRW5ACc2wDklpbLq/VAFTLm5JE+eTOWpU+Dh\nQcxb8wgYOkQ1PRcjNa+ULSdyAK3e2i60HQEBSnp4U3Wnl1Lyyo5X+Oio4r15U+ubeHbgs40mUAQt\nWNRwd9peB/6RylhLg6iT+FA/rmgdClTXnbk6YTNnEP73OQCUbNxE+oN2DBgP/g8qi5W+nT3G2eea\njZj96YUcPnsOUJpYa9gBm7Nz9hHF3bkGnUM7s2D4AgIMAeSV5zFt/TROFjTtRbHbesXgqddRWmnm\n68QzastRH1tWTdxACG+vigRTXp6yo3jiJHh40GLemwQMG6aKlrr4eEcqUkKonyfDOzVXW477o/dQ\nel+D4h9halpOxVJKXt/5OqsOK21rRrcazb+v+Dd6nV5lZfZFCxY13Bu9h1JkDZC4Wuk1pVErd/VT\nVlN/PZZNeoF7OJiF3XsvYbMfBKB440bS5/wdaY+A0bYS2v4GpX+nRq3YdhVbhfnRv2WIymoaCTF9\nIKKzMt7115X5zmGd+XDEh/gb/JWAcd00ThWecrJI1yHEz5ORXZQFwjVusuDlMM6dhWNrlXEvdXYV\nTfn5pEyZSsXx46DX0+KNNwgcPlwVLXVhMlv4dJdyDxvTOwZPD+0jsF3oZTUbLMmGYz+oq8WJSCl5\nY+cbLD+kbFTc0PIGnr/y+UYXKIIWLGo0BmyuqCVZcLTp3Kgules6NyfY14CU8ImLG93UJPy++wh7\n4AEAin/5hbSHHr68gDHzEKTtUMYqfdByJ4orTHydqLQsuLNvrGZsYy+EqHawPPiFYth1Hl3CuvDB\n8A/wN/iTW57LtHXTOF142rk6XYix1gWv/emF7E8rVFmNiiSuBGkG7yDofIvTp68KFI8eVQLF118j\n8LoRTtdRX345mk3mOaXuXcuMsCPBCdBqsDK+wIJXY0RKyZu73mTZIeX/OzJhJC9e9WKjDBRBCxY1\nGgMhLatvVE00Z74heHnoua2X1ehmZypmi/tY0Ic/cD9h980CoHjDBtIfeQRpvMTdZFvaclCs0rdT\no1a+2XuGkkozBr3gb71j1JbTuOh2u9Lj01QO+z698CHh3Zg/fD5+Bj9yynKYtm4aSYVJztXpIgxo\nFULLMKXB9ZodTXR30WKpvod1uwsMPk6dvipQPHIEdDqiX32FwOuvd6qGhmLbie7fMoRW4f51HK3R\nIGwLrid/hoLG/TcppWTernksPbgUgOsSruOlQS/hofNQV5gD0YJFjcaB7UZ1YkOjv1HZA5sF/ZnC\ncjYdy1ZZTcMIe/BBQu+9B4CiH38i/eFLCBiNZbB3tTLuOUHp26lRK7YPWtd1jqxqwaJhJ3yCofOt\nyvg8o5uadA/vzvxr5+Pr4Ut2WTbT1k0j+Vyy83S6CEKIqnvYV3vSKWmKRjenfql+r3OysY0pP5+U\nqdNqBIqvEjRqlFM1NJQzBWX8elQxdRvXXzO2sTsdRoFvKCCr3XkbIVJK3tr9FksOLgFgRPwIXh70\ncqMOFEELFjUaCzVvVHtWqq3G5WkTEUDfBKX5vLvV/QghCJ8zh9CZMwEo+vFH0h99rGEB46GvlEbC\nQlddb6FxUQ6kF7LPmu43rp/2Qcsh2FJRsw5C+q6LHtYjogfzhysBY1ZZFlPXTSXlnHv9DduDv/WK\nwaAXlFSa+WZvEzS6sRnbxPSF5p2dNq25oICUadOoOHxYCRRfeYWgG107UAT4ZGcqFgnNfA1c1zlS\nbTmNDw8v6D5WGe9ZAebGt4AjpeTt3W+z+MBiAIbHD+flqxt/oAhasKjRWPjTjWql0ntKo1ZstuEb\njmSRda5cZTUNQwhB+EN/J3TGdACK1q1rWMC4U1kVpN1IpbGwRq3Y+pIlhPoyoFWoymoaKbH9Iczq\nZrlrSa2H9ozoyfvXvo+Phw9ZpVlMWTelyQWMof5eVR/63W3B67IpzoKj3ytjJ9ZbmwsKlB3FQ7ZA\n8WWCRt/otPkvFbNFVtXn39qzBd4GLZPEIdgWvIrOwvF1qkqxN1JK3tnzDosOLAKUQPGVq1/BoGsa\n7aO0YFGj8WB70zyXDid+UleLG3BD1ygCvD0wWySf7kpTW06DEUIQ/vDDhE6fBjQgYMw6DKl/KOPe\nUxys0v0pqTDx1R7F2GZsvzh0Os3YxiHUNLo58IWy810LvZr3avIBo22Xe29aIQfSm5DRTeIqsJjA\nM6C6T6eDMRcWkjJ1GuWHDoEQRL/8EkGjRztl7stl0/FszhQqC6JjtcwIxxHWFhIGKWPbzncjQErJ\nf/b8h4X7FwJwbdy1TSpQBC1Y1GhMhLeDuCuUcRNx5LocfDz13NqzBQAf7UjB4kZGNzaEEIQ/8kjD\nAkbbm1hQLLRxzV5groRmbONEut8Fei8wlsL+Cxvd1KR3895NOmAc0CqU+FBfQLmHNQkslur3t253\ngKefw6e8YKB4000On9derNmm/G70jg+mXfMAldU0cmwLXsd/bBT+EVJK3k18lwX7FwAwLG4Yr17z\napMKFEELFjUaG7ZC/2NroShDXS1ugC0VNTWvjN9O5qqs5tJoUMBoLFMaBwP0mqgZ29QDWwrqCM3Y\nxvH4hkCnm5VxLUY3NblQwNhUTG90OlF1D/tyzxlKKxtfndRfSNoM+da2KU4wtjEXFpIybTrlBw+C\nEES9NJegm292+Lz2IutcORuOKMY22q6iE+g4uobRzXK11VwWth3FD/d9CMCQ2CG8dvVrTS5QBC1Y\n1GhsdLpZ6TklzY3akctedIoOpHtsM8C9LejrHTAe/NJqbKOHnuNVUOpeaMY2KmALADL2w5k99Tvl\nvIBx6rqpTSZgHNM7Bg+doLjCxLd7z6otx/HYMiOie0JUd4dOVbWjeOCAEijOnUuzW5zfz/Fy+HRX\nGmaLJMDbg1Fdo9SW0/jx8IIe45Tx7hVgvsTWVipjCxRtO4pDYofwxjVvYNA3vUARtGBRo7Fh8Kk2\nutm9TDO6qQdjrc2J1x/MILe4QmU1l069AsZdmrFNQ1hTw9hmoGZs4xzir4TQNsq4AXU/TTVgDA/w\nYkTn5kD1LnijpSQHjnyrjB1sbGMuKCBlytTqHcW5c2l2q3sFihaLrEpPvqVHC3w8tUwSp2DzAijO\nULK83Ayb62nN1NOmHCiCFixqNEZsN6rCVM3oph6M7h6Nn6ceo1nymRsa3dSk1oAx8xCkblMO7KMZ\n29RFSYWJrxKVlgR3acY2zqOm0c3+z6CiqN6nNtWA0ZZemJhawKEz51RW40D2rgFzJRj8oOsYh01j\nys8necrUqhrFqJfcL1AE+O1kLql5ZYCWgupUQltDy6uVsZsZ3dj6KNpcT4fFDeO1a15r0oEiaMGi\nRmMkooOyOg+wc7G6WtwAPy8PbrEa3aze7p5GNzW5YMD48CPIbdbfhaBYaD1URYXuwTd7z1BcYcKg\nF4zRjG2cS/exoDOAsQQOfN6gU/8SMK6dSlJhkmN0ughXtg4jNsQHaMRGN1JWG9t0HQNejjFqMeXn\nK+0xDle3x3C31FMbtsyI7jFBdIoOVFlNE8O2aH9iA+S7x4KVlJJ5u+f9qY/ia9c0zRrF89GCRY3G\nSZ+pytfj66EgVV0tbsDd/eMBSM4tZcuJHJXVXD5/CRh//JG0/3yFNKOkb2nGNnWyRjO2UQ+/MMUo\nAi5pZf5PAWOZssN4qvCUfTW6EDWNbv63O52yykZYfpD8G+QeV8YOMrYx5eeTMmXqnwJFd3I9rUl2\nUQXrDykmd3dpu4rOp8ON4BuGYnTj+u70Ukre3PUmSw4opSpNrY9iXWjBokbjxObIJS1u78jlDDpF\nB9IzTjG6WbXNPVYB66IqYLznHgCKUz1I+y0US+c7VFbm+hxIL2SvZmyjLrZU1DN74Exiw09v3psP\nhn+Ar4cv2WXZTF07lZMFJ+2r0YW4vY9idFNUYeLbfWfUlmN/bIsGkV0hupfdL2/KyyNl0mQqjhyx\nBoqvuE0fxQvxyc5UjGaJv5cHN3XX6tOdjocn9LxbGe9Z6dJGN1JK3tj5BksPLmZ6q7YAACAASURB\nVAVgRPwILVA8D7sGi0IIXyHEYiFEvhCiQAixSAjhU8vxE4UQv1mPzxFC/CCE6Frj9QQhhBRClAgh\niq0P9y6q0nAOHl7Qw3qj2r3cpW9UrsJ46+7iT4ezyLA2MHZ3hBCE/30OYQOVFKTidC/S/m8ulgr3\nNfJxBrZUvnjN2EY9EgZBcEtlfIkr8z0jevLB8A/wM/iRW57L1HVTOZ5/3I4iXYeIAG+u7agY3axp\nbEY3JTlw6Etl3GuSUtdqR0x5eaRMnkLFsWNKoPjaqwSNvtGuczgTs0Wy2tpb8bZeLfDz8lBZURPF\nZsJUnAlHv1dXy0WQUvLaztdYdki5x46IH8HLV7+sBYrnYe+dxbeBDtZHO6Aj8GYtxwcA/wJigBbA\nbmD9BQLM9lJKf+tDK57RqB+2lfniDDj6g6pS3IFR3aII8jFgruEg1xgQWYcJjz9CWBfF+KJk4ybS\nHnhQCxgvQkmFiS/3WI1t+mrGNqqh01WnG+77pEFGNzXpEdGDD4d/iL/Bn7zyPKatm8ax/GN2FOo6\njO2v7ILvTingSEYjMrrZs1IxtvH0h2532vXSppwcUiZNUgJFvZ4Wr79G0KhRdp3D2Ww8lkV6gWJs\nYyux0FCB0NbQarAydkGjGyklL21/iRWHlDZr1yVcp+0oXgS7BYvWAG888IyUMlNKmQU8A0wSQnhf\n6Bwp5XtSyh+llCVSygpgLhCJEmxqaFweoa2h1RBlrBnd1Im3QV9lZPLR9lRMZovKiuyE9U0q/MoQ\nwmc/CEDJ5s2kzboPS3nj2EG1J9/uqza2ub2PtjanKj3Gg94TKouVgPES6RbejQUjFhDgGUB+RT7T\n1k3jaN5ROwp1DQa1CaNFM2Wtec22RrLgZbFUt/zpdgd428+oxZiZRfLESVQcP1EVKAbecIPdrq8W\nK/9QfvZ9E4JpH+kYIyCNemIzujn5M+SdVldLDSzSwvN/PM+aI2sAuKHlDbw86GU8dNou9IWw585i\ne8Ab2FXjud2AD8ouY30YBpQCJ857fpsQIlsI8asQYnBDhQkhQoUQ7YQQ7UwmU0NP13BnbEY3p36B\n3MZbr2MvxllX5jPOlfPzkSyV1diBylLY+5Ey7j2RsPvuI/yhhwAo+e03UmfNwlJWpqJA12P1dsUQ\nakQnzdhGdfzDodPNynjHIsUR8xLpEtaFBSMWEOgZSEFFAdPWT+NQ7iE7CXUNdDrB2H5K39gvdqdT\nUtEI3u9P/gz5Scq4zzS7XdaYkUHKxIlUnjoFHh60mPcmgddfb7frq0VqXim/HFXeu8YP0HYVVaf9\nDeAXroxdxOjGIi089/tzfHrsUwBuan0Tc6+aqwWKtVCvYFEIsdRaO3ixxwsoKaUAhTVOtY3rXAoT\nQrQDFgKPSClt+TY5wECgJZAAfA78IIToVh/dNXgQOAoczcpqBB+ANepP++vBP1IZu2AahKvROtyf\nK1orNWqrGsPK/KEvoaIQhF7ZpQHC7plJxKOPAFD6+x+k3nMvlpISNVW6DAfPFLI3tQDQ+pK5DH2n\nK1+zDlb3Cb1EOod2ZuGIhQR5BVFYUcj09dM5mHPQDiJdhzv6xmLQK0Y3tj6hbs1Opd8bsf0hsotd\nLmlMTyd5wkQqk5MRBgMx77xD4IgRdrm22ny0IwUpIdTPk5FdItWWo+HhCT2V9172rARTpapyzBYz\nz2x9hi+OfwHAbW1v4/krn0evOaTXSn13Fh8Awmt5zAVsAV5QjfNs41qLB4QQnYBfgNellPNtz0sp\ni6WUf0gpK62pqv8BtgC311O3jf+g7Hy2j4iIaOCpGm6N3gC9JijjPSvBpNWp1YWtxmPT8WxScktV\nVnOZ7LSmb7W/HgKjqp4OnT6diMcfB6B0+3ZSZszEXFyshkKXwpa+FR/qW7VooKEysf0horMy3rHw\nsi/XMbQji0YsItgrmKLKImasn8He7L2XfV1XISLAm5FdlL/15b8nIS9jN1Z1ClLh2FplbKddxcq0\nNJInTMSYmorw9CTmvXcJGDrELtdWm0qThY93KJkRt/eJxctDCwBcApvRTUk2HP1ONRkmi4mntjzF\n1ye/BuCOdnfwr4H/Qie0xhB1Ua/vkDVoy6nlUYqyc1cO1PR07gmUARetphdC9AJ+BV6WUr5aDzkW\noEGOC1LKXCnlMSnlMQ8PbZu5ydFrEggdlOXBoa/VVuPyDO/UnDB/L6SE1e7sKph5ENK2K2Nb3UQN\nQqdOofn//R8AZbt3kzJ1GubCwr8c11QoLDPy5Z50QHHG1YxtXAQhoK81UDj4JRRnX/Yl24e0Z9F1\niwjxDqHIWMTM9TPZlbmr7hPdhAnW9MMjGUXsTslXWc1lsGup0v7JJ6Q6HfkyqExOJnn8BIxnziC8\nvIh5/7/4X3315et0EdYdzCCnuBIhtJY/LkVIS2g9VBmrlOFltBh5YvMTfH9acWUd22EsTw94WgsU\n64ndvktSyjJgJfBvIUSEECIC+DewXEp5QRcJIcSVwAbgaeuu4fmvDxBCdBFCeAghvIUQM4FrgP/Z\nS7dGE6BZLLS1ptjYjAI0Loqnh447+yrGJp/uTKXC5KYNrrcvUL42i69+ozqPkAnjiXzuOQDK9+0j\necoUTPlu/OHyMvh8VxplRjNeHjrN2MbV6HYHeAaAxQh7Vtjlkm2D27LkuiWE+4RTaipl1k+z+OPs\nH3a5ttr0TQimfXOlMmbF727aN9ZUWd0juNcEMFzQJ7DeVJw6TfKEiZgyMhA+PsR+8AH+V15pB6Gu\nw8o/lJ/11W3DiQv1VVmNxp+wudOf+tXp/hFGs5HHNz7OuqR1AEzoNIEn+z2JsHMLmsaMvUPqOSi7\niLbHUeAh24tCiKeEEDULJF5ASVV9s0YfxWIhxCDr6y2BL1FqH9OBCcBoKWXjWQLVcA42o5vkrZB1\nRF0tbsBdfeMQAnJLKll3MFNtOQ2nrAD2fayM+05T2hBchOA77yBq7lwQgopDh0mZNBlTbq6ThLoG\nFous+qB1U/domvl6qqxI4094BUB3a8uEnUvAYp8FnFbNWrF05FIi/SIpM5Vx/0/3szlts12urSZC\nCMYPVHYXv9+fQU6xG5YfHPkWSrIAccHMiIZQcfw4yRMnYsrKQufrS9yCD/Eb0N8+Ol2EE1lFbDud\nB2jGNi5J+xvAX+mD6kx3+gpzBQ/9+hA/pfwEwJQuU3isz2NaoNhA7BosSilLpZRTpZTNrI9p1h1H\n2+tzpZSda/x7iJRSV6OHou2x2fr6GillGymln5QyVEo5SEr5oz01azQR2lwLQYpLnra7WDexIb4M\naa/U9676ww1X5veuAWMpeHhDzwl1Ht7stluJfvVV0OupOHaM5ImTMDYhM6zfTuZyKkcx+Zk4MEFd\nMRoXxlazVpgCJ36y22XjAuNYOnIpLfxbUGmpZM4vc/g55We7XV8tbu3ZAj9PPZVmC5/sTFVbTsOx\nfaBuM0xJ47tEyg8dInniJMw5Oej8/YlduBDfPn3sJNJ1sNVbRwd5M7SD5k3hcugN1Ysee1YoTuUO\nptRYygMbHmBj2kYAZnSdwUO9HtICxUtAS9bVaBro9NUNrhPXOOVG5e7cbW2jse10HsczL60huCpY\nLNVGIF3GgG9IvU4LGn0jLd54Azw8qDx5kpQJEzFmZDhQqOuw/PckAHrENqNrTFCtx2qoRPNOEHeF\nMraD0U1NWvi3YOnIpcQHxmO0GHnk10eqUrbcFX8vD27rpaRTr/ojBbPFjYxuso9CknWH9zKMbcr2\n7iV58hTM+fnogoKIW7IY31497STSdSitNPH57jRAcXHWa/XWrknvyaDzgPJC2P+pQ6cqMZb8KbX+\nwZ4PMrvXbC1QvES0YFGj6dBzgnKjqiiEg1+orcblGdw+guggpU7GrdponP4Vcq2tWvtNb9CpgSOv\nI+adtxEGQ5UZRGVamv01uhDpBWX8dFhJNZ6gpW+5Njajm+M/VvfesxORfpEsuW4JrYJaYZImHt/0\nON+c/MauczgbWzpiekEZvx51o0yBHdZ2GYEx0O66S7pEyfbtpEyZiuXcOfQhIcQvW4pP1652FOk6\nfLv3LEXlJjx0gjv7xqotR+NiBEZBx9HKePuCy+obWxuFFYXMXD+T3Vm7AXisz2PM7DbTIXM1FbRg\nUaPpEBCp5M2DU3Pm3RW9TlT12vt8dxpllW5idLPduuvSog9EN3wVPWDoUGL++x7CywtjWhrJd4+n\n4tRpO4t0HVZvS8YiIcTPk1Hdouo+QUM9Ot5kbXAtq9vC2JFw33AWX7eYdsHtsEgL/7fl//j82Od2\nn8dZtI8MoF9LJbNghbuk01eWKGn0AH0mK1kxDaR4y1ZSZ96DpbQUj4gI4lcsx7tDB/vqdCFWblN+\ntiM6Nyci8PKMgDQcTD9r0Ja5H1Lsb6iVV57H9PXT2ZezD4BnBjzDxM4T7T5PU0MLFjWaFjajm/Rd\nkL5bXS1uwJ19Y/HQCYrKTXyzzw0aXBekwLEflHG/GZd8Gf9Bg4j9YD7C1xdTZibJEyZQfvSonUS6\nDhUmc1Vfsjv6xOJt0PqSuTQentDL+sFnzwqH9I0N9Qll0YhFdArthETy7O/PsuKQfRxY1cC2W77x\nWDbJuSUqq6kH+z+DinNKFkzPhn/ILfr5Z9JmzUKWl+MRHUX8yhV4tW7tAKGuwb60AvalKS2PxvfX\nMiNcnriB0LyLMt6xwK6Xzi7NZuraqRzJO4JO6Hjhyhe4o/0ddp2jqaIFixpNi5bXQGhbZbz9Q3W1\nuAERgd6M6Kw4mK10h5X5nYuVvmS+YdDplsu6lN+AAcQtWoguIABzbi7JEydRtm+fnYS6BmsPVPcl\ns9Woarg4vScrfWNLc+HQVw6Zopl3MxaOWEjPCGVn/tUdrzJ/73y3bHB/XefI6r6xrp5OLyXstKag\ndhwNAc0bdPq5H34gbfYcpNGIIT6OhJUr8Yxr3H/XtvelVuF+DGwdqrIajToRAvpay0MOfQVF9vEF\nyCjJYMq6KZwsPIle6Hll0Cvc3Obye5NqKGjBokbTQqeD/vco4wOfQ7Eb1bGohG21dl9aIXtcucG1\nsbxGX7KJl92XDMC3Z0/ily1FHxyMpbCQlMlTKNm+/bKv6yost/agG9o+gtgQrS+ZW9AsDtpa69js\nbHRTkwDPAOZfO58BUQMAeC/xPebtmud2AaOnh45x/ZQ6to93plJudOF0+vTdcHavMm6gsU3Bl1+S\n/sijYDLh2aY18StWYIiOdoBI16GwzMjXe5WMl7v7x2vmJe5CtzvAKwgsJti17LIvl3IuhUk/TCL5\nXDIGnYE3B7/JyJYj7SBUw4YWLGo0PbrfBV6BYK6EXUvVVuPyDGwdSrvm/gAs2ZqkrpjaOPSlstsi\ndNXpxnbAu1Mn4lcsxyM8HEtpKakzZlK82f170R08U8iuZCX4nzBQS99yK2wr86nbIGO/w6bxNfjy\n7rB3GRI7BIAlB5fw4rYXsUiLw+Z0BGP7Kw6ZBaVGvtt3Vm05F8e2qxjWHhKuqvdpeStXcfaJJ8Fi\nwatjR+KXL8cQ0fjbR3yxO41yowVvg44xVudbDTfA0w96jlfGOxeD2XjJlzqWf4xJaydxpuQMXnov\n3hn6DkPjhtpJqIYNLVjUaHp4BVTfqHYsuqwbVVNACMHkK5Q+X9/vP0vmuXKVFV2E7db6h3bXQzP7\nOuJ5tWlD/KqVGKKjkRUVpN53P+fWrbfrHM7Glr4VH+rL1W3DVVaj0SBaD4XgBGVsc850EF56L94Y\n/AY3tFTMwT4++jFPb3kak8Xk0HntSVSQD9d2VIKn5a6aTl+ap2S7gOJ6W49dMiklOfPnk/nCCwB4\nd+9G/NIleITUr12QO2OxyKrMiNHdognyNaisSKNB2JydizPg8KW5Lu/L3seUtVPIKcvBz+DH/Gvn\nc1WL+i+yaNQfLVjUaJr0nQ4I5UbloLqfxsQtPaMJ8jFgskhWueKHrfTdkL5TGTewXUZ98YyLI37V\nSjwTEsBoJP2hhyj8yj1/dwrLjHy5R0nfGt8/Hp3Wl8y90NXYPd/3CZSfc+h0Bp2BuVfNZUy7MQB8\nc+obHtv4GJXmSofOa08mDEgAYG9qAfvSCtQVcyESV4GpHAy+SvZLHUgpyXrtdbLfehsA34EDiF+8\nGH1Q0+iT+uuxLE7nKIZFk69MUFeMRsMJbQ1trlXGl5BOv/3sdqavn865ynM082rGohGL6BPZx84i\nNWxowaJG0yS0NbQdoYy3faCuFjfA19ODu6z9q1ZtS3G9uh/bm01oW2g52GHTGKKs7oLt24PFwpl/\nPEHeipUOm89RfLYrjTKjGS8PHbf30dK33JIe40HvBcYS2PuRw6fT6/T8c8A/mdhJcej8KeUnZv88\nmzJTmcPntgdXtA6lVZgf4IJmXWYTbLMarnW7A7xrD/ik2UzGP/9J3mKlBZT/sGHEzp+Pzs/P0Upd\nhsVbkgDo3zKEztFNI0BudNjaaCRvhYwD9T7t19RfmfXTLMpMZUT4RLB05FI6h3V2kEgN0ILFP2Gx\nWKioqKC8vFx7XOKjsrLSfQwQbEY3adu1Nhr1YMLAeHQCcksq+daV6n7+lL41Xdl1cSAeYWHEL1uK\nd/duAGS++CLZ777nNr/3Fous+rB8U/domvl6qqxI45LwC4UutynjbfPB4vg6QiEEj/Z5lFndZwGw\n9cxWZq6fSWFFocPnvlx0OsHd1jYaXyWeobDUhcoPjn4HhVan1v6zaj1UVlaS/uijFHz6GQBBN99E\nzNtvofPycrRKl+FYZhFbTuQAMOXKliqr0bhk2lwLzaz18vVso/Hdqe/4+y9/p9JSSYx/DMuuX0br\nZo23NYyroAWLVoqKiigsLMRsdrEdEzejsrKSgoIC9/jg3GqI1kajAcQE+zKiUyQAS7aedp2f8Z4V\n1vQtP+gx1ilT6ps1I37xYnwHKk6ROe++S+bcl5BO+MB+uWw9mVOVvjVxYIK6YjQujwHWwCLvJBx3\nTg2tEIL7etzHY30eAyAxO5Ep66aQXZrtlPkvhzG9YvA26KgwWfh4pwu10fjjfeVr66EQ0eGih1nK\nykh94AGKflgLQPDddxP10ksIDw9nqHQZbEZrMcE+DO/UsPYiGi6ETl9t1rXvEyirPT38k6Of8OTm\nJzFLM62DWrPs+mXEBGiZMc5ACxYBk8mElJLg4GB8fX3x9vbWHpf48Pf3x2AwYDS60KrtxfhLGw3X\n/7CjNlOstSEHz5xjZ7ILtNGwmKsNPuqRvmVPdH5+xH7wAQHDhwOQv2IFZ554Auniv/s2U4gesc3o\nGqOlb7k1Ud0h3mro8Md7Tp16YueJPH/l8+iFnuP5x5n4w0RSi1KdqqGhBPkauLWn8uFy2W/JmMwu\nsLiTvhtSflfGA+676GHm4mJSZ8ykZJPixBw6616aP/1/CAdnUrga+SWVfLE7DYBJAxPQa/XW7k3P\n8eDhDcZSSFx9wUOklCzav4jn/3geiaRzaGeWjFxChG/jd/x1FZrWXeYimEwmDAbNScte6PV6LG6w\nwwJobTQaSL+WIXSMCgRgqSu00Tj+IxRY64/6zXD69DpPT1q8NY+gMX8D4NzX35A2ew6Wctd0jE3N\nK2XD4UwAJgzQ2mU0CgZaA4zTmxpU92MPbmlzC28OfhNPnSdpxWlM/GEix/KPOVVDQ5l2VQIA6QVl\nrD1on4bgl8W2+crX0LbQetgFDzHl5JAycRKlOxUTr4jHHiNizpwm2VdwzY4UKkwWfD313NHXvq7X\nGirgGwJdFeMsdiz4Szq9RVp4fefrvLX7LQD6NO/DwhELCfYOdrbSJo0WLFppijddR+FW30uvAOhx\ntzLesVBro1EHQgimXJEAwNqDGZwpUNncwlbnEH8lNFenwF3o9UQ9/zyh0xUr8OJffiF1+gzMRUWq\n6KmNxVtPY5EQ5u/JqG5RasvRsAftRla30bClMzqRoXFDmT98Pn4GP3LKcpi8djKJWYlO11Ff2kQE\nMLi90ipm4ebT6oo5dxYOfKGMB9x7wXrrytRUksbdTfmhQ6DTEfnv5widZr8+su6E0WxhhTUzYkzv\nGIJ8tEX+RkFf60Jv3ik49XPV00aLkae3PM3yQ8sBGBw7mPevfR9/T381VDZptGBRQ6PfDLQ2GvXn\nph7RhPh5YrZIVqjpKph1BE78pIz7OqZdRn0RQhDx6KOEP/IwAKU7d5I8aRKm3FxVddWksMzIJzuU\nNMGJAxPwNuhVVqRhF3T6alOU/Z9AcZbTJfSN7Mui6xYR7BVMUWURM9bPYEv6FqfrqC/TrlJMURJT\nC9ilZjr9zkVgMYJ3M+j+13rr8iNHSBo3DmNKCsJgoMVb8wi+4w4VhLoGaw9kcLZQydqYZF201GgE\nRPeA2P7K2OoKXGYqY87Pc/jmlNKD8ZY2tzBv8Dy8PbzVUtmk0YLFRkBaWhpCCJKSkgCYO3cuo0eP\nVleUO1GzjYZmdFMn3gY9Y/sp6T9rtqdQVqmSKdTv7ypfg+Kg403qaDiPsBkziPz3cyAEFYcOkzRu\nHJWprlHH9dH2FEoqlXYZ47UU1MZFz7ur0+l3LlZFQufQziy7fhlRflGUm8t5cMOD/HD6B1W01MVV\nbcJo3zwAgEVbTqkjwlhW/bPqPRk8/9z2omT7dpLHT8CcnaPURy9YQOCIEc7X6UIs2arsBA9pH07r\ncG13qVFha6NxfD2FZ/Ywc/1MNqcr9blTukzh31f8Gw9d0zJyciW0YLER8tRTT/HNN9845NolJSW0\nbt0aj8bmvtbfeqNK3aa10agHEwYoxgIFpUa+Skx3voCiTNj3sTIeMAv0rvP7GHzHHbSY9yYYDBiT\nU0gaO05JIVMRo9nC0t+SAPhb7xhC/LR2GY0KrwDopfQ/ZMdCMKpTM9syqCXLr19Oy6CWmKSJxzc9\nzopDK1TRUhtCiKrdxbUHMkjNK3W+iP2fQmkuCP1f6q3P/fgjqdNnYCkuRh8WRvyK5fgN6O98jS7E\n3tQCdqcobplau4xGSKebISCaTL2OyT/PIjFbSWV/pPcjPNz7Yfcqb2qEaMGiRoN44oknaNmyEd6o\nWw3V2mg0gMggb67vYmujkeT8Nho7Fii7KF5B0GuCc+euB4EjRxK34EN0fn6Yc3JInjCRkt9/V03P\n9/vPVqVv2T4kazQy+s0EoYOSbDjwmWoyIv0iWTZyGV3DugLw6o5XeXPnm1ika5me3dQjmjB/TyyS\nqoUUpyFldX1pp5shqNr+P//TT0mf83dkZSWG2FgSVq/Cu1Mn5+pzQWy7im0i/BnUNkxlNRp2R28g\nqddYJkQ354S5BL3Q8cKVLzC5y2S1lWmgBYsXxGi2kJxb4tSHsQEW3hkZGdx0000EBQXRrl071q5d\n+6fXn332Wa699tqqfyckJPDCCy8wZMgQ/P396dq1K/v27WPNmjW0adOGoKAgpk+fjslkqnXeTZs2\nsXnzZv7xj3807BvqDmhtNBqMrY3G0cwifj/lxNq8ylJl9wSgz2RlV8UF8RswgPiVK9CHh2EpKSFl\n5j0Ufved03VIKVmwWUm1G9YhQkvfaqwEx0OHG5XxH+8rAYlaUryDWThiIYNaDAJgycElPLXlKYwu\nZCDmbdBXpWN/vCOVonInaju9EbKs2QbWdhlSSnLmzyfjmX+CxYJXhw4krF6FZ1yc83S5KJnnyvl2\n31kAJl+RoO0yNUL2Z+9nYuaPnPXwwMti4e2wQdzc5ma1ZWlYcZ3cLRfiTEEZ17z2q1Pn3PjYYOJD\n/eo+ELj77rsJDAwkJSWFsrIyxowZU+c5y5Yt4+uvv6ZNmzZMmTKFW2+9lWHDhrF3715yc3Pp06cP\nQ4cOZdy4cRc8v7S0lBkzZrBy5UpKSkoa9H9zG7rfBT89B5VFsGsJXPO42opcml5xwXSLCWJfWiFL\ntyZxRWsnrfYmroKyfNB5QL97nDPnJeLdsSMJa9aQOm06lcnJnHnkUcy5uYRMnOg0DdtO53Eg/RwA\n0we1ctq8Giow8H44/DVkHlBaabS6RjUpvgZf3h76Ns/99hxfnfyK7059R15ZHvOGzMPPUL/3Okcz\nfkA8//31JMUVJj7Zmea8XXfbrmKLPhDbF2k2k/nii+SvXgOAb9++xPz3PfQBrrkQ5mxW/pGMySIJ\n8jFwW68WasvRsDMbUzfy6MZHKTeXEyAMvJuRRq/s7+HaUvD0VVueBnbeWRRC+AohFgsh8oUQBUKI\nRUIIn1qOnyyEsAghims81px3TB8hxHYhRKkQ4qQQYrw9Nbsb6enp/Pzzz7z++usEBQURGRnJv/71\nrzrPmzlzJh07dsRgMDBu3DhOnTrFiy++iJ+fH3FxcQwePJgdO3Zc9Pwnn3yS0aNH07dvX3v+d1wL\nrwClQSzA9gWq1f24C0IIJlsd6X48nOmcuh+LGf74rzLuMgaCXP+Dg2dMDPFrVuPdVUnLy5z7Elmv\nv+601F1be4DO0YEMaBXilDk1VCK2P0T3Usa2vxMVMegMPH/l88zoqtTk/X72d6asnUJOWY7KyhTC\n/L24tYdyD1my9TRmixP+JnNPwjFrNtCAWVjKykibPacqUAwYfi2xCxdogaKVcqOZVdtSALirXyy+\nntoeR2Pi02OfMvuX2ZSby2nu25xlQ96hl9Gs1PPuXVP3BTScgr3/6t4GOlgfEvgSeBOYVcs5p6SU\nbS70ghAiCPgBeB0YBFwN/E8IcVJK6bACoOhmPmx8bLCjLn/ROetDWloaAPHx1W6G9akhjIqq7qnm\n6+uLXq8nPDz8T88VXaQv3JYtW/j+++/Zu3dvvTS6NQPuVWoWS7Jg72ro0zT7WdWXUd2imPv9EXKK\nK1j6WxLP3Ojg2pqj3yu9mACueMCxc9kRj5AQ4pcuIe3vD1GyeTO5Cxdhys4m6oUXEAbH9Qo7lV3M\nhiOZAMwY1EpL32rsCKGkNX4xXQlIck8qbs+qShLM7jWbCN8I5m6by+G8w0z4fgLzh88nPlB9V96p\nV7Xk452ppOWXsf5gBtd3dXD/0W3zla8B0ZgiB5E6eTLle/cBEDxuHM3/ucxXZwAAIABJREFU7ymE\nXmtrY+PrxDPklVSi1wkmDkxQW46GnZBS8m7iu3y4T/GIaBvclveHvU9zv+bQ+TalDdDv7ylOwTrt\n70Ft7LazaN1BHA88I6XMlFJmAc8Ak4QQl9oY5TagDHhVSlkhpfwR+B8w0y6iL4JBryM+1M+pD4O+\nfj+KFi2UVdDk5Or+dqdPO7ax8E8//URaWhpxcXGEhYVx8803YzabCQsLc5jrqmoEJ0CX25Tx1neU\nnSyNi+LloWeCte5nzfYUCkorHTvhb9Z2Ga0GQ2RXx85lZ3R+fsT+9z2CblbqMAq/+prUe+7BfJFF\nGnuweOtppITIQG9GdXPwh2AN16DzLRBg/Vnb0h1dgLs63MUbg9/AU+dJWnEaE76fwP7s/WrLon1k\nQJVhysItjn0vpawA9qwCoDLhTpLGT6wKFCMee5TmzzytBYo1kFKy2GpsM7JzJC3quaiu4doYLUae\n3vp0VaDYP7I/y0YuUwJFgCseVL7mnYSjrtl+p6lhzzTU9oA3sKvGc7sBH6BdLefFCiEyhBCpQoiP\nhBA1t8m6A7vln/O1dlufrzdCiFAhRDshRLu6TFxcnZiYGAYPHszjjz/OuXPnyMzM5Pnnn3fonA8/\n/DDHjx8nMTGRxMREFi5ciF6vJzEx8U9GOo2GK/+ufM0/DYe+UleLGzBxYDy+nnpKK80s/z257hMu\nldQdkPqHMh74oOPmcSDCYCDq5ZcInaGk5ZX89jvJ4+7GeOaM3efKL6nks11KJsLkKxPqvSCl4ebo\nDdWtGGz1vS7C8PjhfDD8AwIMAeRX5DN13VQ2JG9QW1ZVreKu5HwSUwscN9GeFWAsoazAn6TX12FM\nTkEYDES/8Tqh06ZpO//n8dvJXI5kKItpNkM1DfemxFjC/T/dz9cnvwZgVKtRvH/t+wR41ki7juqm\nLAgD/PYfp2vU+Cv1+vQghFgqhJC1PF4AbD/pwhqn2saBF7n0JqArEA30BcqBH4UQtur3gPOuB1BQ\ny/UuxoPAUeBoVlZWA091PVavXk1FRQWxsbEMGjSIiQ42ywgMDCQmJqbqYUtfjYmJwcenEa70RXaB\nNsOV8da3VHUVdAeC/TwZ209x7Fuy9TSllQ5akPnd+qYR0QnaDHPMHE5ACEHEIw8T+ey/QKej4vhx\nku68i7KDB+06z6ptyZQbLfh66hnbV3NUbFL0ngIePmAshd3L1VbzJ/pE9mHZ9cuI9Iuk3FzOQ78+\nxPKDy53ffqcG17QLp22E4hK8yFG7i2YTbPuQonQvkjc0w5xfgC4ggNiFCwkaNcoxc7o5//31BADd\nY5vROz5YZTUal0t2aTaT107m97NKFdm0LtOYe9VcDPoLlGLYdhdT/4DU7U5UqXEhRH1u0EIIf5Rd\nw4tRirJ7uAcIllIWWM8LBvKA7lLKffWYx4ASHI6WUm4QQrwFJEgpb6lxzEPABCllrzqFV58TCoQC\ndO/e/WhiYuKfXi8vV4xMvL0vNVtWoyaN4vuZtAWWWt/AJ/wPWg9VV4+Lc7awjKtf/QWjWfKv0Z3s\n3zQ57zT8pxdIC9z8XrURkZtTvHEjaQ89jCwtRfj6EjPvTfyvuXwHywqTmate+YXsogomX5HAszd1\ntoNaDbfi24dg52IIjIE5icqOowuRVZrFAxse4HDeYQDGdhjLP/r+A71K9Ulrtqfw5Bf70esEmx4f\nYv+Ux32fkP/qHDJ2BYEUeERFEffhB3i1bWvfeRoJiakF3PLeVgA+nNCbEZ0jVVakcTkczz/O/Rvu\n52zJWQSCJ/s/ydgOYy9+gpTw/hVKi5mOo+HOlc4T24QQQuySUvap67h67SxKKYullDm1PEpRdu7K\ngZpBXE+UmsNj9dQtrQ9bLsZe6zVq0tP6fL2RUuZKKY9JKY95eGhOWhr1IP5KiLE6v255S10tbkBU\nkA+39lTqaRdsOkWlyc4NuP94XwkU/ZtD19vte20V8b/mGhJWrsAjIgJZWkrqrPvI/+ijy77u14ln\nyC6qQAiYau/AXcM96G/1lTuXBge+UFfLBYjwjWDpyKVcE6Msjqw5soY5v8yh1OgEV+ULcGvPFoT4\neWK2SJb/lmTXa0ujkcyXXiRjZzOQQumh+NFHWqBYC//9RdlVbNfcn2s7NldZjcblsCV9CxN+mMDZ\nkrN46b2YN2Re7YEiKGZdtt3Fw98qZl0aqmG3IhYpZRmwEvi3ECJCCBEB/BtYLqW8YA8CIcQoIUSM\nUAgB3gNyAGthEv8DfIUQjwkhPIUQw1BMbz60l24NjQsiRHXt4umNkL5bXT1uwD3XtEYIOFNYztd7\n7ViDV5oHe6yriv1mgoeX/a7tAnh36kTCxx/h1a4dWCxkPPscma+9hrRcWsAtpaxKpbuuUyRxoVqf\nqiZJeDvocKMy3vw6XOLvkyPxNfjy9pC3qz44bkzbyOS1k8kuzXa6Fm+DnvH9lXTt1dtTKCo32uW6\nlpIS0qbeRd4exfzLr3dX4leuwNA8wi7Xb4wcyyxi/SHFxXnW4NbodFotp7vy0ZGPuH/D/ZQYSwj1\nDmXJdUsYFlfPMpIuY6xmXdIlWgE1ZezteDAHZRfR9jgKPGR7UQjxlBCiZmHOYGA7UAwcREkVHS6l\nLAawprPeANyOkp66ALjXkW0zNDSqaH8DhFm9mbZqu4t10Trcn5HWVKH5G09isVfPsl1LwFgCBt9G\n28rEEBVF/OpV+F1xBQB5ixaT/tDDWMrKGnytLSdyqkwhZlyt7So2aQY9onzNOQaHv1ZXy0XQ6/Q8\n1f8p/tH3HwgEh/MOM+77cRzLr29Ckv2YMDABTw8dReUmu5h1GTMySBo/nuIdhwBo1jOI2GWr0fv7\nX/a1GzPv/6rsIsUE+zC6W7TKajQuBbPFzCvbX+HFbS9ikRbaNGvD6lGr6RreABdzD0/of48y3rMK\nSnIdI1ajTuwaLEopS6WUU6WUzayPadYdR9vrc6WUnWv8+zEpZbSU0k9KGSWlHCOlPHbeNXdIKftJ\nKX2klK2klFrisoZz0OngyjnK+NDXWhpEPZg1WOnpdiKrmB8PZ17+BU0VsM2aSNBzPPg23qbyen9/\nYj+YT9CYvwFQtG4dyRMmYsxsmCnXh5uUPpQ9YpvRK04zhWjStOgFbayO1Zted2mzrvGdxjNvyDy8\n9d5klGQw6YdJbE7b7FQN4QFejLOadS3cfIqSiks36yo7cJCkO+6k4vARQNK8ZyGRL76K0EphaiUl\nt7QqM+Xea1rjobk4ux2lxlLm/DKHlYeVj+tXtriSFdevINr/EgL/3lPA0x9MZbBzkZ2VatQX7a9Q\nQ6M2ut4BAdGAhK1vq63G5ekW04yr2ig9y/7768nLdzjc/xkUZwACBsy6fIEujjAYiHr+ecIfUhIy\nyg8cIOn22yk7UD+n1F3J+Ww+ngPAvde00qz4NeDqx5Svmfvh2Dp1tdTBsLhhLBm5hFDvUIqNxTzw\n8wNOd0q955pWeOp15JcaWbXt0nYXi376ieQJEzBlZSEMgphBeYRc2xXRcpCd1TY+Pth0ErNFEh7g\nxZjeMWrL0WggGSUZTFo7iY1pGwG4s/2dvDv0Xfw9L3E33acZ9JqkjLd9AMYLVrVpOBgtWNTQqA0P\nTxh4vzLeuwaKMtTV4wbYdhf3phbw+6nLSBsxm2DzG8q442gIaWUHda6PEIKwe2bS4j/vIHx8MGVl\nkTx+POfWrq3z3Lc3HAegQ2QAIzpp7oEaQNwASLAGKZtec+ndRYAuYV1YPWo17YLbYZEWXtv5Gs/+\n/ixGs31qCOsiKsiHMX2UIOXDTacoqzTX+1wpJbmLFpH24GxkWRkeYcEkDM0koEUFXP2oUguvcVGy\nzpXz6U6lN+yMQS3xNqjjjKtxaRzMPcjd393Nkbwj6ISOJ/o9wf/1/z88dJe5mz7gXhB6KM1RPodp\nOB0tWNTQqIvek8A7CMyVWpF1PbiidSjdY4KA6tqTS+LAZ5BnPd+2O9KECBw+nITVq/CIikKWl5P+\n94fIfu+9i+6y7EnJZ9MxxRhk9rC2mimERjVXP6p8Td8Jp35VVUp9iPaPZsX1Kxgaq7Qs+uL4F0xf\nP5288jynzD/rmtZ46AQ5xZWs2Z5Sr3MslZWcffppsl5T0n29O3cmYVwo3sEmiOpenQ6scVEWbjlN\npdlCkI+Bcf3j1Zaj0QC+P/U9k36YRFZZFj4ePrwz5B3u7ni3fbJbmsVB51uV8ZZ54KSFI41qtGBR\nQ6MuvAIUF06AnUugvFBdPS6OEKJqd3Hz8Rz2p13C98tsgo2vKOMON0JUNzsqdB+8O3ak5Scf49O9\nOwA5/3mXM488iqX8r6k4tl3F9s0DqoyGNDQAaHlNdSugTa+pq6We+Bp8mTdkHjO6zgBgd9Zuxn3n\nHOOb2BBfbuultAL6YNNJyo217y4as7JImTiJws+VFiUBw4cT/8pDGLI2KQcMekTbVayDgtJKVv6h\npP1OviIBfy+tttMdMFvMvLXrLf6x+R9UmCuI8oti+fXLuSb28vsF/4mrHwUEFCRD4mr7XlujTrRg\nUUOjPvS7Bzy8oeKc0uhao1ZGdIqkVbgfoDijNpj9n0KeYtTC4CfsqMz98AgPJ275MgJvGg3Aue+/\n/4vxTWJqAb8eVXYVHxzWRttV1PgzQlTvzidvhaSt6uqpJzqhY3av2bw86GU8dZ6kF6cz4fsJ/Jr6\nq8Pnvm9wG3QCMs9V8OnO1IseV7ZvH0ljbqcsMRGA0Fn30uLtt9DttGahhLWHDqMdrtfdWfpbEqWV\nZnw99Uy+IkFtORr1oKiyiNm/zGbRAcV4pldEL9aMWkOHkA72nyyiI3S5TRlveg1MlfafQ+OiaMFi\nIyAtLQ0hBElJSQDMnTuX0aO1Nye74h+uuHEC/P5frci6DnQ6wb3XKLuL3x84y6ns4vqfbDbBpleV\nccfRENkAq+1Gis7Li+hXXqk2vtm/n6QxYyjdsweAd6y7im0j/LmhS5RqOjVcmLYjqv+WNr+urpYG\nMqrVKJaOXEqYTxilplJm/zybRfsXOdT4JiHMj1t6KLuL7//6/+ydd3gUZdeH79mS3gtJSEIChIRe\nI9J7VboUKSJIU+woiAVBRWxIef1URFSw0HvoSG8ivdeQDum9bpvvjwkBhJAAu9lsMvd1zbXPTtk5\nmSSzc55zzu+Eo9Hd36cyff16okYUCtnY2uI7bx5V3nwTIekyXN4k7dR2kqSsLVMsOQU6fjsUCcDw\np6vham9lXoNkSiQqM4rhW4azP1aKng8KHsSibotwt3U33UnbvwcIkBEDp+XGCGWJfAergHzwwQeE\nhYUZ9TM7dOiAtbU1Dg4ORcumTZuMeo5yT6vXpSLrnEQ49Ye5rSn39Gvsi4+zDaJ4p51DqTi38k5U\nsX3ljirezW3hG7//+w7Bzg5dUhJRI1/k/ILf2H1ZijK+LtcqyhTH3dHF8N0Qe8K89jwiDTwbsOzZ\nZdRxq4OIyLyT83h337vkanNNds6JHYMQBLiZkc/ak7FF60WdjoQvvuTW1PcRNRrUvr4ELl+GU4/u\n0g4H50ivLgFSY3GZh7Ls32gy8rRYKRWMbVs5hMwsmUNxhxi6eSgRGRGoBBXTWkzj45Yfo1aqTXti\nzxBoMEga758ttdaSKRNkZ1Gm1EybNo3s7OyipVevXuY2qWxxDYQGhV/8+2eDxnQPKRUBK9WdL/41\nJ2O5lVGKBvN6Hey7HVXsA971TWihZeLYpQuBy5ehDqgGWi3KeV/zxqlVhLhZ8WwDOaoo8xBq95bS\nIsHioosA3vbeLOm5hJ6BPQHYEbWDYZuHEZkRaZLzBVVxKPqf+n7vdbR6A/r0dGLGjyd1yRIA7J5+\nmsDVq7AJKbyuKeFwfo00bvMWKOXau4dRoNMXTSYODPXDy8nGzBbJFIcoiiy5sISJuyaSpcnC1dqV\nhd0WMjhkcNkZ0f49EBSQGQcnfy+781ZyZGfxQei1UmSjLJdHUHeKj4+nT58+ODs7ExwczLb/SOrP\nmDGDLl3uKK8FBgYyc+ZMOnbsiIODAw0aNODs2bMsW7aMoKAgnJ2dGTt2LDrd4zcgrjR0mAoKldT7\n79jP5ram3DO0uT+udmq0epHvdl8v+YCzKyAtQhq3f8+0xlkwNsHBVF+1CvHpVgD0jDrKF/t/wJCY\nYGbLZMo1CsUdZdQrWyD+nHnteQxsVbZ81e4rJodORikoCc8IZ+jmoSarY3y9Uy0AYlLz2LbxEBGD\nBpNz+AgAri+8QLVFP6Nydb1zwKF5IBrA0QcaDzeJTRWJNSfiSMwqQCHAhHZyVLG8kqvN5b397zH7\n+GwMooFg12CW9VrGU95Pla0hHkHQcIg0PvCtXBJURshTXg8iIwb+16Rsz/nGqVL3kRs+fDhOTk5E\nR0eTl5fHwIElp7ksWbKEjRs3EhQUxOjRo+nfvz+dO3fmzJkzpKSkEBoaSqdOnRg2bFixnzFv3jzm\nzJmDj48PI0aM4N1330WtNnHaQXnDrQY0HSmJ3BycC81GSW01ZB6InZWKiR2C+HzLJVYci2Fc2xpU\n97B/8M563R2lRjmqWCJKJyfmdpqAV7o9w6/sxPb6ZSKeG4jfvLnYPVXGX+AylkO9AbBnljQpc+Bb\nGLTY3BY9MoIgMLLeSGq71Wby/smk5qfy+u7XebnRy7zS6BUUgvHmwUO8JXVhzdbNVAtbjVavRVCr\n8Z4xA5fnBty7c0YsnC7sA9fqdVBZG82OiohGZ+DHfdIkYu9GVQlwL+a7Qcas3Mi4waQ9kwjPkMTq\nugZ0ZWbrmdip7cxjULvJcHYlZN2CE4ulPowyJkWOLFoYcXFx7N69m9mzZ+Ps7Iy3tzfTp08v8bjx\n48dTp04d1Go1w4YN48aNG3z++efY29tTrVo1OnTowLFjx4o9/osvvuDatWskJSXxyy+/sGjRIj7+\n+GNj/miWQ7spoLKFvDQ4/J25rSn3vNAyAB9nG/QGkdk7rhS/49nld6KKlVwBtTRcuJnBzktJ/Fmn\nO7HvfILC3h59SgpRo18i9c+/TCr+IWPBKFWS6ArAhfWQZPpWFKaiuU9zVvRaQX13aWJpwZkFvLbr\nNTIKjNfeyFBQwOtn1jD55DKs9Fq0bh4E/PH7/Y4iSN8HBi3YuUsTiTIPZdm/0cSk5iEIkvqsTPlj\nR+QOhm4aSnhGOEpBybuh7/Jt+2/N5ygCuNeERkOl8cE5oC1FiYvMEyFHFh+Es78U6Svrc5aC2Fip\nyD4g4E7D2urVq5d4nI/PnVomOzs7lEolnp6e96zLysoq9viWLVsWjVu0aMGnn37K1KlT+eKLL0pl\nd4XCyQeeHg+H5kvKqM0nSGqpMg/ERq3k7S7BTFlzls1nb/Fyuwwa+P0nGqvX3okq1u0LXvXK3lAL\n47YCanUPezq+1BN951BiX3sdzY0bJMycSd7p0/h8MgOFvTxbL/MfGj4Pe7+CzFgpujjgJ3Nb9Nh4\n23uzuOdiZh2dxdprazkQd4Chm4cyt8NcQtxCnuizNbFxxL31Forz5wE46VmL1T3Gs6Zho/t3Tou8\n01apxUSwkv/vHkZWvrboHjagiR8h3o5mtkjmbrQGLfNOzOP3i1JdoIetB9+0+4ZQ71AzW1ZIu3el\nCebsBOn/ruWr5raoQiNHFh+EUi2lG5blUkoVKV9fSco7KiqqaF1ERIRJLsPDUCgUlTty0fotsHYC\nbY70sCXzUAY09aVmYd/Fr7dfvn+HM8ulhy2QFVBLwaVbmWy/INUnvtYxCJVSgXWNGgSuXIFjV6le\nOXPTJiIGDiL/iuVGjmRMhMpKEl8BqU44/rx57XlCrJXWzGg5g49bfoxKoSImK4bhW4az5uqax/6e\nyt6/n8jnniO/0FHUDx/FtFbjOJUlsO1C/P0H7J4Jeg3YV4Gn5bS4kvj5QAQpORqsVAomdQs2tzky\nd5GUm8TY7WOLHMWmVZqystfK8uMoArhVh8aFZVMH54Imx7z2VHBkZ9HC8PPzo0OHDkyZMoXMzEwS\nEhL47LPPTHrO9PR0Nm3aRHZ2NqIocurUKWbMmMGQIUNMet5yjZ0btHpDGh//BdKjzWtPOUelVDC5\nuzTLf+BaMoevJ9/ZeE9UsR941TWDhZbF7Rn5QHc7+jauWrRe6eCA7//+R5UpU0ClQhMRQeSQIaSv\nefyHZpkKStORksIzIuy0/JICQRAYFDyIxT0W423vTYG+gBlHZvDBwQ8eqb2GqNeT9L//ETPhZfQZ\nGSicnfFb8CP1p71HmxAvAObsvIpOf1ffxbiTcG6VNO74Plg7GPNHq3AkZuWz6ICkgDqqVSC+LrZm\ntkjmNsfjjzN402BOJp4E4IW6L7Co+yI87cph9lTbd0GhhpwkOPaLua2p0MjOogWydOlSCgoK8Pf3\np23btowcOdKk59NqtcycORNfX1+cnJwYMmQIw4YNq5wpqHfT4hWw85Bmk/d9ZW5ryj3d63nTyN8F\ngK+2Xb7jvJxZBulRgCAroJaCS7cy2Xpeimy8WhhVvBtBEHB/aTQBf/yOytsbMT+fWx9+xK2p72PI\nldu9yBSisoYuM6Rx+C64/rc5rTEajTwbsarXKtr5tQNg041NDNk0hKtpJUfYdcnJxIwbT/IPP4Io\nYlOvHtXXrMGxQwcA3ukqRcCuJ2az7FiMdJB4l7PtEQxNTPt9XBGY//c1cjV6nGxUTOxQ09zmyAB6\ng54FZxYwZscYkvOSsVPZMbv9bKY8NQW1opwKGboGQJMR0vjQfCjINq89FRihss02h4aGisePH79n\nXX6+JL1rYyP39zEGlep6/vMjbJsq9f2ZeBQ85XSah3E4PJlhPx8F4MfhTelZ1wO+aypFZuv1t0hl\nxrJEFEVG/vovB64lU83Njt3vtL/PWbwbXVoaN997j5z9BwCwCqqJ37x5WAfJYhIySI7OL10h9hhU\nqQcvHwCF0txWGQWDaGDJhSXMPzkfvajHWmnN+83fZ0CtAQiCcN/+2QcPcXPqVPTJUtaDy+DBeH34\nAQrrexVN31p+ivWnb+Jmb8XeyR1wit4NSwv7zD2/DGo/Y/KfzZK5kZRN17n70RtEpvaszcvtZWfR\n3CTmJjL1wFSOxUsihzWcazC3w1xquFhAK5P0GOkZQq+RJr/avG1uiywKQRBOiKJYYn6xHFmUkXkS\nQl+SxIlEA+z53NzWlHta1fSgXbCUzvLNjivo/11UmMIrSCqzMg9l9+VEDlyTHman9qz9UEcRQOXq\niv+CBXhOmgRKJZrr4UQMGkz6+vVlYa5MeUcQoFvhfSvxApxeal57jIhCUDC6/ugS01JFjYbE2bOJ\nGTsWfXIygp0dVb/6Ep9PP7nPUQSY3KM21ioFqTkafth9+U5UMaA1hPQsqx/PYvlm+xX0BhEfZxtG\ntQo0tzmVnv2x+xm4cWCRo/hcredY3mu5ZTiKAC7+Uko9FEYXixdqlHl8ZGdRRuZJUFnfafNwcT3c\nPG1eeyyAKYW1i+lJt9DvKnxQbTRUrlUsAY3OwMzNlwBoXt2NnvW9S3WcoFDgMX4cAUsWo6pSBTEv\nj1tT3yfu3cnoMzNNabKMJVDtaUmBGCSRlgomFNG4SuMHpqVeSrmEJjqayBEvkLJIqneyqVuXGmvX\n4Ny3b7Gf5+tiy7i20oN01uHfIalQsKvrZ5LzLVMsJ6PTilLoJ3UNxkZdMaLYlohWr+WbY9/w6q5X\nSStIw0HtwDftvmFGqxnYqiyshrTNJFBaS+3MDswxtzUVEtlZlJF5Uho+L9WqAOz61Ly2WAD1fZ3p\n1dCHd1SrsNJlIVo5QJeSe4VWdn4/EklEcg6CAB/3qvvAVLqHYRcaSvV1a7Fv0wYoVEvt15/c/6Tl\ny1RCOk+XhCKy4+HI9+a2xui42LjwXafveKfZOygFJZGZkfzf10O40rc3+WfPAuA2ahQBy5dhFRhY\n4ue90qEm/g4ibyhXSivqDQC/Zib8CSwfURT5covkWId4OTKgqZ+ZLaq8RGdGM2LriCK10wYeDVjZ\neyU9qvcws2WPibOvpCEBcOT/ICXcvPZUQGRnUUbmSVGqoOOH0jh8F0QeNK89FsAHTbQMVe4G4F//\nl8CxdFGyykpKdgHzCxVQh4T6U9/XuYQjHozK3R3/hT/h9f5UBLUa7c2bRI18kcR58xC1WmOaLGNJ\nuNeE5uOk8cF5kJVgXntMgEJQMKr+KBa3W8C72214bYMWVZ6GHAcV9vNm4TX1PRRWVqX6LHtrFT/U\nOISXkI5GVHKuzpsmtt7y2X05kX8jUwF4r2cISoUchS1rRFEkLDyMQWGDuJhyEYDR9UazpMcS/B1L\n1+u73NJuMjj6SLWL2z8wtzUVDtlZlJExBnX7gk9jabzrU0k4QubBiCJVj8xAIYhEGLx4NbwFGXmy\no/Iw5uy8Sla+DgdrFe90e7JG44JCgduLLxK4ehXWtWqBwUDKgp+IHDYcTWSkcQyWsTzaTQYbZ6l3\n7N5Z5rbGJOSePIXDhOk0PympJp4NFHh7tMjzqd+wNWJr6T8oO5H6EUsA+EPfjY/25WAwyPf84tAb\nRL7aJkUVm1d3o2NIFTNbVPlIzU9l0t5JUs2uLhc3GzcWdFnApNBJqEvZ57tcY+0AXQszu65ug6s7\nzGtPBcOozqIgCHaCIPwqCEKaIAjpgiD8IghCscnPgiAsEAQh+z+LKAjCpLv2EQVByP3PPo83rS4j\nYyoEATpPk8YxR+HcavPaU565sBaiDwPwlTiS5HxYuF9OGymOS7cyWfav1MfztU5BeDreL7rxONiE\nhBC4aiWuL7wAQP65c9wY8Bzpq1fLPRkrI3ZuksMIcPJ3SLxkXnuMiEGjIfHbb4kaMQJtVDQolXhO\nmoTrD3MwuDmTpc1iyv4pTD0wlSxNKQQy9n6BoM1Bp3bkO10/zsSkE3b2pul/EAtlzclYriZIDvr7\nPWs/cgq9zJOxJ3oP/Tf05+9oqT1Oa9/WrO69mta+rc1smZFpMAiqtZTG294DXYF57alAGDuyOB+o\nXbgEA3WAYqtNRVF8WRRFh9sL0B/QAcv/s2u3u/cTRTHDyHbLyDzbqDX8AAAgAElEQVQ5NTtDrW7S\nePv7UrG1zL1ocmFHoXpgzc7UaPUcAL8cjCA6Re4B+F9EUeSzTRcxiFDNzY7RrQON+vkKGxu8P/wA\n/58XovTwQMzN5dZH04h9/XV0SUlGPZeMBdB8PLhUk9Sdb6t8Wjj5ly4R+dxAUn5eBAYDVkE1CVyx\nAo/x4+hWowdr+6ylhU8LADbf2MxzG5/jePxD6niTrsIJKaqoav8uzWpLrR++3naFfK3e5D+PpZGv\n1TN3p9Tj8pkG3jSp5mpmiyoP2ZpsPj70MW/seYPU/FRsVbZMazGNHzv/iKedp7nNMz6CAD2/llqZ\npd6okPXX5sJozmJhBHEEME0UxQRRFBOBacCLgiCUtuHeBCBMFEV5ik7G8hAEeGY2qGwhJwn+/sTc\nFpU/Ds2DzFhQqKDHl0zoEISHgzX5WgMfrj8nR7T+w86LCRwOTwHgg2fqYK0yjXqgQ9u21Ni4AYeO\nHQHI/nsXN3r1JiNsk/w7qUyorKVeZQDXdkD4HnNa80SIOh3JCxYQMWgwBdeugSDg9tJLVF+zBtv6\n9Yr287L34qeuPzE5dDJqhZpbObd4aftLfHH0i3tabBTx9wwQ9eDkB09P4P1n6qBUCMSl5/HLwYiy\n+wEthF8ORnArIx+lQuDdJ0yhlyk9x+KPMTBsIOuurwOgsWdjVvdezeCQwRU7suvTEJqNlsb7Z0Om\n7E4YA2NGFkMAG+DEXetOArZIUcaHIgiCN9AHWPCAzasEQUgWBOGoIAgDHtUwQRDcBUEIFgQhWKfT\nPerh5Z7Y2FgEQSCysN5o1qxZ9O7d27xGVVZcA6DDe9L4xG8Q86957SlPpEdLfZAAmk8Az2CcbdXM\n6CO1zDhwLZkNp+Ub+20KdHo+3yKlAras4U73el4mPZ/KzQ2/H77H+7NPUTg4oM/I4ObkycS++hra\nxESTnlumHFFvAPgW9mjeMQ0MlhctK7gRQeSw4STNmw86HWp/fwL++B2vKZMf2DtRISgYWW8ky3st\nJ9g1GBGRpZeXMmDjAI7eOnpnx8hDcGWzNO48DdS2BFVxYMTT1QD4Yc91krLk1LfbRCbn8L9CYa6h\nzf2p4elgZosqPvm6fGYfm82Y7WOIy45DpVDxVtO3WNxjMdWcqpnbvLKh00dg6yrVX1eQDAlzUypn\nURCExYW1g8UtMwHHwt3vThG9PXYqxWnGADHAzv+s7wJUB/yQUlr/EgThUfV9XweuAFcSK8FDzwcf\nfEBYWJjRP3f58uU0bNgQe3t7vL29+fxzuQn9A2n5GnjWkcZhb4FeFm8BpAdPXT7YeUD7KUWrn23g\nQ6fakuDBp5sukpajMZeF5YrFhyKJSslFIcDHvR+9VcbjIAgCroMGUSNsI/Zt2wKQvXs3N3r1Jn39\nejnKWBkQBOheeG9POAenl5rXnkdA1OlI+eVXIvr3L2qJ4fL8EGqsX4ddaGiJxwe7BrP82eVMbDQR\nlaAiLjuOsTvG8umRT8nOTYZNb0k7ejeABoOLjnuzSzCONipyNHrmFKZcVnZEUWTq2rMU6Ax4Oloz\nuVttc5tU4fn31r88t/E5llxcgohY9Pc8psEYlIpK1NPSzk1yGAHOrYKow+a1pwJQ2sjia4DnQ5ZZ\nwO2q8LvFZ26PH9r5WRAEBTAOWCj+52lEFMVdoijmFy4rgD+B4aW0+zbfIUU+Q6pUkVW4Hoc//viD\nt99+mzlz5pCRkcG1a9fo06ePuc0qnyjV0HueNE68AP/8YF57ygMRB+Diemnc+WOwdSnaJAgCn/at\nh52VktQcDbO2VBxhjcclKauA73ZfB+D55tWo41Oa+TbjofbxwX/hT/jMmoXC0RFDZia3pr5P7Muv\noE2oeG0VZP5DtRZQp/D+vuNDyLxlXntKQd75C0QMHkziN98gFhSgqlIF/59/xmfGDBT29qX+HLVS\nzSuNX2F5r+XUcZMm/VZdXUW/tc9wMCdaqofqNQ8Udx6f3OyteKNTLQBWHIvm4s2HPvJUCpYfi+Gf\nG1KrjM/61sPZrgIobpZTMgoymH54OmN2jCE6KxqloGRM/TEse3YZIW6VNPW32WjwaiCNt0yxyAyJ\n8kSpnEVRFLNFUUx+yJKLFLnLB5redWgTIA8oaaqtB+AD/FoKcwzAI02xi6KYIoriVVEUr6pUqhL3\n1xq0xGTGlOmiNZQ++hQfH0+fPn1wdnYmODiYbdu23bN9xowZdOnSpeh9YGAgM2fOpGPHjjg4ONCg\nQQPOnj3LsmXLCAoKwtnZmbFjx1Jciq7BYGDq1KlMnz6dLl26oFKpcHR0pEGDBqW2udJRrQU0HSmN\n934ppWBWVvQ62DZVGvs0giYj7tvFz9WOSV2lbPVVJ2I5fD25LC0sd8zZeYXsAh2O1qqi61LWCIKA\ny4D+1NgUhkP79gBk79vHjV69SVu+AtFgMItdMmVEz6+lVhr5GVJErZxGlQ25uSR8+RWRgwdTcFGa\naHIZNJAaYRtxaNvmsT83xC2Epc8u5c2mb6IWVCTo83jFuwof1WlJhmet+/Yf2SqAam52GESYvPoM\nWn3l/f+Iz8hn1mbpd9Gzvjc96vuY2aKKiSiK7IjcQd/1fVl7bS0AddzqsPTZpbzV7C2slKXrG1oh\nUSjhma+lccI5qSxI5rEp2XMqJaIo5gmC8CfwqSAI5wtXfwr8LopifgmHTwDWiqJ4j/yeIAj1ATvg\nNCACzwIvAM8by+4HEZ8dzzPrnjHlKe5jS/8t+DuVrinq8OHDcXJyIjo6mry8PAYOHFjiMUuWLGHj\nxo0EBQUxevRo+vfvT+fOnTlz5gwpKSmEhobSqVMnhg0bdt+xV69e5ebNm2RnZ1O7dm1SU1Np3rw5\n8+bNIygo6JF/1kpDl0/g8hbITYYtk2HocinFq7JxcjEkFN4Sen4t3cQfwKhWgWw4fZNzcRl8sO4c\n295qh426EqXOFHL0RgrLj8UA8EbnWng4GKdVxuOi9vLCb8GPZGzYQMKsLzBkZhI/Ywbp69biM2MG\nNnXqmNU+GRPh5AM9voL1L0t9y84sh8ZDzW3VPWTv30/8jE/Q3pRqna0CA/H+9BPsmzc3yuerFCrG\n1nmBTgd/Zpoih7M21mzIi+HA+j5MajaJPjX7FKWHW6uUfDmgAcMWHeXCzUy+233dbBM95kQURaZt\nOE9WgQ4nGxWf9K1X8kEyj0x8TjyfH/2cvTF7AbBR2vBq41cZUXcEKoXRHu0tm4BWUjuNc6tg90yp\nHtvOzdxWWSTGbp3xJlIU8fZyBXj79kZBED4QBOHC3QcIguCL5AQ+SNjGE/gNSAMSgY+Al0RR3Ghk\nuy2GuLg4du/ezezZs3F2dsbb25vp06eXeNz48eOpU6cOarWaYcOGcePGDT7//HPs7e2pVq0aHTp0\n4NixYw88NjlZivIsWbKErVu3EhkZSbVq1ejdu3ex0UgZpJvS7dqfq9vgkvHrSMs9GbGwq7BRboNB\nUsS1GFRKBV8MaIBSIRCZksv/FaZhViYy87VMWnkGUYTa3o682CrQ3CYBhVHGfv2oERaGYw+pZDz/\nzFkinhtIwhdfoM/OMbOFMiah0fMQXCgRsO29cpOOqktOJu6dd4kZP0FyFFUq3F95meob1hvNUSzi\nwLfUSLjM77cSebfmIGyUNqTmp/LRoY8YtW0U19KuFe3aKsiDUYX/s9/vuc7Z2HTj2mIBbDkXz86L\nUqr6R73qUsWxtGL4MqVBb9Cz7PIy+m3oV+QotvBpwdo+axlVf5TsKP6Xrp+C2l5qZbb7M3NbY7EY\n9a+qMB31pcLlQdtnIdU33r0urjg7RFHcA5T5tJS3gzdb+m8p83OWhtjYWAACAgKK1lWvXr3E43x8\n7qSB2NnZoVQq8fT0vGddVtaDmxE7OkraRW+++WbRuWbNmoWrqytXr16lbt26pbK9UtJwCJz+CyL2\nw9b3oGZHsHYs+biKgEEPaydIaWw2ztJNuwTq+zozpk11Fu6/wYJ94fRuVJUQ70pyvYAZGy4Ql56H\nlVLBvOcbY6Uy9nzek6H2qoLfvLlkH3iO+M8+QxsdTeqS38ncug2vD97HsXv3ii3LXtkQBKk+74en\npf/jsDdh2AqzZUiIOh1py1eQ9N13GDIk/Tzbxo3x/vQTbIJNEMW7dRYOfAuA8ukJvNjmY7o2HstX\n/37F7pjdnEw8yaCwQYyoM4JXGr+Cvdqe93rUZt/VJCKSc5i08gybXm9TaTIk0nI0TN8oZZG0CfJg\nUDM/M1tUsTiRcIIvjn7BlbQrADhbOzM5dPI9EW6Z/+BUFdq9C7s+geO/SbXYNTua2yqLo3w9iZQT\n1Ao1/k7+ZbqoFaUr/vb19QUgKiqqaF1EhGl7O4WEhGBra/vAm5F8gyoBQYBn54DSCrJuwu5KpCB7\naD5EHZTGvf8n3bRLwVtdauHnaovOIKnpGQzls1bK2Gw+e4u1p+IAmNw9hNreZStq8yg4tG1DjY0b\n8Jg4EUGtRpeYSNxbbxMzbjyau+5NMhUAJx8pfRzg2nazqaPm/PMPEf0HkDBzJoaMDBT29nh9PI2A\npX+ZxlHUa2HDq2DQgUuAJMwFVHWoyvxO8/m+8/f4OviiF/UsubiEPuv7sD1yOzZqBbMHNUIhwPXE\nbL7dccX4tpVTZm6+RHK2Blu1kln9G8jPB0YiISeB9/a/x6hto4ocxWdrPMuGvhvoG9RXvs4l0fJV\nScEYEdZNgJzKrYnwOMjOooXh5+dHhw4dmDJlCpmZmSQkJPDZZ6YNrdvY2DB69Gjmz59PTEwMBQUF\nTJs2jXr16hFsii/pioZHLWgzSRr/+xPcPG1ee8qCuBOwp9AxbjIC6vUr9aF2Vipm9qsPwKnodP46\nWvGdj/iMfD5Ydw6QeiqOaVNytoC5UdjY4PnG61TfuAH7Vi0ByDl4kBu9+5A4ezb6YjIVZCyQhkMg\nuKc03vZ+mTa61sTGEfvGm0SPGk3BNSnl07lvH2ps2YLbsGEIChM9xhyaB/FS+w36/h9Y3auo2s6v\nHev7rmdCwwmoFWoScxN5d9+7vPz3y7g4pzKhfU0AFh2M4OiNFNPYWI7YdzWJNSelzKd3ugVTzd3O\nzBZZPhq9hkXnFtF7fW+2REjZbiGuISzusZgv236Ju627mS20EFTWMPA3UNtBdgKsn1huBbvKK7Kz\naIEsXbqUgoIC/P39adu2LSNHjjT5OefMmUObNm1o1KgRvr6+REVFERYWhlJZOdJrnpg2b4NbTRAN\n0my1JtfcFpmOgmxYM06akXerIYlkPCIdQqrQp5EUifxq2xXiM0rSyLJcDAaRyavPkJGnxdFGxbeD\nG6FQWM5MsXX16vj/8gu+c75F5emJqNGQsugXwrt1J23ZMkS5rtnyEQSpHZCNCxRkwMY3TP6wZcjL\nI+l/33Hj2WfJ2rEDAJv69QlYtpSqX32F2suEbbASL8G+wmhq6EtQvd0Dd7NR2fBak9dY13cdraq2\nAuDwzcMM2DiAXMcV1PIREUV4d/UZcgoq7v9BToGOD9ZKk12N/F0Y3br8T3aVd/bH7mfAxgHMPzmf\nPF0eztbOfPT0R6zotYJmXs3MbZ7l4VHr3gyJoz+Z1x4LQ6hsTZZDQ0PF48eP37MuP196ELWxkQux\njYF8PYsh4gAs6Q2I0PB56L+gYqqjbngVTv0JChWM2QG+j/fFlpRVQJc5+8jI09K2lgeLRzdHaUFO\nVGlZfCiCGWEXAZj/fGP6NvY1s0WPjz47h5SffyZ18WLEggIArGrWxGvKZOzbtZPTpSydMytg3Xhp\n3Pf7B7bBeVJEg4HMLVtJ/PZbdLckQR2luztVJr2Nc//+posk3kavg1+7SdkRzv4w8Uip6sxFUWRH\n1A7mnphLXLaUTm6jtCUroR35ya0Z1jyIWf0rZrupGRsvsPhwJGqlwKbX21aqOnNjczn1MvNOzuNQ\n3CEAFIKCQcGDeK3xa7jYuJRwtMxDEUVY/RJcWCuVBo3dBT4NzW2VWREE4YQoiqEl7SdHFmVkyorq\nbaHTR9L47HI4/ot57TEFF9ZLjiJAxw8e21EE8HS0ZnpvSTzpwLVkvtle8Wp/riVk8cXWywD0blTV\noh1FAKWDPVXefouaW7fg1Ls3AJrwcGImvEzMmLHkX6l4v8NKRcPBEFLYVmrb+5ARZ7SPFkWR7P37\niRjwHDfffVdyFFUq3EaNoua2rbg895zpHUWAf36QHEWQoqmlFCQTBIHugd3Z0G8D7zR7B0e1I/n6\nPNQe27GvOZuVl9ex50qCCQ03D4evJ7PkSCQAEzsEyY7iYxKTFcN7+99jUNigIkexaZWmLH92OR+1\n+Eh2FI2BIECvueBSDfQayXHUyErepUGOLCJHwoyNfD0fgsEAK4bDlS2gUMPoreD/lLmtMg4ZsfBj\nK0k1MaANvLix2J6Kj8JH68/x5z/RAPzfsCb0alg6oZzyjkZnoP8Ph7hwMxMfZxu2vdkOZ7vSCV1Z\nCnnnzpHw5VfknSh8+FYocO7XD4+Jr2DlJyslWiRZ8fD905CfDkFdYPjqJ86QyD15ksQ5c8g7fqJo\nnUP79lR5bwrWNWo8qcWl5+Zp+LU76PKlqGnf7x/7o9Lz01lwdgErLq9AJ0opqILGlzldPqRzYJsK\nEWWPSsmh7/eHSM/VEuLlyMbXW2OtkktTHoXkvGQWnl3Iqqur0Bmkv5MApwBeb/I63QK6VYi/k3JH\nzL/waw8Q9dDkBakmuZJS2sii7CwiOzfGRr6eJZCXDj93hNQb4FgVJuwDBxPW35QFBj383hciD0ht\nMl45DM7GcQY0OgPDF/3Dscg0bNVK1k5sRR2f8qsUWlq+3naZH/aGA/DX2KdpHeRhZotMgyiKZO3c\nSeLsb9FGS04/KhUuAwbg8fIE1FUrhvNfqTi7EtaOk8YdP4L2kx/rY/KvXCFp7jyy9+4tWmfbrBlV\nJr2NXbMyrsvKiINFnSHrFjj5wSuHwPbJozlRmVHMPDybfxL2Fq1rWqUpExtPpLl3c4t1BrLytfT/\n4TDXE7NxtlWz4dXWBHrYl3ygDAA52hwWX1jMkgtLyNPlAeBp68nLjV6mf63+pVbIl3lM9n8Du2dK\n44G/Qf0B5rXHTMjOYjHIzqLpka9nKUi4AD93Bl0eBLaFF9aD0oKb6R6YI/UxAhi0GOr1N+rHJ2bl\n0+e7Q8Rn5uPvZsvGV9vgam9l1HOUJcciUxny0xEMIoxpU51pvSp+r1KDRkP68hUkL1yIPrlQulyt\nxmXgc3hMmIDau3S9ZmXKAaIIa8bC+dXS+/4LodGQUh+uiYwk6YcfyAzbVCSUYx0SQpVJb5untrUg\nG37rAfHnpAbeY7YXSu0bj093bmZ5+PcobWOK1jXzasbERhN5yvspi3Ia9QaRsUuOsedKEkqFwB8v\nNadVBZ3sMjbZmmxWXFnB7xd/JzU/FQBHtSMvNXiJYbWHYaeWVWTLhLsnuK2d4eUD4BpQ8nEVDNlZ\nLAbZWTQ98vUsJXfPzrd6A7qZtgWKyYg9IQlCGHTQeAT0e/zUrYdxKjqNIT/9g0ZvoG0tD34b9RQq\npeWVXUcm5zBwwRGSswsI8XJkw2utK03TbpBULtOWryDl55/Rp0oPS4JajcvgwbiPH29alUsZ46HN\nlx62Yv6RUupHrofANg89JP/iRZIX/kzW9u1FTqLa3x/PN97A6dlnyqYm8b8Y9LBihFQagABDl0FI\nT+OfxiAy7o/j7I0+gLXn3w90Gpv7NDf6eU3BrC2XWLj/BgCf9avPCy0q30P2o5Ken86fl/5k6eWl\nZGmktkJWCiuG1RnGmPpj5JpEc5B5UyqdyUsDv+ZSWZAlT9o/BrKzWAyys2h65Ov5CGyZIvVeBBj8\nO9Tta157HpWkq/BbT8hNltpkTDgA1g4mO93KYzFMWSP1PpvQrgbvP1PHZOcyBfEZ+QxccJjYtDwc\nrVWseqUltb0tP6X2cTDk5pK2dCkpi35Bn54OgGBtjfOA/ri/+CJWgYHmNVCmZHJS4JeukBoupZ+P\n2QmeIfftlnv8OMk/LSTnwIGidSovL9wnjMd14EAEKzNmCWz/EI4U1ix1/wJaTjTZqfI0eoYt+odT\n0WlYO16lVu0jRGVfLtoe6hXKuAbjaFm1ZbmNNK46HsPk1dI9eESLaszsVzEVXo1FYm4iv1/4nZVX\nVxalm1oprBhQawBjGozB217OqDArl7fA8qHSuMWr0P3ziqlSXwyys1gMsrNoeuTr+QjoNFI7jZh/\nwMoBxu0Bz2BzW1U6UiMkRzHrlpTGMWpTmchQT1t/nj/+iQLgf0ObFPVjLO+k52oY/NMRriZkY61S\n8MeYp2le3c3cZpkdfXYOaX/9Rcqvv2LIyJBWCgIOnTrhPnoUts2aldsHZxkgJRwWdYG8VEllcOwu\ncKgiqZvu20fKwp/JO3myaHergADcx4/DuXdv8zqJAMd/g01vSePQl+DZOSZ/UEzN0TDwx8PcSM7B\nwVrJ1AGwOWYJ51POF+0T5BLEyLojeabGM1grrU1qz6NwIiqVoQuPotEbaFnDnd/HNEdtgdkdZUFs\nViy/nf+NddfXoTVoAbBT2TGk9hBG1h2Jh62ctltu2DIZ/l0ojdtNgU4fmteeMkR2FotBdhZNj3w9\nH5HMW/BTO8hJBI9gGLe71HLtZiMjTqrxSY+WanxGrgf/skmhulvwxkatYO0rralbtXxH53IKdAxf\ndJTTMekoFQILX2hG5zpe5jarXKHPziZ9+XJS//gTXcKdFgM2DRviPnoUjl27IqgqV4qQxRB9VJr0\n0hdgqNKEDJexpK1YTcFdrVKs69bBY/x46feoLAdp1+F74M/nJEXEmp1g2EpQlo2oSExqLgN+PExS\nVgGejtasebklkXknWHh2IWeSzhTt52bjxvMhzzM4ZDDutu5lYltxxKXn0ff/DpKcrSHA3Y71E1tb\ndN24KRBFkaPxR1l2aRl7Y/diEA0AOFk5MaLOCIbVGYaztbOZrZS5D70WVo2Cy5uk9x0/hPZTzGpS\nWSE7i8UgO4umR76ej0HkIelhS9SD31PSg4tdOY06ZSdKEcWU66CygeGroHq7MjUhKauA3t8dJD4z\nHz9XW9ZObEUVx/L591ag0zN2yXEOXJNEXeYOaUT/JnLbiOIQNRoyt24l5bfFFFy+k6Kn9vXF9YUR\nuPTrh9JFru8pbxTs/Jm07z4lI8IOg/ZOtMkuNBT3CROwb9O6/ESIk67Aoq5QkAGetWHMDimNtgw5\nH5fBkJ+OkKPRU8PTnjUvt8LV3oozSWf44+If7IzaWeRsWCms6F2zNyPqjCDINahM7QRpsmvggiNc\nupWJo7WKda+2IqhKOZ/QLENytDlsDN/IssvLiMiIKFrvbuPOi/VeZHDIYOzVslJsuUangZUvwNVt\n0vsuM6DN2+a0qEyQncViqIjOYmxsLP7+/kRERBAYGMisWbM4cuQIYWFhZrHH0q+n2Tj2C2yeJI09\na8OIteBczpq056ZKTm3CeUnU4vmlENzNLKacjkln8E9H0OgM+Lnasnh0c4KqmK5e8nHQG0TeWHaK\nzeduATC9d11Gt65uZqssA1EUyf3nH1J++42c/Xdq3QQrKxy7dcNl0CDsmluWimRFQ9TpyN67l7Sl\ny8g5fPjOBkHEqbEfrpO/wa5pE/MZ+CBykuHnTpAeBXYeMG4XuAaaxZQD15IY/dsxdAaRZgGu/DX2\n6SKxq7jsOJZeWsqaa2vI0d5pHN60SlP61+pPt4BuZaKcma/V89rSU/x9KQFBgF9ffIqOtWURKoCI\njAiWXV7GxvCN9/yOGng0YGjtoXQL7Fau0ohlSkBXAMuHwfW/pffdPodWr5nXJhMjO4vFUBmcRVPg\n4HDvQ7hWq0Wn05GQkICHx72595Z+Pc3KmeWwfqIUYXT2hxfWgUctc1slkZ8Jf/SDuBMgKKTeRPX6\nmdWkHRfieX3ZKQp0Bpxt1Sx6MZSnAstHRFYURT5Yd45l/0qqh290rsWkrhZSj1rOKLh2jZQlS8jc\ntBmx8P4CoA6ohsvAgbj064fK09OMFlYuNNHRZISFkb56Dbpbt4rWq7y8cGnshIvNAdS2hvL3sJUR\nC8uGQvxZUFpLddZllD5fHOtOxfL2Cin1tGtdLxaMaIZScWcCJFuTzbrr6/jr0l/EZccVrbdX29Mj\nsAcDag2ggUcDk0yaJGUVMOGP45yMlgSoPnymDuPa1TD6eSyJLE0WO6N2EhYexvGEO8+SaoWaHoE9\nGFp7KA08ZdEfi0WbB8uehxt7pfc9v4Gnx5vVJFMiO4vFIDuLxmH48OGkpaWxZcuW+7ZZ+vU0O1e3\nw8oXpR6Mdu4wfDX4NjWvTZpcqb4nujB60P8naPS8eW0q5ERUKmOWHCc9V4uVSsH8IY3p2cDH3Gbx\n1bbL/Lg3HIAXWwYwo089OQr2hOgzM8ncvJm0VasouHjpzgalEoeOHXAZMAD71q1RWMuz+cZGl5pK\n5tatZIZtIu/06Xu22T39NK7DhuHYqSOCQpDUBa/tkDY2GwU9vwaVmX8nkYdg5UhJuRkBnlsEDQaa\n16ZCFuwL58utUsp138ZV+WJAA+ys7q3P1Rl0HL55mLXX1rIvZh86UVe0LcgliH5B/ehVo5fRahsv\nx2cyZvFx4tIlBc+3uwTzRuegSnkP0xq0HLl5hLDwMPbE7KFAX1C0zcvOiyEhQxhQa4DZ60pljIQm\nF5YOlnowAvSaKwlgVUBkZ7EYSuMsilot2rtmS8sCtY8Pgrp0xfXx8fGMHz+effv24eXlxZQpUxg3\nblyRszhjxgwOHjzI339LofTAwEDGjh3Lrl27OHbsGNWrV+evv/7iwoULTJs2jaSkJAYNGsSCBQtQ\nlUJAIiUlBV9fX1auXEmfPn3u2y47i0Yg+igsHQT5GZJK6pA/oWZH89iSnwmrXoTw3dL7Z+fAU2PM\nY0sxhCdlM+q3f4lJzUMQ4KNn6zKmjXnSPXMKdHy26SLLj0kRxb6NqzJ3cGMUisr3kGVK8i5cIH31\najLDNmHIzi5ar7C3x6FTJ5x6dMe+TRvZcXwCDHl5ZO3aTZFmMCIAABeRSURBVGZYGNmHDoHujoOi\ndHPD6ZlncH1+CNZB/6mjK8iW6n9u3zOqNpVaA7n4l6H1hYgiHFsE26ZKvWCtnWDAzxDSo+xtKQZR\nFPkk7CKLD0cCUKuKA98Pb0qw14PrApPzktkUvom119feUyOnFJSEeoXSNaArnQM6P7bi5u7LCby+\n9BQ5Gj3WKgXfDm5Er4aWoTptLERR5GLKRcJuhLE1Yiup+alF29QKNe392tOrZi/a+7VHpZCFtyoc\nBdnSBHnMP9L7Pv8HTV8wr00mQHYWi6E0zqImOprwbt3L1K6aO7ZjVa1aqfbt3LkzTk5OLF68mLy8\nPAYOHMihQ4ce6iyq1Wo2btxIUFAQo0eP5siRI3Tu3Jm5c+eSkpJCaGgo8+bNY9iwYSWef/bs2cyf\nP5/IyEiUD1C1k51FI5FwEf4cILWmUFrBgIVQr3/Z2hC+Bza+DhmFDaS7zYRWr5etDaUkKauAlxYf\n41yc1H5hTJvqfPhMnTJ10k5Fp/H2itNEpuQC0Ll2FRa80EyWlzchhtxcMrfvIH3NavKOn7hnm8Le\nHoeOHXHq2UN2HEuJLjWV7P37yd67j5z9+zHk5hZtE2xtcezcGec+vbFv2fLhE5wGPez9AvZ/I723\ndYOBv5btpJeuADa/A6f+kN6715LqrMtheyKDQeTHfeF8u+MKBhFs1Ao+7VufQc38io3miaLImaQz\nrL22lm2R24r6+AEICDSp0oSuAV3pEtClVP38RFHkl4MRzNpyCYMIno7W/DwylMb+lUNQSqPXcDz+\nOHti9rAvdh+3cu4NGjSp0oReNXrRPbC7rGpaGcjPhD/6Q9xxQICWr0LHD8Cq4ogVyc5iMVi6sxgX\nF4efnx/Xr1+nZs2aAOzcuZNu3bo91Fl89dVXmTx5MgBbtmzh2WefJTExEc/COp/Bgwfj6+vL3Llz\nH3p+URQJCQlh+PDhTJ8+/YH7yM6iEUmLkm5WqeGAIKVzNR9n+qaxBVmw82M4/qv0XqGGrp+atGG1\nMcgp0PHa0pPsuZIEwLMNfPh2cKMi0QhTodMb+H5POP/bfQ29QUSlEHi7azAvt695T/2RjGnRJiSQ\ntX0Hmdu3S7397vp+U9jbY9+qJfatWmHfunWpJ+cqOqIoUnD1Ktl79pK9dy95Z87cc91QKLBv3Rrn\n3r1w7NwZhf0jPihd2QprJ0jKo4JCkqVvMwkUJp5AybwlRTdjj0nvg3tIE25lrHr6qPwbkcoby04R\nnyl9jw5o4stn/epjb/3w6FWuNpcDcQfYGbWT/bH773EcARp6NKSdXztaVG1BPfd690XDNDoD0zee\nL6qxruvjxKIXQ6nqYmvEn678kZ6fzoG4A+yJ2cPhm4fvEaoBqOZYjV41e9GrRi/8Hc0QGZcxL3np\n0jPYzcJesS4B0Hu++TK9jIzsLBaDpaehHj16lBYtWqDVaotSRq9fv06tWrUe6izOnDmTESNGALB3\n7166dOmC7q6UolGjRqFSqVi0aNFDz7979266detGVFQUvr4PVuqUnUUjk50Efw2EW4V1Qr7NoPN0\nqNHeNOe7sQ82vAYZ0dJ7n8bQ70fwqmua8xkZnd7AtA13Hnoa+DozuXsIbWt5mKTeJiolh7dWnOZU\noQhEDQ975j3fmIZ+lWM2vryiTUgka8cOMrdvI+/EvY4jgNrPr8hxtG/xNErn8u1EGBPtrVvknjhJ\n7rFjZO/ff49IDUiKs3YtW+DQvj1O3bqh8njCBuIp4VK9YEJh4/mQZ6R7iq2J/kdijsGKEZAdL71v\nNxk6fGB6B9VIpOZomLTyNHsLJ71qetrz/fCm1PYuXT/ZfF0+h24e4u+ov9kbs5dsbfY92x3UDjzl\n/RQtfFrQomoLHBVVeWPZaY7cSAGgW10v5g5pXKKDaolka7I5nXSaEwknOB5/nLPJZ4talNymrntd\nOvh1oL1/e+q41amUdZoyd6HNlzIkDs2TUtkBGo+Abp+V3xZnpUR2FovB0gVubovZPGpk0VjO4uDB\ng9Fqtaxbt67YfSzpeloMBVmwdjxcuUtQqEYH6Pyx5Dwa5RzZ8Pd0qb4HpGhi+/egzVtl1qzaWIii\nyPd7rjN7x9WidU2qufBm51q0D/Y0ype/KIqsOh7LjLAL5Gr0AIxoUY0PnqlznziFjHnRJiSSvXsX\nOYcPk3Pkn3tqHAFQKLCpVw/bRo2wbVAfmwYNsQoMQLAQ5+JhiAYDBdevk3fyJLnHT5B78gS6m/dP\nhqqqVMGhQwccOnTAvsXTKOyM3JZBkwub3oKzK6T3bjWg+ywI6gpKI/2/JF2RWhCd+A30GlDbQ/8f\noW5f43x+GWIwiCw8cINvtl9BbxCxVimY3rsezz/l/0ip9Rq9hn9u/cPu6N0cuXmEmzk379tH1Dmh\ny66JPr8afes8zWc9u2Krrhjf36n5qZxKOMXxhOOcTDzJ5dTL9zmHVgormvs0p6N/R9r5tStVyq5M\nJST+vFSWczvKaO8pZXvV62/6bC8TYRZnURCEN4DhQAPgpiiKJXaPFQShB/AtUAMIByaJorjjru1B\nwAKgJZAGzBVF8dvHtdHSnUWAjh074ubmxm+//UZeXh6DBg3iwIEDJncWExMT8fPzIywsjO7di0/T\ntbTraVHc2Ae7PpHaV9ymTh/oNO3x63AKsuD6LintND1KWufdUJr5967/5DabkYPXkvl255WiqB9A\nI38X3uwcRMeQKo/lNKbnath9OZF1p+I4cC0ZAA8HK74e2JBOtb2MZruMaRB1OvLOnZMcx0OHpbRL\nvf6+/RSOjtjUr4dt/QbYNGyATZ06UgbIA+q0ywuG3FwKwsMpuHqNgmvXKLh6lbzz5zFkZt63r2Bl\nhW3Dhti1aIFDxw7Y1K1r+ghKkdjM+2DQSuscvKHxMGgyAtxrPvpn6rXSJNqxRRCx/85610B4fpnF\nZEQUx4moVF5feoqbGdL3qq+LLb0bVaVfk6qljjTeRhRFYrJiWH5uF5uvHyBFfwFBmXvffipBRZBr\nEPXc61HXvS71POpRy6UWVkoro/xMpkBv0BObHcvVtKvSkiq9xmbH3revgEAt11o0rdKUFj4taFm1\nZZn0rJSpABj08M+PsHumpFgPUqZEjy/BNcC8tj0G5nIWBwIiUBsYXZKzKAhCDeA8MB5YCQwCFgL1\nRFGMFARBWbj9b+C9ws/dBrwuiuKKx7GxIjiLt27dYty4cRw4cKDUaqjGcBa//PJLFi5cSHh4+EMf\nKiztelocogiXN8GuzyD5irROUECjYdB4qPSQ5OgDimIeag16uHlaUioM3w2x/95JrVCooN0UaDvJ\n4qKJxSGKIgevJzP/72scj0orWt/A15k3OteiTZAHtlYPdwBiUnPZcTGBnRfjORaZht5w577ZpU4V\nvnyuIR4OsniKJaLPyiL333/JPXacvPPnyL9wETEv74H7Cmo16mrVsAoIwCow8M5rYAAqd3eEUqhJ\nPwmiKGLIykIbH48uIQFtfDza2DjJMbx2DW1s7H3ptrdROjtj26wZds2aYtu0KTb16qGwMtPDf+xx\naXIq6tC96wPaQNORULcPqEuolcuKhxNLpChi1l2RUic/CB0FT40zXZprGZOWo2HKmrPsvJhwz/oQ\nL0f6NK5Kn0ZV8Xd7uLOj0xvYej6enw/c4GxsRuFaA57uKTQNSQKbG1xOu3iP6ufdKAQFPvY++Dv6\nU82xGv6O/vg7SWM/Rz9sVaavbdToNSTkJHAr5xbxufHcyr5FXHYc19Ovcz39+n11mrdRCkrqudej\nqVdTmnk1o0mVJrJAjcyTkRohZUrc7scIkoBWYGvpPhbQCpwfXKpVnjBrGqogCKOAj0rhLH4CdBJF\nse1d6w4Af4ui+IkgCB2BzUAVURSzC7d/BrQRRfGxqksrgrNY3pGvZxlh0MOZ5ZLi4G210tso1JJM\nvUuANNvlGii14Ig8KN3c8tP/82GCdHPr+RV4V8yGwqIociQ8hXm7rvFvxL0PRA7WKqo4WuPhaI2n\nozWeDtJrnkbP35cSuByfdc/+VkoFLWu6M6CpL30aVZVrWioQok5HQfgN8s+fI+/sOfLPnSP/6tV7\n2kY8EEFA6eyM0s0NpZsrKle3orHS2RlBrUZQqRFUKgS1CpRK6b1ahajXI+bmYsjNxZCbJ73m5WHI\nzcGQm4suKQldfALahATE3PsjQf9F4eyMTa1aWAfXwjqkNnbNmmJVo0b5S6tNCZeUSk8vhey7HCFr\nZwjqLE1eiXrpXmfQgWiQXrV5EH3kziQXQM1O8NRYqNXdeGmt5YzL8ZlsOH2TjadvFvU/vE1ogCvN\nAlzJ0+rJLtCRU6AjV3NnnJytITVHU7R/XR8nxrerwbMNfYrUmkVRJCE3gQspF7iYclF6Tb5IWkEa\nJeFo5YirtSsu1i44WzvjauOKs7Wz9N7KGZVChUJQoFQoUQrSohAUKAUlelFPjjaHXF0u2ZpscnQ5\n5Ghyil6T8pKIz4knJT+lRDvUCjVBLkHUcq1FsGswIW4hNPRoKEcOZYyPKEr3rh0fQt4D/kdcAyXH\nMbA1BLQul5FHS3EW1wORoii+dde6+YC/KIoDBEF4CxglimLju7b3B34RRbHUVaWCILgD7gCNGjW6\ncvo/DYVl58a4yNezjNEVwPHf4PB3kHl/yk2xOPtLil41O0H19hZfqP0oHAlP4X+7rhUJOpQGZ1s1\nnWpXoWtdL9oFe+JQAcUfZB6MoaAATVQUmsjI/7xGoU9ONo9RgoDKwwOVjw/WQUFY16pVtKiqGKcu\nt8zQ6+DaDslxvLpdchBLg42zJDQR+hJ4lFj1UmEwGERORKex4XQcm8/eIi1XW+pjO4Z4Mq5tDVrW\ndC/V34goitzMucmN9BtEZ0UTkxVDdKb0Gpsdi85QwiSKiXCxdsHH3gdve29qutQk2DWYYNdgqjlV\nQ62oGFkxMhaCNk9SXY48JGVLxPwL+oJ793Hyg7fPl7vaRqM6i4IgLAZefMgun4ui+NFd+4+idM7i\nLuCgKIrT71r3CdBaFMUugiBMA7qIotj+ru0dgZ2iKJb6SU0QhBnAdAAfHx9u3ry3wFt2boyLfD3N\nSH6mVHeYFvWf10jITZXEcGp2kpxE96Byd+Mqa6JTcrmZkUdiVgFJdy/Z0qveYKB1kAdd63rxVKCb\n3C9R5j702dloo6PRpaSgT01Fl5omvaalok9NQ5+Sgj4zE1GnQ9TpoPD17vcoFCjs7IoWwc5WGttK\n71Xu7qi8vVF7e6HyKnz19CyVgrbFkRUPZ5ZJYhIKJQhK6VWhlCKNt9971YN6A8CqckeMNDoDB68n\nEXbmFjfT83CwVmFnrcLBWom91V1jaxVPV3cjqIqj0c6tN+iJz40nJiuGlLwU0gvSpSU//c64IJ3M\ngkx0og6DaMAgGtCLegyGwlfRgCAIOKgdsFfbY6e2w15tf2dR2eNu6463vTfe9t742PvgZeclRwpl\nyi/afElXIuqQlM0V86+UWj9gobktuw9jO4sOwMOe/HNFUcy9a/9RyJHFSot8PWVkZGRkZGRkZCo9\nOg0UZIL9E7YgMgGldRZLFZ0rrBfMLnHHR+cM8N/awybArru2BwuCYC+KYs5d2888yklEUUwBUgBC\nQ0u8JjJPiCiKlpUCJSMjIyMjIyMjI2NsVFagKn+O4qNg1JwqQRBUgiDYAGrprWBT+L44fgdCBUEY\nKgiCWhCEoUAzYEnh9v1AFDBLEARbQRAaAxOAn4xpt0KhQP8A6XSZx0N2FmVk/r+9+w3Vs67jOP7+\n7JyzTi2dbpoV9mcqG0Q0S6kIg8ywMmY5kZIyMrBIIguJfFAPLPojswdRGDlTLAiL1AeVUYlWUyuq\nicuohpMtJIpNPM7l0bn57cF1Hb13rhPTc+5z3zvnvF8w7l3X7xr7DT6c+/fd788lSZK08PV7A87n\ngUma11+c1P7+mSO7knwwyTMzlFW1A9jY/rm97ed5VbWzbT8IbABeSzMzeBuwqapu6menx8bGmJyc\nZD4O+1mKnnzyScYW414aSZIkaQmZl9NQj2QzvToD4MCBA0xMTDA+Ps7IyIgzY7NQVezfv59ly5Zx\n1FH920QvSZIkqX/6umdxKRgdHWX16tU89dRTPP3008PuzoKUhBUrVjA6zy+mliRJkjT/HNX3SMLy\n5cuH3Q1JkiRJGjpfGiZJkiRJ6rBYlCRJkiR1WCxKkiRJkjosFiVJkiRJHUvu1RlJdgO7ht2PaUaA\nE4D/AAeH3BctTmZMg2DONAjmTINgzjTfhp2xV1XV8Yd7aMkVi0eiJGuBfwDrqmr7sPujxceMaRDM\nmQbBnGkQzJnm20LJmMtQJUmSJEkdFouSJEmSpA6LxSPDw8CV7ac0H8yYBsGcaRDMmQbBnGm+LYiM\nuWdRkiRJktThzKIkSZIkqcNiUZIkSZLUYbEoSZIkSeqwWJQkSZIkdVgsSpIkSZI6LBYlSZIkSR0W\ni5IkSZKkDotFSZIkSVKHxeIQJRlJsinJ7iSPJbk5yXHD7pcWriRXJflrkr1J/pVkc5JV0575cJId\nSR5P8ockpw2rv1r4kixLck+SSnJiz31zpr5I8o4kv0+yL8meJNf0tJkzzUmSlyb5YTsWeyTJHUnW\n97SbMT0vST6QZEs7FjswQ/u72rHaZJL7k5w9rf2UJLcn+W+Sh5JcPrjed1ksDtcVwHuBNwFTg6zv\nD687WgQOAh8CVgPraXJ1w1RjkjOAbwOfAI4FbgZuS3L04LuqReIzwOO9N8yZ+iXJ24AfA1fT/Fw7\nEbiubTNn6odrgFXAOuAE4E/AT9MwY5qNR2hy9enpDUlOAm4BvgqsbD9vTfLqtn0E+AnwN+B44Fzg\nc0neP4iOzyRVNay/e8lLsgv4YlV9t70+GXgAWFNVO4fZNy0OSd4D/KCqVrbXNwLLquqi9jrALuAL\nVXXj8HqqhSjJWuDnwPnAvcArquohc6Z+SfI74DdVdcUMbeZMc5ZkG/Ctqrq2vV4H/J1moP51zJhm\nqf3PrturarTn3pXA26vqrT33trTPXZnkTOBnwEuqal/b/iXgjKo6c6D/gJYzi0OSZCXwSuDPU/eq\nagewF3jdsPqlRecsYFvP9XoOzVzRDPLXIz0PSZYB1wOfBSamNZszzVmSFcAbgSeSbG2XoP46yent\nI+ZM/bAJOD/JcUnGgY8Bd1XVHsyY+u+QTLW28mym1gPbpwrFGdoHzmJxeKaWMDw67f5ET5s0a0nO\nBy4BLuu5fRRmTv1xGfDvqrplhjZzpn44lmaccgnwEeDlwC9plgEegzlTf9wNjAC7gX3ARprMgRlT\n/x0uU0dc5iwWh+ex9nPltPvH0MwuSrOW5AJgM3BuVW3taXoMM6c5SnIKcDnwyf/ziDlTP0x9T95Q\nVduqaj/N/p4x4C2YM81Ru0LidmA7TZZeBHwZ2JLkBMyY+u9wmTriMmexOCRVNQH8E3jD1L120+vR\nHLpsUHpeklwMfAfYUFV3Tmu+j0MzF+DU9r70XJ1Bs5/n/iR7aJbIAGxLcinmTH1QVY8CO4GZDlco\nzJnmbhWwBvhmVe2tqv1VdR3N+PjNmDH13yGZar2eZzN1H7C2XYY/U/vAWSwO17U0JxytaU/Wugr4\nhYfbaLaSfIrm1MB3VtXdMzyyGdiY5Kwky2lmh8aBWwfYTS18PwJOphk0nQqc094/G/ge5kz9cw1w\ncZLXJBml2SP7BHAP5kxz1O5L3A5cmmRFktEkH6VZCvgXzJhmIc2r8caB5e31ePsrNN+Rpye5MMlY\nkguB04CpA5N+S3OI0leSvDDJqcDHaSYBhmL08I9oHn2NZk/GH4EXAL+iee2BNFvfAA4AdzY/kxpV\n9eL286525mcz8DKaL8NzqsolNXrOqupxel6X0Q7iodnDuA8wZ+qXq2kG7nfQDNLvBd7dzjqaM/XD\n+2gOudlFs8T5AeCCqnoQeNCMaRYuoue1ZcBk+7mmqnYk2Uhz0u71wIPAeVMTRVV1MMkGmuLwYZr9\nipuq6qZBdX46X50hSZIkSepwGaokSZIkqcNiUZIkSZLUYbEoSZIkSeqwWJQkSZIkdVgsSpIkSZI6\nLBYlSZIkSR0Wi5IkSZKkDotFSZIkSVKHxaIkSZIkqeN/VLa7q2tZ3AEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb26362f588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(15, 5))\n",
    "pe = PositionalEncoding(20, 0)\n",
    "y = pe.forward(Variable(torch.zeros(1, 100, 20)))\n",
    "plt.plot(np.arange(100), y[0, :, 4 : 8].data.numpy())\n",
    "plt.legend([\"dim %d\"%p for p in [4, 5, 6, 7]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Full Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Here we define a function that takes in hyperparameters and produces a full model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_model(src_vocab, tgt_vocab, N=6,\n",
    "               d_model=512, d_ff=2048, h=8, dropout=0.1):\n",
    "    \"Helper: Construct a model from hyperparameters.\"\n",
    "    c = copy.deepcopy\n",
    "    attn = MultiHeadedAttention(h, d_model)\n",
    "    ff = PositionwiseFeedForward(d_model, d_ff, dropout)\n",
    "    position = PositionalEncoding(d_model, dropout)\n",
    "    \n",
    "    model = EncoderDecoder(\n",
    "        Encoder(EncoderLayer(d_model, c(attn), c(ff), dropout), N),\n",
    "        Decoder(DecoderLayer(d_model, c(attn), c(attn), c(ff), dropout), N),\n",
    "        nn.Sequential(Embeddings(d_model, src_vocab), c(position)),\n",
    "        nn.Sequential(Embeddings(d_model, tgt_vocab), c(position)),\n",
    "        Generator(d_model, tgt_vocab)\n",
    "    )\n",
    "    \n",
    "    # This was important from their code. \n",
    "    # Initialize parameters with Glorot / fan_avg.\n",
    "    for p in model.parameters():\n",
    "        if p.dim() > 1:\n",
    "            nn.init.xavier_uniform(p)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EncoderDecoder(\n",
      "  (encoder): Encoder(\n",
      "    (layers): ModuleList(\n",
      "      (0): EncoderLayer(\n",
      "        (self_attn): MultiHeadedAttention(\n",
      "          (linears): ModuleList(\n",
      "            (0): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (1): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (3): Linear(in_features=512, out_features=512, bias=True)\n",
      "          )\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (feed_forward): PositionwiseFeedForward(\n",
      "          (w_1): Linear(in_features=512, out_features=2048, bias=True)\n",
      "          (w_2): Linear(in_features=2048, out_features=512, bias=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (sublayer): ModuleList(\n",
      "          (0): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "          (1): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "        )\n",
      "      )\n",
      "      (1): EncoderLayer(\n",
      "        (self_attn): MultiHeadedAttention(\n",
      "          (linears): ModuleList(\n",
      "            (0): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (1): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (3): Linear(in_features=512, out_features=512, bias=True)\n",
      "          )\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (feed_forward): PositionwiseFeedForward(\n",
      "          (w_1): Linear(in_features=512, out_features=2048, bias=True)\n",
      "          (w_2): Linear(in_features=2048, out_features=512, bias=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (sublayer): ModuleList(\n",
      "          (0): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "          (1): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (norm): LayerNorm()\n",
      "  )\n",
      "  (decoder): Decoder(\n",
      "    (layers): ModuleList(\n",
      "      (0): DecoderLayer(\n",
      "        (self_attn): MultiHeadedAttention(\n",
      "          (linears): ModuleList(\n",
      "            (0): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (1): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (3): Linear(in_features=512, out_features=512, bias=True)\n",
      "          )\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (src_attn): MultiHeadedAttention(\n",
      "          (linears): ModuleList(\n",
      "            (0): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (1): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (3): Linear(in_features=512, out_features=512, bias=True)\n",
      "          )\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (feed_forward): PositionwiseFeedForward(\n",
      "          (w_1): Linear(in_features=512, out_features=2048, bias=True)\n",
      "          (w_2): Linear(in_features=2048, out_features=512, bias=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (sublayer): ModuleList(\n",
      "          (0): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "          (1): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "          (2): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "        )\n",
      "      )\n",
      "      (1): DecoderLayer(\n",
      "        (self_attn): MultiHeadedAttention(\n",
      "          (linears): ModuleList(\n",
      "            (0): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (1): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (3): Linear(in_features=512, out_features=512, bias=True)\n",
      "          )\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (src_attn): MultiHeadedAttention(\n",
      "          (linears): ModuleList(\n",
      "            (0): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (1): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "            (3): Linear(in_features=512, out_features=512, bias=True)\n",
      "          )\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (feed_forward): PositionwiseFeedForward(\n",
      "          (w_1): Linear(in_features=512, out_features=2048, bias=True)\n",
      "          (w_2): Linear(in_features=2048, out_features=512, bias=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (sublayer): ModuleList(\n",
      "          (0): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "          (1): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "          (2): SublayerConnection(\n",
      "            (norm): LayerNorm()\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (norm): LayerNorm()\n",
      "  )\n",
      "  (src_embed): Sequential(\n",
      "    (0): Embeddings(\n",
      "      (lut): Embedding(10, 512)\n",
      "    )\n",
      "    (1): PositionalEncoding(\n",
      "      (dropout): Dropout(p=0.1, inplace=False)\n",
      "    )\n",
      "  )\n",
      "  (tgt_embed): Sequential(\n",
      "    (0): Embeddings(\n",
      "      (lut): Embedding(10, 512)\n",
      "    )\n",
      "    (1): PositionalEncoding(\n",
      "      (dropout): Dropout(p=0.1, inplace=False)\n",
      "    )\n",
      "  )\n",
      "  (generator): Generator(\n",
      "    (proj): Linear(in_features=512, out_features=10, bias=True)\n",
      "  )\n",
      ")\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dongdongdongdong/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:21: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.\n"
     ]
    }
   ],
   "source": [
    "# A small example model\n",
    "tmp_model = make_model(10, 10, 2)\n",
    "print(tmp_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "This section describes the training regime for our models.\n",
    "\n",
    "We stop for a quich interlude to introduce some of the tools needed to train a standard Encoder-Decoder model. First we define a batch object that holds the src and target sentences for training, as well as constructing the masks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batches and Masking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Batch:\n",
    "    \"Object for holding a batch of data with mask during training.\"\n",
    "    def __init__(self, src, trg=None, pad=0):\n",
    "        self.src = src\n",
    "        self.src_mask = (src != pad).unsqueeze(-2)\n",
    "        if trg is not None:\n",
    "            self.trg = trg[:, :-1]\n",
    "            self.trg_y = trg[:, 1:]\n",
    "            self.trg_mask = self.make_std_mask(self.trg, pad)\n",
    "            self.ntokens = (self.trg_y != pad).data.sum()\n",
    "            \n",
    "    @staticmethod\n",
    "    def make_std_mask(tgt, pad):\n",
    "        \"Create a mask to hide padding and future words.\"\n",
    "        tgt_mask = (tgt != pad).unsqueeze(-2)\n",
    "        tgt_mask = tgt_mask & Variable(\n",
    "            subsequent_mask(tgt.size(-1)).type_as(tgt_mask.data))\n",
    "        return tgt_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we create a generic training and scoring function to keep track of losses. We pass in a generic loss compute function that also handles parameter updates.\n",
    "\n",
    "## Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def run_epoch(data_iter, model, loss_compute):\n",
    "    \"Standard training and logging function.\"\n",
    "    start = time.time()\n",
    "    total_tokens = 0\n",
    "    total_loss = 0\n",
    "    tokens = 0\n",
    "    \n",
    "    for i, batch in enumerate(data_iter):\n",
    "        out = model.forward(batch.src, batch.trg, batch.src_mask, batch.trg_mask)\n",
    "        loss = loss_compute(out, batch.trg_y, batch.ntokens)\n",
    "        \n",
    "        total_loss += loss\n",
    "        total_tokens += batch.ntokens\n",
    "        tokens += batch.ntokens\n",
    "        \n",
    "        if i % 50 == 1:\n",
    "            elapsed = time.time() - start\n",
    "            print(\"Epoch step: {}, Loss: {}, Tokens per Sec: {}\".format(i, loss/batch.ntokens, tokens/elapsed))\n",
    "            start = time.time()\n",
    "            tokens = 0\n",
    "    return total_loss / total_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Data and Batching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs. Sentences were encoded using byte-pair encoding, which has a shared source-target vocabulary of about 37000 tokens. For English- French, we used the significantly larger WMT 2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece vocabulary.\n",
    "\n",
    "Sentence pairs were batched together by approximate sequence length. Each training batch contained a set of sentence pairs containing approximately 25000 source tokens and 25000 target tokens.\n",
    "\n",
    "    We will use torch text for batching. This is discussed in more detail below. Here we create batches in a torchtext function that ensures our batch size padded to the maximum batchsize does not surpass a threshold (25000 if we have 8 gpus)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "global max_src_in_batch, max_tgt_in_batch\n",
    "\n",
    "def batch_size_fn(new, count, sofar):\n",
    "    \"Keep augmenting batch and calculate total number of tokens + padding.\"\n",
    "    global max_src_in_batch, max_tgt_in_batch\n",
    "    if count == 1:\n",
    "        max_src_in_batch = 0\n",
    "        max_tgt_in_batch = 0\n",
    "    max_src_in_batch = max(max_src_in_batch,  len(new.src))\n",
    "    max_tgt_in_batch = max(max_tgt_in_batch,  len(new.trg) + 2)\n",
    "    src_elements = count * max_src_in_batch\n",
    "    tgt_elements = count * max_tgt_in_batch\n",
    "    return max(src_elements, tgt_elements)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hardware and Schedule\n",
    "\n",
    "We trained our models on one machine with 8 NVIDIA P100 GPUs. For our base models using the hyperparameters described throughout the paper, each training step took about 0.4 seconds. We trained the base models for a total of 100,000 steps or 12 hours. For our big models, step time was 1.0 seconds. The big models were trained for 300,000 steps (3.5 days)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer\n",
    "\n",
    "We used the Adam optimizer with $β_1 = 0.9$, $β_2 = 0.98$ and $ϵ = 10^{−9}. We varied the learning rate over the course of training, according to the formula:\n",
    "\n",
    "$$ lrate = d^{-0.5}_{model}·\\min(step_num^{-0.5}, step_num·warmup_steps^{-1.5})$$\n",
    "\n",
    "This corresponds to increasing the learning rate linearly for the first $warmup_steps$ raining steps, and decreasing it thereafter proportionally to the inverse square root of the step number. We used $warmup_steps=4000$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Note: This part is very important. Need to train with this setup of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class NoamOpt:\n",
    "    \"Optim wrapper that implements rate.\"\n",
    "    def __init__(self, model_size, factor, warmup, optimizer):\n",
    "        self.optimizer = optimizer\n",
    "        self._step = 0\n",
    "        self.warmup = warmup\n",
    "        self.factor = factor\n",
    "        self.model_size = model_size\n",
    "        self._rate = 0\n",
    "        \n",
    "    def step(self):\n",
    "        \"Update parameters and rate\"\n",
    "        self._step += 1\n",
    "        rate = self.rate()\n",
    "        for p in self.optimizer.param_groups:\n",
    "            p['lr'] = rate\n",
    "        self._rate = rate\n",
    "        self.optimizer.step()\n",
    "        \n",
    "    def rate(self, step = None):\n",
    "        \"Implement `lrate` above\"\n",
    "        if step is None:\n",
    "            step = self._step\n",
    "        return self.factor * \\\n",
    "            (self.model_size ** (-0.5) *\n",
    "            min(step ** (-0.5), step * self.warmup ** (-1.5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_std_opt(model):\n",
    "    return NoamOpt(model.src_embed[0].d_model, 2, 4000,\n",
    "            torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Example of the curves of this model for different model sizes and for optimization hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEECAYAAAAcdIIXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXdYlNe2h989FOkgCAoqgmAFRbH3HvVYEjSJSYwaje2Y\ncm6uUZKT3i3nJDc5qcaTmGiiSSyJNTGosXcQu4iA2BCpQxWY2fePgQkgyEAoDuz3eeaB+dYuawad\n3+z9rb2WkFKiUCgUCkVNo6lrBxQKhULRMFCCo1AoFIpaQQmOQqFQKGoFJTgKhUKhqBWU4CgUCoWi\nVlCCo1AoFIpaQQmOQqFQKGoFJTgKhUKhqBWU4CgUCoWiVrCsawfuBZo0aSJ9fHzq2g2FQqEwK44f\nP54kpXQ3tb0SHMDHx4djx47VtRsKhUJhVgghLlemvdpSUygUCkWtoARHoVAoFLWCSYIjhLAQQiwV\nQtwSQmQIIdYJIZrcpf0oIcQZIUSOEOK0EOK+UnZ/IUSYECJLCHFVCDG/lP1ZIcRhIUS2ECK6nDmm\nCiEuFbY5LIToZsprUSgUCkXdYOo9nBeA+4FeQDLwFbASGF26oRCiNbAemA38CDwEbBBCBEgp44QQ\nFsAmIAwYD7QHfhVCXJVS/lA4zHVgSaFtehlz9Ac+A0KA3cA/gK1CiDZSSq2Jr0mhUNQRBQUFFBQU\n1LUbirug0WiwsrJCCFF9Y5rYbjawWEoZI6VMBxYCo4QQPmW0nQYcl1KuklLmSSm/A8ILrwMMBFoB\nL0ops6WU4cAXwNyiAaSUa6WU64Br5fgzC1gvpdwupbwNLAVuYxAgkxBCuAkh2goh2qp/+ApF7ZGR\nkUF2djaqFte9TX5+PsnJydX6xaDCFY4QwhnwBo4XXZNSXhJCaIHOQFypLkHF2xYSXni9yB4lpcws\nZX+qEn4HASuK+SOFEBHF5jCFZ4DXABITEyvRTaFQVBW9Xk9BQQGNGzeua1cUJmBnZ0dycjJubm7V\nstIxZYXjVPgzvdT1tGK24jhW0LYiuylUxxj/AdoB7Tw8PCrRrWGTkJXAuqh13NbdrmtXFGZIfn4+\njRo1qms3FCYihMDGxob8/PxqGc8Uwcko/Olc6roLUNb9kowK2lZkN4W/PIaUMllKGSWljLK0VMeR\nTOWf+/7J6wdf59X9r9a1KwozREqJRqOCY80JCwsL9Hp9tYxV4V9eSpkGxAPBRdcKAwOcgJNldIks\n3raQroXXi+xthRD25dhNocQcwrDW61LJMRSVJCkniaMJRwHYGruVvVf31rFHCoWipqmLoIFlQKgQ\nwlcI4QQsBn6TUsaV0fZboLsQ4lEhhJUQ4lGgG/BNoX0PcBl4VwhhK4ToAszBEDgAgBDCUghhA1gZ\nngqbwudFfAlMEEIME0JYA/MBG2CDia9HUQX2XN1T4vlbh94iOz+7jrxRKBTmhqmCswhDKPNRDJFj\nFsDjAEKIyUIIYwCAlPISMAF4GcMW18tASJE4SSl1wDggEEOI9VZgqZRyTbH5XgZyMAhd68Lfc4rN\nsQ+Yh0F40oGHgb+pkOiaZVf8LgC8Hb2x1FhyI+sGH0V8VMdeKRQKc8EkwZFS6qSUz0spm0gpHaWU\nE6SUSYW276SUDqXa/yqlDJBS2hb+3F7KHi2lHCaltJNSekkp/1XK/rqUUpR+lGrzrZSydeEcPaWU\npSPjFNVITkEOB28cBGBmp5nM6jQLgO/PfU/kLbWTqah/PPHEE1hZWeHg4GB8fPrpp0Z7ZGQko0eP\nxtPTEyEE+/btK9E/KiqKBx98kObNm+Po6EhAQADLly83ef7Q0FCEEKxatarE9WPHjtGzZ0/s7Ozw\n8/O7w56YmMiECRNwdHTE3d2d0NDQEvdgdDodCxYswN3dHUdHRyZOnEhSUlJl3poqo+7eKUzi4PWD\n3NbdRiAY2GIgMzvNpLVzaySS1/a/pqLWFPWSadOmkZmZaXzMmzfPaLO2tmbChAls3LixzL6pqakM\nGTKEo0ePotVq+eKLL3j++edZv359hfMeOXKEbdu24enpWeJ6eno6o0ePZuLEiaSmpvL5558zd+5c\nDh48aGwzefJkAK5evcrhw4fZsGEDS5cuNdoXLVrEL7/8wuHDh7l69SoAU6ZMMf1N+Quo8CyFSey6\nYthO6+LRBTdbNwDe6PsG036dxqX0S3wY/iELeyysSxcVZki+Ts/1tJyKG1YjXi62WFn89e/aHTp0\noEOHDuXae/XqRa9evYzP+/fvz4gRI9i9ezcTJkwot9/t27d58sknWbZsGY8++mgJ2/r167G1tWXh\nwoUIIRgxYgQhISEsW7aMPn36EBsbS1hYGNHR0Tg7O+Ps7ExoaChvv/02oaGhACxbtoxXX32V1q1b\nA7BkyRL8/f2Ji4ujpsu0KMFRVIhOrzMGDAxuOdh4vYtHF2YEzmD5qeWsPLuSgS0G0tuzdx15qTBH\nrqflMGjpH7U65+4Fg2nlZl9xQ2DdunWsX7+eJk2acP/99/Paa6/h4OBQcccyyM7O5uDBg7zxxhvG\na/PmzSM+Pp7Nmzcbr73++usMHTqUPn363DFGZGQkwcHBJSLHgoODWblypdHu7OyMn59fCXtcXBxa\nrRYpJfHx8XTr9mfqST8/P5ycnDh58mSNC47aUlNUSOStSFJyUwAY0nJICdu8oHm0d20PwMv7Xkab\np+I2FPWDZ555hvPnz5OUlMSGDRvYvXs3s2bNqtJYOp2OKVOm4Ovry9SpU43XP/300xJic+zYMX76\n6SfeeeedMsfJyMjA2bnkEUQXFxe0Wu1d7QBardbY7m5j1CRqhaOokKLtNB8nH3ydfUvYrCyseK//\ne0zaPImb2Td59/C7LBqwqC7cVJghXi627F4wuNbnNIXiq4CAgAA++OADBg8ezIoVKyqVLSE/P5/J\nkydz48YNtm3bhpWVVZnt8vLymD59Op988km5qyhHR0fi4uJKXEtLS8PJycloT09Pv8NeZCvKX1dW\nm6IxahIlOIoK+ePKHwAM8R5Spt2/sT//0+1/WHJ0CVtitjCg+QDGtB5Tix4qzBUrC43J21t1TVGG\nhMokHc3NzeXBBx8kMzOT7du333U77vr165w5c8Z40x8MgQd///vf2bZtG9999x1BQUFs2FDyuGFE\nRARBQYY0kkFBQaSnpxMTE2O8RxMREYGPj49xVePt7U14eDhdunQBICYmBq1WS+fOnU1+XVVFbakp\n7kpMegxx2jgAhrYcWm67yR0mG+/fvHnwTWLTY2vDPYWixlizZo1xdXDx4kXmz5/P+PHjsbExnEGX\nUpKbm0tubi5gWKHk5uai0+kAyMzMZPTo0eTl5bFt27YK7/20bNmS+Ph4Tpw4YXx4eXnx7rvv8tFH\nhvNuISEhZGdns3TpUvLy8tixYwfr169n9uzZAPj6+jJ8+HAWLlyIVqslNjaWxYsXM2fOHOM8s2fP\nZvHixcTGxqLVagkNDWXkyJE1fv8GMLxpDf3RrVs3qSib5SeXy8AVgXLgmoGyQFdw17a3sm/JQWsG\nycAVgTLklxCZk59TS14qzIWcnByZk2Me/y4GDRokGzduLO3s7KSPj4987rnnZHp6utEeGxsrgTse\nX3/9tZRSyhUrVkhA2traSnt7e+Njzpw5xjHmzJkjR40aVa4PrVq1kitXrixx7ciRI7JHjx7SxsZG\n+vr63mG/efOmDAkJkQ4ODtLNzU0uWLBA6nQ6o72goEDOnz9furm5SQcHBxkSEiJv3bpVrg93+5sB\nx2QlPmuFVDUp6N69uzx27Fhdu3FP8vjWx4m8FUmIfwhv9nuzwvaHbxxm9u+z0Us9E9tM5PW+r9e8\nkwqzoWg1ULRKUNz73O1vJoQ4LqXsbupYaktNUS5JOUmcvGXIz1o6Oq08enn2Ym6QoZbeuovr2Byz\nuYIeCoWioaAER1Euu6/sRiKxsbCht5fp52tmd5pNL0/Dgbc3D77JhZQLNeWiQqEwI5TgKMqlKDqt\nj1cfbC1NCyUFsNBYsGjAIjzsPMgpyOHZnc+SmptaQ14qFApzQQmOokyy87ONyTpN3U4rThPbJnw4\n5EOsNdZcz7rO/N3zyddXT9VAhUJhnijBUZTJwRt/Jusc1HJQlcYIbBJoDBo4mnCUpUeX3r2DQqGo\n1yjBUZRJUe2bLh5dcLVxrfI44/zGMbWjIZXH6vOrWRu1tlr8UygU5ocSHMUdFE/WWZXttNI81+05\n+ngaEhG+feht9l/b/5fHVCgU5ocSHMUdnLh1gtTbhpv81SE4lhpLlg5aSmvn1uikjv/94385n3L+\nL4+rUCjMCyU4ijsoik7zdfbFx9mnWsZ0buTMZ8M/o4ltE7ILspkXNo8bmTeqZWyFQmEeKMFRlEBK\nacwOXR2rm+J4OXjxybBPsLW05VbOLebtmKfKGSgUDQglOIoSxKbHcll7Gah+wQHo6NaRfw/6NxbC\ngui0aJ7Z8Qw5BbVb8VGhMIUnnngCKysrHBwcjI9PP/3UaI+MjGT06NF4enoihGDfvn0l+kdFRfHg\ngw/SvHlzHB0dCQgIYPny5XedU6fTERoaSsuWLXF0dKRTp06sXVsy0ObYsWP07NkTOzs7/Pz8WLVq\nVQl7YmIiEyZMwNHREXd3d0JDQ9Hr9SXmWLBgAe7u7jg6OjJx4kSSkpKq+jZVCiU4ihLsvLITAFcb\nVzo16VQjcwxoMYBXer8CQHhiOM/teo48XV6NzKW4x9HlQ0pM7T50pp8HmzZtGpmZmcbHvHnzjDZr\na2smTJjAxo0by+ybmprKkCFDOHr0KFqtli+++ILnn3+e9evXlzvfJ598wsqVKwkLC0Or1fLWW2/x\n2GOPcf684Z5neno6o0ePZuLEiaSmpvL5558zd+5cDh48aByjqLzB1atXOXz4MBs2bGDp0j+PJCxa\ntIhffvmFw4cPc/XqVQCmTJli8nvyV1D1cBQlKNpOG9xyMBYaixqbZ2LbiWTmZ/KvY/9i//X9vLD3\nBZYMXIKlRv2TbFCkX4GPutbunM9GgGvrvzxMhw4d6NChQ7n2Xr160atXL+Pz/v37M2LECHbv3s2E\nCRPK7BMdHc3gwYNp164dAA888ABubm6cPn2a9u3bs379emxtbVm4cCFCCEaMGEFISAjLli2jT58+\nxMbGEhYWRnR0NM7Ozjg7OxMaGsrbb79NaGgoAMuWLePVV1811stZsmQJ/v7+xMXFqRLTitojKSeJ\nU7dOATWznVaaaQHTjIk+f7/8O68deA291FfQS6GoPdatW4erqytt27ZlwYIFZGZmVnms7OxsDh48\nWKLQ2bx58xg7dqzx+axZszh9+jRnz55Fp9Oxdu1aCgoKGDhwIGDYxgsODkYIYewTHBxMZGSk0e7s\n7Iyfn18Je1xcHFqtlvT0dOLj40tUM/Xz88PJyYmTJ09W+bWZivo6qTBSlKzT1tLWWEytppkXNI/M\nvExWnVvFxksbaWTRiJd7v4xGqO9CDQLnloYVR23PaQLPPPMMixcvxt3dnXPnzjF9+nRmzZrF6tWr\nKz2lTqdjypQp+Pr6MnXqVOP14veEAFq3bs2AAQMIDAxEo9HQqFEjVq5ciYeHBwAZGRnGyp1FuLi4\noNVq72oH0Gq1xmqldxujJlGCozBStJ3Wx7MPNpa1U69ECMHCHgvJLshm/cX1/BT1E3qp59U+ryrR\naQhYWFXL9lZNUHwVEBAQwAcffMDgwYNZsWIFjRo1Mnmc/Px8Jk+ezI0bN9i2bRtWVlbltp03bx4X\nL14kNjaWli1bcujQIR544AEcHBy47777cHR0JC4urkSftLQ0nJycAHB0dCQ9Pf0Oe5GtSHDKalM0\nRk2i/kcrAEOyzkM3DgEwxLvmt9OKI4Tg1d6vEuIfAhjq6Ly6/1V0el2t+qFQ3A2NxvBxWZmilbm5\nuYSEhJCYmMj27dvvWFmU5vjx40yZMoVWrVqh0Wjo27cvAwYMYNu2bQAEBQUREVFyRRgREUFQUJDR\nnp6eTkxMTAm7j48Pzs7OuLi44O3tTXh4uNEeExODVqstsdVXUyjBUQBw8LohWadGaBjYYmCtz2+h\nseD1vq/zUNuHAPjl0i+8vP9lCvQFte6LQgGwZs0a4+rg4sWLzJ8/n/HjxxsrX0opyc3NNVbEzMvL\nIzc3F53O8EUpMzOT0aNHk5eXx7Zt23BwcKhwzn79+vHdd99x7do1AA4fPswff/xBcHAwACEhIWRn\nZ7N06VLy8vLYsWMH69evZ/bs2QD4+voyfPhwFi5ciFarJTY2lsWLFzNnzhzjHLNnz2bx4sXExsai\n1WoJDQ1l5MiRNR4wAGByLer6/OjWrVu59bwbCv/c+08ZuCJQTt06tU790Ov18u2Db8vAFYEycEWg\nnP/HfJlXkFenPimqj5ycHJmTk1PXbpjEoEGDZOPGjaWdnZ308fGRzz33nExPTzfaY2NjJXDH4+uv\nv5ZSSrlixQoJSFtbW2lvb298zJkzxzjGnDlz5KhRo4zP09PT5Zw5c6SXl5d0cHCQfn5+8p133inh\n15EjR2SPHj2kjY2N9PX1lStXrixhv3nzpgwJCZEODg7Szc1NLliwQOp0OqO9oKBAzp8/X7q5uUkH\nBwcZEhIib926Ve77cLe/GXBMVuKzVshKLA/rK927d5fHjh2razfqjAJ9AUN+HELa7TTmd5vPE4FP\n1Kk/UkqWHF3CqnOGA219vfryweAPsLOyq1O/FH+dotVA0SpBce9zt7+ZEOK4lLK7qWOpLTUFkbci\nSbtt2Dqo7fs3ZVEUSDCns2Eb4MD1Azz525Ok5KbUsWcKheKvYJLgCCEshBBLhRC3hBAZQoh1Qogm\nd2k/SghxRgiRI4Q4LYS4r5TdXwgRJoTIEkJcFULML2W3E0J8JYRIFUKkCSH+K4SwLWa3F0IsE0Ik\nCCHShRCHhRB1/0lpphTVvmnt3JpWTq3q2BsDQgie7vo0L/Z8EYHgdPJppm2bxvXM63XtmkKhqCKm\nrnBeAO4HegEtCq+tLKuhEKI1sB54D3Au/LlBCOFTaLcANgHnAHdgPBAqhJhUbJgPgfaFj7ZAB+D9\nYva3gN5AN6AxsArYKIRobOLrURQiazBZZ3XwWIfHjBkI4rRxTNk6RZU2UCjMFFMFZzawWEoZI6VM\nBxYCo4pEpBTTgONSylVSyjwp5XdAeOF1gIFAK+BFKWW2lDIc+AKYC1C4knkceEVKeVNKmQi8AkwT\nQhRtIvoDm6WU16SUeuBLwAH483htBQgh3IQQbYUQbQsKGm4kVEx6DPEZ8YAhnc29yCjfUcYs04k5\niUzdNtVYQkGhUJgPFQqOEMIZ8AaOF12TUl4CtEBZgdtBxdsWEl54vcgeJaXMLMfeDrApNUY4YIth\ntQPwETBMCOEthLDEIFbRwOmKXk8xngEuABcSExMr0a1+UbS6cbNxo7N7zcfhV5W+Xn1ZMWoFHrYe\n5BTk8OzOZ/nmzDeVOhOhUCjqFlNWOEXHT9NLXU8rZiuOYwVtTbGXnq/o96I2kUAccBnIBV4GnpBS\n5pb3IsrgPxjErV1R2oiGSNH9m8EtB9/zJ/s7unXk+zHf08G1AxLJv479izcPvUm+3vTsvwqFou4w\n5RMmo/Bn6SOyLhhWOWW1v1tbU+yl5yv6vajNWsAaaIphNTQd2CKECCj3VZRCSpkspYySUkZZWjbM\nDD9JOUmcTDIk7LsX79+URVP7pqwYtYKhLYcCsDZqLXN/n6si2BQKM6BCwZFSpgHxQHDRtcLAACeg\nrPSikcXbFtK18HqRva0Qwr4c+wUMq5bgUvYcIKrweTdgmZQyUUpZIKXcBFwChlf0ehR/UnQfxNbS\nll6eve7e+B7CzsqOD4Z8wPSA6QAcSTjCpM2TOJ1UmR1VhUJR25i6h7IMQySZrxDCCVgM/CaljCuj\n7bdAdyHEo0IIKyHEoxgE4ptC+x4MW2HvCiFshRBdgDkYAgeQUuZgiDp7UwjhIYTwAN4Evi22ZbYf\nmCmEcBVCaIQQY4AAoJbTzpo3Rfdv+nr1rbVkndWFRmj43+7/y7v936WRRSMSshKYum0q66LW1bVr\nCoWiHEwVnEUYQpmPAtcACwyRZAghJgshjAEAhQEFEzDcV9EW/gwpEicppQ4YBwQCycBWYKmUck2x\n+f6BYTVT9LgAPFfMPh3IwxBanQYsAZ6WUu4x8fU0eLLzszl0vTBZp5lsp5XFOL9xrPrbKpo7NCdf\nn8/rB1/n9QOvc1t3u65dU5g5oaGhBAQE4OTkhJeXF7NmzSIl5c+t2xUrVqDRaEqUoH700UdLjJGV\nlcWzzz6Lp6cnDg4OdOjQ4Y7km+UxadKkMktX//rrrwQEBGBra0tgYCDbt28vYY+Ojmb48OHY29vT\nokUL/v3vf5ewZ2dnM2PGDBo3boyLiwtPPvkkOTm1VOa9Mnlw6uujIeZS+z3udxm4IlB2/qazTMlJ\nqWt3/jJpuWlyzu9zjDnYHtr4kIxLj6trtxSlMKdcai+++KIMDw+XeXl5MjExUY4aNUqOHz/eaP/6\n66+ln59fuf31er0cPny4HDdunIyPj5dSSnnp0iWZkJBQ4dzr1q2Tw4YNk4Dcu3ev8fqlS5ekra2t\nXLlypbx9+7ZctWqVtLOzk7GxsVJKQ5609u3by6efflpmZWXJ48ePS3d3d7lmzRrjGDNnzpR9+vSR\nCQkJ8ubNm7JPnz5y7ty55fqicqlVMw0xl9pL+15i46WNBHsE883obyruYAbo9Do+P/k5n0d+DoCd\npR0v936ZcX7j6tgzRRGl83Ll6/NJyEyoVR+aOTTDSlN+TZry2LJlC4899pixlsyKFSt4++23iY6O\nLrP9b7/9RkhICFevXsXV1dXkeZKTk+nRowdhYWH4+fmxd+9e+vfvD8Brr73Gzp072bt3r7H9gAED\nGD58OK+99hq7du1izJgxJCYmGrNTv/LKK+zbt49du3aRk5ODq6srmzdvZtiwYQDs2LGDcePGkZKS\nUma+tOrMpdYww7MaOAX6AvZcNew+DvUeWsfeVB8WGgue6vIUXd278s99/yQ5N5l/7vsnB64f4OXe\nL2NvZV/xIIpaJSEzgb9t+Futzrk1ZCstnUyr+lmcHTt23FEz5sqVKzRr1gwrKyv69evHe++9h6+v\nLwC7du3Cz8+PxYsX8/XXX+Po6MikSZN44403jEXYxo4di7e3d4nKn08//TTPPPMMrVvfWZguMjKy\nRGE4uLPEdNu2bUuUQggODuaTTz4B4MKFC+Tm5pYYIzg4mJycHKKiomq8Js69ffBCUSOcSDzxZ7JO\nM75/Ux59m/dl7fi19PPqB8DmmM08vOlhziSdqWPPFObKunXr+PLLL/nwww+N1wYOHMipU6e4fv06\nR48excbGhhEjRpCVlQVAUlISp0+fRqfTER8fz6+//spPP/3EkiVLjGNs3ry5hNj8/PPPxMTE8I9/\n/KNMP6paYrq4HUqWmC76XZWYVtQIRdFpfs5+eDt517E3NUMT2yZ8OvxTvj3zLR+Gf0h8RjyPb32c\nWZ1nMavzrCptqSiqn2YOzdgasrXW56wMP/30E3PmzGHjxo3GQmhAiRVIs2bN+PLLL3F2dubQoUMM\nGzYMR0dHLCwseOedd2jUqBFt2rThqaee4vvvv+ell166Y56UlBSeffZZtmzZYqwuWprySkhXVGK6\nuB0MJaZdXFyMvwO1UmJaCU4DQxZP1nkPlCKoSTRCwxOBT9C9WXdC94QSnxHPZ5Gf8ceVP3in/zu0\nadymrl1s8FhprKq0vVVbfP3118yfP59NmzbRr1+/u7YVQiCEMKZb6tKlS7ntyuLkyZNcv36dIUNK\n/r8cO3Ysc+fOZdGiRQQFBbFr164S9oiICOP9mKCgIKKiosjKysLe3t5oLypB3a5dO2xsbAgPD2fo\n0KFGu62tLW3btqXGqUyEQX19NKQotYspF42RXCcST9S1O7VGVl6WfPfQu8bX3vXbrnL5yeWyQFdQ\n1641KMwpSu3DDz+Urq6u8siRI2XaN2/eLK9cuSL1er1MTk6Ws2fPlt7e3jIjI0NKKaVWq5XNmjWT\noaGh8vbt2zImJka2bdtWLlq0qMzxcnNz5ZUrV0o8ALl+/XqZmpoqpZQyOjpa2trayu+//17m5eXJ\n77//vswotWeffVZmZ2fLiIgI6eHhIVevXm2cZ+bMmbJfv37y5s2b8ubNm7Jfv34lqpCWpjqj1Or8\nw/5eeDQkwVkWuUwGrgiUg38YLHV6XcUd6hmHrh+SI34aYRSex7Y8Ji+mXKxrtxoM5iQ4gLS0tCxR\nHtre3t5of/7556Wnp6e0s7OTzZo1kxMnTpQXLlwoMUZkZKTs37+/tLOzk97e3vK1116TBQV/fskZ\nNWrUXT/sKRUWLaWU27Ztkx07dpQ2NjayY8eO8rfffithv3jxohw6dKi0tbWVnp6ecunSpSXsWVlZ\ncvr06dLZ2Vk6OzvLGTNmyOzs7HJ9UGHR1UxDCouevGUyJ5NOMrHNRF7v+3pdu1MnZOZlsvTYUtZf\nXA+ApcaS6QHTmRM0h0YWjerYu/qNKjFtfqgS04oqcSv7ljFZZ30Kh64sDtYOvNH3DT4d9ime9p4U\n6Av48tSXTPhlAodvHK5r9xSKeosSnAbEH1f/AMwvWWdNMaDFAH6+/2emdpyKRmiIz4hn5vaZvLTv\nJVJzU+vaPYWi3qEEpwFRVPumn1c/tXVUiJ2VHQt6LGD1mNV0dOsIwMZLGxm7YSxrzq+hQN9wq8Eq\nFNWNEpwGQnZ+tnG76F4tJV2XdHTryHd/+44F3Rdga2mLNk/LO4ffYdLmSRxNOFrX7ikUdUZ13udX\ngtNA2H99P3n6PDRCw8AWA6s8zoWEDD74PYqkzPqXjdlSY8nUgKlsemATY1qPASAqNYoZv81gwe4F\nJGTVbs6v+ogQAr1eX9duKCqBTqcr9yBqZVGC00AoKrbW1aMrjW0aV2kMvV7y1PfhfLjjIo8vP0x6\nTv0s7dzUvimLBizi29Hf0sG1AwC/xv3K+J/H8+mJT8nOz65jD80XKysrbt+uf19W6itSSnJzc425\n3/4qKtNAA6BAX8Duq7uBv5Y7bW90EtGJhtJH5xMymPXNMb59sic2VhbV4ue9RlePrqwes5oN0Rv4\nKPwjUm92H3uyAAAgAElEQVSn8lnkZ/x44Uf+HvR3JrSdoFLkVBKNRoOlpSXp6elYW1uXe+peUbdI\nKdHpdOTm5uLi4lJtfye1wmkARCRGkH7bkC9paMuqh0N/vT+2xPMjcSk8szqCAl393SKx0FjwYNsH\n2RSyiakdp2KlsSI5N5m3D7/NhF8m8Pvl36t1j7sh4OjoiL29vRKbexghBNbW1ri5uWFpWX3rEnXw\nk/p/8HPJ0SWsPLsSfxd/Nty/oUpjRCdmMvx9wyrp48e6Ep+SzZJfLwAwqXtLFk3s1CA+QK5lXuPj\niI/ZErMFieH/Tmf3zjwX/Bzdm5l8/k2hqBeog5+KEkgpjeHQfyU67ZsDcQB4OdswKqAZfx/kx4x+\nhrofPxy7wmsbzzSIb/rNHZrz3oD3+GHsD/Tx7APAyVsnmf7bdGZun0lEomnlgxWKhogSnHpOdFo0\nVzOvAlW/f5Oenc/a44YxpvTxwdJCgxCCl8d0YGJwCwC+PXiZNzadbRCiA9DBrQPL7lvGFyO+MAYW\nHL5xmKnbpjJ7+2xOJJ6oYw8VinsPJTj1nKLoNHdbdwKbBFZpjB+OxZOTr8PGSsOjPf9MJa/RCJY8\n2JkJXZsDsOJAHG9ubjiiA9DXqy8/jP2B/xvyf7Rr3A6AgzcOMmXbFOaGzeXUrVN17KFCce+gBKee\nU1T7ZlDLQWhE5f/cBTo93xy4DMCE4Ba42FmXsFtoBEsfCuL+Ll4AfL2/4YmOEIJh3sP4cdyPfDD4\nA2Odnf3X9vPY1seYvX02h28cblDviUJRFkpw6jGJ2YmcSjJ8w67qdtrvZ29yLS0HgOl9fcpsY6ER\n/PuhIMYH/Sk6C9eerNfRa2WhERqGtxrO2nFr+degf+Hv4g8YVjwzt8/ksS2PEXY5DL1sWO+LQlGE\nEpx6TNF22l9J1vlVYSj0gDZNaNPUsdx2lhYa3n84iAe7Ge7p/HT8KvO+Cyc3X1elec0ZjdAw0mck\n68av44PBHxDoZtjKPJ18muf+eI77f76fDRc3kK+rnwdnFYryUIJTjynaTqtqss5TV9M5GmfImjyj\nv2+F7S0tNCyZ2JknC9tuP3uTGSuOknm7YSbALFrxfD/me5bft9wY1RanjePVA68yav0olp9aTlpu\nWh17qlDUDkpw6ilZ+VnGZJ1DvKu2nVZ00LN1E3sGtXE3qY9GY4hee/4+Q330A5eSmfTFQRLSc6vk\nQ31ACEEvz14su28Za8auYaTPSASCxOxEPgz/kBFrR/DGwTe4lHaprl1VKGoUJTj1lAPXD5Cvz8dC\nWDCweeWTdSZm5LLp5HUApvfzQaMx/VCnEIKnh7bhrfsDEALOXNfywCf7OXM9vdJ+1DcC3AL416B/\nsSlkE4+0ewRbS1tydbmsjVrLA788wOzts9lzdY+6z6OolyjBqacUHfbs6tEVFxuXSvdfdSiefJ3E\n0caSCYVnbSrLlD4+LJvSHVsrCxK0uTz0+UF2nLtZpbHqG62cWvFS75cIeyiM57s/j5e9IeDi4I2D\nPLXjKcb/PJ5VZ1cZUxIpFPUBJTj1kAJ9AXuu7QGqFp2Wm6/j+8OGUOhHe3pj36jquZRGdGzKT3P7\n0NSpEdl5OmZ9e4zle2NUiHAhTtZOTAuYxpYJW/hg8AcEewQDcFl7mcVHFzPsp2G8tO8lTiSeUO+Z\nwuxRglMPKZ6ssyqCsynyOkmZeWgETO3T6i/7E9jcmZ+f6kdHTyf0Et7eco5/rDlBdl7DDCYoC0uN\nJcNbDeeb0d/w49gfud/vfmwsbLitu83GSxuZsm0KEzdNZPX51WTkZdS1uwpFlVCCUw/ZGb8TAH8X\nf1o6taygdUmklHy9Pw6AkQHNaNHYrlp88nS25ae5fRgd2AyAjZHXCfnkAHFJWdUyfn2ig1sH3u7/\nNjse3sELPV8wnue5mHqRdw+/y7CfhvHq/lfVqkdhdpgkOEIICyHEUiHELSFEhhBinRCiyV3ajxJC\nnBFC5AghTgsh7itl9xdChAkhsoQQV4UQ80vZ7YQQXwkhUoUQaUKI/wohbEu1CS4cI6Ow3cbKvPD6\nipTSGA5dldXN4dgUzt7QAjC9X8Wh0JXBvpEln04O5oXR7dEIuHAzg3Ef7yPsrLqvUxZO1k5M7jCZ\n9ePX882obxjbeizWGmtyCnLYEL2BKdumMO7ncXx58ktVjVRhFpi6wnkBuB/oBRTdQV5ZVkMhRGtg\nPfAe4Fz4c4MQwqfQbgFsAs4B7sB4IFQIManYMB8C7QsfbYEOwPvF5mgP7ALWAs0AD+AtE19LvSY6\nLZprmdeAqglOUSh0YHMnevhUrTLo3RBCMHeQHyuf7IWrvTUZuQXM/PYYb20+y+2ChndI1BSEEAQ3\nDea9Ae+x46EdLOi+AD9nP8Bwr+ejiI+4b+19zNo+i80xm8kpyKljjxWKsjGpHo4Q4jLwppTyv4XP\n/YBowFdKGVeq7RvAUCnlgGLX9gJhUso3hBBDgC2Ah5Qys9D+FtBfSjmkcCWTAoyVUu4otA/DIFKu\nUspcIcTqQt8fqfILF8INcAMICgq6cOJE/cjuu+zkMv4T8R/cbd0JeyisUvnTrqRkM3DpLqSEfz8U\nxMRuVYtOM5VraTnM+y6cyCuGg48BXk589GhX/NwdanTe+oCUkrPJZ/k5+me2xm5Fm6c12uyt7Bnp\nM5KxrcfSrWm3KuXQUyhModrr4QghnAFv4HjRNSnlJUALdC6jS1DxtoWEF14vskcViU0Z9naATakx\nwgFbDKsdgCFAghBitxAiWQhxpPS2nQk8A1wALiQmJlay671L8do3lf2g+eZAHFJCE4dGjA3yrAn3\nStDcxZaf5vRhzqDWgOG8ztiP9vHj0Svq3kQFCCEIaBLAS71fYtfDu3h/8PsMajEIC2FBVn4W6y+u\nZ8ZvMxjx0wiWHF3C6aTT6j1V1DmmfCI5Ff4sfSAgrZitOI4VtDXFXnq+ot+L2jQBZgKvYNhS+w/w\nS+HKy1T+g0Hc2nl4eFSi271LYnYip5NPA5XfTsu8XcAPR68A8HhvbxpZWlS7f2VhbanhxdEdWPVk\nL9wdG5GTr2PhupP8fVU4SZm3a8UHc8fawpoRrUbw8bCPjed62jY2fDdLzElk5dmVPLrlUcZsGMNH\n4R8RnRpdxx4rGiqmCE5RDKZzqesuGFY5ZbW/W1tT7KXnK/q9eJufpZR7pJT5UsqVGFYrI+/yOkog\npUyWUkZJKaOqs2Z3XVI8WWdPz56V6rvu+FUybhdgbaFhcq+/HgpdWfq3acKv/xjA0PYG8f/1TAIj\n3t/Npsjr6pt5JWhi24RpAdNYN34dP9//M3M6z8Hb0RuAKxlX+PLUl4RsDGHCxgl8efJL4tLj6tZh\nRYOiQsGRUqYB8UBw0bXCwAAn4GQZXSKLty2ka+H1IntbIYR9OfYLQG6pMboCOUBU4fMTQFmfQg36\nk2nnFUM4dP/m/SuVrFOvl6woLCE9LsgLd8fKJ/qsDtwcGvHfad15N6QT9tYWpGbn88zqCOZ9p1Y7\nVcHPxY+nuz7N5pDNrBmzhqkdp+JhZxD0i6kX+SjiI8b9PI6QX0L4OOJjzqecV+KuqFFMDRp4CZgK\njAKSgf8CjlLKUWW09QNOAU9iiCJ7EFgOBEgp4wqj1E4D2zFEv7UDfgP+IaVcUzjGlxgi0yYUDrse\nOC2lnFtofwj4GrgPOARMKpwjUEoZW9k3oXv37vLYsWOV7XZPkZWfxYA1A8jX5/Nu/3cZ5zfO5L47\nz99kxgrD69/8TH8Cm5degNY+V1OzeWHdKfZFJwHQ2M6KV8Z2JKRrc4QwPa+boiR6qSf8ZjjbYrcR\nFh9GSm5KCXsLhxYMbzWcYd7D6OzeWQUcKO5KZYMGTBUcC2Ax8ATQCPgdmC2lTBJCTAa+kFI6FGs/\nCvg30BqIAZ6TUm4vZvcHvgD6YLh/876U8l/F7HbAx/wpOOuAp6WUOcXaPA0sAFwxhFiHSil3mfrC\ni1MfBGd73Hbm756PhbBg96TdODcyXTQeX36YfdFJ9PR15cc5fWrQy8ohpWT1kSu8s+UsWXmGkOne\nrV15+4FA/D3Kr82jMA2dXkdEYgQ74ncQFh92x1ked1t3hnoPZWjLoXRv1h1rC+tyRlI0VGpEcOo7\n9UFwXtz7IptjNtOjWQ++GvmVyf2ibmZw3weGvGufP96NUYWZAO4lrqXl8NovZwgrTPxpZSGYNaA1\nzwxtg6117QQ31HeklJxJPkPY5TDC4sO4rL1cwm5raUtfr74MajGIAS0G0MS23HPfigaEEpwqYO6C\nk6/PZ/APg9HmaVnYYyFTOk4xue+L60+y+sgVWjS2ZfeCIVhUogxBbbP9TAKvbzzD9cLaOi0a2/Ly\nmI6MDGiqttmqESkll9IuERYfxs74nZxLOXdHm0C3QAa2HMigFoPo4NpBvf8NFCU4VcDcBefIjSM8\nuf1JALZO2EpLR9Pyp6Vm5dH7vR3cLtDz8pgOzBzQuibdrBay8wr4cMdF/rs3lgK94d9uL19XXhnb\n8Z6491QfSchKYO+1vey5sodDNw6RqytZTM/D1oMBLQbQv3l/enr2xMm6rNMSivqIEpwqYO6Cs/jI\nYladW4W/iz8b7t9gcr9PdkWz9LcL2FlbcPDFYTjbWtWgl9XLhYQM3tx8hv3RyQAIARODW7BgZDua\nOtnUsXf1l9yCXI4kHGHP1T3svrr7jvs+GqGhU5NO9PPqRx+vPgQ2CcRSUz+OHSjuRAlOFTBnwZFS\nMnr9aK5lXmNWp1k8G/ysSf3ydXoGLN5FgjaXaX1a8cb9gTXsafUjpWTHuUTe3XqOmMKs07ZWFswe\n2JqZA3xxtDEfATVHpJREpUax5+oe9l7by8lbJ9HJkvnwHK0d6e3Zmz5efejn1Q8vB6868lZREyjB\nqQLmLDhRqVFM3DgRgNVjVhPYxDTh2Bh5nWdXRwCwc/4gWptx/rJ8nZ5Vhy7zf2EXSc/JBwxh1H8f\n7MfUPj7YWKnAgtogIy+DIwlHOHDtAPuv7zcmkS2Oj5MPvTx70bNZT3o060Fjm+pPEKuoPZTgVAFz\nFpwvIr/g4xMf42Hrwe8P/W7yuYmQT/cTEZ/G0PYefPVEjxr2snZIy87jk13RfHPwMnkFegA8HBvx\nzFB/JvXwxtpSnSmpTa5or7D/+n4OXD/AkYQjZOXfWfuoTeM2RvHp3rR7pcL5FXWPEpwqYM6C88jm\nRziTfIaH2z7MK31eMalPRHwqIZ8eAGDlkz0Z0Ma9Jl2sdRLSc/nPzov8cPSKMbCgRWNbnh7iz4Tg\nFkp46oB8fT4nb500iM+NI5xOOk2BLFnxVSBo59qOHs160LNZT7o17YajtTpvdS+jBKcKmKvg3My6\nyfC1wwH4bPhn9G/e36R+z66OYGPkddp4OLD9uYH1NqT1cnIWH4ZdZMOJaxT9M/d0tmHWgNY80rMl\ndtbqZnZdkZ2fTURiBEcTjnI04Shnks/ccf9HIzS0d21PsEcwXT260tWjK+529evLkbmjBKcKmKvg\n/HD+B94+/DZ2lnbsfWSvSSfBE9Jz6b94JwV6ybshnXisl3cteFq3RN3M4KMdF9ly6oZReFztrXmy\nvy+P925lVtF59ZXMvEzCE8M5mnCUIwlHOJd8DllGasSWji3p6tHVIEJNu+Lr5FtvvzCZA0pwqoC5\nCs7csLnsv7afEa1G8P7g9yvuACz97Tyf7LqEi50VB18Y1qBO6sfcyuTz3ZdYH37NuNXm2MiSx3p7\nM62PD14uthWMoKgttHlajiccJzwxnPDEcM4mn6VAX3BHO5dGLnTx6GJcBXV066hS8NQiSnCqgDkK\nTlWSdebm6+jz3g5Ss/P5+2A/Qke1rwVP7z2upeXw5Z4YVh+J53ZhcIGFRvC3Tp7M6OdDV28VOXWv\nkVuQy+mk00QkRhCeGM6JxBNk5mfe0c5KY0UHtw50btKZTk060dm9M80dVMLXmkIJThUwR8H5Le43\nnt/9fKWSda45Es8L609hoRHsCx2Cp3PD/kaflHmbFfvj+O7wZVKz843Xg71deLJ/a0YGNMXSQgUY\n3Ivo9Dqi06KNAhSRGHHHIdQiXG1c6dSkk1GAApsEqmCEakIJThUwR8F5Ye8LbInZQs9mPfnvyP9W\n2F5Kycj/20PUzUzGdvbk48dKlyxquOTm69gQcY2v9sVyMfHPb83NXWx5pEdLJvVoiYfKXnDPcyPz\nBieTTnLy1klOJZ3ibPJZbuvurKMkEPg6+9LZ3bAKCmgSQBuXNmorrgoowakC5iY4+fp8Bv0wiIy8\nDEJ7hPJ4x8cr7LM/OonJyw8DsH5eX4LVttEdSCnZezGJ/+6LZXfULeP1nhZRzHI/Q9Nu4wnsOwaN\nRcO572XO5OvziUqN4tStU5xKOsXJWyeJ08aV2dZSY0kblzZ0dOtIR7eOBLgF0KaxEqGKUIJTBcxN\ncA7fOMzM7TMB2DZhGy0cW1TY58kVR9lxPpGgli788lS/mnbR7IlOzGDVwcs0iljOAvkNlsJwr+eq\naMY1n4m0GTkH12a1X4pb8ddIv53O6aTTxpXQ6aTTpN1OK7OtpbDEv7G/QYRcDULU1rVtparp1ncq\nKzjqIIIZ8seVPwDDKW1TxCY2KYudFxIBmNHPpwY9qz/4u1rzOp8Bq0BAtrDFTubQQibQIvYTdJ99\nSqR9L3RdphA46CGsG6kPIXPAuZEz/Zr3o19zw5cuKSU3sm5wNvlsiUfq7VQKZAHnU85zPuU861kP\nGETIz8WPDm4daNe4He1c29G2cVuVIcFElOCYGVJKdl0xFDYd0nKISX2+ORCHlNDUqRF/6+RZk+7V\nDzIS4IfH4epRw/N2Y7Cb8AWXLpzk1u7ldEz6FSeRTVD2IThwiKQD/yTKfRTu/abg37kfQqMCDcwF\nIQReDl54OXgxvJXhELWUkoSsBM4mn+VM8hnOppzlXPI5UnJTKJAFXEi9wIXUCyXGaWbfjHaNDeJT\nJELejt5YaNT2a3HUlhrmtaV2IeUCD256EIA1Y9YQ0CTgru21ufn0eXcHWXk6Foxsx1ND/GvDTfPl\n6jGD2GTcMDwfFAqDXoBiIpKVmcHpHd/hcOY7AvJOluger2lBQqvxeA+eRrNWDTPsvD4ipeRm9k2D\nACWf5UKKQXTKi4wDQ5VUfxd/owgVCZKDtfkmyi2NuodTBcxJcD6P/JxPTnyCh50HYQ+GVXi+YPne\nGN7eco5GlhoOvjgMV3t1E7RcIr6Dzf8DujywsoeQz6Hj+Lt2uRF7his7v6L51U00lzdL2M5bdSTV\n7378h0zBvWnzmvRcUUek304nKjWKCykXDD9TLxCdGk2ePq/cPp72nvi5+OHv4m98+Dr7YmdlV4ue\nVw9KcKqAOQnOpM2TOJt8lkntJvFy75fv2lanlwz+1y6upOTwSI+WLJrYuZa8NDN0BbD9ZTj8meG5\nSyt4dDU0vfvqsThSr+f8sZ2kHf6Odsm/40qG0VYgNZxtFESO/1jaDJqEa1PTKrIqzJMCfQGXtZeN\nq6ALqReISoniVs6tcvsIDFt7/i7+RjHyc/GjtXNrbCzv3ZB8JThVwFwEJyErgRFrRwCmJev87UwC\nc1YeB+DX/xlA+2aq9O8dZKfAT09A7G7Dc99B8NAKsHOt8pB5t29zdt8GdJE/0jF9L7biz2+7OimI\nsulERusx+A54BHcvn7/kvsJ8SMlNISo1iktpl4hOizb8TI0mIz+j3D4CQUvHlkYRau3SGl9nX3yd\n7o0VkRKcKmAuglOUrNPeyp49k/ZUeEbgkWUHORSTQj9/N76b2buWvDQjbp6B1Y9C2mXD897zYMRb\nYFF9sTS5Wemc37MO/ZmfaZ9xCDtR8iDiWcuOpLYaSYveE2nVplO1zaswD6SUJGYn/ilC6X+KUVn1\ng4rjYeeBr7MvPk4+BhEqFKKm9k1Nrov1V1Fh0fWYoui0fl79KhSbM9fTORSTAsD0vr417pvZcfYX\n2PB3yM8Ci0Yw7v+gy2PVPo2NvTNdRs+A0TPIztJyZM8GxNlf6KA9gIPIoWPBWbh0Fi59wGVNCxKa\nDqJx1/H4Bw9DY6myWNd3hBA0tW9KU/um9G3e13i9KFLOuBIq/BmTHkN2QTYAidmJJGYncvjG4RJj\n2lra0sqpFb5OBhHycTYIUiunVtha1m06K7XCwTxWOJl5mQz4YQAF+gLeG/AeY1uPvWv7BT9F8tPx\nq7Rys2PX/MFoNCp5IQB6PfzxHuxZYnju6AmTvoMW3WrVjdycLKIObCL/9Ab8UvfjQsltlTQcuOTU\nB9l2JP59HsDFTdWBUfy5IorTxhGbHktseqzx9xtZNyrs72nvSSunVnc8Wji0qFIIt9pSqwLmIDi/\nxv3Kgt0LTErWmZR5m77v7SRPp+e1cR2Z3k+tcADI1cKGOXBhq+F5ix4waRU4NqtTt3QFBVwM30Fq\nxEY8E/7AR14tYc+XFlyw7ojWawDuXf+GX6e+Kr2O4g6y87OJz4g3iFB6oSBpY7msvUxOQc5d++56\neBdNbJtUek61pVZP2RVv2E4zpe7794fjydPpcWxkyUPdVUQUAMmXDPdrkgoP7HV9HMa8D5Z1nyHA\nwtKS9j1HQs+RSCmJu3iaG0d/xjE+jHa5p7ASOgLzT8HlU3D5U1J/diLWqTv4DcO7+xiaNFdfKBRg\nZ2VHe9f2tHctef5LL/XczLpJrNawIrqsvUy8Np44bRw3sm5ga2mLm41brfioBMcMyNfns/faXgAG\ntxx817Z5BXpWHjLcBH+oe0scGqk/MdFhsHYG5KaDsIBRi6DnLLgHa6QIIfBp2wmftp2AV8hMT+Hs\nwZ/Ji9qBd8pBmpJMY7Q01u6EiJ0Q8RKxmlbcaNKHRu2G0abHCJycVGJWxZ9ohAZPB088HTzp69W3\nhC1Pl0didmKt1QtSn0ZmQPjNcDLyDHv8Q7zvns5my6nr3Mq4jRDwRF+fWvDuHkZKOPAfCHsNpB5s\nXeHhb8B3YF17ZjIOzq4EjZoBo2Yg9Xriok5w/fgW7K7spl1OJLYiD1/9ZXwTL0PiGvL3WHDeqi1p\nTXvi2G4Ift2GYWOvwuEVZWNtYW1SPsbqQgmOGVAUnda2cVuaO5R/Yl1KyVf74gAY3qEp3m51H6df\nZ+TnwMZn4dSPhudNA+GR76Gx+WZ4FhoNPu2D8WkfDLxCbk42p4+HkXVmO+6J+2mti8FK6GhfcA6u\nnYNr35C/w4Lz1u1I8+iJXbshtA4eioODEiBF3aAE5x5HSmm8f1NRss7jl1M5dS0dgBkNOVAg/Sqs\nmQw3Thied3wAHvgUrO3r1q9qxsbWjsD+46G/If2ONjmBmGPbyY3eTdPko/jqLxsEKP8sXDsL11aQ\nt8OCs1ZtSXMLxsavLz5dhuDqodLuKGoHkwRHCGEBLAKeAGyA7cAcKWVSOe1HAf8GWgOXgP+VUm4v\nZvcHPgf6AKnAB1LKfxez2wEfAyGAANYBT0sp7wi1EEL8ADwMDJBS7jPl9ZgTUalRXM+6DlS8nfbV\n/lgA2jdzpHfrqp+UN2suH4Qfp0DWLUDA0JdhwPx78n5NdePk1owuI6fCyKkAJCZcJe747+hj99As\n5Rg++nishY6OBefg5jm4+R0cgCua5tx0DkK07IVHwCBatAlSGa8VNYKpK5wXgPuBXkAy8BWwEhhd\nuqEQojWwHpgN/Ag8BGwQQgRIKeMKxWsTEAaMB9oDvwohrkopfygc5sPC6+0BCfwMvA/8vdRcE4Da\nCa+oI3Ze2QkYThV3dO1Ybrurqdn8etqQuXZGf99auwl4T3HsK9i6EPT5YO0IE5dDu1F17VWd4dGs\nBR5jpgPTAUhLvMbl8O3kxeyncUoEPvkxWAo9LfXXaJl6DVK3wknDGaA420CyPLrj4N8H3859cXJu\noF9gFNWKSedwhBCXgTellP8tfO4HRAO+Usq4Um3fAIZKKQcUu7YXCJNSviGEGAJsATyklJmF9reA\n/lLKIUIIWyAFGCul3FFoH4ZBpFyllLmF19yAo8BwDKuoSq1wCvu7AQQFBV04ceKEqV1rlYc3Pcy5\nlHMVJut8b+s5vtgTg5u9NftfGIqNVQM6p1GQB7+GGgQHwM0fHlkN7m3r1q97nKyMNGJO7EEbtQ/7\nxGP45Z7FUdx5XkMvBZctWpLkFAjNu9GkfW+82/fAwqruQ8oVdUu1n8MRQjgD3sDxomtSyktCCC3Q\nGYgr1SWoeNtCwguvF9mjisSmmP2pwt/bYdi2O17Kbgu0BYoKkHwM/EdKGVPFb/PPAK8BJCYmVqV/\njZOQlcC5lHPA3e/fZOcVsPpIPACTe3k3LLHJvAU/ToX4A4bn/iMMKxtbl7r1ywywd3Sh04DxMMBw\nD6ggP59L58NJPrcbi2tH8dRG4iVvohESX308vmnxkLYVzsBtaUW0tR+pLp3QtOiGe/u+ePsHYqEO\npCrugilbakUhLemlrqcVsxXHsZy2ARXYnYrZS89X9LsTgBDiAQz3hyZX4Pvd+A/wPYCHh8eFCtrW\nCUWlpO2t7OnRrEe57daFX0ObW4CVheDx3uYbhVVprp8wBAdoC0/m9/sfGPYqqCqLVcLSygq/Tr3w\n69TLeE2bdIPLp/aRFXMYm1uReOeew5UMGol82uWfh1vn4dZPEAFaaUe8tT8ZjTtg6dUF97Y9aNm2\nCxYqJ5yiEFMEpyjJU+nj7S6Atpz2d2trir1ovrRSc2uFEK7AR8AYKaXeBP/LREqZjOF+FN27m7wi\nrFWKwqH7N+9fbrJOvV6yojBYYGxnLzyc7t3aGdXKqbXwy1NQkAuWtnD/x9Dpwbr2qt7h1MSTTkMe\ngiEPAaDX6bkcd57EcwcouHIM59RT+Ny+iJ24jZPIJjD/JCSehMQf4ATkSitirHxJd+6AbNYZ59bd\naNm+O7b2jhXMrKiPVCg4Uso0IUQ8EAycAGNggBN/bm8VJxIovf/TFdhRzN5WCGEvpcwqZo8s/P0C\nkFs4385i9hwgCugNeAG7Sm2lbRZCfC6lfKGi12QOZORlcCThCHD37bQ9F29x6ZbhbZzez6c2XKtb\n9BnnntoAACAASURBVDrY8Qbs/9Dw3LklPPIdeAbdvZ+iWtBYaGjl15FWfh2BmQDoC/KJj4ogKeoQ\nBdcjcUo9S8u8GOxFLjYinzYFUZAcBcm/wBnQbRTEWbQg0b4deU06Yt+iE83adqNZc18VHVfPMTVK\nbRkQKoTYhWFVsBj4rXTAQCHfAguEEI8Ca4EHgW7A1EL7HuAy8K4Q4gUM92zmAP8AkFLmCCFW/X97\ndx5eVXUvfPz7y0nIPJB5ICRATACLgAxOqFSv1F6vto5IQUVb7b0d3j52UHuvPrfXjl5tn9e27622\ndQbFAbXWageuoqDWMjiCJAgESMhI5vnknPX+sfYJJ4cAJyE5J8Pv8zznydl77b32Ois5+5e199pr\nAXeLyMfOPncDjxtjukTkHaAw4JgHsV1xXg/y84x6b1W+Ra+3l0iJPO5Ea4+8VQ7AwoLJnDZlnN+3\n6GyE9V+xQ9UATD0brnkcEnQk5XCKiIxi6uzFTJ29uG+d1+Nh/56PqS3bQk/F+yQ27mRK925SacEl\nhkLvQQpbD0LrBtgHbIIW4qmMKqQ1uZiIrNkkFswlt/h0ElP09zteBBtwfgZMxvYKiwb+BqwCEJGV\nwIPGmATo61BwBfY5nIeBvcDlvuBkjPGIyKXAg9jg1QTca4xZ53e8b2E7BZQ5y+uBW539u4F+w+k6\nLZ06Y0wT44SvO/SCrAXHHKzz09o23iiz09aO+xGh60rt4JsNe+zywi/bMdEijz8vkAqPCJeLguK5\nFBQfaXkar5faqv1Ul75L54H3iKnfQVrHHnI9VUSIIYl2ktw7oH4H1L8AO4BXoJZUqmJm0J58Cq7s\nU0mcOpcpp8whKWmc/4M1Dun0BIy+6QncXjfnrzufVncrdyy+g5WzBu4bceeLH7Hm7wfITY7hzds+\nS6RrnF6OKH0V1t8MPa0QEQWX3AcLVoe7VGqYdHW0UlH2Hg37PsBTtYP45lKyu/eRSeNx96sinbqY\nQjqSpuPKLCEp/1RyZswlKS1nQjzoOxro9ATjwLaabX3znB/r/k1zh5v12yoBuP7swvEZbIyBTffB\naz8GDMRnwvInYKpOlz2exMQlUjTvPJh3ZFBVYwz1tVVU795Oe8VHSO1Oklt3k+cuJwH7rFAO9eR0\n1UPXVqgFnAvwTSRQHTWV1oRp9KYVE5M9k7TCOeQUFBMVpT3mwkkDzijkGzutZHIJuQm5A26zbssB\nOt0eYqNcXLtoHM55090Gf/ianQoaIGee7RyQHLqRbVX4iAjpWbmkZ+UCfrPbGkND1T6q93xIW+UO\nqCsjvnUvWT0HSHc6tabQRop7JzTuhMY/2UfUN9secxWuHBpj8ulOKsSVXkRCbgmZhbNJyy7QDgsh\noAFnlDHG9HWHPtbYab0eL4+9XQ7AFafnkRI3zu5jNJbb52tqnH9ZT1sOl94PUeGdj12NAiKk5k4n\nNXc68MV+SQ31NRza8yFtFTsxdaXENu8lvaucHG81LjHEiNs+wNpxADregmr6WkUdJpqayFyaY/Pp\nSS5E0maQkFNCesFs0rPzNRgNEw04o0xpY2nf3OTHupz21501HGruAsZhV+i9b8Czq6GzASQCLrob\nzvqGXpNXJ5SankVq+kVwxkX91ru7Ozi0byeN+z+mq6aMiMa9JLTtJ919qK9VFCfdTPPsg7Z90PYm\nVNL30Ee7iaHKlUNTTD49SVOR1ELiMqeTOqWErPwiJkVPkGffhoEGnFHGdzktKy6LWamzBtzm4c32\nQc/zijMoyhwnD9AZA+8+CH/5dzAeiEmGqx6BogvDXTI1xkVFx5E/cyH5M4++t93S3ED13h00Ve6i\np2Y3Uc3lJHccILO3klTnWfR46aLIuw869kEHtmW00+7vMUJ1RDoNUTl0xE/Bm1xAZFohCdlFpE0p\nJjVriraO/GjAGWV8l9OW5i8dcMTnDyua2Lrf9t65aby0bnq74eVvw/tr7HLGTDtZWtqM8JZLjXtJ\nyakkzT8X5p97VFpLUz215TtprSzFXbebyKZy4joqSOupIoMGAFxiyDZ1ZPfUQc+HdrKV8iN5dJpJ\n1LgyaZqUR2f8FLwpU4lKLSAxaxrpU2aQlpFHxHjs8HMMGnBGEf/BOi/Iv2DAbXwPek7PiOe8U8bB\nA3EtVfD0Kqh0uqWXXAJXPAjR46TlpsaspJR0kgJ6z/l0d7VTvX83jZVldNbuxTSUM6ntICldlWR5\nqvtG3Y6VHgq9FdBVAV3v2icP9/jlY6KojcigeVIWnXG5mKQpRKXmE5s5jeSsaaTnTSMqevzM3KsB\nZxTxtW4SohIGHKyztqWLlz+0k7HdeM40IiLG+H2Niq22c0CbnceH82+H8+8AvQShRrnomHgKSuZR\nUDLvqDTj9dLYUEd9RRmtVZ/irt9LRPMBYtsOktJTRYanjmhx23zETb45RH73Ieh+z7aQ9vfP7zAp\nHI7MpDU6m574XExyPlFpU0nImMrk7GmkZ+URGTk2TuVjo5QThO/+zZK8JUS5jn5eYM3f9+P2GJJi\nIrny9DE+LfB7a+DlW8HTA1HxcPkDMPuycJdKqZMmERFMTs9icnoWzDv6Up3xemk+XEVd5R5aq/fS\nffgAprmC6LZKErqrSeutJU2OjIucRhNpvU3QWwbt2GeOdh/Jz21cVMlkmqIy6IzOpCc+B0nMYVJq\nPvHp+UzOLmRy9lQiJ4W/c4MGnFGitaeVLTVbgIF7p3W5Pax91855s2LxVOImjdFfnacX/nonvPsb\nu5xSACuegqxTj7+fUuOERESQnJFHckYecPTlOoDW1mbqK/fR4gQkT9NBoloriO2sIqWnhgxvPZOk\nF4Ao8diHYN314P4E2oCao/NsIJlGVzqt0Zn0xGbhTcwlIjmX2LR8ihcvIzpm5C/djdGz1vizuXLz\nkcE6pxw9WOdLHxzicHsPEQLXnTVG57zpaIBnb4B9b9rlaefD1Y9CnE5frJS/xMRkEmfOg5lHX7ID\nMF4PLYerOVxVTmvdAboOH8TbXElEWzVxXdUk9tST7q0nXrr69kmlmVRPM3Tssb3tDh/Jr+W0vRpw\nJhLf/ZsF2QtImtR/XjtjTF9ngYs/k82UyWPwJmL1x7DuS9DkXKA+8+v2GRuX/gkqNVgS4SIpI4+k\njDzgnAG3McbQ0NhAQ9Ve2uoO9gUlV1s10Z3VJPbUkeqpJ8q4SUyaHJJy67d9FHB73Wyu2AwMfDnt\n73sb+KTKXtMdk6NC73gRXvw3cHeAK9qOGjBvRbhLpdS4JiKkpqaRmpoGHHvGYHdPV8ieFdKAMwps\nrd563ME6H3Fm9JyTl8zCgtD8JzIsvF7Y+BN48167nJgDy9fClAXhLZdSqk9UCDsTaMAZBXyX02am\nzjxqsM4Dhzv42yf2DuCN5xQO+DDoqNTVAi98FUpfsctTFsHyNZCYHd5yKaXCRgNOmPUbrHOA1s1j\n75RjDGQkRnPJaTkhLt0QHd5jJ0urL7XL86+DS34OkdHhLZdSKqw04ITZroZdVLfbBx8DA05bdy/P\nbDkIwKozCoiOdIW8fIO2ewOsvwm6mkFcdlbOxTfr4JtKKQ044bbx4EYAsuOzmZk6s1/ac1sP0trd\nyyRXBCvPnBqG0g2CMfD2L2HDD8B4ITYVrnkcph394JtSamLSgBNmfYN1Tuk/WKfXa3jUmfPmsnm5\npCeM4stR7k546Zvw0bN2OeszdvDNyWP0eSGl1IjQgBNGVW1VfYN1Bk629nppLeWHO4BRPudN00F4\neiVUfWCXZ38Rvvg/MCk+vOVSSo06GnDCqN9gnVn9+8k/7HSFPmNaKqfmJoe8bEHZ/zY8cz201wEC\nF94FS76t92uUUgPSgBNGvoBzbt65/QbrLK1u5a1P7bgTNy0ZpQ96bnkIXr0NvL0wKRGu/D2UXBzu\nUimlRjENOGHS0tPC1mo7B0zg5TTfg575qbH806yskJftuHp7bKDZ9ohdTiuCa5+CjOLwlkspNepp\nwAmTtyrfotfYwTrPyTsyFlJDew8vvFcJwA1nFeIaTXPetNXaS2gH3rHLRRfZlk1sSnjLpZQaEzTg\nhIlv7puF2Qv7Ddb51D8O0N3rJX6Si2sW5YereEc79J6dLK3FBkOW3AoX3AURY+DZIKXUqKABJwzc\nHjebKjcB/R/2dHu8PP5OOQBXL8wnKeboSdjC4sNn4aVvQG8XRMbCF34Nc64Kd6mUUmOMBpww2FKz\nhTZ3G9A/4LzyURU1Ld2IwA1nF4apdH68Hvsg59u/tMvJ+XDtWsiZG9ZiKaXGJg04YeC7nDYrdRY5\nCUfGR/PNeXNBSSbT0sP8HEtnI6z/Cny6wS4XnANXPwYJGeEtl1JqzNKAE2LGGDZWbAT6t262H2jk\n/YNNwCiY86Z2F6xbAQ177fKir9gx0Vyj5BKfUmpM0oATYv6DdS7NX9q33te6Kc5K4JyitDCUzLHr\nFXj+FuhphYgouOQ+WLA6fOVRSo0bQU3zJiIuEblXROpEpFVE1otI+nG2v1hEdohIp4h8LCLLAtKL\nRGSDiLSLSIWIfCcgPU5EHhaRRhFpEpGHRCTWL/07IrJdRJpFpEZEnhGRUT66peV72DMnPqdvsM6q\n5k5e+agKsK2bsMx5Ywy8ca9t2fS0QnwmrH5Zg41SatgEO6/oHcAXgDOAKc66JwbaUESmA88DPwWS\nnZ8viEihk+4C/gh8AmQAlwG3i8hyv2zuB2Y6r2JgFvALv/RJwDeBLKAIaAdeDvKzhFXfYJ35Rwbr\nfOKd/Xi8hslxUVw+Py/0hepus8/XvP4ju5w7H27ZCFPPDH1ZlFLjVrAB5xbgHmPMXmNMM3AbcLEv\niAS4AdhmjFljjOkxxqwFtjvrAc4DCoDvG2M6jDHbgQeBfwVwWjKrgLuMMTXGmFrgLuAGEYkBMMb8\n1BjzljGmyxjTCtwHzBGRoOdfFpE0ESkWkeLe3t5gdzsph9oOsathF3Dk/k1nj4cn/3EAgBWLpxIT\nFeLnWhrL4aFl8MlLdvm05XDjq5AchsCnlBrXThhwRCQZmAps860zxuwBWoDTBthlrv+2ju3Oel96\nmTGm7RjpJUBMQB7bgVhsa2cgFwIVxpjGE30eP98ESoHS2traQew2dL7WTWJUIguzFwLw4vuVNHW4\niYwQrjsrxMP5790Iv10KtTtAImDZj+HyByEq9kR7KqXUoAXTwvE9Bt8csL7JL81f4gm2DSY98Hi+\n90cdT0TOBn6M00IahF9hg1tJZmbmIHcdGl/AWTJlCVERURhj+sZN+/ycHHKSQ3SiNwb+/ht44grb\n/TkmGVY+B2d/Q0d6VkqNmGACTqvzM3CM/BRsK2eg7Y+3bTDpgcfzve93PBE5F3vv5hZjzJ+OUf4B\nGWMOG2PKjDFlkZEj31mvpaeFbdW20ea7nPbWp4cpq7ENvZtCNeeNuwv+8HX48x1gPJAxE25+HYou\nDM3xlVIT1gkDjjGmCTgAnO5b53QMSAI+HGCXD/y3dcx31vvSi0Uk/hjppUBXQB7zgU6gzK8Mn8N2\nPviKMeapE32OcNtcsdkO1hkRyZK8JcCROW/m5acwf2rQt5+GrqUKHr0E3l9rl2f+C3xlA6TNGPlj\nK6UmvGA7DfwW25NsmogkAfcAfzHGlA+w7ePAQhFZISJRIrICWAA85qS/CewHfiIisSIyD/gqtuMA\nxphOYA1wt4hkikgmcDfwuDGmC0BErgSeBVYZY54f/McOPd/ltEVZi0iclMi++nZe22XvHYVkzpuD\nW+z9mko7JQLn3wHXPAHRicfdTSmlhkuwAedn2NbEFqAScGF7kiEiK0WkrwOA06HgCuBO7CWwO4HL\nfcHJGOMBLgU+AxwGXgHuNcas8zvet7CtGd+rFLjVL/0+IA5YJyJtfq9R+SyO2+Nmc+Vm4MjcN486\nrZvspBg+/5nskS3Ae2vg0X+GtmqIiofla+Cz34eIYH/9Sil18oK6eeEEie86r8C0tcDagHV/Bv58\nnPw+xfYsO1Z6B3CT8xoofZROgzmwLdX9B+ts7nTz7LYKAK47q4Ao1wid+D1u+Oud8O4DdnlyIVz7\nJGSdOjLHU0qp49ChbULgtYOvAXawzuz4bH6/aS8dPR6iIyP40uIRapS1H4Znb4ByOw0C05fCVY9A\nXOrIHE8ppU5AA84IM8aw8eBGwLZuPF7Do2+XA3DF6XlMjp80/Aet/tgOUdNkHyjlzK/DRXeDS3/d\nSqnw0TPQCPuk4RNqOmoAe//mbztrqGjsBGD12SNwZXDHC/Di18DdAa5ouPR+mLdi+I+jlFKDpHeN\nR5ivd1pufC4lk0v6HvRcUpROSfYw9hDzeuG1H8Gzq22wScyxQ9RosFFKjRLawhlhvsnWluYvZWdV\nC+/uawDgxuF80LOrxU4pUPaqXZ6yGJY/AYkj3PtNKaUGQQPOCKpsq6S0sRSwl9Me2VwOQGFaHJ8t\nGabhdOo/hXVfgnp7HOZfB5f8HCKjhyd/pZQaJhpwRpCvs0BiVCIFcXN46f03AVh9diEREcMwZtnu\nDfDcTdDdDOKCz99jZ+fU8dCUUqOQBpwR5D9Y5zNbDtHj8ZIYHclVC/NPLmNj4K37YcMPAANxaXD1\nYzDt3JMus1JKjRQNOCPEf7DOc3OX8l/r9gNwzaJ8EqJPotp7OuClb8LHz9nlrDlw7VqYHOKpDZRS\napA04IyQTRWb+gbrbG2cQX3bp0SIvZw2ZE0H7f2aamfM1FMvhy/8P5gUf/z9lFJqFNCAM0J8l9MW\nZy3myb/bQTr/aVYW+alxQ8tw/9vw9HXQUQ8IXHgXLPm23q9RSo0ZGnBGQI+np2+wzunxi/lLpZ3G\nZ8ijQm95CF69Dby9EJ0EV/4eij83XMVVSqmQ0IAzArZUb6Hd3Q5A6d6pQA+zcpI4Y9ogxzHr7bGB\nZtsjdjmtCK59CjKONdO2UkqNXhpwRoDvclpRcgkb3+0B7IyeMpjLX2218Mz1cOAdu3zKMrjidxCb\nMtzFVUqpkNCAM8z8B+uMdc/FayAtfhKXzs0NPpPK7fD0KmiptMtLvg0X3AkRruEvsFJKhYgGnGG2\ns2Fn32CdH+2eAsDKMwuIiQoyWHz4jO323NsFkbHwhV/DnKtGqrhKKRUyGnCGmW/stKTITCpbMohy\nCavODGLOG68HNvwnvP0ru5ycb5+vyZk7gqVVSqnQ0YAzzHz3b9ytswDh0tNyyUyMOf5OnY3w3Jdh\nz//a5YJz4JrHIT59ZAurlFIhpAFnGFW0VlDWWAbA4dpTALjxnBN0ha7dZSdLa9hrlxfdDBf/FFxR\nI1lUpZQKOQ04w+iNijcAcJk4PB3TWFQ4mTlTko+9w64/2WkFetogIgouuQ8WrA5NYZVSKsQ04Awj\n3/2brpZiwHXs1o3XC5vug9d/bJfjM+38NVPPDE1BlVIqDDTgDJPm7ma21mwFoLd1NnkpsSybnXX0\nht1t8OK/wScv2eXc+bB8LSTnhbC0SikVehpwhsmmyk14jAdjXPS2F3P95wqIdAXM4N2wD9athNod\ndvm05XDp/RAVG/oCK6VUiGnAGSa+y2me9hnEuuK5dlFAV+i9G+HZ1bZHmkTART+Es76ug28qpSYM\nDTjDwH+wzt7W2SxfkEdynNPLzBh49wH4y3+A8UBMClz9CMy4IIwlVkqp0NOAMwy2VG+ho7cDgN62\nWaw+2+ks4O6Cl2+FD560yxmzYMWTkDo9TCVVSqnw0YAzDHwPe3o68zhvehFFmQnQUgVPr4RKO+sn\nM/8FLn8AohPDWFKllAofDTgnyRjDX8tfA6C3bTY3XTYNDv7DDr7ZZsdU4/w74PzbISLiODkppdT4\npgHnJO08vJPG7joAciIXcF7bq/D0d8DTA1HxcMWDMOvSMJdSKaXCTwPOSfrjp38FwNuTwgNJm5CX\nnrIJkwvtZGlZs8NXOKWUGkWCusYjIi4RuVdE6kSkVUTWi8gxR5YUkYtFZIeIdIrIxyKyLCC9SEQ2\niEi7iFSIyHcC0uNE5GERaRSRJhF5SERiA7b5nohUOnlsEJGw3Il/Zc8GAJZ1tjPrgBNspi+Fm1/X\nYKOUUn6CvalwB/AF4AxgirPuiYE2dE78zwM/BZKdny+ISKGT7gL+CHwCZACXAbeLyHK/bO4HZjqv\nYmAW8Au/Y6wEvgdc6uSxE3jJyTtk9jQeoLH3AAArOirsyjO/DivXQ9wgp5NWSqlxLtiAcwtwjzFm\nrzGmGbgNuNgXRALcAGwzxqwxxvQYY9YC2531AOcBBcD3jTEdxpjtwIPAvwI4LZlVwF3GmBpjTC1w\nF3CDiPjG+b8FeNAYs90Y0wH8OzAdWBLsBxeRNBEpFpHi3t7eYHfr58m/3QNAosfLPDfwxQfg4p+A\nS69UKqVUoBMGHBFJBqYC23zrjDF7gBbgtAF2meu/rWO7s96XXmaMaTtGegkQE5DHdiAW29o56hhO\nXrv98gjGN4FSoLS2tnYQuzm8XnY02oc9z+zyMummV2HeisHno5RSE0QwLZwk52dzwPomvzR/iSfY\nNpj0wOP53gebRzB+hQ1uJZmZmYPYzRERwbc/+2tWdqayeMGPIG/B4PNQSqkJJJhrP63Oz8CJXVKw\nrZyBtj/etsGk+47XFHDsYPM4IWPMYeAwwMKFC4PdrZ/FxeeyuPiNIe2rlFITzQlbOMaYJuAAcLpv\nndMxIAn4cIBdPvDf1jHfWe9LLxaR+GOklwJdAXnMBzqBsoGOISIJwCl+eSillBplgu008FtsT7Jp\nIpIE3AP8xRhTPsC2jwMLRWSFiESJyApgAfCYk/4msB/4iYjEisg84KvYjgMYYzqBNcDdIpIpIpnA\n3cDjxpguv/J8VUTmO50MfgTsAzYP6tMrpZQKmWADzs+wXZm3AJWAC9uTDBFZKSJ9HQCcDgVXAHdi\nL3HdCVzuC07GGA+2O/NnsJe0XgHuNcas8zvet7CtGd+rFLjV7xhrgZ8Df3LymANc5uStlFJqFBJj\nTLjLEHYLFy40W7duDXcxlFJqTBGRbcaYoG+C62iSSimlQkIDjlJKqZDQgKOUUiok9B4OICJ12J5z\nQ+ECsoAaQDstnJjW1+BofQ2O1tfgnGx9FRhjMoLdWAPOSRKRYmwvuhJjTNmJtp/otL4GR+trcLS+\nBifU9aWX1JRSSoWEBhyllFIhoQHn5B0G/sv5qU5M62twtL4GR+trcEJaX3oPRymlVEhoC0cppVRI\naMBRSikVEhpwlFJKhYQGHKWUUiGhAUcppVRIaMBRSikVEhpwlFJKhYQGHKWUUiGhAUcppVRIaMAZ\nIhFxici9IlInIq0isl5E0sNdrlAQkUdFxC0ibX6vrwVsc72I7BGRDhF5V0QWBKQvFJF/OOl7RGRV\nQHqmiDzv1G2diNwjImPi71VErhWRTSLSIiK9A6RfLCI7RKRTRD4WkWUB6UUiskFE2kWkQkS+E5Ae\nJyIPi0ijiDSJyEMiEhuwzfdEpNLJY4OITB+ZTzs8jldnIrJUREzA39vbAdtMmDpzvgs7nLo6JCK/\nE5HUgG1G9Ps35POfMUZfQ3gB/wGUAdOBZGA98Gq4yxWiz/4o8PvjpC8B2oFlQDRwG3a+jSQnPRmo\nA2530i8C2oCz/PL4G/C8s+10p65vD/dnD7J+PgesAG4CegPSpgMdwCpgErDSqatCJ90FfAL8CogD\nTgdqgeV+efwOeBs7j0mm8/43fukrnX1Od/L4JfAx4Ap33QyxzpYGrgtIn1B1BvwEmA9EARnAq8Af\n/NJH/PvHEM9/Ya+8sfrCTtj2Zb/lGYDxnTjG84sTB5zHgCf8lgU4ANzgLN/oLIvfNk8Ajzjvpzl1\nOcMv/cvAvnB/9kHW01EnSuxAiZsC1m0C/tN5/1lsQErwS/8h8LrzPhboBC70S7/Q2SfGWX4D+KFf\neoKTfn6462SIdXbUuoD0iV5nlwDNfssj/v0b6vlvTFyiGG1EJBmYCmzzrTPG7AFagNPCVa4Qu1JE\nGkSkzGlaJ/ilzaV/3RjgPWe9L327s95ne0B6s1On/umFIpI03B8kxPrVjSPws5cZY9qOkV4CxATk\nsR17Ui0e6BhOXrv98hiLXCJyUESqReRPIuL/WSZ6nV0IfOi3PKLfv5M5/2nAGRrfSa85YH2TX9p4\n9itgJpAOXA6cj71k4ZPI8etmqOkw9ut3OOqGgG1874PNY6zZBczD/uc9E3tyfU1Ecp30CVtnInIl\ncDPwLb/VI/39G/L5TwPO0LQ6P5MD1qdgo/y4ZozZZoypMcZ4jTE7gFuBq0Qk2tmklePXzVDTfWlj\n2XDUDQHb+N4Hm8eYYoypNsZ8YIzpNcY0GWO+DzQAn3c2mZB1JiJXY//Ru8wYs90vaaS/f0M+/2nA\nGQJjTBP2GujpvnVOj5Yk+jdtJwqv81Ocnx/Qv24E+x/qB37p8wPymB+QnhzQS2g+UG6MCfyvaqzp\nVzeOwM9eLCLxx0gvBboC8piPvUfhm5M+sP4TgFP88hgPvPT/e5tQdSYiNwIPApcaY14PSB7R799J\nnf/CfcNrrL6wvTRKsc38JOBZ4M/hLleIPvu1QIrz/hRsj5/1fulLsL1eLsT2xPou/XvJpGB7yXzP\nSb+QgXvJPOfU7TSnru8I92cPsn5c2HsGy4Be530M9gQ5A3szegW2l9EKBu6ldj/2HsM8p+6u9cv/\nd8BmbG+rTOf9A37pK5195jt5/F9gB6Owx1WQdXYBUIT9BzkB+AH28k3+RKwz4P9gZ+hcdIz0Ef/+\nMcTzX9grb6y+nD/y+4B6bBPzeSA93OUK0WffiL2k0Q7sA37h+2P22+Z6YC/2v8h/AAsC0hc56zud\n7VYFpGc6ddrq1PF/AxHh/uxB1s9qbI+dwFehk36xczLrdH4uC9i/CPhfbGA6BHw3ID0OeNg56TYB\nDwGxAdvc5uzb4eQ1Y6Q+70jXGfaS7X7n760W+HPgyXYi1ZlTL24nSPS9ArYZ0e8fQzz/6RTTSiml\nQkLv4SillAoJDThKKaVCQgOOUkqpkNCAo5RSKiQ04CillAoJDThKKaVCQgOOUkqpkNCAo5RS32f6\n+wAAAAhJREFUKiT+P8E4udoGbXrYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb263689128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Three settings of the lrate hyperparameters.\n",
    "opts = [NoamOpt(512, 1, 4000, None), \n",
    "        NoamOpt(512, 1, 8000, None),\n",
    "        NoamOpt(256, 1, 4000, None)]\n",
    "plt.plot(np.arange(1, 20000), [[opt.rate(i) for opt in opts] for i in range(1, 20000)])\n",
    "plt.legend([\"512:4000\", \"512:8000\", \"256:4000\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization\n",
    "\n",
    "## Label Sommthing\n",
    "\n",
    "During training, we employed label smoothing of value $ϵ_{ls} = 0.1$. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.\n",
    "\n",
    "We implement label smoothing using the `KL div` loss. Instead of using a one-hot target distribution, we create a distribution that has `confidence` of the correct word and the rest of the `smoothing` mass distributed throughout the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LabelSmoothing(nn.Module):\n",
    "    \"Implement label smoothing.\"\n",
    "    def __init__(self, size, padding_idx, smoothing=0.0):\n",
    "        super(LabelSmoothing, self).__init__()\n",
    "        self.criterion = nn.KLDivLoss(size_average=False)\n",
    "        self.padding_idx = padding_idx\n",
    "        self.confidence = 1.0 - smoothing\n",
    "        self.smoothing = smoothing\n",
    "        self.size = size\n",
    "        self.true_dist = None\n",
    "        \n",
    "    def forward(self, x, target):\n",
    "        assert x.size(1) == self.size\n",
    "        true_dist = x.data.clone()\n",
    "        true_dist.fill_(self.smoothing / (self.size - 2))\n",
    "        true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)\n",
    "        true_dist[:, self.padding_idx] = 0\n",
    "        mask = torch.nonzero(target.data == self.padding_idx)\n",
    "        if mask.dim() > 0:\n",
    "            true_dist.index_fill_(0, mask.squeeze(), 0.0)\n",
    "        self.true_dist = true_dist\n",
    "        return self.criterion(x, Variable(true_dist, requires_grad=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see an example of how the mass is distributed to the words based on confidence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dongdongdongdong/anaconda3/lib/python3.6/site-packages/torch/nn/_reduction.py:43: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
      "  warnings.warn(warning.format(ret))\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAD1CAYAAABDY8L3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEB1JREFUeJzt3X+sX3V9x/Hni5ZSsd6yFnEztbZoqEsmBWRD3YxV4u+p\nUUMiA+OPBGIWK5KJKMHBSPzBNDOmzgwaEcG6ZRFZ4hJo/IHR6ZykNe1sZhvB+gODgyv9BZUKvPfH\n93S9ud72fi733H5ve5+P5OT7PZ/zOee8c/64r37O5/R7UlVIkjSZE4ZdgCTp2GBgSJKaGBiSpCYG\nhiSpiYEhSWpiYEiSmhgYkqQmvQZGkpOT3JTkoSS7knwuyVOO0P8dSZ5Ism/M8s991iRJ6kffI4xP\nA8/rljOAPwb+YZJ97q2qRWOWC3uuSZLUg/l9HagbSVwM/GVV/bpr+zDw1SSXV9Vv+zrXuPMuBZZ2\nq6NVNToT55Gkua63wABWAQuBTWPaNgNPYTDa2HqY/Z6V5H7gd8B3gQ9V1U+ncN61wDUAJzCPkSyZ\nat3HpRo5edglzBqrVjw47BJmje07Tx12CZqF9u2578Gqevpk/ZoCI8nNwNuP0OUjwMbu++4x7Qe/\njxxmv28Dzwd+ApwGfBz4WpLVVfVwS23AOuBLACezaPt5Ob9xt+PbgT//02GXMGvcddP6YZcwa7zs\nXZcMuwTNQt++48qftfRrHWG8B3j/EbY/wmAUAbAY2DXmO8CeiXaqqnvHrN6f5BIGIfNC4BsthXW3\noEYBRxeSNIOaAqOq9gH7jtQnyXbgt8A5wDe75rOB/cCOxnqqW9LYX5J0lPT2lFRV7Qe+CFyX5LQk\npwHXAbccbsI7yeuSLMvAEuAfgQeB7/dVlySpH30/VnsZg9HEwWU7cPnBjUmuSrJtTP81wA8YjF62\nMXja6RXdiEaSNIv0+ZQUVfUI8K5umWj7R4GPjlm/AriizxokSTPDnwaRJDUxMCRJTQwMSVITA0OS\n1MTAkCQ1MTAkSU0MDElSEwNDktTEwJAkNTEwJElNDAxJUhMDQ5LUxMCQJDUxMCRJTQwMSVITA0OS\n1MTAkCQ1MTAkSU0MDElSEwNDktTEwJAkNTEwJElNDAxJUhMDQ5LUpPfASDIvySeSPJBkb5Lbkpx6\nhP6vTrItyf4kP0ryyr5rkiRN30yMMD4IvBE4D1jWtd06UcckpwNfAT4GLO4+b0+yYgbqkiRNw0wE\nxqXA9VV1b1XtBj4AvPowIfB2YFNVfbGqDlTVBmBz194kydIkZyQ5o3iih/IlSRPpNTCSLAaWA5sO\ntlXVPcAe4MwJdlk9tm9nc9feai2wHdh+gEenVK8kqV3fI4yR7nP3uPZdY7aN9bQp9D2cdcAqYNUC\nTprCbpKkqZjf8/H2dp+Lx7WfwmCUMVH/1r4TqqpRYBRgJEtad5MkTVGvI4yq2gX8HDjnYFs3sT0C\nbJ1gly1j+3bO7tolSbPITEx63whcmWRlkhHgemBjVe2coO8twLlJLkxyYpILgRcAX5iBuiRJ0zAT\ngfFx4KvA3cB9wDzgYoAkFyXZd7BjNyH+ZuBqBrehrgbedJhwkSQNUd9zGFTV48D7u2X8tg3AhnFt\ndwJ39l2HJKlf/jSIJKmJgSFJamJgSJKaGBiSpCYGhiSpiYEhSWpiYEiSmhgYkqQmBoYkqYmBIUlq\nYmBIkpoYGJKkJgaGJKmJgSFJamJgSJKaGBiSpCYGhiSpiYEhSWpiYEiSmhgYkqQmBoYkqYmBIUlq\nYmBIkpr0HhhJ5iX5RJIHkuxNcluSUw/Td02SSrJvzPK9vmuSJE3fTIwwPgi8ETgPWNa13XqE/o9X\n1aIxy4tnoCZJ0jTNn4FjXgpcV1X3AiT5APCTJCuqamffJ0uyFFgKsIjFfR9ektTpNTCSLAaWA5sO\ntlXVPUn2AGcCOyfYbV6SXwAndvtdVVVbpnDatcA1AAd49ElWfvy566b1wy5h1njZuy4ZdgnScaHv\nW1Ij3efuce27xmwb68fAWcBK4HnAVuCbSZ45hXOuA1YBqxZw0tSqlSQ16zsw9naf4+8NnQLsGd+5\nqu6vqi1V9VhV7aqqDwG/AV7TesKqGq2qHVW1Iz70JUkzpte/sFW1C/g5cM7BtiSnMxhdbG08zBNA\n+qxLkjR9M/FP8huBK5OsTDICXA9snGjCO8nLkzw3yQlJFiW5FngGsHEG6pIkTcNMBMbHga8CdwP3\nAfOAiwGSXJRk35i+q4FvMLiVdS/wQuAVVfWLGahLkjQNvT9WW1WPA+/vlvHbNgAbxqx/CvhU3zVI\nkvrnLLEkqYmBIUlqYmBIkpoYGJKkJgaGJKmJgSFJamJgSJKaGBiSpCYGhiSpiYEhSWpiYEiSmhgY\nkqQmBoYkqYmBIUlqYmBIkpoYGJKkJgaGJKmJgSFJamJgSJKaGBiSpCYGhiSpiYEhSWpiYEiSmhgY\nkqQmvQZGkrcm+U6SPUkea+h/bpIfJHkkyT1JLu6zHklSf/oeYTwEfBZ432QdkywG7gBuA/4AeDfw\nT0le1HNNkqQezO/zYFW1ESDJmobubwb2A39fVQV8LcntwKXAf7aeM8lSYCnAIhZPtWRJUqNhzmGs\nBjZ3YXHQ5q59KtYC24HtB3i0r9okSeMMMzCeBuwe17YLGJnicdYBq4BVCzipj7okSRPo9ZbUFO0F\nVoxrOwXYM5WDVNUoMAowkiW9FCZJ+n3DHGFsAc4e13Z21y5JmmX6fqx2XpKFwIJufWG3ZILutwMn\nJ7kiyYIk5zOYCL+xz5okSf3oe4TxNgZPPm0E5nXf9wPPTvKSJPuSLAeoql3Aa4ELGMxlrAfeXVXN\nT0hJko6evh+rvRm4+TCbdwKLxvW/G/izPmuQJM0MfxpEktTEwJAkNTEwJElNDAxJUhMDQ5LUxMCQ\nJDUxMCRJTQwMSVITA0OS1MTAkCQ1MTAkSU0MDElSEwNDktTEwJAkNTEwJElNDAxJUhMDQ5LUxMCQ\nJDUxMCRJTQwMSVITA0OS1MTAkCQ1MTAkSU16DYwkb03ynSR7kjw2Sd8VSSrJw0n2dcsv+6xHktSf\n+T0f7yHgs8BTgBsb91lVVQaFJM1yvQZGVW0ESLKmz+MeSZKlwFKARSw+WqeVpDmn7xHGk/FfSRYA\n24Brq+pbU9x/LXANwAEe7bm0Y9ernnnWsEuYNRZw97BLkI4Lw5z0fhB4EbASWAHcBtyR5MwpHmcd\nsApYtYCTei1QknTI0AKjqvZV1fer6kBVPVxV64D/AC6Y4nFGq2pHVe2ID31J0oyZbX9hnwAy7CIk\nSb+v78dq5yVZCCzo1hd2y++FQJIXJvmTJPO7PpcCLwVu77MmSVI/+h5hvA3YD2wE5nXf9wPPTvKS\n7v9aLO/6rgT+DdgN3Nft+/qq2tRzTZKkHqSqhl1Db0aypM7L+cMuQ5KOKV+vL2+qqnMn6zfb5jAk\nSbOUgSFJamJgSJKaGBiSpCYGhiSpiYEhSWpiYEiSmhgYkqQmBoYkqYmBIUlqYmBIkpoYGJKkJgaG\nJKmJgSFJamJgSJKaGBiSpCYGhiSpiYEhSWpiYEiSmhgYkqQmBoYkqYmBIUlqYmBIkpoYGJKkJr0G\nRpLrk2xLsifJr5KsT7Jkkn1e3e2zP8mPkryyz5okSf3oe4TxOHAxsBRYDSwDPn+4zklOB74CfAxY\n3H3enmRFz3VJkqap18Coqquq6odV9buqegD4DLDmCLu8HdhUVV+sqgNVtQHY3LU3SbI0yRlJziie\nmFb9kqTDm+k5jPOBrUfYvhrYNK5tc9feai2wHdh+gEenVp0kqdmMBUaStwCXAJcdodvTgN3j2nYB\nI1M41TpgFbBqASdNqUZJUrv5M3HQJBcANwBvqKrNR+i6l8HcxVinAHtaz1VVo8AowMiR59clSdPQ\n+wgjyTsZhMXrq+quSbpvAc4Z13Z21y5JmkX6fqz2vcAngVdV1XcbdrkFODfJhUlOTHIh8ALgC33W\nJUmavr5HGJ9mMP9wV5J9B5eDG5NcNHa9qu4B3gxczeA21NXAm6pqZ891SZKmqdc5jKrKJNs3ABvG\ntd0J3NlnHZKk/vnTIJKkJgaGJKmJgSFJamJgSJKaGBiSpCYGhiSpiYEhSWpiYEiSmhgYkqQmBoYk\nqYmBIUlqYmBIkpoYGJKkJgaGJKmJgSFJamJgSJKaGBiSpCYGhiSpiYEhSWpiYEiSmhgYkqQmBoYk\nqYmBIUlq0mtgJLk+ybYke5L8Ksn6JEuO0H9Nkkqyb8zyvT5rkiT1o+8RxuPAxcBSYDWwDPj8ZPtU\n1aIxy4t7rkmS1IP5fR6sqq4as/pAks8AX+rzHOMlWcogoFjE4pk8lSTNab0GxgTOB7ZO0mdekl8A\nJwKbgKuqassUzrEWuAZgH7sf+Xp9+X+eVKX9mQc8A/g1gxHXXOa1OMRrcYjX4pDZci2e3dIpVTUj\nZ0/yFuBm4KVVtfkwff6QwcXaBiwCrgQuBZ5fVb9qPM//jzCA0aoanWbp05LkDGA7sKqqdgyzlmHz\nWhzitTjEa3HIsXYtZuQpqSQXAOuBNxwuLACq6v6q2lJVj1XVrqr6EPAb4DWt56qq0ara0S1DDQtJ\nOp71HhhJ3gncALy+qu56Eod4Aki/VUmSpqvvx2rfC3wSeFVVfbeh/8uTPDfJCUkWJbmWwS2qjX3W\ndZSNAn/Xfc51XotDvBaHeC0OOaauRa9zGEkKeAx4dGx7VS3qtl8E3DBm/XLgfcCpwMPAZuDDVXV3\nb0VJknoxY5PekqTjiz8NIklqYmBIkpoYGJKkJgaGJKmJgSFJamJgSJKaGBiSpCYGhiSpiYHRoyTz\nknwiyQNJ9ia5Lcmpw67raEvy1iTf6d68+Niw6xmmqb6F8niX5CNJftpdj/9N8uUky4dd17B0P4v0\nve7No8uGXc9kDIx+fRB4I3Aeg7cNAtw6vHKG5iHgswx+9mWuezJvoTye3QqcVVUjwArg58C/DLWi\n4boceGTYRbTyp0F6lORnwHVV9blu/TnAT4CVVbVzmLUNQ5I1wNeraqZf1HXMSPI64EtVNedfD5nk\nqQx+eO+dVbV0sv7Hm+5dGHcAbwF+CDyrqn453KqOzBFGT5IsBpYzeGsgAFV1D7AHOHNYdWnWaXkL\n5XEtyV8l2Q3sAy4Drh1uRUdfkhOAm4ArgF1DLqeZgdGfke5z97j2XWO2aQ7r3kJ5CYM/knNWVR0c\nYf0Rg7D47+FWNBSXAfdX1VeGXchUeKugP3u7z/G3Gk5hMMrQHNa9hfIGJnkL5VxSVfcnWQ/cm2R5\nVf1m2DUdDUmeC/wNcO6wa5kqRxg9qapdDCbwzjnYluR0BqOLOX0LYq7r4S2Ux7P5wFOBZw67kKPo\nL4CnAz9K8iCD9wABbE3y18Mra3IGRr9uBK5MsjLJCHA9sHGuTXh3jxcvBBZ06wu7Zc69eneqb6E8\nnnWPkL4nyWnd+jLgH4GdwI+HWdtR9q/Ac4CzuuW1XfsrgVuGVVQLn5LqUZJ5DELiHcBJwNeAS6vq\nwWHWdbQleQcTPzo6554Wm+wtlHNJN9H77wxuxTyVwfzet4C/7R4QmZOSrAB+yjHwlJSBIUlq4i0p\nSVITA0OS1MTAkCQ1MTAkSU0MDElSEwNDktTEwJAkNTEwJElN/g/m7wNyF5sS7QAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb262fb5be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Example of label smoothing.\n",
    "crit = LabelSmoothing(5, 0, 0.4)\n",
    "predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],\n",
    "                             [0, 0.2, 0.7, 0.1, 0], \n",
    "                             [0, 0.2, 0.7, 0.1, 0]])\n",
    "v = crit(Variable(predict.log()), \n",
    "         Variable(torch.LongTensor([2, 1, 0])))\n",
    "\n",
    "# Show the target distributions expected by the system.\n",
    "plt.imshow(crit.true_dist)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label smoothing actually starts to penalize the model if it gets very confident about a given choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "crit = LabelSmoothing(5, 0, 0.1)\n",
    "\n",
    "def loss(x):\n",
    "    d = x + 3 * 1\n",
    "    predict = torch.FloatTensor([[0, x / d, 1 / d, 1 / d, 1 / d],\n",
    "                                 ])\n",
    "    #print(predict)\n",
    "    return crit(Variable(predict.log()),\n",
    "                 Variable(torch.LongTensor([1]))).data[0]\n",
    "\n",
    "plt.plot(np.arange(1, 100), [loss(x) for x in range(1, 100)])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A First Example\n",
    "\n",
    "We can begin by trying out a simple copy-task. Given a random set of input symbols from a small vocabulary, the goal is to generate back those same symbols."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synthetic Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def data_gen(V, batch, nbatches):\n",
    "    \"Generate random data for a src-tgt copy task.\"\n",
    "    for i in range(nbatches):\n",
    "        data = torch.from_numpy(np.random.randint(1, V, size=(batch, 10)))\n",
    "        data[:, 0] = 1.0\n",
    "        src = Variable(data, requires_grad=False)\n",
    "        tgt = Variable(data, requires_grad=False)\n",
    "        yield Batch(src, tgt, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Computation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class SimpleLossCompute:\n",
    "    \"A simple loss compute and train function.\"\n",
    "    def __init__(self, generator, criterion, opt=None):\n",
    "        self.generator = generator\n",
    "        self.criterion = criterion\n",
    "        self.opt = opt\n",
    "        \n",
    "    def __call__(self, x, y, norm):\n",
    "        x = self.generator(x)\n",
    "        loss = self.criterion(x.contiguous().view(-1, x.size(-1)), \n",
    "                              y.contiguous().view(-1)) / norm\n",
    "        loss.backward()\n",
    "        if self.opt is not None:\n",
    "            self.opt.step()\n",
    "            self.opt.optimizer.zero_grad()\n",
    "        return loss.data[0] * norm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Greedy Decoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the simple copy task.\n",
    "V = 11\n",
    "criterion = LabelSmoothing(size=V, padding_idx=0, smoothing=0.0)\n",
    "model = make_model(V, V, N=2)\n",
    "model_opt = NoamOpt(model.src_embed[0].d_model, 1, 400,\n",
    "                    torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))\n",
    "\n",
    "for epoch in range(10):\n",
    "    model.train()\n",
    "    run_epoch(data_gen(V, 30, 20), model, \n",
    "              SimpleLossCompute(model.generator, criterion, model_opt))\n",
    "    model.eval()\n",
    "    print(run_epoch(data_gen(V, 30, 5), model, \n",
    "                    SimpleLossCompute(model.generator, criterion, None)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code predicts a translation using greedy decoding for simplicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def greedy_decode(model, src, src_mask, max_len, start_symbol):\n",
    "    memory = model.encode(src, src_mask)\n",
    "    ys = torch.ones(1, 1).fill_(start_symbol).type_as(src.data)\n",
    "    for i in range(max_len-1):\n",
    "        out = model.decode(memory, src_mask, \n",
    "                           Variable(ys), \n",
    "                           Variable(subsequent_mask(ys.size(1))\n",
    "                                    .type_as(src.data)))\n",
    "        prob = model.generator(out[:, -1])\n",
    "        _, next_word = torch.max(prob, dim = 1)\n",
    "        next_word = next_word.data[0]\n",
    "        ys = torch.cat([ys, \n",
    "                        torch.ones(1, 1).type_as(src.data).fill_(next_word)], dim=1)\n",
    "    return ys\n",
    "\n",
    "model.eval()\n",
    "src = Variable(torch.LongTensor([[1,2,3,4,5,6,7,8,9,10]]) )\n",
    "src_mask = Variable(torch.ones(1, 1, 10) )\n",
    "print(greedy_decode(model, src, src_mask, max_len=10, start_symbol=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Real World Example\n",
    "\n",
    "Now we consider a real-world example using the IWSLT German-English Translation task. This task is much smaller than the WMT task considered in the paper, but it illustrates the whole system. We also show how to use multi-gpu processing to make it really fast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#!pip install torchtext spacy\n",
    "#!python -m spacy download en\n",
    "#!python -m spacy download de"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loading\n",
    "\n",
    "We will load the dataset using torchtext and spacy for tokenization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For data loading.\n",
    "from torchtext import data, datasets\n",
    "\n",
    "if True:\n",
    "    import spacy\n",
    "    spacy_de = spacy.load('de')\n",
    "    spacy_en = spacy.load('en')\n",
    "\n",
    "    def tokenize_de(text):\n",
    "        return [tok.text for tok in spacy_de.tokenizer(text)]\n",
    "\n",
    "    def tokenize_en(text):\n",
    "        return [tok.text for tok in spacy_en.tokenizer(text)]\n",
    "\n",
    "    BOS_WORD = '<s>'\n",
    "    EOS_WORD = '</s>'\n",
    "    BLANK_WORD = \"<blank>\"\n",
    "    SRC = data.Field(tokenize=tokenize_de, pad_token=BLANK_WORD)\n",
    "    TGT = data.Field(tokenize=tokenize_en, init_token = BOS_WORD, \n",
    "                     eos_token = EOS_WORD, pad_token=BLANK_WORD)\n",
    "\n",
    "    MAX_LEN = 100\n",
    "    train, val, test = datasets.IWSLT.splits(\n",
    "        exts=('.de', '.en'), fields=(SRC, TGT), \n",
    "        filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and \n",
    "            len(vars(x)['trg']) <= MAX_LEN)\n",
    "    MIN_FREQ = 2\n",
    "    SRC.build_vocab(train.src, min_freq=MIN_FREQ)\n",
    "    TGT.build_vocab(train.trg, min_freq=MIN_FREQ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batching matters a ton for speed. We want to have very evenly divided batches, with absolutely minimal padding. To do this we have to hack a bit around the default torchtext batching. This code patches their default batching to make sure we search over enough sentences to find tight batches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyIterator(data.Iterator):\n",
    "    def create_batches(self):\n",
    "        if self.train:\n",
    "            def pool(d, random_shuffler):\n",
    "                for p in data.batch(d, self.batch_size * 100):\n",
    "                    p_batch = data.batch(\n",
    "                        sorted(p, key=self.sort_key),\n",
    "                        self.batch_size, self.batch_size_fn)\n",
    "                    for b in random_shuffler(list(p_batch)):\n",
    "                        yield b\n",
    "            self.batches = pool(self.data(), self.random_shuffler)\n",
    "            \n",
    "        else:\n",
    "            self.batches = []\n",
    "            for b in data.batch(self.data(), self.batch_size,\n",
    "                                          self.batch_size_fn):\n",
    "                self.batches.append(sorted(b, key=self.sort_key))\n",
    "\n",
    "def rebatch(pad_idx, batch):\n",
    "    \"Fix order in torchtext to match ours\"\n",
    "    src, trg = batch.src.transpose(0, 1), batch.trg.transpose(0, 1)\n",
    "    return Batch(src, trg, pad_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-GPU Training\n",
    "\n",
    "Finally to really target fast training, we will use multi-gpu. This code implements multi-gpu word generation. It is not specific to transformer so I won’t go into too much detail. The idea is to split up word generation at training time into chunks to be processed in parallel across many different gpus. We do this using pytorch parallel primitives:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- replicate - split modules onto different gpus.\n",
    "- scatter - split batches onto different gpus\n",
    "- parallel_apply - apply module to batches on different gpus\n",
    "- gather - pull scattered data back onto one gpu.\n",
    "- `nn.DataParallel` - a special module wrapper that calls these all before evaluating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Skip if not interested in multigpu.\n",
    "class MultiGPULossCompute:\n",
    "    \"A multi-gpu loss compute and train function.\"\n",
    "    def __init__(self, generator, criterion, devices, opt=None, chunk_size=5):\n",
    "        # Send out to different gpus.\n",
    "        self.generator = generator\n",
    "        self.criterion = nn.parallel.replicate(criterion, \n",
    "                                               devices=devices)\n",
    "        self.opt = opt\n",
    "        self.devices = devices\n",
    "        self.chunk_size = chunk_size\n",
    "        \n",
    "    def __call__(self, out, targets, normalize):\n",
    "        total = 0.0\n",
    "        generator = nn.parallel.replicate(self.generator, \n",
    "                                                devices=self.devices)\n",
    "        out_scatter = nn.parallel.scatter(out, \n",
    "                                          target_gpus=self.devices)\n",
    "        out_grad = [[] for _ in out_scatter]\n",
    "        targets = nn.parallel.scatter(targets, \n",
    "                                      target_gpus=self.devices)\n",
    "\n",
    "        # Divide generating into chunks.\n",
    "        chunk_size = self.chunk_size\n",
    "        for i in range(0, out_scatter[0].size(1), chunk_size):\n",
    "            # Predict distributions\n",
    "            out_column = [[Variable(o[:, i:i+chunk_size].data, \n",
    "                                    requires_grad=self.opt is not None)] \n",
    "                           for o in out_scatter]\n",
    "            gen = nn.parallel.parallel_apply(generator, out_column)\n",
    "\n",
    "            # Compute loss. \n",
    "            y = [(g.contiguous().view(-1, g.size(-1)), \n",
    "                  t[:, i:i+chunk_size].contiguous().view(-1)) \n",
    "                 for g, t in zip(gen, targets)]\n",
    "            loss = nn.parallel.parallel_apply(self.criterion, y)\n",
    "\n",
    "            # Sum and normalize loss\n",
    "            l = nn.parallel.gather(loss, \n",
    "                                   target_device=self.devices[0])\n",
    "            l = l.sum()[0] / normalize\n",
    "            total += l.data[0]\n",
    "\n",
    "            # Backprop loss to output of transformer\n",
    "            if self.opt is not None:\n",
    "                l.backward()\n",
    "                for j, l in enumerate(loss):\n",
    "                    out_grad[j].append(out_column[j][0].grad.data.clone())\n",
    "\n",
    "        # Backprop all loss through transformer.            \n",
    "        if self.opt is not None:\n",
    "            out_grad = [Variable(torch.cat(og, dim=1)) for og in out_grad]\n",
    "            o1 = out\n",
    "            o2 = nn.parallel.gather(out_grad, \n",
    "                                    target_device=self.devices[0])\n",
    "            o1.backward(gradient=o2)\n",
    "            self.opt.step()\n",
    "            self.opt.optimizer.zero_grad()\n",
    "        return total * normalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create our model, criterion, optimizer, data iterators, and paralelization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# GPUs to use\n",
    "devices = [0, 1, 2, 3]\n",
    "if True:\n",
    "    pad_idx = TGT.vocab.stoi[\"<blank>\"]\n",
    "    model = make_model(len(SRC.vocab), len(TGT.vocab), N=6)\n",
    "    model.cuda()\n",
    "    criterion = LabelSmoothing(size=len(TGT.vocab), padding_idx=pad_idx, smoothing=0.1)\n",
    "    criterion.cuda()\n",
    "    BATCH_SIZE = 12000\n",
    "    train_iter = MyIterator(train, batch_size=BATCH_SIZE, device=0,\n",
    "                            repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),\n",
    "                            batch_size_fn=batch_size_fn, train=True)\n",
    "    valid_iter = MyIterator(val, batch_size=BATCH_SIZE, device=0,\n",
    "                            repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),\n",
    "                            batch_size_fn=batch_size_fn, train=False)\n",
    "    model_par = nn.DataParallel(model, device_ids=devices)\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we train the model. I will play with the warmup steps a bit, but everything else uses the default parameters. On an AWS p3.8xlarge with 4 Tesla V100s, this runs at ~27,000 tokens per second with a batch size of 12,000."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the System"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#!wget https://s3.amazonaws.com/opennmt-models/iwslt.pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if False:\n",
    "    model_opt = NoamOpt(model.src_embed[0].d_model, 1, 2000,\n",
    "            torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))\n",
    "    for epoch in range(10):\n",
    "        model_par.train()\n",
    "        run_epoch((rebatch(pad_idx, b) for b in train_iter), \n",
    "                  model_par, \n",
    "                  MultiGPULossCompute(model.generator, criterion, \n",
    "                                      devices=devices, opt=model_opt))\n",
    "        model_par.eval()\n",
    "        loss = run_epoch((rebatch(pad_idx, b) for b in valid_iter), \n",
    "                          model_par, \n",
    "                          MultiGPULossCompute(model.generator, criterion, \n",
    "                          devices=devices, opt=None))\n",
    "        print(loss)\n",
    "else:\n",
    "    model = torch.load(\"iwslt.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once trained we can decode the model to produce a set of translations. Here we simply translate the first sentence in the validation set. This dataset is pretty small so the translations with greedy search are reasonably accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i, batch in enumerate(valid_iter):\n",
    "    src = batch.src.transpose(0, 1)[:1]\n",
    "    src_mask = (src != SRC.vocab.stoi[\"<blank>\"]).unsqueeze(-2)\n",
    "    out = greedy_decode(model, src, src_mask, \n",
    "                        max_len=60, start_symbol=TGT.vocab.stoi[\"<s>\"])\n",
    "    print(\"Translation:\", end=\"\\t\")\n",
    "    for i in range(1, out.size(1)):\n",
    "        sym = TGT.vocab.itos[out[0, i]]\n",
    "        if sym == \"</s>\": break\n",
    "        print(sym, end =\" \")\n",
    "    print()\n",
    "    print(\"Target:\", end=\"\\t\")\n",
    "    for i in range(1, batch.trg.size(0)):\n",
    "        sym = TGT.vocab.itos[batch.trg.data[i, 0]]\n",
    "        if sym == \"</s>\": break\n",
    "        print(sym, end =\" \")\n",
    "    print()\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Components: BPE, Search, Averaging\n",
    "\n",
    "So this mostly covers the transformer model itself. There are four aspects that we didn’t cover explicitly. We also have all these additional features implemented in OpenNMT-py.\n",
    "\n",
    "1) BPE/ Word-piece: We can use a library to first preprocess the data into subword units. See Rico Sennrich’s subword- nmt implementation. These models will transform the training data to look like this:\n",
    "\n",
    "    ▁Die ▁Protokoll datei ▁kann ▁ heimlich ▁per ▁E - Mail ▁oder ▁FTP ▁an ▁einen ▁bestimmte n ▁Empfänger ▁gesendet ▁werden .\n",
    "    \n",
    "2) Shared Embeddings: When using BPE with shared vocabulary we can share the same weight vectors between the source / target / generator. To add this to the model simply do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if False:\n",
    "    model.src_embed[0].lut.weight = model.tgt_embeddings[0].lut.weight\n",
    "    model.generator.lut.weight = model.tgt_embed[0].lut.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3) Beam Search: This is a bit too complicated to cover here. See the OpenNMT- py for a pytorch implementation.\n",
    "\n",
    "4) Model Averaging: The paper averages the last k checkpoints to create an ensembling effect. We can do this after the fact if we have a bunch of models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def average(model, models):\n",
    "    \"Average models into model\"\n",
    "    for ps in zip(*[m.params() for m in [model] + models]):\n",
    "        p[0].copy_(torch.sum(*ps[1:]) / len(ps[1:]))## "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2.0 BLEU, establishing a new state-of-the-art BLEU score of 28.4. The configuration of this model is listed in the bottom line of Table 3. Training took 3.5 days on 8 P100 GPUs. Even our base model surpasses all previously published models and ensembles, at a fraction of the training cost of any of the competitive models.\n",
    "\n",
    "On the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0, outperforming all of the previously published single models, at less than 1/4 the training cost of the previous state-of-the-art model. The Transformer (big) model trained for English-to-French used dropout rate Pdrop = 0.1, instead of 0.3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code we have written here is a version of the base model.\n",
    "\n",
    "With the addtional extensions in the last section, the OpenNMT-py replication gets to 26.9 on EN-DE WMT. Here I have loaded in those parameters to our reimplemenation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!wget https://s3.amazonaws.com/opennmt-models/en-de-model.pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model, SRC, TGT = torch.load(\"en-de-model.pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.eval()\n",
    "sent = \"▁The ▁log ▁file ▁can ▁be ▁sent ▁secret ly ▁with ▁email ▁or ▁FTP ▁to ▁a ▁specified ▁receiver\".split()\n",
    "src = torch.LongTensor([[SRC.stoi[w] for w in sent]])\n",
    "src = Variable(src)\n",
    "src_mask = (src != SRC.stoi[\"<blank>\"]).unsqueeze(-2)\n",
    "out = greedy_decode(model, src, src_mask, \n",
    "                    max_len=60, start_symbol=TGT.stoi[\"<s>\"])\n",
    "print(\"Translation:\", end=\"\\t\")\n",
    "trans = \"<s> \"\n",
    "for i in range(1, out.size(1)):\n",
    "    sym = TGT.itos[out[0, i]]\n",
    "    if sym == \"</s>\": break\n",
    "    trans += sym + \" \"\n",
    "print(trans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Visualization\n",
    "\n",
    "Even with a greedy decoder the translation looks pretty good. We can further visualize it to see what is happening at each layer of the attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tgt_sent = trans.split()\n",
    "\n",
    "def draw(data, x, y, ax):\n",
    "    seaborn.heatmap(data, \n",
    "                    xticklabels=x, square=True, yticklabels=y, vmin=0.0, vmax=1.0, \n",
    "                    cbar=False, ax=ax)\n",
    "    \n",
    "for layer in range(1, 6, 2):\n",
    "    fig, axs = plt.subplots(1,4, figsize=(20, 10))\n",
    "    print(\"Encoder Layer\", layer+1)\n",
    "    for h in range(4):\n",
    "        draw(model.encoder.layers[layer].self_attn.attn[0, h].data, \n",
    "            sent, sent if h ==0 else [], ax=axs[h])\n",
    "    plt.show()\n",
    "    \n",
    "for layer in range(1, 6, 2):\n",
    "    fig, axs = plt.subplots(1,4, figsize=(20, 10))\n",
    "    print(\"Decoder Self Layer\", layer+1)\n",
    "    for h in range(4):\n",
    "        draw(model.decoder.layers[layer].self_attn.attn[0, h].data[:len(tgt_sent), :len(tgt_sent)], \n",
    "            tgt_sent, tgt_sent if h ==0 else [], ax=axs[h])\n",
    "    plt.show()\n",
    "    print(\"Decoder Src Layer\", layer+1)\n",
    "    fig, axs = plt.subplots(1,4, figsize=(20, 10))\n",
    "    for h in range(4):\n",
    "        draw(model.decoder.layers[layer].self_attn.attn[0, h].data[:len(tgt_sent), :len(sent)], \n",
    "            sent, tgt_sent if h ==0 else [], ax=axs[h])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Hopefully this code is useful for future research. Please reach out if you have any issues. If you find this code helpful, also check out our other OpenNMT tools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    @inproceedings{opennmt,\n",
    "      author    = {Guillaume Klein and\n",
    "                   Yoon Kim and\n",
    "                   Yuntian Deng and\n",
    "                   Jean Senellart and\n",
    "                   Alexander M. Rush},\n",
    "      title     = {OpenNMT: Open-Source Toolkit for Neural Machine Translation},\n",
    "      booktitle = {Proc. ACL},\n",
    "      year      = {2017},\n",
    "      url       = {https://doi.org/10.18653/v1/P17-4012},\n",
    "      doi       = {10.18653/v1/P17-4012}\n",
    "    }"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}