4 - Packed Padded Sequences, Masking, Inference and BLEU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 - 打包填充序列，掩码，推断和 BLEU\n",
    "\n",
    "## 简介\n",
    "\n",
    "在这个笔记本中，我们将为上一个笔记本中的模型添加一些改进 - 打包填充序列和掩码。打包填充序列用于告诉我们的 RNN 在编码器中跳过 `<pad>` token。掩码明确地强制模型忽略某些值，例如对填充元素的注意力。这两种技术在自然语言处理中常被使用。\n",
    "\n",
    "我们还将看看如何将我们的模型用于推断，通过给它一个句子，查看它的翻译结果以及在翻译每个单词时它关注的具体位置。\n",
    "\n",
    "最后，我们将使用 BLEU 指标来衡量我们翻译的质量。\n",
    "\n",
    "## 数据准备\n",
    "\n",
    "首先，我们将导入所有的模块，与之前一样，还增加了用于查看注意力的 `matplotlib` 模块。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore as ms\n",
    "import mindspore.context as context\n",
    "import mindspore.nn as nn\n",
    "import mindspore.ops as ops\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "import spacy\n",
    "import datasets\n",
    "from tqdm.notebook import tqdm\n",
    "import evaluate\n",
    "from matplotlib import pyplot as plt\n",
    "import matplotlib.ticker as ticker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 1234\n",
    "\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "ms.set_seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "导入数据集，接下来不再赘述"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset json (/root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7abc5f908654ed49eda92b0a864c700",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-39e2a81dd5826464.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-340eed13fdfe768e.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-05a87c7dda3754fe.arrow\n"
     ]
    }
   ],
   "source": [
    "dataset = datasets.load_dataset(\"bentrevett/multi30k\")\n",
    "train_data, valid_data, test_data = dataset[\"train\"], dataset[\"validation\"], dataset[\"test\"]\n",
    "en_nlp = spacy.load(\"en_core_web_sm\")\n",
    "de_nlp = spacy.load(\"de_core_news_sm\")\n",
    "def tokenize_example(\n",
    "    example,\n",
    "    en_nlp,\n",
    "    de_nlp,\n",
    "    max_length,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token\n",
    "):\n",
    "    en_tokens = [token.text for token in en_nlp.tokenizer(example[\"en\"])][:max_length]\n",
    "    de_tokens = [token.text for token in de_nlp.tokenizer(example[\"de\"])][:max_length]\n",
    "    if lower:\n",
    "        en_tokens = [token.lower() for token in en_tokens]\n",
    "        de_tokens = [token.lower() for token in de_tokens]\n",
    "    en_tokens = [sos_token] + en_tokens + [eos_token]\n",
    "    de_tokens = [sos_token] + de_tokens + [eos_token]\n",
    "    return {\"en_tokens\": en_tokens, \"de_tokens\": de_tokens}\n",
    "\n",
    "\n",
    "max_length = 1_000\n",
    "lower = True\n",
    "sos_token = \"<sos>\"\n",
    "eos_token = \"<eos>\"\n",
    "\n",
    "fn_kwargs = {\n",
    "    \"en_nlp\": en_nlp, \n",
    "    \"de_nlp\": de_nlp, \n",
    "    \"max_length\": max_length,\n",
    "    \"lower\": lower,\n",
    "    \"sos_token\": sos_token,\n",
    "    \"eos_token\": eos_token,\n",
    "}\n",
    "\n",
    "\n",
    "train_data = train_data.map(tokenize_example, fn_kwargs=fn_kwargs)\n",
    "valid_data = valid_data.map(tokenize_example, fn_kwargs=fn_kwargs)\n",
    "test_data = test_data.map(tokenize_example, fn_kwargs=fn_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "class Vocab:\n",
    "    \"\"\"一个词汇表的实现\"\"\"\n",
    "    def __init__(self, tokens:list, min_freq=0, reserved_tokens:list=None) -> None:\n",
    "        self.default_index = None\n",
    "        if tokens is not None:\n",
    "            # 当第一个条件满足时，就不会跳到第二个判断，避免了空列表报错的情况。\n",
    "            if len(tokens)!=0 and isinstance(tokens[0], list):\n",
    "                tokens = [i for line in tokens for i in line]\n",
    "        else:\n",
    "            tokens = []\n",
    "        if reserved_tokens is None:\n",
    "            reserved_tokens = []\n",
    "        counter=collections.Counter(tokens)\n",
    "        # 按出现词频从高到低排序\n",
    "        self._token_freqs = sorted(counter.items(), key=lambda x:x[1], reverse=True)\n",
    "        # 通过列表,利用序号访问词元。\n",
    "        self.idx_to_token = [] + reserved_tokens # 未知词元<unk>的索引为0, 保留词元排在最前\n",
    "        self.token_to_idx = {\n",
    "            i: k\n",
    "            for k, i in enumerate(self.idx_to_token) \n",
    "        }\n",
    "        \n",
    "        for token, freq in self._token_freqs:\n",
    "            if freq < min_freq:  # 过滤掉出现频率低于要求的词\n",
    "                break\n",
    "            if token not in self.token_to_idx:  \n",
    "                self.idx_to_token.append(token)\n",
    "                self.token_to_idx[token] = len(self.idx_to_token) - 1\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.idx_to_token)\n",
    "    \n",
    "    def __getitem__(self, input_tokens):\n",
    "        \"\"\"输入单字串或序列, 将其全部转化为序号编码\"\"\"\n",
    "        if isinstance(input_tokens, str):\n",
    "            out =  self.token_to_idx.get(input_tokens, self.default_index)\n",
    "            if out is None:\n",
    "                raise Exception('Please call \"set_default_index\" before getting unknown index')\n",
    "            return out\n",
    "        return [self.__getitem__(token) for token in input_tokens]\n",
    "    \n",
    "    def __repr__(self) -> str:\n",
    "        show_items = 5 if len(self) > 5 else len(self)\n",
    "        out = f\"<Vocab with {len(self)} tokens: \"\n",
    "        for i in range(show_items):\n",
    "            out += f'\"{self.idx_to_token[i]}\", '\n",
    "        out += \"...>\"\n",
    "        return out\n",
    "\n",
    "    def __contains__(self, token:str) -> bool:\n",
    "        return token in self.idx_to_token\n",
    "\n",
    "    def to_tokens(self, input_keys):\n",
    "        \"\"\"输入单s索引或序列, 将其全部转化为词元\"\"\"\n",
    "        if isinstance(input_keys, int):\n",
    "            return self.idx_to_token[input_keys] if input_keys < len(self) else self.idx_to_token[0]\n",
    "        elif isinstance(input_keys, (list, tuple)):\n",
    "            return [self.to_tokens(keys) for keys in input_keys]\n",
    "        else:\n",
    "            return self.idx_to_token[0]\n",
    "    \n",
    "    def get_itos(self):\n",
    "        return self.idx_to_token\n",
    "    \n",
    "    def get_stoi(self):\n",
    "        return self.token_to_idx\n",
    "    \n",
    "    def set_default_index(self, idx):\n",
    "        if isinstance(idx, int):\n",
    "            self.default_index = idx\n",
    "        else:\n",
    "            raise Exception(f\"Only type int allowed, got {type(idx)}\")\n",
    "\n",
    "    def lookup_indices(self, input_tokens):\n",
    "        return self.__getitem__(input_tokens)\n",
    "    \n",
    "    def lookup_tokens(self, idx):\n",
    "        return self.to_tokens(idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-6689c04abfdb721f.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-acab783df695b18d.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-e1088b8589007c26.arrow\n"
     ]
    }
   ],
   "source": [
    "min_freq = 2\n",
    "unk_token = \"<unk>\"\n",
    "pad_token = \"<pad>\"\n",
    "\n",
    "special_tokens = [\n",
    "    unk_token,\n",
    "    pad_token,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    "]\n",
    "\n",
    "def build_vocab_from_iterator(tokens, min_freq, specials):\n",
    "    return Vocab(tokens, min_freq, specials)\n",
    "\n",
    "en_vocab = build_vocab_from_iterator(\n",
    "    train_data[\"en_tokens\"],\n",
    "    min_freq=min_freq,\n",
    "    specials=special_tokens,\n",
    ")\n",
    "\n",
    "de_vocab = build_vocab_from_iterator(\n",
    "    train_data[\"de_tokens\"],\n",
    "    min_freq=min_freq,\n",
    "    specials=special_tokens,  \n",
    ")\n",
    "\n",
    "assert en_vocab[unk_token] == de_vocab[unk_token]\n",
    "assert en_vocab[pad_token] == de_vocab[pad_token]\n",
    "\n",
    "unk_index = en_vocab[unk_token]\n",
    "pad_index = en_vocab[pad_token]\n",
    "\n",
    "en_vocab.set_default_index(unk_index)\n",
    "de_vocab.set_default_index(unk_index)\n",
    "\n",
    "def numericalize_example(example, en_vocab, de_vocab):\n",
    "    en_ids = en_vocab.lookup_indices(example[\"en_tokens\"])\n",
    "    de_ids = de_vocab.lookup_indices(example[\"de_tokens\"])\n",
    "    return {\"en_ids\": en_ids, \"de_ids\": de_ids}\n",
    "\n",
    "fn_kwargs = {\n",
    "\"en_vocab\": en_vocab, \n",
    "\"de_vocab\": de_vocab\n",
    "}\n",
    "\n",
    "train_data = train_data.map(numericalize_example, fn_kwargs=fn_kwargs)\n",
    "valid_data = valid_data.map(numericalize_example, fn_kwargs=fn_kwargs)\n",
    "test_data = test_data.map(numericalize_example, fn_kwargs=fn_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_type = \"numpy\"\n",
    "format_columns = [\"en_ids\", \"de_ids\"]\n",
    "\n",
    "train_data = train_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True\n",
    ")\n",
    "\n",
    "valid_data = valid_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True,\n",
    ")\n",
    "\n",
    "test_data = test_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_sequence(sequences:list, padding_value:int):\n",
    "    '''将序列填充到等长并返回mindspore张量'''\n",
    "    # Find the length of the longest sequence in the batch\n",
    "    max_length = max(len(seq) for seq in sequences)\n",
    "    padded_sequences:ms.Tensor = ops.full((len(sequences), max_length), padding_value, dtype=ms.int64)\n",
    "    # Copy the sequences into the padded array\n",
    "    for i, seq in enumerate(sequences):\n",
    "        padded_sequences[i, :len(seq)] = ms.tensor(seq).astype(np.int64)\n",
    "    # 换轴，保证输出为时序优先\n",
    "    padded_sequences = padded_sequences.swapaxes(0, 1)\n",
    "    return padded_sequences  \n",
    "\n",
    "def get_collate_fn(pad_index):\n",
    "    \n",
    "    def collate_fn(batch):\n",
    "        batch_en_ids = [example[\"en_ids\"] for example in batch]\n",
    "        batch_de_ids = [example[\"de_ids\"] for example in batch]\n",
    "        batch_en_ids = pad_sequence(batch_en_ids, padding_value=pad_index)\n",
    "        batch_de_ids = pad_sequence(batch_de_ids, padding_value=pad_index)\n",
    "        batch = {\n",
    "            \"en_ids\": batch_en_ids,\n",
    "            \"de_ids\": batch_de_ids,\n",
    "        }\n",
    "        return batch\n",
    "    \n",
    "    return collate_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataLoader:\n",
    "    def __init__(self, source, batch_size, shuffle=False, per_batch_map=None):\n",
    "        self.source = source\n",
    "        self.batch_size = batch_size\n",
    "        self.shuffle = shuffle\n",
    "        self.per_batch_map = per_batch_map\n",
    "        self.indices = np.arange(len(source))\n",
    "        self.current_index = 0\n",
    "\n",
    "        if self.shuffle:\n",
    "            np.random.shuffle(self.indices)\n",
    "\n",
    "    def __iter__(self):\n",
    "        return self\n",
    "\n",
    "    def __next__(self):\n",
    "        if self.current_index >= len(self.source):\n",
    "            self.current_index = 0\n",
    "            raise StopIteration\n",
    "\n",
    "        batch_indices = self.indices[self.current_index:self.current_index + self.batch_size]\n",
    "        batch_data = [self.source[int(i)] for i in batch_indices]\n",
    "\n",
    "        if self.per_batch_map:\n",
    "            batch_data = self.per_batch_map(batch_data)\n",
    "\n",
    "        self.current_index += self.batch_size\n",
    "        return batch_data\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.source) // self.batch_size\n",
    "\n",
    "\n",
    "def get_data_loader(dataset, batch_size, pad_index, shuffle=False):\n",
    "    collate_fn = get_collate_fn(pad_index)\n",
    "    dataloader = DataLoader(dataset, batch_size, shuffle=shuffle, per_batch_map=collate_fn)\n",
    "    \n",
    "    return dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "\n",
    "train_data_loader = get_data_loader(train_data, batch_size, pad_index, shuffle=True)\n",
    "valid_data_loader = get_data_loader(valid_data, batch_size, pad_index)\n",
    "test_data_loader = get_data_loader(test_data, batch_size, pad_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建模型\n",
    "\n",
    "### 编码器\n",
    "\n",
    "接下来，我们定义编码器。\n",
    "\n",
    "主要的变化发生在 `construct` 方法中。现在，它不仅接受源句子，还接受源句子的长度。\n",
    "\n",
    "在对源句子进行嵌入后（由迭代器自动填充），我们可以使用 `pack_padded_sequence` 和句子的长度`src_len`。`packed_embedded` 将成为我们的打包填充序列。然后，我们可以像往常一样将其馈送到我们的RNN中，该RNN将返回 `packed_outputs`，其中包含序列中的所有隐藏状态，以及 `hidden`，它只是我们序列的最终隐藏状态。`hidden` 是一个标准张量，没有以任何方式打包，唯一的区别是由于输入是一个打包的序列，此张量来自序列中的最终 **非填充元素**。(用于加快计算速度，目前Mindspore中无此功能)\n",
    "\n",
    "然后，我们使用 `pad_packed_sequence` 解压我们的 `packed_outputs`，它返回 `outputs` 和每个序列的长度，但我们不需要这些长度。\n",
    "\n",
    "`outputs` 的第一个维度是填充序列长度，但由于使用了打包填充序列，当填充token是输入时，张量的值将全部为零。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Cell):\n",
    "    def __init__(self, input_dim, emb_dim, enc_hid_dim, dec_hid_dim, dropout):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.GRU(emb_dim, enc_hid_dim, bidirectional = True)\n",
    "        \n",
    "        self.fc = nn.Dense(enc_hid_dim * 2, dec_hid_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "        \n",
    "    def construct(self, src, src_len):\n",
    "        \n",
    "        #src = [src len, batch size]\n",
    "        #src_len = [batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(src))\n",
    "        \n",
    "        #embedded = [src len, batch size, emb dim]\n",
    "                \n",
    "        #need to explicitly put lengths on cpu!\n",
    "        # packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, src_len.to('cpu'))\n",
    "                \n",
    "        outputs, hidden = self.rnn(embedded)\n",
    "                                 \n",
    "        #packed_outputs is a packed sequence containing all hidden states\n",
    "        #hidden is now from the final non-padded element in the batch\n",
    "            \n",
    "        # outputs, _ = nn.utils.rnn.pad_pacedk_sequence(packed_outputs) \n",
    "            \n",
    "        #outputs is now a non-packed sequence, all hidden states obtained\n",
    "        #  when the input is a pad token are all zeros\n",
    "            \n",
    "        #outputs = [src len, batch size, hid dim * num directions]\n",
    "        #hidden = [n layers * num directions, batch size, hid dim]\n",
    "        \n",
    "        #hidden is stacked [forward_1, backward_1, forward_2, backward_2, ...]\n",
    "        #outputs are always from the last layer\n",
    "        \n",
    "        #hidden [-2, :, : ] is the last of the forwards RNN \n",
    "        #hidden [-1, :, : ] is the last of the backwards RNN\n",
    "        \n",
    "        #initial decoder hidden is final hidden state of the forwards and backwards \n",
    "        #  encoder RNNs fed through a linear layer\n",
    "        hidden = ops.tanh(self.fc(ops.cat((hidden[-2,:,:], hidden[-1,:,:]), axis = 1)))\n",
    "        \n",
    "        #outputs = [src len, batch size, enc hid dim * 2]\n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        \n",
    "        return outputs, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 注意力\n",
    "\n",
    "注意力模块是我们计算源语句上的注意力值的地方。\n",
    "\n",
    "以前，我们允许此模块“关注”源语句中的填充标记。但是，使用*masking*，我们可以强制注意力仅集中在非填充元素上。\n",
    "\n",
    "`construct` 方法现在接受一个 `mask` 输入。这是一个 **[batch size, source sentence length]** 的张量，当源语句标记不是填充标记时为1，当它是填充标记时为0。例如，如果源语句是: [\"hello\", \"how\", \"are\", \"you\", \"?\", `<pad>`, `<pad>`]，那么 mask 将是 [1, 1, 1, 1, 1, 0, 0]。\n",
    "\n",
    "我们在计算注意力后但在通过 `softmax` 函数进行规范化之前应用掩码。它使用 `masked_fill` 进行应用。这会在第一个参数（`mask == 0`）为真的每个元素处，用第二个参数给定的值（`-1e10`）填充张量。换句话说，它将取未规范化的注意值，并将填充元素上的注意值更改为 `-1e10`。由于与其他值相比，这些数字将是微不足道的，因此当通过 `softmax` 层传递时，它们将变为零，确保在源语句中不关注填充标记。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Attention(nn.Cell):\n",
    "    def __init__(self, enc_hid_dim, dec_hid_dim):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.attn = nn.Dense((enc_hid_dim * 2) + dec_hid_dim, dec_hid_dim)\n",
    "        self.v = nn.Dense(dec_hid_dim, 1, has_bias = False)\n",
    "        \n",
    "    def construct(self, hidden, encoder_outputs, mask):\n",
    "        \n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        #encoder_outputs = [src len, batch size, enc hid dim * 2]\n",
    "        \n",
    "        batch_size = encoder_outputs.shape[1]\n",
    "        src_len = encoder_outputs.shape[0]\n",
    "        \n",
    "        #repeat decoder hidden state src_len times\n",
    "        hidden = hidden.unsqueeze(1).tile((1, src_len, 1))\n",
    "  \n",
    "        encoder_outputs = encoder_outputs.permute(1, 0, 2)\n",
    "        \n",
    "        #hidden = [batch size, src len, dec hid dim]\n",
    "        #encoder_outputs = [batch size, src len, enc hid dim * 2]\n",
    "        \n",
    "        energy = ops.tanh(self.attn(ops.cat((hidden, encoder_outputs), axis = 2))) \n",
    "        \n",
    "        #energy = [batch size, src len, dec hid dim]\n",
    "\n",
    "        attention = self.v(energy).squeeze(2)\n",
    "        \n",
    "        #attention = [batch size, src len]\n",
    "        \n",
    "        attention = attention.masked_fill(mask == 0, -1e10)\n",
    "        \n",
    "        return ops.softmax(attention, axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 解码器\n",
    "\n",
    "解码器只需要进行一些小的更改。它需要接受源语句上的掩码并将其传递到注意力模块。由于我们希望在推断期间查看注意力的值，因此我们还返回注意张量。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Cell):\n",
    "    def __init__(self, output_dim, emb_dim, enc_hid_dim, dec_hid_dim, dropout, attention):\n",
    "        super().__init__()\n",
    "\n",
    "        self.output_dim = output_dim\n",
    "        self.attention = attention\n",
    "        \n",
    "        self.embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.GRU((enc_hid_dim * 2) + emb_dim, dec_hid_dim)\n",
    "        \n",
    "        self.fc_out = nn.Dense((enc_hid_dim * 2) + dec_hid_dim + emb_dim, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "        \n",
    "    def construct(self, input, hidden, encoder_outputs, mask):\n",
    "             \n",
    "        #input = [batch size]\n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        #encoder_outputs = [src len, batch size, enc hid dim * 2]\n",
    "        #mask = [batch size, src len]\n",
    "        input = input.unsqueeze(0)\n",
    "        #input = [1, batch size]\n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        #embedded = [1, batch size, emb dim]\n",
    "        a = self.attention(hidden, encoder_outputs, mask)      \n",
    "        #a = [batch size, src len]\n",
    "        a = a.unsqueeze(1)\n",
    "        #a = [batch size, 1, src len]\n",
    "        encoder_outputs = encoder_outputs.permute(1, 0, 2)\n",
    "        #encoder_outputs = [batch size, src len, enc hid dim * 2]\n",
    "        weighted = ops.bmm(a, encoder_outputs)\n",
    "        #weighted = [batch size, 1, enc hid dim * 2]\n",
    "        weighted = weighted.permute(1, 0, 2)\n",
    "        #weighted = [1, batch size, enc hid dim * 2]\n",
    "        rnn_input = ops.cat((embedded, weighted), axis = 2)\n",
    "        #rnn_input = [1, batch size, (enc hid dim * 2) + emb dim]\n",
    "        output, hidden = self.rnn(rnn_input, hidden.unsqueeze(0))\n",
    "        #output = [seq len, batch size, dec hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, dec hid dim]\n",
    "        #seq len, n layers and n directions will always be 1 in this decoder, therefore:\n",
    "        #output = [1, batch size, dec hid dim]\n",
    "        #hidden = [1, batch size, dec hid dim]\n",
    "        #this also means that output == hidden\n",
    "        assert (output == hidden).all()\n",
    "        embedded = embedded.squeeze(0)\n",
    "        output = output.squeeze(0)\n",
    "        weighted = weighted.squeeze(0)\n",
    "        prediction = self.fc_out(ops.cat((output, weighted, embedded), axis = 1))\n",
    "        #prediction = [batch size, output dim]\n",
    "        \n",
    "        return prediction, hidden.squeeze(0), a.squeeze(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seq2Seq\n",
    "\n",
    "为了使用打包填充的序列、掩码和推断，总体 seq2seq 模型还需要一些更改。\n",
    "\n",
    "我们需要告诉它 pad token的索引，并将源语句长度作为输入传递给 `construct` 方法。\n",
    "\n",
    "我们使用 pad token索引创建掩码，方法是创建一个遮罩张量，在源语句不等于 pad token的地方为 1。这一切都在 `create_mask` 函数中完成。\n",
    "\n",
    "序列长度需要传递给编码器以使用打包填充的序列。\n",
    "\n",
    "每个时间步的注意力存储在 `attentions` 中\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Cell):\n",
    "    def __init__(self, encoder:Encoder, decoder:Encoder, src_pad_idx):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.src_pad_idx = src_pad_idx\n",
    "        \n",
    "    def create_mask(self, src):\n",
    "        mask = (src != self.src_pad_idx).astype(ms.int32).permute(1, 0)\n",
    "        return mask\n",
    "        \n",
    "    def construct(self, src, src_len, trg, teacher_forcing_ratio = 0.5):\n",
    "        \n",
    "        batch_size = src.shape[1]\n",
    "        trg_len = trg.shape[0]\n",
    "        trg_vocab_size = self.decoder.output_dim\n",
    "        \n",
    "        #存储解码器输出\n",
    "        outputs = []\n",
    "\n",
    "        # 编码器（encoder）：\n",
    "        # 输入：源序列、源序列长度\n",
    "        # 输出1：编码器中所有前向与反向RNN 的隐藏状态 encoder_outputs\n",
    "        # 输出2：编码器中前向与反向RNN中最后时刻的隐藏状态放入线性层后的输出 hidden\n",
    "        encoder_outputs, hidden = self.encoder(src, src_len)\n",
    "\n",
    "        #解码器的第一个输入是表示序列开始的占位符<bos>\n",
    "        inputs = trg[0,:]\n",
    "\n",
    "        # 标记源序列中<pad>占位符的位置\n",
    "        # shape = [batch size, src len]\n",
    "        mask = self.create_mask(src)\n",
    "                \n",
    "        for t in range(1, trg_len):\n",
    "            # 解码器（decoder）：\n",
    "            # 输入：源句子序列 inputs、前一时刻的隐藏状态 hidden、编码器所有前向与反向RNN的隐藏状态\n",
    "            # 标明每个句子中的<pad>，方便计算注意力权重时忽略该部分\n",
    "            # 输出：预测结果 output、新的隐藏状态 hidden、注意力权重（忽略）\n",
    "            output, hidden, _ = self.decoder(inputs, hidden, encoder_outputs, mask)\n",
    "            if len(outputs) == 0:\n",
    "                outputs.append(ops.zeros(output.shape, dtype=output.dtype))\n",
    "\n",
    "            # 将预测结果放入之前的存储中\n",
    "            outputs.append(output)\n",
    "\n",
    "            #找出对应预测概率最大的词元\n",
    "            top1 = output.argmax(1).astype(ms.int32)\n",
    "\n",
    "            if self.training:\n",
    "                #如果目前为模型训练状态，则按照之前设定的概率使用teacher forcing\n",
    "                teacher_force = random.random() < teacher_forcing_ratio\n",
    "                # 如使用teacher forcing，则将目标序列中对应的词元作为下一个输入\n",
    "                # 如不使用teacher forcing，则将预测结果作为下一个输入\n",
    "                inputs = trg[t] if teacher_force else top1\n",
    "            else:\n",
    "                inputs = top1\n",
    "\n",
    "        # 将所有输出整合为tensor\n",
    "        outputs = ops.stack(outputs, axis=0)\n",
    "\n",
    "        return outputs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim = len(de_vocab)\n",
    "output_dim = len(en_vocab)\n",
    "encoder_embedding_dim = 256\n",
    "decoder_embedding_dim = 256\n",
    "encoder_hidden_dim = 512\n",
    "decoder_hidden_dim = 512\n",
    "encoder_dropout = 0.5\n",
    "decoder_dropout = 0.5\n",
    "SRC_PAD_IDX = de_vocab.get_stoi()[pad_token]\n",
    "\n",
    "\n",
    "attention = Attention(encoder_hidden_dim, decoder_hidden_dim)\n",
    "\n",
    "encoder = Encoder(\n",
    "    input_dim,\n",
    "    encoder_embedding_dim,\n",
    "    encoder_hidden_dim,\n",
    "    decoder_hidden_dim,\n",
    "    encoder_dropout,\n",
    ")\n",
    "\n",
    "decoder = Decoder(\n",
    "    output_dim,\n",
    "    decoder_embedding_dim,\n",
    "    encoder_hidden_dim,\n",
    "    decoder_hidden_dim,\n",
    "    decoder_dropout,\n",
    "    attention,\n",
    ")\n",
    "\n",
    "\n",
    "model = Seq2Seq(encoder, decoder, SRC_PAD_IDX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Seq2Seq<\n",
       "  (encoder): Encoder<\n",
       "    (embedding): Embedding<vocab_size=7853, embedding_size=256, use_one_hot=False, embedding_table=Parameter (name=encoder.embedding.embedding_table, shape=(7853, 256), dtype=Float32, requires_grad=True), dtype=Float32, padding_idx=None>\n",
       "    (rnn): GRU<\n",
       "      (rnn): _DynamicGRUCPUGPU<>\n",
       "      (dropout_op): Dropout<p=0.0>\n",
       "      >\n",
       "    (fc): Dense<input_channels=1024, output_channels=512, has_bias=True>\n",
       "    (dropout): Dropout<p=0.5>\n",
       "    >\n",
       "  (decoder): Decoder<\n",
       "    (attention): Attention<\n",
       "      (attn): Dense<input_channels=1536, output_channels=512, has_bias=True>\n",
       "      (v): Dense<input_channels=512, output_channels=1>\n",
       "      >\n",
       "    (embedding): Embedding<vocab_size=5893, embedding_size=256, use_one_hot=False, embedding_table=Parameter (name=decoder.embedding.embedding_table, shape=(5893, 256), dtype=Float32, requires_grad=True), dtype=Float32, padding_idx=None>\n",
       "    (rnn): GRU<\n",
       "      (rnn): _DynamicGRUCPUGPU<>\n",
       "      (dropout_op): Dropout<p=0.0>\n",
       "      >\n",
       "    (fc_out): Dense<input_channels=1792, output_channels=5893, has_bias=True>\n",
       "    (dropout): Dropout<p=0.5>\n",
       "    >\n",
       "  >"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_weights(m:nn.Cell):\n",
    "    for name, param in m.parameters_and_names():\n",
    "        if \"weight\" in name:\n",
    "            param.set_data(ops.normal(param.shape, ms.Tensor([0.]), ms.Tensor([0.01])))\n",
    "        else:\n",
    "            param.set_data(ops.zeros(param.shape, dtype=param.dtype))\n",
    "\n",
    "model.apply(init_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 20,518,405 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.get_parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = nn.Adam(model.trainable_params())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss(ignore_index=pad_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_fn(src, src_len, trg, teacher_forcing_ratio):\n",
    "    # src = [src length, batch size]\n",
    "    # trg = [trg length, batch size]\n",
    "    output = model(src, src_len, trg, teacher_forcing_ratio)\n",
    "    # output = [trg length, batch size, trg vocab size]\n",
    "    output_dim = output.shape[-1]\n",
    "    output = output[1:].view(-1, output_dim)\n",
    "    # output = [(trg length - 1) * batch size, trg vocab size]\n",
    "    trg = trg[1:].view(-1)\n",
    "    # trg = [(trg length - 1) * batch size]\n",
    "    loss = criterion(output, trg.astype(ms.int32))\n",
    "    return loss\n",
    "\n",
    "grad_fn = ms.value_and_grad(forward_fn, grad_position=None, weights=model.trainable_params())\n",
    "\n",
    "\n",
    "def train_fn(data_loader, optimizer, clip, teacher_forcing_ratio):\n",
    "    epoch_loss = 0.\n",
    "    model.set_train(True)\n",
    "    for i, batch in enumerate(tqdm(data_loader)):\n",
    "        src = batch[\"de_ids\"]\n",
    "        trg = batch[\"en_ids\"]\n",
    "        src_len = src.shape[0]\n",
    "        loss, grads = grad_fn(src, src_len, trg, teacher_forcing_ratio)\n",
    "        # grads = ops.clip_by_norm(grads, max_norm=clip)\n",
    "        optimizer(grads)\n",
    "        epoch_loss += float(loss)\n",
    "\n",
    "    return epoch_loss / len(data_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_fn(model:Seq2Seq, data_loader, criterion):\n",
    "    \"\"\"模型验证\"\"\"\n",
    "    model.set_train(False)\n",
    "    epoch_loss = 0\n",
    "    for i, batch in enumerate(data_loader):\n",
    "        src = batch[\"de_ids\"]\n",
    "        trg = batch[\"en_ids\"]\n",
    "        src_len = src.shape[0]\n",
    "\n",
    "        output = model(src, src_len, trg, 0) # turn off teacher forcing\n",
    "        # output = [trg length, batch size, trg vocab size]\n",
    "        output_dim = output.shape[-1]\n",
    "        output = output[1:].view(-1, output_dim)\n",
    "        # output = [(trg length - 1) * batch size, trg vocab size]\n",
    "        trg = trg[1:].view(-1)\n",
    "        # trg = [(trg length - 1) * batch size]\n",
    "        loss = criterion(output, trg.astype(ms.int32))\n",
    "        epoch_loss += float(loss)\n",
    "\n",
    "    return epoch_loss / len(data_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "451707cfd6334f849097cbcea8c68843",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae5f376a94b8432eb2d3d1c51217cc66",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   5.141 | Train PPL: 170.885\n",
      "\tValid Loss:   5.604 | Valid PPL: 271.597\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f04a39060be49aaa5be8e8ecdae49c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   4.282 | Train PPL:  72.351\n",
      "\tValid Loss:   5.119 | Valid PPL: 167.157\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e0f396de00446c7b0a0b7582a910db9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.757 | Train PPL:  42.802\n",
      "\tValid Loss:   4.548 | Valid PPL:  94.439\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "23da61f1a08946098b6212ec07d7bc44",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.160 | Train PPL:  23.569\n",
      "\tValid Loss:   4.144 | Valid PPL:  63.023\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b718f22b029d471893d6b1c3a36e6a6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.746 | Train PPL:  15.583\n",
      "\tValid Loss:   3.980 | Valid PPL:  53.498\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53ab0bcf669242b7bfe09d3ba666220e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.430 | Train PPL:  11.354\n",
      "\tValid Loss:   3.832 | Valid PPL:  46.163\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1009776e2d0642df92b5acc97222449a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.197 | Train PPL:   8.994\n",
      "\tValid Loss:   3.787 | Valid PPL:  44.122\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2abc1213800c4e3c805eab0d1b3adf7b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   1.971 | Train PPL:   7.179\n",
      "\tValid Loss:   3.899 | Valid PPL:  49.375\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed70df4206af42f0b6c92c11047f5bc4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   1.798 | Train PPL:   6.040\n",
      "\tValid Loss:   3.882 | Valid PPL:  48.506\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d7b41a7b6654f829ffcb68ceb62320e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   1.676 | Train PPL:   5.345\n",
      "\tValid Loss:   3.856 | Valid PPL:  47.257\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 10\n",
    "clip = 1.0\n",
    "teacher_forcing_ratio = 0.5\n",
    "\n",
    "best_valid_loss = float(\"inf\")\n",
    "\n",
    "for epoch in tqdm(range(n_epochs)):\n",
    "        \n",
    "    train_loss = train_fn(\n",
    "        train_data_loader, \n",
    "        optimizer, \n",
    "        clip, \n",
    "        teacher_forcing_ratio, \n",
    "    )\n",
    "    \n",
    "    valid_loss = evaluate_fn(\n",
    "        model, \n",
    "        valid_data_loader, \n",
    "        criterion, \n",
    "    )\n",
    "\n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        ms.save_checkpoint(model, f\"./tut4-model.ckpt\")\n",
    "    \n",
    "    print(f\"\\tTrain Loss: {train_loss:7.3f} | Train PPL: {np.exp(train_loss):7.3f}\")\n",
    "    print(f\"\\tValid Loss: {valid_loss:7.3f} | Valid PPL: {np.exp(valid_loss):7.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们从最佳验证损失中加载参数，并在测试集上获得结果。\n",
    "\n",
    "我们获得了改进的测试困惑度，同时速度几乎快了一倍！\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Test Loss: 3.761 | Test PPL:  42.995 |\n"
     ]
    }
   ],
   "source": [
    "ms.load_checkpoint(f\"tut4-model.ckpt\", model)\n",
    "\n",
    "test_loss = evaluate_fn(model, test_data_loader, criterion)\n",
    "\n",
    "print(f\"| Test Loss: {test_loss:.3f} | Test PPL: {np.exp(test_loss):7.3f} |\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 推理\n",
    "\n",
    "现在我们可以使用训练好的模型生成翻译。\n",
    "\n",
    "**注意：**这些翻译与论文中显示的示例相比可能较差，因为它们使用了1000的隐藏维度大小并训练了4天！它们已经被精选，以展示在足够大的模型上应该是什么样的关注。\n",
    "\n",
    "我们的 `translate_sentence` 将执行以下操作：\n",
    "- 确保我们的模型处于评估模式，这在推理中应始终如此\n",
    "- 如果尚未标记化（是字符串），则标记源句\n",
    "- 数字化源句\n",
    "- 将其转换为张量并添加批处理维度\n",
    "- 获取源句长度并转换为张量\n",
    "- 将源句馈送到编码器\n",
    "- 为源句创建掩码\n",
    "- 创建一个列表以保存输出句子，初始化为 `<sos>` token\n",
    "- 创建一个张量以保存注意力值\n",
    "- 当我们没有达到最大长度时\n",
    "  - 获取输入张量，应该是 `<sos>` 或上次预测的token\n",
    "  - 将输入、所有编码器输出、隐藏状态和掩码馈送到解码器\n",
    "  - 存储注意值\n",
    "  - 获取预测的下一个token\n",
    "  - 将预测添加到当前输出句预测\n",
    "  - 如果预测是 `<eos>` token，则中断\n",
    "- 将输出句从索引转换为标记\n",
    "- 返回输出句（去除 `<sos>` token）和整个序列上的注意力值\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate_sentence(\n",
    "    sentence, \n",
    "    model:Seq2Seq,\n",
    "    en_nlp:spacy.Language,\n",
    "    de_nlp:spacy.Language,\n",
    "    en_vocab:Vocab,\n",
    "    de_vocab:Vocab,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    "    max_output_length=25,\n",
    "):\n",
    "    model.set_train(False)\n",
    "    if isinstance(sentence, str):\n",
    "        tokens = [token.text for token in de_nlp.tokenizer(sentence)]\n",
    "    else:\n",
    "        tokens = [token for token in sentence]\n",
    "    if lower:\n",
    "        tokens = [token.lower() for token in tokens]\n",
    "    tokens = [sos_token] + tokens + [eos_token]\n",
    "    ids = de_vocab.lookup_indices(tokens)\n",
    "\n",
    "    tensor = ms.Tensor(ids).astype(ms.int64).unsqueeze(-1)\n",
    "    encoder_outputs, hidden = model.encoder(tensor, len(ids))\n",
    "    mask = model.create_mask(tensor)\n",
    "\n",
    "\n",
    "    inputs = en_vocab.lookup_indices([sos_token])\n",
    "    attentions = ops.zeros((max_output_length, 1, len(ids)))\n",
    "    for i in range(max_output_length):\n",
    "        inputs_tensor = ms.Tensor(inputs[-1]).unsqueeze(-1).astype(ms.int64)\n",
    "        \n",
    "        output, hidden, attention = model.decoder(inputs_tensor, hidden, encoder_outputs, mask)\n",
    "        attentions[i] = attention\n",
    "        predicted_token = output.argmax(-1).item()\n",
    "        inputs.append(int(predicted_token))\n",
    "        if predicted_token == en_vocab[eos_token]:\n",
    "            break\n",
    "    en_tokens = en_vocab.lookup_tokens(inputs)\n",
    "    return en_tokens, tokens, attentions[:len(en_tokens)-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型可能仅仅是记住了训练集的翻译。因此，我们也应该查看来自验证集和测试集的翻译。\n",
    "\n",
    "首先，让我们从验证集中获取一个例子。我们将从数据集中获取一个源语言句子和对应的目标语言翻译。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Ein Mann mit einem orangefarbenen Hut, der etwas anstarrt.',\n",
       " 'A man in an orange hat starring at something.')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = test_data[0][\"de\"]\n",
    "expected_translation = test_data[0][\"en\"]\n",
    "\n",
    "sentence, expected_translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们将使用 `translate_sentence` 函数来获取我们的预测翻译和注意力分布。\n",
    "它在翻译上下文基本正确。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<sos>',\n",
       " 'a',\n",
       " 'man',\n",
       " 'in',\n",
       " 'an',\n",
       " 'orange',\n",
       " 'hat',\n",
       " 'is',\n",
       " 'something',\n",
       " 'something',\n",
       " '.',\n",
       " '<eos>']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translation, sentence_tokens, attention = translate_sentence(\n",
    "    sentence,\n",
    "    model,\n",
    "    en_nlp,\n",
    "    de_nlp,\n",
    "    en_vocab,\n",
    "    de_vocab,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    ")\n",
    "translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们将创建一个函数，用于显示模型在生成每个目标token时对源句的注意力。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_attention(sentence, translation, attention:Attention):  \n",
    "    fig = plt.figure(figsize=(10,10))\n",
    "    ax = fig.add_subplot(111)\n",
    "    \n",
    "    attention = attention.squeeze(1).numpy()\n",
    "    \n",
    "    cax = ax.matshow(attention, cmap='bone')\n",
    "   \n",
    "    ax.tick_params(labelsize=15)\n",
    "    if isinstance(sentence, str):\n",
    "        tokens = [token.text for token in de_nlp.tokenizer(sentence)]\n",
    "    else:\n",
    "        tokens = [token for token in sentence]\n",
    "    if lower:\n",
    "        tokens = [token.lower() for token in tokens]\n",
    "\n",
    "    x_ticks = [''] + ['<sos>'] + tokens + ['<eos>']\n",
    "    y_ticks = [''] + translation\n",
    "     \n",
    "    ax.set_xticklabels(x_ticks, rotation=45)\n",
    "    ax.set_yticklabels(y_ticks)\n",
    "\n",
    "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "\n",
    "    plt.show()\n",
    "    plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们通过在 x 轴上放置源语言句子，y 轴上放置预测翻译来以图形方式展示这一过程。在两个单词之间的交叉点上，颜色越浅，模型在翻译该目标单词时对源单词的关注越多。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/python-3.7.5/lib/python3.7/site-packages/ipykernel_launcher.py:20: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "/usr/local/python-3.7.5/lib/python3.7/site-packages/ipykernel_launcher.py:21: UserWarning: FixedFormatter should only be used together with FixedLocator\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA5UAAAM2CAYAAABmIKeTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAACfz0lEQVR4nOzdd3QU9dfH8TvphSRUCRBa6EhvAlKlIwhiQxBEEQRBQIr03kEQqYIoKCpNRKodEURAKeoP6T3SSyCV1Pv8wbNjliSUkeymvF/n5Ci7s5s7u5vZ+cy3GaqqAgAAAACABS7OLgAAAAAAkHERKgEAAAAAlhEqAQAAAACWESoBAAAAAJYRKgEAAAAAlhEqAQAAAACWESoBAAAAAJYRKgEAAAAAlhEqAQAAAACWESoBAAAAAJYRKgEAAAAAlhEqAQAAAACWESoBAAAAAJYRKgEAAAAAlhEqAQDI5BITE0VERFWT3QYAwH/l5uwCAABA2klISBBXV1cRETl79qzEx8dL9uzZJVeuXE6uDACQWdBSCQBAJpWYmGgGymnTpkmTJk2katWqUrVqVZk6daocPHjQyRUCADIDQ5P2hQEAAJnOiBEjZNKkSeLn5yclSpSQw4cPy61bt+TJJ5+UoUOHSs2aNZ1dIgAgA6OlEgCATCYhIcH8/3379snSpUulbdu2sm3bNtmzZ49s2LBBnnvuOdmwYYOMGDFCdu7c6cRqAQAZHWMqAQDIRFTV7PIaEREhrq6ucuHCBRk0aJBUrFhRREQaNmwoxYoVEz8/P1m8eLEYhiHjxo2TWrVqObN0AEAGRUslAACZiGEYIiLSq1cv8ff3l08//VRat25tBkZbK2ahQoVkzJgx0rVrV/nxxx9l1KhRtFgCACyhpRIAgEwoNjZWRERmz54tgYGBEhISIgULFjRbMUVE8ufPL2PHjhXDMGTx4sXi6uoqo0aNktq1azurbABABkSoBAAgE/rggw/E19dXZs+eLVeuXJHff/9dChYsaLfEiMjtYDlmzBhxcXGRRYsWSWhoqKxbt04CAwOdWD0AICMhVAIAkAmoqqiquLi4SGxsrHh4eMisWbNE5HZrZZcuXaRYsWJSsWLFFIPlyJEjJSwsTKpXr06gBAA8EJYUAQAgg0pMTBQXl3+nR7h165Z4eXmJqppjK0VE+vfvL7NmzRI/Pz/Ztm1bisFSRCQyMlJ8fX1FRJI9BwAAqSFUAgCQASUNhStWrJCffvpJfv/9dylatKhUrlxZXn31VcmfP7+5/f0GSxECJQDgwRAqAQDIYJK2UA4ZMkSmTZsmrq6ukjdvXomIiJCwsDApVKiQrFq1SmrUqGE+Lmmw3L59u1SoUCHVYAkAwP1iSREAADIYW6B85513ZNq0adK0aVPZsWOHHD58WP755x8ZOHCgnD17Vp599lk5cOCAJCYmiojIzJkzpV+/fhIeHi41atSQffv2ESgBAP8ZoRIAgAzo0KFDsnDhQgkODpbJkydLjRo1JFu2bJItWzbx9fUVNzc3KVmypMTFxYmLi4u5PuXMmTOld+/eEhsbK7t373byXgAAMgNCJQAAGVBISIicOnVKevbsKZUrVzZvHz16tIwdO1aaNm0q06dPN++Li4szt5k9e7b8/PPP0rNnT4fXDQDIfAiVAABkQIcOHZLExETJmTOnedu4ceNk/Pjx0rJlS5k4caIZKENCQmTKlCly7do1c9u6deuKiJhdYwEAsIpQCQBABlSoUCERETlw4ICIiEyYMEHGjBljBsqKFSua23788ccyZ84cCQkJSfY8SZckAQDACjdnFwAAAO7tzjUpK1WqJAEBAbJ27Vo5f/68rFy5Ulq1aiVjx461C5Q//PCDvPfee1KzZk0JCgpyRukAgEyOy5MAAKRDtol1bO5cAaxo0aIycuRIuXDhgqxcuVJq1qwp48aNk8qVK5tdWnfu3CkTJ06UhIQE6datm+TOndth9QMAsg5CJQAA6UzStSOXLFkib7zxhjRr1kwWL14sx44dM7dr0aKFdOrUSdzd3SUyMlKOHj0qIiLh4eGycuVK6d69u/z8888ybtw4adu2rYgkD6cAAPxXhvLtAgBAujR06FCZOnWq3W2NGjWS3r17S5s2bUREZN++fbJkyRJ5//33JSEhQQoVKiQxMTFy6dIlCQgIkIkTJ8obb7whIsm70AIA8DAQKgEASIeWLVsmXbt2lbp168rgwYMlNDRUvv32W1m2bJmULFlSRo4cKe3btxcRkevXr8v+/ftlzpw5cuPGDYmOjpYnn3xSGjRoIPXq1RMRAiUAIO0QKgEASAeSdnkVEenRo4ds3bpVVq9eLeXLlxcRkQsXLsjSpUtl1KhRUrx4cRkzZoy88MIL5mPi4uLE3d1d4uPjxc3t37n4CJQAgLTE7K8AAKQDtkDZt29fKVu2rIiIdO3aVcqXL2+GxHz58knv3r3Fzc1Nhg0bJmPGjBHDMOT5559P9lyqKoZhiAjLhgAA0hahEgCAdGLPnj0yZ84cERHx8/Mzx0ImbXX08/OTHj16iIjIsGHDZPTo0eLi4iLPPvusuLu7i4iYYRIAAEfg0iUAAOlEtWrVZMGCBeLr6yvh4eFy8eJFc2mRpKNVbMFy0qRJcvr0aRk8eLB8+umnziobAJDFESoBAHAyVZX4+HgREXn99dflnXfeERcXF/n444/lo48+EpHbrY8pBcvRo0fLqVOnzLUpAQBwNCbqAQDAwe5n4pzFixdL9+7dRUTkk08+kZdeeklExG6spIhIWFiYHDt2TKpWrZp2BQMAcBeESgAAHCjpLK+//fabnD17Vo4cOSI1a9aUokWLSnBwsLnt/QZLG2Z5BQA4A6ESAAAHSRr6xo0bJ3PmzJFr166JyO3urUWKFJEZM2ZI27ZtzcekFiwJkACA9IJQCQCAg40aNUomTJgglSpVkt69e4unp6fs379fZs6cKSK3g+Srr75qbv/hhx9Kt27dRERkyZIl8vLLLzulbgAAUkKoBADAgTZt2iTPPPOMVK9eXebOnSsVK1Y072vTpo1s2LBBWrVqJR9++KHkzJnT7Cr70UcfyWuvvSYiIgcPHpRSpUqxdAgAIF1gnUoAABzo119/ldjYWBk3bpxdoJwwYYJs2LBB2rRpI0OGDJE8efLYPe7VV1+V6OhoiYuLk9KlSzu6bAAAUkVLJQAADpCQkCCJiYlSu3ZtOXPmjPz999+SO3duMQxDxo0bJ2PGjJGWLVvKxIkTzbC5d+9e8fb2lrJlyyZ7PsZUAgDSC76NAABwAFdXV3F3d5eCBQtKfHy8uLu7i2EYMmbMmBQDpcjtsZUjRoyQyMhIufMaMIESAJBe0P0VAAAHsC0Dkj9/frlx44Z88MEHcuPGDZk8ebK0aNEiWaBcu3atLFy4UIYOHSre3t6MnwQApFt0fwUA4CG6V7fUU6dOSd26deX69ety69Ytad68ucyYMUPKlCljbrNz507p27evXLx4UZYvXy6PP/64I0oHAMAS+s4AAPAfJSYmisjtcZO2QPnXX3/JL7/8Ihs2bJDjx49LQkKCiIjkzZtX3nrrLfH39xdXV1epWrWqXaD85ptvZNCgQbJnzx4ZPXo0gRIAkO7RUgkAgEUhISFSsGBBERGJjY0VDw8PERGZOHGizJs3T65fvy6xsbESFBQkVatWlTlz5khQUJCcP39eFi5cKPPnz5fIyEh5/PHHpWLFinLhwgVZs2aNxMbGyowZM+Stt94SkX+7zgIAkB4RKgEAsGDLli3SuHFjGT9+vAwfPty8fcyYMTJu3DgpUKCAtGrVSv755x/566+/JCQkREqVKiULFiyQBg0ayOXLl2XHjh0yceJE+d///idxcXGSPXt2qVmzprz66qvy7LPPigizvAIA0j9CJQAAFmzevFlatWolIiLTpk2TgQMHysmTJ6VBgwZSrVo1GTt2rJQvX15iYmIkNDRUunXrJps2bZJixYrJ6tWrpVKlSiIicuvWLTl48KDcuHFDChcuLDlz5pQcOXKICIESAJAxECoBAHhAtu6oX3/9tTz55JMiIjJv3jx54oknpHr16vLVV1/JE088ISIi8fHx4ubmJmFhYdKzZ09Zvny5VKtWTbZt2yZeXl6pPjddXgEAGQWhEgCAB2T76jQMw67Fsnnz5nL06FHZs2ePZM+e3QyGthbHqKgoqV+/vuzdu1fmzJkjvXr1IjwCADI8+tQAAPCAEhMTzSDYsmVL+eqrr0Tk9sytV69elb1795rbiYi4uLhIfHy8+Pj4yIABA0RE5PDhwyIiBEoAQIZHqAQA4AEkJiaKq6uriIhMnz5djh8/Lk899ZSsX79eRETCwsJkzZo1IiLi6upqLiVie0zx4sVFROTAgQOiqmbwBAAgoyJUAgDwAGwT54wbN04GDx4svXv3lujoaGnVqpVs2rRJRETef/99mTBhgojcDpNJg+PZs2dFRKRy5cpiGAYtlQCADI9QCQDAfUgaDA8ePCgffvihtGvXTsaOHSve3t6iqtKiRQvZvHmziIiMGjVKhg8fLrdu3ZKEhAQxDEN+/fVXmTFjhoiIOZEPoRIAkNExUQ8AAPeQdDKds2fPym+//Sbt27eXX375RWrWrCki/46zNAxDvvnmG2nZsqWIiNSvX19y5col+fPnl9WrV0toaKhMmTJF+vXr56zdAQDgoSJUAgBwn0aMGCFz586VTp06yd9//y1btmyxC5NJZ4X99ttvpUWLFiIikjNnTmnRooUUKFBAatasKW3bthUR1qEEAGQObs4uAACAjCAuLk6io6MlMTFR5s2bJx4eHnLw4EEpW7asuY0tWKqqNGvWTDZt2iStWrWS69evS8mSJWXkyJHmtgRKAEBmwbcZAAD3wd3dXcaOHStvvfWWFC5cWBISEmT9+vUSGRlpt52tm6xtjKVtuREPDw+77QiUAIDMgu6vAADchW08ZUJCgri6ukpERIRMmzZN5s6dK9myZZMFCxZI8+bNzSVDkj5O5HbIPHPmjBQuXNgZ5QMAkOYIlQAAJHG3bqm2gBkRESHTp0+XGTNmSN68eeX999+XJ554IsVgaesSaxgGXV4BAJkSoRIAgP9na40UEfn555/l4MGDcvbsWalQoYI0a9ZMcubMaW4bGRkp06dPl3feeeeuwRIAgMyOUAkAgNi3UI4dO1amTZsm0dHR5v0NGzaU7t27ywsvvGDedmewXLhwoTRs2JBgCQDIUuiDAwDI8lTVDJQjR46UsWPHSpkyZWTFihWybds2GTZsmGzfvl0mT54s77//vvk4X19fGTRokAwcOFCuXbsmr7zyinzzzTfC9VoAQFbCkiIAgCznhx9+kPPnz0vnzp1F5N8ZW5ctWyazZs2Sli1bysSJE6VixYoiIvLLL7+IiMhff/0lkydPFldXV+nWrZuI/Bss4+PjZdKkSXLlyhXz+QAAyAro/goAyFJ++uknadSokZQoUUK+//57KVSokIiIXLx4Udq3by9nz56Vzz77TGrVqiUxMTEyd+5cGTVqlOTLl0969OghY8eOlbx588rAgQOlR48e5vNGRETIgQMHpGbNms7aNQAAnIKWSgBAlvHXX3/J888/L8WLF5cxY8aYgVLk9iQ9J0+elDfeeENq1aolCQkJsnTpUhk/frzky5dPfv/9d8mRI4ccOHBAPvnkE1mwYIG4uLhI9+7dRUQkW7ZsZqBkllcAGV1cXJy4u7s7uwxkEHzjAQCyjK1bt8q1a9eka9eu8uKLL4qIyKeffipHjhyRAgUKyDfffGN2iT148KDMmTNHAgIC5Pvvv5ccOXKIiMhTTz0lIiKHDh2S0aNHy9y5c5P9HgIlgIwoNDRUYmNjRVXF3d1dwsLCZMCAAXLz5k1nl4Z0jm89AECWceHCBRERc3bWvn37yiuvvCI//vijxMfHS5kyZSR//vwiIrJmzRo5ePCgTJ06VYoWLSoxMTEiIlK1alUpU6aM9O3bVy5duiReXl7O2RkAeIj+/PNPqVChgnz22WdiGIZER0fL448/Lu+++65s2bLF2eUhnaP7KwAgy+jcubOsWLFC5s+fLz/88IN899138swzz8gTTzwhbm63vxJVVQzDkN27d4u7u7uUK1dORMTsBrZ9+3Y5fPiwfPzxx9KtWzcpWbKk0/YHAB6WX375RS5cuCAjRowQwzBk+vTpcurUKZk4caI0b97c2eUhnaOlEgCQJSQkJEiZMmVk2rRpcvnyZfnuu++kWrVqMnLkSCldurQkJibabV+wYEGJi4uTX3/9VURud2ndvXu3zJ8/X6pUqSLBwcFmoLzzsQCQ0XTt2lUWL14ssbGx0q1bNzl27JgMHTpUBg4cKN7e3pKQkODsEpGOESoBAFmCrcvrhg0bJCoqSry9veX8+fPy999/y61bt8TFxcVspRQRadu2rYiI9OjRQ/r16yeDBw+Wjh07yq5du6R79+6SM2dO87kZQwkgI4uLixMvLy9p27atGR49PDykSJEiZi8N2zEUSAnfggCALCMqKkouXLggDRo0kMGDB0tcXJwMGzZMPvvsM4mJiRHDMMwTqpYtW8rHH38s7u7uMnv2bJk+fbpcvXpV5s2bZ65RyapcADK6+Ph4cXd3l+joaFm6dKncvHlTGjduLN7e3tK/f39Zvny5xMbGOrtMpHOsUwkAyBISEhLE1dVVwsPDJSoqSvLmzSsLFy6UkSNHiq+vr4wcOVI6duwonp6eEh8fb46x3LNnjxw9elT8/PykYMGCUqlSJRFh2RAAGZ+td0ZkZKQ89thjUrVqVXnqqaekefPmsmzZMhk1apSoqsydO1eefvpp8fDwEJF/j6eADRP1AAAypTtDn+0EyM/PT/z8/ERE5PnnnxfDMGTEiBEyfvx4EZFkwbJatWpSrVq1uz43AGREtt4ZvXr1kuPHj0uzZs2kRYsW4uPjI127dpWYmBiZOHGi9O7dW0TEDJa24+m2bdukcuXK5jEVWRctlQCATCfpVfTNmzfL//73P9m1a5fUrFlTypYtK61btza3vXHjhqxatUpGjBiRrMWS8AggM7JdNLMd4ypWrChVqlSRuXPniq+vr939c+bMkYkTJ4qqypw5c+S5554TV1dXWbZsmXTv3l1eeOEFWbJkiTkeHVkToRIAkKkkDYIjRoyQd955R+Li4sTFxcUcL9mnTx8ZMmSIBAYGiojIzZs3ZeXKlWawHD16tLz44ovi6enptP0AgLQUGRkpHTp0kLJly8r8+fPlp59+kipVqpjH0KT/nTNnjkyaNEkSEhKkT58+EhoaKitXrpSoqCjZunWrOSwAWRfdXwEAmYotUE6YMEEmTZok9erVk6FDh0q+fPnkzJkzMmXKFJk9e7ZERkbK+PHjJTAwUAICAuSFF14QEZExY8bI8OHDJTo6Wl5//XVaKgFkGkl7caxZs0Y2bNggx44dE09PT4mKirLbNmmwfPPNN8XT01Pmz58vY8aMERGR4OBg+eGHH6Rs2bKO3g2kQ4RKAECms2vXLpk1a5ZUqlRJ3nvvPalYsaKIiFSoUEE2bNggO3fulLNnz9qdRNmCpe1KvJeXF4ESQKZhC5RRUVHyxx9/yJNPPinvvPOODB06VOLi4uTLL7+UOnXq2IXJpP//+uuvS4MGDWTHjh3i7u4ujRo1kgIFCjh7t5BO0P0VAJDpLF68WLp37y5ffPGFtGvXzrx97NixMnbsWGndurWMHDky2QQ8IrfHWP7zzz9Srlw5R5YMAGkuKipKKlasKOfOnZNt27ZJiRIlZOnSpTJo0CBxd3eX+fPny8svvywi9kMJkq7hi4znzvcvMjJSvL29H+qFUy7BItPg+giQvqX0N/qw/25tYyZ37NghIiL58+c37xs3bpyMHTtWWrZsKePGjTMD5Z49e2Tt2rXmdtmzZzcDZWJi4kOtDwAcLelxdtSoURIaGipdu3aVRx99VAICAqRLly4yZcoUiY2NlUmTJsmqVatE5N/uryJCoMzgbO/fP//8I6tXr5ZnnnlG1qxZ81DXH6X7KzK0pGMDDMOQffv2yaVLl6RWrVqSPXt25xYHwGSbSTAhIUGioqLExcVFfH19xTCM/zTD6p1XX23Hg1KlSomISHR0tIjcHic5btw4admypUycONHsDisismrVKlmxYoVUr15dgoKC7J6f7q8AMjLbeVJ8fLyEh4fLvn37pG7duvLuu++Km5ubqKoEBARIt27dJCEhQYYNGyajR48WkdtLLiXt/oqMxfa+xcXFSWxsrMyYMUN+/PFH2b59u4jcvoD65JNPmmuP/leESmRItj8U2wnk6tWr5ddff5X33ntPXF1d5auvvpInn3zSyVUCELl9UuPm5iaRkZHSt29f+d///iceHh5StWpVGT9+vOX1zZKe6ERGRoqvr695W+HChUVEZNq0abJu3TqZPXu2tGzZUiZMmGAXKDdv3izz5s2T5557TnLmzPnfdxZAhmULYJmpq6erq6tER0dLs2bNpHbt2nL27FkZN26cuLm5SVxcnLi7u4vI7fV7e/ToISKSYrDMTK9JVuHi4iLHjx+XhQsXyrfffisHDhyQXLlySZEiReTs2bNSuHBhcXN7eFGQUIkMI+kBzcXFRa5duyY//fSTLFu2TDZs2CAit1srvby8zC5wAJzPNjFEw4YNZc+ePZI9e3a5deuW7NixQ3bv3i2zZ8+WqlWrPvCVcNv2Q4YMkUuXLsmUKVMkb968IiLy4osvymeffSabN2+Wb7/9Vho1aiSzZs2S4sWLm4/ftWuXTJ48WXx9faVDhw7i4+Pz8HYaQIZi601x69YtWbBggYSFhcmQIUMyxbJC27Ztk4MHD8r+/fslMjJSrly5IiJiBkobW7A0DEOGDh0q48ePl5iYGOnUqROBMoP57rvv5IcffpCFCxdKeHi4lC1bViZPnixPPvmkDB06VKKjo+XNN998aK2UIiKiQAZz5MgR/eKLL7R8+fJqGIbmzJlTW7durdOnT9fcuXNrtWrVNCwszNll4j9ISEhwdgl4SBITE1VVdcKECZo7d24dOnSohoSE6MGDB/W5555TwzC0QoUK+ssvv9z3+550u0uXLmmxYsXUzc1N+/btqxcuXDDv+/XXX7VWrVpqGIa2atVKL126ZN731VdfaY0aNdQwDF2wYMFD2lsAGVF8fLyqqkZERGirVq3U1dVV/fz8dM+ePeYxLCOLj4/X5cuXm8fDxx57TI8dO5bq9mFhYTpz5kw1DENr1KjBOVUGERsbqxs2bNBXX31VDcNQwzC0YcOGOm3aNI2JiVFV1bVr16phGDp8+HBVfbjnW4RKZAjx8fF65swZnTp1qhYtWlRdXV01Z86c2rlzZz169Kiqqs6bN08Nw9CvvvpKVQkmGZHtPQsPD3dyJXjY2rZtq40bN9bIyEjztsuXL2v//v3V3d1dy5cvf1/B0nbyp6q6d+9e/eyzz3TgwIHq6uqq+fLl0zfffNMMllFRUbp582Z9/PHH1TAMzZEjhzZu3NgMk15eXvree++Zz8cxA8h6bH/3kZGRWqVKFfXy8tLu3bvrtWvXnFzZw2E7ZsbHx+unn36qVatWVU9PTx03bpxevnw51cfdvHlT58yZowcPHnRUqbAoISFBt23bpnXq1FHDMNTHx0efeOIJXblypUZFRZnbnT9/Xhs0aKCGYejWrVsfeh2ESqRrCQkJGhISot26ddMyZcqoYRgaEBCgEyZM0N27d5vb7d27V/38/LR06dJ68uRJJ1YMqyIjI7Vfv37atm1brVWrli5evFjPnTvn7LJgQdLgZ1O/fn1dtGiReb/tRO7atWs6YMCA+wqWSW8fPny45smTRz09PbVhw4bq5+enLi4u6uPjYxcsY2Nj9Z9//tE+ffroo48+qnnz5tVixYpp9+7dddOmTSk+N4CsJTY2Vjt27Kiurq46atSoe17YTK+tl/dzUe7zzz/XsmXLqr+/v86cOVOvXr2a6vbpdT+R3Oeff66lS5fWZs2a6Y4dO/TixYvmfbbPxY4dO9TT01PfeOONNKmBUIl07ezZs9q4cWM1DENLlCihs2bN0v379yfbbu7cuWoYhs6ZM8fxReI/i4iI0GrVqplX2GzdNp599lndtm2bs8vDA7AFyqioKB03bpx27txZR48erdWrV9chQ4bYbWM7Ybl27ZoOHDjQDJY7duy468nRlClT1DAMbdu2rf7++++qqnr69GmdMWOGli1bVt3c3OyCpc3Vq1c1JCREr1+/bhd8CZRA1nb48GHNlSuX1q9f3+wmqHr7GPX777/rjBkzdPDgwbp8+XKz5Se9Ba64uDhVvR2Qd+zYocuXL9cvv/xSL126ZFerrStsmTJl1N/fX2fMmHHXYImM49ixY3afX9V/P6eRkZHatGlTdXV1NS+oPuzPMKES6d727dt18eLFya4c2k4Er1y5okWLFtUyZcroP//8o6rp72CP1CUkJGjXrl3Vz89P+/TpoyEhIbp69Wp99tln1cXFRRs2bKg//vijs8vEfbD93UVEROhjjz1mXhyw/VSpUsU8IUspWA4YMEB9fHw0KChId+3aleLvuHjxopYuXVoDAwP1wIEDds8VFRWl33//vZYvX149PDySBcukxwWCJACbr7/+Wg3D0JEjR5q3xcXFab9+/bRgwYLmMSxbtmzat29fjY6OdmK1ySUdE/rMM89otmzZzJqrVaums2bNMkOnbfukwfJeLZZIv1I63036/Wb7/23btqm3t7c2adIkzb7/CJXIEGx/NElbFxITEzUhIUGnTZumhmHoW2+95azyYJHtfS1fvrx26NDBbrzdoUOHtF+/furq6kqwzABs72ViYqJ26dJF/fz89I033tDvvvtOFy5cqMWLF1fDMPTpp5/WW7duqWrKwbJHjx76yCOP6IkTJ1L8PX///bc58Y7q7avySb9UY2Njde3atZovXz7NlSuX9u7d2+wGxMUmACnZtWuX+vr6alBQkK5du1Znz56tlStXVsMwtFSpUjp79mydOnWqFi5cWPPnz29e0EoPbAEhIiJCq1atqoZh6BNPPKHz5s3TsWPHap48eTRnzpz61ltvpRosc+XKpRMmTMg040izgpiYGP3iiy9006ZNevPmzXtu37VrV/Xw8NAtW7aoatpcWCVUIt2Jj4/XU6dO6Y0bN1LdJunJYYMGDbRQoULmgHNOHDOGsLAwrVGjhk6ePFmrV6+uf//9t6reDgU2p0+f1rfeektdXV21QYMGBMt0KulEEDExMVq3bl195ZVX7CYIOHXqlJYsWVINw9AXXnjB7KJzZ7C8fv263Sytdzp58qR6eXlprVq1zNvu/Ju/efOmNm3aVA3D0Fy5cumgQYM4WQKQTNJjR8+ePdUwDHVxcVHDMLRkyZI6dOhQu8lsbNusWbPGGeWmKiYmRp9//nn19vbWESNGmD27jh8/rs2aNTMnJhswYECyYLly5UoNDAzUQoUKcZzMICIiIrRdu3bmTL4HDx6867nvDz/8oO7u7lqlShWzR19aIFQiXYmKitLevXubVwbvdfVlwYIFahiG9ujRQ2/dukWXtgxk2bJlahiG5smTR318fPSrr77SxMTEZAfGU6dO2QVL21U2pC+RkZHasmVLHTVqlObOnducWS4hIcH8u7x8+fJ9Bcu7CQsL06JFi6phGLpkyRK7FlLVf8cVLV68WEuUKKFlypRRPz8//eCDD8zeDQCynpQmELvTvHnzdOjQoTp69Gg9ceKE2avC5qmnntLAwMB0NyHgihUrNCAgQDt37mwGyiNHjuhLL72khmFomzZttFChQmavrqQXb+Pj43XNmjV6/PhxZ5WPBxAREaHVq1dXLy8v7dy5c6q9epIaPny4GoahK1euTNPaCJVINyIiIrRevXrq5eWlFStW1D179tgd+O507do1bdKkibq7u6c6/grp2zvvvGOO+5gyZYp5e2rB0tvbWytXrqw///yzo0vFPaxevVoNw9CiRYtqrly57EKl6r9h79KlS3bB0vY3fj8T59i2WbhwoXp6emrjxo11x44d5v1JjxeDBw/WUqVK6erVq7VAgQJavXp1AiWQRdmOP7du3dJPP/1UBw8erIMGDdLVq1ffdabXpMeM+fPnq6enp7Zp0+auPakc4c7vyJ49e6qbm5t5If706dPatWtXNQxDu3fvrqqq69ev14CAAPX399c+ffrYtVgiY4iJidGnn35aXVxcdOTIkXa9ge5ku4h67do1LVu2rFavXl3Pnz+fpvURKpEuREVFaZ06ddTLy0sHDRp0X7Or/frrr+rp6WkeMJExJCYm2n2Z2RZYNgxDv/jiC7vtkjp9+rR2795dc+fOradPn3ZYvbg/169f14ULF2qRIkXUMAzt1KmTOUb2zlbEpMGyRYsWya6a2xw8eFAPHjyYbPzS4cOH9ZlnnlHDMLRly5b63Xff2d2/c+dOrVChgnbt2lUjIiL0+eefV8Mw7JYQAZA1JJ3EpkmTJskmEGvevLleuXJFVTXVC9nTp0/X/Pnza1BQkB47dsxhtafE1noaFxen169fV1XV0NBQXblypSYmJmpMTIz5vdq5c2fzcWfOnNGKFSuqYRjq7++v3bt3J1hmMN99951my5ZN27Rpk+y9u3jxoq5Zs0Y3btyohw4dUtV/v3vnzp2rc+fOTfP6CJVwusTERB01apQahqF9+/ZN8arhlStXNCwszO62LVu2aOnSpXXVqlWOKhUPUdJpr2fNmqWGYaivr69++eWX5u13BsuzZ8/arb2E9OX69ev6/vvva1BQkObMmVMXLlxongDdGSwvX76suXPnVsMwzDEeSVsFJk6cqAULFtTs2bOrl5eXDh8+3G45oV9++UVbtGihLi4uWqRIEX399dd106ZNumDBAq1QoYIahqEfffSRqqpOmjQp2UULAJmf7ZgSGRmpNWvWVMMw9LnnntOtW7fqH3/8obVq1TLHT9rGctuOUTdv3tTVq1drixYt1N3dXUuWLGmO/Xe0o0eP6i+//GL++8aNG9qwYUNdsGCBec6UdFx6yZIltWrVqhoREaGq/wbRbt26acOGDdXb21t9fX3vOn4d6c/kyZPVMAxzzWfV2+/7mDFjzImlDMPQHDly6Pbt21N8jrScd4RQiXShcePGGhgYaDcgPjExUT/55BN96aWXNE+ePFqjRg39/PPP7R5nuxqD9C06Olrff/99nTp1qk6dOjXFyQBmzJhhrlN5t2AJ57rXuKSrV6/q+++/r3ny5NHg4GD96KOPzAsIdwbLK1eupDiOZ+TIkeYkO02bNjWnx2/WrJldq+Tvv/+u/fv31xw5cti1PLi6uurMmTPN7Vq3bq0+Pj4prnELIHOLjY3VV199VX19fXXYsGFmCLt27Zp26NBBXV1dzVlebecgtjD64osvqpeXl3br1k1PnTrllPrPnDmjvr6++thjj+n//vc/vXXrltniOHv27GTd+r/44guzlTIhIcHuAm758uW1X79+euDAAae3uOLBLV++XF1cXLRbt276yy+/6Ndff21eGMmXL5/26NFDn332WXNtd0dfhCdUwukiIiK0SJEimjdvXv3777/15s2beuHCBXNmKx8fH7t1otauXZvsOQge6VdERITWrl3b7qS/ePHiumLFimQTMREs07ek45LWrVunY8eO1bFjx+rSpUv12rVr5snN1atXdcGCBZo7d24tWrRoisEyaThNOlHPgQMHtGDBgtq2bVvdt2+fqqru2LFDu3Xrpm5ublq3bl395ptvzMeGhYXpX3/9paNHj9Y33nhDZ82apV9//bV5v60bWMOGDc2uYgCyju3bt2vu3Lm1bdu2Zsvd0aNHtWPHjmoYhnbt2lWfe+45NQxDS5cubXdxW1V1z549dx13mdZOnDihnTt3Vk9PT33iiSe0aNGi6unpqVOnTjX3J6nffvtNs2XLpm3btrW7fcGCBZozZ067Vi6kf0nPf/78809t0KCBGoahbm5uahiGBgYGas+ePc0u3Kq3G2q8vLz08OHDDq2VUAmnsv2xvP322+ri4qKPP/64NmnSRAsUKKCGYWiTJk30zz//1NDQUH333XfNmV4ZB5AxxMTEmF2HXnjhBV29erW2b99eAwICNE+ePDp37txkJ/q2YBkQEKArVqxwUuW4U9JxSY0aNUo2Lunxxx/XJUuWmOHx+vXr9wyWdzp16pQeOnRIfX197bp6qd5eSuTtt99OMVimZvr06frII49o7ty5Hf7lCiB9+Pjjj7VQoUIaEhKiqqohISHao0cPNQxDX3vtNVW93TJp6zZfokSJdNct9Pr169qtWzd1cXFRV1dX7datW7JJ0GxOnTqlNWrUMCfpWb16tb711luaM2dODQ4O1jNnzjhjF2DRnd+X27Zt06FDh2q9evX0zTff1N9//91ujW9V1Ro1amihQoU0NDTUgZUSKpFO7N+/X7t06aKPPPKIGoahjRs31iVLltj9QRw+fFgNw9AuXbo4r1A8kJCQEM2VK5e+/fbb5uRLt27d0vnz52uxYsU0Z86cKQZL2wWE/Pnza3h4OC2V6URUVJTWrFlTXV1dtUuXLrpv3z7duHGjTpgwQXPkyKGFCxfWuXPnmgH0xo0bZrAsWbKkLliwwK4rVlK9evVSwzD0+eef19q1a5u3J+3adebMGR00aJAZLJN2hU3a2nnu3DmtWbOmenp6avHixdPVQuUA0o4tYN35nfHDDz+Y4wqXLVumnp6e+uKLL5r3JyYmav/+/c01KnPmzGnX8uNs8fHxWr58eXMdzcaNG+tvv/1m3p9S8LBNmmb7KVGihB48eNDRpcOCW7du6ZdffqmjR4/WN998Uz/66CPdvXu33TapTSo1e/Zs9fT01C5duiRbEietESrhUPHx8Xru3DndvXu37t271+7qyrlz5/T48eP6448/pjj1/5AhQ+wm30D6FR4ermPHjtUlS5Zo8eLFzS46tkARExOjn3zyiRYvXjzVYDl37lz966+/HF47UjdixAg1DEMHDBig0dHR5u3Xr1/XokWLau7cuXXatGl2X2ShoaG6cOFCNQxDK1WqlOpU/FOmTFHDMNTT01ODg4P1woULqpp8eZGkwbJBgwYptliePHlS27Vrp6+++up9reEFZGVJ/8YywwW88PBwbdKkSYq9E2JjY7VatWpauHBhc4Iw2/Fq8eLF+uijj2q9evXUMAynr9uY9L0IDQ3VTp066eDBg/X1119XFxcXbdKkid3yWrbtbe/nvn37dM6cOfrmm2/qokWLaKHMIMLDw1PsDVSwYEEdM2aMuV1KPX9mz56t+fLl0yJFijhlLVVCJRwmMjJSu3TpYnf1rGHDhjpmzJgUv8iSdulYtGiR5smTR6tXr67nzp1zZNmwoH379moYhtauXVtLliypoaGh5lU123sdGxt7z2CZESX9LN/PYtsZSb169TQ4ONiuB0FCQoLWqlVLXV1dddiwYeZ9Sf9+r127ph999JEeOXLkrs8/e/Zs89gwb9488/Y7jw9nzpwxLzJVrFgxxZOlqKiou67hhbtL+prf2b2O9T4zj6TjpG0hK6OzrX/csWNH84Km7fN85MgR9fPz00qVKiX7vmnTpo0++eSTqqppvp7fvdi+O+Lj4833xXYh7/z589qzZ091cXHRpk2b2gVLhgZlbFFRUfrYY4+ZLY07duzQxYsXa/fu3c0xlG+++abdY8LDw/XAgQP6zDPPqLe3twYHBzttlmJCJRwiIiJCq1SpooZhaLVq1fTVV1/V8uXLa0BAgBqGoe3atUs2M6Tq7YPooEGDNE+ePBoYGEjXjQzi0KFD5vTW/v7+5hXfpF0UVe2DZd68eXX69OkOHwPwsNj2zXb10HYCEBERYReQMqoLFy5o9uzZtVmzZuZtCQkJ5jT9I0aMMJf9iYmJ0e3bt9uNS7pbC0jS8D1v3jwzWK5cuTLVx58+fVp79eql77333n/eN/zL9l7YPr+2/4aHh9tdJUfGZ7s4YJtMrWHDhpniom1MTIxWq1ZN/f39dfXq1ar67/EjLCxMH330US1QoIBd99GFCxdqYGCg9u7d2257Z7CdA0VFRelbb72lxYoV0y+++MKupuPHj2uvXr1SDJaqqqtXr9alS5fq1atXHVo7rEu6vF7S9dpVbx+XN27cqO7u7moYho4aNcq8b+/evdqgQQP19fXVZ5991qm9cwiVSHOxsbFmy9XYsWPNK4chISG6efNmc2bXtm3bmgfN6Oho/fzzz7VSpUpqGIbWqlWLiTYymGPHjpmTBTRq1Mi8Kmw7kUkaLD/99FPNmTOnFi1aNEO3Vu7YsUNr1aplXiUMDQ3VRx99VPPnz5/qmlEZxeXLlzVXrlxarlw5Vb194pNSoFS9/ff7yCOP6OTJk+2e424tXEnHWs6dO9fsCnu3YJl0RsbM0G0vvdi5c6fmz59f9+7dq6q3P8flypVTV1dXu5l1kfHFxcXpCy+8oF5eXvr222/bdWvPCO7sDWLryrpp0ybNnj27Nm/e3LwvLi5O4+PjdeLEieru7q7ly5fXbt266QsvvKAeHh6aP39+p3d5te1PeHi4NmzYUD09PbVMmTL6zTffJDt+Jg2WzZo1061bt6rq7YmJsmfPrqVKlUpx+S6kXy1bttQ8efKY79udPby+/vprdXNz09y5c5vvt6rqTz/9pD/88IPTL8oTKpHmDh48qPnz59d69eqZB8WkB8dDhw6ZwXLYsGHm7Tt27NCXX35Z33nnHad3RcH9uXNMx7Fjx7RatWrmBEu2A96dwTImJkZXrFiR4dfNeuGFF8xuv3/++adWqlRJ3d3ddeLEiQ4fMJ8WXn75ZfX09NQvv/xS69Spo4Zh6PDhw+0Cpapqv3791DAMs5VA1f7k74cfftB58+bp22+/rQsXLjTHWSY9LsyZMyfFYEnXy7Q3YMAAc6r63bt3a6VKldTT01MnTZqU4UIHkks6wUdkZKS5JIGtZSSj/I3ZjilRUVH67bff2t134cIFffHFF9UwDJ04caLdfSEhIfrmm29qUFCQuYRVtWrVnL7ute37MCoqSh9//HH19PTUt956K9nkZkl7cx09elTffPNNdXV11eDgYG3UqJH6+Pho7ty59Y8//nBo/bAuMTFRb968qQUKFNACBQrolStXkl0wsX0+Bg4cqIZhpMteOoRKpLlVq1apYRjap08fVbVvkbB9eW3ZskVz5MihNWrUsLuyFh4enqnGCKQ2xi6jfImnxHagu379eorjCY8dO2a2OL/88supBsuMLOk+vPTSS2oYhnp7e6u7u7u+9957Zut8en+f7/X5XLJkiRqGoa6ururm5qZTpkxJ1rL8/vvv6yOPPKLNmzc3Z09Mut+jR49WX19fuwkISpUqpTt27Ej2t540WCYNqEh7w4YNM9dCc3d31zlz5pgtw+n9c4x7CwsL0w4dOuhHH32kJUuWNCfGymjjwKOiosyhFoMHD9aTJ0+ax5Fff/1VPT09NW/evHatOqq3x3n//fffumDBAv3555/N/Xe2xMREHTRokNkF8s6lIlQ12frOp06d0ilTpqiLi4tmz55da9So4fSAjAeTmJiocXFxWrduXTUM4649Qmzn1C+88ILGxsamq3MoQiXS3M8//6zu7u7asWNH87Y7/wguX75szrh258E/s7B90UVHR+uaNWt0zpw5um7dOnMMS0Y8UYuKitJBgwZpvXr1NEeOHPrkk0/qBx98YN5ve59TC5YZ7QTmTitWrDC/4G1X/2NiYtTDw0Pd3Nw0f/78ZlfY1JbSSC/u9fm06dmzpxqGoe7u7rpu3Tq7+yZMmKC5c+fWoKAgPXr0qKra/62PHDlSDcPQKlWq6CeffKL79+/XN954Qw3D0KJFi+q6deuSTZNuC5aGYeiyZcvSYteRRNKJW2wXRnx8fMyusOn9c4z7079/fzUMQ4ODg9XPz0937drl7JIsWblypRqGoR4eHuax5Z133jEvgNgm/+rdu7cmJCRoQkJCujoJv9OtW7f0scce06CgIL18+bLdfatWrdLu3btrvnz59KWXXkq2jvOJEyf06NGjdHnNQBITE+0+j5MmTVLDMLRZs2bJLgzYjs2nTp2yW2M1PSFUIs398ccf5uDizZs3m7ff2VVy6NChybbJLJJOiGDrNmj7KVasmN2EARlFRESEOWayYMGCWrVqVfX29lbDMHTgwIHJJuVJGiy7du2a4b/4PvjgAzUMQ5955hnzYB8bG6szZsxQHx8fLVmypDkxlS1gpdcLBw/6+Xz99dfVMAz18vLSpk2bavv27c1uzkWLFk1x5rlPPvlE/fz8tEWLFrp//37z9nfeeUd9fHzMdUm/+uqrZMFl1qxZahiGzpkz5+HvPJKJj4/XZcuWqa+vr1atWtVct8+2xE9GvxiE29q2bWsGslWrVqlqxnxvn3vuOfN7p0KFCmoYhtaoUUP/+OMP/d///qdPPvmkGoah33//vaqm794xly9f1kKFCumjjz6qqrffj6tXr+rzzz+vbm5u6ubmZh4vCxUqpBs3blTV9L1PuH/nz5/XRo0aqbu7u/bs2dOcMT3p36VtMp8FCxaoavp67wmVeKhiY2P1m2++0W3bttndbvsjaNWqld0JatKTx6efflqzZcuWabtt3Lp1Sxs3bqwuLi7atm1bnTRpkrZu3dqcITUjhemoqCht1KiRenp6ap8+fcwxcStWrDBn9O3du3eyQebHjh3T6tWrq2EY+sYbb6TbkHU/9u7dqzVr1tRJkybZ3X7s2DH95ZdfNCEhwTzZqVy5sjkjW3rt9vugn8/Zs2frk08+qa6ururq6qply5bVN998U0+dOpXsuS9duqRNmjTRoKAgcwHn2NhYnTp1qvr4+GixYsV02LBh6uHhoYULF9Z169YlG4P6559/ptm+49+r4LbWnOjoaN2/f7/Gx8ebYyxz5MihBw4cUNV/T3Lu/BvOyH/TWUXS79127dqpYRiaO3duc0x7en0P7+webztGHD9+XAsXLqzt2rXT+Ph4HTNmjObLl09z5MihQ4cO1TfeeEN9fX21fPnyTlm770FERESYYb9p06b6/PPPm3NONGjQQPfs2aN79+7VYcOGqYuLS7Lxokj/oqOjdd68edqjRw9t3bq1jhkzxuwpkJCQoGvXrtUKFSqoh4eHtmnTRrds2WKeS82ePVsDAwO1RIkSGhIS4szdSBGhEg9NZGSkduzYUbNly6a1a9e2m9b4zz//1BYtWqiLi4s+9dRT5hVDmw8++EB9fHy0UaNGqS6OnhEl/RI8fPiw5s6dW4cMGWJ31WnkyJHq5uamfn5+umnTJmeU+UASEhLMcXH9+/c3J2k5dOiQdu3a1WzVMAxD+/btm2yB3sOHD2u9evXMk9OMyLYvtunaw8PDddCgQWaXq5RO2ipVqmTOLJj0fmeGpf/6+UxMTNTTp0/riRMnNCYmxny+O1s7Tp06pdWrVzcnFoiLi9OFCxdqQECABgcHm+OGnn322VS7wt7ZswEPX0REhD7//PO6du3aZCfwffr0UcMwNHv27Pq///1PVe0/xxm1+2Rml9rfS9KLWraLXxUqVDCPUen17yw8PFyXLFlid1tkZKSOGzfObn3b/fv3a8eOHdXDw0NLly6tLi4umitXLp04cWK6mGzqbq/v33//rTVr1lR/f38zTC5YsMBuZs+vv/463XaBROrCw8PNcZM+Pj7q5+dn9gqaOXOm3rp1SxMSEvTLL7/UBg0amPdVqFDB7P2UdFhNekOoxEMRERGh1atXVy8vL23durWeOnUq2UnJ999/r82aNTMnMenVq5dOmTJFO3XqpL6+vponT55MsWzInSfUERER+sEHH+iUKVM0MDDQnGEv6RfblClT1N3dXf38/MzuLOnV+fPntUqVKlqlShUzUB4/flw7depkLsy7f/9+zZEjhxqGof369UsWLO8cN5eRJSYmmmH6xRdfNINl0n1MGixtXWFVb3cJLV++fLLWzrSUlp/PxMREu5OlL774wrx4sGXLFjOEHzhwQMuVK6cFCxbUM2fOmNvv27dPvb291cvLS7Nnz64rV65Mdy26mdmCBQvMcWmbN29O9neaNFgmvRiybNkyDQwMTLYoN5zL9h1sWzd22bJlumTJEv3777/NycNsbMeoypUrp7tgmXTWeNt6108//bT+8MMPdt9BpUqV0pIlS5rdtFVvX7Bu1KiRefLeoEGDFCe/caSkwyX+97//6a5du5ItZRISEqK//fabbty4UePi4pK9F/379082MzbSt+joaK1bt666ublpz5499cSJE3r8+HFdtmyZFi5cWA3D0HfeecfsLXL69GkdPXq0li1bVgMCArRSpUratWtXp65DeS+ESvxnt27d0ubNm6u3t7eOGjXK/LJK6WRwz5495sxmth9/f3+tXbt2hu/2mtI6hAkJCWbrS9u2bbVBgwaq+u8V/qRfFJMnT1Z3d3fNli1bug6WFy5c0I4dO+qOHTtU9fYYENsU1127djW3W7x4sfke9+7dO1NP8LF//3594oknzBnZ7hYsS5curbt379bp06dr/vz51d/fXw8ePJjmNTr682kbIz148OBk7/2YMWPUMAz99NNPVfXf1+n48eOaI0cO8wJF0kmfkPbOnz+vY8eOVT8/P61YseJdg2WOHDn0yy+/1NGjR2v+/Pk1e/bsdifzcC7bxaOIiAht1aqVOQ7PMAzNlSuXPvPMM/rPP//YPSalXhXOHmNp6+IaExOjN2/e1J07d2qbNm3U29tbAwMD9bXXXjO7AX777bfq5uamAwYMsHuOAwcO6Ny5czU4ONjpy2wkfV+aNWtmht2CBQvqkCFDUn1c0r/DRYsWaa5cufSxxx5LN7PW4t5scwP06NHDbo1lVdXSpUurn5+fvv322+aFXZubN2/qpUuXNCEhId2fRxEq8Z8tX75cDcPQjh07Jrv6GRoaqj///LP++OOPdieov/76q65Zs0bfffdd3bFjR7JZzjKa0aNHq2EYyRZ7j4uL09WrV5tTnvv4+Ni1VKmmfOKeM2dO/fLLLx1S+/1KepHg0qVL5qyne/bs0Vy5culTTz1l3h8TE6N79+5Vf39/LVu2rBqGkeyLPqNK+jokfe/+97//af369e8aLDt06GB3QaVQoUJmV8K05IjPZ9Ltdu3apXnz5tVWrVqZs4Ym3e7ZZ59Vd3d33blzp6r++5quXLnSHLfn7JO/rOrixYs6evRozZYtW6rB8u233zYnXzMMQ4sUKZKhu7NnNra/p8jISK1evbq6uLjo008/ratWrdL33ntPmzRpYgaZO8dl2YJl9erVzUlCHO2vv/7Szz//3Px3aGio1qhRQ/v27auqty9+fPbZZ+bEbwUKFNBPP/1Ujxw5om+99Za6uLjoN998k+x500O3V9XbcxLUqlXLfJ3btGljhv6XXnopWeBIaujQoZonTx7Nly9fhr8Qn9W88MILmj17drtluGJjY83PwsiRI80uzknDY9ILO+m95w6hEv+ZbZmApBPwJCYm6pQpU/Txxx83TzxeeOEF3bdvnxMrTTuff/65uri4pLgYbUxMjK5fv958Lbp165bs6mLSE/Jp06apYRgaFBSULKQ7Q1xcnIaHh+v58+dTvL9Hjx5qGIa5+LTti/vGjRtaqFAhHTlypNarVy9TtGLYui2l9pr8+eef9wyWM2fO1H79+umAAQNSnNQmLaT15zPpF11ISIguX75cc+TIkep40c6dO6thGDphwgTzNdqxY4fWqVNHH330UXN9yzt/Nx6OO4cm3NkadT/BcuXKlTpx4kQdP368XRdmpA/x8fH65ptvqmEY+vbbb9t9l8THx2uuXLnUx8dHJ06cqImJiXYnsbYxlvXq1XP4OniXL1/W7Nmza86cOXX9+vUaHR2tFStWVMMwdPr06Xafw+joaO3Vq5fmypVLvby8tF27dtqvXz8tV66cNm7cOF1NypP0b+yzzz5Tf39/HT16tNkSu3fvXrNrb9LvDtv6hfPnz9fg4GAziBIoM474+Hi9du2alihRQoODg80L8gkJCWagHDFihNmVW/X2pIe23mAZCaESDywxMdHuip8tVAwYMECPHTumu3btMk+s8+XLp+3btzen+e7Ro4cTK09btn7uERER+vHHH9vdd+vWLV2/fr1WqlRJs2XLpuPGjUvWOpv05Pndd991SJfIe4mIiNCOHTtqlSpVtHDhwtqnTx/966+/7Mbq2Frf1q5da/fY4cOHa0BAgIaFhTm9C9XDkLTb0p2vyZ9//mmelKUWLO88kXd0WHLE53Po0KFaqVIlfeaZZ7RmzZqqar/ftpPT/fv3a3BwsObKlUufffZZHT58uBYqVEgNw9BFixY9nB3GXd35Objz83jx4kUdM2aM+vr6apUqVXTTpk0pdr1K71fOM7uDBw8mmylZ9XaXuQoVKmiFChXsgtitW7fMiUJGjBhhtprc+T526tTJKb0FIiIidPLkyern56dFixbVoKAg9fT01GnTptldxEr6nbJ582bt2LGjOV+Di4uL5suXTz/66COH158S22sbHR2tf/zxhw4YMEArV65s7oPt7+qvv/4yJ2dJ+t2hevuY+fjjj+vYsWOTrR2M9OfO82RV1RYtWmj27NnNf9esWTPFQKmqWq5cOX3uuecy3PwThEo8sP379+sXX3xh9vs+evSoeQUtMDDQnGSjc+fO5pgN28Qt/v7+evz48Ux7IhIfH28eKGbMmGF3X0xMjG7YsEHLly+vfn5+KZ64p6fwFRkZqY899pjZvcjT09OcxOPzzz83T2Rsk3u88sor5gyQs2bN0oIFC2r9+vXtZqzL6O72mnz66afml0hqwTLpibsz/gbS8vMZGhqq3bt3V8Mw1NfXVytWrGjed+e+RkZG6meffaZlypRRwzDUzc1NH3nkEXPdrZQeg4cnISEhxc/BncHy/Pnz5kXDunXr6saNG5MtE8T75DwjR47UwMBAXb9+vV3gT0xM1F27diW7kJtay8itW7f0559/NltQ0oNp06api4uLurq66nPPPWcee5KeZN/5ef3oo4+0TJky5rJWdevWTTdj0GJiYrRq1apatWpVbdSokfbq1UtV/+3ZY/s7unMYRdKwERYWlm72B3dnO09OOklh79691TAM7dmzp7nG97Bhw5IFysGDB6urq2u6uSjyIAiVeCDx8fFarVo1DQoK0hMnTpgHwt27d2v9+vX10Ucf1Zdeekl//PHHZOMCihcvro8//niyFpvMZuXKlerv769eXl46ffp0u/vu58Q9vZg9e7bmzp1bhw0bppGRkfr333/rsGHDNCAgQIsVK6ZLlizRmJgYPXv2rLm4tL+/vzntdWBgYLpobX2Y7vWafPTRRykGy44dO6abE7a0/HyeOnVKhw0bpt7e3moYhs6dO9e8787wERcXp+fOndNZs2bphg0b9Pfffzfvo8tr2kvtc3Dna3/s2DHzJP2xxx5LcbkROF5ERIQOHjzY7KK8YcMGu8Bx7Ngx9fHx0Weffda8LbWWkfPnz2udOnX0s88+c+g+pCY+Pt6c+MzNzU2Dg4N11apV5v13Oz78/vvvOnLkSC1btqzu37/fAdXen/Pnz2u3bt3Uy8tLDcPQli1bmvfdeYEmabDs2LFjplpmLStI7Tz5/PnzWrx4cTUMQz08PHTw4MHJPsuLFi3SvHnzat26dfXixYvOKP8/IVTivtj+KL7++mt1c3PTunXrJutyEx8fbzcAOal33nlHDcPQt956y+FjNJxh3bp16uPjo56envc8cZ84cWK6PHi0b99ea9SoYTcT2eXLl3Xu3LmaK1cuDQ4O1k8++URVb3fb6devn/r6+mq5cuW0Xbt2yaZIzwzu9ZqkFCwbNWqkhmHoq6++mm4+9//183m3k7pjx47pkCFD1MPDQytXrqxff/21ed/97H96eY0yk9Re09Q+B7b31xZSmjdvbp7k161bN12M9cbtY8+ECRPU399fy5UrZxcsz507pwUKFNCgoCD98ccfzUA5fPjwZC0jr7zyinp4eKQ4Q7QzREVF6fjx43X69Ok6Y8YM9fT01ODgYF22bJm5zZ3HoKSf8ejo6HT5GT19+rR50c3d3d2c/Vo15WCZHr87kLq7nScnJCRoYmKifv755+ZQD9vEU/Hx8RoXF6cjRozQnDlzar58+Zw2SdZ/RajEA+nZs6d6e3vrzz//rKr/HtjvPOAl/ff8+fM1f/78WqpUKYdNTOIId3YFvPPq/ZdffnnPE3fbrJvTpk1LN60ztv145ZVXdOHChapqPxNZaGiozp8/X3PlyqVFihTRZcuWma/FmTNnNCYmJl1+oT+opO/vg7wmdwbLvXv3aqtWrRwyy2tSafX5TPq8Bw4c0O+++05XrVplt8zIhQsXtH///urm5qb16tXT77//3ryPk6O0l/Q1tr3v8fHxGhUVZa4VarN27doUPwdJxwOVKFFC+/Tpo1OmTGGCkHTm0qVLOn78ePXz89OyZcvq+vXrzXUYbV1IAwIC1M3NTUeMGJFsjcb58+drjhw59Omnn3baUIV7DfuYOXOmenp6arFixeyCZdKusHcGZWe6n4tu7u7uWrt27RSPjbb//vHHH0757sB/k9p5surtc4WlS5dqkSJFzAnvqlevrgULFlTDMLRUqVL6999/O6v0/4xQifu2ZcsWNQxDa9SokWx9qztFRUXp33//rV26dFF/f38tUKBAhv5DuZPtRC06Olrnz5+vXbt21WeeeUb79u2rO3fuNLs6rlmz5q4n7mvWrNHatWvr4cOHHb4Pd4qMjNRRo0bp888/rwMHDtTg4GB98cUXUxzDkTREFS1aVD/++GO7k9CMHhxsJzlWX5NixYrp0qVLzRZNR4+DSavP553Li9iuuNp+mjRposuXL9fo6Gi9du2avvXWW+rq6kqwdIKEhAS7z3GPHj20Tp06WrlyZe3bt6+eO3fOvD9psJwwYYLd88yYMUOzZctmLv+C9CHpRaJr167psGHD1M/PTx977DFdt26dxsXF6eHDh7Vly5bq5uamZcuW1R9//NHuOaZNm6Z58+bVokWLOq1nSUrHqqefflr79u2ru3btMsPiu+++m2KwVL29rNnIkSP12LFjDq//Trb9iY2N1d9++00///xz/fHHH+0uqB8/flwHDhx414tutv8yhjJjuZ/z5NjYWD169Ki++OKLWr16dc2fP782atRIx48fr2fPnnVwxQ8XoRL3ZDvxGD58uBqGcc9xF4mJibp3716zu02rVq3SxcH+YUk6C6htGQZPT09zHFnu3Ln1zTffNJebSK0lQPX2weVua1I5SkREhFavXt0uINgOjKldDLCFqMDAQM2RI4fdumIZme3L/GG8Jra/FUeGKEd8PocOHaqGYWjJkiV18ODBOmzYMA0ODlZXV1fNmzevjh07ViMjI/XKlSt2LZY//PBD2r8AWdiQIUN0+PDhdrdFRERo1apV1TAMzZ49u/k5rl27tn777bdm96y1a9eqr6+vGoahHTp00KVLl2rfvn3V399fS5cuzYyT6YgtuEREROibb76pzZs31/z585vj9R577DGz58DWrVu1UaNG6uLiomXLltUePXrosGHDzO7MBQoUcNoao/dzrOrTp485rnvWrFnq6empxYsX1yVLlqiq6pIlSzRPnjwaFBSU6rJXjpJ0f55//nnNkSOHGoahLi4uWr16dZ0/f7657f0ES2QcD3qebHtMdHS0GT7TS2+1/4JQifty9uxZzZMnj9atW9ccNJ70SrjNtWvXNDY2Vm/duqXffPONrl69Oll3q8wgOjpa69evr25ubtqrVy89d+6cHj58WLds2aKVKlVSFxcXff31182rrEnHLt0566azJSYmao8ePTQgIED79eunu3fv1q+++sqc0bdt27apThQQGhqqM2bM0GLFimWKCweJiYnmT0Z+TdLy87lq1Sp1cXHRFi1a2IXrkJAQnTp1qubPn1+zZ8+uM2fO1ISEBD179qwOGDBAvby8tFatWrp58+Y03fes6vDhw+YJ7JQpU8zbe/furdmzZ9dBgwbphQsXdN++ffrKK6+ot7e3litXTtevX28Gy61bt2pgYKDdRZSCBQtmql4mGZ0tcERGRmrVqlU1W7Zs2qpVK50zZ45OnjzZPEaVK1dON23apKq3x3YPHTrUDDmGYWhwcLB26tTJXGrIWe51rLLNYBsZGamRkZE6d+5cc9btSpUqqaurqz7yyCOpronrKLZAEBERodWqVTPX+ZwyZYqOHDlS3dzc1MPDQ4cNG2Y+5s5geWdLMjKWBzlPTnqxNjPNpE2oxF3ZFt61XX0ZP368eXtSmzdv1gEDBmiJEiX0ww8/zPRdNt577z27LzubhIQErVixoubKlSvZ+JWvvvrKnEVx9uzZzig7GdsV78aNG+tzzz1n14X1f//7nzZs2FANw9B27dqlGqJu3LiRKZYNsb0WmeE1ScvPZ//+/dUwDP3uu+9U1X7NuNDQUJ03b57myZNHH330UbPb7MmTJ7Vfv35qGIauXr06LXYZqrpp0ybNkSOHuru76+TJk1VVtWnTptq2bVu79/r8+fM6duxY9ff310cffVTXr19vdtU+fvy4LlmyRIcNG6YLFizQM2fOOGVfkLqEhAR97bXXzIl3kk6ad/z4ce3bt6/6+PiYwdIWeE6cOKE7duzQjRs36qVLl9LF2Pf7OVYNHz7c/HzGx8fr2rVrNSgoSIsVK6YNGzZMN5OaxMTEaLt27dTHx8eu5itXrmiTJk3MQD9q1CjzMbZg6e3trRUrVtStW7c6q3xYZPU8+c51LDMDQiVSlXTB3kqVKumjjz5qFxbDwsL022+/1Zdfftm8Ql6qVCn94IMPnFWyw7Rv315z585t1yUs6RqAw4cPNwNHdHS0GVRWrlyp+fPnTxeTXURERGilSpW0e/fuWrx4cd2yZYuq3g5Vtvf+0KFD5gx0dwtRmUVmeU3S6vMZExOjDRo0UD8/P71w4YKqJu+yc+nSJe3QoYM527PNyZMn083skpnZpk2b1M/PTw3D0AEDBmjp0qXNCwBJr5hfvnxZx40bZ84cmjRYIn0LDQ3VChUqaKFChcyeQPHx8XZLF9gu/lSvXl3XrVuXbIKe9OJBjlW3bt0yj1WXL1/Wq1evppulmlRvj+309/fXLl26mIH9yJEj+tJLL6lhGNq6dWt1c3Mzl3WxOXHihPbo0UNz586tp0+fdlb5sMDqefLixYudVXKaIlTinqZMmaKGYei4ceNU9fYMnz/99JPWrVtXPT091cfHR2vVqqVfffWVXrlyxcnVpr3Y2FitXLmyFi9eXK9du6aqt78EU1pUOjExUX/88UfdunWreUKXHsZQqqr+8ssvahiG5smTRwMCAnTFihWq+u+JZ2ohKj19iT9smeE1ScvPZ0xMjHnF3TamKSU7duxQDw8PbdasWYr3Z4axI+nZpk2bNFu2bJo9e3b19/c3xzvf2Q3r8uXLOn78eDNYbty4MdlSUUh/QkJCNE+ePFqpUqVU1ww9efKkli9fXg3D0Fq1aiVbxzI9sHKs+umnn9LlOqlxcXH66quvau7cuc3vg1OnTmnXrl3VMAzt1q2bqt6eHM3V1VUNw9ChQ4eajz916pReunTJKbXjv+M8+TZCJe7q8uXL+sQTT2i2bNl07969+sUXX+jjjz+unp6e6ubmpq1atdJt27aZrRY2maFveGpiY2P1scceU29vbz1x4oQmJCSkuqi0qmpwcLC+/PLL95w23Rk2b95sdnls3769efudS8UkDVGNGzdOVyHqYcvor0lafz4XLVqk7u7u+vLLLyfbZ9trdPbsWc2ePbtWqFBBY2JiMvXxIL3auHGj+vv7p/o5trEFy1y5cmmBAgX0m2++cXSpeEDXr1/XokWLqqurq906sDa2v7dFixapYRjq5+en+fLlS3fvrZVjVZcuXdLld6nq7VZJ29rNMTExOn36dDUMQ19++WVzm5CQEK1Ro4bZatW/f38nVYuHhfPkfxEqcVcbNmxQwzC0UKFC2rp1a3N2tn79+mXpQeWzZ89WwzB08ODB+thjj5nddO48ybbNkrlgwYJ0ewD55ptvzFkhx44da95+Z4g6fPiwVq1aVV1cXDQkJMQptTpKRn9N0vLz+c8//2iZMmXUMAydMmWK3VpxtufYsGGDurm56YABAx7eTuGB3etzbHPlyhUdMmSIFipUyOkTt+D+jBkzRl1dXbVPnz7JxmbZWvLWrl2rjzzyiL7xxhvp9r3NTN+lqv/2Brh27ZoWLVpUa9WqZXaFtR0r27dvb4ZnwzAydctVVsB58r8IlUhVTEyMPvfcc+aBL1++fDpmzBjdsWOH3Xbp9aphWvrnn3+0RIkSahiGurq66siRI5ONrVu0aJHmy5dP69atm+wKVXrz9ddfq7+/v3p4eOikSZPM2+8MUUePHs0yE3dk5NckrT+ff/zxh93YoKQzL+7YsUPr1Kmj7u7u5uyTcJ57fY5trl69ysltBnLq1ClzndipU6em2LW1d+/eWqFCBQ0JCUm3s7Bntu9S2/fCqlWr1DAMfe211+wmM1NVrVixos6cOVP//PNPPXjwoLNKxUPAebI9QiXu6s8//9QXX3xRx40blyWa7h/Evn37zAkxhg8frhcvXtSIiAiNjIzUYcOGafbs2TUwMNCcATO9s03w4e7uftcQlZVk5NckrT+fe/fu1Zw5c5rLE7Rv315fe+01zZcvnxqGobNmzXrIewSr7vU5xu3JbcaNG5ehxpTu27fPHJ83cOBA3bZtm3nfggULNDAwUNu2bZsuxyAmldm+S1Vvj893cXHRDh062N2+YMECDQgI0Pfff99JleFh4zz5X4RK3FNMTIx5lSWrXG25X3v37tX8+fOb67lVrVrVXCesTJkyGe4qJCefyWXk1yStP5+HDx/WTp06mevG2daOW7p0qblNRnidsoKM/DlOaxEREWZ3xKlTpzq7nAeyb98+c23RgIAAbdiwobkvefPmzTBBLLN9l16/fl0ff/xxcxbmVatW6cCBAzVnzpxarFgxPXv2rLNLzDRiYmL06NGjTq+B82RVQ1VVAFh2+vRpWbJkiXz66aeSkJAgRYoUkcaNG0uXLl0kKCjI2eU9sM2bN0v79u0lISFB+vfvL+PHj3d2SU6XkV+TtP58JiYmyunTp+X8+fMSEBAgjzzyiOTNm9e8z8XF5T//DjwcGflznJbCw8Nl7ty5MmfOHMmXL5/s2rVL3N3dnV3WfTtx4oTMnz9fPvnkE7lx44bkypVLKlasKLNnz5ZSpUo5u7z7ltm+Sw8dOiRt2rSREydOiO1Uu2TJkrJ27VopU6aMk6vLHKKjo6V+/fpSunRp6du3r1StWtXZJWVphErgIYmKihIRER8fHydX8t9988030rJlS8mdO7ccOnRIcuXK5eySnC6jvyaO/nyqqhiG4ZDfhfuX0T/HaSU0NFQGDBggS5culU2bNkmLFi2cXdIDu3jxoly7dk0CAgIke/bski1bNmeXZElm+i49c+aMrF27Vv766y+pVKmStGnTRgoXLuzssjKNU6dOyRNPPCFnzpyR7t27y3vvvSeenp7OLivLIlQCaSAznFB///33UqhQoQx1pTutZZbXJDN8PmFdZvkcP2xnz56VIkWKyNChQ2XixInOLgfCsQr39ueff0rnzp1l+fLlUrZsWWeXk6URKgEAAETkk08+kerVq9M9EchAYmJiaKFMBwiVAAAAAADLmEEBAAAAAGAZoRIAAAAAYBmhEgAAAABgGaESAAAAAGAZoRJOExMTI2PGjJGYmBhnl5LmstK+irC/mVlW2leRrLW/WWlfRdjfzCwr7asI+5uZZaR9ZfZXOE1YWJgEBATIzZs3xd/f39nlpKmstK8i7G9mlpX2VSRr7W9W2lcR9jczy0r7KsL+ZmYZaV9pqQQAAAAAWEaoBAAAAABY5ubsAuB8iYmJcv78efHz8xPDMBz2e8PCwuz+m5llpX0VYX8zs6y0ryJZa3+z0r6KsL+ZWVbaVxH2NzNz1r6qqoSHh0v+/PnFxeX+2iAZUwn5559/pGDBgs4uAwAAAEA6ERISIkFBQfe1LS2VED8/vyT/clxLpTPlzxfs7BIcpve4sc4uwaE+mDTN2SU4TGjoRWeX4FCurh7OLsGhIiNvOLsEh7l1K8LZJSANGUbWGW2lmujsEoCHxj4j3B2hEkm6vBoO7f7qTC4urs4uwWG8fHycXYJDZaX3NiudqInIfXfBySyyyvH4tqy0r1lPVvosq2adfUVmdrsj64P87Watb2gAAAAAwENFqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoFJETJ07I2bNnnV1Ginbu3Cm3bt1ydhkAAAAAkKIsGyrDwsJk8eLFUrduXSlevLjs27fPvC82Nlbmz58v1atXl1y5comPj48UKVJEWrVqJStWrEj2XNeuXZNBgwZJiRIlxMvLS3LmzCnNmzeX7777LsXffebMGenZs6eULFlSfHx8JGfOnPLoo4/K66+/LkeOHLHbdujQoRIYGCivv/66/Prrrw/3RQAAAACA/8jN2QU4UmJiovzwww/y8ccfy9q1ayU6OloMw5D69etL6dKlze06duwoX3zxhfj5+UndunXF399fzp07J7/88otERERI+/btzW3PnTsn9erVk5MnT0qhQoWkbdu2cuXKFfnhhx/k22+/lZkzZ8pbb71lbh8SEiJVqlSR69evS4kSJaRly5aSkJAgZ86ckQ8++EBq1aolpUqVMrdv06aNHD16VBYtWiSLFi2SEiVKyMsvvyydOnWSQoUKOeaFAwAAAIBUGKqqzi4irR0+fFg+/vhjWbZsmZw7d05EREqVKiWdOnWSl156SQoXLmxue+rUKQkODpbChQvL3r17JVeuXOZ9t27dkv3790utWrXM21q3bi0bN26UDh06yJIlS8TDw0NERH755Rdp1qyZxMTEyJ49e6RSpUoiIjJ69GgZN26c9O7dW+bMmWNX59mzZyUuLk6KFStmd3tCQoL88MMP8sknn8hXX30lUVFRYhiGNGzYULp06SLt2rUTX1/f+349YmJiJCYmxvx3WFiYFCxYUEQMMQzjvp8nIyuQv7izS3CY/tOmOLsEh5o3aryzS3CY69cvOLsEh3Jz83B2CQ4VERHq7BIcJjo6wtklIA25uGSdjnGJiYnOLgF4CG7Hw5s3b4q/v/99PSLT/pWHhobK/Pnz5bHHHpMyZcrIlClTJCYmRnr16iW7d++Ww4cPy/Dhw+0CpYjIlStXRESkcuXKdoFSRMTLy8suUJ48eVI2btwo2bJlkzlz5piBUkSkTp060qNHD0lISJB58+Yle/7GjRsnq7lQoULJAqWIiKurqzRr1kw+++wzuXjxoixZskQaNGggP/30k3Tu3FkCAwPllVdeka1bt8r9XCOYPHmyBAQEmD+3AyUAAAAAPLhMGSoHDx4s+fLlk169esmff/4p7dq1k6+++krOnz8vc+fOlRo1aqT62NKlS4uvr69s2rRJpk+fLufPn091219++UVERJo3by45c+ZMdn+nTp1ERGT79u3mbVWrVhURkWHDhsnGjRsfeBIePz8/6dKli2zZskXOnDkjEydOlKCgIFm6dKk0bNhQgoOD7caHpmTo0KFy8+ZN8yckJOSBagAAAAAAm0wZKnfv3i0xMTHi6uoqgwYNknnz5kmbNm3E3d39no/19/eXDz74QDw9PeXtt9+WAgUKSKlSpaRHjx6yY8cOu21tgbNIkSIpPpftdluXWxGRLl26yPPPPy8HDx6U1q1bS44cOaRevXoyadIkuXjx4gPtZ8GCBWXo0KHy/vvvS7Vq1URE5PTp0/ecydbT01P8/f3tfgAAAADAikwZKidPnixdunQRb29vmTBhggQFBUnz5s3ls88+k8jIyHs+/sUXX5STJ0/KBx98IM8995zcuHFDFi5cKHXq1JEBAwbcdx0pjU90dXWVlStXyr59+2T06NFSvXp12b17twwfPlxKlix53zO82rrvFilSRBo0aCB79uyRRx99VKZNmyb169e/7xoBAAAA4L/IlKGyVq1asmTJErl48aJ8/PHHUr9+ffnuu+/kpZdeksDAQOncubN8//33dx1MnSdPHnnttddk1apVcvHiRfn666/F399fZs6cKX///beIiOTPn19Ebi8RkpLTp0+LiEiBAgWS3Ve5cmUZM2aMbNu2Ta5cuSJvvfWWhIeHS79+/VKt6fLlyzJ79mypXr26lClTRiZNmiSRkZHSu3dv+f333+XAgQMyaNAgyZEjx32+UgAAAADw32TKUGnj6+srnTt3lh9//FFOnz4t48ePl3z58smyZcukadOmUrBgQRk0aJD89ddfd30ewzCkefPm8uSTT4qImKGyTp06IiLyzTffyI0bN5I97tNPPxURkbp16971+f39/WXy5MliGIYcOHDA7r7o6GhZuXKltGrVSgoUKCB9+/aVP//8U5566ilZs2aNnD9/XubMmWN2fwUAAAAAR8rUoTKpQoUKyYgRI+To0aOyY8cO6d69u0RFRck777wjFStWlE2bNomIyP79++XLL7+U2NhYu8dfv35ddu/eLSJizpYaHBwsTz75pISHh0vfvn0lLi7O3H7nzp2yYMECcXV1lV69epm3L1u2LFlwFBH5+uuvRVWTzcT61FNPSfv27WXTpk1Svnx5mTVrlpw7d07WrVsn7dq1s5txFgAAAAAczc3ZBThD7dq1pXbt2vLee+/JV199JR9//LG5FMeZM2fkmWeekYCAAKlWrZoEBgbKjRs3ZNu2bRIeHi6tW7e2W1Zk4cKFUrduXfnkk0/k559/llq1asmVK1dk69atkpCQIDNmzDDXqBQRWbNmjXTu3FmKFSsm5cuXF29vbzl16pTs3r1bXFxcZMKECXa15sqVS/r37y9dunSR8uXLO+T1AQAAAID7Zej9LGyYhVy8eFE+/PBD2bJlixw9elQuX74sOXLkkOLFi0vXrl3lpZdeSjaL7LVr12Ty5Mny1VdfSUhIiPj4+EiNGjVkwIAB0rRpU7ttt23bJqtWrZIdO3ZISEiIREZGSv78+c3tndGNNSwsTAICAkTESHFyocyoQP7izi7BYfpPm+LsEhxq3qjxzi7BYa5fv+DsEhzKzS1r9cyIiAh1dgkOEx0d4ewSkIZcXLJMx7i7ztcBZBy34+HNmzfve5UIQiUIlZkcoTLzIlRmboRKZBaESiCjefBQmXX+ygEAAAAADx2hEgAAAABgGaESAAAAAGAZoRIAAAAAYBmhEgAAAABgGaESAAAAAGAZoRIAAAAAYBmhEgAAAABgGaESAAAAAGAZoRIAAAAAYBmhEgAAAABgGaESAAAAAGAZoRIAAAAAYBmhEgAAAABgGaESAAAAAGAZoRIAAAAAYBmhEgAAAABgGaESAAAAAGAZoRIAAAAAYBmhEgAAAABgmZuzCwCc4fKVs84uwWGuX7ju7BIcysPDy9klOIy7u6ezS3Co3LkKOLsEh4q5FensEhwm1tXV2SU4VEJCgrNLcChX16xzuqka5+wSkEZU1dklpGu0VAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1RmUJs2bZJXX31VypQpI/7+/uLr6ysVK1aUSZMmSUxMjLPLAwAAAJBFuDm7AFjTtWtXiY6OlnLlykmFChXk5s2b8ttvv8nw4cPlxx9/lO+++05cXV2dXSYAAACATI5QmUEtXLhQmjZtKt7e3uZt4eHh0qFDB9m4caN89tln0rlz5xQfGxMTY9eaGRYWlub1AgAAAMic6P6aQbVp08YuUIqI+Pn5ybvvvisiIuvWrUv1sZMnT5aAgADzp2DBgmlaKwAAAIDMi5bKDOzYsWOyefNmOX78uERGRkpiYqKoqnlfaoYOHSr9+/c3/x0WFkawBAAAAGAJoTIDUlUZOHCgvPvuu2aIvFN4eHiqj/f09BRPT8+0Kg8AAABAFkL31wxo5cqVMnPmTAkKCpIvvvhCzp07J7GxsaKq5ljJ1MImAAAAADxMtFRmQGvXrhURkQULFsiTTz5pd9/JkyedURIAAACALIqWygwoNDRURESCgoKS3bdq1SpHlwMAAAAgCyNUZkAlS5YUEZFFixbZdXPdvn27TJ8+3VllAQAAAMiCCJUZUJ8+fcTX11fmz58v5cqVkxdffFHq1asn9evXlx49eji7PAAAAABZCKEyAypZsqTs2bNHWrduLVevXpX169dLRESELFy4kJZKAAAAAA7FRD0ZVOnSpWX9+vUp3sfMrwAAAAAchZZKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhqqqs4uAc4WFhUlAQICzy3Aww9kFII2ULVvb2SU4zJXLZ51dgkM1bPyis0twqGw5sjm7BIf5eNE4Z5fgUAkJCc4uwaE8PDydXYLDxMfHObsEh8pKMSIr7avI7X29efOm+Pv739cjaKkEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaofECnT58WwzCkQYMGEhkZKf3795eCBQuKt7e3VKlSRTZs2GBuu3r1annsscfE19dX8ubNK3369JHo6Gi75/vjjz/k7bfflqpVq0qePHnE09NTgoOD5Y033pDz58/f9fdHR0fLkCFDpHDhwuLp6SnFixeXqVOniqqm+esAAAAAACKESstiY2OlUaNG8tlnn0nNmjWlZs2a8ueff8rTTz8tP/zwg7z77rvSoUMH8fPzk2bNmklCQoLMmTNHXnvtNbvnmTJlirz77rsiIlKnTh1p2bKlqKosWLBAqlWrlmKwtP3+pk2bygcffCDVqlWThg0byrlz52TIkCEycuTINN9/AAAAABARMZRmrQdy+vRpKVq0qIiIPPHEE7J+/Xrx9fUVEZGlS5fKK6+8IsWLF5dr167Jd999J9WqVRMRkfPnz0vlypXl8uXLcuLECQkODhYRkZ9++knKli0refPmNX9HYmKiTJgwQUaPHi2vvPKKfPTRRyn+/vr168v69evF399fRET27NkjNWvWFE9PT7l06ZJky5YtxX2IiYmRmJgY899hYWFSsGDBh/USZRCGswtAGilbtrazS3CYK5fPOrsEh2rY+EVnl+BQ2XKkfAzPjD5eNM7ZJThUQkKCs0twKA8PT2eX4DDx8XHOLsGhslKMyEr7KnJ7X2/evGnmjHuhpdIiFxcXWbBggRkoRUQ6d+4suXPnluPHj0uvXr3MQCkikj9/funYsaOIiGzbts28vWHDhnaB0vbco0aNkgIFCsj69etT/f0LFy60e6OrVasmLVq0kKioKNmzZ0+qtU+ePFkCAgLMn6wXKAEAAAA8LG7OLiCjKlKkiJQsWdLuNhcXFylcuLBcvXpVmjZtmuwxttbJCxcu2N1+7do1Wb9+vRw4cEBu3LhhXsGMi4uTa9euyfXr1yVnzpx2jylcuLCUKlUq2e+w1XTn70hq6NCh0r9/f/PfWbOlEgAAAMDDQKi0qECBAinebutymtL9tvuSdj1dvny5dO/eXSIiIlL9XeHh4clCZVBQUIrb+vn5Jfsdd/L09BRPz6zTFQUAAABA2qH7q0UuLnd/6e51v4jImTNnpEuXLhIbGyuzZs2SY8eOSVRUlKiqqKrUqlVLRFLuw30/zw8AAAAAaY2WSifavHmzxMbGysCBA6Vv377J7j958qQTqgIAAACA+0dzlxOFhoaKSMpdWbdt2yaXLl1ydEkAAAAA8EAIlU5km1Tn008/lcjISPP2c+fOSY8ePZxVFgAAAADcN0KlEz311FPy6KOPyp49e6R48eLy7LPPSqtWraRkyZKSI0cOqV0766y3BwAAACBjIlQ6kYeHh2zfvl169uwpXl5esnHjRjl06JC8+eab8v3334u7u7uzSwQAAACAuzI0palFkaWEhYVJQECAs8twMMPZBSCNlC2bdVr4r1w+6+wSHKph4xedXYJDZcuRzdklOMzHi8Y5uwSHsq1HnVV4eGSdZczi4+OcXYJDZaUYkZX2VeT2vt68eVP8/f3v6xG0VAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsc3N2AYBzqLMLQBo5cWK/s0twGE9PH2eX4FBla5d1dgkOdf74eWeX4DCqWe2YnLX2NyEh3tklAA9B1vq7fVC0VAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUpmOGYUiRIkWcXQYAAAAApIpQCQAAAACwzM3ZBSB1hw4dEnd3d2eXAQAAAACpIlSmY6VLl3Z2CQAAAABwV3R/TcdSGlO5detWMQxDunTpItevX5eePXtKvnz5xNPTU8qVKycfffSRc4oFAAAAkCXRUplB3bhxQ2rVqiURERFSt25duXr1qmzbtk26du0qiYmJ8tprrzm7RAAAAABZAC2VGdS6deukSpUqcvLkSVm1apVs2bJFvvjiCxERGT9+/F0fGxMTI2FhYXY/AAAAAGAFoTKD8vf3l7lz54qnp6d5W9u2baVcuXJy9uxZOX36dKqPnTx5sgQEBJg/BQsWdEDFAAAAADIjQmUGVbVqVcmVK1ey20uWLCkiIhcuXEj1sUOHDpWbN2+aPyEhIWlWJwAAAIDMjTGVGVRQUFCKt/v5+YnI7S6uqfH09LRr4QQAAAAAq2ipzKBcXHjrAAAAADgfyQQAAAAAYBmhEgAAAABgGaESAAAAAGAZoRIAAAAAYBmhEgAAAABgGUuKpGOqmuy2Bg0apHi7zdKlS2Xp0qVpWBUAAAAA/IuWSgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZYRKAAAAAIBlhEoAAAAAgGWESgAAAACAZW7OLgAAHqaYmChnl+AwCQnxzi7Boeo8Uc3ZJTjUe98tcHYJDqOqzi4BaSghIcHZJSDN8LeL22ipBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqEwHNm3aJK+++qqUKVNG/P39xdfXVypWrCiTJk2SmJgYu22XLl0qhmHImDFj5OzZs9KhQwfJkyePeHt7S7Vq1WTDhg1O2gsAAAAAWZGbswuASNeuXSU6OlrKlSsnFSpUkJs3b8pvv/0mw4cPlx9//FG+++47cXV1tXvM6dOnpXr16uLn5yeNGjWSs2fPys6dO6Vt27by9ddfS9OmTZ20NwAAAACyEloq04GFCxfKxYsXZceOHbJy5Ur55ptv5MyZM9KqVSvZsmWLfPbZZ8ke8/HHH0unTp3k6NGjsmLFCvn111/l3XfflcTERJkwYYIT9gIAAABAVkSoTAfatGkj3t7edrf5+fnJu+++KyIi69atS/aYokWLyqRJk8TF5d+3sHfv3pIjRw7ZtWuXxMbGpvr7YmJiJCwszO4HAAAAAKyg+2s6cezYMdm8ebMcP35cIiMjJTExUVTVvO9ODRo0EA8PD7vb3NzcpGjRorJv3z65du2a5MuXL8XfNXnyZBk7duzD3wkAAAAAWQ6h0slUVQYOHCjvvvuuGSLvFB4enuy2oKCgFLf18/MTEUk2wU9SQ4cOlf79+5v/DgsLk4IFCz5I2QAAAAAgInR/dbqVK1fKzJkzJSgoSL744gs5d+6cxMbGiqqawTClsJm02+uD8vT0FH9/f7sfAAAAALCClkonW7t2rYiILFiwQJ588km7+06ePOmMkgAAAADgvtFS6WShoaEiknJ31lWrVjm6HAAAAAB4IIRKJytZsqSIiCxatMium+v27dtl+vTpzioLAAAAAO4LodLJ+vTpI76+vjJ//nwpV66cvPjii1KvXj2pX7++9OjRw9nlAQAAAMBdESqdrGTJkrJnzx5p3bq1XL16VdavXy8RERGycOFCWioBAAAApHuGpraOBbKMsLAwCQgIcHYZAB6Qm5vHvTfKRL75Y5+zS3Co94YscHYJDrNxU9bZVxER1URnl+BghrMLQJohRmRmN2/evO9VImipBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhGqAQAAAAAWEaoBAAAAABYRqgEAAAAAFhmqKo6uwg4V1hYmAQEBDi7DAC4Ky+vbM4uwaGio8OdXYLD+PnldHYJDpWV3lsREU9PH2eX4DDx8bHOLsGh4uKyzv6qJjq7BIe7efOm+Pv739e2tFQCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALMs0oXLnzp3Spk0byZMnj3h6ekqRIkXkjTfekPPnz9ttt3TpUjEMQ8aMGSNHjx6V9u3bS968ecXFxUW++uorERE5fvy4jBkzRmrVqiWBgYHi4eEhQUFB0rlzZzl69GiKv98wDClSpIgkJCTI1KlTpWTJkuLp6SkFCxaUwYMHS0xMTIqP++uvv6R169aSPXt28fPzk3r16sn3338vW7duFcMwpEuXLskeo6qyfPlyeeKJJyRHjhzi5eUlZcqUkTFjxkhUVNR/eh0BAAAA4EFkilD56aefSt26dWX9+vVSqlQpadeunXh6esqCBQukSpUqcvjw4WSPOXLkiFSvXl1+++03adiwoTRp0kTc3d1FRGTx4sUybtw4iYyMlOrVq8tTTz0l/v7+smzZMqlevbr89ddfqdbSoUMHmTBhgpQqVUqaNm0q4eHhMm3aNOnatWuybXfu3Cm1atWSjRs3SuHChaVVq1Zy69Ytad68uXz55ZcpPn9iYqJ07NhROnToIL///rtUqlRJWrZsKZGRkTJ27Fhp2LChREdHW3wlAQAAAODBuDm7gP8qJCREunfvLiIi69atk6eeekpEboevAQMGyKxZs6RTp07y+++/2z1uxYoV0rt3b5k1a5a4urra3de2bVt5/fXXpWjRona3L1myRF599VXp16+fbNmyJVktZ86cER8fHzl27JgEBgaKiMipU6ekSpUq8tlnn8nYsWOlWLFiZn1dunSRqKgomThxogwbNsx8ng8//FBee+21FPd3xowZsnz5cmnQoIEsX77c/D2xsbHyxhtvyIcffihjx46VKVOmpPqaxcTE2LWchoWFpbotAAAAANxNhm+pXLx4sURHR8vzzz9vBkoRERcXF5kyZYrkz59f9uzZIzt27LB7XJ48eWTq1KnJAqWISM2aNZMFShGRV155RR5//HHZunWr3Lx5M8V6Zs+ebQY9EZGiRYvKSy+9JCIi27dvN2/fsmWLHD16VEqUKCFDhgyxe46uXbvK448/nuy54+PjZdq0aeLr6ysrVqyw+z0eHh4yZ84cCQwMlEWLFkliYmKK9YmITJ48WQICAsyfggULprotAAAAANxNhg+VtqDWsWPHZPd5enrKc889Z7edTePGjcXHxyfV542IiJDly5fL4MGDpVu3btKlSxfp0qWLXLhwQVRVTpw4kewx7u7u0rBhw2S3lyxZUkRELly4YN5mC7nPPPOMuLgkfxteeOGFZLft27dPrl69KrVr15a8efMmu9/b21uqVq0qoaGhcuzYsVT3bejQoXLz5k3zJyQkJNVtAQAAAOBuMnz3V9tEPEWKFEnxftvt586ds7u9UKFCqT7nli1bpH379nLlypVUtwkPD092W2BgYIotn35+fiIidl1ObQEztVbClOo7ffq0iIh8//33YhhGqrWJiFy9elVKlSqV4n2enp7i6el518cDAAAAwP3I8KHyXlILX15eXineHhERIc8//7xcv35dRo0aJe3bt5fChQuLt7e3GIYhHTp0kOXLl4uqJntsSi2OD5OtS2vx4sVT7B6bVK5cudK0FgAAAAAQyQShMn/+/HLkyBE5c+aMPProo8nut7XuFShQ4L6eb/v27XLt2jV59tlnZezYscnuP3ny5H+q1yZfvnwiIql2PU3p9qCgIBERKV26tCxduvSh1AEAAAAA/0WGH1NZt25dERFZvnx5svtiY2Nl9erVdtvdS2hoqIj8G+CSOn78uOzbt89qqXZsLY1r165NsdVz1apVyW6rXr26BAQEyM8//yzXr19/KHUAAAAAwH+R4UNl165dxdvbW1asWCGbNm0yb09MTJRhw4bJuXPnpGrVqvfsLmpjm1Tnyy+/tBtTeePGDenatavExcU9lLqfeOIJKVGihBw5ckSmTZtmd9/SpUuTTSwkcnss5Ntvvy3h4eHSrl27FFtNz507J8uWLXsoNQIAAADAvWT47q+FChWShQsXSpcuXaR169by+OOPS8GCBWXfvn1y5MgRyZs3r3z66af3/XzVqlWTJk2ayPfffy8lS5aUBg0aiIjI1q1bJXfu3NKmTRtZt27df67bxcVFPv74Y2ncuLEMGTJEli9fLmXLlpUTJ07I77//Lr169ZJ58+aJh4eH3eOGDBkihw8flmXLlkmZMmWkcuXKUrRoUYmNjZUjR47IwYMHpUKFCtKpU6f/XCMAAAAA3EuGb6kUEenUqZNs375dWrVqJYcOHZIvvvhCoqOjpWfPnrJ3714pXbr0Az3funXrZPjw4ZInTx75+uuvZe/evdK+fXvZtWuXZM+e/aHVXatWLfn111+lVatWcurUKVm/fr24u7vL5s2bpVatWiKSfMIdFxcX+eSTT2TdunXSpEkTOXXqlKxZs0Z++eUX8fLykkGDBslHH3300GoEAAAAgLsxNKUBfXC6Hj16yMKFC2XFihUprln5MIWFhUlAQECa/g4A+K+8vLI5uwSHio5OvnRVZuXnl9PZJThUVnpvRUQ8PVNfFzyziY+PdXYJDhUXl3X2VzXR2SU43M2bN8Xf3/++ts0ULZUZ1fXr183ZaZNauXKlLF68WLJnzy6tWrVyfGEAAAAAcJ8y/JjKjOzo0aNSq1YtqVChggQHB4uIyKFDh+TIkSPi6uoqCxcuFF9fXydXCQAAAACpo6XSiYKDg6VXr14SFxcnP/30k2zcuFFu3rwp7dq1k+3bt8vzzz/v7BIBAAAA4K4YUwnGVALIEBhTmXkxpjJzY0xl5sWYysyNMZUAAAAAAIcgVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACxzc3YBAADcj5iYKGeX4FB+fjmdXYLDqCY6uwSHcnFxdXYJDvXRj986uwSHmff2TGeX4FA7d65zdgkOEx8f5+wSHEgf+BG0VAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUAgAAAAAsI1QCAAAAACwjVAIAAAAALCNUprHTp0+LYRjSoEEDZ5cCAAAAAA8doTIT2Lp1qxiGIV26dHF2KQAAAACyGEIlAAAAAMAyQiUAAAAAwDJCpQNFR0fLkCFDpHDhwuLp6SnFixeXqVOniqrabbd9+3bp3bu3VKhQQXLkyCHe3t5SunRpGTJkiNy4ccNu2y5dukjDhg1FROTjjz8WwzDMnzFjxjhozwAAAABkVW7OLiCriI2NlaZNm8rBgwelQYMGEhkZKT///LMMGTJEwsPDZcKECea2gwYNkj///FMqVKggjRo1klu3bsm+fftk6tSpsnHjRtm1a5dky5ZNRETq1KkjFy9elG+//VaKFSsmderUMZ+nUqVKjt5NAAAAAFkModJBdu7cKfXr15dTp06Jv7+/iIjs2bNHatasKe+++64MGTLEDIqjR4+W2rVrS0BAgPn4mJgY6dOnjyxatEhmzpwpo0aNEhGR1157TYoXLy7ffvut1KlTR5YuXXrPWmJiYiQmJsb8d1hY2EPcUwAAAABZCd1fHcTFxUUWLlxoBkoRkWrVqkmLFi0kKipK9uzZY97eokULu0ApIuLp6SmzZs0SNzc3Wbdu3X+qZfLkyRIQEGD+FCxY8D89HwAAAICsi5ZKBylcuLCUKlUq2e0lS5YUEZELFy7Y3X7u3DnZsGGDHD58WMLCwiQxMVFERDw8POTYsWP/qZahQ4dK//79zX+HhYURLAEAAABYQqh0kKCgoBRv9/PzExGx6446c+ZMGTJkiMTFxaVJLZ6enuLp6Zkmzw0AAAAga6H7q4O4uNzfS71r1y4ZMGCA+Pj4yNKlS+X06dNy69YtUVVRVcmXL18aVwoAAAAA94+WynRm7dq1IiIyceJEefnll+3ui46OlosXLzqjLAAAAABIES2V6UxoaKiIpNxddvXq1cnWtBS5Pc5SRCQ+Pj5tiwMAAACAOxAq0xnbxD0ffvih3ZjKgwcPyuDBg1N8TP78+UVE5MiRI2lfIAAAAAAkQahMZ1555RUJDAyUDRs2SKlSpeSFF16QJk2aSKVKlaRu3bpSuHDhZI8pUqSIVKhQQfbs2SM1atSQV155RV577TVZv369E/YAAAAAQFZCqExncuXKJb///rt06NBBYmNjZf369XLu3DkZP368LF++PNXHrVmzRtq2bSsnT56UTz75RD788EPZt2+fAysHAAAAkBUZmtIgPWQpYWFhEhAQ4OwyAOCuDCNrXQf19c06x2XVRGeX4FCxsbecXYJDLdu21dklOMy8t2c6uwSH2rlznbNLcJj4+LRZ6i99uh0Pb968Kf7+/vf1iKz1DQ0AAAAAeKgIlQAAAAAAywiVAAAAAADLCJUAAAAAAMsIlQAAAAAAywiVAAAAAADLCJUAAAAAAMsIlQAAAAAAywiVAAAAAADLCJUAAAAAAMsIlQAAAAAAywiVAAAAAADLCJUAAAAAAMsIlQAAAAAAywiVAAAAAADLCJUAAAAAAMsIlQAAAAAAywiVAAAAAADLCJUAAAAAAMvcnF0AAAD3QzXR2SU4VETEDWeX4DAuLlnrGreqOrsEhzqw429nl+AwzTo85ewSHOrXX9c6uwSkE1nrKA4AAAAAeKgIlQAAAAAAywiVAAAAAADLCJUAAAAAAMsIlQAAAAAAywiVAAAA/9fe3cdoVd75A/48ODBjRbA65U2nEgotGqI0VAqW11XQtlEpRkMsbNGyjZQlJYulok3VkMjPxnYrSyN/rJaYJiRAoV3kxWoNxc5WQK0QWsRVfBnUjVIDKMoIzPz+aJiCA+7MUeYpM9eVPAlz7nPu870PZJjP3Pc5B4DChEoAAAAKEyoBAAAoTKgEAACgMKESAACAwoRKAAAAChMqAQAAKEyoBAAAoDChEgAAgMKESgAAAAoTKgEAAChMqAQAAKAwoRIAAIDChEoAAAAKEyoBAAAoTKgEAACgMKESAACAwoTKU0SpVErfvn3LXQYAAMAxhEoAAAAKqyh3AbTM9u3b07lz53KXAQAAcAyh8hQxcODAcpcAAADQjOWvp4gT3VP53//935kwYULOP//8VFZWplevXhk6dGhuvfXWvPvuu21fKAAA0KEIlaewVatWZeTIkfmv//qv9O7dOxMnTswXv/jFvP3227nnnnuye/fucpcIAAC0c5a/nsLuvffeNDQ0ZPny5bn22muPadu8eXPOOeec4x5XX1+f+vr6pq/37dt3UusEAADaLzOVp7C33norSXL55Zc3a7vkkkty5plnHve4+fPnp3v37k2fmpqak1onAADQfgmVp7AhQ4YkSaZMmZLNmzenoaGhRcfNnTs3e/fubfrU1dWdzDIBAIB2TKg8hd199925+OKLs2rVqgwdOjTV1dW5+uqr85//+Z85cODACY+rrKxMt27djvkAAAAUIVSewmpqavLUU0/lkUceycyZM1NTU5NVq1blX/7lX3LRRRflr3/9a7lLBAAA2jmh8hRXUVGR8ePHZ8GCBdmyZUtefvnl/NM//VP+53/+J/fcc0+5ywMAANo5obKdOf/88/ODH/wgSbJt27YyVwMAALR3QuUp7N///d/zv//7v822r1mzJkk81RUAADjpvKfyFHbXXXfllltuycUXX5wBAwaksbExW7ZsyfPPP5+zzz47t9xyS7lLBAAA2jkzlaew//iP/8ikSZPy3nvvZe3atVm3bl0qKiryb//2b9m6dWsGDBhQ7hIBAIB2zkzlKaKxsbHZtilTpmTKlCllqAYAAOBvzFQCAABQmFAJAABAYUIlAAAAhQmVAAAAFCZUAgAAUJhQCQAAQGFCJQAAAIUJlQAAABQmVAIAAFCYUAkAAEBhQiUAAACFCZUAAAAUJlQCAABQmFAJAABAYUIlAAAAhQmVAAAAFCZUAgAAUJhQCQAAQGFCJQAAAIUJlQAAABRWUe4CAICOrbGxsdwltKmONt4Xnnmh3CW0mav/9epyl9CmKk7rXO4S2kxDQ0O5S2gzf/sW1brvU2YqAQAAKEyoBAAAoDChEgAAgMKESgAAAAoTKgEAAChMqAQAAKAwoRIAAIDChEoAAAAKEyoBAAAoTKgEAACgMKESAACAwoRKAAAAChMqAQAAKEyoBAAAoDChEgAAgMKESgAAAAoTKgEAAChMqAQAAKAwoRIAAIDChEoAAAAKEyoBAAAoTKgEAACgMKGyFdavX59SqZSpU6e26rhSqZS+ffuelJoAAADKSag8yuLFi1MqlXLnnXeWuxQAAIBTQkW5C+gItm/fns6dO5e7DAAAgE+cUNkGBg4cWO4SAAAATopWL3/dtm1bJk+enH79+qWqqiqf+cxnMnjw4MyaNStvvPHGMfuuWbMm48aNy6c//elUVVXlC1/4Qm699dbs2bOnWb933nlnSqVSFi9enKeffjpf/epXc9ZZZ+Xss8/O9ddfn127diVJ9u/fnzlz5qRv376pqqrKoEGDsnz58hPWu3379kydOjU1NTWprKxMz549M2nSpPz5z38+Zr8xY8bkxhtvTJLcddddKZVKTZ/Fixc36/ftt9/O9OnT07t371RWVmbQoEF58MEHj1vD8e6pPPr+zNb0lSQrVqzIsGHD8qlPfSrV1dW57rrr8sILLxxzDQEAANpCq2Yqn3766YwYMSIHDhzIRRddlGuuuSbvvfdedu7cmfvuuy8TJkxI7969kyTz58/PbbfdloqKiowePTrV1dWpra3NPffck5UrV2bDhg3p2bNns3Ns3LgxN998cwYNGpQrrrgizzzzTJYtW5YtW7Zk06ZNGTduXF555ZWMGjUqu3fvzu9///tcf/31Wbt2ba644opj+vr1r3+dSZMmpb6+PoMHD86wYcNSV1eXpUuXZtWqVVm7dm1GjRqVJLnyyitz6NCh1NbW5uKLL87gwYOb+unfv/8x/e7ZsyfDhw/Pu+++m5EjR2b37t3ZsGFDvv3tb6ehoSHTpk1r8TVtbV/33XdfZs2alU6dOmXUqFHp1atXNm7cmKFDh+aqq65q8XkBAAA+Ca0KlQsWLMiBAwdy7733Zvbs2ce0Pffcc+nevXuSZPPmzfnhD3+Yrl275rHHHsuXv/zlJEl9fX2mTJmSZcuWZcaMGcedYVy0aFHuv//+3HzzzUmSgwcP5mtf+1oee+yxXHrppenVq1d27tyZM844I0nywAMPZNq0abn77ruPCZUvv/xyJk+enM6dO+fhhx/O5Zdf3tS2bt26XH311Zk8eXJeeOGFdOnSJbfeemt69eqV2traTJgw4SMf1vOb3/wmkyZNyuLFi1NZWZnkbwH2G9/4RubNm9eqUNmavnbu3Jk5c+akS5cuWbduXcaOHZskOXToUL7zne/kF7/4RYvOWV9fn/r6+qav9+3b1+J6AQAAjtaq5a9vvfVWkhwT0I4YOHBg0yzlwoUL09DQkJkzZzYFyiSprKzMwoULc/rpp2flypWpq6tr1s+IESOaAmWSdO7cOTNnzkzyt+B6//33NwXKJJk6dWqqq6vzxz/+MQcPHmza/rOf/Sz79+/P/Pnzm9V75ZVXZvr06amrq8vq1atbcwmSJN26dcvChQubQmCSTJgwIYMGDcqrr76al19++aT09eCDD+aDDz7IlClTmgJlklRUVOSnP/1punbt2qJzzp8/P927d2/61NTUtLheAACAo7UqVA4ZMiRJMmPGjKxfvz6HDh067n5PPPFEkuSb3/xms7YePXpk/PjxaWhoSG1tbbP28ePHN9vWr1+/JEnfvn3z+c9//pi20047Leeff34OHjyY3bt3N23/7W9/mySZOHHicWscOXJkkmTTpk3Hbf8oQ4YMyTnnnNNs+5HaPnxv6SfV15Hrdd111zXb/6yzzjrutTueuXPnZu/evU2f44V7AACAlmjV8tfvf//7+cMf/pD169dn7Nix6dq1a4YPH56vf/3rmTp1atPy19dffz1Jmj2c5ogj21977bVmbeeee26zbUdm4I7XdnT70Us6j8zwneiYI44Ooi113nnnHXf7mWee2ayOT7KvIwHzRDOLn/3sZ1t0zsrKymNmRgEAAIpqVajs1q1bHn/88dTW1mbVqlVZv359Hn/88Tz66KOZP39+nnjiiQwYMOD/7KdUKp2wrVOnE0+eflTbhzU0NCRJvvWtb33kfkcvz22p1tTRln0BAAC0tVa/p7JUKmXEiBEZMWJEkuTNN9/MrFmzsmTJktx+++1ZunRp+vTpk5deeimvvPJKLrzwwmZ9tHQW8eM477zz8uKLL+YnP/nJcZeXnop69+6dHTt2pK6u7rjX1TJWAACgrX3sabIePXo0PSl127ZtSf5+v+KSJUua7f/WW2/lkUceSalUyle+8pWPe/oTGjduXJJk5cqVLT6mS5cuSXLCe0XL7cj1+tWvftWsbe/evU33kQIAALSVVoXKRYsW5aWXXmq2fc2aNUn+fq/fjBkz0qlTpyxYsCBPPfVU034ffPBBZs6cmffffz8TJ048qU8dnT17dk4//fTccsstWbFiRbP2+vr6LF++PLt27Wra1qdPnyTJjh07TlpdH8eNN96YLl265KGHHsqGDRuath8+fDizZ8/OO++8U8bqAACAjqhVy18XLVqU6dOn58ILL8wFF1yQioqKPPfcc9myZUuqqqryox/9KEkydOjQzJs3L7fffnuGDx+eMWPGpLq6OrW1tamrq8uAAQPy85///KQM6Ij+/ftnyZIlueGGG3Lttdemf//+ueCCC3LGGWfktddeyzPPPJP9+/fnT3/6U9PDcoYNG5YePXpk+fLlGTNmTPr165dOnTrlpptuyqWXXnpS622Jz33uc/nxj3+cWbNmZezYsRk9enR69uyZTZs25e23387kyZPzy1/+smnGFQAA4GRr1UzlvHnzctNNN6VUKuV3v/tdVq1alffffz/Tpk3Ls88+e8xy1ttuuy0PP/xwRo8enc2bN2fFihWprKzMnDlzsnHjxvTs2fMTH8yHXXPNNdm6dWu++93vplQq5dFHH83q1avz5ptv5qqrrsrSpUuPuTexqqoqq1evzrhx4/Lss89m8eLFeeCBB/L888+f9Fpb6nvf+16WL1+eL33pS3nyySfzyCOPZPDgwdm4cWOqqqqSpN3cQwoAAPzjKzU2NjaWuwg+vsOHD+eiiy7K9u3b8/rrr6dXr14tPnbfvn1Nr4MB4B/FiZ+U3t581FPh26OO9qPXpBt+UO4S2szV/3p1uUtoUzeOuazcJbSZDw62/JWBp7q/fY9qzN69e9OtW7cWHeN9FqeYF198MXv27DlmW319febMmZO//OUvueyyy1oVKAEAAD6OVr9ShPJatmxZ7rjjjgwZMiQ1NTXZt29ftmzZkjfeeCPV1dVZuHBhuUsEAAA6EKHyFHPZZZdly5YtefLJJ7N169YcOnQo5557bqZPn565c+ee1CfqAgAAfJhQeYq55JJLjvv+TwAAgHJwTyUAAACFCZUAAAAUJlQCAABQmFAJAABAYUIlAAAAhQmVAAAAFCZUAgAAUJhQCQAAQGFCJQAAAIUJlQAAABQmVAIAAFCYUAkAAEBhQiUAAACFCZUAAAAUJlQCAABQmFAJAABAYRXlLgAAWqJU6li/B+3UqeOMt7GxsdwltKmONt4df3mq3CW0mXO63lDuEtrU2ef0KXcJbWb37l3lLqHNNDY25tChD1p1TMf5HwsAAIBPnFAJAABAYUIlAAAAhQmVAAAAFCZUAgAAUJhQCQAAQGFCJQAAAIUJlQAAABQmVAIAAFCYUAkAAEBhQiUAAACFCZUAAAAUJlQCAABQmFAJAABAYUIlAAAAhQmVAAAAFCZUAgAAUJhQCQAAQGFCJQAAAIUJlQAAABQmVAIAAFCYUAkAAEBhQmUrrF+/PqVSKVOnTm3VcaVSKX379j0pNQEAAJSTUHmUxYsXp1Qq5c477yx3KQAAAKeEinIX0BFs3749nTt3LncZAAAAnzihsg0MHDiw3CUAAACcFK1e/rpt27ZMnjw5/fr1S1VVVT7zmc9k8ODBmTVrVt54441j9l2zZk3GjRuXT3/606mqqsoXvvCF3HrrrdmzZ0+zfu+8886USqUsXrw4Tz/9dL761a/mrLPOytlnn53rr78+u3btSpLs378/c+bMSd++fVNVVZVBgwZl+fLlJ6x3+/btmTp1ampqalJZWZmePXtm0qRJ+fOf/3zMfmPGjMmNN96YJLnrrrtSKpWaPosXL27W79tvv53p06end+/eqayszKBBg/Lggw8et4bj3VN59P2ZrekrSVasWJFhw4blU5/6VKqrq3PdddflhRdeOOYaAgAAtIVWzVQ+/fTTGTFiRA4cOJCLLroo11xzTd57773s3Lkz9913XyZMmJDevXsnSebPn5/bbrstFRUVGT16dKqrq1NbW5t77rknK1euzIYNG9KzZ89m59i4cWNuvvnmDBo0KFdccUWeeeaZLFu2LFu2bMmmTZsybty4vPLKKxk1alR2796d3//+97n++uuzdu3aXHHFFcf09etf/zqTJk1KfX19Bg8enGHDhqWuri5Lly7NqlWrsnbt2owaNSpJcuWVV+bQoUOpra3NxRdfnMGDBzf1079//2P63bNnT4YPH5533303I0eOzO7du7Nhw4Z8+9vfTkNDQ6ZNm9bia9ravu67777MmjUrnTp1yqhRo9KrV69s3LgxQ4cOzVVXXdXi8wIAAHwSWhUqFyxYkAMHDuTee+/N7Nmzj2l77rnn0r179yTJ5s2b88Mf/jBdu3bNY489li9/+ctJkvr6+kyZMiXLli3LjBkzjjvDuGjRotx///25+eabkyQHDx7M1772tTz22GO59NJL06tXr+zcuTNnnHFGkuSBBx7ItGnTcvfddx8TKl9++eVMnjw5nTt3zsMPP5zLL7+8qW3dunW5+uqrM3ny5Lzwwgvp0qVLbr311vTq1Su1tbWZMGHCRz6s5ze/+U0mTZqUxYsXp7KyMsnfAuw3vvGNzJs3r1WhsjV97dy5M3PmzEmXLl2ybt26jB07Nkly6NChfOc738kvfvGLFp2zvr4+9fX1TV/v27evxfUCAAAcrVXLX996660kOSagHTFw4MCmWcqFCxemoaEhM2fObAqUSVJZWZmFCxfm9NNPz8qVK1NXV9esnxEjRjQFyiTp3LlzZs6cmeRvwfX+++9vCpRJMnXq1FRXV+ePf/xjDh482LT9Zz/7Wfbv35/58+c3q/fKK6/M9OnTU1dXl9WrV7fmEiRJunXrloULFzaFwCSZMGFCBg0alFdffTUvv/zySenrwQcfzAcffJApU6Y0BcokqaioyE9/+tN07dq1ReecP39+unfv3vSpqalpcb0AAABHa1WoHDJkSJJkxowZWb9+fQ4dOnTc/Z544okkyTe/+c1mbT169Mj48ePT0NCQ2traZu3jx49vtq1fv35Jkr59++bzn//8MW2nnXZazj///Bw8eDC7d+9u2v7b3/42STJx4sTj1jhy5MgkyaZNm47b/lGGDBmSc845p9n2I7V9+N7ST6qvI9fruuuua7b/WWedddxrdzxz587N3r17mz7HC/cAAAAt0arlr9///vfzhz/8IevXr8/YsWPTtWvXDB8+PF//+tczderUpuWvr7/+epI0ezjNEUe2v/baa83azj333GbbjszAHa/t6Pajl3QemeE70TFHHB1EW+q888477vYzzzyzWR2fZF9HAuaJZhY/+9nPtuiclZWVx8yMAgAAFNWqUNmtW7c8/vjjqa2tzapVq7J+/fo8/vjjefTRRzN//vw88cQTGTBgwP/ZT6lUOmFbp04nnjz9qLYPa2hoSJJ861vf+sj9jl6e21KtqaMt+wIAAGhrrX5PZalUyogRIzJixIgkyZtvvplZs2ZlyZIluf3227N06dL06dMnL730Ul555ZVceOGFzfpo6Szix3HeeeflxRdfzE9+8pPjLi89FfXu3Ts7duxIXV3dca+rZawAAEBb+9jTZD169Gh6Uuq2bduS/P1+xSVLljTb/6233sojjzySUqmUr3zlKx/39Cc0bty4JMnKlStbfEyXLl2S5IT3ipbbkev1q1/9qlnb3r17m+4jBQAAaCutCpWLFi3KSy+91Gz7mjVrkvz9Xr8ZM2akU6dOWbBgQZ566qmm/T744IPMnDkz77//fiZOnHhSnzo6e/bsnH766bnllluyYsWKZu319fVZvnx5du3a1bStT58+SZIdO3actLo+jhtvvDFdunTJQw89lA0bNjRtP3z4cGbPnp133nmnjNUBAAAdUauWvy5atCjTp0/PhRdemAsuuCAVFRV57rnnsmXLllRVVeVHP/pRkmTo0KGZN29ebr/99gwfPjxjxoxJdXV1amtrU1dXlwEDBuTnP//5SRnQEf3798+SJUtyww035Nprr03//v1zwQUX5Iwzzshrr72WZ555Jvv378+f/vSnpoflDBs2LD169Mjy5cszZsyY9OvXL506dcpNN92USy+99KTW2xKf+9zn8uMf/zizZs3K2LFjM3r06PTs2TObNm3K22+/ncmTJ+eXv/xl04wrAADAydaqmcp58+blpptuSqlUyu9+97usWrUq77//fqZNm5Znn332mOWst912Wx5++OGMHj06mzdvzooVK1JZWZk5c+Zk48aN6dmz5yc+mA+75pprsnXr1nz3u99NqVTKo48+mtWrV+fNN9/MVVddlaVLlx5zb2JVVVVWr16dcePG5dlnn83ixYvzwAMP5Pnnnz/ptbbU9773vSxfvjxf+tKX8uSTT+aRRx7J4MGDs3HjxlRVVSVJu7mHFAAA+MdXamxsbCx3EXx8hw8fzkUXXZTt27fn9ddfT69evVp87L59+5peBwPwj6pU6lhPy+5ITwfvaD+KHHlCfUfxxcGXlbuENvP/HvppuUtoU1Ov+Ea5S2gzu3fv+r93aicaGxtz6NAH2bt3b7p169aiYzrO/1jtxIsvvpg9e/Ycs62+vj5z5szJX/7yl1x22WWtCpQAAAAfR6tfKUJ5LVu2LHfccUeGDBmSmpqa7Nu3L1u2bMkbb7yR6urqLFy4sNwlAgAAHYhQeYq57LLLsmXLljz55JPZunVrDh06lHPPPTfTp0/P3LlzT+oTdQEAAD5MqDzFXHLJJcd9/ycAAEA5uKcSAACAwoRKAAAAChMqAQAAKEyoBAAAoDChEgAAgMKESgAAAAoTKgEAAChMqAQAAKAwoRIAAIDChEoAAAAKEyoBAAAoTKgEAACgMKESAACAwoRKAAAAChMqAQAAKKyi3AXwj6SUUqlU7iLaRGNjQ7lL4CQplfyurL2qqOhc7hLa1GmndZzxNjQcKncJberw4Y413vr698pdQpt5+slt5S6hTZ19du9yl9Bm3ntvX7lLaDONjQ3Zt++vrTrGT18AAAAUJlQCAABQmFAJAABAYUIlAAAAhQmVAAAAFCZUAgAAUJhQCQAAQGFCJQAAAIUJlQAAABQmVAIAAFCYUAkAAEBhQiUAAACFCZUAAAAUJlQCAABQmFAJAABAYUIlAAAAhQmVAAAAFCZUAgAAUJhQCQAAQGFCJQAAAIUJlQAAABQmVAIAAFCYUAkAAEBhQiUAAACFCZUAAAAUJlQCAABQmFAJAABAYUIlAAAAhVWUuwDaXn19ferr65u+3rdvXxmrAQAATmVmKjug+fPnp3v37k2fmpqacpcEAACcooTKDmju3LnZu3dv06eurq7cJQEAAKcoy187oMrKylRWVpa7DAAAoB0wUwkAAEBhQiUAAACFCZUAAAAUJlS2M//8z/+cgQMHZuXKleUuBQAA6ACEynbm1VdfzY4dO7J3795ylwIAAHQAQiUAAACFeaVIO7N+/fpylwAAAHQgZioBAAAoTKgEAACgMKESAACAwoRKAAAAChMqAQAAKEyoBAAAoDChEgAAgMKESgAAAAoTKgEAAChMqAQAAKAwoRIAAIDChEoAAAAKEyoBAAAoTKgEAACgMKESAACAwoRKAAAAChMqAQAAKEyoBAAAoDChEgAAgMIqyl0A5dfY2HjkT2n6I5yiGv0jbrc62t9tRxpvRxpr0vHGe/jwoXKX0GYOvP9euUtoUx3p77axsaHcJbSZI2NtzfeqUmNH+85GM7t27UpNTU25ywAAAP5B1NXV5bzzzmvRvkIlaWhoyOuvv54zzzwzpVKpzc67b9++1NTUpK6uLt26dWuz85ZDRxprYrztWUcaa9KxxtuRxpoYb3vWkcaaGG97Vq6xNjY25p133kmfPn3SqVPL7pa0/JV06tSpxb+FOBm6devW7r8pHNGRxpoYb3vWkcaadKzxdqSxJsbbnnWksSbG256VY6zdu3dv1f4e1AMAAEBhQiUAAACFCZWUTWVlZe64445UVlaWu5STriONNTHe9qwjjTXpWOPtSGNNjLc960hjTYy3PTuVxupBPQAAABRmphIAAIDChEoAAAAKEyoBAAAoTKgEAACgMKESAACAwoRKAAAAChMqAQAAKEyoBAAAoLD/D0qqD64MWSvzAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_attention(sentence, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BLEU\n",
    "\n",
    "以前我们只关心模型的损失/困惑度。然而，有一些专门用于测量翻译质量的指标 - 最流行的是 *BLEU*。不详细介绍，BLEU 从 n-gram 的角度考察了预测目标序列与实际目标序列的重叠程度。它会给我们每个序列一个介于 0 和 1 之间的数字，其中 1 表示完全重叠，即完美翻译，尽管通常显示在 0 和 100 之间。BLEU 最初是为每个源序列有多个候选翻译设计的，但在这个数据集中，每个源序列只有一个候选翻译。\n",
    "\n",
    "我们从`evaluate`库中加载BLEU指标： 它用来计算数据集上的 BLEU 分数。该函数为每个源语句创建了实际翻译和预测翻译的语料库，然后计算 BLEU 分数。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "bleu = evaluate.load(\"bleu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "循环遍历我们的 `test_data`，得到模型对每个测试句子的翻译。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e420a86028d245ab9b195cd0152b927e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "translations = [\n",
    "    translate_sentence(\n",
    "        example[\"de\"],\n",
    "        model,\n",
    "        en_nlp,\n",
    "        de_nlp,\n",
    "        en_vocab,\n",
    "        de_vocab,\n",
    "        lower,\n",
    "        sos_token,\n",
    "        eos_token, \n",
    "    )[0] for example in tqdm(test_data)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = [\n",
    "    \" \".join(translation[1:-1]) for translation in translations\n",
    "]\n",
    "\n",
    "references = [\n",
    "    [example[\"en\"]] for example in test_data\n",
    "]\n",
    "\n",
    "def get_tokenizer_fn(nlp, lower):\n",
    "    \n",
    "    def tokenizer_fn(s):\n",
    "        tokens = [token.text for token in nlp.tokenizer(s)]\n",
    "        if lower:\n",
    "            tokens = [token.lower() for token in tokens]\n",
    "        return tokens\n",
    "        \n",
    "    return tokenizer_fn\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bleu': 0.29882945217820545,\n",
       " 'precisions': [0.642085337960407,\n",
       "  0.3826526243685247,\n",
       "  0.23841183631426163,\n",
       "  0.15342494059303646],\n",
       " 'brevity_penalty': 0.970550398459947,\n",
       " 'length_ratio': 0.970975647112881,\n",
       " 'translation_length': 12679,\n",
       " 'reference_length': 13058}"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_fn = get_tokenizer_fn(en_nlp, lower)\n",
    "results = bleu.compute(predictions=predictions, references=references, tokenizer=tokenizer_fn)\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们得到了约 0.299 的 BLEU 分数。如果我们将其与注意力模型试图复制的论文进行比较，与我们的分数相似，但是他们使用的是一个完全不同的数据集，并且他们的模型尺寸要大得多 - 1000 个隐藏维度，需要 4 天时间进行训练！- 因此我们不能真正与其进行比较。\n",
    "\n",
    "这个数字并不真正可解释，我们不能真正从中得出什么结论。BLEU 分数最有用的部分是它可以用来比较同一数据集上的不同模型，其中 BLEU 分数**较高**的模型更“好”。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在接下来的教程中，我们将摆脱使用循环神经网络，并开始探讨构建序列到序列模型的其他方法。具体来说，在下一个教程中，我们将使用卷积神经网络。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}