3 - Neural Machine Translation by Jointly Learning to Align and Translate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 - 通过联合学习对齐和翻译实现神经机器翻译\n",
    "\n",
    "在这个使用Mindspore的序列到序列模型的第三个笔记本中，我们将实现来自[Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473)的模型。与先前模型相比，该模型实现了我们迄今为止的最佳困惑度。\n",
    "\n",
    "## 简介\n",
    "\n",
    "作为提醒，这是一般的编码器-解码器模型：\n",
    "\n",
    "![](assets/seq2seq1.png)\n",
    "\n",
    "在先前的模型中，我们的架构被设置为通过在每个时间步将上下文向量$z$明确传递给解码器，并通过在每个时间步将上下文向量和嵌入的输入词$d(y_t)$以及隐藏状态$s_t$传递给线性层$f$来减少“信息压缩”。\n",
    "\n",
    "![](assets/seq2seq7.png)\n",
    "\n",
    "即使我们减少了一些这种压缩，我们的上下文向量仍然需要包含有关源句的所有信息。本笔记本中实现的模型通过允许解码器在每个解码步骤中查看整个源句（通过其隐藏状态）来避免此压缩！它是如何做到这一点的呢？它使用*注意力*。\n",
    "\n",
    "注意力通过首先计算一个与源句长度相同的注意力向量$a$来工作。注意向量具有每个元素都在0到1之间的属性，并且整个向量总和为1。然后，我们计算我们的源句隐藏状态$H$的加权和，以获得加权源向量$w$。\n",
    "\n",
    "$$w = \\sum_{i}a_ih_i$$\n",
    "\n",
    "在解码时，我们在每个时间步计算一个新的加权源向量，将其用作解码器RNN的输入，以及用于进行预测的线性层。我们将在教程中解释如何完成所有这些。\n",
    "\n",
    "## 准备数据\n",
    "\n",
    "再次，准备工作与上次相似。\n",
    "\n",
    "首先，导入所有所需的模块。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore as ms\n",
    "import mindspore.context as context\n",
    "import mindspore.nn as nn\n",
    "import mindspore.ops as ops\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "import spacy\n",
    "import datasets\n",
    "from tqdm.notebook import tqdm\n",
    "import evaluate\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置随机数种子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 1234\n",
    "\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "ms.set_seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "加载数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset json (/root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09f95624c8344b7ab46d39a22c4750e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset = datasets.load_dataset(\"bentrevett/multi30k\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, valid_data, test_data = dataset[\"train\"], dataset[\"validation\"], dataset[\"test\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_nlp = spacy.load(\"en_core_web_sm\")\n",
    "de_nlp = spacy.load(\"de_core_news_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_example(\n",
    "    example,\n",
    "    en_nlp,\n",
    "    de_nlp,\n",
    "    max_length,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token\n",
    "):\n",
    "    en_tokens = [token.text for token in en_nlp.tokenizer(example[\"en\"])][:max_length]\n",
    "    de_tokens = [token.text for token in de_nlp.tokenizer(example[\"de\"])][:max_length]\n",
    "    if lower:\n",
    "        en_tokens = [token.lower() for token in en_tokens]\n",
    "        de_tokens = [token.lower() for token in de_tokens]\n",
    "    en_tokens = [sos_token] + en_tokens + [eos_token]\n",
    "    de_tokens = [sos_token] + de_tokens + [eos_token]\n",
    "    return {\"en_tokens\": en_tokens, \"de_tokens\": de_tokens}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-39e2a81dd5826464.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-340eed13fdfe768e.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-05a87c7dda3754fe.arrow\n"
     ]
    }
   ],
   "source": [
    "max_length = 1_000\n",
    "lower = True\n",
    "sos_token = \"<sos>\"\n",
    "eos_token = \"<eos>\"\n",
    "\n",
    "fn_kwargs = {\n",
    "    \"en_nlp\": en_nlp, \n",
    "    \"de_nlp\": de_nlp, \n",
    "    \"max_length\": max_length,\n",
    "    \"lower\": lower,\n",
    "    \"sos_token\": sos_token,\n",
    "    \"eos_token\": eos_token,\n",
    "}\n",
    "\n",
    "\n",
    "train_data = train_data.map(tokenize_example, fn_kwargs=fn_kwargs)\n",
    "valid_data = valid_data.map(tokenize_example, fn_kwargs=fn_kwargs)\n",
    "test_data = test_data.map(tokenize_example, fn_kwargs=fn_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "class Vocab:\n",
    "    \"\"\"一个词汇表的实现\"\"\"\n",
    "    def __init__(self, tokens:list, min_freq=0, reserved_tokens:list=None) -> None:\n",
    "        self.default_index = None\n",
    "        if tokens is not None:\n",
    "            # 当第一个条件满足时，就不会跳到第二个判断，避免了空列表报错的情况。\n",
    "            if len(tokens)!=0 and isinstance(tokens[0], list):\n",
    "                tokens = [i for line in tokens for i in line]\n",
    "        else:\n",
    "            tokens = []\n",
    "        if reserved_tokens is None:\n",
    "            reserved_tokens = []\n",
    "        counter=collections.Counter(tokens)\n",
    "        # 按出现词频从高到低排序\n",
    "        self._token_freqs = sorted(counter.items(), key=lambda x:x[1], reverse=True)\n",
    "        # 通过列表,利用序号访问词元。\n",
    "        self.idx_to_token = [] + reserved_tokens # 未知词元<unk>的索引为0, 保留词元排在最前\n",
    "        self.token_to_idx = {\n",
    "            i: k\n",
    "            for k, i in enumerate(self.idx_to_token) \n",
    "        }\n",
    "        \n",
    "        for token, freq in self._token_freqs:\n",
    "            if freq < min_freq:  # 过滤掉出现频率低于要求的词\n",
    "                break\n",
    "            if token not in self.token_to_idx:  \n",
    "                self.idx_to_token.append(token)\n",
    "                self.token_to_idx[token] = len(self.idx_to_token) - 1\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.idx_to_token)\n",
    "    \n",
    "    def __getitem__(self, input_tokens):\n",
    "        \"\"\"输入单字串或序列, 将其全部转化为序号编码\"\"\"\n",
    "        if isinstance(input_tokens, str):\n",
    "            out =  self.token_to_idx.get(input_tokens, self.default_index)\n",
    "            if out is None:\n",
    "                raise Exception('Please call \"set_default_index\" before getting unknown index')\n",
    "            return out\n",
    "        return [self.__getitem__(token) for token in input_tokens]\n",
    "    \n",
    "    def __repr__(self) -> str:\n",
    "        show_items = 5 if len(self) > 5 else len(self)\n",
    "        out = f\"<Vocab with {len(self)} tokens: \"\n",
    "        for i in range(show_items):\n",
    "            out += f'\"{self.idx_to_token[i]}\", '\n",
    "        out += \"...>\"\n",
    "        return out\n",
    "\n",
    "    def __contains__(self, token:str) -> bool:\n",
    "        return token in self.idx_to_token\n",
    "\n",
    "    def to_tokens(self, input_keys):\n",
    "        \"\"\"输入单s索引或序列, 将其全部转化为词元\"\"\"\n",
    "        if isinstance(input_keys, int):\n",
    "            return self.idx_to_token[input_keys] if input_keys < len(self) else self.idx_to_token[0]\n",
    "        elif isinstance(input_keys, (list, tuple)):\n",
    "            return [self.to_tokens(keys) for keys in input_keys]\n",
    "        else:\n",
    "            return self.idx_to_token[0]\n",
    "    \n",
    "    def get_itos(self):\n",
    "        return self.idx_to_token\n",
    "    \n",
    "    def get_stoi(self):\n",
    "        return self.token_to_idx\n",
    "    \n",
    "    def set_default_index(self, idx):\n",
    "        if isinstance(idx, int):\n",
    "            self.default_index = idx\n",
    "        else:\n",
    "            raise Exception(f\"Only type int allowed, got {type(idx)}\")\n",
    "\n",
    "    def lookup_indices(self, input_tokens):\n",
    "        return self.__getitem__(input_tokens)\n",
    "    \n",
    "    def lookup_tokens(self, idx):\n",
    "        return self.to_tokens(idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_freq = 2\n",
    "unk_token = \"<unk>\"\n",
    "pad_token = \"<pad>\"\n",
    "\n",
    "special_tokens = [\n",
    "    unk_token,\n",
    "    pad_token,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    "]\n",
    "\n",
    "def build_vocab_from_iterator(tokens, min_freq, specials):\n",
    "    return Vocab(tokens, min_freq, specials)\n",
    "\n",
    "en_vocab = build_vocab_from_iterator(\n",
    "    train_data[\"en_tokens\"],\n",
    "    min_freq=min_freq,\n",
    "    specials=special_tokens,\n",
    ")\n",
    "\n",
    "de_vocab = build_vocab_from_iterator(\n",
    "    train_data[\"de_tokens\"],\n",
    "    min_freq=min_freq,\n",
    "    specials=special_tokens,  \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert en_vocab[unk_token] == de_vocab[unk_token]\n",
    "assert en_vocab[pad_token] == de_vocab[pad_token]\n",
    "\n",
    "unk_index = en_vocab[unk_token]\n",
    "pad_index = en_vocab[pad_token]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_vocab.set_default_index(unk_index)\n",
    "de_vocab.set_default_index(unk_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def numericalize_example(example, en_vocab, de_vocab):\n",
    "    en_ids = en_vocab.lookup_indices(example[\"en_tokens\"])\n",
    "    de_ids = de_vocab.lookup_indices(example[\"de_tokens\"])\n",
    "    return {\"en_ids\": en_ids, \"de_ids\": de_ids}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-6689c04abfdb721f.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-acab783df695b18d.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-8cca6da32304eb4d/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-e1088b8589007c26.arrow\n"
     ]
    }
   ],
   "source": [
    "fn_kwargs = {\n",
    "    \"en_vocab\": en_vocab, \n",
    "    \"de_vocab\": de_vocab\n",
    "}\n",
    "\n",
    "train_data = train_data.map(numericalize_example, fn_kwargs=fn_kwargs)\n",
    "valid_data = valid_data.map(numericalize_example, fn_kwargs=fn_kwargs)\n",
    "test_data = test_data.map(numericalize_example, fn_kwargs=fn_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_type = \"numpy\"\n",
    "format_columns = [\"en_ids\", \"de_ids\"]\n",
    "\n",
    "train_data = train_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True\n",
    ")\n",
    "\n",
    "valid_data = valid_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True,\n",
    ")\n",
    "\n",
    "test_data = test_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_sequence(sequences:list, padding_value:int):\n",
    "    '''将序列填充到等长并返回mindspore张量'''\n",
    "    # Find the length of the longest sequence in the batch\n",
    "    max_length = max(len(seq) for seq in sequences)\n",
    "    padded_sequences:ms.Tensor = ops.full((len(sequences), max_length), padding_value, dtype=ms.int64)\n",
    "    # Copy the sequences into the padded array\n",
    "    for i, seq in enumerate(sequences):\n",
    "        padded_sequences[i, :len(seq)] = ms.tensor(seq).astype(np.int64)\n",
    "    # 换轴，保证输出为时序优先\n",
    "    padded_sequences = padded_sequences.swapaxes(0, 1)\n",
    "    return padded_sequences  \n",
    "\n",
    "def get_collate_fn(pad_index):\n",
    "    \n",
    "    def collate_fn(batch):\n",
    "        batch_en_ids = [example[\"en_ids\"] for example in batch]\n",
    "        batch_de_ids = [example[\"de_ids\"] for example in batch]\n",
    "        batch_en_ids = pad_sequence(batch_en_ids, padding_value=pad_index)\n",
    "        batch_de_ids = pad_sequence(batch_de_ids, padding_value=pad_index)\n",
    "        batch = {\n",
    "            \"en_ids\": batch_en_ids,\n",
    "            \"de_ids\": batch_de_ids,\n",
    "        }\n",
    "        return batch\n",
    "    \n",
    "    return collate_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataLoader:\n",
    "    def __init__(self, source, batch_size, shuffle=False, per_batch_map=None):\n",
    "        self.source = source\n",
    "        self.batch_size = batch_size\n",
    "        self.shuffle = shuffle\n",
    "        self.per_batch_map = per_batch_map\n",
    "        self.indices = np.arange(len(source))\n",
    "        self.current_index = 0\n",
    "\n",
    "        if self.shuffle:\n",
    "            np.random.shuffle(self.indices)\n",
    "\n",
    "    def __iter__(self):\n",
    "        return self\n",
    "\n",
    "    def __next__(self):\n",
    "        if self.current_index >= len(self.source):\n",
    "            self.current_index = 0\n",
    "            raise StopIteration\n",
    "\n",
    "        batch_indices = self.indices[self.current_index:self.current_index + self.batch_size]\n",
    "        batch_data = [self.source[int(i)] for i in batch_indices]\n",
    "\n",
    "        if self.per_batch_map:\n",
    "            batch_data = self.per_batch_map(batch_data)\n",
    "\n",
    "        self.current_index += self.batch_size\n",
    "        return batch_data\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.source) // self.batch_size\n",
    "\n",
    "\n",
    "def get_data_loader(dataset, batch_size, pad_index, shuffle=False):\n",
    "    collate_fn = get_collate_fn(pad_index)\n",
    "    dataloader = DataLoader(dataset, batch_size, shuffle=shuffle, per_batch_map=collate_fn)\n",
    "    \n",
    "    return dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "\n",
    "train_data_loader = get_data_loader(train_data, batch_size, pad_index, shuffle=True)\n",
    "valid_data_loader = get_data_loader(valid_data, batch_size, pad_index)\n",
    "test_data_loader = get_data_loader(test_data, batch_size, pad_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建Seq2Seq模型\n",
    "\n",
    "### 编码器\n",
    "\n",
    "首先，我们将构建编码器。与之前的模型类似，我们仅使用一个单层GRU，但现在我们使用了*双向RNN*。在双向RNN中，每一层都有两个RNN。一个*RNN向前*从左到右遍历嵌入的句子（下图中为绿色），一个*RNN向后*从右到左遍历嵌入的句子（青色）。在代码中，我们只需设置 `bidirectional = True`，然后像以前一样将嵌入的句子传递给RNN。\n",
    "\n",
    "![](assets/seq2seq8.png)\n",
    "\n",
    "现在我们有：\n",
    "\n",
    "$$\\begin{align*}\n",
    "h_t^\\rightarrow &= \\text{EncoderGRU}^\\rightarrow(e(x_t^\\rightarrow),h_{t-1}^\\rightarrow)\\\\\n",
    "h_t^\\leftarrow &= \\text{EncoderGRU}^\\leftarrow(e(x_t^\\leftarrow),h_{t-1}^\\leftarrow)\n",
    "\\end{align*}$$\n",
    "\n",
    "其中 $x_0^\\rightarrow = \\text{<sos>}, x_1^\\rightarrow = \\text{guten}$ 和 $x_0^\\leftarrow = \\text{<eos>}, x_1^\\leftarrow = \\text{morgen}$。\n",
    "\n",
    "与之前一样，我们只将输入 (`embedded`) 传递给RNN，告诉Mindspore将前向和后向初始隐藏状态（分别为 $h_0^\\rightarrow$ 和 $h_0^\\leftarrow$）初始化为全零张量。我们还会得到两个上下文向量，一个来自前向RNN在看到句子的最后一个词后，$z^\\rightarrow=h_T^\\rightarrow$，一个来自后向RNN在看到句子的第一个词后，$z^\\leftarrow=h_T^\\leftarrow$。\n",
    "\n",
    "RNN返回 `outputs` 和 `hidden`。\n",
    "\n",
    "`outputs` 的大小为 **[src len, batch size, hid dim * num directions]**，其中第三轴的前 `hid_dim` 元素是从顶层前向RNN获得的隐藏状态，最后的 `hid_dim` 元素是从顶层后向RNN获得的隐藏状态。我们可以将第三轴视为前向和后向隐藏状态在一起连接的地方，即 $h_1 = [h_1^\\rightarrow; h_{T}^\\leftarrow]$，$h_2 = [h_2^\\rightarrow; h_{T-1}^\\leftarrow]$，我们可以将所有编码器隐藏状态（前向和后向连接在一起）表示为 $H=\\{ h_1, h_2, ..., h_T\\}$。\n",
    "\n",
    "`hidden` 的大小为 **[n layers * num directions, batch size, hid dim]**，其中 **[-2, :, :]** 给出了顶层前向RNN在最后一个时间步（即在看到句子的最后一个词后）之后的隐藏状态，**[-1, :, :]** 给出了顶层后向RNN在最后一个时间步（即在看到句子的第一个词后）之后的隐藏状态。\n",
    "\n",
    "由于解码器不是双向的，它只需要一个上下文向量 $z$ 用作初始隐藏状态 $s_0$，而我们目前有两个，一个前向的和一个后向的（$z^\\rightarrow=h_T^\\rightarrow$ 和 $z^\\leftarrow=h_T^\\leftarrow$）。我们通过将两个上下文向量连接在一起，将它们传递给一个线性层 $g$ 并应用 $\\tanh$ 激活函数来解决这个问题。\n",
    "\n",
    "$$z=\\tanh(g(h_T^\\rightarrow, h_T^\\leftarrow)) = \\tanh(g(z^\\rightarrow, z^\\leftarrow)) = s_0$$\n",
    "\n",
    "**注意**：这实际上是与论文中的方法有所不同。在论文中，他们仅通过线性层传递第一个后向RNN隐藏状态以获得上下文向量/解码器初始隐藏状态。这对我来说看起来不合理，所以我们进行了修改。\n",
    "\n",
    "由于我们希望我们的模型能够查看整个源句子，我们返回 `outputs`，即源句子中每个标记的堆叠的前向和后向隐藏状态。我们还返回 `hidden`，它充当解码器中的初始隐藏状态。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Cell):\n",
    "    def __init__(self, input_dim, emb_dim, enc_hid_dim, dec_hid_dim, dropout):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_dim, emb_dim)\n",
    "        self.rnn = nn.GRU(emb_dim, enc_hid_dim, bidirectional = True)\n",
    "        self.fc = nn.Dense(enc_hid_dim * 2, dec_hid_dim)\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "        \n",
    "    def construct(self, src):\n",
    "        \n",
    "        #src = [src len, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(src))\n",
    "        \n",
    "        #embedded = [src len, batch size, emb dim]\n",
    "        \n",
    "        outputs, hidden = self.rnn(embedded)\n",
    "                \n",
    "        #outputs = [src len, batch size, hid dim * num directions]\n",
    "        #hidden = [n layers * num directions, batch size, hid dim]\n",
    "        \n",
    "        #hidden is stacked [forward_1, backward_1, forward_2, backward_2, ...]\n",
    "        #outputs are always from the last layer\n",
    "        \n",
    "        #hidden [-2, :, : ] is the last of the forwards RNN \n",
    "        #hidden [-1, :, : ] is the last of the backwards RNN\n",
    "        \n",
    "        #initial decoder hidden is final hidden state of the forwards and backwards \n",
    "        #  encoder RNNs fed through a linear layer\n",
    "        hidden = ops.tanh(self.fc(ops.cat((hidden[-2,:,:], hidden[-1,:,:]), axis=1)))\n",
    "        \n",
    "        #outputs = [src len, batch size, enc hid dim * 2]\n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        \n",
    "        return outputs, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention\n",
    "\n",
    "接下来是注意力层。它将接收到解码器的前一个隐藏状态 $s_{t-1}$ 和编码器的所有堆叠的前向和后向隐藏状态 $H$。该层将输出一个注意力向量 $a_t$，其长度为源句子的长度，其中每个元素在 0 和 1 之间，整个向量的和为 1。\n",
    "\n",
    "直观地说，该层利用我们到目前为止已解码的内容 $s_{t-1}$ 和我们已编码的所有内容 $H$ 来生成一个向量 $a_t$，该向量表示为了正确预测下一个要解码的单词 $\\hat{y}_{t+1}$，我们应该最关注源句子中的哪些单词。\n",
    "\n",
    "首先，我们计算前一个解码器隐藏状态和编码器隐藏状态之间的 *能量*。由于我们的编码器隐藏状态是 $T$ 个张量的序列，而我们的前一个解码器隐藏状态是单个张量，我们首先使用 `repeat` 将前一个解码器隐藏状态重复 $T$ 次。然后，通过将它们连接在一起并通过一个线性层 (`attn`) 和一个 $\\tanh$ 激活函数，计算它们之间的能量 $E_t$。\n",
    "\n",
    "$$E_t = \\tanh(\\text{attn}(s_{t-1}, H))$$\n",
    "\n",
    "这可以看作是计算每个编码器隐藏状态与前一个解码器隐藏状态的 \"匹配程度\"。\n",
    "\n",
    "我们目前对于批处理中的每个示例都有一个 **[dec hid dim, src len]** 张量。我们希望对于批处理中的每个示例，这个张量都是 **[src len]**，因为注意力应该覆盖源句子的长度。这通过将 `energy` 乘以 **[1, dec hid dim]** 张量 $v$ 来实现。\n",
    "\n",
    "$$\\hat{a}_t = v E_t$$\n",
    "\n",
    "我们可以将 $v$ 视为加权求和的权重，这个权重遍历所有编码器隐藏状态的能量。这些权重告诉我们在源序列中应该注意每个标记的程度。 $v$ 的参数是随机初始化的，但会随着模型的其余部分一起通过反向传播进行学习。请注意，$v$ 不依赖于时间，并且在解码的每个时间步骤中都使用相同的 $v$。我们将 $v$ 实现为一个没有偏差的线性层。\n",
    "\n",
    "最后，通过通过一个 $\\text{softmax}$ 层，我们确保注意力向量符合所有元素都在 0 和 1 之间且向量总和为 1 的约束。\n",
    "\n",
    "$$a_t = \\text{softmax}(\\hat{a_t})$$\n",
    "\n",
    "这给我们提供了源句子上的注意力！\n",
    "\n",
    "从图形上看，如下所示。这是用于计算第一个注意力向量的情况，其中 $s_{t-1} = s_0 = z$。绿色/青色块表示前向和后向RNN的隐藏状态，注意力计算完全在粉色块内完成。\n",
    "\n",
    "![](assets/seq2seq9.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Attention(nn.Cell):\n",
    "    def __init__(self, enc_hid_dim, dec_hid_dim):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.attn = nn.Dense((enc_hid_dim * 2) + dec_hid_dim, dec_hid_dim)\n",
    "        self.v = nn.Dense(dec_hid_dim, 1, has_bias = False)\n",
    "        \n",
    "    def construct(self, hidden:ms.Tensor, encoder_outputs:ms.Tensor):\n",
    "        \n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        #encoder_outputs = [src len, batch size, enc hid dim * 2]\n",
    "        \n",
    "        batch_size = encoder_outputs.shape[1]\n",
    "        src_len = encoder_outputs.shape[0]\n",
    "        \n",
    "        #repeat decoder hidden state src_len times\n",
    "        hidden = hidden.unsqueeze(1).tile((1, src_len, 1))\n",
    "        \n",
    "        encoder_outputs = encoder_outputs.permute(1, 0, 2)\n",
    "        \n",
    "        #hidden = [batch size, src len, dec hid dim]\n",
    "        #encoder_outputs = [batch size, src len, enc hid dim * 2]\n",
    "        \n",
    "        energy = ops.tanh(self.attn(ops.cat((hidden, encoder_outputs), axis = 2))) \n",
    "        \n",
    "        #energy = [batch size, src len, dec hid dim]\n",
    "\n",
    "        attention = self.v(energy).squeeze(2)\n",
    "        \n",
    "        #attention= [batch size, src len]\n",
    "        \n",
    "        return ops.softmax(attention, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoder\n",
    "\n",
    "接下来是解码器。\n",
    "\n",
    "解码器包含注意力层 `attention`，该层接收前一个隐藏状态 $s_{t-1}$，所有编码器隐藏状态 $H$，并返回注意力向量 $a_t$。\n",
    "\n",
    "然后，我们使用这个注意力向量来创建加权源向量 $w_t$，表示为 `weighted`，它是使用 $a_t$ 作为权重对编码器隐藏状态 $H$ 进行加权求和。\n",
    "\n",
    "$$w_t = a_t H$$\n",
    "\n",
    "嵌入的输入单词 $d(y_t)$、加权源向量 $w_t$ 和前一个解码器隐藏状态 $s_{t-1}$ 随后都传入解码器 RNN，其中 $d(y_t)$ 和 $w_t$ 被连接在一起。\n",
    "\n",
    "$$s_t = \\text{DecoderGRU}(d(y_t), w_t, s_{t-1})$$\n",
    "\n",
    "然后，我们通过线性层 $f$ 传递 $d(y_t)$、$w_t$ 和 $s_t$，以预测目标句子中的下一个单词 $\\hat{y}_{t+1}$。这通过将它们全部连接在一起来实现。\n",
    "\n",
    "$$\\hat{y}_{t+1} = f(d(y_t), w_t, s_t)$$\n",
    "\n",
    "下图显示了对示例翻译中的第一个单词进行解码。\n",
    "\n",
    "![](assets/seq2seq10.png)\n",
    "\n",
    "绿色/青色块显示了前向/后向编码器RNN，其输出为 $H$；红色块显示了上下文向量 $z = h_T = \\tanh(g(h^\\rightarrow_T,h^\\leftarrow_T)) = \\tanh(g(z^\\rightarrow, z^\\leftarrow)) = s_0$；蓝色块显示了解码器RNN，其输出为 $s_t$；紫色块显示了线性层 $f$，其输出为 $\\hat{y}_{t+1}$；橙色块显示了通过 $a_t$ 对 $H$ 进行加权求和的计算，并输出 $w_t$。没有显示 $a_t$ 的计算。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Cell):\n",
    "    def __init__(self, output_dim:int, emb_dim:int, enc_hid_dim:int, dec_hid_dim:int, dropout, attention:Attention):\n",
    "        super().__init__()\n",
    "\n",
    "        self.output_dim = output_dim\n",
    "        self.attention = attention\n",
    "        \n",
    "        self.embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.GRU((enc_hid_dim * 2) + emb_dim, dec_hid_dim)\n",
    "        \n",
    "        self.fc_out = nn.Dense((enc_hid_dim * 2) + dec_hid_dim + emb_dim, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "        \n",
    "    def construct(self, input:ms.Tensor, hidden:ms.Tensor, encoder_outputs:ms.Tensor):\n",
    "             \n",
    "        #input = [batch size]\n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        #encoder_outputs = [src len, batch size, enc hid dim * 2]\n",
    "        \n",
    "        input = input.unsqueeze(0)\n",
    "        \n",
    "        #input = [1, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        \n",
    "        #embedded = [1, batch size, emb dim]\n",
    "        \n",
    "        a = self.attention(hidden, encoder_outputs)\n",
    "                \n",
    "        #a = [batch size, src len]\n",
    "        \n",
    "        a = a.unsqueeze(1)\n",
    "        \n",
    "        #a = [batch size, 1, src len]\n",
    "        \n",
    "        encoder_outputs = encoder_outputs.permute(1, 0, 2)\n",
    "        \n",
    "        #encoder_outputs = [batch size, src len, enc hid dim * 2]\n",
    "        \n",
    "        weighted = ops.bmm(a, encoder_outputs)\n",
    "        \n",
    "        #weighted = [batch size, 1, enc hid dim * 2]\n",
    "        \n",
    "        weighted = weighted.permute(1, 0, 2)\n",
    "        \n",
    "        #weighted = [1, batch size, enc hid dim * 2]\n",
    "        \n",
    "        rnn_input = ops.cat((embedded, weighted), axis = 2)\n",
    "        \n",
    "        #rnn_input = [1, batch size, (enc hid dim * 2) + emb dim]\n",
    "            \n",
    "        output, hidden = self.rnn(rnn_input, hidden.unsqueeze(0))\n",
    "        \n",
    "        #output = [seq len, batch size, dec hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, dec hid dim]\n",
    "        \n",
    "        #seq len, n layers and n directions will always be 1 in this decoder, therefore:\n",
    "        #output = [1, batch size, dec hid dim]\n",
    "        #hidden = [1, batch size, dec hid dim]\n",
    "        #this also means that output == hidden\n",
    "        assert (output == hidden).all()\n",
    "        \n",
    "        embedded = embedded.squeeze(0)\n",
    "        output = output.squeeze(0)\n",
    "        weighted = weighted.squeeze(0)\n",
    "        \n",
    "        prediction = self.fc_out(ops.cat((output, weighted, embedded), axis = 1))\n",
    "        \n",
    "        #prediction = [batch size, output dim]\n",
    "        \n",
    "        return prediction, hidden.squeeze(0), a.squeeze(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seq2Seq\n",
    "\n",
    "这是第一个不需要使编码器RNN和解码器RNN具有相同隐藏维度的模型，但是编码器必须是双向的。可以通过将所有出现的 `enc_dim * 2` 更改为 `enc_dim * 2 if encoder_is_bidirectional else enc_dim` 来移除此要求。\n",
    "\n",
    "这个seq2seq封装器与前两个类似。唯一的区别是 `encoder` 返回最终隐藏状态（即从前向和后向编码器RNN传递过线性层的最终隐藏状态），用作解码器的初始隐藏状态，以及每个隐藏状态（即前向和后向隐藏状态堆叠在一起）。我们还需要确保将 `hidden` 和 `encoder_outputs` 传递给解码器。\n",
    "\n",
    "简要概述所有步骤：\n",
    "- 创建 `outputs` 张量以保存所有预测 $\\hat{Y}$\n",
    "- 将源序列 $X$ 输入编码器以接收 $z$ 和 $H$\n",
    "- 将初始解码器隐藏状态设置为 `context` 向量，$s_0 = z = h_T$\n",
    "- 使用一批 `<sos>` 标记作为第一个 `input`，$y_1$\n",
    "- 然后在循环内解码：\n",
    "  - 将输入标记 $y_t$、前一个隐藏状态 $s_{t-1}$ 和所有编码器输出 $H$ 插入解码器\n",
    "  - 接收预测 $\\hat{y}_{t+1}$ 和新的隐藏状态 $s_t$\n",
    "  - 然后决定是否使用 teacher force，设置下一个输入为适当的值\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Cell):\n",
    "    def __init__(self, encoder, decoder):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.encoder:Encoder = encoder\n",
    "        self.decoder:Decoder = decoder\n",
    "        \n",
    "    def construct(self, src:ms.Tensor, trg:ms.Tensor, teacher_forcing_ratio = 0.5):\n",
    "        \n",
    "        #src = [src len, batch size]\n",
    "        #trg = [trg len, batch size]\n",
    "        #teacher_forcing_ratio is probability to use teacher forcing\n",
    "        #e.g. if teacher_forcing_ratio is 0.75 we use teacher forcing 75% of the time\n",
    "        \n",
    "        batch_size = src.shape[1]\n",
    "        trg_len = trg.shape[0]\n",
    "        trg_vocab_size = self.decoder.output_dim\n",
    "        \n",
    "        #tensor to store decoder outputs\n",
    "        # outputs = torch.zeros(trg_len, batch_size, trg_vocab_size).to(self.device)\n",
    "        outputs = []\n",
    "        #encoder_outputs is all hidden states of the input sequence, back and forwards\n",
    "        #hidden is the final forward and backward hidden states, passed through a linear layer\n",
    "        encoder_outputs, hidden = self.encoder(src)\n",
    "                \n",
    "        #first input to the decoder is the <sos> tokens\n",
    "        input = trg[0,:]\n",
    "        \n",
    "        for t in range(1, trg_len):\n",
    "            #insert input token embedding, previous hidden state and all encoder hidden states\n",
    "            #receive output tensor (predictions) and new hidden state\n",
    "            output, hidden, _ = self.decoder(input, hidden, encoder_outputs)\n",
    "            if len(outputs) == 0:\n",
    "                outputs.append(ops.zeros(output.shape, dtype=output.dtype))\n",
    "            #place predictions in a tensor holding predictions for each token\n",
    "            # outputs[t] = output\n",
    "            outputs.append(output)\n",
    "            \n",
    "            #get the highest predicted token from our predictions\n",
    "            top1 = output.argmax(1) \n",
    "            if self.training:\n",
    "                teacher_force = random.random() < teacher_forcing_ratio\n",
    "                # get the highest predicted token from our predictions\n",
    "                \n",
    "                # if teacher forcing, use actual next token as next input\n",
    "                # if not, use predicted token\n",
    "                input = trg[t] if teacher_force else top1\n",
    "                # input = [batch size]\n",
    "            else:\n",
    "                input = top1\n",
    "        # return outputs\n",
    "        return ops.stack(outputs, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim = len(de_vocab)\n",
    "output_dim = len(en_vocab)\n",
    "encoder_embedding_dim = 256\n",
    "decoder_embedding_dim = 256\n",
    "encoder_hidden_dim = 512\n",
    "decoder_hidden_dim = 512\n",
    "encoder_dropout = 0.5\n",
    "decoder_dropout = 0.5\n",
    "\n",
    "attention = Attention(encoder_hidden_dim, decoder_hidden_dim)\n",
    "\n",
    "encoder = Encoder(\n",
    "    input_dim,\n",
    "    encoder_embedding_dim,\n",
    "    encoder_hidden_dim,\n",
    "    decoder_hidden_dim,\n",
    "    encoder_dropout,\n",
    ")\n",
    "\n",
    "decoder = Decoder(\n",
    "    output_dim,\n",
    "    decoder_embedding_dim,\n",
    "    encoder_hidden_dim,\n",
    "    decoder_hidden_dim,\n",
    "    decoder_dropout,\n",
    "    attention,\n",
    ")\n",
    "\n",
    "\n",
    "model = Seq2Seq(encoder, decoder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Seq2Seq<\n",
       "  (encoder): Encoder<\n",
       "    (embedding): Embedding<vocab_size=7853, embedding_size=256, use_one_hot=False, embedding_table=Parameter (name=encoder.embedding.embedding_table, shape=(7853, 256), dtype=Float32, requires_grad=True), dtype=Float32, padding_idx=None>\n",
       "    (rnn): GRU<\n",
       "      (rnn): _DynamicGRUCPUGPU<>\n",
       "      (dropout_op): Dropout<p=0.0>\n",
       "      >\n",
       "    (fc): Dense<input_channels=1024, output_channels=512, has_bias=True>\n",
       "    (dropout): Dropout<p=0.5>\n",
       "    >\n",
       "  (decoder): Decoder<\n",
       "    (attention): Attention<\n",
       "      (attn): Dense<input_channels=1536, output_channels=512, has_bias=True>\n",
       "      (v): Dense<input_channels=512, output_channels=1>\n",
       "      >\n",
       "    (embedding): Embedding<vocab_size=5893, embedding_size=256, use_one_hot=False, embedding_table=Parameter (name=decoder.embedding.embedding_table, shape=(5893, 256), dtype=Float32, requires_grad=True), dtype=Float32, padding_idx=None>\n",
       "    (rnn): GRU<\n",
       "      (rnn): _DynamicGRUCPUGPU<>\n",
       "      (dropout_op): Dropout<p=0.0>\n",
       "      >\n",
       "    (fc_out): Dense<input_channels=1792, output_channels=5893, has_bias=True>\n",
       "    (dropout): Dropout<p=0.5>\n",
       "    >\n",
       "  >"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_weights(m:nn.Cell):\n",
    "    for name, param in m.parameters_and_names():\n",
    "        if \"weight\" in name:\n",
    "            param.set_data(ops.normal(param.shape, ms.Tensor([0.]), ms.Tensor([0.01])))\n",
    "        else:\n",
    "            param.set_data(ops.zeros(param.shape, dtype=param.dtype))\n",
    "\n",
    "model.apply(init_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 20,518,405 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.get_parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = nn.Adam(model.trainable_params())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss(ignore_index=pad_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_fn(src, trg, teacher_forcing_ratio):\n",
    "    # src = [src length, batch size]\n",
    "    # trg = [trg length, batch size]\n",
    "    output = model(src, trg, teacher_forcing_ratio)\n",
    "    # output = [trg length, batch size, trg vocab size]\n",
    "    output_dim = output.shape[-1]\n",
    "    output = output[1:].view(-1, output_dim)\n",
    "    # output = [(trg length - 1) * batch size, trg vocab size]\n",
    "    trg = trg[1:].view(-1)\n",
    "    # trg = [(trg length - 1) * batch size]\n",
    "    loss = criterion(output, trg.astype(ms.int32))\n",
    "    return loss\n",
    "\n",
    "grad_fn = ms.value_and_grad(forward_fn, grad_position=None, weights=model.trainable_params())\n",
    "\n",
    "\n",
    "def train_fn(data_loader, optimizer, clip, teacher_forcing_ratio):\n",
    "    epoch_loss = 0.\n",
    "    model.set_train(True)\n",
    "    for i, batch in enumerate(tqdm(data_loader)):\n",
    "        src = batch[\"de_ids\"]\n",
    "        trg = batch[\"en_ids\"]\n",
    "        loss, grads = grad_fn(src, trg, teacher_forcing_ratio)\n",
    "        # grads = ops.clip_by_norm(grads, max_norm=clip)\n",
    "        optimizer(grads)\n",
    "        epoch_loss += float(loss)\n",
    "\n",
    "    return epoch_loss / len(data_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_fn(model:Seq2Seq, data_loader, criterion):\n",
    "    epoch_loss = 0\n",
    "    model.set_train(False)\n",
    "    for i, batch in enumerate(data_loader):\n",
    "        src = batch[\"de_ids\"]\n",
    "        trg = batch[\"en_ids\"]\n",
    "        # src = [src length, batch size]\n",
    "        # trg = [trg length, batch size]\n",
    "        output = model(src, trg, 0) # turn off teacher forcing\n",
    "        # output = [trg length, batch size, trg vocab size]\n",
    "        output_dim = output.shape[-1]\n",
    "        output = output[1:].view(-1, output_dim)\n",
    "        # output = [(trg length - 1) * batch size, trg vocab size]\n",
    "        trg = trg[1:].view(-1)\n",
    "        # trg = [(trg length - 1) * batch size]\n",
    "        loss = criterion(output, trg.astype(ms.int32))\n",
    "        epoch_loss += float(loss)\n",
    "    return epoch_loss / len(data_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2533e892227f4a28ab0acd4fecfde9b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b7dc02ad1d84a71a81ea9b7ef6493a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   5.147 | Train PPL: 171.917\n",
      "\tValid Loss:   5.657 | Valid PPL: 286.161\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e738fd7937b44497b140ec1394836d54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   4.415 | Train PPL:  82.654\n",
      "\tValid Loss:   5.374 | Valid PPL: 215.730\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "442a349e6d024d2d912a555772671c83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   4.082 | Train PPL:  59.263\n",
      "\tValid Loss:   5.066 | Valid PPL: 158.580\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9cb60df4798c4401bcf49e39e9ff7e1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.625 | Train PPL:  37.507\n",
      "\tValid Loss:   4.516 | Valid PPL:  91.457\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a30d255cdd94621a3359d911c583b55",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.090 | Train PPL:  21.984\n",
      "\tValid Loss:   4.117 | Valid PPL:  61.348\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f8760e675ec64896af40d48c141affe4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.668 | Train PPL:  14.413\n",
      "\tValid Loss:   3.962 | Valid PPL:  52.537\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cf8d01a31b9c4fd9852395258b0b63e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.379 | Train PPL:  10.796\n",
      "\tValid Loss:   3.785 | Valid PPL:  44.028\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df209df17af54c1da1c4d014b019de47",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.100 | Train PPL:   8.164\n",
      "\tValid Loss:   3.870 | Valid PPL:  47.959\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae1b8bace7fb4ec6bc5427f1f16f104b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   1.889 | Train PPL:   6.616\n",
      "\tValid Loss:   3.879 | Valid PPL:  48.373\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9912b19ad7794c98b4b91900678a9c55",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   1.715 | Train PPL:   5.558\n",
      "\tValid Loss:   3.869 | Valid PPL:  47.904\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 10\n",
    "clip = 1.0\n",
    "teacher_forcing_ratio = 0.5\n",
    "\n",
    "best_valid_loss = float(\"inf\")\n",
    "\n",
    "for epoch in tqdm(range(n_epochs)):\n",
    "        \n",
    "    train_loss = train_fn(\n",
    "        train_data_loader, \n",
    "        optimizer, \n",
    "        clip, \n",
    "        teacher_forcing_ratio, \n",
    "    )\n",
    "    \n",
    "    valid_loss = evaluate_fn(\n",
    "        model, \n",
    "        valid_data_loader, \n",
    "        criterion, \n",
    "    )\n",
    "\n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        ms.save_checkpoint(model, f\"./tut3-model.ckpt\")\n",
    "    \n",
    "    print(f\"\\tTrain Loss: {train_loss:7.3f} | Train PPL: {np.exp(train_loss):7.3f}\")\n",
    "    print(f\"\\tValid Loss: {valid_loss:7.3f} | Valid PPL: {np.exp(valid_loss):7.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Test Loss: 3.739 | Test PPL:  42.069 |\n"
     ]
    }
   ],
   "source": [
    "ms.load_checkpoint(f\"tut3-model.ckpt\", model)\n",
    "\n",
    "test_loss = evaluate_fn(model, test_data_loader, criterion)\n",
    "\n",
    "print(f\"| Test Loss: {test_loss:.3f} | Test PPL: {np.exp(test_loss):7.3f} |\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们改进了之前的模型，但代价是训练时间翻倍!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型评估\n",
    "### 翻译与可视化\n",
    "\n",
    "我们首先需要使用模型翻译测试集的每个例子，我们使用下面的 `translate_sentence` 函数来实现这一点。\n",
    "## 推断\n",
    "\n",
    "我们的 `translate_sentence` 将执行以下操作：\n",
    "- 确保我们的模型处于评估模式，对于推断来说，它应该一直是这样的\n",
    "- 如果源句子没有被标记，则对其进行标记\n",
    "- 数字化源句子\n",
    "- 将其转换为张量并添加一个批次维度\n",
    "- 获取源句子的长度并转换为张量\n",
    "- 将源句子输入到编码器中\n",
    "- 创建一个列表来保存输出句子，初始化为 `<sos>` token\n",
    "- 创建一个张量来保存 attention 值\n",
    "- 当我们尚未达到最大长度时\n",
    "  - 获取输入张量，应该是 `<sos>` 或最后预测的 token\n",
    "  - 将输入、所有编码器输出、隐藏状态和 mask 传递到解码器中\n",
    "  - 存储 attention 值\n",
    "  - 获取下一个预测的 token\n",
    "  - 将预测添加到当前输出句子预测中\n",
    "  - 如果预测是 `<eos>` token，则中断\n",
    "- 将输出句子从索引转换为 token\n",
    "- 返回输出句子（去除 `<sos>` token）和对整个序列的 attention 值\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate_sentence(\n",
    "    sentence, \n",
    "    model:Seq2Seq,\n",
    "    en_nlp:spacy.Language,\n",
    "    de_nlp:spacy.Language,\n",
    "    en_vocab:Vocab,\n",
    "    de_vocab:Vocab,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    "    max_output_length=25,\n",
    "):\n",
    "    model.set_train(False)\n",
    "    if isinstance(sentence, str):\n",
    "        tokens = [token.text for token in de_nlp.tokenizer(sentence)]\n",
    "    else:\n",
    "        tokens = [token for token in sentence]\n",
    "    if lower:\n",
    "        tokens = [token.lower() for token in tokens]\n",
    "    tokens = [sos_token] + tokens + [eos_token]\n",
    "    ids = de_vocab.lookup_indices(tokens)\n",
    "\n",
    "    tensor = ms.Tensor(ids).astype(ms.int64).unsqueeze(-1)\n",
    "    encoder_outputs, hidden = model.encoder(tensor)\n",
    "    \n",
    "    inputs = en_vocab.lookup_indices([sos_token])\n",
    "    attentions = ops.zeros((max_output_length, 1, len(ids)))\n",
    "    for i in range(max_output_length):\n",
    "        inputs_tensor = ms.Tensor(inputs[-1]).unsqueeze(-1).astype(ms.int64)\n",
    "        \n",
    "        output, hidden, attention = model.decoder(inputs_tensor, hidden, encoder_outputs)\n",
    "        attentions[i] = attention\n",
    "        predicted_token = output.argmax(-1).item()\n",
    "        inputs.append(int(predicted_token))\n",
    "        if predicted_token == en_vocab[eos_token]:\n",
    "            break\n",
    "    en_tokens = en_vocab.lookup_tokens(inputs)\n",
    "    return en_tokens, tokens, attentions[:len(en_tokens)-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这段代码用于绘制注意力矩阵的可视化图。展示模型在翻译时对输入句子的注意力分布。横轴表示输入句子的词汇，纵轴表示翻译结果的词汇，而颜色的深浅表示不同位置的注意力权重。这样的图可以帮助我们理解模型在翻译时的关注点。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_attention(sentence, translation, attention:Attention):\n",
    "    fig, ax = plt.subplots(figsize=(10,10))\n",
    "    attention = attention.squeeze(1).numpy()\n",
    "    cax = ax.matshow(attention, cmap=\"bone\")\n",
    "    ax.set_xticks(ticks=np.arange(len(sentence)), labels=sentence, rotation=90, size=15)\n",
    "    translation = translation[1:]\n",
    "    ax.set_yticks(ticks=np.arange(len(translation)), labels=translation, size=15)\n",
    "    plt.show()\n",
    "    plt.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Ein Mann mit einem orangefarbenen Hut, der etwas anstarrt.',\n",
       " 'A man in an orange hat starring at something.')"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = test_data[0][\"de\"]\n",
    "expected_translation = test_data[0][\"en\"]\n",
    "\n",
    "sentence, expected_translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "translation, sentence_tokens, attention = translate_sentence(\n",
    "    sentence,\n",
    "    model,\n",
    "    en_nlp,\n",
    "    de_nlp,\n",
    "    en_vocab,\n",
    "    de_vocab,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<sos>',\n",
       " 'a',\n",
       " 'man',\n",
       " 'in',\n",
       " 'an',\n",
       " 'orange',\n",
       " 'hat',\n",
       " 'is',\n",
       " 'something',\n",
       " 'to',\n",
       " '<unk>',\n",
       " '.',\n",
       " '<eos>']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<sos>',\n",
       " 'ein',\n",
       " 'mann',\n",
       " 'mit',\n",
       " 'einem',\n",
       " 'orangefarbenen',\n",
       " 'hut',\n",
       " ',',\n",
       " 'der',\n",
       " 'etwas',\n",
       " 'anstarrt',\n",
       " '.',\n",
       " '<eos>']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据可视化结果可知，模型在翻译时，总是会关注在时序上较为临近的token。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA5IAAAOVCAYAAAAbdN+qAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAACGp0lEQVR4nOzdd3wUdeL/8fcmpAFJqBJIQiBCKPKl95aCoChNsNANRcVDOQWk6B2iIginYlc8gUiRIoJ0EYHQOwiiEGogQOglECAhZH5/8GPPmAQykuwk2dfz8djH3c7M7r4nrMm+9zPzGZthGIYAAAAAAMgiF6sDAAAAAADyFookAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAlAJWBwAAIDeJjo7WmjVrFB8fr6SkpAy3sdlsmjhxooOTAQCQe9gMwzCsDgEAgNUuX76s9u3ba+3atbrXn0abzaZbt245KBkAALkPI5IAAEgaOnSo1qxZowoVKqhfv34KCQmRt7e31bEAAMiVGJEEAEBS6dKlJUm///67ihUrZnEaAAByNybbAQBAtw9tbdKkCSUSAIAsoEgCACCpYsWKSkxMtDoGAAB5AkUSAABJL7/8sqKjo3Xw4EGrowAAkOtRJAEAkNS3b18NGDBAoaGhmjx5so4fP251JAAAci0m2wEAQJKrq6skyTAM2Wy2u25rs9mUkpLiiFgAAORKXP4DAABJgYGB9yyQAADgNkYkAQAAAACmcI4kAAAAAMAUiiQAABlISkpSfHy8Lly4YHUUAAByHYokAAB/8vXXX6tWrVoqVKiQAgICNHjwYPu6uXPnqmPHjlwiBADg9CiSAABIunXrlp544gm9+OKL2rt3r6pUqaK/TiNQo0YN/fjjj5o1a5ZFKQEAyB0okgAASPrss880f/58tW7dWkePHtVvv/2WbpsHH3xQFSpU0NKlSy1ICABA7kGRBABAUlRUlEqVKqVZs2apVKlSmW5XtWpVHT161IHJAADIfSiSAABIiomJUYMGDVSoUKG7bleoUCGdPXvWQakAAMidKJIAAEhyc3PTjRs37rndsWPH5O3t7YBEAADkXhRJAAAkPfTQQ9q+fbuuXLmS6TZnzpzRr7/+qpo1azouGAAAuRBFEgAAST169ND58+fVr18/JScnp1t/69Yt9e/fX9euXdOzzz5rQUIAAHIPm/HXuc0BAHBCt27dUsuWLRUdHa2goCA98sgj9mtKNm3aVIsWLdKRI0fUqlUrLV26VDabzerIAABYhiIJAMD/d+PGDQ0aNEjffPONbt68mWadq6urevfurY8//lienp4WJQQAIHegSAIA8Bdnz55VdHS0YmNjlZqaqoCAAIWHh6tMmTJWRwMAIFegSAIAAAAATGGyHQAAAACAKQWsDgAAQG4SHR2tNWvWKD4+XklJSRluY7PZNHHiRAcnAwAg9+DQVgAAJF2+fFnt27fX2rVrda8/jTabTbdu3XJQMgAAch9GJAEAkDR06FCtWbNGFSpUUL9+/RQSEiJvb2+rYwEAkCsxIgkAgKTSpUtLkn7//XcVK1bM4jQAAORuTLYDAIBuH9rapEkTSiQAAFlAkQQAQFLFihWVmJhodQwAAPIEiiQAAJJefvllRUdH6+DBg1ZHAQAg16NIAgAgqW/fvhowYIBCQ0M1efJkHT9+3OpIAADkWky2AwCAJFdXV0mSYRiy2Wx33dZmsyklJcURsQAAyJW4/AcAAJICAwPvWSABAMBtjEgCAAAAAEzhHEkAAAAAgCkUSQAAMpCUlKT4+HhduHDB6igAAOQ6FEkAAP7k66+/Vq1atVSoUCEFBARo8ODB9nVz585Vx44duUQIAMDpUSQBAJB069YtPfHEE3rxxRe1d+9eValSRX+dRqBGjRr68ccfNWvWLItSAgCQO1AkAQCQ9Nlnn2n+/Plq3bq1jh49qt9++y3dNg8++KAqVKigpUuXWpAQAIDcgyIJAICkqKgolSpVSrNmzVKpUqUy3a5q1ao6evSoA5MBAJD7UCQBAJAUExOjBg0aqFChQnfdrlChQjp79qyDUgEAkDtRJAEAkOTm5qYbN27cc7tjx47J29vbAYkAAMi9KJIAAEh66KGHtH37dl25ciXTbc6cOaNff/1VNWvWdFwwAAByIYokAACSevToofPnz6tfv35KTk5Ot/7WrVvq37+/rl27pmeffdaChAAA5B42469zmwMA4IRu3bqlli1bKjo6WkFBQXrkkUfs15Rs2rSpFi1apCNHjqhVq1ZaunSpbDab1ZEBALAMRRIAgP/vxo0bGjRokL755hvdvHkzzTpXV1f17t1bH3/8sTw9PS1KCABA7kCRBADgL86ePavo6GjFxsYqNTVVAQEBCg8PV5kyZayOBgBArkCRBAAAAACYUsDqAAAA5EanT5/WyZMnJUllypRRqVKlLE4EAEDuwaytAAD8f4Zh6JNPPlFISIjKlCmjunXrqm7duipTpowqVqyojz/+WKmpqVbHBADAchzaCgCApKSkJLVt21YrVqyQYRgqWrSogoKCJEnHjh3ThQsXZLPZFBERoUWLFsnDw8PixAAAWIcRSQAAJI0ePVq//PKLHnroIS1dulTnz5/Xjh07tGPHDp07d04//fSTqlWrppUrV2r06NFWxwUAwFKMSAIAIOnBBx/UxYsXdeDAARUvXjzDbc6dO6eQkBAVKVJEhw8fdnBCAAByD0YkAQCQdPLkSbVo0SLTEilJJUqUUEREhOLj4x2YDACA3IciCQCAJH9/fyUnJ99zu5s3b3I9SQCA06NIAgAgqVu3blqxYoWOHj2a6TZHjx7VihUr1LVrVwcmAwAg9+EcSQAAdHuksVOnTtq1a5fefPNNPfPMMypUqJAkKTExUbNnz9Zbb72lGjVqaM6cOXJzc7M4MQAA1qFIAgCcUnBwcLplhmHo2LFj9vtFixaVJF28eNG+rGzZsnJxcdGhQ4dyPiQAALkURRIA4JRcXO7v7I7U1NRsSgIAQN5DkQQAAAAAmMJkOwAASEpISNCVK1esjgEAQJ5AkQQAQFKRIkXUqlUrq2MAAJAnUCQBAJDk6+ub4QQ8AAAgPYokAACSatWqxUysAABkEUUSAABJQ4cO1datWzVnzhyrowAAkOsVsDoAAAC5gZeXl/r27atnnnlGbdq0Udu2bVW2bFl5enpmuH3z5s0dnBAAgNyDy38AAKDb15W02Wy682fRZrPddftbt245IhYAALkSI5IAAEjq2bPnPcsjAAC4jRFJAAAAAIApTLYDAAAAADCFIgkAAAAAMIVzJAEA+JNjx45p4cKFOnDggK5cuaKMzgCx2WyaOHGiBekAAMgdOEcSAID/7+2339Y777yj1NRU+7K/zuJqGIZsNhuztgIAnBqHtgIAIGnWrFkaOXKkAgMD9fXXX6tly5aSpGXLlunLL79UaGioDMPQwIEDtXLlSovTAgBgLUYkAQCQFBoaqs2bNysmJkZBQUHq1auXpkyZkmbkcfz48RoyZIhWrVqlpk2bWpgWAABrMSIJAICk3bt3q3HjxgoKCpKU9lDWO1599VVVqlRJo0aNsiQjAAC5BUUSAABJSUlJ8vPzs9/39PSUJF26dCnNdjVq1NDWrVsdGQ0AgFyHIgkAgKTSpUvrzJkz9vv+/v6SpN9//z3NdsePH2eiHQCA06NIAgAg6f/+7/8UExNjvx8WFibDMPTmm28qMTFRkjR79mytXbtWDz30kFUxAQDIFSiSAABIatu2rU6cOGGfkbVJkyYKDw/XqlWrVLRoUZUoUUJdunSRzWbTv//9b4vTAgBgLWZtBQBAt8+RjI2NVcmSJVWsWDFJUkJCgoYMGaIff/xRFy9eVEhIiIYPH66uXbtanBaAMzp27JgKFy5s/x2VmYsXL+rKlSsqW7asg5LBGVEkAQBOKasfyAAgt3B1dVVkZKQmTpx41+2ee+45TZ48WSkpKQ5KBmfEoa0AAKdUvnx5vfbaa/b7vXv31qRJkyxMBAB3ZxiGsjoGxFgRchpFEgDglAzDUGpqqv1+VFSU1q1bZ2EiAMge586dk5eXl9UxkM8VsDoAAABW8PX1VVxcnNUxAOCu1qxZk+b+qVOn0i27IyUlRTExMVq2bBmzSyPHcY4kAMAptWrVSitXrlSPHj1Uvnx5jRw5UjVr1lSHDh3u+VhmbgXgKC4uLrLZbJJuH0lx5/9n5s4206dPV+fOnR0REU6KIgkAcEo7duxQ27ZtFR8fb/qxNptNt27dyoFUAJBWZGSkvTx+++23qlChgpo0aZLhtu7u7ipTpozatm2r2rVrOzImnBBFEgDgtK5evaqtW7cqLi5OkZGRatq0qfr06ZOlxz777LM5nA4A0nJxcVFkZCQTgyFXoEgCACA+oAHI/T7++GMVKlRIffv2tToKQJEEAECSjh49qsKFC6t48eJWRwGADLm5uemxxx7T/PnzrY4CMGsrAACSFBQUlOb+gQMHdO7cORUvXlwhISEWpQKA//Hz85Onp6fVMQBJXEcSAAC7pKQkvf766ypRooQqV66spk2b6r333rOvnzZtmmrXrq1ff/3VupAAnNYjjzyidevWKTk52eooAEUSAABJun79usLCwjR27Fi5u7vrscce01/P/oiIiNCuXbs0e/Zsi1ICcGbvvvuuXF1d1a1bt7814zSQnTi0FQAASePGjdPmzZvVp08fffrpp/L09JSLS9rvW8uUKaOqVavql19+0ejRoy1KCsBZDR8+XDVq1NDcuXO1ePFi1a5dW2XLls3wcFebzaaJEydakBLOgsl2AACQVLVqVV27dk0HDx5UgQK3v2fNaCbXJ598UuvXr2c0AIDD/fXLrbvherd536+//qpjx44pNDRUvr6+VsdJhxFJAAAkHTlyRI8//ri9RGbG3d1dFy9edFAqAPifVatWWR0BDtSxY0cdPXpU//nPfzRw4ECr46RDkYRlcvu3LACci5eXV5YK4pEjR1S0aFEHJAKAtIoWLSoXFxdVq1bN6ijIYatXr1ZsbKwkKSoqKlcWSSbbgWU6duyoJ554guP3AeQKNWvW1LZt23T27NlMtzly5Ih27typevXqOTAZANxWs2ZNDRgwwOoYcIBvv/1WklS3bl39/vvv2rFjh8WJ0qNIwhJ3vmUxDENRUVFWxwEAPffcc7py5Yq6dOmic+fOpVt/6dIl9e7dWzdv3tTzzz9vQUIAzq5YsWIqXbq01TGQw65du6Y5c+aoSpUqev/992UYhqZMmWJ1rHQ4tBWW+PO3LNu3b9eOHTtUu3Zti1MByMyxY8cUHx+vpKSkTLdp3ry5AxNlvy5dumjhwoWaOXOmgoOD1bhxY0nS+vXr1b59e61evVoJCQnq2bOn2rRpY3FaAM6oYcOG+u2336yOgRw2d+5cXb16VT169FDz5s0VGBio7777Tu+///49z+N3JGZthcNdu3ZNfn5+CgwM1JdffqmwsDANGDBAH330kdXRAPzFxIkT9e677+ro0aP33DY/zA5oGIbef/99/ec//0k3Kunr66shQ4Zo2LBhstlsFiUE4My2bt2qpk2bavTo0Ro0aJDVcZBDHn74YUVHR+vo0aPy9/fXG2+8offee09z585V+/btrY5nR5GEw02bNk09e/bU6NGjNWzYMAUFBen69es6efJkrvqWBXB2X3zxhV5++WUZhqFatWopODhYhQsXznT7yZMnOzBdzrp165Z27Nih2NhYpaamKiAgQPXq1ZO7u7vV0QA4sSlTpmj16tWKiopS9erV9fjjj2d6HUlJ6tmzp4MT4n7FxcWpXLlyCg8P1y+//CJJiomJUZUqVdShQwfNnTvX4oT/Q5GEw+WVb1kAZ1exYkUdP35cixcvVkREhNVxACCda9eu6dy5cypevLgKFSpkX37x4kWNHTtWe/bsUdmyZTVo0CA9+OCDFibNHi4uLrLZbPrzx/eMjpAwDIPrSOZRo0eP1r///W9Nnjw5zRcB9erV02+//aYTJ06oePHiFib8H4okHCovfcsCODsvLy+Fh4dryZIlVkcBgAwNHz5c48aN05YtW1SnTh1JUlJSkqpXr66DBw/aC1eJEiW0a9euPD9RzciRI00dWv/mm2/mYBrkhEqVKunkyZM6depUmi9HPvnkE73yyiv65JNP9NJLL1mY8H84jhAONXXqVElpD7WoVKmS6tSpoyVLluj8+fO55lsWwNmVLVtWXl5eVsdwmN69e2dpO3d3dxUvXlw1a9ZUmzZtnOpnBOQ2K1eu1IMPPmgvkdLtU2gOHDigiIgIDR06VIsXL9Ynn3yi8ePHa9y4cRamvX8jR460OgJy0MaNG3XgwAF169YtTYmUbk8IN3jwYH377be5pkgyIgmHykvfssC8I0eOaO3atXed3dNms+nf//63g5Ph7xg9erTef/99HTx4UMWKFbM6To5zcbl9Raw73/b/9c/jX5fbbDYVLVpUEyZMUKdOnRyYFMAdpUuXVu3atbV48WL7svbt22vRokWKjY1VYGCgJKly5cpyc3NjxlPkav369dN///tf/fTTT2rZsmW69W3atNHSpUu1e/duPfTQQxYkTIsiCYfZuHGjmjRpom7dutlHJu84e/as/P39VaNGDW3dutWihPi7kpOT1bdvX02fPl1S+g/gf8Y5G3nHrVu31KlTJx05ckQff/yxQkND8/VspatXr9bcuXP16aefqkmTJnrmmWdUtmxZSbcPy581a5bWrVunl156SQ0bNtSaNWs0ceJE2Ww2rV27Vg0aNLB4DwDn4+npqU6dOqX5+1OiRAmVLVtWO3futG/3zDPPaNmyZbp06ZJFSYG7S0pKkp+fnwoVKqS4uLgM/97OmjXLPjKZG0bXObQVDvPtt9/KZrNlOINYyZIl1apVKy1dulS///57rviWBVk3YsQITZs2TUWKFFH37t0VEhIib29vq2PhPrm6umrChAlq0aKFWrRoITc3N/n5+dlH7v7MZrPp0KFDFqTMPjdv3tQXX3yhiRMnqlevXunW9+/fX1FRUerbt6/atm2rr776SuHh4erSpYs++OADzZ4924LUgHPz8/PTkSNH7Pe3b9+uixcvqkePHmm2y29fgq1bt07z58/XgQMHdOXKlQy/wLXZbFqxYoUF6fB37Ny5UzVr1lT79u0zfb+2b99eLVq00PHjxx2cLmOMSMIh8uK3LMi6smXL6urVq9q5c6eCgoKsjoNssnfvXoWHh+vs2bN3HWW+IzU11QGpck5oaKiSkpK0adOmu27XsGFDubu7a82aNZKkKlWqKCEhQSdOnHBETAB/0qFDBy1atEhz5sxRixYt1L17dy1atEjLli3Tww8/bN+uTp06un79uv744w8L094/wzDUp08fffvtt2kOs//rLK7M2gpHSP+1MpAD7nzLMnjw4DzzLQuy7syZM2rWrBklMp8ZNGiQzpw5o2effVa7du3SlStXlJqamuktr9u5c2eWLg/w4IMP6tdff7Xfr1Klis6dO5eDyQBkZsiQIZKkTp06qUiRIlq4cKFq1KiR5pJFp0+f1q5du9JMyJNXffXVV4qKilKdOnW0fPlydezYUdLtGfCXLl2qyMhIubi46LXXXtPhw4ctTov8jkNb4RANGzbUqlWr7rqNp6enli9f7qBEyE4UyPxp/fr1ql69uiZNmmR1FIdwdXXN0mjFH3/8IVdX1zTL/jq7HgDHaNy4sebNm6f3339f586dU506dTR69Og0h+DPmDFD3t7eevTRRy1Mmj2ioqJUqFAhLV26VMWLF9e0adMk3b7ub8WKFfXII4/oscce0zPPPKPGjRvz9xk5ihFJAPetd+/eio6O1tmzZ62Ogmzk7u6uqlWrWh3DYZo2bardu3dr7NixmW4zbtw47dq1S82bN7cvO3z4sMqUKeOIiAAy0LZtW61evVq///67pkyZooCAgDTrX3nlFV28eFHdunWzKGH22bt3rxo3bmy/VNqdo7z+fAjrk08+qTp16uj999+3JCP+vgMHDmjKlClpzvuVpE2bNqlhw4YqXLiwqlatmmuuu86IJID79tprr2nnzp0KDw/Xp59+qrCwsHw3sYEzCg8P1+7du62O4TCjR4/WqlWr9Prrr2vy5Mnq1KmT/dIBcXFxmjdvnmJiYlSoUCGNGjVKknT06FHt3r2byxYBcIjU1NQ019suWLCgJOnixYsqUaKEfXnFihXTXBIFecMHH3ygb775RrGxsfZlp0+f1iOPPKIrV67IZrNp3759euaZZ7R582bVrl3burBiRBIOlNe+ZUHWVahQQVu2bNHevXv18MMPy8vLS+XKlVNwcHC6W1bOQUPu8N577+nUqVP617/+5RQTNvzf//2fli9frgoVKmj//v0aM2aM+vfvr/79+2vMmDHat2+fKlSooOXLl+v//u//JEk+Pj7auHEj10YF4BD+/v46efKk/f6dQ1f/fKkTSdq/f78KFGC8KK9Zt26datasmWZUfdKkSbpy5YoGDhyo69eva+7cuUpNTdWHH35oYdLbmLUVDtOvXz/7tyx3/gM5ffq0QkJC7N+yGIYhV1fXXPEtC7Iuo8tB3E1+mJjFGbz99ts6fPiwpk6dquDgYIWFhcnf3z/Ty3/klzJlGIZWrlyp9evXKz4+XtLti543adJEERERjLYDucy1a9f04YcfprkcRkZsNptSUlIcnC57denSRStWrFB8fLxcXV21a9cu1apVSzVq1NCMGTPk7++vr776SkOHDlWLFi2YeyKPKV68uMLCwvTDDz/Yl4WGhmrLli06e/asChcuLOn2ucFnz57VgQMHrIoqiSIJB6pWrZo8PT21bds2+7IxY8bojTfe0MCBAzV69GgtWbJEnTp1UpcuXewnkAOwhouLS7pp5TPDNPMArHD58mU1a9ZMv//+u1xdXeXu7q5r166pdOnSOnXqlP33152Ru78eFZXXzJgxQ926ddOPP/6odu3aSZK6du2qmTNnpvmSy9XVVevWrVP9+vWtioq/oVChQnr88cft1yVOSkpS0aJFVb9+fUVHR9u3u/MeSExMtCjpbYx5w2Hi4+MVFhaWZtlPP/0kDw8PjRw5Uu7u7urQoYMaNGigzZs3WxMSgN3kyZOtjgAAd/Xee+9pz549euGFFzR+/Hj169dPU6dO1YkTJ3Tjxg3Nnj1bw4YNU4MGDTRjxgyr4963Ll26qGPHjmkOW/32229VvXp1/fjjj7p48aJCQkI0ZMgQSmQeFBAQkGZugl9++UU3btxIczkbSbp+/XqumC2cIgmHuXHjRpop85OSkrR161Y1aNDAPlQvSeXLl9euXbusiAjgT5599lmrI1jiyJEjWrt2reLj45WUlJThNvnpUF4gL/vxxx9VpkwZffLJJ3Jzc0szKufp6amePXuqbt26qlWrlj744AMNHjzYwrTZw8PDI819Nzc3DRs2TMOGDbMoEbJLRESEvv76a73yyitq0aKFhg8fLpvNpvbt26fZ7rfffrNPBmcliiQcJq99ywLzrl27pm3btt31A7gk9ezZ04GpgKxJTk5W3759NX36dEm66yG9FEkgdzh69Kgefvhhubm5SfrfOfs3b960L6tatapCQ0MVFRWV54vklClTVKFCBTVu3Piu223atEn79+/n720eM3z4cM2ePVuffvqpPv30UxmGoWeeeUY1atSwb/P777/r0KFDuWK2cIokHCavfcsCc0aMGKHx48fr2rVrmW5jGIZsNht/2PKYlJQULV68WFu2bNG5c+fUoEED9e7dW5J08uRJnTt3TlWrVs3zMwSOGDFC06ZNU5EiRdS9e3eFhITI29vb6lgA7sLT01Oenp72+z4+PpKkU6dOpfksUaxYMa1fv97h+bJbZGSkIiMj71kkJ06cqEmTJvH3No8pW7asdu3apW+++UZnz55VnTp1FBkZmWabnTt3qn379nr66aetCfknefuvPvKUvPYtC7Ju3LhxGjVqlFxdXfX444/zATwfWbdunbp37664uDj7FwE3b960F8mNGzfq6aef1vfff6+OHTtanPb+fPfddypSpIh27txpn5gDQO4WGBiouLg4+/3KlStLklavXq3u3btLuv1l2NatW9NcfzG/S01NZYbpPCogIEAjR47MdH337t3t722rUSThMHntWxZk3X//+195eXlp7dq1XLYlH/njjz/06KOP6ubNm3r55ZfVtGnTdP9ttm3bVgULFtQPP/yQ54vkmTNn9Mgjj1AigTykWbNmmjx5sq5cuSJvb2+1bdtWAwYM0IABA3T16lX5+/tr4sSJio2NVdeuXa2O6zCHDx+2j84COYUiCYfKS9+yIOvi4uIUERFBicxn3nnnHd24cUNLlixRq1atMtzG3d1dtWvXTncx7LyIAgnkPZ07d9b27du1YcMGPfLIIypTpozGjBmjwYMHq3///pJun1bh5+ensWPHWpz273n77bfT3P/111/TLbsjJSVFMTExWrNmjVq2bOmIeMgBu3fv1ueff661a9fqxIkTkiR/f381b95c//jHP1S9enWLE97GdSQB3Ldy5cqpXr16+v77762Ogmzk5+en4OBgbdiwwb7MxcVFkZGRmjRpkn1Z165dtWTJEl26dMmClNln7NixGj16tA4ePKiSJUtaHQfAfdi8ebPmzZtnvxxGr169VKxYMatj/S1/vqZvVq/t+8ADD2jJkiV8wZsHffzxx3rttdd069atDP+tCxQooP/85z/65z//aUG6v2SxOgCcT175lgVZ17lzZ02cOFGJiYnMuJuPXLp0KUsTXyUmJurmzZsOSJSzXnvtNe3cuVPh4eH69NNPFRYWxjlGQB7VoEEDNWjQwOoY2eLONX0Nw1Dv3r3VtGlT9enTJ8Nt3d3dVaZMGTVs2DDdZUKQ+y1fvlyvvvqqChYsqH79+qlHjx4qV66cbDabYmNjNXXqVH311VcaOHCgqlWrphYtWlialxFJOFRe+pYFWXfjxg21atVKbm5umjBhgipUqGB1JGSDsmXLqmTJktq+fbt9WUYjkiEhIXJxcdG+ffusiJltgoODJd2+nIB0+9psfn5+9ssJ/JnNZtOhQ4ccmg9AeufPn3eqSXTCw8PVunVrDRkyxOooyAGtW7fWihUrFB0dnenMvBs3blTz5s3VsmVLLVmyxMEJ06JIwmGWL1+uRx555J7fsly7dk0///yz5d+yIOsiIiKUnJysjRs3ysXFRUFBQQoICMj0A/iKFSssSAmzIiMjNXXqVP3yyy8KDw+XlL5Izps3T506dVL//v316aefWhn3vmX0fr2b1NTUHEoCIKsKFCighx56SKGhoQoNDVVYWJhTFUvkL8WLF1ft2rW1fPnyu27XsmVL7dixQ+fPn3dQsoxRJOEwee1bFmSdmQ/gNptNt27dysE0yC779u1TrVq15O7urvfee09PPPGEypQpo8jISH3wwQeaN2+eBg8erJs3b2r37t0qX7681ZEBOJm6devq119/TXO5i6pVq9pLZWhoaL465/natWs6d+6cihcvnuZUkosXL2rs2LHas2ePypYtq0GDBunBBx+0MCn+Di8vLz3xxBP67rvv7rpd165dNW/ePF2/ft1ByTJGkYTD5LVvWZB1dw4FzCpmx8w7fvzxR/Xo0UPXrl3LcL2np6dmzJihdu3aOTgZgMx07NhRpUuX1ueff251FIdISEjQmjVrFB0drVWrVmnXrl1pimXlypUVFhamsLAwPfXUUxanvT/Dhw/XuHHjtGXLFtWpU0eSlJSUpOrVq+vgwYP204ZKlCihXbt2qXTp0lbGhUkhISG6efOmDhw4oAIFMp7KJiUlRRUrVpSbm5v279/v4IRpmTuOB7gP165dy9K3giVLlsz0Qytyp6CgIFO3vC4iIkLjxo2753bvv/++IiIiHJAo53To0EF79uzRyy+/rMqVK8vT01Pu7u4KDg7WCy+8oN27d1Mi86jg4GANHTr0ntsNHz6ckY08ZsmSJU71ZayPj4/atGmj999/X9u3b9eFCxe0YMECvfrqq6pVq5ZiYmL01VdfqUuXLlZHvW8rV67Ugw8+aC+RkjRt2jQdOHBA4eHhWrZsmQYMGKBz585p/PjxFibF39G+fXsdPXpUvXv3znAm9ISEBD333HM6duyYOnTo4PB8f8WsrXCYwMBAbdy4USkpKXf9lmXjxo1ZmikSsEp0dLTKlSt3z+1iYmK0evXqnA+Uw4KCgvTRRx9ZHcOhrly5okOHDunKlSuZTrXfvHlzB6fKXrGxsTp79uw9tzt37pxiY2NzPhCyTfny5ZWYmGh1DMu4u7urYMGCKliwoLy8vOTq6povZpaWpGPHjqW7pMeCBQtks9k0efJkBQYGqmXLlvrpp5+0dOnSLH3pidxj+PDhmjt3rqZPn6758+fr0UcftX/eOHr0qH766SclJCQoODhYw4cPtzasKJJwoPbt2+uDDz5Q79699cknn6hIkSJp1ickJOif//ynjh07pkGDBlkTEtni0qVLd/0AXrZsWQcnssaNGzcy/dIEudOePXv0yiuvKDo6+p7XanOWc30TExPl5uZmdQyY0KVLF73//vs6deqU/Pz8rI6T427cuKENGzYoOjpa0dHR2rp1q5KTk2UYhoKCgtS1a1f7oa153cWLF9N8fjIMQ+vWrVP16tXTfAlfo0YNLVu2zIKEuB/FihXT2rVr9cILL2jx4sUZXp/78ccf14QJE1S0aFELEqbFJxw4TF77liW7pKSk6Pz580pKSsp0m/xQrE6dOqV//etfWrBgwV0PqbLZbEpJSXFgMmskJCRow4YN+eb8FGd4Hx84cEBNmzZVQkKCmjRpovj4eB05ckSdO3fW4cOHtWPHDqWkpKhdu3bpvgjLj1JTUxUTE6NVq1bl+X9bZzN8+HBt3rxZoaGheu+999SmTZt8+2VA8+bN0xXHzp0724tjfjid4s/8/Px05MgR+/3t27fr4sWL6tGjR5rtuAZu3lWmTBktXLhQR44c0bp163Ty5En78qZNm+aqie2YbAcOdfLkSfu3LBm58y1LmTJlHJws+/3yyy8aNWqUNm3adNdDavJDsYqPj1e9evV08uRJ+fv76+bNmzpz5owaNWqkw4cP6/Tp07LZbGrUqJHc3Ny0atUqqyObducag9LtQwILFy6sEiVKZLhtSkqKTp8+rZSUFL300kv6+OOPHRUz2znT+/jZZ5/VtGnTNGnSJD377LPq1auXpkyZYh95PHjwoPr06aNTp05p06ZNueLbYLNcXV3t/98wjCx92DQMQ//+97/11ltv5WQ0ZKPg4GClpqYqLi5O0u3/Ph944AF5enqm2zavXxPVxcVFNptNDz30kEaMGKFOnTrl6xLVoUMHLVq0SHPmzFGLFi3UvXt3LVq0SMuWLdPDDz9s365OnTq6fv26/vjjDwvTIr+jSMISeeFblvuxaNEiPfHEE7p165aKFi2q8uXLy9vbO9Pt82Kx+rP+/fvryy+/1Ntvv61//etf6T6Ar1mzRi+++KKKFSum5cuXZ/hhJrf78yVObDbbXQ97dHNzU5kyZdSuXTuNGTNGBQsWdETEbOds7+PAwED5+vpqz549kpTufSzdPmw7ODhYXbt21WeffWZV1L/tzrV7pdvnWhUsWDDTL0Tc3d3t7+MBAwakKaHI3ZzpmqgdOnTQ2rVrdfHiRdlsNnl7e6t58+b2S3/Url07XxXLDRs2qHnz5va/QYZhqGbNmtq2bZv93/306dPy9/dXly5dNHXqVCvj4j4dOHDAfrmXkJAQq+OkZwDIdnXr1jVcXFyMjz76yEhJSbE6To4LDg42goOD7fcjIyMNFxeXNNscP37cKFSokPH66687Ol62s9lsRq9evayOkeOc7X3s7u5uPPXUU/b7zz33nOHi4mJcv349zXZPPPGEERQU5OB02c9Z3sfI/1JTU42dO3caH374odGuXTujWLFihs1mM1xcXIwiRYoYbdu2NT744ANj+/btVkfNFgsWLDCaN29uVK1a1ejRo4cRFxeXZv348eONIkWKGNOmTbMoIe7HjRs3jOHDhxvFixc3XFxcDBcXlzS/q6dOnWrUqlXL2Llzp3Uh/z/OkUSusHfvXv3+++8KDAxUgwYNrI5z337//Xc1atRI//znP62O4hAnTpzQ448/br9/Z+QiKSlJHh4ekiR/f3+Fh4dr9uzZevfddy3JmV0mT56sChUqWB0jxznb+7hYsWJpzgEtVqyYpNvncFeqVCnNtmfOnHFotpywatUqp5iIBfmfzWZTzZo1VbNmTb366qsyDEO7du2yT77z888/a/HixfniEHxJatu2rdq2bZvp+ldeeUWvvPKK4wIh21y/fl0RERHasmWLSpUqpcceeyzd6WARERF69tlnNXv2bNWsWdOaoP8f15GEw8yaNUsRERHavHlzmuWDBw9WtWrV9Mwzz6hx48b2Q+nyssKFCzvV5BQ+Pj5p7t+ZiOTEiRNplnt6eqZblhc9++yzatKkidUxcpyzvY/Lly+vo0eP2u/XrFlThmFo1qxZ9mXnzp1TdHR0vvi5hIaGpivIQH4QFxen3bt32283btyQYRj3nIkZsNq4ceO0efNm9e7dW4cPH9bChQvTbVOmTBlVrVpVv/zyiwUJ02JEEg4zbdo0/frrr6pVq5Z92YYNG/Thhx/Kx8dHjz/+uDZt2qQFCxZo+vTp6tmzp4Vp78/DDz+sbdu2WR3DYcqWLatjx47Z71erVk3S7Ytiv/TSS5Kka9euaf369XlyFtM7++bv7y9XV9c0+5oVebV0ONv7uFWrVho1apSOHj2qoKAgtW3bViVKlNDbb7+tP/74Q/7+/po7d64uX75sf1/nZVOmTDG1fV7+neysfv75Z3355ZfasmWLzp07p+7du2vixImSpGXLlmnZsmUaPHhwnp/g7tixY/bRx+joaPsXQoZhyN3dXU2bNlVoaGi+uPzHnznDbNrOZtasWSpbtqy+/PLLu14+rFKlSlq/fr0Dk2XC0gNr4VTKlStnhIaGplnWr18/w8XFxVi2bJlhGIZx/vx5w8fHx2jWrJkFCbPPsWPHjFKlShlDhgwxbt68aXWcHPfaa68Z7u7uxpkzZwzDuP3v6O3tbXh6ehpDhw41PvnkE6N+/fqGi4uL0b9/f4vTmmez2QxXV1cjJibGfv/OeQv3urm6ulqc/u9ztvfxwYMHjWHDhhmbN2+2L1uxYoVRvHhxw2az2W+tWrUybty4YWHS7JHV9/Gd7ZC3DBgwwP7v5+3tne6c2F27dhk2m8348MMPLUx5/8qXL5/mverp6Wk0b97cGDFihLFixYp05zjnB8uXLzdCQ0MNDw+PfPv3x1l5enoanTp1SrMso/PZu3TpYnh4eDgyWoYYkYTDnDlzRo0bN06zbNWqVXrggQfUqlUrSbfPSWrevLm2b99uRcRsM3nyZLVu3Vrvv/++fvjhB4WFhSkgICDDmfRsNpv+/e9/W5Ay+3Tr1k1xcXH6448/FBoaqmLFimnChAnq1auXxo0bZ5/l9KGHHsqT50c2b95cNpvNPvvqnfv5nbO9jx988EGNGTMmzbKIiAgdPXrUPitkSEiI6tSpY1HC7DVixIgM38d3LhuxevVqHTlyRJGRkfnuWnz53ZQpU/Tpp5+qbt26+vrrr1WzZs10/93euYD9woUL9eqrr1qU9P6dPHlSzZo1s183smHDhnlyZvCsMjubNvIWLy8vXbx48Z7bHTlyJFdcgorLf8BhSpQooUaNGtmP946Pj5e/v7+efvppzZw5075djx49NGfOHF2/ft2qqPftznWtsvKfl81my/PnhGbm2LFjWrJkif0DeLt27fLtRbHzI2d7Hy9YsEBubm5q3bq11VFyhZSUFA0cOFCzZs3S1q1bOUQuD2nUqJFiYmIUExOjkiVLSrr933NkZKQmTZpk365t27b67bffFBsba1HS+/fnSd2cQb169bRjxw59+OGHeumll7gsTz4TERGh7du36+DBg5n+t3vkyBFVqVJFrVq10oIFC6yMyzmScJzg4GCtXbtWly5dUpEiRTR9+nTZbDb7aOQdp06d0gMPPGBRyuwxefJkqyPkCmXLllW/fv2sjoG/ydnex0888YRatWpFkfz/ChQooPHjx2vBggUaNmyYvvvuO6sjIYv27Nmj0NBQ+wfRzPj6+ur06dMOSpUzxo4dq5o1a6pdu3Z33W7hwoXauXOnRowY4aBkOcPZZtN2Ns8995yio6PVpUsXzZw5M911fi9duqTevXvr5s2bev755y1K+T8USThMZGSkXnrpJdWpU0c1a9bU4sWLVbhwYbVv396+zc2bN7Vt2zbVrVvXwqT379lnn7U6giViY2O1Zs0axcfHZ3ryf344BPKOrOyvpDz7wcXZ3sclS5bMFYcK5Saurq6qU6eOli9fbnUUmJSVw+9PnjwpLy8vB6TJOSNHjlRkZOQ9i+SCBQs0adKkPPv7+A5nm03b2XTp0kULFy7UzJkzFRwcbD8lbP369Wrfvr1Wr16thIQE9ezZU23atLE4LUUSDvTcc89p1apV+uGHH3TkyBEVKlRIEyZMUPHixe3bLFq0SJcvX1ZERISFSWHWjRs39Nxzz9lHLO52KGR+KJJm9zevf3BxFmFhYdqyZYsMw3CKc2Cz6tSpU0pMTLQ6BkyoWLGiduzYoZs3b2Z6OsGVK1f066+/6qGHHnJwOmvcunUrw/O78xpnm03bGU2fPl21atXSf/7zH/3888+SpAMHDujAgQPy9fXVu+++q2HDhlmc8jaKJBzGzc1N33//vWJjY3X27FlVrlw53Qni5cuX17x589SwYUOLUuLvGDp0qKZPn64HHnhA3bp1U3BwsAoXLmx1rBzjbPvrLN555x3Vq1dPr776qt577718PWFHVqSmpurzzz/Xxo0bVb9+favjwISnnnpKb7zxhoYNG6YPPvggw22GDx+uy5cvq3Pnzg5OZ43ff/89XxxxMHbsWNWrV09Dhw7Vu+++e9dLRCBvstlseu211zRw4EDt2LFDsbGxSk1NVUBAgOrVqyd3d3erI9ox2Q6QDSIiImSz2fTtt98qICDA1IiqzWbTihUrcjBdzvPz81Nqaqp2794tPz8/q+PkuPy6v87+Pn777be1b98+zZo1SyVLltTDDz+ssmXLZlgo88PI+t3+fa9evaojR47owoULstlsWrhwIeeO5iHXr19Xw4YNtWfPHtWvX1/t27fX66+/rmbNmqlDhw6aN2+e1q1bp9q1a2vDhg256oNpVvTu3dv+/6OiolShQgU1bdo0w21TUlIUExOjbdu2qUOHDvrhhx8cFTNHvP322zpy5IimTJmi8uXL5/vZtJG7USThEHFxcdq5c6cqV66skJCQTLdbunSpDMPQY4895sB09+/O7JZ79+5VSEiIqcNn8sNsl4ULF9ajjz6qOXPmWB3FIfLr/jr7+9jZZqm9179vgQIF1KhRI40YMUItWrRwUCpkl7NnzyoyMlJLly7N8H3dsmVLTZs27Z4T8uRGf37vZvW/2erVq2vu3LkKDg7OyWg5ztl+TzmTvPhZmfFwOMTNmzfVoUMHhYeHZzpq8dtvv+nxxx9X69atc8V/HGYcOXJEkuTv75/mvrOoVq2aEhISrI7hMPl1f539fexss9Te7d/X3d1dJUqU4HI9eVjJkiW1ePFi7dq1Sz///HOaw+NatmyZpw9XXrVqlaTb56dHRETo0Ucf1dChQzPc1t3dXWXKlMk310J1tt9TziQvflZmRBIO06xZM23cuFFHjhxRYGBguvVDhgzRBx98oBkzZujpp5+2ICH+ru+//17dunXT5s2bVatWLavj5Dhn218AyK169eqlZs2apTncFcir8tpnZYokHOabb77R888/r9GjR6ebbcowDAUGBur69euKj4/Pc+drZOb8+fOaNm2atmzZonPnzqlFixYaMmSIpNsn/h86dEgPP/ywChYsaHHS+zd+/Hi9++67eumll9SyZUv5+/tneuhcfpi63Jn215nex3ds3LhRa9eu1YkTJyTdHqVt1qyZGjVqZHGy7Hft2jVt27btnpex6dmzpwNTwYwpU6bc1+Pzw7+tM/6eQv6T5z4rG4CDJCQkGAULFjSqVKmSbt3y5csNm81m9OvXz4JkOWP27NmGj4+P4eLiYthsNsPFxcXo1auXff2yZcsMFxcXY+rUqRamzD6//PKLUbFiRcPFxeWuN1dXV6ujZgtn2V9nex/HxMQY9evXt//72Ww2+367uLgY9evXN2JiYqyOmW3+/e9/G4ULF77re/jO/iP3+vN71Mwtv/zbOtvvqdTUVGPq1KnGk08+adSoUcMIDg42ypcvn+4WHBxsdVSYlNc+K3OOJBzG29tbHTp00MyZM7Vt2zbVrVvXvm7q1Kmy2Wz55gLoGzduVNeuXeXj46MPPvhATZs2TXc+SosWLeTr66u5c+eqe/fuFiXNHosWLVLHjh2VkpKiEiVKKCgoKF9fDsNZ9tfZ3sfx8fEKDQ3V6dOnVaZMGT311FMqV66cbDabYmNj9f3332vr1q0KDw/Xtm3bVLp0aasj35dx48Zp1KhRcnV11eOPP66QkJB0l2RC3jBixIh01z49dOiQpk2bpoIFC6pVq1YqV66cJOno0aP6+eeflZiYqO7du+vBBx+0IHH2cbbfU8nJyXr88ce1cuXKTCfcyepkPMh98txnZaubLJzLsmXLDJvNZgwYMMC+LDEx0fD29jYqVapkYbLs1aZNG8Pd3d3Yvn27fZnNZkvzDalhGEaLFi2MihUrOjpetqtdu7ZRoEABIyoqykhNTbU6To5zlv11tvfxP/7xD8NmsxkDBw40kpKS0q1PTk42Bg0aZNhsNuOll16yIGH2qlChglGwYME0/77IH/bv328UKVLE6NGjh3H+/Pl06y9cuGD07NnTKFq0aJ4fYXe231OjR482bDab0a5dO+PgwYNGz549DRcXFyM5OdnYt2+f8dZbbxne3t7GkCFDrI6KvykvfVamSMKhUlNTjYCAAOOBBx4wUlJSDMMwjOnTpxs2m8149913LU6XfYoVK2aEhoamWZbRH7Zu3boZhQsXdmCynOHl5WVERERYHcNhnGV/ne19XK5cOaNy5cp33SY1NdWoXLmyUa5cOQelyjkeHh5G69atrY6BHNCpUyejfPny9r+zGbl586ZRvnx5o2PHjg5Mlv2c7fdUjRo1jOLFixtXr141DMMwIiMj0x2evGbNGsPV1dWYOHGiFRFxn/LSZ+WsXyQMyAY2m03du3fXuXPntHTpUkm3h+pdXFzyxcn+d1y7di1L1+a6ePGiA9LkvBIlSqhEiRJWx3AYZ9lfZ3sfx8fHq3bt2nfdxmazqXbt2oqPj3dQqpzj5+enQoUKWR0DOSA6OloNGzaUq6trptsUKFBADRs21OrVqx2YLPs52++pgwcPqn79+vb/du9M8vbn60U2a9ZMTZo00RdffGFJRtyfvPRZmSIJh4uMjJRhGJoyZYpOnz6tX375ReHh4QoICLA6Wrbx9/fX77//ftdtDMPQnj17VL58eQelyjlPPvmk1qxZoxs3blgdxSGcZX+d7X3s4+OjuLi4e24XFxcnHx8fByTKWZ07d1Z0dLQSExOtjoJsdmdWx3s5depUnv895my/p1xdXeXr62u/f6dQnj17Ns12/v7+iomJcWg2ZJ+88lmZIgmHq1SpkurVq6dFixbpiy++0K1bt3LXicPZ4NFHH1VMTIxmzpyZ6TbffPON4uLi9PjjjzswWc4YNWqUypUrp3bt2unQoUNWx8lxzrK/zvY+btSokdavX6/Fixdnus2SJUu0fv16NW7c2IHJcsbIkSNVpUoVtWvXTgcPHrQ6Tq7Qu3dvvfDCC9qwYYPVUe5L9erVtXbtWv3yyy+ZbrNixQqtWbNG1atXd2Cy7Odsv6f8/f11/Phx+/0KFSpIkjZt2pRmu927d+fLSeCcRV75rMx1JGGJL774Qi+99JIKFCggLy8vnTp1Sl5eXlbHyjbHjx9X9erVdfXqVb366qt64okn1LhxYz311FMaNmyY5s2bp3HjxsnX11e//fabHnjgAasj35eIiAglJydr48aNcnFxUbly5TK9rqLNZtOKFSssSJl9nGV/ne19vHHjRjVv3lw2m03PPPOMunbtmmamyxkzZmjmzJlKTU3V2rVr1bBhQ2sDmxQREZFu2Z/fx0FBQQoICMh372MzXFxc7LOftmzZUm+//Xa6GUDzggULFqhDhw5yd3dX165d9cwzzygoKEjS7ffy7NmzNX36dN28eVPz5s1Tu3btLE789znb76nevXtr3rx5OnXqlDw8PHTw4EFVqlRJgYGB+uqrr+Tv76+vv/5aX3zxhdq2basff/zR6sgOMX/+fF2+fFlS/rguqpRHPitbeH4mnNjFixcNT09Pw8XFxejdu7fVcXLEhg0bjNKlS2d4fS+bzWaUKlXK2LRpk9Uxs8Wda+1l5ZYfrlnmTPvrTO9jwzCMqVOnGgULFsx0fwsWLJhnr0Vn5n2b397HWTVy5EhjxIgRRvv27Y2iRYvm6f3+8ssvDS8vr0zfy56ensbnn39udcxs4Uy/pxYtWmT4+fkZCxYssC8bOHBgmn232WxG4cKF8/yMvGZUrlzZvv/5RV74rMyIJCwzdOhQbdmyRe+9954aNGhgdZwcceXKFU2cOFHLly9XbGysUlNTFRAQoJYtW+qFF15Ic55DXnb06FFT29/5Zjyvcrb9dZb38R3Hjx/Xf//7X61bt04nT56UJJUpU0bNmjVTnz59FBgYaHHCv8fs+/av8vr72CzDMLRz5857TsCUmx07dkwTJ05M814uXbq0mjVrpl69etlH3PMDZ/s99VczZ87Ujz/+qIsXLyokJEQDBgxQxYoVrY7lMD179rSf475q1SqL02Sf3P5ZmSIJAAAAADCFyXYAAAAAAKZQJAEAAAAAplAkAQAAAACmUCRhmaSkJI0cOVJJSUlWR8lxzrSvEvubnznTvkrOtb/OtK8S+5ufOdO+Suxvfpbb95XJdmCZhIQE+fr66vLly/Lx8bE6To5ypn2V2N/8zJn2VXKu/XWmfZXY3/zMmfZVYn/zs9y+r4xIAgAAAABMoUgCAAAAAEwpYHUAWC81NVUnT56Ut7e3bDabw143ISEhzf/mZ860rxL7m585075KzrW/zrSvEvubnznTvkrsb35m1b4ahqErV66oTJkycnHJfNyRcySh48ePKzAw0OoYAAAAAHKJuLg4BQQEZLqeEUnI29vb6ggO997E76yO4DBH9x2zOoJDffPRm1ZHcJibN3PnLG45x3FHTOQOfM8LALDOvToCRRJ/OZzVOT6oeRYsaHUEh/Hw8LQ6gkM58vBsOJaz/dtywBAAwEr3+rvLZDsAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSedTixYvVu3dvValSRT4+PipUqJBq1Kih0aNHKykpyep4AAAAAPKxAlYHwN/Tp08fXb9+XdWqVVP16tV1+fJlbdmyRW+88YZWrFihn3/+Wa6urlbHBAAAAJAPUSTzqAkTJqhVq1by8vKyL7ty5Yq6du2qRYsWafr06erZs6eFCQEAAADkVxzamke1b98+TYmUJG9vb40fP16SNH/+/Ewfm5SUpISEhDQ3AAAAAMgqRiTzsAMHDmjJkiU6ePCgEhMTlZqaKsMw7OsyM2bMGL311luOigkAAAAgn6FI5kGGYWjw4MEaP368vTj+1ZUrVzJ9/PDhwzVw4ED7/YSEBAUGBmZ7TgAAAAD5E4e25kGzZs3Shx9+qICAAM2ZM0cnTpxQcnKyDMOwz9iaWcGUJA8PD/n4+KS5AQAAAEBWMSKZB82bN0+S9OWXX+rxxx9Ps+7w4cNWRAIAAADgRBiRzIMuXrwoSQoICEi3bvbs2Y6OAwAAAMDJUCTzoJCQEEnS119/neYQ1rVr1+o///mPVbEAAAAAOAmKZB40YMAAFSpUSF988YWqVaumLl26qHnz5goNDVW/fv2sjgcAAAAgn6NI5kEhISHatm2b2rZtq3PnzmnBggW6evWqJkyYwIgkAAAAgBzHZDt5VOXKlbVgwYIM191txlYAAAAAuF+MSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMKWB1AOQ2htUBHGLE832sjuAwny2aZ3UEh2q+6WmrIzjM6tWzrI7gUB4eBa2O4FCJiZetjuAwhpFqdQQAgEmMSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUiaFBsbK5vNprCwMCUmJmrgwIEKDAyUl5eXateurYULF9q3/f7779WgQQMVKlRIpUqV0oABA3T9+vU0z/frr79qyJAhqlOnjkqWLCkPDw8FBwfrH//4h06ePHnX179+/bqGDRumoKAgeXh4qEKFCho7dqwMw8jxnwMAAAAA50WR/JuSk5PVokULTZ8+XQ0bNlTDhg21a9cuPfHEE/rll180fvx4de3aVd7e3nrkkUd069Ytffrpp+rbt2+a53nvvfc0fvx4SVLTpk312GOPyTAMffnll6pbt26GZfLO67dq1Ur//e9/VbduXYWHh+vEiRMaNmyY/v3vf+f4/gMAAABwXjaD4StTYmNjVb58eUlSRESEFixYoEKFCkmSoqKi1KtXL1WoUEHnz5/Xzz//rLp160qSTp48qVq1aunMmTM6dOiQgoODJUmrVq1S1apVVapUKftrpKamatSoUXrzzTfVq1cvTZo0KcPXDw0N1YIFC+Tj4yNJ2rZtmxo2bCgPDw+dPn1ahQsXznAfkpKSlJSUZL+fkJCgwMDA7PoR5Qk+3sWtjuAwny2aZ3UEh5ryzjdWR3CY1atnWR3BoTw8ClodwaESEy9bHcFhDCPV6ggAgL+4fPmyvWdkhBHJv8nFxUVffvmlvURKUs+ePVWiRAkdPHhQ/fv3t5dISSpTpoy6desmSVqzZo19eXh4eJoSeee5R4wYIX9/fy1YsCDT158wYUKaf9y6deuqdevWunbtmrZt25Zp9jFjxsjX19d+c7YSCQAAAOD+FLA6QF5Vrlw5hYSEpFnm4uKioKAgnTt3Tq1atUr3mDujkPHx8WmWnz9/XgsWLNCePXt06dIl3bp1S5J08+ZNnT9/XhcuXFCxYsXSPCYoKEiVKlVK9xp3Mv31Nf5s+PDhGjhwoP2+M45IAgAAAPj7KJJ/k7+/f4bL7xxOmtH6O+v+fFjpjBkz9Pzzz+vq1auZvtaVK1fSFcmAgIAMt/X29k73Gn/l4eEhDw+PTNcDAAAAwN1waOvf5OJy9x/dvdZL0tGjRxUZGank5GR99NFHOnDggK5duybDMGQYhho1aiRJGc7CmpXnBwAAAICcwIikhZYsWaLk5GQNHjxY//znP9OtP3z4sAWpAAAAAODuGNay0MWLFyVlfJjqmjVrdPr0aUdHAgAAAIB7okha6M7EONOmTVNiYqJ9+YkTJ9SvXz+rYgEAAADAXVEkLdSuXTs99NBD2rZtmypUqKAnn3xSbdq0UUhIiIoWLarGjRtbHREAAAAA0qFIWsjd3V1r167Viy++KE9PTy1atEh79+7Vyy+/rOXLl8vNzc3qiAAAAACQjs3IaEpQOJWEhAT5+vpaHcOhfLyLWx3BYT5bNM/qCA415Z1vrI7gMKtXz7I6gkN5eBS0OoJDJSZetjqCwxhGqtURAAB/cfnyZfn4+GS6nhFJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApBawOAFgh4coFqyM4TGR4uNURHKprz+FWR3CYwoWLWh3BoQaN/o/VERzqg9dfszqCw1y8eNrqCA5mWB0AAO4bI5IAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokrmYzWZTuXLlrI4BAAAAAGlQJAEAAAAAphSwOgAyt3fvXrm5uVkdAwAAAADSoEjmYpUrV7Y6AgAAAACkw6GtuVhG50hGR0fLZrMpMjJSFy5c0IsvvqjSpUvLw8ND1apV06RJk6wJCwAAAMBpMCKZR126dEmNGjXS1atX1axZM507d05r1qxRnz59lJqaqr59+1odEQAAAEA+xYhkHjV//nzVrl1bhw8f1uzZs7Vy5UrNmTNHkvTOO+9YnA4AAABAfkaRzKN8fHz02WefycPDw76sQ4cOqlatmo4dO6bY2NhMH5uUlKSEhIQ0NwAAAADIKopkHlWnTh0VL1483fKQkBBJUnx8fKaPHTNmjHx9fe23wMDAHMsJAAAAIP+hSOZRAQEBGS739vaWdHvUMTPDhw/X5cuX7be4uLgcyQgAAAAgf2KynTzKxeXvfwfg4eGR5pBYAAAAADCDEUkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKczamosZhpFuWVhYWIbL74iKilJUVFQOpgIAAADg7BiRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhSwOoAgDUMqwM4TGpqqtURHGrx/G+sjuAw5YKqWR3BoWo0eMjqCA5VoIC71REcxmazWR3BoQzDef4GAci/GJEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRzAUWL16s3r17q0qVKvLx8VGhQoVUo0YNjR49WklJSWm2jYqKks1m08iRI3Xs2DF17dpVJUuWlJeXl+rWrauFCxdatBcAAAAAnEUBqwNA6tOnj65fv65q1aqpevXqunz5srZs2aI33nhDK1as0M8//yxXV9c0j4mNjVW9evXk7e2tFi1a6NixY9q4caM6dOigpUuXqlWrVhbtDQAAAID8jhHJXGDChAk6deqU1q9fr1mzZumnn37S0aNH1aZNG61cuVLTp09P95hvv/1WPXr00P79+zVz5kxt2LBB48ePV2pqqkaNGnXX10tKSlJCQkKaGwAAAABkFUUyF2jfvr28vLzSLPP29tb48eMlSfPnz0/3mPLly2v06NFycfnfP+FLL72kokWLatOmTUpOTs709caMGSNfX1/7LTAwMJv2BAAAAIAz4NDWXOLAgQNasmSJDh48qMTERKWmpsowDPu6vwoLC5O7u3uaZQUKFFD58uW1Y8cOnT9/XqVLl87wtYYPH66BAwfa7yckJFAmAQAAAGQZRdJihmFo8ODBGj9+vL04/tWVK1fSLQsICMhwW29vb0lKN0nPn3l4eMjDw+NvpAUAAAAADm213KxZs/Thhx8qICBAc+bM0YkTJ5ScnCzDMOxlMKOC+edDWgEAAADAkRiRtNi8efMkSV9++aUef/zxNOsOHz5sRSQAAAAAuCuGtSx28eJFSRkfqjp79mxHxwEAAACAe6JIWiwkJESS9PXXX6c5hHXt2rX6z3/+Y1UsAAAAAMgURdJiAwYMUKFChfTFF1+oWrVq6tKli5o3b67Q0FD169fP6ngAAAAAkA5F0mIhISHatm2b2rZtq3PnzmnBggW6evWqJkyYwIgkAAAAgFzJZmR2zQk4jYSEBPn6+lodAznGZnUAhypatJTVERymXFA1qyM41NuTxlkdwaF6P9LB6ggOc+7ccasjOJRhpFodAQDu6fLly/Lx8cl0PSOSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMKWB0AQE4zrA7gUJcunbE6gsPsT95qdQSHalOrltURHMrd3cPqCAAAZIoRSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKfmmSG7cuFHt27dXyZIl5eHhoXLlyukf//iHTp48mWa7qKgo2Ww2jRw5Uvv371fnzp1VqlQpubi46Mcff5QkHTx4UCNHjlSjRo3k5+cnd3d3BQQEqGfPntq/f3+Gr2+z2VSuXDndunVLY8eOVUhIiDw8PBQYGKihQ4cqKSkpw8ft3r1bbdu2VZEiReTt7a3mzZtr+fLlio6Ols1mU2RkZLrHGIahGTNmKCIiQkWLFpWnp6eqVKmikSNH6tq1a/f1cwQAAACAe8kXRXLatGlq1qyZFixYoEqVKqljx47y8PDQl19+qdq1a2vfvn3pHhMTE6N69eppy5YtCg8PV8uWLeXm5iZJ+uabb/T2228rMTFR9erVU7t27eTj46OpU6eqXr162r17d6ZZunbtqlGjRqlSpUpq1aqVrly5onHjxqlPnz7ptt24caMaNWqkRYsWKSgoSG3atNGNGzf06KOPau7cuRk+f2pqqrp166auXbtq69atqlmzph577DElJibqrbfeUnh4uK5fv/43f5IAAAAAcG8FrA5wv+Li4vT8889LkubPn6927dpJul24Bg0apI8++kg9evTQ1q1b0zxu5syZeumll/TRRx/J1dU1zboOHTrohRdeUPny5dMsnzx5snr37q1XXnlFK1euTJfl6NGjKliwoA4cOCA/Pz9J0pEjR1S7dm1Nnz5db731lh588EF7vsjISF27dk3vvvuuXn/9dfvzTJw4UX379s1wfz/44APNmDFDYWFhmjFjhv11kpOT9Y9//EMTJ07UW2+9pffeey/LP0MAAAAAMCPPj0h+8803un79up5++ml7iZQkFxcXvffeeypTpoy2bdum9evXp3lcyZIlNXbs2HQlUpIaNmyYrkRKUq9evdSkSRNFR0fr8uXLGeb55JNP7OVOksqXL6/u3btLktauXWtfvnLlSu3fv18VK1bUsGHD0jxHnz591KRJk3TPnZKSonHjxqlQoUKaOXNmmtdxd3fXp59+Kj8/P3399ddKTU3NMJ8kJSUlKSEhIc0NAAAAALIqzxfJO+WsW7du6dZ5eHjoqaeeSrPdHQ8//LAKFiyY6fNevXpVM2bM0NChQ/Xcc88pMjJSkZGRio+Pl2EYOnToULrHuLm5KTw8PN3ykJAQSVJ8fLx92Z1i26lTJ7m4pP9neOaZZ9It27Fjh86dO6fGjRurVKlS6dZ7eXmpTp06unjxog4cOJDpvo0ZM0a+vr72W2BgYKbbAgAAAMBf5flDW+9MplOuXLkM199ZfuLEiTTLy5Ytm+lzrly5Up07d9bZs2cz3ebKlSvplvn5+WU4wunt7S1JaSbcuVMqMytxGeWLjY2VJC1fvlw2my3TbJJ07tw5VapUKcN1w4cP18CBA+33ExISKJMAAAAAsizPF8l7yaxweXp6Zrj86tWrevrpp3XhwgWNGDFCnTt3VlBQkLy8vGSz2dS1a1fNmDFDhmGke2xGI4vZ6c7hqhUqVMjw0Nc/K168eKbrPDw85OHhka3ZAAAAADiPPF8ky5Qpo5iYGB09elQPPfRQuvV3RvH8/f2z9Hxr167V+fPn9eSTT+qtt95Kt/7w4cP3lfeO0qVLS7o9WVBGMloeEBAgSapcubKioqKyJQcAAAAAmJXnz5Fs1qyZJGnGjBnp1iUnJ+v7779Ps929XLx4UdL/StufHTx4UDt27Pi7UdO4M6I4b968DEc3Z8+enW5ZvXr15Ovrq9WrV+vChQvZkgMAAAAAzMrzRbJPnz7y8vLSzJkztXjxYvvy1NRUvf766zpx4oTq1Klzz0NB77gzMc7cuXPTnCN56dIl9enTRzdv3syW3BEREapYsaJiYmI0bty4NOuioqLSTQ4k3T4kdciQIbpy5Yo6duyY4ejoiRMnNHXq1GzJCAAAAAAZyfOHtpYtW1YTJkxQZGSk2rZtqyZNmigwMFA7duxQTEyMSpUqpWnTpmX5+erWrauWLVtq+fLlCgkJUVhYmCQpOjpaJUqUUPv27TV//vz7zu3i4qJvv/1WDz/8sIYNG6YZM2aoatWqOnTokLZu3ar+/fvr888/l7u7e5rHDRs2TPv27dPUqVNVpUoV1apVS+XLl1dycrJiYmL0xx9/qHr16urRo8d9ZwQAAACAjOT5EUlJ6tGjh9auXas2bdpo7969mjNnjq5fv64XX3xR27dvV+XKlU093/z58/XGG2+oZMmSWrp0qbZv367OnTtr06ZNKlKkSLblbtSokTZs2KA2bdroyJEjWrBggdzc3LRkyRI1atRIUvpJc1xcXDRlyhTNnz9fLVu21JEjR/TDDz9o3bp18vT01GuvvaZJkyZlW0YAAAAA+CubkdEJerBcv379NGHCBM2cOTPDa0pmp4SEBPn6+uboawCOYrPli+/HsqRgQW+rIzjU1auXrI7gUAEBIVZHcJiTJ9Nfmzk/M4xUqyMAwD1dvnxZPj4+ma53nk9cudCFCxfss8r+2axZs/TNN9+oSJEiatOmjeODAQAAAMBd5PlzJPOy/fv3q1GjRqpevbqCg4MlSXv37lVMTIxcXV01YcIEFSpUyOKUAAAAAJAWI5IWCg4OVv/+/XXz5k2tWrVKixYt0uXLl9WxY0etXbtWTz/9tNURAQAAACAdRiQt9MADD+izzz6zOgYAAAAAmMKIJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAlAJWBwCA7GQYhtURHCYp6brVERyqUcN2VkdwqOGfjLc6gsN8/db7VkdwqJiYLVZHcChn+13lXJznby7SY0QSAAAAAGAKRRIAAAAAYApFEgAAAABgCkUSAAAAAGAKRRIAAAAAYApFEgAAAABgCkUSAAAAAGAKRRIAAAAAYApFEgAAAABgCkUSAAAAAGAKRRIAAAAAYApFEgAAAABgCkUSAAAAAGAKRRIAAAAAYApFEgAAAABgCkUSAAAAAGAKRRIAAAAAYApFEgAAAABgCkUSAAAAAGAKRRIAAAAAYApFEgAAAABgCkUSAAAAAGAKRTKHxcbGymazKSwszOooAAAAAJAtKJL5QHR0tGw2myIjI62OAgAAAMAJUCQBAAAAAKZQJAEAAAAAplAkHej69esaNmyYgoKC5OHhoQoVKmjs2LEyDCPNdmvXrtVLL72k6tWrq2jRovLy8lLlypU1bNgwXbp0Kc22kZGRCg8PlyR9++23stls9tvIkSMdtGcAAAAAnEkBqwM4i+TkZLVq1Up//PGHwsLClJiYqNWrV2vYsGG6cuWKRo0aZd/2tdde065du1S9enW1aNFCN27c0I4dOzR27FgtWrRImzZtUuHChSVJTZs21alTp7Rs2TI9+OCDatq0qf15atas6ejdBAAAAOAEKJIOsnHjRoWGhurIkSPy8fGRJG3btk0NGzbU+PHjNWzYMHs5fPPNN9W4cWP5+vraH5+UlKQBAwbo66+/1ocffqgRI0ZIkvr27asKFSpo2bJlatq0qaKiohy+bwAAAACcC4e2OoiLi4smTJhgL5GSVLduXbVu3VrXrl3Ttm3b7Mtbt26dpkRKkoeHhz766CMVKFBA8+fPv68sSUlJSkhISHMDAAAAgKxiRNJBgoKCVKlSpXTLQ0JCJEnx8fFplp84cUILFy7Uvn37lJCQoNTUVEmSu7u7Dhw4cF9ZxowZo7feeuu+ngMAAACA86JIOkhAQECGy729vSXdHiW848MPP9SwYcN08+bNHMkyfPhwDRw40H4/ISFBgYGBOfJaAAAAAPIfDm11EBeXrP2oN23apEGDBqlgwYKKiopSbGysbty4IcMwZBiGSpcufd9ZPDw85OPjk+YGAAAAAFnFiGQuM2/ePEnSu+++q2effTbNuuvXr+vUqVNWxAIAAAAAO0Ykc5mLFy9KyvhQ2O+//z7dNSel2+dNSlJKSkrOhgMAAAAAUSRznTuT70ycODHNOZJ//PGHhg4dmuFjypQpI0mKiYnJ+YAAAAAAnB5FMpfp1auX/Pz8tHDhQlWqVEnPPPOMWrZsqZo1a6pZs2YKCgpK95hy5cqpevXq2rZtm+rXr69evXqpb9++WrBggQV7AAAAACC/o0jmMsWLF9fWrVvVtWtXJScna8GCBTpx4oTeeecdzZgxI9PH/fDDD+rQoYMOHz6sKVOmaOLEidqxY4cDkwMAAABwFjYjo5Pu4FQSEhLk6+trdQwgm9isDuAwBQq4WR3BoerWecTqCA7VfcgLVkdwmK/fet/qCA4VE7PF6ggOlZR03eoIyDHUiPzs8uXLd726AyOSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFMokgAAAAAAUyiSAAAAAABTKJIAAAAAAFNshmEYVoeAtRISEuTr62t1DACm2awO4FA2m3Ptr7ubh9URHGbg2x9ZHcGhVs5baHUEh3J397I6gsP8+usKqyM4VGLiZasjOExqaqrVERzodj28fPmyfHx8Mt2KEUkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUyTzCZrOpXLlyVscAAAAAAIokAAAAAMCcAlYHQNbs3btXbm5uVscAAAAAAIpkXlG5cmWrIwAAAACAJA5tzTMyO0dyw4YN6tChg4KCguTh4SE/Pz/Vr19fw4YN09WrVx0fFAAAAEC+R5HMwxYuXKhmzZppwYIFKl26tDp27KhatWrpwoULGjt2rM6dO2d1RAAAAAD5EIe25mHvv/++UlNTNWfOHHXq1CnNuq1bt6p48eIZPi4pKUlJSUn2+wkJCTmaEwAAAED+wohkHnb27FlJ0sMPP5xuXb169eTt7Z3h48aMGSNfX1/7LTAwMEdzAgAAAMhfKJJ5WJ06dSRJPXr00NatW5Wampqlxw0fPlyXL1+23+Li4nIyJgAAAIB8hiKZh40ePVo1atTQwoULVb9+fZUoUULt2rXTN998oxs3bmT6OA8PD/n4+KS5AQAAAEBWUSTzsMDAQG3btk3Lli3Tyy+/rMDAQC1cuFDPPfecqlevrvPnz1sdEQAAAEA+RJHM4woUKKBWrVrpk08+0a5duxQbG6uIiAgdOHBAY8eOtToeAAAAgHyIIpnPBAUFaejQoZKkPXv2WJwGAAAAQH5EkczDxo8fr1OnTqVbvmTJEkliNlYAAAAAOYLrSOZhb731lgYPHqwaNWqoYsWKMgxDu3bt0v79+1WsWDENHjzY6ogAAAAA8iFGJPOwTz/9VJ07d9a1a9e0dOlS/fTTTypQoIAGDhyo3bt3q2LFilZHBAAAAJAPMSKZRxiGkW5Zjx491KNHDwvSAAAAAHBmjEgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATKFIAgAAAABMoUgCAAAAAEyhSAIAAAAATClgdQAAwN9lWB3AoQzn2l0l30yyOoLDfPvJf6yO4FABAZWsjuBQHV7obHUEh4n71z6rIzhUUtI1qyM4THKy8/xOzipGJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJE2Ijo6WzWZTZGSkqcfZbDaVK1cuRzIBAAAAgKNRJP8kKipKNptNI0eOtDoKAAAAAORaBawO4Az27t0rNzc3q2MAAAAAQLagSDpA5cqVrY4AAAAAANnG9KGte/bsUffu3RUcHCxPT0+VLFlSNWvW1CuvvKL4+Pg02y5ZskQtW7ZU0aJF5enpqUqVKmnYsGG6dOlSuucdOXKkbDaboqKitH37drVu3VpFihRRsWLF9PTTT+v48eOSpMTERA0ZMkTlypWTp6enqlWrpjlz5mSad+/evYqMjFRgYKA8PDxUqlQpde7cWb///nua7cLCwtSrVy9J0ltvvSWbzWa/RUVFpXveCxcu6MUXX1Tp0qXl4eGhatWqadKkSRlmyOgcyT+fb2nmuSRp7ty5atiwoQoWLKgSJUroqaee0sGDB9P8DAEAAAAgp5gakdy+fbuaNm2qGzduqHr16mrfvr2uXbumw4cP6+OPP1aHDh1UunRpSdKYMWP0+uuvq0CBAgoNDVWJEiW0fv16jR07VvPmzdOaNWtUqlSpdK+xefNm9evXT9WqVdMjjzyiHTt26Pvvv9euXbu0ZcsWtWzZUkePHlXz5s117tw5rV69Wk8//bSWLl2qRx55JM1z/fjjj+rcubOSkpJUs2ZNNWzYUHFxcZo9e7YWLlyopUuXqnnz5pKkRx99VCkpKVq/fr1q1KihmjVr2p+nQoUKaZ730qVLatSoka5evapmzZrp3LlzWrNmjfr06aPU1FT17ds3yz9Ts8/18ccf65VXXpGLi4uaN28uPz8/bd68WfXr11fbtm2z/LoAAAAA8HeZKpKffPKJbty4offff1+DBg1Ks27fvn3y9fWVJG3dulX/+te/VLhwYf3yyy9q0KCBJCkpKUk9evTQ999/r/79+2c4kvjVV1/pyy+/VL9+/SRJN2/e1GOPPaZffvlFjRs3lp+fnw4fPqxChQpJkiZOnKi+fftq9OjRaYpkbGysunfvLjc3Ny1atEgPP/ywfd1PP/2kdu3aqXv37jp48KDc3d01bNgw+fn5af369erQocNdJ9yZP3++OnfurKioKHl4eEi6XVqfeOIJvfPOO6aKpJnnOnz4sIYMGSJ3d3f99NNPCg8PlySlpKTo+eef1+TJk7P8ugAAAADwd5k6tPXs2bOSlKaU3VG5cmX7aORnn32m1NRUvfzyy/YSKUkeHh767LPP5OXlpXnz5ikuLi7d8zRt2tReIiXJzc1NL7/8sqTbZfXLL7+0l0hJioyMVIkSJbRx40bdvHnTvvyjjz5SYmKixowZky7vo48+qhdffFFxcXFavHixmR+BJMnHx0efffaZvfhJUocOHVStWjUdO3ZMsbGxOfJckyZNUnJysnr06GEvkZJUoEABffjhhypcuHCWXjMpKUkJCQlpbgAAAACQVaaKZJ06dSRJ/fv3V3R0tFJSUjLcbu3atZKkbt26pVv3wAMPqFWrVkpNTdX69evTrW/VqlW6ZcHBwZKkcuXKKSQkJM06V1dXBQUF6ebNmzp37px9+c8//yxJ6tixY4YZmzVrJknasmVLhuvvpk6dOipevHi65Xey/fVc0ex6rjs/r6eeeird9kWKFMnwZ5eRMWPGyNfX134LDAzMcl4AAAAAMFUkX3vtNYWFhWn9+vUKDw9X0aJF1apVK3388ce6fPmyfbuTJ09KUroJZu64s/zEiRPp1vn7+6dbdmekLaN1f16flJRkX3ZnJM/f3z/NxDl3bnfK2J/LZ1YFBARkuNzb2ztdjux8rjulMrPiV7Zs2Sy95vDhw3X58mX7LaORYQAAAADIjKlzJH18fLRy5UqtX79eCxcuVHR0tFauXKnly5drzJgxWrt2rSpWrHjP57HZbJmuc3HJvNvebd1fpaamSpKeffbZu27350Nvs8pMDkc+V1Z5eHikOZQWAAAAAMwwfR1Jm82mpk2bqmnTppKkM2fO6JVXXtGMGTP0xhtvaPbs2SpTpoyOHDmio0ePqmrVqume48+jhTklICBAhw4d0gcffJDhoaN5UenSpRUTE6O4uLgMf66MLAIAAABwhPseDnvggQfsM5zu2bNH0v/OP5wxY0a67c+ePatly5bJZrOpSZMm9/vymWrZsqUkad68eVl+jLu7uyRleu6n1e78vH744Yd06y5fvmw/LxQAAAAAcpKpIvnVV1/pyJEj6ZYvWbJE0v/O3evfv79cXFz0ySefaNu2bfbtkpOT9fLLL+v69evq2LFjjk7yMmjQIHl5eWnw4MGaO3duuvVJSUmaM2eOjh8/bl9WpkwZSVJMTEyO5bofvXr1kru7u6ZMmaI1a9bYl9+6dUuDBg3SlStXLEwHAAAAwFmYOrT1q6++0osvvqiqVauqSpUqKlCggPbt26ddu3bJ09NTI0aMkCTVr19f77zzjt544w01atRIYWFhKlGihNavX6+4uDhVrFhRn3/+eY7s0B0VKlTQjBkz1LVrV3Xq1EkVKlRQlSpVVKhQIZ04cUI7duxQYmKidu7caZ/wpmHDhnrggQc0Z84chYWFKTg4WC4uLurdu7caN26co3mz4sEHH9S4ceP0yiuvKDw8XKGhoSpVqpS2bNmiCxcuqHv37po2bZp9ZBUAAAAAcoKpEcl33nlHvXv3ls1m04oVK7Rw4UJdv35dffv21a+//prmUNXXX39dixYtUmhoqLZu3aq5c+fKw8NDQ4YM0ebNm1WqVKls35m/at++vXbv3q1//OMfstlsWr58uRYvXqwzZ86obdu2mj17dppzDT09PbV48WK1bNlSv/76q6KiojRx4kTt378/x7Nm1T//+U/NmTNHdevW1aZNm7Rs2TLVrFlTmzdvlqenpyTlm3NCAQAAAORONsMwDKtD4P7dunVL1atX1969e3Xy5En5+fll+bEJCQny9fXNwXQAkB0yn/E7P7rbDOf5TenSwVZHcKiAgEpWR3CoZwb0tjqCw3z6r5FWR3CokycPWB3BYZKTs355v7zvdj28fPmyfHx8Mt3K8deewH05dOiQLl26lGZZUlKShgwZoj/++EMtWrQwVSIBAAAAwCzTl/+Atb7//nu9+eabqlOnjgIDA5WQkKBdu3YpPj5eJUqU0GeffWZ1RAAAAAD5HEUyj2nRooV27dqlTZs2affu3UpJSZG/v79efPFFDR8+PEdnwgUAAAAAiSKZ59SrVy/D63MCAAAAgKNwjiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwBSKJAAAAADAFIokAAAAAMAUiiQAAAAAwJQCVgcAAPxdNqsDOJTN5lz7axiG1REc5vTpWKsjONStWylWR3ComC0xVkdwmPBWnayO4FArl31vdQSHiYvbZ3UEhzEMQ4aRes/tGJEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRzIViY2Nls9kUFhZmdRQAAAAASIciCQAAAAAwhSIJAAAAADCFIpnLjBw5UuXLl5ckrV69WjabzX6LjIy0b/fHH3+oW7duKl26tNzd3eXv76+ePXsqJibGouQAAAAAnEUBqwMgrZo1a6pTp0764YcfVKpUKT366KP2dU2bNpUkrVixQm3bttX169dVq1YthYWFad++fZo6darmzZunJUuWqFmzZlbtAgAAAIB8jiKZy3To0EE1a9bUDz/8oMqVKysqKirN+sTERHXr1k3Xr1/XZ599pv79+9vXjR8/XgMHDlTXrl114MABeXp6ZvgaSUlJSkpKst9PSEjIkX0BAAAAkD9xaGseM3v2bJ0+fVqNGjVKUyIl6dVXX1WdOnV0/Phx/fDDD5k+x5gxY+Tr62u/BQYG5nRsAAAAAPkIRTKPWbt2rSSpW7duGa7v3r17mu0yMnz4cF2+fNl+i4uLy/6gAAAAAPItDm3NY06ePClJKleuXIbr7yw/ceJEps/h4eEhDw+P7I4GAAAAwEkwIpnP2Gw2qyMAAAAAyOcoknlMmTJlJElHjx7NcH1sbKwkyd/f31GRAAAAADgZimQu5O7uLklKSUlJt+7OZT1mzJiR4WOnTZuWZjsAAAAAyG4UyVyoRIkScnNz06FDh3Tr1q00655++mmVKlVK69at09dff51m3SeffKJt27bJ399fnTp1cmRkAAAAAE6EyXZyIXd3dz366KNauHChatSoodq1a8vd3V1NmjRRr169NH36dLVt21YvvPCCvv76a4WEhGjfvn3auXOnChcurBkzZmR6DUkAAAAAuF+MSOZS33zzjXr06KHz58/ru+++08SJE7V69WpJUosWLbR161Z16dJFx48f15w5c3Tq1Cl1795d27Zt47BWAAAAADmKEclc6oEHHtCUKVMyXf/QQw/pu+++c2AiAAAAALiNEUkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKRRJAAAAAIApFEkAAAAAgCkUSQAAAACAKQWsDgAA2ctmdQCHcXFxru8CXVxcrY7gULdupVgdwWFcXZ3r44hhGFZHcKg927ZaHcFhajRoYHUEhyrtF2x1BIc5ceKA1REcxjAM3bqVes/tnOtTCAAAAADgvlEkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJAEAAAAAplAkAQAAAACmUCQBAAAAAKZQJB0oKipKNptNI0eOtDoKAAAAAPxtFEkAAAAAgCkUSQAAAACAKU5ZJA8dOqRjx45ZHSNLoqOjZRiG1TEAAAAAwM5pimRCQoK++eYbNWvWTBUqVNCOHTvs68LCwmSz2RQbG5vucbGxsbLZbAoLC0uzfOTIkbLZbIqKitJvv/2mdu3aqWjRoipUqJBCQ0O1YcMGU/k++OADubi4qHLlyoqLi7MvDw8PV3BwsN58800dOnTI1HMCAAAAQE7I10UyNTVVP//8s7p16yY/Pz8999xzWr9+vUJDQ1W5cuVseY1t27apYcOGio2N1SOPPKKKFStqzZo1atGihfbs2ZOl53j99dc1ePBg1alTR+vWrVNgYKB9Xffu3XX27Fm9/fbbqlChgpo3b66JEycqISEhW/IDAAAAgFn5skju27dPw4cPV9myZfXII4/ou+++U9myZTVq1CgdOXJE0dHR2VYkP//8c40ZM0a7d+/WzJkz9euvv+qVV17RjRs3NG7cuLs+NjU1VS+88ILGjBmj8PBwrVy5UiVKlEizzdSpU3Xq1ClFRUUpIiJC69evV9++feXn56du3bpp+fLlSk1NzZZ9AQAAAICsyDdF8uLFi/riiy/UoEEDValSRe+9956SkpLUv39/bd68Wfv27dMbb7yhoKCgbH3dJk2aaMCAAWmW/etf/5IkrVmzJtPHJScnq3Pnzvr666/VoUMHLV26VN7e3hluW7hwYT377LNasWKFjh49qjFjxqh8+fL67rvv1KpVK5UtW1bDhw/Xvn37spQ5KSlJCQkJaW4AAAAAkFX5okgOHTpUpUuXVv/+/bVr1y517NhRP/74o06ePKnPPvtM9evXz7HXbtWqVbplxYsXV7FixRQfH5/hYxITE9WmTRt9//33ioyM1Jw5c+Th4ZGl1wsICNCwYcP0+++/a9u2bRowYICSk5P13nvvqUqVKmrYsKGuX79+1+cYM2aMfH197bc/H0oLAAAAAPdSwOoA2WHz5s1KSkqSq6urXnvtNfXv319+fn4Oee2AgIAMl3t7e+vChQsZrvvoo4+UkpKixx57TJMmTZLNZvtbr12nTh099NBDql+/vgYNGqTTp0/bfxZeXl6ZPm748OEaOHCg/X5CQgJlEgAAAECW5YsRyTFjxigyMlJeXl4aNWqUAgIC9Oijj2r69OlKTEy8r+e+1/mHLi7mf4StW7eWr6+vfv75Z/3www+mH28YhlavXq2+ffuqVKlS6t69u86cOaOIiAh9++238vHxuevjPTw85OPjk+YGAAAAAFmVL4pko0aNNHnyZJ06dUrffvutQkND9fPPP6t79+7y8/NTz5497zopjbu7uyTp6tWr6db9+VIc2aV27dpatmyZChYsqC5dumju3LlZetzevXv1xhtvqFy5cgoLC9PEiRPl5+enUaNGKTY2VitWrFDPnj3/VrkFAAAAgKzKV42jUKFC6tmzp1asWKHY2Fi98847Kl26tKZOnapWrVopMDBQr732mnbv3p3mcaVLl5Yk7d+/P91zLl++PEeyNmjQQD/99JO8vLzUuXNnzZ8/P8Ptzpw5o08++UT16tVT1apVNXr0aCUkJOj555/X+vXrFRMTozfeeENly5bNkZwAAAAA8Ff5qkj+WdmyZfWvf/1L+/fv1/r16/X888/r2rVrev/991WjRg0tXrzYvm1oaKgk6YMPPtC1a9fsy1euXKmPPvooxzI2atRIP/30kzw8PPT0009r4cKF6bbx9/fXP//5T+3cuVOtW7fWzJkzFR8frwkTJqhx48Y5lg0AAAAAMpNvi+SfNW7cWBMmTFB8fLxmzJihRx99VIZh2Nd36dJFlSpV0oYNG1SlShU9+eSTatiwoVq2bKkXX3wxx7MtXbpUbm5uevLJJ9MUXEmqVKmSxo0bp7i4OC1ZskTPPPOMPD09czQTAAAAANyNUxTJOzw9PdW5c2ctXbpUbdq0sS/38vLSihUr1KVLF125ckVLlizRrVu3NGvWLPXv3z/HczVt2lRLliyRm5ubOnXqpJ9++sm+bs+ePXrttdfsh98CAAAAgNVsxp+H5uCUEhIS5Ovra3UMIJv8vcvp5EXONrGWi4ur1REc6tatFKsjOIybm7vVERyqSJFSVkdwqAoValsdwWFqNGhgdQSH2rlhvdURHGbb9mVWR3AYwzB069ZNXb58+a5Xd3CuTyEAAAAAgPtGkQQAAAAAmEKRBAAAAACYQpEEAAAAAJhCkQQAAAAAmEKRBAAAAACYQpH8f+3dcazVdf3H8dcF4QJeQKhNRG4WGNEcJtqPDZ0h2dJMg1ilw67cPyqXuNVyaLQ2sWW4RdmWTYe1hNuGtVwXpyuJOZYRI53l1pxUiNzbJX+FY1wUvOC95/eH825XUPn8kvPNcx6P7Yzdc77nnPfncHe4T77n+70AAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUEZIAAAAUOaXqAaAaLVUPUDctLc2z1qT51ttMRo8aXfUIdTU0NFj1CHXT0tJc/6999OhA1SPU1f8+v7vqEepmXNslVY9QV+MnTKp6hLoZNap53qdqtVoGT+CfoOZ5RQAAAHhbCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKCEkAAACKnFL1ANTfwMBABgYGhr/u7++vcBoAAOCdxh7JJrRmzZpMnjx5+NLe3l71SAAAwDuIkGxCq1atyoEDB4Yvvb29VY8EAAC8g/hoaxNqbW1Na2tr1WMAAADvUPZIAgAAUERIAgAAUERINpjrrrsuc+bMya9+9auqRwEAABqUkGwwPT092blzZw4cOFD1KAAAQIMSkgAAABRx1tYGs3Xr1qpHAAAAGpw9kgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQRkgAAABQ5peoBgJOrVqtVPUJdjRkztuoR6qYlLVWPUFdtE6dUPUJdvXhwf9Uj1M24cadWPUJdnXrqaVWPUFdjxo6reoS62fXU36seoa5efLF53qeGhoaqHqFuTvRnR3skAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKNLUIblr16709PRUPcab2r59e15++eWqxwAAABjWdCHZ39+fH//4x7n44otz9tln58knnxxxe61Wy8aNG/PRj340U6ZMybhx4/LBD34wq1evzqFDh477mC+88EJWrlyZ97///Rk3blymTp2ayy+/PJs3bz7u9nv27MmXv/zlzJ49OxMmTMjUqVNzzjnn5Prrr8/OnTtHbLtq1apMmzYt119/ff7whz+8PS8CAADAf6ApQnJoaCibN2/Otddem2nTpuWLX/xitm3bloULF2bOnDkjtrv22muzbNmyPP744znvvPNyxRVX5KWXXsptt92WRYsW5fDhwyMeu6+vL/Pnz8/atWtz5MiRLFmyJPPmzcuWLVty2WWX5c477xyxfW9vb84///zcc889SZIrrrgiCxcuTGtra+69995s3759xPaLFy/OhAkTsm7dulx00UWZPXt2br/99v/6PakAAEDjaqnVarWqhzhZnnnmmaxfvz5dXV3p6+tLknzgAx9IR0dHPv/5z+ess84asf13v/vd3HzzzbnkkkuycePGTJs2LUly5MiR3HDDDfnJT36SW265JXfcccfwfa666qo89NBDWbZsWX76059m7NixSZLf//73ueyyyzIwMJAnnngi5513XpLk1ltvzbe+9a3ceOON+eEPfzji+Xt6enL06NHMmjVrxPWDg4PZsmVLNmzYkO7u7hw6dCgtLS1ZtGhROjs7s3Tp0px66qkn/LoMDAxkYGBg+Ov+/v60t7ef8P0bQ0vVA3CSjB3bWvUIddPSZN/HbROnVD1CXb14cH/VI9TNuPFtVY9QVxMnvqvqEeqqre20qkeom9mz/6fqEerqn//cVfUIdfOnP22peoS6qdVqGRw8mgMHDmTSpElvuF3DheT+/fuzcePGrF+/Pn/84x+TJO9+97tz9dVX57rrrsv8+fOPe79XXnklZ5xxRg4fPpxdu3bl9NNPH3H74cOHM3PmzAwMDGTfvn0ZNWpUnn322cyaNSttbW3Zs2dPpk6dOuI+N910U77//e/nC1/4Qu69994kyQ033JC777473d3dWbx4cfH6Dh48mAceeCAbNmzI1q1bU6vV0tbWls985jNZvnx5Fi5cmJaWN//hcvXq1bntttuKn7uxNNcP4M1ESDYuIdm4hGRjE5KNS0g2phMNyYb6aOstt9ySM844IytWrMhTTz2VpUuXpru7O3v37s1dd931hhGZJE8++WT27duXCy+88JiITJLx48fnggsuyP79+/O3v/0tyat7HZPk8ssvPyYik6SjoyNJ8thjjw1fd8EFFyRJvvGNb+Shhx4qPpHOxIkT09nZmUcffTR79uzJ7bffnhkzZuS+++7LokWLMnPmzGOO+3y9VatW5cCBA8OX3t7eohkAAIDm1lAhuWPHjgwMDGT06NFZuXJlfvSjH2Xx4sUZM2bMW973ueeeS5L89re/TUtLy3EvDz/8cJJk3759SZK9e/cmSd773vce9zFfu/61j9UmSWdnZz73uc/l6aefzlVXXZUpU6bkIx/5SL7zne/k+eefL1pve3t7Vq1alXvuuScf/vCHh9fxVsdPtra2ZtKkSSMuAAAAJ+qUqgd4O61Zsybr1q3LL3/5y3z729/OmjVr8rGPfSwdHR1ZsmTJmx5HODQ0lCQ5++yzc9FFF73p87zrXSf2kZTjfcR09OjR+fnPf56vf/3r2bRpUx599NHs2LEjjz32WO6444785je/yYUXXviWj/3MM8+kq6srP/vZz4bD8Zxzzhn+eCsAAMDJ0lAhuWDBgixYsCB33XVXHnjggaxfvz6bN2/OI488kra2tnz6059OR0dHLr300owaNXJn7IwZM5Ikc+bMyX333XdCzzd9+vQkr/46j+N5bS/nmWeeecxt8+bNy7x587J69er09/dn9erVufPOO/PVr351+NjO1/vXv/6V+++/P11dXXniiSeSvBq1N954Y5YvXz68VxIAAOBkariT7bxeT09PNmzYkA0bNgwf2zh9+vQsW7YsHR0dOffcc5O8eibT008/PUNDQ3nuueeOe8zj6712sp2JEyemp6cnp5122ojbV65cmbVr14442c4bGRgYyPjx4zNu3LgRv6/y8OHDefDBB9PV1ZVHHnkkr7zySsaMGZNPfOITWb58ea688srhM8X+f/X392fy5Mn/0WO88zTXSUqaiZPtNC4n22lcTrbT2Jxsp3E52U5jasqT7RzPe97znnzzm9/MX//612zbti1f+tKXcujQoaxduzYf+tCHho97bG1tzc0335yDBw9m6dKlefbZZ495rL6+vnR1dQ1/PXPmzHzyk5/MwYMH85WvfCVHjx4dvm379u25++67M3r06KxYsWL4+q6urvzlL3855rF//etfp1arHfNrOD71qU/lmmuuycMPP5y5c+fmBz/4Qfr6+rJp06YsXbr0P45IAACAUg2/R/J4Xn755XR3d2f9+vVZsWJFrrzyyiSvHifZ2dmZrq6ujB07NvPmzcv73ve+HDlyJDt37szTTz+dc889N3/+85+HH6uvry8XX3xxdu/enbPOOisLFizIv//972zdujWDg4P53ve+l6997WvD2y9ZsiSbNm3KrFmzMnfu3IwfPz67d+/Ojh070tLSkvvvvz+f/exnh7e/5pprcuaZZ6azszNz5849Ka+HPZI0EnskG5c9ko3LHsnGZo9k47JHsjE17e+RfDs8+OCDWbduXR5//PHs378/U6ZMSXt7ey699NJcffXVOf/880ds/8ILL2TNmjXp7u5Ob29vJkyYkPnz5+emm27Kxz/+8RHb/u53v8svfvGLbNu2Lb29vXnppZcyffr04e2rOM5RSNJIhGTjEpKNS0g2NiHZuIRkYxKSnDAhSSMRko1LSDYuIdnYhGTjEpKNyTGSAAAAnBRCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCJCEgAAgCKnVD0A1avValWPUIFmXHNzaM7v5+YwNDRU9Qh11Uzfy7Vac/3dDg0NVj1CXQ0OvlL1CHVz9OiRqkeoq8HBo1WPUDfN9Z5cG/HnG2mpNdOrwnH94x//SHt7e9VjAAAA/yV6e3szY8aMN7xdSJKhoaHs3bs3EydOTEtLS92et7+/P+3t7ent7c2kSZPq9rxVaKa1JtbbyJpprUlzrbeZ1ppYbyNrprUm1tvIqlprrVbLwYMHM3369Iwa9cZHQvpoKxk1atSb/m/DyTZp0qSGfyN4TTOtNbHeRtZMa02aa73NtNbEehtZM601sd5GVsVaJ0+e/JbbONkOAAAARYQkAAAARYQklWltbc2tt96a1tbWqkc56ZpprYn1NrJmWmvSXOttprUm1tvImmmtifU2sv/2tTrZDgAAAEXskQQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKCIkAQAAKDI/wHC/bx4Mki0ywAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_attention(sentence_tokens, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过将任何字符串传递给 `translate_sentence` 来使用它来翻译任意句子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"Ein Mann sieht sich einen Film an.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "translation, sentence_tokens, attention = translate_sentence(\n",
    "    sentence,\n",
    "    model,\n",
    "    en_nlp,\n",
    "    de_nlp,\n",
    "    en_vocab,\n",
    "    de_vocab,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<sos>', 'a', 'man', 'looking', 'at', 'a', 'a', '.', '<eos>']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过注意力也可以发现，翻译出现错误的位置，其注意力也较为弥散。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3AAAALWCAYAAAAH2HGQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQ9ElEQVR4nO3dd5icZaE+4Gc2kEIanZBCiRCCIF2keEIIRQJCAM9RCRKKDZUm0iIcBH9IbzYQRKWIFMFADqBSJBCQliOIVIEQEwGBQMiGJKTt/P7A7GFN213Dzr7JfV/XXF7zzTs7z34uk3m+9/veqVSr1WoAAABo9+pqHQAAAIDmUeAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAsFx44oknMnr06EydOrXWUVpNgQMAAJYL+++/f/bbb7/87Gc/q3WUVlPgWMCycGQCAAA+6L777suECRNSrVZz5ZVX1jpOqylwLGBZODIBAAAfdNVVVyVJttlmmzz99NP505/+VONEraPA0cSycmQCAADmmzFjRm666aZsvPHGOf/881OtVnP11VfXOlarKHA0sawcmQAAgPl+85vf5N13381BBx2UQYMGpV+/fvnVr36VuXPn1jpaiylwNFqWjkwAAMB8V155Zerq6nLQQQclSb7whS/krbfeyu23317jZC2nwNFoWToyAQAASTJp0qTce++9GTx4cPr06ZMkGTFiRKrVauPZZyVR4Gi0LB2ZAACAJLnmmmuSvF/a5ttoo42y9dZb54477shbb71Vq2itosCRZNk7MgEAAMn7azystNJK+cxnPtNk+0EHHZTZs2fnuuuuq1Gy1lHgSLLsHZkAAICHHnooL7zwQvbdd9907dq1yWMHHHBAVlhhheImK1aodQDah8UdmTjmmGNy3XXX5YgjjqhROpY1L7/8csaOHZvXXnsts2bNWuiYSqWS//7v/27jZADAsuSqq65KpVJpMkkx3xprrJHdd989v/3tb/P0009nk002qUHClqtUq9VqrUNQWw899FB23HHHHHjggY0zcfO9+eab6dOnTzbffPM89thjNUrIsmL27Nn50pe+lGuvvTZJsri3n0qlknnz5rVVNOAD5s6dm7feemuRB1iSZJ111mnDRAAtN2vWrPTq1Stdu3bNpEmTUqlUFhhzww035IADDshxxx2Xc889twYpW84MHMvkkQnap1NPPTW//OUvs/LKK+cLX/hCBgwYkO7du9c6FvBPd999d84444w8/PDDmTNnziLHVSoVKxQD7d7jjz+eLbbYIsOGDVtoeUuSYcOGZZdddsnf//73Nk7XembglnPL6pEJ2qd11lkn7777bh5//PGsu+66tY4DfMBtt92W/fbbL/Pmzcsqq6yS9ddff7EHWO699942TAfAfGbglnPL6pEJ2qc33ngjn/rUp5Q3aIdOP/30NDQ05KKLLsoRRxyRDh061DoSAAthBg5oMxtttFEGDhyYW2+9tdZRgH+x0korZauttsoDDzxQ6ygALIavEQDazGGHHZYxY8bkzTffrHUU4F9069bNwiTAMueFF17I1VdfnZdffrnJ9ocffjjbbbddunXrlo9+9KP5zW9+U6OELafAAW3m+OOPz9ChQ7Pzzjvn3nvvXewqlLzv6quvzh//+Mcljnv44Ydz9dVXt0EillW77rprxo0bV+sYAEvVBRdckMMOOywrrrhi47bXX389n/rUp/Loo49m5syZee655/K5z30uf/rTn2qYtPkUOJbJIxO0TxtssEEeffTRPPvss9l1113TpUuXrLfeeunfv/8Ct4985CO1jtsuHHLIIbniiiuWOO5nP/tZDj300DZIxLLqnHPOSX19fU488UQrTALLjAceeCBbbLFF+vbt27jt5z//eaZNm5Zjjz02M2fOzG9+85s0NDTkwgsvrGHS5rOICbngggtyxRVXZMKECY3b5h+ZmDZtWiqVSuORiUceeSRbbbVV7cJStA/+jSXvfy/cxIkTaxNmGdPQ0LDIhYigOX7xi19k6NChOf/883PzzTdn8ODB6du3b+rqFjzWW6lU8t///d81SAnQMq+99loGDx7cZNvvfve7dOrUKaeddlo6duyYfffdN5/4xCfyyCOP1CZkCylwLPHIxJlnnpk77rgjn/nMZ3LhhRfml7/8ZQ3TUrKGhoZaR1hmjR8/Pj169Kh1DAp22mmnpVKppFqtZvz48Rk/fvwixypwQCnee++9Jqvqzpo1K4899lg+8YlPpFu3bo3b119//fz5z3+uRcQWU+BYJo9MQMm++93vNrn/xBNPLLBtvrlz5+b555/P/fffn912260t4rGM+sUvflHrCABLXd++ffPkk0823r/77rvz3nvvZciQIU3GzZw5M127dm3reK2iwLFMHpmAkn1wJqRSqeSJJ57IE088sdjnrLnmmjnzzDPbJiDLpIMPPrjWEQCWuiFDhuTyyy/PMccck1122SUjR45MpVLJsGHDmoz7y1/+kn79+tUoZcsocCyTRyZo32bMmJFx48bltddey6xZsxY5bsSIEW2Yqv2YPxNSrVZz2GGH5ZOf/GS++MUvLnRsx44d07t372y33Xbp1KlTW8YEgHZv5MiRufHGG/PDH/4wP/zhD1OtVvO5z30um2++eeOYp59+Oi+99FKOOOKIGiZtPgWOZfLIBO3XqaeemosuuigzZsxY5Jj5M0/La4H74EzIVVddlaFDh5odoc3MnTs3t99+ex599NFMnjw5n/jEJ3LYYYclSV599dVMnjw5H/3oR7PCCj5CAO3fOuuskz//+c+54oor8uabb2brrbfOIYcc0mTM448/nmHDhuWzn/1sbUK2UKXqi5iWexMnTsyWW26Zd955J0kaj0xcd911jWOefvrpfOxjH8sRRxyRH/zgBzVKSunOPffcnHTSSenQoUOGDh2aAQMGpHv37osc/53vfKcN0wEPPPBAvvCFL2TSpEmNB1IOPvjg/PznP0+S3HzzzfnsZz+bX//619l///1rnBZg+eTwGcvkkQnap5/+9Kfp0qVLxo4d6+sooJ155plnsscee2TOnDk58sgj88lPfnKB9/y99947K620Um6++WYFDqBGzMABbaZz584ZMmRI7rjjjlpHKcozzzyTc889N/fff39ee+21zJ49e6HjKpWKL2Cm1Q444ID8+te/zh133JHdd989SVJXV5dDDjmkcQYuSXbaaae8+eabeeaZZ2oVFaDFnnzyyfz4xz/O2LFj88orryRJ+vTpk0GDBuXrX/96NttssxonbD4zcECb6dWrl4VwWuihhx7KrrvumpkzZyZJVl111fTq1avGqcowZsyYxtK7qMVyKpVKfvazn7Vxsvbp3nvvzbbbbttY3halT58+ViQGivL9738/xx9/fObNm5cPzl0999xzee655/Lzn/885513Xo4++ugapmw+BY5Gy9KRCdqnz3/+8/nZz36W6dOnK3LNNHLkyMycOTPHHHNMTjnllKy66qq1jtTuTZ06NcOGDcvYsWOzpJNMFLj/88477zRroarp06dnzpw5bZCoHPfdd1/uu+8+BwugHbrrrrvyzW9+MyuttFIOP/zwHHTQQVlvvfVSqVQyYcKEXHPNNfnJT36SY489Nptuuml22WWXWkdeIqdQkmTRRybmW2GFFYo6MkH79N5772X33XfPiiuumMsuuywbbLBBrSO1e926dcuAAQPypz/9qdZRinH44Yfn8ssvzwYbbJDDDz98iYvl7LTTTm2Yrv1aZ511ssYaa+R///d/G7ct7BTKAQMGpK6uLs8991wtYrYrb7/9dvbbb7888MADzTpYMG/evDZKBsw3dOjQ3HPPPRkzZkx22GGHhY556KGHMmjQoOy2225FXOZhBo5l8sgE7dOee+6ZhoaGjBkzJhtvvHHWXXfd9O3bN3V1dQuMrVQqueeee2qQsn3p2LFjBg4cWOsYRbn11luz1lpr5eGHHzZj2QJDhgzJNddck3vvvTc777zzQseMGjUqL774Yr7xjW+0cbr26Zvf/GbGjh2bTTbZJF/5ylfSv3//dOvWrdaxgA949NFHs9NOOy2yvCXJ9ttvn8GDB+eRRx5pw2StZwaOZfLIBO3Tworaojha/b599tknEydOzBNPPFHrKMVYaaWVsueee+amm26qdZSiPPfcc9lyyy3TsWPHnH322dlvv/3Su3fvHHLIIbngggsyatSoHHfccZkzZ06efPLJrL/++rWOXHOrrrpqunXrlmeeeUZxg3aqS5cu2W+//fKrX/1qseOGDx+eUaNGNV5z3p4pcGS11VbLVlttlbvuumux43bbbbf86U9/yltvvdVGyVjW/O1vf2vR+HXXXfdDSlKOp556Ktttt13OOeccsx7NtPnmm6d379757W9/W+soxbnlllty0EEHZcaMGQt9vHPnzrnuuuuyzz77tHGy9qlnz57ZY489csMNN9Q6CrAIAwYMyJw5c/LCCy9khRUWfvLh3Llzs+GGG2bFFVfMX//61zZO2HJOoSQzZszIGmusscRxa6yxxiL/UYfmUMiW7Oqrr15g26GHHpqjjjoqN954Y3bbbbdFnnaaJCNGjPiwI7Z7Rx55ZI488si8+OKLrrNsoX333TdPPfVULrrootx1112ZMGFCGhoa0rdv3+y222751re+lY985CO1jtlubL311vnHP/5R6xjAYgwbNiwXXHBBDjvssPzgBz/Iyiuv3OTx+vr6HH300Zk4cWK+9a1v1SZkC5mBY5k8MgGlqqurS6VSWWD7B9+qF/W4007/z4knnphf/vKXOeOMMxpLLyxtf/jDH7LHHntk9OjR2WOPPWodB1iIt99+Ox//+MczYcKEdOvWLXvssUfWW2+9JO+fGfS73/0u9fX16d+/fx577LGsssoqtQ3cDGbgWCaPTFCGd955J9OmTVvk6m3rrLNOGyeqvVNPPXWhBY1F69Chw0K3V6vVfOlLX1rsc335Of+OIUOG5Fe/+lVGjBiRPffcM7vttlv69OmzyBnyQYMGtXFCYNVVV83YsWPz1a9+Nbfffnt+/etfLzBmr732ymWXXVZEeUvMwJFl88hEW5o7d27eeuutRX73T7J8FpFF+cc//pFTTjklo0ePXuz1lD5Y01zzV81trZdffnkpplk2eF9rviuvvDInn3xys06lNEMOtfXyyy/ngQceyKuvvpok6d27dz75yU8WtyiTAkeS5NVXX208MrEw849M9O7du42TtV933313zjjjjDz88MOL/VJbReT/vPbaa/n4xz+eV199NX369MmcOXPyxhtvZPvtt8/48ePz+uuvp1KpZPvtt8+KK66Ye++9t9aRYbnifa1lrrzyynzxi19MtVrNlltuucSvEfjFL37RhumAZZVTKEny/hGI//mf/1lmjkx82G677bbst99+mTdvXlZZZZWsv/76i/2iYN53xhln5NVXX813v/vdnHLKKTn00ENz9dVX58EHH0yS3H///fna176WSqViBcGF+POf/5xHH300kydPziabbNK4EuCsWbMya9as9OjRo8YJKZn3tZY799xz06lTp9xxxx0ZPHhwreMAzfDCCy9k8uTJWW211TJgwIBax2mdKtBi22yzTbWurq568cUXV+fOnVvrOMXo379/tX///o33DznkkGpdXV2TMX//+9+rXbt2rX77299u63jt1nPPPVfdfvvtq3V1dY23Qw89tPHxn//859W6urrqb3/72xqmpHTe11quS5cu1T322KPWMYAleO+996ojR46srrbaagv9d/Saa66pbrnlltXHH3+8diFboPnfqsty6dlnn81NN91UzDfTt5Wnn34622+/fY4++uhFLqDAgl555ZVsscUWjffn77sPXmfTp0+f7LzzzrnxxhvbOl67NGnSpAwaNCgPP/xw9t5775x77rkLLPry2c9+Nh07dszNN99co5TtyxVXXJFVV101v/vd7xY55re//W1WXXXVXHnllW0XrJ3zvtZyffr0yUorrVTrGMBizJw5M4MHD84555yTjh07Zs8991zg39EhQ4bkz3/+czGfPRQ4csMNN2TIkCELlLTjjjsum266aT73uc9lhx12aDy1hqRbt24u4G+Ffz3Fb/6Kp6+88kqT7Z07d15g2/Lqu9/9biZPnpwrrrgit9xyy0JXgu3atWu22GILB1r+6frrr0+nTp2y++67L3LM7rvvno4dO+ZXv/pVGyZr37yvtdyIESNy77335u233651FGARzj333DzyyCM57LDDMn78+PzP//zPAmN69+6dj370o7n77rtrkLDlFDjyy1/+Mk888US23HLLxm1//OMfc+GFF6Z79+75/Oc/n/XWWy+jR4/OtddeW8Ok7ceuu+6acePG1TpGcdZZZ51MnDix8f6mm26aJLnjjjsat82YMSMPPvhg1l577TbP1x797ne/y2abbZbDDjtssePWW289pfefnnnmmWy22WaLXMo9eX/2d/PNN88zzzzThsnaN+9rLfftb387O++8c3beeeeMGTNmkV+JwqLNnTs3r7/+eiZOnLjIG/w7brjhhqyzzjq59NJL07lz50WO22ijjTJp0qQ2TNZ6Chx56qmnstlmm6Vjx46N26655ppUKpXceOONufbaa/PYY4+lW7duueKKK2qYtP0455xzUl9fnxNPPNFKbC0wZMiQPPnkk3nzzTeTJPvss0+6du2a448/PieddFJ++MMfZuedd87rr7+eoUOH1jht+/DGG29ko402WuK4OXPmZMaMGW2QqP17++23s/rqqy9x3Oqrr57Jkye3QaIyeF9ruQ033DCPP/54nnrqqeyyyy7p0qVL1ltvvfTv33+B20c+8pFax21X7r777gwePDjdunVL7969s/766y/01r9//1pHpXAvv/xyttlmm6ywwuLXbuzYsWOmTJnSRqn+PVahJG+88UZ22GGHJtvuvfferLnmmo2nIK266qoZNGhQ/vd//7cWEdudX/ziFxk6dGjOP//83HzzzRk8eHD69u270CP+lUol//3f/12DlO3PgQcemEmTJuWZZ57JTjvtlFVXXTWXXXZZDj300Jx77rmpVCqpVqvZZJNN8r3vfa/WcduF1VZbrVlHoP/617+atfyn1VdfPS+88MISx73wwgu+1/IDvK+13IQJE5rcnz17thmjZrDiKW2pS5cuzSpmL7/8cjH/JihwpEuXLqmvr2+8/9prr+Wvf/1rPvvZzzYZt/LKKxdzZOLDdtpppzWWjfHjx2f8+PGLHOuDzv/ZfPPNc9111zXZdsABB2THHXfMHXfckSlTpmTAgAHZZ599suKKK9YoZfuy44475pZbbskTTzzRZAGYD7rvvvvy1FNP5ZBDDmnTbO3VJz/5yfz617/OmDFjFrm0+5gxYzJu3Ljsv//+bRuuHfO+1nINDQ21jlCk008/PQ0NDbnoootyxBFHWDSHD9UWW2yRcePG5c0338waa6yx0DEvv/xyHn/88cVeO92eKHCkf//+GTt2bN55552svPLKufbaa1OpVBb4I/7HP/6RNddcs0Yp2xdfxrp0rbPOOjn88MNrHaNdOu644zJq1KgMGzYsP/nJTxb47/IPf/hDDjnkkKywwgo55phjahOynTn22GNz0003Zd99980pp5ySL3/5y+nZs2eSpL6+Ppdffnm+973vpa6uLt/85jdrnLb98L5GW/ngiqfwYfvyl7+cMWPG5IADDsj111+/wCn277zzTg477LDMmTMnX/nKV2qUsmUqVVfcLvcuueSSHHHEEVl//fWzxRZb5Pbbb0+nTp0yfvz4rLbaaknev75mzTXXzDbbbJO77rqrxokp3YQJE3L//ffntddea/IVAh/kCP//ueSSS3L00UenoaEhK620UmbMmJFu3bqlrq4u9fX1qVQqueSSS4r5h6ctfP/738+xxx7beH/VVVdNkiarBZ533nlNxgBtY80118yuu+5qFVjazPDhw3P99denW7du2WGHHXLnnXdmww03zMCBA3Pfffelvr4+I0aMKOarZRQ4MmfOnAwfPrzxO6S6du2an/70p/n85z/fOGbUqFH5zGc+k+9973sZOXJkraJSuPfeey9f/vKXG//RXtzbT6VS8bUVH/Dwww/n7LPPzh/+8Ie8++67Sd7/uoXBgwfn5JNPzo477ljjhO3P/fffn7PPPjv33XdfZs6cmeT9U8YHDx6cE088MYMGDapxQkrz717f5msa3jd8+PCMGzcuf/3rX2sdheVEtVrN+eefn/POO2+Bxat69uyZE044ISeddFIqlUqNEraMAkejCRMm5M0338zAgQMXuJj4iSeeyN/+9rdst912WWuttWqUkNIdffTR+eEPf5g111wzBx54YPr3759u3botcvzBBx/chunKUK1WM3ny5DQ0NGT11Vd37UgzzJs3L2+99VaS9xc4WdzXC8Di1NXVtfoDXqVSsbrnP02aNCkf//jHc/DBB+d73/veElcHhKVl3rx5+dOf/pQJEyakoaEhffv2zcc//vEmK7GXQIGDZhgyZEgqlUquuuqq9O3bN0OGDGn2cyuVSu65554PMV05evXqlYaGhjz55JPp1atXrePAcs37WssNHjz43zpCf++99y7FNOX67ne/m5dffjlXX3111l9/fSueQgspcMu5SZMm5fHHH8/AgQMzYMCARY777W9/m2q1mj333LMN07Uf84+6PvvssxkwYECLjuA7FfD/dOvWLXvssUduuummWkeB5Z73NWpl/t9ecz6C+lvj37Gsfs41Z72cmzNnTvbdd9/svPPOizya+pe//CV77bVXhg4dWswf9tL28ssvJ0n69OnT5D4ts+mmmzb5ygoWZFak5eyz1vG+Rq1Y8ZS2sqx+zlXglnP9+/fPjjvumPvuuy+TJk1Kv379FhhzzTXXpFKpLNfXI6277rqLvU/zfOtb38qBBx6Yxx9/PFtuuWWt47RLY8aMSaVSyYwZMxrvN1cpF18vbfZZ63hfo1aW588TtK1l9XOuAkcOPvjgPPjgg7n22mtz0kknNXmsWq3mV7/6VVZeeeXsu+++tQnYjr311lv55S9/mUcffTSTJ0/OLrvskhNOOCHJ+99z89JLL2XXXXfNSiutVOOk7cN//dd/5e9//3t22223HHHEEdltt93Sp0+fRZ66tTyu2GZWpOXss6XL+9qi3X///UmSbbfdNp07d26831xWPoW2tyx+znUNHJk2bVp69eqVddddN88880yTx+6+++7svvvu+epXv5pLL720Rgnbp1//+tf50pe+lHfffTfVarXx6M3Pf/7zJMmdd96ZoUOH5qqrrsoXvvCFGqdtP+6555587Wtfy0svvbTYcVZsg7bnfW3xFnbdYEtmcV3L9X+q1Wquvfba3HrrrXnhhRcybdq0hV4TV6lUlvjvBSzOsvg51wwc6d69e/bdd99cf/31GTduXLbZZpvGx0qcVm4LDz30UIYPH54ePXrkggsuyCc/+clsu+22Tcbssssu6dmzZ37zm98slx90Fua2227L/vvvn7lz52b11VfPuuuuu9ivESB5/fXX8/zzz2ejjTZq8hUeL730Uk4++eQ89dRTWWeddXLqqadmu+22q2HS9sM+ax3va0s2/9/Cnj17JklGjBixXJ+G21qzZ8/OXnvtlT/84Q+LXMikuYucwJIsk59zq1CtVn//+99XK5VK9aijjmrcNn369Gr37t2rG220UQ2TtU+f/vSnqx07dqz+7//+b+O2SqVSPfTQQ5uM22WXXaobbrhhW8drt7baaqvqCiusUL3yyiurDQ0NtY5ThGOOOaZaV1dXff755xu3TZ06tdqrV69qXV1dtVKpVCuVSnWllVaq/vWvf61h0vbDPmsd72tLNnjw4Oo555zTeP++++5r8ndG85x55pnVSqVS3WeffaovvvhidcSIEdW6urrq7Nmzq88991z19NNPr3bv3r16wgkn1Doqy4hl7XOubzMlSRqvRbr++usbT/G45ZZb8u6772bEiBE1Ttf+/PGPf8z222+frbbaarHjevXqlddee62NUrV/zz77bAYNGpSDDz7YUetmGjNmTD760Y82Wf74yiuvzOuvv54DDjggzz//fC688MLMnDkzF1xwQQ2Tth/2Wet4X1uy++67L88991zj/Z133jnnnHNODROV6YYbbsiqq66aX/3qV/nIRz7SeB30iiuumI022iinnnpqbr/99lxwwQWNp+/Cv2NZ+5yrwJHk/VMVvvCFL2Ty5Mn57W9/m+T9aeW6uroi/7A/bDNmzMgaa6yxxHFTpkxpgzTlWH311bP66qvXOkZRXnnllfTv37/Jtttvvz0rrLBCLr744my44YY55phjsvnmm+e+++6rUcr2xT5rHe9rS9axY8dMnz698X61WnWaXyu8+OKL2XbbbdO1a9ckaSxwH7xG8D/+4z+y44475pJLLqlJRpYty9rnXAWORoccckiq1WquvvrqvP7667n77ruz8847p2/fvrWO1u706dMnTz/99GLHVKvVPPXUU1l//fXbKFX795//+Z+5//77895779U6SjGmTZvWZLW/efPm5aGHHsrWW2/dpAwPHDgwf//732sRsd2xz1rH+9qSbbDBBrnnnnty3333ZeLEiUmSd999NxMnTmzWjfd16NCh8TrCJI1F7s0332wyrk+fPnn++efbNBvLrmXpc64CR6ONNtooH//4x3Pbbbflkksuybx588q7qLON7LHHHnn++edz/fXXL3LMFVdckUmTJmWvvfZqw2Tt2xlnnJH11lsv++yzj1XFmql3795NTtl64IEH8u6772bw4MFNxs2dOzcdO3Zs43Ttk33WOt7XluwrX/lK3n777QwZMqSxxN58881Zf/31l3j711nh5VmfPn2aHDzZYIMNkiQPP/xwk3FPPvmkha5Yapalz7lWoaSJgw8+OEcccUTOOuusdO/ePZ/5zGdqHaldOumkk/KrX/0qI0aMyOOPP5799tsvSTJ9+vQ8/vjjGTVqVM4999ysscYa+eY3v1njtO3Hpz/96XTo0CH33HNPBg4cmPXWW2+R3wNXqVRyzz331CBl+7L99tvnuuuuy8UXX5xddtklp5xySiqVSvbee+8m45599tnG70Fb3tlnreN9bcmOOuqo9O3bN7feemv+/ve/5957782aa66ZgQMH1jpaUbbbbruMGjUqs2bNSqdOnbLnnnvmm9/8Zo455ph07tw5ffr0yeWXX55nn312gf9uWbxbb701U6dOTZIiTw38sC0zn3NruYIK7c+UKVOqnTt3rtbV1VUPO+ywWsdp1/74xz9W11577WqlUqnW1dU1uVUqlepaa61Vffjhh2sds12Zv/pfc251dXW1jtsuPPXUU9UuXbo0+dsaMmRIkzEvv/xytVKpVL/0pS/VKGX7Yp+1nve1llnYKp0s2W233Vbt1atXdfTo0Y3bjj322CZ/d5VKpdqtWzerfLbQwIEDG/chC1pWPueagaOJlVdeOUcddVQeffTRfOUrX6l1nHZt++23z/PPP5+f/exnueuuuzJhwoQ0NDSkb9++2W233fLVr361yTn+JC+//HKtIxRnk002yQMPPJDvf//7mTx5crbeeuscf/zxTcb8/ve/z+abb5599923NiHbGfus9byvtcx3vvOdbLnllrWOUZy99tprgZVML7jggnz84x/PLbfckilTpmTAgAE56qijsuGGG9YoZZk+/vGPp1evXrWO0W4tK59zK9Wq5ZMAAABKYBETAACAQihwAAAAhVDgAAAACqHAsYBZs2bltNNOy6xZs2odpSj2W8vZZ61jv7WcfdY69lvL2WetY7+1nH3WOsvCfrOICQuor69Pz549M3Xq1PTo0aPWcYphv7WcfdY69lvL2WetY7+1nH3WOvZby9lnrbMs7DczcAAAAIVQ4AAAAArhi7xrpKGhIa+++mq6d++eSqVS6zhN1NfXN/lfmsd+azn7rHXst5azz1rHfms5+6x17LeWs89ap73ut2q1mmnTpqV3796pq1v8HJtr4Grk73//e/r161frGAAAQDsxadKk9O3bd7FjzMDVSPfu3T9wr33NwLVnF11/U60jFOnZh56tdYTiXP79U2odAQBYzjTtCAunwNXI/502WWl3p1C2Z11W6lrrCEXq2KlzrSMUyH+XreOkDgBoreb0AouYAAAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwLXA7bffnsMOOywbb7xxevToka5du2bzzTfPmWeemVmzZtU6HgAAsIxbodYBSvLFL34xM2fOzKabbprNNtssU6dOzaOPPpqTTz4599xzT+6888506NCh1jEBAIBllALXApdddll23333dOnSpXHbtGnTMnz48Nx222259tprM2LEiBomBAAAlmVOoWyBYcOGNSlvSdK9e/dcdNFFSZJbb711kc+dNWtW6uvrm9wAAABawgxcC73wwgu544478uKLL2b69OlpaGhItVptfGxRzjrrrJx++ultFRMAAFgGKXDNVK1Wc9xxx+Wiiy5qLGz/atq0aYt8/siRI3Psscc23q+vr0+/fv2Wek4AAGDZ5RTKZrrhhhty4YUXpm/fvrnpppvyyiuvZPbs2alWq40rUC6q2CVJp06d0qNHjyY3AACAljAD10yjRo1Kklx66aXZa6+9mjw2fvz4WkQCAACWM2bgmmnKlClJkr59+y7w2I033tjWcQAAgOWQAtdMAwYMSJJcfvnlTU6VHDt2bM4777xaxQIAAJYjClwzHXXUUenatWsuueSSbLrppjnggAMyaNCg7LTTTjn88MNrHQ8AAFgOKHDNNGDAgIwbNy577713Jk+enNGjR+fdd9/NZZddZgYOAABoExYxaYGBAwdm9OjRC31scStQAgAALA1m4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFCISrVardY6xPKovr4+PXv2rHWM4vTq1b/WEYr02NOP1TpCcbb56Na1jlCkN96YWOsIxalUKrWOUCQfX1quWm2odQRgCaZOnZoePXosdowZOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKUUSBmzBhQiqVSgYPHpzp06fn2GOPTb9+/dKlS5dstdVW+Z//+Z/Gsb/+9a/ziU98Il27ds1aa62Vo446KjNnzmzy85544omccMIJ2XrrrbPGGmukU6dO6d+/f77+9a/n1VdfXezrz5w5MyeddFLWXXfddOrUKRtssEHOOeecVKvVD30/AAAAy7dKtYDmMWHChKy//vrZfvvt09DQkJdffjmDBg3K5MmTc//996dSqeR3v/td/vKXv+SEE07ITjvtlB49euT+++/PW2+9leHDh+faa69t/Hmf//znc/PNN2ezzTbLOuusk+T9UjdhwoSsvfbaGTduXHr37r3Q1+/QoUOeeeaZxjJ533335b333svJJ5+cM844o9m/U319fXr27Ln0dtJyolev/rWOUKTHnn6s1hGKs81Ht651hCK98cbEWkcoTqVSqXWEIhXw8aXdqVYbah0BWIKpU6emR48eix1TVIFLkiFDhmT06NHp2rVrkuTKK6/MoYcemg022CBvvfVW7rzzzmyzzTZJkldffTVbbrll3njjjbz00kvp3//9D//33ntvPvrRj2attdZqfI2GhoacccYZ+c53vpNDDz00P//5zxf6+jvttFNGjx7duGPHjRuX7bbbLp06dcrrr7+ebt26Net3UuBaR4FrHQWu5RS41lHgWk6Ba50CPr60OwoctH/NKXBFnEI5X11dXS699NLG8pYkI0aMyOqrr54XX3wx3/jGNxrLW5L07t07Bx54YJLk/vvvb9y+8847Nylv83/2qaeemj59+mT06NGLfP3LLrusyU7dZpttMnTo0MyYMSPjxo1bZPZZs2alvr6+yQ0AAKAlVqh1gJZYb731MmDAgCbb6urqsu6662by5MnZfffdF3jO/Fm31157rcn2t956K6NHj85TTz2Vd955J/PmzUuSzJkzJ2+99VbefvvtrLrqqk2es+6662ajjTZa4DXmZ/rX1/igs846K6effnozfksAAICFK6rA9enTZ6Hb55+2uLDH5z82a9asxm3XXXddvvKVr+Tdd99d5GtNmzZtgQLXt2/fhY7t3r37Aq/xr0aOHJljjz228X59fX369eu3yPEAAAD/qrhTKP+dx5Pkb3/7Ww455JDMnj07F198cV544YXMmDEj1Wo11Wo122+/fZKFn1vfnJ+/KJ06dUqPHj2a3AAAAFqiqBm4peGOO+7I7Nmzc9xxx+Xoo49e4PHx48fXIBUAAMCSFTUDtzRMmTIlycJPh7z//vvz+uuvt3UkAACAZlnuCtz8BUd++ctfZvr06Y3bX3nllRx++OG1igUAALBEy12B22effbLJJptk3Lhx2WCDDfKf//mf+fSnP50BAwZklVVWyQ477FDriAAAAAu13BW4jh07ZuzYsfna176Wzp0757bbbsuzzz6bI488MnfddVdWXHHFWkcEAABYqEp1Ycst8qGrr69Pz549ax2jOL169a91hCI99vRjtY5QnG0+unWtIxTpjTcm1jpCcSqVSq0jFMnHl5arVhtqHQFYgqlTpy5xtfrlbgYOAACgVAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACrFCrQOQJJVaByjGG2/8rdYRirRB7361jlCc996bXusIRVp99T61jlCcKVNer3WEIlUq/u1sqWrVPqOtVGsdYJlmBg4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAoRIsLXKVSyXrrrfchRFmywYMHp1KpZMKECc0af8ghh6RSqWTMmDEfai4AAIC2YAYOAACgEMt0gTvrrLPy7LPPZtttt611FAAAgH/bCrUO8GFae+21s/baa9c6BgAAwFKxVGfg7rjjjuy2225ZZZVV0rlz52y00UY56aST8s477yx0/Ny5c/PDH/4wW2+9dbp165Zu3bpl2223zaWXXpp58+Y1+3WnTp2aQYMGpVKp5Mgjj0y1Wk2y6Gvg1ltvvVQqlSTJFVdckc022yxdunRJr1698tWvfnWReSdMmJDhw4dnjTXWSNeuXbPNNtvk+uuvz4QJE1KpVDJ48OBmZwYAAGippVbgzjrrrOy1114ZM2ZMtt566+y7776ZMWNGzjnnnHziE5/I66+/3mT8vHnzMmzYsBx11FF58cUXs9tuu2XXXXfNc889l69//ev5r//6rzQ0NCzxdV9//fUMHjw4Y8eOzamnnpof/vCHjeVsSU444YR84xvfyNprr52hQ4emWq3m8ssvzz777NNYAud78cUXs+222+a6667LyiuvnH322Sddu3bN8OHDc/HFFzd7PwEAALTWUjmF8rHHHsspp5ySbt265e67784nPvGJJMmsWbNy0EEH5de//nW+8Y1v5Kabbmp8zsUXX5w77rgjm2yySe65556stdZaSZLXXnstO++8c0aNGpVLLrkkRxxxxCJfd8KECdltt93y0ksv5Qc/+EGOPPLIFuW+5ppr8uSTT2ajjTZKkkyePDnbb799xo4dm3vvvTdDhgxpHHv44YfnzTffzOGHH54f/ehH6dChQ5Lk97//ffbee+8WvS4AAEBrLJUZuB/96EdpaGjIkUce2VjekqRTp0750Y9+lC5dumTUqFGZNGlS42M/+MEPkiQXXnhhY3lL3r9u7bzzzkuSfP/731/kaz799NPZcccdM2HChFxzzTUtLm9J8v/+3/9rLG9Jsvrqq+fwww9Pktx///2N21988cXcc889WXnllXPeeec1lrck+dSnPpXPfvazS3ytWbNmpb6+vskNAACgJZZKgRs7dmyS5MADD1zgsTXXXDO77757Ghoa8uCDDyZJJk6cmIkTJ2aNNdbI7rvvvsBzPv3pT2fllVfOiy++mH/84x8LPP7www9n0KBBmTJlSkaNGrXQ122Ohb32gAEDkrw/Ezjf/Nx77LFHunXrtsBzPve5zy3xtc4666z07Nmz8davX79WZQYAAJZfS6XAvfrqq0myyC/4nr/9lVdeaTJ+3XXXXej4SqXS+Nj853zQQQcdlLfffjtXXnllPv3pT7c6d9++fRfY1r179yTvz5jNN7/MLap0rbPOOkt8rZEjR2bq1KmNtw/ORgIAADRHm3wPXHMXFWnucw444IAkyamnnrrQGbrmqqtru6/B69SpU3r06NHkBgAA0BJLpcH07t07SfK3v/1toY9PmDAhSdKnT59mjf/gY/Of80Hf/e53c+KJJ+b555/PkCFD8sYbb7Q6e3PM/y65Rc2amU0DAADawlIpcP/xH/+RJLnuuusWeOzNN9/M73//+1Qqley4445J3j/lcJ111smbb76Ze+65Z4Hn3H777ZkyZUo22GCD9OrVa6GvefbZZ+e4447Ls88+myFDhuTNN99cGr/KQu2www5J3l9xcvr06Qs8fuONN35orw0AADDfUilw3/jGN1JXV5cf/OAHGTduXOP22bNn58gjj8zMmTOz//77N7mGbP6qkccee2yT8vWPf/wjxx9/fJLk6KOPXuzrnnfeeTn22GPz9NNPZ5dddsnkyZOXxq+zgA033DC77LJLpkyZkhNPPLHJ99Pddddduf766z+U1wUAAPigpVLgtt122/y///f/Ul9fn+233z677bZbDjjggGywwQa54YYbsuGGG+bHP/5xk+d885vfzNChQ/Pkk09mww03zP7775/99tsvAwYMyLPPPpt99903X//615f42hdccEGOPvro/OUvf8kuu+ySt956a2n8Sgu49NJLs8Yaa+THP/5xBg4cmOHDh2fw4MHZY4898tWvfjVJ0rFjxw/ltQEAAJKluIjJt7/97dx2223Zaaed8thjj+U3v/lNOnXqlBNOOCGPPPJIk+96S5IOHTpk9OjR+f73v5/+/fvn97//fe68885stNFG+fGPf5ybbrqp2YuMXHzxxTnyyCPz5JNPZtddd83bb7+9tH6tRhtuuGEeeeSRHHDAAXn77bdzyy23pL6+PldddVU+//nPJ0lWW221pf66AAAA81Wq1Wq11iFKd/bZZ2fkyJE5++yzc+KJJzbrOfX19enZs+c/77V8lc7lVVuuHLosWXHFTrWOUJz33lvweleWbPXVF1x4isWbMuX1WkdgOfHBS0Dgw6VetNbUqVOXuFq9T8PN9N577+WZZ55ZYPu9996bM888MyussELjTBwAAMCHYYVaByjFO++8k0022SQbbbRRNtxww3Tu3DkvvPBC/vznPydJzj///EV+MTkAAMDSoMA1U8+ePXPcccflrrvuyh//+MfU19dn5ZVXztChQ3PkkUdm6NChtY4IAAAs41wDVyOugWsd18C1jmvgWs41cK3jGriWcw0cbcU1cLQd9aK1XAMHAACwDFHgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVih1gFIkmqtAxSjoaGh1hGKNHv2e7WOUJxTzr+i1hGK9LGPDa51hOI8//wjtY5QpHnz5tY6QnGmT3+n1hGKNHfO7FpHKM7sObNqHaE41Wo1ze0EZuAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwrXTllVemUqnktNNOq3UUAABgOaHAAQAAFEKBAwAAKIQC9wG33357DjvssGy88cbp0aNHunbtms033zxnnnlmZs2a1Thu8ODBOfTQQ5Mkp59+eiqVSuPtyiuvrFF6AABgWbdCrQO0J1/84hczc+bMbLrpptlss80yderUPProozn55JNzzz335M4770yHDh2yxx57ZO7cuXnwwQez+eabZ4sttmj8GRtssEHtfgEAAGCZpsB9wGWXXZbdd989Xbp0adw2bdq0DB8+PLfddluuvfbajBgxIieddFJ69eqVBx98MPvuu6+FTAAAgDbhFMoPGDZsWJPyliTdu3fPRRddlCS59dZbW/2zZ82alfr6+iY3AACAljAD9y9eeOGF3HHHHXnxxRczffr0NDQ0pFqtNj7WWmeddVZOP/30pRUTAABYDilw/1StVnPcccfloosuaixs/2ratGmt/vkjR47Mscce23i/vr4+/fr1a/XPAwAAlj9OofynG264IRdeeGH69u2bm266Ka+88kpmz56darXauALloopdc3Tq1Ck9evRocgMAAGgJM3D/NGrUqCTJpZdemr322qvJY+PHj69FJAAAgCbMwP3TlClTkiR9+/Zd4LEbb7xxgW0dO3ZMksydO/fDDQYAAPBPCtw/DRgwIEly+eWXNzlVcuzYsTnvvPMWGN+7d+8kyfPPP982AQEAgOWeAvdPRx11VLp27ZpLLrkkm266aQ444IAMGjQoO+20Uw4//PAFxm+33XZZc801c9NNN2Xw4ME57LDD8qUvfSl//OMfa5AeAABYHihw/zRgwICMGzcue++9dyZPnpzRo0fn3XffzWWXXbbQGbjOnTvn9ttvz2677ZYnnngiV155ZX72s5/lr3/9aw3SAwAAywOLmHzAwIEDM3r06IU+trAVKLfZZpvceeedH3YsAACAJGbgAAAAiqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBCVarVarXWI5VF9fX169uxZ6xjAInTqtFKtIxSpV6/1ax2hOJ07d611hCIN2n1YrSMUZ+xd/1PrCEWaO3dOrSMUZ8KEv9Q6QnGq1WrmzZuTqVOnpkePHosdawYOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocC1w++2357DDDsvGG2+cHj16pGvXrtl8881z5plnZtasWbWOBwAALONWqHWAknzxi1/MzJkzs+mmm2azzTbL1KlT8+ijj+bkk0/OPffckzvvvDMdOnSodUwAAGAZpcC1wGWXXZbdd989Xbp0adw2bdq0DB8+PLfddluuvfbajBgxooYJAQCAZZlTKFtg2LBhTcpbknTv3j0XXXRRkuTWW29d5HNnzZqV+vr6JjcAAICWMAPXQi+88ELuuOOOvPjii5k+fXoaGhpSrVYbH1uUs846K6effnpbxQQAAJZBClwzVavVHHfccbnooosaC9u/mjZt2iKfP3LkyBx77LGN9+vr69OvX7+lnhMAAFh2OYWymW644YZceOGF6du3b2666aa88sormT17dqrVauMKlIsqdknSqVOn9OjRo8kNAACgJczANdOoUaOSJJdeemn22muvJo+NHz++FpEAAIDljBm4ZpoyZUqSpG/fvgs8duONN7Z1HAAAYDmkwDXTgAEDkiSXX355k1Mlx44dm/POO69WsQAAgOWIAtdMRx11VLp27ZpLLrkkm266aQ444IAMGjQoO+20Uw4//PBaxwMAAJYDClwzDRgwIOPGjcvee++dyZMnZ/To0Xn33Xdz2WWXmYEDAADahEVMWmDgwIEZPXr0Qh9b3AqUAAAAS4MZOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACjECrUOAC1TqXUAlhNz5syqdYQivfnmpFpHKM5+/3VkrSMUad6cubWOUJwZM+prHaFI7703vdYRilOtNtQ6QnGq1Wqzx5qBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBa4Fbr/99hx22GHZeOON06NHj3Tt2jWbb755zjzzzMyaNavW8QAAgGXcCrUOUJIvfvGLmTlzZjbddNNsttlmmTp1ah599NGcfPLJueeee3LnnXemQ4cOtY4JAAAsoxS4Frjsssuy++67p0uXLo3bpk2bluHDh+e2227LtddemxEjRtQwIQAAsCxzCmULDBs2rEl5S5Lu3bvnoosuSpLceuuti3zurFmzUl9f3+QGAADQEmbgWuiFF17IHXfckRdffDHTp09PQ0NDqtVq42OLctZZZ+X0009vq5gAAMAySIFrpmq1muOOOy4XXXRRY2H7V9OmTVvk80eOHJljjz228X59fX369eu31HMCAADLLqdQNtMNN9yQCy+8MH379s1NN92UV155JbNnz061Wm1cgXJRxS5JOnXqlB49ejS5AQAAtIQZuGYaNWpUkuTSSy/NXnvt1eSx8ePH1yISAACwnDED10xTpkxJkvTt23eBx2688ca2jgMAACyHFLhmGjBgQJLk8ssvb3Kq5NixY3PeeefVKhYAALAcUeCa6aijjkrXrl1zySWXZNNNN80BBxyQQYMGZaeddsrhhx9e63gAAMByQIFrpgEDBmTcuHHZe++9M3ny5IwePTrvvvtuLrvsMjNwAABAm7CISQsMHDgwo0ePXuhji1uBEgAAYGkwAwcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKsUKtA5AklVoHKEi11gFYTqy4QsdaRyhSly7dah2hOI8/NqbWEYq04y571jpCcdZcc91aRyjS5Ml/r3WE4rz99mu1jlCgapJ5zRppBg4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACrFCrQMsL2bNmpVZs2Y13q+vr69hGgAAoERm4NrIWWedlZ49ezbe+vXrV+tIAABAYRS4NjJy5MhMnTq18TZp0qRaRwIAAArjFMo20qlTp3Tq1KnWMQAAgIKZgQMAACiEAgcAAFAIBW4pGDFiRAYOHJhRo0bVOgoAALAMU+CWgokTJ+b555/P1KlTax0FAABYhilwAAAAhbAK5VIwZsyYWkcAAACWA2bgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAr1DoA0BYqtQ5QnEpdh1pHKFJdnX9WWmrFFTvWOkKR3pv+Xq0jFKeuznH71pg7d06tI0AT/ksGAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKESbF7iXXnopEydObOuXbZGHHnoo7733Xq1jAAAANNEmBa6+vj5XXHFF/uM//iMbbLBB/vSnPzV5vFqt5rrrrsuQIUOyyiqrpHPnztl4441z2mmnZcaMGQv9mW+99VaOP/74bLjhhuncuXNWXXXV7LHHHrnzzjsXOv5vf/tbvva1r2XAgAFZaaWVsuqqq2aTTTbJV7/61Tz//PNNxo4cOTK9evXKV7/61fzxj39cOjsBAADg3/ShFbiGhobceeedOfDAA9OrV698+ctfzoMPPpiddtopAwcObDLuwAMPzPDhw/PYY49liy22yJ577pnp06fn9NNPz84775yZM2c2+dmvvPJKtt1225x//vmZPXt29t1332y55Za5++6786lPfSoXXXRRk/GTJk3KVlttlZ/85CdJkj333DM77bRTOnXqlJ/+9Kd56KGHmowfNmxYVlpppVx++eXZcccdM2DAgHzve99r9zOHAADAsq1SrVarS/MHPvfcc7nqqqtyzTXX5JVXXkmSbLTRRjnooIPyhS98Ieuuu26T8eedd15OOOGEDB48ONddd1169eqVJJk9e3a+/vWv52c/+1lOPPHEnH322Y3P2XvvvXPbbbdl+PDh+cUvfpGOHTsmSR544IF86lOfyqxZszJu3LhsscUWSZLvfOc7+e53v5sjjjgiP/zhD5u8/sSJEzNnzpx85CMfabJ93rx5ufvuu3P11VfnlltuyYwZM1KpVLLzzjvnkEMOyf7775+uXbu2ej/V19enZ8+e/7xXafXPWf4s1T/X5Yi/sZbq3Ln1/30vz7p3X7XWEYrTu/dHljyIBWy29SdrHaE4zz/1pyUPYgGvvvpSrSMU5x//GF/rCMWpVquZN29Opk6dmh49eix27FIpcFOmTMl1112Xq666Ko8++miSZPXVV8/nPve5jBgxIttuu+1Cnzd37tysvfbamTlzZl566aWstdZaTR6fOXNm+vfvn1mzZmXy5Mmpq6vL+PHj85GPfCTdunXL3/72t6y6atMPC9/61rdy4YUX5ktf+lJ++tOfJkm+/vWv59JLL80tt9ySYcOGtfj3mzZtWm6++eZcffXVGTNmTKrVarp165b//M//zMEHH5yddtoplcriPyDPmjUrs2bNarxfX1+ffv36/fOeD9fNp8C1jr+xllLgWkeBazkFrnUUuJZT4FpHgWs5Ba7lWlLg/u1TKE888cSsvfba+cY3vpE///nP2X///XPLLbfk1VdfzY9+9KNFlrck+dOf/pTJkydnhx12WKC8JUmXLl2y9dZbZ8qUKXnhhReSvD/LliR77LHHAuUtSQ466KAkydixYxu3bb311kmSb3/727nttttavEBJ9+7dc8ghh+QPf/hD/va3v+V73/te+vbtmyuvvDI777xz+vfvv8B1ff/qrLPOSs+ePRtv/1feAAAAmuffLnCPPPJIZs2alQ4dOuT444/Pj3/84wwbNiwrrrjiEp87YcKEJMldd92VSqWy0Nvtt9+eJJk8eXKS5NVXX02SrLfeegv9mfO3zz99M0kOOeSQfPazn80zzzyTvffeO6usskoGDRqUM888M//4xz9a9Pv269cvI0eOzE9+8pNss802jb/Hkq6PGzlyZKZOndp4mzRpUoteFwAAYIV/9wecddZZufzyy3PTTTfljDPOyFlnnZVdd901Bx10UPbdd9/FXifW0NCQJNlggw2y4447LvZ1VltttWblWdipjB06dMgNN9yQk046Kbfeemv+8Ic/5JFHHsnYsWNz9tln53e/+1122GGHJf7s5557Ltdcc01++ctfNha2TTbZpPE0ysXp1KlTOnXq1KzfAQAAYGH+7QK3/fbbZ/vtt8+PfvSj3Hzzzbnqqqty55135ve//326deuW/fbbLwcddFB22WWX1NU1nfDr27dvkmTgwIG58sorm/V6vXv3TvL+1wIszPxZvT59+izw2JZbbpktt9wyp512Wurr63PaaafloosuyjHHHNN47d6/euONN3L99dfnmmuuybhx45K8XyaPOOKIHHzwwY2zcAAAAB+2pb4KZfL+yo5XX311rr766sZr13r37p3hw4fnoIMOymabbZbk/YU91lprrTQ0NGTChAkLvabtX81fxKR79+6ZOHFiVl555SaPH3/88Tn//PObLGKyKLNmzUqXLl3SuXPnJt83N3PmzIwePTrXXHNNfv/732fu3LlZccUVM3To0Bx88MH59Kc/3bjyZWtZhbK1LGLSOv7GWsoiJq1jEZOWs4hJ61jEpOUsYtI6FjFpOYuYtFybLmKyMOuss05OOeWU/PWvf82DDz6Yr3zlK5kxY0bOP//8bL755o3XtXXq1CknnHBCpk2blv333z/jxy/4f/Yrr7ySa665pvF+//79s9dee2XatGk5+uijM2fOnMbHHnrooVx66aXp0KFDvvGNbzRuv+aaa/LUU08t8LN/+9vfplqtLrCgyD777JPPf/7zuf322/Oxj30sF198cV555ZXceuut2X///f/t8gYAANAa//YplEuyww47ZIcddsj3v//93HLLLbnqqqvywUm/k046qfHaso033jhbbrll1l9//cyePTvPP/98nnnmmWy22WaNq0smyWWXXZb/+I//yNVXX5377rsv22+/fd58882MGTMm8+bNywUXXND4HXBJcvPNN2fEiBH5yEc+ko997GPp0qVLXn755TzyyCOpq6vLGWec0STzaqutlmOPPTaHHHJIPvaxj33YuwgAAKBZPpRTKFtj9OjRufzyy/PYY49lypQpWWWVVdKvX7/ssssu+dznPpetttqqyfi33norZ511Vm655ZZMmjQpK620Urbddtt861vfyu67795k7P33358bb7wxDz74YCZNmpTp06end+/ejeNrcR2bUyhbq138uRbI31hLOYWydZxC2XJOoWwdp1C2nFMoW8cplC3nFMqWa/Mv8qblFLjW8ufaOv7GWkqBax0FruUUuNZR4FpOgWsdBa7lFLiWq/k1cAAAACx9ChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKocABAAAUQoEDAAAohAIHAABQCAUOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUAgFDgAAoBAKHAAAQCEUOAAAgEIocAAAAIVQ4AAAAAqhwAEAABRCgQMAACiEAgcAAFAIBQ4AAKAQChwAAEAhFDgAAIBCKHAAAACFUOAAAAAKsUKtAyyvqtXqB+/VLAfLC39jLdX0v1Gaq6GhodYRijNv3txaRyjS7Nmzah2hOPPmzal1hCI1NMyrdYTi+De05ebvs+bsu0rVHq6Jv//97+nXr1+tYwAAAO3EpEmT0rdv38WOUeBqpKGhIa+++mq6d++eSqVS6zhN1NfXp1+/fpk0aVJ69OhR6zjFsN9azj5rHfut5eyz1rHfWs4+ax37reXss9Zpr/utWq1m2rRp6d27d+rqFn+Vm1Moa6Surm6J7brWevTo0a7+sEthv7WcfdY69lvL2WetY7+1nH3WOvZby9lnrdMe91vPnj2bNc4iJgAAAIVQ4AAAAAqhwLGATp065Tvf+U46depU6yhFsd9azj5rHfut5eyz1rHfWs4+ax37reXss9ZZFvabRUwAAAAKYQYOAACgEAocAABAIRQ4AACAQihwAAAAhVDgAAAACqHAAQAAFEKBAwAAKIQCBwAAUIj/DyNYjPrfz/0gAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_attention(sentence_tokens, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "循环遍历我们的 `test_data`，得到模型对每个测试句子的翻译。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e90457593ad24140b1965124668d19f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "translations = [\n",
    "    translate_sentence(\n",
    "        example[\"de\"],\n",
    "        model,\n",
    "        en_nlp,\n",
    "        de_nlp,\n",
    "        en_vocab,\n",
    "        de_vocab,\n",
    "        lower,\n",
    "        sos_token,\n",
    "        eos_token, \n",
    "    )[0] for example in tqdm(test_data)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 评估`bleu`指标\n",
    "从`evaluate`库中加载BLEU指标："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07463d3eb312408e9ab50269b468af20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script:   0%|          | 0.00/5.94k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb6edbb640ff46c0882abe114cb73f6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading extra modules:   0%|          | 0.00/1.55k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f9720f5d7b64ea8a0be93f18b9003e9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading extra modules:   0%|          | 0.00/3.34k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bleu = evaluate.load(\"bleu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将目标句子放入列表中。并将我们的翻译从token列表转换为字符串，方法是在它们之间用空格连接，并摆脱`<sos>`和`<eos>`tocken。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = [\n",
    "    \" \".join(translation[1:-1]) for translation in translations\n",
    "]\n",
    "\n",
    "references = [\n",
    "    [example[\"en\"]] for example in test_data\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义一个函数，该函数对输入字符串进行tocken化。通过将我们的预测tocken与参考tocken进行比较来计算BLEU分数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tokenizer_fn(nlp, lower):\n",
    "    \n",
    "    def tokenizer_fn(s):\n",
    "        tokens = [token.text for token in nlp.tokenizer(s)]\n",
    "        if lower:\n",
    "            tokens = [token.lower() for token in tokens]\n",
    "        return tokens\n",
    "        \n",
    "    return tokenizer_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer_fn = get_tokenizer_fn(en_nlp, lower)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们在测试集上计算BLEU指标！\n",
    "\n",
    "我们将我们的“predictions”、“references”和我们的“tokenizer_fn”传递给BLEU指标的“compute”方法，以获取我们的结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = bleu.compute(predictions=predictions, references=references, tokenizer=tokenizer_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bleu': 0.2633405950067028,\n",
       " 'precisions': [0.5873529411764706,\n",
       "  0.3334920634920635,\n",
       "  0.19836206896551725,\n",
       "  0.12377358490566037],\n",
       " 'brevity_penalty': 1.0,\n",
       " 'length_ratio': 1.0415071220707612,\n",
       " 'translation_length': 13600,\n",
       " 'reference_length': 13058}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们获得了0.263的BLEU分数, 但是训练的时间也翻倍了。\n",
    "\n",
    "\n",
    "通过方法的进步，可以逐渐提高BLEU分数！\n",
    "\n",
    "\n",
    "在下一个笔记本中，我们将使用相同的架构，但使用适用于所有 RNN 架构的一些技巧 - packed padded sequences 和 masking。我们还将实现一些代码，让我们查看 RNN 在解码输出时关注的输入单词。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}