1 - Sequence to Sequence Learning with Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: 首次使用请运行以下命令安装所需的库 ！\n",
    "\n",
    "#!  pip install datasets evaluate\n",
    "#!  pip install git+https://github.com/mindspore-lab/mindnlp.git\n",
    "#!  python -m spacy download de_core_news_sm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - 序列到序列学习与神经网络\n",
    "\n",
    "在本系列中，我们将使用`mindspore`构建一个机器学习模型，实现从一个序列到另一个序列的转换。我们将以德语到英语的翻译为例，但这些模型可以应用于任何涉及从一个序列到另一个序列的问题，比如摘要，即从同一语言中的一个序列到一个较短序列的转换。\n",
    "\n",
    "在这个第一个笔记本中，我们将从简单的开始，实现来自 [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215) 论文的模型以理解这些一般概念。\n",
    "\n",
    "## 引言\n",
    "\n",
    "最常见的序列到序列（seq2seq）模型是*编码器-解码器*模型，通常使用*循环神经网络*（RNN）来将源（输入）句子编码成一个向量。在本笔记本中，我们将这个向量称为*上下文向量*。我们可以将上下文向量看作是整个输入句子的抽象表示。然后，这个向量由第二个RNN进行*解码*，该RNN学会逐个单词生成目标（输出）句子。\n",
    "\n",
    "![Sequence to Sequence](assets/seq2seq1.png)\n",
    "\n",
    "上面的图片显示了一个翻译的例子。输入/源句子 \"guten morgen\" 通过嵌入层（黄色）传递，然后输入到编码器（绿色）。我们还在句子的开头和结尾附加了一个*序列开始*（`<sos>`）和*序列结束*（`<eos>`）标记。在每个时间步，编码器RNN的输入既是当前单词的嵌入 $e(x_t)$，也是上一个时间步的隐藏状态 $h_{t-1}$，编码器RNN输出新的隐藏状态 $h_t$。我们可以将隐藏状态看作是到目前为止句子的向量表示。RNN可以表示为 $e(x_t)$ 和 $h_{t-1}$ 的函数：\n",
    "\n",
    "$$h_t = \\text{EncoderRNN}(e(x_t), h_{t-1})$$\n",
    "\n",
    "虽然这里我们使用RNN一词，但是，它可以是任何循环架构，如LSTM（长短时记忆）或GRU（门控循环单元）。\n",
    "\n",
    "在这里，我们有 $X = \\{x_1, x_2, ..., x_T\\}$，其中 $x_1 = \\text{<sos>}, x_2 = \\text{guten}$，等等。初始隐藏状态 $h_0$ 通常初始化为零或学习参数。\n",
    "\n",
    "一旦最终单词 $x_T$ 通过嵌入层传入RNN，我们使用最终隐藏状态 $h_T$ 作为上下文向量，即 $h_T = z$。这是整个源句子的向量表示。\n",
    "\n",
    "现在我们有了上下文向量 $z$，我们可以开始解码以获得输出/目标句子 \"good morning\"。同样，在目标句子中，我们附加了序列开始和序列结束标记。在每个时间步，解码器RNN（蓝色）的输入是当前单词的嵌入 $d(y_t)$ 和前一个时间步的隐藏状态 $s_{t-1}$，其中初始解码器隐藏状态 $s_0$ 是上下文向量 $s_0 = z = h_T$，即初始解码器隐藏状态是最终编码器隐藏状态。因此，类似于编码器，我们可以表示解码器为：\n",
    "\n",
    "$$s_t = \\text{DecoderRNN}(d(y_t), s_{t-1})$$\n",
    "\n",
    "虽然在图中显示的输入/源嵌入层 $e$ 和输出/目标嵌入层 $d$ 都是黄色的，但它们是具有各自参数的两个不同的嵌入层。\n",
    "\n",
    "在解码器中，我们需要从隐藏状态到实际单词，因此在每个时间步，我们使用 $s_t$ 预测（通过传递给 `Linear` 层，显示为紫色）我们认为是序列中的下一个单词 $\\hat{y}_t$。\n",
    "\n",
    "$$\\hat{y}_t = f(s_t)$$\n",
    "\n",
    "解码器中的单词总是一个接一个地生成的，每个时间步一个。我们始终使用 `<sos>` 作为解码器的第一个输入 $y_1$，但对于后续输入 $y_{t>1}$，我们有时使用实际的、地面真实的下一个单词 $y_t$，有时使用我们的解码器预测的单词 $\\hat{y}_{t-1}$。这称为*教师强迫*，在 [这里](https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/) 有更多关于它的信息。\n",
    "\n",
    "在训练/测试我们的模型时，我们始终知道目标句子中有多少单词，因此一旦生成了这么多单词，我们就停止生成。在推理期间，通常一直生成单词，直到模型输出一个 `<eos>` 标记或生成了一定数量的单词为止。\n",
    "\n",
    "一旦我们有了预测的目标句子 $\\hat{Y} = \\{ \\hat{y}_1, \\hat{y}_2, ..., \\hat{y}_T \\}$，我们将其与实际目标句子 $Y = \\{ y_1, y_2, ..., y_T \\}$ 进行比较，计算损失。然后，我们使用此损失更新模型中的所有参数。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据\n",
    "\n",
    "首先，导入所有必要的库。我们将主要使用以下库：\n",
    "\n",
    "- [mindspore](https://pytorch.org/) 用于创建模型\n",
    "- [spaCy](https://spacy.io/) 用于辅助数据的分词\n",
    "- [datasets](https://huggingface.co/docs/datasets/index) 用于加载和操作数据\n",
    "- [evaluate](https://huggingface.co/docs/evaluate/index) 用于计算指标\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore as ms\n",
    "import mindspore.context as context\n",
    "import mindspore.nn as nn\n",
    "import mindspore.ops as ops\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "import spacy\n",
    "import datasets\n",
    "from tqdm.notebook import tqdm\n",
    "import evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将设置所有可能的随机种子以确保结果的可重复性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 1234\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "ms.set_seed(seed)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集\n",
    "\n",
    "接下来，我们将使用 `datasets` 库加载我们的数据集。在使用 `load_dataset` 函数时，我们传递数据集的名称，即 `bentrevett/multi30k`。\n",
    "\n",
    "我们将使用的数据集是 [Multi30k 数据集](https://github.com/multi30k/dataset) 的一个子集，该数据集托管在 HuggingFace 数据集中心的 [这里](https://huggingface.co/datasets/bentrevett/multi30k)。这个子集包含了使用 [这里](https://github.com/multi30k/dataset/tree/master/data/task1/raw) 的任务 1 原始数据获得的约 30,000 条并行的英语和德语句子。我们使用了测试集的 \"2016\" 版本。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset json (/root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-632b0bad1bafed4c/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "67879e7983b0469b946788b34aaa838d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['en', 'de'],\n",
       "        num_rows: 29000\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['en', 'de'],\n",
       "        num_rows: 1014\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['en', 'de'],\n",
       "        num_rows: 1000\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = datasets.load_dataset(\"bentrevett/multi30k\")\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了方便起见，我们为每个数据拆分创建一个变量。每个变量都是一个 `Dataset` 对象。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, valid_data, test_data = dataset[\"train\"], dataset[\"validation\"], dataset[\"test\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以索引每个 `Dataset` 以查看单个示例。每个示例有两个特征：\"en\" 和 \"de\"，分别是平行的英语和德语句子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'en': 'Two young, White males are outside near many bushes.',\n",
       " 'de': 'Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche.'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  分词器（Tokenizers）\n",
    "\n",
    "接下来，我们将加载包含分词器的 spaCy 模型。\n",
    "\n",
    "分词器用于将字符串转换为构成该字符串的标记列表，例如 \"good morning!\" 变为 [\"good\", \"morning\", \"!\"]。从现在开始，我们将讨论句子是标记序列，而不是说它们是单词序列。有什么区别？嗯，\"good\" 和 \"morning\" 都是单词和标记，但 \"!\" 不是一个单词。我们可以说 \"!\" 是标点符号，但术语标记更一般，包括：单词、标点、数字和任何特殊符号。\n",
    "\n",
    "spaCy 为每种语言都有一个模型（\"de_core_news_sm\" 用于德语，\"en_core_web_sm\" 用于英语），我们需要加载这些模型以便访问每个模型的分词器。\n",
    "\n",
    "**注意**：必须首先使用以下命令在命令行中下载模型：\n",
    "```bash \n",
    "python -m spacy download en_core_web_sm\n",
    "python -m spacy download de_core_news_sm\n",
    "```\n",
    "我们加载模型如下：\n",
    "```python\n",
    "import spacy\n",
    "# 加载英语分词器模型\n",
    "spacy_en = spacy.load('en_core_web_sm')\n",
    "# 加载德语分词器模型\n",
    "spacy_de = spacy.load('de_core_news_sm')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple/\n",
      "Collecting en-core-web-sm==3.7.1\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl (12.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.8/12.8 MB\u001b[0m \u001b[31m561.5 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.8.0,>=3.7.2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from en-core-web-sm==3.7.1) (3.7.2)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.0.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.0.10)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.8)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.9)\n",
      "Requirement already satisfied: thinc<8.3.0,>=8.1.8 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (8.2.2)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.1.2)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.4.8)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.10)\n",
      "Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.3.4)\n",
      "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.9.0)\n",
      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (6.4.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (4.66.1)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.10.13)\n",
      "Requirement already satisfied: jinja2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.1.2)\n",
      "Requirement already satisfied: setuptools in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (41.2.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (23.2)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.3.0)\n",
      "Requirement already satisfied: typing-extensions<4.5.0,>=3.7.4.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (4.4.0)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.21.6)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from catalogue<2.1.0,>=2.0.6->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.15.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2023.11.17)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.7.11)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.1.4)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from typer<0.10.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (8.1.7)\n",
      "Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from weasel<0.4.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from jinja2->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.1.3)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/python-3.7.5/lib/python3.7/site-packages (from click<9.0.0,>=7.1.1->typer<0.10.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (6.7.0)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip3 install --upgrade pip\u001b[0m\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_sm')\n",
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple/\n",
      "Collecting de-core-news-sm==3.7.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-3.7.0/de_core_news_sm-3.7.0-py3-none-any.whl (14.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.6/14.6 MB\u001b[0m \u001b[31m497.6 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.8.0,>=3.7.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from de-core-news-sm==3.7.0) (3.7.2)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.0.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.0.10)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.0.8)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.0.9)\n",
      "Requirement already satisfied: thinc<8.3.0,>=8.1.8 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (8.2.2)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.1.2)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.4.8)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.0.10)\n",
      "Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.3.4)\n",
      "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.9.0)\n",
      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (6.4.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (4.66.1)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.10.13)\n",
      "Requirement already satisfied: jinja2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.1.2)\n",
      "Requirement already satisfied: setuptools in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (41.2.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (23.2)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.3.0)\n",
      "Requirement already satisfied: typing-extensions<4.5.0,>=3.7.4.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (4.4.0)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.21.6)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from catalogue<2.1.0,>=2.0.6->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.15.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2023.11.17)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.7.11)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.1.4)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from typer<0.10.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (8.1.7)\n",
      "Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from weasel<0.4.0,>=0.1.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/python-3.7.5/lib/python3.7/site-packages (from jinja2->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.1.3)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/python-3.7.5/lib/python3.7/site-packages (from click<9.0.0,>=7.1.1->typer<0.10.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (6.7.0)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip3 install --upgrade pip\u001b[0m\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('de_core_news_sm')\n"
     ]
    }
   ],
   "source": [
    "!  python -m spacy download en_core_web_sm\n",
    "!  python -m spacy download de_core_news_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_nlp = spacy.load(\"en_core_web_sm\")\n",
    "de_nlp = spacy.load(\"de_core_news_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以使用 `.tokenizer` 方法为每个 spaCy 模型调用分词器，该方法接受一个字符串并返回一系列 `Token` 对象。我们可以从 token 对象中使用 `text` 属性获取字符串。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['What', 'a', 'lovely', 'day', 'it', 'is', 'today', '!']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string = \"What a lovely day it is today!\"\n",
    "\n",
    "[token.text for token in en_nlp.tokenizer(string)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们将编写一个用于将分词器应用于每个数据拆分中所有示例的函数，并进行一些其他处理。\n",
    "\n",
    "此函数接受来自 `Dataset` 对象的示例，应用英语和德语 spaCy 模型的分词器，将标记列表修剪到最大长度，可选地将每个标记转换为小写，然后将序列开始和序列结束标记附加到标记列表的开头和结尾。\n",
    "\n",
    "这个函数将与 `Dataset` 的 `map` 方法一起使用，该方法需要返回一个包含每个示例中存储输出的特征的名称的字典。由于输出特征名称 \"en_tokens\" 和 \"de_tokens\" 在示例中尚未存在（我们只有 \"en\" 和 \"de\" 特征），因此这将在每个示例中创建两个新特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_example(\n",
    "    example,\n",
    "    en_nlp,\n",
    "    de_nlp,\n",
    "    max_length,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token\n",
    "):\n",
    "    en_tokens = [token.text for token in en_nlp.tokenizer(example[\"en\"])][:max_length]\n",
    "    de_tokens = [token.text for token in de_nlp.tokenizer(example[\"de\"])][:max_length]\n",
    "    if lower:\n",
    "        en_tokens = [token.lower() for token in en_tokens]\n",
    "        de_tokens = [token.lower() for token in de_tokens]\n",
    "    en_tokens = [sos_token] + en_tokens + [eos_token]\n",
    "    de_tokens = [sos_token] + de_tokens + [eos_token]\n",
    "    return {\"en_tokens\": en_tokens, \"de_tokens\": de_tokens}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用以下方式使用 `map` 方法应用 `tokenize_example` 函数。\n",
    "\n",
    "`example` 参数是隐含的，但是 `tokenize_example` 函数的所有额外参数都需要存储在字典中，并传递给 `map` 的 `fn_kwargs` 参数。\n",
    "\n",
    "在这里，我们将所有序列修剪为最大长度为 1000 个标记，将每个标记转换为小写，并分别使用 `<sos>` 和 `<eos>` 作为序列的开始和结束标记。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-632b0bad1bafed4c/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-df1aa7b13386ffa1.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-632b0bad1bafed4c/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-483f10036f73b73e.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-632b0bad1bafed4c/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-855a5188daf48c66.arrow\n"
     ]
    }
   ],
   "source": [
    "max_length = 1_000\n",
    "lower = True\n",
    "sos_token = \"<sos>\"\n",
    "eos_token = \"<eos>\"\n",
    "\n",
    "fn_kwargs = {\n",
    "    \"en_nlp\": en_nlp, \n",
    "    \"de_nlp\": de_nlp, \n",
    "    \"max_length\": max_length,\n",
    "    \"lower\": lower,\n",
    "    \"sos_token\": sos_token,\n",
    "    \"eos_token\": eos_token,\n",
    "}\n",
    "\n",
    "\n",
    "train_data = train_data.map(tokenize_example, fn_kwargs=fn_kwargs)\n",
    "valid_data = valid_data.map(tokenize_example, fn_kwargs=fn_kwargs)\n",
    "test_data = test_data.map(tokenize_example, fn_kwargs=fn_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们可以查看一个示例，确认已添加了两个新特征；这两个特征都是附加了序列开始/结束标记的小写字符串列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'en': 'Two young, White males are outside near many bushes.',\n",
       " 'de': 'Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche.',\n",
       " 'en_tokens': ['<sos>',\n",
       "  'two',\n",
       "  'young',\n",
       "  ',',\n",
       "  'white',\n",
       "  'males',\n",
       "  'are',\n",
       "  'outside',\n",
       "  'near',\n",
       "  'many',\n",
       "  'bushes',\n",
       "  '.',\n",
       "  '<eos>'],\n",
       " 'de_tokens': ['<sos>',\n",
       "  'zwei',\n",
       "  'junge',\n",
       "  'weiße',\n",
       "  'männer',\n",
       "  'sind',\n",
       "  'im',\n",
       "  'freien',\n",
       "  'in',\n",
       "  'der',\n",
       "  'nähe',\n",
       "  'vieler',\n",
       "  'büsche',\n",
       "  '.',\n",
       "  '<eos>']}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 词汇表（Vocabularies）\n",
    "\n",
    "接下来，我们将为源语言和目标语言构建*词汇表*。词汇表用于将数据集中的每个唯一标记与一个索引（整数）关联起来，例如 \"hello\" = 1, \"world\" = 2, \"bye\" = 3, \"hates\" = 4 等。在将文本数据提供给我们的模型时，我们使用词汇表将字符串转换为标记，然后将标记转换为数字，词汇表充当查找表，例如 \"hello world\" 变为 `[\"hello\", \"world\"]`，然后使用给定的索引变为 `[1, 2]`。我们这样做是因为神经网络无法处理字符串，只能处理数字值。\n",
    "\n",
    "我们使用提供的 `build_vocab_from_iterator` 函数从我们的数据集中创建词汇表（每种语言一个），该函数接受一个迭代器，其中每个项都是标记列表。然后，它统计唯一标记的数量，并为每个标记分配一个数字值。\n",
    "\n",
    "理论上，我们的词汇表可能足够大，以便为数据集中的每个唯一标记创建一个索引。然而，如果一个标记存在于我们的验证集和测试集中，但不存在于我们的训练集中，会发生什么？在这种情况下，我们将该标记替换为一个“未知标记”，表示为 `<unk>`，它被赋予自己的索引（通常为索引零）。所有未知标记都将被 `<unk>` 替换，即使这些标记不同，例如，如果标记 \"gilgamesh\" 和 \"enkidu\" 都不在我们的词汇表中，那么字符串 \"gilgamesh hates enkidu\" 将被分词为 `[\"gilgamesh\", \"hates\", \"enkidu\"]`，然后变为 `[0, 24, 0]`（其中 \"hates\" 的索引为 24）。\n",
    "\n",
    "理想情况下，我们希望我们的模型能够处理未知标记，学会使用它们周围的上下文进行翻译。它只有在训练集中也有未知标记时才能学到这一点。因此，在使用 `build_vocab_from_iterator` 创建词汇表时，我们使用 `min_freq` 参数，以便不为在我们的训练集中出现次数少于 `min_freq` 次的标记创建索引。换句话说，使用词汇表时，任何在我们的训练集中出现次数不足两次的标记将在将标记转换为索引时被未知标记索引替换。\n",
    "\n",
    "重要的是要注意，词汇表应仅从训练集构建，而不应从验证集或测试集构建。这可以防止“信息泄漏”到我们的模型中，从而提供人工膨胀的验证/测试分数。\n",
    "\n",
    "我们还使用 `build_vocab_from_iterator` 的 `specials` 参数传递*特殊标记*。这些是我们想要添加到词汇表中但不一定出现在我们的标记化示例中的标记。这些特殊标记将首先出现在词汇表中。我们已经讨论过 `unk_token`、`sos_token` 和 `eos_token`。最后的特殊标记是 `pad_token`，表示为 `<pad>`。\n",
    "\n",
    "在将句子输入模型时，将多个句子一次性传递（称为批处理）而不是一个一个传递更有效。将句子批处理在一起的要求是它们在长度上都必须相同（以标记的数量为准）。我们的大多数句子长度不相同，但我们可以通过在批处理中的每个句子的标记化版本中“填充”（添加 `<pad>` 标记）标记，直到它们的标记数等于批处理中最长的句子。例如，如果我们有两个句子：\"I love pizza\" 和 \"I hate music videos\"。它们将被标记为：`[\"i\", \"love\", \"pizza\"]` 和 `[\"i\", \"hate\", \"music\", \"videos\"]`。然后，第一个标记序列将被填充为 `[\"i\", \"love\", \"pizza\", \"<pad>\"]`。然后，两个序列都可以使用词汇表转换为索引。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "class Vocab:\n",
    "    \"\"\"一个词汇表的实现\"\"\"\n",
    "    def __init__(self, tokens:list, min_freq=0, reserved_tokens:list=None) -> None:\n",
    "        self.default_index = None\n",
    "        if tokens is not None:\n",
    "            # 当第一个条件满足时，就不会跳到第二个判断，避免了空列表报错的情况。\n",
    "            if len(tokens)!=0 and isinstance(tokens[0], list):\n",
    "                tokens = [i for line in tokens for i in line]\n",
    "        else:\n",
    "            tokens = []\n",
    "        if reserved_tokens is None:\n",
    "            reserved_tokens = []\n",
    "        counter=collections.Counter(tokens)\n",
    "        # 按出现词频从高到低排序\n",
    "        self._token_freqs = sorted(counter.items(), key=lambda x:x[1], reverse=True)\n",
    "        # 通过列表,利用序号访问词元。\n",
    "        self.idx_to_token = [] + reserved_tokens # 未知词元<unk>的索引为0, 保留词元排在最前\n",
    "        self.token_to_idx = {\n",
    "            i: k\n",
    "            for k, i in enumerate(self.idx_to_token) \n",
    "        }\n",
    "        \n",
    "        for token, freq in self._token_freqs:\n",
    "            if freq < min_freq:  # 过滤掉出现频率低于要求的词\n",
    "                break\n",
    "            if token not in self.token_to_idx:  \n",
    "                self.idx_to_token.append(token)\n",
    "                self.token_to_idx[token] = len(self.idx_to_token) - 1\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.idx_to_token)\n",
    "    \n",
    "    def __getitem__(self, input_tokens):\n",
    "        \"\"\"输入单字串或序列, 将其全部转化为序号编码\"\"\"\n",
    "        if isinstance(input_tokens, str):\n",
    "            out =  self.token_to_idx.get(input_tokens, self.default_index)\n",
    "            if out is None:\n",
    "                raise Exception('Please call \"set_default_index\" before getting unknown index')\n",
    "            return out\n",
    "        return [self.__getitem__(token) for token in input_tokens]\n",
    "    \n",
    "    def __repr__(self) -> str:\n",
    "        show_items = 5 if len(self) > 5 else len(self)\n",
    "        out = f\"<Vocab with {len(self)} tokens: \"\n",
    "        for i in range(show_items):\n",
    "            out += f'\"{self.idx_to_token[i]}\", '\n",
    "        out += \"...>\"\n",
    "        return out\n",
    "\n",
    "    def __contains__(self, token:str) -> bool:\n",
    "        return token in self.idx_to_token\n",
    "\n",
    "    def to_tokens(self, input_keys):\n",
    "        \"\"\"输入单s索引或序列, 将其全部转化为词元\"\"\"\n",
    "        if isinstance(input_keys, int):\n",
    "            return self.idx_to_token[input_keys] if input_keys < len(self) else self.idx_to_token[0]\n",
    "        elif isinstance(input_keys, (list, tuple)):\n",
    "            return [self.to_tokens(keys) for keys in input_keys]\n",
    "        else:\n",
    "            return self.idx_to_token[0]\n",
    "    \n",
    "    def get_itos(self):\n",
    "        return self.idx_to_token\n",
    "    \n",
    "    def get_stoi(self):\n",
    "        return self.token_to_idx\n",
    "    \n",
    "    def set_default_index(self, idx):\n",
    "        if isinstance(idx, int):\n",
    "            self.default_index = idx\n",
    "        else:\n",
    "            raise Exception(f\"Only type int allowed, got {type(idx)}\")\n",
    "    \n",
    "    def lookup_indices(self, input_tokens):\n",
    "        return self.__getitem__(input_tokens)\n",
    "    \n",
    "    def lookup_tokens(self, idx):\n",
    "        return self.to_tokens(idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Vocab with 5893 tokens: \"<unk>\", \"<pad>\", \"<sos>\", \"<eos>\", \"a\", ...>,\n",
       " <Vocab with 7853 tokens: \"<unk>\", \"<pad>\", \"<sos>\", \"<eos>\", \".\", ...>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_freq = 2\n",
    "unk_token = \"<unk>\"\n",
    "pad_token = \"<pad>\"\n",
    "\n",
    "special_tokens = [\n",
    "    unk_token,\n",
    "    pad_token,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    "]\n",
    "\n",
    "def build_vocab_from_iterator(tokens, min_freq, specials):\n",
    "    return Vocab(tokens, min_freq, specials)\n",
    "\n",
    "\n",
    "en_vocab = build_vocab_from_iterator(\n",
    "    train_data[\"en_tokens\"],\n",
    "    min_freq=min_freq,\n",
    "    specials=special_tokens,\n",
    ")\n",
    "\n",
    "de_vocab = build_vocab_from_iterator(\n",
    "    train_data[\"de_tokens\"],\n",
    "    min_freq=min_freq,\n",
    "    specials=special_tokens,  \n",
    ")\n",
    "\n",
    "en_vocab, de_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们有了词汇表，我们可以查看其中实际包含的内容。\n",
    "\n",
    "我们可以使用 `get_itos` 方法获取我们词汇表中的前十个标记（索引从 0 到 9），其中 itos = \"**i**nt **to** **s**tring\"，它返回一个标记列表。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<unk>', '<pad>', '<sos>', '<eos>', 'a', '.', 'in', 'the', 'on', 'man']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab.get_itos()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'man'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab.get_itos()[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以对德语词汇表执行相同的操作。请注意特殊标记是相同的，且顺序相同（索引从 0 到 3），然而其余的标记是不同的。这是因为尽管它们来自相同的示例，但词汇表实际上是从不同的数据中创建的（一个是英语，一个是德语）。对于不是特殊标记的标记，给定的索引按照出现频率从最高到最低的顺序排序（尽管仍然至少出现 `min_freq` 次）。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<unk>', '<pad>', '<sos>', '<eos>', '.', 'ein', 'einem', 'in', 'eine', ',']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "de_vocab.get_itos()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以使用 `get_stoi`（stoi = \" **s**tring **to** **i**nt \"） 方法从给定的标记获取索引。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab.get_stoi()[\"the\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "作为一种简写，我们可以将词汇表直接用作字典，并传递标记以获取索引。请注意，反过来则不起作用，即 `en_vocab[7]` 不起作用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab[\"the\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每个词汇表的 `len` 给出了每个词汇表中唯一标记的数量。我们可以看到，我们的训练数据中有大约 2000 个更多的德语标记（至少出现两次）比英语数据。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5893, 7853)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(en_vocab), len(de_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们还可以使用 `in` 关键字来获取一个布尔值，指示标记是否在词汇表中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"the\" in en_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "还记得我们是如何将所有标记转换为小写的吗？这意味着在我们的词汇表中不会出现包含任何大写字符的标记。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"The\" in en_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果尝试获取不在词汇表中的标记的索引会发生什么？你会得到 `<unk>`（未知）标记的索引零，对吧？\n",
    "\n",
    "嗯，并不是。`Vocab` 词汇表类的一个怪癖是，您必须手动设置当尝试获取超出词汇表范围的标记的索引时希望您的词汇表返回的值。如果您没有设置此值，那么您将收到一个错误！这样，您就可以在尝试获取不在词汇表中的标记的索引时设置词汇表返回任何值，甚至是 `-100` 等。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "ename": "Exception",
     "evalue": "Please call \"set_default_index\" before getting unknown index",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_6282/2979932904.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0men_vocab\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"The\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/tmp/ipykernel_6282/1101150808.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, input_tokens)\u001b[0m\n\u001b[1;32m     37\u001b[0m             \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m  \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoken_to_idx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_tokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefault_index\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 39\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Please call \"set_default_index\" before getting unknown index'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     40\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     41\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtoken\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtoken\u001b[0m \u001b[0;32min\u001b[0m \u001b[0minput_tokens\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mException\u001b[0m: Please call \"set_default_index\" before getting unknown index"
     ]
    }
   ],
   "source": [
    "en_vocab[\"The\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们已经知道 `<unk>` 标记的索引为零，因为它是我们 `special_tokens` 列表的第一个元素，我们还使用 `get_itos` 进行了手动检查。\n",
    "\n",
    "然而，在这里，我们将以程序化的方式获取它，并检查我们的两个词汇表是否对未知和填充标记具有相同的索引，因为这将简化后面的一些代码。\n",
    "\n",
    "我们还获取 `<pad>` 标记的索引，因为我们稍后将使用它。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert en_vocab[unk_token] == de_vocab[unk_token]\n",
    "assert en_vocab[pad_token] == de_vocab[pad_token]\n",
    "\n",
    "unk_index = en_vocab[unk_token]\n",
    "pad_index = en_vocab[pad_token]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `set_default_index` 方法，我们可以设置在尝试获取词汇表范围之外的标记的索引时返回的值。在这种情况下，即未知标记 `<unk>` 的索引。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_vocab.set_default_index(unk_index)\n",
    "de_vocab.set_default_index(unk_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，我们可以愉快地获取超出词汇表范围的标记的索引，直到心满意足！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab[\"The\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们还可以获取与该索引对应的标记，以证明它是 `<unk>` 标记。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<unk>'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab.get_itos()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "词汇表的另一个有用的特性是 `lookup_indices` 方法。它接受一个标记列表，并返回一个索引列表。在下面的例子中，我们可以看到标记 \"crime\" 在我们的词汇表中不存在，因此被转换为 `<unk>` 标记的索引，即我们传递给 `set_default_index` 方法的: 零。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = [\"i\", \"love\", \"watching\", \"crime\", \"shows\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[951, 2217, 171, 0, 815]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab.lookup_indices(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相反，我们可以使用 `lookup_tokens` 方法使用词汇表将索引列表转换回标记列表。请注意，原始的 \"crime\" 标记现在变成了 `<unk>` 标记。无法确定原始标记序列是什么。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'love', 'watching', '<unk>', 'shows']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab.lookup_tokens(en_vocab.lookup_indices(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "希望我们现在已经掌握了 `Vocab` 类的工作原理。是时候将其付诸实践了！\n",
    "\n",
    "就像我们的 `tokenize_example` 一样，我们创建了一个 `numericalize_example` 函数，将在数据集的 `map` 方法中使用。这将使用词汇表“数字化”（一个说法是将标记转换为索引）每个示例中的标记，并将结果返回到新的 \"en_ids\" 和 \"de_ids\" 特征中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def numericalize_example(example, en_vocab:Vocab, de_vocab:Vocab):\n",
    "    en_ids = en_vocab.lookup_indices(example[\"en_tokens\"])\n",
    "    de_ids = de_vocab.lookup_indices(example[\"de_tokens\"])\n",
    "    return {\"en_ids\": en_ids, \"de_ids\": de_ids}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们应用 `numericalize_example` 函数，将我们的词汇表传递给 `fn_kwargs` 参数中的字典。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-632b0bad1bafed4c/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-b8add894629bd533.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-632b0bad1bafed4c/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-5321fd8a377bbba4.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/bentrevett___json/bentrevett--multi30k-632b0bad1bafed4c/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-e95764a174da5b82.arrow\n"
     ]
    }
   ],
   "source": [
    "fn_kwargs = {\n",
    "    \"en_vocab\": en_vocab, \n",
    "    \"de_vocab\": de_vocab\n",
    "}\n",
    "\n",
    "train_data = train_data.map(numericalize_example, fn_kwargs=fn_kwargs)\n",
    "valid_data = valid_data.map(numericalize_example, fn_kwargs=fn_kwargs)\n",
    "test_data = test_data.map(numericalize_example, fn_kwargs=fn_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "检查一个例子，我们可以看到它有两个新特征：“en_ids”和“de_ids”，都是整数列表，表示它们在相应词汇表中的索引。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'en': 'Two young, White males are outside near many bushes.',\n",
       " 'de': 'Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche.',\n",
       " 'en_tokens': ['<sos>',\n",
       "  'two',\n",
       "  'young',\n",
       "  ',',\n",
       "  'white',\n",
       "  'males',\n",
       "  'are',\n",
       "  'outside',\n",
       "  'near',\n",
       "  'many',\n",
       "  'bushes',\n",
       "  '.',\n",
       "  '<eos>'],\n",
       " 'de_tokens': ['<sos>',\n",
       "  'zwei',\n",
       "  'junge',\n",
       "  'weiße',\n",
       "  'männer',\n",
       "  'sind',\n",
       "  'im',\n",
       "  'freien',\n",
       "  'in',\n",
       "  'der',\n",
       "  'nähe',\n",
       "  'vieler',\n",
       "  'büsche',\n",
       "  '.',\n",
       "  '<eos>'],\n",
       " 'en_ids': [2, 16, 24, 15, 25, 774, 17, 57, 80, 202, 1305, 5, 3],\n",
       " 'de_ids': [2, 18, 26, 253, 30, 84, 20, 88, 7, 15, 110, 5374, 3099, 4, 3]}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以使用相应的词汇表在索引列表上使用 `lookup_tokens` 方法来确认索引是正确的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<sos>',\n",
       " 'two',\n",
       " 'young',\n",
       " ',',\n",
       " 'white',\n",
       " 'males',\n",
       " 'are',\n",
       " 'outside',\n",
       " 'near',\n",
       " 'many',\n",
       " 'bushes',\n",
       " '.',\n",
       " '<eos>']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vocab.lookup_tokens(train_data[0][\"en_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`datasets` 库使用 `Dataset` 类为我们处理的另一件事是将特征转换为正确的类型。每个示例中的索引当前是基本的 Python 整数。但是，为了在 mindspore 中使用它们，它们需要转换为 mindspore 张量。我们可以在将它们传递到模型之前将它们转换，但是现在这样做更加方便。\n",
    "\n",
    "`with_format` 方法将由 `columns` 参数指示的特征转换为给定的 `type`。在这里，我们将类型指定为 \"numpy\"（用于 mindspore），并将列指定为 \"en_ids\" 和 \"de_ids\"（我们要转换为 mindspore 张量的特征）。默认情况下，`with_format` 将删除未在传递给 `columns` 的特征列表中的特征。我们希望保留这些特征，可以使用 `output_all_columns=True`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_type = \"numpy\"\n",
    "format_columns = [\"en_ids\", \"de_ids\"]\n",
    "\n",
    "train_data = train_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True\n",
    ")\n",
    "\n",
    "valid_data = valid_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True,\n",
    ")\n",
    "\n",
    "test_data = test_data.with_format(\n",
    "    type=data_type, \n",
    "    columns=format_columns, \n",
    "    output_all_columns=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以通过检查一个示例并查看 \"en_ids\" 和 \"de_ids\" 特征是否被列为 `array([...])` 来确认这一点。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'en_ids': array([   2,   16,   24,   15,   25,  774,   17,   57,   80,  202, 1305,\n",
       "           5,    3]),\n",
       " 'de_ids': array([   2,   18,   26,  253,   30,   84,   20,   88,    7,   15,  110,\n",
       "        5374, 3099,    4,    3]),\n",
       " 'en': 'Two young, White males are outside near many bushes.',\n",
       " 'de': 'Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche.',\n",
       " 'en_tokens': ['<sos>',\n",
       "  'two',\n",
       "  'young',\n",
       "  ',',\n",
       "  'white',\n",
       "  'males',\n",
       "  'are',\n",
       "  'outside',\n",
       "  'near',\n",
       "  'many',\n",
       "  'bushes',\n",
       "  '.',\n",
       "  '<eos>'],\n",
       " 'de_tokens': ['<sos>',\n",
       "  'zwei',\n",
       "  'junge',\n",
       "  'weiße',\n",
       "  'männer',\n",
       "  'sind',\n",
       "  'im',\n",
       "  'freien',\n",
       "  'in',\n",
       "  'der',\n",
       "  'nähe',\n",
       "  'vieler',\n",
       "  'büsche',\n",
       "  '.',\n",
       "  '<eos>']}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(train_data[0][\"en_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据加载器（Data Loaders）\n",
    "\n",
    "准备数据的最后一步是创建数据加载器。这些加载器可以进行迭代，以返回一批数据，每个批次都是一个包含数值化的英语和德语句子的字典（它们也已被填充）作为 Mindspore 张量。\n",
    "\n",
    "首先，我们需要创建一个将一批示例合并成批次的函数，即 `collate_fn`。下面的 `collate_fn` 接受一个批次作为输入（一个示例列表），然后分离出批次中每个示例的英语和德语索引，并将每个索引传递给 `pad_sequence` 函数。`pad_sequence` 接受一个张量列表，使用 `padding_value`（我们设置为 `pad_index`，我们的 `<pad>` 标记的索引）将每个张量填充到最长张量的长度，然后返回一个形状为 `[max length, batch size]` 的张量，其中 `batch size` 是批次中的示例数，`max length` 是批次中最长张量的长度。我们将每个张量放入一个字典中，然后返回它。\n",
    "\n",
    "`get_collate_fn` 接受填充标记索引，并返回其内部定义的 `collate_fn`。这种在另一个函数内定义并返回的技术被称为 [闭包](https://en.wikipedia.org/wiki/Closure_(computer_programming))。它允许 `collate_fn` 在不创建类或使用全局变量的情况下继续使用它被创建时的 `pad_index` 值。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindspore import dtype as mstype\n",
    "def pad_sequence(sequences:list, padding_value:int):\n",
    "    '''将序列填充到等长并返回mindspore张量'''\n",
    "    # Find the length of the longest sequence in the batch\n",
    "    max_length = max(len(seq) for seq in sequences)\n",
    "    padded_sequences = ops.full((len(sequences), max_length), padding_value, dtype=mstype.int64)\n",
    "    # Copy the sequences into the padded array\n",
    "    for i, seq in enumerate(sequences):\n",
    "        padded_sequences[i, :len(seq)] = ms.tensor(seq).astype(np.int64)\n",
    "    # 换轴，保证输出为时序优先\n",
    "    padded_sequences = padded_sequences.swapaxes(0, 1)\n",
    "    return padded_sequences  \n",
    "\n",
    "\n",
    "def get_collate_fn(pad_index):\n",
    "    def collate_fn(batch):\n",
    "        batch_en_ids = [example[\"en_ids\"] for example in batch]\n",
    "        batch_de_ids = [example[\"de_ids\"] for example in batch]\n",
    "        batch_en_ids = pad_sequence(batch_en_ids, padding_value=pad_index)\n",
    "        batch_de_ids = pad_sequence(batch_de_ids, padding_value=pad_index)\n",
    "        batch = {\n",
    "            \"en_ids\": batch_en_ids,\n",
    "            \"de_ids\": batch_de_ids,\n",
    "        }\n",
    "        return batch\n",
    "    return collate_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们编写使用  `DataLoader` 类创建的数据加载器的函数。\n",
    "\n",
    "`get_data_loader` 是使用 `Dataset`、批次大小、填充标记索引（用于在 `collate_fn` 中创建批次）以及一个布尔值（决定在迭代数据加载器时是否应该对示例进行洗牌）创建的。\n",
    "\n",
    "批次大小定义了一个批次中的示例的最大数量。如果数据集的长度不能被批次大小整除，那么最后一个批次将更小。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataLoader:\n",
    "    def __init__(self, source, batch_size, shuffle=False, per_batch_map=None):\n",
    "        self.source = source\n",
    "        self.batch_size = batch_size\n",
    "        self.shuffle = shuffle\n",
    "        self.per_batch_map = per_batch_map\n",
    "        self.indices = np.arange(len(source))\n",
    "        self.current_index = 0\n",
    "\n",
    "        if self.shuffle:\n",
    "            np.random.shuffle(self.indices)\n",
    "\n",
    "    def __iter__(self):\n",
    "        return self\n",
    "\n",
    "    def __next__(self):\n",
    "        if self.current_index >= len(self.source):\n",
    "            self.current_index = 0\n",
    "            raise StopIteration\n",
    "\n",
    "        batch_indices = self.indices[self.current_index:self.current_index + self.batch_size]\n",
    "        batch_data = [self.source[int(i)] for i in batch_indices]\n",
    "\n",
    "        if self.per_batch_map:\n",
    "            batch_data = self.per_batch_map(batch_data)\n",
    "\n",
    "        self.current_index += self.batch_size\n",
    "        return batch_data\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.source) // self.batch_size\n",
    "\n",
    "\n",
    "def get_data_loader(dataset, batch_size, pad_index, shuffle=False):\n",
    "    collate_fn = get_collate_fn(pad_index)\n",
    "    dataloader = DataLoader(dataset, batch_size, shuffle=shuffle, per_batch_map=collate_fn)\n",
    "    \n",
    "    return dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们创建数据加载器。\n",
    "\n",
    "为了减少训练时间，我们通常希望使用尽可能大的批量大小。在使用GPU时，这意味着使用能够适应GPU内存的最大批量大小。\n",
    "\n",
    "数据的随机打乱可以使训练更加稳定，并可能提高模型的最终性能，但是只需要对训练集进行此操作。无论数据的顺序如何，对验证集和测试集计算出的指标都将是相同的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "\n",
    "train_data_loader = get_data_loader(train_data, batch_size, pad_index, shuffle=True)\n",
    "valid_data_loader = get_data_loader(valid_data, batch_size, pad_index)\n",
    "test_data_loader = get_data_loader(test_data, batch_size, pad_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 建立模型\n",
    "\n",
    "我们将模型分为三个部分构建：编码器、解码器和封装了编码器和解码器的seq2seq模型，并提供了与每个部分的接口方式。\n",
    "\n",
    "### 编码器\n",
    "\n",
    "首先，我们介绍编码器，这是一个2层LSTM。我们正在实现的论文使用了4层LSTM，但为了训练时间，我们将其减少到2层。多层RNN的概念很容易从2层扩展到4层。\n",
    "\n",
    "对于多层RNN，输入句子$X$经过嵌入后进入RNN的底层，隐藏状态$H=\\{h_1, h_2, ..., h_T\\}$由该层输出，并作为上层RNN的输入。因此，使用上标表示每一层，第一层的隐藏状态为：\n",
    "\n",
    "$$h_t^1 = \\text{EncoderRNN}^1(e(x_t), h_{t-1}^1)$$\n",
    "\n",
    "第二层的隐藏状态为：\n",
    "\n",
    "$$h_t^2 = \\text{EncoderRNN}^2(h_t^1, h_{t-1}^2)$$\n",
    "\n",
    "使用多层RNN还需要为每一层提供初始隐藏状态作为输入$h_0^l$，并且我们还将为每一层输出一个上下文向量$z^l$。\n",
    "\n",
    "对于LSTM，这里不详细展开（参见[这篇博客文章](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)以了解更多关于它们的信息），我们只需知道它们是一种类型的RNN，其每个时间步不仅接受一个隐藏状态并返回一个新的隐藏状态，还接受并返回一个*单元状态* $c_t$。\n",
    "\n",
    "$$\\begin{align*}\n",
    "h_t &= \\text{RNN}(e(x_t), h_{t-1})\\\\\n",
    "(h_t, c_t) &= \\text{LSTM}(e(x_t), h_{t-1}, c_{t-1})\n",
    "\\end{align*}$$\n",
    "\n",
    "我们可以将$c_t$视为另一种类型的隐藏状态。类似于$h_0^l$，$c_0^l$将初始化为一个全为零的张量。另外，我们的上下文向量现在既是最终的隐藏状态也是最终的单元状态，即$z^l = (h_T^l, c_T^l)$。\n",
    "\n",
    "将我们的多层方程扩展到LSTMs，我们得到：\n",
    "\n",
    "$$\\begin{align*}\n",
    "(h_t^1, c_t^1) &= \\text{EncoderLSTM}^1(e(x_t), (h_{t-1}^1, c_{t-1}^1))\\\\\n",
    "(h_t^2, c_t^2) &= \\text{EncoderLSTM}^2(h_t^1, (h_{t-1}^2, c_{t-1}^2))\n",
    "\\end{align*}$$\n",
    "\n",
    "请注意，只有来自第一层的隐藏状态作为输入传递给第二层，而不是单元状态。\n",
    "\n",
    "因此，我们的编码器看起来像这样：\n",
    "\n",
    "![](assets/seq2seq2.png)\n",
    "\n",
    "我们通过创建一个`Encoder`模块在代码中实现这一点，这要求我们从`mindspore.nn.Cell`继承并使用`super().__init__()`作为一些样板代码。编码器采用以下参数：\n",
    "\n",
    "- `input_dim`是输入到编码器的one-hot向量的尺寸/维度。这等于输入（源）词汇表的大小。\n",
    "- `embedding_dim`是嵌入层的维度。该层将one-hot向量转换为具有`embedding_dim`维度的密集向量。\n",
    "- `hidden_dim`是隐藏状态和单元状态的维度。\n",
    "- `n_layers`是RNN中的层数。\n",
    "- `dropout`是使用的dropout量。这是一个正则化参数，用于防止过拟合。更多关于dropout的详细信息，请查阅[这里](https://www.coursera.org/lecture/deep-neural-network/understanding-dropout-YaGbR)。\n",
    "\n",
    "在本次教程中，我们不会详细讨论嵌入层。我们需要知道的是，在单词（技术上是指单词的索引）被传递到RNN之前，有一个步骤将单词转换为向量。要了解更多关于词嵌入的信息，请查看以下文章：[1](https://monkeylearn.com/blog/word-embeddings-transform-text-numbers/), [2](http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html), [3](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/), [4](http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/).\n",
    "\n",
    "嵌入层使用`nn.Embedding`创建，LSTM使用`nn.LSTM`创建，dropout层使用`nn.Dropout`创建。要了解更多关于这些的信息，请查阅Mindspore的[文档](https://www.mindspore.cn/docs/zh-CN/master/api_python/nn/mindspore.nn.Dropout.html)。\n",
    "\n",
    "需要注意的是，LSTM的`dropout`参数是在多层RNN的层之间应用dropout的量，即在层$l$输出的隐藏状态与层$l+1$输入的相同隐藏状态之间。\n",
    "\n",
    "在`construct`方法中，我们传入源句子$X$，使用`embedding`层将其转换为密集向量，然后应用dropout。然后将这些嵌入传递到RNN中。由于我们将整个序列传递给RNN，它会自动对整个序列进行递归计算隐藏状态！请注意，我们没有将初始隐藏状态或单元状态传递给RNN。这是因为如文档中所述[文档](https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM)，如果未将隐藏/单元状态传递给RNN，它将自动创建一个初始隐藏/单元状态作为全零张量。\n",
    "\n",
    "RNN返回：`outputs`（每个时间步的顶层隐藏状态），`hidden`（每层的最终隐藏状态，$h_T$，堆叠在一起）和`cell`（每层的最终单元状态，$c_T$，堆叠在一起）。\n",
    "\n",
    "由于我们只需要最终的隐藏状态和单元状态（以制作我们的上下文向量），因此`construct`只返回`hidden`和`cell`。\n",
    "\n",
    "这些张量的尺寸留在代码中的注释中。在这个实现中，`n_directions`始终为1，但是请注意，双向RNN（在教程3中介绍）的`n_directions`为2。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Cell):\n",
    "    def __init__(self, input_dim, embedding_dim, hidden_dim, n_layers, dropout):\n",
    "        super().__init__()\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.n_layers = n_layers\n",
    "        self.embedding = nn.Embedding(input_dim, embedding_dim)\n",
    "        self.rnn = nn.LSTM(embedding_dim, hidden_dim, n_layers, dropout=dropout)\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "        \n",
    "    def construct(self, src):\n",
    "        # src = [src length, batch size]\n",
    "        embedded = self.dropout(self.embedding(src))\n",
    "        # embedded = [src length, batch size, embedding dim]\n",
    "        outputs, (hidden, cell) = self.rnn(embedded)\n",
    "        # outputs = [src length, batch size, hidden dim * n directions]\n",
    "        # hidden = [n layers * n directions, batch size, hidden dim]\n",
    "        # cell = [n layers * n directions, batch size, hidden dim]\n",
    "        # outputs are always from the top hidden layer\n",
    "        return hidden, cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 解码器\n",
    "\n",
    "接下来，我们将构建我们的解码器，它也将是一个2层（论文中是4层）LSTM。\n",
    "\n",
    "![](assets/seq2seq3.png)\n",
    "\n",
    "`Decoder` 类执行解码的单个步骤，即每个时间步输出单个token。第一层将从前一个时间步接收隐藏状态和细胞状态 $(s_{t-1}^1, c_{t-1}^1)$，并通过具有当前嵌入tokens $y_t$ 的 LSTM 进行处理，以生成新的隐藏状态和细胞状态 $(s_t^1, c_t^1)$。后续层将使用下层的隐藏状态 $s_t^{l-1}$ 和其层中的先前隐藏状态和细胞状态 $(s_{t-1}^l, c_{t-1}^l)$。这提供了与编码器中非常相似的方程。\n",
    "\n",
    "$$\\begin{align*}\n",
    "(s_t^1, c_t^1) = \\text{DecoderLSTM}^1(d(y_t), (s_{t-1}^1, c_{t-1}^1))\\\\\n",
    "(s_t^2, c_t^2) = \\text{DecoderLSTM}^2(s_t^1, (s_{t-1}^2, c_{t-1}^2))\n",
    "\\end{align*}$$\n",
    "\n",
    "请记住，解码器的初始隐藏状态和细胞状态是上下文向量，即编码器的相同层的最终隐藏状态和细胞状态，即 $(s_0^l,c_0^l)=z^l=(h_T^l,c_T^l)$。\n",
    "\n",
    "然后，我们通过线性层 $f$ 传递 RNN 的顶层隐藏状态 $s_t^L$，以预测目标（输出）序列中的下一个tokens $\\hat{y}_{t+1}$。\n",
    "\n",
    "$$\\hat{y}_{t+1} = f(s_t^L)$$\n",
    "\n",
    "参数和初始化与 `Encoder` 类相似，除了现在有一个 `output_dim`，它是输出/目标语言词汇表的大小。还有一个额外的 `Linear` 层，用于从顶层隐藏状态进行预测。\n",
    "\n",
    "在 `construct` 方法中，我们接受一批输入tokens、先前的隐藏状态和先前的细胞状态。由于我们一次只解码一个tokens，输入tokens的序列长度始终为1。我们使用 `unsqueeze` 添加一个句子长度维度为1。然后，与编码器类似，我们通过嵌入层并应用 dropout 传递。然后将这批嵌入tokens传递到具有先前隐藏状态和细胞状态的 RNN 中。这会产生一个 `output`（来自 RNN 顶层的隐藏状态）、一个新的 `hidden` 状态（每层一个，堆叠在一起）和一个新的 `cell` 状态（每层一个，堆叠在一起）。然后将 `output`（去除句子长度维度后）通过线性层传递以获得我们的 `prediction`。然后返回 `prediction`、新的 `hidden` 状态和新的 `cell` 状态。\n",
    "\n",
    "**注意**：由于我们始终具有序列长度为1，我们可以使用 `nn.LSTMCell` 而不是 `nn.LSTM`，因为它设计用于处理不一定在序列中的输入批次。`nn.LSTMCell` 只是一个单独的单元，而 `nn.LSTM` 是潜在多个单元的包装器。在这种情况下使用 `nn.LSTMCell` 将意味着我们不需要 `unsqueeze` 添加一个虚假的序列长度维度，但我们需要解码器中每层一个 `nn.LSTMCell`，并确保每个 `nn.LSTMCell` 从编码器接收到正确的初始隐藏状态。所有这些使得代码不太简洁 — 因此决定坚持使用常规的 `nn.LSTM`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Cell):\n",
    "    def __init__(self, output_dim, embedding_dim, hidden_dim, n_layers, dropout):\n",
    "        super().__init__()\n",
    "        self.output_dim = output_dim\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.n_layers = n_layers\n",
    "        self.embedding = nn.Embedding(output_dim, embedding_dim)\n",
    "        self.rnn = nn.LSTM(embedding_dim, hidden_dim, n_layers, dropout=dropout)\n",
    "        self.fc_out = nn.Dense(hidden_dim, output_dim)\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "        \n",
    "    def construct(self, input:ms.Tensor, hidden:ms.Tensor, cell:ms.Tensor):\n",
    "        # input = [batch size]\n",
    "        # hidden = [n layers * n directions, batch size, hidden dim]\n",
    "        # cell = [n layers * n directions, batch size, hidden dim]\n",
    "        # n directions in the decoder will both always be 1, therefore:\n",
    "        # hidden = [n layers, batch size, hidden dim]\n",
    "        # context = [n layers, batch size, hidden dim]\n",
    "        input = input.unsqueeze(0)\n",
    "        # input = [1, batch size]\n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        # embedded = [1, batch size, embedding dim]\n",
    "        output, (hidden, cell) = self.rnn(embedded, (hidden, cell))\n",
    "        # output = [seq length, batch size, hidden dim * n directions]\n",
    "        # hidden = [n layers * n directions, batch size, hidden dim]\n",
    "        # cell = [n layers * n directions, batch size, hidden dim]\n",
    "        # seq length and n directions will always be 1 in this decoder, therefore:\n",
    "        # output = [1, batch size, hidden dim]\n",
    "        # hidden = [n layers, batch size, hidden dim]\n",
    "        # cell = [n layers, batch size, hidden dim]\n",
    "        prediction = self.fc_out(output.squeeze(0))\n",
    "        # prediction = [batch size, output dim]\n",
    "        return prediction, hidden, cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seq2Seq\n",
    "\n",
    "对于实现的最后一部分，我们将实现 seq2seq 模型。这将处理以下任务：\n",
    "- 接收输入/源句子\n",
    "- 使用编码器生成上下文向量\n",
    "- 使用解码器生成预测的输出/目标句子\n",
    "\n",
    "我们的完整模型如下：\n",
    "\n",
    "![](assets/seq2seq4.png)\n",
    "\n",
    "`Seq2Seq` 模型接受 `Encoder`、`Decoder` 和 `device`（用于将张量放置在 GPU 上，如果存在的话）。\n",
    "\n",
    "在这个实现中，我们必须确保 `Encoder` 和 `Decoder` 中的层数和隐藏（和细胞）维度相等。这并不总是成立，我们在序列到序列模型中不一定需要相同数量的层或相同的隐藏维度大小。然而，如果我们做了一些像层数不同的事情，那么我们就需要做出关于如何处理这种情况的决策。例如，如果我们的编码器有 2 层，而我们的解码器只有 1 层，那么如何处理？我们是否对解码器输出的两个上下文向量进行平均？我们是否通过线性层传递两者？我们是否只使用来自最高层的上下文向量？等等。\n",
    "\n",
    "我们的 `forward` 方法接受源句子、目标句子和一个教师强制比率。教师强制比率在训练模型时使用。在解码时，在每个时间步，我们将从先前解码的token $\\hat{y}_{t+1}=f(s_t^L)$ 预测目标序列中的下一个token。以概率等于教师强制比率（`teacher_forcing_ratio`）的情况下，我们将使用实际的地面真实下一个token作为下一个时间步骤中解码器的输入。但以概率 `1 - teacher_forcing_ratio`，我们将使用模型预测的下一个token作为模型的下一个输入，即使它与实际下一个token不匹配。\n",
    "\n",
    "在 `forward` 方法中，首先创建一个将存储所有预测 $\\hat{Y}$ 的 `outputs` 张量。\n",
    "\n",
    "然后将输入/源句子 `src` 传递给编码器，并获得最终的隐藏和细胞状态。\n",
    "\n",
    "解码器的第一个输入是序列开始 (`<sos>`) token。由于我们的 `trg` 张量已经附加了 `<sos>` token（当我们对英语句子进行标记化时），我们通过切片来获取我们的 $y_1$。我们知道目标句子的长度应该是 `trg_length`，因此我们循环这么多次。解码器中的最后一个token输入是 `<eos>` token之前的token —— 永远不会将 `<eos>` token输入到解码器中。\n",
    "\n",
    "在循环的每次迭代中，我们：\n",
    "- 将输入、先前的隐藏状态和先前的细胞状态（$y_t, s_{t-1}, c_{t-1}$）传递到解码器\n",
    "- 从解码器接收预测、下一个隐藏状态和下一个细胞状态（$\\hat{y}_{t+1}, s_{t}, c_{t}$）\n",
    "- 将我们的预测 $\\hat{y}_{t+1}$/`output` 放入我们的预测的张量 $\\hat{Y}$/`outputs`\n",
    "- 决定是否进行“强制教学”\n",
    "    - 如果是，下一个 `input` 是序列中的下一个地面真实token，$y_{t+1}$/`trg[t]`\n",
    "    - 如果不是，下一个 `input` 是模型预测的序列中的下一个token，$\\hat{y}_{t+1}$/`top1`，通过在输出张量上进行 `argmax` 获得\n",
    "    \n",
    "完成所有预测后，我们返回充满预测 $\\hat{Y}$/`outputs` 的张量。\n",
    "\n",
    "**注意**：我们的解码器循环从 1 开始，而不是 0。这意味着我们的 `outputs` 张量的第 0 个元素仍然是全零。因此，我们的 `trg` 和 `outputs` 看起来像：\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{trg} = [<sos>, &y_1, y_2, y_3, <eos>]\\\\\n",
    "\\text{outputs} = [0, &\\hat{y}_1, \\hat{y}_2, \\hat{y}_3, <eos>]\n",
    "\\end{align*}$$\n",
    "\n",
    "稍后，当我们计算损失时，我们将截取每个张量的第一个元素，得到：\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{trg} = [&y_1, y_2, y_3, <eos>]\\\\\n",
    "\\text{outputs} = [&\\hat{y}_1, \\hat{y}_2, \\hat{y}_3, <eos>]\n",
    "\\end{align*}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Cell):\n",
    "    def __init__(self, encoder:Encoder, decoder:Decoder):\n",
    "        super().__init__()\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        assert encoder.hidden_dim == decoder.hidden_dim, \\\n",
    "            \"Hidden dimensions of encoder and decoder must be equal!\"\n",
    "        assert encoder.n_layers == decoder.n_layers, \\\n",
    "            \"Encoder and decoder must have equal number of layers!\"\n",
    "        \n",
    "    def construct(self, src:ms.Tensor, trg:ms.Tensor, teacher_forcing_ratio):\n",
    "        # src = [src length, batch size]\n",
    "        # trg = [trg length, batch size]\n",
    "        # teacher_forcing_ratio is probability to use teacher forcing\n",
    "        # e.g. if teacher_forcing_ratio is 0.75 we use ground-truth inputs 75% of the time\n",
    "        batch_size = trg.shape[1]\n",
    "        trg_length = trg.shape[0]\n",
    "        trg_vocab_size = self.decoder.output_dim\n",
    "        # tensor to store decoder outputs\n",
    "        # outputs = ops.zeros(trg_length, batch_size, trg_vocab_size)\n",
    "        outputs = []\n",
    "        # last hidden state of the encoder is used as the initial hidden state of the decoder\n",
    "        hidden, cell = self.encoder(src)\n",
    "        # hidden = [n layers * n directions, batch size, hidden dim]\n",
    "        # cell = [n layers * n directions, batch size, hidden dim]\n",
    "        # first input to the decoder is the <sos> tokens\n",
    "        input = trg[0,:]\n",
    "        # input = [batch size]\n",
    "        for t in range(1, trg_length):\n",
    "            # insert input token embedding, previous hidden and previous cell states\n",
    "            # receive output tensor (predictions) and new hidden and cell states\n",
    "            output, hidden, cell = self.decoder(input, hidden, cell)\n",
    "            if len(outputs) == 0:\n",
    "                outputs.append(ops.zeros(output.shape, dtype=output.dtype))\n",
    "            # output = [batch size, output dim]\n",
    "            # hidden = [n layers, batch size, hidden dim]\n",
    "            # cell = [n layers, batch size, hidden dim]\n",
    "            # place predictions in a tensor holding predictions for each token\n",
    "            outputs.append(output)\n",
    "            top1 = output.argmax(1)\n",
    "            # outputs[t] = output\n",
    "            # decide if we are going to use teacher forcing or not\n",
    "            if self.training:\n",
    "                teacher_force = random.random() < teacher_forcing_ratio\n",
    "                # get the highest predicted token from our predictions\n",
    "                \n",
    "                # if teacher forcing, use actual next token as next input\n",
    "                # if not, use predicted token\n",
    "                input = trg[t] if teacher_force else top1\n",
    "                # input = [batch size]\n",
    "            else:\n",
    "                input = top1\n",
    "        # return outputs\n",
    "        return ops.stack(outputs, axis=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型\n",
    "\n",
    "现在我们已经实现了我们的模型，我们可以开始训练它。\n",
    "\n",
    "### 模型初始化\n",
    "\n",
    "首先，我们将初始化我们的模型。如前所述，输入和输出维度由词汇表的大小定义。编码器和解码器的嵌入维度和 dropout 可以不同，但是层数和hidden/cell状态的大小必须相同。\n",
    "\n",
    "然后，我们定义编码器、解码器，然后是我们的 Seq2Seq 模型.\n",
    "```python\n",
    "# Model Initialization\n",
    "INPUT_DIM = len(en_vocab)\n",
    "OUTPUT_DIM = len(de_vocab)\n",
    "ENC_EMB_DIM = 256\n",
    "DEC_EMB_DIM = 256\n",
    "HID_DIM = 512\n",
    "N_LAYERS = 2\n",
    "ENC_DROPOUT = 0.5\n",
    "DEC_DROPOUT = 0.5\n",
    "\n",
    "encoder = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, N_LAYERS, ENC_DROPOUT)\n",
    "decoder = Decoder(OUTPUT_DIM, DEC_EMB_DIM, HID_DIM, N_LAYERS, DEC_DROPOUT)\n",
    "\n",
    "model = Seq2Seq(encoder, decoder, device)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim = len(de_vocab)\n",
    "output_dim = len(en_vocab)\n",
    "encoder_embedding_dim = 256\n",
    "decoder_embedding_dim = 256\n",
    "hidden_dim = 512\n",
    "n_layers = 2\n",
    "encoder_dropout = 0.5\n",
    "decoder_dropout = 0.5\n",
    "\n",
    "encoder = Encoder(\n",
    "    input_dim,\n",
    "    encoder_embedding_dim,\n",
    "    hidden_dim,\n",
    "    n_layers,\n",
    "    encoder_dropout,\n",
    ")\n",
    "\n",
    "decoder = Decoder(\n",
    "    output_dim,\n",
    "    decoder_embedding_dim,\n",
    "    hidden_dim,\n",
    "    n_layers,\n",
    "    decoder_dropout,\n",
    ")\n",
    "\n",
    "model = Seq2Seq(encoder, decoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 权重初始化\n",
    "\n",
    "接下来是初始化我们模型的权重。在论文中，他们声明将所有权重初始化为在 $-0.08$ 和 $+0.08$ 之间的均匀分布，即 $\\mathcal{U}(-0.08, 0.08)$。\n",
    "\n",
    "在 mindspore 中，我们通过创建一个应用于模型的函数来初始化权重。使用 `apply` 时，`init_weights` 函数将对我们模型中的每个模块和子模块进行调用。对于每个模块，我们循环遍历所有参数，并从均匀分布中对它们进行采样，使用 `nn.init.uniform_`。\n",
    "```python\n",
    "def init_weights(m):\n",
    "    for name, param in m.named_parameters():\n",
    "        nn.init.uniform_(param.data, -0.08, 0.08)\n",
    "\n",
    "model.apply(init_weights)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Seq2Seq<\n",
       "  (encoder): Encoder<\n",
       "    (embedding): Embedding<vocab_size=7853, embedding_size=256, use_one_hot=False, weight=Parameter (name=encoder.embedding.weight, shape=(7853, 256), dtype=Float32, requires_grad=True), dtype=Float32, padding_idx=None>\n",
       "    (rnn): LSTM<\n",
       "      (rnn): _DynamicLSTMCPUGPU<>\n",
       "      (dropout_op): Dropout<p=0.5>\n",
       "      >\n",
       "    (dropout): Dropout<p=0.5>\n",
       "    >\n",
       "  (decoder): Decoder<\n",
       "    (embedding): Embedding<vocab_size=5893, embedding_size=256, use_one_hot=False, weight=Parameter (name=decoder.embedding.weight, shape=(5893, 256), dtype=Float32, requires_grad=True), dtype=Float32, padding_idx=None>\n",
       "    (rnn): LSTM<\n",
       "      (rnn): _DynamicLSTMCPUGPU<>\n",
       "      (dropout_op): Dropout<p=0.5>\n",
       "      >\n",
       "    (fc_out): Dense<>\n",
       "    (dropout): Dropout<p=0.5>\n",
       "    >\n",
       "  >"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_weights(m:nn.Cell):\n",
    "    for name, param in m.parameters_and_names():\n",
    "        param.set_data(ops.uniform(param.shape, ms.Tensor([-0.08]), ms.Tensor([0.08]), dtype=param.dtype))\n",
    "\n",
    "model.apply(init_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 13,898,501 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model:nn.Cell):\n",
    "    return sum(p.numel() for p in model.get_parameters() if p.requires_grad)\n",
    "\n",
    "print(f\"The model has {count_parameters(model):,} trainable parameters\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 优化器(Optimizer)\n",
    "\n",
    "我们定义了我们的优化器，用于在训练循环中更新我们的参数。查看 [这篇文章](http://ruder.io/optimizing-gradient-descent/) 以获取有关不同优化器的信息。在这里，我们将使用 Adam。\n",
    "```python\n",
    "optimizer = nn.Adam(model.trainable_params())\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = nn.Adam(model.trainable_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 损失函数\n",
    "\n",
    "接下来，我们定义我们的损失函数。`CrossEntropyLoss` 函数同时计算我们预测的对数 softmax 和负对数似然。\n",
    "\n",
    "我们的损失函数计算每个标记的平均损失，但通过将 `<pad>` 标记的索引作为 `ignore_index` 参数传递，我们在目标标记是填充标记时忽略损失。\n",
    "```python\n",
    "criterion = nn.CrossEntropyLoss(ignore_index=pad_index)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss(ignore_index=pad_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练循环\n",
    "\n",
    "接下来，我们将定义我们的训练循环。\n",
    "\n",
    "\n",
    "正如前面所述，我们的解码器循环从 1 开始，而不是 0。这意味着 `outputs` 张量的第 0 个元素保持为全零。因此，我们的 `trg` 和 `outputs` 看起来像：\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{trg} = [<sos>, &y_1, y_2, y_3, <eos>]\\\\\n",
    "\\text{outputs} = [0, &\\hat{y}_1, \\hat{y}_2, \\hat{y}_3, <eos>]\n",
    "\\end{align*}$$\n",
    "\n",
    "在这里，当我们计算损失时，我们切掉了每个张量的第一个元素，得到：\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{trg} = [&y_1, y_2, y_3, <eos>]\\\\\n",
    "\\text{outputs} = [&\\hat{y}_1, \\hat{y}_2, \\hat{y}_3, <eos>]\n",
    "\\end{align*}$$\n",
    "\n",
    "每次迭代时：\n",
    "- 从批处理中获取源和目标句子，$X$ 和 $Y$\n",
    "- 将从上一批次计算得到的梯度清零\n",
    "- 将源和目标传递到模型以获得输出，$\\hat{Y}$\n",
    "- 由于损失函数仅适用于具有 1D 目标的 2D 输入，我们需要使用 `.view` 平铺它们中的每一个\n",
    "    - 我们切掉了上述第一列的输出和目标张量\n",
    "- 使用 `ms.value_and_grad` 计算梯度\n",
    "- 截断梯度以防止爆炸（在 RNN 中常见问题）\n",
    "- 通过执行优化器步骤更新模型的参数\n",
    "- 将损失值总和到运行总计中\n",
    "\n",
    "最后，我们返回在所有批次上平均的损失。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_fn(src, trg, teacher_forcing_ratio):\n",
    "    # src = [src length, batch size]\n",
    "    # trg = [trg length, batch size]\n",
    "    output = model(src, trg, teacher_forcing_ratio)\n",
    "    # output = [trg length, batch size, trg vocab size]\n",
    "    output_dim = output.shape[-1]\n",
    "    output = output[1:].view(-1, output_dim)\n",
    "    # output = [(trg length - 1) * batch size, trg vocab size]\n",
    "    trg = trg[1:].view(-1)\n",
    "    # trg = [(trg length - 1) * batch size]\n",
    "    loss = criterion(output, trg.astype(ms.int32))\n",
    "    return loss\n",
    "\n",
    "grad_fn = ms.value_and_grad(forward_fn, grad_position=None, weights=model.trainable_params())\n",
    "\n",
    "\n",
    "def train_fn(data_loader, optimizer, clip, teacher_forcing_ratio):\n",
    "    epoch_loss = 0.\n",
    "    model.set_train(True)\n",
    "    for i, batch in enumerate(tqdm(data_loader)):\n",
    "        src = batch[\"de_ids\"]\n",
    "        trg = batch[\"en_ids\"]\n",
    "        loss, grads = grad_fn(src, trg, teacher_forcing_ratio)\n",
    "        # grads = ops.clip_by_norm(grads, max_norm=clip)\n",
    "        optimizer(grads)\n",
    "        epoch_loss += float(loss)\n",
    "\n",
    "    return epoch_loss / len(data_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 评估循环\n",
    "\n",
    "我们的评估循环与训练循环相似，但由于我们不会更新任何参数，因此无需传递优化器或截断值。\n",
    "\n",
    "迭代循环类似（不包括参数更新），但我们必须确保在评估时关闭 teacher forcing。这将导致模型仅使用自己的预测来进行句子内的进一步预测，这反映了在部署中使用它的方式。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_fn(model:Seq2Seq, data_loader, criterion):\n",
    "    epoch_loss = 0\n",
    "    model.set_train(False)\n",
    "    for i, batch in enumerate(data_loader):\n",
    "        src = batch[\"de_ids\"]\n",
    "        trg = batch[\"en_ids\"]\n",
    "        # src = [src length, batch size]\n",
    "        # trg = [trg length, batch size]\n",
    "        output = model(src, trg, 0) # turn off teacher forcing\n",
    "        # output = [trg length, batch size, trg vocab size]\n",
    "        output_dim = output.shape[-1]\n",
    "        output = output[1:].view(-1, output_dim)\n",
    "        # output = [(trg length - 1) * batch size, trg vocab size]\n",
    "        trg = trg[1:].view(-1)\n",
    "        # trg = [(trg length - 1) * batch size]\n",
    "        loss = criterion(output, trg.astype(mstype.int32))\n",
    "        epoch_loss += loss.item()\n",
    "    return epoch_loss / len(data_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型训练\n",
    "\n",
    "我们终于可以开始训练我们的模型了！\n",
    "\n",
    "在每个 epoch 中，我们将检查我们的模型是否达到迄今为止的最佳验证损失。如果是这样，我们将更新我们的最佳验证损失并保存我们模型的参数。然后，在测试我们的模型时，我们将使用用于达到最佳验证损失的保存参数。\n",
    "\n",
    "我们将在每个 epoch 中打印出损失和困惑度。困惑度的变化比损失更容易看到，因为这些数字要大得多。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c495e907accc4c058fc129fa23a16aea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7afbf2850c243028393d6428839579c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   4.394 | Train PPL:  81.002\n",
      "\tValid Loss:   5.577 | Valid PPL: 264.345\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6fa01875d0044c64aeb2dafd9949a4bb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   4.064 | Train PPL:  58.218\n",
      "\tValid Loss:   5.136 | Valid PPL: 170.074\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01e4fcda6e254b43b600cfec91468794",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.833 | Train PPL:  46.208\n",
      "\tValid Loss:   4.933 | Valid PPL: 138.865\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37662d062b2843efadd65bc58637bb52",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.627 | Train PPL:  37.607\n",
      "\tValid Loss:   4.864 | Valid PPL: 129.548\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79dc28ed3c3049bdbc5e2c4c084b66e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.439 | Train PPL:  31.165\n",
      "\tValid Loss:   4.641 | Valid PPL: 103.615\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "30c6e36e4d774084bc3d6bc7fc7330dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.270 | Train PPL:  26.303\n",
      "\tValid Loss:   4.570 | Valid PPL:  96.588\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "350b9d3a9f1541239452bd2d584d34c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   3.083 | Train PPL:  21.826\n",
      "\tValid Loss:   4.528 | Valid PPL:  92.619\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d74566aef27d4d1c90c674eec9074aad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.947 | Train PPL:  19.043\n",
      "\tValid Loss:   4.425 | Valid PPL:  83.480\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8440ce7003014ecfab6b806678652e4d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.791 | Train PPL:  16.293\n",
      "\tValid Loss:   4.432 | Valid PPL:  84.134\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d9f968e57cb4fbdaf823e7b859b243b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/226 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Loss:   2.682 | Train PPL:  14.614\n",
      "\tValid Loss:   4.294 | Valid PPL:  73.252\n"
     ]
    }
   ],
   "source": [
    "context.set_context(mode=context.PYNATIVE_MODE, device_target=\"GPU\")\n",
    "n_epochs = 10\n",
    "clip = 1.0\n",
    "teacher_forcing_ratio = 0.5\n",
    "\n",
    "best_valid_loss = float(\"inf\")\n",
    "\n",
    "for epoch in tqdm(range(n_epochs)):\n",
    "\n",
    "    train_loss = train_fn(\n",
    "        train_data_loader, \n",
    "        optimizer, \n",
    "        clip, \n",
    "        teacher_forcing_ratio, \n",
    "    )\n",
    "    \n",
    "    valid_loss = evaluate_fn(\n",
    "        model, \n",
    "        valid_data_loader, \n",
    "        criterion, \n",
    "    )\n",
    "\n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        ms.save_checkpoint(model, f\"./tut1-model.ckpt\")\n",
    "    \n",
    "    print(f\"\\tTrain Loss: {train_loss:7.3f} | Train PPL: {np.exp(train_loss):7.3f}\")\n",
    "    print(f\"\\tValid Loss: {valid_loss:7.3f} | Valid PPL: {np.exp(valid_loss):7.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们现在已经成功训练了一个将德语翻译成英语的模型！但是它的表现如何呢？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先要做的是在测试集上测试模型的性能。\n",
    "\n",
    "我们将加载提供给模型最佳验证损失的参数（`state_dict`），并在测试集上运行它以获取我们的测试损失和困惑度。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Test Loss: 4.290 | Test PPL:  72.986 |\n"
     ]
    }
   ],
   "source": [
    "ms.load_checkpoint(f\"tut1-model.ckpt\", model)\n",
    "\n",
    "test_loss = evaluate_fn(model, test_data_loader, criterion)\n",
    "\n",
    "print(f\"| Test Loss: {test_loss:.3f} | Test PPL: {np.exp(test_loss):7.3f} |\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型性能评估\n",
    "\n",
    "性能与验证性能相当，这是个好迹象。这意味着我们没有在验证集上过拟合。\n",
    "\n",
    "你可能认为在验证集上不可能过拟合，但事实并非如此。每次你调整超参数（例如优化器、学习率、模型架构、权重初始化等），以在验证集上获得更好的结果时，你都在逐渐将这些超参数过拟合到验证集上。你也可能在测试集上进行这样的过拟合！因此，你应该尽可能少地在测试集上评估模型。\n",
    "\n",
    "大多数使用神经网络进行翻译的论文不会提供关于测试集损失和困惑度的结果，它们通常会提供 [BLEU](https://en.wikipedia.org/wiki/BLEU) 分数。与损失/困惑度不同，BLEU 是一个介于零和一之间的值，数值越高越好，并且根据 [BLEU 论文](https://aclanthology.org/P02-1040.pdf) 的说法，它与人类判断有很高的相关性。\n",
    "\n",
    "为了获取我们模型在测试集上的 BLEU 分数，我们首先需要使用模型翻译测试集的每个例子，我们使用下面的 `translate_sentence` 函数来实现这一点。该函数首先将输入的 `sentence` 转换为 `tokens`，可选择将每个 `token` 转换为小写，然后附加起始和结束序列的标记，即 `sos_token` 和 `eos_token`。然后，它使用词汇表将这些 `tokens` 转换为 `ids`，并将这些 `ids` 转换为 `tensor`，添加一个“虚拟”批次维度，然后通过 `encoder` 将 `tensor` 传递以获得 `hidden` 和 `cell` 状态。接下来，我们执行解码，从 `sos_token` 开始，将其转换为 `tensor`，通过 `decoder` 传递，获取我们模型认为在序列中最有可能是下一个的 `predicted_token`，然后将其附加到传递给解码器的 `inputs` 列表中。如果 `predicted_token` 是序列结束的标记，则停止解码；如果不是，则继续循环，使用 `predicted_token` 作为传递给解码器的下一个输入。我们一直解码，直到解码器输出 `eos_token` 或达到 `max_output_length`（我们使用它来避免解码器无限生成标记）。一旦停止解码，我们使用词汇表将输入转换为 `tokens` 并将它们返回。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate_sentence(\n",
    "    sentence, \n",
    "    model:Seq2Seq,\n",
    "    en_nlp:spacy.Language,\n",
    "    de_nlp:spacy.Language,\n",
    "    en_vocab:Vocab,\n",
    "    de_vocab:Vocab,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    "    max_output_length=25,\n",
    "):\n",
    "    if isinstance(sentence, str):\n",
    "        tokens = [token.text for token in de_nlp.tokenizer(sentence)]\n",
    "    else:\n",
    "        tokens = [token for token in sentence]\n",
    "    if lower:\n",
    "        tokens = [token.lower() for token in tokens]\n",
    "    tokens = [sos_token] + tokens + [eos_token]\n",
    "    ids = de_vocab.lookup_indices(tokens)\n",
    "\n",
    "    tensor = ms.Tensor(ids).astype(ms.int64).unsqueeze(-1)\n",
    "    hidden, cell = model.encoder(tensor)\n",
    "    inputs = en_vocab.lookup_indices([sos_token])\n",
    "    for _ in range(max_output_length):\n",
    "        inputs_tensor = ms.Tensor([inputs[-1]]).astype(ms.int64)\n",
    "        output, hidden, cell = model.decoder(inputs_tensor, hidden, cell)\n",
    "        predicted_token = output.argmax(-1).item()\n",
    "        inputs.append(predicted_token)\n",
    "        if predicted_token == en_vocab[eos_token]:\n",
    "            break\n",
    "    tokens = en_vocab.lookup_tokens(inputs)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将传递一个测试示例（模型未经过训练的内容）作为一个要测试的句子，将德语句子传递给我们的 `translate_sentence` 函数，并期望得到类似英语句子的输出。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Ein Mann mit einem orangefarbenen Hut, der etwas anstarrt.',\n",
       " 'A man in an orange hat starring at something.')"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = test_data[0][\"de\"]\n",
    "expected_translation = test_data[0][\"en\"]\n",
    "\n",
    "sentence, expected_translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "translation = translate_sentence(\n",
    "    sentence,\n",
    "    model,\n",
    "    en_nlp,\n",
    "    de_nlp,\n",
    "    en_vocab,\n",
    "    de_vocab,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们的模型似乎已经弄清楚了输入句子提到了一个穿着某种衣物的男人（尽管颜色和物品都猜错了），但它似乎无法弄清楚这个男人在做什么。\n",
    "\n",
    "我们不应该期望令人惊讶的结果，我们的模型相对于我们正在实现的论文中使用的模型来说是相对较小的（他们使用了四层，嵌入和隐藏维度为1000），并且与现代翻译模型相比（具有数十亿的参数）几乎微不足道。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<sos>',\n",
       " 'a',\n",
       " 'man',\n",
       " 'in',\n",
       " 'a',\n",
       " 'orange',\n",
       " 'hat',\n",
       " 'is',\n",
       " 'looking',\n",
       " 'at',\n",
       " 'something',\n",
       " '.',\n",
       " '<eos>']"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型不仅仅可以翻译训练、验证和测试集中的示例。我们可以通过将任何字符串传递给 `translate_sentence` 来使用它来翻译任意句子。\n",
    "\n",
    "请注意，multi30k数据集由已从英语翻译成德语的图像标题组成，而我们的模型已经训练成将德语翻译成英语。因此，只有当句子是可能是图像标题的德语句子时，模型才会输出合理的翻译。（还要强调的是，此处训练的模型相对较小，翻译性能通常较差。）\n",
    "\n",
    "在下面，我们输入了**\"A man is watching a film.\"**的德语翻译。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"Ein Mann sieht sich einen Film an.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "translation = translate_sentence(\n",
    "    sentence,\n",
    "    model,\n",
    "    en_nlp,\n",
    "    de_nlp,\n",
    "    en_vocab,\n",
    "    de_vocab,\n",
    "    lower,\n",
    "    sos_token,\n",
    "    eos_token,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "并且我们收到了我们的翻译..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<sos>', 'a', 'man', 'is', 'a', 'a', '.', '.', '<eos>']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们现在可以循环遍历我们的 `test_data`，得到模型对每个测试句子的翻译。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2d08f1338e14b799ca7a29f56a2f19d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "translations = [\n",
    "    translate_sentence(\n",
    "        example[\"de\"],\n",
    "        model,\n",
    "        en_nlp,\n",
    "        de_nlp,\n",
    "        en_vocab,\n",
    "        de_vocab,\n",
    "        lower,\n",
    "        sos_token,\n",
    "        eos_token,\n",
    "    ) for example in tqdm(test_data)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了计算BLEU，我们将使用`evaluate`库。建议使用度量指标专用的库，以确保在度量计算中没有错误，从而得到潜在的不正确的结果。\n",
    "\n",
    "可以像这样从`evaluate`库中加载BLEU指标：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7a2395212524e3aaf751d50117310bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script: 0.00B [00:00, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "18d32bccb3b844ffae0616dcda8b336a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading extra modules:   0%|          | 0.00/1.55k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d74a6152e404fde9826a588ea331eed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading extra modules: 0.00B [00:00, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bleu = evaluate.load(\"bleu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于BLEU指标的一个怪癖是它期望预测（预测的翻译）是字符串，而参考（实际的英语句子）是句子列表。这是因为BLEU在每个预测中有多个正确句子的情况下起作用，因为可能有多种方法来翻译一句话。在我们的情况下，我们只有一个参考句子，所以我们只需将目标句子放入列表中。我们还将我们的翻译从token列表转换为字符串，方法是在它们之间用空格连接，并摆脱`<sos>`和`<eos>`tocken。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = [\n",
    "    \" \".join(translation[1:-1]) for translation in translations\n",
    "]\n",
    "\n",
    "references = [\n",
    "    [example[\"en\"]] for example in test_data\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('a man in a orange hat is looking at something .',\n",
       " ['A man in an orange hat starring at something.'])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions[0], references[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们还需要定义一个函数，该函数对输入字符串进行tocken化。这将用于通过将我们的预测tocken与参考tocken进行比较来计算BLEU分数。\n",
    "\n",
    "我们将翻译的tocken连接成字符串，然后再次对其进行tocken化，似乎有些奇怪，而且还使用了我们测试数据中的英语字符串而不是现有的tocken（en_tokens），但这是由“evaluate”库提供的BLEU指标的另一个怪癖；“predictions”和“references”必须是字符串，而不是tocken，并且我们必须告诉指标如何对这些字符串进行tocken化。\n",
    "\n",
    "“get_tokenize_fn”返回我们的“tokenizer_fn”，该函数使用我们的“spaCy”分词器，并在必要时将tocken小写。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tokenizer_fn(nlp, lower):\n",
    "    \n",
    "    def tokenizer_fn(s):\n",
    "        tokens = [token.text for token in nlp.tokenizer(s)]\n",
    "        if lower:\n",
    "            tokens = [token.lower() for token in tokens]\n",
    "        return tokens\n",
    "        \n",
    "    return tokenizer_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer_fn = get_tokenizer_fn(en_nlp, lower)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['a',\n",
       "  'man',\n",
       "  'in',\n",
       "  'a',\n",
       "  'orange',\n",
       "  'hat',\n",
       "  'is',\n",
       "  'looking',\n",
       "  'at',\n",
       "  'something',\n",
       "  '.'],\n",
       " ['a', 'man', 'in', 'an', 'orange', 'hat', 'starring', 'at', 'something', '.'])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_fn(predictions[0]), tokenizer_fn(references[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们在测试集上计算BLEU指标！\n",
    "\n",
    "我们将我们的“predictions”、“references”和我们的“tokenizer_fn”传递给BLEU指标的“compute”方法，以获取我们的结果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = bleu.compute(predictions=predictions, references=references, tokenizer=tokenizer_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bleu': 0.16086491322307994,\n",
       " 'precisions': [0.5102489593968429,\n",
       "  0.22134151538395977,\n",
       "  0.1111525202646045,\n",
       "  0.05907736566320764],\n",
       " 'brevity_penalty': 0.9747987608819583,\n",
       " 'length_ratio': 0.9751110430387502,\n",
       " 'translation_length': 12733,\n",
       " 'reference_length': 13058}"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们获得了0.160的BLEU分数！虽然不是很出色，但对于我们的第一个翻译模型来说还不错。\n",
    "\n",
    "在接下来的笔记本中，我们将实现更多的翻译论文，并逐渐提高BLEU分数。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}