transformers/一课transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">一课Transformers</h1> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第一节 介绍与准备工作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 transformers简介及本教程目的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "随着NLP领域预训练模型的盛行，从BERT、GPT到T5、ELECTRA、Longformer、MobileBERT等越来越多的模型涌现了出来。每个模型的作者可能用tf，也可能是pytorch，而且很可能不同的环境版本，这对于学术界、工业界的学习、复现、使用都带来了一定困难。幸好，huggingface公司下的transformers库帮我们解决了这个难题。\n",
    "\n",
    "huggingface是一家公司，transformers[1][2][3]是其公司开发的开源库，已有30k+个star。该库表现为：简单易用；同时支持tf2和pytorch；支持很多预训练模型如BERT, GPT, ALBERT, T5, DialoGPT, ELECTRA等，而且随时维护；提供统一的、标准的Config、Model、Tokenizer、Trainer接口，同时提供标准化的模型方式，方便复现、拓展和实验。\n",
    "\n",
    "目前一些开源库使用huggingface/transformers为基础进行开发，提供一些分类、生成任务，如基于transformers库开发的中文文本生成[4]；基于transformers库进行的科研，如DialoGPT[5]。\n",
    "\n",
    "两种方式去使用，第一种是pipeline方式，高度集成，直接使用，可以用于情感分类、NER标注等；第二种是提供标注的模型，去训练，更符合标准的使用，更能使用我们日常学习、工作中的任务。\n",
    "\n",
    "本篇讲解第二种使用方式，以pytorch版的模型使用为例，希望能通过一节课的时间帮助大家入门transformers库的使用。\n",
    "\n",
    "注：本代码参考和使用了大量的官方样例，该文的所有代码已经通过测试。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 本文结构\n",
    "\n",
    "#### 第一节 介绍与准备工作\n",
    "\n",
    "本节对huggingface/transformers是什么进行了说明，说明本文的目的，章节结构，准备工作等内容。\n",
    "\n",
    "#### 第二节 快速入门\n",
    "\n",
    "一个快速入门的例子，tokenizer、model加载与保存的概念，讲解如何利用tokenizer将文本转换成模型的输入，通过model得到logits与loss等结果。通过这么简单地几步，这是标准的pytorch一个batch的损失计算过程。基本上实现了一个完整的周期迭代。\n",
    "\n",
    "#### 第三节 概念与说明\n",
    "\n",
    "介绍transformers的服务人群、目标、Configuration、Tokenizer、Model、AutoModels、Trainer这几个类的设计目的，从高层次上理解该库的设计思路。\n",
    "\n",
    "#### 第四节 GLUE/MRPC数据集进行文本分类的示例\n",
    "\n",
    "基于上面的知识，进行完整的实战代码演示。这是一个完整的文本分类fine-tune与线上部署的代码，共分为三个部分：\n",
    "\n",
    "第一部分，讲解如何加载数据、模型、tokenizer等，去初始化Trainer，并训练。\n",
    "\n",
    "第二部分，加载第一部分得到的模型结果，进行预测结果。\n",
    "\n",
    "#### 第五节 GPT2训练使用示例\n",
    "\n",
    "类似于第四节。讲解从头预训练GPT2的语言模型，并且使用beam search、top k、top p解码进行文本生成。\n",
    "\n",
    "#### 第六节 总结、思考与展望\n",
    "\n",
    "进行全文总结。\n",
    "\n",
    "#### 第七节 相关网址"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 准备工作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.1 下载预训练模型和数据集\n",
    "\n",
    "下面的讲解使用到bert-base-uncased、gpt2两个预训练模型；同时会使用到glue下的MRPC数据集、wikitext-2-raw数据集。可以直接通过下面的百度云链接和密码去下载。可以将下载好的数据放在与本notebook同级目录下，方便使用。\n",
    "\n",
    "链接：https://pan.baidu.com/s/1w65feQWdUHCn8Bg4ATTIEw \n",
    "\n",
    "提取码：svoz\n",
    "\n",
    "#### 1.3.2 安装相应的环境\n",
    "\n",
    "本教程以pytorch为基础，需要安装pytorch，建议pytorch>=1.4.0。\n",
    "\n",
    "transformers升级至最新版本，升级方式：\n",
    "\n",
    "```\n",
    "pip install --upgrade transformers\n",
    "```\n",
    "\n",
    "可能会需要其他环境，如pandas、xlrd、sklearn等，提示缺少什么包，直接安装即可。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第二节 快速入门\n",
    "\n",
    "通过简单的例子，感受从文本到模型输出的快捷。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m W&B installed but not logged in.  Run `wandb login` or set the WANDB_API_KEY env variable.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch: 1.4.0+cu100\n",
      "transformers: 3.0.2\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import transformers\n",
    "\n",
    "print(f'torch: {torch.__version__}')\n",
    "print(f'transformers: {transformers.__version__}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 所有大写的内容，需要改为自己的实际路径\n",
    "BERT_MODEL_NAME_OR_PATH = 'transformers_data_and_model/bert-base-uncased'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transformers_data_and_model/bert-base-uncased\n",
      "├── config.json\n",
      "├── modelcard.json\n",
      "├── pytorch_model.bin\n",
      "└── vocab.txt\n",
      "\n",
      "0 directories, 4 files\n"
     ]
    }
   ],
   "source": [
    "# 展示文件夹中的内容\n",
    "!tree {BERT_MODEL_NAME_OR_PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at transformers_data_and_model/bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at transformers_data_and_model/bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "# 初始化model和tokenizer\n",
    "#　所有model和tokenizer的初始化都使用from_pretrained的方法，保存都使用save_pretrained的方法\n",
    "# 对from_pretrained:第一个参数是文件夹的路径/文件的路径/模型的short name等几种方法，这里推荐使用文件夹的方式\n",
    "# model初始化默认是eval模式，这里加载的是BERT的tokenizer和分类模型model\n",
    "model = transformers.AutoModelForSequenceClassification.from_pretrained(BERT_MODEL_NAME_OR_PATH)\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(BERT_MODEL_NAME_OR_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [101, 2057, 2024, 2200, 3407, 2000, 2265, 2017, 1996, 100, 19081, 3075, 1012, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}\n"
     ]
    }
   ],
   "source": [
    "# tokenizer的作用：对于给定的文本，经过tokenizer处理成model可以接受的格式\n",
    "# tokenizer最重要的用法是__call__，这个方法可以将文本输出为模型的要的格式\n",
    "# tokenizer还有其他方法，encode/decode，顾名思义，就是将文本转换成input_ids及将input_ids转换为文本\n",
    "# encode/decode与__call__其实无本质区别，只是__call__为了提供统一的处理接口\n",
    "inputs = tokenizer(\"We are very happy to show you the 🤗 Transformers library.\")\n",
    "# input_ids是文本每个词的index；token_type_id是表示文本是第一句/第二句； attention_mask是处理mask用的\n",
    "# 这里处理的是一条样本且只有一句话的例子，如果是多条单句样本，输入为一个文本list即可。\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [101, 7592, 102, 2204, 2851, 102], 'token_type_ids': [0, 0, 0, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1]}\n"
     ]
    }
   ],
   "source": [
    "# 一条样本且有两句话的例子，分别作为第一个和第二个参数输入\n",
    "# 如果是多个样本，每个样本都是两句话，则第一个参数是第一句话的文本list，第二个参数为第二句话的文本list\n",
    "inputs2 = tokenizer('hello', 'good morning')\n",
    "print(inputs2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_ids: [[101, 2057, 2024, 2200, 3407, 2000, 2265, 2017, 1996, 100, 19081, 3075, 1012, 102], [101, 2057, 3246, 2017, 2123, 1005, 1056, 5223, 2009, 1012, 102, 0, 0, 0]]\n",
      "token_type_ids: [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]\n",
      "attention_mask: [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0]]\n"
     ]
    }
   ],
   "source": [
    "# 更多时候，我们需要的model输入是成batch格式的\n",
    "# 第一个输入是文本list，padding设置为True，truncation设置为True可以进行padding和truncation\n",
    "# return_tensors写明了返回的格式，是一个pytorch的tensor\n",
    "pt_batch = tokenizer(\n",
    "     [\"We are very happy to show you the 🤗 Transformers library.\", \"We hope you don't hate it.\"],\n",
    "     padding=True,\n",
    "     truncation=True,\n",
    "     return_tensors=\"pt\"\n",
    ")\n",
    "for key, value in pt_batch.items():\n",
    "     print(f\"{key}: {value.numpy().tolist()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logits:tensor([[-3.5918e-01,  1.7649e-02],\n",
      "        [-3.3124e-01, -1.6445e-04]], grad_fn=<AddmmBackward>)\n",
      "\n",
      "loss: 0.6973519325256348\n",
      "logits: tensor([[-3.5918e-01,  1.7649e-02],\n",
      "        [-3.3124e-01, -1.6445e-04]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "# 对于tokenizer处理后的文本，目标是送入model，用于分类、预测等任务\n",
    "# 上面的pt_batch就是一个batch，增加**直接输入模型\n",
    "# 根据transformers库的规则， 所有model的输出都是元组\n",
    "# 如果只有每个batch的输入，元组输出的第一个是logits\n",
    "# 如果同时传入了labels的参数，则元组输出的第一个是loss，第二个是logits\n",
    "# 一般回归任务loss用的Mean-Square loss，分类任务则是Cross-Entroy\n",
    "pt_outputs = model(**pt_batch)\n",
    "print(f'logits:{pt_outputs[0]}\\n')\n",
    "# 传入labels参数的情况\n",
    "pt_outputs = model(**pt_batch, labels=torch.LongTensor([1, 0]))\n",
    "print(f'loss: {pt_outputs[0]}\\nlogits: {pt_outputs[1]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面就是pytorch版transformers的基本输入逻辑：\n",
    "\n",
    "transformers中的所有model都是pytorch的标准模型类torch.nn.Module。文本可以通过tokenizer调用转换为模型的输入，模型输入这些信息，得到logits，计算loss，进行误差回传backward，进行迭代，就完成了训练/fine-tune。模型输入的时候，如果传入labels的参数，也可以直接得到相应的loss，一样backward，多次迭代，完成训练。\n",
    "\n",
    "除了标准的pytorch方式，transformers还封装了Trainer类来帮助我们简化pytorch代码，后面再讲。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 完成训练/微调后，可以将tokenizer和model保存至相同的文件夹。\n",
    "# 在transformers框架里，一个很好的习惯，将model、tokenizer参数、训练参数等所有存放在同一文件夹。\n",
    "# 这里的model/tokenizer/config的初始化使用from_pretrained，保存使用save_pretrained\n",
    "# save_pretrained传入具体的文件夹名即可\n",
    "SAVE_DIRECTORY = 'transformers_data_and_model/bert_save_example'\n",
    "tokenizer.save_pretrained(SAVE_DIRECTORY)\n",
    "model.save_pretrained(SAVE_DIRECTORY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transformers_data_and_model/bert_save_example\n",
      "├── config.json\n",
      "├── pytorch_model.bin\n",
      "├── special_tokens_map.json\n",
      "├── tokenizer_config.json\n",
      "└── vocab.txt\n",
      "\n",
      "0 directories, 5 files\n"
     ]
    }
   ],
   "source": [
    "# 保存的结果展示\n",
    "!tree {SAVE_DIRECTORY}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 总结：\n",
    "\n",
    "1. 在transformers框架里，提供了model/tokenizer/config通用化的加载和保存，也就是from_pretraiend/save_pretrained；\n",
    "2. tokenizer的作用在于，通过\\_\\_call__将文本进行转换成模型接受的格式，model的输出都是元组，依据这些元组的内容进行计算loss；\n",
    "3. tokenizer包装了Byte-Pair Encoding、WordPiece、SentencePiece等不同的方式；\n",
    "4. model是包装了BERT、GPT2、ALBERT等不同的模型，并且提供标准化的类。等下细说。\n",
    "5. 对于我们来讲，去复现、实验、研究更容易。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第三节 概念与说明"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本节讲解transformers涉及的服务群体、目标、主要概念、AutoModels、Trainer类等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 transformers的服务人群：\n",
    "- 寻找用于使用、学习、扩展大型transformers模型的NLP研究者和教育者\n",
    "- 希望对模型进行微调或在生产生提供服务的实践者\n",
    "- 只想下载预训练模型，并将其用于解决NLP任务的工程师"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 transformers的两个强目标：\n",
    "- 尽可能的简单，快速的使用\n",
    "- 为最新模型提供与原始模型尽可能接近的性能"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 transformers的小目标:\n",
    "- 尽可能的内部接口一致\n",
    "- 纳入主观选择的有前途的工具，以对这些模型进行微调/研究\n",
    "- 在PyTorch和TensorFlow 2.0之间轻松切换，从而允许使用一种框架进行训练，而使用另一种框架进行推理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 主要概念：\n",
    "\n",
    "主要的类有Model、Configuration、Tokenizer这三个类，下面分别介绍。\n",
    "\n",
    "Model类，比如BertModel，均从pytorch models(torch.nn.Module)或者keras models(tf.keras.Model)继承而来，用于处理预训练权重。\n",
    "\n",
    "Configuration类，比如BertConfig，里面保存着建立模型所需要的所有参数。并不是总是我们手动去初始化这个类，尤其是当你使用没有做任何更改的预训练时，model会自动处理好这个类。也就是说，如果自己重新预训练的模型且架构不一致时，是许我们我们去初始化这个类的。\n",
    "\n",
    "Tokenizer类，比如BERTTokenizer，为每个模型保存词典，并将文本进行编码/解码成模型需要的格式——token嵌入的索引。\n",
    "\n",
    "上面的类，都有下面两个方法去实例化类和保存至本地：\n",
    "\n",
    "`from_pretrained()` 允许我们加载预训练模型，可以使用short_name，也可以使用本地的模型，作为第一个参数model_name_or_path传入，可以是文件夹、文件、short_name等。其中文件夹的话，会默认寻找文件夹中的pytorh_model.bin。\n",
    "\n",
    "`save_pretrained()` 允许我们将模型保存至本地，保存的参数是可以是文件夹。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 AutoModels\n",
    "\n",
    "以BERT为例，每个模型都有一个Config（BertConfig）；有1-2个tokenizer，分别是基于rust的快速tokenizer（BertTokenizerFast），一个是基于python原版的tokenizer（BertTokenizer），部分没有提供rust的快速tokenizer；有多个皆有不同head的Model，比如最原始的模型，不含head的BertModel、预训练MLM和NSP的BertForPreTraining、MLM head的BertForMaskedLM，NSP的BertForNextSentencePrediction，用于句子分类的BertForSequenceClassification，用于多选的BertForMultipleChoice，用单词分类的BertForTokenClassification，用于问答的BertForQuestionAnswering。\n",
    "\n",
    "不同的模型，会稍有不同。但是config类都继承自PretrainedConfig；tokenizer都继承自PreTrainedTokenizer或PreTrainedTokenizerFast；model都继承自PreTrainedModel。\n",
    "\n",
    "为了使用方便，AutoConfig、AutoTokenizer、AutoModel、AutoModelForPreTraining、AutoModelWithLMHead、AutoModelForSequenceClassification、AutoModelForQuestionAnswering、AutoModelForTokenClassification等可以用于自动查找模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6 Trainer类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trainer类提供了一个完整的标准训练的API，目前支持语言模型、文本分类、单词分类（NER）等任务。对于前面的config、tokenizer、model，我们可以认为，帮助我们简化的是写模型的这一步，正常生成dataset、dataloader，然后再每个epoch、batch进行训练，得到最终的结果。正常写的话，Trainer类可以不用，Trainer其实是简化的是我们训练的这一步。\n",
    "\n",
    "对于通常的训练过程，写法大致是这样的（下面是伪代码）:\n",
    "\n",
    "```\n",
    "# 加载数据\n",
    "train_data, test_data = get_data()\n",
    "# 转换成features，获得dataset\n",
    "train_dataset = MyDataset(train_data, args)\n",
    "test_dataset = MyDataset(test_data, args)\n",
    "\n",
    "# 转换成dataloader，用于生成batch\n",
    "train_sampler, test_sampler = \n",
    "train_dataloader = Dataloader(train_dataset, sampler=train_sampler, batch_size=batch_size, collate_fn=collate_fn)\n",
    "test_dataloader = Dataloader(train_dataset, sampler=test_sampler, batch_size=batch_size, collate_fn=collate_fn)\n",
    "# 初始化tensorboard\n",
    "tb_writer = SummaryWriter(log_dir=None)\n",
    "# 加载optimizer\n",
    "optimizer = \n",
    "mode.to(GPU)\n",
    "# 开始训练\n",
    "for epoch in range(epochs):\n",
    "    for batch in train_dataloader:\n",
    "        # 切换为train\n",
    "        model.train()\n",
    "        # 传入ＧＰＵ\n",
    "        batch.to(GPU)\n",
    "        # 计算每一步的loss，然后回传\n",
    "        tr_loss = train_step(model, inputs, optimizer)\n",
    "        tr_loss.backword()\n",
    "        model.zero_grad()\n",
    "for batch in test_dataloder:\n",
    "    ***\n",
    "model.save_pretrained(OUTPUT_PATH)\n",
    "```\n",
    "\n",
    "使用了Trainer之后，就能大大的简化训练过程：\n",
    "\n",
    "```\n",
    "# 加载数据\n",
    "train_data, test_data = get_data()\n",
    "# 转换成features，获得dataset\n",
    "train_dataset = MyDataset(train_data, args)\n",
    "test_dataset = MyDataset(test_data, args)\n",
    "# 读入train_args\n",
    "train_args = **\n",
    "\n",
    "\n",
    "#初始化本Trainer\n",
    "trainer = transformers.Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=train_dataset,\n",
    "    eval_dataset=eval_dataset,\n",
    "    compute_metrics=build_compute_metrics_fn(data_args.task_name),\n",
    ")\n",
    "# 训练\n",
    "if training_args.do_train:\n",
    "    trainer.train(\n",
    "        model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None\n",
    "    )\n",
    "    # trainer保存模型，里面调用的还是save_pretrained的方法\n",
    "    trainer.save_model()\n",
    "    # 保存tokenizer至同一个文件夹，方便使用\n",
    "    if trainer.is_world_master():\n",
    "        tokenizer.save_pretrained(training_args.output_dir)\n",
    "```\n",
    "\n",
    "从上面的代码，可以看出，Trainer简化的过程就是训练的过程。对于传统的训练中常见的过程进行了封装，我们只要去初始化Trainer这个类，就很方便的去训练，保存模型使用是save_model，内部调用的还是上文提到的save_pretrained的方法。下面的章节也会用到Trainer类进行训练，更详细的大家可以看Trainer的API说明[8]及源代码。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第四节 GLUE/MRPC数据集进行文本分类的示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 本节说明\n",
    "\n",
    "本节课以简单的例子，说明文本分类的fine-tune详细过程，线上部署代码。\n",
    "\n",
    "4.2节详细实现了文本分类任务的详细代码，4.3节实现了基于4.2节训练好的文本分类模型的线上预测代码，4.4节总结了文本分类的基本经验和transformers使用的基本经验。\n",
    "\n",
    "本节涉及到的模型时bert-base-uncased，涉及到的数据集是glue数据集下的MRPC数据集。\n",
    "\n",
    "glue数据集，共有9个任务，其中STS-B是一个回归任务，MNLI是三分类任务，剩余7类均是二分类任务。更详细的glue数据集的信息，可以参考[6]。九个任务之一的MRPC(The Microsoft Research Paraphrase Corpus，微软研究院释义语料库)，相似性和释义任务，是从在线新闻源中自动抽取句子对语料库，并人工注释句子对中的句子是否在语义上等效。类别并不平衡，其中68%的正样本，所以遵循常规的做法，报告准确率（accuracy）和F1值。样本个数：训练集3, 668个，开发集408个，测试集1, 725个。任务：是否释义二分类，是释义，不是释义两类。评价准则：准确率（accuracy）和F1值。标签为1（正样本，互为释义）的样例（每个样例是两句话，中间用tab隔开）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 模型训练的详细过程 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "transformers提供了config、tokenizer、model等类简化了分词、模型等步骤，同时又有Trainer类简化了训练过程。那么更详细的训练过程是什么呢？本节主要的内容就是实现和讲解模型分类的详细过程。\n",
    "\n",
    "简单的讲，主要分为几个步骤，加载参数，文本处理成Dataset。写collate_fn，用于处理padding。加载config、tokenizer、model等。写metrics，用于评估效果。将这些参数都送入Trainer去初始化此类，然后调用Trainer的train方法去训练。\n",
    "\n",
    "首先要加载参数，这些参数分为三类，一类是model相关参数，用于记录模型的位置等信息，用于初始化模型；一类是数据参数，数据的位置，任务名称等用于提供模型输入前的参数；一类是训练参数，这些是模型在训练过程中的参数，比如learning_rate、epochs等。\n",
    "\n",
    "文本最开始需要载入，可以通过写明一个Processor类，这个类用于提供几个方法：获得训练样本、获得开发样本、获得标签list等。这里的获得的样本是一个list，每个元素都是一个example，example里包含文本和对应的标签（有的不含，比如测试集）。这个processor提供的方法主要是为了Dataset类使用，Dataset实现单个输入的样本。\n",
    "\n",
    "collate_fn是Dataloader类的输入，对于处理好的恒定长度的feature，可以不输入collate_fn，使用默认的collate_fn，对于长度不定涉及到padding的文本，需要自己写此参数。\n",
    "\n",
    "config、tokenizer、model的加载我们已经基本熟悉。\n",
    "\n",
    "metrics的写法可以参考下面的方法。\n",
    "\n",
    "然后将这些参数送入Trainer，就可以训练和评估了。\n",
    "\n",
    "本过程不仅是为了实现MRPC分类，更重要的是，它可以是文本分类的一个标准化流程，也是pytorch使用的标准化流程，可以方便以后按照此思路进行扩展。以上的说法相对比较抽象，但是实际操作的过程中，还是非常明了的，所以请看下面的代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import sys\n",
    "import enum\n",
    "import time\n",
    "from dataclasses import dataclass, field\n",
    "from typing import Callable, Dict, Optional, List, Union, NamedTuple\n",
    "\n",
    "import filelock\n",
    "import torch\n",
    "import numpy as np\n",
    "import transformers\n",
    "from scipy.stats import pearsonr, spearmanr\n",
    "from sklearn.metrics import matthews_corrcoef, f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 下载好的预训练模型位置\n",
    "BERT_MODEL_NAME_OR_PATH = 'transformers_data_and_model/bert-base-uncased'\n",
    "# 下载好的glue数据集中的MRPC数据集的位置\n",
    "MRPC_DATA_DIR = 'transformers_data_and_model/MRPC'\n",
    "# finetune好的model、tokenizer等各种参数存放的位置\n",
    "FINETUNED_MRPC = 'transformers_data_and_model/finetuned-mrpc'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 日志文件\n",
    "logger = logging.getLogger(__name__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义模型参数，包含model、config、tokenizer、cache_dir等\n",
    "# 有三类参数：\n",
    "#   一个是模型参数，这个就是下面的定义；\n",
    "#   一个是模型训练参数， 可以参考transformers/src/train_args.py文件，主要是epoch、batch_size等常见的训练参数，也包含device这种设备参数\n",
    "#   一个数据参数，决定数据处理任务的参数，使用什么数据，数据名称，是否覆盖数据的cache，最长长度，这个长度是用于生成features使用的\n",
    "# 下面这个是模型参数\n",
    "@dataclass\n",
    "class ModelArguments:\n",
    "    \"\"\"\n",
    "    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.\n",
    "    \"\"\"\n",
    "\n",
    "    model_name_or_path: str = field(\n",
    "        metadata={\"help\": \"Path to pretrained model or model identifier from huggingface.co/models\"}\n",
    "    )\n",
    "    config_name: Optional[str] = field(\n",
    "        default=None, metadata={\"help\": \"Pretrained config name or path if not the same as model_name\"}\n",
    "    )\n",
    "    tokenizer_name: Optional[str] = field(\n",
    "        default=None, metadata={\"help\": \"Pretrained tokenizer name or path if not the same as model_name\"}\n",
    "    )\n",
    "    cache_dir: Optional[str] = field(\n",
    "        default=None, metadata={\"help\": \"Where do you want to store the pretrained models downloaded from s3\"}\n",
    "    )\n",
    "        \n",
    "\n",
    "# 数据（训练使用的）参数\n",
    "# 这些参数用于送入Dataset、processor等，用于方便加载数据，完成从开始到模型输入前的参数\n",
    "@dataclass\n",
    "class GlueDataTrainingArguments:\n",
    "    \"\"\"\n",
    "    Arguments pertaining to what data we are going to input our model for training and eval.\n",
    "\n",
    "    Using `HfArgumentParser` we can turn this class\n",
    "    into argparse arguments to be able to specify them on\n",
    "    the command line.\n",
    "    \"\"\"\n",
    "\n",
    "    task_name: str = field(metadata={\"help\": \"The name of the task to train on: MRPC\"})\n",
    "    data_dir: str = field(\n",
    "        metadata={\"help\": \"The input data dir. Should contain the .tsv files (or other data files) for the task.\"}\n",
    "    )\n",
    "    max_seq_length: int = field(\n",
    "        default=128,\n",
    "        metadata={\n",
    "            \"help\": \"The maximum total input sequence length after tokenization. Sequences longer \"\n",
    "            \"than this will be truncated, sequences shorter will be padded.\"\n",
    "        },\n",
    "    )\n",
    "    overwrite_cache: bool = field(\n",
    "        default=False, metadata={\"help\": \"Overwrite the cached training and evaluation sets\"}\n",
    "    )\n",
    "\n",
    "    def __post_init__(self):\n",
    "        self.task_name = self.task_name.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_args = ['--model_name_or_path', BERT_MODEL_NAME_OR_PATH,\n",
    "             '--task_name', 'MRPC',\n",
    "             '--do_train',\n",
    "             '--do_eval',\n",
    "             '--data_dir', MRPC_DATA_DIR,\n",
    "             '--max_seq_length', '128',\n",
    "             '--per_device_train_batch_size', '32',\n",
    "             '--learning_rate', '3e-5',\n",
    "             '--num_train_epochs', '3.0',\n",
    "             '--output_dir', FINETUNED_MRPC,\n",
    "             '--overwrite_cache',\n",
    "             '--overwrite_output_dir']\n",
    "\n",
    "# 以后新的实验，需要实现上面的模型参数和数据参数，基本上照上面格式去写，使用下面的方法转换成参数空间即可\n",
    "# transformers里，有一个HfArguementParser用于解析上面格式的参数，为标准的python参数\n",
    "parser = transformers.HfArgumentParser((ModelArguments, transformers.GlueDataTrainingArguments, transformers.TrainingArguments))\n",
    "# 将三类参数分别解析为对应的空间\n",
    "# 模型本身的参数，数据的参数，训练的参数\n",
    "model_args, data_args, training_args = parser.parse_args_into_dataclasses(input_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ModelArguments(model_name_or_path='transformers_data_and_model/bert-base-uncased', config_name=None, tokenizer_name=None, cache_dir=None)\n",
      "\n",
      "GlueDataTrainingArguments(task_name='mrpc', data_dir='transformers_data_and_model/MRPC', max_seq_length=128, overwrite_cache=True)\n",
      "\n",
      "TrainingArguments(output_dir='transformers_data_and_model/finetuned-mrpc', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=False, per_device_train_batch_size=32, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=3e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Jul27_19-04-01_d1f9a32623b6', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1)\n"
     ]
    }
   ],
   "source": [
    "# 展示一下所有的参数\n",
    "print(f'{model_args}\\n\\n{data_args}\\n\\n{training_args}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 确保output_dir可以用\n",
    "if (\n",
    "    os.path.exists(training_args.output_dir)\n",
    "    and os.listdir(training_args.output_dir)\n",
    "    and training_args.do_train\n",
    "    and not training_args.overwrite_output_dir\n",
    "):\n",
    "    raise ValueError(\n",
    "        f\"Output directory ({training_args.output_dir}) already exists and is not empty. Use --overwrite_output_dir to overcome.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:04:01 - INFO - transformers.training_args -   PyTorch: setting up devices\n",
      "07/27/2020 19:04:01 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 2, distributed training: False, 16-bits training: False\n",
      "07/27/2020 19:04:01 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='transformers_data_and_model/finetuned-mrpc', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=False, per_device_train_batch_size=32, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=3e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Jul27_19-04-01_d1f9a32623b6', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1)\n"
     ]
    }
   ],
   "source": [
    "# 设定日志格式，记录一些关键参数，并且将训练参数打印出来\n",
    "# 一个很重要的感受：使用logger打印中间变量很重要，对于transformers库，还是我们学习、工作中，都是如此\n",
    "logging.basicConfig(\n",
    "    format=\"%(asctime)s - %(levelname)s - %(name)s -   %(message)s\",\n",
    "    datefmt=\"%m/%d/%Y %H:%M:%S\",\n",
    "    level=logging.INFO if training_args.local_rank in [-1, 0] else logging.WARN,\n",
    ")\n",
    "logger.warning(\n",
    "    \"Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s\",\n",
    "    training_args.local_rank,\n",
    "    training_args.device,\n",
    "    training_args.n_gpu,\n",
    "    bool(training_args.local_rank != -1),\n",
    "    training_args.fp16,\n",
    ")\n",
    "logger.info(\"Training/evaluation parameters %s\", training_args)\n",
    "\n",
    "# 设定种子\n",
    "transformers.set_seed(training_args.seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获得标签的个数\n",
    "# 输出的模式，这里是classification与regression两种\n",
    "# 如果适配新任务，我们要的不是去按照这种格式，而是要得到这两个参数\n",
    "num_labels = 2\n",
    "output_mode = 'classification'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:04:01 - INFO - transformers.configuration_utils -   loading configuration file transformers_data_and_model/bert-base-uncased/config.json\n",
      "07/27/2020 19:04:01 - INFO - transformers.configuration_utils -   Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"finetuning_task\": \"mrpc\",\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "07/27/2020 19:04:01 - INFO - transformers.configuration_utils -   loading configuration file transformers_data_and_model/bert-base-uncased/config.json\n",
      "07/27/2020 19:04:01 - INFO - transformers.configuration_utils -   Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   Model name 'transformers_data_and_model/bert-base-uncased' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). Assuming 'transformers_data_and_model/bert-base-uncased' is a path, a model identifier, or url to a directory containing tokenizer files.\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   Didn't find file transformers_data_and_model/bert-base-uncased/added_tokens.json. We won't load it.\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   Didn't find file transformers_data_and_model/bert-base-uncased/special_tokens_map.json. We won't load it.\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   Didn't find file transformers_data_and_model/bert-base-uncased/tokenizer_config.json. We won't load it.\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   Didn't find file transformers_data_and_model/bert-base-uncased/tokenizer.json. We won't load it.\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   loading file transformers_data_and_model/bert-base-uncased/vocab.txt\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:04:01 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:04:01 - INFO - transformers.modeling_utils -   loading weights file transformers_data_and_model/bert-base-uncased/pytorch_model.bin\n",
      "07/27/2020 19:04:04 - WARNING - transformers.modeling_utils -   Some weights of the model checkpoint at transformers_data_and_model/bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "07/27/2020 19:04:04 - WARNING - transformers.modeling_utils -   Some weights of BertForSequenceClassification were not initialized from the model checkpoint at transformers_data_and_model/bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "# 加载config、tokenizer、model这三个。\n",
    "# config是包含层数、dropout参数、head个数、finetune任务等模型相关内容的参数，这个参数加载后只是为了model使用。\n",
    "# config内写入标签的个数num_labels，决定model后面分类使用的全连接的输出的个数\n",
    "config = transformers.AutoConfig.from_pretrained(\n",
    "    model_args.config_name if model_args.config_name else model_args.model_name_or_path,\n",
    "    num_labels=num_labels,\n",
    "    finetuning_task=data_args.task_name,\n",
    "    cache_dir=model_args.cache_dir,\n",
    ")\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(\n",
    "    model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,\n",
    "    cache_dir=model_args.cache_dir,\n",
    ")\n",
    "model = transformers.AutoModelForSequenceClassification.from_pretrained(\n",
    "    model_args.model_name_or_path,\n",
    "    from_tf=bool(\".ckpt\" in model_args.model_name_or_path),\n",
    "    config=config,\n",
    "    cache_dir=model_args.cache_dir,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transformers.DataProcessor是一个基类，需要实现get_train_examples,get_dev_examples, get_test_examples, get_labels等几个函数，\n",
    "# 分别用于提供的InputExample的集和（list）和标签的集和\n",
    "class MrpcProcessor(transformers.DataProcessor):\n",
    "    \"\"\"Processor for the MRPC data set (GLUE version).\"\"\"\n",
    "\n",
    "    def get_example_from_tensor_dict(self, tensor_dict):\n",
    "        \"\"\"See base class.\"\"\"\n",
    "        return InputExample(\n",
    "            tensor_dict[\"idx\"].numpy(),\n",
    "            tensor_dict[\"sentence1\"].numpy().decode(\"utf-8\"),\n",
    "            tensor_dict[\"sentence2\"].numpy().decode(\"utf-8\"),\n",
    "            str(tensor_dict[\"label\"].numpy()),\n",
    "        )\n",
    "\n",
    "    def get_train_examples(self, data_dir):\n",
    "        \"\"\"See base class.\"\"\"\n",
    "        logger.info(\"LOOKING AT {}\".format(os.path.join(data_dir, \"train.tsv\")))\n",
    "        return self._create_examples(self._read_tsv(os.path.join(data_dir, \"train.tsv\")), \"train\")\n",
    "\n",
    "    def get_dev_examples(self, data_dir):\n",
    "        \"\"\"See base class.\"\"\"\n",
    "        return self._create_examples(self._read_tsv(os.path.join(data_dir, \"dev.tsv\")), \"dev\")\n",
    "\n",
    "    def get_test_examples(self, data_dir):\n",
    "        \"\"\"See base class.\"\"\"\n",
    "        return self._create_examples(self._read_tsv(os.path.join(data_dir, \"test.tsv\")), \"test\")\n",
    "\n",
    "    def get_labels(self):\n",
    "        \"\"\"See base class.\"\"\"\n",
    "        return [\"0\", \"1\"]\n",
    "\n",
    "    def _create_examples(self, lines, set_type):\n",
    "        \"\"\"Creates examples for the training, dev and test sets.\"\"\"\n",
    "        examples = []\n",
    "        for (i, line) in enumerate(lines):\n",
    "            if i == 0:\n",
    "                continue\n",
    "            guid = \"%s-%s\" % (set_type, i)\n",
    "            text_a = line[3]\n",
    "            text_b = line[4]\n",
    "            label = None if set_type == \"test\" else line[0]\n",
    "            examples.append(transformers.InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))\n",
    "        return examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将examples转换成features\n",
    "def glue_convert_examples_to_features(\n",
    "    examples: List[transformers.InputExample],\n",
    "    tokenizer: transformers.PreTrainedTokenizer,\n",
    "    max_length: Optional[int] = None,\n",
    "    task=None,\n",
    "    label_list=None,\n",
    "    output_mode=None,\n",
    "):\n",
    "    \"\"\"\n",
    "    Loads a data file into a list of ``InputFeatures``\n",
    "\n",
    "    Args:\n",
    "        examples: List of ``InputExamples`` or ``tf.data.Dataset`` containing the examples.\n",
    "        tokenizer: Instance of a tokenizer that will tokenize the examples\n",
    "        max_length: Maximum example length. Defaults to the tokenizer's max_len\n",
    "        task: GLUE task\n",
    "        label_list: List of labels. Can be obtained from the processor using the ``processor.get_labels()`` method\n",
    "        output_mode: String indicating the output mode. Either ``regression`` or ``classification``\n",
    "\n",
    "    Returns:\n",
    "        If the ``examples`` input is a ``tf.data.Dataset``, will return a ``tf.data.Dataset``\n",
    "        containing the task-specific features. If the input is a list of ``InputExamples``, will return\n",
    "        a list of task-specific ``InputFeatures`` which can be fed to the model.\n",
    "\n",
    "    \"\"\"\n",
    "    if max_length is None:\n",
    "        max_length = tokenizer.max_len\n",
    "\n",
    "    if task is not None:\n",
    "        processor = MrpcProcessor()\n",
    "        if label_list is None:\n",
    "            label_list = processor.get_labels()\n",
    "            logger.info(\"Using label list %s for task %s\" % (label_list, task))\n",
    "        if output_mode is None:\n",
    "            output_mode = glue_output_modes[task]\n",
    "            logger.info(\"Using output mode %s for task %s\" % (output_mode, task))\n",
    "\n",
    "    label_map = {label: i for i, label in enumerate(label_list)}\n",
    "\n",
    "    def label_from_example(example: transformers.InputExample) -> Union[int, float, None]:\n",
    "        if example.label is None:\n",
    "            return None\n",
    "        if output_mode == \"classification\":\n",
    "            return label_map[example.label]\n",
    "        elif output_mode == \"regression\":\n",
    "            return float(example.label)\n",
    "        raise KeyError(output_mode)\n",
    "\n",
    "    labels = [label_from_example(example) for example in examples]\n",
    "\n",
    "    batch_encoding = tokenizer(\n",
    "        [(example.text_a, example.text_b) for example in examples],\n",
    "        max_length=max_length,\n",
    "        padding=\"max_length\",\n",
    "        truncation=True,\n",
    "    )\n",
    "\n",
    "    features = []\n",
    "    for i in range(len(examples)):\n",
    "        inputs = {k: batch_encoding[k][i] for k in batch_encoding}\n",
    "\n",
    "        feature = transformers.InputFeatures(**inputs, label=labels[i])\n",
    "        features.append(feature)\n",
    "\n",
    "    for i, example in enumerate(examples[:5]):\n",
    "        logger.info(\"*** Example ***\")\n",
    "        logger.info(\"guid: %s\" % (example.guid))\n",
    "        logger.info(\"features: %s\" % features[i])\n",
    "\n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Split(enum.Enum):\n",
    "    train = \"train\"\n",
    "    dev = \"dev\"\n",
    "    test = \"test\"\n",
    "    \n",
    "    \n",
    "class GlueDataset(torch.utils.data.dataset.Dataset):\n",
    "    \"\"\"\n",
    "    This will be superseded by a framework-agnostic approach\n",
    "    soon.\n",
    "    \"\"\"\n",
    "\n",
    "    args: GlueDataTrainingArguments\n",
    "    output_mode: str\n",
    "    features: List[transformers.InputFeatures]\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        args: transformers.GlueDataTrainingArguments,\n",
    "        tokenizer: transformers.PreTrainedTokenizer,\n",
    "        limit_length: Optional[int] = None,\n",
    "        mode: Union[str, Split] = Split.train,\n",
    "        cache_dir: Optional[str] = None,\n",
    "        output_mode = \"classification\",\n",
    "    ):\n",
    "        self.args = args\n",
    "        self.processor = MrpcProcessor()\n",
    "        self.output_mode = output_mode\n",
    "        if isinstance(mode, str):\n",
    "            try:\n",
    "                mode = Split[mode]\n",
    "            except KeyError:\n",
    "                raise KeyError(\"mode is not a valid split name\")\n",
    "        # Load data features from cache or dataset file\n",
    "        cached_features_file = os.path.join(\n",
    "            cache_dir if cache_dir is not None else args.data_dir,\n",
    "            \"cached_{}_{}_{}_{}\".format(\n",
    "                mode.value, tokenizer.__class__.__name__, str(args.max_seq_length), args.task_name,\n",
    "            ),\n",
    "        )\n",
    "        label_list = self.processor.get_labels()\n",
    "        self.label_list = label_list\n",
    "\n",
    "        # Make sure only the first process in distributed training processes the dataset,\n",
    "        # and the others will use the cache.\n",
    "        lock_path = cached_features_file + \".lock\"\n",
    "        with filelock.FileLock(lock_path):\n",
    "\n",
    "            if os.path.exists(cached_features_file) and not args.overwrite_cache:\n",
    "                start = time.time()\n",
    "                self.features = torch.load(cached_features_file)\n",
    "                logger.info(\n",
    "                    f\"Loading features from cached file {cached_features_file} [took %.3f s]\", time.time() - start\n",
    "                )\n",
    "            else:\n",
    "                logger.info(f\"Creating features from dataset file at {args.data_dir}\")\n",
    "\n",
    "                if mode == Split.dev:\n",
    "                    examples = self.processor.get_dev_examples(args.data_dir)\n",
    "                elif mode == Split.test:\n",
    "                    examples = self.processor.get_test_examples(args.data_dir)\n",
    "                else:\n",
    "                    examples = self.processor.get_train_examples(args.data_dir)\n",
    "                if limit_length is not None:\n",
    "                    examples = examples[:limit_length]\n",
    "                self.features = glue_convert_examples_to_features(\n",
    "                    examples,\n",
    "                    tokenizer,\n",
    "                    max_length=args.max_seq_length,\n",
    "                    label_list=label_list,\n",
    "                    output_mode=self.output_mode,\n",
    "                )\n",
    "                start = time.time()\n",
    "                torch.save(self.features, cached_features_file)\n",
    "                # ^ This seems to take a lot of time so I want to investigate why and how we can improve.\n",
    "                logger.info(\n",
    "                    \"Saving features into cached file %s [took %.3f s]\", cached_features_file, time.time() - start\n",
    "                )\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.features)\n",
    "\n",
    "    def __getitem__(self, i) -> transformers.InputFeatures:\n",
    "        return self.features[i]\n",
    "\n",
    "    def get_labels(self):\n",
    "        return self.label_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:04:04 - INFO - filelock -   Lock 139748473285376 acquired on transformers_data_and_model/MRPC/cached_train_BertTokenizer_128_mrpc.lock\n",
      "07/27/2020 19:04:04 - INFO - __main__ -   Creating features from dataset file at transformers_data_and_model/MRPC\n",
      "07/27/2020 19:04:04 - INFO - __main__ -   LOOKING AT transformers_data_and_model/MRPC/train.tsv\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   guid: train-1\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 2572, 3217, 5831, 5496, 2010, 2567, 1010, 3183, 2002, 2170, 1000, 1996, 7409, 1000, 1010, 1997, 9969, 4487, 23809, 3436, 2010, 3350, 1012, 102, 7727, 2000, 2032, 2004, 2069, 1000, 1996, 7409, 1000, 1010, 2572, 3217, 5831, 5496, 2010, 2567, 1997, 9969, 4487, 23809, 3436, 2010, 3350, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=1)\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   guid: train-2\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 9805, 3540, 11514, 2050, 3079, 11282, 2243, 1005, 1055, 2077, 4855, 1996, 4677, 2000, 3647, 4576, 1999, 2687, 2005, 1002, 1016, 1012, 1019, 4551, 1012, 102, 9805, 3540, 11514, 2050, 4149, 11282, 2243, 1005, 1055, 1999, 2786, 2005, 1002, 6353, 2509, 2454, 1998, 2853, 2009, 2000, 3647, 4576, 2005, 1002, 1015, 1012, 1022, 4551, 1999, 2687, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   guid: train-3\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 2027, 2018, 2405, 2019, 15147, 2006, 1996, 4274, 2006, 2238, 2184, 1010, 5378, 1996, 6636, 2005, 5096, 1010, 2002, 2794, 1012, 102, 2006, 2238, 2184, 1010, 1996, 2911, 1005, 1055, 5608, 2018, 2405, 2019, 15147, 2006, 1996, 4274, 1010, 5378, 1996, 14792, 2005, 5096, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=1)\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   guid: train-4\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 2105, 6021, 19481, 13938, 2102, 1010, 21628, 6661, 2020, 2039, 2539, 16653, 1010, 2030, 1018, 1012, 1018, 1003, 1010, 2012, 1037, 1002, 1018, 1012, 5179, 1010, 2383, 3041, 2275, 1037, 2501, 2152, 1997, 1037, 1002, 1018, 1012, 5401, 1012, 102, 21628, 6661, 5598, 2322, 16653, 1010, 2030, 1018, 1012, 1020, 1003, 1010, 2000, 2275, 1037, 2501, 5494, 2152, 2012, 1037, 1002, 1018, 1012, 5401, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   guid: train-5\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 1996, 4518, 3123, 1002, 1016, 1012, 2340, 1010, 2030, 2055, 2340, 3867, 1010, 2000, 2485, 5958, 2012, 1002, 2538, 1012, 4868, 2006, 1996, 2047, 2259, 4518, 3863, 1012, 102, 18720, 1004, 1041, 13058, 1012, 6661, 5598, 1002, 1015, 1012, 6191, 2030, 1022, 3867, 2000, 1002, 2538, 1012, 6021, 2006, 1996, 2047, 2259, 4518, 3863, 2006, 5958, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=1)\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   Saving features into cached file transformers_data_and_model/MRPC/cached_train_BertTokenizer_128_mrpc [took 0.834 s]\n",
      "07/27/2020 19:04:08 - INFO - filelock -   Lock 139748473285376 released on transformers_data_and_model/MRPC/cached_train_BertTokenizer_128_mrpc.lock\n",
      "07/27/2020 19:04:08 - INFO - filelock -   Lock 139748862558504 acquired on transformers_data_and_model/MRPC/cached_dev_BertTokenizer_128_mrpc.lock\n",
      "07/27/2020 19:04:08 - INFO - __main__ -   Creating features from dataset file at transformers_data_and_model/MRPC\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   guid: dev-1\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 2002, 2056, 1996, 9440, 2121, 7903, 2063, 11345, 2449, 2987, 1005, 1056, 4906, 1996, 2194, 1005, 1055, 2146, 1011, 2744, 3930, 5656, 1012, 102, 1000, 1996, 9440, 2121, 7903, 2063, 11345, 2449, 2515, 2025, 4906, 2256, 2146, 1011, 2744, 3930, 5656, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=1)\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   guid: dev-2\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 20201, 22948, 2056, 10958, 19053, 4140, 6283, 1996, 8956, 6939, 1998, 2246, 2830, 2000, 2478, 2010, 2146, 2086, 1997, 2731, 1999, 1996, 2162, 1012, 102, 2010, 2564, 2056, 2002, 2001, 1000, 2531, 3867, 2369, 2577, 5747, 1000, 1998, 2246, 2830, 2000, 2478, 2010, 2086, 1997, 2731, 1999, 1996, 2162, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   guid: dev-3\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 1996, 7922, 2001, 2012, 12904, 1012, 6227, 18371, 2114, 1996, 18371, 1010, 4257, 2006, 1996, 5219, 1010, 1998, 2012, 1015, 1012, 27054, 2487, 2114, 1996, 5364, 23151, 2278, 1010, 2036, 4257, 1012, 102, 1996, 7922, 2001, 2012, 12904, 1012, 6275, 18371, 16545, 2100, 1027, 1010, 8990, 4257, 2006, 1996, 5219, 1010, 1998, 2012, 1015, 1012, 23090, 2487, 2114, 1996, 5364, 23151, 2278, 10381, 2546, 1027, 1010, 2091, 1014, 1012, 1015, 3867, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   guid: dev-4\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 1996, 10028, 1011, 25022, 2080, 2003, 3403, 2127, 2255, 2000, 5630, 2065, 2009, 2097, 2203, 5668, 2063, 1037, 4018, 1012, 102, 1996, 10028, 1011, 25022, 2080, 2623, 9317, 2008, 2009, 2097, 5630, 1999, 2255, 3251, 2000, 2203, 5668, 2063, 1037, 4018, 2077, 1996, 27419, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=1)\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   *** Example ***\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   guid: dev-5\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   features: InputFeatures(input_ids=[101, 2053, 5246, 2031, 2042, 2275, 2005, 1996, 2942, 2030, 1996, 4735, 3979, 1012, 102, 2053, 5246, 2031, 2042, 2275, 2005, 1996, 4735, 2030, 2942, 3572, 1010, 2021, 17137, 3051, 2038, 12254, 2025, 5905, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)\n",
      "07/27/2020 19:04:09 - INFO - __main__ -   Saving features into cached file transformers_data_and_model/MRPC/cached_dev_BertTokenizer_128_mrpc [took 0.103 s]\n",
      "07/27/2020 19:04:09 - INFO - filelock -   Lock 139748862558504 released on transformers_data_and_model/MRPC/cached_dev_BertTokenizer_128_mrpc.lock\n"
     ]
    }
   ],
   "source": [
    "# 获得dataset\n",
    "train_dataset, eval_dataset, test_dataset = None, None, None\n",
    "if training_args.do_train:\n",
    "    train_dataset = GlueDataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir, output_mode=output_mode)\n",
    "if training_args.do_eval:\n",
    "    eval_dataset = GlueDataset(data_args, tokenizer=tokenizer, mode=\"dev\", cache_dir=model_args.cache_dir)\n",
    "if training_args.do_predict:\n",
    "    test_dataset = GlueDataset(data_args, tokenizer=tokenizer, mode=\"test\", cache_dir=model_args.cache_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 得到计算结果的函数\n",
    "# 这里计算的是acc和f1\n",
    "# EvalPrediction是预测的结果格式，predictions是预测的，labels_ids是正确的\n",
    "class EvalPrediction(NamedTuple):\n",
    "    \"\"\"\n",
    "    Evaluation output (always contains labels), to be used to compute metrics.\n",
    "\n",
    "    Parameters:\n",
    "        predictions (:obj:`np.ndarray`): Predictions of the model.\n",
    "        label_ids (:obj:`np.ndarray`): Targets to be matched.\n",
    "    \"\"\"\n",
    "\n",
    "    predictions: np.ndarray\n",
    "    label_ids: np.ndarray\n",
    "        \n",
    "# 得到计算函数\n",
    "def compute_metrics_fn(p: EvalPrediction):\n",
    "    if output_mode == \"classification\":\n",
    "        # 预测的结果\n",
    "        preds = np.argmax(p.predictions, axis=1)\n",
    "    # 正确的结果\n",
    "    labels = p.label_ids\n",
    "    \n",
    "    # acc和f1\n",
    "    acc = (preds == labels).mean()\n",
    "    f1 = f1_score(y_true=labels, y_pred=preds)\n",
    "    return {\"acc\": acc, \"f1\": f1, \"acc_and_f1\": (acc + f1) / 2}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:04:13 - INFO - transformers.trainer -   You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.\n"
     ]
    }
   ],
   "source": [
    "#初始化本Trainer\n",
    "trainer = transformers.Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=train_dataset,\n",
    "    eval_dataset=eval_dataset,\n",
    "    compute_metrics=compute_metrics_fn,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:04:13 - INFO - transformers.trainer -   ***** Running training *****\n",
      "07/27/2020 19:04:13 - INFO - transformers.trainer -     Num examples = 3668\n",
      "07/27/2020 19:04:13 - INFO - transformers.trainer -     Num Epochs = 3\n",
      "07/27/2020 19:04:13 - INFO - transformers.trainer -     Instantaneous batch size per device = 32\n",
      "07/27/2020 19:04:13 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 64\n",
      "07/27/2020 19:04:13 - INFO - transformers.trainer -     Gradient Accumulation steps = 1\n",
      "07/27/2020 19:04:13 - INFO - transformers.trainer -     Total optimization steps = 174\n",
      "07/27/2020 19:04:13 - INFO - transformers.trainer -     Starting fine-tuning.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29c3039f19a948f7b66dddd3d9b071fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch', max=3.0, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7bd50a33b13491ab622b72fa383c19b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Iteration', max=58.0, style=ProgressStyle(description_wid…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
      "  warnings.warn('Was asked to gather along dimension 0, but all '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8dc3ebeabcf24ad2aed84dece8d2aa4a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Iteration', max=58.0, style=ProgressStyle(description_wid…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c2c5f9b3712b48bea75c5821a78eab5d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Iteration', max=58.0, style=ProgressStyle(description_wid…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:18 - INFO - transformers.trainer -   \n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "07/27/2020 19:05:18 - INFO - transformers.trainer -   Saving model checkpoint to transformers_data_and_model/finetuned-mrpc\n",
      "07/27/2020 19:05:18 - INFO - transformers.configuration_utils -   Configuration saved in transformers_data_and_model/finetuned-mrpc/config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:19 - INFO - transformers.modeling_utils -   Model weights saved in transformers_data_and_model/finetuned-mrpc/pytorch_model.bin\n"
     ]
    }
   ],
   "source": [
    "# 训练\n",
    "if training_args.do_train:\n",
    "    # 此时传入model_path是为了加载optimiztor，进行继续训练，对于通常的fine-tune来说，model_path可以不传入\n",
    "    # 对于初始化后的Trainer，调用train方法就可以训练了，简化了训练的过程\n",
    "    trainer.train(\n",
    "        model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None\n",
    "    )\n",
    "    # Trainer保存模型调用此方法\n",
    "    trainer.save_model()\n",
    "    \n",
    "    # 为了方便使用起见，将tokenizer的模型参数也存入model同目录\n",
    "    if trainer.is_world_master():\n",
    "        tokenizer.save_pretrained(training_args.output_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:19 - INFO - __main__ -   *** Evaluate ***\n",
      "07/27/2020 19:05:19 - INFO - transformers.trainer -   ***** Running Evaluation *****\n",
      "07/27/2020 19:05:19 - INFO - transformers.trainer -     Num examples = 408\n",
      "07/27/2020 19:05:19 - INFO - transformers.trainer -     Batch size = 16\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "270ae155881e422bb1b587c160446397",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Evaluation', max=26.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:21 - INFO - transformers.trainer -   {'eval_loss': 0.4694334108095903, 'eval_acc': 0.8112745098039216, 'eval_f1': 0.8718801996672212, 'eval_acc_and_f1': 0.8415773547355714, 'epoch': 3.0, 'step': 174}\n",
      "07/27/2020 19:05:21 - INFO - __main__ -   ***** Eval results mrpc *****\n",
      "07/27/2020 19:05:21 - INFO - __main__ -     eval_loss = 0.4694334108095903\n",
      "07/27/2020 19:05:21 - INFO - __main__ -     eval_acc = 0.8112745098039216\n",
      "07/27/2020 19:05:21 - INFO - __main__ -     eval_f1 = 0.8718801996672212\n",
      "07/27/2020 19:05:21 - INFO - __main__ -     eval_acc_and_f1 = 0.8415773547355714\n",
      "07/27/2020 19:05:21 - INFO - __main__ -     epoch = 3.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# 评估结果\n",
    "eval_results = {}\n",
    "if training_args.do_eval:\n",
    "    logger.info(\"*** Evaluate ***\")\n",
    "\n",
    "    # 传入metrics，对于前面初始化已经传入的，此时如果没变化，可以省略此步骤\n",
    "    trainer.compute_metrics = compute_metrics_fn\n",
    "    # 传入评估的dataset\n",
    "    eval_result = trainer.evaluate(eval_dataset=eval_dataset)\n",
    "\n",
    "    output_eval_file = os.path.join(\n",
    "        training_args.output_dir, f\"eval_results_{eval_dataset.args.task_name}.txt\"\n",
    "    )\n",
    "    if trainer.is_world_master():\n",
    "        # 写入本地文件\n",
    "        with open(output_eval_file, \"w\") as writer:\n",
    "            logger.info(\"***** Eval results {} *****\".format(eval_dataset.args.task_name))\n",
    "            for key, value in eval_result.items():\n",
    "                logger.info(\"  %s = %s\", key, value)\n",
    "                writer.write(\"%s = %s\\n\" % (key, value))\n",
    "\n",
    "        eval_results.update(eval_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 测试集的评估\n",
    "if training_args.do_predict:\n",
    "    logging.info(\"*** Test ***\")\n",
    "\n",
    "    predictions = trainer.predict(test_dataset=test_dataset).predictions\n",
    "    if output_mode == \"classification\":\n",
    "        predictions = np.argmax(predictions, axis=1)\n",
    "\n",
    "    output_test_file = os.path.join(\n",
    "        training_args.output_dir, f\"test_results_{test_dataset.args.task_name}.txt\"\n",
    "    )\n",
    "    if trainer.is_world_master():\n",
    "        with open(output_test_file, \"w\") as writer:\n",
    "            logger.info(\"***** Test results {} *****\".format(test_dataset.args.task_name))\n",
    "            writer.write(\"index\\tprediction\\n\")\n",
    "            for index, item in enumerate(predictions):\n",
    "                item = test_dataset.get_labels()[item]\n",
    "                writer.write(\"%d\\t%s\\n\" % (index, item))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4694334108095903, 'eval_acc': 0.8112745098039216, 'eval_f1': 0.8718801996672212, 'eval_acc_and_f1': 0.8415773547355714, 'epoch': 3.0}\n"
     ]
    }
   ],
   "source": [
    "print(eval_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'transformers_data_and_model/finetuned-mrpc'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FINETUNED_MRPC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transformers_data_and_model/finetuned-mrpc\n",
      "├── config.json\n",
      "├── eval_results_mrpc.txt\n",
      "├── pytorch_model.bin\n",
      "├── special_tokens_map.json\n",
      "├── tokenizer_config.json\n",
      "├── training_args.bin\n",
      "└── vocab.txt\n",
      "\n",
      "0 directories, 7 files\n"
     ]
    }
   ],
   "source": [
    "# 查看训练好的模型文件\n",
    "!tree {FINETUNED_MRPC}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为MRPC数据集非常小，任务也比较简单，3个epoch，学习率3e-5，就能轻松的完成了训练，可以看到在验证集上accuracy达到了81.1%，F1值达到了0.87。训练好的模型model和tokenizer都保存在了FINETUNE_MRPC文件夹下了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 线上预测\n",
    "\n",
    "训练好了模型，就可以线上进行预测了，怎么加载训练好的模型进行预测呢？也非常简单。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch: 1.4.0+cu100\n",
      "transformers: 3.0.2\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import transformers\n",
    "\n",
    "print(f'torch: {torch.__version__}')\n",
    "print(f'transformers: {transformers.__version__}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:22 - INFO - transformers.configuration_utils -   loading configuration file transformers_data_and_model/finetuned-mrpc/config.json\n",
      "07/27/2020 19:05:22 - INFO - transformers.configuration_utils -   Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForSequenceClassification\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"finetuning_task\": \"mrpc\",\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "07/27/2020 19:05:22 - INFO - transformers.tokenization_utils_base -   Model name 'transformers_data_and_model/finetuned-mrpc' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). Assuming 'transformers_data_and_model/finetuned-mrpc' is a path, a model identifier, or url to a directory containing tokenizer files.\n",
      "07/27/2020 19:05:22 - INFO - transformers.tokenization_utils_base -   Didn't find file transformers_data_and_model/finetuned-mrpc/added_tokens.json. We won't load it.\n",
      "07/27/2020 19:05:22 - INFO - transformers.tokenization_utils_base -   Didn't find file transformers_data_and_model/finetuned-mrpc/tokenizer.json. We won't load it.\n",
      "07/27/2020 19:05:22 - INFO - transformers.tokenization_utils_base -   loading file transformers_data_and_model/finetuned-mrpc/vocab.txt\n",
      "07/27/2020 19:05:22 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:05:22 - INFO - transformers.tokenization_utils_base -   loading file transformers_data_and_model/finetuned-mrpc/special_tokens_map.json\n",
      "07/27/2020 19:05:22 - INFO - transformers.tokenization_utils_base -   loading file transformers_data_and_model/finetuned-mrpc/tokenizer_config.json\n",
      "07/27/2020 19:05:22 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:05:22 - INFO - transformers.configuration_utils -   loading configuration file transformers_data_and_model/finetuned-mrpc/config.json\n",
      "07/27/2020 19:05:22 - INFO - transformers.configuration_utils -   Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForSequenceClassification\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"finetuning_task\": \"mrpc\",\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "07/27/2020 19:05:22 - INFO - transformers.modeling_utils -   loading weights file transformers_data_and_model/finetuned-mrpc/pytorch_model.bin\n",
      "07/27/2020 19:05:25 - INFO - transformers.modeling_utils -   All model checkpoint weights were used when initializing BertForSequenceClassification.\n",
      "\n",
      "07/27/2020 19:05:25 - INFO - transformers.modeling_utils -   All the weights of BertForSequenceClassification were initialized from the model checkpoint at transformers_data_and_model/finetuned-mrpc.\n",
      "If your task is similar to the task the model of the ckeckpoint was trained on, you can already use BertForSequenceClassification for predictions without further training.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not paraphrase: 38%\n",
      "is paraphrase: 62%\n",
      "classification result: is paraphrase\n"
     ]
    }
   ],
   "source": [
    "# 加载tokenizer和model\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(FINETUNED_MRPC)\n",
    "model = transformers.AutoModelForSequenceClassification.from_pretrained(FINETUNED_MRPC)\n",
    "\n",
    "# 类别标签\n",
    "classes = [\"not paraphrase\", \"is paraphrase\"]\n",
    "\n",
    "sequence_0 = \"The company HuggingFace is based in New York City\"\n",
    "sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n",
    "\n",
    "# 通过tokenizer将文本转成model需要的格式，返回为pytorch的tensor\n",
    "paraphrase = tokenizer(sequence_0, sequence_1, return_tensors=\"pt\")\n",
    "\n",
    "with torch.no_grad():\n",
    "    # 是释义的样例\n",
    "    # 输入模型，模型输出为元组，不输入正确标签labels参数的情况下，第一个logits\n",
    "    paraphrase_classification_logits = model(**paraphrase)[0]\n",
    "    # logits经过softmax得到最终的概率\n",
    "    paraphrase_results = torch.softmax(paraphrase_classification_logits, dim=1).tolist()[0]\n",
    "\n",
    "# 结果应当是释义\n",
    "for i in range(len(classes)):\n",
    "     print(f\"{classes[i]}: {int(round(paraphrase_results[i] * 100))}%\")\n",
    "print(f\"classification result: {classes[paraphrase_results.index(max(paraphrase_results))]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总结：可以看到，线上用于预测的代码也非常简单。加载tokenizer和model，然后通过tokenizer标准化为model的输入，模型输入输出为元组，元组的结果进行softmax，得到各个label的概率，得到最终的结果概率。可以看到，对于训练好的模型，进行线上部署的代码也非常简单和容易。\n",
    "\n",
    "上面为了简单起见，没有使用GPU。如果线上部署需要支持GPU，也很简单，只要将tokenizer输出的结果传入GPU，model也传入GPU即可。一般tokenizer输出结果是一个词典格式，词典的value为tensor，可以转入GPU，请看下面的例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:25 - INFO - transformers.configuration_utils -   loading configuration file transformers_data_and_model/finetuned-mrpc/config.json\n",
      "07/27/2020 19:05:25 - INFO - transformers.configuration_utils -   Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForSequenceClassification\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"finetuning_task\": \"mrpc\",\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "07/27/2020 19:05:25 - INFO - transformers.tokenization_utils_base -   Model name 'transformers_data_and_model/finetuned-mrpc' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). Assuming 'transformers_data_and_model/finetuned-mrpc' is a path, a model identifier, or url to a directory containing tokenizer files.\n",
      "07/27/2020 19:05:25 - INFO - transformers.tokenization_utils_base -   Didn't find file transformers_data_and_model/finetuned-mrpc/added_tokens.json. We won't load it.\n",
      "07/27/2020 19:05:25 - INFO - transformers.tokenization_utils_base -   Didn't find file transformers_data_and_model/finetuned-mrpc/tokenizer.json. We won't load it.\n",
      "07/27/2020 19:05:25 - INFO - transformers.tokenization_utils_base -   loading file transformers_data_and_model/finetuned-mrpc/vocab.txt\n",
      "07/27/2020 19:05:25 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:05:25 - INFO - transformers.tokenization_utils_base -   loading file transformers_data_and_model/finetuned-mrpc/special_tokens_map.json\n",
      "07/27/2020 19:05:25 - INFO - transformers.tokenization_utils_base -   loading file transformers_data_and_model/finetuned-mrpc/tokenizer_config.json\n",
      "07/27/2020 19:05:25 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:05:25 - INFO - transformers.configuration_utils -   loading configuration file transformers_data_and_model/finetuned-mrpc/config.json\n",
      "07/27/2020 19:05:25 - INFO - transformers.configuration_utils -   Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForSequenceClassification\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"finetuning_task\": \"mrpc\",\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "07/27/2020 19:05:25 - INFO - transformers.modeling_utils -   loading weights file transformers_data_and_model/finetuned-mrpc/pytorch_model.bin\n",
      "07/27/2020 19:05:28 - INFO - transformers.modeling_utils -   All model checkpoint weights were used when initializing BertForSequenceClassification.\n",
      "\n",
      "07/27/2020 19:05:28 - INFO - transformers.modeling_utils -   All the weights of BertForSequenceClassification were initialized from the model checkpoint at transformers_data_and_model/finetuned-mrpc.\n",
      "If your task is similar to the task the model of the ckeckpoint was trained on, you can already use BertForSequenceClassification for predictions without further training.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not paraphrase: 38%\n",
      "is paraphrase: 62%\n",
      "classification result: is paraphrase\n"
     ]
    }
   ],
   "source": [
    "# 支持GPU则使用GPU，不支持则使用CPU\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# 加载tokenizer和model\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(FINETUNED_MRPC)\n",
    "model = transformers.AutoModelForSequenceClassification.from_pretrained(FINETUNED_MRPC)\n",
    "# model转入cuda\n",
    "model = model.to(device)\n",
    "\n",
    "# 类别标签\n",
    "classes = [\"not paraphrase\", \"is paraphrase\"]\n",
    "\n",
    "sequence_0 = \"The company HuggingFace is based in New York City\"\n",
    "sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n",
    "# 通过tokenizer将文本转成model需要的格式，返回为pytorch的tensor\n",
    "paraphrase = tokenizer(sequence_0, sequence_1, return_tensors=\"pt\")\n",
    "# paraphrase转入GPU\n",
    "# value是tensor类的，转入GPU进行加速\n",
    "# 除了model和输入的转入GPU，其他无区别\n",
    "for k, v in paraphrase.items():\n",
    "    if isinstance(v, torch.Tensor):\n",
    "        paraphrase[k] = v.to(device)\n",
    "\n",
    "with torch.no_grad():\n",
    "    # 是释义的样例\n",
    "    # 输入模型，模型输出为元组，不输入正确标签labels参数的情况下，第一个logits\n",
    "    paraphrase_classification_logits = model(**paraphrase)[0]\n",
    "    # logits经过softmax得到最终的概率\n",
    "    paraphrase_results = torch.softmax(paraphrase_classification_logits, dim=1).tolist()[0]\n",
    "\n",
    "# 结果应当是释义\n",
    "for i in range(len(classes)):\n",
    "     print(f\"{classes[i]}: {int(round(paraphrase_results[i] * 100))}%\")\n",
    "print(f\"classification result: {classes[paraphrase_results.index(max(paraphrase_results))]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里已经实现了线上部署的全部代码。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 transformers使用总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总体来说，基于transformers框架，写代码会非常简单，总体步骤也非常少。\n",
    "\n",
    "1. 参数的传入，参数包含三种类型的参数：模型的参数，模型训练的参数，数据的参数。\n",
    "2. 数据的处理，从原始数据到模型可接受的数据，最终是将数据分成可以迭代的，定长的、标准的batch，供模型使用。这里面通常标准化为几个步骤，这也是pytorch代码编写的流程：\n",
    "\n",
    "第一步，设立processor，完成加载数据集为examples的集和（list），每个example可以为一个InputExample对象。这里通常还是文本，只是text_a, text_b，label变为InputExample的属性。\n",
    "\n",
    "第二步，转换为torch.utils.data.Dataset，pytorch的Dataset是一个抽象类，我们需要重写__len__,\\_\\_getitem__这两个方法。最终的目标是将文本变成一个个模型可以用的一条条的数据。在这里将每个InputExample转换成feature，而InputExample转换成feature的时候，就需要tokenizer发挥作用了，这里可以进行padding和truncation。\n",
    "\n",
    "第三步，在transformers里，可以将上一步得到的dataset传入Trainer进行使用，Trainer会自动处理此步骤，不必过分操心里面的事情。对于通常的pytorch来说，此步骤是将dataset转换为dataloader，要决定batch_size，是否shuffle，抽样方法，collate_fn聚合方式。对于transformers里，collate_fn可以使用默认或者自己写的传入，可以将batch_size通过1的参数传入。\n",
    "\n",
    "DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,\n",
    "           batch_sampler=None, num_workers=0, collate_fn=None,\n",
    "           pin_memory=False, drop_last=False, timeout=0,\n",
    "           worker_init_fn=None)\n",
    "\n",
    "3. 定义metrics用来评估预测的效果。如果有需要，也需要写collate_fn聚合函数。\n",
    "4. 对于已经处理好的数据train_dataset、dev_dataset、test_dataset，连同model、tokenizer、metric、args一同送入trainer，进行训练，保存。这里使用的是save_model，其实就是调用的标准的save_pretrained的接口。\n",
    "\n",
    "通过本节的学习，已经掌握了文本分类任务的详细训练过程、线上部署的代码；对于文本分类的流程及transformers、pytorch代码的书写，使用已经熟悉；对于新的文本分类任务已经可以独立完成。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第五节 GPT2训练使用示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 本节说明\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLP领域，除了常见的文本分类任务，还有文本生成任务。随着预训练的火热，很多人可能想尝试自己预训练一个GPT2模型或者因业务需要fine-tune GPT2模型。本节解决的就是GPT2的从头预训练及微调，利用自己重头预训练的模型及微调的模型进行句子生成任务。\n",
    "\n",
    "本章节的结构与第四节类似，5.2节是重头训练或者在预训练模型上继续fine-tune GPT2模型；5.3节是使用官方预训练的GPT2或者我们自己训练的GPT2模型进行句子生成，此场景对应于我们的文本生成任务的线上服务部分；本节整体思路和流程是和第四节相似的。\n",
    "\n",
    "本节涉及到的预训练模型是GPT2，涉及到的数据集是WikiText-2数据集[9]，简单的讲，WikiText-2是从维基百科上抓去的大量好的文本，量很大，具体的可以自行查看里面的文本。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 预训练的详细过程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面的是重头开始预训练GPT2的例子。如果想改为在已有的GPT2模型上进行微调，只需要在下面的输入参数input_args增加一行代码'--model_name_or_path', GPT2_MODEL_NAME_OR_PATH即可，更具体看下面代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import math\n",
    "import os\n",
    "import time\n",
    "import pickle\n",
    "from dataclasses import dataclass, field\n",
    "from typing import Optional, List, Dict, Tuple\n",
    "\n",
    "import filelock\n",
    "import torch\n",
    "import transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 初始化日志\n",
    "logger = logging.getLogger(__name__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 同上面的训练一样，这里是模型参数，下面是数据参数\n",
    "@dataclass\n",
    "class ModelArguments:\n",
    "    \"\"\"\n",
    "    Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch.\n",
    "    \"\"\"\n",
    "\n",
    "    model_name_or_path: Optional[str] = field(\n",
    "        default=None,\n",
    "        metadata={\n",
    "            \"help\": \"The model checkpoint for weights initialization. Leave None if you want to train a model from scratch.\"\n",
    "        },\n",
    "    )\n",
    "    model_type: Optional[str] = field(\n",
    "        default=None,\n",
    "        metadata={\"help\": \"If training from scratch, pass a model type from the list: GPT \"},\n",
    "    )\n",
    "    config_name: Optional[str] = field(\n",
    "        default=None, metadata={\"help\": \"Pretrained config name or path if not the same as model_name\"}\n",
    "    )\n",
    "    tokenizer_name: Optional[str] = field(\n",
    "        default=None, metadata={\"help\": \"Pretrained tokenizer name or path if not the same as model_name\"}\n",
    "    )\n",
    "    cache_dir: Optional[str] = field(\n",
    "        default=None, metadata={\"help\": \"Where do you want to store the pretrained models downloaded from s3\"}\n",
    "    )\n",
    "\n",
    "\n",
    "@dataclass\n",
    "class DataTrainingArguments:\n",
    "    \"\"\"\n",
    "    Arguments pertaining to what data we are going to input our model for training and eval.\n",
    "    \"\"\"\n",
    "\n",
    "    train_data_file: Optional[str] = field(\n",
    "        default=None, metadata={\"help\": \"The input training data file (a text file).\"}\n",
    "    )\n",
    "    eval_data_file: Optional[str] = field(\n",
    "        default=None,\n",
    "        metadata={\"help\": \"An optional input evaluation data file to evaluate the perplexity on (a text file).\"},\n",
    "    )\n",
    "    line_by_line: bool = field(\n",
    "        default=False,\n",
    "        metadata={\"help\": \"Whether distinct lines of text in the dataset are to be handled as distinct sequences.\"},\n",
    "    )\n",
    "\n",
    "    mlm: bool = field(\n",
    "        default=False, metadata={\"help\": \"Train with masked-language modeling loss instead of language modeling.\"}\n",
    "    )\n",
    "    mlm_probability: float = field(\n",
    "        default=0.15, metadata={\"help\": \"Ratio of tokens to mask for masked language modeling loss\"}\n",
    "    )\n",
    "    plm_probability: float = field(\n",
    "        default=1 / 6,\n",
    "        metadata={\n",
    "            \"help\": \"Ratio of length of a span of masked tokens to surrounding context length for permutation language modeling.\"\n",
    "        },\n",
    "    )\n",
    "    max_span_length: int = field(\n",
    "        default=5, metadata={\"help\": \"Maximum length of a span of masked tokens for permutation language modeling.\"}\n",
    "    )\n",
    "\n",
    "    block_size: int = field(\n",
    "        default=-1,\n",
    "        metadata={\n",
    "            \"help\": \"Optional input sequence length after tokenization.\"\n",
    "            \"The training dataset will be truncated in block of this size for training.\"\n",
    "            \"Default to the model max input length for single sentence inputs (take into account special tokens).\"\n",
    "        },\n",
    "    )\n",
    "    overwrite_cache: bool = field(\n",
    "        default=False, metadata={\"help\": \"Overwrite the cached training and evaluation sets\"}\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# config, tokenizer, model\n",
    "CONFIG_NAME = '/dfsdata2/yucc1_data/models/huggingface/gpt2'\n",
    "TOKENIZER_NAME = '/dfsdata2/yucc1_data/models/huggingface/gpt2'\n",
    "# 从头进行预训练，不需要指定此参数；继续finetune，需要指定原始的gpt2模型所在位置\n",
    "GPT2_MODEL_NAME_OR_PATH = '/dfsdata2/yucc1_data/models/huggingface/gpt2'\n",
    "# 生成的模型\n",
    "GPT2_OUTPUT_DIR = '/dfsdata2/yucc1_data/output/gpt2-train-new-model'\n",
    "# 训练集的位置\n",
    "TRAIN_DATA_FILE = '/dfsdata2/yucc1_data/datasets/wikitext-2-raw/wiki.train.raw'\n",
    "# 验证集的位置\n",
    "EVAL_DATA_FILE = '/dfsdata2/yucc1_data/datasets/wikitext-2-raw/wiki.test.raw'\n",
    "\n",
    "# num_train_epochs后的参数，此时是3.0，对于实际训练，建议提升epochs对应的值。\n",
    "input_args = [\n",
    "    '--output_dir', GPT2_OUTPUT_DIR,\n",
    "    '--model_type', 'gpt2',\n",
    "    '--config_name', CONFIG_NAME,\n",
    "    '--tokenizer_name', TOKENIZER_NAME,\n",
    "    '--do_train',\n",
    "    '--train_data_file', TRAIN_DATA_FILE,\n",
    "    '--do_eval',\n",
    "    '--eval_data_file', EVAL_DATA_FILE,\n",
    "    '--block_size', '510',\n",
    "    '--save_steps', '5000',\n",
    "    '--num_train_epochs', '3.0',\n",
    "    '--overwrite_cache',\n",
    "    '--overwrite_output_dir',\n",
    "]\n",
    "\n",
    "# 如果不是重头预训练，而是在已有的GPT2预训练模型上微调，则加入下面的一行代码，指定已有的预训练模型即可。\n",
    "# input_args.extend(['--model_name_or_path', GPT2_MODEL_NAME_OR_PATH])\n",
    "\n",
    "\n",
    "parser = transformers.HfArgumentParser((ModelArguments, DataTrainingArguments, transformers.TrainingArguments))\n",
    "model_args, data_args, training_args = parser.parse_args_into_dataclasses(input_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ModelArguments(model_name_or_path=None, model_type='gpt2', config_name='/dfsdata2/yucc1_data/models/huggingface/gpt2', tokenizer_name='/dfsdata2/yucc1_data/models/huggingface/gpt2', cache_dir=None)\n",
      "\n",
      "DataTrainingArguments(train_data_file='/dfsdata2/yucc1_data/datasets/wikitext-2-raw/wiki.train.raw', eval_data_file='/dfsdata2/yucc1_data/datasets/wikitext-2-raw/wiki.test.raw', line_by_line=False, mlm=False, mlm_probability=0.15, plm_probability=0.16666666666666666, max_span_length=5, block_size=510, overwrite_cache=True)\n",
      "\n",
      "TrainingArguments(output_dir='/dfsdata2/yucc1_data/output/gpt2-train-new-model', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Jul27_19-05-28_d1f9a32623b6', logging_first_step=False, logging_steps=500, save_steps=5000, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1)\n"
     ]
    }
   ],
   "source": [
    "# 查看这三类参数\n",
    "print(f'{model_args}\\n\\n{data_args}\\n\\n{training_args}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 确保输入参数正确\n",
    "# do_eval与eval_data_file两个参数统一\n",
    "# 输出文件夹合理\n",
    "if data_args.eval_data_file is None and training_args.do_eval:\n",
    "    raise ValueError(\n",
    "        \"Cannot do evaluation without an evaluation data file. Either supply a file to --eval_data_file \"\n",
    "        \"or remove the --do_eval argument.\"\n",
    "    )\n",
    "\n",
    "if (\n",
    "    os.path.exists(training_args.output_dir)\n",
    "    and os.listdir(training_args.output_dir)\n",
    "    and training_args.do_train\n",
    "    and not training_args.overwrite_output_dir\n",
    "):\n",
    "    raise ValueError(\n",
    "        f\"Output directory ({training_args.output_dir}) already exists and is not empty. Use --overwrite_output_dir to overcome.\"\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:28 - INFO - transformers.training_args -   PyTorch: setting up devices\n",
      "07/27/2020 19:05:28 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 2, distributed training: False, 16-bits training: False\n",
      "07/27/2020 19:05:28 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/dfsdata2/yucc1_data/output/gpt2-train-new-model', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Jul27_19-05-28_d1f9a32623b6', logging_first_step=False, logging_steps=500, save_steps=5000, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1)\n"
     ]
    }
   ],
   "source": [
    "# 设置日志格式，并记录本次训练的重要参数\n",
    "logging.basicConfig(\n",
    "    format=\"%(asctime)s - %(levelname)s - %(name)s -   %(message)s\",\n",
    "    datefmt=\"%m/%d/%Y %H:%M:%S\",\n",
    "    level=logging.INFO if training_args.local_rank in [-1, 0] else logging.WARN,\n",
    ")\n",
    "logger.warning(\n",
    "    \"Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s\",\n",
    "    training_args.local_rank,\n",
    "    training_args.device,\n",
    "    training_args.n_gpu,\n",
    "    bool(training_args.local_rank != -1),\n",
    "    training_args.fp16,\n",
    ")\n",
    "logger.info(\"Training/evaluation parameters %s\", training_args)\n",
    "\n",
    "# 设置seed\n",
    "transformers.set_seed(training_args.seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:28 - INFO - transformers.configuration_utils -   loading configuration file /dfsdata2/yucc1_data/models/huggingface/gpt2/config.json\n",
      "07/27/2020 19:05:28 - INFO - transformers.configuration_utils -   Model config GPT2Config {\n",
      "  \"activation_function\": \"gelu_new\",\n",
      "  \"architectures\": [\n",
      "    \"GPT2LMHeadModel\"\n",
      "  ],\n",
      "  \"attn_pdrop\": 0.1,\n",
      "  \"bos_token_id\": 50256,\n",
      "  \"embd_pdrop\": 0.1,\n",
      "  \"eos_token_id\": 50256,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"layer_norm_epsilon\": 1e-05,\n",
      "  \"model_type\": \"gpt2\",\n",
      "  \"n_ctx\": 1024,\n",
      "  \"n_embd\": 768,\n",
      "  \"n_head\": 12,\n",
      "  \"n_layer\": 12,\n",
      "  \"n_positions\": 1024,\n",
      "  \"resid_pdrop\": 0.1,\n",
      "  \"summary_activation\": null,\n",
      "  \"summary_first_dropout\": 0.1,\n",
      "  \"summary_proj_to_labels\": true,\n",
      "  \"summary_type\": \"cls_index\",\n",
      "  \"summary_use_proj\": true,\n",
      "  \"task_specific_params\": {\n",
      "    \"text-generation\": {\n",
      "      \"do_sample\": true,\n",
      "      \"max_length\": 50\n",
      "    }\n",
      "  },\n",
      "  \"vocab_size\": 50257\n",
      "}\n",
      "\n",
      "07/27/2020 19:05:28 - INFO - transformers.configuration_utils -   loading configuration file /dfsdata2/yucc1_data/models/huggingface/gpt2/config.json\n",
      "07/27/2020 19:05:28 - INFO - transformers.configuration_utils -   Model config GPT2Config {\n",
      "  \"activation_function\": \"gelu_new\",\n",
      "  \"architectures\": [\n",
      "    \"GPT2LMHeadModel\"\n",
      "  ],\n",
      "  \"attn_pdrop\": 0.1,\n",
      "  \"bos_token_id\": 50256,\n",
      "  \"embd_pdrop\": 0.1,\n",
      "  \"eos_token_id\": 50256,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"layer_norm_epsilon\": 1e-05,\n",
      "  \"model_type\": \"gpt2\",\n",
      "  \"n_ctx\": 1024,\n",
      "  \"n_embd\": 768,\n",
      "  \"n_head\": 12,\n",
      "  \"n_layer\": 12,\n",
      "  \"n_positions\": 1024,\n",
      "  \"resid_pdrop\": 0.1,\n",
      "  \"summary_activation\": null,\n",
      "  \"summary_first_dropout\": 0.1,\n",
      "  \"summary_proj_to_labels\": true,\n",
      "  \"summary_type\": \"cls_index\",\n",
      "  \"summary_use_proj\": true,\n",
      "  \"task_specific_params\": {\n",
      "    \"text-generation\": {\n",
      "      \"do_sample\": true,\n",
      "      \"max_length\": 50\n",
      "    }\n",
      "  },\n",
      "  \"vocab_size\": 50257\n",
      "}\n",
      "\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   Model name '/dfsdata2/yucc1_data/models/huggingface/gpt2' not found in model shortcut name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2). Assuming '/dfsdata2/yucc1_data/models/huggingface/gpt2' is a path, a model identifier, or url to a directory containing tokenizer files.\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/models/huggingface/gpt2/added_tokens.json. We won't load it.\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/models/huggingface/gpt2/special_tokens_map.json. We won't load it.\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/models/huggingface/gpt2/tokenizer_config.json. We won't load it.\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/models/huggingface/gpt2/tokenizer.json. We won't load it.\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   loading file /dfsdata2/yucc1_data/models/huggingface/gpt2/vocab.json\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   loading file /dfsdata2/yucc1_data/models/huggingface/gpt2/merges.txt\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:05:28 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:05:28 - INFO - __main__ -   Training new model from scratch\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Embedding(50257, 768)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加载config\n",
    "if model_args.config_name:\n",
    "    config = transformers.AutoConfig.from_pretrained(model_args.config_name, cache_dir=model_args.cache_dir)\n",
    "elif model_args.model_name_or_path:\n",
    "    config = transformers.AutoConfig.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir)\n",
    "else:\n",
    "    config = CONFIG_MAPPING[model_args.model_type]()\n",
    "    logger.warning(\"You are instantiating a new config instance from scratch.\")\n",
    "\n",
    "# 加载tokenizer\n",
    "if model_args.tokenizer_name:\n",
    "    tokenizer = transformers.AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)\n",
    "elif model_args.model_name_or_path:\n",
    "    tokenizer = transformers.AutoTokenizer.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir)\n",
    "else:\n",
    "    raise ValueError(\n",
    "        \"You are instantiating a new tokenizer from scratch. This is not supported, but you can do it from another script, save it,\"\n",
    "        \"and load it from here, using --tokenizer_name\"\n",
    "    )\n",
    "\n",
    "if model_args.model_name_or_path:\n",
    "    model = transformers.AutoModelForCausalLM.from_pretrained(\n",
    "        model_args.model_name_or_path,\n",
    "        from_tf=bool(\".ckpt\" in model_args.model_name_or_path),\n",
    "        config=config,\n",
    "        cache_dir=model_args.cache_dir,\n",
    "    )\n",
    "else:\n",
    "    logger.info(\"Training new model from scratch\")\n",
    "    model = transformers.AutoModelForCausalLM.from_config(config)\n",
    "\n",
    "# 如果tokenizer与model的embedding个数不同，设置为相同\n",
    "model.resize_token_embeddings(len(tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 校验参数，并且设置block_size\n",
    "if data_args.block_size <= 0:\n",
    "    data_args.block_size = tokenizer.max_len\n",
    "    # Our input block size will be the max possible for the model\n",
    "else:\n",
    "    data_args.block_size = min(data_args.block_size, tokenizer.max_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TextDataset(torch.utils.data.dataset.Dataset):\n",
    "    \"\"\"\n",
    "    This will be superseded by a framework-agnostic approach\n",
    "    soon.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self, tokenizer: transformers.PreTrainedTokenizer, file_path: str, block_size: int, overwrite_cache=False,\n",
    "    ):\n",
    "        assert os.path.isfile(file_path)\n",
    "\n",
    "        block_size = block_size - tokenizer.num_special_tokens_to_add(pair=False)\n",
    "\n",
    "        directory, filename = os.path.split(file_path)\n",
    "        cached_features_file = os.path.join(\n",
    "            directory, \"cached_lm_{}_{}_{}\".format(tokenizer.__class__.__name__, str(block_size), filename,),\n",
    "        )\n",
    "\n",
    "        # Make sure only the first process in distributed training processes the dataset,\n",
    "        # and the others will use the cache.\n",
    "        lock_path = cached_features_file + \".lock\"\n",
    "        with filelock.FileLock(lock_path):\n",
    "\n",
    "            if os.path.exists(cached_features_file) and not overwrite_cache:\n",
    "                start = time.time()\n",
    "                with open(cached_features_file, \"rb\") as handle:\n",
    "                    self.examples = pickle.load(handle)\n",
    "                logger.info(\n",
    "                    f\"Loading features from cached file {cached_features_file} [took %.3f s]\", time.time() - start\n",
    "                )\n",
    "\n",
    "            else:\n",
    "                logger.info(f\"Creating features from dataset file at {directory}\")\n",
    "\n",
    "                self.examples = []\n",
    "                with open(file_path, encoding=\"utf-8\") as f:\n",
    "                    text = f.read()\n",
    "\n",
    "                tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))\n",
    "\n",
    "                for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size\n",
    "                    self.examples.append(\n",
    "                        tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size])\n",
    "                    )\n",
    "                # Note that we are losing the last truncated example here for the sake of simplicity (no padding)\n",
    "                # If your dataset is small, first you should loook for a bigger one :-) and second you\n",
    "                # can change this behavior by adding (model specific) padding.\n",
    "\n",
    "                start = time.time()\n",
    "                with open(cached_features_file, \"wb\") as handle:\n",
    "                    pickle.dump(self.examples, handle, protocol=pickle.HIGHEST_PROTOCOL)\n",
    "                logger.info(\n",
    "                    \"Saving features into cached file %s [took %.3f s]\", cached_features_file, time.time() - start\n",
    "                )\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.examples)\n",
    "\n",
    "    def __getitem__(self, i) -> torch.Tensor:\n",
    "        return torch.tensor(self.examples[i], dtype=torch.long)\n",
    "\n",
    "\n",
    "class LineByLineTextDataset(torch.utils.data.dataset.Dataset):\n",
    "    \"\"\"\n",
    "    This will be superseded by a framework-agnostic approach\n",
    "    soon.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, tokenizer: transformers.PreTrainedTokenizer, file_path: str, block_size: int):\n",
    "        assert os.path.isfile(file_path)\n",
    "        # Here, we do not cache the features, operating under the assumption\n",
    "        # that we will soon use fast multithreaded tokenizers from the\n",
    "        # `tokenizers` repo everywhere =)\n",
    "        logger.info(\"Creating features from dataset file at %s\", file_path)\n",
    "\n",
    "        with open(file_path, encoding=\"utf-8\") as f:\n",
    "            lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]\n",
    "\n",
    "        batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)\n",
    "        self.examples = batch_encoding[\"input_ids\"]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.examples)\n",
    "\n",
    "    def __getitem__(self, i) -> torch.Tensor:\n",
    "        return torch.tensor(self.examples[i], dtype=torch.long)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 整理文本至同等长度\n",
    "@dataclass\n",
    "class DataCollatorForLanguageModeling:\n",
    "    \"\"\"\n",
    "    Data collator used for language modeling.\n",
    "    - collates batches of tensors, honoring their tokenizer's pad_token\n",
    "    - preprocesses batches for masked language modeling\n",
    "    \"\"\"\n",
    "\n",
    "    tokenizer: transformers.PreTrainedTokenizer\n",
    "    mlm: bool = True\n",
    "    mlm_probability: float = 0.15\n",
    "\n",
    "    def __call__(self, examples: List[torch.Tensor]) -> Dict[str, torch.Tensor]:\n",
    "        batch = self._tensorize_batch(examples)\n",
    "        if self.mlm:\n",
    "            inputs, labels = self.mask_tokens(batch)\n",
    "            return {\"input_ids\": inputs, \"labels\": labels}\n",
    "        else:\n",
    "            labels = batch.clone().detach()\n",
    "            labels[labels == self.tokenizer.pad_token_id] = -100\n",
    "            return {\"input_ids\": batch, \"labels\": labels}\n",
    "\n",
    "    def _tensorize_batch(self, examples: List[torch.Tensor]) -> torch.Tensor:\n",
    "        length_of_first = examples[0].size(0)\n",
    "        are_tensors_same_length = all(x.size(0) == length_of_first for x in examples)\n",
    "        if are_tensors_same_length:\n",
    "            return torch.stack(examples, dim=0)\n",
    "        else:\n",
    "            if self.tokenizer._pad_token is None:\n",
    "                raise ValueError(\n",
    "                    \"You are attempting to pad samples but the tokenizer you are using\"\n",
    "                    f\" ({self.tokenizer.__class__.__name__}) does not have one.\"\n",
    "                )\n",
    "            return pad_sequence(examples, batch_first=True, padding_value=self.tokenizer.pad_token_id)\n",
    "\n",
    "    def mask_tokens(self, inputs: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:\n",
    "        \"\"\"\n",
    "        Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.\n",
    "        \"\"\"\n",
    "\n",
    "        if self.tokenizer.mask_token is None:\n",
    "            raise ValueError(\n",
    "                \"This tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.\"\n",
    "            )\n",
    "\n",
    "        labels = inputs.clone()\n",
    "        # We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)\n",
    "        probability_matrix = torch.full(labels.shape, self.mlm_probability)\n",
    "        special_tokens_mask = [\n",
    "            self.tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()\n",
    "        ]\n",
    "        probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)\n",
    "        if self.tokenizer._pad_token is not None:\n",
    "            padding_mask = labels.eq(self.tokenizer.pad_token_id)\n",
    "            probability_matrix.masked_fill_(padding_mask, value=0.0)\n",
    "        masked_indices = torch.bernoulli(probability_matrix).bool()\n",
    "        labels[~masked_indices] = -100  # We only compute loss on masked tokens\n",
    "\n",
    "        # 80% of the time, we replace masked input tokens with tokenizer.mask_token ([MASK])\n",
    "        indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices\n",
    "        inputs[indices_replaced] = self.tokenizer.convert_tokens_to_ids(self.tokenizer.mask_token)\n",
    "\n",
    "        # 10% of the time, we replace masked input tokens with random word\n",
    "        indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced\n",
    "        random_words = torch.randint(len(self.tokenizer), labels.shape, dtype=torch.long)\n",
    "        inputs[indices_random] = random_words[indices_random]\n",
    "\n",
    "        # The rest of the time (10% of the time) we keep the masked input tokens unchanged\n",
    "        return inputs, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获得dataset\n",
    "def get_dataset(args: DataTrainingArguments, tokenizer: transformers.PreTrainedTokenizer, evaluate=False):\n",
    "    file_path = args.eval_data_file if evaluate else args.train_data_file\n",
    "    if args.line_by_line:\n",
    "        return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)\n",
    "    else:\n",
    "        return TextDataset(\n",
    "            tokenizer=tokenizer, file_path=file_path, block_size=args.block_size, overwrite_cache=args.overwrite_cache\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:32 - INFO - filelock -   Lock 139747928023392 acquired on /dfsdata2/yucc1_data/datasets/wikitext-2-raw/cached_lm_GPT2Tokenizer_510_wiki.train.raw.lock\n",
      "07/27/2020 19:05:32 - INFO - __main__ -   Creating features from dataset file at /dfsdata2/yucc1_data/datasets/wikitext-2-raw\n",
      "07/27/2020 19:05:47 - INFO - __main__ -   Saving features into cached file /dfsdata2/yucc1_data/datasets/wikitext-2-raw/cached_lm_GPT2Tokenizer_510_wiki.train.raw [took 0.455 s]\n",
      "07/27/2020 19:05:47 - INFO - filelock -   Lock 139747928023392 released on /dfsdata2/yucc1_data/datasets/wikitext-2-raw/cached_lm_GPT2Tokenizer_510_wiki.train.raw.lock\n",
      "07/27/2020 19:05:47 - INFO - filelock -   Lock 139743778328704 acquired on /dfsdata2/yucc1_data/datasets/wikitext-2-raw/cached_lm_GPT2Tokenizer_510_wiki.test.raw.lock\n",
      "07/27/2020 19:05:47 - INFO - __main__ -   Creating features from dataset file at /dfsdata2/yucc1_data/datasets/wikitext-2-raw\n",
      "07/27/2020 19:05:48 - INFO - __main__ -   Saving features into cached file /dfsdata2/yucc1_data/datasets/wikitext-2-raw/cached_lm_GPT2Tokenizer_510_wiki.test.raw [took 0.015 s]\n",
      "07/27/2020 19:05:48 - INFO - filelock -   Lock 139743778328704 released on /dfsdata2/yucc1_data/datasets/wikitext-2-raw/cached_lm_GPT2Tokenizer_510_wiki.test.raw.lock\n"
     ]
    }
   ],
   "source": [
    "# 获取dataset\n",
    "train_dataset = get_dataset(data_args, tokenizer=tokenizer) if training_args.do_train else None\n",
    "eval_dataset = get_dataset(data_args, tokenizer=tokenizer, evaluate=True) if training_args.do_eval else None\n",
    "data_collator = DataCollatorForLanguageModeling(\n",
    "    tokenizer=tokenizer, mlm=data_args.mlm, mlm_probability=data_args.mlm_probability\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:49 - INFO - transformers.trainer -   You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.\n"
     ]
    }
   ],
   "source": [
    "# 初始化Trainer\n",
    "trainer = transformers.Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    data_collator=data_collator,\n",
    "    train_dataset=train_dataset,\n",
    "    eval_dataset=eval_dataset,\n",
    "    prediction_loss_only=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:05:49 - INFO - transformers.trainer -   ***** Running training *****\n",
      "07/27/2020 19:05:49 - INFO - transformers.trainer -     Num examples = 4740\n",
      "07/27/2020 19:05:49 - INFO - transformers.trainer -     Num Epochs = 3\n",
      "07/27/2020 19:05:49 - INFO - transformers.trainer -     Instantaneous batch size per device = 8\n",
      "07/27/2020 19:05:49 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16\n",
      "07/27/2020 19:05:49 - INFO - transformers.trainer -     Gradient Accumulation steps = 1\n",
      "07/27/2020 19:05:49 - INFO - transformers.trainer -     Total optimization steps = 891\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72bac0d17a1d49dc8c5489e64902d91b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch', max=3.0, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f17a2f5608146bcb4084aaf5963e29c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Iteration', max=297.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "89e673e24c2345dbbd01c27a22d975a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Iteration', max=297.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:11:16 - INFO - transformers.trainer -   {'loss': 6.927489006996155, 'learning_rate': 2.1941638608305277e-05, 'epoch': 1.6835016835016834, 'step': 500}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4daab627f714455a15468c2daf5ed97",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Iteration', max=297.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:15:30 - INFO - transformers.trainer -   \n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "07/27/2020 19:15:30 - INFO - transformers.trainer -   Saving model checkpoint to /dfsdata2/yucc1_data/output/gpt2-train-new-model\n",
      "07/27/2020 19:15:30 - INFO - transformers.configuration_utils -   Configuration saved in /dfsdata2/yucc1_data/output/gpt2-train-new-model/config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:15:31 - INFO - transformers.modeling_utils -   Model weights saved in /dfsdata2/yucc1_data/output/gpt2-train-new-model/pytorch_model.bin\n"
     ]
    }
   ],
   "source": [
    "# 训练\n",
    "if training_args.do_train:\n",
    "    model_path = (\n",
    "        model_args.model_name_or_path\n",
    "        if model_args.model_name_or_path is not None and os.path.isdir(model_args.model_name_or_path)\n",
    "        else None\n",
    "    )\n",
    "    trainer.train(model_path=model_path)\n",
    "    trainer.save_model()\n",
    "    # For convenience, we also re-save the tokenizer to the same directory,\n",
    "    # so that you can share your model easily on huggingface.co/models =)\n",
    "    if trainer.is_world_master():\n",
    "        tokenizer.save_pretrained(training_args.output_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:15:31 - INFO - __main__ -   *** Evaluate ***\n",
      "07/27/2020 19:15:31 - INFO - transformers.trainer -   ***** Running Evaluation *****\n",
      "07/27/2020 19:15:31 - INFO - transformers.trainer -     Num examples = 561\n",
      "07/27/2020 19:15:31 - INFO - transformers.trainer -     Batch size = 16\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec38a66545c64a15a4490a6569b301c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Evaluation', max=36.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:15:42 - INFO - transformers.trainer -   {'eval_loss': 6.314453668064541, 'epoch': 3.0, 'step': 891}\n",
      "07/27/2020 19:15:42 - INFO - __main__ -   ***** Eval results *****\n",
      "07/27/2020 19:15:42 - INFO - __main__ -     perplexity = 552.500129656645\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# 评估\n",
    "results = {}\n",
    "if training_args.do_eval:\n",
    "    logger.info(\"*** Evaluate ***\")\n",
    "\n",
    "    eval_output = trainer.evaluate()\n",
    "\n",
    "    perplexity = math.exp(eval_output[\"eval_loss\"])\n",
    "    result = {\"perplexity\": perplexity}\n",
    "\n",
    "    output_eval_file = os.path.join(training_args.output_dir, \"eval_results_lm.txt\")\n",
    "    if trainer.is_world_master():\n",
    "        with open(output_eval_file, \"w\") as writer:\n",
    "            logger.info(\"***** Eval results *****\")\n",
    "            for key in sorted(result.keys()):\n",
    "                logger.info(\"  %s = %s\", key, str(result[key]))\n",
    "                writer.write(\"%s = %s\\n\" % (key, str(result[key])))\n",
    "\n",
    "    results.update(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/dfsdata2/yucc1_data/output/gpt2-train-new-model\n",
      "├── config.json\n",
      "├── eval_results_lm.txt\n",
      "├── merges.txt\n",
      "├── pytorch_model.bin\n",
      "├── special_tokens_map.json\n",
      "├── tokenizer_config.json\n",
      "├── training_args.bin\n",
      "└── vocab.json\n",
      "\n",
      "0 directories, 8 files\n"
     ]
    }
   ],
   "source": [
    "# 查看保存我们自己与训练的模型文件\n",
    "!tree {GPT2_OUTPUT_DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'perplexity': 552.500129656645}\n"
     ]
    }
   ],
   "source": [
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面完成了GPT2的从头开始预训练的过程，epochs为3的时候，ppl大约是， epoch为10的时候，验证集的ppl大约是297，为了更好地实际使用，实际使用的时候，建议增加epoch。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 使用预训练好的模型进行生成"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文本解码的策略有两类[7]：\n",
    "\n",
    "- Argmax Decoding: 主要包括beam search, class-factored softmax等\n",
    "- Stochastic Decoding: 主要包括temperature sampling, top-k sampling等\n",
    "\n",
    "在大多数文本生成任务中，大家都直接采用Argmax Decoding，最常见的就是beam search。但如果我们的vocabulary size较大，达到了50k甚至150k，在softmax层的运算量就会变得非常大，因为要经过softmax。有两种效率更高的方法，Class-factored Softmax和Pointer-generator Network。\n",
    "\n",
    "实际上Argmax Decoding常常会导致模型生成重复的句子，如\"I don't know. I don't know. I don't know....\"。一个可行的解决方案就是在decoding过程中引入randomness，，但是The Curious Case of Neural Text Degeneration这篇论文指出，sampling from full vocabulary distribution生成的句子会非常的杂乱无章，因为当vocabulary size非常大时，每个词的probability都会变得很小，这时模型会有非常高的可能性sample到一个tail distribution中的词，一旦sample到了tail distribution中一个和前文非常不相关的词，很有可能接下来的词都受其影响，使得句子脱离原本的意思。因此，我们需要sampling from truncated vocabulary distribution，比较常见的算法主要有以下几种：(1) Temperature Sampling，(2) Top-k Sampling，(3) Top-p Sampling.\n",
    "\n",
    "本节以最常见的beam search解码方法和top k，top p的抽样方法解码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:15:43 - INFO - transformers.configuration_utils -   loading configuration file /dfsdata2/yucc1_data/output/gpt2-train-new-model/config.json\n",
      "07/27/2020 19:15:43 - INFO - transformers.configuration_utils -   Model config GPT2Config {\n",
      "  \"activation_function\": \"gelu_new\",\n",
      "  \"architectures\": [\n",
      "    \"GPT2LMHeadModel\"\n",
      "  ],\n",
      "  \"attn_pdrop\": 0.1,\n",
      "  \"bos_token_id\": 50256,\n",
      "  \"embd_pdrop\": 0.1,\n",
      "  \"eos_token_id\": 50256,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"layer_norm_epsilon\": 1e-05,\n",
      "  \"model_type\": \"gpt2\",\n",
      "  \"n_ctx\": 1024,\n",
      "  \"n_embd\": 768,\n",
      "  \"n_head\": 12,\n",
      "  \"n_layer\": 12,\n",
      "  \"n_positions\": 1024,\n",
      "  \"resid_pdrop\": 0.1,\n",
      "  \"summary_activation\": null,\n",
      "  \"summary_first_dropout\": 0.1,\n",
      "  \"summary_proj_to_labels\": true,\n",
      "  \"summary_type\": \"cls_index\",\n",
      "  \"summary_use_proj\": true,\n",
      "  \"task_specific_params\": {\n",
      "    \"text-generation\": {\n",
      "      \"do_sample\": true,\n",
      "      \"max_length\": 50\n",
      "    }\n",
      "  },\n",
      "  \"vocab_size\": 50257\n",
      "}\n",
      "\n",
      "07/27/2020 19:15:43 - INFO - transformers.modeling_utils -   loading weights file /dfsdata2/yucc1_data/output/gpt2-train-new-model/pytorch_model.bin\n",
      "07/27/2020 19:15:46 - INFO - transformers.modeling_utils -   All model checkpoint weights were used when initializing GPT2LMHeadModel.\n",
      "\n",
      "07/27/2020 19:15:46 - INFO - transformers.modeling_utils -   All the weights of GPT2LMHeadModel were initialized from the model checkpoint at /dfsdata2/yucc1_data/output/gpt2-train-new-model.\n",
      "If your task is similar to the task the model of the ckeckpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.\n",
      "07/27/2020 19:15:46 - INFO - transformers.configuration_utils -   loading configuration file /dfsdata2/yucc1_data/output/gpt2-train-new-model/config.json\n",
      "07/27/2020 19:15:46 - INFO - transformers.configuration_utils -   Model config GPT2Config {\n",
      "  \"activation_function\": \"gelu_new\",\n",
      "  \"architectures\": [\n",
      "    \"GPT2LMHeadModel\"\n",
      "  ],\n",
      "  \"attn_pdrop\": 0.1,\n",
      "  \"bos_token_id\": 50256,\n",
      "  \"embd_pdrop\": 0.1,\n",
      "  \"eos_token_id\": 50256,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"layer_norm_epsilon\": 1e-05,\n",
      "  \"model_type\": \"gpt2\",\n",
      "  \"n_ctx\": 1024,\n",
      "  \"n_embd\": 768,\n",
      "  \"n_head\": 12,\n",
      "  \"n_layer\": 12,\n",
      "  \"n_positions\": 1024,\n",
      "  \"resid_pdrop\": 0.1,\n",
      "  \"summary_activation\": null,\n",
      "  \"summary_first_dropout\": 0.1,\n",
      "  \"summary_proj_to_labels\": true,\n",
      "  \"summary_type\": \"cls_index\",\n",
      "  \"summary_use_proj\": true,\n",
      "  \"task_specific_params\": {\n",
      "    \"text-generation\": {\n",
      "      \"do_sample\": true,\n",
      "      \"max_length\": 50\n",
      "    }\n",
      "  },\n",
      "  \"vocab_size\": 50257\n",
      "}\n",
      "\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   Model name '/dfsdata2/yucc1_data/output/gpt2-train-new-model' not found in model shortcut name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2). Assuming '/dfsdata2/yucc1_data/output/gpt2-train-new-model' is a path, a model identifier, or url to a directory containing tokenizer files.\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/output/gpt2-train-new-model/added_tokens.json. We won't load it.\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/output/gpt2-train-new-model/tokenizer.json. We won't load it.\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   loading file /dfsdata2/yucc1_data/output/gpt2-train-new-model/vocab.json\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   loading file /dfsdata2/yucc1_data/output/gpt2-train-new-model/merges.txt\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   loading file /dfsdata2/yucc1_data/output/gpt2-train-new-model/special_tokens_map.json\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   loading file /dfsdata2/yucc1_data/output/gpt2-train-new-model/tokenizer_config.json\n",
      "07/27/2020 19:15:46 - INFO - transformers.tokenization_utils_base -   loading file None\n"
     ]
    }
   ],
   "source": [
    "# 本模型与下面的模型加载，二选一即可\n",
    "# 使用刚刚预训练的模型\n",
    "model = transformers.AutoModelForCausalLM.from_pretrained(GPT2_OUTPUT_DIR)\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(GPT2_OUTPUT_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:15:47 - INFO - transformers.configuration_utils -   loading configuration file /dfsdata2/yucc1_data/models/huggingface/gpt2/config.json\n",
      "07/27/2020 19:15:47 - INFO - transformers.configuration_utils -   Model config GPT2Config {\n",
      "  \"activation_function\": \"gelu_new\",\n",
      "  \"architectures\": [\n",
      "    \"GPT2LMHeadModel\"\n",
      "  ],\n",
      "  \"attn_pdrop\": 0.1,\n",
      "  \"bos_token_id\": 50256,\n",
      "  \"embd_pdrop\": 0.1,\n",
      "  \"eos_token_id\": 50256,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"layer_norm_epsilon\": 1e-05,\n",
      "  \"model_type\": \"gpt2\",\n",
      "  \"n_ctx\": 1024,\n",
      "  \"n_embd\": 768,\n",
      "  \"n_head\": 12,\n",
      "  \"n_layer\": 12,\n",
      "  \"n_positions\": 1024,\n",
      "  \"resid_pdrop\": 0.1,\n",
      "  \"summary_activation\": null,\n",
      "  \"summary_first_dropout\": 0.1,\n",
      "  \"summary_proj_to_labels\": true,\n",
      "  \"summary_type\": \"cls_index\",\n",
      "  \"summary_use_proj\": true,\n",
      "  \"task_specific_params\": {\n",
      "    \"text-generation\": {\n",
      "      \"do_sample\": true,\n",
      "      \"max_length\": 50\n",
      "    }\n",
      "  },\n",
      "  \"vocab_size\": 50257\n",
      "}\n",
      "\n",
      "07/27/2020 19:15:47 - INFO - transformers.modeling_utils -   loading weights file /dfsdata2/yucc1_data/models/huggingface/gpt2/pytorch_model.bin\n",
      "07/27/2020 19:15:50 - INFO - transformers.modeling_utils -   All model checkpoint weights were used when initializing GPT2LMHeadModel.\n",
      "\n",
      "07/27/2020 19:15:50 - WARNING - transformers.modeling_utils -   Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at /dfsdata2/yucc1_data/models/huggingface/gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "07/27/2020 19:15:50 - INFO - transformers.configuration_utils -   loading configuration file /dfsdata2/yucc1_data/models/huggingface/gpt2/config.json\n",
      "07/27/2020 19:15:50 - INFO - transformers.configuration_utils -   Model config GPT2Config {\n",
      "  \"activation_function\": \"gelu_new\",\n",
      "  \"architectures\": [\n",
      "    \"GPT2LMHeadModel\"\n",
      "  ],\n",
      "  \"attn_pdrop\": 0.1,\n",
      "  \"bos_token_id\": 50256,\n",
      "  \"embd_pdrop\": 0.1,\n",
      "  \"eos_token_id\": 50256,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"layer_norm_epsilon\": 1e-05,\n",
      "  \"model_type\": \"gpt2\",\n",
      "  \"n_ctx\": 1024,\n",
      "  \"n_embd\": 768,\n",
      "  \"n_head\": 12,\n",
      "  \"n_layer\": 12,\n",
      "  \"n_positions\": 1024,\n",
      "  \"resid_pdrop\": 0.1,\n",
      "  \"summary_activation\": null,\n",
      "  \"summary_first_dropout\": 0.1,\n",
      "  \"summary_proj_to_labels\": true,\n",
      "  \"summary_type\": \"cls_index\",\n",
      "  \"summary_use_proj\": true,\n",
      "  \"task_specific_params\": {\n",
      "    \"text-generation\": {\n",
      "      \"do_sample\": true,\n",
      "      \"max_length\": 50\n",
      "    }\n",
      "  },\n",
      "  \"vocab_size\": 50257\n",
      "}\n",
      "\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   Model name '/dfsdata2/yucc1_data/models/huggingface/gpt2' not found in model shortcut name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2). Assuming '/dfsdata2/yucc1_data/models/huggingface/gpt2' is a path, a model identifier, or url to a directory containing tokenizer files.\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/models/huggingface/gpt2/added_tokens.json. We won't load it.\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/models/huggingface/gpt2/special_tokens_map.json. We won't load it.\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/models/huggingface/gpt2/tokenizer_config.json. We won't load it.\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   Didn't find file /dfsdata2/yucc1_data/models/huggingface/gpt2/tokenizer.json. We won't load it.\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   loading file /dfsdata2/yucc1_data/models/huggingface/gpt2/vocab.json\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   loading file /dfsdata2/yucc1_data/models/huggingface/gpt2/merges.txt\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   loading file None\n",
      "07/27/2020 19:15:50 - INFO - transformers.tokenization_utils_base -   loading file None\n"
     ]
    }
   ],
   "source": [
    "# 使用GPT2官方的预训练好的模型\n",
    "# 与上面自己的训练模型二选一加载即可\n",
    "# 自己的预训练模型并不够好，因为epoch不够多，建议使用GPT2官方模型进行生成\n",
    "# 如果想使用自己的GPT2，一个方法是微调GPT2官方模型，另一个增加epoch\n",
    "model = transformers.AutoModelForCausalLM.from_pretrained(GPT2_MODEL_NAME_OR_PATH)\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(GPT2_MODEL_NAME_OR_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:15:50 - WARNING - transformers.generation_utils -   Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "decode 0: Today the weather is really nice and I am planning on  going to go out on a limb and say that I am going to go out on a limb and say that I am going to go out on a limb and say that I am going to go out on a limb and say that I am going\n",
      "decode 1: Today the weather is really nice and I am planning on  going to go out on a limb and say that I am going to go out on a limb and say that I am going to go out on a limb and say that I am going out on a limb and say that I am going out on\n",
      "decode 2: Today the weather is really nice and I am planning on  going to go out on a limb and say that I am going to go out on a limb and say that I am going out on a limb and say that I am going out on a limb and say that I am going out on a limb\n",
      "decode 3: Today the weather is really nice and I am planning on  going to go out on a limb and say that I am going to go out on a limb and say that I am going to go out on a limb and say that I am going to go out on a limb and that I am going to\n",
      "decode 4: Today the weather is really nice and I am planning on  going to go out on a limb and say that I am going to go out on a limb and say I am going to go out on a limb and say I am going to go out on a limb and say I am going to go out\n"
     ]
    }
   ],
   "source": [
    "# 解码示例一：\n",
    "# 初始文本\n",
    "prompt = \"Today the weather is really nice and I am planning on \"\n",
    "# 转换成tensor\n",
    "inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors=\"pt\")\n",
    "# 使用beam search生成\n",
    "# max_length是最长长度，50表示在输入文本的后面最多生成50个字\n",
    "beam_outputs = model.generate(input_ids=inputs,\n",
    "           max_length=50+inputs.shape[-1],\n",
    "           min_length=2+inputs.shape[-1],\n",
    "           num_beams=20,\n",
    "           num_return_sequences=5,)\n",
    "for i in range(5):\n",
    "    output_ids = beam_outputs[i].tolist()\n",
    "    text = tokenizer.decode(output_ids, skip_special_tokens=True)\n",
    "    print(f'decode {i}: {text}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:16:03 - WARNING - transformers.generation_utils -   Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n",
      "07/27/2020 19:16:05 - WARNING - transformers.generation_utils -   Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Today the weather is really nice and I am planning on iced tea tomorrow night (hopefully) and on Sunday (hopefully).\n",
      "So, this is a little teaser for tomorrow, but I will be making 2 more teas in 1 day!!  (I do not mean only all tea/\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:16:07 - WARNING - transformers.generation_utils -   Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Today the weather is really nice and I am planning on  going for a few days a week to do some work in a field, so it is going to be just fine, but I'm not going to try and start new projects because I'm feeling like I'm not living in the same house\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:16:10 - WARNING - transformers.generation_utils -   Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Today the weather is really nice and I am planning on ices having a nice day.\"\n",
      "\n",
      "— Chris's family, after a long hot summer on the town of Ritz-Carlton (Ritz-Carlton News, August 4, 2001)\n",
      "\n",
      "Photo by Flickr user mike.k\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/27/2020 19:16:12 - WARNING - transformers.generation_utils -   Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Today the weather is really nice and I am planning on  working for the next day and  the next night...so this is definitely a good day for any weather related stuff. I know, I know I should just work my way through the morning and just go to bed. But this is\n",
      "Today the weather is really nice and I am planning on icing this up later this year. Thank you so much,\n",
      "\n",
      "James<|endoftext|>\n"
     ]
    }
   ],
   "source": [
    "# 解码示例二：\n",
    "# 初始文本\n",
    "prompt = \"Today the weather is really nice and I am planning on \"\n",
    "# 转换成tensor\n",
    "inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors=\"pt\")\n",
    "# 给定tensor，生成新的tensor\n",
    "# 此处需要注意的是max_length包含着前面的prompt的长度，也就是前面的长度非常长，超过250，就无法生成了\n",
    "# top_p是概率从大到小排列，最小的个数达到0.95，后面的砍掉不使用；top_k是选择概率最大的60个，后面的砍掉不使用；\n",
    "# 综合起来就是概率相加不超过0.95且最多60个的单词去生成。\n",
    "for i in range(5):\n",
    "    outputs = model.generate(inputs, \n",
    "                             max_length=50+inputs.shape[-1], \n",
    "                             do_sample=True, \n",
    "                             top_p=0.95, \n",
    "                             top_k=60)\n",
    "    generated = tokenizer.decode(outputs[0])\n",
    "    print(generated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "已经完成了两种不同解码方式的结果，可以看到beam search的结果相对重复性比较多。\n",
    "\n",
    "到这里，基于GPT2的重头预训练，并且进行线上预测已经完成。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第六节 总结、思考与展望\n",
    "\n",
    "上面已经完成了基于huggingface/transformers库在文本分类的训练与部署，文本生成的语言模型训练与部署。上面的代码也是一种标准的pytorch模型范式。\n",
    "\n",
    "一些其他的思考：\n",
    "\n",
    "1. 开源代码及工具报的书写方式。setup.py用于pip包写作；tests是测试样例；同名的文件夹下是代码。\n",
    "2. 所有输出日志非常重要且有用。\n",
    "3. typing, dataclasses, classmethod等方法的使用。\n",
    "4. 开源库的学习。第一步，看文档，跑官方examples：第二步，修改代码，仿写自己的代码；第三步，重复前两步；第四步，看源码，修改源码。\n",
    "5. 通过学习源码，学到了很多，希望后续对源码更加清楚，使用更加熟练；希望能在工作、学习、科研上对自己、对大家更有帮助。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第七节 相关网址\n",
    "1. github代码：https://github.com/huggingface/transformers\n",
    "2. doc说明：https://huggingface.co/transformers/index.html\n",
    "3. 模型下载地址：https://huggingface.co/models\n",
    "4. 利用transformers开发的中文chitchat生成：https://github.com/yangjianxin1/GPT2-chitchat\n",
    "5. DialoGPT：https://github.com/microsoft/DialoGPT\n",
    "6. glue数据集介绍：https://zhuanlan.zhihu.com/p/135283598\n",
    "7. 文本解码策略：https://zhuanlan.zhihu.com/p/68383015\n",
    "8. Trainer说明：https://huggingface.co/transformers/main_classes/trainer.html\n",
    "9. WikiText数据集：https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}