Add MPNet Model #869
Merged
Add MPNet Model #869
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gongel
requested changes
Aug 31, 2021
| [MPNet: Masked and Permuted Pre-training for Language Understanding - Microsoft Research](https://www.microsoft.com/en-us/research/publication/mpnet-masked-and-permuted-pre-training-for-language-understanding/) | ||
|
|
||
| **摘要:** | ||
| BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pretraining to address this problem. However, XLNet does not leverage the full position information of a sentence and thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g., BERT, XLNet, RoBERTa) under the same model setting. The code and the pre-trained models are available at: https://github.com/microsoft/MPNet. |
|
|
||
| ## 快速开始 | ||
|
|
||
| ### 模型精度对齐 |
| @@ -0,0 +1,50 @@ | |||
| # Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved. | |||
gongel
requested changes
Sep 10, 2021
| pretrained_resource_files_map = { | ||
| "vocab_file": { | ||
| "mpnet-base": | ||
| "https://paddlenlp.bj.bcebos.com/models/transformers/mpnet/mpnet-base/vocab.txt", |
There was a problem hiding this comment.
另外Tokenizer默认会返回token_type_ids，但是这个在MPNet是不需要的
| embedding_output = self.embeddings(input_ids, position_ids) | ||
|
|
||
| encoder_outputs, _ = self.encoder(embedding_output, | ||
| extended_attention_mask) |
There was a problem hiding this comment.
extended_attention_mask --->attention_mask
| mpnet (:class:`MPNetModel`): | ||
| An instance of MPNetModel. | ||
| num_classes (int, optional): | ||
| The number of classes. Defaults to `2`. |
|
好的，过会改 |
gongel
previously approved these changes
Sep 11, 2021
| return len(cls + token_ids_0 + sep) * [0] | ||
| return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0] | ||
|
|
||
| def __call__(self, |
gongel
reviewed
Sep 11, 2021
| @@ -0,0 +1,903 @@ | |||
| # Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved. | |||
| # Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team. | |||
gongel
approved these changes
Sep 11, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
飞桨论文复现挑战赛（第四期）MPNet: Masked and Permuted Pre-training for Language Understanding 论文复现提交。