Skip to content
No description or website provided.
Branch: master
Clone or download
Latest commit 00c5425 Jul 14, 2019

README.md

UER-py

Build Status codebeat badge

Pre-training has become an essential part for NLP tasks and has led to remarkable improvements. UER-py is a toolkit for pre-training on general-domain corpus and fine-tuning on downstream task. UER-py maintains model modularity and supports research extensibility. It facilitates the use of different pre-training models (e.g. BERT), and provides interfaces for users to further extend upon. UER-py also incorporates many mechanisms for better performance and efficiency. It has been tested on several Chinese datasets and should match or even outperform Google's TF implementation.

Update: Now ELMO (bilstm encoder + bilm target) is supported by UER. ELMO model pre-trained on mixed large corpus will be released soon. Word-based BERT is now available. Context-dependent word embedding (trained by BERT) is in particular suitable for polysemous words.


Table of Contents


Features

UER-py has the following features:

  • Reproducibility. UER-py is able to reproduce the results of existing pre-training models (such as Google BERT).
  • Multi-GPU. UER-py supports CPU mode, single GPU mode, and distributed training mode.
  • Model modularity. UER-py is divided into multiple components: subencoder, encoder, target, and fine-tuning. Ample modules are implemented in each component. Clear and robust interface allows users to combine modules with as few restrictions as possible.
  • Efficiency. UER-py refines its pre-processing, pre-training, and fine-tuning stages, which largely improves speed and needs less memory.
  • SOTA results. Our works further improve the results upon Google BERT, providing new baselines for a range of datasets.
  • Chinese model zoo. We are pre-training models with different corpora, encoders, and targets. Selecting proper pre-training models is beneficial to the performance of downstream tasks.

Requirements

Python3.6 torch>=1.0 argparse


Quickstart

We use BERT model and Douban book review classification dataset to demonstrate how to use UER-py. We firstly pre-train model on book review corpus and then fine-tune it on classification dataset. There are three input files: book review corpus, book review dataset, and vocabulary. All files are encoded in UTF-8 and are included in this project.

The format of the corpus for BERT is as follows:

doc1-sent1
doc1-sent2
doc1-sent3

doc2-sent1

doc3-sent1
doc3-sent2

The book review corpus is obtained by book review dataset. We remove labels and split a review into two parts from the middle (See book_review_bert.txt in corpora folder).

The format of the classification dataset is as follows (label and instance are separated by \t):

label    text_a
1        instance1
0        instance2
1        instance3

We use Google's Chinese vocabulary file, which contains 21128 Chinese characters. The format of the vocabulary is as follows:

word-1
word-2
...
word-n

First of all, we preprocess the book review corpus. We need to specify the model's target in pre-processing stage (--target):

python3 preprocess.py --corpus_path corpora/book_review_bert.txt --vocab_path models/google_vocab.txt --dataset_path dataset.pt \
                      --processes_num 8 --target bert

Pre-processing is time-consuming. Multi-process can largely accelerate the pre-processing speed (--processes_num). The raw text is converted to dataset.pt, which is the input of pretrain.py. Then we download Google's pre-trained Chinese model, and put it into models folder. We load Google's pre-trained model and train it on book review corpus. We should better explicitly specify model's encoder (--encoder) and target (--target). Suppose we have a machine with 8 GPUs.:

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --pretrained_model_path models/google_model.bin \
                    --output_model_path models/book_review_model.bin  --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 20000 --save_checkpoint_steps 5000 --encoder bert --target bert

mv models/book_review_model.bin-20000 models/book_review_model.bin

Notice that the model trained by pretrain.py is attacted with the suffix which records the training step. We could remove the suffix for ease of use. Finally, we do classification. We can use google_model.bin:

python3 classifier.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                      --train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
                      --epochs_num 3 --batch_size 32 --encoder bert

or use our book_review_model.bin, which is the output of pretrain.py:

python3 classifier.py --pretrained_model_path models/book_review_model.bin --vocab_path models/google_vocab.txt \
                      --train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
                      --epochs_num 3 --batch_size 32 --encoder bert

It turns out that the result of Google's model is 87.5; The result of book_review_model.bin is 88.1. It is also noticable that we don't need to specify the target in fine-tuning stage. Pre-training target is replaced with task-specific target.

BERT consists of next sentence prediction (NSP) target. However, NSP target is not suitable for sentence-level reviews since we have to split a review into two parts. UER-py facilitates the use of different targets. Using masked language modeling (MLM) as target could be a properer choice for pre-training of reviews:

python3 preprocess.py --corpus_path corpora/book_review.txt --vocab_path models/google_vocab.txt --dataset_path dataset.pt \
                      --processes_num 8 --target mlm

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --pretrained_model_path models/google_model.bin \
                    --output_model_path models/book_review_mlm_model.bin  --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 20000 --save_checkpoint_steps 5000 --encoder bert --target mlm

mv models/book_review_mlm_model.bin-20000 models/book_review_mlm_model.bin

python3 classifier.py --pretrained_model_path models/book_review_mlm_model.bin --vocab_path models/google_vocab.txt \
                      --train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
                      --epochs_num 3 --batch_size 32 --encoder bert

It turns out that the result of book_review_mlm_model.bin is 88.3.

We could search proper pre-trained models in Chinese model zoo for further improvements. For example, we could download a model pre-trained on Amazon corpus (over 4 million reviews) with BERT encoder and classification (CLS) target. It achieves 88.5 accuracy on book review dataset.

BERT is really slow. It could be great if we can speed up the model and still achieve competitive performance. We select a 2-layers LSTM encoder to substitute 12-layers Transformer encoder. We could download a model pre-trained with LSTM encoder and language modeling (LM) + classification (CLS) targets:

python3 classifier.py --pretrained_model_path models/lstm_reviews_model.bin --vocab_path models/google_vocab.txt \
                      --train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
                      --epochs_num 3  --batch_size 64 --encoder lstm --pooling mean --config_path models/rnn_config.json --learning_rate 1e-3

We can achieve 86.5 accuracy on testset, which is also a competitive result. Using LSTM without pre-training can only achieve 80.2 accuracy. In practice, above model is around 10 times faster than BERT. One can see Chinese model zoo section for more detailed information about above pre-trained LSTM model.

Besides classification, UER-py also provides scripts for other downstream tasks. We could use tagger.py for sequence labeling:

python3 tagger.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                  --train_path datasets/msra/train.tsv --dev_path datasets/msra/dev.tsv --test_path datasets/msra/test.tsv \
                  --epochs_num 5 --batch_size 16 --encoder bert

We could download a model pre-trained on RenMinRiBao (as known as People's Daily, a news corpus) and finetune on it:

python3 tagger.py --pretrained_model_path models/rmrb_model.bin --vocab_path models/google_vocab.txt \
                  --train_path datasets/msra/train.tsv --dev_path datasets/msra/dev.tsv --test_path datasets/msra/test.tsv \
                  --epochs_num 5 --batch_size 16 --encoder bert

It turns out that the result of Google's model is 92.6; The result of rmrb_model.bin is 94.4.


Datasets

This project includes a range of Chinese datasets. Small-scale datasets can be downloaded at datasets_zh.zip. datasets_zh.zip contains 7 datasets: XNLI, LCQMC, MSRA-NER, ChnSentiCorp, and nlpcc-dbqa are obtained from Baidu ERNIE; Book review (from BNU) and Shopping are two sentiment analysis datasets. Large-scale datasets can be found in glyph's github project.


Instructions

UER-py's framework

UER-py is organized as follows:

UER-py/
    |--uer/
    |    |--encoders/: contains encoders such as RNN, CNN, Attention, CNN-RNN, BERT
    |    |--targets/: contains targets such as language modeling, masked language modeling, sentence prediction
    |    |--subencoders/: contains subencoders such as RNN, CNN, and different pooling strategies
    |    |--layers/: contains frequently-used NN layers, such as embedding layer, normalization layer
    |    |--models/: contains model.py, which combines subencoder, embedding, encoder, and target modules
    |    |--utils/: contains frequently-used utilities
    |    |--model_builder.py 
    |    |--model_saver.py
    |    |--trainer.py
    |
    |--corpora/: contains corpora for pre-training
    |--datasets/: contains downstream tasks
    |--models/: contains pre-trained models, vocabularies, and config files
    |--scripts/: contains some useful scripts for pre-training models
    |
    |--preprocess.py
    |--pretrain.py
    |--classifier.py
    |--cloze.py
    |--tagger.py
    |--feature_extractor.py
    |--README.md

The code is well-organized. Users can use and extend upon it with little efforts.

Preprocess the data

usage: preprocess.py [-h] --corpus_path CORPUS_PATH --vocab_path VOCAB_PATH
                     [--dataset_path DATASET_PATH]
                     [--tokenizer {bert,char,space}]
                     [--processes_num PROCESSES_NUM]
                     [--target {bert,lm,cls,mlm,nsp,s2s}]
                     [--docs_buffer_size DOCS_BUFFER_SIZE]
                     [--instances_buffer_size INSTANCES_BUFFER_SIZE]
                     [--seq_length SEQ_LENGTH] [--dup_factor DUP_FACTOR]
                     [--short_seq_prob SHORT_SEQ_PROB] [--seed SEED]

--docs_buffer_size and --instances_buffer_size could be used to control memory consumption in pre-processing and pre-training stages. --preprocesses_num n denotes that n processes are used for pre-processing. The example of pre-processing on a single machine is as follows:

python3 preprocess.py --corpus_path corpora/book_review_bert.txt --vocab_path models/google_vocab.txt \
                      --dataset_path dataset.pt --processes_num 8 --target bert

We need to specify the model's target in pre-processing stage since different targets require different data formats. Currently, UER-py consists of the following target modules:

  • lm_target.py: language model
  • mlm_target.py: masked language model (cloze test)
  • nsp_target.py: next sentence prediction
  • cls_target.py: classification
  • s2s_target.py: supports autoencoder and machine translation
  • bert_target.py: masked language model + next sentence prediction

If multiple machines are available, each machine contains a part of corpus. The command is identical with the single machine case.

Pretrain the model

usage: pretrain.py [-h] [--dataset_path DATASET_PATH] --vocab_path VOCAB_PATH
                   [--pretrained_model_path PRETRAINED_MODEL_PATH]
                   --output_model_path OUTPUT_MODEL_PATH
                   [--config_path CONFIG_PATH] [--total_steps TOTAL_STEPS]
                   [--save_checkpoint_steps SAVE_CHECKPOINT_STEPS]
                   [--report_steps REPORT_STEPS]
                   [--accumulation_steps ACCUMULATION_STEPS]
                   [--batch_size BATCH_SIZE]
                   [--emb_size EMB_SIZE] [--hidden_size HIDDEN_SIZE]
                   [--feedforward_size FEEDFORWARD_SIZE]
                   [--kernel_size KERNEL_SIZE] [--heads_num HEADS_NUM]
                   [--layers_num LAYERS_NUM] [--dropout DROPOUT] [--seed SEED]
                   [--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
                   [--bidirectional] [--target {bert,lm,cls,mlm,nsp,s2s}]
                   [--labels_num LABELS_NUM] [--learning_rate LEARNING_RATE]
                   [--warmup WARMUP] [--world_size WORLD_SIZE]
                   [--gpu_ranks GPU_RANKS [GPU_RANKS ...]]
                   [--master_ip MASTER_IP] [--backend {nccl,gloo}]

Notice that it is recommended to explicitly specify model's encoder and target. UER-py consists of the following encoder modules:

  • rnn_encoder.py: contains (bi-)LSTM and (bi-)GRU
  • cnn_encoder.py: contains CNN and gatedCNN
  • attn_encoder.py: contains attentionNN
  • gpt_encoder.py: contains GPT encoder
  • bert_encoder.py: contains BERT encoder
  • mixed_encoder.py: contains combinations of basic encoders, such as RCNN (RNN+CNN), CRNN (CNN+RNN)

The target should be coincident with the target in pre-processing stage. Users can try different combinations of encoders and targets by --encoder and --target.

There are two strategies for pre-training: 1)random initialization 2)loading a pre-trained model.

Random initialization

The example of pre-training on CPU:

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --output_model_path models/output_model.bin --encoder bert --target bert

The example of pre-training on single GPU (the id of GPU is 3):

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --output_model_path models/output_model.bin --encoder bert --target bert --gpu_ranks 3

The example of pre-training on a single machine with 8 GPUs:

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
                    --output_model_path models/output_model.bin --encoder bert --target bert --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 

The example of pre-training on two machines, each has 8 GPUs (16 GPUs in total):

Node-0 : python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
            --output_model_path models/output_model.bin --encoder bert --target bert --world_size 16 --gpu_ranks 0 1 2 3 4 5 6 7 \
            --master_ip tcp://node-0-addr:port
Node-1 : python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
            --output_model_path models/output_model.bin --encoder bert --target bert --world_size 16 --gpu_ranks 8 9 10 11 12 13 14 15 \
            --master_ip tcp://node-0-addr:port            

Load a pre-trained model

We recommend to load a pre-trained model. We can specify the pre-trained model by --pretrained_model_path . The example of pre-training on CPU and single GPU:

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
                    --pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
                    --encoder bert --target bert
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
                    --pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
                    --encoder bert --target bert --gpu_ranks 3

The example of pre-training on a single machine with 8 GPUs:

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
                    --pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
                    --encoder bert --target bert --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 

The example of pre-training on two machines, each has 8 GPUs (16 GPUs in total):

Node-0 : python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
            --pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
            --encoder bert --target bert --world_size 16 --gpu_ranks 0 1 2 3 4 5 6 7 --master_ip tcp://node-0-addr:port
Node-1 : python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
            --pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
            --encoder bert --target bert --world_size 16 --gpu_ranks 8 9 10 11 12 13 14 15 --master_ip tcp://node-0-addr:port

Try pre-training models with different targets and encoders

UER-py allows users to combine different components (e.g. subencoders, encoders, and targets). Here is an example of trying different targets:

In fact, NSP target and sentence-level reviews are incompatible to some extent. We could replace BERT target with MLM target on book review dataset:

python3 preprocess.py --corpus_path corpora/book_review.txt --vocab_path models/google_vocab.txt --dataset_path dataset.pt --processes_num 8 --target mlm

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 --total_steps 20000 --save_checkpoint_steps 5000 --encoder bert --target mlm

Notice that different targets correspond to different corpus formats. It is important to select proper format for a target. If we want to change encoder, only thing we need to do is to specify --encoder in pretrain.py. Here is an example of using LSTM for pre-training.

python3 preprocess.py --corpus_path corpora/book_review.txt --vocab_path models/google_vocab.txt --dataset_path dataset.pt --processes_num 8 --target lm

python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --output_model_path models/output_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 --total_steps 20000 --save_checkpoint_steps 5000 \ 
                    --encoder lstm --target lm --learning_rate 1e-3 --config_path models/rnn_config.json

Fine-tune on downstream tasks

Currently, UER-py consists of 4 downstream tasks, i.e. classification, sequence labeling, cloze test, feature extractor. The encoder of downstream task should be coincident with the pre-trained model.

Classification

classifier.py adds two feedforward layers upon encoder layer.

usage: classifier.py [-h] [--pretrained_model_path PRETRAINED_MODEL_PATH]
                     [--output_model_path OUTPUT_MODEL_PATH]
                     [--vocab_path VOCAB_PATH] --train_path TRAIN_PATH
                     --dev_path DEV_PATH --test_path TEST_PATH
                     [--config_path CONFIG_PATH] [--batch_size BATCH_SIZE]
                     [--seq_length SEQ_LENGTH]
                     [--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
                     [--bidirectional] [--pooling {mean,max,first,last}]
                     [--subword_type {none,char}]
                     [--sub_vocab_path SUB_VOCAB_PATH]
                     [--subencoder {avg,lstm,gru,cnn}]
                     [--sub_layers_num SUB_LAYERS_NUM]
                     [--tokenizer {bert,char,word,space}]
                     [--learning_rate LEARNING_RATE] [--warmup WARMUP]
                     [--dropout DROPOUT] [--epochs_num EPOCHS_NUM]
                     [--report_steps REPORT_STEPS] [--seed SEED]
                     [--mean_reciprocal_rank]

The example of using classifier.py:

python3 classifier.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                      --train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
                      --epochs_num 3 --batch_size 64 --encoder bert

The example of using classifier.py for pair classification:

python3 classifier.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                      --train_path datasets/lcqmc/train.tsv --dev_path datasets/lcqmc/dev.tsv --test_path datasets/lcqmc/test.tsv \
                      --epochs_num 3 --batch_size 64 --encoder bert

The example of using classifier.py for dbqa:

python3 classifier.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                      --train_path datasets/dbqa/train.tsv --dev_path datasets/dbqa/dev.tsv --test_path datasets/dbqa/test.tsv \
                      --epochs_num 3 --batch_size 64 --encoder bert --mean_reciprocal_rank

Sequence labeling

tagger.py adds a feedforward layer upon encoder layer.

usage: tagger.py [-h] [--pretrained_model_path PRETRAINED_MODEL_PATH]
                 [--output_model_path OUTPUT_MODEL_PATH]
                 [--vocab_path VOCAB_PATH] [--train_path TRAIN_PATH]
                 [--dev_path DEV_PATH] [--test_path TEST_PATH]
                 [--config_path CONFIG_PATH] [--batch_size BATCH_SIZE]
                 [--seq_length SEQ_LENGTH]
                 [--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
                 [--bidirectional] [--subword_type {none,char}]
                 [--sub_vocab_path SUB_VOCAB_PATH]
                 [--subencoder {avg,lstm,gru,cnn}]
                 [--sub_layers_num SUB_LAYERS_NUM]
                 [--learning_rate LEARNING_RATE] [--warmup WARMUP]
                 [--dropout DROPOUT] [--epochs_num EPOCHS_NUM]
                 [--report_steps REPORT_STEPS] [--seed SEED]

The example of using tagger.py:

python3 tagger.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                  --train_path datasets/msra/train.tsv --dev_path datasets/msra/dev.tsv --test_path datasets/msra/test.tsv \
                  --epochs_num 5 --batch_size 32 --encoder bert

Cloze test

cloze.py predicts masked words. Top n words are returned.

usage: cloze.py [-h] [--pretrained_model_path PRETRAINED_MODEL_PATH]
                [--vocab_path VOCAB_PATH] [--input_path INPUT_PATH]
                [--output_path OUTPUT_PATH] [--config_path CONFIG_PATH]
                [--batch_size BATCH_SIZE] [--seq_length SEQ_LENGTH]
                [--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
                [--bidirectional] [--target {bert,lm,cls,mlm,nsp,s2s}]
                [--subword_type {none,char}] [--sub_vocab_path SUB_VOCAB_PATH]
                [--subencoder_type {avg,lstm,gru,cnn}]
                [--tokenizer {bert,char,word,space}] [--topn TOPN]

The example of using cloze.py:

python3 cloze.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                 --input_path datasets/cloze_input.txt --output_path output.txt

Feature extractor

feature_extractor.py extracts sentence embeddings.

usage: feature_extractor.py [-h] --input_path INPUT_PATH --model_path
                            MODEL_PATH --vocab_path VOCAB_PATH --output_path
                            OUTPUT_PATH [--subword_type {none,char}]
                            [--sub_vocab_path SUB_VOCAB_PATH]
                            [--subencoder {avg,lstm,gru,cnn}]
                            [--sub_layers_num SUB_LAYERS_NUM]
                            [--seq_length SEQ_LENGTH]
                            [--batch_size BATCH_SIZE]
                            [--config_path CONFIG_PATH]
                            [--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
                            [--target {bert,lm,cls,mlm,nsp,s2s}]
                            [--tokenizer {char,word,space,mixed}]

The example of using feature_extractor.py:

python3 feature_extractor.py --input_path datasets/cloze_input.txt --pretrained_model_path models/google_model.bin \
                             --vocab_path models/google_vocab.txt --output_path output.npy

Finding nearest neighbours

Pre-trained models can learn high-quality word embeddings. Traditional word embeddings such as word2vec and GloVe assign each word a fixed vector. However, polysemy is a pervasive phenomenon in human language, and the meanings of a polysemous word depend on the context. To this end, we use a the hidden state in pre-trained models to represent a word. It is noticeable that Google BERT is a character-based model. To obtain real word embedding (not character embedding), Users should download our word-based BERT model and vocabulary. The example of using scripts/topn_words_indep.py (finding nearest neighbours for context-independent word embedding):

python3 scripts/topn_words_indep.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                                    --cand_vocab_path models/google_vocab.txt --target_words_path target_characters.txt

Contexct-independent word embedding is obtained model's embedding layer. The format of the target_characters.txt is as follows:

word-1
word-2
...
word-n

The example of using scripts/topn_words_dep.py (finding nearest neighbours for context-dependent word embedding):

python3 scripts/topn_words_dep.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
                                  --cand_vocab_path models/google_vocab.txt --sent_path target_characters_with_sentences.txt --config_path models/google_config.json \
                                  --batch_size 256 --seq_length 32 --tokenizer bert

We substitute the target word with other words in the vocabulary and feed the sentences into the pretrained model. Hidden state is used as the context-dependent embedding of a word. Select proper tokenizer according to the sentence in target_characters_with_sentences.txt. The format of target_words_with_sentences.txt is as follows:

sent1 word1
sent1 word1
...
sentn wordn

Sentence and word are splitted by \t.

Scripts

Scripts Function description
average_model.py Take the average of pre-trained models. A frequently-used ensemble strategy for deep learning models
build_vocab.py Build vocabulary (multi-processing supported)
check_model.py Check the model (single GPU or multiple GPUs)
diff_vocab.py Compare two vocabularies
dynamic_vocab_adapter.py Change the pre-trained model according to the vocabulary. It can save memory in fine-tuning stage since task-specific vocabulary is much smaller than general-domain vocabulary
multi_single_convert.py convert the model (single GPU or multiple GPUs)
topn_words_indep.py Finding nearest neighbours with context-independent word embedding
topn_words_dep.py Finding nearest neighbours with context-dependent word embedding

Experiments

Speed

GPU:Tesla P40

CPU:Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

We use BERT to test the speed of distributed training mode. Google BERT is trained for 1 million steps and each step contains 128,000 tokens. It takes around 18 days to reproduce the experiments by UER-py on 3 GPU machines (24 GPU in total).

#(machine) #(GPU)/machine tokens/second
1 0 276
1 1 7050
1 2 13071
1 4 24695
1 8 44300
3 8 84386

Qualitative evaluation

We qualitatively evaluate pre-trained models by finding words' near neighbours.

Character-based model

Evaluation of context-independent word embedding:

Target word: 苹 Target word: 吃 Target word: 水
0.762 0.539 0.286
apple 0.447 0.475 0.278
iphone 0.400 0.340 water 0.276
0.347 0.324 0.266
ios 0.317 0.322 0.259

Evaluation of context-dependent word embedding:

Target sentence: 其冲积而形成小平原沙土层厚而肥沃,盛产苹果、大樱桃、梨和葡萄。

Target word: 苹
0.822
0.714
0.706
0.704
0.696

Target sentence: 苹果削减了台式Mac产品线上的众多产品

Target word: 苹
0.892
apple 0.788
iphone 0.743
ios 0.720
ipad 0.706

Word-based model

Evaluation of context-independent word embedding:

Target word: 苹果 Target word: 腾讯 Target word: 吉利
苹果公司 0.419 新浪 0.357 沃尔沃 0.277
apple 0.415 网易 0.356 伊利 0.243
苹果电脑 0.349 搜狐 0.356 长荣 0.235
微软 0.320 百度 0.341 天安 0.224
mac 0.298 乐视 0.332 哈达 0.220

Evaluation of context-dependent word embedding:

Target sentence: 其冲积而形成小平原沙土层厚而肥沃,盛产苹果、大樱桃、梨和葡萄。

Target word: 苹果
柠檬 0.734
草莓 0.725
荔枝 0.719
树林 0.697
牡丹 0.686

Target sentence: 苹果削减了台式Mac产品线上的众多产品

Target word: 苹果
苹果公司 0.836
apple 0.829
福特 0.796
微软 0.777
苹果电脑 0.773

Target sentence: 讨吉利是通过做民间习俗的吉祥事,或重现过去曾经得到好结果的行为,以求得好兆头。

Target word: 吉利
仁德 0.749
光彩 0.743
愉快 0.736
永元 0.736
仁和 0.732

Target sentence: 2010年6月2日福特汽车公司宣布出售旗下高端汽车沃尔沃予中国浙江省的吉利汽车,同时将于2010年第四季停止旗下中阶房车品牌所有业务

Target word: 吉利
沃尔沃 0.771
卡比 0.751
永利 0.745
天安 0.741
仁和 0.741

Target sentence: 主要演员有扎克·布拉夫、萨拉·朝克、唐纳德·费森、尼尔·弗林、肯·詹金斯、约翰·麦吉利、朱迪·雷耶斯、迈克尔·莫斯利等。

Target word: 吉利
玛利 0.791
米格 0.768
韦利 0.767
马力 0.764
安吉 0.761

Quantitative evaluation

We use a range of Chinese datasets to evaluate the performance of UER-py. Douban book review, ChnSentiCorp, Shopping, and Tencentnews are sentence-level small-scale sentiment classification datasets. MSRA-NER is a sequence labeling dataset. These datasets are included in this project. Dianping, JDfull, JDbinary, Ifeng, and Chinanews are large-scale classification datasets. They are collected in glyph and can be downloaded at glyph's github project. These five datasets don't contain validation set. We use 10% instances in trainset for validation.

Most pre-training models consist of 2 stages: pre-training on general-domain corpus and fine-tuning on downstream dataset. We recommend 3-stage mode: 1)Pre-training on general-domain corpus; 2)Pre-training on downstream dataset; 3)Fine-tuning on downstream dataset. Stage 2 enables models to get familiar with distributions of downstream tasks. It is sometimes known as semi-supervised fune-tuning.

Hyper-parameter settings are as follows:

  • Stage 1: We train with batch size of 256 sequences and each sequence contains 256 tokens. We load Google's pretrained models and train upon it for 500,000 steps. The learning rate is 2e-5 and other optimizer settings are identical with Google BERT. BERT tokenizer is used.
  • Stage 2: We train with batch size of 256 sequences. For classification datasets, the sequence length is 128. For sequence labeling datasets, the sequence length is 256. We train upon Google's pretrained model for 20,000 steps. Optimizer settings and tokenizer are identical with stage 1.
  • Stage 3: For classification datasets, the training batch size and epochs are 64 and 3. For sequence labeling datasets, the training batch size and epochs are 32 and 5. Optimizer settings and tokenizer are identical with stage 1.

We provide the pre-trained models (using BERT target) on different downstream datasets: book_review_model.bin; chnsenticorp_model.bin; shopping_model.bin; msra_model.bin. Tencentnews dataset and its pretrained model will be publicly available after data desensitization.

Model/Dataset Douban book review ChnSentiCorp Shopping MSRA-NER Tencentnews review
BERT 87.5 94.3 96.3 93.0/92.4/92.7 84.2
BERT+semi_BertTarget 88.1 95.6 97.0 94.3/92.6/93.4 85.1
BERT+semi_MlmTarget 87.9 95.5 97.1 85.1

Pre-training is also important for other encoders and targets. We pre-train a 2-layer LSTM on 1.9G review corpus with language model target. Embedding size and hidden size are 512. The model is much more efficient than BERT in pre-training and fine-tuning stages. We show that pre-training brings significant improvements and achieves competitive results (the differences are not big compared with the results of BERT).

Model/Dataset Douban book review ChnSentiCorp Shopping
BERT 87.5 94.3 96.3
LSTM 80.2 88.3 94.4
LSTM+pre-training 86.6(+6.4) 94.5(+6.2) 96.5(+2.1)

It requires tremendous computional resources to fine-tune on large-scale datasets. For Ifeng, Chinanews, Dianping, JDbinary, and JDfull datasets, we provide their classification models (see Chinese model zoo). Classification models on large-scale datasets allow users to reproduce the results without training. Besides that, classification models could be used for improving other related tasks. More experimental results will come soon.

Ifeng and Chinanews datasets contain news' titles and abstracts. In stage 2, we use title to predict abstract.

Model/Dataset Ifeng Chinanews Dianping JDbinary JDfull
pre-SOTA (Glyph & Glyce) 85.76 91.88 78.46 91.76 54.24
BERT 87.50 93.37 92.37 54.79
BERT+semi+BertTarget 87.65

We also provide the pre-trained models on different corpora, encoders, and targets (see Chinese model zoo). Selecting proper pre-training models is beneficial to the performance of downstream tasks.

Model/Dataset MSRA-NER
Wikizh corpus (Google) 93.0/92.4/92.7
Renminribao corpus 94.4/94.4/94.4

Chinese_model_zoo

With the help of UER, we are pre-training models with different corpora, encoders, and targets.

pre-trained model Link Description
Wikizh+BertEncoder+BertTarget https://share.weiyun.com/5s9AsfQ The training corpus is Wiki_zh, trained by Google
Wikizh(word-based)+BertEncoder+BertTarget Model: https://share.weiyun.com/5s4HVMi Vocab: https://share.weiyun.com/5NWYbYn Word-based BERT model trained on Wikizh. Training steps: 500,000
RenMinRiBao+BertEncoder+BertTarget https://share.weiyun.com/5JWVjSE The training corpus is news data from People's Daily (1946-2017). It is suitable for datasets related with news, e.g. F1 is improved on MSRA-NER from 92.6 to 94.4 (compared with Google BERT). Training steps: 500,000
Webqa2019+BertEncoder+BertTarget https://share.weiyun.com/5HYbmBh The training corpus is WebQA, which is suitable for datasets related with social media, e.g. Accuracy (dev/test) on LCQMC is improved from 88.8/87.0 to 89.6/87.4; Accuracy (dev/test) on XNLI is improved from 78.1/77.2 to 79.0/78.8 (compared with Google BERT). Training steps: 500,000
Google-BERT-en-uncased-base Model: https://share.weiyun.com/5hWivED Vocab: https://share.weiyun.com/5gBxBYD Provided by Google.
Google-BERT-en-cased-base Model: https://share.weiyun.com/5SltATz Vocab: https://share.weiyun.com/5ouUo2q Provided by Google.
Reviews+LstmEncoder+LmTarget https://share.weiyun.com/57dZhqo The training corpus is amazon reviews + JDbinary reviews + dainping reviews (11.4M reviews in total). Language model target is used. It is suitable for datasets related with reviews. It achieves over 5 percent improvements on some review datasets compared with random initialization. Training steps: 200,000; Sequence length: 128
(Mixedlarge corpus & Amazon reviews)+LstmEncoder+(LmTarget & ClsTarget) https://share.weiyun.com/5B671Ik Mixedlarge corpus contains baidubaike + wiki + webqa + RenMinRiBao. The model is trained on it with language model target. And then the model is trained on Amazon reviews with language model and classification targets. It is suitable for datasets related with reviews. It can achieve comparable results with BERT on some review datasets. Training steps: 500,000 + 100,000; Sequence length: 128
IfengNews+BertEncoder+BertTarget https://share.weiyun.com/5HVcUWO The training corpus is news data from Ifeng website. We use news titles to predict news abstracts. Training steps: 100,000; Sequence length: 128
jdbinary+BertEncoder+ClsTarget https://share.weiyun.com/596k2bu The training corpus is review data from JD (jingdong). Classification target is used for pre-training. It is suitable for datasets related with shopping reviews, e.g. accuracy is improved on shopping datasets from 96.3 to 97.2 (compared with Google BERT). Training steps: 50,000; Sequence length: 128
jdfull+BertEncoder+MlmTarget https://share.weiyun.com/5L6EkUF The training corpus is review data from JD (jingdong). Masked LM target is used for pre-training. Training steps: 50,000; Sequence length: 128
Amazonreview+BertEncoder+ClsTarget https://share.weiyun.com/5XuxtFA The training corpus is review data from Amazon (including book reviews, movie reviews, and etc.). Classification target is used for pre-training. It is suitable for datasets related with reviews, e.g. accuracy is improved on Douban book review datasets from 87.6 to 88.5 (compared with Google BERT). Training steps: 20,000; Sequence length: 128
XNLI+BertEncoder+ClsTarget https://share.weiyun.com/5oXPugA Infersent with BertEncoder

We release the classification models on 5 large-scale datasets, i.e. Ifeng, Chinanews, Dianping, JDbinary, and JDfull. Users can use these models to reproduce results, or regard them as pre-training models for other datasets.

Datasets Link
Ifeng https://share.weiyun.com/5ZCp4wU
Chinanews https://share.weiyun.com/5bSfeQ7
Dianping https://share.weiyun.com/5Ls8R02
JDbinary https://share.weiyun.com/5QNu4QF
JDfull https://share.weiyun.com/5bqchN1

Contact information

For communication related to this project, please contact Zhe Zhao (helloworld@ruc.edu.cn; nlpzhezhao@tencent.com) or Xin Zhao (2014201975@ruc.edu.cn).

You can’t perform that action at this time.