Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
Apr 23, 2019

A BERT-Based Machine Reading Comprehension Baseline

This repository maintains a machine reading comprehension baseline based on BERT. The implementations follow the baseline system descriptions in the following two papers.

If you find this code useful, please consider citing the following papers.

@article{sun2019probing,
  title={Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension},
  author={Sun, Kai and Yu, Dian and Yu, Dong and Cardie, Claire},
  journal={CoRR},
  volume={cs.CL/1904.09679v2},
  url={https://arxiv.org/abs/1904.09679v2}
  year={2019}
}

@article{pan2019improving,
  title={Improving Question Answering with External Knowledge},
  author={Pan, Xiaoman and Sun, Kai and Yu, Dian and Ji, Heng and Yu, Dong},
  journal={CoRR},
  volume={cs.CL/1902.00993v1}
  url={https://arxiv.org/abs/1902.00993v1},
  year={2019}
}

Here, we show the usage of this baseline using a demo designed for DREAM, a dialogue-based three-choice machine reading comprehension task.

  1. Download and unzip the pre-trained language model from https://github.com/google-research/bert. and set up the environment variable for BERT by export BERT_BASE_DIR=/PATH/TO/BERT/DIR.
  2. Copy the data folder data from the DREAM repo to bert/.
  3. In bert, execute python convert_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=$BERT_BASE_DIR/bert_model.ckpt --bert_config_file=$BERT_BASE_DIR/bert_config.json --pytorch_dump_path=$BERT_BASE_DIR/pytorch_model.bin
  4. Execute python run_classifier.py --task_name dream --do_train --do_eval --data_dir . --vocab_file $BERT_BASE_DIR/vocab.txt --bert_config_file $BERT_BASE_DIR/bert_config.json --init_checkpoint $BERT_BASE_DIR/pytorch_model.bin --max_seq_length 512 --train_batch_size 24 --learning_rate 2e-5 --num_train_epochs 8.0 --output_dir dream_finetuned --gradient_accumulation_steps 3
  5. The resulting fine-tuned model, predictions, and evaluation results are stored in bert/dream_finetuned.

Results on DREAM:

We run the experiments five times with different random seeds and report the best development set performance and the corresponding test set performance.

Method/Language Model Batch Size Learning Rate Epochs Dev Test
BERT-Base, Uncased 24 2e-5 8 63.4 63.2
BERT-Large, Uncased 24 2e-5 16 66.0 66.8
Human Performance 93.9 95.5
Ceiling Performance 98.7 98.6

Environment: The code has been tested with Python 3.6 and PyTorch 1.0

About

A BERT-Based Machine Reading Comprehension Baseline

Topics

Resources

Releases

No releases published

Packages

No packages published

Languages