BERT for RACE
By: Chenglei Si (River Valley High School)
This work is based on Pytorch implementation of BERT (https://github.com/huggingface/pytorch-pretrained-BERT). I adapted the original BERT model to work on multiple choice machine comprehension.
The code is tested with Python3.6 and Pytorch 1.0.0.
- Download the dataset and unzip it. The default dataset directory is ./RACE
I did some tuning and find the following hyperparameters to work reasonally well:
BERT_base: batch size: 32, learning rate: 5e-5, training epoch: 3
BERT_large: batch size: 8, learning rate: 1e-5 (DO NOT SET IT TOO LARGE), training epoch: 2
You can compare them with other results on the leaderboard.
BERT large achieves the current (Jan 2019) best result. Looking forward to new models that can beat BERT!
I have written a short report in this repo describing the details.