Yuanfudao at SemEval-2018 Task 11: Three-way Attention and Relational Knowledge for Commonsense Machine Comprehension

Model Overview

We use attention-based LSTM networks.

For more technical details, please refer to our paper at https://arxiv.org/abs/1803.00191

For more details about this task, please refer to paper SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge.

Official leaderboard is available at https://competitions.codalab.org/competitions/17184#results (Evaluation Phase)

The overall model architecture is shown below:

How to run

Prerequisite

pytorch 0.2, 0.3 or 0.4 (may have a few warnings, but that's ok)

spacy >= 2.0

Won't work for >= python3.7 due to async keyword conflict.

GPU machine is preferred, training on CPU will be much slower.

Step 1:

Download preprocessed data from Google Drive or Baidu Cloud Disk, unzip and put them under folder data/.

If you choose to preprocess dataset by yourself, please run ./download.sh to download Glove embeddings and ConceptNet, and then run ./run.sh to preprocess dataset and train the model.

Official dataset can be downloaded on hidrive.

We transform original XML format data to Json format with xml2json by running ./xml2json.py --pretty --strip_text -t xml2json -o test-data.json test-data.xml

Step 2:

Train model with python3 src/main.py --gpu 0, the accuracy on development set will be approximately 83% after 50 epochs.

How to reproduce our competition results

Following above instructions you will get a model with ~81.5% accuracy on test set, we use two additional techniques for our official submission (~83.95% accuracy):

1. Pretrain our model with RACE dataset for 10 epochs.

2. Train 9 models with different random seeds and ensemble their outputs.