Skip to content
No description, website, or topics provided.
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bert
esim ESIM implementation for commonsenseQA Jun 3, 2019
README.md

README.md

CommonsenseQA

A question answering dataset for commonsense reasoning.

Check out the website!

Downloading the Data

You can download the data from the website, which also has an evaluation script. The leaderboard is for the random split of the data.

Running ESIM

Our implementation is based on this code. To run it, follow these steps:

  1. Install ESIM dependencies:
    cd esim
    pip install -r requirements.txt
    cd ..
    
  2. Place the dataset in data/ folder.
  3. Set PYTHONPATH to the commonsenseqa directory: export PYTHONPATH=$(pwd)
  4. Run the model either with pre-trained GloVe embeddings:
    python -m allennlp.run train esim/train-glove-csqa.json -s tmp --include-package esim
    
  5. Alternatively, run the model with ELMo pretrained contextual embeddings:
    python -m allennlp.run train esim/train-elmo-csqa.json -s tmp --include-package esim
    

Running BERT

To run BERT on CommonsenseQA, first install the BERT dependencies:

cd bert/
pip install -r requirements.txt

Then, obtain the CommonsenseQA data, and download the pretrained BERT weights. For the paper, we used BERT Large, Uncased. To train BERT Large, you'll most likely need to use a TPU; BERT base can be trained on a standard GPU.

To run training:

GPU

python run_commonsense_qa.py
  --split=$SPLIT \
  --do_train=true \
  --do_eval=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$BERT_DIR/bert_model.ckpt \
  --max_seq_length=128 \
  --train_batch_size=16 \
  --learning_rate=2e-5 \
  --num_train_epochs=3.0 \
  --output_dir=$OUTPUT_DIR

TPU

python run_commonsense_qa.py
  --split=$SPLIT \
  --use_tpu=true \
  --tpu_name=$TPU_NAME \
  --do_train=true \
  --do_eval=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$BERT_DIR/bert_model.ckpt \
  --max_seq_length=128 \
  --train_batch_size=16 \
  --learning_rate=2e-5 \
  --num_train_epochs=3.0 \
  --output_dir=$OUTPUT_DIR

For TPUs, all directories must be in Google Storage. The environment variables have the following meanings:

  • $SPLIT should either be rand or qtoken, depending on the split you'd like to run.
  • $DATA_DIR is a location for the CommonsenseQA data.
  • $BERT_DIR is a location for the pre-trained BERT files.
  • $TPU_NAME is the name of the TPU.
  • $OUTPUT_DIR is the directory to write output to.

To predict on the test set, run:

GPU (only)

python run_commonsense_qa.py \
  --split=$SPLIT \
  --do_predict=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$TRAINED_CHECKPOINT \
  --max_seq_length=128 \
  --output_dir=$OUTPUT_DIR

Prediction must be run on a GPU (including for BERT Large). All environment variables have the same meanings, and the new variable $TRAINED_CHECKPOINT is simply the prefix for your trained checkpoint files from fine-tuning BERT. It should look something like $OUTPUT_DIR/model.ckpt-1830.

You can’t perform that action at this time.