Skip to content
master
Switch branches/tags
Code

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 

CommonsenseQA

A question answering dataset for commonsense reasoning.

Check out the website!

Downloading the Data

You can download the data from the website, which also has an evaluation script. The leaderboard is for the random split of the data.

Running ESIM

Our implementation is based on this code. To run it, follow these steps:

  1. Install ESIM dependencies:
    cd esim
    pip install -r requirements.txt
    cd ..
    
  2. Place the dataset in data/ folder.
  3. Set PYTHONPATH to the commonsenseqa directory: export PYTHONPATH=$(pwd)
  4. Run the model either with pre-trained GloVe embeddings:
    python -m allennlp.run train esim/train-glove-csqa.json -s tmp --include-package esim
    
  5. Alternatively, run the model with ELMo pretrained contextual embeddings:
    python -m allennlp.run train esim/train-elmo-csqa.json -s tmp --include-package esim
    

Running BERT

To run BERT on CommonsenseQA, first install the BERT dependencies:

cd bert/
pip install -r requirements.txt

Then, obtain the CommonsenseQA data, and download the pretrained BERT weights. For the paper, we used BERT Large, Uncased. To train BERT Large, you'll most likely need to use a TPU; BERT base can be trained on a standard GPU.

To run training:

GPU

python run_commonsense_qa.py
  --split=$SPLIT \
  --do_train=true \
  --do_eval=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$BERT_DIR/bert_model.ckpt \
  --max_seq_length=128 \
  --train_batch_size=16 \
  --learning_rate=2e-5 \
  --num_train_epochs=3.0 \
  --output_dir=$OUTPUT_DIR

TPU

python run_commonsense_qa.py
  --split=$SPLIT \
  --use_tpu=true \
  --tpu_name=$TPU_NAME \
  --do_train=true \
  --do_eval=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$BERT_DIR/bert_model.ckpt \
  --max_seq_length=128 \
  --train_batch_size=16 \
  --learning_rate=2e-5 \
  --num_train_epochs=3.0 \
  --output_dir=$OUTPUT_DIR

For TPUs, all directories must be in Google Storage. The environment variables have the following meanings:

  • $SPLIT should either be rand or qtoken, depending on the split you'd like to run.
  • $DATA_DIR is a location for the CommonsenseQA data.
  • $BERT_DIR is a location for the pre-trained BERT files.
  • $TPU_NAME is the name of the TPU.
  • $OUTPUT_DIR is the directory to write output to.

To predict on the test set, run:

GPU (only)

python run_commonsense_qa.py \
  --split=$SPLIT \
  --do_predict=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$TRAINED_CHECKPOINT \
  --max_seq_length=128 \
  --output_dir=$OUTPUT_DIR

Prediction must be run on a GPU (including for BERT Large). All environment variables have the same meanings, and the new variable $TRAINED_CHECKPOINT is simply the prefix for your trained checkpoint files from fine-tuning BERT. It should look something like $OUTPUT_DIR/model.ckpt-1830.

About

Author implementation of the paper "CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge"

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages