The goal of CoQA challenge is to measure the ability of machines to read a passage and answer a series of interconnected questions that appear in a conversation. This repository hosts the code for generating the website and the associated data files. The resulting web files are stored on the gh-pages branch.
CoQA contains data from seven domains. Of these, only five are shared for public and the remaining two are private (for out-of-domain testing).
- Literature and Wikipedia domains are shared under CC BY-SA 4.0 license.
- Children's stories are collected from MCTest.
- Mid/High school exam passages are collected from RACE
- News articles are collected from CNN dataset.
To automate sanity checks, run the following commands
# First create submissions/FILE.txt indicating the worksheet, cl run command, submission name and date, e.g. see submissions/BERT-Augmented-AoA.txt
python print-coqa-eval-commands.py < submissions/FILE.txt > submissions/FILE.sh
sh submissions/FILE.sh
To see if the submission succeeded, run the following command
python populate_results.py < submissions/FILE.txt
To populate the leaderaboard, run the following command
cp out-v1.0.new.json out-v1.0.json
gulp
gulp connect
gulp deploy