Permalink
Find file Copy path
a4af402 Oct 1, 2018
hopkinsm Adapted README.
0 contributors

Users who have contributed to this file

29 lines (21 sloc) 1.87 KB

Getting Started with SemEval 2019, Task 10

The following is a quickstart guide for Task 10:

  1. Fork the Task 10 repository: https://github.com/allenai/semeval-2019-task-10
  2. The available datasets are in the repo directory (the file format is described here):
  • Training data: data/sat.train.json
  • Training diagrams: data/diagrams/diagrams.train.tar.gz
  • Development data: data/sat.dev.json
  • Development diagrams: data/diagrams/diagrams.dev.tar.gz
  1. Create a program that can take data/sat.dev.json as input and output a list of JSON datum { id: <id>, answer: "<answer>"}, where <id> is the integer index of a question and <answer> is the guessed answer (either a choice key or a numeric string). An example program is provided in the repo at baselines/guesser.py. Run python baselines/guesser.py data/sat.dev.json answer.json b-guesser. This program will guess "B" for every multiple choice question (and abstain from direct answer questions). The output is stored in `answer.json'.
  2. Create a CodaLab submission file:
zip submission.zip answer.json 
  1. Submit the file to the Task 10 Codalab competition by:
  • Going to the Participate tab.
  • Going to the Submit/View Results submenu.
  • Clicking the Submit button to upload a new result.
  • Finding submission.zip in your local directory structure and uploading it.
  1. You can then wait until the status of your submission says Completed (you will likely need to click the Refresh Status button or refresh your browser to force a status change).
  2. Your scores should become visible and you are given the option to submit your result to the leaderboard.

To use the gold logical forms and the logical form interpreter, see the following page.