[leaderboards] [registration] [forum] [contact] [SCAI]
Answer a series of contextually-dependent questions like they may occur in natural human-to-human conversations.
- Submission deadline: July 3, 2022
- Results announcement: July 10, 2022
- Workshop presentations: July 15, 2022
File names here refer to the respective files hosted on [Zenodo].
The passage collection (passages.zip) is 27.5GB with 54M passages!
The input format for the task (scai-qrecc21-[toy,training,test]-questions[,-rewritten].json) is a JSON file:
[
{
"Conversation_no": <number>,
"Turn_no": X,
"Question": "<questionX>"
}, ...
]
With X being the number of the question in the conversation. Questions with the same Conversation_no are from the same conversation.
The questions-rewritten.json-files contain human rewritten questions that can be used by systems that do not want to participate in question rewriting.
Register for the task using this form. We will then send you your TIRA login once it is ready.
The challenge is hosted on TIRA. Participants can upload their submission as a single JSON file. Alternatively, participants can upload their code and run the evaluation on the VMs provided by the platform to ensure reproducibility of the results.
The submission format for the task is a JSON file similar to the input (all Model_xxx-fields are optional and you can omit them from the submission, e.g. provide only Conversation_no, Turn_no and Model_answer to get the EM and F1 scores for the generated answers):
[
{
"Conversation_no": <number>,
"Turn_no": X,
"Model_rewrite": "<your-rewrite-of-questionX>",
"Model_passages": {
"<ID-of-your-first-retrieved-passage-for-questionX>": <score-for-that-passage>, ...
},
"Model_answer": "<your-answer-for-questionX>",
"Model_evidence": {"passage_id": "<ID-of-the-passage-containing-the-evidence>", "text": "<evidence-text-from-the-passage>"},
}, ...
]
Example: scai-qrecc21-naacl-baseline.zip
You can use the code of our simple baseline to get started.
You can upload a JSON file as a submission at https://www.tira.io/task/scai-qrecc.
Please click on the submit button to enter the upload formular:
Please click on Uploads and select either scai-qrecc22-dataset (if the run uses the original questions) or scai-qrecc22-rewritten-dataset (if the run uses the rewritten questions) as dataset:
After you have uploaded your run, you can evaluate your run to verify that your run is valid. At the "Uploads" section, you can click on the blue (i)-icon to double-check your upload. You can also download the run from there.
[script]
Once you run your software or uploaded your run, "Run" the evaluator on that run through the TIRA web interface (below the software; works out-of-the-box).
Then go to the "Runs" section below and click on the blue (i)-icon of the evaluator run to see your scores.
We use the QReCC paper annotations in the initial phase, and will update them with alternative answer spans and passages by pooling and crowdsourcing the relevance judgements over the results submitted by the challenge participants (similar to the TREC evaluation setup).
We use the same metrics as the QReCC paper, but may add more for the final evaluation: ROUGE1-R for question rewriting, Mean Reciprocal Rank (MRR) for passage retrieval, and F1 and Exact Match for question answering.
We provide the following baselines for comparison:
- scai-qrecc21-simple-baseline: BM25 baseline for passage retrieval using original conversational questions without rewriting. We recommend to use this code as a boilerplate to kickstart your own submission using the VM.
- scai-qrecc21-naacl-baseline: results for the end-to-end approach using supervised question rewriting and QA models reported in the QReCC paper (accepted at NAACL'21). This sample run is available on Zenodo as scai-qrecc21-naacl-baseline.zip.
Note that the baseline results differ from the ones reported in the paper since we made several corrections to the evaluation script and the ground truth annotations:
-
We excluded the samples for which the ground truth is missing from the evaluation (i.e., no relevant passages or no answer text or no rewrite provided by the human annotators)
-
We removed 5,251 passages judgements annotated by the heuristic as relevant for the short answers with lengths <= 5 since these matches are often trivial and unrelated, e.g., the same noun phrase appearing in different contexts.
Some useful links to get you started on a new conversational open-domain QA system:
[leaderboards] [forum] [contact] [SCAI]
The SCAI-QReCC-21 shared task is over and the results are reported in the overview paper.
Answer a series of contextually-dependent questions like they may occur in natural human-to-human conversations.
- Submission deadline:
September 8, 2021Extended: September 15, 2021 - Results announcement: September 30, 2021
- Workshop presentations: October 8, 2021