Skip to content

Resources for the shared task on conversational question answering SCAI-QReCC 2021

License

Notifications You must be signed in to change notification settings

scai-conf/SCAI-QReCC-21

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SCAI-QReCC-22

[leaderboards] [registration] [forum] [contact] [SCAI]

Answer a series of contextually-dependent questions like they may occur in natural human-to-human conversations.

  • Submission deadline: July 3, 2022
  • Results announcement: July 10, 2022
  • Workshop presentations: July 15, 2022

Data

[Zenodo] [original]

File names here refer to the respective files hosted on [Zenodo].

The passage collection (passages.zip) is 27.5GB with 54M passages!

The input format for the task (scai-qrecc21-[toy,training,test]-questions[,-rewritten].json) is a JSON file:

[
  {
    "Conversation_no": <number>,
    "Turn_no": X,
    "Question": "<questionX>"
  }, ...
]

With X being the number of the question in the conversation. Questions with the same Conversation_no are from the same conversation.

The questions-rewritten.json-files contain human rewritten questions that can be used by systems that do not want to participate in question rewriting.

Submission

Register for the task using this form. We will then send you your TIRA login once it is ready.

The challenge is hosted on TIRA. Participants can upload their submission as a single JSON file. Alternatively, participants can upload their code and run the evaluation on the VMs provided by the platform to ensure reproducibility of the results.

The submission format for the task is a JSON file similar to the input (all Model_xxx-fields are optional and you can omit them from the submission, e.g. provide only Conversation_no, Turn_no and Model_answer to get the EM and F1 scores for the generated answers):

[
  {
    "Conversation_no": <number>,
    "Turn_no": X,
    "Model_rewrite": "<your-rewrite-of-questionX>",
    "Model_passages": { 
      "<ID-of-your-first-retrieved-passage-for-questionX>": <score-for-that-passage>, ...
    },
    "Model_answer": "<your-answer-for-questionX>",
    "Model_evidence": {"passage_id": "<ID-of-the-passage-containing-the-evidence>", "text": "<evidence-text-from-the-passage>"},
  }, ...
]

Example: scai-qrecc21-naacl-baseline.zip

You can use the code of our simple baseline to get started.

Run Submission

You can upload a JSON file as a submission at https://www.tira.io/task/scai-qrecc.

Please click on the submit button to enter the upload formular:

tira-submit

Please click on Uploads and select either scai-qrecc22-dataset (if the run uses the original questions) or scai-qrecc22-rewritten-dataset (if the run uses the rewritten questions) as dataset:

tira-upload

After you have uploaded your run, you can evaluate your run to verify that your run is valid. At the "Uploads" section, you can click on the blue (i)-icon to double-check your upload. You can also download the run from there.

Evaluation

[script]

Once you run your software or uploaded your run, "Run" the evaluator on that run through the TIRA web interface (below the software; works out-of-the-box).

TIRA Interface: Evaluation

Then go to the "Runs" section below and click on the blue (i)-icon of the evaluator run to see your scores.

Ground truth

We use the QReCC paper annotations in the initial phase, and will update them with alternative answer spans and passages by pooling and crowdsourcing the relevance judgements over the results submitted by the challenge participants (similar to the TREC evaluation setup).

Metrics

We use the same metrics as the QReCC paper, but may add more for the final evaluation: ROUGE1-R for question rewriting, Mean Reciprocal Rank (MRR) for passage retrieval, and F1 and Exact Match for question answering.

Baselines

We provide the following baselines for comparison:

  • scai-qrecc21-simple-baseline: BM25 baseline for passage retrieval using original conversational questions without rewriting. We recommend to use this code as a boilerplate to kickstart your own submission using the VM.
  • scai-qrecc21-naacl-baseline: results for the end-to-end approach using supervised question rewriting and QA models reported in the QReCC paper (accepted at NAACL'21). This sample run is available on Zenodo as scai-qrecc21-naacl-baseline.zip.

Note that the baseline results differ from the ones reported in the paper since we made several corrections to the evaluation script and the ground truth annotations:

  • We excluded the samples for which the ground truth is missing from the evaluation (i.e., no relevant passages or no answer text or no rewrite provided by the human annotators)

  • We removed 5,251 passages judgements annotated by the heuristic as relevant for the short answers with lengths <= 5 since these matches are often trivial and unrelated, e.g., the same noun phrase appearing in different contexts.

Resources

Some useful links to get you started on a new conversational open-domain QA system:

Conversational Passage Retrieval

Answer Generation

Passage Retrieval

Conversational Question Reformulation

SCAI-QReCC-21

[leaderboards] [forum] [contact] [SCAI]

The SCAI-QReCC-21 shared task is over and the results are reported in the overview paper.

Answer a series of contextually-dependent questions like they may occur in natural human-to-human conversations.

  • Submission deadline: September 8, 2021 Extended: September 15, 2021
  • Results announcement: September 30, 2021
  • Workshop presentations: October 8, 2021

About

Resources for the shared task on conversational question answering SCAI-QReCC 2021

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •