Clarification on Part 3

[GitianOberhuber]

Hello,
I have some questions regarding Part 3 of the exercise. I read the exercise description as well as the other discussion-post about the topic (#74) but I still can't make sense of it.

• What is the purpose of the 3 provided files qrels.retrieval, qrel.qa-answers and qrel.qa-tupels? These files seem to contain extremely similar information - qrels.retrieval contains pairs of queryId and documentId and relevance grades, qrels.qa-answers contains that (though only the entries with relevance grade 2 or 3) plus text selection and qrels.qa-tuples contains that plus the query- and document-text. On the other discussion-post linked above, it says "The last point asks for an evaluation of the extractive QA pipeline for the msmarco-fira-21.qrels.qa-answers.tsv as queries and the msmarco-fira-21.qrels.qa-tuples.tsv as labels". Why would qrels.qa-answers contain queries while qrels.qa_tuples contains labels? They contain the same information except for the document- and query-text (both files contain the text-passage).

• Lastly, I think I managed to understand what Part 3 is asking, but I would like to confirm it and make sure that my understanding is correct and that I didn't miss anything:
  On part 2, perform re-ranking such that you have a list of (possibly) relevant documents for a query. Take the first (and hopefully most relevant) entry in that list, which is a pair of queryId and documentId. Using this pair, get the document- and query-text, run the QA model for them, take the resulting text-passage and compare that to label (aka the text-passage that can be found in either qrels.qa-answers or qrels.qa-tuples).
  Is this correct? Did I miss something?

[sophiaalthammer]

Hello,

two your points:

• Yes they do contain similar information. The qrels.retrieval file you should use in Part 2 to evaluate your retrieval approach. The qrels.qa-answers and qrels.qa-tuples are label files containing the answer span and should be used to evaluate the QA model in Part 3.
• Yes you understood the assignment correctly. You take the top1 document for each query, run the QA model and then evaluate on both test sets: qrels.qa-answers and qrels.qa-tuples. For both test sets you need to consider the field text-selection as the answer and then evaluate it regarding that.

Is that clear now?

Best,
Sophia

[GitianOberhuber]

Hello,

two your points:

* Yes they do contain similar information. The qrels.retrieval file you should use in Part 2 to evaluate your retrieval approach. The qrels.qa-answers and qrels.qa-tuples are label files containing the answer span and should be used to evaluate the QA model in Part 3.

* Yes you understood the assignment correctly. You take the top1 document for each query, run the QA model and then evaluate on both test sets: qrels.qa-answers and qrels.qa-tuples. For both test sets you need to consider the field text-selection as the answer and then evaluate it regarding that.

Is that clear now?

Best, Sophia

Dear Sophia,

Thanks a lot for the response! It certainly clears some things up. However, there is still one thing that is unclear: As I mentioned, qrels.qa-answers and qrels.qa-tuples contain the same data. That is, they contain the exact same queryId-documentId-textpassage entries, the only difference between those two files is that qrels.qa-tuples contains two additional columns, the docment-text and the query-text. Therefore I don't get why we should, as you mentioned, evaluate our results on both files, because they contain the same data.

[sophiaalthammer]

Hello,

I am sorry, I also got confused and your question is understandable now!
So you should also evaluate the run from the qrels.qa-tuples with the qrels.qa-answers, so its not two tests sets, but always one test set (qrels.qa-answers). First you evaluate your top1 results, then the tuples given in qrels.qa-tuples.

Sorry for the confusion!

Best,
Sophia

[GitianOberhuber]

Ok thanks, I think I understand now :)
Just one (hopefully) last thing I want to confirm after thinking about what you said:
When evaluating the top1 results, we can only consider query-document pairs that are also present within the qrels.qa-answers/qa-tuples files, because otherwise we wouldn't have the label (text-selection column).
This means that our top1 results which we are evaluating are actally a subset of qrels.qa-tuples. Is that correct?

[sophiaalthammer]

No not really, if a query-document pair is nto present in the qrels file, then it should be considered as irrelevant. But I think this is also implemented like that in the predefined functions, but you have to look that up!