Add danFEVER #97
Add danFEVER #97
Conversation
| for claim, evidence, label_id in zip(claims, evidences, labels): | ||
| claim_is_supported = class_labels[label_id] == "Supported" | ||
|
|
||
| sim = 1 if claim_is_supported else 0 # negative for refutes claims - is that what we want? |
There was a problem hiding this comment.
@Muennighoff - The DanFEVER dataset is similar to the FEVER dataset. Just wanted to make sure that this dataset is constructed fairly similar to FEVER.
I use the claim as the query to all the evidence segments as the corpus. The relevance score is then determined by whether the claim is supported.
However, I am unsure if assigning 0 to "not supported" and "not enough evidence" is meaningful.
What are your thoughts?
There was a problem hiding this comment.
However, I am unsure if assigning 0 to "not supported" and "not enough evidence" is meaningful.
If that's the same way it is done for FEVER, then I think it's okay!
There was a problem hiding this comment.
I am unsure how it is done for FEVER (can I find the processing script somewhere?)
There was a problem hiding this comment.
This is what it says in BEIR, so it does seem like they everything that's not the evidence is a 0
FEVER [60] The Fact Extraction and VERification dataset is collected to facilitate the automatic fact checking. We utilize the original paper splits as queries Q and retrieve evidences from the pre-processed Wikipedia Abstracts (June 2017 dump) as our corpus T
Added danfever as a retrieval dataset
fixes #93