This repository contains the data for the HumEval2021 paper "Interrater disagreement resolution; A systematic procedure to reach consensus in annotation tasks" (2021). In this repository one can find (i) the flowchart of section 4 of the paper, (ii) the data statement and relevant data for the first test case, section 5.1, on Epistemology in Quine, and (iii) the data statement and relevant data for the second test case, section 5.2, on literature reviews in the History of Philosophy.
In section 4 of the paper, we present a procedure for annotation, illustrated by this flowchart. All steps of the flowchart are described in the paper.
In section 5.1 of the paper, we present the first test case. For this test case, annotators scored paragraphs in the work of the philosopher Willard Van Orman Quine for relevance on his views on epistemology. See this data statement for details on this dataset, and links to all relevant documents. See also Betti et al., 2020 for more information.
In section 5.2 of the paper, we present the second test case. For this test case, annotators scored articles from the British Journal of History of Philosophy between 2017 and 2019 by looking at their abstracts, introduction and methodology checking for inclusion/exclusion criteria for the literature taken into account, the completeness of the literature reviewed and the scope of the claims it makes. See this data statement for details on this dataset.
- Yvette Oortwijn
- Thijs Ossenkoppele
- Arianna Betti