percentage-refBias
These scripts provide a detailed process of how the percentages of human annotators pairs with significance differences in Kappa coefficients for pairs of annotators shown the same reference translation (SAME), different reference translation (DIFF) or the source language input only (SOURCE).
Requirements:
1. R
2. Python
Usage:
./run.sh
This shell script deals with all things and produces the 'percentages' file.
Publication:
The percentages of human annotator pairs with significant differences is described in detail in: "Further Investigation into Reference Bias in Monolingual Evaluation of Machine Translation", Qingsong Ma, Yvette Graham, Timothy Baldwin, Qun Lin, EMNLP 2017.