Repository navigation
NorEval v1.2
NorEval v1.2 restructures the multiple-choice tasks into three formulations, adds new metrics and samples the generative tasks (#7).
Changelog
- New prompts: Every multiple-choice task is now evaluated in three formulations: cloze, multiple-choice & hybrid. Each multiple-choice dataset now has 5 prompt stems, each with 3 formulations (instead of 5 mixed prompts), giving 15 subtasks with tags
<task>_cf,<task>_hybrid,<task>_mcf. - New metrics: All multiple-choice tasks are now evaluated with all possible loglikelihood normalizations (none, per-character and PMI), as well as with new soft probabilistic metrics.
- New metrics: Standard errors are now computed consistently for every metric.
- All generative tasks are now sampled, without any temperature adjustment, to better estimate the expected performance and measurement uncertainty. To reduce variance, we repeat the sampling multiple times.
- The whole repository has been restructured so that all tasks are defined consistently. Directories now follow
<task>/<language>/<task>_<language>_p<N>.yamlformat throughout. - BERTScore is now pinned to
device="cpu". - Shuffled order of NorTruthfulQA options.
- Task names now end with a language code (
norquad→norquad_nob), and translation tasks are namedtranslation_<source>_<target>(tatoeba_eng_nob→translation_eng_nob). Commands written for v1.1 need the new names.