Skip to content

NorEval v1.2

Choose a tag to compare

@vmkhlv vmkhlv released this 10 Nov 12:01
· 6 commits to main since this release

NorEval v1.2 restructures the multiple-choice tasks into three formulations, adds new metrics and samples the generative tasks (#7).

Changelog

  • New prompts: Every multiple-choice task is now evaluated in three formulations: cloze, multiple-choice & hybrid. Each multiple-choice dataset now has 5 prompt stems, each with 3 formulations (instead of 5 mixed prompts), giving 15 subtasks with tags <task>_cf, <task>_hybrid, <task>_mcf.
  • New metrics: All multiple-choice tasks are now evaluated with all possible loglikelihood normalizations (none, per-character and PMI), as well as with new soft probabilistic metrics.
  • New metrics: Standard errors are now computed consistently for every metric.
  • All generative tasks are now sampled, without any temperature adjustment, to better estimate the expected performance and measurement uncertainty. To reduce variance, we repeat the sampling multiple times.
  • The whole repository has been restructured so that all tasks are defined consistently. Directories now follow <task>/<language>/<task>_<language>_p<N>.yaml format throughout.
  • BERTScore is now pinned to device="cpu".
  • Shuffled order of NorTruthfulQA options.
  • Task names now end with a language code (norquad → norquad_nob), and translation tasks are named translation_<source>_<target> (tatoeba_eng_nob → translation_eng_nob). Commands written for v1.1 need the new names.

Known issues, fixed for v1.3

  • NorBelebele prompt 2 shows no passage (#8), and some cloze prompts refer to answer options that aren't shown (#9).
  • NorQuAD fails to import with transformers 5.