Skip to content

NorEval v1.3

Latest

Choose a tag to compare

@vmkhlv vmkhlv released this 10 Nov 16:07

NorEval v1.3 fixes prompts, moves NRK-Quiz-QA to the filtered version 2.0 of the dataset, changes the primary metric of the multiple-choice question-answering tasks and adds task metadata (#9).

Changelog

  • Prompt fixes: NorBelebele prompt 2 includes the passage (#8). Cloze prompts no longer refer to answer options that aren't shown (#9). Hybrid prompts end with the same answer cue ("Svar:") as the other formulations. A Bokmål → Nynorsk translation prompt no longer asks for a Sámi translation. Typos, Nynorsk forms and list punctuation are corrected.
  • SLIDE prompt 2 asks the model to identify the language of the text, in all three formulations.
  • NRK-Quiz-QA uses version 2.0 of the dataset, which removes 140 questions (84 Bokmål, 56 Nynorsk) and changes 11 others.
  • Primary metric: the multiple-choice question-answering tasks (NorBelebele, NorCommonsenseQA, NorOpenBookQA with and without facts, NorTruthfulQA multiple choice, NRK-Quiz-QA) and SLIDE use per-character normalized accuracy (acc_norm). The nob_world_* tool groups aggregate acc_norm.
  • MultiBLiMP: multiblimp_ltg_sme is renamed to multiblimp_noreval_sme (configs in tasks/multiblimp). It is now a group that reports every phenomenon and a size-weighted overall score. The 1 -> 2|3 phenomenon is dropped, as its two sentence pairs are identical.
  • Task metadata: every task config has a metadata block with its category, language(s), primary metric, random baseline and version. noreval_metadata.json, built by tools/build_metadata.py, collects them and replaces meta.md. All tasks are at version 1.3.
  • NorQuAD works with transformers 5, and NorSumm needs bert_score only for BERTScore.
  • The README task table is grouped by category.