Repository navigation
NorEval v1.3 fixes prompts, moves NRK-Quiz-QA to the filtered version 2.0 of the dataset, changes the primary metric of the multiple-choice question-answering tasks and adds task metadata (#9).
Changelog
- Prompt fixes: NorBelebele prompt 2 includes the passage (#8). Cloze prompts no longer refer to answer options that aren't shown (#9). Hybrid prompts end with the same answer cue ("Svar:") as the other formulations. A Bokmål → Nynorsk translation prompt no longer asks for a Sámi translation. Typos, Nynorsk forms and list punctuation are corrected.
- SLIDE prompt 2 asks the model to identify the language of the text, in all three formulations.
- NRK-Quiz-QA uses version 2.0 of the dataset, which removes 140 questions (84 Bokmål, 56 Nynorsk) and changes 11 others.
- Primary metric: the multiple-choice question-answering tasks (NorBelebele, NorCommonsenseQA, NorOpenBookQA with and without facts, NorTruthfulQA multiple choice, NRK-Quiz-QA) and SLIDE use per-character normalized accuracy (
acc_norm). Thenob_world_*tool groups aggregateacc_norm. - MultiBLiMP:
multiblimp_ltg_smeis renamed tomultiblimp_noreval_sme(configs intasks/multiblimp). It is now a group that reports every phenomenon and a size-weighted overall score. The1 -> 2|3phenomenon is dropped, as its two sentence pairs are identical. - Task metadata: every task config has a
metadatablock with its category, language(s), primary metric, random baseline and version.noreval_metadata.json, built bytools/build_metadata.py, collects them and replacesmeta.md. All tasks are at version 1.3. - NorQuAD works with transformers 5, and NorSumm needs
bert_scoreonly for BERTScore. - The README task table is grouped by category.