Repository navigation
Releases
NorEval v1.3
NorEval v1.3 fixes prompts, moves NRK-Quiz-QA to the filtered version 2.0 of the dataset, changes the primary metric of the multiple-choice question-answering tasks and adds task metadata (#9).
Changelog
- Prompt fixes: NorBelebele prompt 2 includes the passage (#8). Cloze prompts no longer refer to answer options that aren't shown (#9). Hybrid prompts end with the same answer cue ("Svar:") as the other formulations. A Bokmål → Nynorsk translation prompt no longer asks for a Sámi translation. Typos, Nynorsk forms and list punctuation are corrected.
- SLIDE prompt 2 asks the model to identify the language of the text, in all three formulations.
- NRK-Quiz-QA uses version 2.0 of the dataset, which removes 140 questions (84 Bokmål, 56 Nynorsk) and changes 11 others.
- Primary metric: the multiple-choice question-answering tasks (NorBelebele, NorCommonsenseQA, NorOpenBookQA with and without facts, NorTruthfulQA multiple choice, NRK-Quiz-QA) and SLIDE use per-character normalized accuracy (
acc_norm). Thenob_world_*tool groups aggregateacc_norm. - MultiBLiMP:
multiblimp_ltg_smeis renamed tomultiblimp_noreval_sme(configs intasks/multiblimp). It is now a group that reports every phenomenon and a size-weighted overall score. The1 -> 2|3phenomenon is dropped, as its two sentence pairs are identical. - Task metadata: every task config has a
metadatablock with its category, language(s), primary metric, random baseline and version.noreval_metadata.json, built bytools/build_metadata.py, collects them and replacesmeta.md. All tasks are at version 1.3. - NorQuAD works with transformers 5, and NorSumm needs
bert_scoreonly for BERTScore. - The README task table is grouped by category.
NorEval v1.2
NorEval v1.2 restructures the multiple-choice tasks into three formulations, adds new metrics and samples the generative tasks (#7).
Changelog
- New prompts: Every multiple-choice task is now evaluated in three formulations: cloze, multiple-choice & hybrid. Each multiple-choice dataset now has 5 prompt stems, each with 3 formulations (instead of 5 mixed prompts), giving 15 subtasks with tags
<task>_cf,<task>_hybrid,<task>_mcf. - New metrics: All multiple-choice tasks are now evaluated with all possible loglikelihood normalizations (none, per-character and PMI), as well as with new soft probabilistic metrics.
- New metrics: Standard errors are now computed consistently for every metric.
- All generative tasks are now sampled, without any temperature adjustment, to better estimate the expected performance and measurement uncertainty. To reduce variance, we repeat the sampling multiple times.
- The whole repository has been restructured so that all tasks are defined consistently. Directories now follow
<task>/<language>/<task>_<language>_p<N>.yamlformat throughout. - BERTScore is now pinned to
device="cpu". - Shuffled order of NorTruthfulQA options.
- Task names now end with a language code (
norquad→norquad_nob), and translation tasks are namedtranslation_<source>_<target>(tatoeba_eng_nob→translation_eng_nob). Commands written for v1.1 need the new names.
Known issues, fixed for v1.3
NorEval v1.1.1
NorEval v1.1.1 is a reproducibility fix for v1.1. The tasks, prompts and metrics are unchanged.
Changelog
- Every task now pins its Hugging Face dataset to a specific commit with
dataset_kwargs: revision: <commit hash>, so one version of NorEval always evaluates on exactly the same data, regardless of later updates to the datasets (#6). - Fixes the Tatoeba English ↔ Bokmål/Nynorsk tasks (
tatoeba_eng_nob,tatoeba_eng_nno,tatoeba_nob_eng,tatoeba_nno_eng), which no longer work in v1.1: Helsinki-NLP/tatoeba_mt was replaced with a new benchmark release that no longer has theeng-nobandeng-nnosubsets, so v1.1 fails withBuilderConfig 'eng-nob' not found. They are now pinned to the last version of the dataset before that change. - NRK-Quiz-QA stays on version v1.0 of ltg/nrk_quiz_qa, now pinned by commit hash.
Note: Tatoeba requires accepting the dataset's terms
Helsinki-NLP/tatoeba_mt is now a gated dataset. Before running the four tasks above, open the dataset page and accept its terms of use.
NorEval v1.1
Warning
The Tatoeba English ↔ Bokmål/Nynorsk tasks (tatoeba_eng_nob, tatoeba_eng_nno, tatoeba_nob_eng, tatoeba_nno_eng) no longer work in v1.1 and fail with BuilderConfig 'eng-nob' not found: v1.1 loads the latest version of Helsinki-NLP/tatoeba_mt, which was replaced with a new benchmark release on 29 September 2026 that no longer has these subsets. Please use v1.1.1, which pins every dataset to a fixed version.
NorEval v1.1 updates the task configs to the versions used in our Norwegian LLM dashboard, covering all 35 tasks. The tasks are plain LM Evaluation Harness configs: clone this repository and pass it to lm_eval with --include_path (see the README).
Changelog
- New metadata: Every task now explicitly declares the primary metric, random-baseline performance, category and language(s). The order of metrics is changed to list the primary metric first.
- New task: SLIDE is a language identification task with five choices: Bokmål, Nynorsk, Danish, Swedish, or other language. One sentence can be compatible with multiple languages, a prediction is considered correct if it matches any of them.
- New task: NoCoLA, BLiMP-like Norwegian linguistic acceptability task.
- New task: Northern Sámi MultiBLiMP is actually a group of tasks, we evaluate each linguistic feature separately to get more informative results (unlike the official
lm-evalMultiBLiMP). - New task: NorOpenBookQA without facts is a simple rephrasing of the original NorOpenBookQA so that it evaluates knowledge rather then reading comprehension.
- New task: Bokmål
↔️ Nynorsk translation based on the NorSumm dataset, which contains multi-sentence manual translations. - New task: Bokmål
↔️ Northern Sámi translation based on ltg/saami-tatoeba. - NorBelebele is now based on the corrected Bokmål passages from FLORES+ -- ltg/norbelebele.
- NorBelebele has two additional prompts, so that it's also evaluated in cloze formulation.
- ASK-GEC is now scored inside the harness, not in an extra post-production step.
- NorIdiom gets a new primary metric
em_firstand a less aggressive stopping criterion. - The references in NorSumm are cleaned -- we strip newlines from the reference summaries, which broke few-shot evaluation previously.
- Hard-coded
num_fewshot: 0are removed fromncb,noridiom,norsumm,nortruthfulqa_gen,nortruthfulqa_mcandnrk_quiz_qa. num_beams: 1is removed from all generative tasks, which previously broke evaluation with vLLM.nortruthfulqa_gen'sdoc_to_targetis changed from" "tobest_answer, to work properly with few-shot evaluation.- NRK-Quiz-QA is pinned to version
v1.0of ltg/nrk_quiz_qa withdataset_kwargs: revision: v1.0, so results stay comparable to previous evaluations. The dataset's new v2.0 removes 140 examples.
NorEval v1.0
First versioned release of NorEval, the Norwegian language understanding and generation evaluation benchmark described in our ACL 2025 Findings paper.
This release includes:
- 24 datasets for Norwegian Bokmål and Nynorsk, integrated into LM Evaluation Harness (
lm_eval/tasks/noreval) - Annotation guidelines for human baselines, prompt writing, NorRewrite-Instruct and NorSummarize-Instruct
- Example task groups and result-processing scripts in
tools/