Skip to content

Releases

NorEval v1.3

Choose a tag to compare

@vmkhlv vmkhlv released this 10 Nov 16:07

NorEval v1.3 fixes prompts, moves NRK-Quiz-QA to the filtered version 2.0 of the dataset, changes the primary metric of the multiple-choice question-answering tasks and adds task metadata (#9).

Changelog

  • Prompt fixes: NorBelebele prompt 2 includes the passage (#8). Cloze prompts no longer refer to answer options that aren't shown (#9). Hybrid prompts end with the same answer cue ("Svar:") as the other formulations. A Bokmål → Nynorsk translation prompt no longer asks for a Sámi translation. Typos, Nynorsk forms and list punctuation are corrected.
  • SLIDE prompt 2 asks the model to identify the language of the text, in all three formulations.
  • NRK-Quiz-QA uses version 2.0 of the dataset, which removes 140 questions (84 Bokmål, 56 Nynorsk) and changes 11 others.
  • Primary metric: the multiple-choice question-answering tasks (NorBelebele, NorCommonsenseQA, NorOpenBookQA with and without facts, NorTruthfulQA multiple choice, NRK-Quiz-QA) and SLIDE use per-character normalized accuracy (acc_norm). The nob_world_* tool groups aggregate acc_norm.
  • MultiBLiMP: multiblimp_ltg_sme is renamed to multiblimp_noreval_sme (configs in tasks/multiblimp). It is now a group that reports every phenomenon and a size-weighted overall score. The 1 -> 2|3 phenomenon is dropped, as its two sentence pairs are identical.
  • Task metadata: every task config has a metadata block with its category, language(s), primary metric, random baseline and version. noreval_metadata.json, built by tools/build_metadata.py, collects them and replaces meta.md. All tasks are at version 1.3.
  • NorQuAD works with transformers 5, and NorSumm needs bert_score only for BERTScore.
  • The README task table is grouped by category.

NorEval v1.2

Choose a tag to compare

@vmkhlv vmkhlv released this 10 Nov 12:01

NorEval v1.2 restructures the multiple-choice tasks into three formulations, adds new metrics and samples the generative tasks (#7).

Changelog

  • New prompts: Every multiple-choice task is now evaluated in three formulations: cloze, multiple-choice & hybrid. Each multiple-choice dataset now has 5 prompt stems, each with 3 formulations (instead of 5 mixed prompts), giving 15 subtasks with tags <task>_cf, <task>_hybrid, <task>_mcf.
  • New metrics: All multiple-choice tasks are now evaluated with all possible loglikelihood normalizations (none, per-character and PMI), as well as with new soft probabilistic metrics.
  • New metrics: Standard errors are now computed consistently for every metric.
  • All generative tasks are now sampled, without any temperature adjustment, to better estimate the expected performance and measurement uncertainty. To reduce variance, we repeat the sampling multiple times.
  • The whole repository has been restructured so that all tasks are defined consistently. Directories now follow <task>/<language>/<task>_<language>_p<N>.yaml format throughout.
  • BERTScore is now pinned to device="cpu".
  • Shuffled order of NorTruthfulQA options.
  • Task names now end with a language code (norquad → norquad_nob), and translation tasks are named translation_<source>_<target> (tatoeba_eng_nob → translation_eng_nob). Commands written for v1.1 need the new names.

Known issues, fixed for v1.3

  • NorBelebele prompt 2 shows no passage (#8), and some cloze prompts refer to answer options that aren't shown (#9).
  • NorQuAD fails to import with transformers 5.

NorEval v1.1.1

Choose a tag to compare

@vmkhlv vmkhlv released this 03 Nov 12:44
68b2a09

NorEval v1.1.1 is a reproducibility fix for v1.1. The tasks, prompts and metrics are unchanged.

Changelog

  • Every task now pins its Hugging Face dataset to a specific commit with dataset_kwargs: revision: <commit hash>, so one version of NorEval always evaluates on exactly the same data, regardless of later updates to the datasets (#6).
  • Fixes the Tatoeba English ↔ Bokmål/Nynorsk tasks (tatoeba_eng_nob, tatoeba_eng_nno, tatoeba_nob_eng, tatoeba_nno_eng), which no longer work in v1.1: Helsinki-NLP/tatoeba_mt was replaced with a new benchmark release that no longer has the eng-nob and eng-nno subsets, so v1.1 fails with BuilderConfig 'eng-nob' not found. They are now pinned to the last version of the dataset before that change.
  • NRK-Quiz-QA stays on version v1.0 of ltg/nrk_quiz_qa, now pinned by commit hash.

Note: Tatoeba requires accepting the dataset's terms

Helsinki-NLP/tatoeba_mt is now a gated dataset. Before running the four tasks above, open the dataset page and accept its terms of use.

NorEval v1.1

Choose a tag to compare

@vmkhlv vmkhlv released this 17 Oct 15:26

Warning

The Tatoeba English ↔ Bokmål/Nynorsk tasks (tatoeba_eng_nob, tatoeba_eng_nno, tatoeba_nob_eng, tatoeba_nno_eng) no longer work in v1.1 and fail with BuilderConfig 'eng-nob' not found: v1.1 loads the latest version of Helsinki-NLP/tatoeba_mt, which was replaced with a new benchmark release on 29 September 2026 that no longer has these subsets. Please use v1.1.1, which pins every dataset to a fixed version.

NorEval v1.1 updates the task configs to the versions used in our Norwegian LLM dashboard, covering all 35 tasks. The tasks are plain LM Evaluation Harness configs: clone this repository and pass it to lm_eval with --include_path (see the README).

Changelog

  • New metadata: Every task now explicitly declares the primary metric, random-baseline performance, category and language(s). The order of metrics is changed to list the primary metric first.
  • New task: SLIDE is a language identification task with five choices: Bokmål, Nynorsk, Danish, Swedish, or other language. One sentence can be compatible with multiple languages, a prediction is considered correct if it matches any of them.
  • New task: NoCoLA, BLiMP-like Norwegian linguistic acceptability task.
  • New task: Northern Sámi MultiBLiMP is actually a group of tasks, we evaluate each linguistic feature separately to get more informative results (unlike the official lm-eval MultiBLiMP).
  • New task: NorOpenBookQA without facts is a simple rephrasing of the original NorOpenBookQA so that it evaluates knowledge rather then reading comprehension.
  • New task: Bokmål ↔️ Nynorsk translation based on the NorSumm dataset, which contains multi-sentence manual translations.
  • New task: Bokmål ↔️ Northern Sámi translation based on ltg/saami-tatoeba.
  • NorBelebele is now based on the corrected Bokmål passages from FLORES+ -- ltg/norbelebele.
  • NorBelebele has two additional prompts, so that it's also evaluated in cloze formulation.
  • ASK-GEC is now scored inside the harness, not in an extra post-production step.
  • NorIdiom gets a new primary metric em_first and a less aggressive stopping criterion.
  • The references in NorSumm are cleaned -- we strip newlines from the reference summaries, which broke few-shot evaluation previously.
  • Hard-coded num_fewshot: 0 are removed from ncb, noridiom, norsumm, nortruthfulqa_gen, nortruthfulqa_mc and nrk_quiz_qa.
  • num_beams: 1 is removed from all generative tasks, which previously broke evaluation with vLLM.
  • nortruthfulqa_gen's doc_to_target is changed from " " to best_answer, to work properly with few-shot evaluation.
  • NRK-Quiz-QA is pinned to version v1.0 of ltg/nrk_quiz_qa with dataset_kwargs: revision: v1.0, so results stay comparable to previous evaluations. The dataset's new v2.0 removes 140 examples.

NorEval v1.0

Choose a tag to compare

@vmkhlv vmkhlv released this 23 Sep 15:37

First versioned release of NorEval, the Norwegian language understanding and generation evaluation benchmark described in our ACL 2025 Findings paper.

This release includes:

  • 24 datasets for Norwegian Bokmål and Nynorsk, integrated into LM Evaluation Harness (lm_eval/tasks/noreval)
  • Annotation guidelines for human baselines, prompt writing, NorRewrite-Instruct and NorSummarize-Instruct
  • Example task groups and result-processing scripts in tools/