Skip to content

NorEval v1.1

Choose a tag to compare

@vmkhlv vmkhlv released this 17 Oct 15:26
· 47 commits to main since this release

Warning

The Tatoeba English ↔ Bokmål/Nynorsk tasks (tatoeba_eng_nob, tatoeba_eng_nno, tatoeba_nob_eng, tatoeba_nno_eng) no longer work in v1.1 and fail with BuilderConfig 'eng-nob' not found: v1.1 loads the latest version of Helsinki-NLP/tatoeba_mt, which was replaced with a new benchmark release on 29 September 2026 that no longer has these subsets. Please use v1.1.1, which pins every dataset to a fixed version.

NorEval v1.1 updates the task configs to the versions used in our Norwegian LLM dashboard, covering all 35 tasks. The tasks are plain LM Evaluation Harness configs: clone this repository and pass it to lm_eval with --include_path (see the README).

Changelog

  • New metadata: Every task now explicitly declares the primary metric, random-baseline performance, category and language(s). The order of metrics is changed to list the primary metric first.
  • New task: SLIDE is a language identification task with five choices: Bokmål, Nynorsk, Danish, Swedish, or other language. One sentence can be compatible with multiple languages, a prediction is considered correct if it matches any of them.
  • New task: NoCoLA, BLiMP-like Norwegian linguistic acceptability task.
  • New task: Northern Sámi MultiBLiMP is actually a group of tasks, we evaluate each linguistic feature separately to get more informative results (unlike the official lm-eval MultiBLiMP).
  • New task: NorOpenBookQA without facts is a simple rephrasing of the original NorOpenBookQA so that it evaluates knowledge rather then reading comprehension.
  • New task: Bokmål ↔️ Nynorsk translation based on the NorSumm dataset, which contains multi-sentence manual translations.
  • New task: Bokmål ↔️ Northern Sámi translation based on ltg/saami-tatoeba.
  • NorBelebele is now based on the corrected Bokmål passages from FLORES+ -- ltg/norbelebele.
  • NorBelebele has two additional prompts, so that it's also evaluated in cloze formulation.
  • ASK-GEC is now scored inside the harness, not in an extra post-production step.
  • NorIdiom gets a new primary metric em_first and a less aggressive stopping criterion.
  • The references in NorSumm are cleaned -- we strip newlines from the reference summaries, which broke few-shot evaluation previously.
  • Hard-coded num_fewshot: 0 are removed from ncb, noridiom, norsumm, nortruthfulqa_gen, nortruthfulqa_mc and nrk_quiz_qa.
  • num_beams: 1 is removed from all generative tasks, which previously broke evaluation with vLLM.
  • nortruthfulqa_gen's doc_to_target is changed from " " to best_answer, to work properly with few-shot evaluation.
  • NRK-Quiz-QA is pinned to version v1.0 of ltg/nrk_quiz_qa with dataset_kwargs: revision: v1.0, so results stay comparable to previous evaluations. The dataset's new v2.0 removes 140 examples.