Repository navigation
NorEval v1.1
Warning
The Tatoeba English ↔ Bokmål/Nynorsk tasks (tatoeba_eng_nob, tatoeba_eng_nno, tatoeba_nob_eng, tatoeba_nno_eng) no longer work in v1.1 and fail with BuilderConfig 'eng-nob' not found: v1.1 loads the latest version of Helsinki-NLP/tatoeba_mt, which was replaced with a new benchmark release on 29 September 2026 that no longer has these subsets. Please use v1.1.1, which pins every dataset to a fixed version.
NorEval v1.1 updates the task configs to the versions used in our Norwegian LLM dashboard, covering all 35 tasks. The tasks are plain LM Evaluation Harness configs: clone this repository and pass it to lm_eval with --include_path (see the README).
Changelog
- New metadata: Every task now explicitly declares the primary metric, random-baseline performance, category and language(s). The order of metrics is changed to list the primary metric first.
- New task: SLIDE is a language identification task with five choices: Bokmål, Nynorsk, Danish, Swedish, or other language. One sentence can be compatible with multiple languages, a prediction is considered correct if it matches any of them.
- New task: NoCoLA, BLiMP-like Norwegian linguistic acceptability task.
- New task: Northern Sámi MultiBLiMP is actually a group of tasks, we evaluate each linguistic feature separately to get more informative results (unlike the official
lm-evalMultiBLiMP). - New task: NorOpenBookQA without facts is a simple rephrasing of the original NorOpenBookQA so that it evaluates knowledge rather then reading comprehension.
- New task: Bokmål
↔️ Nynorsk translation based on the NorSumm dataset, which contains multi-sentence manual translations. - New task: Bokmål
↔️ Northern Sámi translation based on ltg/saami-tatoeba. - NorBelebele is now based on the corrected Bokmål passages from FLORES+ -- ltg/norbelebele.
- NorBelebele has two additional prompts, so that it's also evaluated in cloze formulation.
- ASK-GEC is now scored inside the harness, not in an extra post-production step.
- NorIdiom gets a new primary metric
em_firstand a less aggressive stopping criterion. - The references in NorSumm are cleaned -- we strip newlines from the reference summaries, which broke few-shot evaluation previously.
- Hard-coded
num_fewshot: 0are removed fromncb,noridiom,norsumm,nortruthfulqa_gen,nortruthfulqa_mcandnrk_quiz_qa. num_beams: 1is removed from all generative tasks, which previously broke evaluation with vLLM.nortruthfulqa_gen'sdoc_to_targetis changed from" "tobest_answer, to work properly with few-shot evaluation.- NRK-Quiz-QA is pinned to version
v1.0of ltg/nrk_quiz_qa withdataset_kwargs: revision: v1.0, so results stay comparable to previous evaluations. The dataset's new v2.0 removes 140 examples.