Skip to content

HPLT-e v2.0

Latest

Choose a tag to compare

@vmkhlv vmkhlv released this 08 Nov 16:10

HPLT-e v2.0 updates the task configurations for recent versions of LM Evaluation Harness (tested with v0.4.13), datasets (v5) and transformers (v5). The version used in our LREC 2026 paper is v1.0.

Compatibility

  • Each task family is a single folder with its own utils.py: recent LM Evaluation Harness versions resolve !function only relative to the YAML file, so references such as !function ../utils.fn no longer load.
  • The SQuAD-style metrics (compute_exact, compute_f1) are included in utils.py; transformers.data.metrics was removed in transformers v5.
  • No dataset needs a loading script any more (datasets v4 and later cannot run them), and dataset IDs are updated to their current names on the Hugging Face Hub.

Tags

All prompts of a task can be run with its tag hplte_<language>_<task> (e.g. --tasks hplte_spa_xnli); the tags and their tasks are listed in configs/tags.json. The Czech BenCzechMark tags follow the same scheme. Global-MMLU, EusExams and Umimeto have a group per prompt instead.

Datasets

All datasets are pinned with dataset_kwargs.revision to the data used for the HPLT-e evaluations: the last commit before 2025-11-18, when the results were published. FLORES and MGSM-es use a later commit with identical data, because the earlier one only provides a loading script; CTKFacts is read from the Hub's Parquet conversion, which is identical to the output of its loading script.

Fixes

  • coqcat_p2 and klokan_qa_p3 declared the names of other prompts (coqcat_p1, klokan_qa_p2).
  • eusbelebele_p2 had a template typo ({{mc_answer3]}}).
  • VeritasQA MC2 with the second prompt (Catalan and Spanish) failed on items with more than 14 answer options.
  • EusExams has separate subtasks for each prompt (eus_exams_eu_<exam>_p<N>); the three prompt groups shared subtask names before.
  • Umimeto has one group per prompt (umimeto_p0-umimeto_p4), as in the published results.
  • Prompt fixes.

Removed

  • WNLI-es and QNLIeu, which are not part of the published results.
  • Unused BenCzechMark templates and scripts/run.sh.

Documentation

  • Paper links, the NorEval v1.0 release, the FinBench v2 paper and homepage, a simplified Norwegian task table, and corrected prompt ranges (CTKFacts p[0-7], Tatoeba p[0-3]) and task names (flores_en-gl, Czech Global-MMLU p[0-2]).

Verification

Every task was compared with the v1.0 configuration (run with LM Evaluation Harness v0.4.9.1 and datasets v3.6) at the same dataset revisions: the evaluation documents, all zero-shot requests and the metric outputs are identical, apart from the fixes listed above.