Repository navigation
HPLT-e v2.0 updates the task configurations for recent versions of LM Evaluation Harness (tested with v0.4.13), datasets (v5) and transformers (v5). The version used in our LREC 2026 paper is v1.0.
Compatibility
- Each task family is a single folder with its own
utils.py: recent LM Evaluation Harness versions resolve!functiononly relative to the YAML file, so references such as!function ../utils.fnno longer load. - The SQuAD-style metrics (
compute_exact,compute_f1) are included inutils.py;transformers.data.metricswas removed intransformersv5. - No dataset needs a loading script any more (
datasetsv4 and later cannot run them), and dataset IDs are updated to their current names on the Hugging Face Hub.
Tags
All prompts of a task can be run with its tag hplte_<language>_<task> (e.g. --tasks hplte_spa_xnli); the tags and their tasks are listed in configs/tags.json. The Czech BenCzechMark tags follow the same scheme. Global-MMLU, EusExams and Umimeto have a group per prompt instead.
Datasets
All datasets are pinned with dataset_kwargs.revision to the data used for the HPLT-e evaluations: the last commit before 2025-11-18, when the results were published. FLORES and MGSM-es use a later commit with identical data, because the earlier one only provides a loading script; CTKFacts is read from the Hub's Parquet conversion, which is identical to the output of its loading script.
Fixes
coqcat_p2andklokan_qa_p3declared the names of other prompts (coqcat_p1,klokan_qa_p2).eusbelebele_p2had a template typo ({{mc_answer3]}}).- VeritasQA MC2 with the second prompt (Catalan and Spanish) failed on items with more than 14 answer options.
- EusExams has separate subtasks for each prompt (
eus_exams_eu_<exam>_p<N>); the three prompt groups shared subtask names before. - Umimeto has one group per prompt (
umimeto_p0-umimeto_p4), as in the published results. - Prompt fixes.
Removed
- WNLI-es and QNLIeu, which are not part of the published results.
- Unused BenCzechMark templates and
scripts/run.sh.
Documentation
- Paper links, the NorEval v1.0 release, the FinBench v2 paper and homepage, a simplified Norwegian task table, and corrected prompt ranges (CTKFacts
p[0-7], Tatoebap[0-3]) and task names (flores_en-gl, Czech Global-MMLUp[0-2]).
Verification
Every task was compared with the v1.0 configuration (run with LM Evaluation Harness v0.4.9.1 and datasets v3.6) at the same dataset revisions: the evaluation documents, all zero-shot requests and the metric outputs are identical, apart from the fixes listed above.