terflip

lm-evaluation-harness attaches a higher_is_better flag to every metric and uses it to decide which of two scores is better and which direction arrow to print. The TER metric is registered with the wrong flag: its own docstring says lower is better, but it is registered higher_is_better=True. Because the harness returns the TER score un-negated, a translation system that makes more errors is reported as the better one.

The contradiction

In lm_eval/api/metrics.py the three machine-translation metrics are defined with a direction note in their docstrings and registered a few hundred lines later with a higher_is_better flag:

metric	docstring note	returns	registered flag	consistent
bleu (L84, L93, L354)	Higher is better	corpus_bleu(...).score	higher_is_better=True	yes
chrf (L102, L108, L364)	Higher is better	corpus_chrf(...).score	higher_is_better=True	yes
ter (L117, L124, L374)	Lower is better	corpus_ter(...).score	higher_is_better=True	no

TER is the Translation Error Rate, an edit-distance error rate returned without negation, so lower is better, exactly as the ter docstring states. BLEU and chrF really are higher-is-better and are flagged correctly. The ter registration carries the same True as its two neighbours, which is a copy of the wrong direction.

What it does

The harness compares scores through the flag and prints an arrow to match. With higher_is_better=True on an un-negated error rate:

TER scores (lower is better): {system_A: 24.0, system_B: 41.0}
harness picks best: system_B          (should be system_A, which has fewer errors)
results-table arrow: up               (should be down)

The system that translates worse is selected as the better one, and an up arrow is printed next to an error rate. Group-level aggregation does not catch it: bleu, chrf, and ter all carry higher_is_better=True, so a group that mixes them looks internally consistent and no conflict is flagged. The bug bites any translation benchmark that reports TER (WMT, FLORES, IWSLT and the FLORES language-pair suites all inherit the registry flag).

Layout

• excerpt.py: the relevant lines of lm_eval/api/metrics.py (MIT), quoted with line numbers so the docstring direction and the registered flag can be compared without installing the harness.
• contradiction.py: for each metric, the direction its docstring implies versus the direction it is registered with; only ter disagrees.
• consequence.py: the harness's own compare-and-arrow logic, showing the winner and the arrow both invert for TER.
• test_terflip.py: ter contradicts and bleu/chrf do not, the ranking inverts, and the arrow inverts.

Reproduce

python contradiction.py
python consequence.py
python test_terflip.py

The direction notes and flags are quoted from lm_eval/api/metrics.py on the main branch; the fix is a one-character change, higher_is_better=False for the ter registration.