chrfshort

nltk's chrF averages the character-F-score over the n-gram orders 1 to max_len (6 by default), but it divides the summed per-order scores by max_len regardless of how many orders actually have n-grams. A string shorter than max_len characters has fewer than max_len non-empty orders, so its score is deflated by the ratio of present orders to max_len. A hypothesis identical to its reference, which must score 1.0, scores length / 6: the one-character string "c" scores 0.1667, not 1.0. The reference chrF (chrF++ and sacrebleu) averages only over the orders that have n-grams, so it returns 1.0.

The code

nltk/translate/chrf_score.py, corpus_chrf, current develop (identical in 3.10.0):

for n in range(min_len, max_len + 1):
    prec, rec, fscore, tp = chrf_precision_recall_fscore_support(reference, hypothesis, n, beta=beta)
    ngram_fscores[n].append(fscore)          # appended for every order, even empty ones
...
num_ngram_sizes = len(ngram_fscores)          # 6, all orders 1..max_len are keys
total_scores = [sum(fscores) for n, fscores in ngram_fscores.items()]
return (sum(total_scores) / num_ngram_sizes) / num_sents   # divides by 6

An order with no n-grams (a string shorter than that order) returns a near-zero epsilon from the helper (except ZeroDivisionError: prec = rec = fscore = epsilon), and that epsilon is still appended and counted in the average. So the denominator is always max_len, the number of requested orders, not the number of orders that actually have n-grams. chrF is defined to average over the effective order, the orders that have n-grams, which is what chrF++ and sacrebleu do; the nltk docstring even claims equivalence to chrF++.py -nw 0 -b 3.

What it does

Driving real nltk against the reference sacrebleu chrF on identical hypothesis and reference:

'c' (len 1): nltk 0.1667, sacrebleu 1.0  WRONG (should be 1.0)
'OK' (len 2): nltk 0.3333, sacrebleu 1.0  WRONG (should be 1.0)
'Yes' (len 3): nltk 0.5, sacrebleu 1.0  WRONG (should be 1.0)
'hello' (len 5): nltk 0.8333, sacrebleu 1.0  WRONG (should be 1.0)
'abcdef' (len 6): nltk 1.0, sacrebleu 1.0  match
'identical' (len 9): nltk 1.0, sacrebleu 1.0  match

A perfect match of a string of length L < 6 scores L / 6: the present orders each score 1, so the sum is L, but nltk divides by 6. The reference returns 1.0. The base case confirms the harness and isolates the fault: a string of at least max_len = 6 characters has all six orders present, so the denominator 6 equals the true order count and nltk matches sacrebleu exactly, which is why the doctests (long sentences) pass and the bug hides. Short model outputs, "Yes", "No", "OK", short chat or question-answering replies, and short-segment translation, all sit below six characters and are scored too low.

Scope

The bug is on current develop and fires on the default parameters (max_len = 6) for any hypothesis or reference shorter than six characters, non-adversarial ordinary input. The effect is a deflation by present_orders / max_len, up to sixfold. The fix is to divide by the number of orders that have n-grams, the effective order, as chrF++ and sacrebleu do.

Layout

• excerpt.py: the corpus_chrf averaging and the empty-order epsilon quoted with the two flags (Apache-2.0).
• chrf.py: the per-order F-scores of a perfect match and the average over the requested orders (nltk) versus the effective order (correct).
• consequence.py: the real nltk chrF versus sacrebleu on identical strings, the one-character case, and the length-six base case.
• test_chrfshort.py: a length-L perfect match is L / max_len under nltk and 1.0 under the effective order, lengths at least max_len match, and the real nltk short strings are deflated while sacrebleu is 1.0.

Reproduce

python chrf.py
python consequence.py
python test_chrfshort.py

The averaging is quoted from current develop; the values are produced by the real nltk chrF and the reference sacrebleu. The fix is to divide by the effective order, the number of n-gram orders that have n-grams.