Companion to v0.1.7 — finalizes all system evaluations against the new certified-translator FLORES references.
Changes
- Re-ran NLLB-200-distilled-600M for all 8 extension pairs (101 translations, 371 skipped via
skip_no_ref); scored against v0.1.7 refs → 16/16 pair coverage - Re-ran Helsinki-NLP/opus-mt for all 8 extension pairs (101 translations, 1 207 s total) → 16/16 pair coverage
- Re-scored DeepL and identity-baseline against v0.1.7 manifest
- Fixed
skip_no_refbug inrun_nllb(parameter declared but unused in loop body) - Rebuilt leaderboard (chrF); COMET-Kiwi re-score pending
Scores (chrF, oracle-layout, v0.1.7)
| System | LTB-100 [95% CI] | chrF | Coverage |
|---|---|---|---|
| DeepL | 78.20 [75.2, 81.7] | 56.40 | 6/16 |
| NLLB-200-distilled-600M | 73.71 [72.5, 74.9] | 47.61 | 16/16 |
| Helsinki-NLP/opus-mt | 68.60 [67.1, 70.0] | 37.26 | 16/16 |
| identity-baseline | 50.38 [50.2, 50.6] | 0.62 | 16/16 |
Notable per-pair findings
- en-uz: chrF ≈ 0 for both NLLB (0.07) and opus-mt (0.04) — Uzbek-Latin is a structural zero for 600M-class open models
- en-ko: NLLB 36.88 chrF vs opus-mt 2.58 — NLLB wins decisively; Helsinki opus-mt-en-ko model is inadequate
- en-id / en-ru: both systems score well (chrF 60–70+), confirming FLORES refs are clean
All scores: 1 000-resample percentile bootstrap, seed 42.