Skip to content

LTB v0.1.6.4 — 16 language pairs + commercial-safe MT runner

Choose a tag to compare

View all tags
@Lawrenzho-bit Lawrenzho-bit released this 20 May 16:36
· 16 commits to main since this release
v0.1.6.4
1e7d1cd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

LayoutTranslateBench (LTB) is the first public benchmark for document translation that scores layout fidelity and reading order alongside translation quality. This release is the v0.1.6 series checkpoint — the canonical citeable artifact for the 16-pair coverage tier.

What landed in v0.1.6

Sub-release Headline
v0.1.6 LangPair extended from 8 to 16. Added: en-ru, en-ko, en-vi, en-id, en-ur, en-uz, en-kk, en-zh-tw. CORE_LANG_PAIRS frozen at the v0.1 core 8 so author-curated docs keep clean coverage. Language-detection gate gets script rules for non-Latin extension pairs. NLLB code map covers all 16.
v0.1.6.1 NLLB-200 extension rerun + macro/micro aggregation consistency fix.
v0.1.6.2 Helsinki-NLP/opus-mt runner — 14 models, full 16/16 coverage, commercial-safe (Apache-2.0 / CC-BY-4.0). Ship-able where NLLB-200's CC-BY-NC-4.0 is not.
v0.1.6.3 Methodology note documenting opus-mt vs NLLB per-pair quality variance — competitive on European pairs, dramatically weaker on Asian / Central Asian.
v0.1.6.4 fugumt en-ja swap (replaced opus-mt-en-jap with staka/fugumt-en-ja). Script-validation gate at ingest + post-hoc scripts/validate_extension_refs.py drops rileykim refs whose tgt_text script doesn't match the expected target-language script. Surfaced en-ru rileykim rows that were actually Chinese.

Coverage after v0.1.6.4

Group Pairs N per pair
Core 8 en-es, en-de, en-ar, en-fr, en-th, en-ms N=20
Core 8 (extended) en-ja N=28
Core 8 (extended) en-zh N=27
Extension 7 (surviving) en-ko, en-vi, en-id, en-ur, en-uz, en-kk, en-zh-tw N=3 each
Extension 1 (dropped) en-ru N=0 (rileykim labeling bug, refs in Chinese)

Leaderboard — oracle-layout (COMET-Kiwi-22, higher = better)

System LTB-100 [95% CI] Coverage
nllb-text-oracle-nllb-200-distilled-600m 86.95 [85.8, 88.0] 15/16
deepl-text-oracle 84.89 [81.4, 88.0] 6/16
opus-mt-text-oracle 80.46 [78.9, 82.1] 15/16

Leaderboard — end-to-end (COMET-Kiwi-22)

System LTB-100 [95% CI] Coverage
identity-baseline 50.66 [50.4, 51.0] 15/16
qwen3-vl-2b-instruct 22.79 [18.6, 27.1] 8/16

Methodology

  • Two parallel leaderboards (chrF + COMET-Kiwi-22)
  • Bootstrap 95% CIs via 1000 resamples (seed 42)
  • Coverage-aware Kendall τ (no free credit for single-region fallbacks)
  • Language-detection penalty on chrF (latin bleed-through scores 0)
  • Oracle-layout vs end-to-end systems segregated on the leaderboard
  • Ingest-time + post-hoc script validation for mined references

Full critique tracker at docs/methodology-roadmap.md.

Citation

@misc{ltbench2026,
  title  = {LayoutTranslateBench: A Benchmark for Document Translation with Layout Preservation},
  year   = {2026},
  url    = {https://github.com/Lawrenzho-bit/LayoutTranslateBench},
  note   = {Version 0.1.6.4}
}

What's next (v0.2 scope)

  • Extension pair coverage to N≥10 each (certified or industry-grade refs)
  • Multi-reference scoring (2 certified-translator refs per doc on the core 8)
  • DA / SQM human evaluation on 50 doc-pair outputs across 4+ systems
  • LPIPS visual-fidelity metric (10% weight) + OCR round-trip metric (10% weight)
  • Held-out split rotation (20% private, refreshed quarterly)

Links

Assets 2
Loading