WMT2017 Czech-English machine translation challenge for news

Translate news articles from Czech into English.

This is WMT2017 news challenge reformatted as a Gonito.net challenge, all the data were taken from http://www.statmt.org/wmt17/translation-task.html.

BLEU is used as the evaluation metric.

Directory structure

• README.md — this file
• config.txt — configuration file
• train/ — directory with training data
• train/commoncrawl.tsv.xz — Common Crawl parallel corpus
• train/news-commentary-v12.tsv.xz — News Commentary parallel corpus
• train/europarl-v7.tsv.xz — Europarliament parallel corpus
• dev-0/ — directory with dev (test) data (Newstest 2013)
• dev-0/in.tsv — German input text for the dev set
• dev-0/expected.tsv — English reference translation for the dev set
• test-A — directory with test data
• test-A/in.tsv — German input data for the test set (WMT2017 test set)
• test-A/expected.tsv — English reference translation for the test set

Training sets

All training sets were compressed with xz, use xzcat to decompress:

$ xzcat train/*.tsv.xz | ...

The pairs where German or English side is empty were removed from the training sets.

Test sets

Reference English translations in the dev and test sets is not tokenised.

Monolingual data

Monolingual data was not included here.