problem TranslateEnzhWmt8k

[mosesli]

I run the Chinese-to-English translation task as the default set, the "t2t-decoder" scripts can not translate Chinese segment into English sentence, but only outputs some short Chinese sentence:

INFO:tensorflow:Inference results INPUT: 雇主们 希望 继续 积极 探讨 其他 重要 话题 —— 如 部署 、 培训 的 灵活性 、 针对 那些 在 职业 生涯 中断 后重 上岗 之 人 的 其他 培训 、 培训 成本 、 对 课程 提纲 的 共同 认知 、 进修 假 和 医疗 领域 的 性别 工资 差异 ， 这些 说明 了 雇主 、 英格兰 健康 教育 组织 ( Health Education England ) 和 卫生部 对 11月 、 2月 及 5月 与 BMA 达成 的 协议 很 重视 。
INFO:tensorflow:Inference results OUTPUT: 这 是 一 个 错误 。
INFO:tensorflow:Inference results INPUT: 在 该 法 院本 周 的 听证会 上 ， 加莱 政府 辩称 法国 政府 应 有 权 拆除 该 难民营 内 总计 72 间 临时 搭建 的 商店 。 民众 在 难民营 中的 这些 商店 里 销售 从 香烟 到 能量 饮料 在内 的 商品 、 提供 理发 服务 ， 还 在 临时 搭建 的 餐馆 中 以 50 美分 （ 40 便士 ） 的 价格 卖茶 或 以 几 欧 元 的 价格 售卖 基本 餐点 。
INFO:tensorflow:Inference results OUTPUT: 但 这 一 比例 并非 如此 。

although the outputs of "t2t-datagen" and "t2t-trainer" scripts seems fine:

INFO:tensorflow:Saving dict for global step 250000: global_step = 250000, loss = 1.29398, metrics-translate_enzh_wmt8k/accuracy = 0.622824, metrics-translate_enzh_wmt8k/accuracy_per_sequence = 0.0, metrics-translate_enzh_wmt8k/accuracy_top5 = 0.892174, metrics-translate_enzh_wmt8k/approx_bleu_score = 0.452572, metrics-translate_enzh_wmt8k/neg_log_perplexity = -1.56626, metrics-translate_enzh_wmt8k/rouge_2_fscore = 0.756268, metrics-translate_enzh_wmt8k/rouge_L_fscore = 0.538652

I have two question:
(1) Why the Chinese-to-English translation task do not work, and the problem name "TranslateEnzhWmt8k" seems to mean translating English to Chinese.
(2) Why not split the Chinese sentence into Chinese words before training and decoding~

Thanks in advance!

[colmantse]

Hi, have you tried feed in English sentences to it and see if it works properly?
tokenization is done in the script with a vocab size of 8k.

[martinpopel]

I agree it is misleading (and I would suggest to fix it) that "TranslateEnzhWmt8k" internally uses _ZHEN_TRAIN_DATASETS, source_vocab is zh and target_vocab is en. However, there is a trick where instead of "lang1, lang2", it uses "lang2, lang1". So in the end I guess translate_enzh_wmt8k is (as the name suggests) translation from English to Chinese. If you need Chinese to English, try translate_enzh_wmt8k_rev.
(I am not the author. I have not even tried zh translation. This is just my guess based on the source code.)

[mosesli]

@colmantse @martinpopel, thanks for your kind help.
I also try translate English to Chinese using the "t2t-decoder" script with the same trained model, but the outputs are also not right:

2017-09-21 20:48:13.634873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /home/mosesli/T4/transformer/z2e_train/translate_enzh_wmt8k/transformer-transformer_base_single_gpu/model.ckpt-250000
INFO:tensorflow:Inference results INPUT: employers were hopeful that the continued positive engagement on other important topics - such as deployment , flexibility in training , additional training for those returning from career breaks , costs of training , mutual recognition of syllabus , study leave and the gender pay gap in medicine - were a sign of how serious employers , health education england and the department of health were about honouring the agreements reached with the bma in november , february and may .
INFO:tensorflow:Inference results OUTPUT: 这 是 一 个 错误 。
INFO:tensorflow:Inference results INPUT: " it 's either that they had no diverse representation of people of color on their staff to the point where they 're able to make decisions like this , " ms. zhu , who is chinese-american , said , " or they do have some people of color who are working there , but they 're not on positions where they feel safe or comfortable to speak up . "
INFO:tensorflow:Inference results OUTPUT: 这 是 一 个 错误 。

I notice that the corresponding codes in "wmt.py" is change from lines start with "-" to lines start with "+" on 17 Aug by rsepassi, but I donot understand it.

-@registry.register_problem("wmt_zhen_tokens_8k") # line 443
-class WMTZhEnTokens8k(WMTProblem): #line 444
+@registry.register_problem
+class TranslateEnzhWmt8k(TranslateProblem):
+"""Problem spec for WMT Zh-En translation."""

-return bi_vocabs_token_generator(data_path + ".lang1", data_path + ".lang2",
-#We generate English->X data by convention, to train reverse translation
+# just add the "_rev" suffix to the problem name, e.g., like this.
+#--problems=translate_enzh_wmt8k_rev
+return bi_vocabs_token_generator(data_path + ".lang2", data_path + ".lang1", source_vocab, target_vocab, EOS)

[colmantse]

• means remove, + means add. It looks like you have trained a chi --> en model, what is your loss and bleu?

[mosesli]

Since the translation outputs are obviously wrong, I do not measure the BLEU score on the test set, but the results when training are as follows:

INFO:tensorflow:loss = 0.434695, step = 249601 (17.925 sec)
INFO:tensorflow:global_step/sec: 5.56531
INFO:tensorflow:loss = 0.395525, step = 249701 (17.968 sec)
INFO:tensorflow:global_step/sec: 5.55805
INFO:tensorflow:loss = 0.370602, step = 249801 (17.992 sec)
INFO:tensorflow:global_step/sec: 5.54219
INFO:tensorflow:loss = 0.401325, step = 249901 (18.044 sec)
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236], 'window_size': 720, 'max_length': 1000000000, 'batch_sizes': [240, 180, 180, 180, 144, 144, 144, 120, 120, 120, 90, 90, 90, 90, 80, 72, 72, 60, 60, 48, 48, 48, 40, 40, 36, 30, 30, 24, 24, 20, 20, 18, 18, 16, 15, 12, 12, 10, 10, 9, 8, 8], 'shuffle_queue_size': 270}
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Unsetting shared_embedding_and_softmax_weights.
INFO:tensorflow:Doing model_fn_body took 1.501 sec.
INFO:tensorflow:This model_fn took 1.705 sec.
INFO:tensorflow:Starting evaluation at 2017-09-13-16:05:08
2017-09-14 00:05:09.127576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /home/mosesli/T4/transformer/z2e_train/translate_enzh_wmt8k/transformer-transformer_base_single_gpu/model.ckpt-248416
INFO:tensorflow:Evaluation [1/10]
INFO:tensorflow:Evaluation [2/10]
INFO:tensorflow:Evaluation [3/10]
INFO:tensorflow:Evaluation [4/10]
INFO:tensorflow:Evaluation [5/10]
INFO:tensorflow:Evaluation [6/10]
INFO:tensorflow:Evaluation [7/10]
INFO:tensorflow:Evaluation [8/10]
INFO:tensorflow:Evaluation [9/10]
INFO:tensorflow:Evaluation [10/10]
INFO:tensorflow:Finished evaluation at 2017-09-13-16:05:22
INFO:tensorflow:Saving dict for global step 248416: global_step = 248416, loss = 1.32257, metrics-translate_enzh_wmt8k/accuracy = 0.627322, metrics-translate_enzh_wmt8k/accuracy_per_sequence = 0.0, metrics-translate_enzh_wmt8k/accuracy_top5 = 0.888772, metrics-translate_enzh_wmt8k/approx_bleu_score = 0.451552, metrics-translate_enzh_wmt8k/neg_log_perplexity = -1.60012, metrics-translate_enzh_wmt8k/rouge_2_fscore = 0.758087, metrics-translate_enzh_wmt8k/rouge_L_fscore = 0.539738
INFO:tensorflow:Validation (step 250000): loss = 1.32257, metrics-translate_enzh_wmt8k/neg_log_perplexity = -1.60012, metrics-translate_enzh_wmt8k/accuracy = 0.627322, metrics-translate_enzh_wmt8k/accuracy_per_sequence = 0.0, metrics-translate_enzh_wmt8k/rouge_2_fscore = 0.758087, metrics-translate_enzh_wmt8k/approx_bleu_score = 0.451552, metrics-translate_enzh_wmt8k/accuracy_top5 = 0.888772, metrics-translate_enzh_wmt8k/rouge_L_fscore = 0.539738, global_step = 248416
INFO:tensorflow:Saving checkpoints for 250000 into /home/mosesli/T4/transformer/z2e_train/translate_enzh_wmt8k/transformer-transformer_base_single_gpu/model.ckpt.
...
INFO:tensorflow:Finished evaluation at 2017-09-13-16:05:42
INFO:tensorflow:Saving dict for global step 250000: global_step = 250000, loss = 1.29398, metrics-translate_enzh_wmt8k/accuracy = 0.622824, metrics-translate_enzh_wmt8k/accuracy_per_sequence = 0.0, metrics-translate_enzh_wmt8k/accuracy_top5 = 0.892174, metrics-translate_enzh_wmt8k/approx_bleu_score = 0.452572, metrics-translate_enzh_wmt8k/neg_log_perplexity = -1.56626, metrics-translate_enzh_wmt8k/rouge_2_fscore = 0.756268, metrics-translate_enzh_wmt8k/rouge_L_fscore = 0.538652

  The final loss is 1.29398, and the BLEU score is 0.452572.

[colmantse]

hi, i am not sure what has gone wrong, but it might be due to small data size that it did not work so well? I suggest you use some dev set sentence and check how they perform, at least then you would be able to figure if it output chi or en.

[mosesli]

@colmantse @martinpopel OK, thank you very much!
Finally, I change the line 502 in data_generators/wmt.py from

-#return bi_vocabs_token_generator(data_path + ".lang2", data_path + ".lang1", source_vocab, target_vocab, EOS)

to

+return bi_vocabs_token_generator(data_path + ".lang1", data_path + ".lang2", source_vocab, target_vocab, EOS)

then it can translate Chinese to English, and the BLEU score on the test set of newstest2017-zhen is about 11.07 with the default datasets and parameters.

[yynil]

I think I can answer the question why the words split is not done. From our experiments, the character based Chinese encoding leads to better results than the one using Chinese word split. More corpus you feed, better results you will get. The transformer model can learn how to form words from characters and it's a better way to do this in domain specific corpus from my point of view.

[lukaszkaiser]

I suppose we should change the name from translate_enzh_wmt8k to translate_enzh_wmt8k right? Is there anything else we should do to correct this in the next release? Thanks for your suggestions!

[martinpopel]

change the name from translate_enzh_wmt8k to translate_enzh_wmt8k

I guess, you mean from translate_enzh_wmt8k to translate_zhen_wmt8k.
I have another suggestion: follow the style of other datasets, that is:

• change _ZHEN_TRAIN_DATASETS to _ENZH_TRAIN_DATASETS
• get rid of the hack where instead of "lang1, lang2", it uses "lang2, lang1"
• This way the default will stay en->zh as it is now. Whoever wants zh->en, must use translate_enzh_wmt8k_rev.