Problem in Running Example Code for translate_enzh_wmt8k

[heronyang]

I tried to run the example commands in the README file on translate_enzh_wmt8k. But, I was not able to get Chinese words after decoding. The result contains some byte characters separated by white spaces but Chinese characters.

Commands I used for data generating, training and decoding:

sudo apt-get update
sudo apt-get install -y python-pip
pip install --upgrade pip
pip install --user tensorflow tensor2tensor

PROBLEM=translate_enzh_wmt8k
MODEL=transformer
HPARAMS=transformer_base_single_gpu

DATA_DIR=$HOME/t2t_data
TMP_DIR=/tmp/t2t_datagen
TRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS

mkdir -p $DATA_DIR $TMP_DIR $TRAIN_DIR

t2t-datagen \
  --data_dir=$DATA_DIR \
  --tmp_dir=$TMP_DIR \
  --problem=$PROBLEM

t2t-trainer \
  --data_dir=$DATA_DIR \
  --problems=$PROBLEM \
  --model=$MODEL \
  --hparams_set=$HPARAMS \
  --output_dir=$TRAIN_DIR

DECODE_FILE=$DATA_DIR/decode_this.txt
echo "Hello world" >> $DECODE_FILE
echo "Goodbye world" >> $DECODE_FILE

BEAM_SIZE=4
ALPHA=0.6

t2t-decoder \
  --data_dir=$DATA_DIR \
  --problems=$PROBLEM \
  --model=$MODEL \
  --hparams_set=$HPARAMS \
  --output_dir=$TRAIN_DIR \
  --decode_hparams="beam_size=$BEAM_SIZE,alpha=$ALPHA" \
  --decode_from_file=$DECODE_FILE

The decoded result:

$ cat ~/t2t_data/decode_this.txt.transformer.transformer_base_single_gpu.translate_enzh_wmt8k.beam4.alpha0.6.decodes
� � � � � � � � � � � � � � � � � � � � � � � � \ 2 0 1 9 9
� � � � � � � � � � � � � � � � � � � � � � � � \ 2 0 1 9 9
$ od ~/t2t_data/decode_this.txt.transformer.transformer_base_single_gpu.translate_enzh_wmt8k.beam4.alpha0.6.decodes
0000000 020344 020270 020252 020344 020270 020252 020344 020273
0000020 020247 020344 020273 020247 020344 020273 020247 020344
0000040 020275 020264 020344 020275 020264 020344 020275 020264
0000060 020134 020062 020060 020061 020071 005071 020344 020270
0000100 020252 020344 020270 020252 020344 020273 020247 020344
0000120 020270 020252 020344 020273 020247 020344 020275 020264
0000140 020344 020275 020264 020344 020275 020264 020134 020062
0000160 020060 020061 020071 005071
0000170

[twairball]

TranslateEnzhWmt8k actually has the datasets defined as Chinese -> English. So that's quite confusing.

See #315, you can try PROBLEM=translate_enzh_wmt8k_rev

[colmantse]

another possibility is the chcp code. change to chcp 65001 and see.

[lukaszkaiser]

I suppose we should change the name from translate_enzh_wmt8k to translate_enzh_wmt8k right? Is there anything else we should do to correct this in the next release? Thanks for your suggestions!

[heronyang]

I am still not getting any meaningful result when taking Chinese as input on PROBLEM=translate_enzh_wmt8k. I ran the decode part when the trainer reports the loss is around 1.x. It's easy to reproduce the problem I have by using the script listed above.

I got multiple white spaces if the input is "早安" (good morning).

[colmantse]

Hi, I have a similar problem, but I think it should be data problem, I got better results with webcrawl data, at least with variety and sometimes making sense. I suppose the lack of data and training steps contribute to the problem.

[twairball]

The author also noted that the results are quite bad: #151

My opinion is that while the loss, perplexity and estimated BLEU scores look terrible they are consistent with what I've seen building models on fairseq using same limited dataset.

I think it should be data problem, not due to Transformer or tensor2tensor itself, but why or what to do to improve it is still a question.

[colmantse]

Interestingly seems zhen formal data such as UN data seems often underperform comparing with web crawl data. Would be great if someone could verify if this is the case as well.

[rsepassi]

Seems that this is fixed now.