[BUG] Decode with attention_lm (with or without MOE)

[vince62s]

This issue has been reported in various ways in #302 #337 #53 #86 #299

It is unclear whether t2t-decoder is supposed to work with attenion_lm models.

As I have observed it does not work.
Sunsequent question is, what is the ouput supposed to be ?
in a "normal" situation, the output should be the input string shifted one token right.

Can someone @lukaszkaiser @rsepassi check this issue ?

output for attention_lm is nonsense
output for attention_lm_moe is nonsense AND log carries some strange warnings.

Thanks !

[rsepassi]

Can you please provide training and decoding command lines?

[alexcdot]

I'm seeing a similar problem with t2t-decoder and attention_lm.
tensorflow-gpu 1.4.1
tensor2tensor 1.3.2
Model trained to checkpoint 142959, using: December 1st, Simple English Wikipedia as opposed to the default /enwiki/20170620/ data source (the articles and titles are almost the same format, just simplewiki is a smaller dataset by x10).

t2t-trainer   --generate_data   --data_dir=~/t2t_wiki_data   --problems=languagemodel_wiki_full32k   --model=attention_lm   --hparams_set=attention_lm_base   --output_dir=~/t2t_train/wiki_base --hparams='batch_size=5'

trains the model to negative log perplexity of -15. Using

t2t-decoder  --data_dir=~/t2t_wiki_data   --problems=languagemodel_wiki_full32k   --model=attention_lm   --hparams_set=attention_lm_base   --output_dir=~/t2t_train/wiki_base --decode_beam_size=1 --decode_alpha=0.6 --decode_batch_size=2 --decode_interactive

From an input such as "Food", or "Computer Science" I will always get this output:
"< < < < < < < < < <..."

The training loss on the model seems to be decreasing, but I'm not sure why the decoding is non-sensical.

[rsepassi]

For others in #496 and #536, the outputs are empty. This does seem to be a bug in how LMs are dealt with in decoding.

[wachaong]

@rsepassi when can this bug be fixed ？seems no one give an answer

[rasmi]

Is this also related to #435? I'm getting individual outputs properly, but when I set beam_size and return_beams=True all the results are empty. Could it be an issue with how the beam search is returning results? @nshazeer @rsepassi @lukaszkaiser let me know if I can help debug.

[twilightdema]

Did you guys manage to overcome the issue?
I am facing the same situation here. Every transformer_moe, both Encoder-Decoder or Decoder only generates strange output using t2t-decoder.

In my case, I am trying to do Language Modeling using Decoder only...