Add mask is set for Attn during training.

[yuntang]

In Decoder.forward, no mask is set for attention model before attention computation. The softmax will has 0 (padding value) as input and the output will be exp(0)/sum exp(x_i) != 0

[donglixp]

The mask is used in: https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/Translator.py#L130

[magic282]

It seems that this apply mask is not used during training.

[donglixp]

@magic282 There's no mask in during training in the implementation. I'm not sure whether it would make a huge difference.

[jekbradbury]

I don't think it does, but I also haven't run any comparison tests

[vene]

I assumed since the sentences are sorted by length, with small enough batches and large enough datasets, training batches will be fully filled out? Now I'm not sure anymore...

[magic282]

@vene But with option -extra-shuffle, I guess things will be different.

[nelson-liu]

Anecdotally speaking, I ran an informal comparison and it made almost no difference, since as @vene said my dataset was large enough and the batch size was small enough that the majority of batches had no padding.

[vene]

Thanks for checking @nelson-liu, that makes sense!

I wonder if skipping the masking really saves a lot of time during training. With -extra-shuffle it indeed seems like this is a bug, as @magic282 points out. Even with sorted batches, and with a huge number of sentences for each length bin, there will be some unfortunate batches with one sentence of length d+1 and N-1 sentences of length d, where the code does not correctly reflect the intended model, then.

[vince62s]

old thread, if someone is motivated to implement, just reopen.