TransformerEncoder is not causal

[bigheary]

Add Link

https://pytorch.org/tutorials/beginner/transformer_tutorial.html

for language modeling， src_mask should be mask future words

Describe the bug

is there anything wrong?

Describe your environment

colab

cc @pytorch/team-text-core @Nayef211 @sekyondaMeta @svekars @carljparker @NicolasHug @kit1980 @subramen

[jorendorff]

Yes, this has to be a bug. I tried using the resulting model to do some basic predictions and it was never able to give a prediction for the next word that made any sense, despite reaching a perplexity of 2.9 on the validation and test data. Without a mask, every word except the last (the one that isn't present in the input) should be easy for the model to predict flawlessly.

[jorendorff]

You can play with it here (a copy of the notebook with a cell added at the end for making predictions)

[xanderex-sid]

/assigntome

[nvs-abhilash]

/assigntome

[xanderex-sid]

/assigntome