Issue with BERT implementation

[ajfisch]

Hi,

It seems that you're trying to decode auto-regressively using BERT representations as a drop-in replacement for word embeddings. But BERT is bi-directional; the representation at token i has information about all tokens j > i. So, your model already knows what it needs to predict, before it predicts it.

In order for this to be correct you need to mask attention to all tokens j > i, which I don't think you do currently.

[leyuan]

@ajfisch I think you are right, by any chance you have fixed the issue?

[enes3774]

@ajfisch I think you are right, by any chance you have fixed the issue?
I think you have to use autoregressive models such as gpt2. I think https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning is a high enough model for image captioning.