Is the model a transformer encoder-decoder or just decoder?

[ehsan-soe]

Hi,

Thank you for the very nice and interesting work.
I have a question regarding the model. In the paper it s mentioned that you used the same architecture as GPT, which is a transformer decoder. However, you also talked about input encoder and how you configure the input to encoder.
I was a bit confused, whether you have both encoder and decoder? or it is a language model and then it would be just decoder. And if it is a decoder, then how do you encode your [X_s, X_r] ?
Could you please clarify this. Many thanks

[atcbosselut]

It's a transformer language model. The confusion stems from the fact that the transformer language model is a mix of the original transformer encoder and decoder from the original transformer paper: https://arxiv.org/abs/1706.03762

It has the same architecture as the transformer encoder since there's no cross-attention, but it also masks future tokens as a transformer decoder would do. I used the term "encoder" since the architecture of each block of the transformer language model is the same as the transformer encoder.

The input during training is [X_s, X_r, X_o] where future tokens are masked when computing the self-attention of a particular token.

During generation, the input is [X_s, X_r, Y_o] where Y_o is the set of generated tokens of the phrase object up to that point.

[ehsan-soe]

@atcbosselut Thanks
Can you correct me if I am wrong? During the training you take [X_s; X_r] as seed text and you start predicting from end of X_r? I mean you compute self attention for all tokens up to the current time step but only take into account X_o in the loss function, right?

[atcbosselut]

In theory, this is what happens. In practice, it's all computed in parallel so a loss value is computed for all tokens (except the first token of X_s) in [X_s, X_r, X_o] and you just mask the loss values computed for tokens of X_s and X_r (i.e., multiply them by 0) before performing backpropagation, effectively zeroing the gradients of those intermediate losses.

[ehsan-soe]

@atcbosselut Can you kindly point me to where you did mask out the loss computed for tokens in x_s and x_r in your code? Many thanks

[atcbosselut]

comet-commonsense/src/train/batch.py

Line 46 in 0a8a94b

final_loss = (loss * loss_mask).sum(1)

[guotong1988]

Same question. Thank you.

[atcbosselut]

Is your question answered by the discussion above?

[guotong1988]

Yes. Thank you.