Is hidden state missing in UniXcoder implementation?

[oathaha]

Normally, a transformer model will take the last hidden state of an encoder as the input to a decoder. However, I saw that the hidden state obtained from the encoder of UniXcoder is not used as the decoder's input to generate output. Actually, I only saw that the variable past_hidden_state is used. So I'm not sure if the implementation is correct.

[guoday]

Normally, a transformer model will take the last hidden state of an encoder as the input to a decoder.
For encoder-decoder model, yes, it's right. However, UniXcoder is based on Unilm which leverages mask attention matrix for generation. The last hidden state is not used as the decoder's input but for predicting next token.

[oathaha]

I see. Thanks.