Transformer Decoder Support

[hwidong-na]

Hi,

I'm looking for the implementation of the transformer decoder with cross attention to the encoder in DeepSpeed.
I found the pull request #933 was a candidate for this purpose.
Would #933 be also enough to support the transformer decoder?

Best regards,
Hwidong

[RezaYazdaniAminabadi]

Hi @hwidong-na

I think there is possibility of supporting decoder for transformer kernel. I am currently investigating the differences other than the softmax masking as you mentioned. I have seen that HuggingFace is also using position-biasing at the decoder which is borrowed from google implementation. Other than that, the API needs to change in order to send out more outputs.
I will update further on this line.
Thanks,
Reza