Separate weights for word embedding and lm-head?

[zeyun-zhong]

Hi, thanks for providing the code.

I have a question regarding the word embedding and lm-head. In your code, both functions shard the same weight. I wonder if they can have separate weights?

Thanks for your help!

[summmeer]

Hi,
The lm-head weights (word embedding) are used for decoding. You can add extra weighting parameters before lm-head to build a better decoder.

[zeyun-zhong]

Hi,
thank you for the reply!
Wish you a happy new year :).