Are the weights of the lm head of the model tied with the word embeddings?

[lvcc2018]

Thanks for the amazing work.
I wonder whether the weights of the lm head of the model are tied with the word embeddings of the model. From the code, it seems that they are not tied.

[lvcc2018]

And I am curious about the difference between tied or untied word embeddings. Does it have an effect on training stability?

[benob]

Long ago at the time of LSTMs, I used to train a lot of LMs and tying embedding weights was a game changer: much faster convergence and less memory usage. I don't know about nowadays though.

To answer your first question, I looked at the weights of the tok_embeddings and output tensors in the checkpoint files and they differ. So tying was not used for training.

[glample]

if by "lm head" you are referring to the output layer on top of the transformer (the Linear(hidden_dim, vocab_size)), then no, they are not shared with the input word embeddings.

[wuwuwuxxx]

I want to know if there are some particular reasons to not use weight tying? @glample