Layer Normalization

[conceptofmind]

Hi all,

Do you use Layer Normalization with or without bias? From my understanding PaLM used bias less pre and post layer normalization.

Additionally, do you apply DeepNorm to the feedforward, attention, and fused attention feedforward projection layers? Or just the feedforward and attention layers?

Thank you,

Enrico

[Sengxian]

Hello @conceptofmind, thank you for your attention!

1. We use LayerNorm with bias.
2. Our implementation of DeepNorm generally follows its paper: xavier normal initialization with a $(2N)^{-1/2}$ scaling factor is applied to ffn, v_proj, out_proj.

[conceptofmind]

Hi @Sengxian,

I greatly appreciate the additional information. The performance of GLM is phenomenal and exactly what you would expect from top researchers at Tsinghua!

I had a few additional questions about the structure of the GLM architecture as new developments in Pathways-like models are quite exciting.

1. The PaLM paper used embedding weight tied to the projection out logits. Are the input and output embedding matrices shared in GLM as well?
2. Was bias used in any of the dense/linear kernels?
3. Are the key/value projections shared for each head as they are in PaLM? If so did you isolate the v_proj for DeepNorm or apply it to the shared k/v projection?

I have been working on a basic GLM "implementation" in PyTorch for my own understanding and education.

Thank you again for taking the time to answer my questions.

Enrico

[Sengxian]

1. Yes, input and output embedding matrices are shared in GLM-130B.
2. Yes, all dense layer have bias.
3. No, we use a standard multi-head self-attention instead of sharing key/value projections.

You can check out our model implementation for more details. Thank you again for your interest in GLM-130B!

[conceptofmind]

Hi @Sengxian ,

I appreciate the additional information. I will take a look.

Thank you for all of your help.

Enrico