DeepNorm question

[ezeli]

annotated_deep_learning_paper_implementations/labml_nn/normalization/deep_norm/__init__.py

Line 105 in 25ad4d6

self.layer_norm = LayerNorm(normalized_shape, eps=eps, elementwise_affine=elementwise_affine)

The self.layer_norm is not used in the forward function. And the self.alpha should be multiplied by x, not gx.

The correct forward function might be:

def forward(self, x: torch.Tensor, gx: torch.Tensor):
    return self.layer_norm(self.alpha * x + gx)

[vpj]

Thanks 🙏

Fixed it 0eb74d7

[everks]

thanks for your great work, but I think you are not fixed this? the alpha should be multiplied by x, not gx?