Question About LayerNorm Implementation

[Guaishou74851]

Hi, your work is really inspiring and interesting!
I am wondering why you re-implement LayerNorm (2D), instead of using PyTorch modules like nn.GroupNorm(1, channels)?
Are there some differences between them (e.g. final performance or function differences)?

[mayorx]

Hi, Guaishou74851,
Thanks for your attention to NAFNet.
You can regard the code as a re-implementation of https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py#L138-L143, which behaves differently from nn.GroupNorm(1, channels). We re-implement it to save GPU memory cost.

[Luciennnnnnn]

@mayorx Hi, I observe a different optimization behavior between your implementation and connext's, how it occurs?