Why are there so many variables named layrnorm in the codebase?

[jere357]

Running

grep -R -n --color=auto "layrnorm" .

gives these results when ran in src/tranformers

./models/idefics/vision.py:441:        self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
./models/idefics/vision.py:468:        hidden_states = self.pre_layrnorm(hidden_states)
./models/idefics/vision_tf.py:506:        self.pre_layrnorm = tf.keras.layers.LayerNormalization(epsilon=config.layer_norm_eps, name="pre_layrnorm")
./models/idefics/vision_tf.py:534:        hidden_states = self.pre_layrnorm(hidden_states)
./models/idefics/vision_tf.py:564:        if getattr(self, "pre_layrnorm", None) is not None:
./models/idefics/vision_tf.py:565:            with tf.name_scope(self.pre_layrnorm.name):
./models/idefics/vision_tf.py:566:                self.pre_layrnorm.build([None, None, self.embed_dim])
./models/altclip/modeling_altclip.py:1140:        self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
./models/altclip/modeling_altclip.py:1168:        hidden_states = self.pre_layrnorm(hidden_states)
./models/git/convert_git_to_pytorch.py:88:    rename_keys.append((f"{prefix}image_encoder.ln_pre.weight", "git.image_encoder.vision_model.pre_layrnorm.weight"))
./models/git/convert_git_to_pytorch.py:89:    rename_keys.append((f"{prefix}image_encoder.ln_pre.bias", "git.image_encoder.vision_model.pre_layrnorm.bias"))
./models/git/modeling_git.py:997:        self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
./models/git/modeling_git.py:1025:        hidden_states = self.pre_layrnorm(hidden_states)
./models/clipseg/modeling_clipseg.py:849:        self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
./models/clipseg/modeling_clipseg.py:874:        hidden_states = self.pre_layrnorm(hidden_states)
./models/clipseg/convert_clipseg_original_pytorch_to_hf.py:87:        name = name.replace("visual.ln_pre", "vision_model.pre_layrnorm")
./models/chinese_clip/convert_chinese_clip_original_pytorch_to_hf.py:84:    copy_linear(hf_model.vision_model.pre_layrnorm, pt_weights, "visual.ln_pre")
./models/chinese_clip/modeling_chinese_clip.py:1097:        self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
./models/chinese_clip/modeling_chinese_clip.py:1124:        hidden_states = self.pre_layrnorm(hidden_states)
./models/clip/modeling_tf_clip.py:719:        self.pre_layernorm = keras.layers.LayerNormalization(epsilon=config.layer_norm_eps, name="pre_layrnorm")
./models/clip/modeling_clip.py:1073:        self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
./models/clip/modeling_clip.py:1101:        hidden_states = self.pre_layrnorm(hidden_states)
./models/clip/convert_clip_original_pytorch_to_hf.py:96:    copy_linear(hf_model.vision_model.pre_layrnorm, pt_model.visual.ln_pre)
./models/clip/modeling_flax_clip.py:584:        self.pre_layrnorm = nn.LayerNorm(epsilon=self.config.layer_norm_eps, dtype=self.dtype)
./models/clip/modeling_flax_clip.py:603:        hidden_states = self.pre_layrnorm(hidden_states)
./models/kosmos2/modeling_kosmos2.py:748:        self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
./models/kosmos2/modeling_kosmos2.py:770:        hidden_states = self.pre_layrnorm(hidden_states)
./models/kosmos2/modeling_kosmos2.py:1440:            module.pre_layrnorm.bias.data.zero_()
./models/kosmos2/modeling_kosmos2.py:1441:            module.pre_layrnorm.weight.data.fill_(1.0)
./models/kosmos2/convert_kosmos2_original_pytorch_checkpoint_to_pytorch.py:16:    "ln_pre": "pre_layrnorm",

Why are there so many layernorm variables named layrnorm? Is it a typo or is this intended?

[Rocketknight1]

Seems perfectly nrmal to me

Real answer: I have no idea why it's like this, but it's probably a typo in one original codebase that was copied by other models. Fixing it would unfortunately break saved checkpoints, so we'll probably just leave it!