Replace BertLayerNorm with LayerNorm

[eldarkurtic]

Running Movement pruning experiments with the newest HuggingFace would crash due to non-existing BertLayerNorm.

[VictorSanh]

lgtm, thank you @eldarkurtic !

[yeeeqichen]

@VictorSanh @eldarkurtic

Hello guys!

Thanks for your effort for such a great repo!

I'm wondering why the implementation of LayerNorm in BERT seems not to be identical to what's commonly discussed version of LayerNorm.
I've read several tutorials about LayerNorm, and according to their explanation, the implementation seems to be like following:

input_x = torch.rand(batch_size, sequence_length, hidden_size)
layer_norm = torch.nn.LayerNorm([sequence_length, hidden_size])
output = layer_norm(input_x)

In above implementation, the normalization is calculated through the dim of sequence_length as well as hidden_size, but the implementation in this repo only normalize the tensors in the last dim, which I think is somehow identical to InstanceNorm?