The Encoder implementation is different from the original "Attention is all need" paper?

[chaoyanghe]

Hi, I checked your code at

ViT-pytorch/models/modeling.py

Line 154 in 878ebc5

class Block(nn.Module):

.

Your implementation is: Attention(LayerNorm(x)) + x, but the original Transformer is: LayerNorm(x +Attention(x)). Is this an error or deliberately implemented like this?

[jeonsworld]

As in paper On Layer Normalization in the Transformer Architecture, the position of Layer Normalization in Transformer implementation is used as pre-LN and post-LN. For example, Transformer Encoder-based BERT uses post-LN, but Vision Transformer uses pre-LN.
In conclusion, that implementation is correct.

[jeonsworld]

Additional comments.
Attention is all you need uses post-LN.

[jeonsworld]

I believe the issue has been answered and close the issue.

[chaoyanghe]

Thank you!