LayerNorm(x + Sublayer(x))

[bkj]

In the text (between cells 7 and 8), you say the output of each sub-layer is:

layer_norm(x + dropout(sublayer(x)))

but in the code it looks like it's implemented as

x + dropout(sublayer(layer_norm(x))

Am I understanding correctly? (I'm guessing this doesn't matter in practice, but just checking)

Thanks

[srush]

So their notation is a little confusing, from their code it seems like the x you add in does not have the previous norm applied.

At the end we return x_3 = layer_norm(x_2) where

x_2 = x_1 + dropout(sublayer(layer_norm(x_1)) where
x_1 = x_0 + dropout(sublayer(layer_norm(x_0))

So you can think of the layer_norm being applied last, but not to the residual you add.

[sth4k]

Hi I have the same question here. So is there a reason why the annotated transformer code uses different layernorm+residual as the one in the original paper?

[imleibao]

is that won't be a problem? since you also normalized the input embedding by coding like that.