About Transformer model details

[tanggang1997]

Hello, I read the source code of this project and I have two questions, the first one is: Are the a1 and a2 operations omitted in the Transformer block, I don't seem to see them, the second one is why three linear layers + activation function are used in the Pointwise Feedforward, and also after Transformer block is the Layer Norm and Linear and Reshape omitted.

[tanggang1997]

[Plachtaa]

This is SwiGLU activation from Llama3