|
Why do modern LLMs prefer SwiGLU in feed-forward blocks? |
Answered by
Ultron09
Aug 28, 2026
Replies: 1 comment
|
SwiGLU provides bilinear gating Swish(xW) * (xV) which improves gradient expressiveness per parameter. |
0 replies
Answer selected by
Ultron09
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
SwiGLU provides bilinear gating Swish(xW) * (xV) which improves gradient expressiveness per parameter.