Repository navigation
Implemented rotary positional embedding and swiglu activation #101
aezazi
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Building on the tutorial code, I implemented rotary positional embedding and swiglu activation. Attaching Hellaswag accuracy of the base case and the rotary implementation. Next goal: implement mixture of experts.

All reactions