Feature Request: Add OLMo November 2024 #10316
Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
Add support for the updated family of OLMo models will be released in November.
Motivation
It would be great to have llama.cpp support of the new OLMo models at release.
Possible Implementation
The new OLMo models has a few small architecture changes compared to the existing models:
- RMSNorm is used instead of standard layer norm.
- Norm is applied to attention queries and keys.
- Norm is applied after attention/feedforward rather than before.
The PR for adding it to transformers is approved: huggingface/transformers#34551