K2 Horizon: Pre-release llama.cpp support

[rayendito]

K2 Horizon is a series of LLMs by MBZUAI IFM. It has varying sizes spanning dense, sparse MoE architectures, including a novel MoVA architecture (Mixtures of Values). Every size is released as open weights.

• Dense 0.9B, 3.7B, 7B, 32B
• MoE + MoVA 36B
• Sparse MoE 375B (support for this model intentionally not implemented)

Links

Draft llama.cpp support implementation: https://github.com/MBZUAI-IFM/llama.cpp/tree/model/K2Horizon

Proposed Contributions

K2 Horizon introduces a new Mixture of Values architecture. Main architectural difference is in the attention layer. Conceptually it is similar to MoE; given a hidden state x, instead of the x going into the value matrix, it goes to a value router instead. This value router would then select expert value matrices and pass x through them and aggregate, giving the final value tensors.

(details in the upcoming technical report).

Known Issues

3.7B and 7B are using a diff. transformers version, which doesn’t match llama.cpp’s pinned version