Feature Request: Support for NVidia Nemotron Nano v2

[pwilkin]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Support the new https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2 model.

Motivation

This seems like the first real hybrid Mamba-transformer model out there that potentially performs at SOTA levels for its size. Might be worth pursuing.

Possible Implementation

No response

[0wwafa]

Yep! Same here:

python /content/llama.cpp/convert_hf_to_gguf.py --outtype f16 NVIDIA-Nemotron-Nano-9B-v2 --outfile NVIDIA-Nemotron-Nano-9B-v2.f16.gguf

INFO:hf-to-gguf:Loading model: NVIDIA-Nemotron-Nano-9B-v2
WARNING:hf-to-gguf:Failed to load model config from NVIDIA-Nemotron-Nano-9B-v2: The repository NVIDIA-Nemotron-Nano-9B-v2 contains custom code which must be executed to correctly load the model. You can inspect the repository content at /content/NVIDIA-Nemotron-Nano-9B-v2 .
 You can inspect the repository content at https://hf.co/NVIDIA-Nemotron-Nano-9B-v2.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:hf-to-gguf:Model architecture: NemotronHForCausalLM
ERROR:hf-to-gguf:Model NemotronHForCausalLM is not supported

[ggerganov]

Should be relatively easy to support since we can now implement hybrid architectures. cc @compilade @gabe-l-hart

[gabe-l-hart]

I did some initial investigation. The main difference between nemotronh and the granitehybrid (the one I know best) is in how the blocks are configured. In granitehybrid, each layer is either mamba2 or attention, and each layer has a FFN or MoE block afterwards. In nemotronh, the layer blocks can be either mamba2, attention, or mlp (here), and the mamba2/attention blocks to not have an FFN block after them. This difference means that my first-whack attempt to simply convert from nemotronh to granitehybrid in convert_hf_to_gguf.py is not going to work. Instead, we'll need a new model class (no big issue, just more code).

The biggest question now will be how to differentiate between three possible layer block types for a hybrid model. The current hybrid models rely on checking hparams.is_recurrent (eg here) which gets populated by checking hparams.n_head_kv (eg here).

[gabe-l-hart]

It looks like n_ff_arr might be what we need

[gabe-l-hart]

(broken) first pass at architecture support is on https://github.com/gabe-l-hart/llama.cpp/tree/gabe-l-hart/nvidia-nemotron-nano-15409

The conversion works and the model generates tokens, but produces garbage. Putting this down for now, but will get back to it soon.

[ggerganov]

(broken) first pass at architecture support is on https://github.com/gabe-l-hart/llama.cpp/tree/gabe-l-hart/nvidia-nemotron-nano-15409

The conversion works and the model generates tokens, but produces garbage. Putting this down for now, but will get back to it soon.

Is it correct to not have RoPE in the attention?

[gabe-l-hart]

The modeling_nemotron_h.py included with the model has no reference to RoPE (similar to modeling_granitemoehybrid.py). When compared to a model like modeling_granite.py that explicitly does use RoPE, I believe I'm reading Nemotron H correctly that it does not use RoPE.

[gabe-l-hart]

I've open a draft PR for implementation discussion

#15507

[ThiloteE]

For completeness sake, alongside the already mentioned model in the initial issue description, Nvidia also released the following two models along with it:

• https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base
• https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base

While having "Base" in their name, them scoring high on benchmarks, indicates they might be instruction tuned already, which differentiates them from traditional base models.

[suhara]

Hi @gabe-l-hart

Thank you for creating the PR!

I believe I'm reading Nemotron H correctly that it does not use RoPE.

Correct. It does not use RoPE.

[gabe-l-hart]

I tried to post a comment with a bunch of debug logging, but managed to exceed the maximum comment length, so I'll have to try again later.

Looking at tensor traces comparing transformers and my branch here, they start diverging at the conv1d operation. My current guess is that I'm missing an initialization token because the activation after the first conv1d is close, but distinctly different and the transformers output has 2 tokens in the batch while the llama.cpp batch has a single token that matches the earlier activations in the second token position from transformers.

[pwilkin]

I tried to post a comment with a bunch of debug logging, but managed to exceed the maximum comment length, so I'll have to try again later.

In my experience it's best to post those long debug snippets to gist and just link them here (I'll try to help however I can, I'm by no means an expert but maybe by comparing with the reference implementation I'll spot something).