Feature Request: Nemotron-3-Nano-30B-A3B model (moe on nemotron_h)

[mattepiu]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Support for Nvidia Nemotron-3-Nano-30B-A3B and architecture nemotron_h_moe which should be a moe extension of the existing nemotron_h.

Motivation

Benchmarks are very good for its size (i.e.: 38.8 in swe-bench) and speed is quite better than similar sized qwen3 models.

Possible Implementation

No response

[gabe-l-hart]

Looks like this is already addressed in #18058

[BVEsun]

b7415

llama-server.exe

unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q6_K_XL.gguf

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'nemotron_h_moe'

Looks like this is already addressed in #18058

just waiting merge

Sorry for missing the above comment

[aussetg]

2995341

:)

[engrtipusultan]

It is not faster than Qwen3 A3B on vulkan. But it is certainly faster on higher contexts due to Mamba2. Maybe there will be an impact after #18062

Qwen3-Instruct-30B-A3B

bash  /home/tipu/Applications/llamacpp/llama-bench -m /home/tipu/AI/models/ggml-org/Qwen3-Instruct-30B-A3B/qwen3-30b-a3b-instruct-2507-q8_0.gguf -ngl 99 --ubatch-size 512 --batch-size 2048 --mmap 0 -fa 0,1 --prio 3 -p 512 -n 256 -d 0,1024,8096,16546
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so

model	size	params	backend	ngl	fa	mmap	test	t/s
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	0	0	pp512	100.47 ± 1.50
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	0	0	tg256	12.66 ± 0.16
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	0	0	pp512 @ d1024	91.31 ± 1.30
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	0	0	tg256 @ d1024	12.00 ± 0.10
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	0	0	pp512 @ d8096	51.45 ± 0.29
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	0	0	tg256 @ d8096	5.78 ± 0.03
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	0	0	pp512 @ d16546	24.99 ± 0.12
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	0	0	tg256 @ d16546	3.56 ± 0.02
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	1	0	pp512	94.87 ± 0.90
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	1	0	tg256	12.81 ± 0.01
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	1	0	pp512 @ d1024	68.48 ± 0.25
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	1	0	tg256 @ d1024	12.07 ± 0.01
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	1	0	pp512 @ d8096	19.47 ± 0.06
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	1	0	tg256 @ d8096	8.85 ± 0.05
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	1	0	pp512 @ d16546	9.93 ± 0.04
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	1	0	tg256 @ d16546	6.62 ± 0.02

build: 5806286 (7444)

Nemotron-3-Nano

bash  /home/tipu/Applications/llamacpp/llama-bench -m /home/tipu/AI/models/unsloth/Nemotron-3-Nano/Nemotron-3-Nano-30B-A3B-Q8_0.gguf -ngl 99 --ubatch-size 512 --batch-size 2048 --mmap 0 -fa 0,1 --prio 3 -p 512 -n 256 -d 0,1024,8096,16546
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so

model	size	params	backend	ngl	fa	mmap	test	t/s
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	0	0	pp512	99.25 ± 0.35
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	0	0	tg256	11.80 ± 0.01
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	0	0	pp512 @ d1024	97.11 ± 0.48
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	0	0	tg256 @ d1024	11.59 ± 0.01
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	0	0	pp512 @ d8096	87.66 ± 0.34
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	0	0	tg256 @ d8096	9.01 ± 0.00
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	0	0	pp512 @ d16546	71.58 ± 0.22
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	0	0	tg256 @ d16546	7.18 ± 0.01
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	1	0	pp512	98.36 ± 0.72
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	1	0	tg256	11.97 ± 0.04
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	1	0	pp512 @ d1024	94.47 ± 0.14
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	1	0	tg256 @ d1024	11.93 ± 0.00
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	1	0	pp512 @ d8096	68.51 ± 0.71
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	1	0	tg256 @ d8096	11.21 ± 0.06
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	1	0	pp512 @ d16546	48.98 ± 0.27
nemotron_h_moe 31B.A3.5B Q8_0	31.27 GiB	31.58 B	Vulkan	99	1	0	tg256 @ d16546	10.57 ± 0.01

build: 5806286 (7444)

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.