Requesting converter + runtime support for the Nemotron-H architecture
(model_type: nemotron_h), e.g. nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16.
Architecture (config.json):
- 52 layers: 23× Mamba-2 SSM + 23× MoE + 6× full attention (layers_block_type)
- MoE: 128 routed experts (n_routed_experts), 6 per token, grouped top-k
(n_groups=8, topk_group=1), routed_scaling_factor=2.5, norm_topk_prob,
1 shared expert (3712 wide), mlp_hidden_act=relu2 (experts have no gate_proj)
- Mamba-2: ssm_state_size=128, conv_kernel=4, chunk_size=128
- MTP present (mtp_layers_block_type=[attention, moe], 2 layers)
- 32 attn heads / 2 KV, head_dim 128, full RoPE
Why it matters: beyond MTP, NVIDIA ships official D-Flash and D-Spark
speculative-decoding variants for this exact model
(nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash / -DSpark,
standalone qwen3-arch draft models). D-Flash block-parallel drafting
reports 2-3x speedups over Eagle-3 and up to 6.08x peak single-stream
(MATH-500) / 15x production throughput (Blackwell); D-Spark adds a
sequential Markov head + confidence head and reports 60-85% production
throughput gains (live DeepSeek-V4). baseRT currently has no
speculative-decoding path (MTP is dropped at convert; decode is
bandwidth-bound), so Nemotron-H support is the prerequisite for this
optimization class on Apple Silicon.
For agent/harness workloads (input-dominated, prefill-heavy), MoE is the
productive shape: measured on M5 Pro with Q4, a 35B-A3B MoE prefills at
3028 t/s vs 417 t/s for a 27B dense (~7x), with minimal intelligence
trade-off. For this usage class, prefill throughput and concurrency
matter more than decode — and MoE wins decisively on both.
Current state: base-convert rejects it at the mapper stage
(HF model_type "nemotron_h" not supported yet); the converter's
to_canonical_name prefix chain (model.language_model./model.) does not
cover the backbone.layers.N.mixer.* naming, MoE experts are up/down-only,
the SSM tensors (mixer.A_log/D/dt_bias) fall outside the existing
GDN/ssm f32-protection predicates and would be forced to f16 by the
1-D norm-like rule, and no runtime model class can execute Mamba-2 blocks.
Full evidence pack (tensor inventory with shapes, converter gap list with
file:line refs, runtime requirements): prepared in my fork at
https://github.com/chaoshou-coder/baseRT/blob/docs/nemotron-h-evidence/docs/nemotron-h-support.md
llama.cpp already ships a reference implementation
(LLM_ARCH_NEMOTRON_H, GGUF layer_types 0=SSM/1=ATTN/2=FFN).
Happy to contribute the converter-side mapper once runtime support is planned.
Requesting converter + runtime support for the Nemotron-H architecture
(
model_type: nemotron_h), e.g. nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16.Architecture (config.json):
(n_groups=8, topk_group=1), routed_scaling_factor=2.5, norm_topk_prob,
1 shared expert (3712 wide), mlp_hidden_act=relu2 (experts have no gate_proj)
Why it matters: beyond MTP, NVIDIA ships official D-Flash and D-Spark
speculative-decoding variants for this exact model
(nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash / -DSpark,
standalone qwen3-arch draft models). D-Flash block-parallel drafting
reports 2-3x speedups over Eagle-3 and up to 6.08x peak single-stream
(MATH-500) / 15x production throughput (Blackwell); D-Spark adds a
sequential Markov head + confidence head and reports 60-85% production
throughput gains (live DeepSeek-V4). baseRT currently has no
speculative-decoding path (MTP is dropped at convert; decode is
bandwidth-bound), so Nemotron-H support is the prerequisite for this
optimization class on Apple Silicon.
For agent/harness workloads (input-dominated, prefill-heavy), MoE is the
productive shape: measured on M5 Pro with Q4, a 35B-A3B MoE prefills at
3028 t/s vs 417 t/s for a 27B dense (~7x), with minimal intelligence
trade-off. For this usage class, prefill throughput and concurrency
matter more than decode — and MoE wins decisively on both.
Current state: base-convert rejects it at the mapper stage
(
HF model_type "nemotron_h" not supported yet); the converter'sto_canonical_name prefix chain (model.language_model./model.) does not
cover the
backbone.layers.N.mixer.*naming, MoE experts are up/down-only,the SSM tensors (mixer.A_log/D/dt_bias) fall outside the existing
GDN/ssm f32-protection predicates and would be forced to f16 by the
1-D norm-like rule, and no runtime model class can execute Mamba-2 blocks.
Full evidence pack (tensor inventory with shapes, converter gap list with
file:line refs, runtime requirements): prepared in my fork at
https://github.com/chaoshou-coder/baseRT/blob/docs/nemotron-h-evidence/docs/nemotron-h-support.md
llama.cpp already ships a reference implementation
(LLM_ARCH_NEMOTRON_H, GGUF layer_types 0=SSM/1=ATTN/2=FFN).
Happy to contribute the converter-side mapper once runtime support is planned.