Repository navigation
v0.8.31 — Nemotron 3.5 quantization (non-gated MoE experts)
Quantization support for the Nemotron 3.5 family — NVIDIA-Nemotron-3.5-Super-VL-120B-A12B
and the text models (Lightning-30B-A3B, Super-120B-A12B, Nano-30B-A3B).
What this fixes
These models' routed experts are stacked 3-D parameters with no gate — the forward is
up -> relu**2 -> down — and every stacked-expert path in GLQ was written against a gated
gate_up_proj. Neither weight walk could see them: 210.0 of 231.8 GiB, 90.6% of the VL
checkpoint. 0.8.30 would have produced a "3 bpw" output heavier than its bf16 input, from a
run that reported success.
0.8.31 adds the non-gated path: a per-expert Hessian hook whose down-projection input is
act(up(x)) rather than act(gate) * up, the per-expert virtual-Linear expansion, a state
converter that merges the checkpoint's per-expert 2-D tensors into the module's two 3-D
parameters, and a non-gated branch in the resume replay.
It also narrows rather than removes the guard introduced in 0.8.30: a stacked expert
container whose architecture profile omits stacked_experts is still refused — gated or not —
because a bare 3-D parameter has no nn.Linear children and the stacked walk is opt-in. The
error now names the one-line fix.
Three profile bugs of one family
transformers renames backbone. to model. when loading this architecture, so the module
path and the checkpoint path differ — and the profile had collapsed them into a single value
three times over:
layers_attr/embed_attrsaidbackbone.*and resolved nothing on transformers
5.17+, which is an empty quantize rather than an error;forward_kwargs: 'nemotron_h'passedcache_params=Noneinto aforwardwith no such
parameter (it collides on every Mamba block), and forced eager attention, whose causality
here comes only from a mask the calibration forward never passes;- the embedding key was built from a module path, so a run died with
KeyError: 'model.embeddings.weight'. Measured on Lightning-30B: 6,242 keys under
backbone., zero undermodel..
trust_remote_code is now False for these architectures — they are native in transformers,
and the load path is safetensors streaming throughout.
Two decode fixes
_replace_nemotron_h_expertsread the top-level config, but a vision-language wrapper keeps
the MoE geometry onconfig.text_configonly. The substitution raisedAttributeErrorand
surfaced only as the loader's "the model will load DENSE in its original dtype".- The fused MoE dispatch mapped an unrecognised activation to
activation_type = 0, which is
gated silu — asking a gated kernel to evaluate a non-gated module. Not a crash: a
different function, computed with confidence. It now declines the fused path and takes the
Python loop, which applies the activation itself.
Also
- Peak-memory fix on the shared quantize path: the per-expert Hessian hand-off releases each
GPU Gram as soon as its host copy exists. On a 512-expert layer that removed 15.8 GiB of
device Hessians coexisting with 15.8 GiB of host copies. Output unchanged. - Model cards: the installer one-liner, a
glq-codeexample that actually serves tool
calling, a plugin-aware install branch, and a closing-fence fix that had been leaving a
code block open.
Existing checkpoints
Unaffected. No encode-path change reaches a gated-expert model: the new collectors key on
a 3-D up_proj with no gate_up_proj, so for gemma-4, Qwen4Exp, Kolibri-1 and sarvam the new
code is unreachable. The Hessian hand-off change alters peak memory, not output.
Validation
CI green on 3.12 / 3.13 / 3.14. A failing control was run for each new mechanism — each
reverted in turn with the intended test confirmed failing — which also caught three tests that
had been passing vacuously, including one that asserted through a code path which declines on
CPU for an unrelated reason.
distro-matrix was not run and does not apply: this release touches no install.sh, no
pre-flight, and nothing under glq/csrc/, so the kernel build is unchanged.
Not included
No published Nemotron checkpoint, and no measured quality or decode number for the new
architecture — so the README's comparison tables are unchanged. An end-to-end build is in
progress; quantization support landing is not the same as a validated checkpoint, and nothing
here should be read as a quality claim.