spec(#810 A2-Q): split the quantized arms into A2-Q1 and A2-Q2, because their blockers differ (#517)

[localai-bot]

Splits A2-Q of .agents/specs/nemotron-h-abi-e2e.md into two units with their own specs. Issue #810, parent row #517. Records only: no product code, no lifecycle change.

Per the governing spec's §1.4, the implementation is a separate PR by a different agent and the reviewer is a third. This spec is written by the agent that scoped the work and will not implement either half — a fresh implementer is not fresh if it authored the spec it works from.

Why split, and why it is not about size

A2-Q was briefed as one unit covering the mamba, MoE and lm_head device arms. Scoping measured three independent quantization mechanisms with three independent failure modes — but the decisive reason is that the two halves have different blockers:

Unit	Spec	Blocker
A2-Q1 — FP8 W8A8 mamba	nemotron-h-a2q1-fp8-mamba.md	#960, declared as a base. Cannot run on Thor at all until it lands
A2-Q2 — NVFP4 MoE + lm_head	nemotron-h-a2q2-nvfp4-moe-lmhead.md	none for the build; the Thor leg is PENDING #962

Bundling them makes the NVFP4 half wait on a build-file fix it does not need.

The Thor finding, measured

Recorded in full at A2-Q1 §3 because it inverted twice during scoping. Thor has half an fp8 arm, denied by a build accident rather than by silicon.

The GEMM half works. A standalone probe mirroring BuildFp8Plan's exact key (src/vt/cuda/cuda_matmul.cu:533) — A = e4m3 op T, B = e4m3 op N, CUBLAS_COMPUTE_32F, scale CUDA_R_32F, epilogue DEFAULT — on real sm_110 silicon (NVIDIA Thor, driver 13020, cublasLt 130002):

-- D=bf16 --
  m=1     n=10304  k=2688   status=0 returned=1  -> FP8 PLAN AVAILABLE
  m=8     n=10304  k=2688   status=0 returned=1  -> FP8 PLAN AVAILABLE
  m=512   n=10304  k=2688   status=0 returned=1  -> FP8 PLAN AVAILABLE
  m=1     n=2688   k=4096   status=0 returned=1  -> FP8 PLAN AVAILABLE
  m=512   n=2688   k=4096   status=0 returned=1  -> FP8 PLAN AVAILABLE
  m=64    n=4096   k=4096   status=0 returned=1  -> FP8 PLAN AVAILABLE
-- D=f32 --   (same six shapes)                  -> FP8 PLAN AVAILABLE x6
RESULT: 0 of 12 probes had NO fp8 plan

The six shapes are NemotronH's own in_proj and out_proj at decode, small-batch and prefill widths, plus a square control.

The activation-quant half is missing:

kQuantFp8Static registered ONLY at    cuda_matmul_fp8_cutlass.cu:376
that TU compiled only if              VT_CUTLASS_FP8_ARCHS   (CMakeLists.txt:1668-1669)
CUTLASS refs in the kernel body:      0                      (:353-370)
kMatmulFp8CublasLt registered         cuda_matmul.cu:920 -- unconditional

MatmulFp8CutlassD's guard (dense_fp8_gemm.h:130) keys on kMatmulFp8CublasLt, which IS registered, so the guard passes on Thor and the code crashes one call later on the portable reference tier — which is what #960 measured as a SIGSEGV. The relocation unblocks every non-CUTLASS CUDA arch, so it is dispatched as its own unit; A2-Q1 declares it a base rather than carrying it.

The dependency is written against the issue, not a branch name. git ls-remote --heads origin on 2026-08-16 showed no ref for the dispatched row — it is working locally, pre-push. A base has to be something git can reach.

Two corrections to the brief, recorded rather than quietly applied

The MoE arm does not take a per-expert [K,N] pointer array. That is kMoeGroupedGemmBf16 (include/vt/ops.h:1642). MoeGroupedGemmNvfp4Marlin (:1685) takes a rank-3 strided arena [E, K/16, N*8/pack], validated at src/vt/ops.cpp:884, with processed scales and global scales. That needs a load-time repack, which the pointer array does not — measured at ~16.5 GB device-resident across the 23 MoE layers, with the per-slab arithmetic in A2-Q2 §3.

PrepareNemotronHForCausalLM is a pure no-op (nemotron_h_registry.cpp:113-118, three (void) casts) and must be implemented to host that repack pre-capture, as PrepareQwen3_5Dense (qwen3_5_dense.cpp:104-115) does. That makes it a shared-seam change, not a model-local detail.

The lm_head residency is now a decision, not a default

There are two functions named MarlinDenseResidentFor, different safety, selected by translation unit:

dense_nvfp4_gemm.h:379   static std::unordered_map<const Nvfp4Weight*, MarlinDenseResident> cache;   ADDRESS-KEYED
qwen3_5.cpp:2429         return ResidentIn<MarlinDenseResident>(w->resident_marlin);                 SLOT-KEYED, safe

Each is used by the MatmulNvfp4W4A16D in its own TU (:730 and :2521). The shared seam holds the unsafe one, which inverts the incentive the seam rule creates: following the seam gets you the defect, hand-rolling gets you the fix, and the safe behaviour is not discoverable from the header. Filed as #984. A2-Q2 is exactly the second-consumer condition an address key cannot survive — lm_head plus 5935 expert projections through one header — so A2-Q2 §4.3 requires the choice to be argued in the commit rather than defaulted.

G-SAFE and reachability

Both specs keep all three clauses at src/vllm/model_executor/models/nemotron_h_registry.cpp:162 — attn_kv.empty() && gdn_state.empty() && num_reqs <= 1 — fully intact. Neither unit creates paging, carried state or batching. A2-P is where it narrows.

Both inherit A2-R's unreached posture: NemotronHDeviceForward is called only from test_nemotron_h_forward.cpp:1805, and A2-P owns wiring it through ModelRegistry::Forward. Each spec states this under ## Owed rather than leaving it to be discovered. A2-Q2 adds the nuance that its Prepare hook is reached from production load even while the forward that consumes it is not.

Gates

scripts/agent-preflight.sh green on this branch. Baseline re-measured at the pinned base 0e1bee42f in a clean detached worktree: all 26 record gates green including the seven #873 ones, Release build 0 warnings 1441/1441, ctest -R "nemotron_h|ops_mamba2|ops_fp8|linear_method" 13/13.

Not touched

.agents/issue-index.md is unchanged deliberately. #810 already has a row, and it is linked from both new specs and from this body, so the three places agree. #960, #962 and #984 were filed or dispatched by others; appending rows for them here risks the clean-but-wrong duplicate that append-only union merges produce when two branches record the same intent. They are instead listed under each spec's ## Owed, which is what check-agent-record.py reads.

Closes nothing. Unblocks the dispatch of both implementations to fresh agents.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]