[Model request] KAT-Coder-V2.5-Dev NVFP4 community exports (compressed-tensors MoE): dense-branch misroute + GDN granularity

[minatoAI]

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

Checkpoints

• sakamakismile/KAT-Coder-V2.5-Dev-NVFP4 — uniform NVFP4 (llm-compressor recipe
  targets Linear, bf16 lm_head, single model.safetensors)
• sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16 — NVFP4A16 (recipe ignore carries
  re:.*\.linear_attn\..*, so all GDN projections stay bf16)

Both are Qwen3.5-MoE (Qwen3_5MoeForConditionalGeneration, hidden 2048, 40
layers, 256 experts / top-8, MoE-512), quant_method: compressed-tensors,
format: nvfp4-pack-quantized.

Environment

• GPU: RTX 5070Ti 16GB (Blackwell) / driver 610 / Windows 11
• FreeToken Desktop 0.1.2+g141c31a8d (Windows bundle)
• Commands: ft checkpoint (via Desktop) then
  ft serve --model <id> --moe-backend auto

Problem 1: MoE+compressed-tensors falls into the dense loader

_expert_quant() only recognizes ModelOpt NVFP4 / MIXED_PRECISION, so
expert_quant falls back to none and iter_weights() takes the
dense (no routed experts) branch. Same bug class as #381.

Problem 2: one attn knob cannot cover both GDN layouts

After routing experts to the NVFP4 banks, serve fails per-variant:

• A16 (sahilchachra): KeyError: 'model.layers.0.linear_attn.out_proj.weight_scale'
  (nvfp4_linear.py load_state_dict pops a scale the bf16 GDN never had).
• uniform (sakamakismile): AssertionError at
  layers/base.py: assert param.shape == item.shape and param.dtype == item.dtype
  (bf16-built GDN out_proj vs the NVFP4 triplet in the FTW).

The engine needs full-attention NVFP4 + GDN bf16 for the first, uniform NVFP4
for the second.

Subtlety: the ignore list needs probe-level matching

saka's exported config has a 10,491-entry ignore including 60
linear_attn hits — but all are bare module paths
(...layers.N.linear_attn[.norm]), which match no projection, while its
in_proj/out_proj are genuinely packed (verified in the source headers).
A substring check on linear_attn misfires here; only evaluating entries with
their own semantics (re: as regex, otherwise fnmatch) against the probe
...linear_attn.out_proj distinguishes the two layouts.

Local validation (Desktop bundle, real GPU)

Implemented locally as: MoE-CT detection -> expert_quant=nvfp4; CT expert
bank spec (weight_packed->weight, weight_global_scale->weight_scale_2,
global_reciprocal=True); MoE dense pass (native W4A16 attn/shared, GDN
in_proj dequant, experts skipped); single-file index materialization; plus a
linear_attn_quant flag (probe-based) wired to the GDN constructor. Results:

• saka: 723 weight + 240 experts_bank, 20.62 GiB FTW, exit 0; serves and
  decodes normally (wired into a coding-agent harness, project-scan task).
• sahil: 663 weight + 240 experts_bank, 20.96 GiB FTW, exit 0; serves and
  decodes normally.
• No regression on the previously working nvidia/Qwen3.6-35B-A3B-NVFP4.

Relation to existing work

Same root-cause family as #296 (CT NVFP4 experts + block-fp8 dense side),
#390 (mixed FP8-attn + NVFP4 experts) and #381 — this covers the remaining
uniform-NVFP4 and NVFP4A16 dense-side variants. Happy to shape the local change
into a PR if the direction looks right.

How did you install FreeToken

pip / uv wheel

FreeToken version

FreeToken Desktop 0.1.2+g141c31a8d (Windows bundle)

OS

Windows 11

OS details

No response

GPU and driver

RTX 5070 Ti, driver 610.62

CPU and system RAM

AMD Ryzen 7 9700X 8-Core Processor , 32GB

Checkpoint

sakamakismile/KAT-Coder-V2.5-Dev-NVFP4 ; sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16

Command

Full log

Anything else

No response

[Romeriz]

Hi, I have a working, hardware-tested patch for this exact family and I don't want to
duplicate you or the open PRs. I got AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4,
doth4580/Kwaipilot-KAT-Coder-V2.5-Dev-NVFP4-MIXED and
Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 to convert and serve cleanly.

The layouts it handles that are distinct from PR #390 (mixed FP8-attn + NVFP4 experts):

• per-expert llm-compressor experts (...experts.E.{gate,up,down}_proj.weight_packed,
  one global per expert) -- AEON and Ttimms;
• stacked per-layer experts (...experts.gate_up_proj / .down_proj as one
  [E*rows, cols] tensor + a per-layer scalar global) with a dedicated bank loader --
  Kwaipilot;
• format-only configs (no config_groups) detected via
  format: nvfp4-pack-quantized;
• GDN granularity: ignore-listed GDN stays bf16 (AEON / Ttimms / your sahilchachra
  case) and a gdn:fp8 recipe GDN is dequantized to bf16 (Kwaipilot), while
  full-attention projections stay NVFP4 -- resolved with a probe, matching your point 2;
• NVFP4 embed_tokens dequantized (no NVFP4 embedding kernel) and lm_head kept
  native NVFP4 when the export quantized it (Kwaipilot);
• single-file no-index checkpoints (AEON is one 23 GB model.safetensors) -- the
  reciprocal global is read from the header, which PR fix(models): better support for mixed-precision compressed-tensors NVFP4  #390's index-only probe misses.

It's a one-branch change (fix(models): support llm-compressor NVFP4 MoE export variants) with weight-free unit tests.

Since you said you'd shape your local change into a PR and PR #390 is open in the same
files, I'd like to coordinate rather than race: are you still planning to open it? My
per-expert plumbing and #390's FP8-attn routing touch the same
iter_weights/_CT_NVFP4_FUSE/_expert_quant spots, so the cleanest is to reconcile
them (ideally a per-variant dense dispatch: .weight_packed NVFP4 dense MoE CT -> the
CT reader, FP8-config_groups MoE CT -> the fp8 reader). Happy to rebase mine on
whichever lands first and drop the overlap.

Verified locally (CPU, weight-free):

• AEON config -> expert_quant/attn_quant/dense_quant=nvfp4, gdn_quant=none, lm_head_quant=none; its dense read matches the model state dict exactly (663 tensors,
  0 missing/extra) and is unchanged from before this patch.
• Kwaipilot config -> expert_quant/attn_quant/dense_quant=nvfp4, gdn_quant=none, lm_head_quant=nvfp4; dense read matches the model state dict exactly (665 tensors,
  0 missing/extra).
• On the GPU box all three ft checkpoint + ft serve successfully (AEON FTW per my
  log: 663 weight + 240 experts_bank, 20.96 GiB). Happy to paste exact hardware and
  commands in the PR.

[minatoAI]

Thanks @Romeriz — I won't open a competing PR.

#413 matches both root causes in this issue: MoE compressed-tensors routing to
expert_quant=nvfp4, and separate GDN granularity with probe-level
ignore/recipe handling. #390 looks complementary rather than a replacement for
these two checkpoints.

My exact checkpoints remain:

• sakamakismile/KAT-Coder-V2.5-Dev-NVFP4
• sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16

My locally validated results were saka 723 weight + 240 experts_bank, 20.62 GiB
and sahil 663 weight + 240 experts_bank, 20.96 GiB, both convert + serve clean.
If useful, I can rerun the #413 branch against those two exact checkpoints and
post logs/counts for Windows/Desktop verification.