Qwen3.8-27B (VLM): Lightning MTP 0% acceptance on bf16 checkpoint, 71.5% on identical oQ8e — suspected non-quantized verify path

[PT68]

Environment

• oMLX 0.6.0.dev1 (venv install from git tag, custom kernels built, macOS 26.x, MacBook M5 Max 128 GB)
• mlx-vlm 0.6.3 (vendored)
• Model: Qwen/Qwen3.8-27B (released 2026-08-14; model_type: qwen3_5, Qwen3_5ForConditionalGeneration, 15 mtp.* tensors in HF checkpoint, text_config.mtp_num_hidden_layers = 1, mtp_use_dedicated_embeddings = false)

Summary
Lightning MTP produces 0/N acceptance (0.0%) on a bf16 MLX conversion of Qwen3.8-27B, while an oQ8e conversion of the same source checkpoint reaches 71.5% acceptance (tok/cycle 2.55, 35.2 tok/s) on the same machine, same prompt, same oMLX build. MTP-head tensors of both artifacts were verified identical in structure, naming, and norm calibration (means match to 3 decimal places). The only remaining difference between the artifacts is quantization, which points at the non-quantized (bf16) MTP verify path — possibly related to qwen35_verify_qmm being an affine 4/8-bit path (MTP verify qmm patch applied (M=2..6 affine 4/8-bit)), with the bf16 fallback branch broken or untested for this architecture.

Reproduction

1. bf16 artifact: mlx_vlm.convert --dtype bfloat16 from HF checkpoint (sanitize drops mtp.*), then re-attach the 15 original mtp.* tensors under language_model.mtp.* (bf16 preserved, mx.eval + read-back verification), then apply the MLX RMSNorm gamma convention (+1) to all 7 head norms — final means match the oQ8e artifact exactly (see table).
2. oQ8e artifact: built via oMLX oQ Quantizer from the same source, "Preserve MTP weights" ON, non-quant dtype bfloat16.
3. Load either model (VLMBatchedEngine, no warnings, Speculative backend: Lightning MTP (model_type=qwen3_5, active)), send a short chat request.

Results (same prompt, same build)

artifact | accept | tok/cycle | decode -- | -- | -- | -- bf16 + reattached MTP | 0/38 (0.0%), depth[d1=0/38,d2=0/0,d3=0/0] | 1.05 | 8.9 tok/s oQ8e-mtp | 929/1300 (71.5%), depth[d1=439/590,d2=299/426,d3=191/284] | 2.55 | 35.2 tok/s

MTP head norm means (identical on both artifacts, after +1 convention)
input_layernorm +1.036 · post_attention +1.206 · k_norm +1.779 · q_norm +1.791 · mtp.norm +2.252 · pre_fc_norm_embedding +0.539 · pre_fc_norm_hidden +0.843
(raw HF means for reference: +0.036 / +0.206 / +0.780 / +0.791 / +1.252 / −0.461 / −0.157 — i.e. Qwen3.8 ships zero-centered gammas on all seven head norms, including input_layernorm and pre_fc_norm_*, which norm_repair.py currently assumes are always converted correctly; with a raw re-attached head, repair_legacy_head_norms fixes only a subset — worth noting for the heuristic, though it is not the cause of this issue: the 0% result reproduces with fully corrected norms and repair active or disabled.)

Also observed (minor)

• mlx_vlm.convert drops mtp.* tensors for qwen3_5 VLM (expected upstream behavior, mentioned for context).
• oQ "Preserve MTP weights" works as advertised for this brand-new checkpoint — nice.

Ask
Could the bf16 (non-qmm) MTP verify branch be checked for qwen3_5 with mtp_use_dedicated_embeddings=false (Qwen3.8)? Happy to run diagnostics/patches on the bf16 artifact — we keep it as a reference build and can reproduce on demand.

[CharlesJS]

How are you testing that? I've also got some oQ quants that I've been curious about.

[PT68]

eQ qants are OK, problem is with MTP at bf16 MLX converted model.