unsloth/Qwen3.8-27B-NVFP4 via oMLX?

[zaskara]

In oMLX 0.6.0 release it says:
"Expanded Qwen3.8 support. oMLX now loads blockwise FP8, embedded MTP, and the mixed ModelOpt NVFP4 unsloth/Qwen3.8-27B-NVFP4 checkpoint directly. NAX prompt throughput for common group-size-128 checkpoints improved from 513 to 925 tok/s in the reported workload."

Does it mean that I can run on my mac 'unsloth/Qwen3.8-27B-NVFP4'? On unsloth web-site it says: "NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark (see Qwen3.8), B200, B300 GPUs. For older GPUs, our GGUFs work well! You can run NVFP4 quants in vLLM only for now (SGLang is not supported)."
Please explain who knows. And if yes, I wonder — what will be better to use (faster/accurate) on m1 Pro 32 gb - this one or 'Qwen3.8-27B-oQ4e-fp16-mtp' from HF: scottlowry/Qwen3.8-27B-oQ4e-fp16-mtp?

[tt-a1i]

Yes, with oMLX 0.6.0 or newer. The warning on Unsloth's page describes the CUDA runtimes they support; oMLX added a separate MLX loader for this exact checkpoint. It keeps the packed NVFP4 and FP8 weights, puts them in MLX's native quantized carriers, and applies the checkpoint's original extra scales after each matmul (loader, quantized layer). The real-model test is explicitly restricted to Apple Silicon and exercises both text and vision generation with unsloth/Qwen3.8-27B-NVFP4 (test, generation checks). So this is not vLLM somehow running on the Mac; oMLX is loading the same checkpoint through MLX.

For an M1 Pro with 32 GB, I would start with the oQ4e-fp16-mtp model. There is at least a published M1 Pro 32 GB run for a Qwen3.8-27B oQ4e-fp16-mtp checkpoint: it peaked around 17.8–18.2 GB without MTP and 20.7–21.1 GB with MTP. Decode went from about 10 tok/s to 11.7–12.3 tok/s with MTP in that test (measurements). That is real evidence for this class of checkpoint on the machine you have.

The two repositories are also quite different in size: Hugging Face reports about 23.4 GB stored for the Unsloth NVFP4 checkpoint and about 17.9 GB for the oQ4e checkpoint. The smaller one leaves noticeably more of the 32 GB unified memory for macOS, runtime buffers, and context.

I would not claim which one is more accurate yet. The NVFP4 support PR validates strict loading plus text and vision generation, but does not publish a same-prompt accuracy comparison against oQ4e, and the 513 → 925 tok/s number in the 0.6.0 notes was measured on an M5 Max with a group-size-128 affine checkpoint, not on this NVFP4 model or an M1. For daily use on a 32 GB M1, oQ4e is the safer choice. NVFP4 is worth trying if you want to compare it and can keep the context modest, but the oMLX Accuracy Bench is the honest way to decide quality for your workload.

[zaskara]

Thank you for the detailed explanation. Yes, oQ4e-fp16-mtp gives me indeed about 11 tok/s. Not usable on my Mac. I will wait for 3.8 MoE.