[RFC]: NVFP4 Quantization Support for Diffusion Models

[lishunyang12]

Motivation

NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell GPUs (SM100+). It uses two-level scaling (FP8 micro-block scale per 16 values + FP32 tensor-level scale), achieving ~1.8x memory reduction vs FP8 and ~3.5x vs FP16 with near-FP8 accuracy. NVIDIA demonstrated up to 10.2x speedup on FLUX.2 with NVFP4 on Blackwell.

This issue tracks pre-RFC research for adding NVFP4 support to vllm-omni diffusion models.

Research Links

NVIDIA Technical Resources

• Introducing NVFP4 for Efficient and Accurate Low-Precision Inference — core algorithm, two-level scaling, accuracy analysis
• Scaling NVFP4 Inference for FLUX.2 on Blackwell Data Center GPUs — FLUX.2 FP4 optimization, 6.3x speedup, multi-GPU scaling
• TensorRT Unlocks FP4 Image Generation for Blackwell GeForce RTX 50 Series — consumer GPU FP4, FLUX.1 quantization
• Optimizing FLUX.1 Kontext for Image Editing with Low-Precision Quantization — FP4 for image editing models
• NVFP4 KV Cache Optimization for Long Context — FP4 KV cache on Blackwell
• Quantization-Aware Distillation for NVFP4 Accuracy Recovery (paper) — QAD technique for recovering FP4 accuracy loss

vLLM Upstream (LLM side, already merged)

• PR #12784 — Support NVFP4 quantization — original FP4 quant support by @kaixih
• PR #13571 — Support NVFP4 CUTLASS GEMM — FP4 GEMM kernel using CUTLASS 3.8
• PR #21465 — Fix CompressedTensor NVFP4 illegal memory access — critical bug fix for MoE FP4
• Issue #31085 — SM120 support for NVFP4 MoE kernels — RTX 6000 Pro Blackwell support
• Issue #27231 — Expert Parallelism with FP4 — EP + FP4 for large MoE models
• vLLM NVFP4 KV Cache discussion — FP4 in attention KV cache

Tooling & Quantization

• NVIDIA Model-Optimizer (ModelOpt) — unified PTQ/QAT library, produces NVFP4 checkpoints
• ModelOpt diffusion quantization examples — quantize.py --format fp4 for FLUX, SDXL, LTX-2
• ModelOpt diffusers integration (HuggingFace docs) — diffusers + ModelOpt pipeline
• llm-compressor NVFP4 example (vLLM docs) — W4A4 FP4 quantization workflow
• ModelOpt PR #782 — FLUX NVFP4 quantization fix — community fix for FLUX FP4

Pre-Quantized Models

• nvidia/DeepSeek-R1-NVFP4 — reference NVFP4 checkpoint format

Hardware Requirements

• Requires NVIDIA Blackwell GPU (SM100+ for quantization, SM100/SM120 for inference)
• B200/B100 (data center)

Open Questions

1. Weight-only vs W4A4? — vLLM upstream supports both NVFP4 weight-only (W4A16) and W4A4. Which is more practical for DiT models where activations have high dynamic range?
2. Calibration approach — ModelOpt provides PTQ + optional QAD (Quantization-Aware Distillation). Do we need calibration data for diffusion models?
3. Integration path — Should this go through the unified quantization framework ([Core] Unified quantization framework #1764) or as a standalone method?
4. Kernel availability — vLLM's FP4 kernels use CUTLASS 3.8. Does vllm-omni's build include this?
5. Quality metrics — NVIDIA showed FP4 needs QAD for FLUX quality recovery. Do we need model-specific fine-tuning per diffusion model?

Related

• [RFC]: Continuous Quantization Support  #1854 (Quantization Roadmap)
• [Core] Unified quantization framework #1764 (Unified Quantization Framework)
• [RFC]: Low-Precision Quantization (MXFP4) Support for Multimodal Models in vLLM-Omni #1733 (MXFP4 RFC — related but different format)

CC

@david6666666 @Isotr0py