Skip to content

[Metal] NVFP4 tensor-scale (global_scale) not implemented — throws runtime error on Metal backend #3911

Description

@dahai80

Summary

PR #3022 added per-tensor scale (global_scale) support for NVFP4 quantization on CUDA and CPU backends. However, the Metal backend explicitly rejects it:

https://github.com/ml-explore/mlx/blob/main/mlx/backend/metal/quantized.cpp#L1725-L1730

if (mode_ == QuantizationMode::Nvfp4 &&
    static_cast<int>(inputs.size()) > base_size) {
  throw std::runtime_error(
      "[QQMatmul] Global scale (tensor-scale nvfp4) is not supported "
      "on the Metal backend.");
}

Impact

Without tensor-scale support, NVFP4 on Metal has ~137x less dynamic range than NVIDIA Blackwell (unsigned UE4M3 vs signed E4M3 scales), which was the original concern in #2962. The fix (PR #3022) landed for CUDA/CPU but Metal users still hit the error.

This blocks NVFP4 quantization for Apple Silicon users running MoE models (DeepSeek-V3/V4, GLM-5.1, etc.) where tensor-scale is critical for expert routing accuracy.

Request

Please add Metal kernel support for consuming the global_scale inputs in the qqmm / fp_quantized Metal kernel paths, matching the CUDA/CPU behavior from PR #3022.

Environment

  • MLX commit: 973e27f
  • Platform: Apple Silicon (M-series)
  • Backend: Metal

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions