Skip to content

NVFP4EffMA 0.1.0

Choose a tag to compare

@srose69 srose69 released this 20 Oct 05:31
· 14 commits to main since this release

NVFP4EffMA 0.1.0

NVFP4 GEMM, GEMV and quantization for NVIDIA GPUs without native FP4 (sm_50 … sm_90).

Install

PyTorch: pick the wheel matching your Python (cpXY), PyTorch (torchA.B) and its CUDA (cuNNN):

pip install nvfp4effma-0.1.0+<cuNNNtorchA.B>-<cpXY>-<cpXY>-linux_x86_64.whl

Other combinations: pip install ./binds/python --no-build-isolation from the source archive.

C / C++: unpack nvfp4effma-0.1.0-linux-x86_64-cuda12.tar.gz and link lib/libnvfp4effma.a (see its README).

Every artifact carries SASS for sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

  • nvfp4effma-0.1.0-linux-x86_64-cuda12.tar.gz (3.9M)
  • nvfp4effma-0.1.0-src.tar.gz (88K)
  • nvfp4effma-0.1.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (6.7M)

What's in it

First release. The NVFP4 sibling of E4M3EffMA and
E5M2EffMA, with the same build, tuning, test and benchmark setup.

  • GEMM D = α·A·Bᵀ + bias + β·C: A NVFP4 (E2M1 values, E4M3 scale per 16 values, FP32 global scale), B ∈ {NVFP4, BF16, FP16, FP32}, D ∈ {BF16, FP16, FP32}, accumulation into D (e.g. an FP32 main_grad), amax of the result.
  • NVFP4 × NVFP4 (W4A4) on DP4A from sm_61: twice an E2M1 value is an exact int8, so block sums of 16 are exact integers and each block is rounded once. Below sm_61 (Maxwell, Tesla P100) an FFMA kernel produces the same bits.
  • NVFP4 × BF16 / FP16 / FP32 (W4A16): the block scale is applied when the weights are decoded; exact products, one FFMA per MAC.
  • GEMV path for up to 8 columns (decoding), picked automatically; DP4A per block for NVFP4 activations.
  • Quantization / dequantization from and to BF16, FP16 and FP32, with the rounding of ModelOpt / Transformer Engine and the checkpoint layout (weight, weight_scale, weight_scale_2).
  • PyTorch: quantize / dequantize / linear, NVFP4Tensor, and NVFP4Linear with quantize_linear_ to swap the nn.Linear layers of a model. Imports alongside e4m3effma / e5m2effma.

On a GTX 1080: NVFP4 × NVFP4 GEMM 2.01× cuBLAS SGEMM (9.72 ms at 4096³, 7.1 TMAC/s; 1.98× when accumulating into FP32), NVFP4 × BF16 at SGEMM speed with 7× fewer weight bytes, GEMV n = 1 at 84 % of DRAM bandwidth (7.4× cuBLAS SGEMV), quantization at 89 %. Details: docs/performance.md, benches/.

Tested and tuned by the author on a GTX 1080 (sm_61) only, plus the code path of GPUs without DP4A (SASS built for sm_60, run on the GTX 1080). Other architectures are compiled, not run. No stack or local memory in any kernel, except on Jetson TX1 / TX2 (sm_53 / sm_62, 32 K registers per block), where the heaviest kernels spill.

The wheel here is built for Python 3.12, PyTorch 2.7 and CUDA 12.6.