Repository navigation
NVFP4EffMA 0.1.0
NVFP4EffMA 0.1.0
NVFP4 GEMM, GEMV and quantization for NVIDIA GPUs without native FP4 (sm_50 … sm_90).
Install
PyTorch: pick the wheel matching your Python (cpXY), PyTorch (torchA.B) and its CUDA (cuNNN):
pip install nvfp4effma-0.1.0+<cuNNNtorchA.B>-<cpXY>-<cpXY>-linux_x86_64.whlOther combinations: pip install ./binds/python --no-build-isolation from the source archive.
C / C++: unpack nvfp4effma-0.1.0-linux-x86_64-cuda12.tar.gz and link lib/libnvfp4effma.a (see its README).
Every artifact carries SASS for sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).
Assets
nvfp4effma-0.1.0-linux-x86_64-cuda12.tar.gz(3.9M)nvfp4effma-0.1.0-src.tar.gz(88K)nvfp4effma-0.1.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl(6.7M)
What's in it
First release. The NVFP4 sibling of E4M3EffMA and
E5M2EffMA, with the same build, tuning, test and benchmark setup.
- GEMM
D = α·A·Bᵀ + bias + β·C: A NVFP4 (E2M1 values, E4M3 scale per 16 values, FP32 global scale), B ∈ {NVFP4, BF16, FP16, FP32}, D ∈ {BF16, FP16, FP32}, accumulation into D (e.g. an FP32main_grad),amaxof the result. - NVFP4 × NVFP4 (W4A4) on DP4A from sm_61: twice an E2M1 value is an exact int8, so block sums of 16 are exact integers and each block is rounded once. Below sm_61 (Maxwell, Tesla P100) an FFMA kernel produces the same bits.
- NVFP4 × BF16 / FP16 / FP32 (W4A16): the block scale is applied when the weights are decoded; exact products, one FFMA per MAC.
- GEMV path for up to 8 columns (decoding), picked automatically; DP4A per block for NVFP4 activations.
- Quantization / dequantization from and to BF16, FP16 and FP32, with the rounding of ModelOpt / Transformer Engine and the checkpoint layout (
weight,weight_scale,weight_scale_2). - PyTorch:
quantize/dequantize/linear,NVFP4Tensor, andNVFP4Linearwithquantize_linear_to swap thenn.Linearlayers of a model. Imports alongsidee4m3effma/e5m2effma.
On a GTX 1080: NVFP4 × NVFP4 GEMM 2.01× cuBLAS SGEMM (9.72 ms at 4096³, 7.1 TMAC/s; 1.98× when accumulating into FP32), NVFP4 × BF16 at SGEMM speed with 7× fewer weight bytes, GEMV n = 1 at 84 % of DRAM bandwidth (7.4× cuBLAS SGEMV), quantization at 89 %. Details: docs/performance.md, benches/.
Tested and tuned by the author on a GTX 1080 (sm_61) only, plus the code path of GPUs without DP4A (SASS built for sm_60, run on the GTX 1080). Other architectures are compiled, not run. No stack or local memory in any kernel, except on Jetson TX1 / TX2 (sm_53 / sm_62, 32 K registers per block), where the heaviest kernels spill.
The wheel here is built for Python 3.12, PyTorch 2.7 and CUDA 12.6.