Repository navigation
NVFP4EffMA 0.5.0
NVFP4EffMA 0.5.0
NVFP4 GEMM, GEMV and quantization for NVIDIA GPUs without native FP4 (sm_50 … sm_90).
Install
PyTorch: pick the wheel matching your Python (cpXY), PyTorch (torchA.B) and its CUDA (cuNNN):
pip install nvfp4effma-0.5.0+<cuNNNtorchA.B>-<cpXY>-<cpXY>-linux_x86_64.whlOther combinations: pip install ./binds/python --no-build-isolation from the source archive.
C / C++: unpack nvfp4effma-0.5.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libnvfp4effma.a (see its README).
Every artifact carries SASS for sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).
Assets
nvfp4effma-0.5.0-linux-x86_64-cuda12.tar.xz(8,7M)nvfp4effma-0.5.0-src.tar.gz(168K)nvfp4effma-0.5.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl(59M)
What's in it
Tensor cores for sm_75 … sm_89.
nvfp4a_gemm_tc/nvfp4a_gemv_tc: W4A4 and W4A8 on INT8mmawith the same exact block sums as the DP4A kernel (onemmaper block of 16, the 1.5 · 2²³ accumulator trick), W4A16 on BF16mmawith the weight decoded toa · scaleexactly (sm_80+). FP32 accumulation throughout; the order is the tensor cores', so the values matchnvfp4a_gemmwithin the rounding of an FP32 accumulation, not bit for bit.- One image of the weight (
nvfp4a_tc_image, kindsNVFP4A_TC_A16for BF16 activations andNVFP4A_TC_A8for INT8 / NVFP4), built once from any NVFP4 variant, serves both the GEMM and the GEMV; activations go throughNVFP4A_TC_Bper GEMM call, the GEMV takes them as they are. - GEMM: persistent CTAs with stream-K for the last wave (
nvfp4a_tc_workspace_bytes, zeroed once), rastered tiles; on sm_80+ acp.asynccopier and an mbarrier pipeline,ldmatrixfor BF16 activations; on sm_75 producer warps and sequence flags. - GEMV (1 … 8 columns): the weight's image streamed from DRAM once in 16-byte loads (L1 bypassed, evict-first in L2), the k sums kept in registers to the end, one store per output.
- PyTorch:
TCWeight(weight, act="exact" | "int8" | "nvfp4")holds the image;linear(x, tw)runs the tensor-core GEMV up to 8 rows and the GEMM beyond. NVFP4A_DP4A=1in the environment turns the tensor-core paths off.nvfp4a_gemmandlinearon a plainNVFP4Tensorare unchanged (DP4A / FFMA).
Tested by the author on a GTX 1080 (DP4A / FFMA paths, as before) and an L40S (sm_89): the C tests check every GEMM / GEMV case also through the tensor-core functions against a float64 reference, and make test-sm75 runs the Turing variant on the L40S from compute_75 PTX. The tensor-core GEMM's integer paths keep a few values in local memory at k-tile boundaries; every other kernel has none (Jetson TX1 / TX2 as before).
The wheel here is built for Python 3.12, PyTorch 2.7 and CUDA 12.6.