Skip to content

NVFP4EffMA 0.5.0

Choose a tag to compare

@srose69 srose69 released this 26 Oct 03:25
· 7 commits to main since this release

NVFP4EffMA 0.5.0

NVFP4 GEMM, GEMV and quantization for NVIDIA GPUs without native FP4 (sm_50 … sm_90).

Install

PyTorch: pick the wheel matching your Python (cpXY), PyTorch (torchA.B) and its CUDA (cuNNN):

pip install nvfp4effma-0.5.0+<cuNNNtorchA.B>-<cpXY>-<cpXY>-linux_x86_64.whl

Other combinations: pip install ./binds/python --no-build-isolation from the source archive.

C / C++: unpack nvfp4effma-0.5.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libnvfp4effma.a (see its README).

Every artifact carries SASS for sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

  • nvfp4effma-0.5.0-linux-x86_64-cuda12.tar.xz (8,7M)
  • nvfp4effma-0.5.0-src.tar.gz (168K)
  • nvfp4effma-0.5.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (59M)

What's in it

Tensor cores for sm_75 … sm_89.

  • nvfp4a_gemm_tc / nvfp4a_gemv_tc: W4A4 and W4A8 on INT8 mma with the same exact block sums as the DP4A kernel (one mma per block of 16, the 1.5 · 2²³ accumulator trick), W4A16 on BF16 mma with the weight decoded to a · scale exactly (sm_80+). FP32 accumulation throughout; the order is the tensor cores', so the values match nvfp4a_gemm within the rounding of an FP32 accumulation, not bit for bit.
  • One image of the weight (nvfp4a_tc_image, kinds NVFP4A_TC_A16 for BF16 activations and NVFP4A_TC_A8 for INT8 / NVFP4), built once from any NVFP4 variant, serves both the GEMM and the GEMV; activations go through NVFP4A_TC_B per GEMM call, the GEMV takes them as they are.
  • GEMM: persistent CTAs with stream-K for the last wave (nvfp4a_tc_workspace_bytes, zeroed once), rastered tiles; on sm_80+ a cp.async copier and an mbarrier pipeline, ldmatrix for BF16 activations; on sm_75 producer warps and sequence flags.
  • GEMV (1 … 8 columns): the weight's image streamed from DRAM once in 16-byte loads (L1 bypassed, evict-first in L2), the k sums kept in registers to the end, one store per output.
  • PyTorch: TCWeight(weight, act="exact" | "int8" | "nvfp4") holds the image; linear(x, tw) runs the tensor-core GEMV up to 8 rows and the GEMM beyond.
  • NVFP4A_DP4A=1 in the environment turns the tensor-core paths off. nvfp4a_gemm and linear on a plain NVFP4Tensor are unchanged (DP4A / FFMA).

Tested by the author on a GTX 1080 (DP4A / FFMA paths, as before) and an L40S (sm_89): the C tests check every GEMM / GEMV case also through the tensor-core functions against a float64 reference, and make test-sm75 runs the Turing variant on the L40S from compute_75 PTX. The tensor-core GEMM's integer paths keep a few values in local memory at k-tile boundaries; every other kernel has none (Jetson TX1 / TX2 as before).

The wheel here is built for Python 3.12, PyTorch 2.7 and CUDA 12.6.