Skip to content

NVFP4EffMA 0.2.0

Choose a tag to compare

View all tags
@srose69 srose69 released this 20 Oct 07:08
· 11 commits to main since this release
v0.2.0
2ebad24

NVFP4EffMA 0.2.0

NVFP4 GEMM, GEMV and quantization for NVIDIA GPUs without native FP4 (sm_50 … sm_90).

Install

PyTorch: pick the wheel matching your Python (cpXY), PyTorch (torchA.B) and its CUDA (cuNNN):

pip install nvfp4effma-0.2.0+<cuNNNtorchA.B>-<cpXY>-<cpXY>-linux_x86_64.whl

Other combinations: pip install ./binds/python --no-build-isolation from the source archive.

C / C++: unpack nvfp4effma-0.2.0-linux-x86_64-cuda12.tar.gz and link lib/libnvfp4effma.a (see its README).

Every artifact carries SASS for sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

  • nvfp4effma-0.2.0-linux-x86_64-cuda12.tar.gz (5.6M)
  • nvfp4effma-0.2.0-src.tar.gz (96K)
  • nvfp4effma-0.2.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (8.4M)

What's new

  • W4A8: NVFP4 weights × int8 activations with an FP32 scale per 16 values (NVFP4A_INT8; nvfp4a_quantize produces them). Exact integer block sums on DP4A (|p| ≤ 24384), one rounding per block, the same bits on the FFMA path below sm_61. 9.3–9.4 ms at 4096³ (7.3 TMAC/s, 2.1× cuBLAS SGEMM) — the accuracy class of llama.cpp's int8 activations.
  • Small and medium batches: the GEMV kernel now also runs beyond 8 columns, in groups of 8, up to where the tiled kernel takes over (48–56 columns, 32 for FP32); a 64-wide DP4A tile (two blocks per SM) covers 57–64 and 129–192 columns.
  • PyTorch: linear(..., act=) with "auto" (default: exact for up to 2 rows, int8 activations beyond, about twice as fast), "exact", "int8", "nvfp4"; the same option on NVFP4Linear / quantize_linear_.
  • Against llama.cpp's NVFP4 kernels on the same GTX 1080 (interleaved runs, int8 activations on both sides): faster on every shape measured — 1.47× at 4096³, 3.9–4.5× for 8-token decoding, 1.45–1.66× at 512 tokens. Table and method: docs/performance.md.
  • GEMM candidates with 128-thread blocks no longer spill (register budget of 4 × 128 threads).
  • The GEMV kernel's NVFP4-activation path computes the exact block sums of the tiled kernels (DP4A per block), 2–3× faster for 4–8 columns.

API change: nvfp4a_dtype gains NVFP4A_INT8 = 4; ld_scale of INT8 operands counts floats. Everything else is source-compatible with 0.1.0.

Tested and tuned by the author on a GTX 1080 (sm_61) only, plus the code path of GPUs without DP4A (SASS built for sm_60, run on the GTX 1080). Other architectures are compiled, not run. No stack or local memory in any kernel, except on Jetson TX1 / TX2 (sm_53 / sm_62, 32 K registers per block), where the heaviest kernels spill.

The wheel here is built for Python 3.12, PyTorch 2.7 and CUDA 12.6.

Assets 6
Loading