Skip to content

NVFP4EffMA 0.6.0

Choose a tag to compare

View all tags
@srose69 srose69 released this 28 Oct 14:46
· 2 commits to main since this release
v0.6.0
3b8cadf

NVFP4EffMA 0.6.0

NVFP4 GEMM, GEMV and quantization for NVIDIA GPUs without native FP4 (sm_50 … sm_90).

Install

PyTorch: pick the wheel matching your Python (cpXY), PyTorch (torchA.B) and its CUDA (cuNNN):

pip install nvfp4effma-0.6.0+<cuNNNtorchA.B>-<cpXY>-<cpXY>-linux_x86_64.whl

Other combinations: pip install ./binds/python --no-build-isolation from the source archive.

C / C++: unpack nvfp4effma-0.6.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libnvfp4effma.a (see its README).

Every artifact carries SASS for sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

  • nvfp4effma-0.6.0-linux-x86_64-cuda12.tar.xz (12M)
  • nvfp4effma-0.6.0-src.tar.gz (208K)
  • nvfp4effma-0.6.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (75M)

What's in it

A transformer layer's epilogue in the call, and the calls' work as items.

  • nvfp4a_gemm_ex, nvfp4a_gemv_tc_ex, nvfp4a_gemm_tc_ex extend the plain calls (B INT8 / BF16,
    A's scales linear E4M3) by any combination of: the deferred RMSNorm column scale (the parts of each
    column's sum of squares read through L2, one warp of the CTA reduces them and shares the scales), SwiGLU
    of interleaved gate / up rows, and a second weighted output with the parts of its sum of squares — the
    input of the next layer's norm. Fixed summation orders, no atomics, on the CUDA-core GEMV, the DP4A /
    FFMA tiles and the tensor-core paths alike.
  • nvfp4effma_items.h: the same work as items for an executor of one's own — the number of items,
    their CTA shape and shared memory, per-item read / write ranges (with boundary fallbacks) and device
    entries; coherent intra-launch reads of mutable activations through L2, and the executor's publication
    protocol (waits / signals stay with the executor). Works from NVRTC too; the pairing of a compute_75
    library with a different-architecture executor is checked through cudaFuncAttributes::ptxVersion.
  • Weight images on any GPU (nvfp4a_tc_image): the image — and the A16 ↔ A8 and image → tensor
    conversions — build wherever the weights are, not only where the tensor cores are.
  • GEMV extended-epilogue performance: the NORM scales are computed by one warp while the others load
    the epilogue's inputs, and the INT8 kernels keep the plain call's k-loop schedule (a ptxas register
    allocation that had halved the loop's bandwidth is pinned by launch bounds): against the plain call,
    n = 8 tails cost ×1.0 … ×1.2 (was ×1.4 … ×1.9 on INT8), BF16 tiles ≈ ×1.0, tensor cores ×0.97 … ×1.23
    by mask (bench_epilogue, bench_gemm / bench_gemv with the new Extended epilogue sections;
    benches/1080.md and benches/l40s.md).

Tested by the author on a GTX 1080 (sm_61 and sm_60 builds: the full C suite, all architectures' ordinary
kernels byte-identical where nothing changed) and an L40S (sm_89 and compute_75: the full suite, items and
epilogue); the wheel installs and passes its pytest on both (the tensor-core tests skip below sm_75).

The wheel here is built for Python 3.12, PyTorch 2.7 and CUDA 12.6.

Assets 6
Loading