Skip to content

v0.2.4 — differentiable grouped NF4 GEMM (training) + NVMe arena surface

Choose a tag to compare

@pjordanandrsn pjordanandrsn released this 30 Jul 00:23
v0.2.4

v0.2.4 — the differentiable grouped NF4 GEMM (training half) + NVMe arena surface

nf4_qlora makes the fused kernel trainable: an autograd.Function whose backward
re-decodes one expert at a time, plus fused_grouped_lora composing the frozen
4-bit projection with the trainable B(Ax) delta at the pre-activation point.

Verified on real hardware before this tag:

  • gradient correctness -- 5/5 on sm_86 and sm_89, gradient relative error
    exactly 0.0 against the dequantize-then-matmul reference, forward 0.0027
  • offload safety at 48 layers (Qwen3-30B-A3B, RTX 4090): 0.756x the reference
    peak, no OOM, where holding weights on the ctx OOMed at 22.41 GB
  • frozen experts bit-identical after training -- 192 tensors, 16.3 GB hashed,
    zero placeholders, with a positive control proving a single flipped byte IS
    detected
  • loss parity with the reference on varied data and a held-out eval:
    0.337073 vs 0.340271 (0.95%), train median delta 0.0136 (band 0.05)

Also: triton is now a Linux-only dependency, so the pure-torch provenance and
NVMe-arena surface installs on macOS/Windows; nvme_arena, nvme_bake_nf4 and
nvme_reader ship in the wheel with the smoke exercising the CLI and the O_DIRECT
alignment guard.

0.2.3 was committed but never tagged or published; PyPI goes 0.2.2 -> 0.2.4.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
-----BEGIN PGP SIGNATURE-----

iJEEABYKADkWIQQ7n/u5ZVSf3n/PS1XqrS1lj4+/ngUCamqZMRsUgAAAAAAEAA5t
YW51MiwyLjUrMS4xMiwwLDMACgkQ6q0tZY+Pv54NXAD9FrbYfM6cdN3SNnF2kVep
CuWRNZjY02/kcwG8andZxU4A/2BTNbZs6adxX9nNjtfhSRXKMDp+/R5Z4hwZdODC
hzEG
=n67z
-----END PGP SIGNATURE-----