Skip to content

mlops-attention 0.3.0

Choose a tag to compare

@srose69 srose69 released this 25 Oct 17:23
· 7 commits to main since this release

mlops-attention 0.3.0

Part of mlops: CUDA operations for NVIDIA GPUs sm_61 … sm_90.

Install

Nothing here downloads anything. Install first:

  • PyTorch (CUDA build);

Then, in the directory with the downloaded wheel (cpXY = your Python, torchA.B / cuNNN = your PyTorch and its CUDA):

pip install --no-index -f . mlops-attention

--no-index takes mlops-attention from this directory and its dependencies from what is installed, never from PyPI.
Other combinations: ./setup.sh attention from the source archive.

C / C++: unpack mlops-attention-0.3.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libmlops_attention.a (see its README).

Every artifact carries SASS for sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

  • mlops-attention-0.3.0-linux-x86_64-cuda12.tar.xz (1,2M)
  • mlops-attention-0.3.0-src.tar.gz (152K)
  • mlops_attention-0.3.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (6,7M)

Changes

The cache format and the decode / prefill API are unchanged. One ABI change: mlops_fbla_cache has a new trailing
field, tc_images (null: none), so programs built against 0.2.x headers must be recompiled. Python:
FBLACache.images, set by empty(); prefill_p128 in mlops.attention.fbla_reference.

  • Prefill on tensor cores (sm_75 .. sm_89). The closed groups are attended through int8 tensor-core products of
    the group's q-hat and the 6-bit codes (one softmax block of 128 tokens a group; the sink and the row's own group
    in BF16 with BF16 P, blocks of 64), any q_heads / kv_heads ratio. sm_61 .. sm_72 and sm_90 stay on the DP4A
    kernel. Against flash-attn 2.8.3 (BF16) on an L40S the prefill call (close, transcode, attention, finish) is
    1.3 .. 1.9× faster, except an empty cache (0.85 .. 1.04×). The output differs from decode's (blocks of 16)
    within the BF16 rounding of the output; fbla/tests/test_fbla.cu and prefill_p128 model it in float64.
  • Persistent images (mlops_fbla_cache.tc_images, mlops_fbla_images_bytes, empty() allocates them on
    tensor-core GPUs: 34 KB per group and KV head, only with a nonzero capacity). A chunked prefill transcodes each
    closed group once instead of every closed group at every call: 256K context, chunks of 2048: 1.37 -> 0.024 ms.
  • Decode: the grid no longer depends on the context. The CTAs of the coarse and refine passes are the device's
    resident CTAs split over the heads, and the partials are combined by a separate kernel that spreads a row over
    SMs by channel. KV 1 / GQA 4 at 16K: 92 -> 31 us (flash-attn 2: 43 us); KV 8 is unchanged. The decode scratch
    needs no initial contents (the CTA counters are gone) and is smaller by one counter per KV head.
  • Quality and benchmarks on WikiText-2. bench/fbla_lm.py, fbla_token_metrics.py and capture_qkv.py use the
    WikiText-2 test split by default (the tables measured on the repository's READMEs changed with them), and
    measure the prefill kernels. bench/fbla_vs_sdpa.py times decode and prefill against flash-attn 2 (else
    PyTorch's flash / memory-efficient SDPA). Qwen3-0.6B, 11 windows of 16K: FBLA 3+3 / 4+2 decode +0.0013 / +0.0014,
    tensor-core prefill -0.0013 against BF16 SDPA (loss 2.784; windows differ by ±0.005), llama.cpp q8_0 +0.0011,
    KIVI-4 +0.0010, q4_0 +0.20 (docs/overview.md).
  • Wheel and distribution. The wheel (linux_x86_64, not manylinux) is built on Ubuntu 24.04 (GCC 13) and needs
    GLIBCXX_3.4.32; on older systems (Ubuntu 22.04 has 3.4.30) use ./setup.sh attention from the source archive.
  • Builds with nvcc 12.0 .. 13.4: the sm_75 tensor-core path avoids an early-clobber asm output nvcc 12.0 rejects.
  • Tested on a GTX 1080 (sm_61, CUDA 12.0; this release's static library and wheel): the C++ tests (64), pytest (16),
    samples 01 and 02. On an L40S (sm_89, CUDA 13.4): this release's static library passes the C++ tests (88, the
    tensor-core prefill with and without persistent images), compute-sanitizer memcheck, synccheck and racecheck report no errors, hazards or warnings;
    the sm_75 specialization was run as compute_75 PTX on the L40S (JIT; no sm_75 GPU was available), and the
    Python wheel only on the GTX 1080 (the DP4A prefill).