Repository navigation
mlops-attention 0.3.0
mlops-attention 0.3.0
Part of mlops: CUDA operations for NVIDIA GPUs sm_61 … sm_90.
Install
Nothing here downloads anything. Install first:
- PyTorch (CUDA build);
Then, in the directory with the downloaded wheel (cpXY = your Python, torchA.B / cuNNN = your PyTorch and its CUDA):
pip install --no-index -f . mlops-attention--no-index takes mlops-attention from this directory and its dependencies from what is installed, never from PyPI.
Other combinations: ./setup.sh attention from the source archive.
C / C++: unpack mlops-attention-0.3.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libmlops_attention.a (see its README).
Every artifact carries SASS for sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).
Assets
mlops-attention-0.3.0-linux-x86_64-cuda12.tar.xz(1,2M)mlops-attention-0.3.0-src.tar.gz(152K)mlops_attention-0.3.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl(6,7M)
Changes
The cache format and the decode / prefill API are unchanged. One ABI change: mlops_fbla_cache has a new trailing
field, tc_images (null: none), so programs built against 0.2.x headers must be recompiled. Python:
FBLACache.images, set by empty(); prefill_p128 in mlops.attention.fbla_reference.
- Prefill on tensor cores (sm_75 .. sm_89). The closed groups are attended through int8 tensor-core products of
the group's q-hat and the 6-bit codes (one softmax block of 128 tokens a group; the sink and the row's own group
in BF16 with BF16 P, blocks of 64), any q_heads / kv_heads ratio. sm_61 .. sm_72 and sm_90 stay on the DP4A
kernel. Against flash-attn 2.8.3 (BF16) on an L40S the prefill call (close, transcode, attention, finish) is
1.3 .. 1.9× faster, except an empty cache (0.85 .. 1.04×). The output differs fromdecode's (blocks of 16)
within the BF16 rounding of the output;fbla/tests/test_fbla.cuandprefill_p128model it in float64. - Persistent images (
mlops_fbla_cache.tc_images,mlops_fbla_images_bytes,empty()allocates them on
tensor-core GPUs: 34 KB per group and KV head, only with a nonzerocapacity). A chunked prefill transcodes each
closed group once instead of every closed group at every call: 256K context, chunks of 2048: 1.37 -> 0.024 ms. - Decode: the grid no longer depends on the context. The CTAs of the coarse and refine passes are the device's
resident CTAs split over the heads, and the partials are combined by a separate kernel that spreads a row over
SMs by channel. KV 1 / GQA 4 at 16K: 92 -> 31 us (flash-attn 2: 43 us); KV 8 is unchanged. The decode scratch
needs no initial contents (the CTA counters are gone) and is smaller by one counter per KV head. - Quality and benchmarks on WikiText-2.
bench/fbla_lm.py,fbla_token_metrics.pyandcapture_qkv.pyuse the
WikiText-2 test split by default (the tables measured on the repository's READMEs changed with them), and
measure the prefill kernels.bench/fbla_vs_sdpa.pytimes decode and prefill against flash-attn 2 (else
PyTorch's flash / memory-efficient SDPA). Qwen3-0.6B, 11 windows of 16K: FBLA 3+3 / 4+2 decode +0.0013 / +0.0014,
tensor-core prefill -0.0013 against BF16 SDPA (loss 2.784; windows differ by ±0.005), llama.cppq8_0+0.0011,
KIVI-4 +0.0010,q4_0+0.20 (docs/overview.md). - Wheel and distribution. The wheel (
linux_x86_64, not manylinux) is built on Ubuntu 24.04 (GCC 13) and needs
GLIBCXX_3.4.32; on older systems (Ubuntu 22.04 has 3.4.30) use./setup.sh attentionfrom the source archive. - Builds with nvcc 12.0 .. 13.4: the sm_75 tensor-core path avoids an early-clobber asm output nvcc 12.0 rejects.
- Tested on a GTX 1080 (sm_61, CUDA 12.0; this release's static library and wheel): the C++ tests (64), pytest (16),
samples 01 and 02. On an L40S (sm_89, CUDA 13.4): this release's static library passes the C++ tests (88, the
tensor-core prefill with and without persistent images), compute-sanitizer memcheck, synccheck and racecheck report no errors, hazards or warnings;
the sm_75 specialization was run as compute_75 PTX on the L40S (JIT; no sm_75 GPU was available), and the
Python wheel only on the GTX 1080 (the DP4A prefill).