Repository navigation
MagiAttention V1.1.8
This release adds index-sparse DSA kernels for Blackwell, learnable attention sinks in the CuTeDSL backend, and overlap-aware range scheduling. It also fixes Top-k compatibility and concurrency issues, removes backward range-contract device-to-host reads, and improves sink preprocessing and sparse-attention execution.
New Features
-
Index-Sparse DSA Kernels (#388)
- Add native SM100 forward/backward and SM120 forward entry points using per-query-block K-token indices.
- Add packed-layout Triton DSA attention, a grouped-layout Triton backward alternative, and CuTeDSL radix-select Top-k helpers with deterministic index ordering and tie-breaking.
- Support LSE-output gradients in native SM100 index-sparse backward and packed Triton autograd.
-
Learnable Attention Sinks (#367)
- Support multiple learnable sink logits and sink gradients in the CuTeDSL backend, including dense and range attention.
- Integrate sink handling into distributed attention and FP32 partial-gradient accumulation.
-
Communication-Overlap Scheduling (#374)
- Add
sm_marginand dynamic range scheduling on SM100/SM110 to reserve execution resources for concurrent communication. - Enable Q/O communication with CuTeDSL and provide guidance for NCCL CTA-cluster configuration.
- Add
Enhancements and Performance
- Bypass redundant
logsumexpfor a single attention sink while retaining FP32 conversion and contiguous preprocessing; preserve the existing multi-sink reduction (#388). - Add asynchronous V gathers and L2-sized K passes to SM100 index-sparse backward, with FP32 dQ accumulation across passes (#388).
- Treat packed Triton KV length as a runtime argument to avoid specializing solely on
SKV(#388).
Bug Fixes
- Fix CuTeDSL Top-k compilation with DSL 4.4.2 by passing raw LLVM pointers to memory operations and using compatible vector extraction (#388).
- Isolate Top-k synchronization workspaces by CUDA stream and launch through the current Torch stream, preventing ordinary multi-stream calls from sharing mutable synchronization state (#388).
- Return zero output and negative-infinity LSE for empty packed-Triton sparse rows, with zero query gradients, instead of dividing by zero (#388).
- Remove first-backward device-to-host range reads by deriving range-contract flags from CPU metadata. Preserve the actual reordered-range semantics, including Q sortedness, Q union coverage, and K partition checks (#388).
- Correct persistent-backward tile handoffs and FP32 forward shared-memory reuse on SM100/SM110 (#374).
- Correct sink handling for empty-key and skipped distributed stages, including asymmetric K/V head dimensions (#367).
Dependency Updates
-
Extend native CuTeDSL support through DSL 4.8.1. Supported pairs remain explicitly validated (#388):
nvidia-cutlass-dslquack-kernels4.4.2 0.4.1 4.6.2 0.6.4 4.7.0–4.8.1 0.6.5 DSL 4.6.0 and 4.6.1 remain excluded;
libs-basemust match the DSL version. Independent pip ranges do not imply support for arbitrary DSL/Quack combinations. -
FA4 retains its separate DSL
>=4.4.2,<4.6.0requirement and tested DSL 4.4.2 / Quack 0.4.1 pair.
CI and Build
- Pin the lint runner to Ubuntu 24.04 for a stable CI environment (#388).
Compatibility Notes
- Native index-sparse kernels are low-level entry points with FP16/BF16 and head-dimension-128 constraints; they do not automatically replace public attention dispatch. Native forward requires
q_block_size * (num_heads_q / num_heads_kv) == 128. SM120 index-sparse backward is not included. - Packed Triton attention returns log2 LSE; native index-sparse attention uses natural-log LSE. Their layouts and index contracts differ and require explicit adaptation when combining backends.
sm_marginapplies to supported SM100/SM110 range paths; forward head dimension 192 does not honor the reservation. For NCCL communication overlap, configureNCCL_CGA_CLUSTER_SIZE=1before creating process groups.- Removing backward range-contract reads does not guarantee that every metadata-construction or backend path is globally free of device-to-host transfers.