Repository navigation
Added
- [Torch] Optimized Triangle Multiplicative Update kernels.
cuet.triangle_multiplicative_updatenow runs new prebuilt optimized kernels on the shapes each kernel was validated on. All other calls use the 0.12 implementation, unchanged. CUDA 13 wheels only. See "Optimized TriMul kernels" below.- Inference on H100 SXM, H200, GH200 480GB, B200 and RTX PRO 6000 Blackwell; training on H100 SXM, H200, GH200 480GB and B200.
- Pair width C and hidden width D: 128/128, 256/256, 384/256 and 512/512, with and without projection bias. The exact supported shapes are listed under "Supported configurations" below.
- [JAX]
cuex.triangle_attentionacceptsmask=Nonefor unmasked attention, ascuet.triangle_attentiondoes. The call then runs the unmasked kernels, without an all-True mask. This is supported throughjax.gradandjax.vmap. With an oldercuequivariance-ops-jax, the frontend makes an all-True mask and passes it. (#307)
Bug Fixes
- [Torch/JAX] Triangle Attention on RTX PRO 6000 Blackwell, CUDA 13 wheels: fixed a register spill in the masked forward kernels at head dimension 64 (BF16 and FP16) that made the masked forward slower than in older builds. Results are bit-identical.
- [JAX] Unmasked Triangle Attention calls no longer run the Boolean-mask kernels. Pass
mask=Noneto get the unmasked forward and backward kernels. - [Torch] Fixed a failure with PyTorch 2.14 in training: the fused gated dual GEMM backward returned too many gradients, and PyTorch 2.14 raises an error for this. Every training call that reached this kernel failed, including the 0.12 TriMul training route.
Breaking Changes
- None.
Notes
- Supported PyTorch versions: 2.11, 2.12 and 2.13. PyTorch 2.14 is not yet supported. A known PyTorch 2.14 training failure is fixed in this release (see Bug Fixes), but full 2.14 support comes in a future release.
- The newly optimized TriMul torch kernels run only with the CUDA 13 wheels (
cuequivariance-ops-torch-cu13). The CUDA 12 wheels do not contain the optimized TriMul kernels and instead use the 0.12 implementation. - The optimized TriMul kernels run only on these GPUs: H100 SXM (80 GB HBM3), H200, GH200 480GB, B200 and RTX PRO 6000 Blackwell (Server and Workstation Edition). All other GPUs use the 0.12 implementation. This includes H100 PCIe, H100 NVL, B300 / GB300, A100, L40(S) and Rubin.
- Recommended usage is unchanged:
torch.compile(model, mode="reduce-overhead"). Warm up for a few iterations first. - The optimized kernels were added only on shapes where they passed the accuracy release gate and were faster than the 0.12 implementation in our CI.
Optimized TriMul kernels
Terms used in this section. The names refer to the arguments of cuet.triangle_multiplicative_update:
| Term | Meaning | Where it is in the call |
|---|---|---|
| B | Batch size | x.shape[0] |
| N | Sequence length (number of tokens or residues) | x.shape[1] and x.shape[2] |
| C | Pair width: the channel size of the pair representation (c_z in AlphaFold-style models) |
x.shape[3] |
| D | Hidden width: the channel size of the projected pair inside the update (c_hidden) |
p_in_weight.shape[0] // 2 |
| Bias | The four projection biases p_in_bias, g_in_bias, p_out_bias and g_out_bias |
"yes" = all four given; "no" = all four None |
The tensor shapes that the optimized kernels expect:
| Argument | Shape |
|---|---|
x |
(B, N, N, C), exactly one batch dimension |
mask |
(B, N, N) or None |
norm_in_weight, norm_in_bias |
(C,) |
p_in_weight, g_in_weight |
(2D, C) |
p_in_bias, g_in_bias |
(2D,) |
norm_out_weight, norm_out_bias |
(D,) |
p_out_weight |
(C, D) |
g_out_weight |
(C, C) |
p_out_bias, g_out_bias |
(C,) |
All tensors must be contiguous and on the same GPU. A call that gives only some of the four projection biases, has more than one leading batch dimension, or has other shapes uses the 0.12 implementation. For C = D (128, 256, 512), the shapes are the same as in the function's docstring, where both widths are called D. For C = 384, D = 256, use the shapes in the table above.
Where the optimized kernels run
| GPU | Inference cells | Training cells |
|---|---|---|
| H100 SXM (80 GB HBM3) | 131 | 54 |
| H200 | 131 | 54 |
| GH200 480GB | 131 | 54 |
| B200 | 222 | 66 |
| RTX PRO 6000 Blackwell (Server and Workstation Edition) | 190 | — (inference only) |
A cell is one combination of pair width C, hidden width D, bias or no bias, batch size B and sequence length N. The optimized kernels run only on the exact cells listed under "Supported configurations". On all other cells, the call uses the 0.12 implementation. In inference, the incoming and outgoing directions are both supported on every listed cell.
How to use the optimized kernels
- Install the CUDA 13 wheels.
- Call
cuet.triangle_multiplicative_updatein BF16: the pair and all parameters must be BF16. Alternatively, pass FP32 inputs withprecision="bfloat16". Mixed precision (autocast) keeps the 0.12 implementation in 0.13. See "Precision and mixed precision" below. - Under
torch.compile, each listed shape gets its own graph. For inference, pad or bucket your inputs to a small set of supported shapes. To use the optimized kernels on many shapes:- compile with
dynamic=False; - set
torch._dynamo.config.recompile_limithigher than the number of shapes.
- compile with
With the default (automatic dynamic shapes), at most recompile_limit − 2 shapes per process use the optimized kernels (6 by default). The optimized kernels and the 0.12 implementation share the recompile budget of the frame. With many different sequence lengths, the frame can reach recompile_limit. On B200 and RTX PRO 6000 at C=384, D=256, this occurs at about the 9th new shape. PyTorch then runs that frame in eager mode for all further new shapes. cuEquivariance 0.12.0 stayed compiled in this case. A fixed set of shapes, or dynamic=False with a higher recompile_limit, prevents this.
Precision and mixed precision
The optimized kernels take BF16 inputs only. The input dtypes and the precision argument decide which implementation runs:
| Your call | Implementation in 0.13 |
|---|---|
| Pair and all parameters BF16 (with or without BF16 autocast) | Optimized |
FP32 pair and FP32 parameters under BF16 autocast (for example torch.autocast("cuda", dtype=torch.bfloat16), Lightning bf16-mixed) |
0.12 |
| BF16 pair with FP32 parameters (with or without autocast) | 0.12 |
FP32 inputs with precision="bfloat16" (with or without autocast) |
Optimized |
FP32 without autocast, FP16 inputs, autocast to FP16, precision="ieee" or "mxfp8" |
0.12 |
"Optimized" in this table applies only to the shapes listed under "Supported configurations". A call with any other shape uses the 0.12 implementation, whatever its dtypes and precision.
- Under autocast, the LayerNorms run in FP32. The 0.12 implementation normalizes the FP32 pair with FP32 LayerNorm parameters. Only the GEMM operands become BF16. This is why autocast calls stay on 0.12 in 0.13: the optimized kernels need BF16 LayerNorm inputs, and the results would change.
precision="bfloat16"rounds all inputs to BF16 before the call. This includes the pair, the mask, the LayerNorm weights and biases, and the projection weights. The LayerNorms then read BF16 values. The rounding is differentiable, so gradients still reach FP32 master weights. 0.12 applies the same rounding for this argument, so the optimized kernels and 0.12 get the same inputs.- Accuracy: compared with autocast,
precision="bfloat16"adds the rounding error of the pair at the LayerNorm input. If your model runs under autocast today, check its accuracy before you change toprecision="bfloat16"to get the optimized kernels. - Numerics: the optimized kernels' outputs agree with the 0.12 implementation within its numerical tolerance, but they are not bitwise identical.
Known issues
-
Small-sequence training on B200. A few B200 training cells at short sequence lengths (N=128) can run slightly slower than the 0.12 implementation. We are working to resolve this in a future release.
-
Exported models use the 0.12 implementation.
torch.export(strict and non-strict), ONNX export (and so TensorRT), and traces with symbolic sizes outsidetorch.compile(make_fx,FakeTensorMode) never record an optimized kernel. -
Eager-mode routing cost. In eager mode, calls that do not take an optimized kernel pay a small routing check on each call. The cost is most visible at small shapes, and most of all on RTX PRO 6000. Under
torch.compile, the check costs nothing. -
Second backward and double backward. In training, a second backward through one forward (
retain_graph=True):- is exact in eager mode, but needs about 1.9× the memory of the first backward;
- gives NaN gradients when it is replayed from a CUDA graph. This is by design: the result is visibly wrong, never silently wrong. The first backward of every replay is exact.
Double backward (
create_graph=True) raises an error. -
Exclusive GPU. Some optimized kernels use software grid barriers and expect exclusive use of the GPU. Under MPS sharing without SM partitions, they can hang.
-
FP32 TriMul on GPUs with about 100 KB shared memory per block (L40S and other Ada GPUs, RTX PRO 6000 and other sm_120 GPUs): with Triton newer than 3.6 (PyTorch 2.14 ships 3.8), the FP32 0.12 Triton kernel at C=128 fails with
OutOfResources. This is not an issue of the optimized kernels; 0.12.0 has the same problem. BF16 calls are not affected. -
Per-rank compile limit under DDP. Under DDP or multi-node training, each rank applies the recompile limit separately. Ranks that see shapes in a different order can run the optimized kernels on different shapes. Results are not affected, but a small speed difference between ranks is possible.
Supported configurations
The optimized kernels run on these exact cells: (C, D, bias) and, for each batch size B, the listed sequence lengths N. A batch size or sequence length that is not listed for that row uses the 0.12 implementation. The list is the same for the incoming and outgoing directions.
H100 SXM, H200, GH200 480GB, inference (131 cells)
| C | D | Bias | Batch size B: sequence lengths N |
|---|---|---|---|
| 128 | 128 | no | B=1: 256, 384, 512, 640, 768, 1024, 1280, 1536, 2048; B=4: 384, 512, 768 |
| 128 | 128 | yes | B=1: 256, 384, 512, 640, 768, 1024, 1280 |
| 256 | 256 | no | B=1: 128, 256, 384, 512, 768, 1024, 1536, 2048; B=4: 512 |
| 256 | 256 | yes | B=1: 128, 256, 384, 512, 768, 1024, 2048; B=2: 128, 256, 384, 512, 768, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=8: 128, 256, 384, 512, 768; B=16: 128, 256, 384, 512 |
| 384 | 256 | no | B=1: 128, 160, 192, 224, 256, 288, 320, 352, 384, 448, 512, 640, 768, 896, 1024; B=2: 128, 256, 384, 512, 768; B=4: 128, 256, 384, 512, 768; B=5: 256, 320, 384, 448, 512, 560, 608; B=8: 128, 256, 384, 512; B=16: 128, 256, 384, 512 |
| 384 | 256 | yes | B=1: 128, 256, 384, 512, 768; B=2: 128, 256, 384, 512, 768; B=4: 128, 256, 384, 512, 768; B=8: 128, 256, 384, 512; B=16: 128, 256, 384, 512 |
| 512 | 512 | no | B=1: 128, 256, 384, 512, 768, 1024 |
| 512 | 512 | yes | B=1: 128, 256, 384, 512, 768, 1024 |
H100 SXM, H200, GH200 480GB, training (54 cells)
| C | D | Bias | Batch size B: sequence lengths N |
|---|---|---|---|
| 128 | 128 | no | B=1: 256, 384, 512, 640, 768; B=4: 384, 512, 768 |
| 128 | 128 | yes | B=1: 256, 384, 512, 640, 768 |
| 256 | 256 | no | B=1: 256, 384, 512; B=4: 512 |
| 256 | 256 | yes | B=1: 256, 384, 512; B=2: 128, 256, 384, 512; B=4: 128, 256, 384, 512 |
| 384 | 256 | no | B=1: 256, 384; B=2: 256, 384; B=4: 256, 384; B=8: 256, 384; B=16: 256, 384 |
| 384 | 256 | yes | B=1: 256, 384; B=2: 256, 384; B=4: 256, 384; B=8: 256, 384; B=16: 256, 384 |
| 512 | 512 | no | B=1: 128, 256, 384 |
| 512 | 512 | yes | B=1: 128, 256, 384 |
B200, inference (222 cells)
| C | D | Bias | Batch size B: sequence lengths N |
|---|---|---|---|
| 128 | 128 | no | B=1: 64, 98, 128, 200, 256, 384, 512, 640, 768, 1024, 1280, 1536, 2048; B=2: 98; B=4: 384, 512, 768; B=8: 512; B=16: 128 |
| 128 | 128 | yes | B=1: 64, 98, 128, 200, 256, 384, 512, 640, 768, 1024, 1280, 1536, 2048; B=2: 98; B=16: 128 |
| 256 | 256 | no | B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024, 1536, 2048; B=2: 98; B=4: 512; B=8: 512; B=16: 128 |
| 256 | 256 | yes | B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024, 2048; B=2: 98, 128, 256, 384, 512, 768, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=8: 128, 256, 384, 512, 768; B=16: 128, 256, 384, 512 |
| 384 | 256 | no | B=1: 32, 48, 64, 80, 96, 98, 112, 128, 130, 160, 192, 200, 224, 256, 257, 288, 300, 320, 352, 384, 448, 500, 512, 640, 768, 896, 1000, 1024; B=2: 64, 98, 128, 200, 256, 384, 512, 768, 1000, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=5: 256, 320, 384, 448, 512, 560, 608; B=8: 128, 256, 257, 300, 384, 500, 512, 768; B=16: 32, 48, 64, 80, 96, 98, 112, 128, 130, 200, 256, 384, 512 |
| 384 | 256 | yes | B=1: 32, 48, 64, 80, 96, 98, 112, 128, 130, 200, 256, 257, 300, 384, 385, 500, 512, 768, 1000, 1024; B=2: 64, 98, 128, 200, 256, 384, 385, 512, 768, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=8: 128, 256, 384, 512, 768; B=16: 128, 256, 384, 512 |
| 512 | 512 | no | B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024; B=2: 98; B=8: 512; B=16: 128 |
| 512 | 512 | yes | B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024; B=2: 98; B=8: 512; B=16: 128 |
B200, training (66 cells)
| C | D | Bias | Batch size B: sequence lengths N |
|---|---|---|---|
| 128 | 128 | no | B=1: 128, 256, 384, 512, 640, 768, 1024; B=4: 384, 512, 768 |
| 128 | 128 | yes | B=1: 128, 256, 384, 512, 640, 768, 1024 |
| 256 | 256 | no | B=1: 128, 256, 384, 512, 768; B=4: 512 |
| 256 | 256 | yes | B=1: 128, 256, 384, 512, 768; B=2: 128, 256, 384, 512, 768; B=4: 128, 256, 384, 512, 768 |
| 384 | 256 | no | B=1: 256, 384; B=2: 256, 384; B=4: 256, 384; B=8: 256, 384; B=16: 256, 384 |
| 384 | 256 | yes | B=1: 256, 384; B=2: 256, 384; B=4: 256, 384; B=8: 256, 384; B=16: 256, 384 |
| 512 | 512 | no | B=1: 128, 256, 384, 512 |
| 512 | 512 | yes | B=1: 128, 256, 384, 512 |
RTX PRO 6000 Blackwell (Server and Workstation Edition), inference (190 cells)
| C | D | Bias | Batch size B: sequence lengths N |
|---|---|---|---|
| 128 | 128 | no | B=1: 64, 98, 128, 200, 256, 384, 512, 640, 768, 1024, 1280, 1536, 2048; B=2: 98; B=4: 384, 512, 768; B=8: 512; B=16: 128 |
| 128 | 128 | yes | B=1: 64, 98, 128, 200, 256, 384, 512, 640, 768, 1024, 1280, 1536; B=2: 98; B=8: 512; B=16: 128 |
| 256 | 256 | no | B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024, 1536, 2048; B=2: 98; B=4: 512; B=8: 512; B=16: 128 |
| 256 | 256 | yes | B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024, 2048; B=2: 98, 128, 256, 384, 512, 768, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=8: 128, 256, 384, 512, 768; B=16: 128, 256, 384, 512 |
| 384 | 256 | no | B=1: 32, 48, 64, 80, 96, 98, 112, 128, 130, 160, 192, 200, 224, 256, 257, 288, 300, 320, 352, 384, 385, 448, 500, 512, 640, 768, 896, 1024; B=2: 64, 98, 128, 200, 256, 384, 385, 512, 768; B=4: 128, 256, 384, 512, 768; B=5: 256, 320, 384, 448, 512, 560, 608; B=8: 128, 256, 257, 300, 384, 385, 500, 512; B=16: 32, 48, 64, 80, 96, 98, 112, 128, 130, 200, 256 |
| 384 | 256 | yes | B=1: 32, 48, 64, 80, 96, 98, 112, 128, 130, 200, 256, 257, 300, 384, 385, 500, 512; B=2: 64, 98, 200, 385; B=8: 512; B=16: 128 |
| 512 | 512 | no | B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024; B=2: 98; B=8: 512; B=16: 128 |
| 512 | 512 | yes | B=1: 128, 256, 384, 512, 768, 1024 |