Skip to content

v0.13.0

Latest

Choose a tag to compare

@jomitchellnv jomitchellnv released this 15 Oct 19:52
· 3 commits to main since this release
eaafc3b

Added

  • [Torch] Optimized Triangle Multiplicative Update kernels. cuet.triangle_multiplicative_update now runs new prebuilt optimized kernels on the shapes each kernel was validated on. All other calls use the 0.12 implementation, unchanged. CUDA 13 wheels only. See "Optimized TriMul kernels" below.
    • Inference on H100 SXM, H200, GH200 480GB, B200 and RTX PRO 6000 Blackwell; training on H100 SXM, H200, GH200 480GB and B200.
    • Pair width C and hidden width D: 128/128, 256/256, 384/256 and 512/512, with and without projection bias. The exact supported shapes are listed under "Supported configurations" below.
  • [JAX] cuex.triangle_attention accepts mask=None for unmasked attention, as cuet.triangle_attention does. The call then runs the unmasked kernels, without an all-True mask. This is supported through jax.grad and jax.vmap. With an older cuequivariance-ops-jax, the frontend makes an all-True mask and passes it. (#307)

Bug Fixes

  • [Torch/JAX] Triangle Attention on RTX PRO 6000 Blackwell, CUDA 13 wheels: fixed a register spill in the masked forward kernels at head dimension 64 (BF16 and FP16) that made the masked forward slower than in older builds. Results are bit-identical.
  • [JAX] Unmasked Triangle Attention calls no longer run the Boolean-mask kernels. Pass mask=None to get the unmasked forward and backward kernels.
  • [Torch] Fixed a failure with PyTorch 2.14 in training: the fused gated dual GEMM backward returned too many gradients, and PyTorch 2.14 raises an error for this. Every training call that reached this kernel failed, including the 0.12 TriMul training route.

Breaking Changes

  • None.

Notes

  • Supported PyTorch versions: 2.11, 2.12 and 2.13. PyTorch 2.14 is not yet supported. A known PyTorch 2.14 training failure is fixed in this release (see Bug Fixes), but full 2.14 support comes in a future release.
  • The newly optimized TriMul torch kernels run only with the CUDA 13 wheels (cuequivariance-ops-torch-cu13). The CUDA 12 wheels do not contain the optimized TriMul kernels and instead use the 0.12 implementation.
  • The optimized TriMul kernels run only on these GPUs: H100 SXM (80 GB HBM3), H200, GH200 480GB, B200 and RTX PRO 6000 Blackwell (Server and Workstation Edition). All other GPUs use the 0.12 implementation. This includes H100 PCIe, H100 NVL, B300 / GB300, A100, L40(S) and Rubin.
  • Recommended usage is unchanged: torch.compile(model, mode="reduce-overhead"). Warm up for a few iterations first.
  • The optimized kernels were added only on shapes where they passed the accuracy release gate and were faster than the 0.12 implementation in our CI.

Optimized TriMul kernels

Terms used in this section. The names refer to the arguments of cuet.triangle_multiplicative_update:

Term Meaning Where it is in the call
B Batch size x.shape[0]
N Sequence length (number of tokens or residues) x.shape[1] and x.shape[2]
C Pair width: the channel size of the pair representation (c_z in AlphaFold-style models) x.shape[3]
D Hidden width: the channel size of the projected pair inside the update (c_hidden) p_in_weight.shape[0] // 2
Bias The four projection biases p_in_bias, g_in_bias, p_out_bias and g_out_bias "yes" = all four given; "no" = all four None

The tensor shapes that the optimized kernels expect:

Argument Shape
x (B, N, N, C), exactly one batch dimension
mask (B, N, N) or None
norm_in_weight, norm_in_bias (C,)
p_in_weight, g_in_weight (2D, C)
p_in_bias, g_in_bias (2D,)
norm_out_weight, norm_out_bias (D,)
p_out_weight (C, D)
g_out_weight (C, C)
p_out_bias, g_out_bias (C,)

All tensors must be contiguous and on the same GPU. A call that gives only some of the four projection biases, has more than one leading batch dimension, or has other shapes uses the 0.12 implementation. For C = D (128, 256, 512), the shapes are the same as in the function's docstring, where both widths are called D. For C = 384, D = 256, use the shapes in the table above.

Where the optimized kernels run

GPU Inference cells Training cells
H100 SXM (80 GB HBM3) 131 54
H200 131 54
GH200 480GB 131 54
B200 222 66
RTX PRO 6000 Blackwell (Server and Workstation Edition) 190 — (inference only)

A cell is one combination of pair width C, hidden width D, bias or no bias, batch size B and sequence length N. The optimized kernels run only on the exact cells listed under "Supported configurations". On all other cells, the call uses the 0.12 implementation. In inference, the incoming and outgoing directions are both supported on every listed cell.

How to use the optimized kernels

  1. Install the CUDA 13 wheels.
  2. Call cuet.triangle_multiplicative_update in BF16: the pair and all parameters must be BF16. Alternatively, pass FP32 inputs with precision="bfloat16". Mixed precision (autocast) keeps the 0.12 implementation in 0.13. See "Precision and mixed precision" below.
  3. Under torch.compile, each listed shape gets its own graph. For inference, pad or bucket your inputs to a small set of supported shapes. To use the optimized kernels on many shapes:
    • compile with dynamic=False;
    • set torch._dynamo.config.recompile_limit higher than the number of shapes.

With the default (automatic dynamic shapes), at most recompile_limit − 2 shapes per process use the optimized kernels (6 by default). The optimized kernels and the 0.12 implementation share the recompile budget of the frame. With many different sequence lengths, the frame can reach recompile_limit. On B200 and RTX PRO 6000 at C=384, D=256, this occurs at about the 9th new shape. PyTorch then runs that frame in eager mode for all further new shapes. cuEquivariance 0.12.0 stayed compiled in this case. A fixed set of shapes, or dynamic=False with a higher recompile_limit, prevents this.

Precision and mixed precision

The optimized kernels take BF16 inputs only. The input dtypes and the precision argument decide which implementation runs:

Your call Implementation in 0.13
Pair and all parameters BF16 (with or without BF16 autocast) Optimized
FP32 pair and FP32 parameters under BF16 autocast (for example torch.autocast("cuda", dtype=torch.bfloat16), Lightning bf16-mixed) 0.12
BF16 pair with FP32 parameters (with or without autocast) 0.12
FP32 inputs with precision="bfloat16" (with or without autocast) Optimized
FP32 without autocast, FP16 inputs, autocast to FP16, precision="ieee" or "mxfp8" 0.12

"Optimized" in this table applies only to the shapes listed under "Supported configurations". A call with any other shape uses the 0.12 implementation, whatever its dtypes and precision.

  • Under autocast, the LayerNorms run in FP32. The 0.12 implementation normalizes the FP32 pair with FP32 LayerNorm parameters. Only the GEMM operands become BF16. This is why autocast calls stay on 0.12 in 0.13: the optimized kernels need BF16 LayerNorm inputs, and the results would change.
  • precision="bfloat16" rounds all inputs to BF16 before the call. This includes the pair, the mask, the LayerNorm weights and biases, and the projection weights. The LayerNorms then read BF16 values. The rounding is differentiable, so gradients still reach FP32 master weights. 0.12 applies the same rounding for this argument, so the optimized kernels and 0.12 get the same inputs.
  • Accuracy: compared with autocast, precision="bfloat16" adds the rounding error of the pair at the LayerNorm input. If your model runs under autocast today, check its accuracy before you change to precision="bfloat16" to get the optimized kernels.
  • Numerics: the optimized kernels' outputs agree with the 0.12 implementation within its numerical tolerance, but they are not bitwise identical.

Known issues

  • Small-sequence training on B200. A few B200 training cells at short sequence lengths (N=128) can run slightly slower than the 0.12 implementation. We are working to resolve this in a future release.

  • Exported models use the 0.12 implementation. torch.export (strict and non-strict), ONNX export (and so TensorRT), and traces with symbolic sizes outside torch.compile (make_fx, FakeTensorMode) never record an optimized kernel.

  • Eager-mode routing cost. In eager mode, calls that do not take an optimized kernel pay a small routing check on each call. The cost is most visible at small shapes, and most of all on RTX PRO 6000. Under torch.compile, the check costs nothing.

  • Second backward and double backward. In training, a second backward through one forward (retain_graph=True):

    • is exact in eager mode, but needs about 1.9× the memory of the first backward;
    • gives NaN gradients when it is replayed from a CUDA graph. This is by design: the result is visibly wrong, never silently wrong. The first backward of every replay is exact.

    Double backward (create_graph=True) raises an error.

  • Exclusive GPU. Some optimized kernels use software grid barriers and expect exclusive use of the GPU. Under MPS sharing without SM partitions, they can hang.

  • FP32 TriMul on GPUs with about 100 KB shared memory per block (L40S and other Ada GPUs, RTX PRO 6000 and other sm_120 GPUs): with Triton newer than 3.6 (PyTorch 2.14 ships 3.8), the FP32 0.12 Triton kernel at C=128 fails with OutOfResources. This is not an issue of the optimized kernels; 0.12.0 has the same problem. BF16 calls are not affected.

  • Per-rank compile limit under DDP. Under DDP or multi-node training, each rank applies the recompile limit separately. Ranks that see shapes in a different order can run the optimized kernels on different shapes. Results are not affected, but a small speed difference between ranks is possible.

Supported configurations

The optimized kernels run on these exact cells: (C, D, bias) and, for each batch size B, the listed sequence lengths N. A batch size or sequence length that is not listed for that row uses the 0.12 implementation. The list is the same for the incoming and outgoing directions.

H100 SXM, H200, GH200 480GB, inference (131 cells)
C D Bias Batch size B: sequence lengths N
128 128 no B=1: 256, 384, 512, 640, 768, 1024, 1280, 1536, 2048; B=4: 384, 512, 768
128 128 yes B=1: 256, 384, 512, 640, 768, 1024, 1280
256 256 no B=1: 128, 256, 384, 512, 768, 1024, 1536, 2048; B=4: 512
256 256 yes B=1: 128, 256, 384, 512, 768, 1024, 2048; B=2: 128, 256, 384, 512, 768, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=8: 128, 256, 384, 512, 768; B=16: 128, 256, 384, 512
384 256 no B=1: 128, 160, 192, 224, 256, 288, 320, 352, 384, 448, 512, 640, 768, 896, 1024; B=2: 128, 256, 384, 512, 768; B=4: 128, 256, 384, 512, 768; B=5: 256, 320, 384, 448, 512, 560, 608; B=8: 128, 256, 384, 512; B=16: 128, 256, 384, 512
384 256 yes B=1: 128, 256, 384, 512, 768; B=2: 128, 256, 384, 512, 768; B=4: 128, 256, 384, 512, 768; B=8: 128, 256, 384, 512; B=16: 128, 256, 384, 512
512 512 no B=1: 128, 256, 384, 512, 768, 1024
512 512 yes B=1: 128, 256, 384, 512, 768, 1024
H100 SXM, H200, GH200 480GB, training (54 cells)
C D Bias Batch size B: sequence lengths N
128 128 no B=1: 256, 384, 512, 640, 768; B=4: 384, 512, 768
128 128 yes B=1: 256, 384, 512, 640, 768
256 256 no B=1: 256, 384, 512; B=4: 512
256 256 yes B=1: 256, 384, 512; B=2: 128, 256, 384, 512; B=4: 128, 256, 384, 512
384 256 no B=1: 256, 384; B=2: 256, 384; B=4: 256, 384; B=8: 256, 384; B=16: 256, 384
384 256 yes B=1: 256, 384; B=2: 256, 384; B=4: 256, 384; B=8: 256, 384; B=16: 256, 384
512 512 no B=1: 128, 256, 384
512 512 yes B=1: 128, 256, 384
B200, inference (222 cells)
C D Bias Batch size B: sequence lengths N
128 128 no B=1: 64, 98, 128, 200, 256, 384, 512, 640, 768, 1024, 1280, 1536, 2048; B=2: 98; B=4: 384, 512, 768; B=8: 512; B=16: 128
128 128 yes B=1: 64, 98, 128, 200, 256, 384, 512, 640, 768, 1024, 1280, 1536, 2048; B=2: 98; B=16: 128
256 256 no B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024, 1536, 2048; B=2: 98; B=4: 512; B=8: 512; B=16: 128
256 256 yes B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024, 2048; B=2: 98, 128, 256, 384, 512, 768, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=8: 128, 256, 384, 512, 768; B=16: 128, 256, 384, 512
384 256 no B=1: 32, 48, 64, 80, 96, 98, 112, 128, 130, 160, 192, 200, 224, 256, 257, 288, 300, 320, 352, 384, 448, 500, 512, 640, 768, 896, 1000, 1024; B=2: 64, 98, 128, 200, 256, 384, 512, 768, 1000, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=5: 256, 320, 384, 448, 512, 560, 608; B=8: 128, 256, 257, 300, 384, 500, 512, 768; B=16: 32, 48, 64, 80, 96, 98, 112, 128, 130, 200, 256, 384, 512
384 256 yes B=1: 32, 48, 64, 80, 96, 98, 112, 128, 130, 200, 256, 257, 300, 384, 385, 500, 512, 768, 1000, 1024; B=2: 64, 98, 128, 200, 256, 384, 385, 512, 768, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=8: 128, 256, 384, 512, 768; B=16: 128, 256, 384, 512
512 512 no B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024; B=2: 98; B=8: 512; B=16: 128
512 512 yes B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024; B=2: 98; B=8: 512; B=16: 128
B200, training (66 cells)
C D Bias Batch size B: sequence lengths N
128 128 no B=1: 128, 256, 384, 512, 640, 768, 1024; B=4: 384, 512, 768
128 128 yes B=1: 128, 256, 384, 512, 640, 768, 1024
256 256 no B=1: 128, 256, 384, 512, 768; B=4: 512
256 256 yes B=1: 128, 256, 384, 512, 768; B=2: 128, 256, 384, 512, 768; B=4: 128, 256, 384, 512, 768
384 256 no B=1: 256, 384; B=2: 256, 384; B=4: 256, 384; B=8: 256, 384; B=16: 256, 384
384 256 yes B=1: 256, 384; B=2: 256, 384; B=4: 256, 384; B=8: 256, 384; B=16: 256, 384
512 512 no B=1: 128, 256, 384, 512
512 512 yes B=1: 128, 256, 384, 512
RTX PRO 6000 Blackwell (Server and Workstation Edition), inference (190 cells)
C D Bias Batch size B: sequence lengths N
128 128 no B=1: 64, 98, 128, 200, 256, 384, 512, 640, 768, 1024, 1280, 1536, 2048; B=2: 98; B=4: 384, 512, 768; B=8: 512; B=16: 128
128 128 yes B=1: 64, 98, 128, 200, 256, 384, 512, 640, 768, 1024, 1280, 1536; B=2: 98; B=8: 512; B=16: 128
256 256 no B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024, 1536, 2048; B=2: 98; B=4: 512; B=8: 512; B=16: 128
256 256 yes B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024, 2048; B=2: 98, 128, 256, 384, 512, 768, 1024; B=4: 128, 256, 384, 512, 768, 1024; B=8: 128, 256, 384, 512, 768; B=16: 128, 256, 384, 512
384 256 no B=1: 32, 48, 64, 80, 96, 98, 112, 128, 130, 160, 192, 200, 224, 256, 257, 288, 300, 320, 352, 384, 385, 448, 500, 512, 640, 768, 896, 1024; B=2: 64, 98, 128, 200, 256, 384, 385, 512, 768; B=4: 128, 256, 384, 512, 768; B=5: 256, 320, 384, 448, 512, 560, 608; B=8: 128, 256, 257, 300, 384, 385, 500, 512; B=16: 32, 48, 64, 80, 96, 98, 112, 128, 130, 200, 256
384 256 yes B=1: 32, 48, 64, 80, 96, 98, 112, 128, 130, 200, 256, 257, 300, 384, 385, 500, 512; B=2: 64, 98, 200, 385; B=8: 512; B=16: 128
512 512 no B=1: 64, 98, 128, 200, 256, 384, 512, 768, 1024; B=2: 98; B=8: 512; B=16: 128
512 512 yes B=1: 128, 256, 384, 512, 768, 1024