Support for blackwell architecture

[EnricoBeltramo]

I tried to use flash attention but I have follow error:
FA version 3 is not supported due to FA3 is only supported on devices with compute capability >= 8 excluding 8.6 and 8.9 and Blackwell archs (>=10)
I have a blackwell, (RTX pro 4500 blackwell), is there any expected timeline for when FA3 kernels will be available for compute capability 10.x?

[johnnynunez]

I tried to use flash attention but I have follow error: FA version 3 is not supported due to FA3 is only supported on devices with compute capability >= 8 excluding 8.6 and 8.9 and Blackwell archs (>=10) I have a blackwell, (RTX pro 4500 blackwell), is there any expected timeline for when FA3 kernels will be available for compute capability 10.x?

you have to wait for flash attention 4 (around 3 weeks)

[tridao]

RTX pro 4500 is sm_120, which is quite different from Blackwell datacenter cards like B200 (sm_100). The Ampere code (FA2) should run fine on sm_120.

https://developer.nvidia.com/cuda-gpus

[johnnynunez]

RTX pro 4500 is sm_120, which is quite different from Blackwell datacenter cards like B200 (sm_100). The Ampere code (FA2) should run fine on sm_120.

https://developer.nvidia.com/cuda-gpus

i think that 11.0 Thor can shared some kernels with sm_100:
From CUDA 13.0, the Blackwell SM101 for Thor GPUs is renamed to SM110.
- For CUDA toolkit version < 13.0, SM101 is still used for Thor GPUs.
- For CUDA toolkit version >= 13.0, SM110 is used for Thor GPUs and SM101 is no longer valid.

[triple-mu]

RTX pro 4500 is sm_120, which is quite different from Blackwell datacenter cards like B200 (sm_100). The Ampere code (FA2) should run fine on sm_120.
https://developer.nvidia.com/cuda-gpus

i think that 11.0 Thor can shared some kernels with sm_100: From CUDA 13.0, the Blackwell SM101 for Thor GPUs is renamed to SM110. - For CUDA toolkit version < 13.0, SM101 is still used for Thor GPUs. - For CUDA toolkit version >= 13.0, SM110 is used for Thor GPUs and SM101 is no longer valid.

Very urgent, I would greatly appreciate it if you have a better solution!

[mcr-ksh]

+1

[Dimitri-Ingenieur]

I tried to use flash attention but I have follow error: FA version 3 is not supported due to FA3 is only supported on devices with compute capability >= 8 excluding 8.6 and 8.9 and Blackwell archs (>=10) I have a blackwell, (RTX pro 4500 blackwell), is there any expected timeline for when FA3 kernels will be available for compute capability 10.x?

you have to wait for flash attention 4 (around 3 weeks)

I get following error with flash-attn-4 (b15): python -c "
import torch
from flash_attn.cute import flash_attn_func
q=torch.randn(1,512,8,128,device='cuda',dtype=torch.bfloat16)
out = flash_attn_func(q,q,q)
o = out[0] if isinstance(out, tuple) else out
print('FA4 OK | Typ:', type(out).name, '| Output-Shape:', tuple(o.shape))
"
Traceback (most recent call last):
File "", line 5, in
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/flash_attn/cute/interface.py", line 2151, in flash_attn_func
return FlashAttnFunc.apply(
^^^^^^^^^^^^^^^^^^^^
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/torch/autograd/function.py", line 596, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/flash_attn/cute/interface.py", line 1943, in forward
out, lse = _flash_attn_fwd(
^^^^^^^^^^^^^^^^
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/flash_attn/cute/interface.py", line 981, in _flash_attn_fwd
_flash_attn_fwd.compile_cache[compile_key] = cute.compile(
^^^^^^^^^^^^^
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/flash_attn/cute/flash_fwd.py", line 737, in call
).launch(
^^^^^^^^^^^
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/flash_attn/cute/flash_fwd.py", line 1067, in kernel
self.epilogue(
^^^^^^^^^^^^^^^^^^
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/flash_attn/cute/flash_fwd.py", line 405, in epilogue
store_O, _, _ = copy_utils.tma_get_copy_fn(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/quack/copy_utils.py", line 788, in tma_get_copy_fn
s, g = cpasync.tma_partition(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/dima/miniconda3/envs/lyra2/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/cute/nvgpu/cpasync/helpers.py", line 585, in tma_partition
atom._trait.value,
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '_trait'