Support for NVIDIA Blackwell (sm_120) architectures — any planned release date?

[FrankZhangRp]

Hello,

Thank you for maintaining flash-attention and for offering cutting-edge support for new GPU architectures.

I am using an NVIDIA RTX PRO 6000 Blackwell Server Edition (sm_120) machine with CUDA 12.8 and PyTorch 2.8.0+cu128.
I encountered the following error when using flash-attention’s forward/backward kernels:

CUDA error (/__w/xformers/xformers/third_party/flash-attention/hopper/flash_fwd_launch_template.h:188): invalid argument

I believe this is because flash-attention currently supports Ampere, Ada or Hopper architectures (per README) but does not yet support Blackwell.

Could you please share the roadmap for support of Blackwell (sm_120) architectures? Specifically:

1. Is there a release or patch version of flash-attention planned that fully supports sm_120 / Blackwell?
2. If so, is there a tentative date or milestone for that release?
3. Are there any recommended workarounds in the meantime (e.g., disabling flash kernels, using fallback kernels) to enable stable training on Blackwell GPUs?

Thank you very much for your time and your work.
Best regards

[weireweire]

I think fa2 should support blackwell, fa3 is for hopper. fa2 don't support fp8 qkv and performance is not good for blackwell though. I wander plan for blackwell too.

[tdev36]

I NEED flash attention for blackwell nvidia 6000 pro when will the flash attention for it be available my cuda version is 13 when is their any release date or something i have to work on video generation projects

[johnnynunez]

I NEED flash attention for blackwell nvidia 6000 pro when will the flash attention for it be available my cuda version is 13 when is their any release date or something i have to work on video generation projects

it is not will be released soon.

[fernandaspets]

I also have rtx 6000s. is this related to the same problem? ValueError: No valid attention backend found for cuda with head_size: 576, dtype: torch.bfloat16, kv_cache_dtype: auto, block_size: 64, use_mla: True, has_sink: False, use_sparse: True. Reasons: {FLASH_ATTN_MLA: [sparse not supported, compute capability not supported, FlashAttention MLA not supported on this device], FLASHMLA: [sparse not supported, compute capability not supported, vllm._flashmla_C is not available, likely was not compiled due to insufficient nvcc version or a supported arch was not in the list of target arches to compile for.], FLASHINFER_MLA: [sparse not supported, compute capability not supported], TRITON_MLA: [sparse not supported], FLASHMLA_SPARSE: [compute capability not supported]}.

[tdev36]

is flash attention or sage attention for blackwell 6000 been released or not

[Berengames]

Any updates on flash attention fro blackwell 6000?

[johnnynunez]

Any updates on flash attention fro blackwell 6000?

#2222 a lot of improvement here anyways, i tried to explain the best i can

[tdev36]

so when are expecting fa3 for blackwell

[johnnynunez]

so when are expecting fa3 for blackwell

I think that you are confused. Flash Attention is Hardware Aware Programming, so it means that the algorithms fits exclusively for the architecture that you are programming.

FA: https://arxiv.org/abs/2205.14135

FA2 agnostic to arch. https://arxiv.org/abs/2307.08691
FA3 hopper and sm80 in some cases https://arxiv.org/abs/2407.08608
FA4 B200/B300 (FA4 paper is coming)

C++ code it will not maintained i think so, due hard compile in devices, and now move everything to cute dsl, where you can "compile" x50 faster.

git clone --recursive https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/flash_attn/cute
uv build --wheel . -v --no-build-isolation --out-dir ./flash-attention/wheels
uv pip install /opt/flash-attention/wheels/flash_attn_cute*.whl

That includes FA2, FA3 and new FA4 in cute dsl: https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/cute/flash_fwd.py

header of the file

# Copyright (c) 2025, Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao.
# A reimplementation of
# https://github.com/Dao-AILab/flash-attention/blob/main/hopper/flash_fwd_kernel_sm80.h
# and https://github.com/Dao-AILab/flash-attention/blob/main/hopper/flash_fwd_kernel_sm90.h
# from Cutlass C++ to Cute-DSL.
# Built on Cute-DSL example: https://github.com/NVIDIA/cutlass/blob/main/examples/python/CuTeDSL/ampere/flash_attention_v2.py

In my case, i was trying to support like a hybrid Flash attention 2,3,4 to sm12x, due that blackwell geforce != blackwell enterprise. I explained it in my PR #2222 . Now i am waiting for cutlass team to the release of cutlass v4.4.0-v4.5.0 stable release, and upgrade Pytorch to CUDA 13.2 with better support on blackwell geforce devices in cutlass cute DSL

For me the new cute DSL is easier to develop, because i didn't touch C++ from long time, but i am not very expert in hardware aware programming, i do it because i like and i would like to learn more in depth. One of the best ways to know the status and development is GPU Mode(https://github.com/gpu-mode) and https://gau-nernst.github.io/ where he implemented FA in C++ for blackwell https://github.com/gau-nernst/learn-cuda

[geraldstanje1]

hi any updated to flash attention for blackwell 6000 pro?

[johnnynunez]

hi any updated to flash attention for blackwell 6000 pro?

#2268