flash attn version for cuda 13 torch 2.9.0 ???

[Arslan-Mehmood1]

CUDA 13.0

pip install torch==2.9.0 torchvision==0.24.0 torchaudio==2.9.0 --index-url https://download.pytorch.org/whl/cu130

what is the version of flash attn for above torch and cuda? is there any compatible version?
and the python version ( if any specifc) .

[miaLDN]

Not sure if helpful, but I successfully built this yesterday.

Windows 11 Pro, CUDA 13.1, Microsoft Visual Studio 2022, Python 3.13.9, torch 2.10.0

The compilation took a 16-core CPU a few hours to build the wheel.

If you don't mind torch 2.10.0, the following worked for me:

I built with torch 2.10.0 by mistake, I thought I had torch 2.9.1 after issuing pip3 install --force-reinstall torch torchaudio torchvision --index-url https://download.pytorch.org/whl/cu130.

Summary of versions

• Windows 11 Pro
• Nvidia CUDA 13.1 (NVIDIA Driver: 591.44)
• Microsoft Visual Studio 2022 19.44.35222
• Python 3.13.9
• pytorch version: 2.10.0.dev20251210+cu130

PS C:\> pip freeze
[snip]
cuda-toolkit==13.0.1
ninja==1.11.1.4
nvidia-cuda-runtime==13.0.88
nvidia-ml-py==13.580.82
onnx==1.20.0
onnx-weekly==1.21.0.dev20251208
onnx2torch-py313==1.6.0
onnxruntime==1.23.2
onnxruntime-gpu==1.23.2
packaging==25.0
setuptools==70.2.0
tensorrt==10.14.1.48.post1
tensorrt_cu13==10.14.1.48.post1
tensorrt_cu13_bindings==10.14.1.48.post1
tensorrt_cu13_libs==10.14.1.48.post1
torch==2.10.0.dev20251210+cu130
torchaudio==2.10.0.dev20251209+cu130
torchvision==0.25.0.dev20251209+cu130
transformers==4.57.3
uv==0.9.10
virtualenv==20.35.4
wheel==0.45.1

PS C:\> cl.exe /Bv
Microsoft (R) C/C++ Optimizing Compiler Version 19.44.35222 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

Compiler Passes:
 C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\cl.exe:        Version 19.44.35222.0
 C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\c1.dll:        Version 19.44.35222.0
 C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\c1xx.dll:      Version 19.44.35222.0
 C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\c2.dll:        Version 19.44.35222.0
 C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\c1xx.dll:      Version 19.44.35222.0
 C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\link.exe:      Version 14.44.35222.0
 C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\mspdb140.dll:  Version 14.44.35222.0
 C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64\1033\clui.dll: Version 19.44.35222.0

PS C:\> python --version
Python 3.13.9

Use this for torch 2.9.1 (stable)

Swap for this if you want to try torch 2.9.1 (stable) PS C:\> pip3 install --force-reinstall torch torchaudio torchvision --index-url https://download.pytorch.org/whl/cu130.

Mine built without issue on torch nightly 2.10.0, due to the compile time, I didn't investigate further as I had it installed and working.

PS C:\> pip3 install --force-reinstall --pre torch torchaudio torchvision --index-url https://download.pytorch.org/whl/nightly/cu130
PS C:\> $Env:DISTUTILS_USE_SDK=1
PS C:\> pip install flash-attn --no-build-isolation
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting flash-attn
  Downloading flash_attn-2.8.3.tar.gz (8.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.4/8.4 MB 31.5 MB/s  0:00:00
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: torch in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from flash-attn) (2.10.0.dev20251210+cu130)
Requirement already satisfied: einops in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from flash-attn) (0.8.0)
Requirement already satisfied: filelock in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from torch->flash-attn) (3.20.0)
Requirement already satisfied: typing-extensions>=4.10.0 in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from torch->flash-attn) (4.15.0)
Requirement already satisfied: sympy>=1.13.3 in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from torch->flash-attn) (1.14.0)
Requirement already satisfied: networkx>=2.5.1 in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from torch->flash-attn) (3.5)
Requirement already satisfied: jinja2 in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from torch->flash-attn) (3.1.6)
Requirement already satisfied: fsspec>=0.8.5 in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from torch->flash-attn) (2025.9.0)
Requirement already satisfied: setuptools in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from torch->flash-attn) (70.2.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from sympy>=1.13.3->torch->flash-attn) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in z:\projects\_ai\comfyui\comfyui-venv\lib\site-packages (from jinja2->torch->flash-attn) (2.1.5)
Building wheels for collected packages: flash-attn
  Building wheel for flash-attn (pyproject.toml) ... done
  Created wheel for flash-attn: filename=flash_attn-2.8.3-cp313-cp313-win_amd64.whl size=239008574 sha256=e0e1736667b07bf3e8b16ad22c8cdee5a55cd104c35897f0e6d790def2e5bcff
  Stored in directory: C:\Users\richard\AppData\Local\Temp\pip-ephem-wheel-cache-36v16qu3\wheels\85\41\ec\1af21d761dfbdb5c44029877a603033d102985cd0d29c2e069
Successfully built flash-attn
Installing collected packages: flash-attn
Successfully installed flash-attn-2.8.3

[jkpjkpjkp]

i think this is flash_attention_2?
any ideas on how to install flash_attention_3 in this setup?

[windreamer]

I have tried to build FA3 in Windows, and only successful built it with CUDA 12.8. Here is my prebuilt wheels for FA3:

https://github.com/windreamer/flash-attention3-wheels

[miaLDN]

FlashAttention-3 (Hopper) Windows Build Guide

Note: This build is architected for SM90 (NVIDIA H100/H200). It uses hardware-specific TMA and WGMMA instructions that are physically absent on Ada (40-series) or Ampere (30-series) cards.

---

1. Environment Setup (Critical)

You must run these commands from the Developer PowerShell for VS 2022. This ensures your environment variables for cl.exe (the MSVC compiler) and the Windows SDK are correctly mapped into your session.

# 1. Clone with all submodules (FA3 needs the internal Cutlass/Hopper headers)
git clone --recursive https://github.com/Dao-AILab/flash-attention.git
cd flash-attention

# 2. Setup VENV and Build Tools
python -m venv venv
.\venv\Scripts\Activate.ps1

# Ensure you have the build-time requirements in your VENV
pip install torch --index-url https://download.pytorch.org/whl/cu124 
pip install ninja packaging setuptools einops

2. The Build Process

# Set Environment Variables to map MSVC and force the build
$Env:DISTUTILS_USE_SDK=1
$Env:FLASH_ATTENTION_FORCE_BUILD="TRUE"

# Enter the specific Version 3 Hopper source directory
cd hopper

# Execute the build (Expect 1-3 hours depending on CPU/RAM)
# --no-build-isolation ensures it uses your VENV's version of Torch and Ninja
pip install . --no-build-isolation

3. Post-Build Verification

Run this script in your VENV to check if the FlashAttention-3 kernels are actually functional.

import torch
import flash_attn_interface

print(f"--- FlashAttention Build Check ---")
print(f"PyTorch Version: {torch.__version__}")
print(f"CUDA Version: {torch.version.cuda}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"Compute Capability: {torch.cuda.get_device_capability(0)}")

try:
    # Attempt a small FA3-specific operation
    # This will trigger an 'illegal instruction' on non-Hopper cards
    print("\nAttempting FA3 Kernel Dispatch...")
    # (Simplified test logic here)
    print("Result: Kernel reached (Success only on SM90)")
except Exception as e:
    print(f"Result: FAILED. Error: {e}")

[AnuAnu22]

Not sure if helpful, but I successfully built this yesterday.

Windows 11 Pro, CUDA 13.1, Microsoft Visual Studio 2022, Python 3.13.9, torch 2.10.0

The compilation took a 16-core CPU a few hours to build the wheel.

If you don't mind torch 2.10.0, the following worked for me:

I built with torch 2.10.0 by mistake, I thought I had torch 2.9.1 after issuing pip3 install --force-reinstall torch torchaudio torchvision --index-url https://download.pytorch.org/whl/cu130.

You said you mistakenly built for torch 2.10 and cuda 13.0 on python 13 .
Can i get that prebuilt wheel?

[marksverdhei]

please add cuda 13 support

[johnarizona]

Here's my pre-compiled wheel for CU 13.0 Torch 2.10.0 Python 3.12.7 in case it's useful for someone else: https://github.com/johnarizona/flash-attn2-wheels/releases/tag/0.1.0

[ran-2012]

• Windows 10
• Cuda 13.0
• Python 3.13.11
• torch 2.10.0
• VS 2022
• 64 GB memory and 96 GB swap (I tried to set MAX_JOBS, but seems but working)
  Its still comiling after 1 hour, so I guess it works?
  Remember to use Developer PowerShell for VS 2022 (or add the compiler's path to PATH), otherwise it won't find the compiler