flash-attn 3 - Windows ROCm pre-built wheels for AMD GPUs (aiter Triton backend)

Triton-backend only build of flash-attn 3 for Windows ROCm users who don't want to build from source.

Install

pip install https://github.com/0xDELUXA/flash-attention/releases/download/v3.0.0_win-rocm/flash_attn_3-3.0.0-py3-none-any.whl
pip install https://github.com/0xDELUXA/flash-attention/releases/download/v2.8.4_win-rocm/amd_aiter-0.0.0-py3-none-win_amd64.whl

Example

import torch
from flash_attn_interface import flash_attn_func

q = torch.randn(1, 1, 4, 16, device="cuda")
out = flash_attn_func(q, q, q)
print(out.shape)

Example output

UserWarning: flash_attn_3._C (which has ROCm/HIP kernels) not found, falling back to Triton implementation
[aiter] Windows: CK and HIP ops are not available. Triton ops only.
torch.Size([1, 1, 4, 16])

flash-attn 2.8.4 - Windows ROCm pre-built wheels for AMD GPUs (aiter Triton backend)

Triton-backend only build of flash-attn 2.8.4 for Windows ROCm users who don't want to build from source.

Tested on

• GPUs: gfx1030, gfx1150 and gfx1200
• Python: 3.11, 3.12 and 3.13
• ROCm: 7.12.0a and 7.13.0a
• PyTorch: 2.9.1, 2.10.0, and 2.12.0a0

Expected to work on

• Other AMD GPUs which are supported by aiter
• Other Python versions
• Other TheRock ROCm/PyTorch builds

Requirements

• Windows 10/11
• Python
• ROCm & PyTorch from TheRock

Install

pip install https://github.com/0xDELUXA/flash-attention/releases/download/v2.8.4_win-rocm/flash_attn-2.8.4-py3-none-win_amd64.whl
pip install https://github.com/0xDELUXA/flash-attention/releases/download/v2.8.4_win-rocm/amd_aiter-0.0.0-py3-none-win_amd64.whl

Notes

• CK backend is not yet available on Windows - Triton only
• NVIDIA GPUs are not supported by these wheels