Skip to content

v1.7 / flash_attn 2.9.2.post1

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 12 Aug 03:03
· 14 commits to main since this release
v1.7
6513475

Fork tag: v1.7
Package: flash_attn 2.9.2.post1
Date: 2026-08-12
Tag commit: 65134752eca23e52bf902946a9bb461931c320fd

This is a patch on the 2.9.2 split_align line (fork v1.6). It does not add new public APIs. It changes the FA2 forward softmax rescale skip (A-1) to the FA4 16-bit threshold with exact max-lagging, and it allows the compiler to schedule the A-2 packed FMA (Delta 4).

Windows FA2 remains the production path for this fork (PyTorch 2.10+ / CUDA 13+ / MSVC). The Blackwell / legacy wheel split from v1.6 is unchanged.

1. Why 2.9.2.post1 exists

Layer Meaning
v1.6 / 2.9.2 Official split_align merge: 24 flash_fwd_split_align_*.cu units; bat2 Blackwell (100;120;121) vs bat3 Ampere/Hopper (80;89;90) to stay under MSVC LNK1189 (2 GB linker).
v1.7 / 2.9.2.post1 Same split_align + wheel split. A-1: float RescaleThreshold (call sites 8.0f) instead of bool / -0.01f skip; on skip, restore row_max to the previous max so O, row_sum, and P share one base (exact output and LSE). A-2: fma_f32x2 uses asm( (not asm volatile() so fma.rn.f32x2 may be scheduled.

1.1 Kernel deltas in this release

Delta Change Files
Delta 1 bool Use_rescale_threshold → float RescaleThreshold=0.0f; skip when scaled_diff >= -RescaleThreshold; call sites pass 8.0f csrc/flash_attn/src/softmax.h, csrc/flash_attn/src/flash_fwd_kernel.h
Delta 4 asm volatile( → asm( in fma_f32x2 csrc/flash_attn/src/utils.h
Delta 5 Comment: documents RescaleThreshold scope csrc/flash_attn/src/softmax.h
Delta 6 FA4-style max lagging: on skip, row_max(mi) = scores_max_prev(mi) csrc/flash_attn/src/softmax.h

1.2 A-1 — rescale threshold and max lagging

Aspect 2.9.0 / 2.9.1 (original A-1) 2.9.2.post1 (this release)
Template bool Use_rescale_threshold float RescaleThreshold=0.0f
Threshold kSoftmaxRescaleSkipThreshold = -0.01f 8.0f at Is_first=false call sites (FA4 16-bit default)
Skip condition scaled_diff >= -0.01 scaled_diff >= -8.0
Skip semantics Keep the new max; approximate (~0.7% error at 0.01) Revert row_max to the previous max; running O, row_sum, and P stay on one base (exact)

Why 8.0f: FA4 on Blackwell uses rescale_threshold = 8.0 when Q is 16-bit (q_dtype.width == 16), else 0.0.

Why Delta 6 is required: Skipping rescale but keeping the new max is bounded (~0.7%) at threshold 0.01. At 8.0, the same pattern can overweight stale block contributions by up to ~256×. Max lagging restores row_max(mi) = scores_max_prev(mi) on skip so output and LSE stay exact.

P saturation: With max lagging, P can reach exp2(8) = 256 before fp16/bf16 conversion. fp16 max is 65504; bf16 max is 3.39e38. No overflow at this threshold.

1.3 A-2 — packed FMA scheduling (Delta 4)

On sm_100+, scale_apply_exp2 uses fma_f32x2 (fma.rn.f32x2). Removing volatile lets the compiler schedule that instruction.

On sm_120, ptxas may still lower fma.rn.f32x2 to scalar FFMA (see §4.5). Accuracy and determinism tests still pass.

2. What did not change

  • Public APIs: flash_attn_func, flash_attn_varlen_func, flash_attn_with_kvcache, window, ALiBi, softcap.
  • v1.6 split_align (24 compilation units) and the bat2 / bat3 arch split.
  • FA3 (hopper/) and FA4 (flash_attn/cute/) Windows runtime: out of scope for this wheel line (same as v1.6).
  • No new FP8 path (A-3). No extra public kwargs.

3. Version and Windows wheels

Package version is 2.9.2.post1 (import flash_attn / pip show flash-attn). flash_attn/__init__.py reads installed metadata and falls back to "2.9.2.post1". setup2.py sets public_version = "2.9.2.post1".

Script Archs Wheel suffix
WindowsWhlBuilder_cuda_2.bat (setup2.py) 100;120;121 .blackwell
WindowsWhlBuilder_cuda_3.bat 80;89;90 .legacy

Validated wheel (this release):

flash_attn-2.9.2.post1+cu132torch2.13.0cxx11abiTRUE.blackwell-cp314-cp314-win_amd64.whl
PYD size 1,313,640,960 bytes (1.3 GB).

4. Validation (measured)

Hardware / software (one environment):

Item Value
GPU NVIDIA GeForce RTX 5060 Ti (sm_120, capability 12.0)
CUDA (build) 13.2
PyTorch 2.13.0+cu132
Python 3.14.6
OS Windows 11 (10.0.26200)
Installed package flash-attn 2.9.2.post1

Reference for accuracy: PyTorch scaled_dot_product_attention in fp32.
Gates: fp16 rel_Linf <= 2e-2, bf16 rel_Linf <= 1e-2, cosine >= 0.9995.

4.1 Seven public API paths (fp16, B=2, S=512, H=8, D=64)

# Test NaN Inf Verdict
1 flash_attn_func (base, non-causal) False False PASS
2 flash_attn_func (causal) False False PASS
3 flash_attn_varlen_func False False PASS
4 flash_attn_with_kvcache False False PASS
5 flash_attn_func window_size=(32,32) False False PASS
6 flash_attn_func alibi_slopes False False PASS
7 flash_attn_func softcap=30.0 False False PASS

flash_attn_triton: not tested (Triton not installed in that venv).
flash_attn.cute: not tested (Windows FA4 runtime out of scope).

4.2 Accuracy — 8-case sweep (S=512–2048, D=64)

dtype S D causal rel_Linf cosine Verdict
fp16 512 64 True 2.95e-04 0.99999994 PASS
fp16 1024 64 True 2.75e-04 1.00000000 PASS
fp16 2048 64 True 5.81e-04 1.00000000 PASS
bf16 512 64 True 2.37e-03 0.99999809 PASS
bf16 1024 64 True 2.48e-03 0.99999815 PASS
bf16 2048 64 True 1.92e-03 0.99999815 PASS
fp16 512 64 False 7.89e-04 0.99999994 PASS
bf16 512 64 False 6.54e-03 0.99999774 PASS

4.3 Accuracy — 10-shape sweep (D=128, D=256, S=8192, B=1)

dtype S D B causal max_abs rel_Linf cosine Verdict
fp16 1024 64 2 True 1.22e-04 2.45e-04 0.99999994 PASS
fp16 1024 64 2 False 7.63e-06 6.23e-04 0.99999988 PASS
fp16 1024 128 2 True 1.22e-04 3.42e-04 0.99999988 PASS
fp16 1024 128 2 False 7.63e-06 7.49e-04 0.99999994 PASS
bf16 4096 64 2 True 9.77e-04 2.58e-03 0.99999809 PASS
bf16 4096 64 2 False 3.05e-05 4.41e-03 0.99999768 PASS
bf16 4096 128 2 True 9.77e-04 3.36e-03 0.99999815 PASS
bf16 4096 128 2 False 3.05e-05 5.88e-03 0.99999785 PASS
fp16 2048 256 1 False 3.81e-06 7.94e-04 1.00000000 PASS
fp16 8192 64 2 True 1.22e-04 3.02e-04 0.99999994 PASS

Combined 18 cases: fp16 12/12, bf16 6/6.
Worst fp16 rel_Linf = 7.94e-04. Worst bf16 rel_Linf = 6.54e-03. Both inside gates.

4.4 Backward

Finiteness (fp16/bf16, S=2048, causal): dQ, dK, dV all finite.

Gradient vs fp32 (fp16, S=1024, causal):

Gradient max_abs_diff vs fp32
dQ 2.10e-05
dK 1.86e-05
dV 1.47e-03

dV is larger because V gradients accumulate softmax-weighted values. All finite; training path is usable on this config.

Varlen backward (fp16): dQ/dK/dV finite (no NaN/Inf). Ranges: dQ [-1.06, 1.44], dK [-1.82, 2.47], dV [0.00, 1.88].

4.5 Latency (fp16 causal, 50 warmup + 50 timing)

S D ms/iter
1024 64 0.079
1024 128 0.151
4096 64 0.839
4096 128 1.725
8192 64 3.162
8192 128 6.504

Scales as expected (quadratic in S, linear in D). No catastrophic regression vs the 2.9.0 sm_120 baseline on this machine.

4.6 Edge / feature probes

Test Result
return_attn_probs=True (LSE) shape [2, 8, 512], finite, range [-0.0244, 6.2399]
Determinism (two identical calls) max abs diff 0.00e+00, bit-identical (Delta 4 did not add nondeterminism)
S=1 finite
S=2 vs fp32 max_abs_diff 1.22e-04
H=1 vs fp32 max_abs_diff 1.22e-04
D=256 bf16 rel_Linf 3.73e-03, cosine 0.99999821, PASS
softcap + causal finite
window_size=(0,0) vs V max_abs_diff 0.00e+00 (output == V)
window_size=(128,128) finite

4.7 SASS on sm_120 (96 cubins)

Pattern Count
FFMA.X2 not found
FFMA (scalar) 5,760
FMUL.FTZ 12,854
MUFU.EX2 6,862

fma.rn.f32x2 PTX was lowered to scalar FFMA on this sm_120 / CUDA 13.2 build. Correctness is covered by the tables above, not by FFMA.X2 presence.

5. Not in this release

Item Status
Triton (flash_attn_triton) Not tested (module not installed)
CuTe / FA4 (flash_attn.cute) Windows runtime out of scope
FP8 path (A-3) Not implemented
Multi-GPU / multi-node Not in this validation
A-2-off latency ablation Not measured (A-2-on binary only)

This is an unofficial fork build. Use at your own risk. The numbers above are from one Windows + RTX 5060 Ti + CUDA 13.2 + torch 2.13.0+cu132 environment.

6. Commits (kernel + docs on the tag)

Commit Role
5b068de A-1: rescale threshold + FA4-style max lagging (Deltas 1/5/6)
9abf358 A-2: remove volatile from fma_f32x2 (Delta 4)
b6dad32 version 2.9.2.post1
a293812 validation guide title
a30a82f fork changelog v1.7
6513475 merge HEAD tagged v1.7

Full measured tables: https://github.com/ussoewwin/flash-attention/blob/v1.7/md/2.9.2.post1_COMPLETE_TEST_AND_VALIDATION_GUIDE.md

Assets 2
Loading