Skip to content

Fork Release v1.8 — PyTorch 2.14.0+cu132 Compatibility & Upstream Sync

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 04 Sep 05:28
· 6 commits to main since this release
v1.8
9b45d73

Overview

Fork release v1.8 provides verified compatibility for PyTorch 2.14.0+cu132 under Windows MSVC, resolves linker symbol mismatches (LNK2001 / LNK1120), synchronizes the latest official upstream improvements (ce088ab / FA4 CuTe DSL), and strictly maintains all fork-specific architectural features (24 split_align kernels, MSVC 2GB COFF limit bypass via 3:3 SM split, PEP 427 wheel naming, and complete CI workflow suppression).

1. PyTorch 2.14.0+cu132 C++ ABI Compatibility & Linker Fix

Root Cause of MSVC LNK1120 / LNK2001

When building against PyTorch 2.14.0 under MSVC, link.exe could fail with exit status 1120:

flash_api.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) void __cdecl c10::cuda::c10_cuda_check_implementation(int,char const *,char const *,unsigned int,bool)" (__imp_?c10_cuda_check_implementation@cuda@c10@@YAXHPEBD0I_N@Z)
  Hint on matching symbol:
    "__declspec(dllimport) void __cdecl c10::cuda::c10_cuda_check_implementation(int,char const *,char const *,unsigned int,bool,class c10::cuda::CUDAErrorLogCapture *)" (__imp_?c10_cuda_check_implementation@cuda@c10@@YAXHPEBD0I_NPEAVCUDAErrorLogCapture@12@@Z)
build\lib.win-amd64-cpython-314\flash_attn_2_cuda.cp314-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals

Technical Breakdown

  • Signature Expansion in PyTorch 2.14.0:
    Starting in PyTorch 2.14.0, CUDA error handling was enhanced with c10::cuda::CUDAErrorLogCapture. The function signature of c10_cuda_check_implementation in c10/cuda/CUDAException.h expanded from 5 parameters to 6 parameters: 
    // PyTorch 2.14.0+ signature:
C10_CUDA_API void c10_cuda_check_implementation(
    const int32_t err,
    const char* filename,
    const char* function_name,
    const uint32_t line_number,
    const bool include_device_assertions,
    CUDAErrorLogCapture* error_log = nullptr);
  • ABI Mismatch Mechanism:
    PyTorch 2.14.0's c10_cuda.lib exports only the 6-parameter mangled symbol (?c10_cuda_check_implementation...PEAVCUDAErrorLogCapture...). If an existing flash_api.obj compiled under PyTorch 2.13.0 headers remained in the build tree, ninja skipped recompilation because flash_api.cpp had no source diff, causing the linker to request the obsolete 5-parameter symbol.
  • Resolution:
    Purging intermediate build artifacts (build/ directory) ensures flash_api.cpp compiles freshly against PyTorch 2.14.0's header, correctly emitting references to the 6-parameter symbol with default nullptr and resolving cleanly without any external symbol gaps.

2. Upstream Sync (Dao-AILab/flash-attention @ ce088ab)

The latest changes from the authoritative upstream repository were cleanly merged without touching any FlashAttention-2 code:

  • SM100 Scalar Mask Compilation Speedup (Dao-AILab#2819):
    Replaced static Python loop expansion in flash_attn/cute/mask.py with dynamic CuTeDSL unrolling for 128-element scalar mask-mod loops, reducing cold first-call compilation latency by ~31%.
  • CUTLASS DSL Requirement Relaxation (Dao-AILab#2798):
    Updated pyproject.toml dependency floor to nvidia-cutlass-dsl>=4.6.2 to support newer QuACK releases and CUTLASS DSL 4.7.
  • QuACK Packed Subtraction Compatibility (Dao-AILab#2787):
    Adapted flash_attn/cute/flash_bwd_sm100.py and utils.py to invoke CuTe DSL primitives directly following QuACK's deprecation of sub_packed_f32x2.

3. Preserved Fork Architecture & Windows Optimizations

All fork-specific optimizations remain 100% active and untouched:

  • 24 split_align Forward Kernels:
    Independent translation units (flash_fwd_split_align_*.cu) for num_splits == 1 sequences eliminate ptxas compiler timeouts.
  • MSVC 2GB COFF Linker Limit Bypass (3:3 SM Split):
    • WindowsWhlBuilder_cuda_2.bat: Blackwell targets (100;120;121).
    • WindowsWhlBuilder_cuda_3.bat: Legacy targets (80;89;90), with MAX_JOBS=5.
  • PEP 427 Compliant Wheel Naming:
    Uses .blackwell and .legacy dot-suffixes to avoid pip Invalid build number errors.
  • Dynamic Runtime Versioning:
    flash_attn/__init__.py uses importlib.metadata to dynamically reflect package version (2.9.1 vs 2.9.2).
  • CI Workflow Suppression:
    .github and tools/ci remain purged from the repository.

4. Pre-built Windows Wheels

Pre-compiled wheels are published on Hugging Face:
👉 ussoewwin/Flash-Attention-2_for_Windows

Quick Install Command

pip install https://huggingface.co/ussoewwin/Flash-Attention-2_for_Windows/resolve/main/<matching_wheel_file>.whl
Assets 2
Loading