Skip to content

add blackwell support filter for 9.7<=cudnn<9.18.1 - #2775

Merged
sudhakarsingh27 merged 10 commits into
NVIDIA:mainfrom
sudhakarsingh27:fix_fp8_determinism_check
Mar 24, 2026
Merged

add blackwell support filter for 9.7<=cudnn<9.18.1#2775
sudhakarsingh27 merged 10 commits into
NVIDIA:mainfrom
sudhakarsingh27:fix_fp8_determinism_check

Conversation

@sudhakarsingh27

Copy link
Copy Markdown
Member

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
pre-commit-ci Bot and others added 2 commits March 17, 2026 23:09
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@greptile-apps

greptile-apps Bot commented Mar 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes a one-line C++ clarity fix in nvte_get_fused_attn_backend: the third architecture condition for the F16_arbitrary_seqlen backend is tightened from sm_arch_ >= 80 to sm_arch_ >= 100, making it explicit that this branch is intended for Blackwell GPUs only. The change is functionally equivalent — architectures with sm_arch 80–99 (Ampere, Ada, Hopper) are already covered by the second condition (cudnn_runtime_version >= 8903 && sm_arch_ >= 80 && sm_arch_ < 100), so the old overlap had no effect. This is a code clarity/correctness cleanup that aligns the C++ condition structure with its documented intent.

Key points:

  • No behavioral change for any existing architecture.
  • The broader "filter for Blackwell with cuDNN 9.7–9.18.1" described in the PR title is handled at the Python layer (utils.py), which was addressed in prior review rounds.
  • The C++ code follows the same pattern as the FP8 path at line 261 (sm_arch_ >= 100 for Blackwell with cuDNN ≥ 9.7.0) — consistent with the rest of the backend selection logic.
  • A small concern: unlike the FP8 path which has an explicit inline comment/exclusion for cudnn_runtime_version != 91000 to document a known bug, the F16 path has no such note explaining why there is no upper-bound guard for the Blackwell bug range. Callers using the C++ API directly (bypassing Python) would receive F16_arbitrary_seqlen for Blackwell with cuDNN in [9.7.0, 9.18.1), which is the range known to have a determinism bug.

Confidence Score: 4/5

  • Safe to merge — the single-line change is functionally equivalent for all architectures and improves code clarity; the only open question is whether a C++ inline comment or bug-exclusion guard is warranted for the Blackwell [9.7, 9.18.1) range.
  • The change is a pure clarity/correctness fix: sm_arch_ >= 80sm_arch_ >= 100 removes a logically redundant overlap between conditions 2 and 3 and makes the Blackwell intent explicit. No regression is possible. Score is 4 rather than 5 only because the Python-layer-only filtering means C++ API users on Blackwell with cuDNN 9.7–9.18.1 are unprotected, and a brief inline comment explaining this design decision would close the loop.
  • No files require special attention — the single changed file contains a straightforward, low-risk edit.

Important Files Changed

Filename Overview
transformer_engine/common/fused_attn/fused_attn.cpp Single-line clarity fix: changes the third architecture condition for F16_arbitrary_seqlen from sm_arch_ >= 80 to sm_arch_ >= 100, making it explicit that the cuDNN 9.7+ branch targets Blackwell only. Functionally equivalent (sm_arch 80–99 is already covered by condition 2). No upper-bound filter (< 9.18.1) is added at the C++ level; that filtering is handled in the Python layer.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["nvte_get_fused_attn_backend (F16/BF16 path)"] --> B{Architecture condition}
    B --> C["Cond 1: cuDNN < 8.9.0.3\nAND sm_arch == 80 or 90"]
    B --> D["Cond 2: cuDNN >= 8.9.0.3\nAND 80 <= sm_arch < 100\n(Ampere / Ada / Hopper)"]
    B --> E["Cond 3 (updated): cuDNN >= 9.7.0\nAND sm_arch >= 100\n(Blackwell only)"]
    C --> F[Check further conditions...]
    D --> F
    E --> F
    F --> G{All conditions pass?}
    G -- Yes --> H[Return F16_arbitrary_seqlen]
    G -- No --> I[Return No_Backend / F16_max512_seqlen]

    style E fill:#d4edda,stroke:#28a745
    style C fill:#fff3cd,stroke:#ffc107
    style D fill:#fff3cd,stroke:#ffc107
Loading

Reviews (5): Last reviewed commit: "Merge branch 'main' of github.com:NVIDIA..." | Re-trigger Greptile

Comment thread transformer_engine/pytorch/attention/dot_product_attention/utils.py Outdated
Comment on lines +1093 to +1103
if (
fused_attention_backend == FusedAttnBackend["F16_arbitrary_seqlen"]
and deterministic
and (cudnn_version >= (9, 7) and cudnn_version < (9, 18, 1))
and device_compute_capability >= (10, 0)
):
logger.debug(
"Determinism not supported on Blackwell for BF16 with 9.7 <= cuDNN < 9.18.1"
)
use_fused_attention = False
fused_attention_backend = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Missing is_training guard — may incorrectly disable FusedAttention for inference

Every other determinism filter in this same if use_fused_attention and deterministic: block guards against is_training (see lines 1070–1080 for FP8 and 1081–1092 for F16_arbitrary_seqlen), conveying that those non-determinism issues are backward-pass–specific. The new Blackwell / cuDNN-version filter does not include and is_training, so it will also disable FusedAttention during deterministic inference on Blackwell GPUs with cuDNN 9.7–9.18.1.

If the cuDNN bug only manifests during training (backward pass), the filter is overly broad and will unnecessarily fall back to a slower backend during inference. If it truly affects the forward pass as well, a comment explaining that would help reviewers and future maintainers understand the deviation from the existing pattern.

Consider either:

        if (
            fused_attention_backend == FusedAttnBackend["F16_arbitrary_seqlen"]
            and is_training
            and (cudnn_version >= (9, 7) and cudnn_version < (9, 18, 1))
            and device_compute_capability >= (10, 0)
        ):

or, if inference is also affected, add a comment explaining why is_training is deliberately omitted.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, check again

Comment thread transformer_engine/pytorch/attention/dot_product_attention/utils.py Outdated
Comment thread transformer_engine/pytorch/attention/dot_product_attention/utils.py Outdated
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Comment thread transformer_engine/pytorch/attention/dot_product_attention/utils.py Outdated
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
cyanguwa
cyanguwa previously approved these changes Mar 20, 2026
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27
sudhakarsingh27 merged commit 4013c6c into NVIDIA:main Mar 24, 2026
10 of 12 checks passed
KshitijLakhani pushed a commit that referenced this pull request Mar 25, 2026
* add blackwell support filter for 9.7<=cudnn<9.18.1

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* simplify conditionals

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* fix conditionals again

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* fix conditionals again

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* update the error log

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* remove the python filter and correct the cpp filter

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

---------

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
vthumbe1503 pushed a commit to ksivaman/TransformerEngine-1 that referenced this pull request Apr 1, 2026
* add blackwell support filter for 9.7<=cudnn<9.18.1

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* simplify conditionals

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* fix conditionals again

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* fix conditionals again

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* update the error log

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* remove the python filter and correct the cpp filter

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

---------

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants