Skip to content

[PyTorch] Skip KV cache for sm89 and cuDNN < 9.12#1895

Merged
cyanguwa merged 3 commits intoNVIDIA:mainfrom
cyanguwa:skip_sm89_kv_cache
Jun 26, 2025
Merged

[PyTorch] Skip KV cache for sm89 and cuDNN < 9.12#1895
cyanguwa merged 3 commits intoNVIDIA:mainfrom
cyanguwa:skip_sm89_kv_cache

Conversation

@cyanguwa
Copy link
Copy Markdown
Collaborator

@cyanguwa cyanguwa commented Jun 26, 2025

Description

The bug for KV caching on sm89 won't be fixed until cuDNN 9.12. This PR adds that condition to the backend checks.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

See above.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@cyanguwa
skip kv cache for sm89, cudnn < 9.12
b8e5498 
Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>
@cyanguwa cyanguwa requested a review from ksivaman June 26, 2025 00:04
@cyanguwa
Copy link
Copy Markdown
Collaborator Author

cyanguwa commented Jun 26, 2025

/te-ci pytorch L0

cyanguwa added 2 commits June 26, 2025 11:41
@cyanguwa
fix test_numerics
4162e6d 
Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>
@cyanguwa
Merge branch 'main' into skip_sm89_kv_cache
cd33a8b
@cyanguwa
Copy link
Copy Markdown
Collaborator Author

cyanguwa commented Jun 26, 2025

/te-ci pytorch L0

@cyanguwa cyanguwa added the 2.5.0 label Jun 26, 2025
ksivaman
ksivaman approved these changes Jun 26, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@ksivaman ksivaman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cyanguwa cyanguwa merged commit cc0cb35 into NVIDIA:main Jun 26, 2025
21 checks passed
KshitijLakhani pushed a commit that referenced this pull request Jun 27, 2025
@cyanguwa @KshitijLakhani
[PyTorch] Skip KV cache for sm89 and cuDNN < 9.12 (#1895)
8382eed 
* skip kv cache for sm89, cudnn < 9.12

Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>

* fix test_numerics

Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>

---------

Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>
chtruong814 pushed a commit to chtruong814/TransformerEngine that referenced this pull request Jul 17, 2025
@cyanguwa @chtruong814
[PyTorch] Skip KV cache for sm89 and cuDNN < 9.12 (NVIDIA#1895)
1e97ef5 
* skip kv cache for sm89, cudnn < 9.12

Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>

* fix test_numerics

Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>

---------

Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ksivaman ksivaman ksivaman approved these changes

Assignees

No one assigned

Labels

2.5.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cyanguwa @ksivaman