Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Why fused attention is only applicable on Ampere GPUs? #1279

Open
rayleizhu opened this issue Mar 5, 2023 · 1 comment
Open

Why fused attention is only applicable on Ampere GPUs? #1279

rayleizhu opened this issue Mar 5, 2023 · 1 comment

Comments

@rayleizhu
Copy link

Hi, I'm writing my operator using fused attention as a template. However, I found that fused attention requires an Ampere arch:

https://github.com/openai/triton/blob/d376020f90002757eea3ea9475d4f7cfc2ec5ead/python/triton/ops/flash_attention.py#L200

I do not understand this.

  • Does it mean this template uses some arch-specific operators?
  • To use it on Volta GPU, how should I modify it?

Besides, it seems that only head_dim=64 is supported, right? How can I fix it for the head_dim=32 case?

https://github.com/openai/triton/blob/d376020f90002757eea3ea9475d4f7cfc2ec5ead/python/triton/ops/flash_attention.py#L207
@ptillet
Copy link
Collaborator

ptillet commented Mar 6, 2023 •
edited

there is some more information in #616

ZzEeKkAa pushed a commit to ZzEeKkAa/triton that referenced this issue Aug 5, 2024
@gshimansky
Enable upstream pytorch (no ipex) for e2e accuracy workflow (triton-l…
b8c8d3a 
…ang#1279)

The core Triton is a small number of people, and we receive many PRs
(thank
you!).  To help us review your code more quickly, **if you are a new
contributor (less than 3 PRs merged) we ask that you fill out the
following
checklist and include it in your PR description.**

Fill out the checklist by replacing `[ ]` with `[x]`.

- [x] I am not making a trivial change, such as fixing a typo in a
comment.

- [x] I have written a PR description following these
  [rules](https://cbea.ms/git-commit/#why-not-how).

- [ ] I have used an LLM to copyedit my PR description and and code
comments.

- [x] I have run `pre-commit run --from-ref origin/main --to-ref HEAD`.

- Select one of the following.
  - [ ] I have added tests.
    - `/test` for `lit` tests
    - `/unittest` for C++ tests
    - `/python/test` for end-to-end tests
- [x] This PR does not need a test because `it doesn't change any triton
code`.

- Select one of the following.
  - [x] I have not added any `lit` tests.
- [ ] The `lit` tests I have added are "minimal" -- they contain only
the
instructions necessary to exercise the bug. (Usually running Python code
    and using the instructions it generates is not minimal.)

---------

Signed-off-by: Gregory Shimansky <gshimansky@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ptillet @rayleizhu