-
Notifications
You must be signed in to change notification settings - Fork 463
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add GQA attention in experimental/gen_ai #2504
Conversation
This pull request was exported from Phabricator. Differential Revision: D56110657 |
✅ Deploy Preview for pytorch-fbgemm-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Summary: This diff open sources `mqa_attn_splitk`, which is a multi-query attention operator for LLM inference for the decoding phase. The operator supports the following: - BF16 input query and output types - BF16/INT4 KV cache types - 16,384 max context length - Fixed head dimension of 128 - Arbitrary query head size The INT4 KV `mqa_attn_splitk` is ~1.7x faster than the BF16 KV counterpart. Differential Revision: D56110657
This pull request was exported from Phabricator. Differential Revision: D56110657 |
Summary: This diff open sources `mqa_attn_splitk`, which is a multi-query attention operator for LLM inference for the decoding phase. The operator supports the following: - BF16 input query and output types - BF16/INT4 KV cache types - 16,384 max context length - Fixed head dimension of 128 - Arbitrary query head size The INT4 KV `mqa_attn_splitk` is ~1.7x faster than the BF16 KV counterpart. Differential Revision: D56110657
This pull request was exported from Phabricator. Differential Revision: D56110657 |
Summary: This diff open sources `mqa_attn_splitk`, which is a multi-query attention operator for LLM inference for the decoding phase. The operator supports the following: - BF16 input query and output types - BF16/INT4 KV cache types - 16,384 max context length - Fixed head dimension of 128 - Arbitrary query head size The INT4 KV `mqa_attn_splitk` is ~1.7x faster than the BF16 KV counterpart. Differential Revision: D56110657
This pull request was exported from Phabricator. Differential Revision: D56110657 |
Summary: This diff open sources `mqa_attn_splitk`, which is a multi-query attention operator for LLM inference for the decoding phase. The operator supports the following: - BF16 input query and output types - BF16/INT4 KV cache types - 16,384 max context length - Fixed head dimension of 128 - Arbitrary query head size The INT4 KV `mqa_attn_splitk` is ~1.7x faster than the BF16 KV counterpart. Differential Revision: D56110657
This pull request was exported from Phabricator. Differential Revision: D56110657 |
Summary: This diff open sources `mqa_attn_splitk`, which is a multi-query attention operator for LLM inference for the decoding phase. The operator supports the following: - BF16 input query and output types - BF16/INT4 KV cache types - 16,384 max context length - Fixed head dimension of 128 - Arbitrary query head size The INT4 KV `mqa_attn_splitk` is ~1.7x faster than the BF16 KV counterpart. Differential Revision: D56110657
This pull request was exported from Phabricator. Differential Revision: D56110657 |
Summary: This diff open sources `gqa_attn_splitk`, which is a grouped-query attention operator for LLM inference for the decoding phase. The operator supports the following: - BF16 input query and output types - BF16/INT4 KV cache types - 16,384 max context length - Fixed head dimension of 128 - Arbitrary number of query heads The INT4 KV `mqa_attn_splitk` is ~1.7x faster than the BF16 KV counterpart. Reviewed By: jianyuh, amylittleyang Differential Revision: D56110657
Summary: This diff open sources `gqa_attn_splitk`, which is a grouped-query attention operator for LLM inference for the decoding phase. The operator supports the following: - BF16 input query and output types - BF16/INT4 KV cache types - 16,384 max context length - Fixed head dimension of 128 - Arbitrary number of query heads The INT4 KV `mqa_attn_splitk` is ~1.7x faster than the BF16 KV counterpart. Reviewed By: jianyuh, amylittleyang Differential Revision: D56110657
This pull request was exported from Phabricator. Differential Revision: D56110657 |
This pull request was exported from Phabricator. Differential Revision: D56110657 |
This pull request has been merged in e839246. |
Summary:
This diff open sources
gqa_attn_splitk
, which is a grouped-queryattention operator for LLM inference for the decoding phase. The
operator supports the following:
The INT4 KV
gqa_attn_splitk
is ~1.7x faster than the BF16 KVcounterpart.
Differential Revision: D56110657