Dont quantize the current token for attention #5715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

facebook-github-bot merged 10 commits into gh/kimishpatel/114/base from gh/kimishpatel/114/head

Dec 2, 2024

Contributor

kimishpatel commented Sep 26, 2024 •

edited

Loading

Stack from ghstack (oldest at bottom):

Differential Revision: D63497872


          Dont quantize the current token for attention

8b5cd86

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

pytorch-bot bot commented Sep 26, 2024 •

edited

Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5715

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c7c9ffe with merge base c726a9b ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot added the CLA Signed label

kimishpatel mentioned this pull request

[ExecuTorch] Add quantized kv cache to llama #5664

Closed

Contributor

facebook-github-bot commented Sep 26, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

kimishpatel mentioned this pull request

Refactor custom SDPA op to separate kv cache update from the custom sdpa op #5665

Closed

facebook-github-bot added the fb-exported label

This was referenced Sep 26, 2024

Add update_quantized_cache op #5527

Closed

[Executorch][llama] Update SDPA op to use quantized kv cache #5666

Closed

[Executorch][llama] Refactoring sdpa #5667

Closed

[Executorch] Update EXECUTORCH_LIBRARY macro #5668

Closed

[Executorch][llama] Add custom_sdpa and use that instead of sdpa_with_kv_cache #5669

Closed

[Executorch][quant] Optimize per channel dequantize #5670

Merged

kimishpatel added a commit that referenced this pull request


          Dont quantize the current token for attention

9f43ab7

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

ghstack-source-id: 244978598
Pull Request resolved: #5715


          Update on "Dont quantize the current token for attention"

48e045e

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 30, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

kimishpatel added a commit that referenced this pull request


          Dont quantize the current token for attention

4c0ece4

Pull Request resolved: #5715


ghstack-source-id: 245478028
@exported-using-ghexport

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

digantdesai approved these changes

View reviewed changes


          Update on "Dont quantize the current token for attention"

d9986d4

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 1, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

kimishpatel added a commit that referenced this pull request


          Dont quantize the current token for attention

a572c04

Pull Request resolved: #5715


ghstack-source-id: 245703478
@exported-using-ghexport

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)


          Update on "Dont quantize the current token for attention"

9ed141f

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 1, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

kimishpatel added a commit that referenced this pull request


          Dont quantize the current token for attention

f237471

Pull Request resolved: #5715


ghstack-source-id: 245718144
@exported-using-ghexport

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)


          Update on "Dont quantize the current token for attention"

ae88ec7

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 1, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

kimishpatel added a commit that referenced this pull request


          Dont quantize the current token for attention

adfc140

Pull Request resolved: #5715


ghstack-source-id: 245751541
@exported-using-ghexport

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)


          Update on "Dont quantize the current token for attention"

b54362e

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

kimishpatel added a commit that referenced this pull request


          Dont quantize the current token for attention

6f57124

Pull Request resolved: #5715


ghstack-source-id: 246121068
@exported-using-ghexport

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

Contributor

facebook-github-bot commented Oct 3, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872


          Update on "Dont quantize the current token for attention"

a25f5bf

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

kimishpatel mentioned this pull request

[Executorch][llama] Rename update_quantized_cache to update_cache #6914

Merged

Contributor

facebook-github-bot commented Nov 16, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

This was referenced Nov 20, 2024

[Executorch][BE] Rename sdpa_with_kv_cache.py to custom_ops.py #6996

Merged

[Executorch] Add quantized kv cache to oss ci #6997

Merged


          Update on "Dont quantize the current token for attention"

622e928

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Nov 21, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

kimishpatel added the topic: not user facing label


          Update on "Dont quantize the current token for attention"

f187574

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Nov 22, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

This was referenced Nov 22, 2024

[Executorch][custom ops] Change lib loading logic to account for package dir #7038

Merged

[Executorch][CI] Fix qnn runner ci job scripts #7049

Closed


          Update on "Dont quantize the current token for attention"

c7c9ffe

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Nov 28, 2024

This pull request was exported from Phabricator. Differential Revision: D63497872

facebook-github-bot merged commit 3b4a931 into gh/kimishpatel/114/base

41 of 43 checks passed

facebook-github-bot deleted the gh/kimishpatel/114/head branch

December 2, 2024 16:19

facebook-github-bot temporarily deployed to cherry-pick-bot

December 2, 2024 16:19

— with

GitHub Actions Inactive

pytorchbot mentioned this pull request

Dont quantize the current token for attention #7140

Closed

kirklandsign pushed a commit that referenced this pull request


          Dont quantize the current token for attention

418ede3

Pull Request resolved: #5715


ghstack-source-id: 255730816
@exported-using-ghexport

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

kirklandsign mentioned this pull request

Dont quantize the current token for attention #7164

Merged

kirklandsign added a commit that referenced this pull request


          Dont quantize the current token for attention

73b823b

Pull Request resolved: #5715


ghstack-source-id: 255730816
@exported-using-ghexport

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

Co-authored-by: Kimish Patel <kimishpatel@fb.com>

kedarnath03 pushed a commit to kedarnath03/executorch that referenced this pull request


          Dont quantize the current token for attention

93fc7d5

Pull Request resolved: pytorch/executorch#5715


ghstack-source-id: 254987268
@exported-using-ghexport

Differential Revision: [D63497872](https://our.internmc.facebook.com/intern/diff/D63497872/)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed fb-exported topic: not user facing