Skip to content

[webgpu] Make the GQA's intermediate buffer static #25091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 24, 2025
Merged

[webgpu] Make the GQA's intermediate buffer static #25091

merged 2 commits into from
Jun 24, 2025

Conversation

qjia7
Copy link
Contributor

@qjia7 qjia7 commented Jun 17, 2025

Description

This PR makes the intermediate generated buffers static in GQA for the static kv cache so that it's possible to use the graph capture capability on llm.

The changes may improve the buffer cache hit rate but also slightly increase the average gpu memory usage.

@guschmue guschmue added the ep:WebGPU ort-web webgpu provider label Jun 17, 2025
@qjia7 qjia7 marked this pull request as ready for review June 18, 2025 07:39
@qjia7 qjia7 merged commit 15f6bde into main Jun 24, 2025
89 checks passed
@qjia7 qjia7 deleted the gqa_static branch June 24, 2025 04:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ep:WebGPU ort-web webgpu provider
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants