Skip to content

[Perf] Fix and reapply move apply w8a8 block fp8 linear to class #25696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 65 commits into from
Oct 2, 2025
Merged

[Perf] Fix and reapply move apply w8a8 block fp8 linear to class #25696

merged 65 commits into from
Oct 2, 2025

Conversation

ElizaWszola
Copy link
Contributor

@ElizaWszola ElizaWszola commented Sep 25, 2025 •
edited by github-actions bot
Loading

This PR resolves the issue (#25623) raised in #25607 by using a single padding-cutlass custom op on Hopper.

We are waiting to merge this after v0.11.0 release out of an abundance of caution.

@ElizaWszola
Move apply_w8a8_block_fp8_linear to an op class
c9ca102 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Remove TODO, bring back old one
eef4349 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
CUDA graphs fix
dd53183 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Clean up
bb24881 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Create linear op objects conditionally, move some arch checks to bloc…
1ba47cd 
…ked op constructor

Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
format
02793b9 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
clean up repetitive code
b72c9f2 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
More aggressive dispatch of blockscale ops
d51f35c 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
fix
a6ae689 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Deep_gemm fix
3238ff6 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
f9c79aa 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
23341c2 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Post-merge fixes, better dispatch
9b09b60 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
small fixes
e6b0028 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
9b5c552 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Fix cutlass compilation issue on Hopper
ef6f1e2 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Cleanup bad transpose
77335de 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
5eaf155 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Wrap w8a8_block_fp8_matmul
e036dac 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Rename padded_cutlass to padded_cutlass_scaled_mm, add todo
233e874 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Cleanup dispatch_w8a8_blockscale_func
1edfedc 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
35a0236 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Deep gemm warmup fix
0ac3a1e 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Fix deep gemm support function
9a48100 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Feedback
b6a8fb8 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Pre-commit fixes
e89ecd8 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Pre-commit fixes 2
00cb05c 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Feedback
66c89e6 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
fix type issue
d9b4121 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Add use_ue8m0 support to _quantize_group_native
1bc81a1 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@mergify mergify bot removed the needs-rebase label Sep 29, 2025
ElizaWszola
ElizaWszola commented Sep 29, 2025
View reviewed changes
ProExpertProg
ProExpertProg reviewed Sep 29, 2025
View reviewed changes
@ProExpertProg ProExpertProg mentioned this pull request Sep 29, 2025
Open
1 task
@mergify Mergify
Copy link

mergify bot commented Sep 30, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ElizaWszola.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 30, 2025
@ElizaWszola
remove dispatch key
e450042 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
c6bbf49 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@mergify mergify bot removed the needs-rebase label Sep 30, 2025
@ProExpertProg ProExpertProg enabled auto-merge (squash) September 30, 2025 11:41
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 30, 2025
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
a7b6a16 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
2d input in deep_gemm
301dc11 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
auto-merge was automatically disabled September 30, 2025 19:28

Head branch was pushed to by a user without write access

@ProExpertProg ProExpertProg removed the ready ONLY add when PR is ready to merge/full CI is needed label Sep 30, 2025
@ElizaWszola
Better dims in test_quantfp8_group_multidimensional, fix dim order
497ea78 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Clean up padded_cutlass op register
fec861d 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
ProExpertProg
ProExpertProg reviewed Oct 1, 2025
View reviewed changes
@ElizaWszola
More register custom op cleanup
1b36b18 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
ProExpertProg
ProExpertProg approved these changes Oct 1, 2025
View reviewed changes
Copy link
Collaborator

@ProExpertProg ProExpertProg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just two logic simplification comments, otherwise LGTM!

@ElizaWszola
nits
7b7956d 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ProExpertProg ProExpertProg enabled auto-merge (squash) October 1, 2025 14:40
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 1, 2025
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
398c8b3 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class
710d86b 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ProExpertProg ProExpertProg merged commit 502640c into vllm-project:main Oct 2, 2025
54 checks passed
@ProExpertProg ProExpertProg mentioned this pull request Oct 2, 2025
Merged
pdasigi pushed a commit to pdasigi/vllm that referenced this pull request Oct 2, 2025
@ElizaWszola @ProExpertProg
@mgoin @pdasigi
[Perf] Fix and reapply move apply w8a8 block fp8 linear to class (vll…
2d7cbf4 
…m-project#25696)

Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: ElizaWszola <elizaw.9289@gmail.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>
@ProExpertProg
Copy link
Collaborator

@ElizaWszola now that cutlass 4.2 has landed, can you remove the hopper padding in a follow-up?

yewentao256 pushed a commit that referenced this pull request Oct 3, 2025
@ElizaWszola @ProExpertProg
@mgoin @yewentao256
[Perf] Fix and reapply move apply w8a8 block fp8 linear to class (#25696
abc55b1 
)

Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: ElizaWszola <elizaw.9289@gmail.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ProExpertProg ProExpertProg ProExpertProg approved these changes

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo simon-mo is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@houseroad houseroad Awaiting requested review from houseroad houseroad is a code owner

@hmellor hmellor Awaiting requested review from hmellor hmellor is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
performance Performance-related issues ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
vllm==v0.12.0/torch==2.9.0 compilatio...
Development

Successfully merging this pull request may close these issues.
3 participants
@ElizaWszola @ProExpertProg @mgoin