[Performance] Move apply_w8a8_block_fp8_linear to an op class #24666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

vllm-bot merged 50 commits into vllm-project:main from neuralmagic:move-apply_w8a8_block_fp8_linear-to-class

Sep 23, 2025

Contributor

ElizaWszola commented Sep 11, 2025 •

edited by github-actions bot

Loading

This PR unwraps apply_w8a8_block_fp8_linear custom op by implementing a class that we can instantiate and run from Fp8LinearMethod. Part of work outlined in #24629

Testing:

E2E tested with on Blackwell and Hopper

llm = LLM(model="Qwen/Qwen3-30B-A3B-FP8")
llm = LLM(model="RedHatAI/Qwen3-0.6B-FP8-BLOCK")

ElizaWszola added 3 commits

September 11, 2025 13:38


          Move apply_w8a8_block_fp8_linear to an op class

c9ca102

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Remove TODO, bring back old one

eef4349

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          CUDA graphs fix

dd53183

Signed-off-by: ElizaWszola <ewszola@redhat.com>

ProExpertProg reviewed

View reviewed changes

Collaborator

ProExpertProg left a comment

A few overall notes

vllm/model_executor/layers/quantization/fp8.py Outdated Show resolved Hide resolved

vllm/model_executor/layers/quantization/utils/fp8_utils.py Outdated Show resolved Hide resolved

vllm/model_executor/layers/quantization/utils/fp8_utils.py Outdated Show resolved Hide resolved

vllm/model_executor/layers/quantization/utils/fp8_utils.py Show resolved Hide resolved

ProExpertProg reviewed

View reviewed changes

vllm/model_executor/layers/quantization/fp8.py Outdated Show resolved Hide resolved

ElizaWszola added 2 commits

September 11, 2025 15:13


          Clean up

bb24881

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Create linear op objects conditionally, move some arch checks to bloc…

1ba47cd

…ked op constructor

Signed-off-by: ElizaWszola <ewszola@redhat.com>

ProExpertProg mentioned this pull request

[torch.compile][Performance]: Unwrap custom ops and improve fusion (Inductor and custom) #24629

Open

1 task


          format

02793b9

Signed-off-by: ElizaWszola <ewszola@redhat.com>

mergify bot commented Sep 12, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ElizaWszola.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify bot added the needs-rebase label

ElizaWszola added 7 commits

September 12, 2025 04:48


          clean up repetitive code

b72c9f2

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          More aggressive dispatch of blockscale ops

d51f35c

Signed-off-by: ElizaWszola <ewszola@redhat.com>

fix

a6ae689

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Deep_gemm fix

3238ff6

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class

f9c79aa

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class

23341c2

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Post-merge fixes, better dispatch

9b09b60

Signed-off-by: ElizaWszola <ewszola@redhat.com>

mergify bot removed the needs-rebase label

ElizaWszola added 6 commits

September 12, 2025 11:25


          small fixes

e6b0028

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class

9b5c552

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Fix cutlass compilation issue on Hopper

ef6f1e2

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Cleanup bad transpose

77335de

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class

5eaf155

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Wrap w8a8_block_fp8_matmul

e036dac

Signed-off-by: ElizaWszola <ewszola@redhat.com>

ElizaWszola marked this pull request as ready for review

September 17, 2025 12:38

ElizaWszola requested review from mgoin, robertgshaw2-redhat, tlrmchlsmth and yewentao256 as code owners

September 17, 2025 12:38

ElizaWszola mentioned this pull request

[Bug]: torch.compile fails for Gemma3n on pytorch 2.8 #24547

Open

1 task

ElizaWszola added 2 commits

September 23, 2025 00:17


          Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class

15cf30e

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          CUDA condition for compressed tensors and H100

ebdcb10

Signed-off-by: ElizaWszola <ewszola@redhat.com>

mergify bot removed the needs-rebase label

ElizaWszola added 3 commits

September 23, 2025 02:47


          Fix quantfp8 test

2e3d206

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Test scales_col vs. scales_native

bd32cb9

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class

efa4446

Signed-off-by: ElizaWszola <ewszola@redhat.com>

mgoin approved these changes

View reviewed changes

tests/kernels/quantization/test_fp8_quant_group.py Show resolved Hide resolved

ElizaWszola added 2 commits

September 23, 2025 10:20


          Add compressed tensors model test

1f00804

Signed-off-by: ElizaWszola <ewszola@redhat.com>


          Extra asserts, don't use enabled()

e895df6

Signed-off-by: ElizaWszola <ewszola@redhat.com>

ProExpertProg approved these changes

View reviewed changes

Collaborator

ProExpertProg left a comment

Really clean, thanks for the great work!

ProExpertProg enabled auto-merge (squash)

September 23, 2025 15:37

ProExpertProg reviewed

View reviewed changes

Collaborator

ProExpertProg left a comment

Gonna submit this to always use the cuda path - we can enable torch for Blackwell in a follow-up as suggested by @mgoin, he also has improvements to the torch path for H100.

tests/quantization/test_compressed_tensors.py Outdated Show resolved Hide resolved

tests/quantization/test_compressed_tensors.py Outdated Show resolved Hide resolved

vllm/config/__init__.py Outdated Show resolved Hide resolved

vllm/config/__init__.py Outdated Show resolved Hide resolved

ProExpertProg added 2 commits

September 23, 2025 11:48


          CUDA path for quant

9806cf8

Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>


          Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class

2ae1ef9

mergify bot commented Sep 23, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ElizaWszola.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify bot added the needs-rebase label


          Merge branch 'main' into move-apply_w8a8_block_fp8_linear-to-class

00bd638

Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

mergify bot removed the needs-rebase label

vllm-bot merged commit 6340025 into vllm-project:main

48 of 49 checks passed

ProExpertProg added this to the vllm==v0.12.0/torch==2.9.0 compilation improvements milestone

eicherseiji mentioned this pull request

[Bug] Qwen3 DPTPEP cutlass_scaled_mm: RuntimeError: m must be divisible by 4 #25527

Closed

1 task

tlrmchlsmth added a commit to tlrmchlsmth/vllm that referenced this pull request


          Revert "[Performance] Move apply_w8a8_block_fp8_linear to an op class (…

7c6404d

…vllm-project#24666)"

This reverts commit 6340025.

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>

tlrmchlsmth mentioned this pull request

Revert "[Performance] Move apply_w8a8_block_fp8_linear to an op class… #25607

Merged

ProExpertProg mentioned this pull request

[Bug]: torch.compile padding causes IMA on Hopper + DBO #25623

Open

1 task

FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request


          [Performance] Move apply_w8a8_block_fp8_linear to an op class (vllm-p…

a691a6d

…roject#24666)

Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: ElizaWszola <elizaw.9289@gmail.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>

This was referenced Sep 25, 2025

New PR number #25693 #25688

Closed

[Rocm][torch.compile] Adding layernorm + fp8 block quant and silu + fp8 block quant for Aiter #25693

Open

yewentao256 pushed a commit that referenced this pull request


          [Performance] Move apply_w8a8_block_fp8_linear to an op class (#24666)

b672b8c

Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: ElizaWszola <elizaw.9289@gmail.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

ProExpertProg ProExpertProg approved these changes

mgoin mgoin approved these changes

robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

simon-mo Awaiting requested review from simon-mo simon-mo is a code owner

WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

houseroad Awaiting requested review from houseroad houseroad is a code owner

hmellor Awaiting requested review from hmellor hmellor is a code owner

Labels

performance ready