Skip to content

[Bugfix][Benchmark] Fix Marlin benchmark #19929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jun 23, 2025
Merged

[Bugfix][Benchmark] Fix Marlin benchmark #19929

merged 5 commits into from
Jun 23, 2025

Conversation

22quinn
Copy link
Collaborator

@22quinn 22quinn commented Jun 21, 2025 •
edited by github-actions bot
Loading

Essential Elements of an Effective PR Description Checklist

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

benchmark_marlin.py is outdated. A few updates to the marlin ops have been introduced, breaking the benchmark script. This PR fix the benchmark and also expands it to test more quantization data types.
Fix #19909

Test Plan

python benchmarks/kernels/benchmark_marlin.py --batch-sizes 1

Test Result

[----------------------------------------------------------------------------------------------------- Quant Matmul -----------------------------------------------------------------------------------------------------]
                                                                                                        |  pytorch_gemm  |  gptq_marlin_gemm  |  gptq_marlin_gemm_fp32  |  gptq_marlin_24_gemm  |  gptq_marlin_repack
1 threads: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=-1, MKN=(1x4096x12288)         |      52.7      |           11.5          |           11.4          |          11.3         |         49.3       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=32, MKN=(1x4096x12288)         |      52.3      |           13.6          |           13.6          |                       |         49.6       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=64, MKN=(1x4096x12288)         |      52.2      |           12.6          |           12.5          |                       |         49.7       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=128, MKN=(1x4096x12288)        |      52.7      |           12.2          |           12.1          |          12.1         |         49.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=-1, MKN=(1x4096x12288)       |      52.3      |           22.3          |           22.7          |          13.5         |         89.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=32, MKN=(1x4096x12288)       |      52.7      |           27.0          |           27.4          |                       |         91.2       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=64, MKN=(1x4096x12288)       |      52.7      |           22.3          |           22.7          |                       |         91.2       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=128, MKN=(1x4096x12288)      |      52.3      |           21.5          |           21.7          |          13.6         |         88.6       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float8_e4m3fn, g=-1, MKN=(1x4096x12288)   |      52.3      |           22.1          |           22.4          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float8_e4m3fn, g=128, MKN=(1x4096x12288)  |      52.7      |           22.3          |           23.0          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float4_e2m1f, g=16, MKN=(1x4096x12288)    |      55.8      |           15.4          |           15.3          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=-1, MKN=(1x4096x12288)           |      52.7      |           12.0          |           11.9          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=32, MKN=(1x4096x12288)           |      52.7      |           15.6          |           15.5          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=64, MKN=(1x4096x12288)           |      52.7      |           13.5          |           13.4          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=128, MKN=(1x4096x12288)          |      52.6      |           13.5          |           13.4          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=-1, MKN=(1x4096x12288)          |      52.2      |           11.5          |           11.4          |          11.4         |         49.6       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=32, MKN=(1x4096x12288)          |      52.7      |           13.7          |           13.7          |                       |         49.4       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=64, MKN=(1x4096x12288)          |      52.7      |           12.6          |           12.5          |                       |         49.2       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=128, MKN=(1x4096x12288)         |      52.2      |           12.2          |           12.1          |          12.1         |         49.5       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=-1, MKN=(1x4096x12288)        |      52.7      |           22.7          |           22.9          |          13.5         |         87.7       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=32, MKN=(1x4096x12288)        |      52.2      |           27.0          |           27.1          |                       |         87.3       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=64, MKN=(1x4096x12288)        |      52.2      |           22.6          |           22.7          |                       |         86.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=128, MKN=(1x4096x12288)       |      52.7      |           21.3          |           21.7          |          13.6         |         91.2       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float8_e4m3fn, g=-1, MKN=(1x4096x12288)    |      52.7      |           22.5          |           22.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float8_e4m3fn, g=128, MKN=(1x4096x12288)   |      52.2      |           23.5          |           23.9          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float4_e2m1f, g=16, MKN=(1x4096x12288)     |      56.6      |           15.4          |           15.2          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=-1, MKN=(1x4096x12288)            |      52.3      |           12.0          |           12.0          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=32, MKN=(1x4096x12288)            |      52.2      |           15.6          |           15.5          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=64, MKN=(1x4096x12288)            |      52.2      |           13.5          |           13.4          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=128, MKN=(1x4096x12288)           |      52.2      |           13.5          |           13.4          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=32, MKN=(1x4096x12288)          |      52.7      |           45.9          |           45.9          |                       |         77.6       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=64, MKN=(1x4096x12288)          |      52.7      |           35.3          |           35.4          |                       |         77.4       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=128, MKN=(1x4096x12288)         |      52.1      |           28.3          |           28.3          |          12.1         |         77.6       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=32, MKN=(1x4096x12288)        |      52.7      |           71.1          |           71.7          |                       |         80.2       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=64, MKN=(1x4096x12288)        |      52.7      |           52.4          |           52.8          |                       |         82.4       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=128, MKN=(1x4096x12288)       |      52.3      |           40.5          |           40.6          |          13.5         |         84.3       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=32, MKN=(1x4096x12288)           |      52.7      |           17.1          |           17.1          |                       |         77.5       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=64, MKN=(1x4096x12288)           |      52.7      |           16.0          |           15.9          |                       |         77.4       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=128, MKN=(1x4096x12288)          |      52.2      |           15.5          |           15.5          |          12.1         |         77.7       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=32, MKN=(1x4096x12288)         |      52.6      |           31.8          |           32.9          |                       |         80.3       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=64, MKN=(1x4096x12288)         |      52.7      |           28.7          |           29.1          |                       |         82.2       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=128, MKN=(1x4096x12288)        |      52.2      |           26.2          |           26.5          |          13.8         |         85.7       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=-1, MKN=(1x4096x4096)          |      11.7      |            9.9          |           10.4          |           9.3         |         16.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=32, MKN=(1x4096x4096)          |      11.7      |           10.0          |           10.4          |                       |         16.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=64, MKN=(1x4096x4096)          |      11.9      |            9.8          |           10.3          |                       |         16.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=128, MKN=(1x4096x4096)         |      11.9      |            9.9          |           10.4          |           9.5         |         16.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=-1, MKN=(1x4096x4096)        |      11.9      |           11.4          |           11.3          |           9.9         |         32.3       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=32, MKN=(1x4096x4096)        |      11.8      |           11.6          |           11.5          |                       |         33.6       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=64, MKN=(1x4096x4096)        |      11.7      |           11.5          |           11.3          |                       |         32.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=128, MKN=(1x4096x4096)       |      11.7      |           11.4          |           11.3          |          10.1         |         31.6       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float8_e4m3fn, g=-1, MKN=(1x4096x4096)    |      11.9      |           11.8          |           12.1          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float8_e4m3fn, g=128, MKN=(1x4096x4096)   |      11.7      |           11.8          |           12.2          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float4_e2m1f, g=16, MKN=(1x4096x4096)     |      11.7      |           10.9          |           11.5          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=-1, MKN=(1x4096x4096)            |      11.9      |           11.0          |           11.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=32, MKN=(1x4096x4096)            |      11.9      |           11.1          |           11.6          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=64, MKN=(1x4096x4096)            |      11.9      |           11.0          |           11.5          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=128, MKN=(1x4096x4096)           |      11.8      |           11.0          |           11.5          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=-1, MKN=(1x4096x4096)           |      11.8      |            9.8          |           10.3          |           9.3         |         16.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=32, MKN=(1x4096x4096)           |      11.9      |           10.0          |           10.4          |                       |         16.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=64, MKN=(1x4096x4096)           |      11.9      |            9.8          |           10.3          |                       |         16.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=128, MKN=(1x4096x4096)          |      11.9      |            9.8          |           10.4          |           9.5         |         16.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=-1, MKN=(1x4096x4096)         |      11.9      |           11.4          |           11.3          |           9.9         |         32.4       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=32, MKN=(1x4096x4096)         |      11.9      |           11.6          |           11.5          |                       |         32.0       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=64, MKN=(1x4096x4096)         |      11.9      |           11.3          |           11.2          |                       |         32.0       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=128, MKN=(1x4096x4096)        |      11.7      |           11.5          |           11.4          |          10.1         |         32.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float8_e4m3fn, g=-1, MKN=(1x4096x4096)     |      11.8      |           11.8          |           12.1          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float8_e4m3fn, g=128, MKN=(1x4096x4096)    |      11.9      |           11.9          |           12.1          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float4_e2m1f, g=16, MKN=(1x4096x4096)      |      11.9      |           11.0          |           11.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=-1, MKN=(1x4096x4096)             |      11.8      |           11.1          |           11.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=32, MKN=(1x4096x4096)             |      11.9      |           11.1          |           11.6          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=64, MKN=(1x4096x4096)             |      11.9      |           11.0          |           11.6          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=128, MKN=(1x4096x4096)            |      11.8      |           11.1          |           11.6          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=32, MKN=(1x4096x4096)           |      11.7      |           24.0          |           23.8          |                       |         27.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=64, MKN=(1x4096x4096)           |      11.7      |           20.0          |           19.8          |                       |         27.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=128, MKN=(1x4096x4096)          |      11.9      |           17.1          |           16.9          |           9.5         |         27.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=32, MKN=(1x4096x4096)         |      11.9      |           25.1          |           24.9          |                       |         28.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=64, MKN=(1x4096x4096)         |      17.6      |           21.4          |           21.2          |                       |         28.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=128, MKN=(1x4096x4096)        |      11.9      |           18.5          |           18.3          |          10.0         |         28.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=32, MKN=(1x4096x4096)            |      11.7      |           13.5          |           13.3          |                       |         27.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=64, MKN=(1x4096x4096)            |      11.7      |           13.0          |           12.9          |                       |         27.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=128, MKN=(1x4096x4096)           |      11.7      |           12.9          |           12.7          |           9.5         |         27.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=32, MKN=(1x4096x4096)          |      11.7      |           15.0          |           14.8          |                       |         28.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=64, MKN=(1x4096x4096)          |      11.9      |           15.3          |           14.9          |                       |         28.1       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=128, MKN=(1x4096x4096)         |      11.7      |           14.8          |           14.7          |          10.1         |         27.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=-1, MKN=(1x4096x22016)         |      84.8      |           19.1          |           19.2          |          16.7         |         85.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=32, MKN=(1x4096x22016)         |      84.3      |           24.3          |           24.6          |                       |         86.0       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=64, MKN=(1x4096x22016)         |      83.9      |           21.3          |           21.6          |                       |         85.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=128, MKN=(1x4096x22016)        |      84.0      |           19.7          |           19.7          |          18.4         |         85.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=-1, MKN=(1x4096x22016)       |      84.9      |           48.0          |           48.8          |          31.3         |        165.6       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=32, MKN=(1x4096x22016)       |      84.4      |           48.0          |           49.4          |                       |        166.2       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=64, MKN=(1x4096x22016)       |      84.2      |           46.6          |           47.7          |                       |        165.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=128, MKN=(1x4096x22016)      |      84.2      |           46.1          |           47.3          |          32.3         |        166.0       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float8_e4m3fn, g=-1, MKN=(1x4096x22016)   |      84.2      |           48.4          |           49.1          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float8_e4m3fn, g=128, MKN=(1x4096x22016)  |      84.3      |           46.7          |           47.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float4_e2m1f, g=16, MKN=(1x4096x22016)    |      90.9      |           26.4          |           26.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=-1, MKN=(1x4096x22016)           |      85.0      |           19.7          |           19.9          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=32, MKN=(1x4096x22016)           |      84.4      |           27.3          |           27.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=64, MKN=(1x4096x22016)           |      83.9      |           22.7          |           23.4          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=128, MKN=(1x4096x22016)          |      84.0      |           21.7          |           21.8          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=-1, MKN=(1x4096x22016)          |      85.0      |           19.2          |           19.2          |          16.7         |         85.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=32, MKN=(1x4096x22016)          |      84.2      |           24.3          |           24.8          |                       |         85.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=64, MKN=(1x4096x22016)          |      83.5      |           21.3          |           21.5          |                       |         85.7       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=128, MKN=(1x4096x22016)         |      84.4      |           19.7          |           19.8          |          18.5         |         85.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=-1, MKN=(1x4096x22016)        |      84.9      |           48.0          |           48.8          |          31.2         |        165.5       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=32, MKN=(1x4096x22016)        |      83.5      |           48.1          |           49.4          |                       |        171.4       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=64, MKN=(1x4096x22016)        |      84.2      |           46.7          |           47.7          |                       |        165.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=128, MKN=(1x4096x22016)       |      84.2      |           46.1          |           47.2          |          32.2         |        166.1       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float8_e4m3fn, g=-1, MKN=(1x4096x22016)    |      84.2      |           48.4          |           49.1          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float8_e4m3fn, g=128, MKN=(1x4096x22016)   |      84.2      |           46.7          |           47.6          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float4_e2m1f, g=16, MKN=(1x4096x22016)     |      90.7      |           26.3          |           26.4          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=-1, MKN=(1x4096x22016)            |      84.8      |           19.7          |           19.9          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=32, MKN=(1x4096x22016)            |      84.2      |           27.3          |           27.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=64, MKN=(1x4096x22016)            |      84.1      |           22.7          |           23.6          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=128, MKN=(1x4096x22016)           |      84.1      |           21.8          |           21.9          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=32, MKN=(1x4096x22016)          |      84.7      |           97.7          |           97.7          |                       |        136.4       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=64, MKN=(1x4096x22016)          |      84.6      |           71.3          |           71.5          |                       |        136.5       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=128, MKN=(1x4096x22016)         |      84.7      |           52.1          |           51.9          |          18.4         |        136.7       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=32, MKN=(1x4096x22016)        |      83.9      |          110.6          |          111.1          |                       |        138.1       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=64, MKN=(1x4096x22016)        |      84.6      |           82.7          |           83.4          |                       |        141.2       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=128, MKN=(1x4096x22016)       |      84.6      |           68.5          |           69.5          |          32.3         |        144.8       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=32, MKN=(1x4096x22016)           |      83.8      |           29.4          |           29.8          |                       |        136.4       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=64, MKN=(1x4096x22016)           |      84.5      |           26.2          |           26.4          |                       |        136.4       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=128, MKN=(1x4096x22016)          |      84.6      |           24.6          |           24.6          |          18.4         |        136.5       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=32, MKN=(1x4096x22016)         |      84.6      |           52.9          |           54.0          |                       |        138.6       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=64, MKN=(1x4096x22016)         |      83.4      |           51.3          |           52.3          |                       |        141.3       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=128, MKN=(1x4096x22016)        |      84.5      |           51.2          |           52.2          |          32.3         |        146.3       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=-1, MKN=(1x11008x4096)         |      47.3      |           13.7          |           13.7          |          13.6         |         49.4       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=32, MKN=(1x11008x4096)         |      47.0      |           15.6          |           15.5          |                       |         49.5       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=64, MKN=(1x11008x4096)         |      47.0      |           14.9          |           14.7          |                       |         49.5       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4b8, g=128, MKN=(1x11008x4096)        |      47.3      |           14.7          |           14.5          |          14.4         |         49.4       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=-1, MKN=(1x11008x4096)       |      46.9      |           18.8          |           18.7          |          14.8         |         86.7       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=32, MKN=(1x11008x4096)       |      47.3      |           20.1          |           20.0          |                       |         86.9       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=64, MKN=(1x11008x4096)       |      47.3      |           19.3          |           19.2          |                       |         83.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint8b128, g=128, MKN=(1x11008x4096)      |      46.9      |           19.1          |           19.0          |          15.5         |         85.0       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float8_e4m3fn, g=-1, MKN=(1x11008x4096)   |      47.1      |           19.6          |           19.5          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float8_e4m3fn, g=128, MKN=(1x11008x4096)  |      47.4      |           19.2          |           19.1          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=float4_e2m1f, g=16, MKN=(1x11008x4096)    |      49.9      |           17.0          |           17.0          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=-1, MKN=(1x11008x4096)           |      47.3      |           13.8          |           13.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=32, MKN=(1x11008x4096)           |      47.3      |           17.4          |           17.2          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=64, MKN=(1x11008x4096)           |      47.3      |           15.7          |           15.6          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=False, q=uint4, g=128, MKN=(1x11008x4096)          |      47.3      |           15.8          |           15.6          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=-1, MKN=(1x11008x4096)          |      46.9      |           13.8          |           13.6          |          13.5         |         49.3       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=32, MKN=(1x11008x4096)          |      47.3      |           15.5          |           15.5          |                       |         49.5       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=64, MKN=(1x11008x4096)          |      47.3      |           14.8          |           14.7          |                       |         49.5       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4b8, g=128, MKN=(1x11008x4096)         |      46.9      |           14.7          |           14.6          |          14.4         |         49.5       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=-1, MKN=(1x11008x4096)        |      47.3      |           19.5          |           19.4          |          14.8         |         83.8       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=32, MKN=(1x11008x4096)        |      47.0      |           19.9          |           19.7          |                       |         83.4       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=64, MKN=(1x11008x4096)        |      47.0      |           19.0          |           18.8          |                       |         86.5       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint8b128, g=128, MKN=(1x11008x4096)       |      47.3      |           19.0          |           18.8          |          15.6         |         86.3       
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float8_e4m3fn, g=-1, MKN=(1x11008x4096)    |      47.4      |           19.4          |           19.3          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float8_e4m3fn, g=128, MKN=(1x11008x4096)   |      47.0      |           19.2          |           19.2          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=float4_e2m1f, g=16, MKN=(1x11008x4096)     |      50.4      |           17.1          |           17.0          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=-1, MKN=(1x11008x4096)            |      46.9      |           13.8          |           13.7          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=32, MKN=(1x11008x4096)            |      47.0      |           17.5          |           17.4          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=64, MKN=(1x11008x4096)            |      47.0      |           15.6          |           15.5          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=False k_full=True, q=uint4, g=128, MKN=(1x11008x4096)           |      47.0      |           15.9          |           15.8          |                       |                    
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=32, MKN=(1x11008x4096)          |      47.4      |           50.2          |           50.2          |                       |         80.2       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=64, MKN=(1x11008x4096)          |      47.4      |           40.0          |           39.8          |                       |         80.1       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint4b8, g=128, MKN=(1x11008x4096)         |      47.0      |           33.4          |           33.3          |          14.4         |         80.2       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=32, MKN=(1x11008x4096)        |      47.4      |           69.4          |           69.2          |                       |         94.0       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=64, MKN=(1x11008x4096)        |      47.4      |           52.4          |           52.3          |                       |         93.9       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=False, q=uint8b128, g=128, MKN=(1x11008x4096)       |      47.0      |           42.6          |           42.8          |          15.4         |         93.4       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=32, MKN=(1x11008x4096)           |      47.4      |           21.5          |           21.4          |                       |         80.2       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=64, MKN=(1x11008x4096)           |      47.3      |           20.8          |           20.7          |                       |         80.3       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint4b8, g=128, MKN=(1x11008x4096)          |      47.0      |           20.5          |           20.3          |          14.4         |         80.3       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=32, MKN=(1x11008x4096)         |      47.4      |           30.1          |           30.0          |                       |         94.6       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=64, MKN=(1x11008x4096)         |      47.4      |           29.3          |           29.2          |                       |         93.8       
      meta-llama/Llama-2-7b-hf/TP1, act=True k_full=True, q=uint8b128, g=128, MKN=(1x11008x4096)        |      47.1      |           28.2          |           28.1          |          15.4         |         93.0       

Times are in microseconds (us).

(Optional) Documentation Update

22quinn added 2 commits June 20, 2025 12:36
@22quinn
play
20d2876 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn
update
45ee838 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jun 21, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @22quinn, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on enhancing the benchmark_marlin.py script. My primary goal was to fix the existing benchmark, which had become outdated due to recent updates in Marlin operations. Additionally, I've expanded its capabilities to test a broader range of quantization data types, including FP4 and FP8, ensuring a more robust and relevant performance assessment for Marlin kernels.

Highlights

  • Benchmark Script Fixes: I've updated the benchmark_marlin.py script to align with recent changes in Marlin operations, addressing a bug that caused the previous benchmark to break. This ensures the benchmark accurately reflects the current state of Marlin performance.
  • Expanded Quantization Type Coverage: The benchmark now includes support for additional quantization data types, specifically float4_e2m1f (FP4) and float8_e4m3fn (FP8). This provides a more comprehensive performance evaluation across various quantization schemes.
  • Improved Quantization Logic Handling: I've refined the bench_run function to correctly handle different Marlin quantization scenarios, including those with zero-points (uint4, uint8), varying group sizes, and activation order. This involves conditional execution paths for different quantization methods (e.g., AWQ Marlin, Marlin_24, GPTQ repack) and updated function signatures for gptq_marlin_gemm.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jun 21, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request fixes and extends the Marlin benchmark script by addressing outdated logic and expanding test coverage to include FP4, FP8, and AWQ-style quantization. The changes ensure that only supported configurations are benchmarked. I've provided suggestions to improve code clarity and reduce redundancy. Overall, this is a solid contribution.

@22quinn
update
29c32ad 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn 22quinn changed the title [Bugfix][Benchmarks] Fix Marlin benchmark [Bugfix][Benchmark] Fix Marlin benchmark Jun 21, 2025
houseroad
houseroad reviewed Jun 21, 2025
View reviewed changes
benchmarks/kernels/benchmark_marlin.py Outdated
)

marlin_zp = torch.empty(0, dtype=torch.int, device=b.device)
marlin_24_w_ref = None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: shall we just move this to a dedicated function to improve the code readability?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactored a bit

22quinn added 2 commits June 21, 2025 21:26
@22quinn
refactor
46d1684 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn
refactor
bd66e86 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn 22quinn added perf-benchmarks ready ONLY add when PR is ready to merge/full CI is needed labels Jun 22, 2025
@22quinn 22quinn mentioned this pull request Jun 22, 2025
Merged
4 tasks
@mergify mergify bot added the performance Performance-related issues label Jun 22, 2025
@aarnphm
Copy link
Collaborator

aarnphm commented Jun 22, 2025

Will wait for @mgoin to take a look

mgoin
mgoin approved these changes Jun 23, 2025
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really appreciate this, thanks

@mgoin mgoin merged commit 4671ac6 into vllm-project:main Jun 23, 2025
57 checks passed
gmarinho2 pushed a commit to gmarinho2/vllm that referenced this pull request Jun 26, 2025
@22quinn @gmarinho2
[Bugfix][Benchmark] Fix Marlin benchmark (vllm-project#19929)
36f1f8b
xjpang pushed a commit to xjpang/vllm that referenced this pull request Jun 30, 2025
@22quinn
[Bugfix][Benchmark] Fix Marlin benchmark (vllm-project#19929)
f5692d1
wseaton pushed a commit to wseaton/vllm that referenced this pull request Jun 30, 2025
@22quinn
[Bugfix][Benchmark] Fix Marlin benchmark (vllm-project#19929)
b3a841c 
Signed-off-by: Will Eaton <weaton@redhat.com>
wseaton pushed a commit to wseaton/vllm that referenced this pull request Jun 30, 2025
@22quinn
[Bugfix][Benchmark] Fix Marlin benchmark (vllm-project#19929)
faca98b
wwl2755-google pushed a commit to wwl2755-google/vllm that referenced this pull request Jul 1, 2025
@22quinn @wwl2755-google
[Bugfix][Benchmark] Fix Marlin benchmark (vllm-project#19929)
46cf59e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@houseroad houseroad

@gemini-code-assist gemini-code-assist[bot]

@mgoin mgoin

@aarnphm aarnphm

@LucasWilkinson LucasWilkinson

@alexm-redhat alexm-redhat

Assignees
No one assigned
Labels
perf-benchmarks performance Performance-related issues ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: file in vllm/benchmarks/kernels/benchmark_marlin.py cannot execute
4 participants
@22quinn @aarnphm @mgoin @houseroad