Support serving DeepSeek-R1-Channel-INT8 with 32 L40S.

[solrex]

Motivation

Add L40S support for serving channel quant model: meituan/DeepSeek-R1-Channel-INT8.

Modifications

• Modify extend attention BLOCK_M/N for sm_89; Fix test/srt/test_triton_attention_kernels.py test failure on L40S/L20.
• Add fused_moe_triton config for DSv3 & L40S.
• Modify int8 gemm kernel to support sm_89 gemm according to vllm implemetation.
• Add int8 gemm test case for Ms in 16, 32, 64. Because the main branch ./test_int8_gemm.py fails after M=64, N=4096, K=16384 on L40S/L20.

$ python3 test/srt/test_triton_attention_kernels.py 
...
======================================================================
ERROR: test_extend_attention (__main__.TestTritonAttention)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sgl-workspace/sglang/test/srt/test_triton_attention_kernels.py", line 178, in test_extend_attention
    self._test_extend_attention_once(19, 12331, 12, 4, value)
  File "/sgl-workspace/sglang/test/srt/test_triton_attention_kernels.py", line 141, in _test_extend_attention_once
    extend_attention_fwd(
  File "/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 356, in extend_attention_fwd
    _fwd_kernel[grid](
  File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 345, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 691, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
  File "/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py", line 381, in __getattribute__
    self._init_handles()
  File "/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py", line 374, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 102400, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

----------------------------------------------------------------------
Ran 4 tests in 37.889s

FAILED (errors=1)

$ python tests/test_int8_gemm.py
...
M=64, N=4096, K=4096, with_bias=False, out_dtype=torch.bfloat16: OK
M=64, N=4096, K=8192, with_bias=True, out_dtype=torch.float16: OK
M=64, N=4096, K=8192, with_bias=True, out_dtype=torch.bfloat16: OK
M=64, N=4096, K=8192, with_bias=False, out_dtype=torch.float16: OK
M=64, N=4096, K=8192, with_bias=False, out_dtype=torch.bfloat16: OK
M=64, N=4096, K=16384, with_bias=True, out_dtype=torch.float16: OK
M=64, N=4096, K=16384, with_bias=True, out_dtype=torch.bfloat16: OK
M=64, N=4096, K=16384, with_bias=False, out_dtype=torch.float16: OK
M=64, N=4096, K=16384, with_bias=False, out_dtype=torch.bfloat16: OK
E
======================================================================
ERROR: test_accuracy (__main__.TestInt8Gemm.test_accuracy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/sglang/sgl-kernel/tests/test_int8_gemm.py", line 50, in test_accuracy
    self._test_accuracy_once(
  File "/workspace/sglang/sgl-kernel/tests/test_int8_gemm.py", line 32, in _test_accuracy_once
    o = int8_scaled_mm(a, b, scale_a, scale_b, out_dtype, bias)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/sgl_kernel/gemm.py", line 14, in int8_scaled_mm
    return torch.ops.sgl_kernel.int8_scaled_mm(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/_ops.py", line 1116, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: gemm executioin failed, error: Error Internal

----------------------------------------------------------------------
Ran 1 test in 2.133s

FAILED (errors=1)

Benchmark result

$ python3 -m sglang.bench_serving \
    --dataset-path /workspace/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json \
    --dataset-name random  --random-input 128 --random-output 128 --num-prompts 1000 \
    --request-rate 64 --random-range-ratio 1.0

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    64.0      
Max reqeuest concurrency:                not set   
Successful requests:                     1000      
Benchmark duration (s):                  254.92    
Total input tokens:                      128000    
Total generated tokens:                  128000    
Total generated tokens (retokenized):    127745    
Request throughput (req/s):              3.92      
Input token throughput (tok/s):          502.12    
Output token throughput (tok/s):         502.12    
Total token throughput (tok/s):          1004.24   
Concurrency:                             786.40    
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   200468.88 
Median E2E Latency (ms):                 194789.87 
---------------Time to First Token----------------
Mean TTFT (ms):                          56819.31  
Median TTFT (ms):                        52114.21  
P99 TTFT (ms):                           197165.86 
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1131.19   
Median ITL (ms):                         840.09    
P95 ITL (ms):                            1101.30   
P99 ITL (ms):                            1331.69   
Max ITL (ms):                            90948.58  
==================================================

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhaochenyang20]

@solrex great work! We do not have L40s, I will find someone to check this locally.

[zhyncs]

It's cool! Thanks for your contribution. Please join the https://slack.sglang.ai!

[HandH1998]

Thanks for your contribution! LGTM, only need to provide the benchmark results of int8_gemm in sgl_kernel before and after.

[zhaochenyang20]

@ispobock @yizhang2077 could you plz take a look? Lianmin said it used the features from yours.

[zhyncs]

Thanks! @solrex

[zhyncs]

FYI @solrex It has been merged in latest main #4522