perf: Improve vLLM backend performance by using a separate thread for responses

[Tabrizian]

What does the PR do?

Triton's output token throughput for generate endpoint increases by 18% for concurrency 50. There is still a small gap between vLLM-only and vLLM + Triton solution.

The model is llama-2-7b.

Changes:

• Use add_request instead of generate for vLLM backend. It appears that with generate , the token generation is delayed until the next iteration of the loop.
• Delegate response sending to a separate thread.

Next steps:

• In the long term it might be better to have an async_send API to avoid creating a separate thread for sending the responses.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

N/A

Where should the reviewer start?

N/A

Test plan:

This is a performance improvement, existing test cases should be sufficient at covering any possible issues.

• CI Pipeline ID: 16917857 16926866

Caveats:

N/A

Background

N/A

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

[tanmayv25]

@Tabrizian Can you add description for the code changes in the PR? Also including the performance improvement you observed and in what cases?

[Tabrizian]

@tanmayv25 I updated the PR description.

[oandreeva-nv]

@kthui So you've received close to Iman's perf results after sync?

[kthui]

@kthui So you've received close to Iman's perf results after sync?

yes

[oandreeva-nv]

LGTM!

[zhaotyer]

I test Qwen2-7B-chat whith a A100*80G，The pr did not work, The gap compared to vllm deployments is nearly 40% for concurrency 64

vllm

============ Serving Benchmark Result ============
Successful requests:                     64        
Benchmark duration (s):                  15.35     
Total input tokens:                      14970     
Total generated tokens:                  15304     
Request throughput (req/s):              4.17      
Input token throughput (tok/s):          975.53    
Output token throughput (tok/s):         997.30    
---------------Time to First Token----------------
Mean TTFT (ms):                          1222.87   
Median TTFT (ms):                        1168.27   
P99 TTFT (ms):                           2284.71   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          55.58     
Median TPOT (ms):                        55.96     
P99 TPOT (ms):                           56.88     
---------------Inter-token Latency----------------
Mean ITL (ms):                           53.51     
Median ITL (ms):                         51.57     
P99 ITL (ms):                            78.99     
==================================================

triton+llm+stream

============ Serving Benchmark Result ============
Successful requests:                     64        
Benchmark duration (s):                  24.85     
Total input tokens:                      14970     
Total generated tokens:                  15178     
Request throughput (req/s):              2.58      
Input token throughput (tok/s):          602.44    
Output token throughput (tok/s):         610.81    
---------------Time to First Token----------------
Mean TTFT (ms):                          1762.53   
Median TTFT (ms):                        1687.10   
P99 TTFT (ms):                           3342.62   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          91.16     
Median TPOT (ms):                        91.76     
P99 TPOT (ms):                           93.63     
---------------Inter-token Latency----------------
Mean ITL (ms):                           138.93    
Median ITL (ms):                         93.49     
P99 ITL (ms):                            368.58    
==================================================

triton+llm+no stream

============ Serving Benchmark Result ============
Successful requests:                     64        
Benchmark duration (s):                  18.48
Total input tokens:                      14784     
Total generated tokens:                  15232
Request throughput (req/s):              3.46
Input token throughput (tok/s):          801.91
Output token throughput (tok/s):         826.32

[zhaotyer]

I test Qwen2-7B-chat whith a A100*80G，The pr did not work, The gap compared to vllm deployments is nearly 40% for concurrency 64

vllm

============ Serving Benchmark Result ============
Successful requests:                     64        
Benchmark duration (s):                  15.35     
Total input tokens:                      14970     
Total generated tokens:                  15304     
Request throughput (req/s):              4.17      
Input token throughput (tok/s):          975.53    
Output token throughput (tok/s):         997.30    
---------------Time to First Token----------------
Mean TTFT (ms):                          1222.87   
Median TTFT (ms):                        1168.27   
P99 TTFT (ms):                           2284.71   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          55.58     
Median TPOT (ms):                        55.96     
P99 TPOT (ms):                           56.88     
---------------Inter-token Latency----------------
Mean ITL (ms):                           53.51     
Median ITL (ms):                         51.57     
P99 ITL (ms):                            78.99     
==================================================

triton+llm+stream

============ Serving Benchmark Result ============
Successful requests:                     64        
Benchmark duration (s):                  24.85     
Total input tokens:                      14970     
Total generated tokens:                  15178     
Request throughput (req/s):              2.58      
Input token throughput (tok/s):          602.44    
Output token throughput (tok/s):         610.81    
---------------Time to First Token----------------
Mean TTFT (ms):                          1762.53   
Median TTFT (ms):                        1687.10   
P99 TTFT (ms):                           3342.62   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          91.16     
Median TPOT (ms):                        91.76     
P99 TPOT (ms):                           93.63     
---------------Inter-token Latency----------------
Mean ITL (ms):                           138.93    
Median ITL (ms):                         93.49     
P99 ITL (ms):                            368.58    
==================================================

triton+llm+no stream

============ Serving Benchmark Result ============
Successful requests:                     64        
Benchmark duration (s):                  18.48
Total input tokens:                      14784     
Total generated tokens:                  15232
Request throughput (req/s):              3.46
Input token throughput (tok/s):          801.91
Output token throughput (tok/s):         826.32

llm+triton gpu utilization is significantly lower than vllm