[Performance]: Multi-Modal Benchmark on NVIDIA A100 – Qwen2.5-VL / MiniCPM-V-4 / InternVL3_5-4B / InternVL3_5-2B

[player0718]

Proposal to improve performance

Performance Bottleneck in Video Inference

My tests indicate a significant performance bottleneck when processing multi-frame video inputs with vLLM.

• High Time-to-First-Token (TTFT): In the Prefill Stage, vLLM encodes all frames of the entire video before generating the first text token. For long videos, this results in an extremely high Time-to-First-Token (TTFT) and a significant peak GPU memory footprint[cite: 9].
• CPU Bottleneck & GPU Underutilization: Test results on an NVIDIA A100 GPU show that when processing video, CPU utilization can spike to 100%, while GPU utilization remains at only about 20%. This indicates that video frame decoding and ViT feature extraction on the CPU side become bottlenecks, leading to idle GPU computing.
• Performance Comparison (Qwen2.5-VL-7B on 1xA100):
  • Single Image Input: Generates responses at an average speed of approximately 132 tokens/s.
  • Multi-frame (~30) Video Input: The average generation speed drops to only about 10 tokens/s.

Proposal: Streaming/Chunk-level Vision Encoding

To address this issue, I propose implementing Streaming or Chunk-level Vision Encoding. Instead of encoding the entire video at once during the prefill stage, vLLM could process the video in smaller chunks or as a stream. This would:

1. Significantly reduce the Time-to-First-Token (TTFT).
2. Lower the peak GPU memory requirement.
3. Allow for better pipelining between CPU-based video decoding and GPU-based inference, improving overall hardware utilization.

This would make vLLM more viable for real-time long-video analysis and interactive applications.

Report of performance regression

N/A. This is not a regression from a previous vLLM release but rather new benchmarking data for previously untested multi-modal video workloads.

Misc discussion on performance

I have conducted a comprehensive benchmark of four popular multi-modal models using the vllm bench toolkit. The goal was to evaluate and compare their inference performance for both image and video inputs as a baseline so that future changes can be compared against them.

Hardware / Environment: single NVIDIA A100 40 GB, CUDA 12.x, vLLM latest main .
Datasets: ShareGPT4Video and ShareGPT4V images.
Benchmark commands:

vllm serve Qwen/Qwen2.5-VL-7B-Instruct  --allowed-local-media-path pathToData/datasets/ShareGPT4Video --port 8000  --dtype bfloat16 

vllm bench serve --backend openai-chat --model Qwen/Qwen2.5-VL-7B-Instruct  --dataset-name sharegpt  --dataset-path pathToData/datasets/ShareGPT4Video/llava_v1_5_mix665k_with_video_chatgpt72k_share4video28.json   --num-prompts 50 --request-rate inf --max-concurrency 1 

The tested models are:

• Qwen/Qwen2.5-VL-7B-Instruct
• openbmb/MiniCPM-V-4
• OpenGVLab/InternVL3_5-4B
• OpenGVLab/InternVL3_5-2B

1. Performance with Unlimited Request Rate (request-rate=inf)

This test sends all requests instantaneously to measure the maximum Queries Per Second (QPS) limit.

Image Input Performance

Metric	Qwen/Qwen2.5-VL-7B-Instruct	openbmb/MiniCPM-V-4	OpenGVLab/InternVL3_5-4B	OpenGVLab/InternVL3_5-2B
Request Throughput (req/s)	13.24	17.59	15.61	26.26
Output Token Throughput (tok/s)	1517.21	2320.44	2674.70	4734.68
Mean TTFT (ms)	130.48	97.31	98.98	99.80
Mean TPOT (ms)	15.45	9.93	11.57	6.57

Video Input Performance

Metric	Qwen/Qwen2.5-VL-7B-Instruct	openbmb/MiniCPM-V-4	OpenGVLab/InternVL3_5-4B	OpenGVLab/InternVL3_5-2B
Request Throughput (req/s)	5.19	8.24	7.18	11.82
Output Token Throughput (tok/s)	1795.85	1272.71	2120.87	4143.84
Mean TTFT (ms)	95.16	115.24	104.24	101.77
Mean TPOT (ms)	15.67	12.18	11.18	6.74

2. Performance with Fixed Concurrency (max-concurrency = 1, 10, 50)

This test evaluates the system's performance under controlled, increasing levels of concurrent requests.

Image Input - Fixed Concurrency

Model	Concurrency	Req/s	Output tok/s	Mean TTFT (ms)	Mean TPOT (ms)
Qwen/Qwen2.5-VL-7B	10	6.08	713.45	28.28	13.69
	50	20.89	2451.80	58.31	16.36
openbmb/MiniCPM-V-4	1	1.07	126.22	15.16	7.88
	10	9.52	1114.85	19.11	8.64
	50	32.73	3822.97	51.44	10.54
OpenGVLab/InternVL3.5-2B	1	1.09	202.74	11.12	4.89
	10	9.13	1691.45	17.53	5.66
	50	31.93	5912.53	48.34	6.65

Video Input - Fixed Concurrency

Model	Concurrency	Req/s	Output tok/s	Mean TTFT (ms)	Mean TPOT (ms)
Qwen/Qwen2.5-VL-7B	1	0.21	76.46	21.98	13.04
	10	1.98	709.00	28.54	13.77
	50	7.35	2632.88	63.79	16.25
openbmb/MiniCPM-V-4	1	0.69	126.05	14.95	7.89
	10	6.04	1065.36	22.33	8.67
	50	18.95	3239.55	65.85	10.95
OpenGVLab/InternVL3.5-2B	1	0.59	199.48	13.55	4.97
	10	4.72	1683.27	18.37	5.80
	50	17.40	6016.16	48.43	7.00

Conclusion

1. Throughput Scaling: For all models, increasing concurrency significantly boosts request throughput (req/s) and output token throughput (tok/s). This demonstrates vLLM's strong ability to handle high-concurrency scenarios, likely due to its continuous batching innovation.
2. Latency vs. Concurrency Trade-off: As expected, Mean Time to First Token (TTFT) increases with higher concurrency. This is a classic trade-off where the system achieves higher overall throughput at the cost of slightly longer wait times for individual requests.
3. Model Size Impact: Smaller models like OpenGVLab/InternVL3.5-2B consistently deliver the highest throughput and lowest per-token latency (TPOT) across all concurrency levels, making them highly efficient choices.
4. Input Modality Impact: Video inputs consistently result in lower request throughput compared to image inputs at the same concurrency level. For instance, at a concurrency of 50, Qwen/Qwen2.5-VL-7B handles 20.89 req/s for images but only 7.35 req/s for videos, highlighting the significant computational overhead of video processing.
5. Overall Suitability: For production environments, vLLM is suitable for short video clips where the initial latency is acceptable. For long videos, the high latency makes it better suited for offline, asynchronous tasks.

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

• This indicates that video frame decoding and ViT feature extraction on the CPU side become bottlenecks, leading to idle GPU computing.

+1

I also found this problem

[david6666666]

+1

cc @ywang96 @DarkLight1337 @Isotr0py

[ywang96]

Hey @player0718! Thanks for all the benchmarking efforts and analysis. Long video inference bottleneck is definitely what we have been aware of and streaming video frames as input is something on our mind!

There have been quite a few people who are interested in improving this. We're very happy to collaborate!

[noooop]

I will further test the Multi-Modal CPU bottleneck issue when I have time.

[wwl2755]

Thanks so much for this.

This indicates that video frame decoding and ViT feature extraction on the CPU side become bottlenecks, leading to idle GPU computing.

+1. I also observed the CPU bottleneck recently and was hoping torch.compile can largely help.

I will further test the Multi-Modal CPU bottleneck issue when I have time.

That would be nice. Thanks for that!

[knlnguyen1802]

@player0718 Thanks for raise the question.
I have found the problem and same idea also propose here #22695.
But I do not have time to complete it and I can share some of result on it.
I found that for LLaVa architechture model, we can do chunk vision encoding, and need to modify the implement of RoPE (or MRoPE) for each model and there is no acuracy loss.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!