[Multi Modal] Configurable MM Profiling

[wwl2755]

Fix #14438

LAST EDIT: 09/26

1. Basic idea to make the profiling mm data to be more configurable.

main:
limit_mm_per_prompt={"image": 1, "video": 1}

PR:
limit_mm_per_prompt={"image": {"count": 1, "width": 512, "height": 512}}

2. Backward compatibility is provided. Fix format is also supported.

limit_mm_per_prompt={"image": {"count": 1, "width": 512, "height": 512}, "video": 0}

3. For image, we can specify count, width, height.
   For video, we can sepcify count, width, height, num_frames.
   For audio, we can specify count, length.

Known issues/TODOs:

• Update related doc and tests
• Only tested with image, will test on videos and audio soon
• The width and height are used directly without resize/reshape (not sure if it is good)
• Currently the change is only applied to qwen2_vl, need to adapt to other models

Test

Tested on A100-40GB

#main
vllm serve Qwen/Qwen2.5-VL-3B-Instruct
Available KV cache memory: 5.26 GiB

#PR
vllm serve Qwen/Qwen2.5-VL-3B-Instruct
Available KV cache memory: 5.26 GiB

vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--limit-mm-per-prompt '{"image": 1}'
Available KV cache memory: 5.26 GiB

vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--limit-mm-per-prompt '{"image": 1, "video": 0}'
Available KV cache memory: 25.05 GiB

vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--limit-mm-per-prompt.image 1 --limit-mm-per-prompt.video 0
Available KV cache memory: 25.05 GiB

vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--limit-mm-per-prompt '{"image": 0, "video": 0}'
Available KV cache memory: 29.08 GiB

vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--limit-mm-per-prompt '{"image": {"count": 1, "width": 512, "height": 512}, "video": 0}'
Available KV cache memory: 27.82 GiB

vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--limit-mm-per-prompt '{"image": {"count": 1, "width": 5120, "height": 5120}, "video": 0}'
Available KV cache memory: 25.05 GiB

vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--limit-mm-per-prompt '{"image": {"count": 1, "width": 5120, "height": 5120}, "video": {"count": 1, "num_frames": 5, "width": 5120, "height": 5120}}'
Available KV cache memory: 18.45 GiB

#audio
vllm serve Qwen/Qwen2-Audio-7B-Instruct
Available KV cache memory: 19.30 GiB

vllm serve Qwen/Qwen2-Audio-7B-Instruct \
--limit-mm-per-prompt '{"image": 0, "video": 0, "audio": {"count": 1, "length": 1000}}'
Available KV cache memory: 19.31 GiB

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wwl2755.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[gemini-code-assist]

Code Review

This pull request introduces a valuable feature for making multimodal profiling more configurable, allowing for more precise memory allocation. The changes, which include new dataclasses for modality options and updated configuration parsing, are well-structured and maintain backward compatibility. However, I've identified a critical issue with the use of overly broad exception handling in the profiling logic. This could mask underlying bugs and lead to incorrect memory profiling, potentially causing runtime errors. My review includes specific suggestions to address this by narrowing the exception scope and improving debuggability.

[DarkLight1337]

Thanks for working on this, can you fix the merge conflicts?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wwl2755.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

Feel free to update each model now

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wwl2755.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

/gemini review

[DarkLight1337]

Thanks, I'm waiting for upstream CI to be fixed before adding ready label

[DarkLight1337]

Can you merge from main again to get the CI running?

[DarkLight1337]

Tests pass so LGTM, thanks!