[Usage]: Unreasonable value of token per image for multi-modal memory profiling

[marcmk6]

Your current environment

Run the official demo script for using Qwen 2.5 VL 3B.
When specifying limit_mm_per_prompt={"image": 1}, I got

WARNING 06-26 07:12:07 [profiling.py:245] The sequence length used for profiling (max_num_batched_tokens / max_num_seqs = 5120) is too short to hold the multi-modal embeddings in the worst case (32768 tokens in total, out of which {'image': 16384, 'video': 16384} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.

Where does this (approx.) 16384 tokens per image come from??
If I remember correctly, Qwen 2.5 VL uses 28*28 patch for an image token. 16384*28*28 is an image 3584*3584 image. This is an extremely unreasonbaly large size for VL LLM use cases.
This very large hard-coded value blocks me from using multiple images as it will cause OOM during memory profiling. But it's acutally ok during inference since I can resize my images.
How can I set it by myself?

How would you like to use vllm

either a way to completely disable memory profiling, or a way to sepcify a reasonable value of tokens per image

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[marcmk6]

The result reported in the main post was with v0 engine. Now with v1 engine, it seems to profile with 1 image (16384 tokens) irrelevant to limit_mm_per_prompt={"image": x}. with x≥1. But still, it's 16384.

[DarkLight1337]

You can set the HF processing kwargs (refer to the example script) to limit the max possible image size

[marcmk6]

With kwargs

engine_args = EngineArgs(
        model=MODEL_NAME,
        max_model_len=2048,
        max_num_seqs=5,
        limit_mm_per_prompt={"image": 16},
        dtype="bfloat16",
        gpu_memory_utilization=0.5 * 0.75,
        mm_processor_kwargs={
            "min_pixels": 28 * 28,
            "max_pixels": 32 * 28 * 28,
        },
    )

I got

Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 82 video items of the maximum feature size.

Without kwargs

engine_args = EngineArgs(
        model=MODEL_NAME,
        max_model_len=2048,
        max_num_seqs=5,
        limit_mm_per_prompt={"image": 16},
        dtype="bfloat16",
        gpu_memory_utilization=0.5 * 0.75,
    )

I got

Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.

I still can't figure out where do the two numbers 16384 and 82 come from

[DarkLight1337]

The token counts are based on the dummy data calculated here:

vllm/vllm/model_executor/models/qwen2_vl.py

Lines 991 to 1015 in 2c5302f

	def get_dummy_mm_data(
	self,
	seq_len: int,
	mm_counts: Mapping[str, int],
	) -> MultiModalDataDict:
	num_images = mm_counts.get("image", 0)
	num_videos = mm_counts.get("video", 0)
	target_width, target_height = \
	self.info.get_image_size_with_most_features()
	target_num_frames = \
	self.info.get_num_frames_with_most_features(seq_len, mm_counts)
	return {
	"image":
	self._get_dummy_images(width=target_width,
	height=target_height,
	num_images=num_images),
	"video":
	self._get_dummy_videos(
	width=target_width,
	height=target_height,
	num_frames=target_num_frames,
	num_videos=num_videos,
	)

The dummy data is intentionally constructed to take up as much memory as possible given the processor constraints so users can't crash the engine during inference by sending adversarial inputs that exceed the expected memory usage.

[marcmk6]

got it, thanks! close this for now