[Usage]: Can vllm multimodal generate use preprocessed image?

[pjgao]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

When using the generate method in vLLM for inference with multimodal data, we can directly pass Image objects, but is there an interface that allows us to directly pass preprocessed pixel_values and image_grid_thw (generated by a processor) instead, to avoid performing image preprocessing within vLLM?

MODEL_PATH = "Qwen2-VL-7B-Instruct/"
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "./demo.jpeg",
                "min_pixels": 224 * 224,
                "max_pixels": 1280 * 28 * 28,
            },
            {"type": "text", "text": "describe this picture?"},
        ],
    },
]
from transformers import AutoProcessor, AutoTokenizer
prompt = AutoProcessor.from_pretrained(MODEL_PATH).apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
from qwen_vl_utils import process_vision_info
image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)

llm_inputs = {
    "prompt_token_ids": prompt_token_ids,
    "multi_modal_data":  {"image": image_inputs},
    "mm_processor_kwargs": video_kwargs,
}
outputs = llm.generate(prompts=[llm_inputs], sampling_params=sampling_params)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

You will have to define your own multi-modal processor to handle that. See #14281

[pjgao]

You will have to define your own multi-modal processor to handle that. See #14281

Thanks for your kindly reply, I will check it❤️