Qwen2.5Omni Repeatedly Calculates Audio/Video/Image Features

[CharlesGong12]

System Info

• transformers version: 4.54.1
• Platform: Linux-5.10.134-16.3.al8.x86_64-x86_64-with-glibc2.39
• Python version: 3.10.15
• Huggingface_hub version: 0.34.3
• Safetensors version: 0.5.3
• Accelerate version: 1.7.0
• Accelerate config: not found
• DeepSpeed version: 0.16.3
• PyTorch version (accelerator?): 2.6.0+cu126 (CUDA)
• Tensorflow version (GPU?): 2.16.2 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

My code is:

model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
args.model_name_or_path,
    torch_dtype=torch.bfloat16,
#   attn_implementation="flash_attention_2",
    attn_implementation="sdpa",
#   attn_implementation="eager",
    device_map="auto"
    )

conversation = [
    {
        'role': 'system',
        'content': [
            {'type': 'text', 'text': 'You are a helpful assistant.'}
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "xxx.mp4", 'max_frames': 80, "max_pixels": 50176},
            {"type": "text", "text": "xxx"}
        ],
    }
]

text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios, images, videos = process_mm_info(conversation, use_audio_in_video=USE_AUDIO_IN_VIDEO)
inputs = processor(text=text, audio=audios, images=images, videos=videos, return_tensors="pt", padding=True, use_audio_in_video=USE_AUDIO_IN_VIDEO)
inputs = inputs.to(model.device).to(model.dtype)

# Inference: Generation of the output text and audio
text_ids = model.generate(**inputs, use_audio_in_video=USE_AUDIO_IN_VIDEO, return_audio=False, thinker_max_new_tokens=100)

text = processor.batch_decode(text_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)

Expected behavior

In the qwen2.5-omni code, Qwen2_5OmniThinkerForConditionalGeneration repeatedly calculates the feature of audio/video/image input for each text token generated. Does this cause significant redundancy? As shown here, this code segment does not check whether it's in the prefill phase. Consequently, every time a text token is generated, the forward function computes the same audio/image/video feature as in previous token generation phases.

[zucchini-nlp]

@CharlesGong12 When using generate() we skip multimodal inputs after pre-fill phase and the merging/encoding is not called. You can see it here

transformers/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py

Lines 2029 to 2032 in ff8b88a

	if cache_position[0] != 0:
	model_inputs["pixel_values"] = None
	model_inputs["pixel_values_videos"] = None
	model_inputs["input_features"] = None

If you are doing any manual generation loop with forward calls, it is recommended to not pass multimodal inputs in decoding phase as long as the model is generating with cache

[CharlesGong12]

@zucchini-nlp Hi, thanks for your reply! I notice the code you mentioned and print pixel_values, pixel_values_videos, input_features in Qwen2_5OmniThinkerForConditionalGeneration, by adding print(f'{input_features is not None}, {pixel_values is not None}, {pixel_values_videos is not None}') here of 4.54.1. I also print cache_position in prepare_inputs_for_generation. However, the output is:

cache_position: tensor([   0,    1,    2,  ..., 1365, 1366, 1367], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1366, 1367, 1368], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1367, 1368, 1369], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1368, 1369, 1370], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1369, 1370, 1371], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1370, 1371, 1372], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1371, 1372, 1373], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1372, 1373, 1374], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1373, 1374, 1375], device='cuda:0')
True, False, True
cache_position: tensor([   0,    1,    2,  ..., 1374, 1375, 1376], device='cuda:0')
True, False, True

As you can see, cache_position[0] is always 0 and thus the multimodal encoding will be repeatedly called.

[zucchini-nlp]

make sure you are generating with cache, otherwise there is no way for us to save pre-computed features. In generate pass in use_cache=True explicitly, as it seems that model is generating in no cache mode

[CharlesGong12]

I add use_cache=True here, in thinker_kwargs of self.thinker.generate. However, an error occurs:

Traceback (most recent call last):
  File "xxx/tools/hf_infer.py", line 228, in <module>
    text_ids = model.generate(**inputs, use_audio_in_video=USE_AUDIO_IN_VIDEO, return_audio=False, thinker_max_new_tokens=5)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 3879, in generate
    thinker_result = self.thinker.generate(input_ids=input_ids, **thinker_kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/generation/utils.py", line 2633, in generate
    result = self._sample(
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/generation/utils.py", line 3617, in _sample
    outputs = model_forward(**model_inputs, return_dict=True)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 1951, in forward
    outputs = self.model(
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 1638, in forward
    layer_outputs = decoder_layer(
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__
    return super().__call__(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/accelerate/hooks.py", line 175, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 1485, in forward
    hidden_states, self_attn_weights = self.self_attn(
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/accelerate/hooks.py", line 175, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 1397, in forward
    attn_output, attn_weights = attention_interface(
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
RuntimeError: q must have shape (batch_size, seqlen_q, num_heads, head_size_og)

If I do not add use_cache=True, the inference is normal but the Audio/Video/Image Features are repeatedly encoded.

[zucchini-nlp]

If I do not add use_cache=True, the inference is normal but the Audio/Video/Image Features are repeatedly encoded.

This is expected and we don't have a way to save precomputed audio/video without a cache object

I add use_cache=True here, in thinker_kwargs of self.thinker.generate. However, an error occurs:

Can you try just passing use_cache=True in generate()? It definitely works and I didn't see issues the last time

[CharlesGong12]

Thanks for your reply. To confirm, I passed use_cache=True in generate() and ran it. Using SDPA still throws the shape error of q, but using eager attention execution turns out to be normal.

[zucchini-nlp]

weird, I will check what is wrong with SDPA later on, Keeping the issue open to not forget about it

[zucchini-nlp]

Cant reproduce the issue with official model weights, closing as resolved. Might be an issue in users' model setup, but if the issue persists feel free to re-open and tag me :)

import torch
from transformers import Qwen2_5OmniForConditionalGeneration, AutoProcessor

model_id = "Qwen/Qwen2.5-Omni-7B"
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    model_id,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

conversation = [
    {
        'role': 'system',
        'content': [
            {'type': 'text', 'text': 'You are a helpful assistant.'}
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "/raid/raushan/karate.mp4"},
            {"type": "text", "text": "What is this video about?"}
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    load_audio_from_video=False,
    fps=2,
).to(device=model.device, dtype=model.dtype)

text_ids = model.generate(**inputs, use_audio_in_video=False, return_audio=False, use_cache=True, thinker_max_new_tokens=100)
text = processor.batch_decode(text_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)

[CharlesGong12]

Weird. I directly copy and run your code, and the shape of error of q still occurs. Could that be an issue of different dependencies version or the cuda drive? @zucchini-nlp

[zucchini-nlp]

@CharlesGong12 interesting, can you check the shapes of queries and keys before they are fed to torch SDPA? Btw, I am also using the latest torch/transformers versions, which might also be the reason

[CharlesGong12]

@zucchini-nlp I print the shapes of queries and keys and there are lots of outputs. The latest two outputs before the error are:

before attention_interface, query_states: torch.Size([1, 16, 5501, 128]), key_states: torch.Size([1, 2, 5501, 128])
In sdpa_attention_forward, query: torch.Size([1, 16, 5501, 128]), key: torch.Size([1, 2, 5501, 128])
before attention_interface, query_states: torch.Size([1, 16, 1, 128]), key_states: torch.Size([1, 2, 5502, 128])
In sdpa_attention_forward, query: torch.Size([1, 16, 1, 128]), key: torch.Size([1, 2, 5502, 128])

before attention_interface is here, and In sdpa_attention_forward is here

I also tried transformers==4.55.4 and the same error occured. The env is:

• transformers version: 4.55.4
• Platform: Linux-5.10.134-16.3.al8.x86_64-x86_64-with-glibc2.39
• Python version: 3.10.15
• Huggingface_hub version: 0.34.3
• Safetensors version: 0.5.3
• Accelerate version: 1.7.0
• Accelerate config: not found
• DeepSpeed version: 0.16.3
• PyTorch version (accelerator?): 2.6.0+cu126 (CUDA)
• Tensorflow version (GPU?): 2.16.2 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: Yes