[Bug]: Error when using Qwen3-VL/Qwen3.5 with video input and num_frames

[danigarciaoca]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.9.1+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.12 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 20:16:04) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-6.8.0-101-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version        : 580.126.09
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Arquitectura:                            x86_64
modo(s) de operación de las CPUs:        32-bit, 64-bit
Address sizes:                           39 bits physical, 48 bits virtual
Orden de los bytes:                      Little Endian
CPU(s):                                  28
Lista de la(s) CPU(s) en línea:          0-27
ID de fabricante:                        GenuineIntel
Nombre del modelo:                       Intel(R) Core(TM) i7-14700KF
Familia de CPU:                          6
Modelo:                                  183
Hilo(s) de procesamiento por núcleo:     2
Núcleo(s) por «socket»:                  20
«Socket(s)»                              1
Revisión:                                1
CPU MHz máx.:                            5600,0000
CPU MHz mín.:                            800,0000
BogoMIPS:                                6835.20
Indicadores:                             fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities ibpb_exit_to_user
Virtualización:                          VT-x
Caché L1d:                               768 KiB (20 instances)
Caché L1i:                               1 MiB (20 instances)
Caché L2:                                28 MiB (11 instances)
Caché L3:                                33 MiB (1 instance)
Modo(s) NUMA:                            1
CPU(s) del nodo NUMA 0:                  0-27
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Mitigation; Clear Register File
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.0
[pip3] nvidia-cutlass-dsl-libs-base==4.4.0
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.3.20
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.9.1+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.9.1+cu129
[pip3] torchcodec==0.9.1
[pip3] torchvision==0.24.1+cu129
[pip3] transformers==4.57.6
[pip3] triton==3.5.1
[conda] flashinfer-python                    0.6.1            pypi_0           pypi
[conda] numpy                                2.2.6            pypi_0           pypi
[conda] nvidia-cublas-cu12                   12.9.1.4         pypi_0           pypi
[conda] nvidia-cuda-cupti-cu12               12.9.79          pypi_0           pypi
[conda] nvidia-cuda-nvrtc-cu12               12.9.86          pypi_0           pypi
[conda] nvidia-cuda-runtime-cu12             12.9.79          pypi_0           pypi
[conda] nvidia-cudnn-cu12                    9.10.2.21        pypi_0           pypi
[conda] nvidia-cudnn-frontend                1.18.0           pypi_0           pypi
[conda] nvidia-cufft-cu12                    11.4.1.4         pypi_0           pypi
[conda] nvidia-cufile-cu12                   1.14.1.1         pypi_0           pypi
[conda] nvidia-curand-cu12                   10.3.10.19       pypi_0           pypi
[conda] nvidia-cusolver-cu12                 11.7.5.82        pypi_0           pypi
[conda] nvidia-cusparse-cu12                 12.5.10.65       pypi_0           pypi
[conda] nvidia-cusparselt-cu12               0.7.1            pypi_0           pypi
[conda] nvidia-cutlass-dsl                   4.4.0            pypi_0           pypi
[conda] nvidia-cutlass-dsl-libs-base         4.4.0            pypi_0           pypi
[conda] nvidia-ml-py                         13.590.48        pypi_0           pypi
[conda] nvidia-nccl-cu12                     2.27.5           pypi_0           pypi
[conda] nvidia-nvjitlink-cu12                12.9.86          pypi_0           pypi
[conda] nvidia-nvshmem-cu12                  3.3.20           pypi_0           pypi
[conda] nvidia-nvtx-cu12                     12.9.79          pypi_0           pypi
[conda] pyzmq                                27.1.0           pypi_0           pypi
[conda] torch                                2.9.1+cu129      pypi_0           pypi
[conda] torch-c-dlpack-ext                   0.1.5            pypi_0           pypi
[conda] torchaudio                           2.9.1+cu129      pypi_0           pypi
[conda] torchcodec                           0.9.1            pypi_0           pypi
[conda] torchvision                          0.24.1+cu129     pypi_0           pypi
[conda] transformers                         4.57.6           pypi_0           pypi
[conda] triton                               3.5.1            pypi_0           pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.15.1
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
  	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-27	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

🐛 Describe the bug

Hi,

I found a bug in the Qwen3-VL model executor when using num_frames via mm_processor_kwargs in LLM.chat() / LLM.generate().

What happens

I have observed that when passing num_frames in mm_processor_kwargs from ll.chat() or generate() methods, an error is raised from qwen3_vl.py model executor:

assert len(timestamps) == grid_thw[0], (
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: The timestamps length(10) should be equal video length (16).

The error is triggered around this part, where the executor reads fps from hf_processor_mm_kwargs:

video, metadata = mm_items["video"][item_idx]
do_sample_frames = hf_processor_mm_kwargs.get("do_sample_frames")
sampled_fps = hf_processor_mm_kwargs.get("fps")

Some details

When num_frames is provided, fps is None, so _get_video_second_idx falls back to the default FPS from video_processor = self.get_video_processor() (see _get_video_second_idx). For Qwen3VLVideoProcessor this default is 2 FPS (see HF implementation).

This produces a mismatch between the reconstructed timestamps length and the number of frames represented by video_grid_thw (which reflects the num_frames value actually used to sample the input video), and this triggers the assertion:

  timestamps = self.info._get_video_second_idx(
      metadata, out_item, do_sample_frames, sampled_fps
  )

  assert len(timestamps) == grid_thw[0], (
      f"The timestamps length({len(timestamps)}) should be equal "
      f"video length ({grid_thw[0]})."
  )

Affected models

• Qwen3-VL
• Qwen3.5-VL (shares the same video processor / code path)

Scope: video modality when sampling is configured via mm_processor_kwargs={"num_frames": ...} (i.e., fps not set).

Proposed solutions

a) When num_frames is provided, compute the equivalent sampling FPS and populate it in hf_processor_mm_kwargs (or add it to the video metadata, e.g. sampled_fps).

b) Alternatively, read both fps and num_frames (defaulting to None) and pass both into _get_video_second_idx, so it can reconstruct timestamps correctly when only num_frames is set.

Steps to reproduce

I’m including a minimal repro snippet below:

from vllm import LLM, SamplingParams

if __name__ == '__main__':
    checkpoint_path = "Qwen/Qwen3-VL-8B-Instruct"

    llm = LLM(
        model=checkpoint_path,
        media_io_kwargs={"video": {"video_backend": "opencv", "num_frames": -1}},
        allowed_local_media_path="<path to local media>",
        seed=0,
    )

    sampling_params = SamplingParams(
        temperature=0.7,
        top_p=0.8,
        top_k=20,
        max_tokens=1024,
        seed=42,
        repetition_penalty=1.0,
        presence_penalty=1.5,
    )

    # OpenAI-style chat payload for VLM input
    conversation = [
        {
            "role": "user",
            "content": [
                {
                    "type": "video_url",
                    "video_url": {
                        "url": f"file://<path to local video file>"
                    },
                },
                {
                    "type": "text",
                    "text": "Describe the video"
                },
            ],
        }
    ]

    # Run chat-style generation on the multimodal payload (apply chat template and generate a response)
    outputs = llm.chat(
        conversation,
        sampling_params=sampling_params,
        mm_processor_kwargs={
            "num_frames": 32,
            "fps": None,
            "do_sample_frames": True,
        },
    )

    for i, output in enumerate(outputs):
        prompt = conversation[i]["content"][-1]["text"]
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}")
        print('=' * 40)
        print(f"Generated text: {generated_text}")

Thanks in advance!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[alezan1020]

Workaround for Qwen3-VL / Qwen3.5-VL timestamps length != video length assertion

I hit this exact bug and resolved it with a server-side configuration workaround.

Root cause

The bug has two interacting layers:

1. vLLM I/O loader extracts frames from the video container (controlled by media_io_kwargs).
2. HF Qwen3VL Processor post-processes those frames (controlled by mm_processor_kwargs).

When num_frames is provided but fps is None, the model executor's _get_video_second_idx falls back to the HF Qwen3VLVideoProcessor default of 2 FPS. This reconstructs timestamps at 2 FPS, but video_grid_thw reflects the actual number of frames sampled (via num_frames or the I/O loader's fps), causing the mismatch:

AssertionError: The timestamps length(10) should be equal video length (16).

There is a second, related pitfall: do_sample_frames defaults to True in the HF processor. If the vLLM I/O loader already sampled frames at a specific FPS, the processor re-samples them at its own default FPS (2), producing fewer frames than the I/O metadata records — triggering the same assertion.

Workaround

The fix is to always explicitly set fps and disable double-sampling in mm_processor_kwargs:

# Layer 1 — vLLM I/O loader (media_io_kwargs)
# Controls how frames are decoded from the container.
media_io_kwargs = {
    "video": {
        "video_backend": "opencv_dynamic",
        "num_frames": 512,
        "fps": 4,
        "max_duration": 128,
    },
}

# Layer 2 — HF Qwen3VL Processor (mm_processor_kwargs)
# MUST be FLAT — no nested images_kwargs/videos_kwargs dicts.
# Reason: vLLM injects `do_sample_frames` at the TOP level of kwargs
# (from I/O metadata). If your config also has `do_sample_frames`
# inside a nested `videos_kwargs` dict, HF's _merge_kwargs raises
# ValueError("passed two times").
mm_processor_kwargs = {
    "min_pixels": 65536,
    "max_pixels": 33554432,
    "do_resize": True,
    "fps": 4.0,                # MUST match the I/O loader fps
    "do_sample_frames": False,  # CRITICAL — prevent double-sampling
}

Why this works

Parameter	Purpose
fps: 4.0 in mm_processor_kwargs	Ensures _get_video_second_idx uses the correct FPS instead of falling back to the HF default (2.0). The value must match the FPS used by the I/O loader.
do_sample_frames: False	Prevents the HF processor from re-sampling frames that the vLLM I/O loader already extracted. Without this, the processor applies its own default FPS (2), producing fewer frames than grid_thw expects.
Flat kwargs (no nesting)	Avoids ValueError("passed two times") from HF's _merge_kwargs when vLLM also injects do_sample_frames at the top level.

vLLM serve CLI equivalent

vllm serve Qwen/Qwen3-VL-8B-Instruct \
  --media-io-kwargs '{"video": {"video_backend": "opencv_dynamic", "num_frames": 512, "fps": 4, "max_duration": 128}}' \
  --mm-processor-kwargs '{"min_pixels": 65536, "max_pixels": 33554432, "do_resize": true, "fps": 4.0, "do_sample_frames": false}'

Suggested upstream fix

The workaround above requires users to keep fps synchronized across two config layers manually. The proper fix in vLLM would be:

When num_frames is provided but fps is None, compute the effective FPS from the video metadata (num_frames / duration) and pass it into _get_video_second_idx, rather than silently falling back to the HF processor's class default of 2.0.

Environment

• vLLM: 0.16.0
• Transformers: 4.57.6
• PyTorch: 2.9.1+cu129
• Affected models: Qwen3-VL, Qwen3.5-VL (shared code path)

Hope this helps!

[danigarciaoca]

Hi @alezan1020!

Thanks for the quick response and for sharing the workaround. I actually tried that approach already, but it doesn’t fit my use case.

In my setup, I need to process videos with variable durations. Because of that, using a fixed num_frames=512 ends up implying a different effective FPS for each video:

• Video 1: 30s @ 25 fps = 750 total frames → num_frames=512 ≈ sampling at ~17 fps
• Video 2: 60s @ 25 fps = 1500 total frames → num_frames=512 ≈ sampling at ~8.5 fps

So, in order to use your workaround, I would have to restart the vLLM server with a different fps param on each request to adapt to each input video, which isn’t practical.

That’s why I think the right direction is to fix this so that num_frames can be handled properly at request time (both for offline inference and online serving), and the backend can derive consistent timestamps / sampling behavior without requiring server-side reconfiguration per video.

To consider

By the way, be aware that your proposed approach can lead to unintended behavior.

If you pass both num_frames and fps (as in your example):

# Layer 1 — vLLM I/O loader (media_io_kwargs)
media_io_kwargs = {
    "video": {
        "video_backend": "opencv_dynamic",
        "num_frames": 512,
        "fps": 4,
        "max_duration": 128,
    },
}

vLLM effectively uses the minimum number of frames between the FPS-based sampling (fps) and the max-total-frames setting (num_frames). See the implementation here:

# resample video to target num_frames and fps
# - the minimum of the two will be used
num_frames_to_sample = total_frames_num
if num_frames > 0:
    num_frames_to_sample = min(num_frames, total_frames_num)
if fps > 0:
    num_frames_to_sample = min(num_frames_to_sample, math.floor(duration * fps))
num_frames_to_sample = max(1, num_frames_to_sample)  # at least one sample

As a result, if your inputs videos have variable durations, the effective sampling rate can vary across videos (i.e., the effective FPS may differ from the configured fps, leading to incorrect timestamps).

---

Hoping this gets addressed in an upcoming release! 🚀