[https://nvbugs/6395830][fix] Qwen-VL mRoPE: move seq-slot delta cache update to model device

[nv-guomingz]

Background / motivation

Running the Qwen3.6-27B dense VL checkpoint text-only through the PyTorch backend crashes at startup during KV-cache capacity estimation:

RuntimeError: Expected all tensors to be on the same device, but got source
is on cpu, different from other tensors on cuda:0
  (when checking argument in method wrapper_CUDA_index_copy_)

Stack (Qwen3.6-27B-NVFP4, tp1, CUTEDSL, text prompt):

py_executor_creator.create_py_executor
  -> kv_cache_creator.configure_kv_cache_capacity   # dummy-request probe
  -> model.forward -> modeling_qwen3vl.prepare_mrope_config
  -> modeling_qwen2vl._prepare_qwen_vl_mrope_config
  -> mrope_position_deltas_cache.index_copy_(0, seq_slots, deltas)  # CRASH

Root cause

Checkpoints carrying language_model_only: false with the dense Qwen3_5ForConditionalGeneration architecture match is_qwen_image_bench_config and are routed to QwenImageBenchModel. That model's multimodal_data_device_paths was missing the mrope_config.mrope_position_ids / mrope_config.mrope_position_deltas entries that its sibling _Qwen3_5VLModel lists, so the engine's H2D move (MultimodalParams.to_device with target_keywords) skips the mRoPE tensors. The deltas then reach the GPU seq-slot cache write (introduced in #11943) still CPU-resident, and the device-strict index_copy_ fails.

Why other Qwen-VL configs are unaffected:

• MoE VL checkpoints (Qwen3_5MoeForConditionalGeneration, e.g. Qwen3.6-35B-A3B) can never match the image-bench detection (it requires the dense arch), route to Qwen3_5MoeVLModel whose device paths include the mRoPE keys, and run clean — verified on the pre-fix tree with the identical single-GPU command.
• Dense VL checkpoints without language_model_only: false route to Qwen3_5VLModel, which also lists the mRoPE keys.

Summary

Two complementary changes:

1. QwenImageBenchModel.multimodal_data_device_paths: add the two mrope_config.* entries, matching _Qwen3_5VLModel, so mRoPE tensors ride the engine's pinned async H2D move like every other Qwen-VL model (covers real multimodal requests as well as the text-only dummy probe).
2. _prepare_qwen_vl_mrope_config (shared Qwen-VL mRoPE library): normalize deltas onto the cache device before index_copy_ as a defensive backstop for any path that still reaches the write branch with CPU tensors. No-op when devices already match.

Impact

• Fixes text-only startup (and CPU-origin mRoPE tensors generally) for the image-bench-routed dense VL checkpoints.
• No behavior change for models whose device paths already cover mRoPE (.to() is a no-op).
• Negligible cost: deltas is one element per seq slot.

Test

• Qwen3.6-27B-NVFP4 (dense VL -> image-bench route), single GPU, --moe_backend CUTEDSL, text prompts: previously crashed in configure_kv_cache_capacity; with this PR startup completes and generation produces coherent output.
• Qwen3.6-35B-A3B-NVFP4 (MoE VL route), identical command on the pre-fix tree: runs clean, confirming the failure is specific to the image-bench route's missing device paths.

🤖 Generated with Claude Code

[coderabbitai]

📝 Walkthrough

Walkthrough

Updates Qwen2-VL MRoPE preparation to move concatenated per-request delta tensors to the cache device before indexed copying, supporting CPU-resident delta inputs.

Changes

Qwen2-VL MRoPE preparation

Layer / File(s)	Summary
Normalize MRoPE delta device tensorrt_llm/_torch/models/modeling_qwen2vl.py	Moves concatenated deltas to mrope_position_deltas_cache.device with non-blocking transfer before index_copy_. Adds documentation for CPU-resident per-request data.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: wanli-jiang

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly states the NVBugs fix and the Qwen-VL mRoPE device-move change.
Description check	✅ Passed	The description explains the bug, root cause, fix, impact, and tests, covering the required essentials.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}