[Bugfix] Allow Only SDPA Backend for ViT on B200 for Qwen3-VL

[yewentao256]

Purpose

Temple fix for #25582

Special thanks to @ywang96 for the context

The default XFORMERS backend has problem and we will meet repeated token issue

The FlashAttn backend has issue too, we will meet This flash attention build does not support headdim not being a multiple of 32.', please check the stack trace above for the root cause

Let's land this temple fix first to require user to use torch SDPA, and we can have other issues fix in following up PRs.

Test

vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct --tensor-parallel-size 4 --enforce_eager

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-VL-235B-A22B-Instruct",
    "temperature": 0.1,
    "max_tokens": 1024,
    "messages": [
      {
        "role": "user",
        "content": [
          {"type":"image_url","image_url":{"url":"https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}},
          {"type":"text","text":"Read all the text in the image."}
        ]
      }
    ]
  }' | jq .

Origin

Generated text: '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'

Now

Wentao Ye
  [下午 5:34](https://vllm-dev.slack.com/archives/D09HJSC0VC4/p1758922499070009)
        "role": "assistant",
        "content": "Auntie Anne's\n\nCINNAMON SUGAR\n1 x 17,000                      17,000\nSUB TOTAL                       17,000\n\nGRAND TOTAL                     17,000\nCASH IDR                        20,000\nCHANGE DUE                      3,000",
        "refusal": null,

[gemini-code-assist]

Code Review

This pull request introduces a temporary fix for an issue in Qwen3-VL by enforcing the use of the TORCH_SDPA attention backend. While the intent is correct, the implementation uses an assert statement for configuration validation, which is not robust as it can be disabled. My review suggests replacing this with a ValueError to ensure the check is always performed, and also points out a potential logic flaw in the backend selection that could negatively impact user experience.

[yewentao256]

@tlrmchlsmth CC

[ywang96]

I cleaned up the PR a bit - thanks for fixing this!