Serving Qwen3-Omni with vLLM on a Blackwell GPU: setup notes and two gotchas

[sungbin1015]

Notes from serving Qwen3-Omni-30B-A3B-Instruct with vLLM on an AWS g7e.4xlarge (one RTX PRO 6000 Blackwell, 96 GB). Two things cost us time, so I am writing them down.

Environment

OS	Amazon Linux 2023
Python	3.12 (uv venv)
torch	2.11.0+cu130
vLLM	0.22.0

1. Blackwell needs a torch wheel with sm_120

Install torch first and check the arch list before anything else:

uv pip install torch==2.11.0 torchaudio==2.11.0 torchvision==0.26.0 \
    --index-url https://download.pytorch.org/whl/cu130
python -c "import torch; print(torch.cuda.get_arch_list())"
# ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120']

Then pin it while installing vLLM, otherwise the vLLM install can replace torch:

cat > /tmp/torch-constraint.txt <<'TXT'
torch==2.11.0+cu130
torchaudio==2.11.0+cu130
torchvision==0.26.0+cu130
TXT
uv pip install vllm==0.22.0 \
    --constraint /tmp/torch-constraint.txt \
    --extra-index-url https://download.pytorch.org/whl/cu130 \
    --index-strategy unsafe-best-match

2. Audio input returns 400 until the audio decoders are installed

With a plain uv pip install vllm, requests that include audio fail with:

400 "Invalid or unsupported audio file"

Video-only requests work, which made this confusing. The audio decoders are not installed by default:

uv pip install soundfile librosa av

Restart the server afterwards. To have the audio track of an mp4 processed together with the video, the request also needs mm_processor_kwargs: {"use_audio_in_video": true}.

Serve command

vllm serve /mnt/nvme/models/Qwen3-Omni-30B-A3B-Instruct \
    --served-model-name qwen \
    --port 8000 --host 0.0.0.0 \
    --dtype bfloat16 \
    --max-model-len 16384 \
    --max-num-seqs 8 \
    --gpu-memory-utilization 0.90 \
    --mm-encoder-attn-backend TORCH_SDPA \
    --moe-backend triton \
    --allowed-local-media-path /mnt/nvme/vod \
    --limit-mm-per-prompt '{"image":1,"video":1,"audio":1}' \
    --tensor-parallel-size 1 \
    --trust-remote-code

A known issue with use_audio_in_video

On vLLM 0.21.0 we hit HTTP 400 (StopIteration in the processor) when two requests had different video frames but a byte-identical audio track, for example BGM-only clips from the same episode. It comes from the multimodal processor cache; --mm-processor-cache-gb 0 avoids it. Reproduction: vllm-project/vllm#44538

Full write-up (Korean): https://doc.scenemaker.solbox.com/docs/install/1