[copilot] Qwen3-VL Support

[justinchuby]

[copilot] Feature Request: Native Qwen3-VL Support in onnxruntime-genai

Summary

Add native support for Qwen3-VL (Qwen/Qwen3-VL-2B-Instruct, Qwen/Qwen3-VL-8B-Instruct)
as a vision-language model in onnxruntime-genai. Currently, Qwen3-VL can run using the
"qwen2_5_vl" model type as a workaround (the 3-model I/O contract is identical for
text-only generation), but native support would enable proper image processing and
model-type-specific optimizations.

Motivation

Qwen3-VL is the latest vision-language model from the Qwen series. We have verified that
the 3-model split (decoder + vision encoder + embedding) works end-to-end with
onnxruntime-genai using the "qwen2_5_vl" model type, but this workaround has limitations:

1. The image processor (QwenImageProcessor) may not handle Qwen3-VL's different
   patch size (16x16 vs 14x14) and image preprocessing correctly.
2. The processor_config.json format may differ between Qwen2.5-VL and Qwen3-VL.
3. Model-type-specific optimizations and configuration cannot be applied.

Working Prototype

We have a working prototype in the onnx-genai-models package
(https://github.com/justinchuby/onnx-genai-models) that exports Qwen3-VL
as a 3-model package compatible with onnxruntime-genai:

• model.onnx (decoder): inputs_embeds, attention_mask, position_ids[3,B,S],
  past_kv → logits, present_kv
• vision.onnx: pixel_values[N, CT_pP*P], cu_seqlens, rotary_pos_ids,
  grid_thw → image_features
• embedding.onnx: input_ids, image_features → inputs_embeds

Text-only generation works by loading with model.type = "qwen2_5_vl" in
genai_config.json. The model generates correct, coherent text.

Architectural Differences from Qwen2.5-VL

Feature	Qwen2.5-VL	Qwen3-VL
Patch size	14×14	16×16
Vision attention	Windowed + full-att	Packed (block-diag)
Vision MLP	Gated (SiLU)	Standard (GELU)
Vision normalization	RMSNorm	LayerNorm
Feature fusion	Single MLP merger	DeepStack (multi-layer)
Text RoPE	MRoPE [16,24,24]	Interleaved MRoPE
MRoPE section	[16,24,24]	[24,20,20]
Text QK norm	No	Yes (RMSNorm on Q,K)
Merger structure	Sequential(fc,gelu,fc)	Linear(fc,norm,gelu,fc)
HF weight prefix	model.{layers,visual}	model.{language_model,visual}
tie_word_embeddings	True	True
image_token_id	151655	151655

Proposed Changes

1. Model Type Registration (model_type.h)

Add "qwen3_vl" to the VLM array:

static constexpr std::array<std::string_view, 5> VLM = {
    "fara", "gemma3", "phi3v", "qwen2_5_vl", "qwen3_vl"
};

Update IsQwen25VL() to also handle Qwen3-VL (both use 3D MRoPE position_ids):

static bool IsQwen25VL(const std::string& model_type) {
    return model_type == "fara"
        || model_type == "qwen2_5_vl"
        || model_type == "qwen3_vl";
}

2. Image Processor

Option A: Reuse QwenImageProcessor with config-driven patch size.
The spatial_merge_size is already read from config. If the image preprocessing
pipeline is compatible (just different patch_size), the existing processor may
work with minimal changes.

Option B: Create a Qwen3VLImageProcessor subclass that handles the 16x16 patch
size and any Qwen3-VL-specific preprocessing (e.g., different image normalization
constants, different grid computation).

3. MultiModal Processor Factory (model.cpp)

Register Qwen3-VL in the processor factory:

{"qwen3_vl", Processor::Create<QwenImageProcessor>}  // or Qwen3VLImageProcessor

4. genai_config.json

Standard 3-model config with model.type = "qwen3_vl":

{
    "model": {
        "type": "qwen3_vl",
        "decoder": {
            "filename": "model.onnx",
            "head_size": 128,
            "hidden_size": 2048,
            "num_attention_heads": 16,
            "num_hidden_layers": 28,
            "num_key_value_heads": 8,
            ...
        },
        "vision": {
            "filename": "vision.onnx",
            "spatial_merge_size": 2,
            ...
        },
        "embedding": {
            "filename": "embedding.onnx",
            ...
        }
    }
}

Testing

We have verified the following with our prototype:

• Text-only generation produces coherent output (tested with Qwen3-VL-2B-Instruct)
• All ONNX model weights are correctly assigned (625 weight tensors)
• Decoder logits match HuggingFace PyTorch reference (rtol=1e-3, atol=1e-3)
• 3-model split loads and runs with onnxruntime-genai 0.12.0

Test script:

import onnxruntime_genai as og
model = og.Model("path/to/qwen3vl/")
tokenizer = og.Tokenizer(model)
input_ids = tokenizer.encode("The capital of France is")
params = og.GeneratorParams(model)
params.set_search_options(max_length=50)
generator = og.Generator(model, params)
generator.append_tokens(input_ids)
while not generator.is_done():
    generator.generate_next_token()
    print(tokenizer.decode([generator.get_next_tokens()[0]]), end="")

Output: "The capital of France is Paris, and the capital of Spain is Madrid."

Note on DeepStack

Qwen3-VL introduces DeepStack, which injects intermediate vision features into
early text decoder layers. The 3-model split does NOT include DeepStack features
(only the final merged vision features are passed through the embedding model).
Supporting DeepStack would require extending the MultiModalLanguageModel pipeline
to pass additional tensors from vision → decoder, which is a larger architectural
change. The current 3-model split without DeepStack still produces high-quality
text generation.

Detailed Changes Required for DeepStack Support

What DeepStack Does

During the vision encoder forward pass, hidden states are extracted at specific
intermediate transformer layers (e.g., layers 5, 11, 17 for Qwen3-VL-2B's
24-layer ViT). Each extracted hidden state is passed through a dedicated
PatchMerger (with its own learned weights), producing a feature tensor of
shape (num_merged_tokens, hidden_size).

These features are then injected into the first N text decoder layers during
forward pass. At each injection point, the features are scattered at image
token positions and added to the hidden states:

# In text decoder layer loop:
for layer_idx, layer in enumerate(self.layers):
    hidden_states = layer(hidden_states, ...)

    if layer_idx < len(deepstack_visual_embeds):
        # Scatter deepstack features at image_token_id positions
        mask = (input_ids == image_token_id)
        indices = cumsum(mask) - 1  # Map to flat feature indices
        scattered = gather(deepstack_embeds[layer_idx], indices)
        hidden_states += where(mask, scattered, 0.0)

For Qwen3-VL-2B (28 text layers, 24 vision layers):

• deepstack_visual_indexes = [5, 11, 17] → 3 intermediate features
• Each feature: (num_merged_tokens, 2048)
• Injected into text decoder layers 0, 1, 2

Current ORT GenAI Pipeline (prompt stage)

pixel_values, grid_thw  ──→  [vision.onnx]  ──→  image_features
                                                       │
input_ids + image_features  ──→  [embedding.onnx]  ──→  inputs_embeds
                                                              │
inputs_embeds + position_ids + past_kv  ──→  [model.onnx]  ──→  logits

The pipeline has three sequential stages. Between stages, ORT GenAI passes
tensors via shared buffers (ReuseFeaturesBuffer / ReuseEmbeddingsBuffer).
The decoder model only receives inputs_embeds — no additional vision features.

Proposed Pipeline with DeepStack

pixel_values, grid_thw  ──→  [vision.onnx]  ──→  image_features
                                              ├──→  deepstack_0
                                              ├──→  deepstack_1
                                              └──→  deepstack_2
                                                       │
input_ids + image_features  ──→  [embedding.onnx]  ──→  inputs_embeds
                                                              │
inputs_embeds + deepstack_{0..2}  ──→  [model.onnx]  ──→  logits
            + position_ids + past_kv

Required C++ Changes

1. Vision Model Outputs (vision.onnx)

The vision ONNX model gains additional outputs:

Inputs:   pixel_values, cu_seqlens, rotary_pos_ids, grid_thw
Outputs:  image_features              (num_merged, hidden_size)
          deepstack_features_0        (num_merged, hidden_size)
          deepstack_features_1        (num_merged, hidden_size)
          deepstack_features_2        (num_merged, hidden_size)

2. Decoder Model Inputs (model.onnx)

The decoder ONNX model gains additional inputs:

Inputs:   inputs_embeds               (batch, seq, hidden_size)
          attention_mask              (batch, total_seq)
          position_ids               (3, batch, seq)
          deepstack_features_0        (num_merged, hidden_size)  [prompt only]
          deepstack_features_1        (num_merged, hidden_size)  [prompt only]
          deepstack_features_2        (num_merged, hidden_size)  [prompt only]
          past_key_values.{i}.key/value

The deepstack inputs are only used during the prompt stage. During generation
steps they should be empty tensors with dim-0 = 0.

3. Config Schema (config.h)

Add DeepStack config under the Vision struct:

struct Vision {
    ...
    // DeepStack: intermediate feature injection
    std::vector<int> deepstack_layer_indexes;  // e.g. [5, 11, 17]

    struct Outputs {
        std::string image_features{"image_features"};
        std::vector<std::string> deepstack_features;  // output names
    } outputs;
};

struct Decoder {
    ...
    struct Inputs {
        ...
        std::vector<std::string> deepstack_features;  // input names
    } inputs;
};

4. genai_config.json

{
    "model": {
        "type": "qwen3_vl",
        "vision": {
            "filename": "vision.onnx",
            "deepstack_layer_indexes": [5, 11, 17],
            "outputs": {
                "image_features": "image_features",
                "deepstack_features": [
                    "deepstack_features_0",
                    "deepstack_features_1",
                    "deepstack_features_2"
                ]
            }
        },
        "decoder": {
            "filename": "model.onnx",
            "inputs": {
                "inputs_embeds": "inputs_embeds",
                "deepstack_features": [
                    "deepstack_features_0",
                    "deepstack_features_1",
                    "deepstack_features_2"
                ],
                ...
            }
        }
    }
}

5. VisionState (multi_modal.cpp)

Add deepstack feature buffers alongside image_features:

class VisionState {
    std::unique_ptr<MultiModalFeatures> image_features_;
    std::vector<std::unique_ptr<MultiModalFeatures>> deepstack_features_;
    // ...
};

void VisionState::SetExtraInputs(...) {
    // Existing image_features setup
    image_features_ = ...;
    // New: deepstack output buffers
    for (auto& name : config.model.vision.outputs.deepstack_features) {
        deepstack_features_.push_back(
            std::make_unique<MultiModalFeatures>(*this, Mode::Output, name, ...));
    }
}

6. DecoderState (multi_modal.cpp)

Accept deepstack features as additional prompt-only inputs:

class DecoderState {
    std::vector<std::unique_ptr<MultiModalFeatures>> deepstack_features_;
    // ...
};

void DecoderState::SetDeepStackInputs(
    const std::vector<std::unique_ptr<MultiModalFeatures>>& vision_ds) {
    for (size_t i = 0; i < vision_ds.size(); ++i) {
        deepstack_features_[i]->ReuseFeaturesBuffer(*vision_ds[i]);
    }
}

7. Pipeline Orchestration (MultiModalPipelineState::Run)

Thread deepstack features from vision to decoder during prompt stage:

if (is_prompt_) {
    if (num_image_tokens_ > 0 && vision_state_) {
        vision_state_->Run(...);
    }

    // Existing: pass image_features to embedding
    embedding_state_->image_features_->ReuseFeaturesBuffer(
        *vision_state_->image_features_);

    // New: pass deepstack features to decoder
    decoder_state_->SetDeepStackInputs(vision_state_->deepstack_features_);

    embedding_state_->Run(...);
    decoder_state_->Run(...);

    // Cleanup: deepstack only needed for prompt
    for (auto& ds : decoder_state_->deepstack_features_) {
        ds->ClearBuffer();
    }
}

Complexity Estimate

The DeepStack support is a medium-sized change:

• Config parsing: ~50 lines (new vector fields)
• VisionState: ~30 lines (additional output buffers)
• DecoderState: ~40 lines (additional input buffers, prompt-only logic)
• Pipeline: ~20 lines (threading features through)
• Total: ~140 lines of C++ across 3-4 files

The ONNX model export side (onnx-genai-models) already supports DeepStack in the
single-model variant; the 3-model split just needs to expose the intermediate
features as additional vision outputs and decoder inputs.

References

• Qwen3-VL paper: https://arxiv.org/abs/2502.13923
• HuggingFace model: https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct
• onnx-genai-models package: https://github.com/justinchuby/onnx-genai-models
• Export code: src/onnx_genai_models/models/qwen.py (Qwen3VL3ModelCausalLMModel)

Additional Discussion: Prefill vs Decode Model Separation

What it means

Instead of one model.onnx handling both phases with dynamic sequence length, you'd have:

• prefill.onnx — processes full input sequence (seq_len = N), populates KV cache
• decode.onnx — generates one token at a time (seq_len = 1), reads/appends KV cache

Pros

1. Better kernel selection — Prefill is compute-bound (large MatMuls over full sequence); decode is memory-bandwidth-bound (single token, large KV cache reads).
   ONNX Runtime can select different kernels/fusions for each.
2. Simpler decode graph — The decode model doesn't need multimodal logic (image token scattering, DeepStack injection, embedding lookups). It's just: embed one
   token → N transformer layers → logits.
3. Flash/paged attention — Prefill benefits from flash attention (large Q×K); decode benefits from paged KV cache. Separate graphs make it easier to use
   different attention implementations.
4. Reduced branching — No if seq_len == 1 dynamic shapes; each model has more predictable shapes, enabling better static optimization.
5. Speculative decoding — Draft model generates candidates; verification uses the prefill model to score them in parallel. Natural fit.

Cons

1. Weight duplication — Both models share ~99% of weights. Either duplicate on disk (2× storage, e.g. 6GB→12GB for a 3B model) or need weight-sharing at runtime
   (adds runtime complexity).
2. Pipeline complexity — ORT GenAI currently assumes one decoder model. Splitting requires changes to GeneratorState to switch models after prefill, transfer KV
   cache, and handle the prompt/generation boundary.
3. KV cache handoff — The prefill model's KV cache output format must exactly match the decode model's KV cache input format. Any mismatch (layout, padding,
   paged vs contiguous) causes correctness issues.
4. Continuous batching friction — In serving scenarios where prefill and decode requests are batched together, separate models make this harder (vLLM/TRT-LLM
   handle this with unified models + different CUDA kernels).
5. Maintenance burden — Two model graphs to export, validate, and keep in sync. Any architectural change (e.g., adding LoRA) must be applied to both.
6. Marginal gain with good runtime — ONNX Runtime already handles dynamic shapes well. The performance gap between split and unified models is smaller than in
   frameworks without good dynamic shape support.

[fwaris]

Justin Chu (@justinchuby)

FYI. In the interim, I was able to port Qwen3 VLM to ONNX by separating out the graphs and stitching them together in code. This includes DeepStack handling. Intermediate values use OrtValue so the tensors stay on the GPU.

See:
https://github.com/fwaris/ArDroneVlm/tree/main/src/Qwen3Vl.Onnx

As an aside the, processing rate on the (aging) RTX 3080 is ~ 3 sec per image