MNN Model Support: Qwen3.5

[wangzhaode]

Introduction

Qwen3.5 was officially open-sourced on Chinese New Year's Eve, and the MNN team quickly completed comprehensive adaptation and optimization, simultaneously releasing Qwen3.5 MNN models that can be efficiently deployed on edge devices.

Qwen3.5 is the latest generation of large language models in the Qwen (Tongyi Qianwen) series. From Qwen to Qwen2, Qwen2.5, and Qwen3, each generation's upgrades have primarily focused on training data, model scale, and fine-tuning strategies, while the core architecture has always remained within the standard Transformer Decoder framework. Qwen3.5, however, represents the most significant architectural overhaul in this series, introducing disruptive designs across almost every key module:

• Attention Mechanism: Introduces Hybrid Attention, alternating between standard Attention and linear attention DeltaNet
• Normalization Layer: Introduces Gated RMSNorm, applied to both linear attention layers and standard Attention layers
• MoE Architecture: Adds a Shared Expert mechanism
• Model Form: Released by default as a multimodal Vision-Language Model (VLM)
• Position Encoding: Adds Partial Rotary, continuing to use Interleaved M-RoPE
• Vocabulary: Expanded from 151,936 to 248,320
• Speculative Decoding: Built-in MTP (Multi-Token Prediction) support

This article provides a detailed introduction to the core adaptation work and technical implementations MNN has made to support Qwen3.5.

Overview of Qwen3.5 Architectural Changes

Feature	Qwen3 / Qwen3-VL	Qwen3.5
Attention Mechanism	Full Transformer Attention	Hybrid Attention (3:1 Linear + Full)
Normalization	RMSNorm	RMSNorm + Gated RMSNorm
MoE	Expert Routing	Expert Routing + Shared Expert
Default Modality	Text-only / VL released separately	Default Multimodal (VLM)
RoPE	Standard / Interleaved M-RoPE	Partial Rotary + Interleaved M-RoPE
Vocabulary Size	151,936	248,320
Speculative Decoding	-	MTP (Multi-Token Prediction)

1. DeltaNet: Linear Attention Mechanism

1.1 Comparison with Standard Attention

Standard Transformer Attention and DeltaNet linear attention are both sequence modeling mechanisms at their core, but they differ fundamentally in computation patterns and resource consumption:

	Standard Attention	DeltaNet Linear Attention
Core Operation	Softmax(QK^T/√d)V	Recurrent state matrix recurrence
Per-Step Inference Complexity	O(L), requires traversing all historical KV	O(1), only updates fixed-size state
Memory Overhead	KV Cache grows linearly with sequence length	Fixed-size state [H, d_k, d_v]
Position Encoding	Depends on RoPE	Causal convolution implicitly encodes position
Information Access	Can precisely access any historical position	Indirectly accessed via compressed memory
Suitable Scenarios	Requires precise long-range dependencies	Scenarios where information can be progressively compressed

Qwen3.5 combines these two attention types at a 3:1 ratio -- out of every 4 layers, 3 use DeltaNet linear attention and 1 uses standard Full Attention. This design significantly reduces inference costs while relying on the periodically inserted standard Attention layers to maintain precise capture of critical information.

1.2 Algorithm Principles

The core of DeltaNet is the Gated Delta Rule, which maintains a Key-Value memory matrix S ∈ ℝ^{d_k × d_v}, updating at each time step through the following five steps:

Input: qkv [B, D, L], gate [B, L, H], beta [B, L, H], conv_weight [D, 1, K]

Step 1: Depthwise Conv1D + SiLU
    Pass qkv through a stateful causal convolution and SiLU activation

Step 2: Split Q/K/V + GQA Expansion
    Separate Q, K, V from convolution output, and handle GQA (Grouped-Query Attention)

Step 3: Optional L2 Normalization
    Apply per-head L2 normalization to Q and K

Step 4: Scale Q
    Q = Q / √d_k

Step 5: Gated Delta Rule Recurrence (Core)
    For each time step t:
    S_t = S_{t-1} × exp(gate_t)         // Gated decay of old memory
    v_pred = S_t^T @ k_t                // Predict value using current key
    delta = beta_t × (v_t - v_pred)     // Compute prediction error
    S_t = S_t + k_t @ delta^T           // Outer product update of memory
    o_t = S_t^T @ q_t                   // Query memory with query

Here, gate (decay factor) and beta (learning rate) control the forgetting and update strength of memory, enabling the model to maintain fine-grained control over contextual information while preserving linear complexity.

Notably, the computation of gate is carefully parameterized to ensure it is always non-positive (guaranteeing memory decay):

beta = torch.sigmoid(b)                                                  # ∈ (0, 1)
gate = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)  # always ≤ 0

Both A_log and dt_bias are learnable parameters. exp(A_log) ensures positivity, softplus(...) ensures non-negativity, and the leading negative sign ensures gate is always ≤ 0, yielding a decay coefficient in (0, 1] after applying exp(gate).

1.3 Reverse GQA

Qwen3.5's Linear Attention features a unique design: the number of V heads exceeds the number of K/Q heads (e.g., in the 7B model, num_v_heads=32, num_k_heads=16). This is the opposite of standard GQA (Grouped-Query Attention), where K/V heads are fewer than Q heads. Before computation, K and Q are expanded via repeat_interleave to match the number of V heads:

if self.num_v_heads > self.num_k_heads:
    factor = self.num_v_heads // self.num_k_heads
    query = query.repeat_interleave(factor, dim=2)
    key = key.repeat_interleave(factor, dim=2)

This means the recurrent state S has dimensions [B, num_v_heads, d_k, d_v], with each V head possessing its own independent memory matrix. More V heads give the model richer expressive capacity at the output side, while keeping the computational cost of K/Q low.

1.4 MNN Implementation

To support DeltaNet, MNN introduced a new OpType_LinearAttention operator (ID=305), with a new LinearAttentionParam parameter table in the FlatBuffers Schema:

table LinearAttentionParam {
    attn_type: string;       // "gated_delta_rule"
    num_k_heads: int;        // Number of K/Q heads
    num_v_heads: int;        // Number of V heads
    head_k_dim: int;         // Dimension per K head
    head_v_dim: int;         // Dimension per V head
    use_qk_l2norm: bool;     // Whether to apply L2 normalization to Q/K
}

This operator has complete implementations across four backends:

• CPU Backend (CPULinearAttention.cpp): Leverages multi-threading parallelism, channel-level parallel computation for Conv1D+SiLU, head-level parallel computation for the Gated Delta Rule, and uses MNN's built-in optimized functions such as MNNSiLu, MNNScaleAndAddBiasScalar, and MNNComputeMatMulForE_1 to accelerate core computations.

• Metal Backend (MetalLinearAttention.mm): Implements three Metal Compute Kernels -- linear_attn_conv_silu, linear_attn_conv_state_update, linear_attn_gated_delta_rule -- compiled via inline shaders using R"metal(...)", with a Pipeline caching mechanism to avoid redundant compilation.

• OpenCL Backend (LinearAttentionBufExecution.cpp): Three OpenCL Kernels are written as .cl files, automatically converted to C++ source code and embedded during compilation via a Codegen script. All use float precision to ensure numerical stability in recurrence.

• Vulkan Backend (VulkanLinearAttention.cpp): Three GLSL Compute Shaders compiled to SPIR-V, registered via VulkanShaderMap, following the Vulkan backend's standard Descriptor Set + Uniform Buffer pattern.

All four backends share the same algorithmic logic and must manage two persistent states:

• Conv State [B, D, K-1]: Sliding window history for causal convolution
• Recurrent State [B, H, d_k, d_v]: Key-Value memory matrix for the Delta Rule

These states are allocated using Backend::STATIC, zero-initialized on the first inference step, and persist across Decode steps.

1.5 Hybrid Attention Decoder Structure

Qwen3.5 adopts a hybrid architecture alternating 3 layers of Linear Attention with 1 layer of Full Attention. In the Decoder, each layer automatically selects the attention type based on the original model's configuration:

class Decoder:
    def __init__(self, ...):
        if hasattr(self, 'self_attn'):
            self.self_attn = Attention(...)       # Standard KV Cache Attention
            self.layer_type = 'full_attention'
        if hasattr(self, 'linear_attn'):
            self.self_attn = LinearAttention(...)  # DeltaNet linear attention
            self.layer_type = 'linear_attention'

Differences between the two layer types during inference:

• Full Attention layers: Require RoPE position encoding and Attention Mask, maintain KV Cache.
• Linear Attention layers: Do not use RoPE (position information is implicitly encoded by causal convolution), do not require KV Cache, and only maintain a fixed-size recurrent state.

This means Qwen3.5's KV Cache size is only about 1/4 that of a pure Transformer model, significantly reducing memory overhead for long-sequence inference.

2. Gated RMSNorm

Qwen3.5 introduces Gated RMSNorm, an extension of standard RMSNorm that additionally receives a gating signal, multiplying the normalized result by a SiLU-activated gate value:

class RMSNorm:
    def forward(self, hidden_states, gate=None):
        # Standard RMSNorm
        variance = hidden_states.pow(2).mean(-1, keepdim=True)
        hidden_states = hidden_states * torch.rsqrt(variance + eps)
        hidden_states = weight * hidden_states
        # Gated: Multiply normalized result by SiLU(gate)
        if gate is not None:
            hidden_states = hidden_states * F.silu(gate)
        return hidden_states

Gated RMSNorm is applied to both attention layer types in Qwen3.5:

In Linear Attention layers, the gate comes from an independent in_proj_z projection:

z = self.in_proj_z(hidden_states)       # [B, L, value_dim]
attn_out = self.norm(attn_out, z)       # Gated RMSNorm: norm(x) * silu(z)
output = self.out_proj(attn_out)

In Full Attention layers, the gate comes from the expanded output of q_proj. When q_proj's output dimension is 2 * num_heads * head_dim, the output is split into query and gate parts:

query_states = self.q_proj(hidden_states)
if self.q_proj.out_features == 2 * self.num_heads * self.head_dim:
    query_states, gate = torch.split(reshaped, self.head_dim, dim=-1)
    ...
    # After Attention computation, output is multiplied by sigmoid(gate)
    attn_output = attn_output * torch.sigmoid(gate)

Note that the two layer types use different gating activation functions: Linear Attention layers use SiLU (norm(x) * silu(z)), while Full Attention layers use Sigmoid (attn_output * sigmoid(gate)). SiLU outputs in the range (-0.28, +∞), allowing mild suppression; Sigmoid outputs are strictly in (0, 1), serving purely as a scaling mechanism.

3. MoE Architecture Changes: Shared Expert

3.1 What Changed

Qwen3.5's MoE (Mixture of Experts) layer introduces a Shared Expert mechanism. Unlike traditional MoE where each token is routed only to Top-k experts, the Shared Expert is a common expert that all tokens pass through, and its output is added to the routed experts' output:

class Mlp:
    def forward(self, hidden_states):
        # Shared Expert: Shared foundational capability for all tokens
        shared_expert_output = F.sigmoid(self.shared_expert_gate(x)) * self.shared_expert(x)

        # Routed Experts: Top-k routed sparse experts
        router_logits = self.gate(x)
        routing_weights = F.softmax(router_logits, dim=-1)
        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)
        routing_weights /= routing_weights.sum(dim=-1, keepdim=True)  # Normalize
        expert_output = self.custom_moe(x, routing_weights, selected_experts)

        # Final output = Routed expert output + Shared expert output
        final_output = expert_output + shared_expert_output
        return final_output

3.2 Significance of Shared Expert

• Baseline Capability: Shared Expert ensures every token receives at least one complete FFN computation, preventing information loss due to poor routing.
• Knowledge Sharing: General knowledge is stored in the Shared Expert, while specialized knowledge is handled by routed experts, forming a hierarchical knowledge structure.
• Training Stability: Reduces the routing collapse problem common in MoE training.

3.3 MNN Adaptation

In MNN's export pipeline, the Shared Expert is exported as an independent linear layer, and its output is modulated by shared_expert_gate (Sigmoid gating) before being added to the MoE output. The model_mapper.py adds corresponding mappings:

qwen3_5_moe_mlp = {
    'num_experts': 'experts.num_experts',
    'top_k': 'gate.top_k',
    'gate': 'gate',
    'experts': 'experts',
    'shared_expert_gate': 'shared_expert_gate',    # New
    'shared_expert': 'shared_expert'               # New
}

Additionally, the Expert weight format for MoE modules changes from ModuleList to PackedMLP format -- all expert weights are stored in a unified 3D tensor [num_experts, ...], accessible by index slicing (gate_up_proj.data[i] can be used directly without .transpose(0, 1)), simplifying the export process. Furthermore, Qwen3.5 MoE routing weights are normalized after Top-k selection (routing_weights /= routing_weights.sum()), ensuring the sum of all expert contributions equals 1.

4. Default Multimodal: Native Vision Support

4.1 Architectural Features

Unlike Qwen3, which released text-only and vision-language models separately, Qwen3.5 is released by default as a multimodal Vision-Language Model (VLM). This means:

• Model configuration uses a nested text_config structure (e.g., text_config.hidden_size), distinguishing text and vision configurations.
• Model paths access language model layers via model.language_model.layers rather than model.layers.
• The Embedding layer is located at model.language_model.embed_tokens.
• The Visual Encoder is located at model.visual.

4.2 Vision Encoder (Qwen3_5Vision)

Qwen3.5's vision encoder is largely consistent with Qwen3-VL, inheriting its ViT + Merger architecture design, Interleaved M-RoPE position encoding scheme, learnable position encoding (pos_embed + bilinear interpolation), and dynamic resolution handling. The main difference is:

• Deepstack Removed: The Deepstack feature introduced in Qwen3-VL (deepstack_merger_list, used for multi-scale deep fusion of visual tokens) has been removed in Qwen3.5, simplifying the vision encoder structure.

4.3 Interleaved M-RoPE Adaptation for Text-Only Scenarios

Qwen3.5 inherits the Interleaved M-RoPE position encoding scheme introduced in Qwen3-VL. This scheme alternates frequency components across time (T), height (H), and width (W) dimensions, rather than concatenating them along the feature dimension as in earlier M-RoPE:

# Traditional M-RoPE: [T_freq | H_freq | W_freq]  (concatenated)
# Interleaved M-RoPE: [T0, H0, W0, T1, H1, W1, ...]  (interleaved)
freq_idx = torch.arange(0, 3 * half_rotary).reshape(3, 1, half_rotary)
mrope_reindex = apply_interleaved_mrope(freq_idx, mrope_section).flatten()

In vision processing, the T/H/W dimensions each carry genuine spatiotemporal position information. For text-only inference, since text lacks spatial dimensions, the MNN inference engine replicates the same position ID three times (T=H=W), automatically handling this via the is_mrope configuration flag:

// Text-only inference: set all three-dimensional position IDs to the same sequence position
if (mConfig->is_mrope()) {
    positionIds = _Input({3, seq_len}, NCHW, halide_type_of<int>());
    auto ptr = positionIds->writeMap<int>();
    for (int i = 0; i < seq_len; i++) {
        ptr[0 * seq_len + i] = i + all_seq_len;  // T = pos
        ptr[1 * seq_len + i] = i + all_seq_len;  // H = pos
        ptr[2 * seq_len + i] = i + all_seq_len;  // W = pos
    }
}

4.4 Partial Rotary

Qwen3.5 introduces a partial_rotary_factor parameter, applying rotary position encoding to only a subset of dimensions within head_dim, while the remaining dimensions pass through unchanged:

if 'partial_rotary_factor' in config.rope_parameters:
    self.partial_rotary_factor = config.rope_parameters['partial_rotary_factor']
    self.rotary_dim = int(self.rotary_dim * self.partial_rotary_factor)

During apply_rotary_pos, this mechanism follows the same approach used by the Phi series models -- splitting the feature vector into two parts, applying rotation only to the first rotary_dim dimensions, and concatenating the remaining dimensions as-is:

def phi_rotary_pos(self, x, cos, sin):
    x, x_pass = x[..., :self.rotary_dim], x[..., self.rotary_dim:]
    x = (x * cos) + (rotate_half(x) * sin)
    return torch.cat((x, x_pass), dim=-1)

This differs from models like LLaMA, which apply rotation to all dimensions. Partial rotary allows the model to preserve position information encoding capability while maintaining a portion of "position-invariant" feature dimensions. Qwen3.5's model_type is routed into the Phi branch for RoPE processing:

if self.model_type in ['phi-msft', 'qwen3_5', 'qwen3_5_moe']:
    return self.phi_rotary_pos(x, cos, sin)

Note that partial rotary is only applied to Full Attention layers. LinearAttention layers do not use RoPE; position information is provided entirely by causal convolution.

5. Tokenizer Upgrade

Qwen3.5 significantly expands its vocabulary from the 151,936 tokens used throughout the Qwen series to 248,320, an increase of over 63%. A larger vocabulary better covers token distributions across multiple languages and specialized domains, reducing fragmentation during tokenization. This results in shorter token sequences for the same text, improving inference efficiency.

6. MTP (Multi-Token Prediction)

Qwen3.5's configuration includes mtp_num_hidden_layers: 1, indicating the model has built-in MTP (Multi-Token Prediction) capability. MTP is a speculative decoding technique -- building on standard LLMs that predict only the next token per step, additional MTP layers parallelly predict multiple subsequent tokens, generating several candidate tokens in a single forward pass, significantly boosting inference throughput.

The MTP module is a lightweight Decoder layer that takes the main model's final layer hidden states and the current token's embedding as input:

class MTP:
    def forward(self, input_embeds, hidden_states, attention_mask, position_ids, past_key_values):
        # Normalize and fuse hidden states and token embeddings
        input_embeds = self.token_layernorm(input_embeds)
        hidden_states = self.hidden_layernorm(hidden_states)
        hidden_states = self.input_proj(torch.cat([hidden_states, input_embeds], dim=-1))

        # A complete Decoder layer (Self-Attention + MLP)
        hidden_states = self.self_attn(hidden_states, ...)
        hidden_states = self.mlp(hidden_states)
        hidden_states = self.final_layernorm(hidden_states)

        # Share the main model's lm_head for prediction
        logits = self.lm_head(hidden_states)
        return logits

MNN's export framework has built a complete export channel for MTP (export_mtp), supporting exporting the MTP layer as an independent ONNX model and converting it to MNN format. The MTP layer shares lm_head weights with the main model and maintains its own independent KV Cache.

7. MNN Framework-Level Improvements

During the Qwen3.5 adaptation, MNN's LLM export framework underwent significant refactoring:

• Removed explicit past_key_values passing: KV Cache management changed from external model input to internal self-management (self.past_key_value), simplifying both export and inference pipelines.
• Simplified Attention output: The Decoder's forward return value changed from (hidden_states, present_key_value) to simply hidden_states.
• Unified Fused Attention export: All model types now have export_fused_attn enabled, without distinguishing MoE models.
• FusedAttention adds kv_cache attribute: Distinguishes standard Attention layers (which need KV Cache) from LinearAttention layers (which do not need KV Cache).

8. Quick Start

Exporting the Model

# Export Qwen3.5 model
python llmexport.py --path /path/to/model --export mnn --hqq

Inference

# Inference using MNN LLM engine
./llm_demo /path/to/exported/model/config.json prompt.txt

Summary

The transition from Qwen3 to Qwen3.5 represents the most significant architectural overhaul in the Qwen series to date. MNN's support for Qwen3.5 covers all of the model's core innovations:

1. DeltaNet Linear Attention: New LinearAttention operator implemented across CPU, Metal, OpenCL, and Vulkan backends; the 3:1 Linear + Full hybrid architecture reduces KV Cache to 1/4.
2. Gated RMSNorm: Extended RMSNorm to support gated modulation, applied to both linear attention and standard Attention layers.
3. MoE Shared Expert: Supports hybrid computation of shared experts and routed experts.
4. Native Multimodal: Adapts to Qwen3.5's default VLM architecture (based on Qwen3-VL, with Deepstack removed), supporting Interleaved M-RoPE and partial rotary position encoding.
5. Tokenizer Upgrade: Supports the expanded vocabulary from 151,936 to 248,320.
6. MTP Multi-Token Prediction: Accelerates generation via built-in MTP head for speculative sampling.

These efforts enable MNN to efficiently deploy Qwen3.5 series models on mobile and embedded devices.

Resource Links

• Github Commit: 1fdc7d3
• HuggingFace: https://huggingface.co/collections/taobao-mnn/qwen35-mnn
• ModelScope: https://modelscope.cn/collections/MNN/Qwen35-MNN

We welcome developers to try, provide feedback, and contribute, together advancing the development of on-device large language model inference technology.