[Mobile] Dynamic Shape Challenge: Enabling LLM on QNN-HTP

[DakeQQ]

Describe the issue

In the ONNX Runtime documentation, it's stated that QNN-HTP doesn't support dynamic shapes.

My questions are:

1. Are the Slice, Split, and Gather operators considered dynamic shapes?
2. Is there a plan to update the dynamic shape computation? The QNN document now declares support for it.

According to both the ONNX Runtime and Qualcomm official documentation, QNN-HTP supports these operators. However, we encountered a failure during the compilation stage when testing an LLM model or a simple test model with the correct ONNX Runtime and QNN-HTP backend settings (Android 14, ORT==1.20.1, QNN_SDK=2.28.*, HTP=v73, SOC=43).

We aim to run an LLM model on QNN-HTP, but we're having trouble with dynamic input/output.

Attempts made:

1. Using a dynamic shape input tensor resulted in expected compilation failure.
2. Using a static shape input tensor with Slice, Split, and Gather operators also failed during compilation.
3. Using a static large shape input tensor with a mask (processing the full tensor regardless of input word count) led to heavy computation and extremely low performance.
4. Modifying the code to process with static shapes on HTP while moving other operations to the CPU resulted in processing one token at a time during prefill stage, which was extremely slow.

We are looking for solutions or ideas to address these issues and believe the Microsoft team is brilliant and knowledgeable about ONNX Runtime.

To reproduce

# Export the Slice model for testing Android-QNN-HTP backend.

import torch
import torch.nn as nn


class SliceModel(nn.Module):
    def forward(self, data, start):
        return data[:, :, :, start:]   # Slice the data from the 'start' to the end.


# Instantiate the model
model = SliceModel()
exported_path = "slice_model.onnx"

# Create dummy inputs:
data = torch.zeros((1, 1, 1, 100), dtype=torch.int16)   # NCHW format
start = torch.tensor([10], dtype=torch.int64)

# Export the model to ONNX
torch.onnx.export(
    model,
    (data, start),
    exported_path,
    input_names=["data", "start"],
    output_names=["slice_output"],
    opset_version=17
)

print("Exported the model to slice_model.onnx")

Urgency

No response

Platform

Android

OS Version

14

ONNX Runtime Installation

Built from Source

Compiler Version (if 'Built from Source')

CMake=3.31.2, NDK=26.3.*

Package Name (if 'Released Package')

onnxruntime-android

ONNX Runtime Version or Commit ID

1.20.1

ONNX Runtime API

C++/C

Architecture

ARM64

Execution Provider

Other / Unknown, SNPE

Execution Provider Library Version

QNN SDK 2.28.*

[skottmckay]

Dynamic shape isn't operator specific. It's whether the ONNX model has fixed or dynamic dimensions for an input. You can view the ONNX model in Netron to check.

e.g. in this model the input and output has a dynamic batch_size dimension that could be any value at runtime.

If you know it will always be say 1 during inferencing you can update the model to replace batch_size with 1 so the shape is fixed.

https://onnxruntime.ai/docs/tutorials/mobile/helpers/make-dynamic-shape-fixed.html

[HectorSVC]

QNN only has limit support for dynamic shape, especially with HTP. ORT QNN EP doesn't support dynamic shape for now.
Could you share which LLM model you are working one. We recommend to try to run the part which you can make it static shape. For example the Transformer part, maybe the 1st token gen use seq_len = 64, the rest of token gen use seq_len = 521. Then you can create to models with this static shape.

[brian-at-pieces]

If dyanmic shapes aren't allowed, then are the input and ouput sequence lengths fixed for the Phi and Llama models from the docs page: Run SLMs on Snapdragon devices with NPUs ? If this were the case I'd expect latency and memory to stay constant, but I'm seeing it scale with input size.

[DakeQQ]

Good day~ @HectorSVC
Thanks for the reply. We are trying to get Qwen running on QNN-HTP. Although most large language models (LLMs) consist of operations like matmul, add, transpose, and reshape, I believe there isn’t much difference between them.

My key question is this: while it's possible to set a static shape on the NPU and run other operations on the CPU, this approach incurs significant costs related to CPU-RAM-NPU communication and data transfer. Even with I/O binding, transferring data between the CPU and NPU is still necessary. Therefore, we are looking for a solution that can fully utilize the NPU for all operations.

Would there be any additional bold ideas or suggestions that could be shared?