[QNN] GPU support for executorch QNN delegate

[vybhav-ibr]

🚀 The feature, motivation and pitch

qnn sdk can address the GPU as well as NPU(per the docs)
Currently If i want to use both the GPU and NPU on a snapdragon chip, I would need to build my own arr with both the vulcan back-end and the QNN back-end, same with model quantitation.

Ideally the QNN must enable usage with both the GPU and the NPU

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @cccclai @cbilgin @abhinaykukkadapu

[abhinaykukkadapu]

@vybhav-ibr We were discussing this just recently to support QNN GPU as fallback, can we know more about your usecase, isn't QNN HTP sufficient for your usecase? CC: @shewu-quic

[ecccccsgo]

hello @abhinaykukkadapu, recently i am stduying the accuracy of quantilized models, I found low SQNR results with the recipe in qwen3-0.6b with SA8295:

class Qwen3_0_6BQuantRecipe(StaticLLMQuantRecipe):
    default_quant_dtype = QuantDtype.use_16a8w

    def __init__(self, verbose: bool = False):
        super().__init__()

        self.recipe = (
            QuantRecipe(
                self.default_quant_dtype,
                False,
                act_observer=MinMaxObserver,
                granularity=QuantGranularity.PER_TENSOR,
                verbose=verbose,
            )
            .add_node_target(
                {
                    torch.ops.aten.conv2d.default,
                },
                QuantDtype.use_16a8w,
                False,
                act_observer=MinMaxObserver,
                granularity=QuantGranularity.PER_CHANNEL,
            )
            .add_regex(
                {
                    r"layers\..*\.feed_forward\.w2_conv",
                },
                QuantDtype.use_16a8w,
                False,
                act_observer=MinMaxObserver,
                granularity=QuantGranularity.PER_CHANNEL,
            )
        )
        self.recipe.custom_quant_annotations.append(annotate_kv_8bit)

the sqnr are:

SQNR Eval Score between CPU QDQ and QNN: 9.11525821685791
SQNR Eval Score between FP32 nn.Module and CPU QDQ: 8.545727729797363

Some things I know for the low accuracy with the old v68 arch:

• the second input of matmul and mul (v73);
• dont support per block quant (v69);

and some other things:- quant layernorm with same w8a16

can I fallback layernorm or other sensitive layer to CPU/GPU in this case?
And can you share us some advice to increse the accuracy.

[vybhav-ibr]

Hi @abhinaykukkadapu,
I would like to run a 'large' llm/vlm on the HTP, and a small CV model(like yolo nano, may be even smaller) on the GPU, I am guessing this will make the inference faster for both the models,but frankly i don't know if this will work as expected. GPU fallback is also a valid reason

[yinrun]

I have another question: when using only HTP, what is the theoretical DDR bandwidth utilization it can reach? Roughly 60% to 70%?

[nil-is-all]

Hi @abhinaykukkadapu, I would like to run a 'large' llm/vlm on the HTP, and a small CV model(like yolo nano, may be even smaller) on the GPU, I am guessing this will make the inference faster for both the models,but frankly i don't know if this will work as expected. GPU fallback is also a valid reason

Hi @vybhav-ibr, does Vulkan work for your use case of running smaller CV models? Have you tried it out?

[abhinaykukkadapu]

hello @abhinaykukkadapu, recently i am stduying the accuracy of quantilized models, I found low SQNR results with the recipe in qwen3-0.6b with SA8295:

class Qwen3_0_6BQuantRecipe(StaticLLMQuantRecipe):
default_quant_dtype = QuantDtype.use_16a8w

def __init__(self, verbose: bool = False):
    super().__init__()

    self.recipe = (
        QuantRecipe(
            self.default_quant_dtype,
            False,
            act_observer=MinMaxObserver,
            granularity=QuantGranularity.PER_TENSOR,
            verbose=verbose,
        )
        .add_node_target(
            {
                torch.ops.aten.conv2d.default,
            },
            QuantDtype.use_16a8w,
            False,
            act_observer=MinMaxObserver,
            granularity=QuantGranularity.PER_CHANNEL,
        )
        .add_regex(
            {
                r"layers\..*\.feed_forward\.w2_conv",
            },
            QuantDtype.use_16a8w,
            False,
            act_observer=MinMaxObserver,
            granularity=QuantGranularity.PER_CHANNEL,
        )
    )
    self.recipe.custom_quant_annotations.append(annotate_kv_8bit)

the sqnr are:

SQNR Eval Score between CPU QDQ and QNN: 9.11525821685791
SQNR Eval Score between FP32 nn.Module and CPU QDQ: 8.545727729797363

Some things I know for the low accuracy with the old v68 arch:

• the second input of matmul and mul (v73);
• dont support per block quant (v69);

and some other things:- quant layernorm with same w8a16

can I fallback layernorm or other sensitive layer to CPU/GPU in this case? And can you share us some advice to increse the accuracy.

Yeah GPU fallback is never tested as QNN GPU is experimental, you can try fallback to CPU using skip_node_op_set / skip_annotation if you only care about accuracy.

Also for Float vs QDQ accuracy, i feel like you can try 16a8w with per channel default as it is a small model? Also try #18969 This is exactly the tool that auto-identifies which layers need to be promoted to higher precision. CC: @DannyYuyang-quic if you want feedback for the tool.