Qualcomm AI Engine Direct - gpu support part1

[haowhsu-quic]

Summary

• rename folders in backends/qualcomm/runtime/backends
• add gpu infra

Test plan

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNFloatingPointOperator.test_qnn_backend_conv2d -b build-android/ -m SM8750 -s 5f396958 --online_prepare --backend gpu

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12165

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit 5a7b62b with merge base 1fe59c8 ():

NEW FAILURE - The following job has failed:

• pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
  RuntimeError: Command docker exec -t d084affc2af0048404b315e23131ec9d06dd9a48c32744f3a9a8883327b2c1e6 /exec failed with exit code 1

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

• pull / android / run-emulator (gh) (#16137)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[haowhsu-quic]

@pytorchbot label "release notes: qualcomm"

[cccclai]

I'm slightly worried about the runtime size increase, that usually is a requirement for production. Do we know how much size increase with this PR? If I have a model runs on HTP only, can the runtime include HTP only?

[haowhsu-quic]

The libqnn_executorch_backend.so grows from 630984 to 652672 bytes. We'll deprecate few files in next PR, hopefully it could further reduce the number.

[cccclai]

What files will be deprecated in next PR?

[haowhsu-quic]

I think it will be aot/ir and runtime/backend/CustomProtocol*. We now switch to QNN IR backend (DLC) for online-prepare path, the qcir and the legacy code for multi-method compilation can be fully deprecated.
But it would break backward compatibility since we used to wrap preprocess result with custom protocol. Probably will let you to decide when will be the right time to apply the change.

[haowhsu-quic]

Hi, I was thinking wrong about the impact of deprecating files. We still need to keep the custom protocol implementation to make multi-graph path work.
The change is in #12583 now and will guarantee BC.

[cccclai]

Sorry I need to spend a bit more time on this, because we don't have CI to test the pllm model and I'm worried it will cause breakage

[haowhsu-quic]

Sorry I need to spend a bit more time on this, because we don't have CI to test the pllm model and I'm worried it will cause breakage

No worries, I think GA decoder models is way more important than this. This PR is mainly a proof of concept that we can extend the capability of QNN backend.

[cccclai]

Can we prioritize the stories.pte as part of CI to prevent BC breakage? Otherwise it's hard to catch failure

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[cccclai]

We should be good to continue this PR, what do you think?

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[cccclai]

Should we rebase this PR are land it?

[haowhsu-quic]

Should we rebase this PR are land it?

We're doing some final checking and will submit PR next week. We will also verify all the enabled models and try to support them with GPU after this PR.

[meta-codesync]

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D87936306.

[cccclai]

I'm getting the error message

executorch/backends/qualcomm/tests/utils.py", line 222, in get_backend_type
return getattr(QnnExecuTorchBackendType, f"k{self.backend.title()}Backend")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'QnnExecuTorchBackendType' has no attribute 'kBackend'. Did you mean: 'kDspBackend'?

Can you help fixing it?

[chenweng-quic]

I'm getting the error message

executorch/backends/qualcomm/tests/utils.py", line 222, in get_backend_type
return getattr(QnnExecuTorchBackendType, f"k{self.backend.title()}Backend")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'QnnExecuTorchBackendType' has no attribute 'kBackend'. Did you mean: 'kDspBackend'?

Can you help fixing it?

Hi @cccclai,
Could you provide the command to reproduce the issue?
The self.backend supposed to not become an empty string and cause the error per my understanding.

[cccclai]

Sorry was out for 3 days last week, seems like you push new commits, let me check again

[cccclai]

It seems like an internal test failing, I will forward fix