Build QNN in android_llm_demo.sh and perf #5105
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5105
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit c9bf893 with merge base 83d92ff ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
| PYTHON_EXECUTABLE=python bash .ci/scripts/setup-linux.sh cmake | ||
| export ARTIFACTS_DIR_NAME=artifacts-to-be-uploaded | ||
| if [[ ${{ matrix.delegate }} == "qnn" ]]; then |
There was a problem hiding this comment.
I just notice that the job fail https://github.com/pytorch/executorch/actions/runs/10713455099/job/29705837269#step:12:6342, you will need to define the matrix strategy for this job with
strategy:
matrix:
delegate: ${{ fromJson(needs.set-parameters.outputs.delegates) }}
fail-fast: false
There was a problem hiding this comment.
I just realized that we're building backend-specific apps in the android benchmark workflow and it seems like the changes are introduced here. @kirklandsign @huydhn I think we should not build different apps for each backend. There are several reasons for it:
- It will complicate the benchmarking workflow as you will have to add additional logics to match model with the app. Because we are benchmarking the perf of a model instead of the app, the app size is not a concern in this case.
- Delegates to hybrid backends will be supported for better perf at some point though today nobody is proactively working on it. Ideally QNN unsupported ops should fallback to XNNPACK instead of Portable, otherwise we may end up with QNN delegated model run slower than a pure CPU model w/ XNNPACK. Our stack should have supported building an app with multiple backends/delegates.
There was a problem hiding this comment.
This is an addition to xnnpack when the QNN_SDK_ROOT is set
(see https://github.com/pytorch/executorch/pull/5105/files#diff-20ef4ec16dbbb8f0311f92eb54751e47274a250335422f71a5107c4a053a1275R44-R45)
There was a problem hiding this comment.
Right now we build this XNNPACK+QNN when matrix has qnn. https://github.com/pytorch/executorch/blob/main/.github/workflows/android-perf.yml#L147-L150
We can build a single flavor (XNNPACK+QNN+otherbackends) regardless of matrix as well
There was a problem hiding this comment.
The requirement to do always build QNN is we need to run
PYTHON_EXECUTABLE=python bash .ci/scripts/setup-qnn-deps.sh
PYTHON_EXECUTABLE=python bash .ci/scripts/build-qnn-sdk.sh
all the time. Probably takes some time and stability. Ideally we have a docker image.
|
Error: File /tmp/qnn/include/QNN/QnnTypes.h does not exist. |
Should not export separately! .ci/scripts/build-qnn-sdk.sh: export QNN_SDK_ROOT=/tmp/qnn/2.23.0.240531
|
I'm not sure whether the error is from this PR or already there. https://github.com/pytorch/executorch/actions/runs/10715134533/job/29711462394 |
No description provided.