Qualcomm AI Engine Direct - Suite Operator Test Support (Part1)

[winskuo-quic]

Summary

• Support Add/Sub with alpha values
• Support Conv3d
• Support TransposeConv3d

Test plan

UT added

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14618

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [Maintenance] MacOS runners update

❌ 5 New Failures

As of commit d6c6a28 with merge base 5fd66ee ():

NEW FAILURES - The following jobs have failed:

• pull / test-vulkan-operators-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 449b97a7183d5771ec96c8cf2d93179b33e8004de77a9e399ca80e6b540bb2b6 /exec failed with exit code 127
• Test CUDA Builds / check-all-cuda-builds (gh)
  Process completed with exit code 1.
• Test CUDA Builds / test-executorch-cuda-build-12.6 / linux-job (gh)
  RuntimeError: Command docker exec -t a5b6928392ebf1fe2d9267ccf875d3872da9921af0cb8b186fcb47e46c030e59 /exec failed with exit code 1
• Test CUDA Builds / test-executorch-cuda-build-12.8 / linux-job (gh)
  RuntimeError: Command docker exec -t e2d6156414a2057682d3aea19ab1e47de0145d4824188a07646317ba72c08183 /exec failed with exit code 1
• Test CUDA Builds / test-executorch-cuda-build-13.0 / linux-job (gh)
  RuntimeError: Command docker exec -t 62d82199561f6fd8fc4e263417db770f5e521eb559cab78ac3b28fd6651701eb /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D83314665.

[winskuo-quic]

Hi @cccclai,
This PR should resolve some suite operator's failure.
It would be appreciated if you could guide me through the following:

1. How could I see the most up-to-date suite operator results in mainline, like the pdf file you shared with us through email.
2. Could you share the command to run the suite operator tests? The command I use is:
   python -m executorch.backends.test.suite.runner operators --flow qnn --filter "test_conv3d_basic_qnn".
   I can see op validation and preprocess is passing, however, I keep getting an unknown error at the end, causing the test to skip, as shown in the image below:

[cccclai]

Hi @cccclai, This PR should resolve some suite operator's failure. It would be appreciated if you could guide me through the following:

1. How could I see the most up-to-date suite operator results in mainline, like the pdf file you shared with us through email.
2. Could you share the command to run the suite operator tests? The command I use is:
   python -m executorch.backends.test.suite.runner operators --flow qnn --filter "test_conv3d_basic_qnn".
   I can see op validation and preprocess is passing, however, I keep getting an unknown error at the end, causing the test to skip, as shown in the image below:

@GregoryComer can you share the steps?

[cccclai]

@winskuo-quic from the latest run, it seems the number improves https://github.com/pytorch/executorch/actions/runs/18053810243

[cccclai]

@winskuo-quic from the latest run, it seems the number improves https://github.com/pytorch/executorch/actions/runs/18053810243

[winskuo-quic]

@pytorchbot cherry-pick --onto release/1.0 -c regression

[pytorchbot]

Cherry picking #14618

The cherry pick PR is at #14742 and it is recommended to link a regression cherry pick PR with an issue. The following tracker issues are updated:

• [v1.0.0] Release Tracker #14288 (comment)

Details for Dev Infra team

Raised by workflow job

[GregoryComer]

https://github.com/pytorch/executorch/actions/runs/18053810243

Thanks for the fixes! I updated the README here with some more repro examples: https://github.com/pytorch/executorch/blob/main/backends/test/suite/README.md. I've also updated to use pytest, which should make it easier to run the tests.

Regarding skips on conv3d, that's effectively a pass. What's happening is that the operator or model has an undelegated conv3d, which we don't have a portable implementation for. If nothing is delegated, it gets considered a pass, as the backend worked as intended. If it's partially delegated and has missing portable ops, it's considered skipped as it can't test the model. But, from the backend perspective, it's passing. I should maybe just update the test logic to mark as pass instead of skip for this case.

[cccclai]

I should maybe just update the test logic to mark as pass instead of skip for this case.

@GregoryComer yes let's do that