Skip to content

NXP backend: Resolve limitations of uncertain tensor formats. #14576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

NXP backend: Resolve limitations of uncertain tensor formats. #14576

wants to merge 7 commits into from

Conversation

MartinPavella
Copy link
Collaborator

@MartinPavella MartinPavella commented Sep 25, 2025 •
edited
Loading

Summary

This PR resolves format related issues by inferring the format (NCHW/NHWC) for all nodes before partitioning. These formats are then used by the NeutronPartitioner to accurately determine which nodes are supported on Neutron.

Test plan

Unit tests provided, and correct function is tested by nearly every test in the nxp backend.

cc @kimishpatel

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Sep 25, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14576

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 1ecfe36 with merge base 70ea661 (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 25, 2025
@MartinPavella
Copy link
Collaborator Author

@pytorchbot label "module: nxp" "release notes: nxp"

@pytorch-bot pytorch-bot bot added module: nxp Issues related to NXP Neutron NPU delegation and code under backends/nxp/ release notes: nxp Changes to the NXP Neutron backend delegate labels Sep 25, 2025
This was referenced Sep 25, 2025
Merged
Merged
Draft
Draft
kimishpatel
kimishpatel reviewed Sep 26, 2025
View reviewed changes
backends/nxp/TARGETS
deps = [
":neutron_sdk",
":aten_passes",
":_passes",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add neutron_backend build to this https://github.com/pytorch/executorch/blob/main/.ci/scripts/unittest-buck2.sh. Thats the one that was failing last time. This way we get signal on the pr itself

@facebook-github-bot
Copy link
Contributor

@kimishpatel has imported this pull request. If you are a Meta employee, you can view this in D83311730.

@kimishpatel
Copy link
Contributor

i have imported it internally. There is an nxp test failing as well

digantdesai
digantdesai approved these changes Sep 26, 2025
View reviewed changes
Copy link
Contributor

@digantdesai digantdesai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.

@MartinPavella MartinPavella force-pushed the upstream/main-nxp/EIEX-392-resolve-limitations-of-uncertain-tensor-formats branch 2 times, most recently from bb48aad to e1893f5 Compare October 1, 2025 07:41
@MartinPavella MartinPavella marked this pull request as draft October 1, 2025 12:22
@MartinPavella
Copy link
Collaborator Author

MartinPavella commented Oct 1, 2025 •
edited
Loading

@kimishpatel I have fixed the failing nxp test, and added the nxp backend build to the unittest-buck.sh. The build is now failing with:

1: From load at backends/nxp/TARGETS:4
2: File not found: shim_et//arvr/tools/build_defs/oxx_python.bzl

These lines were in the file already before my changes. Do you know what is causing the issue/how to fix it?

@kimishpatel
Copy link
Contributor

@kimishpatel I have fixed the failing nxp test, and added the nxp backend build to the unittest-buck.sh. The build is now failing with:

1: From load at backends/nxp/TARGETS:4 2: File not found: shim_et//arvr/tools/build_defs/oxx_python.bzl

These lines were in the file already before my changes. Do you know what is causing the issue/how to fix it?

I guess it is triggering your build in oss and we have never really had this test in oss so its trying to pull in internal files. I would say roll back that change for now just to be able to land.

@MartinPavella MartinPavella force-pushed the upstream/main-nxp/EIEX-392-resolve-limitations-of-uncertain-tensor-formats branch from e1893f5 to 2672277 Compare October 3, 2025 06:58
@MartinPavella
NXP backend: Store inferred node format in the node.meta.
92fb3f8
@MartinPavella
NXP backend: Update pass which removes GetItem nodes, to preserve the…
ce2553f 
… node format.

The pass `RemoveGetItemPass` replaces a `max_pool2d_with_indices` node with a `max_pool2d` node, that doesn't require a GetItem afterward. The new operator must, however, preserve the original node format. Therefore, a copy of the pass was created in `backends/nxp/_passes`, where it was modified. The new directory was created, because the pass doesn't follow the `NeutronEdgePass` interface.
@MartinPavella
NXP backend: Perform node format inference before partitioning.
e62c9aa 
Before, the format inference was done during conversion to NeutronIR (after partitioning), so the partitioner didn't yet know the formats. Now, the partitioner has the format data, which can be used to accurately select nodes for delegation.
@MartinPavella
NXP backend: Improve cat delegation by using inferred node formats.
54bfece
@MartinPavella
NXP backend: Improve constant_pad_nd delegation by using inferred n…
b3d6433 
…ode formats.
@MartinPavella
NXP backend: Do not infer format for unknown nodes.
29f1137
@MartinPavella
Fix rebase.
1ecfe36
@MartinPavella MartinPavella force-pushed the upstream/main-nxp/EIEX-392-resolve-limitations-of-uncertain-tensor-formats branch from 2672277 to 1ecfe36 Compare October 3, 2025 07:15
@MartinPavella MartinPavella marked this pull request as ready for review October 3, 2025 08:34
@MartinPavella
Copy link
Collaborator Author

@kimishpatel I have fixed the failing nxp test, and added the nxp backend build to the unittest-buck.sh. The build is now failing with:
1: From load at backends/nxp/TARGETS:4 2: File not found: shim_et//arvr/tools/build_defs/oxx_python.bzl
These lines were in the file already before my changes. Do you know what is causing the issue/how to fix it?

I guess it is triggering your build in oss and we have never really had this test in oss so its trying to pull in internal files. I would say roll back that change for now just to be able to land.

@kimishpatel thanks for the response. I have removed the build as you said. Can you please now verify that your internal build passes?

@kimishpatel
Copy link
Contributor

doing

@MartinPavella
Copy link
Collaborator Author

doing

@kimishpatel any updates? Is the build passing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kimishpatel kimishpatel kimishpatel left review comments

@digantdesai digantdesai digantdesai approved these changes

@robert-kalmar robert-kalmar Awaiting requested review from robert-kalmar robert-kalmar is a code owner

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: nxp Issues related to NXP Neutron NPU delegation and code under backends/nxp/ release notes: nxp Changes to the NXP Neutron backend delegate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@MartinPavella @facebook-github-bot @kimishpatel @digantdesai