Register a Cortex-M flow in the backend test suite - #21854
Conversation
The shared suite at backends/test/suite runs 23 torchvision and torchaudio models against every other major backend; Cortex-M had none, so its model coverage was hand-written one file at a time. This adds one, running on the Corstone-300 FVP. Five models pass and 15 are listed in CORTEX_M_SKIPS with a comment recording why each fails; read that list as a to-do. A pass means the model lowered and ran rather than that it was accurate, for the reason the module docstring gives. The suite drives to_edge_transform_and_lower, so CortexMTester grows a stage of that name: to_edge with no partitioner followed by CortexMPassManager, since Cortex-M rewrites operators in place rather than delegating a subgraph. The flow converts 4D inputs to channels_last, without which CortexMConv2DCheck rejects every convolution in every model and raises the FVP timeout, which an ImageNet-sized model exceeds at the 120 second default. CI goes through test_backend.sh like every other backend, path-gated because every case installs the Arm toolchain, builds a runner and drives the FVP. The operator suite is excluded and the exclusion says why. Authored with Claude Code.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21854
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit 584deba with merge base d18ca0f ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
Erik-Lundell
left a comment
Erik-Lundell
left a comment
There was a problem hiding this comment.
A couple of comments
Review feedback: a skip list is the wrong way to record what does not work. The suite already classifies every failure and writes the category, the runtime error, the undelegated op histogram and the SNR into the job summary, so the reasons in that list were duplicating generated output with something that cannot stay accurate -- and had not. Two of the fifteen entries were already wrong: efficientnet_b4 and efficientnet_v2_s pass, and the list was hiding it. Conformer's first failure had moved as well. So they run now, and are marked xfail instead. An xfailed case still executes, and pytest_json_runtest_metadata reads the results whatever the outcome, so every one of them keeps its row in the report; only the verdict changes. The marker is strict, which is the part that makes this self-correcting: the two efficientnets would have been reported the day they started passing rather than sitting in a list for weeks. Three stay skipped, because running them teaches nothing. vit_b_16 and wide_resnet50_2 are over the runner's 60 MiB pool at 86.6M and 68.9M parameters, which no amount of lowering fixes -- vit_b_16 already measures 1.01 bytes per parameter. Conformer is not a size problem at all: its program is under a megabyte and it is the activation arena that overruns, crossing the pool at a sequence length of about 353 while the test draws lengths from randint(1, 400). It fits roughly one run in three, so it cannot be a strict xfail. wav2letter looked like the same class of problem and is not -- at exactly 4.00 bytes per parameter none of its convolutions lower, and at int8 it would be 26 MiB, so it is an xfail waiting on conv1d. xfail_patterns is the only shared change: one field on TestFlow beside skip_patterns, and one branch in pytest_collection_modifyitems. Other flows default to an empty list. Also, on the same review: the flow docstring said the host runtime carries no cortex_m kernels, which read as a claim about the FVP runner rather than the point, which is that CMSIS-NN kernels are compiled for the device and cannot run on a host CPU at all. The runner does register the cortex_m kernels, and the Ethos-U flows reach the FVP the same way. And the timeout the flow passes is the FVP's --timelimit, not a serialization budget; it is now 900s, under the suite conftest's 1200s pytest timeout, so an overrun is reported by the FVP rather than killed by pytest. Authored with Claude Code.
| StageType.QUANTIZE: CortexMQuantize, | ||
| StageType.RUN_PASSES: CortexMRunPasses, | ||
| StageType.TO_EDGE: CortexMToEdge, | ||
| StageType.TO_EDGE_TRANSFORM_AND_LOWER: CortexMToEdgeTransformAndLower, |
| uses: ./.github/workflows/_get-changed-files.yml | ||
|
|
||
| test-cortex-m: | ||
| needs: changed-files |
There was a problem hiding this comment.
nice to have this filter, do you know how long this takes on an FVP?
There was a problem hiding this comment.
Looks like total job time was ~20m: https://github.com/pytorch/executorch/actions/runs/32047574862/job/95438943767?pr=21854
But I would add that there are many tests skipped right now (notably the operator tests). Definitely something to keep an eye on and remove from pull requests if it gets too long.
Erik-Lundell
left a comment
Erik-Lundell
left a comment
There was a problem hiding this comment.
I am still unsure why the job turns red when tests are failing, I believe we have many fails in the Ethos-U and vgf tests.. But thanks for addressing my comments, lgtm
Do you mean why the job doesn't turn red? I think this is because the test suite is designed to generate a report and not gate CI. It looks like there are a couple failures on the Arm backend: https://github.com/pytorch/executorch/actions/runs/32047574626?pr=21854 |
3 participants
Summary
The shared suite at backends/test/suite runs 23 torchvision and torchaudio models against every other major backend; Cortex-M had none, so its model coverage was hand-written one file at a time. This adds one, running on the Corstone-300 FVP. Five models pass and 15 are listed in CORTEX_M_SKIPS with a comment recording why each fails; read that list as a to-do. A pass means the model lowered and ran rather than that it was accurate, for the reason the module docstring gives.
The suite drives to_edge_transform_and_lower, so CortexMTester grows a stage of that name: to_edge with no partitioner followed by CortexMPassManager, since Cortex-M rewrites operators in place rather than delegating a subgraph. The flow converts 4D inputs to channels_last, without which CortexMConv2DCheck rejects every convolution in every model and raises the FVP timeout, which an ImageNet-sized model exceeds at the 120 second default.
Test plan
CI goes through test_backend.sh like every other backend, path-gated because every case installs the Arm toolchain, builds a runner and drives the FVP. The operator suite is excluded and the exclusion says why.
Authored with Claude Code.