(CMSIS-NN Integration) Testing strategy #13902
Description
Let's use this issue to align on the testing for the Cortex-M backend. I suggest the following tests, but I am happy to hear other opinions:
Kernel tests
All kernels should have GTests similar to the quantization ops to separate testing kernel logic from the full executorch framework. These tests should be easily be built and run on an FVP through some helper script. This should simplify debugging of kernels significantly.
Operator integration tests
We should write a cortex_m implementation of https://github.com/pytorch/executorch/blob/main/backends/test/harness/tester.py to get the kernel integration tests for free, stealing some code from the arm backend to run pte:s on FVP using semihosting.
Python code tests
We should have some level of unittesting of the python code. We can reuse the tester for easily counting the expected number of ops before/ after passes. For testing the dialect ops we should write some helper functions for asserting that they produce expected shapes and results, if they do not exist already?
Network tests
In principle we can reuse the tester framework also for running full network tests instead of having to keep integrating cortex_m support into the run.sh script/ arm_aot_compiler. We should still have some proper full flow test which is also useful as an example for users, I would suggest something close to the https://github.com/pytorch/executorch/blob/main/examples/arm/ethos_u_minimal_example.ipynb but for cortex_m in that case.
Thoughts?