[CoreML EP] Implement Unary & Reduce operators

[ShukantPal]

Description

This change is a follow-up to #15327. It adds Unary operators (Sqrt, Reciprocal) and Reduce operators (ReduceSum, ReduceMean). I've tried to follow existing patterns in the code :-)

Motivation and Context

This reduces fragmentation across EPs when using CoreML on macOS, thereby speeding up execution.

[edgchen1]

Thanks for your contribution!

[ShukantPal]

Thanks for your review @edgchen1 ! I've followed up on all of your comments/suggestions. Please let me know how it looks now.

[edgchen1]

/azp run MacOS CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[natke]

Hi @ShukantPal, thank you for this contribution! We're interested in learning more about your use case. Can you tell us a little bit about how you are using ONNX Runtime? Or if you would prefer you can reach me at nakersha@microsoft.com

[ShukantPal]

Hi @natke, I'm developing a goofy macOS virtual camera that uses different video filters like FaceMesh, CenterFace, DFL, etc. To get real-time frame rates, executing on CoreML / ANE is necessary on M1 MacBooks. I wanted to keep my application cross-platform to run on Intel Macs (with discrete GPUs) and Windows in the future. That's why I chose ONNX runtime, but having full CoreML support is still necessary for satisfactory performance on M1.

[natke]

Sounds pretty awesome. Is it published anywhere? I'd love to see a demo

[ShukantPal]

@natke Haha, not yet − I've been working on it. Can send you a beta build when ready :-)

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[ShukantPal]

• I ran the linter on Windows and fixed the linting issues.
• It seems like TensorRT, OpenVINO, and DNN don't support ReduceMean opset 18 with axes input, so I disabled those providers in the new tests (copying that from similar tests for ReduceMean).

But I need help understanding why the test_layer_normalization* tests are failing on CoreML/macOS. The ReduceMean op-tests pass, but still there's some mismatch with downstream "Reshape" layers in the test models.

I don't have this issue when testing the ReduceMean layers in my own models.

[ShukantPal]

@skottmckay

Given the testing complications, I've commented out the line registering ReduceMean. Hopefully, rest of the PR provides enough value for it to be merged :-)

[skottmckay]

Can you clarify which tests are failing?

I pulled your changes, uncommented the ReduceMean line in onnxruntime/core/providers/coreml/builders/op_builder_factory.cc and did a build on a mac with CoreML enabled and the unit tests passed.

[skottmckay]

/azp run Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,MacOS CI Pipeline

[skottmckay]

/azp run orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-python-checks-ci-pipeline,onnxruntime-binary-size-checks-ci-pipeline

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[ShukantPal]

@skottmckay Here were the failing tests: https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=1012534&view=logs&j=07136875-e4b3-5a15-ff50-026d13f57d01&t=b16e8f5d-fb27-5dd3-278d-cb1fb09fb81a

They come from the --build_wheel option.

[skottmckay]

Definitely want to get it added, but I think we need to figure out the root cause given we're calling AddReductionParams the same way for both ReduceMean and ReduceSum, so if there's an issue I would expect it potentially applies to both operator types and it's possibly that there's no test showing the issue for ReduceSum.

The failing tests are ONNX test cases. We create a binary called onnx_test_runner that can be used to execute/debug. The ONNX tests are in ./cmake/external/onnx/onnx/backend/test/data/node. Specify the EP with '-e'. CPU passes, CoreML fails.

e.g. ./build/MacOS/Debug/onnx_test_runner -e coreml ./cmake/external/onnx/onnx/backend/test/data/node/test_layer_normalization_2d_axis0_expanded

I don't think this is the root cause, but the last param for the CoreML parms is reduceAll which might not be 1:1 with noop_with_empty_axes.

https://apple.github.io/coremltools/mlmodel/Format/NeuralNetwork.html#reducemeanlayerparams

CoreML reduceAll says to ignore the axes parameter and reduce all.

So if axes is empty, reduceAll is true, but if noop_with_empty_axes is also true the ONNX spec says do nothing which doesn't seem to have an equivalent in CoreML. In that case we could probably drop the node in CoreML or insert an Identity node to do the value rename given it's turned into a no-op.

[skottmckay]

Sorry - I overlooked the early exit for noop_with_empty_axes. It would be good to clarify the implementation in ReductionOpBuilder::AddToModelBuilderImpl though and set a reduceAll value instead of passing noop_with_empty_axes into the AddReductionParms call.

[skottmckay]

Might be an issue with Flatten. There's not a lot of places in this particular test where this combination of shapes is possible to get wrong. One path is the Flatten of the model input, which should result in (1, 4) not (4, 1)

2023-05-19T06:58:33.0585720Z ERROR: test_layer_normalization_2d_axis1_expanded_cpu

2023-05-19T06:58:33.0593450Z onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Error compiling model compiler error: Espresso exception: "Invalid blob shape": generic_elementwise_kernel: cannot broadcast:
2023-05-19T06:58:33.0593910Z (3, 4)
2023-05-19T06:58:33.0594210Z (4, 1)

I ran a couple of tools to a) add shape info to all the nodes, and b) run the 'basic' level optimizations like constant folding so we can see what nodes are around when the CoreML EP is creating the model.

python -m onnxruntime.tools.symbolic_shape_infer --input D:\src\github\ort\cmake\external\onnx\onnx\backend\test\data\node\test_layer_normalization_2d_axis1_expanded\model.onnx --output D:\src\github\ort\cmake\external\onnx\onnx\backend\test\data\node\test_layer_normalization_2d_axis1_expanded\model.ssi.onnx
python -m onnxruntime.tools.optimize_onnx_model --opt_level basic D:\src\github\ort\cmake\external\onnx\onnx\backend\test\data\node\test_layer_normalization_2d_axis1_expanded\model.ssi.onnx D:\src\github\ort\cmake\external\onnx\onnx\backend\test\data\node\test_layer_normalization_2d_axis1_expanded\model.ssi.basic.onnx

If I comment out Flatten in op_builder_factory.cc the tests pass.

[skottmckay]

And this is the issue:

onnxruntime/onnxruntime/core/providers/coreml/builders/impl/flatten_op_builder.cc

Line 44 in 684e900

const int64_t axis = helper.Get("axis ", 1);

Rogue space in the attribute name so it wasn't reading the actual value from the node of 0.

I'll add a test and create a separate PR for that fix, but please test out your changes with just the fix ("axis " -> "axis") so we can check all CIs pass with that.

[skottmckay]

Should be addressed by #16046.

[ShukantPal]

I can confirm that patching the rogue space in Flatten fixes this! Amazing.

[skottmckay]

/azp run Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,MacOS CI Pipeline

[skottmckay]

/azp run orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-python-checks-ci-pipeline,onnxruntime-binary-size-checks-ci-pipeline

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[skottmckay]

/azp run Windows ARM64 QNN CI Pipeline,Linux QNN CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).