Explore Executorch as another model conversion option

[himanshunaidu]

Executorch:

https://pytorch.org/executorch/stable/index.html

[himanshunaidu]

Introduction

ExecuTorch is an end-to-end solution for enabling on-device inference capabilities across mobile and edge devices including wearables, embedded devices and microcontrollers.

Why Executorch

It is PyTorch. It supports export, quantization, Core ATen ops.
It supports delegation (core of how ExecuTorch works) that enables partnerships with different companies to support all kinds of edge devices.
Different lowering techniques and paths to hardware and optimized kernels
Has a first-party SDK support to help debug.

Overview

First uses torch.export to generate a semantically equivalent graph. Then the graph is compiled to target a specific hardware to generate a Executorch program.

Benefits

1. Export-based instead of torchscript based
2. No intermediate conversion for any platform
3. Modularity
4. 3rd Party Compilers (The partnering vendors including Apple support the Pytorch 2.x export Intermediate Representation)
5. Portable and Lightweight Runtime

Further Reading (if you have access): https://www.notion.so/Executorch-1d22472f25ef805f8973debe562d21b5?pvs=4

[himanshunaidu]

Issues (Some are fixed)

1. Not sure how reliable this would be in tandem with Apple, thanks to the delegates concept
   The support of this package obviously depends on Apple's support of the backend delegate

2. The entire conversion and runtime process doesn’t seem to be as seamless as coremltools for iOS (FIXED)
   CoreMLTools gives us the .mlpackage format, which is much easier to use with both Swift and Objective C++. One can also use Xcode to directly get the performance metrics.
   For the Executorch .pte format on the other hand, one can only use Objective C++ and requires a lot of dependencies. One cannot use Xcode to get any performance metrics for .pte files.

Turns out we CAN extract the .mlpackage format from the .ple file.

3. The library still has a long way to go before being a stable package

4. Noticeable performance drop during conversion (FIXED)
   Several warnings and noticeable accuracy drop when running semantic segmentation model through CoreML backend pytorch/executorch#10179

[himanshunaidu]

Conclusion (Current)

Until we are using only iOS, CoreMLTools is definitely the better option for deploying models to iOS devices. It's specifically built for Apple's ecosystem and supports seamless integration with Core ML, which is designed to leverage Apple’s hardware acceleration.

Executorch would be more appropriate if you're looking to deploy PyTorch models on edge devices that aren't necessarily part of the Apple ecosystem (e.g., microcontrollers, Raspberry Pi, etc.). While you could technically try to use Executorch for iOS, it wouldn’t provide the level of optimization, ease of integration, or hardware acceleration that Core ML offers.

[himanshunaidu]

Weirdly enough, CoreML and Executorch have some difference in how they handle the scale and bias
Executorch, predictably enough, requires 1/255 scaling only if the input is scaled to 255
CoreML, if we use ImageType for input, seems to require the scaling even if the input is not scaled to 1
CoreML, if the input is scaled to 255, even then it requires the scaling
It is possible that internally, CoreML is scaling the input to 1/255 if the input is not scaled to 1