[Draft] Initial implementation of executorch tensor filter

[leeeryboy]

Future Work and Implementation

This draft pull request lays the groundwork for integrating the executorch tensor filter into nnstreamer. Although the current implementation is not yet complete and might have some errors or inefficiencies, we believe it can serve as a valuable starting point and reference for future work on this feature.

Build and Performance Considerations

During the development of this draft PR, we encountered significant performance issues with the current stable branch of executorch (v0.2.1). Despite our efforts to investigate and resolve these issues, we were unable to identify the root cause of the performance degradation.

As a result, we made the decision to build the executorch main branch instead of the stable v0.2.1 branch. While we acknowledge that using the main branch may introduce instability and potential compatibility issues, we found that it provided better performance compared to the stable branch.

Building ExecuTorch:

# Assume conda is installed

# Create and activate a conda environment named "executorch"
conda create -yn executorch python=3.10.0
conda activate executorch

# Clone the ExecuTorch repo from GitHub
git clone https://github.com/pytorch/executorch.git
cd executorch

# Update and pull submodules
git submodule sync
git submodule update --init

# Install ExecuTorch pip package and its dependencies, as well as
# development tools like CMake.
# If developing on a Mac, make sure to install the Xcode Command Line Tools first.
./install_requirements.sh

rm -rf cmake-out
mkdir cmake-out

# Configure cmake
cmake \
    -DCMAKE_INSTALL_PREFIX=cmake-out \
    -DCMAKE_BUILD_TYPE=Release \
    -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
    -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
    -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
    -DEXECUTORCH_ENABLE_LOGGING=ON \
    -DPYTHON_EXECUTABLE=python \
    -Bcmake-out .

cmake --build cmake-out -j9 --target install --config Release

Integrating ExecuTorch to NNStreamer:

1. Create executorch.pc in /usr/lib/pkgconfig

includedir=/home/leeeryboy
libdir=${includedir}/executorch/cmake-out/lib

Name: executorch
Description: Executorch
Version: 0.2.0
Libs: -L${libdir} -lexecutorch_no_prim_ops -lexecutorch -lextension_data_loader -lextension_module -lextension_module_static -lextension_runner_util -lportable_kernels -lportable_ops_lib
Cflags: -I${includedir}

2. Manually reslove naming conflicts between two files below.
   (NEED FUTURE IMPROVEMENT: We could not find out a nice way to register kernels when using executorch runtime in nnstreamer)

#include <executorch/cmake-out/kernels/portable/portable_ops_lib/RegisterCodegenUnboxedKernelsEverything.cpp>
#include <executorch/kernels/prim_ops/register_prim_ops.cpp>

3. Build and install nnstreamer
   ref. https://github.com/nnstreamer/nnstreamer/blob/main/Documentation/getting-started-meson-build.md

Self evaluation:

1. Build test: [*]Passed [ ]Failed [ ]Skipped
2. Run test: [ ]Passed [ ]Failed [*]Skipped

Example Image classfication test script:

gst-launch-1.0 \
    textoverlay name=overlay font-desc=Sans,24 ! videoconvert ! ximagesink name=img_test \
    v4l2src name=cam_src ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=RGB ! tee name=t_raw \
    t_raw. ! queue ! overlay.video_sink \
    t_raw. ! queue leaky=2 max-size-buffers=2 ! videoconvert ! videoscale ! video/x-raw,width=224,height=224,format=RGB ! tensor_converter ! \
    tensor_transform mode=arithmetic option=typecast:float32,per-channel:true@2,div:255.0,add:-0.485@0,add:-0.456@1,add:-0.406@2,div:0.229@0,div:0.224@1,div:0.225@2 ! \
    tensor_transform mode=dimchg option=0:2 ! \
    tensor_filter framework=executorch model=executorch_model/mobilenetv3_large.pte input=224:224:3 inputtype=float32 output=1000:1 outputtype=float32 ! \
    tensor_decoder mode=image_labeling option1=executorch_model/imagenet_classes.txt ! \
    overlay.text_sink

[anyj0527]

@leeeryboy This looks as a good start of supporting ExecuTorch, great Thanks!
Could you check your commits to be signed off? git rebase --signoff HEAD~X will do it.

[anyj0527]

LGTM👍
I tested/verified this locally. Great thanks!