Build iOS binaries with GPU delegate

Author: @applikationsprogramvara

Building TF Lite

Once again here is the start point.

The recommended option is ending up with issue https://github.com/tensorflow/tensorflow/issues/48464, so no outcome here: bazel build --config=ios_fat -c opt //tensorflow/lite/ios:TensorFlowLiteC_framework

To build version 2.4.1 and earlier I used (by adding experimental/): bazel build --config=ios_fat -c opt //tensorflow/lite/experimental/ios:TensorFlowLiteC_framework

To build the latest version for arm64 only (by replacing ios_fat via ios_arm64): bazel build --config=ios_arm64 -c opt //tensorflow/lite/ios:TensorFlowLiteC_framework

The second and the third options produced TensorFlowLiteC_framework.zip as described here, but running the app ended up with error regarding missing TFLGpuDelegateCreate symbol.

So I assumed, that GPU delegate metal function is not enabled in the framework. The idea was to include metal_delegate.h into the compiled framework.

Building GPU delegate metal

Unfortunately I didn't find a way to include the GPU delegate metal function to TensorFlowLiteC_framework. At the end I stumbled on this line in tensorflow, which gave an idea, that the function can be built separately.

So to build GPU delegate metal function I used: bazel build --config=ios_arm64 -c opt //tensorflow/lite/ios:TensorFlowLiteCMetal_framework And got a separate TensorFlowLiteCMetal_framework.zip.

Enabling GPU delegate metal function

On the next step I placed TensorFlowLiteCMetal.framework into plugin ios folder and added it into tflite_flutter.podspec:

s.ios.vendored_frameworks = 'TensorFlowLiteC.framework', 'TensorFlowLiteCMetal.framework' s.xcconfig = { 'OTHER_LDFLAGS' => '-framework TensorFlowLiteC -all_load -framework TensorFlowLiteCMetal -all_load' }

To apply the changes I cleaned and rebuilt the flutter project: flutter clean flutter build ios

After that the error (magically) disappeared. The app is working all right and more than twice faster, than without GPU acceleration.

Observations

The option suggested in Readme is working all right:

final gpuDelegate = GpuDelegate(
      options: GpuDelegateOptions(true, TFLGpuDelegateWaitType.active),
    );
var interpreterOptions = InterpreterOptions()..addDelegate(gpuDelegate);

The other option though mentioned here https://github.com/am15h/tflite_flutter_plugin/issues/60#issuecomment-752367647 and also present in the documentation is not working all right.