Build with XNNPACK

[asus4]

https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/xnnpack

XNNPACK is a highly optimized library of floating-point neural network inference operators for ARM, x86, and WebAssembly architectures in Android, iOS, Windows, Linux, macOS, and Emscripten environments. This document describes how to use the XNNPACK library as an inference engine for TensorFlow Lite.

[TheBricktop]

Could this boost the performance?
I've tried mediapipe hands on gpu on android (s8) and they get about 20-25 fps but current hands implementation are very slow on pc. (i7 16 gb ram gti 960 ti)

[asus4]

Hi @TheBricktop, This is because the GPU delegate (a GPU acceleration feature of the TensorFlow Lite) is not supported on Window at the moment. I would like to support it eventually.

[TheBricktop]

What ones should do to implement it? Recompile tf lite library?

[asus4]

It looks like to be not officially supported.
tensorflow/tensorflow#40325

[TheBricktop]

So on android the performance would be higher?

[asus4]

Yes, if you are interested in using MediaPipe without the GPU delegate, please refer XNNPACK(this issue) or the integer quantized model

[TheBricktop]

Would exporting tflite models to onnx and running in barracuda improve the performance?

[asus4]

Yes, if all ops are supported in Barracuda, could improve it. Please refer to the supported ops in Barracuda

[asus4]

Now, XNNPACK options are enabled in v2.4 libraries

[tonysung]

@asus4 I suspect XNNPACK is not correctly enabled based on the following observations:

1. Currently the CPU mode performance of tf-lite-unity-sample is closer to what I get from the official benchmark tool (https://www.tensorflow.org/lite/performance/measurement) with use_xnnpack=false.
2. From Tensorflow source code, I think TfLiteXNNPackDelegateCreate must be called in order for XNNPACK to be actually enabled. I don't see it being called anywhere in tf-lite-unity-sample.
3. When running the official benchmark tool with use_xnnpack=true, there will be a line in log that says Created TensorFlow Lite XNNPACK delegate for CPU. It's missing when tf-lite-unity-sample is run.

Would you like to take a look?

[asus4]

Thanks, @tonysung

I mistakenly thought it would be automatically enabled on the CPU mode. I will add XNNPack delegate.

https://github.com/tensorflow/tensorflow/blob/3f878cff5b698b82eea85db2b60d65a2e320850e/tensorflow/lite/delegates/xnnpack/xnnpack_delegate.h#L48