[Question]TFLite gpu acceleration.

[Nanamare]

Current, my model's inference speed (mate 10) 300ms
We have applied all the structural transformations that can be done in the model, algorithm
So, I am looking for a way to improve the performance of the Android platform.
Example of Render Script, OpenCL...
But, don't know if these ways really improve inference performance dramatically.
I wonder if these methods are applied to Tensorflow Mobile(including TFLite) at present.
I also wonder which method is the most appropriate.

Have a nice day :)

[VincentLin78]

There's standard channel for hw acceleration called NNAPI in Android 8.1.
https://developer.android.com/ndk/guides/neuralnetworks/index.html

TFL is gonna built on top of it and dispatch model to it.
Depends on how SOC vendor implemented NN HAL, the dispatched model may or may not have performance gain.

Before doing that, make sure if your model can be converted to tflite format by TOCO.
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/toco
If yes, then you can run your converted tflite model with TFL.

[Nanamare]

@VincentLin78
I try to convert pb to tflite, but encountered unsupported operation error (here is my posting). I converted 5 pb to tflite, but there was no dramatic effect. (mobilenet, inception, ssd mobilenet etc)
also
NNAPI document
I'm wondering if you need to write code that references the model(load graph, open session), or additionally needs to add other parts(model operation add, mul, activation function etc).

[aselle]

@Nanamare, we are working on various acceleration strategies (including GPU), and we will update this post when we can.

Thanks!