Interpreter API (Java) - GpuDelegateV2 support

[cfasana]

Hi,
I am trying to run a TFLite model on the GPU of an Android device.
According to this documentation, it is possible to use both the Interpreter API and the Native c++ API to achieve this.

At the moment, I am using the following dependencies:

implementation 'org.tensorflow:tensorflow-lite:2.15.0'
implementation 'org.tensorflow:tensorflow-lite-select-tf-ops:2.15.0'
implementation 'org.tensorflow:tensorflow-lite-support:0.4.4'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.15.0'
implementation 'org.tensorflow:tensorflow-lite-gpu-api:2.15.0'
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin:0.4.4'

I was able to successfully run my model using the GPUDelegate provided by the Java Interpreter API. However, this delegate does not allow to specify inference priority options (TFLITE_GPU_INFERENCE_PRIORITY_MIN_LATENCY, TFLITE_GPU_INFERENCE_PRIORITY_MIN_MEMORY_USAGE, TFLITE_GPU_INFERENCE_PRIORITY_MAX_PRECISION).

These options can be specified if the Native C++ API is used given the presence of GpuDelegateV2. However, at the moment I don't see this option in the Interpreter API since there is no class named GpuDelegateV2.

Is there a way to make use of this new delegate without the need of using the Native C++ API?

[LakshmiKalaKadali]

Hi @cfasana,

The Java Interpreter API currently doesn't have GpuDelegateV2 directly. However if you would like to achieve the faster inference speed you can use GpuDelegate class by setting the isPrecisionLossAllowed flag to true in the following way as a workaround. But for memory usage and max precision, feature requests will be raised. Thanks for letting us know.

GpuDelegateOptions options = new GpuDelegateOptions();
options.isPrecisionLossAllowed = true;  
GpuDelegate gpuDelegate = new GpuDelegate(options);
InterpreterOptions interpreterOptions = new InterpreterOptions();
interpreterOptions.addDelegate(gpuDelegate);
Interpreter interpreter = new Interpreter(modelBuffer, interpreterOptions);

or
Also try with tflite_flutter library which will provide access to GpuDelegateV2 through DART API.

Hi @pkgoogle,
As @cfasana mentioned, GpuDelegateV2 need to be included in Java Interpreter API with the support of (TFLITE_GPU_INFERENCE_PRIORITY_MIN_LATENCY, TFLITE_GPU_INFERENCE_PRIORITY_MIN_MEMORY_USAGE, TFLITE_GPU_INFERENCE_PRIORITY_MAX_PRECISION). Raised a feature request.

Thank You

[cfasana]

Hi @LakshmiKalaKadali,
thanks for the feedback.
I will proceed as you suggested while awaiting the Java Interpreter API update.

[pkgoogle]

Hi @sirakiin, can you please take a look a this feature request? Thanks.