TFLite GPU crashes when not all ops are supported by delegate?

[bazinac]

System information

• I have custom code, however GPU Delegate part I have used as described in here provided sample
• Ubuntu 18.04
• Samsung Galaxy J5 2015 (crashes), Galaxy S7 and LG G6 (works fine),
• TF build from 'org.tensorflow:tensorflow-lite:0.0.0-gpu-experimental'

Describe the current behavior
Using TF Lite with GPU developer preview, when I run inference on my mobilenetV2 retrained model (with two outputs) on most devices it works nice. However on Galaxy J5 (odler device), it crashes with:

    E/AndroidRuntime: FATAL EXCEPTION: inference
Process: bazinac.aplikacenahouby, PID: 12367
java.lang.IllegalArgumentException: Internal error: Failed to apply delegate: WARNING: op code #40 cannot be handled by this delegate.  Only the first 67 ops will run on the GPU, and the remaining 10 on the CPU.GpuDelegate Prepare: No EGL error, but eglCreatePbufferSurface failedNode number 77 (GpuDelegate) failed to prepare.

    at org.tensorflow.lite.NativeInterpreterWrapper.applyDelegate(Native Method)_

Describe the expected behavior
I would expect that when isGpuDelegateAvailable() method provides only such a delegate, that is capable preparing and running all the ops. It should return false otherwise, I guess. Is there some way how to tell this in advance, or should I manually check for these kind of internal errors and try to fall back by recreating Interpret without GPU support?

Code to reproduce the issue
` protected void runInference() {

    Log.d(TAG, "Inference starts ");
    Object[] inputArray = {imgData};

    Map<Integer, Object> outputMap = new HashMap<>();
    outputMap.put(0, outp1);
    outputMap.put(1, outp2);
    tflite.runForMultipleInputsOutputs(inputArray, outputMap);
    Log.d(TAG, "Inference ends ");
}`

Other info / logs
Thank you guys for your effort to make this available to us, you are doing very good job!

[impjdi]

@bazinac

isGpuDelegateAvailable essentially only checks whether AAR is available. It is not very safe to assume that function is foolproof. In fact, there can be various other reasons why GPU initialization might fail. We are trying hard to improve that method though.

re: if the whole graph cannot be run, return false, is against the current philosophy of delegates API. The idea is to provide a seamless experience to the user, so that, if GPU delegate cannot be applied, it will be covered by the CPU automatically. It has pros and cons. Pros is, if the last couple of ops are not covered by the GPU, it will automagically work. The cons is, that the users miss the WARNING message, and think the graph is executed on the GPU, but it was actually run on the CPU. In that case, you don't see any speedup whatsoever. We definitely need a better way of notifying the user, rather than just a WARNING message in the logs.

re: the PBbuffer surface thing. The OpenGL-based backend needs to create a GL context and keep it around. We try various things to create a GL context. First, we try to create a configless context. If that fails, we try to created a surfaceless context. If that fails, the final step is to create a PBuffer context. If even that failed, we cannot proceed. Unfortunately, your Galaxy J5 doesn't seem to pass this context creation phase. I'm not 100% certain, but according to:

https://compubench.com/device.jsp?benchmark=compu20&os=Android&api=rs&D=Samsung+Galaxy+J5+%28SM-J500x%29&testgroup=info

Galaxy J5 2015 only seems to have GL ES 3.0. The requirement for our GL-based backend is GL ES 3.1. Could you please check whether you have GL ES 3.1 available on your J5?

[bazinac]

Hi guys, thanks for providing extensive info on the topic.

You are right, this J5 has GL ES 3.0 only, which is why it does not run on it. Maybe it would be good idea to throw some warning about required version inavailability right at the TF init time (when GPU options are passed). Thanks for expanation!