Android: GPU delegate fails with a YOLOv4 model

[arturdryomov]

System Information

• Custom code: none, using the upstream benchmark
• OS: Android 12
• Device: Google Pixel 4a
• TensorFlow version: nightly release benchmark build (definite version unspecified in the URL)

Steps to Reproduce

1. Enable developer options and USB debugging.
2. Execute the script below — it will download the TF benchmark and a model.

Please note that we have an internal YOLOv4 model we cannot share. I’ve found an existing one. However, the result is more or less the same so I’m gonna guess something is up with the model architecture. Also — the model executes fine on CPU / NPU.

Results:

• Expected: no error messages.
• Actual: a lot of error messages — seem to be an error for each model node. See the output below.

#!/bin/bash
set -eou pipefail


MODEL_FILE_NAME="model.tflite"

BENCHMARK_PATH="$(mktemp -d)"
BENCHMARK_FILE_NAME="tensorflow-benchmark"

DEVICE_PATH="/data/local/tmp"


echo ":: Fetch benchmark..."
curl \
  --location "https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model" \
  --output "${BENCHMARK_PATH}/${BENCHMARK_FILE_NAME}"

echo ":: Fetch model..."
curl \
  --location "https://github.com/theAIGuysCode/tensorflow-yolov4-tflite/raw/master/android/app/src/main/assets/yolov4-416-fp32.tflite" \
  --output "${MODEL_FILE_NAME}"


echo ":: Move benchmark to the device..."
adb push "${BENCHMARK_PATH}/${BENCHMARK_FILE_NAME}" "${DEVICE_PATH}"
adb shell chmod +x "${DEVICE_PATH}/${BENCHMARK_FILE_NAME}"

echo ":: Move model to the device..."
adb push "${MODEL_FILE_NAME}" "${DEVICE_PATH}"

echo ":: Run benchmark..."
adb shell taskset f0 "${DEVICE_PATH}/${BENCHMARK_FILE_NAME}" \
  --graph="${DEVICE_PATH}/${MODEL_FILE_NAME}" \
  --use_gpu=true

echo ":: Remove benchmark..."
adb shell rm "${DEVICE_PATH}/${BENCHMARK_FILE_NAME}"
rm -rf "${BENCHMARK_PATH}"

echo ":: Remove model..."
adb shell rm "${DEVICE_PATH}/${MODEL_FILE_NAME}"
rm -rf "${MODEL_FILE_NAME}"

:: Fetch benchmark...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6029k  100 6029k    0     0  11.8M      0 --:--:-- --:--:-- --:--:-- 11.8M
:: Fetch model...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   196  100   196    0     0    336      0 --:--:-- --:--:-- --:--:--   335
100 23.1M  100 23.1M    0     0  8255k      0  0:00:02  0:00:02 --:--:-- 23.7M
:: Move benchmark to the device...
/var/folders/d8/zmkczjms4jxbtbw24wt7qzbw0000gp/T/tmp.Yg5JHGh1/tens...ark: 1 file pushed, 0 skipped. 99.8 MB/s (6174376 bytes in 0.059s)
:: Move model to the device...
model.tflite: 1 file pushed, 0 skipped. 36.6 MB/s (24279948 bytes in 0.632s)
:: Run benchmark...
STARTING!
Log parameter values verbosely: [0]
Graph: [/data/local/tmp/model.tflite]
Use gpu: [1]
Loaded model /data/local/tmp/model.tflite
INFO: Initialized TensorFlow Lite runtime.
GPU delegate created.
INFO: Created TensorFlow Lite delegate for GPU.
INFO: Replacing 144 node(s) with delegate (TfLiteGpuDelegateV2) node, yielding 1 partitions.
INFO: Initialized OpenCL-based API.
INFO: Created 1 GPU delegate kernels.
Explicitly applied GPU delegate, and the model graph will be completely executed by the delegate.
The input model file size (MB): 24.2799
Initialized session in 2394.17ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
ERROR: TfLiteGpuDelegate Invoke: Given object is not valid
ERROR: Node number 144 (TfLiteGpuDelegateV2) failed to invoke.
ERROR: TfLiteGpuDelegate Invoke: Given object is not valid
ERROR: Node number 144 (TfLiteGpuDelegateV2) failed to invoke.
ERROR: TfLiteGpuDelegate Invoke: Given object is not valid
ERROR: Node number 144 (TfLiteGpuDelegateV2) failed to invoke.
ERROR: TfLiteGpuDelegate Invoke: Given object is not valid
ERROR: Node number 144 (TfLiteGpuDelegateV2) failed to invoke.
ERROR: TfLiteGpuDelegate Invoke: Given object is not valid
ERROR: Node number 144 (TfLiteGpuDelegateV2) failed to invoke.
ERROR: TfLiteGpuDelegate Invoke: Given object is not valid
>>> THIS CONTINUES FOR A WHILE <<<
count=852 first=237 curr=175 min=17 max=381 avg=176.995 std=47

Benchmarking failed.

[sushreebarsa]

@arturdryomov
In order to expedite the trouble-shooting process here,Could you please fill the issue template,
Thanks!

[arturdryomov]

@sushreebarsa, added a bit more relevant information.

[sushreebarsa]

@arturdryomov
In order to reproduce the issue reported here, could you please provide the complete code and the dataset , tensorflow version you are using. Thanks!

[arturdryomov]

@sushreebarsa, PTAL at the original issue. It contains the script downloading the official TensorFlow benchmark and the model. There is no custom code / project needed to reproduce the issue.

[ingura]

I have the same issue on tfLite 2.8.0, target sdk 32. So far the only workaround I have is to downgrade tensorflow to 2.5.0 if I would like to keep GPU support for yolo v4 , otherwise I can go with tf 2.8.0.

It would be great if someone would develop a more future ready solution