TensorflowLite Android OpenCL delegate may produce invalid Conv2D result

[dev0x13]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Android
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Huawei P30 Lite
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): 2.4.0
• Python version: -
• Bazel version (if compiling from source): 3.1.0
• GCC/Compiler version (if compiling from source): Android NDK 21.3.6528147
• CUDA/cuDNN version: -
• GPU model and memory: Mali-G51 MP4 as per gsmarena on an Android smartphone

I have been trying to run inference of some CNN model using TFLite 2.4.0 with OpenCL GPU delegate enabled and found that Conv2D operator may produce NaNs, Infs and other invalid values when running on the Mali-G51 MP4 GPU if precision loss is allowed (I assume that getting NaNs is not considered as a reasonable precision loss) and Cond2D padding is set to same. For valid padding model produces valid results.
I've created a simple Conv2D-only (simple_conv.zip, shown on the illustration below) model to test via inference_diff util:

Here are some sample outputs of the inference_diff/run_eval util obtained using the described model on the Huawei P30 Lite (Mali-G51 MP4 GPU) smartphone:

$ adb shell /data/local/tmp/run_eval --model_file=/data/local/tmp/simple_conv.tflite --num_runs=1 --delegate=gpu
INFO: Initialized TensorFlow Lite runtime.
INFO: Created TensorFlow Lite delegate for GPU.
GPU delegate is created.
INFO: Initialized OpenCL-based API.
INFO: Created 1 GPU delegate kernels.
native : inference_profiler_stage.cc:77 Test interpreter has been initialized.
native : tflite_inference_stage.cc:128 
native : inference_profiler_stage.cc:91 Reference interpreter (1 thread on CPU) has been initialized.
Num evaluation runs: 1
Reference run latency: avg=55112(us), std_dev=17548(us)
Test run latency: avg=11990(us), std_dev=1488(us)
OutputDiff[0]: avg_error=inf, std_dev=nan

After disabling precision loss manually, model seemed to be working fine, but much slower obviously:

$ adb shell /data/local/tmp/run_eval --model_file=/data/local/tmp/simple_conv.tflite --num_runs=1 --delegate=gpu --gpu_precision_loss_allowed=false
INFO: Initialized TensorFlow Lite runtime.
INFO: Created TensorFlow Lite delegate for GPU.
GPU delegate is created.
INFO: Initialized OpenCL-based API.
INFO: Created 1 GPU delegate kernels.
native : inference_profiler_stage.cc:77 Test interpreter has been initialized.
native : tflite_inference_stage.cc:128 
native : inference_profiler_stage.cc:91 Reference interpreter (1 thread on CPU) has been initialized.
Num evaluation runs: 1
Reference run latency: avg=121364(us), std_dev=20829(us)
Test run latency: avg=28716(us), std_dev=1158(us)
OutputDiff[0]: avg_error=0.000120298, std_dev=0

After further investigation I found out that this kind of behavior may be fixed by manually commenting the piece of code responsible for selecting Winograd algorithm kernel as a Conv2D node implementation (i.e. SelectConvolution branch is always used). After this fix, model seemed to be working fine:

$ adb shell /data/local/tmp/run_eval --model_file=/data/local/tmp/simple_conv.tflite --num_runs=1 --delegate=gpu
GPU delegate is created.
INFO: Initialized TensorFlow Lite runtime.
INFO: Created TensorFlow Lite delegate for GPU.
INFO: Initialized OpenCL-based API.
INFO: Created 1 GPU delegate kernels.
native : inference_profiler_stage.cc:77 Test interpreter has been initialized.
native : tflite_inference_stage.cc:128 
native : inference_profiler_stage.cc:91 Reference interpreter (1 thread on CPU) has been initialized.
Num evaluation runs: 1
Reference run latency: avg=113876(us), std_dev=17465(us)
Test run latency: avg=30590(us), std_dev=3084(us)
OutputDiff[0]: avg_error=0.304206, std_dev=0

Thus I assume that Winograd algorithm implementation for OpenCL delegate is the root cause of the issue. To sum up, here is the list of conditions to reproduce the bug, at least on the Mali-G51 MP4 GPU:

1. Create Conv2D node which is suitable for using Winograd algorithm as per check.
2. Use same padding in Conv2D node.
3. Use OpenCL backend.
4. Allow precision loss.

The same behavior was also observed when running on Samsung Galaxy M31 (Mali-G72 MP3 GPU) and Huawei P20 (Mali-G72 MP12 GPU). However, running default build (i.e. without disabling Winograd manually) on Samsung Galaxy S20+ (Mali-G77 MP11 GPU) was successful.

Please let me know, if you need more details/logs/code etc.

[srjoglekar246]

@impjdi , could you take a look at this? If its a bug I can help file an internal tracker.

[impjdi]

@dev0x13

Forwarding the message from our OpenCL specialist. Due to COVID-19 and restricted office visits, he could only test on a device that he had, Mali T880MP4, but it seems to work correctly and he couldn't reproduce the issue.

Can you try to run tensorflow/tensorflow/lite/delegates/gpu/cl/kernels/winograd_test.cc ?

[dev0x13]

@impjdi

This test is passing, here is the output just in case:

$ adb shell /data/local/tmp/winograd_test
Running main() from gmock_main.cc
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from OpenCLOperationTest
[ RUN      ] OpenCLOperationTest.Winograd4x4To36
[       OK ] OpenCLOperationTest.Winograd4x4To36 (2942 ms)
[ RUN      ] OpenCLOperationTest.Winograd36To4x4
[       OK ] OpenCLOperationTest.Winograd36To4x4 (1472 ms)
[----------] 2 tests from OpenCLOperationTest (4415 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (4415 ms total)
[  PASSED  ] 2 tests.

Thank you and OpenCL specialist for checking this issue. Too bad that you're not able to reproduce it with available hardware.
I'll try to debug this kernel myself using my hardware and will share results here if there will be any.

[impjdi]

Thank you!

[dev0x13]

Well, I managed to find the root cause of this behavior, and it seems to be a serious system problem from my perspective. Description is below.

If Conv2D operator is suitable for applying Winograd it's been decomposed to three operators: Winograd4x4To36 -> ConvBuffer1x1 (at least at Mali GPU) -> Winograd36To4x4 in operation selection while applying OpenCL delegate.

After brief inspection of OpenCL code produced by ConvBuffer1x1 I have noticed that the computation core consists of accumulating multiplication results. However, accumulator type for half-precision computation mode (which is default for OpenCL delegate) is also float16, so it may represent at most 65504 and therefore is very vulnerable for overflow. This may be easily proven by changing bias value in tensorflow/tensorflow/lite/delegates/gpu/cl/kernels/conv_buffer_1x1.cc from zero to 65503 (totally valid value even for a half-precision float). Apparently the test will fail, but one may notice that in this case ConvBuffer1x1 will produce inf values for CalculationsPrecision::F16 case, which are eventually becoming NaNs if used for arithmetic operations in the next operator (e.g. Winograd). If we will increase bias to, say, 10^5 (still valid float value), then we will obtain Infs even for CalculationsPrecision::F32_F16.

I believe that the same behavior can be reproduced using this recipe for Winograd ops as well. I also believe that this is observable only at Mali G51 and a couple other GPUs because of the implementation details, since Inf on float overflow is not guaranteed by the specification.

I understand that the given example of incorrect behavior is kind of unrealistic, but apparently this may happen in some other cases, e.g. in the one above, which is actually delivered from the real-world production model.

To sum up:

• In my opinion the root cause of this issue is half-precision floating point values overflow which happens in kernels code.
• If so, then this may also happen in other kernels where accumulation is used as well. This is very tricky since there are no overflow checks in the TF Lite GPU delegate code for half-precision computation mode, however for the limit this low (65504) this may happen easily.

I am not really into production OpenCL coding, so could you clarify whether this is expected or should be considered as a bug? Hope this helps and thank you in advance.

[impjdi]

Thank you very much for your time looking into this issue! The overflow issue makes sense; it is one of the most frequent reasons why one cannot use FP16. I'll forward your findings to the person who implemented the Winograd conv2d, maybe we can do something about it. It may take a bit to hear back as he's on vacation right now.

[dev0x13]

Thank you! I will try to make a workaround for my model for now.