GPU delegate gives different result from CPU

[ShiyongL]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution:Linux Ubuntu 18.04
• Mobile device if the issue happens on mobile device: Pixel 3 XL
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): r2.2.0 and master
• Python version: 3.6
• Bazel version (if compiling from source): 3.0.0
• GCC/Compiler version (if compiling from source): default
• CUDA/cuDNN version: 10.0
• GPU model and memory:

Describe the current behavior
GPU delegate gives very different result from CPU.
I was able to hard-code the source (in tensorflow/lite/delegates/gpu/common/model_builder.cc) to allow some operations to be delegated to GPU. Out of 270+ operations:

• Delegating just one Conv_2d will produce very similar result as the one by CPU only.
• Delegating a few more operations seem to produce bigger difference.
• Delegating just the first MUL operation will produce very different result.

Describe the expected behavior
result should be close

Standalone code to reproduce the issue
I personally hijacked tflite tools/benchmark code and give an sample image as deterministic input, instead of random input.
I would love to provide my code change if it helps.

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

The tflite model was converted from InsightFace/ArcFace MXNet model
https://github.com/deepinsight/insightface/wiki/Model-Zoo (3.2 model)
link to download the tflite
https://drive.google.com/file/d/1pJX2I8btskVy-QHiF-mcUF7HF6ZFaQuf/view?usp=sharing

Also attached the graph of above model:
visualized_official_arcface_no_sub.zip

[srjoglekar246]

Can confirm that I see output diff errors for GPU vs CPU.

@ShiyongL , for future ease, feel free to check out our inference_diff tool to compare raw delegate outputs with CPU, and modify the delegate node partitioner to select particular nodes by their index in their execution plan (which can be visualized with this tool) :-).

Looking into this...

[srjoglekar246]

The culprit (atleast partially) seems to be the GPU delegate's low-precision mode, which is enabled by default in our benchmark tooling.

AFAIK, the delegate uses fp16 (instead of fp32) precision in that case, and that seems to be the cause of your output diffs. I could not pinpoint a single node/nodes that cause the output errors; they seem to be increasing with number of delegate nodes as you suggested.

See the outputs with inference_diff with the mode enabled/disabled:

$ adb shell /data/local/tmp/run_eval   --model_file=/data/local/tmp/gpu_debug_model.tflite --delegate=gpu --num_runs=5 --gpu_precision_loss_allowed=true
...
OutputDiff[0]: avg_error=nan, std_dev=nan
OutputDiff[1]: avg_error=0, std_dev=0

vs

$ adb shell /data/local/tmp/run_eval   --model_file=/data/local/tmp/gpu_debug_model.tflite --delegate=gpu --num_runs=5 --gpu_precision_loss_allowed=false
...
OutputDiff[0]: avg_error=0.0824622, std_dev=0.000453001
OutputDiff[1]: avg_error=0, std_dev=0

There are still some errors in the latter case, but they are lesser than the earlier one. (Note that the latency goes to ~80ms from ~50ms on a Pixel 3, when the mode is disabled).

In our Java API, the mode can be disabled by doing the following instead of the usual:

GpuDelegate delegate = new GpuDelegate(new GpuDelegate.Options().setPrecisionLossAllowed(false));

CC'ing Juhyun who might know if such output diffs are expected - I guess Convolutions & Fully-Connecteds can amplify previous error.

[impjdi]

Usually, it's either the model's fault of numerical instability (e.g. using squared diff on an error that is below a very small threshold such as 1e-6, often visible in instance norms of X - MEAN followed by SQUARE_DIFF, and then doing math based on those values) or a bug in our shaders' accumulators, i.e. some accumulators needs to be in FP32 regardless of the precision mode such as SOFTMAX. Unfortunately, I don't think I will have time to look into this, it would be nice if you could pinpoint what the issue is.