TensorFlow C++ API is significantly slower than Python API in inference

[RothLuo]

System information
***Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
No
***OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux Kernel 4.4.103, LUbuntu 16.04
***TensorFlow installed from (source or binary):
Python API is the official release; C++ API was complied via Makefile with all available optimization flags (linked as a static library)
***TensorFlow version (use command below):
1.10.1
***Python version:
2.7
***CUDA/cuDNN version:
Only CPU no GPU
***Bazel version:
did not use Bazel to build
***GPU model and memory:
did not use GPU for inference
***Mobile device:
platform with a RK3399 ARM processor

Steps to reproduce:

1. Trained a MobileNetSSDv1 model via Google Detection API (Python);
2. Froze the graph via the official tool provided by Detection API with batch_size = 1, and the input tensor size was [1, 150, 150, 3];
3. Used the frozen graph (.pb file) along with TensorFlow Python and C++ API for inference, respectively. All C++ optimization flags on;
4. The inference time with Python API was ~200 ms while C++ API takes ~550 ms.

Any thoughts would be appreciated!

[dhingratul]

Is this after ignoring the first few inferences? They are usually very time-consuming.

[RothLuo]

Yes, performance was measured after warming up.

[dhingratul]

Could be model/hparam specific, for my model I see equivalent inference times with C++ API and Python

[RothLuo]

Does your model run on an ARM processor? As I understand, the Python API also calls the same machine code with the C++ API. So it makes sense that the inference time is similar.

[dhingratul]

Ah! Missed that part, my model runs on GPU.

[reedwm]

If you are referring to the TensorFlow Object Detection API, can you file an issue in that repo instead?

[Harshini-Gadige]

If you are referring to the TensorFlow Object Detection API, can you file an issue in that repo instead?

@RothLuo Can you pls answer ?

[tensorflowbutler]

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

[ymodak]

Closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!