TF v1.3 slower than v1.2 when used with ResNets

[eldar]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  It's a custom code with fully convolutional ResNet using tf.slim implementation (with diluted kernels)
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Debian 8.9
• TensorFlow installed from (source or binary):
  Binary (pip)
• TensorFlow version (use command below):
  1.3
• Python version:
  Python 3.4
• Bazel version (if compiling from source):
  N/A
• CUDA/cuDNN version:
  CUDA 8.0/cuDNN 6.0
• GPU model and memory:
  NVIDIA K40m 12Gb

I run fully convolutional ResNet-101 on the images which vary in size. When moving from TF 1.2 to TF 1.3 inference became about 3x slower. With TF1.2 I use CUDA 8.0 and cudnn 5.1. To make sure variable sized images are processed fast I set env variable TF_CUDNN_USE_AUTOTUNE=0 to switch off auto-tuning of convolutions.

In case it is not to do with convolutions, but the data loading, here's how I feed the input data (numpy arrays) into the convnet:

outputs_np = sess.run(outputs, feed_dict={inputs: batch})

Could you suggest how I can troubleshoot that?

[ppwwyyxx]

The training performance has been stable in the official benchmarks (https://benchmarks-dot-tensorflow-testing.appspot.com/test/tf-cnn-benchmark-resnet50). But it uses fixed size image, with autotune enabled. Maybe you can try fixed-size autotune as well to narrow down the causes.

[poxvoculi]

What makes you think the slowdown is due to data loading, as opposed to convolutional computation? What kinds of measurements have you made? I notice that you're using a pip binary instead of building from source. Could the slowdown be due to lack of SIMD support for CPU operations?

[eldar]

@poxvoculi Hi, I may have not been precise. I only indicated the possibility it is not the computations that slow it down, but the data processing part. I am not sure how to benchmark it, because from the user point of view sess.run() does both compute and data feeding, so I cannot evaluate them individually. I use GPU version and not CPU, and in both cases (TF 1.2 and 1.3) I use pre-combiled builds, so it's comparable setting.

@ppwwyyxx I'll try to narrow down the the issue..