Resnet: Default AvgPoolingOp only supports NHWC. #263

andreas128 · 2017-05-09T16:35:40Z

When running python examples/ResNet/cifar10-resnet.py I get following error message InvalidArgumentError (see above for traceback): Default AvgPoolingOp only supports NHWC.

I tried it using tensorflow 1.0 and 1.1. Also with tag 0.1.8 and 0.1.9. The full error message is as following:

  0%|                                                                                                           |0/390[00:00<?,?it/s]
E tensorflow/core/common_runtime/executor.cc:594] Executor failed to create kernel. Invalid argument: Default AvgPoolingOp only suppo
rts NHWC.
         [[Node: tower0/res2.0/pool/output = AvgPool[T=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="VALID", strides=[1,
 1, 2, 2], _device="/job:localhost/replica:0/task:0/cpu:0"](tower0/res1.17/add)]]

Traceback (most recent call last):
[0509 17:26:58 @input_data.py:125] EnqueueThread Exited.
  File "examples/ResNet/cifar10-resnet.py", line 179, in <module>
    SyncMultiGPUTrainer(config).train()
  File "/media/Data/face3d/tensorpack/tensorpack/train/base.py", line 94, in train
    self.main_loop()
  File "/media/Data/face3d/tensorpack/tensorpack/train/base.py", line 164, in main_loop
    self.run_step()  # implemented by subclass
  File "/media/Data/face3d/tensorpack/tensorpack/train/feedfree.py", line 43, in run_step
    self.hooked_sess.run(self.train_op)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 891, in run
    run_metadata=run_metadata)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 744, in run
    return self._sess.run(*args, **kwargs)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
    run_metadata_ptr)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
    feed_dict_string, options, run_metadata)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
    target_list, options, run_metadata)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Default AvgPoolingOp only supports NHWC.
         [[Node: tower0/res2.0/pool/output = AvgPool[T=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="VALID", strides=[1,
 1, 2, 2], _device="/job:localhost/replica:0/task:0/cpu:0"](tower0/res1.17/add)]]

Caused by op u'tower0/res2.0/pool/output', defined at:
  File "examples/ResNet/cifar10-resnet.py", line 179, in <module>
    SyncMultiGPUTrainer(config).train()
  File "/media/Data/face3d/tensorpack/tensorpack/train/base.py", line 93, in train
    self.setup()
  File "/media/Data/face3d/tensorpack/tensorpack/train/base.py", line 108, in setup
    self._setup()   # subclass will setup the graph
  File "/media/Data/face3d/tensorpack/tensorpack/train/multigpu.py", line 132, in _setup
    self.config.tower, lambda: self._get_cost_and_grad()[1])
  File "/media/Data/face3d/tensorpack/tensorpack/train/multigpu.py", line 41, in _multi_tower_grads
    grad_list.append(get_tower_grad_func())
  File "/media/Data/face3d/tensorpack/tensorpack/train/multigpu.py", line 132, in <lambda>
    self.config.tower, lambda: self._get_cost_and_grad()[1])
  File "/media/Data/face3d/tensorpack/tensorpack/train/feedfree.py", line 66, in _get_cost_and_grad
    self.build_train_tower()
  File "/media/Data/face3d/tensorpack/tensorpack/train/feedfree.py", line 35, in build_train_tower
    f()
  File "/media/Data/face3d/tensorpack/tensorpack/train/feedfree.py", line 28, in f
    self.model.build_graph(inputs)
  File "/media/Data/face3d/tensorpack/tensorpack/models/model_desc.py", line 116, in build_graph
    self._build_graph(model_inputs)
  File "examples/ResNet/cifar10-resnet.py", line 83, in _build_graph
    l = residual('res2.0', l, increase_dim=True)
  File "examples/ResNet/cifar10-resnet.py", line 68, in residual
    l = AvgPooling('pool', l, 2)
  File "/media/Data/face3d/tensorpack/tensorpack/models/common.py", line 97, in wrapped_func
    outputs = func(*args, **actual_args)
  File "/media/Data/face3d/tensorpack/tensorpack/models/pool.py", line 64, in AvgPooling
    data_format=data_format)
  File "/media/Data/face3d/tensorpack/tensorpack/models/pool.py", line 28, in _Pooling
    name='output')
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1765, in avg_pool
    name=name)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 50, in _avg_pool
    data_format=data_format, name=name)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
    op_def=op_def)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/andreas/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Default AvgPoolingOp only supports NHWC.
         [[Node: tower0/res2.0/pool/output = AvgPool[T=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="VALID", strides=[1,
 1, 2, 2], _device="/job:localhost/replica:0/task:0/cpu:0"](tower0/res1.17/add)]]

The text was updated successfully, but these errors were encountered:

ppwwyyxx · 2017-05-09T16:45:24Z

TensorFlow only supports NHWC on cpu.
If you really want to use CPU, you can change data_format to 'NHWC' at this line, and remove the layout transform at this line.

andreas128 · 2017-05-09T17:04:11Z

Thank you very much, that solved the Issue.

abhijitnathwani · 2017-11-29T11:08:27Z

Hello,
Even I faced the same error, and on solving with the method as suggested by @ppwwyyxx I solved that error. However, i guess changing the data_format also changed some shapes.

Now I'm getting error:

ValueError: Dimensions must be equal, but are 3 and 16 for 'tower0/res1.0/add' (op: 'Add') with input shapes: [?,3,32,3], [?,3,32,16].

Please help me for the same. I need to train cifar10-resnet.py on CPU. I don't have GPU with me.

ppwwyyxx · 2017-11-29T11:09:50Z

cifar10-resnet.py is written for GPU.
You need to change the relevant symbolic code after changing data_format.

abhijitnathwani · 2017-11-29T11:13:10Z

Can you please help me with changes @ppwwyyxx ? I'm stuck on this for long now. Don't know where to make the changes for this to run on CPU. All the help is appreciated!

ppwwyyxx · 2017-11-29T11:14:58Z

If you can understand tensorflow symbolic code, the changes are easy to identify in build_graph. If not I suggest you learn basic tensorflow symbolic programming before using tensorpack.

abhijitnathwani · 2017-11-29T12:30:51Z

@ppwwyyxx I already made the necessary changes in build_graph to bypass GPU checks. I still cannot get why the

ValueError: Dimensions must be equal, but are 3 and 16 for 'tower0/res1.0/add' (op: 'Add') with input shapes: [?,3,32,3], [?,3,32,16].

error occurs.

Shapes should have nothing to do with GPU/CPU I guess.

ppwwyyxx · 2017-11-29T12:32:13Z

Shapes should have something to do with data_format, for sure.

ppwwyyxx · 2017-11-29T12:38:45Z

diff --git i/examples/ResNet/cifar10-resnet.py w/examples/ResNet/cifar10-resnet.py
index d2cbc6c..764ca75 100755
--- i/examples/ResNet/cifar10-resnet.py
+++ w/examples/ResNet/cifar10-resnet.py
@@ -48,12 +48,10 @@ class Model(ModelDesc):
     def _build_graph(self, inputs):
         image, label = inputs
         image = image / 128.0
-        assert tf.test.is_gpu_available()
-        image = tf.transpose(image, [0, 3, 1, 2])
 
         def residual(name, l, increase_dim=False, first=False):
             shape = l.get_shape().as_list()
-            in_channel = shape[1]
+            in_channel = shape[3]
 
             if increase_dim:
                 out_channel = in_channel * 2
@@ -68,12 +66,12 @@ class Model(ModelDesc):
                 c2 = Conv2D('conv2', c1, out_channel)
                 if increase_dim:
                     l = AvgPooling('pool', l, 2)
-                    l = tf.pad(l, [[0, 0], [in_channel // 2, in_channel // 2], [0, 0], [0, 0]])
+                    l = tf.pad(l, [[0, 0], [0, 0], [0, 0], [in_channel // 2, in_channel // 2]])
 
                 l = c2 + l
                 return l
 
-        with argscope([Conv2D, AvgPooling, BatchNorm, GlobalAvgPooling], data_format='NCHW'), \
+        with argscope([Conv2D, AvgPooling, BatchNorm, GlobalAvgPooling], data_format='NHWC'), \
                 argscope(Conv2D, nl=tf.identity, use_bias=False, kernel_shape=3,
                          W_init=variance_scaling_initializer(mode='FAN_OUT')):
             l = Conv2D('conv0', image, 16, nl=BNReLU)

abhijitnathwani · 2017-11-29T13:08:21Z

@ppwwyyxx Thanks a ton! You're a savior. Works well now.

Superlee506 · 2018-05-08T21:29:38Z

@ppwwyyxx
When I tested the "roi_align" function, I also meet this problem and I used GPU.
2018-05-08 17:24:44.425156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7) 2018-05-08 17:24:44.624317: E tensorflow/core/common_runtime/executor.cc:643] Executor failed to create kernel. Invalid argument: Default AvgPoolingOp only supports NHWC. [[Node: roi_align/AvgPool = AvgPool[T=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="SAME", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:GPU:0"](roi_align/crop_and_resize/transpose_1)]]

ppwwyyxx · 2018-05-08T22:20:55Z

@Superlee506 When reporting problems please following the issue template. Thanks!

Superlee506 · 2018-05-08T22:43:30Z

@ppwwyyxx Ok

andreas128 closed this as completed May 9, 2017

ppwwyyxx added the examples label Aug 25, 2017

msbauer mentioned this issue Sep 26, 2017

NCHW on CPU #429

Closed

ppwwyyxx added the unrelated unrelated to tensorpack / invalid questions / questions that are not helpful to others label May 8, 2018

Superlee506 mentioned this issue May 8, 2018

roi_align: Default AvgPoolingOp only supports NHWC #753

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resnet: Default AvgPoolingOp only supports NHWC. #263

Resnet: Default AvgPoolingOp only supports NHWC. #263

andreas128 commented May 9, 2017

ppwwyyxx commented May 9, 2017

andreas128 commented May 9, 2017

abhijitnathwani commented Nov 29, 2017 •

edited

ppwwyyxx commented Nov 29, 2017

abhijitnathwani commented Nov 29, 2017

ppwwyyxx commented Nov 29, 2017

abhijitnathwani commented Nov 29, 2017

ppwwyyxx commented Nov 29, 2017

ppwwyyxx commented Nov 29, 2017

abhijitnathwani commented Nov 29, 2017 •

edited

Superlee506 commented May 8, 2018

ppwwyyxx commented May 8, 2018

Superlee506 commented May 8, 2018

Resnet: Default AvgPoolingOp only supports NHWC. #263

Resnet: Default AvgPoolingOp only supports NHWC. #263

Comments

andreas128 commented May 9, 2017

ppwwyyxx commented May 9, 2017

andreas128 commented May 9, 2017

abhijitnathwani commented Nov 29, 2017 • edited

ppwwyyxx commented Nov 29, 2017

abhijitnathwani commented Nov 29, 2017

ppwwyyxx commented Nov 29, 2017

abhijitnathwani commented Nov 29, 2017

ppwwyyxx commented Nov 29, 2017

ppwwyyxx commented Nov 29, 2017

abhijitnathwani commented Nov 29, 2017 • edited

Superlee506 commented May 8, 2018

ppwwyyxx commented May 8, 2018

Superlee506 commented May 8, 2018

abhijitnathwani commented Nov 29, 2017 •

edited

abhijitnathwani commented Nov 29, 2017 •

edited