demogen: loading resnet models fails

[uricohen]

After working-around a problem in example.py as described in previous issue I could load NIN models but not resnet models. The error is below:

I0716 13:20:53.269759 140097876026944 saver.py:1280] Restoring parameters from data/demogen_models/RESNET_CIFAR10/resnet_wide_1.0x_batchnorm__decay_0.0_1/model.ckpt-150000
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "~/google-research/demogen/example.py", line 62, in <module>
    tf.app.run(main)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "~/.local/lib64/python2.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "~/.local/lib64/python2.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "~/google-research/demogen/example.py", line 57, in main
    load_and_run(model_config, root_dir)
  File "~/google-research/demogen/example.py", line 44, in load_and_run
    sess.run(logits)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnimplementedError: Generic conv implementation only supports NHWC tensor format for now.
         [[node resnet/conv2d/Conv2D (defined at /tmp/tmpdCZJAJ.py:12) ]]

Errors may have originated from an input operation.
Input Source operations connected to node resnet/conv2d/Conv2D:
 transpose (defined at demogen/data_util.py:79)
 resnet/conv2d/kernel/read (defined at demogen/models/resnet.py:136)

Original stack trace for u'resnet/conv2d/Conv2D':
  File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "~/google-research/demogen/example.py", line 62, in <module>
    tf.app.run(main)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "~/.local/lib64/python2.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "~/.local/lib64/python2.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "~/google-research/demogen/example.py", line 57, in main
    load_and_run(model_config, root_dir)
  File "~/google-research/demogen/example.py", line 41, in load_and_run
    logits = model_fn(image, is_training=False)
  File "demogen/models/resnet.py", line 391, in __call__
    strides=self.conv_stride, data_format=self.data_format)
  File "demogen/models/resnet.py", line 136, in conv2d_fixed_padding
    data_format=data_format)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/layers/convolutional.py", line 424, in conv2d
    return layer.apply(inputs)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1479, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/layers/base.py", line 537, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 634, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/autograph/impl/api.py", line 146, in wrapper
    ), args, kwargs)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/autograph/impl/api.py", line 450, in converted_call
    result = converted_f(*effective_args, **kwargs)
  File "/tmp/tmpdCZJAJ.py", line 12, in tf__call
    outputs = ag__.converted_call('_convolution_op', self, ag__.ConversionOptions(recursive=True, force_conversion=False, optional_features=(), internal_convert_user_code=True), (inputs, self.kernel), None)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/autograph/impl/api.py", line 356, in converted_call
    return _call_unconverted(f, args, kwargs)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/autograph/impl/api.py", line 255, in _call_unconverted
    return f(*args)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1079, in __call__
    return self.conv_op(inp, filter)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 635, in __call__
    return self.call(inp, filter)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 234, in __call__
    name=self.name)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1953, in conv2d
    name=name)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1071, in conv2d
    data_format=data_format, dilations=dilations, name=name)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "~/.local/lib64/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

[yidingjiang]

Hi @uricohen, thank you for using the code base!
By any chance you are running this on CPU? The ResNet I use is optimized for performance with channel first format but I think the NCHW format is only supported on GPU.

[uricohen]

OK, solved after pip install tensorflow-gpu and properly installing the needed versions of CUDA, CuDNN.

[yidingjiang]

Great! Can you close the issue if it has been properly addressed?

[SUmrao]

How can we run in CPU? I am using macbook and does not have GPU@yidingjiang