Conv2DCustomBackpropInputOp only supports NHWC.

[QiyaoWei]

For completeness, the entirety of the error message is as below--

2020-07-01 13:36:02.995480: E tensorflow/core/common_runtime/executor.cc:642] Executor failed to create kernel. Invalid argument: Conv2DCustomBackpropInputOp only supports NHWC.
[[{{node G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose}}]]
Traceback (most recent call last):
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.
[[{{node G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "generate_images.py", line 209, in
main()
File "generate_images.py", line 205, in main
fake = tflib.run(fake_images_out, {latents:lats,noise:noise_})
File "/home/region/Desktop/DisentangledFaceGAN/dnnlib/tflib/tfutil.py", line 26, in run
return tf.get_default_session().run(*args, **kwargs)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.
[[node G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose (defined at /home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose':
File "generate_images.py", line 209, in
main()
File "generate_images.py", line 186, in main
fake_images_out = truncate_generation(Gs,INPUTcoeff_w_noise,dlatent_average_id=average_w_id)
File "generate_images.py", line 89, in truncate_generation
fake_images_out = Gs.components.synthesis.get_output_for(dlatent_out_final, randomize_noise = False)
File "/home/region/Desktop/DisentangledFaceGAN/dnnlib/tflib/network.py", line 222, in get_output_for
out_expr = self._build_func(*final_inputs, **build_kwargs)
File "", line 570, in G_synthesis
File "", line 569, in grow
File "", line 563, in
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1224, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1061, in BuildCondBranch
original_result = fn()
File "", line 568, in
File "", line 569, in grow
File "", line 563, in
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1224, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1061, in BuildCondBranch
original_result = fn()
File "", line 568, in
File "", line 569, in grow
File "", line 563, in
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1224, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1061, in BuildCondBranch
original_result = fn()
File "", line 568, in
File "", line 569, in grow
File "", line 563, in
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1224, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1061, in BuildCondBranch
original_result = fn()
File "", line 568, in
File "", line 565, in grow
File "", line 534, in block
File "", line 191, in upscale2d_conv2d
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/nn_ops.py", line 2204, in conv2d_transpose
name=name)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/nn_ops.py", line 2275, in conv2d_transpose_v2
name=name)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_nn_ops.py", line 1407, in conv2d_backprop_input
name=name)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/region/anaconda3/envs/tod1/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

I searched online, and many seem to suggest that this is a CPU/GPU problem. However, I encountered the same issue on CPU and on GeForce RTX 2080 Ti. Any help would be appreciated. Thanks!

[YuDeng]

Hi, I suggest you run the script with python 3.6 and tensorflow 1.12 to see if the error still occurs.

[QiyaoWei]

Hi Yu,

Thanks for the quick reply! However, I did make sure that my environment is as defined by the repo, but neither CPU or GPU worked.

I did notice, however, that there was a slightly longer error message when I created a new environment of tensorflow-gpu==1.12. It is as follows--

InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInputOp only supports NHWC.
[[node G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose (defined at :191) = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose/output_shape, G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/AddN, G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose/Switch:1)]]

I have read online that I might be able to get away with simply changing all the places where it says "NCHW" to "NHWC", but I spent some time searching without results.

P.S. Not sure if tf==1.12 is too old, but when I created a new environment of python==3.6.2 and tf==1.12 I had to explicitly do "pip install tensorflow_estimator==1.10.12" in order to import tensorflow. I doubt that has anything to do with the problem tho

[rangmiao]

hello, i have the same problem when running this code . my environment is python 3.6.10 and tensorflow 1.12.0
InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInputOp only supports NHWC.
[[node G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose (defined at :191) = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose/output_shape, G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/AddN, G_synthesis_2/cond/cond/cond/cond/128x128/Conv0_up/conv2d_transpose/Switch:1)]]