This repository was archived by the owner on Nov 13, 2024. It is now read-only.
This repository was archived by the owner on Nov 13, 2024. It is now read-only.
Error: Conv2DCustomBackpropFilterOp only supports NHWC. #937
Description
Everything with my images appear to be in order. I've tried reducing batch sizes and looking for this same error elsewhere on the internet for a quick fix but the error persists. It always happens after the samples from the dst and src have been 100% loaded and it is about to start training.
I'm on Linux so I'm using the scripts here namely the ./6_train_SAEHD.sh scipt when I encounter this error.
================ Model Summary =================
== ==
== Model name: new_SAEHD ==
== ==
== Current iteration: 0 ==
== ==
==-------------- Model Options ---------------==
== ==
== resolution: 128 ==
== face_type: f ==
== models_opt_on_gpu: True ==
== archi: df ==
== ae_dims: 128 ==
== e_dims: 32 ==
== d_dims: 32 ==
== d_mask_dims: 16 ==
== masked_training: True ==
== eyes_prio: False ==
== uniform_yaw: False ==
== lr_dropout: n ==
== random_warp: True ==
== gan_power: 0.0 ==
== true_face_power: 0.0 ==
== face_style_power: 0.0 ==
== bg_style_power: 0.0 ==
== ct_mode: none ==
== clipgrad: False ==
== pretrain: False ==
== autobackup_hour: 4 ==
== write_preview_history: False ==
== target_iter: 0 ==
== random_flip: True ==
== batch_size: 4 ==
== ==
==---------------- Running On ----------------==
== ==
== Device index: 0 ==
== Name: GeForce GTX 1080 Ti ==
== VRAM: 10.91GB ==
== Device index: 1 ==
== Name: GeForce GTX 1080 Ti ==
== VRAM: 10.92GB ==
== ==
================================================
Starting. Press "Enter" to stop training and save model.
Trying to do the first iteration. If an error occurs, reduce the model parameters.
You are training the model from scratch. It is strongly recommended to use a pretrained model to speed up the training and improve the quality.
Error: Conv2DCustomBackpropFilterOp only supports NHWC.
[[node gradients/Conv2D_27_grad/Conv2DBackpropFilter (defined at media/iii/Q2/tor/face/DeepFaceLab/core/leras/ops/__init__.py:55) ]]
Errors may have originated from an input operation.
Input Source operations connected to node gradients/Conv2D_27_grad/Conv2DBackpropFilter:
Pad_26 (defined at media/iii/Q2/tor/face/DeepFaceLab/core/leras/layers/Conv2D.py:97)
Original stack trace for 'gradients/Conv2D_27_grad/Conv2DBackpropFilter':
File "home/iii/miniconda3/envs/face4/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "home/iii/miniconda3/envs/face4/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "home/iii/miniconda3/envs/face4/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "media/iii/Q2/tor/face/DeepFaceLab/mainscripts/Trainer.py", line 45, in trainerThread
model = models.import_model(model_class_name)(
File "media/iii/Q2/tor/face/DeepFaceLab/models/ModelBase.py", line 189, in __init__
self.on_initialize()
File "media/iii/Q2/tor/face/DeepFaceLab/models/Model_SAEHD/Model.py", line 471, in on_initialize
gpu_G_loss_gvs += [ nn.gradients ( gpu_G_loss, self.src_dst_trainable_weights ) ]
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/ops/__init__.py", line 55, in tf_gradients
grads = gradients.gradients(loss, vars, colocate_gradients_with_ops=True )
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gradients_impl.py", line 169, in gradients
return gradients_util._GradientsHelper(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py", line 668, in _GradientsHelper
in_grads = _MaybeCompile(grad_scope, op, func_call,
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py", line 336, in _MaybeCompile
return grad_fn() # Exit early
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py", line 669, in <lambda>
lambda: grad_fn(op, *out_grads))
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/nn_grad.py", line 597, in _Conv2DGrad
gen_nn_ops.conv2d_backprop_filter(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1132, in conv2d_backprop_filter
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
ret = Operation(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
self._traceback = tf_stack.extract_stack()
...which was originally created as op 'Conv2D_27', defined at:
File "home/iii/miniconda3/envs/face4/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
[elided 3 identical lines from previous traceback]
File "media/iii/Q2/tor/face/DeepFaceLab/models/ModelBase.py", line 189, in __init__
self.on_initialize()
File "media/iii/Q2/tor/face/DeepFaceLab/models/Model_SAEHD/Model.py", line 337, in on_initialize
gpu_pred_src_src, gpu_pred_src_srcm = self.decoder_src(gpu_src_code)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/models/ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/archis/DeepFakeArchi.py", line 190, in forward
m = self.upscalem2(m)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/models/ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/archis/DeepFakeArchi.py", line 57, in forward
x = self.conv1(x)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/layers/LayerBase.py", line 14, in __call__
return self.forward(*args, **kwargs)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/layers/Conv2D.py", line 99, in forward
x = tf.nn.conv2d(x, weight, self.strides, 'VALID', dilations=self.dilations, data_format=nn.data_format)
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/nn_ops.py", line 2264, in conv2d
return gen_nn_ops.conv2d(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 975, in conv2d
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
Traceback (most recent call last):
File "/home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropFilterOp only supports NHWC.
[[{{node gradients/Conv2D_27_grad/Conv2DBackpropFilter}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/media/iii/Q2/tor/face/DeepFaceLab/mainscripts/Trainer.py", line 123, in trainerThread
iter, iter_time = model.train_one_iter()
File "/media/iii/Q2/tor/face/DeepFaceLab/models/ModelBase.py", line 462, in train_one_iter
losses = self.onTrainOneIter()
File "/media/iii/Q2/tor/face/DeepFaceLab/models/Model_SAEHD/Model.py", line 636, in onTrainOneIter
src_loss, dst_loss = self.src_dst_train (warped_src, target_src, target_srcm_all, warped_dst, target_dst, target_dstm_all)
File "/media/iii/Q2/tor/face/DeepFaceLab/models/Model_SAEHD/Model.py", line 497, in src_dst_train
s, d, _ = nn.tf_sess.run ( [ src_loss, dst_loss, src_dst_loss_gv_op],
File "/home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 957, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1180, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1358, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropFilterOp only supports NHWC.
[[node gradients/Conv2D_27_grad/Conv2DBackpropFilter (defined at media/iii/Q2/tor/face/DeepFaceLab/core/leras/ops/__init__.py:55) ]]
Errors may have originated from an input operation.
Input Source operations connected to node gradients/Conv2D_27_grad/Conv2DBackpropFilter:
Pad_26 (defined at media/iii/Q2/tor/face/DeepFaceLab/core/leras/layers/Conv2D.py:97)
Original stack trace for 'gradients/Conv2D_27_grad/Conv2DBackpropFilter':
File "home/iii/miniconda3/envs/face4/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "home/iii/miniconda3/envs/face4/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "home/iii/miniconda3/envs/face4/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "media/iii/Q2/tor/face/DeepFaceLab/mainscripts/Trainer.py", line 45, in trainerThread
model = models.import_model(model_class_name)(
File "media/iii/Q2/tor/face/DeepFaceLab/models/ModelBase.py", line 189, in __init__
self.on_initialize()
File "media/iii/Q2/tor/face/DeepFaceLab/models/Model_SAEHD/Model.py", line 471, in on_initialize
gpu_G_loss_gvs += [ nn.gradients ( gpu_G_loss, self.src_dst_trainable_weights ) ]
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/ops/__init__.py", line 55, in tf_gradients
grads = gradients.gradients(loss, vars, colocate_gradients_with_ops=True )
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gradients_impl.py", line 169, in gradients
return gradients_util._GradientsHelper(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py", line 668, in _GradientsHelper
in_grads = _MaybeCompile(grad_scope, op, func_call,
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py", line 336, in _MaybeCompile
return grad_fn() # Exit early
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py", line 669, in <lambda>
lambda: grad_fn(op, *out_grads))
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/nn_grad.py", line 597, in _Conv2DGrad
gen_nn_ops.conv2d_backprop_filter(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1132, in conv2d_backprop_filter
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
ret = Operation(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
self._traceback = tf_stack.extract_stack()
...which was originally created as op 'Conv2D_27', defined at:
File "home/iii/miniconda3/envs/face4/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
[elided 3 identical lines from previous traceback]
File "media/iii/Q2/tor/face/DeepFaceLab/models/ModelBase.py", line 189, in __init__
self.on_initialize()
File "media/iii/Q2/tor/face/DeepFaceLab/models/Model_SAEHD/Model.py", line 337, in on_initialize
gpu_pred_src_src, gpu_pred_src_srcm = self.decoder_src(gpu_src_code)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/models/ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/archis/DeepFakeArchi.py", line 190, in forward
m = self.upscalem2(m)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/models/ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/archis/DeepFakeArchi.py", line 57, in forward
x = self.conv1(x)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/layers/LayerBase.py", line 14, in __call__
return self.forward(*args, **kwargs)
File "media/iii/Q2/tor/face/DeepFaceLab/core/leras/layers/Conv2D.py", line 99, in forward
x = tf.nn.conv2d(x, weight, self.strides, 'VALID', dilations=self.dilations, data_format=nn.data_format)
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/nn_ops.py", line 2264, in conv2d
return gen_nn_ops.conv2d(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 975, in conv2d
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "home/iii/miniconda3/envs/face4/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
^Z
[2]+ Stopped ./6_train_SAEHD.sh
(face4) iii@iii-MS-7A32:/media/iii/Q2/tor/face/scripts$ python