Error using SSD-mobilenet-v1-coco: Incompatible shapes: [3,1917] vs. [16,1]

[ZhenyF]

System information

• What is the top-level directory of the model you are using:tensorflow/models
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Win10
• TensorFlow installed from (source or binary):binary
• TensorFlow version (use command below):1.9
• CUDA/cuDNN version: CUDA8.0
• GPU model and memory:GTX 1080TI 11G

Describe the problem

I tried train a facial detector using object detection API. The dataset is WIDER Face dataset and the conversion code (to tfrecord) is from here. The pretrained model is ssd_mobilenet_v1_coco and the config file is ssd_mobilenet_v1_focal_loss_pets. I didn't do any modification except setting the number of class to 1, imcreasing the batch size from 25 to 32 and changing the address of label map, checkpoint, and tfrecords. The initialisation process looks fine but I got an error after that:

`I0927 11:59:49.965208 7440 tf_logging.py:115] Restoring parameters from C:/Users/mhb14150/Documents/TF_object_api/My_model/check_point/ssd_mobilenet_v1_coco/model.ckpt
INFO:tensorflow:Running local_init_op.
I0927 11:59:50.130070 7440 tf_logging.py:115] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0927 11:59:50.387650 7440 tf_logging.py:115] Done running local_init_op.
INFO:tensorflow:Starting Session.
I0927 11:59:57.981596 7440 tf_logging.py:115] Starting Session.
INFO:tensorflow:Saving checkpoint to path C:/Users/mhb14150/Documents/TF_object_api/My_model/mode_trained\model.ckpt
I0927 11:59:58.153400 12356 tf_logging.py:115] Saving checkpoint to path C:/Users/mhb14150/Documents/TF_object_api/My_model/mode_trained\model.ckpt
INFO:tensorflow:Starting Queues.
I0927 11:59:58.169021 7440 tf_logging.py:115] Starting Queues.
INFO:tensorflow:global_step/sec: 0
I0927 12:00:09.622732 8096 tf_logging.py:159] global_step/sec: 0
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Incompatible shapes: [3,1917] vs. [16,1]
[[Node: Loss/Match_25/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match_25/cond/one_hot/_4475, Loss/Match_25/cond/Cast_2)]]
[[Node: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/BatchNorm/AssignMovingAvg_1/mul/_3975 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5635_...gAvg_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'Loss/Match_25/cond/mul_4', defined at:
File "train.py", line 184, in
tf.app.run()
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\util\deprecation.py", line 250, in new_func
return func(*args, **kwargs)
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\legacy\trainer.py", line 290, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, **kwargs)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\legacy\trainer.py", line 205, in _create_losses
losses_dict = detection_model.loss(prediction_dict, true_image_shapes)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\meta_architectures\ssd_meta_arch.py", line 680, in loss
keypoints, weights)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\meta_architectures\ssd_meta_arch.py", line 853, in _assign_targets
groundtruth_weights_list)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\target_assigner.py", line 483, in batch_assign_targets
anchors, gt_boxes, gt_class_targets, unmatched_class_label, gt_weights)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\target_assigner.py", line 182, in assign
valid_rows=tf.greater(groundtruth_weights, 0))
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\matcher.py", line 241, in match
return Match(self._match(similarity_matrix, valid_rows),
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\matchers\argmax_matcher.py", line 194, in _match
_match_when_rows_are_non_empty, _match_when_rows_are_empty)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\util\deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 2040, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 1890, in BuildCondBranch
original_result = fn()
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\matchers\argmax_matcher.py", line 175, in _match_when_rows_are_non_empty
tf.cast(tf.expand_dims(valid_rows, axis=-1), dtype=tf.float32))
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\math_ops.py", line 847, in binary_op_wrapper
return func(x, y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1091, in _mul_dispatch
return gen_math_ops.mul(x, y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 5066, in mul
"Mul", x=x, y=y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
op_def=op_def)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [3,1917] vs. [16,1]
[[Node: Loss/Match_25/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match_25/cond/one_hot/_4475, Loss/Match_25/cond/Cast_2)]]
[[Node: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/BatchNorm/AssignMovingAvg_1/mul/_3975 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5635_...gAvg_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

I0927 12:00:12.403529 7440 tf_logging.py:115] Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Incompatible shapes: [3,1917] vs. [16,1]
[[Node: Loss/Match_25/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match_25/cond/one_hot/_4475, Loss/Match_25/cond/Cast_2)]]
[[Node: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/BatchNorm/AssignMovingAvg_1/mul/_3975 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5635_...gAvg_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'Loss/Match_25/cond/mul_4', defined at:
File "train.py", line 184, in
tf.app.run()
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\util\deprecation.py", line 250, in new_func
return func(*args, **kwargs)
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\legacy\trainer.py", line 290, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, **kwargs)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\legacy\trainer.py", line 205, in _create_losses
losses_dict = detection_model.loss(prediction_dict, true_image_shapes)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\meta_architectures\ssd_meta_arch.py", line 680, in loss
keypoints, weights)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\meta_architectures\ssd_meta_arch.py", line 853, in _assign_targets
groundtruth_weights_list)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\target_assigner.py", line 483, in batch_assign_targets
anchors, gt_boxes, gt_class_targets, unmatched_class_label, gt_weights)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\target_assigner.py", line 182, in assign
valid_rows=tf.greater(groundtruth_weights, 0))
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\matcher.py", line 241, in match
return Match(self._match(similarity_matrix, valid_rows),
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\matchers\argmax_matcher.py", line 194, in _match
_match_when_rows_are_non_empty, _match_when_rows_are_empty)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\util\deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 2040, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 1890, in BuildCondBranch
original_result = fn()
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\matchers\argmax_matcher.py", line 175, in _match_when_rows_are_non_empty
tf.cast(tf.expand_dims(valid_rows, axis=-1), dtype=tf.float32))
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\math_ops.py", line 847, in binary_op_wrapper
return func(x, y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1091, in _mul_dispatch
return gen_math_ops.mul(x, y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 5066, in mul
"Mul", x=x, y=y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
op_def=op_def)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [3,1917] vs. [16,1]
[[Node: Loss/Match_25/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match_25/cond/one_hot/_4475, Loss/Match_25/cond/Cast_2)]]
[[Node: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/BatchNorm/AssignMovingAvg_1/mul/_3975 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5635_...gAvg_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Traceback (most recent call last):
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
return fn(*args)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [3,1917] vs. [16,1]
[[Node: Loss/Match_25/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match_25/cond/one_hot/_4475, Loss/Match_25/cond/Cast_2)]]
[[Node: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/BatchNorm/AssignMovingAvg_1/mul/_3975 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5635_...gAvg_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\util\deprecation.py", line 250, in new_func
return func(*args, **kwargs)
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\legacy\trainer.py", line 415, in train
saver=saver)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 770, in train
sess, train_op, global_step, train_step_kwargs)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 487, in train_step
run_metadata=run_metadata)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
run_metadata_ptr)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
run_metadata)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [3,1917] vs. [16,1]
[[Node: Loss/Match_25/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match_25/cond/one_hot/_4475, Loss/Match_25/cond/Cast_2)]]
[[Node: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/BatchNorm/AssignMovingAvg_1/mul/_3975 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5635_...gAvg_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'Loss/Match_25/cond/mul_4', defined at:
File "train.py", line 184, in
tf.app.run()
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\util\deprecation.py", line 250, in new_func
return func(*args, **kwargs)
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\legacy\trainer.py", line 290, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, **kwargs)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\legacy\trainer.py", line 205, in _create_losses
losses_dict = detection_model.loss(prediction_dict, true_image_shapes)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\meta_architectures\ssd_meta_arch.py", line 680, in loss
keypoints, weights)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\meta_architectures\ssd_meta_arch.py", line 853, in _assign_targets
groundtruth_weights_list)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\target_assigner.py", line 483, in batch_assign_targets
anchors, gt_boxes, gt_class_targets, unmatched_class_label, gt_weights)
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\target_assigner.py", line 182, in assign
valid_rows=tf.greater(groundtruth_weights, 0))
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\core\matcher.py", line 241, in match
return Match(self._match(similarity_matrix, valid_rows),
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\matchers\argmax_matcher.py", line 194, in _match
_match_when_rows_are_non_empty, _match_when_rows_are_empty)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\util\deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 2040, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 1890, in BuildCondBranch
original_result = fn()
File "C:\Users\mhb14150\Documents\TF_object_api\models\research\object_detection\matchers\argmax_matcher.py", line 175, in _match_when_rows_are_non_empty
tf.cast(tf.expand_dims(valid_rows, axis=-1), dtype=tf.float32))
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\math_ops.py", line 847, in binary_op_wrapper
return func(x, y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1091, in _mul_dispatch
return gen_math_ops.mul(x, y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 5066, in mul
"Mul", x=x, y=y, name=name)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
op_def=op_def)
File "C:\Users\mhb14150\Desktop\winpython\WinPython-64bit-3.5.3.1Qt5\python-3.5.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [3,1917] vs. [16,1]
[[Node: Loss/Match_25/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match_25/cond/one_hot/_4475, Loss/Match_25/cond/Cast_2)]]
[[Node: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/BatchNorm/AssignMovingAvg_1/mul/_3975 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5635_...gAvg_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]`

Can someone help me with this? Thanks!

[Aitul]

I am havving the same issue today

[javad314]

Unfortunately, I am having the same error and despite all my efforts, I have not find the reason. Like you, I have one class, and I am using ssd_mobilenet_v1_pets model. The interesting part is, with every run, I am getting this error with different shapes! like [2,1917] vs. [4,1] and [3,1917] vs. [5,1] and so on!

[hi-ilkin]

@javad314 I also had same problem, with changing bounding box valuas from float to int had solved my problem. I'm using open image dataset

UPDATE: For us this only worked for faster_rcnn_resnet50. @ZhenyF The bounding box info that we downloaded from OID was not normilized.

[ZhenyF]

@huseynlilkin How do you assign it to int? The coordinates anchor are normalised to [0,1]

[linlin860320]

I have the same problem with @javad314 when using ssd_mobilenet_v1_pets.config. After run 29 steps the problem occurs.
InvalidArgumentError :Incompatible shapes: [2,1917] vs. [4,1]

and when I use ssd_mobilenet_v2_coco_configs has the similar problem with Incompatible shapes: [6,1917] vs. [7,1]

When using ssd models to train have the problem with shapes, but I don't know how to fix it.

[faizan1041]

Anyone found solution? Here's my question: https://stackoverflow.com/questions/52623733/tensorflow-incompatible-shapes-error-while-training?noredirect=1#comment92189808_52623733
I was able to do this successfully with 1 class, now I have added 2 classes and a few more images to the dataset, I have regenerated xml to csv and tf records by deleting old ones, pbtxt file is correct and also deleted old checkpoints inside the training directory but when I run the training job like this:

python legacy/train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config

following this tutorial . I get the following error:

InvalidArgumentError (see above for traceback): Incompatible shapes: [2,1917] vs. [4,1]
[[Node: Loss/Match/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match/cond/one_hot, Loss/Match/cond/Cast_2)]]

[michaelcamba]

I am encountering the same issue. after several steps I encounter this error:

InvalidArgumentError (see above for traceback): Incompatible shapes: [2,1917] vs. [3,1]
[[Node: Loss/Match/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Loss/Match/cond/one_hot, Loss/Match/cond/Cast_2)]]

Can someone help me with this? Thanks in advance

[faizan1041]

I solved this by changing model to faster_rcnn_inception_v2. However with ssd_mobile_net_v1_coco and ssd_mobile_net_v2_coco I'm not able to do that.

[w5688414]

It works fine in faster rcnn, but doestn't work in ssd mobile net? can anyone solve this problem?

[grofattila]

Same here:

SSD with Mobilenet v1

INFO:tensorflow:Error reported to Coordinator: Incompatible shapes: [2,1917] vs. [3,1]
[[Node: Loss/Match_14/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Loss/Match_14/cond/one_hot, Loss/Match_14/cond/Cast_2)]]

[hi-ilkin]

@ZhenyF can you solve this problem for SSD ?

[hi-ilkin]

I used model_main.py instead of legacy/train.py and problem solved. I can successfully trained SSD models. Running model_main.py:

python3 model_main.py
--pipeline_config_path=PRETRAINED_CONFIG
--model_dir=TRAINING_OUTPUT
--num_train_steps=TRAINING_STEP_COUNT
--alsologtostderr
--sample_1_of_n_eval_examples=1

If nothing printed on console add tf.logging.set_verbosity(tf.logging.INFO) after imports in model_main.py .

[w5688414]

@huseynlilkin my command is
python3 model_main.py --pipeline_config_path=ssd_mobilenet_v1_coco.config --model_dir=training/ssd_mobilenet_v1_coco --num_train_steps=100000 --alsologtostderr --sample_1_of_n_eval_examples=1
but got another error
`084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9841 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
Traceback (most recent call last):
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 1 values, but the requested shape requires a multiple of 13
[[Node: Reshape_11 = Reshape[T=DT_BOOL, Tshape=DT_INT32, _device="/device:CPU:0"](Cast, Reshape_13/shape)]]
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[8], [8,300,300,3], [8,2], [8,3], [8,100], [8,100,4], [8,100,90], [8,100,90], [8,100], [8,100], [8,100], [8]], output_types=[DT_INT32, DT_FLOAT, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "model_main.py", line 109, in
tf.app.run()
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "model_main.py", line 105, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 447, in train_and_evaluate
return executor.run()
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 531, in run
return self.run_local()
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 669, in run_local
hooks=train_hooks)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 366, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1119, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1135, in _train_model_default
saving_listeners)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1336, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 577, in run
run_metadata=run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1053, in run
run_metadata=run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1144, in run
raise six.reraise(*original_exc_info)
File "/home/eric/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1129, in run
return self._sess.run(*args, **kwargs)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1201, in run
run_metadata=run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 981, in run
return self._sess.run(*args, **kwargs)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 1 values, but the requested shape requires a multiple of 13
[[Node: Reshape_11 = Reshape[T=DT_BOOL, Tshape=DT_INT32, _device="/device:CPU:0"](Cast, Reshape_13/shape)]]
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[8], [8,300,300,3], [8,2], [8,3], [8,100], [8,100,4], [8,100,90], [8,100,90], [8,100], [8,100], [8,100], [8]], output_types=[DT_INT32, DT_FLOAT, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
`
so how about your tensorflow versions and other configurations