InvalidArgumentError (see above for traceback): Reduction axis 1 is empty in shape [1,0]

[gulingfengze]

• System information

What is the top-level directory of the model you are using:object_detection
Have I written custom code (as opposed to using a stock example script provided in TensorFlow):No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): 1.8
Bazel version (if compiling from source):
CUDA/cuDNN version: 9.0
GPU model and memory:GTX 1080
Exact command to reproduce:

• Describe the problem

I am using the fasterrcnn-resnet50 training data to report an error:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "/home/wangyutang/models-master/research/object_detection/trainer.py", line 399, in train
saver=saver)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 784, in train
ignore_live_threads=ignore_live_threads)
File "/home/wangyutang/anaconda3/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 1000, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 828, in stop
ignore_live_threads=ignore_live_threads)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 495, in run
self.run_loop()
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 1030, in run_loop
self._sv.global_step])
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Reduction axis 1 is empty in shape [1,0]
[[Node: Loss/RPNLoss/Match/cond/ArgMax_1 = ArgMax[T=DT_FLOAT, Tidx=DT_INT32, output_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/RPNLoss/Match/cond/Shape_1/Switch:1, Loss/RPNLoss/Match/cond/ArgMax_1/dimension)]]
[[Node: Loss/RPNLoss/objectness_loss/_961 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3809_Loss/RPNLoss/objectness_loss", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'Loss/RPNLoss/Match/cond/ArgMax_1', defined at:
File "train.py", line 184, in
tf.app.run()
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "/home/wangyutang/models-master/research/object_detection/trainer.py", line 275, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "/home/wangyutang/models-master/research/slim/deployment/model_deploy.py", line 193, in create_clones
outputs = model_fn(args, **kwargs)
File "/home/wangyutang/models-master/research/object_detection/trainer.py", line 200, in _create_losses
losses_dict = detection_model.loss(prediction_dict, true_image_shapes)
File "/home/wangyutang/models-master/research/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 1596, in loss
groundtruth_classes_with_background_list)
File "/home/wangyutang/models-master/research/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 1650, in _loss_rpn
groundtruth_boxlists, len(groundtruth_boxlists)[None])
File "/home/wangyutang/models-master/research/object_detection/core/target_assigner.py", line 447, in batch_assign_targets
anchors, gt_boxes, gt_class_targets, gt_weights)
File "/home/wangyutang/models-master/research/object_detection/core/target_assigner.py", line 162, in assign
match = self._matcher.match(match_quality_matrix, **params)
File "/home/wangyutang/models-master/research/object_detection/core/matcher.py", line 239, in match
return Match(self._match(similarity_matrix, **params),
File "/home/wangyutang/models-master/research/object_detection/matchers/argmax_matcher.py", line 190, in _match
_match_when_rows_are_non_empty, _match_when_rows_are_empty)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2063, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1913, in BuildCondBranch
original_result = fn()
File "/home/wangyutang/models-master/research/object_detection/matchers/argmax_matcher.py", line 169, in _match_when_rows_are_non_empty
output_type=tf.int32)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 220, in argmax
return gen_math_ops.arg_max(input, axis, name=name, output_type=output_type)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 783, in arg_max
name=name)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/home/wangyutang/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Reduction axis 1 is empty in shape [1,0]
[[Node: Loss/RPNLoss/Match/cond/ArgMax_1 = ArgMax[T=DT_FLOAT, Tidx=DT_INT32, output_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/RPNLoss/Match/cond/Shape_1/Switch:1, Loss/RPNLoss/Match/cond/ArgMax_1/dimension)]]
[[Node: Loss/RPNLoss/objectness_loss/_961 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3809_Loss/RPNLoss/objectness_loss", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

I need help for this error pls

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

[gulingfengze]

This is my configuration file：

model {
faster_rcnn {
num_classes: 1
image_resizer {
keep_aspect_ratio_resizer {
min_dimension: 600
max_dimension: 1024
}
}
feature_extractor {
type: 'faster_rcnn_resnet50'
first_stage_features_stride: 16
}
first_stage_anchor_generator {
grid_anchor_generator {
scales: [0.25, 0.5, 1.0, 2.0]
aspect_ratios: [0.5, 1.0, 2.0]
height_stride: 16
width_stride: 16
}
}
first_stage_box_predictor_conv_hyperparams {
op: CONV
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
truncated_normal_initializer {
stddev: 0.01
}
}
}
first_stage_nms_score_threshold: 0.0
first_stage_nms_iou_threshold: 0.7
first_stage_max_proposals: 300
first_stage_localization_loss_weight: 2.0
first_stage_objectness_loss_weight: 1.0
initial_crop_size: 14
maxpool_kernel_size: 2
maxpool_stride: 2
second_stage_box_predictor {
mask_rcnn_box_predictor {
use_dropout: false
dropout_keep_probability: 1.0
fc_hyperparams {
op: FC
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
variance_scaling_initializer {
factor: 1.0
uniform: true
mode: FAN_AVG
}
}
}
}
}
second_stage_post_processing {
batch_non_max_suppression {
score_threshold: 0.0
iou_threshold: 0.6
max_detections_per_class: 100
max_total_detections: 300
}
score_converter: SOFTMAX
}
second_stage_localization_loss_weight: 2.0
second_stage_classification_loss_weight: 1.0
}
}

train_config: {
batch_size: 1
optimizer {
momentum_optimizer: {
learning_rate: {
manual_step_learning_rate {
initial_learning_rate: 0.0003
schedule {
step: 900000
learning_rate: .00003
}
schedule {
step: 1200000
learning_rate: .000003
}
}
}
momentum_optimizer_value: 0.9
}
use_moving_average: false
}
gradient_clipping_by_norm: 10.0
fine_tune_checkpoint: "/home/glf/models-master/research/object_detection/faster_rcnn_resnet50_coco_2018_01_28/model.ckpt"
from_detection_checkpoint: true

num_steps: 200000
data_augmentation_options {
random_horizontal_flip {
}
}
}

train_input_reader: {
tf_record_input_reader {
input_path: "/home/glf/projects/train.record"
}
label_map_path: "/home/glf/projects/ng.pbtxt"
}

eval_config: {
num_examples: 8000

max_evals: 10
}

eval_input_reader: {
tf_record_input_reader {
input_path: "/home/glf/projects/test.record"
}
label_map_path: "/home/glf/projects/ng.pbtxt"
shuffle: false
num_readers: 1
}

My training image size is 5000x250, which will cause this error。But changing the size of the training data to 500x250，its will allow it to train normally.

[safooray]

Could someone please provide some clarification here? I have seen this error message posted several times but people have offered different solutions none of which solve my problem:

https://stackoverflow.com/questions/49272943/reduction-axis-1-is-empty-in-shape-9-0
https://stackoverflow.com/questions/44995679/reduction-axis-1-is-empty-in-shape-x-0?rq=1
broadinstitute/keras-rcnn#169
@afathi3 @jch1
My images are 64 * 64.

[pkulzc]

Sorry for the late response. Is this issue still happening?

[gulingfengze]

@pkulzc I made the following changes:

image_resizer { keep_aspect_ratio_resizer{ min_dimension:250 max_dimension:5000 } }
to
image_resizer { fixed_shape_resizer{ height:250 widith:5000 } }
, It can be trained normally.

[kujta1]

@pkulzc I made the following changes:

image_resizer { fixed_shape_resizer{ height:250 widith:5000 } }
to
image_resizer { fixed_shape_resizer{ height:250 widith:5000 } }
, It can be trained normally.
you have the same text as before changing and after changing. Can you please clarify this ?

[gulingfengze]

@kujta1 @pkulzc I'm very sorry that I made a mistake in the previous content. Now it has been corrected.

[xor007]

@pkulzc I made the following changes:

image_resizer { keep_aspect_ratio_resizer{ min_dimension:250 max_dimension:5000 } }
to
image_resizer { fixed_shape_resizer{ height:250 widith:5000 } }
, It can be trained normally.

This is the only way I was able to avoid #170, while training a different dataset.

[biendltb]

@pkulzc I made the following changes:
image_resizer { keep_aspect_ratio_resizer{ min_dimension:250 max_dimension:5000 } }
to
image_resizer { fixed_shape_resizer{ height:250 widith:5000 } }
, It can be trained normally.

This is the only way I was able to avoid #170, while training a different dataset.

This solution doesn't work for my case. The issue still presents in the current version. My image size is kinda small (~50 - 70 pixels) which might cause the issue. I use the pre-trained model faster_rcnn_resnet101_coco to re-train my object detection model.

However, this problem didn't happen to ssd_mobilenet_v1_coco with a similar configuration. This brought me to the thought that it's an image-size-related issue of Faster R-CNN models.

[KuznetsovIllya]

Hi there. Maybe solution I suggested on similar question will help: matterport/Mask_RCNN#494 (comment)
Probably resizing the image affects the size of the anchor boxes, generated on the first stage - I suppose, that the size of the anchor boxes is the source of the issue.

[zonasw]

I had the same problem. This question has been bothering me.

[zonasw]

I solved this problem by deleting some images that contained small annotations.

[viralparekh]

@zonasw how small annotations did you remove?
smaller than 16x16 or some other number?

[adonisDias]

I solved the problema adjusting the fixed_shape_resizer {
height: 256
width: 256
}
before was 1024, 1024.