Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[10] = 10 is not in [0, 0) #9067

Open
magedhelmy1 opened this issue Aug 7, 2020 · 15 comments
Assignees
Labels

Comments

@magedhelmy1
Copy link

magedhelmy1 commented Aug 7, 2020

Prerequisites

Please answer the following questions for yourself before submitting an issue.

  • [x ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
  • [x ] I am reporting the issue to the correct repository. (Model Garden official or research directory)
  • [x ] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/using_your_own_dataset.md

2. Describe the bug

I am trying to write the equivalent of this code which converts CSV to TF records but instead, I am trying to convert from JSON to TFrecords. I am trying to generate TFrecords for using it in object detection API.

Here is my full error message

Traceback (most recent call last):
  File "model_main_tf2.py", line 113, in <module>
    tf.compat.v1.app.run()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main_tf2.py", line 109, in main
    record_summaries=FLAGS.record_summaries)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\model_lib_v2.py", line 561, in train_loop
    unpad_groundtruth_tensors)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\model_lib_v2.py", line 342, in load_fine_tune_checkpoint
    features, labels = iter(input_dataset).next()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 645, in next
    return self.__next__()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 649, in __next__
    return self.get_next()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 694, in get_next
    self._iterators[i].get_next_as_list_static_shapes(new_name))
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1474, in get_next_as_list_static_shapes
    return self._iterator.get_next()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\data\ops\multi_device_iterator_ops.py", line 581, in get_next
    result.append(self._device_iterators[i].get_next())
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 825, in get_next
    return self._next_internal()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 764, in _next_internal
    return structure.from_compatible_tensor_list(self._element_spec, ret)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\eager\context.py", line 2105, in execution_mode
    executor_new.wait()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\eager\executor.py", line 67, in wait
    pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[12] = 12 is not in [0, 0)
         [[{{node GatherV2_4}}]]
         [[MultiDeviceIteratorGetNextFromShard]]
         [[RemoteCall]]

And here is my code, which is an attempt to convert JSON files into TFrecords

Sample JSON file

{
  "0.jpg59329": {
    "filename": "0.jpg",
    "size": 59329,
    "regions": [{
      "shape_attributes": {
        "name": "rect",
        "x": 412,
        "y": 130,
        "width": 95,
        "height": 104
      },
      "region_attributes": {}
    }, {
      "shape_attributes": {
        "name": "rect",
        "x": 521,
        "y": 82,
        "width": 126,
        "height": 106
      },
      "region_attributes": {}
    }
}

My Python Code

# Ref 1: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/using_your_own_dataset.md
# Ref 2: https://github.com/datitran/raccoon_dataset/blob/master/generate_tfrecord.py

import json
import glob
from object_detection.utils import dataset_util
import tensorflow as tf
from pathlib import Path

flags = tf.compat.v1.app.flags
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
FLAGS = flags.FLAGS


def json_to_tf(jsonFile, im):
    with open(im, "rb") as image:
        encoded_image_data = image.read()

    with open(jsonFile) as json_file:
        data = json.load(json_file)

        for key, value in data.items():
            width = 1920
            height = 1080
            filename = value["filename"]
            filename = filename.encode('utf8')
            image_format = b'jpeg'
            xmins = []
            xmaxs = []
            ymins = []
            ymaxs = []
            classes_text = []
            classes = []

            for x in value["regions"]:
                xmins.append(x["shape_attributes"]['x'])
                xmaxs.append(x["shape_attributes"]['width'] + x["shape_attributes"]['x'])
                ymins.append(x["shape_attributes"]['y'])
                ymaxs.append(x["shape_attributes"]['height'] + x["shape_attributes"]['y'])
                classes_text.append("cars".encode('utf8'))
                classes.append(1)

            tf_example = tf.train.Example(features=tf.train.Features(feature={
                'image/height': dataset_util.int64_feature(height),
                'image/width': dataset_util.int64_feature(width),
                'image/filename': dataset_util.bytes_feature(filename),
                'image/source_id': dataset_util.bytes_feature(filename),
                'image/encoded': dataset_util.bytes_feature(encoded_image_data),
                'image/format': dataset_util.bytes_feature(image_format),
                'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
                'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
                'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
                'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
                'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
                'image/object/class/label': dataset_util.int64_list_feature(classes),
            }))

            return tf_example


writer = tf.compat.v1.python_io.TFRecordWriter("train.record")

for fn in glob.glob("annotation_refined\\*.json"):
    for img in glob.glob("images\\*.jpg"):
        if Path(fn).stem == Path(img).stem:
            tf_example_1 = json_to_tf(fn, img)
            writer.write(tf_example_1.SerializeToString())

writer.close()

Can someone give any tips on what is going wrong?

3. Steps to reproduce

I execute the code above to convert JSON to TF records

4. Expected behavior

A TF record that works which I can use when executing the following

out_dir=output/
mkdir -p $out_dir
python model_main_tf2.py --alsologtostderr --model_dir=$out_dir --checkpoint_every_n=1  \
                         --pipeline_config_path=mask_rcnn_inception_resnet_v2.config \
                         --eval_on_train_data 2>&1 | tee $out_dir/train.log

Here is my config file

# Mask R-CNN with Inception Resnet v2 (no atrous)
# Sync-trained on COCO (with 8 GPUs) with batch size 16 (1024x1024 resolution)
# Initialized from Imagenet classification checkpoint
#
# Train on GPU-8
#
# Achieves 40.4 box mAP and 35.5 mask mAP on COCO17 val

model {
  faster_rcnn {
    number_of_stages: 3
    num_classes: 1
    image_resizer {
      fixed_shape_resizer {
        height: 1024
        width: 1024
      }
    }
    feature_extractor {
      type: 'faster_rcnn_inception_resnet_v2_keras'
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        scales: [0.25, 0.5, 1.0, 2.0]
        aspect_ratios: [0.5, 1.0, 2.0]
        height_stride: 16
        width_stride: 16
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.01
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.7
    first_stage_max_proposals: 300
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 17
    maxpool_kernel_size: 1
    maxpool_stride: 1
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        use_dropout: false
        dropout_keep_probability: 1.0
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
        mask_height: 33
        mask_width: 33
        mask_prediction_conv_depth: 0
        mask_prediction_num_conv_layers: 4
        conv_hyperparams {
          op: CONV
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            truncated_normal_initializer {
              stddev: 0.01
            }
          }
        }
        predict_instance_masks: true
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
    second_stage_mask_prediction_loss_weight: 4.0
    resize_masks: false
  }
}

train_config: {
  batch_size: 16
  num_steps: 2
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        cosine_decay_learning_rate {
          learning_rate_base: 0.008
          total_steps: 2
          warmup_learning_rate: 0.0
          warmup_steps: 1
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint_version: V2
  fine_tune_checkpoint: "mask_rcnn_inception_resnet_v2_1024x1024_coco17_gpu-8/checkpoint/ckpt-0"
  fine_tune_checkpoint_type: "detection"
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
}

train_input_reader: {
  label_map_path: "capillary_labelmap.pbtxt"
  tf_record_input_reader {
    input_path: "train.record"
  }
  load_instance_masks: true
  mask_type: PNG_MASKS
}

eval_config: {
  metrics_set: "coco_detection_metrics"
  metrics_set: "coco_mask_metrics"
  eval_instance_masks: true
  use_moving_averages: false
  batch_size: 512
  include_metrics_per_category: true
}

eval_input_reader: {
  label_map_path: "capillary_labelmap.pbtxt"
  shuffle: false
  num_epochs: 1
  tf_record_input_reader {
    input_path: "train.record"
  }
  load_instance_masks: true
  mask_type: PNG_MASKS
}

5. Additional context

2020-08-07 10:38:35.337851: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-07 10:38:38.463844: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-08-07 10:38:38.513432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.74GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-08-07 10:38:38.513852: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-07 10:38:38.518276: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-07 10:38:38.522366: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-07 10:38:38.524155: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-07 10:38:38.529030: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-07 10:38:38.531871: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-07 10:38:38.540810: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-07 10:38:38.542364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-07 10:38:38.543192: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-07 10:38:38.554233: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x21c5e9cf1e0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-07 10:38:38.554563: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-08-07 10:38:38.555417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.74GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-08-07 10:38:38.555804: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-07 10:38:38.556098: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-07 10:38:38.556281: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-07 10:38:38.556479: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-07 10:38:38.556678: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-07 10:38:38.556864: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-07 10:38:38.557049: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-07 10:38:38.558452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-07 10:38:39.469766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-07 10:38:39.470008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-08-07 10:38:39.470183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-08-07 10:38:39.471856: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8583 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:09:00.0, compute capability: 7.5)
2020-08-07 10:38:39.474601: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x21c05d6bb10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-07 10:38:39.474889: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0807 10:38:39.477294 26156 mirrored_strategy.py:341] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: None
I0807 10:38:39.482280 26156 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0807 10:38:39.482280 26156 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0807 10:38:39.949601 26156 dataset_builder.py:83] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\builders\dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.

W0807 10:38:39.953629 26156 deprecation.py:323] From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\builders\dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.

WARNING:tensorflow:From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\builders\dataset_builder.py:175: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0807 10:38:39.996484 26156 deprecation.py:323] From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\builders\dataset_builder.py:175: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\data_decoders\tf_example_decoder.py:703: calling map_fn (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
W0807 10:38:41.664895 26156 deprecation.py:506] From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\data_decoders\tf_example_decoder.py:703: calling map_fn (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
WARNING:tensorflow:From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\util\dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W0807 10:38:47.547355 26156 deprecation.py:323] From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\util\dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\inputs.py:259: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0807 10:38:50.558459 26156 deprecation.py:323] From C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\inputs.py:259: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Traceback (most recent call last):
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\eager\context.py", line 2102, in execution_mode
    yield
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 758, in _next_internal
    output_shapes=self._flat_output_shapes)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2610, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\framework\ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[10] = 10 is not in [0, 0)
         [[{{node GatherV2_4}}]]
         [[MultiDeviceIteratorGetNextFromShard]]
         [[RemoteCall]] [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "model_main_tf2.py", line 113, in <module>
    tf.compat.v1.app.run()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main_tf2.py", line 109, in main
    record_summaries=FLAGS.record_summaries)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\model_lib_v2.py", line 561, in train_loop
    unpad_groundtruth_tensors)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\object_detection\model_lib_v2.py", line 342, in load_fine_tune_checkpoint
    features, labels = iter(input_dataset).next()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 645, in next
    return self.__next__()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 649, in __next__
    return self.get_next()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 694, in get_next
    self._iterators[i].get_next_as_list_static_shapes(new_name))
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1474, in get_next_as_list_static_shapes
    return self._iterator.get_next()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\data\ops\multi_device_iterator_ops.py", line 581, in get_next
    result.append(self._device_iterators[i].get_next())
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 825, in get_next
    return self._next_internal()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 764, in _next_internal
    return structure.from_compatible_tensor_list(self._element_spec, ret)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\eager\context.py", line 2105, in execution_mode
    executor_new.wait()
  File "C:\ProgramData\anaconda3\envs\4_SOA_OD_v2\lib\site-packages\tensorflow\python\eager\executor.py", line 67, in wait
    pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[10] = 10 is not in [0, 0)
         [[{{node GatherV2_4}}]]
         [[MultiDeviceIteratorGetNextFromShard]]
         [[RemoteCall]]

6. System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
  • TensorFlow installed from (source or binary): Using pip install
  • TensorFlow version (use command below): v2.3.0-rc2-23-gb36436b087 2.3.0
  • Python version: 3.6
  • CUDA/cuDNN version: cudart64_101.dll
  • GPU model and memory: GTX 2080TI 11GB
== check python ===================================================
python version: 3.6.11
python branch: 
python build version: ('default', 'Aug  5 2020 19:41:03')
python compiler version: MSC v.1916 64 bit (AMD64)
python implementation: CPython


== check os platform ===============================================
os: Windows
os kernel version: 10.0.17763
os release version: 10
os platform: Windows-10-10.0.17763-SP0
linux distribution: ('', '', '')
linux os distribution: ('', '', '')
mac version: ('', ('', '', ''), '')
uname: uname_result(system='Windows', node='DESKTOP-8I4BAP8', release='10', version='10.0.17763', machine='AMD64', processor='Intel64 Family 6 Model 158 Stepping 10, GenuineIntel')
architecture: ('64bit', 'WindowsPE')
machine: AMD64


== are we in docker =============================================
No

== compiler =====================================================
c++.exe (MinGW.org GCC Build-20200227-1) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


== check pips ===================================================
numpy                         1.19.1
protobuf                      3.12.3
tensorflow                    2.3.0
tensorflow-addons             0.11.0
tensorflow-datasets           3.2.1
tensorflow-estimator          2.3.0
tensorflow-hub                0.8.0
tensorflow-metadata           0.22.2
tensorflow-model-optimization 0.4.1

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.version.VERSION = 2.3.0
tf.version.GIT_VERSION = v2.3.0-rc2-23-gb36436b087
tf.version.COMPILER_VERSION = MSVC 192628806
2020-08-07 10:33:58.725634: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll

== env ==========================================================
LD_LIBRARY_PATH is unset
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
Fri Aug 07 10:34:01 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208... WDDM  | 00000000:09:00.0  On |                  N/A |
| 25%   30C    P8    28W / 300W |   1352MiB / 11264MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1352    C+G   ...mmersiveControlPanel\SystemSettings.exe N/A      |
|    0      1756    C+G   ...2.0_x64__8wekyb3d8bbwe\WinStore.App.exe N/A      |
|    0      1964    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0      5040    C+G   Insufficient Permissions                   N/A      |
|    0      5424    C+G   C:\Windows\explorer.exe                    N/A      |
|    0     11260    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0     12224    C+G   ...lmy\AppData\Roaming\Spotify\Spotify.exe N/A      |
|    0     12868    C+G   ...osoft.LockApp_cw5n1h2txyewy\LockApp.exe N/A      |
|    0     12900    C+G   C:\Program Files\Hue Sync\HueSync.exe      N/A      |
|    0     13396    C+G   ...1.95.0_x64__8wekyb3d8bbwe\YourPhone.exe N/A      |
|    0     13492    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
|    0     13556    C+G   ...0.6242.0_x64__8wekyb3d8bbwe\GameBar.exe N/A      |
|    0     15188    C+G   ...x64__8wekyb3d8bbwe\Microsoft.Photos.exe N/A      |
|    0     16064    C+G   Insufficient Permissions                   N/A      |
|    0     16748    C+G   ...oftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe N/A      |
|    0     17708    C+G   ...dqpraam7r\AcrobatNotificationClient.exe N/A      |
|    0     17744    C+G   ...rosoft Office\root\Office16\OUTLOOK.EXE N/A      |
|    0     18160    C+G   ...AppData\Local\slack\app-4.7.0\slack.exe N/A      |
|    0     19316    C+G   ...DIA GeForce Experience\NVIDIA Share.exe N/A      |
|    0     21808    C+G   ...16211.0_x64__8wekyb3d8bbwe\Video.UI.exe N/A      |
|    0     22076    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0     24124    C+G   ... Files (x86)\Dropbox\Client\Dropbox.exe N/A      |
+-----------------------------------------------------------------------------+

== cuda libs  ===================================================

== tensorflow installed from info ==================
Name: tensorflow
Version: 2.3.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: c:\programdata\anaconda3\envs\4_soa_od_v2\lib\site-packages
Required-by: tf-models-official

== python version  ==============================================
(major, minor, micro, releaselevel, serial)
(3, 6, 11, 'final', 0)

== bazel version  ===============================================

== check python ===================================================
python version: 3.6.11
python branch: 
python build version: ('default', 'Aug  5 2020 19:41:03')
python compiler version: MSC v.1916 64 bit (AMD64)
python implementation: CPython


== check os platform ===============================================
os: Windows
os kernel version: 10.0.17763
os release version: 10
os platform: Windows-10-10.0.17763-SP0
linux distribution: ('', '', '')
linux os distribution: ('', '', '')
mac version: ('', ('', '', ''), '')
uname: uname_result(system='Windows', node='DESKTOP-8I4BAP8', release='10', version='10.0.17763', machine='AMD64', processor='Intel64 Family 6 Model 158 Stepping 10, GenuineIntel')
architecture: ('64bit', 'WindowsPE')
machine: AMD64


== are we in docker =============================================
No

== compiler =====================================================
c++.exe (MinGW.org GCC Build-20200227-1) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


== check pips ===================================================
numpy                         1.19.1
protobuf                      3.12.3
tensorflow                    2.3.0
tensorflow-addons             0.11.0
tensorflow-datasets           3.2.1
tensorflow-estimator          2.3.0
tensorflow-hub                0.8.0
tensorflow-metadata           0.22.2
tensorflow-model-optimization 0.4.1

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.version.VERSION = 2.3.0
tf.version.GIT_VERSION = v2.3.0-rc2-23-gb36436b087
tf.version.COMPILER_VERSION = MSVC 192628806
2020-08-07 10:35:32.098298: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll

== env ==========================================================
LD_LIBRARY_PATH is unset
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
Fri Aug 07 10:35:34 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208... WDDM  | 00000000:09:00.0  On |                  N/A |
| 25%   30C    P8    26W / 300W |   1344MiB / 11264MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1352    C+G   ...mmersiveControlPanel\SystemSettings.exe N/A      |
|    0      1756    C+G   ...2.0_x64__8wekyb3d8bbwe\WinStore.App.exe N/A      |
|    0      1964    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0      5040    C+G   Insufficient Permissions                   N/A      |
|    0      5424    C+G   C:\Windows\explorer.exe                    N/A      |
|    0     11260    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0     12224    C+G   ...lmy\AppData\Roaming\Spotify\Spotify.exe N/A      |
|    0     12868    C+G   ...osoft.LockApp_cw5n1h2txyewy\LockApp.exe N/A      |
|    0     12900    C+G   C:\Program Files\Hue Sync\HueSync.exe      N/A      |
|    0     13396    C+G   ...1.95.0_x64__8wekyb3d8bbwe\YourPhone.exe N/A      |
|    0     13492    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
|    0     13556    C+G   ...0.6242.0_x64__8wekyb3d8bbwe\GameBar.exe N/A      |
|    0     15188    C+G   ...x64__8wekyb3d8bbwe\Microsoft.Photos.exe N/A      |
|    0     16064    C+G   Insufficient Permissions                   N/A      |
|    0     16748    C+G   ...oftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe N/A      |
|    0     17708    C+G   ...dqpraam7r\AcrobatNotificationClient.exe N/A      |
|    0     17744    C+G   ...rosoft Office\root\Office16\OUTLOOK.EXE N/A      |
|    0     18160    C+G   ...AppData\Local\slack\app-4.7.0\slack.exe N/A      |
|    0     19316    C+G   ...DIA GeForce Experience\NVIDIA Share.exe N/A      |
|    0     21808    C+G   ...16211.0_x64__8wekyb3d8bbwe\Video.UI.exe N/A      |
|    0     22076    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0     24124    C+G   ... Files (x86)\Dropbox\Client\Dropbox.exe N/A      |
+-----------------------------------------------------------------------------+

== cuda libs  ===================================================

== tensorflow installed from info ==================
Name: tensorflow
Version: 2.3.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: c:\programdata\anaconda3\envs\4_soa_od_v2\lib\site-packages
Required-by: tf-models-official

== python version  ==============================================
(major, minor, micro, releaselevel, serial)
(3, 6, 11, 'final', 0)

== bazel version  ===============================================
@magedhelmy1 magedhelmy1 added models:research models that come under research directory type:bug Bug in the code labels Aug 7, 2020
@ravikyram ravikyram assigned ravikyram, tombstone, jch1 and pkulzc and unassigned ravikyram Aug 7, 2020
@waccasuzair
Copy link

same error
Traceback (most recent call last):
File "model_main_tf2.py", line 113, in
tf.compat.v1.app.run()
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "c:\programdata\anaconda5\lib\site-packages\absl\app.py", line 300, in run
_run_main(main, args)
File "c:\programdata\anaconda5\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "model_main_tf2.py", line 110, in main
record_summaries=FLAGS.record_summaries)
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\object_detection-0.1-py3.7.egg\object_detection\model_lib_v2.py", line 569, in train_loop
unpad_groundtruth_tensors)
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\object_detection-0.1-py3.7.egg\object_detection\model_lib_v2.py", line 350, in load_fine_tune_checkpoint
features, labels = iter(input_dataset).next()
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\distribute\input_lib.py", line 645, in next
return self.next()
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\distribute\input_lib.py", line 649, in next
return self.get_next()
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\distribute\input_lib.py", line 694, in get_next
self._iterators[i].get_next_as_list_static_shapes(new_name))
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\distribute\input_lib.py", line 1474, in get_next_as_list_static_shapes
return self._iterator.get_next()
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\data\ops\multi_device_iterator_ops.py", line 581, in get_next
result.append(self._device_iterators[i].get_next())
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 825, in get_next
return self._next_internal()
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 764, in _next_internal
return structure.from_compatible_tensor_list(self._element_spec, ret)
File "C:\ProgramData\Anaconda5\lib\contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\context.py", line 2105, in execution_mode
executor_new.wait()
File "C:\Users\Lenovo\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\executor.py", line 67, in wait
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.284775376][0.262416959][0.104731433]...] [[0.387153327][0.36479491][0.182397455]...]
[[{{node Assert/AssertGuard/else/_25/Assert/AssertGuard/Assert}}]]
[[MultiDeviceIteratorGetNextFromShard]]
[[RemoteCall]]

@maxpaynestory
Copy link

same here

pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.408593744]] [[0.384375]]
	 [[{{node Assert/AssertGuard/else/_49/Assert/AssertGuard/Assert}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
	 [[RemoteCall]]

@waccasuzair
Copy link

same here

pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.408593744]] [[0.384375]]
	 [[{{node Assert/AssertGuard/else/_49/Assert/AssertGuard/Assert}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
	 [[RemoteCall]]

Resolved. For me the problem was that in some cases 'xmin' were larger than 'xmax'.

To my experience this error appears when there is something wrong with the data.

@maxpaynestory
Copy link

same here

pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.408593744]] [[0.384375]]
	 [[{{node Assert/AssertGuard/else/_49/Assert/AssertGuard/Assert}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
	 [[RemoteCall]]

Resolved. For me the problem was that in some cases 'xmin' were larger than 'xmax'.

To my experience this error appears when there is something wrong with the data.

Thanks, it solved my problem after fixing my annotations.

@mashalahmad
Copy link

I got the same error with mask rcnn inception reset. can you please tell how did u fix your annotations?

@maxpaynestory
Copy link

I got the same error with mask rcnn inception reset. can you please tell how did u fix your annotations?

annotations should be from top left to bottom right. Take a look at video i am sharing below

https://drive.google.com/file/d/1pSLMu6lEqpdvYr2XqaDaEUcVyIp-YEOi/view?usp=sharing

@mashalahmad
Copy link

can u tell me the name of software you are using to create the annotation?

@maxpaynestory
Copy link

Labelme

https://github.com/wkentaro/labelme

@mashalahmad
Copy link

one last question is it making mask labels? and have you try it with mask rcnn?

@maxpaynestory
Copy link

No it is not mask rcnn. It is a normal way of annotating objects inside a dataset.

I haven't tried mask rcnn.

@Anishauday
Copy link

If the same set of annotations works for SSD Mobilenet/ FastRCNNResnet50 then why is there issue with MaskRCNN?
In my case training happens for mobilenet/fastrcnn but not for MaskRCNN....

@jaeyounkim jaeyounkim added models:research:odapi ODAPI and removed models:research models that come under research directory labels Jun 25, 2021
@Nileshhampiholi
Copy link

Hello, I have the same issue with mask rcnn. I am using labelme to generate masks and converting json files from label me format to coco format json file. And then using the coco format json file to generate tf records.

I get the same error message.

Has any one resolved it?

@Im-JimmyHu
Copy link

the same error,the the error happen in maskrcnn train while no problem in other model like ssd,I use the labelimg the picture to xml,to csv,to record,
my xml_csv like this

import os, sys

import glob

import pandas as pd

import xml.etree.ElementTree as ET

def xml_to_csv(_path, _out_file):

    xml_list = []

    for xml_file in glob.glob(_path + '/*.xml'):

        tree = ET.parse(xml_file)

        root = tree.getroot()

        for member in root.findall('object'):

            value = (root.find('filename').text,

                     int(root.find('size')[0].text),

                     int(root.find('size')[1].text),

                     member[0].text,

                     int(member[4][0].text),

                     int(member[4][1].text),

                     int(member[4][2].text),

                     int(member[4][3].text)

                     )

            xml_list.append(value)

    column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax']

    xml_df = pd.DataFrame(xml_list, columns=column_name)

    xml_df.to_csv(_out_file, index=None)

    print('Successfully converted xml to csv.')

 

if __name__ == '__main__':

    # convert
    xml_to_csv(sys.argv[1], sys.argv[2])

csv_tfrecord.py like this

import os
import io
import pandas as pd
import tensorflow.compat.v1 as tf
 
from PIL import Image
from object_detection.utils import dataset_util
 
flags = tf.app.flags
flags.DEFINE_string('csv_input', '', 'Path to the CSV input')
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
flags.DEFINE_string('image_path','image/train','Path to the image_dir input')
FLAGS = flags.FLAGS
 
def class_text_to_int(row_label):
    if row_label == 'Jam':
        return 1
    if row_label == 'Target':
        return 2
    if row_label == 'Clutter':
        return 3

def create_tf_example(row):
    #full_path = os.path.join(os.getcwd(), 'images', '{}'.format(row['filename']))
    full_path = os.path.join(os.path.dirname(FLAGS.csv_input), FLAGS.image_path, '{}'.format(row['filename']))
    with tf.gfile.GFile(full_path, 'rb') as fid:
        encoded_jpg = fid.read()
    encoded_jpg_io = io.BytesIO(encoded_jpg)
    image = Image.open(encoded_jpg_io)
    width, height = image.size
 
    filename = row['filename'].encode('utf8')
    image_format = b'jpg'
    xmins = [row['xmin'] / width]
    xmaxs = [row['xmax'] / width]
    ymins = [row['ymin'] / height]
    ymaxs = [row['ymax'] / height]
    classes_text = [row['class'].encode('utf8')]
    classes = [class_text_to_int(row['class'])]
 
    tf_example = tf.train.Example(features=tf.train.Features(feature={
        'image/height': dataset_util.int64_feature(height),
        'image/width': dataset_util.int64_feature(width),
        'image/filename': dataset_util.bytes_feature(filename),
        'image/source_id': dataset_util.bytes_feature(filename),
        'image/encoded': dataset_util.bytes_feature(encoded_jpg),
        'image/format': dataset_util.bytes_feature(image_format),
        'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
        'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
        'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
        'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
        'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
        'image/object/class/label': dataset_util.int64_list_feature(classes),
    }))
    return tf_example
 
def main(_):
    writer = tf.python_io.TFRecordWriter(FLAGS.output_path)
    examples = pd.read_csv(FLAGS.csv_input)
    for index, row in examples.iterrows():
        tf_example = create_tf_example(row)
        writer.write(tf_example.SerializeToString())
    writer.close()
 
if __name__ == '__main__':
    tf.app.run()

I also view the record by the scrpit below:

"""
view_records.py:
Consume and display data from a tfrecord file: pulls image and bounding boxes for display
so you can make sure things look reasonabloe, e.g., after augmentation.
Hit 'n' for 'next' image, or 'esc' to quit.
Part of tensorflow-view repo: https://github.com/EricThomson/tfrecord-view
"""

import cv2
import numpy as np
import tensorflow.compat.v1 as tf
tf.enable_eager_execution()
import warnings
warnings.filterwarnings('ignore', category = FutureWarning)  #tf 1.14 and np 1.17 are clashing: temporary solution

def cv_bbox(image, bbox, color = (255, 255, 255), line_width = 2):
    """
    use opencv to add bbox to an image
    assumes bbox is in standard form x1 y1 x2 y2
    """

    cv2.rectangle(image, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color, line_width)
    return


def parse_record(data_record):
    """
    parse the data record from a tfrecord file, typically pulled from an iterator,
    in this case a one_shot_iterator created from the dataset.
    """
    feature = {'image/encoded': tf.FixedLenFeature([], tf.string),
               'image/object/class/label': tf.VarLenFeature(tf.int64),
               'image/object/bbox/xmin': tf.VarLenFeature(tf.float32),
               'image/object/bbox/ymin': tf.VarLenFeature(tf.float32),
               'image/object/bbox/xmax': tf.VarLenFeature(tf.float32),
               'image/object/bbox/ymax': tf.VarLenFeature(tf.float32),
               'image/filename': tf.FixedLenFeature([], tf.string)
               }
    return tf.parse_single_example(data_record, feature)


def view_records(file_path, class_labels, stride = 1, verbose = 1):
    """
    peek at the data using opencv and tensorflow tools.
    Inputs:
        file_path: path to tfrecord file (usually has 'record' extension)
        class_labels: dictionary of labels with name:number pairs (start with 1)
        stride (default 1): how many records to jump (you might have thousands so skip a few)
        verbose (default 1): display text output if 1, display nothing except images otherwise.
    Usage:
    Within the image window, enter 'n' for next image, 'esc' to stop seeing images.
    """
    dataset = tf.data.TFRecordDataset([file_path])
    record_iterator = dataset.make_one_shot_iterator()
    num_records = dataset.reduce(np.int64(0), lambda x, _: x + 1).numpy()

    if verbose:
        print(f"\nGoing through {num_records} records with a stride of {stride}.")
        print("Enter 'n' to bring up next image in record.\n")
    for im_ind in range(num_records):

        #Parse and process example

        parsed_example = parse_record(record_iterator.get_next())
        if im_ind % stride != 0:
            continue

        fname = parsed_example['image/filename'].numpy()
        encoded_image = parsed_example['image/encoded']
        image_np = tf.image.decode_image(encoded_image, channels=3).numpy()

        labels =  tf.sparse_tensor_to_dense(parsed_example['image/object/class/label'], default_value=0).numpy()
        x1norm =  tf.sparse_tensor_to_dense( parsed_example['image/object/bbox/xmin'], default_value=0).numpy()
        x2norm =  tf.sparse_tensor_to_dense( parsed_example['image/object/bbox/xmax'], default_value=0).numpy()
        y1norm =  tf.sparse_tensor_to_dense( parsed_example['image/object/bbox/ymin'], default_value=0).numpy()
        y2norm =  tf.sparse_tensor_to_dense( parsed_example['image/object/bbox/ymax'], default_value=0).numpy()

        num_bboxes = len(labels)

        #% Process and display image
        height, width = image_np[:, :, 1].shape
        image_copy = image_np.copy()
        image_rgb = cv2.cvtColor(image_copy, cv2.COLOR_BGR2RGB)

        if num_bboxes > 0:
            x1 = np.int64(x1norm*width)
            x2 = np.int64(x2norm*width)
            y1 = np.int64(y1norm*height)
            y2 = np.int64(y2norm*height)
            for bbox_ind in range(num_bboxes):
                    bbox = (x1[bbox_ind], y1[bbox_ind], x2[bbox_ind], y2[bbox_ind])
                    label_name = list(class_labels.keys())[list(class_labels.values()).index(labels[bbox_ind])]
                    label_position = (bbox[0] + 5, bbox[1] + 20)
                    cv_bbox(image_rgb, bbox, color = (250, 250, 150), line_width = 2)
                    cv2.putText(image_rgb,
                                label_name,
                                label_position,
                                cv2.FONT_HERSHEY_SIMPLEX,
                                1, (10, 10, 255), 2); #scale, color, thickness

        if verbose:
            print(f"\nImage {im_ind}")
            print(f"    {fname}")
            print(f"    Height/width: {height, width}")
            print(f"    Num bboxes: {num_bboxes}")
        cv2.imshow("bb data", image_rgb)
        k = cv2.waitKey()
        if k == 27:
            break
        elif k == ord('n'):
            continue
    cv2.destroyAllWindows()
    if verbose:
        print("\n\ntfrecord-view: done going throug the data.")


if __name__ == '__main__':
    class_labels =  {"Jam" : 1, "Target": 2,"Clutter": 3 }
    #Make the following using voc_to_tfr.py
    data_path = r"train.record"

    verbose = 1
    stride = 1
    view_records(data_path, class_labels, stride = stride, verbose = verbose)

that look like no problem,and no problem in othet object detect model.
have somebody solved the problem?

@Im-JimmyHu
Copy link

one last question is it making mask labels? and have you try it with mask rcnn?

have you sloved it, i face the same error in mask_rcnn

@aino-gautam
Copy link

Same problem here. Has anyone found a solution yet. It seems there is a compatibility issue in loading the pre-trained checkpoint.

Traceback (most recent call last): File "/Users/_dga/ml-git/tf-ark/Tensorflow/workspace/training_demo/model_main_tf2.py", line 126, in <module> tf.compat.v1.app.run() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/platform/app.py", line 36, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/absl/app.py", line 308, in run _run_main(main, args) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/Users/_dga/ml-git/tf-ark/Tensorflow/workspace/training_demo/model_main_tf2.py", line 117, in main model_lib_v2.train_loop( File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/object_detection/model_lib_v2.py", line 605, in train_loop load_fine_tune_checkpoint( File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/object_detection/model_lib_v2.py", line 401, in load_fine_tune_checkpoint _ensure_model_is_built(model, input_dataset, unpad_groundtruth_tensors) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/object_detection/model_lib_v2.py", line 161, in _ensure_model_is_built features, labels = iter(input_dataset).next() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 260, in next return self.__next__() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 264, in __next__ return self.get_next() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 325, in get_next return self._get_next_no_partial_batch_handling(name) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 361, in _get_next_no_partial_batch_handling replicas.extend(self._iterators[i].get_next_as_list(new_name)) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/distribute/input_lib.py", line 1427, in get_next_as_list return self._format_data_list_with_options(self._iterator.get_next()) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/data/ops/multi_device_iterator_ops.py", line 553, in get_next result.append(self._device_iterators[i].get_next()) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 867, in get_next return self._next_internal() File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 777, in _next_internal ret = gen_dataset_ops.iterator_get_next( File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 3028, in iterator_get_next _ops.raise_from_not_ok_status(e, name) File "/Users/_dga/anaconda3/envs/tf-ark/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6656, in raise_from_not_ok_status raise core._status_to_exception(e) from None # pylint: disable=protected-access tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__IteratorGetNext_output_types_18_device_/job:localhost/replica:0/task:0/device:GPU:0}} indices[0] = 0 is not in [0, 0) [[{{node GatherV2_7}}]] [[MultiDeviceIteratorGetNextFromShard]] [[RemoteCall]] [Op:IteratorGetNext] name:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests