RuntimeError: Groundtruth tensor boxes has not been provided #9738

azzubair01 · 2021-02-19T03:26:28Z

Prerequisites

Please answer the following questions for yourself before submitting an issue.

[/ ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
[/ ] I am reporting the issue to the correct repository. (Model Garden official or research directory)
[ /] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_detection

2. Describe the bug

Since 18/2/2021, i got this error when I am training my object detection model using faster_rcnn_resnet50_v1_640x640_coco17_tpu-8.tar.gz

3. Steps to reproduce

I follow the steps done by this blog : https://blog.tensorflow.org/2021/01/custom-object-detection-in-browser.html. However, since 18/2/2021, I got the runtime error when beginning model training.

4. Expected behavior

I expect that I can train my model successfully
2021-02-15 08:21:37.118183: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-15 08:21:39.552141: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-15 08:21:39.553005: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-15 08:21:39.582898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:39.583503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2021-02-15 08:21:39.583541: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-15 08:21:39.585143: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-02-15 08:21:39.585220: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2021-02-15 08:21:39.587085: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-15 08:21:39.587452: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-15 08:21:39.589528: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-15 08:21:39.590427: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-02-15 08:21:39.594231: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2021-02-15 08:21:39.594360: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:39.594974: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:39.595513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-15 08:21:39.595952: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-15 08:21:39.596069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:39.596612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2021-02-15 08:21:39.596641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-15 08:21:39.596685: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-02-15 08:21:39.596710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2021-02-15 08:21:39.596732: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-15 08:21:39.596752: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-15 08:21:39.596779: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-15 08:21:39.596800: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-02-15 08:21:39.596820: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2021-02-15 08:21:39.596891: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:39.597453: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:39.597954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-15 08:21:39.597994: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-15 08:21:40.117665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-15 08:21:40.117717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-02-15 08:21:40.117732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-02-15 08:21:40.117947: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:40.118568: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:40.119151: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-15 08:21:40.119655: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-02-15 08:21:40.119702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13994 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0215 08:21:40.121417 139690246182784 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: 7500
I0215 08:21:40.125437 139690246182784 config_util.py:552] Maybe overwriting train_steps: 7500
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0215 08:21:40.125594 139690246182784 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/model_lib_v2.py:531: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W0215 08:21:40.189840 139690246182784 deprecation.py:339] From /usr/local/lib/python3.6/dist-packages/object_detection/model_lib_v2.py:531: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
INFO:tensorflow:Reading unweighted datasets: ['/content/dataset/train.record']
I0215 08:21:40.201230 139690246182784 dataset_builder.py:163] Reading unweighted datasets: ['/content/dataset/train.record']
INFO:tensorflow:Reading record datasets for input file: ['/content/dataset/train.record']
I0215 08:21:40.201407 139690246182784 dataset_builder.py:80] Reading record datasets for input file: ['/content/dataset/train.record']
INFO:tensorflow:Number of filenames to read: 1
I0215 08:21:40.201493 139690246182784 dataset_builder.py:81] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0215 08:21:40.201568 139690246182784 dataset_builder.py:88] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:105: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
W0215 08:21:40.211044 139690246182784 deprecation.py:339] From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:105: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:237: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map() W0215 08:21:40.235985 139690246182784 deprecation.py:339] From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:237: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map()
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead.
W0215 08:21:46.300727 139690246182784 deprecation.py:339] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/inputs.py:282: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0215 08:21:48.916735 139690246182784 deprecation.py:339] From /usr/local/lib/python3.6/dist-packages/object_detection/inputs.py:282: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2021-02-15 08:21:51.409052: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-02-15 08:21:51.438530: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2199995000 Hz
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py:434: UserWarning: tf.keras.backend.set_learning_phase is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the training argument of the __call__ method of your layer or model.
warnings.warn('tf.keras.backend.set_learning_phase is deprecated and '
INFO:tensorflow:depth of additional conv before box predictor: 0
I0215 08:21:57.088716 139686635575040 convolutional_keras_box_predictor.py:154] depth of additional conv before box predictor: 0
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/utils/model_util.py:57: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.
W0215 08:22:03.989516 139686635575040 deprecation.py:339] From /usr/local/lib/python3.6/dist-packages/object_detection/utils/model_util.py:57: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

W0215 08:22:08.459299 139686635575040 deprecation.py:339] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

2021-02-15 08:22:18.668204: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-02-15 08:22:20.100653: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._groundtruth_lists
W0215 08:22:27.445933 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._groundtruth_lists
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv
W0215 08:22:27.446708 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor
W0215 08:22:27.446813 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._maxpool_layer
W0215 08:22:27.446888 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._maxpool_layer
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor
W0215 08:22:27.446954 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._batched_prediction_tensor_names
W0215 08:22:27.447016 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._batched_prediction_tensor_names
WARNING:tensorflow:Unresolved object in checkpoint: (root).model.endpoints
W0215 08:22:27.447077 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model.endpoints
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer_with_weights-0
W0215 08:22:27.447151 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer_with_weights-0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer-1
W0215 08:22:27.447220 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer-1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer-2
W0215 08:22:27.447281 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer-2
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads
W0215 08:22:27.447340 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._sorted_head_names
W0215 08:22:27.447400 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._sorted_head_names
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._shared_nets
W0215 08:22:27.447459 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._shared_nets
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head
W0215 08:22:27.447518 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head
W0215 08:22:27.447578 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._third_stage_heads
W0215 08:22:27.447637 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._third_stage_heads
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer_with_weights-0._inbound_nodes
W0215 08:22:27.447877 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer_with_weights-0._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer_with_weights-0.kernel
W0215 08:22:27.448033 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer_with_weights-0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer_with_weights-0.bias
W0215 08:22:27.448127 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer_with_weights-0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer-1._inbound_nodes
W0215 08:22:27.448197 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer-1._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer-2._inbound_nodes
W0215 08:22:27.448281 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor_first_conv.layer-2._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings
W0215 08:22:27.448348 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background
W0215 08:22:27.448421 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._shared_nets.0
W0215 08:22:27.448482 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._shared_nets.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers
W0215 08:22:27.448541 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers
W0215 08:22:27.448601 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0
W0215 08:22:27.448707 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0
W0215 08:22:27.448767 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.0
W0215 08:22:27.448831 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.1
W0215 08:22:27.448891 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.2
W0215 08:22:27.448951 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.2
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.0
W0215 08:22:27.449010 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.1
W0215 08:22:27.449069 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.2
W0215 08:22:27.449140 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.2
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0._box_encoder_layers
W0215 08:22:27.449199 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers
W0215 08:22:27.449264 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.1.kernel
W0215 08:22:27.449331 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.1.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.1.bias
W0215 08:22:27.449391 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._box_prediction_head._box_encoder_layers.1.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.1.kernel
W0215 08:22:27.449450 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.1.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.1.bias
W0215 08:22:27.449509 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._mask_rcnn_box_predictor._class_prediction_head._class_predictor_layers.1.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0
W0215 08:22:27.449568 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0
W0215 08:22:27.449627 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0.kernel
W0215 08:22:27.449687 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0.bias
W0215 08:22:27.449747 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0.kernel
W0215 08:22:27.449807 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0.bias
W0215 08:22:27.449869 139690246182784 util.py:161] Unresolved object in checkpoint: (root).model._first_stage_box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0.bias
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
W0215 08:22:27.449932 139690246182784 util.py:169] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.818649 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.819849 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.821680 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.822472 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.824238 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.824966 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.827125 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.827858 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.829257 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0215 08:22:27.829981 139690246182784 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py:605: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
W0215 08:22:35.116770 139686635575040 deprecation.py:537] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py:605: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
INFO:tensorflow:Step 100 per-step time 0.428s loss=1.241
I0215 08:23:33.098545 139690246182784 model_lib_v2.py:659] Step 100 per-step time 0.428s loss=1.241
INFO:tensorflow:Step 200 per-step time 0.439s loss=1.431
I0215 08:24:17.019295 139690246182784 model_lib_v2.py:659] Step 200 per-step time 0.439s loss=1.431
INFO:tensorflow:Step 300 per-step time 0.437s loss=1.627
I0215 08:25:01.111969 139690246182784 model_lib_v2.py:659] Step 300 per-step time 0.437s loss=1.627
INFO:tensorflow:Step 400 per-step time 0.451s loss=0.982
I0215 08:25:45.355300 139690246182784 model_lib_v2.py:659] Step 400 per-step time 0.451s loss=0.982
INFO:tensorflow:Step 500 per-step time 0.447s loss=0.983
I0215 08:26:29.711425 139690246182784 model_lib_v2.py:659] Step 500 per-step time 0.447s loss=0.983
INFO:tensorflow:Step 600 per-step time 0.439s loss=0.883
I0215 08:27:14.057378 139690246182784 model_lib_v2.py:659] Step 600 per-step time 0.439s loss=0.883
INFO:tensorflow:Step 700 per-step time 0.455s loss=1.194
I0215 08:27:58.709164 139690246182784 model_lib_v2.py:659] Step 700 per-step time 0.455s loss=1.194
INFO:tensorflow:Step 800 per-step time 0.442s loss=0.848
I0215 08:28:43.209079 139690246182784 model_lib_v2.py:659] Step 800 per-step time 0.442s loss=0.848
INFO:tensorflow:Step 900 per-step time 0.435s loss=0.548
I0215 08:29:27.878699 139690246182784 model_lib_v2.py:659] Step 900 per-step time 0.435s loss=0.548
INFO:tensorflow:Step 1000 per-step time 0.445s loss=0.551
I0215 08:30:12.481833 139690246182784 model_lib_v2.py:659] Step 1000 per-step time 0.445s loss=0.551
INFO:tensorflow:Step 1100 per-step time 0.448s loss=0.719
I0215 08:30:57.579932 139690246182784 model_lib_v2.py:659] Step 1100 per-step time 0.448s loss=0.719
INFO:tensorflow:Step 1200 per-step time 0.452s loss=0.836
I0215 08:31:42.285844 139690246182784 model_lib_v2.py:659] Step 1200 per-step time 0.452s loss=0.836
INFO:tensorflow:Step 1300 per-step time 0.438s loss=0.565
I0215 08:32:27.014727 139690246182784 model_lib_v2.py:659] Step 1300 per-step time 0.438s loss=0.565
INFO:tensorflow:Step 1400 per-step time 0.454s loss=0.470
I0215 08:33:11.437570 139690246182784 model_lib_v2.py:659] Step 1400 per-step time 0.454s loss=0.470
INFO:tensorflow:Step 1500 per-step time 0.453s loss=0.524
I0215 08:33:56.149427 139690246182784 model_lib_v2.py:659] Step 1500 per-step time 0.453s loss=0.524
INFO:tensorflow:Step 1600 per-step time 0.476s loss=0.818
I0215 08:34:40.956308 139690246182784 model_lib_v2.py:659] Step 1600 per-step time 0.476s loss=0.818
INFO:tensorflow:Step 1700 per-step time 0.438s loss=1.120
I0215 08:35:25.636198 139690246182784 model_lib_v2.py:659] Step 1700 per-step time 0.438s loss=1.120
INFO:tensorflow:Step 1800 per-step time 0.442s loss=0.421
I0215 08:36:10.108073 139690246182784 model_lib_v2.py:659] Step 1800 per-step time 0.442s loss=0.421
INFO:tensorflow:Step 1900 per-step time 0.452s loss=1.001
I0215 08:36:54.813591 139690246182784 model_lib_v2.py:659] Step 1900 per-step time 0.452s loss=1.001
INFO:tensorflow:Step 2000 per-step time 0.436s loss=0.814
I0215 08:37:39.525192 139690246182784 model_lib_v2.py:659] Step 2000 per-step time 0.436s loss=0.814
INFO:tensorflow:Step 2100 per-step time 0.437s loss=0.596
I0215 08:38:24.556893 139690246182784 model_lib_v2.py:659] Step 2100 per-step time 0.437s loss=0.596
INFO:tensorflow:Step 2200 per-step time 0.452s loss=0.791
I0215 08:39:08.973331 139690246182784 model_lib_v2.py:659] Step 2200 per-step time 0.452s loss=0.791
INFO:tensorflow:Step 2300 per-step time 0.446s loss=0.435
I0215 08:39:53.345946 139690246182784 model_lib_v2.py:659] Step 2300 per-step time 0.446s loss=0.435
INFO:tensorflow:Step 2400 per-step time 0.455s loss=0.452
I0215 08:40:37.624874 139690246182784 model_lib_v2.py:659] Step 2400 per-step time 0.455s loss=0.452
INFO:tensorflow:Step 2500 per-step time 0.443s loss=1.045
I0215 08:41:22.092401 139690246182784 model_lib_v2.py:659] Step 2500 per-step time 0.443s loss=1.045
INFO:tensorflow:Step 2600 per-step time 0.448s loss=0.527
I0215 08:42:06.632725 139690246182784 model_lib_v2.py:659] Step 2600 per-step time 0.448s loss=0.527
INFO:tensorflow:Step 2700 per-step time 0.446s loss=1.170
I0215 08:42:51.225370 139690246182784 model_lib_v2.py:659] Step 2700 per-step time 0.446s loss=1.170
INFO:tensorflow:Step 2800 per-step time 0.443s loss=0.338
I0215 08:43:35.560664 139690246182784 model_lib_v2.py:659] Step 2800 per-step time 0.443s loss=0.338
INFO:tensorflow:Step 2900 per-step time 0.442s loss=0.562
I0215 08:44:20.041495 139690246182784 model_lib_v2.py:659] Step 2900 per-step time 0.442s loss=0.562
INFO:tensorflow:Step 3000 per-step time 0.435s loss=0.696
I0215 08:45:04.365055 139690246182784 model_lib_v2.py:659] Step 3000 per-step time 0.435s loss=0.696
INFO:tensorflow:Step 3100 per-step time 0.441s loss=0.261
I0215 08:45:49.667834 139690246182784 model_lib_v2.py:659] Step 3100 per-step time 0.441s loss=0.261
INFO:tensorflow:Step 3200 per-step time 0.440s loss=0.306
I0215 08:46:34.367719 139690246182784 model_lib_v2.py:659] Step 3200 per-step time 0.440s loss=0.306
INFO:tensorflow:Step 3300 per-step time 0.442s loss=0.325
I0215 08:47:18.731243 139690246182784 model_lib_v2.py:659] Step 3300 per-step time 0.442s loss=0.325
INFO:tensorflow:Step 3400 per-step time 0.442s loss=0.765
I0215 08:48:03.319004 139690246182784 model_lib_v2.py:659] Step 3400 per-step time 0.442s loss=0.765
INFO:tensorflow:Step 3500 per-step time 0.454s loss=0.830
I0215 08:48:47.534738 139690246182784 model_lib_v2.py:659] Step 3500 per-step time 0.454s loss=0.830
INFO:tensorflow:Step 3600 per-step time 0.439s loss=0.253
I0215 08:49:32.038966 139690246182784 model_lib_v2.py:659] Step 3600 per-step time 0.439s loss=0.253
INFO:tensorflow:Step 3700 per-step time 0.443s loss=0.233
I0215 08:50:16.291156 139690246182784 model_lib_v2.py:659] Step 3700 per-step time 0.443s loss=0.233
INFO:tensorflow:Step 3800 per-step time 0.442s loss=0.249
I0215 08:51:00.660153 139690246182784 model_lib_v2.py:659] Step 3800 per-step time 0.442s loss=0.249
INFO:tensorflow:Step 3900 per-step time 0.441s loss=0.383
I0215 08:51:45.064616 139690246182784 model_lib_v2.py:659] Step 3900 per-step time 0.441s loss=0.383
INFO:tensorflow:Step 4000 per-step time 0.437s loss=0.290
I0215 08:52:29.419125 139690246182784 model_lib_v2.py:659] Step 4000 per-step time 0.437s loss=0.290
INFO:tensorflow:Step 4100 per-step time 0.444s loss=0.617
I0215 08:53:14.411015 139690246182784 model_lib_v2.py:659] Step 4100 per-step time 0.444s loss=0.617
INFO:tensorflow:Step 4200 per-step time 0.458s loss=0.455
I0215 08:53:58.611593 139690246182784 model_lib_v2.py:659] Step 4200 per-step time 0.458s loss=0.455
INFO:tensorflow:Step 4300 per-step time 0.458s loss=0.154
I0215 08:54:43.082449 139690246182784 model_lib_v2.py:659] Step 4300 per-step time 0.458s loss=0.154
INFO:tensorflow:Step 4400 per-step time 0.435s loss=0.389
I0215 08:55:27.612740 139690246182784 model_lib_v2.py:659] Step 4400 per-step time 0.435s loss=0.389
INFO:tensorflow:Step 4500 per-step time 0.435s loss=0.364
I0215 08:56:11.912635 139690246182784 model_lib_v2.py:659] Step 4500 per-step time 0.435s loss=0.364
INFO:tensorflow:Step 4600 per-step time 0.440s loss=0.507
I0215 08:56:56.278250 139690246182784 model_lib_v2.py:659] Step 4600 per-step time 0.440s loss=0.507
INFO:tensorflow:Step 4700 per-step time 0.447s loss=0.356
I0215 08:57:40.885407 139690246182784 model_lib_v2.py:659] Step 4700 per-step time 0.447s loss=0.356
INFO:tensorflow:Step 4800 per-step time 0.439s loss=0.198
I0215 08:58:25.201893 139690246182784 model_lib_v2.py:659] Step 4800 per-step time 0.439s loss=0.198
INFO:tensorflow:Step 4900 per-step time 0.458s loss=0.194
I0215 08:59:09.507776 139690246182784 model_lib_v2.py:659] Step 4900 per-step time 0.458s loss=0.194
INFO:tensorflow:Step 5000 per-step time 0.436s loss=0.244
I0215 08:59:53.909693 139690246182784 model_lib_v2.py:659] Step 5000 per-step time 0.436s loss=0.244
INFO:tensorflow:Step 5100 per-step time 0.433s loss=0.395
I0215 09:00:39.148197 139690246182784 model_lib_v2.py:659] Step 5100 per-step time 0.433s loss=0.395
INFO:tensorflow:Step 5200 per-step time 0.446s loss=0.095
I0215 09:01:23.480309 139690246182784 model_lib_v2.py:659] Step 5200 per-step time 0.446s loss=0.095
INFO:tensorflow:Step 5300 per-step time 0.449s loss=0.243
I0215 09:02:07.736174 139690246182784 model_lib_v2.py:659] Step 5300 per-step time 0.449s loss=0.243
INFO:tensorflow:Step 5400 per-step time 0.458s loss=0.409
I0215 09:02:52.198406 139690246182784 model_lib_v2.py:659] Step 5400 per-step time 0.458s loss=0.409
INFO:tensorflow:Step 5500 per-step time 0.436s loss=0.540
I0215 09:03:36.533442 139690246182784 model_lib_v2.py:659] Step 5500 per-step time 0.436s loss=0.540
INFO:tensorflow:Step 5600 per-step time 0.444s loss=0.132
I0215 09:04:20.785590 139690246182784 model_lib_v2.py:659] Step 5600 per-step time 0.444s loss=0.132
INFO:tensorflow:Step 5700 per-step time 0.441s loss=0.249
I0215 09:05:05.162519 139690246182784 model_lib_v2.py:659] Step 5700 per-step time 0.441s loss=0.249
INFO:tensorflow:Step 5800 per-step time 0.440s loss=0.255
I0215 09:05:49.580168 139690246182784 model_lib_v2.py:659] Step 5800 per-step time 0.440s loss=0.255
INFO:tensorflow:Step 5900 per-step time 0.457s loss=0.192
I0215 09:06:33.944940 139690246182784 model_lib_v2.py:659] Step 5900 per-step time 0.457s loss=0.192
INFO:tensorflow:Step 6000 per-step time 0.461s loss=0.282
I0215 09:07:18.423235 139690246182784 model_lib_v2.py:659] Step 6000 per-step time 0.461s loss=0.282
INFO:tensorflow:Step 6100 per-step time 0.439s loss=0.276
I0215 09:08:03.347150 139690246182784 model_lib_v2.py:659] Step 6100 per-step time 0.439s loss=0.276
INFO:tensorflow:Step 6200 per-step time 0.452s loss=0.330
I0215 09:08:47.703232 139690246182784 model_lib_v2.py:659] Step 6200 per-step time 0.452s loss=0.330
INFO:tensorflow:Step 6300 per-step time 0.442s loss=0.259
I0215 09:09:31.859435 139690246182784 model_lib_v2.py:659] Step 6300 per-step time 0.442s loss=0.259
INFO:tensorflow:Step 6400 per-step time 0.473s loss=0.102
I0215 09:10:16.377394 139690246182784 model_lib_v2.py:659] Step 6400 per-step time 0.473s loss=0.102
INFO:tensorflow:Step 6500 per-step time 0.455s loss=0.163
I0215 09:11:00.698708 139690246182784 model_lib_v2.py:659] Step 6500 per-step time 0.455s loss=0.163
INFO:tensorflow:Step 6600 per-step time 0.451s loss=0.227
I0215 09:11:45.162684 139690246182784 model_lib_v2.py:659] Step 6600 per-step time 0.451s loss=0.227
INFO:tensorflow:Step 6700 per-step time 0.436s loss=0.134
I0215 09:12:29.465272 139690246182784 model_lib_v2.py:659] Step 6700 per-step time 0.436s loss=0.134
INFO:tensorflow:Step 6800 per-step time 0.441s loss=0.229
I0215 09:13:13.592250 139690246182784 model_lib_v2.py:659] Step 6800 per-step time 0.441s loss=0.229
INFO:tensorflow:Step 6900 per-step time 0.434s loss=0.120
I0215 09:13:58.131253 139690246182784 model_lib_v2.py:659] Step 6900 per-step time 0.434s loss=0.120
INFO:tensorflow:Step 7000 per-step time 0.459s loss=0.177
I0215 09:14:42.605149 139690246182784 model_lib_v2.py:659] Step 7000 per-step time 0.459s loss=0.177
INFO:tensorflow:Step 7100 per-step time 0.439s loss=0.232
I0215 09:15:27.663410 139690246182784 model_lib_v2.py:659] Step 7100 per-step time 0.439s loss=0.232
INFO:tensorflow:Step 7200 per-step time 0.476s loss=0.142
I0215 09:16:12.025357 139690246182784 model_lib_v2.py:659] Step 7200 per-step time 0.476s loss=0.142
INFO:tensorflow:Step 7300 per-step time 0.444s loss=0.080
I0215 09:16:56.625972 139690246182784 model_lib_v2.py:659] Step 7300 per-step time 0.444s loss=0.080
INFO:tensorflow:Step 7400 per-step time 0.450s loss=0.175
I0215 09:17:40.988873 139690246182784 model_lib_v2.py:659] Step 7400 per-step time 0.450s loss=0.175
INFO:tensorflow:Step 7500 per-step time 0.443s loss=0.131
I0215 09:18:25.365577 139690246182784 model_lib_v2.py:659] Step 7500 per-step time 0.443s loss=0.131

5. Additional context

2021-02-19 03:01:40.807306: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-19 03:01:43.340863: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-19 03:01:43.341734: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-19 03:01:43.374163: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.374804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2021-02-19 03:01:43.374843: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-19 03:01:43.376354: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-02-19 03:01:43.376456: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2021-02-19 03:01:43.378162: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-19 03:01:43.378504: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-19 03:01:43.380057: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-19 03:01:43.381148: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-02-19 03:01:43.384804: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2021-02-19 03:01:43.384944: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.385564: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.386084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-19 03:01:43.386535: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-19 03:01:43.386665: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.387226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2021-02-19 03:01:43.387255: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-19 03:01:43.387290: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-02-19 03:01:43.387318: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2021-02-19 03:01:43.387347: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-19 03:01:43.387371: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-19 03:01:43.387397: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-19 03:01:43.387435: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-02-19 03:01:43.387462: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2021-02-19 03:01:43.387535: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.388131: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.388672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-19 03:01:43.388719: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-19 03:01:43.913912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-19 03:01:43.913962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-02-19 03:01:43.913978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-02-19 03:01:43.914199: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.914902: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.915537: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-19 03:01:43.916066: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-02-19 03:01:43.916117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13994 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0219 03:01:43.917932 139986759083904 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: 7500
I0219 03:01:43.922563 139986759083904 config_util.py:552] Maybe overwriting train_steps: 7500
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0219 03:01:43.922743 139986759083904 config_util.py:552] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.023690 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.025247 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.028743 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.029536 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.073875 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.077457 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.095414 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.096279 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.097664 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0219 03:01:44.098390 139986759083904 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
2021-02-19 03:01:45.480024: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2021-02-19 03:01:46.659904: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
INFO:tensorflow:depth of additional conv before box predictor: 0
I0219 03:01:47.375232 139986759083904 convolutional_keras_box_predictor.py:154] depth of additional conv before box predictor: 0
Traceback (most recent call last):
File "/content/models/research/object_detection/model_main_tf2.py", line 113, in
tf.compat.v1.app.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/content/models/research/object_detection/model_main_tf2.py", line 110, in main
record_summaries=FLAGS.record_summaries)
File "/usr/local/lib/python3.6/dist-packages/object_detection/model_lib_v2.py", line 523, in train_loop
dummy_prediction_dict = detection_model.predict(dummy_image, dummy_shapes)
File "/usr/local/lib/python3.6/dist-packages/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 830, in predict
**side_inputs))
File "/usr/local/lib/python3.6/dist-packages/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 999, in _predict_second_stage
image_shape, true_image_shapes)
File "/usr/local/lib/python3.6/dist-packages/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 729, in _proposal_postprocess
anchors, image_shape_2d, true_image_shapes)
File "/usr/local/lib/python3.6/dist-packages/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 1754, in _postprocess_rpn
) = self._format_groundtruth_data(image_shapes)
File "/usr/local/lib/python3.6/dist-packages/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 1891, in _format_groundtruth_data
self.groundtruth_lists(fields.BoxListFields.boxes))
File "/usr/local/lib/python3.6/dist-packages/object_detection/core/model.py", line 118, in groundtruth_lists
field))
RuntimeError: Groundtruth tensor boxes has not been provided

6. System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab
Mobile device name if the issue happens on a mobile device:
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version: 3.6.9
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 11.2
GPU model and memory:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 41C P8 9W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

tf.__version__ = 2.4.1

--> tf.git_version = v2.4.1-0-g85c8b2a817f

The text was updated successfully, but these errors were encountered:

azzubair01 · 2021-02-19T08:15:03Z

Custom Object Detection using Tensorflow 2:

https://github.com/azzubair01/Invoice-Segmentation

dangiiii · 2021-02-19T13:22:42Z

Did it work for you before? In the same configuration? I noticed that I don't get the error on my local machine with CPU-only, but as soon as I use the server (same tensorflow version, same training data, same labelmaps, same config files, same everything) and GPUs this error will pop up.
Tested different models, same result.

azzubair01 · 2021-02-19T13:25:54Z

Yes, it worked for me with the same config, and version before 18/2/2021. After that, when I want to run again, I got that error. However, this error only happen when I use FRCNN. When I used other models such as EfficientDet, MobileNet, everything seems fine.

dangiiii · 2021-02-19T13:47:45Z

Did you update anything? I didn't update anything yet these errors occur which is what I don't really get.

farecomhalima · 2021-02-19T13:54:46Z

Same problem and i didn't change anything

dangiiii · 2021-02-19T14:03:36Z

Just tried copying an older version of the models folder combined with an older version of model_main_tf2.py. Didn't lead to anything (except one bug concerning multiple graphics cards vanished somehow with the older version(!?)) on this issue.

I just can't understand how this could occur without changing anything? Is there any "live" stuff that tensorflow checks online when running? Cause otherwise there shouldn't be any errors using the old files

Update: using the older configuration I ran into "NotImplementedError: Cannot convert a symbolic Tensor (cond_2/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
".
This lead me to tensorflow/tensorflow#47242 (comment) , where people report the same strange behavior of these bugs on previously working configurations (also during the last 48 hours). I would be really interested in knowing if, how and why this stuff is connected and how these errors can occur without updating anything

dangiiii · 2021-02-19T15:15:56Z

I got it working again.
I downloaded the most current clone of tensorflow and then first took the advice from #9735 (comment) and replaced the mentioned file. Then I ran into this tensorflow/tensorflow#47242 error, which seems to be related to the numpy version (https://stackoverflow.com/questions/58479556/notimplementederror-cannot-convert-a-symbolic-tensor-2nd-target0-to-a-numpy).
I downgraded my numpy to 1.19.2 which lead to problems with pycocotools
("File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
"). I fixed this by doing steps 1-4 in open-mmlab/mmdetection3d#301 (comment). Maybe installing an older version of pycocotools would solve that error as well, hope using this installation won't lead to further problems down the road.

But at least I was finally able to train again.

CUDA 11.0
cuDNN 8.0.3
tensorflow 2.4.1
numpy 1.19.2
mmpycocotools 12.0.3
python 3.8.5

google-ml-butler · 2021-02-20T09:30:49Z

Are you satisfied with the resolution of your issue?
Yes
No

servercalap · 2021-02-20T19:32:15Z

I get the same error when I try to start training with faster rcnn(Object Detection Api).

faster_rcnn_resnet152_v1_1024x1024_coco17_tpu-8

-Additional context
RuntimeError: Groundtruth tensor boxes has not been provided

--System Conf.
Google Colab
Runtime- TPU

tanishkthomas · 2021-02-21T19:15:19Z

I get the same error when I try to start training with faster rcnn(Object Detection Api).

faster_rcnn_resnet152_v1_1024x1024_coco17_tpu-8

-Additional context
RuntimeError: Groundtruth tensor boxes has not been provided

--System Conf.
Google Colab
Runtime- TPU

Same Issue

RITIK-12 · 2021-02-23T13:08:34Z

Yes, it worked for me with the same config, and version before 18/2/2021. After that, when I want to run again, I got that error. However, this error only happen when I use FRCNN. When I used other models such as EfficientDet, MobileNet, everything seems fine.

same error!

azzubair01 · 2021-02-23T13:29:45Z

Specifically for Faster-RCNN, you guys may git clone previous commits of tensorflow to solve this problem. Use the model_lib_v2.py script before 18/2/2021. In short you may use this code:

%%bash

#clone tensorflow object detection api & use working commit
git clone https://github.com/tensorflow/models.git
cd models
git fetch origin 8a06433
git reset --hard FETCH_HEAD

RITIK-12 · 2021-02-24T07:42:03Z

Specifically for Faster-RCNN, you guys may git clone previous commits of tensorflow to solve this problem. Use the model_lib_v2.py script before 18/2/2021. In short you may use this code:

%%bash

#clone tensorflow object detection api & use working commit
git clone https://github.com/tensorflow/models.git
cd models
git fetch origin 8a06433
git reset --hard FETCH_HEAD

Thanks, the code threw some errors but I was able get things running with this code :

%%bash
git clone https://github.com/tensorflow/models.git
cd models
git checkout 8a06433

dragonnikkirocks · 2021-03-02T15:41:18Z

Specifically for Faster-RCNN, you guys may git clone previous commits of tensorflow to solve this problem. Use the model_lib_v2.py script before 18/2/2021. In short you may use this code:
%%bash
#clone tensorflow object detection api & use working commit
git clone https://github.com/tensorflow/models.git
cd models
git fetch origin 8a06433
git reset --hard FETCH_HEAD

Thanks, the code threw some errors but I was able get things running with this code :

%%bash
git clone https://github.com/tensorflow/models.git
cd models
git checkout 8a06433

This didnt work for me :/

ghost · 2021-03-05T15:45:59Z

Is this problem solved? This error only happens when I use FRCNN.

awaisbajwaml · 2021-03-09T01:57:54Z

I am having the same problem, I am thinking if there is a possibility that the dataset has a problem? if yes, is there any way we can verify the dataset before start doing the training.? any comment.

Isfet666 · 2021-03-09T03:26:09Z

I get the same error.
Is there any bugfix for this issue?

jaekookang · 2021-03-09T04:59:05Z

Me too. I still get the same error after doing git checkout 8a06433.

GiorgosBetsos · 2021-03-12T09:24:58Z

Specifically for Faster-RCNN, you guys may git clone previous commits of tensorflow to solve this problem. Use the model_lib_v2.py script before 18/2/2021. In short you may use this code:

%%bash

#clone tensorflow object detection api & use working commit
git clone https://github.com/tensorflow/models.git
cd models
git fetch origin 8a06433
git reset --hard FETCH_HEAD

This didn't work for me

meghamgarg · 2021-03-17T04:18:18Z

I get the same error when I try to start training with faster rcnn(Object Detection Api)
faster_rcnn_resnet152_v1_1024x1024_coco17_tpu-8
-Additional context
RuntimeError: Groundtruth tensor boxes has not been provided
--System Conf.
Google Colab
Runtime- TPU

Same Issue

s902888597 · 2021-03-17T05:19:27Z

I tried all the faster rcnn provided on tf2 model zoo. The same error appears. RuntimeError: Groundtruth tensor boxes has not been provided

farecomhalima · 2021-03-17T10:02:20Z

git clone https://github.com/tensorflow/models.git
cd models
git checkout 31e86e8
I try to get an older version before 02/02/2021
It works and i don't have any more the error "RuntimeError: Groundtruth tensor boxes has not been provided"

s902888597 · 2021-03-17T19:09:02Z

git clone https://github.com/tensorflow/models.git
cd models
git checkout 31e86e8
I try to get an older version before 02/02/2021
It works and i don't have any more the error "RuntimeError: Groundtruth tensor boxes has not been provided"

This didn't work for me

abrarum · 2021-03-18T18:25:57Z

Same above mentioned issue still persists.

marooned20 · 2021-03-22T10:55:31Z

The issue still persists.

If it helps other for debugging purposes as it definitely did to me (still on-going), I noticed that on 18th Feb, following change was introduced!
852e098#diff-d6241ffb0103c18b5bea432efe07a10a271943fbf526fd87b50922a28de1cac5

richardkippers · 2021-03-23T07:56:36Z

Make sure to reboot your kernel and re-install TF Object Detection API after using git checkout

christinasunnegardh · 2021-03-23T09:55:22Z

Should this issue really be closed if the solution is to revert to an old commit? 🤔

haridas · 2021-03-23T11:12:23Z

after reverting model_lib_v2.py to the old commit mentioned above and then do the following,

pip uninstall object_detection

# And follow the installation steps mentioned in Readme - 
# https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2.md#python-package-installation
#
cd models/research
# Compile protos.
protoc object_detection/protos/*.proto --python_out=.
# Install TensorFlow Object Detection API.
cp object_detection/packages/tf2/setup.py .
python -m pip install --use-feature=2020-resolver .

wegmatho · 2021-04-05T17:42:42Z

Issue seems to be resolved by a recent bugfix and merge into master. After pulling and reinstall it worked for me (Windows and Linux). Now only more GPU power is left to wish for ;)

# pull newest master
cd models/research
git pull
# reinstall object_detection API (must be in models/research dir)
pip uninstall object_detection
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .
# test installation
python object_detection/builders/model_builder_tf2_test.py

ghost · 2021-04-07T21:05:49Z

This solution works for me. I also don't know whether it will cause potential risks.

#9808 (comment)

servercalap · 2021-04-08T14:25:39Z

This solution for me. The object detection code I modified is in my repository. You can download and run from there.

!git clone --depth 1 https://github.com/servercalap/object_detection.git
!protoc object_detection/protos/*.proto --python_out=.
!cp object_detection/packages/tf2/setup.py .
!python -m pip install .
!rm setup.py

note: google colaboratory
OR

git clone https://github.com/servercalap/object_detection.git

cd models/research

Compile protos.

protoc object_detection/protos/*.proto --python_out=.

Install TensorFlow Object Detection API.

cp object_detection/packages/tf2/setup.py .
python -m pip install --use-feature=2020-resolver .

best regards

EdenBelouadah · 2022-05-10T11:51:45Z

I am facing the same problem with CenterNet. Any solution without reverting the commits? thanks

SkepticRaven · 2022-12-01T21:06:33Z

Don't know if anyone is still running into this, but the following worked for me. Was able to train a segmentation model with CenterNet (mobilenet v2 backbone).
In the project config add the fields (for both train/eval readers):

train_input_reader {
...
  load_instance_masks: true
  mask_type: PNG_MASKS
}

azzubair01 added models:research models that come under research directory type:bug Bug in the code labels Feb 19, 2021

google-ml-butler bot assigned ravikyram Feb 19, 2021

ravikyram assigned tombstone, jch1 and pkulzc and unassigned ravikyram Feb 19, 2021

dangiiii mentioned this issue Feb 19, 2021

ODAPI failed to initiate by running into runtime error. #9735

Open

1 task

azzubair01 closed this as completed Feb 20, 2021

TannerGilbert mentioned this issue Apr 4, 2021

RuntimeError: Groundtruth tensor boxes has not been provided TannerGilbert/Tensorflow-Object-Detection-API-train-custom-Mask-R-CNN-model#8

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RuntimeError: Groundtruth tensor boxes has not been provided #9738

RuntimeError: Groundtruth tensor boxes has not been provided #9738

azzubair01 commented Feb 19, 2021

azzubair01 commented Feb 19, 2021 •

edited

Loading

dangiiii commented Feb 19, 2021 •

edited

Loading

azzubair01 commented Feb 19, 2021 via email •

edited

Loading

dangiiii commented Feb 19, 2021

farecomhalima commented Feb 19, 2021 •

edited

Loading

dangiiii commented Feb 19, 2021 •

edited

Loading

dangiiii commented Feb 19, 2021 •

edited

Loading

google-ml-butler bot commented Feb 20, 2021

servercalap commented Feb 20, 2021

tanishkthomas commented Feb 21, 2021

RITIK-12 commented Feb 23, 2021

azzubair01 commented Feb 23, 2021 •

edited

Loading

RITIK-12 commented Feb 24, 2021

dragonnikkirocks commented Mar 2, 2021

ghost commented Mar 5, 2021

awaisbajwaml commented Mar 9, 2021

Isfet666 commented Mar 9, 2021

jaekookang commented Mar 9, 2021

GiorgosBetsos commented Mar 12, 2021

meghamgarg commented Mar 17, 2021

s902888597 commented Mar 17, 2021

farecomhalima commented Mar 17, 2021

s902888597 commented Mar 17, 2021

abrarum commented Mar 18, 2021

marooned20 commented Mar 22, 2021

richardkippers commented Mar 23, 2021

christinasunnegardh commented Mar 23, 2021

haridas commented Mar 23, 2021

wegmatho commented Apr 5, 2021

ghost commented Apr 7, 2021

servercalap commented Apr 8, 2021

EdenBelouadah commented May 10, 2022

SkepticRaven commented Dec 1, 2022

RuntimeError: Groundtruth tensor boxes has not been provided #9738

RuntimeError: Groundtruth tensor boxes has not been provided #9738

Comments

azzubair01 commented Feb 19, 2021

Prerequisites

1. The entire URL of the file you are using

2. Describe the bug

3. Steps to reproduce

4. Expected behavior

5. Additional context

6. System information

azzubair01 commented Feb 19, 2021 • edited Loading

dangiiii commented Feb 19, 2021 • edited Loading

azzubair01 commented Feb 19, 2021 via email • edited Loading

dangiiii commented Feb 19, 2021

farecomhalima commented Feb 19, 2021 • edited Loading

dangiiii commented Feb 19, 2021 • edited Loading

dangiiii commented Feb 19, 2021 • edited Loading

google-ml-butler bot commented Feb 20, 2021

servercalap commented Feb 20, 2021

tanishkthomas commented Feb 21, 2021

RITIK-12 commented Feb 23, 2021

azzubair01 commented Feb 23, 2021 • edited Loading

RITIK-12 commented Feb 24, 2021

dragonnikkirocks commented Mar 2, 2021

ghost commented Mar 5, 2021

awaisbajwaml commented Mar 9, 2021

Isfet666 commented Mar 9, 2021

jaekookang commented Mar 9, 2021

GiorgosBetsos commented Mar 12, 2021

meghamgarg commented Mar 17, 2021

s902888597 commented Mar 17, 2021

farecomhalima commented Mar 17, 2021

s902888597 commented Mar 17, 2021

abrarum commented Mar 18, 2021

marooned20 commented Mar 22, 2021

richardkippers commented Mar 23, 2021

christinasunnegardh commented Mar 23, 2021

haridas commented Mar 23, 2021

wegmatho commented Apr 5, 2021

ghost commented Apr 7, 2021

servercalap commented Apr 8, 2021

Compile protos.

Install TensorFlow Object Detection API.

EdenBelouadah commented May 10, 2022

SkepticRaven commented Dec 1, 2022

azzubair01 commented Feb 19, 2021 •

edited

Loading

dangiiii commented Feb 19, 2021 •

edited

Loading

azzubair01 commented Feb 19, 2021 via email •

edited

Loading

farecomhalima commented Feb 19, 2021 •

edited

Loading

dangiiii commented Feb 19, 2021 •

edited

Loading

dangiiii commented Feb 19, 2021 •

edited

Loading

azzubair01 commented Feb 23, 2021 •

edited

Loading