"TypeError: data type not understood" error in Official Object Detection Tutorial

[mburakbozbey]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [+ ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [+] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [+ ] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/utils/visualization_utils.py

2. Describe the bug

In object_detection.utils.ops, "reframe_box_masks_to_image_masks" function returns TypeError

3. Steps to reproduce

Run official object detection colab tutorial notebook, instance segmentation part fails to produce segmentation mask when using show_inference function
https://colab.research.google.com/github/tensorflow/models/blob/master/research/object_detection/colab_tutorials/object_detection_tutorial.ipynb

4. Expected behavior

Image with segmentation mask was available in 2-3 days ago.
from object_detection.utils import visualization_utils as vis_util
vis_util.visualize_boxes_and_labels_on_image_array() function gives type error when image with segmentation mask is requested.

5. Additional context

"""
TypeError Traceback (most recent call last)
in ()
----> 1 show_inference(masking_model, "/content/Screenshot_3.jpg")

2 frames
/usr/local/lib/python3.6/dist-packages/object_detection/utils/ops.py in reframe_box_masks_to_image_masks(box_masks, boxes, image_height, image_width, resize_method)
823 def reframe_box_masks_to_image_masks_default():
824 """The default function when there are more than 0 box masks."""
--> 825 def transform_boxes_relative_to_boxes(boxes, reference_boxes):
826 boxes = tf.reshape(boxes, [-1, 2, 2])
827 min_corner = tf.expand_dims(reference_boxes[:, 0:2], 1)

TypeError: data type not understood
""""

6. System information

• Official colab tutorial notebook of object detection API
  https://colab.research.google.com/github/tensorflow/models/blob/master/research/object_detection/colab_tutorials/object_detection_tutorial.ipynb

[mburakbozbey]

Problem solved after changing:

detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(output_dict['detection_masks'], output_dict['detection_boxes'], image.shape[0], image.shape[1])   

to:

detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(tf.convert_to_tensor(output_dict['detection_masks']), tf.convert_to_tensor(output_dict['detection_boxes'])[tf.newaxis,...], image.shape[0], image.shape[1])   

Detection part, run_inference_for_single_image function in: https://colab.research.google.com/github/tensorflow/models/blob/master/research/object_detection/colab_tutorials/object_detection_tutorial.ipynb

PR: #8814

[magedhelmy1]

Replace

detection_masks_reframed=utils_ops.reframe_box_masks_to_image_masks(output_dict['detection_masks'],output_dict['detection_boxes'],image.shape[0],image.shape[1])

with

detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(tf.convert_to_tensor(output_dict['detection_masks']),tf.convert_to_tensor(output_dict['detection_boxes']), image.shape[0], image.shape[1])   

[hichemfelouat]

I tried to use it with mask rcnn but it did not work?

InvalidArgumentError: Incompatible shapes: [1,2,2] vs. [1,1,1,100,4] [Op:Sub]

[arcivi]

Below code works for me with TensorFlow 2.3

'if 'detection_masks' in detections:

we need to convert np.arrays to tensors

detection_masks = tf.convert_to_tensor(detections['detection_masks'][0])
detection_boxes = tf.convert_to_tensor(detections['detection_boxes'][0])

Reframe the the bbox mask to the image size.

detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(detection_masks, detection_boxes,image_np.shape[0], image_np.shape[1])
detection_masks_reframed = tf.cast(detection_masks_reframed > 0.5,tf.uint8)
detections['detection_masks_reframed'] = detection_masks_reframed.numpy()`

[hichemfelouat]

I solved the problem of mask-RCNN in FT2 Object detection API :

def run_inference_for_single_image(image_path):

print("Running inference for : ",image_path)
image_np = load_image_into_numpy_array(image_path)

# The input needs to be a tensor, convert it using `tf.convert_to_tensor`.
input_tensor = tf.convert_to_tensor(image_np)
# The model expects a batch of images, so add an axis with `tf.newaxis`.
input_tensor = input_tensor[tf.newaxis, ...]

# input_tensor = np.expand_dims(image_np, 0)
detections = my_detection_model(input_tensor)

# All outputs are batches tensors.
# Convert to numpy arrays, and take index [0] to remove the batch dimension.
# We're only interested in the first num_detections.
num_detections = int(detections.pop("num_detections"))

# detections = {key: value[0, :num_detections].numpy()
#             for key, value in detections.items()}

import itertools
detections = dict(itertools.islice(detections.items(), num_detections))

detections["num_detections"] = num_detections

# detection_classes should be ints.
# detections["detection_classes"] = detections["detection_classes"].astype(np.int64)

image_np_with_detections = image_np.copy()

# Handle models with masks:
if "detection_masks" in detections:
    # Reframe the the bbox mask to the image size.
    detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(
          detections["detection_masks"][0], detections["detection_boxes"][0],
           image_np.shape[0], image_np.shape[1])      
    detection_masks_reframed = tf.cast(detection_masks_reframed > 0.5,
                                   tf.uint8)
    detections["detection_masks_reframed"] = detection_masks_reframed.numpy()

boxes = np.asarray(detections["detection_boxes"][0])
classes = np.asarray(detections["detection_classes"][0]).astype(np.int64)
scores = np.asarray(detections["detection_scores"][0])
mask = np.asarray(detections["detection_masks_reframed"])

# Visualizing the results
viz_utils.visualize_boxes_and_labels_on_image_array(
    image_np_with_detections,
    boxes,
    classes,
    scores,
    category_index,
    instance_masks=mask,
    use_normalized_coordinates=True,
    line_thickness=3)

cv2_imshow(image_np_with_detections)
display(Image.fromarray(image_np_with_detections))
print("Done")

[jamerico]

That worked for me @ravikyram. I am using TF 2.1.3

[ravikyram]

@mburakbozbey

Any updates on the issue, please. Thanks!

[sniper0110]

@hichemfelouat's solution worked for me as well on Tensorflow 2.3.1!