object_detector.create operands could not be broadcast together

[res0nat0r]

Hi all,

I'm trying to follow the object detection tutorial on some image data I have, I have
an SFrame with my images labeled with annotations. explore() shows the bounding
boxes properly setup, but I'm getting the error below when trying to create a model.

Any suggestions would be great, or if this is a bug I'd be happy to help debug further.

In [20]: images
Out[20]:
Columns:
	image	Image
	annotations	list
	truth	Image

Rows: 23

Data:
+------------------------+-------------------------------+
|         image          |          annotations          |
+------------------------+-------------------------------+
| Height: 100 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 104 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 100 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 100 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 100 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 100 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 104 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 104 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 104 Width: 100 | [{'coordinates': {'y': 50,... |
| Height: 104 Width: 100 | [{'coordinates': {'y': 50,... |
+------------------------+-------------------------------+
+------------------------+
|         truth          |
+------------------------+
| Height: 100 Width: 100 |
| Height: 104 Width: 100 |
| Height: 100 Width: 100 |
| Height: 100 Width: 100 |
| Height: 100 Width: 100 |
| Height: 100 Width: 100 |
| Height: 104 Width: 100 |
| Height: 104 Width: 100 |
| Height: 104 Width: 100 |
| Height: 104 Width: 100 |
+------------------------+
[23 rows x 3 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.

In [21]: images[0]
Out[21]:
{'annotations': [{'coordinates': {'height': 95, 'width': 95, 'x': 50, 'y': 50},
   'label': 'elixir'}],
 'image': Height: 100px
 Width: 100px
 Channels: 4,
 'truth': Height: 100px
 Width: 100px
 Channels: 4}

Model failure:

model = tc.object_detector.create(images,feature='image', annotations='annotations')

...

-> 1428         return fn_array(lhs, rhs)
   1429     else:
   1430         raise TypeError('type %s not supported' % str(type(rhs)))

/Users/hovland/.pyenv/versions/2.7.10/envs/turicreate/lib/python2.7/site-packages/mxnet/ndarray.pyc in broadcast_mul(lhs, rhs, out, name, **kwargs)

/Users/hovland/.pyenv/versions/2.7.10/envs/turicreate/lib/python2.7/site-packages/mxnet/_ctypes/ndarray.pyc in _imperative_invoke(handle, ndargs, keys, vals, out)
     87         ctypes.c_int(len(keys)),
     88         c_array(ctypes.c_char_p, [c_str(key) for key in keys]),
---> 89         c_array(ctypes.c_char_p, [c_str(str(val)) for val in vals])))
     90
     91     if original_output is not None:

/Users/hovland/.pyenv/versions/2.7.10/envs/turicreate/lib/python2.7/site-packages/mxnet/base.pyc in check_call(ret)
    127     """
    128     if ret != 0:
--> 129         raise MXNetError(py_str(_LIB.MXGetLastError()))
    130
    131 if sys.version_info[0] < 3:

MXNetError: [13:55:21] src/operator/tensor/./elemwise_binary_broadcast_op.h:66: Check failed: l == 1 || r == 1 operands could not be broadcast together with shapes (416,416,4) (1,1,3)

Stack trace returned 7 entries:
[bt] (0) 0   libmxnet.so                         0x0000000116600ad8 _ZN4dmlc15LogMessageFatalD2Ev + 40
[bt] (1) 1   libmxnet.so                         0x000000011685c990 _ZN5mxnet2op20BinaryBroadcastShapeERKN4nnvm9NodeAttrsEPNSt3__16vectorINS1_6TShapeENS5_9allocatorIS7_EEEESB_ + 1392
[bt] (2) 2   libmxnet.so                         0x0000000116cf8fcc _Z12SetShapeTypePKN4nnvm2OpERKNS_9NodeAttrsERKN5mxnet7ContextERKNSt3__16vectorINS6_7NDArrayENSA_9allocatorISC_EEEEPSF_ + 1260
[bt] (3) 3   libmxnet.so                         0x0000000116cff650 _Z20ImperativeInvokeImplRKN5mxnet7ContextERKN4nnvm9NodeAttrsEPNSt3__16vectorINS_7NDArrayENS7_9allocatorIS9_EEEESD_ + 688
[bt] (4) 4   libmxnet.so                         0x0000000116d007f1 MXImperativeInvoke + 433
[bt] (5) 5   _ctypes.so                          0x000000010f0b7627 ffi_call_unix64 + 79
[bt] (6) 6   ???                                 0x00007ffee1653470 0x0 + 140732679926896


In [23]:

[gustavla]

Thanks for trying out the object detector! I really appreciate the detailed report of this bug, since it made it very easy for me to understand the issue. The problem is that we are not handling images with 4 channels (RGB + opacity channel) correctly.

I will work on a fix and in the meantime let's discuss work-arounds to get your model working. If you want to train the model to leverage the opacity channel, then the current version will unfortunately not support that. However, if you get rid of the opacity channel then this error message will disappear. You can either manually convert your images before loading them, or you can do it through Turi Create:

def drop_alpha(image):
    return tc.Image(_image_data=image.pixel_data[..., :3].tobytes(),
                    _width=image.width,
                    _height=image.height,
                    _channels=3,
                    _format_enum=2,
                    _image_data_size=image.width * image.height * 3)

sf['image_rgb'] = sf['image'].apply(drop_alpha)

Removing the opacity channel of an image can be done in several ways. The question is, do we replace the transparent background with white or black, or something else? The above method is blunt and simply drops the opacity channel. This will work if most of your images do not actually use the extra channel. However, if they do use it, it may lead to visual artifacts so I encourage you to check the new images manually with explore(). Let me know how it works out!

[res0nat0r]

Hi,

Thanks for the feedback. I can look into manually fixing the images as a workaround. I've tried the fix above but am also getting the below error:

#!/usr/bin/env python

import turicreate as tc

def drop_alpha(image):
  return tc.Image(_image_data=image.pixel_data[..., :3].tobytes(),
      _width=image.width,
      _height=image.height,
      _channels=3,
      _format_enum=2,
      _image_data_size=image.width * image.height * 3)

images = tc.SFrame('clash.sframe')

images['rgb'] = images['image'].apply(drop_alpha)

$ ./convert.py
Traceback (most recent call last):
  File "./convert.py", line 16, in <module>
    images['rgb'] = images['image'].apply(drop_alpha)
  File "/Users/hovland/.pyenv/versions/turicreate/lib/python2.7/site-packages/turicreate/data_structures/sarray.py", line 1881, in apply
    return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_na, seed))
  File "/Users/hovland/.pyenv/versions/turicreate/lib/python2.7/site-packages/turicreate/cython/context.py", line 49, in __exit__
    raise exc_type(exc_value)
RuntimeError: Runtime Exception. Exception in python callback function evaluation:
UnpicklingError('NEWOBJ class argument has NULL tp_new',):
Traceback (most recent call last):
  File "turicreate/cython/cy_pylambda_workers.pyx", line 398, in turicreate.cython.cy_pylambda_workers._init_lambda
  File "turicreate/cython/cy_pylambda_workers.pyx", line 138, in turicreate.cython.cy_pylambda_workers.lambda_evaluator.__init__
UnpicklingError: NEWOBJ class argument has NULL tp_new

[gustavla]

That looks like another bug unfortunately (thanks for finding these!). I can't seem to reproduce it though. There is probably something different between your images and my test images. Perhaps it will give us a more informative error message (or none at all) if you run this as a slower for loop:

rgbs = []
for row in images:
    rgb = drop_alpha(row['image'])
    rgbs.append(rgb)
images['rgb'] = rgbs

Sorry again for this and thanks for your patience! Any additional information that can help us track this down is greatly appreciated.

[res0nat0r]

Interestingly the code above seems to work fine. I'm able to run it and then modify my model create statement to point to the rgb column and I see the model generation output status updates now.

Here is just a print of the images['rgb'] column after running the above:

['Height: 100 Width: 100', 'Height: 104 Width: 100', 'Height: 100 Width: 100', 'Height: 100 Width: 100', 'Height: 100 Width: 100', 'Height: 100 Width: 100', 'Height: 104 Width: 100', 'Height: 104 Width: 100', 'Height: 104 Width: 100', 'Height: 104 Width: 100', 'Height: 104 Width: 100', 'Height: 104 Width: 100', 'Height: 100 Width: 100', 'Height: 100 Width: 100', 'Height: 104 Width: 100', 'Height: 104 Width: 100', 'Height: 104 Width: 100', 'Height: 100 Width: 100', 'Height: 104 Width: 100', 'Height: 100 Width: 100', 'Height: 100 Width: 100', 'Height: 100 Width: 100', 'Height: 100 Width: 100']```

[gustavla]

Great! Does that mean the detector is training now?

I would love to investigate the second bug some more. If you have time, it would be great if you could check if sf['image'][:1].apply(drop_alpha) also throws an error. If it doesn't, then try to adjust [:1] to find a subset that reproduces the error.

If you find an SFrame with a single image that breaks it, I would love to get more info on that particular image (or ideally the file itself, but I understand completely if you do not want to share it). Thanks again for finding these things, @res0nat0r!

[res0nat0r]

Yep, I was able to get a model to generate now. :)

Actually here is the entire SFrame if you'd like to download it and look. It is just a handful
of iOS game images and is only ~1.2MB so I have no problem sharing it.

Feel free to download below, let me know if you need anything else. Thanks for the help.

https://stefhen.s3.amazonaws.com/clash.sframe.tar

[gustavla]

That is super helpful, thank you so much @res0nat0r. I am now able to reproduce the error. I have filed a separate issue #124 so as not to confuse it with the original issue of not being able to use the detector on RGBA images.

[res0nat0r]

Awesome thanks!

[UHemarajU]

Hi res0nat0r,

I am trying to generate model for object detection. While generating model i am also getting same error as you got.
i.e. MXNetError: [13:55:21] src/operator/tensor/./elemwise_binary_broadcast_op.h:66: Check failed: l == 1 || r == 1 operands could not be broadcast together with shapes (416,416,4) (1,1,3)

i tried to add the code you and gustavla discussed above but i am still getting the error. Following is my convert.py code. Can you help me how to solve this issue.

import sys
import pandas as pd
import math
import os

if len(sys.argv) < 2:
quit("Require input file")

fileIn = sys.argv[1]

objectLabel = 'Object'

csv = pd.read_csv(fileIn, names = ["image", "id", "label", "xMin", "xMax", "yMin", "yMax", "annotations"],
dtype={"annotations": str})

for i, item in csv.iterrows():
height = csv.iat[i, 6] - csv.iat[i, 5]
width = csv.iat[i, 4] - csv.iat[i, 3]
x = csv.iat[i, 3] + math.floor(width / 2)
y = csv.iat[i, 5] + math.floor(height / 2)

props = {'label': objectLabel, 'type': 'rectangle'}
props['coordinates'] = {'height': height, 'width': width, 'x': x, 'y': y}
csv.iat[i, 7] = [props]

csv.to_csv('annotations.csv')

Thank you.