Check failed: l == 1 || r == 1 operands could not be broadcast together with shapes [416,416,4] [1,1,3]

[henrytkirk]

Hi,
I'm having the following issue when I run this command:
model = tc.object_detector.create(data, feature='image', annotations='annotations', max_iterations=5)

Downloading https://docs-assets.developer.apple.com/turicreate/models/darknet.params
Download completed: /var/folders/k5/x_hplcg9495b5sqj_rvxym540000gn/T/model_cache/darknet.params
Setting 'batch_size' to 16
Using GPU to create model (NVIDIA GeForce GT 750M)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/anaconda2/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/anaconda2/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/anaconda2/lib/python2.7/site-packages/turicreate/toolkits/object_detector/object_detector.py", line 482, in sframe_worker
    for batch in loader:
  File "/anaconda2/lib/python2.7/site-packages/turicreate/toolkits/object_detector/_sframe_loader.py", line 333, in next
    return self._next()
  File "/anaconda2/lib/python2.7/site-packages/turicreate/toolkits/object_detector/_sframe_loader.py", line 405, in _next
    image, bbox = aug(image, bbox)
  File "/anaconda2/lib/python2.7/site-packages/mxnet/image/detection.py", line 86, in __call__
    src = self.augmenter(src)
  File "/anaconda2/lib/python2.7/site-packages/mxnet/image/image.py", line 659, in __call__
    src = t(src)
  File "/anaconda2/lib/python2.7/site-packages/mxnet/image/image.py", line 698, in __call__
    gray = src * self.coef
  File "/anaconda2/lib/python2.7/site-packages/mxnet/ndarray/ndarray.py", line 235, in __mul__
    return multiply(self, other)
  File "/anaconda2/lib/python2.7/site-packages/mxnet/ndarray/ndarray.py", line 2566, in multiply
    None)
  File "/anaconda2/lib/python2.7/site-packages/mxnet/ndarray/ndarray.py", line 2379, in _ufunc_helper
    return fn_array(lhs, rhs)
  File "<string>", line 46, in broadcast_mul
  File "/anaconda2/lib/python2.7/site-packages/mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke
    ctypes.byref(out_stypes)))
  File "/anaconda2/lib/python2.7/site-packages/mxnet/base.py", line 146, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
MXNetError: [09:11:17] src/operator/tensor/./elemwise_binary_broadcast_op.h:67: Check failed: l == 1 || r == 1 operands could not be broadcast together with shapes [416,416,4] [1,1,3]

Stack trace returned 8 entries:
[bt] (0) 0   libmxnet.so                         0x0000000a0da942cf libmxnet.so + 58063
[bt] (1) 1   libmxnet.so                         0x0000000a0da9406f libmxnet.so + 57455
[bt] (2) 2   libmxnet.so                         0x0000000a0de3a7f8 libmxnet.so + 3885048
[bt] (3) 3   libmxnet.so                         0x0000000a0ea9b427 MXNDListFree + 420935
[bt] (4) 4   libmxnet.so                         0x0000000a0ea9a029 MXNDListFree + 415817
[bt] (5) 5   libmxnet.so                         0x0000000a0ea1c277 MXCustomFunctionRecord + 16119
[bt] (6) 6   libmxnet.so                         0x0000000a0ea1d444 MXImperativeInvokeEx + 164
[bt] (7) 7   libffi.6.dylib                      0x00000001031e7884 ffi_call_unix64 + 76

I have OSX Mojave Beta, MacBook Pro (Retina, 15-inch, Late 2013). Sometimes the processing works, sometimes not. Can you point me to what might be wrong? Did I install anything incorrectly? I'm using the mxnet v1.1.0 as mentioned in another post.

Thanks for the help.

[srikris]

That's not expected. Can you share a sample of your dataset and annotations with us for us to be able to reproduce it?

[henrytkirk]

Sure, where can I securely send them to?

[henrytkirk]

I think it's something to do with the JPEG images. I converted them to PNG's and I did not get the error.

Anything special with how JPEGs need to be created/compressed? Color profiles?

[concertman]

Hello, I'm getting something similar to this using 5.0b3

I can package up my env and jupyter notebook to send over. Just let me know where to send it.

From the command model = tc.object_detector.create(train, "annotations", max_iterations=5000) I'm getting...

Using 'image' as feature column
Setting 'batch_size' to 32
Using CPU to create model
---------------------------------------------------------------------------
MXNetError                                Traceback (most recent call last)
<ipython-input-6-d1599f5a5b12> in <module>()
      1 train, test = data.random_split(0.8)
----> 2 model = tc.object_detector.create(train, "annotations", max_iterations=5000)

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/turicreate/toolkits/object_detector/object_detector.pyc in create(dataset, annotations, feature, model, classes, batch_size, max_iterations, verbose, **kwargs)
    570         trainer = _mx.gluon.Trainer(net.collect_params(), 'sgd', options)
    571 
--> 572         for batch in loader:
    573             data = _mx.gluon.utils.split_and_load(batch.data[0], ctx_list=ctx, batch_axis=0)
    574             label = _mx.gluon.utils.split_and_load(batch.label[0], ctx_list=ctx, batch_axis=0)

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/turicreate/toolkits/object_detector/_sframe_loader.pyc in next(self)
    331 
    332     def next(self):
--> 333         return self._next()
    334 
    335     @property

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/turicreate/toolkits/object_detector/_sframe_loader.pyc in _next(self)
    403             for aug in self.augmentations:
    404                 try:
--> 405                     image, bbox = aug(image, bbox)
    406                 except ValueError:
    407                     # It is extremely rare, but mxnet can fail for some reason.

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/image/detection.pyc in __call__(self, src, label)
     84     def __call__(self, src, label):
     85         """Augmenter implementation body"""
---> 86         src = self.augmenter(src)
     87         return (src, label)
     88 

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/image/image.pyc in __call__(self, src)
    657         random.shuffle(self.ts)
    658         for t in self.ts:
--> 659             src = t(src)
    660         return src
    661 

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/image/image.pyc in __call__(self, src)
    696         """Augmenter body"""
    697         alpha = 1.0 + random.uniform(-self.contrast, self.contrast)
--> 698         gray = src * self.coef
    699         gray = (3.0 * (1.0 - alpha) / gray.size) * nd.sum(gray)
    700         src *= alpha

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/ndarray/ndarray.pyc in __mul__(self, other)
    233     def __mul__(self, other):
    234         """x.__mul__(y) <=> x*y <=> mx.nd.multiply(x, y) """
--> 235         return multiply(self, other)
    236 
    237     def __neg__(self):

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/ndarray/ndarray.pyc in multiply(lhs, rhs)
   2564         operator.mul,
   2565         _internal._mul_scalar,
-> 2566         None)
   2567     # pylint: enable= no-member, protected-access
   2568 

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/ndarray/ndarray.pyc in _ufunc_helper(lhs, rhs, fn_array, fn_scalar, lfn_scalar, rfn_scalar)
   2377         return lfn_scalar(lhs, float(rhs))
   2378     elif isinstance(rhs, NDArray):
-> 2379         return fn_array(lhs, rhs)
   2380     else:
   2381         raise TypeError('type %s not supported' % str(type(rhs)))

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/ndarray/register.pyc in broadcast_mul(lhs, rhs, out, name, **kwargs)

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/_ctypes/ndarray.pyc in _imperative_invoke(handle, ndargs, keys, vals, out)
     90         c_str_array(keys),
     91         c_str_array([str(s) for s in vals]),
---> 92         ctypes.byref(out_stypes)))
     93 
     94     if original_output is not None:

/Users/editing/Desktop/aux-phase5/Prototypes/object-recognition/lib/python2.7/site-packages/mxnet/base.pyc in check_call(ret)
    144     """
    145     if ret != 0:
--> 146         raise MXNetError(py_str(_LIB.MXGetLastError()))
    147 
    148 

MXNetError: [08:52:55] src/operator/tensor/./elemwise_binary_broadcast_op.h:67: Check failed: l == 1 || r == 1 operands could not be broadcast together with shapes [416,416,4] [1,1,3]

Stack trace returned 8 entries:
[bt] (0) 0   libmxnet.so                         0x00000001176ec7cf libmxnet.so + 59343
[bt] (1) 1   libmxnet.so                         0x00000001176ec56f libmxnet.so + 58735
[bt] (2) 2   libmxnet.so                         0x0000000117a92cf8 libmxnet.so + 3886328
[bt] (3) 3   libmxnet.so                         0x00000001186f6e37 MXNDListFree + 434551
[bt] (4) 4   libmxnet.so                         0x00000001186f5a39 MXNDListFree + 429433
[bt] (5) 5   libmxnet.so                         0x0000000118674757 MXCustomFunctionRecord + 16119
[bt] (6) 6   libmxnet.so                         0x0000000118675924 MXImperativeInvokeEx + 164
[bt] (7) 7   libffi.dylib                        0x00007fff6584ef64 ffi_call_unix64 + 76

[znation]

@henrytkirk You can send your JPEGs that consistently repro this issue to the email address in my profile. We can investigate from there, thanks!

[concertman]

@znation Can I send you my set? I've got a lot of images and it's been really hard trying to figure out which ones might be causing the issue. My zipped set is about 125MB so email won't work too well for that.

[concertman]

@znation Apologies for the comment spam but I was able to get a smaller replication set and I have emailed it to you. Please let me know if there is more information you need.

[asinugobi]

@henrytkirk @concertman ran into the same problem today. Converting the images from RGBA (4 channels) to RGB (3 channels) solved my issue. Hope this helps: #114

[praeclarum]

@znation I am also running into this issue and am not sure why #114 was closed. It repros with pretty much every set of images I throw at it.

[yousifKashef]

I stopped getting this bug when I downgraded by 2015 iMac to high Sierra. This bug must have something to do with the GPU support in Mojave. I hope this helps.

[concertman]

@yousifKashef I was getting this on High Sierra as well so the Mojave thing might be a red herring.