HDF5 error at end of run

[braymp]

Run the attached pipeline from source(c0a74b1) to reproduce:

Traceback (most recent call last):
  File "D:\Trunk\CellProfiler\cellprofiler\utilities\thread_excepthook.py", line 27, in run_with_except_hook
    run_old(*args, **kw)
  File "C:\Python27\lib\threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "D:\Trunk\CellProfiler\cellprofiler\analysis.py", line 448, in interface
    self.copy_recieved_measurements(recd_measurements, measurements, image_numbers)
  File "D:\Trunk\CellProfiler\cellprofiler\analysis.py", line 552, in copy_recieved_measurements
    = recd_measurements[o, feature, image_numbers]
  File "D:\Trunk\CellProfiler\cellprofiler\measurements.py", line 338, in __setitem__
    self.add_measurement(key[0], key[1], value, image_set_number=key[2])
  File "D:\Trunk\CellProfiler\cellprofiler\measurements.py", line 790, in add_measurement
    self.hdf5_dict[object_name, feature_name, image_set_number] = data
  File "D:\Trunk\CellProfiler\cellprofiler\utilities\hdf5_dict.py", line 529, in __setitem__
    dataset = feature_group.create_dataset('data', **kwds)
  File "C:\Python27\lib\site-packages\h5py\_hl\group.py", line 71, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
  File "C:\Python27\lib\site-packages\h5py\_hl\dataset.py", line 89, in make_new_dset
    tid = h5t.py_create(dtype, logical=1)
  File "h5t.pyx", line 1361, in h5py.h5t.py_create (h5py\h5t.c:12754)
  File "h5t.pyx", line 1433, in h5py.h5t.py_create (h5py\h5t.c:12601)
TypeError: Object dtype dtype('O') has no native HDF5 equivalent

Interestingly, this forum user noticed the same thing: http://cellprofiler.org/forum/viewtopic.php?f=18&t=3510&p=10262#p10263

Attachments:
http://cellprofiler.org/issues/uploaded/OIGqk_/issue.cppipe

[dlogan]

The Forum user, who, by the way, has this friendly comment:

Kudos by the way on the Metadata and Groups modules--they're great.

is also having trouble getting the '-l' switch to not crash with CP 2.1. BatchProfiler seems to handle it, though?
Note that the user is trying to run from the Amazon cloud.

[LeeKamentsky]

They still get the error after updating to include the fix to #1011?

[braymp]

I'll let them know about the fix.

[dlogan]

Another user is having trouble with '-l'. Starting here the user 'cscott', on this Forum thread. This time it's on Linux though so it may be other issues related to the OS @ljosa .

[LeeKamentsky]

I think I may have the fix for this. It only takes the code path that causes the bug if you use the -l switch. I posted to the forum to tell the user how to apply the patch that fixes the release for them.

[ljosa]

Does this user have his own build, or should I make a new RPM?

[LeeKamentsky]

It sure looks like he is using our build. I would not make an RPM until he
confirms that the patch works (or you can replicate the problem with the
build previous to the fix and fix it with the fix).

On Tue, Feb 18, 2014 at 2:13 PM, Vebjorn Ljosa notifications@github.comwrote:

Does this user have his own build, or should I make a new RPM?

Reply to this email directly or view it on GitHubhttps://github.com//issues/995#issuecomment-35421387
.

[dlogan]

The 'cscott' Linux cluster admin says "Previously I have installed cellProfiler using the yum depot on centos 6.4". Because of the issue, they are trying to use the Makefile without success -- so if all looks good after the fix, then I'd assume they'd appreciate a new rpm.

[braymp]

Assigned to @dlogan for verification of the -l issue; the original HDF5 issue seems to be fixed. Scheduled for patch release.

[pontus]

One of our users had a similar issue (although she received a

Traceback (most recent call last):
  File "/usr/cellprofiler/src/CellProfiler/cellprofiler/utilities/hdf5_dict.py", line 293, in __init__
    maxshape = (None, ))
  File "/usr/cellprofiler/lib/python2.7/site-packages/h5py/_hl/group.py", line 94, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
  File "/usr/cellprofiler/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 76, in make_new_dset
    tid = h5t.py_create(dtype, logical=1)
  File "h5t.pyx", line 1379, in h5py.h5t.py_create (h5py/h5t.c:12683)
  File "h5t.pyx", line 1451, in h5py.h5t.py_create (h5py/h5t.c:12533)
TypeError: Object dtype dtype('object') has no native HDF5 equivalent

instead of dtype('O') referenced in the original report. The corresponding fix for this issue fixes these issues as well.

[LeeKamentsky]

I found this problem in Fiocruz. There is another path where this happens. I made it work by looping and setting the dataset by hand

check for dataset.dtype.kind = "O"
create dataset empty with correct length
set each element by hand.