None in dataset makes converter fail

[talonchandler]

The following conversion:
iohub convert -i /hpc/projects/compmicro/rawdata/mantis/2022_07_08\ zebrafish\ imaging/fish2_withGFP_1/ -o /hpc/projects/compmicro/sandbox/Talon/test-converter/
failed after ~3 minutes with

Traceback

  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/talon.chandler/iohub/iohub/cli/cli.py", line 84, in convert
    converter.run()
  File "/home/talon.chandler/iohub/iohub/convert.py", line 340, in run
    zarr_img[coord_reorder[1:]] = img_raw
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/zarr/core.py", line 1391, in __setitem__
    self.set_basic_selection(pure_selection, value, fields=fields)
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/zarr/core.py", line 1486, in set_basic_selection
    return self._set_basic_selection_nd(selection, value, fields=fields)
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/zarr/core.py", line 1790, in _set_basic_selection_nd
    self._set_selection(indexer, value, fields=fields)
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/zarr/core.py", line 1818, in _set_selection
    check_array_shape('value', value, sel_shape)
  File "/home/talon.chandler/.conda/envs/recorder-test/lib/python3.9/site-packages/zarr/util.py", line 551, in check_array_shape
    raise ValueError('parameter {!r}: expected array with shape {!r}, got {!r}'
ValueError: parameter 'value': expected array with shape (1024, 1024), got ()

I suspect this is similar to recOrder #320 with an incomplete pycromanager dataset that contains None values?

The (soon deprecated) recOrder converter also failed, so I am opening a new issue and not blocking the iohub converter from merging.

[ziw-liu]

The ND-TIFF reader will return None from get_image() if the image does not exist:
https://github.com/czbiohub/iohub/blob/eeaad0861bf92e53a8b931596fd4002ddd7d5860/iohub/ndtiff.py#L111-L117

Is it the desired behavior to write zeros instead of failing?

@ieivanov and @edyoshikun may have more context about the specific dataset.

[talonchandler]

Nice find. I think writing zeros (and possibly printing a warning?) is a good solution.

[ieivanov]

That makes sense. The pycromanager converter was written for "standard" datasets, i.e. assuming there aren't missing indices. Zarr datasets can be sparse, correct? We can either write zeros, or leave these indices empty also, if that doesn't cause downstream problems

[ziw-liu]

We can either write zeros, or leave these indices empty also

These are equivalent if the array is chunked along the incomplete axis and the fill value is left at default (which is 0).