DIALS behaviour with incomplete Eiger data sets

[graeme-winter]

If a data collection is aborted with an Eiger the master file will contain references to data files which will never exist i.e. if a 3600 image data set is planned but aborted some way through the master file becomes "broken"

What should the correct behaviour be?

[graeme-winter]

Suggestion from @dagewa is to use image_range= as a workaround - reasonable, will see if this works.

[graeme-winter]

Error is

Traceback (most recent call last):
  File "/Users/graeme/svn/cctbx/build/../modules/dials/command_line/dials_import.py", line 832, in <module>
    halraiser(e)
  File "/Users/graeme/svn/cctbx/build/../modules/dials/command_line/dials_import.py", line 830, in <module>
    script.run()
  File "/Users/graeme/svn/cctbx/build/../modules/dials/command_line/dials_import.py", line 693, in run
    params, options = self.parser.parse_args(show_diff_phil=False)
  File "/Users/graeme/svn/cctbx/modules/dials/util/options.py", line 835, in parse_args
    quick_parse=quick_parse)
  File "/Users/graeme/svn/cctbx/modules/dials/util/options.py", line 553, in parse_args
    format_kwargs=format_kwargs)
  File "/Users/graeme/svn/cctbx/modules/dials/util/options.py", line 219, in __init__
    format_kwargs)
  File "/Users/graeme/svn/cctbx/modules/dials/util/options.py", line 285, in try_read_datablocks_from_images
    format_kwargs=format_kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 935, in from_filenames
    format_kwargs=format_kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 464, in __init__
    format_kwargs=format_kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 634, in _create_single_file_imageset
    format_kwargs=format_kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/FormatMultiImage.py", line 184, in get_imageset
    reader = Class.get_reader()(filenames, num_images=num_images, **format_kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/FormatMultiImage.py", line 15, in __init__
    self._num_images = self.read_num_images()
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/FormatMultiImage.py", line 31, in read_num_images
    format_instance = self.format_class.get_instance(self._filename, **self.kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/Format.py", line 307, in get_instance
    Class._current_instance_ = Class(filename, **kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/FormatHDFEigerNearlyNexus.py", line 292, in __init__
    FormatHDF5.__init__(self, image_file, **kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/FormatHDF5.py", line 17, in __init__
    Format.__init__(self, image_file, **kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/Format.py", line 199, in __init__
    self.setup()
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/Format.py", line 209, in setup
    self._start()
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/FormatHDFEigerNearlyNexus.py", line 341, in _start
    self._raw_data = DataFactory(data).model
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/format/nexus.py", line 1604, in __init__
    if obj.handle[key].ndim == 1:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/Users/graeme/svn/cctbx/base/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_hl/group.py", line 167, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Please report this error to dials-support@lists.sourceforge.net: Unable to open object (unable to open file: name = '/Users/graeme/data/h5/200Hz_0.1dpf_1_data_000036.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)"

if try to import master with missing block

[graeme-winter]

image_range= does not work around this problem...

[graeme-winter]

Also turns out to dig a lot of holes - the resulting data structures are pretty universally inconsistent.

Almost certainly better idea is to be able to rewrite the master file $somehow with no references to the missing files...

[biochem-fan]

I think this issue should be addressed not only by DIALS but also by EIGER detector software. How about bringing this to the HDRMX mailing list?

[graeme-winter]

@biochem-fan exactly what I am doing literally at this instant 🙂

[JieNanMAXIV]

Even with rewriting the master file, the feature with data range is still appreciated in some occasions. E.x. with SSX file, a dataset is often quite large, over 100K images. It takes a looooong time to load the whole dataset though one may just want to analyze a small fraction of images.

[JieNanMAXIV]

As Aaron and David requested, I uploaded an example dataset, https://drive.google.com/open?id=1QALbjV7KYquhuMlGOeU1Akswlhxhf0F5. The experiment was set to collect 1M images for jet SSX, and aborted after some time. test-lyso_1_master.h5 and test-lyso_1_data_000001.h5 are the original images, which doesn't work with dials.import. test-lyso_1_master_new.h5 is the new master file, removing the datalinks by h5py, and dials.import works then.

[dagewa]

I confirm that this remains a problem using @JieNanMAXIV's example dataset

[stale]

This issue has been automatically marked as stale because it has not had recent activity. The label will be removed automatically if any activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. The label will be removed automatically if any activity occurs. Thank you for your contributions.

[jbeilstenedmands]

Please reopen if this is found to still be an occuring issue on recent data collections