Format needs get_static_mask

[dagewa]

Following discussion in #65, Format needs a get_static_mask method. This will:

• Return either a mask or None
• Be set in the ImageSet during instantiation (imageset.external_lookup.mask.data = ImageBool(mask_flex_array_or_tuple_of_flex_arrays))
• Allow merging of a user-provided mask with the mask from the format class
• Be exercised by the dxtbx or DIALS tests

[rjgildea]

The following is my attempt at rationalising the current state of masking in format classes/imagesets:

Dynamic masking

I.e. a mask that varies image-to-image, with a dependence on the goniometer/scan.

Overload the get_goniometer_shadow_masker() method in the format class, e.g.:

    def get_goniometer_shadow_masker(self, goniometer=None):
        if goniometer is None:
            goniometer = self.get_goniometer()

		return GoniometerMaskerFactory.smargon(goniometer)

(see

dxtbx/format/Format.py

Lines 500 to 504 in df55d9a

	def get_goniometer_shadow_masker(self, goniometer=None):
	"""Overload this method to allow generation of dynamic goniometer shadow
	masks to be used during spotfinding or integration."""
	return None

)

and set the _dynamic_shadowing attribute of the format class to True.

This method is called by Format.get_masker() if the format class has the _dynamic_shadowing
attribute set to True and goniometer is an instance of MultiAxisGoniometer:

dxtbx/format/Format.py

Lines 269 to 281 in df55d9a

	def get_masker(self, goniometer=None):
	"""
	Return a masker class
	"""
	if (
	isinstance(goniometer, MultiAxisGoniometer)
	and hasattr(self, "_dynamic_shadowing")
	and self._dynamic_shadowing
	):
	masker = self.get_goniometer_shadow_masker(goniometer=goniometer)
	else:
	masker = None
	return masker

Format.get_masker() is called from Format.get_imageset(), where it is passed as an argument to ImageSetData during the construction of the ImageSweep object:

dxtbx/format/Format.py

Lines 427 to 447 in df55d9a

	# Create the masker
	if format_instance is not None:
	masker = format_instance.get_masker(goniometer=goniometer)
	else:
	masker = None
	# Create the sweep
	iset = ImageSweep(
	ImageSetData(
	reader=reader,
	masker=masker,
	vendor=vendor,
	params=params,
	format=Class,
	template=template,
	),
	beam=beam,
	detector=detector,
	goniometer=goniometer,
	scan=scan,
	)

ImageSweep.get_mask() calls ImageSweep.get_dynamic_mask(), which then calls the `get_mask() method of the dynamic masker object:

dxtbx/imageset.h

Lines 1244 to 1267 in 2086723

	/**
	* Get the dynamic mask for the requested image
	* @param index The image index
	* @returns The image mask
	*/
	virtual Image<bool> get_dynamic_mask(std::size_t index) {
	// Get the masker
	ImageSetData::masker_ptr masker = data_.masker();
	// Create return buffer
	Image<bool> dyn_mask;
	// Get the image data object
	if (masker != NULL) {
	DXTBX_ASSERT(scan_ != NULL);
	DXTBX_ASSERT(detector_ != NULL);
	double scan_angle = rad_as_deg(
	scan_->get_angle_from_image_index(index + scan_->get_image_range()[0]));
	dyn_mask = masker->get_mask(*detector_, scan_angle);
	}
	// Return the dynamic mask
	return get_trusted_range_mask(get_static_mask(dyn_mask), index);
	}

ImageSweep.get_dynamic_mask() also applies ImageSet.get_trusted_range_mask():

dxtbx/imageset.h

Line 1266 in 2086723

return get_trusted_range_mask(get_static_mask(dyn_mask), index);

dxtbx/imageset.h

Lines 856 to 872 in 2086723

	/**
	* Get the trusted range mask for the index
	* @param mask The mask to write into
	* @param index The image index
	* @returns The mask
	*/
	Image<bool> get_trusted_range_mask(Image<bool> mask, std::size_t index) {
	Detector detector = detail::safe_dereference(get_detector_for_image(index));
	Image<double> data = get_raw_data(index).as_double();
	DXTBX_ASSERT(mask.n_tiles() == data.n_tiles());
	DXTBX_ASSERT(data.n_tiles() == detector.size());
	for (std::size_t i = 0; i < detector.size(); ++i) {
	detector[i].apply_trusted_range_mask(data.tile(i).data().const_ref(),
	mask.tile(i).data().ref());
	}
	return mask;
	}

Goniometer shadow masker

Goniometer shadow maskers should be an instance of dxtbx.masking.GoniometerShadowMasker (or subclass):

https://github.com/cctbx/dxtbx/blob/master/masking/goniometer_shadow_masking.h#L33-L225

There is a GoniometerMaskerFactory class that provides methods for constructing varying different types of GoniometerShadowMasker that can be re-used in different format classes:

dxtbx/masking/__init__.py

Lines 31 to 213 in 2086723

	class GoniometerMaskerFactory(object):
	@staticmethod
	def mini_kappa(goniometer, cone_opening_angle=43.60281897270362):
	"""Construct a GoniometerShadowMasker for a mini-kappa goniometer.
	This is modelled a simple cone with the opening angle specified by
	`cone_opening_angle`.
	Args:
	goniometer (`dxtbx.model.Goniometer`): The goniometer instance.
	cone_opening_angle (float): The opening angle of the cone (in degrees).
	Returns:
	`dxtbx.masking.GoniometerShadowMasker`
	"""
	assert isinstance(goniometer, MultiAxisGoniometer)
	assert len(goniometer.get_axes()) == 3
	# Simple model of cone around goniometer phi axis
	# Exact values don't matter, only the ratio of height/radius
	height = 50 # mm
	radius_height_ratio = math.tan(1 / 2 * cone_opening_angle * math.pi / 180)
	radius = radius_height_ratio * height
	steps_per_degree = 1
	theta = (
	flex.double(range(360 * steps_per_degree))
	* math.pi
	/ 180
	* 1
	/ steps_per_degree
	)
	y = radius * flex.cos(-theta) # x
	z = radius * flex.sin(-theta) # y
	x = flex.double(theta.size(), height) # z
	coords = flex.vec3_double(zip(x, y, z))
	coords.insert(0, (0, 0, 0))
	return GoniometerShadowMasker(goniometer, coords, flex.size_t(len(coords), 0))
	@staticmethod
	def dls_i23_kappa(goniometer):
	"""Construct a GoniometerShadowMasker for the DLS I23 Kappa goniometer.
	Args:
	goniometer (`dxtbx.model.Goniometer`): The goniometer instance.
	Returns:
	`dxtbx.masking.GoniometerShadowMasker`
	"""
	coords = flex.vec3_double(((0, 0, 0),))
	alpha = flex.double_range(0, 190, step=10) * math.pi / 180
	r = flex.double(alpha.size(), 40)
	x = flex.double(r.size(), 107.61)
	y = -r * flex.sin(alpha)
	z = -r * flex.cos(alpha)
	coords.extend(flex.vec3_double(x, y, z))
	coords.extend(
	flex.vec3_double(
	(
	# fixed
	(107.49, 7.84, 39.49),
	(107.39, 15.69, 38.97),
	(107.27, 23.53, 38.46),
	(107.16, 31.37, 37.94),
	(101.76, 33.99, 36.25),
	(96.37, 36.63, 34.56),
	(90.98, 39.25, 33.00),
	(85.58, 41.88, 31.18),
	(80.89, 47.06, 31.00),
	(76.55, 51.51, 31.03),
	(72.90, 55.04, 31.18),
	(66.86, 60.46, 31.67),
	(62.10, 64.41, 32.25),
	)
	)
	)
	alpha = flex.double_range(180, 370, step=10) * math.pi / 180
	r = flex.double(alpha.size(), 33)
	x = flex.sqrt(flex.pow2(r * flex.sin(alpha)) + 89.02 ** 2) * flex.cos(
	(50 * math.pi / 180) - flex.atan(r / 89.02 * flex.sin(alpha))
	)
	y = flex.sqrt(flex.pow2(r * flex.sin(alpha)) + 89.02 ** 2) * flex.sin(
	(50 * math.pi / 180) - flex.atan(r / 89.02 * flex.sin(alpha))
	)
	z = -r * flex.cos(alpha)
	coords.extend(flex.vec3_double(x, y, z))
	coords.extend(
	flex.vec3_double(
	(
	# fixed
	(62.10, 64.41, -32.25),
	(66.86, 60.46, -31.67),
	(72.90, 55.04, -31.18),
	(76.55, 51.51, -31.03),
	(80.89, 47.06, -31.00),
	(85.58, 41.88, -31.18),
	(90.98, 39.25, -33.00),
	(96.37, 36.63, -34.56),
	(101.76, 33.99, -36.25),
	(107.16, 31.37, -37.94),
	(107.27, 23.53, -38.46),
	(107.39, 15.69, -38.97),
	(107.49, 7.84, -39.49),
	(107.61, 0.00, -40.00),
	)
	)
	)
	# I23 end station coordinate system:
	# X-axis: positive direction is facing away from the storage ring (from
	# sample towards goniometer)
	# Y-axis: positive direction is vertically up
	# Z-axis: positive direction is in the direction of the beam (from
	# sample towards detector)
	# K-axis (kappa): at an angle of +50 degrees from the X-axis
	# K & phi rotation axes: clockwise rotation is positive (right hand
	# thumb rule)
	# Omega-axis: along the X-axis; clockwise rotation is positive
	# End station x-axis is parallel to ImgCIF x-axis
	# End station z-axis points in opposite direction to ImgCIF definition
	# (ImgCIF: The Z-axis is derived from the source axis which goes from
	# the sample to the source)
	# Consequently end station y-axis (to complete set following right hand
	# rule) points in opposite direction to ImgCIF y-axis.
	# Kappa arm aligned with -y in ImgCIF convention
	R = align_reference_frame(
	matrix.col((1, 0, 0)),
	matrix.col((1, 0, 0)),
	matrix.col((0, 1, 0)),
	matrix.col((0, -1, 0)),
	)
	coords = R.elems * coords
	return GoniometerShadowMasker(goniometer, coords, flex.size_t(len(coords), 1))
	@staticmethod
	def smargon(goniometer):
	"""Construct a SmarGonShadowMasker for the SmarGon goniometer.
	Args:
	goniometer (`dxtbx.model.Goniometer`): The goniometer instance.
	Returns:
	`dxtbx.masking.SmarGonShadowMasker`
	"""
	return SmarGonShadowMasker(goniometer)
	@staticmethod
	def diamond_anvil_cell(goniometer, cone_opening_angle):
	radius_height_ratio = math.tan(1 / 2 * cone_opening_angle)
	height = 10 # mm
	radius = radius_height_ratio * height
	steps_per_degree = 1
	theta = (
	flex.double([list(range(360 * steps_per_degree))])
	* math.pi
	/ 180
	* 1
	/ steps_per_degree
	)
	x = radius * flex.cos(theta) # x
	z = radius * flex.sin(theta) # y
	y = flex.double(theta.size(), height) # z
	coords = flex.vec3_double(zip(x, y, z))
	coords.extend(flex.vec3_double(zip(x, -y, z)))
	coords.insert(0, (0, 0, 0))
	return GoniometerShadowMasker(
	goniometer, coords, flex.size_t(len(coords), 0), True
	)

Static mask

I.e. a mask that is common to all images in the imageset.

The method ImageSet.get_static_mask() combines the mask generated by Imageset.get_untrusted_rectangle_mask() with an optional external mask that is set to the external_lookup_ attribute of the ImageSet:

dxtbx/imageset.h

Lines 838 to 854 in 2086723

	/**
	* Get the static mask common to all images
	* @param mask The input mask
	* @returns The mask
	*/
	Image<bool> get_static_mask(Image<bool> mask) {
	return get_untrusted_rectangle_mask(
	get_external_mask(mask.empty() ? get_empty_mask() : mask));
	}
	/**
	* Get the static mask common to all images
	* @returns The mask
	*/
	Image<bool> get_static_mask() {
	return get_static_mask(Image<bool>());
	}

dxtbx/imageset.h

Lines 816 to 836 in 2086723

	/**
	* Apply the external mask
	* @param mask The input mask
	* @returns The external mask
	*/
	Image<bool> get_external_mask(Image<bool> mask) {
	Image<bool> external_mask = external_lookup().mask().get_data();
	if (!external_mask.empty()) {
	DXTBX_ASSERT(external_mask.n_tiles() == mask.n_tiles());
	for (std::size_t i = 0; i < mask.n_tiles(); ++i) {
	scitbx::af::ref<bool, scitbx::af::c_grid<2> > m1 = mask.tile(i).data().ref();
	scitbx::af::const_ref<bool, scitbx::af::c_grid<2> > m2 =
	external_mask.tile(i).data().const_ref();
	DXTBX_ASSERT(m1.accessor().all_eq(m2.accessor()));
	for (std::size_t j = 0; j < m1.size(); ++j) {
	m1[j] = m1[j] && m2[j];
	}
	}
	}
	return mask;
	}

dxtbx/imageset.h

Lines 802 to 814 in 2086723

	/**
	* Get the untrusted rectangle mask
	* @param mask The mask to write into
	* @returns The mask
	*/
	Image<bool> get_untrusted_rectangle_mask(Image<bool> mask) const {
	Detector detector = detail::safe_dereference(get_detector_for_image(0));
	DXTBX_ASSERT(mask.n_tiles() == detector.size());
	for (std::size_t i = 0; i < detector.size(); ++i) {
	detector[i].apply_untrusted_rectangle_mask(mask.tile(i).data().ref());
	}
	return mask;
	}

imageset.external_lookup.mask.data is set in e.g. dials.import or dials.generate_mask:

https://github.com/dials/dials/blob/eea37fbc02fac730169e700b53a0f7e060902bb7/command_line/dials_import.py#L587-L589

https://github.com/dials/dials/blob/c7753bedcf7ac6c8fc575103833df4527f52ad8d/command_line/generate_mask.py#L144-L146

The functionality that is currently missing is that there is no way for a format class to override/update the static mask. What is proposed in #65 (comment) and #70 is to add a get_static_mask() method to Format which returns either a mask or None and then set that in the ImageSet when it is instantiated. This can be done as follows:

imageset.external_lookup.mask.data = ImageBool(mask_flex_array_or_tuple_of_flex_arrays)

[biochem-fan]

override/update the static mask

Where does "merging of a user-provided mask" happen?

[rjgildea]

@biochem-fan this i

Where does "merging of a user-provided mask" happen?

This happens in the call to ImageSet.get_static_mask() which combines it with the untrusted rectangle mask:

dxtbx/imageset.h

Lines 838 to 846 in 2086723

	/**
	* Get the static mask common to all images
	* @param mask The input mask
	* @returns The mask
	*/
	Image<bool> get_static_mask(Image<bool> mask) {
	return get_untrusted_rectangle_mask(
	get_external_mask(mask.empty() ? get_empty_mask() : mask));
	}

The external_lookup.mask.data attribute can be set in e.g. dials.import or dials.generate_mask. What is being suggested is that we create a mechanism that allows the format class to set/modify imageset.external_lookup.mask.data when constructing the imageset.

[biochem-fan]

the format class to set/modify imageset.external_lookup.mask.data

Does it mean Format.get_static_mask() should take an existing user-provided mask, merge it with the format specific mask and return a new mask? That is:

imageset.external_lookup.mask.data = format.get_static_mask(imageset.external_lookup.mask.data)

Then ImageSet.get_static_mask(mask) further merges it with the untrusted rectangle mask?

Shouldn't the first merging step with the user-provided mask (external_lookup.mask) happen in ImageSet?

[rjgildea]

the format class to set/modify imageset.external_lookup.mask.data

Does it mean Format.get_static_mask() should take an existing user-provided mask, merge it with the format specific mask and return a new mask? That is:

imageset.external_lookup.mask.data = format.get_static_mask(imageset.external_lookup.mask.data)

Then ImageSet.get_static_mask(mask) further merges it with the untrusted rectangle mask?

Yes, that is what I understand of @jmp1985's suggestion.

Shouldn't the first merging step with the user-provided mask (external_lookup.mask) happen in ImageSet?

This issue with this approach is that it would require further modification to Imageset and likely other places in dxtbx/imageset.h to pass and store a format class-specified static mask, whereas the above suggestion would only need changes to Format.py.

[rjgildea]

@biochem-fan do you have a small example file (a single image would suffice) that we can use to exercise the format class-defined mask in dxtbx/format/FormatHDF5SaclaMPCCD.py? If this was small enough we might be able to add it to https://github.com/dials/data-files, or possibly better on zenodo as long as the files aren't compressed (see e.g. https://github.com/dials/data/blob/8b7f34c9467997fc49dedf2c563b946a2b770d33/dials_data/definitions/vmxi_thaumatin.yml).

[phyy-nx]

There's one in dials_regression. test_stills_process.py uses it. cd dials_regression/image_examples/SACLA_MPCCD_Cheetah dxtbx.print_header run266702-0-subset.h5 | head === run266702-0-subset.h5 === Using header reader: FormatHDF5SaclaMPCCD

…

On Fri, Aug 9, 2019 at 10:08 AM Richard Gildea ***@***.***> wrote: @biochem-fan <https://github.com/biochem-fan> do you have a small example file (a single image would suffice) that we can use to exercise the format class-defined mask in dxtbx/format/FormatHDF5SaclaMPCCD.py? If this was small enough we might be able to add it to https://github.com/dials/data-files, or possibly better on zenodo as long as the files aren't compressed (see e.g. https://github.com/dials/data/blob/8b7f34c9467997fc49dedf2c563b946a2b770d33/dials_data/definitions/vmxi_thaumatin.yml ). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#70?email_source=notifications&email_token=ADGY5SRDZ4DQ53WYRZCSLZDQDWP27A5CNFSM4II5XFR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD37HS6A#issuecomment-519993720>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADGY5SRAXEX4VGADSHR6FITQDWP27ANCNFSM4II5XFRQ> .

[biochem-fan]

@rjgildea Isn't it simply modifying ImageSet.get_static_mask(mask)? My worry is that by delegating the merging step to implementation in each Format class, a buggy class might ignore the user-specified mask. It also causes code duplication among many classes.

@phyy-nx That image was created by an early version of the pipeline and does not contain the mask. Please try https://drive.google.com/open?id=0ByqmCYlnqv6UU1RXb053MG1ucmc (11 MB).

[rjgildea]

@rjgildea Isn't it simply modifying ImageSet.get_static_mask(mask)? My worry is that by delegating the merging step to implementation in each Format class, a buggy class might ignore the user-specified mask. It also causes code duplication among many classes.

No, the merging step would be in the Format baseclass, probably somewhere around this point in Format.get_imageset() which would call Format.get_static_mask():

https://github.com/dials/dxtbx/blob/master/format/Format.py#L427-L447

In the case of FormatHDF5SaclaMPCCD it would look pretty much the same as the existing (currently unused) FormatHDF5SaclaMPCCDget_mask() function:

https://github.com/dials/dxtbx/blob/b3c9a071a22f1d64da2e42b042b9d9c2c9bc3435/format/FormatHDF5SaclaMPCCD.py#L400-L404

@phyy-nx That image was created by an early version of the pipeline and does not contain the mask. Please try https://drive.google.com/open?id=0ByqmCYlnqv6UU1RXb053MG1ucmc (11 MB).

Thanks!

[biochem-fan]

@rjgildea OK, this makes sense.

[rjgildea]

Unfortunately it's not quite as simple as I'd hoped. Even if the imageset is returned here with the imageset.external_lookup.mask.data set by the format class:

dxtbx/model/experiment_list.py

Lines 227 to 230 in 2086723

	if imageset_data["__id__"] == "ImageSet":
	imageset = self._make_stills(
	imageset_data, format_kwargs=format_kwargs
	)

the mask is then overwritten again here:

dxtbx/model/experiment_list.py

Lines 249 to 280 in 2086723

	if imageset is not None:
	# Set the external lookup
	if mask is None:
	mask = ImageBool()
	else:
	mask = ImageBool(mask)
	if gain is None:
	gain = ImageDouble()
	else:
	gain = ImageDouble(gain)
	if pedestal is None:
	pedestal = ImageDouble()
	else:
	pedestal = ImageDouble(pedestal)
	if dx is None:
	dx = ImageDouble()
	else:
	dx = ImageDouble(dx)
	if dy is None:
	dy = ImageDouble()
	else:
	dy = ImageDouble(dy)
	imageset.external_lookup.mask.data = mask
	imageset.external_lookup.mask.filename = mask_filename
	imageset.external_lookup.gain.data = gain
	imageset.external_lookup.gain.filename = gain_filename
	imageset.external_lookup.pedestal.data = pedestal
	imageset.external_lookup.pedestal.filename = pedestal_filename
	imageset.external_lookup.dx.data = dx
	imageset.external_lookup.dx.filename = dx_filename
	imageset.external_lookup.dy.data = dy
	imageset.external_lookup.dy.filename = dy_filename

[rjgildea]

I've made a start towards this here 0e36727. This works if you run dials.image_viewer directly on the image file:

$ dials.image_viewer MPCCD-Phase3-21528-5images.h5 show_mask=True

However it doesn't work when loading from an experiments json file, due to the imageset.external_lookup.mask being overwritten as mentioned above:

$ dials.import MPCCD-Phase3-21528-5images.h5
$ dials.image_viewer imported.expt show_mask=True

[rjgildea]

@biochem-fan @phyy-nx do you also have example images for testing the masking in FormatHDF5SaclaRayonix.py and FormatPYunspecified.py?

[biochem-fan]

@rjgildea Currently the Rayonix detector at SACLA does not need a detector specific mask. The get_mask() function is not used now (self.mask is always None). But they might become relevant in future (i.e. when the detector gets damaged over time).

[phyy-nx]

For FormatPYunspecified, see dials_regression/image_examples/LCLS_CXI/shot-s00-2011-12-02T21_07Z29.723_00569.pickle and shot-s04-20111204004533388.pickle. This format class is having issues in py3 which I have to sort out at some point, but the file format itself (serialized pickle dictionaries) is considered legacy at this point. We aren't making any more of these files at LCLS and haven't been for some time. But they are good examples of get_mask.

…

On Mon, Aug 12, 2019 at 9:44 AM biochem_fan ***@***.***> wrote: @rjgildea <https://github.com/rjgildea> Currently the Rayonix detector at SACLA does not need a detector specific mask. The get_mask() function is not used now (self.mask is always None). But they might become relevant in future (i.e. when the detector gets damaged over time). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70?email_source=notifications&email_token=ADGY5SXTSWSU3QJ6HAQY3EDQEGHQNA5CNFSM4II5XFR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4DD3CQ#issuecomment-520502666>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADGY5SSVC4KQIITNIQZPA33QEGHQNANCNFSM4II5XFRQ> .

[rjgildea]

@rjgildea Currently the Rayonix detector at SACLA does not need a detector specific mask. The get_mask() function is not used now (self.mask is always None). But they might become relevant in future (i.e. when the detector gets damaged over time).

It would probably still be a good idea to have an example dataset available so we can ensure test coverage of this format class.

[biochem-fan]

@keitaroyam or @phyy-nx, do you have a test image from Rayonix we can make public?

[rjgildea]

@phyy-nx are the FormatPYunspecifiedInMemory/FormatPYunspecifiedStillInMemory classes still needed/expected to work? As far as I can tell they don't work as it stands:

5cdffe4#diff-e59100cf97d8b4af3427ae450ca1476eR34-R46

$ pytest --regression tests/format/test_FormatPYunspecified.py::test_FormatPYunspecifiedStillInMemory --runxfail
...
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_FormatPYunspecifiedStillInMemory ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

dials_regression = '/Users/rjgildea/software/cctbx/modules/dials_regression'

    @pytest.mark.xfail
    def test_FormatPYunspecifiedStillInMemory(dials_regression):
        filename = os.path.join(
            dials_regression,
            "image_examples/LCLS_CXI/shot-s00-2011-12-02T21_07Z29.723_00569.pickle",
        )
        assert not FormatPYunspecifiedStillInMemory.understand(filename)
        with open(filename, "rb") as f:
            d = pickle.load(f)
        assert FormatPYunspecifiedStillInMemory.understand(d)
>       mem_imageset = FormatPYunspecifiedStillInMemory.get_imageset(d)  # noqa F841

tests/format/test_FormatPYunspecified.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
format/Format.py:321: in get_imageset
    format_instance = Class(filenames[0], **format_kwargs)
format/FormatPYunspecifiedStill.py:70: in __init__
    FormatPYunspecifiedInMemory.__init__(self, data, **kwargs)
format/FormatPYunspecified.py:227: in __init__
    FormatPYunspecified.__init__(self, data, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dxtbx.format.FormatPYunspecifiedStill.FormatPYunspecifiedStillInMemory object at 0x117146890>, image_file = '/Users/rjgildea/software/cctbx/modules/dxtbx/DISTANCE', kwargs = {}

    def __init__(self, image_file, **kwargs):
        """Initialise the image structure from the given file."""
    
        if not self.understand(image_file):
>           raise IncorrectFormatError(self, image_file)
E           IncorrectFormatError: (<dxtbx.format.FormatPYunspecifiedStill.FormatPYunspecifiedStillInMemory object at 0x117146890>, '/Users/rjgildea/software/cctbx/modules/dxtbx/DISTANCE')

format/FormatPYunspecified.py:46: IncorrectFormatError

[phyy-nx]

Hi, it's used in xtc_process, where the get_imageset method is never called. Instead, this code pattern is used: from libtbx import easy_pickle from dxtbx.format.FormatPYunspecifiedStill import FormatPYunspecifiedStillInMemory from dxtbx.imageset import ImageSet, ImageSetData, MemReader d = easy_pickle.load('/net/dials/raid1/aaron/dialsbuild/modules/dials_regression/image_examples/LCLS_CXI/shot-s04-20111204004533388.pickle') img = FormatPYunspecifiedStillInMemory(d) imgset = ImageSet(ImageSetData(MemReader([img]), None)) Again, it's legacy code but it does still work I believe. The image dictionary in shot-s04-20111204004533388.pickle was assembled in memory earlier in xtc_process, and the MemReader is used so the image doesn't need to be written to disk. Same principle with the cspad cbf in memory code that we use now.

…

On Tue, Aug 13, 2019 at 8:16 AM Richard Gildea ***@***.***> wrote: @phyy-nx <https://github.com/phyy-nx> are the FormatPYunspecifiedInMemory/FormatPYunspecifiedStillInMemory classes still needed/expected to work? As far as I can tell they don't work as it stands: 5cdffe4#diff-e59100cf97d8b4af3427ae450ca1476eR34-R46 <5cdffe4#diff-e59100cf97d8b4af3427ae450ca1476eR34-R46> $ pytest --regression tests/format/test_FormatPYunspecified.py::test_FormatPYunspecifiedStillInMemory --runxfail ... ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_FormatPYunspecifiedStillInMemory ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― dials_regression = '/Users/rjgildea/software/cctbx/modules/dials_regression' @pytest.mark.xfail def test_FormatPYunspecifiedStillInMemory(dials_regression): filename = os.path.join( dials_regression, "image_examples/LCLS_CXI/shot-s00-2011-12-02T21_07Z29.723_00569.pickle", ) assert not FormatPYunspecifiedStillInMemory.understand(filename) with open(filename, "rb") as f: d = pickle.load(f) assert FormatPYunspecifiedStillInMemory.understand(d) > mem_imageset = FormatPYunspecifiedStillInMemory.get_imageset(d) # noqa F841 tests/format/test_FormatPYunspecified.py:46: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ format/Format.py:321: in get_imageset format_instance = Class(filenames[0], **format_kwargs) format/FormatPYunspecifiedStill.py:70: in __init__ FormatPYunspecifiedInMemory.__init__(self, data, **kwargs) format/FormatPYunspecified.py:227: in __init__ FormatPYunspecified.__init__(self, data, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <dxtbx.format.FormatPYunspecifiedStill.FormatPYunspecifiedStillInMemory object at 0x117146890>, image_file = '/Users/rjgildea/software/cctbx/modules/dxtbx/DISTANCE', kwargs = {} def __init__(self, image_file, **kwargs): """Initialise the image structure from the given file.""" if not self.understand(image_file): > raise IncorrectFormatError(self, image_file) E IncorrectFormatError: (<dxtbx.format.FormatPYunspecifiedStill.FormatPYunspecifiedStillInMemory object at 0x117146890>, '/Users/rjgildea/software/cctbx/modules/dxtbx/DISTANCE') format/FormatPYunspecified.py:46: IncorrectFormatError — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70?email_source=notifications&email_token=ADGY5SVY35WFIF5THJIGT6TQELF6FA5CNFSM4II5XFR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4F72AA#issuecomment-520879360>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADGY5SUZ55LXHV42E7TZ5CLQELF6FANCNFSM4II5XFRQ> .