Zoom data reduction #50

SimonHeybrock · 2024-01-16T11:58:19Z

This adds a data reduction workflow for Zoom. The main addition is handling of file I/O and reading masks from a series of XML files.

esssans.isis.mantidio provides I/O wrapper depending on Mantid.
For the docs we are using pooch as usual, bypassing Mantid.
I found a way of hiding the Pooch get_path stuff from the docs notebooks, but adding a provider that converts filenames into paths. This can either be configured to use a configured data folder, or pooch, or anything else someone might come up with (such as search known locations, ...).

nvaytet · 2024-01-18T13:10:12Z

src/esssans/isis/__init__.py

@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: BSD-3-Clause


Should the sans2d submodule be part of isis?

Probably, but I want to keep any unrelated changes out of this PR. We can perform such restructuring and rollout of changes in a follow-up.

docs/examples/zoom.ipynb

nvaytet · 2024-01-18T13:14:51Z

docs/examples/zoom.ipynb

+    "    sans.isis.transmission_from_background_run,\n",
+    "    sans.isis.transmission_from_sample_run,\n",


Why are those not part of the isis.providers?

Because someone might want to set a separate transmission run, and in that case this should not be added.

nvaytet · 2024-01-18T13:15:55Z

docs/examples/zoom.ipynb

+   "metadata": {},
+   "source": [
+    "If Mantid is available, we can use it to load data files.\n",
+    "**You must configure the `DataFolder` below to point to the directory containing the data files.**\n",


DataFolder renders strangely in the docs

nvaytet · 2024-01-18T13:18:32Z

docs/examples/zoom.ipynb

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "da = iofq.compute()\n",


Maybe we should silence the pooch warnings everywhere?

nvaytet · 2024-01-18T13:20:02Z

src/esssans/isis/data.py

@@ -0,0 +1,66 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2023 Scipp contributors (https://github.com/scipp)


Suggested change

# Copyright (c) 2023 Scipp contributors (https://github.com/scipp)

# Copyright (c) 2024 Scipp contributors (https://github.com/scipp)

nvaytet · 2024-01-18T13:21:34Z

src/esssans/isis/data.py

+        'andru_test.xml': 'md5:c59e0c4a80640a387df7beca4857e66f',
+        'left_beg_18_2.xml': 'md5:5b24a8954d4d8a291f59f5392cd61681',
+        'right_beg_18_2.xml': 'md5:fae95a5056e5f5ba4996c8dff83ec109',
+        'small_bs_232.xml': 'md5:6d67dea9208193c9f0753ffcbb50ed83',
+        'small_BS_31032023.xml': 'md5:3c644e8c75105809ab521773f9c0c85b',
+        'tube_1120_bottom.xml': 'md5:fe577bf73c16bf5ac909516fa67360e9',
+        'tubes_beg_18_2.xml': 'md5:2debde8d82c383cc3d592ea000552300',


Do you know what the different masks represent? (i.e. what they are trying to mask?)

Not in detail. Some tubes, beamstop, sample holder, but there appear to be multiple varieties. Why?

I was thinking since we started adding short comments describing what the different files are, we could add some descriptions here?

I extended the description.

nvaytet · 2024-01-18T13:23:45Z

src/esssans/isis/io.py

+DataFolder = NewType('DataFolder', str)
+
+
+class Path(sciline.Scope[FilenameType, str], str):


Path will very commonly refer to pathlib.Path. Can we avoid using the same name (especially as we sometimes from types import *?)

Renamed to FilePath.

especially as we sometimes from types import *?

I want to get away from that. As you can see I have kept providers and types more structured here, no placing anything in a "meaningless" types.py. Still not there yet, but I hope it will work nice eventually.

nvaytet · 2024-01-18T13:25:47Z

src/esssans/isis/io.py

+    return f'{path}/{filename}'
+
+
+def read_xml_detector_masking(filename: Path[PixelMaskFilename]) -> MaskedDetectorIDs:


Is the format of those xml mask files specific to Mantid or are they also used by other software? If the former, should this move into mantidio?

Well, XML is used elsewhere, but the files are afaik pecific to Mantid.

nvaytet · 2024-01-18T13:28:23Z

src/esssans/isis/mantidio.py

+        data_ws = loaded
+    else:
+        # Separate data and monitor workspaces
+        data_ws = loaded.OutputWorkspace


How do we get hold of the monitor data in this case?

Mantid stores a link to the monitor workspace in the data workspace and that is where ScippNeutron pulls the monitors from, i.e., the fact that it returns both is redundant.

nvaytet · 2024-01-23T08:14:42Z

src/esssans/isis/visualization.py

+import scipp as sc
+
+
+def plot_instrument(da: sc.DataArray, pixels_per_tube: int = 512, **kwargs: Any) -> Any:


Are you thinking this is generic enough that it can be used for other isis instruments? I guess it would work for sans2d as well. But then it should also work for other flat panels, not just isis panels?
Should we call it something else that implies it is plotting a flat panel? As it wouldn't work for something else.

Are there any ISIS SANS instruments that do not have tubes that we have to worry about?

I renamed it.

nvaytet · 2024-01-23T08:15:19Z

src/esssans/isis/visualization.py

+    Parameters
+    ----------
+    da:
+        The data array to plot. Must have a 'position' coord and a single dimension.


Is it not possible to flatten if there are more than one dimension?

Sure, but this is for ISIS data, which has been loaded with Mantid, so why complicate things? It can be extended once/if required?

I had considered doing this, but then it would complicate since we would need to maybe support summing some dims but flattening others, I didn't think it was worth it at this point. After all, this is not an ESS instrument.

nvaytet · 2024-01-23T08:17:15Z

src/esssans/isis/visualization.py

+        folded.coords['y'] = y.min('x')
+    else:
+        raise ValueError(
+            'Cannot plot 2-D instrument view of data array with non-constant '


Do we really have to raise or would it just make a ragged detector panel, like the other plots we have with 2d coords?

Ah maybe it's because the x coord is already 2d?

Yes, Scipp (and plotting, I guess?) do not support multiple 2-D coords. Or is Plopp supporting this?

Or is Plopp supporting this?

No, we are not.

nvaytet · 2024-01-23T08:19:28Z

src/esssans/isis/visualization.py

+            'Cannot plot 2-D instrument view of data array with non-constant '
+            'y coordinate along tubes. Use scippneutron.instrument_view instead.'
+        )
+    plot_kwargs = dict(aspect='equal', norm='log', figsize=(6, 10))


I think the aspect='equal' make sense for this, but I find it a little strange that it has log as a default, and also changes the figsize? This size would not work e.g. for the sans2d panel.

SimonHeybrock · 2024-01-23T09:36:11Z

There are still some position offsets to handle (from user file), but since I do not know when we will get the information on what to implement I will merge and handle the remaining bits in a follow-up.

SimonHeybrock marked this pull request as draft January 16, 2024 11:58

Base automatically changed from beam-direct-v2 to main January 17, 2024 14:01

SimonHeybrock added 13 commits January 18, 2024 07:47

Draft Zoom example

d7d1aff

Begin adding isis module with Mantid loaders

59b9a7b

Add run loader

1139da5

Add masking

fdd5200

Update Zoom notebook draft

a574970

Hack to get correct spectrum numbers for masks

1ec5499

Continue work on Zoom notebook

c84fedb

More cleanup

11b88fd

Get gravity direction via reference frame

2b2e5e9

Cleanup

2bf3b34

Rewrite mantidio using Mantid more directly

78d9dd7

Update todo

644258d

Minor cleanup

cddd003

SimonHeybrock force-pushed the zoom branch from caaf05a to cddd003 Compare January 18, 2024 06:47

SimonHeybrock added 4 commits January 18, 2024 08:42

Refactor to read xml directly

0577a8d

Update docs

ae45bf9

Use Mantid if available, otherwise pooch

1cbcd01

Add Zoom page to docs

c4e0f02

SimonHeybrock requested a review from nvaytet January 18, 2024 10:10

SimonHeybrock marked this pull request as ready for review January 18, 2024 10:10

SimonHeybrock added 3 commits January 18, 2024 11:12

Unzip if required

a62f630

Remove now unused suffix

3be36c6

Fix pooch path handling

1c0706b

nvaytet reviewed Jan 18, 2024

View reviewed changes

SimonHeybrock added 4 commits January 22, 2024 05:12

Avoid markdown render problem

30ba0cf

Copyright date

a52783f

Fix file comment

db6cc19

Path -> FilePath

88ba28c

SimonHeybrock added 2 commits January 23, 2024 05:48

Extend file description

ec3d713

Add simply 2-D instrument view

a541769

nvaytet reviewed Jan 23, 2024

View reviewed changes

SimonHeybrock added 2 commits January 23, 2024 09:34

Move args to notebook

7f945d4

Improve function naming and docstring

818bf47

nvaytet approved these changes Jan 23, 2024

View reviewed changes

SimonHeybrock merged commit 394c5be into main Jan 23, 2024
3 checks passed

SimonHeybrock deleted the zoom branch January 23, 2024 09:36

SimonHeybrock mentioned this pull request Feb 5, 2024

fix: separate path and filename providers scipp/essreflectometry#25

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zoom data reduction #50

Zoom data reduction #50

SimonHeybrock commented Jan 16, 2024 •

edited

Loading

nvaytet Jan 18, 2024

SimonHeybrock Jan 22, 2024

nvaytet Jan 18, 2024

SimonHeybrock Jan 22, 2024

nvaytet Jan 18, 2024

nvaytet Jan 18, 2024

nvaytet Jan 18, 2024

nvaytet Jan 18, 2024

SimonHeybrock Jan 22, 2024

nvaytet Jan 22, 2024

SimonHeybrock Jan 23, 2024

nvaytet Jan 18, 2024

SimonHeybrock Jan 22, 2024

nvaytet Jan 18, 2024

SimonHeybrock Jan 22, 2024 •

edited

Loading

nvaytet Jan 18, 2024

SimonHeybrock Jan 22, 2024 •

edited

Loading

nvaytet Jan 23, 2024

SimonHeybrock Jan 23, 2024

SimonHeybrock Jan 23, 2024

nvaytet Jan 23, 2024

SimonHeybrock Jan 23, 2024

SimonHeybrock Jan 23, 2024

nvaytet Jan 23, 2024

nvaytet Jan 23, 2024

SimonHeybrock Jan 23, 2024

nvaytet Jan 23, 2024

nvaytet Jan 23, 2024

SimonHeybrock commented Jan 23, 2024

		" sans.isis.transmission_from_background_run,\n",
		" sans.isis.transmission_from_sample_run,\n",

		@@ -0,0 +1,66 @@
		# SPDX-License-Identifier: BSD-3-Clause
		# Copyright (c) 2023 Scipp contributors (https://github.com/scipp)

	# Copyright (c) 2023 Scipp contributors (https://github.com/scipp)
	# Copyright (c) 2024 Scipp contributors (https://github.com/scipp)

		DataFolder = NewType('DataFolder', str)


		class Path(sciline.Scope[FilenameType, str], str):

		return f'{path}/{filename}'


		def read_xml_detector_masking(filename: Path[PixelMaskFilename]) -> MaskedDetectorIDs:

		import scipp as sc


		def plot_instrument(da: sc.DataArray, pixels_per_tube: int = 512, **kwargs: Any) -> Any:

Zoom data reduction #50

Zoom data reduction #50

Conversation

SimonHeybrock commented Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimonHeybrock Jan 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimonHeybrock Jan 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimonHeybrock commented Jan 23, 2024

SimonHeybrock commented Jan 16, 2024 •

edited

Loading

SimonHeybrock Jan 22, 2024 •

edited

Loading

SimonHeybrock Jan 22, 2024 •

edited

Loading