Basic dream workflow #44

jl-wynen · 2024-03-12T15:26:55Z

Fixes #35

This obviously needs a lot of work. But it is meant as a skeleton to add the rest of the milestone.

The data is currently private. I'm waiting for an ok from the instrument team before publishing it.

SimonHeybrock · 2024-03-14T08:35:56Z

src/ess/dream/beamline.py

+def flatten_detector_dimensions(
+    data: RawDetectorData[RunType],
+) -> FlatDetectorData[RunType]:
+    """Flatten logical detector dimensions to a single ``spectrum``.


Isn't RawDetectorData (from ESSreduce) flat already?

No, it's shaped like the logical dims. At least this is what the GEANT4 loader produces. Would you rather we load a flat array and have a separate step for folding afterwards?

My question is why you flatten it. We want to fold the flat version we get from ESSreduce, that is what also ESSsans will do.

Removed flattening

SimonHeybrock · 2024-03-14T08:37:54Z

src/ess/dream/beamline.py

+        Flattened detector data with a ``spectrum`` dimension.
+    """
+    logical_dims = {'module', 'segment', 'counter', 'wire', 'strip', 'sector'}
+    actual_dims = tuple(dim for dim in data.dims if dim in logical_dims)
+    return FlatDetectorData[RunType](data.flatten(dims=actual_dims, to='spectrum'))


Why do we even want to flatten?

See also https://scipp.github.io/essreduce/user-guide/reduction-workflow-guidelines.html#s-10-detector-banks-shall-be-loaded-with-their-logical-dimensions-if-possible.

Some providers didn't work. In particular, the focussing step needs a spectrum dim. Should I make that instrument-specific so that it can use the logical dimensions?

Also, isn't there a massive performance penalty when binning in an additional dim like tof?

The steps should be fixed to work with any dims. If this is too much work, we should look into it.

SimonHeybrock · 2024-03-14T08:43:22Z

src/ess/powder/conversion.py

+def _to_dspacing_impl(
+    data: sc.DataArray, base_graph: dict[str, Callable[..., Any]]
+) -> sc.DataArray:
+    out = _restore_tof_if_in_wavelength(data)


Seems to contradict the "Attention" section in the docstring above?

No. _restore_tof_if_in_wavelength is a bit funky, but it removes the wavelength and makes tof aligned. The latter part is no longer needed, this code is a leftover from when we used attrs. So I guess we can just delete the wavelength coord instead.

SimonHeybrock · 2024-03-14T08:44:33Z

src/ess/powder/conversion.py


-    out = out.transform_coords('dspacing', graph=graph, keep_intermediate=False)
+    out = data.transform_coords('dspacing', graph=graph, keep_intermediate=False)
    out.coords.pop('_tag_positions_consumed', None)


What exactly do you ensure with this mechanism? That a conversion actually happens?

It ensures that positions in the output are unaligned. The conversion with calibration doesn't use positions directly but conceptually, d is derived from positions.

Can you explain this in a comment please?

@jl-wynen Has this been addressed somewhere? Can't see in the diff.

Done now. I actually simplified the conversion with positions because it doesn't need this mechanism.

SimonHeybrock · 2024-03-14T09:36:22Z

src/ess/powder/types.py

+class FlatDetectorData(sciline.Scope[RunType, sc.DataArray], sc.DataArray):
+    """Data (events / histogram) of a RawDetector flattened to ``detector_number``."""


See above, not sure why this is needed.

SimonHeybrock · 2024-03-18T04:57:54Z

src/ess/dream/beamline.py

+    base = ('module', 'segment', 'counter', 'wire', 'strip')
+    if detector_name == DetectorName.high_resolution:
+        return DetectorDimensions(base + ('sector',))


Can you double check if this is actually correct? I remember that at least one detector was not prepresentable with these base dims.

I have not encountered this. But I only have simulated data. Do you have a NeXus file where it doesn't work? If so, we need to update the csv loader.

See https://github.com/scipp/essdiffraction/blob/main/src/ess/dream/io/nexus.py.

The endcap detectors have a irregular structure that cannot be fully folded.
This is not a problem but note again that the counter may be reversed. It is
currently not clear if this is a bug in the file.

What do I do with this information? Do I just not fold the endcaps? Or do I only fold part of them, which part?
Or Do we just wait because it seems to be unclear whether real files will be like this?

I do not think it is unclear. Real file are (and will be) like this. It is impossible to (fully) fold. I guess we should do what is done in the linked file?

Comparing geant4 and ICD names and sizes (https://confluence.esss.lu.se/pages/viewpage.action?pageId=462000005), I don't think we can really represent both simulated and measured data in the same way. Some names are different as well as some sizes. And out test data doesn't even conform to the specs on the confluence page (e.g., the mantle has 7 modules where the pages says there should be 5).

So I would say we just live with the discrepancy. I can change the provider you commented on to check which dims exist in the data and return either the current set or the dims produced by the nexus loader. Got any better idea?

I am fine with any discrepancy, provided that this code is ensured to be Geant-4 specific.

SimonHeybrock · 2024-03-18T04:59:11Z

src/ess/dream/data.py

+    """Path to a GEANT4 CSV file for a diamond sample.
+
+    **Sample**:
+
+    - Diamond powder
+    - radius=0.0059 m
+    - yheight=0.05-1e-9 m
+    - material
+        reflections of powder from "C_Diamond.laz" (McStas file)
+        incoherent process with sigma=0.001 barns, packing_factor=1,
+        unit cell volume=45.39 angstrom^3
+        absorption of 0.062 1/m
+
+    **Container**:
+
+    - radius=0.006 m
+    - yheight=0.05 m
+    - material
+        reflections of powder from "V.laz" (McStas file)
+        incoherent process with sigma=4.935 barns, packing factor=1,
+        unit cell volume=27.66 angstrom^3
+        absorption of 36.73 1/m


Great idea to include this info in the docstrings, this makes the data module much more useful.

SimonHeybrock · 2024-03-18T13:45:55Z

src/ess/dream/beamline.py

+        'counter',
+        'strip',
+        'sector',
+        'sumo_cass_cst',


Typo? See

essdiffraction/src/ess/dream/io/nexus.py

Line 99 in 0a375c1

'sumo_cass_ctr': -1, # sumo*cassette*counter, irregular, cannot fold

jl-wynen · 2024-03-20T08:30:29Z

Any more comments?

jl-wynen changed the base branch from main to dream-instrument-view March 12, 2024 15:42

jl-wynen mentioned this pull request Mar 14, 2024

Configurable variance handling in broadcasting #37

Closed

SimonHeybrock reviewed Mar 14, 2024

View reviewed changes

Base automatically changed from dream-instrument-view to main March 14, 2024 11:53

jl-wynen force-pushed the basic-dream-workflow branch from a0a3466 to 1acddf9 Compare March 14, 2024 16:04

jl-wynen added 16 commits March 15, 2024 09:26

Turn GEANT4 loader into a provider

de1770f

Wrap geant4 data in data group

6c1da42

Depend on essreduce

0e3dc41

Add extract_geant4_detector_data

1b3ad51

Test geant4 providers

ec0cb59

Add to_dspacing_with_positions

6759382

Add flatten_detector_dimensions

070b19d

Support binned data in crop_tof

e828bd5

Add broadcast_uncertainties

4cd5062

Update POWGEN workflow

479454a

Add dream data reduction notebook

4b7dca0

Fix instrument view notebook

d2b06e1

Save reduced data

ae31360

Simpler _restore_tof_if_in_wavelength

11d1b8a

Add simulated data to dream.data

0117f3a

Keep logical detector dims

2d97220

jl-wynen force-pushed the basic-dream-workflow branch from 1acddf9 to 2d97220 Compare March 15, 2024 08:27

Fix test

dd03be1

SimonHeybrock reviewed Mar 18, 2024

View reviewed changes

jl-wynen added 2 commits March 18, 2024 15:01

Support logical dims from NeXus

414c073

Match dim order of data

a5415e1

jl-wynen force-pushed the basic-dream-workflow branch from 5151940 to a5415e1 Compare March 18, 2024 14:54

Explain _tag_positions_consumed

e8e0dd3

SimonHeybrock approved these changes Mar 20, 2024

View reviewed changes

Merge branch 'main' into basic-dream-workflow

1fdfe2d

jl-wynen enabled auto-merge March 20, 2024 13:01

jl-wynen merged commit 465ecfa into main Mar 20, 2024

jl-wynen deleted the basic-dream-workflow branch March 20, 2024 13:08

jl-wynen mentioned this pull request Mar 20, 2024

[Requirement] Remove low resolution ToF #23

Closed

		class FlatDetectorData(sciline.Scope[RunType, sc.DataArray], sc.DataArray):
		"""Data (events / histogram) of a RawDetector flattened to ``detector_number``."""

Basic dream workflow #44

Basic dream workflow #44

Uh oh!

Conversation

jl-wynen commented Mar 12, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SimonHeybrock Mar 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jl-wynen commented Mar 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SimonHeybrock Mar 18, 2024 •

edited

Loading