Dexela support for live data page #51

jilavsky · 2021-06-04T00:45:42Z

At this time Dexela detector data are not reduced and posted to live data graph. It is using same Nexus as Pilatus, but there are small differences. Sounds like code needs to be modified to handle also Dexela.

jilavsky · 2021-12-13T17:37:25Z

Recent test proven usefulness of this detector and future Linkam experiments from some users should be using it for regular work. Users expressed need for the web support with this detector. We have plenty of data now available.

prjemian · 2021-12-13T17:46:33Z

PV prefix for the Dexela?

jilavsky · 2021-12-13T17:55:03Z

9idcDEX, it is off line now. It was used until Sunday morning only, Sunday users needed Pilatus again.

jilavsky · 2021-12-13T17:57:28Z

Test data: USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs, any image there. Same metadata as Pilatus, should be proper Nexus file.

prjemian · 2021-12-14T00:16:04Z

@jilavsky Is the Dexela image transposed (as the Pilatus detectors)?

livedata/reduceAreaDetector.py

Lines 46 to 48 in 25113a7

    
           # image is transposed, consider that here 
        
           'y_image_center_pixels' : '/entry/instrument/detector/beam_center_x', 
        
           'x_image_center_pixels' : '/entry/instrument/detector/beam_center_y',

prjemian · 2021-12-14T00:29:29Z

Need to have this set as in the other detectors (it is empty in the Dexela's HDF5 files):

        x_pixel_size:NX_FLOAT64[] = 
        y_pixel_size:NX_FLOAT64[] =

jilavsky · 2021-12-14T17:43:02Z

That is confusing, I checked random file with HDFView and it has proper value there (0.0748mm). q It should, this is in layout:

  <dataset name="x_pixel_size"   value="0.0748"  source="constant" type="float"/>
  <dataset name="y_pixel_size"   value="0.0748"  source="constant" type="float"/>

Check again, please and let me know where the problem is.

prjemian · 2021-12-14T19:52:09Z

What is the HDF5 address to that value you saw in HDFView?

prjemian · 2021-12-14T19:53:18Z

These lines define the configuration that will be used:

        'x_pixel_size_mm'       : '/entry/instrument/detector/x_pixel_size',
        'y_pixel_size_mm'       : '/entry/instrument/detector/y_pixel_size',

prjemian · 2021-12-14T21:20:11Z

Aha! That information is smoewhere else:

bash-4.2$ punx tree 2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ti64Dex_975C_244min_000437.hdf | grep -i pixel
      waxs_ccd_center_x_pixel:NX_FLOAT64 = 83.18774884799959
        @NDAttrName = waxs_ccd_center_x_pixel
        @NDAttrDescription = horizontal position of beam center on CCD, pixels
      waxs_ccd_center_y_pixel:NX_FLOAT64 = 1539.139546368824
        @NDAttrName = waxs_ccd_center_y_pixel
        @NDAttrDescription = vertical position of beam center on CCD, pixels
      waxs_ccd_pixel_size_x:NX_FLOAT64 = 0.0748
        @NDAttrName = waxs_ccd_pixel_size_x
        @NDAttrDescription = CCD pixel size, horizontal, mm
      waxs_ccd_pixel_size_y:NX_FLOAT64 = 0.0748
        @NDAttrName = waxs_ccd_pixel_size_y
        @NDAttrDescription = CCD pixel size, vertical, mm
          @NDAttrName = waxs_ccd_center_x_pixel
          @NDAttrDescription = horizontal position of beam center on CCD, pixels
          @NDAttrName = waxs_ccd_center_y_pixel
          @NDAttrDescription = vertical position of beam center on CCD, pixels
        x_pixel_size:NX_FLOAT64[] = 
        y_pixel_size:NX_FLOAT64[] =

Using h5dump, we can see that links are missing in this file:

bash-4.2$ h5dump -n 2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ti64Dex_975C_244min_000437.hdf | grep pixel
 dataset    /entry/Metadata/waxs_ccd_center_x_pixel
 dataset    /entry/Metadata/waxs_ccd_center_y_pixel
 dataset    /entry/Metadata/waxs_ccd_pixel_size_x
 dataset    /entry/Metadata/waxs_ccd_pixel_size_y
 dataset    /entry/instrument/detector/beam_center_x -> /entry/Metadata/waxs_ccd_center_x_pixel
 dataset    /entry/instrument/detector/beam_center_y -> /entry/Metadata/waxs_ccd_center_y_pixel
 dataset    /entry/instrument/detector/x_pixel_size
 dataset    /entry/instrument/detector/y_pixel_size

prjemian · 2021-12-14T21:24:18Z

Looking at how the EPICS AreaDetector configures these files:

bash-4.2$ pwd
/share1/AreaDetectorConfig
bash-4.2$ git grep -i link | grep -vi linkam | grep pixel
Old/SAXS/layout_saxs.xml:        <hardlink name="beam_center_x" target="/entry/Metadata/pin_ccd_center_x_pixel"/>
Old/SAXS/layout_saxs.xml:        <hardlink name="beam_center_y" target="/entry/Metadata/pin_ccd_center_y_pixel"/>
Old/SAXS/layout_saxs.xml:        <hardlink name="x_pixel_size" target="/entry/Metadata/pin_ccd_pixel_size_x"/>
Old/SAXS/layout_saxs.xml:        <hardlink name="y_pixel_size" target="/entry/Metadata/pin_ccd_pixel_size_y"/>
SAXS_config/hdf5_plugin/layout.xml:        <hardlink name="beam_center_x" target="/entry/Metadata/pin_ccd_center_x_pixel"/>
SAXS_config/hdf5_plugin/layout.xml:        <hardlink name="beam_center_y" target="/entry/Metadata/pin_ccd_center_y_pixel"/>
SAXS_config/hdf5_plugin/layout.xml:        <hardlink name="x_pixel_size" target="/entry/Metadata/pin_ccd_pixel_size_x"/>
SAXS_config/hdf5_plugin/layout.xml:        <hardlink name="y_pixel_size" target="/entry/Metadata/pin_ccd_pixel_size_y"/>
WAXS_config/hdf5_plugin/layout_waxs.xml:        <hardlink name="beam_center_x" target="/entry/Metadata/waxs_ccd_center_x_pixel"/>
WAXS_config/hdf5_plugin/layout_waxs.xml:        <hardlink name="beam_center_y" target="/entry/Metadata/waxs_ccd_center_y_pixel"/>
WAXS_config/hdf5_plugin/layout_waxs.xml:        <hardlink name="x_pixel_size" target="/entry/Metadata/waxs_ccd_pixel_size_x"/>
WAXS_config/hdf5_plugin/layout_waxs.xml:        <hardlink name="y_pixel_size" target="/entry/Metadata/waxs_ccd_pixel_size_y"/>

no such configuration for the Dexela detector.

bash-4.2$ git grep hardlink | grep -i dexela
bash-4.2$

prjemian · 2021-12-14T21:37:48Z

Just this:

bash-4.2$ grep -i pixel Dexela/layout_waxs.xml 
        <hardlink name="beam_center_x" target="/entry/Metadata/waxs_ccd_center_x_pixel"/>
        <hardlink name="beam_center_y" target="/entry/Metadata/waxs_ccd_center_y_pixel"/>
        <dataset name="x_pixel_size"   value="0.0748"  source="constant" type="float"/>
        <dataset name="y_pixel_size"   value="0.0748"  source="constant" type="float"/>
bash-4.2$

h5dump's view:

            DATASET "x_pixel_size" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SCALAR
               DATA {
               (0): 0.0748
               }
            }
            DATASET "y_pixel_size" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SCALAR
               DATA {
               (0): 0.0748
               }
            }

punx's view:

        x_pixel_size:NX_FLOAT64[] = 
        y_pixel_size:NX_FLOAT64[] =

h5py's view

In [1]: import h5py

In [2]: f = h5py.File("2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ti64Dex_975C_244min_000437.hdf", "r")

In [3]: g = f["/entry/instrument/detector"]

In [4]: "x_pixel_size" in g
Out[4]: True

In [5]: g["x_pixel_size"]
Out[5]: <HDF5 dataset "x_pixel_size": shape (), type "<f8">

In [6]: g["y_pixel_size"]
Out[6]: <HDF5 dataset "y_pixel_size": shape (), type "<f8">

prjemian · 2021-12-14T21:39:22Z

In summary, the declaration of a constant float in the layout.xml file is not working. These two are the only such declarations in that file, all the other constants are string.

bash-4.2$ grep constant ./Dexela/layout_waxs.xml | grep float
        <dataset name="x_pixel_size"   value="0.0748"  source="constant" type="float"/>
        <dataset name="y_pixel_size"   value="0.0748"  source="constant" type="float"/>
bash-4.2$ grep constant ./Dexela/layout_waxs.xml | head -5
    <attribute name="NX_class"      value="NXentry" source="constant" type="string"/>
    <attribute name="default"       value="data"    source="constant" type="string"/>
    <dataset  name="definition"     value="NXsas" source="constant" type="string"/>
    <dataset  name="program_name"   value="EPICS areaDetector" source="constant" type="string"/>
      <attribute name="NX_class" value="NXmonitor" source="constant" type="string"/>

prjemian · 2021-12-14T21:56:30Z

Investigating further, these are written to the HDF5 file as scalar values rather than arrays (like absolutely all the other numerical data in NeXus HDF5 files ever is written - assumed to be some sort of rule).

Now, we can see some sort of difference here:

In [16]: g["beam_center_x"]
Out[16]: <HDF5 dataset "beam_center_x": shape (1,), type "<f8">

In [17]: g["x_pixel_size"]
Out[17]: <HDF5 dataset "x_pixel_size": shape (), type "<f8">

In [18]: g["beam_center_x"].dtype
Out[18]: dtype('float64')

In [19]: g["x_pixel_size"].dtype
Out[19]: dtype('float64')

In [20]: g["beam_center_x"].shape
Out[20]: (1,)

In [21]: g["x_pixel_size"].shape
Out[21]: ()

which means a new rule to access scalar values in HDF5 files:

if dataset.shape == (1, ):  # historical representation of scalar value
    value = dataset[0]
elif  dataset.shape == ():  # scalar representation
    value = dataset.value
else:  # array of some sort
    value = numpy.array(dataset)

prjemian · 2021-12-14T21:59:03Z

Moving ahead as if the HDF5 files (and associated XML files used to create them) do not need to be changed, the livedata/reduceAreaDetector.py will be changed for this new development.

prjemian · 2021-12-14T23:17:59Z

@jilavsky - the livedata/reduceAreaDetector.py code runs without failing on the USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ti64Dex_975C_99min_000399.hdf file.

prjemian · 2021-12-14T23:18:45Z

I'm trying to force the Dexela data from USAXS_data/2021-12/12_10_DexHeater.dat to plot.

prjemian · 2021-12-15T00:08:58Z

Plots are not getting updated because of too many digits in when AD writes the file name:

bash-4.2$ ll /share1/USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ag*
-rw-rw-r-- 1 usaxs 24519664 Dec 10 17:22 /share1/USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/AgBehenateLaB6_000026.hdf
-rw-rw-r-- 1 usaxs    37497 Dec 10 17:21 /share1/USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/AgBehenateLaB6_0026.jpg
bash-4.2$ ll '/share1/USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/AgBehenateLaB6_0026.hdf'

SPEC writes the name of the HDF5 file on the command line. With %04d format for the USAXS data number.
The JPEG detector also writes its image file with %04d format.
It is clear the Dexela IOC writes its image file with %06d format resulting in a file not found and no plot of the Dexela image.

prjemian · 2021-12-15T00:19:57Z

From the Dexela's bluesky setup:

    det.hdf1.stage_sigs["file_template"] = "%s%s_%6.6d.h5"

When the file name is written by SPEC on the command line, it should agree with the formatting of 9idcDEX:HDF1:FileTemplate PV.

jilavsky · 2021-12-15T15:25:03Z

I fixed spec to use 6 digit formatting. This was oversight.
With template, looks like I messed up something. I thought I did minimum changes from Pilatus configuration. Dexela configuration are attributes.xml and layout.xml in AreaDetectorConfig/Dexela on share1. If these need to be fixed, work with me.

prjemian · 2021-12-15T15:41:28Z

Try it with just the change to the FileTemplate PV. Leave the XML files as-is for now. If we really want to plot the Dexela WAXS data from Dec 10, we can either edit the SPEC file (my choice) or make soft links to the correct image files.

…

On Wed, Dec 15, 2021, 9:25 AM Jan Ilavsky ***@***.***> wrote: I fixed spec to use 6 digit formatting. This was oversight. With template, looks like I messed up something. I thought I did minimum changes from Pilatus configuration. Dexela configuration are attributes.xml and layout.xml in AreaDetectorConfig/Dexela on share1. If these need to be fixed, work with me. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMUMCNKBKEFZUTVY2AJRTURCXNVANCNFSM46BY65EQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

jilavsky · 2021-12-16T16:19:27Z

We will not run Dexela in December again. I edited spec to make sure it works next time = February.

prjemian · 2021-12-16T16:52:26Z

Can you test it after the run ends to verify it works as expected?

jilavsky added the enhancement New feature or request label Jun 4, 2021

jilavsky assigned prjemian Jun 4, 2021

jilavsky added this to the 2022-01 milestone Dec 13, 2021

prjemian pushed a commit that referenced this issue Dec 14, 2021

ENH #51 describe Dexela N2315

75197b1

prjemian pushed a commit that referenced this issue Dec 14, 2021

DBG #51

18cedda

prjemian pushed a commit that referenced this issue Dec 14, 2021

ENH #51 new scalar data access rule

960ef98

prjemian mentioned this issue Dec 14, 2021

getting the value of a scalar dataset prjemian/punx#172

Open

prjemian closed this as completed Dec 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dexela support for live data page #51

Dexela support for live data page #51

jilavsky commented Jun 4, 2021

jilavsky commented Dec 13, 2021

prjemian commented Dec 13, 2021

jilavsky commented Dec 13, 2021

jilavsky commented Dec 13, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

jilavsky commented Dec 14, 2021 •

edited

Loading

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021 •

edited

Loading

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 15, 2021

prjemian commented Dec 15, 2021

jilavsky commented Dec 15, 2021

prjemian commented Dec 15, 2021 via email

jilavsky commented Dec 16, 2021

prjemian commented Dec 16, 2021

Dexela support for live data page #51

Dexela support for live data page #51

Comments

jilavsky commented Jun 4, 2021

jilavsky commented Dec 13, 2021

prjemian commented Dec 13, 2021

jilavsky commented Dec 13, 2021

jilavsky commented Dec 13, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

jilavsky commented Dec 14, 2021 • edited Loading

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021 • edited Loading

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 14, 2021

prjemian commented Dec 15, 2021

prjemian commented Dec 15, 2021

jilavsky commented Dec 15, 2021

prjemian commented Dec 15, 2021 via email

jilavsky commented Dec 16, 2021

prjemian commented Dec 16, 2021

jilavsky commented Dec 14, 2021 •

edited

Loading

prjemian commented Dec 14, 2021 •

edited

Loading