HDF5 NDAttribute datasets to support "extra" dimensions like detector datasets

[ulrikpedersen]

The HDF5 writer allow writing detector (NDArray) data into HDF5 datasets with up to 3 "extra" or "virtual" dimensions (in addition to the NDArray dimensions). However, datasets with NDAttribute data are currently just organised in long linear 1D arrays with 1 entry per stored NDArray. Our users (scientific software developers working with data post-processing) would like to have the NDAttribute datasets organised with the same "extra" dimensions as the detector datasets.

Alan Greer is working on this at the moment and has highlighted a tricky issue: how to deal with (i.e. where to store the NDAttribute value) in the case where the incoming NDArrays can go to different detector datasets? We recently added this support so that NDArrays can be re-directed to different datasets - in principle to support writing dark or flatfield images separately - but it could also be used to support storing multiple scans into a single HDF5 file.

Anyone with interest in this, please feel free to pitch in with ideas on how to solve this. I would think that includes @ajgdls, @aglowacki and @prjemian at least... 😃

Note: please don't mistake this ticket for an attempt to support multi-dimensional data in NDAttributes. This ticket/task only relates to how the scalar NDAttribute data values are stored in HDF5 datasets!

[ajgdls]

Myself and @ulrikpedersen have discussed this and have come up with the following idea which we believe will provide the behaviour expected.

A parameter will be added that offers the ability for attribute datasets to follow the real dataset dimensionality. The default value of the parameter will be interpreted as retaining the current behaviour, and in this case attribute datasets will continue to be stored as 1D arrays with 1 entry per NDArray. If the parameter is set for following dimensionality then the attribute datasets will be saved with the same dimensionality as the specified default dataset, in the same order and any NDArrays that are redirected to the non-default datasets will lose their corresponding attribute data items as there would be no-where to store them.

Are there any comments on this proposed behaviour?

[ulrikpedersen]

@ajgdls and I think this was actually implemented quite a while ago.

As per the documentation:

asyn interface	Access	Description	drvInfo string	EPICS record name	EPICS record type
asynInt32	r/w	Turn on or off NDAttribute dataset dimensions (1 = On, 0 = Off). When switched on NDAttribute datasets are forced to have the same dimensionality as the main dataset.	HDF5_dimAttDatasets	$(P)$(R)DimAttDatasets, $(P)$(R)DimAttDatasets_RBV	bo, bi

Closing the ticket.