New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge attributes of key group and value dataset for CONTROL keys #498
Conversation
Fixes #495 |
extra_data/keydata.py
Outdated
|
||
if self.is_control and self.key.endswith('.value'): | ||
# For CONTROL sources, most of the attributes are saved on | ||
# the parent group rather than the .value dataset. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's also mention here that at least currently, some overlapping attributes differ, and in that case the group level attribute seems to be the one that's right.
(I have a nasty foreboding that some day we'll come across a case where the group attribute is wrong and the dataset attribute is right, and we'll have to figure out how to wrangle that. But hopefully that's just pessimism. 🤞 )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(I have a nasty foreboding that some day we'll come across a case where the group attribute is wrong and the dataset attribute is right, and we'll have to figure out how to wrangle that. But hopefully that's just pessimism. 🤞 )
I don't think that's a possibility.
The attributes come from the device schema. The HDF5 datasets .value
and .timestamp
don't exist on the device. The parent group of those datasets, is actually (on the device) the property leaf that holds the value and attributes on the device we're saving.
So the attributes on these dataset must be copied from the parent group (or have some default). I guess the wrong default could as well be on the parent group, but I don't see how this could be wrong on the parent group and correct on the dataset.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, added that clarification.
In general though, I wouldn't worry much. I'll get in touch with the DAQ team and asked them to verify our assumptions here.
We may even be able to remove the dataset attributes entirely again and shift EXtra-data to the ones on the group. Frankly I doubt users would notice.
Other than that, LGTM. 👍 |
f057e96
to
a736812
Compare
Thanks, LGTM! |
Thanks for review. |
Unfortunately I noticed that the HDF attributes mapping to Karabo properties at the time of recording are a bit all over the place in the case of
CONTROL
keys. For example,SA3_XTD10_XGM/XGM/DOOCS.beamposition.ixPos
for a recent run:After scanning through attributes across a large number of sources, I found:
minSize
, not shown here), the keys onvalue
are a true subset of the parentvalue
either have the same value (e.g.alias
above) or a wrong, likely constant value (e.g.daqPolicy
)timestamp
always has exactly the four attributes with exactly those values as shown aboveBased on that, this PR merges the attributes on the parent group into those returned from
KeyData.attributes()
for thevalue
dataset of a key, specifically overwriting any existing key.