Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save padding/offset of fields in compound types #273

Closed
mattjala opened this issue Oct 12, 2023 · 3 comments
Closed

Save padding/offset of fields in compound types #273

mattjala opened this issue Oct 12, 2023 · 3 comments
Assignees

Comments

@mattjala
Copy link
Contributor

Currently, HSDS removes all bytes of padding from compound types. This can lead to a re-opened datatype having a different total size, and different arrangement in memory than the type that a user originally wrote.

In order for HSDS to preserve the offsets of fields within compound types, it will probably need to accept an optional "offset" field for each field in a compound type, return this information as part of datatype reads, and then the client will need to use those (optional) offsets to re-assemble the datatype.

@jreadey
Copy link
Member

jreadey commented Oct 19, 2023

In numpy there's no way it seems to specify the exact offsets of the fields. There's an alignment option: https://numpy.org/doc/stable/reference/generated/numpy.dtype.html, that can be used to flip between packed and compiler specified alignment. That might be the best we can do for now (though there could be potential issues sharing data between machines with different architectures)

@jreadey
Copy link
Member

jreadey commented Oct 27, 2023

Another alternative would be for clients that care about padding (C specifically) to add some extra fixed char elements as needed to adjust the alignment.

@mattjala
Copy link
Contributor Author

This should be handled on the client's side with padding - see HDFGroup/vol-rest#91

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants