Fix check for swmr compatibility in h5 files#368
Open
pblowey wants to merge 2 commits into
Open
Conversation
Previous check of SWMR compatibility was trying to open the file with old HDF5 library versions, to see if an error was raised. With HDF5>2.0 this no longer works because a fix was implemented to only raise an error when trying to create or write new objects that are incompatible rather than when opening the file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The validation service checks files to ensure that the h5 files are not compatible with HDF5 versions <= 1.8, with the intention of ensuring that files are written in a SWMR compatible file format (SWMR was introduced in HDF5 1.10). The check (added in #25) operated by trying to open the file using library version bounds of ("earliest", "v108") and would determine a file to be incompatible if an
OSErrorwas returned.With the release of hdf5 library versions > 2.0, which the dials build now uses, the behaviour of
with h5py.File(filename, "r", libver=("earliest", "v108")):has changed and will now successfully open files even if they contain/use objects/features incompatible with the older libraries, including swmr. This is because the intention of the libver bounds is to ensure library compatibility of new objects created thereafter and is not supposed to affect initial file access, but this was not implemented correctly in hdf5 < 2.0 (see the discussion here for details).This PR fixes the check by instead finding the super block version of the hdf5 file. SWMR requires a superblock version >= 3, so this is used to determine whether the file is swmr compatible. The check does not establish whether the file is SWMR mode is enabled in the file.
This PR also adds a test for the SWMR compatibility check.