Update HDF5 format compatibility

[Dave-Allured]

This is a proposal to update netCDF4/HDF5 file format controls within netCDF-C. The recent HDF5 release 1.10.2 in March 2018 provides new options for fine grained control of the HDF5 internal file format. HDF5 documentation hints at possible performance gains and bug avoidance.

The best format version for universal compatibility is the one created by the latest HDF5 1.8.* library version, currently 1.8.20. This is the traditional netCDF4 format that was made by all previous netCDF4 library versions, until HDF5 1.10.0 was released in March 2016. This is the so-called "superblock 2" format.

Objectives

1. Maintain full backward read and write format compatibility. Files created through this proposed change should be readable and writable by older software builds, specifically older versions of netCDF and HDF5 libraries.

2. When possible, avoid "earliest" HDF5 file format selection, loosely known as "superblock 0" format. This variation is unavoidable when using HDF5 library versions 1.10.0 and 1.10.1, due to lack of fine grained controls. However, this format variation is avoidable with both older (1.8.*) and newer (1.10.2 and later) HDF5 library versions.

History of HDF5 format control

Current and previous HDF5 versions provide file format control through this function, which takes arguments "low" and "high":

https://portal.hdfgroup.org/display/HDF5/H5P_SET_LIBVER_BOUNDS

All netCDF-C library versions 4.0 through 4.4.0, linked with HDF5 1.8.*, created the traditional "V1.8" or "superblock 2" format using these settings:

low  = H5F_LIBVER_LATEST
high = H5F_LIBVER_LATEST

With the advent of HDF5 1.10.0, more recent netCDF-C versions 4.4.1 through 4.6.1 used the following settings to preserve backward format compatibility. See #250 for the original discussion. This created "superblock 0" files:

low  = H5F_LIBVER_EARLIEST
high = H5F_LIBVER_LATEST

Proposed format control for new netCDF-C releases

Select format controls based on the detected HDF5 library version within the current netcdf-C library build.

All HDF5 1.8.* versions: Use the traditional settings to select "V1.8" file format:

low  = H5F_LIBVER_LATEST
high = H5F_LIBVER_LATEST

HDF5 1.10.0 and 1.10.1: These versions lack the new fine grained control. Select "earliest" as the only available backward compatible setting. This will create "superplock 0" files, and avoid making incompatible "superblock 3" files:

low  = H5F_LIBVER_EARLIEST
high = H5F_LIBVER_LATEST

HDF5 1.10.2 (new release) and later: Select the traditional "V1.8" or "superblock 2" format with this newly available configuration:

low  = H5F_LIBVER_V18
high = H5F_LIBVER_LATEST

Note: "high = H5F_LIBVER_LATEST" is retained in all cases. This setting allows the possible development of new HDF5 features such as SWMR, without additional complication of the format control code.

[edhartnett]

This makes sense.

[Dave-Allured]

For anyone interested, here is more detailed documentation on HDF5 format controls and the internal implementation. This came from the HDF help desk. The version on the HDF5 website is currently out of date.

RFC-bounds.v4.2018-jan-19.docx

[DennisHeimbigner]

I have recollection that when we used the original 1.10 mechanism,
which was feature based, we discovered that we never actually generated
any superblock versions other than 0 (zero).
Ward- do you remember this?

[WardF]

I do recall that that was the case, but that was with the original 1.10 version (as you mention). I do not recall if this was expected behavior or a bug. We should re-investigate.

[Dave-Allured]

Release announcement for HDF5 1.10.2, 2016 March 30:
https://forum.hdfgroup.org/t/release-of-hdf5-1-10-2-newsletter-160/4247

Full release notes:
https://portal.hdfgroup.org/display/support/HDF5+1.10.2

[Dave-Allured]

HDF5 format compatibility was documented in the first netCDF-C release announcement following HDF5-1.10.0. This was netCDF-C 4.4.1, release date 2016 June 28:
https://www.unidata.ucar.edu/blogs/news/entry/netcdf-4-4-1

[Dave-Allured]

Format version control is an HDF5 file access property, not a file creation property. This means that version settings are transient, not permanently embedded somehow in the file. In particular the superblock number is not a reliable indicator of internal HDF5 object versions, if a file was ever updated after initial creation.

https://portal.hdfgroup.org/display/HDF5/H5P_SET_LIBVER_BOUNDS

For this proposal to work as expected, format settings must be selected every time a file is re-opened for writing, not just during initial creation. Otherwise you can get a non-optimal mix of object versions within the same file. That is probably safe, but non-optimal.

In the current netcdf-C release 4.6.1, H5Pset_libver_bounds is called only during create, and not when a file is re-opened. Please add to this proposal, format control should be selected in both create and read/write modes. Apply the same three-way algorithm above, for both of these open modes.

[DennisHeimbigner]

This raises an interesting question. Can the superblock version number change
over time. You seem to be saying it can, which surprises me. Have you run a test
case to verify this?

[Dave-Allured]

@DennisHeimbigner, my understanding is that the superblock is written one time on file creation, and never again rewritten during normal read or write operations. There are a few dynamic fields within the superblock that get updated, such as end of file pointer and file lock status. But the superblock version number and some of the other fields appear to be permanent.

[Dave-Allured]

I am withdrawing this proposal. I will submit a new proposal to include an optional control for 1.10 format forward compatibility.

[Dave-Allured]

Re-opening. This part, update to v1.8 format compatibility, is good as it stands now. It would be better to get this part done first, then consider a 1.10 compatibility option later. 1.10 compatibility will be problematic.

[brtnfld]

@Dave-Allured @WardF @DennisHeimbigner

Was there anymore work done for this, i.e. a new proposal submitted? I need to introduce a mechanism for enabling the latest HDF5 version as an option. Currently, the latest is always set to 1.8:

#ifdef HAVE_H5PSET_LIBVER_BOUNDS
#if H5_VERSION_GE(1,10,2)
    if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_EARLIEST, H5F_LIBVER_V18) < 0)
#else
        if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_EARLIEST,
                                 H5F_LIBVER_LATEST) < 0)
#endif
            BAIL(NC_EHDFERR);
#endif

This will not work with the DAOS VOL connector we have developed, which is only able to produce 1.12 compatible versions. I can make it, if NetCDF is creating a file using the DAOS connector, to automatically use the latest HDF5 version. However, if the DAOS container is converted to native HDF5, it will not be readable by HDF5 1.8 or 1.10.

[WardF]

I need to review this issue so I may be repeating some things addressed above as I think out loud below; feel free to tell me that this is a non-issue (I'd welcome such news, frankly :)).

The issue we'll need to consider internally, before accepting this change, would be the impact resulting when files created this way are distributed. From a practical perspective, we want to avoid a situation where netCDF-4 files are being created and distributed (say, through the Unidata THREDDS server) which cannot be read, failing in a non-transparent way, on systems which are using an older libhdf5 install. The last time I tested this (which was a while ago and will bear re-testing, but it was right after the 1.10.0 release, which caught us off guard), libhdf5 1.8.x-based libnetcdf installs simply reported a generic HDF5 file error.

So as I return to speed on this issue, we will need to particularly focus on the potential impact of broad compatibility, both to avoid creating and distributing files which are indistinguishable from corrupted files, and to avoid the administrative/support overhead on the already-stretched Unidata staff and their support time.

Having stated that I'll start digging in to this, and will also ask @lesserwhirls and @dopplershift for comment.

[Dave-Allured]

@brtnfield, no, I never submitted a new proposal. What I outlined above should still be good for maximal forward and backward compatibility.

I think there is actually a mistake in the current implementation (4.7.4). This line which you quoted above:

   if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_EARLIEST, H5F_LIBVER_V18) < 0)

Should actually be:

   if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_V18, H5F_LIBVER_LATEST) < 0)

This would correspond with the third part of my suggestion above under "Proposed format control", for HDF5 1.10.2 and above.

There may be a reason for the current implementation, but I do not understand it. This current implementation, combined with newer HDF5 versions 1.10.2 and above, will create only netcdf-4 files with "v1.6 compatibility", represented as superblock zero. This suggested one-line change should create the desired netcdf-4 files with "v1.8 compatibility", represented as superblock 2.

Also, if the third argument is changed to the suggested H5F_LIBVER_LATEST, then advanced HDF5 features will be unblocked. This might achieve compatibility with your DAOS VOL connector, without needing further changes. Would you mind patching netcdf 4.7.4 with this one-line change, and see if this by itself solves your compatibility needs?

[edwardhartnett]

I think @Dave-Allured has a great suggestion above. Can you submit a PR with this one-line change?

There is not a great reason for the current implementation. I think Dave's makes more sense.

[gsjaardema]

The CGNS library has the following:

    /* Compatibility with V1.8 */
    H5Pset_libver_bounds(g_propfileopen,
#if H5_VERSION_GE(1,10,3)
          H5F_LIBVER_V18, H5F_LIBVER_V18);
#else
          H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);
#endif

If compatability with hdf5-1.8.X is desired and the hdf5 library is 1.10.3 or newer, then use H5F_LIBVER_V18 for both arguments. If the library is older or we don't need compatibility with hdf5-1.8.X then use H5F_LIBVER_LATEST for both arguments.

We also are looking at using VOL capabilities...

[brtnfld]

To be clear, the current behavior by specifying H5F_LIBVER_EARLIEST, H5F_LIBVER_V18 is the superblock will be v0 (v1.6) if no v1.8 format features are being used. If v1.8 format features are being used, which is possible with NetCDF, then the superblock will be v2.

The proposal is to use H5F_LIBVER_V18, H5F_LIBVER_LATEST, which will create, by default, v2 superblock. If a newer format feature is used, which requires a higher format (SWMR, VDS, new references), the lowest (H5F_LIBVER_V18) will be set to V110. The superblock will be v3.

CGNS users had an issue with readers built with hdf5 1.8 not reading the CGNS files (we had used H5Pset_file_space_strategy, a format upgrade), so CGNS is currently enforcing 1.8 compatibilities. This policy will need to be changed because it would be beneficial to use some of the new HDF5 features.

NetCDF users might have the same issue with readers not reading the netCDF files with this change. It should be fine unless NetCDF adds an API that always enforces a v3 superblock (like CGNS did); otherwise, it should, for the most part, always create a v2 superblock.

A good summary of the libverbounds behavior can be found at:

https://www.hdfgroup.org/2018/04/why-should-i-care-about-the-hdf5-1-10-2-release/rfc-setting-bounds-for-object-creation-in-hdf5-1-10-0-update/

[Dave-Allured]

@gsjaardema et al, I will go into the CGNS case because it is a good example for netcdf. The SECOND function arg (low =) is the important one for this part of the discussion. You said:

    /* Compatibility with V1.8 */
    H5Pset_libver_bounds(g_propfileopen,
#if H5_VERSION_GE(1,10,3)
          H5F_LIBVER_V18, H5F_LIBVER_V18);
#else
          H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);
#endif

My understanding is that this will result in v1.10 (superblock 3) files when used with HDF5 library versions 1.10.0, 1.10.1, and 1.10.2, in other words files that are NOT v1.8 compatible, therefore NOT universally readable. This is because the second arg "low = H5F_LIBVER_LATEST" takes effect for these particular cases, therefore v1.10 format is engaged.

If the CGNS project does not care about possible use of older HDF5 versions, then just mandate HDF5 1.10.3 or later, and there is no problem. If they DO care, as I do for netcdf, then I suggest a change along the lines of my original proposal above. At minimum, change the LOW arg in the ELSE clause as follows.

#if H5_VERSION_GE(1,10,3)
          H5F_LIBVER_V18, H5F_LIBVER_V18);
#else
          H5F_LIBVER_EARLIEST, H5F_LIBVER_LATEST);
#endif

This will result in v1.6 (superblock 0) files, which are universally readable, when used with older HDF5 library versions, specifically all 1.8.X versions, also 1.10.0 through 1.10.2.

Now I am no fan of unnecessary downgrading, so I suggest this more fine grained solution would be better. I think I will also invert the conditionals so that the code will progress sensibly from earliest to latest HDF5 versions. This corresponds to my original proposal above, except for the final "high = " arg which relates only to advanced features, to be discussed later. Notice also the tweak for lib version 1.10.2.

#if H5_VERSION_LT(1,10,0)
          H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);     /* all HDF5 1.8 lib versions */
#else
#if H5_VERSION_LE(1,10,1)
          H5F_LIBVER_EARLIEST, H5F_LIBVER_LATEST);    /* lib versions 1.10.0, 1.10.1 */
#else
          H5F_LIBVER_V18, H5F_LIBVER_V18);      /* lib versions 1.10.2 and higher */
#endif
#endif

This will result in the optimal v1.8 (superblock 2) files with ALL HDF5 library versions, older and newer, EXCEPT for library versions 1.10.0 and 1.10.1, which will create v1.6 (superblock 0) files. Both of these format versions are universally readable. Reminder, this middle exception occurs only because of a compatibility option that is missing in only those two library versions.

This part of the discussion deals only with basic file compatibility across library versions, and does not address advanced features. Do you agree with this analysis so far?

[DennisHeimbigner]

I think I need some clarification vis-a-vis netcdf backward compatibility.
At the moment, we are implicitly committed to superblock <= 2 so that netcdf
combined with hdf5 1.8.21 (our minimal recommended version) can read all
currently created files. This change appears to replace this commitment with one
in which netcdf can only read certain netcdf files (with superblock > 2) if the
netcdf library is built using some version of HDF5 with version greater than 1.8.21.
Is this correct?

[Dave-Allured]

@DennisHeimbigner, no, not correct. For normal netcdf operations, the governing of file format during create and write operations is the SECOND function arg, known as "low =" in HDF5 documentation.

When linked with all 1.8.X HDF5 lib versions, "low = H5F_LIBVER_LATEST" results in v1.8 (superblock 2) files.

When linked with lib versions 1.10.0 and 1.10.1, "low = H5F_LIBVER_EARLIEST" results in v1.6 (superblock 0) files. For optimal results, use "low = H5F_LIBVER_EARLIEST" only when required to link with one of these now-obsolescent lib versions.

When linked with lib versions 1.10.2 and greater, "low = H5F_LIBVER_V18" results in v1.8 (superblock 2) files.

All three together, via these conditionals, should always result in the superblock <= 2 condition that you want for normal netcdf creates and writes.

The third arg "high =" governs format only when some discrete operation invokes an advanced HDF5 feature, such as SWMR. My limited understanding of the netcdf core library is that such advanced features are never invoked in normal operations. This is something that would only be done, and very explicitly, in support of a new capability, such as the DAOS VOL connector discussed above.

All this is subject to oversight and testing, to be sure. I have not lately run any advanced test cases for this.

[brtnfld]

... if the netcdf library is built using some version of HDF5 with version greater than 1.8.21.

The minimum superblock value for earliest is not dependent on what version of hdf5 is used; it is possible that a user can build netCDF with HDF5 1.12 and have a file with superblock v0. It only depends on what APIs are called. For example, if NetCDF calls H5Pset_attr_creation_order, that would force the superblock to be v2.

Currently, if NetCDF had the upper bound to be lastest, it will never have a superblock > v2 since it does not use any feature requiring greater than that, independent of the HDF5 version. The commitment to <=2 is still valid with these changes; it will only be an issue in the future if NetCDF starts to use v3 dependent APIs. It is possible for a user to not have version > 2 assuming they don't use any new NetCDF feature that needs v3 superblock.

[brtnfld]

@Dave-Allured I guess I don't understand why low is set to H5F_LIBVER_EARLIEST for some cases and not for others; why do you want to allow for v0 with a certain version of HDF5 and not others? That would change the current behavior of what NetCDF allows now.

I could see doing that if Netcdf finds that v2 should always be used because it performs better when used with a certain version of the HDF5 library.

[Dave-Allured]

@brtnfld, the short answer is that the function H5Pset_libver_bounds and its enum parameters have changed several times over the history of HDF5. You can find traces of this in the function doc page and the release notes. There are a lot of gory details if you dive into the full history.

The proposed three-way conditional represents the most appropriate method to coax a "universally readable" file format (superblock <= 2), at each of three different stages in the API evolution.

H5F_LIBVER_EARLIEST is necessary only with HDF5 library versions 1.10.0 and 1.10.1, because unfortunately v1.8 format compatibility was neglected in those versions. This meant that netcdf had to resort to a downgrade from v1.8 to v1.6 compatility (superblock 0) following the 1.10.0 HDF5 release in 2016 April. See #250 for history.

The desired symbol H5F_LIBVER_V18 was added in HDF5 library version 1.10.2. This remains the optimal choice for the "low =" parameter through the remainder of 1.10.X, and into 1.12.X library versions.

Unfortunately I never got around to following up on restoring low = H5F_LIBVER_V18, which is why netcdf has been stuck on v1.6 compatibility (superblock 0), until now.

[edwardhartnett]

Are we done here? Or is there more work to be done?

[brtnfld]

I'm done with this issue.

[Dave-Allured]

Ignore that first commit. I wish I understood how to back out mistakes more gracefully than this. The PR is good.