Windows builds should use HDF5 1.10.1

[bks]

The Windows binaries (as installed with pip) are still distributing HDF5 version 1.8.18, which does not support SWMR. Please upgrade the build environment and distribution to provide version 1.10.1.

[tacaswell]

This is at the discretion of @cgohlke .

[cgohlke]

I'm not sure that should be at my discretion. Isn't it true that "HDF5-1.8 may not be able to read HDF5-1.10 files"? Maybe this should be coordinated with binaries for other platforms, pytables and conda folks?

I put binaries for testing at http://www.lfd.uci.edu/~gohlke/pythonlibs/#h5py. The vc9 binaries for Python 2.7 crash during the tests.

[bks]

Quoting from the HDF5 documentation:

If an application built on HDF5 Release 1.10 avoids use of the new features and does not request use of the latest format, applications built on HDF5 Release 1.8.x will be able to read files the first application created. In addition, applications originally written for use with HDF5 Release 1.8.x can be linked against a suitably configured HDF5 Release 1.10.x library, thus taking advantage of performance improvements in 1.10.

So if you set libver='earliest', you are guaranteed compatibility with 1.8.x. If you set libver='latest' and blindly enable new features then the 1.10 format will not be readable by 1.8.x without conversion.

[cgohlke]

HDF5 1.10 dropped support for Visual Studio 2008 and 2010 required for Python 2.7 and 3.4. Even though HDF5 builds with those compilers and the HDF5 tests are OK, it does not seem to be a good idea to use them for h5py. In fact h5py+hdf5-1.10 crashes on Python 2.7.

That leaves Python 3.5 and 3.6 using Visual Studio 2015. H5py+hdf5-1.10 builds and tests pass. Unless libver='latest' is set, files written by those builds could be read with older software. It is not possible to import both h5py+hdf5-1.10 and current tables+hdf5-1.8 because the HDF5 DLLs are no longer compatible...

[vasole]

@cgohlke

I have h5py+hdf5-1.10.0 patch1 working on Python 2.7. Perhaps I do not use the conflicting features?

My experience is that files generated with 1.10 can be read with 1.8 as far as no 1.10 features are used.

I have formally requested the HDF Group to supply official binaries of the latest version (1.10.1) compiled with Visual Studio 2008 (64-bit). I expect to test them this week.

[cgohlke]

According to the release notes, Visual Studio 2008 and 2010 are not listed as supported. I am not using their official binaries, but that probably means things can break any 1.10.x release (?)
Just tested PyTables-3.4.0: works with HDF5-1.10.1. Netcdf-c-4.4.1.1 should also work. Could not find any info on GDAL yet.

[vasole]

@cgohlke

Probably I should use direct mail ...
Please take a look at the binaries supplied to us by the HDF group. As I said, I will test them this week, but you can also give them a try:

All binaries can be downloaded from ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/ESRF/

VS2008 and VS2015 binaries contain versions of the standard distributions (C,C++, FORTRAN, JAVA, zlib and szip compressions) and thread-safe distributions of just HDF5 C library (file names have "TS").

[vasole]

Sorry guys, I was not supposed to present those files as officially supported by the HDF Group.

I can attempt to provide those (or similar ones) under my responsibility thus not being any different from what Christoph is doing (no engagement or warranty from the HDF group)

[aragilar]

Uh, I forgot about ABI breakage :(

The 1.10.0 and 1.10.0-patch1 have VS2008 and VS2010 under "More Tested Platforms" (I'm not sure if that's platforms HDF group tested, or feedback from downstream...), but there aren't any Windows platforms under "More Tested Platforms" for 1.10.1, so it looks like VS2008 and VS2010 support can't be relied on (and why @vasole has a working 1.10.0-patch1 setup). It would make sense then to document (when we make the switch) if people want to use features from the HDF5 1.10 series to use Python 3.5 and up (at least on Windows), which would give another reason for people to switch to Python 3.

I agree with @cgohlke we would need to coordinate with other HDF5-providing packages, though how to avoid breakage is going to be hard, as there's no equivalent to apt/dpkg's breaks unfortunately to make sure all packages HDF5-providing are upgraded at the same time (we'll probably have the same problem on the other OSs too). In any case, we should add testing HDF5 1.10 to the CIs, so we can flag any possible problems before trying to release with 1.10 (and thanks @cgohlke for testing things, sorry for generating more work for you).

[vasole]

Anybody needing to read files generated with the SWMR is going to need 1.10.x

Honestly, I do not see much of a problem if things are shipped with 1.10 because I would expect:

• Users generating files using 1.8 will not use 1.10 features therefore generating 1.8 compatible files.
• Users using 1.10 can read files generated by 1.8
• Users having 1.8 already have a problem reading files generated using 1.10 features

In fact I had to provide 1.10 solutions for VS2008 and VS2015 because of the later.

[cgohlke]

I now got msvc9 builds of HDF5-1.10.1 that work stable with h5py. The previous builds picked up compatibility headers (stdint.h, inttypes.h) during the configuration step.
I have updated my h5py, pytables, and netcdf4 builds with hdf5-1.10.1. All tests pass.

[tacaswell]

Users having 1.8 already have a problem reading files generated using 1.10 features

I do not agree with this point. Many 1.8 users are not having problems because most packagers have started shipping 1.10 binaries yet so there are very few (accidental) 1.10 hdf5 files in the world.

[vasole]

@tacaswell

• The fact 1.8 users already have problems has nothing to do with packagers or the amount of files. DIAMOND is generating files with SWMR and 1.8 users cannot read them. That is not accidental at all and it is just a fact. BTW you put the word many, not me.

• You comment does not go against my post that it is just trying to support the point of view of the issue opener: windows wheels should support HDF5 1.10.1

[vasole]

@tacaswell

What is the rationale behind delaying (again) shipping builds with HDF5 1.10.x?

Please note that:

• Code currently generating files using h5py with HDF5 1.8.x works smoothly with HDF5 1.10.x
• Files generated by that same code but using 1.10.x are readable using h5py with HDF5 1.8.x
• Anaconda is shipping h5py with HDF5 1.10.x

Your argument of generating accidentally incompatible files with 1.8 does not hold. For that to happen the code using HDF5 1.10.x has to use HDF5 1.10 features and that does not happen accidentally. One does not create virtual datasets by accident nor one uses SWMR by accident.

[cgohlke]

The h5py-2.7.1 wheels for Windows are using HDF5-1.10.1.

[vasole]

Then I guess this issue should closed and my comments transferred to #972