Update to use BinaryProvider

[staticfloat]

Let's see if this passes tests!

[musm]

closes
#539
#526
#519
#521
#504
#483
#481
#447
#410
#334
#97

award for most issues closed by a single PR ? 😆

[musm]

Can we also now remove https://github.com/JuliaIO/HDF5.jl/blob/master/.travis.yml the hdf5 package downloads there ?

[staticfloat]

There's a problem with Windows; investigating. It's more complex than just missing the MSVCR90 runtime, unfortunately.

[musm]

Do these release notes at all help

from https://portal.hdfgroup.org/display/support/HDF5%201.10.4#releasenotes

New Features
============
 
    Configuration:
    -------------
    - Add toolchain and cross-compile support
 
      Added info on using a toolchain file to INSTALL_CMAKE.txt. A
      toolchain file is also used in cross-compiling, which requires
      CMAKE_CROSSCOMPILING_EMULATOR to be set. To help with cross-compiling
      the fortran configure process, the HDF5UseFortran.cmake file macros
      were improved. Fixed a Fortran configure file issue that incorrectly
      used #cmakedefine instead of #define.

[staticfloat]

Windows issues may be resolved through a new BP release, which is currently making its way into General.

[musm]

HDF5 version is 1.8 on windows, is this expected?

[staticfloat]

Yes. We're getting it via msys2, which only builds version 1.8. I think that's not a problem for this package; HDF5.jl can deal with both versions.

[musm]

Ok I'm working on trying to build HDF5 using a Cygwin Host environment and mingw, the build is just taking forever. Would we be able to utilize those? I believe it should be compatible with Julia.

[staticfloat]

Likely, yes. When you finish building it, upload them somewhere and I'll shove them through a BB audit pass to double-check.

[staticfloat]

The windows tests work locally for me, but they're failing on Travis 32-bit, so I'm trying to figure out why that is.

[staticfloat]

Okay so these are working now across all platforms. Yay! Here are the details:

• We use PyPI to source binaries for Linux and MacOS. These pretty much "just work"; we rewrite the dependency chain to link against our BB-provided libz, which is the only real change.

• We use MSYS2 to source binaries for Win32/Win64, because the PyPI binaries use Visual Studio to build theirs, and we can't mix-and-match C runtime libraries. This means that Windows is using 1.8.X whereas Linux and Mac are using 1.10.X, but that's not the end of the world. Also, the win32 binaries depend on a different version of libgcc (they use libgcc_s_dw2-1.dll instead of the sjl variant) which I have had to bundle along inside the tarball. I don't know what the ramifications of having multiple libgcc_s.dll files around are, but it seems to work, so I'm rolling with it for now.

In the future, it may be worthwhile to just manually compile tarballs with the compilers we know and trust, and bundle those up. Long-term, we should continue to improve BB to add in emulation capabilities such that the HDF5 cross-compilation process can actually be completed within the BB environment.

[simonbyrne]

In the future, it may be worthwhile to just manually compile tarballs with the compilers we know and trust, and bundle those up. Long-term, we should continue to improve BB to add in emulation capabilities such that the HDF5 cross-compilation process can actually be completed within the BB environment.

If there are only small changes required to the HDF5 build process, we could also try to upstream those changes as well.

[staticfloat]

If there are only small changes required to the HDF5 build process, we could also try to upstream those changes as well.

The problem is that the HDF5 build process requires a lot of bootstrapped execution; they use their (compiled for target) binaries to generate source code that must then be compiled. The "manual compilation" here is not because the build system is broken, it's because we can't do it inside of BB, because we can't run things like windows executables inside of BB (yet).

[simonbyrne]

The problem is that the HDF5 build process requires a lot of bootstrapped execution; they use their (compiled for target) binaries to generate source code that must then be compiled.

😱

[aviks]

steven johnson has been attempting to remove the compile-and-run checks to autoconf checks. He thinks it's feasible, but its a long slog. see https://github.com/stevengj/HDF5builder and its forks for progress.

[ViralBShah]

Pinging @stevengj so he knows about this.

[musm]

I've been trying to build windows binaries using a cygwin host and mingw-w64 compiler

$ ./configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --prefix=/cygdrive/c/Users/Mus/work/hdf5/out
$ make 

Unfortunately the build stalls at

  CC       H5HFtest.lo
  CC       H5HFtiny.lo
  CC       H5HG.lo
  CC       H5HGcache.lo
  CC       H5HGdbg.lo
  CC       H5HGquery.lo
  CC       H5HL.lo
  CC       H5HLcache.lo
  CC       H5HLdbg.lo
  CC       H5HLint.lo
  CC       H5HLprfx.lo
  CC       H5HLdblk.lo
  CC       H5HP.lo
  CC       H5I.lo
H5I.c: In function ‘H5I__id_dump_cb’:
H5I.c:2125:47: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
     HDfprintf(stderr, "   obj   = 0x%08lx\n", (unsigned long)(item->obj_ptr));
                                               ^
  CC       H5Itest.lo
  CC       H5L.lo
  CC       H5Lexternal.lo
  CC       H5make_libsettings.o
  CCLD     H5make_libsettings.exe```

[simonbyrne]

https://github.com/live-clones/hdf5/blob/f97ea92fe5450cefd4c4d9fb266e7f42a952f2f9/src/H5I.c#L2341

wow, they seem to just assume that unsigned long is the same size as the pointer?

[staticfloat]

@musm have you tried doing a verbose build? Try setting V=1 or VERBOSE=1 or something like that.

[simonbyrne]

LGTM!

[musm]

Unfortunately the build keeps freezing. I will try through MSYS2 and see if that works.

Thanks so much for this fantastic PR.