Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDF5: Update to 1.14.3 #7908

Merged
merged 16 commits into from
Jan 8, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
166 changes: 31 additions & 135 deletions H/HDF5/build_tarballs.jl
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,28 @@ const YGGDRASIL_DIR = "../.."
include(joinpath(YGGDRASIL_DIR, "platforms", "mpi.jl"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the idea is that we would remove this line and replace it with a RuntimeDependency, but I'm a bit confused as well.

@giordano ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why replacing that line? That'd break everything. At most you want to try and change

push!(dependencies, Dependency(PackageSpec(name="MPIPreferences", uuid="3da0fdf6-3ccc-4f1b-acd9-58baa6c99267"); compat="0.1", top_level=true))
to a runtime dep, but I seem to remember that had some problems.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm starting to follow. You're saying we need to edit Yggdrasil/platforms/mpi.jl by perhaps reverting c4069bf.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only if it solves #6417 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking we should leave this for another pull request since fixing that will probably hit multiple recipes.

I'm good to merge this as is now if Erik is good.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think #7908 (comment) was addressed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@giordano That's the comment regarding OpenMP? What is unresolved there? I think we all agree that OpenMP isn't used and isn't mentioned, and since @mkitti removed the respective dependency all is good?

Apart from this: Yes, we can merge.


name = "HDF5"
version = v"1.14.2"
version = v"1.14.3"

# Collection of sources required to complete build
sources = [
ArchiveSource("https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-$(version.major).$(version.minor)/hdf5-$(version)/src/hdf5-$(version).tar.bz2",
"ea3c5e257ef322af5e77fc1e52ead3ad6bf3bb4ac06480dd17ee3900d7a24cfb"),
"9425f224ed75d1280bb46d6f26923dd938f9040e7eaebf57e66ec7357c08f917"),
DirectorySource("./bundled"),

# We don't build HDF5 on Windows; instead, we use packages from msys there:

# 32-bit Windows from https://packages.msys2.org/package/mingw-w64-i686-hdf5
ArchiveSource("https://mirror.msys2.org/mingw/mingw32/mingw-w64-i686-hdf5-1.14.2-2-any.pkg.tar.zst",
"ab053fdafb3e0c456751fed9fe5cc2fa339042046b4de677ce2848cd8b6d3b3f"; unpack_target="i686-w64-mingw32"),
ArchiveSource("https://mirror.msys2.org/mingw/mingw32/mingw-w64-i686-hdf5-1.14.3-1-any.pkg.tar.zst",
"09c6bf048eada4d3ad945c641111e7394169aa69904e5c4dd8de5c4b41143440"; unpack_target="i686-w64-mingw32"),
ArchiveSource("https://mirror.msys2.org/mingw/mingw32/mingw-w64-i686-zlib-1.3-1-any.pkg.tar.zst",
"21bacf3a43073749a4cbdf407c7f1da92bab56c80925b1205f7c4cb289c724a1"; unpack_target="i686-w64-mingw32"),
# We need some special compiler support libraries from mingw for i686 (libgcc_s_dw2)
ArchiveSource("https://mirror.msys2.org/mingw/mingw32/mingw-w64-i686-gcc-libs-13.2.0-2-any.pkg.tar.zst",
"2dae8189318a91067cca895572b2b46183bfd2ee97a55127a84f4f418f0b32f3"; unpack_target="i686-w64-mingw32"),

# 64-bit Windows from https://packages.msys2.org/package/mingw-w64-x86_64-hdf5
ArchiveSource("https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-hdf5-1.14.2-2-any.pkg.tar.zst",
"19a0a28d32c8990a29e001b77fe2deeb4946ff6c7d0949dbf756dfe1b9b40e8a"; unpack_target="x86_64-w64-mingw32"),
ArchiveSource("https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-hdf5-1.14.3-1-any.pkg.tar.zst",
"a6a69866725504cba9212a616c8734ade54c8de7d5f42074e33a7b62c7a31428"; unpack_target="x86_64-w64-mingw32"),
ArchiveSource("https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-zlib-1.3-1-any.pkg.tar.zst",
"254a6c5a8a27d1b787377a3e70a39cceb200b47c5f15f4ab5bfa1431b718ef98"; unpack_target="x86_64-w64-mingw32"),

Expand Down Expand Up @@ -68,102 +68,10 @@ atomic_patch -p1 ${WORKSPACE}/srcdir/patches/mpi.patch
# Idea:
# - provide the registered filter plugins (BZIP2, JPEG, LZF, BLOSC, MAFISC, LZ4, Bitshuffle, and ZFP)

# Building via `configure` instead of via `cmake` has one advantage:
# The `h5cc` etc. scripts are generated, and some other packages might expect these.
if false; then

# Option 1: Build with cmake
# This is now outdated, e.g. it doesn't enable C++ nor Fortran.

# Patch `CMakeLists.txt`:
# - HDF5 would also try to build and run `H5detect` to collect ABI information.
# We know this information, and thus can provide it manually.
# - HDF5 would try to build and run `H5make_libsettings` to collect
# build-time information. That information seems entirely optional, so
# we do mostly nothing instead.
atomic_patch -p1 ${WORKSPACE}/srcdir/patches/CMakeLists.txt.patch

# Prepare the pre-generated file `H5Tinit.c` that cmake will expect:
case "${target}" in
aarch64-apple-darwin*)
cat ../../files/H5Tinit-darwin-arm64v8.c
;;
aarch64-linux-*)
cat ../../files/H5Tinit-debian-arm64v8.c
;;
arm-linux-*)
cat ../../files/H5Tinit-debian-arm32v7.c
;;
i686-linux-*)
cat ../../files/H5Tinit-debian-i386.c
;;
i686-w64-mingw32)
# sizeof(long double) == 12
# layout seems to be 16-bit sign+exponent and 64-bit mantissa
# same as for Linux
cat ../../files/H5Tinit-debian-i386.c
;;
powerpc64le-linux-*)
cat ../../files/H5Tinit-debian-ppc64le.c
;;
x86_64-apple-darwin*)
cat ../../files/H5Tinit-darwin-amd64.c
;;
x86_64-linux-* | x86_64-*-freebsd*)
cat ../../files/H5Tinit-debian-amd64.c
;;
x86_64-w64-mingw32)
# sizeof(long double) == 16
# layout seems to be 16-bit sign+exponent and 64-bit mantissa
# same as for Linux
cat ../../files/H5Tinit-debian-amd64.c
;;
*)
echo "Unsupported target architecture ${target}" >&2
exit 1
;;
esac >src/H5Tinit.c
echo 'char H5libhdf5_settings[]="";' >src/H5lib_settings.c

mkdir build
pushd build

cmake \
-DCMAKE_FIND_ROOT_PATH=${prefix} \
-DCMAKE_INSTALL_PREFIX=${prefix} \
-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} \
-DBUILD_STATIC_LIBS=OFF \
-DBUILD_TESTING=OFF \
-DHDF5_BUILD_EXAMPLES=OFF \
-DHDF5_BUILD_HL_LIB=ON \
-DHDF5_BUILD_TOOLS=ON \
-DHAVE_IOEO_EXITCODE=0 \
-DTEST_LFS_WORKS_RUN=0 \
-DH5_LDOUBLE_TO_LONG_SPECIAL_RUN=1 \
-DH5_LDOUBLE_TO_LONG_SPECIAL_RUN__TRYRUN_OUTPUT= \
-DH5_LONG_TO_LDOUBLE_SPECIAL_RUN=1 \
-DH5_LONG_TO_LDOUBLE_SPECIAL_RUN__TRYRUN_OUTPUT= \
-DH5_LDOUBLE_TO_LLONG_ACCURATE_RUN=0 \
-DH5_LDOUBLE_TO_LLONG_ACCURATE_RUN__TRYRUN_OUTPUT= \
-DH5_LLONG_TO_LDOUBLE_CORRECT_RUN=0 \
-DH5_LLONG_TO_LDOUBLE_CORRECT_RUN__TRYRUN_OUTPUT= \
-DH5_DISABLE_SOME_LDOUBLE_CONV_RUN=1 \
-DH5_DISABLE_SOME_LDOUBLE_CONV_RUN__TRYRUN_OUTPUT= \
..
cmake --build . --config RelWithDebInfo --parallel ${nproc}
cmake --build . --config RelWithDebInfo --parallel ${nproc} --target install

popd

else

# Option 2: Build with configure
# This is the currently supported way.

# Patch `configure.ac`:
atomic_patch -p1 ${WORKSPACE}/srcdir/patches/configure.ac.patch

# Prepare the files `H5init.c` and `config.saved` that contain predetermined
# Prepare the file `config.saved` that contains predetermined
# configuration information
mkdir saved
case "${target}" in
Expand All @@ -182,7 +90,6 @@ case "${target}" in
i686-w64-mingw32)
# sizeof(long double) == 12
# layout seems to be 16-bit sign+exponent and 64-bit mantissa
# same as for Linux
cp ../files/msys2-mingw32/* saved
;;
powerpc64le-linux-*)
Expand All @@ -195,13 +102,12 @@ case "${target}" in
cp ../files/debian-amd64/* saved
;;
x86_64-*-freebsd*)
# same as for Linux
# no __float128
cp ../files/freebsd-amd64/* saved
;;
x86_64-w64-mingw32)
# sizeof(long double) == 16
# layout seems to be 16-bit sign+exponent and 64-bit mantissa
# same as for Linux
cp ../files/msys2-mingw64/* saved
;;
*)
Expand All @@ -211,6 +117,7 @@ case "${target}" in
esac
cp ../files/get_config_setting saved


env \
HDF5_ACLOCAL=/usr/bin/aclocal \
HDF5_AUTOHEADER=/usr/bin/autoheader \
Expand Down Expand Up @@ -253,45 +160,39 @@ elif [[ ${target} == *-w64-mingw32 ]]; then
fi

# Configure MPI
ENABLE_PARALLEL=yes
if grep -q MPICH_NAME ${prefix}/include/mpi.h; then
# MPICH
export CC=mpicc
export CXX=mpicxx
export FC=mpifort
elif grep -q MPITRAMPOLINE_MPI_H ${prefix}/include/mpi.h; then
# MPItrampoline
export MPITRAMPOLINE_CC="$(which $CC)"
export MPITRAMPOLINE_CXX="$(which $CXX)"
export MPITRAMPOLINE_FC="$(which $FC)"
export CC=mpicc
export CXX=mpicxx
export FC=mpifort
elif grep -q MSMPI_VER ${prefix}/include/mpi.h; then
if grep -q MSMPI_VER ${prefix}/include/mpi.h; then
# Microsoft MPI
if [[ ${target} == i686-* ]]; then
# 32-bit system
# Do not enable MPI; the function MPI_File_close is not defined
# in the 32-bit version of Microsoft MPI 10.1.12498.18
ENABLE_PARALLEL=no
:
else
# Hide static libraries
rm ${prefix}/lib/msmpi*.lib
# Make shared libraries visible
ln -s msmpi.dll ${libdir}/libmsmpi.dll
export FCFLAGS="$FCFLAGS -I${prefix}/src -I${prefix}/include -fno-range-check"
ENABLE_PARALLEL=yes
export FCFLAGS="${FCFLAGS} -I${prefix}/src -I${prefix}/include -fno-range-check"
export LIBS="-L${libdir} -lmsmpi"
fi
elif grep -q OMPI_MAJOR_VERSION ${prefix}/include/mpi.h; then
# OpenMPI
else
ENABLE_PARALLEL=yes
export MPITRAMPOLINE_CC="${CC}"
export MPITRAMPOLINE_CXX="${CXX}"
export MPITRAMPOLINE_FC="${FC}"
export CC=mpicc
export CXX=mpicxx
export FC=mpifort
else
# Unknown MPI
exit 1
fi

# This is likely a bug in HDF5; see
# <https://github.com/HDFGroup/hdf5/issues/3925>. The file
# `config/freebsd` includes `config/classic-fflags` which is
# missing. This file is supposed to contain Fortran flags for the
# classic Intel compiler.
: >../config/classic-fflags

../configure \
--prefix=${prefix} \
--build=${MACHTYPE} \
Expand Down Expand Up @@ -337,14 +238,15 @@ fi
"$(../saved/get_config_setting H5CONFIG_F_NUM_IKIND ../saved/config.status)" \
"$(../saved/get_config_setting H5CONFIG_F_IKIND ../saved/config.status)"


# Patch the generated `Makefile`:
# (We could instead patch `Makefile.in`, or maybe even `Makefile.am`.)
# - HDF5 would also try to build and run `H5detect` to collect ABI information.
# We know this information, and thus can provide it manually.
# - HDF5 would try to build and run `H5make_libsettings` to collect
# build-time information. That information seems entirely optional, so
# we do mostly nothing instead.
atomic_patch -p1 ${WORKSPACE}/srcdir/patches/src-Makefile.patch
#atomic_patch -p1 ${WORKSPACE}/srcdir/patches/src-Makefile.patch
atomic_patch -p1 ${WORKSPACE}/srcdir/patches/fortran-src-Makefile.patch
atomic_patch -p1 ${WORKSPACE}/srcdir/patches/hl-fortran-src-Makefile.patch

Expand All @@ -357,8 +259,6 @@ make install

popd

fi

install_license COPYING
"""

Expand All @@ -376,7 +276,7 @@ platforms = expand_cxxstring_abis(platforms)
platforms = expand_gfortran_versions(platforms)

# TODO: Don't require MPI for Windows since we're using the non-MPI msys libraries there.
platforms, platform_dependencies = MPI.augment_platforms(platforms; MPItrampoline_compat="5.3.0")
platforms, platform_dependencies = MPI.augment_platforms(platforms; MPItrampoline_compat="5.3.1", OpenMPI_compat="4.1.6")

# Avoid platforms where the MPI implementation isn't supported
# OpenMPI
Expand Down Expand Up @@ -421,12 +321,8 @@ products = [

# Dependencies that must be installed before this package can be built
dependencies = [
# For OpenMP we use libomp from `LLVMOpenMP_jll` where we use LLVM as compiler (BSD
# systems), and libgomp from `CompilerSupportLibraries_jll` everywhere else.
Dependency(PackageSpec(name="CompilerSupportLibraries_jll", uuid="e66e0078-7015-5450-92f7-15fbd957f2ae");
platforms=filter(!Sys.isbsd, platforms)),
Dependency(PackageSpec(name="LLVMOpenMP_jll", uuid="1d63c593-3942-5779-bab2-d838dc0a180e");
platforms=filter(Sys.isbsd, platforms)),
Comment on lines -424 to -429
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@giordano OpenMP was removed as a dependency.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't see that addressed in the comments after my comment, I missed it had been changed before! The diff was quite large 🥲

# To ensure that the correct version of libgfortran is found at runtime
Dependency(PackageSpec(name="CompilerSupportLibraries_jll", uuid="e66e0078-7015-5450-92f7-15fbd957f2ae")),
eschnett marked this conversation as resolved.
Show resolved Hide resolved
Dependency("LibCURL_jll"),
# The msys Windows libraries require OpenSSL@3
Dependency("OpenSSL_jll"; compat="3.0.8"),
Expand All @@ -443,4 +339,4 @@ ENV["MPITRAMPOLINE_DELAY_INIT"] = "1"
# Build the tarballs, and possibly a `build.jl` as well.
# GCC 5 reports an ICE on i686-linux-gnu-libgfortran3-cxx11-mpi+mpich
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies;
augment_platform_block, julia_compat="1.6", preferred_gcc_version=v"6")
augment_platform_block, clang_use_lld=false, julia_compat="1.6", preferred_gcc_version=v"6")