HDF5 build error (VEGA) #4

mattzett · 2022-06-03T14:54:42Z

gcc 8.3 and cmake 3.20:

[ 88%] Building C object fortran/src/CMakeFiles/H5match_types.dir/H5match_types.c.o
In file included from /home3/zettergm/Projects/external/build/hdf5-prefix/src/hdf5/fortran/src/H5match_types.c:40:
/home3/zettergm/Projects/external/build/hdf5-prefix/src/hdf5/fortran/src/H5match_types.c: In function 'main':
/home3/zettergm/Projects/external/build/hdf5-prefix/src/hdf5-build/fortran/H5fort_type_defines.h:23:43: error: subscripted value is neither array nor pointer nor vector
 #define H5_FORTRAN_INTEGER_KINDS_SIZEOF {1[1654265863.761701][vegaln1:195718:0]sys.c:744MXMWARNConflictingCPUfrequenciesdetected,using:3404.63,2[1654265864.588095][vegaln1:195734:0]sys.c:744MXMWARNConflictingCPUfrequenciesdetected,using:3296.37,4[1654265865.459181][vegaln1:195749:0]sys.c:744MXMWARNConflictingCPUfrequenciesdetected,using:2860.09,8[1654265866.256815][vegaln1:195763:0]sys.c:744MXMWARNConflictingCPUfrequenciesdetected,using:3430.05,16[1654265867.060529][vegaln1:195779:0]sys.c:744MXMWARNConflictingCPUfrequenciesdetected,using:3471.47}
                                           ^
/home3/zettergm/Projects/external/build/hdf5-prefix/src/hdf5/fortran/src/H5match_types.c:154:31: note: in expansion of macro H5_FORTRAN_INTEGER_KINDS_SIZEOF'
     int  IntKinds_SizeOf[]  = H5_FORTRAN_INTEGER_KINDS_SIZEOF;

The text was updated successfully, but these errors were encountered:

scivision · 2022-06-04T00:07:34Z

What HDF5 version do you have in cmake/libraries.json?

external/cmake/libraries.json

Line 68 in 7274e75

"tag": "hdf5-1_12_2"

You can put like 1_10_6, 1_10_7, 1_12_1, to see if there is a working version. HDF Group has been overhauling build of HDF5. 1_12_2 was supposed to fix a number of substantial HDF5 build problems and boost HDF5 performance, but maybe they broke something for older systems.

I.e. what HDF5 version are you now trying to build? If before 1_12_2 maybe git pull this gemini3d/external repo to try HDF5 1.12.2. Otherwise, manually trying older HDF5 versions in cmake/libraries.json may help. I could make this a CMake command line options.

With older HDF5 versions, there were other build issues I saw, but the error you noted is not an error I've seen yet.

mattzett · 2022-06-28T17:07:50Z

So I updated to cmake 3.23.2 and that seemed to not trigger the same error but now I get this:

/home3/zettergm/Projects/external/build/h5fortran-prefix/src/h5fortran/src/write_scalar.f90:31:111:

   call hdf_create(self, dname, H5T_NATIVE_CHARACTER, dims, filespace_id, dset_id, dtype_id, charlen=len(value))
                                                                                                               1
Error: Type mismatch in argument 'charlen' at (1); passed INTEGER(8) to INTEGER(4)
gmake[5]: *** [CMakeFiles/h5fortran.dir/src/write_scalar.f90.o] Error 1
gmake[4]: *** [CMakeFiles/h5fortran.dir/all] Error 2
gmake[3]: *** [all] Error 2
gmake[2]: *** [h5fortran-prefix/src/h5fortran-stamp/h5fortran-build] Error 2
gmake[1]: *** [CMakeFiles/h5fortran.dir/all] Error 2
gmake: *** [all] Error 2

mattzett · 2022-06-29T13:34:20Z

I just realized that this is now an issue with h5fortran apparently... How do I set the version for that; does it make sense to try an earlier version since this did work before.

mattzett · 2022-06-29T14:08:17Z

Okay I found a workaround for VEGA (gcc 8.3). It's a two part process:

compile and install cmake 3.23.2 - this fixes all the processor frequency warnings
export CC=gcc && export CXX=g++ && export FC=gfortran
Additionally clear out any other environment variables that point to compilers, e.g. CXXCPP, F77, etc.
pull external and edit ./cmake/libraries.json . Change the h5fortran version to 4.4.4 and then try to build

mattzett · 2022-06-29T14:14:34Z

@scivision Could you take a look at this and see if you can reproduce with gcc 8.3?

paulinchin · 2022-06-30T15:38:16Z

[pineyrob@vegaln1 GEMININEWEST3]$ module list
Currently Loaded Modulefiles:

craype-network-infiniband 3) gcc/8.3.0 5) python/3.9.2
craype-broadwell 4) openmpi/gcc8/64/3.1.2
[pineyrob@vegaln1 GEMININEWEST3]$ git clone https://github.com/gemini3d/external.git
Cloning into 'external'...
remote: Enumerating objects: 730, done.
remote: Counting objects: 100% (219/219), done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 730 (delta 149), reused 133 (delta 73), pack-reused 511
Receiving objects: 100% (730/730), 197.33 KiB | 0 bytes/s, done.
Resolving deltas: 100% (491/491), done.
[pineyrob@vegaln1 GEMININEWEST3]$ cd external/cmake/
[pineyrob@vegaln1 cmake]$ nano libraries.json
[pineyrob@vegaln1 cmake]$ cd ..
[pineyrob@vegaln1 external]$ cd ..
[pineyrob@vegaln1 GEMININEWEST3]$ ls
external
[pineyrob@vegaln1 GEMININEWEST3]$ /cm/shared/apps/cmake/3.23.2/bin/cmake -B external/build -S external -DCMAKE_INSTALL_PREFIX=./libgem
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- The Fortran compiler identification is GNU 8.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/bin/gfortran - skipped
-- Found Python: /cm/shared/apps/admin/python/3.9.2/bin/python3.9 (found suitable version "3.9.2", minimum required is "3.7") found components: Interpreter
-- checking that C, C++, and Fortran compilers can link
-- checking that C, C++, and Fortran compilers can link - OK
-- Found Autotools: /usr/bin/autoconf
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1")
-- Checking for one of the modules 'hwloc'
-- Could NOT find HWLOC (missing: HWLOC_LIBRARY HWLOC_TOPO_LOAD) (Required is at least version "2.4")
-- Found LibXml2: /usr/lib64/libxml2.so (found version "2.9.1")
-- Found MPI_C: /cm/shared/apps/openmpi/gcc/64/3.1.2_gcc8/lib/libmpi.so (found version "3.1")
-- Found MPI_Fortran: /cm/shared/apps/openmpi/gcc/64/3.1.2_gcc8/lib/libmpi_usempif08.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: C Fortran
-- Could NOT find HDF5 (missing: HDF5_C_LIBRARIES HDF5_links C Fortran HL)
-- Configuring done
-- Generating done
-- Build files have been written to: /scratch/pineyrob/GEMININEWEST3/external/build
[pineyrob@vegaln1 GEMININEWEST3]$ /cm/shared/apps/cmake/3.23.2/bin/cmake --build external/build
[ 1%] Creating directories for 'zlib'
[ 2%] Performing download step (git clone) for 'zlib'
Cloning into 'zlib'...
HEAD is now at b56a2fd... Version 2.0.6 - Fix hangs on macOS #1031 - Fix minideflate write buffers being overwritten #1060 - Fix deflateBound and compressBound returning too small size estimates #1049 #1071 - Fix incorrect function declaration warning #1080 - Fix build problems when building outside of source dir #1049 - Fix build problems on arm2-7 #1030 - Fixed some compile warnings #1020 #1036 #1037 #1048 - Improved posix memalign support #888 - Improvements to testing #637 #1026 #1032 #1035 #1049 #1051 #1056 #1063 #1067 #1079 - Improvements for integration into other projects #1022 #1042 - Code style fixes #637 #1040 #1050 #1075
[ 3%] Performing update step for 'zlib'
[ 4%] No patch step for 'zlib'
[ 5%] Performing configure step for 'zlib'
-- Using CMake version 3.23.2
-- ZLIB_HEADER_VERSION: 1.2.11
-- ZLIBNG_HEADER_VERSION: 2.0.6
-- The C compiler identification is GNU 8.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Arch detected: 'x86_64'
-- Basearch of 'x86_64' has been detected as: 'x86'
-- Performing Test FNO_LTO_AVAILABLE
-- Performing Test FNO_LTO_AVAILABLE - Success
-- Architecture supports unaligned reads
-- Architecture supports unaligned reads of > 4 bytes
-- Looking for sys/sdt.h
-- Looking for sys/sdt.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for strerror
-- Looking for strerror - found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Performing Test HAVE_NO_INTERPOSITION
-- Performing Test HAVE_NO_INTERPOSITION - Failed
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN - Success
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL - Success
-- Performing Test HAVE_BUILTIN_CTZ
-- Performing Test HAVE_BUILTIN_CTZ - Success
-- Performing Test HAVE_BUILTIN_CTZLL
-- Performing Test HAVE_BUILTIN_CTZLL - Success
-- Performing Test HAVE_PTRDIFF_T
-- Performing Test HAVE_PTRDIFF_T - Success
-- Performing Test HAVE_SSE2_INTRIN
-- Performing Test HAVE_SSE2_INTRIN - Success
-- Performing Test HAVE_SSSE3_INTRIN
-- Performing Test HAVE_SSSE3_INTRIN - Success
-- Performing Test HAVE_SSE42CRC_INLINE_ASM
-- Performing Test HAVE_SSE42CRC_INLINE_ASM - Success
-- Performing Test HAVE_SSE42CRC_INTRIN
-- Performing Test HAVE_SSE42CRC_INTRIN - Success
-- Performing Test HAVE_SSE42CMPSTR_INTRIN
-- Performing Test HAVE_SSE42CMPSTR_INTRIN - Success
-- Performing Test HAVE_PCLMULQDQ_INTRIN
-- Performing Test HAVE_PCLMULQDQ_INTRIN - Success
-- Performing Test HAVE_AVX2_INTRIN
-- Performing Test HAVE_AVX2_INTRIN - Success
-- Architecture-specific source files: arch/x86/x86.c;arch/x86/slide_avx.c;arch/x86/chunkset_avx.c;arch/x86/compare258_avx.c;arch/x86/adler32_avx.c;arch/x86/insert_string_sse.c;arch/x86/compare258_sse.c;arch/x86/chunkset_sse.c;arch/x86/slide_sse.c;arch/x86/adler32_ssse3.c;arch/x86/crc_folding.c
-- The following features have been enabled:

CMAKE_BUILD_TYPE, Build type: Release (selected)
WITH_GZFILEOP, Compile with support for gzFile related functions
ZLIB_COMPAT, Compile with zlib compatible API
WITH_OPTIM, Build with optimisation
WITH_NEW_STRATEGIES, Use new strategies
WITH_UNALIGNED, Support unaligned reads on platforms that support it
WITH_AVX2, Build with AVX2
WITH_SSE2, Build with SSE2
WITH_SSSE3, Build with SSSE3
WITH_SSE4, Build with SSE4
WITH_PCLMULQDQ, Build with PCLMULQDQ
AVX2_SLIDEHASH, Support AVX2 optimized slide_hash, using "-mavx2"
AVX_CHUNKSET, Support AVX optimized chunkset, using "-mavx2"
AVX2_COMPARE258, Support AVX2 optimized compare258, using "-mavx2"
AVX2_ADLER32, Support AVX2-accelerated adler32, using "-mavx2"
SSE42_CRC, Support SSE4.2 optimized CRC hash generation, using "-msse4"
SSE42_COMPARE258, Support SSE4.2 optimized compare258, using "-msse4"
SSSE3_ADLER32, Support SSSE3-accelerated adler32, using "-mssse3"
PCLMUL_CRC, Support CRC hash generation using PCLMULQDQ, using "-mssse3 -msse4 -mpclmul"

-- The following features have been disabled:

ZLIB_ENABLE_TESTS, Build test binaries
ZLIB_DUAL_LINK, Dual link tests against system zlib
WITH_FUZZERS, Build test/fuzz
WITH_NATIVE_INSTRUCTIONS, Instruct the compiler to use the full instruction set on this host (gcc/clang -march=native)
WITH_MAINTAINER_WARNINGS, Build with project maintainer warnings
WITH_CODE_COVERAGE, Enable code coverage reporting
WITH_INFLATE_STRICT, Build with strict inflate distance checking
WITH_INFLATE_ALLOW_INVALID_DIST, Build with zero fill for inflate invalid distances
INSTALL_UTILS, Copy minigzip and minideflate during install

-- Configuring done
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:

BUILD_TESTING
CMAKE_CXX_COMPILER
CMAKE_Fortran_COMPILER

-- Build files have been written to: /scratch/pineyrob/GEMININEWEST3/external/build/zlib-prefix/src/zlib-build
[ 6%] Performing build step for 'zlib'
[ 2%] Building C object CMakeFiles/zlib.dir/adler32.c.o
[ 5%] Building C object CMakeFiles/zlib.dir/chunkset.c.o
[ 8%] Building C object CMakeFiles/zlib.dir/compare258.c.o
[ 11%] Building C object CMakeFiles/zlib.dir/compress.c.o
[ 14%] Building C object CMakeFiles/zlib.dir/crc32.c.o
[ 17%] Building C object CMakeFiles/zlib.dir/crc32_comb.c.o
[ 20%] Building C object CMakeFiles/zlib.dir/deflate.c.o
[ 22%] Building C object CMakeFiles/zlib.dir/deflate_fast.c.o
[ 25%] Building C object CMakeFiles/zlib.dir/deflate_medium.c.o
[ 28%] Building C object CMakeFiles/zlib.dir/deflate_quick.c.o
[ 31%] Building C object CMakeFiles/zlib.dir/deflate_slow.c.o
[ 34%] Building C object CMakeFiles/zlib.dir/functable.c.o
[ 37%] Building C object CMakeFiles/zlib.dir/infback.c.o
[ 40%] Building C object CMakeFiles/zlib.dir/inffast.c.o
[ 42%] Building C object CMakeFiles/zlib.dir/inflate.c.o
[ 45%] Building C object CMakeFiles/zlib.dir/inftrees.c.o
[ 48%] Building C object CMakeFiles/zlib.dir/insert_string.c.o
[ 51%] Building C object CMakeFiles/zlib.dir/trees.c.o
[ 54%] Building C object CMakeFiles/zlib.dir/uncompr.c.o
[ 57%] Building C object CMakeFiles/zlib.dir/zutil.c.o
[ 60%] Building C object CMakeFiles/zlib.dir/arch/x86/x86.c.o
[ 62%] Building C object CMakeFiles/zlib.dir/arch/x86/slide_avx.c.o
[ 65%] Building C object CMakeFiles/zlib.dir/arch/x86/chunkset_avx.c.o
[ 68%] Building C object CMakeFiles/zlib.dir/arch/x86/compare258_avx.c.o
[ 71%] Building C object CMakeFiles/zlib.dir/arch/x86/adler32_avx.c.o
[ 74%] Building C object CMakeFiles/zlib.dir/arch/x86/insert_string_sse.c.o
[ 77%] Building C object CMakeFiles/zlib.dir/arch/x86/compare258_sse.c.o
[ 80%] Building C object CMakeFiles/zlib.dir/arch/x86/chunkset_sse.c.o
[ 82%] Building C object CMakeFiles/zlib.dir/arch/x86/slide_sse.c.o
[ 85%] Building C object CMakeFiles/zlib.dir/arch/x86/adler32_ssse3.c.o
[ 88%] Building C object CMakeFiles/zlib.dir/arch/x86/crc_folding.c.o
[ 91%] Building C object CMakeFiles/zlib.dir/gzlib.c.o
[ 94%] Building C object CMakeFiles/zlib.dir/gzread.c.o
[ 97%] Building C object CMakeFiles/zlib.dir/gzwrite.c.o
[100%] Linking C static library libz.a
[100%] Built target zlib
[ 7%] Performing install step for 'zlib'
Consolidate compiler generated dependencies of target zlib
[100%] Built target zlib
Install the project...
-- Install configuration: "Release"
-- Installing: /scratch/pineyrob/GEMININEWEST3/libgem/lib/libz.a
-- Installing: /scratch/pineyrob/GEMININEWEST3/libgem/include/zlib.h
-- Installing: /scratch/pineyrob/GEMININEWEST3/libgem/include/zconf.h
-- Installing: /scratch/pineyrob/GEMININEWEST3/libgem/lib/pkgconfig/zlib.pc
[ 8%] No test step for 'zlib'
[ 9%] Completed 'zlib'
[ 9%] Built target zlib
[ 10%] Creating directories for 'ffilesystem'
[ 11%] Performing download step (git clone) for 'ffilesystem'
Cloning into 'ffilesystem'...
HEAD is now at 025bc5e... test C: linker_language CXX for Intel and GCC < 8
[ 12%] Performing update step for 'ffilesystem'
[ 13%] No patch step for 'ffilesystem'
[ 14%] Performing configure step for 'ffilesystem'
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- The Fortran compiler identification is GNU 8.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/bin/gfortran - skipped
-- Performing Test HAS_Fortran_2018
-- Performing Test HAS_Fortran_2018 - Success
-- Looking for __has_include
-- Looking for __has_include - not found
CMake Error at cmake/compilers.cmake:58 (message):
C++ compiler not C++17 capable GNU 8.3.0
Call Stack (most recent call first):
CMakeLists.txt:14 (include)

-- Configuring incomplete, errors occurred!
See also "/scratch/pineyrob/GEMININEWEST3/external/build/ffilesystem-prefix/src/ffilesystem-build/CMakeFiles/CMakeOutput.log".
See also "/scratch/pineyrob/GEMININEWEST3/external/build/ffilesystem-prefix/src/ffilesystem-build/CMakeFiles/CMakeError.log".
gmake[2]: *** [ffilesystem-prefix/src/ffilesystem-stamp/ffilesystem-configure] Error 1
gmake[1]: *** [CMakeFiles/ffilesystem.dir/all] Error 2
gmake: *** [all] Error 2

scivision · 2022-07-01T23:36:44Z

@paulinchin I think this issue is resolved now? It was a general issue with blanking out unwanted environment variables on Vega?

scivision · 2022-07-01T23:37:18Z

However, there may still be an issue with h5fortran > v4.4.4, but not the more severe issue shown later

scivision · 2022-07-06T17:51:52Z

I confirm on CentOS 7 with GCC 8.3.0 that the compiler bug where

   call hdf_create(self, dname, H5T_NATIVE_CHARACTER, dims, dims, file_space_id, dset_id, dtype_id, charlen=len(value))

                  1
Error: Type mismatch in argument 'charlen' at (1); passed INTEGER(8) to INTEGER(4)

is still present in h5fortran main branch. I will move that part to h5fortran repo.

scivision · 2022-07-06T18:06:45Z

I am closing this as the GCC 8.3.0 workaround is implemented in a new version of h5fortran. This is a definite GCC 8.3.0 bug that was fixed by GCC 8.5.0.

The rest of the reported issues here were resolved by clearing out unwanted compiler environment variables on Vega.

mattzett assigned scivision Jun 3, 2022

scivision added the HDF5 Build issues with HDF5 label Jun 4, 2022

scivision mentioned this issue Jul 6, 2022

GCC 8.3.0 bug (GCC 8.5.0 and others work) geospace-code/h5fortran#31

Closed

scivision closed this as completed Jul 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDF5 build error (VEGA) #4

HDF5 build error (VEGA) #4

mattzett commented Jun 3, 2022

scivision commented Jun 4, 2022 •

edited

mattzett commented Jun 28, 2022

mattzett commented Jun 29, 2022

mattzett commented Jun 29, 2022 •

edited

mattzett commented Jun 29, 2022 •

edited

paulinchin commented Jun 30, 2022 •

edited

scivision commented Jul 1, 2022

scivision commented Jul 1, 2022

scivision commented Jul 6, 2022

scivision commented Jul 6, 2022

HDF5 build error (VEGA) #4

HDF5 build error (VEGA) #4

Comments

mattzett commented Jun 3, 2022

scivision commented Jun 4, 2022 • edited

mattzett commented Jun 28, 2022

mattzett commented Jun 29, 2022

mattzett commented Jun 29, 2022 • edited

mattzett commented Jun 29, 2022 • edited

paulinchin commented Jun 30, 2022 • edited

scivision commented Jul 1, 2022

scivision commented Jul 1, 2022

scivision commented Jul 6, 2022

scivision commented Jul 6, 2022

scivision commented Jun 4, 2022 •

edited

mattzett commented Jun 29, 2022 •

edited

mattzett commented Jun 29, 2022 •

edited

paulinchin commented Jun 30, 2022 •

edited