Compilation issues with the cubeGenerator in 'preprocessing/meshing/cube_c'

[montrie]

Hi,

I've been trying to compile the cubeGenerator from the cube_c preprocessing directory. I searched the existing documentation for hints on how to compile the cubeGenerator, but I couldn't find something that would help me.

Running cmake as seen below leads to successfully generated build files:

[ga27koz2@cs1 SeisSol]$ cd preprocessing/meshing/cube_c
[ga27koz2@cs1 cube_c]$ mkdir build && cd build
[ga27koz2@cs1 build]$ CC=gcc CXX=mpiCC FC=gfortran cmake ..
-- The C compiler identification is GNU 11.0.0
-- The CXX compiler identification is GNU 11.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/sw/a64fx/gcc/11.0.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/mpi/gcc/openmpi-4.0.4rc3/bin/mpiCC - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found HDF5: /home/ga27koz2/lib/libhdf5.a;/usr/lib64/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.10.8") found components: C HL
-- Found MPI_C: /usr/mpi/gcc/openmpi-4.0.4rc3/lib64/libmpi.so (found version "3.1")
-- Found MPI_CXX: /usr/mpi/gcc/openmpi-4.0.4rc3/bin/mpiCC (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ga27koz2/SeisSol/preprocessing/meshing/cube_c/build

However, when i try to compile an executable, i.e. run either make, make all, or make cubeGenerator, I get the following error:

[ga27koz2@cs1 build]$ make cubeGenerator
Scanning dependencies of target cubeGenerator
[ 50%] Building CXX object CMakeFiles/cubeGenerator.dir/src/main.cpp.o
[100%] Linking CXX executable cubeGenerator
/opt/cray/pe/cce-sve/10.0.1/binutils/aarch64/aarch64-unknown-linux-gnu/bin/ld: /home/ga27koz2/lib/libhdf5.a(H5Dchunk.o): in function `H5D__chunk_cmp_addr':
H5Dchunk.c:(.text+0xfc4): undefined reference to `MPI_Aint_diff'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/cubeGenerator.dir/build.make:111: cubeGenerator] Error 1
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/cubeGenerator.dir/all] Error 2
make: *** [Makefile:103: all] Error 2

I looked through existing issues/discussions in this repository for a similar problem, but I couldn't find anything related.

Any help is appreciated!

Additional context
Current branch:

[ga27koz2@cs1 cube_c]$ git status
On branch fugaku
Your branch is up to date with 'origin/fugaku'.

Commit id: 70232f8

SeisSol is installed on a cluster system using the A64FX cpu. I can successfully compile and run the SeisSol-proxy executable, so I think the dependencies as described in the documentation are installed correctly.
Below are the loaded modules.

[ga27koz2@cs1 build]$ module list
Currently Loaded Modulefiles:
 1) cpe-cray                  3) craype/2.7.0(default)   5) craype-network-infiniband        7) cray-mvapich2_nogpu/2.3.4(default)   9) cmake/3.18.3
 2) cce-sve/10.0.1(default)   4) craype-arm-nsp1         6) cray-libsci/20.10.1.2(default)   8) gcc/11.0.0                          10) likwid/likwid-5.2.1-gcc-11.0.0

[sebwolf-de]

MPI_Aint_diff is a MPI-3.1 feature. It seems to me that you've found MPI v3.1 on your system, but apparently it is not linked correctly.
The proxy does not have a HDF5 nor an MPI dependency, so probably that's why you didn't run into this problem earlier. Can you compile the proper SeisSol executable?
Otherwise, I'd try to find the location of the MPI library and check, whether it supports MPI_Aint_diff:
On the LRZ linux cluster, e.g.:

09:01 $ nm /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/x86_64/intel-oneapi-mpi/2021.5.1-gcc-2pu5diq/mpi/2021.5.1//lib/release/libmpi.so.12 | grep -i aint
00000000000f5090 W MPI_Aint_add
00000000000f5160 W MPI_Aint_diff
000000000020f9d0 t MPID_Aint_add
000000000020f9e0 t MPID_Aint_diff
0000000000892170 t MPII_Datatype_get_contents_aints
00000000000f5090 T PMPI_Aint_add
00000000000f5160 T PMPI_Aint_diff

[krenzland]

Yes, I'd also assume that this is caused by HDF5 linking with a different MPI library. The cube generator should not have a direct dependency on MPI, only indirect via netcdf/hdf5

[montrie]

Thank you for the response!

So I noticed that I turned HDF5 off when compiling SeisSol. After turning it on, calling make ends with a similar error:

...
[ 99%] Linking CXX executable SeisSol_proxy_Release_da64fx_6_elastic
/opt/cray/pe/cce-sve/10.0.1/binutils/aarch64/aarch64-unknown-linux-gnu/bin/ld: /home/ga27koz2/lib/libhdf5.a(H5Dchunk.o): in function `H5D__chunk_cmp_addr':
H5Dchunk.c:(.text+0xfc4): undefined reference to `MPI_Aint_diff'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/SeisSol-proxy.dir/build.make:284: SeisSol_proxy_Release_da64fx_6_elastic] Error 1
make[1]: *** [CMakeFiles/Makefile2:106: CMakeFiles/SeisSol-proxy.dir/all] Error 2
make: *** [Makefile:103: all] Error 2

When searching the libmpi.so as linked above (see: Found MPI_C:), I found that it doesn't support MPI_Aint_diff

 nm /usr/mpi/gcc/openmpi-4.0.4rc3/lib64/libmpi.so | grep -i aint
000000000007b760 T mpi_aint_add_f90
000000000007b778 T mpi_aint_add_f90_
000000000007b790 T mpi_aint_add_f90__
000000000007b748 T MPI_AINT_ADD_F90
000000000007b7c0 T mpi_aint_diff_f90
000000000007b7d8 T mpi_aint_diff_f90_
000000000007b7f0 T mpi_aint_diff_f90__
000000000007b7a8 T MPI_AINT_DIFF_F90
000000000002b848 T ompi_attr_create_keyval_aint
000000000002c318 T ompi_attr_get_aint
000000000002bee0 T ompi_attr_set_aint
00000000000f2c58 D ompi_mpi_aint

However, there seem to be 2 versions of mpicc:

[ga27koz2@cs1 build]$ whereis mpicc
mpicc: /opt/cray/pe/cray-mvapich2_nogpu/2.3.4/infiniband/cray/10.0/bin/mpicc /usr/mpi/gcc/openmpi-4.0.4rc3/bin/mpicc

When I search through the cray version of mpicc, I find that it does support MPI_Aint_diff:

[ga27koz2@cs1 build]$ nm /opt/cray/pe/cray-mvapich2_nogpu/2.3.4/infiniband/cray/10.0/lib/libmpi.so.12 | grep -i aint
000000000032aad8 W MPI_Aint_add
000000000032ab18 W MPI_Aint_diff
00000000003c5d28 T MPID_Aint_add
00000000003c5d30 T MPID_Aint_diff
000000000042cf28 T MPIDI_Datatype_get_contents_aints
000000000032aad8 T PMPI_Aint_add
000000000032ab18 T PMPI_Aint_diff

Calling which mpicc returns a path to the version of mpicc that does support MPI_Aint_diff. I manually installed HDF5 as described in the documentation, so I would expect HDF5 to be linked with the MPI version that does support MPI_Aint_diff. When I set CC=mpicc like I did for HDF5, the MPI_C field points to the version of mpicc that support MPI_Aint_diff, but MPI_CXX still points to the openmpi-4.0.4rc3 version. (Omitting the CXX completely from the cmake call leads to more undefined MPI types.)

[ga27koz2@cs1 build]$ CC=mpicc CXX=mpiCC FC=mpif90 cmake ..
-- The C compiler identification is GNU 11.0.0
-- The CXX compiler identification is GNU 11.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/cray/pe/cray-mvapich2_nogpu/2.3.4/infiniband/cray/10.0/bin/mpicc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/mpi/gcc/openmpi-4.0.4rc3/bin/mpiCC - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found HDF5: /home/ga27koz2/lib/libhdf5.a;/usr/lib64/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.10.8") found components: C HL
-- Found MPI_C: /opt/cray/pe/cray-mvapich2_nogpu/2.3.4/infiniband/cray/10.0/bin/mpicc (found version "3.1")
-- Found MPI_CXX: /usr/mpi/gcc/openmpi-4.0.4rc3/bin/mpiCC (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ga27koz2/SeisSol/preprocessing/meshing/cube_c/build

Finally, i tried specifying CXX=mpicxx, since that compiler seems to point to a path that is similar to the mpicc compiler. However, cmake now complains that the language dialect CXX17 is required:

[ga27koz2@cs1 build]$ CC=mpicc CXX=mpicxx cmake ..
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is Cray 10.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/cray/pe/cray-mvapich2_nogpu/2.3.4/infiniband/cray/10.0/bin/mpicc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/cray/pe/cray-mvapich2_nogpu/2.3.4/infiniband/cray/10.0/bin/mpicxx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found HDF5: /home/ga27koz2/lib/libhdf5.a;/usr/lib64/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.10.8") found components: C HL
-- Found MPI_C: /opt/cray/pe/cray-mvapich2_nogpu/2.3.4/infiniband/cray/10.0/bin/mpicc (found version "3.1")
CMake Error in /home/ga27koz2/SeisSol/preprocessing/meshing/cube_c/build/CMakeFiles/CMakeTmp/CMakeLists.txt:
  Target "cmTC_3bbb1" requires the language dialect "CXX17" (with compiler
  extensions), but CMake does not know the compile flags to use to enable it.


CMake Error at /home/sw/a64fx/cmake/3.18.3/share/cmake-3.18/Modules/FindMPI.cmake:1213 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /home/sw/a64fx/cmake/3.18.3/share/cmake-3.18/Modules/FindMPI.cmake:1264 (_MPI_try_staged_settings)
  /home/sw/a64fx/cmake/3.18.3/share/cmake-3.18/Modules/FindMPI.cmake:1427 (_MPI_check_lang_works)
  CMakeLists.txt:40 (find_package)


-- Configuring incomplete, errors occurred!

So my guess would be that the (MPI) compiler I specify in the CXX flag simply doesn't support the MPI_Aint_diff type. However, I'm not quite sure whether the A64FX on the LRZ linux cluster provides a suitable compiler.