Master fails to Build Under Ubuntu 22-04 LTS

[OPMUSER]

I get this today when trying to build master:

[akva2]

If u want to build with mpi support, you need the mpi enabled hdf5 library. And a clean build directory.

[OPMUSER]

@akva2

If u want to build with mpi support, you need the mpi enabled hdf5 library. And a clean build directory.

Easy for you, not so easy for me :-) What is the command to install the mpi enabled hdf5 library:

sudo apt-get install XXXXXX

Also, does it require a clean build in all four repositories?

[blattms]

If u want to build with mpi support, you need the mpi enabled hdf5 library. And a clean build directory.

Does this mean that hdf5 is a required dependency?

[alfbr]

If u want to build with mpi support, you need the mpi enabled hdf5 library. And a clean build directory.

Does this mean that hdf5 is a required dependency?

Indeed, this has bitten us twice now :)

[akva2]

No it is not a required but serial hdf5 library will not work with a mpi build.

apt install libhdf5-openmpi-dev. Onyl simulators needs clean build.

[OPMUSER]

@akva2 Thanks, will try this now.

[bska]

Does this mean that hdf5 is a required dependency?

Indeed, this has bitten us twice now :)

For what it's worth, If you don't need (or want) to test/enable the experimental serialisation/perfect restart support you can disable all of HDF5 by setting

CMAKE_DISABLE_FIND_PACKAGE_HDF5=True

when configuring the build. That will allow you to build both sequential and MPI versions of the simulator irrespective of what HDF5 libraries may or may not be available on your system.

If you do want to enable the serialisation support in an MPI build then you'll need the parallel HDF5 libraries as mentioned earlier.

[OPMUSER]

Does not work, I get this based on a clean build directory and installing the mpi library as stated above:

[akva2]

Show me the cmake output for opm-simulators.

[OPMUSER]

@akva2 you need to be a bit more specific, as there are many cmake files

[akva2]

Don't know how to be more specific. Show me the cmake output when you configure opm-simulators. In particular what is printed for hdf5.

[OPMUSER]

Is this it.

CMakeFiles.zip

[akva2]

No, i mean terminal output. Only on my phone as i am on vacation so hard to open zip files.

[OPMUSER]

Is this what you need:

CMakeError.log
CMakeOutput.log

[blattms]

Let's cut this short:
@OPMUSER I think @akva2 meant that you need to install the mpi version of hdf5: apt-get install libhdf5-mpi-dev
The output requested was the logfile cmake.log that you can create with cmake . &> cmake.log in the build directory

@akva2 Hands away from the keyboard on vacation...

[OPMUSER]

It's installed

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libhdf5-mpi-dev is already the newest version (1.10.7+repack-4ubuntu2).
The following packages were automatically installed and are no longer required:
  libann0 libcdt5 libcgraph6 libgts-0.7-5 libgts-bin libgvc6 libgvpr2 liblab-gamut1 libllvm13 libpathplan4 linux-headers-5.15.0-50
  linux-headers-5.15.0-50-generic linux-headers-5.15.0-52 linux-headers-5.15.0-52-generic linux-headers-5.15.0-53 linux-headers-5.15.0-53-generic
  linux-headers-5.15.0-56 linux-headers-5.15.0-56-generic linux-headers-5.15.0-57 linux-headers-5.15.0-57-generic linux-headers-5.15.0-58
  linux-headers-5.15.0-58-generic linux-image-5.15.0-50-generic linux-image-5.15.0-52-generic linux-image-5.15.0-53-generic
  linux-image-5.15.0-56-generic linux-image-5.15.0-57-generic linux-image-5.15.0-58-generic linux-modules-5.15.0-50-generic
  linux-modules-5.15.0-52-generic linux-modules-5.15.0-53-generic linux-modules-5.15.0-56-generic linux-modules-5.15.0-57-generic
  linux-modules-5.15.0-58-generic linux-modules-extra-5.15.0-50-generic linux-modules-extra-5.15.0-52-generic linux-modules-extra-5.15.0-53-generic
  linux-modules-extra-5.15.0-56-generic linux-modules-extra-5.15.0-57-generic linux-modules-extra-5.15.0-58-generic
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.

And I keep on getting this:

-- Generating debug symbols: -ggdb3
-- Looking for strip utility
-- Looking for strip utility - found
-- Performing Test HAVE_DYNAMIC_BOOST_TEST
-- Performing Test HAVE_DYNAMIC_BOOST_TEST - Success
-- Writing config file "/home/david/OPM/GitHub/opm-simulators/build/config.h"...
-- This build defaults to installing in /usr/local
-- Writing version information to local header project-version.h
-- Configuring done
-- Generating done
-- Build files have been written to: /home/david/OPM/GitHub/opm-simulators/build
[  0%] Updating version information
[  0%] Built target update-version
Consolidate compiler generated dependencies of target opmsimulators
[  0%] Building CXX object CMakeFiles/opmsimulators.dir/opm/simulators/utils/HDF5File.cpp.o
/home/david/OPM/GitHub/opm-simulators/opm/simulators/utils/HDF5File.cpp: In constructor ‘Opm::HDF5File::HDF5File(const string&, Opm::HDF5File::OpenMode, Opm::Parallel::Communication)’:
/home/david/OPM/GitHub/opm-simulators/opm/simulators/utils/HDF5File.cpp:63:9: error: ‘H5Pset_fapl_mpio’ was not declared in this scope; did you mean ‘H5Pset_fapl_stdio’?
   63 |         H5Pset_fapl_mpio(acc_tpl, comm_, info);
      |         ^~~~~~~~~~~~~~~~
      |         H5Pset_fapl_stdio
make[2]: *** [CMakeFiles/opmsimulators.dir/build.make:1630: CMakeFiles/opmsimulators.dir/opm/simulators/utils/HDF5File.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:225: CMakeFiles/opmsimulators.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
END

[GitPaean]

Hei, this file is an example what @akva2 want to see.

It is the screen output for cmake,

cmake_log.txt

from inside, you can see the following lines,

-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so;/usr/lib/x86_64-linux-gnu/libcrypto.so;/usr/lib/x86_64-linux-gnu/libcurl.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.a;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.7")  

[GitPaean]

Very likely your cmake found the wrong HDF5 header file, probably the serial HDF5 header file? That was how I got the exactly the same compilation error message with you.

Deleting the content in the build folder and redo the cmake helped me to fix the compilation.

[GitPaean]

And if I did not clean the build folder, just redo cmake, I obtained the following,

-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libcrypto.so;/usr/lib/x86_64-linux-gnu/libcurl.so;/usr/lib/x86_64-linux-gnu/libpthread.a;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.a;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.7")  

and I got the same compilation failure.

[blattms]

OK, I have verified this and the buildsystem does not detect if it is an MPI build but there is just a serial hdf5 library (package libhdf5-dev with the mpi counterpart). I think that is the situation @OPMUSER was in.
I guess we should make cmake fail with a decent error message in this case instead of trying to compile things.

On the other hand not having restart in a serial build seems rather unfortunate. Maybe we should think about requiring MPI?

[akva2]

You can have both libraries installed at the same time. But yes, we should fail/disable if mpi is enabled and only the serial lib was found

[blattms]

Are you saying that with serial hdf5 and serial OPM we have restart?

[akva2]

Yes.

[blattms]

@OPMUSER Indeed you have to install libhdf5-mpi-dev (apt-get install libhdf5-mpi-dev) and do a clean build of opm-simulators (make -j4 clean all) to fix this.

[OPMUSER]

Okay, I still cannot build master. I have deleted the build directory and re-created it, and I have run cmake .. -DBUILD_TESTING=OFF &> cmake.log. And I have done this several times with the same result, As you can see below, cmake only finds the serial version.

-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libcrypto.so;/usr/lib/x86_64-linux-gnu/libcurl.so;/usr/lib/x86_64-linux-gnu/libpthread.a;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.a;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.7")  
-- Setting CUDA host compiler CMAKE_CUDA_HOST_COMPILER to /usr/bin/c++ to prevent incompatibilities. Note that this might report that there is not CUDA compiler if your system's CUDA compiler does not support /usr/bin/c++.

And both versions of HD5 are installed.

cmake.log

[OPMUSER]

On the other hand not having restart in a serial build seems rather unfortunate. Maybe we should think about requiring MPI?

Dumb question. Why do we need to support serial build? Why not just support mpi, as all modern computers have multiple CPU's.

[OPMUSER]

blattms added a commit to blattms/opm-simulators that referenced this issue Feb 21, 2023
@blattms
[fix] For parallel build with serial hdf5 issue warning and skip hdf5

I don't think this will work in my case. The problem is cmake finds the wrong version of the hdf5 libraries, when both serial and mpi versions are installed. So somehow we need to link a serial build with the serial hdf5 libraries, similarly for the mpi libraries. If the correct library is not found, then terminate the build.

If I do the following:

1. Uninstall the serial version of the hdf5 libraries (sudo apt-get remove --auto-remove libhdf5-dev).
2. Delete the build directory and re-created it.
3. Run cmake .. -DBUILD_TESTING=OFF &> cmake.log

Then everything works.

cmake.log

[GitPaean]

Yes, now you found the mpi version of HDF5,

-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so;/usr/lib/x86_64-linux-gnu/libcrypto.so;/usr/lib/x86_64-linux-gnu/libcurl.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.a;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.7")  

not totally sure whether removing the serial HDF5 was necessary, but I removed it also on my side.

[OPMUSER]

Closing as I have a workaround, but we may still have a build issue.

[blattms]

I don't think this will work in my case.

Well if the test in older versions of cmake don't work, then there is not much we can do. At least my fix fixes the case where there is only serial hdf5 and a parallel build.