Support for MPI.jl if configured to use the system MPI library.

[jmark]

I prefer for several reasons to use the system MPI library as backend for MPI.jl
Trixi should fully support this.

When using Trixi I get the following warning:

Warning: Error requiring `MPICH_jll` from `MPI`
[...]
MPICH_jll cannot be loaded: MPI.jl is configured to use the system MPI library
[...]

Some examples, e.g., the default_example() nevertheless run.

[sloede]

I am not sure if this is a Trixi error or an upstream issue. We at HLRS can run Trixi with the system MPI library without issue on multiple systems. On what kind of system did you get this error and is it possible for you to create an MWE to reproduce this behavior?

[jmark]

My system is on my workstatioin with "Arch Linux". I use OpenMPI as MPI implementation. So, using MPICH_jll, which is a dependency of Trixi, might no like this.

[jmark]

It seems the warning stems from using P4est, which aparently does not support OpenMPI according to https://juliahub.com/ui/Packages/P4est_jll/o3fDR/2.8.0+0?page=0

[sloede]

Ah, yes. I can see the problem with P4est and I can reproduce it as follows:

# Start in empty directory
mkdir mwe-mpi && cd mwe-mpi

# Install P4est, MPI
julia --project=. -e 'using Pkg; Pkg.add(["P4est", "MPI"])'

# Configure MPI to use default MPI JLL
julia --project=. -e 'ENV["JULIA_MPI_BINARY"]=""; using Pkg; Pkg.build("MPI"; verbose=true)'
# Output:
#      Building MPI → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/d56a80d8cf8b9dc3050116346b3d83432b1912c0/build.log`
# [ Info: using default MPI jll                        ]  0/1

# Works:
julia --project=. -e 'using P4est'

# Configure MPI to use system binary
julia --project=. -e 'ENV["JULIA_MPI_BINARY"]="system"; using Pkg; Pkg.build("MPI"; verbose=true)'
# Output:
#     Building MPI → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/d56a80d8cf8b9dc3050116346b3d83432b1912c0/build.log`
# [ Info: using system MPI                             ]  0/1
# ┌ Info: Using implementation
# │   libmpi = "libmpi"
# │   mpiexec_cmd = `mpiexec`
# └   MPI_LIBRARY_VERSION_STRING = "Open MPI v4.0.3, package: Debian OpenMPI, ident: 4.0.3, repo rev: v4.0.3, Mar 03, 2020\0"
# ┌ Info: MPI implementation detected
# │   impl = OpenMPI::MPIImpl = 2
# │   version = v"4.0.3"
# └   abi = "OpenMPI"
# Precompiling project...
#   2 dependencies successfully precompiled in 10 seconds (15 already precompiled)

# Triggers error now:
julia --project=. -e 'using P4est'
# Output:
# ┌ Warning: Error requiring `MPICH_jll` from `MPI`
# │   exception =
# │    MPICH_jll cannot be loaded: MPI.jl is configured to use the system MPI library
# │    Stacktrace:
# │      [1] error(s::String)
# │        @ Base ./error.jl:33
# │      [2] top-level scope
# │        @ none:1
# │      [3] eval
# │        @ ./boot.jl:373 [inlined]
# │      [4] eval
# │        @ ~/.julia/packages/MPI/08SPr/src/MPI.jl:1 [inlined]
# │      [5] (::MPI.var"#3#12")()
# │        @ MPI ~/.julia/packages/Requires/Z8rfN/src/require.jl:101
# │      [6] macro expansion
# │        @ timing.jl:299 [inlined]
# │      [7] err(f::Any, listener::Module, modname::String, file::String, line::Any)
# │        @ Requires ~/.julia/packages/Requires/Z8rfN/src/require.jl:47
# │      [8] (::MPI.var"#2#11")()
# │        @ MPI ~/.julia/packages/Requires/Z8rfN/src/require.jl:100
# │      [9] withpath(f::Any, path::String)
# │        @ Requires ~/.julia/packages/Requires/Z8rfN/src/require.jl:37
# │     [10] (::MPI.var"#1#10")()
# │        @ MPI ~/.julia/packages/Requires/Z8rfN/src/require.jl:99
# │     [11] #invokelatest#2
# │        @ ./essentials.jl:716 [inlined]
# │     [12] invokelatest
# │        @ ./essentials.jl:714 [inlined]
# │     [13] foreach(f::typeof(Base.invokelatest), itr::Vector{Function})
# │        @ Base ./abstractarray.jl:2694
# │     [14] loadpkg(pkg::Base.PkgId)
# │        @ Requires ~/.julia/packages/Requires/Z8rfN/src/require.jl:27
# │     [15] #invokelatest#2
# │        @ ./essentials.jl:716 [inlined]
# │     [16] invokelatest
# │        @ ./essentials.jl:714 [inlined]
# │     [17] _tryrequire_from_serialized(modkey::Base.PkgId, build_id::UInt64, modpath::String)
# │        @ Base ./loading.jl:787
# │     [18] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
# │        @ Base ./loading.jl:843
# │     [19] _tryrequire_from_serialized(modkey::Base.PkgId, build_id::UInt64, modpath::String)
# │        @ Base ./loading.jl:783
# │     [20] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
# │        @ Base ./loading.jl:843
# │     [21] _require(pkg::Base.PkgId)
# │        @ Base ./loading.jl:1097
# │     [22] require(uuidkey::Base.PkgId)
# │        @ Base ./loading.jl:1013
# │     [23] require(into::Module, mod::Symbol)
# │        @ Base ./loading.jl:997
# │     [24] eval
# │        @ ./boot.jl:373 [inlined]
# │     [25] exec_options(opts::Base.JLOptions)
# │        @ Base ./client.jl:268
# │     [26] _start()
# │        @ Base ./client.jl:495
# └ @ Requires ~/.julia/packages/Requires/Z8rfN/src/require.jl:51

I don't think it is possible to just change the underlying MPI implementation in MPI and continue to use Trixi, since this will clash with P4est's use of MPI. Generally speaking, this is a known issue with MPI.jl and using MPI-dependent packages in Julia, which will (hopefully, someday) be resolved by using something like MPItrampoline.

More specifically, however, we should consider adding a check somewhere in the P4est.jl stack that produces a more legible error message (if that's possible at all). Maybe @lchristm knows more about this issue and/or how to fix it?

[ranocha]

Currently, the workaround is to build P4est.jl also using the system MPI, correct?

[lchristm]

Yes, they must use the same. That is, in order to fix this, one has to configure P4est.jl to use a custom build of p4est (built against the system MPI), which is quite cumbersome at the moment (see https://github.com/trixi-framework/P4est.jl/blob/main/README.md for details).

I'll check next week if there's a way to provide a more helpful error message/warning.

[ranocha]

I don't think we should invest time into trying to get a better error message right now. We will change P4est.jl completely soon (as soon as possible), so I would like to redirect all free work time to setting up the new infrastructure instead.

[ranocha]

Could you built MPI.jl and P4est.jl using your local versions, @jmark?

[jmark]

I tried according to Using a custom build of p4est several weeks ago and failed. I got lots of error messages. But I do not rember the details, anymore.

Trying to do this today, for example, gives me:

ERROR: LoadError: Unsatisfiable requirements detected for package Clang_jll [0ee61d77]:
 Clang_jll [0ee61d77] log:
 ├─possible versions are: 9.0.1-14.0.5 or uninstalled
 ├─restricted by julia compatibility requirements to versions: 9.0.1-12.0.1 or uninstalled
 ├─restricted by compatibility requirements with CBindingGen [308a6e0c] to versions: 9.0.1-11.0.1
 │ └─CBindingGen [308a6e0c] log:
 │   ├─possible versions are: 0.1.0-0.4.5 or uninstalled
 │   └─restricted to versions 0.4.5 by an explicit requirement, leaving only versions 0.4.5
 └─restricted by compatibility requirements with libLLVM_jll [8f36deef] to versions: 12.0.0-12.0.1 or uninstalled — no versions left
   └─libLLVM_jll [8f36deef] log:
     └─possible versions are: 12.0.1 or uninstalled

Anyway, I simply bypass this issue and concentrate my efforts on supporting t8code in Trixi.

[ranocha]

Can you rebuild the bindings with Julia v1.6? Currently, the version of CBinding that P4est.jl uses does not work with Julia v1.7 or newer.

[ranocha]

@jmark Could you please try with the new versions of MPI.jl and P4est.jl? The build process should be significantly simpler.

[jmark]

It works as intented. Closing the issue.