Could Not find MPI Fortran (missing: MPI Fortran WORKS) when building solver

[DavidChampion24]

Could Not find MPI Fortran (missing: MPI Fortran WORKS) when building solver

Dear developers,

I am a new user of the MFC. When I try to build the solver (using NVIDIA's NVHPC SDK) using the command,
./mfc.sh build --gpu -j 8.
The following error was hinted at. I want to ask how to solve this issue when building the solver.
Thanks for your time and help.

[sbryngelson]

Hi @DavidChampion24 — thanks for opening this issue. To help us figure out what’s going wrong with finding MPI Fortran, could you provide a few more details? Otherwise, it’s hard to reproduce or diagnose. Some useful info would be:

1. Operating System
   Which OS are you using (e.g., Ubuntu version, CentOS, etc.)?
2. MPI implementation & version
   What MPI are you using (e.g., OpenMPI, MPICH, Intel MPI, etc.)? Which version?
3. Fortran compiler
   Which Fortran compiler (and version) is in use? Is it from the NVHPC SDK, or a system/compiler-provided one?
4. Environment settings
   Are there any environment variables set that might affect MPI or compilers (e.g., MPI_HOME, PATH, LD_LIBRARY_PATH, FC, CC, MPIF90, etc.)?
5. Command used & configuration options
   We see you used ./mfc.sh build --gpu -j 8. Did you also supply any other flags? Did you try disabling GPU or other options to isolate MPI Fortran?

If you can share this info, we’ll have a much better chance of diagnosing what’s missing or misconfigured.

[DavidChampion24]

Dear prof. Bryngelson,

Thanks for your reply @sbryngelson .

The following are the details,

1. Operating System, Kubuntu 24.04 (based on ubuntu).
2. MPI implementation & version, OpenMPI 4.1.6
3. Fortran compiler, it's from the NVHPC SDK, version 25.7
4. I have set some environment variables as follows,

added for openmpi
export PATH=/usr/lib/openmpi/bin:$PATH
export LD_LIBRARY_PATH=/usr/lib/openmpi/lib:$LD_LIBRARY_PATH

NVIDIA HPC Compilers
export NVHPC_ROOT=/opt/nvidia/hpc_sdk/Linux_x86_64/25.7
export PATH=$NVHPC_ROOT/compilers/bin:$PATH
set for nvc, nvc++, nvfortran
export C=$NVHPC_ROOT/compilers/bin/nvc
export CXX=$NVHPC_ROOT/compilers/bin/nvc++
export FC=$NVHPC_ROOT/compilers/bin/nvfortran

5. Command used & configuration options,
   just tested by ./mfc.sh build --gpu -j 8 (no other flags)
   btw, I tested using ./mfc.sh build --no-mpi to build the solver. The following error was hinted at.

[sbryngelson]

There's a bug in the latest compiler. Downgrade to nvhpc 24.5. Also, your first post suggests you did not have FC set.

[DavidChampion24]

Hi @sbryngelson . After I downgraded to the NVHPC 24.5 version, the same error was hinted at as in the first post. I guess it's the problem of environment settings. Are there some recommended environment settings after installing the NVHPC? For example, the MPI is OpenMPI.

[sbryngelson]

If you build with --no-mpi you don't need any MPI wrappers at all. Did you try following the instructions here (https://mflowcode.github.io/documentation/md_getting-started.html)?

[DavidChampion24]

Thanks @sbryngelson . Yes, I have followed the Getting Started instructions. Now, the solver can be build successfully with --no-mpi and --no-mpi --gpu.

btw, when I try to run the first case with ./mfc.sh run examples/0D_bubblecollapse_adap/case.py. The following error was hinted at. It suggest the --computer option.

[sbryngelson]

So what exactly is the problem at the moment? I've lost track of what works and what does not. Are you on a SLURM+GPU cluster or your laptop?

[DavidChampion24]

Sorry for not describing the problem clearly. I use my desktop. Now the problem is that when I try to run the example with ./mfc.sh run examples/0D_bubblecollapse_adap/case.py, the following error is shown.

[sbryngelson]

delete build directory, rebuild without gpu, and run again

[DavidChampion24]

Hi @sbryngelson . After deleting build directory, rebuilding without gpu (also no-mpi), and runing. The following error is still shown,

[sbryngelson]

The error Could NOT find MPI Fortran (missing: MPI_Fortran_WORKS) usually means CMake is testing MPI with nvfortran, but the mpif90/mpifort on your PATH was built with a different Fortran compiler (e.g., system gfortran).

Fix: use the OpenMPI that ships with the NVIDIA HPC SDK (or rebuild MPI with NVHPC), and make sure CMake sees the matching compilers/wrappers.

1) Verify compilers & MPI wrappers

which nvfortran && nvfortran --version
which mpifort && mpifort --show
ompi_info | head -n 20

You should see the NVHPC paths (e.g., /opt/nvidia/hpc_sdk/.../comm_libs/mpi/bin/mpifort) and nvfortran in the mpifort --show output.
If not, load NVHPC or export PATHs (adjust version/path as installed):

# Example default install path; change 25.7 to your version
export NVHPC=/opt/nvidia/hpc_sdk
export PATH=$NVHPC/Linux_x86_64/25.7/compilers/bin:$PATH
export PATH=$NVHPC/Linux_x86_64/25.7/comm_libs/mpi/bin:$PATH
export LD_LIBRARY_PATH=$NVHPC/Linux_x86_64/25.7/comm_libs/mpi/lib:$LD_LIBRARY_PATH
# Make sure the MPI wrappers use NVHPC compilers
export OMPI_CC=nvc
export OMPI_CXX=nvc++
export OMPI_FC=nvfortran

2) Reconfigure & build MFC with explicit compilers

Start from a clean build and set compilers/wrappers for CMake via mfc.sh:

./mfc.sh clean
CC=nvc CXX=nvc++ FC=nvfortran MPI_Fortran_COMPILER=mpifort \
CMAKE_PREFIX_PATH="$NVHPC/Linux_x86_64/25.7/comm_libs/mpi" \
./mfc.sh build --gpu -j 8

(If you have a system OpenMPI installed, make sure the NVHPC MPI bin/ comes first on your PATH so CMake finds the right one.)

3) If detection still fails, capture logs

Re-run configure with CMake’s find-debugging:

./mfc.sh clean
MFC_CMAKE_FLAGS="--trace-expand --debug-find" \
CC=nvc CXX=nvc++ FC=nvfortran MPI_Fortran_COMPILER=mpifort \
CMAKE_PREFIX_PATH="$NVHPC/Linux_x86_64/25.7/comm_libs/mpi" \
./mfc.sh build --gpu -j 8 2>&1 | tee cmake-findmpi.log

Attach cmake-findmpi.log so we can see exactly why MPI_Fortran_WORKS is failing.

4) Sanity check

Try a tiny MPI Fortran test with the same toolchain:

cat > hello_mpi.f90 <<'EOF'
program main
  use mpi
  integer :: ierr, rank
  call MPI_Init(ierr)
  call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)
  print *, 'Hello from rank', rank
  call MPI_Finalize(ierr)
end program
EOF
mpifort hello_mpi.f90 -o hello_mpi
mpirun -np 2 ./hello_mpi

If this uses nvfortran and runs, the environment is correct and MFC should configure.

[DavidChampion24]

Hi @sbryngelson . After the steps 1-3, the following error is shown,

For the step 4, sanity test, the following is shown,

Attached is the cmake-findmpi.log file.

cmake-findmpi.log

[sbryngelson]

It's running with error because it doesn't want to execute on your particular GPU for whatever reason. Specify --no-gpu and it should work.

[henryleberre]

If you export MFC_CUDA_CC=120 and rebuild, syscheck should pass. See

MFC/CMakeLists.txt

Line 213 in eb61616

if (DEFINED ENV{MFC_CUDA_CC})

[DavidChampion24]

Hi @henryleberre . I have tried exporting MFC_CUDA_CC=120, and it still fails as follows.

For the error (as follows), may I ask how to support compile files with -gpu=cc120? Are some modifications needed for /src/syscheck/syscheck.fpp or other recommendations?

Accelerator Fatal Error: This file was compiled: -acc=gpu -gpu=cc50 -gpu=cc60 -gpu=cc60 -gpu=cc70 -gpu=cc75 -gpu=cc80 -gpu=cc80 -gpu=cc86 -gpu=cc80 -gpu=cc89 -
Rebuild this file with -gpu=cc120 to use NVIDIA Tesla GPU 0