Library operates incorrectly for a mixed Fortran/C++ code

[william-dawson]

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Environment (conda list):

``` $ conda list _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge binutils 2.36.1 hdd6e379_2 conda-forge binutils_impl_linux-64 2.36.1 h193b22a_2 conda-forge binutils_linux-64 2.36 hf3e587d_1 conda-forge c-compiler 1.3.0 h7f98852_0 conda-forge compilers 1.3.0 ha770c72_0 conda-forge cxx-compiler 1.3.0 h4bd325d_0 conda-forge fortran-compiler 1.3.0 h1990efc_0 conda-forge gcc 9.4.0 h192d537_1 conda-forge gcc_impl_linux-64 9.4.0 h03d3576_11 conda-forge gcc_linux-64 9.4.0 h391b98a_1 conda-forge gfortran 9.4.0 h2018a41_1 conda-forge gfortran_impl_linux-64 9.4.0 h0003116_11 conda-forge gfortran_linux-64 9.4.0 hf0ab688_1 conda-forge gxx 9.4.0 h192d537_1 conda-forge gxx_impl_linux-64 9.4.0 h03d3576_11 conda-forge gxx_linux-64 9.4.0 h0316aca_1 conda-forge kernel-headers_linux-64 2.6.32 he073ed8_14 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge libgcc-devel_linux-64 9.4.0 hd854feb_11 conda-forge libgcc-ng 11.2.0 h1d223b6_11 conda-forge libgfortran-ng 11.2.0 h69a702a_11 conda-forge libgfortran5 11.2.0 h5c6108e_11 conda-forge libgomp 11.2.0 h1d223b6_11 conda-forge libsanitizer 9.4.0 h79bfe98_11 conda-forge libstdcxx-devel_linux-64 9.4.0 hd854feb_11 conda-forge libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge mpi 1.0 mpich conda-forge mpich 3.4.2 h846660c_100 conda-forge sysroot_linux-64 2.12 he073ed8_14 conda-forge ```

Details about conda and system ( conda info ):

$ conda info
     active environment : test_conda
    active env location : /home/wddawson/miniconda3/envs/test_conda
            shell level : 2
       user config file : /home/wddawson/.condarc
 populated config files :
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.7.6.final.0
       virtual packages : __linux=4.4.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/wddawson/miniconda3  (writable)
      conda av data dir : /home/wddawson/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/wddawson/miniconda3/pkgs
                          /home/wddawson/.conda/pkgs
       envs directories : /home/wddawson/miniconda3/envs
                          /home/wddawson/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.3 requests/2.26.0 CPython/3.7.6 Linux/4.4.0-19041-Microsoft ubuntu/20.04.3 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

I am having trouble using the MPICH that comes with conda to compile a mixed C++/Fortran program. Below I will detail a minimal example.

An environment can be setup like this (environment.yml):

name: test_conda
channels:
  - conda-forge
dependencies:
  - compilers
  - mpich

conda env create -f environment.yml  
conda activate test_conda

I created a Fortran subroutine that might be called from C++ (flib.f90):

MODULE FLIB
CONTAINS
  SUBROUTINE MyRoutine() BIND(C, name="MyRoutine")
    USE MPI
    IMPLICIT NONE
    INTEGER :: foo, err
    foo = 1
    WRITE(*,*) ">>", foo
    CALL MPI_Allreduce(MPI_IN_PLACE, foo, 1, &
       & MPI_INT, MPI_SUM, MPI_COMM_WORLD, err)
    WRITE(*,*) "<<", foo
  END SUBROUTINE MyRoutine
END MODULE

It can be driven from Fortran (fdriv.f90):

PROGRAM FDriv
  USE FLIB
  IMPLICIT NONE
  INTEGER :: err
  CALL MPI_Init(err)
  CALL MyRoutine()
  CALL MPI_Finalize(err)
END PROGRAM

And driven from C++ (cdriv.cc):

#include <mpi.h>
#include <iostream>

extern "C" {
  void MyRoutine();
}

void CVersion() {
  int foo;
  foo = 1;
  MPI_Allreduce(MPI_IN_PLACE, &foo, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
  std::cout << " C >> " << foo << std::endl;
}

int main(int argc, char *argv[]) {
  MPI_Init(&argc, &argv);
  MyRoutine();
  CVersion();
  MPI_Finalize();
  return 0;
}

Here is the Makefile:

ALL: fort cxx
flib.o: flib.f90
        mpif90 -c $< -o $@
fdriv.o: fdriv.f90 flib.o
        mpif90 -c $< -o $@
fort: fdriv.o flib.o
        mpif90 $^ -o $@
cdriv.o: cdriv.cc
        mpicxx -c $< -o $@ -lstdc++
cxx: cdriv.o flib.o
        mpif90 $^ -o $@
test:
        mpirun -np 1 ./fort
        mpirun -np 1 ./cxx
clean:
        rm *.o *.mod fort cxx

When I run the code I get the following output:

mpirun -np 1 ./fort
 >>           1
 <<           1
mpirun -np 1 ./cxx
 >>           1
 <<           0
 C >> 1

The 0 result is incorrect, it should be 1.

Note that if I switch mpich for openmpi, the code operates correctly. The code also operates correctly if I use the MPICH which is installed via apt.

[dalcinl]

This is most likely related to the following issue pmodels/mpich#5589. It was fixed in PR pmodels/mpich#5590. The patch looks trivial, although I don't know if it will apply to current MPICH release.

If you can try a manual build of MPICH with the patch applied and confirm things works on your side, then I can handle adding the patch to the conda-forge recipe and pushing a new build.

[william-dawson]

Thank you very much @dalcinl! So this is an upstream issue with mpich. I think it is fine if you just close this issue. It is not something that I need fixed urgently since the C++ bindings of my library are rarely used.