Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

We state that MPI is optional but -DUSE_MPI=false fails #17

Closed
beddalumia opened this issue Oct 1, 2022 · 3 comments
Closed

We state that MPI is optional but -DUSE_MPI=false fails #17

beddalumia opened this issue Oct 1, 2022 · 3 comments

Comments

@beddalumia
Copy link
Contributor

beddalumia commented Oct 1, 2022

If we force plain gfortran (instead of mpif90) as:

$ export FC=gfortran

The build fails even if we explicitly request for not including mpi:

$ cmake -DUSE_MPI=false ..
click to see output
-- The Fortran compiler identification is GNU 12.2.0
-- Check for working Fortran compiler: /home/linuxbrew/.linuxbrew/bin/gfortran
-- Check for working Fortran compiler: /home/linuxbrew/.linuxbrew/bin/gfortran  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /home/linuxbrew/.linuxbrew/bin/gfortran supports Fortran 90
-- Checking whether /home/linuxbrew/.linuxbrew/bin/gfortran supports Fortran 90 -- yes
-- OS: Linux 5.15.0-46-generic
-- BUILD_TYPE=RELEASE
-- MKL found at: /opt/intel/compilers_and_libraries_2020.4.304/linux/mkl
-- MKL supports Scalapack+Blacs
-- Fortran Compiler id   = GNU
-- Fortran Compiler ver. = 12.2.0
-- Fortran Compiler options = -cpp -ffree-line-length-none -fPIC -w  -fallow-argument-mismatch -O3   -funroll-loops
-- Found Git: /usr/bin/git (found version "2.25.1") 
-- Git SHA1: 4f49e7d81a3067c84dcaf8ee6bf9b768e7665a3e
-- Git branch: master
-- Git tag: 4.9.6
-- Testing P-ARPACK patch: zdotc.patch
-- Patch applied: zdotc.patch
-- SF compilation lines: -L${libdir} -lscifor -L/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -L/usr/lib/x86_64-linux-gnu -lpthread -lm -ldl
-- Version file: /home/gbellomia/Dropbox/SISSA/QcmPlab/SciFortran/buildNinja/etc/modules/scifor/gnu/.version
-- Module file: scifor/gnu/4.9.6
-- Library version: 4.9.6
-- Library will be installed in: /home/gbellomia/opt/scifor/gnu/4.9.6
-- 
>> TO CONCLUDE INSTALLATION <<
Compile with:
$ make
Install with:
$ make install

Uninstall with:
$ make uninstall

-- Configuring done
-- Generating done
-- Build files have been written to: /home/gbellomia/Dropbox/SISSA/QcmPlab/SciFortran/build
$ make
click to see output
Scanning dependencies of target SF_PARSE_INPUTLIB
[  0%] Building Fortran object src/SF_PARSE_INPUT/CMakeFiles/SF_PARSE_INPUTLIB.dir/LIST_INPUT.f90.o
[  1%] Building Fortran object src/SF_PARSE_INPUT/CMakeFiles/SF_PARSE_INPUTLIB.dir/SF_PARSE_INPUT.f90.o
[  1%] Built target SF_PARSE_INPUTLIB
Scanning dependencies of target SF_INITLIB
[  2%] Building Fortran object src/SF_INIT/CMakeFiles/SF_INITLIB.dir/SF_ARRAYS.f90.o
[  2%] Building Fortran object src/SF_INIT/CMakeFiles/SF_INITLIB.dir/SF_COLORS.f90.o
[  3%] Building Fortran object src/SF_INIT/CMakeFiles/SF_INITLIB.dir/SF_CONSTANTS.f90.o
[  3%] Building Fortran object src/SF_INIT/CMakeFiles/SF_INITLIB.dir/SF_FONTS.f90.o
[  3%] Building Fortran object src/SF_INIT/CMakeFiles/SF_INITLIB.dir/SF_MISC.f90.o
[  4%] Building Fortran object src/SF_INIT/CMakeFiles/SF_INITLIB.dir/SF_PAULI.f90.o
[  4%] Building Fortran object src/SF_INIT/CMakeFiles/SF_INITLIB.dir/SF_TIMER.f90.o
[  4%] Built target SF_INITLIB
Scanning dependencies of target SF_MPILIB
[  4%] Building Fortran object src/SF_MPI/CMakeFiles/SF_MPILIB.dir/SF_BLACS.f90.o
[  5%] Building Fortran object src/SF_MPI/CMakeFiles/SF_MPILIB.dir/SF_MPI.f90.o
[  5%] Built target SF_MPILIB
Scanning dependencies of target SF_IOTOOLSLIB
[  5%] Building Fortran object src/SF_IOTOOLS/CMakeFiles/SF_IOTOOLSLIB.dir/IOFILE.f90.o
[  5%] Building Fortran object src/SF_IOTOOLS/CMakeFiles/SF_IOTOOLSLIB.dir/IOPLOT.f90.o
[  6%] Building Fortran object src/SF_IOTOOLS/CMakeFiles/SF_IOTOOLSLIB.dir/IOREAD.f90.o
[  6%] Building Fortran object src/SF_IOTOOLS/CMakeFiles/SF_IOTOOLSLIB.dir/SF_IOTOOLS.f90.o
[  6%] Built target SF_IOTOOLSLIB
Scanning dependencies of target SF_DERIVATELIB
[  7%] Building Fortran object src/SF_DERIVATE/CMakeFiles/SF_DERIVATELIB.dir/SF_DERIVATE.f90.o
[  7%] Built target SF_DERIVATELIB
Scanning dependencies of target SF_INTEGRATELIB
[  8%] Building Fortran object src/SF_INTEGRATE/CMakeFiles/SF_INTEGRATELIB.dir/GAUSS_QUADRATURE.f90.o
[  8%] Building Fortran object src/SF_INTEGRATE/CMakeFiles/SF_INTEGRATELIB.dir/SF_INTEGRATE.f90.o
[  8%] Built target SF_INTEGRATELIB
Scanning dependencies of target SF_INTERPOLATELIB
[  9%] Building Fortran object src/SF_INTERPOLATE/CMakeFiles/SF_INTERPOLATELIB.dir/SF_INTERPOLATE.f90.o
[  9%] Built target SF_INTERPOLATELIB
Scanning dependencies of target SF_LINALGLIB
[ 10%] Building Fortran object src/SF_LINALG/CMakeFiles/SF_LINALGLIB.dir/SF_LINALG.f90.o
linalg_blacs_aux.f90:3:0:

Fatal Error: Cannot open included file 'mpif.h'
compilation terminated.
make[2]: *** [src/SF_LINALG/CMakeFiles/SF_LINALGLIB.dir/build.make:63: src/SF_LINALG/CMakeFiles/SF_LINALGLIB.dir/SF_LINALG.f90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:823: src/SF_LINALG/CMakeFiles/SF_LINALGLIB.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Where I want to highlight the actual fortran error:

linalg_blacs_aux.f90:3:0:

Fatal Error: Cannot open included file 'mpif.h'
compilation terminated.

The very same error is encountered by ninja (I first discovered it here, but then cross-checked with make)

$ cmake -DUSE_MPI=false -GNinja ..
[...]
$ ninja
[417/475] Building Fortran object src/SF_LINALG/CMakeFiles/SF_LINALGLIB.dir/SF_LINALG.f90.o
FAILED: src/SF_LINALG/CMakeFiles/SF_LINALGLIB.dir/SF_LINALG.f90.o include/sf_linalg.mod 
/home/linuxbrew/.linuxbrew/bin/gfortran  -I../src/SF_LINALG  -Iinclude -cpp -ffree-line-length-none -fPIC -w  -fallow-argument-mismatch -O3   -funroll-loops -Jinclude   -fpreprocessed -c src/SF_LINALG/CMakeFiles/SF_LINALGLIB.dir/SF_LINALG.f90-pp.f90 -o src/SF_LINALG/CMakeFiles/SF_LINALGLIB.dir/SF_LINALG.f90.o
linalg_blacs_aux.f90:3:0:

Fatal Error: Cannot open included file 'mpif.h'
compilation terminated.
[426/475] Building Fortran object src/SF_INTERPOLATE/CMakeFiles/SF_INTERPOLATELIB.dir/SF_INTERPOLATE.f90.o
ninja: build stopped: subcommand failed.

So, having it with both, I guess it is not a build-system (dependency-related) error.


Could it come from commit d9c6758, where the incriminate source file has moved from a USE MPI statement to include mpif.h? As previously discussed in 71b8aea this is crucial to avoid version mismatches but indeed it requires the mpif90 wrapper, to provide its own version of the mpif.h header. Well, I'm not actually sure it would have worked with a use statement neither: what happens to the MPI module if the compiler is not mpi-aware?

I imagine that PARPACK does not raise an issue since we do not request to compile it with the -DUSE_MPI=false option, but for sure we need to compile linalg_blacs_aux.f90 and there MPI is explicitly requested:
https://github.com/QcmPlab/SciFortran/blob/4f49e7d81a3067c84dcaf8ee6bf9b768e7665a3e/src/SF_LINALG/linalg_blacs_aux.f90#L2-L3

I actually believe that all the work needed to carefully disentangle the MPI and non-MPI features (and somehow document what you can actually use with a non-MPI install...) would not really be rewarded by any notable benefit. [See later comments, it's easier than what I expected]

@beddalumia
Copy link
Contributor Author

Side note: apparently ifort has no problems whatsoever

1. it embeds MPI functionality, so explicitly selecting mpif90 is not needed for a fully functional MPI build

click to see details
$ export FC=ifort
$ ifort --version
ifort (IFORT) 19.1.3.304 20200925
Copyright (C) 1985-2020 Intel Corporation.  All rights reserved.
$ cmake -GNinja ..
-- The Fortran compiler identification is Intel 19.1.3.20200925
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/sissa/compiler/intel/2020.4/compilers_and_libraries_2020.4.304/linux/bin/intel64/ifort - skipped
-- OS: Linux 3.10.0-1062.el7.x86_64
-- BUILD_TYPE=RELEASE
-- Found MPI_Fortran: /opt2/ohpc/pub/mpi/openmpi3-intel/3.1.4/lib/libmpi_usempif08.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- Set Fortran compiler FC to /opt/ohpc/pub/mpi/openmpi3-intel/3.1.4/bin/mpif90, ID=Intel
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - not found
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - found
-- BLAS found as: /usr/lib64/libblas.so
-- LAPACK found as: /usr/lib64/liblapack.so;/usr/lib64/libblas.so
-- Finding SCALAPACK
-- Fortran Compiler id   = Intel
-- Fortran Compiler ver. = 19.1.3.20200925
-- Fortran Compiler options = -fpp -O3 -ftz
-- Found Git: /opt/rh/rh-git218/root/usr/bin/git (found version "2.18.4") 
-- Git SHA1: 4f49e7d81a3067c84dcaf8ee6bf9b768e7665a3e
-- Git branch: master
-- Git tag: 4.9.6
-- Testing P-ARPACK patch: zdotc.patch
-- Patch applied: zdotc.patch
-- SF compilation lines: -L${libdir} -lscifor /usr/lib64/liblapack.so /usr/lib64/libblas.so
-- Version file: /home/gbellomi/QcmPlab/SciFortran/build_intel/etc/modules/scifor/intel/.version
-- Module file: scifor/intel/4.9.6
-- Library version: 4.9.6
-- Library will be installed in: /home/gbellomi/opt/scifor/intel/4.9.6
-- 
>> TO CONCLUDE INSTALLATION <<
Compile with:
$ make
Install with:
$ make install

Uninstall with:
$ make uninstall

-- Configuring done
-- Generating done
-- Build files have been written to: /home/gbellomi/QcmPlab/SciFortran/build_intel

$ ninja
[521/543] Building Fortran object src/SF_OPTIMIZE/CMakeFiles/SF_OPTIMIZELIB.dir/lbfgsb.f.o
src/SF_OPTIMIZE/CMakeFiles/SF_OPTIMIZELIB.dir//home/gbellomi/QcmPlab/SciFortran/src/SF_OPTIMIZE/lbfgsb.f(2802): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3))
----------------------------------------------------^
src/SF_OPTIMIZE/CMakeFiles/SF_OPTIMIZELIB.dir//home/gbellomi/QcmPlab/SciFortran/src/SF_OPTIMIZE/lbfgsb.f(2893): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-')
----------------------------------------------------^
[543/543] Linking Fortran static library libscifor.a

2. even if we request -DUSE_MPI=false it somehow manages to compile

click to see details
$ cmake -GNinja -DUSE_MPI=false ..
-- The Fortran compiler identification is Intel 19.1.3.20200925
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/sissa/compiler/intel/2020.4/compilers_and_libraries_2020.4.304/linux/bin/intel64/ifort - skipped
-- OS: Linux 3.10.0-1062.el7.x86_64
-- BUILD_TYPE=RELEASE
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - not found
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - found
-- BLAS found as: /usr/lib64/libblas.so
-- LAPACK found as: /usr/lib64/liblapack.so;/usr/lib64/libblas.so
-- Finding SCALAPACK
-- Fortran Compiler id   = Intel
-- Fortran Compiler ver. = 19.1.3.20200925
-- Fortran Compiler options = -fpp -O3 -ftz
-- Found Git: /opt/rh/rh-git218/root/usr/bin/git (found version "2.18.4") 
-- Git SHA1: 4f49e7d81a3067c84dcaf8ee6bf9b768e7665a3e
-- Git branch: master
-- Git tag: 4.9.6
-- Testing P-ARPACK patch: zdotc.patch
-- Patch applied: zdotc.patch
-- SF compilation lines: -L${libdir} -lscifor /usr/lib64/liblapack.so /usr/lib64/libblas.so
-- Version file: /home/gbellomi/QcmPlab/SciFortran/build_intel/etc/modules/scifor/intel/.version
-- Module file: scifor/intel/4.9.6
-- Library version: 4.9.6
-- Library will be installed in: /home/gbellomi/opt/scifor/intel/4.9.6
-- 
>> TO CONCLUDE INSTALLATION <<
Compile with:
$ make
Install with:
$ make install

Uninstall with:
$ make uninstall

-- Configuring done
-- Generating done
-- Build files have been written to: /home/gbellomi/QcmPlab/SciFortran/build_intel

$ ninja
[454/475] Building Fortran object src/SF_OPTIMIZE/CMakeFiles/SF_OPTIMIZELIB.dir/lbfgsb.f.o
src/SF_OPTIMIZE/CMakeFiles/SF_OPTIMIZELIB.dir//home/gbellomi/QcmPlab/SciFortran/src/SF_OPTIMIZE/lbfgsb.f(2802): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3))
----------------------------------------------------^
src/SF_OPTIMIZE/CMakeFiles/SF_OPTIMIZELIB.dir//home/gbellomi/QcmPlab/SciFortran/src/SF_OPTIMIZE/lbfgsb.f(2893): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-')
----------------------------------------------------^
[475/475] Linking Fortran static library libscifor.a

(this is a mistery to me, is it maybe buiding an MPI-aware library nonentheless? Note that the total number of targets has gone from 543 to 475, so the CMake option is indeed being recognized...)

@beddalumia
Copy link
Contributor Author

No ok, they do not seem to be a lot, except for PARPACK, that is already handled by:

https://github.com/QcmPlab/SciFortran/blob/4f49e7d81a3067c84dcaf8ee6bf9b768e7665a3e/src/arpack/CMakeLists.txt#L187-L192

In fact grep tells us that:

$ git grep "USE MPI"
src/SF_MPI/SF_MPI.f90:  ! USE MPI
$ git grep mpif.h
click to see the long output
src/SF_LINALG/linalg_blacs_aux.f90:  include 'mpif.h'
src/SF_LINALG/linalg_blacs_aux.f90:  include 'mpif.h'
src/SF_LINALG/linalg_blacs_aux.f90:  include 'mpif.h'
src/SF_LINALG/linalg_blacs_aux.f90:  include 'mpif.h'
src/SF_MPI/SF_MPI.f90:  include 'mpif.h'
src/SF_SP_LINALG/SF_SP_LINALG.f90:  include 'mpif.h'
src/arpack/parpack/src/pcgetv0.f:      include   'mpif.h'
src/arpack/parpack/src/pcnaitr.f:      include   'mpif.h'
src/arpack/parpack/src/pcnaup2.f:      include   'mpif.h'
src/arpack/parpack/src/pcnaupd.f:      include  'mpif.h'
src/arpack/parpack/src/pdgetv0.f:      include   'mpif.h'
src/arpack/parpack/src/pdlamch10.f:      include "mpif.h"
src/arpack/parpack/src/pdlarnv.f:      include  'mpif.h'
src/arpack/parpack/src/pdnaitr.f:      include   'mpif.h'
src/arpack/parpack/src/pdnaup2.f:      include   'mpif.h'
src/arpack/parpack/src/pdnaupd.f:      include  'mpif.h'
src/arpack/parpack/src/pdnorm2.f:      include   'mpif.h'
src/arpack/parpack/src/pdsaitr.f:      include   'mpif.h'
src/arpack/parpack/src/pdsapps.f:      include  'mpif.h'
src/arpack/parpack/src/pdsaup2.f:      include   'mpif.h'
src/arpack/parpack/src/pdsaupd.f:      include  'mpif.h'
src/arpack/parpack/src/pdznorm2.f:      include   'mpif.h'
src/arpack/parpack/src/pscnorm2.f:      include   'mpif.h'
src/arpack/parpack/src/psgetv0.f:      include   'mpif.h'
src/arpack/parpack/src/pslamch10.f:      include "mpif.h"
src/arpack/parpack/src/pslarnv.f:      include  'mpif.h'
src/arpack/parpack/src/psnaitr.f:      include   'mpif.h'
src/arpack/parpack/src/psnaup2.f:      include   'mpif.h'
src/arpack/parpack/src/psnaupd.f:      include  'mpif.h'
src/arpack/parpack/src/psnorm2.f:      include   'mpif.h'
src/arpack/parpack/src/pssaitr.f:      include   'mpif.h'
src/arpack/parpack/src/pssaup2.f:      include   'mpif.h'
src/arpack/parpack/src/pssaupd.f:      include  'mpif.h'
src/arpack/parpack/src/pzgetv0.f:      include   'mpif.h'
src/arpack/parpack/src/pznaitr.f:      include   'mpif.h'
src/arpack/parpack/src/pznaup2.f:      include   'mpif.h'
src/arpack/parpack/src/pznaupd.f:      include  'mpif.h'
src/arpack/parpack/util/pcmout.f:      include  'mpif.h'
src/arpack/parpack/util/pcvout.f:      include  'mpif.h'
src/arpack/parpack/util/pdmout.f:      include  'mpif.h'
src/arpack/parpack/util/pdvout.f:      include  'mpif.h'
src/arpack/parpack/util/pivout.f:      include  'mpif.h'
src/arpack/parpack/util/psmout.f:      include  'mpif.h'
src/arpack/parpack/util/psvout.f:      include  'mpif.h'
src/arpack/parpack/util/pzmout.f:      include  'mpif.h'
src/arpack/parpack/util/pzvout.f:      include  'mpif.h'

Only src/SF_LINALG/linalg_blacs_aux.f90, src/SF_MPI/SF_MPI.f90 and src/SF_SP_LINALG/SF_SP_LINALG.f90 include mpif.h (other than PARPACK) and the latter two resolve the issue by preprocessor directives:

https://github.com/QcmPlab/SciFortran/blob/4f49e7d81a3067c84dcaf8ee6bf9b768e7665a3e/src/SF_MPI/SF_MPI.f90#L3-L6

https://github.com/QcmPlab/SciFortran/blob/4f49e7d81a3067c84dcaf8ee6bf9b768e7665a3e/src/SF_SP_LINALG/SF_SP_LINALG.f90#L5-L11

So probably we just need to add an #ifdef _MPI outer block here:

https://github.com/QcmPlab/SciFortran/blob/4f49e7d81a3067c84dcaf8ee6bf9b768e7665a3e/src/SF_LINALG/SF_LINALG.f90#L506-L508

beddalumia added a commit that referenced this issue Oct 1, 2022
Subroutines contained in linalg_blacs_aux.f90 include the mpif.h header,
which is not available in a non-MPI environment (of course). The problem
is readily solved by constraining the compilation of the

       include "linalg_blacs_aux.f90"

statement in SF_LINALG.f90 and the related interface declarations, to an
active -D_MPI flag, via suitable preprocessor directives.

------------------------------------------------------------------------
I also added yet another explicit dependency in the top level CMakeLists
file, since both ninja and parallel "make -j 8" builds were failing, due
to SF_SPECIAL being compiled before SF_INTEGRATE.
------------------------------------------------------------------------
@beddalumia
Copy link
Contributor Author

Of course this was also needed:

https://github.com/QcmPlab/SciFortran/blob/03228bcdf93859d31417c0df6038b5daa04e51a7/src/SF_LINALG/SF_LINALG.f90#L358-L370

Now I manage to compile with both gfortran and ifort, so the issue appears to be solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant