PETSc was configured with MPICH but now appears to be compiling using a non-MPICH mpi.h

[richman2024]

Hi,

When I ran "make -j 6" for test after installed MOOSE following the Build and Test MOOSE, I got 2 problems as below:

/public/home/ghfund3_a47/mambaforge3/envs/moose/include/petscsys.h:235:6: error: #error "PETSc was configured with MPICH but now appears to be compiling using a non-MPICH mpi.h"`
`/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/stl_algobase.h:378:57: error: no type named 'value_type' in 'struct std::iterator_traits<infix_ostream_iterator<std::basic_string<char> > >'`

What should I do to fix them?

Thank you for your help!

[GiudGiud]

Hello

for the first one it seems you are mixing mpi distributions. Can you paste the output of the diagnostics script in moose/scripts? So we can check how the mixing is happening

Guillaume

[richman2024]

Hi GiudGiud,

The output after ran diagnostics.sh as below:

System Arch: LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: NFSServer Description: NFS Server release 3.2 (RTM1) Release: 3.2 Codename: RTM1

MOOSE Package Version: Custom Build

CPU Count: 128

Memory Free: 174553.957 MB

$CC not set

MPICC:
which mpicc:
        /opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/bin/mpicc
mpicc -show:
        gcc -I/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/include -I/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/include/openmpi -I/opt/hpc/software/mpi/hwloc/include -pthread -L/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/lib -L/opt/hpc/software/mpi/hwloc//lib -L/usr//lib -Wl,-rpath -Wl,/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/lib -Wl,-rpath -Wl,/opt/hpc/software/mpi/hwloc//lib -Wl,-rpath -Wl,/usr//lib -Wl,--enable-new-dtags -lmpi

COMPILER gcc:
gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Python:
        /public/home/ghfund3_a47/mambaforge3/bin/python
        Python 3.10.6

MODULES:
Currently Loaded Modulefiles:
  1) compiler/devtoolset/7.3.1   3) compiler/dtk/22.10.1
  2) mpi/hpcx/2.11.0/gcc-7.3.1

PETSC_DIR not set

ENVIRONMENT:
CPLUS_INCLUDE_PATH=/public/software/compiler/dtk-22.10.1/include:/public/software/compiler/dtk-22.10.1/llvm/include:/opt/rh/devtoolset-7/root/usr/include/c++/7:/opt/rh/devtoolset-7/root/usr/include/c++/7/x86_64-redhat-linux
MANPATH=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/share/man:/opt/rh/devtoolset-7/root/usr/share/man:/opt/gridview/slurm/share/man:/opt/gridview/munge/share/man:
LDFLAGS=-Wl,-rpath=/opt/rh/devtoolset-7/root/usr/lib64 -Wl,-rpath=/opt/rh/devtoolset-7/root/usr/lib
XDG_SESSION_ID=1012616
SHMEM_HOME=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1
HOSTNAME=login01
AMDGPU_TARGETS=gfx906
HPCX_MPI_DIR=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1
OMPI_MCA_btl_openib_allow_ib=1
SHELL=/bin/bash
TERM=xterm
HISTSIZE=1000
SSH_CLIENT=10.10.2.2 55752 22
PERL5LIB=/opt/rh/devtoolset-7/root/usr/lib64/perl5/vendor_perl:/opt/rh/devtoolset-7/root/usr/share/perl5/vendor_perl:/public/home/ghfund3_a47/perl5/lib/perl5:
LIBRARY_PATH=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/sharp/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/hcoll/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/ucx_without_rocm/lib
OMPI_MCA_coll_hcoll_enable=0
CONDA_SHLVL=1
HSA_PATH=/opt/rocm/hsa
UCX_MAX_EAGER_LANES=4
CONDA_PROMPT_MODIFIER=(base) 
QTDIR=/usr/lib64/qt-3.3
UCX_MAX_RNDV_LANES=4
PERL_MB_OPT=--install_base /public/home/ghfund3_a47/perl5
QTINC=/usr/lib64/qt-3.3/include
UCX_ZCOPY_THRESH=16384
SSH_TTY=/dev/pts/4
CLUSCONF_HOME=/opt/clusconf
OMPI_MCA_pml=ucx
QT_GRAPHICSSYSTEM_CHECKED=1
USER=ghfund3_a47
HISTORY_FILE=/var/log/user-log/user.log
HPCX_DIR=/opt/hpc/software/mpi/hpcx/v2.11.0
OPAL_PREFIX=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
LD_LIBRARY_PATH=/public/software/compiler/dtk-22.10.1/lib:/public/software/compiler/dtk-22.10.1/lib64:/public/software/compiler/dtk-22.10.1/hip/lib:/public/software/compiler/dtk-22.10.1/llvm/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/sharp/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/hcoll/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/ucx_without_rocm/lib:/opt/rh/devtoolset-7/root/usr/lib64:/opt/rh/devtoolset-7/root/usr/lib:/opt/gridview/slurm/lib:/opt/gridview/slurm/lib64:/opt/gridview/munge/lib
UCX_RNDV_THRESH=16384
UCX_IB_PCI_BW=mlx5_0:50Gbs,mlx5_1:50Gbs,mlx5_2:50Gbs,mlx5_3:50Gbs
CONDA_EXE=/public/home/ghfund3_a47/mambaforge3/bin/conda
CPATH=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/include:/opt/hpc/software/mpi/hpcx/v2.11.0/ucx_without_rocm/include:/opt/hpc/software/mpi/hpcx/v2.11.0/sharp/include:/opt/hpc/software/mpi/hpcx/v2.11.0/hcoll/include
HPCX_HOME=/opt/hpc/software/mpi/hpcx/v2.11.0
UCX_IB_ADDR_TYPE=ib_global
_CE_CONDA=
HPCX_SHARP_DIR=/opt/hpc/software/mpi/hpcx/v2.11.0/sharp
UCX_NET_DEVICES=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1
PATH=/public/home/ghfund3_a47/mambaforge3/bin:/public/home/ghfund3_a47/mambaforge3/condabin:/public/software/compiler/dtk-22.10.1/bin:/public/software/compiler/dtk-22.10.1/llvm/bin:/public/software/compiler/dtk-22.10.1/hip/bin:/public/software/compiler/dtk-22.10.1/hip/bin/hipify:/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/bin:/opt/hpc/software/mpi/hpcx/v2.11.0/hcoll/bin:/opt/hpc/software/mpi/hpcx/v2.11.0/ucx_without_rocm/bin:/opt/rh/devtoolset-7/root/usr/bin:/usr/lib64/qt-3.3/bin:/public/home/ghfund3_a47/perl5/bin:/opt/rocm/bin:/opt/rocm/profiler/bin:/opt/rocm/opencl/bin/x86_64:/opt/rocm/hcc/bin:/opt/rocm/hip/bin:/opt/gridview/slurm/bin:/opt/gridview/slurm/sbin:/opt/gridview/munge/bin:/opt/gridview/munge/sbin:/opt/clusconf/sbin:/opt/clusconf/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin:/public/home/ghfund3_a47/.local/bin:/public/home/ghfund3_a47/bin
MAIL=/var/spool/mail/ghfund3_a47
NFSCONF=/opt/clusconf/etc/nfs.cfg
ROCM_PATH=/public/software/compiler/dtk-22.10.1
HPCX_HCOLL_DIR=/opt/hpc/software/mpi/hpcx/v2.11.0/hcoll
C_INCLUDE_PATH=/public/software/compiler/dtk-22.10.1/include:/public/software/compiler/dtk-22.10.1/llvm/include:/opt/rh/devtoolset-7/root/usr/include/c++/7:/opt/rh/devtoolset-7/root/usr/include/c++/7/x86_64-redhat-linux:/opt/gridview/slurm/include:/opt/gridview/munge/include
CONDA_PREFIX=/public/home/ghfund3_a47/mambaforge3
PWD=/public/home/ghfund3_a47/projects/moose-next/scripts
OSHMEM_HOME=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1
_LMFILES_=/opt/hpc/software/modules/compiler/devtoolset/7.3.1:/opt/hpc/software/modules/mpi/hpcx/2.11.0/gcc-7.3.1:/public/software/modules/compiler/dtk/22.10.1
IPMICONF=/opt/clusconf/etc/ipmi.cfg
LANG=en_US.UTF-8
LOG_HOME=/var/log/user-log
HPCX_UCX_DIR=/opt/hpc/software/mpi/hpcx/v2.11.0/ucx_without_rocm
MODULEPATH=/public/software/modules:/opt/hpc/software/modules
AUTOCLUSCONF=/opt/clusconf/etc/autoconf.cfg
KDEDIRS=/usr
LOADEDMODULES=compiler/devtoolset/7.3.1:mpi/hpcx/2.11.0/gcc-7.3.1:compiler/dtk/22.10.1
GRIDVIEW_HOME=/opt/gridview
OMPI_MCA_btl_openib_warn_default_gid_prefix=0
HCC_HOME=/opt/rocm/hcc
SLURM_HOME=/opt/gridview/slurm
OMPI_HOME=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1
HISTCONTROL=ignoredups
_CE_M=
HOME=/public/home/ghfund3_a47
SHLVL=2
HPCX_OSHMEM_DIR=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1
UCX_RC_MLX5_TIMEOUT=1000ms
OMPI_MCA_btl_openib_warn_no_device_params_found=0
UCX_DC_MLX5_TIMEOUT=1000ms
USER_ACTION=ghfund3_a47(10.10.2.2)
PERL_LOCAL_LIB_ROOT=:/public/home/ghfund3_a47/perl5
LOGNAME=ghfund3_a47
CONDA_PYTHON_EXE=/public/home/ghfund3_a47/mambaforge3/bin/python
STARTWAITTIME=300
QTLIB=/usr/lib64/qt-3.3/lib
SSH_CONNECTION=10.10.2.2 55752 10.10.2.1 22
XDG_DATA_DIRS=/public/home/ghfund3_a47/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
MODULESHOME=/usr/share/Modules
MPI_HOME=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1
OMPI_MCA_coll_hcoll_np=0
LESSOPEN=||/usr/bin/lesspipe.sh %s
CONDA_DEFAULT_ENV=base
PROMPT_COMMAND={ date "+[%F %T `hostname` `whoami`] `history 1 | { read x cmd; echo " -> $USER_ACTION   $cmd";}`"; }>> $HISTORY_FILE
INFOPATH=/opt/rh/devtoolset-7/root/usr/share/info
XDG_RUNTIME_DIR=/run/user/3777
MUNGE_HOME=/opt/gridview/munge
HIP_PATH=/public/software/compiler/dtk-22.10.1/hip
QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
INCLUDE=/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/include:/opt/hpc/software/mpi/hpcx/v2.11.0/ucx_without_rocm/include:/opt/hpc/software/mpi/hpcx/v2.11.0/sharp/include:/opt/hpc/software/mpi/hpcx/v2.11.0/hcoll/include
PERL_MM_OPT=INSTALL_BASE=/public/home/ghfund3_a47/perl5
OMPI_MCA_btl=^uct
BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`
}
_=/usr/bin/env

[GiudGiud]

You have mamba and a local MPI compiler loaded.

I think you are on a HPC machine so you should not use mamba. You should be following our HPC instructions

[richman2024]

You are right!
I tried again following HPC Cluster, when I was runing 'update_and_rebuild_petsc.sh' got below report:

INFO: Checking for HDF5...
INFO: HDF5 library not detected, opting to download via PETSc...
python3: can't open file '/public/home/ghfund3_a47/projects/moose-next/petsc/./configure': [Errno 2] No such file or directory
There was an error. Exiting...

It seems without HDF5 on the HPC, but I haven't permission to install HDF5.
Any other ways can I use to install MOOSE?

[milljm]

If you have write access to your home directory, and access to the internet from that account: then you have permission to install everything you need.

The error in question may stem from an earlier error. Perhaps failing to download HDF5? The modules I am seeing loaded, as well as the version of GCC in use, is below the minimum version we require (MOOSE requires GCC 7.5.0 or newer). So you will need to take care of that before continuing (PETSc may build fine, then libMesh... but then MOOSE will fail, wasting your time!).

It's most likely that you have newer versions or GCC available via modules. My advice is to run: module avail, peruse over that list, and choose modules which satisfy MOOSE's requirements. They need to fall into:

• Python: >=3.8 <3.11

  • 3.10.x being the stable standard at the moment.

• GCC >= 7.5.0 < 11.x

  • Try to stick with a version in between the range above. It's safer to stay away from bleeding edge compilers. Most of our testing is done on GCC 8.x - 10.x.
  • Intel compilers are not supported.

• MPI

  • This can be OpenMPI, MPICH, MVAPICH. You'll likely see all three available on your cluster. However, it may be best to ask your HPC admins which MPI stack to use. Clusters are usually tailor made, and therefor have a very optimized MPI stack. This is why we ask users to avoid using our Conda packages, when operating on an HPC cluster.

Basically, I am asking you to remove Conda from the equation, and start over, by figuring out what "modules" are available on your cluster.