Crash with Intel MPI 2019.1

[prckent]

It is not clear if this is a problem with Intel MPI, support libraries, or QMCPACK, but on oxygen there is a crash on startup that did not occur with the .0 release:

$ export I_MPI_FABRICS="ofi"
$ yum list libfabric
Installed Packages
libfabric.x86_64                                                  1.6.1-2.el7                                                     @rhel-x86_64-workstation-7
$ mpirun -n 1 ../../../build_intel2019/bin/qmcpack simple-LiH.xml
Abort(1094543) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(639)......:
MPID_Init(860).............:
MPIDI_NM_mpi_init_hook(689): OFI addrinfo() failed (ofi_init.h:689:MPIDI_NM_mpi_init_hook:No data available)

Most of the Intel tests on https://cdash.qmcpack.org/CDash/viewProjects.php will fail until this is fixed or we decide to downgrade the software.

[ye-luo]

Was export I_MPI_FABRICS="ofi" used with old Intel compiler installation on oxygen?
Could you try "export I_MPI_FABRICS=shm"?
Enabling I_MPI_DEBUG may give us some insight.

[prckent]

We used to never set this. i.e. Behavior changed between 2019.0 and .1

"shm" is no longer supported. "shm:ofi" also fails. This setting might be a decoy but I explored it due to reported problems on the Intel support site.

[prckent]

$ export I_MPI_FABRICS=shm:ofi
$ export I_MPI_DEBUG=1
$ ../../../build_intel2019/bin/qmcpack
[0] MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
[0] MPI startup(): libfabric version: 1.7.0a1-impi
Abort(1094543) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(639)......:
MPID_Init(860).............:
MPIDI_NM_mpi_init_hook(689): OFI addrinfo() failed (ofi_init.h:689:MPIDI_NM_mpi_init_hook:No data available)

Interesting inconsistency in fabrics version.

[prckent]

MPI hello world fails. Happily this is therefore not a QMCPACK problem.

https://software.intel.com/en-us/node/799716?page=0

[prckent]

Fixed via https://software.intel.com/en-us/forums/intel-clusters-and-hpc-technology/topic/799716

Added export FI_PROVIDER=tcp to the oxygen scripts to bypass this bug in Intel MPI 2019.1