"Other MPI error" with Intel MPI

[jtamir]

Hi,
I have used mpi4py with Intel MPI in the past. I am now trying to use on a fresh Ubuntu 22.04.1 LTS with Intel MP andh mpi4py 3.1.4 and get the following error upon trying to import MPI:

~/mpi4py/mpi4py-3.1.4 $ python demo/helloworld.py
Abort(1090959) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(176)........:
MPID_Init(1538)..............:
MPIDI_OFI_mpi_init_hook(1473):
(unknown)(): Other MPI error
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1090959
:
system msg for write_line failure : Bad file descriptor
Abort(1090959) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(176)........:
MPID_Init(1538)..............:
MPIDI_OFI_mpi_init_hook(1473):
(unknown)(): Other MPI error
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1090959
:
system msg for write_line failure : Bad file descriptor
Segmentation fault (core dumped)

Here is how I am set up:

Intel MPI

Intel MPI is installed through the OneAPI package:

$ mpicc -v
mpigcc for the Intel(R) MPI Library 2021.7 for Linux*
Copyright Intel Corporation.
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)

I can confirm that Intel MPI is working with their test programs:

$ mpirun -n 4 -ppn 1 -f mpihosts.txt ./myprog
Hello world: rank 0 of 4 running on intel01
Hello world: rank 1 of 4 running on intel02
Hello world: rank 2 of 4 running on intel03
Hello world: rank 3 of 4 running on intel04

As well as:

$ mpiexec -n 4 -perhost 1 -hostfile mpihosts.txt -prepend-rank echo test
[1] test
[3] test
[2] test
[0] test

mpi4py install

I have installed mpi4py 3.1.4 by downloading the release and setting up mpi.cfg as follows:

# Default configuration
# ---------------------
[mpi]

 mpi_dir              = /opt/intel/oneapi/mpi/latest/
 mpicc                = %(mpi_dir)s/bin/mpicc
 mpicxx               = %(mpi_dir)s/bin/mpicxx
 include_dirs         = %(mpi_dir)s/include
 libraries            = mpi
 library_dirs         = %(mpi_dir)s/lib:%(mpi_dir)s/lib/release
 #library_dirs         = %(mpi_dir)s/lib/release
 runtime_library_dirs = %(mpi_dir)s/lib

python setup.py build
python setup.py install

Then in a Python interpreter (Python 3.9.12 (main, Apr 5 2022, 06:56:58)), I get this:

>>> import mpi4py
>>> mpi4py.__version__
'3.1.4'
>>> mpi4py.get_config()
{'mpicc': '/opt/intel/oneapi/mpi/latest//bin/mpicc', 'mpicxx': '/opt/intel/oneapi/mpi/latest//bin/mpicxx', 'include_dirs': '/opt/intel/oneapi/mpi/latest//include', 'libraries': 'mpi', 'library_dirs': '/opt/intel/oneapi/mpi/latest//lib:/opt/intel/oneapi/mpi/latest//lib/release', 'runtime_library_dirs': '/opt/intel/oneapi/mpi/latest//lib'}
>>> from mpi4py import MPI
Abort(1090959) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(176)........:
MPID_Init(1538)..............:
MPIDI_OFI_mpi_init_hook(1473):
(unknown)(): Other MPI error
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1090959
:
system msg for write_line failure : Bad file descriptor
Abort(1090959) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(176)........:
MPID_Init(1538)..............:
MPIDI_OFI_mpi_init_hook(1473):
(unknown)(): Other MPI error
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1090959
:
system msg for write_line failure : Bad file descriptor
Segmentation fault (core dumped)

Thank you for your help

[dalcinl]

I have not idea what could be going on, but this is most very likely not mpi4py's fault. What do you get if you export I_MPI_DEBUG=2 before ? Did you source setvars.sh as instructed? Is your Intel MPI using an external libfabric?

https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux/top/environment-variable-reference/other-environment-variables.html

You should really ask Intel MPI folks, they may provide more informed advice and further tips on how to debug the issue.

[jtamir]

Thanks for your reply. I will ask on the Intel forums to see if they can help. I am not sure exactly what to ask because I am able to compile and run MPI programs myself, I am just now able to use it within mpi4py.

Edit -- here is the output:

jtamir@intel01 ~/mpi4py/mpi4py-3.1.4 $ export I_MPI_DEBUG=30
jtamir@intel01 ~/mpi4py/mpi4py-3.1.4 $ python demo/helloworld.py
MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.
               Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.
MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.
[0] MPI startup(): Intel(R) MPI Library, Version 2021.7  Build 20220909 (id: 6b6f6425df)
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
Abort(1090959) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(176)........:
MPID_Init(1538)..............:
MPIDI_OFI_mpi_init_hook(1473):
(unknown)(): Other MPI error
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1090959
:
system msg for write_line failure : Bad file descriptor
Abort(1090959) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(176)........:
MPID_Init(1538)..............:
MPIDI_OFI_mpi_init_hook(1473):
(unknown)(): Other MPI error
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1090959
:
system msg for write_line failure : Bad file descriptor
Segmentation fault (core dumped)

[jtamir]

Thanks again for your help. Indeed, this was not an mpi4py issue and is likely due to the non-conventional way I am setting environment variables for my convoluted setup. Your advice helped me find more information online. I was able to get things running in the following way:

$ I_MPI_FABRICS=shm FI_PROVIDER=shm python demo/helloworld.py
Hello, World! I am process 0 of 1 on intel01.