Assertion failure on MPI_Finalize with libpsm_infinipath

[lpouillo]

Thank you for taking the time to submit an issue!

Background information

What verson of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v4.1.1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

• using EasyBuild v4.4.2
• using tarball on top of EasyBuild module file
  1. GCCcore/10.3.0 8) libpciaccess/0.16-GCCcore-10.3.0
  2. zlib/1.2.11-GCCcore-10.3.0 9) hwloc/2.4.1-GCCcore-10.3.0
  3. binutils/2.36.1-GCCcore-10.3.0 10) OpenSSL/1.1
  4. GCC/10.3.0 11) libevent/2.1.12-GCCcore-10.3.0
  5. numactl/2.0.14-GCCcore-10.3.0 12) UCX/1.10.0-GCCcore-10.3.0
  6. XZ/5.2.5-GCCcore-10.3.0 13) libfabric/1.12.1-GCCcore-10.3.0
  7. libxml2/2.9.10-GCCcore-10.3.0 14) PMIx/3.2.3-GCCcore-10.3.0
• using tarball on top of system gcc

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

no

Please describe the system on which you are running

• Operating system/version: Centos 7.9
• Computer hardware: Dell C6320 with dual Xeon E5-2640 v3
• Network type: QLogic Corp. IBA7322 QDR InfiniBand HCA
  Installed Packages
  Name : infinipath-psm
  Arch : x86_64
  Version : 3.3
  Release : 26_g604758e_open.2.el7
  Size : 434 k
  Repo : installed
  From repo : anaconda
  Summary : QLogic PSM Libraries
  URL : https://www.openfabrics.org/
  License : BSD or GPLv2
  Description : The PSM Messaging API, or PSM API, is QLogic's low-level
  : user-level communications interface for the Truescale
  : family of products. PSM users are enabled with mechanisms
  : necessary to implement higher level communications
  : interfaces in parallel environments.

---

Details of the problem

• configure, make and make install run without error.
• ompi_info show the correct mtl, btl, pml

ompi_info | grep -e mtl -e btl -e pml
                 MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: ofi (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: openib (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: uct (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: usnic (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                 MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.1)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA mtl: psm (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.1.1)
                 MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.1.1)

• compilation of the examples are ok.

On execution of all examples, there is an Assertion failure

lpouillo@host:~/openmpi-4.1.1/examples$ mpirun -np 2 ./hello_c
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   host
  Local device: qib0
--------------------------------------------------------------------------
Hello, world, I am 0 of 2, (Open MPI v4.1.1, package: Open MPI lpouillo@host Distribution, ident: 4.1.1, repo rev: v4.1.1, Apr 24, 2021, 118)
Hello, world, I am 1 of 2, (Open MPI v4.1.1, package: Open MPI lpouillo@host Distribution, ident: 4.1.1, repo rev: v4.1.1, Apr 24, 2021, 118)
host.17542Assertion failure at psm_ep.c:1074: ep->mctxt_master == ep
host.17543Assertion failure at psm_ep.c:1074: ep->mctxt_master == ep
[host:17542] *** Process received signal ***
[host:17542] Signal: Aborted (6)
[host:17542] Signal code:  (-6)
[host:17542] [ 0] /lib64/libpthread.so.0(+0xf630)[0x7f95a370f630]
[host:17542] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7f95a33683d7]
[host:17542] [ 2] /lib64/libc.so.6(abort+0x148)[0x7f95a3369ac8]
[host:17542] [ 3] /lib64/libpsm_infinipath.so.1(+0x19b4a)[0x7f95955e3b4a]
[host:17542] [ 4] /lib64/libpsm_infinipath.so.1(+0x19fb1)[0x7f95955e3fb1]
[host:17542] [ 5] /lib64/libpsm_infinipath.so.1(psm_ep_close+0x3cf)[0x7f95955e2b1f]
[host:17542] [ 6] /lib64/libpsm_infinipath.so.1(__psm_finalize+0x4e)[0x7f95955ea92e]
[host:17542] [ 7] /home/lpouillo/openmpi_system/lib/openmpi/mca_mtl_psm.so(ompi_mtl_psm_finalize+0x40)[0x7f9589bc5d60]
[host:17542] [ 8] /home/lpouillo/openmpi_system/lib/libmpi.so.40(ompi_mpi_finalize+0x628)[0x7f95a396f218]
[host:17542] [haswell-t16-44:17543] *** Process received signal ***
[host:17543] Signal: Aborted (6)
[host:17543] Signal code:  (-6)
[ 9] ./hello_c[0x4008af]
[host:17542] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f95a3354555]
[host:17542] [11] ./hello_c[0x400759]
[host:17542] *** End of error message ***
[host:17543] [ 0] /lib64/libpthread.so.0(+0xf630)[0x7fe41c8f0630]
[host:17543] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7fe41c5493d7]
[host:17543] [ 2] /lib64/libc.so.6(abort+0x148)[0x7fe41c54aac8]
[host:17543] [ 3] /lib64/libpsm_infinipath.so.1(+0x19b4a)[0x7fe40a608b4a]
[host:17543] [ 4] /lib64/libpsm_infinipath.so.1(+0x19fb1)[0x7fe40a608fb1]
[host:17543] [ 5] /lib64/libpsm_infinipath.so.1(psm_ep_close+0x3cf)[0x7fe40a607b1f]
[host:17543] [ 6] /lib64/libpsm_infinipath.so.1(__psm_finalize+0x4e)[0x7fe40a60f92e]
[host:17543] [ 7] /home/lpouillo/openmpi_system/lib/openmpi/mca_mtl_psm.so(ompi_mtl_psm_finalize+0x40)[0x7fe402d88d60]
[host:17543] [ 8] /home/lpouillo/openmpi_system/lib/libmpi.so.40(ompi_mpi_finalize+0x628)[0x7fe41cb50218]
[host:17543] [ 9] ./hello_c[0x4008af]
[host:17543] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fe41c535555]
[host:17543] [11] ./hello_c[0x400759]
[host:17543] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 0 on node haswell-t16-44 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
[haswell-t16-44:17537] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init
[haswell-t16-44:17537] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

The example compiles and runs perfectly with OpenMPI 3.1.1

Thank you if you have any idea on howto solve this issue.

Best regards,

Laurent

[jsquyres]

@mwheinz Has anyone had a chance to look at this?

[jsquyres]

@lpouillo Given the age of the hardware, we may not have been testing this particular code path for a while. Can you do me a favor and check to see if Open MPI v4.1.0 works? And if it doesn't, can you also check Open MPI v4.0.0?

[lpouillo]

@jsquyres I'll do the check as soon as possible. Thanks for your feedback (and sorry for the prehistoric fabric, but it is still working).

[jsquyres]

"It ain't broke (and it's fully paid for), so keep using it!"

Ok, that's not the usual quote. But it's close enough. 😛

[DanCalugaru]

Hello OMPI folk (I'am @lpouillo colleague),

I did the tests you requested and here are the answers :

• Open MPI v4.1.0 : it doesn't work (same errors as v4.1.1, but without the normal messages "Hello, world, I am 0 of 2...")
• Open MPI v4.0.0 : it works (no output error)

Therefore, it seems the problem appears from 4.1.0.

As reminder, the test was : mpirun -np 2 ./hello_c

I think (need to confirm or not...) that this is related with this entry on the release notes for 4.1.0 :

• OFI/libfabric: Added btl for one-sided support

Indeed, for v4.1.1 and v4.1.0, the above test works well by adding these MCA options :
--mca mtl psm --mca btl self
but doesn't work if these MCA options are used :
--mca mtl psm --mca btl self,ofi
or
--mca btl self,ofi alone

As above, for v4.1.0 when it doesn't work, the messages "Hello, world, I am 0 of 2..." are no longer displayed at all, but only error messages "Assertion failure..."

I have also tried to see what happens when v4.1.1 is build with --enable-mca-no-build=btl-ofi and in this case, all seems OK (no error if hello_c is launched without MCA options).

Final remark : all is fine for all versions if ofi is used as mtl, I mean :
mpirun -np 2 --mca mtl ofi --mca btl self ./hello_c

I'm a little confused why ofi was also added in btl framework in v4.1.0 (in addition of mtl framework)

Thank you very much,

Dan

[mwheinz]

Dan, the addition of the OFI BTL is intended as a replacement for the previously used pt2pt BTL which provided support for one-sided communications via other MTLs such as PSM and OFI.

Your question about OFI causing the problem is interesting, and the discontinuing of the pt2pt is important because there's no PSM BTL: applications that require one-sided communications need to use a different transport, such as OFI or sockets to work - but if you don't need such support, specifying "btl self" will certainly solve the problem.

I should let you know that while I am in some way the "descendent" of the team that wrote the original PSM support I do not have access to QLogic DDR or QDR hardware or even the source code for libpsm-infinipath. We're probably going to be very limited in how much support we can give you.

[jsquyres]

Based on the above, --enable-mca-no-build=btl-ofi is likely your best bet (or simply rm -f $prefix/lib/openmpi/mca_btl_ofi.* after you install).

FWIW, if you want an explanation of BTLs vs. MTLs, you can check out https://www.open-mpi.org/video/?category=general#abcs-of-open-mpi-part-2, especially slides 28-42.

[DanCalugaru]

Many thanks Jeff & Michael for your answers (and links) which complement each other well.
So we will try to reinstall by adding --enable-mca-no-build=btl-ofi

P.S. : I see that PSM is still considered in (recent) OpenMPI presentations (e.g. page 40-41 in the Jeff's link for 4.x), in addition of course to PSM2. This link https://github.com/intel/psm seemed to me to be the repository for libpsm-infinipath, but I may be wrong

Have a pleasant weekend

[mwheinz]

P.S. : I see that PSM is still considered in (recent) OpenMPI presentations (e.g. page 40-41 in the Jeff's link for 4.x), in addition of course to PSM2. This link https://github.com/intel/psm seemed to me to be the repository for libpsm-infinipath, but I may be wrong

Officially PSM is still a part of 4.x, but it was dropped from Open MPI 5.x and, as a practical matter, we don't have any way to test it. I didn't know they'd put the source on github, that's good to know. All the Omnipath stuff (the successor to Infinipath) was transferred to Cornelis Networks when they spun us back out of Intel. (It's a long and complicated story...)

[jsquyres]

FWIW: Open MPI has backwards guarantees in a major version number series. According to https://www.open-mpi.org/software/ompi/versions/timeline.php, Open MPI v4.0.0 was released in 2018. Our backwards compat guarantees state that we won't break something in a later release where the major release number is the same. So PSM is still included in all 4.0.x and 4.1.x releases.

[jsquyres]

@DanCalugaru Did --enable-mca-no-build=btl-ofi work for you?

[DanCalugaru]

Hello Jeff & Michael ,
Yes --enable-mca-no-build=btl-ofi solved the problem and all tests was OK (as well as some benchmarks).
You can close this issue, thank you for your assistance and additional informations.
Dan