openib btl fails to run with rdmacm

[alinask]

The following command line fails when running on 2 hosts with ompi master branch.
(one process on each node).
The same command line runs fine with ompi's v1.10 release branch.

mpirun -np 2 --bind-to core  --display-map -mca pml ob1 -mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm  --map-by node  ./IMB/src/IMB-MPI1 pingpong

The output is:

--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           vegas10
  Local device:         mlx5_0
  Local port:           1
  CPCs attempted:       rdmacm
--------------------------------------------------------------------------
 benchmarks to run pingpong 
[vegas11:27048] mca_bml_base_btl_array_get_next: invalid array size
[vegas11:27048] *** Process received signal ***
[vegas11:27048] Signal: Segmentation fault (11)
[vegas11:27048] Signal code: Address not mapped (1)
[vegas11:27048] Failing at address: 0x8
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications.  This means that no Open MPI device has indicated
that it can be used to communicate between these processes.  This is
an error; Open MPI requires that all MPI processes be able to reach
each other.  This error can sometimes be the result of forgetting to
specify the "self" BTL.

  Process 1 ([[65106,1],0]) is on host: vegas11
  Process 2 ([[65106,1],1]) is on host: vegas10
  BTLs attempted: self

Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------

When using udcm the test passes.

[jsquyres]

@alinask Is this related to the recent mpool/rcache change, perchance? I.e., could you git bisect to find out where it broke?

[alinask]

I just checked this with the master branch after taking it back to many commits ago (bofore the rcache change) and got the same result. Looks like this might have been broken for a while.

[rhc54]

rdmacm has been known to be broken for years - I don't believe it was ever fully functional. That's why we use udcm instead.

[alinask]

the release branch (v1.10) seems to work with it though.

[rhc54]

it does? miracles never cease 😄

[jladd-mlnx]

@alinask What about the 2.x branch? Please track down the offending commit.

[hjelmn]

We now default to having only SRQs on IB hardware. rdmacm will not work unless the first QP is per-peer. Try adding a per-peer QP to btl_openib_receive_queues. If that works I will close this because this is a known limitation of rdmacm.

[hjelmn]

BTW, per-peer QPs provide no known benefit on mlx hardware. They have a huge downside in that they scale horribly.

[jsquyres]

@alinask @hjelmn Perhaps a check should be added to rdmacm to ensure that the first RQ in the openib BTL is a per-peer RQ? That would avoid confusing users like this (i.e., assuming we can detect the error and print a useful error message). Or maybe rdmacm can simply emit a BTL verbose message saying that it chose to disqualify itself because there were no PP RQs. Then the situation would be discovered if the use increases the btl_base_verbose. Shrug. Just my $0.02.

[alinask]

@hjelmn I tried adding P,65536,256,192,128 to the default value of btl_openib_receive_queues and to run only with P,65536,256,192,128.
Both failed with:

[[8857,1],1][connect/btl_openib_connect_rdmacm.c:440:rdmacm_find_endpoint] can't find suitable endpoint for this peer

[alinask]

Branch v2.x currently doesn't work, same as the master.
Looks like the commit that started causing the failure is the one changing 'mpi_add_procs_cutoff' from 1024 to 0.

In v2.x:

commit 949b77f603e6b77afc67bf19f57f76121d03f217
Merge: fef0010 d7c21da
Author: Howard Pritchard <hppritcha@gmail.com>
Date:   Wed Feb 10 15:41:55 2016 -0700

    Merge pull request #955 from jsquyres/v2.x

    v2.x: ompi_mpi_params.c: set mpi_add_procs_cutoff default to 0

Increasing mpi_add_procs_cutoff, from the command line, to a value higher than the np value makes the test run (along with not using SRQ). Both in v2.x and in the master branch.

mpirun  -np 2 --bind-to core  --display-map -mca pml ob1 -mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm  --map-by node -mca btl_openib_receive_queues  P,65536,256,192,128 -mca mpi_add_procs_cutoff 4  ./IMB/src/IMB-MPI1 pingpong

[hjelmn]

Yup. So there were two problems. 1: you can not run rdmacm without a per-peer QP, 2: rdmacm has not been updated for dynamic add_procs. I can add a flag to indicate if rdmacm is in use not to use dynamic add_procs. I have no interest in fixing rdmacm to work with dynamic add_procs so its unlikely I will do it. I can give pointers if anyone else wants to take a crack at it.

[jladd-mlnx]

@hjelmn can you, please, add the flag to disqualify dynamic add-procs if rdmacm is selected. In my opinion that's the only blocker here.

[hjelmn]

@jladd-mlnx Yeah, plan to add the flag today. Let me know if any pointers are needed for updating rdmacm.