btl/openib: update rdmacm for dynamic add_procs #1038

hjelmn · 2016-03-21T18:15:14Z

This commit adds the data necessesary for supporting dynamic add_procs
to the rdmacm message (opal_process_name_t). The endpoint lookup
function has been updated to match the code in udcm.

Closes open-mpi/ompi#1468.

:bot:assign: @jladd-mlnx
:bot:milestone:v2.0.0
:bot🏷️bug

Signed-off-by: Nathan Hjelm hjelmn@lanl.gov

(cherry picked from open-mpi/ompi@645bd9d)

Signed-off-by: Nathan Hjelm hjelmn@lanl.gov

This commit adds the data necessesary for supporting dynamic add_procs to the rdma message (opal_process_name_t). The endpoint lookup function has been updated to match the code in udcm. Closes open-mpi/ompi#1468. Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov> (cherry picked from open-mpi/ompi@645bd9d) Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>

jsquyres · 2016-03-21T18:21:09Z

@alinask Could you verify that this works for you?

mellanox-github · 2016-03-21T18:53:29Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ompi-release-pr/1458/ for details.

alinask · 2016-03-22T09:10:55Z

@hjelmn , thanks for this fix!

I tested on the master branch, where mpi_add_procs_cutoff default is set to 0.
The following command lines work:

rdmacm, only per-peer QP:

mpirun -np 2 --bind-to core --map-by node  --display-map -mca pml ob1 -mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm -mca btl_openib_receive_queues  P,65536,256,192,128  ./IMB/src/IMB-MPI1 pingpong

rdmacm, first QP is per-peer, rest are SRQ:

mpirun -np 2 --bind-to core --map-by node  --display-map -mca pml ob1 -mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm -mca btl_openib_receive_queues  P,65536,256,192,128:S,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64 ./IMB-MPI1 pingpong

udcm:

mpirun -np 2 --bind-to core --map-by node  --display-map -mca pml ob1 -mca btl self,sm,openib --mca btl_openib_cpc_include udcm   ./IMB/src/IMB-MPI1 pingpong

Since the default of btl_openib_receive_queues is to only use SRQ QPs (in the master branch), rdmacm won't work without changing this mca parameter from the command line. I added a comment stating this in the FAQ but maybe a warning about this should be added to the code like @jsquyres suggested or the default should have a per-peer QP at the beginning like the release branch (v1.10) has?
Without adding a per-peer QP to the command line, there is a segmentation fault and the output doesn't imply anything about what needs to be done.

mpirun  -np 8 --bind-to core --map-by node  --display-map -mca pml ob1 -mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm ./IMB/src/IMB-MPI1 pingpong

--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           vegas07
  Local device:         mlx5_0
  Local port:           1
  CPCs attempted:       rdmacm
--------------------------------------------------------------------------
 benchmarks to run pingpong alltoall 
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications.  This means that no Open MPI device has indicated
that it can be used to communicate between these processes.  This is
an error; Open MPI requires that all MPI processes be able to reach
each other.  This error can sometimes be the result of forgetting to
specify the "self" BTL.

  Process 1 ([[19697,1],0]) is on host: vegas06
  Process 2 ([[19697,1],1]) is on host: vegas07
  BTLs attempted: self

Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
[vegas06:19044] *** Process received signal ***
[vegas06:19044] Signal: Segmentation fault (11)
[vegas06:19044] Signal code: Address not mapped (1)
[vegas06:19044] Failing at address: 0x8
[vegas06:19044] mca_bml_base_btl_array_get_next: invalid array size
[vegas06:19044] [ 0] /lib64/libpthread.so.0[0x3d2e60f710]
[vegas06:19044] [ 1] /labhome/alinas/workspace/ompi/ompi-github/ompi/install/lib/openmpi/mca_pml_ob1.so(+0xf028)[0x7ffff1b47028]
[vegas06:19044] [ 2] /labhome/alinas/workspace/ompi/ompi-github/ompi/install/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_isend+0xc0)[0x7ffff1b47303]
[vegas06:19044] [ 3] /labhome/alinas/workspace/ompi/ompi-github/ompi/install/lib/libmpi.so.0(ompi_coll_base_bcast_intra_generic+0x1e4)[0x7ffff7d51a74]
[vegas06:19044] [ 4] /labhome/alinas/workspace/ompi/ompi-github/ompi/install/lib/libmpi.so.0(ompi_coll_base_bcast_intra_binomial+0x188)[0x7ffff7d526b6]
[vegas06:19044] [ 5] /labhome/alinas/workspace/ompi/ompi-github/ompi/install/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_bcast_intra_dec_fixed+0x162)[0x7ffff0c6b0a1]
[vegas06:19044] [ 6] /labhome/alinas/workspace/ompi/ompi-github/ompi/install/lib/libmpi.so.0(MPI_Bcast+0x2e3)[0x7ffff7cf021b]
[vegas06:19044] [ 7] /labhome/alinas/workspace/imb_test/IMB/src/IMB-MPI1[0x403de4]
[vegas06:19044] [ 8] /labhome/alinas/workspace/imb_test/IMB/src/IMB-MPI1[0x401cc7]
[vegas06:19044] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3d2de1ed5d]
[vegas06:19044] [10] /labhome/alinas/workspace/imb_test/IMB/src/IMB-MPI1[0x401b69]
[vegas06:19044] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 19044 on node vegas06 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

alinask · 2016-03-24T15:52:46Z

An enhancement of the warning "rdmacm CPC only supported when the first QP is a PP QP" is here:
open-mpi/ompi#1488

jladd-mlnx · 2016-03-24T15:57:20Z

👍

bot:assign: @jsquyres
bot:label:reviewed

Please merge.

ompiteam-bot · 2016-03-24T15:57:23Z

OMPIBot error: Label "reviewed" is already set on issue 1038.

jsquyres · 2016-03-24T16:13:35Z

Am I understanding this PR correctly:

this PR fixes the rdmacm CPC to work with the new dynamic add procs
openib still segv's if there is no PP QP with no discernable error message
there is no PP QP by default

alinask · 2016-03-24T16:31:36Z

There is an error message if btl_base_verbose is set (open-mpi/ompi#1488) and the current default is no PP QP (before this PR).

jsquyres · 2016-03-24T16:34:58Z

If we're still segv'ing by default, this seems like half a solution.

hjelmn · 2016-03-24T17:18:43Z

The SEGV still needs to be fixed in r2. I plan to take a look later today.

hjelmn · 2016-03-24T18:41:39Z

Kind of. The SEGV requires the user to specifically ask for rdmacm which is suboptimal on IB systems. I would prefer this get merged regardless of the warning message and r2 fix.

hjelmn · 2016-03-24T18:42:59Z

Also, there is a really clear error message about openib not working. It calls out the CPCs tried.

jsquyres · 2016-03-24T18:45:30Z

How long to get the 2nd segv fix?

I'm only trying to prevent the "I promise to commit the 2nd half of the fix!" ...but then get too busy/forget to do it kind of scenario.

This commit adds code to detect when procs are unreachable when using the dynamic add_procs functionality. Fixes #1501 Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov> (cherry picked from open-mpi/ompi@9d5eeec) Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>

ompiteam-bot · 2016-03-28T21:41:19Z

OMPIBot error: Label "-pushed-back" does not exist.

hjelmn · 2016-03-28T21:41:51Z

bot:nolabel:pushed-back

hjelmn · 2016-03-28T21:42:18Z

@jsquyres The SEGV is now fixed.

mellanox-github · 2016-03-28T22:19:33Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ompi-release-pr/1482/ for details.

jsquyres · 2016-03-29T10:30:19Z

@alinask Can you confirm that this PR now fixes everything for you / review the PR? Thanks.

alinask · 2016-03-29T13:52:46Z

sure, I'll check.
@hjelmn was this fix pushed to the master branch as well or should I test it only on v2.x?

jsquyres · 2016-03-29T15:31:33Z

@hjelmn says yes, it was pushed to master as well (he said this verbally on the call today).

jladd-mlnx · 2016-03-29T16:18:00Z

@alinask, please ack ASAP. We are trying to get this release out.

alinask · 2016-03-30T08:30:50Z

I checked the v2.x branch with the patch from this PR with the command lines from above.

When using rdmacm without a per-peer QP, the output is:

$mpirun -H vegas08,vegas09 -np 2 --bind-to core --map-by node  --display-map -mca pml ob1 -mca btl self,sm,openib --mca btl_openib_cpc_include rdmacm   ./IMB/src/IMB-MPI1 pingpong 

 ========================   JOB MAP   ========================

 Data for node: vegas08 Num slots: 1    Max slots: 0    Num procs: 1
    Process OMPI jobid: [42738,1] App: 0 Process rank: 0 Bound: socket 0[core 0[hwt 0]]:[B/././././././.][./././././././.]

 Data for node: vegas09 Num slots: 1    Max slots: 0    Num procs: 1
    Process OMPI jobid: [42738,1] App: 0 Process rank: 1 Bound: socket 0[core 0[hwt 0]]:[B/././././././.][./././././././.]

 =============================================================
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'vegas08', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           vegas08
  Local device:         mlx5_0
  Local port:           1
  CPCs attempted:       rdmacm
--------------------------------------------------------------------------
 benchmarks to run pingpong 
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications.  This means that no Open MPI device has indicated
that it can be used to communicate between these processes.  This is
an error; Open MPI requires that all MPI processes be able to reach
each other.  This error can sometimes be the result of forgetting to
specify the "self" BTL.

  Process 1 ([[42738,1],0]) is on host: vegas08
  Process 2 ([[42738,1],1]) is on host: vegas09
  BTLs attempted: self

Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
[vegas08:14207] *** An error occurred in MPI_Bcast
[vegas08:14207] *** reported by process [2800877569,140733193388032]
[vegas08:14207] *** on communicator MPI_COMM_WORLD
[vegas08:14207] *** MPI_ERR_INTERN: internal error
[vegas08:14207] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[vegas08:14207] ***    and potentially your MPI job)
[vegas08:14197] 1 more process has sent help message help-mpi-btl-openib.txt / default subnet prefix
[vegas08:14197] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[vegas08:14197] 3 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port

So there is no segmentation fault anymore. Thanks @hjelmn !
The other command lines worked (rdmacm with a PP QP/ udcm) while mpi_add_procs_cutoff is set to 0.

jsquyres · 2016-03-30T10:36:47Z

Thanks @alinask. I'm thinking that's an implicit 👍 :-)

jsquyres · 2016-03-30T10:36:57Z

@hppritcha Good to go.

ompiteam-bot assigned jladd-mlnx Mar 21, 2016

ompiteam-bot added this to the v2.0.0 milestone Mar 21, 2016

ompiteam-bot added the bug label Mar 21, 2016

hppritcha assigned alinask and unassigned jladd-mlnx Mar 21, 2016

ompiteam-bot assigned jsquyres and unassigned alinask Mar 24, 2016

ompiteam-bot added the reviewed label Mar 24, 2016

jsquyres added the RM-Approved label Mar 24, 2016

jsquyres removed the RM-Approved label Mar 24, 2016

jsquyres added pushed-back and removed reviewed labels Mar 25, 2016

ompiteam-bot removed the pushed-back label Mar 28, 2016

ompiteam-bot added the reviewed label Mar 30, 2016

hppritcha added the RM-Approved label Mar 30, 2016

hppritcha merged commit a9cecde into open-mpi:v2.x Mar 30, 2016

btl/openib: update rdmacm for dynamic add_procs #1038

btl/openib: update rdmacm for dynamic add_procs #1038

Uh oh!

Conversation

hjelmn commented Mar 21, 2016

Uh oh!

jsquyres commented Mar 21, 2016

Uh oh!

mellanox-github commented Mar 21, 2016

Uh oh!

alinask commented Mar 22, 2016

Uh oh!

alinask commented Mar 24, 2016

Uh oh!

jladd-mlnx commented Mar 24, 2016

Uh oh!

ompiteam-bot commented Mar 24, 2016

Uh oh!

jsquyres commented Mar 24, 2016

Uh oh!

alinask commented Mar 24, 2016

Uh oh!

jsquyres commented Mar 24, 2016

Uh oh!

hjelmn commented Mar 24, 2016

Uh oh!

hjelmn commented Mar 24, 2016

Uh oh!

hjelmn commented Mar 24, 2016

Uh oh!

jsquyres commented Mar 24, 2016

Uh oh!

ompiteam-bot commented Mar 28, 2016

Uh oh!

hjelmn commented Mar 28, 2016

Uh oh!

hjelmn commented Mar 28, 2016

Uh oh!

mellanox-github commented Mar 28, 2016

Uh oh!

jsquyres commented Mar 29, 2016

Uh oh!

alinask commented Mar 29, 2016

Uh oh!

jsquyres commented Mar 29, 2016

Uh oh!

jladd-mlnx commented Mar 29, 2016

Uh oh!

alinask commented Mar 30, 2016

Uh oh!

jsquyres commented Mar 30, 2016

Uh oh!

jsquyres commented Mar 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants