--MCA flag does not activate btl framework to work with specified components (TCP/IB)

[abeltre1]

Background information

1. installed openmpi 3.1.2 after Mellanox OFED
2. Tested with mpirun and a large number for --mca parameters

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

1. V3.1.2
2. MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.5-x86_64
3. CentOS 7.5

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

1. source/distribution tarball

Please describe the system on which you are running

• Operating system/version: Centos 7.5
• Computer hardware: Intel Xion
• Network type: IB

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

Note: If you include verbatim output (or a code block), please use a GitHub Markdown code block like below:

shell$  mpirun -mca mtl_mxm_np 0 --mca btl tcp -np 48 --map-by node --hostfile hostfile  ./osu_alltoallv 

# OSU MPI All-to-Allv Personalized Exchange Latency Test v3.8
# Size       Avg Latency(us)
1                     357.03
2                     357.91
4                     351.81
8                     352.57
16                    352.21
32                    350.65
64                    355.97
128                   369.26
256                   383.22
512                   398.14
1024                  425.57
2048                  485.46
4096                  680.91
8192                 1204.47
16384                2094.76
32768                3836.11
65536                8636.47
131072              17042.69
262144              31665.86
524288              63652.95
1048576            124922.32

mpirun -mca mtl_mxm_np 0 --mca btl openib -np 48 --map-by node --hostfile hostfile  ./osu_alltoallv

# OSU MPI All-to-Allv Personalized Exchange Latency Test v3.8
# Size       Avg Latency(us)
1                     351.64
2                     357.78
4                     354.88
8                     351.91
16                    353.21
32                    348.08
64                    356.01
128                   368.82
256                   383.04
512                   396.46
1024                  424.53
2048                  484.94
4096                  681.93
8192                 1211.97
16384                2104.20
32768                3852.04
65536                8563.86
131072              16901.64
262144              31549.93
524288              62248.82
1048576            124405.48

This is a latency test for which I expect the values for each individual component to yield a lower latency number for each message size.

[ggouaillardet]

btl/tcp and btl/openib are only used when pml/ob1 is used. Also, it is mandatory to use btl/self when pml/ob1 is used.

Based on that, I think pml/cm and mtl/mxm are used on your system.

I do invite you to test again with

mpirun --mca pml ob1 --mca btl tcp,self ...
mpirun --mca pml ob1 --mca btl openib,self ...

You might as well want to use btl/vader for intra node communications (otherwise you will use btl/tcp or btl/openib as the "loopback" device, which is suboptimal.

[abeltre1]

@ggouaillardet I have ran all the tests below to see if you can shine more light on the particualr issues.

Using MXM -mca pml cm -mca mtl mxm

mpirun  -mca mtl_mxm_np 0 -mca pml cm -mca mtl mxm -x MXM_RDMA_PORTS=mlx4_0:1 -mca btl_openib_if_include mlx4_0:1 -mca rmaps_base_dist_hca mlx4_0   -hostfile ~/hostfile --map-by node -np 48  ./osu_alltoallv 

# OSU MPI All-to-Allv Personalized Exchange Latency Test v5.3.2
# Size       Avg Latency(us)
1                     359.18
2                     359.80
4                     358.13
8                     359.06
16                    358.16
32                    357.18
64                    368.31
128                   381.27
256                   389.58
512                   404.07
1024                  438.92
2048                  531.51
4096                  819.45
8192                 1643.60
16384                2497.37
32768                4261.66
65536                9289.27
131072              18060.99
262144              32590.61
524288              64060.68
1048576            127826.50

TCP --mca pml ob1 --mca btl tcp,self

mpirun  --mca mtl ^mxm  --mca pml ob1 --mca btl tcp,self  -hostfile ~/hostfile --map-by node -np 48  ./osu_alltoallv 
--------------------------------------------------------------------------
WARNING: Open MPI accepted a TCP connection from what appears to be a
another Open MPI process but cannot find a corresponding process
entry for that peer.

This attempted connection will be ignored; your MPI job may or may not
continue properly.

  Local host: kube-infi-1
  PID:        121465
--------------------------------------------------------------------------
[kube-infi-1][[38282,1],15][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],11]
[kube-infi-1][[38282,1],3][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],5]
--------------------------------------------------------------------------
WARNING: Open MPI failed to TCP connect to a peer MPI process.  This
should not happen.

Your Open MPI job may now fail.

  Local host: kube-infi-1
  PID:        121442
  Message:    connect() to 172.17.0.1:1026 failed
  Error:      Operation now in progress (115)
--------------------------------------------------------------------------
[kube-infi-1][[38282,1],1][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],15]
[kube-infi-1][[38282,1],11][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],9]
[kube-infi-2][[38282,1],2][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],12]
[kube-infi-1][[38282,1],19][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],21]
[kube-infi-1][[38282,1],21][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],25]
[kube-infi-1][[38282,1],25][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],27]
[kube-infi-2][[38282,1],8][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],10]
[kube-infi-2][[38282,1],14][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],0]
[kube-infi-1][[38282,1],7][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],17]
[kube-infi-1][[38282,1],17][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],19]
[kube-infi-2][[38282,1],26][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],24]
[kube-infi-2][[38282,1],10][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],14]
[kube-infi-1][[38282,1],27][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],29]
[kube-infi-2][[38282,1],28][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],26]
[kube-infi-2][[38282,1],30][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],28]
[kube-infi-1][[38282,1],29][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],31]
[kube-infi-1][[38282,1],31][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],33]
[kube-infi-2][[38282,1],16][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],6]
[kube-infi-2][[38282,1],24][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],20]
[kube-infi-2][[38282,1],20][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],18]
[kube-infi-2][[38282,1],18][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],16]
[kube-infi-1][[38282,1],13][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],3]
[kube-infi-2][[38282,1],4][btl_tcp_endpoint.c:626:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[38282,1],2]
[kube-infi-1.novalocal:121404] 1 more process has sent help message help-mpi-btl-tcp.txt / server accept cannot find guid
[kube-infi-1.novalocal:121404] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[kube-infi-1.novalocal:121404] 4 more processes have sent help message help-mpi-btl-tcp.txt / client connect fail

OPENIB --mca pml ob1 --mca btl openib,self

mpirun  --mca mtl ^mxm  --mca pml ob1  --mca btl openib,self  -hostfile ~/hostfile --map-by node -np 48  ./osu_alltoallv 

# OSU MPI All-to-Allv Personalized Exchange Latency Test v5.3.2
# Size       Avg Latency(us)
1                     359.18
2                     359.22
4                     360.99
8                     360.02
16                    361.16
32                    358.74
64                    369.90
128                   380.24
256                   389.49
512                   403.87
1024                  439.09
2048                  531.12
4096                  821.04
8192                 1631.51
16384                2477.50
32768                4274.88
65536                9264.67
131072              18034.40
262144              32574.76
524288              64094.50
1048576            127919.69

I am not able to determine whether this is normal behavior for IB in those two nodes. In other words, is IB fully functioning in this case scenario? On the other hand, TCP blocks us from understanding whether or not IB connectivity is happening since it fails, there is no way to compare to IB.

[ggouaillardet]

If you only want to double check Infiniband is used between two nodes, then I'd rather suggest you run osu_latency with 2 nodes and 1 task per node. (Infiniband latency is 1us, TCP is much higher).

If you do want to use btl/tcp, then I suggest you restrict it to a single subnet/interface.
For example, you can

mpirun --mca pml ob1 --mca btl tcp,self --mca btl_tcp_if_include 10.0.0.0/24 ...

if appropriate, you can also use the --mca btl_tcp_if_include eth0 syntax.
(that should just work, plus you can compare TCP over Ethernet vs Infiniband)