verbs broken on new InfiniBand machines

[jcphill]

Errors during startup.

On a user's machine:

mlx5: atl1-1-01-013-3-l.pace.gatech.edu: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
0000000b 00000000 00000000 00000000
00000000 00008a12 0a00e2dd 000025d2
[0] Assertion "0" failed in file machine-ibverbs.c line 1522.
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: Assertion "0" failed in file machine-ibverbs.c line 1522.

And on a TACC Frontera:

[0] wc[0] status 9 wc[i].opcode 0

mlx5: c101-011.frontera.tacc.utexas.edu: got completion with error:

00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000

00000002 00000000 00000000 00000000

00000000 00008a12 0a03eb2b 0026c1d2

------------- Processor 48 Exiting: Called CmiAbort ------------

Reason: Work completion error in sendCq

[48] Stack Traceback:

[48:0] _Z14CmiAbortHelperPKcS0_S0_ii+0x56 [0x15e7616]

[48:1] CmiAbort+0xd7 [0x15e75b7]

[48:2] _Z24LrtsAdvanceCommunicationi+0x12c4 [0x15d4b14]

[48:3] _Z25CommunicationServerThreadi+0xb [0x15d37eb]

[48:4] [0x15d5676]

[48:5] [0x15d35c3]

[48:6] +0x7dd5 [0x2b337815edd5]

[48:7] clone+0x6d [0x2b3378e7bead]

Charmrun> error on request socket to node 0 '127.0.0.1'--

Socket closed before recv.

[jcphill]

I can reproduce this on Frontier with megatest, building as:

./build charm++ verbs-linux-x86_64 smp iccstatic --no-build-shared --with-production -xCORE-AVX512 -j8 
~/sep2019/charmrun-verbs ++mpiexec ++remote-shell ibrun ++p 52 ++ppn 13 ~/sep2019/megatest-verbs +showcpuaffinity +pemap 4-55:2,5-55:2 +commap 0,2,1,3

[evan-charmworks]

#2555 (now merged) included a fix for ++mpiexec launching. Is this issue affected at all?

[davidhardy]

Any updates on this? We have users from Georgia Tech that are impacted by this issue.

[evan-charmworks]

[0] wc[0] status 9 wc[i].opcode 0

Status 9 after a ibv_poll_cq call indicates IBV_WC_REM_INV_REQ_ERR.

https://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf:

This event is generated when the receive buffer is smaller than the incoming send. It is generated on the sender side of the connection. It may also be generated if the QP attributes are not set correctly, particularly those governing MR access.

[evan-charmworks]

@jcphill Does this fix anything?

diff --git a/src/arch/verbs/machine-ibverbs.C b/src/arch/verbs/machine-ibverbs.C
index 1347320dc..9b9392037 100644
--- a/src/arch/verbs/machine-ibverbs.C
+++ b/src/arch/verbs/machine-ibverbs.C
@@ -866,7 +866,7 @@ void createLocalQps(struct ibv_device *dev,int ibPort, int myNode,int numNodes,s
 		attr.qp_state        = IBV_QPS_INIT;
 		attr.pkey_index      = 0;
 		attr.port_num        = ibPort;
-		attr.qp_access_flags = IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE;
+		attr.qp_access_flags = IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE;
 
 /*		MACHSTATE1(3,"context->pd %p",context->pd);
 		struct ibv_qp *qp = ibv_create_qp(context->pd,&initAttr);

The manual I linked above states,

If IBV_ACCESS_REMOTE_WRITE or IBV_ACCESS_REMOTE_ATOMIC is set, then IBV_ACCESS_LOCAL_WRITE must be set as well.

I can't reproduce the problem on iForge, the only Verbs machine I have access to.

[jcphill]

Frontera is single-user through Wednesday so I can't test until then.
Jim

[evan-charmworks]

@davidhardy Any chance the users from Georgia Tech could try the above change?

[andrewytp]

I am Andrew Pang from JC Gumbart's lab, and the user from Georgia Tech getting this error. Unfortunately, the change you suggested did not solve the issue. I am getting the same error. But I find that if I compile with icc instead of iccstatic, it gives a more complete Stack Traceback as copied below. Hope it helps!

test 0: initiated [groupring (milind)]
[0] wc[0] status 9 wc[i].opcode 0
mlx5: login-hive2.pace.gatech.edu: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000
00000000 00008a12 0a00105a 003665d2
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: Work completion error in sendCq
[0] Stack Traceback:
[0:0] pgm 0x69b307 CmiAbortHelper(char const*, char const*, char const*, int, int)
[0:1] pgm 0x69b2a7 CmiAbort
[0:2] pgm 0x68af06 LrtsAdvanceCommunication(int)
[0:3] pgm 0x68b4bb
[0:4] pgm 0x6a16f3 CcdRaiseCondition
[0:5] pgm 0x69cbfb CsdScheduleForever
[0:6] pgm 0x69cb2e CsdScheduler
[0:7] pgm 0x69b9b9 ConverseInit
[0:8] pgm 0x58efe7 charm_main
[0:9] libc.so.6 0x7ffff46373d5 __libc_start_main
[0:10] pgm 0x4eb1e9
Charmrun> error on request socket to node 0 '127.0.0.1'--
Socket closed before recv.

[evan-charmworks]

Thanks for trying that out. Until someone from Charmworks or PPL can investigate this issue directly we may not be able to make any progress with it. For now, I would recommend trying the UCX machine layer which also makes use of Infiniband hardware: https://charm.readthedocs.io/en/latest/charm++/manual.html#ucx

[evan-charmworks]

I am now able to reproduce the issue on Thor. I built Charm with ./build charm++ verbs-linux-x86_64 smp icc --no-build-shared -g -j8 and ran with megatest +p3.

[evan-charmworks]

This is proving extremely difficult to debug. I can reproduce it with GCC instead of ICC, which makes debugging with GDB a little bit easier. It happens in non-SMP mode as well as SMP mode.

In addition to the .status 9 (IBV_WC_REM_INV_REQ_ERR), there is a .vendor_err field with value 138 (or 0x8a), which I cannot find any information on despite searching the internet and looking at both kernel and userland sources of the mlx5 driver stack.

In the libmlx5 userland code there is a struct mlx5_err_cqe with a .vendor_err_synd with the same 0x8a value, and .syndrome field with value 0x12 (MLX5_CQE_SYNDROME_REMOTE_INVAL_REQ_ERR). This structure is the hex dump following "got completion with error".

Running dmesg after a failure shows the following:

[145291.904971] mlx5_core 0000:04:00.0: mlx5_cmd_check:795:(pid 9993): RTR2RTS_QP(0x504) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x4bb88f)
[145291.904974] mlx5_core 0000:04:00.0: mlx5_cmd_check:795:(pid 9992): RTR2RTS_QP(0x504) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x4bb88f)
[145291.904985] mlx5_core 0000:04:00.0: mlx5_cmd_check:795:(pid 9994): RTR2RTS_QP(0x504) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x4bb88f)
[145291.905192] mlx5_core 0000:04:00.0: mlx5_cmd_check:795:(pid 9993): RTR2RTS_QP(0x504) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x4bb88f)
[145291.905235] mlx5_core 0000:04:00.0: mlx5_cmd_check:795:(pid 9994): RTR2RTS_QP(0x504) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x4bb88f)
[145291.905238] mlx5_core 0000:04:00.0: mlx5_cmd_check:795:(pid 9992): RTR2RTS_QP(0x504) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x4bb88f)

However, this is a red herring, as there is code in initInfiOtherNodeData that detects error 22 from ibv_modify_qp and tries again with different parameters. This is where the message comes from and it happens even during successful runs.

I enabled MACHINE_DEBUG and set MACHINE_DEBUG_PRIO to 1 (the highest verbosity) and got no useful information that way either.