I am seeing the following intermittent errors on our new HPE Apollo system with Open MPI 4.0.5 and UCX 1.9.0:
[r40c1t7n3:108757:0:108757] ib_mlx5_log.c:136 Transport retry count exceeded on mlx5_1:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[r40c1t7n3:108757:0:108757] ib_mlx5_log.c:136 RC QP 0x33945 wqe[386]: SEND s-e [va 0x14ebbaf7d680 len 474 lkey 0x70f41]
[r40c1t7n3:108756:0:108756] ib_mlx5_log.c:136 Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[r40c1t7n3:108756:0:108756] ib_mlx5_log.c:136 RC QP 0x32efe wqe[385]: SEND s-e [va 0x151ff55be680 len 474 lkey 0x10334f]
Some runs succeed, some runs fail. It's not always reproducible. I do not see any previous errors in the logs. Any idea what might cause such errors?
$ ucx_info -v
# UCT version=1.9.0 revision cd9efd3
# configured with: --prefix=$HOME/opt/ucx-1.9.0 --enable-mt --disable-backtrace-detail --enable-stats --without-mpi --with-ib-hw-tm --with-mlx5-dv --with-dc --with-ud --with-rc --with-dm --with-cm --with-avx --enable-devel-headers --enable-experimental-api CC=gcc CXX=g++
GCC 9.2.0 was used.
$ uname -a
Linux hawk-login05 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ rpm -q libibverbs
libibverbs-41mlnx1-OFED.4.7.0.0.2.47329.x86_64
$ ofed_info -s
MLNX_OFED_LINUX-4.7-3.2.9.0:
$ ibv_devinfo -vv
hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 20.27.6202
node_guid: b883:03ff:ffa3:f05c
sys_image_guid: b883:03ff:ffa3:f058
vendor_id: 0x02c9
vendor_part_id: 4123
hw_ver: 0x0
board_id: HPE0000000051
phys_port_cnt: 1
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffffffffff000
max_qp: 262144
max_qp_wr: 32768
device_cap_flags: 0xe17e1c36
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
XRC
Unknown flags: 0xe16e0000
device_cap_exp_flags: 0x5648F8F100000000
EXP_DC_TRANSPORT
EXP_CROSS_CHANNEL
EXP_MR_ALLOCATE
EXT_ATOMICS
EXT_SEND NOP
EXP_UMR
EXP_ODP
EXP_RX_CSUM_TCP_UDP_PKT
EXP_RX_CSUM_IP_PKT
EXP_DC_INFO
EXP_MASKED_ATOMICS
EXP_RX_TCP_UDP_PKT_TYPE
EXP_PHYSICAL_RANGE_MR
EXP_UMR_FIXED_SIZE
EXP_PACKET_BASED_CREDIT_MODE
Unknown flags: 0x200000000000
max_sge: 30
max_sge_rd: 30
max_cq: 16777216
max_cqe: 4194303
max_mr: 16777216
max_pd: 16777216
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 4194304
max_qp_init_rd_atom: 16
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
log atomic arg sizes (mask) 0x8
masked_log_atomic_arg_sizes (mask) 0x3c
masked_log_atomic_arg_sizes_network_endianness (mask) 0x34
max fetch and add bit boundary 64
log max atomic inline 5
max_ee: 0
max_rdd: 0
max_mw: 16777216
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 2097152
max_mcast_qp_attach: 240
max_total_mcast_qp_attach: 503316480
max_ah: 2147483647
max_fmr: 0
max_srq: 8388608
max_srq_wr: 32767
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 16
hca_core_clock: 156250
max_klm_list_size: 65536
max_send_wqe_inline_klms: 20
max_umr_recursion_depth: 4
max_umr_stride_dimension: 1
general_odp_caps:
ODP_SUPPORT
ODP_SUPPORT_IMPLICIT
max_size: 0xFFFFFFFFFFFFFFFF
rc_odp_caps:
SUPPORT_SEND
SUPPORT_RECV
SUPPORT_WRITE
SUPPORT_READ
SUPPORT_SRQ_RECV
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
SUPPORT_SEND
dc_odp_caps:
SUPPORT_SEND
SUPPORT_WRITE
SUPPORT_READ
xrc_odp_caps:
NO SUPPORT
raw_eth_odp_caps:
NO SUPPORT
max_dct: 262144
max_device_ctx: 1020
Multi-Packet RQ supported
Supported for objects type:
IBV_EXP_MP_RQ_SUP_TYPE_SRQ_TM
IBV_EXP_MP_RQ_SUP_TYPE_WQ_RQ
Supported payload shifts:
2 bytes
Log number of strides for single WQE: 3 - 16
Log number of bytes in single stride: 6 - 13
rx_pad_end_addr_align: 64
tso_caps:
max_tso: 0
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
ooo_caps:
ooo_rc_caps = 0x1
ooo_xrc_caps = 0x1
ooo_dc_caps = 0x1
ooo_ud_caps = 0x0
SUPPORT_RC_RW_DATA_PLACEMENT
SUPPORT_XRC_RW_DATA_PLACEMENT
SUPPORT_DC_RW_DATA_PLACEMENT
sw_parsing_caps:
supported_qp:
max_rndv_hdr_size: 0x40
max_num_tags: 0x7f
max_ops: 0x8000
max_sge: 0x1
capability_flags:
IBV_EXP_TM_CAP_RC
IBV_EXP_TM_CAP_DC
tunnel_offloads_caps:
UMR fixed size:
max entity size: 2147483648
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 10752
port_lid: 7564
port_lmc: 0x00
link_layer: InfiniBand
max_msg_sz: 0x40000000
port_cap_flags: 0x2241e848
max_vl_num: 4 (3)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 128
gid_tbl_len: 8
subnet_timeout: 18
init_type_reply: 0
active_width: 4X (2)
active_speed: 50.0 Gbps (64)
phys_state: LINK_UP (5)
GID[ 0]: fe80:0000:0000:0744:b883:03ff:ffa3:f05c
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 20.27.6202
node_guid: b883:03ff:ffa3:f058
sys_image_guid: b883:03ff:ffa3:f058
vendor_id: 0x02c9
vendor_part_id: 4123
hw_ver: 0x0
board_id: HPE0000000051
phys_port_cnt: 1
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffffffffff000
max_qp: 262144
max_qp_wr: 32768
device_cap_flags: 0xe17e1c36
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
XRC
Unknown flags: 0xe16e0000
device_cap_exp_flags: 0x5648F8F100000000
EXP_DC_TRANSPORT
EXP_CROSS_CHANNEL
EXP_MR_ALLOCATE
EXT_ATOMICS
EXT_SEND NOP
EXP_UMR
EXP_ODP
EXP_RX_CSUM_TCP_UDP_PKT
EXP_RX_CSUM_IP_PKT
EXP_DC_INFO
EXP_MASKED_ATOMICS
EXP_RX_TCP_UDP_PKT_TYPE
EXP_PHYSICAL_RANGE_MR
EXP_UMR_FIXED_SIZE
EXP_PACKET_BASED_CREDIT_MODE
Unknown flags: 0x200000000000
max_sge: 30
max_sge_rd: 30
max_cq: 16777216
max_cqe: 4194303
max_mr: 16777216
max_pd: 16777216
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 4194304
max_qp_init_rd_atom: 16
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
log atomic arg sizes (mask) 0x8
masked_log_atomic_arg_sizes (mask) 0x3c
masked_log_atomic_arg_sizes_network_endianness (mask) 0x34
max fetch and add bit boundary 64
log max atomic inline 5
max_ee: 0
max_rdd: 0
max_mw: 16777216
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 2097152
max_mcast_qp_attach: 240
max_total_mcast_qp_attach: 503316480
max_ah: 2147483647
max_fmr: 0
max_srq: 8388608
max_srq_wr: 32767
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 16
hca_core_clock: 156250
max_klm_list_size: 65536
max_send_wqe_inline_klms: 20
max_umr_recursion_depth: 4
max_umr_stride_dimension: 1
general_odp_caps:
ODP_SUPPORT
ODP_SUPPORT_IMPLICIT
max_size: 0xFFFFFFFFFFFFFFFF
rc_odp_caps:
SUPPORT_SEND
SUPPORT_RECV
SUPPORT_WRITE
SUPPORT_READ
SUPPORT_SRQ_RECV
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
SUPPORT_SEND
dc_odp_caps:
SUPPORT_SEND
SUPPORT_WRITE
SUPPORT_READ
xrc_odp_caps:
NO SUPPORT
raw_eth_odp_caps:
NO SUPPORT
max_dct: 262144
max_device_ctx: 1020
Multi-Packet RQ supported
Supported for objects type:
IBV_EXP_MP_RQ_SUP_TYPE_SRQ_TM
IBV_EXP_MP_RQ_SUP_TYPE_WQ_RQ
Supported payload shifts:
2 bytes
Log number of strides for single WQE: 3 - 16
Log number of bytes in single stride: 6 - 13
rx_pad_end_addr_align: 64
tso_caps:
max_tso: 0
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
ooo_caps:
ooo_rc_caps = 0x1
ooo_xrc_caps = 0x1
ooo_dc_caps = 0x1
ooo_ud_caps = 0x0
SUPPORT_RC_RW_DATA_PLACEMENT
SUPPORT_XRC_RW_DATA_PLACEMENT
SUPPORT_DC_RW_DATA_PLACEMENT
sw_parsing_caps:
supported_qp:
max_rndv_hdr_size: 0x40
max_num_tags: 0x7f
max_ops: 0x8000
max_sge: 0x1
capability_flags:
IBV_EXP_TM_CAP_RC
IBV_EXP_TM_CAP_DC
tunnel_offloads_caps:
UMR fixed size:
max entity size: 2147483648
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 10752
port_lid: 1974
port_lmc: 0x00
link_layer: InfiniBand
max_msg_sz: 0x40000000
port_cap_flags: 0x2251e848
max_vl_num: 4 (3)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 128
gid_tbl_len: 8
subnet_timeout: 18
init_type_reply: 0
active_width: 4X (2)
active_speed: 50.0 Gbps (64)
phys_state: LINK_UP (5)
GID[ 0]: fe80:0000:0000:0744:b883:03ff:ffa3:f058
ucx_info -d
#
# Memory domain: posix
# Component: posix
# allocate: unlimited
# remote key: 29 bytes
# rkey_ptr is supported
#
# Transport: posix
# Device: memory
#
# capabilities:
# bandwidth: 12179.00 + 0.00 MB/sec
# latency: 80 nsec
# overhead: 10 nsec
# put_short: <= 4294967295
# put_bcopy: unlimited
# get_bcopy: unlimited
# am_short: <= 92
# am_bcopy: <= 8K
# domain: cpu
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# priority: 0
# device address: 8 bytes
# iface address: 16 bytes
# error handling: none
#
#
# Memory domain: sysv
# Component: sysv
# allocate: unlimited
# remote key: 24 bytes
# rkey_ptr is supported
#
# Transport: sysv
# Device: memory
#
# capabilities:
# bandwidth: 12179.00 + 0.00 MB/sec
# latency: 80 nsec
# overhead: 10 nsec
# put_short: <= 4294967295
# put_bcopy: unlimited
# get_bcopy: unlimited
# am_short: <= 92
# am_bcopy: <= 8K
# domain: cpu
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# priority: 0
# device address: 8 bytes
# iface address: 16 bytes
# error handling: none
#
#
# Memory domain: self
# Component: self
# register: unlimited, cost: 0 nsec
# remote key: 0 bytes
#
# Transport: self
# Device: memory
#
# capabilities:
# bandwidth: 6911.00 + 0.00 MB/sec
# latency: 0 nsec
# overhead: 10 nsec
# put_short: <= 4294967295
# put_bcopy: unlimited
# get_bcopy: unlimited
# am_short: <= 8K
# am_bcopy: <= 8K
# domain: cpu
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# priority: 0
# device address: 0 bytes
# iface address: 8 bytes
# error handling: none
#
#
# Memory domain: tcp
# Component: tcp
# register: <= 0, cost: 0 nsec
# remote key: 0 bytes
#
# Transport: tcp
# Device: ib0
#
# capabilities:
# bandwidth: 0.00 + 22285.02 MB/sec
# latency: 5203 nsec
# overhead: 50000 nsec
# am_short: <= 8K
# am_bcopy: <= 8K
# am_zcopy: <= 128K, up to 6 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 2044
# am header: <= 8037
# connection: to iface
# priority: 1
# device address: 4 bytes
# iface address: 2 bytes
# error handling: none
#
# Transport: tcp
# Device: ib1
#
# capabilities:
# bandwidth: 0.00 + 22285.02 MB/sec
# latency: 5203 nsec
# overhead: 50000 nsec
# am_short: <= 8K
# am_bcopy: <= 8K
# am_zcopy: <= 128K, up to 6 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 2044
# am header: <= 8037
# connection: to iface
# priority: 1
# device address: 4 bytes
# iface address: 2 bytes
# error handling: none
#
# Transport: tcp
# Device: bond0
#
# capabilities:
# bandwidth: 0.00 + 113.16 MB/sec
# latency: 5776 nsec
# overhead: 50000 nsec
# am_short: <= 8K
# am_bcopy: <= 8K
# am_zcopy: <= 128K, up to 6 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 1500
# am header: <= 8037
# connection: to iface
# priority: 1
# device address: 4 bytes
# iface address: 2 bytes
# error handling: none
#
#
# Memory domain: sockcm
# Component: sockcm
# supports client-server connection establishment via sockaddr
# < no supported devices found >
#
# Memory domain: mlx5_1
# Component: ib
# register: unlimited, cost: 90 nsec
# remote key: 8 bytes
# local memory handle is required for zcopy
#
# Transport: rc
# Device: mlx5_1:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 600 nsec + 1 * N
# overhead: 75 nsec
# put_short: <= 124
# put_bcopy: <= 8K
# put_zcopy: <= 1G, up to 8 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8K
# get_zcopy: 65..1G, up to 8 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 123
# am_bcopy: <= 8191
# am_zcopy: <= 8191, up to 7 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 127
# domain: device
# atomic_add: 64 bit
# atomic_fadd: 64 bit
# atomic_cswap: 64 bit
# connection: to ep
# priority: 50
# device address: 19 bytes
# ep address: 3 bytes
# error handling: peer failure
#
#
# Transport: rc_mlx5
# Device: mlx5_1:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 600 nsec + 1 * N
# overhead: 40 nsec
# put_short: <= 2K
# put_bcopy: <= 8K
# put_zcopy: <= 1G, up to 8 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8K
# get_zcopy: 65..1G, up to 8 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 2046
# am_bcopy: <= 8190
# am_zcopy: <= 8190, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 186
# domain: device
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to ep
# priority: 50
# device address: 19 bytes
# ep address: 7 bytes
# error handling: buffer (zcopy), remote access, peer failure
#
#
# Transport: dc_mlx5
# Device: mlx5_1:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 660 nsec
# overhead: 40 nsec
# put_short: <= 2K
# put_bcopy: <= 8K
# put_zcopy: <= 1G, up to 8 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8K
# get_zcopy: 65..1G, up to 8 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 2046
# am_bcopy: <= 8190
# am_zcopy: <= 8190, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 138
# domain: device
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# priority: 50
# device address: 19 bytes
# iface address: 5 bytes
# error handling: buffer (zcopy), remote access, peer failure
#
#
# Transport: ud
# Device: mlx5_1:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 610 nsec
# overhead: 105 nsec
# am_short: <= 116
# am_bcopy: <= 4088
# am_zcopy: <= 4088, up to 7 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 3984
# connection: to ep, to iface
# priority: 50
# device address: 19 bytes
# iface address: 3 bytes
# ep address: 6 bytes
# error handling: peer failure
#
#
# Transport: ud_mlx5
# Device: mlx5_1:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 610 nsec
# overhead: 80 nsec
# am_short: <= 180
# am_bcopy: <= 4088
# am_zcopy: <= 4088, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 132
# connection: to ep, to iface
# priority: 50
# device address: 19 bytes
# iface address: 3 bytes
# ep address: 6 bytes
# error handling: peer failure
#
#
# Transport: cm
# Device: mlx5_1:1
#
# capabilities:
# bandwidth: 0.00 + 18658.85 MB/sec
# latency: 600 nsec
# overhead: 1200 nsec
# am_bcopy: <= 214
# connection: to iface
# priority: 50
# device address: 19 bytes
# iface address: 4 bytes
# error handling: none
#
#
# Memory domain: mlx5_0
# Component: ib
# register: unlimited, cost: 90 nsec
# remote key: 8 bytes
# local memory handle is required for zcopy
#
# Transport: rc
# Device: mlx5_0:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 600 nsec + 1 * N
# overhead: 75 nsec
# put_short: <= 124
# put_bcopy: <= 8K
# put_zcopy: <= 1G, up to 8 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8K
# get_zcopy: 65..1G, up to 8 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 123
# am_bcopy: <= 8191
# am_zcopy: <= 8191, up to 7 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 127
# domain: device
# atomic_add: 64 bit
# atomic_fadd: 64 bit
# atomic_cswap: 64 bit
# connection: to ep
# priority: 50
# device address: 19 bytes
# ep address: 3 bytes
# error handling: peer failure
#
#
# Transport: rc_mlx5
# Device: mlx5_0:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 600 nsec + 1 * N
# overhead: 40 nsec
# put_short: <= 2K
# put_bcopy: <= 8K
# put_zcopy: <= 1G, up to 8 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8K
# get_zcopy: 65..1G, up to 8 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 2046
# am_bcopy: <= 8190
# am_zcopy: <= 8190, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 186
# domain: device
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to ep
# priority: 50
# device address: 19 bytes
# ep address: 7 bytes
# error handling: buffer (zcopy), remote access, peer failure
#
#
# Transport: dc_mlx5
# Device: mlx5_0:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 660 nsec
# overhead: 40 nsec
# put_short: <= 2K
# put_bcopy: <= 8K
# put_zcopy: <= 1G, up to 8 iov
# put_opt_zcopy_align: <= 512
# put_align_mtu: <= 4K
# get_bcopy: <= 8K
# get_zcopy: 65..1G, up to 8 iov
# get_opt_zcopy_align: <= 512
# get_align_mtu: <= 4K
# am_short: <= 2046
# am_bcopy: <= 8190
# am_zcopy: <= 8190, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 138
# domain: device
# atomic_add: 32, 64 bit
# atomic_and: 32, 64 bit
# atomic_or: 32, 64 bit
# atomic_xor: 32, 64 bit
# atomic_fadd: 32, 64 bit
# atomic_fand: 32, 64 bit
# atomic_for: 32, 64 bit
# atomic_fxor: 32, 64 bit
# atomic_swap: 32, 64 bit
# atomic_cswap: 32, 64 bit
# connection: to iface
# priority: 50
# device address: 19 bytes
# iface address: 5 bytes
# error handling: buffer (zcopy), remote access, peer failure
#
#
# Transport: ud
# Device: mlx5_0:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 610 nsec
# overhead: 105 nsec
# am_short: <= 116
# am_bcopy: <= 4088
# am_zcopy: <= 4088, up to 7 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 3984
# connection: to ep, to iface
# priority: 50
# device address: 19 bytes
# iface address: 3 bytes
# ep address: 6 bytes
# error handling: peer failure
#
#
# Transport: ud_mlx5
# Device: mlx5_0:1
#
# capabilities:
# bandwidth: 0.00 + 23588.47 MB/sec
# latency: 610 nsec
# overhead: 80 nsec
# am_short: <= 180
# am_bcopy: <= 4088
# am_zcopy: <= 4088, up to 3 iov
# am_opt_zcopy_align: <= 512
# am_align_mtu: <= 4K
# am header: <= 132
# connection: to ep, to iface
# priority: 50
# device address: 19 bytes
# iface address: 3 bytes
# ep address: 6 bytes
# error handling: peer failure
#
#
# Transport: cm
# Device: mlx5_0:1
#
# capabilities:
# bandwidth: 0.00 + 18658.85 MB/sec
# latency: 600 nsec
# overhead: 1200 nsec
# am_bcopy: <= 214
# connection: to iface
# priority: 50
# device address: 19 bytes
# iface address: 4 bytes
# error handling: none
#
#
# Memory domain: cma
# Component: cma
# register: unlimited, cost: 9 nsec
#
# Transport: cma
# Device: memory
#
# capabilities:
# bandwidth: 11145.00 + 0.00 MB/sec
# latency: 80 nsec
# overhead: 400 nsec
# put_zcopy: unlimited, up to 16 iov
# put_opt_zcopy_align: <= 1
# put_align_mtu: <= 1
# get_zcopy: unlimited, up to 16 iov
# get_opt_zcopy_align: <= 1
# get_align_mtu: <= 1
# connection: to iface
# priority: 0
# device address: 8 bytes
# iface address: 4 bytes
# error handling: none
#
Describe the bug
I am seeing the following intermittent errors on our new HPE Apollo system with Open MPI 4.0.5 and UCX 1.9.0:
Some runs succeed, some runs fail. It's not always reproducible. I do not see any previous errors in the logs. Any idea what might cause such errors?
Steps to Reproduce
GCC 9.2.0 was used.
Setup and versions
ibstatoribv_devinfo -vvcommandAdditional information (depending on the issue)
ucx_info -dto show transports and devices recognized by UCXI was unable to reproduce this issue with logging enabled.