The support for libibverbs is essentially the same as for Python
<py-ibverbs>
, with the same limitations. The programmatic interface is via the :cppspead2::recv::udp_ibv_reader
and :cppspead2::send::udp_ibv_stream
classes:
spead2::recv::udp_ibv_config
spead2::recv::udp_ibv_reader
spead2::send::udp_ibv_config
spead2::send::udp_ibv_stream
The pointer given to :cppspead2::send::udp_ibv_config::add_memory_region
is passed to :cpp!ibv_reg_mr
. When using a Mellanox NIC, this can be a pointer that is handled by PeerDirect, such as a GPU device pointer. This can be used to transfer data directly from a GPU to the network without passing though the CPU.
This approach does need some care, because the spead2 implementation will fall back to copying if a packet contains too many discontiguous pieces of memory. It will be safe as long as there is only one item in a heap that uses a registered memory region, or as long as all such items are at least as big as the packet size.
For an example of this, see examples/gpudirect_example.cu
in the spead2 source distribution.