Skip to content

CUDA RNDV protocols

Devendar Bureddy edited this page May 16, 2020 · 2 revisions

The following are default rndv protocols with default UCX_RNDV_SCHEME=auto

Inter-node:

GPUDIrectRDMA enabled

  • D-D

    Sender stages cuda fragments and PUTs to destination Device buffer
  • D-H

    Sender stages cuda fragments and PUTs to destination Host buffer
  • H-D

    put_zcopy protocol
  • H-H

    get-zcopy protocol

GPUDIrectRDMA disabled

  • D-D

    Receiver requests fragments from Sender (frag rtr) Sender stages fragments and PUTs to the receiver staging buffer and notifies (frag ATP) Receiver PUTs fragment to Device buffer.
  • D-H

    Sender pipelines cuda fragments and PUTs to destination Host buffer
  • H-D

    Receiver GETs host fragments and PUTs to destination Device buffer
  • H-H

    get-zcopy protocol

Intra-node

CUDA-IPC

  • D-D

    put_zcopy protocol
  • D-H

    Sender stages cuda fragments and PUTs to destination Host buffer
  • H-D

    put_zcopy protocol
  • H-H

    get-zcopy protocol

No CUDA_IPC

  • D-D

    Sender stages cuda fragments and PUTs(IB)to destination Device buffer
  • D-H

    Sender pipelines cuda fragments and PUTs to destination Host buffer
  • H-D

    put_zcopy protocol (IB).
  • H-H

    get-zcopy protocol
Clone this wiki locally