Home

SparkRDMA is a high-performance, scalable and efficient ShuffleManager plugin for Apache Spark. It utilizes RDMA (Remote Direct Memory Access) technology to reduce CPU cycles needed for Shuffle data transfers. It reduces memory usage by reusing memory for transfers instead of copying data multiple times down the traditional TCP-stack.

RDMA is supported on various types of networks, such as traditional Ethernet with RoCE (RDMA over Converged Ethernet), Infiniband and more.

SparkRDMA is build to provide the best performance out of the box. However, if one wishes to squeeze the most out of its installation, we provide multiple configuration properties to precisely tune SparkRDMA on a per-job basis. For more information on how to tune a system, please refer to the guides offered in this wiki and to the following Mellanox community posts:

• Reference Deployment Guide for RDMA over Ethernet (RoCE) accelerated Apache Spark 2.2.0 over Mellanox 100 GbE Network
• Description of SparkRDMA's configuration properties
• Recommended performance tuning steps for Mellanox Network Adapters
• General performance tips
• Troubleshooting guide
• Running HiBench suite