SparkRDMA Release Version 3.1

• Improve ODP (on demand paging) performance by using explicit MR registration
• Spark-2.4 support.
• Bug fixes and performance improvements.

SparkRDMA Release Version 3.0

• All-new shuffle protocol that essentially removes the driver from being the bottleneck for serving shuffle blocks metadata. Increases scalability and performance when using higher numbers of mappers, reducers, executors and/or nodes.
• Many bug fixes and performance improvements.

SparkRDMA Release Version 2.0

All-new implementation of SparkRDMA, redesigned from the ground up to further increase scalability, robustness and most importantly - performance.

Among the new features and capabilities introduced in this release:

• All-new Metadata (Map Output) fetching protocol - now allows scaling to the tens of thousands of partitions, with superior performance and recoverability
• Software-level flow control in RdmaChannel - eliminates pause storms in the fabric
• ODP (On-Demand Paging) support - improves memory efficiency

Attached are pre-built binaries. Please follow the README page for instructions.

SparkRDMA Release Version 1.0

This is the first production ready release of SparkRDMA.
Release v1.0 went through rigorous testing for performance and functionality, as well as customer-side pre-production testing.
This version addresses many bugs, and it is considered the best version yet to be published.

Attached are pre-built binaries. Please follow the README page for instructions.

Pre-release version

Pre-release version of SparkRDMA, precursor for version 1.0 that will include a set of fixes and improvements.