MPI-enabled PaddlePaddle

[seiriosPlus]

By using MPI API, we enables PaddlePaddle to take advantage of high performance low latency networks such as Infiniband.
There are two benefits:

1. Enable RDMA with PaddlePaddle, which bring high performance low latency networks.
2. Enable GPUDriect with PaddlePaddle, which bring highest throughput and lowest latency GPU read and write.

[wangkuiyi]

It looks to me that in order to utilize InfiniBank and GPDirect via MPI, we need to call MPI_AllReduce?

MPI_AllReduce is mutually exclusive with parameter server, fault tolerance, and elastic scheduling. It would be important to draft a design doc to make sure that Fluid supports both modes -- AllReduce and ParameterServer.

[wangkuiyi]

Also, I noticed that the current distributed computing solution includes a transpiler that generates the ProgramDesc messages for trainers and parameters. If we are going to use MPI_AllReduce to replace parameter servers, do we need a new transpiler that generates the ProgramDesc for trainers in the AllReduce mode?

[seiriosPlus]

We current target speed up PaddlePaddle with Distribution. And for that, We need to call API MPI_Isend and MPI_Irecv to send and receive data between nodes in MPI cluster.
Introduce Open MPI API to PaddlePaddle, which can bring two benefits to PaddlePaddle:

1. Enable RDMA with PaddlePaddle, which bring high-performance low latency networks.
2. Enable GPUDriect with PaddlePaddle, which bring the highest throughput and lowest latency GPU read and write.