Add CUDA Allgather implementation to support tensor fusion

[jessebenson]

Allgather with CUDA tensors does not always work with tensor fusion/CUDA.

1. Build Horovod without HOROVOD_GPU_*

import torch
import horovod.torch as hvd

hvd.init()

// The following allgather will succeed because it will run on the CPU (manually cudaMemcpy to CPU, run CPU allgather) in pytorch/tensorflow/etc.
tensor = torch.cuda.FloatTensor(*([4])).fill_(hvd.rank())
result = hvd.allgather(tensor)

// Tensor fusion works because the CUDA memory is manually copied to/from CPU
// and AllgatherOp uses std::memcpy for tensor fusion
tensor1 = torch.cuda.FloatTensor(*([4])).fill_(hvd.rank())
tensor2 = torch.cuda.FloatTensor(*([4])).fill_(hvd.rank())
handle1 = hvd.allgather_async(tensor1)
handle2 = hvd.allgather_async(tensor2)
result1 = hvd.synchronize(handle1)
result2 = hvd.synchronize(handle2)

2. Build OpenMPI with CUDA support. Build Horovod with HOROVOD_GPU_ALLREDUCE=MPI and HOROVOD_GPU_ALLGATHER=MPI.

import torch
import horovod.torch as hvd

hvd.init()

// The following allgather will fail because CUDA is not initialized
tensor = torch.cuda.FloatTensor(*([4])).fill_(hvd.rank())
result = hvd.allgather(tensor)

// The following allgather will succeed because MPI_CUDAAllreduce will first initialize CUDA
tensor = torch.cuda.FloatTensor(*([4])).fill_(hvd.rank())
dummy = hvd.allreduce(tensor)
result = hvd.allgather(tensor)

// The following allgather using tensor fusion will fail
// This will invoke AllgatherOp, which unconditionally calls std::memcpy on the pointers, even if they are allocated on a CUDA device
tensor1 = torch.cuda.FloatTensor(*([4])).fill_(hvd.rank())
tensor2 = torch.cuda.FloatTensor(*([4])).fill_(hvd.rank())
handle1 = hvd.allgather_async(tensor1)
handle2 = hvd.allgather_async(tensor2)
result1 = hvd.synchronize(handle1)
result2 = hvd.synchronize(handle2)

[tgaddair]

Very nice! Adding @abditag2 who worked on the original Allgather tensor fusion implementation.

[jessebenson]

I am also preparing a pull request to add "Reducescatter" as a first-class operation type. There's a paper about a memory optimization for distributed training. Instead of doing 'allreduce -> optimizer' they do 'reducescatter -> optimizer -> allgather'. But to do that, I need reducescatter in Horovod :)

https://arxiv.org/abs/1910.02054

[jessebenson]

I rebased on the latest master commit, and added the CUDAOpContext class for CUDAAllreduce and CUDAAllgather to share CUDA init logic.

[tgaddair]

LGTM! Nice work!