data parallel distributed training

[SunDoge]

I'm a pytorch and mxnet user and Flux looks promising to me. I have 8 GPUs on the server and I want to train my model faster. Unfortunately, I see no document about parallel training on multiple GPUs. Is it possible to copy the model and initialize them on multiple GPUs and split the input data for them?

I find a pr #154 and it suggests that there may be difficulties on deep copy. Is there any progress now?

[MikeInnes]

I don't think it's too difficult to do if you're willing to get your hands dirty; you'll want to look at NCCL and make copies of the model for each GPU, and run a training loop over all of them. If you get anywhere, it'd be great to have additions to the docs etc.

Unfortunately we don't have anything out of the box yet, though.

[ToucheSir]

One can now choose between https://github.com/avik-pal/FluxMPI.jl, https://github.com/FluxML/DaggerFlux.jl/ and https://github.com/DhairyaLGandhi/ResNetImageNet.jl. The general consensus seems to be that this is not something that should be handled in core (at least for now), so further discussion should happen outside of the issue tracker.

[CarloLucibello]

I'd like to reopen this since all the packages mentioned are very experimental and not ready for prime time and also since I think that a distributed training solution should be a concern of Flux.jl. Maybe the solution will be implemented in a separate package, but I think we should at the very least document and give some examples of the "recommended" way to do data-parallel distributed training. We can discuss here what that recommended way should be.

[AriMKatz]

@tkf can we have an easy win with transducers here ?

[CarloLucibello]

@avik-pal I like the simplicity of https://github.com/avik-pal/FluxMPI.jl interface. What is the package currently missing?
A (minor?) annoyance is having to run a training script with mpiexecjl -n <np> julia --project=. <filename>.jl, I would prefer to be able to run it in a standard julia session.

[ToucheSir]

For one we still lack a CUDA-aware MPI JLL: JuliaPackaging/Yggdrasil#2063. I agree with you about the niceness of the interface though, I think it's also the only solution that handles efficient multi-GPU reductions right now.

[avik-pal]

A (minor?) annoyance is having to run a training script with mpiexecjl -n julia --project=. .jl, I would prefer to be able to run it in a standard julia session.

If we want to use MPI, we need to start the julia session like that. The alternative would be to use Distributed.jl for performing the reduction (this would allow us to use a standard julia session)

[CarloLucibello]

The alternative would be to use Distributed.jl for performing the reduction (this would allow us to use a standard julia session)

Ok, but we can think to that later, it's not too important. Anything important to iron out before advertising it and working a bit on the docs/examples?

[avik-pal]

Nothing particular that I am working on at the moment. I have already tested it out for a paper where we scale training on 6 processes.

[AriMKatz]

I think dagger.jl would be better than distributed. Seems more robust to things like worker failure, different topologies, smarter scheduling. Any thoughts @jpsamaroo ?

[avik-pal]

For one we still lack a CUDA-aware MPI JLL

Just to put it out there. In case the compiled MPI is not CUDA aware, FluxMPI will do the reductions on CPU. Though yeah it would be great have a compiled JLL.

[jpsamaroo]

Parallel/distributing training is a large design space, and MPI covers a big portion of it, but also does so in a rigid manner that makes interactive development more cumbersome, and has certain assumptions about how programs will be written, which makes it difficult to do anything that's inherently task-parallel (which is often desired for logging, visualization, or integration with other Julia libraries which use multitasking).

I'm all for pushing FluxMPI.jl as one possible means to parallelize training (and the currently recommended approach), but I wouldn't advertise it (MPI) as the best way to do so generally, and would point out that there are other potential alternatives in incubation.