Possible using torch DDP(DistributedDataParallel)?

[ericxsun]

Is your feature request related to a problem? Please describe.
How could we train model with torch DDP?

[tgaddair]

Hey @ericxsun, adding DDP is something we've been considering lately, but haven't prioritized it since Horovod is currently supported and it's not clear what the added benefit of DDP would be for most users. Is there a reason you would prefer to use DDP instead of Horovod for distributed training in Ludwig?

[ericxsun]

Thanks @tgaddair

Just be familiar with DDP and technical stack in the group is native PyTorch。Change to Horovod may not be possible.

Could you give me some points, so I can do it in Ludwig with PyTorch DDP? Thank you very much

[tgaddair]

Hey @ericxsun, are you also able to use Ray with Ludwig? If so, it shouldn't be a problem integrating the existing distributed training into your stack. The only issue would be making sure the environment has the right dependencies, however, we provide a Docker image (ludwig-ray-gpu) that has all the dependencies, including Horovod, pre-installed.

If you're not able to use Ray, you can still do distributed training, but not distributed data preprocessing, which limits the scale of the data you can use for training to something that can fit entirely in memory.

To integrate DDP into Ludwig there are really two main touch points. The first is the Ray Trainer, which would need to be changed to support Ray's TorchConfig here. The other would the in trainer.py where all the calls to self.horovod would need to be changed to the DDP equivalent API calls.

Let me know if you'd like to discuss this further, I'd be happy to put together a quick prototype using DDP at some point soon if it would be helpful.

[tgaddair]

Hey @ericxsun, I've implement an initial version of DDP integration with Ray in #2890. Let me know if this works for your use case. I will do some benchmarking to see how it compares with Horovod in terms of performance before landing.

[ericxsun]

Hi @tgaddair, sorry for late reply. Thank you so much. I'll try the your implementation of DDP integration with Ray in #2890. Thank you again.

[tgaddair]

Thanks @ericxsun, let me know if it works for you. I haven't done any testing to verify correctness yet beyond small local tests, so let me know if you encounter any issues!

[tgaddair]

I made some updates earlier that fixed gpu support. After running benchmarks, the performance was almost identical to Horovod, with and without AMP, so should be good to merge in the PR once it's reviewed.