RFC: Validate Trainer settings against cluster environment settings

[awaelchli]

🚀 Feature

Introduce a validation check that verifies the Trainer settings like gpus, and number of nodes are set compatible with the cluster environment currently used.

Motivation

There have been several issues where users did not set the number of devices or the number of nodes correctly on clusters (#10098, #8993, #4612). The result: Processes hang for unknown reason. The user is confused, does not know what to do. It's also hard to debug for us because often the user just reports "DDP hangs, help!", and there could be a million other reasons why ddp could stall.

Pitch

Introduce a ClusterEnvironment.validate_distributed_settings method (name up for negotiation):

class ClusterEnvironment:

    def validate_distributed_settings(num_devices, num_nodes, ...):
        # implementation may vary depending on environment
        if self.world_size() != num_devices * num_nodes:
            # raise warning, e.g., user should check SLURM settings for num nodes, num tasks per node, etc.

This would be an optional method on the ClusterEnvironment. The argument list could include also other arguments.

In the accelerator connector, when setting up the environments, we would call this method. Roughly here:
https://github.com/PyTorchLightning/pytorch-lightning/blob/c9bc10ce8473a2249ffa4e00972c0c3c1d2641c4/pytorch_lightning/trainer/connectors/accelerator_connector.py#L794-L807

Alternatives

Alternative suggestions welcome.

Additional context

This would resolve #10098, #8993, #4612, and perhaps others.

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

• Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

• Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

[tchaton]

Dear @awaelchli,

Sounds like a good idea !

As a site note:

Do you think we should refactor:

 def select_cluster_environment(self) -> ClusterEnvironment: 
     if self._cluster_environment is not None: 
         return self._cluster_environment 
     if self.is_slurm_managing_tasks: 
         env = SLURMEnvironment() 
     elif TorchElasticEnvironment.is_using_torchelastic(): 
         env = TorchElasticEnvironment() 
     elif KubeflowEnvironment.is_using_kubeflow(): 
         env = KubeflowEnvironment() 
     elif LSFEnvironment.is_using_lsf(): 
         env = LSFEnvironment() 
     else: 
         env = LightningEnvironment() 
     return env 

as by providing a is_of_cluster_type function

 def select_cluster_environment(self) -> ClusterEnvironment: 
     if self._cluster_environment is not None: 
         return self._cluster_environment 
     if self.is_slurm_managing_tasks: 
         env = SLURMEnvironment() 
     elif TorchElasticEnvironment.is_of_cluster_type(): 
         env = TorchElasticEnvironment() 
     elif KubeflowEnvironment.is_of_cluster_type(): 
         env = KubeflowEnvironment() 
     elif LSFEnvironment.is_of_cluster_type(): 
         env = LSFEnvironment() 
     else: 
         env = LightningEnvironment() 
     return env

[awaelchli]

Yes, generalizing the vairous is_using_xyz to a single method has crossed my mind too! I think that's a next logical step as we are evolving here :)