Validate environment variables in SLURMEnvironment and warn user about incompatible settings

[awaelchli]

🚀 Feature

Check slurm environment settings and print warnings if needed.

Motivation

If SLURM srun variables are set incorrectly, the processes can hang and the user will not know why.

Examples:

• Code stuck on "initalizing ddp" when using more than one gpu  #4612 (comment)
• single-node multi-gpu training never starts #15141

Pitch

In the SLURM cluster environment, check for example:

if os.environ.get("SLURM_CPUS_PER_TASK") > 1:
    warn(
        "You asked SLURM for multiple CPUs but we are not sure your machine has multiple CPUs. "
        "If your process hangs, consider setting --cpus-per-task=1"
    )

Alternatives

Do nothing. Users will keep submitting issues :)

Additional context

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

• Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

• Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @Borda @awaelchli

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[awaelchli]

No!

[tchaton]

Great idea! Let's do it.