Make the default auto-requeue signal in SLURM environments configurable

[martenlienen]

Description & Motivation

I have to use a requeue-signal other than SIGUSR1 for technical reasons, which is the hard coded default at the moment. While I can use SLURMEnvironment(requeue_signal=signal.SIGUSR2) as a plugin, I would like to only set that in cases where I actually am in a SLURM environment without having to re-implement the environment detection inside of lightning. In other words, I still want to be able to just run my script locally in a LightningEnvironment.

Pitch

I want to move the hard-coded constant here into a class constant, so that I can change it from the outside.

Alternatives

Re-implementing environment detection myself so that I can pass an argument to just the SLURM environment, but that seems like too much work and needless duplication.

Additional context

No response

cc @lantiga

[tuanmp]

In the same context, if the slurm training job is running in a container that does not contain scontrol, automatic requeue would throw an error /bin/sh: 1: scontrol: not found and the job exits without requeuing.

This is caused by lightning trying to run scontrol requeue job_id (

pytorch-lightning/src/lightning/pytorch/trainer/connectors/signal_connector.py

Line 93 in 79ffe50

log.info(f"requeing job {job_id}...")

).

Is there a workaround ?

[martenlienen]

Lightning needs access to scontrol to requeue your job, so I don't see a workaround. But let's keep this issue on topic.