Size of #SBATCH --array larger than MaxSubmitJobs

[mhesselbarth]

Hey,

first of all, thank you for this interesting package. I am only starting to look into it, but so far I like it a lot.

Currently, I am running into the problem that the array in submit.sh is larger then the MaxSubmitJobs allowed by the HPC resulting in sbatch: error: Batch job submission failed: Invalid job array specification.

Is there any way if this happens that rslurm automatically splits the job submission into several arrays but still writes results into the same _rslurm_* folder?

Thanks a lot.

Edit:
Here is how my submit.sh looks like and MaxSubmitJobs is 5,000

#!/bin/bash

#SBATCH --job-name=variability_cont
#SBATCH --array=0-6049
#SBATCH --cpus-per-task=1
#SBATCH --partition=standard
#SBATCH --time=00:10:00
#SBATCH --mem-per-cpu=7G
#SBATCH --output=rslurm.log
#SBATCH --error=rslurm.log

module load gcc/8.2.0
module load R/4.0.2

/sw/arcts/centos7/stacks/gcc/8.2.0/R/4.0.2/lib64/R/bin/Rscript --vanilla slurm_run.R

[qdread]

That's an interesting issue. Currently there is no way to automatically split up the array in rslurm. To implement this, we would have to have slurm_apply() check the value of MaxSubmitJobs and submit multiple job arrays if the number of requested tasks is greater than the maximum. Tricky but feasible. Unfortunately, I currently don't have time to work on adding new features to rslurm. You are always welcome to implement this yourself and submit a pull request. Thanks for submitting the issue.

[qdread]

Following up, I just looked into the Slurm documentation a little bit. It seems like the relevant configuration value here is MaxArraySize, not MaxSubmitJobs, or am I misunderstanding?

[mhesselbarth]

Yes, I think you are right.