Allow users to pass extra options to the job submission command in the partition config

[vkarak]

This is requested by EESSI/test-suite#68 as well as #2968. #2968 introduces a solution that is LSF-specific (in terms of naming), but the idea is to generalise it to the rest of job schedulers.

@boegel @casparvl Would that work in your case?

[boegel]

@vkarak Sorry for not getting back to this earlier....

Yes, a way to pass additional options to the sbatch command should help (at least, as long as options like --partition override environment variables like $SBATCH_PARTITION, which I guess is the case, I didn't check)

[laraPPr]

I was finally able to test it and passing --partition in the sbatch command works on hortense.

[laraPPr]

To give a little more context I will clarify some parts of the setup of hortense and how I tested that passing --partition in the sbatch command would resolve our current issue.

The setup of Hortense:
To help our users we set the environments of the partitions in sticky modules. So that when users want to submit jobs to a specific partition they just have to swap to the right sticky module corresponding with the partition the user wants to submit to.

With ReFrame however we want to be able to submit to all partitions and not just the partition which is set in the environment.

We tried first to set the partition in the config file of hortense. However here the sbatch option is overruled by the environment that is set.

'access': hortense_access + ['--partition=cpu_rome'],

An other option was to see if passing the sbatch option in the command rather than the job script would overrule the environment. To test this I ran a jobscript that was made by a previous run of the ReFrame test suite. The goal was to submit the job to the cpu_milan partition while the cpu_rome environment is set.

The set environment during the test:

Currently Loaded Modules:
  1) env/vsc/dodrio/cpu_rome (S)   2) env/slurm/dodrio/cpu_rome (S)   3) env/software/dodrio/cpu_rome (S)   4) cluster/dodrio/cpu_rome (S)

the sbatch command:

sbatch --partition cpu_milan rfm_job.sh

With the following sbatch command the rfm_job.sh was correctly submitted to the cpu_milan partition.

[vkarak]

Yes, indeed, Slurm precedence is command-line > environment variable > job script. Another possibility for reframe to support this would be to add a configuration parameter in sched_options to instruct reframe to pass the access options in command line instead of the script. This is probably a cleaner solution.