Snakemake 8 not usable with Slurm clusters offering GPUs

[w8jcik]

In order to use GPUs one needs to be able to ask Slurm for a GPU.

The usual way to do this is --gres gpu:1 or --gpus 1 switches, which both go to slurm_extra in the Snakemake profile.

slurm_extra does not work in Snakemake 8, it is rejected.

You might be aware of the issue, but maybe you think it is already fixed.

I am creating this issue as a reminder.

Prior discussion snakemake/snakemake-executor-plugin-slurm#18

Some workflows really need GPUs, otherwise they do not finish in a feasible time.

[cmeesters]

As far as I can see, a profile with

slurm_extra: "'--gres gpu:1'" 

gets accepted on submission, but fails in a job context, right?

[cmeesters]

I will prepare a simple PR.

[w8jcik]

As far as I can see, a profile with

slurm_extra: "'--gres gpu:1'" 

gets accepted on submission, but fails in a job context, right?

That is correct.

[w8jcik]

Should this PR #2711 fix the issue?

various bug fixes for resource parsing

I have tried the change by updating Snakemake to 8.4.12

1. With following profile:

default-resources:
  ...
  slurm_extra: "'--gres=gpu:1'"

it fails when it gets a slot in the Slurm queue

Error:
  WorkflowError:
    Failed to evaluate resources value '--gres=gpu:1'.
        String arguments may need additional quoting. E.g.: --default-resources "tmpdir='/home/user/tmp'" or --set-resources "compute1:slurm_extra='--nice=100'".
    SyntaxError: invalid syntax (<string>, line 1)

2. With following profile

default-resources:
  ...
  slurm_extra: "--gres=gpu:1"

it fails right away

Error:
  WorkflowError:
    Failed to evaluate resources value '--gres=gpu:1'.
        String arguments may need additional quoting. E.g.: --default-resources "tmpdir='/home/user/tmp'" or --set-resources "compute1:slurm_extra='--nice=100'".

[cmeesters]

Please try: slurm_extra: "'--gres=gpu:1'". Why? Because slurm_extra: "'--gres=gpu:1 --nice=100'" ought to work, too.

However, it seems to me that this is only a partial fix. Human-readable runtime specs in a workflow profile still raise an error.

[w8jcik]

Please try: slurm_extra: "'--gres=gpu:1'".

I am not sure if I understand. This is exactly what I used and described in the previous message. Am I missing some detail?

[cmeesters]

In these threads there might be someone overlooking something and this someone might be me. Such things just happen. I only looked at point two, when writing. Sorry.

I am, however, confused altogether regarding the parsing state right now. Therefore, I will submit a few additional test cases.

[aryarm]

does this issue still exist? I'm still running into it with a profile like this:

default-resources:
    slurm_partition: condo
    slurm_extra: "'--qos=condo'"

downgrading to snakemake 7 seems to fix the issue

[cmeesters]

no, the issue is fixed. I do not run into any issues in this regard. If you run into any, please report in the slurm-executor repo - be sure to be running snakemake +v8.6

[sahuno]

this should work!
in your config.yaml file, specify gpu for individual rules slurm_extra: "'--gpus=2'"

set-resources:
    rule_name:
        #....
        slurm_extra: "'--gpus=2'"