Improving handling of extra resources with PBS scheduler

[giordano]

As far as I understand, with the PBS Pro scheduler only the last -l select=....:.... option survives. This is problematic with ReFrame, because it makes it harder than necessary to have GPU jobs. I initially had this in my reframe configuration

                    'resources': [
                        {
                            'name': 'gpu',
                            'options': ['-l ngpus=1'],
                        },
                    ],

This would create a new -l ngpus=1 option, but

• this is wrong, because it should be -l select={num_nodes}:ngpus=1,
• even if I used -l select={num_nodes}:ngpus=1, this would override the default -l select={num_nodes}:ncpus={num_cpus} line, in particular it'd override the previous ncpus={num_nodes} setting. In the best case, there would be lots of needless duplication.

There should really be a single -l select line, with all extra resources appended to the existing -l select line, instead of creating a new line.

Also, note that ngpus is a custom resource name I need to use on Isambard 2, I don't think this is a standard name, it isn't documented in the man page of qsub, other system may have other custom names for other resources, so any solution to this problem shouldn't be specific to ngpus resources.

[giordano]

Actually, do I understand correctly I should use

'options': ['ngpus=1'],

to append to the select options?

[vkarak]

Are you also setting the extra_resources in your test? Because the resources in your config are not emitted if not requested with extra_resources by the test. If you want to use that as an access option you should instead pass it as ngpus=1. This will be appended to the single -l select line. There is a note about this also in the resources configuration reference docs (but not in the access docs :-( ):

For the pbs and torque backends, options accepted in the access and resources attributes may either refer to actual qsub options or may be just resources specifications to be passed to the -l option. The backend assumes a qsub option, if the options passed in these attributes start with a -.

[giordano]

Are you also setting the extra_resources in your test?

Yes, I simplified the example above too much by omitting that, but I was indeed setting also extra_resources, otherwise I wouldn't have got the extra line in the job script, which was the whole problem.

If you want to use that as an access option you should instead pass it as ngpus=1. This will be appended to the single -l select line.

Yes, that's what I figured out what I had to do. Probably it's just me, but that wasn't clear from the docs, but it became clearer after reading the code

reframe/reframe/core/schedulers/pbs.py

Lines 95 to 101 in 2811360

	for opt in (*job.sched_access, *job.options, *job.cli_options):
	if opt.startswith('-'):
	rem_opts.append(opt)
	elif opt.startswith('#'):
	verb_opts.append(opt)
	else:
	select_opt += ':' + opt

[vkarak]

I'll keep it open so that we improve at least the docs.