[bugfix] Fix use of sched_options by the backend schedulers

[vkarak]

This PR fixes three bugs related to sched_options. Two were introduced in 4.0 and one was present since the introduction of the use_nodes_option parameter.

1. Scheduler backends were not accessing the correct configuration section due in order to get the partition's sched_options due to a typo! This is fixed, unit tests are added to verify it and the configuration schema for accessing the sched_options has become stricter in order to avoid typos going undetected in the future.
2. Fix the configuration schema and allow sched_options to be defined inside the systems section, as it is also stated in the docs.
3. Fix crash of the use_nodes_option when then num_tasks_per_node was not set. This bug was present since the introduction of the use_nodes_option. If num_tasks_per_node is not set in this case, it is considered as 1.

Fixes #2794.
Fixes #2795.

Fyi @smoors

[codecov-commenter]

Codecov Report

Base: 86.59% // Head: 86.58% // Decreases project coverage by -0.01% ⚠️

Coverage data is based on head (b13d9c4) compared to base (039023b).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2796      +/-   ##
==========================================
- Coverage   86.59%   86.58%   -0.01%     
==========================================
  Files          60       60              
  Lines       11232    11233       +1     
==========================================
  Hits         9726     9726              
- Misses       1506     1507       +1     

Impacted Files	Coverage Δ
reframe/core/schedulers/__init__.py	98.39% <ø> (ø)
reframe/core/schedulers/slurm.py	52.92% <100.00%> (+0.12%)	⬆️
reframe/frontend/executors/policies.py	92.54% <0.00%> (-0.28%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[giordano]

Shouldn't the docs be updated to reflect the fact sched_options can't be used at all in practice in v4.0.0?

[vkarak]

I'm not sure that this would help, as the docs for 4.0 (not the latest or stable to) would still claim that sched_options work.

[giordano]

I was reading stable docs and assumed I could use sched_options with v4.0.0 and discovered the hard way that's not true.