[bugfix] Keep updating the node list during the job

[rsarm]

Closes #2567

[codecov-commenter]

Codecov Report

Base: 86.26% // Head: 86.34% // Increases project coverage by +0.08% 🎉

Coverage data is based on head (f64f88c) compared to base (6e68deb).
Patch coverage: 93.10% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2634      +/-   ##
==========================================
+ Coverage   86.26%   86.34%   +0.08%     
==========================================
  Files          60       60              
  Lines       10989    11062      +73     
==========================================
+ Hits         9480     9552      +72     
- Misses       1509     1510       +1     

Impacted Files	Coverage Δ
reframe/core/schedulers/slurm.py	53.47% <75.00%> (+0.92%)	⬆️
reframe/utility/__init__.py	92.94% <100.00%> (+0.21%)	⬆️
reframe/core/variables.py	95.75% <0.00%> (-0.56%)	⬇️
reframe/core/meta.py	99.08% <0.00%> (ø)
reframe/core/pipeline.py	93.40% <0.00%> (ø)
reframe/frontend/cli.py	72.41% <0.00%> (+0.04%)	⬆️
reframe/utility/osext.py	84.67% <0.00%> (+0.12%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[rsarm]

Here the idea is to use the command scontrol show hostname {nodespec}, which I don't think it looks at all to the Slurm database. I think it's just an utility to expand a node list string. For instance

> scontrol show hostname xxx0[00-05]
xxx000
xxx001
xxx002
xxx003
xxx004
xxx005

I was considering checking if len(job.nodelist) is equal to number of nodes and in that case, stop updating the node list. I'm a bit confused though with job.num_tasks_per_node in the case of flexible allocations. If job.num_tasks_per_node is None it means that there's only one node? Or that's not valid for flexible allocations?

[vkarak]

@rsarm I think the best approach is to simply store the nodespec as we get it back from the sacct or squeue and then convert it to a nodelist using scontrol once at the end after the job has finished. This is the easiest implementation, imho, until we implement the inverse of nodelist_abbrev.

[vkarak]

I've updated the PR according to my last suggestion. We save the nodespec with every sacct and we expand it into the nodelist the first time this is requested. I have also added a utility function to expand a nodespec into a nodelist.