Job that gets cancelled prematurely by ReFrame

[casparvl]

I think I found a bug in the SLURM scheduler implementation of ReFrame. The situation is the following: I'm writing a test that runs on a specific node, i.e. that passes --nodelist=<somenode> to SLURM. I have a couple of variants of this test. Now, if you submit more than two jobs to the same node using --nodelist, you see something like this in the queue:

             JOBID            PARTITION                 NAME     USER    STATE     TIME TIME_LIMI  NODES   PRIORITY           START_TIME                                   NODELIST(REASON)
           8724936         gpu_titanrtx                sleep  <myuser> PENDING     0:00      5:00      1     689130  2022-02-01T18:32:40          (ReqNodeNotAvail, UnavailableNodes:r34n5)
           8724937         gpu_titanrtx                sleep  <myuser> PENDING     0:00      5:00      1     689129  2022-02-01T18:32:40   (ReqNodeNotAvail, May be reserved for other job)
           8724938         gpu_titanrtx                sleep  <myuser> PENDING     0:00      5:00      1     689129  2022-02-01T18:32:40   (ReqNodeNotAvail, May be reserved for other job)
           8724935         gpu_titanrtx                sleep  <myuser> RUNNING     1:22      5:00      1     689100  2022-02-01T18:27:40                                              r34n5

I.e. one job is running, the next one that is eligible gives REASON: ReqNodeNotAvail, UnavailableNodes:r34n5 and any job after that gives REASON: ReqNodeNotAvail, May be reserved for other job.

The issue is that ReFrame will cancel all jobs that get the ReqNodeNotAvail, May be reserved for other job because it ends up in this part of the logic. I.e. I see a test failure with:

 * Reason: job blocked error: [jobid=8724916] job cancelled because it was blocked due to a perhaps non-recoverable reason: ReqNodeNotAvail,  May be reserved for other job

In my opinion, this test should not fail. The test is fully eligible, the node is not down, and if it remains in the queue long enough it will get scheduled once the other jobs that were submitted to that specific node finish.

My suggested fix would be to catch this REASON in a similar way as ReqNodeNotAvail, UnavailableNodes:r34n5 is caught here and check if the node is offline. If not, we can assume the job will be scheduled at some point and it should not be cancelled by the ReFrame framework.

I can probably come up with a PR for this if that's helpful, but I probably won't have time the coming days. The fix should be pretty quick and straightforward though...

[vkarak]

The solution to your problem already exists: Set RFM_IGNORE_REQNODENOTAVAIL=y. This will do exactly what you want and you will not get that error. Also if I remember correctly, the Slurm backend does query the nodes in UnavailableNodes to check if they are indeed down. What we consider as "down" states are the following:

https://github.com/eth-cscs/reframe/blob/535d694480ba653f14ea2a8b8713203671014e84/reframe/core/schedulers/slurm.py#L649-L650

Can you check what does Slurm report as a status of r34n5. You might be getting ReqNodeNotAvail if the Slurm backfill algorithm is running and the system is loaded. Generally, Slurm is not so reliable with the ReqNodeNotAvail message, but there are cases that nodes are indeed not available and you don't want to block forever, that's why the default behavior is not to ignore this.

Check also #1049.

[casparvl]

Also if I remember correctly, the Slurm backend does query the nodes in UnavailableNodes to check if they are indeed down.

It only does so if the reason matches the pattern r'UnavailableNodes:(?P<node_names>\S+)?'. As you can see in my original post the first pending job indeed matches that, since it has reason (ReqNodeNotAvail, UnavailableNodes:r34n5). The second pending job does not match that, since it has reason (ReqNodeNotAvail, May be reserved for other job).Note that this happens if the nodes that are requested are part of a (magnetic) reservation (one that I am eligible to submit too).

My proposal was to add this reason as an additional match, i.e. something like

if reason == 'ReqNodeNotAvail' and reason_details:
    self.log('Job blocked due to ReqNodeNotAvail')
    node_match = re.match(
        r'UnavailableNodes:(?P<node_names>\S+)?',
        reason_details.strip()
    )
    node_match2 = re.match(
        r'May be reserved for other job\S+)?',
        reason_details.strip()
    )
    if node_match or node_match2:
        <check_if_node_is_down...>

The issue that I overlooked is that in the first case, you capture the node name in the capture group, so that you can check if it is down. In the second case, the node name is not mentioned in the reason - hence you cannot check capture the nodename from the reason. The only way to still determine for which node the state would have to be checked is to get the ReqNodeList

casparl@login3:~/reframe-surfsara/production_tests/synthetic/gpu_burn/src$ scontrol show job 8725381 | grep ReqNodeList
   ReqNodeList=r34n5 ExcNodeList=(null)

Does the SLURM scheduler in ReFrame have the capability to do scontrol checks? If so, we could parse that output with a regexp, capture the node name, and check if it is down or not. That would be a bit more fool-proof than just setting RFM_IGNORE_REQNODENOTAVAIL=y, since that will (according to this) simply remove the ReqNodeNotAvail as a reason to cancel the job alltogether (i.e. no check on the node state will be done).

[vkarak]

I agree that this is indeed a bug, but the reason is in the following if condition:

https://github.com/eth-cscs/reframe/blob/b04eae513f49d3dea2ea73c94ae48e491f3856ab/reframe/core/schedulers/slurm.py#L496-L523

The problem is the job is cancelled even in the case that there is no node_match because the control will fall through the if node_match and go to self.cancel(). I think adding an else: return to the following condition will make it work:

https://github.com/eth-cscs/reframe/blob/b04eae513f49d3dea2ea73c94ae48e491f3856ab/reframe/core/schedulers/slurm.py#L502