ReqNodeNotAvail reason code causes ReFrame to incorrectly kill jobs on Cori at NERSC

[bcfriesen]

The Slurm scheduler on the Cori system at NERSC takes a long time to compute backfill - up to 15 minutes per backfill cycle, due to having a huge number of enqueued jobs (several 10,000s). During this process, Slurm will often temporarily change the reason code for an enqueued job to ReqNodeNotAvail, UnavailableNodes:nid[...], where the list of nodes is essentially the entire system, around 11,000 nodes. Although ReFrame attempts to check if the nodes are healthy (cf. #303 and resulting PR), this check seems to fail on Cori while the backfill cycle is running. I don't understand why it fails, but wonder if Slurm refuses to service the scontrol show node command with such a large list of nodes.

I have worked around this problem by simply removing ReqNodeNotAvail from the _cancel_reasons list in reframe/core/schedulers/slurm.py in NERSC's fork of ReFrame, but this is probably not the best long-term solution. If it is true that Slurm is refusing to service the large scontrol command, can you think of a way to work around this problem?

[bcfriesen]

@brandongc points out that the scontrol show node logic may cause problems in the future, e.g., if SchedMD decides to change the meaning of generic reason codes like ReqNodeNotAvail. If this reason code is transient in most cases, as it seems to be on Cori, then would it be sufficient to remove that from the _cancel_reasons list and simply retry the sacct and squeue queries for a while until ultimately timing out?

[vkarak]

Hi @bcfriesen, this is something that has bugged us as well some time ago. Indeed, checking the ReqNodeNotAvail reason code is not the most robust solution (its documentation is also not so clear). Our key goal was to avoid blocking indefinitely on the queue if nodes could not come back, because they are for example reserved, out of service etc. Then we went into the issue #303 that was giving us random failures on one of our systems, which we fixed by explicitly looking in the node list of ReqNodeNotAvail, which again if you do scontrol show node <huge-nodelist> is not ideal since it can take too long. Did scontrol time out or did it give you an error?

A potential solution that I see is implementing #881 and giving the option to ignore ReqNodeNotAvail. Again this is not ideal, because you don't know what a reasonable max queue time is. If it's too low, you may cancel tests that would otherwise run and if it's too long, you will just block even if there is no change to run.

Waiting for sacct or squeue to time out does not solve the problem, because these don't time out due to ReqNodeNotAvail. They will always report the job as pending, so there is no way for ReFrame to differentiate whether this is normal or not. These commands usually time out, when Slurm is overloaded.

[bcfriesen]

Thanks for the context @vkarak, this sounds like a problem with no easy solution. For NERSC at least, I think #881 would be a sufficient workaround, i.e., provide a non-default option to ignore ReqNodeNotAvail and allow the user to specify a maximum wait time. Again, this is because in 99% of cases on Cori, the ReqNodeNotAvail reason is due purely to the long backfill time, and not to nodes actually being down.