slurm is_down check doesn't look like it's handling MAINTENANCE mode properly

[lagerhardt]

At NERSC we (unfortunately) have a non-negligible pool of nodes awaiting hardware servicing (aka "debug nodes"). We put these nodes into a special debug reservation tagged with the MAINTENANCE flag so that our on-site staff can investigate issues while still keeping user jobs away from them. During system maintenances we use reframe to run full system tests in a reservation that contains all the nodes including the debug nodes. We rely on the MAINTENANCE tag to keep jobs off the debug nodes awaiting service. However, we run into a problem because these nodes have a state of IDLE+MAINTENANCE+RESERVED. Currently this isn't working in 4.3.2 because the is_down function was looking for MAINT instead of MAINTENANCE but we can make a local patch for that. I came to make a bug, but wanted to check if this was fixed in more recent code. However, looking at the current head of the code (https://github.com/reframe-hpc/reframe/blob/8e9036fa2f2ccbf56e4c608c5104de390f7bf131/reframe/core/schedulers/slurm.py#L662C5-L670C35)
it looks like our debug nodes would still be included as viable when they aren't because they would match for IDLE because the states are split on the "+" symbol.

def in_statex(self, state):
        return self._states == set(state.upper().split('+'))

    def is_avail(self):
        return any(self.in_statex(s)
                   for s in ('ALLOCATED', 'COMPLETING', 'IDLE'))

    def is_down(self):
        return not self.is_avail()

Am I missing something? Is there a way to exclude nodes tagged with MAINTENANCE when doing runs with num_tasks = 0?

[vkarak]

Up until #3140 state check for flexible tests and the --distribute option was not exclusive, so this would match any node that has IDLE in its states. Now, by default, the check is exclusive, so this would only match nodes strictly in IDLE state. If you want to match IDLE + any other state, you would have to do --distribute=idle* (similarly for the --flex-alloc-nodes). The same is true for any other state. Now, the avail you see above is mostly to support the new pseudo-state named avail that matches any node where a job could possibly run.

[dmargala]

thanks @vkarak !

With the new changes, I'm still not quite sure what value we'd want to pass to --flex-alloc-nodes for the use case @lagerhardt mentioned. We want to include nodes that have a state in ('ALLOCATED', 'COMPLETING', 'IDLE') while also excluding any nodes that have a state like IDLE+MAINTENANCE.

• the new default is too restrictive if we want to include nodes that have any "available" state, anything in ('ALLOCATED', 'COMPLETING', 'IDLE').
• --flex-alloc-nodes=idle* seems too permissive and will match nodes with IDLE+MAINTENANCE which we do not want.
• --flex-alloc-nodes=avail is also too permissive, it would not exclude nodes with IDLE+MAINTENANCE.

[vkarak]

@dmargala Sorry for the late response. We were quite busy with the 4.6 release.

The avail pseudo-state is not so permissive: it will only pick nodes that are exclusively in any of the aforementioned states. So it will exclude a node in IDLE+MAINTENANCE as the node is not exclusively in IDLE state.

Have you had the chance to try out this new feature?

[dmargala]

Thanks @vkarak, we haven't had a chance to test this out yet but hope to do so soon.

Looking at the recent change in the code again, I agree with you that --flex-alloc-nodes=avail should do what we want and is not too permissive (I also disagree with my 3rd bullet point in the comment above now).