Some Slurm configuration changes lead to invalid node state

[priteau]

Take a configuration such as:

openhpc_nodegroups:
  - name: cpu
    node_params:
      CPUSpecList: 92-95

Initial deployment will work fine. However, if CPUSpecList is changed (e.g. to 90-95), the deployment of the new configuration will lead affected nodes to invalid state with Reason=CoreSpec differ. This happens as soon as slurmctld is restarted, probably due to a mismatch between slurmctld and slurmd.

This can be fixed by forcing another configuration update elsewhere in the Slurm configuration.

There must be some safe way to roll out this change? Maybe stop slurmd services first, then restart slurmctld and finally start all slurmd services?

[sjpb]

I have seen this, but normally just as a warning in the logs that there's a difference. This is because on a config change, the role a) restarts slurmctld, then b) restarts slurmd. So between the restarts, the slurmd is running with a cached config, which doesn't match. So a bit of a limitation of configless in a way.

We could try to scontrol reconfigure for the cases (called out in man slurm.conf) where that's possible - it used not to be for lots of changes, but that restriction has gradually been removed. However for cases where a restart is necessary, I'm not clear if doing a slurmd stop / slurmctld restart / slurmd start is actually a better result!