SCHED-380: Skip maintenance handling based on node labels

[ali-sattari]

Problem

Soperator's handling of maintenance events is geared toward worker nodes and works by deleting the node to be recreated. For non-replicated stateful applications such as NFS we want to stop/start the node instead. Managed K8S control plane already handles maintenance the way we need for these applications.

Solution

Get a list of labels as input and ignore matching nodes when there is a maintenance condition.

Testing

• Deploy a cluster with a separate node group for NFS in K8S
• Config Soperator to pass labels for those nodes (e.g. slurm.nebius.ai/nodeset=nfs
• Listen to logs of soperator-check-checks deployment
• Manually set condition on node using kubectl proxy and curl (example below)
• You should see skipping draincondition processing due to ignored labels in logs
• Soperator should not take any action on the node matching above labels

Manually setting node condition:

curl -X PATCH http://127.0.0.1:8001/api/v1/nodes/computeinstance-xxx/status \
 -H "Content-Type: application/merge-patch+json" \
 --data '{
  "status": {
   "conditions": [
    {
     "type": "MaintenanceScheduled",
     "status": "True",
     "message": "Maintenance scheduled for node",
     "lastTransitionTime": "'"$(date -u +"%Y-%m-%dT%H:%M:%SZ")"'"
    }
   ]
  }
}'

Log line example:

2025-11-14T12:29:01.388403986Z manager {"level":"info","ts":"2025-11-14T12:29:01Z","logger":"K8SNodesController.processDrainCondition","msg":"skipping draincondition processing due to ignored labels","controller":"soperatorchecks.k8snodes","controllerGroup":"","controllerKind":"Node","Node":{"name":"computeinstance-xxx"},"namespace":"","name":"computeinstance-xxx","reconcileID":"yyyy","node":"computeinstance-xxx","nodeLabels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"cpu-d3","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"eu-north1","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"computeinstance-xxx","kubernetes.io/os":"linux","nebius.com/node-group-id":"mk8snodegroup-zzz","nebius.com/resource-preset":"32vcpu-128gb","node.kubernetes.io/instance-type":"cpu-d3","slurm.nebius.ai/nodeset":"nfs","slurm.nebius.ai/workload":"cpu","topology.kubernetes.io/region":"eu-north1"},"ignoredLabels":{"slurm.nebius.ai/nodeset":"nfs"}}

Release Notes

Skip maintenance handling based on node labels. Useful to let other controllers (such as mk8s in Nebius) handle maintenance events.