Feature Request: Quota Bypass for Rolling Updates in Machine Deployments

[ronissac88]

What happened?

When implementing Project Quotas, we're running into a problem where rolling restarts of machine deployments fail due to quota validation. The issue occurs because during a rolling update, the machine controller temporarily creates new machines before deleting old ones, causing a brief period where the total resource usage exceeds the quota limits.

Error Message:

failed to sync Machineset replicas: admission webhook "machines.cluster.k8c.io" denied the request: requested CPU "6" would exceed current quota (quota/used "20"/"18")

Root Cause:

• Machine controller uses a rolling update strategy with maxSurge to maintain availability
• New machines are created before old machines are deleted
• Quota validation happens at admission webhook level before the old machines are removed
• This creates a temporary overage that fails quota validation

Expected behavior

The machine controller should be able to perform rolling updates without failing quota validation when:

1. The replica count of the MachineDeployment is not being increased (i.e., it's a rolling restart/update, not a scale-up)
2. The temporary overage is within the maxSurge limits of the rolling update strategy
3. The total resource usage will return to normal levels once the rolling update completes

Proposed Solution:

• Add a quota bypass mechanism for machines created during rolling updates
• Use an annotation kubermatic.io/bypass-quota-validation=true on machines created during rolling updates
• Modify the admission webhook to detect quota-related errors and bypass validation when the annotation is present
• Ensure the bypass only applies to quota-related errors, not other validation failures

How to reproduce the issue?

1. Setup a MachineDeployment with quota limits:

   apiVersion: "cluster.k8s.io/v1alpha1"
   kind: MachineDeployment
   metadata:
     name: test-deployment
   spec:
     replicas: 3
     strategy:
       type: RollingUpdate
       rollingUpdate:
         maxSurge: 1
         maxUnavailable: 1
     template:
       spec:
         providerSpec:
           value:
             cloudProvider: "azure"
             cloudProviderSpec:
               vmSize: "Standard_D2s_v3"  # 2 vCPUs per machine

2. Set up resource quotas:

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: compute-quota
   spec:
     hard:
       requests.cpu: "6"  # Total of 3 machines * 2 CPUs = 6 CPUs

3. Trigger a rolling update by changing the machine template (e.g., update the OS image or kubelet version)

4. Observe the failure when the machine controller tries to create the 4th machine (3 existing + 1 new for rolling update)

Additional details

Current Flow:

MachineDeployment Update → MachineSet Scaling → Machine Creation → Quota Check → FAIL

Proposed Flow:

MachineDeployment Update → MachineSet Scaling → Machine Creation (with bypass annotation) → Quota Check → BYPASS → SUCCESS

[csengerszabo]

Internal reference: 8382

[kron4eg]

Should we add a new webhook to remove the kubermatic.io/bypass-quota-validation=true annotation? 😂 I'm pretty sure running at 100% of the quota will make troubles at any system. My "solution" would be to increase the quota, or decrease the replicas -1 before the rollout.

[steled]

@kron4eg I think a manual approach maybe works for 3 projects but not for >10.
To manually update temporarily the project quota for over 10 projects doesn't feel right.
Also decreasing the replicas -1 does not feel very well.

It would be great to have a programmatic solution that new nodes during a md rolling restart are not counted into the project quota.

[kron4eg]

@steled I meant that userclusters / projects in general should not run at the 100% quota. Or if they need more, then the quota is not correctly set.

[steled]

For some customers, we do not sell on demand, they just buy a specific node count.
And I think it should be possible to run on 100% quota but still be able to do a rolling restart of machine deployment wihtout temporarily updating the project quota.

I'm also running into a problem that I can't restart machine deployment even when the project quota has a buffer of 1 additional node.
I have a machine deployment with 3 nodes (2 CPUs, 2048 MB RAM) and a project quota of 8 CPUs, 9 GB RAM.