Resource Quotas: Possibility to not enforce limits.cpu per pod

[scenox]

What would you like to be added?

Currently, for cpu (and memory) resources, ResourceQuotas enforce that every (new) pod in that namespace sets a limit for that resource.

There should be a new option for the limits.cpu ResourceQuota at namespace level:
enforceLimitPerPod: true (default) / false

If false, it's optional to define limits.cpu for each pod in this namespace. limits.cpu of the namespace minus limits.cpu of each pod (where defined) should be used to limit CPU usage across all other pods in this namespace.

Why is this needed?

According to https://kubernetes.io/docs/concepts/policy/resource-quotas/,
you can limit the total sum of compute resources that can be requested in a given namespace. limits.cpu for the namespace ist computed as follows: Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.

This is a problem with workloads which have a low base CPU usage with short cpu spikes randomly distributed across pods and time. A lot of assigned CPU is wasted in such case.
For such workload scenarios, it would make more sense to limit only the sum of CPU usage of all pods in the namespace instead of limiting each pod separately.

Example:
limits.cpu namespace: 8
16 containers with limits.cpu 500m

Each of these containers are limited to 500m CPU which could affect application performance negatively. If specification of limits.cpu is not enforced in this namespace, each pod could use up to 7.985 CPU If each other pod in the observed timeslot currently has a CPU usage of 1m.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pacoxu]

Did you try LimitRange to add some default cpu limit?
https://kubernetes.io/docs/concepts/policy/limit-range/

[scenox]

I considered LimitRange but I don't think it solves the issue.

Some context:
Cluster operators often use CPU and memory limits per namespace to put a price tag on providing their infrastructure. They don't want to care about the number of pods, the CPU requirements of each pod in the namespace or any other application-specific requirements on this level - they only care about the CPU usage of the namespace and want to limit that.
In the case above with LimitRange, the cluster operator would have to define a max limit of 8 CPU per pod and theoretically, each pod could then use 8 CPU which isn't what the cluster operator wants. He wants to limit the CPU usage of the set of all pods in the namespace to 8, which should be possible via cgroup.

[liggitt]

He wants to limit the CPU usage of the set of all pods in the namespace to 8, which should be possible via cgroup.

cgroups are local to the node running the pod, but pods in a namespace are not all on the same node. That means real time CPU usage would have to be reported back into the API, observed by other nodes, and cgroup rules updated in response, at high frequency, for this to work. You need some indicator on each pod how the CPU limits should be set to inform the node how to set its cgroups.

[scenox]

Why not distributing available CPU to the set of non-limited pods on each node relative to the number of non-limited pods?
E.g.
Namespace cpu limit: 8
Pods in namespace: 16 (all without cpu limit)
Node A: 12 pods -> cgroup set to 6 CPU for the whole set of 12 pods
Node B: 4 pods: -> cgroup set to 2 CPU for the whole set of 4 pods

[scenox]

/sig node

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.