Add support for shared burstable pod resource limits

[liorokman]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR enhances the Burstable QoS level for pods by allowing deployments to define resource limits which are enforced on the Pod level, rather than only on the Container level.

While the current Burstable QoS level allows creating a resource limit for each containers, it doesn't allow defining a pod with an enforced resource limit that can be shared between multiple containers. Enforcing a limit on the pod level means that developers can decide the overall amount of resources assigned to a pod without having to micro-manage the split of resources between containers in the pod. This allows supporting a more dynamic range of applications.

This PR offers this new functionality as an opt-in feature. In order to use it, the PodSpec must set the ShareBurstableLimits to true. When this is done, the pod level cgroup is set to the sum of all the container limits which were actually defined. Pods can then:

1. Define specific limits for those containers that really need them
2. Limit the overall pod resource limits for those containers that don't define explicit limits.

Which issue(s) this PR fixes:

Fixes #84340 , #87365, #60834

Special notes for your reviewer:

Does this PR introduce a user-facing change?:
Yes.

Allow Pod in the `Burstable` QoS class to define resource limits on the pod level, which are shared by multiple containers in the pod.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[KEP]: https://github.com/liorokman/enhancements/blob/f33d45643fe50649852e6f15ce1c1f44fa250c76/keps/sig-node/20200303-shared-pod-limits.md

[k8s-ci-robot]

Hi @liorokman. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liorokman]

This PR replaces #88722. (I made the mistake of merging instead of rebasing)

[fejta-bot]

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

[dashpole]

You should open the KEP as a PR to kubernetes/enhancements first so we can discuss. I would also highly recommend putting this on the agenda for the weekly sig-node meeting.

/hold

[liorokman]

You should open the KEP as a PR to kubernetes/enhancements first so we can discuss.

Here's the PR: kubernetes/enhancements#1592

I would also highly recommend putting this on the agenda for the weekly sig-node meeting.

How do I do this?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[liorokman]

/remove-lifecycle stale

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: liorokman
To complete the pull request process, please assign erictune, vishh
You can assign the PR to them by writing /assign @erictune @vishh in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• api/OWNERS
• pkg/api/OWNERS
• pkg/apis/OWNERS
• pkg/kubelet/cm/OWNERS
• staging/src/k8s.io/api/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[JiaminZhu]

This is a very useful feature, looking forward to see this feature get released!

[k8s-ci-robot]

@liorokman: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.