Support memory qos with cgroups v2

[xiaoxubeii]

Enhancement Description

• One-line enhancement description (can be used as a release note): Support memory qos with cgroups v2
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2570-memory-qos
• Discussion Link: https://docs.google.com/document/d/1r9Fx2omdMLP8SJ59np-C6uBqeuS4TSbQc5ChmQWMYJ8/edit?usp=sharing
• Primary contact (assignee): @xiaoxubeii
• Responsible SIGs: node
• Enhancement target (which target equals to which milestone):
  • Alpha1 release target (x.y): v1.22
  • Alpha2 release target (x.y): v1.27
  • Beta release target (x.y): blocked for https://docs.google.com/document/d/1mY0MTT34P-Eyv5G1t_Pqs4OWyIH-cg9caRKWmqYlSbI/edit?usp=sharing
  • Stable release target (x.y):
• Alpha1
  • KEP (k/enhancements) update PR(s):
    • KEP-2570: Support memory qos with cgroups v2 #2571
  • Code (k/k) update PR(s):
    • Feature: add unified on CRI to support cgroup v2 kubernetes#102578
    • Feature: Support memory qos with cgroups v2 kubernetes#102970
  • Docs (k/website) update PR(s):
    • Support Memory QoS with cgroups v2 for 1.22 #2570 website#28566
• Alpha2
  • KEP (k/enhancements) update PR(s):
    • KEP-2570: Memory QOS Alpha Update and Production Readiness #3818
  • Code (k/k) update PR(s):
    • default memoryThrottlingFactor to 0.9 and optimize the memory.high formulas kubernetes#115371
  • Docs (k/website) update PR(s):
    • Memory QOS Alpha2 KEP 2570 blog post website#39853
    • [zh] Sync blog 2023-05-05-memory-qos-cgroups-v2 website#43183

[xiaoxubeii]

/sig node

[xiaoxubeii]

/assign @xiaoxubeii

[MadhavJivrajani]

Hi! This sounds really interesting and I'd love to help out, please let me know how I can help out with this!

[ehashman]

/stage alpha
/milestone v1.22

[gracenng]

Hi @xiaoxubeii 👋 1.22 Enhancements Shadow here.

This enhancement is in good shape, some minor change requests in light of Enhancement Freeze on Thursday May 13th:

• Update kep.yaml file to the latest template
• In kep.yaml, status is currently provisional instead of implementable
• Alpha graduation criteria missing
• KEP not merged to master

Thanks!

[gracenng]

Hi @xiaoxubeii 👋 1.22 Enhancements shadow here.

To help SIG's be aware of their workload, I just wanted to check to see if SIG-Node will need to do anything for this enhancement and if so, are they OK with it?
Thanks!

[xiaoxubeii]

@gracenng Hey grace, I have updated necessary contents as follows:

• update kep.yaml for prr approval
• add Alpha graduation criteria

sig-node approvers @derekwaynecarr @mrunalp are reviewing for that. I am waiting for lgtm/approve and merge as implementable.

[xiaoxubeii]

@gracenng sig-node approvers(Derek and Mrunal) have gave lgtm/approve. There are few prr review requests, I have updated and am waiting for next review round. I think we can catch up with the freeze day :)

[gracenng]

Hi @xiaoxubeii , looks like your PRR was approved and the requested changes are all here. I have updated the status of this enhancement to tracked
Thank you for keeping me updated!

[xiaoxubeii]

Thanks for your help. Also thanks very much to a lot of valuable review suggestions and helps from @derekwaynecarr @mrunalp @bobbypage @giuseppe @odinuge @johnbelamaric @ehashman
Really appreciate that :)

[ritpanjw]

Hello @xiaoxubeii 👋 , 1.22 Docs Shadow here.

This enhancement is marked as Needs Docs for 1.22 release.
Please follow the steps detailed in the documentation to open a PR against dev-1.22 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Fri July 9, 11:59 PM PDT.
Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thank you!

[xiaoxubeii]

@ritpanjw OK, thanks for reminding.

[gracenng]

Hi @xiaoxubeii 🌞 1.22 enhancements shadow here.

In light of Code Freeze on July 8th, this enhancement current status is tracked, and we're currently tracking kubernetes/kubernetes#102578 kubernetes/kubernetes/pull/102970

Please let me know if there is other code PR associated with this enhancement.

Thanks

[xiaoxubeii]

Hi @xiaoxubeii 🌞 1.22 enhancements shadow here.

In light of Code Freeze on July 8th, this enhancement current status is tracked, and we're currently tracking kubernetes/kubernetes#102578 kubernetes/kubernetes/pull/102970

Please let me know if there is other code PR associated with this enhancement.

Thanks

@gracenng It is all here, thanks.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pacoxu]

/remove-lifecycle stale
/reopen

[k8s-ci-robot]

@pacoxu: Reopened this issue.

In response to this:

/remove-lifecycle stale
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[pacoxu]

@ndixita in https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2570-memory-qos#latest-update-stalled, the kep mentioned that we may use PSI here.

#4205 is PSI kep and we may wait for that KEP implementation, IIUC.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ndixita]

/reopen

[k8s-ci-robot]

@ndixita: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[pacoxu]

/remove-lifecycle rotten

[pacoxu]

Thanks for @ndixita detailed test: https://docs.google.com/document/d/1mY0MTT34P-Eyv5G1t_Pqs4OWyIH-cg9caRKWmqYlSbI/edit?usp=sharing.

🚨🚨 Sometimes the application pod is stuck for throttling memory. This is a worse behavior than OOM kill. So we decided to postpone promoting this feature until we can gracefully handle this issue.

See kubernetes/kubernetes#118699 (comment) as well.

So KEP needs an update.

For the blocker issue, it is a kernel behavior which may need fix or we need something like alibaba cloud memory high watermark which only trigger memory reclaim without throttling.

Some new thoughts on this.

The throttling behavior may not be suitable to all applications. Should we change the default behavior of this to be just disabled? Or I'm afraid this will be something forever alpha. 🤔

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[haircommander]

I chatted with a colleague who works on the memory subsystem about this, and he is looking into adding a knob to the kernel to tune how aggressive the kernel is with reclaiming memory. This could allow us to reduce the aggression which could let the kernel OOM kill in this case

[haircommander]

/remove-lifecycle stale