Node memory swap support

[ehashman]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubernetes nodes support swap memory.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node-swap
• Discussion Link: https://docs.google.com/document/d/1CZtRtC8W8FwW_VWQLKP9DW_2H-hcLBcH9KBbukie67M/edit# Kubelet/Kubernetes should work with Swap Enabled kubernetes#53533
• Primary contact (assignee): @iholder101 @pacoxu
• Responsible SIGs: Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.22
  • Beta release target (x.y): 1.28
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s): Node system swap support #2602
  • Code (k/k) update PR(s):
    • CI/test infra changes: Setup new node image with swap enabled test-infra#22602 Add swap periodics for node test-infra#22809
    • CRI and kubelet changes Alpha node swap support kubernetes#102823
  • Docs (k/website) update PR(s): KEP-2400: Alpha node support for swap memory website#28838
• Beta1
  • KEP (k/enhancements) update PR(s):
    • KEP-2400: Update swap KEP for 1.23 beta #2858
    • KEP-2400: Graduate swap to Beta 1 #3957
  • Issues and Code (k/k) update PR(s):
    • Support cgroups v2 with auto-calculated LimitedSwap for Burstable QoS pods
      • [KEP-2400] Enable Swap Support for Cgroup v2 Only kubernetes#119427
      • [KEP-2400] Enable auto-calculated LimitedSwap for Burstable QoS pods kubernetes#119428
      • Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes#118764
    • Add Swap stats to /stats/summary and /metrics/resource
      • [KEP-2400] Add swap memory to the Kubelet stats API kubernetes#119425
      • Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes#118865
    • Determine a set of metrics for measuring node performance with swap enabled
      • [KEP-2400] Determine a set of metrics for measuring node performance with swap enabled kubernetes#119424
      • Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes#118865
      • Add a metric for machine swap capacity google/cadvisor#3339
    • Improve test coverage and make sure tests are green
      • [KEP-2400] Swap: improve test coverage and make sure tests are green kubernetes#119430
      • Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes#118764
  • Docs (k/website) update(s):
    • Swap graduates to Beta1: add a blog-post website#42009
    • Swap graduates to Beta1: add documentation and update feature gates website#42167
  • Misc
    • Bugfix: MachineInfo Clone() to also clone "SwapCapacity" google/cadvisor#3293
    • Bump cadvisor version to v0.47.3 kubernetes#119225
  • Archive / Outdated
• Beta2
• Future (related PRs, can be assigned to a specific track later)
  • [KEP-2400] [Swap] Add the ability to set a system-reserved quantity of swap on a node kubernetes#120802
  • [KEP-2400] [Swap] [Discussion]: Add swap as a ResourceName for pod spec resources API kubernetes#120801
  • [KEP-2400] [Swap]: Verify memory pressure behavior with swap enabled kubernetes#120800
  • [KEP-2400] [Swap]: SwapConformance e2e testing kubernetes#120798
  • [KEP-2400] Address swap impact on memory-backed volumes kubernetes#105978
  • Add the ability to set a system-reserved quantity of swap on a node
    • [KEP-2400] Add the ability to set a system-reserved quantity of swap on a node kubernetes#105019
    • add --system-reserved support for swap(cgroup v2 only): only for burstable pods kubernetes#105271
  • Fail Kubelet at startup if swap is configured with cgroup v1 kubernetes#119327

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig node

[karan]

@cookieisaac

[ehashman]

/stage alpha
/milestone v1.22

[jrsapi]

Greetings @ehashman!
Enhancement shadow checking in with a few reminders. 1.22 Enhancements Freeze starts at 23:59:59 pst on Thursday, May 13. A few items needing review for this KEP:

• Enhancement README

• PRR approved

• Node memory swap support #2400 updated with current target and alpha information.

Thanks!

[ehashman]

All of this is covered in #2602

[jrsapi]

Greetings @ehashman!
Thanks for the follow-up. After reviewing the KEP and PRR everything looks on target. The enhancement is marked at risk but once merged we can move its status to tracked. A reminder that tomorrow 5/13 the Enhancement freeze starts at 23:59:59 PST.

Thanks!

[ehashman]

Work breakdown for 1.22

• Documentation for enabling swap (TBD - @ehashman?)
• CI environment/test updates (@ike-ma)
  • Build images with swap for 2 Linux distros
  • Add jobs to test-infra that use the images and enable the swap feature flag/kubelet option for node e2e suite
  • See also Test Plan
• CRI and kubelet changes (@ehashman)
  • Complete API changes per KEP
  • Add e2e tests as appropriate

External to k8s but still need to happen:

• Containerd update to use new CRI
• CRI-O update to use new CRI

Once the above CRI updates happen, ensure that CI environment is using latest container runtimes with updated CRI.

[jrsapi]

Greetings @ehashman ,
Enhancement shadow checking with a reminder that we are 2 weeks away from code freeze (July 8, 2021). Can you like the k/k PR/s that are needed to implement this enhancement for the 1.22 milestone?

[jrsapi]

Greetings @ehashman,
A friendly reminder that code freeze is this Thursday, July 8th and we're tracking the following k/k PR:

• Alpha node swap support kubernetes#102823
  The PR has to be merged by code freeze and currently, this KEP is marked "At Risk".
  Also, Docs PR placeholder deadline is Friday, July 9th. Here are instructions on opening a PR in dev-1.22 branch.

Thanks!

[ehashman]

PR just merged. Docs placeholder is kubernetes/website#28838

[jrsapi]

Awesome! Thanks for the update. Moving this to "Tracked".

[ehashman]

We are good to go for 1.22! Docs complete.

[ehashman]

/milestone v1.23

[ehashman]

/stage beta

[npolshakova]

/label lead-opted-in

[jan-kantert]

Hey @jan-kantert! Thanks for reaching out.
You're right. This is indeed something we need to look into. I guess (?) that all needs to be done is to subtract the swap space from workingSet and simply ignore swap space for eviction decisions. But as said I still need to explore this area.

Would there be a scenario where swap + workingSet > node.status.capacity[memory] ? and would that lead to having negative value for memory.available?

I my understanding this is currently the case. However, in practice that is where eviction kicks in so that value stays around the eviction threshold. If you look at our nodes you will see that they are in fact only using around 50% of their available RAM (and bunch of swap) but memory.available indicates that they are almost our of memory (which they are not). On a plus side nodes will use the spare RAM as cache which is better spend than keeping unused stuff in RAM.

We are currently investigating with --experimental-allocatable-ignore-eviction to disable memory evictions. I will report back once we gained some experience with that.

[iholder101]

Hey all.
I've made some cleaning w.r.t. old issues that are outdated, inactive and that I can't update their descriptions.

The issues can be tracked here:

• [KEP-2400] [Swap] Add the ability to set a system-reserved quantity of swap on a node kubernetes#120802
• [KEP-2400] [Swap] [Discussion]: Add swap as a ResourceName for pod spec resources API kubernetes#120801
• [KEP-2400] [Swap]: Verify memory pressure behavior with swap enabled kubernetes#120800
• [KEP-2400] [Swap]: SwapConformance e2e testing kubernetes#120798
• [KEP-2400] Address swap impact on memory-backed volumes kubernetes#105978

From now on I plan to keep them organized and up-to-date.
@pacoxu you're welcome to add these under "future" in this issue's description (since unfortunately I can't edit it as well).

[rayandas]

Hello @ehashman 👋, v1.29 Enhancements team here.

Just checking in as we approach enhancements freeze on 01:00 UTC, Friday, 6th October, 2023.

This enhancement is targeting for stage beta for v1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

• Update the latest-milestone in kep.yaml

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[pacoxu]

@iholder101 do you think this feature needs to be tracked by the release team in v1.29 release cycle?

• If yes, do you have time to update the KEP status?
• If no, we can just push forward swap related tasks and update the KEP in next release cycle.

[deads2k]

Is this planning to move to stable in 1.29?

[SergeyKanzhelev]

Is this planning to move to stable in 1.29?

no. We have critical issues with evction with the current implementation and we haven't even enabled it on by default. We will try to fix the eviction problem in 1.29 and see if there is a demand for more APIs.

[rayandas]

Hi @ehashman checking in once more as we approach the 1.29 enhancement freeze deadline on 01:00 UTC, Friday, 6th October, 2023. The status of this enhancement is marked as at risk for enhancement freeze.
It looks like updating the latest milestone in kep.yaml will help this enhancement move to tracked for enhancement freeze. Let me know if I missed anything. Thanks.

[harche]

Hi @ehashman checking in once more as we approach the 1.29 enhancement freeze deadline on 01:00 UTC, Friday, 6th October, 2023. The status of this enhancement is marked as at risk for enhancement freeze. It looks like updating the latest milestone in kep.yaml will help this enhancement move to tracked for enhancement freeze. Let me know if I missed anything. Thanks.

I created #4275 to bump swap to beta2 in 1.29

[rayandas]

As the PR #4275 is merged and latest-milestone has been updated, marking this as Tracked for Enhancements Freeze. 🚀

[drewhagen]

Hello @iholder101 @pacoxu @ehashman @SergeyKanzhelev 👋, v1.29 Docs Shadow here.
Does this enhancement work planned for v1.29 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023.
Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[harche]

Hello @iholder101 @pacoxu @ehashman @SergeyKanzhelev 👋, v1.29 Docs Shadow here. Does this enhancement work planned for v1.29 require any new docs or modification to existing docs? If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023. Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release. Thank you!

Ack. Thanks.

[drewhagen]

Hi @harche @iholder101 @pacoxu @ehashman @SergeyKanzhelev! The deadline to open a placeholder PR against k/website for required documentation is Thursday, 19 October. Could you please update me on the status of docs for this enhancement? Thank you!

[harche]

Hi @harche @iholder101 @pacoxu @ehashman @SergeyKanzhelev! The deadline to open a placeholder PR against k/website for required documentation is Thursday, 19 October. Could you please update me on the status of docs for this enhancement? Thank you!

@drewhagen I have created a placehold PR here - kubernetes/website#43571

[jan-kantert]

Hey @jan-kantert! Thanks for reaching out.
You're right. This is indeed something we need to look into. I guess (?) that all needs to be done is to subtract the swap space from workingSet and simply ignore swap space for eviction decisions. But as said I still need to explore this area.

Would there be a scenario where swap + workingSet > node.status.capacity[memory] ? and would that lead to having negative value for memory.available?

I my understanding this is currently the case. However, in practice that is where eviction kicks in so that value stays around the eviction threshold. If you look at our nodes you will see that they are in fact only using around 50% of their available RAM (and bunch of swap) but memory.available indicates that they are almost our of memory (which they are not). On a plus side nodes will use the spare RAM as cache which is better spend than keeping unused stuff in RAM.

We are currently investigating with --experimental-allocatable-ignore-eviction to disable memory evictions. I will report back once we gained some experience with that.

We ran this setting in production for a while and it seems to perform well. We do not use swap as workload memory currently (i.e. pods reserve actual RAM for their workload instead of a mixture of swap and memory). However, this helps a lot with preventing evictions. Our clusters are almost free of evictions which is really great. It also allows us to use vertical pod autoscaler (VPA) for memory in more cases as pods might temporarily use more memory without fear of instant eviction (which we had previously on tightly packed clusters). So overall this seems to allow us to pack even tighter and get our average memory usage from 20-30% to 50-60% which is a roughly 50% cost reduction.

As a downside we see a lot more instances of random probe failures (similar to kubernetes/kubernetes#89898). This seems to be caused by better packing and higher CPU load. We generally see high load (i.e. 50+) but not necessarily cpu usage. So this might be connected with swap in/out of memory. Guess since kubelet has a general issues around probe timeout/slowness this is something which is kind of expected when you pack tighter. As a workaround we use a mutating webhook to patch all livenessProbe timeouts to at least 30s since a lot of software (i.e. linkerd) uses the 1s default timeout. Without the workaround we often see pods (such as linkerd with a 1s probe) getting terminated by kubelet. No issues with the workaround so far.

[rayandas]

Hey again @ehashman @pacoxu @SergeyKanzhelev 👋, 1.29 Enhancements team here,

Just checking in as we approach code freeze at 01:00 UTC Wednesday 1st November 2023: .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).

• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

The status of this KEP is currently at risk for code freeze.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.

As always, we are here to help if any questions come up. ✌️ Thanks!

[kcmartin]

Hi @ehashman @pacoxu @SergeyKanzhelev ! 👋 from the v1.29 Release Team-Communications! We would like to check if you have any plans to publish a blog for this KEP regarding new features, removals, and deprecations for this release.

If so, you need to open a PR placeholder in the website repository.
The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review)

Here's the 1.29 Calendar

[harche]

Hi @ehashman @pacoxu @SergeyKanzhelev ! 👋 from the v1.29 Release Team-Communications! We would like to check if you have any plans to publish a blog for this KEP regarding new features, removals, and deprecations for this release.

If so, you need to open a PR placeholder in the website repository. The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review)

Here's the 1.29 Calendar

I am afraid, due to pending issues we won't be able to graduate swap to beta2 in 1.29.

[npolshakova]

Hello @ehashman @pacoxu @SergeyKanzhelev 👋 1.29 Enhancements lead here,

Unfortunately, the implementation (code related) PR(s) associated with this enhancement is not in the merge-ready state by code-freeze and hence this enhancement is now removed from the 1.29 milestone.

If you still wish to progress this enhancement in 1.29, please file an exception request. Thanks!

/milestone clear