[Swap][Tests][KEP2400] Add swap serial stress tests, improve NodeConformance tests and adapt NoSwap behavior

[iholder101]

What type of PR is this?

/kind feature

Fixes #120798

What this PR does / why we need it:

This PR does the following:

1. Havily refactors and cleans swap node-e2e tests.
2. Adapt swap NodeConformance test to [KEP-2400] add no swap as the default option for swap #122745 which drops UnlimitedSwap and introduces NoSwap behavior.
3. Adds serial tests for swap.

To expand on (3), two serial tests are now added which validate the following scenarios:

1. "should be able over-commit the node memory" - a pod is deployed which stresses memory. The test expects that once the pod uses all of the node's memory some of its memory would be swapped away.
2. "should be able to use more memory than memory limits" - a pod with memory limits stresses memory. Eventually it goes beyond its memory limits and instead of being OOM killed some of its memory is swapped away keeping it alive.

Special notes for your reviewer:

The original version of this PR was here: #120430

Due to unfortunate situations I had to be out of work for a while, therefore my PR was closed and continued here: #122175

Fortunately I was able to come back to work and therefore the PR is now continued here.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: iholder101
Once this PR has been reviewed and has the lgtm label, please assign dchen1107 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[iholder101]

/test pull-kubernetes-node-swap-fedora
/test pull-kubernetes-node-swap-fedora-serial
/test pull-kubernetes-node-swap-ubuntu-serial

[iholder101]

PR is not ready to review yet.

/uncc @dchen1107 @andrewsykim

[kannon92]

/cc

when it is ready

[iholder101]

/test pull-kubernetes-node-swap-conformance-fedora-serial
/test pull-kubernetes-node-swap-conformance-ubuntu-serial

[iholder101]

@kannon92 could you please have another look now that kubernetes/test-infra#32276 is merged and the swap conformance label is removed?

[kannon92]

/test pull-kubernetes-node-swap-fedora-serial
/test pull-kubernetes-node-swap-ubuntu-serial

[kannon92]

/test pull-kubernetes-node-swap-conformance-fedora-serial
/test pull-kubernetes-node-swap-conformance-ubuntu-serial

[kannon92]

tests look good now

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 69dd57e69a7ebb36963cc8ba100ca09ad1a10717

[kannon92]

/triage accepted
/priority important-soon

Let's get this into 1.31 as soon as it opens.

[iholder101]

@mrunalp Would you mind taking a look?

[kannon92]

/assign @mrunalp @SergeyKanzhelev @dchen1107

[SergeyKanzhelev]

/retest

[k8s-ci-robot]

@iholder101: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-node-swap-ubuntu-serial	fd26dac	link	false	/test pull-kubernetes-node-swap-ubuntu-serial
pull-kubernetes-node-swap-fedora-serial	fd26dac	link	false	/test pull-kubernetes-node-swap-fedora-serial

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[iholder101]

/retest

Hey @SergeyKanzhelev!
Please note that the regular pull-kubernetes-node-swap-*-serial lanes are flaky.

We've introduced the new pull-kubernetes-node-swap-conformance-*-serial lanes, which are stable and run only swap-stress related tests, for that purpose. These tests pass continuously on this PR.

[pohly]

Is someone working on fixing this flakiness?

Fixing flakes should have more priority than adding more tests or features.

[iholder101]

Is someone working on fixing this flakiness?

Fixing flakes should have more priority than adding more tests or features.

While I agree it's important, I think it's out-of-scope for this PR.

This PR does not affect the tests that these lanes are running, but rather adds new serial stress tests to run under pull-kubernetes-node-swap-conformance-*-serial lanes.

[kannon92]

@pohly We are working on fixing flakyness in #sig-node-ci but bandwidth is limited. I don't think we can block all features in sig-node until we have a green CI.

Fixing flakes is complicated because each test can have a different reason. For example, we have a flake in our eviction tests (for at least two years) and we spent at least a few months working on fixing it. It turns out that there was a limitation in our cadvisor stats code which was reporiting wrong processes. We debated about fixing at the cadvisor level or at the stats level.