Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kubeproxy/ipvs] New sysctls to improve pod termination #71834

Merged
merged 1 commit into from
Dec 10, 2018

Conversation

lbernail
Copy link
Contributor

@lbernail lbernail commented Dec 7, 2018

What type of PR is this?
/kind bug

What this PR does / why we need it:
This PR enables two IPVS sysctls:

  • net/ipv4/vs/expire_nodest_conn: delete connections associated to a real server that has been deleted. This is not important with graceful termination (because real servers are removed when connections have terminated/expired) but very important without it, see IPVS default sysctls and configuration #71358 (so this sysctl should be backported to 1.12 and 1.11)
  • net/ipv4/vs/expire_quiescent_template: expire persistent connections to a real server when its weight has been set to 0 (otherwise future connections from a client with the same IP will be sent to an endpoint that is no longer available. In addition, if a client keeps trying to send traffic, the real server will not be removed until persistency expires which by default takes 3h).

Which issue(s) this PR fixes:
Fixes: #71809
Partially addresses: #71358

Special notes for your reviewer:
expire_quiescent_template defaults to 0 because when using ClientIP affinity it makes sense to continue sending traffic to the same backend even its weight is set to 0. However in Kubernetes pod shutdown is (usually) pretty fast and new connections will be blackholed. In addition, if the client retries to connect often, the number of connections will never reach 0 for this backend and the real server will not be removed until the persistency timer expires.

Does this PR introduce a user-facing change?:

kube-proxy in IPVS mode will stop initiating connections to terminating pods for services with sessionAffinity set.

/sig network
/area ipvs
/assign @m1093782566

@k8s-ci-robot k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Dec 7, 2018
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Dec 7, 2018
@k8s-ci-robot
Copy link
Contributor

Hi @lbernail. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. sig/network Categorizes an issue or PR as relevant to SIG Network. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 7, 2018
@m1093782566
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 7, 2018
@m1093782566
Copy link
Contributor

A general question:

Does "net/ipv4/vs/expire_nodest_conn" and "net/ipv4/vs/expire_quiescent_template" exist in all versions of kernel?

@lbernail
Copy link
Contributor Author

lbernail commented Dec 7, 2018

Does "net/ipv4/vs/expire_nodest_conn" and "net/ipv4/vs/expire_quiescent_template" exist in all versions of kernel?

I just checked, and they are available in 2.6 kernels. What is the oldest kernel version we want to support?

@m1093782566
Copy link
Contributor

Thanks! I don't think we should support kernel < 2.6.

/lgtm

/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 10, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lbernail, m1093782566

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 10, 2018
@m1093782566
Copy link
Contributor

/retest

@k8s-ci-robot k8s-ci-robot merged commit a69b565 into kubernetes:master Dec 10, 2018
@m1093782566
Copy link
Contributor

💯

k8s-ci-robot added a commit that referenced this pull request Dec 19, 2018
…4-upstream-release-1.12

Automated cherry pick of #71834 / #71114 upstream release 1.12
k8s-ci-robot added a commit that referenced this pull request Dec 27, 2018
…4-upstream-release-1.11

Automated cherry pick of #71834 / #71114 upstream release 1.11
k8s-ci-robot added a commit that referenced this pull request Jan 5, 2019
…4-upstream-release-1.13

Automated cherry pick of #71834 upstream release 1.13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ipvs cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/network Categorizes an issue or PR as relevant to SIG Network. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

When using IPVS with graceful termination, connections to services with sessionAffinity are not expired
3 participants