Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kubeproxy/ipvs] New sysctls to improve pod termination #71834

Merged
merged 1 commit into from Dec 10, 2018

Conversation

@lbernail
Copy link
Contributor

commented Dec 7, 2018

What type of PR is this?
/kind bug

What this PR does / why we need it:
This PR enables two IPVS sysctls:

  • net/ipv4/vs/expire_nodest_conn: delete connections associated to a real server that has been deleted. This is not important with graceful termination (because real servers are removed when connections have terminated/expired) but very important without it, see #71358 (so this sysctl should be backported to 1.12 and 1.11)
  • net/ipv4/vs/expire_quiescent_template: expire persistent connections to a real server when its weight has been set to 0 (otherwise future connections from a client with the same IP will be sent to an endpoint that is no longer available. In addition, if a client keeps trying to send traffic, the real server will not be removed until persistency expires which by default takes 3h).

Which issue(s) this PR fixes:
Fixes: #71809
Partially addresses: #71358

Special notes for your reviewer:
expire_quiescent_template defaults to 0 because when using ClientIP affinity it makes sense to continue sending traffic to the same backend even its weight is set to 0. However in Kubernetes pod shutdown is (usually) pretty fast and new connections will be blackholed. In addition, if the client retries to connect often, the number of connections will never reach 0 for this backend and the real server will not be removed until the persistency timer expires.

Does this PR introduce a user-facing change?:

kube-proxy in IPVS mode will stop initiating connections to terminating pods for services with sessionAffinity set.

/sig network
/area ipvs
/assign @m1093782566

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2018

Hi @lbernail. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@m1093782566

This comment has been minimized.

Copy link
Member

commented Dec 7, 2018

/ok-to-test

@m1093782566

This comment has been minimized.

Copy link
Member

commented Dec 7, 2018

A general question:

Does "net/ipv4/vs/expire_nodest_conn" and "net/ipv4/vs/expire_quiescent_template" exist in all versions of kernel?

@lbernail

This comment has been minimized.

Copy link
Contributor Author

commented Dec 7, 2018

Does "net/ipv4/vs/expire_nodest_conn" and "net/ipv4/vs/expire_quiescent_template" exist in all versions of kernel?

I just checked, and they are available in 2.6 kernels. What is the oldest kernel version we want to support?

@m1093782566

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

Thanks! I don't think we should support kernel < 2.6.

/lgtm

/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Dec 10, 2018

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Dec 10, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lbernail, m1093782566

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@m1093782566

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

/retest

@k8s-ci-robot k8s-ci-robot merged commit a69b565 into kubernetes:master Dec 10, 2018

18 checks passed

cla/linuxfoundation lbernail authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@m1093782566

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

💯

k8s-ci-robot added a commit that referenced this pull request Dec 19, 2018

Merge pull request #72101 from DataDog/automated-cherry-pick-of-#7183…
…4-upstream-release-1.12

Automated cherry pick of #71834 / #71114 upstream release 1.12

k8s-ci-robot added a commit that referenced this pull request Dec 27, 2018

Merge pull request #72100 from DataDog/automated-cherry-pick-of-#7183…
…4-upstream-release-1.11

Automated cherry pick of #71834 / #71114 upstream release 1.11

k8s-ci-robot added a commit that referenced this pull request Jan 5, 2019

Merge pull request #72099 from DataDog/automated-cherry-pick-of-#7183…
…4-upstream-release-1.13

Automated cherry pick of #71834 upstream release 1.13

@lbernail lbernail referenced this pull request May 13, 2019

Closed

REQUEST: New membership for lbernail #811

6 of 6 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.