Enable graceful termination for UDP flows when using kube-proxy in IPVS mode #71515

lbernail · 2018-11-28T17:26:48Z

What type of PR is this?
/kind bug

What this PR does / why we need it:
The current graceful termination logic is to delete a RS if the number of active connections is 0. This makes sense for TCP but for UDP the number of active connections is always 0. This is an issue for DNS queries because the RS will be deleted but the IPVS connection will remain until it expires (5mn by default) and if there are a lot of DNS queries, the port will be reused and queries blackholed.

Of course for this to work properly the service needs to continue to serve queries until the connections expire (this works fine with the lameduck option of coredns).

Which issue(s) this PR fixes *
Fixes #71514

Special notes for your reviewer:
In addition to this fix, the PR also includes a small fix in the delete function which was never removing entries from the graceTerminateRSList and a small improvement in logging to use the full RS name in logs to identify the service associated with it (helpful for DNS because when a pod is deleted both the TCP and UDP endpoints are removed). I can of course create separate PRs if needed

Does this PR introduce a user-facing change?:

UDP connections now support graceful termination in IPVS mode

/assign @m1093782566

Help distinguish UDP and TCP RS (useful for DNS which uses both)

The current logic is to delete a RS if the number of active connections is 0. This makes sense for TCP but for UDP the number of active connections is always 0. This is an issue for DNS queries because the RS will be deleted but the IPVS connection will remain until it expires (5mn by default) and if there are a lot of DNS queries, the port will be reused and queries blackholed. Of course for this to work properly the service needs to continue to serve queries until the connections expire (this works fine with the lameduck option of coredns).

k8s-ci-robot · 2018-11-28T17:26:57Z

Hi @lbernail. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

m1093782566 · 2018-11-29T01:55:11Z

/ok-to-test

m1093782566 · 2018-11-29T01:59:35Z

pkg/proxy/ipvs/graceful_termination.go

@@ -75,10 +75,10 @@ func (q *graceTerminateRSList) remove(rs *listItem) bool {

 	uniqueRS := rs.String()
 	if _, ok := q.list[uniqueRS]; ok {
-		return false
+		delete(q.list, uniqueRS)


@Lion-Wei Can you check this piece of code?

m1093782566 · 2018-11-29T04:12:42Z

LGTM

/milestone v1.13

/approve

k8s-ci-robot · 2018-11-29T04:12:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lbernail, m1093782566

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/proxy/ipvs/OWNERS~~ [m1093782566]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

m1093782566 · 2018-11-29T04:13:24Z

/retest

m1093782566 · 2018-11-29T04:16:40Z

Adding

/priority critical-urgent

as there are some related outstanding issues.

/lgtm

@Lion-Wei given the priority, I am going to merge this PR. Please feel free to put your comments here.

Lion-Wei · 2018-11-29T06:07:48Z

@lbernail @m1093782566 I'm so sorry for this mistake, thanks for caching this.
All looks good to me.

lbernail · 2018-11-29T10:38:48Z

I don't think we have tests for graceful termination yet. Happy to help work on them

m1093782566 · 2018-11-29T11:09:27Z

We can limit this PR to UDP if you prefer and discuss TCP later?

Sounds like a good plan. We need to fix the UDP graceful termination first as there are some outstanding issues.

I don't think we have tests for graceful termination yet. Happy to help work on them

Great!

m1093782566 · 2018-11-29T11:14:18Z

/hold cancel

m1093782566 · 2018-11-29T11:14:50Z

@lbernail gofmt please.

/lgtm

Will LGTM again when you push the update to fix go fmt.

lbernail · 2018-11-29T12:34:48Z

go fmt fixed
sorry about that

m1093782566 · 2018-11-29T12:38:51Z

/lgtm

AishSundar · 2018-11-29T14:38:12Z

/hold

AishSundar · 2018-11-29T14:40:36Z

@lbernail @m1093782566 was this PR meant for 1.13? we no longer accept merges into 1.13 branch and only extremely critical urgent fixes can to be CP'ed if needed.

nikopen · 2018-11-29T17:58:54Z

@lbernail @m1093782566 this fix looks rather important, but we need to assess the criticality of this fix and how stable it is to merge for 1.13.0 (release is on Monday). Let us know about the details:

Is this a new bug on 1.13-alpha or is it longstanding / should be backported to previous releases?
How well has it been tested / what are the possibilities of release-blocking CI tests failing?
Can it be merged for 1.13.1?

I think this can go in, and be reverted if it introduces any CI problems due to lack of time.

m1093782566 · 2018-11-30T06:10:51Z

I think it can be merged in 1.13.1.

AishSundar · 2018-11-30T06:14:44Z

@tpepper and @aleksandra-malinowska to consider this as a possible 1.13.1 patch release candidate.

/milestone clear

/hold cancel

…5-upstream-release-1.12 Automated cherry pick of #71515 upstream release 1.12

…5-upstream-release-1.13 Automated cherry pick of #71515 upstream release 1.13

…5-upstream-release-1.11 Automated cherry pick of #71515 upstream release 1.11

lbernail added 3 commits November 28, 2018 18:02

[kube-proxy/ipvs] Display RS complete name in logs

7acb5a6

Help distinguish UDP and TCP RS (useful for DNS which uses both)

[kube-proxy/ipvs] Fix logic of delete function

b955634

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Nov 28, 2018

k8s-ci-robot assigned m1093782566 Nov 28, 2018

k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 28, 2018

k8s-ci-robot requested review from Lion-Wei and thockin November 28, 2018 17:27

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 29, 2018

m1093782566 reviewed Nov 29, 2018

View reviewed changes

k8s-ci-robot added this to the v1.13 milestone Nov 29, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 29, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 29, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 29, 2018

[kube-proxy/ipvs] Generalize handling of InactiveConn to TCP

b11233a

lbernail force-pushed the lbernail/udp-graceful branch from 5024875 to b11233a Compare November 29, 2018 12:34

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 29, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 29, 2018

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 29, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 30, 2018

k8s-ci-robot removed this from the v1.13 milestone Nov 30, 2018

k8s-ci-robot merged commit 25c9ac6 into kubernetes:master Nov 30, 2018

m1093782566 mentioned this pull request Nov 30, 2018

In IPVS mode, kube-proxy doesn't handle graceful termination of UDP flows #71514

Closed

Quentin-M mentioned this pull request Dec 7, 2018

kube-proxy ipvs mode hangs after a few hours, needs manual restart #71071

Closed

This was referenced Dec 7, 2018

Automated cherry pick of #71515 upstream release 1.12 #71847

Merged

Automated cherry pick of #71515 upstream release 1.11 #71848

Merged

Automated cherry pick of #71515 upstream release 1.13 #71881

Merged

k8s-ci-robot added a commit that referenced this pull request Dec 10, 2018

Merge pull request #71847 from DataDog/automated-cherry-pick-of-#7151…

51b823e

…5-upstream-release-1.12 Automated cherry pick of #71515 upstream release 1.12

k8s-ci-robot added a commit that referenced this pull request Dec 10, 2018

Merge pull request #71881 from DataDog/automated-cherry-pick-of-#7151…

5bcfca3

…5-upstream-release-1.13 Automated cherry pick of #71515 upstream release 1.13

k8s-ci-robot added a commit that referenced this pull request Dec 10, 2018

Merge pull request #71848 from DataDog/automated-cherry-pick-of-#7151…

f8eaf95

…5-upstream-release-1.11 Automated cherry pick of #71515 upstream release 1.11

lbernail mentioned this pull request May 13, 2019

REQUEST: New membership for lbernail kubernetes/org#811

Closed

6 tasks

Enable graceful termination for UDP flows when using kube-proxy in IPVS mode #71515

Enable graceful termination for UDP flows when using kube-proxy in IPVS mode #71515

lbernail commented Nov 28, 2018

k8s-ci-robot commented Nov 28, 2018

m1093782566 commented Nov 29, 2018

m1093782566 Nov 29, 2018

m1093782566 commented Nov 29, 2018

k8s-ci-robot commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

m1093782566 commented Nov 29, 2018 •

edited

Lion-Wei commented Nov 29, 2018

lbernail commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

m1093782566 commented Nov 29, 2018 •

edited

lbernail commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

AishSundar commented Nov 29, 2018

AishSundar commented Nov 29, 2018

nikopen commented Nov 29, 2018

m1093782566 commented Nov 30, 2018

AishSundar commented Nov 30, 2018

Enable graceful termination for UDP flows when using kube-proxy in IPVS mode #71515

Enable graceful termination for UDP flows when using kube-proxy in IPVS mode #71515

Conversation

lbernail commented Nov 28, 2018

k8s-ci-robot commented Nov 28, 2018

m1093782566 commented Nov 29, 2018

m1093782566 Nov 29, 2018

Choose a reason for hiding this comment

m1093782566 commented Nov 29, 2018

k8s-ci-robot commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

m1093782566 commented Nov 29, 2018 • edited

Lion-Wei commented Nov 29, 2018

lbernail commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

m1093782566 commented Nov 29, 2018 • edited

lbernail commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

AishSundar commented Nov 29, 2018

AishSundar commented Nov 29, 2018

nikopen commented Nov 29, 2018

m1093782566 commented Nov 30, 2018

AishSundar commented Nov 30, 2018

m1093782566 commented Nov 29, 2018 •

edited

m1093782566 commented Nov 29, 2018 •

edited