Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ipvs: fix string check for IPVS protocol during graceful termination #78999

Merged
merged 1 commit into from Jun 15, 2019

Conversation

@andrewsykim
Copy link
Member

commented Jun 13, 2019

Signed-off-by: Andrew Sy Kim kiman@vmware.com

What type of PR is this?
/kind bug

What this PR does / why we need it:
In #77802 we removed graceful termination for UDP (causes many issues when there are too many connections/real servers). It turns out that in pkg/util/ipvs we actually convert the protocol values from netlink to upper case strings ("TCP", "UDP", etc, see protocolToString) even though the string passed into docker/libnetwork/ipvs is lower case string (see stringToProtocol).

This PR fixes this bug by always checking upper case values. As a follow-up we should consolidate the string casing for utils/ipvs to ensure this doesn't happen again. The PR for this will be larger so wanted to get the bug fix only change in before we cut v1.15 and for easier cherry-picking for previous versions.

I'm missing context as to why we use lower case values for stringToProtocol and upper case values for protocolToString, would appreciate some background there @m1093782566 @lbernail.

Which issue(s) this PR fixes:
Fixes ##78993

DNS service in the reported issue went down due to too many stale real server entries being held by graceful termination.

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fix a string comparison bug in IPVS graceful termination where UDP real servers are not deleted.
@andrewsykim

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

@andrewsykim

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

/priority critical-urgent
/milestone v1.15

@k8s-ci-robot k8s-ci-robot added this to the v1.15 milestone Jun 13, 2019

@k8s-ci-robot k8s-ci-robot requested review from cmluciano and dnardo Jun 13, 2019

@andrewsykim andrewsykim changed the title ipvs: fix string check for IPVS protocol ipvs: fix string check for IPVS protocol graceful termination Jun 13, 2019

@andrewsykim andrewsykim changed the title ipvs: fix string check for IPVS protocol graceful termination ipvs: fix string check for IPVS protocol during graceful termination Jun 13, 2019

ipvs: fix string check for IPVS protocol during graceful termination
Signed-off-by: Andrew Sy Kim <kiman@vmware.com>

@andrewsykim andrewsykim force-pushed the andrewsykim:ipvs-graceful-term-fix branch from 388308c to bb95143 Jun 13, 2019

@andrewsykim

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

More proper fix for a follow-up PR would be to change the string values returned in protocolToString to lower case values to be consistent with values passed into libnetwork/ipvs. Didn't do that in this PR since I'm not sure what the implications are yet.

@andrewsykim

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

Also created #79001 re: e2e coverage for IPVS graceful termination

@andrewsykim

This comment has been minimized.

Copy link
Member Author

commented Jun 14, 2019

/retest

@lbernail

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2019

Nice catch
I tested with "!=tcp" before we decided to disable it only for UDP. So this worked (but would have broken TCP graceful termination...)

I agree that we should standardize on lower or upper everywhere to avoid this type of issue

@lbernail

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Jun 14, 2019

@lbernail

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2019

We'll need to cherry-pick this in 1.14, 1.13 and 1.12

@bazzargh

This comment has been minimized.

Copy link

commented Jun 14, 2019

(for the future cleanup, not this patch) Would suggest moving those strings into constants for consistency.

@dims

This comment has been minimized.

Copy link
Member

commented Jun 14, 2019

/assign @johnbelamaric

@liggitt

This comment has been minimized.

Copy link
Member

commented Jun 14, 2019

needs approval. once approved, go ahead and open the cherrypick against 1.15 if you want to make v1.15.0, since the master merge pool is full

@tpepper

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2019

@liggitt

This comment has been minimized.

Copy link
Member

commented Jun 14, 2019

@kubernetes/patch-release-team

note that this is targeting v1.15.0

@andrewsykim

This comment has been minimized.

Copy link
Member Author

commented Jun 14, 2019

Created #79044 for faster turn over into 1.15 if/when this one gets approved (please approve this one first).

@thockin

This comment has been minimized.

Copy link
Member

commented Jun 14, 2019

Thanks!

/lgtm
/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@andrewsykim

This comment has been minimized.

Copy link
Member Author

commented Jun 14, 2019

@thockin can you approve the cherry-pick PR as well #79044, thank you

k8s-ci-robot added a commit that referenced this pull request Jun 15, 2019

Merge pull request #79044 from andrewsykim/automated-cherry-pick-of-#…
…78999-origin-release-1.15

Automated cherry pick of #78999: ipvs: fix string check for IPVS protocol during graceful

@k8s-ci-robot k8s-ci-robot merged commit 101f9ff into kubernetes:master Jun 15, 2019

23 checks passed

cla/linuxfoundation andrewsykim authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-iscsi Skipped.
pull-kubernetes-e2e-gce-iscsi-serial Skipped.
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details

k8s-ci-robot added a commit that referenced this pull request Jun 18, 2019

Merge pull request #79103 from andrewsykim/automated-cherry-pick-of-#…
…78999-origin-release-1.12

Automated cherry pick of #78999: ipvs: fix string check for IPVS protocol during graceful

k8s-ci-robot added a commit that referenced this pull request Jun 20, 2019

Merge pull request #79101 from andrewsykim/automated-cherry-pick-of-#…
…78999-origin-release-1.14

Automated cherry pick of #78999: ipvs: fix string check for IPVS protocol during graceful

k8s-ci-robot added a commit that referenced this pull request Jun 22, 2019

Merge pull request #79102 from andrewsykim/automated-cherry-pick-of-#…
…78999-origin-release-1.13

Automated cherry pick of #78999: ipvs: fix string check for IPVS protocol during graceful
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.