Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Kubernetes aware of the LoadBalancer behaviour #1392

Merged
merged 1 commit into from Jun 18, 2020

Conversation

Sh4d1
Copy link
Member

@Sh4d1 Sh4d1 commented Dec 5, 2019

First draft of KEP to allow an option to change kube proxy behaviour regarding the LoadBalancer IPs

Signed-off-by: Patrik Cyvoct patrik@ptrk.io

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes needs-ok-to-test labels Dec 5, 2019
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Dec 5, 2019

Welcome @Sh4d1!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Dec 5, 2019

Hi @Sh4d1. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/M label Dec 5, 2019
@k8s-ci-robot k8s-ci-robot added kind/kep sig/network labels Dec 5, 2019
@Sh4d1
Copy link
Member Author

Sh4d1 commented Dec 5, 2019

/assign @caseydavenport

Copy link

@jcodybaker jcodybaker left a comment

I'm an engineer with DigitalOcean (called out in the KEP's motivation section), and added some clarification in comments on the motivation.

keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
1. Some cloud provider (DigitalOcean, Scaleway, ...) are using the LB's external IP as egress IP. This is a probem in the ipvs mode of kube proxy since the IP is bouded to an interface and healtchecks from the LB are never coming back.
2. Some cloud provider are also having features at the LB level (TLS termination, PROXY protocol, ...) and bypassing the LB means missing these features when the packet arrives to the service (leading to protocols errors).

So giving an options to these cloud to disable the actual beahviour would be very valuable.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional motivation:

Load-balancers which set status.loadBalancer.ingress[*].hostname instead of setting status.loadBalancer.ingress[*].ip don't exhibit this kube-proxy behavior. Because iptables/ipvs don't support rules based upon potentially dynamic hostnames, they leave the traffic untouched.

DigitalOcean has used this behavioral difference as a workaround the issue for customers who want to hairpin traffic through the LB (for example to get the ProxyProtocol headers appended). But it's hacky, and we much prefer the idea of addressing this functionality as the KEP proposes.

Copy link
Member Author

@Sh4d1 Sh4d1 Dec 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep AWS does the same! Forgot to add it 😅

@Sh4d1 Sh4d1 changed the title First draft of the KEP Give the ability to the LoadBalancer provisioner to disable the LB IP binding Dec 5, 2019
@Sh4d1
Copy link
Member Author

Sh4d1 commented Dec 21, 2019

@caseydavenport @thockin the first draft looks good to me

@Sh4d1
Copy link
Member Author

Sh4d1 commented Jan 24, 2020

@thockin any updates on this?

@sftim
Copy link
Contributor

sftim commented Feb 22, 2020

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Feb 22, 2020
@Sh4d1 Sh4d1 force-pushed the kube-proxy-disable-bind branch from b013220 to 46d6d37 Compare Feb 23, 2020
Copy link
Member

@MorrisLaw MorrisLaw left a comment

Great work! I've found some typos. I've crossed out and highlighted my suggestions to hopefully make it easier to make the changes. LGTM after you go through the comments 👍

keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
@Sh4d1 Sh4d1 force-pushed the kube-proxy-disable-bind branch from 0d81559 to f63cb1d Compare Mar 31, 2020
@Sh4d1
Copy link
Member Author

Sh4d1 commented Mar 31, 2020

Thanks for the review @MorrisLaw 😄 I think it's all resolved!

@Sh4d1
Copy link
Member Author

Sh4d1 commented Mar 31, 2020

@thockin any updates? I have a working code that seems to work 😄 Just need to choose the right naming!

@Sh4d1 Sh4d1 changed the title Give the ability to the LoadBalancer provisioner to disable the LB IP binding Make Kubernetes aware of the LoadBalancer behaviour Jun 13, 2020
@k8s-ci-robot k8s-ci-robot added size/L and removed size/M labels Jun 13, 2020
@Sh4d1
Copy link
Member Author

Sh4d1 commented Jun 13, 2020

KEP is updated with a bit more detail. Not sure what else I can add, open to suggestions 😄

If this were a flag on kube-proxy, I would encourage (for example) GKE to set
the flag correctly and configure nodes to not do this bypass (or to only do it
sometimes).

Devil's advocate: If we are adding a flag to kube-proxy, why not just make that the ONLY mechanism?

As stated at the end of the KEP, it works for managed solutions. If a user uses a self hosted cluster with a cloud's CCM, he then needs to be aware of the LB behaviour and act accordingly. That's why I tend to prefer the cloud controller manager solution.
That said, in fact an additional flag to kube proxy (maybe not in the scope of this kep though?) might be useful as well.

Copy link
Member

@thockin thockin left a comment

I am OK with this - @uablrek can you say whether this speaks to your need?

/approve

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jun 15, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Sh4d1, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved label Jun 15, 2020
@andrewsykim
Copy link
Member

andrewsykim commented Jun 18, 2020

@Sh4d1 can you squash your commits please?

Signed-off-by: Patrik Cyvoct <patrik@ptrk.io>
@Sh4d1 Sh4d1 force-pushed the kube-proxy-disable-bind branch from 0778bfa to 615efc2 Compare Jun 18, 2020
@Sh4d1
Copy link
Member Author

Sh4d1 commented Jun 18, 2020

@andrewrynhard squashed!

@andrewrynhard
Copy link

andrewrynhard commented Jun 18, 2020

@andrewsykim ^ 🙂

@Sh4d1
Copy link
Member Author

Sh4d1 commented Jun 18, 2020

Oh 🙄 totally sorry for that HL 😅

@MorrisLaw
Copy link
Member

MorrisLaw commented Jun 18, 2020

This should fit DigitalOcean's use case, lgtm 👍

@k8s-ci-robot k8s-ci-robot added do-not-merge/hold and removed do-not-merge/hold labels Jun 18, 2020
@MorrisLaw
Copy link
Member

MorrisLaw commented Jun 18, 2020

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Jun 18, 2020
@k8s-ci-robot k8s-ci-robot merged commit 8b9b994 into kubernetes:master Jun 18, 2020
2 of 3 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Jun 18, 2020
@thockin
Copy link
Member

thockin commented Jun 26, 2020

@tgraf You might want to track this, since you have a kube-proxy replacement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved cncf-cla: yes kind/kep lgtm ok-to-test sig/architecture sig/network size/L
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet