Why kube-proxy add external-lb's address to node local iptables rule?

[BSWANG]

/kind friction

What happened:
I have a LoadBalancer type service A of address 1.1.1.1. The external loadbalancer of service A is a TLS decoder, it will convert https requests to http hostport and endpoint. But since the kube-proxy add the external-lb's address to local iptables rule. Requests of https//1.1.1.1 will hijack to local http endpoints. Then https request failed.

What you expected to happen:
Kube-proxy don't add external-lb's address to local iptables. And the request will go through external-lb.

Environment:

• Kubernetes version (use kubectl version):
  1.10.4
• Cloud provider or hardware configuration:
  Alibaba Cloud
• OS (e.g. from /etc/os-release):
  Centos 7.4
• Kernel (e.g. uname -a):
  3.10.0-693
• Install tools:
  kubeadm

[BSWANG]

/sig network

[Lion-Wei]

I think the reason is, traffic from in cluster have no need to go out side(lb) then come back.
And probably lb should not do TLS decoder work.

[BSWANG]

@Lion-Wei Will it be an option to config this behavior？Some functions like TLS,Monitor,Logging doing on lb are more reasonable and high performance.
If kube-proxy add this option, I would like to commit a PR to it.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[BSWANG]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[BSWANG]

/remove-lifecycle stale

[jcodybaker]

We're also seeing this as an issue at DigitalOcean. It's a concern not just for load-balancer TLS termination, but also for supporting proxy protocol as encouraged in ( https://kubernetes.io/docs/tutorials/services/source-ip/ ). Traffic addressed to the LB ip from within the cluster never reaches the load-balancer, and the required proxy header isn't applied, causing a protocol violation.

The in-tree AWS service type loadbalancer supports proxy protocol and TLS termination, but because they populate status.loadbalancer.ingress.hostname rather than .ip they avoid this bug/optimization.

We're willing to put together a PR to address this there's interest from sig-network to accept it. We've considered a kube-proxy flag to disable the optimization, or the more complex option of extending v1.LoadBalancerIngress to include feedback from the cloud provider.

[bowei]

It seems like what is being implemented here is not modeled all that well in the current kube-proxy construct; your LB is not transparent from an L3/4 perspective and lives somewhere other than the node (it is terminating TLS, adding a proxy protocol field). It seems possible for nothing to break if we remove the external LB IPs from the node as a provider-specific setting, but it will require some thought.

[snuxoll]

If I’m being quite honest this behavior goes against the principle of least surprise. If I want kube-proxy to route my traffic I would use a ClusterIP service or similar and configure the application to use it, if I am hitting a LoadBalancer provided by a cloud provider there is likely to be a specific reason for that and kube-proxy shouldn’t try to second guess it.

[johnl]

We hit this same problem at Brightbox with our cloud controller manager. Our Load Balancers can terminate SSL and support the PROXY protocol. Confirmed with both ipvs and iptables kube-proxy modes. Any IPs we provide as LoadBalancerIngress are used to redirect outgoing connections and keep them within the cluster, breaking things.

To put it more clearly (imo), the problem is that kube-proxy assumes all load balancer services are straight TCP/UDP and that it can optimize them internally but that is not the case.

So either the assumption is wrong and kube-proxy needs fixing, or the assumption is right and external load balancers should not be doing these fancy things. I obviously think kube-proxy is wrong here :)

As @jcodybaker says above, AWS avoids this problem simply but not listing IP addresses as LoadBalancerIngress, only hostnames, which kube-proxy doesn't resolve.

We've changed our cloud controller to do the same, listing only hostnames in there, and it's fixed our problem too.

This feels a bit like a hack, but maybe it could be argued that's the way for a cloud controller to enable/disable the optimisation - a bit misguiding though.

We were also exposing the IPv6 addresses of our load balancers too, which actually didn't look properly supported further down the stack at all, so things do seem to be a bit of a mess here generally.

So a kube-proxy flag to disable the optimization would be a great start, but I think this all needs a closer look!

Plus, it's worth noting that in ipvs mode, the IP address is actually added locally to each node so it's not possible to selectively support this per-port. As soon as a load balancer lists the IP, it's completely unreachable directly.

[whereisaaron]

If I create an external loadbalancer I don't expect kube-proxy to hijack traffic addressed to that external loadbalancer.

That completely disregards any functionality the load balancer provides, including TLS termination, proxy protocol, logging, metrics, DPI firewalling, and... the actual the load balancing algorithm!

I'm perplexed that this is default behavior? It only seems useful if your external load balancer doesn't actually do anything useful 😄

As @jcodybaker says above, AWS avoids this problem simply but not listing IP addresses as LoadBalancerIngress, only hostnames, which kube-proxy doesn't resolve. I'd like to be able to lock this out on kube-proxy in general for cluster deployments. This behavior is only going to break things.

Thanks for raising this @BSWANG. This submerged iceberg could have spoiled my day one day, but now I know to watch out for this 👀

[k8s-ci-robot]

@BSWANG: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BSWANG]

reopen due this pr #96454 reverted

[aojea]

reopen due this pr #96454 reverted

the revert was done because that PR merged accidentally, to avoid misunderstandings, is not that the project doesn't want to implement, is that there are some process to add new features and that PR didn't follow them.
The KEP to implement the change has been approved https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1860-kube-proxy-IP-node-binding

@Sh4d1 are you still working on this #97681? if you want to include this in 1.22, the process has changed, please see https://groups.google.com/g/kubernetes-sig-network/c/3NYVEgvE1Uc/m/o68JAIt0DAAJ for more details

[Sh4d1]

@aojea yep! Layed the groundwork on #101027 but I was waiting on Tim's review, tough I understood he's been a bit busy! ( You're welcome if you want to review!).
Once this one is merged, I can rebase #97681 and it should be good.

Thanks, I'll read that 😄

[ryanlelek]

Commenting for support/+1 that self-referencing the cluster from within would be useful

@mkreidenweis-schulmngr 's hair-pin solution (kind of) worked but didn't pick up the correct name of the pod(s) to reroute to and would have worked with further digging.
Setting up a separate cluster solved the issue in the meantime.

Thanks for your work on this

[thockin]

#101027 was not assigned to me, so I missed it.

[thockin]

I'm going to close this as a dup, since we do have the KEP open

[liudongbo123]

how is the progress ？

[dcardellino]

Is there any solution for this?

[thockin]

Unfortunately the work to fix this sort of stalled for lack of someone to drive it to completion. Help wanted?

…

On Tue, Oct 25, 2022 at 8:51 AM Dominic Cardellino ***@***.***> wrote: Is there any solution for this? — Reply to this email directly, view it on GitHub <#66607 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVAQTN5RNEIQRTB7IQLWE7QVJANCNFSM4FL7W5EQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[timoreimann]

Hey @thockin 👋

I'd be happy to help pushing this over the finishing line. Is there a place to refresh my memory on where we stand right now and what remains to be done?

I vaguely remember that time was spent on a KEP but don't recall the details.

Appreciate any starting pointers.

[Sh4d1]

@timoreimann 👋 you can read kubernetes/enhancements#1860 (comment)
and Tim opened #106242 instead of #101027 .

So if nothing really changed, my last message still stands I guess!