Please enable proxy protocol in k3s ingress

[sandys]

This issue is related to similar issues discussed in the docker swarm world - moby/moby#39465 and moby/moby#25526

we need source ip address to be passed on through ingress. The only standards compliant way to do this and be compatible with all kinds of upstream LB (including cloud loadbalancers) is by using proxy protocol

https://github.com/containous/traefik/blob/master/docs/content/routing/entrypoints.md

Traefik already has support for proxyprotocol - this needs to be enabled in the right way for both upstream loadbalancers like ELB to pass on proxy protocol info. And in case clients directly connect, then traefik must inject the appropriate headers

[sandys]

hi guys,
any update on this ? Since you use traefik already as ingress, this shouldnt be too hard right ?

we are having trouble in a production deploy of k3s on AWS

1. behind a load balancer (it should pass through the headers)
2. while just using k3s (it should inject source headers)

[sunpeinju]

hi, is there any update on this issue?

[niklaskorz]

I suppose this would have to be implemented in https://github.com/rancher/klipper-lb @ibuildthecloud ? It looks like Amazon supports this behind an annotation for LoadBalancer services: service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*". Maybe something similar could be done for k3s' servicelb. The PROXY protocol is specified at http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt, I might be willing to get my hands on this task.

[brandond]

Not sure about how it works with klipper-lb (the thing behind k3s's servicelb), but with metallb at least you have to use externalTrafficPolicy: Local in the Service entry for the Traefik Ingress, otherwise inbound traffic will come in through other nodes and you'll lose the original source address as it passes through the other node before hitting the node that's actually running the pod.

[niklaskorz]

As far as I know, the goal of the PROXY protocol is to not make this a problem, as the load balancer keeps track of the source IP to forward it properly to the service.

[sandys]

Agreed. Proxy protocol injects this and can pass through several nodes. Each node can optionally inject additional layers into the header to capture the whole chain (for debugging purposes) This is scalable and supported across all clouds.

…

On Fri, 24 Apr, 2020, 00:32 Niklas Korz, ***@***.***> wrote: As far as I know, the goal of the PROXY protocol is to not make this a problem, as the load balancer keeps track of the source IP to forward it properly to the service. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#852 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYU5VS4ENVR6HUR42WW3ROCGC5ANCNFSM4I4MXPRQ> .

[brandond]

Yes, PROXY protocol is supposed to pass the information along. That only works if the first entrypoint into the system supports it. If you pass it through kubelet NAT or kube-proxy tunnel before hitting the Ingress Service which grabs the original layer 3 source and destination, that information is lost. Your LB layer needs to either keep the inbound traffic local (MetalLB) or add the Proxy information itself (ELB).

[ibuildthecloud]

The solution to this issue is going to be flipping some settings in the traefik chart we install by default or special annotation on your ingress resource. I don't know what that is, but I'd start with first finding out how to use PROXY protocol with Traefiks ingress controller. klipper-lb should have nothing to do with this because it's just a TCP pass through using iptables. If you are just using a service load balancer and not ingress, proxy protocol will work already.

[sandys]

we have had trouble doing this in traefik, so for our current production effort (we are deploying to AWS using k3s), we are going with haproxy.

…

On Sat, Apr 25, 2020 at 12:35 AM Darren Shepherd ***@***.***> wrote: The solution to this issue is going to be flipping some settings in the traefik chart we install by default or special annotation on your ingress resource. I don't know what that is, but I'd start with first finding out how to use PROXY protocol with Traefiks ingress controller. klipper-lb should have nothing to do with this because it's just a TCP pass through using iptables. If you are just using a service load balancer and not ingress, proxy protocol will work already. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#852 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYUZECAWQZC4JZDQIINTROHPJNANCNFSM4I4MXPRQ> .

[cjellick]

@ibuildthecloud do you think we should update our traefik chart to support this or is this "on the user" to configure?

[davidnuzik]

Is this configurable now because of HelmChartConfig CRD? (To be discussed in our next design call)

[brandond]

@davidnuzik yes and doing so is actually covered in the example docs!

https://rancher.com/docs/k3s/latest/en/helm/#customizing-packaged-components-with-helmchartconfig - see the proxyProtocol: bit.

[davidnuzik]

Test as low priority. Leverage the docs that Brad outlined.

[davidnuzik]

Should not block v1.19.3 release.

[rancher-max]

Validated in v1.19.3+k3s1 using the example mentioned in the docs

• Viewing logs of the traefik pod shows enabling of the proxyprotocol with the specified trusted IPs.:

{"level":"info","msg":"Enabling ProxyProtocol for trusted IPs [10.0.0.0/8]","time":"2020-10-16T17:43:55Z"}
{"level":"info","msg":"Preparing server https \u0026{Address::443 TLS:0xc0003d17a0 Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:0xc00079b7e0 ForwardedHeaders:0xc00079b8c0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2020-10-16T17:43:55Z"}
{"level":"info","msg":"Enabling ProxyProtocol for trusted IPs [10.0.0.0/8]","time":"2020-10-16T17:43:55Z"}

Notice the difference when not including the config manifest has mention of proxyprotocol but without any trusted ips:

$ k logs -n kube-system traefik-5dd496474-btz5n | grep -i proxy
{"level":"info","msg":"Preparing server https \u0026{Address::443 TLS:0xc0006157a0 Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0005e4320} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2020-10-16T18:01:00Z"}
{"level":"info","msg":"Preparing server prometheus \u0026{Address::9100 TLS:\u003cnil\u003e Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:false ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0005e4460} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2020-10-16T18:01:00Z"}
{"level":"info","msg":"Preparing server http \u0026{Address::80 TLS:\u003cnil\u003e Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0005e4300} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2020-10-16T18:01:00Z"}

• All pods come up successfully and appear to be functional

[arctica]

This is only half working. Traefik supports Proxy Protocol but it needs the actual proxy data from the upstream injected into the TCP stream. Loadbalancers from AWS et al do this for you but the Loadbalancer service (Klipper) from k3s does not.

Klipper is essentially two lines of iptables and isn't capable of prepending data into the TCP stream. It took me some good time to realize this.

@rancher-max Your test only checks if Traefik enables Proxy Protocol but it probably did not check if Traefik actually sees the real client IP or was using a loadbalancer from AWS etc. Enabling the setting without having the actual Proxy Protocol data being sent will not do much.

[sandys]

Same here. We had to rip out load balancers and traefik and put haproxy everywhere. Not ideal at all.

…

On Sun, 29 Nov, 2020, 00:07 arctica, ***@***.***> wrote: This is only half working. Traefik supports Proxy Protocol but it needs the actual proxy data from the upstream injected into the TCP stream. Loadbalancers from AWS et al do this for you but the Loadbalancer service (Klipper) from k3s does not. Klipper is essentially two lines of iptables and isn't capable of prepending data into the TCP stream. It took me some good time to realize this. @rancher-max <https://github.com/rancher-max> Your test only checks if Traefik enables Proxy Protocol but it probably did not check if Traefik actually sees the real client IP or was using a loadbalancer from AWS etc. Enabling the setting without having the actual Proxy Protocol data being sent will not do much. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#852 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYUZFDDFE7TRYFZTMBJDSSE7M5ANCNFSM4I4MXPRQ> .

[arctica]

@sandys Yes we also ripped out the k3s built in loadbalancer and just exposed Traefik directly via HostPort (note this only works if you configure the pod explicitly to run as root as the NET_BIND_SERVICE capability is not working either). Many places suggest not to do this but I can't see the benefit of the Klipper loadblancer in this case or why there need to be svclb-traefik pods running on every node.

[sandys]

The cleaner thing to do here would have been for the loadbalancer to inject/pass-on proxy protocol headers. In which case, everything downstream would have worked properly

…

On Sun, 29 Nov, 2020, 00:24 arctica, ***@***.***> wrote: @sandys <https://github.com/sandys> Yes we also ripped out the k3s built in loadbalancer and just exposed Traefik directly via HostPort (note this only works if you configure the pod explicitly to run as root as the NET_BIND_SERVICE capability is not working either). Many places suggest not to do this but I can't see the benefit of the Klipper loadblancer in this case or why there need to be svclb-traefik pods running on every node. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#852 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYU6453OZDY5N5I2NEP3SSFBO3ANCNFSM4I4MXPRQ> .

[jon-nfc]

starting to notice a theme here with k3s, a suggestion is made to fix/make standards compliant and nothing happens except the closing of the issue.

Of note, the validate post above was undone by #852 (comment), this issue needs to be addressed.

Considering that knowing the IP address of the actual endpoint(s) is a must for security, this issue should not have been closed and an actual solution, like OP's suggestion added. I have a barebones cluster and use nginx ingress which also supports proxy protocol however serviceLB doesn't add it and like everyone else can't get the real IP because of that.

Same here. We had to rip out load balancers and traefik and put haproxy everywhere. Not ideal at all.

@sandys, looks like i may have to do the same. I Know it's an old post, any major issues doing this on a production cluster?

[brandond]

Please don't bump a 3+ year old closed issue. If you'd like to reopen the topic, please create a new issue or discussion.