Get proxy protocol working correctly

[rjbaat]

Hi All,
I am trying to get proxy protocol enabled on my Traefik setup on DigitalOcean k8s.
The hw load balancer has proxy protocol. I can enable it manually, but then I get a error 400 respons from my ingressRoute.

I tried to add an additional argument to the entry point web with the trusted ip. I tried it with the internal subnet of DO and 127.0.0.1/32 like stated in the documentation. But on my 3 node cluster it sometimes gives the node ip of one of the 3 cluster nodes, instead of my ISP ip. I think this is because it is the node that Traefik container actual lives on.

--entryPoints.web.proxyProtocol.trustedIPs=127.0.0.1/32,<INTERNAL_IP_SUBNET>

I noticed that it only truly works when I add the additional argument like:

--entryPoints.web.proxyProtocol.insecure

Any idea how to get proxy protocol working with the DO load balancer entry points, without getting the node ip?

[tanandy]

hi, are you using externalTrafficPolicy to local ? (check the lb from traefik)

fyi, you get 400 response when you setup proxy protocol only 1 side . (remember you need to add it at LB lvl + at your ingress level)

[rjbaat]

Hi, when I check the Traefik LB it stated:

externalTrafficPolicy: Cluster

[tanandy]

ok sorry, you dont try to get the real source ip , do you ? or you just want proxy protocol enable .
because to retrieve the real source ip with the proxy protocol, you will need to use traffic policy to local afaik

[rjbaat]

No I actually do want the see client ip and not the ip's in between.
Aha i will test it with setting it to local then.

[tanandy]

could you describe the LB created to see if we have some annotation on your lb for the proxy protocol

[tanandy]

Aha i will test it with setting it to local then.

ok then if you need also to retrieve the real ip, you will need to be in local at least

[tanandy]

for your information i put a working example here, you can get some insights with it on OVH and adapt it to DO

https://github.com/tanandy/helm-ovh-ingress/blob/main/ingress/traefik/values.yaml
https://github.com/tanandy/helm-ovh-ingress/blob/main/ingress/traefik/values-upgrade-ips.yaml

[tanandy]

https://docs.digitalocean.com/products/kubernetes/how-to/configure-load-balancers/#:~:text=Enabling%20the%20PROXY%20protocol%20allows,information%20from%20the%20load%20balancer.

service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" in your value as i did for ovh https://github.com/tanandy/helm-ovh-ingress/blob/main/ingress/traefik/values.yaml#L12

[rjbaat]

I tried to do the

externalTrafficPolicy: Local

But then only 1 node is healthy on the hw load balancer. Because it will not forward traffic to the other nodes. So I don't think that is the solution.

What I did was adding this to the helm chart as extra answers:

additionalArguments:
  - "--entryPoints.web.proxyProtocol.trustedIPs=127.0.0.1/32,10.133.0.0/16"
service:
  annotations: 
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

This is what I will test atm, to see if it solves it.

[rjbaat]

@tanandy thnx!
I think that this indeed solves it. I only see my own ip now.
Stupid, since I had this annotation for NGINX before aswel 🙈.

[tanandy]

you have 2 different things :

1/ you need to activate proxy protocol on ingress + lb (to support proxy protocol)

2/ you need to use externalTrafficPolicy local (to be able to retrieve real ip)

[rjbaat]

Number 2 isn't needed, since it then breaks the LB.
I guess that would only work if Traefik runs as a DaemonSet.

Edit i was to soon. The problem stil occurs :(

[tanandy]

really, i always needed to setup externaltrafficpolicy to local to get the real ip of the user. you may be lucky
https://blog.getambassador.io/externaltrafficpolicy-local-on-kubernetes-e66e498212f9
https://www.asykim.com/blog/deep-dive-into-kubernetes-external-traffic-policies

[rjbaat]

Well if I do that, 2 of the 3 nodes in the LB pool will show unhealthy.