X-Forwarded-For & X-Real-IP = Addr of GKE Ingress Controller

[DocBradfordSoftware]

When using the example for NGINX on GCE, everything seems to work, but I am not getting the headers from the original client. It appears that the values for
X-Real-IP & X-Forwarded-For are the private ip addr of the gke ingress controller.

[tyranron]

@DocBradfordSoftware if you are using NGINX example then you are actually using Nginx Ingress Controller, not GCE Ingress Controller. But traffic is forwarded to your Nginx Ingress Controller via GCE load balancer, I suppose.
If so, take a look at Proxy protocol usage for Nginx Ingress Controller.

[DocBradfordSoftware]

At aws I would add this to my config map

use-proxy-protocol: "true"

If I try that with google container engine I get this error.

while reading PROXY protocol, client: 10.0.0.1, server: 0.0.0.0:443
2016-11-15T21:24:41.460205161Z 2016/11/15 21:24:41 [error] 198#198: *69 broken header

This is probably a gke issue, so I will head over there.

[kenng]

have the same error once the use-proxy-protocol is enabled. Does it means that we have to setup haproxy to use it properly if the nginx service is set as NodePort?

[simonswine]

GCE Load Balancers don't support proxy protocol. To get the real IP addresses with an nginx ingress, you have to rely on this alpha feature introduced in kubernetes 1.4:

https://kubernetes.io/docs/user-guide/load-balancer/#loss-of-client-source-ip-for-external-traffic

Closing this as it's not a kube-lego bug/limitation. Let me know how it goes...

[kenng]

Work like a charm!
need to set the type as LoadBalancer.
@simonswine so there is no way to avoid using LoadBalancer if we need to forward the client IP to the ingress? As the Network Load Balancing rule is kinda expensive for small project.

I have already disabled the httpLoadBalancing, still saw one loadbalancing rule got created once the type type is set as LoadBalancer.
gcloud container clusters describe my-project

addonsConfig:
  httpLoadBalancing:
    disabled: true

[alex88]

@kenng have you found a solution? My nginx ingress still shows 127.0.0.1 only as client ip

[DocBradfordSoftware]

@alex88, per the reference above, you need to add an annotation to the nginx controller service

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: nginx-ingress
  annotations:
    service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
  - port: 443
    name: https
  selector:
    app: nginx

[alex88]

@DocBradfordSoftware unfortunately that didn't help. :(
I've a domain pointing to the GCE load balancer IP with an A record, if I use curl with the domain name, nginx logs shows: 127.0.0.1 - [127.0.0.1], if I use curl with the load balancer IP address it correctly shows my client IP :/

[DocBradfordSoftware]

I used the echo service in the example. and got these values
X-Forwarded-For=70.197.xxx.xxx
X-Real-IP=70.197.xxx.xxx

double checking with whatsmyip
http://www.whatsmyip.org/
i got the same ip: 70.197.xxx.xxx

[alex88]

Lucky you :) I still can't get my real ip if I use a domain to connect to the load balancer

[DocBradfordSoftware]

I am not sure if what i am doing is correct, but the way I find my ip to hook up to the dns record, is that I go to gcloud console networking -> loadbalancing. there should be one instance, click on that and it will show the external ip.
Also if you look at the External IP Addresses tab, there should be one IP that points to a Forwarding Rule. Clicking on the forwarding rule, should bring you to the loadbalancing page above.

[alex88]

@DocBradfordSoftware I've found the discriminant, if I use port 443, I get 127.0.0.1 in the nginx ingress logs, if I use port 80 I get the real client IP

[artushin]

@alex88 Did you find a solution to your issue? I'm seeing the same problem. What version of the Nginx Ingress are you on?

[alex88]

@artushin nope, still the same issue, maybe it's because of the ports that the tcp load balancer can use the proxy protocol with, I don't really know.

[artushin]

Just upgraded to 0.9.0-beta.8 (https://github.com/kubernetes/ingress/releases). Looks good now.

Looks like it was probably this: kubernetes/ingress-nginx#233

[alex88]

@artushin Sorry forgot to mention I was using beta-5, I'll try with the beta 8. Anway, do you have any special config in the service? Like the service.beta.kubernetes.io/external-traffic: OnlyLocal annotation?

[artushin]

I do, but it's a one node dev cluster, and I'm running the nginx pods in a daemonset, so it would probably work even without it. But yeah, based on the k8s docs, if you have nodes that are not running an nginx pod, you should use that annotation if you need source IPs.

[alex88]

Oh yeah it works! That's awesome! :D

[alex88]

Now it's just a matter of having nginx trust the forwarded ip and that's it :)

[bviolier]

Did somebody also get it working when using a GCLB through an ingress in front of Nginx? The service.beta.kubernetes.io/external-traffic: OnlyLocal doesn't seem to work, because the service is just a NodePort.

Sorry for using this thread, but best way to get a solution :-)

[DocBradfordSoftware]

I would check out this documentation:
https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer.

kubectl patch svc loadbalancer -p '{"spec":{"externalTrafficPolicy":"Local"}}'

replace loadbalancer with your nginx service and add --namespace=xxx if you need to.

[bviolier]

@DocBradfordSoftware Problem is that our services is a type:NodePort, because we have an ingress in front of it. But I am now assuming that nobody in this thread has it setup that way.

PS. We use the ingress controller with GCP Http Loadbalancer, so we can use:

• TLS termination at the HTTP loadbalancer at Google
• ingress.gcp.kubernetes.io/pre-shared-cert
• kubernetes.io/ingress.global-static-ip-name

[DocBradfordSoftware]

You are correct, mine is on GKE, but I followed the kube-lego/examples/nginx and so the service is Type:LoadBalancer

[khash]

@bviolier we have the same setup and issue and been talking to Google support for about a week to find a solution. So far the best they have been able to offer is to setup an L2 load balancer with type:LoadBalancer and host nginx ourselves including all the ingress routing, SSL termination, etc in the nginx configuration.

[olala7846]

@bviolier we are using the same setup as you Ingress + NodePort as the GKE document suggested. with externalTrafficPolicy: Local set to the NotePort config, we no longer get source NAT'd as this document promised.
But the problem is, the health check of GCE loadbalancer simply not responding fast enough to catch up with the pod change while scaling, So it keeps sending traffic to nodes that has no healthy pods on it and then the requets get dropped directly 🙁
Does anyone managed to workaround this issue?

Update:
It turns out using Google L7 LB > Ingress > NodePort > nginx the IP was correctly written in the X-Forwared-For correctly, all I have to do is filter the GKE node IP address.

[ghost]

See my solution posted in kubernetes/ingress-nginx#808 (comment)