x-forwarded-for does not set correct ip

[chinglinwen]

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Kong Ingress controller version:

kong:0.13.1-centos

Kubernetes version (use kubectl version):

# kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[root@dfkjyph kong]# uname -a
Linux dfkjyph 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@dfkjyph kong]# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core) 
[root@dfkjyph kong]# 

What happened:

Can't get the correct x-forwarded-for ip

What you expected to happen:

Set correct x-forwarded-for header for correct ip

How to reproduce it (as minimally and precisely as possible):

[root@dfkjyph kong]# cat dummy-application.yaml 

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: http-svc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: http-svc
  template:
    metadata:
      labels:
        app: http-svc
    spec:
      containers:
      - name: http-svc
        image: chinglinwen/echoserver:1.8
        ports:
        - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP

---

apiVersion: v1
kind: Service
metadata:
  name: http-svc
  labels:
    app: http-svc
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: http-svc

---
[root@dfkjyph kong]# cat dummy.ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: http-svc
spec:
  rules:
  - host: dummy.service
    http:
      paths:
      - path: /
        backend:
          serviceName: http-svc 
          servicePort: http
[root@dfkjyph kong]# 

access is from browser ( kong have nodePort type service, port 80 on every node )

map a name specified in dummy ingress to the ip of any node

get the following html page

Hostname: http-svc-55dd675888-wbvqx

Pod Information:
	node name:	dfkjyph-46-122
	pod name:	http-svc-55dd675888-wbvqx
	pod namespace:	default
	pod IP:	172.28.235.32

Server values:
	server_version=nginx: 1.13.3 - lua: 10008

Request Information:
	client_address=172.28.233.36
	method=GET
	real path=/
	query=
	request_version=1.1
	request_uri=http://172.28.235.32:8080/

Request Headers:
	accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
	accept-encoding=gzip, deflate
	accept-language=en,en-US;q=0.9,zh;q=0.8,zh-CN;q=0.7
	cache-control=no-cache
	connection=keep-alive
	host=172.28.235.32:8080
	pragma=no-cache
	upgrade-insecure-requests=1
	user-agent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
	x-forwarded-for=172.28.46.122
	x-forwarded-host=dummy.service
	x-forwarded-port=8000
	x-forwarded-proto=http
	x-real-ip=172.28.46.122

Request Body:
	-no body in request-

172.28.46.122 is node ip

expect x-forwarded-for be my desktop ip 172.28.66.71

Anything else we need to know:

[Ngob]

Isn't it related to https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport ? Did you try kubectl patch svc kong-proxy -p '{"spec":{"externalTrafficPolicy":"Local"}}

[ghost]

+1

[hbagdi]

This is related to the Kubernetes network configuration.
Please refer https://kubernetes.io/docs/tutorials/services/source-ip and setup your service accordingly.

Thank you for opening this issue.

[joubertredrat]

@hbagdi hi. Is possible to be a bug in kong? I tested today here with nginx-ingress kops addon and displayed correct x-real-ip, instead this, kong ingress displayed internal cluster ip or ec2 subnet ip on x-real-ip, as images below.

Ingress with kong

Ingress with nginx

In both examples I used mendhak/http-https-echo image

[raittes]

@joubertredrat, try setting trusted_ips and real_ip_header to use 'X-Forwarded-For' on Kong configuration, this can fix the x-real-ip:

trusted_ips: 0.0.0.0/0,::0
real_ip_recursive: "on"
real_ip_header: X-Forwarded-For

https://docs.konghq.com/1.3.x/configuration/#trusted_ips
https://docs.konghq.com/1.3.x/configuration/#real_ip_header

[Pierre-Malherbe]

Hello @raittes , i have the same issue with your configuration, I think the proxy does not support modification

[NICK-DUAN]

I have the same problem, too.

And I think it's important to me, In my Cluster, I will use kong to forward some requests to external services, and these external services will check x-forward-for header, Because they only support some ip address which in their white ip list.

We add our node ip to this white ip list, so I want to my request x-forward-for is a node ip.

[NICK-DUAN]

Hey @Pierre-Malherbe

I got this plugin at kong-ingress-controller version: 2.1.x
You can read this #issue

And this is my plugin:

kind: KongPlugin
apiVersion: configuration.konghq.com/v1
config:
  functions:
    - ngx.var.upstream_x_forwarded_for=nil
metadata:
  annotations:
    global: 'true'
    kubernetes.io/ingress.class: kong
  name: egress-plugin-remove-x-forwarded-for
plugin: pre-function

With this config on pre-function, it will make X-Forwarded-For to be my node ip in my kubernetes cluster.

[yaobo-lab]

I have the same problem. How was it finally resolved?

[NICK-DUAN]

I have the same problem. How was it finally resolved?

apiVersion: v1
kind: Service
metadata:
  annotations:
    konghq.com/override: cc     # you need this, cc is the name of kongingress
    kubernetes.io/ingress.class: kong   # ingressController
  name: cc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
      k8s-app: cc
      qcloud-app: cc
---
apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  annotations:
    kubernetes.io/ingress.class: kong
  name: cc
route:
  preserve_host: false  # this must be false
upstream:
  host_header: cc.isd.com   # this is the finally host that client get
--- 
# ingress yaml detail
# ---
# deployment yaml detail