Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DO Proxy Protocol broken header #3996

Open
dottodot opened this issue Apr 11, 2019 · 27 comments
Open

DO Proxy Protocol broken header #3996

dottodot opened this issue Apr 11, 2019 · 27 comments

Comments

@dottodot
Copy link

@dottodot dottodot commented Apr 11, 2019

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version:
0.24.0

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: Digital Ocean

What happened:
Digital Ocean now allows for use of Proxy Protocol
https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/#proxy-protocol
So I've added the annotation to my service

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

and updated my config as follows

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-proxy-protocol: "true"
  enable-brotli: "true"
  enable-vts-status: "true"

However once I've applied these changes I get lots of errors such as the following

6���Zك7�g̮["\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�
�98" while reading PROXY protocol, client: 10.244.35.0, server: 0.0.0.0:443
2019/04/11 13:02:57 [error] 265#265: *4443 broken header: "����p�����ўL��k+
rbO-
/�Ç���y�8\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�
�98" while reading PROXY protocol, client: 10.244.41.0, server: 0.0.0.0:443
2019/04/11 13:02:57 [error] 265#265: 4444 broken header: "���5�Kk��4 ��b�pxLJw�]��G�V��� �
\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�
�98" while reading PROXY protocol, client: 10.244.41.0, server: 0.0.0.0:443

Digital Ocean's response to these error was

This type of response is typically caused by Nginx not properly accepting the PROXY protocol. You should be able to simply append proxy_protocol to your listen directive in your server definition. More information on this can be seen in Nginx's documentation available here:

https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/

I can't see what I'm missing

What you expected to happen:
No errors

@aledbf

This comment has been minimized.

Copy link
Member

@aledbf aledbf commented Apr 11, 2019

However once I've applied these changes I get lots of errors such as the following
6���Zك7�g̮["\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�

It seems HTTPS traffic is being sent to the HTTP port, please check the port mapping in the ingres-nginx service and your DO console

@dottodot

This comment has been minimized.

Copy link
Author

@dottodot dottodot commented Apr 11, 2019

@aledbf my service ports are

  ports:
  - name: http
    nodePort: 32342
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31346
    port: 443
    protocol: TCP
    targetPort: https

and the port forwarding on my load balancer is set to
TCP on port 80 > TCP on port 32342
TCP on port 443 > TCP on port 31346

@Routhinator

This comment has been minimized.

Copy link

@Routhinator Routhinator commented Apr 13, 2019

Can confirm this issue, same configuration here across the board.

@tlaverdure

This comment has been minimized.

Copy link

@tlaverdure tlaverdure commented May 23, 2019

Looking for a solution to this as well. All public requests work but internal traffic to a host of the ingress fail.

2019/05/23 19:59:51 [error] 411#411: *870261 broken header: "��*EpS�M;I��K��WT}�:^ͼ�0���0�,�(�$��
����kjih9876�����2�.�*�&���=5" while reading PROXY protocol, client: 10.244.2.1, server: 0.0.0.0:443
@dperetti

This comment has been minimized.

Copy link

@dperetti dperetti commented May 23, 2019

It works for me using the helm chart with the following values:

## nginx configuration
## https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml

## https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol
## https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-loadbalancer
controller:
  config:
    use-proxy-protocol: "true"

  service:
    externalTrafficPolicy: "Local"
    annotations:
      # https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/
      # https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
@tlaverdure

This comment has been minimized.

Copy link

@tlaverdure tlaverdure commented May 24, 2019

I've got a similar setup. Which is working if I access the host publicly. However, accessing the host from within the cluster seems to fail (i.e. a server side request from one pod to another pod using the host)

Ingress

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: ingress
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
spec:
  tls:
  - hosts:
    - 'example.com'
    - '*.example.com'
    secretName: example-tls
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-app
          servicePort: 80
  - host: api.example.com
    http:
      paths:
      - backend:
          serviceName: example-backend
          servicePort: 80
...

Config

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  compute-full-forwarded-for: "true"
  use-forwarded-headers: "true"
  use-proxy-protocol: "true"

Curl Example Error

$ curl -v https://api.example.com
* Rebuilt URL to: https://api.example.com/
*   Trying (123.456.789.000...
* TCP_NODELAY set
* Connected to api.example.com (123.456.789.000) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to api.example.com:443 
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to api.example.com:443 

Log from failed request

[error] 661#661: *981009 broken header: "�/�9��ނ���R�6ަ�@%Qe�lG�3.���0�,�(�$��
����kjih9876�����2�.�*�&���=5" while reading PROXY protocol, client: 10.244.3.1, server: 0.0.0.0:443
@dperetti

This comment has been minimized.

Copy link

@dperetti dperetti commented May 24, 2019

I don't think service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" should be set in the Ingress annotation. It has probably no effect here.
My understanding is it must be set on the nginx-ingress service. I think it simply tells DO to activate the "Use Proxy Protocol" setting when the load balancer is created.

@aledbf

This comment has been minimized.

Copy link
Member

@aledbf aledbf commented May 24, 2019

However, accessing the host from within the cluster seems to fail (i.e. a server side request from one pod to another pod using the host)

@tlaverdure that's because you are no specifying the flag --haproxy-protocol in the curl command. If you enable proxy protocol in the ingress controller you need to decode it.

@aledbf

This comment has been minimized.

Copy link
Member

@aledbf aledbf commented May 24, 2019

I don't think service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" should be set in the Ingress annotation. It has probably no effect here.

That is correct.

@dottodot

This comment has been minimized.

Copy link
Author

@dottodot dottodot commented May 24, 2019

@dperetti I have service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" on the nginx-ingress service and get the same issue as @tlaverdure

@aledbf I'm not using curl command but get the same errors.

@dperetti

This comment has been minimized.

Copy link

@dperetti dperetti commented May 24, 2019

I also have: externalTrafficPolicy: Local on the nginx-ingress-controller service and of course use-proxy-protocol: "true" in the configmap.
My Kubernetes version is 1.14.1-do.2 .

@tlaverdure

This comment has been minimized.

Copy link

@tlaverdure tlaverdure commented May 24, 2019

@dperetti thanks for the tip. I think I added it initially when testing and forgot to remove after I added that annotation to the ingress-nginx service.

@aledbf I'm experiencing this with any type of server side http request. Curl was used to verify the issue, but any server side scripting language that makes a http request to the host (i.e. node) is failing.

@dottodot

This comment has been minimized.

Copy link
Author

@dottodot dottodot commented May 24, 2019

I also have externalTrafficPolicy: Local so don't think that's related.

@dperetti

This comment has been minimized.

Copy link

@dperetti dperetti commented May 24, 2019

Stupid check, but if you go to the load balancer's settings in the DO admin panel, it's enabled, right?
Because if it's not while use-proxy-protocol = true you end up with the same kind of encoding mismatch.
image

@tlaverdure

This comment has been minimized.

Copy link

@tlaverdure tlaverdure commented May 24, 2019

Yes, Proxy Protocol is enabled.

@tlaverdure

This comment has been minimized.

Copy link

@tlaverdure tlaverdure commented May 24, 2019

Just tested setting use-proxy-protocol: "false" in the ConfigMap. This kills my ability to reach the host externally but allows me to access the host within the cluster.

@dottodot

This comment has been minimized.

Copy link
Author

@dottodot dottodot commented Jun 12, 2019

OK I've had some advise back from Digital Ocean

Current options for workaround are to have pods access other DOKS services through their resolvable service names or by using the desired services clusterIP.

Using the service hostname as described below or the service cluster IP could be used for traffic originating inside the cluster.
kubectl get svc

Will return a list of your services and their clusterIP. You can use this IP to reach your service within the cluster. Note this IP will only work from within the cluster.

You can find documentation for accessing services properly in kubernetes here: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#services

The usage for accessing a service is: my-svc.my-namespace.svc.cluster.local:443

Using either of these methods will mean that that traffic no longer needs to go outside the traffic and interact with the proxy and can just get direct access to the service.

Only problem is I'm not entirely sure how to find which pods have the issue and need updating.

@dottodot

This comment has been minimized.

Copy link
Author

@dottodot dottodot commented Jun 13, 2019

Also when i turn on proxy protocol the logs suggest that not all requests have a broken header so how do I identify what's causing the broken header and fix it.

@MichaelJCole

This comment has been minimized.

Copy link

@MichaelJCole MichaelJCole commented Jun 15, 2019

I ran into this and at one point, I deleted the ingress service and recreated it and it worked. I got the broken headers issue when I had the nginx configmap set, but not the annotations on the ingress service that created the DO LB. Manually configuring "Proxy Protocol" on the LB w/ the Web UI didn't work for me.

Anyways, here is a config that worked for me:

mandatory.yaml

# Please note this file has been customized from the original at: 
# https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml  

#... stuff in the mandatory.yaml file that doesn't need to be customized.

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
# https://www.digitalocean.com/community/questions/how-to-set-up-nginx-ingress-for-load-balancers-with-proxy-protocol-support?answer=50244
data:
  use-forwarded-headers: "true"
  compute-full-forwarded-for: "true"
  use-proxy-protocol: "true"

# ... more stuff in the mandatory.yaml file that doesn't need to be customized.

cloud-generic.yaml

# Please note this file has been customized from the original at: 
# https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml 

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    # https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: https
      port: 443
      targetPort: https

ingress.yaml (I wanted a whitelist for only CloudFlare)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: allup
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "2400:cb00::/32, 2606:4700::/32, 2803:f800::/32, 2405:b500::/32, 2405:8100::/32, 2a06:98c0::/29, 2c0f:f248::/32, 173.245.48.0/20, 103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22, 141.101.64.0/18, 108.162.192.0/18, 190.93.240.0/20, 188.114.96.0/20, 197.234.240.0/22, 198.41.128.0/17, 162.158.0.0/15, 104.16.0.0/12, 172.64.0.0/13, 131.0.72.0/22"
spec:
  tls:
    - secretName: cloudflare-tls-cert
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: app-www-service
              servicePort: http
@w1ndy

This comment has been minimized.

Copy link

@w1ndy w1ndy commented Jul 9, 2019

Is it possible to allow nginx to listen on the same http/https port with and without proxy protocol like this setup?

@aledbf

This comment has been minimized.

Copy link
Member

@aledbf aledbf commented Jul 9, 2019

@lingxiankong

This comment has been minimized.

Copy link

@lingxiankong lingxiankong commented Aug 6, 2019

The same issue happens in Kubernetes cluster on top of OpenStack cloud (using openstack-cloud-controller-manager).

The Ingress service could be accessed from outside the cluster, but not from the cluster node or in the pod.

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Nov 4, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@lingxiankong

This comment has been minimized.

Copy link

@lingxiankong lingxiankong commented Nov 4, 2019

/remove-lifecycle stale

@davecranwell-vocovo

This comment has been minimized.

Copy link

@davecranwell-vocovo davecranwell-vocovo commented Nov 22, 2019

I'm a bit unclear whether this issue is the same one we just encountered, but perhaps the following is helpful:

We have a microservice architecture, where services talk to each other via the axios library, all inside the same cluster. What we'd misconfigured was the URL by which the services talk to each other. We had one service talk to the other via the external dns record by which the target service was known e.g foo.domain.com, causing traffic for it to go all the way out and back into the cluster again. When nginx tried to handle the request, the header looked broken because the request wasn't preceded by the instruction PROXY TCP4 10.8.0.18 [TARGET_IP] 55966 443 (which is what happens when you curl --haproxy-protocol, and is what happens to all inbound traffic handled by the ingress controller when you enable the "Proxy Protocol" setting)

By changing the URL of the target service to the internal dns record e.g http://[container].[namespace].svc.cluster.local by which it was known, traffic was sent directly to the target, not back through the ingress controller.

@peteychuk

This comment has been minimized.

Copy link

@peteychuk peteychuk commented Nov 27, 2019

Temporary solved it by using Cloudflare proxy mode for subdomains.
In this case, all traffic goes via Cloudflare proxy and works the same way for all traffic (Internal inside the cluster and external traffic).

Looking forward to the resolution of this issue.

@timoreimann

This comment has been minimized.

Copy link

@timoreimann timoreimann commented Dec 5, 2019

For users of DigitalOcean's managed Kubernetes offering (DOKS) and possibly others: you are most likely running into a specific bypassing behavior of kube-proxy that causes requests to never leave the cluster but go straight to the internal pod. For receiving pods that expect a certain protocol to be implemented by the load balancer (like proxy protocol or TLS termination), this breaks work flows.

Pointers to the related upstream issue and workarounds are described in our CCM README. The tracking issue to address the problem long-term for DOKS is at digitalocean/DOKS#8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.