kubernetes ingress-nginx 500 Internal Server Error

[shinebayar-g]

Expected Behavior

Successfully running oauth2-proxy with ingress-nginx

Current Behavior

I'm running oauth2-proxy 6.1.1 & ingress-nginx 0.42.0 on DigitalOcean Kubernetes 1.19.3

Steps to Reproduce (for bugs)

oauth2-proxy is deployed with following config:

args:
  - --cookie-domain=.internal.mydomain.com
  - --authenticated-emails-file=/etc/oauth2_proxy/auth_users
  - --http-address=0.0.0.0:4180
  - --provider=google
  - --reverse-proxy=true
  - --upstream=file:///dev/null
  - --whitelist-domain=.internal.mydomain.com
  - --set-xauthrequest=true

env:
  - name: OAUTH2_PROXY_COOKIE_SECRET
    valueFrom:
      secretKeyRef:
        name:  oauth2-proxy
        key: cookie-secret
  - name: OAUTH2_PROXY_CLIENT_ID
    valueFrom:
      secretKeyRef:
        name:  oauth2-proxy
        key: client-id
  - name: OAUTH2_PROXY_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name:  oauth2-proxy
        key: client-secret

This is my Ingress config:

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: test-nginx
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-url: "https://oauth.internal.mydomain.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://oauth.internal.mydomain.com/oauth2/start?rd=https://$host$escaped_request_uri"
    nginx.ingress.kubernetes.io/auth-response-headers: Authorization
    nginx.ingress.kubernetes.io/configuration-snippet: |
      auth_request_set $name_upstream_1 $upstream_cookie__oauth2_proxy_1;

      access_by_lua_block {
        if ngx.var.name_upstream_1 ~= "" then
          ngx.header["Set-Cookie"] = "_oauth2_proxy_1=" .. ngx.var.name_upstream_1 .. ngx.var.auth_cookie:match("(; .*)")
        end
      }

      auth_request_set $user   $upstream_http_x_auth_request_user;
      auth_request_set $email  $upstream_http_x_auth_request_email;
      proxy_set_header X-User  $user;
      proxy_set_header X-Email $email;
spec:
  rules:
    - host: test.internal.mydomain.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: test-nginx
                port:
                  number: 80

Currently Nginx is not passing any traffic to oauth2-proxy container.

Nginx logs:

2020/12/27 06:54:06 [error] 1688#1688: *45647 broken header: "����������4q��8�]1ڊL���N����R=#��Q�'�8�,�0�̨̩̪�+�/��$�(k�#�'g�
��9�    ��3��=<5/��{" while reading PROXY protocol, client: 10.244.4.165, server: 0.0.0.0:443
2020/12/27 06:54:06 [error] 1688#1688: *45645 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: MY-PUBLIC-IP, server: mydomain.com, request: "GET /headers HTTP/1.1", subrequest: "/_external-auth-Lw-Prefix", upstream: "https://LOAD-BALANCER-IP:443/oauth2/auth", host: "mydomain.com"
2020/12/27 06:54:06 [error] 1688#1688: *45645 auth request unexpected status: 502 while sending to client, client: MY-PUBLIC-IP, server: mydomain.com, request: "GET /headers HTTP/1.1", host: "mydomain.com"
MY-PUBLIC-IP - - [27/Dec/2020:06:54:06 +0000] "GET /headers HTTP/1.1" 502 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.107 Safari/537.36" 0 0.001 [default-test-nginx-80] [] LOAD-BALANCER-IP:443 0 0.000 502 acbc0cf25d4a84c4733068ee03d6c0a4
MY-PUBLIC-IP - - [27/Dec/2020:06:54:06 +0000] "GET /headers HTTP/1.1" 500 572 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.107 Safari/537.36" 652 0.001 [default-test-nginx-80] [] - - - - acbc0cf25d4a84c4733068ee03d6c0a4
2020/12/27 06:54:06 [error] 1688#1688: *45651 broken header: "��������������1��ǻ
������9�������,"�8�,�0�̨̩̪�+�/��$�(k�#�'g�
��9�    ��3��=<5/��{" while reading PROXY protocol, client: 10.244.4.165, server: 0.0.0.0:443
2020/12/27 06:54:06 [error] 1688#1688: *45649 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: MY-PUBLIC-IP, server: mydomain.com, request: "GET /favicon.ico HTTP/1.1", subrequest: "/_external-auth-Lw-Prefix", upstream: "https://LOAD-BALANCER-IP:443/oauth2/auth", host: "mydomain.com", referrer: "https://mydomain.com/headers"
2020/12/27 06:54:06 [error] 1688#1688: *45649 auth request unexpected status: 502 while sending to client, client: MY-PUBLIC-IP, server: mydomain.com, request: "GET /favicon.ico HTTP/1.1", host: "mydomain.com", referrer: "https://mydomain.com/headers"
MY-PUBLIC-IP - - [27/Dec/2020:06:54:06 +0000] "GET /favicon.ico HTTP/1.1" 502 0 "https://mydomain.com/headers" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.107 Safari/537.36" 0 0.000 [default-test-nginx-80] [] LOAD-BALANCER-IP:443 0 0.000 502 19eae2728ae3925934ec320a6f6cd2bb
MY-PUBLIC-IP - - [27/Dec/2020:06:54:06 +0000] "GET /favicon.ico HTTP/1.1" 500 572 "https://mydomain.com/headers" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.107 Safari/537.36" 536 0.000 [default-test-nginx-80] [] - - - - 19eae2728ae3925934ec320a6f6cd2bb

SSL is terminated at DigitalOcean Load Balancer not on ingress-nginx.
Any help would be appreciated.

[shinebayar-g]

Ugh, it fixed by setting service.beta.kubernetes.io/do-loadbalancer-hostname: "mydomain.com" annotation on the ingress-nginx-controller service.

[JoelSpeed]

Glad you fixed it, but I'm not sure entirely why that helps? Seems that Nginx was maybe getting an invalid https response when trying to connect, was it actually going through the load balancer? What effect does the new annotation have?

[shinebayar-g]

I guess It has something to do with the broken proxy protocol header and looks like this is the work around to fix the issue. I can also disable the proxy protocol setting which was enabled by default on them.

[atmosx]

You kind of saved me. I spent most of the day trying to debug this problem. I was adding annotation to the ingresses instead of the nginx-controller service.

Ty!

[a0s]

@atmosx adding the annotation to nginx-controller service (instead of ingress) in result disable authentification.