login loop using ingress nginx, auth_request always returning 401

[jurgenweber]

Expected Behavior

Go to the ingress hostname for the first time, be greeted with google login.
Select google account, redirect to application setup to be behind hostname.

Current Behavior

Go to the ingress hostname for the first time, be greeted with google login.
Select google account..... Select google account.... Select Google account.

I can see in ingress nginx that the auth_request requests are always returning a 401.

"/oauth2/auth" HTTP/1.1 "Go-http-client/1.1" 401 21 0.000
2019/02/25 02:37:26 oauthproxy.go:796: 100.97.114.11:35790 ("54.79.36.100") Cookie "_oauth2_proxy" not present

Steps to Reproduce (for bugs)

I am using the helm chart, here is the values.yaml deployed:

config:
  clientID: "asdfasdfasdfasdf.apps.googleusercontent.com"
  clientSecret: "asdf"
  cookieSecret: "asdfasdf=="
  configFile: |-
    pass_basic_auth = false
    pass_access_token = true
    set_authorization_header = true
    pass_authorization_header = true

image:
  repository: "quay.io/pusher/oauth2_proxy"
  tag: "v3.1.0"
  pullPolicy: "IfNotPresent"

extraArgs:
  provider: "google"
  email-domain: "example.com.au"
  whitelist-domain: ".stuff.example.com.au"
  upstream: "file:///dev/null"
  http-address: "0.0.0.0:4180"

authenticatedEmailsFile:
  enabled: false
  template: ""
  restricted_access: ""

ingress:
  enabled: true
  path: /
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
  hosts:
    - oauth2.k8s-url.com.au
  tls:
    - secretName: wildcard.k8s-url.com.au
      hosts:
        - oauth2.k8s-url.com.au

I think have the ingress annotation config of:

    nginx.ingress.kubernetes.io/auth-response-headers: Authorization
    nginx.ingress.kubernetes.io/auth-signin: https://oauth2.k8s-url.com.au/oauth2/start?rd=https://$host$request_uri$is_args$args
    nginx.ingress.kubernetes.io/auth-url: https://oauth2.k8s-url.com.au/oauth2/auth
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    #nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri
    #nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
    nginx.ingress.kubernetes.io/configuration-snippet: |
      auth_request_set $name_upstream_1 $upstream_cookie_name_1;

      access_by_lua_block {
        if ngx.var.name_upstream_1 ~= "" then
          ngx.header["Set-Cookie"] = "name_1=" .. ngx.var.name_upstream_1 .. ngx.var.auth_cookie:match("(; .*)")
        end
      }

I have gone through many iterations to get to this point.

Context

I am try to use this auth proxy without luck. To me it just looks like the nginx auth-request always returns a 401. I realise this maybe an ingress nginx issue but I thought I would start here.

Your Environment

kops 1.11 maintained k8s cluster, k8s version 1.11.6

• Version used: "v3.1.0"

[JoelSpeed]

We really need to improve the documentation here, apologies!

Can you check which domain the set-cookie header is being set for when you complete your authentication on the proxy?

I think you might need to set cookie-domain to match your whitelist-domain flag, can you try that and see if that helps?

[jurgenweber]

ok, I had a wildcard in for the whitelist but for testing sake to see it work, I just hard coded both cookie-domain and whitelist-domain to the same value... This seems to have made things worse. :)

403 Permission Denied
http: named cookie not present

[jurgenweber]

aaah, so with your clue and a little bit of experimentation...
URL/Domain I am logging into:

https://my.thing.example.com

cookie-domain: "example.com"
whitelist-domain: ".thing.example.com"

and this worked!

Yeah, some more documentation would help... It took me all morning to work out that the whitelist-domain was mandatory and that is why everything I was logging into was going to "/". :)

[JoelSpeed]

Yes I believe your findings match our configuration here, I'm out of office right now so can't check unfortunately

If you have some time to update the docs I would appreciate it. The whitelist domain flag is, as you say, mandatory if (and only if) you wish to host the oauth2 proxy on a separate domain to the ones you are protecting. A short guide for using this with the nginx Auth request mode would be good

[jurgenweber]

ok, another question.

The power of having the oauth-proxy on its own domain is that you can have many things use it to login. Is there a way to have per domain ACL on it?

domain1.k8s.example.com
--> allow email domain example2.com
domain2.k8s.example.com
--> Allow email user bob@example2.com, rob@example2.com

does that make sense? Or so I need to have a separate proxy for each one?

[jurgenweber]

yeah, so I am trying a another proxy install for the site I want to have email users:

403 Permission Denied
http: named cookie not present

is what I get, I matched the cookie domain and hostname to be the same.

[JoelSpeed]

domain1.k8s.example.com
--> allow email domain example2.com
domain2.k8s.example.com
--> Allow email user bob@example2.com, rob@example2.com

does that make sense? Or so I need to have a separate proxy for each one?

At present you will need a separate proxy for each one

yeah, so I am trying a another proxy install for the site I want to have email users:

403 Permission Denied
http: named cookie not present

is what I get, I matched the cookie domain and hostname to be the same.

Can you inspect the responses when setting the cookie and check that the browser is storing the cookie you expect?

[jurgenweber]

Yeah, I had to create another proxy on another domain and match all the settings for the two authentication scopes I was after.

[tamsky]

Yeah, some more documentation would help... It took me all morning to work out that the whitelist-domain was mandatory

A bit more logging here would be super helpful. I hit this required setting today as well.

A log.Printf near the end of IsValidRedirect [1] would be super useful after none of the whitelistDomains have returned true.

https://github.com/pusher/oauth2_proxy/blob/bd64aeb7ee859aab0d85b9a95e517c6e20787b94/oauthproxy.go#L626-L630

[s1apped]

Hi,

I have the similar issue

Proxy oauth logs:

GET - "/oauth2/auth" HTTP/1.1 "Go-http-client/1.1" 401 21 0.000
GET - "/oauth2/start?rd=https://my.domain.eu/.well-known/acme-challenge/XXXXXXXX" HTTP/1.1 "Go-http-client/1.1" 302 456 0.000
2019/11/15 13:24:53 oauthproxy.go:830: Cookie "_oauth2_proxy" not present

Ingress controller:

"GET /oauth2/auth HTTP/1.1" 401 21 "http://app.test.eu/.well-known/acme-challenge/xxxxxxxxxxxxxxxxxxxxxxxxxx" "Go-http-client/1.1" 576 0.001 [XXX-proxy-4180] [] xx.xxx.x.xxx:4180 21 0.000 401 xxxxxxxxxxxxxxxxxxxxxxxxxx
"GET /.well-known/acme-challenge/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 HTTP/1.1" 401 0 "http://app.test.eu/.well-known/acme-challenge/xxxxxxxxxxxxxxxxxxxxxxx" 
"Go-http-client/1.1" 0 0.005 [pact-pact-80] [] 13.80.147.2:443 0 0.004 401 xxxxxxxxxxxxxxxxxxxxxxx

I'm using Azure. This is the setup:

Application:

    ingress:
      enabled: true
      annotations:
        certmanager.k8s.io/cluster-issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
        nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
        nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
      hostnames:
      - app.test.eu
      tls:
        enabled: true

Proxy:

  values:
    config:
      clientID: xxxxxxxxxxxxxxxxx
      clientSecret: xxxxxxxxxxxxxxxxxxxxxxxxx
      cookieSecret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      configFile: |
        provider = "azure"
        azure_tenant = "xxxxxxxxxxxxxxxxxxxx"
        email_domains = [ "test.eu" ]
    service:
      port: 4180
    readinessProbe:
      periodSeconds: 240
    extraArgs:
      whitelist-domain: ".test.eu"
      cookie-domain: "test.eu"
    ingress:
      enabled: true
      path: /oauth2
      hosts:
        - app.test.eu
      annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
      tls:
        - hosts:
            - app.test.eu
          secretName: test-tls

I tried multiple combination and can't figure it out. Any ideas?

[JoelSpeed]

@s1apped Try your cookie domain with a . at the beginning, I believe it has to have this to be applicable to subdomains

cookie-domain: ".test.eu"

[s1apped]

@JoelSpeed unfortunatelly it didn't help.

[infinitydon]

Has anyone been able to fix this? Constantly getting 401 Unauthorized error.

@s1apped -- Were you able to find a work around?

I have tried all kinds of combinations for the cookie-domain and whitelisting, oauth2_proxy keeps throwing the Cookie "_oauth2_proxy" not present

[JoelSpeed]

If you are using azure, I would recommend trying the redid session storage mechanism. The cookie storage does not work very well with Azure due to the size of the ID Tokens they generate

[github-actions]

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

[drewwells]

I am able to reproduce this. We are using the 3.0.0 helm chart

          - --cookie-domain=.test.example.com
          - --email-domain=*
          - --github-team=example
          - --provider=github
          - --redirect-url=https://oauth.test.example.com
          - --upstream=file:///dev/null
          - --whitelist-domain=.test.example.com

[JoelSpeed]

@drewwells Your redirect-url is invalid, it should point to the OAuth2 Proxy callback eg https://oauth.test.example.com/oauth2/callback

[psreddy22]

@JoelSpeed
Is there any doc to configure traefik (ingress Controller), oauth2-proxy & DEX (OIDC).
We configured k8s dashboard & oauth2
Issues are :- we are able to redirect and authenticate with DEX & landing again with dashboard login page.

authResponseHeaders:
  - X-Forwarded-User
  - X-Auth-User
  - X-Secret