v2.9.56 breaks oidc auth with keycloak

[rsch136]

When updating from v2.9.45 to v2.9.56, our existing integration with keycloak breaks.

The error shown in the docker logs is time="2024-03-24T17:38:23Z" level=error msg="oauth2: \"invalid_grant\" \"Code not valid\"". On the UI side, signing in with SSO just returns you to the login page but does not actually log the user in.

A downgrade to 2.9.45 fixes the issue.

[fiftin]

Hi @rsch136 , thanks a lot

[fiftin]

@rsch136 Could you provide your provider configuration (without secrets), I need it to reproduce the issue?

@tboerger can you help? :)

[fiftin]

@rsch136 configuration which works for me:

"keycloak": {
	"display_name":"Sign in with keycloak",
	"provider_url":"http://localhost:8080/realms/master",
	"client_id":"test",
	"client_secret":"A0G8Awc7NTHrfllbXABbNK9kRQgQq7X9",
	"redirect_url": "http://localhost:8082/api/auth/oidc/keycloak/redirect"
}

[fiftin]

I found this by the issue: https://keycloak.discourse.group/t/getting-invalid-grant-during-client-token-request-using-authorization-code/2042/4

@rsch136
I need more info about your configuration. Do you have query string (/redirect?query_sting) in your redirect_url?

[rsch136]

This is our config:

{
"web_host": "https://semaphore......de",
  "oidc_providers": {
    "keycloak": {  
      "display_name": "SSO",
      "provider_url": "https://keycloak.../realms/network",
      "client_id": "semaphore",
      "client_secret": "{{ semaphore_keycloak_client_secret }}"
      }
  }
}

We do not specify a redirect url here.

[fiftin]

@rsch136 where you specify redirect_url? I have following error without it:

[bodzio4749]

Actually, 2.9.56 breaks much more. After the upgrade I got meself a surprise page too. Granted I am running inside the docker.

it was caused (I am guessing here) by my attempt to downgrade to 2.9.45 . There reason of this downgrade attempt was an
error in running my simple ansible job of pining some servers, which worked fine up till now. The error message in the task log
indicates problems with ... ssh-agent having problems reading socks (I wonder who game them under the Christmas tree). Here is the following screen snapshot.

SSH-AGENT is responsible for feeding passphrase when a SSH private key is pass protected. I am not using such protection for Semaphore. I created a special set of keys to and distributed them to all servers to avoid the problem.
I hope this helps. In the meantime, back to the last night snapshot for recovery.

[fiftin]

@bodzio4749 please create separate issue for your problem with details. Because I don't see whole error log.

PS: SSH-AGENT used in Semaphore starts form v2.9.45. There is no changes for this in v2.9.58.

[fiftin]

@bodzio4749 do you have issue with OpenID connect?

[rsch136]

@rsch136 where you specify redirect_url? I have following error without it:

Which version of keycloak are you using? I remember having this kind of issue when switching between keycloak version 14 and 19. We're currently using 19.0.3.

I'm not specifying any redirect url. I will deploy it later tonight with an explicit redirect url in the config.

[fiftin]

@rsch136 can we contact via Telegram? It is critical bug and I want to fix is ASAP. https://t.me/semaphoreui. Thank you

[mhzawadi]

FYI: 2.9.56 with Kycloak 24.0.1 is working fine, can provide any configs you need

[mhzawadi]

@bodzio4749 when you get that error, just re-login

[rsch136]

I've deployed it again with a redirect url and latest version but it does not fix the error. I guess it could now be a problem with our version of keycloak. Unfortunately, I will not be able to update that anytime soon.

FYI: 2.9.56 with Kycloak 24.0.1 is working fine, can provide any configs you need

@mhzawadi yes, please

@fiftin Since I am the only one who seems to be having this problem, I would be ok with closing the issue. I can open a new one if it still doesn't work once I've updated keycloak.

[fiftin]

Please don't close the issue :)

[bodzio4749]

@mhzawadi, @fiftin - yes, regular login with user and password. No OpenId.

[jsievertde]

Hmm everything over v2.9.45 seems to also break oidc with authentik as IDP.
Here is the relevant part of our config.json

"oidc_providers": {
        "XXX": {
            "display_name": "Authentik",
            "provider_url": "https://authentik.XXX.de/application/o/XXX/",
            "client_id": "XXX",
            "client_secret": "XXX",
            "redirect_url": "https://automation.XXX.de/api/auth/oidc/XXX/redirect/",
            "scopes": ["openid", "profile", "email"],
            "username_claim": "preferred_username",
            "name_claim": "preferred_username"
            }
        },

[mhzawadi]

@mhzawadi, @fiftin - yes, regular login with user and password. No OpenId.

My semaphore config below, then screen grabs of Keycloak. let me know if you need anything else

   "oidc_providers":{
      "horwood":{
         "display_name":"Sign in with keycloak",
         "provider_url":"https://keycloak/realms/PVE01",
         "client_id":"semaphore-real",
         "client_secret":"SECRETKEY",
         "redirect_url": "https://semaphore/api/auth/oidc/horwood/redirect"
      }
   },

[fiftin]

@mhzawadi it doesn't work for 2.9.58, but works for 2.9.45?

Could you provide error log?

[fiftin]

@jsievertde could you provide error log?

[pushpinders1ngh]

also an issue with using Okta and using the latest docker image. happy to provide more info as needed

error log:

time="2024-03-27T14:06:34Z" level=error msg="oauth2: \"invalid_grant\" \"The authorization code is invalid or has expired.\""

configuration:

    "oidc_providers": {
        "okta": {
            "display_name": "Sign in with Okta",
            "provider_url": "https://11111111.okta.com/oauth2/default",
            "client_id": "SOME_CLIENT_ID",
            "client_secret": "SOME_CLIENT_SECRET",
	    "name_claim": "name"
        }
    }

[fiftin]

@pushpinders1ngh just created Okta acc. Works to me. Can't reproduced.

Okta has a log. Can you share it?

[fiftin]

@jsievertde please share the logs.

[pushpinders1ngh]

@fiftin all successful logins

[fiftin]

Please try v2.9.64.

[pushpinders1ngh]

Unfortunately, the same result with v2.9.64.

[rsch136]

I'm also getting the same result but there is an extra error now: http: panic serving 172.18.0.4:50116: securecookie: hash key is not set

time="2024-03-27T17:58:22Z" level=error msg="oauth2: \"invalid_grant\" \"Code not valid\""
2024/03/27 17:59:35 http: panic serving 172.18.0.4:50116: securecookie: hash key is not set
goroutine 71 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0xbfd580?, 0xc0001a9b00?})
        /usr/local/go/src/runtime/panic.go:920 +0x270github.com/ansible-semaphore/semaphore/api.createSession({0x1e6ce60, 0xc000546000}, 0xc0004fa200, {0x1, {0x0, 0xedd926561, 0x0}, {0xc00040e340, 0x1d}, {0xc000013478, ...}, ...})
        /go/src/github.com/ansible-semaphore/semaphore/api/login.go:151 +0x491github.com/ansible-semaphore/semaphore/api.oidcRedirect({0x1e6ce60, 0xc000546000}, 0xc0004fa200)
        /go/src/github.com/ansible-semaphore/semaphore/api/login.go:655 +0x9d0
net/http.HandlerFunc.ServeHTTP(0xbdc5e0?, {0x1e6ce60?, 0xc000546000?}, 0xc?)
        /usr/local/go/src/net/http/server.go:2136 +0x29github.com/ansible-semaphore/semaphore/api.JSONMiddleware.func1({0x1e6ce60, 0xc000546000}, 0x45442b?)
        /go/src/github.com/ansible-semaphore/semaphore/api/router.go:44 +0xf6
net/http.HandlerFunc.ServeHTTP(0xb2de00?, {0x1e6ce60?, 0xc000546000?}, 0x774e8c?)
        /usr/local/go/src/net/http/server.go:2136 +0x29github.com/ansible-semaphore/semaphore/api.StoreMiddleware.func1.1()
        /go/src/github.com/ansible-semaphore/semaphore/api/router.go:35 +0x28github.com/ansible-semaphore/semaphore/db.StoreSession({0x1e791f0, 0xc0000666a0}, {0xc0002d42e0, 0xc}, 0xc00059f7b8)
        /go/src/github.com/ansible-semaphore/semaphore/db/Store.go:432 +0x5fgithub.com/ansible-semaphore/semaphore/api.StoreMiddleware.func1({0x1e6ce60?, 0xc000546000}, 0xc0004fa200)
        /go/src/github.com/ansible-semaphore/semaphore/api/router.go:34 +0xf5
net/http.HandlerFunc.ServeHTTP(0xc0004fa200?, {0x1e6ce60?, 0xc000546000?}, 0xbe6f80?)
        /usr/local/go/src/net/http/server.go:2136 +0x29github.com/ansible-semaphore/semaphore/cli/cmd.runService.func1.1({0x1e6ce60, 0xc000546000}, 0xc0002121e0?)
        /go/src/github.com/ansible-semaphore/semaphore/cli/cmd/root.go:74 +0xea
net/http.HandlerFunc.ServeHTTP(0xc000600000?, {0x1e6ce60?, 0xc000546000?}, 0xc00059f8e8?)
        /usr/local/go/src/net/http/server.go:2136 +0x29github.com/ansible-semaphore/semaphore/api.Route.CORSMethodMiddleware.func1.1({0x1e6ce60, 0xc000546000}, 0xc0002120f0?)
        /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/mux/middleware.go:51 +0x88
net/http.HandlerFunc.ServeHTTP(0xc0004fa100?, {0x1e6ce60?, 0xc000546000?}, 0xc00059f990?)
        /usr/local/go/src/net/http/server.go:2136 +0x29github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000180, {0x1e6ce60, 0xc000546000}, 0xc0004fa000)
        /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/mux/mux.go:212 +0x1c5github.com/ansible-semaphore/semaphore/cli/cmd.runService.ProxyHeaders.func2({0x1e6ce60, 0xc000546000}, 0xc0004fa000)
        /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/handlers/proxy_headers.go:59 +0x143
net/http.HandlerFunc.ServeHTTP(0xc000180104?, {0x1e6ce60?, 0xc000546000?}, 0x444140?)
        /usr/local/go/src/net/http/server.go:2136 +0x29github.com/ansible-semaphore/semaphore/cli/cmd.runService.cropTrailingSlashMiddleware.func3({0x1e6ce60, 0xc000546000}, 0xc0004fa000)
        /go/src/github.com/ansible-semaphore/semaphore/cli/cmd/server.go:27 +0xbd
net/http.HandlerFunc.ServeHTTP(0x235f100?, {0x1e6ce60?, 0xc000546000?}, 0xc00059fb50?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
net/http.serverHandler.ServeHTTP({0xc0004ba930?}, {0x1e6ce60?, 0xc000546000?}, 0x6?)
        /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc000126090, {0x1e6ec58, 0xc0004ba810})
        /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 1
        /usr/local/go/src/net/http/server.go:3086 +0x5cb
time="2024-03-27T17:59:35Z" level=error msg="oauth2: \"invalid_grant\" \"Code not valid\""

I've also changed the set-up in keycloak like recommended here: #1864 (comment)
But this did not change anything with v2.9.56. I assume it also has no affect on v.2.9.64

[fiftin]

@pushpinders1ngh please add "redirect_url": "https://semaphore/api/auth/oidc/okta/redirect" to your config file.

[fiftin]

@rsch136 looks like you have broken config. Please check that you have cookie_hash in your config file or SEMAPHORE_COOKIE_HASH environment variable. It must be generated by command head -c32 /dev/urandom | base64.

[rsch136]

@rsch136 looks like you have broken config. Please check that you have cookie_hash in your config file or SEMAPHORE_COOKIE_HASH environment variable. It must be generated by command head -c32 /dev/urandom | base64.

@fiftin we were following this docu: https://docs.semui.co/administration-guide/openid

Didn't previously have any issue with cookie hashes. Do you have a link?

[pushpinders1ngh]

Thank you @fiftin. Very generous of you to provide this quick solution. adding the redirect_url fixed it. Just curious how is it different from omitting the redirect_url and letting it use the default as suggested in the doc: ${web_host}/api/auth/oidc/${provider}/redirect, shouldn't this expand to the same value as long as web_host is defined correctly? did the behaviour of the default values changed in the newer versions?

i've in my config:
web_host: https://semaphore.example.com
redirect_url: https://semaphore.example.com/api/auth/oidc/okta/redirect

[fiftin]

@pushpinders1ngh I can't understand how it worked in 2.9.45. I tested with it and got error "invalid redirect_url".

[fiftin]

${web_host}/api/auth/oidc/${provider}/redirect

@phishkapsch Hm, it is really strange

[mhzawadi]

I have updated to 2.9.65 and all is working fine for me

[jsievertde]

@fiftin sorry for the delay. I copied the relevant error from our authentik-backend:

{
    "message": "Invalid redirect URI used by provider",
    "expected": [
        "https://automation.XXX.de/api/auth/oidc/XXX-authentik/redirect/"
    ],
    "provider": {
        "pk": 7,
        "app": "authentik_providers_oauth2",
        "name": "automation.XXX.de",
        "model_name": "oauth2provider"
    },
    "http_request": {
        "args": {},
        "path": "/application/o/token/",
        "method": "POST",
        "user_agent": "Go-http-client/2.0"
    },
    "redirect_uri": "https://automation.XXX.de/api/auth/oidc/XXX-authentik/redirect//api/auth/oidc/XXX-authentik/redirect"
}
User
{
    "pk": 1,
    "email": "",
    "username": "AnonymousUser",
    "is_anonymous": true
}

I'm just deploying the new version of ansible-semaphore to check if it is already fixed.

[fiftin]

Hi @jsievertde It is what I fixed in 2.9.64. Didn't help?

[jsievertde]

Sorry @fiftin deployment is done and the service has been restarted.
It's also fixed now for us.
Thanks for your work.

[rsch136]

Adding the cookie_hash to the env fixed the issue and I can now sign in using keycloak. I updated keycloak to 24.0.0 before adding the cookie_hash but this did not change anything. It also works without adding the redirect url to the config.

Thanks for the help!