"invalid session token: failed to verify signature: failed to verify id token signature" with argocd v2.5.2 and dex v2.35.x

[mconigliaro]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I found #11219, which supposedly fixes #11071 by using ghcr.io/dexidp/dex:v2.35.3 instead of ghcr.io/dexidp/dex:v2.35.3-distroless. When I try using any of these v2.35.x images, I get the following error in the webui when attempting to log in via SSO:

invalid session token: failed to verify signature: failed to verify id token signature

The latest version of dex that works for me is v2.31.2, so I'm working around this in my helm chart (but this leaves us exposed to: #10939):

dex:
  image:
    tag: v2.31.2

Possible regression of #1113?

To Reproduce

Use dex v2.35.x (e.g. v2.35.3) with argo-cd v2.5.2. Here's my dex.config:

configs:
  cm:
    url: ${var.okta_argocd_url}
    dex.config: |
      logger:
        level: debug
      connectors:
        - type: saml
          id: okta
          name: Okta
          config:
            ssoURL: ${var.okta_sso_url}
            caData: |
                ${var.okta_ca_data}
            redirectURI: ${var.okta_sso_redirect}
            usernameAttr: email
            emailAttr: email
            groupsAttr: group

Expected behavior

SSO should work.

Version

argocd: v2.5.2+148d8da
  BuildDate: 2022-11-07T16:42:47Z
  GitCommit: 148d8da7a996f6c9f4d102fdd8e688c2ff3fd8c7
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64

Logs

Interestingly, auth seems to be working if I look at the dex server logs:

time="2022-11-21T20:35:12Z" level=info msg="parsed and verified saml response attributes <REDACTED>"
time="2022-11-21T20:35:12Z" level=info msg="login successful: connector \"okta\" <REDACTED>"

[huangyutongs]

I encountered the same problem, I integrated with keycloak, click logout on the UI interface, and re-use keycloak to log in to the browser, the following error will appear

failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"unauthorized_client","error_description":"Invalid client secret"}

I need to fail 3 times before I can log in successfully

[davidtaddei]

I have also encountered this using the Dex GitHub Connector. The Dex server logs appear to have logged in successfully and I can see my GH user information however the argocd server then cannot validate the token.

[VazeeKevin]

same issue with Github connector after upgrading ArgoCD from v2.4 to v2.5.10.
argocd-dex-server log:

time="2023-02-06T10:41:48Z" level=info msg="login successful: connector \"github\", [...]

argocd-server log:

time="2023-02-06T10:42:07Z" level=info msg="Initializing OIDC provider (issuer: https://[argocdDomain]/api/dex)"
time="2023-02-06T10:42:07Z" level=warning msg="Failed to verify token: failed to verify token: Failed to query provider \"https://[argocdDomain]/api/dex\": 404 Not Found: Not Found\n"

client log:

{"error":"invalid session: failed to verify the token","code":16,"message":"invalid session: failed to verify the token"}

[michaelfedell]

Anyone able to identify if this is a bug in the current version of the connector, or is it simply a configuration error?

I am seeing the same issue (Dex logs report successful login, ArgoCD server reports "failed to verify") with the a similar config (GitHub Dex Connector, config snippet below)

Dex Connector Config

  dex.config: |
    connectors:
      - type: github
        id: github
        name: GitHub
        config:
          clientID: <redacted>
          clientSecret: <redacted>
          orgs:
            - name: <my-org>

[BernardoABC]

@michaelfedell I was experiencing the same issue, fixed by deleting the argocd-server pod. After a new pod was ready I was able to log in.

[VazeeKevin]

@BernardoABC thanks a lot for your feedback, it's now working properly after deleting argocd-server pods

[D3vl0per]

Oh my God! That's worked! Thanks @BernardoABC !

[mconigliaro]

FWIW, I just upgraded to helm chart version 5.22.1 and I'm not having this problem anymore.

[michaelfedell]

hah - I kind of hate it when problems just solve themselves, but either way, it's resolved! thanks for sharing your experience

[purduemike]

Not sure where this should happen. But, the pod(s) that need restarting should auto restart by some trigger. I'm using a helm chart to deploy ArgoCD.

[Vinaum8]

argocd app list
FATA[0000] rpc error: code = Unauthenticated desc = invalid session: signature is invalid

[Vinaum8]

argocd account list
FATA[0000] rpc error: code = Unauthenticated desc = invalid session: signature is invalid

[Vinaum8]

argocd login $ARGOCD_HOST_PRODUCTION --username admin --server $ARGOCD_HOST_PRODUCTION --grpc-web --config $ARGOCD_CONFIG_HOME/$ARGOCD_HOST_PRODUCTION
Password:
'admin:login' logged in successfully
Context 'argocd-manager.appnext.fit' updated

Same error, but login is successfully.
Argocd version 2.7.7

[sspreitzer]

I have this error every time rolling out a fresh Kubernetes cluster and ArgoCD installation. After one restart of the ArgoCD server pods everything works fine.

I would hate to implement a workaround in my Terraform manifests just to fix this.

I am not using dex. Any idea to what could cause this problem?

CA on the IdP is letsencrypt.

time="2023-07-31T08:41:37Z" level=info msg="Initializing OIDC provider (issuer: https://id.xxxx.xxx/auth/realms/master)"
time="2023-07-31T08:41:37Z" level=warning msg="Failed to verify token: failed to verify token: Failed to query provider \"https://id.xxxx.xxx/auth/realms/master\": Get \"https://id.xxxx.xxx/auth/realms/master/.well-known/openid-configuration\": x509: certificate signed by unknown authority"
time="2023-07-31T08:41:37Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: failed to verify the token" grpc.code=Unauthenticated grpc.method=List grpc.service=cluster.ClusterService grpc.start_time="2023-07-31T08:41:37Z" grpc.time_ms=15.149 span.kind=server system=grpc
time="2023-07-31T08:41:37Z" level=info msg="Initializing OIDC provider (issuer: https://id.xxxx.xxx/auth/realms/master)"
time="2023-07-31T08:41:38Z" level=warning msg="Failed to verify token: failed to verify token: Failed to query provider \"https://id.xxxx.xxx/auth/realms/master\": Get \"https://id.xxxx.xxx/auth/realms/master/.well-known/openid-configuration\": x509: certificate signed by unknown authority"
time="2023-07-31T08:41:38Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.content= grpc.service=session.SessionService grpc.start_time="2023-07-31T08:41:37Z" span.kind=server system=grpc
time="2023-07-31T08:41:38Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2023-07-31T08:41:37Z" grpc.time_ms=15.4 span.kind=server system=grpc

[Vinaum8]

I was logging in incorrectly.
I logged in as admin, then changed my user's password and logged in with the new user.

After that, if I had the error, I closed the terminal and redid the login.

[suryaval]

Issue occured when trying to add target cluster using argocd-cli

{"level":"fatal","msg":"rpc error: code = Unauthenticated desc = invalid session: failed to verify the token","time":"2023-12-18T16:18:59-06:00"}

Resolved by resetting the argocd context i.e. re-login to argocd cluster via SSO. This issue happens when the ArgoCD server is restarted and argocd context gets invalidated.