Eclipse che Dashboard issue while deploying on AKS cluster (Failed to fetch the user profile data. Unable to get user profile data: Unauthorized: 401 error) #22665

venkatesh0822 · 2023-11-09T05:07:21Z

Summary

Hi All,
I am installing eclipse che on AKS cluster using chectl command (chectl server:deploy --platform k8s --che-operator-cr-patch-yaml cr-patch.yaml --domain --skip-oidc-provider-check) and using OIDC as Azure application details.

Installation completed Successfully and after hitting che dashboard URL Azure login page is appearing and able to login with AD credentials, after the oidc authentication it redirecting to che dashboard successful but I am getting below errors:

'Failed to fetch available workspaces, reason: Failed to fetch the list of devWorkspaces. Unable to list devworkspaces: Unauthorized'
'Failed to fetch the user profile data. Unable to get user profile data: Unauthorized'

Relevant information

Details:

Installation Command: chectl server:deploy --platform k8s --che-operator-cr-patch-yaml cr-patch.yaml --domain --skip-oidc-provider-check

cr-patch.yaml:

spec:
networking:
auth:
identityProviderURL: https://sts.windows.net/{TENANT_ID}/v2.0/
oAuthClientName: {CLIENT_ID}
oAuthSecret: {CLIENT_SECRET}
components:
cheServer:
extraProperties:
CHE_OIDC_AUTH__SERVER__URL: https://sts.windows.net/{TENANT_ID}/v2.0/
CHE_OIDC_USERNAME__CLAIM: name
(I have tried both 'CHE_OIDC_USERNAME__CLAIM: name' and 'CHE_OIDC_EMAIL__CLAIM: email')

OIDC details:

I have created application in Azure and used as OIDC
ClientID and TENANTID

CLIENT_SECRET

Authentication:

che-dashboard- logs:

body: {
kind: 'Status',
apiVersion: 'v1',
metadata: {},
status: 'Failure',
message: 'Unauthorized',
reason: 'Unauthorized',
code: 401
},
statusCode: 401
}

che-gateway-xxxx -c oauth-proxy logs:

No error messages

che logs:

2023-11-09 04:20:44,259[nio-8080-exec-2] [ERROR] [o.a.c.c.C.[.[.[/api].[default] 175] - Servlet.service() for servlet [default] in context with path [/api] threw exception
io.jsonwebtoken.ExpiredJwtException: JWT expired at 2023-11-09T04:07:24Z. Current time: 2023-11-09T04:20:44Z, a difference of 797258 milliseconds. Allowed clock skew: 3000 milliseconds.
at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:448)
at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:550)
at io.jsonwebtoken.impl.DefaultJwtParser.parseClaimsJws(DefaultJwtParser.java:610)
at io.jsonwebtoken.impl.ImmutableJwtParser.parseClaimsJws(ImmutableJwtParser.java:173)
at org.eclipse.che.multiuser.oidc.filter.OidcTokenInitializationFilter.processToken(OidcTokenInitializationFilter.java:80)
at org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:127)
at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:738)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:676)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:400)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:840)
2023-11-09 04:20:58,893[nio-8080-exec-5] [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198] - Unsupported OAuth provider gitlab
2023-11-09 04:20:58,915[nio-8080-exec-5] [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198] - Unsupported OAuth provider github
2023-11-09 04:20:58,937[nio-8080-exec-5] [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198] - Unsupported OAuth provider bitbucket

Errors after accessing che dashboard:

Error While creating workspace in che dashboard:

Can someone please help on this I have tried different below approaches to resolve this issue, but the issue is still appearing

Adding role assignment to cluster (az role assignment create --assignee "user" --role "Custom AKS Developer" --scope /subscriptions/xxx/resourcegroups/xx/providers/Microsoft.ContainerService/managedClusters/xx)
Enabling aad and providing AD access to user (az aks update -g xx -n xx --enable-aad --aad-admin-group-object-ids , [--aad-tenant-id ])
Deployed Cluster role and role-binding in AKS
Integrating AKS cluster (AKS-managed Microsoft Entra integration)
Deployed with enabling Auth mode and devWorkspace (Eclipse Che installation on aks with DevWorkspace #21023)
Created cluster with 'Azure AD authentication with Kubernetes RBAC' and 'Created cluster with Azure AD authentication with Azure RBAC'
Deployed Roles and Role Binding in AKS to users to access Che dashboard

venkatesh0822 · 2023-11-13T05:17:05Z

@tolusha I have followed this document to install Eclipse-che on AKS https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html

But as per document while using 'oAuthClientName' and 'oAuthScope' in below spec (Both are different app ID's ), we are Getting 403 (Login Failed: The upstream identity provider returned an error: invalid_client)
spec:
networking:
auth:
identityProviderURL: https://sts.windows.net/{TENANT_ID}/v2.0/
identityToken: access_token
oAuthClientName: {CLIENT_ID}
oAuthSecret: {CLIENT_SECRET}
oAuthScope: openid email profile xxxxx/user.read

If we deploy Eclipse-che without below values under spec we are able to install and login into che, but we are getting previously mentioned 401 Authorization error
identityToken: access_token
oAuthScope: openid email profile xxxx/user.read

ibuziuk · 2023-11-13T17:09:52Z

@karatkep hi, maybe you can take a look and provide some guidance based on your experience with https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html ?

karatkep · 2023-11-13T20:09:20Z

Hello @venkatesh0822,

Small hint: oAuthScope is a constant for AKS case.
oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read

I need following to help you:

Please make sure that your CheCluster custom resource contains:

spec:
  components:
    cheServer:
      debug: false
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: >-
          https://login.microsoftonline.com/{TENANT_ID}/v2.0/
        CHE_OIDC_EMAIL__CLAIM: unique_name
        CHE_OIDC_USERNAME__CLAIM: unique_name
  networking:
    auth:
      identityProviderURL: https://sts.windows.net/{TENANT_ID}/v2.0/
      identityToken: access_token
      oAuthClientName: {CLIENT_ID}
      oAuthSecret: {CLIENT_SECRET}
      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
      gateway:
        deployment:
          containers:
            - env:
                - name: OAUTH2_PROXY_OIDC_GROUPS_CLAIM
                  value: name
                - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
                  value: 'true'

please share oauth-proxy logs from che-gateway pod in case error.

venkatesh0822 · 2023-11-14T05:06:22Z

Hi @karatkep
I have tried 2 scenarios

1: I have used below spec and constant value of `oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read`

spec:
  components:
    cheServer:
      debug: false
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://login.microsoftonline.com/xxxxx/v2.0/
        CHE_OIDC_EMAIL__CLAIM: email
        CHE_OIDC_USERNAME__CLAIM: name
  networking:
    auth:
      identityProviderURL: https://sts.windows.net/xxxxxx/v2.0/
      identityToken: access_token
      oAuthClientName: xxxxx
      oAuthSecret: xxxxxx
      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
      gateway:
        deployment:
          containers:
            - env:
                - name: OAUTH2_PROXY_OIDC_GROUPS_CLAIM
                  value: name
                - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
                  value: 'true'

Microsoft login page appearing, but after login I'm getting 500 Internal Server Error

oauth-proxy logs:

[2023/11/14 04:35:34] [oauthproxy.go:959] No valid authentication in request. Initiating login.
10.244.0.24:56388 - 43ecdd30-32a7-4722-beb7-a8199384700e - - [2023/11/14 04:35:34] eclipse-che.centralus.cloudapp.azure.com GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.76" 302 445 0.000
[2023/11/14 04:35:36] [oauthproxy.go:823] Error redeeming code during OAuth2 callback: could not get claim "groups": failed to fetch claims from profile URL: error making request to profile URL: unexpected status "401": {"error":{"code":"InvalidAuthenticationToken","message":"Access token validation failure. Invalid audience.","innerError":{"date":"2023-11-14T04:35:36","request-id":"b0f5d00f-8bc9-4d21-bc91-c6f212cccedb","client-request-id":"b0f5d00f-8bc9-4d21-bc91-c6f212cccedb"}}}
10.244.0.24:56400 - bc05b95c-0421-4d57-ad67-f932eb6deb46 - - [2023/11/14 04:35:35] eclipse-che.centralus.cloudapp.azure.com GET - "/oauth/callback?code=0.ASsAyw9fyY7xoEaw3NSQ-T78Joa1adiJjPpFt9XDX5TElGDCAAA.AgABAAIAAAAmoFfGtYxvRrNriQdPKIZ-AgDs_wUA9P8haYf_xr6ZRrM5D7Ym8UH-DZ-jbNNFNmz95UNZOQXSasAhsCRGK8pw4PcRhx-Xl7pFlyBBvbjZkwJFc87iqaXm_jbPhXwQvZja9I8H6Tf7ZXSJrmm0ECU6eNW3Y-aPt35S41eiMoYsy2h2t-ToOGxBdVAthOLtAms9XD7GVo_dGqrGiYAybGy2hCUrwPn6H6x_FmJbiX4VeALrx1C2fQ_07yzkXj5w3E6C9fsZ1sfXcdpUzkeGUReYZ8lg2jRmu4D2l7mnysnSPFr9AsCDnWvRCjGHyjj4FvSA12k0bmUvWGmjwxi3OeANmLIN-HRguoABSDBX6GRt3Q2oQVHvwfWZy3ANkWnsgF3W0JuPMBvjzm6Ro9OpQBlBj-XtMe62UzyQ7esEVbrDSR7LBKOYq7WGcyEH1_mdo5l-rK0nsjHRNkO8f5ByXIlfNsufico_gfJdO86m_CM2o1iof6d8lYsgugBbz4J44E4FRM5X7S-ERbjR9O01hGAEuPmYD795qNb0GxQaX0eNK53blC78Zj21EE4wJkcSgkEm_Fs41CtbE9i04jPvM_Ql2vhe2QnfkbQmxi82TbjWP7eVYC1MpUay0bYIEE5k_724ot-CpM0icjaSREdM8TChPNfVJNGKOkDtAYOxp-ep6IXMR1Y1q-bRy3JBYxpYc_UEyrMyyOIvmFWaakPPgqmR-wiW9ikZFK4ia3J6EQSgtcX1jzJc_-raMpAqmLKns8eIM3enPqeZ6zTU5JmCn7F-kt38Sc7V4PnCAQTKDqZDNdDBsX3O4p6LoYzx5-RRgZPgJ8PKqAxS21Ce3ZlXgH1yq3_A0rd9QYkJK3FuGlxP_GNX_Qq2HYCkfs3HAfz0DEjHoViInLVz_e12gCI0sBjXE5poyt41kHK9BWHe5x6fEw-Ow7QqkyqLcyZEpv0GyU4R9eNmfxzKjb0wsuemFNInv9Wlu8-t2-J1MHLI628aapFVVNrN58h-uHdcMYqmTkrEN34gv3KPnH5VdAe8e4gcJK_7u_5ZKAXGg01dDj_1k70mHqlvB9QwpYCHTP74yryKxek4wLjxp_t-E35Cui6uHrsME1J32GF2AQCLrPNA6mAqhAS3oDrz98VISbinhJ-f349p9actyNxQORHbe0TcB8KWhFxhEcvjf_f9pPGQ_i4VFjFO3zCHazkVFX0swCuNnHpv_rTFaVg_n_-R_OD5OF9RuLVqdgMc3qXdm2SWQ6CqXTldKB_astRpyeMIqu-BJ1F1EiQ_lYlIPrtW0ZoFaRsKufwXyG_TKP3RjM3f_l7JnSbDKTCqb0IN2OBITg1SIG-5SkhLGC0yUKXQIIaR&state=K3A3QQAHZ_DS7kEpD0O-gGkf1_4g29IxUg49XivSvjw%3a%2ffavicon.ico&session_state=528c7c25-b434-4fdd-8b6c-f3a3bca669fd" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.76" 500 2840 0.385

2.

I have tried by removing identityToken: access_token from spec and deployed Eclipse-che, but there also I'm getting same 500 Internal Server Error and oauth-proxy logs.

karatkep · 2023-11-14T08:26:37Z

Hello @venkatesh0822 , could you please check if Token configuration contains groups claim?

venkatesh0822 · 2023-11-14T08:42:16Z

@karatkep, No claims in my token configuration. I have created new group claim now and deployed eclipse-che with below claim values
CHE_OIDC_EMAIL__CLAIM: unique_name
CHE_OIDC_USERNAME__CLAIM: unique_name

oauth-proxy logs:

[2023/11/14 08:36:37] [oauthproxy.go:959] No valid authentication in request. Initiating login.
10.244.0.24:33508 - 78710ba6-1844-4444-92ff-b3db934cff22 - - [2023/11/14 08:36:37] eclipse-che.centralus.cloudapp.azure.com GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.76" 302 445 0.000
[2023/11/14 08:36:39] [oauthproxy.go:823] Error redeeming code during OAuth2 callback: could not get claim "email_verified": failed to fetch claims from profile URL: error making request to profile URL: unexpected status "401": {"error":{"code":"InvalidAuthenticationToken","message":"Access token validation failure. Invalid audience.","innerError":{"date":"2023-11-14T08:36:39","request-id":"d134c314-85e3-450b-a0f8-69fa2ed58ff8","client-request-id":"d134c314-85e3-450b-a0f8-69fa2ed58ff8"}}}

karatkep · 2023-11-14T09:04:46Z

@venkatesh0822 , both CHE_OIDC_EMAIL__CLAIM and CHE_OIDC_USERNAME__CLAIM are part of cheServer component and not related to oauth-proxy.

By some reason, oauth-proxy configuration below has been not applied in your case

                - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
                  value: 'true'

You can see that in your logs:

could not get claim "email_verified"

Let's double check it. Please run echo command in oauth-proxy container:

echo $OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL

venkatesh0822 · 2023-11-14T09:31:59Z

@karatkep No value for echo $OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL in oauth-proxy container and in che-gateway deployment.

karatkep · 2023-11-14T09:34:18Z

@venkatesh0822,
so, we need to find a way to add envvar OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL=true to oauth-proxy container. Is there any idea?

venkatesh0822 · 2023-11-14T09:36:13Z

Looking on it.

venkatesh0822 · 2023-11-14T09:37:57Z

@karatkep I have tried by adding that value directly in deployment file, but it is not updating.

karatkep · 2023-11-14T09:40:04Z

@venkatesh0822 , right, che-operator will prevent direct modification in deployment file.

venkatesh0822 · 2023-11-14T09:42:31Z

Yes @karatkep, any idea how to update ?

karatkep · 2023-11-14T09:46:28Z

@venkatesh0822, I think the first step is to back to documentation and try to find some information about it... In case no luck, please let me know

venkatesh0822 · 2023-11-14T09:49:18Z

Ok @karatkep, I have followed this document https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html, do we have any other document also?

karatkep · 2023-11-14T09:52:56Z

@venkatesh0822, one moment, let me please try google.

karatkep · 2023-11-14T09:57:47Z

According to the google:

oauth-proxy documentation: https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/overview
che documentation to set envvar to the container: https://eclipse.dev/che/docs/stable/administration-guide/checluster-custom-resource-fields-reference/#checluster-custom-resource-components-common-deployment-containers-settings

venkatesh0822 · 2023-11-14T09:59:11Z

Thanks @karatkep, I will go through this documents.

venkatesh0822 · 2023-11-16T08:26:23Z

Hi @karatkep, I have tried different approaches to deploy che with this env value echo $OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL, but the value is not appearing in oauth-proxy container and below auth values also not appearing in any containers while using echo command.

networking:
    auth:
      identityProviderURL: https://sts.windows.net/xxxxxx/v2.0/
      identityToken: access_token
      oAuthClientName: xxxxx
      oAuthSecret: xxxxxx
      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
      gateway:
        deployment:
          containers:
            - env:
                - name: OAUTH2_PROXY_OIDC_GROUPS_CLAIM
                  value: name
                - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
                  value: 'true'

Still getting same below 500 error, can you help on this ?

Error redeeming code during OAuth2 callback: could not get claim "email_verified": failed to fetch claims from profile URL: error making request to profile URL: unexpected status "401": {"error":{"code":"InvalidAuthenticationToken","message":"Access token validation failure. Invalid audience."

karatkep · 2023-11-16T09:26:38Z

What Eclipse Che version do you use?

venkatesh0822 · 2023-11-16T09:37:58Z

Installing default version.
I'm using this command to install chectl server:deploy --platform k8s --che-operator-cr-patch-yaml cr-patch.yaml --domain xxxx --skip-oidc-provider-check and current installed version was 7.77.0

karatkep · 2023-11-16T09:45:47Z

Could you please run and share?

kubectl describe CheCluster -n eclipse-che

Important: please mask/hide secret information from the output before sharing

venkatesh0822 · 2023-11-16T09:54:12Z

#### kubectl describe CheCluster -n eclipse-che
Name:         eclipse-che
Namespace:    eclipse-che
Labels:       <none>
Annotations:  che.eclipse.org/checluster-defaults-cleanup:
                {"containers.resources":"true","spec.components.dashboard.headerMessage":"true","spec.components.pluginRegistry.openVSXURL":"true","spec.d...
API Version:  org.eclipse.che/v2
Kind:         CheCluster
Metadata:
  Creation Timestamp:  2023-11-16T09:34:45Z
  Finalizers:
    checluster.che.eclipse.org
    cluster-resources.finalizers.che.eclipse.org
    cheGateway.clusterpermissions.finalizers.che.eclipse.org
    dashboard.clusterpermissions.finalizers.che.eclipse.org
  Generation:        1
  Resource Version:  70517
  UID:               xxxxxx
Spec:
  Components:
    Che Server:
      Debug:  false
      Extra Properties:
        CHE_OIDC_AUTH__SERVER__URL:  https://login.microsoftonline.com/xxxxxx/v2.0/
        CHE_OIDC_EMAIL__CLAIM:       email
        CHE_OIDC_USERNAME__CLAIM:    name
      Log Level:                     INFO
    Dashboard:
      Log Level:  ERROR
    Dev Workspace:
    Devfile Registry:
    Image Puller:
      Enable:  false
      Spec:
    Metrics:
      Enable:  true
    Plugin Registry:
  Container Registry:
  Dev Environments:
    Container Build Configuration:
      Open Shift Security Context Constraint:  container-build
    Default Namespace:
      Auto Provision:                      true
      Template:                            <username>-che
    Disable Container Build Capabilities:  true
    Max Number Of Workspaces Per User:     -1
    Seconds Of Inactivity Before Idling:   1800
    Seconds Of Run Before Idling:          -1
    Security:
    Start Timeout Seconds:  300
    Storage:
      Pvc Strategy:  per-user
  Git Services:
  Networking:
    Auth:
      Gateway:
        Config Labels:
          App:                che
          Component:          che-gateway-config
      Identity Provider URL:  https://sts.windows.net/xxxxxxxxxxxx/v2.0/
      Identity Token:         access_token
      O Auth Client Name:     xxxxx
      O Auth Scope:           openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
      O Auth Secret:          xxxxxxx
    Domain:                   xxxxxxx
    Tls Secret Name:          che-tls

karatkep · 2023-11-16T09:57:27Z

As we can see envvar is missing here... it should look like this:

Spec:
  Networking:
    Auth:
      Gateway:
        Deployment:
          Containers:
            Env:
              Name:           OAUTH2_PROXY_OIDC_GROUPS_CLAIM
              Value:          name
              Name:           OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
              Value:          true

Please update it accordingly.

venkatesh0822 · 2023-11-16T10:41:43Z

Used below cr-patch.yaml file

apiVersion: org.eclipse.che/v2
spec:
  components:
    cheServer:
      debug: false
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://login.microsoftonline.com/xxxx/v2.0/
        CHE_OIDC_EMAIL__CLAIM: email
        CHE_OIDC_USERNAME__CLAIM: name
  networking:
    auth:
      identityProviderURL: https://sts.windows.net/xxx/v2.0/
      identityToken: access_token
      oAuthClientName: xxx
      oAuthSecret: xxx
      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
      gateway:
        deployment:
          containers:
            - env:
                - name: OAUTH2_PROXY_OIDC_GROUPS_CLAIM
                - value: name
                - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
                - value: 'true'

Output of kubectl describe CheCluster -n eclipse-che, but still getting same could not get claim "email_verified": failed to fetch claims from profile URL error.

Name:         eclipse-che
Namespace:    eclipse-che
Labels:       <none>
Annotations:  che.eclipse.org/checluster-defaults-cleanup:
                {"containers.resources":"true","spec.components.dashboard.headerMessage":"true","spec.components.pluginRegistry.openVSXURL":"true","spec.d...
API Version:  org.eclipse.che/v2
Kind:         CheCluster
Metadata:
  Creation Timestamp:  2023-11-16T10:31:52Z
  Finalizers:
    checluster.che.eclipse.org
    cluster-resources.finalizers.che.eclipse.org
    cheGateway.clusterpermissions.finalizers.che.eclipse.org
    dashboard.clusterpermissions.finalizers.che.eclipse.org
  Generation:        1
  Resource Version:  88533
  UID:               03363e73-e1bc-49c4-a5d1-d2a76d276ef0
Spec:
  Components:
    Che Server:
      Debug:  false
      Extra Properties:
        CHE_OIDC_AUTH__SERVER__URL:  https://login.microsoftonline.com/xxx/v2.0/
        CHE_OIDC_EMAIL__CLAIM:       email
        CHE_OIDC_USERNAME__CLAIM:    name
      Log Level:                     INFO
    Dashboard:
      Log Level:  ERROR
    Dev Workspace:
    Devfile Registry:
    Image Puller:
      Enable:  false
      Spec:
    Metrics:
      Enable:  true
    Plugin Registry:
  Container Registry:
  Dev Environments:
    Container Build Configuration:
      Open Shift Security Context Constraint:  container-build
    Default Namespace:
      Auto Provision:                      true
      Template:                            <username>-che
    Disable Container Build Capabilities:  true
    Max Number Of Workspaces Per User:     -1
    Seconds Of Inactivity Before Idling:   1800
    Seconds Of Run Before Idling:          -1
    Security:
    Start Timeout Seconds:  300
    Storage:
      Pvc Strategy:  per-user
  Git Services:
  Networking:
    Auth:
      Gateway:
        Config Labels:
          App:        che
          Component:  che-gateway-config
        Deployment:
          Containers:
            Env:
              Name:           OAUTH2_PROXY_OIDC_GROUPS_CLAIM
              Name:
              Value:          name
              Name:           OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
              Name:
              Value:          true
      Identity Provider URL:  https://sts.windows.net/xxx/v2.0/
      Identity Token:         access_token
      O Auth Client Name:     xxx
      O Auth Scope:           openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
      O Auth Secret:          xxxx
    Domain:                   xxxx
    Tls Secret Name:          che-tls
Status:
  Che Phase:              Active
  Che URL:                https://xxx
  Che Version:            7.77.0
  Devfile Registry URL:   https://xxxx/devfile-registry
  Gateway Phase:          Established
  Plugin Registry URL:    https://xxxx/plugin-registry/v3
  Workspace Base Domain:  xxxx

venkatesh0822 · 2023-11-16T15:29:30Z

@karatkep I will try to resolve it and let you know.

venkatesh0822 · 2023-11-16T16:09:49Z

The error logs of oauth-proxy and Che dashboard:

oauth-proxy Logs:

[error_page.go:93] Error proxying to upstream server: context canceled

che-dashboard: Logs

ERROR [15:56:02 UTC]: HTTP request failed
    err: {
      "type": "ae",
      "message": "HTTP request failed",
      "stack":
          HttpError: HTTP request failed
              at F._callback (/backend/server/backend.js:2:1305027)
              at t._callback.t.callback.t.callback (/backend/server/backend.js:8:1138802)
              at F.emit (node:events:513:28)
              at F.<anonymous> (/backend/server/backend.js:8:1151380)
              at F.emit (node:events:513:28)
              at IncomingMessage.<anonymous> (/backend/server/backend.js:8:1150212)
              at Object.onceWrapper (node:events:627:28)
              at IncomingMessage.emit (node:events:525:35)
              at endReadableNT (node:internal/streams/readable:1359:12)
              at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
      "response": {
        "statusCode": 401,
        "body": {
          "kind": "Status",
          "apiVersion": "v1",
          "metadata": {},
          "status": "Failure",
          "message": "Unauthorized",
          "reason": "Unauthorized",
          "code": 401
        },

As per che-dashboard logs, I think there was authorization issue.

I have updated below data, to resolve Failed to fetch the user profile data. timeout of 5000ms exceeded issue, but still the same timeout issue appearing in dashboard.

proxy-connect-timeout:
proxy-read-timeout:
proxy-buffer-size:
proxy-buffering:
proxy-buffers:

karatkep · 2023-11-16T16:38:36Z

@venkatesh0822 , could you please share oauth-proxy logs?

venkatesh0822 · 2023-11-17T07:33:41Z

Hi @karatkep oauth-proxy logs:

karatkep · 2023-11-17T08:11:02Z

Hello @venkatesh0822 , Could you please check and share logs from che pod?

venkatesh0822 · 2023-11-17T08:14:29Z

che pod logs

17-Nov-2023 07:25:19.218 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [24891] milliseconds
2023-11-17 07:28:02,385[://10.0.0.1/...]  [ERROR] [f.k.c.d.i.AbstractWatchManager 291]  - Unhandled exception encountered in watcher event handler
java.util.concurrent.RejectedExecutionException: null
        at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:48)
        at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager$SerialWatcher.eventReceived(AbstractWatchManager.java:70)
        at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.eventReceived(AbstractWatchManager.java:287)
        at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.onMessage(AbstractWatchManager.java:372)
        at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onMessage(WatcherWebSocketListener.java:52)
        at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$1.onMessage(OkHttpWebSocketImpl.java:116)
        at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:322)
        at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
        at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
        at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:273)
        at okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:209)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
2023-11-17 07:28:02,666[nio-8080-exec-9]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider gitlab
2023-11-17 07:28:02,702[nio-8080-exec-9]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider github
2023-11-17 07:28:02,728[nio-8080-exec-9]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider github_2
2023-11-17 07:28:02,755[nio-8080-exec-9]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider bitbucket
2023-11-17 07:28:02,755[nio-8080-exec-9]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider azure-devops
2023-11-17 07:28:03,332[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider gitlab
2023-11-17 07:28:03,354[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider github
2023-11-17 07:28:03,376[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider github_2
2023-11-17 07:28:03,404[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider bitbucket
2023-11-17 07:28:03,404[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider azure-devops
2023-11-17 07:28:06,083[nio-8080-exec-7]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider gitlab
2023-11-17 07:28:06,107[nio-8080-exec-7]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider github
2023-11-17 07:28:06,134[nio-8080-exec-7]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider github_2
2023-11-17 07:28:06,155[nio-8080-exec-7]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider bitbucket
2023-11-17 07:28:06,156[nio-8080-exec-7]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider azure-devops
2023-11-17 08:01:10,603[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider gitlab
2023-11-17 08:01:10,621[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider github
2023-11-17 08:01:10,639[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider github_2
2023-11-17 08:01:10,657[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider bitbucket
2023-11-17 08:01:10,657[io-8080-exec-10]  [WARN ] [o.e.c.s.oauth.EmbeddedOAuthAPI 198]  - Unsupported OAuth provider azure-devops

karatkep · 2023-11-17T09:35:59Z

@venkatesh0822 Could you please check in your browser: what is cookie size when request goes to /dashboard/api/... ?

venkatesh0822 · 2023-11-17T09:45:33Z

I'm getting 401 Unauthorized error:

Che-dashboard pod logs

      "body": {
        "type": "Object",
        "message": "Unauthorized",
        "stack":

        "kind": "Status",
        "apiVersion": "v1",
        "metadata": {},
        "status": "Failure",
        "reason": "Unauthorized",
        "code": 401
      },
      "statusCode": 401,
      "name": "HttpError"
    }
ERROR [09:42:39 UTC]: HTTP request failed
    err: {
      "type": "ae",
      "message": "HTTP request failed",
      "stack":
          HttpError: HTTP request failed
              at F._callback (/backend/server/backend.js:2:1328247)
              at t._callback.t.callback.t.callback (/backend/server/backend.js:8:1138802)
              at F.emit (node:events:513:28)
              at F.<anonymous> (/backend/server/backend.js:8:1151380)
              at F.emit (node:events:513:28)
              at IncomingMessage.<anonymous> (/backend/server/backend.js:8:1150212)
              at Object.onceWrapper (node:events:627:28)
              at IncomingMessage.emit (node:events:525:35)
              at endReadableNT (node:internal/streams/readable:1359:12)
              at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
      "response": {
        "statusCode": 401,
        "body": {
          "kind": "Status",
          "apiVersion": "v1",
          "metadata": {},
          "status": "Failure",
          "message": "Unauthorized",
          "reason": "Unauthorized",
          "code": 401
        },

karatkep · 2023-11-17T10:27:24Z

Hi @venkatesh0822,
I just wanted to check request size, not response size. Unfortunately, provided by you screenshots do not have such information.

We need to check request header size for calls /dashboard/api/...

venkatesh0822 · 2023-11-17T11:04:48Z

@venkatesh0822 Could you please check in your browser: what is cookie size when request goes to /dashboard/api/... ?

I found that 401 error and shared above details for your reference.

cookie details of api/

cookie deatils of /api/...

venkatesh0822 · 2023-11-17T11:23:33Z

Errors in console

FYI I'm using nginx-ingress-controller loadbalncer as a eclipse-che domain.
There are some SSL certificate errors in above console details, but I didn't find any SSL errors in all pod logs.

Only some warn logs available in nginx-ingress-controller pod logs

[warn] 93#93: *618 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/01/0000000011 while reading upstream,

karatkep · 2023-11-17T12:54:42Z

@venkatesh0822, not sure, but let's try to use unique_name instead of email:

CHE_OIDC_EMAIL__CLAIM: unique_name

venkatesh0822 · 2023-11-17T13:11:11Z

@venkatesh0822, not sure, but let's try to use unique_name instead of email:
CHE_OIDC_EMAIL__CLAIM: unique_name

I have tired this, but still same issue.
For authorization error, do we have any spec or values ?

karatkep · 2023-11-17T14:12:35Z

@venkatesh0822 I have no such documentation. My suggestion: to work on investigation, identify the error and fix it.

venkatesh0822 · 2023-11-20T05:48:19Z

Hi @karatkep, can you please share below details, so I can compare with my configurations.

Azure application configurations (App registrations > Application)
Kubernetes RBAC (for Authorization error)

karatkep · 2023-11-20T14:45:50Z

Hello @venkatesh0822, I haven't already been working on that for year and months.

Long story short... I had a task to install Che on Azure on my project about two yeas ago. I performed installation according to the instruction, and it did not work. I met and resolved lots of issues to make it workable. Most interesting things were documented in my blog post. That is it.

I think, I will try to repeat Che installation on Azure in the future. But not sure about timing due to high loading on my current project.

venkatesh0822 · 2023-11-21T04:04:23Z

Hello @venkatesh0822, I haven't already been working on that for year and months.

Long story short... I had a task to install Che on Azure on my project about two yeas ago. I performed installation according to the instruction, and it did not work. I met and resolved lots of issues to make it workable. Most interesting things were documented in my blog post. That is it.

I think, I will try to repeat Che installation on Azure in the future. But not sure about timing due to high loading on my current project.

Thanks @karatkep, I will try to debug this issue.
If you got anytime to install Che on Azure or any cloud, please let me know it will useful for me.

tolusha · 2023-12-14T11:59:43Z

@venkatesh0822 @karatkep
The correct configuration for oauth-proxy container is the following:

  networking:
    auth:
      gateway:
        deployment:
          containers:
          - env:
            - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
              value: "true"
            name: oauth-proxy

I was able to deploy Eclipse Che on Azure Kubernetes Service (following the https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html)
And will be glad to answer any questions.

tolusha · 2023-12-14T14:03:56Z

@karatkep

Do you mind if I create an article [1] How to deploy Eclipse Che on AKS based on your blogpost with some adjustments ?
[1] https://eclipse.dev/che/docs/stable/administration-guide/installing-che/

karatkep · 2023-12-14T14:31:37Z

Do you mind if I create an article [1] How to deploy Eclipse Che on AKS based on your blogpost with some adjustments ?

@tolusha, sure, please do it.

tolusha · 2024-01-11T08:03:26Z

Fixed by eclipse-che/che-docs#2670

karatkep · 2024-01-19T05:54:18Z

Fixed by eclipse-che/che-docs#2670

@tolusha , could you please share a link to that documentation?

tolusha · 2024-01-19T12:14:48Z

@karatkep
It make me realized that we have an issue with doc publishing.
I'll let you know when it will be available on [1]
[1]h ttps://eclipse.dev/che/docs/stable/overview/introduction-to-eclipse-che/

tolusha · 2024-01-20T13:49:10Z

@karatkep
https://eclipse.dev/che/docs/next/administration-guide/installing-che-on-microsoft-azure/

venkatesh0822 added the kind/question Questions that haven't been identified as being feature requests or bugs. label Nov 9, 2023

che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Nov 9, 2023

tolusha added the area/install Issues related to installation, including offline/air gap and initial setup label Nov 9, 2023

ibuziuk added status/analyzing An issue has been proposed and it is currently being analyzed for effort and implementation approach and removed status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. labels Nov 13, 2023

ibuziuk mentioned this issue Nov 28, 2023

Dev Tools Week Q4 2023 #22670

Closed

14 tasks

svor assigned tolusha Dec 20, 2023

tolusha mentioned this issue Dec 21, 2023

feat: Deploy Eclipse Che on AKS eclipse-che/che-docs#2670

Merged

5 tasks

tolusha closed this as completed Jan 11, 2024

tolusha removed the status/analyzing An issue has been proposed and it is currently being analyzed for effort and implementation approach label Jan 11, 2024

ibuziuk added the new&noteworthy/che-only Like 'new&noteworthy' but which do not apply to downstream (eg., plugins or devfiles) label Jan 19, 2024

Eclipse che Dashboard issue while deploying on AKS cluster (Failed to fetch the user profile data. Unable to get user profile data: Unauthorized: 401 error) #22665

Eclipse che Dashboard issue while deploying on AKS cluster (Failed to fetch the user profile data. Unable to get user profile data: Unauthorized: 401 error) #22665

Comments

venkatesh0822 commented Nov 9, 2023

Summary

Relevant information

cr-patch.yaml:

OIDC details:

che-dashboard- logs:

che-gateway-xxxx -c oauth-proxy logs:

che logs:

Can someone please help on this I have tried different below approaches to resolve this issue, but the issue is still appearing

venkatesh0822 commented Nov 13, 2023 • edited Loading

ibuziuk commented Nov 13, 2023

karatkep commented Nov 13, 2023 • edited Loading

venkatesh0822 commented Nov 14, 2023 • edited Loading

1: I have used below spec and constant value of oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read

2.

karatkep commented Nov 14, 2023

venkatesh0822 commented Nov 14, 2023

karatkep commented Nov 14, 2023

venkatesh0822 commented Nov 14, 2023

karatkep commented Nov 14, 2023

venkatesh0822 commented Nov 14, 2023

venkatesh0822 commented Nov 14, 2023

karatkep commented Nov 14, 2023

venkatesh0822 commented Nov 14, 2023

karatkep commented Nov 14, 2023

venkatesh0822 commented Nov 14, 2023 • edited Loading

karatkep commented Nov 14, 2023

karatkep commented Nov 14, 2023 • edited Loading

venkatesh0822 commented Nov 14, 2023

venkatesh0822 commented Nov 16, 2023

karatkep commented Nov 16, 2023

venkatesh0822 commented Nov 16, 2023

karatkep commented Nov 16, 2023

venkatesh0822 commented Nov 16, 2023

karatkep commented Nov 16, 2023 • edited Loading

venkatesh0822 commented Nov 16, 2023

venkatesh0822 commented Nov 16, 2023

venkatesh0822 commented Nov 16, 2023

oauth-proxy Logs:

che-dashboard: Logs

karatkep commented Nov 16, 2023

venkatesh0822 commented Nov 17, 2023

karatkep commented Nov 17, 2023

venkatesh0822 commented Nov 17, 2023

che pod logs

karatkep commented Nov 17, 2023

venkatesh0822 commented Nov 17, 2023

I'm getting 401 Unauthorized error:

Che-dashboard pod logs

karatkep commented Nov 17, 2023

venkatesh0822 commented Nov 17, 2023

cookie details of api/

cookie deatils of /api/...

venkatesh0822 commented Nov 17, 2023

Errors in console

karatkep commented Nov 17, 2023

venkatesh0822 commented Nov 17, 2023

karatkep commented Nov 17, 2023

venkatesh0822 commented Nov 20, 2023

karatkep commented Nov 20, 2023

venkatesh0822 commented Nov 21, 2023

tolusha commented Dec 14, 2023

tolusha commented Dec 14, 2023

karatkep commented Dec 14, 2023 • edited Loading

tolusha commented Jan 11, 2024

karatkep commented Jan 19, 2024

tolusha commented Jan 19, 2024

tolusha commented Jan 20, 2024

venkatesh0822 commented Nov 13, 2023 •

edited

Loading

karatkep commented Nov 13, 2023 •

edited

Loading

venkatesh0822 commented Nov 14, 2023 •

edited

Loading

1: I have used below spec and constant value of `oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read`

venkatesh0822 commented Nov 14, 2023 •

edited

Loading

karatkep commented Nov 14, 2023 •

edited

Loading

karatkep commented Nov 16, 2023 •

edited

Loading

karatkep commented Dec 14, 2023 •

edited

Loading