[Bug]: 500 error when trying to log in to dashboard

[asanzgom]

Is there an existing issue for this?

• I have searched the existing issues

Deploy type

Downstream version (eg. RHODS 1.29)

Version

2.4.0

Current Behavior

On a cluster that has

There is this error in the oauth-proxy container of the rhods dashboard pods:

2023/11/07 14:19:35 provider.go:631: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2023/11/07 14:19:35 provider.go:671: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
"issuer": "https://oauth-openshift.apps.gcp-4-13-2-4.mzx2.s2.devshift.org/",
"authorization_endpoint": "https://oauth-openshift.apps.gcp-4-13-2-4.mzx2.s2.devshift.org/oauth/authorize",
"token_endpoint": "https://oauth-openshift.apps.gcp-4-13-2-4.mzx2.s2.devshift.org/oauth/token",
"scopes_supported": [
"user:check-access",
"user:full",
"user:info",
"user:list-projects",
"user:list-scoped-projects"
],
"response_types_supported": [
"code",
"token"
],
"grant_types_supported": [
"authorization_code",
"implicit"
],
"code_challenge_methods_supported": [
"plain",
"S256"
]
}
2023/11/07 14:19:35 oauthproxy.go:654: error redeeming code (client:10.129.2.8:52702): got 400 from "https://oauth-openshift.apps.gcp-4-13-2-4.mzx2.s2.devshift.org/oauth/token" {"error":"unauthorized_client","error_description":"The client is not authorized to request a token using this method."}
2023/11/07 14:19:35 oauthproxy.go:445: ErrorPage 500 Internal Error Internal Error

Expected Behavior

It should be possible to log into the dashboard seamlessly when upgrading or installing RHODS 2.4 without any disabling/enabling process

Steps To Reproduce

1. clean the cluster with: https://gitlab.cee.redhat.com/data-hub/olminstall/-/blob/main/cleanup.sh?ref_type=heads
2. manually remove:
   2.1) https://console-openshift-console.apps.ods-qe-03.rhods.ccitredhat.com/k8s/cluster/customresourcedefinitions[…]tions.dscinitialization.opendatahub.io
   2.2) https://console-openshift-console.apps.ods-qe-03.rhods.ccitredhat.com/k8s/cluster/customresourcedefinitions[…]ters.datasciencecluster.opendatahub.io
3. install RHODS 2.4
4. you will see that DSC Initialization is present
5. create a Data Science Cluster
   you will have the error above

Workaround (if any)

Disabling the dashboard component from the DSC and enabling it again fixed the problem

[zdtsw]

/transfer opendatahub-operator

[openshift-ci]

@zdtsw: Something went wrong or the destination repo opendatahub-io/operator does not exist.

In response to this:

/transfer operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[andrewballantyne]

/transfer opendatahub-operator

[VaishnaviHire]

The cause for this issue is stale oauth-client secret.

Possible Fix:
The secret controller doesn't watch for oauth-client secret modifications. Update secret controller to watch for changes in secret and compare values to regenerate Oauth Client resource.

Ref: https://github.com/opendatahub-io/opendatahub-operator/blob/incubation/controllers/secretgenerator/secretgenerator_controller.go#L84

[zdtsw]

The cause for this issue is stale oauth-client secret.

Possible Fix: The secret controller doesn't watch for oauth-client secret modifications. Update secret controller to watch for changes in secret and compare values to regenerate Oauth Client resource.

Ref: https://github.com/opendatahub-io/opendatahub-operator/blob/incubation/controllers/secretgenerator/secretgenerator_controller.go#L84

I dont follow this.
in both v1 and v2, the secret is the same, from dashboard's manifests https://github.com/opendatahub-io/odh-dashboard/blob/main/manifests/base/oauth.secret.yaml
when v2 enables dashboard component, it wont see any updates comparing with the one already in the cluster (from v1), so no updates made in this case, right? (similar to from v2.X to v2.X+1)

so why not we just force a r.deleteOAuthClient when setupmanager ?

[VaishnaviHire]

The cause for this issue is stale oauth-client secret.
Possible Fix: The secret controller doesn't watch for oauth-client secret modifications. Update secret controller to watch for changes in secret and compare values to regenerate Oauth Client resource.
Ref: https://github.com/opendatahub-io/opendatahub-operator/blob/incubation/controllers/secretgenerator/secretgenerator_controller.go#L84

I dont follow this. in both v1 and v2, the secret is the same, from dashboard's manifests https://github.com/opendatahub-io/odh-dashboard/blob/main/manifests/base/oauth.secret.yaml when v2 enables dashboard component, it wont see any updates comparing with the one already in the cluster (from v1), so no updates made in this case, right? (similar to from v2.X to v2.X+1)

so why not we just force a r.deleteOAuthClient when setupmanager ?

Yea I did think about this as well. Not sure though, if doing so will require dashboard restart. Looking at the manifests, I think we are just mounting the secrets, so it should be fine.

[kpouget]

Hello @zdtsw , I see that this issue was closed yesterday, do you know in which RHOAI/RHODS version this fix will be available? the notebook scale test is failing because of it ...

[zdtsw]

Hello @zdtsw , I see that this issue was closed yesterday, do you know in which RHOAI/RHODS version this fix will be available? the notebook scale test is failing because of it ...

hi, it should be already in the rhods 2.4 RC2 (officially RHODS 2.4)

[kpouget]

ack thanks
... but 2.4 RC2 hasn't been advertised yet AFAICS

[kpouget]

@zdtsw , I launched test with 2.4RC2, but I still see this failure :/

 RHODS 2.4.0-2023-11-15 running on OCP v4.12.44 

[asanzgom]

Reopening the issue as the bug is still reproducible on v2.4 RC2:

https://redhat-internal.slack.com/archives/C05NXTEHLGY/p1700135335630479?thread_ts=1700121604.676009&cid=C05NXTEHLGY

[AjayJagan]

This is now migrated to JIRA: https://issues.redhat.com/browse/RHOAIENG-88. Hence closing it here.