[Backport v2.6] [BUG] Potential panic in keycloakoidc client caused by uncaught error #40554

rancherbot · 2023-02-15T17:06:05Z

This is a backport issue for #38683, automatically created via rancherbot by @anupama2501

Original issue description:

Rancher Server Setup

Rancher version: v2.6.7
Installation option (Docker install/Helm Chart): Docker
Proxy/Cert Details: no proxy, self-sigend cert

Describe the bug
rancher with keycloak auth configured constantly crashes with the following backtrace:

2022/08/22 06:35:05 [INFO] Starting management.cattle.io/v3, Kind=SamlToken controller
E0822 06:35:05.202301      52 runtime.go:79] Observed a panic: runtime.boundsError{x:1, y:1, signed:true, code:0x0} (runtime error: index out of range [1] with length 1)
goroutine 5333 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3f478c0, 0xc005166120})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0009088c0})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:49 +0x75
panic({0x3f478c0, 0xc005166120})
	/usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.getSearchURL({0xc00537af90, 0x4})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:172 +0xe5
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*KeyCloakClient).getFromKeyCloakByID(0xc00260a998, {0xc0019fc6d4, 0x24}, {0xc0019fc6cd, 0xeda95187f}, 0xc006571a40)
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:135 +0xd3
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*keyCloakOIDCProvider).GetPrincipal(_, {_, _}, {{{0xc00260a95a, 0x5}, {0xc0032e3680, 0x17}}, {{0xc00260a9a0, 0xb}, {0xc00260a960, ...}, ...}, ...})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_provider.go:150 +0x2e7
github.com/rancher/rancher/pkg/auth/providers.GetPrincipal({_, _}, {{{0xc00260a95a, 0x5}, {0xc0032e3680, 0x17}}, {{0xc00260a9a0, 0xb}, {0xc00260a960, 0x6}, ...}, ...})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/providers.go:164 +0x10e
github.com/rancher/rancher/pkg/auth/providerrefresh.(*refresher).refreshAttributes(0xc002c7f780, 0xc001274580)
	/go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/refresher.go:256 +0xc08
github.com/rancher/rancher/pkg/auth/providerrefresh.RefreshAttributes(0xc001274580)
	/go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/daemon.go:83 +0x99
github.com/rancher/rancher/pkg/controllers/management/auth.(*UserAttributeController).sync(0xc006fc1c30, {0xc00713feb0, 0x0}, 0x696c6163203a656d)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/auth/user_attribute_handler.go:39 +0x5f
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.(*userAttributeController).AddHandler.func1({0xc00274ee10, 0xa}, {0x418c100, 0xc001274580})
	/go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_user_attribute_controller.go:155 +0x42
github.com/rancher/norman/controller.(*genericController).AddHandler.func1({0xc00274ee10, 0xa}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20220627222520-b74009fac3ff/controller/generic_controller.go:60 +0x191
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc001064860, {0xc00274ee10, 0x40d214}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedcontroller.go:29 +0x38
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000c17e00, {0xc00274ee10, 0xa}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedhandler.go:75 +0x23f
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc000aafc30, {0xc00274ee10, 0xa})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:233 +0x93
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc000aafc30, {0x37c5e40, 0xc0009088c0})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:214 +0x10e
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc000aafc30)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:191 +0x46
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:180
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f630febe2a0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00399f680, {0x4b09600, 0xc0044b6480}, 0x1, 0xc0019d01e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0039a0960, 0x3b9aca00, 0x0, 0xf0, 0xc0039a10e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0xc0039a1680, 0xc0039a1860, 0xc0008fee00)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/rancher/lasso/pkg/controller.(*controller).run
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:148 +0x2c6
panic: runtime error: index out of range [1] with length 1 [recovered]
	panic: runtime error: index out of range [1] with length 1

goroutine 5333 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0009088c0})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:56 +0xd8
panic({0x3f478c0, 0xc005166120})
	/usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.getSearchURL({0xc00537af90, 0x4})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:172 +0xe5
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*KeyCloakClient).getFromKeyCloakByID(0xc00260a998, {0xc0019fc6d4, 0x24}, {0xc0019fc6cd, 0xeda95187f}, 0xc006571a40)
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:135 +0xd3
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*keyCloakOIDCProvider).GetPrincipal(_, {_, _}, {{{0xc00260a95a, 0x5}, {0xc0032e3680, 0x17}}, {{0xc00260a9a0, 0xb}, {0xc00260a960, ...}, ...}, ...})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_provider.go:150 +0x2e7
github.com/rancher/rancher/pkg/auth/providers.GetPrincipal({_, _}, {{{0xc00260a95a, 0x5}, {0xc0032e3680, 0x17}}, {{0xc00260a9a0, 0xb}, {0xc00260a960, 0x6}, ...}, ...})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/providers.go:164 +0x10e
github.com/rancher/rancher/pkg/auth/providerrefresh.(*refresher).refreshAttributes(0xc002c7f780, 0xc001274580)
	/go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/refresher.go:256 +0xc08
github.com/rancher/rancher/pkg/auth/providerrefresh.RefreshAttributes(0xc001274580)
	/go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/daemon.go:83 +0x99
github.com/rancher/rancher/pkg/controllers/management/auth.(*UserAttributeController).sync(0xc006fc1c30, {0xc00713feb0, 0x0}, 0x696c6163203a656d)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/auth/user_attribute_handler.go:39 +0x5f
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.(*userAttributeController).AddHandler.func1({0xc00274ee10, 0xa}, {0x418c100, 0xc001274580})
	/go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_user_attribute_controller.go:155 +0x42
github.com/rancher/norman/controller.(*genericController).AddHandler.func1({0xc00274ee10, 0xa}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20220627222520-b74009fac3ff/controller/generic_controller.go:60 +0x191
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc001064860, {0xc00274ee10, 0x40d214}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedcontroller.go:29 +0x38
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000c17e00, {0xc00274ee10, 0xa}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedhandler.go:75 +0x23f
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc000aafc30, {0xc00274ee10, 0xa})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:233 +0x93
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc000aafc30, {0x37c5e40, 0xc0009088c0})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:214 +0x10e
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc000aafc30)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:191 +0x46
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:180
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f630febe2a0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00399f680, {0x4b09600, 0xc0044b6480}, 0x1, 0xc0019d01e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0039a0960, 0x3b9aca00, 0x0, 0xf0, 0xc0039a10e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0xc0039a1680, 0xc0039a1860, 0xc0008fee00)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/rancher/lasso/pkg/controller.(*controller).run
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:148 +0x2c6

The text was updated successfully, but these errors were encountered:

samjustus · 2023-03-15T16:58:18Z

sure-5506

MbolotSuse · 2023-03-23T14:59:29Z

Validation Template

Root Cause

The keycloak integration logic assumed that there was a /auth/ somewhere in the keycloak url. This lead to a panic when newer versions of keycloak, which use the quarkus distribution and don't include an /auth/ prefix, were integrated with Rancher.

What was fixed, or what change have occurred

The logic to form the keycloak url can now handle cases where there is no /auth/ value in the keycloak url, allowing it to properly form search urls for keycloak versions using both Quarkus and Widlfly.

Areas or cases that should be tested

Basic keycloak integration on both Quarkus (verison 19 and up standard) and Wildfly (-legacy or version 16) distributions, including:

Group search
User search
Permissions granted just to a user
Permissions granted to a group, and inherited by a user in that group

What areas could experience regressions

Keycloak integration for older version (Wildfly based) including:

Group search
User search
Permissions granted just to a user
Permissions granted to a group, and inherited by a user in that group

Are the repro steps accurate/minimal?

Yes, they are included here for convenience:

Start keycloak version 19.0.2
Integrate with rancher, following our docs
Notice that the rancher pods are restarting with panics, and group/user search does not work

anupama2501 · 2023-04-06T18:25:27Z

Verified on v2.6-head a52c574

Created a rancher server on docker install
Enabled keycloak version 1.19 and verified no panics were seen in the rancher logs.
Created a downstream cluster and added a user from keycloak to it and verified the user search works fine
Verified group search works fine too.

anupama2501 · 2023-04-12T10:55:35Z

Verified on v2.6-head 4485a0d

Created a rancher server on docker install
Enabled keycloak version 1.19 and verified no panics were seen in the rancher logs.
Created a downstream cluster and added a user from keycloak to it and verified the user search works fine
Verified group search works fine too.
Added a group as a restricted admin and upon logging by adding the following config for groups
Verified logging in as a user from the group and nested group works fine.

MbolotSuse · 2023-04-21T16:40:44Z

Release Note:

Credit goes to github user @jamhed for developing a fix to this issue.

Rancher attempted to form a search URL for keycloak by splitting based on the pattern of /auth/. Newer keycloak versions, namely those using the Quarkus Distribution, did not include this value, causing a panic to occur when users attempted to integrate keycloak with Rancher. Rancher has been updated to properly form this search url for Quarkus Based distributions.

rancherbot added kind/bug Issues that are defects reported by users or that we know have reached a real release team/area1 labels Feb 15, 2023

rancherbot added this to the v2.6.x milestone Feb 15, 2023

rancherbot assigned anupama2501 Feb 15, 2023

cbron modified the milestones: v2.6.x, 2023-Q2-v2.6.x Feb 22, 2023

samjustus mentioned this issue Feb 27, 2023

[BUG] [REGRESSION] crash (and loop crash) when using keycloakoidc authconfig #40631

Closed

sowmyav27 added the QA/S label Mar 1, 2023

samjustus added the [zube]: Next Up label Mar 13, 2023

samjustus assigned MbolotSuse Mar 13, 2023

samjustus added the JIRA To be used in correspondence with the internal ticketing system. label Mar 15, 2023

MbolotSuse mentioned this issue Mar 22, 2023

Keycloak Issuer url fix 2.6 #40956

Merged

MbolotSuse added the [zube]: To Test label Mar 23, 2023

zube bot removed the [zube]: Next Up label Mar 23, 2023

sowmyav27 added [zube]: QA Next up and removed [zube]: To Test labels Apr 3, 2023

anupama2501 added [zube]: QA Working and removed [zube]: QA Next up labels Apr 6, 2023

anupama2501 closed this as completed Apr 12, 2023

zube bot added [zube]: Done and removed [zube]: QA Working labels Apr 12, 2023

MbolotSuse added the release-note Note this issue in the milestone's release notes label Apr 21, 2023

zube bot removed the [zube]: Done label Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport v2.6] [BUG] Potential panic in keycloakoidc client caused by uncaught error #40554

[Backport v2.6] [BUG] Potential panic in keycloakoidc client caused by uncaught error #40554

rancherbot commented Feb 15, 2023

samjustus commented Mar 15, 2023

MbolotSuse commented Mar 23, 2023

anupama2501 commented Apr 6, 2023

anupama2501 commented Apr 12, 2023

MbolotSuse commented Apr 21, 2023

[Backport v2.6] [BUG] Potential panic in keycloakoidc client caused by uncaught error #40554

[Backport v2.6] [BUG] Potential panic in keycloakoidc client caused by uncaught error #40554

Comments

rancherbot commented Feb 15, 2023

samjustus commented Mar 15, 2023

MbolotSuse commented Mar 23, 2023

Validation Template

Root Cause

What was fixed, or what change have occurred

Areas or cases that should be tested

What areas could experience regressions

Are the repro steps accurate/minimal?

anupama2501 commented Apr 6, 2023

anupama2501 commented Apr 12, 2023

MbolotSuse commented Apr 21, 2023