Incorrect ServiceAccount used for Multi-cluster when using OpenID without RBAC

[BeryJu]

Hi, I've noticed that in my setup since some time the graphs didn't work and would always just get an "Unauthorised" log message from kiali.

After quite a bit of digging I found this:

• It worked with v1.65.0 and stopped working with v1.66.0
• I'm not using the new script to setup Multicluster but rather I'm using the istiod remote cluster ServiceAccount/Secrets, with an additional ClusterRoleBinding that gives it the same permissions (I tested with the new multicluster setup too and got the same result)
• Taking the Kubeconfig from that secret works correctly
• On the remote cluster there are errors in the kube-apiserver logs complaining about an invalid bearer token, which after quite some testing turns out to be the Kiali local ServiceAccount token
• Switching to anonymous authentication makes all these issues go away and the graph works as expected

I'm using Kiali in a multi-primary multi-cluster istio setup, configured with OpenID authentication with disable_rbac: true set. From my understanding of the docs here https://kiali.io/docs/configuration/authentication/openid/#set-up-with-no-namespace-access-control-support, with this flag disabled, the multi-cluster graphs should behave the same as with anonymous authentication, or am I missing something?

Excuse the lack of more details in the descriptions, I forgot to take notes while debugging this, let me know if anything is unclear

Testing with a local build, this diff seems to fix the issue, however there's probably a reason the current logic works the way it does

diff --git a/kubernetes/client_factory.go b/kubernetes/client_factory.go
index a6b21f942..edafed37b 100644
--- a/kubernetes/client_factory.go
+++ b/kubernetes/client_factory.go
@@ -207,7 +207,8 @@ func (cf *clientFactory) newClient(authInfo *api.AuthInfo, expirationTime time.D
                        var remoteConfig *rest.Config
                        var err2 error
                        // In auth strategy should we use SA token
-                       if cfg.Auth.Strategy == kialiConfig.AuthStrategyAnonymous {
+                       if cfg.Auth.Strategy == kialiConfig.AuthStrategyAnonymous ||
+                               (cfg.Auth.Strategy == kialiConfig.AuthStrategyOpenId && cfg.Auth.OpenId.DisableRBAC) {
                                remoteConfig, err2 = GetConfigForRemoteClusterInfo(clusterInfo[cluster])
                        } else {
                                remoteConfig, err2 = GetConfigWithTokenForRemoteCluster(clusterInfo[cluster].Cluster,

[jmazzitelli]

Just briefly looking at the proposed fix, that seems correct.

@nrfox you are familiar with this stuff - what do you think?

[jmazzitelli]

I'm using Kiali in a multi-primary

I also know most of the recent testing done for the Kiali multi-cluster support has been with primary-remote mode. There is going to be work in the future to ensure things work better in multi-primary -- this might be one of those things.

[jmazzitelli]

I was able to see a similar error when I did the following:

1. Install test primary-remote setup on two minikube clusters:

./hack/istio/multicluster/install-primary-remote.sh

2. Edit the Kiali ConfigMap in east cluster kubectl edit cm -n istio-system --context=east kiali and add the disable_rbac: true to auth.strategy.openid section.
3. Restart the kiali pod to pick up the new change: kubectl delete pod -n istio-system -l app=kiali
4. Start the Kiali UI found on the east cluster: xdg-open https://$(minikube ip --profile east)/kiali/console
5. Go to the Graph page, select "bookinfo" namespace.

I get this

2023-06-29T17:27:54Z ERR Unauthorized: goroutine 1712 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.20.5/x64/src/runtime/debug/stack.go:24 +0x65
github.com/kiali/kiali/handlers.handlePanic({0x230dac0, 0xc000cd6750})
	/home/runner/work/kiali/kiali/handlers/graph.go:86 +0x219
panic({0x1c12740, 0xc0015b8900})
	/opt/hostedtoolcache/go/1.20.5/x64/src/runtime/panic.go:884 +0x213
github.com/kiali/kiali/graph.CheckError(...)
	/home/runner/work/kiali/kiali/graph/util.go:38
github.com/kiali/kiali/graph/telemetry/istio/appender.getServiceList({0xc000e63e7c, 0x4}, {0xc0003589d4, 0x8}, 0xc000b12720)
	/home/runner/work/kiali/kiali/graph/telemetry/istio/appender/appender.go:291 +0x314
github.com/kiali/kiali/graph/telemetry/istio/appender.getServiceLists(0x8?, {0xc0003589d4, 0x8}, 0xc0006ac120?)
	/home/runner/work/kiali/kiali/graph/telemetry/istio/appender/appender.go:264 +0x33e
github.com/kiali/kiali/graph/telemetry/istio/appender.IstioAppender.AppendGraph({0x2011804?}, 0xc000b12780, 0xc000efc240?, 0xc000cd7980)
	/home/runner/work/kiali/kiali/graph/telemetry/istio/appender/istio_details.go:46 +0x65
github.com/kiali/kiali/graph/telemetry/istio.BuildNamespacesTrafficMap({_, _}, {0xc000b120f0, {0x0, {0xc00016d920, 0x6, 0x6}}, 0x0, 0x1, 0xc0006bff50, ...}, ...)
	/home/runner/work/kiali/kiali/graph/telemetry/istio/istio.go:80 +0x6a3
github.com/kiali/kiali/graph/api.graphNamespacesIstio({_, _}, _, _, {{0x201717d, 0x9}, {0x2011804, 0x5}, {{0xc000358939, 0x15}, ...}, ...})
	/home/runner/work/kiali/kiali/graph/api/api.go:52 +0x126
github.com/kiali/kiali/graph/api.GraphNamespaces({_, _}, _, {{0x201717d, 0x9}, {0x2011804, 0x5}, {{0xc000358939, 0x15}, {0xdf8475800, ...}}, ...})
	/home/runner/work/kiali/kiali/graph/api/api.go:33 +0x2fb
github.com/kiali/kiali/handlers.GraphNamespaces({0x230dac0, 0xc000cd6750}, 0xc0008ecd00)
	/home/runner/work/kiali/kiali/handlers/graph.go:51 +0x165
net/http.HandlerFunc.ServeHTTP(0x201f4c8?, {0x230dac0?, 0xc000cd6750?}, 0x7f5002683808?)
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/routing.metricHandler.func1({0x231c670?, 0xc000256230}, 0xc000485301?)
	/home/runner/work/kiali/kiali/routing/router.go:145 +0x13b
net/http.HandlerFunc.ServeHTTP(0x231daa8?, {0x231c670?, 0xc000256230?}, 0xc000485370?)
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/handlers.AuthenticationHandler.Handle.func1({0x231c670, 0xc000256230}, 0xc0008ec900)
	/home/runner/work/kiali/kiali/handlers/authentication.go:79 +0x3fd
net/http.HandlerFunc.ServeHTTP(0x231daa8?, {0x231c670?, 0xc000256230?}, 0x22f1e10?)
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/server.plainHttpMiddleware.func1({0x231c670?, 0xc000256230?}, 0xc0006bf770?)
	/home/runner/work/kiali/kiali/server/server.go:154 +0x65
net/http.HandlerFunc.ServeHTTP(0xc0008ec800?, {0x231c670?, 0xc000256230?}, 0x2?)
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000f00, {0x231c670, 0xc000256230}, 0xc0008ec700)
	/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x231c520, 0xc000b901c0}, 0x0?)
	/home/runner/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:336 +0x24e
net/http.HandlerFunc.ServeHTTP(0xc000bb2e80?, {0x231c520?, 0xc000b901c0?}, 0x40d92a?)
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc0003588e0?, {0x231c520, 0xc000b901c0}, 0xc0008ec700)
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2500 +0x149
net/http.serverHandler.ServeHTTP({0xc0016060f0?}, {0x231c520, 0xc000b901c0}, 0xc0008ec700)
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc0007b7b00, {0x231daa8, 0xc000bb84b0})
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
	/opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:3089 +0x5ed

[BeryJu]

I forgot to add it to the initial post, but that's the same stacktrace I was getting, The other error I was getting was along the lines of "Error getting namespaces from cluster ...: Unauthorized"

[nrfox]

@BeryJu this is a bug and thanks for taking the time to investigate and identify a fix for this. The patch you posted looks fine to me since openid auth strategy with rbac disabled uses the kiali service account when communicating with the kube API the same way that anonymous mode does. We should even be able to just return the existing kiali SA client here rather than creating a new one. Would you like to open a PR for this change?

[herzcthu]

I'm having similar issue with token strategy. All resources from remote cluster showing unauthorized error.

[nrfox]

@herzcthu the token auth strategy is not supported for multi-cluster since service accounts are scoped to a specific cluster. Only anonymous and openid are currently supported: https://kiali.io/docs/configuration/multi-cluster/#requirements.

[herzcthu]

Thanks for pointing out. I misread the documentation. Now I changed it to anonymous and working well. Will change to openid later.