Kiali working with no Istiod access #5642

josunect · 2022-11-24T15:23:25Z

Proposal as a first version to have Kiali running with no Istiod.

Details in: https://docs.google.com/document/d/1XjkodS5Uu14C_1l3-j7iNaWfLufQFycY-cKMlagIxyc

Changes:

Not perform some calls if there is no access to istiod. They were making the system really slow (When istiod was not available):
- get pod proxy status for workloads
- Get registry services for services
- Fetch registry services
Istio Config: Show a message indicating that Istiod is not available

Some tests to review performance: Frontend tests were showing a degraded performance so a new commit was sent to internally cache the istioAccess value

How this could be tested
Documentation PR: New section kiali working with no istiod kiali.io#613
Operator PR: Add new configuration option istio_api_enabled kiali-operator#595

Fixes #5626

Xunzhuo · 2022-11-24T15:35:38Z

Great! If you want some reviews, ping me after this is ready! @josunect

josunect · 2022-11-24T15:40:15Z

Great! If you want some reviews, ping me after this is ready! @josunect

Thanks, @Xunzhuo ! It would be great! I'll let you know.

josunect · 2022-11-25T15:48:45Z

Changes when istiod is not available:

Istio no available messages

This remains the same. Should we change it to just a warning instead of error?

Service mesh version unknown

Workload status warning because no proxy pod sync

Istio Config not available

Show a message instead

No error requests
There are no error or time out requests.

Running locally with no istiod
I had to add some code in kubernetes/token to run in local environment. But not sure yet what would be the best solution to not have to modify the code all the time.

Probably it would require some more testing but I think it is ready for feedback and reviews

nrfox

Thanks for picking up this investigation!

nrfox · 2022-11-28T19:04:19Z

kubernetes/istio.go

+func (in *K8SClient) HasIstioAccess() bool {
+	componentStatus, err := in.CanConnectToIstiod()
+	if err != nil {
+		return false
+	}
+	if len(componentStatus) == 0 {
+		return false
+	}
+	return true
+}


I think this would be better as a config option for the registry where you can enable/disable the registry altogether if you'd like rather than having it just be a fallback option: https://github.com/kiali/kiali/blob/master/config/config.go#L208.

@hhovsepy may be looking into this but you should be able to get the istio config from the kube api rather than the registry when there's no istiod access.

What will be the impact on the controlplane namespace card once there is no direct istiod access? It might still be possible to get some details, like controlplane revision, about the controlplane from the injection webhook since this may be present in the cluster even if the controlplane is not.

Funny, I was just looking at the same thing. I also think it makes sense to configure Kiali for this scenario. By default Kiali would expect to be able to connect to istiod's debug endpoint and proceed normally. If it can't connect then that would be a FAILURE scenario for the control plane card. If configured to ignore istiod then it would be a DEGRADED scenario for the control plane card (even if it is accessible we won't connect, this could be for performance reasons, or whatever).

draft solution of retrieving configs when istiod is not available #5649

draft solution of retrieving configs when istiod is not available #5649

Thanks, @hhovsepy !

business/istio_validations.go

jshaughn

Good progress @josunect , I think I agree with @nrfox about making this a more explicit configuration option for Kiali (see comments). What do you think?

josunect · 2022-11-29T08:21:47Z

Good progress @josunect , I think I agree with @nrfox about making this a more explicit configuration option for Kiali (see comments). What do you think?

Thanks, @jshaughn and @nrfox for the review! I'll approach the changes and submit.

I've though in do it this way as it can be more robust in the sense that it can work even if Istiod is or is not available, but it is true that it is not possible to know the reason, can be intentional or not - a failure. So it make sense to do it more explicit as part of the configuration.

josunect · 2022-11-29T12:15:33Z

I've also added the following change:

View for CP card if Istio API is disabled:

And enabled:

Xunzhuo

I have reviewed the backend codes, did not find any issues. LGTM. Wait for other reviews in front part :)

josunect · 2022-11-29T13:15:14Z

I have reviewed the backend codes, did not find any issues. LGTM. Wait for other reviews in front part :)

Thank you, @Xunzhuo !

hhovsepy · 2022-11-30T19:52:35Z

Is it supposed, that the case when 'istiod' is not running/accessible and the config "istio_api_enabled: true" will be handled? Because it throws "open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" error message while selecting "istiod" workload.

josunect · 2022-12-01T08:03:45Z

Is it supposed, that the case when 'istiod' is not running/accessible and the config "istio_api_enabled: true" will be handled? Because it throws "open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" error message while selecting "istiod" workload.

In that case, it won't be handled - That should be the default behavior. But, I will reproduce it.

hhovsepy · 2022-12-02T12:58:43Z

Facing a different behavior on OCP cluster, gateway timeouts on all requests:

Many errors in Kiali pod log:
1202 12:56:50.277723 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot list resource "pods" in API group "" at the cluster scope E1202 12:56:50.277759 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot list resource "pods" in API group "" at the cluster scope W1202 12:56:51.683977 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta1.AuthorizationPolicy: authorizationpolicies.security.istio.io is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot list resource "authorizationpolicies" in API group "security.istio.io" at the cluster scope

config/config.go

business/workloads.go

business/istio_validations.go

jmazzitelli · 2022-12-02T14:02:15Z

Many errors in Kiali pod log: `1202 12:56:50.277723 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot list resource "pods" in API group "" at the cluster scope

@hhovsepy What is your deployment.accessible_namespaces set to? Is it ['**'] or a list of namespaces?

That error message says it wants permission to list pods AT THE CLUSTER SCOPE. Which means it expects (for some reason) a ClusterRole. The Kiali SA only gets bound to a ClusterRole when accessible_namespaces is set to ['**']. Otherwise, there are just Roles for each namespace listed in accessible_namespaces.

We never before required ClusterRole access to look at pods. So this error message seems strange. We should avoid this requirement if possible.

hhovsepy · 2022-12-02T17:09:05Z

When setting accessible_namespaces: [ "**" ], Kiali pod fails to get ready, but this is not PR related:
W1202 17:07:02.688572 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot list resource "deployments" in API group "apps" at the cluster scope E1202 17:07:02.688614 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot list resource "deployments" in API group "apps" at the cluster scope W1202 17:07:03.229890 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot list resource "statefulsets" in API group "apps" at the cluster scope E1202 17:07:03.229935 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:istio-system:kiali-service-account" cannot list resource "statefulsets" in API group "apps" at the cluster scope

…bled

…aryMasthead.tsx Co-authored-by: Nick Fox <nick@nrfox.com>

…ConfigOverview.tsx Co-authored-by: Nick Fox <nick@nrfox.com>

josunect · 2023-01-10T15:04:48Z

It will need to test further after the merge. I see some errors like:

2023-01-10T15:01:54Z WRN Error getting proxy status from istiod. Proxy status may be stale. Err: unable to proxy Istiod pods. Make sure your Kubernetes API server has access to the Istio control plane through 8080 port

2023-01-10T15:03:54Z ERR Failed to call Istiod endpoint /debug/syncz error: unable to proxy Istiod pods. Make sure your Kubernetes API server has access to the Istio control plane through 8080 port

frontend/src/components/IstioWizards/ServiceWizard.tsx

Co-authored-by: Nick Fox <nick@nrfox.com>

kubernetes/cache/proxy_status.go

nrfox

LGTM

josunect self-assigned this Nov 25, 2022

josunect added the enhancement This is the preferred way to describe new end-to-end features. label Nov 25, 2022

josunect requested review from jshaughn, nrfox and Xunzhuo November 25, 2022 15:30

josunect marked this pull request as ready for review November 25, 2022 15:50

nrfox reviewed Nov 28, 2022

View reviewed changes

jshaughn reviewed Nov 28, 2022

View reviewed changes

business/istio_validations.go Outdated Show resolved Hide resolved

jshaughn reviewed Nov 28, 2022

View reviewed changes

josunect added requires operator PR It requires update in operator code requires helm chart PR labels Nov 29, 2022

Xunzhuo reviewed Nov 29, 2022

View reviewed changes

josunect force-pushed the issue5626 branch from 30f6589 to 788d017 Compare November 30, 2022 14:33

josunect removed the requires helm chart PR label Nov 30, 2022

josunect mentioned this pull request Nov 30, 2022

Add new configuration option istio_api_enabled kiali/kiali-operator#595

Merged

josunect requested review from jshaughn and nrfox December 1, 2022 18:00

hhovsepy self-requested a review December 2, 2022 10:19

nrfox reviewed Dec 2, 2022

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

business/workloads.go Outdated Show resolved Hide resolved

business/istio_validations.go Outdated Show resolved Hide resolved

josunect and others added 12 commits January 10, 2023 13:15

Remove read only tag in istio config

b62fd81

Use all space for canary upgrade status if Istio API is disabled

6c4a677

Center canary status information, remove indicator proxy status unknown

f704363

Small CP card when canary status is not present and istio API is disa…

2ca8f5a

…bled

Check istio_api_enabled in istio_status

417ce02

Do not fetch registry services when istiod is disabled. (#7)

78ef3cb

Not perform validations if Istio api is disabled

6ae1a35

Add message when Create/edit Istio configs if Istio API is disabled

872b210

Update frontend/src/components/DefaultSecondaryMasthead/DefaultSecond…

414f408

…aryMasthead.tsx Co-authored-by: Nick Fox <nick@nrfox.com>

Update frontend/src/pages/IstioConfigDetails/IstioObjectDetails/Istio…

e1f6f21

…ConfigOverview.tsx Co-authored-by: Nick Fox <nick@nrfox.com>

Apply last changes

915f964

Remove redundant var

9981a08

josunect force-pushed the issue5626 branch from eb03015 to 9981a08 Compare January 10, 2023 13:38

Update CP card if istiod is disabled

35fc1af

Check if istio api is enabled before start polling istiod for cache

9d1e2a4

josunect added the docs It requires changes on kiali.io label Jan 10, 2023

josunect added 2 commits January 10, 2023 15:52

go imports

48b3a08

lint

35428ce

josunect requested a review from nrfox January 11, 2023 08:01

nrfox previously approved these changes Jan 11, 2023

View reviewed changes

frontend/src/components/IstioWizards/ServiceWizard.tsx Outdated Show resolved Hide resolved

Update frontend/src/components/IstioWizards/ServiceWizard.tsx

af64f75

Co-authored-by: Nick Fox <nick@nrfox.com>

josunect dismissed nrfox’s stale review via af64f75 January 12, 2023 08:13

josunect requested a review from nrfox January 12, 2023 10:18

nrfox reviewed Jan 12, 2023

View reviewed changes

kubernetes/cache/proxy_status.go Outdated Show resolved Hide resolved

Move if istio api is enabled to cache

5eec947

nrfox approved these changes Jan 12, 2023

View reviewed changes

josunect merged commit 7e95fbc into kiali:master Jan 13, 2023

josunect deleted the issue5626 branch January 13, 2023 06:57

josunect mentioned this pull request Jan 16, 2023

[CI] Kiali working with No Istiod access test #5722

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kiali working with no Istiod access #5642

Kiali working with no Istiod access #5642

josunect commented Nov 24, 2022 •

edited

Loading

Xunzhuo commented Nov 24, 2022

josunect commented Nov 24, 2022

josunect commented Nov 25, 2022 •

edited

Loading

nrfox left a comment

nrfox Nov 28, 2022

jshaughn Nov 28, 2022

hhovsepy Nov 28, 2022

josunect Nov 29, 2022

jshaughn left a comment

josunect commented Nov 29, 2022

josunect commented Nov 29, 2022 •

edited

Loading

Xunzhuo left a comment

josunect commented Nov 29, 2022

hhovsepy commented Nov 30, 2022

josunect commented Dec 1, 2022

hhovsepy commented Dec 2, 2022

jmazzitelli commented Dec 2, 2022

hhovsepy commented Dec 2, 2022

josunect commented Jan 10, 2023

nrfox left a comment

Kiali working with no Istiod access #5642

Kiali working with no Istiod access #5642

Conversation

josunect commented Nov 24, 2022 • edited Loading

Xunzhuo commented Nov 24, 2022

josunect commented Nov 24, 2022

josunect commented Nov 25, 2022 • edited Loading

Probably it would require some more testing but I think it is ready for feedback and reviews

nrfox left a comment

Choose a reason for hiding this comment

nrfox Nov 28, 2022

Choose a reason for hiding this comment

jshaughn Nov 28, 2022

Choose a reason for hiding this comment

hhovsepy Nov 28, 2022

Choose a reason for hiding this comment

josunect Nov 29, 2022

Choose a reason for hiding this comment

jshaughn left a comment

Choose a reason for hiding this comment

josunect commented Nov 29, 2022

josunect commented Nov 29, 2022 • edited Loading

Xunzhuo left a comment

Choose a reason for hiding this comment

josunect commented Nov 29, 2022

hhovsepy commented Nov 30, 2022

josunect commented Dec 1, 2022

hhovsepy commented Dec 2, 2022

jmazzitelli commented Dec 2, 2022

hhovsepy commented Dec 2, 2022

josunect commented Jan 10, 2023

nrfox left a comment

Choose a reason for hiding this comment

josunect commented Nov 24, 2022 •

edited

Loading

josunect commented Nov 25, 2022 •

edited

Loading

josunect commented Nov 29, 2022 •

edited

Loading