Could not fetch health

[riasadul-hoq]

Hi,
Promethus and Grafana dashboard configured in different namesapce is unable to communicate with kiali dashboard. Hence graph is not loading. Configured cr.yaml file but URL is still pointing to istio-system namespace.

Error: Could not fetch health: Error while fetching app health: Post "http://prometheus.istio-system:9090/api/v1/query": dial tcp: lookup prometheus.istio-system on 10.69.0.10:53: no such host

Any idea?

[jmazzitelli]

(converted to github discussion)

If prometheus is in a different namespace, and you are getting that error message (which indicates Kiali is still trying to find prometheus in istio-system namespace) then your Kiali CR is still not configured correctly.

Please see this and make sure the configuration is correct: https://kiali.io/docs/configuration/p8s-jaeger-grafana/#prometheus-configuration

You did not post your Kiali CR yaml so we cannot confirm whether what you have is correct or not - but one thing is to always check the indentation. We often see Kiali CRs simply misconfigured due to poor/misaligned indentation. To confirm the validity of your Kiali CR, see: https://kiali.io/docs/configuration/kialis.kiali.io/#validating-your-kiali-cr

[riasadul-hoq]

Hi @jmazzitelli
Thanks for responding.
Please find the attached cr file.
kiali-cr.yaml.zip

[jmazzitelli]

Following the docs I linked to above, your CR has a problem:

$ bash <(curl -sL https://raw.githubusercontent.com/kiali/kiali-operator/master/crd-docs/bin/validate-kiali-cr.sh) \
>   -crd https://raw.githubusercontent.com/kiali/kiali-operator/master/crd-docs/crd/kiali.io_kialis.yaml \
>   --kiali-cr-file ./kiali-cr.yaml 
=== SETTINGS ===
KIALI_CRD_LOCATION=https://raw.githubusercontent.com/kiali/kiali-operator/master/crd-docs/crd/kiali.io_kialis.yaml
KIALI_CR_FILE=./kiali-cr.yaml
KIALI_CR_NAME=
NAMESPACE=default
PRINT_CRD=false
=== SETTINGS ===
Using 'oc' located here: /home/jmazzite/bin/oc
...
Validating the CR:
----------
error: error validating "STDIN": error validating data: [ValidationError(TestKiali.spec.external_services.grafana.auth): unknown field 
"component_status" in io.kiali.v1alpha1.TestKiali.spec.external_services.grafana.auth, 
ValidationError(TestKiali.spec.external_services.grafana.dashboards[2].variables): unknown field "var-cluster" in 
io.kiali.v1alpha1.TestKiali.spec.external_services.grafana.dashboards.variables, 
ValidationError(TestKiali.spec.external_services.grafana.dashboards[2].variables): unknown field "var-datasource" in 
io.kiali.v1alpha1.TestKiali.spec.external_services.grafana.dashboards.variables, 
ValidationError(TestKiali.spec.external_services.grafana.dashboards[2].variables): unknown field "var-instance" in 
io.kiali.v1alpha1.TestKiali.spec.external_services.grafana.dashboards.variables, 
ValidationError(TestKiali.spec.external_services.prometheus): unknown field "component_status" in 
io.kiali.v1alpha1.TestKiali.spec.external_services.prometheus, ValidationError(TestKiali.spec.external_services.prometheus): unknown 
field "custom_metrics_url" in io.kiali.v1alpha1.TestKiali.spec.external_services.prometheus, ValidationError(TestKiali.spec): unknown field 
"istio_component_namespaces" in io.kiali.v1alpha1.TestKiali.spec]; if you choose to ignore these errors, turn validation off with --
validate=false
----------
ERROR! Validation failed for Kiali CR [./kiali-cr.yaml]

Now, that said, you did not say what version of Kiali you are using. I am assuming the latest version. If you are using an older version, that would be important to know and would change how to go about trying to fix your problem.

[jmazzitelli]

The spec.external_services.prometheus.url looks correct at least. So something else must be going on.

A couple things to look at:

1. Look at the logs of your kiali pod - see what other errors occurred (particularly at startup) regarding prometheus initialization
2. Look at the Kiali CR status field - that might tell you if the operator had a problem (e.g. kubectl get kiali kiali -n istio-system -o jsonpath='{.status}' - see if there are any error messages
3. Confirm your ConfigMap contains the proper setting. The operator should have done this for you - but its possible someone/something manually modified it... kubectl get cm kiali -n istio-system and look at the external_services.prometheus section of the configmap yaml and confirm it is what you expect. If it is not, then either the operator failed to update it (the Kiali CR status field should tell you if this is the case) or someone manually modified it after the operator installed Kiali. We highly recommend no one manually modify the ConfigMap - the operator should be the only entity modifying the kiali ConfigMap.

[riasadul-hoq]

Hi @jmazzitelli

I am using version 1.49.0.
Can you guide me on how to fix the validation error on cr.yaml file?

In terms of modification I can confirm there has not been any manual modification. Also the installation is standalone kiali installation.

[jmazzitelli]

Based on the error (which would also be the same error on 1.49) your component_status is incorrect. Please see this:

https://kiali.io/docs/configuration/kialis.kiali.io/#.spec.external_services.istio.component_status

Your component status fields should be there. Just read the error message carefully, and look at your CR - specifically the component_status field and other fields mentioned by the error message. You need to put these in the correct place.

That said, I do not think this affects your prometheus error. But you should fix the CR anyway, because it isn't doing what you think the configuration is going to do.

[jmazzitelli]

Here's the 1.49 FAQ that is related that might help: https://v1-49.kiali.io/docs/faq/istio-component-status/

Look at the 1.49 example CR and search for "component_status" to see an example: https://v1-49.kiali.io/docs/configuration/kialis.kiali.io/#example-cr

[riasadul-hoq]

Error from pod

2022-07-08T09:08:55.992586006Z2022-07-08T09:08:55Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup tracing.istio-system on 10.69.0.10:53: no such host"

Received following output
kubectl get cm kiali -n istio-system
NAME DATA AGE
kiali 1 2d

kubectl kiali kiali -n istio-system -o jsonpath='{.spec}'
error: unknown command "kiali" for "kubectl"

[lucasponce]

2022-07-08T09:08:55.992586006Z2022-07-08T09:08:55Z ERR Error fetching availability of the tracing service: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup tracing.istio-system on 10.69.0.10:53: no such host"

This error is related to Jaeger, as it seems it's not installed under the "istio-system" namespace, is that the case?

[riasadul-hoq]

@lucasponce that's correct. We have not installed Jaeger.

[lucasponce]

Would you mind installing it? Or if it's not installed, "disable" it on the Kiali configuration.
You'll miss features, so the recommendation is to have it enabled, but I'd like to see if you saw the same problem.

One thing that happens is the default config, Kiali is expecting to communicate with it? So it's possible that error is triggering what you see.

[riasadul-hoq]

We would prefer not to install it. Are you referring to kiali cr file as Kiali configuration? If yes, we don't have any reference of jaeger in the cr file. I have shared the cr file already, You can have a look.

[lucasponce]

Could you disable it and check if you keep having the same error?
This is the configuration flag you should adjust in your side:
https://kiali.io/docs/configuration/kialis.kiali.io/#.spec.external_services.tracing.enabled

[jmazzitelli]

kubectl kiali kiali -n istio-system -o jsonpath='{.status}'
error: unknown command "kiali" for "kubectl"

I made a mistake in my previous instructions - I forgot the command "get" and you should examine the "status" (not spec) field.

So run this:

kubectl get kiali kiali -n istio-system -o jsonpath='{.status}'

(this assumes the CR is in the namespace istio-system)

[jmazzitelli]

Yes. This is commonly known occurrence and it is because you did not delete the CR first before you attempted to uninstall the operator. This is mentioned specifically in our uninstall documentation here: https://kiali.io/docs/installation/installation-guide/install-with-helm/#known-problem-uninstall-hangs-unable-to-delete-the-kiali-cr ... in there you will read:

| If you fail to delete the Kiali CRs before uninstalling the operator, a proper cleanup may not be done.

Known problem: uninstall hangs (unable to delete the Kiali CR)

Typically this happens if not all Kiali CRs are deleted prior to uninstalling the operator.
To force deletion of a Kiali CR, you need to clear its finalizer. For example:

$ kubectl patch kiali kiali -n istio-system -p '{"metadata":{"finalizers": []}}' --type=merge

| This forces deletion of the Kiali CR and will skip uninstallation of the Kiali Server.
| Remnants of the Kiali Server may still exist in your cluster which you will need to manually remove.

So, long story short - please make sure you follow the documentation when you are installing and uninstalling Kiali (either standalone or with the operator) - it will help you avoid all of these issues you are having.

[jmazzitelli]

I would like to point out another thing - our install docs document the fact that you should uninstall Kiali before trying to install it via another mechanism (i.e. if you are going from a standalone install to an operator install, you should uninstall the standalone first before moving to the operator installation). You can read this here: https://kiali.io/docs/installation/installation-guide/ where it says:

| If you previously installed Kiali via a different mechanism, you must first uninstall Kiali using the original mechanism’s uninstall procedures.

[riasadul-hoq]

Now its looking better

riasadulhoq@ITG-MBP-001783 Downloads % kubectl patch kiali kiali -n istio-system -p '{"metadata":{"finalizers": []}}' --type=merge
kiali.kiali.io/kiali patched
riasadulhoq@ITG-MBP-001783 Downloads % kubectl delete kiali --all --all-namespaces
No resources found
riasadulhoq@ITG-MBP-001783 Downloads %

Can we do any further checks to ensure all instances of kiali has been uninstalled successfully

[jmazzitelli]

Can we do any further checks to ensure all instances of kiali has been uninstalled successfully

You can look for any object in any namespace with the label app.kubernetes.io/name=kiali or app.kubernetes.io/name=kiali-operator. Also look for cluster resources like roles and role bindings with the same labels. Nothing should exist. Something like this will check (all of these should return an empty list):

1. kubectl get all,secrets,sa,configmaps,deployments,roles,rolebindings,ingresses,horizontalpodautoscalers --selector="app.kubernetes.io/name=kiali" --all-namespaces
2. kubectl get all,secrets,sa,configmaps,deployments,roles,rolebindings,ingresses,horizontalpodautoscalers --selector="app.kubernetes.io/name=kiali-operator" --all-namespaces
3. kubectl get clusterroles,clusterrolebindings --selector="app.kubernetes.io/name=kiali"
4. kubectl get clusterroles,clusterrolebindings --selector="app.kubernetes.io/name=kiali-operator"

Finally, because helm does not delete CRDs (this is mentioned in the docs, too) you have to uninstall the CRD as well. So make sure the CRD is deleted via kubectl delete crd kialis.kiali.io

[riasadul-hoq]

All those returns "No resources found" so we are good.
Now I need to sort out the CR file and finally reinstall.

[riasadul-hoq]

I have made a few changes to the CR file.
Can you kindly review?
kiali-cr.yaml.zip

[jmazzitelli]

Did you run the validation tool? Please run that first. Instructions are here: https://kiali.io/docs/configuration/kialis.kiali.io/#validating-your-kiali-cr

[riasadul-hoq]

Yes I did. Not sure how to fix it

=== SETTINGS ===
KIALI_CRD_LOCATION=https://raw.githubusercontent.com/kiali/kiali-operator/master/crd-docs/crd/kiali.io_kialis.yaml
KIALI_CR_FILE=./kiali-cr.yaml
KIALI_CR_NAME=
NAMESPACE=default
PRINT_CRD=false
=== SETTINGS ===
Using 'kubectl' located here: /usr/local/bin/kubectl
...
Validating the CR:
----------
error: error validating "STDIN": error validating data: [ValidationError(TestKiali.spec.external_services.grafana.auth): unknown field "component_status" in io.kiali.v1alpha1.TestKiali.spec.external_services.grafana.auth, ValidationError(TestKiali.spec.external_services.prometheus): unknown field "component_status" in io.kiali.v1alpha1.TestKiali.spec.external_services.prometheus, ValidationError(TestKiali.spec): unknown field "istio_component_namespaces" in io.kiali.v1alpha1.TestKiali.spec]; if you choose to ignore these errors, turn validation off with --validate=false
----------
ERROR! Validation failed for Kiali CR [./kiali-cr.yaml]```

[jmazzitelli]

Reading the error message you posted, I see this:

... spec.external_services.grafana.auth): unknown field "component_status"
... spec.external_services.prometheus): unknown field "component_status"
... spec): unknown field "istio_component_namespaces"

spec.external_services.grafana.auth.component_status, spec.external_services.prometheus.component_status, and spec.istio_component_namespaces are all unknown/invalid settings as you can see in the Kiali CR reference page which is here -- this is the latest Kiali CR reference for our current v1.53 release: https://kiali.io/docs/configuration/kialis.kiali.io/

All of those settings are related to the component status stuff ... I don't know why those are in your CR - those component_status fields you have were removed almost 2 years ago from the Kiali config (and actually, spec.external_services.grafana.auth.component_status has never been valid - there was never a component_status field under the auth field for grafana). They were removed starting at version 1.27 (see kiali/kiali-operator#195) and the istio_component_namespaces has also been obsolete/removed for a long time (see #4109)

The docs have a small features section and an FAQ regarding component status as they are implemented in the more recent releases:

• https://kiali.io/docs/faq/istio-component-status/
• https://kiali.io/docs/features/istio-component-status/

If I were you, I would just remove those settings from your Kiali CR and worry about configuring istio component statuses later. Get the Kiali CR to validate properly and get your Kiali installed successfully so you can see your mesh graph and data. Once that is done, then you can worry about configuring Kiali to check your components' statuses. The docs I link above talk about configuring that.

[riasadul-hoq]

Now when I run the following
kubectl get kiali kiali -n istio-system -o jsonpath='{.status}'
error: the server doesn't have a resource type "kiali"

But kiali has installed successfully

[israel-hdez]

I think you installed kiali-server rather than the kiali-operator, which means that Kiali is already running and you don't need to create a KialiCR.

The operator is what you need to install to manage the Kiali installation via the operator, rather than directly with helm.

[jmazzitelli]

Right. If you do not have a Kiali CR schema (the CRD), then you are not installing what you think you are.

Read the install docs and pick one helm chart to install (the server itself, or the operator) - https://kiali.io/docs/installation/installation-guide/install-with-helm/

You did not tell us what command you used to install - we can only go by what you are describing. If you provide more details on HOW you installed, it might be easier to help. At this point, I don't know what you installed or what version of Kiali you installed. But based on your screenshot, I guess you installed 1.49 (which is fairly old).

I really strongly suggest you read and understand our install docs fully. It looks like you are going in circles.

[riasadul-hoq]

Okay from the top,

I am using the following commands to install standalone kiali
helm upgrade --install --namespace istio-system kiali-server -f kiali-cr.yaml kiali/kiali-server

I am installing the latest version.
Our grafana promethus is installed in the monitoring namespace.

If for standalone installation there is not any CR file then how kiali will point to prom grafana on different namespace.

[israel-hdez]

I do agree with @jmazzitelli that you should read carefully Kiali docs.
Under this section: https://kiali.io/docs/installation/installation-guide/install-with-helm/#standalone-kiali-installation, there is a note:

The kiali-server Helm Chart mirrors all settings of the Kiali CR as chart values that you can configure using regular --set flags.

And then an example is given about how you would configure a setting via Helm when using kiali-standalone. So, for your question:

If for standalone installation there is not any CR file then how kiali will point to prom grafana on different namespace.

It is done via --set flags or via a values file (specified by a -f values-file.yaml argument) on the regular helm install or helm upgrade command.

And BTW, from in the command you provided helm upgrade --install --namespace istio-system kiali-server -f kiali-cr.yaml kiali/kiali-server I may think you are passing a KialiCR file to helm as if it were a Helm values file. It is not the same! So, perhaps that's the first thing to fix is if you are going to use "kiali-standalone".

[riasadul-hoq]

do you have any reference for value.yaml file? So that I can point kiali to different namespace for prom and grafna

[riasadul-hoq]

URL now is being picked up from values.yaml

Now receiving the following error

This is my simple values.yaml file

auth:
  strategy: anonymous

external_services:
  custom_dashboards:
    enabled: true
  prometheus:
      auth:
        namespace: 'monitoring'
      url: http://grafana-prometheus.monitoring:80
  grafana:
      auth:
        username: 'admin'
        password: 'prom-operator'
        namespace: 'monitoring'
      url: http://grafana-prometheus.monitoring:80
      enabled: true

[nrfox]

Looks like both your grafana/prom urls are the same: http://grafana-prometheus.monitoring:80. Is that correct?

[riasadul-hoq]

@nrfox it was just to find out if either of the URL is picked up by kiali dashboard as it was not the case before.

[riasadul-hoq]

@jmazzitelli do you have any idea about my latest issue?

[jmazzitelli]

cannot load graph: client_error - cannot successfully connect to Prometheus
client_error: 401 means unauthorized - missing credentials.

[jmazzitelli]

I am installing the latest version.

I don't see how you are installing the latest version. The latest version of Kiali is 1.53.0 - see https://github.com/kiali/kiali/releases and the helm chart repo here https://kiali.org/helm-charts/index.yaml

Yet, your screenshot shows v1.49.0:

So I still think you are not installing what you think you are installing.

(I hate to harp on the "read the docs" thing - but... did you update your local helm repo as per the Kiali installation docs here: https://kiali.io/docs/installation/installation-guide/install-with-helm/#adding-the-kiali-helm-charts-repository where it says, "If you already added the repository, you may want to update your local cache to fetch latest definitions by running: $ helm repo update")

[riasadul-hoq]

This is an old screenshot.

Have a look at the latest installation:

values.yaml file sorted most of the issues.
Kindly review the latest issue.
#5295 (reply in thread)

[riasadul-hoq]

Sorry trial and error. If either works!

…

On Tue, 19 Jul 2022, 19:07 Nick Fox, ***@***.***> wrote: Looks like both your grafana/prom urls are the same: http://grafana-prometheus.monitoring:80. Is that correct? — Reply to this email directly, view it on GitHub <#5295 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIWJIBS3T56S7O45JEG2URDVU3VHFANCNFSM526CGETQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[riasadul-hoq]

This issue can be closed.