Kiali cannot get prometheus data when prometheus enable https

[clyang82]

From kiali document:
prometheus_service_url The URL used to access and query the Prometheus Server. It must be accessible from Kiali pod. (default is http://prometheus.istio-system:9090)

When I configured prometheus_service_url=https://prometheus.istio-system:9090, the kiali cannot collect prometheus data and there has exception thrown from kiali pod

E0117 09:47:28.726655       1 graph.go:853] Get https://prometheus.istio-system:9090/api/v1/query?query=round%28sum%28rate%28istio_requests_total%7Breporter%3D%22destination%22%2Csource_workload%3D%22unknown%22%2Cdestination_service_namespace%3D%22cert-manager%22%2Cresponse_code%3D~%22%5B2345%5D%5B0-9%5D%5B0-9%5D%22%7D+%5B60s%5D%29%29+by+%28source_workload_namespace%2Csource_workload%2Csource_app%2Csource_version%2Cdestination_service_namespace%2Cdestination_service_name%2Cdestination_workload%2Cdestination_app%2Cdestination_version%2Cresponse_code%29%2C0.001%29&time=2019-01-17T09%3A47%3A28Z: x509: certificate signed by unknown authority: goroutine 21 [running]:
runtime/debug.Stack(0xc42063a280, 0x26e, 0xc42069c3a0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
github.com/kiali/kiali/handlers.handlePanic(0x192a100, 0xc4206e8b60)
	/root/go/src/github.com/kiali/kiali/handlers/graph.go:853 +0xa6
panic(0x1005f20, 0xc42069c3a0)
	/usr/local/go/src/runtime/panic.go:491 +0x283
github.com/kiali/kiali/handlers.checkError(0x1913d80, 0xc42026c900)
	/root/go/src/github.com/kiali/kiali/handlers/graph.go:836 +0xaa
github.com/kiali/kiali/handlers.promQuery(0xc42056a900, 0x17a, 0x0, 0xed3d24630, 0x19a1460, 0x1920d80, 0xc420223e10, 0x0, 0x0, 0x0)
	/root/go/src/github.com/kiali/kiali/handlers/graph.go:822 +0x468
github.com/kiali/kiali/handlers.buildNamespaceTrafficMap(0xc420678e10, 0xc, 0xc420234d80, 0xc4203b2a80, 0x6, 0x8, 0xdf8475800, 0x0, 0xc420121000, 0xb, ...)
	/root/go/src/github.com/kiali/kiali/handlers/graph.go:166 +0x2be
github.com/kiali/kiali/handlers.graphNamespaces(0xc420234d80, 0xc4203b2a80, 0x6, 0x8, 0xdf8475800, 0x0, 0xc420121000, 0xb, 0x10, 0x0, ...)
	/root/go/src/github.com/kiali/kiali/handlers/graph.go:86 +0x38d
github.com/kiali/kiali/handlers.graphNamespace(0x192a100, 0xc4206e8b60, 0xc420120700, 0xc4208db530)
	/root/go/src/github.com/kiali/kiali/handlers/graph.go:70 +0xb1

Can we support this feature?

[lucasponce]

cc @jmazzitelli @jotak

[jmazzitelli]

probably has to do with a self-signed cert that kiali would need to accept. We'd need to look at this more closely - we have to make sure we don't blindly accept certs that do not have a valid root CA.

If that is not the case (if the prometheus cert has a valid root CA) then it may mean kiali has to be told to trust the root CA.

Would need more information about what certificate is served by the Prometheus https endpoint.

[clyang82]

My case is Prometheus enables https with a self-signed cert.

[jmazzitelli]

OK, that might be a problem. I'm not sure we should allow kiali to blindly accept certs without a valid root CA via the InsecureSkipVerify: true setting.

There is talk about getting internal network communications within the Istio mesh to be secured (this conversation was going on in the istio community). Hopefully, those discussions will help address something like this (that is, you won't have to have your own self-signed cert in front of Prometheus, you can install it with the Citadel-generated cert which presumably will let kiali access it).

I would be curious to know what the community thinks of having kiali turn on InsecureSkipVerify for communications to the prometheus server over https.

[pilhuhn]

@clyang82 How do you set up Istio telemetry to report to that Prometheus with your cert?
I think Kiali can go the same route then.

I Istio-mTLS is set up, there are DestinationRules in place that disable mTLS for certain source-destination pairs. This may also be a way to go here.

[aljesusg]

Consider that kiali, jaeger, prometheus and grafana(if it is enabled) should be with the same CA if we want to communicate with these services

[clyang82]

@clyang82 How do you set up Istio telemetry to report to that Prometheus with your cert?
I think Kiali can go the same route then.

I Istio-mTLS is set up, there are DestinationRules in place that disable mTLS for certain source-destination pairs. This may also be a way to go here.

@pilhuhn I am using the external prometheus to collect istio components metrics so that the kiali fetches the data from this external prometheus.

[clyang82]

any news on this? Thanks.

[ghost]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jotak]

For information, there's now a JIRA ticket to track this issue: https://issues.jboss.org/browse/KIALI-2960

My 2 cts: We should by default reject any untrusted secure connection (like what we do today), however we can have a config flag to explicitly allow untrusted certificate. So an admin would have to knowingly configure it.

[israel-hdez]

@clyang82 I'm closing this issue because #1112 has been merge and Kiali can now accept self-signed certificates. Fix will be available when version 0.21 is released.

[clyang82]

Thanks @israel-hdez and @jotak

[mindcrime]

Hey all, I'm using Kiali with Prometheus running on https, where the cert is signed by our internal CA, and seeing this issue despite setting external_services.prometheus.insecure_skip_verify as mentioned in #1112

This is with Kiali 1.22.1, which I would have thought was recent enough to have this change included, based on the date that 1112 was merged. Can anybody comment on this? Do I need a newer version of Kiali, or might something else be going on?

[lucasponce]

@mindcrime just to confirm, following the settings described in https://github.com/kiali/kiali-operator/blob/master/deploy/kiali/kiali_cr.yaml#L457 is not enough for the version you comment ?

[mindcrime]

@lucasponce - thanks, I think you pointed me to the answer. I had the insecure_skip_verify setting directly under the "prometheus" key, but I see here:

https://github.com/kiali/kiali-operator/blob/master/deploy/kiali/kiali_cr.yaml#L602

that it belongs under the "auth" key under "prometheus". I was going off the verbiage from the #1112 PR which mentioned the key being "external_services.prometheus.insecure_skip_verify" (no "auth" in there).

I'll update my config to move that setting and I'm guessing that will fix the problem.

Edit:

Yes, that was the issue. It's working now, thanks for the pointer to that sample config.