I can't configure my custom prometheus

[winds6206]

Describe the bug
Hi , I install istio in my GKE and everything is fine, but I wanna let my kiali to scrape my custom prometheus so modify default YAML file from istio as below

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali
  namespace: istio-system
  labels:
    helm.sh/chart: kiali-server-1.29.0
    app: kiali
    app.kubernetes.io/name: kiali
    app.kubernetes.io/instance: kiali-server
    version: "v1.29.0"
    app.kubernetes.io/version: "v1.29.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: "kiali"
data:
  config.yaml: |
    auth:
      openid: {}
      openshift:
        client_id_prefix: kiali
      strategy: anonymous
    deployment:
      accessible_namespaces:
      - '**'
      additional_service_yaml: {}
      affinity:
        node: {}
        pod: {}
        pod_anti: {}
      custom_dashboards:
        excludes:
        - ""
        includes:
        - '*'
      hpa:
        api_version: autoscaling/v2beta2
        spec: {}
      image_name: quay.io/kiali/kiali
      image_pull_policy: Always
      image_pull_secrets: []
      image_version: v1.29
      ingress_enabled: false
      logger:
        log_format: text
        log_level: info
        sampler_rate: "1"
        time_field_format: 2006-01-02T15:04:05Z07:00
      namespace: istio-system
      node_selector: {}
      override_ingress_yaml:
        metadata: {}
      pod_annotations:
        sidecar.istio.io/inject: "false"
      pod_labels: {}
      priority_class_name: ""
      replicas: 1
      resources: {}
      secret_name: kiali
      service_annotations: {}
      service_type: ""
      tolerations: []
      version_label: v1.29.0
      view_only_mode: false
    #external_services:
    #  custom_dashboards:
    #    enabled: true
    external_services:
      custom_dashboards:
        prometheus:
          url: http://prometheus.monitoring.svc.cluster.local:9090
        namespace_label: "monitoring"
    identity:
      cert_file: ""
      private_key_file: ""
    istio_namespace: istio-system
    login_token:
      signing_key: CHANGEME
    server:
      metrics_enabled: true
      metrics_port: 9090
      port: 20001
      web_root: /kiali

I modify this block, and deploy

    #external_services:
    #  custom_dashboards:
    #    enabled: true
    external_services:
      custom_dashboards:
        prometheus:
          url: http://prometheus.monitoring.svc.cluster.local:9090
        namespace_label: "monitoring"

when I open kiali UI again, all of about graph are pop up

Cannot load the graph: Post "http://prometheus.istio-system:9090/api/v1/query": dial tcp: lookup prometheus.istio-system on 10.55.240.10:53: no such host

why kiali still scrape default prometheus, I try many method, but I still have no idea...

Versions used
Kiali: 1.29.0
Istio: 1.9.5
Kubernetes flavour and version: GKE 1.18.17-gke.700

Expected behavior

I expect Kiali can scrape my custom prometheus

[lucasponce]

It seems related to your config, I give you an example of personalized one that I use for development:

api:
  namespaces:
    exclude:
      - istio-operator
      - kube.*
      - openshift.*
      - ibm.*
      - kiali-operator
    label_selector: ""
[...]
external_services:
  istio:
    url_service_version: http://localhost:15014/version
  prometheus:
    url: http://localhost:19090
    cache_enabled: true
  tracing:
    enabled: true
    in_cluster_url: http://localhost:16686/jaeger
    url: http://localhost:16686/jaeger
    namespace: istio-system
    port: 443
    service: jaeger-query
    auth:
      insecure_skip_verify: true
      password: [...]
      type: basic
      use_kiali_token: false
      username: internal
  grafana:
    in_cluster_url: http://localhost:13000
    url: http://localhost:13000

Full detailed config doc is under this link https://github.com/kiali/kiali-operator/blob/master/deploy/kiali/kiali_cr.yaml#L423

[jshaughn]

You edited the Prometheus location only for custom dashboard queries but not the Prometheus location for Istio telemetry. Kiali allows you to configure a separate Prom instance for custom dashboards. Also, Kiali does not "scrape" telemetry, Prometheus scrapes the Istio telemetry, Kiali queries that telemetry from Prometheus.

[jmazzitelli]

In other words, you need to define this:
https://github.com/kiali/kiali-operator/blob/v1.29.0/deploy/kiali/kiali_cr.yaml#L516-L547

You need to define external_services.prometheus ... that points to the primary Prometheus that is scraping the Istio telemetry and the main one Kiali needs.

The setting you set was external_services.custom_dashboards.prometheus and that is only to point to a secondary prometheus (in case you have a second one) that is responsible for scraping your applications.