[prometheus-kube-stack] Prometheus not scraping Istio mTLS, advice please.

[DanoThom]

Describe the bug a clear and concise description of what the bug is.

EDIT:

I think this issue may be related to what I'm experiencing:
istio/istio#21402
and maybe
#533

I'm seeing these 3 prometheus annotations on my workloads, but not seeing the prometheus.io/scheme: https annotation which seems to be crucial:

So what is responsible for adding that annotation that I seem to be missing? Also do I need to enhance my custom istio scrape configs by adding custom relablings? The Istio docs don't seem to indicate so (which is where I got my info from):
https://istio.io/latest/docs/ops/integrations/prometheus/#option-2-customized-scraping-configurations

Ok, so on my journey into getting Prometheus working with Istio using STRICT mutual TLS, here is where I'm up to:

• I've managed to get the promstack deployed successfully with an istio sidecar correctly mounting the Istio certs:
  

• I've also managed to get the Istio additional scrape config to work as I can now see the endpoints showing up in the Service Discovery:

However, I know something is not working 100% because I can't see any of the istio related queries in the graph query dropdown i.e 'istio_request_totals' etc and because the Istio Targets aren't working with the error 'server gave HTTP response to HTTPS client':

I've also tried ServiceMonitors and PodMonitors instead of the custom scrapers but same result. I feel like I'm pretty close to getting this working but after perusing what must be 100+ Github issues and other related posts I'm a little stuck and in need of community advice.

Do I need to be changing a value or annotation somewhere in the prometheus operator or CR spec to fix this http -> https issue? Any advice would be welcome on the issue!

Here are my relevant files:
prometheus-stack-values.txt
istio.custom.operator_ha.txt

What's your helm version?

version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"19bca9208454ad7ac76f6fc71b5a2520922e395b", GitTreeState:"clean", BuildDate:"2022-03-10T18:21:44Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

prometheus-community/kube-prometheus-stack

What's the chart version?

34.1.1

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

prometheusOperator:
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
admissionWebhooks:
patch:
podAnnotations:
sidecar.istio.io/inject: "false"

grafana:
adminPassword: strongpassword
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"

prometheus:
serviceMonitor:
scheme: "https"
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true
prometheusSpec:
logLevel: info
additionalScrapeConfigs:
- job_name: 'istiod'
scheme: https
tls_config:
ca_file: /etc/istio-output-certs/root-cert.pem
key_file: /etc/istio-output-certs/key.pem
cert_file: /etc/istio-output-certs/cert-chain.pem
insecure_skip_verify: true # Prometheus does not support Istio security naming, thus skip verifying target pod ceritifcate
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- istio-system
relabel_configs:
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: istiod;http-monitoring
- job_name: 'envoy-stats'
scheme: https
tls_config:
ca_file: /etc/istio-output-certs/root-cert.pem
key_file: /etc/istio-output-certs/key.pem
cert_file: /etc/istio-output-certs/cert-chain.pem
insecure_skip_verify: true # Prometheus does not support Istio security naming, thus skip verifying target pod ceritifcate
metrics_path: /stats/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: '.*-envoy-prom'
alertingEndpoints:
- name: promstack-prometheus-operator-alertmanager
namespace: default
port: web
scheme: https
pathPrefix: /
apiVersion: v2
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true
volumeMounts:
- name: istio-certs-dir
mountPath: /etc/istio-output-certs/
readOnly: true
volumes:
- name: istio-certs-dir
emptyDir:
medium: Memory
podMetadata:
annotations:
sidecar.istio.io/inject: "true"
sidecar.istio.io/rewriteAppHTTPProbers: "true"
traffic.sidecar.istio.io/includeInboundPorts: "" # do not intercept any inbound ports
traffic.sidecar.istio.io/includeOutboundIPRanges: "" # do not intercept any outbound traffic
proxy.istio.io/config: | # configure an env variable OUTPUT_CERTS to write certificates to the given folder
proxyMetadata:
OUTPUT_CERTS: /etc/istio-output-certs
sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs-dir", "mountPath": "/etc/istio-output-certs"}]' # mount the shared volume at sidecar proxy
labels:
sidecar.istio.io/inject: "true"
# securityContext:
# runAsGroup: 1337
# runAsNonRoot: true
# runAsUser: 1337
# fsGroup: 1337

alertmanager:
alertmanagerSpec:
podMetadata:
annotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"

serviceMonitor:
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true

kube-state-metrics:
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"

Enter the command that you execute and failing/misfunctioning.

helm install promstack prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace --values prometheus-stack-values.yaml

Anything else we need to know?

No response

[tomasAlabes]

@DanoThom were you able to find a solution for this? I'm facing the same.

[snuggie12]

I think you just have it backwards. You need to setup https for your application metrics which have istio sidecars. The metrics ports for istiod and envoy themselves don't run mTLS.

[snuggie12]

Here are my two setups for non-application metrics. I'll note I added stuff to the sidecar scraper and it may still be incomplete (like possibly harden it to ensure only the istio-proxy container is scraped?)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-1-10-6
  namespace: istio-system
spec:
  endpoints:
  - path: /metrics
    port: http-monitoring
  selector:
    matchLabels:
      app: istiod
      istio.io/rev: 1-10-6

$ kg prometheus -n istio-system istio-sidecar-scraper -o yaml | yq e '.spec.additionalScrapeConfigs'
key: scrape-config.yaml
name: prometheus-additional-scrape-configs

$ kg secrets -n istio-system prometheus-additional-scrape-configs -o yaml | yq e '.data."scrape-config.yaml"' | base64 -D
- job_name: 'istio-envoy-stats'
  kubernetes_sd_configs:
    - role: pod
  metrics_path: /stats/prometheus
  relabel_configs:
    - action: keep
      regex: '.*-envoy-prom'
      source_labels: [__meta_kubernetes_pod_container_port_name]
    - source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      target_label: namespace
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_pod_name]
      separator: ;
      regex: (.*)
      target_label: pod
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_pod_container_name]
      separator: ;
      regex: (.*)
      target_label: container
      replacement: $1
      action: replace

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.