Not able to configure Thanos with Istio

[prashantcfc]

What happened:

I've a setup where I've Thanos Querier deployed alongside Istio which talks to Thanos sidecar that sits alongside prometheus deployed using kube-prometheus-stack.

However, communication between Thanos Querier and Thanos sidecar fails due to Istio in between.

Is deployment of Thanos supported with Istio (to communicate with thanos sidecar) ?

Full logs to relevant components:

upstream connect error or disconnect/reset before headers. reset reason: connection termination

Thanos: v0.31.0
Prometheus: v2.40

[philgladman]

@prashantcfc we are experiencing the same issue. Any luck?

[prashantcfc]

@prashantcfc we are experiencing the same issue. Any luck?

Yes, I was able to get stuff working.

[philgladman]

@prashantcfc we are experiencing the same issue. Any luck?

Yes, I was able to get stuff working.

How were you able to get it to work, any advice?

I have gotten it to work intermittently by setting istio mtls to permissive. However, what is weird for us is that it will take almost 30 minutes after restarting the thanos-query pod for the sidecar endpoint to finally get registered/connect. We also see a bunch of these TLS errors coming out of the istio-proxy side car of the thanos-query pod when trying to connect to the thanos-sidecar. These errors continue even after the thanos-sidecar gets registered.

"TLS_error:|:SSL_routines:OPENSSL_internal:WRONG_VERSION_NUMBER:TLS_error_end:TLS_error_end"

[philgladman]

We were able to fix this by adding an istio DestinationRule. Found this from an istio issue here.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: thanos-query-to-thanos-sidecar
  namespace: monitoring
spec:
  host: monitoring-monitoring-kube-thanos-discovery.monitoring.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE