Readiness probe fails with mTLS enabled and "RequireAndVerifyClientCert" clientAuthType

[rzetelskik]

What happened?
Deploying a Prometheus instance with mTLS configured and spec.web.tlsConfig.clientAuthType set to "RequireAndVerifyClientCert" results in the Deployment never reaching readiness. The operator sets the probes behind the authenticated endpoints, which prohibits kubelet from reaching them, since the readiness probes do not support setting client certificates at this point.

$ kubectl logs  pod/prometheus-example-0
...
ts=2023-03-16T18:03:23.514Z caller=stdlib.go:105 level=error component=web caller="http: TLS handshake error from [::1" msg="]:38722: tls: client didn't provide a certificate"

$ kubectl get events
...
17s         Warning   Unhealthy                   pod/prometheus-test-0                           Startup probe failed: Get "https://10.244.0.37:9090/-/ready": remote error: tls: bad certificate

Did you expect to see something different?
The readiness probe should not fail in this scenario. The readiness probe shouldn't be set up behind an authenticated endpoint.

How to reproduce it (as minimally and precisely as possible):
Prerequisites: prometheus-operator deployed in your cluster

1. Create a TLS certificate and key for the server and a CA certificate for client certificate authentication to the server.
2. Create a example-prometheus-client-ca ConfigMap with ca.crt key set to the CA certificate for client authentication.
3. Create a TLS Secret example-prometheus-serving-certs containing the serving certificate and key under keys tls.crt and tls.key correspondingly.
4. Create a minimal Prometheus instance:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: example
spec:
  web:
    tlsConfig:
      cert:
        secret:
          name: "example-prometheus-serving-certs"
          key: "tls.crt"
      keySecret:
        name: "example-prometheus-serving-certs"
        key: "tls.key"
      clientAuthType: "RequireAndVerifyClientCert"
      client_ca:
        configMap:
          name: "example-prometheus-client-ca"
          key: "ca.crt"
  serviceMonitorSelector:
    matchLabels: {}
  ruleSelector:
    matchLabels: {}

Environment

• Prometheus Operator version:
  v0.61.1

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:49:09Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, etc.

• Manifests:
  as above

• Prometheus Operator Logs:
  N/A

Anything else we need to know?:

[JoaoBraveCoding]

I might be wrong but I think that for this we would need prometheus/prometheus#9166

[rzetelskik]

@JoaoBraveCoding thanks, I haven't seen that one. It surely looks like a prerequisite. Do you happen to know an ETA for that issue?