Metrics Service setup only exposes metrics from a single control plane instance

[tobiasgiese]

Hi 👋

I’ve just tested the metrics exposure for Kamaji. Since we’re not using prometheus-operator (yet), I had to adapt the configuration to use classic Prometheus scrape configs — that part is all good.

However, while doing so I noticed an issue:
Kamaji creates a Service of type ClusterIP and configures scraping from it. This results in Prometheus only scraping metrics from one control plane instance, rather than all of them.

This is a problem because:

• While the Kubernetes API is stateless, the metrics are not aggregated between control planes.
• Scraping through a ClusterIP service results in Prometheus hitting just one of the backing Pods.

Suggested improvements:

• Use a headless Service instead.
• For classic Prometheus setups, leverage the dns_sd_configs feature to discover all Pod endpoints behind the headless Service.
• For prometheus-operator, this might be solvable with endpoints or podMonitor, but I currently can’t test this.

Maybe related: prometheus-operator/prometheus-operator#1899

[prometherion]

I think the best option here would be to provide an additional headless Service for metrics.

Maybe toggable to avoid the sprawl of non required resources.

[tobiasgiese]

Maybe it makes also sense to automateically add the ServiceMonitor/PodMonitor as well - toggable of course.
In this case the kamaji controller will create all the necessary resources to enable Prometheus fetch the metrics from the kamaji control planes.

[prometherion]

I'm a bit worried about importing the Prometheus Operator code base just for their API: that could lead to a Go replacement directives nightmare.

I need to test this; the more automation we put in place, the better.

[bsctl]

@tobiasgiese, I was able to scrape from different clusters using a Service Monitor and ClusterIP service for each tenant control plane. But I've used the Prometheus Operator.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: sample
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2025-04-19T14:51:50Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    cluster.x-k8s.io/cluster-name: sample
    release: kube-prometheus-stack
  name: sample-service-monitor
  namespace: default
  resourceVersion: "8008354"
  uid: 7d7373f0-0b85-4870-9bd3-9a755431dfc9
spec:
  endpoints:
  - interval: 15s
    metricRelabelings:
    - action: drop
      regex: apiserver_request_duration_seconds_bucket;(0.15|0.2|0.3|0.35|0.4|0.45|0.6|0.7|0.8|0.9|1.25|1.5|1.75|2|3|3.5|4|4.5|6|7|8|9|15|25|40|50)
      sourceLabels:
      - __name__
      - le
    path: /metrics
    port: kube-apiserver-metrics
    relabelings:
    - action: replace
      replacement: sample
      targetLabel: cluster
    - action: replace
      replacement: apiserver
      targetLabel: job
    scheme: https
    scrapeTimeout: 10s
    tlsConfig:
      cert:
        secret:
          key: apiserver-kubelet-client.crt
          name: sample-api-server-kubelet-client-certificate
      insecureSkipVerify: true
      keySecret:
        key: apiserver-kubelet-client.key
        name: sample-api-server-kubelet-client-certificate
  - interval: 15s
    path: /metrics
    port: kube-controller-manager-metrics
    relabelings:
    - action: replace
      replacement: sample
      targetLabel: cluster
    - action: replace
      replacement: kube-controller-manager
      targetLabel: job
    scheme: https
    scrapeTimeout: 10s
    tlsConfig:
      cert:
        secret:
          key: apiserver-kubelet-client.crt
          name: sample-api-server-kubelet-client-certificate
      insecureSkipVerify: true
      keySecret:
        key: apiserver-kubelet-client.key
        name: sample-api-server-kubelet-client-certificate
  - interval: 15s
    path: /metrics
    port: kube-scheduler-metrics
    relabelings:
    - action: replace
      replacement: sample
      targetLabel: cluster
    - action: replace
      replacement: kube-scheduler
      targetLabel: job
    scheme: https
    scrapeTimeout: 10s
    tlsConfig:
      cert:
        secret:
          key: apiserver-kubelet-client.crt
          name: sample-api-server-kubelet-client-certificate
      insecureSkipVerify: true
      keySecret:
        key: apiserver-kubelet-client.key
        name: sample-api-server-kubelet-client-certificate
  selector:
    matchLabels:
      kamaji.clastix.io/name: sample-metrics

and

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: sample
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2025-04-19T14:51:50Z"
  labels:
    app.kubernetes.io/managed-by: Helm
    cluster.x-k8s.io/cluster-name: sample
    kamaji.clastix.io/name: sample-metrics
  name: sample-metrics
  namespace: default
  resourceVersion: "8008303"
  uid: a4ec6a06-60d8-48e6-9cdb-d0b4ab6552f5
spec:
  clusterIP: 10.96.226.232
  clusterIPs:
  - 10.96.226.232
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: kube-apiserver-metrics
    port: 6443
    protocol: TCP
    targetPort: 6443
  - name: kube-controller-manager-metrics
    port: 10257
    protocol: TCP
    targetPort: 10257
  - name: kube-scheduler-metrics
    port: 10259
    protocol: TCP
    targetPort: 10259
  selector:
    kamaji.clastix.io/name: sample
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

[bsctl]

I would suggest to keep the configuration of metric service and related Service/Pod monitors outside Kamaji since there are many flavours of doing the job, e.g. traditional Prometheus, Prometheus Operator, and others. What we can do is toggle the creation of the metric service and the creation of a dedicated certificate to scrape the metrics instead to use the <tcp_name>-api-server-kubelet-client-certificate secret.

[tobiasgiese]

I was able to scrape from different clusters using a Service Monitor and ClusterIP service for each tenant control plane.

@bsctl this isn’t about scraping metrics from multiple Kamaji clusters, but rather about scraping all control plane Pods within a single tenant cluster.

With a ClusterIP Service, Prometheus only scrapes one of the Pods behind it. Since metrics aren’t aggregated between control plane instances, this means we’re missing data from the others.

Using a headless Service would allow Prometheus to discover and scrape each Pod individually via DNS, giving a complete view of all control planes.

Hope that helps explain the issue — and thanks for digging into it! 🙂

[tobiasgiese]

I'm a bit worried about importing the Prometheus Operator code base just for their API: that could lead to a Go replacement directives nightmare.

@prometherion this was just an idea earlier, but after thinking it through again, I’d actually recommend not adding Prometheus Operator CRs to the Kamaji controllers. We should avoid coupling the controller logic to external CRDs — especially ones we don’t own. Depending on the Prometheus Operator APIs would be a maintenance headache and something we should steer clear of.

[bsctl]

@tobiasgiese As per our conversation on slack, Service Monitors are used by Prometheus to discovery the endpoints of a service, i.e. the pods, so Prometheus can access directly to the endpoints, i.e. the pods, also called "targets" and it is not passing through the iptables, so there is no round-robin. I'm going to close this issue.

[tobiasgiese]

Sorry for the delay @bsctl. I could also verify that without ServiceMonitors we only have a single Endpoint which leads to round-robin connections and no full metrics collection.

Maybe it's worth adding this information somewhere? Or do you think it's okay to not mention it as we describe how to use metrics with ServiceMonitors?

[Sea-you]

What you could also do, if you want to see the relevant metrics in the Tenant cluster after setting up the ServiceMonitor as above, is the following:

In your Tenant Prometheus, set the externalLabels as:

    externalLabels:
      cluster: sample

If you use kube-prometheus-stack, disable relevant ServiceMonitors, you won't need them:

kubeApiServer:
  enabled: false

kubeControllerManager:
  enabled: false

kubeEtcd:
  enabled: false

kubeScheduler:
  enabled: false

Set your Prometheus to remote_read from the Prometheus of the management cluster:

    # When reading from the Management Prometheus, the externalLabels are added automatically
    # to the query.
    remoteRead:
    - url: https://prometheus-management.tld/api/v1/read
      readRecent: true
      filterExternalLabels: true # the default setting

Now you'll be able to see metrics, alert on them, etc, from the Tenant cluster.