How to add multiple ServiceMonitor to PrometheusOperator object

[gr8Adakron]

What did you do?
I have configured prometheus operator with 2 service monitors:
My service monitors files

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gateway-service-monitor
  labels:
    app: frontend
spec:
  namespaceSelector:
    matchNames:
    - abcd-efg
  selector:
    matchLabels:
      prometheus_operator_service_monitor: allow
  endpoints:
  - targetPort: 8877
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hijk-lmno
  labels:
    app: backend
spec:
  namespaceSelector:
    matchNames:
    - ambassador-log
  selector:
    matchLabels:
      prometheus_operator_service_monitor: allow
  endpoints:
  - targetPort: 80

Now I am refering the service monitor in the prometheus-main-object (Note: below YAML contain just issue-relevant field)

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus  
spec:
  serviceMonitorSelector:
    matchLabels:
      app: frontend

How do I add app: backend as well to the main object, I try doing like this:

spec:
  serviceMonitorSelector:
    matchLabels:
      app: frontend
      app: backend

But this didnt work as matchLabels works as AND operator.

Any help is appreaciated. THanks

[paulfantom]

But this didnt work as matchLabels works as AND operator.

Yes, that is how label matches work in kubernetes.

Solution is to use another label in ServiceMonitors (we most commonly use prometheus: <name_of_prometheus_resource>). This way you have a label dedicated for this purpose and you have a clear notion of which prometheus is scraping this ServiceMonitor (there can be multiple prometheus servers governed by prometheus-operator).

[gr8Adakron]

Thanks for the response @paulfantom

there can be multiple prometheus servers governed by prometheus-operator

Let me know if I am wrong but this means creating two prometheus-main object with each concentrating on different service monitor.

For example:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-frontend
spec:
  serviceMonitorSelector:
    matchLabels:
      app: frontend
...
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-backend
spec:
  serviceMonitorSelector:
    matchLabels:
      app: backend
...

[paulfantom]

No. Solution is to use different labels, adding another Prometheus object will spin up another set of prometheus servers (unless this is what you want).

Use one selector in Prometheus object, like:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-backend
spec:
  serviceMonitorSelector:
    matchLabels:
      prometheus: application-monitoring
...

and use prometheus: application-monitoring label in all ServiceMonitors which need to be handled by that prometheus.

Additionally, if you have only one Prometheus server and wish for this server to collect metrics from all ServiceMonitors in the cluster, then just use {} as serviceMonitorSelector. Like in https://github.com/thaum-xyz/ankhmorpork/blob/master/apps/monitoring/manifests/prometheus/prometheus.yaml#L64 (Feel free to poke around the repository to check a working configuration)

[stale]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[stale]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.

[wajika]

@paulfantom Hello brother, I would like to ask a question, how to determine whether the data is collected after the ServiceMonitor is configured? I can't search on prometheus /graph

[beingnishas]

@gr8Adakron : Did you make it work? I am in a same situation, I tried:

serviceMonitorSelector:
  matchExpressions:
    - key: this-key-should-exist-name
      operator: Exists

[wajika]

@gr8Adakron : Did you make it work? I am in a same situation, I tried:

serviceMonitorSelector:
  matchExpressions:
    - key: this-key-should-exist-name
      operator: Exists

Sorry, I haven't followed up on github recently, my issue has been resolved, thank you.

[EarthlingDavey]

The only way to have multiple targets 'ORed' is to use matchExpressions with the In operator. And both of the expressions must use the same lable key. e.g release.

To answer the original question you could have

serviceMonitorSelector:
  matchExpressions:
    - key: app
      operator: In
      values: 
        - frontend
        - backend

Or, if you're running kube-prometheus-stack and you want to keep using the default service monitors then you must embrace the label key of release and change your app labels from app: frontend & app: backend to release: frontend-app & release: backend-app and set your selector like so:

serviceMonitorSelector:
  # Default value:
  # matchLabels:
  #   release: kube-prometheus-stack
  matchExpressions:
    - key: release
      operator: In
      values: 
        - kube-prometheus-stack
        - frontend-app
        - backend-app