PodMonitor vs ServiceMonitor what is the difference?

[mandreasik]

Hi, I'm new to prometheus-operator and I do not see difference between ServiceMonitor and PodMonitor.

Documentation describes them as:

ServiceMonitor, which declaratively specifies how groups of services should be monitored. The Operator automatically generates Prometheus scrape configuration based on the definition.

PodMonitor, which declaratively specifies how groups of pods should be monitored. The Operator automatically generates Prometheus scrape configuration based on the definition.

But this does not explain when to use particular one.

I played with them a little bit. Both of them use selector rules to find pods.

So you can either define PodMonitor that search for pods with eg. label app: myAPP, or if your app is behind service that is labeled eg. app:myAPP, you create ServiceMonitor that will scrape metrics from all pods in service with this label. Both of them will produce the same result in prometheus target tab in prometheus UI.

So I would really appreciate if someone will explain to me when one should use PodMonitor and when ServiceMonitor?

[pgier]

I think ServiceMonitor should be the default choice unless you have some reason to use PodMonitor. For example, you may want to scrape a set of pods which all have a certain label which is not consistent between different services.

There is some helpful discussion in the original podmonitor issue #38, but I agree that the docs could be improved here.

[brancz]

The way I see it, ServiceMonitors are the perfect fit if you already have a Service for your pods anyways. However, if it doesn't make sense for other reasons to have a Service for your component, then the PodMonitor is the right choice. It's essentially about avoiding (failure prone) duplicating pod label selectors when possible.

[mandreasik]

@pgier @brancz Thank you for the explanation :) (now it's more clear to me when to use them)

I hope this issue will help other prometheus-operator newbies understand this matter better :)

[winningjai]

@pgier @brancz,
I have a doubt about this.

1. Consider I need to collect metrics from all the pods under a particular service. What I need to prefer PodMonitor or ServiceMonitor.

2. If I used ServiceMonitor, will Prometheus server scrape all the nodes under the service or it will scrape just the service endpoint alone.

Thanks in advance for your time.

[brancz]

It will scrape all pods behind the service, because the Service maintains an Endpoints object. That's where Prometheus discovers the targets from.

[yann-soubeyrand]

Hello,

I should say that I'm a little bit confused too: when talking about ServiceMonitor, I would expect my service to be monitored, not the associated Endpoints (otherwise I would have been looking for an EndpointsMonitor). This would be particularly useful when one has a highly available exporter and wants only one of the exporter's pods to be scraped each time to avoid having duplicate metrics and to avoid uselessly loading the service which the exporter requests.

Without using PrometheusOperator, one would use a kubernetes_sd_config with a role of type service in this case. Prometheus would then scrape the service IP, which would randomly scrape one of the exporter's pods. That's precisely what I thought the ServiceMonitor did in the first place due to its naming (ServiceMonitor let one think that it will be translated to kubernetes_sd_config with service role).

How can we achieve this with the actual ServiceMonitor and PodMonitor? Is it too late to rename the actual ServiceMonitor to EndpointsMonitor and create a “real” ServiceMonitor?

[paulfantom]

@yann-soubeyrand scraping metric endpoints via a load balancer is an antipattern in prometheus. Best practice is to scrape directly and this is why ServiceMonitor is examining all endpoints in a Service and scraping them this way.

[yann-soubeyrand]

@paulfantom so how do you achieve high availability for your exporters? Do you scrape all the replicas of your exporters and then get duplicated metrics and useless load on your applications (elasticsearch or postgresql exporters can be quite demanding on the server they get the metrics from)?

[yann-soubeyrand]

Hi @paulfantom, that's a real question, I'd really like to know what's the right approach for the scraping of highly available exporters if scraping them through a load balancer is an antipattern?

[paulfantom]

Highly available exporter is an anti-pattern in itself and that has a consequence of #3119 (comment). The best practice is to have an exporter near to an instance that is being monitored.

However, if you need to do this then additionalScrapeConfigs can be helpful

[yann-soubeyrand]

Thanks for your answer @paulfantom. In case one cannot have the exporter near the instance being monitored (like a PostgreSQL instance on AWS RDS), what's the best practice for the deployment of the exporter? Should one deploy a single instance of it and use a PodMonitor (and accept that there can be holes in the metrics when the exporter gets unavailable, which can happen for various reasons like a k8s node being drained in response to cluster scale-in)?

[paulfantom]

In such case, you need to deploy at least 2 instances and deduplicate data on prometheus level.