How to decrease ServiceMonitor reload interval?

[SoerenHenning]

We observed that after adding a ServiceMonitor to Kubernetes, it takes some time before Prometheus lists a new target in its configuration (up to 1-2 minutes in my observation).

While this seems to be reasonable for production deployments, we deploy short-living ServiceMonitors as part of our benchmarking tool Theodolite, where we would like to get metrics as soon as possible to decrease the overall benchmarking time.

Is it possible to decrease the reload interval in the Prometheus kind resource?

[paulfantom]

ServiceMonitors are registered by the operator as soon as they are available (operator is using list watchers for the objects). After the operator picks up ServiceMonitor it regenerates config and puts it as a Secret. Next, config-reloader is registering change to the Secret and sends a request to reload prometheus. After this prometheus does its own service discovery and starts scraping targets (bear in mind that Pod needs first to be in a ready state for prometheus to start scraping it). It is highly unlikely you can get this whole process to happen in times shorter than 1 minute, especially when the k8s API server is under load.

However, if you want to investigate more, the potential areas of performance improvements would be as follows:

1. reducing time between adding ServiceMonitor to cluster and registering it by the operator
2. reducing time in which config-reloader notices new Secret and reloads prometheus
3. speeding up prometheus service discovery mechanism

---

IMHO if you need to have targets present as soon as possible, you might be better of with pre-defining them by using additionalScrapeConfig option.

[SoerenHenning]

Thank you for the answer. It helped a lot in understanding how the chain of communication works. I was now able to find out that the delay is mainly caused by the config-reloader container. From looking at the logs, I can see that Prometheus Operator immediately writes a sync prometheus message, while it takes up to 2-3 minutes until the config-reloader container of my Prometheus pod prints a Reload triggered message.

I now found that in prometheus-config-reloader/main.go a watch-interval can be configured, which seems to be set to 3 minutes per default. Unfortunately, I did not found any documentation on this option. Could reducing this interval help here or does it configure something different? Is it possible to configure it from my Prometheus object?

[simonpasquier]

Most of the delay is due to the fact that the updated secret isn't propagated immediately by the kubelet. The config reloader uses fsnotify under the hood so once the secret is updated in the container, the container detects it.

[SoerenHenning]

Okay, so that means a shorter watch-interval would have no effect?

[simonpasquier]

correct