Add ability to set a limit of the number of targets per scrape config

[lilic]

Proposal

Original proposal idea: Add ability to set a limit of total series/samples per scrape config. This would override any sample_limit set per scrape config target and set the limit of series instead proportional for each scrape target. This would be an optional field, so would be compatible with any current configuration. (Also exposing this value as a metric to potentially alert on this with the scrape_samples_scraped metric, might be a good idea.).

New proposal idea: Add ability to set a limit of the number of targets per scrape config. This would work together with the sample_limit per scrape config target and would provide a way to capacity plan better.

Feedback or alternative ideas welcome! :)

Use case. Why is this important?
Due to the dynamic nature of service discovery, it’s practically impossible to do accurate capacity planning, having an overall limit regardless of the number of scrape targets would be very helpful from an admin perceptive to prevent any accidental bad actor by limiting the number of ingested series.

[paulfantom]

Possibly a duplicate of #2137

[brian-brazil]

This would only really make sense when targets within a scrape config are homogeneous, otherwise you have to set it so high to allow for a smaller number of massive targets that it'd not be of much use. Given that such a thing is likely niche I can see this causing more outages and confusion than cases where it'd help.

(Also exposing this value as a metric to potentially alert on this with the scrape_samples_scraped metric, might be a good idea.).

Let's not add more samples to be ingested on every single ingestion. The fact that up is 0 would be enough.

Possibly a duplicate of #2137

This builds on that.

[lilic]

@brian-brazil thanks for the feedback!

Given that such a thing is likely niche I can see this causing more outages and confusion than cases where it'd help.

Curious about the outage part, can you elaborate a bit more on that, thanks! Our actual use case would be in prometheus-operator, where we have homogeneous targets, and I sure there are other users who do it in the same way.

Let's not add more samples to be ingested on every single ingestion. The fact that up is 0 would be enough.

So that would already signal we stopped scrapping, I would like an alert that is warning saying you are about to hit your total limit of series/samples per target. But maybe this is an additional use case, more of a informational one, but goes hand in hand with the capacity planning aspect (you inform users that they should do something before the scrapping fails.)

I am trying to somehow limit the series/samples a scrape config can produce from an admin perspective, maybe limiting the number of targets would be another option?

[brian-brazil]

Curious about the outage part, can you elaborate a bit more on that, thanks!

Both false positives and negatives could result in an outage, and taking out an entire scrape_config may not be what users expect/realise is the only way this could work.

But maybe this is an additional use case, more of a informational one, but goes hand in hand with the capacity planning aspect

You already have the target labels on scrape_samples_scraped to work off.

maybe limiting the number of targets would be another option?

That also came to mind, less math for users. In reality that's what the original proposal is limiting.

[lilic]

You already have the target labels on scrape_samples_scraped to work off.

That would work, thanks! We can expose the limits in prom-operator as well and create alert based on the labels, thanks!

That also came to mind, less math for users. In reality that's what the original proposal is limiting.

Yes, my thinking as well here. Would this be something that Prometheus would consider having? Should I go ahead and update this issue here or close and open a new one?

[brian-brazil]

I see no reason for a new issue. The idea seems sane, the question for me is mainly if users would end up using it and not realising that the only way such a thing could sanely work is basically disabling entire scrape configs at once.

[roidelapluie]

It seems that would bring flapping scrapes that would be hard to debug.

Job total limit: 4000
target1: t=1s 1000 t=5s 800 t=9s [1000] t=13s 1000 t=17s 1000 t=21s [1100]
target2: t=2s 1000 t=6s 1000 t=10s 1000 1=14s 1100 t=18s 1000 t=22s 2000
target3: t=3s 1000 t=7s 1000 t=11s 1000 t=15s 900 t=19s [1100] t=23s [9000]
target4: t=4s [1100] t=8s 1100 t=12s [1100] t=16s 1000 t=20s 2000 t=24s 1000

[lilic]

@roidelapluie are you referring to introduction of the limit of total series/sample or the new suggestion of limiting the number of targets?

[brian-brazil]

We already have that with sample_limit, and if your limit is that close to the actual value then you're already asking for trouble. It's a emergency relief valve, not a quota mechanism.

[lilic]

The idea seems sane, the question for me is mainly if users would end up using it and not realising that the only way such a thing could sanely work is basically disabling entire scrape configs at once.

We would be interested in using this in prometheus-operator, but I am not sure if one use case is enough here?

[brian-brazil]

I wouldn't consider the operator itself as a use case, as it's your users who would have the use case.

My concern would be more if this ends up as an attractive nuisance, as using this correctly would be non-trivial - though fairly simple for the operator.

[brancz]

I recall from the last prometheus team meeting, that there seemed to be consensus on having this, or some version of this ("this" being some mechanism to limit amount of targets).