[Feature request] config parameter (or option) to limit max acceptable samples/size/lines in single scrape

[KevinAMurray]

Proposal

A config parameter, possibly global or even as command line option, but ideally for each target that specifies the maximum size (time series, data or lines would do) that is acceptable for a single scrape. If the target exceeds that value, the scrape is ignored. (e.g. if ContentLength is greater than a certain value, then the scrape is treated as failed.)

We have had OOM issues with scraping that have been caused by the target returning "too much" data. To have a bit more resilence it would be good to have prometheus "reject" metric where the volume returned is too large. (Whilst for some targets the time taken to generate the data causes a timeout which has the same effect, that is not always the case, especially if the timeout is deliberately set to a large value).

Occasionally we can identify the issue via alerts on scrape_samples_scraped, once the problem has been tripped, prometheus gets into a crash loop.

[brian-brazil]

You're looking for sample_limit.

[KevinAMurray]

Doh! Apologies -- never spotted that!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.