Consider per-scrape sample limit as abuse/overload mitigation

[brian-brazil]

Currently if a client does something silly and starts returning large numbers of samples Prometheus will take it, and will ultimately fall over or get throttled if the throughput is too high.

While we're unlikely to ever fully solve this, and I think it'd be unwise to go down the rabbit hole of trying to get it perfect I think there are simple things that are practical to do.

I propose that we add a per-scrape_config option to limit the number of samples returned by individual targets in that scrape_config. This would not be relabelable, and on a target exceeding this none of the scraped samples would be ingested (to avoid weirdness from partial ingestion) and up would be set to 0.

This would be a very blunt tool, but should catch the worst problems caused by a sudden increase in time series count. The disadvantage is that the user has to maintain the limit, but I think some level of additional administration is inevitable if a user chooses to use features in this area.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.