[Feature Request] Allow retention config per scrape job

[d-shi]

I searched existing issues and didn't see this anywhere. We would like to keep our metrics scraped from a dev kubernetes cluster for only maybe a day or a few days, since pod churn there is huge (we create upwards of 50k new pods per day, each with hundreds of time series associated with them). However, we would like to keep metrics from production clusters for much longer. We want to use the same Prometheus instance to hold all the metrics, and this instance has different scrape jobs set up for the different clusters. It would be nice to be able to configure a different retention policy per scrape job, similar to scrape_interval.

[brian-brazil]

In this case you want two different Prometheus servers. In general dev and prod should have distinct monitoring setups, and usually you'd also want a Prometheus per k8 cluster.

[d-shi]

In our current state we would like to not have to run multiple prometheus servers. Perhaps eventually we will split into multiple servers, but for now we would like to have all our metrics in one place. There are likely many other use cases where having different retentions would be helpful. Can you comment on whether or not you would consider implementing this?

[brian-brazil]

Dupe of #1381, don't expect it anytime soon. The recommended architecture is as I described above.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.