Reloading Prometheus 2.0 has significant memory spikes

[fabxc]

Since every scraper has a rather large cache now for all staleness handling and usage of AddFast a reload causes severe memory spikes of up to 2x.

On reload we terminate all scrapers as the configuration might have changed. In 2.0, each scraper carries a rather large cache around for staleness handling and usage of AddFast. Those caches are dropped and get re-created for all the "new" targets. The same happens with the known buffer sizes for each scraper.

We could investigate not purging all scrapers and instead try to hot-reload them as well where necessary. This needs some very careful consideration though.

@brian-brazil

[siavashs]

I noticed that the number of open files are doubled during reload (kept open ~5 minutes)

^ markers: start, reload

[julianvmodesto]

Looks like when deploying Prometheus Operator, my Pod is failing to start because the ConfigMap reloader will cause memory spikes that will hit my memory requests / limits and prevent my Pod from becoming Ready due to this issue.