Add ability to track series churn

[brancz]

Proposal

Use case. Why is this important?

In order to identify targets producing high series-churn it would be good to track the number of time-series appearing/disappearing across scrapes of a target. I haven't looked into an implementation too much, but I think this is relatively close to the staleness tracking, so we could use those existing mechanisms to produce this metric.

I'd propose a metric called something along the lines of:

prometheus_target_scrape_series_created_total

This metric would basically be increased whenever a sample of a scrape of a target causes a new time-series to be created. Essentially a more detailed prometheus_tsdb_head_series_created_total.

@brian-brazil @krasi-georgiev @simonpasquier

cc @mxinden @squat @s-urbaniak

[brian-brazil]

Per-target information doesn't belong on /metrics, anything like that would need to be one of the scrape_ gauges.

If piggy backing on existing logic, this will underestimate as it'll missing data around failed scraped.

[brancz]

Can you be more specific what you mean by "scrape_ gauges"?

If piggy backing on existing logic, this will underestimate as it'll missing data around failed scraped.

Yes that's what I could tell from the code as well, I think an estimation is better than nothing and while it seems a little more complicated I think it might also be possible to get the exact value.

[brian-brazil]

Things like scrape_samples_scraped.

[brancz]

Got it, that makes sense and works just as well. Just for my own understanding and for anyone else reading this, the "report metrics" are written directly to tsdb upon scrapes of a target. This is more suitable in this case, because this is actually information of the target and therefore does not belong in the metrics exposed by Prometheus itself. Right?

[brian-brazil]

Yes.