Ruler never unregisters group metrics

[bboreham]

As a new group is created (e.g. on resharding when a ruler starts or stops) it registers metrics with Prometheus, but there is no code to unregister them when the group stops being used (e.g. on another resharding). Over time this will build up a substantial number of useless metrics.

This makes it hard to observe how well the ruler is keeping up, since time()-cortex_prometheus_rule_group_last_evaluation_timestamp_seconds is ever-increasing for the left-behind metrics.

The metric registration is done in Prometheus code; ruler calls Update() with a list of files, but nobody is checking which files have disappeared since last update.

[bboreham]

Upstream issue prometheus/prometheus#6689

[jtlisi]

prometheus/prometheus#6693

Now that this has been merged, we should be able to update our vendored prometheus version to solve this issue.

[bboreham]

NB we had a request to vendor only tagged versions of Prometheus, which would imply waiting for 2.16 to ship.

[pracucci]

NB we had a request to vendor only tagged versions of Prometheus, which would imply waiting for 2.16 to ship.

We had that request, but I feel we haven't talked much about it. I'm personally dubious about it. Forcing the version in master to always match a tagged version of Thanos and Prometheus may significantly slow down the development, especially on experimental features like the blocks storage.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.