Expire stale exported metrics

[mmckeen]

Is your feature request related to a problem? Please describe.

For advanced metrics with high-cardinality (pod-level, port, IP, etc..) many metrics can become stale and not change for a long period of time. These metrics end up remaining exported to Prometheus and take up a significant amount of resources being scraped. By default, prometheus will mark metrics as stale after 5 minutes of not changing anyways.

We should remove these metrics from the exported set if they have not changed in a significant period of time to save significant resource overhead on serving stale metrics.

Describe the solution you'd like

Stale metrics after some configurable time period (say 15 minutes by default) are no longer exported to Prometheus for advanced metrics.

Describe alternatives you've considered

N/A

Additional context

N/A