Forgetting stale instrument values

[stormshield-fabs]

Is there a recommended way to forget "stale" instruments that have not been updated in a long time? We have a system that collects metrics for several subsystems that can be created and deleted at runtime. We are facing a monotonic increase in the number of samples collected because stale values are collected "forever". This also leads to some incoherencies in aggregate queries.

I know this could be solved using the Delta temporality but this is not an acceptable solution for us due to other constraints (our ingestion pipeline uses Prometheus Remote Write which is not really compatible with the Delta temporality).

Another solution I came up with was to instantiate a provider per subsystem, but this is rather cumbersome since we need to propagate it to the components that use instruments, instead of relying on the global provider.

Related : #955