Incorporate Data Resampling and Destruction Policy

[matttproud]

The datastore grows ad infinitum right now. We need a couple of capabilities:

1. The capability of specifying a reduction policy along …
   • an interval of a given size (e.g., one hour),
   • with a reduction method (e.g., mean, median, minimum, maximum), and
   • on samples with a timestamp subject to a certain predicate condition (e.g., older than one day from now).
2. A reduction policy should be specifiable on a …
   • global basis (e.g., a median is OK for most things), and
   • a per-metric basis (e.g., downsample input pertaining to a SLA with the most pessimistic method like a minimum or maximum).

[matttproud-soundcloud]

We effectively have a destruction policy now, which defaults to ten days. Through the curator, it would be possible to implement a resampler easily.

[fabxc]

Retention exists, downsampling is no longer a goal IIRC.
Thoughts?

At least I have a hard time seeing the benefit of downsampling at our level of compression. It could stretch the retention window a bit but the only thing actually solving the underlying problem is distributed remote storage from which Prometheus components can read.

[brian-brazil]

I think we're reasonably agreed on no downsampling, but we may allow more granular configuration of retention.

[fabxc]

That's sufficiently different that a new issue with the respective requirements should be filed.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.