[Rollup] Support for filters in rollup job to rollup counts of documents for unstructured text queries into metrics

[mikeh-elastic]

One possible way to rollup unstructured text could be to rollup the document counts for predefined filter aggregations on the data so that dashboards could leverage this rolled up filter count for long term analytics without requiring to store the original raw data.

This would allow better support for rollup log aggregation use cases where parsing of the logs is not complete and the message (or analogous) field is used to drive analytics via filters rather than terms aggregations on structured fields.

[elasticmachine]

Pinging @elastic/es-search-aggs

[wchaparro]

With the 8.7 release of Elasticsearch, we have made a new downsampling capability associated with the new time series datastreams functionality generally available (GA). This capability was in tech preview in ILM since 8.5. Downsampling provides a method to reduce the footprint of your time series data by storing it at reduced granularity. The downsampling process rolls up documents within a fixed time interval into a single summary document. Each summary document includes statistical representations of the original data: the min, max, sum, value_count, and average for each metric. Data stream time series dimensions are stored unchanged.

Downsampling is superior to rollup because:

• Downsampled indices are searched through the _search API
• It is possible to query multiple downsampled indices together with raw data indices
• The pre-aggregation is based on the metrics and time series definitions in the index mapping so very little configuration is required (i.e. much easier to add new time serieses)
• Downsampling is managed as an action in ILM
• It is possible to downsample a downsampled index, and reduce granularity as the index ages
• The performance of the pre-aggregation process is superior in downsampling, as it builds on the time_series index mode infrastructure

Because of the introduction of this new capability, we are deprecating the rollups functionality, which never left the Tech Preview/Experimental status, in favor of downsampling and thus we are closing this issue. We encourage you to migrate your solution to downsampling and take advantage of the new TSDB functionality.