Aggregations: add serial differencing pipeline aggregation

[polyfractal]

No need for assignment or review yet, still need to write tests!

Serial Differencing

Serial differencing (or just differencing) is a technique where values in a time series are subtracted from itself at different time lags or periods. For example, the datapoint f(x) = f(x_t) - f(x_t-n), where n is the period being used.

A period of 1 is equivalent to a derivative: it is simply the change from one point to the next. Single periods are useful for removing constant, linear trends.

Single periods are also useful for transforming data into a stationary series. In this example, the Dow Jones is plotted over ~250 days. The raw data is not stationary, which would make it difficult to use with some techniques.

But once we plot the first-difference, it becomes a stationary series (we know this because the first difference is randomly distributed around zero, and doesn't seem to exhibit any pattern/behavior). The transformation reveals that the dataset is a random-walk model, which allows us to use further analysis.

Larger periods can be used to remove seasonal / cyclic behavior. In this example, a population of lemmings was synthetically generated with a sine wave + constant linear trend + random noise. The sine wave has a period of 30 days.

The first-difference removes the constant trend, leaving just a sine wave. The 30th-difference is then applied to the first-difference to remove the cyclic behavior, leaving a stationary series which is amenable to other analysis.

(Old PR and comments: #10190)

API

{
   "aggs": {
      "my_date_histo": {
         "date_histogram": {
            "field": "timestamp",
            "interval": "day"
         },
         "aggs": {
            "the_sum": {
               "sum": {
                  "field": "lemmings"
               }
            },
            "first_difference": {
               "serial_diff": {
                  "buckets_path": "the_sum",
                  "lag" : 1
               }
            },
            "thirtieth_difference": {
               "serial_diff": {
                  "buckets_path": "first_difference",
                  "lag" : 30
               }
            }
         }
      }
   }
}

[polyfractal]

@colings86 Low priority, but this is up for review whenever you have a few spare minutes. It is blissfully simple compared to moving_avg :)

Open question: currently, if there is not enough data (or the lag is too large), you just don't get any serial_diff metric values. We could also throw an exception, but that seems like poor behavior (the rest of your aggs may work fine). Thoughts?

[colings86]

Could use PipelineAggregator.AGGREGATION_TRANFORM_FUNCTION instead of this?

[colings86]

@polyfractal left some comment but I really like this aggregation and the documentation for it is great :)

To your question: I am struggling to decide what is best. As you say, throwing an exception seems unfriendly and would be different behaviour to other aggregations. But equally if we just don't output anything then it can easily confuse users as to why the aggregation is not working since there will be no message or indication anywhere of what caused the aggregation to not output any data.

[polyfractal]

@colings86 cleaned up, ready at your leisure :)

[colings86]

LGTM