Aggregations: Add serial differencing aggregation

[polyfractal]

This is still a WIP, just putting it up for discussion. We may want to roll this functionality into a different Agg.

Serial Differencing

Serial differencing (or just differencing) is a technique where values in a time series are subtracted from itself at different time lags or periods. For example, the datapoint f(x) = f(x_t) - f(x_t-n), where n is the period being used.

A period of 1 is equivalent to a derivative: it is simply the change from one point to the next. Single periods are useful for removing constant, linear trends.

Single periods are also useful for transforming data into a stationary series. In this example, the Dow Jones is plotted over ~250 days. The raw data is not stationary, which would make it difficult to use with some techniques.

But once we plot the first-difference, it becomes a stationary series (we know this because the first difference is randomly distributed around zero, and doesn't seem to exhibit any pattern/behavior). The transformation reveals that the dataset is a random-walk model, which allows us to use further analysis.

Larger periods can be used to remove seasonal / cyclic behavior. In this example, a population of lemmings was synthetically generated with a sine wave + constant linear trend + random noise. The sine wave has a period of 30 days.

The first-difference removes the constant trend, leaving just a sine wave. The 30th-difference is then applied to the first-difference to remove the cyclic behavior, leaving a stationary series which is amenable to other analysis.

API

{
   "aggs": {
      "my_date_histo": {
         "date_histogram": {
            "field": "timestamp",
            "interval": "day"
         },
         "aggs": {
            "the_sum": {
               "sum": {
                  "field": "lemmings"
               }
            },
            "first_difference": {
               "diff": {
                  "bucketsPath": "the_sum",
                  "periods" : 1
               }
            },
            "thirtieth_difference": {
               "diff": {
                  "bucketsPath": "first_difference",
                  "periods" : 30
               }
            }
         }
      }
   }
}

TODO

• Tests :)
• Javadocs, cleanup, etc
• How does this interact with Derivative? The first difference is technically a derivative. We could roll this behavior into Deriv, but if Deriv ever gets time normalization this will get weird. We could also tell people to just use Diff with first-period, but I quite like having a single, simple Deriv agg

[polyfractal]

Just a note: I think I want to rename periods parameter to lags. More standardized naming, and I think a bit more descriptive.

[polyfractal]

Just a note: I think I want to rename periods parameter to lags. More standardized naming, and I think a bit more descriptive.

[colings86]

+1 to renaming periods to lags, but what is the affect of having multiple lags? or is that not the reason why it's plural? We can still only get one output per bucket right?

[colings86]

+1 to renaming periods to lags, but what is the affect of having multiple lags? or is that not the reason why it's plural? We can still only get one output per bucket right?

[polyfractal]

Oh, good point. It should be lag. I think supporting multiple lags would be very confusing and unnecessary.

[polyfractal]

Oh, good point. It should be lag. I think supporting multiple lags would be very confusing and unnecessary.

[colings86]

Agreed, we should not try to support multiple lags in one reducer

[colings86]

Agreed, we should not try to support multiple lags in one reducer

[polyfractal]

Closing since this is against an outdated branch (feature/aggs_2_0)

[polyfractal]

Closing since this is against an outdated branch (feature/aggs_2_0)

[kingaj]

any java example if have then please give me a link please

[kingaj]

any java example if have then please give me a link please