rolling functions, rolling aggregates, sliding window, moving average

[jangorecki]

To gather requirements in single place and refresh ~4 years old discussions creating this issue to cover rolling functions feature (also known as rolling aggregates, sliding window or moving average/moving aggregates).

rolling functions

• rollmean
• rollsum
• rollmin (more rolling functions #5682)
• rollmax (rolling functions: adaptive left, frollmax, frollapply adaptive, partial #5441)
• rollmedian (rolling median #5692)
• rollprod (more rolling functions #5682)
• rollsd
• rollvar
• rollapply (user provided FUN) (rewritten for more features in frollapply rewritten, supports by.column=F #5575)

features

• multiple columns at once
• multiple windows at once
• multiple columns and multiple windows at once
• atomic vectors input and single window returns atomic vectors
• various length list of vectors
• align: left/center/right
• handling NAs
• handling Inf (rolling functions: adaptive left, frollmax, frollapply adaptive, partial #5441)
• fill constant
• long vector support
• partial window support (frollmean partial argument #4968, PR rolling functions: adaptive left, frollmax, frollapply adaptive, partial #5441)
• adaptive window support
• use openmp to parallelize calculation of multiple columns/windows
• rounding error correction
• timing in verbose mode from parallel region (blocked by testing output using patterns disabled when parentheses present #3422, print uint64 using inttypes.h PRIu64 instead of llu #3423)
• partial & adaptive support (rolling functions: adaptive left, frollmax, frollapply adaptive, partial #5441)
• give.names argument, same as shift has (rolling functions: adaptive left, frollmax, frollapply adaptive, partial #5441)
• frollapply by.column=FALSE (issue add by.column=F argument in frollapply #4887, PR frollapply rewritten, supports by.column=F #5575)
• unevenly spaced time series (rolling funs / shift could support logical window #3241, PR adapt uneven time series rolling window #5576)

[jangorecki]

@mattdowle answering questions from PR

Why are we doing this inside data.table? Why are we integrating it instead of contributing to existing packages and using them from data.table?

1. There were 3 different issues created asking for that functionality in data.table. Also multiple SO questions tagged data.table. Users expects that to be in scope of data.table.
2. data.table fits perfectly for time-series data and rolling aggregates are pretty useful statistic there.

my guess is it comes down to syntax (features only possible or convenient if built into data.table; e.g. inside [...] and optimized) and building data.table internals into the rolling function at C level; e.g. froll* should be aware and use data.table indices and key. If so, more specifics on that are needed; e.g. a simple short example.

For me personally it is about speed and lack of chain of dependencies, nowadays not easy to achieve.
Key/indices could be useful for frollmin/frollmax, but it is unlikely that user will create index on measure variable. It is unlikely that user will make index on measure variable, also we haven't made this optimization for min/max yet. I don't see much sense for GForce optimization because allocated memory is not released after roll* call but returned as answer (as opposed to non-rolling mean, sum, etc.).

If there is no convincing argument for integrating, then we should contribute to the other packages instead.

I listed some above, if you are not convinced I recommend you to fill a question to data.table users, ask on twitter, etc. to check response. This feature was long time requested and by many users. If response won't convince you then you can close this issue.

[jangorecki]

An example of how vectorized x/n arguments can impact performance.
AdrianAntico/AutoQuant@d837071#r34769837
less loops, code easier to read, much faster. Code using frollmean in a loop vs passing lists/vectors to frollmean, result 10x-36x speedup.

[jangorecki]

frollapply ready: #3600

    ### fun             mean     sum  median
    # rollfun          8.815   5.151  60.175
    # zoo::rollapply  34.373  27.837  88.552
    # zoo::roll[fun]   0.215   0.185      NA
    # frollapply       5.404   1.419  56.475
    # froll[fun]       0.003   0.002      NA

[jangorecki]

rollcor
rollcov
rollrank
rollunqn
rolllm

went out of scope as of current moment. All can work using frollapply (not master branch but PRs), just not super fast. We could consider adding them to scope in future. For the current moment the following set of sum mean prod min max sd var median feels fine and complete to me.