CursorSeries implementation

[buybackoff]

WIP

This issue is to track (re)implementation of CursorSeries. The new approach makes all CursorSeries implementations structs or sealed classes and allows for cursor type specialization, which often improves performance significantly.

Unary Transformations

• Unary Arithmetic Tested and benchmarked vs sealed class impl (struct is much faster, Constrained calls to IComparable<T> instead of virtual calls IComparer<T> #100, CursorSeries as structs #113). TODO Overloads for ContainerSeries

• Unary comparison (LT, LE, EQ, NEQ, GE, GT). Very simple, but not tested. TODO Overloads for ContainerSeries

• Range Used as a value of Window.

• MapValues

• Fill

• Filter

• Repeat, RepeatWithKey

• Cast - should be OK to have just an extension that uses Map.

• OfType - should be OK to have just an extension that uses Filter.

• Window<...>(int count, bool allowIncomplete), Window<TKey,...>(TKey width, Lookup lookup) Each window is a range constructed from a cloned lagged cursor. TODO Window with while predicate and accumulator function with a limit. Consider adding an option for keys at the start of each window.

• Window<TKey,...>(TKey width, Lookup.GT)] has a bug - it expands to max possible, the intent to get minimum possible range but GT. Same with LT/LE - it returns a single element, but the intent is max possible but LT/LE than width.

• Lag, ZipLag is implemented as Lag().Map() composition in extension methods. (TODO A simple fast lag method without current point, probably Shift is proper name and support both directions. It should be fused, e.g. Shift(-1).Map(fun x -> ...).Shift(-3) is the same as Shift(-4).Map(fun x -> ...). See Running time exponential in number of operators on stream #127 )

• Cache/CacheInto

• Scan

• Moving statistics - SMA, EWMA, MovingMedian, MovingRegression, etc. Could generalize the calculations in the same way SMA is implemented currently, but should specialize for hot cursors such as SMA. Watch for precision loss, e.g. see this for StDev.

• Stat1 - Count and Sum over a Window. Mean could be a property of Stat1 struct or calculated on the fly. This is kind of done, need to refactor the implementation and make it similar to Stat2. TODO add Product, it is cheap and allows to calculate geometric mean

• Stat2 - Stat1 + Variance + double StDev as property + start key + end key + end value. Allows to calculate ZScore and annualized StDev for DateTime keys.

• Stat3/Stat4 - Skewness/Kurtosis. Forward-only accumulators are here, need to handle outgoing values.

• Chunk - review naming. We split series into adjacent chunks by some function: Func<TKey, TWhatever>. While the function returns the same value all items go to the same chunk. Non-adjacent chunks could have the same TWhatever value. E.g. DateTime.TimeOfDay.Ticks / TimeSpan.TicksPerHour gives same values for each day, but equals values are 24 hours apart. In this example DateTime.Ticks / TimeSpan.TicksPerHour works without repeating values, but the result is the same. Consider adding an option for keys at the start of each chunk. Count/Width are easy to add as well but such chunks depend on location - e.g. where to move from MoveAt to build a chunk?

Binary transformations (via Zip)

• Binary arithmetic - boilerplate is done. Not tested. Risk of copy-paste errors, too many similar blocks of code. TODO Replace Map with Op2, that will remove a virtual call. Compare perf with Map.
• Binary comparison - boilerplate is done. Not tested. Risk of copy-paste errors, too many similar blocks of code.
• Lookup - see a comment in Zip constructor.

Aggregations

• Fold
• Sum/Product/SumProduct/Average/etc. - should specialize on most important inputs (foreach optimization). Should match MovingStatistics.
• Async versions of aggregations that return a LastValueSeries or IObservable directly.
• Stat1
• Stat2

Zip

Older ZipN that worked on an array of series is deprecated in favor of a specialized Zip that works over two series. For binary arithmetic operators performance is 5+x times faster.

• ICursor<,> members implementation.
• ZipN should use Zip recursively (in a fold-like manner). POC is done. With new[] everything is simple and due to lazyness the new arrays are allocated only when CurrentValue is evaluated. But this still causes a lot of GC0. Consider an additional method that works with PreservedBuffers (finalizing PooledOwnedBuffers should work OK if we forget to dispose PBs).
• ZipN should not accept lambda, it should be internal and then wrapped into any specialized implementations.
• Zip2 for arithmetic ops with two series should be specialized.
• Rewrite async move using Updated property of inputs.
• Support indexed series. See Lookup cursor
• Support parallel calculations for large N (review the idea of adding complexity attribute to each series). This would only be beneficial when we will calculate batches in parallel, otherwise synchronization and method calls will kill all potential benefits. (There was a funny and interesting article recently on how a guy wrote a code in C that was faster on his laptop on a single thread than a Hadoop cluster - we should optimize for single-thread performance first!)
• Random test for forward-only evaluation.
• Test MoveAt correctness in the random tests by evaluating it at every possible key from min to max and comparing to the expected series (all directions).
• CPU and allocations profiling. The main private methods must be inlined.

Interop

• RCursor - prepare some data structure that could be consumed by Spreads.R and provide a function name. Will be done in the R project.
• ~~~RactorCursor - same as R, but sends data to Ractor~~~
• AsyncMap() is enough for cases that to not require pinned blittable data, any complex work could be done in a lambda.

General issues

• Try to make CursorSeries structs, at least stateless one. Stateless are the ones whose state is defined at construction time, e.g. Range start and end values, and that state doesn't change during cursor moves. Mutable structs are evil and there will be a lot of problems, but in very limited cases (Window) that could be useful.
• Recovery logic is tested only for several cases.