v3.0.0 preview | Streaming indicators and quotes

[DaveSkender]

We’ll be releasing a series of progressive pre-release preview versions to evaluate and stabilize features for streaming indicators from live WebSockets and other incremental quote sources. As we evolve our approach, your feedback is appreciated. Runnable examples are available.

Warning
We expect possibly 5-10 preview pre-release versions before we have a sufficiently stable core to make an official v3 package. These preview versions are all experimental and will be quite volatile with breaking changes and conventions.

To do

• create core quote provider, handling
• prototype with EMA and SMA indicators
• prototype with basic .Use(..) chain
• release 3.0.0-preview.1014.15
• release 3.0.0-preview.1014.12 (newer than .15)
• get initial feedback
• release 3.0.0-preview.1014.25 (maintenance update only)
• add isolated increment methods #1091 for all indicators - @martonb feedback and example in Ema.Increment()
  • Idea: Increment approach for ADX #1062
• auto-prune in-memory stream data #1095 on EMA/SMA base cases only - @elpht
• quote-stream auto-aggregator #1093 to [handle different stream flavors] (Indicator source taxonomy #1069 (comment)) - @mihakralj
• stream through full chains #1096 on EMA/SMA base cases only
• handle out-of-sequence late-arrival quotes #1094 on EMA/SMA base cases only
• increment base documentation
• get feedback
• more performance tuning and usability testing
• implement widely for most common moving average indicators
• implement widely for most common other indicators
• get and incorporate final feedback
• publish initial stable v3 release
• plan to implement remaining indicators, incrementally

[DaveSkender]

The latest preview is 3.0.0-preview.1014.25 for evaluation and community feedback.

This preview version has this usage model:

// initialize quoteProvider provider,
// can be extended to integrate with your
// quote stream or WebSocket source
QuoteProvider quoteProvider = new();

// subscribe individual indicators as observers
EmaObserver ema = quoteProvider.GetEma(20);
SmaObserver sma = quoteProvider.GetSma(13);
[..]

// these results are automatically updated
IEnumerable<EmaResult> emaResults = ema.Results;
IEnumerable<SmaResult> smaResults = sma.Results;

// when quoteProvider are added in a separate process
quoteProvider.Add(quote);

// and also works with `.Use` chain
EmaObserver emaAlt = quoteProvider
    .Use(CandlePart.HL2)
    .GetEma(20);

// stop streaming
// by unsubscribing each observer
ema.Unsubscribe();

// stop providing quoteProvider to everyone
// will unsubscribe all gracefully
quoteProvider.EndTransmission();

// feedback appreciated

[DaveSkender]

For those looking to contribute, here’s an explanation of this versions file organizational structure: #1069 (comment), subject to change as we stabilize our final approach.

[mo3in]

What you think about using Rx.net for streaming data and processing indicators?

[DaveSkender]

I did look at Rx.NET when working on #824. I left some notes there. Seems like a solid library overall.

It’s probably better for people using this library than for us to use it internal to the library. The native C# .NET observables are easy enough to use and seem to work well for what we’re doing, so it’s not needed. For us and the features we need, Rx.NET appeared to only add a layer of complexity and performance costs.

With that said, I don’t know it extremely well. What are your thoughts on it?

[DaveSkender]

I was just checking out the new v6 for Rx.NET (aka reactive extensions). Much of the core reactive observable framework is part of .NET core. From what I can tell, you’d really only need these extensions for advanced use cases. It’s an excellent overall framework for handling modern data streams, worth exploring, and will be fully compatible with v3 of our library.

[mo3in]

Yes, but some of extensions likes Buffer, Window, Groupby[Until] can useful.

And What your Opponent about Subjects?
They have different use cases and is it possible to use them?

[DaveSkender]

I don’t have any opposition to any of those specific functions and may take another look. My only hesitation for use inside this indicator library isn’t about utility, it’s more about need and performance costs that often come with “helper” libraries like these.

Many of the basic elements in the Rx.NET library are similar to LINQ in it has useful, simpler developer syntax that’s excellent for 99% of its intended users. If small performance costs aren't going to make a difference in your external code, I'd recommend using it.

Given our general guiding principles for this library, I’d consider adding some of these if there are valuable pass-through usability benefits to end users to offset these small performance costs.

Generally speaking, for a base math-style library like ours, it’s often better for end users when we internally use the ugly native C# syntax than the simpler developer-friendly ones from third-party libraries. For example, using a for loop to iteratively sum an array is faster than the LINQ myarray.Sum() extension, which has no residual end user functionality benefits, so we use the uglier, faster looping technique.

[DaveSkender]

I’ll probably upgrade to use Rx.NET in my own personal trading system for processing and interacting with this indicators library. ML.NET is another that users seem to like as well, in the .NET ecosystem.

[elpht]

Hi, I'm right now trying out v3.0.0 preview and wanted to ask whether there is or will be an option to limit the size of the data collections held by both, observable and observer, as I didn't find any yet. Thanks!

[DaveSkender]

The idea of these collections growing in an uncontrolled manner has crossed my mind. Since these are instantiated classes, I think we can add an optional setting so you can supply a max size that'll trigger some auto-pruning after new data arrives. Makes sense.

• auto-prune in-memory stream data #1095

[elAndyG]

Just wanted to share a note, for "determine best approach for handling out of sequence quotes", or the "auto-aggregator", you may need a parameter to determine what to do with missing candles. I found that in premarket or sometimes just after open, some candles will be missing because of extremely limited volume.

I believe the correct way to do it is to use the last available value as that candle's value. For instance, if you have candles for 10:01, 10:02, 10:04, 10:05, the missing 10:03 could potentially mess up any calculations. So 10:03 can just have the same value as 10:02. Or, the avg between 10:02 and 10:04 is another approach ive seen.

great library! and great work!

[DaveSkender]

Yes, I think this'll be interesting to figure out. And yes, probably part of an optional and configurable auto-aggregator utility tool. I think most users will want visibility and control over anything that creates magic numbers through extrapolation or interpolations.

Inside an aggregate window / current period

Inside a current aggregate window is relatively easy since you're just doing min/max/first/last, resend. We handle this "update of last" well already, but do need to insert the new aggregator into that process.

• quote-stream auto-aggregator #1093

Outside a current window (historical)

The challenging part here is when, outside of a current period, there's a late historical change, once that has increasingly larger impact the further back it goes. This can happen with an upstream correction or similarly for splits / dividend adjustments, an even more insidious impact.

For example, say you're streaming all day long with a single quote streams and 100 indicators and 30 with 2 layers of chaining. If an old or updated quote comes in for 500 periods ago you now have a rather big situation where there's potentially large and complex chain of cleanup work, $500(100+30\times2)=80,000$ in this case. It's possible to do, but if this happens frequently in large datasets, you can imagine the spiking loads that can occur.

I say potential, because at some point it leans more towards moot than useful, it's ancient history a user may not want to address at all. The option to do this may fall into the same settings category as @elpht recommends.

• handle out-of-sequence late-arrival quotes #1094

[MhAllan]

Let me share some modeling with you, I did for series of data I had before

This design should help you write indicators for real time series very easy and quickly, and should handle series size, and improve performance by reducing arrays allocation

Cache methods results, this interface for caching methods using their names and an object that represent the method parameters

public interface IMethodCacher
{
    bool TryGetFromCache<TRet>(string key, object parameters, out TRet result);
    void SetCache<TRet>(string key, object parameters, TRet value);
    TRet ExecuteAndCache<TRet>(object obj, Func<TRet> func, [CallerMemberName] string methodName = "");
    Task<TRet> ExecuteAndCacheAsync<TRet>(object parameters, Func<Task<TRet>> func, [CallerMemberName] string methodName = "");
}

Implement it, I chose concurrent dictionary here

Two dictionaries one for the method name, the other inside it is for that method parameters

public abstract class MethodCacher : IMethodCacher
{
    readonly ConcurrentDictionary<string, ConcurrentDictionary<object, object>> _c =
        new ConcurrentDictionary<string, ConcurrentDictionary<object, object>>();

    public bool TryGetFromCache<TRet>(string key, object parameters, out TRet result)
    {
        result = default;
        if (!_c.TryGetValue(key, out var dic))
            return false;
        if (!dic.TryGetValue(parameters, out var obj))
            return false;

        result = (TRet)obj;
        return true;
    }

    public void SetCache<TRet>(string key, object parameters, TRet value)
    {
        if (!_c.TryGetValue(key, out var dic))
            dic = _c.AddOrUpdate(key, new ConcurrentDictionary<object, object>(), (k, v) => v);

        dic.AddOrUpdate(parameters, value, (k, v) => v);
    }

    public async Task<TRet> ExecuteAndCacheAsync<TRet>(object parameters, Func<Task<TRet>> func,
        [CallerMemberName] string caller = "")
    {
        if (!_c.TryGetValue(caller, out var dic))
        {
            dic = _c.AddOrUpdate(caller, new ConcurrentDictionary<object, object>(), (k, v) => v);
        }

        if (!dic.TryGetValue(parameters, out var result))
        {
            result = await func();
            dic.AddOrUpdate(parameters, result, (k, v) => v);
        }

        return (TRet)result;
    }

    public TRet ExecuteAndCache<TRet>(object obj, Func<TRet> func, [CallerMemberName] string caller = "")
    {
        if(!_c.TryGetValue(caller, out var dic))
        {
            dic = _c.AddOrUpdate(caller, new ConcurrentDictionary<object, object>(), (k, v) => v);
        }

        if(!dic.TryGetValue(obj, out var result))
        {
            result = func();
            dic.AddOrUpdate(obj, result, (k, v) => v);
        }

        return (TRet)result;
    }

    public bool TryGetFromCache<TRet>(string key, out TRet result) => 
        TryGetFromCache(key, new MethodCacheParams(), out  result);

    public void SetCache<TRet>(string key, TRet value) =>
        SetCache(key, new MethodCacheParams());
}

Now extend it, so it looks like method call

Define IEquatable to compare parameters and work as dictionary key for the method parameters, go for up to 6 parameters

record struct MethodCacheParams();
record struct MethodCacheParams<T>(T a) where T : struct, INumber<T>;
record struct MethodCacheParams<T1, T2>(T1 a1, T2 a2)
   where T1 : struct, INumber<T1>
   where T2 : struct, INumber<T2>;

record struct MethodCacheParams<T1, T2, T3>(T1 a1, T2 a2, T3 a3)
   where T1 : struct, INumber<T1>
   where T2 : struct, INumber<T2>
   where T3 : struct, INumber<T3>;

// and so on until T6

Write the extensions

public static class MethodCacherExtensions
{
public static TRet ExecuteAndCache<TRet>(this IMethodCacher mc, Func<TRet> func, [CallerMemberName] string caller = "") 
    => mc.ExecuteAndCache(new MethodCacheParams(), func, caller);

public static TRet ExecuteAndCache<TRet, T1>(this IMethodCacher mc, T1 a1, Func<TRet> func, [CallerMemberName] string caller = "") where T1 : struct, INumber<T1>
    => mc.ExecuteAndCache(new MethodCacheParams<T1>(a1), func, caller);

public static TRet ExecuteAndCache<TRet, T1, T2>(this IMethodCacher mc, T1 a1, T2 a2, Func<TRet> func, [CallerMemberName] string caller = "")
    where T1 : struct, INumber<T1>
    where T2 : struct, INumber<T2>
    => mc.ExecuteAndCache(new MethodCacheParams<T1, T2>(a1, a2), func, caller);

// and so on until T6
. . . 

Usage example, this is how the results holder will look like

static void Main(string[] args)
{
    var obj = new MyObj();
    obj.X = "Hello world!";

    var x = obj.GetX(1, 2, 3);
    Console.WriteLine(x);

    obj.X = "Good bye";

    x = obj.GetX(2, 3, 4);
    Console.WriteLine(x);

    x = obj.GetX(1, 2, 3);
    Console.WriteLine(x);

    Console.ReadLine();
}

public class MyObj : MethodCacher
{
    public string X { get; set; }

    public MyResult GetX(int a, int b, int c)
    {
        var func = () => new MyResult(X);

        return this.ExecuteAndCache(a, b, c, func);
    }
}

Implement series navigation

Define series

This model will have

• _isCapped: should be set to a positive number in real time series, it will cap the size the of the collection, once above the cap, old items will be deleted
• Prev: give the previous item of a given item
• EnumeratePrev(): enumerate previous items of a given item

public interface IReadOnlySeries<T> : IReadOnlyList<T>
{
    IEnumerable<T> EnumeratePrevious(T item, bool includeItem);
    IEnumerable<T> EnumerateNext(T item, bool includeItem);
    T Prev(T item);
    T Next(T item);
    int IndexOf(T item);
}

public interface ISeries<T> : IReadOnlySeries<T> 
{
    void Add(T item);
}

public class Series<T> : ISeries<T>
{
    readonly int _maxItems;
    readonly List<T> _items;
    readonly bool _isCapped;
    readonly object _lock;

    public Series(int max = -1)
    {
        _maxItems = max;
        _isCapped = max > 0;
        if(_isCapped)
        {
            _items = new List<T>(max);
            _lock = new object();
        }
        else
        {
            _items = new List<T>();
        }
    }

    public T this[int index] => _items[index];
    public int Count => _items.Count;

    public void Add(T item)
    {
        if(_lock != null)
            Monitor.Enter(_lock);
        try
        {
            if (_isCapped && _items.Count == _maxItems)
            {
                var first = _items[0];
                _items.RemoveAt(0);
                OnItemOutdated(first);
            }
            
            _items.Add(item);
            OnItemAdded(item);
        }
        finally
        {
            if(_lock != null)
                Monitor.Exit(_lock);
        }
    }

    public int IndexOf(T item) => _items.IndexOf(item);

    public IEnumerator<T> GetEnumerator() => _items.GetEnumerator();

    IEnumerator IEnumerable.GetEnumerator() => _items.GetEnumerator();

    public IEnumerable<T> EnumeratePrevious(T item, bool includeItem)
    {
        var index = _items.IndexOf(item);
        index += includeItem ? 1 : 0;
        return _items.Take(index);
    }

    public IEnumerable<T> EnumerateNext(T item, bool includeItem)
    {
        var index = _items.IndexOf(item);
        index += includeItem ? 0 : 1;
        return _items.Skip(index);
    }

    public T Prev(T item)
    {
        var index = _items.IndexOf(item);
        if (index <= 0)
            return default;
        return _items[index - 1];
    }

    public T Next(T item)
    {
        var index = _items.IndexOf(item);
        if (index < 0 || index >= _items.Count - 1)
            return default;
        return _items[index + 1];
    }

    protected virtual void OnItemAdded(T item) { }
    protected virtual void OnItemOutdated(T item) { }
}

Define the candle model

This model will have

• Index: gives an idea where are you in the series, Index + 1 is the count of item an indicator is interested in when calculating for this candle
• Prev: go to previous candle and access it's indicator results using the MethodCacher mechanism above
• EnumeratePrev(): allows you to get the enumeration of all candles in the series that are not outdated (if the series is capped) and prior to this candle

public interface ICandle
{
    public int Index { get; } // => _series.IndexOf(this);
    public ICandle Prev(); // => _series.Prev(this);
    public ICandle Next(); // => _series.Next(this);
    public IEnumerable<ICandle> EnumeratePrev(bool includeSelf = true); // => _series.EnumeratePrevious(this, includeSelf);
    public IEnumerable<ICandle> EnumerateNext(bool includeSelf = true); // => _series.EnumerateNext(this, includeSelf);
    public void SetSeries(ISeriece<ICandle> series);

Create the candles series, this will hold candles, set a cap so old ones will be removed, and allow them to navigate by Next() and Prev() methods

public interface ICandleSeries : ISeries<ICandle>
{
}

public class CandleSeries : Series<ICandle>, ICandleSeries
{
    public CandleSeries(int max) : base(max)
    {
    }

    protected override void OnItemAdded(ICandle c)
    {
        c.SetSeries(this); // so in the Indicators we can do c.Prev() and c.Next()
 . . .

Implement the indicators using cachable methods that return indicators results, and implement the navigation in the candle as well. All in the Candle model.

public class Candle : MethodCacher, ICandle
{
     ICandleSeries _series;
    //cached method
    public EmaResult GetEma(int period)
    {
        var func = () => EmaCalculation(period);
        
        //use method cacher
        return ExecuteAndCache(period, func);
    }

    private EmaResult EmaCalculation(int perdiod)
    {
         //get previous item
        var prev = this.Prev();
        if(prev == null) 
                  . . . 
         var prevEma = prev.GetEma(period) // This will come from memory, not calculated

        //Calculate
    }

   //series methods
    public void SetSeries(ICandleSeries series) => _series = series;
    public ICandle Prev() => _series.Prev(this);
    public ICandle Next() => _series.Next(this);
    public IEnumerable<ICandle> EnumeratePrev(bool includeSelf = true) => _series.EnumeratePrevious(this, includeSelf);
}

Or write an Indicator

public static class SomeIndicator
{
        public static SomeResult Calculate(ICandle c, int p1, double p2, ...)
       {
              //prev candles
              var prevCandles = c.EnumeratePrev().TakeLast(p1) ...

              //prev
              var prev = c.Prev();

              //prev of prev
              var prev_prev = c.Prev().Prev();

               //cached result
               var r = prev.GetResult(p1, p2, ...) //cached according to method name and parameters.

              // and so on.
       }
}

Hope it helps, I might have copied inconsistent pieces, let me know if it is not clear.

[DaveSkender]

Nice. Thanks for the contributions. I'll have to digest this a bit when I get back into my open PR for the next preview version. This is timely and potentially helpful, but I'll have to review and understand it better first.

[MhAllan]

You welcome, I edited it with more explanation. if you have questions reach me.