Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Full chain streaming, increment prototypes #1092

Draft
wants to merge 111 commits into
base: v3
Choose a base branch
from
Draft

Conversation

DaveSkender
Copy link
Owner

@DaveSkender DaveSkender commented Oct 12, 2023

Description

As the next step in our v3 preview release plan, this implements:

Problems to resolve

  • get full chain observation to work in isolated EMA/SMA base cases, proof of concept for stream through full chains #1096
  • refactor to obviate dual cache, and other EMA/SMA performance tuning
    • refactor performance tests to group Static or Stream, remove redundant series tests from stream grouping
    • abstract QuoteObserver, ChainObserver, and generic Cache methods with generic <T> result storage
  • add missing use cases for dispositions Acts, rebuild / cache busting scenarios, in preparation for wider implementation of handle out-of-sequence late-arrival quotes #1094
  • allow non-chainable IResult to be an observable » all result forms should be observable, even if they're not chainable (for comprehensive end-use streaming).
  • consider: forcing concrete types for observables, cached quotes; I don't think record struct is going to work well with larger types, and has too many stack copies (stack overflow risks). Allowing TQuote is only problematic on observing and caching; maybe just do a quick transform OnArrival();

Additional testing

  • test: Stack overflow scenarios, others #1208
    • change forward-cascading updates to use iterative Resend() instead of recursive OnNext()
    • one QuoteProvider with all indicators subscribed (massive parallel)
    • one QuoteProvider with max possible chain depth (massive serial)
  • find and fix other, hidden fixed-period initialization indexes

Refactor interfaces

  • remove all forms to Tuple interfaces. These are no longer needed due to wider use and compatibility to the IReusable interface; further reducing re-writing transforms and memory utilization.
  • convert all public class types to record struct and remove unneeded AbstractResult class. Use of struct type and IObservable has largely limited inheritance options.
    • refactor all uses to "write once", since these are now immutable value types.

Documentation

@DaveSkender DaveSkender added this to the v3 milestone Oct 12, 2023
@DaveSkender DaveSkender changed the title update alpaca package isolated increment methods Oct 12, 2023
@DaveSkender DaveSkender linked an issue Oct 12, 2023 that may be closed by this pull request
4 tasks
@DaveSkender

This comment was marked as outdated.

Signed-off-by: Dave Skender <8432125+DaveSkender@users.noreply.github.com>
@@ -2,7 +2,7 @@

| name | type | notes
| -- |-- |--
| `Date` | DateTime | Date
Copy link
Owner Author

@DaveSkender DaveSkender Mar 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • add Date alias with [Obsolete], mapped to IQuote.Timestamp
  • same for all ISeries classes

Signed-off-by: Dave Skender <8432125+DaveSkender@users.noreply.github.com>
@DaveSkender
Copy link
Owner Author

DaveSkender commented May 30, 2024

Not sure why I keep flip-flopping on the [Serializable] attribute. Microsoft has deprecated all BinaryFormatter related code in .NET 8 due to security flaws.

I think it’s still relevant for other distributed asynchronous messaging frameworks, despite its ill-fated relationship to BinaryFormatter, worth reviewing again to natively support more user environments.

  • add [Serializable] attribute to custom exception types
  • check to see if this is even needed for new record struct types
  • add to any other uncovered return types, if needed

@DaveSkender
Copy link
Owner Author

For consideration; but may want to implement this in a v3.1

    /// <summary>
    /// Optional.  Use externally provided cache.
    /// </summary>
    /// <param name="externalCache"></param>
    /// <remarks>
    /// Allow users to provide their own cache storage location.
    /// </remarks>
    protected AbstractCache(
        List<TSeries> externalCache)
    {
        Cache = externalCache;
        throw new NotImplementedException();
    }

@DaveSkender

This comment was marked as outdated.

@DaveSkender
Copy link
Owner Author

  • restore MacOS to build matrix, investigate StackOverflowException

@DaveSkender
Copy link
Owner Author

DaveSkender commented Jul 1, 2024

  • add QuoteProvider() that defaults to built-in Quote class? In addition to QuoteProvider<T>().

  • consider using only QuoteProvider() if struct paradigm doesn’t work well with memory management and stack overflow, and we need to revert to record class-only approach, which can’t be well constrained for generic T.

Signed-off-by: Dave Skender <8432125+DaveSkender@users.noreply.github.com>
Signed-off-by: Dave Skender <8432125+DaveSkender@users.noreply.github.com>
@DaveSkender
Copy link
Owner Author

relative speed, on local PC)

Before these changes:

Method Mean Error StdDev
EmaHub 33.77 us 0.644 us 0.689 us
SmaHub 34.77 us 0.482 us 0.451 us

Current state:

Method Mean Error StdDev
EmaHub 1.783 ms 0.0107 ms 0.0089 ms
SmaHub 1.766 ms 0.0156 ms 0.0130 ms

» significant performance refactoring needed
Current changes (needs performance tuning)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

None yet

1 participant