Modern marketstore, the summary

[goodboy]

There's been a ton of action over at marketstore and we want in.

We already have a psuedo-async-wrapped client that can probably be used just fine until we write our own async-grpc version (see last bullet).

Currently, my opinion is to work on these in the order presented.

Notably the following really new stuff has been added:

• tick(trade)-to-bar data aggregation, for which there appears to be no docs - likely have to read through this PR

  • input array schema seems to be defined here and is something like "SPY/1Sec/TRADE" for the new table key which then get's aggregated to a /OHLCV key, right?
  • originally asked for in this issue
  • PR contains the loop logic for bucketting which we'll likely need to reverse eng to understand usage
    • made comment below but the lowest timeframe supported seems to be 1s OHLC
  • they seems to allow up to 4 trade condition fields
  • ideally we can store both the trade data and the aggregates without having to keep just the 1Sec data
    • @guilledk so in other words I think we should store both the techtonic style ticks as well as the 1s bucketting but not sure yet if we can get both by only writing one column series (more comments below).

• grpc support for faster IPC was in these PRs

  • there's a flag in the test set to enable
  • command to generate buf files: python -m grpc_tools.protoc -I./ --python_out=./ --grpc_python_out=./ ./pymarketstore/proto/marketstore.proto
  • pymarketstore defined client
    • with PR introducing it
  • purepc is like what we'll want to deliver on the issue I made

• a new replication feature which now has docs

  • requires TLS pem certs
  • it's write only on replicas
  • replicas can't be written directly

---

Feature list:

• async grpc client with anyio is now done thanks to @guilledk as anyio-marketstore 🏄🏼
• tick ingest via the new TRADES/1s key + whatever "conditional" fields we decide to include
• tick to OHLCV bars aggregation using the new support codified in ENH: tick to bar conversion alpacahq/marketstore#399
• try out the binance ingestor to see how well (and at what granularity) it operates
• supervisor for spawning the docker instance with appropriate config and making all of it included as a dependency

I think it makes sense to break these tasks out per dev and nail em down 😎

[goodboy]

Heads up @guilledk they have a bulit-in feeder for binance: https://github.com/alpacahq/marketstore/tree/master/contrib/binancefeeder

Might be worth benchmarking against?

[goodboy]

Digging into some points for the tick-trade aggregation requirements from PR 399:

• It appears as though they define the set of aggregate-able timeframes here: with the lowest being a 1s OHLC.