Start piker.storage subsys: cross-(ts)db middlewares

README.rst

@@ -13,13 +13,14 @@ computational trading targeted at `hardcore Linux users <comp_trader>`_ .
 we use as much bleeding edge tech as possible including (but not limited to):
 
 - latest python for glue_
-- trio_ for `structured concurrency`_
-- tractor_ for distributed, multi-core, real-time streaming
-- marketstore_ for historical and real-time tick data persistence and sharing
-- techtonicdb_ for L2 book storage
+- trio_ & tractor_ for our distributed, multi-core, real-time streaming
+  `structured concurrency`_ runtime B)
 - Qt_ for pristine high performance UIs
 - pyqtgraph_ for real-time charting
-- ``numpy`` and ``numba`` for `fast numerics`_
+- ``polars`` ``numpy`` and ``numba`` for `fast numerics`_
+- `apache arrow and parquet`_ for time series history management
+  persistence and sharing
+- (prototyped) techtonicdb_ for L2 book storage
 
 .. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
     :target: https://travis-ci.org/pikers/piker
@@ -31,6 +32,7 @@ we use as much bleeding edge tech as possible including (but not limited to):
 .. _Qt: https://www.qt.io/
 .. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
 .. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
+.. _apache arrow and parquet: https://arrow.apache.org/faq/
 .. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
 .. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
 
@@ -58,23 +60,20 @@ building the best looking, most reliable, keyboard friendly trading
 platform is the dream; join the cause.
 
 
-install
-*******
-``piker`` is currently under heavy pre-alpha development and as such
-should be cloned from this repo and hacked on directly.
+sane install with `poetry`
+**************************
+TODO!
 
-for a development install::
 
-    git clone git@github.com:pikers/piker.git
-    cd piker
-    virtualenv env
-    source ./env/bin/activate
-    pip install -r requirements.txt -e .
+rigorous install on ``nixos`` using ``poetry2nix``
+**************************************************
+TODO!
 
-install for nixos
-*****************
-for users of `NixOS` we offer a development shell envoirment that can be
-loaded with::
+
+hacky install on nixos
+**********************
+`NixOS` is our core devs' distro of choice for which we offer
+a stringently defined development shell envoirment that can be loaded with::
 
     nix-shell develop.nix
 
@@ -85,91 +84,19 @@ run::
 
 once after loading the shell
 
-install for tinas
-*****************
-for windows peeps you can start by installing all the prerequisite software:
-
-- install git with all default settings - https://git-scm.com/download/win
-- install anaconda all default settings - https://www.anaconda.com/products/individual
-- install microsoft build tools (check the box for Desktop development for C++, you might be able to uncheck some optional downloads)  - https://visualstudio.microsoft.com/visual-cpp-build-tools/
-- install visual studio code default settings - https://code.visualstudio.com/download
-
-
-then, `crack a conda shell`_ and run the following commands::
-
-    mkdir code # create code directory
-    cd code # change directory to code
-    git clone https://github.com/pikers/piker.git # downloads piker installation package from github
-    cd piker # change directory to piker
-
-    conda create -n pikonda # creates conda environment named pikonda
-    conda activate pikonda # activates pikonda
-
-    conda install -c conda-forge python-levenshtein # in case it is not already installed
-    conda install pip # may already be installed
-    pip # will show if pip is installed
-
-    pip install -e . -r requirements.txt # install piker in editable mode
-
-test Piker to see if it is working::
-
-    piker -b binance chart btcusdt.binance # formatting for loading a chart
-    piker -b kraken -b binance chart xbtusdt.kraken
-    piker -b kraken -b binance -b ib chart qqq.nasdaq.ib
-    piker -b ib chart tsla.nasdaq.ib
-
-potential error::
-
-    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Roaming\\piker\\brokers.toml'
-
-solution:
-
-- navigate to file directory above (may be different on your machine, location should be listed in the error code)
-- copy and paste file from 'C:\\Users\\user\\code\\data/brokers.toml' or create a blank file using notepad at the location above
-
-Visual Studio Code setup:
-
-- now that piker is installed we can set up vscode as the default terminal for running piker and editing the code
-- open Visual Studio Code
-- file --> Add Folder to Workspace --> C:\Users\user\code\piker (adds piker directory where all piker files are located)
-- file --> Save Workspace As --> save it wherever you want and call it whatever you want, this is going to be your default workspace for running and editing piker code
-- ctrl + shift + p --> start typing Python: Select Interpetter --> when the option comes up select it --> Select at the workspace level --> select the one that shows ('pikonda')
-- change the default terminal to cmd.exe instead of powershell (default)
-- now when you create a new terminal VScode should automatically activate you conda env so that piker can be run as the first command after a new terminal is created
-
-also, try out fancyzones as part of powertoyz for a decent tiling windows manager to manage all the cool new software you are going to be running.
-
-.. _conda installed: https://
-.. _C++ build toolz: https://
-.. _crack a conda shell: https://
-.. _vscode: https://
-
-.. link to the tina guide
-.. _setup a coolio tiled wm console: https://
-
-provider support
-****************
-for live data feeds the in-progress set of supported brokers is:
-
-- IB_ via ``ib_insync``, also see our `container docs`_
-- binance_ and kraken_ for crypto over their public websocket API
-- questrade_ (ish) which comes with effectively free L1
-
-coming soon...
-
-- webull_ via the reverse engineered public API
-- yahoo via yliveticker_
-
-if you want your broker supported and they have an API let us know.
-
-.. _IB: https://interactivebrokers.github.io/tws-api/index.html
-.. _container docs: https://github.com/pikers/piker/tree/master/dockering/ib
-.. _questrade: https://www.questrade.com/api/documentation
-.. _kraken: https://www.kraken.com/features/api#public-market-data
-.. _binance: https://github.com/pikers/piker/pull/182
-.. _webull: https://github.com/tedchou12/webull
-.. _yliveticker: https://github.com/yahoofinancelive/yliveticker
-.. _coinbase: https://docs.pro.coinbase.com/#websocket-feed
+
+install wild-west style via `pip`
+*********************************
+``piker`` is currently under heavy pre-alpha development and as such
+should be cloned from this repo and hacked on directly.
+
+for a development install::
+
+    git clone git@github.com:pikers/piker.git
+    cd piker
+    virtualenv env
+    source ./env/bin/activate
+    pip install -r requirements.txt -e .
 
 
 check out our charts

develop.nix

@@ -6,12 +6,22 @@ stdenv.mkDerivation {
     # System requirements.
     readline
 
-    # Python requirements (enough to get a virtualenv going).
-    python310Full
+    # TODO: hacky non-poetry install stuff we need to get rid of!!
     virtualenv
     setuptools
-    pyqt5
     pip
+
+    # obviously, and see below for hacked linking
+    pyqt5
+
+    # Python requirements (enough to get a virtualenv going).
+    python310Full
+
+    # numerics deps
+    python310Packages.python-Levenshtein
+    python310Packages.fastparquet
+    python310Packages.polars
+
   ];
   src = null;
   shellHook = ''

docs/tina_readme.rst

@@ -0,0 +1,91 @@
+### NOTE this is likely out of date given it was written some
+(years) time ago by a user that has since not really partaken in
+contributing since.
+
+install for tinas
+*****************
+for windows peeps you can start by installing all the prerequisite software:
+
+- install git with all default settings - https://git-scm.com/download/win
+- install anaconda all default settings - https://www.anaconda.com/products/individual
+- install microsoft build tools (check the box for Desktop development for C++, you might be able to uncheck some optional downloads)  - https://visualstudio.microsoft.com/visual-cpp-build-tools/
+- install visual studio code default settings - https://code.visualstudio.com/download
+
+
+then, `crack a conda shell`_ and run the following commands::
+
+    mkdir code # create code directory
+    cd code # change directory to code
+    git clone https://github.com/pikers/piker.git # downloads piker installation package from github
+    cd piker # change directory to piker
+
+    conda create -n pikonda # creates conda environment named pikonda
+    conda activate pikonda # activates pikonda
+
+    conda install -c conda-forge python-levenshtein # in case it is not already installed
+    conda install pip # may already be installed
+    pip # will show if pip is installed
+
+    pip install -e . -r requirements.txt # install piker in editable mode
+
+test Piker to see if it is working::
+
+    piker -b binance chart btcusdt.binance # formatting for loading a chart
+    piker -b kraken -b binance chart xbtusdt.kraken
+    piker -b kraken -b binance -b ib chart qqq.nasdaq.ib
+    piker -b ib chart tsla.nasdaq.ib
+
+potential error::
+
+    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Roaming\\piker\\brokers.toml'
+
+solution:
+
+- navigate to file directory above (may be different on your machine, location should be listed in the error code)
+- copy and paste file from 'C:\\Users\\user\\code\\data/brokers.toml' or create a blank file using notepad at the location above
+
+Visual Studio Code setup:
+
+- now that piker is installed we can set up vscode as the default terminal for running piker and editing the code
+- open Visual Studio Code
+- file --> Add Folder to Workspace --> C:\Users\user\code\piker (adds piker directory where all piker files are located)
+- file --> Save Workspace As --> save it wherever you want and call it whatever you want, this is going to be your default workspace for running and editing piker code
+- ctrl + shift + p --> start typing Python: Select Interpetter --> when the option comes up select it --> Select at the workspace level --> select the one that shows ('pikonda')
+- change the default terminal to cmd.exe instead of powershell (default)
+- now when you create a new terminal VScode should automatically activate you conda env so that piker can be run as the first command after a new terminal is created
+
+also, try out fancyzones as part of powertoyz for a decent tiling windows manager to manage all the cool new software you are going to be running.
+
+.. _conda installed: https://
+.. _C++ build toolz: https://
+.. _crack a conda shell: https://
+.. _vscode: https://
+
+.. link to the tina guide
+.. _setup a coolio tiled wm console: https://
+
+provider support
+****************
+for live data feeds the in-progress set of supported brokers is:
+
+- IB_ via ``ib_insync``, also see our `container docs`_
+- binance_ and kraken_ for crypto over their public websocket API
+- questrade_ (ish) which comes with effectively free L1
+
+coming soon...
+
+- webull_ via the reverse engineered public API
+- yahoo via yliveticker_
+
+if you want your broker supported and they have an API let us know.
+
+.. _IB: https://interactivebrokers.github.io/tws-api/index.html
+.. _container docs: https://github.com/pikers/piker/tree/master/dockering/ib
+.. _questrade: https://www.questrade.com/api/documentation
+.. _kraken: https://www.kraken.com/features/api#public-market-data
+.. _binance: https://github.com/pikers/piker/pull/182
+.. _webull: https://github.com/tedchou12/webull
+.. _yliveticker: https://github.com/yahoofinancelive/yliveticker
+.. _coinbase: https://docs.pro.coinbase.com/#websocket-feed
+
+

piker/_cacheables.py

@@ -20,22 +20,15 @@
 '''
 
 from collections import OrderedDict
-from contextlib import (
-    asynccontextmanager as acm,
-)
 from typing import (
     Awaitable,
     Callable,
     ParamSpec,
     TypeVar,
 )
 
-from tractor.trionics import maybe_open_context
-
-from .brokers import get_brokermod
 from .log import get_logger
 
-
 log = get_logger(__name__)
 
 T = TypeVar("T")
@@ -104,21 +97,3 @@ async def decorated(
         return decorated
 
     return decorator
-
-
-# TODO: move this to `.brokers.utils`..
-@acm
-async def open_cached_client(
-    brokername: str,
-) -> 'Client':  # noqa
-    '''
-    Get a cached broker client from the current actor's local vars.
-
-    If one has not been setup do it and cache it.
-
-    '''
-    brokermod = get_brokermod(brokername)
-    async with maybe_open_context(
-        acm_func=brokermod.get_client,
-    ) as (cache_hit, client):
-        yield client

piker/accounting/_mktinfo.py

@@ -121,7 +121,7 @@ class Asset(Struct, frozen=True):
 
     # NOTE: additional info optionally packed in by the backend, but
     # should not be explicitly required in our generic API.
-    info: dict = {}  # make it frozen?
+    info: dict | None = None
 
     # TODO?
     # _to_dict_skip = {'info'}