Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Binancial secs: futes for the peepz! #520

Open
wants to merge 73 commits into
base: storage_middleware_layer
Choose a base branch
from

Conversation

goodboy
Copy link
Contributor

@goodboy goodboy commented Jun 9, 2023

Replacement for #182 keeping as much of the work from @guilledk as possible!

This is the first proto which will begin to complete the tasks listed in #519 🏄🏼

More then likely I am going to rebase this onto #486 as well since I'm somewhat hoping that lands before this onto mainline.

ping @ebisu4


To defer to new issues/PRs


To CHERRY from downstream PRs that relate to this

  • 2efc34a: drop config get/set/del api stuff from legacy PR
  • 674f99a: symcache methods and derivative qualified asset keys
  • 416dbc93: use .get_assets() in .get_mkt_info()
  • 82fd785: make default binance section in config use paper
    engine and disable testnets.

ToDo Summary:

  • adds first proto support for futures testnet endpoints, rest API and drafted functionality to support live order control via our EMS:

    _url = 'https://api.binance.com'
    _sapi_url = 'https://api.binance.com'
    _fapi_url = 'https://testnet.binancefuture.com'
    • thanks @guilledk for all that work from 2 years ago 😂
  • general cleanups / factoring in the API client

    • don't think we need separate ._api, ._sapi, ._fapi
      right? simply do the right endpoints parent call implicitly
      based on venue selection?
      • factored and simplified in 24ef09a
  • futures live data feeds via ws api as per requirements in binance live (futures) trading support #519

    • paper engine trading using these feeds alongside spot feeds
  • ideally basic limit order control through emsd with accompanying tests

@goodboy goodboy requested a review from guilledk June 9, 2023 19:23
@goodboy goodboy added feature-request New feature or request data-layer real-time and historical data processing and storage broker-backend `brokerd`/`datad` related backend tech lurker_sponsored Paid (or traded) for by your local anon fanbois labels Jun 9, 2023
@goodboy goodboy changed the base branch from master to storage_middleware_layer June 10, 2023 00:19
goodboy added a commit that referenced this pull request Jun 20, 2023
@goodboy
Copy link
Contributor Author

goodboy commented Jun 20, 2023

Just adding a skip on the marketstore docker test and then CI should run clean as well 🏄🏼

goodboy added a commit that referenced this pull request Jun 26, 2023
After #520 we've moved to better supporting explicit venues for cex
backends which is important where a provider offers both spot and
derivatives markets (kraken, binance, kucoin) and we need to distinguish
which is being traded given a common asset pair (eg. BTC/USDT). So, make
this work for `kraken`'s brokerd such that requests and pre-existing
live order are (un)packed to/from EMS messaging form.
@goodboy goodboy force-pushed the binancial_secs branch 2 times, most recently from a5ae776 to e322e5e Compare June 26, 2023 19:26
Guillermo Rodriguez and others added 8 commits June 27, 2023 13:42
First draft originally by @guilledk but update by myself 2 years later
xD. Will crash at runtime but at least has the machinery to setup signed
requests for auth-ed endpoints B)

Also adds a generic `NoSignature` error for when credentials are not
present in `brokers.toml` but user is trying to access auth-ed eps with
the client.
Instead of having a buncha logic branches for 'get', 'post', etc. just
pass the `method: str` and do a attr lookup on the `asks` sesh.

Also, adjust the `trades_dialogue()` ep to switch to paper mode when no
client API key is detected/loaded.
Patch again originally from @guilledk and adds a sesh for futures
testnet as well as a order canceller method B)
Use dynamic lookups instead by mapping to the correct http session and
endpoints path using the venue routing/mode key. This let's us simplify
from 3 methods down to a single `Client._api()` which either can be
passed the `venue: str` explicitly by the caller (as is needed in the
`._cache_pairs()` case) or falls back to the client's current
`.mkt_mode: str` setting B)

Deatz:
- add couple more tables to suffice all authed-endpoint use cases:
  - `.venue2configkey: dict[str, str]` which maps the venue key to the
    `brokers.toml` subsection which should be used for auth creds and
    testnet config.
  - `.confkey2venuekeys: dict[str, list[str]]` which maps each config
    subsection key to the list of venue name keys for doing config to
    venues lookup.
- always build out testnet sessions for spot and futes venues (though if
  not set the sessions obviously won't ever be used).
- add and use new `config.ConfigurationError` custom exceptions when api
  creds are missing.
- rename `action: str` to `method: str` in `._api()` since it's the
  proper ReST term and switch what was "method" to be `endpoint: str`.
- mask out `.get_positions()` since we can get that from a user stream
  wss request (and are doing that).
- (in theory) import and use spot testnet url as necessary.
Drop the older `dict[str, ChainMap]` prototype we had since the new
`OrderDialogs` built-out while adding `binance` order support is more
refined and general. Also, handle new and now expect `.spot` venue token
in FQMEs since kraken too has futes markets that we'll likely want to
support eventually.
Since crypto backends now also may expand an FQME like `xbteur.kraken`
-> `xbteur.spot.kraken` (by filling in the venue token), we need to use
this identifier when looking up per-market order dialogs or submitting
new requests. The simple fix is to simply look up that expanded from
from the `Feed.flumes` table which is always keyed by the `MktPair.fqme:
str` - the expanded form.
After #520 we've moved to better supporting explicit venues for cex
backends which is important where a provider offers both spot and
derivatives markets (kraken, binance, kucoin) and we need to distinguish
which is being traded given a common asset pair (eg. BTC/USDT). So, make
this work for `kraken`'s brokerd such that requests and pre-existing
live order are (un)packed to/from EMS messaging form.
We might as well start standardizing on `brokerd` init such that it can
be used more generally in client code (such as the `.accounting.cli`
stuff).

Deats of `broker_init()` impl:
- loads appropriate py pkg module,
- reads any declared `__enable_modules__: listr[str]` which will be
  passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
- loads the `.brokers._daemon._setup_persistent_brokerd

As expected the `accounting.cli` tools can now import directly from this
new location and use the common daemon fixture definition.
Should be the final production backend to switch this over B)

Also tidy up the `update_and_audit_msgs()` validator to log vs. raise
when `validate: bool` is set; turn it off by default to avoid raises
until we figure out wtf is up with ib ledger processing or wtv..
Like you'd think:
- `load_ledger()` -> ._ledger
- `load_accounrt()` -> ._pos

Also fixup the old `load_pps_from_ledger()` and expose it from a new
`.accounting.cli.disect` cli cmd for trying to figure out why pp calcs
are totally mucked on stupid ib..
Allows for tracking paper engine orders despite the ems not necessarily
being opened by the current order mode instance (UI) in "paper"
execution mode; useful for tracking bots/strats running against the same
EMS daemon.
goodboy added a commit that referenced this pull request Jun 27, 2023
@goodboy goodboy mentioned this pull request Jun 27, 2023
8 tasks
@goodboy goodboy requested a review from ebisu4 June 27, 2023 19:41
guilledk pushed a commit that referenced this pull request Jan 18, 2024
Apparently publishing futures contracts that aren't yet trading AND
changing their contract type `str` format/schema was necessary (such
that's there's a f@#$in space in it..)?

I honestly have no idea where they found their "data engineers" XD

TO CHERRY to #520
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
broker-backend `brokerd`/`datad` related backend tech data-layer real-time and historical data processing and storage feature-request New feature or request lurker_sponsored Paid (or traded) for by your local anon fanbois
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant