Broker bumpz #294
Merged
Broker bumpz #294
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guilledk
approved these changes
Apr 11, 2022
goodboy
force-pushed
the
broker_bumpz
branch
2 times, most recently
from
dcee927
to
5bc1e03
Compare
This adds a new client manager-factory: `open_client_proxy()` which uses the newer `tractor.to_asyncio.open_channel_from()` (and thus the inter-loop-task-channel style) a `aio_client_method_relay()` and a re-implemented `MethodProxy` wrapper to allow transparently calling `asyncio` client methods from `trio` tasks. Use this proxy in the history backfiller task and add a new (prototype) `open_history_client()` which will be used in the new storage management layer. Drop `get_client()` which was the portal wrapping equivalent of the same proxy but with a one-task-per-call approach. Oh, and `Client.bars()` can take `datetime`, so let's use it B)
To start we only have futes working but this allows both searching and loading multiple expiries of the same instrument by specifying different expiries with a `.<expiry>` suffix in the symbol key (eg. `mnq.globex.20220617`). This also paves the way for options contracts which will need something similar plus a strike property. This change set also required a patch to `ib_insync` to allow retrieving multiple "ambiguous" contracts from the `IB.reqContractDetailsAcync()` method, see erdewit/ib_insync#454 for further discussion since the approach here might change. This patch also includes a lot of serious reworking of some `trio`-`asyncio` integration to use the newer `tractor.to_asyncio.open_channel_from()` api and use it (with a relay task) to open a persistent connection with an in-actor `ib_insync` `Client` mostly for history requests. Deats, - annot the module with a `_infect_asyncio: bool` for `tractor` spawning - add a futes venu list - support ambiguous futes contracts lookups so that all expiries will show in search - support both continuous and specific expiry fute contract qualification - allow searching with "fqsn" keys - don't crash on "data not found" errors in history requests - move all quotes msg "topic-key" generation (which should now be a broker-specific fqsn) and per-contract quote processing into `normalize()` - set the fqsn key in the symbol info init msg - use `open_client_proxy()` in bars backfiller endpoint - include expiry suffix in position update keys
Make the throttle error propagate through to `trio` again by adding `dict`-msg support between the two loops such that errors can be re-raised on the `trio` side. This is all integrated into the `MethoProxy` and accompanying result relay task. Further fix a longer standing issue where sometimes the `ib_insync` order entry method will raise a weird assertion error because it detects some internal order-id state issue.. Just ignore those and make relay back an error to the ems in such cases. Add a bunch of notes for todos surrounding data feed reset hackery.
Found an issue (that was predictably brushed aside XD) where the `ib_insync.util.df()` helper was changing the timestamps on bars data to be way off (probably a `pandas.Timestamp` timezone thing?). Anyway, dropped all that (which will hopefully let us drop `pandas` as a hard dep) and added a buncha timestamp checking as well as start/end datetime return values using `pendulum` so that consumer code can know which "slice" is output. Also added some WIP code to work around "no history found" request errors where instead now we try to increment backward another 200 seconds - not sure if this actually correct yet.
ib has a throttle limit for "hft" bars but contained in here is some hackery using ``xdotool`` to reset data farms auto-magically B) This copies the working script into the ib backend mod as a routine and now uses `trio.run_process()` and calls into it from the `get_bars()` history retriever and then waits for "data re-established" events to be received from the client before making more history queries. TL;DR summary of changes: - relay ib's "system status" events (like for data farm statuses) as a new "event" msg that can be processed by registers of `Client.inline_errors()` (though we should probably make a new method for this). - add `MethodProxy.status_event()` which allows a proxy user to register for a particular "system event" (as mentioned above), which puts a `trio.Event` entry in a small table can be set by an relay task if there are any detected waiters. - start a "msg relay task" when opening the method proxy which does the event setting mentioned above in the background. - drop the request error handling around the proxy creation, doesn't seem necessary any more now that we have better error propagation from `asyncio`. - add event waiting logic around the data feed reset hackzorin. - change the order relay task to only log system events for now (though we need to do some better parsing/logic to get tws-external order updates to work again..
As per erdewit/ib_insync#454 the more correct way to do this is with `.reqContractDetailsAsync()` which we wrap with `Client.con_deats()` and which works just as well. Further drop all the `dict`-ifying that was being done in that method and instead always return `ContractDetails` object in an fqsn-like explicitly keyed `dict`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
it's a bit to explain but this more or less adds some super hackzorz updates to the
ibbackend and some general backend api changes to all backends to supportfqsn: fully qualified symbol names in effort to begin to support broker-agnostic derivative symbols.I don't want to go into too too much detail about the hackery for
ib(since this is all public) but for those that know the api let's just say this makes getting HFT data pretty smooth 馃弰馃徏This is a pre-cursor to #93 and #289.
Oh right and this also include changes to
data._sharedmem.pymostly for larger sizing for those users who want to test the larger data set history / graphics performance versus what's coming in #289 馃弰馃徏UPDATE:
oh right i almost forgot this requires a
tractorpatch which adds atractor.to_asyncio.LinkedTaskChanel.subscribe()so that we can have multiple tasks pull on the trio side of the inter-loop stream.LinkedTaskChannel.subscribe(), explicitasynciotask cancel logging,test_trioisms.py聽goodboy/tractor#304TODO (WIP list will fill out better with links later):
ib_insyncauthor instead of the change made in our fork...