CCXT integration

[cjdsellers]

Lets push ahead with setting up some scaffolding for a CCXT integration.

I'm proposing we keep all broker and exchange integration modules in nautilus_trader/adapters. I'm adding a ccxt module folder on the next push.

Basically what needs to happen is implementations of DataClient and ExecClient need to be written.

The data classes are about to get refactored to ease this whole process though, I'm thinking of separating DataClient from a DataManager type class (for lack of a better name). Then we can have as many DataClients as we like all piping back to DataManager.

Thoughts everyone?

[scoriiu]

Sounds good to me.

[cjdsellers]

So in an initial refactoring DataClient has been separated from a new common DataEngine. This allows many different DataClient implementations to be registered with the engine. BacktestDataEngine overrides some methods for now to keep the current behaviour. TradingNode still hard codes a LiveDataEngine and LiveExecEngine. The former will be an implementation for Nautilus Enterprise (C# distributed backend), and will likely be moved into adapters/enterprise for example.

For live trading my current intention is to allow as many DataClients as the user wishes to be specified in the config.json. These will then be instantiated and registered with the DataEngine as the system spools up.

[cjdsellers]

Also in a future push, many different ExecutionClients can be registered with the common ExecutionEngine so that multiple venues can be traded from a single TradingNode.

Once domain objects are sent into the system we're limited by the GIL. However asyncio usage in the clients will definitely be necessary to avoid any network IO bounds for these new features.

[mLewisLogic]

Just a small FYI that streaming in CCXT is only supported in the paid pro version: https://ccxt.pro
That said, I agree that CCXT is the right library here. Well organized and provides an insane amount of flexibility. Esp in crypto, cross-exchange stat arb and mean-reversion strategies are major categories.

[scoriiu]

In order to use asyncio efficiently in python, the code has to run in entirely in async. Scheduling async coroutines from a non-async context doesn't work well in my experience. However, in this case async coroutines will be called from cython, which I don't know yet how will it work because I didn't use it in this way before.

[scoriiu]

We discussed about using CCXT in a separate thread and also mentioned CCXT PRO there. The PRO version is nice, but it needs more love in order to be production ready. For streaming I suggest having our own implementation based on asyncio.

[mLewisLogic]

Interesting. I've got no skin in the game on CCXT PRO. I've used CCXT in the past (and a handrolled Bitmex streamer), but in this case I'm actually trading equities (so just a spectator on the crypto stuff).

Amazingly, seems like there is some prior art for async in Cython: https://github.com/MagicStack/uvloop/blob/d6c67e7a7283f433e314e96a3bae2085e1d72e8f/uvloop/server.pyx#L51-L75

I'd imagine that TradingStrategy is the most likely place that a user would want to drop new coroutines onto the loop, so would just need to make sure the loop is accessible there. Fingers crossed that Cython doesn't complicate the ability to pass a coroutine to a create_task() call.

[mLewisLogic]

I think that as long as your top level event handling and orchestration is async you're good. Not much of a problem if you call down into the meat of the Cython code that's sync. Assuming you don't snag on any network or disk calls.

[mLewisLogic]

(sorry for sidetracking the CCXT issue. Guess streaming/async is really its own general animal)

[scoriiu]

I'm not sure how the design will be, it depends if the exec client can be called from a non-async context and be able to successfully fire async request using ccxt. In this case there will be no need to have async coroutines in the strategy. The flow that I see here is the following, e.g. order submission

0. Start listening on the private websocket stream to order/position/balance topics (in exec client?)
1. Submit an order from the strategy
2. The order submission request is is passed to the exec client, which will fire a http request (using async if possible) and will return immediately
3. The order update is received upon succesful http request in the websocket stream and an on_event is triggered in the strategy
4. Optionally, the position is maintained independently in the trader as opposed to using the position updates from the exchange. This has to be discussed

[scoriiu]

As an inspiration I recommend having a look at https://github.com/CoinAlpha/hummingbot/. It looks like they managed to call async coroutines from cython. e.g. https://github.com/CoinAlpha/hummingbot/blob/03125f2435a8076af342a4648e8cdc41da080e43/hummingbot/connector/exchange/kucoin/kucoin_market.pyx

[scoriiu]

Cython:

Interesting project, uvloop. They claim to be 2-4x faster than the asyncio stock loop. This is good to know. Thanks.

[cjdsellers]

I think we're settled that the CCXT integration is a good idea, whilst leaving open the option for the PRO streaming also. Regardless having our own stand alone efficient streaming solution is needed.

That general control flow you mention above sounds like the right sort of thing. Note however that currently the platform runs backtests single threaded with messages being handled in sequence. LiveExecutionEngine sits on top of the common ExecutionEngine implementation, its simple less than 100 lines, handler methods for commands and events place them on a queue - with a processing thread pulling them off and handling them one at a time.

I'll keep pushing the platform towards something similar for the DataEngine with some kind of a live version utilizing a thread. Its then up to the clients to make their async requests and pass objects back into the handler methods as fast as possible... something like that.

A different architecture may emerge as I iterate towards the above. Remember the DataEngine holds shared memory which strategies may want to call at any time (granted there is some GIL safety). So any updating of the data really has to be single threaded (blazingly fast with Cython anyway). Lets keep the async stuff as far to the periphery as possible probably just in the clients and their caller. Also note Redis is single threaded (but again, burning C level fast) which is where the ExecutionDatabase holds the state for the engine (in the live case).

uvloop does look cool. It would also be in keeping with one of the points of difference for this platform being the mostly Cython production code, and consequently fast performance.

[mLewisLogic]

I won't have time to really read into it, but is there an off-chance that the hummingbot classes are generic/decoupled enough that they could be used themselves for broker communication (and tied into the Nautilus exec model once it's set)?
Just asking because integrations can be a major time suck, but also can be make-or-break for adoption. Shortcuts that don't compromise elegance would help with speed to market. Worst case they're replaced with something custom later as cycles allow. Of course if the interface doesn't match well then custom may be the only rout.
Guess this is more a question for @scoriiu since you're familiar with hummingbot interface design.

[scoriiu]

It's a good proposition, I had the same dilemma some time back when I was researching for robust webosocket streams and came across hummingbot. Back then the code was not as separated as it is nowadays, so I had to write my own implementation being inspired by it. I think now it would be fairly easy to take the code and rework it a bit. They still lack the connector for Bitmex, so that one we probably have to write ourselves. Will have to have a deeper look in their code since it changed quite a lot since last time I checked.

[cjdsellers]

It's a good point @mLewisLogic, I've spent weeks before getting FIX integrations right. I'll have a deeper dive into hummingbot to see whats what.

Further to the above I also need to read the asyncio docs.

[cjdsellers]

The architecture for the execution stack has been finalized for now. ExecutionEngine sits on top of ExecutionCache which sits on top of ExecutionDatabase with a seam there for any specific database adapter. A LiveExecutionEngine can inherit ExecutionEngine which will be one call site for the uvloop event loop to utilise coroutines.

Across on the data side refactorings are about to begin to end up with the same patterns, with another uvloop call site.

The exact interaction with the clients and uvloop remains to be determined, will update during first client implementation.

[cjdsellers]

Update

The asyncio implementation has largely been figured out, and work is currently in progress and making some good headway.

My approach is to first get the unified API integrated, data is working well with order book capability pending but at the top of the backlog. Order management is getting there.

Currently developing live on Binance and the BitMEX testnet.

Later on there could be scope for allowing more advanced per exchange functionality by detecting which exchange the ccxtpro client is for, and utilizing the params for requests and info for responses.

[ian-wazowski]

Update

The asyncio implementation has largely been figured out, and work is currently in progress and making some good headway.

My approach is to first get the unified API integrated, data is working well with order book capability pending but at the top of the backlog. Order management is getting there.

Currently developing live on Binance and the BitMEX testnet.

Later on there could be scope for allowing more advanced per exchange functionality by detecting which exchange the ccxtpro client is for, and utilizing the params for requests and info for responses.

Just out of interest, How do you think of various network latency by different underlying http lib(i.e. curl, aiohttp, requests).

I do not think that ccxtpro did not design for latency-sensitive.

There are also significant latency differences between Python network libraries.

[cjdsellers]

This is a great question.

The decision to integrate with CCXT Pro was to get a number of crypto exchanges running as quickly as possible, it's by no means an optimal solution though.

The vision is to eventually start including exchange specific adapters which have all of the exchange features implemented, and can be optimized as much as possible (including the use of Cython and efficient libraries).

[cjdsellers]

Even with optimized adapters there's still a good place for the CCXT Pro integration as it would be near impossible to write individual adapters for every possible exchange or even keep pace with the efforts there. That project is continually adding exchanges and capability, with that it leaves the door open for more arb strategies taking data from exchanges one isn't as latency sensitive to.

[ian-wazowski]

Even with optimized adapters there's still a good place for the CCXT Pro integration as it would be near impossible to write individual adapters for every possible exchange or even keep pace with the efforts there. That project is continually adding exchanges and capability, with that it leaves the door open for more arb strategies taking data from exchanges one isn't as latency sensitive to.

Agree, Your vision is great.

[cjdsellers]

The current state of play here is that CCXT will continued to be supported, in particular for those users with a CCXT Pro license this is an extremely useful feature.

What needs to be tested now is that the adapter in its current state can be used with the 27 exchanges marked with the ccxt pro badge in the CCXT README.md.

https://github.com/ccxt/ccxt

There are currently specific adapters for Binance and BitMEX which require CCXT Pro.

Going forward a new network module will be added with some generic WebSocket and HTTP client base classes. These can then be leveraged to write adapters which don't depend on CCXT, starting with those exchanges.

[yohplala]

Going forward a new network module will be added with some generic WebSocket and HTTP client base classes. These can then be leveraged to write adapters which don't depend on CCXT, starting with those exchanges.

Hi there, just discovered Nautilus.
I read your comment. Only for information, did you already stumbled on cryptofeed?
I am a user of this (high level user, really not expert / not low level). I believe the work done to manage in a uniform way both websocket or rest APIs is terrific.
Maybe this can be something interesting for you?
Bests

[ian-wazowski]

Going forward a new network module will be added with some generic WebSocket and HTTP client base classes. These can then be leveraged to write adapters which don't depend on CCXT, starting with those exchanges.

Hi there, just discovered Nautilus.
I read your comment. Only for information, did you already stumbled on cryptofeed?
I am a user of this (high level user, really not expert / not low level). I believe the work done to manage in a uniform way both websocket or rest APIs is terrific.
Maybe this can be something interesting for you?
Bests

Hi

I implemented integration with cryptofeed using zeromqCallback(seperated two processor(like Microservice architecture), maintaining low latency as possible.). and this code is under live testing on upbit exchange.

I hope the this part will be done and merged within this weekend.

[yohplala]

I implemented integration with cryptofeed using zeromqCallback(seperated two processor(like Microservice architecture), maintaining low latency as possible.). and this code is under live testing on upbit exchange.

Hey, terrific! Is there any PR that can be reviewed? (just out of curiosity)

[ian-wazowski]

@yohplala

Yet there is no pr under reviewing.

PR will be pushed within 6 hours? Maybe...

By the way. There is old commit that cjseller implemented about cryptofeed integration. I hope you can get help there

[yohplala]

By the way. There is old commit that cjseller implemented about cryptofeed integration. I hope you can get help there

Hi, any link toward that? I am not sure to understand, a PR in cryptofeed, or a PR in nautilus?
Sorry for the question, I am a bit confused :).

[ian-wazowski]

By the way. There is old commit that cjseller implemented about cryptofeed integration. I hope you can get help there

Hi, any link toward that? I am not sure to understand, a PR in cryptofeed, or a PR in nautilus?
Sorry for the question, I am a bit confused :).

PR in nautilus_trader. I remember about 3months ago commits there was a working implementation(but deleted)

Give me a secs

[ian-wazowski]

By the way. There is old commit that cjseller implemented about cryptofeed integration. I hope you can get help there

Hi, any link toward that? I am not sure to understand, a PR in cryptofeed, or a PR in nautilus?
Sorry for the question, I am a bit confused :).

Sorry I can not find the commits with my mobile phone :(

[yohplala]

Sorry I can not find the commits with my mobile phone :(

No worries, I will have a look shortly :)
Thanks for your help, really appreciated!

[ian-wazowski]

Sorry I can not find the commits with my mobile phone :(

No worries, I will have a look shortly :)
Thanks for your help, really appreciated!

ian-wazowski@6407ff2

This is working example.

and the cryptofeeder.py is

from cryptofeed import FeedHandler
from cryptofeed.callback import BookCallback, TradeCallback
from cryptofeed.defines import BID, ASK, L2_BOOK, TRADES
from cryptofeed.backends.zmq import BookZMQ, TradeZMQ
from cryptofeed.exchanges import Upbit, Binance

from multiprocessing import Process
from yapic import json

def receiver(port):
    import zmq
    addr = 'tcp://127.0.0.1:{}'.format(port)
    ctx = zmq.Context.instance()
    s = ctx.socket(zmq.SUB)
    # empty subscription for all data, could be book for just book data, etc
    s.setsockopt(zmq.SUBSCRIBE, b'')

    s.bind(addr)
    while True:
        data = s.recv_string()
        key, msg = data.split(" ", 1)
        print(key)
        print(json.loads(msg))

def main():
    # p = Process(target=receiver, args=(5678,))
    # p.start()
    symbols = [symbol for symbol in Upbit.info()['symbols'] if symbol.split("-")[1] == "KRW"]
    # print(Binance.info()['symbols'])
    from pprint import pprint
    pprint(symbols)
    f = FeedHandler()
    #f.add_feed(Upbit(symbols=symbols, channels=[TRADES], callbacks={TRADES: TradeCallback(trade)}))
    #f.add_feed(Upbit(symbols=symbols, channels=[L2_BOOK], callbacks={L2_BOOK: BookCallback(book)}))
    f.add_feed(Upbit(symbols=symbols, channels=[TRADES, L2_BOOK], callbacks={TRADES: TradeZMQ(port=5678), L2_BOOK: BookZMQ(port=5678)}))
    f.run()

if __name__ == '__main__':
    main()

[cjdsellers]

Hi @yohplala

Thank you for your interest! I'm aware of cryptofeed and have had several discussions with the author Bryant. It took some modifications to be able to integrate it into NautilusTrader as our architecture is designed so that the live system runs on a single event loop - cryptofeed was previously running on its own internal event loop.

Also at the time, not many private endpoints had been implemented - for a full NautilusTrader integration we need access to the private account and trade streams at the very least. I know cryptofeed has since implemented some of these but I haven't had a look in detail yet.

However I do like how they handle the feeds by abstracting over both http2 and websockets and plan to introduce something similar into the network subpackage.

Expect some contributions from several sources soon including @ian-wazowski and @limx0 and we'll combine and refactor to something solid we can build integrations on top of.

[cjdsellers]

Lets direct future discussion around this to #231