New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Websocket support #57

Open
dparlevliet opened this Issue Sep 17, 2017 · 58 comments

Comments

Projects
None yet
@dparlevliet

dparlevliet commented Sep 17, 2017

For some reason someone asked me on the nodejs package how to do websockets in python, they provided some code to get started and I was able to fix it so I thought you guys might be interested in using it dparlevliet/node.bittrex.api#24

@skyl

This comment has been minimized.

Collaborator

skyl commented Sep 25, 2017

Is there anywhere I can read up on corehub = connection.register_hub('coreHub')? Where can I get a listing of possible values for eg 'updateSummaryState', 'updateExchangeState'? Is this stuff documented?

@dparlevliet

This comment has been minimized.

dparlevliet commented Sep 25, 2017

No, it's not documented yet, the Websocket API is still in development by Bittrex. There are other methods, but they require authentication and there's no current way to authenticate yet. Bittrex does this on the website when you login.

So, coreHub and those states are the only entry types available to us at this time.

@skyl

This comment has been minimized.

Collaborator

skyl commented Sep 25, 2017

The code works. I adapted it to a little class:

from requests import Session
from signalr import Connection


class WebsocketAPI(object):

    url = 'http://socket.bittrex.com/signalr'

    def __init__(self, market, timeout=120000):
        with Session() as session:
            self.connection = Connection(self.url, session)
        self.corehub = self.connection.register_hub('coreHub')
        self.connection.received += self.debug
        self.connection.error += self.error
        self.connection.start()
        self.corehub.server.invoke('SubscribeToExchangeDeltas', market)
        self.corehub.client.on('updateSummaryState', self.ticker)
        self.corehub.client.on('updateExchangeState', self.market)
        self.connection.wait(timeout)

    def error(self, error):
        print(error)

    def debug(self, *args, **kwargs):
        print('debug', args, kwargs)

    def ticker(self, *args, **kwargs):
        print('ticker', args, kwargs)

    def market(self, *args, **kwargs):
        print('market', args, kwargs)

Useful. Thanks for pointing this out. This is a nice firehose for me. Much appreciated.

@dparlevliet

This comment has been minimized.

dparlevliet commented Sep 25, 2017

No problem, happy to help.

@skyl

This comment has been minimized.

Collaborator

skyl commented Sep 25, 2017

I think this issue should stay open; we should get support for this into this library one way or the other. Maybe @ericsomdahl wants to weigh in. This is the future though ;P

@dparlevliet

This comment has been minimized.

dparlevliet commented Sep 25, 2017

Ah, my mistake. I didn't realise you weren't the maintainer.

@dparlevliet dparlevliet reopened this Sep 25, 2017

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Sep 26, 2017

@dparlevliet and @skyl thank you for the code! Are you able to clarify what the different 'Type' are? E.g 'Type': 0 or 'Type': 1.

[Edit]
I was able to find the answer. Thanks to ilyacherevkov in this thread: n0mad01/node.bittrex.api#23

0 – new order entries at matching price, you need to add it to the orderbook
1 – cancelled / filled order entries at matching price, you need to delete it from the orderbook
2 – changed order entries at matching price (partial fills, cancellations), you need to edit it in the orderbook

@dparlevliet

This comment has been minimized.

dparlevliet commented Sep 27, 2017

Sorry. Yep, that's correct.

@tundeanderson

This comment has been minimized.

tundeanderson commented Oct 27, 2017

Hi, almost finished my implementation.... now just working on getting a snapshot of the market to then add the deltas to, for this I'm doing hub.server.invoke('QueryExchangeState', market). I get the snapshot back successfully, but it always comes back with an empty MarketName, so matching these up with the correct deltas is difficult. Has anyone else experienced this?

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Oct 27, 2017

Get an order book snapshot from the API and compare one of the sides (Bid/Ask) to that of the Websocket snapshot.

@skyl

This comment has been minimized.

Collaborator

skyl commented Nov 12, 2017

Is anyone else experiencing this?

HTTPError: 503 Server Error: Service Temporarily Unavailable for url: http://socket.bittrex.com/signalr/negotiate?connectionData=%5B%7B%22name%22%3A+%22coreHub%22%7D%5D&clientProtocol=1.5

I don't think I've changed anything. I'm thinking maybe Bittrex is under heavy load in the last couple of days and shut it down. Is it just me?

@skyl

This comment has been minimized.

Collaborator

skyl commented Nov 12, 2017

I didn't notice the connection token parameter before. This is what gets sent from the website now:

wss://socket.bittrex.com/signalr/connect?transport=webSockets&clientProtocol=1.5&connectionToken=28%2BotNt6hw6w%2BaNTgvve4ME4aTHBf5Mk2niqkjoyMXRA9c7KU088N8q%2F3DYcoOlNdE4W8tuqy1vR8ssLLaMQ0YKfUkCJ%2B0UsiI%2B8A6b&connectionData=%5B%7B%22name%22%3A%22corehub%22%7D%5D&tid=1
@dparlevliet

This comment has been minimized.

dparlevliet commented Nov 12, 2017

I will reference it here, too, for continuity n0mad01/node.bittrex.api#67 (comment)

@skyl

This comment has been minimized.

Collaborator

skyl commented Nov 12, 2017

@dparlevliet is my hero.

So, altogether, I have working, as of right now, for ticker data for example:

class BittrexSignalr(object):

    url = 'http://socket.bittrex.com/signalr'

    def __init__(self):
        # important! imports go inside so that gevent can patch first. hrm.
        # https://github.com/gevent/gevent/issues/941#issuecomment-281497659
        import cfscrape
        from signalr import Connection

        with cfscrape.create_scraper() as session:
            self.connection = Connection(self.url, session)
        self.corehub = self.connection.register_hub('coreHub')

    def start(self, timeout=120000):
        self.connection.start()
        self.connection.wait(timeout)

    def listen(self):
        raise NotImplementedError()


class BittrexTickerFeed(BittrexSignalr):
    """
    Subclass and override update method to handle deltas feed.
    """

    def listen(self):
        self.corehub.client.on('updateSummaryState', self.update)

    def update(self, ticker):
        """
        print(ticker['Deltas'][0])
        {
            'MarketName': 'BTC-ADX',
            'High': 0.00026361,
            'Low': 0.00018862,
            'Volume': 4721661.42599791,
            'Last': 0.0001916,
            'BaseVolume': 1047.95908808,
            'TimeStamp': '2017-10-09T04:07:15.64',
            'Bid': 0.00018966,
            'Ask': 0.00019157,
            'OpenBuyOrders': 666,
            'OpenSellOrders': 7204,
            'PrevDay': 0.00026087,
            'Created': '2017-07-11T22:42:17.78'
        }
        """
        print(ticker['Deltas'])

Then, I subclass BittrexTickerFeed to do something real with the ticker deltas on update.

# call listen before start
myfeed = MyBittrexTickerFeed()
myfeed.listen()
myfeed.start()
@p0nt

This comment has been minimized.

p0nt commented Nov 19, 2017

python script is not working anymore as of today, something wrong after changes they've made on their website this morning.

https://twitter.com/BittrexExchange/status/932173635447431168

@tundeanderson

This comment has been minimized.

tundeanderson commented Nov 19, 2017

Just checked mine, and mine still works

@p0nt

This comment has been minimized.

p0nt commented Nov 19, 2017

any chance you can leave your code here?
Because the above code, is not working (copy paste, no changes)

@p0nt

This comment has been minimized.

p0nt commented Nov 19, 2017

Did some more testing, it seems SubscribeToExchangeDeltas does work but updateSummaryState does not, do you have updateSummaryState working with python atm? I am very interested in this please

@tundeanderson

This comment has been minimized.

tundeanderson commented Nov 19, 2017

Ah, I'm not using updateSummaryState, only updateExchangeState, SubscribeToExchangeDeltas and QueryExchangeState. Seems like they might have changed updateSummaryState then :(

@p0nt

This comment has been minimized.

p0nt commented Nov 19, 2017

yeah thought so :( it is not working anymore damnit.

Anyone found a solution yet?

@skyl

This comment has been minimized.

Collaborator

skyl commented Nov 19, 2017

Hrm. Yeah, confirmed that I'm not getting any data anymore. The connection seems to happen just fine; but, self.corehub.client.on('updateSummaryState', self.update) event never fires :/

@skyl

This comment has been minimized.

Collaborator

skyl commented Nov 19, 2017

Paging Dr. @dparlevliet ... any idea where 'updateSummaryState' got off to?

@p0nt

This comment has been minimized.

p0nt commented Nov 19, 2017

Ty so much for confirming skyl, I have been looking for around 3 hours but I can not find anything. Lets hope dparlevliet has an idea.

@developerwok

This comment has been minimized.

developerwok commented Nov 20, 2017

the last time i received something from updateSummaryState seems to be around "2017-11-19T07:38:35.93Z"

@p0nt

This comment has been minimized.

p0nt commented Nov 20, 2017

Yeah that is exactly when mine stopped as well, also this might be related to the Bittrex tweet around that time.

https://twitter.com/BittrexExchange/status/932173635447431168

So I guess they've changed the endpoint to something else, because there must be something that they use themselfs to get volume, high, low etc etc

@p0nt

This comment has been minimized.

p0nt commented Nov 20, 2017

taken from the refered post:

Thanks. It seems Bittrex killed "updateSummaryState" 👎
#57

Edit: looking at the source log from the referenced issue, "updateSummaryState" works if you're using "https://socket-stage.bittrex.com" as socket feed Uri.

To mimic what this does in python, probably you need to make a simple HTTP get request to "https://www.bittrex.com". Get the returned headers and cookies from that request and open the feed to the "socket-stage" uri, using those headers / cookies. By simple I mean using the cloudflare package, of course.

@p0nt

This comment has been minimized.

p0nt commented Nov 20, 2017

@p0nt

This comment has been minimized.

p0nt commented Nov 20, 2017

no one yet?

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Nov 24, 2017

@p0nt I've created a basic websocket client. You can check it here python-bittrex-websocket.
@ericsomdahl, perhaps we can merge them at certain point.

@nanvel

This comment has been minimized.

nanvel commented Nov 24, 2017

import asyncio
import json
import time

import aiohttp


SOCKET_URL = 'https://socket.bittrex.com/signalr/'
SOCKET_HUB = 'corehub'


async def socket(tickers):
    """
    Uses signalr protocol: https://github.com/TargetProcess/signalr-client-py
    https://github.com/slazarov/python-bittrex-websocket/blob/master/bittrex_websocket/websocket_client.py
    """
    conn_data = json.dumps([{'name': SOCKET_HUB}])
    async with aiohttp.ClientSession() as session:
        url = SOCKET_URL + 'negotiate' + '?' + urlencode({
            'clientProtocol': '1.5',
            'connectionData': conn_data,
            '_': round(time.time() * 1000)
        })
        async with session.get(url) as r:
            socket_conf = await r.json()

        socket_url = SOCKET_URL.replace('https', 'wss') + 'connect' + '?' + urlencode({
            'transport': 'webSockets',
            'clientProtocol': socket_conf['ProtocolVersion'],
            'connectionToken': socket_conf['ConnectionToken'],
            'connectionData': conn_data,
            'tid': 3
        })
        async with session.ws_connect(socket_url) as ws:
            for n, ticker in enumerate(tickers, start=1):
                message = {
                    'H': SOCKET_HUB,
                    'M': 'SubscribeToExchangeDeltas',
                    'A': [ticker],
                    'I': n
                }
                await ws.send_str(json.dumps(message))
            async for msg in ws:
                print(msg.type, msg.data)


if __name__ == '__main__':
    ioloop = asyncio.get_event_loop()
    ioloop.run_until_complete(socket(['BTC-ETH', 'BTC-NEO']))
@skyl

This comment has been minimized.

Collaborator

skyl commented Nov 24, 2017

Is there any updateSummaryState equivalent?

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Nov 24, 2017

@skyl

This comment has been minimized.

Collaborator

skyl commented Nov 25, 2017

@slazarov works for me.

@p0nt

This comment has been minimized.

p0nt commented Nov 29, 2017

awesome man looks good, so this means you can not use "updateSummaryState" anymore without mentioning tickers, which previously just updated all bittrex tickets instead of chosen one?

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Nov 29, 2017

@p0nt I was testing it yesterday. Trying to invoke 'SubscribeToSummaryDeltas' with the specific tickers results in an error. In another words, you will receive the summary state deltas for all the updated tickers on the exchange. Bittrex seems to constantly change it.

@p0nt

This comment has been minimized.

p0nt commented Dec 3, 2017

Ah thats great! I am looking for that to just get all tickers summary info instead of just the one I subscribe to.

Thank yuu!

@hippylover

This comment has been minimized.

hippylover commented Dec 8, 2017

For QueryExchangeState, is the solution really to get the orderbook from rest api and compare the colums?

I suppose i need to do a QueryExchangeState and not just a rest call in order to get the right nonces?

A weird thing too is that when i do a QueryExchangeState i also receive updateExchangeState stuff as if i had called it...(i guess i don't have to call it then, if i can get queryexchangestate to start telling me anyway and i can only figure out which market it is...)

@hippylover

This comment has been minimized.

hippylover commented Dec 8, 2017

Also this was the only way i managed to be notified of what QueryExchangeState spits out:

connection.received += getOrderBooks
#corehub.client.on('queryExchangeState', getOrderBooks) this doesnt work

Any ideas?

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Dec 8, 2017

@hippylover there are two solutions. Firstly, you can get the order book from the REST API and compare it. Secondly, you can track the QueryExchangeState invokes and know which order book snapshot to expect next. Both have pros and cons.

REST API

  • Pro: for larger ticker subscriptions you will sync the nounces faster because you can request asynchronously
  • Con: you are using the rest API, so it's not a native method

Websocket Invoke

  • Pro: you are doing it natively
  • Con: the invokes are FIFO and that's how you have to sync them. You must make sure that your order nounces match or are lower than the order book snapshot nounce for the sync to work.

In any case, if you scroll up you would see that I have posted a link to a websocket library which you can test.

@ericsomdahl I don't want to hijack the thread by referring people to my library, not my point really. My idea is to get people to test the library which is solely written for that, iron out bugs/issues and combine at some point.

@ericsomdahl

This comment has been minimized.

Owner

ericsomdahl commented Dec 8, 2017

@slazarov no worries. I hope for your lib to merge here eventually... Or at least become the "officially recognized" python implementation for websocket support, depending on your preference.

@skyl

This comment has been minimized.

Collaborator

skyl commented Dec 9, 2017

@slazarov this code was working for a while but doesn't work anymore:

from bittrex_websocket.websocket_client import BittrexSocket


class BittrexTickerSender(BittrexSocket):
    exchange = Bittrex()

    def on_debug(self, **kwargs):
        pass

    def on_open(self):
        self.client_callbacks = ['updateSummaryState']

    def on_message(self, ticker):
        # Bittrex keeps breaking me :/
        print('UPDATe', ticker)

# ....
# ....
if __name__ == `__main__`:
        bts = BittrexTickerSender()
        bts.run()
        while True:
            print('ok..')
            time.sleep(5)

I'm guess Bittrex changed something again? When I print/log in on_debug, I only see 'updateExchangeState' and no updateSummaryState

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Dec 9, 2017

@skyl I have released v0.0.2, please check the repo, reinstall through pip, read the new documentation and tell me if you are experiencing the issue.

@skyl

This comment has been minimized.

Collaborator

skyl commented Dec 10, 2017

@slazarov I have something working again. I'm using master / 0.0.3. It seems like if I try to subscribe to all the markets I get a lot of:

Traceback (most recent call last):
  File "/.../env/lib/python3.6/site-packages/gevent/hub.py", line 863, in switch
    assert getcurrent() is self.hub, "Can only use Waiter.switch method from the Hub greenlet"
AssertionError: Can only use Waiter.switch method from the Hub greenlet
Sun Dec 10 21:40:05 2017 <io at 0x110d1e7b8 fd=30 events=READ active callback=<bound method Waiter.switch of <gevent.hub.Waiter object at 0x110d40ea0>> args=(<object object at 0x10e8adc60>,)> failed with AssertionError

Do you know what this is referring to?

If I just subscribe to a couple of markets this doesn't seem to happen ... I guess we could take this conversation up in your repo ...

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Dec 10, 2017

@skyl Move the issue to the correct repo and provide the code you are using so I could replicate it (it's important to know which subscribe method you are using). The error might be caused by a lot of factors, conflicts with gevent or maybe something outdate within the SignalR library which uses gevent.

@halcyonjuly7

This comment has been minimized.

halcyonjuly7 commented Dec 20, 2017

@nanvel have you gotten your async code to work still? it seems that right now the DDOS protection page is screwing it up.

@nanvel

This comment has been minimized.

nanvel commented Dec 20, 2017

@halcyonjuly7 it was working before. I don't use this API anymore. DDOS protection page should have no relation to the API IMHO.

@nanvel

This comment has been minimized.

nanvel commented Dec 20, 2017

@halcyonjuly7 I see "503 Service Temporarily Unavailable" in response. So, probably, will be available soon.

@halcyonjuly7

This comment has been minimized.

halcyonjuly7 commented Dec 20, 2017

@nanvel one would think that. but using the code you provided a month ago.

you called await r.json() it was throwing an error because it wasn't returning json anymore so I checked the text instead await r.text() and I got the html saying "blah blah this happens automatically please wait up to 5 seconds"

so what endpoint are you using now?

@nanvel

This comment has been minimized.

nanvel commented Dec 20, 2017

@halcyonjuly7 I see the same page "Checking your browser before accessing bittrex.com." now.
Have no good idea how to deal with it. Worst case you can try selenium, but there must be better solutions. I don't use the socket, only these: https://bittrex.com/home/api

@skyl

This comment has been minimized.

Collaborator

skyl commented Dec 21, 2017

@slazarov Have you found a workaround for this? I'm now seeing:

DEBUG:requests.packages.urllib3.connectionpool:https://socket-stage.bittrex.com:443 "GET /signalr/negotiate?connectionData=%5B%7B%22name%22%3A+%22coreHub%22%7D%5D&clientProtocol=1.5 HTTP/1.1" 503 None

We had something going with cfscrape before but that got broken too ...

@slazarov

This comment has been minimized.

Collaborator

slazarov commented Dec 23, 2017

@skyl it should be fixed now. Update to v0.0.5.1
Issue: slazarov/python-bittrex-websocket#12

@zaycker

This comment has been minimized.

zaycker commented Dec 23, 2017

Guys I used ws feature with https://github.com/zolzolwik/node.bittrex.api for a few days and there was a moment when it was died :( And they use cloudscraper to bypass cloudflare protection and it didn't help. I don't remember http status code but I made a decision that good scheduler + get_market_summaries more reliable than ws. Or you have to use it as a fallback to hedge your bets :)

@panamantis

This comment has been minimized.

panamantis commented Jan 17, 2018

I've confirmed that a websocket connection can be make to an authenticated/private call (ie. chat.server.invoke('QueryBalanceState'))
Transfer cookies + agent name from ie/ logged in selenium session -> cfscrape sesh. I'll elaborate later.

Though making this headless is a bit of a challenge re: recaptcha + google authenticator. Cookies seem to last 2-3 hours and could possibly be refreshed though.

@MoroZvlg

This comment has been minimized.

MoroZvlg commented Feb 27, 2018

Sorry, i don't really understand what should i do with connection.wait() if i want listen sockets forever. Can anybody explain?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment