Data received gets less and less.

[fsteckel]

Hello again,

I report a bug. python version 3.7.5, pip 18.1, ubuntu 19.10 in a local docker container with unicorn binance 1.11.0.

I'm sorry, this is rather unspecific: I posted another issure before and attached a log file. There you stated, you would rather put all the streams of the symbols into one single stream in oppose to my seperating each and every stream. Then I did not rember why I seperated them in the first place. So I did unseperate them and now have two streams, one for depth, one for trade. Each containing all the symbols.

Something strange happens:
No exceptions. Everything seams ok. I collect all the message frames and pack them into sql files (one per day). I started off on the first day with a file of about 6-8GB. Second day 400MB, third 300, fourth 250MB. Less and less data is being packed into the sql files.
This (sql) system has worked before without this error, thats why I suspect the unicorn package (which otherwise is great).
Encountering this behavior in the past drove me into seperating the streams in order to check each streams
stream_info['status'] != 'running'
and also
time.time() - stream_info['last_heartbeat'] > self.stream_heartbeat_tolerance.

One (maybe) reason:
Is the 'running' flag set to running if at least one of the symbols in the stream is still being updated? Then it would yield a false sense of running. Same goes for the last heartbeat. This should better be the last heartbeat of the least updated symbol.

My suspicions may be totally off and I appologize for this rather unspecific 'bug'.
Thanks for any helf you can provide Oliver!

[fsteckel]

This is a very regular error message after a while:
CRITICAL:root:BinanceWebSocketApiSocket->start_socket(f6295724-2949-4caa-89d6-25430b3389ac, ['depth@100ms'], ['gvtbtc', 'funbtc', 'sntbtc', ..., 'tntbtc', 'dockusdt', 'funusdt', 'stormbtc', 'winusdt', 'perlbtc', 'etcbtc', 'renusdt', 'ontbtc', 'qtumbtc', 'dentusdt', 'engbtc']) Exception ConnectionClosed Info: code = 1006 (connection closed abnormally [internal]), no reason CRITICAL:root:BinanceWebSocketApiManager->stream_is_crashing(f6295724-2949-4caa-89d6-25430b3389ac) INFO:logger_logger:Mar 15 2020 20:22:17 UTC: Stream depth@100ms: Status != "running". Stream wird neu gestartet. API request wird gestellt.
Last mgs being my code checking the stream status and deciding it has crashed.

[oliver-zehentleitner]

Hi Flo!

I think the first error is ok. 1006 is very common and handled by the lib!
Is the status running determined from your code or is it a result of get_stream_info()?
Would be great if you could add timestamps to the logs :)

I think I know where the problem is happening, I was observing similar behaviour and its a new issue which was not there up to 1.9.x

In 1.10.0 I started releasing methods for subscriptions which are using websocket.send() and I think the lib is catching a specific exception (no error trace in stdin) but the code doesnt handle it the right way. I will add more logging to the code on level critical, so we might get an insight then.

Best regards!

[oliver-zehentleitner]

I gues its about this section of code: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/ff5fff1801baf518bbda746d365107f1773dab40/unicorn_binance_websocket_api/unicorn_binance_websocket_api_connection.py#L251

async def send(self, data):
        try:
            await self.handler_binance_websocket_api_manager.websocket_list[self.stream_id].send(data)
            self.handler_binance_websocket_api_manager.increase_transmitted_counter(self.stream_id)
        except websockets.exceptions.ConnectionClosed as error_msg:
            logging.error("BinanceWebSocketApiSocket->send(" + str(self.stream_id) + ", " +
                          str(self.channels) + ", " + str(self.markets) + ") Exception ConnectionClosed "
                          "Info: " + str(error_msg))
        except RuntimeError as error_msg:
            logging.error("BinanceWebSocketApiSocket->send(" + str(self.stream_id) + ", " +
                          str(self.channels) + ", " + str(self.markets) + ") Exception RuntimeError "
                          "Info: " + str(error_msg))
        except IndexError as error_msg:
            logging.error("BinanceWebSocketApiSocket->send(" + str(self.stream_id) + ", " +
                          str(self.channels) + ", " + str(self.markets) + ") Exception IndexError "
                          "Info: " + str(error_msg))
        except KeyError as error_msg:
            logging.error("BinanceWebSocketApiSocket->send(" + str(self.stream_id) + ", " +
                          str(self.channels) + ", " + str(self.markets) + ") Exception KeyError "
                          "Info: " + str(error_msg))

I set loglevels to critical, with level error its to noisy in my configuration.
pip install https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/tarball/master --upgrade

Maybe then we can see the reason and let the lib handle the restart.

[oliver-zehentleitner]

I posted another issure before and attached a log file. There you stated, you would rather put all the streams of the symbols into one single stream in oppose to my seperating each and every stream. Then I did not rember why I seperated them in the first place. So I did unseperate them and now have two streams, one for depth, one for trade. Each containing all the symbols.

I remember that and from the view of the OS it seems to be right. But as I mentioned I am observing a similar issue with this script on a test server: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/dev_test_cex_full_non_stop.py

All streams are fine all the time, but the mega stream is dying but remains as "running" in the stream_info output. Logfiles are not referring to a reason.

So maybe its possible that a stream is becoming to big for any reason - yes we can handle it with an exception, but why its not happening with small streams in my setup, only the one big one...

Till now we didnt create streams with a subscription list longer than 8004 chars. In my test i am using 22k chars i guess.

I recognized a bug when sending a subscription via websocket.send() to binance. The binance websocket server splitted to big requests into smaller requests and was not able anymore to interpret the json structure, it always sent back a msg about invalid JSON syntax... so I decided to split the subscriptions into small JSON packages (max 350 items per subscription send) then it worked fine. Could be good to investigate if binance is sending a too request to us, maybe including a list of all subscriptions ...

finally: I am not shure anymore if my thesis is really allways valid! Maybe its not good to use a single stream for every kline, but maybe its also good to make a stream not too big.

[fsteckel]

Thanks I'll check out this version with the big and the single streams. Ill post the results here.

[oliver-zehentleitner]

Maybe this solves this issue: #47 (comment)

[oliver-zehentleitner]

Regarding to Binance API Changelog of 2020-04-23 (thx to @jemeraldo for reprting)

WEB SOCKET STREAM

WebSocket connections have a limit of 5 incoming messages per second. A message is considered:
    A PING frame
    A PONG frame
    A JSON control message (e.g. subscribe, unsubscribe)
A connection that goes beyond the limit will be disconnected; IPs that are repeatedly disconnected may be banned.
A single connection can listen to a maximum of 1024 streams.

... we have two tasks:

1. slow down subscriptions to a max of 5 per second
2. handle the max of 1024 subscriptions per stream