Stream signals callback

[papapon]

Hi everyone,

I just activate the enable_stream_signal_buffer option which works perfectly. Thank you very much for this work which simplifies the recovery in case of loss of connection.

I use the process_stream_data option to have a callback to process the data. I was wondering if it was possible to have another callback to handle signals. I think it would be more efficient than coming to read the buffer at regular intervals.

Is this an option that is being considered?

[oliver-zehentleitner]

Thanks for the feedback! You are the first user of the stream_signals who reported this to me!

I personally think the callback is not good and the use of the buffers are a benefit.
It would be just a few lines of code to add a callback support, ...

I would prefer to develop an asyncio ready buffer that supports await.

[papapon]

What is the advantage of a buffer over a callback?

Even with the asyncio version, you will have to do an asyncio.sleep to watch at regular intervals.

[oliver-zehentleitner]

with the buffer you can better handle errors like a not available database (just put the data back to the stream_buffer and try later again).

also if there is a receiving peak when you receive more than you can process the buffer is saving you from loses and runtime errors. but this argument is for the stream receives, not for the signals since they are only a few msg.

and i had problems troubleshooting errors within the callback function, but maybe there is a better way to implement the callback than we do it now.

I dont think that we need an async.sleep.

[amaioli]

Oliver, about the advantage of buffer vs callback to manage peak of messages theoretically we can rely on websocket buffer, can't we?

[oliver-zehentleitner]

Oliver, about the advantage of buffer vs callback to manage peak of messages theoretically we can rely on websocket buffer, can't we?

Yes, but if to big it will reconnect. which does not happen if we move the data to the stream_buffer.

[amaioli]

Hi see sometime reconnection event during volatility peak. So that could be due to the websocket buffer size? Il Mer 7 Lug 2021, 13:39 Oliver Zehentleitner ***@***.***> ha scritto:

…

Oliver, about the advantage of buffer vs callback to manage peak of messages theoretically we can rely on websocket buffer, can't we? Yes, but if to big it will reconnect. which does not happen if we move the data to the stream_buffer. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVPK2T22HQMGRFA5PXBZ2DTWQ4F7ANCNFSM42PCTVVA> .

[oliver-zehentleitner]

yes, thats why i created the stream_buffer.

[amaioli]

Thanks Oliver. I was reading that the default buffer size of the websocket is only 32 frames. I'll switch my bot to the stream_buffer from real time handling. Il giorno mer 7 lug 2021 alle ore 13:55 Oliver Zehentleitner < ***@***.***> ha scritto:

…

yes, they why i created the stream_buffer. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVPK2WTEBQR76VLOFRQQT3TWQ6E7ANCNFSM42PCTVVA> .

[amaioli]

Hi Oliver, I've some problem with the buffer_stream. i don't know why in some cases the loop don't see event emitted by Binance and I don't have any error on your library. Supposing I want to come back to real time handling, do you know if is possible increase the size of the standard websocket buffer? Il giorno mer 7 lug 2021 alle ore 13:55 Oliver Zehentleitner < ***@***.***> ha scritto:

…

yes, they why i created the stream_buffer. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVPK2WTEBQR76VLOFRQQT3TWQ6E7ANCNFSM42PCTVVA> .

[oliver-zehentleitner]

Sorry i dont know!
You mean you loose received data?

[amaioli]

I've in my bot the typical loop you mention in your example. It works but some BOTS after some hours stop retrieving events from the buffer without any errors. In that case I've to restart the BOT. def trade_loop(binance_websocket_api_manager, handler, request_client): """ Main loop. It handle event from websocket """ """ Main loop. It handle event from websocket """ while True: if binance_websocket_api_manager.is_manager_stopping(): exit(0) buffer = binance_websocket_api_manager.pop_stream_data_from_stream_buffer() if buffer is False: time.sleep(0.1) else: buffer = json.loads(buffer) .... Il giorno gio 8 lug 2021 alle ore 14:19 Oliver Zehentleitner < ***@***.***> ha scritto:

…

Sorry i dont know! You mean you loose received data? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVPK2UOSDXXJ4JQSZB24RDTWWJUNANCNFSM42PCTVVA> .

[oliver-zehentleitner]

Hm, i have had an issue in the past where my websockets was to slow (at least my internet connection was...)

So binance stopped streaming to me and sent a message that they had a to big backlog and stop streaming to me...

Maybe you can find hints in https://oliver-zehentleitner.github.io/unicorn-binance-websocket-api/unicorn_binance_websocket_api.html?highlight=error#unicorn_binance_websocket_api.unicorn_binance_websocket_api_manager.BinanceWebSocketApiManager.get_errors_from_endpoints