How to restart a coroutine after a websocket stream stops receiving data? (1006 error code) #14086
Comments
Kinzowa
changed the title
|
Hello @Kinzowa, First, it is not expected to get stalled streams after some time running it, we will take a look at it to see if is something related to our code or something related to the exchange's server. Second, currently, there is no good way to restart specific streams (unsubscribe is not yet implemented) so the only thing you can do is shut down all the connections in the current instance upon detecting a stalled stream/invalid response/etc (by invoking |
|
Hello @carlosmiei, thank you for your response. Please let us know if you find anything in the code that could cause the stream to crash. On my end, I'll try to catch an exception or something that might give a clue as to what's going on. |
|
Hi @carlosmiei , just to let you know that I catched an exception with error code |
|
@Kinzowa thank you |
|
Looks like the problem is discussed here and here. In my case I have very few disconnection (one every day, or less), probably because the code is executed by a quality server in a datacenter. Also the code is executed by a That would be great if a fix could be implemented. |
|
Hi @carlosmiei, hi @kroitor, Please find below the traceback of the exceptions that occured this morning in the Six exceptions concerned public method Do you think there is something we could do ? Yesterday I configured the Celery worker with only 2 processes instead of 4, to match the number of CPUs, but unfortunnatly it doesn't prevent the error from occuring. Thanks! |
|
Hi @Kinzowa, let me share with you my experience since I am dealing with the same problem.
Unfortunately practice and theory are different and error 1006 happens quite often. I am using Binance, OKX, Bitmex and BTSE ( BTSE is not supported by CCXT) and my code runs on AWS server so I should not have any connection issue. Binance and OKX are the worst as far as error 1006 is concerned.. Honestly, after researching it on google, I have only understood 1006 is a NetworkError and I know CCXT tries to resubscribe the channel automatically. All other explanations I found online did not convince me. If somebody of CCXT staff could give us more info about this error I think the community would appreciate it. In any case, every time an exception is raised, I put it in an As for your second question:
remember that you are Running Tasks Concurrently¶
here you can find info about restarting individual task in a a gather() In your case, since you are using a single exchange (Binance)and unsubscribe is not implemented, as @carlosmiei pointed out you will have to close the connection and restart all the task. You can still use the above for automating it. In case you are using more then one exchange you can design your code in a way that let you close and restart only the Exchange that failed. Another option for you would be defining the tasks with more granularity in the main so that every task is related to a single and well defined exchange/user/method/symbol and every task subscribes a single channel. This will result in a more verbose and less elegant code but it will help you catching the exception and eventually restart only a specific coroutine. I am obviously assuming that after error 1006 you can resubscribe a channel without previously unsubscribe it ( does error 1006 unsubscribe a channel?) final thought:
Professional market makers with a team of engineers working in London do not go to the pub while their algos ( usually co-located within the exchange ) execute thousands of trades. I hope this can help you and I would be happy to continue the discussion. |
|
Hi @ShieldTrade, first thank you for your suggestions and informations. The SF link you pointed out for restarting an individual task is much appreciated, although I'm still a beginner with I must say that I implemented an automatic ping to the exchange and since then I haven't had the |
|
@Kinzowa, CCXT has its own keepalive mechanism which sends ping-pong messages to the exchange. I do not think this is the problem. In any case let me know if your ping solves the issue. I am also considering this malicious scenario: sometimes exchange servers are too busy and they simply drop a few non VIP clients connections. |
Hi, sure, I know there is a ping pong mecanism to maintains the connection alive, but maybe there is an issue with the timer. Since I implemented a 5 sec. ping I had no disconnection since the last 2 days. This solution is also discussed online in several forums. I will post again if/when an occurence happens. |
|
How exactly did you implement the ping please? |
I created a Celery task (see above) in my Django application and schedule it every 5 seconds with beat. It seems to solve the issue. Hi, @carlosmiei, do you think it coul be implemented natively in ccxt ? or maybe the ping mechanism could be improved for Binance and some other exchanges ? |
Kinzowa
changed the title
|
@Kinzowa, CCXT has an integer The default for OKX is 20000 mls. You can override it to 5000 so you would have the same effect as with your Celery task. |
|
Just to let you know that I had zero disconnection since I implemented the periodic ping few days ago and I assume it's a working solution. I think this ticket can be closed now. |
|
@Kinzowa thanks for your feedback! |
Hello,
I'm writing an asyncio application to monitor prices and trade/order events, but for an unknown reason some streams stop receiving data after few hours. I'm not familiar with the
asynciopackage and I would appreciate help in finding a solution.Basically, the code below establishs websocket connections with Binance to listen streams of six symbols (ETH/USD, BTC/USD, BNB/USD for both spot and future) and trades events from two accounts (user1, user2). The application uses the library ccxtpro. The public method
watch_ohlcvget price steams, while private methodswatchMyTradesandwatchOrdersget new orders and trades events at account level.The problem is that one or several streams are interrupted after few hours, and the object
responseget empty orNone. I would like to detect and restart these streams after they stops working, how can I do that ?The text was updated successfully, but these errors were encountered: