BinanceWebSocketApiManager._create_stream_thread() stream_id=7bb5ef452f01-7200-c1f7-310c-f7e4ee8e - RuntimeError error: 11 - error_msg: cannot schedule new futures after interpreter shutdown

[WJ-Thanefield]

Version of this library.

Latest

Solution to Issue cannot be found in the documentation or other Issues and also occurs in the latest version of this library.

• I checked the documentation and other Issues. I am using the latest version of this library.

Hardware?

Local server/workstation

Operating System?

Windows

Python version?

Python3.9

Installed packages

No response

Logging output

No response

Processing method?

stream_buffer

Used endpoint?

binance.com

Issue

BinanceWebSocketApiManager.stream_is_crashing(7bb5ef452f01-7200-c1f7-310c-f7e4ee8e)
BinanceWebSocketApiManager._create_stream_thread() stream_id=7bb5ef452f01-7200-c1f7-310c-f7e4ee8e - RuntimeError error: 11 - error_msg: cannot schedule new futures after interpreter shutdown - Please create an issue: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/issues/new/choose

[oliver-zehentleitner]

Thank you! We need log files and a script to reproduce the error...

[WJ-Thanefield]

Thanks Oliver, this time I did not manage to produce a log file. But once I encounter it again will send the log file here. Thanks again.

[AnimusXCASH]

@oliver-zehentleitner I am getting same error.

Is there anyway how it can be handled with exceptions?

BinanceWebSocketApiManager.stream_is_crashing(4e0592aab0ff-03ed-5592-8b4d-a54234ae)
BinanceWebSocketApiManager._create_stream_thread() stream_id=4e0592aab0ff-03ed-5592-8b4d-a54234ae  - RuntimeError `error: 11` - error_msg:  cannot schedule new futures after interpreter shutdown - Please create an issue: 

Ill try to log it and post what i get out

[oliver-zehentleitner]

Hello!

Affected lines of code:
https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/manager.py#L529

Script to replicate (Thanks to nhmartens):
https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/files/10300596/error_replication_v01.txt

Same issues:
#282, #297, #299

I know and understand this issue, but when we get in touch with it the first time (#131) it happened also in other code parts which was very easy to reproduce within seconds and i did not want to "over fix it" with code parts that may be not necessary to solve it. I thought its possible that since we fixed it in #131 i could also avoid it here in this code block.

To implement a CLEAN solution i would need one log file in INFO or even DEBUG logging mode... to see what exactly is happening inside ubwa before the exception occurs.

If you gives give me such a log file, i will start updating it on 2.1.2023 and prepare a fixed release.

[nhmartens]

This is the log file of my replication run, I hope this helps.

error_replication.py.log.zip

[nhmartens]

@oliver-zehentleitner is there any update on this? Is the log file I provided what you were looking for or do you need anything else?

[oliver-zehentleitner]

I will pick this up at next, but i get to tired for this now :/

[dima-dmytruk23]

Up plz

[rockyrobinls29]

Does anyone have any suggestions for a temporary fix for this until it gets resolved?

[oliver-zehentleitner]

i should be able to find the time today or tomorrow. i will get back to you with the update.

[rockyrobinls29]

Thank you! Your work is very much appreciated 😎

[oliver-zehentleitner]

tomorrow, sorry, was too busy today :/

[oliver-zehentleitner]

This error message occurs when a shutdown process has been initiated by the Python interpreter.

This means that the interpreter is about to exit, and thus cannot accept any new tasks.

This happens when exit()/quit() or similar are called, the program window is closed or when the main thread is terminated. Therefore create_stream() is also blocking if no "high_performance=True" is activated. This is how we got a handle on the problem here: #131.

I can't replicate the problem so far.

I don't know if it's possible to start new threads again after this error, doesn't sound like it actually.

Basically, I also find that when the shutdown process is initiated, the code does exactly what it is supposed to. it shuts down UBWA cleanly and exits.

unicorn-binance-websocket-api/unicorn_binance_websocket_api/manager.py

Line 530 in 1c731fd

if "cannot schedule new futures after interpreter shutdown" in str(error_msg):

So the problem must/should be solved in the program logic.

Does this code work for you and avoid the error?

from unicorn_binance_websocket_api import BinanceWebSocketApiManager
import asyncio
import logging
import os

logging.getLogger("unicorn_binance_websocket_api")
logging.basicConfig(level=logging.DEBUG,
                    filename=os.path.basename(__file__) + '.log',
                    format="{asctime} [{levelname:8}] {process} {thread} {module}: {message}",
                    style="{")

class Test:
    def __init__(self, apikey, apisecret):
        self.apikey = apikey
        self.apisecret = apisecret
        self.ubwa = None

    async def start(self):
        def handle_socket_message(data):
            print(f"msg: {data}")

        self.ubwa = BinanceWebSocketApiManager(exchange='binance.com-margin', output_default='dict')
        self.ubwa.create_stream(["arr"], ["!userData"], process_stream_data=handle_socket_message,
                                api_key=self.apikey, api_secret=self.apisecret)
        print('Started')

        while True:
            await asyncio.sleep(1)


if __name__ == "__main__":
    test = Test("key", "sec")
    try:
        asyncio.run(test.start())
    except KeyboardInterrupt:
        print("Gracefully stopping the websocket manager...")
        test.ubwa.stop_manager_with_all_streams()

[dima-dmytruk23]

@oliver-zehentleitner

while True:
await asyncio.sleep(1)

I don't think this solution will 100% work.

To reproduce the error, I raised the websockets.exceptions.ConnectionClosed exception to start_socket method of the BinanceWebSocketApiSocket somewhere around this place https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/sockets.py#L116

As a result, the problem was reproduced, but I did not manage to fix it correctly. In the end, I just removed from this block

                             self.manager.stream_is_crashing(self.stream_id, error_msg)
                             self.manager.set_restart_request(self.stream_id)
                             sys.exit(1)

and added raise Exception(str(error_msg)) which I pass and handle in BinanceWebSocketApiManager _create_stream_thread method. There, in the finally block, I removed loop.close() and added self.specific_process_stream_data[stream_id]({'need_restart_process': True})

And in the callback where I process the message I do this (I just restart the process)

if 'need_restart_process' in data:
             os.execl('/usr/local/bin/python', 'python', *sys.argv) # nosec B606:start_process_with_no_shell

This is very crutch, but I did not find another solution.

[oliver-zehentleitner]

Actually a simple

while True:
    time.sleep(1)

at the end of your script (before it finishs) should fix this issue... have you ever tryed?

[dima-dmytruk23]

Ok. I will try it.

[dima-dmytruk23]

Actually a simple

while True:
    time.sleep(1)

at the end of your script (before it finishs) should fix this issue... have you ever tryed?

Not works

logs

[nhmartens]

Actually a simple

while True:
    time.sleep(1)

at the end of your script (before it finishs) should fix this issue... have you ever tryed?

I also tried this, and it did not solve the issue for me.

[nhmartens]

Hello!

Affected lines of code: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/manager.py#L529

Script to replicate (Thanks to nhmartens): https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/files/10300596/error_replication_v01.txt

Same issues: #282, #297, #299

I know and understand this issue, but when we get in touch with it the first time (#131) it happened also in other code parts which was very easy to reproduce within seconds and i did not want to "over fix it" with code parts that may be not necessary to solve it. I thought its possible that since we fixed it in #131 i could also avoid it here in this code block.

To implement a CLEAN solution i would need one log file in INFO or even DEBUG logging mode... to see what exactly is happening inside ubwa before the exception occurs.

If you gives give me such a log file, i will start updating it on 2.1.2023 and prepare a fixed release.

The script mentioned here replicates the error for me. However, it usually takes between two and four days until the issue occurs and the script exits.

[dima-dmytruk23]

However, it usually takes between two and four days until the issue occurs and the script exits.

You don't have to wait a few days. You can reproduce the bug by throwing an websockets.exceptions.ConnectionClosed exception to start_socket method of the BinanceWebSocketApiSocket somewhere around this place https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/sockets.py#L116

[oliver-zehentleitner]

Hello!
Affected lines of code: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/manager.py#L529
Script to replicate (Thanks to nhmartens): https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/files/10300596/error_replication_v01.txt
Same issues: #282, #297, #299
I know and understand this issue, but when we get in touch with it the first time (#131) it happened also in other code parts which was very easy to reproduce within seconds and i did not want to "over fix it" with code parts that may be not necessary to solve it. I thought its possible that since we fixed it in #131 i could also avoid it here in this code block.
To implement a CLEAN solution i would need one log file in INFO or even DEBUG logging mode... to see what exactly is happening inside ubwa before the exception occurs.
If you gives give me such a log file, i will start updating it on 2.1.2023 and prepare a fixed release.

The script mentioned here replicates the error for me. However, it usually takes between two and four days until the issue occurs and the script exits.

I used it and shorted it, then i added a possible fix - this is the result: #299 (comment)

Please can you test it?

[oliver-zehentleitner]

However, it usually takes between two and four days until the issue occurs and the script exits.

You don't have to wait a few days. You can reproduce the bug by throwing an websockets.exceptions.ConnectionClosed exception to start_socket method of the BinanceWebSocketApiSocket somewhere around this place https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/sockets.py#L116

Please test #299 (comment) with the latest and unmodified lib version.

In your simulation of the bug, can you set this to False: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/77f2f7294b4b5d7fe21bcda50eb7c8e1beb41f10/unicorn_binance_websocket_api/sockets.py#LL73C45-L73C45

And tell me what then happens?

[dima-dmytruk23]

However, it usually takes between two and four days until the issue occurs and the script exits.

You don't have to wait a few days. You can reproduce the bug by throwing an websockets.exceptions.ConnectionClosed exception to start_socket method of the BinanceWebSocketApiSocket somewhere around this place https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/sockets.py#L116

Please test #299 (comment) with the latest and unmodified lib version.

In your simulation of the bug, can you set this to False: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/77f2f7294b4b5d7fe21bcda50eb7c8e1beb41f10/unicorn_binance_websocket_api/sockets.py#LL73C45-L73C45

And tell me what then happens?

It seems updating to version 1.42.0 and setting self.manager.socket_is_ready[self.stream_id] = False solved the problem.

[dima-dmytruk23]

Is it worth waiting for a fix inside the library?

[oliver-zehentleitner]

We easily could add a flag to bypass "self.manager.socket_is_ready[self.stream_id] = True". So a fix would be pretty easy. But adding this has also a downside, becouse now you have an unlimited blocking create_stream() function isnt it?

And i really beflief its fixed on the wrong end.... i bet that now also a code structure similar to #299 (comment) fixes the problem.

[dima-dmytruk23]

И я действительно верю, что это исправлено не на том конце... Держу пари, что теперь также структура кода, подобная # 299 (комментарий), устраняет проблему.

No, I didn't even use while True: sleep(1), but the problem is solved)

[dima-dmytruk23]

But adding this has also a downside, becouse now you have an unlimited blocking create_stream() function isnt it?

Maybe add it to __init__ ?

[oliver-zehentleitner]

И я действительно верю, что это исправлено не на том конце... Держу пари, что теперь также структура кода, подобная # 299 (комментарий), устраняет проблему.

No, I didn't even use while True: sleep(1), but the problem is solved)

Ok, but the framework I provided in #299 (comment) is maybe the smarter way than hacking the lib 😄

[dima-dmytruk23]

И я действительно верю, что это исправлено не на том конце... Держу пари, что теперь также структура кода, подобная # 299 (комментарий), устраняет проблему.

No, I didn't even use while True: sleep(1), but the problem is solved)

Ok, but the framework I provided in #299 (comment) is maybe the smarter way than hacking the lib 😄

Agreed, but the while True: sleep(1) solution seems very dirty too

[oliver-zehentleitner]

I also prefer:

        while True:
            await asyncio.sleep(1)

[oliver-zehentleitner]

I leave this for now as it is and wait for more feedback of the community...

[oliver-zehentleitner]

any news, did my suggestion solve your issues?

[dima-dmytruk23]

any news, did my suggestion solve your issues?

Yes, for more than a week about 10 stages have been working continuously.

[oliver-zehentleitner]

Cool. Then i will close this. Reopen if needed. Thanks!