module 'asyncio.base_futures' has no attribute 'InvalidStateError'

[Flowelfox]

Check this or we will delete your issue. (fill in the checkbox with an X like so: [x])

• I have searched for other issues with the same problem or similar feature requests.
  (I found one old issue with same problem asyncio.base_futures.InvalidStateError: invalid state #18, but bug still exists.)

Select one:

• Bug
• Feature Request
• Technical Help
• Other

Environment

• Are you using the module on a VPS or other Cloud hosting?
• If so, have you tried running the code on your local machine?
• Are you using the module on a Raspberry Pi?

Operating System? (include version)

• macOS
• Windows
• Linux (Ubuntu 18.04.4 LTS)

Python Version Requirement

• [X ] I am using Python 3.6.1 or above

Exact Python Version?

Python 3.8.3

Pip Version?

pip 20.1.1

Dependencies

Package                       Version    Location
----------------------------- ---------- ---------
alembic                       1.4.2
aniso8601                     8.0.0
attrs                         19.3.0
autobahn                      20.4.3
Automat                       20.2.0
certifi                       2020.4.5.1
cffi                          1.14.0
chardet                       3.0.4
cheroot                       8.3.0
click                         7.1.2
colorama                      0.4.3
constantly                    15.1.0
cryptography                  2.9.2
dateparser                    0.7.4
Flask                         1.1.2
Flask-RESTful                 0.3.8
flask-restx                   0.2.0
Flask-SQLAlchemy              2.4.3
hyperlink                     19.0.0
idna                          2.9
incremental                   17.5.0
inprofitapi                   0.1        /inprofit
itsdangerous                  1.1.0
jaraco.functools              3.0.1
Jinja2                        2.11.2
jsonschema                    3.2.0
Mako                          1.1.3
MarkupSafe                    1.1.1
more-itertools                8.3.0
pathlib                       1.0.1
pip                           20.1.1
psutil                        5.7.0
psycopg2-binary               2.8.5
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.20
PyHamcrest                    2.0.2
pyOpenSSL                     19.1.0
pyrsistent                    0.16.0
python-binance                0.7.5
python-dateutil               2.8.1
python-editor                 1.0.4
pytz                          2020.1
regex                         2020.5.14
requests                      2.23.0
service-identity              18.1.0
setuptools                    47.1.1
six                           1.15.0
SQLAlchemy                    1.3.17
Twisted                       20.3.0
txaio                         20.4.1
tzlocal                       2.1
ujson                         3.0.0
unicorn-binance-websocket-api 1.15.0
unicorn-fy                    0.4.1
urllib3                       1.25.9
websocket-client              0.57.0
websockets                    8.1
Werkzeug                      1.0.1
wheel                         0.34.2
zope.interface                5.1.0

UNICORN Binance WebSocket API Version?

Did you upgrade to the latest release version with `pip install unicorn-binance-websocket-api --upgrade`?
No. I am using version 1.15.0

Description of your issue

Debug log when streams crashed:

2020-06-08 16:05:14,701 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 16:35:14,721 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 16:35:15,461 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 16:53:47,277 - root - DEBUG - BinanceWebSocketApiSocket->start_socket(76622917-8447-4e2a-b723-c2df11feebc2, ['kline_1m', 'kline_3m', 'kline_5m', 'kline_15m', 'kline_30m', 'kline_1h', 'kline_2h', 'kline_4h', 'kline_6h', 'kline_8h', 'kline_12h', 'kline_1d'], ['btcusdt', 'ethusdt', 'bchusdt', 'xrpusdt', 'eosusdt', 'ltcusdt', 'trxusdt', 'etcusdt', 'linkusdt', 'xlmusdt', 'adausdt', 'xmrusdt', 'dashusdt', 'zecusdt', 'xtzusdt', 'bnbusdt', 'atomusdt', 'ontusdt', 'iotausdt', 'batusdt', 'vetusdt', 'neousdt', 'qtumusdt', 'iostusdt']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'
2020-06-08 17:05:15,521 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 17:05:15,928 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 17:35:15,973 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 17:35:16,543 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 17:46:30,840 - root - DEBUG - BinanceWebSocketApiSocket->start_socket(27264e1f-532d-4293-933c-38e6f02cb0ad, ['arr'], ['!miniTicker']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'
2020-06-08 17:46:31,427 - root - DEBUG - BinanceWebSocketApiSocket->start_socket(2ca0094e-8dea-46cf-af8d-8ac704876335, ['arr'], ['!userData']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'
2020-06-08 18:05:16,621 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 18:05:17,024 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 18:35:17,065 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 18:35:17,539 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 19:05:17,635 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)

Kline stream, and then other websocket streams (miniTicker, UserData) will shutdown without any error, exception after some time of work. It can happen in hours or days. Last time it worked almost 5 days.
I found some info about what happened in debug log, i added log above.
Last candle received in 16:53. After that no new data was added to database.
I assume that i must get exception from unicorn_binance_websocket_api_connection.py:268, but somehow i didn't have InvalidStateError.
Also it worries me because it was shutted down silently. No errors, no critical error or anything else. I was 

I ask for help to understand what caused this issue, my code or something wrong in library?
How i can avoid getting this error and stream crashing?

[Flowelfox]

I found that in Python 3.8.0 and above InvalidStateError does not exists:

(.venv3.8) flowelcat@Flowelcat-PC:~/PycharmProjects/inprofitapi$ python test.py 
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    from asyncio.futures import InvalidStateError
ImportError: cannot import name 'InvalidStateError' from 'asyncio.futures' (/usr/lib/python3.8/asyncio/futures.py)
(.venv3.8) flowelcat@Flowelcat-PC:~/PycharmProjects/inprofitapi$ python --version
Python 3.8.3

[oliver-zehentleitner]

Thanks for the good report!
I added a restart handling for this error, please can you test it?
pip install https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/tarball/master --upgrade
Please proof that you are on 1.16.1.dev!

I am not shure if the code I added is really good, maybe we have to filter out some other attribute errors... lets test it :)

Best regards,
Oliver

[Flowelfox]

Hi, updated library and restarted application recently.

Version proof:

inprofitapi | 2020-06-09 20:04:38,817 - root - INFO - Initializing flask application...
inprofitapi | 2020-06-09 20:04:38,891 - root - INFO - Setting up kline subscription...
inprofitapi | 2020-06-09 20:04:38,892 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1 started ...
inprofitapi | 2020-06-09 20:04:39,031 - root - INFO - Starting missing candles loader...
inprofitapi | 2020-06-09 20:04:39,036 - root - INFO - Setting up mark price subscription...
inprofitapi | 2020-06-09 20:04:39,038 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1 started ...
inprofitapi | 2020-06-09 20:04:39,183 - root - INFO - Setting up mini ticker subscription...
inprofitapi | 2020-06-09 20:04:39,184 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1 started ...
inprofitapi | 2020-06-09 20:04:39,284 - root - INFO - Setting up user data subscription...
inprofitapi | 2020-06-09 20:04:39,300 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1 started ...
inprofitapi | 2020-06-09 20:04:39,431 - root - INFO - Starting flask application...

And

flowelcat@vps729526:~/inprofit$ docker container exec -it inprofitapi python
Python 3.8.3 (default, Jun  3 2020, 19:49:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from unicorn_binance_websocket_api.unicorn_binance_websocket_api_manager import BinanceWebSocketApiManager
>>> 
>>> binance_websocket_api_manager = BinanceWebSocketApiManager()
>>> print(binance_websocket_api_manager.get_version())
1.16.1
>>> 

A will monitor logs and candles from stream and notify here if something goes wrong.

[oliver-zehentleitner]

1.16.1 is WRONG! It must be 1.16.1.dev!

Try to deinstall all unicorn installations via pip and then run the installation of the dev stage again!

[Flowelfox]

Ok, changed version to 1.16.1.dev.

inprofitapi | 2020-06-10 09:51:42,228 - root - INFO - Initializing flask application...
inprofitapi | 2020-06-10 09:51:42,313 - root - INFO - Setting up kline subscription...
inprofitapi | 2020-06-10 09:51:42,315 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1.dev started ...
inprofitapi | 2020-06-10 09:51:42,593 - root - INFO - Starting missing candles loader...
inprofitapi | 2020-06-10 09:51:42,598 - root - INFO - Setting up mark price subscription...
inprofitapi | 2020-06-10 09:51:42,599 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1.dev started ...
inprofitapi | 2020-06-10 09:51:42,738 - root - INFO - Setting up mini ticker subscription...
inprofitapi | 2020-06-10 09:51:42,740 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1.dev started ...
inprofitapi | 2020-06-10 09:51:42,858 - root - INFO - Setting up user data subscription...
inprofitapi | 2020-06-10 09:51:42,886 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1.dev started ...
inprofitapi | 2020-06-10 09:51:43,047 - root - INFO - Starting flask application...

[oliver-zehentleitner]

This fix is included now in 1.16.2 release!

[oliver-zehentleitner]

Can we close this issue? I think its successfully fixed.

[Flowelfox]

I think yes, no problems so far. Thank you for quick fix)

[oliver-zehentleitner]

Thanks for the good report, I think this fix made the lib stable, I was hunting this problem since a while :D (#45)

[oliver-zehentleitner]

Its so weired. 8 days no issue, then I write its fixed and then it happens again:

4 streams stopped. I investigate it asap.

[alekkhalex]

I have same issue. A stream is crashing time after time:

ERROR:root:BinanceWebSocketApiSocket->start_socket(d1f4bcc1-092d-4e63-bbda-2003ec246d48, ['arr'], ['!userData']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'
CRITICAL:root:BinanceWebSocketApiManager->stream_is_crashing(d1f4bcc1-092d-4e63-bbda-2003ec246d48)

Is it possible to fix it?

Python 3.8.5
unicorn-binance-websocket-api 1.17.0
Windows 10

[tahaEntesari]

same issue here. using 1.16.9 with python 3.8.1 on windows 10

[oliver-zehentleitner]

I am not sure, but it seems the InvalidSateError happens if you have to less system powser (CPU).

Can you test it by making less streams or use a more powerful system?

[tahaEntesari]

I am running this program on a server and the cpu useage is typically low. It won't go beyong 50% no matter what happens.
I have only two streams per user and these are the userData streams for SPOT and FUTURES
At the moment there are 5 users that my program is keeping track of though. So, there are 5 different instances of websocket running in parallel each containing 2 streams.
I don't think that I can personally test it. I mean, the program is under load and trading in binance and I can't take it down, at least not in a few days

[oliver-zehentleitner]

i am running different test systems of this lib since one year. on my systems i recognized even 50% on average is very much. sometimes are peaks that are too much then.

Running on a raspberry pi doenst work in general, dont know why.

You just need one instance per endpoint (spot/futures/...), doesnt matter how much users you have.

So your best setup should be two instances, one for spot and one for futures.

then you can create 5 spot userData streams within one instance and 5 futures userData streams.

you can split the stream_buffer, look at this example https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/example_multiple_userdata_streams.py so you know which stream has received the data.

Till now you have had to use set_private_api_config() but its not thread safe and i added the possibility to add the key and secret to create_stream() see here: https://oliver-zehentleitner.github.io/unicorn-binance-websocket-api/unicorn_binance_websocket_api.html?highlight=create_stream#unicorn_binance_websocket_api.unicorn_binance_websocket_api_manager.BinanceWebSocketApiManager.create_stream

BUT: As you can see here #109 the new way of providing key and secret doenst work now, i try to fix it this weekend.

[oliver-zehentleitner]

Its fixed providing key and secret via create_stream()!

Just update to 1.17.1

[alekkhalex]

Still facing same issue. A stream is crashing and reconnecting time after time:

ERROR:root:BinanceWebSocketApiSocket->start_socket(d1f4bcc1-092d-4e63-bbda-2003ec246d48, ['arr'], ['!userData']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'
CRITICAL:root:BinanceWebSocketApiManager->stream_is_crashing(d1f4bcc1-092d-4e63-bbda-2003ec246d48)

Is it possible to fix it?

Python 3.8.5
unicorn-binance-websocket-api 1.17.1
Windows 2019 Server, 6 GB RAM, 4x 3.4ГГц CPU

[oliver-zehentleitner]

I will remove the InvalidStateError code part - it was a fix for an other problem. Lets make one step back.

[tahaEntesari]

@oliver-zehentleitner
The odd thing is that this process is somewhat random. I mean, there are times that it happens like once every few hours but other times, its not that frequent.
Since the stream is crashing each time, I think the websocket gets disconnected and thus any data sent in between the reconnect is lost (I have had inconsistent data only once in like 2 month but, I only use userData so I don't expect that much frequent data). So I think this is a somewhat important bug.
One more thing. You added the parameters apiKey and apiSecret to create_stream but you have not added them to replace_stream(). I didn't check other places that it might have been useful.

[alekkhalex]

get_stream_info(stream_id) returns 'reconnects' = 1 after any qty of 'InvalidStateError' reconnects. Each time 'reconnects' = 1.

'logged_reconnects' = [..last reconnect timestamp..] - has only one element in the array of the last reconnection. May be it should be a list of timestamps of all reconnects?

Python 3.8.5
unicorn-binance-websocket-api 1.17.1
Windows 2019 Server

[oliver-zehentleitner]

@tahaEntesari
Thanks for the hint about replace_stream() - i add it to the todo list.
About disconnects in general: #42
But I am with you, we want it as stable as possible :)

@alekkhalex
i recogniced that too, this bugs is new since 1.17.0 - also added to todo list.

Its much more handy if there is one issue for one topic. Please think about that in the future! And open new one if needed!

[oliver-zehentleitner]

InvalidStateError: #110

[oliver-zehentleitner]

@tahaEntesar
@alekkhalex

Both fixes are released now in 1.17.4

[alekkhalex]

Now all that reconnects have another message:

CRITICAL:root:BinanceWebSocketApiSocket->start_socket(09655d52-c65d-4ea3-9df7-5e242dc2577d, ['arr'], ['!userData']) Exception ConnectionClosed Info: code = 1006 (connection closed abnormally [internal]), no reason

From Binance documentation:
-1006 UNEXPECTED_RESP
An unexpected response was received from the message bus. Execution status unknown.

Python 3.8.5
unicorn-binance-websocket-api 1.17.4
Windows 2019 Server

[oliver-zehentleitner]

#113