Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data received gets less and less. #45

Closed
fsteckel opened this issue Mar 9, 2020 · 66 comments
Closed

Data received gets less and less. #45

fsteckel opened this issue Mar 9, 2020 · 66 comments
Assignees
@oliver-zehentleitner
Labels
bug Something isn't working
Projects
Todo

Comments

@fsteckel
Copy link

fsteckel commented Mar 9, 2020

Hello again,

I report a bug. python version 3.7.5, pip 18.1, ubuntu 19.10 in a local docker container with unicorn binance 1.11.0.

I'm sorry, this is rather unspecific: I posted another issure before and attached a log file. There you stated, you would rather put all the streams of the symbols into one single stream in oppose to my seperating each and every stream. Then I did not rember why I seperated them in the first place. So I did unseperate them and now have two streams, one for depth, one for trade. Each containing all the symbols.

Something strange happens:
No exceptions. Everything seams ok. I collect all the message frames and pack them into sql files (one per day). I started off on the first day with a file of about 6-8GB. Second day 400MB, third 300, fourth 250MB. Less and less data is being packed into the sql files.
This (sql) system has worked before without this error, thats why I suspect the unicorn package (which otherwise is great).
Encountering this behavior in the past drove me into seperating the streams in order to check each streams
stream_info['status'] != 'running'
and also
time.time() - stream_info['last_heartbeat'] > self.stream_heartbeat_tolerance.

One (maybe) reason:
Is the 'running' flag set to running if at least one of the symbols in the stream is still being updated? Then it would yield a false sense of running. Same goes for the last heartbeat. This should better be the last heartbeat of the least updated symbol.

My suspicions may be totally off and I appologize for this rather unspecific 'bug'.
Thanks for any helf you can provide Oliver!
@fsteckel fsteckel added the bug Something isn't working label Mar 9, 2020
@fsteckel
Copy link
Author

fsteckel commented Mar 16, 2020 •
edited
Loading

This is a very regular error message after a while:
CRITICAL:root:BinanceWebSocketApiSocket->start_socket(f6295724-2949-4caa-89d6-25430b3389ac, ['depth@100ms'], ['gvtbtc', 'funbtc', 'sntbtc', ..., 'tntbtc', 'dockusdt', 'funusdt', 'stormbtc', 'winusdt', 'perlbtc', 'etcbtc', 'renusdt', 'ontbtc', 'qtumbtc', 'dentusdt', 'engbtc']) Exception ConnectionClosed Info: code = 1006 (connection closed abnormally [internal]), no reason CRITICAL:root:BinanceWebSocketApiManager->stream_is_crashing(f6295724-2949-4caa-89d6-25430b3389ac) INFO:logger_logger:Mar 15 2020 20:22:17 UTC: Stream depth@100ms: Status != "running". Stream wird neu gestartet. API request wird gestellt.
Last mgs being my code checking the stream status and deciding it has crashed.

@oliver-zehentleitner
Copy link
Member

Hi Flo!

I think the first error is ok. 1006 is very common and handled by the lib!
Is the status running determined from your code or is it a result of get_stream_info()?
Would be great if you could add timestamps to the logs :)

I think I know where the problem is happening, I was observing similar behaviour and its a new issue which was not there up to 1.9.x

In 1.10.0 I started releasing methods for subscriptions which are using websocket.send() and I think the lib is catching a specific exception (no error trace in stdin) but the code doesnt handle it the right way. I will add more logging to the code on level critical, so we might get an insight then.

Best regards!

@oliver-zehentleitner
Copy link
Member

I gues its about this section of code: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/ff5fff1801baf518bbda746d365107f1773dab40/unicorn_binance_websocket_api/unicorn_binance_websocket_api_connection.py#L251

async def send(self, data):
        try:
            await self.handler_binance_websocket_api_manager.websocket_list[self.stream_id].send(data)
            self.handler_binance_websocket_api_manager.increase_transmitted_counter(self.stream_id)
        except websockets.exceptions.ConnectionClosed as error_msg:
            logging.error("BinanceWebSocketApiSocket->send(" + str(self.stream_id) + ", " +
                          str(self.channels) + ", " + str(self.markets) + ") Exception ConnectionClosed "
                          "Info: " + str(error_msg))
        except RuntimeError as error_msg:
            logging.error("BinanceWebSocketApiSocket->send(" + str(self.stream_id) + ", " +
                          str(self.channels) + ", " + str(self.markets) + ") Exception RuntimeError "
                          "Info: " + str(error_msg))
        except IndexError as error_msg:
            logging.error("BinanceWebSocketApiSocket->send(" + str(self.stream_id) + ", " +
                          str(self.channels) + ", " + str(self.markets) + ") Exception IndexError "
                          "Info: " + str(error_msg))
        except KeyError as error_msg:
            logging.error("BinanceWebSocketApiSocket->send(" + str(self.stream_id) + ", " +
                          str(self.channels) + ", " + str(self.markets) + ") Exception KeyError "
                          "Info: " + str(error_msg))

I set loglevels to critical, with level error its to noisy in my configuration.
pip install https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/tarball/master --upgrade

Maybe then we can see the reason and let the lib handle the restart.

@oliver-zehentleitner
Copy link
Member

I posted another issure before and attached a log file. There you stated, you would rather put all the streams of the symbols into one single stream in oppose to my seperating each and every stream. Then I did not rember why I seperated them in the first place. So I did unseperate them and now have two streams, one for depth, one for trade. Each containing all the symbols.

I remember that and from the view of the OS it seems to be right. But as I mentioned I am observing a similar issue with this script on a test server: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/dev_test_cex_full_non_stop.py

All streams are fine all the time, but the mega stream is dying but remains as "running" in the stream_info output. Logfiles are not referring to a reason.

So maybe its possible that a stream is becoming to big for any reason - yes we can handle it with an exception, but why its not happening with small streams in my setup, only the one big one...

Till now we didnt create streams with a subscription list longer than 8004 chars. In my test i am using 22k chars i guess.

I recognized a bug when sending a subscription via websocket.send() to binance. The binance websocket server splitted to big requests into smaller requests and was not able anymore to interpret the json structure, it always sent back a msg about invalid JSON syntax... so I decided to split the subscriptions into small JSON packages (max 350 items per subscription send) then it worked fine. Could be good to investigate if binance is sending a too request to us, maybe including a list of all subscriptions ...

finally: I am not shure anymore if my thesis is really allways valid! Maybe its not good to use a single stream for every kline, but maybe its also good to make a stream not too big.

@fsteckel
Copy link
Author

Thanks I'll check out this version with the big and the single streams. Ill post the results here.

@oliver-zehentleitner
Copy link
Member

oliver-zehentleitner commented Apr 19, 2020 •
edited
Loading

Maybe this solves this issue: #47 (comment)

Closed
8 tasks
@oliver-zehentleitner
Copy link
Member

oliver-zehentleitner commented Apr 25, 2020 •
edited
Loading

Regarding to Binance API Changelog of 2020-04-23 (thx to @jemeraldo for reprting)

WEB SOCKET STREAM

WebSocket connections have a limit of 5 incoming messages per second. A message is considered:
    A PING frame
    A PONG frame
    A JSON control message (e.g. subscribe, unsubscribe)
A connection that goes beyond the limit will be disconnected; IPs that are repeatedly disconnected may be banned.
A single connection can listen to a maximum of 1024 streams.

... we have two tasks:

  1. slow down subscriptions to a max of 5 per second
  2. handle the max of 1024 subscriptions per stream

oliver-zehentleitner added a commit that referenced this issue Apr 25, 2020
@oliver-zehentleitner
working on max limit issue #45 - not ready
85f52f6
oliver-zehentleitner added a commit that referenced this issue Apr 27, 2020
@oliver-zehentleitner
working on max limit issue #45 - not ready
255e8c5
oliver-zehentleitner added a commit that referenced this issue Apr 27, 2020
@oliver-zehentleitner
https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api…
32dc540 
…/issues/45
@oliver-zehentleitner
Copy link
Member

I released version 1.12.0 on pypi. I think we can close this issue.

This was referenced Apr 27, 2020
Closed
Closed
@oliver-zehentleitner
Copy link
Member

oliver-zehentleitner commented May 1, 2020 •
edited
Loading

Unfortunately the 1.12.0 update didnt fix this problem.

I set up a test server that is running example_stream_everything.py and the console output is shown here.

The first version was streaming < 1024 subscriptions per stream but i made multichannel streams (all kind of channels * subset of markets =<1024).

Now I changed it to 1 kind of channel * all available markets (~740) and run the test again.

@oliver-zehentleitner oliver-zehentleitner added this to Suspended in Todo May 2, 2020
@oliver-zehentleitner
Copy link
Member

I made a few tests and it seems to depend on how you create the streams!

If I use one channel type mixed with all markets its fully stable.

https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/example_stream_everything.py

@oliver-zehentleitner oliver-zehentleitner moved this from Suspended to Done in Todo May 6, 2020
@oliver-zehentleitner
Copy link
Member

3 days streaming everything without a problem.

print_summary

Now 3 streams are still alive but 0 receives after a restart.

  • trade stream with all markets
  • kline_6h stream with all markets
  • kline_1w stream with all markets

All 3 streams got restarted within the same second, but it looks like they never started streaming again.

Error msg of all 3 streams on restart:
Exception ConnectionClosed Info: code = 1006 (connection closed abnormally [internal]), no reason

No error msg received from binance.

The log looks like, there are 3 frequent checks threads running, but it should be only one!

The log file:example_stream_everything.py.log

@oliver-zehentleitner oliver-zehentleitner moved this from Done to Suspended in Todo May 7, 2020
@fsteckel
Copy link
Author

fsteckel commented May 8, 2020

This very much aligns to what I am seeing. Streams are restarted from time to time and my received data becomes less and less. Thanks Oliver. I now know, that I still am sane ;)

@oliver-zehentleitner
Copy link
Member

Your welcome :D

I am implementing new methods for a better management and more options to debug this things: #62

I am on it ...

@fsteckel
Copy link
Author

Oh ok. Was about to answer. Sorry for the long delay! Will completely uninstall and then reinstall the dev0. Will start again and post log messages.

@fsteckel
Copy link
Author

Oh ok. Was about to answer. Sorry for the long delay! Will completely uninstall and then reinstall the dev0. Will start again and post log messages.

Or should I install 1.16.2?

@oliver-zehentleitner
Copy link
Member

the best is this:
pip install https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/tarball/master --upgrade
1.6.2.dev

@fsteckel
Copy link
Author

Ok. Installed and checked 1.6.2.dev. Running now.

@fsteckel
Copy link
Author

fsteckel commented Jul 6, 2020

Error in my code. Restarting with unicorn-binance-websocket-api-1.16.3.dev0

@oliver-zehentleitner
Copy link
Member

so it worked for 19 days and the error was in your app not in the unicorn lib?

@fsteckel
Copy link
Author

fsteckel commented Jul 6, 2020

nope. It crashed because out of memory due to unicorn not relaying the packets (I guess) first. Then I restarted (with the new version) and went on holyday. On the first day it crashed because stream_info['last_heartbeat'] was None while my code expected a float.
So it went 2 Weeks without doing anything.

@fsteckel
Copy link
Author

fsteckel commented Jul 9, 2020

Ok, ran for 4 days now. Now again RAM usage is increasing while data-amount saved is decreasing. I get the following errors from time to time:

CRITICAL:root:BinanceWebSocketApiManager->stream_is_crashing(ba7d67...8890a)

CRITICAL:root:BinanceWebSocketApiSocket->start_socket(ba7d67...8890a, ['depth@100ms'], ['mithbtc', ... 'ftmusdt']) Exception ConnectionClosed Info: code = 1006 (connection closed abnormally [internal]), no reason

CRITICAL:root:BinanceWebSocketApiConnection->await._conn.aenter(ba7d67...8890a, ['depth@100ms'], ['mithbtc', ...'ftmusdt']) - OSError - [Errno -3] Temporary failure in name resolution

ERROR:root:binance_websocket_api_connection->aexit(*args, **kwargs): AttributeError - 'Connect' object has no attribute 'ws_client'

@oliver-zehentleitner
Copy link
Member

i think we solved a lot of bugs till now :D

And i can still see 2 issues.

  1. after every restart of a stream the RAM increase to much! this needs an investigation.
  2. I had 3 streams some days ago, that did a restart, get result msg from binance - subscriptions are successfull but then the stream thread exits without any message/info.

Would be really cool if someone could help me out with create a pyflame or backfire log the see the size of vars during the runtime. With this info its easy to solve problem 1.

@oliver-zehentleitner
Copy link
Member

print_summary

@oliver-zehentleitner
Copy link
Member

In 1.16.4 I have no issues anymore and also RAM usage does not go wild...
no-error

@oliver-zehentleitner
Copy link
Member

Ok, ran for 4 days now. Now again RAM usage is increasing while data-amount saved is decreasing. I get the following errors from time to time:

CRITICAL:root:BinanceWebSocketApiManager->stream_is_crashing(ba7d67...8890a)

CRITICAL:root:BinanceWebSocketApiSocket->start_socket(ba7d67...8890a, ['depth@100ms'], ['mithbtc', ... 'ftmusdt']) Exception ConnectionClosed Info: code = 1006 (connection closed abnormally [internal]), no reason

CRITICAL:root:BinanceWebSocketApiConnection->await._conn.aenter(ba7d67...8890a, ['depth@100ms'], ['mithbtc', ...'ftmusdt']) - OSError - [Errno -3] Temporary failure in name resolution

error 1006 happens and the lib is restarting the stream in this case...

"OSError Temporary failure in name resolution" the operating system was not able to resolve the hostname from binance websocket endpoints (no internet connection, DNS down, ...)

@fsteckel
Copy link
Author

Ok merci :) will start test tomorrow.

@oliver-zehentleitner oliver-zehentleitner moved this from Suspended to Done in Todo Jul 27, 2020
@oliver-zehentleitner
Copy link
Member

oliver-zehentleitner commented Jul 28, 2020 •
edited
Loading

After 13 days I had an issue in.

the depth@100ms stream received only 10 receives in average instead of ~1000 per second. I did set_restart_request(stream_id) and after the restart it worked fine again.

I am investigating this!

@fsteckel
Copy link
Author

Jap. And I got a new error:

CRITICAL:root:BinanceWebSocketApiSocket->start_socket(5eea96b3-5b55-4fc3-86f5-d15822985931, ['depth@100ms'], ['manabtc', ..., 'iotabtc']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'

@oliver-zehentleitner
Copy link
Member

Errors are happening, I cant change that!!!

What I can try is to catch them and handle them!

The error you posted should have been catched and the stream should be restarted again! Isnt it?

@fsteckel
Copy link
Author

Sorry, there were not ment to complain, but merely as a - somewhat tiny - chance to guide to the source of the problem. Yes it did restart afterwards. However as you said, the RAM usage still increased. And not as in your case after 13 days, but after one day.

@oliver-zehentleitner
Copy link
Member

Hi Florian!

No worries! I didnt understand it as a complaint! I just thought its not a problem of unicorn lib, we handle it and we cant do anything more. But now I looked into the code and I see I was wrong :)

I think the error is catched here:
https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/unicorn_binance_websocket_api_socket.py#L118

The error asyncio.base...blahblah... InvalidStateError is happening inside await websocket.receive() in the try-block a few lines above which leads to: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/unicorn_binance_websocket_api_connection.py#L260

I will continue the investigation in a couple of days, I am in holiday mode :)

Thanks for reporting it, even if I didnt recognized its value in the first reading!

The RAM ... I need to know which objects got bloated or are bloating the RAM usage. There are a couple of ways to do that. Maybe within an iPython shell or with blackfire or pyflame. Are you able help with a research when its happening again on your system?

That the RAM is growing over time is logical, because the lib is logging a lot of stuff, some vars are stacks which start cleaning after 500 entries for example. For sure thats something we can optimize.

best regards,
Oliver

@fsteckel
Copy link
Author

fsteckel commented Aug 5, 2020

I'll try to do these pyflame things in a coup'la days!

@oliver-zehentleitner
Copy link
Member

A bug happend on ubwa demo:
print_summary

Its the 3rd time and allways ends with 480 Threads instead of 58...

@oliver-zehentleitner
Copy link
Member

This is the root error msg:

 2020-08-02 03:22:22,550 [ERROR   ] 902 140336486795008 base_events: Task exception was never retrieved
future: <Task finished coro=<Application._handle_exception.<locals>.in_term() done, defined at /usr/local/lib/python3.6/dist-packages/prompt_toolkit/application/application.py:829> exception=RuntimeError('Task <Task pending coro=<Application._handle_exception.<locals>.in_term() running at /usr/local/lib/python3.6/dist-packages/prompt_toolkit/application/application.py:830>> got Future <Future pending> attached to a different loop',)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit/application/application.py", line 830, in in_term
    async with in_terminal():
  File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit/eventloop/async_context_manager.py", line 57, in __aenter__
    return await self.gen.__anext__()
  File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit/application/run_in_terminal.py", line 86, in in_terminal
    await previous_run_in_terminal_f
RuntimeError: Task <Task pending coro=<Application._handle_exception.<locals>.in_term() running at /usr/local/lib/python3.6/dist-packages/prompt_toolkit/application/application.py:830>> got Future <Future pending> attached to a different loop
2020-08-02 03:22:22,557 [CRITICAL] 902 140336495187712 unicorn_binance_websocket_api_connection: BinanceWebSocketApiConnection->await._conn.__aenter__(3a9ee23a-f82c-4624-a98c-483a95727a61, ['!userData'], ['arr']) - OSError - Multiple exceptions: [Errno 110] Connect call failed ('3.115.32.220', 9443), [Errno 24] Too many open files, [Errno 24] Too many open files, [Errno 24] Too many open files, [Errno 24] Too many open files, [Errno 24] Too many open files, [Errno 24] Too many open files, [Errno 24] Too many open files

I try to investigate what this open files are ...

@oliver-zehentleitner
Copy link
Member

oliver-zehentleitner commented Aug 14, 2020 •
edited
Loading

This are my open files of ubwa after a fresh start:

root@ubwa-demo:~# lsof -p 923
COMMAND PID USER   FD      TYPE             DEVICE SIZE/OFF   NODE NAME
python3 923 root  cwd       DIR                8,1     4096    269 /root
python3 923 root  rtd       DIR                8,1     4096      2 /
python3 923 root  txt       REG                8,1  4526456   6942 /usr/bin/python3.6
python3 923 root  mem       REG                8,1   101168   3233 /lib/x86_64-linux-gnu/libresolv-2.27.so
python3 923 root  mem       REG                8,1    26936   1418 /lib/x86_64-linux-gnu/libnss_dns-2.27.so
python3 923 root  mem       REG                8,1    47568   1419 /lib/x86_64-linux-gnu/libnss_files-2.27.so
python3 923 root  mem       REG                8,1  2123696 395718 /usr/local/lib/python3.6/dist-packages/regex/_regex.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    21328   6451 /usr/lib/python3.6/lib-dynload/_lsprof.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1  1082648   6254 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
python3 923 root  mem       REG                8,1    92280   6455 /usr/lib/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    15368   6466 /usr/lib/python3.6/lib-dynload/resource.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    24968   6467 /usr/lib/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    66008 528977 /usr/local/lib/python3.6/dist-packages/psutil/_psutil_posix.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1   108640 528978 /usr/local/lib/python3.6/dist-packages/psutil/_psutil_linux.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    38875 528187 /usr/local/lib/python3.6/dist-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    74664   6450 /usr/lib/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    27112   2391 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
python3 923 root  mem       REG                8,1    31032   7786 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
python3 923 root  mem       REG                8,1   126816   6444 /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1   227944   8049 /usr/lib/x86_64-linux-gnu/libmpdec.so.2.4.2
python3 923 root  mem       REG                8,1   175008   6449 /usr/lib/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    96616   1584 /lib/x86_64-linux-gnu/libgcc_s.so.1
python3 923 root  mem       REG                8,1  1594864   1583 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
python3 923 root  mem       REG                8,1   524265  15284 /usr/local/lib/python3.6/dist-packages/ujson.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    24744 393380 /usr/local/lib/python3.6/dist-packages/websockets/speedups.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1   577312   5997 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
python3 923 root  mem       REG                8,1   120088   3771 /usr/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    70400   6434 /usr/lib/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1     6280   3770 /usr/lib/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    15856   6454 /usr/lib/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1  2917216   5996 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
python3 923 root  mem       REG                8,1    29752   3769 /usr/lib/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1   153984   1747 /lib/x86_64-linux-gnu/liblzma.so.5.2.2
python3 923 root  mem       REG                8,1    33592   6452 /usr/lib/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1    66728   1668 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
python3 923 root  mem       REG                8,1    22040   6435 /usr/lib/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so
python3 923 root  mem       REG                8,1  1700792   1411 /lib/x86_64-linux-gnu/libm-2.27.so
python3 923 root  mem       REG                8,1   116960   3495 /lib/x86_64-linux-gnu/libz.so.1.2.11
python3 923 root  mem       REG                8,1   202880   4042 /lib/x86_64-linux-gnu/libexpat.so.1.6.7
python3 923 root  mem       REG                8,1    10592   3236 /lib/x86_64-linux-gnu/libutil-2.27.so
python3 923 root  mem       REG                8,1    14560   1410 /lib/x86_64-linux-gnu/libdl-2.27.so
python3 923 root  mem       REG                8,1   144976   3232 /lib/x86_64-linux-gnu/libpthread-2.27.so
python3 923 root  mem       REG                8,1  2030544   1407 /lib/x86_64-linux-gnu/libc-2.27.so
python3 923 root  mem       REG                8,1   170960   1402 /lib/x86_64-linux-gnu/ld-2.27.so
python3 923 root  mem       REG                8,1  1683056   2316 /usr/lib/locale/locale-archive
python3 923 root  mem       REG                8,1    26376   3970 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
python3 923 root    0u      CHR              136,0      0t0      3 /dev/pts/0
python3 923 root    1u      CHR              136,0      0t0      3 /dev/pts/0
python3 923 root    2u      CHR              136,0      0t0      3 /dev/pts/0
python3 923 root    3w      CHR                1,3      0t0      6 /dev/null
python3 923 root    4w      REG                8,1  3257360  16421 /root/server.py.log
python3 923 root    5u     IPv4              17638      0t0    TCP static.236.194.216.95.clients.your-server.de:55052->server-13-33-246-143.hel50.r.cloudfront.net:https (CLOSE_WAIT)
python3 923 root    6r      CHR                1,9      0t0     11 /dev/urandom
python3 923 root    7r      CHR                1,8      0t0     10 /dev/random
python3 923 root    8u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root    9u     unix 0xffff8a47b5908000      0t0  18837 type=STREAM
python3 923 root   10u     unix 0xffff8a47b5908800      0t0  18838 type=STREAM
python3 923 root   11u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   12u     unix 0xffff8a47b590b000      0t0  18839 type=STREAM
python3 923 root   13u     unix 0xffff8a47b5908400      0t0  18840 type=STREAM
python3 923 root   14u     IPv4              18925      0t0    TCP static.236.194.216.95.clients.your-server.de:41542->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   15u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   16u     IPv4              17647      0t0    TCP static.236.194.216.95.clients.your-server.de:41500->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   17u     unix 0xffff8a47b5909800      0t0  18841 type=STREAM
python3 923 root   18u     unix 0xffff8a47b590b800      0t0  18842 type=STREAM
python3 923 root   19u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   20u     unix 0xffff8a47b5909000      0t0  18843 type=STREAM
python3 923 root   21u     unix 0xffff8a47b590a800      0t0  18844 type=STREAM
python3 923 root   22u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   23u     unix 0xffff8a47f5f0a800      0t0  17648 type=STREAM
python3 923 root   24u     unix 0xffff8a47f5f09400      0t0  17649 type=STREAM
python3 923 root   25u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   26u     unix 0xffff8a47f5f0bc00      0t0  17650 type=STREAM
python3 923 root   27u     unix 0xffff8a47f5f09800      0t0  17651 type=STREAM
python3 923 root   28u     IPv4              18848      0t0    TCP static.236.194.216.95.clients.your-server.de:41502->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   29u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   30u     unix 0xffff8a47f595b000      0t0  18849 type=STREAM
python3 923 root   31u     unix 0xffff8a47b5908c00      0t0  18850 type=STREAM
python3 923 root   32u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   33u     unix 0xffff8a47f5f0b000      0t0  17655 type=STREAM
python3 923 root   34u     unix 0xffff8a47f5f08000      0t0  17656 type=STREAM
python3 923 root   35u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   36u     unix 0xffff8a47f5f08c00      0t0  17657 type=STREAM
python3 923 root   37u     unix 0xffff8a47f5f08400      0t0  17658 type=STREAM
python3 923 root   38u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   39u     unix 0xffff8a47f5f0b800      0t0  17659 type=STREAM
python3 923 root   40u     unix 0xffff8a47f5f0ac00      0t0  17660 type=STREAM
python3 923 root   41u     IPv4              17664      0t0    TCP static.236.194.216.95.clients.your-server.de:41504->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   42u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   43u     unix 0xffff8a47f61e5c00      0t0  17668 type=STREAM
python3 923 root   44u     unix 0xffff8a47f61e6400      0t0  17669 type=STREAM
python3 923 root   45u     IPv4              18851      0t0    TCP static.236.194.216.95.clients.your-server.de:41506->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   46u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   47u     unix 0xffff8a47f90d7800      0t0  18852 type=STREAM
python3 923 root   48u     unix 0xffff8a47b5d47000      0t0  18853 type=STREAM
python3 923 root   49u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   50u     unix 0xffff8a47b5d46400      0t0  18854 type=STREAM
python3 923 root   51u     unix 0xffff8a47b5d46000      0t0  18855 type=STREAM
python3 923 root   52u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   53u     unix 0xffff8a47f61e7000      0t0  17670 type=STREAM
python3 923 root   54u     unix 0xffff8a47f61e4c00      0t0  17671 type=STREAM
python3 923 root   55u     IPv4              17674      0t0    TCP static.236.194.216.95.clients.your-server.de:41508->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   56u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   57u     unix 0xffff8a47f61e5000      0t0  17672 type=STREAM
python3 923 root   58u     unix 0xffff8a47f61e5400      0t0  17673 type=STREAM
python3 923 root   59u     IPv4              18859      0t0    TCP static.236.194.216.95.clients.your-server.de:41510->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   60u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   61u     unix 0xffff8a47b5eac800      0t0  18860 type=STREAM
python3 923 root   62u     unix 0xffff8a47b5ead400      0t0  18861 type=STREAM
python3 923 root   63u     IPv4              17681      0t0    TCP static.236.194.216.95.clients.your-server.de:41512->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   64u     IPv4              18865      0t0    TCP static.236.194.216.95.clients.your-server.de:41514->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   65u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   66u     unix 0xffff8a47f61d7800      0t0  18866 type=STREAM
python3 923 root   67u     unix 0xffff8a47b5eacc00      0t0  18867 type=STREAM
python3 923 root   68u     IPv4              18874      0t0    TCP static.236.194.216.95.clients.your-server.de:41518->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   69u     IPv4              17682      0t0    TCP static.236.194.216.95.clients.your-server.de:41516->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   70u     IPv4              17683      0t0    TCP static.236.194.216.95.clients.your-server.de:41520->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   71u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   72u     unix 0xffff8a47f5276400      0t0  18878 type=STREAM
python3 923 root   73u     unix 0xffff8a47f5277400      0t0  18879 type=STREAM
python3 923 root   74u     IPv4              18880      0t0    TCP static.236.194.216.95.clients.your-server.de:41522->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   75u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   76u     unix 0xffff8a47f82fa400      0t0 268257 type=STREAM
python3 923 root   77u     unix 0xffff8a47f82f9000      0t0 268258 type=STREAM
python3 923 root   78u     IPv4              18883      0t0    TCP static.236.194.216.95.clients.your-server.de:41524->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   79u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   80u     unix 0xffff8a47f5ea9800      0t0  17690 type=STREAM
python3 923 root   81u     unix 0xffff8a47f5ea8000      0t0  17691 type=STREAM
python3 923 root   82u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   83u     unix 0xffff8a47f5221400      0t0  17692 type=STREAM
python3 923 root   84u     unix 0xffff8a47f5223000      0t0  17693 type=STREAM
python3 923 root   85u     IPv4              17697      0t0    TCP static.236.194.216.95.clients.your-server.de:41526->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   86u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   87u     unix 0xffff8a47f5277000      0t0  18884 type=STREAM
python3 923 root   88u     unix 0xffff8a47f5274400      0t0  18885 type=STREAM
python3 923 root   89u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   90u     unix 0xffff8a47f5277c00      0t0  18886 type=STREAM
python3 923 root   91u     unix 0xffff8a47f5275800      0t0  18887 type=STREAM
python3 923 root   92u     IPv4              17701      0t0    TCP static.236.194.216.95.clients.your-server.de:41528->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   93u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   94u     unix 0xffff8a47f5274c00      0t0  18888 type=STREAM
python3 923 root   95u     unix 0xffff8a47f5276c00      0t0  18889 type=STREAM
python3 923 root   96u     IPv4              17706      0t0    TCP static.236.194.216.95.clients.your-server.de:41530->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   97u     IPv4              18910      0t0    TCP static.236.194.216.95.clients.your-server.de:41532->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root   98u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root   99u     unix 0xffff8a47f5bd6800      0t0  18911 type=STREAM
python3 923 root  100u     unix 0xffff8a47f5bd4400      0t0  18912 type=STREAM
python3 923 root  101u     IPv4              17710      0t0    TCP static.236.194.216.95.clients.your-server.de:41534->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root  102u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root  103u     unix 0xffff8a47f5bd7c00      0t0  18913 type=STREAM
python3 923 root  104u     unix 0xffff8a47f5bd5800      0t0  18914 type=STREAM
python3 923 root  105u     IPv4              17711      0t0    TCP static.236.194.216.95.clients.your-server.de:41536->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root  106u      REG                8,1    53248 524102 /root/.ipython/profile_default/history.sqlite
python3 923 root  107u     IPv4             268266      0t0    TCP static.236.194.216.95.clients.your-server.de:40822->ec2-13-114-69-146.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root  108u      REG                8,1    53248 524102 /root/.ipython/profile_default/history.sqlite
python3 923 root  109u     IPv4              17712      0t0    TCP static.236.194.216.95.clients.your-server.de:41540->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root  110u     IPv4              17722      0t0    TCP static.236.194.216.95.clients.your-server.de:41546->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root  111u     IPv4              18926      0t0    TCP static.236.194.216.95.clients.your-server.de:41544->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root  112u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root  113u     IPv4              17723      0t0    TCP static.236.194.216.95.clients.your-server.de:41548->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root  114u     unix 0xffff8a47f5bd7400      0t0  18930 type=STREAM
python3 923 root  115u     unix 0xffff8a47f5bd4800      0t0  18931 type=STREAM
python3 923 root  116u     IPv4              18932      0t0    TCP static.236.194.216.95.clients.your-server.de:41550->ec2-18-176-110-170.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 923 root  117u  a_inode               0,13        0  10602 [eventpoll]
python3 923 root  118u     unix 0xffff8a47f5bd7000      0t0  18936 type=STREAM
python3 923 root  119u     unix 0xffff8a47f5bd6000      0t0  18937 type=STREAM


root@ubwa-demo:~# lsof -p 912 | grep IP
python3 912 root    5u     IPv4              16993      0t0    TCP static.236.194.216.95.clients.your-server.de:38968->server-13-33-246-143.hel50.r.cloudfront.net:https (ESTABLISHED)
python3 912 root   14u     IPv4              18551      0t0    TCP static.236.194.216.95.clients.your-server.de:40772->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   21u     IPv4              18464      0t0    TCP static.236.194.216.95.clients.your-server.de:40726->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   22u     IPv4              17005      0t0    TCP static.236.194.216.95.clients.your-server.de:40724->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   32u     IPv4              17010      0t0    TCP static.236.194.216.95.clients.your-server.de:40728->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   45u     IPv4              18477      0t0    TCP static.236.194.216.95.clients.your-server.de:40730->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   49u     IPv4              17017      0t0    TCP static.236.194.216.95.clients.your-server.de:40732->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   56u     IPv4              17023      0t0    TCP static.236.194.216.95.clients.your-server.de:40734->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   57u     IPv4              18486      0t0    TCP static.236.194.216.95.clients.your-server.de:40736->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   65u     IPv4              18503      0t0    TCP static.236.194.216.95.clients.your-server.de:40738->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   68u     IPv4              18506      0t0    TCP static.236.194.216.95.clients.your-server.de:40740->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   69u     IPv4              17026      0t0    TCP static.236.194.216.95.clients.your-server.de:40744->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   70u     IPv4              17027      0t0    TCP static.236.194.216.95.clients.your-server.de:40742->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   71u     IPv4              18517      0t0    TCP static.236.194.216.95.clients.your-server.de:40746->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   72u     IPv4              18518      0t0    TCP static.236.194.216.95.clients.your-server.de:40748->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   73u     IPv4              17028      0t0    TCP static.236.194.216.95.clients.your-server.de:40750->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   74u     IPv4              18519      0t0    TCP static.236.194.216.95.clients.your-server.de:40752->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root   87u     IPv4              18526      0t0    TCP static.236.194.216.95.clients.your-server.de:40754->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  100u     IPv4              18534      0t0    TCP static.236.194.216.95.clients.your-server.de:40756->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  101u     IPv4              18538      0t0    TCP static.236.194.216.95.clients.your-server.de:40758->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  105u     IPv4              17041      0t0    TCP static.236.194.216.95.clients.your-server.de:40760->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  107u     IPv4              18544      0t0    TCP static.236.194.216.95.clients.your-server.de:40762->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  108u     IPv4              17045      0t0    TCP static.236.194.216.95.clients.your-server.de:40764->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  110u     IPv4              17046      0t0    TCP static.236.194.216.95.clients.your-server.de:40766->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  111u     IPv4              18550      0t0    TCP static.236.194.216.95.clients.your-server.de:40770->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  112u     IPv4              18548      0t0    TCP static.236.194.216.95.clients.your-server.de:40768->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
python3 912 root  113u     IPv4              18552      0t0    TCP static.236.194.216.95.clients.your-server.de:40774->ec2-13-115-126-224.ap-northeast-1.compute.amazonaws.com:9443 (ESTABLISHED)
root@ubwa-demo:~# lsof -p 912 | wc -l
167

lsof | wc -l
14118

I will compare it after running a couple of days

@oliver-zehentleitner
Copy link
Member

after 12 days the amount of open files didnt change. I guess this just happening during a lot and fast reconnects.

@oliver-zehentleitner
Copy link
Member

Jap. And I got a new error:

CRITICAL:root:BinanceWebSocketApiSocket->start_socket(5eea96b3-5b55-4fc3-86f5-d15822985931, ['depth@100ms'], ['manabtc', ..., 'iotabtc']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'

I am not sure about this!
In #18 @DaWe35 that the gets the error asyncio.base_futures.InvalidStateError: invalid state and I added an exception to https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/unicorn_binance_websocket_api_connection.py#L272 which solved the problem, but then sometimes we got an AttributeError during receiving so I added this https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/unicorn_binance_websocket_api_socket.py#L118 what is showing the log you have posted. Its handled and should be restarted after a couple of seconds.

@DaWe35
Copy link
Contributor

DaWe35 commented Aug 28, 2020

It was a long time ago, but as I remember increasing CPU and/or using the buffer helped me out. I can recommend the buffer if you not already use.

@oliver-zehentleitner
Copy link
Member

InvalidStateError: #110

Pyflame: #65

BestCryptoKnight pushed a commit to BestCryptoKnight/unicorn-binance-websocket-api that referenced this issue Jul 10, 2022
https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api…
3743e50 
…/issues/45
Seven-112 pushed a commit to Seven-112/unicorn-binance-websocket-api that referenced this issue May 16, 2023
https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api…
a8e414b 
…/issues/45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@oliver-zehentleitner oliver-zehentleitner

Labels
bug Something isn't working
Projects
Todo
  
Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DaWe35 @fsteckel @oliver-zehentleitner