['receives_statistic_last_second'] dict is changing size during iteration.

[fsteckel]

Hi, great library.
I report a bug. python version 3.7.5, pip 18.1, ubuntu 19.10 in a local docker container.

After some days my streams encounter exceptions in threads I suspect the unicorn binance manager spawned:
Exception in thread Thread-1816: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/dist-packages/unicorn_binance_websocket_api/unicorn_binance_websocket_api_manager.py", line 301, in _frequent_checks for timestamp_key in self.stream_list[stream_id]['receives_statistic_last_second']['entries']: RuntimeError: dictionary changed size during iteration
This happened after I inserted a regular check for binance_websocket_api_manager.get_stream_info(stream_id).
Any idea what might cause this? The ['receives_statistic_last_second'] dict seems to be read by a thread while another is reading it. Do I need to do something before triggering the get_stream_info method?

Cheers

Florian

[oliver-zehentleitner]

hi Florian!

Thanks for the report! No its not on you to solve this. I must make the section in get_stream_info() thread safe or a deepcopy ... depends on the exact reason whats happening.

I am preparing a new release with subscribe/unsubscribe methods for all endpoints (cex and dex) and i can fix this issue too.

I post you a message when its ready!

Best regards,
Oliver

[oliver-zehentleitner]

One question: Which version of the library are you using? 1.9.1 or the dev-stage?

[oliver-zehentleitner]

To fix: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/a8ab50b1d9aabe5b73a0059c14a740ff0f7e1cce/unicorn_binance_websocket_api/unicorn_binance_websocket_api_manager.py#L301

[fsteckel]

Hey, thanks for the swift reply.
I am using version 1.9.1. I do not quite understand the fix. This line (an the following ones) look exactly like this in the lib.

[oliver-zehentleitner]

Hello!
Its not fixed, i just set a reminder for me :)
When i release the 1.10.0 this fix will be included!
LG nach Berlin ;)

[oliver-zehentleitner]

Its fixed now:
https://pypi.org/project/unicorn-binance-websocket-api/
https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/releases/latest

Please reopen if you still experience this issue!

Best regards!

[fsteckel]

Great, thanks for the support!

Leiwand! Danke vielmals!

[fsteckel]

Sadly it also happens with the new deepcopy changes:
Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/dist-packages/unicorn_binance_websocket_api/unicorn_binance_websocket_api_manager.py", line 292, in _freque nt_checks temp_entries = copy.deepcopy(self.stream_list[stream_id]['receives_statistic_last_second']['entries']) File "/usr/lib/python3.7/copy.py", line 150, in deepcopy y = copier(x, memo) File "/usr/lib/python3.7/copy.py", line 239, in _deepcopy_dict for key, value in x.items(): RuntimeError: dictionary changed size during iteration
I seem to not be able to reopen. Request reopening.
Danke :)

[oliver-zehentleitner]

OK, I am going to make it thread safe asap Am 15. Februar 2020 12:56:56 MEZ schrieb Florian Steckel <notifications@github.com>:

Sadly it also happens with the new deepcopy changes: ` Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/dist-packages/unicorn_binance_websocket_api/unicorn_binance_websocket_api_manager.py", line 292, in _freque nt_checks temp_entries = copy.deepcopy(self.stream_list[stream_id]['receives_statistic_last_second']['entries']) File "/usr/lib/python3.7/copy.py", line 150, in deepcopy y = copier(x, memo) File "/usr/lib/python3.7/copy.py", line 239, in _deepcopy_dict for key, value in x.items(): RuntimeError: dictionary changed size during iteration ` -- You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub: #37 (comment)

Oliver Zehentleitner

[oliver-zehentleitner]

i added locking now in the dev stage!

can you test it with this installation:
https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api#from-the-latest-source-dev-stage-with-pip-from-github

[oliver-zehentleitner]

the "solution" triggers a new error ...

[oliver-zehentleitner]

So it works now: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/releases