async for message in self.websocket: blocks the event loop

[mmertc]

Hello,

I'm developing a trading application and the problem I'm currently experiencing is that after running without a problem for a while, sending and receiving lots of messages successfully, the event loop gets stuck on a line with "async for message in self.websocket". I only see this is the case when I KeyboardInterrupt and it shows the traceback otherwise it looks like whole operation stopped at a random point and nothing gets printed to the console anymore. I had this happen multiple times so far and in another instance the traceback showed further up to the line: "await asyncio.shield(self.connection_lost_waiter)" line 321 in connection.py of websockets/asyncio where it got similarly stuck indefinitely.

I would really appreciate suggestions and help, and I can post part of the code if requested, I did not paste it initially since it is quite long.

[aaugustin]

It's hard to tell what's happening with just the information that you provided.

Perhaps you're receiving messages faster than you can process them, leading to your receive buffers filling up. In that case, eventually, you will exceed the maximum latency defined by ping_timeout and, since the buffers are full, the closing handshake gets stuck.

The connection should get closed after close_timeout, though.

[mmertc]

Thanks for you quick reply. I will rerun with max_size = 10 * (2**20) and see if it fixes anything. If not I can maybe provide a much more detailed explanation and also paste code snippets.

[aaugustin]

max_size controls the size of individual messages. It's unlikely to help. Try max_queue and/or ping_timeout. That being said, if messages are sent faster than you can process them, eventually you lose...

[mmertc]

I have tinkered with max_queue and setting to about 10000 made some improvement while increasing max_size made it worse interestingly. Would I be correct to assume this confirms your hypothesis that the problem is queue overflowing? Also would enabling the debug logs allow me to see details such as queue being full etc., if not, is there a way for me to make them visible for easing the debugging?

[aaugustin]

Enabling asyncio's debug mode should tell you when websockets pauses the transport and stops reading:
https://github.com/python/cpython/blob/b9f0943c1e31acd0895a4b57b4af57a5a1e0a2b1/Lib/asyncio/selector_events.py#L851