Handling Intermittent ValueError in _handle_eio_message with Multithreading #1269
-
Hello python-socketio community, I've integrated python-socketio into a PyTorch DataLoader, which functions as a socketio client. It's designed to request and receive data in a multi-threaded setup using a ThreadPoolExecutor. However, I'm intermittently encountering a ValueError that I'm struggling to debug or handle gracefully. Here's the pattern I'm using:
The server responds to "request" with dict data packaged in a JSON format (note that one of the values of the dict is bytes) Occasionally, the client throws the following exception:
This error is problematic because:
I suspect the issue might be related to one of the following:
I am looking for advice on two fronts:
Additional context: The error seems more frequent when I enable multi-processing, likely due to the increased packet exchange volume. It occurs with and without multi-processing enabled, though. Any guidance or suggestions would be greatly appreciated. Théo |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
This happens because you are receiving invalid data. The Socket.IO client is thread-safe, but you can't use the same client object concurrently on different threads. For example, if you emit on the same client instance on different threads at about the same time, the packets corresponding to these two emits may be sent in an incorrect order and mixed up. If you are going to use the client from multiple threads, what you should do is protect the |
Beta Was this translation helpful? Give feedback.
This happens because you are receiving invalid data. The Socket.IO client is thread-safe, but you can't use the same client object concurrently on different threads. For example, if you emit on the same client instance on different threads at about the same time, the packets corresponding to these two emits may be sent in an incorrect order and mixed up.
If you are going to use the client from multiple threads, what you should do is protect the
emits()
with a lock.