[perf] Use zmq.asyncio.Context to accelerate notify_data_update process#38
[perf] Use zmq.asyncio.Context to accelerate notify_data_update process#380oshowero0 merged 5 commits intoAscend:mainfrom
zmq.asyncio.Context to accelerate notify_data_update process#38Conversation
Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
CLA Signature Pass0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
There was a problem hiding this comment.
Pull request overview
This PR refactors the ZMQ socket management in the storage manager to improve performance of the notify_data_update process. It switches from a persistent synchronous socket approach to a dynamic asynchronous socket pattern using zmq.asyncio.Context.
Changes:
- Replaced persistent
data_status_update_socketwith dynamic per-request socket creation innotify_data_update - Changed from synchronous zmq.Poller to async
asyncio.wait_forfor receiving ACK responses - Separated handshake socket (sync) from data update operations (async) for better resource management
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
CLA Signature Pass0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
CLA Signature Pass0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
CLA Signature Pass0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Description
Problem: The
notify_data_updatemethod was declared as async but internally used synchronouszmq.Contextand blockingzmq.Poller.poll() / socket.send_multipart() / socket.recv_multipart()calls, which blocked the asyncio event loop and degraded concurrency performance.Solution:
Changes
transfer_queue/storage/managers/base.py:
data_status_update_socketwith per-call dynamic zmq.asyncio sockets innotify_data_updatezmq.Poller-based polling withasyncio.wait_fortimeout mechanism