-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace scheduler.py main loop by asyncio loop #2866
Comments
Agreed on principle. Just another emphasis on the logic that writes to the primary SQLite file. It is important that we design the asynchronous to block the suite work flow to ensure the integrity of the persistent data for unscheduled stop+restart. Obviously, we can still design this logic to be asynchronous relative to the web server, but have enough logic to stop the suite workflow from moving on until we have a successful write to the primary SQLite file. |
I think this might be superseded by a new issue to do the same but with focus on pyzmq/ZeroMQ, and priority queues? If that's the case, we can close this one 👍 |
Very different scope here, I suppose. The main loop is not just about communication, so we may want to repurpose this issue to have an emphasis on the parts that are not to do with 0mq. Unless the new issue is going to fully supersede those as well? |
Superseded by #3495 |
When working on the Tornado PoC (#2825) one of the tasks involved making the
scheduler.py
main loop to run alongside Tornado's main loop (i.e. an asyncio loop since Python 3.x).It worked after replacing
scheduler.py
main loop by Tornado's loop. However, there are several blocking calls in Cylc's Suite Server Program main loop; calls to read and write files to disk, calls to sqlite, even network calls. These could all be parallelized through coroutines with asyncio.I spent some time looking at the code, and I think it could work. Though I would need more time with the code in order to do some further tests and come up with some benchmarks and pros/cons.
The text was updated successfully, but these errors were encountered: