-
Notifications
You must be signed in to change notification settings - Fork 221
[Bug]: Pipelines not working on Python 3.9 and 3.10 #3735
Description
Steps to reproduce
- Run dstack server on Python 3.9 and 3.10.
- Try submitting any resource and see it being stuck in submitted/pending.
Actual behaviour
The module-level instantiation _pipeline_manager = PipelineManager() creates asyncio primitives outside of a running event loop on Python 3.9 and 3.10. On Python 3.11+ these primitives no longer bind to a loop in __init__() — they lazily get the running loop when first used, so the module-level instantiation works fine. PipelineManager() should not be done at the module level but initialized lazily.
Update: The real root cause is that on Python before 3.11, timeout error is asyncio.Timeout but we were handling only TimeoutError:
dstack/src/dstack/_internal/server/background/pipeline_tasks/base.py
Lines 298 to 303 in c0c011c
| await asyncio.wait_for( | |
| self._fetch_event.wait(), | |
| timeout=self._next_fetch_delay(empty_fetch_count), | |
| ) | |
| except TimeoutError: | |
| pass |
Still a good idea to move pipeline manager init out of module init.
Expected behaviour
No response
dstack version
master
Server logs
Additional information
No response