Skip to content

[Bug]: Pipelines not working on Python 3.9 and 3.10 #3735

@r4victor

Description

@r4victor

Steps to reproduce

  1. Run dstack server on Python 3.9 and 3.10.
  2. Try submitting any resource and see it being stuck in submitted/pending.

Actual behaviour

The module-level instantiation _pipeline_manager = PipelineManager() creates asyncio primitives outside of a running event loop on Python 3.9 and 3.10. On Python 3.11+ these primitives no longer bind to a loop in __init__() — they lazily get the running loop when first used, so the module-level instantiation works fine. PipelineManager() should not be done at the module level but initialized lazily.

Update: The real root cause is that on Python before 3.11, timeout error is asyncio.Timeout but we were handling only TimeoutError:

await asyncio.wait_for(
self._fetch_event.wait(),
timeout=self._next_fetch_delay(empty_fetch_count),
)
except TimeoutError:
pass

Still a good idea to move pipeline manager init out of module init.

Expected behaviour

No response

dstack version

master

Server logs

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions