Skip to content

[Bug]: Job provisioning fails if already attached volume is requested in the run configuration #3815

@un-def

Description

@un-def

Steps to reproduce

# volume.dstack.yml
type: volume
name: volume-aws
backend: aws
region: eu-north-1
size: 10GB
# run.dstack.yml
type: dev-environment
ide: vscode
volumes:
  - volume-aws:/volume
  1. dstack apply -f volume.dstack.yml
  2. dstack apply -f run.dstack.yml, wait for provisioning status, detach
  3. dstack apply -f run.dstack.yml

Actual behaviour

Once the first run/job is provisioned, the second one gets stuck in an exception loop. Stopping the run won't work, it gets stuck in terminating status.

Expected behaviour

No response

dstack version

2a936a2

Server logs

INFO     dstack._internal.server.services.events:205 Emitting event: Instance status changed IDLE -> BUSY. Event targets:
         instance(18eaed)cloud-2. Actor: system
INFO     dstack._internal.server.services.events:205 Emitting event: Job assigned to instance. Instance blocks: 1/1 busy. Event
         targets: job(0f92ee)devenv-2-0-0, instance(18eaed)cloud-2. Actor: system
DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item 0f92eebd-d9e4-4ba6-82bf-df1b5ab37527 in
         0.275
DEBUG    dstack._internal.server.app:296 Processed request POST http://localhost:3000/api/project/main/fleets/list in 0.009673s.
         Status: 200
DEBUG    dstack._internal.server.app:296 Processed request POST http://localhost:3000/api/project/main/fleets/list in 0.009523s.
         Status: 200
DEBUG    dstack._internal.server.app:296 Processed request POST http://localhost:3000/api/project/main/runs/get in 0.013155s.
         Status: 200
DEBUG    dstack._internal.server.app:296 Processed request POST http://localhost:3000/api/project/main/fleets/list in 0.009959s.
         Status: 200
DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item 0f92eebd-d9e4-4ba6-82bf-df1b5ab37527
DEBUG    dstack._internal.server.background.pipeline_tasks.jobs_submitted:316 job(0f92ee)devenv-2-0-0: provisioning has started
ERROR    dstack._internal.server.background.pipeline_tasks.base:361 Unexpected exception when processing item
         Traceback (most recent call last):
           File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/base.py", line 359, in start
             await self.process(item)
           File "/home/def/dev/dstack/src/dstack/_internal/server/utils/sentry_utils.py", line 28, in wrapper
             return await f(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
           File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/jobs_submitted.py", line 317, in
         process
             provisioning = await _process_provisioning(item=item, context=context)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/jobs_submitted.py", line 1003, in
         _process_provisioning
             return await _process_existing_instance_provisioning(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
             ...<3 lines>...
             )
             ^
           File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/jobs_submitted.py", line 1106, in
         _process_existing_instance_provisioning
             volume_attachment_result = await _process_volume_attachments(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
             ...<5 lines>...
             )
             ^
           File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/jobs_submitted.py", line 1501, in
         _process_volume_attachments
             volume = volume_model_to_volume(volume_model)
           File "/home/def/dev/dstack/src/dstack/_internal/server/services/volumes.py", line 391, in volume_model_to_volume
             instance=instance_model_to_volume_instance(instance),
                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
           File "/home/def/dev/dstack/src/dstack/_internal/server/services/volumes.py", line 451, in
         instance_model_to_volume_instance
             fleet_name=instance_model.fleet.name if instance_model.fleet else None,
                                                     ^^^^^^^^^^^^^^^^^^^^
           File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/orm/attributes.py", line 569, in __get__
             return self.impl.get(state, dict_)  # type: ignore
                    ~~~~~~~~~~~~~^^^^^^^^^^^^^^
           File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/orm/attributes.py", line 1096, in get
             value = self._fire_loader_callables(state, key, passive)
           File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/orm/attributes.py", line 1131, in
         _fire_loader_callables
             return self.callable_(state, passive)
                    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
           File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/orm/strategies.py", line 922, in
         _load_for_state
             raise orm_exc.DetachedInstanceError(
             ...<3 lines>...
             )
         sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <InstanceModel at 0x747be55ab4d0> is not bound to a Session;
         lazy load operation of attribute 'fleet' cannot proceed (Background on this error at: https://sqlalche.me/e/20/bhk3)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingvolumes

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions