Steps to reproduce
# volume.dstack.yml
type: volume
name: volume-aws
backend: aws
region: eu-north-1
size: 10GB
# run.dstack.yml
type: dev-environment
ide: vscode
volumes:
- volume-aws:/volume
dstack apply -f volume.dstack.yml
dstack apply -f run.dstack.yml, wait for provisioning status, detach
dstack apply -f run.dstack.yml
Actual behaviour
Once the first run/job is provisioned, the second one gets stuck in an exception loop. Stopping the run won't work, it gets stuck in terminating status.
Expected behaviour
No response
dstack version
2a936a2
Server logs
INFO dstack._internal.server.services.events:205 Emitting event: Instance status changed IDLE -> BUSY. Event targets:
instance(18eaed)cloud-2. Actor: system
INFO dstack._internal.server.services.events:205 Emitting event: Job assigned to instance. Instance blocks: 1/1 busy. Event
targets: job(0f92ee)devenv-2-0-0, instance(18eaed)cloud-2. Actor: system
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item 0f92eebd-d9e4-4ba6-82bf-df1b5ab37527 in
0.275
DEBUG dstack._internal.server.app:296 Processed request POST http://localhost:3000/api/project/main/fleets/list in 0.009673s.
Status: 200
DEBUG dstack._internal.server.app:296 Processed request POST http://localhost:3000/api/project/main/fleets/list in 0.009523s.
Status: 200
DEBUG dstack._internal.server.app:296 Processed request POST http://localhost:3000/api/project/main/runs/get in 0.013155s.
Status: 200
DEBUG dstack._internal.server.app:296 Processed request POST http://localhost:3000/api/project/main/fleets/list in 0.009959s.
Status: 200
DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item 0f92eebd-d9e4-4ba6-82bf-df1b5ab37527
DEBUG dstack._internal.server.background.pipeline_tasks.jobs_submitted:316 job(0f92ee)devenv-2-0-0: provisioning has started
ERROR dstack._internal.server.background.pipeline_tasks.base:361 Unexpected exception when processing item
Traceback (most recent call last):
File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/base.py", line 359, in start
await self.process(item)
File "/home/def/dev/dstack/src/dstack/_internal/server/utils/sentry_utils.py", line 28, in wrapper
return await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/jobs_submitted.py", line 317, in
process
provisioning = await _process_provisioning(item=item, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/jobs_submitted.py", line 1003, in
_process_provisioning
return await _process_existing_instance_provisioning(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
)
^
File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/jobs_submitted.py", line 1106, in
_process_existing_instance_provisioning
volume_attachment_result = await _process_volume_attachments(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<5 lines>...
)
^
File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/jobs_submitted.py", line 1501, in
_process_volume_attachments
volume = volume_model_to_volume(volume_model)
File "/home/def/dev/dstack/src/dstack/_internal/server/services/volumes.py", line 391, in volume_model_to_volume
instance=instance_model_to_volume_instance(instance),
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/def/dev/dstack/src/dstack/_internal/server/services/volumes.py", line 451, in
instance_model_to_volume_instance
fleet_name=instance_model.fleet.name if instance_model.fleet else None,
^^^^^^^^^^^^^^^^^^^^
File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/orm/attributes.py", line 569, in __get__
return self.impl.get(state, dict_) # type: ignore
~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/orm/attributes.py", line 1096, in get
value = self._fire_loader_callables(state, key, passive)
File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/orm/attributes.py", line 1131, in
_fire_loader_callables
return self.callable_(state, passive)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/def/dev/dstack/.venv/lib/python3.13/site-packages/sqlalchemy/orm/strategies.py", line 922, in
_load_for_state
raise orm_exc.DetachedInstanceError(
...<3 lines>...
)
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <InstanceModel at 0x747be55ab4d0> is not bound to a Session;
lazy load operation of attribute 'fleet' cannot proceed (Background on this error at: https://sqlalche.me/e/20/bhk3)
Steps to reproduce
dstack apply -f volume.dstack.ymldstack apply -f run.dstack.yml, wait for provisioning status, detachdstack apply -f run.dstack.ymlActual behaviour
Once the first run/job is provisioned, the second one gets stuck in an exception loop. Stopping the run won't work, it gets stuck in terminating status.
Expected behaviour
No response
dstack version
2a936a2
Server logs