[Bug]: Error when detaching volume from AWS instance when project is not the same as the server (usually main) #1968
Description
Steps to reproduce
-create a new project
-create a new user and assign to the project
-create aws volume
-start run with aws volume
-when the run stops, it will have an error
Actual behaviour
No response
Expected behaviour
No response
dstack version
0.18.23rc2
Server logs
`[17:14:57] INFO dstack._internal.server.services.jobs:248 Detaching volumes:
['hcarrasco-cache-volume', 'hcarrasco-dev-volume']
ERROR apscheduler.executors.default:36 Job
"process_terminating_jobs (trigger: interval[0:00:04], next
run at: 2024-11-06 17:15:01 UTC)" raised an exception
Traceback (most recent call last):
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/apschedule
r/executors/base_py3.py", line 30, in run_coroutine_job
retval = await job.func(*job.args, **job.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/dstack/_in
ternal/server/background/tasks/process_terminating_jobs.py",
line 38, in process_terminating_jobs
await _process_job(session=session, job_model=job_model)
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/dstack/_in
ternal/server/background/tasks/process_terminating_jobs.py",
line 44, in _process_job
await process_terminating_job(session, job_model)
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/dstack/_in
ternal/server/services/jobs/__init__.py", line 249, in
process_terminating_job
await detach_volumes_from_instance(
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/dstack/_in
ternal/server/services/jobs/__init__.py", line 354, in
detach_volumes_from_instance
volume = volume_model_to_volume(volume_model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/dstack/_in
ternal/server/services/volumes.py", line 277, in
volume_model_to_volume
user=volume_model.user.name,
^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/attributes.py", line 566, in __get__
return self.impl.get(state, dict_) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/attributes.py", line 1086, in get
value = self._fire_loader_callables(state, key, passive)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/attributes.py", line 1121, in _fire_loader_callables
return self.callable_(state, passive)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/strategies.py", line 967, in _load_for_state
return self._emit_lazyload(
^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/strategies.py", line 1068, in _emit_lazyload
return loading.load_on_pk_identity(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/loading.py", line 694, in load_on_pk_identity
session.execute(
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/session.py", line 2362, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/session.py", line 2247, in _execute_internal
result: Result[Any] =
compile_state_cls.orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/orm/context.py", line 293, in orm_execute_statement
result = conn.execute(
^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/engine/base.py", line 1418, in execute
return meth(
^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/sql/elements.py", line 515, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/engine/base.py", line 1640, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/engine/base.py", line 1846, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/engine/base.py", line 1986, in _exec_single_context
self._handle_dbapi_exception(
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/engine/base.py", line 2358, in _handle_dbapi_exception
raise exc_info[1].with_traceback(exc_info[2])
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/engine/base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/engine/default.py", line 941, in do_execute
cursor.execute(statement, parameters)
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/dialects/sqlite/aiosqlite.py", line 147, in execute
self._adapt_connection._handle_exception(error)
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/dialects/sqlite/aiosqlite.py", line 298, in
_handle_exception
raise error
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/dialects/sqlite/aiosqlite.py", line 124, in execute
_cursor = self.await_(self._connection.cursor())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/ubuntu/dstack/lib/python3.12/site-packages/sqlalchemy
/util/_concurrency_py3k.py", line 123, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been
called; can't call await_only() here. Was IO attempted in an
unexpected place? (Background on this error at:
https://sqlalche.me/e/20/xd2s)`Additional information
No response