[Bug]: Migration 82b32a135ea2 fails if there is more than one volume with the same project #1961
Description
Steps to reproduce
- Checkout to 003d30b
- Create more than one volume within one project, doesn't matter if deleted or not.
dstack server
Actual behaviour
sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.CardinalityViolationError'>: more than one row returned by a subquery used as an expression
[SQL: UPDATE volumes SET user_id = (SELECT owner_id FROM projects JOIN volumes ON projects.id = volumes.project_id) WHERE user_id IS NULL]
Expected behaviour
No response
dstack version
Server logs
ERROR: Traceback (most recent call last):
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 534, in _prepare_and_execute
self._rows = deque(await prepared_stmt.fetch(*parameters))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
data = await self.__bind_execute(args, 0, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 267, in __bind_execute
data, status, _ = await self.__do_execute(
^^^^^^^^^^^^^^^^^^^^^^^^
lambda protocol: protocol.bind_execute(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
self._state, args, '', limit, True, timeout))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 256, in __do_execute
return await executor(protocol)
^^^^^^^^^^^^^^^^^^^^^^^^
File "asyncpg/protocol/protocol.pyx", line 206, in bind_execute
asyncpg.exceptions.CardinalityViolationError: more than one row returned by a subquery used as an expression
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
~~~~~~~~~~~~~~~~~~~~~~~^
cursor, str_statement, effective_parameters, context
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
cursor.execute(statement, parameters)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 568, in execute
self._adapt_connection.await_(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self._prepare_and_execute(operation, parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
return current.parent.switch(awaitable) # type: ignore[no-any-return,attr-defined] # noqa: E501
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
value = await result
^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 546, in _prepare_and_execute
self._handle_exception(error)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 497, in _handle_exception
self._adapt_connection._handle_exception(error)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 780, in _handle_exception
raise translated_error from error
sqlalchemy.dialects.postgresql.asyncpg.AsyncAdapt_asyncpg_dbapi.Error: <class 'asyncpg.exceptions.CardinalityViolationError'>: more than one row returned by a subquery used as an expression
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/starlette/routing.py", line 693, in lifespan
async with self.lifespan_context(app) as maybe_state:
~~~~~~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
~~~~~~~~~~~~~~~~^^^^^
File "/home/def/.local/share/pyenv/versions/3.13.0/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/def/dev/dstack/src/dstack/_internal/server/app.py", line 85, in lifespan
await migrate()
File "/home/def/dev/dstack/src/dstack/_internal/server/db.py", line 62, in migrate
await connection.run_sync(_run_alembic_upgrade)
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/ext/asyncio/engine.py", line 886, in run_sync
return await greenlet_spawn(
^^^^^^^^^^^^^^^^^^^^^
fn, self._proxied, *arg, _require_await=False, **kw
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
result = context.switch(value)
File "/home/def/dev/dstack/src/dstack/_internal/server/db.py", line 86, in _run_alembic_upgrade
command.upgrade(alembic_cfg, "head")
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/command.py", line 406, in upgrade
script.run_env()
~~~~~~~~~~~~~~^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/script/base.py", line 586, in run_env
util.load_python_file(self.dir, "env.py")
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
module = load_module_py(module_id, path)
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
spec.loader.exec_module(module) # type: ignore
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "<frozen importlib._bootstrap_external>", line 1022, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/home/def/dev/dstack/src/dstack/_internal/server/migrations/env.py", line 94, in <module>
main()
~~~~^^
File "/home/def/dev/dstack/src/dstack/_internal/server/migrations/env.py", line 89, in main
run_migrations_online()
~~~~~~~~~~~~~~~~~~~~~^^
File "/home/def/dev/dstack/src/dstack/_internal/server/migrations/env.py", line 55, in run_migrations_online
run_migrations(connection)
~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/home/def/dev/dstack/src/dstack/_internal/server/migrations/env.py", line 72, in run_migrations
context.run_migrations()
~~~~~~~~~~~~~~~~~~~~~~^^
File "<string>", line 8, in run_migrations
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
self.get_context().run_migrations(**kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/runtime/migration.py", line 628, in run_migrations
step.migration_fn(**kw)
~~~~~~~~~~~~~~~~~^^^^^^
File "/home/def/dev/dstack/src/dstack/_internal/server/migrations/versions/82b32a135ea2_.py", line 37, in upgrade
op.execute(
~~~~~~~~~~^
"UPDATE volumes SET user_id = (SELECT owner_id FROM projects JOIN volumes ON projects.id = volumes.project_id) WHERE user_id IS NULL"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "<string>", line 8, in execute
File "<string>", line 3, in execute
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/operations/ops.py", line 2551, in execute
return operations.invoke(op)
~~~~~~~~~~~~~~~~~^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/operations/base.py", line 442, in invoke
return fn(self, operation)
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/operations/toimpl.py", line 236, in execute_sql
operations.migration_context.impl.execute(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
operation.sqltext, execution_options=operation.execution_options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/ddl/impl.py", line 217, in execute
self._exec(sql, execution_options)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/alembic/ddl/impl.py", line 210, in _exec
return conn.execute(construct, params)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
return meth(
self,
distilled_parameters,
execution_options or NO_OPTIONS,
)
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
return connection._execute_clauseelement(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self, distilled_params, execution_options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
ret = self._execute_context(
dialect,
...<8 lines>...
cache_hit=cache_hit,
)
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
return self._exec_single_context(
~~~~~~~~~~~~~~~~~~~~~~~~~^
dialect, context, statement, parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
self._handle_dbapi_exception(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
e, str_statement, effective_parameters, cursor, context
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
~~~~~~~~~~~~~~~~~~~~~~~^
cursor, str_statement, effective_parameters, context
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
cursor.execute(statement, parameters)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 568, in execute
self._adapt_connection.await_(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self._prepare_and_execute(operation, parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
return current.parent.switch(awaitable) # type: ignore[no-any-return,attr-defined] # noqa: E501
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
value = await result
^^^^^^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 546, in _prepare_and_execute
self._handle_exception(error)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 497, in _handle_exception
self._adapt_connection._handle_exception(error)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
File "/home/def/.local/share/virtualenvs/dstack/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 780, in _handle_exception
raise translated_error from error
sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.CardinalityViolationError'>: more than one row returned by a subquery used as an expression
[SQL: UPDATE volumes SET user_id = (SELECT owner_id FROM projects JOIN volumes ON projects.id = volumes.project_id) WHERE user_id IS NULL]
(Background on this error at: https://sqlalche.me/e/20/dbapi)
ERROR: Application startup failed. Exiting.Additional information
No response