Skip to content

[Bug]: Unreleased global_exports lock on Postgres #3881

@jvstme

Description

@jvstme

Steps to reproduce

Do any of these actions several times:

  • Create a project.
  • Create a global export.
  • Promote an existing export to global.

Actual behaviour

The first action succeeds. A few more may succeed. But at some point, all successive attempts start to hang indefinitely.

$ dstack export --project test-1 create test-1 --global
 NAME    FLEETS  GATEWAYS  IMPORTERS 
 test-1  -       -         *         

$ dstack export --project test-1 create test-2 --global
 NAME    FLEETS  GATEWAYS  IMPORTERS 
 test-2  -       -         *         

$ dstack export --project test-1 create test-3 --global
 NAME    FLEETS  GATEWAYS  IMPORTERS 
 test-3  -       -         *         

$ dstack export --project test-1 create test-4 --global
⠧ Creating export...

Expected behaviour

No hanging.

dstack version

master

Server logs

Additional information

Postgres logs show that at some point the global_exports advisory lock fails to be released, because it is attempted to be released by a connection different than the one that acquired it.

2026-05-14 12:08:35.800 UTC [404] LOG:  execute __asyncpg_stmt_ab__: SELECT pg_advisory_lock($1::BIGINT) AS pg_advisory_lock_1
2026-05-14 12:08:35.800 UTC [404] DETAIL:  Parameters: $1 = '6333502497592769007'
2026-05-14 12:08:35.840 UTC [404] LOG:  execute __asyncpg_stmt_2a6__: SELECT projects.id, projects.name, projects.owner_id, users_1.id AS id_1, users_1.name AS name_1, users_1.created_at, users_1.token, users_1.token_hash, users_1.global_role, users_1.active, users_1.deleted, users_1.original_name, users_1.ssh_private_key, users_1.ssh_public_key, users_1.email, users_1.projects_quota 
	FROM projects LEFT OUTER JOIN users AS users_1 ON users_1.id = projects.owner_id 
	WHERE projects.deleted = false
2026-05-14 12:08:35.897 UTC [404] LOG:  execute __asyncpg_stmt_2a7__: UPDATE exports SET is_global=$1::BOOLEAN WHERE exports.id = $2::UUID
2026-05-14 12:08:35.897 UTC [404] DETAIL:  Parameters: $1 = 't', $2 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7'
2026-05-14 12:08:35.898 UTC [400] LOG:  statement: COMMIT;
2026-05-14 12:08:35.917 UTC [404] LOG:  execute __asyncpg_stmt_2a8__: INSERT INTO exported_gateways (id, export_id, gateway_id) VALUES ($1::UUID, $2::UUID, $3::UUID)
2026-05-14 12:08:35.917 UTC [404] DETAIL:  Parameters: $1 = 'e429087b-c12c-4b07-93ff-d9f6d1d8dfe3', $2 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7', $3 = '2dc8a0c4-55df-4063-805c-db58ac557fec'
2026-05-14 12:08:35.951 UTC [404] LOG:  execute __asyncpg_stmt_2a9__: INSERT INTO imports (id, project_id, export_id, created_at) VALUES ($1::UUID, $2::UUID, $3::UUID, $4::TIMESTAMP WITHOUT TIME ZONE), ($5::UUID, $6::UUID, $7::UUID, $8::TIMESTAMP WITHOUT TIME ZONE), ($9::UUID, $10::UUID, $11::UUID, $12::TIMESTAMP WITHOUT TIME ZONE)
2026-05-14 12:08:35.951 UTC [404] DETAIL:  Parameters: $1 = '3d5c5944-150d-4307-8a4d-96125a2a1911', $2 = '9be9ffb9-2e03-4e09-a363-5aee2cb0d890', $3 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7', $4 = '2026-05-14 12:08:35.943889', $5 = '6d8558be-6579-4609-a429-b432a13b231d', $6 = '4a556b2a-0d47-4cfb-ba7e-d595a14351c4', $7 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7', $8 = '2026-05-14 12:08:35.943904', $9 = 'a11df706-92bf-46d6-be55-d4d132f523d3', $10 = '6adacd1a-fb78-4685-b460-1f8ea9d60adc', $11 = 'f3c448fe-19c2-4d8e-a945-276001dac9a7', $12 = '2026-05-14 12:08:35.943915'
2026-05-14 12:08:35.961 UTC [404] LOG:  statement: COMMIT;
2026-05-14 12:08:35.976 UTC [395] LOG:  statement: BEGIN;
2026-05-14 12:08:35.983 UTC [395] LOG:  execute __asyncpg_stmt_113__: SELECT pg_advisory_unlock($1::BIGINT) AS pg_advisory_unlock_1
2026-05-14 12:08:35.983 UTC [395] DETAIL:  Parameters: $1 = '6333502497592769007'
2026-05-14 12:08:35.983 UTC [395] WARNING:  you don't own a lock of type ExclusiveLock

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions