workspaces which cannot be scheduled to a cluster get stuck in Pending #11397
Description
Bug description
Workspaces which cannot be scheduled to a cluster get stuck in Pending.
Steps to reproduce
In a preview environment, kubectl cordon the node. Try starting a workspace. It'll stay stuck in Pending for-ev-er!
Workspace affected
n/a
Expected behavior
The workspace should fail, and land in a permanent Failed phase
Example repository
n/a
Anything else?
This becomes a problem when you're trying to delete a cluster, because ws-manager-bridge reports that there are still active workspaces in the database for the cluster, even though there are no workspace pods in the governed workspace cluster.
This prevented the delete of us53 (which was empty, and had no workspace clusters), until we ran the following queries:
-- find pending workspaces that aren't really pending
SELECT dbwi.id
FROM gitpod.d_b_workspace dbw
inner join gitpod.d_b_workspace_instance dbwi
on dbw.id = dbwi.workspaceId
WHERE
dbwi.phasePersisted not in ('stopped','failed')
# change to match a region you're interested in
and dbwi.region = 'us53'
ORDER by dbwi.creationTime asc
;
-- force stop the pending workspaces
UPDATE gitpod.d_b_workspace_instance AS wsi
SET
wsi.stoppedTime = IF(wsi.startedTime = '', wsi.creationTime, wsi.startedTime),
wsi.stoppingTime = IF(wsi.startedTime = '', wsi.creationTime, wsi.startedTime),
wsi.phasePersisted = 'stopped',
wsi.status = '{"phase": "stopped", "conditions": {}}'
WHERE wsi.id IN ('fb52614b-927b-4412-b583-dd3cc7bc3087',
'04eedb5b-1f35-4e4b-a8b3-56ea13c0f2dd',
'9b3d3928-59dd-46f0-9876-cc2bab905a2e',
'261bbde5-100f-4a19-8085-125ef82a0f0a',
'7878cc2a-544e-4f24-82ee-62a24c4f9c34',
'5dda17bf-1fe9-4cc6-b5ae-82aa6603049c',
'ad40bce6-a74c-47a9-b9a4-d9490afbfd41',
'028d826e-49b2-45ce-b829-8b4f693e93d3',
'99b74f2b-0a49-45c6-9c59-c4339a6dcb48',
'3695365a-0b4c-4df2-96f3-700598fe3baf')
;- Delete this page once resolved
Metadata
Metadata
Assignees
Labels
Type
Projects
Status