Skip to content

Errored workspaces aren't removed after PVC cleanup failure #873

@AObuchow

Description

@AObuchow

Description

This issue was initially found by @amisevsk here.

If the common PVC cleanup job fails and all workspaces are deleted, the workspace which had the failed PVC cleanup job will be stuck in an errored state and won't be removed.

How To Reproduce

First, ensure the common PVC cleanup job will fail by modifying the PVC cleanup job spec in cleanup.go:

	Args: []string{
		"-c",
- 		fmt.Sprintf(cleanupCommandFmt, path.Join(pvcClaimMountPath, workspaceId)),
+		 "exit 1",
	},

Then do the following:

  1. oc apply -f samples/theia-next.yaml
  2. yq '.metadata.name="theia-next-2"' samples/theia-next.yaml | kubectl apply -f -
  3. Wait for workspaces to start/get finalizers at least
  4. oc delete dw theia-next
  5. Wait for deletion to hit error
  6. oc delete dw --all
  7. The workspace which had the PVC cleanup job that failed should be stuck in the error state, visible when doing kubectl get devworkspace -n $NAMESPACE

(Thank you to @amisevsk for the instructions on how to reproduce)

Expected behavior

The workspaces which are in the error state (from the PVC cleanup job failing) should be removed automatically.

Additional context

This bug arose from #846 and will only be apparent when that PR is merged.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions