Retry the finalizer loop when the ws-daemon pod not found

[jenting]

Description

Retry the function doFinalize when the ws-daemon pod is not found so that we can reach the condition

gitpod/components/ws-manager/pkg/manager/monitor.go

Lines 1272 to 1278 in 444ccf0

	if (err != nil && strings.Contains(err.Error(), context.DeadlineExceeded.Error())) ||
	st.Code() == codes.Unavailable ||
	st.Code() == codes.Canceled {
	// service is currently unavailable or we did not finish in time - let's wait some time and try again
	time.Sleep(wsdaemonRetryInterval)
	continue
	}

Related Issue(s)

Fixes #10682

How to test

1. Create a workspace.
2. Create a large file.

    dd if=/dev/urandom of=4G bs=16M count=256

3. Stop the workspace Gitpod: stop workspace and immediately go to step 5.
4. During the workspace stopping, keep deleting the ws-daemon pod for at least 1 minute. Note, we probably would hit the issue Restart the ws-daemon cause the workspace pod sometimes Failed #11606, but we'd like to tackle it in separate PR.

   while true; do kubectl delete pod -l component=ws-daemon; done

5. Cancel step 5 (ctrl+c).
6. Check the workspace changes to the stopped state and the backup content persistent.

Release Notes

None

Documentation

None

Werft options:

• /werft with-preview