Retry doFinalize if gRPC status code is Unavailable

[jenting]

Description

Retry doFinalize if gRPC status code is Unavailable

Related Issue(s)

Fixes #11861

How to test

None

Release Notes

None

Documentation

None

Werft options:

• /werft with-preview

[kylos101]

@jenting can you share in #11861 why this PR is related to but does not fix the issue, and maybe next steps for #11861, after this PR is merged?

[jenting]

@jenting can you share in #11861 why this PR is related to but does not fix the issue, and maybe next steps for #11861, after this PR is merged?

Well, I was thinking that should we handle a case that the backup thru. ws-daemon keeps failure

gitpod/components/ws-manager/pkg/manager/monitor.go

Lines 1326 to 1335 in bd16387

	if backupError != nil {
	tracing.LogError(span, backupError)
	log.WithError(backupError).Warn("internal error while disposing workspace content")
	m.manager.metrics.totalBackupFailureCounterVec.WithLabelValues(wsType, strconv.FormatBool(pvcFeatureEnabled), wso.Pod.Labels[workspaceClassLabel]).Inc()
	if dataloss {
	disposalStatus.BackupFailure = backupError.Error()
	}
	}

, should we remove the annotation gitpod.io/startedDisposal to make the ws-manager be able to call finalizerWorkspaceContent again?

gitpod/components/ws-manager/pkg/manager/monitor.go

Lines 495 to 497 in bd16387

	_, startedDisposal := wso.Pod.Annotations[startedDisposalAnnotation]
	span.LogKV("startedDisposal", startedDisposal)
	if terminated && !startedDisposal {

But it has a drawback that it keeps calling ws-daemon APIs even if the ws-daemon is gone.

[sagor999]

, should we remove the annotation gitpod.io/startedDisposal to make the ws-manager be able to call finalizerWorkspaceContent again?

Keep in mind, that ws-manager will act ONLY if there was some sort of an update to the pod. If there was none, then actOnPodEvent will not be called, and so ws-manager will not do anything.

[jenting]

/werft run with-preview

👍 started the job as gitpod-build-jenting-11861.4
(with .werft/ from main)

[jenting]

/werft run with-preview with-clean-slate-deployment

👍 started the job as gitpod-build-jenting-11861.5
(with .werft/ from main)