util/retry: create dynamic retry mechanism

[kev-cao]

This commit creates a dynamic retry mechanism wrapper around retry.Retry. This mechanism allows for multiple retry policies that the caller can switch between after each attempt.

Epic: None

Release note: None

[cockroach-teamcity]

This change is 

[kev-cao]

To see this in action, check out:

cockroach/pkg/backup/restore_job.go

Lines 233 to 327 in f46a232

	const singletonState = "_"
	r := retry.NewDynamicRetry[roachpb.RowCount](
	"restore", &execCtx.ExecCfg().Settings.SV,
	).WithStates(map[string]retry.Options{
	singletonState: retryOpts,
	}).WithStrategy(
	func(
	d *retry.DynamicRetryState, res roachpb.RowCount, err error,
	) retry.RetryTransition {
	if errors.HasType(err, &kvpb.InsufficientSpaceError{}) {
	return retry.FastFail(jobs.MarkPauseRequestError(errors.UnwrapAll(err)))
	}
	if shouldFastFailRestore(err) {
	return retry.FastFail(jobs.MarkAsPermanentJobError(err))
	}
	// If we are draining, it is unlikely we can start a
	// new DistSQL flow. Exit with a retryable error so
	// that another node can pick up the job.
	if execCtx.ExecCfg().JobRegistry.IsDraining() {
	return retry.FastFail(jobs.MarkAsRetryJobError(
	errors.Wrapf(err, "job encountered retryable error on draining node"),
	))
	}
	var currProgress jobspb.RestoreFrontierEntries = resumer.job.
	Progress().Details.(*jobspb.Progress_Restore).Restore.Checkpoint
	if !currProgress.Equal(prevPersistedSpans) {
	prevPersistedSpans = currProgress
	// If the previous persisted spans are different than the current, it
	// implies that further progress has been persisted.
	return retry.Reset()
	}
	if d.CurrentStateAttempt() >= maxRestoreRetryFastFail &&
	resumer.job.FractionCompleted() <= progThreshold {
	return retry.FastFail(jobs.MarkAsPermanentJobError(
	errors.Wrap(err, "restore job exhausted max retries without making progress"),
	))
	}
	return retry.Continue()
	},
	).OnContinue(
	func(d *retry.DynamicRetryState, res roachpb.RowCount, err error) error {
	if logThrottler.ShouldProcess(crtime.NowMono()) {
	// We throttle the logging of errors to the jobs messages table to avoid
	// flooding the table during the hot loop of a retry.
	if err := execCtx.ExecCfg().InternalDB.Txn(ctx, func(ctx context.Context, txn isql.Txn) error {
	return resumer.job.Messages().Record(
	ctx, txn, "error", fmt.Sprintf("restore encountered error: %v", err),
	)
	}); err != nil {
	log.Dev.Warningf(ctx, "failed to record job error message: %v", err)
	}
	}
	testingKnobs := execCtx.ExecCfg().BackupRestoreTestingKnobs
	if testingKnobs != nil && testingKnobs.RunAfterRetryIteration != nil {
	if err := testingKnobs.RunAfterRetryIteration(err); err != nil {
	return err
	}
	}
	return nil
	},
	)
	res, err := r.DoCtx(
	ctx,
	func(ctx context.Context) (roachpb.RowCount, error) {
	return restore(
	ctx,
	execCtx,
	backupManifests,
	backupLocalityInfo,
	endTime,
	dataToRestore,
	resumer,
	encryption,
	kmsEnv,
	)
	},
	"_",
	)
	if err != nil {
	if r.LastTransition() == retry.TransitionFastFail {
	// Failure was a fast-fail, return the error as is.
	return res, err
	}
	// Since the restore was able to make some progress before exhausting the
	// retry counter, we will pause the job and allow the user to determine
	// whether or not to resume the job or discard all progress and cancel.
	return res, jobs.MarkPauseRequestError(errors.Wrap(err, "exhausted retries"))
	}
	return res, nil

This doesn't use the multiple states functionality, but it does take advantage of the ability to reset the retry mechanism or fast fail.

The purpose of this mechanism it to facilitate the work for #159388, namely the approach outlined here.

[kev-cao]

This would be so much more succinct with Rust enums, wouldn't need the separate TransitionKind or all this boilerplate with interfaces 😢

[github-actions]

Potential Bug(s) Detected

The three-stage Claude Code analysis has identified potential bug(s) in this PR that may warrant investigation.

Next Steps:
Please review the detailed findings in the workflow run.

Note: When viewing the workflow output, scroll to the bottom to find the Final Analysis Summary.

After you review the findings, please tag the issue as follows:

• If the detected issue is real or was helpful in any way, please tag the issue with O-AI-Review-Real-Issue-Found
• If the detected issue was not helpful in any way, please tag the issue with O-AI-Review-Not-Helpful

[kev-cao]

Wrote up what this would look like on backup:
https://github.com/kev-cao/cockroach/blob/63b49d530ef232ad0449c6327f232ce3f1024ca1/pkg/backup/backup_job.go#L674-L773