How to wait for some k8s cluster state properly?

[igkon]

Type of question

Best practices

Question

What did you do?

I write Operators in Go-style and I want to ask a question. How can we wait for something to be happened on the cluster in Operator code? For example, I delete pods from Operator and once they completely deleted, I want to perform other actions. I know that blocking of Reconcile is a bad practice. I have two ways to reach behavior I need.

1. As soon as it's needed to wait for something, we can start a goroutine with this waiting and related actions. An example in pseudo Go code:

func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (res ctrl.Result, err error) {
    ...
    if err := r.Delete(ctx, pod); err != nil {
        return ctrl.Result{}, err
    }
    go func() {
        r.WaitForPodToBeDeleted(ctx, pod)
        // Additional actions when pod deleted
        r.DeployBiggerPod(ctx)
        ...
    }()
}

There are a lot of cons: it's needed to control amount of goroutines; it's impossible/hard to requeue Reconcile in case of error, etc.

2. The second approach is to do something like this:

func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (res ctrl.Result, err error) {
    ...
    if !r.IfPodWasDeleted(ctx, pod) {
        if err := r.Delete(ctx, pod); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }
    // This code will be invoked only after pod deletion    
    r.DeployBiggerPod(ctx)
    ...
}

What did you expect to see?

Is one of my approaches valid? What are best practices for described situation?

Environment

Any

[asmacdo]

I vote we add this one to the FAQ. Thanks @hoklet for this well written issue that I hope will be part of our docs soon!

[jberkhahn]

Number 2 is the generally accepted way to do this sort of thing. I agree we should add it to the FAQ though.