Graceful shutdown doesn't wait for reconcile workers to complete

[RobEarl]

Following the release of #967, I was hoping it would allow me to set a timeout big enough for any ongoing reconcile to complete. On testing, however, I discovered the reconcile still exits immediately.

I have tried injecting the stop channel into my reconciler then using a sync.WaitGroup to block until any ongoing reconcile completes:

var waitGroup sync.WaitGroup

func (r *MyReconciler) InjectStopChannel(stopChannel <-chan struct{}) error {
	logger := r.log.WithValues("Test", "Request.Name")

	go func() {
		<-stopChannel
		logger.Info("Shutdown signaled. Block until reconcile completes")
		// Stop signalled. Stall until any ongoing reconcile ends.
		waitGroup.Wait()
	}()

	return nil
}

func (r *MyReconciler) Reconcile(context context.Context, request reconcile.Request) (reconcile.Result, error) {
	waitGroup.Add(1)
	defer waitGroup.Done()

	reqLogger := r.log.WithValues("Request.Namespace", request.Namespace, "Request.Name", request.Name)
	reqLogger.Info("Reconciling", "type", r.instanceFactory.Name())
    
	for i := 1; i <= 30; i++ {
		reqLogger.Info("Waiting")
		time.Sleep(time.Second)
	}

	return reconcile.Result{}, nil
}

If I run this and hit CTRL+C in the middle of a reconcile, I do see the waitGroup.Wait() function log output but the process still exits immediately:

2021-03-12T10:39:57.628Z	INFO	controller-runtime.manager.controller.my	Starting Controller	{"reconciler group": "mygroup", "reconciler kind": "MyKind"}
2021-03-12T10:39:57.628Z	INFO	controller-runtime.manager.controller.my	Starting workers	{"reconciler group": "mygroup", "reconciler kind": "MyKind", "worker count": 1}
2021-03-12T10:39:57.628Z	INFO	controller_my	Reconciling	{"Request.Namespace": "test", "Request.Name": "test", "type": "MyKind"}
2021-03-12T10:39:57.628Z	INFO	controller_my	Waiting	{"Request.Namespace": "test", "Request.Name": "test"}
2021-03-12T10:39:58.629Z	INFO	controller_my	Waiting	{"Request.Namespace": "test", "Request.Name": "test"}
^C2021-03-12T10:39:59.291Z	INFO	controller-runtime.manager.controller.my	Stopping workers	{"reconciler group": "mygroup", "reconciler kind": "MyKind"}
2021-03-12T10:39:59.292Z	INFO	controller_my	Shutdown signaled. Block until reconcile completes	{"Test": "Request.Name"}