Operator assumes all pods are ready when there are no pods at all

[colega]

We're currently checking hasStatefulSetNotReadyPods before considering the zone reconciled:

rollout-operator/pkg/controller/controller.go

Lines 382 to 410 in 0b99175

	func (c *RolloutController) hasStatefulSetNotReadyPods(sts *v1.StatefulSet) (bool, error) {
	// We can quickly check the number of ready replicas reported by the StatefulSet.
	// If they don't match the total number of replicas, then we're sure there are some
	// not ready pods.
	if sts.Status.Replicas != sts.Status.ReadyReplicas {
	return true, nil
	}
	// The number of ready replicas reported by the StatefulSet matches the total number of
	// replicas. However, there's still no guarantee that all pods are running. For example,
	// a terminating pod (which we don't consider "ready") may have not yet failed the
	// readiness probe for the consecutive number of times required to switch its status
	// to not-ready. For this reason, we list all StatefulSet pods and check them one-by-one.
	notReadyPods, err := c.listNotReadyPodsByStatefulSet(sts)
	if err != nil {
	return false, err
	}
	if len(notReadyPods) == 0 {
	return false, nil
	}
	// Log which pods have been detected as not-ready. This may be useful for debugging.
	level.Info(c.logger).Log(
	"msg", "StatefulSet status is reporting all pods ready, but the rollout operator has found some not-Ready pods",
	"statefulset", sts.Name,
	"not_ready_pods", strings.Join(util.PodNames(notReadyPods), " "))
	return true, nil
	}

However, we've seen a case where all pods of a statefulset were deleted, but no new pods were created yet. This caused the listNotReadyPodsByStatefulSet method return no pods at all, as there were no pods in any state.

This caused an outage, as next zone was terminated immediately, before no pods were available in the first one.

I think that the code should check that there are at least as many pods as replicas desired by the statefulset, and then it should check whether some of them are unready.