[Question] What production cases motivated the conservative rolling-update strategy and the Pending pod special handling?

[LuckyGrx]

What is the user interaction of your feature

This is a question/discussion rather than a feature request.

As a user running KubeBlocks in production, I'd like to understand the design rationale behind two related behaviours in the rolling-update reconciler (pkg/controller/instanceset/reconciler_update.go).

---

Is your feature request related to a problem? Please describe.

Observation 1 – Conservative "block on unavailable" strategy

When KubeBlocks detects that a pod being upgraded is unavailable (not ready / not available), it halts the upgrade of that pod and all subsequent pods. This is more conservative than some community approaches (e.g., plain cloneset rolling update which keeps going as long as maxUnavailable allows).

Observation 2 – Pending pod special handling (added ~April 2025)

A Pending pod that needs updating is now deleted immediately and the reconciler returns early, bypassing the normal PodUpdatePolicy:

// treat old and Pending pod as a special case, as they can be updated without a consequence
// PodUpdatePolicy is ignored here since in-place update for a pending pod doesn't make much sense.
for i, pod := range oldPodList {
    if i >= rollingUpdateQuota {
        break
    }
    updatePolicy, _, _, err := getPodUpdatePolicy(its, pod)
    if err != nil {
        return kubebuilderx.Continue, err
    }
    if isPodPending(pod) && updatePolicy != noOpsPolicy {
        err = tree.Delete(pod)
        // wait another reconciliation, so that the following update process won't be confused
        return kubebuilderx.Continue, err
    }
}

[leon-ape]

The two behaviors address different concerns:

1. Stop a rollout when an instance fails to recover. maxUnavailable limits concurrency, while the readiness/availability checks prevent a failed update from spreading to subsequent instances. For example, when rolling out to 10 instances with concurrency 2, if the first two fail to become available, the remaining eight should wait instead of continuing the rollout. This was an explicit gradual-rollout requirement in #7913.

2. Allow recovery from a Pending pod that needs a spec change. For example, a vertical scaling operation may request more resources than the cluster can schedule. The pod stays Pending, and waiting for it to become Ready before applying a rollback would leave the operation stuck. Recreating it with the corrected spec lets recovery proceed; preserving the pod for an in-place update is not the priority in this case. This scenario is described in #9070. #9312 separated this recovery path from normal rolling updates while keeping Pending pods counted as unavailable. Returning after deletion lets the next reconciliation observe the result before proceeding.

[LuckyGrx]

Regarding Issue 1, the current behavior blocks the upgrade even when instances of the old version are unavailable—not just when instances of the new version are unavailable. However, upgrades are most commonly performed precisely because there are issues with the old-version instances; why doesn't the system support this scenario?