Feature Request: Scale in StatefulSet at an arbitrary position

[cofyc]

What would you like to be added:

We encountered a problem in production to run databases with local PVs and StatefulSet. It's inconvenient to scale in the database when pods on the node to delete are not the last ordinal indices of the StatefulSet.

I hope we can scale in StatefulSet at an arbitrary position, e.g. delete pod-1 but keep pod-2 in a StatefulSet of 3 replicas.

Why is this needed:

Although pods created from the same spec in StatefulSet, but are not interchangeable because each of them has associated persistent storage.

node A	node B	node C
pod-0	pod-2	pod-1

Using local PVs, pods are tied to nodes. When we scale in the node pool or take down a node in the on-premise data center. If the pod on that node is not the last ordinal of the StatefulSet, it's hard to scale in the StatefulSet to free the node. We need at the first scale in the StatefulSet to free the node B, and then migrate the pod-1 to node B.

It requires to migrate data between nodes. It's often unacceptable when the amount of data on the node is large, e.g. databases.

Current StatefulSet has some limitations when pods cannot be rescheduled freely. So, I think it's time to implement an alternative indexing method (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/apps/stateful-apps.md#how-identities-are-managed).

An intuitive solution is to move off the deleted slots. The network identifiers are ordinal, stable the same as before except holes can exist.

I've done a quick hack on builtin StatefulSet using annotations: cofyc/kubernetes@1aea0dd...cofyc:scale-in-statefulset-at-arbitrary-position and verified that basic operations like create, scale in/out work.

[Joseph-Irving]

/sig apps

[msau42]

@kubernetes/sig-apps-feature-requests

[tedyu]

basic operations like create, stale in/out work

Did you mean scale in/out ?

[cofyc]

Yes, sorry, my typo.

[cofyc]

FYI, I've implemented this feature in CRD based on Kubernetes 1.16.0 StatefulSet code: https://github.com/cofyc/advanced-statefulset

edit: new home https://github.com/pingcap/advanced-statefulset

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[haosdent]

@cofyc I think your CRD solution is preferred, so this issue could be closed?

[cofyc]

this can be closed

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.