StatefulSet with podManagementPolicy: Parallel should create pods in parallel

[aleksandra-malinowska]

What would you like to be added?

When podManagementPolicy: Parallel is set, StatefulSetController should issue create requests for replicas in batches (similar to how other controllers work, e.g. ReplicaSetController) when creating pods for the first time and scaling up.

Why is this needed?

Currently even with Parallel podManagementPolicy StatefulSet the replicas are still created sequentially, even though they can start to run in any order.

This severely limits pod throughput when working with large StatefulSets (~thousands of replicas). For example, 10 StatefulSets with 5k pods each achieve a total of ~6 pods/s, while 1k StatefulSets with 50 pods each allows for creating ~120 pods/s.

Parallel policy suggests the pods can be created sequentially (https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/stateful-set-v1/#StatefulSetSpec):

The alternative policy is Parallel which will create pods in parallel to match the desired scale without waiting, and on scale down will delete all pods at once.

Relevant code (Parallel policy controls the value of monotonic): https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/statefulset/stateful_set_control.go#L424)

[aleksandra-malinowska]

/sig apps

cc @soltysh @wojtek-t @mborsz

[aleksandra-malinowska]

/assign @aleksandra-malinowska

[aleksandra-malinowska]

Given that there is a mismatch between documented and actual behavior, it's likely a bugfix and not a feature request.

/kind bug

[aleksandra-malinowska]

/remove-kind feature

[soltysh]

/triage accepted