New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot autoscale StatefulSet via kubectl #48591

Closed
gyliu513 opened this Issue Jul 7, 2017 · 10 comments

Comments

Projects
None yet
6 participants
@gyliu513
Member

gyliu513 commented Jul 7, 2017

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug
/sig cli

What happened:

I can create HPA policy for StatefulSet but cannot scale the StatefulSet via kubectl.

root@pc001:~/istio-0.1.6# kubectl autoscale statefulset mysql --cpu-percent=50 --min=1 --max=10
error: cannot autoscale a {apps StatefulSet}

What you expected to happen:

I should be able to scale the StatefulSet via kubectl.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration**:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@georgebuckerfield

This comment has been minimized.

Show comment
Hide comment
@georgebuckerfield

georgebuckerfield Jul 7, 2017

Contributor

It looks like this isn't supported yet?
https://kubernetes.io/docs/user-guide/kubectl/v1.6/#autoscale

Related #44033

Contributor

georgebuckerfield commented Jul 7, 2017

It looks like this isn't supported yet?
https://kubernetes.io/docs/user-guide/kubectl/v1.6/#autoscale

Related #44033

@gyliu513

This comment has been minimized.

Show comment
Hide comment
@gyliu513

gyliu513 Jul 7, 2017

Member

It's weird, the document already claim support https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/ , let me check more

Member

gyliu513 commented Jul 7, 2017

It's weird, the document already claim support https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/ , let me check more

@georgebuckerfield

This comment has been minimized.

Show comment
Hide comment
@georgebuckerfield

georgebuckerfield Jul 7, 2017

Contributor

AFAIK, scaling is a separate thing to autoscaling. You should be able to do:

kubectl scale statefulset mysql --replicas=5

But not:

kubectl autoscale statefulset mysql ...
Contributor

georgebuckerfield commented Jul 7, 2017

AFAIK, scaling is a separate thing to autoscaling. You should be able to do:

kubectl scale statefulset mysql --replicas=5

But not:

kubectl autoscale statefulset mysql ...
@gyliu513

This comment has been minimized.

Show comment
Hide comment
@gyliu513

gyliu513 Jul 7, 2017

Member

Ah, I see, thanks @georgebuckerfield

Member

gyliu513 commented Jul 7, 2017

Ah, I see, thanks @georgebuckerfield

@gyliu513 gyliu513 changed the title from Cannot scale StatefulSet via kubectl to Cannot autoscale StatefulSet via kubectl Jul 7, 2017

@kow3ns

This comment has been minimized.

Show comment
Hide comment
@kow3ns

kow3ns Jul 11, 2017

Member

@gyliu513 This isn't a bug. Autoscaling isn't implemented for StatefulSets or DaemonSets. Auto-scaling a stateless application based on cpu utilization generally makes sense. Autoscaling a stateful workload based on that criteria can often do more harm (at least in the short term) than good. If you are asking for an implementation of horizontal Pod auto-scaling for StatefulSet you should open this as a feature request.

Member

kow3ns commented Jul 11, 2017

@gyliu513 This isn't a bug. Autoscaling isn't implemented for StatefulSets or DaemonSets. Auto-scaling a stateless application based on cpu utilization generally makes sense. Autoscaling a stateful workload based on that criteria can often do more harm (at least in the short term) than good. If you are asking for an implementation of horizontal Pod auto-scaling for StatefulSet you should open this as a feature request.

@gyliu513

This comment has been minimized.

Show comment
Hide comment
@gyliu513

gyliu513 Jul 11, 2017

Member

Thanks @kow3ns , can you please explain more for why scale a statefulset often do more harm than good?

Member

gyliu513 commented Jul 11, 2017

Thanks @kow3ns , can you please explain more for why scale a statefulset often do more harm than good?

@kow3ns

This comment has been minimized.

Show comment
Hide comment
@kow3ns

kow3ns Jul 12, 2017

Member

HPA based on CPU makes two assumptions.

  1. CPU varies with direct proportion to QPS
  2. QPS is load balanced, and when auto scaling occurs, the increased capacity of a new Pod will decrease the per Pod QPS, and therefore will decrease the per Pod CPU utilization.

Stateful workloads may violate .1. For example, log structured merge trees and append only btrees may cause CPU spikes during compaction that are only loosely correlated with QPS and that do not vary with direct proportion to the current QPS.

More importantly, for distributed storage applications, when you increase the number of processes in the system by increasing the number of Pods, you may trigger replication or replica re-balancing (depending on the nature of the stateful application). This is likely to increase network and CPU utilization in the short term. Only after the re-balance or replication is complete would you be able to make use of the additional capacity offered by the new Pods.

For synchronously and asynchronously replicated ACID systems, reducing the number or replicas based on CPU utilization might be safe, but, for BASE systems, scaling down will also trigger a re-balance (so the systems data are not under-replicated), and potentially require global anti-entropy repair.

That isn't to say that Autoscaling is not possible, but it is not likely to have a trivial, general solution for StatefulSet. That is why I suggest opening this as a feature request.

Member

kow3ns commented Jul 12, 2017

HPA based on CPU makes two assumptions.

  1. CPU varies with direct proportion to QPS
  2. QPS is load balanced, and when auto scaling occurs, the increased capacity of a new Pod will decrease the per Pod QPS, and therefore will decrease the per Pod CPU utilization.

Stateful workloads may violate .1. For example, log structured merge trees and append only btrees may cause CPU spikes during compaction that are only loosely correlated with QPS and that do not vary with direct proportion to the current QPS.

More importantly, for distributed storage applications, when you increase the number of processes in the system by increasing the number of Pods, you may trigger replication or replica re-balancing (depending on the nature of the stateful application). This is likely to increase network and CPU utilization in the short term. Only after the re-balance or replication is complete would you be able to make use of the additional capacity offered by the new Pods.

For synchronously and asynchronously replicated ACID systems, reducing the number or replicas based on CPU utilization might be safe, but, for BASE systems, scaling down will also trigger a re-balance (so the systems data are not under-replicated), and potentially require global anti-entropy repair.

That isn't to say that Autoscaling is not possible, but it is not likely to have a trivial, general solution for StatefulSet. That is why I suggest opening this as a feature request.

@gyliu513

This comment has been minimized.

Show comment
Hide comment
@gyliu513

gyliu513 Jul 12, 2017

Member

Thanks @kow3ns for the detailed explanation, there is already a feature request #44033 , so I will close this one.

Member

gyliu513 commented Jul 12, 2017

Thanks @kow3ns for the detailed explanation, there is already a feature request #44033 , so I will close this one.

@gyliu513 gyliu513 closed this Jul 12, 2017

@sjbarrio

This comment has been minimized.

Show comment
Hide comment
@sjbarrio

sjbarrio commented Apr 10, 2018

@YueHonghui

This comment has been minimized.

Show comment
Hide comment
@YueHonghui

YueHonghui Apr 20, 2018

@kow3ns I think Statefulset can't meet the requirements of applications like database、distributed storage service or even spark. Those applications is too complicate to fit in statefulset. They should be manipulated by customized controller through CRD, like what so many Operators do.
We need a controller which provides persistent identifier for pods, so we can create reference between pods and other resource persistently. It is reasonable to provide ability of horizontally auto scaling for such types of applications.

YueHonghui commented Apr 20, 2018

@kow3ns I think Statefulset can't meet the requirements of applications like database、distributed storage service or even spark. Those applications is too complicate to fit in statefulset. They should be manipulated by customized controller through CRD, like what so many Operators do.
We need a controller which provides persistent identifier for pods, so we can create reference between pods and other resource persistently. It is reasonable to provide ability of horizontally auto scaling for such types of applications.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment