-
Notifications
You must be signed in to change notification settings - Fork 267
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Support For Pods Auto Scaling #800
Comments
Can you share the details of when you trigger the autoscaling and how you define it right now? It's definitely something we can consider but it's also not something that is as easy as it sounds (making sure it's triggered at the right time) and in many cases doesn't actually provide any/significant benefit (in fact, it can make the performance worse since queues need to be rebalanced, based on the policies additional mirrors/followers may need to be created, etc). |
This would be the HorizontalPodAutoScaler.yaml which defines the policy for scaling. apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: $(ENVIRONMENT)-rabbitmq-stateful-set-autoscaler
namespace: $(ENVIRONMENT)-rabbitmq
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: $(ENVIRONMENT)-rabbitmq-stateful-set
minReplicas: 3
maxReplicas: 9
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 75
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 75 As can be seen a scale operation occurs when 75% of memory or cpu resources are utilized in all pods combined. And in the StatefulSet.yaml im just using VolumeClaimTemplates to claim volumes for each created pod. volumeClaimTemplates:
- metadata:
name: $(ENVIRONMENT)-rabbitmq-data
namespace: $(ENVIRONMENT)-rabbitmq
spec:
storageClassName: $(STORAGE_CLASS)
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 1Gi The way I see it, correct me if im wrong, every node has a claim for its own volume, and data is not shared between nodes, thus rebalancing is not needed when a new node is going live. (it will just connect to the cluster and as soon as it's ready it will be able to handle incoming messages) The only thing I can think of that might need special treatment is when a scale down operation occurs, but it can be achieved without rebalancing. Maybe im not aware enough of other edge cases but from my tests I didn't run in to problems even without special handling for scale downs. |
So there are two key aspects:
Finally, do you have any metrics showing that what you configured, helped to handle your traffic? Have you seen increased throughput and/or lower latency after additional nodes were deployed? Have you seen CPU and/or RAM usage going down after the scaling event? You can read more about what's involved just to handle a "simple" scale-down operation here: #223. Best, |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
I want to use the operator to deploy a RabbitMQ cluster on Kubernetes but I couldn't find any way to define auto scaling.
Is there any reason it's not supported ? is it planned for the future ?
When I create my custom deployment im just defining a HorizontalPodAutoscaler object and it takes care of the rest, so i thought it should be supported in the official operator as it is not a complicated matter.
Insights into it would be appreciated.
The text was updated successfully, but these errors were encountered: