-
Notifications
You must be signed in to change notification settings - Fork 831
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scaling of SeldonSeployments #884
Comments
Hi, I saw similar issue. In case of kubernetes deployments if we use updateStrategy Rolling it works the following way:
In case of SeldonDeployment with single model (in 100 instances):
Is it the normal behavior, or I missed something during configuration? |
Related to #1078 |
I suspect this fits the pattern described in #1110. The predictor spec is fed into all the pods for the engine to read.If that changes, then it prompts a rolling update. And the replicas are part of the predictor spec so this makes sense. A fix could be to omit the number of replicas from the spec that is fed to the engine as the engine doesn't really need to know that bit. |
Yes it seems that the value of env variable ENGINE_PREDICTOR changes at every replica change |
The number of replicas should not be in the ENGINE_PREDICTOR. And also we should indeed look at the rolling update strategy. This may be because the Deployment name changes as it based on the hash of the spec. This needs to be reviewed. |
Seldon 4.0
When I change the replicas field in a SeldonDeployment e.g. from 3 to 4, then I can see that basically 4 new pods are created and the old 3 are terminated.
This behavior seems very strange.
When I do a scaling operation on the deployment level (which belongs to the SeldonDeployment), it is handled smoothly, i.e. only 1 new pod is created in order to achieve the new number of replicas. Although I guess that this kind of scaling should not be performed manually, only via the SeldonDeployment resource.
What is the reason for this? Am I missing some config setting?
Thanks,
Toni
The text was updated successfully, but these errors were encountered: