Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manual scale doesn't work if hpaSpec is set #2816

Closed
stingshen opened this issue Jan 7, 2021 · 4 comments
Closed

Manual scale doesn't work if hpaSpec is set #2816

stingshen opened this issue Jan 7, 2021 · 4 comments
Assignees
Labels

Comments

@stingshen
Copy link

stingshen commented Jan 7, 2021

Describe the bug

Manual scale doesn't work if hpaSpec is set.

To reproduce

Apply the following yaml to k8s:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  replicas: 5
  predictors:
  - componentSpecs:
    - hpaSpec:
        maxReplicas: 3
        metrics:
        - resource:
            name: cpu
            targetAverageUtilization: 10
          type: Resource
        minReplicas: 1
      replicas: 2
      spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.3
          imagePullPolicy: IfNotPresent
          name: classifier
          resources:
            requests:
              cpu: '0.2'
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: example

Manual replica setting doesn't work. There is only one replica created.

Expected behaviour

There are 2 replicas created.

Environment

seldon 1.3.0 with K8s 1.18.4

@stingshen stingshen added bug triage Needs to be triaged and prioritised accordingly labels Jan 7, 2021
@ukclivecox
Copy link
Contributor

Are you saying for a Deployment + HPA with above setting if you try to manually set replicas to 5 it will go to 3? In this case I would expect maybe 2 as you set that for replicas in predictor?

@axsaucedo axsaucedo added this to Triage in Backlog Jan 7, 2021
@stingshen
Copy link
Author

stingshen commented Jan 8, 2021

Sorry, I wrote the wrong expected replica number. The expected behavior is 2 replicas are created. The actual behavior is there is only one replica created by seldon deployment controller.

@ukclivecox ukclivecox removed the triage Needs to be triaged and prioritised accordingly label Jan 14, 2021
@ukclivecox ukclivecox moved this from Triage to Priority p2 in Backlog Jan 14, 2021
@ukclivecox ukclivecox moved this from Priority p2 to priority 1 in Backlog Jan 21, 2021
@ukclivecox ukclivecox moved this from priority 1 to Next Sprint in Backlog Jan 21, 2021
@axsaucedo axsaucedo added this to To do in Sprint 3 via automation Feb 8, 2021
@axsaucedo axsaucedo removed this from Next Sprint in Backlog Feb 8, 2021
@ukclivecox ukclivecox added this to To do in MLOPs Sprint 1 via automation Feb 8, 2021
@ukclivecox ukclivecox self-assigned this Feb 11, 2021
@ukclivecox ukclivecox moved this from To do to In progress in MLOPs Sprint 1 Feb 18, 2021
@ukclivecox
Copy link
Contributor

We are following the advice here kubernetes/kubernetes#25238 (comment)
Replica setting is ignored when an HPA is applied. With normal Deployments you should not set replicas when using an HPA.

Please reopen if you think this is not correct.

MLOPs Sprint 1 automation moved this from In progress to Done Feb 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Development

No branches or pull requests

3 participants
@stingshen @ukclivecox and others