Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HPA scales up deployment to max replicas if using custom metrics that are exposed by the application and the pods fail to deploy with rolling update #87199

Open
hallgeirl opened this issue Jan 14, 2020 · 1 comment

Comments

@hallgeirl
Copy link

@hallgeirl hallgeirl commented Jan 14, 2020

What happened:
We're trying to use Prometheus+Prometheus Adapter+windows performance counters to expose the number of HTTP requests to IIS for use in autoscaling of our Windows-based application in Kubernetes. When the containers start up properly, everything is fine - our application is scaled according to the config we have set. However, if we perform a rolling update, and the updated deployment crashes, the HPA starts spinning up new pods until it reaches the limit.

When I then look at the ReplicaSets, it seems to scale the NEW ReplicaSet (where all the containers are crashing), and leaving the old one alone. If I directly create a deployment that fails, without any previously running containers, then it will just fail with one pod, and not scale up.

What you expected to happen:
I expect the deployment not to scale out during a rolling update, at least not on the new ReplicaSet where all pods are failing. When none of the pods are up, there's no basis for deciding how many pods should be available.

How to reproduce it (as minimally and precisely as possible):
It took some time, but I've found a minimal repro for this. I've attached a zip file with a minimal .NET Core app running on Windows nano server, as well as YAML files for Kubernetes.
All this .NET Core app does, is to serve a /metrics endpoint on port 5000, reporting a metric "iis_requests_per_second" which has been set static to 10. The HPA scaling limit has been set to 80.
testapp.zip

  1. Download the zip, edit build-and-push.ps1, kube.yaml and kube_crash.yaml and change the container registry hostname to one you have access to.
  2. Run build-and-publish.ps1 (in Windows Powershell) to build the image and publish it to the container registry.
  3. Deploy the "non-crashing" deployment.
    > kubectl apply -f .\kube.yaml
  4. Change the default namespace to "test":
    > kubectl config set-context --namespace test --current
  5. Deploy the HPA
    > kubectl apply -f .\hpa.yaml
  6. Verify that the pod count is the same as before (should be one), and that the HPA is available.
  > kubectl get hpa
  NAME               REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  autoscaler-myapp   Deployment/myapp   10/80     1         3         1          50s

After a couple of minutes the count should still be the same. And as you see, the metric is correctly retrieved.

  1. Then the fun begins... try deploying the kube_crash.yaml file:
    > kubectl apply -f .\kube_crash.yaml

The ONLY difference is that it provides a command line parameter to the container that makes it crash before the metrics page is up.
8. Now wait until the pod has crashed (should take a minute or so). Then, after a few more seconds, this happens:

  > kubectl get hpa
  NAME               REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  autoscaler-myapp   Deployment/myapp   10/80     1         3         2          5m38s
  
  > kubectl get pods 
  NAME                     READY   STATUS              RESTARTS   AGE
  myapp-7955776b7c-lj2f7   1/1     Running             0          7m47s
  myapp-8dc77668f-5wpdz    0/1     CrashLoopBackOff    2          91s
  myapp-8dc77668f-7zjgn    0/1     ContainerCreating   0          22s

After a few more seconds, yet another one is spawned:

  > kubectl get hpa; kubectl get pods
  NAME               REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  autoscaler-myapp   Deployment/myapp   10/80     1         3         3          6m25s

  NAME                     READY   STATUS             RESTARTS   AGE
  myapp-7955776b7c-lj2f7   1/1     Running            0          8m33s
  myapp-8dc77668f-5wpdz    0/1     CrashLoopBackOff   3          2m17s
  myapp-8dc77668f-7zjgn    0/1     CrashLoopBackOff   1          68s
  myapp-8dc77668f-85jld    0/1     CrashLoopBackOff   1          38s

Note that NONE of these ever reached "Ready" state. And if I check the details on the HPA:

  > kubectl describe hpa autoscaler-myapp
  Name:                                 autoscaler-myapp
  Namespace:                            test
  Labels:                               <none>
  Annotations:                          kubectl.kubernetes.io/last-applied-configuration:
                                          {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"autoscaler-myapp","namespace":"t...
  CreationTimestamp:                    Tue, 14 Jan 2020 14:28:07 +0100
  Reference:                            Deployment/myapp
  Metrics:                              ( current / target )
    "iis_requests_per_second" on pods:  10 / 80
  Min replicas:                         1
  Max replicas:                         3
  Deployment pods:                      3 current / 3 desired
  Conditions:
    Type            Status  Reason            Message
    ----            ------  ------            -------
    AbleToScale     True    ReadyForNewScale  recommended size matches current size
    ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from pods metric iis_requests_per_second
    ScalingLimited  True    TooManyReplicas   the desired replica count is more than the maximum replica count
  Events:
    Type    Reason             Age    From                       Message
    ----    ------             ----   ----                       -------
    Normal  SuccessfulRescale  2m33s  horizontal-pod-autoscaler  New size: 2; reason: pods metric iis_requests_per_second above target
    Normal  SuccessfulRescale  2m3s   horizontal-pod-autoscaler  New size: 3; reason: pods metric iis_requests_per_second above target

It does not say that it couldn't get metrics - it says that the metrics are above target. Which they CAN'T be because they're hard-coded. And as you see, the reported metrics on the HPA is 10/80, so WELL below the threshold.
If I now deploy the one that doesn't crash, it will deploy 3 replicas, then after a while start scaling back down to 1. This takes 5 minutes or so due to the cooldown on the HPA.

  > kubectl apply -f .\kube.yaml
  > kubectl get pods
  NAME                     READY   STATUS    RESTARTS   AGE
  myapp-7955776b7c-d579m   1/1     Running   0          49s
  myapp-7955776b7c-lj2f7   1/1     Running   0          14m
  myapp-7955776b7c-xfxkj   1/1     Running   0          69s

After about 5 minutes:

  > kubectl describe hpa autoscaler-myapp
  Name:                                 autoscaler-myapp
  Namespace:                            test
  Labels:                               <none>
  Annotations:                          kubectl.kubernetes.io/last-applied-configuration:
                                          {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"autoscaler-myapp","namespace":"t...
  CreationTimestamp:                    Tue, 14 Jan 2020 14:28:07 +0100
  Reference:                            Deployment/myapp
  Metrics:                              ( current / target )
    "iis_requests_per_second" on pods:  10 / 80
  Min replicas:                         1
  Max replicas:                         3
  Deployment pods:                      1 current / 1 desired
  Conditions:
    Type            Status  Reason              Message
    ----            ------  ------              -------
    AbleToScale     True    ReadyForNewScale    recommended size matches current size
    ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from pods metric iis_requests_per_second
    ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
  Events:
    Type    Reason             Age    From                       Message
    ----    ------             ----   ----                       -------
    Normal  SuccessfulRescale  14m    horizontal-pod-autoscaler  New size: 2; reason: pods metric iis_requests_per_second above target
    Normal  SuccessfulRescale  13m    horizontal-pod-autoscaler  New size: 3; reason: pods metric iis_requests_per_second above target
    Normal  SuccessfulRescale  2m57s  horizontal-pod-autoscaler  New size: 2; reason: All metrics below target
    Normal  SuccessfulRescale  2m41s  horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
    

If I delete the whole namespace and deploy the failing deployment directly (i.e., there's no running pods), the HPA seems to report that there's no metrics available, and it won't scale (as I would assume is the correct behavior):

 > kubectl create namespace test
 > kubectl apply -f .\hpa.yaml
 > kubectl apply -f .\kube_crash.yaml

 > kubectl describe hpa autoscaler-myapp
  Name:                                 autoscaler-myapp
  Namespace:                            test
  Labels:                               <none>
  Annotations:                          kubectl.kubernetes.io/last-applied-configuration:
                                          {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"autoscaler-myapp","namespace":"t...
  CreationTimestamp:                    Tue, 14 Jan 2020 14:51:38 +0100
  Reference:                            Deployment/myapp
  Metrics:                              ( current / target )
    "iis_requests_per_second" on pods:  <unknown> / 80
  Min replicas:                         1
  Max replicas:                         3
  Deployment pods:                      1 current / 0 desired
  Conditions:
    Type           Status  Reason               Message
    ----           ------  ------               -------
    AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
    ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric iis_requests_per_second: no metrics returned from custom metrics API
  Events:
    Type     Reason                        Age                From                       Message
    ----     ------                        ----               ----                       -------
    Warning  FailedGetPodsMetric           2s (x9 over 2m3s)  horizontal-pod-autoscaler  unable to get metric iis_requests_per_second: no metrics returned from custom metrics API
    Warning  FailedComputeMetricsReplicas  2s (x9 over 2m3s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric iis_requests_per_second: no metrics returned from custom metrics API

Then it does NOT scale up, which is what I would expect.

Anything else we need to know?:
I would expect the HPA to not try to scale the new ReplicaSet before any metrics has been reported. I would at least expect it to not scale to max when it can't find metrics for the containers in that ReplicaSet.

Environment:

  • Kubernetes version (use kubectl version):
    kubectl version
    Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"windows/amd64"}
    Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-13T17:24:45Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration: Azure (AKS)

  • OS (e.g: cat /etc/os-release): We're using Windows containers on Windows node pools.

  • Kernel (e.g. uname -a): Windows 1809 I believe is running currently on AKS

  • Install tools:

  • Network plugin and version (if this is a network-related bug):

  • Others:
    Not sure if this is relevant because Prometheus/prometheus-adapter seems to serve the correct metrics. But for completeness:
    Prometheus: Image: prom/prometheus:v2.2.1
    Prometheus-adapter: Installed through Helm - v0.5.0

@hallgeirl

This comment has been minimized.

Copy link
Author

@hallgeirl hallgeirl commented Jan 14, 2020

/sig autoscaling

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.