Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nil checks for hpa object target type values #121015

Merged

Conversation

Lukasz-AWS
Copy link
Contributor

@Lukasz-AWS Lukasz-AWS commented Oct 6, 2023

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fixes a bug where if the corresponding value (value, averageValue) of the the type of target (Value, AverageValue) is not set but the opposite type is, HPA object would pass validation but then cause kube-controller-manager to crash loop due to nil pointer dereference looking for that corresponding value. This PR adds a nil check to make sure kube-controller-manager doesn't crash. The change to fix the validation is being worked on here #120373

Which issue(s) this PR fixes:

N/A
Fixes #

Special notes for your reviewer:

Observed in 1.27 and 1.28, possibly might be an issue in earlier versions as well.
Can be replicated by deploying an hpa with a valid scaleTargetRef and adding this object to the metrics. On sync kcm will crash due to nil pointer dereference.

...
- apiVersion: autoscaling/v2
  kind: HorizontalPodAutoscaler
  spec:
    maxReplicas: 10
    metrics:
    - type: Object
      object:
        metric:
          name: requests-per-second
        describedObject:
          apiVersion: networking.k8s.io/v1beta1
          kind: Ingress
          name: main-route
        target:
          type: AverageValue
          value: 1
...

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 6, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 6, 2023
@k8s-ci-robot
Copy link
Contributor

Hi @Lukasz-AWS. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 6, 2023
@liggitt
Copy link
Member

liggitt commented Oct 6, 2023

/lgtm
/approve
/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 6, 2023
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 6, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 308ca288672b5a830a7059e8d77174e05e35ff14

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, Lukasz-AWS

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 6, 2023
@Lukasz-AWS
Copy link
Contributor Author

/retest

1 similar comment
@Lukasz-AWS
Copy link
Contributor Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit 0f16895 into kubernetes:master Oct 6, 2023
16 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.29 milestone Oct 6, 2023
@Lukasz-AWS
Copy link
Contributor Author

Not cherry picking to 1.26, as 1.26 will forcibly change the type.
i.e. if I apply

      target:
        type: AverageValue
        value: "1"

it becomes with no crashes

      target:
        type: Value
        value: "1"

@liggitt
Copy link
Member

liggitt commented Oct 6, 2023

Not cherry picking to 1.26, as 1.26 will forcibly change the type.

wait, what?

is that true when creating a brand new object?

@Lukasz-AWS
Copy link
Contributor Author

yes appears so, this is how I tested against the other versions which led to no changes and kcm crashing:

dev-dsk-luukasz-2b-3776d48b % kubectl version --short
...
Server Version: v1.26.9-eks-f8587cb
...

dev-dsk-luukasz-2b-3776d48b % kubectl get hpa
No resources found in default namespace.

dev-dsk-luukasz-2b-3776d48b % cat hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: test-126
  namespace: default
spec:
  maxReplicas: 10
  metrics:
  - object:
      describedObject:
        kind: Pod
        name: '*'
      metric:
        name: correlator_queue
      target:
        type: AverageValue
        value: "1"
    type: Object
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache

dev-dsk-luukasz-2b-3776d48b % kubectl apply -f hpa.yaml
horizontalpodautoscaler.autoscaling/test-126 created

dev-dsk-luukasz-2b-3776d48b % kubectl get hpa test-126 -oyaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"autoscaling/v2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"test-126","namespace":"default"},"spec":{"maxReplicas":10,"metrics":[{"object":{"describedObject":{"kind":"Pod","name":"*"},"metric":{"name":"correlator_queue"},"target":{"type":"AverageValue","value":"1"}},"type":"Object"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"php-apache"}}}
  creationTimestamp: "2023-10-06T21:19:18Z"
  name: test-126
  namespace: default
  resourceVersion: "254409"
  uid: 933c8c32-ea4a-40be-b0af-99c91d98c26a
spec:
  maxReplicas: 10
  metrics:
  - object:
      describedObject:
        kind: Pod
        name: '*'
      metric:
        name: correlator_queue
      target:
        type: Value
        value: "1"
    type: Object
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
status:
  currentMetrics: null
  desiredReplicas: 0

@liggitt
Copy link
Member

liggitt commented Oct 10, 2023

oh.... I bet that's because of v1 conversion logic, which infers the type from which *value field is non-nil (because there's no dedicated type field in v1):

var metricType autoscaling.MetricTargetType
if value == nil {
metricType = autoscaling.AverageValueMetricType
} else {
metricType = autoscaling.ValueMetricType
}

if in.AverageValue == nil {
metricType = autoscaling.ValueMetricType
} else {
metricType = autoscaling.AverageValueMetricType
}

var metricType autoscaling.MetricTargetType
if utilization == nil {
metricType = autoscaling.AverageValueMetricType
} else {
metricType = autoscaling.UtilizationMetricType
}

prior to #114358 which was included in 1.27, HPA objects were persisted in etcd in v1 format, so every HPA object would interact with the v1 conversion

@liggitt
Copy link
Member

liggitt commented Oct 10, 2023

go ahead and backport the change to 1.26 so the 1.26 controller is resilient to data it could encounter speaking to a 1.27 server

k8s-ci-robot added a commit that referenced this pull request Oct 12, 2023
…121015-upstream-release-1.27

Automated cherry pick of #121015: Add nil checks for hpa object target type values
k8s-ci-robot added a commit that referenced this pull request Oct 12, 2023
…121015-upstream-release-1.28

Automated cherry pick of #121015: Add nil checks for hpa object target type values
k8s-ci-robot added a commit that referenced this pull request Oct 12, 2023
…121015-upstream-release-1.26

Automated cherry pick of #121015: Add nil checks for hpa object target type values
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

3 participants