Arbitrary/Custom Metrics in the Horizontal Pod Autoscaler

[DirectXMan12]

Arbitrary/Custom Metrics in the Horizontal Pod Autoscaler

• One-line feature description (can be used as a release note): Reworked Horizontal Pod Autoscaler functionality, including support for custom metrics and status conditions.
• Primary contact (assignee): @DirectXMan12
• Responsible SIGs: @kubernetes/sig-autoscaling-feature-requests
• Design proposal links (community repo):
  • HPA v2
  • HPA v2 status conditions
  • HPA v2 external metrics
  • HPA v2 metrics specificity improvements
• Testing
  • Unit tests: https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/podautoscaler
  • https://github.com/kubernetes/kubernetes/blob/master/test/e2e/autoscaling/custom_metrics_stackdriver_autoscaling.go
  • Custom metrics autoscaling e2e adapter tests kubernetes#62351
• Reviewer(s): @mwielgus @MaciekPytel
• Approver (likely from SIG/area to which feature belongs): @mwielgus
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y): 1.6
  • Beta release target (x.y): 1.8, 1.12 (pending)
  • Stable release target (x.y)

[DirectXMan12]

cc @kubernetes/autoscaling @jszczepkowski @derekwaynecarr @smarterclayton

[idvoretskyi]

@DirectXMan12 any updates on this issue? Can you provide the actual status of it and update the checkboxes above?

[DirectXMan12]

The proposal is posted, but has not been approved yet. We only recently reached general consensus about the design. Still finalizing the exact semantics. It should be removed from the 1.5 milestone, since no code will have gone into 1.5

[idvoretskyi]

@DirectXMan12 thank you for clarifying.

[idvoretskyi]

@DirectXMan12 please, provide us with the release notes and documentation PR (or links) at https://docs.google.com/spreadsheets/d/1nspIeRVNjAQHRslHQD1-6gPv99OcYZLMezrBe3Pfhhg/edit#gid=0

[DirectXMan12]

@DirectXMan12 please, provide us with the release notes and documentation PR (or links)

done :-)

[bgrant0607]

@DirectXMan12 @mwielgus What is planned for HPA in 1.8?

[fgrzadkowski]

@MaciekPytel @kubernetes/sig-autoscaling-misc

[evmin]

Looking forward to the beta launch!

[DirectXMan12]

@bgrant0607 we're hoping to move to v2 to beta in 1.8 (so just stabilization :-) ).

[davidopp]

Regarding the functionality - am I correct in understanding that if you wanted to scale on a load indicator that is fundamentally external to Kubernetes, you would need to

1. create some kind of proxy API object inside the cluster (maybe a CRD?) that reflects the load indicator in a manner that the Kubernetes metrics pipeline can consume
2. create an HPA with MetricSourceType == "Object" and point to the proxy object

[DirectXMan12]

@davidopp That is incorrect. In order to scale on a load indicator that's not one of the metrics provided by the resource metrics API (CPU, memory), you need to have some implementation of the custom metrics API (see k8s.io/metrics and kubernetes-incubator/custom-metrics-apiserver).

Then, you can either use the "pods" source type, if the metric describes the pods controlled the the target scalable of the HPA (e.g. network throughput), or the "object" source type, if the metric describes an unrelated object (for instance, you might scale on a queue length metric attached to the namespace).

In either case, the HPA controller will the query the custom metrics API accordingly. It is up to cluster admins, etc to actually provide a method to collect the given metrics and expose an implementation of the custom metrics API.

[palnabarun]

/remove-lifecycle stale

[jeremyrickard]

Hey there @mwielgus and @josephburnett -- 1.18 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to stable in 1.18 or having a major change in it's current level?

The current release schedule is:
Monday, January 6th - Release Cycle Begins
Tuesday, January 28th EOD PST - Enhancements Freeze
Thursday, March 5th, EOD PST - Code Freeze
Monday, March 16th - Docs must be completed and reviewed
Tuesday, March 24th - Kubernetes 1.18.0 Released

To be included in the release, this enhancement must have a merged KEP in the implementable status. The KEP must also have graduation criteria and a Test Plan defined.
If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍
We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements
Thanks!

[jeremyrickard]

Hey there @mwielgus and @josephburnett, Enhancements Team reaching out again. We're about a week out from Enhancement Freeze on the 28th. Let us know if you think there will be any activity on this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[palnabarun]

Hey there @mwielgus @josephburnett -- 1.19 Enhancements Lead here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

1. The KEP PR must be merged in an implementable state
2. The KEP must have test plans
3. The KEP must have graduation criteria.

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released
• Thursday, August 20: Week 19 - Release Retrospective

If you do, I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[palnabarun]

Hi @mwielgus @josephburnett, pinging back again as a reminder. 🙂

[palnabarun]

Hi @mwielgus @josephburnett

Tomorrow, Tuesday May 19 EOD Pacific Time is Enhancements Freeze

Will this enhancement be part of the 1.19 release cycle?

[palnabarun]

@mwielgus @josephburnett -- Unfortunately, the deadline for the 1.19 Enhancement freeze has passed. For now, this is being removed from the milestone and 1.19 tracking sheet. If there is a need to get this in, please file an enhancement exception.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[kikisdeliveryservice]

Hi @mwielgus @josephburnett

Enhancements Lead here. Any plans for this to graduate in 1.20?

Thanks!
Kirsten

[kikisdeliveryservice]

Hi @mwielgus @josephburnett

Any updates on whether this will be included in 1.20?

Enhancements Freeze is October 6th and by that time we require:

The KEP must be merged in an implementable state
The KEP must have test plans
The KEP must have graduation criteria
The KEP must have an issue in the milestone

I note that your design proposals are quite old, please consider updating to the new KEP format. See: https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template

Thanks
Kirsten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[pytimer]

@DirectXMan12 I read the Enhance HPA Metrics Specificity doc, i also use the hpa with metricLabelSelector, but i found because of metricLabelSelector is labels.Selector, so if my metric label value is /v1/task, i cannot use selector, becasue label value only allow (([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?'), could consider support filter any value, not only k8s label selector regexp?

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.