Arbitrary/Custom Metrics in the Horizontal Pod Autoscaler

[DirectXMan12]

Arbitrary/Custom Metrics in the Horizontal Pod Autoscaler

• One-line feature description (can be used as a release note): Reworked Horizontal Pod Autoscaler functionality, including support for custom metrics and status conditions.
• Primary contact (assignee): @DirectXMan12
• Responsible SIGs: @kubernetes/sig-autoscaling-feature-requests
• Design proposal links (community repo):
  • HPA v2
  • HPA v2 status conditions
  • HPA v2 external metrics
  • HPA v2 metrics specificity improvements
• Testing
  • Unit tests: https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/podautoscaler
  • https://github.com/kubernetes/kubernetes/blob/master/test/e2e/autoscaling/custom_metrics_stackdriver_autoscaling.go
  • kubernetes/kubernetes#62351
• Reviewer(s): @mwielgus @MaciekPytel
• Approver (likely from SIG/area to which feature belongs): @mwielgus
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y): 1.6
  • Beta release target (x.y): 1.8, 1.12 (pending)
  • Stable release target (x.y)

[DirectXMan12]

cc @kubernetes/autoscaling @jszczepkowski @derekwaynecarr @smarterclayton

[idvoretskyi]

@DirectXMan12 any updates on this issue? Can you provide the actual status of it and update the checkboxes above?

[DirectXMan12]

The proposal is posted, but has not been approved yet. We only recently reached general consensus about the design. Still finalizing the exact semantics. It should be removed from the 1.5 milestone, since no code will have gone into 1.5

[idvoretskyi]

@DirectXMan12 thank you for clarifying.

[idvoretskyi]

@DirectXMan12 please, provide us with the release notes and documentation PR (or links) at https://docs.google.com/spreadsheets/d/1nspIeRVNjAQHRslHQD1-6gPv99OcYZLMezrBe3Pfhhg/edit#gid=0

[DirectXMan12]

@DirectXMan12 please, provide us with the release notes and documentation PR (or links)

done :-)

[bgrant0607]

@DirectXMan12 @mwielgus What is planned for HPA in 1.8?

[fgrzadkowski]

@MaciekPytel @kubernetes/sig-autoscaling-misc

[evmin]

Looking forward to the beta launch!

[DirectXMan12]

@bgrant0607 we're hoping to move to v2 to beta in 1.8 (so just stabilization :-) ).

[davidopp]

Regarding the functionality - am I correct in understanding that if you wanted to scale on a load indicator that is fundamentally external to Kubernetes, you would need to

1. create some kind of proxy API object inside the cluster (maybe a CRD?) that reflects the load indicator in a manner that the Kubernetes metrics pipeline can consume
2. create an HPA with MetricSourceType == "Object" and point to the proxy object

[DirectXMan12]

@davidopp That is incorrect. In order to scale on a load indicator that's not one of the metrics provided by the resource metrics API (CPU, memory), you need to have some implementation of the custom metrics API (see k8s.io/metrics and kubernetes-incubator/custom-metrics-apiserver).

Then, you can either use the "pods" source type, if the metric describes the pods controlled the the target scalable of the HPA (e.g. network throughput), or the "object" source type, if the metric describes an unrelated object (for instance, you might scale on a queue length metric attached to the namespace).

In either case, the HPA controller will the query the custom metrics API accordingly. It is up to cluster admins, etc to actually provide a method to collect the given metrics and expose an implementation of the custom metrics API.

[DirectXMan12]

Awesome, looking forward to this! However, any plans for stable?

eventually. Realistically, I want to give a release or two for feedback on the new stuff, so that we can be sure we're happy with the shape of the API, then we'll look towards stable.

[mistyhacks]

@DirectXMan12 please fill out the appropriate line item of the
1.11 feature tracking spreadsheet
and open a placeholder docs PR against the
release-1.11 branch
by 5/25/2018 (tomorrow as I write this) if new docs or docs changes are
needed and a relevant PR has not yet been opened.

[DirectXMan12]

done.

[mistyhacks]

Thank you 🙌

[DirectXMan12]

@kubernetes/sig-release-misc due to some last-minute hiccups, the corresponding PR for this feature's updates did not get merged for 1.11, so there's no new developments for this feature in 1.11.

CC @kubernetes/sig-autoscaling-misc

[idvoretskyi]

FYI @justaugustus ^

[justaugustus]

Thanks for the update on this, @DirectXMan12!

[justaugustus]

@DirectXMan12 @kubernetes/sig-autoscaling-feature-requests --
This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y)
  • Beta release target (x.y)
  • Stable release target (x.y)

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

[justaugustus]

This feature current has no milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y)
  • Beta release target (x.y)
  • Stable release target (x.y)

Set the following:

• Description
• Assignee(s)
• Labels:
  • stage/{alpha,beta,stable}
  • sig/*
  • kind/feature

Once this feature is appropriately updated, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.

---

Please note that Features Freeze is tomorrow, July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

P.S. This was sent via automation

[MaciekPytel]

@DirectXMan12 Are the changes to API (adding metricselector to pods/object metrics, etc) still in scope for 1.12? If so this should probably be added to milestone.

@mwielgus Anything else that should be added here for 1.12?

[DirectXMan12]

It should be in the milestone. I believe the description is up to date. If it's not, please let me know.

/milestone 1.12

[justaugustus]

Thanks for the updates! I've added this to the tracking sheet for 1.12.

[zparnold]

Hey there! @DirectXMan12 I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

[tomkerkhove]

Would be great if we can include this one as well - kubernetes/kubernetes#66988

[DirectXMan12]

@zparnold already did that quite some time ago: kubernetes/website#8757

[DirectXMan12]

just need to rebase it ;-)

[zparnold]

Thank you Solly!

…

On Tue, Aug 21, 2018 at 12:54 PM Solly Ross ***@***.***> wrote: just need to rebase it ;-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#117 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE81SC_ot8w-RDn1UDWAUZpny6CuaRa9ks5uTElngaJpZM4KRcvs> .

[claurence]

Kubernetes 1.13 is going to be a 'stable' release since the cycle is only 10 weeks. We encourage no big alpha features and only consider adding this feature if you have a high level of confidence it will make code slush by 11/09. Are there plans for this enhancement to graduate to alpha/beta/stable within the 1.13 release cycle? If not, can you please remove it from the 1.12 milestone or add it to 1.13?

We are also now encouraging that every new enhancement aligns with a KEP. If a KEP has been created, please link to it in the original post. Please take the opportunity to develop a KEP

[kacole2]

@DirectXMan12 I'm following up on @claurence's post to see if there are any plans for this to graduate to 1.13 or if we can close out the 1.12 milestone?

This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Thanks!

[DirectXMan12]

hey sorry, I've been out the past couple of weeks, and will likely have limited availability the next couple (GitHub needs out-of-office :-P). I'll try and sync with @MaciekPytel and @mwielgus to make sure we're all set on the sig-autoscaling side.