New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arbitrary/Custom Metrics in the Horizontal Pod Autoscaler #117

Open
DirectXMan12 opened this Issue Oct 7, 2016 · 54 comments

Comments

Projects
None yet
@DirectXMan12
Contributor

DirectXMan12 commented Oct 7, 2016

Arbitrary/Custom Metrics in the Horizontal Pod Autoscaler

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Oct 7, 2016

cc @kubernetes/autoscaling @jszczepkowski @derekwaynecarr @smarterclayton

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Nov 15, 2016

@DirectXMan12 any updates on this issue? Can you provide the actual status of it and update the checkboxes above?

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Nov 16, 2016

The proposal is posted, but has not been approved yet. We only recently reached general consensus about the design. Still finalizing the exact semantics. It should be removed from the 1.5 milestone, since no code will have gone into 1.5

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Nov 16, 2016

@DirectXMan12 thank you for clarifying.

@idvoretskyi idvoretskyi modified the milestones: next-milestone, v1.5 Nov 16, 2016

@idvoretskyi idvoretskyi modified the milestones: v1.6, next-milestone Jan 30, 2017

k8s-merge-robot added a commit to kubernetes/kubernetes that referenced this issue Feb 10, 2017

Merge pull request #36033 from DirectXMan12/feature/hpa-v2
Automatic merge from submit-queue (batch tested with PRs 40796, 40878, 36033, 40838, 41210)

HPA v2 (API Changes)

**Release note**:
```release-note
Introduces an new alpha version of the Horizontal Pod Autoscaler including expanded support for specifying metrics.
```

Implements the API changes for kubernetes/enhancements#117.

This implements #34754, which is the new design for the Horizontal Pod Autoscaler.  It includes improved support for custom metrics (and/or arbitrary metrics) as well as expanded support for resource metrics.  The new HPA object is introduces in the API group "autoscaling/v1alpha1".

Note that the improved custom metric support currently is limited to per pod metrics from Heapster -- attempting to use the new "object metrics" will simply result in an error.  This will change once #34586 is merged and implemented.

k8s-merge-robot added a commit to kubernetes/kubernetes that referenced this issue Feb 20, 2017

Merge pull request #41272 from DirectXMan12/feature/hpa-v2-controller
Automatic merge from submit-queue

Convert HPA controller to support HPA v2 mechanics

This PR converts the HPA controller to support the mechanics from HPA v2.
The HPA controller continues to make use of the HPA v1 client, but utilizes
the conversion logic to work with autoscaling/v2alpha1 objects internally.

It is the follow-up PR to #36033 and part of kubernetes/enhancements#117.

**Release note**:
```release-note
NONE
```
@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Mar 6, 2017

@DirectXMan12 please, provide us with the release notes and documentation PR (or links) at https://docs.google.com/spreadsheets/d/1nspIeRVNjAQHRslHQD1-6gPv99OcYZLMezrBe3Pfhhg/edit#gid=0

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Mar 6, 2017

@DirectXMan12 please, provide us with the release notes and documentation PR (or links)

done :-)

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Jul 20, 2017

@DirectXMan12 @mwielgus What is planned for HPA in 1.8?

@fgrzadkowski

This comment has been minimized.

@evmin

This comment has been minimized.

evmin commented Jul 21, 2017

Looking forward to the beta launch!

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Jul 21, 2017

@bgrant0607 we're hoping to move to v2 to beta in 1.8 (so just stabilization :-) ).

@davidopp

This comment has been minimized.

Member

davidopp commented Jul 21, 2017

Regarding the functionality - am I correct in understanding that if you wanted to scale on a load indicator that is fundamentally external to Kubernetes, you would need to

  1. create some kind of proxy API object inside the cluster (maybe a CRD?) that reflects the load indicator in a manner that the Kubernetes metrics pipeline can consume
  2. create an HPA with MetricSourceType == "Object" and point to the proxy object
@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Jul 23, 2017

@davidopp That is incorrect. In order to scale on a load indicator that's not one of the metrics provided by the resource metrics API (CPU, memory), you need to have some implementation of the custom metrics API (see k8s.io/metrics and kubernetes-incubator/custom-metrics-apiserver).

Then, you can either use the "pods" source type, if the metric describes the pods controlled the the target scalable of the HPA (e.g. network throughput), or the "object" source type, if the metric describes an unrelated object (for instance, you might scale on a queue length metric attached to the namespace).

In either case, the HPA controller will the query the custom metrics API accordingly. It is up to cluster admins, etc to actually provide a method to collect the given metrics and expose an implementation of the custom metrics API.

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented May 7, 2018

Awesome, looking forward to this! However, any plans for stable?

eventually. Realistically, I want to give a release or two for feedback on the new stuff, so that we can be sure we're happy with the shape of the API, then we'll look towards stable.

@mistyhacks

This comment has been minimized.

mistyhacks commented May 24, 2018

@DirectXMan12 please fill out the appropriate line item of the
1.11 feature tracking spreadsheet
and open a placeholder docs PR against the
release-1.11 branch
by 5/25/2018 (tomorrow as I write this) if new docs or docs changes are
needed and a relevant PR has not yet been opened.

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented May 25, 2018

done.

@mistyhacks

This comment has been minimized.

mistyhacks commented May 25, 2018

Thank you 🙌

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Jun 6, 2018

@kubernetes/sig-release-misc due to some last-minute hiccups, the corresponding PR for this feature's updates did not get merged for 1.11, so there's no new developments for this feature in 1.11.

CC @kubernetes/sig-autoscaling-misc

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Jun 6, 2018

@justaugustus justaugustus removed this from the v1.11 milestone Jun 6, 2018

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jun 6, 2018

Thanks for the update on this, @DirectXMan12!

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jul 17, 2018

@DirectXMan12 @kubernetes/sig-autoscaling-feature-requests --
This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jul 30, 2018

This feature current has no milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Set the following:

  • Description
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

Once this feature is appropriately updated, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.


Please note that Features Freeze is tomorrow, July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

P.S. This was sent via automation

@MaciekPytel

This comment has been minimized.

MaciekPytel commented Jul 31, 2018

@DirectXMan12 Are the changes to API (adding metricselector to pods/object metrics, etc) still in scope for 1.12? If so this should probably be added to milestone.

@mwielgus Anything else that should be added here for 1.12?

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Jul 31, 2018

It should be in the milestone. I believe the description is up to date. If it's not, please let me know.

/milestone 1.12

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jul 31, 2018

Thanks for the updates! I've added this to the tracking sheet for 1.12.

@justaugustus justaugustus added this to the v1.12 milestone Jul 31, 2018

@zparnold

This comment has been minimized.

Member

zparnold commented Aug 20, 2018

Hey there! @DirectXMan12 I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

@tomkerkhove

This comment has been minimized.

tomkerkhove commented Aug 21, 2018

Would be great if we can include this one as well - kubernetes/kubernetes#66988

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Aug 21, 2018

@zparnold already did that quite some time ago: kubernetes/website#8757

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Aug 21, 2018

just need to rebase it ;-)

@zparnold

This comment has been minimized.

Member

zparnold commented Aug 25, 2018

@claurence

This comment has been minimized.

claurence commented Oct 5, 2018

Kubernetes 1.13 is going to be a 'stable' release since the cycle is only 10 weeks. We encourage no big alpha features and only consider adding this feature if you have a high level of confidence it will make code slush by 11/09. Are there plans for this enhancement to graduate to alpha/beta/stable within the 1.13 release cycle? If not, can you please remove it from the 1.12 milestone or add it to 1.13?

We are also now encouraging that every new enhancement aligns with a KEP. If a KEP has been created, please link to it in the original post. Please take the opportunity to develop a KEP

@kacole2

This comment has been minimized.

Contributor

kacole2 commented Oct 8, 2018

@DirectXMan12 I'm following up on @claurence's post to see if there are any plans for this to graduate to 1.13 or if we can close out the 1.12 milestone?

This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

  • Docs (open placeholder PRs): 11/8
  • Code Slush: 11/9
  • Code Freeze Begins: 11/15
  • Docs Complete and Reviewed: 11/27

Thanks!

@kacole2 kacole2 added tracked/no and removed tracked/yes labels Oct 8, 2018

@mwielgus mwielgus removed this from the v1.12 milestone Oct 11, 2018

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Oct 25, 2018

hey sorry, I've been out the past couple of weeks, and will likely have limited availability the next couple (GitHub needs out-of-office :-P). I'll try and sync with @MaciekPytel and @mwielgus to make sure we're all set on the sig-autoscaling side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment