Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specifying window over which metrics for HPA are collected #57660

Open
discordianfish opened this issue Dec 27, 2017 · 27 comments
Open

Allow specifying window over which metrics for HPA are collected #57660

discordianfish opened this issue Dec 27, 2017 · 27 comments

Comments

@discordianfish
Copy link
Contributor

@discordianfish discordianfish commented Dec 27, 2017

/kind feature

Use case

The docs say "The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization." without specifying the window over which this was observed. Haven't found where it's defined but it seems like it's ~1-5 minutes.

Now imaging a traffic pattern with short dips. If there is a dip in the calculation window, it will cause the HPA to scale down aggressively. Same applies the other way around, if there is a spike spanning the calculation window, it will add lots of replicas even if the spike is gone already.

The only options to tweak is the cooldown period which doesn't help since it will only limit the number of scale operations, no the amount by which to scale. With longer cooldown the time of over/underprovisioning is just longer and shorter cooldown leads to trashing due to the number of scale operations.

Instead it should made possible to specify the observation interval in the HPA spec.

Implementation

From what I can tell, the HPA gets the metrics to calculate the number of replicas to add or remove from the metrics API without having any way to specify the observation window. Looking at the API here: https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1beta1/types.go it seems like the metrics available via the Metrics api are already preprocessed and the window is determined by the reporter (kubelet/cadvisor?).

With this design it seems impossible to get utilization over different periods of time ad-hoc. Was this discussed for the metrics api design? If the NodeMetrics contained the raw CPU seconds, the HPA could use them to calculate the utilization over a user provided window X (by keeping the value from X ago).

It could be argued that such calcuation have no place in the HPA either and it should defer this to a metrics-server, but I don't know how this would work in the 'push model' we have right now where the kubelets report metrics instead of having the HPA/controller pull them when needed while specifying the window.

@discordianfish

This comment has been minimized.

Copy link
Contributor Author

@discordianfish discordianfish commented Dec 27, 2017

/sig instrumentation
@kubernetes/sig-autoscaling

@brancz

This comment has been minimized.

Copy link
Member

@brancz brancz commented Jan 2, 2018

@kubernetes/sig-instrumentation-feature-requests @kubernetes/sig-api-machinery-api-reviews

Note that by design the core/custom metrics APIs are currently intentionally not historical. They only return a single value. Historical metrics APIs are still to be done. Possibly as extensions to the existing APIs or entirely new APIs - there are a lot of open questions.

Calculations/aggregations are a non-goal of the metrics APIs, it's up to the backing monitoring system to perform these. Otherwise it wouldn't be a backing monitoring system but a backing tsdb, and we'd be back to heapster.

More precisely, the custom-metrics-api returns a single scalar, rather than a vector and series of data points. Meaning the "query" should be able to be anything that obliges to that rule.

@brancz

This comment has been minimized.

Copy link
Member

@brancz brancz commented Jan 2, 2018

@discordianfish

This comment has been minimized.

Copy link
Contributor Author

@discordianfish discordianfish commented Jan 2, 2018

Okay, so the types I linked to would be returned by the backing monitoring system based on a "query"? So Window wouldn't be set by the reporter (as I assumed) but by the monitoring system?

So in the case of the HPA, it would require configuration of a query (whether abstracted or specific to the backing system)? That would work I assume.

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Jan 3, 2018

So Window wouldn't be set by the reporter (as I assumed) but by the monitoring system?

Correct -- it's more intended as information that consumers can use to judge the freshness of rate metrics (like CPU).

Historically, we've shyed away from exposing the interval knob directly to user because it was felt that implementation detail, and that we'd like to do things to make it not necessary to set that knob at all (e.g. automatically adjusting the refresh interval, or something). Additionally, for the spike case, we could implement better algorithms that use more than one data point, to calculate a better derivative, use a PID loop, etc. However, I could definitely buy that the above is highly hypothetical, and that we should have a more concrete solution to problems now.

@lavalamp

This comment has been minimized.

Copy link
Member

@lavalamp lavalamp commented Jan 4, 2018

I think this is not api machinery?

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Jan 10, 2018

I think this is not api machinery?

correct, I think the API review request added that tag.

@brancz

This comment has been minimized.

Copy link
Member

@brancz brancz commented Jan 10, 2018

Sorry about that, that was my mistake.

@piosz

This comment has been minimized.

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Apr 11, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@discordianfish

This comment has been minimized.

Copy link
Contributor Author

@discordianfish discordianfish commented Apr 11, 2018

/remove-lifecycle stale
/lifecycle freeze

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Jul 10, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Jul 16, 2018

/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Oct 14, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@brancz

This comment has been minimized.

Copy link
Member

@brancz brancz commented Oct 15, 2018

/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Jan 13, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@discordianfish

This comment has been minimized.

Copy link
Contributor Author

@discordianfish discordianfish commented Jan 17, 2019

It's beyond me how this bot is acceptable to anyone. I'm going to just unsubscribe here.

@brancz

This comment has been minimized.

Copy link
Member

@brancz brancz commented Jan 18, 2019

/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Apr 18, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@brancz

This comment has been minimized.

Copy link
Member

@brancz brancz commented Apr 18, 2019

/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Jul 17, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@Bessonov

This comment has been minimized.

Copy link

@Bessonov Bessonov commented Jul 17, 2019

Activity!

@Bessonov

This comment has been minimized.

Copy link

@Bessonov Bessonov commented Jul 17, 2019

/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Oct 15, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@Bessonov

This comment has been minimized.

Copy link

@Bessonov Bessonov commented Oct 15, 2019

/remove-lifecycle stale

@matthyx

This comment has been minimized.

Copy link
Contributor

@matthyx matthyx commented Jan 5, 2020

I just came after reading an article linking to here...
@brancz if you really want this to advance you should open a KEP and find a sponsoring SIG.
This is the standard way to propose API changes.

@brancz

This comment has been minimized.

Copy link
Member

@brancz brancz commented Jan 6, 2020

I’m fully aware how this project works and am actively thinking about this also in larger scope than hpa but potentially iterating on the metrics APIs we defined as sig instrumentation.

I’m just also involved in 3 other large KEPs so this is more of a backlog type stage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.