Ignore starting pod cpu usage in horizontal autoscaler

[JorritSalverda]

We're running java applications that use up to 2 vcpu's during start and after that never go above 2% of 1 vcpu. For this I've lowered the cpu request and limit around that 2%. This is fine as hard-capping doesn't seem to affect the large spike in cpu usage at the start, so it still starts up fast.

However when this happens the autoscaler kicks in and in no time I have 10 instead of 3 pods, all spiking in cpu usage and eating up all memory for no real good reason.

Is there a way - just like with the liveness probe - to delay a pod being evaluated for hpa?

[dchen1107]

cc/ @fgrzadkowski

[wojtek-t]

@jszczepkowski @piosz @mwielgus

[jszczepkowski]

This should be one of cases of #29799: when pod is not READY or was created less then a minute ago, w do not autoscale and do not create error event.

[JorritSalverda]

Isn't the cpu prior to the pod being ready taken into account in that first minute when it's ready? We see a steep scale up as soon as the first extra pod is added by the HPA. Our cpu usage before readiness is about 20x as high as during normal operation - our limit is more than 20x as high as request - , so if it's in any form taken into account you can see why it leads to scale up where it isn't really needed.

[JorritSalverda]

This works reasonably well today. At time of creating this ticket I wasn't aware it is cpu request value that's used for hpa, I thought it was cpu limit. For our applications with cpu intensive startup we still see slightly more autoscaling than we would like, but not too big of an issue.