Etcd of shoot cluster when initializing is taking up more resources than actual resource load

[chungtd203338]

When I initialized the shoot in HA mode, there were 6 ETCD pods including 3 etcd-main pods and 3 etcd-events pods. With the default request resource level of 300m CPU and 1G RAM, it would take up to 1.8 CPU and 6G RAM of the seed system, a number too large for initializing many shoot clusters. In fact, when I just finished initializing, I measured that etcd-main only consumed about 150m CPU / 200Mi RAM and etcd-events consumed about 100m CPU / 100Mi RAM (kubectl top pod).

I know that the ETCD component has autoscale using VPA, but the cluster must operate for a long time after that, etcd-main and etcd-events will scale down the request if there is really no load. However, the initial initialization level is quite expensive, not counting many components in the control plane of the shoot cluster.

Is it possible to reduce the initial request of ETCD in the shoot cluster?

[shreyas-s-rao]

Well, we usually set the initial resources requests to a "safe" initial value to avoid restarts of the etcd when a shoot cluster starts off with heavier loads, which we normally observe. This is because initialization workloads on a shoot cluster may be resource-intensive and sensitive to control plane downtimes/latencies, and we want to avoid VPA-triggered restarts of the etcd pods during this time.

We do have a minAllowed configuration in the shoot API (as detailed here) that allows setting minAllowed for shoots with even heavier workloads to begin with, but I don't think this would be helpful in your case.

@timuthy what would be the best path forward here? I do understand that the etcd may never need 1Gi memory at all, if the shoot cluster is very small for the entire duration of its lifecycle, so setting it to 1Gi to begin with may not be optimal.

[chungtd203338]

Well, we usually set the initial resources requests to a "safe" initial value to avoid restarts of the etcd when a shoot cluster starts off with heavier loads, which we normally observe. This is because initialization workloads on a shoot cluster may be resource-intensive and sensitive to control plane downtimes/latencies, and we want to avoid VPA-triggered restarts of the etcd pods during this time.

We do have a minAllowed configuration in the shoot API (as detailed here) that allows setting minAllowed for shoots with even heavier workloads to begin with, but I don't think this would be helpful in your case.

@timuthy what would be the best path forward here? I do understand that the etcd may never need 1Gi memory at all, if the shoot cluster is very small for the entire duration of its lifecycle, so setting it to 1Gi to begin with may not be optimal.

Yeah, i'm looking to sizing my seed cluster. when i do resource calculations for the controlplane component of a shoot cluster i see the request for a shoot cluster (HA enabled) is quite large up to 8-9G RAM however the actual load is only about 2-3G RAM and i notice that the ETCD component (6 Pods) is taking up more system requests than its actual capacity. if i reduce the request of ectd i can provision more shoots than my seed cluster can provide. If the cluster shoots a 2vCPU/4GRAM worker and it takes up 8-9G RAM for controlplane, it's really not very optimal 😅

[rfranzke]

Can't we generally shrink the default requests now that we allow configuring minAllowed? If people want to avoid early restarts, they can configure minAllowed. Otherwise, if etcd load becomes heavier, it auto-scales.

[chungtd203338]

I think it is better to prioritize minAllowed (configuration in the shoot API) configuration for initial etcd request instead of being configured if minAllowed is greater than 300m/1G so that users can proactively configure according to their cluster size.

[chungtd203338]

@rfranzke @timuthy I have created a pull request to prioritize the configuration request for etcd and the apiserver of the main shoot cluster is minallowed in the shoot api so that users can proactively configure. Can you review it?

[chungtd203338]

Is anyone here?

[rfranzke]

@chungtd203338 I have invited some folks for review (#11911 (comment)), please be patient. Many things are going on in parallel and we cannot just abandon the sailing ship :) Give it a couple of more days.