Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce a new autoscaling mode (VPAAndHPA) for Shoot Kubernetes API servers #9678

Merged
merged 10 commits into from
May 15, 2024
Merged

Introduce a new autoscaling mode (VPAAndHPA) for Shoot Kubernetes API servers #9678

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
041b5b8
Add VPAAndHPAForAPIServer feature gate
ialidzhikov Apr 25, 2024
f4d6c59
Introduce consts for API server autoscaling modes
ialidzhikov Apr 25, 2024
946789d
Add VPAAndHPA autoscaling mode
ialidzhikov Apr 25, 2024
e3e96bb
Add docs for kube-apiserver autoscaling
ialidzhikov Apr 26, 2024
2caae51
Address review comments from voelzmo
ialidzhikov May 8, 2024
f205c8f
Address review comments from rfranzke
ialidzhikov May 8, 2024
d6c7852
Address review comments from voelzmo (2)
ialidzhikov May 9, 2024
756e37e
Minor nits
ialidzhikov May 10, 2024
1433c5f
Address review comments from voelzmo (3)
ialidzhikov May 14, 2024
5da9401
Address review comments from plkokanov
ialidzhikov May 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

* Components
* [Gardener API server](concepts/apiserver.md)
* [In-Tree admission plugins](concepts/apiserver_admission_plugins.md)
* [In-Tree admission plugins](concepts/apiserver-admission-plugins.md)
* [Gardener Controller Manager](concepts/controller-manager.md)
* [Gardener Scheduler](concepts/scheduler.md)
* [Gardener Admission Controller](concepts/admission-controller.md)
Expand Down
0 docs/concepts/apiserver_admission_plugins.md → docs/concepts/apiserver-admission-plugins.md
View file
Open in desktop
File renamed without changes.
8 changes: 0 additions & 8 deletions docs/concepts/etcd.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,6 @@ When a [`gardenlet`](gardenlet.md) reconciles a `Shoot` resource or a [`gardener
`etcd-druid` needs to manage the lifecycle of the desired etcd instance (today `main` or `events`).
Likewise, when the `Shoot` or `Garden` is deleted, `gardenlet` or `gardener-operator` deletes the `Etcd` resources and [etcd Druid](https://github.com/gardener/etcd-druid/) takes care of cleaning up all related objects, e.g. the backing `StatefulSet`s.

## Autoscaling

Gardenlet maintains [`HVPA`](https://github.com/gardener/hvpa-controller/blob/master/config/samples/autoscaling_v1alpha1_hvpa.yaml) objects for etcd `StatefulSet`s if the corresponding [feature gate](../deployment/feature_gates.md) is enabled.
This enables a vertical scaling for etcd.
Downscaling is handled more pessimistically to prevent many subsequent etcd restarts.
Thus, for `production` and `infrastructure` shoot clusters (or all garden clusters), downscaling is deactivated for the main etcd.
For all other shoot clusters, lower advertised requests/limits are only applied during a shoot's maintenance time window.

## Backup

If `Seed`s specify backups for etcd ([example](../../example/50-seed.yaml)), then Gardener and the respective [provider extensions](../extensions/overview.md) are responsible for creating a bucket on the cloud provider's side (modelled through a [BackupBucket resource](../extensions/backupbucket.md)).
Expand Down
26 changes: 14 additions & 12 deletions docs/deployment/feature_gates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ The following tables are a summary of the feature gates that you can set on diff
| UseNamespacedCloudProfile | `false` | `Alpha` | `1.92` | |
| ShootManagedIssuer | `false` | `Alpha` | `1.93` | |
| VPAForETCD | `false` | `Alpha` | `1.94` | |
| VPAAndHPAForAPIServer | `false` | `Alpha` | `1.95` | |

## Feature Gates for Graduated or Deprecated Features

Expand Down Expand Up @@ -190,15 +191,16 @@ A *General Availability* (GA) feature is also referred to as a *stable* feature.

## List of Feature Gates

| Feature | Relevant Components | Description |
|------------------------------------|-----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| HVPA | `gardenlet`, `gardener-operator` | Enables simultaneous horizontal and vertical scaling in garden or seed clusters. |
| HVPAForShootedSeed | `gardenlet` | Enables simultaneous horizontal and vertical scaling in managed seed (aka "shooted seed") clusters. |
| DefaultSeccompProfile | `gardenlet`, `gardener-operator` | Enables the defaulting of the seccomp profile for Gardener managed workload in the garden or seed to `RuntimeDefault`. |
| CoreDNSQueryRewriting | `gardenlet` | Enables automatic DNS query rewriting in shoot cluster's CoreDNS to shortcut name resolution of fully qualified in-cluster and out-of-cluster names, which follow a user-defined pattern. Details can be found in [DNS Search Path Optimization](../usage/dns-search-path-optimization.md). |
| IPv6SingleStack | `gardener-apiserver`, `gardenlet` | Allows creating seed and shoot clusters with [IPv6 single-stack networking](../usage/ipv6.md) enabled in their spec ([GEP-21](../proposals/21-ipv6-singlestack-local.md)). If enabled in gardenlet, the default behavior is unchanged, but setting `ipFamilies=[IPv6]` in the `seedConfig` is allowed. Only if the `ipFamilies` setting is changed, gardenlet behaves differently. |
| MutableShootSpecNetworkingNodes | `gardener-apiserver` | Allows updating the field `spec.networking.nodes`. The validity of the values has to be checked in the provider extensions. Only enable this feature gate when your system runs provider extensions which have implemented the validation. |
| ShootForceDeletion | `gardener-apiserver` | Allows forceful deletion of Shoots by annotating them with the `confirmation.gardener.cloud/force-deletion` annotation. |
| UseNamespacedCloudProfile | `gardener-apiserver` | Enables usage of `NamespacedCloudProfile`s in `Shoot`s. |
| ShootManagedIssuer | `gardenlet` | Enables the shoot managed issuer functionality described in GEP 24. |
| VPAForETCD | `gardenlet`, `gardener-operator` | Enables VPA for `etcd-main` and `etcd-events`, regardless of HVPA enablement. |
| Feature | Relevant Components | Description |
|---------------------------------|-----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| HVPA | `gardenlet`, `gardener-operator` | Enables simultaneous horizontal and vertical scaling in garden or seed clusters. |
| HVPAForShootedSeed | `gardenlet` | Enables simultaneous horizontal and vertical scaling in managed seed (aka "shooted seed") clusters. |
| DefaultSeccompProfile | `gardenlet`, `gardener-operator` | Enables the defaulting of the seccomp profile for Gardener managed workload in the garden or seed to `RuntimeDefault`. |
| CoreDNSQueryRewriting | `gardenlet` | Enables automatic DNS query rewriting in shoot cluster's CoreDNS to shortcut name resolution of fully qualified in-cluster and out-of-cluster names, which follow a user-defined pattern. Details can be found in [DNS Search Path Optimization](../usage/dns-search-path-optimization.md). |
| IPv6SingleStack | `gardener-apiserver`, `gardenlet` | Allows creating seed and shoot clusters with [IPv6 single-stack networking](../usage/ipv6.md) enabled in their spec ([GEP-21](../proposals/21-ipv6-singlestack-local.md)). If enabled in gardenlet, the default behavior is unchanged, but setting `ipFamilies=[IPv6]` in the `seedConfig` is allowed. Only if the `ipFamilies` setting is changed, gardenlet behaves differently. |
| MutableShootSpecNetworkingNodes | `gardener-apiserver` | Allows updating the field `spec.networking.nodes`. The validity of the values has to be checked in the provider extensions. Only enable this feature gate when your system runs provider extensions which have implemented the validation. |
| ShootForceDeletion | `gardener-apiserver` | Allows forceful deletion of Shoots by annotating them with the `confirmation.gardener.cloud/force-deletion` annotation. |
| UseNamespacedCloudProfile | `gardener-apiserver` | Enables usage of `NamespacedCloudProfile`s in `Shoot`s. |
| ShootManagedIssuer | `gardenlet` | Enables the shoot managed issuer functionality described in GEP 24. |
| VPAForETCD | `gardenlet`, `gardener-operator` | Enables VPA for `etcd-main` and `etcd-events`, regardless of HVPA enablement. |
| VPAAndHPAForAPIServer | `gardenlet` | Enables an autoscaling mechanism for shoot kube-apiserver where it is scaled simultaneously by VPA and HPA on the same metric (CPU and memory usage). The pod-trashing cycle between VPA and HPA scaling on the same metric is avoided by configuring the HPA to scale on average usage (not on average utilization) and by picking the target average utilization values in sync with VPA's allowed maximums. The feature gate takes precedence over the `HVPA` feature gate when they are both enabled. |
71 changes: 71 additions & 0 deletions docs/development/autoscaling-specifics-for-components.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
title: Autoscaling Specifics for Components
---

## Overview

This document describes the used autoscaling mechanism for several components.

## Garden or Shoot Cluster etcd

By default, if none of the autoscaling modes is requested the `etcd` is deployed with static resources, without autoscaling.

However, there are two supported autoscaling modes for the Garden or Shoot cluster etcd.

- `HVPA`

In `HVPA` mode, the etcd is scaled by the [hvpa-controller](https://github.com/gardener/hvpa-controller). The gardenlet/gardener-operator is creating an `HVPA` resource for the etcd (`main` or `events`).
The `HVPA` enables a vertical scaling for etcd.

The `HVPA` mode is the used autoscaling mode when the `HVPA` feature gate is enabled (and the `VPAForETCD` feature gate is disabled).

- `VPA`

In `VPA` mode, the etcd is scaled by a native `VPA` resource.

The `VPA` mode is the used autoscaling mode when the `VPAForETCD` feature gate is enabled (takes precedence over the `HVPA` feature gate).

For both of the autoscaling modes downscaling is handled more pessimistically to prevent many subsequent etcd restarts. Thus, for `production` and `infrastructure` Shoot clusters (or all Garden clusters), downscaling is deactivated for the main etcd. For all other Shoot clusters, lower advertised requests/limits are only applied during the Shoot's maintenance time window.
ialidzhikov marked this conversation as resolved.
Show resolved Hide resolved

## Shoot Kubernetes API Server

There are three supported autoscaling modes for the Shoot Kubernetes API server.

- `Baseline`

In `Baseline` mode, the Shoot Kubernetes API server is scaled by active HPA and VPA in passive, recommend-only mode.

The API server resource requests are computed based on the Shoot's minimum Nodes count:
| Range | Resource Requests |
|-------------|-------------------|
| [0, 2] | `800m`, `800Mi` |
| (2, 10] | `1000m`, `1100Mi` |
| (10, 50] | `1200m`, `1600Mi` |
| (50, 100] | `2500m`, `5200Mi` |
| (100, inf.) | `3000m`, `5200Mi` |

The `Baseline` mode is the used autoscaling mode when the `HVPA` and `VPAAndHPAForAPIServer` feature gates are not enabled.

- `HVPA`

In `HVPA` mode, the Shoot Kubernetes API server is scaled by the [hvpa-controller](https://github.com/gardener/hvpa-controller). The gardenlet is creating an `HVPA` resource for the API server. The `HVPA` resource is backed by HPA and VPA both in recommend-only mode. The hvpa-controller is responsible for enabling simultaneous horizontal and vertical scaling by incorporating the recommendations from the HPA and VPA.

The initial API server resource requests are `500m` and `1Gi`.
HVPA's HPA is scaling only on CPU (average utilization 80%). HVPA's VPA max allowed values are `8` CPU and `25G`.

The `HVPA` mode is the used autoscaling mode when the `HVPA` feature gate is enabled (and the `VPAAndHPAForAPIServer` feature gate is disabled).

- `VPAAndHPA`

In `VPAAndHPA` mode, the Shoot Kubernetes API server is scaled simultaneously by VPA and HPA on the same metric (CPU and memory usage). The pod-trashing cycle between VPA and HPA scaling on the same metric is avoided by configuring the HPA to scale on average usage (not on average utilization) and by picking the target average utilization values in sync with VPA's allowed maximums. This makes possible VPA to first scale vertically on CPU/memory usage. Once all Pods' average CPU/memory usage is close to exceed the VPA's allowed maximum CPU/memory (the HPA's target average utilization, 1/7 less than VPA's allowed maximums), HPA is scaling horizontally (by adding a new replica).

The `VPAAndHPA` mode is introduced to address disadvantages with HVPA: additional component; modifies the deployment triggering unnecessary rollouts; vertical scaling only at max replicas; stuck vertical resource requests when scaling in again; etc.

The initial API server resource requests are `250m` and `500Mi`.
VPA's max allowed values are `7` CPU and `28G`. HPA's average target usage values are `6` CPU and `24G`.

The `VPAAndHPA` mode is the used autoscaling mode when the `VPAAndHPAForAPIServer` feature gate is enabled (takes precedence over the `HVPA` feature gate).

The API server's replica count in all scaling modes varies between 2 and 3. The min replicas count of 2 is imposed by the [High Availability of Shoot Control Plane Components](../development/high-availability.md#control-plane-components).

The gardenlet sets the initial API server resource requests only when the Deployment is not found. When the Deployment exists, it is not overwriting the kube-apiserver container resources.