Skip to content

[enterprise-4.15] OSDOCS7357 Docs for OCPSTRAT-802 Make VPA usable for large cluster #70784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mburke5678 merged 1 commit into from
Jan 24, 2024
Merged

[enterprise-4.15] OSDOCS7357 Docs for OCPSTRAT-802 Make VPA usable for large cluster #70784

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion modules/nodes-pods-vertical-autoscaler-about.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,18 @@
[id="nodes-pods-vertical-autoscaler-about_{context}"]
= About the Vertical Pod Autoscaler Operator

The Vertical Pod Autoscaler Operator (VPA) is implemented as an API resource and a custom resource (CR). The CR determines the actions the Vertical Pod Autoscaler Operator should take with the pods associated with a specific workload object, such as a daemon set, replication controller, and so forth, in a project.
The Vertical Pod Autoscaler Operator (VPA) is implemented as an API resource and a custom resource (CR). The CR determines the actions that the VPA Operator should take with the pods associated with a specific workload object, such as a daemon set, replication controller, and so forth, in a project.

The VPA Operator consists of three components, each of which has its own pod in the VPA namespace:

Recommender::
The VPA recommender monitors the current and past resource consumption and, based on this data, determines the optimal CPU and memory resources for the pods in the associated workload object.

Updater::
The VPA updater checks if the pods in the associated workload object have the correct resources. If the resources are correct, the updater takes no action. If the resources are not correct, the updater kills the pod so that they can be recreated by their controllers with the updated requests.

Admission controller::
The VPA admission controller sets the correct resource requests on each new pod in the associated workload object, whether the pod is new or was recreated by its controller due to the VPA updater actions.

You can use the default recommender or use your own alternative recommender to autoscale based on your own algorithms.

Expand Down
162 changes: 162 additions & 0 deletions modules/nodes-pods-vertical-autoscaler-tuning.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
// Module included in the following assemblies:
//
// * nodes/nodes-vertical-autoscaler.adoc

:_mod-docs-content-type: CONCEPT
[id="nodes-pods-vertical-autoscaler-tuning_{context}"]
= Performance tuning the VPA Operator

As a cluster administrator, you can tune the performance of your Vertical Pod Autoscaler Operator (VPA) to limit the rate at which the VPA makes requests of the Kubernetes API server and to specify the CPU and memory resources for the VPA recommender, updater, and admission controller component pods.

Additionally, you can configure the VPA Operator to monitor only those workloads that are being managed by a VPA custom resource (CR). By default, the VPA Operator monitors every workload in the cluster. This allows the VPA Operator to accrue and store 8 days of historical data for all workloads, which the Operator can use if a new VPA CR is created for a workload. However, this causes the VPA Operator to use significant CPU and memory, which could cause the Operator to fail, particularly on larger clusters. By configuring the VPA Operator to monitor only workloads with a VPA CR, you can save on CPU and memory resources. One trade-off is that if you have a workload that has been running, and you create a VPA CR to manage that workload, the VPA Operator does not have any historical data for that workload. As a result, the initial recommendations are not as useful as those after the workload had been running for some time.

These tunings allow you to ensure the VPA has sufficient resources to operate at peak efficiency and to prevent throttling and a possible delay in pod admissions.

You can perform the following tunings on the VPA components by editing the `VerticalPodAutoscalerController` custom resource (CR):

* To prevent throttling and pod admission delays, set the queries-per-second (QPS) and burst rates for VPA requests of the Kubernetes API server by using the `kube-api-qps` and `kube-api-burst` parameters.

* To ensure sufficient CPU and memory, set the CPU and memory requests for VPA component pods by using the standard `cpu` and `memory` resource requests.

* To configure the VPA Operator to monitor only workloads that are being managed by a VPA CR, set the `memory-saver` parameter to `true` for the recommender component.

The following example VPA controller CR sets the VPA API QPS and burts rates, configures the component pod resource requests, and sets `memory-saver` to `true` for the recommender:

.Example `VerticalPodAutoscalerController` CR
[source,yaml]
----
apiVersion: autoscaling.openshift.io/v1
kind: VerticalPodAutoscalerController
metadata:
name: default
namespace: openshift-vertical-pod-autoscaler
spec:
deploymentOverrides:
admission: <1>
container:
args: <2>
- '--kube-api-qps=30.0'
- '--kube-api-burst=40.0'
resources:
requests: <3>
cpu: 40m
memory: 40Mi
recommender: <4>
container:
args:
- '--kube-api-qps=20.0'
- '--kube-api-burst=60.0'
- '--memory-saver=true' <5>
resources:
requests:
cpu: 60m
memory: 60Mi
updater: <6>
container:
args:
- '--kube-api-qps=20.0'
- '--kube-api-burst=80.0'
resources:
requests:
cpu: 80m
memory: 80Mi
minReplicas: 2
podMinCPUMillicores: 25
podMinMemoryMb: 250
recommendationOnly: false
safetyMarginFraction: 0.15
----
<1> Specifies the tuning parameters for the VPA admission controller.
<2> Specifies the API QPS and burst rates for the VPA admission controller.
+
--
* `kube-api-qps`: Specifies the queries per second (QPS) limit when making requests to Kubernetes API server. The default is `5.0`.
* `kube-api-burst`: Specifies the burst limit when making requests to Kubernetes API server. The default is `10.0`.
--
<3> Specifies the CPU and memory requests for the VPA admission controller pod.
<4> Specifies the tuning parameters for the VPA recommender.
<5> Specifies that the VPA Operator monitors only workloads with a VPA CR. The default is `false`.
<6> Specifies the tuning parameters for the VPA updater.

You can verify that the settings were applied to each VPA component pod.

.Example updater pod
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
name: vpa-updater-default-d65ffb9dc-hgw44
namespace: openshift-vertical-pod-autoscaler
# ...
spec:
containers:
- args:
- --logtostderr
- --v=1
- --min-replicas=2
- --kube-api-qps=20.0
- --kube-api-burst=80.0
# ...
resources:
requests:
cpu: 80m
memory: 80Mi
# ...
----

.Example admission controller pod
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
name: vpa-admission-plugin-default-756999448c-l7tsd
namespace: openshift-vertical-pod-autoscaler
# ...
spec:
containers:
- args:
- --logtostderr
- --v=1
- --tls-cert-file=/data/tls-certs/tls.crt
- --tls-private-key=/data/tls-certs/tls.key
- --client-ca-file=/data/tls-ca-certs/service-ca.crt
- --webhook-timeout-seconds=10
- --kube-api-qps=30.0
- --kube-api-burst=40.0
# ...
resources:
requests:
cpu: 40m
memory: 40Mi
# ...
----

.Example recommender pod
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
name: vpa-recommender-default-74c979dbbc-znrd2
namespace: openshift-vertical-pod-autoscaler
# ...
spec:
containers:
- args:
- --logtostderr
- --v=1
- --recommendation-margin-fraction=0.15
- --pod-recommendation-min-cpu-millicores=25
- --pod-recommendation-min-memory-mb=250
- --kube-api-qps=20.0
- --kube-api-burst=60.0
- --memory-saver=true
# ...
resources:
requests:
cpu: 60m
memory: 60Mi
# ...
----
2 changes: 2 additions & 0 deletions nodes/pods/nodes-pods-vertical-autoscaler.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ include::modules/nodes-pods-vertical-autoscaler-install.adoc[leveloffset=+1]

include::modules/nodes-pods-vertical-autoscaler-using-about.adoc[leveloffset=+1]

include::modules/nodes-pods-vertical-autoscaler-tuning.adoc[leveloffset=+2]

include::modules/nodes-pods-vertical-autoscaler-custom.adoc[leveloffset=+2]

include::modules/nodes-pods-vertical-autoscaler-configuring.adoc[leveloffset=+1]
Expand Down