Skip to content
Permalink
master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

autoscaler/vertical-pod-autoscaler/FAQ.md

Go to file
 
 
Cannot retrieve contributors at this time
Vertical Pod Autoscaler FAQ Contents VPA restarts my pods but does not modify CPU or memory settings How can I use Prometheus as a history provider for the VPA recommender I get recommendations for my single pod replicaSet but they are not applied What are the parameters to VPA recommender? What are the parameters to VPA updater?
207 lines (166 sloc) 10.3 KB
Raw Blame
Open in GitHub Desktop

Vertical Pod Autoscaler FAQ

Contents

VPA restarts my pods but does not modify CPU or memory settings

First check that the VPA admission controller is running correctly:

$ kubectl get pod -n kube-system | grep vpa-admission-controller

vpa-admission-controller-69645795dc-sm88s 1/1 Running 0 1m

Check the logs of the admission controller:

$ kubectl logs -n kube-system vpa-admission-controller-69645795dc-sm88s

If the admission controller is up and running, but there is no indication of it actually processing created pods or VPA objects in the logs, the webhook is not registered correctly.

Check the output of:

$ kubectl describe mutatingWebhookConfiguration vpa-webhook-config

This should be correctly configured to point to the VPA admission webhook service. Example:

Name:         vpa-webhook-config
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  admissionregistration.k8s.io/v1beta1
Kind:         MutatingWebhookConfiguration
Metadata:
  Creation Timestamp:  2019-01-18T15:44:42Z
  Generation:          1
  Resource Version:    1250
  Self Link:           /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/vpa-webhook-config
  UID:                 f8ccd13d-1b37-11e9-8906-42010a84002f
Webhooks:
  Client Config:
    Ca Bundle: <redacted>
    Service:
      Name:        vpa-webhook
      Namespace:   kube-system
  Failure Policy:  Ignore
  Name:            vpa.k8s.io
  Namespace Selector:
  Rules:
    API Groups:

    API Versions:
      v1
    Operations:
      CREATE
    Resources:
      pods
    API Groups:
      autoscaling.k8s.io
    API Versions:
      v1beta1
    Operations:
      CREATE
      UPDATE
    Resources:
      verticalpodautoscalers

If the webhook config doesn't exist, something got wrong with webhook registration for admission controller. Check the logs for more info.

From the above config following part defines the webhook service:

Service:
  Name:        vpa-webhook
  Namespace:   kube-system

Check that the service actually exists:

$ kubectl describe -n kube-system service vpa-webhook

Name:              vpa-webhook
Namespace:         kube-system
Labels:            <none>
Annotations:       <none>
Selector:          app=vpa-admission-controller
Type:              ClusterIP
IP:                <some_ip>
Port:              <unset>  443/TCP
TargetPort:        8000/TCP
Endpoints:         <some_endpoint>
Session Affinity:  None
Events:            <none>

You can also curl the service's endpoint from within the cluster to make sure it is serving.

Note: the commands will differ if you deploy VPA in a different namespace.

How can I use Prometheus as a history provider for the VPA recommender

Configure your Prometheus to get metrics from cadvisor. Make sure that the metrics from the cadvisor have the label job=kubernetes-cadvisor

Set the flags --storage=prometheus and --prometheus-address=<your-prometheus-address> in the deployment for the VPA recommender. The args for the container should look something like this:

spec:
  containers:
  - args:
    - --v=4
    - --storage=prometheus
    - --prometheus-address=http://prometheus.default.svc.cluster.local:9090

In this example, Prometheus is running in the default namespace.

Now deploy the VPA recommender and check the logs.

$ kubectl logs -n kube-system vpa-recommender-bb655b4b9-wk5x2

Here you should see the flags that you set for the VPA recommender and you should see: Initializing VPA from history provider

This means that the VPA recommender is now using Prometheus as the history provider.

I get recommendations for my single pod replicaSet but they are not applied

By default, the --min-replicas flag on the updater is set to 2. To change this, you can supply the arg in the deploys/updater-deployment.yaml file:

spec:
  containers:
  - name: updater
    args:
    - "--min-replicas=1"

and then deploy it manually if your vpa is already configured.

What are the parameters to VPA recommender?

The following startup parameters are supported for VPA recommender:

Name Type Description Default
recommendation-margin-fraction Float64 Fraction of usage added as the safety margin to the recommended request 0.15
pod-recommendation-min-cpu-millicores Float64 Minimum CPU recommendation for a pod 25
pod-recommendation-min-memory-mb Float64 Minimum memory recommendation for a pod 250
checkpoints-timeout Duration Timeout for writing checkpoints since the start of the recommender's main loop time.Minute
min-checkpoints Int Minimum number of checkpoints to write per recommender's main loop 10
memory-saver Bool If true, only track pods which have an associated VPA false
recommender-interval Duration How often metrics should be fetched 1*time.Minute
checkpoints-gc-interval Duration How often orphaned checkpoints should be garbage collected 10*time.Minute
prometheus-address String Where to reach for Prometheus metrics ""
prometheus-cadvisor-job-name String Name of the prometheus job name which scrapes the cAdvisor metrics "kubernetes-cadvisor"
address String The address to expose Prometheus metrics. ":8942"
kubeconfig String Path to a kubeconfig. Only required if out-of-cluster. ""
kube-api-qps Float64 QPS limit when making requests to Kubernetes apiserver 5.0
kube-api-burst Float64 QPS burst limit when making requests to Kubernetes apiserver 10.0
storage String Specifies storage mode. Supported values: prometheus, checkpoint (default) ""
history-length String How much time back prometheus have to be queried to get historical metrics "8d"
history-resolution String Resolution at which Prometheus is queried for historical metrics "1h"
prometheus-query-timeout String How long to wait before killing long queries "5m"
pod-label-prefix String Which prefix to look for pod labels in metrics "pod_label_"
metric-for-pod-labels String Which metric to look for pod labels in metrics "up{job="kubernetes-pods"}"
pod-namespace-label String Label name to look for pod namespaces "kubernetes_namespace"
pod-name-label String Label name to look for pod names "kubernetes_pod_name"
container-namespace-label String Label name to look for container namespaces "namespace"
container-pod-name-label String Label name to look for container pod names "pod_name"
container-name-label String Label name to look for container names "name"
vpa-object-namespace String Namespace to search for VPA objects and pod stats. Empty means all namespaces will be used. apiv1.NamespaceAll
memory-aggregation-interval Duration The length of a single interval, for which the peak memory usage is computed. Memory usage peaks are aggregated in multiples of this interval. In other words there is one memory usage sample per interval (the maximum usage over that interval model.DefaultMemoryAggregationInterval
memory-aggregation-interval-count Int64 The number of consecutive memory-aggregation-intervals which make up the MemoryAggregationWindowLength which in turn is the period for memory usage aggregation by VPA. In other words, MemoryAggregationWindowLength = memory-aggregation-interval * memory-aggregation-interval-count. model.DefaultMemoryAggregationIntervalCount
memory-histogram-decay-half-life Duration The amount of time it takes a historical memory usage sample to lose half of its weight. In other words, a fresh usage sample is twice as 'important' as one with age equal to the half life period. model.DefaultMemoryHistogramDecayHalfLife
cpu-histogram-decay-half-life Duration The amount of time it takes a historical CPU usage sample to lose half of its weight. model.DefaultCPUHistogramDecayHalfLife
cpu-integer-post-processor-enabled Bool Enable the CPU integer recommendation post processor false

What are the parameters to VPA updater?

The following startup parameters are supported for VPA updater:

Name Type Description Default
pod-update-threshold Float64 Ignore updates that have priority lower than the value of this flag 0.1
in-recommendation-bounds-eviction-lifetime-threshold Duration Pods that live for at least that long can be evicted even if their request is within the [MinRecommended...MaxRecommended] range time.Hour*12
evict-after-oom-threshold Duration Evict pod that has only one container and it OOMed in less than evict-after-oom-threshold since start. 10*time.Minute
updater-interval Duration How often updater should run 1*time.Minute
min-replicas Int Minimum number of replicas to perform update 2
eviction-tolerance Float64 Fraction of replica count that can be evicted for update, if more than one pod can be evicted. 0.5
eviction-rate-limit Float64 Number of pods that can be evicted per seconds. A rate limit set to 0 or -1 will disable the rate limiter. -1
eviction-rate-burst Int Burst of pods that can be evicted. 1
address String The address to expose Prometheus metrics. ":8943"
kubeconfig String Path to a kubeconfig. Only required if out-of-cluster. ""
kube-api-qps Float64 QPS limit when making requests to Kubernetes apiserver 5.0
kube-api-burst Float64 QPS burst limit when making requests to Kubernetes apiserver 10.0
use-admission-controller-status Bool If true, updater will only evict pods when admission controller status is valid. true
vpa-object-namespace String Namespace to search for VPA objects. Empty means all namespaces will be used. apiv1.NamespaceAll