VPA does not perform in-place pod updates despite InPlaceOrRecreate policy

[robertbotez]

What happened?

I deployed Vertical Pod Autoscaler with updateMode: InPlaceOrRecreate, but the pods are never updated in place. Instead, the VPA updater logs show:

I1004 13:14:39.847429 1 pods_inplace_restriction.go:97] "Can't in-place update pod, but not falling back to eviction. Waiting for next loop" pod="default/app-569758fbc5-jzzjc"

• In-place scaling does work manually (kubectl patch with CPU change updates the pod without restart).
• VPA recommends higher CPU values, but it never applies them in-place.
• No eviction happens either — the pod just stays unchanged.

What did you expect to happen?

With updateMode: InPlaceOrRecreate and resizePolicy.restartPolicy: NotRequired, I expect VPA to patch the running Pod’s CPU requests in-place when the recommendation is above the current request.

How to reproduce it

1. Enable feature gates on API server + kubelet:
   --feature-gates=InPlacePodVerticalScaling=true
2. Run VPA with feature gate:
   FEATURE_GATES="InPlaceOrRecreate=true" ./hack/vpa-up.sh
3. Create deployment:

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: app  
spec:  
  replicas: 1  
  selector:  
    matchLabels:  
      app: app  
  template:  
    metadata:  
      labels:  
        app: app  
    spec:  
      containers:  
      - name: nginx  
        image: nginx  
        resources:  
          limits:  
            cpu: 200m 
          requests:  
            cpu: 100m
        resizePolicy:  
        - resourceName: cpu  
          restartPolicy: NotRequired

4. Create VPA:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       app
  updatePolicy:
    updateMode: "InPlaceOrRecreate"
  resourcePolicy:
    containerPolicies:
      - containerName: "nginx"
        minAllowed:
          cpu: "200m"
        maxAllowed:
          cpu: "700m"
        controlledResources: ["cpu"]

5. Apply CPU load in pod (e.g. with stress-ng) until recommendation > current requests.
6. Check VPA updater logs → see error above.

Environment

• Kubernetes version: 1.33.5
• VPA version: 1.5.0
• Feature gates: InPlacePodVerticalScaling=true (API server, scheduler, controller-manager, kubelet)
• Pod resizePolicy: restartPolicy: NotRequired
• Verified that manual kubectl patch can update pod CPU in-place

Additional info

• VPA gives recommendations (visible in status), so recommender is working.
• Updater log confirms it attempted in-place update but refused with:
  Can't in-place update pod, but not falling back to eviction. Waiting for next loop
• No events are created on the pod.
• It seems the restriction logic in pods_inplace_restriction.go prevents updates even when they are possible.

[adrianmoisey]

/area vertical-pod-autoscaler

[adrianmoisey]

I1004 13:14:39.847429 1 pods_inplace_restriction.go:97] "Can't in-place update pod, but not falling back to eviction. Waiting for next loop" pod="default/app-569758fbc5-jzzjc"

Do you see a log message like this, just above the one you pasted?

I1004 18:29:01.786193       1 pods_restriction_factory.go:212] "Too few replicas" kind="ReplicaSet" object="default/app-6d84cf7789" livePods=1 requiredPods=2 globalMinReplicas=2
I1004 18:29:01.786898       1 pods_inplace_restriction.go:97] "Can't in-place update pod, but not falling back to eviction. Waiting for next loop" pod="default/app-6d84cf7789-6ctg5"

In my testing I see it. I think we left the restriction that a Deployment can handle disruption for in-place to work.

If you set spec.updatePolicy.minReplicas to 1 in the VPA resource, does that fix the issue?

[robertbotez]

Yes, I saw that log message. I set spec.updatePolicy.minReplicas and it worked. Thank you.

But I still don't understand something. I successfully scaled a pod during an apt update operation, but when I try to force some load with stress-ng, like stress-ng --cpu 2 --cpu-load 100 --timeout 600s I keep getting:

I1005 18:25:39.847738 1 update_priority_calculator.go:131] "Not updating a short-lived pod, request within recommended range" pod="default/app-5d5fd59855-bhh85"

These are the pod limits:

Limits:
      cpu:  300m
    Requests:
      cpu:        25m

and VPA config:

minAllowed:
          cpu: "20m"
        maxAllowed:
          cpu: "600m"

Also, the VPA recommended:

  Recommendation:
    Container Recommendations:
      Container Name:  nginx
      Lower Bound:
        Cpu:  25m
      Target:
        Cpu:  25m
      Uncapped Target:
        Cpu:  25m
      Upper Bound:
        Cpu:  25m

Why the pod limits doesn't change when I induce load with stress-ng?

[adrianmoisey]

I'm not too sure I understand the question.

This is saying that the CPU requests is within the correct range:

I1005 18:25:39.847738 1 update_priority_calculator.go:131] "Not updating a short-lived pod, request within recommended range" pod="default/app-5d5fd59855-bhh85"

Which you'll notice here:

Requests:
 cpu:        25m

It matches:

Recommendation:
Container Recommendations:
Container Name: nginx
Target:
Cpu: 25m

[robertbotez]

I'm trying to dynamically change cpu limits with VPA by inducing load with stress-ng. I thought that if my container limit is 300m, and the maxAllowed field in VPA is 600m, in case of load the cpu limit will be dynamically changed.

[adrianmoisey]

VPA doesn't control limits directly, limits is a ratio of the incoming pod's requests/limits ratio.
So if the raw Pod spec is 10m CPU requests and 20m CPU limits, the VPA should maintain a 1:2 ratio.

However, I haven't double checked what in-place does here, so it may be a bug.
Do you see requests being changed?

[robertbotez]

Something is strange and I can't explain (or understand) the behaviour. For example, with the configurations below, the following happens to me:

VPA:
...
minAllowed:
          cpu: "20m"
        maxAllowed:
          cpu: "900m"
...

Deployment:
...
resources:
          limits:
            cpu: 300m
          requests:
            cpu: 25m

Initially, the limits are those imposed in the deployment. After I ran the stress command, the limits were updated to:

Limits:
      cpu:  10800m
    Requests:
      cpu:        900m

Aren't these values ​​too high? The ratio is indeed 1:12, and 900x12 is 10800m, but how the requests.cpu is calculated?

Also, after I stopped the stress command and the load on the pod is approx. 0%, after how long do the resource limits scale down? These values ​​remained high, after scaling up and now I get the following logs in the updater:

I1006 07:23:39.858125 1 update_priority_calculator.go:131] "Not updating a short-lived pod, request within recommended range" pod="default/app-test-f7b7f66bf-d5wtn"

[robertbotez]

Nevermind, I understand now what you said about maintaining the ratio, in this case 1:12. But still, the question is why scaling down doesn't work.

[adrianmoisey]

Also, after I stopped the stress command and the load on the pod is approx. 0%, after how long do the resource limits scale down?

How long are you waiting for this? What do you have configured for

autoscaler/vertical-pod-autoscaler/pkg/recommender/main.go

Line 105 in 524270b

cpuHistogramDecayHalfLife = flag.Duration("cpu-histogram-decay-half-life", model.DefaultCPUHistogramDecayHalfLife, `The amount of time it takes a historical CPU usage sample to lose half of its weight.`)

?

[robertbotez]

Well, nothing happened already in approx. 2h. I didn't set anything for that parameter.

[adrianmoisey]

It defaults to 24h. Give it time, I think it should move after about 12 hours of a CPU spike

[robertbotez]

I decreased it to 1h and now it's working. Thank you very much. I think this can be closed.

[adrianmoisey]

/close