Which component are you using?:

Vertical Pod Autoscaler

What version of the component are you using?:

Component version: v0.13.0

What k8s version are you using (kubectl version)?:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.16", GitCommit:"60e5135f758b6e43d0523b3277e8d34b4ab3801f", GitTreeState:"archive", BuildDate:"2023-05-30T23:39:56Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.16", GitCommit:"60e5135f758b6e43d0523b3277e8d34b4ab3801f", GitTreeState:"clean", BuildDate:"2023-01-18T15:54:23Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"

What environment is this in?:

Self hosted on EC2

What did you expect to happen?:

When running multiple recommenders, only checkpoints that correspond with that recommender are eligible for deletion during garbage collection. I.e., for a checkpoint to be deleted by RecommenderA, the checkpoint needs to be associated with RecommenderA.

What happened instead?:

RecommenderA filters out any VPAs that aren’t associated with it, and then in the garbage collection process those checkpoints would be considered ‘orphaned’ and deleted.

The following logs could be found:

I1216 06:14:17.567806     571 cluster_feeder.go:331] Orphaned VPA checkpoint cleanup - deleting some-namespace/some-vpa-checkpoint.

Which only showed for VPA checkpoints not associated with the recommender.

This can also be seen in code:

1. VPA objects are refreshed inside the LoadVPAs function
2. Inside that function, after retrieving all of the VPA objects in that cluster, it does a filterVPAs

Inside this function it includes any VPAs that match the recommender’s name, filtering out any VPAs that have a different recommender name.

We can see this happening based on logs “Ignoring vpaCRD vpa-something in namespace some-namespace as new-recommender recommender doesn't process CRDs implicitly destined to default recommender”

So we know that all the VPAs are able to be loaded at this point, but are being filtered out.

3. After filtering, it processes the VPAs that match the current recommender, and does an important call to AddOrUpdateVpa [2]

Inside this function it will either simply update the VPA object inside the cluster state, or it will create and update it if it’s not present yet.

This is the only place the clusterState.Vpas object is added too.

4. Later in the recommender tick, we get to the MaintainCheckpoints [2] function
5. If enough time has elapsed since the last checkpoint garbage collection, which by default is 10min, we initiate the GarbageCollectCheckpoints [2] function.

Inside this function we once again do the LoadVPAs function, to make sure we are working on fresh data (remember that inside this function we filter!)

We then grab all checkpoints, in all namespaces (which is not filtered), and check if the VPA for that checkpoint exists in the clusterState.

Since we know from (3) that the clusterState only contains the filtered list of VPAs, and this function checks all VPA checkpoints, it starts deleting the VPA checkpoints not associated with that recommender.

We can validate that this happened based on seeing the logs: “I1216 06:14:17.567806 571 cluster_feeder.go:331] Orphaned VPA checkpoint cleanup - deleting some-namespace/some-vpa-checkpoint.”

How to reproduce it (as minimally and precisely as possible):

This can easily be reproduced by spinning up two Recommenders in the same cluster, keep one as the default recommender (i.e., don’t override the recommender name) and add a new one.

Create a few VPAs for each of the corresponding Recommenders, and then start watching for the ‘Orphaned VPA checkpoint cleanup*’ log.

After getting them both running, list the checkpoints in that cluster and you should see them being deleted every ~10 minutes (i.e., the garbage collection interval).

Anything else we need to know?:

The fix should be relatively simple, I can put that together if we come to agreement that this is actually a bug and not intended behavior.

The work around is to set the garbage collection interval to something very large, and replace your recommender pods before that interval is elapsed. However this does have the downside that you are effectively turning off the checkpoint garbage collection logic.