PVC Autoscaler: PoC to automate downscaling of `PersistentVolumeClaims`

[plkokanov]

How to categorize this topic?

/area storage
/area auto-scaling
/kind enhancement
/kind poc

/label teamsize/medium

What is the topic about?:
GEP-0038: Autoscaling PersistentVolumeClaims proposes changes to the pvc-autoscaler controller and how it will be used in Gardener runtime and Seed clusters to scale up observability components. Downscaling is listed as a non-goal because it is a complex and risky operation and its research and implementation would delay the immediate value of scaling up PVCs from a smaller starting size for new clusters.

However, support for downscaling is a potential future enhancement that can be added after the implementation of GEP-0038 is complete.

The goal of this task would be dig deeper into how PVCs can be downscaled and how the operation can be automated reliably - risk of data loss removed and downtime reduced as much as possible.

As there is no native way to downscale volumes in k8s, one way to achieve it is with the following steps:

1. The workload is scaled down.
2. A temporary PVC with a smaller size is created.
3. Data is copied over from the larger PVC to the smaller one, potentially with rsync.
4. The smaller PersistentVolume's persistentVolumeReclaimPolicy is changed to Retain.
5. Both the larger (original) and smaller (temporary) PVCs are deleted - this results in the larger volume being deleted, but the smaller one remaining.
6. The original PVC is re-created, now with the smaller size and volumeName pointing to the smaller PersistentVolume.
7. The claimRef in the PersistentVolume is edited to point to the PVC from the previous step.
8. The workload is scaled-up

Note that VolumeSnapshots cannot be used, because PVCs cannot be created with a smaller size than the one which was used to create the VolumeSnapshot.

The implementation to downscale PVCs could be done in a new controller introduced to the pvc-autoscaler.