Can't downsize to the current size after failed volume upsize

[sebgl]

The following situation happened to an Azure user:

1. User resize their volume claims up
2. Operator propagates the change to PVC & StatefulSet
3. AKS can't resize the volume (see PVC resizing doesn't work as expected Azure/AKS#1477):

error expanding volume "elastic-system/elasticsearch-data-elastic-eck-es-data-0" of plugin "kubernetes.io/azure-disk": compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Cannot resize disk kubernetes-dynamic-pvc-cc190c75-488b-40f7-94ba-9ec8dc3b9e5c while it is attached to running VM /subscriptions/*****************/resourceGroups/******************/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-93507671-vmss/virtualMachines/aks-agentpool-93507671-vmss_1. Resizing a disk of an Azure Virtual Machine requires the virtual machine to be deallocated. Please stop your VM and retry the operation."

4. Users try to set back the initial size. This is rejected by the operator:

Failed to apply spec change: handle volume expansion: decreasing storage size is not supported: an attempt was made to decrease storage size for claim elasticsearch-data

At this point it's pretty hard for the user to figure out a way to get back to a clean state, since the StatefulSet has been recreated by ECK with the new size already.
Should the operator catch the PVC event error and allow a downsize to go through in that particular case?

[leosunmo]

Just in case someone encounters this like I did, using AKS, Azure managed Kubernetes, here's my work around in rough steps.
So far I've only tried to apply new disk size while ECK was running which led to the above error, but I think you can avoid these errors by changing the Elasticsearch CRD desired disk size at step 6 only instead.

1. Scale down ECK to 0
   This is to make sure it doesn't interfere with any of the termination or Elasticsearch settings.
   kubectl scale statefulset elastic-operator -n elastic-system --replicas 0
2. Stop Elasticsearch shard allocations and sync
   This is steps 1,2, and optionally 3 from https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-cluster.html#restart-cluster-full
3. Delete StatefulSet with orphan cascade option
   Next you need to remove the ECK-managed StatefulSet for your NodeSet to make sure it doesn't re-create Pods.
   Important to note here is that we use --cascade=orphan to stop it from removing the Pods.
   kubectl delete statefulset my-nodeSet --cascade=orphan
   It's important that you do NOT delete any Services associated with the StatefulSet.
4. Delete the Pods
   Next we can remove the Pods. This will leave the PVCs unbound which will cause them to get detached from the Azure VM, which is what the official docs suggest you do, https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/known-issues/sizegrow.md.
   If you're cautious, delete one Pod first and check that the disk is resized, then move on to the next Pod and so on. I just verified with one Pod/PVC, then deleted all other Pods in my NodeSet.
5. (Optional) Resize Disks
   I had a problem where the volume manager was stuck in the errored state and wouldn't resize my disks even after the Pod was deleted and volume detached from the VM, so I edited the PVC again at this point and increased the size by a single GB just to kick-start the resize process, which seemed to work.
   I had other PVCs eventually resize without requiring a second edit, but it took 10-20 minutes...
6. Update Elasticsearch CRD
   Make sure the Elasticsearch CRD is reflecting the new PVC sizes if you had to change them a second time at step 5, or if you didn't resize them before starting the process.
7. Start ECK again
   Start ECK again which should trigger it to create a new StatefulSet for the NodeSet, which will reuse the existing PVCs, and boom, you have bigger disks.
8. Enable shard allocation again
   After a while (depending on size of cluster) the Cluster will be in a Yellow state again (up from Red when it first starts up). This should be because all primary shards are available, but none of the replica ones, since we disabled shard allocation in step 2. Re-enable it by following step 8 in https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-cluster.html#restart-cluster-full. The cluster should become green after a while, depending on number of indices/shards.

Hopefully this works for others. I am running Elasticsearch 6.8.13 with ECK 1.7.0 in this scenario.

[BenjaminHerbert]

Hello there, I just ran into this issue as well, as I wanted to resize a disk to 4Ti. This failed, as the storage-class I am using was using cachingmode=ReadOnly, which is only supported <4Ti (4095Gi is okay...)

Resetting to the old value or 4095Gi does not work.

Is the above workaround still the way to go, @sebgl?

[JulienOrain]

@BenjaminHerbert I am in the same situation than you. Did you find a better alternative by any chance ?

[BenjaminHerbert]

Sorry for the late reply, @JulienOrain.

IIRC, I created a new Kubernetes storage class

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: no-caching-
parameters:
  cachingmode: None
  ...
provisioner: disk.csi.azure.com
...

Added a new nodeSet with increased size using the newly created storage class, migrated the data from the old nodeSet, and deleted the original after successful transfer.

[JulienOrain]

Hi @BenjaminHerbert !
yes, I applied the same strategy :)
thanks for your answer 🙏

[naemono]

We may want to update our documentation: https://www.elastic.co/docs/troubleshoot/deployments/cloud-on-k8s/common-problems for this common problem, and potential ways out of this issue such as the simplest which is:

1. rename the nodeSet, which will create a new nodeSet/STS at the larger size, and migrate all data over, and eventually delete the old nodeSet/STS.