Support volume expansion

[candlerb]

local-zfs-provisioner sets the "quota" attribute on the dataset to the selected size in the pvc. I was expecting that changing the pvc definition and reapplying would update the quota, but this is currently rejected.

Tested with: microk8s 1.17 (2 node cluster) on Ubuntu 18.04.

To reproduce: after creating the example pvc and pod, edit example/pvc.yaml, change storage: 2Gi to storage: 4Gi and reapply.

# kubectl apply -f pvc.yaml
Error from server (Forbidden): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{},\"name\":\"local-zfs-pvc\",\"namespace\":\"default\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"4Gi\"}},\"storageClassName\":\"local-zfs\"}}\n"}},"spec":{"resources":{"requests":{"storage":"4Gi"}}}}
to:
Resource: "/v1, Resource=persistentvolumeclaims", GroupVersionKind: "/v1, Kind=PersistentVolumeClaim"
Name: "local-zfs-pvc", Namespace: "default"
Object: &{map["apiVersion":"v1" "kind":"PersistentVolumeClaim" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{},\"name\":\"local-zfs-pvc\",\"namespace\":\"default\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"2Gi\"}},\"storageClassName\":\"local-zfs\"}}\n" "pv.kubernetes.io/bind-completed":"yes" "pv.kubernetes.io/bound-by-controller":"yes" "volume.beta.kubernetes.io/storage-provisioner":"asteven/local-zfs" "volume.kubernetes.io/selected-node":"nuc2"] "creationTimestamp":"2020-01-18T10:58:59Z" "finalizers":["kubernetes.io/pvc-protection"] "name":"local-zfs-pvc" "namespace":"default" "resourceVersion":"1458580" "selfLink":"/api/v1/namespaces/default/persistentvolumeclaims/local-zfs-pvc" "uid":"0885b2e2-b76e-4c00-ae4e-3df08df0f161"] "spec":map["accessModes":["ReadWriteOnce"] "resources":map["requests":map["storage":"2Gi"]] "storageClassName":"local-zfs" "volumeMode":"Filesystem" "volumeName":"pvc-0885b2e2-b76e-4c00-ae4e-3df08df0f161"] "status":map["accessModes":["ReadWriteOnce"] "capacity":map["storage":"2Gi"] "phase":"Bound"]]}
for: "pvc.yaml": persistentvolumeclaims "local-zfs-pvc" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

(Note: it makes no difference if you delete the pod first)

Workaround: I could manually set the "quota" attribute on the dataset. But this isn't in the spirit of k8s declarative configuration.

# zfs get all -r zfs/k8s | grep local
zfs/k8s/pvc-0885b2e2-b76e-4c00-ae4e-3df08df0f161  quota                 2G                                                                       local
zfs/k8s/pvc-0885b2e2-b76e-4c00-ae4e-3df08df0f161  mountpoint            /var/lib/local-zfs-provisioner/pvc-0885b2e2-b76e-4c00-ae4e-3df08df0f161  local

Observations:

• StorageClass has an attribute allowVolumeExpansion, and this is what controls whether the resize can take place. Unfortunately it never allows shrinking.
• local-zfs-provisioner already has a comment wondering whether create might be called on an existing volume

[asteven]

Thanks for collecting all the infos.
Happy to accept any PR's.

[candlerb]

I'm really just finding my way round k8s storage. My test cluster has zpools, therefore local-zfs-provisioner is really useful as it is.

Browsing code, I discovered that local-zfs-provisioner is not using either CSI or FlexVolume interfaces, but external volume provisioning. That spec doesn't talk about growing existing volumes.

The other thing I found is that most of the controller logic is using sigs.k8s.io/sig-storage-lib-external-provisioner/controller. That doesn't contain the words "grow" or "resize" either.

So it's unclear to me whether external provisioners are even allowed to handle growing volumes, and if they can, whether the controller library supports it.

Even the core k8s documentation seems out of line here, where it says:

An external static provisioner can be run separately for improved management of the local volume lifecycle. Note that this provisioner does not support dynamic provisioning yet.

However, local-zfs-provisioner is clearly doing dynamic provisioning, and the volume-provisioning doc clearly talks about creating volumes, so I think this is out-of-date.

[candlerb]

Resize support in sig-storage-lib-external-provisioner is ruled out at kubernetes-sigs/sig-storage-lib-external-provisioner#43.