Volume can be deleted in the middle of cloning

[jsafrane]

We use finalizers to prevent volumes from being deleted while a snapshot is being taken. Should we use the same precaution to protect volumes that are being cloned? Or do we assume that the cloning is faster / easier and storage backend will sort it out somehow?

@msau42 @xing-yang, what do you think?

[xing-yang]

Actually snapshotting is usually faster than cloning if no upload is needed. I think it is a good idea to add a finalizer to prevent the source volume from being deleted in cloning.

[jsafrane]

Looking at the external provisioner, it's simple to add a finalizer to PVC that is being cloned.

But it's quite hard to remove it reliably. Here is a brief design:

• During ProvisionExt(), the provisioner adds a finalizer to PVC before calling cloning CreateVolumeRequest.

• CSI external-provisioner must watch PVCs via a new workqueue.

  • When PVC X has DeletionTimestamp && it has the finalizer:
    • If all PVCs that have X as its data source are not Pending, the finalizer is removed.
    • If there is a Pending PVC that has X as its data source (i.e. cloning may be in progress), the PVC is added back to the workqueue with exp. backoff, so the PVC is re-evaluated periodically, until it is cloned.
  • Therefore CSI external-provisioner must instantiate and run its own PV/PVC/StorageClass shared informers and pass them to provisioning library as options.
  • As side effect, if a PVC that has X as data source is wrong (has wrong storage class, size, ...), it effectively blocks X from being deleted, even though there is no cloning in progress.
    • Checking if cloning is in progress for a PVC is currently impossible, as the cloning may have been started by a previous provisioner instance that has just been restarted. Any in-memory cache will not survive this.

• During ProvisionExt(), the provisioner clones a PVC even if the source PVC has DeletionTimestamp: the cloning operation may have been started when the source PVC was "ok" and the operation timed out. The provisioner must call CreateVolume again to get the final error code or provisioned volume. [This is actually current behavior, posting here for completeness].

[jsafrane]

/help

[k8s-ci-robot]

@jsafrane:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Danil-Grigorev]

/assign

[jsafrane]

/help cancel