Infrastructure volume not deleted if I delete the PV on AWS, Azure, GCP, OpenStack

[vlerenc]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: When I delete a PV, the backing infrastructure volume is not deleted (and the PVC turns to lost). Only if I delete the PVC that led to the PV and the backing infrastructure volume in the first place, all three entities, i.e. the PVC and PV in Kubernetes and the backing infrastructure volume are deleted.

What you expected to happen: If I delete a PV (which I did accidentally, which is why I never noticed this behaviour before), I expect it to delete the backing infrastructure volume, otherwise it becomes orphan, remains in the infrastructure and generates costs.

How to reproduce it (as minimally and precisely as possible):

kubectl create -f pvc.yaml
kubectl get pv
kubectl delete pv PV_NAME

I used the following super simple pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: foobar
spec:
  storageClassName: default
  resources:
    requests:
      storage: 17Gi
  accessModes:
  - ReadWriteOnce

Console output:

$ k create -f pvc.yaml
persistentvolumeclaim "foobar" created

$ ka get pvc
NAMESPACE   NAME      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
default     foobar    Bound     pvc-03145d94-e5fc-11e7-994a-0a580af406f9   17Gi       RWO           default        3m

$ k get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM            STORAGECLASS   REASON    AGE
pvc-03145d94-e5fc-11e7-994a-0a580af406f9   17Gi       RWO           Delete          Bound     default/foobar   default                  3m

$ k delete pv pvc-03145d94-e5fc-11e7-994a-0a580af406f9
persistentvolume "pvc-03145d94-e5fc-11e7-994a-0a580af406f9" deleted

$ k get pv
No resources found.

$ k get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
foobar    Lost      pvc-03145d94-e5fc-11e7-994a-0a580af406f9   0                        default        6m

Anything else we need to know?:
I had self-created vanilla Kubernetes clusters on AWS, Azure, GCP and OpenStack. It happened on all four infrastructures. Deleting the PVC always deleted itself, the PV and the backing infrastructure volume. Deleting the PV directly nowhere deleted the backing infrastructure volume.

Environment:

• Kubernetes version (use kubectl version):
  • AWS, OpenStack: 1.8.4
  • Azure: 1.8.5
  • GCP: 1.8.2
• Cloud provider or hardware configuration: See above.
• OS (e.g. from /etc/os-release): CoreOS Container Linux

[vlerenc]

/sig storage

[mlmhl]

Hi @vlerenc , it is a intentional behavior to keep the backing infrastructure volume exist. Deleting the backing infrastructure volume is a dangerous operation because it may lead to data loss. Only when you create a PVC with dynamic volume provisioning mechanism and the ReclaimPolicy set to Delete, the backing infrastructure volume will be deleted after the PVC is deleted.

As you can see from your pvc.yaml, you use a storage class named default, and I guess its ReclaimPolicy must be Delete(which is the default value), so the PV and backing infrastructure volume are deleted after you delete the PVC.

You cane get a detailed description of dynamic volume provisioning from these docs:
StorageClass
PV reclaiming.
dynamic volume provisioning

[vlerenc]

Who am I to argue, but I find that surprising/misleading. If I delete a PV, I expect the backing volume to be gone. I am sure there are better ways to let the user control the behaviour. Also, e.g. Bosh keeps such disks for a few (configurable) days around and a GC deletes them after a grace period. The way PVs are implemented in Kubernetes right now, people don't even notice they have disks a.) with potentially sensitive data that they have forgotten about and b.) generating unintended costs.

I think discussions like #23032 prove the point, that the current behaviour is counter-intuitive.

Also, the above was only a shortcut. What I actually did at first was to delete the PVC, while its PV was mounted by a pod. Obviously, the backing volume couldn't be deleted, but the deletion of the PVC passed without an error. That surprised me. When I deleted the pod and then the PV I noticed nothing happened. By then Kubernetes had forgotten about the backing volume. Then I repeated the experiment, but this time I deleted the PV right away. This all makes no sense to me, especially if the idea is to bind the PV to the PVC. It is far to easy to end up in a situation with orphan volumes, at least for my liking.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close