Deletion of PVCs on statefulsetscale-up due to low intervals

[JacobAmar]

Issue with node-cleanup-controller

Automatic deletion of PVC (PersistentVolumeClaim) /PV (PersistentVolume) on scale-up of statefulset pods.
Each time a statefulset pod is evicted, a PVC is required to be deleted by setting a low discovery interval and deletion delay time. However, the issue arises when the system is scaled up. During a statefulset pod scale-up, the PVC/PV are first created and due to the short discovery interval and deletion delay time, they are automatically deleted.

Expected Behavior:

When a statefulset pod scaling-up operation occurs, there should be no deletion of PVC/PV.

Steps to Reproduce:

To reproduce the issue, please follow the steps below:

Set a discovery interval to 5 seconds and deletion delay to 1 second.
Deploy a statefulset with PVC.
Attempt to scale it up.
After the scale-up, you will notice that the newly created PVC/PV are deleted automatically due to low interval times.
Anything else we need to know?:

Environment:

• CSI Driver version: we are using a local nvme disk on ec2
• Kubernetes version (use kubectl version): 1.23
• OS (e.g. from /etc/os-release): "Amazon Linux 2"
• Kernel (e.g. uname -a): 5.10.186-179.751.amzn2.x86_64

[JacobAmar]

Update:
it seems like a bug in version 2.6.0, the real problem is that the controller tried to delete a pv from active node

[mauriciopoppe]

/cc @msau42

[msau42]

The deletion logic should only be triggered when the Node doesn't exist. Can you clarify in the scale up case, when does the Node object get created?

It is possible that there could also be a race condition where the controller does not see a new Node yet when it's processing a PV.

[msau42]

One thing that may help is to make sure we only process PVCs that are Bound.

cc @mattcary

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[msau42]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale