HostPath PV is stuck at terminating when claimed from StatefulSet

[ganchandrasekaran]

What happened:
Trying to delete the deployment, but the PV is stuck at Terminating status forever. However claims made from Kind: Deployment and the PV claimed for it doesn't have any problem terminating successfully.

What you expected to happen:
All pods, services and PV disappear after terminating.

How to reproduce it (as minimally and precisely as possible):

# kubectl describe pv elasticsearch-data-001
Name:            elasticsearch-data-001
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    esdata
Status:          Terminating (lasts <invalid>)
Claim:           abc/data-elasticsearch-data-0
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        2Gi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/k8s-pv/es-data-001
    HostPathType:
Events:            <none>

Make a claim for it from StatefulSet

kind: StatefulSet
...
  volumeClaimTemplates:
  - metadata:
      name: abc
    spec:
      accessModes:
        - ReadWriteOnce
      volumeMode: Filesystem
      storageClassName: esdata
      resources:
        requests:
          storage: 1Gi

Anything else we need to know?:
Ofcourse as suggested in bugReport solves the issue.

kubectl patch pvc db-pv-claim -p '{"metadata":{"finalizers":null}}'
kubectl patch pod db-74755f6698-8td72 -p '{"metadata":{"finalizers":null}}'

But this creates two problem.
1, I am not able to redploy the same chart without manually deleting these PVs as the chart produces the same PV names.
2, After I manually delete the PVs, I change the name of PV on the chart and redeploy. But the Kubernetes assigns the OLD PV that doesn't exist anymore to my Claim. THIS MEANS THAT THE ABOVE STEP DOESN'T REMOVE THE PV COMPLETELY.

Any insight would be appreciated! Thanks

@saad-ali
@childsb

[athenabot]

/sig storage

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nemobis]

/reopen Still valid