Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to remove cstorVolumeReplicas after uninstall #2374

Closed
cjyar opened this issue Feb 4, 2019 · 6 comments · Fixed by openebs/maya#955
Closed

Unable to remove cstorVolumeReplicas after uninstall #2374

cjyar opened this issue Feb 4, 2019 · 6 comments · Fixed by openebs/maya#955
Assignees
@AmitKumarDas @umamukkara @ranjithwingrider @sonasingh46
Labels
BUG Cleanup OpenEBS-Archive ORG Transfer into OpenEBS-Archive ORG type: Debuggability
Milestone
0.8.2

Comments

@cjyar
Copy link

cjyar commented Feb 4, 2019

Description

After uninstalling OpenEBS according to the instructions at https://docs.openebs.io/docs/next/uninstall.html, the openebs namespace still contains 3 undeletable CVRs.

Expected Behavior

Removing the openebs namespace should have removed all the resources in it, including these CVRs.

Current Behavior

$ kubectl get cvr --all-namespaces
NAMESPACE   NAME                                                       AGE
openebs     pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-1yi8   18d
openebs     pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-ij6z   18d
openebs     pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-np5b   18d
$ kubectl delete cvr -n openebs --all --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
cstorvolumereplica.openebs.io "pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-1yi8" force deleted
cstorvolumereplica.openebs.io "pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-ij6z" force deleted
cstorvolumereplica.openebs.io "pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-np5b" force deleted
^C
$ kubectl get cvr --all-namespaces
NAMESPACE   NAME                                                       AGE
openebs     pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-1yi8   18d
openebs     pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-ij6z   18d
openebs     pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-np5b   18d

Detail on one of those CVRs:

$ kubectl get cvr -n openebs pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-1yi8 -o yaml
apiVersion: openebs.io/v1alpha1
kind: CStorVolumeReplica
metadata:
  annotations:
    cstorpool.openebs.io/hostname: blahblahblah
  creationTimestamp: 2019-01-17T16:57:11Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2019-02-04T20:57:51Z
  finalizers:
  - cstorvolumereplica.openebs.io/finalizer
  generation: 2
  labels:
    cstorpool.openebs.io/name: cstor-disk-1yi8
    cstorpool.openebs.io/uid: 5c640750-1a78-11e9-8797-00259013594a
    cstorvolume.openebs.io/name: pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e
    openebs.io/cas-template-name: cstor-volume-create-default-0.8.0
    openebs.io/persistent-volume: pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e
    openebs.io/version: 0.8.0
  name: pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-1yi8
  namespace: openebs
  resourceVersion: "281990513"
  selfLink: /apis/openebs.io/v1alpha1/namespaces/openebs/cstorvolumereplicas/pvc-ebe9459b-1a78-11e9-82a9-002590d74e1e-cstor-disk-1yi8
  uid: ee765816-1a78-11e9-8797-00259013594a
spec:
  capacity: 8G
  targetIP: 10.3.0.116
status:
  capacity:
    totalAllocated: 6K
    used: 6K
  phase: Invalid

Steps to Reproduce

Hard to say. I installed OpenEBS from the operator ("kubectl") instructions, played around with it for a while, and then followed the uninstall instructions.

Your Environment
@sonasingh46
Copy link
Contributor

Hi @cjyar
Did you delete PVC first before deleting OpenEBS namespace as pointed in docs?

@sonasingh46
Copy link
Contributor

To get out of this --
kubectl edit cvr -n openebs

And remove finalizers.

@cjyar
Copy link
Author

cjyar commented Feb 5, 2019

@sonasingh46 Thanks for the help. After deleting the finalizers, the CVRs went away, and so did the namespace and the CRD.

Yes, I deleted the PVC before deleting the OpenEBS namespace. It's possible that the PVC was never fully created, since it took me a while to get all the settings right. So the PVC may have been in a weird state when I deleted it.

@AmitKumarDas
Copy link
Member

AmitKumarDas commented Feb 6, 2019 •
edited

Thanks @cjyar for further explanations.
Meanwhile, I have labelled this issue under docs to have the docs improved.

@AmitKumarDas AmitKumarDas added this to the 0.8.2 milestone Feb 22, 2019
@kmova
Copy link
Member

kmova commented Feb 22, 2019

From @gila, if there are multiple CVRs to be deleted:

CRD=`kubectl get crd | grep cstorvolumereplica | cut -d" " -f1` && kubectl patch crd $CRD -p '{"metadata":{"finalizers": [null]}}' --type=merge

AmitKumarDas pushed a commit to AmitKumarDas/maya that referenced this issue Feb 27, 2019
fix(cvr delete, finalizer): remove finalizers to ensure successful de…
7603668 
…letion

This commit removes the finalizers from a cstor volume replica object. This
ensures removal of cstor volume replica object from kubernetes. In addition,
it will try to delete the underlying volume if cvr controller gets the delete
event. With the removal of finalizer there is no more retry mechanism to
actually delete the underlying volume if there were issues encountered during
earlier deletes.

Why was finalizer put in first place?
CstorVolumeReplica aliased CVR has finalizers that prohibits deletion unless
the volume gets deleted. This is a check to ensure proper cleanup even if
delete events are missed & / or underlying volume was not deleted in earlier
attempts.

Observations:
There were observations that point to the fact that when the namespace
(that contains these volumes) as well when the operator (that contains the
CRD schema definitions) are deleted, CVR finalizers pose a significant problem
to the overall deletion & later re-creation procedure. In fact the PODs that
are responsible for actual volume deletion & confirming the delete status is
no more available when its namespace or the openebs operator get deleted.

Future Work:
A better design needs to be implemented that ensures volume cleanup & hence
space reclamation.

fixes openebs/openebs#2374
fixes openebs/openebs#2349

Signed-off-by: AmitKumarDas <amit.das@mayadata.io>
@AmitKumarDas AmitKumarDas mentioned this issue Feb 27, 2019
Merged
9 tasks
AmitKumarDas pushed a commit to AmitKumarDas/maya that referenced this issue Feb 27, 2019
fix(cvr delete, finalizer): remove finalizers to ensure successful de…
3f1aed4 
…letion

This commit removes the finalizers from a cstor volume replica object. This
ensures removal of cstor volume replica object from kubernetes. In addition,
it will try to delete the underlying volume if cvr controller gets the delete
event. With the removal of finalizer there is no more retry mechanism to
actually delete the underlying volume if there were issues encountered during
earlier deletes.

Why was finalizer put in first place?
CstorVolumeReplica aliased CVR has finalizers that prohibits deletion unless
the volume gets deleted. This is a check to ensure proper cleanup even if
delete events are missed & / or underlying volume was not deleted in earlier
attempts.

Observations:
There were observations that point to the fact that when the namespace
(that contains these volumes) as well when the operator (that contains the
CRD schema definitions) are deleted, CVR finalizers pose a significant problem
to the overall deletion & later re-creation procedure. In fact the PODs that
are responsible for actual volume deletion & confirming the delete status is
no more available when its namespace or the openebs operator get deleted.

Future Work:
A better design needs to be implemented that ensures volume cleanup & hence
space reclamation.

fixes openebs/openebs#2374
fixes openebs/openebs#2349

Signed-off-by: AmitKumarDas <amit.das@mayadata.io>
vishnuitta pushed a commit to openebs/maya that referenced this issue Feb 28, 2019
@vishnuitta
fix(cvr delete, finalizer): remove finalizers to ensure successful de…
d04834e 
…letion (#955)

This commit removes the finalizers from a cstor volume replica object. This
ensures removal of cstor volume replica object from kubernetes. In addition,
it will try to delete the underlying volume if cvr controller gets the delete
event. With the removal of finalizer there is no more retry mechanism to
actually delete the underlying volume if there were issues encountered during
earlier deletes.

Why was finalizer put in first place?
CstorVolumeReplica aliased CVR has finalizers that prohibits deletion unless
the volume gets deleted. This is a check to ensure proper cleanup even if
delete events are missed & / or underlying volume was not deleted in earlier
attempts.

Observations:
There were observations that point to the fact that when the namespace
(that contains these volumes) as well when the operator (that contains the
CRD schema definitions) are deleted, CVR finalizers pose a significant problem
to the overall deletion & later re-creation procedure. In fact the PODs that
are responsible for actual volume deletion & confirming the delete status is
no more available when its namespace or the openebs operator get deleted.

Future Work:
A better design needs to be implemented that ensures volume cleanup & hence
space reclamation.

fixes openebs/openebs#2374
fixes openebs/openebs#2349

Signed-off-by: AmitKumarDas <amit.das@mayadata.io>
@ranjithwingrider
Copy link
Member

This is documented. https://docs.openebs.io/docs/next/troubleshooting.html#cvr-deletion-unsuccessful

AmitKumarDas pushed a commit to AmitKumarDas/maya that referenced this issue Mar 12, 2019
fix(cvr delete, finalizer): remove finalizers to ensure successful de…
b109aec 
…letion (openebs#955)

This commit removes the finalizers from a cstor volume replica object. This
ensures removal of cstor volume replica object from kubernetes. In addition,
it will try to delete the underlying volume if cvr controller gets the delete
event. With the removal of finalizer there is no more retry mechanism to
actually delete the underlying volume if there were issues encountered during
earlier deletes.

Why was finalizer put in first place?
CstorVolumeReplica aliased CVR has finalizers that prohibits deletion unless
the volume gets deleted. This is a check to ensure proper cleanup even if
delete events are missed & / or underlying volume was not deleted in earlier
attempts.

Observations:
There were observations that point to the fact that when the namespace
(that contains these volumes) as well when the operator (that contains the
CRD schema definitions) are deleted, CVR finalizers pose a significant problem
to the overall deletion & later re-creation procedure. In fact the PODs that
are responsible for actual volume deletion & confirming the delete status is
no more available when its namespace or the openebs operator get deleted.

Future Work:
A better design needs to be implemented that ensures volume cleanup & hence
space reclamation.

fixes openebs/openebs#2374
fixes openebs/openebs#2349

Signed-off-by: AmitKumarDas <amit.das@mayadata.io>
vishnuitta pushed a commit to openebs/maya that referenced this issue Mar 12, 2019
@vishnuitta
fix(cvr delete, finalizer): remove finalizers to ensure successful de…
aa5f62e 
…letion (#955)

This commit removes the finalizers from a cstor volume replica object. This
ensures removal of cstor volume replica object from kubernetes. In addition,
it will try to delete the underlying volume if cvr controller gets the delete
event. With the removal of finalizer there is no more retry mechanism to
actually delete the underlying volume if there were issues encountered during
earlier deletes.

Why was finalizer put in first place?
CstorVolumeReplica aliased CVR has finalizers that prohibits deletion unless
the volume gets deleted. This is a check to ensure proper cleanup even if
delete events are missed & / or underlying volume was not deleted in earlier
attempts.

Observations:
There were observations that point to the fact that when the namespace
(that contains these volumes) as well when the operator (that contains the
CRD schema definitions) are deleted, CVR finalizers pose a significant problem
to the overall deletion & later re-creation procedure. In fact the PODs that
are responsible for actual volume deletion & confirming the delete status is
no more available when its namespace or the openebs operator get deleted.

Future Work:
A better design needs to be implemented that ensures volume cleanup & hence
space reclamation.

fixes openebs/openebs#2374
fixes openebs/openebs#2349

Signed-off-by: AmitKumarDas <amit.das@mayadata.io>
prateekpandey14 pushed a commit to prateekpandey14/maya that referenced this issue Apr 3, 2019
@prateekpandey14
fix(cvr delete, finalizer): remove finalizers to ensure successful de…
2d217dd 
…letion (openebs#955)

This commit removes the finalizers from a cstor volume replica object. This
ensures removal of cstor volume replica object from kubernetes. In addition,
it will try to delete the underlying volume if cvr controller gets the delete
event. With the removal of finalizer there is no more retry mechanism to
actually delete the underlying volume if there were issues encountered during
earlier deletes.

Why was finalizer put in first place?
CstorVolumeReplica aliased CVR has finalizers that prohibits deletion unless
the volume gets deleted. This is a check to ensure proper cleanup even if
delete events are missed & / or underlying volume was not deleted in earlier
attempts.

Observations:
There were observations that point to the fact that when the namespace
(that contains these volumes) as well when the operator (that contains the
CRD schema definitions) are deleted, CVR finalizers pose a significant problem
to the overall deletion & later re-creation procedure. In fact the PODs that
are responsible for actual volume deletion & confirming the delete status is
no more available when its namespace or the openebs operator get deleted.

Future Work:
A better design needs to be implemented that ensures volume cleanup & hence
space reclamation.

fixes openebs/openebs#2374
fixes openebs/openebs#2349

Signed-off-by: AmitKumarDas <amit.das@mayadata.io>
vishnuitta pushed a commit to openebs/maya that referenced this issue Apr 4, 2019
@vishnuitta
fix(cvr delete, finalizer): remove finalizers to ensure successful de…
1c02bf7 
…letion (#955)

This commit removes the finalizers from a cstor volume replica object. This
ensures removal of cstor volume replica object from kubernetes. In addition,
it will try to delete the underlying volume if cvr controller gets the delete
event. With the removal of finalizer there is no more retry mechanism to
actually delete the underlying volume if there were issues encountered during
earlier deletes.

Why was finalizer put in first place?
CstorVolumeReplica aliased CVR has finalizers that prohibits deletion unless
the volume gets deleted. This is a check to ensure proper cleanup even if
delete events are missed & / or underlying volume was not deleted in earlier
attempts.

Observations:
There were observations that point to the fact that when the namespace
(that contains these volumes) as well when the operator (that contains the
CRD schema definitions) are deleted, CVR finalizers pose a significant problem
to the overall deletion & later re-creation procedure. In fact the PODs that
are responsible for actual volume deletion & confirming the delete status is
no more available when its namespace or the openebs operator get deleted.

Future Work:
A better design needs to be implemented that ensures volume cleanup & hence
space reclamation.

fixes openebs/openebs#2374
fixes openebs/openebs#2349

Signed-off-by: AmitKumarDas <amit.das@mayadata.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AmitKumarDas AmitKumarDas

@umamukkara umamukkara

@ranjithwingrider ranjithwingrider

@sonasingh46 sonasingh46

Labels
BUG Cleanup OpenEBS-Archive ORG Transfer into OpenEBS-Archive ORG type: Debuggability
Projects
None yet
Milestone
0.8.2
Development

Successfully merging a pull request may close this issue.

fix(cvr delete, finalizer): remove finalizers to ensure successful deletion AmitKumarDas/maya
6 participants
@cjyar @AmitKumarDas @kmova @umamukkara @ranjithwingrider @sonasingh46