Removing stuck with the message "Waiting on cluster-scoped-gc"

[oed-guzym]

Rancher Server Setup

• Rancher version: v2.5.8
• Installation option: Docker

Information about the Cluster

• Kubernetes version: v1.19.4
• Cluster Type: Downstream
  • Custom

Describe the bug
In the Rancher UI, a cluster is stuck with the message "Waiting on cluster-scoped-gc". We don't get the cluster removed and we can't interact with it anymore. If we manually go over the "Delete" button nothing happens. Via the API we see the following transition message:

"state": "removing", "transitioning": "error", "transitioningMessage": "waiting on cluster-scoped-gc"

When we click on the cluster in the UI, we get the message:

ClusterUnavailable 503: cluster not found

Due to lack of ideas, we have already tried to rebuild the cluster using the same automated way, but we get the following error message:

Bad response statusCode [422]. Status [422 Unprocessable Entity]. Body: [code=MissingRequired, fieldName=ClusterTemplateRevision, message=this cluster is created from a clusterTemplateRevision, please pass the clusterTemplateRevision, baseType=error]

A connection via "kubectl" can also no longer be established.

The infrastructure of the cluster no longer exists. This is only the entry in the interface, which cannot be removed. We currently have no idea how to get the cluster removed.

To Reproduce
Unfortunately, we do not know exactly how we got into this state. The problem appeared after we automatically deployed and removed the cluster for testing purposes.

Result
The result of the error is that we see an orphaned cluster in the interface and unfortunately we cannot get the entry removed.

Expected Result
It would be good if there was some way to force deletion of orphaned artifacts / clusters and all associated references in the UI.

Screenshots

[bartcee]

I encountered the same issue on Rancher v.2.5.9. The cluster was imported with the rancher CLI tool and deleted with rancher cluster rm cluster-name. Stuck with the Waiting on cluster-scoped-gc error message.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[Heiko-san]

Any workaround to fix clusters in that state ? :(

[vichaos]

Switch to rancher-cluster 's kube-context and run following command (Replace ${STUCK_CLUSTER} with the stuck cluster's name or id )

kubectl get clusters.management.cattle.io $(kubectl get clusters.management.cattle.io \
-o jsonpath='{range .items[*]}{@.spec.displayName}:{@.metadata.name}{"\n"}{end}'| \
grep ${STUCK_CLUSTER} | cut -d ":" -f2) -o=json|jq '.metadata.finalizers = null' | kubectl apply -f -

Any workaround to fix clusters in that state ? :(

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.