controller: garbage collect abandoned clusters

[benesch]

Feature request

If the drop of a cluster is successfully recorded in the catalog, but not in Kubernetes, the cluster will be forever orphaned in Kubernetes. The controller needs to learn to look for abandoned clusters in Kubernetes (via Orchestrator::list_services) on startup and remove them.

[umanwizard]

What happens if the environment is permanently deleted and never comes back? Does something in the cloud repo clean up all the associates pods?

[benesch]

Yep, we delete the Kubernetes namespace, which deletes all resources in the namespace.