Need an API call for "teardown all external resources"

[zmerlynn]

See #4627 / #4530: These are both the wrong approach, as also noted in #4411 (comment). We need to delete these things on the master, prior to deleting the VM itself. For system add-ons, this is basically the API hook necessary for #3579 cleanup, but it's also required for any user services that were created as well.

[zmerlynn]

cc @roberthbailey @jlowdermilk

[fgrzadkowski]

cc @jszczepkowski

Can you please explain what this API would look like? I'm not sure that widening API for a central component is the right approach.
If external resource is tightly related to a resource (e.g. service) it'd be strange to have a separate API call just to remove it's external parts (e.g. load balancer). Also it may leave a resource (service) in an inconsistent state.
Additionally you will still need something like kube-down.sh to clear other things (e.g. remove machines etc.) so what's the benefit?

[fgrzadkowski]

Just to be clear - I think that removing of internal services should be handled by master itself, but I see no reason why we need an API call to delete external resources for user defined services.

[zmerlynn]

First, the meta-point: kube-down.sh isn't the only setup and teardown
"client". (GKE is, others might come along.) That was the meta-point for
#3579 as well. To the extent that we can push this logic onto the server,
we should.

The second meta-point: today ELBs are the biggest issue. There are
discussions around, say, firewall rules - those would also need teardown.
Or xyz cloud provider network widget. The point is, that shell script is
going to keep creeping.

As far as what the API looks like, we were envisioning something like:
"kubectl clusterteardown", that made one API call to the master (call it
teardown, or destroy, or destroyExternalResources or whatever) that could
even just have a loop on Go very similar to rejected PR #4530. (And yes,
we'd make one addition to kube-down.sh, right before the VM itself was
annihilated.)

The API should probably be best-effort semantics, since it's going down
anyways.
On Feb 23, 2015 12:08 AM, "Filip Grzadkowski" notifications@github.com
wrote:

cc @jszczepkowski https://github.com/jszczepkowski

Can you please explain what this API would look like? I'm not sure that
widening API for a central component is the right approach.
If external resource is tightly related to a resource (e.g. service) it'd
be strange to have a separate API call just to remove it's external parts
(e.g. load balancer). Also it may leave a resource (service) in an
inconsistent state.
Additionally you will still need something like kube-down.sh to clear
other things (e.g. remove machines etc.) so what's the benefit?

—
Reply to this email directly or view it on GitHub
#4630 (comment)
.

[zmerlynn]

I just saw your next comment. Why a distinction between user services and
add-ons here? They both represent external resources owned by cluster
services. Some might be owned by the user, so you could argue that we need
policy bits like "don't delete on teardown", but that doesn't mean the
client should delete them.
On Feb 23, 2015 6:42 AM, "Zachary Loafman" zml@google.com wrote:

First, the meta-point: kube-down.sh isn't the only setup and teardown
"client". (GKE is, others might come along.) That was the meta-point for
#3579 as well. To the extent that we can push this logic onto the server,
we should.

The second meta-point: today ELBs are the biggest issue. There are
discussions around, say, firewall rules - those would also need teardown.
Or xyz cloud provider network widget. The point is, that shell script is
going to keep creeping.

As far as what the API looks like, we were envisioning something like:
"kubectl clusterteardown", that made one API call to the master (call it
teardown, or destroy, or destroyExternalResources or whatever) that could
even just have a loop on Go very similar to rejected PR #4530. (And yes,
we'd make one addition to kube-down.sh, right before the VM itself was
annihilated.)

The API should probably be best-effort semantics, since it's going down
anyways.
On Feb 23, 2015 12:08 AM, "Filip Grzadkowski" notifications@github.com
wrote:

cc @jszczepkowski https://github.com/jszczepkowski

Can you please explain what this API would look like? I'm not sure that
widening API for a central component is the right approach.
If external resource is tightly related to a resource (e.g. service) it'd
be strange to have a separate API call just to remove it's external parts
(e.g. load balancer). Also it may leave a resource (service) in an
inconsistent state.
Additionally you will still need something like kube-down.sh to clear
other things (e.g. remove machines etc.) so what's the benefit?

—
Reply to this email directly or view it on GitHub
#4630 (comment)
.

[zmerlynn]

Also, to be clear, the API can just outright delete the services, too. I don't see a reason it has to delete the underlying resources versus the services themselves, since it's running in the shutdown path. If we really want to be delicate, we can terminate all objects (#1535) first.

[jszczepkowski]

I'll be happy working on this. I hope no one is working on it now.

[alex-mohr]

We need to support deletion of clusters, both for GKE and for e.g. e2e tests. The master will create and delete various CloudProvider objects, so it should own those objects. We need a way to (a) crease accepting new objects, (b) change desired state for existing objects to does-not-exist, (c) block until the CloudProvider reconciler (or equivalent) finishes actually deleting all of those, then (d) the master can be deleted.

Given the master knows what it created and has code to delete such things (for whatever version of k8s it's running), we should use the master itself to clean up a cluster that needs to be deleted, not require some out-of-band tool to do so.

[bgrant0607]

Discussion is occurring in #5025.

[brendandburns]

I don't think that this makes the 1.0 cut.

[alex-mohr]

I don't think that this makes the 1.0 cut.

@brendandburns Without this, we orphan resources in GCE on cluster delete. And if e.g. user spins up a new cluster with the same name, there will be all sorts of fun from dangling rules. I think this falls under operational reliability.

[lavalamp]

I don't think a 'protected' field is necessary for this. Today we have RBAC, finalizers, and GC. I think a client w/ super admin powers could delete all namespaces and then wait for the namespace count to go to 0. (There's probably a corner case or two around the default and kube-system namespaces that this would turn up.)

[zmerlynn]

I might be missing it, but I haven't found a way to delete default and kube-system. If you come up with one, I'll close the bug. :)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[roberthbailey]

/remove-lifecycle stale

[roberthbailey]

It still isn't possible to delete the following namespaces: kube-system, kube-public, default. So we can't rely on finalizers / GC for resources in those namespaces.

We also don't have a way to put the apiserver into a lame duck mode to prevent new namespaces from being created during cluster teardown.

[bgrant0607]

/lifecycle frozen

[neolit123]

seems like a FR for api-machinery, that can eventually land in kubectl (sig-cli).
sig-cluster-lifecycle tools can adapt it via client-go if they need to.

/sig cli api-machinery