Application Garbage Collection

[kow3ns]

There are at least two ways to deal with garbage collection of application resources. The first, and simplest, approach is to inject owner reference into each component of the application using the Name, APIVersion, and Kind of the Application CRD. This would work as follows. There is no need to worry about injecting OwnerReferences into intermediate or children objects (e.g. If an OwnerReference is injected into a Deployment, chain garbage collection will delete its ReplicaSets and their Pods). When the Application is deleted the Garbage Collector will delete all Orphaned children. While this has the benefit of simplicity, it has a few drawbacks.

1. The Application object must be crated prior to the creation of any owned children. Otherwise, the Garbage Collector may delete the children prior to creation of the Application.
2. Adopting existing workloads
3. All tools would have to implement this uniformly and correctly.

Another approach would be to construct a graph based controller using the Application's Selector and appropriate labels on children objects to indicate ownership. This controller would watch for the creation of all objects in the system and establish ownership by automatically injecting OwnerReferences. As above, when the Application object is deleted, the Garbage Collector will delete its children.This mitigates the need for tools to inject OwnerReferences, and it ensures that the Garbage Collector will not delete an Application's children prior to creation of the Application (i.e. there need not be a serialized ordering between observation of Application creation and the creation of its children), but it has drawbacks as well.

1. The graph based Application controller is comparable in complexity to the Garbage Collector itself.
2. It still requires that an Application and its children are appropriately labeled (i.e. It does not fully mitigate the potential for erroneous behavior do to user errors).
3. It will require mem and cpu resources on par with the Garbage Collector, and it will require that the Application controller watch (potentially) nearly all creation and deletion events in the system.

Another option is to begin with user tool injected OwnerReferences, and, when the full requirements of Garbage Collection become well understood after some period of production usage, automate the process by adding OwnerReference marking in the Application controller. The controller can be implemented in a way that is strictly backward compatible with user injected OwnerReference. In fact, in order function properly it must be compatible with user injected OwnerReferences.

[mattfarina]

@kow3ns thanks for bring this up here and in the KEP. It's inspired a number of great conversations I've had the opportunity to have with folks.

The approach where the controller monitors and adds owners refs to create/delete events has some complexities. For example,

• What if you have a database used by two applications (or more). You delete one application but still have the other so the database needs to be kept around. How do we handle that situation? Does this play into the dependency proposal?
• What if people want to suspend cleanup of just some dependencies? With the setting of owner references themselves they can control that. How would that happen here and be configurable?

I'm not sure of the right answer but I'd like the more opinionated approaches to be layered on (optional in some form) rather than required by all unless all want them.

[konryd]

@mattfarina In the scenario with shared database, I'd say the db should not be marked as being a part of one application or another, but a separate entity (possibly having its own application).

Between the two approaches given by @kow3ns, I'm in favor of the second one (controller attaching ownerrefs), as most user-friendly and able to cover the adoption scenario out of the box.

Ideally the users would be prevented from deploying overlapping applications, but I understand this is currently not supported due to race conditions. In that case, it would be the controller's job to:

1. Mark the problematic situation (eg. set a condition field in applications: "there are apps with conflicting selectors").
2. Set arbitrary app as the owner of conflicting resources, or follow some other defined behavior.

One more complication in the controller's implementation would be watching changes to application's selector field and re-adopting or un-owning resources accordingly.

[ant31]

I'm in favor of the controller attaching ownerrefs.

About the case of shared resources, we may still want to select them via the 'ApplicationCRD'. So what about adding a field on the components so users can protect them, like garbageCollect: false.
Also, this can be useful in general, e.g. I want to delete/create all component of an application except a secret.

@kow3ns , If the The graph based Application controller is comparable in complexity to the Garbage Collector itself, why not re-using/implement ing the garbage collector in the AppController. It seems silly and duplicated work but would mitigate drawbacks pointed:

• One more complication in the controller's implementation would be watching changes to application's selector field and re-adopting or un-owning resources accordingly.
• It will require mem and cpu resources on par with the Garbage Collector, and it will require that the Application controller watch (potentially) nearly all creation and deletion events in the system.

[jzelinskie]

Building the Operator Lifecycle Manager, we basically discovered that we want a queryable GC API for asking questions of the ownership tree. We have some internal proposals and I know some of the CoreOS upstream contributors poked around seeing if it was a reasonable initiative.

[mattfarina]

@konryd I have some thoughts...

@mattfarina In the scenario with shared database, I'd say the db should not be marked as being a part of one application or another, but a separate entity (possibly having its own application).

We are now discussing opinions on app operation and resource "linking". Kubernetes is not a PaaS and there's intent around not being opinionated as a PaaS would be. That way many can build their own processes on top of k8s. I'd argue that the default solution for describing an application, provided by the k8s project, is also not opinionated on decisions like this.

Ideally the users would be prevented from deploying overlapping applications, but I understand this is currently not supported due to race conditions.

This is another opinionated design. Kubernetes has been focused on infra. Why should the k8s take up an opinion like this?

Remember, this project is trying to help solve tool interoperability for apps. It is not trying to solve app operation which is more complicated than any one tool can cover. Operators are a great example of the need for different business logic for different apps.

@kow3ns When it comes to garbage collection, in the general non-opinionated sense, what felt user problem are we trying to solve in a v1?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.