Workload controllers should support templating of custom resources

[fabiand]

Today workload controllers usually contain templates for pods.

The application survey showed on slide 9 that multiple parties are interested to allow the controllers to support custom resource templating as well.

/kind feature

In our use case - KubeVirt - this would allow us to stamp out VMs.
Obviously these VMs would be treated by the workload controller like pods - which would be fine.

[fabiand]

@kubernetes/sig-apps

/sig apps

[fabiand]

@mattfarina maybe you are aware if there is already a similar issue or if it's tracked elsewhere?

[liggitt]

It's unclear what this is asking for. Are you suggesting that an existing kube controller should start creating a new type of custom resource based on another custom resource? I would expect each pair of owner/child resources to be managed by its own controller, both for least privilege reasons, and because each pairing is likely to have domain specific edge cases that are cleaner to deal with in isolation

[fabiand]

It's unclear what this is asking for. Are you suggesting that an existing kube controller should start creating a new type of custom resource based on another custom resource?

Yes.

I would expect each pair of owner/child resources to be managed by its own controller, both for least privilege reasons, and because each pairing is likely to have domain specific edge cases that are cleaner to deal with in isolation

I see your point and I agree. If you want to correct treatment (semantically for a given resource) then you will need a custom controller. However, we could turn this around and say: If you are fine getting your resource being treated like a pod - including in the edge cases. Then it should be fine to reuse an existing controller.

[liggitt]

However, we could turn this around and say: If you are fine getting your resource being treated like a pod - including in the edge cases. Then it should be fine to reuse an existing controller.

We should not expand the scope of existing controllers to do that. The more unknown dependents a controller has, the less ability it has to make changes in the future. A controller that can reason about how changes would affect a replicaset<->pod relationship would have no idea what that would mean for a customcontrollertype <-> customleaftype relationship.

cc @kubernetes/sig-apps-feature-requests

[fabiand]

I understand this approach. However, I still wanted to raise it, as it seems that this appeared more often in the survey.

[enisoc]

If you want something similar to one of the built-in workload controllers, but you want it to stamp out some other resource instead of Pods, you might be interested in Metacontroller. One of the original design goals was to make it easy to write custom workload controllers.

[liggitt]

One of the original design goals was to make it easy to write custom workload controllers.

I am very much in favor of tools to make it simple to write best-practice controllers, over extending existing controllers' responsibilities

[liggitt]

/assign @kow3ns @mattfarina

[fabiand]

Hm. I understand.

[kow3ns]

The work with extension mechanisms and tools like kubebuilder are their to provide users with the ability to extend the API server and add controller for custom resources. I second @liggitt's notion above.
/close