Support for controllers spanning across clusters

[loewenstein]

We have a use case for a controller responsible for a (custom) resource A in one cluster that kind of owns another (custom) resource B in another cluster. I say "kind of owns" because there are of course no owner references that span clusters, but it can be imitated with labels and a mapping event handler rather easily.

The problem is that there is no easy way to configure the controller to watch for Bs in that there client.

• A 'Kind' source would look for Bs in the wrong cluster.
• I suppose it could work with a separate manager and controller with no reconciler who's only purpose is to setup a watch and provide an Informer for an Informer source.
  Not sure if that would really work though, we would have a controller that prepends to reconcile Bs and we would need a dummy reconciler. Feels like a hack.
• we could probably use a Channel source, create our own client using client-go and feed events. However, this sounds like a lot of work/custom code, I am not sure how well it would integrate with the rest of controller-runtime

Could we instead have a manager that can have n clients and way that kind watches can specify which client to use?

[alvaroaleman]

This is already possible by using multiple managers and injecting the correct cache into source.Kind, although this doesn't have tests, hence it's not officially supported, see #650

[loewenstein]

I was made aware of #650 as well (after opening this one).

It describes what is probably the best option currently available. However, as managers drive the lifecycle of controllers (and other parts of the controller-runtime), I would hesitate to call it a solution.

Do you think a multi-cluster setup as first-class citizen of the controller-runtime is a viable and feasible option?

[vincepri]

/kind design
/help

[k8s-ci-robot]

@vincepri:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

Details

In response to this:

/kind design
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[akutz]

A project with which I am involved is doing this. I will share the design here once I figure out what I am allowed to share.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[alvaroaleman]

xref POC on how this could look like: #950
/remove-lifecycle stale

[adrienjt]

JFYI: I forked controller-runtime back in November 2018 and built this: https://github.com/admiraltyio/multicluster-controller

Merging it back would be a breaking change to controller-runtime's API though.

The folks at Orange use admiraltyio/multicluster-controller for their multi-cluster Cassandra operator: https://github.com/Orange-OpenSource/casskop/blob/master/multi-casskop/Readme.md

I use it in https://github.com/admiraltyio/multicluster-scheduler, but I also write sample-controller-like multi-cluster controllers, as explained here: https://youtu.be/Fv2PKKDgjIQ (KubeCon Barcelona talk)

@alvaroaleman I'll have a look at your POC #950. Back when I started multicluster-controller, controller-runtime's "dependency injection" was coupling packages quite tightly (Java-style vs. idiomatic), which is why I had to fork more than I wanted, but things may have changed.