Multi-cluster UI: aka "control plane"

[alexec]

Summary

Users would like the UI to be a single place to manage all workflows.

Use Cases

Reduces the operational burden of installing the UI in many places

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

Relates to #3523.

[meln5674]

This doesn't appear to have gone anywhere in the last year. I am in need of this feature, and I'm willing to implement it. I can see there has been some work done on having multi-cluster workflows, but I'd argue that is a far, far, more difficult problem, and solving a simple but high value feature like this independently first is worthwhile.

I'd like to propose the following:

1. Have a separate workflow controller for each cluster, in addition to the standard one that is deployed now for the "home" cluster. These can be deployed in the same cluster, or deployed in their own clusters. While this does result in additional resources being consumed, it does mean that changing the list of clusters does not impact existing clusters, and it also partitions resources, potentially mitigating a noisy neighbor cluster. It also means that no changes to the workflow controller codebase needs to be made.
2. For each "remote" cluster, have a "landing zone" namespace where the service accounts for SSO are to be found.
3. Have a single server (or single server deployment) whose API is able to interact with a pre-defined set of clusters, provided by a configuration file, args, environment variables, etc. Each remote cluster is expected to be defined as an arbitrary name, a "landing zone" namespace, a list of namespaces to manage (or to manage all namespaces in that cluster), and a kubeconfig file with appropriate access.
4. Add a new API prefix api/v1/remote/${cluster} which duplicates the various per-resource endpoints (e.g. api/v1/workflows/...). This will preserve API compatibility, but allow for requests to be made which target the name of a pre-configured cluster.
5. In the helm chart, have a new section where the clusters are defined, and their kubeconfigs provided as refernces to secret keys. Use a {{ range }} block to duplicate the workflow controller deployment, but mount the kubeconfig and set the KUBECONFIG environment variable for each additional cluster. Remote workflow controllers would be given a dummy service account, as they (I don't believe) would need to interact with the "home" cluster. Add an additional flag to the chart that would result in only the CRDs and RBAC resources being created, this would facilitate quickly deploying the "landing zone" namespace.

If this sounds acceptable, I'd be happy to start work on it right away.