Workflows Operator (ex. Super-controller)

[alexec]

Summary

When managing Argo Workflows you must choose

• Cluster-scoped install - single controller with lots of permissions and must be large for many workflows, but easy to manage
• Namespace-scope - scales with the number of namespaces, and secured within the user's namespace, but hard to manage

What about a super-controller that installs the workflow controller just-in-time (i.e. whenever it sees a workflow created in a namespace) and uninstalls it when all workflows in that namespace are complete.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

This is similar to what Argo-event does. Create the controller/executor on the namespace to process the resource.
In the workflow-controller scenario, it should deploy controller and RBACs on that namespace. Technically, Supercontroller should install the namespaced workflow-controller on each namespace if the namespace has a workflow

[alexec]

@sarabala1979 exactly what I think. I have an alternative solution too:

• NamespaceSet - install a set of manifests into each namespace of a cluster
• Zero-pod Autoscaler (ZPA) - scales a controller to zero if there are zero resources to control (back to 1 if there are). HPA then takes over.

These are deployment management tools. The downside is that by becoming general, we lose the ability to be specific.

[sarabala1979]

@alexec I think this approach is better than the master-slave/sharding approach. This can scale with any load with a distributing approach.

[sarabala1979]

#4318

[alexec]

I think this is just an https://coreos.com/operators/

[dcherman]

Interesting idea! Off the top of my head, here's some initial thoughts:

• This does make scaling easier if your workflows are (relatively) evenly distributed between namespaces, however if you end up with a hot namespace running the bulk of your workloads, you'd likely still end up needing to shard the workflow controller for that namespace.
• How are secrets managed? If you have workflow archiving enabled, we would end up needing to create/destroy the secrets for s3/postgres/etc. in the corresponding namespaces, unless we just have these controllers be responsible for the executor of workflows, then have a separate deployment be responsible for workflow archiving.

[alexec]

hot namespace running the bulk of your workloads

I actually think "hot namespace" is the default. I don't think this feature can address scaling issues.

[alexec]

How are secrets managed?

I'm don't sure it can. A controller must have access to secret data (e.g. MySQL login), so either the operator can install this in any namespace, or you don't use it.

[alexec]

Some thoughts:

• We only install the controller in explicitly labeled namespaces, rather than all. workflows.argoproj.io/controller: "true"
• We can use annotations as extension point for customization:
  • workflows.argoproj.io/argo-server-ingress: "true" - install an ingress
  • workflows.argoproj.io/scale-to-zero: false - never scale to zero

[alexec]

This is basically going be the agent.

[alexec]

See #9990