Feature-request: Simplify control plane with a consolidated deployment

[skonto]

Describe the feature

Istio long ago evaluated their deployment strategy and they switched to a more consolidated model.

Right now we have a number of components that are installed by default and could be merged into one.

For example:

• domain-mapping
• domainmapping-webhook
• net-*-controller
• net-istio-webhook
• webhook
• controller
• autoscaler

With a consolidated model we get some immediate benefits:

• closer to edge support (Feature-Request: add memory utilization to activator-hpa #13843)
• improved resource utilization due to cache sharing (Feature-Request: add memory utilization to activator-hpa #13843). Smaller footprint for
  people who use Serving as a Serverless layer (eg. KServe), looks more appealing in general.
• easier configuration/maintenance
• faster startup

cc @ReToCode @dprotaso @rhuss @evankanderson @psschwei @nak3

[dprotaso]

I know various controllers have filtered informers setup with labels/namespaces - does it work when collapsing everyting into a single process?

I'm wondering if we have some limits in how we do dependency injection

[skonto]

I know various controllers have filtered informers setup with labels/namespaces - does it work when collapsing everyting into a single process?

We will have to adjust the informers to filter a set of labels instead of one per controller. I have tried that before at the net-istio side so it should work.

Previous work in this area https://github.com/mattmoor/mink/blob/b9148a39b2d8bbc69ca9aaf5e89a7613c0b179d8/config/core/deployments/dataplane.yaml merged things in one pod as a first step. We have some options but I would like to explore the idea of having everything in a process at least for the control plane as some stuff seem too much to be on their own.
This does not need to happen at one step, could be done incrementally. Initially we could have controller, webhook, domain webhook merged and solve any issues one by one.

In general I see a good opportunity here for using Knative in other environments with small footprint requirements like edge.

[skonto]

/assign skonto