Consider separating webhooks into separate binaries

[atoulme]

Component(s)

No response

Is your feature request related to a problem? Please describe.

The current manager is running all the webhooks against the kubernetes API, and is exposing a set of concerns together rather than separating them.

While this solution may seem expedient, it presents significant problems in terms of composability and maintenance of the chart.

Portions of the chart have no interactions, such as the opampbridge CRD vs the collector CRD vs the instrumentation CRDs.

Describe the solution you'd like

Move specific controlers under separate binaries to be packaged as separate Docker images and managed separately.

Describe alternatives you've considered

We now have enough feature flags to manage the state of the manager and to enable certain webhooks based on CRD availability, but this is not foolproof.

Additional context

No response

[swiatekm]

I'm not against moving the webhook server into its own binary. This is a pattern many operators follow, and its largest benefit is decoupling components with different scaling characteristics.

I don't like the idea of separating the per-CRD functionality this way. There's both resource and architectural overhead involved in increasing the number of deployed services necessary for the operator's default feature set. And from the operator maintainers' side, there's the overhead of managing all of the additional release artifacts. If we find that half of our users disable everything but autoinstrumentation, that's an argument in favor of splitting it out into its own binary, but I'm not ready to accept such a change without any evidence that it's meaningful.