Resolve HA / multi-thread concerns around Gateway updates

[evankanderson]

Currenty, we assume a single Gateway resource; when using auto-TLS, we need to update Spec.Listeners for each Knative Route.

The default controller implementation will attempt to reconcile multiple KIngress resources at once, which means that it's likely that we will get some number of update collisions on the Gateway. We should figure out a mechanism for spreading updates across multiple Gateways to enable HA / multi-threaded controllers without bottlenecking on a single Gateway instance.

This is extracted from this PR comment

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[knative-prow-robot]

This issue or pull request is stale because it has been open for 90 days with no activity.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close

/lifecycle stale

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[evankanderson]

/remove-lifecycle stale

I'll take a look and see if this is still necessary

[dprotaso]

Sorta a dupe of #363

Two potential options to explore:

1. From It is hard to programmatically manage `Spec.Listeners` kubernetes-sigs/gateway-api#1248 - add a conformance test to assert Gateway merges and push for this to be part of the CORE.
2. See if Server side apply gives us some granularity here - maybe we use distinct field managers for listener entries - though this implies there isn't some validation preventing duplicate listeners (which is trying to be added via Webhook: Validate unique port, protocol, and hostname per listener kubernetes-sigs/gateway-api#847) which I'm trying to halt - Webhook: Validate unique port, protocol, and hostname per listener kubernetes-sigs/gateway-api#847 (comment) (ignore as hostname would be unique in each listener)

Related to 1. I think we should formalize a different way of merging Gateways beyond having the same addresses. I'll make a GEP for this.

[dprotaso]

Upstream GEP issue
kubernetes-sigs/gateway-api#1713

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[ReToCode]

/remove-lifecycle stale
/triage accepted

[dprotaso]

An alternative as well is using server side apply -

I've added the client API here - kubernetes-sigs/gateway-api#2537

It'll need more exploration once the client changes land