GEP-1713: Standard Mechanism to Merge Multiple Gateways

[dprotaso]

Note - this refactors out the 'merging' aspect of Gateways from #1757 - but tweaks the approach

What type of PR is this?

/kind gep

What this PR does / why we need it:

Outlines a mechanism to merge Gateways

Which issue(s) this PR fixes:

Fixes #1713

Does this PR introduce a user-facing change?:

[hzxuzhonghu]

How can validate the invalid gateways conflicts?

[dprotaso]

How can validate the invalid gateways conflicts?

Can you provide some examples that come to mind?

Generally, I think a first step is to decide the expectations for gateway ports and listeners (within a single gateway) - #1842

Then this GEP should decide if merging across gateways allows us to relax some of those requirements.
ie. Multiple listeners on the same port should definitely be allowed. Knative need this to support adding certificates.

[hzxuzhonghu]

Can you provide some examples that come to mind?

Just come to mind because we met this in istio, istio's gateway has server fields which is like listener here. If all the listners in one gateway, it can be validated at least. But if they are in at least two gateways, once they are created at the same time, we have no way to validate them. This is the mechanism of eventual consistence limitation, the only way is to detect any conflict on running

[dprotaso]

This is the mechanism of eventual consistence limitation, the only way is to detect any conflict on running

I agree - I think when it comes to the scenario you mentioned I would suggest that if a child gateway cannot merge into a parent gateway then the child gateway status conditions Accepted and Ready should be set to False

[dprotaso]

Can folks take another look at the GEP - I hope to have addressed the topics in the discussions and I've flushed out a lot of the semantics.

[howardjohn]

A few small comments but LGTM!

[shaneutt]

One of the teams I work with does some merging of Gateways which may be relevant here and I would be interested to have their review on the subject:

/cc @mlavacca @pmalek @rainest

If we could give a little bit of time for them to check this out they might have some good thoughts on the matter.

/hold

[pmalek]

No substiantial comments from my side but I just want to ask this to be sure: (maybe I've missed this in the comments somewhere):

Can a child attach to a parent in a different namespace? My gut tells me that this is not meant to be allowed but it wasn't explicitly mentioned here.

[dprotaso]

Can a child attach to a parent in a different namespace? My gut tells me that this is not meant to be allowed but it wasn't explicitly mentioned here.

I added a sentence to make this explicit. The API uses an LocalObjectReference which means it can only reference Gateway in the same namespace.

[updated Feb 7 2024] I added support for cross namespace

[dprotaso]

/hold cancel

[keithmattix]

This looks right to me; will approve later once I can go through in more detail

[shaneutt]

/cc @shaneutt @robscott @youngnick

[mikemorris]

This feels pretty solid to me now, and I believe all the dependent issues this has spawned have been resolved. Left a few questions but overall happy with where this is at and excited to see if move forward!

/approve

[mikemorris]

This feels logically safe because it gates the removal of the existing restriction on explicitly setting a new optional field, so my understanding is that this wouldn't risk breaking older implementations (a concern previously cited in #1817 (comment) and #1596 (comment)).

@robscott @youngnick thoughts?

[louiscryan]

I'm definitely support of this but its worth pointing out that this does not represent a guarantee that the merged set produces MinItems=1 since there may be no children. Thus we cannot completely state that runtime merging is the exact equivalent of a single merged resource.

Given that runtime handling of empty lists by controllers was given as a reason for rejecting

#1596
#1817

we may have to reconsider the logic of that decision given the desire to support merging. E.g.

"if controllers want to support merging as per the spec they must tolerate empty listeners even if merging is not used"

It seems like we are making a similar constraint loosening decision around certs for TLS listeners
#2721

[robscott]

Agree with @louiscryan here. I think it's important to note this tradeoff in the GEP. Even implementations that didn't support Gateway merging would still need to be aware of this potential configuration and ensure that it didn't break them. If we move forward with this approach, we'll need to take extra efforts to communicate the disruptive nature of this change.

[cornfeedhobo]

If I may toss in an opinion here.. I've been watching this proposal because I like the use case of service-based teams and trusting them to manage any needed domains. It would be ideal if these teams didn't have to disrupt core infrastructure resources (e.g. a top-level 80/443 gateway) before deploying their domain.

Could this be more specific about the way allowedChildren should be used? Could it specify an intent to support regex and/or broader matching, e.g. all gateways in a namespace.?

[mikemorris]

@cornfeedhobo I'd expect allowedChildren to look very similar to AllowedRoutes, and specifically the namespaces field type RouteNamespaces, which yes would support matching all gateways in one or more namespaces (Same namespace, All namespaces, or by using a label selector match).

[mikemorris]

This feels functionally quite similar to ParentReference - is the rationale for introducing a new type here just to avoid any confusion regarding the optional sectionName and port fields on ParentReference, or some other reason?

[dprotaso]

sectionName and port (that are fields in ParentReference) don't really mean anything in this context

[dprotaso]

We have a bunch of these here actually - https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1/object_reference_types.go

Big thing is to have smart defaulting - eg. in this scenario we want the kind/type to be a Gateway.

[mikemorris]

Hmm, ParentReference does already default to kind: Gateway (although the API reference docs should probably be updated to mention this) and group: gateway.networking.k8s.io (which GatewayObjectReference should likely change to if it feels worth keeping distinct).

Understood that sectionName and port wouldn't be meaningful, just trying to see if we can limit proliferation of similar-enough structs.

[howardjohn]

I think the general advice in k8s apis is to not excessively reuse types when they are not the same thing. Past example was Ingress and Service shared a type ofr status and a field accidentally got added to both thinking it was just one (oops!).

In this case, there are already irrelevant fields so I think its best to keep it a new type

[mikemorris]

In that case, would a more generalized name like AttachmentReference (still defaulting to group: gateway.networking.k8s.io and kind: Gateway) feel a bit less awkward if it were to ever be used for attachment to non-Gateway kinds?

[dprotaso]

The reason why the type has a Gateway prefix wasn't to imply that it only attaches to Gateways but that it's the Gateway that wants to attach to something else.

I'm flexible here - I can see how it can be read as "Attach to this other Gateway"

If we go for AttachmentReference then I'd probably drop the defaulting of the gateway kind. This would open up the type to be re-used in the future. eg. HTTPRoute attaching to another one.

Getting more input here would be useful

[mikemorris]

For purposes of discoverability, is there any way to determine just by examining status on a parent Gateway, which child Gateways have successfully attached to it? (Agreed that pushing up listener conditions from child to parent would be undesirable though.)

[dprotaso]

A list of children on the parent's status could be an option here. Alternatively, could follow what routes do and have a count on the parent status.

I'm flexible here - would solicit what people want

[howardjohn]

I would not put a list, its possible unbounded?

For routes we just have a count.. which is still somewhat problematic with high churn but better than a list

[mikemorris]

Another path may be surfacing these relationships through gwctl similar to policy attachments, I don't think we need to add this if it's problematic due to potential churn and/or not helpful enough as just a count.

[robscott]

I'd say a count is probably the least bad option here, but this thread does highlight the overall complexity this GEP would be adding.

[louiscryan]

Is this required or can the child simply omit. Omission is likely better if the root gateway is infrastructure that is being managed centrally. The children don't actually need to know or care about the class.

[dprotaso]

I think it would great to not require it.

I'm not familiar if it's possible with CEL to make a field optional if some other property is set (eg. infrastructure.attachTo`

@robscott do you know - I'd consider you the CEL expert.

[dprotaso]

A thing to note is that getting the gateway would show an empty class

gateway-api/apis/v1/gateway_types.go

Line 27 in df978ef

// +kubebuilder:printcolumn:name="Class",type=string,JSONPath=`.spec.gatewayClassName`

Unless we made a change so that it appears in the status

[howardjohn]

You can do it, you just need to attach the rule to some common root (here that would be spec), rather than on gatewayClassName

[mikemorris]

It does make sense that the class needs to be managed centrally (by the parent Gateway administrator), butspec.gatewayClassName is a required field on Gateway and while the field doesn't appear to actually be immutable, it's recommended that the specified GatewayClass be used as a template when creating a Gateway, so while it's a bit of extra boilerplate I don't expect it's something that should ever be changing dynamically.

What we may want to add is a clarification that about the GatewayClass "template snapshotting" recommendation - this should likely be tied to the parent Gateway rather than attempting to reconcile potentially drifting GatewayClass configuration snapshots from multiple child Gateways created at different points in time.

[louiscryan]

While I get the templating recommendation in principle I think it presents a toil problem for users when you attempt to apply it to the specification of Gateway.gatewayClassName.

Since the compositional model for routes & policies is back-reference the toil fanout of forcing Gateway immutability is high on users. In this situation any class transition (infrastructure, version etc.) would force a reference update on every inbound policy since the user would have to create a new Gateway resource.

Even immutability is somewhat of an illusion since the recommendation can't prevent the deletion & recreation of the resource with the same name and a different value. The user is simply left with the side-effects of their workaround.

[mikemorris]

For clarity, I would be in favor of allowing an empty gatewayClassName contingent on attachTo being configured if that's permissible - I just think it's a nicety rather than a necessity because changing a gatewayClassName will often be a destructive action and as such not a significantly different, safer, or less downtime operation than deleting and recreating all child and parent resources.

[sunjayBhatia]

Left a few comments, there may be a few more status edge cases I've missed and other comments will cover but overall I think this sounds like it would be useful for a lot of implementations (and implementable) so I'm a fan of getting this GEP in!

[keithmattix]

LGTM - will continue conversation in the other threads

[robscott]

Thanks @dprotaso! Sorry for the delay in getting to this review. Generally appreciate that so many people have shown support for this GEP and that the overall concept of merging Gateways is already widely implemented.

I've got some concerns about the risks to safety here and the overall complexity this introduces to the API. Specifically the idea of a complexity budget comes to mind, and this specific area feels like it could end up having a significant cost when considering all the various knock-on effects like policy attachment, conflict resolution, cross-namespace merging, etc.

With all of that said, I recognize that Gateway merging is already very common, and I would like to see something like this work, I just want to make sure we can find sufficient guardrails to keep the complexity and safety costs down.

[robscott]

Some other potential alternatives that are worth covering here:

• Encouraging users to not use overly large Gateways to minimize the blast radius of any issues
• Increase the number of max listeners that Gateways can support
• Move more of this logic into Routes since those already support ~infinite merging with Gateways

I'm not necessarily recommending any of these, but it would be good to write them out along with why they would or would not work.

[dprotaso]

Move more of this logic into Routes since those already support ~infinite merging with Gateways

I was under the impression this was off the table - but if it's on the table that could simplify Knative's use case tremendously.

[robscott]

For this and other alternatives it would be helpful to have some information about why we chose not to use this specific alternative.

[robscott]

Agree with @louiscryan here. I think it's important to note this tradeoff in the GEP. Even implementations that didn't support Gateway merging would still need to be aware of this potential configuration and ensure that it didn't break them. If we move forward with this approach, we'll need to take extra efforts to communicate the disruptive nature of this change.

[robscott]

Outside of the generated listeners use case, it seems like a primary use case here would be to delegate domains to different namespaces. Unfortunately this doesn't provide any kind of controls for that so it seems like the first namespace to claim a domain would win. I'm concerned that this would become rather chaotic and confusing in practice. A key idea of Gateway listeners was that they allowed you to delegate domains to routes in different namespaces, this seems to make that distinction less clear.

Are there any controls/guardrails we can offer here? Is cross-namespace delegation a key goal? If not, could/should we start without it?

[robscott]

How would you enforce this? In the event of conflicts, which one wins?

[robscott]

A lot of these statements could be enforced with CEL, it would be helpful to rephrase them to state that these checks will be enforced that way.

[robscott]

I'd say a count is probably the least bad option here, but this thread does highlight the overall complexity this GEP would be adding.

[robscott]

One detail I'm missing here - why do we need to have so many listeners attached to a single VIP/Gateway? Is there some kind of number where it makes sense to just have more than one Gateway and/or push some of this complexity down to Routes?

[bravenut]

One argument would be our use case (c.f. #2869). We have an edge gateway acting as a load balancer delegating requests to multiple inner gateways, where a different set of EnvoyFilters is applied to each inner gateway. We will need to support at least >100 listeners on the edge gateway in the future. We will be hitting the limit of 64 listeners with that approach pretty soon.

[dprotaso]

@bravenut curious in your use case what are you configuring for each listener? Also how are you delegating the requests?

[dprotaso]

@bravenut just following up again to my earlier question

[dprotaso]

Is there some kind of number where it makes sense to just have more than one Gateway and/or push some of this complexity down to Routes?

@robscott Introducing more gateways (thus more IPs and underlying infra) incurs a cost and it's not reasonable for some customers.

[mikemorris]

We didn't really touch on it in the call today but another use case here is for managed providers who may want to consolidate separate logical gateways onto multi-tenant infrastructure for efficiency reasons.

[bravenut]

@dprotaso We delegate requests based on the host name on the edge gateway. Then there's one HTTP route per inner gateway applied to the edge gateway routing requests to the inner gateways based on host name + path prefix.

Example scenario: One edge gateway, 3 inner gateways.
The edge gateway has two host names configured, say some.domain & some.other.domain.

Then there's three HTTP routes applied to the edge gateway:
HTTPRoute 1: hostname "some.domain", pathPrefix: "/v1", targeting: service of inner gateway "some-domain-v1"
HTTPRoute 2: hostname "some.domain", pathPrefix: "/v2", targeting: service of inner gateway "some-domain-v2"
HTTPRoute 3: hostname "some.other.domain", pathPrefix: "/v1", targeting: service of inner gateway "some-other-domain-v3"

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dprotaso, howardjohn, keithmattix, mikemorris, mlavacca, sunjayBhatia
Once this PR has been reviewed and has the lgtm label, please assign youngnick for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• geps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[interpixelstudios]

How would I handle SSL on a Kubernetes Gateway for multiple clients? My gateway currently handles HTTP traffic for [example] 20 different domains, using HTTPRoutes. On my gateway, HTTP would realistically just forward to HTTPS. No additional listeners, just the two.

However, on reading how to incorporate SSL into the gateway, I cannot quite figure it out. On my old Ingress Controller [HAProxy], I could just tie the cert-manager issuer to the ingress resource itself and it would issue and enforce the cert. When reading about Gateways and their deployment, it seems like you only attach SSL to the Gateway itself. Wouldn’t that lock down the SSL encompassment to only the domains defined in the cert? What would happen if I need to add domain 21, wouldn’t it make more sense to be able to attach an issuer to the HTTPRoute itself and have it resolve SSL in there? I’m just very confused on how to migrate my existing stack from Ingress to Gateway.

My Class is gke-l7-global-external-managed-mc, thanks!