CFP: Support TCPRoute and UDPRoute in Cilium Gateway API

[sayboras]

Cilium Feature Proposal

The first cut for Gateway API was done in #21749, however, we have a couple of follow-up tasks to support more features from Gateway API upstream.

This is to track the work to support TCPRoute and UDPRoute

Goals

• Support TCPRoute and UDPRoute whereever Cilium is used

Background

When talking about supporting TCPRoute and UDPRoute in Cilium, there are some important things to note:

• The functionality is basically equivalent to creating a Loadbalancer type Service
• Because of that, theoretically, Cilium's LB-IPAM feature, combined with the BGP Control Plane feature, could be used to handle Gateway address allocation as well, and then route traffic as directed with TCPRoute and UDPRoute.
• Current Gateway API support (for HTTPRoute and soon, TLSRoute) is implemented by having the Cilium Operator provision a Loadbalancer Service (in shared LB mode, or one per Gateway in dedicated mode) that points to the per-node proxy deployment that Cilium uses for Layer 7 routing.

Because of this, if we build support for TCPRoute and UDPRoute Gateways using Loadbalancer Services, then we may conceivably need to do a migration in the future for folks who are using LB-IPAM. We can make this migration possible using the ownerRefs that we already apply to created Service objects.

Because of this, we'll look at Gateway API interactions with LB-IPAM later on.

Proposed Solution

For now, this CFP proposes creating a Loadbalancer Service per Gateway that has TCPRoutes and/or UDPRoutes attached as a first cut.

We will look at if we can collapse multiple Gateways onto a smaller set of Loadbalancer Services, but it's much more likely that conflicts will arise with the Layer 4 constructs than with Layer 7 constructs. kubernetes-sigs/gateway-api#1863 will probably end up being relevant here, as we may be able to use those mechanisms to cut down on the number of looad balancers created.

Cilium will not support having a single Gateway that mixes Layer 4 constructs like TCPRoute and UDPRoute with higher-layer constructs like HTTPRoute, TLSRoute, or GRPCRoute. Such Gateways will be marked as incompatible using Gateway API's standard methods and not accepted for processing.

Listener Ports will be translated to ports in the Loadbalancer Service, with the backends set as specified in the associated TCPRoute or UDPRoute.

Since the spec of both TCPRoute and UDPRoute is quite straightforward, that should be sufficient to get us started.

Tasks

Exact tasks TBD, needs a code review to get some idea of what the implementation will need to look like.

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[aojea]

/cc

[youngnick]

#9207 may also be relevant here, because if a Gateway has UDPRoutes and TCPRoutes attached to the same Gateway on the same port number (could be a different Listener though), then #9207 may be relevant.

Going to do some testing today to check.

[aojea]

#9207 may also be relevant here, because if a Gateway has UDPRoutes and TCPRoutes attached to the same Gateway on the same port number (could be a different Listener though), then #9207 may be relevant.

The interesting thing with #9207 is that most people (like me) assumes that both protocols will not work at the same time, but it seems that the effect is that it will ALWAYS forward both protocols, despite only one is configured, see e2e test reproducing the issue kubernetes/kubernetes#116333

Just a heads up, since this can present a network security problem, we are going to discuss next sig-network meeting to promote this test to Conformance https://docs.google.com/document/d/1_w77-zG_Xj0zYvEMfQZTQ-wPP4kXkpGD8smVtW_qqWM/edit

[Sammcb]

Hi all! I was hoping to use a TCPRoute with Cilium, and was curious if there was a potential rough timeline for this. Will this depend on these Routes reaching beta phase in gateway-api? Really excited for all these new options, thanks so much!

[eminaktas]

/cc

[a1994sc]

Is there any chance this will be included in the v1.5.0 release? Thank you for your hard work!

[danieljkemp]

Cilium will not support having a single Gateway that mixes Layer 4 constructs like TCPRoute and UDPRoute with higher-layer constructs like HTTPRoute, TLSRoute, or GRPCRoute. Such Gateways will be marked as incompatible using Gateway API's standard methods and not accepted for processing.

Is this not planned on the initial implementation, or are there technical limitations of Cilium that prevent mixing TCPRoute/UDPRoute with higher layers?

[youngnick]

@danieljkemp there are two reasons:

• it's not really the intended way to use a Gateway. You shouldn't be afraid of having multiple Gateways, and keeping the separate use cases separate.
• it makes the implementation a lot more complex to have to figure out which ports need to be redirected to Envoy, and which should be handled by the datapath directly. Will we never do this? It's possible we might, but the extra complexity cost would need to be worth the tradeoff, and I currently don't see that.

[danieljkemp]

The main use case that I would currently have is for stacking the Gitlab SSH port on the same IP as the ingress/tlsroute gateway. Managing a secondary hostname for gitlab's ssh is not nearly as clean of a setup. I am sure there are other use cases out there, that is just the biggest one I've personally encountered, and it isn't a deal-breaker. I am currently using the tcpproxy feature of nginix-ingress to do this, but could go back to a split hostname setup if needed, and I don't think I need tcproute in that case. (I could also just disable gitlab ssh and tell users to use only https haha)

[rouke-broersma]

The main use case that I would currently have is for stacking the Gitlab SSH port on the same IP as the ingress/tlsroute gateway. Managing a secondary hostname for gitlab's ssh is not nearly as clean of a setup. I am sure there are other use cases out there, that is just the biggest one I've personally encountered, and it isn't a deal-breaker. I am currently using the tcpproxy feature of nginix-ingress to do this, but could go back to a split hostname setup if needed, and I don't think I need tcproute in that case. (I could also just disable gitlab ssh and tell users to use only https haha)

I think in that case you would have two gateways but the same public IP.

[igor-borisoglebski]

The main use case I have for this is to be able to use the Proxy Protocol which if I'm not mistaken is only supported for ingress and api gw. Please correct me if I'm wrong

[ToroNZ]

The main use case I have for this is to be able to use the Proxy Protocol which if I'm not mistaken is only supported for ingress and api gw. Please correct me if I'm wrong

I've replied to you here on this same subject: #30089 (comment)