Is it possible to add additional Envoy configs to the Gateway API managed resources(Ingress controller)?

[heruscode]

I’m trying to understand if its possible to add additional Envoy configs to the Gateway API managed resources(Ingress controller).

I currently have GKE managed cluster, with Cilium enabled ingress controller and I’m using Gateway API resources, I wanted to add some custom Envoy configs that Gateway API doesn’t seem to support(like adding some custom URL Rewriting and Direct Responses)

I tried to follow the official doc examples, but it only seems to affect the traffic within the service-mesh(a Service talking to another Service). What I’m trying to achieve is to customize Ingress traffic(traffic that is coming from real users on the internet)

I see that the Gateway resource auto creates/manages a CiliumEnvoyConfig/cilium-gateway-cilium-gateway that is responsible for the ingress service. Manually editing that manifests would be a no-go, I tried to create a secondary CiliumEnvoyConfig with a new listener pointing to the same ingress service but I had no success:

level=warning msg="Failed to redirect K8s services to Envoy" ciliumClusterwideEnvoyConfigName=cilium-extra error="Service \"dev2-cilium/cilium-gateway-cilium-gateway\" already registered for L7 LB redirection via CiliumEnvoyConfig \"dev2-cilium/cilium-gateway-cilium-gateway\""

Is there anyway to append Envoy configs to the GatewayAPI managed resources?

[youngnick]

Hi @heruscode, thanks for this question.

Currently, there is no way to add extra Envoy config to Gateway API objects - we're hoping to build out this functionality in Cilium's Gateway API support instead.

Could you add a bit more detail about what URL Rewriting and Direct Response config you'd like? We may then be able to turn this into a feature request. :)

[heruscode]

I have a use case for controlling maintenance pages at ingress level, for example, when I want to turn on maintenance-mode for some service, I can use a nginx config to return a direct response with an static html page on all paths or 503 error for api calls.

Would also be useful to be able to rate-limit requests coming from the internet or doing some url rewrite to mutate the request before it reaches the backend

[youngnick]

Okay, so the two requests are:

• direct response
• rate limits

Is that correct?

[heruscode]

correct

[youngnick]

We're hoping to add both of these things via Gateway API, but we need to define the language in upstream for them first.

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[youngnick]

I'm counting this as a feature request for Gateway API to be able to handle the two features above:

• direct response
• rate limits

These are definitely on the roadmap upstream, so we should be able to talk more about them post-GA for Gateway API.

[eminaktas]

@youngnick, thank you for your interest in this request. It would be great to have extending the Gateway API with custom Envoy configs. I can do it with manipulating the auto-generated envoy config but it is not convenient method. I want to add an oauth2 flow in the Gateway API.

Is this feature planned?

[larssb]

There IS a workaround. By no means b e a utiful ... However, here it is:

1. Create your Gateway API gateway instance, httproute and so on
2. apply your custom CiliumEnvoyConfig to the cluster. One that listens on the Gateway API gateway instance you just created
3. Now, in order to get rid of the default auto generated CilimEnvoyConfig for the Gateway API gateway instance you need to:
   1. Delete the cilium Gateway Class from the cluster
   2. Then delete the default auto generated CiliumEnvoyConfig
4. Now Cilium won't auto generate a default CiliumEnvoyConfig covering your Gateway API gateway instance

N.B. this is a rotten but working workaround. Rotten because:

• to create working gateway api gateway instances in the future you need to:

1. Apply a working kind: GatewayClass by the name cilium ( have a backup of the default Cilium created cilium Kind: GatewayClass
2. Create your new Gateway API gateway instance HTTPRoute and so on
3. when the Gateway API gateway instance is fully instantiated delete the Cilium GatewayClass
4. Now delete the auto generated CiliumEnvoyConfig For all Gateway API gateway instances on the cluster.

---

N.B. it should also be perfectly possible to wait with creating you custom CiliumEnvoyConfig until after you:

• deleted the auto generated CiliumEnvoyConfig
• and then the cilium GatewayClass

Enjoy 🥇

[vladimirfx]

IMO parametersRef in GatewayClass is provided for exactly such cases. Gateway API evolving slowly (good thing) but implementors should provide advanced features now.
Envoy Gateway implementation provides an ability to attach custom envoy config (https://gateway.envoyproxy.io/v0.6.0/user/customize-envoyproxy/#add-gatewayclass-parametersref).

It would be nice to have analogical functionality in Cillium Gateway API impl.