How About Graduating the Gateway API Support to GA?

[maelvls]

Support for the Gateway API was added as an "experimental" feature in 1.5. Since 1.14, cert-manager supports v1 Gateway API.

It seems like a good time to start thinking about make this feature generally available, and it seems like there aren't any notable changes that we would like to make before going GA.

Beta then GA, or GA directly?

The feature gate ExperimentalGatewayAPISupport says that it is an "alpha" feature in features.go.

I'd suggest graduating it to "beta" in 1.15 and turn it on by default in 1.16. Having the term "experimental" in the feature flag makes this process somewhat annoying, since it will sound like this feature is experimental forever... But I'd prefer we keep backwards compatibility with existing cert-manager installations by adding a new GatewayAPISupport flag that does the same as the old one, and we graduate both flags to "beta".

I propose that we discuss that Thursday 4th April during the dev biweekly meeting.

/kind feature

[mamachanko]

x-posting key-points from the related Slack 🧵

The docs say there’s limited testing:

🚧 cert-manager 1.14+ is tested with v1 Kubernetes Gateway API. It should also work with v1beta1 because of resource conversion, but has not been tested with it.

When the feature is toggled the controller won’t start unless the Gateway APIs are present.

[hawksight]

A little feedback from the bi-weekly call yesterday. We have not set a date for 1.15 cert-manager because we want to discuss whether GA for GatewayAPI support is part of that release. There was also a related discussion on release cadence, for another thread.

There will be an open discussion on what GA for GatewayAPI looks like for cert-manager. I'll follow up next week once this is pencilled in. Please make it known if you'd like to be part of that discussion.

One idea I proposed is that we:

1. Make what we have now GA.
2. Remove the feature flag / add feature flag / do something around the CRDs not being present in clusters.
3. Look at a new, "better named" feature flag to have more support around different Route resources, which I think is described more here: https://github.com/cert-manager/cert-manager/blob/master/design/20230601.gateway-route-hostnames.md. Looks like someone has some work around this already here: master...tommie:cert-manager:httproute

[mamachanko]

I would like to add another option for transitioning into out-of-the-box support of Gateway APIs; dynamic reconfiguration.

Instead of - or maybe in addition to - controlling Gateway API support through feature gates, cert-manager's components could inspect the available APIs via the discovery client. If gateway.networking.k8s.io is present, then support it but let explicit user configuration take precedence.

In theory it's simple. cert-manager supports Gateway APIs when they are present, otherwise not. But in practise it's complicated. Is gateway.networking.k8s.io installed after or before cert-manager? Worse, maybe it's installed while cert-manager rolls out and some of its components see it at start-up others don't. Maybe components continuously look for gateway.networking.k8s.io and restart themselves when it appears. Yuck!

One of cert-manager's superpowers is its stability. This approach may joepardize it.

I am only reluctantly suggesting this because reconfiguration at runtime has many perils, but it might be worth considering on the path to a world paved with gateways.

[hawksight]

Summarising our plan of action from today's call:

Scope

The following relates only to the existing cert-manager GatewayAPI support enabled through ExperimentalGatewayAPISupport feature flag.

Intent

ExperimentalGatewayAPISupport is currently alpha status for cert-manager. We will make this a "beta" feature flag.

This means the feature will be "on" by default. This signals our intent that what we have right now, is ready for mainstream usage and we are actively looking for feedback from the wider community before graduating to GA. Full definition from website:

Beta: feature is almost stable but might still change in the future. Beta features are enabled by default and can be disabled by the user (if any issues are encountered).

Issue(s)

GatewayAPI CRDs Not Found

Moving this feature to "on" by default would be problematic for many cert-manager users who do not have the relevant CRDs installed in their environment. This is unique as other feature flagged code generally does not require third part Custom Resource Definitions (CRDs). For this reason we will implement a new configuration options / command line parameter, that you must set in order to enable cert-manager to work on Gateway resources.

We decided against being clever and "detecting" the Gateway CRDs as this would introduce the need to watch and reconcile the presence or absence of those CRDs over time, and potentially introduce many more issues trying to resolve what was present or not at any given time.

It seems most sensible to enable this support when you are confident those CRDs are present in your cluster. You can resolve dependency order using your installation tool(s) of choice, just as you would for anything else.

User Experience

1. Those who do not currently use GatewayAPI. No change. It will be off by default.
2. Those who do use GatewayAPI currently. Will need to set a new config parameter.
3. Those who do not currently use GatewayAPI but enable it in the future, AND set all helm values explicitly. Set new parameter and ensure to update ExperimentalGatewayAPISupport option to true when upgrading cert-manager, if this is currently set to false

Timeline

We aim to include this change in v1.15 of cert-manager.

Potential Future Timeline

We will assess how this goes in 1.15 as a beta feature. We will look to making this core / GA in cert-manager in later versions.

Enhanced features around GatewayAPI

For expanded or new features relating to GatewayAPI we will continue to assess them on a case by case basis, just like any new feature enhancement.

If it proves to be a big / risky change, new functionality may go behind another feature flag in the future.

Smaller changes may just be implemented in regular release cadence.

Feedback / Questions

We will give a summary of this discussion on the next cert-manager bi-weekly call, Thursday 18th April 2024. That call will be recorded for anyone interested.

If you have q's or concerns or want to help drive this feature set, please join that call or feedback on this thread. Or of course, join us in slack.