Add support for Gateway API within Cilium

[jValdron]

I'm looking to add support for gateway API, mostly for Cilium, but there's no way to manage Cilium myself in kOps (AFAIK).

As @hakman mentioned below, it sounds like we're not quite ready to bundle the Gateway API CRDs with kOps yet, so this will require the CRDs to be deployed either manually or through a custom addon. This isn't that big of an issue, even if you enable gateway API in Cilium without them, as it will simply complain that the CRDs can't be found but keep going.

I've added a flag to Cilium to enable Gateway API support (configures it through a change in the config map and adds RBAC):

spec:
  networking:
    cilium:
      gatewayAPI:
        enabled: true

I have tested the new feature using https://kops.sigs.k8s.io/contributing/adding_a_feature/#testing

Cilium status seems to be good after enabling the gateway API under networking and Cilium:

❯ cilium --context $CLUSTER status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 2
                       cilium-operator    Running: 1
Cluster Pods:          5/5 managed by Cilium
Helm chart version:    
Image versions         cilium             quay.io/cilium/cilium:v1.16.7@sha256:294d2432507fed393b26e9fbfacb25c2e37095578cb34dabac7312b66ed0782e: 2
                       cilium-operator    quay.io/cilium/operator:v1.16.7@sha256:bac2496ba4348267ca5f16c2dd73ba7be76330cdd0eef0a6958c260a3bf5951d: 1

The expected RBAC is created and the config map is updated with enable-gateway-api as expected.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: jValdron / name: Jason Valdron (eef68ba)

[k8s-ci-robot]

Hi @jValdron. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[hakman]

/ok-to-test

[hakman]

/retest

[jValdron]

Looks like a retest fixed pull-kops-e2e-aws-upgrade-k130-ko130-to-klatest-kolatest-many-addons, but seems like the e2e tests around Cilium are failing.

I'll try to run them locally to see what happens.

[hakman]

"Pod scheduling timeout." is a known issue with the test infra, no worries.
/retest

[hakman]

@jValdron Instead of spec.networking.gatewayAPIEnabled: true, could you use custom addons. We had a chat about this in our last community meeting and the maintainers are not yet ready to add the Gateway API manifest as a dependency.

[jValdron]

@jValdron Instead of spec.networking.gatewayAPIEnabled: true, could you use custom addons. We had a chat about this in our last community meeting and the maintainers are not yet ready to add the Gateway API manifest as a dependency.

So you're suggesting we keep the flag in Cilium (simply to enable the flag in the config map) but that we require a custom addon to utilize it? I can look into that.

[hakman]

@jValdron Instead of spec.networking.gatewayAPIEnabled: true, could you use custom addons. We had a chat about this in our last community meeting and the maintainers are not yet ready to add the Gateway API manifest as a dependency.

So you're suggesting we keep the flag in Cilium (simply to enable the flag in the config map) but that we require a custom addon to utilize it? I can look into that.

Sounds good, thanks!

[jValdron]

I still have to test the changes on an actual cluster, will try and get that done next week.

[hakman]

I think you have a few docs files that should not be touched and upup/models/cloudup/resources/addons/gateway.networking.k8s.io/k8s-1.32.yaml.template that should be removed.

[jValdron]

I think you have a few docs files that should not be touched and upup/models/cloudup/resources/addons/gateway.networking.k8s.io/k8s-1.32.yaml.template that should be removed.

Docs update are definitely unexpected. Rebasing off the latest master made things much worse :/

Edit: I see what happened, my origin wasn't set proper.

Edit 2: Should be all fixed up.

[hakman]

/retest

[hakman]

/retest

[hakman]

/test pull-kops-e2e-cni-cilium-eni

[hakman]

@jValdron please update the PR description.
/lgtm
/hold for squashing the commits

[jValdron]

Alright, PR title/description updated and squashed the commits into a single commit. Thanks for all the reviews, etc :)

[hakman]

/hold cancel
/lgtm
/approve
/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

Alright, PR title/description updated and squashed the commits into a single commit. Thanks for all the reviews, etc :)

Awesome, thanks @jValdron!

[jValdron]

/retest

[hakman]

/retest