How should non-SNI traffic, such as kubernetes.default.svc, be handled when exposing kube-apiserver via Gateway API?

[hkajikawa]

Hi there,

I am currently validating a configuration to expose the kube-apiserver using the Gateway API.
I have set the Gateway's IP address in spec.networkProfile.advertiseAddress. This IP address corresponds to the kind: Gateway hostname configured as spec.controlPlane.gateway.hostname: *.cluster.dev.

The issue I am facing is related to kubernetes.default.svc, which is also discussed here.
Because traffic from Pods to kubernetes.default.svc is IP-address-based rather than SNI-based, they fail to connect to the Control Plane.
How can I address this issue? If there are any common solutions or workarounds, please let me know.
Currently, even the initially deployed CoreDNS cannot connect to the Control Plane, which effectively leaves me stuck and unable to proceed with anything.

I have confirmed a working workaround by setting --endpoint-reconciler-type=none on the kube-apiserver to disable the automatic creation of EndpointSlices for the kubernetes service, and then rewriting the kubernetes service into an ExternalName Service like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
spec:
  externalName: tcp1.cluster.dev
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 7443
  sessionAffinity: None
  type: ExternalName

While this is one potential approach, rewriting it to an ExternalName Service prevents the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables from being injected into Pods. Consequently, any Pods that rely on these environment variables (which are quite common, including CoreDNS) will fail to function.

What other approaches or solutions could be considered to resolve this issue?

[prometherion]

How can I address this issue? If there are any common solutions or workarounds, please let me know

We're solving that with our commercial offering, with an SNI-based solution: https://kamaji.clastix.io/enterprise-addons/

It's currently based on Ingress, but it can be ported to the Gateway API: get in touch with CLASTIX if your business is up for such a commitment.

[hkajikawa]

Thank you for your reply. So it is possible in the production version.

[prometherion]

Yes, as we explain on the Kamaji documentation website, all the stable releases are released "AS IS", meaning we're not providing backports.

To help the company in developing and maintaining Kamaji, we offer a Stable release channel: entirely based on upstream, with support, feature prioritisation, and the availability of commercial addons.

Speaking of the Ingress Addon, it's currently used in production by some of our customers, and it plays a role in our managed Kubernetes offering for OpenStack.