Wildcard ingress routing error

[michael-riha]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When applying an Ingress with a namespace-Issuer I get this Error in the logs

time="2023-05-24T12:47:25Z" level=error msg="could not update kong admin" error="posting new config to /config: HTTP status 400 (message: "declarative config is invalid: {services={[4]={routes={{name=\"invalid value 'basic-www.wildcard-ingress.basic-www.*.k8s.domain.com.80': the only accepted ascii characters are alphanumerics or ., -, _, and ~\"}}}}}")" subsystem=dataplane-synchronizer

Issuer

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-wildcard
  namespace: basic-www
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: email@domain.com
    privateKeySecretRef:
      name: letsencrypt-wildcard
    solvers:
    - selector:
        dnsZones:
          - "domain.com"
      dns01:
        route53:
          region: eu-central-1
          accessKeyID: <Key>
          secretAccessKeySecretRef:
            name: prod-route53-credentials-secret
            key: secret-access-key

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wildcard-ingress
  annotations:
    kubernetes.io/ingress.class: kong
    cert-manager.io/issuer: letsencrypt-wildcard
spec:
  tls:
    - hosts:
        - "*.k8s.domain.com"
      secretName: my-tls-secret
  rules:
    - host: "*.k8s.domain.com"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: basic-www
                port:
                  number: 80

Expected Behavior

When I apply provided Ingress any subdomain such as

• something.k8s.domain.com
• whatever.k8s.domain.com
• ...

Should be routed to the Service with SSL/TLS termination from Kong/KIC

...
   backend:
              service:
                name: basic-www
                port:
                  number: 80

Steps To Reproduce

1. deploy Kong

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/v2.8.0/deploy/single/all-in-one-dbless.yaml

2. point DNS record `*.k8s.domain.com` to Kong-Proxy External IP
3. Deploy a simple echo-deployment `basic-www`
4. Deploy the `Issuer` & `Ingress` as shown above
5. watch logs `kubectl logs pod/ingress-kong-<hash> -n kong -c ingress-controller -f`
> time="2023-05-24T12:47:25Z" level=error msg="could not update kong admin" error="posting new config to /config: HTTP status 400 (message: \"declarative config is invalid: {services={[4]={routes={{name=\\\"invalid value 'basic-www.wildcard-ingress.basic-www.*.k8s.domain.com.80': the only accepted ascii characters are alphanumerics or ., -, _, and ~\\\"}}}}}\")" subsystem=dataplane-synchronizer

Kong Ingress Controller version

v2.8.0

Kubernetes version

1.24.0

Anything else?

I saw some wildcard related release notes

https://docs.konghq.com/gateway/3.3.x/how-kong-works/routing-traffic/#using-wildcard-hostnames
Not sure if this will fix it?
We can update if this is producing the error!

But ...

Latest KIC is 2.9.3

• https://github.com/Kong/kubernetes-ingress-controller/tree/v2.9.3
• https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/v2.6.3/deploy/single/all-in-one-dbless.yaml
  • ... image: kong:3.2 ... direct link

Question:

• Is the Error from this Issue to be solved by "the fix" in 3.3.x?
• Is it safe to deploy KIC v2.9.3 with 3.3.xinstead of 3.2.x which seems to be the preferred version?
• Do we have an Error or miss something?

Thank you guys in advance!

[randmonkey]

@michael-riha This is a bug in translating route names in CombinedRoute mode, which is enabled by default in KIC 2.8 and above. This bug has no relationship with Issuer. This is fixed in KIC 2.8.1, and KIC 2.9+: #3311 (fixed in KIC 2.8.1 by PR #3312). You can fix by either option of the following:

• upgrade KIC to 2.8.1/2.8.2/2.9+ including the fix
• configure feature gate CombinedRoutes to false by adding CombinedRoutes=false in env CONTROLLER_FEATURE_GATES of container
• downgrade KIC to 2.7, which disables CombinedRoutes feature gate by default.

[michael-riha]

@randmonkey appreciate your feedback, thank you.

Just to make sure, this is how I would patch the default deploy/single/all-in-one-dbless.yaml for KIC 2.8.1 ++, correct?

...
        - env:
          - name: CONTROLLER_KONG_ADMIN_URL
            value: https://127.0.0.1:8444
          - name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
            value: "true"
          - name: CONTROLLER_PUBLISH_SERVICE
            value: kong/kong-proxy
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: CONTROLLER_FEATURE_GATES
            value: CombinedRoutes=false
          image: kong/kubernetes-ingress-controller:2.8
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          name: ingress-controller
...

So just on the ingress-controller-container, or on others as well?

Sources:

• https://docs.konghq.com/kubernetes-ingress-controller/latest/deployment/install-gateway-apis/#using-alpha-apis

---

⚠️ Amount of containers in Deployment (name: ingress-kong) has changed from 2.8 - 2.9

---

kustomize example for 2.8.1

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/v2.8.1/deploy/single/all-in-one-dbless.yaml

#FIX: https://github.com/Kong/kubernetes-ingress-controller/issues/4067#issuecomment-1562175250
patches:
- patch: |-
    - op: add
      path: "/spec/template/spec/containers/1/env/-" 
      value:
        name: CONTROLLER_FEATURE_GATES
        value: "CombinedRoutes=false"
  target:
    kind: Deployment
    namespace: kong
    name: ingress-kong

kustomize example for 2.9.x

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/v2.9.3/deploy/single/all-in-one-dbless.yaml

#FIX: https://github.com/Kong/kubernetes-ingress-controller/issues/4067#issuecomment-1562175250
patches:
- patch: |-
    - op: add
      path: "/spec/template/spec/containers/0/env/-"
      value:
        name: CONTROLLER_FEATURE_GATES
        value: "CombinedRoutes=false"
  target:
    kind: Deployment
    namespace: kong
    name: ingress-kong

[randmonkey]

@michael-riha You do not need to turn of CombinedRoutes if you are using KIC 2.8.1+ or 2.9+. The bug exists in 2.8.0, and fixed in 2.8.1 and later versions.