KongUpstreamPolicy controller depends on HTTPRoute CRD presence

[carlosrmendes]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

After upgrading to version 3.1, I'm receiving errors while validating KongUpstreamPolicies with the following error:
2024-02-28T09:24:23Z error failed fetching KongUpstreamPolicy: KongUpstreamPolicy mynamespace/kong-health-check not found {"name": "myservice", "namespace": "mynamespace", "GVK": "/v1, Kind=Service", "error": "resource processing failed"}

Expected Behavior

Existing KongUpstreamPolicies should be fetched correctly by the kong ingress-controller

Steps To Reproduce

1. update to kong kubernetes-ingress-controller v3.1
2. create a KongUpstreamPolicy
3. reference that KongUpstreamPolicy in an existing Kubernetes Service (same namespace) with the `konghq.com/upstream-policy: <service_name>` annotation

Kong Ingress Controller version

v3.1

Kubernetes version

v1.28

Anything else?

No response

[chlislb]

I met the same issue.

[girishadat]

Our deployments are also affected by this and hence stuck @ Kong 3.5. This specific version of Kong has a slightly problematic dns_no_sync flag value for a Kubernetes-like environment where the upstream can fluctuate very much with HPA.

[randmonkey]

@carlosrmendes I tried to reproduce the problem with your steps (install kong/ingress with KIC 3.0 and Kong 3.5 then upgrade to KIC 3.1 and Kong 3.6), but I cannot reproduce the issue. Could you please provide more details about the environment, including:

• How did you upgrade you KIC and Kong? What is your helm command and the used values?
• How did you create your kongupstreampolicy and service? It is better to know the detailed manifests.

[dschniepp]

@randmonkey Sorry for jumping in @skybalsamoan created another issue with the full description on how to replicate the issue Kong/kong#12661.

[randmonkey]

Got the reason why KIC3.1 has the problem:
KIC 3.1 added updating of ancestor status of KongUpstreamPolicy, and in the change the controller of KongUpstreamPolicy added the dependency of HTTPRoute CRD in gateway APIs. If HTTPRoute does not exist, the KongUpstreamPolicy controller will not start. The issue will be fixed when you install CRDs in gateway APIs.
@carlosrmendes @dschniepp Can you try installing gateway API CRDs in standard channel by

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

?

[carlosrmendes]

after that apply I got this error on KIC:

E0321 15:48:46.062059       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1.HTTPRoute: failed to list *v1.HTTPRoute: httproutes.gateway.networking.k8s.io is forbidden: User "system:serviceaccount:kelvin:kong-gateway" cannot list resource "httproutes" in API group "gateway.networking.k8s.io" at the cluster scope

[carlosrmendes]

I've installed and updated the kong gateway and kic using the kong helm chart

[randmonkey]

@carlosrmendes Can you try install gateway CRDs first (apply the yaml) then upgrade/install KIC bye helm?
The charts will only install RBAC rules for accessing resources in gateway APIs in the ClusterRole when the CRDs in gateway APIs exists in the cluster.

[mflendrich]

Changed the title of this issue to pinpoint the reason as identified by @randmonkey.

The workaround seems to be installing the Gateway API CRDs (pending verification).
The fix should come in a patch release.

[rainest]

This stems from #5444, which feels pretty non-critical. Ancestor status info is nice, but we can probably live without it. IMO supporting existing Kong functionality on the (probably, at this point) majority of clusters that lack GWAPI CRDs outweighs providing the additional status info. I don't know of an obvious reason we'd need to trigger reconciles outside that, since all the actual application of the upstream config happens in the translators.

Can we gate the required HTTPRoute CRD in the KUP controller definition and additional watch using a feature flag, and default it off pending wider availability of GWAPI CRDs on major providers? IDK if we have a clear use case that requires surfacing ancestor status yet (other than it being part of the policy attachment design spec), so it seems a reasonable candidate for something you can turn on if you want it.

---

The specific trigger mentioned (reconciling based on Services used by an HTTPRoute that have upstream configuration) sounds a bit odd in itself (shouldn't the Service be the ancestor in that case? the KUP settings that affect Services don't affect HTTPRoutes), but ultimately you can also apply KUP route config to HTTPRoutes directly, so I assume that relationship would exist regardless.

[tongsama]

@randmonkey
I was also experiencing the problem of not being able to upgrade from chart version: 0.10.2.
Has KIC fixed the problem?
I verified today with KIC-3.1.3 image and the problem occurred as before.
It doesn't seem to be the CRD dependency issue in the title.

For simplicity I will describe the reproduction procedure with k3s.
(The same error occurs in the operational k8s environment)

Is there anything wrong with my procedures?

k3s(single server cluster) setup with k3d

• k3d-config.yaml

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: kongCheck
servers: 1
agents: 0
kubeAPI:
  host: "0.0.0.0"
  hostIP: "0.0.0.0"
  hostPort: "6443"
image: rancher/k3s:v1.29.3-k3s1
subnet: "10.32.0.0/16"
token: superSecretToken
options:
  k3s:
    extraArgs:
      - arg: "--tls-san=127.0.0.1"
        nodeFilters:
          - server:*
      - arg: "--disable-network-policy"
        nodeFilters:
          - server:*
      - arg: "--disable=traefik"
        nodeFilters:
          - server:*
  runtime: # runtime (docker) specific options
    ulimits:
      - name: nofile
        soft: 26677
        hard: 26677
      - name: nproc
        soft: 26677
        hard: 26677

• setting up cluster

k3d cluster create --config	k3d-config.yaml

apply gatewayapi

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

install kong chart (Kong:3.4 KIC:3.0)

• values.yaml

controller:
  ingressController:
    enabled: false
gateway:
  admin:
    http:
      enabled: true
  manager:
    type: ClusterIP
  ingressController:
    enabled: true
  env:
    role: traditional
    database: "off"

• install kong

helm repo add kong https://charts.konghq.com

helm upgrade --install \
kong kong/ingress \
-n kong --create-namespace \
--version 0.10.2 -f values.yaml

check status

• wait for pod running

kubectl get pod -n kong
NAME                           READY   STATUS    RESTARTS   AGE
kong-gateway-698fbc76f-kfwtz   2/2     Running   0          55s

kubectl get svc/kong-gateway-proxy -n kong
NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
kong-gateway-proxy   LoadBalancer   10.43.77.241   10.32.0.3     80:31292/TCP,443:31920/TCP   3m19s

create gatwayclass&gateway

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: 'true'
spec:
  controllerName: konghq.com/kic-gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong-gateway
  namespace: kong
spec:
  gatewayClassName: kong

  listeners:
  - name: http-proxy
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
EOF

apply application

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: echoapp
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    konghq.com/upstream-policy: consistent-hashing
  labels:
    app: http-echoserver
    service: http-echo-1
  name: http-echo
  namespace: echoapp
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: http-echoserver
    version: v1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: http-echoserver
    version: v1
  name: http-echo
  namespace: echoapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-echoserver
      version: v1
  template:
    metadata:
      labels:
        app: http-echoserver
        version: v1
    spec:
      containers:
      - image: registry.k8s.io/echoserver:1.4
        imagePullPolicy: IfNotPresent
        name: http-echoserver
        ports:
        - containerPort: 8080
EOF

apply KongUpstreamPolicy(consistent-hashign) & httproute

cat <<EOF | kubectl apply -f -
apiVersion: configuration.konghq.com/v1beta1
kind: KongUpstreamPolicy
metadata:
  name: consistent-hashing
  namespace: echoapp
spec:
  algorithm: consistent-hashing
  hashOn:
    input: ip
  hashOnFallback:
    header: X-Forwarded-For
  healthchecks:
    active:
      healthy:
        interval: 5
        successes: 3
      httpPath: /
      type: http
      unhealthy:
        httpFailures: 1
        interval: 5
    passive:
      healthy:
        successes: 3
      unhealthy:
        httpFailures: 3
  slots: 500
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  annotations:
    konghq.com/strip-path: "true"
  name: echo
  namespace: echoapp
spec:
  parentRefs:
  - name: kong-gateway
    namespace: kong
    sectionName: http-proxy
  rules:
  - backendRefs:
    - kind: Service
      name: http-echo
      port: 8080
    matches:
    - path:
        type: PathPrefix
        value: /
EOF

check logs

no issue.

kubectl logs deployment/kong-gateway -n kong |grep -i upstreampolicy

Defaulted container "ingress-controller" out of: ingress-controller, proxy, clear-stale-pid (init)
2024-04-25T07:54:39Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.KongUpstreamPolicy"}
2024-04-25T07:54:39Z    info    controllers.KongUpstreamPolicy  Starting Controller     {"v": 0}
2024-04-25T07:54:39Z    info    controllers.KongUpstreamPolicy  Starting workers        {"v": 0, "worker count": 1}

check resources

no issue.

kubectl port-forward -nkong svc/kong-gateway-admin  8001:8001 &

curl -s localhost:8001/upstreams/ | jq '.data[] | .algorithm'
Handling connection for 8001
"consistent-hashing"

Upgrade kong chart (Kong:3.6 KIC:3.1)

helm upgrade --install \
kong kong/ingress \
-n kong --create-namespace \
--version 0.12.0 -f values.yaml

check status

• wait for rollingout

kubectl get pod -n kong

NAME                            READY   STATUS     RESTARTS   AGE
kong-gateway-698fbc76f-kfwtz    2/2     Running    0          6m55s
kong-gateway-5d6dfb78f7-nk5dn   0/2     Init:0/1   0          9s

NAME                            READY   STATUS    RESTARTS   AGE
kong-gateway-5d6dfb78f7-nk5dn   2/2     Running   0          62s

check logs

• Got errors
  KongUpstreamPolicy controller seems to be started, but it cannot get Policy

kubectl logs deployment/kong-gateway -n kong |grep -i upstreampolicy

Defaulted container "ingress-controller" out of: ingress-controller, proxy, clear-stale-pid (init)
2024-04-25T08:03:23Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1.Service"}
2024-04-25T08:03:23Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1.HTTPRoute"}
2024-04-25T08:03:23Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.KongUpstreamPolicy"}
2024-04-25T08:03:23Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "channel source: 0xc000965280"}
2024-04-25T08:03:23Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "channel source: 0xc0009652c0"}
2024-04-25T08:03:23Z    info    controllers.KongUpstreamPolicy  Starting Controller     {"v": 0}
2024-04-25T08:03:23Z    info    controllers.KongUpstreamPolicy  Starting workers        {"v": 0, "worker count": 1}
2024-04-25T08:03:31Z    error   failed fetching KongUpstreamPolicy: KongUpstreamPolicy echoapp/consistent-hashing not found     {"name": "http-echo", "namespace": "echoapp", "GVK": "/v1, Kind=Service", "error": "resource processing failed"}
2024-04-25T08:03:31Z    error   recording a Warning event for object    {"name": "http-echo", "namespace": "echoapp", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy echoapp/consistent-hashing not found", "error": "object failed to apply"}
2024-04-25T08:03:34Z    error   failed fetching KongUpstreamPolicy: KongUpstreamPolicy echoapp/consistent-hashing not found     {"name": "http-echo", "namespace": "echoapp", "GVK": "/v1, Kind=Service", "error": "resource processing failed"}
2024-04-25T08:03:34Z    error   recording a Warning event for object    {"name": "http-echo", "namespace": "echoapp", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy echoapp/consistent-hashing not found", "error": "object failed to apply"}

check resources

Policy is not applied

kubectl port-forward -nkong svc/kong-gateway-admin  8001:8001 &

curl -s localhost:8001/upstreams/ | jq '.data[] | .algorithm'

Handling connection for 8001
"round-robin"

cleanup

k3d cluster delete --config	k3d-config.yaml

Perhaps the problem is not caused by an upgrade, but a new installation will have the same problem.

I rewrite the KIC image tag to 3.0 on the 0.12.0 chart, the error no longer occurs and it works correctly.

Thanks.