Multiple ingress: not stable ipaddress on ingress

[Leopol123]

Hello,

I am testing multiple ingress controllers on one cluster.

My wish:

• one default ingress controller that manage a default ingress class or without ingressclass defined
• at least one other controller that manage a specific ingress class

I read with attention: https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/

I deploy the default ingress controller using helm chart:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  chart: ingress-nginx
  targetNamespace: kube-system
  repo: https://kubernetes.github.io/ingress-nginx/
  version: 4.12.3
  valuesContent: |
    controller:
      replicaCount: 1
      ingressClassResource:
        enabled: true
        default: true
      service:
        type: LoadBalancer

And another controller with his own class:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: ingress-nginx-domain-a
  namespace: kube-system
spec:
  chart: ingress-nginx
  targetNamespace: kube-system
  repo: https://kubernetes.github.io/ingress-nginx/
  version: 4.12.3
  valuesContent: |
    controller:
      replicaCount: 1
      ingressClassResource:
        name: domain-a
        enabled: true
        default: false
        controllerValue: k8s.io/domain-a
      ingressClass: domain-a
      service:
        type: LoadBalancer

Each controller has his own IP deliver by an external load balancer.

Then I create two ingresses:

• one with ingressClassName: nginx
• the second with ingressClassName: domain-a

What you expected to happen:

Each ingress should obtain the ipaddress of the ingress-controller (that will be update in our global DNS using external-dns)

What happened:

Ingresses have the correct IP on first seconds then it quickly flip flap with nodes IP:

kubectl get ingress -n default

NAME       CLASS      HOSTS               ADDRESS         PORTS   AGE
whoami     nginx      whoami.test.lan     192.168.1.192   80      16h
whoami-a   domain-a   whoami-a.test.lan   192.168.1.194   80      16h

NAME       CLASS      HOSTS               ADDRESS                     PORTS   AGE
whoami     nginx      whoami.test.lan     192.168.1.19,192.168.1.20   80      16h
whoami-a   domain-a   whoami-a.test.lan   192.168.1.194               80      16h

192.168.1.192/194 are IP of ingress controllers so a good value.
192.168.1.19/20 are my nodes IP
Only the ingress with class nginx is flipflap.

**What do you think went wrong? **:

Each time the wrong value is set, I have this log on controller domain-A:

W0606 07:47:14.456797       8 controller.go:336] ignoring ingress whoami in default based on annotation : no object matching key "nginx" in local store
I0606 07:47:14.456900       8 main.go:107] "successfully validated configuration, accepting" ingress="default/whoami"

Then after few seconds, the default controller fix the ipaddress to the correct one and it repeat again and again...

NGINX Ingress controller version: 1.12.3 (using 4.12.3 helm chart)

Kubernetes version (use kubectl version): v1.29.15+rke2r1

Environment:

• Cloud provider or hardware configuration: RKE2 using Rancher

• OS (e.g. from /etc/os-release): Rocky9

• How was the ingress-nginx-controller installed: Helm chart (not RKE2 version)

Maybe, I do a wrong config.... Thanks for your help !

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Gacko]

Have you considered changing the election ID? I'm not 100% sure, but this could be a reason. Please also try installing them in separate namespaces, just to make it easier to find the root cause.

[Leopol123]

Thanks for your answer.

Election-ID is not the same on my controllers, it seems to build with name of the controller, confirmed in values.yaml:

  # -- Election ID to use for status update, by default it uses the controller name combined with a suffix of 'leader'
  electionID: ""

I just deploy controllers on differents namespaces (ingress-nginx and ingress-nginx2) as suggested, same error. The second controller seems to clean "address" field on an ingressclass that is not managed 😞

[Gacko]

Can you please post the Ingress resources as full YAML as well as the both the Ingress NGINX deployments?

[Leopol123]

Of course.

Ingress-ressources:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app.kubernetes.io/instance: whoami
    app.kubernetes.io/name: whoami
    app.kubernetes.io/version: 1.10.3
  name: whoami
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: whoami.test.lan
    http:
      paths:
      - backend:
          service:
            name: whoami
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app.kubernetes.io/instance: whoami-a
    app.kubernetes.io/name: whoami
    app.kubernetes.io/version: 1.10.3
  name: whoami-a
  namespace: default
spec:
  ingressClassName: domain-a
  rules:
  - host: whoami-a.test.lan
    http:
      paths:
      - backend:
          service:
            name: whoami-a
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    field.cattle.io/publicEndpoints: '[{"addresses":["192.168.1.192"],"port":80,"protocol":"TCP","serviceName":"ingress-nginx:ingress-nginx-controller","allNodes":false},{"addresses":["192.168.1.192"],"port":443,"protocol":"TCP","serviceName":"ingress-nginx:ingress-nginx-controller","allNodes":false}]'
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.12.3
    helm.sh/chart: ingress-nginx-4.12.3
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.12.3
        helm.sh/chart: ingress-nginx-4.12.3
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: LD_PRELOAD
          value: /usr/local/lib/libmimalloc.so
        image: registry.k8s.io/ingress-nginx/controller:v1.12.3@sha256:ac444cd9515af325ba577b596fe4f27a34be1aa330538e8b317ad9d6c8fb94ee
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 90Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          readOnlyRootFilesystem: false
          runAsGroup: 82
          runAsNonRoot: true
          runAsUser: 101
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/local/certificates/
          name: webhook-cert
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: ingress-nginx
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
      - name: webhook-cert
        secret:
          defaultMode: 420
          secretName: ingress-nginx-admission

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    field.cattle.io/publicEndpoints: '[{"addresses":["192.168.1.193"],"port":80,"protocol":"TCP","serviceName":"ingress-nginx2:ingress-nginx-domain-a-controller","allNodes":false},{"addresses":["192.168.1.193"],"port":443,"protocol":"TCP","serviceName":"ingress-nginx2:ingress-nginx-domain-a-controller","allNodes":false}]'
    meta.helm.sh/release-name: ingress-nginx-domain-a
    meta.helm.sh/release-namespace: ingress-nginx2
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx-domain-a
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.12.3
    helm.sh/chart: ingress-nginx-4.12.3
  name: ingress-nginx-domain-a-controller
  namespace: ingress-nginx2
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx-domain-a
      app.kubernetes.io/name: ingress-nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx-domain-a
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.12.3
        helm.sh/chart: ingress-nginx-4.12.3
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-domain-a-controller
        - --election-id=ingress-nginx-domain-a-leader
        - --controller-class=k8s.io/domain-a
        - --ingress-class=domain-a
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-domain-a-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: LD_PRELOAD
          value: /usr/local/lib/libmimalloc.so
        image: registry.k8s.io/ingress-nginx/controller:v1.12.3@sha256:ac444cd9515af325ba577b596fe4f27a34be1aa330538e8b317ad9d6c8fb94ee
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 90Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          readOnlyRootFilesystem: false
          runAsGroup: 82
          runAsNonRoot: true
          runAsUser: 101
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/local/certificates/
          name: webhook-cert
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: ingress-nginx-domain-a
      serviceAccountName: ingress-nginx-domain-a
      terminationGracePeriodSeconds: 300
      volumes:
      - name: webhook-cert
        secret:
          defaultMode: 420
          secretName: ingress-nginx-domain-a-admission

[longwuyuan]

Ignore the URL you used as docs. Try this https://kubernetes.github.io/ingress-nginx/faq/#multiple-controller-in-one-cluster and update results

/remove-kind bug
/kind support

[Leopol123]

Hello, thanks for your answer, we have three days off in France.

I let default ingress controller with default values:

---
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  chart: ingress-nginx
  targetNamespace: ingress-nginx
  repo: https://kubernetes.github.io/ingress-nginx/
  version: 4.12.3
  valuesContent: |
    controller:
      replicaCount: 2
      ingressClassResource:
        enabled: true
        default: true
      service:
        type: LoadBalancer
      config:
        enable-debug: "true"
      logLevel: debug

And the second one in another namespace :

---
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx-2

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: ingress-nginx-domain-a
  namespace: kube-system
spec:
  chart: ingress-nginx
  targetNamespace: ingress-nginx-2
  repo: https://kubernetes.github.io/ingress-nginx/
  version: 4.12.3
  valuesContent: |
    controller:
      replicaCount: 2
      ingressClassResource:
        name: nginx-two
        enabled: true
        default: false
        controllerValue: example.com/ingress-nginx-2
      ingressClass: nginx-two
      ingressClassByName: true
      service:
        type: LoadBalancer
      config:
        enable-debug: "true"
      logLevel: debug

I still have the flip-flap on ipaddress:

kubectl get ingress
NAME       CLASS       HOSTS               ADDRESS                     PORTS   AGE
whoami     nginx       whoami.test.lan     10.152.15.12,10.152.15.13   80      5m7s
whoami-a   nginx-two   whoami-a.test.lan   10.152.15.192               80      5m7s

kubectl get ingress
NAME       CLASS       HOSTS               ADDRESS         PORTS   AGE
whoami     nginx       whoami.test.lan     10.152.15.193   80      5m8s
whoami-a   nginx-two   whoami-a.test.lan   10.152.15.192   80      5m8s

I only have a strange log in nginx-controller-2:

W0610 07:44:45.134632       7 controller.go:336] ignoring ingress whoami in default based on annotation : no object matching key
I0610 07:44:45.134677       7 main.go:107] "successfully validated configuration, accepting" ingress="default/whoami"

It seems controller-2 is looking at ingress default/whoami, but the default controller doesn't log anything for ingress default/whoami-a, juste one line to ignore it:

I0610 07:36:29.192674       6 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="default/whoami-a" error="no object matching key \"nginx-two\" in local store"

Thanks,

[Leopol123]

@longwuyuan , @Gacko, thanks for your help.

We found the issue. We have the default rke-ingress installed by-default as demonset installed on the cluster. I suppress it before running my multi-instances tests, but it was added again probably when I reapply terraform cluster provisionning.

Thanks again for your quick responses ! I close the ticket