Ingress stuck in initializing even though the name resolves

[sfss5362]

What kind of request is this (question/bug/enhancement/feature request):

bug

Steps to reproduce (least amount of steps as possible):

rancher ingress Initializing

Other details that may be helpful:

rancher 2.2.0 rc13
k3s 0.3.0-rc1

Although the state of the ingress is inilization, the domain name is accessible after being resolved to the node.

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
• Installation option (single install/HA):

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported):
• Machine type (cloud/VM/metal) and specifications (CPU/memory):
• Kubernetes version (use kubectl version):

root@ubuntu-s-rks-master:~# k3s kubectl  version 
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4-k3s.1", GitCommit:"7f72ee72d6dce73e2d36dbd3402413d146fad01e", GitTreeState:"clean", BuildDate:"2019-03-04T01:05+00:00Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4-k3s.1", GitCommit:"7f72ee72d6dce73e2d36dbd3402413d146fad01e", GitTreeState:"clean", BuildDate:"2019-03-04T01:05+00:00Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

• Docker version (use docker version):

Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        6247962
 Built:             Tue Feb 26 23:56:24 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       6247962
  Built:            Tue Feb 12 22:47:29 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[vkukk]

I'm having indentical issue with Rancher 2.2.8 Gitlab installation running on OVH public cloud (Openstack, does not have any loadbalancer). Following nginx-ingress config is in use:

nginx-ingress:
  enabled: true
  controller:
    hostNetwork: true
    kind: DaemonSet
    service:
      type: ClusterIP
      externalTrafficPolicy: false

EDIT: By identical I meant that ingress is working but is in initalizing state forever.

[vkukk]

For me, setting kind to nodePort and specifying externalIPs for service, solved issue.

[jhughes2112]

Here's how I solved it correctly for my setup. When I went to the node that is running the controller and checked the logs on it, like so:

docker logs -f k8s_nginx-ingress-controller_nginx-ingress-controller-etcetcetc

A new log line popped up every 30 seconds or so that looks like this:

W0120 03:42:59.332901       6 status.go:278] error updating ingress rule: ingresses.networking.k8s.io "rancher-demo" is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot update resource "ingresses/status" in API group "networking.k8s.io" in the namespace "default"

I had an issue where the nginx ingress ClusterRole was incorrectly configured to have less permissions than it needs to update the ingress/status field with the IP address it got from the LoadBalancer, which is why Rancher UI shows it not being initialized. I had to edit the ClusterRole for my ingress-nginx controller and make sure the part that gives permissions to the ingresses/status looks like this:

- apiGroups:
  - "networking.k8s.io"
  resources:
  - ingresses/status
  verbs:
  - update

It's worth noting that my ingress-nginx is in a different namespace than my deployments, so I think other people may not have exactly the same problems, but it's worth checking.

Good luck!

[ChrisMcKee]

Had a similar issue with the Haproxy helm chart (official one not the default one in the library which didnt work)
using the manually added catalogue and the following values worked fine

controller:
  containerPort:
    http: 80
    https: 443
    stat: 1024
  daemonset:
    hostPorts:
      http: 80
      https: 443
      stat: 1024
    useHostPort: true
  defaultTLSSecret:
    enabled: true
    secret: null
  ingressClass: null
  kind: DaemonSet
  livenessProbe:
    failureThreshold: 3
    initialDelaySeconds: 0
    path: /healthz
    periodSeconds: 10
    port: 1042
    scheme: HTTP
    successThreshold: 1
    timeoutSeconds: 1
  logging:
    level: info
  name: haproxycontroller
  publishService:
    enabled: true
    pathOverride: ""
  readinessProbe:
    failureThreshold: 3
    initialDelaySeconds: 0
    path: /healthz
    periodSeconds: 10
    port: 1042
    scheme: HTTP
    successThreshold: 1
    timeoutSeconds: 1
  replicaCount: 2
  resources:
    requests:
      cpu: 100m
      memory: 64Mi
      
  ## Controller Service configuration
  service:
    annotations: {}
    enablePorts:
      http: true
      https: false
      stat: true
    ports:
      http: 80
      https: 443
      stat: 1024
    targetPorts:
      http: http
      https: https
      stat: stat
    type: NodePort

podSecurityPolicy:
  annotations: {}
  enabled: false
rbac:
  create: true
serviceAccount:
  create: true
  
defaultBackend:
  containerPort: 8080
  image:
    pullPolicy: IfNotPresent
    repository: k8s.gcr.io/defaultbackend-amd64
    runAsUser: 65534
    tag: 1.5
  name: default-backend
  replicaCount: 1
  resources:
    requests:
      cpu: 10m
      memory: 16Mi
  service:
    port: 8080
  serviceAccount:
    create: true

Turning off publish service stopped the load balancing state updating; but routing still worked fine.

[Jacq]

I have just experience this, for future visitors now in 2020 in my case I fixed adding the --publish-service arg to the container which was missing in the YAML definition of my running haproxy ingress:
Before:

spec:
        containers:
        - args:
          - --default-ssl-certificate=default/mycontroller-kubernetes-ingress-default-cert
          - --configmap=default/mycontroller-kubernetes-ingress
          - --default-backend-service=default/mycontroller-kubernetes-ingress-default-backend
          - --log=info

After:

spec:
        containers:
        - args:
          - --default-ssl-certificate=default/mycontroller-kubernetes-ingress-default-cert
          - --configmap=default/mycontroller-kubernetes-ingress
          - --default-backend-service=default/mycontroller-kubernetes-ingress-default-backend
          - --log=info
          - --publish-service=default/mycontroller-kubernetes-ingress

The documentation about this paramenter can be found here:
https://www.haproxy.com/documentation/kubernetes/1.4.5/configuration/controller/
https://github.com/haproxytech/kubernetes-ingress/blob/master/documentation/controller.md

I do not know why this was not already set during the installation, I tried both rancher haproxy library app and also general haproxy helm chart. Both showed the same issue of the "initializing" stuck.
Cheers.
Jacq

[KeithTt]

same problem here...