ingress-nginx Loadbalancer targets too many nodes

[lloesche]

I followed the instructions in the Wiki to set up an nginx ingress and certmanager. Everything is working fine so far, however, the Hetzner load balancer lists all of my k3s cluster's nodes as backends, instead of just the three that run the nginx ingress.

This is the load balancer, that was automatically created:

And this is the configuration:

minimi:~ lukas$ kubectl -n ingress-nginx get all
NAME                                 READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-controller-27h78   1/1     Running   0          34m
pod/ingress-nginx-controller-7k4bt   1/1     Running   0          34m
pod/ingress-nginx-controller-p7jm9   1/1     Running   0          34m

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP         PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.43.78.165    lb.eu.fixcloud.io   80:30997/TCP,443:31020/TCP   34m
service/ingress-nginx-controller-admission   ClusterIP      10.43.208.220   <none>              443/TCP                      34m
service/ingress-nginx-controller-metrics     ClusterIP      10.43.62.70     <none>              10254/TCP                    34m

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/ingress-nginx-controller   3         3         3       3            3           kubernetes.io/os=linux   34m

When I check the loadbalancer configuration, the selector looks good:
kubectl -n ingress-nginx get service/ingress-nginx-controller -o yaml

...
spec:
...
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer
...

I can run this selector manually using the API and get exactly the three controller pods returned:

In [31]: pods = api_instance.list_namespaced_pod("ingress-nginx", label_selector="app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx")

In [32]: [p.spec.containers[0].image for p in pods.items]
Out[32]:
['registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd',
 'registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd',
 'registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd']

I feel like this might be a question for hetznercloud/hcloud-cloud-controller-manager but since I followed the tutorial here I figured you might know what I'm doing wrong or where the issue lies.

[vitobotta]

If you install ingress-nginx as a daemonset then it's expected that all the worker nodes would become backends. The weird thing here is if there are nginx pods only on some but not all the nodes. Are you using any taints/tolerations?

I'll convert this to a discussion since it's not an issue with my tool.

[lloesche]

Hi, thank you for looking into this!

That the DaemonSet only gets scheduled on those three nodes is correct, all other nodes have taints that would prevent K8s from scheduling anything on them.

Anyhow, I have now switched from a DaemonSet to a Deployment with 3 replicas and the issue is the same:

minimi:conf lukas$ kubectl -n ingress-nginx get all
NAME                                            READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-controller-6fcfdf46b9-927xc   1/1     Running   0          44m
pod/ingress-nginx-controller-6fcfdf46b9-bl6v8   1/1     Running   0          44m
pod/ingress-nginx-controller-6fcfdf46b9-czclp   1/1     Running   0          44m

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP         PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.43.138.156   lb.eu.fixcloud.io   80:30825/TCP,443:32557/TCP   44m
service/ingress-nginx-controller-admission   ClusterIP      10.43.216.4     <none>              443/TCP                      44m
service/ingress-nginx-controller-metrics     ClusterIP      10.43.56.170    <none>              10254/TCP                    44m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   3/3     3            3           44m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-6fcfdf46b9   3         3         3       44m

You can see the three pods, and when I check the spec for the LoadBalancer service it has the following selector:

spec:
...
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx

Now when I manually query for this selector I get the three pods:

minimi:conf lukas$ kubectl -n ingress-nginx get pods -l app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-6fcfdf46b9-927xc   1/1     Running   0          45m
ingress-nginx-controller-6fcfdf46b9-bl6v8   1/1     Running   0          45m
ingress-nginx-controller-6fcfdf46b9-czclp   1/1     Running   0          45m

However the Hetzner Cloud LoadBalancer still points to all nodes in the cluster.

This is will become an issue, once the cluster grows to more than the 25 (or 100 in the highest tier) nodes, that a Hetzner Cloud LB supports.

As mentioned, I haven't looked into the hcloud-cloud-controller-manager yet, since I followed the ingress-nginx/certmanager tutorial in this Wiki. I recon I'll have to dig for the answer there. But if you have any idea what I might be doing wrong, any help is much appreciated!

[lloesche]

After raising this issue in the hcloud-cloud-controller-manager repo it turns out this is a known shortcoming of the Kubernetes Cloud Controller Manager for Hetzner Cloud.

[lloesche]

For reference, this issue has been resolved via hetznercloud/hcloud-cloud-controller-manager#514
which introduces a new label load-balancer.hetzner.cloud/node-selector.