NGINX ingress controller - make ingress.class option configurable

[pieterlange]

The newly added "ingress class" option allows cluster administrators to select which set of ingress controllers serves certain Ingress objects by setting an kubernetes.io/ingress.class annotation. Currently the NGINX controller is hardcoded to look for ingresses with an "nginx" value or nonexistent/empty annotation.

If we make this option configurable through the configmap settings, cluster admins would be able to run different sets of ingress controllers and have external network policing on the ingresses. EG one could run some Ingress controllers for "consumer" traffic and another set of ingress controllers that only serve cluster admin/monitoring services. One could also imagine this being used for slow/safe replacements of the ingress controller set itself.

[pieterlange]

I might send a pull request for this, but had to write something down to remind myself as this usecase has been spinning in my head for a while and i just noticed the new option which could be (ab)used for this. Comments welcome.

[aledbf]

Currently the NGINX controller is hardcoded to look for ingresses with an "nginx" value or nonexistent/empty annotation.

The GCE controller also uses this annotation limitations.
Right now is used by Ingress controllers to ignore one or more Ingress rules.
Why not use the --watch-namespace to reduce the scope of the Ingress rules to watch and enable the use of multiple controllers?

[pieterlange]

I didn't know about --watch-namespace :). But i'm not convinced that's a good option for me either, as i don't know which namespaces will exist at the time i deploy the ingress controller. Even if i did, i would have to run a set of controllers for each namespace, or can i watch multiple ones in the same controller? That would get expensive fast, as you'd need to run a cloud provider loadbalancer (ELB in my case) for each set of controllers.

It sounds much more easy to me to be able to select the ingresses i'd want for a particular set of controllers. The kube-native way of doing that would be with labels/annotations and ingress.class sounded right to me :).

[bprashanth]

Yeah you are hinting at the same thing that was initially proposed in the ingress doc: https://github.com/kubernetes/kubernetes/pull/12827/files#diff-41f2bde570ebc813183b7cd0a96a7e04R106, users can request different classes of ingress without bothering what backs each one. You can serve class "gold" and "silver" both from nginx, just that in one case the nginx gets 500m vs 50m cpu limits.

I'm fine doing this through a config map, but I think there is still room for a more solid api level construct. I'd encourage anyone reading this to write up a proposal for kubernetes/kubernetes#30151

[vreon]

This would be cool. I want to run two nginx ingress controllers on my cluster (one behind a public ELB and one behind an internal ELB) and right now they're both looking for the same ingress class.

At first glance this seems like a natural fit for labels rather than an annotation; an ingress controller could be configured with a label selector in the same way that a service watches for labeled pods.

[pieterlange]

@vreon Yes, this is the kind of usecase i imagined. I also like the "gold" and "silver" QoS class usecase. I'm sure there will be more. :)

[roboll]

#1880 makes ingress class configurable

[whereisaaron]

@vreon @pieterlange my use case also: different ingress controllers for internal and external load balancers. Right now you need to either:

• use two different implementations on an ingress controller just so the annotation 'ingress-class' value is different
• use --watch-namespace and have a nginx controller for every namespace, since you can only watch all or one namespace

Matching Ingress resources to Ingress Controllers seems ideal even natural for a label selector, so I too am confused by annotations are preferred? Using a label selector would make Ingress controllers implementation interchangeable without updating all the Ingress resources.

[puja108]

I think this is solved by now. You can set the --ingress-class option like explained in https://github.com/kubernetes/ingress-nginx#running-multiple-ingress-controllers

[pieterlange]

It is solved - i'm using it in production for over a year now.