Drop traefik and go to nginx controller

[ibuildthecloud]

Nginx controller should be supporting armv7 again soon, so we should switch to nginx controller. Nginx controller is more popular controller and a bit more compatible. For example we can't run SDPY behind traefik which is required for k8s port forwarding.

[warmchang]

BTW, which one (kubernetes community or Nginx official)? I guess it should be the previous one.

[sandys]

i vote for the haproxy ingress which also injects proxy protocol headers so that source ip is not lost.
https://www.haproxy.com/blog/dissecting-the-haproxy-kubernetes-ingress-controller/

also the ingress controller is officially supported by haproxy inc. https://github.com/haproxytech/kubernetes-ingress

[nickbp]

In a Raspberry Pi 4 cluster running Raspbian, I've disabled the default K3s traefik ingress controller and am instead using ingress-nginx using their stock ARM7 image provided here. Works fine, even have it fetching certificate secrets generated by an ARM7 build of cert-manager. I am using it in host networking mode since I just have a NAT port forwarding HTTP/HTTPS into one of the nodes of the cluster.

[thewilli]

Why is there a default ingress controller included at all? Should be much easier to not just not care about this (not having to maintain this), and point users to the appropriate YAML to kubectl apply or Helm Charts to install. Doesn't take much time, and absolute beginners might still use NodePort services.

[sandys]

@thewilli while i understand your opinion, I would like to politely and strongly oppose this.

The one thing that k3s must not become is k8s. Kubernetes is a brilliant piece of infrastructure, but there is a huge learning curve to set it up, precisely because it is not batteries included (and unopinionated).

A lot of us (and me included) would like to see a highly opinionated distro that i can run on my laptop and cloud. The goal is to achieve a configuration simple enough that most people using docker-compose.yml move to k3s.
I dont want to think about ingress, network plugins, dns, etc. I like haproxy a lot (and personally think it is superior to traefik and nginx). However, I will happily take it if I get a zero configuration kubernetes in exchange.

I want a kuberentes that gets out of the way. Ingress is a very important (and painful) step towards it.

[kamilgregorczyk]

If anyone of you is wondering on how to do it by yourself then:

1. Add --no-deploy traefik to k3s.service that systemd uses
2. Add k8s repo (on helm 3) helm repo add stable https://kubernetes-charts.storage.googleapis.com/
3. Run helm install nginx stable/nginx-ingress --namespace kube-system --set rbac.create=true,controller.image.repository="quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm",defaultBackend.image.repository="k8s.gcr.io/defaultbackend-arm"

I also replaced servicelb with metalb to have better performance

[sandys]

there is an active pull request now, and i would like to reiterate my request to not use nginx and go for haproxy.

First, the haproxy ingress is actively supported by the org - https://github.com/haproxytech/kubernetes-ingress

Secondly, haproxy officially supports proxy protocol injection (and not just supporting upstream proxy protocol tags), which is the only widespread way of preserving source ip injection.

this has been one of the sources of a huge, long-term issue in Docker Swarm - moby/moby#25526

the haproxy dataplane api is also built in - https://www.haproxy.com/blog/new-haproxy-data-plane-api/ . Similar functionality on nginx side is commercial only - https://www.nginx.com/free-trial-request-nginx-controller/

you can read more about the haproxy ingress here - https://www.haproxy.com/blog/dissecting-the-haproxy-kubernetes-ingress-controller/

[pasikarkkainen]

There's also the https://github.com/jcmoraisjr/haproxy-ingress which has been around for a longer time compared to the haproxytech kubernetes-ingress.

[davidnuzik]

Docs issue: #1480

[orange888]

Change the default ingress, but can you let user override it using k3s server command?
It is done to ensure existing/pro-traefik users can continue using traefik until they decide to migrate.

[sandys]

Change the default ingress, but can you let user override it using k3s server command?
It is done to ensure existing/pro-traefik users can continue using traefik until they decide to migrate.

Following up on this - it will be awesome if the doc includes how to switch to haproxy or traefik.
While it may not be the default, it will really help the rest of us.

[cjellick]

We jumped the gun on this one and decided to revert the change:
#1519

Our next release will still have Traefik as the default ingress controller.

[cjellick]

Going to keep this open for now, but our ultimately solution likely will not be "drop Traefik and go to Nginx." It'll probably involve supporting more options.

[davidnuzik]

Moving out of "To Test"

[ibuildthecloud]

Also traefik does support SPDY now which helps.

[billimek]

It will be great when traefik 2.x once again supports ingress configuration via annotations and not just their CRD.

[dxlr8r]

Going to keep this open for now, but our ultimately solution likely will not be "drop Traefik and go to Nginx." It'll probably involve supporting more options.

Not saying it's the wrong decision, but it would be interesting to hear why you continue with Traefik 1.x.

[davidnuzik]

After input from the community and @ibuildthecloud we're sticking with Traefik.

[joakimr-axis]

To me, it seems Traefik is still hogging port 80 (which ruins things for the native web server running on my node that should not have to be affected) even if I try the different tricks with adding a custom Traefik config and such. Am I doing things wrong and is there now a possibility to reconfigure the ports for Traefik in k3s? To me this is a dealbreaker in using Traefik with k3s for my current use case.

[imba-tjd]

@joakimr-axis You can use --disable=traefik

[joakimr-axis]

@joakimr-axis You can use --disable=traefik

That I do already (to avoid the port 80 issue at the cost of not having the ingress functionality, but now I need the latter), but will it give me nginx?
Or is the trick to run --disable=traefik and then manually install nginx-ingress (this I am experimenting with now but stuck with webhooks not working followed by using helm instrad and then stuck on helm apparently not working properly when behind a corporate proxy...).

[imba-tjd]

I don't have much experience, either. As I understand, with --disable=traefik, the 80 and 443 can be used by normal web servers, but in my tests that won't take effect before my os restart. I don't know how to use nginx-ingress, but I think it's correct to setup a new ingress by yourself.

[northlander]

@joakimr-axis manually installing another ingress won't make any difference. Either use a different node port for HTTP (8080 or something) or setup a something like metallb and aquire a separate IP for the ingress (by disable servicelb and set service type to loadbalancer) which does not affect the host server at all. The bundled Traefik can be configured. Anyhow, either way I usually just disable the bundled traefik and install a separate ingress controller using helm to get control over the custom settings. Typically Traefik 2.x but installing Nginx is pretty much the same thing.

[biels]

Anyone knows where I can find good / official instructions on how to disable traefik and deploy ingress-nginx in 1.18.6+k3s1? Is it supported?

[dkeightley]

@biels this might be a related write up for what you want. In short, disable traefik with the --no-deploy-traefik k3s argument, and follow your preferred option to install ingress-nginx. It's possible to automate the ingress-nginx helm chart install with a HelmChart or k8s manifest as well, once in place k3s will install it for you.